Disks on Opensource internals

How ClickHouse Spreads Parts Across Disks: the Storage Policy

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: A storage policy is a list of volumes, each holding one or more disks. Writing a part calls reserve() down that ladder: the policy tries volumes in priority order, a volume tries its disks (round-robin or least-used), and a disk does the actual accounting. A volume can refuse to hold parts larger than max_data_part_size, which is how a 100 GiB part skips the NVMe tier and lands on the cheap one.

A MergeTree table has to answer two questions continuously: where does the next part go? and when does an old part move to cheaper storage? Answering them well means the hot data is on fast local disks and the cold data is on cheap object storage, without filling a disk to 100%.

The naive design — “put every part on the disk with the most free space” — fails in both directions. It ignores the fact that the tiers have different costs, and it makes placement depend on a moment-to-moment measurement, so the same part lands on different disks depending on when it was written.

ClickHouse’s design is a three-level ladder with an explicit priority order, and the interesting decisions are all about refusing work rather than accepting it.

The three levels

 text
 
 
 StoragePolicy "hot_to_cold"
 |
 +-- Volume "hot" volume_priority = 1
 | +-- Disk "nvme1"
 | +-- Disk "nvme2"
 |
 +-- Volume "cold" volume_priority = 2
 +-- Disk "s3"

A policy is an ordered list of volumes. It is what a table names in SETTINGS storage_policy = '...'.
A volume is a set of disks that are interchangeable for a given part — a JBOD. Its volume_priority says how early it is tried.
A disk is where bytes actually land: a local directory, an S3 bucket, an encrypted wrapper, a cache layer.

The policy’s volumes are sorted by priority, and the priority values are validated to be a dense range:

 cpp
 
 
// src/Disks/StoragePolicy.cpp:85
 if (!volume_priorities.empty())
 {
 /// Check that priority values cover the range from 1 to N (lowest explicit priority)
 if (*volume_priorities.begin() != 1 || *volume_priorities.rbegin() != volume_priorities.size())
 throw Exception(
 ErrorCodes::INVALID_CONFIG_PARAMETER,
 "volume_priority values must cover the range from 1 to N (lowest priority specified) without gaps");

 ::stableSort(
 volumes.begin(), volumes.end(),
 [](const VolumePtr a, const VolumePtr b) { return a->volume_priority < b->volume_priority; });
 }

Two rules hide here. Priorities must be unique (checked when the volumes are built), and they must form a gapless range 1..N — the priority is used as an index (“start looking for space at volume index k”), so a gap would make that index meaningless.

Reserving space: the ladder

Writing a part does not write to a disk directly. It reserves space first, and the reservation is what makes concurrent inserts safe: two inserts that both see “10 GiB free” must not both plan to write 10 GiB.

The top of the ladder is the policy, and it walks volumes in priority order:

 cpp
 
 
// src/Disks/StoragePolicy.cpp:269
ReservationPtrOrError StoragePolicy::reserve(UInt64 bytes, size_t min_volume_index) const
{
 bool all_volumes_read_only = true;
 for (size_t i = min_volume_index; i < volumes.size(); ++i)
 {
 const auto & volume = volumes[i];
 if (volume->isReadOnly())
 continue;
 all_volumes_read_only = false;
 auto reservation = volume->reserve(bytes);
 if (reservation)
 return reservation;
 }

 if (all_volumes_read_only)
 return std::unexpected(
 ReservationError(ErrorCodes::READONLY, "Could not reserve {} because all disk volumes are readonly", ReadableSize(bytes)));

 LOG_TRACE(log, "Could not reserve {} from volume index {}, total volumes {}", ReadableSize(bytes), min_volume_index, volumes.size());
 return std::unexpected(ReservationError(ErrorCodes::NOT_ENOUGH_SPACE, "Cannot reserve {}, not enough space", ReadableSize(bytes)));
}

Note min_volume_index: the caller can say “start at volume 2”. That is what the part-mover uses to push data down the ladder — see below.

The middle of the ladder is the volume, and this is where the interesting refusal lives:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:127
ReservationPtr VolumeJBOD::reserveImpl(UInt64 bytes, const std::optional<ReservationConstraints> & constraints)
{
 /// This volume can not store data which size is greater than `max_data_part_size`
 /// to ensure that parts of size greater than that go to another volume(s).
 if (max_data_part_size != 0 && bytes > max_data_part_size)
 return {};

That single line is the tiering mechanism. A volume configured with max_data_part_size_bytes = 10737418240 (10 GiB) will refuse any reservation for a bigger part, so the policy falls through to the next volume. Small parts land on fast disks; the occasional huge part (a big OPTIMIZE, a large mutation result) lands on the cheap tier.

The volume then picks a disk, by one of two algorithms:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:144
 switch (load_balancing)
 {
 case VolumeLoadBalancing::ROUND_ROBIN:
 {
 size_t disks_num = disks.size();
 for (size_t i = 0; i < disks_num; ++i)
 {
 size_t start_from = last_used.fetch_add(1u, std::memory_order_acq_rel);
 size_t index = start_from % disks_num;

 ReservationPtr reservation = try_reserve(disks[index]);
 if (reservation)
 return reservation;
 }
 return {};
 }

Round-robin is a single atomic counter — no lock, no filesystem query. It spreads parts evenly regardless of how full the disks are, which is what you want when the disks are identical.

LEAST_USED is the alternative, and it is deliberately cached:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:160
 case VolumeLoadBalancing::LEAST_USED:
 {
 std::lock_guard lock(mutex);

 ReservationPtr reservation;
 for (size_t i = 0; i < disks.size() && !reservation; ++i)
 {
 if (i == 0 && (!least_used_ttl_ms || least_used_update_watch.elapsedMilliseconds() >= least_used_ttl_ms))
 {
 disks_by_size = LeastUsedDisksQueue(disks.begin(), disks.end());
 least_used_update_watch.restart();

 DiskWithSize disk = disks_by_size.top();
 reservation = try_reserve(disk.disk);
 }

The ordering is by free space, and read-only disks always sort last:

 cpp
 
 
// src/Disks/VolumeJBOD.h:89
 bool operator<(const DiskWithSize & rhs) const
 {
 if (disk->isReadOnly())
 return true;
 if (rhs.disk->isReadOnly())
 return false;
 return free_size < rhs.free_size;
 }

The TTL (60 seconds by default) is the important part:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:81
 least_used_ttl_ms = config.getUInt64(config_prefix + ".least_used_ttl_ms", 60'000);

Querying free space means a statvfs on every disk, on every insert. The cache re-uses the answer for a minute and then refreshes it. In between, the “free space” of a disk is updated by the reservations themselves:

 cpp
 
 
// src/Disks/VolumeJBOD.h:98
 ReservationPtr reserve(UInt64 bytes)
 {
 ReservationPtr reservation = disk->reserve(bytes);
 if (!reservation)
 return {};

 /// Not just subtract bytes, but update the value,
 /// since some reservations may be done directly via IDisk, or not by ClickHouse.
 free_size = reservation->getUnreservedSpace();
 return reservation;
 }

getUnreservedSpace() is the disk’s own accounting, so the cached number tracks ClickHouse’s own reservations exactly, and drifts only from reservations made outside ClickHouse — which is why the periodic refresh exists.

The bottom of the ladder: what a reservation actually is

At the disk level, a reservation is bytes subtracted from a counter, under a mutex:

 cpp
 
 
// src/Disks/DiskLocal.cpp:202
std::optional<UInt64> DiskLocal::tryReserve(UInt64 bytes, const std::optional<ReservationConstraints> & constraints)
{
 std::lock_guard lock(DiskLocal::reservation_mutex);

 auto available_space = getAvailableSpace();

 UInt64 unreserved_space = available_space
 ? *available_space - std::min(*available_space, reserved_bytes)
 : std::numeric_limits<UInt64>::max();

and the commit is the counter increment:

 cpp
 
 
// src/Disks/DiskLocal.cpp:274
 ++reservation_count;
 reserved_bytes += bytes;
 return {unreserved_space - bytes};

The reservation object is a token: it holds the disk’s bytes until it is destroyed, and the storage policy hands the token to the part, which holds it for the lifetime of the part. That is how the space is held while the part is being written — and it is why the free space reported by system.disks is lower than the free space reported by df.

ReservationConstraints adds a second, policy-level condition: reserve only if enough space would remain afterwards.

 cpp
 
 
// src/Disks/DiskLocal.cpp:223
 UInt64 free_bytes_after = unreserved_space - bytes;

 /// Check min_bytes constraint
 if (constraints->min_bytes > 0 && free_bytes_after < constraints->min_bytes)
 {
 LOG_TRACE(logger, "Could not reserve {} ({} bytes) on disk {}. Free space after reservation {} bytes ({}) would be less than min_bytes {} bytes ({})",
 ReadableSize(bytes), bytes, backQuote(name), free_bytes_after, ReadableSize(free_bytes_after), constraints->min_bytes, ReadableSize(constraints->min_bytes));
 return {};
 }

This is how ClickHouse refuses to fill a disk completely. The insert path turns it into a user-visible error:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataWriter.cpp:1005
 if (!is_system_database && (min_bytes_to_perform_insert > 0 || min_ratio_to_perform_insert > 0.0f))
 {
 ReservationConstraints constraints(min_bytes_to_perform_insert, min_ratio_to_perform_insert);

 /// Try to reserve space on volume with constraints
 reservation = volume->reserve(expected_size, constraints);

 if (!reservation)
 {
 throw Exception(
 ErrorCodes::NOT_ENOUGH_SPACE,
 "Could not perform insert. "
 "None of the disks in volume '{}' have enough free space to meet the configured threshold. "

so the insert fails before writing anything, rather than filling the disk to zero and failing later.

TTL rules beat policy rules

Before the policy ladder is consulted, the TTL move rules get a chance. If a table has TTL ts + INTERVAL 30 DAY TO VOLUME 'cold', and the inserted data is already expired, the part goes straight to the cold volume:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:12185
 auto move_ttl_entry = selectTTLDescriptionForTTLInfos(metadata_snapshot->getMoveTTLs(), ttl_infos.moves_ttl, time_of_move, true);

 if (move_ttl_entry)
 {
 LOG_TRACE(log, "Trying to reserve {} to apply a TTL rule. Will try to reserve in the destination", ReadableSize(expected_size));
 SpacePtr destination_ptr = getDestinationForMoveTTL(*move_ttl_entry);
 bool perform_ttl_move_on_insert = is_insert && destination_ptr && shouldPerformTTLMoveOnInsert(destination_ptr);

and if the destination cannot take it, the insert falls back to the policy ladder. Note the perform_ttl_move_on_insert flag: a volume can opt out of having expired data inserted into it, in which case the data is written normally and moved later by the TTL merge.

Moving parts down the ladder

Placement alone is not enough: data written to the fast tier today must be moved to the cold tier tomorrow, when the TTL says so. The mover picks parts to move by asking the policy for space starting after the source disk’s volume:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartsMover.cpp:196
 for (auto && move : need_to_move)
 {
 auto min_volume_index = policy->getVolumeIndexByDiskName(move.first->getName()) + 1;
 for (auto && part : move.second.getAccumulatedParts())
 {
 auto reservation = policy->reserve(part->getBytesOnDisk(), min_volume_index).value_or(nullptr);
 if (!reservation)
 {
 /// Next parts to move from this disk has greater size and same min volume index.
 /// There are no space for them.
 /// But it can be possible to move data from other disks.
 break;
 }

min_volume_index = source volume index + 1 means: never move data to the volume it is already in, or to a faster one. Data flows in one direction only — down the priority ladder — which keeps the mover from thrashing.

The move itself is a clone plus an atomic rename, as covered in the article on part publication. What matters here is that it is not atomic across disks, and that recovery is defined to handle that: a crash mid-move leaves two copies, and the loader keeps one and marks the other outdated.

You can also move parts by hand:

 sql
 
 
-- Move every part of partition 2024_05 to the 's3' disk of the 'cold' volume.
ALTER TABLE events MOVE PARTITION 2024_05 TO DISK 's3';

-- Or to a whole volume.
ALTER TABLE events MOVE PARTITION 2024_05 TO VOLUME 'cold';

Inspecting the policy

The policy is fully introspectable:

 sql
 
 
SELECT
 policy_name,
 volume_name,
 volume_priority,
 disks,
 max_data_part_size,
 load_balancing,
 perform_ttl_move_on_insert
FROM system.storage_policies
ORDER BY policy_name, volume_priority;

and system.disks shows, per disk, the free space, the total space and the keep_free_space floor configured for it.

The `move_factor`, and what it does now

Every policy has a move_factor, described in the system table as “when the amount of available space gets lower than this factor, data automatically starts to move on the next volume if any”:

 cpp
 
 
// src/Disks/StoragePolicy.cpp:113
 const double default_move_factor = volumes.size() > 1 ? 0.1 : 0.0;
 move_factor = config.getDouble(config_prefix + ".move_factor", default_move_factor);

It is worth being precise about this: in the current code the factor is reported (system.storage_policies.move_factor) but no longer drives an automatic background move. Data moves down the tiers because of TTL rules, or because you ask it to. The default of 0.1 for a multi-volume policy and 0.0 for a single-volume one is a leftover from the era when the factor was the trigger for automatic rebalancing.

Trade-offs

What the design buys:

Deterministic placement. A part’s volume is decided by the priority ladder and the size of the part, not by a transient free-space reading.
Big parts skip the fast tier. max_data_part_size is a one-line refusal that buys a whole tiering policy.
Reservations, not hope. Concurrent inserts cannot overcommit a disk, and the “keep N bytes free” constraint makes the refusal explicit and early.
One-direction data flow. The mover refuses to move data up the ladder.

What it costs:

Free space is read at the wrong granularity. least_used mode caches statvfs results for up to a minute, so placement can be slightly wrong. Round-robin avoids the question by ignoring free space entirely.
Reservations leak into the numbers. system.disks.free_space and df will disagree, and a long merge can hold a reservation for a long time.
The tier boundary is a cliff, not a slope. A part of 11 GiB goes entirely to the cold tier because the hot tier’s limit is 10 GiB; there is no “mostly hot” placement.
Moving is expensive and not atomic across disks. A move is a full copy plus a rename, and a crash in the middle leaves duplicates to reconcile.

Wrapping up

The one thing to remember: the placement rule is “first volume that accepts the reservation”, and a volume accepts or refuses based on the size of the part and the free space on its disks. Everything else — round-robin, least-used, TTL rules — is a detail of how a volume picks among its own disks.

The part people get wrong: a multi-volume policy does not spread data by usage. It spreads it by part size and by TTL. A table with two volumes and no TTL rule keeps all its new parts on volume 1 forever.

What to go do about it: look at system.storage_policies and check that max_data_part_size on your fast volume is smaller than your typical part size but larger than your typical merge result — otherwise big merges will silently land on your slow disks.

ClickHouse Encrypts Files at Rest with a 64-Byte Header

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: Every encrypted file starts with a 64-byte header holding the signature ENC, a version, the algorithm, a SipHash fingerprint of the key and a random 16-byte IV. The payload is AES-CTR ciphertext, and the CTR counter is derived from the plaintext offset — so the plaintext and ciphertext offsets stay exactly 64 bytes apart, which is what makes random access, appending, and per-file key rotation possible.

Encrypting data at rest looks like a solved problem until you try to do it inside a database. A database does not have one big blob to encrypt; it has millions of small files that it seeks around in, appends to, rewrites, and copies between disks. Each of those operations has to keep working.

That rules out most “obvious” designs. Encrypt the whole file with CBC or GCM and you lose random access: to read byte 1000 you must decrypt bytes 0..1000. Use one key for everything and you cannot rotate the key without rewriting the dataset. Store the key id in a sidecar file and you now have a sidecar to keep consistent with the data it describes.

ClickHouse’s design solves all three with a small, self-describing header prepended to each file, and a cipher mode chosen specifically for its appendability.

Why CTR, and not CBC or GCM

The choice is documented in the source, and it is the load-bearing decision of the whole design:

 cpp
 
 
// src/IO/FileEncryptionCommon.h:17
/// Encryption algorithm.
/// We chose to use CTR cipther algorithms because they have the following features which are important for us:
/// - No right padding, so we can append encrypted files without deciphering;
/// - One byte is always ciphered as one byte, so we get random access to encrypted files easily.
enum class Algorithm : uint8_t
{
 AES_128_CTR, /// Size of key is 16 bytes.
 AES_192_CTR, /// Size of key is 24 bytes.
 AES_256_CTR, /// Size of key is 32 bytes.
 MAX
};

Both properties matter enormously for a database:

No padding means ciphertext and plaintext have the same length, so getFileSize is just “size minus 64”, and a part’s byte count on disk is predictable.
Stream cipher with a counter means byte N of the plaintext is encrypted with counter IV + N/16. You can therefore seek to any offset and decrypt from there, without touching the bytes before it.

The cost of CTR is the one you always pay: it is malleable. An attacker who can write to your disk can flip bits in the ciphertext and flip the same bits in the plaintext. ClickHouse does not use an authenticated cipher here — there is no HMAC, no AEAD tag. What protects the data in practice is the header itself: the fingerprint inside it must match a configured key and the signature must be ENC, so a corrupted or tampered file usually fails loudly rather than decrypting to garbage.

The header

The header layout is a fixed 64-byte structure at the very beginning of every encrypted file:

 cpp
 
 
// src/IO/FileEncryptionCommon.h:106
/// File header which is stored at the beginning of encrypted files.
///
/// The format of that header is following:
/// +--------+------+--------------------------------------------------------------------------+
/// | offset | size | description |
/// +--------+------+--------------------------------------------------------------------------+
/// | 0 | 3 | 'E', 'N', 'C' (file's signature) |
/// | 3 | 2 | version of this header (1..2) |
/// | 5 | 2 | encryption algorithm (0..2, 0=AES_128_CTR, 1=AES_192_CTR, 2=AES_256_CTR) |
/// | 7 | 16 | fingerprint of encryption key (SipHash) |
/// | 23 | 16 | initialization vector (randomly generated) |
/// | 39 | 25 | reserved for future use |
/// +--------+------+--------------------------------------------------------------------------+

 cpp
 
 
// src/IO/FileEncryptionCommon.h:120
struct Header
{
 /// Versions:
 /// 1 - Initial version
 /// 2 - The header of an encrypted file contains the fingerprint of a used encryption key instead of a pair {key_id, very_small_hash(key)}.
 /// The header is always stored in little endian.
 static constexpr const UInt16 kCurrentVersion = 2;

 UInt16 version = kCurrentVersion;

 /// Encryption algorithm.
 Algorithm algorithm = Algorithm::AES_128_CTR;

 /// Fingerprint of a key.
 UInt128 key_fingerprint = 0;

 InitVector init_vector;

 /// The size of this header in bytes, including reserved bytes.
 static constexpr const size_t kSize = 64;

 void read(ReadBuffer & in);
 void write(WriteBuffer & out) const;
};

Three things are packed into those 64 bytes: the 3-byte signature ENC (a magic that lets the reader reject a non-encrypted file instead of decrypting garbage), the 16-byte key fingerprint (which is what makes per-file key rotation possible — the file names the key that encrypted it), and the 16-byte random IV. The remaining 25 bytes are reserved and written as zeros.

The key fingerprint: rotation without rewriting

The server keeps every configured key in a map keyed by its fingerprint:

 cpp
 
 
// src/Disks/DiskEncrypted.cpp:259
 for (const auto & [key_id, key_entry] : keys_by_id)
 {
 const auto key = key_entry.plain;
 auto fingerprint = calculateKeyFingerprint(key);
 res->all_keys[fingerprint] = key;

 /// Version 1 used key fingerprints based on the key id.
 /// We have to add such fingerprints to the map too to support reading files encrypted by version 1.
 auto v1_fingerprint = calculateV1KeyFingerprint(key, key_id);
 res->all_keys[v1_fingerprint] = key;
 }

The fingerprint is a keyed SipHash-128, not a bare hash:

 cpp
 
 
// src/IO/FileEncryptionCommon.cpp:467
UInt128 calculateKeyFingerprint(const String & key)
{
 const UInt64 seed0 = 0x4368456E63727970ULL; // ChEncryp
 const UInt64 seed1 = 0x7465644469736B46ULL; // tedDiskF
 return sipHash128Keyed(seed0, seed1, key.data(), key.size());
}

The seed is a fixed constant, so this is a keyed hash in the SipHash sense (a random-looking key chosen once), not a secret. Its job is to be a compact identifier that is practically collision-free across a handful of keys, not to be a KDF. Note the v1 fingerprints are added to the same map, so a server upgraded from the old header format can still read files written by the old version — the header version and the fingerprint scheme were versioned together.

When a file is opened, the fingerprint in its header selects the key:

 cpp
 
 
// src/Disks/DiskEncryptedTransaction.cpp:42
String DiskEncryptedSettings::findKeyByFingerprint(UInt128 key_fingerprint, const String & path_for_logs) const
{
 auto it = all_keys.find(key_fingerprint);
 if (it == all_keys.end())
 {
 throw Exception(
 ErrorCodes::DATA_ENCRYPTION_ERROR,
 "Not found an encryption key required to decipher file {}",
 quoteString(path_for_logs));
 }
 return it->second;
}

So the rotation procedure is: add the new key to the configuration, mark it current_key, and new writes use it. Old files keep working because they name the old key in their own headers. Nothing needs to be rewritten, and the removal of an old key is a deliberate, auditable act — the moment you remove it, every file that used it becomes unreadable.

The offset trick: plaintext and ciphertext live in different coordinates

Here is the heart of the design. The file on disk is:

 text
 
 
 +--------------------------------------------------+
 0 | header (64 bytes) | file offset 0
 +--------------------------------------------------+
64 | ciphertext | file offset 64
 | ciphertext = AES-CTR(key, IV + plaintext_off/16) |
 +--------------------------------------------------+

The encryptor’s counter is computed from the plaintext offset, not from the file offset:

 cpp
 
 
// src/IO/FileEncryptionCommon.cpp:299
 auto current_iv = (init_vector + blocks(offset)).toString();

where offset is the plaintext offset and blocks(pos) = pos / 16.

That single decision buys three things: appending (new plaintext at offset N is encrypted with counter IV + N/16, so nothing is re-encrypted), random access (to read plaintext at offset N, seek the file to N + 64 and start the counter at IV + N/16 — the bytes before N are never touched), and a uniform offset space (every layer above the cipher — the disk abstraction, getFileSize, truncateFile, the read-until-position machinery — works in plaintext offsets, and only the innermost buffer translates).

The translation happens in one place, on the seek path:

 cpp
 
 
// src/IO/ReadBufferFromEncryptedFile.cpp:255
void ReadBufferFromEncryptedFile::performSeekAndSetReadUntilPosition()
{
 std::optional<off_t> in_position;
 std::optional<off_t> new_in_position;
 if (need_seek)
 {
 in_position = in->getPosition();
 new_in_position = offset + FileEncryption::Header::kSize;
 chassert(*new_in_position >= static_cast<off_t>(FileEncryption::Header::kSize));
 }

 auto do_seek = [&]
 {
 chassert(new_in_position.has_value());
 offset = in->seek(*new_in_position, SEEK_SET) - FileEncryption::Header::kSize;
 chassert(offset >= 0);
 need_seek = false;
 };

offset is the plaintext position; in->seek(offset + 64) is the file position. The difference is the header, and it is re-derived after every seek rather than being tracked separately, so the two can never drift.

Above the cipher, everything is in plaintext units. The disk reports a plaintext size by subtracting the header:

 cpp
 
 
// src/Disks/DiskEncrypted.cpp:527
size_t DiskEncrypted::getFileSize(const String & path) const
{
 auto wrapped_path = wrappedPath(path);
 size_t size = delegate->getFileSize(wrapped_path);
 return size > FileEncryption::Header::kSize ? (size - FileEncryption::Header::kSize) : 0;
}

Writing: new IV on rewrite, same IV on append

The write path is where the header is born. DiskEncryptedTransaction picks between two modes:

 cpp
 
 
// src/Disks/DiskEncryptedTransaction.cpp:89
 auto wrapped_path = wrappedPath(path);
 FileEncryption::Header header;
 String key;
 UInt64 old_file_size = 0;
 if (mode == WriteMode::Append && delegate_disk->existsFile(wrapped_path))
 {
 size_t size = delegate_disk->getFileSize(wrapped_path);
 old_file_size = size > FileEncryption::Header::kSize ? (size - FileEncryption::Header::kSize) : 0;
 if (old_file_size)
 {
 /// Append mode: we continue to use the same header.
 auto read_buffer = delegate_disk->readFile(wrapped_path, getReadSettings().adjustBufferSize(FileEncryption::Header::kSize));
 header = readHeader(*read_buffer);
 key = current_settings.findKeyByFingerprint(header.key_fingerprint, path);
 }
 }
 if (!old_file_size)
 {
 /// Rewrite mode: we generate a new header.
 header.algorithm = current_settings.current_algorithm;
 key = current_settings.current_key;
 header.key_fingerprint = current_settings.current_key_fingerprint;
 header.init_vector = FileEncryption::InitVector::random();
 }

Note the asymmetry. Append reuses the existing header — same IV, same key — and sets the encryptor’s offset to the old plaintext size, so the CTR stream continues exactly where it left off. Rewrite generates a fresh random IV and uses the current key. Append mode is also key-aware: it reads the old header to find out which key encrypted the file and keeps using it, so appending to a file written under an old key does not silently re-key half the file.

The header is not written eagerly; it is flushed lazily, on the first buffer of actual data:

 cpp
 
 
// src/IO/WriteBufferFromEncryptedFile.cpp:19
 : WriteBufferDecorator<WriteBufferFromFileBase>(std::move(out_), adaptiveBufferInitialSize(use_adaptive_buffer_size_, adaptive_buffer_initial_size, buffer_size_), nullptr, 0)
 , header(header_)
 , flush_header(!old_file_size)
 , encryptor(header.algorithm, key_, header.init_vector)
{
 enableAdaptiveBufferGrowth(use_adaptive_buffer_size_, buffer_size_);
 encryptor.setOffset(old_file_size);
}

 cpp
 
 
// src/IO/WriteBufferFromEncryptedFile.cpp:53
void WriteBufferFromEncryptedFile::nextImpl()
{
 if (!offset())
 return;

 if (flush_header)
 {
 header.write(*out);
 flush_header = false;
 }

 encryptor.encrypt(working_buffer.begin(), offset(), *out);

 growAdaptiveBufferAfterFlush();
}

flush_header is false in append mode (the header is already in the file) and true otherwise; the header is emitted before the first ciphertext block, so the file always starts with exactly 64 bytes of header.

Copying ciphertext without decrypting it

A MergeTree replica, a BACKUP, or a move between disks will copy files around. If the source and destination are both encrypted with the same keys, there is no reason to decrypt and re-encrypt: the ciphertext can be copied byte for byte, and DiskEncrypted::copyFile does exactly that when from_settings->all_keys == to_settings->all_keys — the whole key set, not just the current key. If the two disks know different key sets, the copy falls through to the decrypting path, because the destination may need to re-encrypt under a key the source does not have.

The header cache: skipping a read per open

Reading a file requires reading its 64-byte header first, which for object storage means an extra ranged GET. EncryptionHeaderCache keeps those bytes in memory, keyed by a hash of the storage path.

The interesting part is when it is allowed to be used:

 cpp
 
 
// src/Disks/DiskEncrypted.cpp:514
 /// Only cache encryption headers when the backend assigns a fresh blob path to every write
 /// (`areBlobPathsRandom`): then a rewrite / replace / rename never rebinds an existing path to
 /// different ciphertext, so the cache can never serve a stale header and needs no invalidation.
 /// Deterministic-path backends (plain / plain-rewritable, local, web) reuse the path on rewrite
 /// and are excluded.
 if (delegate->areBlobPathsRandom())
 {
 if (auto global_context = Context::getGlobalContextInstance())
 if (auto cache = global_context->getEncryptionHeaderCache())
 pipeline.needEncryptionHeaderCache(std::move(cache));
 }

This is a self-invalidating cache by construction. On S3-style object storage every write produces a new blob path, so a cached header can only go stale if the path is reused — which it never is. On a local disk a rewrite reuses the path, so those backends are excluded.

The cache stores opaque bytes rather than parsed headers, so it does not depend on OpenSSL, and its weight function charges the payload plus a fixed 256-byte per-entry overhead, so the byte budget bounds real memory rather than just the header payload.

Configuring it

An encrypted disk is a wrapper around another disk. The key length must match the algorithm exactly — a 32-byte key for aes_256_ctr, a 16-byte key for aes_128_ctr — and the check is strict:

 text
 
 
<storage_configuration>
 <disks>
 <encrypted_disk>
 <type>encrypted</type>
 <disk>s3_disk</disk>
 <path>encrypted/</path>
 <key>0123456789abcdef0123456789abcdef</key>
 <current_key_id>0</current_key_id>
 <algorithm>aes_256_ctr</algorithm>
 </encrypted_disk>
 </disks>
</storage_configuration>

Trade-offs

What the design buys:

Random access. Reading a byte range never decrypts the bytes before it.
Appendability. Appends continue the CTR stream; no rewrite, no re-encryption of the prefix.
Per-file key rotation. The header names the key, so old and new keys coexist in one dataset and rotation is a configuration change.
Self-description. The file knows its own algorithm and key, so there is no manifest that can disagree with the data.

What it costs:

No authentication. CTR is malleable; there is no tag, no HMAC, no AEAD. Integrity is not a property of the format.
64 bytes of overhead per file. For a table of many small files this is measurable, and for a 1-byte file it is 64× the data.
The IV is only 16 bytes of entropy per file. One IV collision between two files encrypted with the same key is catastrophic for those two files.
The fingerprint is not a KDF. It is a fixed-seed SipHash, adequate as an identifier, but it is not a password hash.
A 64-byte prefix on every read. For object storage that is an extra ranged request unless the header cache applies, which is why the cache is restricted to random-path backends.

Wrapping up

The one thing to remember: the encryption is a wrapper around the file format, not around the data. Everything above the cipher works in plaintext offsets; only the innermost buffer knows that the file on disk has 64 extra bytes at the front.

The part people get wrong: key rotation does not rewrite anything. Files written with an old key stay encrypted with it, and removing that key from the configuration makes them unreadable — immediately and permanently.

What to go do about it: check that the key you removed is really not needed any more by looking for the error in the server log (Not found an encryption key required to decipher file), and keep a copy of the old key material for as long as any file encrypted with it might still exist — including in backups.

Disks on Opensource internals

How ClickHouse Spreads Parts Across Disks: the Storage Policy

The three levels

Reserving space: the ladder

The bottom of the ladder: what a reservation actually is

TTL rules beat policy rules

Moving parts down the ladder

Inspecting the policy

The move_factor, and what it does now

Trade-offs

Wrapping up

ClickHouse Encrypts Files at Rest with a 64-Byte Header

Why CTR, and not CBC or GCM

The header

The key fingerprint: rotation without rewriting

The offset trick: plaintext and ciphertext live in different coordinates

Writing: new IV on rewrite, same IV on append

Copying ciphertext without decrypting it

The header cache: skipping a read per open

Configuring it

Trade-offs

Wrapping up

The `move_factor`, and what it does now