Opensource internals

ClickHouse Encrypts Files at Rest with a 64-Byte Header

12 min read Jay Hubbert

ClickHouse's encrypted disk prepends a 64-byte self-describing header to every file and runs AES-CTR over the payload, keeping the plaintext and ciphertext offsets apart by exactly one header.

TL;DR: Every encrypted file starts with a 64-byte header holding the signature ENC, a version, the algorithm, a SipHash fingerprint of the key and a random 16-byte IV. The payload is AES-CTR ciphertext, and the CTR counter is derived from the plaintext offset — so the plaintext and ciphertext offsets stay exactly 64 bytes apart, which is what makes random access, appending, and per-file key rotation possible.

Encrypting data at rest looks like a solved problem until you try to do it inside a database. A database does not have one big blob to encrypt; it has millions of small files that it seeks around in, appends to, rewrites, and copies between disks. Each of those operations has to keep working.

That rules out most “obvious” designs. Encrypt the whole file with CBC or GCM and you lose random access: to read byte 1000 you must decrypt bytes 0..1000. Use one key for everything and you cannot rotate the key without rewriting the dataset. Store the key id in a sidecar file and you now have a sidecar to keep consistent with the data it describes.

ClickHouse’s design solves all three with a small, self-describing header prepended to each file, and a cipher mode chosen specifically for its appendability.

Why CTR, and not CBC or GCM

The choice is documented in the source, and it is the load-bearing decision of the whole design:

cpp 
// src/IO/FileEncryptionCommon.h:17
/// Encryption algorithm.
/// We chose to use CTR cipther algorithms because they have the following features which are important for us:
/// - No right padding, so we can append encrypted files without deciphering;
/// - One byte is always ciphered as one byte, so we get random access to encrypted files easily.
enum class Algorithm : uint8_t
{
    AES_128_CTR, /// Size of key is 16 bytes.
    AES_192_CTR, /// Size of key is 24 bytes.
    AES_256_CTR, /// Size of key is 32 bytes.
    MAX
};

Both properties matter enormously for a database:

  • No padding means ciphertext and plaintext have the same length, so getFileSize is just “size minus 64”, and a part’s byte count on disk is predictable.
  • Stream cipher with a counter means byte N of the plaintext is encrypted with counter IV + N/16. You can therefore seek to any offset and decrypt from there, without touching the bytes before it.

The cost of CTR is the one you always pay: it is malleable. An attacker who can write to your disk can flip bits in the ciphertext and flip the same bits in the plaintext. ClickHouse does not use an authenticated cipher here — there is no HMAC, no AEAD tag. What protects the data in practice is the header itself: the fingerprint inside it must match a configured key and the signature must be ENC, so a corrupted or tampered file usually fails loudly rather than decrypting to garbage.

The header

The header layout is a fixed 64-byte structure at the very beginning of every encrypted file:

cpp 
// src/IO/FileEncryptionCommon.h:106
/// File header which is stored at the beginning of encrypted files.
///
/// The format of that header is following:
/// +--------+------+--------------------------------------------------------------------------+
/// | offset | size | description                                                              |
/// +--------+------+--------------------------------------------------------------------------+
/// |      0 |    3 | 'E', 'N', 'C' (file's signature)                                         |
/// |      3 |    2 | version of this header (1..2)                                            |
/// |      5 |    2 | encryption algorithm (0..2, 0=AES_128_CTR, 1=AES_192_CTR, 2=AES_256_CTR) |
/// |      7 |   16 | fingerprint of encryption key (SipHash)                                  |
/// |     23 |   16 | initialization vector (randomly generated)                               |
/// |     39 |   25 | reserved for future use                                                  |
/// +--------+------+--------------------------------------------------------------------------+
cpp 
// src/IO/FileEncryptionCommon.h:120
struct Header
{
    /// Versions:
    /// 1 - Initial version
    /// 2 - The header of an encrypted file contains the fingerprint of a used encryption key instead of a pair {key_id, very_small_hash(key)}.
    ///     The header is always stored in little endian.
    static constexpr const UInt16 kCurrentVersion = 2;

    UInt16 version = kCurrentVersion;

    /// Encryption algorithm.
    Algorithm algorithm = Algorithm::AES_128_CTR;

    /// Fingerprint of a key.
    UInt128 key_fingerprint = 0;

    InitVector init_vector;

    /// The size of this header in bytes, including reserved bytes.
    static constexpr const size_t kSize = 64;

    void read(ReadBuffer & in);
    void write(WriteBuffer & out) const;
};

Three things are packed into those 64 bytes: the 3-byte signature ENC (a magic that lets the reader reject a non-encrypted file instead of decrypting garbage), the 16-byte key fingerprint (which is what makes per-file key rotation possible — the file names the key that encrypted it), and the 16-byte random IV. The remaining 25 bytes are reserved and written as zeros.

The key fingerprint: rotation without rewriting

The server keeps every configured key in a map keyed by its fingerprint:

cpp 
// src/Disks/DiskEncrypted.cpp:259
            for (const auto & [key_id, key_entry] : keys_by_id)
            {
                const auto key = key_entry.plain;
                auto fingerprint = calculateKeyFingerprint(key);
                res->all_keys[fingerprint] = key;

                /// Version 1 used key fingerprints based on the key id.
                /// We have to add such fingerprints to the map too to support reading files encrypted by version 1.
                auto v1_fingerprint = calculateV1KeyFingerprint(key, key_id);
                res->all_keys[v1_fingerprint] = key;
            }

The fingerprint is a keyed SipHash-128, not a bare hash:

cpp 
// src/IO/FileEncryptionCommon.cpp:467
UInt128 calculateKeyFingerprint(const String & key)
{
    const UInt64 seed0 = 0x4368456E63727970ULL; // ChEncryp
    const UInt64 seed1 = 0x7465644469736B46ULL; // tedDiskF
    return sipHash128Keyed(seed0, seed1, key.data(), key.size());
}

The seed is a fixed constant, so this is a keyed hash in the SipHash sense (a random-looking key chosen once), not a secret. Its job is to be a compact identifier that is practically collision-free across a handful of keys, not to be a KDF. Note the v1 fingerprints are added to the same map, so a server upgraded from the old header format can still read files written by the old version — the header version and the fingerprint scheme were versioned together.

When a file is opened, the fingerprint in its header selects the key:

cpp 
// src/Disks/DiskEncryptedTransaction.cpp:42
String DiskEncryptedSettings::findKeyByFingerprint(UInt128 key_fingerprint, const String & path_for_logs) const
{
    auto it = all_keys.find(key_fingerprint);
    if (it == all_keys.end())
    {
        throw Exception(
            ErrorCodes::DATA_ENCRYPTION_ERROR,
            "Not found an encryption key required to decipher file {}",
            quoteString(path_for_logs));
    }
    return it->second;
}

So the rotation procedure is: add the new key to the configuration, mark it current_key, and new writes use it. Old files keep working because they name the old key in their own headers. Nothing needs to be rewritten, and the removal of an old key is a deliberate, auditable act — the moment you remove it, every file that used it becomes unreadable.

The offset trick: plaintext and ciphertext live in different coordinates

Here is the heart of the design. The file on disk is:

text 
    +--------------------------------------------------+
 0  |  header (64 bytes)                              |  file offset 0
    +--------------------------------------------------+
64  |  ciphertext                                      |  file offset 64
    |  ciphertext = AES-CTR(key, IV + plaintext_off/16) |
    +--------------------------------------------------+

The encryptor’s counter is computed from the plaintext offset, not from the file offset:

cpp 
// src/IO/FileEncryptionCommon.cpp:299
    auto current_iv = (init_vector + blocks(offset)).toString();

where offset is the plaintext offset and blocks(pos) = pos / 16.

That single decision buys three things: appending (new plaintext at offset N is encrypted with counter IV + N/16, so nothing is re-encrypted), random access (to read plaintext at offset N, seek the file to N + 64 and start the counter at IV + N/16 — the bytes before N are never touched), and a uniform offset space (every layer above the cipher — the disk abstraction, getFileSize, truncateFile, the read-until-position machinery — works in plaintext offsets, and only the innermost buffer translates).

The translation happens in one place, on the seek path:

cpp 
// src/IO/ReadBufferFromEncryptedFile.cpp:255
void ReadBufferFromEncryptedFile::performSeekAndSetReadUntilPosition()
{
    std::optional<off_t> in_position;
    std::optional<off_t> new_in_position;
    if (need_seek)
    {
        in_position = in->getPosition();
        new_in_position = offset + FileEncryption::Header::kSize;
        chassert(*new_in_position >= static_cast<off_t>(FileEncryption::Header::kSize));
    }

    auto do_seek = [&]
    {
        chassert(new_in_position.has_value());
        offset = in->seek(*new_in_position, SEEK_SET) - FileEncryption::Header::kSize;
        chassert(offset >= 0);
        need_seek = false;
    };

offset is the plaintext position; in->seek(offset + 64) is the file position. The difference is the header, and it is re-derived after every seek rather than being tracked separately, so the two can never drift.

Above the cipher, everything is in plaintext units. The disk reports a plaintext size by subtracting the header:

cpp 
// src/Disks/DiskEncrypted.cpp:527
size_t DiskEncrypted::getFileSize(const String & path) const
{
    auto wrapped_path = wrappedPath(path);
    size_t size = delegate->getFileSize(wrapped_path);
    return size > FileEncryption::Header::kSize ? (size - FileEncryption::Header::kSize) : 0;
}

Writing: new IV on rewrite, same IV on append

The write path is where the header is born. DiskEncryptedTransaction picks between two modes:

cpp 
// src/Disks/DiskEncryptedTransaction.cpp:89
    auto wrapped_path = wrappedPath(path);
    FileEncryption::Header header;
    String key;
    UInt64 old_file_size = 0;
    if (mode == WriteMode::Append && delegate_disk->existsFile(wrapped_path))
    {
        size_t size = delegate_disk->getFileSize(wrapped_path);
        old_file_size = size > FileEncryption::Header::kSize ? (size - FileEncryption::Header::kSize) : 0;
        if (old_file_size)
        {
            /// Append mode: we continue to use the same header.
            auto read_buffer = delegate_disk->readFile(wrapped_path, getReadSettings().adjustBufferSize(FileEncryption::Header::kSize));
            header = readHeader(*read_buffer);
            key = current_settings.findKeyByFingerprint(header.key_fingerprint, path);
        }
    }
    if (!old_file_size)
    {
        /// Rewrite mode: we generate a new header.
        header.algorithm = current_settings.current_algorithm;
        key = current_settings.current_key;
        header.key_fingerprint = current_settings.current_key_fingerprint;
        header.init_vector = FileEncryption::InitVector::random();
    }

Note the asymmetry. Append reuses the existing header — same IV, same key — and sets the encryptor’s offset to the old plaintext size, so the CTR stream continues exactly where it left off. Rewrite generates a fresh random IV and uses the current key. Append mode is also key-aware: it reads the old header to find out which key encrypted the file and keeps using it, so appending to a file written under an old key does not silently re-key half the file.

The header is not written eagerly; it is flushed lazily, on the first buffer of actual data:

cpp 
// src/IO/WriteBufferFromEncryptedFile.cpp:19
    : WriteBufferDecorator<WriteBufferFromFileBase>(std::move(out_), adaptiveBufferInitialSize(use_adaptive_buffer_size_, adaptive_buffer_initial_size, buffer_size_), nullptr, 0)
    , header(header_)
    , flush_header(!old_file_size)
    , encryptor(header.algorithm, key_, header.init_vector)
{
    enableAdaptiveBufferGrowth(use_adaptive_buffer_size_, buffer_size_);
    encryptor.setOffset(old_file_size);
}
cpp 
// src/IO/WriteBufferFromEncryptedFile.cpp:53
void WriteBufferFromEncryptedFile::nextImpl()
{
    if (!offset())
        return;

    if (flush_header)
    {
        header.write(*out);
        flush_header = false;
    }

    encryptor.encrypt(working_buffer.begin(), offset(), *out);

    growAdaptiveBufferAfterFlush();
}

flush_header is false in append mode (the header is already in the file) and true otherwise; the header is emitted before the first ciphertext block, so the file always starts with exactly 64 bytes of header.

Copying ciphertext without decrypting it

A MergeTree replica, a BACKUP, or a move between disks will copy files around. If the source and destination are both encrypted with the same keys, there is no reason to decrypt and re-encrypt: the ciphertext can be copied byte for byte, and DiskEncrypted::copyFile does exactly that when from_settings->all_keys == to_settings->all_keys — the whole key set, not just the current key. If the two disks know different key sets, the copy falls through to the decrypting path, because the destination may need to re-encrypt under a key the source does not have.

The header cache: skipping a read per open

Reading a file requires reading its 64-byte header first, which for object storage means an extra ranged GET. EncryptionHeaderCache keeps those bytes in memory, keyed by a hash of the storage path.

The interesting part is when it is allowed to be used:

cpp 
// src/Disks/DiskEncrypted.cpp:514
    /// Only cache encryption headers when the backend assigns a fresh blob path to every write
    /// (`areBlobPathsRandom`): then a rewrite / replace / rename never rebinds an existing path to
    /// different ciphertext, so the cache can never serve a stale header and needs no invalidation.
    /// Deterministic-path backends (plain / plain-rewritable, local, web) reuse the path on rewrite
    /// and are excluded.
    if (delegate->areBlobPathsRandom())
    {
        if (auto global_context = Context::getGlobalContextInstance())
            if (auto cache = global_context->getEncryptionHeaderCache())
                pipeline.needEncryptionHeaderCache(std::move(cache));
    }

This is a self-invalidating cache by construction. On S3-style object storage every write produces a new blob path, so a cached header can only go stale if the path is reused — which it never is. On a local disk a rewrite reuses the path, so those backends are excluded.

The cache stores opaque bytes rather than parsed headers, so it does not depend on OpenSSL, and its weight function charges the payload plus a fixed 256-byte per-entry overhead, so the byte budget bounds real memory rather than just the header payload.

Configuring it

An encrypted disk is a wrapper around another disk. The key length must match the algorithm exactly — a 32-byte key for aes_256_ctr, a 16-byte key for aes_128_ctr — and the check is strict:

text 
<storage_configuration>
    <disks>
        <encrypted_disk>
            <type>encrypted</type>
            <disk>s3_disk</disk>
            <path>encrypted/</path>
            <key>0123456789abcdef0123456789abcdef</key>
            <current_key_id>0</current_key_id>
            <algorithm>aes_256_ctr</algorithm>
        </encrypted_disk>
    </disks>
</storage_configuration>

Trade-offs

What the design buys:

  • Random access. Reading a byte range never decrypts the bytes before it.
  • Appendability. Appends continue the CTR stream; no rewrite, no re-encryption of the prefix.
  • Per-file key rotation. The header names the key, so old and new keys coexist in one dataset and rotation is a configuration change.
  • Self-description. The file knows its own algorithm and key, so there is no manifest that can disagree with the data.

What it costs:

  • No authentication. CTR is malleable; there is no tag, no HMAC, no AEAD. Integrity is not a property of the format.
  • 64 bytes of overhead per file. For a table of many small files this is measurable, and for a 1-byte file it is 64× the data.
  • The IV is only 16 bytes of entropy per file. One IV collision between two files encrypted with the same key is catastrophic for those two files.
  • The fingerprint is not a KDF. It is a fixed-seed SipHash, adequate as an identifier, but it is not a password hash.
  • A 64-byte prefix on every read. For object storage that is an extra ranged request unless the header cache applies, which is why the cache is restricted to random-path backends.

Wrapping up

The one thing to remember: the encryption is a wrapper around the file format, not around the data. Everything above the cipher works in plaintext offsets; only the innermost buffer knows that the file on disk has 64 extra bytes at the front.

The part people get wrong: key rotation does not rewrite anything. Files written with an old key stay encrypted with it, and removing that key from the configuration makes them unreadable — immediately and permanently.

What to go do about it: check that the key you removed is really not needed any more by looking for the error in the server log (Not found an encryption key required to decipher file), and keep a copy of the old key material for as long as any file encrypted with it might still exist — including in backups.

ClickhouseInternalsEncryptionSecurityDisks