Mergetree on Opensource internals

ClickHouse Stores a Part's Metadata in Plain Text Files Beside the Data

Jay Hubbert — Sat, 24 Oct 2026 09:00:00 +0000

TL;DR: A MergeTree part is a directory. Inside it, the data is a pile of .bin/.mrk files, and the metadata is a pile of small text files: columns.txt, columns_substreams.txt, checksums.txt, count.txt, uuid.txt, metadata_version.txt, serialization.json. They are written atomically (temp file → fsync → rename), parsed back at load time, and then shared between parts in RAM so that a thousand parts cost one copy.

Every columnar database keeps some description of its data files next to the data. The interesting question is not whether — it is what format, and when it is read.

ClickHouse has a choice here that most engines do not: it could store the schema of a part in a binary blob, parsed with a hand-written deserializer, the way it stores column data. Or it could store it as text.

It stores it as text. And the consequences of that decision run through the entire MergeTree implementation.

What a part directory actually contains

The list of files that belong to a part’s metadata — as opposed to its data — is written down once, in one place, because several subsystems need to agree on it (the replication protocol, the backup code, the orphan-part sweeper):

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.h:851
 inline static const Strings COMMON_METADATA_FILES =
 {
 "uuid.txt",
 "checksums.txt",
 "columns.txt",
 "columns_substreams.txt",
 "count.txt",
 "metadata_version.txt",
 "default_compression_codec.txt",
 "serialization.json",
 "partition.dat",
 "ttl.txt",
 };

Ten names. Six of them are plain text (uuid.txt, columns.txt, columns_substreams.txt, count.txt, metadata_version.txt, default_compression_codec.txt), one is JSON (serialization.json), and three are binary (checksums.txt, partition.dat, ttl.txt). A typical part directory looks like this:

 text
 
 
store/3a8/3a83f2c1-.../all_1_7_0/
├── columns.txt <- the physical column list: names + types
├── columns_substreams.txt <- which substreams each column was written as
├── checksums.txt <- size + SipHash-128 of every file
├── count.txt <- the row count, as a decimal integer
├── uuid.txt <- the part's UUID
├── metadata_version.txt <- the table's metadata version at write time
├── default_compression_codec.txt<- "LZ4" (or whatever the default was)
├── serialization.json <- per-column serialization kinds (Sparse etc.)
├── partition.dat <- the partition tuple, binary
├── ttl.txt <- the TTL expressions and their min/max
├── primary.idx <- data
├── primary.mrk <- data
├── event_date.mrk2 <- data
├── event_date.bin <- data
├── ...

Only the first ten are metadata. The rest is data.

`columns.txt`: the physical column list, and nothing else

Here is the whole writer:

 cpp
 
 
// src/Core/NamesAndTypes.cpp:115
void NamesAndTypesList::writeText(WriteBuffer & buf) const
{
 writeString("columns format version: 1\n", buf);
 DB::writeText(size(), buf);
 writeString(" columns:\n", buf);
 for (const auto & it : *this)
 {
 writeBackQuotedString(it.name, buf);
 writeChar(' ', buf);
 writeString(it.type->getName(), buf);
 writeChar('\n', buf);
 }
}

And the whole reader:

 cpp
 
 
// src/Core/NamesAndTypes.cpp:90
void NamesAndTypesList::readText(ReadBuffer & buf, bool check_eof)
{
 const DataTypeFactory & data_type_factory = DataTypeFactory::instance();

 assertString("columns format version: 1\n", buf);
 size_t count = 0;
 DB::readText(count, buf);
 assertString(" columns:\n", buf);

 String column_name;
 String type_name;
 for (size_t i = 0; i < count; ++i)
 {
 readBackQuotedStringWithSQLStyle(column_name, buf);
 assertChar(' ', buf);
 readString(type_name, buf);
 assertChar('\n', buf);

 emplace_back(column_name, data_type_factory.get(type_name));
 }

 if (check_eof)
 assertEOF(buf);
}

There is no length prefix, no offset table, no varint. A file on disk looks like this:

 text
 
 
columns format version: 1
5 columns:
`event_date` Date
`event_time` DateTime
`user_id` UInt64
`url` String
`referer` String

Which is worth pausing on, because it is a deliberate narrowing of what a part records. It is tempting to imagine columns.txt as a snapshot of the table’s schema: the column names, the types, the compression codec, the DEFAULT/MATERIALIZED kind, the default expression, the TTL, the comment. It is not. A part records only the physical columns it actually stores, and their types. Everything else — the codec, the default expression, the column TTL, the comment — lives in the table’s metadata (metadata/<db>.<table>.sql), not in the part.

That is a real design decision, and it is the reason ALTER TABLE ... MODIFY COLUMN does not have to rewrite parts. A part’s columns.txt is a statement about bytes on disk, not about schema. When you add a column, old parts simply do not have it, and the reader substitutes defaults. When you change a codec, old parts keep the codec they were written with, and the codec pipeline in the data file’s frames says so.

This is also why system.parts_columns reports default_kind and default_expression from the table metadata, not from the part:

 cpp
 
 
// src/Storages/System/StorageSystemPartsColumns.cpp:110
 auto metadata_snapshot = info.storage->getInMemoryMetadataPtr(context, false);
 size_t metadata_column_number = 0;
 for (const auto & column : metadata_snapshot->getColumns())
 {
 /// The prepass alone can take a long time on a table with many columns.
 /// A partially filled `columns_info` would report wrong defaults, so give up the whole storage instead.
 ++metadata_column_number;
 slowDownSystemPartsMetadataEnumeration(info.table, metadata_column_number);
 if (query_status && metadata_column_number % COLUMNS_CANCELLATION_CHECK_PERIOD == 0 && !query_status->checkTimeLimit())
 return;

 ColumnInfo column_info;
 if (column.default_desc.expression)
 {
 column_info.default_kind = toString(column.default_desc.kind);
 column_info.default_expression = column.default_desc.expression->formatForLogging();
 }

 columns_info[column.name] = column_info;
 }

Note the prepass: the defaults are computed once per table, not once per part per column. The part only contributes its column list; the table contributes the semantics. So the two views of a column’s metadata look like this:

 sql
 
 
-- The table's metadata: the semantic definition of the column.
SELECT name, type, default_kind, default_expression, compression_codec
FROM system.columns
WHERE database = 'default' AND table = 'events';

-- The part's metadata: the physical bytes that exist on disk.
SELECT part_name, column, type, column_bytes_on_disk
FROM system.parts_columns
WHERE database = 'default' AND table = 'events' AND active;

The first query is about the schema; the second is about the storage. A column added yesterday appears in the first query and in the second query only for parts created since.

`checksums.txt`: a compressed varint table

checksums.txt is the metadata file that is not text you would want to cat. It is written in format version 4, which is a compressed binary encoding:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:247
void MergeTreeDataPartChecksums::write(WriteBuffer & to) const
{
 writeString("checksums format version: 4\n", to);

 CompressedWriteBuffer out{to, CompressionCodecFactory::instance().getDefaultCodec(), 1 << 16};

 writeVarUInt(files.size(), out);

 for (const auto & [name, sum] : files)
 {
 writeStringBinary(name, out);
 writeVarUInt(sum.file_size, out);
 writeBinaryLittleEndian(sum.file_hash, out);
 writeBinaryLittleEndian(sum.is_compressed, out);

 if (sum.is_compressed)
 {
 writeVarUInt(sum.uncompressed_size, out);
 writeBinaryLittleEndian(sum.uncompressed_hash, out);
 }
 }

 out.finalize();
}

So the file is: a text version banner, then a compressed stream of (name, size, hash) triples. The older formats (v2) were human-readable; v3 switched to binary; v4 added the compression. The reader is symmetric:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:160
bool MergeTreeDataPartChecksums::read(ReadBuffer & in)
{
 files.clear();

 assertString("checksums format version: ", in);
 size_t format_version = 0;
 readText(format_version, in);
 assertChar('\n', in);

 read(in, format_version);
 return true;
}

and the version dispatch is explicit, because a part written by a 2019 server must still load on a 2026 one:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:143
bool MergeTreeDataPartChecksums::read(ReadBuffer & in, size_t format_version)
{
 switch (format_version)
 {
 case 1:
 return false;
 case 2:
 return readV2(in);
 case 3:
 return readV3(in);
 case 4:
 return readV4(in);
 default:
 throw Exception(ErrorCodes::UNKNOWN_FORMAT, "Bad checksums format version: {}", DB::toString(format_version));
 }
}

readV2 is the old text format, still supported:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:173
bool MergeTreeDataPartChecksums::readV2(ReadBuffer & in)
{
 size_t count = 0;

 readText(count, in);
 assertString(" files:\n", in);

 for (size_t i = 0; i < count; ++i)
 {
 String name;
 Checksum sum;

 readString(name, in);
 assertFileNameIsRelativeAndContained(name);
 assertString("\n\tsize: ", in);
 readText(sum.file_size, in);
 assertString("\n\thash: ", in);
 readText(sum.file_hash.low64, in);
 assertString(" ", in);
 readText(sum.file_hash.high64, in);
 assertString("\n\tcompressed: ", in);
 readText(sum.is_compressed, in);

Notice the assertFileNameIsRelativeAndContained(name) call. File names in checksums.txt are used verbatim to build paths inside the part directory (when checking sizes, when removing the part), so a hostile or corrupted checksums.txt must not be able to escape it:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:120
void assertFileNameIsRelativeAndContained(const String & name)
{
 if (name.empty())
 throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART, "Empty file name in checksums of data part");

 if (name.starts_with('/'))
 throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART, "Absolute file name '{}' in checksums of data part", name);

 /// A NUL byte would be kept inside a single path component here, but the local disk layer
 /// passes the joined path to C APIs which truncate at the first NUL, so "..\0/x" would act as "..".
 if (name.contains('\0'))
 throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART, "File name '{}' in checksums of data part contains a NUL byte", name);

 for (const auto & component : std::filesystem::path(name))
 {
 if (component == "." || component == "..")
 throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART,
 "File name '{}' in checksums of data part contains '{}' path component", name, component.string());
 }
}

This is a nice illustration of what “the metadata is just a file on disk” costs you: the file is trusted input from the point of view of the storage layer, and every consumer of it has to validate it. The .. check, the NUL check, the absolute-path check — three lines of paranoia that exist only because the format is a text file that anyone with write access to the data directory can edit.

The other sidecars

columns_substreams.txt records, for every column, the list of substreams it was written as (a LowCardinality(String) column, for instance, is written as three substreams: the dictionary, the dictionary size, and the index):

 cpp
 
 
// src/Storages/MergeTree/ColumnsSubstreams.cpp:208
void ColumnsSubstreams::writeText(WriteBuffer & buf) const
{
 writeString("columns substreams version: 1\n", buf);
 DB::writeText(columns_substreams.size(), buf);
 writeString(" columns:\n", buf);
 for (const auto & entry : columns_substreams)
 {
 DB::writeText(entry->substreams.size(), buf);
 writeString(" substreams for column ", buf);
 writeBackQuotedString(entry->column, buf);
 writeCString(":\n", buf);
 for (const auto & substream : entry->substreams)
 {
 writeChar('\t', buf);
 writeString(substream, buf);
 writeChar('\n', buf);
 }
 }
}

count.txt is the row count, and it is astonishingly small — a single decimal integer:

 cpp
 
 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:410
 write_hashed_file("count.txt", [&](auto & buffer)
 {
 writeIntText(rows_count, buffer);
 });

uuid.txt is the part’s UUID, written only if one was assigned:

 cpp
 
 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:360
 if (!new_part->isProjectionPart())
 {
 if (new_part->uuid != UUIDHelpers::Nil)
 {
 write_hashed_file(IMergeTreeDataPart::UUID_FILE_NAME, [&](auto & buffer)
 {
 writeUUIDText(new_part->uuid, buffer);
 });
 }

Note write_hashed_file vs write_plain_file. count.txt, ttl.txt, serialization.json and uuid.txt are hashed — they appear in checksums.txt and are therefore covered by the part’s integrity check. columns.txt, columns_substreams.txt, metadata_version.txt, default_compression_codec.txt and checksums.txt itself are plain — they are not in checksums.txt, because checksums.txt cannot checksum itself, and because columns.txt must remain readable even when the checksums are lost (see below).

The distinction matters. Here is the full set of writes at part finalization:

 cpp
 
 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:450
 write_plain_file("columns.txt", [&](auto & buffer)
 {
 new_part->getColumns().writeText(buffer);
 });

 /// Merge columns substreams from current writer and additional columns substreams
 /// from other writers (that could be used during vertical merge).
 /// If there are no additional columns substreams we still need to call merge
 /// so we will keep only columns that are present in new_part->getColumns().
 /// It may happen that new_part->getColumns() has less columns then columns substreams
 /// from writer because of expired TTL.
 auto columns_substreams = ColumnsSubstreams::merge(
 writer->getColumnsSubstreams(),
 gathered_data.columns_substreams,
 new_part->getColumns().getNames());

 if (!columns_substreams.empty())
 {
 write_plain_file(IMergeTreeDataPart::COLUMNS_SUBSTREAMS_FILE_NAME, [&](auto & buffer)
 {
 columns_substreams.writeText(buffer);
 });

 new_part->setColumnsSubstreams(columns_substreams);
 }

Writing them: atomicity is not optional

A part directory is created, filled, and then renamed into place — the visibility trick that another article covers. But the metadata files themselves are written after that rename, in some cases, because they describe a part that is already live. So they cannot rely on an enclosing rename for atomicity, and they get their own:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2280
void IMergeTreeDataPart::writeMetadata(const String & filename, const WriteSettings & settings, Writer && writer)
{
 auto & data_part_storage = getDataPartStorage();

 /// Rewrite the metadata file atomically and durably (write to a temp file, fsync, rename),
 /// because it goes into an already-committed part directory with no enclosing rename to rely
 /// on. An interrupted in-place write could otherwise leave an empty checksums.txt, which
 /// bricks the whole part. Mirrors VersionMetadataOnDisk::storeInfoToDataPartStorage.
 /// The temp name carries the thread id so two concurrent repairs of the same part (from
 /// separate CHECK TABLE queries) do not rename each other's half-written file. It keeps the
 /// .tmp suffix so checkDataPart skips it if a crash leaves it behind.
 const String tmp_filename = fmt::format("{}.{}.tmp", filename, getThreadId());

 try
 {
 auto out = data_part_storage.writeFile(tmp_filename, 4096, settings);
 writer(*out);
 out->finalize();
 out->sync();

 SyncGuardPtr sync_guard;
 if ((*storage.getSettings())[MergeTreeSetting::fsync_part_directory])
 sync_guard = data_part_storage.getDirectorySyncGuard();
 data_part_storage.replaceFile(tmp_filename, filename);
 }
 catch (...)
 {
 try
 {
 data_part_storage.removeFileIfExists(tmp_filename);
 }
 catch (...)
 {
 tryLogCurrentException(storage.log, __PRETTY_FUNCTION__);
 }
 throw;
 }
}

Temp file named <file>.<thread_id>.tmp, written, finalize()d, sync()ed (a real fsync), and only then rename()d over the target. The thread id is in the temp name so that two concurrent CHECK TABLE queries repairing the same part do not rename each other’s half-written file. The .tmp suffix is kept so that checkDataPart skips it if a crash leaves it behind.

Two details worth calling out:

finalize() before sync(). For a compressed output stream, finalize() flushes the compressed frame and writes the checksum footer. Calling sync() first would sync a half-written compressed stream.
The directory sync is conditional. fsync_part_directory is a setting; without it, the rename is durable but the directory entry might not survive a power loss.

Reading them back: the load path

When a part is attached — at server startup, or on an ATTACH — the metadata is read once and cached in the part object. loadColumnsChecksumsIndexes is the entry point, and its ordering is not arbitrary:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1628
void IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool require_columns_checksums, bool check_consistency, bool load_metadata_version)
{
 /// Memory should not be limited during ATTACH TABLE query.
 /// This is already true at the server startup but must be also ensured for manual table ATTACH.
 /// Motivation: memory for index is shared between queries - not belong to the query itself.
 MemoryTrackerBlockerInThread temporarily_disable_memory_tracker;

 /// Long-lived per-part metadata (columns substreams, checksums, index granularity, patch part
 /// index, primary index, per-column sizes, partition / minmax index, TTL infos, projections)
 /// is routed into the dedicated parts arena by the inner block below. Deliberately kept OUT
 /// of that arena:
 /// - `loadColumns`: its file read and text/JSON parsing are short-lived scratch; the
 /// persistent columns/serializations it produces are arena-scoped inside `setColumns`.
 /// - `checkConsistency`: pure file-existence/size verification, allocates nothing persistent.
 /// - `loadDefaultCompressionCodec`: a tiny per-part codec pointer.
 /// These paths churn many short-lived allocations; keeping them in the default per-CPU arenas
 /// avoids serializing that churn on the single arena's locks under many concurrent merges.

 try
 {
 if (!isStoredOnReadonlyDisk())
 loadUUID();

 loadColumns(require_columns_checksums, load_metadata_version);

 bool has_broken_projections = false;
 {
 ScopedJemallocThreadArena mergetree_arena_scope(JemallocMergeTreeArena::getArenaIndex());

 loadColumnsSubstreams();
 loadInvalidatedSystemColumns();
 loadChecksums(require_columns_checksums);
 loadIndexGranularity();

 /// Load `source_parts.dat` before the primary index: a v2 patch's rebuilt metadata takes
 /// the sort-key columns from it, and the index cannot be deserialized without them.
 loadPatchPartIndex();

 /// It's important to load index after index granularity.
 if (!(*storage.getSettings())[MergeTreeSetting::primary_key_lazy_load])
 index = loadIndex();

 loadRowsCount(); /// Must be called after loadIndexGranularity() as it uses the value of `index_granularity`.

Read that comment: loadRowsCount() must come after loadIndexGranularity() because the row count is derived from the number of marks times the granularity when the part is in the “adaptive granularity” format. The dependency chain of the metadata files is real, and it is documented at every link.

loadColumns is where the interesting recovery logic lives. It handles the case where columns.txt is missing or empty:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2887
void IMergeTreeDataPart::loadColumns(bool require, bool load_metadata_version)
{
 String path = fs::path(getDataPartStorage().getRelativePath()) / "columns.txt";

 NamesAndTypesList loaded_columns;
 bool is_readonly_storage = getDataPartStorage().isReadonly();

 /** A power loss can leave a file that was never `fsync`ed at length zero while the rest of the part
 * survives - the file's inode is persisted, its data block is not - and neither `columns.txt` nor
 * `metadata_version.txt` is covered by the part checksums, so nothing else notices that one of them
 * is gone. Both have a safe path for being absent: the column list is regenerated from the table
 * metadata, the version falls back to the table's. An empty file carries exactly as much as an
 * absent one, so take the same path for it instead of failing to parse it and detaching the whole
 * part - with all of its rows - as broken.
 */
 auto read_non_empty_file_if_exists = [this](const String & file_name)
 {
 auto in = readFileIfExists(file_name);
 if (in && in->eof())
 return std::unique_ptr<ReadBuffer>{};
 return in;
 };

 auto columns_file = readFileIfExists("columns.txt");
 bool columns_file_is_empty = columns_file && columns_file->eof();

 if (columns_file && !columns_file_is_empty)
 {
 loaded_columns.readText(*columns_file);

 for (auto & column : loaded_columns)
 setVersionToAggregateFunctions(column.type, true);

 if (!info.isPatch())
 attachQuantizeSerializations(loaded_columns, getMetadataSnapshot()->getColumns());
 }

This is the payoff of the “no schema in the part” decision. Because columns.txt only records which physical columns are stored, and because the part’s column list can be reconstructed from the table’s metadata cross-checked against the substreams file, an empty columns.txt is not a fatal error. The part rebuilds its own column list:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2923
 /// If there is no column.txt or it's empty, try to recover it from the metadata.
 else
 {
 /// We can get list of columns only from columns.txt in compact parts.
 if (part_type == Type::Compact || info.isPatch())
 throw Exception(ErrorCodes::NO_FILE_IN_DATA_PART, "No columns.txt in part {}, expected path {} on disk {}",
 name, path, getDataPartStorage().getDiskName());

 auto metadata_snapshot = getMetadataSnapshot();

 /// No (or empty) columns.txt: rebuild it.

 /// Load columns substreams, so we can check if specific column exists in the data part via getFirstFileNameForColumn correctly.
 loadColumnsSubstreams(/*validate_against_loaded_columns=*/false);

and then verifies the reconstruction against columns_substreams.txt, which is a second, independent record of the same fact:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2987
 /// loadColumnsSubstreams() requires exactly this list once loadColumns() returns; checking it
 /// here keeps the incomplete list off disk.
 auto recorded_columns = getColumnsSubstreams().getColumnNames();
 if (!recorded_columns.empty() && recorded_columns != loaded_columns.getNames())
 throw Exception(ErrorCodes::NO_FILE_IN_DATA_PART,
 "Cannot rebuild columns.txt of part {}: it stores columns [{}], the list rebuilt from the table metadata is [{}]",
 name, fmt::join(recorded_columns, ", "), fmt::join(loaded_columns.getNames(), ", "));

 if (!is_readonly_storage)
 writeColumns(loaded_columns, {});
 }

Two metadata files recording the same fact, cross-checked against each other, is how a torn write is turned into a recoverable situation rather than a detached part. The rebuilt list is written back to disk, so the next load takes the fast path.

The same trick applies to checksums.txt. If it is missing or empty, the part is re-checksummed from the data files:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2566
void IMergeTreeDataPart::loadChecksums(bool require)
{
 /// Arena-agnostic: the real part-load callers (`loadColumnsChecksumsIndexes`, the projection load)
 /// already run inside the parts-arena scope, while transient checksum-only probes must stay on the
 /// default arenas. So the caller picks the arena rather than pinning it here.
 auto buf = readFileIfExists("checksums.txt");

 /// An existing but empty checksums.txt is treated like an absent one (recalculated from the
 /// data files), not as fatal: a zero-byte file left by an interrupted repair would otherwise
 /// brick an intact part, which is strictly worse than the missing-file state that self-heals.
 if (buf && !buf->eof())
 {
 if (checksums.read(*buf))
 {
 assertEOF(*buf);
 bytes_on_disk = checksums.getTotalSizeOnDisk();
 bytes_uncompressed_on_disk = checksums.getTotalSizeUncompressedOnDisk();
 }
 else
 bytes_on_disk = getDataPartStorage().calculateTotalSizeOnDisk();
 }
 else
 {
 if (require)
 throw Exception(ErrorCodes::NO_FILE_IN_DATA_PART, "No checksums.txt in part {}", name);

 /// If the checksums file is not present (or empty), calculate the checksums and write
 /// them to disk. Check the data while we are at it.
 LOG_WARNING(storage.log, "Checksums for part {} not found or empty. Will calculate them from data on disk.", name);

require is require_part_metadata, set by StorageReplicatedMergeTree: a replicated part with missing metadata must not be guessed at, because the replica can re-fetch it from a healthy peer. A local MergeTree part has nowhere to re-fetch from, so it self-heals.

The metadata is read once — then it is shared

Here is the part that surprises people. When a part is loaded, its column list is turned into a NamesAndTypesList, a name→position map, a ColumnsDescription, a serialization map, and a substream list. That is a lot of memory, and it is identical for every part of a table with the same schema. A thousand parts would have a thousand copies.

ClickHouse does not do that. The bundle is interned once per table:

 cpp
 
 
// src/Storages/MergeTree/SharedPartColumns.h:21
/// Schema-derived metadata shared by all data parts of a table that store the same set of columns.
///
/// Each active data part used to carry its own copy of this metadata (`columns`, the name-to-position
/// map, the `ColumnsDescription`s, the `serializations` map and `columns_substreams`), all built in
/// `IMergeTreeDataPart::setColumns` and immutable afterwards. All of it scales with the number of
/// columns and is identical across parts with the same stored schema, so for tables with many parts
/// (and especially wide tables) the per-part copies dominated the server RSS. Instead, parts now hold
/// a `shared_ptr` to one immutable `SharedPartColumns` bundle interned in a per-`MergeTreeData` cache
/// keyed by the stored column list (see `MergeTreeData::getSharedPartColumnsForColumns`).

The interning key is not just the column list, though — NamesAndTypesList equality is not enough, because it compares types with IDataType::equals, which ignores custom names at any depth:

 cpp
 
 
// src/Storages/MergeTree/SharedPartColumns.h:129
 /// What makes two column lists interchangeable for a data part, and therefore the interning key of a
 /// bundle: the names of the columns, their full type names and the identities of the custom
 /// serializations attached to them from outside the type. `NamesAndTypesList` equality is not enough:
 /// it compares types with `IDataType::equals`, which ignores custom names at any depth
 /// (`Nullable(Bool)` against `Nullable(UInt8)`) although a part persists the name it was declared
 /// with, and no type name shows a serialization attached by a codec (`Quantized` attaches one).
 static String describeColumns(const NamesAndTypesList & columns);

And the serialization half of the bundle is interned in a second-level cache, because two parts can agree on their column list but disagree on whether a column was written sparsely:

 cpp
 
 
// src/Storages/MergeTree/SharedPartColumns.h:39
/// A serialization kind is an `ISerialization::Kind` (`Default`, `Sparse`, `Detached`, `Replicated`),
/// chosen per column when a part is written and stored in its `serialization.json`. Kinds stack
/// (`Detached` over `Sparse` over `Default`) and every subcolumn of a nested type has its own stack, so
/// a column's serialization is identified by a whole recursive kind stack.

That is the reason serialization.json exists at all. If every part always wrote every column densely, the reader could reconstruct the serialization from the type alone and the file would be unnecessary. The file exists because the choice is made per part at write time, and it must be recoverable.

The whole picture

 text
 
 
 write time read time (attach)
 --------- -----------------

 MergeTreeDataWriter loadColumnsChecksumsIndexes()
 | |
 | rows, columns, types | loadUUID() <- uuid.txt
 v | loadColumns() <- columns.txt
 MergedBlockOutputStream | loadColumnsSubstreams()
 finalizePartOnDisk() | loadChecksums() <- checksums.txt
 | | loadIndexGranularity()
 | write count.txt (hashed) | loadIndex() <- primary.idx
 | write ttl.txt (hashed) | loadRowsCount() <- count.txt
 | write uuid.txt (hashed) | loadPartitionAndMinMaxIndex()
 | write serialization.json (hashed) | loadTTLInfos() <- ttl.txt
 | write columns.txt (plain) | loadDefaultCompressionCodec()
 | write columns_substreams.txt (plain) | loadProjections()
 | write metadata_version.txt (plain) |
 | write default_compression_codec.txt |
 | write checksums.txt (plain) v
 | setColumns()
 | |
 | v
 | SharedPartColumns (interned, shared
 | by every part with the same schema)
 v
 part directory renamed into place

Trade-offs

What the text format buys you:

A part is inspectable with cat and grep. When a merge goes wrong and you want to know what a part actually contains, head columns.txt is a complete answer. There is no tool to download and no format to decode.
Forward compatibility is explicit. Every file starts with a version banner (columns format version: 1, checksums format version: 4, columns substreams version: 1). Adding a field is a new version, and the reader dispatches on it. There is no “unknown trailing bytes” hazard.
Partial corruption is recoverable. Because the metadata is small, redundant and cross-checked, a torn write to one file can be repaired from the others. A monolithic binary header has no such redundancy: one bad byte loses the whole part.
Cheap to write. columns.txt for a 500-column table is about 15 KB of text. It is written with a plain writeFile and no compression stage — compressing 15 KB would cost more in CPU than it saves in I/O.

What it costs:

Parsing at load time. Every part load parses columns.txt into DataType objects via the DataTypeFactory, and checksums.txt into a hash map. For a server with 10 000 parts, that is 10 000 parses of the same 15 KB. This is exactly why the parsed result is interned — the parse happens once per distinct schema, and the rest is a pointer copy.
The trust boundary. As assertFileNameIsRelativeAndContained shows, a text file that names other files is an input to path construction and has to be validated. A single binary blob that is length-prefixed and checksummed has fewer ways to be hostile.
Space. Text is bigger than binary. A part with 500 columns spends roughly 15 KB on columns.txt (about 30 bytes a line) and a few KB on columns_substreams.txt. For a table with 10 000 tiny parts, that is hundreds of MB of metadata — which is why columns.txt and columns_substreams.txt are not hashed into checksums.txt and why the in-memory representation is shared.
Self-consistency has to be maintained by hand. columns.txt and columns_substreams.txt both record the column list. The loader cross-checks them explicitly, and there is a comment explaining why. That redundancy is a feature (it enables recovery) but it is also an invariant that a future change can break.

Wrapping up

The one thing to remember: a MergeTree part’s metadata is a handful of small text files next to the data, and it is reconstructed rather than trusted wherever possible. The schema of a part is not stored in the part; only the list of physical columns is. Everything else is derived.

The part people get wrong: columns.txt does not contain the codec, the default expression, or the column TTL. Those live in the table metadata. If you cat columns.txt and expect to see CODEC(ZSTD(3)), you will not find it — and system.parts_columns will not show it either.

What to go do about it: go look at a part on disk.

 sql
 
SELECT name, path FROM system.parts WHERE active AND table = 'your_table' LIMIT 1;

then cat the columns.txt in that directory. It is four lines long, it is the truth about what is stored, and it is the first thing to check when a query returns a column you did not expect.

ClickHouse Names a MergeTree Part After Its Partition and Block Range

Jay Hubbert — Sat, 24 Oct 2026 09:00:00 +0000

TL;DR: A part’s directory name is <partition_id>_<min_block>_<max_block>_<level>[_<mutation>] — for example all_1_7_0 or 202401_42_57_3_9. The partition id is in the name, so ls on the data directory tells you which partition each part belongs to, which block range it covers, and how many merges it has been through — without opening a single file.

Every storage engine needs a way to answer “what do I have?”. For a MergeTree, the answer is a set of immutable directories of column files, and the question is where the index over those directories lives.

ClickHouse’s answer is that there is no separate index. The directory name is the index. It is a compact, human-readable encoding of the part’s identity, and every piece of the engine that needs to reason about parts — the merge selector, the partition pruner, the orphan sweeper, the replication protocol — reads it straight out of the name.

The name

text

<partition_id>_<minimum_block_number>_<maximum_block_number>_<level>_<data_version>

The engine’s own documentation of the system.parts.name column says exactly this:

 cpp
 
 
// src/Storages/System/StorageSystemParts.cpp:71
 {"name", std::make_shared<DataTypeString>(), R"(
Name of the data part. The part naming structure can be used to determine many aspects of the data, ingest, and merge patterns. The part naming format is the following:

```text
<partition_id>_<minimum_block_number>_<maximum_block_number>_<level>_<data_version>
```

* Definitions:
 - `partition_id` - identifies the partition key
 - `minimum_block_number` - identifies the minimum block number in the part. ClickHouse always merges continuous blocks
 - `maximum_block_number` - identifies the maximum block number in the part
 - `level` - incremented by one with each additional merge on the part. A level of 0 indicates this is a new part that has not been merged. It is important to remember that all parts in ClickHouse are always immutable
 - `data_version` - optional value, incremented when a part is mutated (again, mutated data is always only written to a new part, since parts are immutable)
)"},

And here is the entire writer:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.cpp:223
String MergeTreePartInfo::getPartNameV1() const
{
 WriteBufferFromOwnString wb;

 writeString(partition_id, wb);
 writeChar('_', wb);
 writeIntText(min_block, wb);
 writeChar('_', wb);
 writeIntText(max_block, wb);
 writeChar('_', wb);
 if (use_legacy_max_level)
 {
 chassert(level == MAX_LEVEL);
 writeIntText(LEGACY_MAX_LEVEL, wb);
 }
 else
 {
 writeIntText(level, wb);
 }

 if (mutation)
 {
 writeChar('_', wb);
 writeIntText(mutation, wb);
 }

 return wb.str();
}

Thirty lines, no length prefixes, no checksum, no escaping. The name is the metadata, and it round-trips through a parser that is equally simple:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.cpp:64
std::optional<MergeTreePartInfo> MergeTreePartInfo::tryParsePartName(
 std::string_view part_name, MergeTreeDataFormatVersion format_version)
{
 ReadBufferFromString in(part_name);

 String partition_id;

 if (format_version < MERGE_TREE_DATA_MIN_FORMAT_VERSION_WITH_CUSTOM_PARTITIONING)
 {
 UInt32 min_yyyymmdd = 0;
 UInt32 max_yyyymmdd = 0;

 if (!tryReadIntText(min_yyyymmdd, in)
 || !checkChar('_', in)
 || !tryReadIntText(max_yyyymmdd, in)
 || !checkChar('_', in))
 {
 return std::nullopt;
 }

 partition_id = toString(min_yyyymmdd / 100);
 }
 else
 {
 while (!in.eof())
 {
 char c = 0;
 readChar(c, in);
 if (c == '_')
 break;

 partition_id.push_back(c);
 }
 }

 /// Sanity check
 if (partition_id.empty())
 return std::nullopt;

 Int64 min_block_num = 0;
 Int64 max_block_num = 0;
 UInt32 level = 0;
 UInt32 mutation = 0;

 if (!tryReadIntText(min_block_num, in)
 || !checkChar('_', in)
 || !tryReadIntText(max_block_num, in)
 || !checkChar('_', in)
 || !tryReadIntText(level, in))
 {
 return std::nullopt;
 }

 /// Sanity check
 if (min_block_num > max_block_num)
 return std::nullopt;

 if (!in.eof())
 {
 if (!checkChar('_', in)
 || !tryReadIntText(mutation, in)
 || !in.eof())
 {
 return std::nullopt;
 }
 }

The partition id is “everything up to the first underscore”. That is why partition ids are restricted to alphanumerics and dashes:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.cpp:51
 else
 {
 auto is_valid_char = [](char c) { return c == '-' || isAlphaNumericASCII(c); };
 if (!std::all_of(partition_id.begin(), partition_id.end(), is_valid_char))
 throw Exception(ErrorCodes::INVALID_PARTITION_VALUE, "Invalid partition format: {}", partition_id);
 }

No underscore in a partition id. Ever. That invariant is what makes the name parseable without a separator that needs escaping.

The partition id

The partition id is a string derived from the partition key’s value. How it is derived depends on the types involved:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartition.cpp:231
String MergeTreePartition::getID(const Block & partition_key_sample) const
{
 if (value.size() != partition_key_sample.columns())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Invalid partition key size: {}", value.size());

 if (value.empty())
 return "all"; /// It is tempting to use an empty string here. But that would break directory structure in ZK.

 /// In case all partition fields are represented by integral types, try to produce a human-readable ID.
 /// Otherwise use a hex-encoded hash.
 /// NOTE It will work in unexpected way if some partition key column is Nullable:
 /// are_all_integral will be false if some value is NULL. Maybe we should fix it.
 bool are_all_integral = true;
 for (const Field & field : value)
 {
 if (field.getType() != Field::Types::UInt64 && field.getType() != Field::Types::Int64 && field.getType() != Field::Types::IPv4)
 {
 are_all_integral = false;
 break;
 }
 }

Three cases:

No partition key at all → the literal string all. This is where the familiar all_1_7_0 comes from — not from “wide parts”, as is often assumed, but from a table with no partition expression. The comment explains why it is not the empty string: “It is tempting to use an empty string here. But that would break directory structure in ZK.”

All-integral partition key → a human-readable id. A Date column becomes YYYYMMDD, so PARTITION BY event_date gives you 20240115; a toYYYYMM(event_date) gives 202401; a UInt64 gives the number itself. Multiple columns are joined with -:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartition.cpp:253
 if (are_all_integral)
 {
 String result;
 FieldVisitorToString to_string_visitor;

 for (size_t i = 0; i < value.size(); ++i)
 {
 if (i > 0)
 result += '-';

 if (typeid_cast<const DataTypeDate *>(partition_key_sample.getByPosition(i).type.get()))
 result += toString(
 DateLUT::serverTimezoneInstance().toNumYYYYMMDD(
 DayNum(static_cast<DayNum::UnderlyingType>(value[i].safeGet<UInt64>()))));
 else if (typeid_cast<const DataTypeIPv4 *>(partition_key_sample.getByPosition(i).type.get()))
 result += toString(value[i].safeGet<IPv4>().toUnderType());
 else
 result += applyVisitor(to_string_visitor, value[i]);

 /// It is tempting to output DateTime as YYYYMMDDhhmmss, but that would make partition ID
 /// timezone-dependent.
 }

 return result;
 }

Note the last comment: a DateTime partition value is not rendered as a timestamp, because that would make the partition id depend on the server’s timezone — and partition ids are written to ZooKeeper and to directory names, so they must not change when the timezone does.

Anything else → a hex-encoded SipHash-128 of the partition value. So PARTITION BY city gives you a3f2b1... — 32 hex characters.

The name is enough to list a table

Here is the payoff. When ClickHouse starts and attaches a MergeTree table, it walks the data directory and parses the names:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:3319
 runner.enqueueAndKeepTrack([&expected_parts, &unexpected_disk_parts, &disk_parts, this, disk_ptr, component_name = Coordination::getCurrentComponent()]()
 {
 auto local_component_guard = Coordination::setCurrentComponent(component_name);
 for (auto it = disk_ptr->iterateDirectory(relative_data_path); it->isValid(); it->next())
 {
 /// Skip temporary directories, file 'format_version.txt' and directory 'detached'.
 if (startsWith(it->name(), "tmp")
 || it->name() == MergeTreeData::FORMAT_VERSION_FILE_NAME
 || it->name() == DETACHED_DIR_NAME)
 continue;

 if (auto part_info = MergeTreePartInfo::tryParsePartName(it->name(), format_version))
 {
 if (expected_parts && !expected_parts->contains(it->name()))
 unexpected_disk_parts.emplace_back(*part_info, it->name(), disk_ptr);
 else
 disk_parts.emplace_back(*part_info, it->name(), disk_ptr);
 }
 }
 }, Priority{0});

A readdir, a name parse, and a MergeTreePartInfo struct. No file is opened, no columns.txt is read, no checksum is verified. The engine knows, from the name alone:

which partition the part belongs to (and therefore can prune it),
the range of block numbers it covers (and therefore can decide whether two parts overlap),
the merge level (and therefore how “deep” it is),
the mutation version (and therefore which mutations have been applied).

That last point is why the name needs the mutation suffix. A mutated part has the same block range as its parent but a different data version, and the engine has to be able to tell them apart:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.h:82
 /// Get block number that can be used to determine which mutations we still need to apply to this part
 /// (all mutations with version greater than this block number).
 Int64 getDataVersion() const { return mutation ? mutation : min_block; }

The name encodes the merge rule

The most interesting consequence of putting the block range and the level in the name is that the overlap rule becomes a string comparison. Two parts overlap if and only if their names say they do:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.h:86
 /// True if contains rhs (this part is obtained by merging rhs with some other parts or mutating rhs)
 bool contains(const MergeTreePartInfo & rhs) const
 {
 /// Containing part may have equal level iff block numbers are equal (unless level is MAX_LEVEL)
 /// (e.g. all_0_5_2 does not contain all_0_4_2, but all_0_5_3 or all_0_4_2_9 do)
 bool strictly_contains_block_range = (min_block == rhs.min_block && max_block == rhs.max_block)
 || level > rhs.level
 || level == MAX_LEVEL
 || level == LEGACY_MAX_LEVEL;

 return partition_id == rhs.getPartitionId() /// Parts for different partitions are not merged
 && min_block <= rhs.min_block
 && max_block >= rhs.max_block
 && level >= rhs.level
 && mutation >= rhs.mutation
 && strictly_contains_block_range;
 }

Read the comment: “all_0_5_2 does not contain all_0_4_2, but all_0_5_3 or all_0_4_2_9 do”. The level bump is what makes a merge result strictly contain its inputs; a mutation (the _9 suffix) bumps the version instead.

And the disjointness test, which is the hot path of the merge selector:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.h:120
 /// True if parts do not intersect in any way.
 bool isDisjoint(const MergeTreePartInfo & rhs) const
 {
 return partition_id != rhs.getPartitionId()
 || min_block > rhs.max_block
 || max_block < rhs.min_block;
 }

No part sets, no range trees, no indexes. Two MergeTreePartInfo structs and four comparisons.

Which means you can read the merge tree of a table straight out of the names, without opening anything:

 sql
 
 
SELECT name, partition_id, level, rows, min_block_number, max_block_number
FROM system.parts
WHERE database = 'default' AND table = 'events' AND active
ORDER BY partition_id, min_block_number;

 text
 
 
all_1_1_0 level 0 1 row
all_2_2_0 level 0 1 row
all_3_3_0 level 0 1 row
all_1_3_1 level 1 3 rows

The last line is the merge of the first three: same partition, block range 1..3, level 1. The merge selector will next look for another level-1 part with an adjacent block range.

Patch parts: a partition id with a prefix

Patch parts — the parts created by lightweight UPDATE — need a partition id of their own, because they live in their own partition (they are applied to the regular parts at read time). The partition id is a prefix plus a hash of the patch’s structure:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.h:167
 static constexpr std::string_view PATCH_PART_PREFIX = "patch-";
 /// The full prefix of patch part is "patch-<hash>-".
 /// The size of hash is 32 chars plus 1 char for extra dash.
 static constexpr UInt64 PATCH_PART_PREFIX_SIZE = PATCH_PART_PREFIX.size() + 32 + 1;

The hash is computed by a dedicated internal SQL function, which is where the “patch-” prefix and the 32 hex characters come from:

 cpp
 
 
// src/Functions/patchPartitionID.cpp:51
 String patch_prefix{MergeTreePartInfo::PATCH_PART_PREFIX};
 patch_prefix += column_structure_hash->getDataAt(0);
 patch_prefix += "-";

 if (patch_prefix.size() != MergeTreePartInfo::PATCH_PART_PREFIX_SIZE)
 throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid prefix of patch part's partition id: {}", patch_prefix);

And the kind of a part — regular or patch — is derived from the partition id, not stored anywhere:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.h:34
 static Kind getKind(const String & partition_id)
 {
 return partition_id.starts_with(PATCH_PART_PREFIX) ? Kind::Patch : Kind::Regular;
 }

So a patch part’s directory name looks like patch-8f14e45fceea167a5a36dedd4bea2543-202401_1_2_0, and the engine knows it is a patch part, and which partition it patches, purely from the name.

The legacy format

Before custom partitioning keys, the name was YYYYMMDD_YYYYMMDD_min_max_level — the min and max dates of the part:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.cpp:253
String MergeTreePartInfo::getPartNameV0(DayNum left_date, DayNum right_date) const
{
 const auto & date_lut = DateLUT::serverTimezoneInstance();

 /// Directory name for the part has form: `YYYYMMDD_YYYYMMDD_N_N_L`.

 unsigned left_date_id = date_lut.toNumYYYYMMDD(left_date);
 unsigned right_date_id = date_lut.toNumYYYYMMDD(right_date);

This is the historical origin of the “min date” that people remember from part names. The V0 format was tied to a PARTITION BY toYYYYMM(...) table and encoded the month directly, which meant the name was not usable for arbitrary partition expressions. The V1 format generalised it: the partition id is just the partition id, and a Date partition key still gives you a date-looking prefix.

Reading a V0 name is still supported, and the parser branches on the format version:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.cpp:71
 if (format_version < MERGE_TREE_DATA_MIN_FORMAT_VERSION_WITH_CUSTOM_PARTITIONING)
 {
 UInt32 min_yyyymmdd = 0;
 UInt32 max_yyyymmdd = 0;

 if (!tryReadIntText(min_yyyymmdd, in)
 || !checkChar('_', in)
 || !tryReadIntText(max_yyyymmdd, in)
 || !checkChar('_', in))
 {
 return std::nullopt;
 }

 partition_id = toString(min_yyyymmdd / 100);
 }

20240115_20240131_0_5_1 parses back to partition id 202401.

The `LEGACY_MAX_LEVEL` scar

There is one truly ugly corner in the format, and it is worth reading because it shows what “immutable on-disk format” really costs.

Until 21.6, ClickHouse had two different “maximum levels”, and parts written with the old one could intersect parts written with the new one. The fix was not to renumber the old parts — that would have meant rewriting millions of directory names on disk — but to detect the old value at parse time and normalise it:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.cpp:137
 if (level == LEGACY_MAX_LEVEL)
 {
 /// We (accidentally) had two different max levels until 21.6 and it might cause logical errors like
 /// "Part 20170601_20170630_0_2_999999999 intersects 201706_0_1_4294967295".
 /// So we replace unexpected max level to make contains(...) method and comparison operators work
 /// correctly with such virtual parts. On part name serialization we will use legacy max level to keep the name unchanged.
 part_info.use_legacy_max_level = true;
 level = MAX_LEVEL;
 }

and the writer puts the old value back, so the name on disk does not change:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.cpp:233
 if (use_legacy_max_level)
 {
 chassert(level == MAX_LEVEL);
 writeIntText(LEGACY_MAX_LEVEL, wb);
 }

The two constants:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartInfo.h:161
 static constexpr UInt32 MAX_LEVEL = 999999999;
 /// The maximum block number cannot be higher than the maximum sequence number that Keeper can handle.
 /// The sequence number was Int32 originally, it was recently changed to Int64 (https://github.com/ClickHouse/ClickHouse/pull/99120).
 /// To be safe, we keep it as Int32, but more than doubled than the old maximum block number (999999999).
 /// Once we are sure bigger block numbers don't cause any issues, we can increase it even higher.
 static constexpr UInt32 MAX_BLOCK_NUMBER = static_cast<UInt32>(std::numeric_limits<Int32>::max());

Temporary names, and the one place a random suffix appears

Live parts have the canonical name. Everything else is prefixed:

prefix	meaning
`tmp_insert_`	a part being written by an INSERT
`tmp_merge_`	a part being written by a merge
`tmp_mutation_`	a part being written by a mutation
`tmp-fetch_`	a part being fetched from a replica
`delete_tmp_`	a part being deleted (renamed first, then removed)
`<reason>_`	a detached part: `broken_`, `unexpected_`, `noquorum_`, `ignored_`, `broken-on-start_`, `clone_`, `attaching_`, `deleting_`, `tmp-fetch_`, `covered-by-broken_`, …

The delete prefix is applied by renaming the part first, so that a crash during the (slow, recursive) deletion leaves a recognisable directory rather than a half-deleted part:

 cpp
 
 
// src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:827
 /// NOTE We rename part to delete_tmp_<relative_path> instead of delete_tmp_<name> to avoid race condition
 /// when we try to remove two parts with the same name, but different relative paths,
 /// for example all_1_2_1 (in Deleting state) and tmp_merge_all_1_2_1 (in Temporary state).
 fs::path from = fs::path(root_path) / part_dir;
 // TODO directory delete_tmp_<name> is never removed if server crashes before returning from this function

and the recovery sweeper knows which prefixes to clean up:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.h:1115
 /// Root-level temporary directory prefixes used by startup/drop recovery and by ownership checks.
 /// Includes `delete_tmp_` because it can be left by interrupted part removal.
 static constexpr std::array<std::string_view, 3> ROOT_TEMPORARY_DIRECTORY_PREFIXES_FOR_RECOVERY = {"tmp_", "delete_tmp_", "tmp-fetch_"};

Now, the “random suffix” — it exists in exactly one place, and the comment explaining it is one of the best in the codebase:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:11856
 /// 48 random bits, written as 12 hexadecimal digits, are kept out of the generated UUID, so the
 /// whole basename is a fixed `tmp_merge_dr_<12 hex>`: 25 bytes of the filename limit, 36 once
 /// `IMergeTreeDataPart::remove` prepends `delete_tmp_`, which is 6 bytes more than the merge of
 /// the shortest possible pair of parts takes and no more than the merge of any part whose name
 /// is 15 characters or longer. The temporary directory of a dry run lives only for that one query, and
 /// there are at most a handful of them at a time, so 48 bits leave the collision probability at
 /// nothing.
 const String dry_run_suffix = String(MergeTask::DRY_RUN_TEMP_INFIX)
 + getHexUIntLowercase(UUIDHelpers::getLowBytes(UUIDHelpers::generateV4())).substr(4);

Read the constraint: the random suffix has to be short (because the filename limit is 255 bytes and delete_tmp_ eats 10 of them) but collision-free (because two dry runs sharing a directory name would collide). 48 bits is the answer, and it is derived from a UUID — but from the middle of it, so the prefix has a fixed length.

This is the opposite of the part name itself, which is completely deterministic. The distinction is deliberate: a live part’s name must be reproducible (the merge selector computes it, the replication protocol transmits it, the directory name must match the ZK entry), while a temporary directory only has to be unique among the processes currently alive on that disk.

The part UUID

The name is not unique: two parts on two different servers can have the same name. So there is a separate UUID, off by default:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:2289
 DECLARE(Bool, assign_part_uuids, false, R"(
When enabled, a unique part identifier will be assigned for every new part.
Before enabling, check that all replicas support UUID version 4.
)", 0) \

Assigned once, at part creation:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:15131
 if ((*settings)[MergeTreeSetting::assign_part_uuids])
 new_data_part->uuid = UUIDHelpers::generateV4();

and generated with a fast, non-cryptographic RNG — the same one used for query ids:

 cpp
 
 
// src/Core/UUID.cpp:11
 UUID generateV4()
 {
 UUID uuid;
 getHighBytes(uuid) = (thread_local_rng() & 0xffffffffffff0fffull) | 0x0000000000004000ull;
 getLowBytes(uuid) = (thread_local_rng() & 0x3fffffffffffffffull) | 0x8000000000000000ull;

 return uuid;
 }

The masks set the version nibble (0x4000 → version 4) and the variant bits (0x8000... → RFC 4122 variant), so the value is a well-formed UUIDv4 even though the entropy comes from thread_local_rng() rather than /dev/urandom.

The UUID is written to uuid.txt next to the data, and it is what the delete-bitmap cache uses as its identity:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2124
std::string IMergeTreeDataPart::getDeleteBitmapCacheIdentity() const
{
 if (uuid == UUIDHelpers::Nil)
 return getDataPartStorage().getDiskName() + ":" + getRelativePathOfActivePart();
 return toString(uuid);
}

Note the fallback: with no UUID, the identity is the disk name plus the path. With a UUID, it is the UUID alone — which means the identity survives the part being moved between disks.

The whole picture

 text
 
 
 partition key value
 |
 | MergeTreePartition::getID()
 | no key -> "all"
 | all integral -> "202401" / "20240115" / "7"
 | otherwise -> sipHash128 hex
 v
 partition_id
 |
 | + min_block, max_block, level, [mutation]
 v
 MergeTreePartInfo::getPartNameV1()
 |
 v
 "all_1_7_0" "202401_42_57_3" "202401_42_57_3_9"
 "patch-8f14...-202401_1_2_0"
 |
 v
 directory name on disk == directory name in ZooKeeper == part->name
 |
 | tryParsePartName() at startup: readdir + parse, no file opened
 v
 MergeTreePartInfo -> partition pruning, overlap detection, merge selection

Trade-offs

What putting the index in the name buys you:

Listing a table is a readdir. No separate manifest, no catalogue file, no chance of the catalogue and the filesystem disagreeing. The filesystem is the catalogue. If a part is on disk, it is in the list.
Crash recovery is trivial. A part that was renamed into place is there; a part that was not is not. There is no “committed but not recorded” state, because there is no separate record.
The name is self-describing. ls on the data directory tells you which partition, which blocks, which level. You can debug a merge by looking at a directory listing.
The merge rule is a comparison. Overlap detection is four integer comparisons on structs that were parsed from strings.

What it costs:

The name is a compatibility surface. A change to the format is a change to every existing part on every existing server. That is why LEGACY_MAX_LEVEL exists, and why the MAX_LEVEL / LEGACY_MAX_LEVEL distinction is preserved on the write path so the name never changes.
The partition id must be filename-safe. No underscores, no slashes, limited to alphanumerics and dashes. A partition key that produces something else gets hashed — which loses readability but keeps safety.
The hash loses information. PARTITION BY city gives you 32 hex characters, not berlin. You cannot tell from the name which partition a part belongs to; you have to open the part (or read partition.dat).
Deterministic names need a global counter. Block numbers come from a per-table counter, and the merge level from the merge tree’s structure. Getting a unique name for a temporary directory, in contrast, needs randomness — and the randomness has to be short enough to fit in a filename. The tmp_merge_dr_<12 hex> comment is a whole essay in twelve characters.

Wrapping up

The one thing to remember: a ClickHouse part’s directory name is <partition_id>_<min_block>_<max_block>_<level>[_<mutation>], and it is a complete, parseable index. The engine reads the name to decide what to prune, what to merge and what to replicate — and it does that at startup without opening a single file of the part.

The part people get wrong: the all in all_1_7_0 is not about wide parts and not about “all columns”. It is the partition id of a table with no partition key. And the date-looking prefix in 202401_42_57_3 is a partition id that happens to be readable because the partition key was integral.

What to go do about it: look at the directory.

 sql
 
SELECT name, partition_id, level, rows FROM system.parts WHERE table = 'your_table' AND active;

then compare with ls of the table’s directory. Every name matches, and you can now predict what the merge selector will do next: it will merge the parts of the same partition whose block ranges are adjacent and whose levels are equal.

ClickHouse Chooses What Goes into PREWHERE by Cost per Rejected Row

Jay Hubbert — Sat, 24 Oct 2026 09:00:00 +0000

TL;DR: Every conjunct of your WHERE is scored by bytes read per rejected row — bytes_per_row × total_rows / rejected_rows — and the best-scoring ones are moved into PREWHERE, which is evaluated during the read. The PREWHERE is then split into several read steps, so a step that rejects 99 % of the rows lets the next step read its columns for 1 % of the rows.

Here is the shape of a MergeTree read.

 sql
 
SELECT user_id, event_type, payload FROM events WHERE user_id = 42;

user_id is narrow, event_type is narrow, payload is a wide String. A naive execution reads all three columns for every row in the table, then throws away 99.99 % of them. A PREWHERE execution reads user_id for every row, keeps 1 % of them, and reads event_type and payload only for those.

That is the whole idea of PREWHERE, and it is a columnar idea: the filter is pushed down far enough that it can gate the reading of the other columns.

The hard part is deciding which conditions to move. A condition is only worth moving if:

it can be evaluated during the read at all (no arrayJoin, no GLOBAL IN),
the columns it needs are cheap to read,
and it actually rejects a lot of rows.

ClickHouse scores this with a single number — bytes per rejected row — and that number is one of the nicer pieces of cost modelling in the codebase.

Where the decision is made

The transformation happens in the query-plan optimizer, not in the storage engine:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizePrewhere.cpp:236
void optimizePrewhere(QueryPlan::Node & parent_node, const bool remove_unused_columns, const bool suppress_for_vector_search)
{
 /// Assume that there are at least 2 nodes:
 /// 1. FilterNode - parent_node
 /// 2. SourceStepWithFilterNode - child_node

The pattern is a FilterStep sitting directly on top of a read step. The optimizer peels the filter’s conjuncts, scores them, moves the good ones into the read step’s PrewhereInfo, and leaves the rest as a normal filter above.

The whole thing is controlled by four settings, all on by default:

 cpp
 
 
// src/Core/Settings.cpp:1165
 DECLARE(Bool, move_all_conditions_to_prewhere, true, R"(
Move all viable conditions from WHERE to PREWHERE
)", 0) \
 DECLARE(Bool, enable_multiple_prewhere_read_steps, true, R"(
Move more conditions from WHERE to PREWHERE and do reads from disk and filtering in multiple steps if there are multiple conditions combined with AND
)", 0) \
 DECLARE(Bool, move_primary_key_columns_to_end_of_prewhere, true, R"(
Move PREWHERE conditions containing primary key columns to the end of AND chain. It is likely that these conditions are taken into account during primary key analysis and thus will not contribute a lot to PREWHERE filtering.
)", 0) \
 DECLARE(Bool, allow_reorder_prewhere_conditions, true, R"(
When moving conditions from WHERE to PREWHERE, allow reordering them to optimize filtering
)", 0, \

The third one is the most interesting and the least obvious: a condition on a primary key column has probably already been used by the primary index to skip granules, so evaluating it again during the read is nearly free and nearly useless — it should go last, after the conditions that actually filter rows.

Step 1: splitting the WHERE into conjuncts

The filter expression is flattened into its top-level AND chain, and each conjunct is analysed separately:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:400
void MergeTreeWhereOptimizer::analyzeImpl(Conditions & res, const RPNBuilderTreeNode & node, const WhereOptimizerContext & where_optimizer_context, std::set<Int64> & pk_positions) const
{
 /// Flatten the top-level AND chain into individual conjuncts.
 std::vector<RPNBuilderTreeNode> conjuncts;
 collectConjuncts(node, conjuncts);

Each conjunct gets a set of referenced columns and a verdict on whether it is viable — that is, whether it can move to PREWHERE at all:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:438
 /// Is it possible to use primary index for this condition? For conditions like `lower(country) = 'xx'`, may_use_primary_index will be false.
 info.viable =
 !info.has_invalid_column
 /// Condition depends on some column. Constant expressions are not moved.
 && !info.columns.empty()
 && !cannotBeMoved(conjunct, where_optimizer_context)
 /// When use final, do not take into consideration the conditions with non-sorting keys. Because final select
 /// need to use all sorting keys, it will cause correctness issues if we filter other columns before final merge.
 && (!where_optimizer_context.is_final || isDeterministicExpressionOverSortingKey(conjunct, where_optimizer_context.context))
 /// Some identifiers can unable to support PREWHERE (usually because of different types in Merge engine)
 && columnsSupportPrewhere(info.columns)
 /// Do not move conditions involving all queried columns,
 /// unless the condition can be used as a string filter during the scan:
 /// then it is beneficial on its own, because the reader skips copying the non-matching values.
 && (info.columns.size() < queried_columns.size()
 || (where_optimizer_context.apply_string_filters_during_scan && isConditionSuitableForStringValueFilter(conjunct, where_optimizer_context)));

Read the last clause carefully: a condition that mentions every queried column is normally pointless to move, because the reader has to read all the columns anyway. The exception is a substring search on a String column, where the reader can skip copying the non-matching values during deserialisation — so it is worth moving even though it touches everything.

Grouping by column set

Conditions that need the same columns are grouped and scored together, because their combined selectivity is what matters, and because the reader will have to read those columns once anyway:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:458
 /// Group viable conjuncts by their required storage column set so same-column conditions
 /// are treated as a single unit and their combined selectivity can be estimated.

The grouping is by storage column, not by the exact column name, so that subcolumns of the same column (map.key_k0 and map.key_k1) stay together.

Step 2: the cost model

Here is the number. For a condition that rejects rejected_rows rows out of total_rows, the cost is the bytes that have to be read per rejected row:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:563
 /// Combine I/O cost with selectivity using the classic conjunctive filter ordering rule:
 /// sort by cost / (1 - selectivity), i.e. cost per rejected row.
 const double rejected_rows = static_cast<double>(total_rows) - static_cast<double>(cond.estimated_row_count);
 if (total_rows == 0)
 /// No statistics: fall back to pure I/O cost.
 cond.bytes_per_rejected_row = static_cast<double>(cond.columns_size);
 else if (rejected_rows <= 0)
 /// Rejects no rows, so it is useless in PREWHERE regardless of its cost: schedule it last.
 cond.bytes_per_rejected_row = std::numeric_limits<double>::infinity();
 else if (total_size_of_queried_columns == 0)
 /// Nothing measured (compact parts): the type estimate is too coarse to outrank
 /// selectivity, e.g. `Nullable(Int64)` would beat `Int64` on the null byte.
 cond.bytes_per_rejected_row = static_cast<double>(cond.estimated_row_count);
 else
 cond.bytes_per_rejected_row = approximateBytesPerRow(cond.table_columns) * static_cast<double>(total_rows) / rejected_rows;

This is the classic “cost per rejected tuple” rule from the database literature (the Selinger-style ordering of conjunctive filters), and it is exactly the right metric for a columnar reader:

A cheap column that rejects many rows → few bytes per rejected row → evaluated first, because it lets the reader skip the expensive columns for most of the table.
An expensive column that rejects few rows → many bytes per rejected row → evaluated last, because the reader has to read a lot to save a little.

Note the three degenerate cases, which are all about not lying:

No statistics at all → fall back to the pure I/O cost of the condition’s columns.
A condition that rejects nothing → infinity, so it sorts last. There is no point paying to read a column early in order to reject zero rows.
Compact parts, where column sizes are not measurable → charge the estimated row count instead, so the metric stays comparable across conditions.

Where the byte estimate comes from

The per-column size is measured, not guessed, when the parts allow it:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:747
double MergeTreeWhereOptimizer::approximateBytesPerRowAndColumn(const String & column) const
{
 chassert(total_rows > 0);

 if (auto it = column_sizes.find(column); it != column_sizes.end() && it->second != 0)
 return static_cast<double>(it->second) / static_cast<double>(total_rows);

and only guessed when it cannot be measured:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:66
/// Uncompressed, unlike the stored column sizes, so it over-charges - the safe direction for a
/// column whose real cost is unknown.
double approximateBytesPerValueForType(const IDataType & type)
{
 if (type.haveMaximumSizeOfValue())
 return static_cast<double>(type.getMaximumSizeOfValueInMemory());
 if (WhichDataType(type).isString())
 return default_bytes_per_string_value;
 return default_bytes_per_complex_value;
}

with default_bytes_per_string_value = 64 and default_bytes_per_complex_value = 128. Over-charging an unknown column is the safe direction: it makes the condition sort later, not earlier.

The rejection count

estimated_row_count — the number of rows that survive the condition — comes from the statistics-based selectivity estimator, when statistics are available:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:547
 if (where_optimizer_context.use_statistics)
 {
 cond.good = true;
 cond.estimated_row_count = estimator->estimateRelationProfile(storage_metadata, cond.nodes).rows;
 LOG_DEBUG(log, "Condition group ({}) has estimated row count {}", cond.toString(), cond.estimated_row_count);
 }

Without statistics, the older heuristic takes over: a condition of the form column = N with |N| > 2 is assumed to be selective.

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:46
/// Conditions like "x = N" are considered good if abs(N) > threshold.
/// This is used to assume that condition is likely to have good selectivity.
constexpr auto threshold = 2;

Step 3: the ordering

The conditions are compared with a lexicographic tuple, and the tuple is where several independent rules are packed into one comparator:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.h:123
 auto tuple() const
 {
 return std::make_tuple(
 !viable, !good, -min_position_in_primary_key, is_runtime_filter, bytes_per_rejected_row, table_columns.size());
 }

Reading it in order:

!viable — non-viable conditions sort last (and the loop stops there).
!good — conditions without a selectivity hint sort after those with one.
-min_position_in_primary_key — conditions touching a later primary key column come first, so primary-key conditions end up at the tail.
is_runtime_filter — a join runtime filter is known to pass every row, so it sorts last.
bytes_per_rejected_row — the real cost metric.
table_columns.size() — fewer columns first, as a tie-break.

The primary-key rule is not just “sort PK conditions last” — it is subtler, and the code explains why:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:622
 /// E.g., if the primary key is (a, b, c) but the condition is a = 1 and c = 1,
 /// we should only put (a = 1) to the tail of PREWHERE,
 /// and treat (c = 1) as a normal column.

The point is that the primary index can only use a prefix of the sorting key. A condition on a = 1 (the first key column) is very likely already handled by the index; a condition on c = 1 (the third) is not, because the index cannot use c without a and b. So a = 1 goes to the tail and c = 1 stays where it is. The code computes the longest valid prefix:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:625
 if (where_optimizer_context.move_primary_key_columns_to_end_of_prewhere)
 {
 Int64 min_valid_pk_pos = -1;
 for (auto pk_pos : pk_positions)
 {
 if (pk_pos != min_valid_pk_pos + 1)
 break;
 min_valid_pk_pos = pk_pos;
 }

Step 4: how much to move

Scoring is not enough — moving everything is not free either, because the PREWHERE columns are read for every row of the table. The optimizer stops once the moved conditions account for a reasonable share of the read:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeWhereOptimizer.cpp:681
 /// Move conditions unless the ratio of total_size_of_moved_conditions to the total_size_of_queried_columns is less than some threshold.
 while (!where_conditions.empty())
 {
 /// Move the best condition to PREWHERE if it is viable.
 auto it = std::min_element(where_conditions.begin(), where_conditions.end());

 if (!it->viable)
 break;

 if (!where_optimizer_context.move_all_conditions_to_prewhere)
 {
 bool moved_enough = false;
 if (total_size_of_queried_columns > 0)
 {
 /// If we know size of queried columns use it as threshold. 10% ratio is just a guess.
 moved_enough = total_size_of_moved_conditions > 0
 && (total_size_of_moved_conditions + it->columns_size) * 10 > total_size_of_queried_columns;
 }
 else
 {
 /// Otherwise, use number of moved columns as a fallback.
 /// It can happen, if table has only compact parts. 25% ratio is just a guess.
 moved_enough = total_number_of_moved_columns > 0
 && (total_number_of_moved_columns + it->table_columns.size()) * 4 > queried_columns.size();
 }

 if (moved_enough)
 break;
 }

The thresholds are honestly labelled: “10 % ratio is just a guess”, and 25 % of the columns when the byte sizes are not measurable (compact parts).

With move_all_conditions_to_prewhere = true (the default) this whole early-exit is skipped and everything viable moves.

Step 5: splitting PREWHERE into read steps

This is the part that makes PREWHERE more than “filter earlier”. The PREWHERE condition is an AND of several conjuncts, and each conjunct may need different columns. The reader can therefore split the read into several steps, each reading its own columns and filtering the block:

 text
 
 
 granule of 8192 rows, columns: user_id | event_type | payload

 step 1: read user_id only ──► filter user_id = 42 ──► 82 rows survive
 step 2: read event_type ──► filter event_type = 'x' ──► 30 rows survive
 step 3: read payload ──► (no more filters) ──► 30 rows out

 bytes read: 8192*sizeof(user_id) + 82*sizeof(event_type) + 30*sizeof(payload)

The splitting logic is in MergeTreeSplitPrewhereIntoReadSteps.cpp, and its docstring is the design:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSplitPrewhereIntoReadSteps.cpp:262
/// We want to build a sequence of steps that will compute parts of the prewhere condition.
/// Each step reads some new columns and computes some new expressions and a filter condition.
/// The last step computes the final filter condition and the remaining expressions that are required for the main query.
/// The goal of this is to, when it is possible, filter out many rows in early steps so that the remaining steps will
/// read less data from the storage.
/// NOTE: The result of executing the steps is exactly the same as if we would execute the original DAG in single step.

The grouping rule is “adjacent conditions that read the same physical columns”, and the word adjacent is load-bearing:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSplitPrewhereIntoReadSteps.cpp:341
 /// Only adjacent conditions are merged to preserve the user's explicit PREWHERE evaluation order.
 /// Non-adjacent conditions on the same storage column are kept in separate steps even though this
 /// may cause redundant reads, because the user may have intentionally interleaved a guard predicate
 /// (e.g. `PREWHERE tags['safe'] != '' AND value > 0 AND toUInt64(tags['unsafe']) > 0` — the
 /// `value > 0` step must filter rows before evaluating the potentially-throwing conversion).

The user’s explicit PREWHERE order is a contract, and the splitter will not reorder it.

The throwing-condition guard

That example — toUInt64(tags['unsafe']) — is the reason for the may_throw machinery. If a condition can throw (a cast that can fail, a function that can fail), it must not be evaluated on rows that a preceding filter would have rejected. The splitter tracks this per condition:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSplitPrewhereIntoReadSteps.cpp:114
 /// Reuse the predicate that the short-circuit machinery uses to decide which nodes must be
 /// guarded (findLazyExecutedNodes in ExpressionActions.cpp). IExecutableFunction::canThrow
 /// delegates to it as well, through FunctionToExecutableFunctionAdaptor. It is imprecise in
 /// both directions: it reports true for merely expensive functions, and false for some
 /// functions that do throw while parsing row values (for example addDays(String, ...) via
 /// FunctionDateOrDateTimeAddInterval). There is no sound can-throw oracle in the tree yet,
 /// see the TODO on canThrow in IFunctionAdaptors.h.

and a group whose successor may throw is marked as “must materialise its filter”, so the filter is actually applied before the next step runs:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSplitPrewhereIntoReadSteps.cpp:501
 /// A step that precedes a may_throw condition must materialize its filter, so that
 /// the throwing condition is not evaluated on the rows this step rejects.
 .need_filter = force_short_circuit_execution || groups_requiring_filtered_input.contains(step_index + 1),

Read-ahead of the same columns

There is one more optimisation: two steps that read the same storage column would normally deserialise it twice. The splitter can instead have the first step read it and pass it along:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSplitPrewhereIntoReadSteps.cpp:358
 /// For every group, the index of the group whose step reads its columns. A group that was split
 /// from the previous one only because it may throw reads the same storage columns, so with
 /// `read_ahead_columns` its columns are read by the first step of that run and only the evaluation
 /// waits for the filter. Otherwise every such step would deserialize the same storage column again
 /// (for example a whole `Map` for each of its keys). Reading is safe only when the reader does not
 /// convert the column or fill it from a default expression, because that runs on the whole block
 /// before any filter, so the caller decides (see `MergeTreeReaderSettings::read_ahead_prewhere_columns`).

The read-column split itself

The final piece is in the block-reading utilities, where the read task’s column list is partitioned into “columns read by a PREWHERE step” and “columns read after PREWHERE”:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeBlockReadUtils.cpp:502
 auto add_step = [&](const PrewhereExprStep & step)
 {
 /// Computation results from previous steps might be used in the current step as well. In such a case these
 /// computed columns will be present in the current step inputs. They don't need to be read from the disk so
 /// exclude them from the list of columns to read. This filtering must be done before injecting required
 /// columns to avoid adding unnecessary columns or failing to find required columns that are computation
 /// results from previous steps.
 /// Example: step1: sin(a)>b, step2: sin(a)>c

and the accounting of “already known” columns:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeBlockReadUtils.cpp:536
 /// More columns could have been added, filter them as well by the list of columns from previous steps.
 Names columns_to_read_in_step;
 for (const auto & name : step_column_names)
 {
 if (columns_from_previous_steps.emplace(name).second)
 columns_to_read_in_step.push_back(name);
 }

 /// Add results of the step to the list of already "known" columns so that we don't read or compute them again.
 if (step.actions)
 {
 for (const auto & name : step.actions->getActionsDAG().getNames())
 columns_from_previous_steps.insert(name);
 }

 result.pre_columns.push_back(storage_snapshot->getColumnsByNames(options, columns_to_read_in_step));

and finally the columns that the main read needs, minus everything a PREWHERE step already read:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeBlockReadUtils.cpp:573
 /// Remove columns read in prewehere from the list of columns to read.
 Names post_column_names;
 for (const auto & name : column_to_read_after_prewhere)
 {
 if (!columns_from_previous_steps.contains(name))
 post_column_names.push_back(name);
 }

 result.columns = storage_snapshot->getColumnsByNames(options, post_column_names);

The read task then creates one reader per step, each with its own column list:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeReadTask.cpp:288
 for (const auto & pre_columns_per_step : read_info->task_columns.pre_columns)
 {

A worked example

 sql
 
 
CREATE TABLE events
(
 user_id UInt32,
 event_type LowCardinality(String),
 ts DateTime,
 payload String
)
ENGINE = MergeTree
ORDER BY (ts, user_id);

 sql
 
 
SELECT user_id, payload FROM events
WHERE user_id = 42
 AND event_type = 'purchase'
 AND ts > now() - INTERVAL 1 DAY;

What the optimizer does:

ts > now() - INTERVAL 1 DAY touches the first primary key column, so it goes to the tail of the PREWHERE chain — the primary index has probably already used it to skip granules.
user_id = 42 touches the second primary key column, so it is not a prefix-usable condition and stays a normal filter.
event_type = 'purchase' touches a LowCardinality(String) column, which is cheap to read.

The result is a two-step read: one step reads user_id (and event_type) for the whole granule, and the main read fetches payload only for the survivors.

Trade-offs

The metric is a heuristic with a clear shape. Bytes per rejected row is dimensionally correct — it is “how many bytes must I read to throw away one row” — and it is the standard rule for ordering conjunctive filters. It is not a model of the actual I/O: it ignores decompression cost, cache effects, and the fact that a LowCardinality(String) column is much cheaper to decompress than a String payload.

The thresholds are guesses, and the code says so. “10 % ratio is just a guess”, “25 % ratio is just a guess”, threshold = 2 for the “good condition” heuristic. They are heuristics with a decade of empirical tuning behind them, not derivable from first principles.

Statistics make the difference between a good plan and a bad one. Without statistics, the selectivity of a condition is unknown, the metric degenerates to “bytes per row”, and the ordering becomes “cheapest column first” — which is usually but not always right. With statistics, the metric is what it claims to be.

Splitting PREWHERE into steps is not free. Each step is a separate reader with its own decompression state, and each step boundary is a place where a condition’s semantics can be violated. The code is careful about the throwing-condition case and about the user’s explicit evaluation order, and those are the two places where a clever transformation would have produced wrong results.

The escape hatches.

 sql
 
 
-- Turn the whole thing off.
SELECT * FROM events WHERE user_id = 42 SETTINGS optimize_move_to_prewhere = 0;

-- Keep the conditions in their original order.
SELECT * FROM events PREWHERE user_id = 42 AND event_type = 'purchase' SETTINGS allow_reorder_prewhere_conditions = 0;

-- Do it in one read step instead of several.
SELECT * FROM events WHERE user_id = 42 AND event_type = 'purchase'
SETTINGS enable_multiple_prewhere_read_steps = 0;

And to see where the conditions ended up:

 sql
 
EXPLAIN actions = 1 SELECT * FROM events WHERE user_id = 42 AND event_type = 'purchase';

The plan shows a Prewhere filter line with the condition that was moved (and, in non-compact mode, the whole PREWHERE expression graph), plus the read step’s Output: list, which no longer contains the columns that only the PREWHERE needed.

How SummingMergeTree and AggregatingMergeTree Merge Rows

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: These engines do not aggregate at query time and they do not have a separate “aggregate” step. Their merge algorithm walks the sorted streams, and whenever the sorting key changes it finishes the current group: the summed columns are added up, the aggregate states are merged with the aggregate function’s own merge, and the group collapses to one row.

A MergeTree table is a stream of immutable parts, each sorted by the sorting key. The natural way to build a “pre-aggregated” table on top of it is to run a GROUP BY in a periodic job and write the result into another table. That works, but it doubles the storage, it decouples the result from the source, and the result is only as fresh as the last job run.

SummingMergeTree and AggregatingMergeTree take a different route: the merge itself does the aggregation. Because the parts are sorted by the same key, all the rows that belong together are adjacent in the merge’s input, so the merge can collapse them on the fly — no hash table, no second table, no job. The result is a part that is already aggregated, and SELECT becomes a plain scan.

The interesting part is what “collapse” means for two very different kinds of columns: a number that can simply be added, and an aggregate function state — an opaque blob that only the aggregate function itself knows how to combine.

The engine arguments decide what is summed

SummingMergeTree takes an optional list of columns to sum. It is the last engine argument, and the parser recognises it by the fact that it is not a literal (a literal would be the index_granularity, or a replica name):

 cpp
 
 
// src/Storages/MergeTree/registerStorageMergeTree.cpp:651
 else if (merging_params.mode == MergeTreeData::MergingParams::Summing || merging_params.mode == MergeTreeData::MergingParams::Coalescing)
 {
 /// If the last element is not index_granularity or replica_name (a literal), then this is a list of summable columns.
 if (arg_cnt && !engine_args[arg_cnt - 1]->as<ASTLiteral>())
 {
 merging_params.columns_to_sum = extractColumnNames(engine_args[arg_cnt - 1]);
 --arg_cnt;
 }
 }

So ENGINE = SummingMergeTree(d, (k1, k2), 8192, (a, b)) sums a and b, and ENGINE = SummingMergeTree() sums every summable column that is not part of the sorting key.

The list is a filter, not a declaration of what to do. The actual decision — which columns are numeric enough to be summed, which are part of the key, which are AggregateFunction columns — is made by the merge algorithm, once, when the merge starts, by walking the header:

 cpp
 
 
// src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp:289
 /** Fill in the column numbers, which must be summed.
 * This can only be numeric columns that are not part of the sort key.
 * If a non-empty column_names_to_sum is specified, then we only take these columns.
 * Some columns from column_names_to_sum may not be found. This is ignored.
 */
 for (size_t i = 0; i < num_columns; ++i)
 {

“Summable” is a property of the type, and it is deliberately narrow:

 cpp
 
 
// src/DataTypes/IDataType.h:299
 /** Values of data type can be summed (possibly with overflow, within the same data type).
 * Example: numbers, even nullable. Not Date/DateTime. Not Enum.
 * Enums can be passed to aggregate function 'sum', but the result is Int64, not Enum, so they are not summable.
 */
 virtual bool isSummable() const { return false; }

Only DataTypeNumber and Decimal say yes; Date, DateTime, DateTime64, Enum and String all say no. That is not an accident — summing a DateTime would produce a number that is not a date, and summing an Enum would produce an Int64 that is not an Enum, so the merge could not write the result back into the column without changing the schema.

The per-column decision, in the merge algorithm:

 cpp
 
 
// src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp:410
 {
 bool is_agg_func = WhichDataType(column.type).isAggregateFunction();

 /// There are special const columns for example after prewhere sections
 /// or when skip index expressions produce constants.
 if (isColumnConst(*column.column))
 {
 def.column_numbers_not_to_aggregate.push_back(i);
 continue;
 }

 if (!aggregate_all_columns)
 {
 if (!column.type->isSummable() && !is_agg_func && !simple)
 {
 def.column_numbers_not_to_aggregate.push_back(i);
 continue;
 }
 }
 else if (column.type->getTypeId() == TypeIndex::Tuple)
 {
 def.column_numbers_not_to_aggregate.push_back(i);
 continue;
 }

 /// Are they inside the sorting key or partition key? Check both to ignore columns with order expression.
 if (isInSortingKey(description, column.name)
 || isColumnOrAncestorInNames(i, header_flatten, flatten_ancestors, partition_and_sorting_required_columns))
 {
 def.column_numbers_not_to_aggregate.push_back(i);
 continue;
 }

 if (column_names_to_sum.empty()
 || isColumnOrAncestorInNames(i, header_flatten, flatten_ancestors, column_names_to_sum))
 {
 // Create aggregator to sum this column

Read top to bottom, the rules are: a constant column is never summed; a column whose type is not summable is never summed (unless it is an AggregateFunction column or a SimpleAggregateFunction); a column that is part of the sorting key or the partition key is never summed, because the merge would destroy the ordering it is relying on; and the engine’s columns list, if given, filters what is left.

Everything that is not summed goes into column_numbers_not_to_aggregate, and for those columns the merge keeps the first value of the group. That is the “any” semantics of a dimension column: SummingMergeTree collapses the rows but keeps one representative value of everything that is not a measure.

Walking the group

The merge algorithm is a k-way merge over the sorted inputs, and the group boundary is simply “the sorting key changed”. The whole algorithm is about starting and finishing groups:

 cpp
 
 
// src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp:1027
 {
 detail::RowRef current_key;
 setRowRef(current_key, current);

 key_differs = last_key.empty() || rowsHaveDifferentSortColumns(last_key, current_key);

 last_key = current_key;
 last_chunk_sort_columns.clear();
 }

 if (key_differs)
 {
 if (merged_data.isGroupStarted())
 /// Write the data for the previous group.
 merged_data.finishGroup();

 if (merged_data.hasEnoughRows())
 {
 /// The block is now full and the last row is calculated completely.
 last_key.reset();
 return Status(merged_data.pull());
 }

 merged_data.startGroup(current->all_columns, current->getRow());
 }
 else
 merged_data.addRow(current->all_columns, current->getRow());

startGroup creates one aggregate state per summed column; addRow feeds every row of the group into those states; finishGroup writes the results out. No rows are buffered beyond the current group, which is why a SummingMergeTree merge is O(1) in memory regardless of how many rows it collapses.

Summing: the state is the sum

For a plain numeric column, the “aggregate state” is the sumWithOverflow aggregate function’s state, which is a running total. The merge creates one such function per column at start-up:

 cpp
 
 
// src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp:466
 else if (!is_agg_func)
 {
 desc.init(sum_function_name.c_str(), {column.type});
 }

where sum_function_name is "sumWithOverflow" — with overflow, because the result has to fit back into the column’s own type, and a merge must not fail because a sum exceeded UInt64. (An ordinary sum would promote to Int64 or Decimal128 and change the column’s type, which a merge cannot do.)

When the group finishes, the state is finalised into the output column, and — here is the subtle part — the row is rolled back if every summed value came out zero:

 cpp
 
 
// src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp:815
void SummingSortedAlgorithm::SummingMergedData::finishGroup()
{
 is_group_started = false;

 /// We have nothing to aggregate. It means that it could be non-zero, because we have columns_not_to_aggregate.
 if (def.columns_to_aggregate.empty())
 current_row_is_zero = false;

 for (auto & desc : def.columns_to_aggregate)
 {
 // Do not insert if the aggregation state hasn't been created
 if (desc.created)
 {
 if (desc.is_agg_func_type)
 {
 current_row_is_zero = false;
 }
 else
 {
 try
 {
 desc.function->insertResultInto(desc.state.data(), *desc.merged_column, def.arena.get());
 /// Update zero status of current row
 if (!desc.is_simple_agg_func_type && desc.column_numbers.size() == 1 && desc.remove_default_values)
 {
 // Flag row as non-empty if at least one column number if non-zero
 current_row_is_zero = current_row_is_zero
 && desc.merged_column->isDefaultAt(desc.merged_column->size() - 1);
 }
 else
 {
 /// It is sumMapWithOverflow aggregate function.
 /// Assume that the row isn't empty in this case
 /// (just because it is compatible with previous version)
 current_row_is_zero = false;
 }
 }
 catch (...)
 {
 desc.destroyState();
 throw;
 }
 }
 desc.destroyState();
 }
 else
 desc.merged_column->insertDefault();
 }

 /// If it is "zero" row, then rollback the insertion
 /// (at this moment we need rollback only cols from columns_to_aggregate)
 if (current_row_is_zero)
 {
 for (auto & desc : def.columns_to_aggregate)
 desc.merged_column->popBack(1);

 return;
 }

The popBack(1) is the whole trick: the value was inserted into the output column and then taken back out, so a group whose measures all summed to zero produces no row at all. This is what makes a SummingMergeTree table shrink over time instead of accumulating zero rows — a common pattern for “insert an initial value, then a correction” workloads.

Note the asymmetry: a group is dropped only if every summed column is zero. A row with hits = 0 but bytes = 1024 survives. And a group that has columns which are not summed is never dropped, because those columns are dimensions whose value might be meaningful.

The same rule is not applied by SELECT ... FINAL, and the header says why:

 cpp
 
 
// src/Processors/Merges/SummingSortedTransform.h:29
 /// Whether to remove the rows in which all summed columns are zero after summing.
 /// Merges do it, but `SELECT ... FINAL` does not: a query may read only a subset of
 /// the summed columns, so it cannot decide whether a row is zero the way a merge does.
 bool remove_zero_rows = true

This is a real semantic difference between “what the merge would produce” and “what FINAL produces”, and it is a deliberate one: FINAL may project away the column that would have told it the row was zero.

Aggregating: merging opaque states

AggregatingMergeTree is the same idea for arbitrary aggregate functions. The column’s type is AggregateFunction(sum, UInt64) and its value on disk is a serialised aggregate state. The merge cannot add those states up itself — only the aggregate function knows how to combine two of them — so the merge asks the function:

 cpp
 
 
// src/Processors/Merges/Algorithms/AggregatingSortedAlgorithm.cpp:227
void AggregatingSortedAlgorithm::AggregatingMergedData::addRow(SortCursor & cursor)
{
 if (!is_group_started)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't add a row to the group because it was not started.");

 for (auto & desc : def.columns_to_aggregate)
 desc.column->insertMergeFrom(*cursor->all_columns[desc.column_number], cursor->getRow());

 for (auto & desc : def.columns_to_simple_aggregate)
 {
 auto & col = cursor->all_columns[desc.column_number];
 desc.add_function(desc.function.get(), desc.state.data(), &col, cursor->getRow(), def.arena.get());
 }
}

insertMergeFrom is the whole of it, and it ends up in the column:

 cpp
 
 
// src/Columns/ColumnAggregateFunction.cpp:602
void ColumnAggregateFunction::insertMergeFrom(ConstAggregateDataPtr place)
{
 func->merge(data.back(), place, &createOrGetArena());
}

void ColumnAggregateFunction::insertMergeFrom(const IColumn & from, size_t n)
{
 insertMergeFrom(assert_cast<const ColumnAggregateFunction &>(from).data[n]);
}

func->merge(...) is the aggregate function’s own combine operator. For sum it adds the two totals; for uniq it merges the hash sets; for topK it merges the heaps. The merge algorithm does not know or care which — and that is the point: the state is opaque, and the state knows how to combine itself.

The two flavours of aggregate column are handled differently:

AggregateFunction(...) columns keep their state on disk. The merge starts the group by inserting a default (empty) state and merging every row’s state into it:

 cpp
 
 
// src/Processors/Merges/Algorithms/AggregatingSortedAlgorithm.cpp:181
void AggregatingSortedAlgorithm::AggregatingMergedData::startGroup(const ColumnRawPtrs & raw_columns, size_t row)
{
 /// We will write the data for the group. We copy the values of ordinary columns.
 for (auto column_number : def.column_numbers_not_to_aggregate)
 columns[column_number]->insertFrom(*raw_columns[column_number], row);

 /// Add the empty aggregation state to the aggregate columns. The state will be updated in the `addRow` function.
 for (auto & column_to_aggregate : def.columns_to_aggregate)
 column_to_aggregate.column->insertDefault();

SimpleAggregateFunction(...) columns keep only the final value of the aggregate, not a state. The merge keeps a real state in memory during the group and writes the result out at the end — which is why the file format of a SimpleAggregateFunction column is just the underlying type.

The `sumMap` special case

SummingMergeTree has one more trick: a Nested map column whose name ends in Map is summed by key, not as a whole. The key and the value columns of the map are discovered by name convention — the first column, or one ending in ID, Key or Type, is the key; the rest are values:

 cpp
 
 
// src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp:510
 column_num_it = map.second.begin();
 for (; column_num_it != map.second.end(); ++column_num_it)
 {
 const ColumnWithTypeAndName & key_col = header_flatten.safeGetByPosition(*column_num_it);
 const String & name = key_col.name;
 const DataTypePtr nested_type_holder = recursiveRemoveLowCardinality(assert_cast<const DataTypeArray &>(*key_col.type).getNestedType());
 const IDataType & nested_type = *nested_type_holder;

 if (column_num_it == map.second.begin()
 || endsWith(name, "ID")
 || endsWith(name, "Key")
 || endsWith(name, "Type"))
 {
 if (!nested_type.isValueRepresentedByInteger() &&
 !isStringOrFixedString(nested_type) &&
 !typeid_cast<const DataTypeIPv6 *>(&nested_type) &&
 !typeid_cast<const DataTypeUUID *>(&nested_type))
 break;

 map_desc.key_col_nums.push_back(*column_num_it);
 }
 else
 {
 if (!nested_type.isSummable())
 break;

 map_desc.val_col_nums.push_back(*column_num_it);
 }

With exactly one key column, the whole map is handed to sumMapWithOverflow, a single aggregate function that sums the values of matching keys. With a composite key, the engine falls back to the legacy mergeMap path, which merges the maps row by row in C++.

The reason a map needs its own treatment is that the keys are data, not structure: two rows with the same sorting key may have different map keys, and summing the two Array(Array(...)) columns element-wise would produce nonsense. A sumMap adds the values that share a key.

Choosing the algorithm

The engine mode selects the merge algorithm, and the mode comes from the engine name:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3017
 case MergeTreeData::MergingParams::Summing:
 merged_transform = std::make_shared<SummingSortedTransform>(
 header, input_streams_count, sort_description, merging_params.columns_to_sum, partition_and_sorting_required_columns, merge_block_size_rows, merge_block_size_bytes, max_dynamic_subcolumns, merging_params.allow_tuple_element_aggregation);
 break;

 case MergeTreeData::MergingParams::Aggregating:
 merged_transform = std::make_shared<AggregatingSortedTransform>(header, input_streams_count, sort_description, merge_block_size_rows, merge_block_size_bytes, max_dynamic_subcolumns, merging_params.allow_tuple_element_aggregation);
 break;

Every other mode — Ordinary, Collapsing, Replacing, Graphite, VersionedCollapsing, Coalescing — has its own algorithm in the same switch. Collapsing and VersionedCollapsing use the sign-column trick (a +1 row and a -1 row cancel), Replacing keeps the row with the highest version, Coalescing keeps the last value of each column.

Trade-offs

What this design buys:

No extra storage and no extra job. The pre-aggregated data lives in the same table, in the same parts, and is refreshed by the merges that were going to happen anyway.
O(1) memory. Only the current group is buffered, and the group is collapsed as soon as the sorting key changes.
Correctness by construction for the state case. The aggregate function’s own merge is used, so uniq, topK, quantiles and friends behave exactly as they would in a GROUP BY.
The engine’s columns list is a filter, not a contract. The merge refuses to sum a column that is not summable or that is part of the key, so a mistake in the DDL degrades to “not summed” rather than to a corrupt part.

What it costs:

The result depends on the merge. Until the parts are merged, the table still holds the un-aggregated rows, and a query sees a mix. SELECT ... FINAL forces the aggregation at read time — and then it must decide, on its own, which rows to drop.
FINAL and a merge do not agree on zero rows. A merge drops a group whose measures all sum to zero; FINAL does not, because it may not be reading the column that would have revealed the zero.
The set of summed columns is decided by types. A DateTime or an Enum measure is silently not summed, which surprises people who expected “all numeric columns”.
sumWithOverflow means overflow is silent. The sum is computed in the column’s own type, so a counter that exceeds UInt64 wraps rather than failing. That is the price of not changing the column’s type.
Dimension columns keep the first value of the group, not the last and not an aggregate. For a dimension that varies inside a group, the value in the merged row is arbitrary.

How ClickHouse Recompresses a Part When Its TTL Expires

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: TTL expr RECOMPRESS CODEC(ZSTD(17)) is a scheduled, whole-part rewrite with a different codec. The TTL expression is evaluated at insert time and only its maximum is kept; when every row of a part has passed that maximum, the merge selector picks the part, the merge writes it out with the new codec, and the codec is written into the part’s own metadata so the next comparison knows whether a rewrite would actually change anything.

Compression is a bet on the data. A column of timestamps and a column of free-text URLs want opposite codecs: the first wants Delta, ZSTD, the second wants LZ4 or nothing at all. ClickHouse normally resolves this once, at write time, and the choice is then baked into the part forever.

But a table’s data is not homogeneous over its lifetime. The recent partition — last week’s events, still being written — has a very different entropy profile from the archive partition, which was written a year ago and will never change again. A codec chosen for the recent data is wrong for the archive, and vice versa. The usual remedy is a manual OPTIMIZE with a CODEC clause, or a migration, or accepting the waste.

The RECOMPRESS TTL makes it automatic and, more importantly, scheduled: the rule says “after this date, this part should be stored with that codec”, and the background merger honours it the same way it honours a delete TTL. This article follows the whole mechanism: how the rule is parsed and validated, how the per-part expiry bound is computed, how the merge selector decides that a rewrite is worth it, how the codec reaches the column streams, and how the part records the codec it was written with.

The syntax

The TTL grammar has an optional action per expression, and RECOMPRESS is one of them, sitting next to DELETE, TO DISK and TO VOLUME:

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:3007
 else if (s_group_by.ignore(pos, expected))
 {
 mode = TTLMode::GROUP_BY;
 }
 else if (s_recompress.ignore(pos, expected))
 {
 mode = TTLMode::RECOMPRESS;
 }
 else
 {
 /// DELETE is the default mode.
 s_delete.ignore(pos, expected);
 mode = TTLMode::DELETE;
 }

and the codec itself is parsed by the ordinary codec parser, so anything that is a valid CODEC(...) in a column declaration is a valid codec here:

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:3058
 else if (mode == TTLMode::RECOMPRESS)
 {
 if (!s_codec.ignore(pos, expected))
 return false;

 if (!parser_codec.parse(pos, recompression_codec, expected))
 return false;
 }

The documented shape of a table TTL is therefore:

 sql
 
 
-- src/Storages/MergeTree/registerStorageMergeTree.cpp:2060
TTL expr
 [DELETE|RECOMPRESS codec_name1|TO DISK 'xxx'|TO VOLUME 'xxx'][, DELETE|RECOMPRESS codec_name2|TO DISK 'aaa'|TO VOLUME 'bbb'] ...
 [WHERE conditions]
 [GROUP BY key_expr [SET v1 = aggr_func(v1) [, v2 = aggr_func(v2) ...]] ]

and a complete example from the engine’s own documentation:

 sql
 
 
-- src/Storages/MergeTree/registerStorageMergeTree.cpp:2125
CREATE TABLE table_for_recompression
(
 d DateTime,
 key UInt64,
 value String
) ENGINE MergeTree()
ORDER BY tuple()
PARTITION BY key
TTL d + INTERVAL 1 MONTH RECOMPRESS CODEC(ZSTD(17)), d + INTERVAL 1 YEAR RECOMPRESS CODEC(LZ4HC(10))
SETTINGS min_rows_for_wide_part = 0, min_bytes_for_wide_part = 0;

Note the sentence in the same documentation block that defines the semantics precisely (src/Storages/MergeTree/registerStorageMergeTree.cpp:2058):

For parts moving or recompressing, all rows of a part must satisfy the TTL expression criteria.

This is not the semantics of a row TTL. A row TTL fires per row; a recompression TTL fires per part, and only once every row of the part has passed the expression. Recompression is a statement about a part, not about a row, because the codec is a property of the part’s column streams.

Validation

The codec is validated at CREATE time like any other codec, with one relaxation: on ATTACH the checks are relaxed, so that a table created by an older version still loads even if its codec would now be rejected:

 cpp
 
 
// src/Storages/TTLDescription.cpp:1611
 else if (ttl_element->mode == TTLMode::RECOMPRESS)
 {
 /// On `ATTACH` (loading stored metadata) the codec checks are relaxed the same way column codecs are:
 /// a table created on an earlier version must still load even if its recompression codec would now be
 /// rejected at `CREATE`, otherwise the server could fail to start after an upgrade. A create with
 /// `allow_suspicious_ttl_expressions` also skips them, matching `checkTTLExpression` below.
 result.recompression_codec =
 CompressionCodecFactory::instance().validateCodecAndGetPreprocessedAST(
 ttl_element->recompression_codec, {},
 skip_validation ? CodecValidationSettings::trusted() : CodecValidationSettings(context->getSettingsRef()));
 }

The rules of a table are then sorted into buckets by their action — DELETE without a WHERE becomes the table’s single unconditional row TTL, DELETE with a WHERE becomes a conditional row TTL, RECOMPRESS rules go into their own list:

 cpp
 
 
// src/Storages/TTLDescription.cpp:1701
 else if (ttl.mode == TTLMode::RECOMPRESS)
 {
 result.recompression_ttl.emplace_back(std::move(ttl));
 }
 else if (ttl.mode == TTLMode::GROUP_BY)
 {
 result.group_by_ttl.emplace_back(std::move(ttl));
 }
 else
 {
 result.move_ttl.emplace_back(std::move(ttl));
 }

The per-part bound

Like every other TTL, a recompression rule is evaluated at insert time by TTLCalcTransform, and the result is reduced to a min/max pair in the part’s ttl.txt. The rule is stored under its own map, keyed by the result column of the expression, and — importantly — it does not feed the part’s overall part_min_ttl / part_max_ttl:

 cpp
 
 
// src/Processors/TTL/TTLUpdateInfoAlgorithm.cpp:35
void TTLUpdateInfoAlgorithm::finalize(const MutableDataPartPtr & data_part) const
{
 if (ttl_update_field == TTLUpdateField::RECOMPRESSION_TTL)
 {
 data_part->ttl_infos.recompression_ttl[ttl_update_key] = new_ttl_info;
 }
 else if (ttl_update_field == TTLUpdateField::MOVES_TTL)
 {
 data_part->ttl_infos.moves_ttl[ttl_update_key] = new_ttl_info;
 }

That is right: a recompression rule must never make a part look like it has expired rows, or the delete-TTL machinery would start rewriting it for the wrong reason. The recompression map is consulted only by the recompression selector.

Because the semantics are “all rows of the part must have expired”, the selector needs the maximum expiry of the part, and the part-level bound is the smallest of those maxima over all the rules:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:235
time_t MergeTreeDataPartTTLInfos::getMinimalMaxRecompressionTTL() const
{
 time_t max = std::numeric_limits<time_t>::max();
 for (const auto & [name, info] : recompression_ttl)
 if (info.max != 0)
 max = std::min(info.max, max);

 if (max == std::numeric_limits<time_t>::max())
 return 0;

 return max;
}

So with two rules — one at one month, one at one year — the part becomes eligible for the first rule’s codec after one month (its max for the one-month rule has passed) and for the second rule’s codec after one year.

Choosing the rule

When several rules are due, the engine picks the one that “went into action last”, i.e. the one with the largest expiry that is still in the past:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:333
std::optional<TTLDescription> selectTTLDescriptionForTTLInfos(const TTLDescriptions & descriptions, const TTLInfoMap & ttl_info_map, time_t current_time, bool use_max)
{
 time_t best_ttl_time = 0;
 TTLDescriptions::const_iterator best_entry_it;
 for (auto ttl_entry_it = descriptions.begin(); ttl_entry_it != descriptions.end(); ++ttl_entry_it)
 {
 auto ttl_info_it = ttl_info_map.find(ttl_entry_it->result_column);

 if (ttl_info_it == ttl_info_map.end())
 {
 ttl_info_it = tryToFindTTLExpressionInMapByASTMatching(ttl_info_map, ttl_entry_it->result_column);
 if (ttl_info_it == ttl_info_map.end())
 continue;
 }

 time_t ttl_time = 0;

 if (use_max)
 ttl_time = ttl_info_it->second.max;
 else
 ttl_time = ttl_info_it->second.min;

 /// Prefer TTL rule which went into action last.
 if (ttl_time <= current_time
 && best_ttl_time <= ttl_time)
 {
 best_entry_it = ttl_entry_it;
 best_ttl_time = ttl_time;
 }
 }

 return best_ttl_time ? *best_entry_it : std::optional<TTLDescription>();
}

(The tryToFindTTLExpressionInMapByASTMatching fallback exists because the map is keyed by a string rendering of the expression; expired + toIntervalSecond(20) and plus(expired, toIntervalSecond(20)) are the same rule, so the lookup parses both and compares tree hashes.)

A useful side effect of this function is isExplicitRecompression, which answers “is a non-default codec due right now?”:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:367
bool isExplicitRecompression(
 const TTLDescriptions & recompression_ttl_entries, const TTLInfoMap & recompression_ttl_info, time_t current_time)
{
 auto best_ttl_entry
 = selectTTLDescriptionForTTLInfos(recompression_ttl_entries, recompression_ttl_info, current_time, /*use_max=*/true);
 return best_ttl_entry && !CompressionCodecFactory::isDefaultCodec(best_ttl_entry->recompression_codec);
}

“Would this rewrite change anything?”

A part that is already stored with ZSTD(17) must not be rewritten with ZSTD(17) again: the merge would read every column, recompress it identically, and write it back — burning I/O and CPU to produce a byte-identical part. So the selector compares the codec the rule wants with the codec the part has:

 cpp
 
 
// src/Storages/MergeTree/Compaction/PartProperties.cpp:34
std::optional<PartProperties::RecompressTTLInfo> buildRecompressTTLInfo(StorageMetadataPtr metadata_snapshot, MergeTreeDataPartPtr part, time_t current_time)
{
 if (!metadata_snapshot->hasAnyRecompressionTTL())
 return std::nullopt;

 const auto & recompression_ttls = metadata_snapshot->getRecompressionTTLs();
 const auto ttl_description = selectTTLDescriptionForTTLInfos(recompression_ttls, part->ttl_infos.recompression_ttl, current_time, true);

 if (ttl_description)
 {
 /// If the part's own default codec could not be recovered exactly (see
 /// `IMergeTreeDataPart::default_codec_is_approximate`), the comparison below cannot be trusted
 /// either way: treat the codec as unknown and always let the merge selector reconsider the
 /// part, rather than risk a wrong guess suppressing a recompression that is still needed.
 if (part->default_codec_is_approximate)
 return PartProperties::RecompressTTLInfo{
 .will_change_codec = true,
 .next_recompress_ttl = part->ttl_infos.getMinimalMaxRecompressionTTL(),
 };

 /// FIXME: Implement in other way -- not string comparison
 const std::string next_codec = astToString(ttl_description->recompression_codec);
 const std::string current_codec = astToString(part->default_codec->getFullCodecDescription());

 return PartProperties::RecompressTTLInfo{
 .will_change_codec = (next_codec != current_codec),
 .next_recompress_ttl = part->ttl_infos.getMinimalMaxRecompressionTTL(),
 };
 }

 return std::nullopt;
}

Two things are worth noting. The comparison is a string comparison of the codec descriptions — the source even carries a FIXME about it. And the “approximate codec” case is handled by giving up and assuming the codec will change: a part whose codec could not be recovered exactly is always reconsidered, because wrongly concluding “no change” would permanently suppress a recompression that is still needed.

The selector then refuses to consider a part whose codec would not change:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/TTLMergeSelector.cpp:281
TTLRecompressMergeSelector::TTLRecompressMergeSelector(const PartitionIdToTTLs & merge_due_times_, time_t current_time_)
 : ITTLMergeSelector(&merge_due_times_, current_time_)
{
}

time_t TTLRecompressMergeSelector::getTTLForPart(const PartProperties & part) const
{
 return part.recompression_ttl_info->next_recompress_ttl;
}

bool TTLRecompressMergeSelector::canConsiderPart(const PartProperties & part) const
{
 if (part.is_in_volume_where_merges_avoid)
 return false;

 if (!part.recompression_ttl_info.has_value())
 return false;

 /// Allow part recompression only if it will change codec. Otherwise there will be no difference in bytes size.
 return part.recompression_ttl_info->will_change_codec;
}

getTTLForPart returns the part’s maximum expiry, and the generic TTL range builder only includes a part whose TTL has passed, so a part is only rewritten once every one of its rows has expired. The recompression selector is the fourth priority in the TTL merge scheduler, after drop, row delete and column delete:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectorApplier.cpp:113
 /// Recompression - 4 priority
 if (!ctx.merge_constraints.empty() && ctx.metadata_snapshot.hasAnyRecompressionTTL())
 {
 TTLRecompressMergeSelector recompress_ttl_selector(ctx.next_recompress_times, ctx.current_time);

 if (auto merge_ranges = recompress_ttl_selector.select(ctx.ranges, ctx.merge_constraints, ctx.range_filter); !merge_ranges.empty())
 return pack(ctx, std::move(merge_ranges), MergeType::TTLRecompress);
 }

Recompression is the last priority because it changes nothing about the data — it is pure housekeeping — while the first three change what the data means.

Like the delete TTL, recompression is throttled per partition, with its own timeout setting:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:225
 case MergeType::TTLRecompress:
 {
 const time_t next_time = current_time + (*settings)[MergeTreeSetting::merge_with_recompression_ttl_timeout];
 last_recompress_ttl_merge_time_advance[partition_id]
 = {.installed = next_time, .previous = getTTLMergeTime(next_recompress_ttl_merge_times_by_partition, partition_id)};
 next_recompress_ttl_merge_times_by_partition[partition_id] = next_time;
 break;
 }

The codec reaches the column streams

The merge itself is an ordinary merge; the only special thing about it is the codec that the writer uses. The merge asks the table which codec the new part should have, and the answer is the TTL’s codec if a recompression rule is due:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:12305
MergeTreeData::PartCompressionCodec MergeTreeData::getCompressionCodecForPart(
 const StorageMetadataPtr & metadata_snapshot,
 size_t part_size_compressed,
 const IMergeTreeDataPart::TTLInfos & ttl_infos,
 time_t current_time) const
{
 const auto & recompression_ttl_entries = metadata_snapshot->getRecompressionTTLs();
 auto best_ttl_entry = selectTTLDescriptionForTTLInfos(recompression_ttl_entries, ttl_infos.recompression_ttl, current_time, true);

 if (best_ttl_entry)
 return {
 .codec = CompressionCodecFactory::instance().get(best_ttl_entry->recompression_codec, {}),
 .is_explicit_recompression = !CompressionCodecFactory::isDefaultCodec(best_ttl_entry->recompression_codec)};

 auto codec_setting = (*getSettings())[MergeTreeSetting::default_compression_codec].value;
 if (!codec_setting.empty())
 return {.codec = CompressionCodecFactory::instance().get(codec_setting)};

The rest of the function is the ordinary, adaptive codec selection: if no TTL rule applies, the table-level default_compression_codec setting wins, and otherwise the codec is chosen from the ratio of the part’s size to the table’s total size. The TTL is simply the first rule in that chain.

The merge records the choice on the part, together with the reason:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:1169
 else
 {
 auto part_compression_codec = global_ctx->data->getCompressionCodecForPart(
 global_ctx->metadata_snapshot,
 global_ctx->merge_list_element_ptr->total_size_bytes_compressed,
 global_ctx->new_data_part->ttl_infos,
 global_ctx->time_of_merge);
 global_ctx->compression_codec = std::move(part_compression_codec.codec);
 global_ctx->is_explicit_recompression = part_compression_codec.is_explicit_recompression;
 }

 /// Record the chosen codec on the part so that its projections (merged by the sub-merge above,
 /// or rebuilt via `writeTempProjectionPart`) inherit the same codec, together with whether it
 /// was asked for by an explicit `RECOMPRESS` TTL.
 global_ctx->new_data_part->default_codec = global_ctx->compression_codec;
 global_ctx->new_data_part->default_codec_is_explicit_recompression = global_ctx->is_explicit_recompression;

The is_explicit_recompression flag is what stops the adaptive codec selection from overriding the user’s explicit choice. The projections of the part inherit the parent’s codec — and the flag — so that a projection is not silently downgraded to LZ4 because it is small.

The writer then uses the part’s default codec for every column that does not declare its own:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp:157
ASTPtr IMergeTreeDataPartWriter::getCodecDescriptionOrDefault(const String & column_name, CompressionCodecPtr default_codec) const
{
 /// The `default_codec` is already resolved by `MergeTreeData::getCompressionCodecForPart`, which
 /// honors the table-level `default_compression_codec` setting as well as `RECOMPRESS` TTL codecs.
 /// We must trust it here: re-reading `default_compression_codec` and overriding `default_codec`
 /// would make a `RECOMPRESS` TTL merge write column streams with the setting's codec while the
 /// part metadata (`default_compression_codec.txt`) records the TTL codec, so the metadata and the
 /// actual on-disk data would diverge and recompression would not be applied.
 ASTPtr default_codec_desc = default_codec->getFullCodecDescription();

 if (const auto * column_desc = metadata_snapshot->columns.tryGet(column_name))
 return column_desc->codec ? column_desc->codec : default_codec_desc;

The comment explains exactly the failure mode the design avoids: if the writer re-derived the codec from the setting while the metadata recorded the TTL’s codec, the two would disagree, and the next recompression decision would be made against a fiction.

The part remembers its codec

The chosen codec is written into the part’s own metadata, in default_compression_codec.txt:

 cpp
 
 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:484
 if (default_codec != nullptr)
 {
 write_plain_file(IMergeTreeDataPart::DEFAULT_COMPRESSION_CODEC_FILE_NAME, [&](auto & buffer)
 {
 if (new_part->default_codec_is_approximate)
 writeText(IMergeTreeDataPart::UNKNOWN_DEFAULT_COMPRESSION_CODEC, buffer);
 else
 writeText(default_codec->getFullCodecDescription()->formatWithSecretsOneLine(), buffer);
 });
 }

This file is what makes the whole thing idempotent. Without it, the merge scheduler could not tell “this part is already ZSTD(17)” from “this part is ZSTD(3)”, and every part would be rewritten over and over.

The file can be missing or damaged — a part from an older version, a part whose metadata was truncated — and then the codec has to be recovered from the data itself, from the checksums of the column streams. When even that is impossible, the part’s codec is marked approximate:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2186
 else
 {
 /// The `default_compression_codec.txt` file is missing. When no column proves the codec
 /// (every column has an explicit `CODEC`), only a legacy pre-change part can be recovered
 /// safely. A post-change part must contain this file; otherwise its actual default is no
 /// longer recorded on disk, so fail closed instead of reporting a guess.
 default_codec = detectDefaultCompressionCodec([&] { return detectDefaultCompressionCodecFromChecksums(); });
 }

and the “approximate” flag is what the recompression selector respects by assuming the codec will change.

Trade-offs

What the mechanism buys:

A per-part-age codec policy, expressed in one line of DDL. The archive partition ends up compressed the way archive data should be compressed, without anyone running a migration.
No useless rewrites. The will_change_codec check means a part is only rewritten when the codec really differs, and the part’s own metadata is the source of truth for that comparison.
Idempotence and convergence. After the rewrite, the part records the new codec, so the next scheduler pass sees nothing to do. Two replicas that both run the merge end up with the same codec.
No interference with the data semantics. A recompression merge changes bytes on disk, never rows. It is the lowest-priority TTL, so it never delays a delete TTL.

What it costs:

A full rewrite of the part. The rows are read, decompressed, recompressed and written back. The saving is in the steady state — the smaller part costs less to read and less to store — not in the rewrite itself. A part that is rewritten every month pays that I/O every month.
The “all rows must have expired” rule. A part whose rows straddle the boundary is not eligible, so the archive is only recompressed once it is genuinely frozen. For a table with a single active partition, that may never happen.
A string comparison decides whether a rewrite is needed. The source carries a FIXME about it; two spellings of the same codec would be treated as different, and the part would be rewritten once too often.
The four-hour throttle applies, so a rule that fires for many partitions at once will be honoured gradually, partition by partition.
It only changes the default codec. A column that declares its own CODEC(...) keeps it, which is usually what you want but occasionally is not what a user expects when they write RECOMPRESS CODEC(ZSTD(17)).

In short: RECOMPRESS is the compression analogue of the row TTL. It does not do anything at a precise moment — it declares a policy, and the background merger applies it opportunistically, to whole parts, when it is certain that the rewrite will actually change something.

How ClickHouse Picks a Projection to Read

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: Every projection of a table is a candidate. For each one, the planner asks the same question the base table’s read asks — how many marks will this read touch? — and adds the marks of the parts the projection cannot serve, because those parts still have to be read the old way. The cheapest candidate wins, with ties broken by whether the projection’s sort order serves the query’s ORDER BY.

A skip index is a filter: it tells the reader “these granules cannot contain matching rows”, and the reader skips them. A projection is something else entirely. It is a second copy of the data, stored inside the part, with its own sorting key, its own columns and its own aggregate states. When a query matches a projection, the engine can read the projection instead of the table — and if the projection is sorted by something the query wants ordered, the engine can even skip the ORDER BY entirely.

That makes projections a substitution mechanism, and it makes their selection an optimisation problem rather than a filter problem. The planner has to answer: for this query, is reading the projection cheaper than reading the table? And “cheaper” is not a single number, because a projection may only exist for some of the parts.

What a projection is, physically

The syntax is a SELECT inside the table definition:

 sql
 
 
-- src/Storages/MergeTree/registerStorageMergeTree.cpp:1331
 PROJECTION projection_name_1 (SELECT <COLUMN LIST EXPR> [GROUP BY] [ORDER BY]),
 PROJECTION projection_name_2 (SELECT <COLUMN LIST EXPR> [GROUP BY] [ORDER BY])

and the storage model is spelled out in the engine’s own documentation (src/Storages/MergeTree/registerStorageMergeTree.cpp:1974): the projection is stored inside the part directory, in a subdirectory that holds an anonymous MergeTree table’s part, induced by the definition query of the projection. If the projection has a GROUP BY clause, the underlying storage engine becomes AggregatingMergeTree and all aggregate functions are converted to AggregateFunction; if it has an ORDER BY clause, the MergeTree table uses it as its primary key expression. During the merge process the projection part is merged via its storage’s merge routine.

So a projection is a full MergeTree table, living in a subdirectory of the parent’s part directory, with its own sorting key and its own merge behaviour. A projection with a GROUP BY is an AggregatingMergeTree; a projection with an ORDER BY is sorted by that key, and a query that wants that order can read it in order.

Projections are per-part, and that is the whole difficulty

A projection part is created per parent part. On insert, the writer computes the projection for every block and stores the result next to the parent part:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataWriter.cpp:1180
 if ((*data.getSettings())[MergeTreeSetting::materialize_projections_on_insert])
 {
 for (const auto & projection : metadata_snapshot->getProjections())
 {
 /// Commit-order projections use `_block_number` which is only finalized at commit time.
 /// Skip during insert; they will be built correctly during the first merge.
 if (projection.with_block_number)
 continue;

 Block projection_block;
 {
 ProfileEventTimeIncrement<Microseconds> watch(ProfileEvents::MergeTreeDataWriterProjectionsCalculationMicroseconds);
 projection_block = projection.calculate(block, 0, context, perm_ptr);
 LOG_DEBUG(
 log, "Spent {} ms calculating projection {} for the part {}", watch.elapsed() / 1000, projection.name, new_data_part->name);
 }

 if (projection_block.rows())
 {
 auto proj_temp_part
 = writeProjectionPart(data, projection_block, projection, new_data_part.get(), compression_codec, /*merge_is_needed=*/false, context);
 new_data_part->addProjectionPart(projection.name, std::move(proj_temp_part->part));

The setting is on by default (materialize_projections_on_insert, new in 26.6), so a fresh part has its projections. But a part that was inserted before the projection existed does not, and a part whose projection is broken does not either. The planner cannot assume “the projection exists” — it has to check, per part.

That is exactly what the candidate analysis does. For every part the base-table read would touch, it asks whether the part has a usable projection part:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/projectionsCommon.cpp:446
 RangesInDataParts projection_parts;
 size_t parent_parts_sum_marks = 0;
 for (const auto & part_with_ranges : parent_reading_select_result.parts_with_ranges)
 {
 const auto & created_projections = part_with_ranges.data_part->getProjectionParts();
 auto it = created_projections.find(candidate.projection->name);
 MergeTreeDataPartPtr projection_part = it != created_projections.end() && !it->second->is_broken ? it->second : nullptr;
 if (!projection_part && hypothetical_projection && candidate.projection == &hypothetical_projection->projection)
 projection_part = hypothetical_projection->findPart(part_with_ranges.data_part->name);
 /// A projection part returns every matching row of its part, so it cannot replace a read cut by exact counting.
 if (projection_part
 && !(parent_reading_select_result.exact_ranges_excluded && !part_with_ranges.exact_ranges.empty())
 && projectionPartHasRequiredColumns(
 *projection_part, *part_with_ranges.data_part, *candidate.projection, parent_metadata, required_column_names)
 && !partHasPendingMetadataMutationsOnColumns(
 parent_mutations_snapshot, part_with_ranges.data_part, *candidate.projection, required_column_names))
 {
 projection_parts.push_back(RangesInDataPart(
 projection_part,
 part_with_ranges.data_part,
 part_with_ranges.part_index_in_query,
 part_with_ranges.part_starting_offset_in_query));
 }
 else
 {
 candidate.parent_parts.emplace(part_with_ranges.data_part.get());
 parent_parts_sum_marks += part_with_ranges.getMarksCount();
 }
 }

Read the else branch: when a part has no usable projection, that part is pinned to the base table, and its marks are added to the projection’s cost. That single line is the heuristic. A projection that covers 90 % of the parts still costs the marks of the remaining 10 %, because those parts must be read the old way.

The cost is then the sum of two terms:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/projectionsCommon.cpp:516
 candidate.merge_tree_projection_select_result_ptr = std::move(projection_result_ptr);
 candidate.sum_marks += candidate.merge_tree_projection_select_result_ptr->selected_marks + parent_parts_sum_marks;

 return true;

selected_marks is the number of granules the projection read would touch; parent_parts_sum_marks is the number of granules the uncovered parts would touch. The unit is deliberately the same one the base-table read uses, so the comparison is apples to apples.

Note also the early exit at the top of the function:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/projectionsCommon.cpp:476
 if (projection_parts.empty())
 return false;

A projection that no part materialises is not a candidate at all. The planner’s own message for this case is explicit:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeUseNormalProjection.cpp:664
 if (!analyzed)
 {
 result.projection_reject_reasons.try_emplace(projection->name, "no selected part has the projection materialized, or its analysis exceeded the read limits");
 continue;
 }

The comparison

Now the selection itself. The candidate is compared against the base table’s read, and the tie-break is the sort order:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeUseNormalProjection.cpp:697
 /// Consider projections with equal read cost only if:
 /// - `force_optimize_projection` or `prefer_optimize_projection` is enabled, or
 /// - the parent reading's `selected_marks` becomes zero, or
 /// - the projection's sort order matches the query's ORDER BY,
 if (!relax_projection_checks && candidate.sum_marks > parent_reading_marks)
 {
 stat.description = fmt::format(
 "Projection {} is usable but requires reading {} marks, which is not better than the original table with {} marks",
 candidate.projection->name,
 candidate.sum_marks,
 parent_reading_marks);

 result.projection_reject_reasons.try_emplace(projection->name, stat.description);
 LOG_DEBUG(logger, "{}", stat.description);
 continue;
 }
 else if (candidate.sum_marks == parent_reading_marks && parent_reading_marks > 0 && !relax_projection_checks && !sort_order_helps)
 {
 stat.description = fmt::format(
 "Projection {} is usable but requires reading {} marks and does not help with sorting, which is not better than the original table",
 candidate.projection->name,
 candidate.sum_marks);

 result.projection_reject_reasons.try_emplace(projection->name, stat.description);
 LOG_DEBUG(logger, "{}", stat.description);
 continue;
 }

 if (best_candidate == nullptr
 || candidate.sum_marks < best_candidate->sum_marks
 || (candidate.sum_marks == best_candidate->sum_marks && sort_order_helps && !projection_sort_order_useful(best_candidate->projection)))
 best_candidate = &candidate;

 /// All parts has been filtered out; no need to analyze further projections.
 if (parent_reading_marks == 0)
 break;

The rules, distilled:

A projection that reads more marks than the base table is rejected. No amount of “it would be nicer” beats reading more data.
A projection that reads exactly as many marks is accepted only if its sort order serves the query’s ORDER BY — because then it saves the sort, which is worth something even at equal I/O.
Among the survivors, the one with the fewest marks wins; ties are broken in favour of the projection whose sort order helps.

Two escape hatches relax rule 1 and 2: the force_optimize_projection and prefer_optimize_projection settings. The latter is what the streaming read path uses when it has already decided which projection it wants:

 cpp
 
 
// src/Storages/MergeTree/Streaming/ReadingPlan/ReadRoundContext.cpp:20
ContextPtr makeStreamingContext(ContextPtr context_, const ProjectionDescription * projection)
{
 auto copy = Context::createCopy(context_);
 copy->makeQueryContext();
 copy->setQueryMetadataCache(nullptr);

 if (projection)
 {
 copy->setSetting("preferred_optimize_projection_name", projection->name);
 copy->setSetting("prefer_optimize_projection", true);
 }

 return copy;
}

There is also a hypothetical mode: the planner can be asked “what would happen if this projection existed?”, and it answers with the marks it would read, the rows it would read, and whether the optimiser would actually choose it. That is what the EXPLAIN output for projections reports.

Two rejections that are not about cost

Before the cost comparison, two structural rejections happen, and they are worth separating because they are about correctness, not efficiency.

The first is the WHERE implication check. A projection may carry its own WHERE, which means it only stores a subset of the rows. If the query’s filter does not imply the projection’s filter, the projection is missing rows that the query might need, and it cannot be used:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeUseNormalProjection.cpp:610
 /// Skip projections whose WHERE condition is not implied by the query's filter (Issue #74234).
 /// A projection with WHERE stores only a subset of rows, so we can only use it
 /// if the query's filter guarantees it won't need rows outside that subset.
 if (projection->where_clause_ast)
 {
 if (!doesQueryFilterImplyProjectionWhere(query.filter_node, projection->where_clause_ast, projection->query_ast, context))
 {
 LOG_DEBUG(logger, "Projection {} skipped: query WHERE does not imply projection WHERE", projection->name);
 result.projection_reject_reasons.try_emplace(projection->name, "the query WHERE does not imply the projection WHERE");
 continue;
 }
 }

The second is the column check: the projection must contain every column the query needs. If it does not, the projection cannot serve the query — but it may still be useful for filtering, so the planner does not simply give up:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeUseNormalProjection.cpp:623
 if (!has_all_required_columns(projection))
 {
 result.projection_reject_reasons.try_emplace(projection->name, "the projection does not contain all columns required by the query");

 /// Check if projection can be used to filter parts or building projection index filters
 if (query.filter_node && optimize_use_projection_filtering)
 {
 MergeTreeDataSelectExecutor reader(reading->getMergeTreeData(), projection);
 filterPartsAndCollectProjectionCandidates(
 *reading,
 *projection,
 reader,
 empty_mutations_snapshot,
 *parent_reading_select_result,
 projection_query_info,
 query.filter_node,
 context);
 }

 continue;
 }

A projection that is missing a column can still prune parts and granules, which is the “projection as a filter” mode.

The projection’s own key must be used

There is one more rule, and it is easy to miss because it is documented in the settings rather than in the planner. A projection is sorted by its own key, so a query that does not restrict by that key cannot use the projection’s ordering to narrow anything — and the engine will not pretend otherwise:

 cpp
 
 
// src/Core/Settings.cpp:2190
 DECLARE(Bool, force_primary_key, false, R"(
Disables query execution if indexing by the primary key is not possible.

Works with tables in the MergeTree family.

If `force_primary_key=1`, ClickHouse checks to see if the query has a primary key condition that can be used for restricting data ranges. If there is no suitable condition, it throws an exception. However, it does not check whether the condition reduces the amount of data to read. For more information about data ranges in MergeTree tables, see [MergeTree](/reference/engines/table-engines/mergetree-family/mergetree).

The check applies to the primary key of the table itself. A projection serves the query only when the query also restricts data ranges by that projection's own key; otherwise the projection is skipped and the table is read instead. A projection forced with [force_optimize_projection](#force_optimize_projection) whose key is not used makes the query throw an exception.
)", 0) \

The same sentence is the reason a projection is usually declared with an ORDER BY that matches the query’s filter: without it, the projection can only help if the query happens to filter by the same column.

The two “relaxation” settings are documented next to each other, and the difference between them is only what happens when no projection can be used:

 cpp
 
 
// src/Core/Settings.cpp:6815
 DECLARE(Bool, force_optimize_projection, false, R"(
Enables or disables the obligatory use of [projections](/reference/engines/table-engines/mergetree-family/mergetree#projections) in `SELECT` queries, when projection optimization is enabled (see [optimize_use_projections](#optimize_use_projections) setting).

When enabled, a projection that can serve the query is used even if it requires reading more marks than the table itself, and the query fails with the `PROJECTION_NOT_USED` error when no projection can be used.

Possible values:

- 0 — Projection optimization is not obligatory.
- 1 — Projection optimization is obligatory.
)", 0) \
 DECLARE(Bool, prefer_optimize_projection, false, R"(
Makes the projection optimization prefer [projections](/reference/engines/table-engines/mergetree-family/mergetree#projections) over the table in `SELECT` queries, when projection optimization is enabled (see [optimize_use_projections](#optimize_use_projections) setting).

When enabled, a projection that can serve the query is used even if it requires reading more marks than the table itself, the same as with [force_optimize_projection](#force_optimize_projection), but the query does not fail when no projection can be used.

Possible values:

- 0 — Projections are chosen by their estimated cost.
- 1 — A usable projection is chosen regardless of its estimated cost.
)", 0, \
 {"26.10", false, false, "New setting: choose a usable projection regardless of its estimated cost, like `force_optimize_projection`, but without failing the query when no projection is used."}) \

Keeping the projection parts alive

Writing a projection is only half the job: the projection parts have to be merged, or the table fills up with thousands of tiny projection parts. The projection merge is a cascade of levels, and each level merges a bounded number of parts at a time:

 cpp
 
 
// src/Storages/MergeTree/MergeProjectionPartsTask.cpp:33
bool MergeProjectionPartsTask::executeStep()
{
 auto & current_level_parts = level_parts[current_level];
 auto & next_level_parts = level_parts[next_level];

 MergeTreeData::MutableDataPartsVector selected_parts;
 while (selected_parts.size() < max_parts_to_merge_in_one_level && !current_level_parts.empty())
 {
 selected_parts.push_back(std::move(current_level_parts.back()));
 current_level_parts.pop_back();
 }

 if (selected_parts.empty())
 {
 if (next_level_parts.empty())
 {
 LOG_WARNING(log, "There is no projection parts merged");

 /// Task is finished
 return false;
 }
 current_level = next_level;
 ++next_level;
 updatePartsRemaining();
 }
 else if (selected_parts.size() == 1)
 {
 if (next_level_parts.empty())
 {
 LOG_DEBUG(log, "Merged a projection part in level {}", current_level);
 selected_parts[0]->renameTo(projection.name + ".proj", true);
 selected_parts[0]->setName(projection.name);
 selected_parts[0]->is_temp = false;
 selected_parts[0]->temp_projection_block_number.reset();
 new_data_part->addProjectionPart(name, std::move(selected_parts[0]));

 /// Task is finished
 return false;
 }

 LOG_DEBUG(log, "Forwarded part {} in level {} to next level", selected_parts[0]->name, current_level);
 next_level_parts.push_back(std::move(selected_parts[0]));
 updatePartsRemaining();
 }

It is a merge tree of merge trees: a level that ends up with a single part forwards it to the next level, and a level that ends up with several parts merges them into one. The projection’s own merge mode follows its definition — Ordinary for a plain projection, Aggregating for one with a GROUP BY:

 cpp
 
 
// src/Storages/MergeTree/MergeProjectionPartsTask.cpp:87
 MergeTreeData::MergingParams projection_merging_params;
 projection_merging_params.mode = MergeTreeData::MergingParams::Ordinary;
 if (projection.type == ProjectionDescription::Type::Aggregate)
 projection_merging_params.mode = MergeTreeData::MergingParams::Aggregating;

Because projections are per-part, the ordinary merge of the parent table also has to rebuild the projections of the parts it merges. During a merge, the projection is recomputed from the merged rows and written as a temporary projection part, and the codec of the parent is inherited so that a projection of a ZSTD parent is not silently downgraded to LZ4.

Finally, the merge predicate refuses to merge two parts whose projection sets differ:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergePredicates/MergeTreeMergePredicate.cpp:118
 if (left.projection_names != right.projection_names)
 {
 return std::unexpected(
 PreformattedMessage::create(
 "Parts have different projection sets: {} in '{}' and {} in '{}'",
 left.projection_names,
 left.name,
 right.projection_names,
 right.name));
 }

A part “has” a projection only if the projection part exists and is not broken:

 cpp
 
 
// src/Storages/MergeTree/Compaction/PartProperties.cpp:67
std::set<std::string> getCalculatedProjectionNames(const MergeTreeDataPartPtr & part)
{
 std::set<std::string> projection_names;

 for (auto && [name, projection_part] : part->getProjectionParts())
 if (!projection_part->is_broken)
 projection_names.insert(name);

 return projection_names;
}

which is why a projection added to a table does not immediately appear in old parts: the old parts have a different projection set, they will not merge with new ones, and the projection is backfilled by an explicit ALTER TABLE ... MATERIALIZE PROJECTION (or by materialize_projections_on_merge = 1).

Trade-offs

What the mechanism buys:

A second read path for free. The projection is stored once, at write time, and it is chosen per query by cost. No extra query, no extra table, no materialised-view pipeline to keep running.
A cost model in the same unit as the base read. Comparing “marks to read” against the base table’s “marks to read” means the decision automatically accounts for the fact that a projection may not cover all parts.
Order elimination as a tie-break. A projection that is sorted the way the query wants can win even at equal I/O, because it removes the Sort step entirely.
Graceful degradation. A projection that is missing, broken, or too incomplete is not an error — the planner falls back to the base table, and can still use the projection to prune.

What it costs:

Storage is doubled for the projected columns, and the write path pays for computing the projection on every insert.
The projection may not be usable at all. If a table has many small parts and the projection was only materialised on the new ones, every candidate costs “projection marks + the marks of the uncovered parts”, and the projection usually loses.
The cost model is a proxy. Marks are a good proxy for I/O, but not for decompression cost, cache behaviour, or the cost of the GROUP BY that an aggregate projection needs at read time.
The WHERE implication check is conservative. It compares the query’s filter with the projection’s filter structurally, and a mismatch rejects the projection even when the projection would in fact be correct.
Parts with different projection sets do not merge, which fragments the table until a MATERIALIZE PROJECTION backfill catches up.
Everything is per part. Adding a projection to a live table does not make it usable until every part has one.

In short, a projection is a bet that some queries will want a different shape of the data, and the planner’s job is to decide per query whether the bet paid off. The cost model is deliberately simple — marks — because a projection that cannot beat the base read on marks is rarely worth using at all, whatever else it might offer.

ClickHouse Deletes Expired TTL Rows Only When Parts Merge

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: A TTL is not a delete job. At insert time ClickHouse evaluates the TTL expression and remembers only its minimum and maximum per part, in the part’s ttl.txt. The rows actually disappear when the part is merged, and even then the merge does not rewrite them: it computes a UInt8 mask and the merging algorithm simply never copies the expired rows into the output part.

Every time-to-live feature is a promise about the future that a storage engine has to keep. The obvious implementation of “delete rows older than 30 days” is a scheduled job that finds the offending rows and removes them. In a row store that is a DELETE and a vacuum. In a columnar store it is something much worse: a part is an immutable, compressed, column-wise file set, and removing 1 % of its rows means decompressing every column, filtering it, and recompressing the result. Do that every hour and the write amplification dwarfs the workload the user actually asked for.

ClickHouse takes the opposite route. It refuses to do work on a schedule at all. A TTL is a property of the part, not of the row: the engine evaluates the TTL expression once, when the block is written, and distils the result down to two timestamps — the earliest expiry in the part and the latest. Those two numbers live in the part’s metadata. The rows themselves are not touched again until a merge rewrites the part, and then the deletion is done with a trick that costs almost nothing: a bit mask that the merging algorithm consults while it copies rows.

This article walks the whole path: how a TTL is stored, how the background merger decides which parts to rewrite because of a TTL, how the mask is built, how the merge consumes it, and why the effect of a TTL is invisible until a merge happens.

What a part remembers about its TTL

The TTL expression is evaluated at write time by TTLCalcTransform, which attaches one TTLUpdateInfoAlgorithm per TTL rule to the insert pipeline. The algorithm does not look at individual rows after the fact; it accumulates a minimum and a maximum:

 cpp
 
 
// src/Processors/TTL/TTLUpdateInfoAlgorithm.cpp:20
void TTLUpdateInfoAlgorithm::execute(Block & block)
{
 if (block.empty())
 return;

 auto ttl_column = executeExpressionAndGetColumn(ttl_expressions.expression, block, description.result_column);

 const size_t rows = block.rows();
 PaddedPODArray<Int64> timestamps;
 extractTimestamps(ttl_column.get(), timestamps);

 for (size_t i = 0; i < rows; ++i)
 new_ttl_info.update(timestamps[i]);
}

and update is the entire “TTL semantics” of a part:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:17
void MergeTreeDataPartTTLInfo::update(time_t time)
{
 if (time && (!min || time < min))
 min = time;

 max = std::max(time, max);
}

So a part with a TTL ts + INTERVAL 30 DAY rule and rows spread over three months remembers min = the oldest row’s expiry and max = the newest row’s expiry. That pair is enough to answer the only two questions the merge scheduler ever asks:

max <= now → every row in this part has expired; the part can be dropped whole.
min <= now → some row in this part has expired; the part has to be rewritten.

Everything else — the per-row expiry instants, the WHERE clause of the rule — is recomputed at merge time, because the merge has to read the rows anyway.

The two numbers are written into the part as ttl.txt, a small JSON document with a format version in front of it:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:148
void MergeTreeDataPartTTLInfos::write(WriteBuffer & out) const
{
 writeString("ttl format version: 1\n", out);
 writeString("{", out);
 if (!columns_ttl.empty())
 {
 writeString("\"columns\":[", out);
 for (auto it = columns_ttl.begin(); it != columns_ttl.end(); ++it)
 {
 if (it != columns_ttl.begin())
 writeString(",", out);

 writeString("{\"name\":", out);
 writeString(doubleQuoteString(it->first), out);
 writeString(",\"min\":", out);
 writeIntText(it->second.min, out);
 writeString(",\"max\":", out);
 writeIntText(it->second.max, out);
 writeString(R"(,"finished":)", out);
 writeIntText(static_cast<uint8_t>(it->second.finished()), out);
 writeString("}", out);
 }
 writeString("]", out);
 }

The file is read back lazily, only when the table actually has a TTL:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2847
void IMergeTreeDataPart::loadTTLInfos()
{
 auto metadata_snapshot = getMetadataSnapshot();
 if (metadata_snapshot->hasAnyTTL())
 {
 if (auto in = readFileIfExists("ttl.txt"))
 {
 assertString("ttl format version: ", *in);
 size_t format_version = 0;
 readText(format_version, *in);
 assertChar('\n', *in);

 if (format_version == 1)
 {
 try
 {
 ttl_infos.read(*in);
 }
 catch (const JSONException &)
 {
 throw Exception(ErrorCodes::BAD_TTL_FILE, "Error while parsing file ttl.txt in part: {}", name);
 }
 }
 else
 throw Exception(ErrorCodes::BAD_TTL_FILE, "Unknown ttl format version: {}", toString(format_version));
 }
 }
}

Note the shape of the metadata: for a table TTL there is a single {"min":…,"max":…} pair; for column TTLs there is one pair per column; for WHERE-conditioned TTLs the expression string is the key. The expression is stored as a string, and the lookup has to be forgiving about how it is spelled — expired + toIntervalSecond(20) and plus(expired, toIntervalSecond(20)) are the same rule but different strings, so the lookup falls back to parsing both and comparing tree hashes.

Which parts get rewritten because of a TTL

The merge scheduler has four different TTL selectors, and they are tried in a fixed priority order. The order is the whole design: the cheapest way to honour a TTL wins.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectorApplier.cpp:78
MergeSelectorChoices tryChooseTTLMerge(const ChooseContext & ctx)
{
 /// Drop parts - 1 priority
 if (!ctx.merge_constraints.empty())
 {
 /// The size of the completely expired part of TTL drop is not affected by the merge pressure and the size of the storage space.
 std::vector<MergeConstraint> ttl_constraints(ctx.merge_constraints.size(), {std::numeric_limits<size_t>::max(), std::numeric_limits<size_t>::max()});
 TTLPartDropMergeSelector drop_ttl_selector(ctx.current_time, ctx.merge_tree_settings[MergeTreeSetting::max_parts_to_merge_at_once]);

 if (auto merge_ranges = drop_ttl_selector.select(ctx.ranges, ttl_constraints, ctx.range_filter); !merge_ranges.empty())
 return pack(ctx, std::move(merge_ranges), MergeType::TTLDrop);
 }

 /// Delete rows - 2 priority
 if (!ctx.merge_constraints.empty() && !ctx.merge_tree_settings[MergeTreeSetting::ttl_only_drop_parts])
 {
 TTLRowDeleteMergeSelector delete_ttl_selector(ctx.next_delete_times, ctx.current_time);

 if (auto merge_ranges = delete_ttl_selector.select(ctx.ranges, ctx.merge_constraints, ctx.range_filter); !merge_ranges.empty())
 return pack(ctx, std::move(merge_ranges), MergeType::TTLDelete);
 }

The four priorities are: drop the whole part (nothing to rewrite — the part is unlinked), delete rows (rewrite the part, drop the expired rows), delete columns (a column TTL can only be honoured by rewriting the part; there is no cheaper alternative), and recompress (see the companion article on TTL … RECOMPRESS CODEC(…)).

Priority 1 is the cheapest by far, and it is why the TTL merge selector is built the way it is. TTLPartDropMergeSelector looks only at the maximum expiry of the part, and it does not care about the size limits at all:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/TTLMergeSelector.cpp:6
static bool canIncludeToRange(size_t part_size, size_t part_rows, time_t part_ttl, time_t current_time, size_t usable_memory, size_t usable_rows)
{
 return (0 < part_ttl && part_ttl <= current_time) && usable_memory >= part_size && usable_rows >= part_rows;
}

Contrast that with the row-delete selector, which looks at the minimum expiry — the part qualifies as soon as a single row has expired, so the merge has to actually rewrite it:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/TTLMergeSelector.cpp:244
time_t TTLRowDeleteMergeSelector::getTTLForPart(const PartProperties & part) const
{
 return part.general_ttl_info->part_min_ttl;
}

bool TTLRowDeleteMergeSelector::canConsiderPart(const PartProperties & part) const
{
 if (part.is_in_volume_where_merges_avoid)
 return false;

 if (!part.general_ttl_info.has_value())
 return false;

 return part.general_ttl_info->has_any_non_finished_row_ttls;
}

The has_any_non_finished_* flag matters: once every row of a part has expired and the TTL has been applied, the part’s TTL info is marked finished and the selector must not pick it again. A part that was fully expired and dropped does not come back; a part that was partially rewritten records the new minimum, and the selector moves on to the next one.

The range building itself is a small greedy algorithm. For each candidate centre part (a part whose TTL has expired), the selector walks left and right while the accumulated size and row count fit the merge constraint, and it refuses to include a neighbour that has not itself expired — that is the meaning of the part_ttl <= current_time term in canIncludeToRange. A TTL merge therefore only ever merges parts whose data is actually due for deletion, and it drags along the minimum number of extra parts needed to make a legal merge.

The four-hour throttle

A TTL merge is not free — it rewrites the part — and a table whose TTL expires continuously would otherwise be rewritten continuously. ClickHouse throttles this per partition: after a TTL delete merge in a partition, the next one in that partition is not scheduled for four hours.

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:194
void MergeTreeDataMergerMutator::updateTTLMergeTimes(const MergeSelectorChoices & choices, const MergeTreeSettingsPtr & settings, time_t current_time)
{
 for (const auto & choice : choices)
 {
 chassert(!choice.range.empty());
 const String & partition_id = choice.range.front().info.getPartitionId();

 switch (choice.merge_type)
 {
 case MergeType::Regular:
 case MergeType::TTLDrop:
 {
 /// Do not update anything for regular and drop merges.
 break;
 }
 case MergeType::TTLDelete:
 {
 const time_t next_time = current_time + (*settings)[MergeTreeSetting::merge_with_ttl_timeout];

and the setting itself:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:2199
 DECLARE(Int64, merge_with_ttl_timeout, 3600 * 4, R"(
Minimum delay in seconds before repeating a merge with delete TTL.
)", 0) \
 DECLARE(Int64, merge_with_recompression_ttl_timeout, 3600 * 4, R"(
Minimum delay in seconds before repeating a merge with recompression TTL.
)", 0) \
 DECLARE(Bool, ttl_only_drop_parts, false, R"(
Controls whether data parts are fully dropped in MergeTree tables when all
rows in that part have expired according to their `TTL` settings.

ttl_only_drop_parts is the other side of the same coin: by default ClickHouse rewrites a part to remove the expired rows from it, but with this setting on it only drops parts whose rows have all expired and leaves partially expired parts alone. It is the knob for the “I want the disk space back, I do not care about query-level precision” user — and, not coincidentally, the knob that makes TTL processing much cheaper, because no part is ever rewritten for a row TTL.

The mask

Now the interesting part. When the scheduler decides that a part must be rewritten because some of its rows have expired, the merge has to find out which rows. The naive implementation reads every column, evaluates the TTL expression, and builds a filter — which is what the horizontal merge path does:

 cpp
 
 
// src/Processors/TTL/TTLDeleteAlgorithm.cpp:32
 IColumn::Filter filter(rows);
 size_t removed = 0;

 for (size_t i = 0; i < rows; ++i)
 {
 Int64 cur_ttl = timestamps[i];
 bool where_filter_passed = !where_column || where_column->getBool(i);
 bool remove = isTTLExpired(cur_ttl) && where_filter_passed;

 filter[i] = !remove;

 if (remove)
 {
 ++removed;
 }
 else if (where_filter_passed)
 {
 /// Update ttl info only if row passes the filter.
 /// Rows that don't pass the filter should not affect TTL.
 new_ttl_info.update(cur_ttl);
 }
 }

 rows_removed += removed;

 if (removed == 0)
 return;

 for (auto & column : block)
 column.column = column.column->filter(filter, rows - removed);

That is a real physical deletion: every column is filtered and the block shrinks. It is the correct thing to do, and it is what a horizontal merge does.

But there is a much cheaper way, and it is the one ClickHouse actually uses whenever it can. The vertical merge already reads the rows, sorts them, and copies them into the output part column by column. If the merge knows in advance which rows are expired, it does not have to filter anything at all — it just does not copy them. The information is carried as a side-channel: a bit mask attached to the chunk.

 cpp
 
 
// src/Processors/Transforms/TTLDeleteFilterTransform.h:13
/// Evaluates TTL delete expressions and attaches the resulting mask to the chunk, without
/// filtering the block or changing its header. The merging algorithm reads the mask to set skip
/// flags in row sources for a vertical merge.
class TTLDeleteFilterTransform final : public ISimpleTransform

The mask itself is a RowFilterInfo, a chunk-info object that rides next to the data:

 cpp
 
 
// src/Processors/Merges/Algorithms/RowFilterInfo.h:10
/// Rows a merge must not emit, computed upstream and carried beside the data
struct RowFilterInfo : public ChunkInfoCloneable<RowFilterInfo>
{
 explicit RowFilterInfo(IColumnFilter mask_) : mask(std::move(mask_)) {}

 /// `IColumnFilter` is move-only, so `clone`'s deep copy has to be written by hand.
 RowFilterInfo(const RowFilterInfo & other) { mask.assign(other.mask); }

 /// Not supported: nothing combines two chunks that each carry a mask.
 Ptr merge(const Ptr &) const override;

 IColumnFilter mask;
};

The transform that builds it is deliberately minimal: it evaluates the TTL expression once per block, converts the result into a flat Int64 timestamp array (one type-dispatch, then a tight loop), and folds the expiry test and the WHERE test into a single UInt8 per row:

 cpp
 
 
// src/Processors/Transforms/TTLDeleteFilterTransform.cpp:117
 IColumnFilter filter_vec(num_rows, 1);

 auto chunk_infos = std::move(chunk.getChunkInfos());

 auto block = getInputPort().getHeader().cloneWithColumns(chunk.detachColumns());

 for (const auto & entry : shared_state->entries)
 {
 /// Phase 1: extract typed TTL column into a flat Int64 timestamp array.
 auto ttl_column = ITTLAlgorithm::executeExpressionAndGetColumn(
 entry.expressions.expression, block, entry.description.result_column);
 extractTimestamps(ttl_column.get());

 /// Phase 2: apply TTL expiration and WHERE filter to produce the filter mask.
 auto where_column = ITTLAlgorithm::executeExpressionAndGetColumn(
 entry.expressions.where_expression, block, entry.description.where_result_column);

 for (size_t i = 0; i < num_rows; ++i)
 {
 if (!filter_vec[i])
 continue;

 bool where_filter_passed = !where_column || where_column->getBool(i);
 if (isTTLExpired(timestamps[i], shared_state->current_time) && where_filter_passed)
 {
 filter_vec[i] = 0;
 }
 }
 }

 chunk = Chunk(block.getColumns(), num_rows);
 chunk.setChunkInfos(std::move(chunk_infos));
 attachRowFilter(chunk, std::move(filter_vec));

Two details are worth pausing on. First, the mask is not a filter of the block: the block still has all its rows and all its columns, and the header is unchanged. Second, the mask is not applied to the rows that a WHERE-conditioned TTL skips: filter_vec[i] starts at 1 and is only cleared when the row both expired and passed the WHERE test, so the semantics match the horizontal path exactly.

Before building the mask, the transform checks whether the whole part is expired. If the maximum expiry of the part is in the past and the TTL has no WHERE clause, every single row is expired and the whole part is garbage:

 cpp
 
 
// src/Processors/Transforms/TTLDeleteFilterTransform.cpp:56
 if (force || isTTLExpired(old_ttl_infos.table_ttl.min, current_time))
 {
 auto expressions = buildTTLExpressions(rows_ttl, subqueries, context);

 if (isTTLExpired(old_ttl_infos.table_ttl.max, current_time) && !rows_ttl.where_expression_ast)
 state->all_data_dropped = true;

 state->entries.push_back({std::move(expressions), rows_ttl});
 }

all_data_dropped short-circuits the transform to a mask of all zeros — “emit nothing” — and the merge emits no rows at all for that part.

The merge consumes the mask

The mask has to reach the merging algorithm. The algorithm resolves it once per chunk, from either of two sources: a stored _row_exists column (the lightweight delete) or the chunk’s RowFilterInfo (a TTL delete):

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:162
const IColumnFilter * MergingSortedAlgorithm::resolveRowFilterMask(const Chunk & chunk) const
{
 if (filter_column_position != -1)
 {
 /// One merge filters on one predicate: `canVerticalTTLDelete` gives up when a source part
 /// has a lightweight delete, so a merge never gets a mask from both directions.
 chassert(!chunk.getChunkInfos().has<RowFilterInfo>());
 return &assert_cast<const ColumnUInt8 &>(*chunk.getColumns()[filter_column_position]).getData();
 }

 return getRowFilterMask(chunk);
}

The comment says it all: the two mechanisms are mutually exclusive, because a merge filters on exactly one predicate.

From there the mask is consulted once per row, and the row is simply not copied:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:444
void MergingSortedAlgorithm::insertRow(const SortCursorImpl & current)
{
 auto write_row_source = [&](bool skipped)
 {
 if (out_row_sources_buf)
 {
 RowSourcePart row_source(current.order, skipped);
 out_row_sources_buf->write(row_source.data);
 }
 };

 size_t current_row = current.getRow();

 if (const auto * mask = source_row_filter_masks[current.order])
 {
 if ((*mask)[current_row])
 merged_data.insertRow(current.all_columns, current_row, current.rows);

 write_row_source(!(*mask)[current_row]);
 }
 else
 {
 merged_data.insertRow(current.all_columns, current_row, current.rows);
 write_row_source(false);
 }
}

That is the whole trick. The expired row is never written to the output column, so the output part is physically smaller — but nothing was filtered and no column was re-serialised because of a row that was dropped. The mask is a UInt8 per row, computed once, and it is read once.

The batch path does the same, and additionally records the skip in the row-source file, which is what lets the vertical phase of a merge know that a row of a gathering column has no counterpart in the merged column set:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:471
void MergingSortedAlgorithm::insertRows(const SortCursorImpl & current, size_t num_rows)
{
 if (const auto * mask = source_row_filter_masks[current.order])
 {
 size_t start_index = current.getRow();
 RowSourcePart row_source(current.order, false);
 RowSourcePart row_source_skipped(current.order, true);

 for (size_t i = start_index; i < start_index + num_rows; ++i)
 {
 if ((*mask)[i])
 {
 merged_data.insertRow(current.all_columns, i, current.rows);
 out_row_sources_buf->write(row_source.data);
 }
 else
 {
 out_row_sources_buf->write(row_source_skipped.data);
 }
 }
 }

The RowSourcePart with the skipped flag set is exactly the same mechanism the lightweight delete uses, so a TTL delete and a lightweight delete end up producing the same kind of part.

When the mask is not enough

The mask trick only works when the merge is allowed to not copy a row, and that has preconditions. canVerticalTTLDelete is the gate:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:1454
bool MergeTask::canVerticalTTLDelete(const GlobalRuntimeContext & global_ctx)
{
 switch (global_ctx.merging_params.mode)
 {
 case MergeTreeData::MergingParams::Ordinary:
 case MergeTreeData::MergingParams::Replacing:
 case MergeTreeData::MergingParams::Collapsing:
 case MergeTreeData::MergingParams::VersionedCollapsing:
 break;
 case MergeTreeData::MergingParams::Summing:
 case MergeTreeData::MergingParams::Aggregating:
 case MergeTreeData::MergingParams::Coalescing:
 case MergeTreeData::MergingParams::Graphite:
 return false;
 }

 if (!(*global_ctx.data_settings)[MergeTreeSetting::vertical_merge_optimize_ttl_delete])
 return false;

 if (global_ctx.metadata_snapshot->hasAnyGroupByTTL())
 return false;

 if (global_ctx.metadata_snapshot->hasAnyColumnTTL())
 return false;

 if (hasLightweightDelete(global_ctx.future_part))
 return false;

 return global_ctx.metadata_snapshot->hasRowsTTL() || global_ctx.metadata_snapshot->hasAnyRowsWhereTTL();
}

The list of exclusions is the list of reasons a row cannot simply be skipped:

Summing / Aggregating / Coalescing / Graphite — these engines combine rows. A skipped row would change the aggregate, so the merge has to see it and decide. (For a GROUP BY TTL the same is true: the TTL is applied to an aggregate of the rows, so the rows must be present when the aggregate is formed.)
Column TTL — a column TTL resets a column’s value to its default. The rows stay, the column changes, so the part must be rewritten with different data, not with fewer rows.
A lightweight delete in the same merge — one merge filters on one predicate.

When the gate is closed, the merge falls back to the horizontal path, where TTLTransform physically filters the block and the rows are really gone.

There is one more case worth mentioning: a part whose rows have all expired. The merge does not need to read anything at all — it just writes an empty part and drops the old one. ClickHouse short-circuits the whole pipeline for that:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:1261
 /// Reset all ttl_infos and mark table TTL as finished, matching what
 /// TTLTransform::finalize produces in the normal all_data_dropped path:
 /// it clears everything with `data_part->ttl_infos = {}`, then
 /// TTLDeleteAlgorithm::finalize sets table_ttl = {0, 0, finished}.
 global_ctx->new_data_part->ttl_infos = {};
 global_ctx->new_data_part->ttl_infos.table_ttl = {0, 0, true};

The gotcha: TTL is invisible until a merge

The consequence of all this is the thing that surprises everybody the first time: a TTL does not delete anything on its own. If you insert a row that is already expired and then query the table, the row is still there. It will still be there after an hour, after a day. The row disappears when a merge rewrites the part that contains it — and a merge happens when the merge scheduler decides it should, which may be minutes or may be days.

The same is true in the other direction: if a part is never merged (because it is the only part in its partition and is too small to be worth merging), the expired rows stay in it forever. TTL is a merge-triggered deletion, not a timer.

If you need the effect now, the tool is MATERIALIZE TTL, which is a mutation:

 sql
 
 
-- Force the TTL to be applied to every part, right now.
ALTER TABLE events MATERIALIZE TTL;

-- Recompute only the stored TTL metadata, without removing any data.
-- (merge_tree setting `materialize_ttl_recalculate_only = 1`)
ALTER TABLE events MATERIALIZE TTL;

The mutation rewrites every part with the TTL rules applied — the same TTLTransform machinery, but driven by the mutation engine instead of the merge scheduler. And it is expensive, which is why it is not what the background merger does: the background merger prefers to piggyback the deletion on a merge that was going to happen anyway.

There is a second, subtler gotcha in the same area: the TTL bound of a part is only valid if the TTL expression was evaluated for it. A part written by an older version, or attached from a backup, may not have a ttl.txt at all. The merge detects this and forces the calculation:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:689
 if (global_ctx->metadata_snapshot->hasAnyTTL() && !part->checkAllTTLCalculated(global_ctx->metadata_snapshot))
 {
 LOG_INFO(ctx->log, "Some TTL values were not calculated for part {}. Will calculate them forcefully during merge.", part->name);
 ctx->need_remove_expired_values = true;
 ctx->force_ttl = true;
 }

Trade-offs

What this design buys:

No write amplification from a timer. A TTL costs one timestamp array per inserted block, and two integers per part on disk. The expensive work is folded into merges that were going to happen anyway.
Deletion is proportional to the merge, not to the expiry rate. A table with a TTL that expires 1 % of its rows per day does 1 % less work than one that expires 100 %, because the second one is rewritten far more often — but the per merge cost is the same.
The mask is nearly free. One UInt8 per row, computed once, read once. The rows are not filtered, re-serialised, or re-compressed; they are simply not copied.

What it costs:

The semantics are merge-dependent. “Rows expired 30 days ago” is not a guarantee about a query result; it is a guarantee about the next merge. Two replicas that have merged different amounts of data can legitimately return different row counts for the same query.
Latency. A part that is never merged never expires. MATERIALIZE TTL exists precisely because the user sometimes needs the guarantee now, and it is expensive.
Throttling. The four-hour merge_with_ttl_timeout means that even a table whose data expires continuously will keep a few hours’ worth of expired rows around, deliberately.
The mask trick has a whitelist. Summing, Aggregating, Coalescing, Graphite and GROUP BY TTLs cannot use it, because their rows have to be seen to be combined or aggregated. For those, the deletion is a real rewrite.

The design is a deliberate trade: ClickHouse gives up the guarantee that an expired row is gone, and in exchange makes the common case (a TTL that is a background housekeeping rule) essentially free. The MATERIALIZE TTL mutation is the escape hatch that restores the strong guarantee when a user needs it, and the price of that escape hatch is a full rewrite of every part.

How ReplacingMergeTree Collapses Duplicates and What SELECT FINAL Really Does

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: A ReplacingMergeTree merge keeps one row per sorting key: the one with the highest version, or the last one inserted if the versions tie. A delete is a row with is_deleted = 1 and a high version. SELECT FINAL does not look at the merged state — it re-runs the same collapse at query time over every part, which is why it is expensive.

The problem: an update is a delete plus an insert

ClickHouse’s storage model has no UPDATE. A MergeTree part is immutable; the only way to change a row is to write a new row and let a merge throw the old one away.

For a slowly-changing dimension — a user’s profile, a product’s price, a device’s last-seen timestamp — that is the natural model. The user wants:

For each key, show me the newest version of the row.

ReplacingMergeTree is the engine that does this. Its merge algorithm is simple to state and fiddly to implement:

Group rows by sorting key.
Within a group, keep exactly one.
Which one? The one with the highest version. If versions tie, the last inserted.

The engine declaration

ReplacingMergeTree is a MergeTree with two extra optional parameters: a version column and an is_deleted column.

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.h:478
 struct MergingParams
 {
 /// Merging mode. See above.
 enum Mode
 {
 Ordinary = 0, /// Enum values are saved. Do not change them.
 Collapsing = 1,
 Summing = 2,
 Aggregating = 3,
 Replacing = 5,
 Graphite = 6,
 VersionedCollapsing = 7,
 Coalescing = 8,
 };

 Mode mode{};

 /// For Collapsing and VersionedCollapsing mode.
 String sign_column;

 /// For Replacing mode. Can be empty for Replacing.
 String is_deleted_column;

 /// For Summing mode. If empty - columns_to_sum is determined automatically.
 Names columns_to_sum;

 /// For Replacing and VersionedCollapsing mode. Can be empty for Replacing.
 String version_column;

And the parameters are parsed positionally from the engine arguments, which is a small parsing puzzle — the last argument may be the version column or the is_deleted column, and you cannot tell them apart without knowing how many there are:

 cpp
 
 
// src/Storages/MergeTree/registerStorageMergeTree.cpp:633
 else if (merging_params.mode == MergeTreeData::MergingParams::Replacing)
 {
 // if there is args and number of optional parameter is higher than 1
 // is_deleted is not allowed with the 'allow_deprecated_syntax_for_merge_tree' settings
 if (arg_cnt - arg_num == 2 && !engine_args[arg_cnt - 1]->as<ASTLiteral>() && is_extended_storage_def)
 {
 if (!tryGetIdentifierNameInto(engine_args[arg_cnt - 1], merging_params.is_deleted_column))
 throw Exception(ErrorCodes::BAD_ARGUMENTS, "is_deleted column name must be an identifier {}", verbose_help_message);
 --arg_cnt;
 }

 /// If the last element is not index_granularity or replica_name (a literal), then this is the name of the version column.
 if (arg_cnt && !engine_args[arg_cnt - 1]->as<ASTLiteral>())
 {
 if (!tryGetIdentifierNameInto(engine_args[arg_cnt - 1], merging_params.version_column))
 throw Exception(ErrorCodes::BAD_ARGUMENTS, "Version column name must be an identifier {}", verbose_help_message);
 --arg_cnt;
 }
 }

Note the order: is_deleted is peeled off first (when there are two identifier arguments), then version. But the documented syntax is ReplacingMergeTree(ver, is_deleted) — so the last argument is is_deleted and the one before it is the version. The code peels is_deleted from the end and then version from the new end, which is consistent.

The types are checked at creation time:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:2410
 if (mode == MergingParams::Replacing)
 {
 if (!version_column.empty() && version_column == is_deleted_column)
 throw Exception(ErrorCodes::BAD_ARGUMENTS, "The version and is_deleted column cannot be the same column ({})", version_column);

 check_is_deleted_column(true, "ReplacingMergeTree");
 check_version_column(true, "ReplacingMergeTree");
 }

is_deleted must be UInt8 and the version column must satisfy canBeUsedAsVersion() — an integer or a date type. is_deleted also requires a version column:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:2316
 /// Check that if the is_deleted column is needed, it exists and is of type UInt8. If exist, version column must be defined too but version checks are not done here.
 auto check_is_deleted_column = [this, & columns](bool is_optional, const std::string & storage)
 {
 if (is_deleted_column.empty())
 {
 if (is_optional)
 return;

 throw Exception(ErrorCodes::LOGICAL_ERROR, "`is_deleted` ({}) column for storage {} is empty", is_deleted_column, storage);
 }

 if (version_column.empty() && !is_optional)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Version column ({}) for storage {} is empty while is_deleted ({}) is not.",
 version_column, storage, is_deleted_column);

That is a semantic requirement, not a syntactic one: a delete row without a version cannot be ordered against the rows it is supposed to replace.

A typical table looks like this:

 sql
 
 
CREATE TABLE user_profile
(
 user_id UInt64,
 name String,
 email String,
 updated_at DateTime, -- the version column
 is_deleted UInt8 -- the tombstone flag
)
ENGINE = ReplacingMergeTree(updated_at, is_deleted)
ORDER BY user_id;

An update is then two inserts: the new row with a higher updated_at, and — if the user was deleted — a row with is_deleted = 1 and an even higher updated_at.

The merge algorithm

The merge is implemented as a merging algorithm over the sorted streams of the parts being merged. The heart of it is the version comparison:

 cpp
 
 
// src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:269
 bool replace_with_current_row = false;
 if (selected_row.empty())
 {
 replace_with_current_row = true;
 }
 else
 {
 /// Three-way comparison of the current row's version with the selected row's version.
 /// Without a version column all rows count as having equal versions, so the selection
 /// falls through to the physical-order rule below.
 int version_cmp = version_column_number == -1 ? 0
 : current->all_columns[version_column_number]->compareAt(
 current->getRow(), selected_row.row_num,
 *(*selected_row.all_columns)[version_column_number],
 /* nan_direction_hint = */ 1);

 if (version_cmp > 0)
 {
 replace_with_current_row = true;
 }
 else if (version_cmp == 0)
 {
 /// Rows with equal versions are selected by their physical order: the row written last wins.
 /// The queue emits rows with equal sort key ordered by source index, i.e. by data part
 /// (parts are ordered from the oldest to the newest one), and within one source in the reading order.
 /// In the direct reading order the current row is always "newer", so it replaces the selected one
 /// (a non-strict comparison in terms of the version). In the reverse reading order rows within
 /// one source arrive backwards, so the current row replaces the selected one only when
 /// it comes from a newer data part.
 chassert(current_row.source_stream_index >= selected_row.source_stream_index);
 replace_with_current_row = !read_in_reverse || current_row.source_stream_index > selected_row.source_stream_index;
 }
 }

Three things to unpack.

The comparison is three-way, not “greater than”. compareAt returns -1, 0 or +1, and the tie case is handled separately. A naive if (current_version > selected_version) would keep the first row on a tie; the three-way form lets the tie fall through to a physical-order rule.

The tie-break is physical order, and the direction matters. The queue emits rows with equal sort keys ordered by source index, and source index 0 is the oldest part. So “the current row replaces the selected one” means “the newer part wins”, which is the intuitive behaviour. But in reverse reading order, rows within one source arrive backwards, so the naive rule would pick the oldest row of the newest part. The read_in_reverse flag flips the comparison accordingly: within one part, the first row encountered in reverse order wins.

The nan_direction_hint = 1 matters. The version column’s type only has to satisfy canBeUsedAsVersion(), and DataTypeNumber<T> — which includes Float32 and Float64 — says yes:

 cpp
 
 
// src/DataTypes/DataTypesNumber.h:26
 bool canBeUsedAsVersion() const override { return true; }

So a float version column is legal, and float versions can be NaN. The nan_direction_hint tells compareAt how to order NaN against other values, so that a NaN version does not produce an inconsistent ordering between the merge and the query path.

Now, the delete flag. A row with is_deleted = 1 is a tombstone, and with cleanup enabled it is dropped rather than kept:

 cpp
 
 
// src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:89
/// True when the group's winning row must not reach the output: a `CLEANUP` tombstone, or a row
/// the filter rejects.
bool ReplacingSortedAlgorithm::isSelectedRowSkipped() const
{
 if (cleanup && is_deleted_column_number != -1
 && assert_cast<const ColumnUInt8 &>(*(*selected_row.all_columns)[is_deleted_column_number]).getData()[selected_row.row_num])
 return true;

 return isRowFiltered(selected_row);
}

Note cleanup &&. Without cleanup, the tombstone wins the group and is emitted as the surviving row — the row is still there, marked deleted. This is the documented default behaviour, and it is deliberate:

 cpp
 
 
// src/Storages/MergeTree/registerStorageMergeTree.cpp:3166
By default, ClickHouse will keep the last row for a key even if that row is a delete row. This is so that any future rows with lower versions can
be safely inserted and the delete row will still be applied.

The reasoning is about out-of-order arrival. If a delete with version 100 arrives, and later a row with version 50 arrives (a late-arriving update), the delete must still win — otherwise the resurrect would be applied. Keeping the tombstone preserves the ability to re-apply it. OPTIMIZE ... FINAL CLEANUP (or the allow_experimental_replacing_merge_with_cleanup setting) is what actually removes the tombstone rows.

The skip-run optimization

The merge algorithm has a fast path that is worth understanding, because it explains a lot of ClickHouse’s performance characteristics.

If there is no version column, no is_deleted column, and no row-source bookkeeping, then processing a row has no effect other than replacing selected_row with it. So a whole run of equal keys can be skipped — jump straight to the last row of the run:

 cpp
 
 
// src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:56
 /// With a version or an is_deleted column every row of a run must be examined, and row
 /// sources for a vertical merge must be recorded per row. Without them the only effect of
 /// processing a row is replacing `selected_row` with it, so runs can be fast-forwarded.
 /// In the reverse reading order the first row of a run within a source wins instead of the
 /// last one, so the fast-forward to the last row of the run does not apply either.
 can_skip_to_run_end = version_column_number == -1 && is_deleted_column_number == -1
 && out_row_sources_buf == nullptr && !enable_vertical_final && !read_in_reverse;
 uses_runs_of_equal_keys = can_skip_to_run_end;

And the actual jump:

 cpp
 
 
// src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:310
 /// All rows of one batch come consecutively from the same cursor. When only the last
 /// row of a run of equal keys is kept and processing a row has no other effects, jump
 /// to the last row of the run of the current key within the batch: the intermediate
 /// rows would each merely replace `selected_row` with the next one. Sources with a
 /// non-zero part level are excluded to keep the exact behavior of
 /// `rowsHaveDifferentSortColumns`, which does not compare rows of such sources at all.
 if (skip_runs_of_equal_keys && current_batch_size > 1 && !current->permutation
 && sources_origin_merge_tree_part_level[current->order] == 0)
 {
 size_t run_begin = current->getPos();
 size_t run_bound = run_begin + current_batch_size;

 /// The last row of the cursor may need to be skipped, leave it to the per-row check.
 if (run_bound == current->getSize() && skipLastRowFor(current->order))
 --run_bound;

 if (run_begin + 1 < run_bound)
 {
 auto & key_runs = source_key_runs[current->order];
 if (key_runs.end != run_bound)
 {
 key_runs.runs.reset(current->desc.size());
 key_runs.end = run_bound;
 }
 size_t run_end = getEqualRangeEndAssumeSorted(key_runs.runs, current->sort_columns, current->desc, run_begin, run_bound);
 if (run_end > run_begin + 1)
 {
 /// Jump to the last row of the run; the loop processes it as usual.
 queue.next(run_end - 1 - run_begin);
 continue;
 }
 }
 }

The sources_origin_merge_tree_part_level[current->order] == 0 guard is subtle and important: a part at level > 0 was produced by a merge, so it already has one row per key — there are no duplicate runs in it to skip, and skipping them would change the result of rowsHaveDifferentSortColumns.

The skip_runs_of_equal_keys flag has a further precondition:

 cpp
 
 
// src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:67
void ReplacingSortedAlgorithm::initialize(Inputs inputs)
{
 IMergingAlgorithmWithSharedChunks::initialize(std::move(inputs));

 /// Skipping runs needs the queue to actually detect batches. A batch longer than one row is
 /// not evidence of that on its own: a queue with one cursor left always reports its whole
 /// remainder as one batch, since there is nothing to compare it against. Without this
 /// condition the probe for the end of a run would run on every row of a single-input merge -
 /// which is every `INSERT` into a `ReplacingMergeTree` with `optimize_on_insert` (on by
 /// default), where `MergeTreeDataWriter::mergeBlock` merges one already sorted block. When
 /// that block holds no runs of equal keys, the detection is disabled and the merge has to
 /// cost exactly what it costs with the plain heap.
 skip_runs_of_equal_keys = can_skip_to_run_end && batch_detection_enabled;
 source_key_runs.assign(sources.size(), {});
}

This is a good example of the kind of detail that makes a fast path fast: the probe for “where does this run end” is itself O(run length) in the worst case, so it must be disabled when there is nothing to gain. A single-input merge — which is what optimize_on_insert does — would otherwise pay the probe cost on every row.

What `SELECT FINAL` actually does

Here is the crucial point, and it is the one most users get wrong.

SELECT FINAL does not read the merged state of the table. The parts on disk are exactly as they were written. FINAL re-runs the collapse at query time, over the raw parts, and only then applies the predicate.

The pipe construction makes it explicit — FINAL reads all parts and inserts a merging transform:

 cpp
 
 
// src/Processors/QueryPlan/MergeTreeFinalMerge.cpp:84
 case MergeTreeData::MergingParams::Replacing:
 return std::make_shared<ReplacingSortedTransform>(header, num_outputs,
 sort_description, merging_params.is_deleted_column, merging_params.version_column, max_block_size_rows, /*max_block_size_bytes=*/0, /*max_dynamic_subcolumns*/std::nullopt, /*out_row_sources_buf_*/ nullptr, /*use_average_block_sizes*/ false, /*cleanup*/ !merging_params.is_deleted_column.empty(), enable_vertical_final, read_in_reverse);

and the full-merge pipe reads everything in order and merges it:

 cpp
 
 
// src/Processors/QueryPlan/MergeTreeFinalMerge.cpp:164
Pipe buildFullFinalMergePipe(
 const StorageMetadataPtr & metadata_snapshot,
 MergeTreeData::MergingParams merging_params,
 size_t max_block_size_rows,
 bool enable_vertical_final,
 ContextPtr context,
 std::optional<ActionsDAG> & out_projection,
 const std::function<Pipe()> & read_all_parts_in_order)
{
 auto sorting_expr = metadata_snapshot->getSortingKey().expression;
 auto sort_description = buildFinalSortDescription(metadata_snapshot, context->getSettingsRef());

 Pipe pipe = read_all_parts_in_order();
 if (pipe.empty())
 return pipe;

 pipe.addSimpleTransform([sorting_expr](const SharedHeader & header)
 { return std::make_shared<ExpressionTransform>(header, sorting_expr); });

So FINAL costs a full merge of every part at query time, with all the memory and CPU that implies, and it happens on every query. OPTIMIZE TABLE ... FINAL pays the same cost once, in the background, and writes the result down.

There is a special case for is_deleted, though. When a part range does not intersect any other part range — no key can have a duplicate — the merge is skipped, and the is_deleted filter has to be applied by hand, because the merging algorithm would have applied it:

 cpp
 
 
// src/Processors/QueryPlan/MergeTreeFinalMerge.cpp:279
Pipe readNonIntersectingFinalWithEngineFilter(
 const MergeTreeData::MergingParams & merging_params,
 const Names & origin_column_names,
 ContextPtr context,
 const std::function<Pipe(const Names & columns)> & read)
{
 /// `Collapsing` does not expose unmatched negative-sign rows on FINAL, and `Replacing` with an is-deleted
 /// column hides deleted rows; a non-intersecting range skips the merge, so add that filter here. Other
 /// engines drop nothing on FINAL beyond the deduplication a single part already did.
 if (merging_params.mode == MergeTreeData::MergingParams::Collapsing)
 {
 auto columns_with_sign = origin_column_names;
 if (std::ranges::find(columns_with_sign, merging_params.sign_column) == columns_with_sign.end())
 columns_with_sign.push_back(merging_params.sign_column);
 Pipe pipe = read(columns_with_sign);
 auto [expression, filter_name] = createExpressionForPositiveSign(merging_params.sign_column, pipe.getHeader(), context);
 pipe.addSimpleTransform([&](const SharedHeader & header)
 { return std::make_shared<FilterTransform>(header, expression, filter_name, true); });
 return pipe;
 }
 if (!merging_params.is_deleted_column.empty())
 {
 auto columns_with_is_deleted = origin_column_names;
 if (std::ranges::find(columns_with_is_deleted, merging_params.is_deleted_column) == columns_with_is_deleted.end())
 columns_with_is_deleted.push_back(merging_params.is_deleted_column);
 Pipe pipe = read(columns_with_is_deleted);
 auto [expression, filter_name] = createExpressionForIsDeleted(merging_params.is_deleted_column, pipe.getHeader(), context);
 pipe.addSimpleTransform([&](const SharedHeader & header)
 { return std::make_shared<FilterTransform>(header, expression, filter_name, true); });
 return pipe;
 }
 return read(origin_column_names);
}

is_deleted = 0 is the filter. A tombstone is hidden on FINAL — which is the opposite of the merge’s default behaviour, where the tombstone wins the group and is emitted.

Vertical FINAL

The default FINAL implementation copies the surviving row’s data into the output. enable_vertical_final avoids that: instead of merging the data, the algorithm records which row indices survived, and a later transform picks them out of the original chunks:

 cpp
 
 
// src/Processors/Merges/Algorithms/ReplacingSortedAlgorithm.cpp:127
void ReplacingSortedAlgorithm::insertRowImpl()
{
 flushCurrentRowSources(/*keep_selected_row=*/ true);

 if (enable_vertical_final)
 {
 /// We just record the position to be selected in the chunk
 if (!selected_row.owned_chunk->replace_final_selection)
 selected_row.owned_chunk->replace_final_selection = ColumnUInt64::create();

 selected_row.owned_chunk->replace_final_selection->insert(selected_row.row_num);

 /// This is the last row we can select from `selected_row.owned_chunk`, keep it to emit later
 if (selected_row.current_cursor == nullptr)
 to_be_emitted.push(std::move(selected_row.owned_chunk));
 }
 else
 {
 merged_data->insertRow(*selected_row.all_columns, selected_row.row_num, selected_row.owned_chunk->getNumRows());
 }

 selected_row.clear();
}

The chunks carry a side-channel column of surviving row indices, and a SelectByIndicesTransform materializes the result at the end:

 cpp
 
 
// src/Processors/QueryPlan/MergeTreeFinalMerge.cpp:106
 pipe.addTransform(get_merging_processor());
 if (enable_vertical_final)
 pipe.addSimpleTransform([](const SharedHeader & header_)
 { return std::make_shared<SelectByIndicesTransform>(header_); });

The win: the merge does not have to copy the surviving rows’ column data. The data stays in the original chunks, and only an index is recorded. For a table where most rows survive, that saves a full copy of the table.

The cost: the can_skip_to_run_end fast path is disabled, because a vertical final has to record the position of every surviving row, and the row-source bookkeeping has to be per-row.

The lazy FINAL: replacing a merge with an aggregation

There is a much more aggressive optimization, and it is the most interesting part of this story. It is enabled when the query has a WHERE clause with a selective primary-key condition, and the engine can prove that only a small set of keys can possibly have duplicates.

Instead of merging all the parts, the engine:

Reads only the rows matching the primary-key condition, projecting just the primary key columns.
Builds a Set of those keys.
Uses that set to figure out which parts actually need to be merged.
For the parts that do, computes the winner with an argMax aggregation rather than a merge.

The tiebreaker is the interesting bit. The winner of a group is “the row with the highest version; on a tie, the one inserted last”. Insert order is captured by a global row index, and the two are packed into a single comparable value:

 cpp
 
 
// src/Processors/Sources/LazyReadReplacingFinalSource.cpp:143
 /// When there's a version column, compute a tiebreaker for argMax so that
 /// equal versions resolve to the last inserted row (highest __global_row_index).
 /// For version types ≤ 64 bits, pack (version, __global_row_index) into UInt128.
 /// For wider types, use tuple(version, __global_row_index).
 /// When there's no version column, just use max(__global_row_index).
 bool has_version = !merging_params.version_column.empty();
 static constexpr auto tiebreaker_column_name = "__lazy_final_tiebreaker";

The packing, for a 64-bit-or-narrower version:

 cpp
 
 
// src/Processors/Sources/LazyReadReplacingFinalSource.cpp:184
 const auto * version_128 = &dag.addFunction(to_uint128, {version_node}, {});
 const auto * row_index_128 = &dag.addFunction(to_uint128, {row_index_node}, {});
 auto shift_type = std::make_shared<DataTypeUInt8>();
 auto shift_column = shift_type->createColumnConst(0, Field(UInt8(64)));
 const auto * shift_amount = &dag.addColumn(std::move(shift_column), std::move(shift_type), "__shift_64");
 const auto * shifted = &dag.addFunction(bit_shift_left, {version_128, shift_amount}, {});
 const auto * tiebreaker = &dag.addFunction(plus_func, {shifted, row_index_128}, {});
 tiebreaker = &dag.addAlias(*tiebreaker, tiebreaker_column_name);

(version << 64) + row_index in a UInt128. The version occupies the high 64 bits, the row index the low 64, so a plain unsigned comparison of the tiebreaker is exactly “compare versions, then compare insertion order”.

Signed versions need a fixup, because the sign bit would otherwise make every negative version look larger than every positive one:

 cpp
 
 
// src/Processors/Sources/LazyReadReplacingFinalSource.cpp:168
 /// For signed types (and Date32 which is Int32 internally),
 /// widen to Int64 first, then flip the sign bit
 /// to convert signed order to unsigned order.
 if (which.isNativeInt() || which.isDate32())
 {
 auto to_int64 = FunctionFactory::instance().get("toInt64", nullptr);
 auto reinterpret_func = FunctionFactory::instance().get("reinterpretAsUInt64", nullptr);
 auto bitxor_func = FunctionFactory::instance().get("bitXor", nullptr);
 auto sign_bit_type = std::make_shared<DataTypeUInt64>();
 auto sign_bit_column = sign_bit_type->createColumnConst(0, Field(UInt64(1) << 63));
 const auto * sign_bit_node = &dag.addColumn(std::move(sign_bit_column), std::move(sign_bit_type), "__sign_bit");
 const auto * version_int64 = &dag.addFunction(to_int64, {version_node}, {});
 const auto * version_uint64 = &dag.addFunction(reinterpret_func, {version_int64}, {});
 version_node = &dag.addFunction(bitxor_func, {version_uint64, sign_bit_node}, {});
 }

x ^ (1 << 63) is the standard “flip the sign bit” transform that maps signed order onto unsigned order. For versions wider than 64 bits, the tiebreaker falls back to a tuple:

 cpp
 
 
// src/Processors/Sources/LazyReadReplacingFinalSource.cpp:194
 else
 {
 auto tuple_func = FunctionFactory::instance().get("tuple", query_context);
 const auto * tiebreaker = &dag.addFunction(tuple_func, {version_node, row_index_node}, {});
 tiebreaker = &dag.addAlias(*tiebreaker, tiebreaker_column_name);
 dag.getOutputs().push_back(tiebreaker);
 }

And the aggregation itself:

 cpp
 
 
// src/Processors/Sources/LazyReadReplacingFinalSource.cpp:215
 if (has_version)
 {
 auto tiebreaker_type = header->getByName(tiebreaker_column_name).type;
 argmax_row_index_name = fmt::format("argMax(__global_row_index, {})", tiebreaker_column_name);

 /// argMax(__global_row_index, __tiebreaker)
 AggregateDescription desc;
 desc.function = AggregateFunctionFactory::instance().get(
 "argMax", NullsAction::EMPTY, {uint64_type, tiebreaker_type}, {}, properties);
 desc.argument_names = {"__global_row_index", tiebreaker_column_name};
 desc.column_name = argmax_row_index_name;
 aggregates.push_back(std::move(desc));

 /// argMax(is_deleted, __tiebreaker) if is_deleted column exists
 if (!merging_params.is_deleted_column.empty())
 {
 argmax_is_deleted_name = fmt::format("argMax({}, {})", merging_params.is_deleted_column, tiebreaker_column_name);
 auto is_deleted_type = header->getByName(merging_params.is_deleted_column).type;

Two argMax aggregates: one for the winning row’s global index, one for the winning row’s is_deleted flag. Both use the same tiebreaker, so they agree on which row won. The result is a grouped aggregation over the primary key with an argMax — which is a hash-table aggregate, not a sort-merge.

The whole optimization is gated on being able to prove the merge is unnecessary for most parts. The optimizeLazyFinal pass splits the parts into “intersecting” (key ranges overlap, so a merge is needed) and “non-intersecting” (no overlap, so no duplicate is possible):

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLazyFinal.cpp:489
 /// Split parts into non-intersecting (unique key ranges, no FINAL needed) and
 /// intersecting (overlapping, need FINAL). This avoids running the expensive
 /// aggregation-based FINAL on parts that have no duplicates.
 /// When all parts are non-intersecting, replaceNodeWithPlan is called inside
 /// and fully_replaced is set — in that case we're done.
 auto split_result = trySplitNonIntersectingParts(
 reading_step, analyzed_result, filter_step, read_node, query_plan, /*allow_partial_split=*/ !stops_reading_early);

The gate is a ratio — the optimization only pays off if enough of the table is pruned:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLazyFinal.cpp:509
 /// Use parts from the analyzed result (possibly narrowed to intersecting-only by the split).
 /// These are PK-filtered parts with narrowed mark ranges from selectRangesToRead.
 auto parts_for_set = std::make_shared<RangesInDataParts>(
 analyzed_result ? analyzed_result->parts_with_ranges : reading_step->getParts());

The set is built from the primary key columns only — the PK is a prefix of the sorting key, so the remaining sorting key columns are useless for deciding which keys have duplicates:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLazyFinal.cpp:514
 /// Build the set for primary key columns only (PK is a prefix of sorting key;
 /// the remaining sorting key columns are useless for index analysis).
 SizeLimits set_size_limits(optimization_settings.max_rows_for_lazy_final, optimization_settings.max_bytes_for_lazy_final, OverflowMode::BREAK);

And there is a hard limit on the set size, so the optimization cannot blow up memory on a broad query.

Reverse-order reading

One more modern wrinkle. SELECT FINAL with ORDER BY on a descending sorting key can be answered by reading the parts backwards and taking the first row of each group rather than the last:

 cpp
 
 
// src/Processors/QueryPlan/ReadFromMergeTree.cpp:2481
 const bool read_in_reverse = reader_settings.read_in_order && query_info.input_order_info && query_info.input_order_info->direction < 0;
 const auto read_type = read_in_reverse ? ReadType::InReverseOrder : ReadType::InOrder;

and the merging algorithm has to be told, because the tie-break rule flips:

 cpp
 
 
// src/Processors/QueryPlan/MergeTreeFinalMerge.cpp:56
 /// Only the Replacing merging algorithm supports reading in the reverse order
 /// (see `ReadFromMergeTree::requestReadingInOrder`).
 if (read_in_reverse && merging_params.mode != MergeTreeData::MergingParams::Replacing)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Reading in reverse order with FINAL is supported only for ReplacingMergeTree");

Only Replacing supports it, because only Replacing’s tie-break rule is “the row written last wins” — a rule that can be evaluated in either direction. Collapsing and Summing aggregate, and an aggregate is not direction-independent.

Trade-offs

ReplacingMergeTree is eventually consistent. The collapse happens during merges, which happen in the background, on a schedule you do not control. Until a merge happens, the table physically contains both the old and the new row. A query without FINAL sees both. The documentation says it plainly:

 cpp
 
 
// src/Storages/MergeTree/registerStorageMergeTree.cpp:3239
At merge time, the ReplacingMergeTree identifies duplicate rows, using the values of the `ORDER BY` columns (used to create the table) as a unique identifier, and retains only the highest version. This, however, offers eventual correctness only - it does not guarantee rows will be deduplicated, and you should not rely on it. Queries can, therefore, produce incorrect answers due to update and delete rows being considered in queries.

FINAL is expensive. It re-merges at query time. The lazy-FINAL optimization helps a lot when the query is selective, but a FINAL over the whole table is a full merge on every query.

FINAL changes what predicates mean. A predicate is applied after the collapse, so SELECT ... WHERE version = 5 FINAL finds the row whose winning version is 5 — not a row that had version 5 and lost. Without FINAL, the same predicate might match the loser.

The difference is visible in one query:

 sql
 
 
-- Without FINAL, both versions are present until a merge happens.
SELECT user_id, updated_at FROM user_profile WHERE user_id = 42;

-- With FINAL, only the winner survives.
SELECT user_id, updated_at FROM user_profile WHERE user_id = 42 FINAL;

The version must be monotonic. “No matter the operation on the data, the version should be increased.” A version that goes backwards resurrects stale data, and there is no way for ClickHouse to detect that.

Deletes are two-phase. By default the tombstone is kept (so that late lower-version rows cannot resurrect the row), and only OPTIMIZE ... FINAL CLEANUP removes it. Space is held until then.

The bottom line

ReplacingMergeTree is the smallest interesting member of the MergeTree family: one row per sorting key, the highest version wins, ties broken by insertion order. The implementation is a merging algorithm that has to be careful about three things — the three-way version comparison, the tie-break rule’s dependence on read direction, and the “no false skip” requirement that makes the run-skipping fast path safe.

SELECT FINAL is where the interesting engineering lives, because it has to reproduce a merge’s result at query time. ClickHouse’s progression here is instructive: from “merge everything on every query”, to “merge everything but copy only the survivors” (vertical final), to “prove that most parts need no merge at all, and compute the winners with a grouped argMax over a UInt128 tiebreaker” (lazy final). Each step trades a specific, well-understood correctness argument for a large constant-factor win — and the tiebreaker packing, (version << 64) + row_index, with the sign-bit flip for signed versions, is the kind of detail that only works because someone reasoned about it very carefully.

How ClickHouse Spreads Parts Across Disks: the Storage Policy

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: A storage policy is a list of volumes, each holding one or more disks. Writing a part calls reserve() down that ladder: the policy tries volumes in priority order, a volume tries its disks (round-robin or least-used), and a disk does the actual accounting. A volume can refuse to hold parts larger than max_data_part_size, which is how a 100 GiB part skips the NVMe tier and lands on the cheap one.

A MergeTree table has to answer two questions continuously: where does the next part go? and when does an old part move to cheaper storage? Answering them well means the hot data is on fast local disks and the cold data is on cheap object storage, without filling a disk to 100%.

The naive design — “put every part on the disk with the most free space” — fails in both directions. It ignores the fact that the tiers have different costs, and it makes placement depend on a moment-to-moment measurement, so the same part lands on different disks depending on when it was written.

ClickHouse’s design is a three-level ladder with an explicit priority order, and the interesting decisions are all about refusing work rather than accepting it.

The three levels

 text
 
 
 StoragePolicy "hot_to_cold"
 |
 +-- Volume "hot" volume_priority = 1
 | +-- Disk "nvme1"
 | +-- Disk "nvme2"
 |
 +-- Volume "cold" volume_priority = 2
 +-- Disk "s3"

A policy is an ordered list of volumes. It is what a table names in SETTINGS storage_policy = '...'.
A volume is a set of disks that are interchangeable for a given part — a JBOD. Its volume_priority says how early it is tried.
A disk is where bytes actually land: a local directory, an S3 bucket, an encrypted wrapper, a cache layer.

The policy’s volumes are sorted by priority, and the priority values are validated to be a dense range:

 cpp
 
 
// src/Disks/StoragePolicy.cpp:85
 if (!volume_priorities.empty())
 {
 /// Check that priority values cover the range from 1 to N (lowest explicit priority)
 if (*volume_priorities.begin() != 1 || *volume_priorities.rbegin() != volume_priorities.size())
 throw Exception(
 ErrorCodes::INVALID_CONFIG_PARAMETER,
 "volume_priority values must cover the range from 1 to N (lowest priority specified) without gaps");

 ::stableSort(
 volumes.begin(), volumes.end(),
 [](const VolumePtr a, const VolumePtr b) { return a->volume_priority < b->volume_priority; });
 }

Two rules hide here. Priorities must be unique (checked when the volumes are built), and they must form a gapless range 1..N — the priority is used as an index (“start looking for space at volume index k”), so a gap would make that index meaningless.

Reserving space: the ladder

Writing a part does not write to a disk directly. It reserves space first, and the reservation is what makes concurrent inserts safe: two inserts that both see “10 GiB free” must not both plan to write 10 GiB.

The top of the ladder is the policy, and it walks volumes in priority order:

 cpp
 
 
// src/Disks/StoragePolicy.cpp:269
ReservationPtrOrError StoragePolicy::reserve(UInt64 bytes, size_t min_volume_index) const
{
 bool all_volumes_read_only = true;
 for (size_t i = min_volume_index; i < volumes.size(); ++i)
 {
 const auto & volume = volumes[i];
 if (volume->isReadOnly())
 continue;
 all_volumes_read_only = false;
 auto reservation = volume->reserve(bytes);
 if (reservation)
 return reservation;
 }

 if (all_volumes_read_only)
 return std::unexpected(
 ReservationError(ErrorCodes::READONLY, "Could not reserve {} because all disk volumes are readonly", ReadableSize(bytes)));

 LOG_TRACE(log, "Could not reserve {} from volume index {}, total volumes {}", ReadableSize(bytes), min_volume_index, volumes.size());
 return std::unexpected(ReservationError(ErrorCodes::NOT_ENOUGH_SPACE, "Cannot reserve {}, not enough space", ReadableSize(bytes)));
}

Note min_volume_index: the caller can say “start at volume 2”. That is what the part-mover uses to push data down the ladder — see below.

The middle of the ladder is the volume, and this is where the interesting refusal lives:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:127
ReservationPtr VolumeJBOD::reserveImpl(UInt64 bytes, const std::optional<ReservationConstraints> & constraints)
{
 /// This volume can not store data which size is greater than `max_data_part_size`
 /// to ensure that parts of size greater than that go to another volume(s).
 if (max_data_part_size != 0 && bytes > max_data_part_size)
 return {};

That single line is the tiering mechanism. A volume configured with max_data_part_size_bytes = 10737418240 (10 GiB) will refuse any reservation for a bigger part, so the policy falls through to the next volume. Small parts land on fast disks; the occasional huge part (a big OPTIMIZE, a large mutation result) lands on the cheap tier.

The volume then picks a disk, by one of two algorithms:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:144
 switch (load_balancing)
 {
 case VolumeLoadBalancing::ROUND_ROBIN:
 {
 size_t disks_num = disks.size();
 for (size_t i = 0; i < disks_num; ++i)
 {
 size_t start_from = last_used.fetch_add(1u, std::memory_order_acq_rel);
 size_t index = start_from % disks_num;

 ReservationPtr reservation = try_reserve(disks[index]);
 if (reservation)
 return reservation;
 }
 return {};
 }

Round-robin is a single atomic counter — no lock, no filesystem query. It spreads parts evenly regardless of how full the disks are, which is what you want when the disks are identical.

LEAST_USED is the alternative, and it is deliberately cached:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:160
 case VolumeLoadBalancing::LEAST_USED:
 {
 std::lock_guard lock(mutex);

 ReservationPtr reservation;
 for (size_t i = 0; i < disks.size() && !reservation; ++i)
 {
 if (i == 0 && (!least_used_ttl_ms || least_used_update_watch.elapsedMilliseconds() >= least_used_ttl_ms))
 {
 disks_by_size = LeastUsedDisksQueue(disks.begin(), disks.end());
 least_used_update_watch.restart();

 DiskWithSize disk = disks_by_size.top();
 reservation = try_reserve(disk.disk);
 }

The ordering is by free space, and read-only disks always sort last:

 cpp
 
 
// src/Disks/VolumeJBOD.h:89
 bool operator<(const DiskWithSize & rhs) const
 {
 if (disk->isReadOnly())
 return true;
 if (rhs.disk->isReadOnly())
 return false;
 return free_size < rhs.free_size;
 }

The TTL (60 seconds by default) is the important part:

 cpp
 
 
// src/Disks/VolumeJBOD.cpp:81
 least_used_ttl_ms = config.getUInt64(config_prefix + ".least_used_ttl_ms", 60'000);

Querying free space means a statvfs on every disk, on every insert. The cache re-uses the answer for a minute and then refreshes it. In between, the “free space” of a disk is updated by the reservations themselves:

 cpp
 
 
// src/Disks/VolumeJBOD.h:98
 ReservationPtr reserve(UInt64 bytes)
 {
 ReservationPtr reservation = disk->reserve(bytes);
 if (!reservation)
 return {};

 /// Not just subtract bytes, but update the value,
 /// since some reservations may be done directly via IDisk, or not by ClickHouse.
 free_size = reservation->getUnreservedSpace();
 return reservation;
 }

getUnreservedSpace() is the disk’s own accounting, so the cached number tracks ClickHouse’s own reservations exactly, and drifts only from reservations made outside ClickHouse — which is why the periodic refresh exists.

The bottom of the ladder: what a reservation actually is

At the disk level, a reservation is bytes subtracted from a counter, under a mutex:

 cpp
 
 
// src/Disks/DiskLocal.cpp:202
std::optional<UInt64> DiskLocal::tryReserve(UInt64 bytes, const std::optional<ReservationConstraints> & constraints)
{
 std::lock_guard lock(DiskLocal::reservation_mutex);

 auto available_space = getAvailableSpace();

 UInt64 unreserved_space = available_space
 ? *available_space - std::min(*available_space, reserved_bytes)
 : std::numeric_limits<UInt64>::max();

and the commit is the counter increment:

 cpp
 
 
// src/Disks/DiskLocal.cpp:274
 ++reservation_count;
 reserved_bytes += bytes;
 return {unreserved_space - bytes};

The reservation object is a token: it holds the disk’s bytes until it is destroyed, and the storage policy hands the token to the part, which holds it for the lifetime of the part. That is how the space is held while the part is being written — and it is why the free space reported by system.disks is lower than the free space reported by df.

ReservationConstraints adds a second, policy-level condition: reserve only if enough space would remain afterwards.

 cpp
 
 
// src/Disks/DiskLocal.cpp:223
 UInt64 free_bytes_after = unreserved_space - bytes;

 /// Check min_bytes constraint
 if (constraints->min_bytes > 0 && free_bytes_after < constraints->min_bytes)
 {
 LOG_TRACE(logger, "Could not reserve {} ({} bytes) on disk {}. Free space after reservation {} bytes ({}) would be less than min_bytes {} bytes ({})",
 ReadableSize(bytes), bytes, backQuote(name), free_bytes_after, ReadableSize(free_bytes_after), constraints->min_bytes, ReadableSize(constraints->min_bytes));
 return {};
 }

This is how ClickHouse refuses to fill a disk completely. The insert path turns it into a user-visible error:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataWriter.cpp:1005
 if (!is_system_database && (min_bytes_to_perform_insert > 0 || min_ratio_to_perform_insert > 0.0f))
 {
 ReservationConstraints constraints(min_bytes_to_perform_insert, min_ratio_to_perform_insert);

 /// Try to reserve space on volume with constraints
 reservation = volume->reserve(expected_size, constraints);

 if (!reservation)
 {
 throw Exception(
 ErrorCodes::NOT_ENOUGH_SPACE,
 "Could not perform insert. "
 "None of the disks in volume '{}' have enough free space to meet the configured threshold. "

so the insert fails before writing anything, rather than filling the disk to zero and failing later.

TTL rules beat policy rules

Before the policy ladder is consulted, the TTL move rules get a chance. If a table has TTL ts + INTERVAL 30 DAY TO VOLUME 'cold', and the inserted data is already expired, the part goes straight to the cold volume:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:12185
 auto move_ttl_entry = selectTTLDescriptionForTTLInfos(metadata_snapshot->getMoveTTLs(), ttl_infos.moves_ttl, time_of_move, true);

 if (move_ttl_entry)
 {
 LOG_TRACE(log, "Trying to reserve {} to apply a TTL rule. Will try to reserve in the destination", ReadableSize(expected_size));
 SpacePtr destination_ptr = getDestinationForMoveTTL(*move_ttl_entry);
 bool perform_ttl_move_on_insert = is_insert && destination_ptr && shouldPerformTTLMoveOnInsert(destination_ptr);

and if the destination cannot take it, the insert falls back to the policy ladder. Note the perform_ttl_move_on_insert flag: a volume can opt out of having expired data inserted into it, in which case the data is written normally and moved later by the TTL merge.

Moving parts down the ladder

Placement alone is not enough: data written to the fast tier today must be moved to the cold tier tomorrow, when the TTL says so. The mover picks parts to move by asking the policy for space starting after the source disk’s volume:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartsMover.cpp:196
 for (auto && move : need_to_move)
 {
 auto min_volume_index = policy->getVolumeIndexByDiskName(move.first->getName()) + 1;
 for (auto && part : move.second.getAccumulatedParts())
 {
 auto reservation = policy->reserve(part->getBytesOnDisk(), min_volume_index).value_or(nullptr);
 if (!reservation)
 {
 /// Next parts to move from this disk has greater size and same min volume index.
 /// There are no space for them.
 /// But it can be possible to move data from other disks.
 break;
 }

min_volume_index = source volume index + 1 means: never move data to the volume it is already in, or to a faster one. Data flows in one direction only — down the priority ladder — which keeps the mover from thrashing.

The move itself is a clone plus an atomic rename, as covered in the article on part publication. What matters here is that it is not atomic across disks, and that recovery is defined to handle that: a crash mid-move leaves two copies, and the loader keeps one and marks the other outdated.

You can also move parts by hand:

 sql
 
 
-- Move every part of partition 2024_05 to the 's3' disk of the 'cold' volume.
ALTER TABLE events MOVE PARTITION 2024_05 TO DISK 's3';

-- Or to a whole volume.
ALTER TABLE events MOVE PARTITION 2024_05 TO VOLUME 'cold';

Inspecting the policy

The policy is fully introspectable:

 sql
 
 
SELECT
 policy_name,
 volume_name,
 volume_priority,
 disks,
 max_data_part_size,
 load_balancing,
 perform_ttl_move_on_insert
FROM system.storage_policies
ORDER BY policy_name, volume_priority;

and system.disks shows, per disk, the free space, the total space and the keep_free_space floor configured for it.

The `move_factor`, and what it does now

Every policy has a move_factor, described in the system table as “when the amount of available space gets lower than this factor, data automatically starts to move on the next volume if any”:

 cpp
 
 
// src/Disks/StoragePolicy.cpp:113
 const double default_move_factor = volumes.size() > 1 ? 0.1 : 0.0;
 move_factor = config.getDouble(config_prefix + ".move_factor", default_move_factor);

It is worth being precise about this: in the current code the factor is reported (system.storage_policies.move_factor) but no longer drives an automatic background move. Data moves down the tiers because of TTL rules, or because you ask it to. The default of 0.1 for a multi-volume policy and 0.0 for a single-volume one is a leftover from the era when the factor was the trigger for automatic rebalancing.

Trade-offs

What the design buys:

Deterministic placement. A part’s volume is decided by the priority ladder and the size of the part, not by a transient free-space reading.
Big parts skip the fast tier. max_data_part_size is a one-line refusal that buys a whole tiering policy.
Reservations, not hope. Concurrent inserts cannot overcommit a disk, and the “keep N bytes free” constraint makes the refusal explicit and early.
One-direction data flow. The mover refuses to move data up the ladder.

What it costs:

Free space is read at the wrong granularity. least_used mode caches statvfs results for up to a minute, so placement can be slightly wrong. Round-robin avoids the question by ignoring free space entirely.
Reservations leak into the numbers. system.disks.free_space and df will disagree, and a long merge can hold a reservation for a long time.
The tier boundary is a cliff, not a slope. A part of 11 GiB goes entirely to the cold tier because the hot tier’s limit is 10 GiB; there is no “mostly hot” placement.
Moving is expensive and not atomic across disks. A move is a full copy plus a rename, and a crash in the middle leaves duplicates to reconcile.

Wrapping up

The one thing to remember: the placement rule is “first volume that accepts the reservation”, and a volume accepts or refuses based on the size of the part and the free space on its disks. Everything else — round-robin, least-used, TTL rules — is a detail of how a volume picks among its own disks.

The part people get wrong: a multi-volume policy does not spread data by usage. It spreads it by part size and by TTL. A table with two volumes and no TTL rule keeps all its new parts on volume 1 forever.

What to go do about it: look at system.storage_policies and check that max_data_part_size on your fast volume is smaller than your typical part size but larger than your typical merge result — otherwise big merges will silently land on your slow disks.

ClickHouse Makes a MergeTree Part Visible with One Directory Rename

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: A part is written into tmp_insert_<part_name>/, one file per column, each through a stack of WriteBuffer decorators ending in a 1 MiB file buffer. When every byte is on disk, the whole directory is renamed to its final name with one rename(2) — the atomic step that makes the part visible. The fsyncs are ordered child-before-parent so the rename can never point at a not-yet-durable directory.

An INSERT into a MergeTree table does not “add rows to a table”. It materialises a directory containing a dozen files, and then makes that directory visible atomically. Everything hard about MergeTree’s durability model follows from that framing.

The naive implementation writes all_1_1_1_0/ file by file and updates the part list when it is done. That breaks in two ways. A reader that sees the directory mid-write reads a truncated .bin file. A crash between the last write() and the metadata update leaves a directory that looks like a part but is not in the part list.

ClickHouse’s answer has three parts: write under a temporary name, make the name change the atomic commit point, and fsync in an order that makes the rename durable.

 text
 
 
 INSERT commit readers
 | | |
 v v v
 tmp_insert_all_1_1_1_0/ --rename--> all_1_1_1_0/ ------> visible
 (part is written here) (atomic) (the commit point)

The write buffer stack

A part is not one file. In the Wide part format it is one file per column, plus marks, plus metadata, plus checksums. Each is written through a stack of WriteBuffer decorators.

The bottom of the stack is a file buffer, 1 MiB by default:

 cpp
 
 
// src/Core/Defines.h:23
static constexpr auto DBMS_DEFAULT_BUFFER_SIZE = 1048576ULL;

which is also the default of the disk-level write API:

 cpp
 
 
// src/Disks/IDisk.h:271
 /// Open the file for write and return WriteBufferFromFileBase object.
 virtual std::unique_ptr<WriteBufferFromFileBase> writeFile( /// NOLINT
 const String & path,
 size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE,
 WriteMode mode = WriteMode::Rewrite,
 const WriteSettings & settings = {}) = 0;

The WriteBuffer protocol is small. write() has a fast path that copies into the buffer if there is room:

 cpp
 
 
// src/IO/WriteBuffer.h:95
 void write(const char * from, size_t n)
 {
 /// `closed` is checked first: after `finalize` some buffers reset their storage to null (`set(nullptr, 0)`),
 /// and the pointer arithmetic in `available` must not be evaluated then.
 /// The comparison is unsigned as in `available`: for a constant `n` it is a single `cmp`.
 if (!closed && n <= available()) [[likely]]
 {
 copyBytes(pos, from, n);
 pos += n;
 return;
 }
 writeSlow(from, n);
 }

and next() is the flush, which calls the subclass’s nextImpl() and then resets the cursor to the beginning of the buffer:

 cpp
 
 
// src/IO/WriteBuffer.h:54
 try
 {
 nextImpl();
 ++flush_count;
 }
 catch (CurrentBufferExhausted &)
 {
 pos = working_buffer.begin();
 bytes += bytes_in_buffer;
 throw;
 }
 catch (...)
 {
 /** If the nextImpl() call was unsuccessful, move the cursor to the beginning,
 * so that later (for example, when the stack was expanded) there was no second attempt to write data.
 */
 pos = working_buffer.begin();
 bytes += bytes_in_buffer;

 cancel();

 throw;
 }

 bytes += bytes_in_buffer;
 pos = working_buffer.begin() + nextimpl_working_buffer_offset;
 nextimpl_working_buffer_offset = 0;
 }

Two details matter. First, a failed flush cancels the whole buffer stack — an error in the innermost layer must not be retried by an outer layer. Second, bytes counts the bytes handed to the subclass, not the bytes that reached the disk; that distinction is why the hashing buffer can report a checksum for bytes that are still in the OS buffer.

At the bottom, the file buffer turns buffer-full into a write(2):

 cpp
 
 
// src/IO/WriteBufferFromFileDescriptor.cpp:52
 size_t bytes_written = 0;
 while (bytes_written != offset())
 {
 ProfileEvents::increment(ProfileEvents::WriteBufferFromFileDescriptorWrite);

 ssize_t res = 0;
 {
 CurrentMetrics::Increment metric_increment{CurrentMetrics::Write};
 res = ::write(fd, working_buffer.begin() + bytes_written, offset() - bytes_written);
 }

Note the loop: a short write is not an error, it is a retry. The same loop shape appears in every layer of the stack, which is why one decorator works for files, sockets and object storage alike.

Hashing without a second pass

The interesting decorator is HashingWriteBuffer. A part’s checksums are computed over the bytes as they are compressed, on the way out:

 cpp
 
 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:340
 auto write_hashed_file = [&](const auto & filename, auto && writer)
 {
 auto out = new_part->getDataPartStorage().writeFile(filename, 4096, writer_settings.query_write_settings);
 HashingWriteBuffer out_hashing(*out);
 writer(out_hashing);
 out_hashing.finalize();
 checksums.files[filename].file_size = out_hashing.count();
 checksums.files[filename].file_hash = out_hashing.getHash();
 out->preFinalize();
 written_files.emplace_back(std::move(out));
 };

The stack here is HashingWriteBuffer → WriteBufferFromFileBase, and the checksum is taken from the hashing layer after it is finalized, so the hash covers exactly the bytes that were written — no re-read, no second pass over the compressed data. (Note that the part writer asks for a 4 KiB file buffer, not the 1 MiB default: the hash and compression layers above it do the buffering, and a part is many small files.)

Note out->preFinalize(). This is the hook that makes object storage bearable — it is documented in WriteBuffer as “similar to prefetch”:

 cpp
 
 
// src/IO/WriteBuffer.h:123
 /// The idea is similar to prefetch. In case if all data is written, we can flush the buffer
 /// and start sending data asynchronously. It may improve writing performance in case you have
 /// multiple files to finalize. Mainly, for blob storage, finalization has high latency,
 /// and calling preFinalize in a loop may parallelize it.
 virtual void preFinalize() { next(); }

A part has many files; each file’s finalization on S3 is a separate PUT with its own latency. preFinalize in a loop lets those PUTs overlap instead of serialising them.

Finally, all files are finalized and then synced in parallel:

 cpp
 
 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:160
void MergedBlockOutputStream::Finalizer::Impl::finish()
{
 writer.finish(sync);

 /// Finalize all files first (writes any pending bytes to the OS),
 /// then sync them in parallel — fsync of independent files can run concurrently
 /// and is a major contributor to part finalization latency when many small files are involved.
 for (auto & file : written_files)
 file->finalize();

 if (sync)
 {
 std::vector<WriteBufferFromFileBase *> files_to_sync;
 files_to_sync.reserve(written_files.size());
 for (auto & file : written_files)
 files_to_sync.push_back(file.get());
 parallelSyncFiles(files_to_sync);
 }

Finalize-all-then-sync-all, rather than finalize-and-sync-each: fsync of independent files can run concurrently, and with a wide part that is a dozen files, the difference is the difference between one fsync latency and a dozen serial ones.

The temporary directory

The part directory does not get its final name until the data is in. The writer builds a name with a tmp_insert_ prefix:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataWriter.cpp:855
 std::string temp_prefix = "tmp_insert_";
 const auto & temp_postfix = data.getPostfixForTempInsertName();
 if (!temp_postfix.empty())
 temp_prefix += temp_postfix + "_";

 std::string part_dir = temp_prefix + part_name;

and then claims that name, so that two concurrent inserts that compute the same part name cannot fight over the same directory. MergeTreeData::claimTemporaryPartDirectory takes the claim (and, if a leftover may exist, reclaims the stale directory), and the claim itself is a name in a set, released by a scope guard:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:4539
scope_guard MergeTreeData::getTemporaryPartDirectoryHolder(const String & part_dir_name) const
{
 temporary_parts.add(part_dir_name);
 return [this, part_dir_name]() { temporary_parts.remove(part_dir_name); };
}

and it also makes leftover cleanup safe. If a previous run crashed after creating tmp_insert_all_1_1_1_0/, the next insert with the same part name reclaims it — but only because it owns the name:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:4561
 /// Only temporary names may be auto-reclaimed. Elsewhere (e.g. "detached/<dir>" during ATTACH) the
 /// directory contents are the payload, and reclaiming would destroy user data.
 if (!startsWith(part_dir_name, "tmp") || part_dir_name.contains('/'))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot reclaim {}: not a temporary part directory name", part_dir_name);

The startsWith("tmp") check is a safety interlock: only a directory whose name provably looks temporary may be deleted automatically. A directory named detached/xyz — whose contents are user data being attached — can never be reclaimed.

The part builder is then told that the directory is guaranteed clean (PartDirIntent::CreateFresh, whose comment says “Directory is guaranteed clean by the claim”), so it does not have to probe the disk to find out.

The rename is the commit point

When the data is written, the part is renamed into place. The code says why outright:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:7699
 /// All checks are passed. Now we can rename the part on disk.
 /// So, we maintain invariant: if a non-temporary part in filesystem then it is in data_parts
 preparePartForCommit(part, out_transaction, lock, /* need_rename= */ true, rename_in_transaction);

and the rename itself:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeData.cpp:7601
 if (need_rename && !rename_in_transaction)
 part->renameTo(part->name, true);

renameTo is a directory rename; the destination is computed from the final relative path and handed to the part storage:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:3187
 auto old_projection_root_path = getDataPartStorage().getRelativePath();
 auto to = fs::path(relative_path) / new_relative_path;

 getDataPartStorage().rename(to.parent_path(), to.filename(), storage.log.load(), remove_new_dir_if_exists, fsync_dir);

which ends in DiskLocal::moveDirectory, i.e. one rename(2):

 cpp
 
 
// src/Disks/DiskLocal.cpp:346
void DiskLocal::moveDirectory(const String & from_path, const String & to_path)
{
 fs::rename(fs::path(disk_path) / from_path, fs::path(disk_path) / to_path);
}

rename(2) within a filesystem is atomic: a concurrent reader listing the table directory sees either tmp_insert_all_1_1_1_0 or all_1_1_1_0, never both and never a half-renamed mixture. There is no window in which the part directory exists under its final name but is incomplete — because it was complete under the temporary name all along.

The same trick is used elsewhere for a slightly different reason: when removing a file, ClickHouse renames it to a random 32-character name first and deletes that (RemoveRecursiveOperation::execute calls disk.moveFile(path, path_to) with getRandomASCIIString(32)). Renaming to a random name makes the removal atomic from the reader’s point of view: a reader that had opened the old path keeps its inode, and no new reader can open the path at all.

fsync ordering: child before parent

A rename is only durable if the contents it points at are durable, and if the directory entry of the rename is durable. The order matters:

 cpp
 
 
// src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:785
 /// Only after moveDirectory() since before the directory does not exist.
 if (fsync_part_dir)
 {
 /// Sync child before parent so a parent dentry never points at a not-yet-synced
 /// child: the moved dir, then the parent(s) holding the renamed dentry.
 { SyncGuardPtr to_sync_guard = volume->getDisk()->getDirectorySyncGuard(to); }

 /// parent_path() twice: step past the trailing slash on `to`/`from`, then to the container.
 const String to_parent = fs::path(to).parent_path().parent_path().string();
 const String from_parent = fs::path(from).parent_path().parent_path().string();

 if (!to_parent.empty())
 { SyncGuardPtr to_parent_sync_guard = volume->getDisk()->getDirectorySyncGuard(to_parent); }

 if (!from_parent.empty() && from_parent != to_parent)
 { SyncGuardPtr from_parent_sync_guard = volume->getDisk()->getDirectorySyncGuard(from_parent); }
 }
 });

The comment is the whole lesson: sync child before parent, so a parent dentry never points at a not-yet-synced child. If you fsync the parent first and crash, the parent may durably reference a directory whose contents are not on disk — an empty or truncated part that looks valid.

The sync of the moved directory happens after moveDirectory(), and the comment says why: before the rename the directory did not exist at to, so there was nothing to sync there.

The invariant, and what happens when it breaks

The commit sequence establishes a clean invariant:

if a non-temporary part exists in the filesystem, then it is in data_parts.

The reverse is not true, and that asymmetry is deliberate: a part can be in data_parts (in memory) but not yet on disk, because the metadata commit and the data write are separate steps. The dangerous direction is the one the invariant forbids.

The check is asserted in the code: a part whose directory still has a tmp_ or tmp-fetch_ prefix must be registered in temporary_parts (MergeTreeData::preparePartForCommit asserts exactly that), otherwise it is a leftover from a crash and may be cleaned up.

On startup, the server reconciles reality with the invariant: tmp_* directories are leftovers and get removed; directories that look like real parts but are not in the part list get attached or detached, depending on how complete they are.

Moving a part between disks

The same rename trick is the last step of moving a part to another disk, which is how a storage policy rebalances tiers. The mover clones the part into the destination’s moving/ directory, and only then swaps:

 cpp
 
 
// src/Storages/MergeTree/MergeTreePartsMover.cpp:364
 renameClonedPart(*cloned_part.part);

 /// If server goes down here we will get two copy of the part with the same name on different disks.
 /// And on the next ClickHouse startup during loading parts the first copy (in the order of defining disks
 /// in the storage policy) will be loaded as Active, the second one will be loaded as Outdated and removed as duplicate.
 /// See MergeTreeData::loadDataParts().
 data->swapActivePart(cloned_part.part, part_lock);

The comment is honest about the failure mode: a crash between the clone and the swap leaves two copies of the part, and the startup loader resolves it by loading the first and marking the second outdated. The move is not atomic across disks — it cannot be — but it is idempotent under recovery, which is the property that actually matters.

Trade-offs

What the design buys:

No half-visible parts. A reader either sees the whole part or none of it, because visibility is a single directory rename.
Crash-safe cleanup. Anything still named tmp_* after a restart is garbage by construction, and is reclaimable.
No second pass for checksums. The hash is computed on the way out, inside the write stack.
Parallel durability. Files are fsynced concurrently, so a wide part costs one fsync latency rather than N.

What it costs:

A directory rename per part. On object storage this is a metadata operation rather than a filesystem rename, but it is still a round trip.
Temporary directory churn. Every insert creates and renames a directory; on a slow-metadata filesystem this is measurable.
fsync is expensive and configurable. fsync_part_directory and the table-level fsync_after_insert trade durability for latency, and turning them off means a power loss can lose the last inserted part even though the server said the insert succeeded.
The invariant is checked, not enforced. A part can be in memory but not on disk; recovery has to reconcile.

Wrapping up

The one thing to remember: the rename is the transaction. Everything before it is preparation, and everything after it is bookkeeping.

The part people get wrong: a successful INSERT does not mean the data is on the platter. With fsync_after_insert = 0, “on the platter” means “in the page cache”. A power loss can lose an insert that the server already acknowledged.

What to go do about it: if you are chasing insert latency, look at fsync_after_insert and at how many files your part format produces. A Wide part with 50 columns is 50 files, 50 fsyncs (parallel, but still) and 50 times the metadata churn of a Compact part.

Mergetree on Opensource internals

ClickHouse Stores a Part's Metadata in Plain Text Files Beside the Data

What a part directory actually contains

columns.txt: the physical column list, and nothing else

checksums.txt: a compressed varint table

The other sidecars

Writing them: atomicity is not optional

Reading them back: the load path

The metadata is read once — then it is shared

The whole picture

Trade-offs

Wrapping up

ClickHouse Names a MergeTree Part After Its Partition and Block Range

The name

The partition id

The name is enough to list a table

The name encodes the merge rule

Patch parts: a partition id with a prefix

The legacy format

The LEGACY_MAX_LEVEL scar

Temporary names, and the one place a random suffix appears

The part UUID

The whole picture

Trade-offs

Wrapping up

ClickHouse Chooses What Goes into PREWHERE by Cost per Rejected Row

Where the decision is made

Step 1: splitting the WHERE into conjuncts

Grouping by column set

Step 2: the cost model

Where the byte estimate comes from

The rejection count

Step 3: the ordering

Step 4: how much to move

Step 5: splitting PREWHERE into read steps

The throwing-condition guard

Read-ahead of the same columns

The read-column split itself

A worked example

Trade-offs

How SummingMergeTree and AggregatingMergeTree Merge Rows

The engine arguments decide what is summed

Walking the group

Summing: the state is the sum

Aggregating: merging opaque states

The sumMap special case

Choosing the algorithm

Trade-offs

How ClickHouse Recompresses a Part When Its TTL Expires

The syntax

Validation

The per-part bound

Choosing the rule

“Would this rewrite change anything?”

The codec reaches the column streams

The part remembers its codec

Trade-offs

How ClickHouse Picks a Projection to Read

What a projection is, physically

Projections are per-part, and that is the whole difficulty

The comparison

Two rejections that are not about cost

The projection’s own key must be used

Keeping the projection parts alive

Trade-offs

ClickHouse Deletes Expired TTL Rows Only When Parts Merge

What a part remembers about its TTL

Which parts get rewritten because of a TTL

The four-hour throttle

The mask

The merge consumes the mask

When the mask is not enough

The gotcha: TTL is invisible until a merge

Trade-offs

How ReplacingMergeTree Collapses Duplicates and What SELECT FINAL Really Does

The problem: an update is a delete plus an insert

The engine declaration

The merge algorithm

The skip-run optimization

What SELECT FINAL actually does

`columns.txt`: the physical column list, and nothing else

`checksums.txt`: a compressed varint table

The `LEGACY_MAX_LEVEL` scar

The `sumMap` special case

What `SELECT FINAL` actually does

The `move_factor`, and what it does now