Merge-Tree on Opensource internals

ClickHouse's Query Condition Cache: Skip Granules You Don't Need to Read

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse caches, per (table, part, predicate), a bitmap of granules that a filter proved empty. Repeated queries skip those granules entirely — no index analysis, no reading, no decompression. The cache only records negative results (“this granule matched nothing”), never positive ones, and it salts the key with settings, time zones, and skip-index profiles so a wrong verdict can never leak between queries.

Index analysis is not free. Every time a query runs, ClickHouse has to figure out which granules of which parts could possibly contain a matching row. That means evaluating the predicate against the primary key index, the minmax indexes, the skip indexes, and — in the hard cases — a recursive exclusion search over the marks.

Now think about a real workload: a monitoring dashboard that runs the same query every 30 seconds, a service filtering on tenant_id = 42 over and over, a batch job re-reading yesterday’s data hourly. The predicate is the same every time. The data isn’t — but the data is also mostly not matching.

That’s a textbook case for memoization, and ClickHouse implements it as the QueryConditionCache: a global LRU that remembers, for each (table UUID, part name, predicate hash) triple, a bitmap of the granules the predicate emptied during the last scan.

What the cache actually stores

The class doc is unusually explicit:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:26
/// An implementation of predicate caching a la https://doi.org/10.1145/3626246.3653395
///
/// Given the table, part name and a hash of a predicate as key, caches which marks definitely don't match the predicate and which marks may
/// match the predicate. This allows to skip the scan if the same predicate is evaluated on the same data again. Note that this doesn't work
/// the other way round: we can't tell if _all_ rows in the mark match the predicate.
class QueryConditionCache
{
public:
 /// False means none of the rows in the mark match the predicate. We can skip such marks.
 /// True means at least one row in the mark matches the predicate. We need to read such marks.
 using MatchingMarks = std::vector<bool>;

Two things to internalize:

It’s one-directional. The cache records “this granule matched nothing.” It never records “this granule matched everything” — because a scan can’t establish that. If a granule produced rows, the cache can’t tell whether all of them matched.

It’s allowed to be imprecise — in one direction only. The bitmap’s true means “may match.” A granule that matched one row in a million is marked true — the whole granule has to be read. The cache never produces a false negative, which is the only thing that matters for correctness.

MatchingMarks is a std::vector<bool> — a bit-packed boolean array, one bit per granule. A part with 100,000 granules costs 12.5 KB per predicate.

The whole mechanism is controlled by use_query_condition_cache, which defaults to true.

The key: three components, hashed

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:45
private:
 /// A hash of the table id, part name and condition id.
 using Key = UInt128;

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:237
QueryConditionCache::Key QueryConditionCache::makeKey(const UUID & table_id, const String & part_name, UInt64 condition_hash)
{
 SipHash hash;
 hash.update(table_id);
 hash.update(part_name);
 hash.update(condition_hash);
 return hash.get128();
}

The key is the part, so a cached verdict about one part never leaks into another. The part name is qualified by the table UUID, so two tables with the same part name (very common in MergeTree — part names come from block ranges and partition values) don’t collide.

There’s a wrinkle for file-backed engines, where “part” is a file:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:93
 static String makeFilePartName(const String & path, std::string_view version_token);

This folds in a content-version token (ETag for remote objects, mtime+inode+size for local files) so an in-place rewrite of the file yields a different key rather than a stale hit. For a file-backed table, the “part” can change under you, and a stale “this file matches nothing” verdict would be a correctness bug.

The safety property: everything starts as “may match”

This is the detail that makes the whole thing safe. When an entry is created, every granule is marked true — “may match” — so a fresh entry skips nothing:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:396
QueryConditionCache::Entry::Entry(size_t mark_count)
 : matching_marks(mark_count, true)
{
}

The header explains why that default is a correctness requirement, not a performance one: if a query is cancelled mid-scan and the cache had been updated optimistically (“these granules match nothing”), future scans would skip granules the predicate would in fact have matched — wrong results, silently, forever.

By initializing everything to “may match” and only ever clearing bits after the scan has proven the granule empty, a crash can only lose information (the cache knows less than it could), never gain it.

Writing: only clear bits, never set them

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:272
void QueryConditionCache::write(
 const UUID & table_id, const String & part_name, UInt64 condition_hash, const String & condition,
 const MarkRanges & mark_ranges, size_t marks_count, bool has_final_mark)
{
 if (table_id == UUIDHelpers::Nil)
 return;

 Key key = makeKey(table_id, part_name, condition_hash);

 auto load_func = [&](){ return std::make_shared<Entry>(marks_count); };

 auto [entry, inserted] = cache.getOrSet(key, load_func);

getOrSet is atomic: two concurrent scans of the same part with the same predicate can’t create two entries; they share one.

Then the update — note the direction:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:311
 {
 std::lock_guard lock(entry->mutex);

 for (const auto & mark_range : mark_ranges)
 std::fill(entry->matching_marks.begin() + mark_range.begin, entry->matching_marks.begin() + mark_range.end, false);

 if (has_final_mark)
 entry->matching_marks[marks_count - 1] = false;
 }

std::fill(..., false) — the entry is only ever narrowed. The mark ranges passed in are the ones the scan proved empty. Nothing is ever set back to true, because “this granule matched something” isn’t a fact the cache can rely on: a later scan of the same part with the same predicate might read a different set of marks (another query’s cached verdict pruned some!), and those un-read marks must stay “may match.”

That last point is subtle and important: the cache is a cache of negative results only, and the negative results are monotone. Once a granule is known empty for a predicate, it stays known empty for as long as the entry lives.

There’s also a fast path that avoids taking the exclusive lock when there’s nothing to do — most writes are no-ops because the same query run repeatedly finds the same granules empty and the bits are already clear.

Why the entry has its own mutex

The CacheBase lock protects the map — the mapping from key to entry. It does not protect the contents of the entry. And the contents are mutated concurrently: a query with 16 streams reads 16 disjoint mark ranges of the same part simultaneously, and each writes back the ranges it proved empty.

Without a per-entry lock, two threads could be doing std::fill(begin, mid, false) and std::fill(mid, end, false) simultaneously on the same std::vector<bool>. std::vector<bool> packs eight booleans into a byte, so the two fills could land in the same byte and lose one another’s writes — a lost update, which means a granule proven empty stays “may match.” Not a correctness bug (only a lost optimization), but a real one.

The SharedMutex lets the many readers of a cached verdict proceed in parallel while the rare writer excludes them.

Reading: turning a bitmap back into mark ranges

The consult side is in MergeTreeDataSelectExecutor::filterPartsByQueryConditionCache. After the cache lookup, the bitmap is converted back into MarkRanges — the surviving granules:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1944
 for (const auto & mark_range : part_with_ranges.ranges)
 {
 size_t begin = mark_range.begin;
 for (size_t mark_it = begin; mark_it < mark_range.end;)
 {
 if (!matching_marks[mark_it])
 {
 // ... collapse consecutive zeros into skip ranges ...
 }
 else
 ++mark_it;
 }
 }

0 means “skip this granule,” 1 means “must read it.” Leading zeros are skipped. A gap of zeros between two ones is dropped — unless the gap is smaller than min_marks_for_seek, in which case reading through it is cheaper than seeking past it.

And if every granule of the part was eliminated, the part is dropped from the query entirely:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1994
 if (ranges.empty())
 it = parts_with_ranges.erase(it);

A part the cached verdict proves empty is not opened at all. No marks read, no decompression, no filters evaluated.

The predicate hash is not just the predicate

The most delicate part of the design is the key. If two different predicates hashed to the same value, one query’s verdict would be applied to another query’s data — silently returning wrong results.

The base hash is the ActionsDAG hash of the filter expression, covering function names, result types, children, and constant values. But that’s not enough:

 sql
 
SELECT * FROM t WHERE ts >= now() - INTERVAL 10 DAY;

The DAG hash changes every time the query runs (the constant changes), so this predicate never gets a cache hit. Fine — but there’s a subtler problem in the other direction: a setting can change a function’s result without changing the DAG. formatDateTime with different flags produces different strings from the same DAG. least/greatest propagate NULLs differently. h3ToGeo swaps its arguments.

So the cache key includes a salt computed from every setting a function might capture at build time — settings like session_timezone, formatdatetime_f_prints_single_zero, least_greatest_legacy_null_behavior, h3togeo_lon_lat_result_order, and dozens more.

Two sessions in different time zones produce the same DAG for toDateTime(s), but parse s differently. Without the salt, one session’s verdict would be served to the other. This is the kind of bug that’s nearly impossible to find by testing and completely obvious once you’ve read the rule.

The time-condition trick

now() in a predicate is a special case, handled with a derived condition rather than a salt.

The problem: WHERE ts >= now() - INTERVAL 10 DAY has a different DAG hash every second, so it never caches. But it’s a very common predicate.

The solution: round the time constant onto a grid. On the write side, the condition is strengthened; on the read side, it’s weakened:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/updateQueryConditionCache.cpp:107
 /// Non-deterministic conditions must not be cached under their own hash.
 /// For conditions non-deterministic only because of folded current-time constants,
 /// a deterministic condition can be derived by rounding the constants onto a time grid.

Read the logic carefully, because the asymmetry is the whole trick:

Write side (strengthen): the actual filter used now() - 10 days. The derived condition is a rounded version, e.g. rounded up to the next grid point — a narrower condition. If the narrow condition matched no rows in this granule, the wider (actual) condition certainly matched no rows either. So the entry (written under the derived hash) is sound.
Read side (weaken): the query probes with the rounded-down version, which is wider. “The wider condition matched no rows” implies “the narrower condition matched no rows” — so skipping is sound.

The two coincide for grid-aligned constants and are one grid cell apart otherwise. The grid factor is a setting (query_condition_cache_time_condition_grid_factor).

This is genuinely elegant reasoning: it takes a predicate that’s not cacheable (its value changes every second) and makes it cacheable by proving that a slightly different, grid-aligned predicate implies the right thing in both directions.

The skip-index profile: another salt

Some cached verdicts come from skip index analysis — “the minmax index on this column says no row in this granule can match.” But skip index analysis depends on which indexes exist and are enabled. If a query runs with use_skip_indexes = 0 and another runs with them on, the verdicts differ.

So skip-index verdicts are stored under a profiled key — the condition hash salted with the profile of the index set that produced the verdict:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1771
 /// Each condition is looked up under two keys:
 /// * the bare condition hash, holding row-level exclusions (always sound);
 /// * a hash salted with the effective skip-index profile (skip-index exclusions).

A mark may be skipped iff either verdict says it doesn’t match — the merge is an AND of the two “may match” bitmaps:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1881
 matching_marks[i] = matching_marks[i] && (*marks_opt)[i];

AND of the “may match” bits: a granule is skipped only if both verdicts agree it cannot match. Conservative in the right direction.

Where the “empty granule” fact comes from

The write side is the read pipeline itself. When the PREWHERE filter processes a granule and the filter removes every row of it, the read task records the granule as “unmatched”:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeReadTask.cpp:590
void MergeTreeReadTask::addPrewhereUnmatchedMarks(const MarkRanges & mark_ranges_)
{
 prewhere_unmatched_marks.insert(prewhere_unmatched_marks.end(), mark_ranges_.begin(), mark_ranges_.end());
}

And the write happens when the task is finished:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSelectProcessor.cpp:617
 if (reader_settings.use_query_condition_cache && task && prewhere_info
 && !task->readersChainCanSkipMarksBeforePrewhere(prewhere_filters_by_top_k_threshold)
 && !task->appliesMutationsBeforePrewhere()
 && !row_level_filter
 && task->getInfo().data_part_info->getDataPart())

Every one of those four extra conditions is a soundness condition:

!task->readersChainCanSkipMarksBeforePrewhere(...) — if a skip index upstream of PREWHERE could have filtered marks before PREWHERE saw them, attributing them to the PREWHERE predicate would be a lie.
!task->appliesMutationsBeforePrewhere() — on-the-fly mutations filter rows ahead of PREWHERE, for the same reason.
!row_level_filter — a row-level security filter is prepended before PREWHERE, but the cache key covers only the query’s PREWHERE hash. A mark hidden by a row policy must not be attributed to the PREWHERE predicate.

That last one is a beautiful example of a cache whose key doesn’t cover everything that influenced the result — the fix is to not write in that case, rather than to make the key bigger.

The whole flow

 text
 
 
 query with predicate Q
 │
 ▼
 hash(Q) + settings salt + [skip-index profile]
 │
 ├──► QueryConditionCache::read(table_uuid, part, hash)
 │ │
 │ └──► bitmap of "may match" granules
 │ │
 │ ▼
 │ surviving MarkRanges per part
 │ (parts with no surviving ranges are dropped entirely)
 │
 ▼
 index analysis (primary key, minmax, skip indexes)
 │
 ▼
 scan the surviving granules
 │
 ├── granules where the filter removed everything ──► cache.write(cleared bits)
 │
 └── rows ──► result

What this buys you, and what it costs

It buys you the elimination of repeated index analysis. The expensive part of “which granules can match” isn’t the bitmap lookup — it’s the KeyCondition evaluation and the exclusion search. On a warm cache, that work is replaced by a hash lookup and a bit scan.

It buys you the elimination of reading granules that match nothing. The index analysis can only prove a granule is empty using the index; a predicate on a non-key column can’t be resolved from the index at all. But once a scan has observed that a granule contains no matching rows, that fact is cached, and the next run doesn’t read it.

It costs you memory. One bit per granule per (table, part, predicate). The cache is a CacheBase LRU with a max_size_bytes.

It costs you correctness vigilance. Everything in this design is about not serving a verdict computed under different circumstances: a different predicate, a different set of skip indexes, a different time zone, a different row-level policy. The salt functions, the profiles, the derived time conditions, and the !row_level_filter guard are all the same idea applied four times.

It gives up on one direction. The cache can’t record “this granule matches everything,” so a highly selective predicate matching a few rows in every granule buys nothing. The cache is for predicates that are granule-level selective — which is most of them, since a granule is 10 MB of data.

Seeing it in action

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN ('QueryConditionCacheHits', 'QueryConditionCacheMisses');

A high, stable hit/miss ratio is the signature of a workload with repeated predicates over stable data. A ratio near zero means the predicates are always new — dashboards with per-request filters, for instance — and the cache is pure overhead.

The cache is also inspectable:

 sql
 
 
SELECT *
FROM system.query_condition_cache
LIMIT 10;

And to see it working, run the same query twice and compare the SelectedMarks and SelectedRanges counters in system.query_log:

 sql
 
 
SELECT
 query_id,
 ProfileEvents['SelectedMarks'] AS marks,
 ProfileEvents['SelectedRanges'] AS ranges,
 ProfileEvents['QueryConditionCacheHits'] AS qcc_hits
FROM system.query_log
WHERE query LIKE '%tenant_id = 42%'
ORDER BY event_time;

The second run should show dramatically fewer selected marks — and, if the predicate empties whole granules, the same query won’t even open the part.

ClickHouse Skips Data: Granules, Marks, and Bit-Packed Indexes

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: A MergeTree scan isn’t a full scan. Parts are cut into granules, each granule gets a bit-packed mark, and the primary key index stores one key value per granule — so ClickHouse can prove a granule can’t match and skip it entirely.

A SELECT on a MergeTree table is, mechanically, a scan of a set of files. The only reason it isn’t a full scan is that the files are chopped into pieces, and each piece carries enough metadata for the engine to prove that the piece cannot contain a matching row.

ClickHouse calls those pieces granules, the metadata is called a mark, and the mechanism that ties them together is the primary key index — one key value per granule, held in memory.

This is the single most important performance idea in MergeTree, and it’s built out of three small, very specific tricks:

Granules are sized in bytes, not rows, so a granule of fat rows isn’t a granule of a million rows.
Marks are bit-packed in memory — three bytes per mark for an integer column, instead of the sixteen bytes a naive {offset, offset_in_block} pair would take.
The primary key index is trimmed: if a key column is nearly constant within a part, the key columns after it aren’t kept in memory at all, because they can’t narrow anything down.

Granules: sized in bytes, not rows

The classic description of MergeTree granularity is “8192 rows”, and that’s the default value of the index_granularity setting. But that’s only the cap. The actual mechanism computes the number of rows per granule from the size of the data being written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:115
size_t computeIndexGranularity(
 size_t rows,
 size_t bytes_uncompressed,
 size_t index_granularity_bytes,
 size_t fixed_index_granularity_rows,
 bool blocks_are_granules,
 bool can_use_adaptive_index_granularity)
{
 size_t index_granularity_for_block = 0;

 if (!can_use_adaptive_index_granularity)
 {
 index_granularity_for_block = fixed_index_granularity_rows;
 }
 else
 {
 if (blocks_are_granules)
 {
 index_granularity_for_block = rows;
 }
 else if (index_granularity_bytes != 0 && bytes_uncompressed >= index_granularity_bytes)
 {
 /// index_granularity_bytes == 0 disables adaptive sizing upstream, so this branch is
 /// unreachable with a zero divisor; the explicit check makes that visible to static analysis.
 size_t granules_in_block = bytes_uncompressed / index_granularity_bytes;
 index_granularity_for_block = rows / granules_in_block;
 }
 else
 {
 size_t size_of_row_in_bytes = std::max(bytes_uncompressed / rows, 1UL);
 index_granularity_for_block = index_granularity_bytes / size_of_row_in_bytes;
 }
 }

 /// We should be less or equal than fixed index granularity.
 /// But if block size is a granule size then do not adjust it.
 /// Granularity greater than fixed granularity might come from compact part.
 if (!blocks_are_granules)
 index_granularity_for_block = std::min(fixed_index_granularity_rows, index_granularity_for_block);

 /// Very rare case when index granularity bytes less than single row.
 if (index_granularity_for_block == 0)
 index_granularity_for_block = 1;

 return index_granularity_for_block;
}

Read the interesting branch — the else at the bottom. If the block is smaller than index_granularity_bytes, then

text

 rows_per_granule = index_granularity_bytes / average_row_size

The granule stays a constant number of bytes. The default index_granularity_bytes is 10 MB, so a granule is 10 MB of data whether its rows are 10 bytes wide or 10 KB wide.

Why does that matter? Because a granule is the unit of skipping. If granules were a fixed 8192 rows, then a table of 1 KB rows would have 8 MB granules (fine), but a table of 100 KB rows would have 800 MB granules (useless — you can’t skip anything smaller than 800 MB). Sizing in bytes makes the granule a predictable, bounded amount of IO, and therefore a predictable, bounded amount of waste when you have to read a granule that does contain matches.

The price is that the row count of a granule is no longer constant, so it has to be remembered. The adaptive granularity object stores partial sums:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularityAdaptive.h:9
/// Class that stores adaptive index granularity.
/// Inside it contains vector of partial sums of rows after mark:
/// |-----|---|----|----|
/// | 5 | 8 | 12 | 16 |
class MergeTreeIndexGranularityAdaptive final : public MergeTreeIndexGranularity

One size_t per mark, and the row count of any mark is a subtraction. The alternative — a constant number of rows per granule — is available as use_const_adaptive_granularity, and its setting description explains the trade-off exactly:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:2174
 DECLARE(Bool, use_const_adaptive_granularity, false, R"(
Non-const adaptive granularity (the default, `index_granularity_bytes` is not `0`) keeps
granules at a constant size in bytes, so their row count varies: for every
written block, rows per granule is `index_granularity_bytes` divided by the
average size of a row in that block, capped at `index_granularity`. Because it differs
from granule to granule, the row count of every granule in the part has to be kept in
memory, 8 bytes each, which on large tables adds up to tens of gigabytes.

“Tens of gigabytes” — that’s the cost of the byte-granularity guarantee, and it’s the reason the constant-row variant exists at all. On a table with 100 million parts × 100 marks each, 8 bytes per mark is 80 GB of pure bookkeeping.

The marks themselves are appended as the block is written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:162
void fillIndexGranularityForWidePart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 index_granularity.appendMark(index_granularity_for_block);
}

A Wide part gets exactly index_granularity_for_block rows per mark. A Compact part, where all columns share one file, has to be more careful:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:169
void fillIndexGranularityForCompactPart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 {
 size_t rows_left_in_block = rows_in_block - current_row;

 /// Try to extend last granule if block is large enough
 /// or it isn't first in granule (index_offset != 0).
 if (rows_left_in_block < index_granularity_for_block &&
 (rows_in_block >= index_granularity_for_block || index_offset != 0))
 {
 // If enough rows are left, create a new granule. Otherwise, extend previous granule.
 // So, real size of granule differs from index_granularity_for_block not more than 50%.
 if (rows_left_in_block * 2 >= index_granularity_for_block)
 index_granularity.appendMark(rows_left_in_block);
 else
 index_granularity.addRowsToLastMark(rows_left_in_block);
 }
 else
 {
 index_granularity.appendMark(index_granularity_for_block);
 }
 }
}

The comment “real size of granule differs from index_granularity_for_block not more than 50%” is the invariant: a short tail is either given its own granule (if it’s at least half a granule) or merged into the previous one.

Marks: where in the file does each granule start?

A granule is a range of rows. To read it, you need to know where in the compressed .bin file its rows live. That’s the mark:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:14
/** Mark is the position in the compressed file. The compressed file consists of adjacent compressed blocks.
 * Mark is a tuple - the offset in the file to the start of the compressed block, the offset in the decompressed block to the start of the data.
 */
struct MarkInCompressedFile
{
 size_t offset_in_compressed_file;
 size_t offset_in_decompressed_block;

Two numbers, both size_t, so sixteen bytes per mark in the naive layout. For a part with 100 marks that’s nothing. For a table with 100 000 parts of 1 GB each (100 TB), 100 marks per part × 16 bytes × 100 000 parts = 160 MB of index — and that’s only if every mark of every column is resident.

Which is why the in-memory representation isn’t a struct array:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:30
/**
 * In-memory representation of an array of marks.
 *
 * Uses an ad-hoc compression scheme that decreases memory usage while allowing
 * random access in O(1) time.
 * This is independent from the marks *file* format, which may be uncompressed
 * or use a different compression method.
 *
 * Typical memory usage:
 * * ~3 bytes/mark for integer columns
 * * ~5 bytes/mark for string columns
 * * ~0.3 bytes/mark for trivial marks in auxiliary dict files of LowCardinality columns
 */
class MarksInCompressedFile

Three bytes per mark. How?

The trick is that marks are not random. The comment in the header spells out the regularities the compression exploits:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:51
 /** We need to store a sequence of marks, each consisting of two 64-bit integers:
 * offset_in_compressed_file and offset_in_decompressed_block. We'll call them x and y for
 * convenience, since compression doesn't care what they mean. The compression exploits the
 * following regularities:
 * * y is usually zero.
 * * x usually increases steadily.
 * * Differences between x values in nearby marks usually fit in much fewer than 64 bits.
 *
 * We split the sequence of marks into blocks, each containing MARKS_PER_BLOCK marks.
 * (Not to be confused with data blocks.)
 * For each mark, we store the difference [value] - [min value in the block], for each of the
 * two values in the mark. Each block specifies the number of bits to use for these differences
 * for all marks in this block.
 * The smaller the blocks the fewer bits are required, but the bigger the relative overhead of
 * block headers.
 *
 * Packed marks and block headers all live in one contiguous array.
 */

So: 256 marks per block. Per block, store the minimum x and y and the number of bits needed to represent the spread within the block. Then store every mark as a delta from the block minimum, in that many bits.

The block header computation:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:122
void MarksInCompressedFile::Builder::flushBlock(const MarkInCompressedFile * data, size_t count)
{
 chassert(count > 0 && count <= MARKS_PER_BLOCK);

 size_t block_idx = marks_flushed / MARKS_PER_BLOCK;
 BlockInfo & block = blocks[block_idx];
 block.bit_offset_in_packed_array = packed_bits;

 /// Compute block metadata: min values, bit widths.
 size_t max_x = 0;
 size_t max_y = 0;
 for (size_t i = 0; i < count; ++i)
 {
 block.min_x = std::min(block.min_x, data[i].offset_in_compressed_file);
 max_x = std::max(max_x, data[i].offset_in_compressed_file);
 block.min_y = std::min(block.min_y, data[i].offset_in_decompressed_block);
 max_y = std::max(max_y, data[i].offset_in_decompressed_block);
 block.trailing_zero_bits_in_y
 = std::min(block.trailing_zero_bits_in_y, static_cast<UInt8>(getTrailingZeroBits(data[i].offset_in_decompressed_block)));
 }

 block.bits_for_x = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits(max_x - block.min_x));
 block.bits_for_y
 = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits((max_y - block.min_y) >> block.trailing_zero_bits_in_y));

Two ideas in six lines.

Delta encoding per block. bits_for_x is the number of bits needed to represent the range of x values inside the block, not the magnitude of x itself. If a block of 256 marks spans 64 KB of compressed file, that’s 19 bits per mark, not 64.

Trailing-zero stripping for y. y is the offset inside the decompressed block, and it’s usually zero — a granule normally starts at the beginning of a compressed block. The code computes the number of trailing zero bits common to all the y values in the block and shifts them all right by that amount before packing. For a block where every y is zero, the shift is 63 and bits_for_y becomes zero: the y coordinate costs no bits at all.

The write path, showing both encodings:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:153
 /// Write bit-packed deltas.
 size_t bit_offset = packed_bits;
 for (size_t i = 0; i < count; ++i)
 {
 writeBitsPacked64(packed.data(), bit_offset, data[i].offset_in_compressed_file - block.min_x);
 writeBitsPacked64(
 packed.data(),
 bit_offset + block.bits_for_x,
 (data[i].offset_in_decompressed_block - block.min_y) >> block.trailing_zero_bits_in_y);
 bit_offset += block.bits_for_x + block.bits_for_y;
 }

And the read path, which has to be O(1) — a mark lookup happens for every granule the reader touches:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:47
std::tuple<const MarksInCompressedFile::BlockInfo *, size_t> MarksInCompressedFile::lookUpMark(size_t idx) const
{
 size_t block_idx = idx / MARKS_PER_BLOCK;
 const BlockInfo & block = blocks[block_idx];
 size_t offset = block.bit_offset_in_packed_array + (idx - block_idx * MARKS_PER_BLOCK) * (block.bits_for_x + block.bits_for_y);
 return {&block, offset};
}

Divide by 256 to find the block header, then a shift-and-mask to find the mark inside it. That’s the whole random-access structure: one array division and one multiply. Compare that to a compressed list of marks, which would need a binary search.

Where do the marks come from at write time?

The mark is written when the compressed block is flushed, so it’s simply the state of the hashing/compression stream at that moment:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:351
 MarkInCompressedFile mark{stream->plain_hashing.count(), stream->compressed_hashing.offset()};

plain_hashing.count() is the number of uncompressed bytes written so far — that’s y, the offset inside the decompressed block. And compressed_hashing.offset() is the number of compressed bytes written so far — that’s x, the offset of the compressed block in the file.

The mark cache

Marks are read from .mrk files and cached in a global LRU keyed by the file path:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeMarksLoader.cpp:313
 if (mark_cache)
 {
 auto key = MarkCache::hash(data_part_storage->getDiskName() + ":" + (fs::path(data_part_storage->getFullPath()) / mrk_path).string());

 if (save_marks_in_cache)
 {
 auto callback = [this] { return loadMarksImpl(); };
 loaded_marks = mark_cache->getOrSet(key, callback);
 }
 else
 {
 loaded_marks = mark_cache->getWithoutMetrics(key);
 if (!loaded_marks)
 loaded_marks = loadMarksImpl();
 }
 }

Note the two paths. getOrSet records hits and misses and inserts the loaded marks into the cache. getWithoutMetrics only looks: it’s used when the marks are wanted for a merge or a mutation, which shouldn’t evict the query’s working set or pollute the cache statistics.

That’s a small but real piece of cache hygiene — background work must not steal the cache from foreground queries.

The cache itself weights each entry by its compressed size plus a fixed overhead:

 cpp
 
 
// src/Storages/MergeTree/MarkCache.h:23
/// Estimate of number of bytes in cache for marks.
struct MarksWeightFunction
{
 /// We spent additional bytes on key in hashmap, linked lists, shared pointers, etc ...
 static constexpr size_t MARK_CACHE_OVERHEAD = 128;

 size_t operator()(const MarksInCompressedFile & marks) const
 {
 return marks.approximateMemoryUsage() + MARK_CACHE_OVERHEAD;
 }
};

The 128-byte constant is the per-entry overhead of the hash map node, the list node, the shared pointer, and the MarksInCompressedFile object itself. For a part with 100 marks, the compressed marks are ~300 bytes and the overhead is 128 — so the accounting is honest about the fact that the bookkeeping dominates for small parts.

The primary key index: one key value per granule

The marks tell you where the bytes are. The primary key index tells you what’s in them. It’s a separate file, primary.idx, containing the sorting key values of the first row of every granule:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1935
std::shared_ptr<IMergeTreeDataPart::Index> IMergeTreeDataPart::loadIndex() const
{
 auto component_guard = Coordination::setCurrentComponent("IMergeTreeDataPart::loadIndex");
 /// Memory for index must not be accounted as memory usage for query, because it belongs to a table.
 MemoryTrackerBlockerInThread temporarily_disable_memory_tracker;

and the loop that reads it is the whole format — marks_count rows of key_size values, serialized one after another:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1972
 FormatSettings format_settings;
 for (size_t i = 0; i < marks_count; ++i)
 {
 for (size_t j = 0; j < key_size; ++j)
 key_serializations[j]->deserializeBinary(*loaded_index[j], *index_file, format_settings);
 }

marks_count rows of key_size values, serialized one after another. The in-memory form is a set of ColumnPtrs — one column per key column, each holding marks_count values — which is exactly the layout a KeyCondition wants to evaluate a predicate against.

The trimming trick

Now the interesting part. The primary key may be (a, b, c). If a is almost constant within a part — say, a low-cardinality column — then keeping b and c in memory is a waste. A query filtering on b can’t be answered from the index anyway, because a doesn’t narrow anything down. So the index is trimmed:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1895
void IMergeTreeDataPart::optimizeIndexColumns(size_t marks_count, Columns & index_columns) const
{
 if (marks_count == 0)
 {
 chassert(isEmpty());
 return;
 }

 /// Do not optimize index for patch parts because patch parts
 /// use manual index analysis which requires all index columns.
 if (info.isPatch())
 return;

 size_t key_size = index_columns.size();
 Float64 ratio_to_drop_suffix_columns = static_cast<double>((*storage.getSettings())[MergeTreeSetting::primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns]);

 /// Cut useless suffix columns, if necessary.
 if (key_size > 1 && ratio_to_drop_suffix_columns > 0 && ratio_to_drop_suffix_columns < 1)
 {
 for (size_t j = 0; j < key_size - 1; ++j)
 {
 size_t num_changes = 0;
 for (size_t i = 1; i < marks_count; ++i)
 {
 if (0 != index_columns[j]->compareAt(i, i - 1, *index_columns[j], 0))
 ++num_changes;
 }

 if (static_cast<Float64>(num_changes) / static_cast<Float64>(marks_count) >= ratio_to_drop_suffix_columns)
 {
 key_size = j + 1;
 index_columns.resize(key_size);
 break;
 }
 }

The rule: while the fraction of granules in which key column j changes is at least primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns (default 0.9), column j is informative, so keep it and move on. The first column that changes in fewer than 90% of granules is “nearly constant” within this part — the part is probably all one value of it — and then everything after it is dropped.

This is a per-part decision. A part that covers one day of data might have a constant country column and a varying user_id; the index keeps country and drops user_id. Another part, spanning many countries, keeps both.

The consequence is that index analysis must know which columns are actually available — and the code accounts for it, treating the missing suffix columns as “constant over the whole universe”, which relaxes the analysis to a safe over-approximation.

The primary index cache

For very large tables, keeping every part’s primary index in memory isn’t possible, so it can be cached in an LRU instead:

 cpp
 
 
// src/Storages/MergeTree/PrimaryIndexCache.h:29
/** Cache of primary index for MergeTree tables.
 * Primary index is a list of columns from primary key
 * that store first row for each granule of data part.
 */
class PrimaryIndexCache : public CacheBase<UInt128, PrimaryIndex, UInt128TrivialHash, PrimaryIndexWeightFunction>

and the part consults it lazily:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:706
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::getIndex() const
{
 std::scoped_lock lock(index_mutex);

 if (index)
 return index;

 if (auto index_cache = storage.getPrimaryIndexCache())
 return loadIndexToCache(*index_cache);

 index = loadIndex();
 return index;
}

with the key being the disk name plus the part’s path, so that two parts with the same name on different disks don’t collide:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:726
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::loadIndexToCache(PrimaryIndexCache & index_cache) const
{
 auto key = PrimaryIndexCache::hash(getDataPartStorage().getDiskName() + ":" + getRelativePathOfActivePart());
 auto callback = [this] { return loadIndex(); };
 return index_cache.getOrSet(key, callback);
}

Now the payoff: two different search algorithms

With marks and the primary index in hand, a query has to answer “which granules can contain a matching row?”. ClickHouse has two algorithms for this, and it picks between them based on the shape of the predicate.

The decision point is whether the predicate describes a continuous range of the sorting key:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2705
 bool key_condition_exact_range = !key_condition_useful || key_condition.matchesExactContinuousRange();
 bool part_offset_condition_exact_range = !part_offset_condition_useful || part_offset_condition->matchesExactContinuousRange();
 bool total_offset_condition_exact_range = !total_offset_condition_useful || total_offset_condition->matchesExactContinuousRange();
 const String & part_name = part->isProjectionPart() ? fmt::format("{}.{}", part->name, part->getParentPart()->name) : part->name;

 if (!key_condition_exact_range || !part_offset_condition_exact_range || !total_offset_condition_exact_range)
 {
 // Do exclusion search, where we drop ranges that do not match

Algorithm 1: binary search (continuous range)

If the predicate describes a single continuous interval of the sorting key — WHERE key >= 100 AND key < 200 — then the matching granules form a contiguous range in the index, and two binary searches find its boundaries:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2801
 /// Invariant: !check_in_range(part_range.begin..searched_left).can_be_true
 /// check_in_range(part_range.begin..searched_right).can_be_true
 /// (part_range.end + 1 is a sentinel out-of-bounds index, such that by definition
 /// check_in_range(x..part_range.end+1).can_be_true is true. The binary search will never
 /// actually call check_in_range with this out-of-bounds index.)
 /// If the condition is not true anywhere, we'll end up with searched_left == part_range.end.
 size_t searched_left = part_range.begin;
 size_t searched_right = part_range.end + 1;

 while (searched_left + 1 < searched_right)
 {
 const size_t middle = (searched_left + searched_right) / 2;
 MarkRange range(0, middle);
 if (check_in_range(range, BoolMask::consider_only_can_be_true).can_be_true)
 searched_right = middle;
 else
 searched_left = middle;
 ++steps;
 }
 result_range.begin = searched_left;
 LOG_TEST(log, "Found (LEFT) boundary mark: {}", searched_left);

check_in_range(0, middle) asks: “could any row in granules [0, middle) match?” The KeyCondition evaluates the predicate against the index values at the two ends of the range and the monotonicity of the key columns. Because the key is sorted, the answer is monotone in middle, so binary search is valid. O(log n) index checks for the whole part.

Algorithm 2: generic exclusion search (everything else)

Now consider WHERE key2 = 5 where key2 is the second column of the sorting key, or WHERE key = 5 OR key = 900000. The matching granules are not contiguous, so a binary search can’t be used.

ClickHouse falls back to an exclusion search: split the mark range recursively, drop the parts that cannot match.

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:61
void searchUnlimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 for (const auto & initial_range : initial_ranges)
 {
 std::vector<MarkRange> ranges_stack = {initial_range};
 while (!ranges_stack.empty())
 {
 MarkRange range = ranges_stack.back();
 ranges_stack.pop_back();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 appendWithMaxGap(result.ranges, range, search_settings.min_marks_for_seek);

 /// Unlike `ranges`, exact ranges must never absorb the gap between two accepted
 /// ranges: every mark of an exact range has to fully match the condition, while the
 /// skipped marks in between do not.
 if (collect_exact_ranges && exact)
 appendWithMaxGap(result.exact_ranges, range, 0);
 }
 else
 {
 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { ranges_stack.push_back(subrange); });
 }
 }
 }
}

This is a depth-first search over mark ranges. The check returns a BoolMask with two bits: can_be_true and can_be_false.

!can_be_true → the whole range cannot match → drop it.
!can_be_false → every row in the range matches → keep it whole.
otherwise → the range is a mixture → split it into merge_tree_coarse_index_granularity (default 8) subranges and recurse.

Splitting into 8 rather than 2 is deliberate. It’s a breadth-ish compromise: splitting in half is the classic binary split, but each split costs one check_in_range call, and each check isn’t free — it evaluates a KeyCondition against the index columns at the range boundaries. Eight subranges per step means the search converges in about log_8(n) steps while doing 8× the work per step.

The step budget

The unlimited search has no bound. A pathological predicate could make it check thousands of ranges, so there’s a budget:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:119
void searchLimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 std::vector<MarkRange> heap_storage(initial_ranges.begin(), initial_ranges.end());
 std::priority_queue<MarkRange, std::vector<MarkRange>, LessBySize> queue(LessBySize{}, std::move(heap_storage));

 /// Ranges are accepted out of spatial order, so they are collected raw and merged after sorting.
 std::vector<MarkRange> accepted;
 std::vector<MarkRange> accepted_exact;

 while (!queue.empty())
 {
 MarkRange range = queue.top();
 queue.pop();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 accepted.push_back(range);
 if (collect_exact_ranges && exact)
 accepted_exact.push_back(range);
 continue;
 }

 if (result.num_steps + queue.size() > search_settings.max_steps)
 {
 result.reached_step_limit = true;
 accepted.push_back(range);
 continue;
 }

 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { queue.push(subrange); });
 }

Two things change relative to the unlimited version.

A priority queue, largest range first. The budget is spent where it buys the most: splitting a huge range is what removes the most granules, so the biggest range is examined first. The comparator is explicit:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:102
/// Orders the queue of the step-limited search: the largest range comes first.
struct LessBySize
{
 bool operator()(const MarkRange & lhs, const MarkRange & rhs) const
 {
 size_t lhs_size = lhs.end - lhs.begin;
 size_t rhs_size = rhs.end - rhs.begin;
 return std::tie(lhs_size, rhs.begin) < std::tie(rhs_size, lhs.begin);
 }
};

When the budget runs out, the range is accepted whole. This is the important bit: the search never returns an incorrect answer. When result.num_steps + queue.size() > max_steps, the range is pushed into accepted as-is — meaning “we don’t know, so read it”. The result is a superset of the true matching granules, so the query is correct; it just reads more data than an unlimited search would have.

That’s what the setting says:

 cpp
 
 
// src/Core/Settings.cpp:1936
 DECLARE(UInt64, merge_tree_generic_exclusion_search_max_steps, 0, R"(
When a filter cannot be evaluated as a single continuous range of the primary key, for example when it uses key columns other than the first one, ClickHouse runs an iterative generic exclusion search algorithm over the index marks. The same algorithm is used for the analysis of the text index. This setting limits the number of steps (index checks) the algorithm spends on each data part.

The budget is spent on the largest remaining mark ranges first. When it is exhausted, the ranges that were not fully analyzed are accepted as a whole, so the query stays correct but may read more granules than an unlimited search would select. A lower budget speeds up index analysis at the cost of reading more data. The limit is approximate rather than a strict cap on the analysis cost: the search can exceed it by roughly one round of splitting, and when the part is already divided into many ranges (for example, by the query condition cache), each of them is checked at least once regardless of the limit.

The default of 0 means unlimited steps. So the default behaviour is exact and unbounded; the budget is an escape hatch for pathological cases.

The seek heuristic

One more knob, and it’s about IO, not about correctness. When two accepted ranges are separated by a small gap of marks, it’s cheaper to read the gap sequentially than to seek past it. appendWithMaxGap merges the two ranges when the gap is at most max_gap marks wide.

The gap threshold comes from merge_tree_min_rows_for_seek and merge_tree_min_bytes_for_seek, converted into marks by the granularity of the part. Both default to 0, i.e. “never bridge a gap” — but on a spinning disk or an object store with a high per-request cost, setting them turns “read 8 KB from here, seek, read 8 KB from there” into “read 40 KB contiguously”.

Note the asymmetry that the comment points out: exact_ranges are merged with max_gap = 0, always. An exact range is a range where every row matches, and bridging a gap into it would claim that the skipped marks match too — which is false. So the gap-bridging optimization applies only to the “may match” ranges, never to the “definitely matches” ones.

Putting it together

 text
 
 
 part on disk
 ┌───────────────────────────────────────────────────────────────┐
 │ granule 0 granule 1 granule 2 granule 3 │
 │ rows 0..8191 rows 8192..16383 rows ... rows ... │
 └───────────────────────────────────────────────────────────────┘
 │ │ │ │
 ▼ ▼ ▼ ▼
 marks: (x0,y0) (x1,y1) (x2,y2) (x3,y3)
 bit-packed, ~3 bytes each, one per column file
 │
 ▼
 primary.idx: key values of the FIRST ROW of each granule
 │
 ▼
 KeyCondition.checkInRange(...)
 │
 ├── predicate is a continuous key range → binary search (2·log₂ n checks)
 │
 └── predicate is anything else → exclusion search (recursive, budgeted)
 │
 ▼
 MarkRanges: the granules that must be read

What this buys you, and what it costs

It buys you a scan that’s proportional to the data you asked for, not to the data you stored. A SELECT with a range predicate on a sorted key reads roughly the granules that overlap the range, plus one granule of granularity on each side.

It buys you an index that fits in memory. Three bytes per mark, and the primary key index trimmed to the columns that can actually narrow things down. Without those two tricks, a 10 TB table would need tens of gigabytes of index.

It costs you a fixed granularity of waste. A predicate that matches a single row still reads a whole granule — 10 MB by default. This is the central trade of the whole design, and it’s why index_granularity_bytes is a setting you may want to lower for a table with a very selective predicate and small rows.

It costs you an in-memory structure that has to be built and kept. The primary index is loaded per part (or cached), and the marks are loaded per column file that the query touches. Both are cached, and both have their own LRU, but the first query against a cold part pays for them.

It costs you complexity in the analysis. Two algorithms, a budget, a trimming heuristic that makes the analysis conservative in some cases, and the requirement that the analysis be sound (never skip a granule that could match) even when it isn’t complete.

Seeing it in action

The profile events tell you which algorithm ran:

 sql
 
 
SELECT event, value
FROM system.events
WHERE event IN (
 'IndexBinarySearchAlgorithm',
 'IndexGenericExclusionSearchAlgorithm',
 'IndexGenericExclusionSearchStepLimitReached',
 'FilteringMarksWithPrimaryKeyProcessedMarks',
 'FilteringMarksWithPrimaryKey'
);

FilteringMarksWithPrimaryKeyProcessedMarks is the number of marks examined. FilteringMarksWithPrimaryKey is the number of parts where the primary key was used at all. The ratio between the two, and the ratio of marks read to marks examined, is your “how selective is my index” metric.

And the parts themselves report the granularity they were written with:

 sql
 
 
SELECT
 name,
 rows,
 marks,
 round(rows / marks) AS rows_per_mark,
 formatReadableSize(bytes_on_disk) AS size
FROM system.parts
WHERE active AND rows > 0
ORDER BY marks DESC
LIMIT 10;

A rows_per_mark far below 8192 means the table has fat rows and the byte-based granularity kicked in. A rows_per_mark of exactly 8192 means use_const_adaptive_granularity is on, or index_granularity_bytes was 0 when the part was written.

Wrapping up

The one thing to remember: a MergeTree granule is the unit of skipping, and everything else — marks, the primary index, the two search algorithms — exists to answer one question about it: can this granule possibly match?

What to go do about it: if a query on your table isn’t skipping granules, look at the shape of the predicate before you touch anything else. A predicate on the first sorting-key column with a continuous range gets the fast binary search. A predicate on a later column, or an OR across values, gets the recursive exclusion search — and there you should be watching IndexGenericExclusionSearchStepLimitReached.

If you see that event firing, the search ran out of budget and fell back to reading more granules than it needed to. Either raise merge_tree_generic_exclusion_search_max_steps, or — better — add a projected column and query that instead.

ClickHouse Merges Columns One at a Time: The Vertical Merge Algorithm

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse’s vertical merge splits a merge into two stages: merge the key and record where each output row came from (one byte per row), then gather each payload column separately. Peak memory drops from “all columns at once” to “one column at a time” — which is the difference between a merge that OOMs and one that doesn’t.

Every MergeTree table is a pile of immutable parts. Every part is a pile of immutable columns. Background merges take a contiguous range of parts and rewrite them as one bigger part. That sounds like a simple copy, and for a table with three narrow columns it is one.

Now consider a table with two hundred columns, one of which is a 40 MB String of raw JSON and the rest are UInt64 counters.

The obvious merge implementation reads all two hundred columns of the source parts, merges the sorting key, and writes two hundred columns out. It has to hold, in memory at once, the merged key plus every payload column of the rows it is currently holding — which for a wide table means the peak memory of a merge is proportional to the sum of all column widths, not to the width of the widest column.

That is a bad trade. The payload columns are almost always much larger than the key, and they are almost always independent of the key: merge the keys first and separately, and you can then reconstruct each payload column without ever holding two payload columns in memory at the same time.

ClickHouse implements exactly that. It is called the vertical merge algorithm, and it is one of the more elegant pieces of engineering in the MergeTree code base. The trick that makes it work is a one-byte-per-row side-channel file that records where each output row came from.

The problem, stated precisely

Suppose we merge three parts, each sorted by the key k:

 text
 
 
 part A (10 rows) part B (10 rows) part C (10 rows)
 k=1 a1=... k=1 b1=... k=2 c1=...
 k=1 a2=... k=2 b2=... k=2 c2=...
 k=3 a3=... k=2 b3=... k=5 c3=...
 ... ... ...

A horizontal merge reads all three parts’ k columns, merges them into one sorted stream, and then — because MergeTree guarantees that all rows with the same key are adjacent and, within a key, appear in source-part order — it can reconstruct every other column by a gather: walk the merged key stream, and for each row, take the next value of that column from the part the row came from.

The output order of the merged key is not the concatenation of the inputs; it is the k-way merge of them. So for each output row we need to know which input part it came from.

That is the entire information content of the vertical merge’s side channel.

And it is exactly one small integer per row.

The horizontal stage: merging the key and writing the row sources

The vertical merge is not a separate code path that replaces the normal merge. It is a second stage that runs after the normal merge has merged the key. The normal merge is called the “horizontal” stage, and the code says so explicitly — the stage class is called ExecuteAndFinalizeHorizontalPart:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.h:63
/**
 * Overview of the merge algorithm
 *
 * Each merge is executed sequentially block by block.
 * The main idea is to make a merge not a subroutine which is executed
 * in a thread pool and may occupy a thread for a period of time,
 * but to make a merge a coroutine which can suspend the execution
 * in some points and then resume the execution from this point.
 *
 * A perfect point where to suspend the execution is after the work over a block is finished.
 * The task itself will be executed via BackgroundJobExecutor.
 *
 * The interface of the task is simple.
 * The main method is `execute()` which will return true, if the task wants to be executed again and false otherwise.
 *
 * With this kind of task we can give a merge a priority.
 * A priority is simple - the lower the size of the merge, the higher priority.
 * So, if ClickHouse wants to merge some really big parts into a bigger part,
 * then it will be executed for a long time, because the result of the merge is not really needed immediately.
 * It is better to merge small parts as soon as possible.
*/

Note the second half of that comment: the merge is a coroutine. It is resumed by the background pool between blocks, which is what makes it possible to give a small merge priority over a huge one without pinning a thread. The vertical stage is expressed in the same idiom: a list of sub-tasks, each of which returns true when it wants to be resumed.

The side channel is a real file, managed by a small class:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:257
/// Manages the "rows_sources" temporary file that is used during vertical merge.
class RowsSourcesTemporaryFile : public ITemporaryFileLookup
{
public:
 /// A logical name of the temporary file under which it will be known to the plan steps that use it.
 static constexpr auto FILE_ID = "rows_sources";

 explicit RowsSourcesTemporaryFile(TemporaryDataOnDiskScopePtr temporary_data_on_disk_)
 : temporary_data_on_disk(temporary_data_on_disk_->childScope({CurrentMetrics::TemporaryFilesForMerge}))
 {
 }

and it is deliberately re-openable, so that several columns can be gathered in parallel from independent file handles — getTemporaryFileForReading returns tmp_data_buffer->read(), a fresh reader each time, with the comment “Reopen the file for each read so that multiple reads can be performed in parallel and there is no need to seek to the beginning.”

That matters: the row-source stream is consumed once per column, and each column’s gather starts from the beginning of the file. If the file were a single sequential stream, gathering column #2 would have to seek back to offset 0 and re-read column #1’s worth of bytes. Re-opening gives every column its own independent read cursor.

The encoding: one byte per row

The row source is not a struct with a source number and a flag in two fields. It is a single byte with two bit-fields packed into it:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:15
/// Tiny struct, stores number of a Part from which current row was fetched, and insertion flag.
struct RowSourcePart
{
 UInt8 data = 0;

 RowSourcePart() = default;

 explicit RowSourcePart(size_t source_num, bool skip_flag = false)
 {
 static_assert(sizeof(*this) == 1, "Size of RowSourcePart is too big due to compiler settings");
 setSourceNum(source_num);
 setSkipFlag(skip_flag);
 }

 size_t getSourceNum() const { return data & MASK_NUMBER; }

 /// In CollapsingMergeTree case flag means "skip this rows"
 bool getSkipFlag() const { return (data & MASK_FLAG) != 0; }

 void setSourceNum(size_t source_num)
 {
 data = (data & MASK_FLAG) | (static_cast<UInt8>(source_num) & MASK_NUMBER);
 }

 void setSkipFlag(bool flag)
 {
 data = flag ? data | MASK_FLAG : data & ~MASK_FLAG;
 }

 static constexpr size_t MAX_PARTS = 0x7F;
 static constexpr UInt8 MASK_NUMBER = 0x7F;
 static constexpr UInt8 MASK_FLAG = 0x80;
};

The top bit is the “skip this row” flag (used when a lightweight delete mask removes rows during the merge), and the low seven bits are the source part number. Seven bits means at most 127 source parts per merge — and you can watch that limit propagate all the way into the merge-algorithm choice (see “When is it worth it?” below).

The static_assert is not decoration. If the compiler ever made this struct two bytes, the row-source file would double in size and every write to it would become a two-byte write. The assert turns a silent performance regression into a build failure.

Who writes the bytes

The horizontal stage’s merge transform writes one byte per output row as it emits it. For the Ordinary merging mode that is the plain k-way merge algorithm:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:444
void MergingSortedAlgorithm::insertRow(const SortCursorImpl & current)
{
 auto write_row_source = [&](bool skipped)
 {
 if (out_row_sources_buf)
 {
 RowSourcePart row_source(current.order, skipped);
 out_row_sources_buf->write(row_source.data);
 }
 };

 size_t current_row = current.getRow();

 if (const auto * mask = source_row_filter_masks[current.order])
 {
 if ((*mask)[current_row])
 merged_data.insertRow(current.all_columns, current_row, current.rows);

 write_row_source(!(*mask)[current_row]);
 }
 else
 {
 merged_data.insertRow(current.all_columns, current_row, current.rows);
 write_row_source(false);
 }
}

current.order is the index of the source stream the row came from. The merged algorithm already knows this — it is the whole reason it can merge — so emitting it costs one byte write per row. One byte per row of overhead on a merge that is about to rewrite gigabytes.

Note the ordering: merged_data.insertRow(...) happens first, then the row source is written. The write buffer is only non-null for a vertical merge — the merge step is constructed with the file name only when the vertical algorithm was chosen:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3654
 auto merge_step = std::make_unique<MergePartsStep>(
 merge_parts_query_plan.getCurrentHeader(),
 sort_description,
 partition_and_sorting_required_columns,
 global_ctx->merging_params,
 (is_vertical_merge ? RowsSourcesTemporaryFile::FILE_ID : ""), /// rows_sources' temporary file is used only for vertical merge
 filter_column_name,
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size],
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size_bytes],
 max_dynamic_subcolumns,
 is_vertical_merge,
 sorting_queue_strategy,
 cleanup,
 global_ctx->time_of_merge);

so a horizontal merge pays nothing for the side channel.

The vertical stage: gathering one column at a time

Now the second stage. It does not merge anything. It gathers. For each non-key column, in turn:

open a read stream for that column in every source part;
union those streams;
run a ColumnGathererTransform over them, which uses the row-source file to decide which source’s value belongs at each output position.

Here is the step that wires the gatherer into the plan:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2172
 createReadFromPartStep(
 MergeTreeSequentialSourceType::Merge,
 *plan_for_part,
 *global_ctx->data,
 global_ctx->storage_snapshot,
 RangesInDataPart(global_ctx->future_part->parts[part_num], nullptr, part_num, part_starting_offset),
 global_ctx->alter_conversions[part_num],
 global_ctx->merged_part_offsets,
 Names{column_name},
 global_ctx->input_rows_filtered,
 apply_deleted_mask,
 /*filter=*/ std::nullopt,
 ctx->read_with_direct_io,
 ctx->use_prefetch,
 global_ctx->context,
 getLogger("VerticalMergeStage"));

Names{column_name} — exactly one column is read per part. The read plan for a vertical merge is a column read plan, not a row read plan. Every source part is opened, and only the column of interest is decoded.

Then the per-part streams are unioned, and a ColumnGathererStep is added to the plan. The gatherer is a merging transform, so it plugs into the same IMergingTransform machinery as the key merge: its input is one column per source part, its output is one merged column, and the row-source file is handed to it as a ReadBuffer by name.

The gather loop

The heart of it is ColumnGathererStream::gather. It reads the row-source bytes and copies values out of the per-part source columns:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:131
template <typename Column>
void ColumnGathererStream::gather(Column & column_res)
{
 row_sources_buf.nextIfAtEnd();
 RowSourcePart * row_source_pos = reinterpret_cast<RowSourcePart *>(row_sources_buf.position());
 RowSourcePart * row_sources_end = reinterpret_cast<RowSourcePart *>(row_sources_buf.buffer().end());

 if (next_required_source == -1)
 {
 /// Start new column.
 /// Actually reserve works only for fixed size columns.
 /// So it's safe to ignore preferred size in bytes and call reserve for number of rows.
 size_t size_to_reserve = std::min(static_cast<size_t>(row_sources_end - row_source_pos), block_preferred_size_rows);
 column_res.reserve(size_to_reserve);
 }

 next_required_source = -1;

 /// We use do ... while here to ensure there will be at least one iteration of this loop.
 /// Because the column_res.byteSize() could be bigger than block_preferred_size_bytes already at this point.
 do
 {
 if (row_source_pos >= row_sources_end)
 break;

 RowSourcePart row_source = *row_source_pos;
 size_t source_num = row_source.getSourceNum();
 Source & source = sources[source_num];
 bool source_skip = row_source.getSkipFlag();

 if (source.pos >= source.size) /// Fetch new block from source_num part
 {
 next_required_source = source_num;
 return;
 }

 ++row_source_pos;

 /// Consecutive optimization. TODO: precompute lengths
 size_t len = 1;
 size_t max_len = std::min(static_cast<size_t>(row_sources_end - row_source_pos), source.size - source.pos); // interval should be in the same block

 while (len < max_len && row_source_pos->data == row_source.data)
 {
 ++len;
 ++row_source_pos;
 }

 row_sources_buf.position() = reinterpret_cast<char *>(row_source_pos);

 if (!source_skip)
 {
 /// Whole block could be produced via copying pointer from current block
 if (source.pos == 0 && source.size == len)
 {
 /// If current block already contains data, return it.
 /// Whole column from current source will be returned on next read() iteration.
 source_to_fully_copy = &source;
 return;
 }
 if (len == 1)
 column_res.insertFrom(*source.column, source.pos);
 else
 column_res.insertRangeFrom(*source.column, source.pos, len);
 }

 source.pos += len;
 } while (column_res.size() < block_preferred_size_rows && column_res.byteSize() < block_preferred_size_bytes);
}

Three optimizations are hiding in there.

Run-length coalescing. The inner while loop scans forward over consecutive row-source bytes that are identical. Since the byte encodes “which source part”, a run of identical bytes means “these N output rows came from the same source part, consecutively” — which, because the source part’s own column is read in order, means “these N output rows are exactly source part’s rows [pos, pos+N)”. So the whole run is copied with a single insertRangeFrom call instead of N single-element inserts. The comment /// Consecutive optimization. TODO: precompute lengths is honest about the remaining inefficiency: the run length is rediscovered for every column, N times.

Whole-column pointer steal. If the run covers the entire remaining source column (source.pos == 0 && source.size == len), the output column can simply be the source column — no copy at all. The transform returns the source ColumnPtr directly and sets source_to_fully_copy, and the merge() method hands it out:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.cpp:114
 else
 {
 res.addColumn(source_to_fully_copy->column);
 }

For a merge where one part dominates — the common case after a few rounds of merging, where you merge one big part with one small one — most columns of the big part are copied by pointer. The “merge” of such a column is essentially free.

Sparse output. The source_skip flag comes from the lightweight-delete mask: rows deleted by a DELETE are recorded in the row-source stream with the skip bit set, and the gatherer simply does not copy them out. The horizontal stage does the same filtering, so the two stages agree on which rows exist without either having to re-evaluate the delete predicate.

Writing the gathered column

The gathered column is written by MergedColumnOnlyOutputStream — the vertical stage’s counterpart of MergedBlockOutputStream, except that it writes exactly one column. Its constructor carries the invariant that makes the whole thing safe:

 cpp
 
 
// src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:36
 /// Granularity is never recomputed while writing only columns.
 MergeTreeWriterSettings writer_settings(

The horizontal stage has already decided where the granule boundaries are (it wrote the marks for the key columns), and the payload columns must use the same boundaries, or the part would be corrupt.

When is it worth it?

Vertical merging is not always a win. It has costs: an extra pass over the row-source file per column, an extra file in the temporary space, and a pipeline per column. So chooseMergeAlgorithm() gates it behind a set of conditions:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3760
MergeAlgorithm MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm() const
{
 const size_t total_rows_count = global_ctx->merge_list_element_ptr->total_rows_count;
 const size_t total_size_bytes_uncompressed = global_ctx->merge_list_element_ptr->total_size_bytes_uncompressed;
 const auto & merge_tree_settings = global_ctx->data_settings;

 if (global_ctx->deduplicate)
 return MergeAlgorithm::Horizontal;
 if ((*merge_tree_settings)[MergeTreeSetting::enable_vertical_merge_algorithm] == 0)
 return MergeAlgorithm::Horizontal;
 if (ctx->need_remove_expired_values)
 {
 /// `TTLTransform` stops reading after the first block when the rows TTL has expired for the whole part,
 /// while a vertical merge reads every key row to write `rows_sources`.
 if (global_ctx->future_part->merge_type == MergeType::TTLDrop && global_ctx->metadata_snapshot->hasRowsTTL())
 return MergeAlgorithm::Horizontal;
 if (!canVerticalTTLDelete(*global_ctx))
 return MergeAlgorithm::Horizontal;
 }
 if (global_ctx->future_part->part_format.part_type != MergeTreeDataPartType::Wide)
 return MergeAlgorithm::Horizontal;
 if (global_ctx->cleanup)
 return MergeAlgorithm::Horizontal;

and then the quantitative thresholds:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3793
 bool is_supported_storage =
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Ordinary ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Collapsing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Replacing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::VersionedCollapsing;

 bool enough_ordinary_cols = global_ctx->gathering_columns.size() >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_columns_to_activate];

 bool enough_total_rows = total_rows_count >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_rows_to_activate];

 bool enough_total_bytes = total_size_bytes_uncompressed >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_bytes_to_activate];

 bool no_parts_overflow = global_ctx->future_part->parts.size() <= RowSourcePart::MAX_PARTS;

 auto merge_alg = (is_supported_storage && enough_total_rows && enough_total_bytes && enough_ordinary_cols && no_parts_overflow) ?
 MergeAlgorithm::Vertical : MergeAlgorithm::Horizontal;

 return merge_alg;
}

Read those conditions as a description of when the vertical algorithm is the right shape.

Wide parts only, because a Compact part stores all columns in one data file and you cannot read one column of it without reading the others. No deduplication, because deduplication compares whole rows and so needs all columns at once. Only Ordinary/Collapsing/Replacing/VersionedCollapsing merging modes, because Summing/Aggregating/Coalescing merge the values of equal keys and so have to merge the payload rather than gather it. At least 11 non-key columns (vertical_merge_algorithm_min_columns_to_activate) and at least 16·8192 rows (vertical_merge_algorithm_min_rows_to_activate) — below those, the extra machinery is not worth it. And at most 127 source parts, because RowSourcePart::MAX_PARTS is 0x7F and the one-byte encoding cannot express more.

The default thresholds live in MergeTreeSettings: enable_vertical_merge_algorithm = 1, vertical_merge_algorithm_min_rows_to_activate = 16 * 8192, vertical_merge_algorithm_min_bytes_to_activate = 0, and vertical_merge_algorithm_min_columns_to_activate = 11.

Note the framing of the setting description: “potentially at the cost of merge speed”. Vertical merging is a memory optimization, not a speed one. It trades IO locality (one column at a time, so each source file is opened and read once per column rather than once for all columns) for peak memory. For a table where one column dominates the size, that trade is usually right; for a table with many equally-sized columns, the re-reading can dominate.

The `Wide`-only restriction, and the escape hatch

The Wide restriction is the interesting one. A Compact part interleaves all columns of a row in a single .bin file, so “read only column X” is impossible — you would have to decode the whole file. The sources may still be Compact while the result is Wide, and that case is allowed by allow_vertical_merges_from_compact_to_wide_parts (default true):

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3784
 if (!(*merge_tree_settings)[MergeTreeSetting::allow_vertical_merges_from_compact_to_wide_parts])
 {
 for (const auto & part : global_ctx->future_part->parts)
 {
 if (!isWidePart(part))
 return MergeAlgorithm::Horizontal;
 }
 }

A Compact source is read once per column during the vertical stage, so its file gets decoded N times. That is expensive but it is bounded — and it buys the memory reduction for wide tables whose legacy parts happen to be Compact.

Progress accounting

A merge is a long-running background job, and system.merges shows a progress bar. For a vertical merge, “progress” spans two axes: first the key columns are merged (the horizontal stage), then each payload column is gathered (the vertical stage). The horizontal stage is weighted by the size of the key columns relative to the whole part; the vertical stage is weighted by the size of each individual column:

 cpp
 
 
// src/Storages/MergeTree/ColumnSizeEstimator.h:9
/* Allow to compute more accurate progress statistics */
class ColumnSizeEstimator
{
 using ColumnToSize = std::map<String, UInt64>;
 ColumnToSize map;
public:

 /// Stores approximate size of columns in bytes
 /// Exact values are not required since it used for relative values estimation (progress).
 size_t sum_total = 0;
 size_t sum_index_columns = 0;
 size_t sum_ordinary_columns = 0;

 ColumnSizeEstimator(ColumnToSize && map_, const NamesAndTypesList & key_columns, const NamesAndTypesList & ordinary_columns)
 : map(std::move(map_))
 {
 for (const auto & [name, _] : key_columns)
 if (!map.contains(name)) map[name] = 0;
 for (const auto & [name, _] : ordinary_columns)
 if (!map.contains(name)) map[name] = 0;

 for (const auto & [name, _] : key_columns)
 sum_index_columns += map.at(name);

 for (const auto & [name, _] : ordinary_columns)
 sum_ordinary_columns += map.at(name);

 sum_total = std::max(static_cast<decltype(sum_index_columns)>(1), sum_index_columns + sum_ordinary_columns);
 }

 Float64 columnWeight(const String & column) const
 {
 return static_cast<Float64>(map.at(column)) / static_cast<Float64>(sum_total);
 }

 Float64 keyColumnsWeight() const
 {
 return static_cast<Float64>(sum_index_columns) / static_cast<Float64>(sum_total);
 }
};

The weights are proportional to bytes, not rows, so a merge that is 90% one giant String column reports 90% progress for most of its duration and then jumps. That’s the honest answer for a byte-oriented operation. The sizes come from the parts themselves (accumulateColumnSizes, a cheap per-part statistic), so the whole estimator costs one map construction per merge.

The horizontal stage’s contribution: the row count assertion

There is one more piece of rigor worth pointing out. After the horizontal stage finishes and the row-source file is closed, the code checks that the file contains exactly one byte per input row:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2055
 size_t rows_sources_count = ctx->rows_sources_temporary_file->finalizeWriting();

 /// In special case, when there is only one source part, and no rows were skipped, we may have
 /// skipped writing rows_sources file. Otherwise rows_sources_count must be equal to the total
 /// number of input rows.
 /// Note that only one byte index is written for each row, so number of rows is equals to the number of bytes written.
 if ((rows_sources_count > 0 || global_ctx->future_part->parts.size() > 1) && sum_input_rows_exact != rows_sources_count + input_rows_filtered)
 throw Exception(
 ErrorCodes::LOGICAL_ERROR,
 "Number of rows in source parts ({}) excluding filtered rows ({}) differs from number "
 "of bytes written to rows_sources file ({}). It is a bug.",
 sum_input_rows_exact, input_rows_filtered, rows_sources_count);

This is a cheap, total consistency check on the side channel. If the horizontal stage and the vertical stage ever disagree about which rows exist, the merge fails loudly rather than producing a part whose payload columns are misaligned with its key — which would be a silent data corruption.

Note the subtlety in the condition: with exactly one source part and no filtered rows, the file may legitimately be empty (a single-source merge does not need to record where rows came from), so the check is skipped in that one case.

Prefetching the next column

Because the vertical stage reads one column at a time, and each column read is a separate pipeline over the source parts, the obvious optimization is to start reading column N+1 while column N is still being written. The code does exactly that, and only for remote filesystems, where the latency of the first read of a column is the dominant cost:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2076
 bool all_parts_on_remote_disks = std::ranges::all_of(global_ctx->future_part->parts, [](const auto & part) { return part->isStoredOnRemoteDisk(); });
 ctx->use_prefetch = all_parts_on_remote_disks && storage_settings[MergeTreeSetting::vertical_merge_remote_filesystem_prefetch];

 if (ctx->use_prefetch && ctx->it_name_and_type != global_ctx->gathering_columns.end())
 ctx->prepared_pipeline = createPipelineForReadingOneColumn(ctx->it_name_and_type->name);

The “prefetching” is simply that building the pipeline — which opens the parts and issues the first read requests — starts the IO. It is not a separate prefetch thread; it is eager pipeline construction, one column ahead (prepareVerticalMergeForOneColumn builds the next column’s pipeline while the current one runs).

The whole flow

 text
 
 
 source parts (Wide)
 ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
 │ k, a, b, c │ │ k, a, b, c │ │ k, a, b, c │
 └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
 │ │ │
 └────────────┬────┴─────────────────┘
 │ HORIZONTAL STAGE
 │ k-way merge of the key columns
 │ emits: merged k + rows_sources (1 byte/row)
 ▼
 ┌──────────────────────────────────────────┐
 │ rows_sources: 00 00 01 01 02 00 01 ... │
 └──────────────────────────────────────────┘
 │
 ┌─────────────────┼──────────────────┐
 ▼ ▼ ▼
 gather col a gather col b gather col c VERTICAL STAGE
 read a from read b from read c from one column at a time
 each part, each part, each part,
 copy by copy by copy by
 rows_sources rows_sources rows_sources
 │ │ │
 ▼ ▼ ▼
 └────────────┬────┴──────────────────┘
 ▼
 result part (Wide): k, a, b, c

What the hack buys you, and what it costs

It buys you peak memory proportional to the widest column, not the sum of the columns. For a table with a 40 MB String column and fifty UInt64 columns, a horizontal merge holds the payload of all the columns at once; a vertical merge holds one column at a time. That is the difference between a merge that OOMs and one that does not — and it is what makes the “wide part” format practical at all.

It costs you IO locality. Each column is a separate pass over the source parts, so a Compact source file is decoded once per column — which is why allow_vertical_merges_from_compact_to_wide_parts defaults to true but carries a warning in its description.

It costs you a temporary file. One byte per row of scratch space: 100 MB for a 100-million-row merge. Trivial compared to the data being merged, but not zero.

It costs you the ability to merge every kind of table. Summing, Aggregating and Coalescing merges have to combine the values of equal keys, which is a merge, not a gather. They stay horizontal, permanently.

It is not automatically faster. The setting description says it plainly: “potentially at the cost of merge speed”. The vertical algorithm optimizes memory, and only optimizes IO when a whole column happens to come from a single source part and can be stolen by pointer.

Seeing it in action

You can see which algorithm a merge used from the profile events:

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN (
 'MergeVerticalStageExecuteMilliseconds',
 'MergeHorizontalStageTotalMilliseconds',
 'MergeVerticalStageTotalMilliseconds'
);

MergeVerticalStageExecuteMilliseconds is only non-zero for tables where the vertical algorithm actually ran. If it is zero on your wide table, check enable_vertical_merge_algorithm, the part format (Compact parts cannot be vertically merged), and the vertical_merge_algorithm_min_* thresholds.

And to see the tables that would benefit:

 sql
 
 
SELECT
 database,
 table,
 formatReadableSize(sum(bytes_on_disk)) AS size,
 sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC
LIMIT 10;

A table with hundreds of columns and a large String payload is exactly the shape the vertical merge was built for. A table with ten narrow columns is not, and the thresholds will keep it horizontal.

Wrapping up

The one thing to remember: the vertical merge is a memory optimization, not a speed one. It trades IO locality for peak memory, and it wins big exactly when one column dominates your table’s width.

The part people get wrong: the whole thing hinges on a one-byte-per-row side channel. If you ever see a merge fail with “It is a bug”, that assertion is the row-count check, and it means the two stages disagreed about which rows exist.

What to go do about it: count your columns. Under 11, or under 16·8192 rows, the vertical algorithm will never kick in — and that is deliberate, not a bug.

ClickHouse Decides Which Parts to Merge: A Cost Model for Write Amplification

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse picks merge candidates with a tiny, explicit cost model: score each contiguous range as sum_size / (count - 1.9), pick the cheapest admissible one, and only ever look at the newest 1000 parts so backlogs cannot snowball. The knobs that control it are merge_selector_base and merge_selector_window_size.

A MergeTree table is a set of immutable parts. Every INSERT adds one. Background merges take contiguous runs of parts and rewrite them as one bigger part, which cuts the number of parts a SELECT has to open — and costs a full rewrite of the data involved.

That trade-off has no free lunch, and the two directions pull against each other:

Merge eagerly and the part count stays low, but you rewrite the same bytes over and over. A part merged into a bigger part, which is merged again, is written twice.
Merge lazily and write amplification is low, but the part count climbs, and every query has to open every part.

ClickHouse’s answer is a small, explicit cost model. And — unusually — the model is documented in the source, together with its own limitations. Which means you can read the reasoning, not just guess at it.

Let’s take the score function, the “base of the logarithm” knob, the sliding window that stops merge backlogs from snowballing, and the background scheduler’s exponential backoff.

The design note

The most valuable thing about ClickHouse’s merge selector is that the authors wrote down what they were trying to do — and where they thought they were wrong. The header of SimpleMergeSelector.h is a design document:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:6
/**
We have a set of data parts that is dynamically changing - new data parts are added and there is background merging process.
Background merging process periodically selects continuous range of data parts to merge.

It tries to optimize the following metrics:
1. Write amplification: total amount of data written on disk (source data + merges) relative to the amount of source data.
It can be also considered as the total amount of work for merges.
2. The number of data parts in the set at the random moment of time (average + quantiles).

Also taking the following considerations:
1. Older data parts should be merged less frequently than newer data parts.
2. Larger data parts should be merged less frequently than smaller data parts.
3. If no new parts arrive, we should continue to merge existing data parts to eventually optimize the table.
4. Never allow too many parts, because it will slow down SELECT queries significantly.
5. Multiple background merges can run concurrently but not too many.

It is not possible to optimize both metrics, because they contradict to each other.
To lower the number of parts we can merge eagerly but write amplification will increase.
Then we need some balance between optimization of these two metrics.

But some optimizations may improve both metrics.

For example, we can look at the "merge tree" - the tree of data parts that were merged.
If the tree is perfectly balanced then its depth is proportional to the log(data size),
the total amount of work is proportional to data_size * log(data_size)
and the write amplification is proportional to log(data_size).
If it's not balanced (e.g. every new data part is always merged with existing data parts),
its depth is proportional to the data size, total amount of work is proportional to data_size^2.
*/

That last paragraph is the key insight of the whole design, so read it twice.

The merges form a tree: leaves are inserted parts, internal nodes are merge results. If the tree is balanced, its depth is log_base(total_size) and total write amplification is proportional to that depth. If the tree degenerates into a chain — every new part merged immediately into the current biggest part — the depth is proportional to the data size, and total write amplification is O(data_size). Every byte gets rewritten once per level.

So the selector’s real job is keeping the merge tree balanced. And the knob that controls balance is the arity of the tree: how many parts get merged at once.

The base of the logarithm

That arity is called base, and the header explains how it generalizes to parts of unequal size:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:35
We can also control the "base of the logarithm" - you can look it as the number of data parts
that are merged at once (the tree "arity"). But as the data parts are of different size, we should generalize it:
calculate the ratio between total size of merged parts to the size of the largest part participated in merge.
For example, if we merge 4 parts of size 5, 3, 1, 1 - then "base" will be 2 (10 / 5).

Base of the logarithm (simply called `base` in `SimpleMergeSelector`) is the main knob to control the write amplification.
The more it is, the less is write amplification but we will have more data parts on average.

To fit all the considerations, we also adjust `base` depending on total parts count,
parts size and parts age, with linear interpolation (then `base` is not a constant but a function of multiple variables,
looking like a section of hyperplane).

So the base of a candidate merge is sum_size / max_size: total size of the range divided by the size of its largest part. Merge four parts of sizes 5, 3, 1, 1 and you get a base of 10/5 = 2.

Now merge a huge part with a tiny one. Base is barely above 1 — and that is bad. That is the degenerate chain, the exact shape the selector exists to avoid.

base is exposed as a MergeTree setting, default 5:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1090
 DECLARE(Float, merge_selector_base, 5.0, R"(Affects write amplification of
 assigned merges (expert level setting, don't change if you don't understand
 what it is doing). Works for Simple and StochasticSimple merge selectors
)", 0) \

The admissibility test: is this range allowed at all?

Before any candidate is scored, it has to pass an admissibility test. The test is allow(), and its central quantity is the lowered base — a base adjusted downward depending on how stale the parts are, how many parts the partition has, and how big the range is.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:241
 /// Map size to 0..1 using logarithmic scale
 /// Use log(1 + x) instead of log1p(x) because our sum_size is always integer.
 /// Also log1p seems to be slow and significantly affect performance of merges assignment.
 double size_normalized = mapPiecewiseLinearToUnit(log(1 + sum_size), min_size_to_lower_base_log, max_size_to_lower_base_log);
 /// Calculate boundaries for age
 double min_age_to_lower_base = interpolateLinear(static_cast<double>(settings.min_age_to_lower_base_at_min_size), static_cast<double>(settings.min_age_to_lower_base_at_max_size), size_normalized);
 double max_age_to_lower_base = interpolateLinear(static_cast<double>(settings.max_age_to_lower_base_at_min_size), static_cast<double>(settings.max_age_to_lower_base_at_max_size), size_normalized);
 /// Map age to 0..1
 double age_normalized = mapPiecewiseLinearToUnit(min_age, min_age_to_lower_base, max_age_to_lower_base);
 /// Map partition_size to 0..1
 double num_parts_normalized = mapPiecewiseLinearToUnit(partition_size, static_cast<double>(settings.min_parts_to_lower_base), static_cast<double>(settings.max_parts_to_lower_base));
 /// The ratio should be within [0, 1]
 double combined_ratio = std::min(1.0, age_normalized + num_parts_normalized);

 double lowered_base = interpolateLinear(settings.base, 2.0, combined_ratio);

Three inputs, three effects:

Input	Effect on `lowered_base`
`age_normalized` — how old the youngest part in the range is	Older parts are cheaper to merge (they are not going to be queried as freshly-inserted data), so the base is lowered and the merge happens sooner.
`num_parts_normalized` — how many parts the partition has	A partition with too many parts is a problem for queries, so the base is lowered and merges get more aggressive.
`size_normalized` — how big the range is	Big ranges need to be older before they are merged, because merging them is expensive.

The mapping is deliberately non-linear in size (logarithmic), and the comment explains why log(1+x) rather than log1p(x): log1p is slow enough to matter, and the selector runs on every selection round.

Then the actual test:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:271
 return (sum_size + static_cast<double>(size) * static_cast<double>(settings.size_fixed_cost_to_add)) / (max_size + static_cast<double>(settings.size_fixed_cost_to_add)) >= lowered_base;

Read it as: “is sum_size / max_size at least lowered_base?” — modulo the fixed cost of merging (size_fixed_cost_to_add, default 5 MB) added to both numerator and denominator, which models the fact that merging even two tiny parts costs a fixed amount of work.

Two escape hatches come before it:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:215
 if (settings.min_age_to_force_merge && min_age >= static_cast<double>(settings.min_age_to_force_merge))
 return true;

 if (settings.min_partition_age_to_force_merge && min_partition_age > 0)
 if (min_partition_age >= static_cast<double>(settings.min_partition_age_to_force_merge))
 return true;

If the parts are old enough — or the whole partition is old enough — the range is merged no matter what the base says. This is the safety valve: the cost model can be wrong, but a part that has been sitting there for a day must eventually be merged.

The score function

Now, given that several ranges are admissible, which one do you pick? This is the score() function, and it is where the “cost model” really lives:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:122
 static double score(double count, double sum_size, double sum_size_fixed_cost)
 {
 /** Consider we have two alternative ranges of data parts to merge.
 * Assume time to merge a range is proportional to sum size of its parts.
 *
 * Cost of query execution is proportional to total number of data parts in a moment of time.
 * Let define our target: to minimize average (in time) total number of data parts.
 *
 * Let calculate integral of total number of parts, if we are going to do merge of one or another range.
 * It must be lower, and thus we decide, what range is better to merge.
 *
 * The integral is lower iff the following formula is lower:
 *
 * sum_size / (count - 1)
 *
 * But we have some tunes to prefer longer ranges.
 */
 return (sum_size + sum_size_fixed_cost * count) / (count - 1.9);
 }

That comment is a genuine, if informal, calculus argument. Merge a range of count parts and the part count drops by count - 1 (you replace count parts with one). The merge itself takes time proportional to sum_size. So the “average number of parts over time” is sum_size / (count - 1) — and lower is better. The range with the smallest score is the one to merge.

Now the - 1.9. With a plain - 1 the denominator vanishes for a two-part merge, giving an infinite score. The authors replaced - 1 with - 1.9, which softens the singularity: merging two parts is penalized less than the pure model would suggest, and merging many parts is penalized more. The comment’s own summary is “we have some tunes to prefer longer ranges”.

size_fixed_cost_to_add (5 MB by default) enters the numerator as sum_size + fixed * count — every part in the range carries the fixed cost of being merged, so a range of ten tiny parts is far more expensive than its byte sum suggests.

The candidate enumeration walks every contiguous sub-range of the partition and scores each one:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:108
 double current_score = score(static_cast<double>(end - begin), static_cast<double>(sum_size), static_cast<double>(settings.size_fixed_cost_to_add));

 if (settings.enable_heuristic_to_align_parts
 && static_cast<double>(size_prev_at_left) > static_cast<double>(sum_size) * settings.heuristic_to_align_parts_min_ratio_of_sum_size_to_prev_part)
 {
 double difference = std::abs(log2(static_cast<double>(sum_size) / static_cast<double>(size_prev_at_left)));
 if (difference < settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two)
 current_score *= interpolateLinear(settings.heuristic_to_align_parts_max_score_adjustment, 1,
 difference / settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two);
 }

The “align parts” heuristic prefers ranges whose total size is roughly a power of two times the size of the part to their left. Why? Because a balanced merge tree wants each level to double the size. If the part on the left is 100 MB and the candidate range is 100 MB, the range “fits” the tree; at 30 MB or 300 MB the resulting tree is lopsided and the next merge has to rewrite more. The score is scaled down by up to 25% (heuristic_to_align_parts_max_score_adjustment = 0.75) when the sizes sit close to a power of two apart.

The right-tail heuristic

One more adjustment, applied before scoring, and it is the most “heuristic” of them all:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:51
 const bool should_remove_small_parts_at_right = settings.enable_heuristic_to_remove_small_parts_at_right
 && (!settings.merge_selector_min_age_to_disable_right_tail_heuristic
 || min_age < settings.merge_selector_min_age_to_disable_right_tail_heuristic);
 if (should_remove_small_parts_at_right)
 {
 size_t size_delta = 0;
 size_t rows_delta = 0;
 while (end >= begin + 3 && static_cast<double>((end - 1)->size) < settings.heuristic_to_remove_small_parts_at_right_max_ratio * static_cast<double>(sum_size))
 {
 size_delta += (end - 1)->size;
 rows_delta += (end - 1)->rows;
 --end;
 }

 sum_size -= size_delta;
 sum_rows -= rows_delta;
 }

The idea: a range that ends with a couple of very small parts is a bad merge. Those small parts are probably new parts that just arrived, and folding them into a large merge means they sit in the merge queue for a long time. Worse, it means freshly-arrived data is not queryable quickly. So the heuristic trims the right tail of the range — the small parts at the newest end — and scores what remains.

The default ratio is 1%: a part at the right end smaller than 1% of the range’s total gets trimmed, as long as at least three parts remain. And there is an age-based escape hatch (merge_selector_min_age_to_disable_right_tail_heuristic): once the range is old enough the tail is kept, because at that point staleness is worse than freshness.

The sliding window: anti-snowball protection

The single most valuable piece of engineering in the selector is this:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:307
 /// If the parts in the parts vector are sorted by block number,
 /// it may not be ideal to only select parts for merging from the first N ones.
 /// This is because if there are more than N parts in the partition,
 /// we will not be able to assign a merge for newly created parts.
 /// As a result, the total number of parts within the partition could
 /// grow uncontrollably, similar to a snowball effect.
 /// To address this we will try to assign a merge taking into consideration
 /// only last N parts.
 const size_t parts_threshold = settings.window_size;
 size_t begin = 0;
 if (parts_count >= parts_threshold)
 {
 if (settings.enable_stochastic_sliding)
 begin = calculateRangeWithStochasticSliding(parts_count, parts_threshold);
 else
 begin = parts_count - parts_threshold;
 }

window_size defaults to 1000. The comment describes the failure mode it prevents, and it is a real one:

If the parts in the parts vector are sorted by block number, it may not be ideal to only select parts for merging from the first N ones. This is because if there are more than N parts in the partition, we will not be able to assign a merge for newly created parts. As a result, the total number of parts within the partition could grow uncontrollably, similar to a snowball effect.

Picture it: if the selector always preferred the oldest parts, then in a partition with 10 000 parts it would forever be merging parts 1..100 — ancient, useless candidates — while parts 9000..10000, the ones that arrived in the last minute, never get merged. The part count never goes down. The merge queue is permanently busy with work that does not help.

By only ever considering the last N parts, every merge consumes recent parts, so recent parts get compacted and the partition’s part count actually falls. One line of code, and it kills a self-reinforcing failure mode.

The stochastic variant (StochasticSimpleMergeSelector) randomizes the window position, so different replicas do not all merge the same parts:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:275
size_t calculateRangeWithStochasticSliding(size_t parts_count, size_t parts_threshold)
{
 auto mean = static_cast<double>(parts_count);
 std::normal_distribution<double> distribution{mean, mean / 4};

 size_t right_boundary = static_cast<size_t>(distribution(thread_local_rng));
 if (right_boundary > parts_count)
 right_boundary = 2 * parts_count - right_boundary;

 right_boundary = std::max(right_boundary, parts_threshold);

 return right_boundary - parts_threshold;
}

A normal distribution with mean parts_count and standard deviation parts_count / 4, folded back into range by mirroring at parts_count. And note the std::max(right_boundary, parts_threshold) at the end: the window never shrinks below window_size. The anti-snowball guarantee is never traded away for randomization.

The blurry base

StochasticSimpleMergeSelector adds one more trick: instead of a sharp threshold on the base, it uses a probability distribution around the lowered base.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:256
 if (settings.use_blurry_base)
 {
 double partition_fill_factor = std::max(0., 1 - partition_size / static_cast<double>(settings.parts_to_throw_insert));
 /// Scale factor controls when (relativelty to the number of parts in partition)
 /// do we activate our special algorithm.
 /// With standard parameters the logic kicks in starting from 80% empty factor.
 /// The division by 2 is due to the fact that for normal distribution nearly 95.4%
 /// of all observations fall within two standard deviations.
 double scaling_factor = std::pow(partition_fill_factor, settings.blurry_base_scale_factor) / 2;
 /// The base lower than 1 doesn't make sense, so we try to avoid it.
 std::normal_distribution<double> distribution{lowered_base, (lowered_base - 1) * scaling_factor};
 /// The threshold should be strictly bigger than 1, because we don't allow to merge the part with itself.
 lowered_base = std::min(distribution(thread_local_rng), std::max(1.01, lowered_base));
 }

Instead of “merge if base ≥ 5.0”, it says “merge if base ≥ a randomly-drawn threshold around 5.0”. A merge whose base is 4.9 might happen this round and not the next. The result: the distribution of merges is smoother, no sharp cliff at a particular base value, and — critically — the merge decisions of different replicas do not line up.

The width of the distribution grows with the emptiness of the partition: partition_fill_factor is 0 when the partition is full (parts_to_throw_insert parts) and approaches 1 when it is empty. blurry_base_scale_factor (default 42) makes the effect very nonlinear, so the blur only kicks in when the partition is nearly empty — exactly when you want merges to be infrequent and gentle.

The final std::min(distribution(...), std::max(1.01, lowered_base)) is a careful clamp: the drawn threshold can be lower than the base, making the merge more eager, but never lower than 1.01, because a base ≤ 1 would mean merging a part with itself.

Picking the winner

The scored candidates go into a max-heap, and the best admissible, non-overlapping one wins:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:142
 std::optional<PartsRange> buildMergeRange(const MergeConstraint & constraint)
 {
 constexpr static auto range_compare = [](const ScoredRange & lhs, const ScoredRange & rhs)
 {
 /// If the ranges have the same score, use the range that is most likely to have the lower merge level -
 /// the right one, according to the block number sorting.
 return lhs.score > rhs.score || (lhs.score == rhs.score && lhs.range_begin < rhs.range_begin);
 };

Note the tie-break: “the right one”, i.e. the newer parts. That is the opposite of what a naive reading of “oldest first” would suggest, and it is consistent with the sliding window.

The disjoint_set.addRangeIfPossible call in the pop loop enforces that two concurrent merges never claim overlapping ranges of parts. The disjoint set is implemented as a std::set of range boundaries per independent range, with an upper_bound lookup:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/DisjointPartsRangesSet.cpp:113
bool DisjointPartsRangesSet::addRangeIfPossible(RangesIterator range_it, PartsIterator range_begin, PartsIterator range_end)
{
 chassert(sanityCheck(uncovered_ranges, range_it, range_begin, range_end));

 auto & sorted_ranges = disjoint[range_it.base()];
 PartsRangeBoundaries range_boundaries{std::move(range_begin), std::move(range_end)};

 if (!isDisjoint(range_boundaries, sorted_ranges))
 return false;

 sorted_ranges.insert(std::move(range_boundaries));
 return true;
}

From parts to a merge: the whole pipeline

The selector is one step of a longer pipeline. The entry point is MergeTreeDataMergerMutator::selectPartsToMerge:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:328
std::expected<MergeSelectorChoices, SelectMergeFailure> MergeTreeDataMergerMutator::selectPartsToMerge(
 const PartsCollectorPtr & parts_collector,
 const MergePredicatePtr & merge_predicate,
 const MergeSelectorApplier & selector,
 const std::optional<PartitionIdsHint> & partitions_hint)
{
 const auto context = data.getContext();
 const auto settings = data.getSettings();
 const auto metadata_snapshot = data.getInMemoryMetadataPtr(context, false);
 const auto storage_policy = data.getStoragePolicy();
 const time_t current_time = std::time(nullptr);
 const bool can_use_ttl_merges = !ttl_merges_blocker.isCancelled();
 LogSeriesLimiter series_log(log, 1, /*interval_s_=*/60 * 30);

 auto collected = collectAllPossibleRanges(parts_collector, metadata_snapshot, storage_policy, current_time, partitions_hint, series_log);

Three phases:

collect — find the ranges of parts that could be merged (same partition, compatible schema, not already covered by a running merge);
split by merge predicate — cut those ranges wherever a merge is forbidden (a part that overlaps a running merge, a part with a different schema version, and so on);
choose — run the selector’s cost model over what is left.

The LogSeriesLimiter is a nice touch: the trace log of this function is rate-limited to one message per 30 minutes, because the function runs on every selection round and can produce a lot of output.

The background scheduler’s backoff

Selection is cheap; the merge is not. The background scheduler that runs the selection loop uses an exponential backoff when there is nothing to do:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:62
void BackgroundJobsAssignee::postpone()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 no_work_done_count += 1;
 double random_addition = std::uniform_real_distribution<double>(0, sleep_settings.task_sleep_seconds_when_no_work_random_part)(rng);

 size_t next_time_to_execute = static_cast<size_t>(
 1000 * (data.getBiasBackoffSeconds() + std::min(
 sleep_settings.task_sleep_seconds_when_no_work_max,
 sleep_settings.thread_sleep_seconds_if_nothing_to_do * std::pow(sleep_settings.task_sleep_seconds_when_no_work_multiplier, no_work_done_count))
 + random_addition));

 holder->scheduleAfter(next_time_to_execute, false);
}

with the settings:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.h:18
struct BackgroundTaskSchedulingSettings
{
 double thread_sleep_seconds_random_part = 1.0;
 double thread_sleep_seconds_if_nothing_to_do = 0.1;
 double task_sleep_seconds_when_no_work_max = 600;
 /// For exponential backoff.
 double task_sleep_seconds_when_no_work_multiplier = 1.1;

 double task_sleep_seconds_when_no_work_random_part = 1.0;

So: start at 0.1 s, multiply by 1.1 per idle round, cap at 600 s, plus up to 1 s of random jitter. An idle table’s selection loop slows to one attempt every ten minutes after about 130 idle rounds, and the jitter stops a fleet of servers from synchronizing their wake-ups.

And when work does appear, the backoff is unwound geometrically rather than reset:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:48
void BackgroundJobsAssignee::trigger()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 /// Do not reset backoff factor if some task has appeared,
 /// but decrease it exponentially on every new task.
 no_work_done_count /= 2;
 /// We have background jobs, schedule task as soon as possible
 holder->schedule();
}

no_work_done_count /= 2 rather than = 0. Halving means that after one new task the sleep time is back to roughly its previous value, and it takes several rounds of activity to fully unwind. A table that receives one part per minute therefore does not flip between “poll every 0.1 s” and “poll every 600 s” — it settles at a rate proportional to the arrival rate.

Here’s the thing: that is genuinely nice control theory hiding inside what looks like three lines of code.

The whole picture

 text
 
 
 parts in a partition (by block number)
 ┌────┬────┬────┬────┬────┬──────────────┬────┬────┐
 │ p1 │ p2 │ p3 │ p4 │ p5 │ ... │pn-1│ pn │
 └────┴────┴────┴────┴────┴──────────────┴────┴────┘
 ↑ ↑
 └──────── sliding window (last N=1000) ─────┘
 (only these are eligible)

 for each contiguous sub-range of the window:
 base = sum_size / max_size
 if base < lowered_base(age, partition_size, range_size):
 skip
 score = (sum_size + 5MB * count) / (count - 1.9)
 trim small parts off the right tail
 adjust score for power-of-two alignment with the part on the left

 pick the lowest-scoring admissible range that does not overlap
 an already-assigned merge

What this buys you, and what it costs

It buys you a merge tree that stays balanced, which is what keeps write amplification at O(log n) instead of O(n). The base knob is the lever, and the whole selector is a machine for keeping base near its target.

It buys you a system that self-corrects. When a partition fills up, num_parts_normalized rises, the base is lowered, and merges become more eager — without anybody tuning anything. When a partition sits idle, the opposite happens.

It buys you an anti-snowball guarantee. The sliding window is the difference between a table whose part count stabilizes and one that grows without bound.

It costs you a cost model that is admittedly approximate. The header comment says so:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:57
It's still unclear if this algorithm is good or optimal at all. It's unclear if this algorithm is using the optimal coefficients.

and it describes how the coefficients were tuned (an insert/merge simulator, replayed against production system.part_log data), and how the research found slightly better coefficients that were not used because the test data’s representativeness was in question.

It costs you a - 1.9. The score function’s singularity is papered over with a constant rather than derived. It is a fudge factor, honestly labelled as one.

It costs you a lot of tuning surface. merge_selector_base, merge_selector_window_size, parts_to_throw_insert, max_parts_to_merge_at_once, min_parts_to_merge_at_once, merge_selector_small_parts_*, the align-parts heuristic, the right-tail heuristic, the blurry base, the stochastic sliding window. Every one of these is a lever that was pulled for a reason, and the reason is usually written in a comment somewhere in SimpleMergeSelector.h.

Seeing it in action

The merge list shows what was selected and why:

 sql
 
 
SELECT
 database,
 table,
 elapsed,
 progress,
 num_parts,
 formatReadableSize(total_size_bytes_compressed) AS size,
 formatReadableSize(memory_usage) AS mem,
 result_part_name
FROM system.merges
ORDER BY elapsed DESC;

And the parts-per-partition histogram is the metric the selector is ultimately optimizing:

 sql
 
 
SELECT
 database,
 table,
 partition,
 count() AS parts,
 sum(rows) AS rows,
 formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active
GROUP BY database, table, partition
ORDER BY parts DESC
LIMIT 20;

A partition with hundreds of parts means the selector is not keeping up — usually because the merge pool is saturated, or because the parts are too large to merge under parts_to_throw_insert pressure, or because merge_selector_base is set too high for the workload.

And if you want to watch the cost model work, the trace log has it:

 sql
 
 
SET log_level = 'trace';
SELECT count() FROM hits WHERE URL LIKE '%very-selective-prefix%';

The MergeTreeDataMergerMutator::selectPartsToMerge trace messages show which ranges were considered and which one was chosen.

Wrapping up

The one thing to remember: the merge tree is a tree, and a balanced tree costs O(log n) to write while a chain costs O(n). merge_selector_base is the knob that decides which one you get.

The part people get wrong: the sliding window is not an optimization, it is a correctness guard. Without it, a busy partition can keep a permanent merge queue full of work that never reduces the part count.

What to go do about it: run the parts-per-partition query against your biggest tables. If a partition is crawling toward parts_to_throw_insert, the merge pool is the bottleneck, not the selector — and no amount of knob twiddling will fix that.

Merge-Tree on Opensource internals

ClickHouse's Query Condition Cache: Skip Granules You Don't Need to Read

What the cache actually stores

The key: three components, hashed

The safety property: everything starts as “may match”

Writing: only clear bits, never set them

Why the entry has its own mutex

Reading: turning a bitmap back into mark ranges

The predicate hash is not just the predicate

The time-condition trick

The skip-index profile: another salt

Where the “empty granule” fact comes from

The whole flow

What this buys you, and what it costs

Seeing it in action

ClickHouse Skips Data: Granules, Marks, and Bit-Packed Indexes

Granules: sized in bytes, not rows

Marks: where in the file does each granule start?

Where do the marks come from at write time?

The mark cache

The primary key index: one key value per granule

The trimming trick

The primary index cache

Now the payoff: two different search algorithms

Algorithm 1: binary search (continuous range)

Algorithm 2: generic exclusion search (everything else)

The step budget

The seek heuristic

Putting it together

What this buys you, and what it costs

Seeing it in action

Wrapping up

ClickHouse Merges Columns One at a Time: The Vertical Merge Algorithm

The problem, stated precisely

The horizontal stage: merging the key and writing the row sources

The encoding: one byte per row

Who writes the bytes

The vertical stage: gathering one column at a time

The gather loop

Writing the gathered column

When is it worth it?

The Wide-only restriction, and the escape hatch

Progress accounting

The horizontal stage’s contribution: the row count assertion

Prefetching the next column

The whole flow

What the hack buys you, and what it costs

Seeing it in action

Wrapping up

ClickHouse Decides Which Parts to Merge: A Cost Model for Write Amplification

The design note

The base of the logarithm

The admissibility test: is this range allowed at all?

The score function

The right-tail heuristic

The sliding window: anti-snowball protection

The blurry base

Picking the winner

From parts to a merge: the whole pipeline

The background scheduler’s backoff

The whole picture

What this buys you, and what it costs

Seeing it in action

Wrapping up

The `Wide`-only restriction, and the escape hatch