Internals on Opensource internals

How ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A Variant column is a tagged union: each row holds a value of one of several types, and the column itself knows the full list of alternatives. Dynamic is the same idea with an open-ended list — the set of types is discovered from the data.

The obvious implementation is a struct-of-arrays: one sub-column per alternative, each holding only the rows that belong to it, plus a tag per row saying which sub-column to look in. That is what ClickHouse does. What makes the implementation interesting is not the tag — it is the set of awkward consequences that fall out of it:

The order of the alternatives must not matter (Variant(String, UInt64) and Variant(UInt64, String) are the same type), but the storage of a discriminator is a small integer, so the order does matter on disk.
The type must be extendable at runtime without rewriting the data, which means a column’s internal ordering may differ from the type’s ordering.
The number of alternatives must be bounded, but a Dynamic column cannot know its alternatives in advance, so it needs an escape hatch.
Reading a row must not require a virtual call and a branch on the tag.

Let’s look at how ClickHouse 26.9 solves all four.

The layout

The header comment of ColumnVariant is a complete specification, including a worked example:

 cpp
 
 
// src/Columns/ColumnVariant.h:12
/**
 * Column for storing Variant(...) type values.
 * Variant type represents a union of other data types.
 * For example, type Variant(T1, T2, ..., TN) means that each row of this type
 * has a value of either type T1 or T2 or ... or TN or none of them (NULL value)
 *
 * ColumnVariant stores:
 * - The discriminators column, which determines which variant is stored in each row.
 * - The offsets column, which determines the offset in the corresponding variant column in each row.
 * - The list of variant columns with only real values (so the sizes of variant columns can be different).
 * Discriminator is an index of a variant in the variants list, it also has special value called NULL_DISCRIMINATOR
 * that indicates that the value in the row is NULL.
 *
 * We want to be able to extend Variant column for free without rewriting the data, but as we don't care about the
 * order of variants during Variant creation (we want Variant(T1, T2) to be the same as Variant(T2, T1)), we support
 * some global order of nested types inside Variant during type creation, so after extension the order of variant types
 * (and so their discriminators) can change. For example: Variant(T1, T3) -> Variant(T1, T2, T3).
 * To avoid full rewrite of discriminators column on Variant extension, we differentiate local order of variants
 * inside a column and global order of variants created during type creation. So, ColumnVariant stores only local
 * discriminators and additionally stores the mapping between global and local discriminators.
 * So, when we need to extend Variant column with new variant, we can just append it to a list of variant columns
 * with new local discriminator and update mapping from global to local orders.
 *
 * Note that two instances of ColumnVariant can have different local orders, so we should always use global
 * discriminators during inter-column interactions.
 *
 * Let's take an example with type Variant(UInt32, String, Array(UInt32)):
 * During type creation we will sort types by their names and get the global order: Array(UInt32), String, UInt32.
 * So, type Array(UInt32) will have global discriminator 0, String - 1 and UInt32 - 2.
 * Let's say we have a column with local order (String, UInt32, Array(UInt32)) and values:
 * 'Hello', 42, NULL, 'World', 43, [1, 2, 3], NULL, 44
 *
 * Let's see how these values will be stored in ColumnVariant:
 *
 * local_to_global_discriminators: {0 : 1, 1 : 2, 2 : 0}
 * global_to_local_discriminators: {0 : 2, 1 : 0, 2 : 1}
 * local_discriminators offsets String UInt32 Array(UInt32)
 * 0 0 'Hello' 42 [1, 2, 3]
 * 1 0 'World' 43
 * NULL_DISCRIMINATOR 0 44
 * 0 1
 * 1 1
 * 2 0
 * NULL_DISCRIMINATOR 0
 * 1 2
 *
 */

That diagram is worth staring at, because it shows the two central ideas:

text

 row: 0 1 2 3 4 5 6 7
 local: 0 1 NULL 0 1 2 NULL 1
 offsets: 0 0 0 1 1 0 0 2

 String: 'Hello' 'World' (2 values)
 UInt32: 42 43 44 (3 values)
 Array: [1,2,3] (1 value)

The discriminator is one byte per row. It is a local index into the column’s own list of variant columns.
The offset is a row index within that particular variant column. Row 3 has local discriminator 0 (String) and offset 1, so its value is String[1] == 'World'. Row 5 has local discriminator 2 (Array) and offset 0, so its value is Array[0].

So per row you pay one byte of tag plus one Offset (UInt64) of position. The values themselves are stored densely, one after another, inside their own sub-column — a String sub-column holds only the strings, a UInt32 sub-column holds only the integers, and neither has holes.

The types involved:

 cpp
 
 
// src/Columns/ColumnVariant.h:62
 using Discriminator = UInt8;
 using Discriminators = PaddedPODArray<Discriminator>;
 using ColumnDiscriminators = ColumnVector<Discriminator>;
 using ColumnOffsets = ColumnVector<Offset>;

 static constexpr UInt8 NULL_DISCRIMINATOR = std::numeric_limits<Discriminator>::max(); /// 255
 static constexpr size_t MAX_NESTED_COLUMNS = std::numeric_limits<Discriminator>::max(); /// 255

NULL_DISCRIMINATOR is 255 — one past the largest legal alternative index. That means a column can hold at most 255 alternatives, and 255 is reserved for NULL. The value is not stored in any sub-column; it is only a tag.

Global vs local order

The tag stored in the column is a local discriminator. The global discriminator is the position in the type’s canonical ordering. The canonical ordering is by type name — that is how Variant(T1, T2) and Variant(T2, T1) become the same type:

 cpp
 
 
// src/DataTypes/DataTypeVariant.cpp:46
DataTypeVariant::DataTypeVariant(const DataTypes & variants_)
{
 /// Sort nested types by their full names and squash identical types.
 std::map<String, DataTypePtr> name_to_type;
 for (const auto & type : variants_)
 {
 checkAllowedInsideVariant(type);

 /// Don't use Nothing type as a variant.
 if (!isNothing(type))
 name_to_type[type->getName()] = type;
 }

 variants.reserve(name_to_type.size());
 for (const auto & [_, type] : name_to_type)
 variants.push_back(type);

 checkVariantsNotEmptyAndNotTooMany(variants);
}

A std::map keyed by the type name gives the sorted, de-duplicated order for free.

Now, why do we need two orders at all? Because a Dynamic column is created with one set of alternatives and then extended as new types arrive. When Variant(T1, T3) becomes Variant(T1, T2, T3), the canonical (global) order of T3 may change — T2 might sort before it. If the column stored global discriminators, every stored tag would have to be rewritten.

The fix is to keep the column’s local order stable and just maintain a mapping:

 cpp
 
 
// src/Columns/ColumnVariant.h:160
 Discriminator ALWAYS_INLINE globalDiscriminatorByLocal(Discriminator local_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return local_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : local_to_global_discriminators[local_discr];
 }

 Discriminator ALWAYS_INLINE localDiscriminatorByGlobal(Discriminator global_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return global_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : global_to_local_discriminators[global_discr];
 }

and the extension operation is a pure metadata change:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1904
void ColumnVariant::extend(const VectorWithMemoryTracking<Discriminator> & old_to_new_global_discriminators, VectorWithMemoryTracking<std::pair<MutableColumnPtr, Discriminator>> && new_variants_and_discriminators)
{
 /// Update global discriminators for current variants.
 for (Discriminator & global_discr : local_to_global_discriminators)
 global_discr = old_to_new_global_discriminators[global_discr];

 /// Add new variants.
 variants.reserve(variants.size() + new_variants_and_discriminators.size());
 local_to_global_discriminators.reserve(local_to_global_discriminators.size() + new_variants_and_discriminators.size());
 for (auto & new_variant_and_discriminator : new_variants_and_discriminators)
 {
 variants.emplace_back(std::move(new_variant_and_discriminator.first));
 local_to_global_discriminators.push_back(new_variant_and_discriminator.second);
 }

 /// Update global -> local discriminators matching.
 global_to_local_discriminators.resize(local_to_global_discriminators.size());
 for (Discriminator local_discr = 0; local_discr != local_to_global_discriminators.size(); ++local_discr)
 global_to_local_discriminators[local_to_global_discriminators[local_discr]] = local_discr;
}

Nothing in that function touches the discriminator column or the offsets column. The existing rows’ local tags stay valid; only the interpretation of the tags changes. That is the “extend Variant column for free without rewriting the data” promise, kept.

The invariant is that local discriminators are always dense: local discriminator k is the index of variants[k]. Global discriminators are sparse in the sense that a given column may not contain all of them.

Reading a row

 cpp
 
 
// src/Columns/ColumnVariant.cpp:415
Field ColumnVariant::operator[](size_t n) const
{
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return Null();
 return (*variants[discr])[offsetAt(n)];
}

Two loads and a branch. The branch on NULL_DISCRIMINATOR is unavoidable; what you want to avoid is a virtual call per row, and variants[discr] is an IColumn&, so operator[] on it is a virtual call. That is why the hot paths in ClickHouse do not use operator[] but instead reach into the concrete sub-column via getDataAt, and why ColumnVariant also exposes getVariantByLocalDiscriminator so that callers can hoist the type test out of the loop.

Note also that the offsets array is not a byte offset into a shared arena. It is an element index into the sub-column. That is what allows each sub-column to be an ordinary ClickHouse column with its own type-specific representation (a ColumnString with offsets+chars, a ColumnVector<UInt32>, a ColumnArray, …), and it is why byteSizeAt has to consult the sub-column:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1564
size_t ColumnVariant::byteSizeAt(size_t n) const
{
 size_t res = sizeof(Offset) + sizeof(Discriminator);
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return res;

 return res + variants[discr]->byteSizeAt(offsetAt(n));
}

So a Variant row costs 9 bytes of overhead (1 discriminator + 8 offset) plus the payload. The offsets are the expensive part, and that is deliberate: an offset into a typed sub-column is what lets the sub-column be stored in its native, compressible layout.

Building the offsets

The offsets column is derived from the discriminators. constructOffsetsFromDiscriminators computes it, and it has three special cases:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:69
void ColumnVariant::constructOffsetsFromDiscriminators()
{
 const ColumnDiscriminators * discriminators_concrete = typeid_cast<const ColumnDiscriminators *>(local_discriminators.get());
 Offsets & offsets_data = typeid_cast<ColumnOffsets *>(offsets.get())->getData();
 offsets_data.clear();
 offsets_data.reserve(discriminators_concrete->size());
 /// If we have only NULLs, offsets column will not contain any real offsets.
 if (hasOnlyNulls())
 {
 /// Offsets of NULL rows are never read for their value, but compression reads the whole buffer.
 offsets_data.resize_fill(discriminators_concrete->size(), 0);
 }
 /// If we have only one non empty variant and no NULLs,
 /// offsets column will contain just sequential offsets 0, 1, 2, ...
 else if (getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 for (size_t i = 0; i != discriminators_concrete->size(); ++i)
 offsets_data.push_back(i);
 }
 /// Otherwise we should iterate through discriminators and
 /// remember current offset for each variant column.
 else
 {
 VectorWithMemoryTracking<Offset> nested_offsets;
 nested_offsets.resize(variants.size());
 for (Discriminator discr : discriminators_concrete->getData())
 {
 if (discr == NULL_DISCRIMINATOR)
 {
 offsets_data.emplace_back();
 }
 else
 {
 offsets_data.push_back(nested_offsets[discr]);
 ++nested_offsets[discr];
 }
 }
 }
}

The general case is a running counter per variant — a “rank within the sub-column” computation. The special cases matter for the serializer: if every row is NULL, the offsets buffer is all zeros (and the comment notes that compression reads the whole buffer, so it cannot be empty); if there is exactly one non-empty variant, the offsets are simply 0, 1, 2, ... and the whole offsets column is unnecessary information.

Insertion keeps the two arrays in sync

The discriminator and the offsets column must always have the same length. Because an allocation failure in the middle of an insert could leave them inconsistent, the code is careful:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:737
void ColumnVariant::insertDefault()
{
 getLocalDiscriminators().push_back(NULL_DISCRIMINATOR);
 /// Keep local_discriminators and offsets in sync even if appending to offsets throws
 /// (e.g. on a memory limit), otherwise popBack would over-pop the shorter one.
 try
 {
 getOffsets().emplace_back();
 }
 catch (...)
 {
 getLocalDiscriminators().popBack();
 throw;
 }
}

That is exception-safety hygiene that you will see repeatedly in this code base: the two columns are a matched pair, and one of them can throw (memory allocation) while the other cannot.

The insert path

Inserting a value from another variant column is where the global/local mapping earns its keep:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:501
void ColumnVariant::insertFromImpl(const DB::IColumn & src_, size_t n, const VectorWithMemoryTracking<ColumnVariant::Discriminator> * global_discriminators_mapping)
{
 const size_t num_variants = variants.size();
 const ColumnVariant & src = assert_cast<const ColumnVariant &>(src_);

 if (!global_discriminators_mapping && src.variants.size() != num_variants)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot insert value of Variant type with different number of types");

 Discriminator src_global_discr = src.globalDiscriminatorAt(n);
 Discriminator global_discr = src_global_discr;
 if (global_discriminators_mapping && src_global_discr != NULL_DISCRIMINATOR)
 global_discr = (*global_discriminators_mapping)[src_global_discr];
 Discriminator local_discr = localDiscriminatorByGlobal(global_discr);
 getLocalDiscriminators().push_back(local_discr);
 if (local_discr == NULL_DISCRIMINATOR)
 {
 getOffsets().emplace_back();
 }
 else
 {
 getOffsets().push_back(variants[local_discr]->size());
 variants[local_discr]->insertFrom(src.getVariantByGlobalDiscriminator(src_global_discr), src.offsetAt(n));
 }
}

Read it as a three-step translation: source local tag → source global tag → destination local tag. The source’s local tag means nothing to the destination; the global tag is the portable currency; the destination’s local tag is what gets stored.

Serialization: compact discriminators

On disk, the discriminator stream has two modes:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.h:54
 struct DiscriminatorsSerializationMode
 {
 enum Value
 {
 BASIC = 0, /// Store the whole discriminators column.
 COMPACT = 1, /// Don't write discriminators in granule if all of them are the same.
 };

In BASIC mode the whole discriminator column is written. In COMPACT mode a granule whose discriminators are all the same writes one discriminator plus a marker byte instead of 65 536 of them. The writer has to detect those three situations (uniform non-empty, all NULL, mixed) and there is a fast path for each:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:343
 /// Write number of rows in this granule in compact mode.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 writeVarUInt(UInt64(limit), *discriminators_stream);

 /// If column has only one none empty discriminators and no NULLs we don't need to
 /// calculate limits for variants and use provided offset/limit.
 if (auto non_empty_local_discr = col.getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 auto non_empty_global_discr = col.globalDiscriminatorByLocal(*non_empty_local_discr);

 /// In compact mode write the format of the granule and single non-empty discriminator.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 {
 writeBinaryLittleEndian(UInt8(CompactDiscriminatorsGranuleFormat::COMPACT), *discriminators_stream);
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }
 /// For basic mode just serialize this discriminator limit times.
 else
 {
 for (size_t i = 0; i < limit; ++i)
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }

The “one non-empty variant” fast path also lets the value streams be serialized with the same offset/limit as the whole column, because in that case the sub-column has exactly the same number of rows as the variant column:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:370
 /// We can use the same offset/limit as for whole Variant column
 if (i == non_empty_global_discr)
 {
 const auto & variant_column = col.getVariantByGlobalDiscriminator(i);
 if (variant_column.size() < offset + limit)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Variant {} has less rows ({}) than expected rows to serialize ({})", variant_names[i], variant_column.size(), offset + limit);

 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(variant_column, offset, limit, settings, variant_state->variant_states[i]);
 }
 else
 {
 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(col.getVariantByGlobalDiscriminator(i), col.getVariantByGlobalDiscriminator(i).size(), 0, settings, variant_state->variant_states[i]);
 }

In the general case each sub-column is serialized with its own length, which is the whole point of the dense layout: the String sub-column is written as a String column with only the strings in it, and so it compresses as well as a plain string column would.

Dynamic: the open-ended case

Dynamic is a Variant whose alternative list is discovered at runtime. It is implemented as a ColumnVariant plus a VariantInfo record:

 cpp
 
 
// src/Columns/ColumnDynamic.h:15
/**
 * Column for storing Dynamic type values.
 * Dynamic column allows to insert and store values of any data types inside.
 * Inside it stores:
 * - Variant column with all inserted values of different types.
 * - Information about currently stored variants.
 *
 * When new values are inserted into Dynamic column, the internal Variant
 * type and column are extended if the inserted value has new type.
 * When the limit on number of dynamic types is exceeded, all values
 * with new types are inserted into special shared variant with type String
 * that contains values and their types in binary format.
 */
class ColumnDynamic final : public COWHelper<IColumnHelper<ColumnDynamic>, ColumnDynamic>
{
public:
 /// Maximum limit on dynamic types. We use ColumnVariant to store all the types,
 /// so the limit cannot be greater then ColumnVariant::MAX_NESTED_COLUMNS.
 /// We also always have reserved variant for shared variant.
 static constexpr size_t MAX_DYNAMIC_TYPES_LIMIT = ColumnVariant::MAX_NESTED_COLUMNS - 1;
 static constexpr const char * SHARED_VARIANT_TYPE_NAME = "SharedVariant";

The escape hatch is a shared variant: one of the alternatives is always a String column, and it holds the type name and the binary-serialized value of anything that did not fit into a real alternative.

It has to be a String with a custom name, because a plain String alternative might legitimately be in the same column:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:71
/// Shared variant will contain String values but we cannot use usual String type
/// because we can have regular variant with type String.
/// To solve it, we use String type with custom name for shared variant.
const DataTypePtr & ColumnDynamic::getSharedVariantDataType()
{
 static thread_local DataTypePtr type = DataTypeFactory::instance().getCustom("String", std::make_unique<DataTypeCustomDesc>(std::make_unique<DataTypeCustomFixedName>(getSharedVariantTypeName())));
 return type;
}

and a fresh Dynamic column starts life with only the shared variant:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:80
ColumnDynamic::ColumnDynamic(size_t max_dynamic_types_) : max_dynamic_types(max_dynamic_types_), global_max_dynamic_types(max_dynamic_types)
{
 /// Create Variant with shared variant.
 setVariantType(std::make_shared<DataTypeVariant>(DataTypes{getSharedVariantDataType()}));
}

The insert path

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:257
void ColumnDynamic::insert(const Field & x)
{
 if (x.isNull())
 {
 insertDefault();
 return;
 }

 auto & variant_col = getVariantColumn();
 auto shared_variant_discr = getSharedVariantDiscriminator();
 /// Check if we can insert field into existing variants and avoid Variant extension.
 for (ColumnVariant::Discriminator i = 0; i != variant_col.getNumVariants(); ++i)
 {
 if (i != shared_variant_discr && variant_col.getVariantByGlobalDiscriminator(i).tryInsert(x))
 {
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(i));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(i).size() - 1);
 return;
 }
 }

 /// If we cannot insert field into current variant column, extend it with new variant for this field from its type.
 /// Use LeastSupertypeOnError::Dynamic so that arrays with incompatible element types (e.g. ["text", {"k":1}])
 /// are typed as Array(Dynamic) rather than throwing NO_COMMON_TYPE. Dynamic can hold any element value.
 auto field_data_type = applyVisitor(FieldToDataType<LeastSupertypeOnError::Dynamic>(), x);
 auto field_data_type_name = field_data_type->getName();
 if (addNewVariant(field_data_type, field_data_type_name))
 {
 /// Insert this field into newly added variant.
 auto discr = variant_info.variant_name_to_discriminator[field_data_type_name];
 variant_col.getVariantByGlobalDiscriminator(discr).insert(x);
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(discr));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(discr).size() - 1);
 }
 else
 {
 /// We reached maximum number of variants and couldn't add new variant.
 /// In this case we add the value of this new variant into special shared variant.
 /// We store values in shared variant in binary form with binary encoded type.
 auto & shared_variant = getSharedVariant();
 auto & chars = shared_variant.getChars();
 {
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());
 }
 shared_variant.getOffsets().push_back(chars.size());
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(shared_variant_discr));
 variant_col.getOffsets().push_back(shared_variant.size() - 1);
 }
}

Three outcomes, in order:

The value fits an existing alternative — use it, no extension.
The value has a new type and there is room — extend the variant list and add a real alternative.
The value has a new type and the list is full — encode the type and the value into the shared variant’s character buffer.

Case 3 is the interesting one. The shared variant is a ColumnString, so writing into it means writing into its chars buffer directly. The type is encoded first, then the value:

 cpp
 
 
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());

and reading it back is the mirror image:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:323
void ColumnDynamic::get(size_t n, Field & res) const
{
 /// Object, Dynamic and Variant values nest into each other to a depth that comes from the data, not from the declared type.
 checkStackSize();

 const auto & variant_col = getVariantColumn();
 /// Check if value is not in shared variant.
 if (variant_col.globalDiscriminatorAt(n) != getSharedVariantDiscriminator())
 {
 variant_col.get(n, res);
 if (!res.isNull())
 res = convertFieldToType(res, *getTypeAt(n));
 return;
 }

 /// We should deserialize value from shared variant.
 const auto & shared_variant = getSharedVariant();
 auto value_data = shared_variant.getDataAt(variant_col.offsetAt(n));
 ReadBufferFromMemory buf(value_data);
 auto type = decodeDataType(buf);
 type->getDefaultSerialization()->deserializeBinary(res, buf, getFormatSettings());
}

Note checkStackSize(): because Dynamic can nest inside Dynamic inside Dynamic, the depth is a function of the data, and a malicious input could otherwise blow the stack. That is a real concern for a type whose shape is discovered at runtime, and it shows up in the code as an explicit guard.

Adding a variant

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:145
bool ColumnDynamic::addNewVariant(const DataTypePtr & new_variant, const String & new_variant_name)
{
 /// Check if we already have such variant.
 if (variant_info.variant_name_to_discriminator.contains(new_variant_name))
 return true;

 /// Check if we reached maximum number of variants.
 if (!canAddNewVariant())
 {
 /// Dynamic column should always have shared variant.
 if (!variant_info.variant_name_to_discriminator.contains(getSharedVariantTypeName()))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Maximum number of variants reached, but no shared variant exists");

 return false;
 }

 const DataTypes & current_variants = assert_cast<const DataTypeVariant &>(*variant_info.variant_type).getVariants();
 DataTypes all_variants = current_variants;
 all_variants.push_back(new_variant);
 auto new_variant_type = std::make_shared<DataTypeVariant>(all_variants);
 updateVariantInfoAndExpandVariantColumn(new_variant_type);
 return true;
}

The new type is appended to the type, and then the column is expanded to match. updateVariantInfoAndExpandVariantColumn is where ColumnVariant::extend gets called with the old-to-new global discriminator mapping.

Seeing it in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `payload` Dynamic
)
ENGINE = MergeTree
ORDER BY ts;

INSERT INTO events VALUES
 ('2026-01-01 00:00:00', 42),
 ('2026-01-01 00:00:01', 'hello'),
 ('2026-01-01 00:00:02', [1, 2, 3]),
 ('2026-01-01 00:00:03', NULL);

-- The type of each value is remembered.
SELECT payload, dynamicType(payload) FROM events;

The dynamicType function is the user-visible face of the discriminator: it reads the tag and maps it to the type name.

Trade-offs

What the design buys you.

Each alternative is stored in its native layout, so a String alternative compresses like a string column, an UInt64 alternative compresses like a UInt64 column, and the whole thing benefits from per-column codecs. A Dynamic column that only ever saw strings is physically almost identical to a LowCardinality(String) or a plain String column — one byte of tag per row plus the payloads.

The global/local split means adding a type to a Dynamic column is O(1) in the number of rows. Nothing is rewritten; only two small mapping arrays change length.

What it costs you.

Nine bytes per row of pure overhead (one discriminator, one UInt64 offset), versus zero for a plain column. For a Variant with two alternatives and short values that overhead can exceed the payload.

The offsets are the expensive part, and they are the price of the dense sub-column layout. A “one big union buffer with a tag” design would pay one byte per row but would force every alternative into a common representation, losing per-type compression and per-type codecs.

The escape hatch is a real cost. Once a Dynamic column overflows into the shared variant, those values become opaque byte strings with an embedded type name. Reading one back requires decoding the type and then running the right deserializer — a dynamic dispatch on data, with a checkStackSize() guard because the nesting depth is attacker-controlled. And the shared variant is a String column, so it compresses like text even when it holds binary payloads.

The complexity is concentrated in one place. The mapping from global to local discriminator is the only genuinely subtle piece, and it exists purely so that Variant(T1, T2) and Variant(T2, T1) can be the same type while keeping the on-disk tags stable across extensions. Everything else — the offsets, the compact serialization, the NULL discriminator — is a fairly direct consequence of “one sub-column per alternative, one tag per row”.

How ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every column-oriented database eventually meets the same data: a column of country codes, of status strings, of user agents, of enum-like labels. There are a hundred thousand rows and twelve distinct values, repeated over and over. Storing the value in every row is absurd; storing the position of the value in every row is not.

ClickHouse’s answer is a column type called LowCardinality(T), and the implementation is one of the more interesting pieces of the whole storage engine, because it has to solve three separate problems at once:

how to represent the dictionary and the index in memory,
how to make the same dictionary serve many different blocks, granules and parts, so that the dictionary is stored once and not once per block,
how to serialize that arrangement on disk so that a reader can rebuild it, including the case where the dictionary had to change halfway through.

This article walks through all three, quoting the actual source from ClickHouse 26.9 (04eac2ea3).

The shape of the thing

The header comment of ColumnLowCardinality states the design in four lines:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:17
/**
 * How data is stored (in a nutshell):
 * we have a dictionary @e reverse_index in ColumnUnique that holds pairs (DataType, UIntXX) and a column
 * with UIntXX holding actual data indices.
 * To obtain the value's index, call #getOrFindIndex.
 * To operate on the data (so called indices column), call #getIndexes.
 *
 * @note The indices column always contains the default value (empty std::string_view) with the first index.
 */

The class holds exactly two members:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:398
 Dictionary dictionary;
 ColumnIndex idx;

Dictionary is a thin wrapper around a ColumnUnique plus a shared flag:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:366
private:
 class Dictionary
 {
 public:
 Dictionary(const Dictionary & other) = default;
 explicit Dictionary(MutableColumnPtr && column_unique, bool is_shared);
 explicit Dictionary(ColumnPtr column_unique, bool is_shared);

 const WrappedPtr & getColumnUniquePtr() const { return column_unique; }
 WrappedPtr & getColumnUniquePtr() { return column_unique; }

 const IColumnUnique & getColumnUnique() const { return static_cast<const IColumnUnique &>(*column_unique); }
 IColumnUnique & getColumnUnique() { return static_cast<IColumnUnique &>(*column_unique); }

 /// Dictionary may be shared for several mutable columns.
 /// Immutable columns may have the same column unique, which isn't necessarily shared dictionary.
 void setShared(const ColumnPtr & column_unique_);
 bool isShared() const { return shared; }

 /// Create new dictionary with only keys that are mentioned in indexes.
 void compact(MutableColumnPtr & indexes);
 /// Create nullable dictionary with only keys that are mentioned in indexes.
 void compactToNullable(MutableColumnPtr & indexes);

 static MutableColumnPtr compact(const IColumnUnique & column_unique, MutableColumnPtr & indexes);
 static MutableColumnPtr compactToNullable(const IColumnUnique & column_unique, MutableColumnPtr & indexes);

 private:
 WrappedPtr column_unique;
 bool shared = false;
 };

So a LowCardinality column is:

text

 +---------------------------+
 row 0 -----> | dictionary[ idx[0] ] | idx is a ColumnVector<UInt8|UInt16|UInt32|UInt64>
 row 1 -----> | dictionary[ idx[1] ] | dictionary is a ColumnUnique<ColumnString> (etc.)
 row 2 -----> | dictionary[ idx[2] ] |
 ... +---------------------------+

The values are in the dictionary; the rows are indices. Getting row n is two loads: getIndexes().getUInt(n) then a lookup in the dictionary. That second load is a random access into a (hopefully tiny) array — and the whole point is that the dictionary array is tiny enough to stay in L1/L2 while the index array streams.

The index column grows its own type

The index column is not fixed-width. It starts as UInt8 and is widened to UInt16, UInt32, UInt64 as the dictionary outgrows the range. The widening logic lives in ColumnIndex:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:135
void ColumnIndex::expandType()
{
 auto expand = [&](auto type)
 {
 using CurIndexType = decltype(type);
 constexpr auto next_size = NumberTraits::nextSize(sizeof(CurIndexType));
 if (next_size == sizeof(CurIndexType))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't expand indexes type for ColumnLowCardinality from type: {}",
 demangle(typeid(CurIndexType).name()));

 using NewIndexType = typename NumberTraits::Construct<false, false, next_size>::Type;
 convertIndexes<NewIndexType>();
 };

 callForType(std::move(expand), size_of_type);
}

and the bound that decides when to widen:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:152
size_t ColumnIndex::getMaxIndexForCurrentType() const
{
 chassert(size_of_type == 1 || size_of_type == 2 || size_of_type == 4 || size_of_type == 8);
 /// Shifting by 64 is undefined, and this is called per inserted value, so no `callForType` here.
 if (size_of_type == sizeof(UInt64))
 return std::numeric_limits<UInt64>::max();
 return (1ULL << (8 * size_of_type)) - 1;
}

The per-value insert path is where the widening happens, and note the comment about avoiding a typeid_cast on the hot path:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:198
void ColumnIndex::insertIndex(size_t index)
{
 while (index > getMaxIndexForCurrentType())
 expandType();

 auto insert = [&]<typename CurIndexType>(CurIndexType /*type_value*/)
 {
 /// `size_of_type` tracks the type of `indexes` and `expandType` keeps the two in sync, so
 /// this is `static_cast` in release. `getIndexesData` would run a `typeid_cast` instead, and
 /// this is the per-value path.
 assert_cast<ColumnVector<CurIndexType> *>(indexes.get())->getData().push_back(static_cast<CurIndexType>(index));
 };

 callForType(std::move(insert), size_of_type);

 /// Same invariant, so re-deriving it from `indexes` is a debug-only check here.
 chassert(size_of_type == getSizeOfIndexType(*indexes, size_of_type));
}

callForType is a hand-rolled dispatch over the four possible widths — no virtual call, no switch on a runtime type id, just four instantiations of a lambda. This is the pattern used throughout ClickHouse’s hot column code: the virtual IColumn interface is for the cold, structural paths; the per-row paths are devirtualized by dispatching on the concrete type once and then running a monomorphic loop.

Reading an index back has the same shape, and is marked ALWAYS_INLINE because it is called from the tightest loops in the engine:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:326
 ALWAYS_INLINE size_t getIndexAt(size_t row) const
 {
 const IColumn * indexes = &getIndexes();

 switch (idx.getSizeOfIndexType())
 {
 case sizeof(UInt8): return assert_cast<const ColumnUInt8 *>(indexes)->getElement(row);
 case sizeof(UInt16): return assert_cast<const ColumnUInt16 *>(indexes)->getElement(row);
 case sizeof(UInt32): return assert_cast<const ColumnUInt32 *>(indexes)->getElement(row);
 case sizeof(UInt64): return assert_cast<const ColumnUInt64 *>(indexes)->getElement(row);
 default: throwUnexpectedLowCardinalityIndexType(idx.getSizeOfIndexType());
 }
 }

The dictionary: ColumnUnique and its reverse index

ColumnUnique is the dictionary. It is not a column type you can select; the comment says so:

 cpp
 
 
// src/Columns/ColumnUnique.h:36
/** Stores another column with unique values
 * and also an index that allows to find position by value.
 *
 * This column is not used on it's own but only as implementation detail of ColumnLowCardinality.
 */
template <typename ColumnType>
class ColumnUnique final : public COWHelper<IColumnUnique, ColumnUnique<ColumnType>>

Its members are the values plus the lookup structure:

 cpp
 
 
// src/Columns/ColumnUnique.h:203
private:
 IColumn::WrappedPtr column_holder;
 bool is_nullable;
 size_t size_of_value_if_fixed = 0;

 ReverseIndex<UInt64, ColumnType> reverse_index;

 /// For DataTypeNullable, stores null map.
 IColumn::WrappedPtr nested_null_mask;
 IColumn::WrappedPtr nested_column_nullable;

Two reserved slots

The first row of the dictionary is special: it is the default value of the type. If the dictionary is nullable, row 0 is the NULL and row 1 is the default value of the inner type:

 cpp
 
 
// src/Columns/ColumnUnique.h:74
 size_t getDefaultValueIndex() const override { return 0; }
 size_t getNullValueIndex() const override;
 size_t getNestedTypeDefaultValueIndex() const override { return is_nullable ? 1 : 0; }
 bool canContainNulls() const override { return is_nullable; }

and

 cpp
 
 
// src/Columns/ColumnUnique.h:370
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::getNullValueIndex() const
{
 if (!is_nullable)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ColumnUnique can't contain null values.");

 return 0;
}

The count of reserved slots is therefore one or two:

 cpp
 
 
// src/Columns/ColumnUnique.h:238
 static size_t numSpecialValues(bool is_nullable) { return is_nullable ? 2 : 1; }
 size_t numSpecialValues() const { return numSpecialValues(is_nullable); }

Those reserved slots are not in the reverse index, which is why the insert path has to special-case them by hand:

 cpp
 
 
// src/Columns/ColumnUnique.h:513
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsertData(const char * pos, size_t length)
{
 /// The reserved prefix slots are not in the reverse index, so the default value is matched here.
 /// Comparing the first byte first keeps the `memcmp` call out of this path for the values that
 /// are not the default, which is all of them in a typical dictionary.
 const size_t default_index = getNestedTypeDefaultValueIndex();
 const std::string_view default_value = getRawColumnPtr()->getDataAt(default_index);
 if (default_value.size() == length && (length == 0 || (default_value[0] == pos[0] && default_value == std::string_view(pos, length))))
 return default_index;

 auto insertion_point = reverse_index.insert({pos, length});

 updateNullMask();

 return insertion_point;
}

The comment is a nice piece of micro-optimisation reasoning: comparing the first byte before the whole value keeps the expensive memcmp off the path for the (overwhelmingly common) case where the value is not the default.

The reverse index stores row numbers, not values

ReverseIndex is the interesting part. A naive dictionary would be a HashMap<value, index>, i.e. the values would be stored twice: once in the column, once as hash-table keys. ReverseIndex inverts that: the hash table stores the row number in the dictionary column, and comparisons dereference back into the column:

 cpp
 
 
// src/Columns/ReverseIndex.h:96
 /// Special case when we want to compare with something not in index_column.
 /// When we compare something inside column default keyEquals checks only that row numbers are equal.
 bool keyEquals(std::string_view object, size_t hash_ [[maybe_unused]], const State & state) const
 {
 auto index = key;
 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return hash_ == (*state.saved_hash_column)[index] && object == state.index_column->getDataAt(index);
 else
 return object == state.index_column->getDataAt(index);
 }

The hash table therefore holds 8-byte integers, and the dictionary column holds the only copy of the data. For a LowCardinality(String) with 1000 distinct values that is the difference between a hash table with 1000 string keys (plus their hashes, plus their lengths) and a hash table with 1000 UInt64s.

For string columns the hash is expensive, so it is precomputed once and kept in a parallel ColumnUInt64:

 cpp
 
 
// src/Columns/ReverseIndex.h:43
template <typename ColumnType>
struct ReverseIndexHashTableState<ColumnType, /* with_saved_hash = */ true, /* has_base_index */ false>
{
 constexpr static bool with_saved_hash = true;
 constexpr static bool has_base_index = false;

 ColumnType * index_column;
 typename ColumnVector<UInt64>::Container * saved_hash_column;
};

and the hash function for a stored row reads the precomputed hash rather than re-hashing the string:

 cpp
 
 
// src/Columns/ReverseIndex.h:108
 size_t getHash(const Hash & hash) const
 {
 auto index = key;

 /// Hack. HashTable is Hash itself.
 const auto & state = static_cast<const State &>(static_cast<const HashTable &>(hash));

 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return (*state.saved_hash_column)[index];
 else
 {
 using ValueType = typename ColumnType::ValueType;
 ValueType value = unalignedLoad<ValueType>(state.index_column->getDataAt(index).data());
 return DefaultHash<ValueType>()(value);
 }
 }

The index is built lazily, and the hash table is filled in one pass

The reverse index is not built at construction time; it is built on the first insert, and the build is a single pass over the dictionary:

 cpp
 
 
// src/Columns/ReverseIndex.h:435
template <typename IndexType, typename ColumnType>
void ReverseIndex<IndexType, ColumnType>::buildIndex()
{
 if (index)
 return;

 if (!column)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ReverseIndex can't build index because index column wasn't set.");

 auto size = column->size();
 index = std::make_unique<IndexMapType>(size);

 if constexpr (use_saved_hash)
 saved_hash = calcHashes();

 auto & state = index->getState();
 state.index_column = column;
 state.base_index = base_index;
 if constexpr (use_saved_hash)
 state.saved_hash_column = &saved_hash->getData();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 for (auto row : collections::range(num_prefix_rows_to_skip, size))
 {
 UInt64 hash = 0;
 if constexpr (use_saved_hash)
 hash = saved_hash->getElement(row);
 else
 hash = getHash(column->getDataAt(row));

 index->reverseIndexEmplace(row + base_index, iterator, inserted, hash, column->getDataAt(row));

 if (!inserted)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Duplicating keys found in ReverseIndex.");
 }
}

Note num_prefix_rows_to_skip: the reserved NULL/default slots are skipped, because they are matched by hand in uniqueInsertData.

The insert path is the mirror image, and it is written to be branch-light for the two different column kinds:

 cpp
 
 
// src/Columns/ReverseIndex.h:490
template <typename IndexType, typename ColumnType>
UInt64 ReverseIndex<IndexType, ColumnType>::insert(std::string_view data)
{
 if (!index)
 buildIndex();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 auto hash = getHash(data);
 UInt64 num_rows = size();

 if constexpr (use_saved_hash)
 {
 auto & column_data = saved_hash->getData();
 if (column_data.size() <= num_rows)
 column_data.resize(num_rows + 1);
 column_data[num_rows] = hash;
 }
 else
 column->insertData(data.data(), data.size());

 index->reverseIndexEmplace(num_rows + base_index, iterator, inserted, hash, data);

 if constexpr (use_saved_hash)
 {
 if (inserted)
 column->insertData(data.data(), data.size());
 }
 else
 {
 if (!inserted)
 column->popBack(1);
 }

 return iterator->getValue();
}

For strings the hash is appended optimistically and the value is appended only if the key was new. For fixed-size values the value is appended optimistically and popped back if the key turned out to be a duplicate. Each kind pushes and pops the cheaper thing.

Float NaNs are canonicalized

A LowCardinality(Float64) dictionary must not contain two NaNs, because NaN == NaN is false but all NaNs must compare equal in the dictionary semantics. So NaN is normalized at insert time:

 cpp
 
 
// src/Columns/ColumnUnique.h:389
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsert(const Field & x)
{
 if (x.isNull())
 return getNullValueIndex();

 // NaN can contain different sign or mantissa bits, but we need to consider all NaNs equal.
 if constexpr (is_float_vector_v<ColumnType>)
 {
 if (isNaN(x.safeGet<typename ColumnType::ValueType>()))
 {
 auto nan = NaNOrZero<typename ColumnType::ValueType>();
 return uniqueInsertData(reinterpret_cast<char *>(&nan), sizeof(nan));
 }
 }

Here is where the design pays off. A LowCardinality column can point at a dictionary owned by another column. The shared flag on Dictionary records this, and byteSize() deliberately does not count a shared dictionary:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:200
 /// Don't count the dictionary size as it can be shared between different blocks.
 size_t byteSize() const override { return idx.getIndexes()->byteSize() + (isSharedDictionary() ? 0 : getDictionary().byteSize()); }

 size_t byteSizeAt(size_t n) const override { return getDictionary().byteSizeAt(getIndexes().getUInt(n)); }
 size_t allocatedBytes() const override { return idx.getIndexes()->allocatedBytes() + getDictionary().allocatedBytes(); }

Note the asymmetry: byteSize() (the logical size, used for statistics and for system.parts_columns) excludes the shared dictionary, but allocatedBytes() (the physical size, used for memory accounting) includes it. That distinction is what makes the accounting correct when a part has a hundred blocks all pointing at one dictionary.

Sharing also propagates through every operation that produces a new column from an old one — cut, filter, permute, index, replicate all hand the same dictionary pointer to the result:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:130
 ColumnPtr filter(const Filter & filt, ssize_t result_size_hint) const override
 {
 return ColumnLowCardinality::create(
 dictionary.getColumnUniquePtr(), getIndexes().filter(filt, result_size_hint), isSharedDictionary());
 }

But a shared dictionary is immutable

The moment you want to mutate a column whose dictionary is shared with somebody else, the dictionary has to be un-shared. This happens in compactIfSharedDictionary(), which is called at the top of every mutating insert:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:146
void ColumnLowCardinality::insert(const Field & x)
{
 compactIfSharedDictionary();
 idx.insertIndex(getDictionary().uniqueInsert(x));
}

bool ColumnLowCardinality::tryInsert(const Field & x)
{
 compactIfSharedDictionary();

 size_t index = 0;
 if (!dictionary.getColumnUnique().tryUniqueInsert(x, index))
 return false;

 idx.insertIndex(index);
 return true;
}

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:894
void ColumnLowCardinality::compactIfSharedDictionary()
{
 if (dictionary.isShared())
 compactInplace();
}

Compaction rebuilds a private dictionary containing only the keys that are actually referenced by this column’s indexes, and remaps the indexes accordingly:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:1002
MutableColumnPtr ColumnLowCardinality::Dictionary::compact(const IColumnUnique & unique, MutableColumnPtr & indexes)
{
 auto new_column_unique = unique.cloneEmpty();
 auto & new_unique = static_cast<IColumnUnique &>(*new_column_unique);

 auto unique_indexes = mapUniqueIndex(*indexes);
 auto sub_keys = unique.getNestedColumn()->index(*unique_indexes, 0);
 auto new_indexes = new_unique.uniqueInsertRangeFrom(*sub_keys, 0, sub_keys->size());

 indexes = IColumn::mutate(new_indexes->index(*indexes, 0));
 return new_column_unique;
}

mapUniqueIndex is the piece that does the renumbering. It is a counting-sort-like trick that works in O(n) with a flat array, and it has an overflow path for when the dictionary is shared and the indexes do not fit the “max index < number of rows” assumption:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:79
template <typename T>
MutableColumnPtr mapUniqueIndexImpl(PaddedPODArray<T> & index)
{
 if (index.empty())
 return ColumnVector<T>::create();

 auto size = index.size();

 T max_val = index[0];
 for (size_t i = 1; i < size; ++i)
 max_val = std::max(max_val, index[i]);

 /// May happen when dictionary is shared.
 if (max_val > size)
 return mapUniqueIndexImplRef(index);

 auto map_size = static_cast<UInt64>(max_val) + 1;
 PaddedPODArray<T> map(map_size, 0);
 T zero_pos_value = index[0];
 index[0] = 0;
 T cur_pos = 0;
 for (size_t i = 1; i < size; ++i)
 {
 T val = index[i];
 if (val != zero_pos_value && map[val] == 0)
 {
 ++cur_pos;
 map[val] = cur_pos;
 }

 index[i] = map[val];
 }

The “overflow” case (max_val > size) means the column references dictionary entries that belong to a bigger dictionary — exactly what happens when the dictionary is shared. Then a hash map is used instead of a flat array:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:53
template <typename T>
MutableColumnPtr mapUniqueIndexImplRef(PaddedPODArray<T> & index)
{
 PaddedPODArray<T> copy(index.cbegin(), index.cend());

 HashMap<T, T> hash_map;
 for (auto val : index)
 hash_map.insert({val, static_cast<T>(hash_map.size())});

 auto res_col = ColumnVector<T>::create();
 auto & data = res_col->getData();

 data.resize(hash_map.size());
 for (const auto & val : hash_map)
 data[val.getMapped()] = val.getKey();

 for (auto & ind : index)
 ind = hash_map[ind];

 for (size_t i = 0; i < index.size(); ++i)
 if (data[index[i]] != copy[i])
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected {}, but got {}", toString(data[index[i]]), toString(copy[i]));

 return res_col;
}

That last loop is a self-check: after the renumbering, the new index must still resolve to the same value. Cheap insurance against an off-by-one in a data structure that is otherwise very hard to test.

Inserting from a column with the same dictionary

Once dictionaries are shared, a very common operation becomes trivial: copying a row from another LowCardinality column that happens to use the same dictionary is just a copy of an index.

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:180
 size_t position = low_cardinality_src->getIndexes().getUInt(n);

 if (&low_cardinality_src->getDictionary() == &getDictionary())
 {
 /// Dictionary is shared with src column. Insert only index.
 idx.insertIndex(position);
 }
 else
 {
 compactIfSharedDictionary();
 const auto & nested = *low_cardinality_src->getDictionary().getNestedColumn();
 idx.insertIndex(getDictionary().uniqueInsertFrom(nested, position));
 }

The test is a pointer comparison of the dictionary addresses. If they are the same object, the index is valid as-is; otherwise the value has to be re-inserted into this column’s dictionary and the resulting index used.

On disk: one dictionary per part, versioned per granule

Now the hard part. In a MergeTree part, a LowCardinality column is written as two streams: DictionaryKeys and DictionaryIndexes.

The keys stream holds the dictionary; the indexes stream holds, per granule, a header and then the index data. The dictionary is not rewritten per granule — it accumulates until it hits a size limit, at which point it is flushed and a new one starts.

The version byte

The keys stream starts with a version:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:102
struct KeysSerializationVersion
{
 enum Value
 {
 /// Version is written at the start of <name.dict.bin>.
 /// Dictionary is written as number N and N keys after them.
 /// Dictionary can be shared for continuous range of granules, so some marks may point to the same position.
 /// Shared dictionary is stored in state and is read once.
 SharedDictionariesWithAdditionalKeys = 1,
 };

 Value value;

 static void checkVersion(UInt64 version)
 {
 if (version != SharedDictionariesWithAdditionalKeys)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Invalid version for SerializationLowCardinality key column.");
 }

 explicit KeysSerializationVersion(UInt64 version) : value(static_cast<Value>(version)) { checkVersion(version); }
};

The single legal version is called SharedDictionariesWithAdditionalKeys, and the comment tells you the whole story in three lines: the dictionary is written as a count followed by that many keys, it can be shared across a continuous range of granules, and the shared dictionary is kept in the serialization state and read once.

The per-granule header

Every granule’s index data begins with a header that encodes the index width plus three flags:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:124
/// Version is stored at the start of each granule. It's used to store indexes type and flags.
struct IndexesSerializationType
{
 using SerializationType = UInt64;
 /// Need to read dictionary if it wasn't.
 static constexpr SerializationType NeedGlobalDictionaryBit = 1u << 8u;
 /// Need to read additional keys. Additional keys are stored before indexes as value N and N keys after them.
 static constexpr SerializationType HasAdditionalKeysBit = 1u << 9u;
 /// Need to update dictionary. It means that previous granule has different dictionary.
 static constexpr SerializationType NeedUpdateDictionary = 1u << 10u;

 enum Type
 {
 TUInt8 = 0,
 TUInt16,
 TUInt32,
 TUInt64,
 };

 Type type;
 bool has_additional_keys;
 bool need_global_dictionary;
 bool need_update_dictionary;

 static constexpr SerializationType resetFlags(SerializationType type)
 {
 return type & (~(HasAdditionalKeysBit | NeedGlobalDictionaryBit | NeedUpdateDictionary));
 }

The three flags are the entire state machine:

NeedGlobalDictionaryBit — “there is a dictionary in the keys stream that I have not read yet”.
HasAdditionalKeysBit — “this granule carries keys that did not fit in the shared dictionary; they are stored inline, before the index data”.
NeedUpdateDictionary — “the dictionary changed since the previous granule”.

Serialization and deserialization are symmetric and trivial:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:162
 void serialize(WriteBuffer & buffer) const
 {
 SerializationType val = type;
 if (has_additional_keys)
 val |= HasAdditionalKeysBit;
 if (need_global_dictionary)
 val |= NeedGlobalDictionaryBit;
 if (need_update_dictionary)
 val |= NeedUpdateDictionary;
 writeBinaryLittleEndian(val, buffer);
 }

 void deserialize(ReadBuffer & buffer, const ISerialization::DeserializeBinaryBulkSettings & settings)
 {
 SerializationType val = 0;
 readBinaryLittleEndian(val, buffer);

 checkType(val);
 has_additional_keys = (val & HasAdditionalKeysBit) != 0;
 need_global_dictionary = (val & NeedGlobalDictionaryBit) != 0;
 need_update_dictionary = (val & NeedUpdateDictionary) != 0;
 type = static_cast<Type>(resetFlags(val));

 if (settings.native_format)
 {
 if (need_global_dictionary)
 throw Exception(ErrorCodes::INCORRECT_DATA,
 "LowCardinality indexes serialization type for Native format "
 "cannot use global dictionary");
 }
 }

The low four bits carry the index width; bits 8, 9, 10 are the flags. Bits 4–7 are unused — a hole left for future flags.

Writing: the dictionary accumulates, then overflows

The writer keeps the accumulating dictionary in the serialization state:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:232
struct SerializeStateLowCardinality : public ISerialization::SerializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 MutableColumnUniquePtr shared_dictionary;

 explicit SerializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}
};

The write path is where the interesting decisions live:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:561
 const ColumnLowCardinality & low_cardinality_column = typeid_cast<const ColumnLowCardinality &>(column);

 auto * low_cardinality_state = checkAndGetState<SerializeStateLowCardinality>(state);
 auto & global_dictionary = low_cardinality_state->shared_dictionary;
 KeysSerializationVersion::checkVersion(low_cardinality_state->key_version.value);

 size_t max_limit = column.size() - offset;
 limit = limit ? std::min(limit, max_limit) : max_limit;

 /// Do not write anything for empty column. (May happen while writing empty arrays.)
 if (limit == 0)
 return;

 bool need_update_dictionary = global_dictionary == nullptr;
 if (need_update_dictionary)
 global_dictionary = DataTypeLowCardinality::createColumnUnique(*dictionary_type);

 auto sub_column = low_cardinality_column.cutAndCompact(offset, limit);
 ColumnPtr positions = sub_column->getIndexesPtr();
 ColumnPtr keys = sub_column->getDictionary().getNestedColumn();

 if (settings.low_cardinality_max_dictionary_size)
 {
 /// Insert used_keys into global dictionary and update sub_index.
 auto indexes_with_overflow = global_dictionary->uniqueInsertRangeWithOverflow(*keys, 0, keys->size(),
 settings.low_cardinality_max_dictionary_size);

 if (global_dictionary->size() > settings.low_cardinality_max_dictionary_size)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Got dictionary with size {} but max dictionary size is {}",
 global_dictionary->size(), settings.low_cardinality_max_dictionary_size);

 positions = indexes_with_overflow.indexes->index(*positions, 0);
 keys = std::move(indexes_with_overflow.overflowed_keys);

 if (global_dictionary->size() < settings.low_cardinality_max_dictionary_size && !keys->empty())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Has additional keys, but dict size is {} which is less "
 "then max dictionary size ({})", global_dictionary->size(),
 settings.low_cardinality_max_dictionary_size);
 }

cutAndCompact is the first step: it takes the slice of the column being written, and produces a private dictionary containing only the keys that slice actually uses. That is what keeps the accumulated dictionary small — without it, every granule would re-insert the whole dictionary.

Then the keys are merged into the global dictionary, and anything that does not fit within low_cardinality_max_dictionary_size becomes “overflowed keys” — keys carried inline by that granule.

The flush happens when the accumulated dictionary reaches the limit:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:602
 bool need_additional_keys = !keys->empty();
 bool need_dictionary = settings.low_cardinality_max_dictionary_size != 0;
 bool need_write_dictionary = !settings.low_cardinality_use_single_dictionary_for_part
 && global_dictionary->size() >= settings.low_cardinality_max_dictionary_size;

 IndexesSerializationType index_version(*positions, need_additional_keys, need_dictionary, need_update_dictionary);
 index_version.serialize(*indexes_stream);

 if (need_write_dictionary)
 {
 const auto & nested_column = global_dictionary->getNestedNotNullableColumn();
 UInt64 num_keys = nested_column->size();
 writeBinaryLittleEndian(num_keys, *keys_stream);
 dict_inner_serialization->serializeBinaryBulk(*nested_column, *keys_stream, 0, num_keys);
 low_cardinality_state->shared_dictionary = nullptr;
 }

 if (need_additional_keys)
 {
 UInt64 num_keys = keys->size();
 writeBinaryLittleEndian(num_keys, *indexes_stream);
 dict_inner_serialization->serializeBinaryBulk(*keys, *indexes_stream, 0, num_keys);
 }

 UInt64 num_rows = positions->size();
 writeBinaryLittleEndian(num_rows, *indexes_stream);
 auto index_serialization = index_version.getDataType()->getDefaultSerialization();
 index_serialization->serializeBinaryBulk(*positions, *indexes_stream, 0, num_rows);

The default limit is 8192 keys:

 cpp
 
 
// src/Core/Settings.cpp:5045
 DECLARE(UInt64, low_cardinality_max_dictionary_size, 8192, R"(
Sets a maximum size in rows of a shared global dictionary for the [LowCardinality](/reference/data-types/lowcardinality) data type that can be written to a storage file system. This setting prevents issues with RAM in case of unlimited dictionary growth. All the data that can't be encoded due to maximum dictionary size limitation ClickHouse writes in an ordinary method.

Possible values:

- Any positive integer.
)", 0) \

So: with an 8192-key limit, a part whose column has 100 distinct values writes the dictionary once; a part with 100 000 distinct values writes the dictionary roughly every 8192 keys and starts a new one.

Note that need_update_dictionary is global_dictionary == nullptr — i.e. “this granule starts a new dictionary”. And the dictionary is flushed after the header was written, so the header says “read the dictionary” and the dictionary follows in the keys stream.

Reading: the reader has to reconstruct the same state

The read side keeps the shared dictionary in the deserialize state, along with a flag that survives granule skips:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:240
struct DeserializeStateLowCardinality : public ISerialization::DeserializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 ColumnUniquePtr global_dictionary;

 IndexesSerializationType index_type{};
 ColumnPtr additional_keys;
 ColumnPtr null_map;
 UInt64 num_pending_rows = 0;

 /// If dictionary should be updated.
 /// Can happen is some granules was skipped while reading from MergeTree.
 /// We should store this flag in State because
 /// in case of long block of empty arrays we may not need read dictionary at first reading.
 bool need_update_dictionary = false;

 /// True if this part has a single dictionary for the entire column. The prefix
 /// reader first checks `has_uniform_marks_callback` as a cheap prefilter, then
 /// verifies it by reading the first dictionary and checking that `DictionaryKeys`
 /// reached EOF. When true, non-continuous reads (granule skips) do not invalidate
 /// the cached dictionary, and the dictionary can be shared across all threads via `clone`.
 bool single_dictionary_for_part = false;

 explicit DeserializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}

 ISerialization::DeserializeBinaryBulkStatePtr cloneImpl(ISerialization::DeserializeBinaryBulkStateCloneContext &) const override
 {
 return std::make_shared<DeserializeStateLowCardinality>(*this);
 }
};

The need_update_dictionary comment is the interesting one: when reading is not continuous (i.e. some granules were skipped), the reader cannot assume that the dictionary it holds is the one the next granule refers to, so it must force a re-read. That flag lives in the state, not on the stack, precisely because a skipped granule may be discovered long after the state was created.

The main read loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:752
 if (!settings.continuous_reading)
 {
 low_cardinality_state->num_pending_rows = 0;

 /// Some granules were skipped; the next granule header may signal a dictionary
 /// update. For multi-dict parts this forces a re-read at the next boundary.
 low_cardinality_state->need_update_dictionary = true;
 }

 while (limit)
 {
 if (low_cardinality_state->num_pending_rows == 0)
 {
 if (indexes_stream->eof())
 break;

 auto & index_type = low_cardinality_state->index_type;
 auto & global_dictionary = low_cardinality_state->global_dictionary;

 index_type.deserialize(*indexes_stream, settings);

 bool need_update_dictionary =
 !global_dictionary || index_type.need_update_dictionary || low_cardinality_state->need_update_dictionary;
 if (index_type.need_global_dictionary && need_update_dictionary)
 {
 /// For single-dict parts the in-memory dictionary is valid for the entire part,
 /// so skip re-reading it even if the stream was seeked back to the beginning.
 if (!low_cardinality_state->single_dictionary_for_part || !global_dictionary)
 read_dictionary();
 low_cardinality_state->need_update_dictionary = false;
 }

 if (low_cardinality_state->index_type.has_additional_keys)
 read_additional_keys();
 else
 low_cardinality_state->additional_keys = nullptr;

 readBinaryLittleEndian(low_cardinality_state->num_pending_rows, *indexes_stream);
 }

 size_t num_rows_to_read = std::min<UInt64>(limit, low_cardinality_state->num_pending_rows);
 read_indexes(num_rows_to_read);
 limit -= num_rows_to_read;
 low_cardinality_state->num_pending_rows -= num_rows_to_read;
 }

num_pending_rows is what makes a granule’s worth of rows readable in chunks: the header is read once, then the reader consumes num_pending_rows rows from the index stream, and only when that counter hits zero does it look for the next header.

The single-dictionary-for-part fast path

There is a special case worth highlighting: a part where the column has exactly one dictionary for the whole part. The prefix reader detects it by reading the first dictionary and checking whether the DictionaryKeys stream reached EOF:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:348
 if (might_have_single_dictionary && !stream->eof())
 {
 /// For `Wide` parts that might have a single dictionary, the stream is
 /// now positioned right at the dictionary data (after the version).
 /// Read the first dictionary and check EOF afterwards. If this proves
 /// that there are no more dictionaries, all threads sharing this prefix
 /// state via `DeserializationPrefixesCache` get the dictionary through
 /// a `shared_ptr` clone instead of each reading it independently.

and

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:377
 if (stream->eof())
 {
 new_state->single_dictionary_for_part = true;
 new_state->global_dictionary = std::move(global_dictionary);
 }
 else
 {
 /// The mark heuristic can be fooled by several dictionaries written
 /// inside one mark interval. Reset the `DictionaryKeys` stream back
 /// to the normal post-prefix position and let data deserialization
 /// read dictionary updates from the stream.
 if (!settings.seek_to_start_callback)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot reset LowCardinality dictionary stream without seek_to_start_callback");

 settings.seek_to_start_callback(dictionary_keys_path);

 UInt64 repeated_keys_version = 0;
 readBinaryLittleEndian(repeated_keys_version, *stream);
 if (repeated_keys_version != keys_version)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Inconsistent version while resetting LowCardinality dictionary stream");
 }

This is a heuristic with a verification step. The cheap check (has_uniform_marks_callback) says “all marks point at the same position in the keys stream, so maybe there is only one dictionary”. The expensive check (actually reading the dictionary and seeing EOF) confirms it. If the heuristic lied, the stream is rewound and the normal path takes over. That combination — cheap prefilter, expensive confirmation, safe fallback — is a pattern you will see repeatedly in ClickHouse’s MergeTree reader.

When the fast path holds, cloneImpl copies the state (including the dictionary pointer) so every thread reading the part gets the same dictionary without re-reading it.

The overflow path on the read side

When a granule carries inline “additional keys”, the reader has to merge two dictionaries: the global one and the granule-local one. mapIndexWithAdditionalKeys does that with two flat maps — one for the global dictionary’s keys and one for the overflow keys:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:515
 /// Update column and return map with old indexes.
 /// Let N is the number of distinct values which are less than max_size;
 /// old_column - column before function call;
 /// new_column - column after function call:
 /// * if old_column[i] < max_size, than
 /// dictionary_map[new_column[i]] = old_column[i]
 /// * else
 /// additional_keys_map[new_column[i]] = old_column[i] - dict_size + N
 IndexMapsWithAdditionalKeys mapIndexWithAdditionalKeys(IColumn & column, size_t dict_size)

and then the keys are stitched back together:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:726
 else
 {
 auto maps = mapIndexWithAdditionalKeys(*indexes_column, global_dictionary->size());

 auto used_keys = IColumn::mutate(global_dictionary->getNestedColumn()->index(*maps.dictionary_map, 0));

 if (!maps.additional_keys_map->empty())
 {
 if (additional_keys == nullptr)
 throw Exception(ErrorCodes::INCORRECT_DATA, "No additional keys found.");

 auto used_add_keys = additional_keys->index(*maps.additional_keys_map, 0);

 if (dictionary_type->isNullable())
 {
 ColumnPtr null_map = ColumnUInt8::create(used_add_keys->size(), static_cast<UInt8>(0));
 used_add_keys = ColumnNullable::create(used_add_keys, null_map);
 }

 used_keys->insertRangeFrom(*used_add_keys, 0, used_add_keys->size());
 }

 low_cardinality_column.insertRangeFromDictionaryEncodedColumn(*used_keys, *indexes_column);
 }

The final call re-inserts the merged keys into the destination column’s own dictionary and remaps the indexes one last time.

The whole thing in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `user_id` UInt64,
 `kind` LowCardinality(String), -- 12 distinct values in 100M rows
 `country` LowCardinality(FixedString(2))
)
ENGINE = MergeTree
ORDER BY (ts, user_id);

 sql
 
 
-- The dictionary is shared across every block of the part, so the per-column
-- cost is one byte per row (UInt8 index) plus the dictionary once.
SELECT kind, count() FROM events GROUP BY kind;

The type-level guarantee is that LowCardinality is only worth it while the dictionary stays small; the documentation says so plainly:

text

// src/DataTypes/DataTypeLowCardinality.cpp:198
The efficiency of using `LowCardinality` data type depends on data diversity. If a dictionary contains less than 10,000 distinct values, then ClickHouse mostly shows higher efficiency of data reading and storing. If a dictionary contains more than 100,000 distinct values, then ClickHouse can perform worse in comparison with using ordinary data types.

Trade-offs

What the hack buys you.

A LowCardinality(String) column with 12 distinct values costs one byte per row plus a few hundred bytes of dictionary, instead of ~10 bytes per row of strings. It also means the scan is a byte scan, not a string scan: filters, GROUP BY, and ORDER BY all operate on UInt8 indexes, and the dictionary is touched only for the rows that survive the filter.

What it costs you.

Every value access is two dependent loads: read the index, then read the dictionary entry. That is a pointer chase that a plain String column does not have — although for a plain string column you also pay a pointer chase into the character arena, so the two are more comparable than they look.

The dictionary is a global per part, and the index stream has to carry a per-granule header with three flags and a row count, plus the possibility of inline overflow keys. A plain column has none of that. For a column with 100 000+ distinct values you have bought complexity and lost compression.

The sharing is also asymmetric: byteSize() excludes a shared dictionary while allocatedBytes() includes it. That is correct but subtle, and it means the two numbers in system.parts_columns can disagree in a way that looks like a bug until you know why.

The design decision that makes it work. The dictionary is not stored per block, and not per granule; it is stored per part, capped at 8192 keys, and flushed when it overflows. Everything else — the three header bits, the need_update_dictionary flag, the single_dictionary_for_part heuristic — exists to make that one decision survive partial reads, granule skips and multi-threaded readers.

How ClickHouse Splits GROUP BY into 256 Buckets for Parallel Merges

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A GROUP BY in ClickHouse is usually computed by several threads at once, each with its own hash table, and the tables are then merged. Merging N hash tables of M entries each is not something you want to do on one thread: it is a serial phase in an otherwise parallel query, and it is the phase where the memory high-water mark is reached.

The way out is to make the merge divisible. If the keys are partitioned into a fixed number of buckets by a deterministic function of the key, then two tables can be merged bucket by bucket, and different buckets can be merged by different threads, with no coordination at all: bucket 17 of table A can only collide with bucket 17 of table B.

That is what TwoLevelHashTable in src/Common/HashTable/TwoLevelHashTable.h does, and it is what the group_by_two_level_threshold setting controls.

The structure

The two-level table is literally an array of ordinary hash tables:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:9
/** Two-level hash table.
 * Represents 256 (or 1ULL << BITS_FOR_BUCKET) small hash tables (buckets of the first level).
 * To determine which one to use, one of the bytes of the hash function is taken.
 *
 * Usually works a little slower than a simple hash table.
 * However, it has advantages in some cases:
 * - if you need to merge two hash tables together, then you can easily parallelize it by buckets;
 * - delay during resizes is amortized, since the small hash tables will be resized separately;
 * - in theory, resizes are cache-local in a larger range of sizes.
 *
 * With `BITS_FOR_BUCKET = 0` there is a single bucket: routing folds to a constant.
 * Lookups and inserts compile down to what the single-level table does.
 *
 * Buckets share no state.
 * Threads may fill different buckets at the same time when every bucket is written under its own lock.
 */

The bucket index comes out of the hash:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:58
 static constexpr UInt32 NUM_BUCKETS = 1ULL << BITS_FOR_BUCKET;
 static constexpr UInt32 MAX_BUCKET = NUM_BUCKETS - 1;

 size_t hash(const Key & x) const { return Hash::operator()(x); }

 /// NOTE Bad for hash tables with more than 2^32 cells.
 static size_t getBucketFromHash(size_t hash_value) { return (hash_value >> (32 - BITS_FOR_BUCKET)) & MAX_BUCKET; }

BITS_FOR_BUCKET defaults to 8, so 256 buckets, and the bucket is taken from the high bytes of the 64-bit hash:

 text
 
 
 64-bit hash
 |<-------------------------->|
 | bucket | sub-table index |
 | 8 bits | (mask & (size-1)) |
 +-----------+---------------------+
 | 0..255 | depends on the bucket's own size |
 +-----------+---------------------+

 hash >> 24 & 255 -> which of the 256 sub-tables
 hash & (2^k - 1) -> which cell inside that sub-table

The choice of the high bits is not arbitrary and is the subtle part of the whole design. The single-level table uses the low bits of the hash to index its cells:

 cpp
 
 
// src/Common/HashTable/HashTable.h:238
 /// From the hash value, get the cell number in the hash table.
 size_t place(size_t x) const { return x & mask(); }

If the bucket were also selected from the low bits, the two uses would collide: every bucket would receive keys whose low bits are a fixed value, and the sub-table’s own probe would be confined to a single cell. The sub-tables would degenerate into 256 linked lists. Taking the bucket from the top of the hash and the cell from the bottom keeps the two independent, which is exactly what a good hash function provides.

The default bucket count is 8 bits, and the comment “Bad for hash tables with more than 2^32 cells” is about the other direction: the shift hash_value >> (32 - BITS_FOR_BUCKET) throws away everything above bit 32, so the bucket selection only sees 32 bits of the hash. For a table with more than 2^32 cells, the top bits would carry real information and throwing them away would lose it.

The sub-tables are ordinary tables

Each bucket is a plain HashTable, so all the machinery of the previous article — zeroed cells, linear probing, Robin Hood ordering — applies inside a bucket. The two-level table only adds the routing:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:316
 template <typename KeyHolder>
 void ALWAYS_INLINE emplace(KeyHolder && key_holder, LookupResult & it, bool & inserted, size_t hash_value)
 {
 size_t buck = getBucketFromHash(hash_value);
 impls[buck].emplace(key_holder, it, inserted, hash_value);
 }

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:333
 LookupResult ALWAYS_INLINE find(Key x, size_t hash_value)
 {
 size_t buck = getBucketFromHash(hash_value);
 return impls[buck].find(x, hash_value);
 }

and the storage is a plain array of sub-tables:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:100
 Impl impls[NUM_BUCKETS];

 /// `bucket_cells_prefix[b]` is the number of cells in the buckets before `b`. See `computeBucketPrefix`.
 std::vector<size_t> bucket_cells_prefix;

Impl impls[NUM_BUCKETS] is 256 sub-tables by value, not by pointer. Each sub-table owns its buffer, so the whole structure is 256 small allocations plus the bookkeeping of each.

How a bucket grows

The buckets do grow, but they grow in a different shape from the single-level table. The grower used by the two-level table caps the quadratic-growth phase at 2^15 cells and switches to the Robin Hood mode above that:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:26
/// The buckets grow by 4 times while they are small, and switch to the Robin Hood mode when they become large,
/// see `HashTableGrowerWithPrecalculationAndRobinHood`.
template <size_t initial_size_degree = 8, size_t robin_hood_size_degree = 16>
using TwoLevelHashTableGrower = HashTableGrowerWithPrecalculationAndRobinHood<initial_size_degree, 15, robin_hood_size_degree>;

The template parameters are initial_size_degree = 8, max_size_degree_quadratic = 15, robin_hood_size_degree = 16. So:

up to 2^15 cells per bucket, the sub-table quadruples its size on overflow (a degree increment of two);
from 2^15 to 2^16 it doubles;
above 2^16 cells per bucket it switches to the Robin Hood mode with a 90% load factor, and from then on it keeps doubling like any other table — but at 90% load instead of 50%.

The effect of the switch is memory, not capacity: a bucket that grows large stops paying the factor-of-two penalty of the 0.5 load factor, because the same number of keys now needs half as many cells. That is also what makes the external-memory path possible: when the aggregation spills to disk, it spills bucket by bucket, and each bucket’s data is self-contained.

Resizing is amortised

A single-level table resizes by re-inserting every element: at the moment of the resize, the whole table has to be walked and every cell re-placed. In a two-level table, each bucket resizes on its own, and since the buckets are filled roughly uniformly, the individual resizes happen at different times. The total work is the same, but it is spread out, and — crucially — the pause is per-bucket rather than table-wide.

The comment in the header says it exactly:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:14
 * - delay during resizes is amortized, since the small hash tables will be resized separately;

When is the two-level table used?

The decision is a setting, and it is a pair of settings now — one counting keys, one counting bytes:

 cpp
 
 
// src/Core/Settings.cpp:1500
 DECLARE(UInt64, group_by_two_level_threshold, 100000, R"(
From what number of keys, a two-level aggregation starts. 0 - the threshold is not set.
)", 0) \
 DECLARE(UInt64, group_by_two_level_threshold_bytes, 50000000, R"(
From what size of the aggregation state in bytes, a two-level aggregation begins to be used. 0 - the threshold is not set. Two-level aggregation is used when at least one of the thresholds is triggered.
)", 0) \

So by default a GROUP BY switches to the two-level structure once it holds 100 000 keys, or once its state grows to 50 MB — whichever happens first. Both thresholds can be forced from SQL:

 sql
 
 
SELECT user_id, count()
FROM events
GROUP BY user_id
SETTINGS group_by_two_level_threshold = 1000000,
 group_by_two_level_threshold_bytes = 50000000;

Setting either threshold to 0 disables that trigger — which is exactly what the pipeline does when a two-level aggregation would be useless.

The predicate that implements this lives in src/Interpreters/Aggregator.cpp:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:118
bool worthConvertToTwoLevel(
 size_t group_by_two_level_threshold, size_t result_size, size_t group_by_two_level_threshold_bytes, auto result_size_bytes)
{
 // params.group_by_two_level_threshold will be equal to 0 if we have only one thread to execute aggregation (refer to AggregatingStep::transformPipeline).
 return (group_by_two_level_threshold && result_size >= group_by_two_level_threshold)
 || (group_by_two_level_threshold_bytes && result_size_bytes >= static_cast<Int64>(group_by_two_level_threshold_bytes));
}

Note the && in each disjunct: a threshold of 0 disables that particular trigger, which is why the settings can be turned off individually.

The conversion itself happens during the aggregation, when a block has been added and the table is re-examined:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:2668
 bool worth_convert_to_two_level = worthConvertToTwoLevel(
 params.group_by_two_level_threshold, result_size, params.group_by_two_level_threshold_bytes, result_size_bytes);

 /** Converting to a two-level data structure.
 * It allows you to make, in the subsequent, an effective merge - either economical from memory or parallel.
 */
 if (result.isConvertibleToTwoLevel() && worth_convert_to_two_level)
 result.convertToTwoLevel();

and the conversion itself rebuilds the variant with the two-level type:

 cpp
 
 
// src/Interpreters/AggregatedDataVariants.cpp:263
void AggregatedDataVariants::convertToTwoLevel()
{
 ProfileEvents::increment(ProfileEvents::AggregationConvertedToTwoLevel);

 if (aggregator)
 LOG_TRACE(aggregator->log, "Converting aggregation data to two-level.");

 switch (type)
 {
#define M(NAME) \
 case Type::NAME: \
 NAME ## _two_level = std::make_unique<decltype(NAME ## _two_level)::element_type>(*(NAME)); \
 (NAME ## _two_level)->top_k_heap = std::move((NAME)->top_k_heap); \
 (NAME).reset(); \
 type = Type::NAME ## _two_level; \
 break;

 APPLY_FOR_VARIANTS_CONVERTIBLE_TO_TWO_LEVEL(M)

#undef M

 default:
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Wrong data variant passed.");
 }
}

The TwoLevelHashTable has a constructor that takes a single-level table and distributes it, which is what makes the conversion a one-liner:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:116
 template <typename Source>
 requires(!std::is_arithmetic_v<Source>)
 explicit TwoLevelHashTable(const Source & src)
 {
 typename Source::const_iterator it = src.begin();

 /// It is assumed that the zero key (stored separately) is first in iteration order.
 if (it != src.end() && it.getPtr()->isZero(src))
 {
 insert(it->getValue());
 ++it;
 }

 for (; it != src.end(); ++it)
 {
 const Cell * cell = it.getPtr();
 size_t hash_value = cell->getHash(src);
 size_t buck = getBucketFromHash(hash_value);
 impls[buck].insertUniqueNonZero(cell, hash_value);
 }
 }

The zero key is handled explicitly, because it lives outside the buffer and does not follow the bucket rule — it goes to bucket 0.

The merge is the point

Now the reason the whole thing exists. The aggregation pipeline decides whether a two-level aggregation is even allowed:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:503
 bool allow_to_use_two_level_group_by = pipeline.getNumStreams() > 1 || params.max_bytes_before_external_group_by != 0;

Two conditions: either there is more than one stream (so the merge is worth parallelising), or the query is allowed to spill to disk (so the bucket structure is needed to spill incrementally). If neither holds, the thresholds are zeroed out and the aggregation stays single-level:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:520
 if (!allow_to_use_two_level_group_by)
 {
 params.group_by_two_level_threshold = 0;
 params.group_by_two_level_threshold_bytes = 0;
 }

When the two-level table is used, the merge stage is a source that hands out one bucket at a time, and the pipeline runs several of them in parallel:

 cpp
 
 
// src/Processors/Transforms/AggregatingTransform.cpp:208
/// Worker which merges buckets for two-level aggregation.
/// Atomically increments bucket counter and returns merged result.
class ConvertingAggregatedToChunksWithMergingSource final : public ISource
{
public:
 static constexpr UInt32 NUM_BUCKETS = 256;

The worker takes the next bucket with an atomic counter and merges it across all the per-thread tables:

 cpp
 
 
// src/Processors/Transforms/AggregatingTransform.cpp:272
 Chunk generate() override
 {
 UInt32 bucket_num = shared_data->next_bucket_to_merge.fetch_add(1);

 if (bucket_num >= NUM_BUCKETS)
 {
 data.reset();
 return {};
 }

The atomic counter is the entire coordination mechanism: there is no lock, no work queue, no barrier. A worker that finishes bucket 3 immediately grabs bucket 4, and the workers end up balanced by the cost of the buckets they happen to pick up. Since the buckets partition the key space, no two workers ever touch the same key.

The actual merge of one bucket, in Aggregator::mergeBucketImpl, is “for each thread’s table, merge its bucket into the first thread’s bucket”:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5200
void NO_INLINE Aggregator::mergeBucketImpl(
 ManyAggregatedDataVariants & data, Int32 bucket, Arena * arena, std::atomic<bool> & is_cancelled) const
{
 /// We merge all aggregation results to the first.
 AggregatedDataVariantsPtr & res = data[0];

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5213
 auto & dst = getDataVariant<Method>(*res).data.impls[bucket];

Note that the bucket is an index into the impls[] array of the two-level table — the same array the routing used. That is the whole contract: the bucket is a function of the key, therefore the bucket is a partition of the key space.

The same trick appears in the uniqExact aggregate function, which keeps a single-level set and converts it to a two-level one when it grows:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:386
 static bool worthConvertingToTwoLevel(size_t size) { return size > 100'000; }

and merges it bucket-wise in parallel:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:225
 auto thread_func = [&two_level_ptrs, &first_two_level, next_bucket_to_merge, &is_cancelled]()
 {
 while (true)
 {
 if (is_cancelled.load(std::memory_order_seq_cst))
 return;

 const auto bucket = next_bucket_to_merge->fetch_add(1);
 if (bucket >= NUM_BUCKETS)
 return;

 for (size_t j = 1; j < two_level_ptrs.size(); ++j)
 {
 if (is_cancelled.load(std::memory_order_seq_cst))
 return;

 first_two_level.impls[bucket].merge(two_level_ptrs[j]->impls[bucket]);
 }
 }
 };

with the same bounded-threads trick:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:246
 const size_t max_threads_to_enqueue = std::min<size_t>(thread_pool.getMaxThreads(), NUM_BUCKETS);

There is no point in queueing more workers than there are buckets.

The string flavour

Strings get their own two-level table, because the string table is not a HashTable but a composite of five sub-tables (one per key size class). The routing therefore has to pick both the bucket and the size class:

 cpp
 
 
// src/Common/HashTable/TwoLevelStringHashTable.h:77
 /// The bucket-aware size-class dispatch: the shared implementation in `StringHashTable`
 /// classifies and packs the key and computes its hash; the resolver here picks the bucket
 /// from that hash and hands out the bucket's submap of the right class.
 template <typename Self, typename Func, typename KeyHolder>
 static auto ALWAYS_INLINE dispatch(Self & self, KeyHolder && key_holder, Func && func)
 {
 return Impl::dispatchOnKeyClass(
 [&](auto key_class, size_t hash_value) -> auto &
 { return self.impls[getBucketFromHash(hash_value)].template submapForClass<decltype(key_class)::value>(); },
 std::forward<KeyHolder>(key_holder),
 StringHashTableHash{},
 std::forward<Func>(func));
 }

256 buckets times 5 size classes is 1280 sub-tables. That is a lot of bookkeeping for a table that is usually not huge, which is why the string tables are only converted to two-level when the aggregation really needs it.

Offsets across buckets

One thing the two-level table has to solve is numbering the cells globally. The aggregation code refers to cells by an offset into the whole table, and with 256 separate buffers there is no single array to index into. The solution is a prefix sum over the buckets:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:435
 /// Prefix sums that `offsetInternal` uses to number cells across all buckets.
 /// Call it after the last insert and before reading offsets, never while another thread reads them.
 void computeBucketPrefix()
 {
 bucket_cells_prefix.assign(NUM_BUCKETS, 0);
 size_t run = 0;
 for (UInt32 i = 0; i < NUM_BUCKETS; ++i)
 {
 bucket_cells_prefix[i] = run;
 run += impls[i].getBufferSizeInCells();
 }
 }

and the offset itself:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:448
 /// Number of the cell over all buckets. The zero cell gets 0. Any other cell gets its position in the
 /// concatenated bucket buffers plus one. So the number fits an array of `getBufferSizeInCells() + 1`.
 size_t offsetInternal(ConstLookupResult ptr) const
 {
 if constexpr (NUM_BUCKETS == 1)
 return impls[0].offsetInternal(ptr);

 const size_t buck = getBucketFromHash(ptr->getHash(*this));
 if (ptr->isZero(impls[buck]))
 return 0;

 chassert(!bucket_cells_prefix.empty(), "computeBucketPrefix must run before an offset is read");
 return bucket_cells_prefix[buck] + (ptr - impls[buck].buf) + 1;
 }

The “+1” is the zero key’s slot, which is numbered 0.

Trade-offs

What the two-level table buys:

A parallelisable merge. The merge of N per-thread tables becomes 256 independent merges with an atomic counter and no locks. The serial tail of a parallel GROUP BY mostly disappears.
Incremental spilling. Because a bucket is self-contained, the aggregation can write one bucket to disk and continue, and the merge can read buckets back one at a time.
Amortised resize pauses. The expensive rehash is per-bucket, so no single pause is as long.
A memory-efficient large bucket. The Robin Hood switch at 2^16 cells means a bucket that grows large halves its memory overhead per key, at the cost of a little CPU for the chain ordering.

What it costs:

One extra indirection on every probe. The single-level table computes hash & mask; the two-level table computes a shift, an array index, and then the mask. The header admits it: “Usually works a little slower than a simple hash table.”
256 allocations instead of one. Each sub-table has its own buffer, its own grower, its own size counter.
A weaker guarantee for large tables. The bucket is selected from 32 bits of the hash, so a table with more than 2^32 cells is explicitly out of scope.
Complexity at the boundaries. The zero key does not belong to a bucket; the prefix sums must be recomputed after growth; the string variant has to route on two axes at once.

The design is a good example of a structural optimisation: rather than making the hash table faster, ClickHouse changes the shape of the data so that the expensive part of the query — the merge — can be decomposed into 256 embarrassingly parallel pieces.

How ClickHouse Sorts in Parallel: Partial Sort, Merge Sorted Streams and External Sort

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

ORDER BY is the operation every database is worst at. It is the one requirement that is inherently global: the first row of the result depends on every row of the input. Everything else — filtering, projection, aggregation — is local and streams beautifully. Sorting does not.

The classic answer is “sort in chunks, then merge the sorted chunks”. That is what every system does, but there are two quite different versions of it, and ClickHouse implements both, in two quite different classes.

One version assumes the input is already sorted on a prefix of the sort key (a MergeTree table read in primary-key order, or the output of a previous ORDER BY). Then the merge is cheap and the only work is stitching together runs that are already in order.

The other version assumes nothing: it sorts each chunk independently and then does a k-way merge. This is the general case, and it is where the memory pressure lives — a k-way merge of N sorted runs needs all N runs resident at once.

This is a tour of both, of the trick that lets ClickHouse throw away rows it knows are useless, and of the machinery that spills sorted runs to disk when memory runs out.

The plan: what gets built

SortingStep decides which of the two regimes applies. The difference is visible in the query:

 sql
 
 
-- Full sort: the input is in no particular order, so each stream is sorted
-- on its own and the sorted streams are merged.
SELECT * FROM hits ORDER BY UserID, EventTime LIMIT 100;

 sql
 
 
-- Finish sorting: the table is sorted by (a, b) and the query asks for
-- (a, b) too, so the merge only has to stitch runs together.
SELECT * FROM hits WHERE a = 1 ORDER BY a, b LIMIT 100;

The “finish sorting” case is when the input is already sorted on a prefix:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:381
void SortingStep::finishSorting(
 QueryPipelineBuilder & pipeline, const SortDescription & input_sort_desc, const SortDescription & result_sort_desc, const UInt64 limit_)
{
 pipeline.addSimpleTransform(
 [&](const SharedHeader & header, QueryPipelineBuilder::StreamType stream_type) -> ProcessorPtr
 {
 if (stream_type != QueryPipelineBuilder::StreamType::Main)
 return nullptr;

 return std::make_shared<PartialSortingTransform>(header, result_sort_desc, limit_, threshold_tracker);
 });

Two transforms per stream: a PartialSortingTransform that sorts each chunk as it arrives, and a FinishSortingTransform that merges the sorted chunks. The “full sort” case is the same first transform followed by a MergeSortingTransform:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:522
 if (!skip_partial_sort || limit_)
 {
 pipeline.addSimpleTransform(
 [&](const SharedHeader & header, QueryPipelineBuilder::StreamType stream_type) -> ProcessorPtr
 {
 if (stream_type != sorted_stream_type)
 return nullptr;

 return std::make_shared<PartialSortingTransform>(header, result_sort_desc, limit_, threshold_tracker);
 });

Note the condition !skip_partial_sort || limit_. When there is a LIMIT, the partial sort is always applied, even if the input is already sorted, because the partial sort is what makes the limit cheap — it discards everything but the top limit rows of each chunk, so the merge only ever sees limit rows per chunk instead of the whole chunk.

`PartialSortingTransform`: sort the chunk, throw away the rest

The transform itself is short, and the interesting part is the threshold filter at the top:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:104
void PartialSortingTransform::transform(Chunk & chunk)
{
 if (chunk.getNumRows())
 {
 // The following code works with Blocks and will lose the number of
 // rows when there are no columns. We shouldn't get such block, because
 // we have to sort by at least one column.
 chassert(chunk.getNumColumns());
 }

 if (read_rows)
 read_rows->add(chunk.getNumRows());

 auto block = getInputPort().getHeader().cloneWithColumns(chunk.detachColumns());

 /// Materialize sort key columns that are ColumnReplicated to avoid index indirection during comparison and sorting.
 for (const auto & col_desc : description)
 {
 auto & column_entry = block.getByName(col_desc.column_name);
 column_entry.column = convertToFullColumnIfReplicationNotUseful(column_entry.column);
 }

 /** If we've saved columns from previously blocks we could filter all rows from current block
 * which are unnecessary for sortBlock(...) because they obviously won't be in the top LIMIT rows.
 */
 if (!sort_description_threshold_columns.empty() && !threshold_tracker)
 {
 UInt64 rows_num = block.rows();
 auto block_columns = extractRawColumns(block, description_with_positions);

 size_t result_size_hint = getFilterMask(
 block_columns, sort_description_threshold_columns,
 description, rows_num, filter, rows_to_compare, compare_results);

 /// Everything was filtered. Skip whole chunk.
 if (result_size_hint == 0)
 return;

 if (result_size_hint < rows_num)
 {
 for (auto & column : block)
 column.column = column.column->filter(filter, result_size_hint);
 }
 }

 sortBlock(block, description, limit);

Two things happen here, and the order matters.

First, the threshold filter. sort_description_threshold_columns holds the sort key of the current N-th best row seen so far — the “cutoff”. Every row in the new chunk that compares greater than the cutoff provably cannot be in the final top-N, so it is dropped before the sort. The filter is computed with a vectorised column comparison:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:27
size_t getFilterMask(const ColumnRawPtrs & raw_block_columns, const Columns & threshold_columns,
 const SortDescription & description, size_t num_rows, IColumn::Filter & filter,
 PaddedPODArray<UInt64> & rows_to_compare, PaddedPODArray<Int8> & compare_results)
{
 filter.resize(num_rows);
 compare_results.resize(num_rows);

 if (description.size() == 1)
 {
 /// Fast path for single column
 raw_block_columns[0]->compareColumn(*threshold_columns[0], 0, nullptr, compare_results,
 description[0].direction, description[0].nulls_direction);
 }
 else
 {
 rows_to_compare.resize(num_rows);
 iota(rows_to_compare.data(), num_rows, UInt64(0));

 size_t size = description.size();
 for (size_t i = 0; i < size; ++i)
 {
 raw_block_columns[i]->compareColumn(*threshold_columns[i], 0, &rows_to_compare, compare_results,
 description[i].direction, description[i].nulls_direction);

 if (rows_to_compare.empty())
 break;
 }
 }

 size_t result_size_hint = 0;

 for (size_t i = 0; i < num_rows; ++i)
 {
 /// Leave only rows that are less then row from rhs.
 filter[i] = compare_results[i] < 0;
 result_size_hint += filter[i];
 }

 return result_size_hint;
}

compareColumn is the columnar comparison primitive: it compares a whole column against a single-row column and writes a per-row sign into compare_results. For a single sort column it is one vectorised pass; for multiple columns it narrows the candidate set on each column in turn (the rows_to_compare array is the survivor list, and it is emptied when no rows survive).

Second, sortBlock(block, description, limit). The limit argument makes the sort itself partial: ClickHouse sorts the chunk but only materialises the first limit rows of the permutation.

 cpp
 
 
// src/Interpreters/sortBlock.cpp:384
void sortBlock(Block & block, const SortDescription & description, UInt64 limit, IColumn::PermutationSortStability stability)
{
 IColumn::Permutation permutation;

#ifndef NDEBUG
 block.checkNumberOfRows();
#endif
 getBlockSortPermutationImpl(block, description, stability, limit, permutation);

#ifndef NDEBUG
 checkSortedWithPermutation(block, description, limit, permutation);
#endif

 if (permutation.empty())
 return;

 bool is_identity_permutation = isIdentityPermutation(permutation, limit);
 if (is_identity_permutation && limit == 0)
 return;

 size_t output_rows = limit ? std::min(static_cast<size_t>(limit), permutation.size()) : permutation.size();
 Columns columns = block.getColumns();
 transformColumnsWithSharedIndex(
 columns,
 [&](const ColumnPtr & col) { return is_identity_permutation ? col->cut(0, output_rows) : col->permute(permutation, limit); });
 block.setColumns(columns);
}

Note the output_rows clamp: with a limit of 10 and a chunk of 100 000 rows, the output chunk has 10 rows. The sort still had to consider all 100 000 rows (the top 10 of an unsorted chunk are not the first 10), but the permutation only materialises 10 entries and the columns are permuted with that.

So the memory cost of ORDER BY ... LIMIT 10 is proportional to the chunk size, not to the limit. Each chunk is sorted in full, truncated to 10 rows, and the merge only ever sees 10 rows per chunk.

Updating the threshold

After sorting, the transform updates the threshold to the current chunk’s limit-th row:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:151
 /// Check if we can use this block for optimization.
 if ((min_limit_for_partial_sort_optimization <= limit || threshold_tracker) && limit <= block.rows())
 {
 /** If we filtered more than limit rows from block take block last row.
 * Otherwise take last limit row.
 *
 * If current threshold value is empty, update current threshold value.
 * If min block value is less than current threshold value, update current threshold value.
 */
 size_t min_row_to_compare = limit - 1;
 auto raw_block_columns = extractRawColumns(block, description_with_positions);

 if (sort_description_threshold_columns.empty() ||
 compareWithThreshold(raw_block_columns, min_row_to_compare, sort_description_threshold_columns, description))
 {
 size_t raw_block_columns_size = raw_block_columns.size();
 Columns sort_description_threshold_columns_updated(raw_block_columns_size);

 for (size_t i = 0; i < raw_block_columns_size; ++i)
 {
 MutableColumnPtr sort_description_threshold_column_updated = raw_block_columns[i]->cloneEmpty();
 sort_description_threshold_column_updated->insertFrom(*raw_block_columns[i], min_row_to_compare);
 /// Without this, `assertTypeEquality` may fail when comparing a ColumnNullable against a ColumnReplicated (received from JOIN).
 sort_description_threshold_columns_updated[i]
 = removeSpecialRepresentations(sort_description_threshold_column_updated->getPtr());
 }

 sort_description_threshold_columns = std::move(sort_description_threshold_columns_updated);

The threshold is monotonically non-increasing (it only ever gets better), so the filter gets more aggressive over time. The min_limit_for_partial_sort_optimization guard exists because for very small limits the bookkeeping is not worth it.

There is a subtlety worth noticing: previous_row_chunk style stability is not preserved by this filter. When a row ties with the threshold, the filter keeps it (compare_results[i] < 0 is false for equal, so equal rows are… let me be careful). compare_results[i] < 0 keeps rows that are strictly less than the threshold. So a row equal to the threshold is dropped. That is fine for ORDER BY ... LIMIT without WITH TIES, and it is why LimitTransform has a completely separate mechanism for WITH TIES.

`FinishSortingTransform`: merging streams that are already sorted

When the input is already sorted on a prefix of the key, the merge does not need a heap at all — it just has to concatenate runs. The one hard part is handling a key that spans a chunk boundary.

 cpp
 
 
// src/Processors/Transforms/FinishSortingTransform.cpp:83
 /// Find the position of last already read key in current chunk.
 if (!chunks.empty())
 {
 size_t size = chunk.getNumRows();
 const auto & last_chunk = chunks.back();

 ssize_t low = -1;
 ssize_t high = size;
 while (high - low > 1)
 {
 ssize_t mid = (low + high) / 2;
 if (!less(last_chunk.getColumns(), chunk.getColumns(), last_chunk.getNumRows() - 1, mid, description_with_positions))
 low = mid;
 else
 high = mid;
 }

 size_t tail_pos = high;

 /// We need to save tail of chunk, because next chunk may starts with the same key as in tail
 /// and we should sort these rows in one portion.
 if (tail_pos != size)
 {
 auto source_columns = chunk.detachColumns();
 Columns tail_columns;

 for (auto & source_column : source_columns)
 {
 tail_columns.emplace_back(source_column->cut(tail_pos, size - tail_pos));
 source_column = source_column->cut(0, tail_pos);
 }

 chunks.emplace_back(std::move(source_columns), tail_pos);
 tail_chunk.setColumns(std::move(tail_columns), size - tail_pos);

 stage = Stage::Generate;
 return;
 }
 }

The binary search finds the first row of the new chunk whose key is greater than the last key of the previous chunk. Everything from there on has a key that might tie with the tail, so it is cut off into tail_chunk and held back. The tail is re-merged with the next chunk, so that rows with equal keys end up in one sorted run rather than being split across two and appearing in the wrong order.

The stage = Stage::Generate is the switch that makes this work: the transform has to emit what it has so far, then go back to consuming.

`MergeSortingTransform`: accumulate, remerge, spill, merge

The general case is MergeSortingTransform, and it is a state machine with three stages — Consume, Serialize, Generate — declared in the base class:

 cpp
 
 
// src/Processors/Transforms/SortingTransform.h:126
 enum class Stage : uint8_t
 {
 Consume = 0,
 Generate,
 Serialize,
 };

 Stage stage = Stage::Consume;

The base class prepare() is the dispatcher:

 cpp
 
 
// src/Processors/Transforms/SortingTransform.cpp:293
IProcessor::Status SortingTransform::prepare()
{
 if (stage == Stage::Serialize)
 {
 if (!processors.empty())
 return Status::UpdatePipeline;

 auto status = prepareSerialize();
 if (status != Status::Finished)
 return status;

 stage = Stage::Consume;
 }

 if (stage == Stage::Consume)
 {
 auto status = prepareConsume();
 if (status != Status::Finished)
 return status;

 stage = Stage::Generate;
 }

 /// stage == Stage::Generate

 if (!generated_prefix || !chunks.empty())
 return Status::Ready;

 if (!processors.empty())
 return Status::UpdatePipeline;

 return prepareGenerate();
}

The Status::UpdatePipeline returns are the graph-rewriting mechanism from the first article: when the sort decides it needs to spill to disk, it returns UpdatePipeline, the executor asks it for updatePipeline(), and the graph is rewired to insert a file sink and a file source.

Consume: accumulate chunks

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:94
void MergeSortingTransform::consume(Chunk chunk)
{
 /** Algorithm:
 * - read to memory blocks from source stream;
 * - if too many of them and if external sorting is enabled,
 * - merge all blocks to sorted stream and write it to temporary file;
 * - at the end, merge all sorted streams from temporary files and also from rest of blocks in memory.
 */

 /// If there were only const columns in sort description, then there is no need to sort.
 /// Return the chunk as is.
 if (description.empty())
 {
 generated_chunk = std::move(chunk);
 return;
 }

 removeConstColumns(chunk);
 compactReplicatedColumns(chunk);

 sum_rows_in_blocks += chunk.getNumRows();
 sum_bytes_in_blocks += chunk.allocatedBytes();
 chunks.push_back(std::move(chunk));

 /** If significant amount of data was accumulated, perform preliminary merging step.
 */
 if ((chunks.size() > 1
 && limit
 && limit * 2 < sum_rows_in_blocks /// 2 is just a guess.
 && remerge_is_useful
 && max_bytes_before_remerge
 && sum_bytes_in_blocks > max_bytes_before_remerge) || (threshold_tracker && (static_cast<double>(sum_rows_in_blocks) > static_cast<double>(limit) * 1.5)))
 {
 remerge();
 }

 /** If too many of them and if external sorting is enabled,
 * will merge blocks that we have in memory at this moment and write merged stream to temporary (compressed) file.
 * NOTE. It's possible to check free space in filesystem.
 */
 if (max_bytes_in_block_before_external_sort && sum_bytes_in_blocks > max_bytes_in_block_before_external_sort)
 {
 Int64 query_memory = getCurrentQueryMemoryUsage();
 if (!max_bytes_in_query_before_external_sort || query_memory > static_cast<Int64>(max_bytes_in_query_before_external_sort))
 {
 auto sink = prepareSpill();
 auto source = std::make_shared<BufferingFromFileSource>(sink->getPort().getSharedHeader(), sink->getHolder(), log);
 processors.emplace_back(source);
 processors.emplace_back(sink);
 stage = Stage::Serialize;
 }
 }
}

Three behaviours in one function.

Remerge. When there is a LIMIT and enough data has accumulated, the in-memory chunks are merged now rather than later. This is the classic top-K trick: after merging, only limit rows survive, so the memory footprint of the accumulator collapses from “all data so far” to “the top limit rows”. The condition limit * 2 < sum_rows_in_blocks means “we have at least twice as many rows as we need”, so the remerge is worth it.

Spill. When the accumulated bytes exceed max_bytes_in_block_before_external_sort (and the whole query’s memory exceeds max_bytes_in_query_before_external_sort), the accumulated chunks are merged and written to a temporary file. The transform enters the Serialize stage, and the new processors — the file sink and the file source — are queued for updatePipeline().

The limit on remerge. remerge_is_useful is a latch:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:333
void MergeSortingTransform::remerge()
{
 LOG_DEBUG(log, "Re-merging intermediate ORDER BY data ({} blocks with {} rows) to save memory consumption", chunks.size(), sum_rows_in_blocks);

 /// NOTE Maybe concat all blocks and partial sort will be faster than merge?
 MergeSorter remerge_sorter(std::make_shared<const Block>(header_without_constants), std::move(chunks), description, max_merged_block_size, limit);

 Chunks new_chunks;
 size_t new_sum_rows_in_blocks = 0;
 size_t new_sum_bytes_in_blocks = 0;

 while (auto chunk = remerge_sorter.read())
 {
 new_sum_rows_in_blocks += chunk.getNumRows();
 new_sum_bytes_in_blocks += chunk.allocatedBytes();
 new_chunks.emplace_back(std::move(chunk));
 }

 LOG_DEBUG(log, "Memory usage is lowered from {} to {}", ReadableSize(sum_bytes_in_blocks), ReadableSize(new_sum_bytes_in_blocks));

 /// If the memory consumption was not lowered enough - we will not perform remerge anymore.
 if (remerge_lowered_memory_bytes_ratio > 0.0 && (static_cast<double>(new_sum_bytes_in_blocks) * remerge_lowered_memory_bytes_ratio > static_cast<double>(sum_bytes_in_blocks)))
 {
 remerge_is_useful = false;
 LOG_DEBUG(log, "Re-merging is not useful (memory usage was not lowered by remerge_sort_lowered_memory_bytes_ratio={})", remerge_lowered_memory_bytes_ratio);
 }

The default ratio is 2.0 (remerge_sort_lowered_memory_bytes_ratio), so the remerge must halve the memory to be considered worth doing. If it does not, the flag is cleared and no further remerges happen — because a remerge that does not reduce memory is pure loss: you paid for a merge and got nothing.

The default threshold is max_bytes_before_remerge_sort = 1000000000, one gigabyte.

Serialize and Generate

Serialize is the stage where the accumulated chunks are drained into the file sink, one sorted run at a time:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:252
void MergeSortingTransform::serialize()
{
 current_chunk = merge_sorter->read();
 if (!current_chunk)
 merge_sorter.reset();
 /// A read of `MergeSorter::Mode::MergeUniqueChunks` that consumes only duplicates returns no rows. The
 /// run gets no block for it, and the next call reads on.
 else if (!current_chunk.hasRows())
 current_chunk.clear();
}

Generate is where the final merge happens. If nothing was spilled, the answer is a plain in-memory merge of the accumulated chunks:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:263
void MergeSortingTransform::generate()
{
 if (!generated_prefix)
 {
 if (temporary_files_num == 0)
 {
 merge_sorter = std::make_unique<MergeSorter>(
 std::make_shared<const Block>(header_without_constants),
 std::move(chunks),
 description,
 max_merged_block_size,
 limit,
 merge_mode);
 }
 else
 {
 ProfileEvents::increment(external_merge_event);
 LOG_INFO(log, "There are {} temporary sorted parts to merge", temporary_files_num);

 processors.emplace_back(
 std::make_shared<MergeSorterSource>(
 std::make_shared<const Block>(header_without_constants),
 std::move(chunks),
 description,
 max_merged_block_size,
 limit,
 merge_mode));
 }

 generated_prefix = true;
 }

If there were temporary files, a MergeSorterSource is added to the pipeline and the graph is rewired so that the spilled runs (read back from disk) and the in-memory tail are merged together by a MergingSortedTransform — which is a proper k-way merge with a heap.

The k-way merge

MergingSortedTransform uses MergingSortedAlgorithm, which keeps one SortCursor per input stream in a heap and pops the smallest key. The interesting optimisation is the “totally less” shortcut:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:547
template <typename TSortingHeap>
IMergingAlgorithm::Status MergingSortedAlgorithm::mergeImpl(TSortingHeap & queue)
{
 /// Take rows in required order and put them into `merged_data`, while the rows are no more than `max_block_size`
 while (queue.isValid())
 {
 if (merged_data.hasEnoughRows())
 return Status(merged_data.pull());

 auto current = queue.current();

 if (current.impl->isLast() && current_inputs[current.impl->order].skip_last_row)
 {
 /// Get the next block from the corresponding source, if there is one.
 queue.removeTop();
 return Status(current.impl->order);
 }

 if (current.impl->isFirst()
 && !current_inputs[current.impl->order].skip_last_row /// Ignore optimization if last row should be skipped.
 && (queue.size() == 1
 || (queue.size() >= 2 && current.totallyLessOrEquals(queue.nextChild()))))
 {
 /** This is special optimization if current cursor is totally less than next cursor.
 * We want to insert current cursor chunk directly in merged data.
 *
 * First if merged_data is not empty we need to flush it.
 * We will get into the same condition on next mergeBatch call.
 *
 * Then we can insert chunk directly in merged data.
 */

 if (merged_data.mergedRows() != 0)
 return Status(merged_data.pull());

 /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl)
 size_t source_num = current.impl->order;

 auto chunk_num_rows = current_inputs[source_num].chunk.getNumRows();

 UInt64 total_merged_rows_after_insertion = merged_data.mergedRows() + chunk_num_rows;
 bool limit_reached = limit && total_merged_rows_after_insertion >= limit;

 if (limit && total_merged_rows_after_insertion > limit)
 chunk_num_rows -= total_merged_rows_after_insertion - limit;

 insertChunk(source_num);

 /// We will get the next block from the corresponding source, if there is one.
 queue.removeTop();

 auto status = Status(merged_data.pull(), limit_reached);

 if (!limit_reached)
 status.required_source = source_num;

 return status;
 }

totallyLessOrEquals asks: “is the last key of this stream less than or equal to the first key of the next-smallest stream?” If yes, then the entire chunk of the current stream comes before everything in the other streams, and it can be copied wholesale instead of row by row.

That turns the merge from O(rows log streams) comparisons into a memcpy for the common case. When the data is nearly sorted — which it usually is, because each run came from an already-sorted source — most chunks take the fast path.

Trade-offs

What this buys you.

The LIMIT optimisation is the big one. ORDER BY x LIMIT 10 over a hundred million rows costs: sort each chunk (in memory, chunk by chunk), truncate to 10 rows, merge 10 rows per chunk. Memory is O(chunk size × threads), not O(dataset). And the threshold filter improves on that further: once the cutoff is tight, whole chunks are discarded without being sorted at all — if (result_size_hint == 0) return;.

The remerge trick generalises the same idea beyond LIMIT. Even without a limit, merging periodically keeps the accumulator’s memory bounded and converts a “merge N runs at the end” problem into a series of smaller merges, which is also better for the page cache.

The external sort path is honest about the fact that ORDER BY is a blocking operation: if the data does not fit, it goes to disk, in sorted runs, and the final merge is a k-way merge over those runs. The runs are written with the configured temporary-files codec (usually LZ4), so the disk traffic is compressed.

What it costs.

Sorting is still the slowest thing in the query, and the partial sort does not help the general case: without a LIMIT, every chunk must be fully sorted, and the merge must touch every row. sortBlock allocates a permutation of the chunk’s row indices — 8 bytes per row — and the column permutation that follows is a random-access pass over the column data, which is cache-hostile for wide rows.

The threshold filter has a cost of its own: a full vectorised comparison of the chunk against the cutoff, even when it rejects nothing. For a LIMIT query where the data is not sorted, the filter will usually reject nothing on the first chunk and progressively more later, so the average benefit depends heavily on the data.

The remerge is a gamble. It costs a merge, and it only pays off if the merge actually reduces memory by the configured ratio. The remerge_is_useful latch means the first unhelpful remerge disables all future ones — but the first one was already paid for. And the threshold for even attempting a remerge is one gigabyte of accumulated data, so the memory spike before the first remerge is real.

Finally, the tail-splitting logic in FinishSortingTransform is subtle and easy to get wrong. If a key spans a chunk boundary and the tail is not held back, the rows with equal keys are emitted in two separate sorted runs and the final order among them is wrong. The binary search that finds the split point is O(log n) per chunk, which is cheap, but the reason it exists is not obvious from the code — which is why the comment is there.

How ClickHouse Skips Data: Granules, Marks, and Bit-Packed Indexes

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A SELECT on a MergeTree table is, mechanically, a scan of a set of files. The only reason it is not a full scan is that the files are chopped into pieces, and each piece carries enough metadata for the engine to prove that the piece cannot contain a matching row.

ClickHouse calls those pieces granules, the metadata is called a mark, and the mechanism that ties them together is the primary key index — one key value per granule, held in memory.

This is the single most important performance idea in MergeTree, and it is built out of three small, very specific tricks:

Granules are sized in bytes, not rows, so a granule of fat rows is not a granule of a million rows.
Marks are bit-packed in memory — three bytes per mark for an integer column, instead of the sixteen bytes a naive {offset, offset_in_block} pair would take.
The primary key index is trimmed: if a key column is nearly constant within a part, the key columns after it are not kept in memory at all, because they cannot narrow anything down.

Granules: sized in bytes, not rows

The classic description of MergeTree granularity is “8192 rows”, and that is the default value of the index_granularity setting. But that is only the cap. The actual mechanism computes the number of rows per granule from the size of the data being written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:115
size_t computeIndexGranularity(
 size_t rows,
 size_t bytes_uncompressed,
 size_t index_granularity_bytes,
 size_t fixed_index_granularity_rows,
 bool blocks_are_granules,
 bool can_use_adaptive_index_granularity)
{
 size_t index_granularity_for_block = 0;

 if (!can_use_adaptive_index_granularity)
 {
 index_granularity_for_block = fixed_index_granularity_rows;
 }
 else
 {
 if (blocks_are_granules)
 {
 index_granularity_for_block = rows;
 }
 else if (index_granularity_bytes != 0 && bytes_uncompressed >= index_granularity_bytes)
 {
 /// index_granularity_bytes == 0 disables adaptive sizing upstream, so this branch is
 /// unreachable with a zero divisor; the explicit check makes that visible to static analysis.
 size_t granules_in_block = bytes_uncompressed / index_granularity_bytes;
 index_granularity_for_block = rows / granules_in_block;
 }
 else
 {
 size_t size_of_row_in_bytes = std::max(bytes_uncompressed / rows, 1UL);
 index_granularity_for_block = index_granularity_bytes / size_of_row_in_bytes;
 }
 }

 /// We should be less or equal than fixed index granularity.
 /// But if block size is a granule size then do not adjust it.
 /// Granularity greater than fixed granularity might come from compact part.
 if (!blocks_are_granules)
 index_granularity_for_block = std::min(fixed_index_granularity_rows, index_granularity_for_block);

 /// Very rare case when index granularity bytes less than single row.
 if (index_granularity_for_block == 0)
 index_granularity_for_block = 1;

 return index_granularity_for_block;
}

Read the interesting branch — the else at the bottom. If the block is smaller than index_granularity_bytes, then

text

 rows_per_granule = index_granularity_bytes / average_row_size

That is: keep the granule a constant number of bytes. The default index_granularity_bytes is 10 MB, so a granule is 10 MB of data whether its rows are 10 bytes wide or 10 KB wide.

Why does this matter? Because a granule is the unit of skipping. If granules were a fixed 8192 rows, then a table of 1 KB rows would have 8 MB granules (fine) but a table of 100 KB rows would have 800 MB granules (useless — you cannot skip anything smaller than 800 MB). Sizing in bytes makes the granule a predictable, bounded amount of IO, and therefore a predictable, bounded amount of waste when you have to read a granule that does contain matches.

The price is that the row count of a granule is no longer constant, so it has to be remembered. The adaptive granularity object stores partial sums:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularityAdaptive.h:9
/// Class that stores adaptive index granularity.
/// Inside it contains vector of partial sums of rows after mark:
/// |-----|---|----|----|
/// | 5 | 8 | 12 | 16 |
class MergeTreeIndexGranularityAdaptive final : public MergeTreeIndexGranularity

One size_t per mark, and the row count of any mark is a subtraction. The alternative — a constant number of rows per granule — is available as use_const_adaptive_granularity, and its setting description explains the trade-off exactly:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:2174
 DECLARE(Bool, use_const_adaptive_granularity, false, R"(
Non-const adaptive granularity (the default, `index_granularity_bytes` is not `0`) keeps
granules at a constant size in bytes, so their row count varies: for every
written block, rows per granule is `index_granularity_bytes` divided by the
average size of a row in that block, capped at `index_granularity`. Because it differs
from granule to granule, the row count of every granule in the part has to be kept in
memory, 8 bytes each, which on large tables adds up to tens of gigabytes.

“Tens of gigabytes” — that is the cost of the byte-granularity guarantee, and it is the reason the constant-row variant exists at all. On a table with 100 million parts × 100 marks each, 8 bytes per mark is 80 GB of pure bookkeeping.

The marks themselves are appended as the block is written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:162
void fillIndexGranularityForWidePart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 index_granularity.appendMark(index_granularity_for_block);
}

A Wide part gets exactly index_granularity_for_block rows per mark. A Compact part, where all columns share one file, has to be more careful:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:169
void fillIndexGranularityForCompactPart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 {
 size_t rows_left_in_block = rows_in_block - current_row;

 /// Try to extend last granule if block is large enough
 /// or it isn't first in granule (index_offset != 0).
 if (rows_left_in_block < index_granularity_for_block &&
 (rows_in_block >= index_granularity_for_block || index_offset != 0))
 {
 // If enough rows are left, create a new granule. Otherwise, extend previous granule.
 // So, real size of granule differs from index_granularity_for_block not more than 50%.
 if (rows_left_in_block * 2 >= index_granularity_for_block)
 index_granularity.appendMark(rows_left_in_block);
 else
 index_granularity.addRowsToLastMark(rows_left_in_block);
 }
 else
 {
 index_granularity.appendMark(index_granularity_for_block);
 }
 }
}

The comment “real size of granule differs from index_granularity_for_block not more than 50%” is the invariant: a short tail is either given its own granule (if it is at least half a granule) or merged into the previous one.

Marks: where in the file does each granule start?

A granule is a range of rows. To read it, you need to know where in the compressed .bin file its rows live. That is the mark:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:14
/** Mark is the position in the compressed file. The compressed file consists of adjacent compressed blocks.
 * Mark is a tuple - the offset in the file to the start of the compressed block, the offset in the decompressed block to the start of the data.
 */
struct MarkInCompressedFile
{
 size_t offset_in_compressed_file;
 size_t offset_in_decompressed_block;

Two numbers, both size_t, so sixteen bytes per mark in the naive layout. For a part with 100 marks that is nothing; for a table with 100 000 parts of 1 GB each (100 TB), 100 marks per part × 16 bytes × 100 000 parts = 160 MB of index — and that is only if every mark of every column is resident.

Which is why the in-memory representation is not a struct array:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:30
/**
 * In-memory representation of an array of marks.
 *
 * Uses an ad-hoc compression scheme that decreases memory usage while allowing
 * random access in O(1) time.
 * This is independent from the marks *file* format, which may be uncompressed
 * or use a different compression method.
 *
 * Typical memory usage:
 * * ~3 bytes/mark for integer columns
 * * ~5 bytes/mark for string columns
 * * ~0.3 bytes/mark for trivial marks in auxiliary dict files of LowCardinality columns
 */
class MarksInCompressedFile

Three bytes per mark. How?

The trick is that marks are not random. The comment in the header spells out the regularities the compression exploits:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:51
 /** We need to store a sequence of marks, each consisting of two 64-bit integers:
 * offset_in_compressed_file and offset_in_decompressed_block. We'll call them x and y for
 * convenience, since compression doesn't care what they mean. The compression exploits the
 * following regularities:
 * * y is usually zero.
 * * x usually increases steadily.
 * * Differences between x values in nearby marks usually fit in much fewer than 64 bits.
 *
 * We split the sequence of marks into blocks, each containing MARKS_PER_BLOCK marks.
 * (Not to be confused with data blocks.)
 * For each mark, we store the difference [value] - [min value in the block], for each of the
 * two values in the mark. Each block specifies the number of bits to use for these differences
 * for all marks in this block.
 * The smaller the blocks the fewer bits are required, but the bigger the relative overhead of
 * block headers.
 *
 * Packed marks and block headers all live in one contiguous array.
 */

So: 256 marks per block; per block, store the minimum x and y, and the number of bits needed to represent the spread within the block; then store every mark as a delta from the block minimum, in that many bits.

The block header computation:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:122
void MarksInCompressedFile::Builder::flushBlock(const MarkInCompressedFile * data, size_t count)
{
 chassert(count > 0 && count <= MARKS_PER_BLOCK);

 size_t block_idx = marks_flushed / MARKS_PER_BLOCK;
 BlockInfo & block = blocks[block_idx];
 block.bit_offset_in_packed_array = packed_bits;

 /// Compute block metadata: min values, bit widths.
 size_t max_x = 0;
 size_t max_y = 0;
 for (size_t i = 0; i < count; ++i)
 {
 block.min_x = std::min(block.min_x, data[i].offset_in_compressed_file);
 max_x = std::max(max_x, data[i].offset_in_compressed_file);
 block.min_y = std::min(block.min_y, data[i].offset_in_decompressed_block);
 max_y = std::max(max_y, data[i].offset_in_decompressed_block);
 block.trailing_zero_bits_in_y
 = std::min(block.trailing_zero_bits_in_y, static_cast<UInt8>(getTrailingZeroBits(data[i].offset_in_decompressed_block)));
 }

 block.bits_for_x = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits(max_x - block.min_x));
 block.bits_for_y
 = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits((max_y - block.min_y) >> block.trailing_zero_bits_in_y));

Two ideas in six lines.

Delta encoding per block. bits_for_x is the number of bits needed to represent the range of x values inside the block, not the magnitude of x itself. If a block of 256 marks spans 64 KB of compressed file, that is 19 bits per mark, not 64.

Trailing-zero stripping for y. y is the offset inside the decompressed block, and it is usually zero — a granule normally starts at the beginning of a compressed block. The code computes the number of trailing zero bits common to all the y values in the block and shifts them all right by that amount before packing. For a block where every y is zero, the shift is 63 and bits_for_y becomes zero: the y coordinate costs no bits at all.

The write path, showing both encodings:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:153
 /// Write bit-packed deltas.
 size_t bit_offset = packed_bits;
 for (size_t i = 0; i < count; ++i)
 {
 writeBitsPacked64(packed.data(), bit_offset, data[i].offset_in_compressed_file - block.min_x);
 writeBitsPacked64(
 packed.data(),
 bit_offset + block.bits_for_x,
 (data[i].offset_in_decompressed_block - block.min_y) >> block.trailing_zero_bits_in_y);
 bit_offset += block.bits_for_x + block.bits_for_y;
 }

And the read path, which has to be O(1) — a mark lookup happens for every granule the reader touches:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:47
std::tuple<const MarksInCompressedFile::BlockInfo *, size_t> MarksInCompressedFile::lookUpMark(size_t idx) const
{
 size_t block_idx = idx / MARKS_PER_BLOCK;
 const BlockInfo & block = blocks[block_idx];
 size_t offset = block.bit_offset_in_packed_array + (idx - block_idx * MARKS_PER_BLOCK) * (block.bits_for_x + block.bits_for_y);
 return {&block, offset};
}

Divide by 256 to find the block header, then a shift-and-mask to find the mark inside it. That is the whole random-access structure: one array division and one multiply. Compare that to a compressed list of marks, which would need a binary search.

Where do the marks come from at write time?

The mark is written when the compressed block is flushed, so it is simply the state of the hashing/compression stream at that moment:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:351
 MarkInCompressedFile mark{stream->plain_hashing.count(), stream->compressed_hashing.offset()};

plain_hashing.count() is the number of uncompressed bytes written so far — that is y, the offset inside the decompressed block. And compressed_hashing.offset() is the number of compressed bytes written so far — that is x, the offset of the compressed block in the file.

The mark cache

Marks are read from .mrk files and cached in a global LRU keyed by the file path:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeMarksLoader.cpp:313
 if (mark_cache)
 {
 auto key = MarkCache::hash(data_part_storage->getDiskName() + ":" + (fs::path(data_part_storage->getFullPath()) / mrk_path).string());

 if (save_marks_in_cache)
 {
 auto callback = [this] { return loadMarksImpl(); };
 loaded_marks = mark_cache->getOrSet(key, callback);
 }
 else
 {
 loaded_marks = mark_cache->getWithoutMetrics(key);
 if (!loaded_marks)
 loaded_marks = loadMarksImpl();
 }
 }

Note the two paths. getOrSet records hits and misses and inserts the loaded marks into the cache. getWithoutMetrics only looks: it is used when the marks are wanted for a merge or a mutation, which should not evict the query’s working set nor pollute the cache statistics. This is a small but real piece of cache hygiene — background work must not steal the cache from foreground queries.

The cache itself weights each entry by its compressed size plus a fixed overhead:

 cpp
 
 
// src/Storages/MarkCache.h:23
/// Estimate of number of bytes in cache for marks.
struct MarksWeightFunction
{
 /// We spent additional bytes on key in hashmap, linked lists, shared pointers, etc ...
 static constexpr size_t MARK_CACHE_OVERHEAD = 128;

 size_t operator()(const MarksInCompressedFile & marks) const
 {
 return marks.approximateMemoryUsage() + MARK_CACHE_OVERHEAD;
 }
};

The 128-byte constant is the per-entry overhead of the hash map node, the list node, the shared pointer, and the MarksInCompressedFile object itself. For a part with 100 marks, the compressed marks are ~300 bytes and the overhead is 128 — so the accounting is honest about the fact that the bookkeeping dominates for small parts.

The primary key index: one key value per granule

The marks tell you where the bytes are. The primary key index tells you what is in them. It is a separate file, primary.idx, containing the sorting key values of the first row of every granule:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1935
std::shared_ptr<IMergeTreeDataPart::Index> IMergeTreeDataPart::loadIndex() const
{
 auto component_guard = Coordination::setCurrentComponent("IMergeTreeDataPart::loadIndex");
 /// Memory for index must not be accounted as memory usage for query, because it belongs to a table.
 MemoryTrackerBlockerInThread temporarily_disable_memory_tracker;

and the loop that reads it is the whole format — marks_count rows of key_size values, serialized one after another:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1972
 FormatSettings format_settings;
 for (size_t i = 0; i < marks_count; ++i)
 {
 for (size_t j = 0; j < key_size; ++j)
 key_serializations[j]->deserializeBinary(*loaded_index[j], *index_file, format_settings);
 }

marks_count rows of key_size values, serialized one after another. The in-memory form is a set of ColumnPtrs — one column per key column, each holding marks_count values — which is exactly the layout a KeyCondition wants to evaluate a predicate against.

The trimming trick

Now the interesting part. The primary key may be (a, b, c). If a is almost constant within a part — say, a low-cardinality column — then keeping b and c in memory is a waste: a query filtering on b cannot be answered from the index anyway, because a does not narrow anything down. So the index is trimmed:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1895
void IMergeTreeDataPart::optimizeIndexColumns(size_t marks_count, Columns & index_columns) const
{
 if (marks_count == 0)
 {
 chassert(isEmpty());
 return;
 }

 /// Do not optimize index for patch parts because patch parts
 /// use manual index analysis which requires all index columns.
 if (info.isPatch())
 return;

 size_t key_size = index_columns.size();
 Float64 ratio_to_drop_suffix_columns = static_cast<double>((*storage.getSettings())[MergeTreeSetting::primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns]);

 /// Cut useless suffix columns, if necessary.
 if (key_size > 1 && ratio_to_drop_suffix_columns > 0 && ratio_to_drop_suffix_columns < 1)
 {
 for (size_t j = 0; j < key_size - 1; ++j)
 {
 size_t num_changes = 0;
 for (size_t i = 1; i < marks_count; ++i)
 {
 if (0 != index_columns[j]->compareAt(i, i - 1, *index_columns[j], 0))
 ++num_changes;
 }

 if (static_cast<Float64>(num_changes) / static_cast<Float64>(marks_count) >= ratio_to_drop_suffix_columns)
 {
 key_size = j + 1;
 index_columns.resize(key_size);
 break;
 }
 }

The rule: while the fraction of granules in which key column j changes is at least primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns (default 0.9), column j is informative, so keep it and move on. The first column that changes in fewer than 90% of granules is “nearly constant” within this part — the part is probably all one value of it — and then everything after it is dropped.

This is a per-part decision. A part that covers one day of data might have a constant country column and a varying user_id; the index keeps country and drops user_id. Another part, spanning many countries, keeps both. The consequence is that index analysis must know which columns are actually available — and the code accounts for it, treating the missing suffix columns as “constant over the whole universe”, which relaxes the analysis to a safe over-approximation.

The primary index cache

For very large tables, keeping every part’s primary index in memory is not possible, so it can be cached in an LRU instead:

 cpp
 
 
// src/Storages/MergeTree/PrimaryIndexCache.h:29
/** Cache of primary index for MergeTree tables.
 * Primary index is a list of columns from primary key
 * that store first row for each granule of data part.
 */
class PrimaryIndexCache : public CacheBase<UInt128, PrimaryIndex, UInt128TrivialHash, PrimaryIndexWeightFunction>

and the part consults it lazily:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:706
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::getIndex() const
{
 std::scoped_lock lock(index_mutex);

 if (index)
 return index;

 if (auto index_cache = storage.getPrimaryIndexCache())
 return loadIndexToCache(*index_cache);

 index = loadIndex();
 return index;
}

with the key being the disk name plus the part’s path, so that two parts with the same name on different disks do not collide:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:726
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::loadIndexToCache(PrimaryIndexCache & index_cache) const
{
 auto key = PrimaryIndexCache::hash(getDataPartStorage().getDiskName() + ":" + getRelativePathOfActivePart());
 auto callback = [this] { return loadIndex(); };
 return index_cache.getOrSet(key, callback);
}

Now the payoff: two different search algorithms

With marks and the primary index in hand, a query has to answer “which granules can contain a matching row?”. ClickHouse has two algorithms for this, and it picks between them based on the shape of the predicate.

The decision point is whether the predicate describes a continuous range of the sorting key:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2705
 bool key_condition_exact_range = !key_condition_useful || key_condition.matchesExactContinuousRange();
 bool part_offset_condition_exact_range = !part_offset_condition_useful || part_offset_condition->matchesExactContinuousRange();
 bool total_offset_condition_exact_range = !total_offset_condition_useful || total_offset_condition->matchesExactContinuousRange();
 const String & part_name = part->isProjectionPart() ? fmt::format("{}.{}", part->name, part->getParentPart()->name) : part->name;

 if (!key_condition_exact_range || !part_offset_condition_exact_range || !total_offset_condition_exact_range)
 {
 // Do exclusion search, where we drop ranges that do not match

Algorithm 1: binary search (continuous range)

If the predicate describes a single continuous interval of the sorting key — WHERE key >= 100 AND key < 200 — then the matching granules form a contiguous range in the index, and two binary searches find its boundaries:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2801
 /// Invariant: !check_in_range(part_range.begin..searched_left).can_be_true
 /// check_in_range(part_range.begin..searched_right).can_be_true
 /// (part_range.end + 1 is a sentinel out-of-bounds index, such that by definition
 /// check_in_range(x..part_range.end+1).can_be_true is true. The binary search will never
 /// actually call check_in_range with this out-of-bounds index.)
 /// If the condition is not true anywhere, we'll end up with searched_left == part_range.end.
 size_t searched_left = part_range.begin;
 size_t searched_right = part_range.end + 1;

 while (searched_left + 1 < searched_right)
 {
 const size_t middle = (searched_left + searched_right) / 2;
 MarkRange range(0, middle);
 if (check_in_range(range, BoolMask::consider_only_can_be_true).can_be_true)
 searched_right = middle;
 else
 searched_left = middle;
 ++steps;
 }
 result_range.begin = searched_left;
 LOG_TEST(log, "Found (LEFT) boundary mark: {}", searched_left);

check_in_range(0, middle) asks: “could any row in granules [0, middle) match?” The KeyCondition evaluates the predicate against the index values at the two ends of the range and the monotonicity of the key columns. Because the key is sorted, the answer is monotone in middle, so binary search is valid. O(log n) index checks for the whole part.

Algorithm 2: generic exclusion search (everything else)

Now consider WHERE key2 = 5 where key2 is the second column of the sorting key, or WHERE key = 5 OR key = 900000. The matching granules are not contiguous. A binary search cannot be used. ClickHouse falls back to an exclusion search: split the mark range recursively, drop the parts that cannot match.

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:61
void searchUnlimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 for (const auto & initial_range : initial_ranges)
 {
 std::vector<MarkRange> ranges_stack = {initial_range};
 while (!ranges_stack.empty())
 {
 MarkRange range = ranges_stack.back();
 ranges_stack.pop_back();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 appendWithMaxGap(result.ranges, range, search_settings.min_marks_for_seek);

 /// Unlike `ranges`, exact ranges must never absorb the gap between two accepted
 /// ranges: every mark of an exact range has to fully match the condition, while the
 /// skipped marks in between do not.
 if (collect_exact_ranges && exact)
 appendWithMaxGap(result.exact_ranges, range, 0);
 }
 else
 {
 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { ranges_stack.push_back(subrange); });
 }
 }
 }
}

This is a depth-first search over mark ranges. The check returns a BoolMask with two bits: can_be_true and can_be_false.

!can_be_true → the whole range cannot match → drop it.
!can_be_false → every row in the range matches → keep it whole.
otherwise → the range is a mixture → split it into merge_tree_coarse_index_granularity (default 8) subranges and recurse.

Splitting into 8 rather than 2 is deliberate: it is a breadth-ish compromise. Splitting in half is the classic binary split, but each split costs one check_in_range call, and each check is not free — it evaluates a KeyCondition against the index columns at the range boundaries. Eight subranges per step means the search converges in about log_8(n) steps while doing 8× the work per step.

The step budget

The unlimited search has no bound: a pathological predicate could make it check thousands of ranges. So there is a budget:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:119
void searchLimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 std::vector<MarkRange> heap_storage(initial_ranges.begin(), initial_ranges.end());
 std::priority_queue<MarkRange, std::vector<MarkRange>, LessBySize> queue(LessBySize{}, std::move(heap_storage));

 /// Ranges are accepted out of spatial order, so they are collected raw and merged after sorting.
 std::vector<MarkRange> accepted;
 std::vector<MarkRange> accepted_exact;

 while (!queue.empty())
 {
 MarkRange range = queue.top();
 queue.pop();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 accepted.push_back(range);
 if (collect_exact_ranges && exact)
 accepted_exact.push_back(range);
 continue;
 }

 if (result.num_steps + queue.size() > search_settings.max_steps)
 {
 result.reached_step_limit = true;
 accepted.push_back(range);
 continue;
 }

 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { queue.push(subrange); });
 }

Two things change relative to the unlimited version.

A priority queue, largest range first. The budget is spent where it buys the most: splitting a huge range is what removes the most granules, so the biggest range is examined first. The comparator is explicit:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:102
/// Orders the queue of the step-limited search: the largest range comes first.
struct LessBySize
{
 bool operator()(const MarkRange & lhs, const MarkRange & rhs) const
 {
 size_t lhs_size = lhs.end - lhs.begin;
 size_t rhs_size = rhs.end - rhs.begin;
 return std::tie(lhs_size, rhs.begin) < std::tie(rhs_size, lhs.begin);
 }
};

When the budget runs out, the range is accepted whole. This is the important bit: the search never returns an incorrect answer. When result.num_steps + queue.size() > max_steps, the range is pushed into accepted as-is — meaning “we do not know, so read it”. The result is a superset of the true matching granules, so the query is correct; it just reads more data than an unlimited search would have.

That is what the setting says:

 cpp
 
 
// src/Core/Settings.cpp:1936
 DECLARE(UInt64, merge_tree_generic_exclusion_search_max_steps, 0, R"(
When a filter cannot be evaluated as a single continuous range of the primary key, for example when it uses key columns other than the first one, ClickHouse runs an iterative generic exclusion search algorithm over the index marks. The same algorithm is used for the analysis of the text index. This setting limits the number of steps (index checks) the algorithm spends on each data part.

The budget is spent on the largest remaining mark ranges first. When it is exhausted, the ranges that were not fully analyzed are accepted as a whole, so the query stays correct but may read more granules than an unlimited search would select. A lower budget speeds up index analysis at the cost of reading more data. The limit is approximate rather than a strict cap on the analysis cost: the search can exceed it by roughly one round of splitting, and when the part is already divided into many ranges (for example, by the query condition cache), each of them is checked at least once regardless of the limit.

“The default (0) means unlimited steps.” So the default behaviour is exact and unbounded; the budget is an escape hatch for pathological cases.

The seek heuristic

One more knob, and it is about IO, not about correctness. When two accepted ranges are separated by a small gap of marks, it is cheaper to read the gap sequentially than to seek past it: appendWithMaxGap merges the two ranges when the gap is at most max_gap marks wide.

The gap threshold comes from merge_tree_min_rows_for_seek and merge_tree_min_bytes_for_seek, converted into marks by the granularity of the part. Both default to 0, i.e. “never bridge a gap” — but on a spinning disk or an object store with a high per-request cost, setting them turns “read 8 KB from here, seek, read 8 KB from there” into “read 40 KB contiguously”.

Note the asymmetry that the comment points out: exact_ranges are merged with max_gap = 0, always. An exact range is a range where every row matches; bridging a gap into it would claim that the skipped marks match too, which is false. So the gap-bridging optimization applies only to the “may match” ranges, never to the “definitely matches” ones.

Putting it together

 text
 
 
 part on disk
 ┌───────────────────────────────────────────────────────────────┐
 │ granule 0 granule 1 granule 2 granule 3 │
 │ rows 0..8191 rows 8192..16383 rows ... rows ... │
 └───────────────────────────────────────────────────────────────┘
 │ │ │ │
 ▼ ▼ ▼ ▼
 marks: (x0,y0) (x1,y1) (x2,y2) (x3,y3)
 bit-packed, ~3 bytes each, one per column file
 │
 ▼
 primary.idx: key values of the FIRST ROW of each granule
 │
 ▼
 KeyCondition.checkInRange(...)
 │
 ├── predicate is a continuous key range → binary search (2·log₂ n checks)
 │
 └── predicate is anything else → exclusion search (recursive, budgeted)
 │
 ▼
 MarkRanges: the granules that must be read

What this buys you, and what it costs

It buys you a scan that is proportional to the data you asked for, not to the data you stored. A SELECT with a range predicate on a sorted key reads roughly the granules that overlap the range, plus one granule of granularity on each side.

It buys you an index that fits in memory. Three bytes per mark, and the primary key index trimmed to the columns that can actually narrow things down. Without those two tricks, a 10 TB table would need tens of gigabytes of index.

It costs you a fixed granularity of waste. A predicate that matches a single row still reads a whole granule — 10 MB by default. This is the central trade of the whole design, and it is why index_granularity_bytes is a setting you may want to lower for a table with a very selective predicate and small rows.

It costs you an in-memory structure that has to be built and kept. The primary index is loaded per part (or cached), and the marks are loaded per column file that the query touches. Both are cached, and both have their own LRU, but the first query against a cold part pays for them.

It costs you complexity in the analysis. Two algorithms, a budget, a trimming heuristic that makes the analysis conservative in some cases, and the requirement that the analysis be sound (never skip a granule that could match) even when it is not complete.

Seeing it in action

The profile events tell you which algorithm ran:

 sql
 
 
SELECT event, value
FROM system.events
WHERE event IN (
 'IndexBinarySearchAlgorithm',
 'IndexGenericExclusionSearchAlgorithm',
 'IndexGenericExclusionSearchStepLimitReached',
 'FilteringMarksWithPrimaryKeyProcessedMarks',
 'FilteringMarksWithPrimaryKey'
);

FilteringMarksWithPrimaryKeyProcessedMarks is the number of marks examined; FilteringMarksWithPrimaryKey is the number of parts where the primary key was used at all. The ratio between the two, and the ratio of marks read to marks examined, is the “how selective is my index” metric.

And the parts themselves report the granularity they were written with:

 sql
 
 
SELECT
 name,
 rows,
 marks,
 round(rows / marks) AS rows_per_mark,
 formatReadableSize(bytes_on_disk) AS size
FROM system.parts
WHERE active AND rows > 0
ORDER BY marks DESC
LIMIT 10;

A rows_per_mark far below 8192 means the table has fat rows and the byte-based granularity kicked in. A rows_per_mark of exactly 8192 means use_const_adaptive_granularity is on, or index_granularity_bytes was 0 when the part was written.

How ClickHouse Runs a Query Pipeline: Ports, Status Codes and the Executing Graph

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every query engine eventually has to answer the same question: given a graph of operators — read these files, filter these rows, group them, sort them, write them out — in what order do you actually run them?

The textbook answer is the Volcano iterator model: every operator exposes a next() that returns one tuple at a time, and the executor is a nest of while (child.next()) loops. It is beautifully simple and it is also unbelievably slow, because every tuple crosses every operator through a virtual call, and nobody ever gets to see a whole batch.

ClickHouse’s answer is the opposite extreme. Operators exchange blocks — tens of thousands of rows at a time — through ports, and the executor never calls operators in a fixed order. Instead every processor is asked, in effect, “given the state of your neighbours, what should happen next?”, and it answers with one of five status codes. The executor is a scheduler that walks the graph, and the graph tells it where to go.

This is a tour of that machinery: the IProcessor interface, the one-slot mailbox that is a Port, the five status codes, the version counters that let a processor notice that a neighbour changed, and the ExecutingGraph that ties them together.

The building block: `IProcessor`

A processor is a node with zero or more input ports and zero or more output ports. The taxonomy in the header comment of src/Processors/IProcessor.h still describes every processor in the tree — sources, sinks, simple transformations, squashing transformations (DISTINCT, WHERE), accumulating transformations (ORDER BY, GROUP BY), and limiting transformations.

The crucial design decision is that there is no next(). A processor never asks its input for data; it declares whether it needs data, and the executor arranges for it to arrive. And a processor never decides what runs next — it returns a status, and the executor interprets it. Here is the entire status vocabulary:

 cpp
 
 
// src/Processors/IProcessor.h:150
 enum class Status : uint8_t
 {
 /// Processor needs some data at its inputs to proceed.
 /// You need to run another processor to generate required input and then call 'prepare' again.
 NeedData,

 /// Processor cannot proceed because output port is full or not isNeeded().
 /// You need to transfer data from output port to the input port of another processor and then call 'prepare' again.
 PortFull,

 /// All work is done (all data is processed or all output are closed), nothing more to do.
 Finished,

 /// You may call 'work' method and processor will do some work synchronously.
 Ready,

 /// You may call 'schedule' method and processor will return a descriptor.
 /// You need to poll this descriptor and call work() afterwards.
 Async,

 /// Processor wants to add new processors and/or remove finished neighbours.
 /// Update must be obtained by updatePipeline() call.
 UpdatePipeline,
 };

NeedData and PortFull are the two “I am blocked” answers, and they are the whole point. They tell the executor which neighbour has to run next: NeedData means “run my upstream”, PortFull means “run my downstream”. Ready means “I have work, call work() on me”. Async means “I have a file descriptor, poll it”. UpdatePipeline means “I changed the graph”.

The contract for prepare() is spelled out carefully, because it is where most of the subtlety lives:

 cpp
 
 
// src/Processors/IProcessor.h:177
 /** Method 'prepare' is responsible for all cheap ("instantaneous": O(1) of data volume, no wait) calculations.
 *
 * It may access input and output ports,
 * indicate the need for work by another processor by returning NeedData or PortFull,
 * or indicate that processing has finished by returning `Finished`,
 * it may pull data from input ports and push data to output ports.
 *
 * The method is not thread-safe and must be called from a single thread in one moment of time,
 * even for different connected processors.
 *
 * Instead of all long work (CPU calculations or waiting) it should just prepare all required data and return Ready or Async.
 *
 * Thread safety and parallel execution:
 * - no methods (prepare, work, schedule) of single object can be executed in parallel;
 * - method 'work' can be executed in parallel for different objects, even for connected processors;
 * - method 'prepare' cannot be executed in parallel even for different objects,
 * if they are connected (including indirectly) to each other by their ports;
 */
 virtual Status prepare();

Read that last bullet twice. It is the fundamental law of the model: work() may run in parallel with anything; prepare() may not run in parallel with anything it is connected to. The port connections define the happens-before edges of the whole execution, and the executor honours them by construction.

The port: a mailbox with the flags in the pointer

A port is where two processors meet. It is shared state, and it must be readable and writable from two different threads (the producer and the consumer), so it is an atomic pointer to a Data slot — plus three flag bits that say what the slot currently holds.

 cpp
 
 
// src/Processors/Port.h:75
 /// Flags for Port state.
 /// Will store them in least pointer bits.

 /// Port was set finished or closed.
 static constexpr std::uintptr_t IS_FINISHED = 1;
 /// Block is not needed right now, but may be will be needed later.
 /// This allows to pause calculations if we are not sure that we need more data.
 static constexpr std::uintptr_t IS_NEEDED = 2;
 /// Check if port has data.
 static constexpr std::uintptr_t HAS_DATA = 4;

 static constexpr std::uintptr_t FLAGS_MASK = IS_FINISHED | IS_NEEDED | HAS_DATA;
 static constexpr std::uintptr_t PTR_MASK = ~FLAGS_MASK;

Packing the flags into the low bits of the pointer itself is the trick. It means the entire port state — the slot pointer and the three flags — is a single atomic word, so a push and a “set not needed” can be one CAS operation, and there is never a window in which the flags disagree with the pointer.

The little DataPtr class exists to make that safe. The swap() method is the whole implementation of a push or a pull:

 cpp
 
 
// src/Processors/Port.h:117
 uintptr_t ALWAYS_INLINE swap(std::atomic<Data *> & value, std::uintptr_t flags, std::uintptr_t mask) /// NOLINT
 {
 Data * expected = nullptr;
 Data * desired = getPtr(flags | getUInt(data));

 while (!value.compare_exchange_weak(expected, desired))
 desired = getPtr((getUInt(expected) & FLAGS_MASK & (~mask)) | flags | getUInt(data));

 /// It's not very safe. In case of exception after exchange and before assignment we will get leak.
 /// Don't know how to make it better.
 data = getPtr(getUInt(expected) & PTR_MASK);

 return getUInt(expected) & FLAGS_MASK;
 }

The mask argument says which flags the caller intends to change and which ones it wants to preserve from whatever is currently in the word. The CAS loop re-computes desired from expected on every failure, so concurrent flag updates by the two sides are merged rather than lost.

push() and pull() are then thin wrappers:

 cpp
 
 
// src/Processors/Port.h:154
 void ALWAYS_INLINE push(DataPtr & data_, std::uintptr_t & flags)
 {
 flags = data_.swap(data, HAS_DATA, HAS_DATA);

 /// It's possible to push data into finished port. Will just ignore it.
 /// if (flags & IS_FINISHED)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to finished port.");

 /// It's possible to push data into port which is not needed now.
 /// if ((flags & IS_NEEDED) == 0)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which is not needed.");

 if (unlikely(flags & HAS_DATA))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which already has data");
 }

 void ALWAYS_INLINE pull(DataPtr & data_, std::uintptr_t & flags, bool set_not_needed = false)
 {
 uintptr_t mask = HAS_DATA;

 if (set_not_needed)
 mask |= IS_NEEDED;

 flags = data_.swap(data, 0, mask);

 /// It's ok to check because this flag can be changed only by pulling thread.
 if (unlikely((flags & IS_NEEDED) == 0) && !set_not_needed)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which is not needed");

 if (unlikely((flags & HAS_DATA) == 0))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which has no data");
 }

Note the semantics of IS_NEEDED. It is not “the consumer wants data”; it is “the consumer might want data later”. A producer may push into a port that is currently IS_NEEDED == 0 — the data simply sits in the slot — but it may not pull from a port that is not needed, because pulling implies “give me the next thing” and the consumer has explicitly said it does not want more right now.

canPush() is the producer’s gate:

 cpp
 
 
// src/Processors/Port.h:478
 bool ALWAYS_INLINE canPush() const
 {
 assumeConnected();
 auto flags = state->getFlags();
 return (flags & State::IS_NEEDED) && ((flags & State::HAS_DATA) == 0);
 }

Both conditions matter: if the consumer said “not needed”, the producer must wait, and if the slot already has data the producer must wait, because the port holds exactly one chunk.

This “not needed” handshake is what makes the pull side of the model honest. The consumer says input.setNotNeeded() while it is busy, and the producer’s canPush() goes false, so the producer stops. That is how a LIMIT transform can stop a system.numbers_mt source from producing infinity: it closes the port.

 sql
 
 
-- The Limit transform closes its input port once the limit is reached,
-- so the source stops generating rows instead of the query reading forever.
SELECT * FROM system.numbers_mt WHERE number = 1000000 LIMIT 1;

The shape of the resulting graph is visible with EXPLAIN PIPELINE, which prints the processors and the ports the executor will walk:

 sql
 
 
EXPLAIN PIPELINE
SELECT number FROM numbers(10) WHERE number > 5 LIMIT 2;

`prepare()` and `work()`

The canonical implementation of the two-phase protocol is ISimpleTransform, the base class of nearly every operator in the tree (filters, expressions, DISTINCT, window functions, LIMIT BY…).

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:23
ISimpleTransform::Status ISimpleTransform::prepare()
{
 /// Check can output.

 if (output.isFinished())
 {
 input.close();
 onFinish();
 return Status::Finished;
 }

 if (!output.canPush())
 {
 input.setNotNeeded();
 return Status::PortFull;
 }

 /// Output if has data.
 if (has_output)
 {
 output.pushData(std::move(output_data));
 has_output = false;

 if (!no_more_data_needed)
 return Status::PortFull;

 }

 /// Stop if don't need more data.
 if (no_more_data_needed)
 {
 input.close();
 output.finish();
 return Status::Finished;
 }

 /// Check can input.
 if (!has_input)
 {
 if (input.isFinished())
 {
 output.finish();
 onFinish();
 return Status::Finished;
 }

 input.setNeeded();

 if (!input.hasData())
 return Status::NeedData;

 input_data = input.pullData(set_input_not_needed_after_read);
 has_input = true;

 if (input_data.exception)
 /// No more data needed. Exception will be thrown (or swallowed) later.
 input.setNotNeeded();
 }

 /// Now transform.
 return Status::Ready;
}

Every branch here is a decision about who runs next:

 text
 
 
 output.isFinished() -> input.close(); output.finish() -> Finished
 !output.canPush() -> input.setNotNeeded() -> PortFull
 has_output -> output.pushData() -> PortFull
 no_more_data_needed -> input.close(); output.finish() -> Finished
 input.isFinished() -> output.finish() -> Finished
 !input.hasData() -> input.setNeeded() -> NeedData
 otherwise -> pullData(); transform() -> Ready

Note that prepare() does push: output.pushData(...) and input.pullData(...) both happen inside prepare(). The division of labour is not “prepare moves no data”; it is “prepare does no expensive work”. Pulling a chunk is O(1) — it is a pointer swap. Actually computing on it is not, and that is what work() is for:

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:86
void ISimpleTransform::work()
{
 if (input_data.exception)
 {
 transform(input_data.exception);

 output_data = std::move(input_data);
 has_input = false;
 has_output = true;
 return;
 }

 try
 {
 transform(input_data.chunk, output_data.chunk);
 }
 catch (DB::Exception &)
 {
 output_data.exception = std::current_exception();
 has_output = true;
 has_input = false;
 return;
 }

 has_input = !needInputData();

 if (!skip_empty_chunks || output_data.chunk)
 has_output = true;

 if (has_output && !output_data.chunk && !getOutputPort().getHeader().empty())
 /// Support invariant that chunks must have the same number of columns as header.
 output_data.chunk = Chunk(getOutputPort().getHeader().cloneEmpty().getColumns(), 0);
}

Three things are worth noting.

Exceptions are data. A failure in the upstream processor is pushed into the port as an exception_ptr alongside the chunk, not thrown across the thread boundary. work() picks it up, hands it to transform(), and pushes it on. The exception then travels downstream with the stream, which means the error surfaces in the right place and every processor upstream of the failure gets to clean up.

work() cannot block. That is why the Status::Async case exists: a processor that needs to wait for a socket or a file descriptor returns Async from prepare() and hands out an fd from schedule().

The empty-chunk invariant. Every chunk that crosses a port must have exactly as many columns as the port’s header, even if it has zero rows. ISimpleTransform enforces it at the bottom of work() by synthesising an empty chunk with the right shape.

Version counters: how a processor learns that a neighbour changed

The executor must know which processors to wake up. Polling every processor in the graph after every chunk would be O(V) per chunk, which for a twenty-node graph and a million chunks is a lot of pointless work.

ClickHouse’s answer is a version counter per connection, with a twist: the counter lives in the UpdateInfo of the port, and it is compared against the value the neighbour last saw.

 cpp
 
 
// src/Processors/Port.h:34
 struct UpdateInfo
 {
 using UpdateList = VectorWithMemoryTracking<void *>;

 UpdateList * update_list = nullptr;
 void * id = nullptr;
 UInt64 version = 0;
 UInt64 prev_version = 0;

 void inline ALWAYS_INLINE update()
 {
 if (version == prev_version && update_list)
 update_list->push_back(id);

 ++version;
 }

 void inline ALWAYS_INLINE trigger() { prev_version = version; }
 };

update() is called whenever the port’s state changes (a push, a pull, a setNeeded). It bumps version, and — this is the interesting part — if version == prev_version, meaning nobody has consumed the change yet, it appends the port’s identity to a list of “ports that changed”. trigger() marks the change as seen by setting prev_version = version.

So the protocol is: a port that changes while nobody was looking records itself on a list; the executor drains that list and re-prepares exactly those neighbours. If a port changes twice before the neighbour is re-prepared, the neighbour is only told once — it does not need to know how many times it changed, only that it did.

This is the mechanism that replaced the old “wake up the neighbour immediately” design, and it is why prepare() can be called on a subset of the graph instead of on all of it.

The `ExecutingGraph`: nodes, edges, and who owns what

The executor’s view of the pipeline is ExecutingGraph, in src/Processors/Executors/Runtime/V1/. It holds one Node per processor and two kinds of Edge per connection: a direct edge from an output port to the consuming processor’s input port, and a backward edge from an input port back to the producing processor’s output port.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:31
 /// Edge represents connection between OutputPort and InputPort.
 /// For every connection, two edges are created: direct and backward (it is specified by backward flag).
 struct Edge
 {
 Edge(Node * to_, bool backward_,
 InputPort * input_port_, OutputPort * output_port_,
 VectorWithMemoryTracking<void *> * update_list)
 : to(to_), backward(backward_)
 , input_port(input_port_), output_port(output_port_)
 {
 update_info.update_list = update_list;
 update_info.id = this;
 }

 /// Node this edge points to.
 /// It is the node with output_port for direct edge or the node with input_port for backward edge.
 Node * to = nullptr;
 bool backward;
 /// Ports of this connection. Both pointers are set regardless of direction:
 /// for a direct edge (out -> in) input_port belongs to `to`; for a back edge the reverse.
 InputPort * input_port = nullptr;
 OutputPort * output_port = nullptr;

 /// Edge version is increased when port's state is changed (e.g. when data is pushed). See Port.h for details.
 /// To compare version with prev_version we can decide if neighbour processor need to be prepared.
 Port::UpdateInfo update_info;
 };

The Node carries a status with a very specific ownership protocol:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:62
 /// Status for processor.
 /// Can be owning or not. Owning means that executor who set this status can change node's data and nobody else can.
 enum class ExecStatus : uint8_t
 {
 Idle, /// prepare returned NeedData or PortFull. Non-owning.
 Preparing, /// some executor is preparing processor, or processor is in task_queue. Owning.
 Executing, /// prepare returned Ready or Async and task is executing. Owning.
 Finished, /// prepare returned Finished. Non-owning.
 };

“Owning” is the key word. Two threads may never call prepare() on the same processor, and may never call prepare() on connected processors at the same time. The status is how the graph enforces that: a node in Preparing or Executing is owned by exactly one thread, and the per-node std::mutex status_mutex is taken when ownership is transferred.

The traversal itself is updateNode(). It starts from one processor and spreads along the edges, re-preparing whoever was affected:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:524
ExecutingGraph::UpdateNodeStatus ExecutingGraph::updateNode(IProcessor & initial, Queue & queue, Queue & async_queue, UpdateNodeBuffers & buffers)
{
 Processors delayed_destruction;
 auto & updated_edges = buffers.updated_edges;
 auto & updated_processors = buffers.updated_processors;
 auto & pending_expansion = buffers.pending_expansion;

 /// The buffers may be left non-empty by a previous call that exited early or threw.
 buffers.reset();

 std::shared_lock read_lock(nodes_mutex);

 /// Traversal starts from the node related to updated processor
 updated_processors.push_back(processors_map.at(&initial));

 while (!updated_processors.empty() || !updated_edges.empty())
 {
 std::optional<std::unique_lock<std::mutex>> stack_top_lock;

 if (updated_processors.empty())
 {
 auto * edge = updated_edges.front();
 updated_edges.pop_front();

 /// Here we have ownership on edge, but node can be concurrently accessed.

 auto & node = *edge->to;

 std::unique_lock lock(node.status_mutex);

 ExecutingGraph::ExecStatus status = node.status;

 if (status != ExecutingGraph::ExecStatus::Finished)
 {
 if (edge->backward)
 node.updated_output_ports.push_back(edge->output_port);
 else
 node.updated_input_ports.push_back(edge->input_port);

 if (status == ExecutingGraph::ExecStatus::Idle)
 {
 node.status = ExecutingGraph::ExecStatus::Preparing;
 updated_processors.push_front(edge->to);
 stack_top_lock = std::move(lock);
 }
 else
 edge->to->processor()->onUpdatePorts();
 }
 }

Two work lists are at work here, and the distinction is worth spelling out. updated_processors is a queue of nodes to call prepare() on. updated_edges is a queue of connections whose port state changed and whose far side therefore needs to know. The loop interleaves them: an edge whose far side is Idle promotes that node to Preparing and puts it on the processor queue; if the far side is already Preparing or Executing, the edge is simply recorded in that node’s updated_input_ports / updated_output_ports list and the node will pick it up on its next prepare().

That is the “which processor am I at” trick in its modern form: there is no cursor. There is no “current processor” that remembers where it was. Instead, each processor holds the set of ports that changed since it was last prepared, and the graph walks edges until it reaches a fixed point.

The core of the traversal is the prepare() call and the dispatch on its result:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:621
 switch (*node.last_processor_status)
 {
 case IProcessor::Status::NeedData:
 case IProcessor::Status::PortFull:
 {
 node.status = ExecutingGraph::ExecStatus::Idle;
 if (spill_state.memory_reservation)
 {
 /// A request may have arrived while this node was owned by normal execution.
 for (auto * spillable : processor.getSpillables())
 if (spill_state.scheduleSpill(node, spillable, queue))
 break;
 }
 break;
 }
 case IProcessor::Status::Finished:
 {
 node.status = ExecutingGraph::ExecStatus::Finished;
 if (last_status != IProcessor::Status::Finished)
 spill_state.unregisterSpillables(node);
 accountFinishedProcessorInGroup(*node.processor_iter);
 break;
 }
 case IProcessor::Status::Ready:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 queue.push(node.processor());
 break;
 }
 case IProcessor::Status::Async:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 async_queue.push(node.processor());
 break;
 }
 case IProcessor::Status::UpdatePipeline:
 {
 need_update_pipeline = true;
 pending_expansion.push_back(current);
 break;
 }
 }

Ready puts the processor on the work queue, to be picked up by a worker thread; NeedData/PortFull put the node back to Idle, meaning “somebody else must move first”. Finished retires the node for good. And then the edges that this processor disturbed get re-armed:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:666
 /// If you wonder why edges are pushed in reverse order,
 /// it is because updated_edges is a stack, and we prefer to get from stack
 /// input ports firstly, and then outputs, both in-order.
 ///
 /// Actually, there should be no difference in which order we process edges.
 /// However, some tests are sensitive to it (e.g. something like SELECT 1 UNION ALL 2).
 /// Let's not break this behaviour so far.

 for (auto it = node.post_updated_output_ports.rbegin(); it != node.post_updated_output_ports.rend(); ++it)
 {
 auto * edge = static_cast<ExecutingGraph::Edge *>(*it);
 updated_edges.push_front(edge);
 edge->update_info.trigger();
 }

edge->update_info.trigger() is the prev_version = version from Port.h, and it is called after the edge is queued — the ordering matters, because trigger() says “the neighbour has now seen this change” and the queue entry is what makes it true.

The graph can change shape at runtime

The last status, UpdatePipeline, is unusual: a processor can, while running, decide that it needs more processors. The classic case is external sort: MergeSortingTransform starts as a single-input, single-output processor, and when it decides to spill to disk it must insert a BufferingToFileSink and a BufferingFromFileSource into the middle of its own pipeline.

The mechanism is updatePipeline(), which returns a PipelineUpdate:

 cpp
 
 
// src/Processors/IProcessor.h:262
 struct PipelineUpdate
 {
 Processors to_add;
 Processors to_remove;
 };
 virtual PipelineUpdate updatePipeline();

The graph handles the insertion by rebuilding the edges of every node and scheduling the affected ones for re-preparation. The interesting part is the error handling, which encodes a hard-won lesson about allocation failures:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:324
 try
 {
 return updatePipelineImpl(stack, cur_node, update);
 }
 catch (...)
 {
 expansion_failed = true;
 /// The ports of the pipeline's processors point at the new processors, and the pipeline is read after the graph is gone.
 std::lock_guard guard(processors_mutex);
 for (auto it = update.to_add.begin(); it != update.to_add.end();)
 {
 auto next = std::next(it);
 if (std::ranges::find(*processors, *it) == processors->end())
 processors->splice(processors->end(), update.to_add, it);
 it = next;
 }
 throw;
 }

The processor has already connected its ports to the new processors before updatePipeline() is even called. So if the graph then fails to record them — say, addNode throws MEMORY_LIMIT_EXCEEDED — the ports point at processors the graph does not know about, and the graph is permanently inconsistent. The handler keeps the new processors alive (so the ports stay valid) and sets expansion_failed, which makes every later expansion bail out instead of walking into the inconsistency.

Trade-offs

What this buys you.

The model is fully data-parallel by construction. Any node of the graph may be scheduled on any thread, because the only shared state between processors is a port, and a port is a single atomic word. There is no notion of “this operator belongs to this thread”, which is what lets ClickHouse run a SELECT over a MergeTree with 16 streams on 16 cores without any of the 16 streams knowing about the other 15.

The graph also reshapes itself. External sort, GLOBAL JOIN, subquery materialisation, and LIMIT pushdown all insert or remove processors while the query is running, and none of them require a separate “compile the plan” phase.

Finally, the split between prepare() (cheap, may not run concurrently with connected peers) and work() (expensive, may run concurrently with everything) is precisely the split that lets the executor overlap computation without any locking on the hot path.

What it costs.

prepare() cannot run in parallel with connected processors. For a wide graph — say a UNION ALL of twenty subqueries, each with a dozen operators — this means the scheduling phase is effectively serialised per connection. That is why the pipeline shape matters so much: every extra operator in a chain adds a serialisation point between the parallel parts. This is also why Pipe::resize() has a comment about splitting the resize processor when multiple threads are allocated:

 cpp
 
 
// src/QueryPipeline/Pipe.cpp:781
 /// Performance bottleneck identified: Severe lock contention for ExecutingGraph::Node::status_mutex.
 /// Issues observed:
 /// 1. Increased latency of ExecutingGraph::updateNode, lengthening overall query latency.
 /// 2. Unnecessary CPU cycle consumption in native_queued_spin_lock_slowpath (kernel).
 /// 3. Decreased CPU utilization.
 ///
 /// Proposed solution: Split ResizeProcessor when multiple threads are allocated to execute a query.

The second cost is the per-node status_mutex. Every edge traversal takes it. On a hot path with many small chunks this is measurable, and the comment above is the record of someone measuring it.

The third cost is that the model is hard to reason about in the presence of prepare() side effects. ISimpleTransform::prepare() pushes data; LimitTransform::prepare() closes ports; SortingTransform::prepare() can return UpdatePipeline and rebuild the graph. There is no single “execute this operator” entry point — which is exactly why ClickHouse’s test infrastructure is built around assembling a whole Pipe.

How ClickHouse Runs a Query Pipeline on N Threads: Task Queues, Work Stealing and Growing the Pool

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A query plan in ClickHouse is not one pipeline. It is a stack of pipelines: read from storage, filter, project, aggregate, sort, format. Each stage produces many parallel streams, and consecutive stages are connected by ResizeProcessors that shuffle chunks between them.

Running that means answering three questions at once:

Which processor should run next? (the graph — see the previous article on ExecutingGraph)
Which thread should run it?
How many threads should exist at all?

The third question is the interesting one. A query that reads one small table does not want sixteen threads. A query that scans a hundred partitions does. And a query that has been running for thirty seconds may need fewer threads than it started with, because the server is under load from other queries.

ClickHouse’s answer, in src/Processors/Executors/Runtime/V1/, is a system that starts with one thread, measures demand as the pipeline runs, and grows the pool only when there is genuinely parallel work that idle threads cannot absorb.

Two kinds of parallelism

Before the machinery, it is worth being precise about what “parallel” means here, because ClickHouse does two different things and they are often conflated.

Data parallelism is the obvious one: SELECT count() FROM t reads from N partitions, each partition is its own stream, and N streams are processed at once. This is a property of the plan — the number of streams is fixed when the plan is built.

Pipeline parallelism is the subtler one. Even in a pipeline with a single stream, the stage that is reading from disk is not blocked while the stage that is formatting the output is busy. A chunk is handed from one processor to the next, and while the consumer works on chunk k, the producer is already building chunk k+1.

The executor supports both, and the interesting bit is how it decides how many threads to use:

 sql
 
 
-- Sixteen streams of data parallelism; the number of threads that actually
-- run them is decided at execution time, not by the plan.
SELECT count() FROM numbers_mt;

 sql
 
 
-- The ceiling. The pool starts at one thread and grows only if there is
-- parallel work that idle threads cannot absorb.
SETTINGS max_threads = 16;
SELECT count() FROM numbers_mt;
``` Note that the two are not the same thing: a pipeline
with 16 streams needs 16 threads to be *fully* parallel, but it will still
make progress — just less of it — on one thread.

## The shape of the executor

`PipelineExecutor` owns three things:

```cpp
// src/Processors/Executors/Runtime/V1/PipelineExecutor.h:49
private:
 /// Outlives graph registrations, including when construction fails.
 QueryStatusPtr process_list_element;
 ExecutingGraphPtr graph;

 ExecutorTasks tasks;

 /// Concurrency control related
 SlotAllocationPtr cpu_slots;
 AcquiredSlotPtr single_thread_cpu_slot; // cpu slot for single-thread mode to work using executeUntil()
 std::unique_ptr<ThreadPool> pool;
 std::mutex spawn_mutex;

graph is the processor graph (the subject of the previous article). tasks is the scheduler: the queues, the per-thread contexts, the counters. cpu_slots and pool are the resources: how many threads are allowed to exist, and the threads themselves.

The entry point is execute(), which is unremarkable apart from its comment about output formats:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:231
void PipelineExecutor::execute(size_t num_threads, bool concurrency_control)
{
 /// `QueryStatus` may release its reservation before this executor is destroyed.
 SCOPE_EXIT({ detachSpillables(); });
 if (process_list_element && !process_list_element->checkTimeLimit())
 cancel(IProcessor::CancelReason::CancelledByTimeout);

 num_threads = std::max<size_t>(num_threads, 1);

and executeImpl() is where the threads actually appear:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:783
void PipelineExecutor::executeImpl(size_t num_threads, bool concurrency_control)
{
 initializeExecution(num_threads, concurrency_control);

 try
 {
 if (num_threads > 1)
 {
 {
 std::lock_guard lock(spawn_mutex);
 // Start at least one thread, could block to acquire the first CPU slot
 spawnThreads(cpu_slots->acquire());
 }
 tasks.processAsyncTasks();
 pool->wait();
 }
 else
 {
 auto slot = cpu_slots->acquire();
 tasks.upscale(slot->slot_id);
 executeSingleThread(slot->slot_id, WorkloadResources(slot.get(), process_list_element));
 }
 }

Note pool->wait(): the master thread does not execute pipeline work in the multi-threaded case. It spawns worker threads and then parks in processAsyncTasks(), which is the epoll loop for processors that returned Status::Async. The master thread is a scheduler, not a worker — and that distinction matters for deadlock avoidance, which is why the master is never downscaled:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:591
 // The first thread is called master thread.
 // It is NOT the thread that handles async tasks (unless query has max_threads=1).
 // Master thread is different from other threads due to special role in scheduling:
 // 1. During query start, master thread spawns worker threads, so starting it fast is important.
 // 2. It should never be downscaled to avoid query deadlock (0 threads to process tasks), although it can be preempted.
 // 3. When using ConcurrencyControl, master thread is granted immediately and may not request CPU slot (see 'fair_round_robin').
 // 4. When using resource scheduler, master thread may use different resource link than worker threads.
 const auto master_threads = 1uz;

Task queues with work stealing

The scheduler’s central data structure is TaskQueue, and it is per-thread-first:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/TasksQueue.h:17
/// A set of task queues for multithreaded processing.
/// Every threads has its dedicated queue and uses it preferably.
/// When there are no tasks left in dedicated queue it steals tasks from other threads
template <typename Task>
class TaskQueue
{
public:
 void init(size_t num_threads) { queues.resize(num_threads); }

 /// Push a task into thread's dedicated queue
 void push(Task * task, size_t thread_num)
 {
 queues[thread_num].push(task);
 use_queues = std::max(use_queues, thread_num + 1);
 ++num_tasks;
 }

 /// Returns thread number to pop task from.
 /// First it check dedicated queue, and only if it is empty, it steal from other threads
 size_t getAnyThreadWithTasks(size_t from_thread)
 {
 if (num_tasks == 0)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");

 if (from_thread >= use_queues)
 from_thread = 0;

 for (size_t i = 0, max_iterations = use_queues; i < max_iterations; ++i)
 {
 if (!queues[from_thread].empty())
 return from_thread;

 ++from_thread;
 if (from_thread == use_queues)
 {
 use_queues--; // lazy update: less expensive than updating on every pop
 from_thread = 0;
 }
 }

 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");
 }

Each thread has a private FIFO. A thread that has work prefers its own queue; only when it is empty does it walk the other threads’ queues looking for something to steal. use_queues is a lazily-maintained upper bound on how many queues are non-empty, so the scan is O(number of active threads) and not O(total threads).

The per-thread affinity matters for cache reasons: a task that runs on thread 3 tends to produce data that thread 3 will consume next, and the data stays in that core’s L1/L2.

There are three queues, not one:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.h:43
 /// Queue with pointers to tasks. Each thread will concurrently read from it until finished flag is set.
 /// Stores processors need to be prepared. Preparing status is already set for them.
 TaskQueue<IProcessor> task_queue;

 /// Async tasks should be processed with higher priority, but also require task stealing logic.
 /// So we have a separate queue specifically for them.
 TaskQueue<IProcessor> fast_task_queue;
 std::atomic_bool has_fast_tasks = false; // Required only to enable local task optimization
 std::atomic_bool spill_requested = false;

 /// Queue which stores tasks where processors returned Async status after prepare.
 /// If multiple threads are used, main thread will wait for async tasks.
 /// For single thread, will wait for async tasks only when task_queue is empty.
 PollingQueue async_task_queue;

fast_task_queue holds processors that came back from an asynchronous operation — a socket that became readable, a file that finished being written. They get priority because they are latency-sensitive: the data is already there and the client is waiting.

async_task_queue is different in kind: it is not a task queue at all but an epoll wrapper that owns the file descriptors.

The local-task optimization

Here is the single most important performance trick in the executor, and it is small. When a processor finishes, the next processor to run is often known immediately — it is the downstream consumer of the chunk that was just produced. Rather than pushing it onto a shared queue (where another thread would have to wake up, take the mutex, and pop it), the executor hands it directly to the thread that is already running.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:192
size_t ExecutorTasks::pushTasks(Queue & queue, Queue & async_queue, ExecutionThreadContext & context)
{
 /// Take local task from queue if has one.
 if (!queue.empty() && !has_fast_tasks.load(std::memory_order_relaxed)
 && context.num_scheduled_local_tasks < ExecutionThreadContext::max_scheduled_local_tasks)
 {
 ++context.num_scheduled_local_tasks;
 context.setTask(queue.front());
 queue.pop();
 }
 else
 {
 context.num_scheduled_local_tasks = 0;
 context.setTask(nullptr);
 }

and the constant that bounds it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutionThreadContext.h:55
 /// There is a performance optimization that schedules a task to the current thread, avoiding global task queue.
 /// Optimization decreases contention on global task queue but may cause starvation.
 /// See 01104_distributed_numbers_test.sql
 /// This constant tells us that we should skip the optimization
 /// if it was applied more than `max_scheduled_local_tasks` in a row.
 constexpr static size_t max_scheduled_local_tasks = 128;

128 in a row. The comment names the failure mode: starvation. If one thread keeps grabbing work locally while other threads sit idle with nothing to do, the pipeline runs on one core while the other max_threads - 1 cores idle. The counter is reset as soon as the local optimization is skipped, so a thread that has been running alone for 128 tasks will put the next one on the global queue and let somebody else take it.

The referenced test, 01104_distributed_numbers_test.sql, is a good illustration of why this matters: SELECT * FROM numbers(1000000000) with distributed_product_mode and a LIMIT — the limit transform stops the source, and if the source is not rebalanced the other threads never wake up.

The has_fast_tasks check in the condition is the second half of the guard: if there is an asynchronous task waiting, it takes priority over a local task, because asynchronous tasks are latency-critical.

Measuring demand: `pushTasks` returns a number

Now the interesting part. pushTasks does not just enqueue work — it measures how much parallel work was created, and returns it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:208
 if (!queue.empty() || !async_queue.empty())
 {
 std::unique_lock lock(mutex);

 /// Measure the parallelism of the push: how many threads could run the new tasks
 /// concurrently. Used by `PipelineExecutor` to size the slot-allocation ceiling via
 /// `ISlotAllocation::setMax` on the pipeline's `cpu_slots` handle.
 const size_t pushed = queue.size() + async_queue.size();

and then, at the end:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:232
 /// Capture state under the lock — tryWakeUp may unlock it internally to wake a
 /// thread, after which `total_slots` could change.
 const size_t idle_before = threads_queue.size();
 const size_t total_slots_now = total_slots;
 const size_t num_threads_now = num_threads; // const after init but read for clarity

 /// Wake up at least one thread that will wake up other threads if required
 tryWakeUpAnyOtherThreadWithTasks(context, lock);

 /// Return the number of additional pipeline threads the caller should target: pushed
 /// tasks that idle threads can't cover, clamped to remaining pipeline-width headroom
 /// (`num_threads - total_slots`). `PipelineExecutor` adds this delta to the
 /// `cpu_slots->setMax(...)` target so the slot allocator grows in step with demand.
 const size_t coverable = std::min(pushed, idle_before);
 const size_t uncovered = pushed - coverable;
 const size_t headroom = (num_threads_now > total_slots_now) ? (num_threads_now - total_slots_now) : 0;
 return std::min(uncovered, headroom);
 }
 return 0; // No new tasks -- no need for new threads

The arithmetic is: pushed tasks were created. idle threads can absorb that many without being woken up. The remainder is genuine demand for new threads — clamped to the pipeline’s own width ceiling (num_threads, from max_threads), because the plan has a fixed number of streams and there is no point running more threads than there is parallelism.

This is the demand signal. It is computed from two observable facts — how many tasks were pushed, how many threads are parked — and no profiling or heuristics.

The executor then accumulates that demand and acts on it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:492
 /// Upscaling: `pushTasks` tells us how many additional threads would usefully
 /// parallelize the tasks it just enqueued. Publish our spawn_count into the shared
 /// `pending_demand` counter; the thread that actually holds `spawn_mutex` drains it
 /// so no demand is lost when multiple threads push concurrently.
 if (pool && spawn_count > 0)
 pending_demand.fetch_add(spawn_count, std::memory_order_relaxed);

 // Drain pending_demand under `spawn_mutex`. A thread that fails `try_lock`
 // must not exit while `pending_demand > 0` -- otherwise its add could race
 // with the owner's final `exchange(0)` and be left stranded after the owner
 // unlocks. The outer loop holds the thread until either it drains the demand
 // itself or observes the counter return to zero (meaning the owner caught it).
 while (pool && pending_demand.load(std::memory_order_acquire) > 0)
 {
 if (!spawn_mutex.try_lock())
 {
 std::this_thread::yield();
 continue;
 }
 try
 {
 std::lock_guard lock(spawn_mutex, std::adopt_lock);
 while (true)
 {
 size_t accumulated = pending_demand.exchange(0, std::memory_order_acq_rel);
 if (accumulated == 0)
 break;
 size_t target = std::min<size_t>(max_pipeline_threads, desired_threads + accumulated);
 if (target > desired_threads)
 {
 desired_threads = target;
 cpu_slots->setMax(target);
 }
 }
 spawnThreads({});
 }

This is a small and rather careful piece of concurrent code. The pending_demand counter is atomic so that threads which fail to take spawn_mutex do not lose their demand. But a naive fetch_add would race with the owner’s exchange(0): thread A adds 1, the owner exchanges and gets 0, the owner unlocks, thread A’s 1 sits in the counter forever. The outer while loop is the fix — a thread that cannot get the mutex keeps spinning (yielding) until either it drains the demand itself or sees the counter back at zero, which means the owner already caught it.

The target computation is the second piece of policy: demand is accumulated into desired_threads, capped at max_pipeline_threads (the plan’s width), and applied by raising the slot allocator’s ceiling with cpu_slots->setMax(target).

spawnThreads then converts slots into actual OS threads:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:743
void PipelineExecutor::spawnThreads(AcquiredSlotPtr slot)
{
 while (cpu_slots)
 {
 if (!slot)
 slot = cpu_slots->tryAcquire();
 if (!slot)
 return;

 size_t thread_num = slot->slot_id;

 /// Count of threads in use should be updated for proper finish() condition.
 /// `upscale` returns the remaining headroom for spawning. Even when >0 we spawn
 /// one thread per iteration and re-enter `upscale` after the next `tryAcquire`.
 const size_t remaining_capacity = tasks.upscale(thread_num);

 /// Start new thread
 pool->scheduleOrThrowOnError([this, thread_num, thread_group = CurrentThread::getGroup(), my_slot = std::move(slot)]
 {
 ThreadGroupSwitcher switcher(thread_group, ThreadName::QUERY_ASYNC_EXECUTOR);

 try
 {
 executeSingleThread(thread_num, WorkloadResources(my_slot.get(), process_list_element));
 }
 catch (...)
 {
 /// In case of exception from executor itself, stop other threads.
 finish();
 tasks.getThreadContext(thread_num).setException(std::current_exception());
 }
 });

The tryAcquire is the gate: the slot allocator may refuse to hand out a slot if the server is out of CPU budget. If it refuses, no thread is spawned and the pipeline simply runs with fewer threads. The thread count is not max_threads; it is max_threads subject to what the scheduler is willing to give you.

CPU slots: `max_threads` is a ceiling, not a promise

The slot allocation is where the concurrency-control policy lives. There are four cases, and the comment enumerates them:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:605
 // There are a few possible modes of concurrency control.
 // 1) If we have `CREATE RESOURCE cpu (MASTER THREAD, WORKER THREAD)`:
 // all thread should be allocated through scheduler using the same resource link
 // 2) If we have `CREATE RESOURCE cpu (WORKER THREAD)`:
 // master thread uses "free" noncompeting CPU slot, the other threads are allocated through resource scheduler
 // 3) If we have two resources `CREATE RESOURCE master_cpu (MASTER THREAD)` and `CREATE RESOURCE worker_cpu (WORKER THREAD)`:
 // all thread should be allocated through scheduler using different resource links
 // 4) If we have no cpu-related resources:
 // the ConcurrencyControl class is used instead of resource scheduler

In the default case (case 4) the allocation is deliberately lazy:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:672
 else
 {
 /// Allocate CPU slots from concurrency control with guaranteed master thread slot.
 /// Default path (lazy_allocation=true): start at max=master_threads and grow on
 /// demand via setMax from the upscaling block. Zero-waste per idle pipeline.
 /// Rollback (lazy_allocation=false): pre-branch eager allocation — request the
 /// full ceiling upfront and never call setMax.
 SlotCount initial_max = lazy_allocation ? master_threads : num_threads;
 return ConcurrencyControl::instance().allocate(master_threads, initial_max);
 }

“Zero-waste per idle pipeline” is the point. A server running a thousand concurrent queries, each of which is a small scan, does not want a thousand times max_threads slots reserved. It wants one slot each, growing if the query turns out to be wide.

The lazy flag is read exactly once, and the reason is a real bug:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:696
 /// Read the flag ONCE. A config reload mid-initialization would otherwise split a
 /// single query between lazy and eager strategies (e.g. allocate lazy with max=1 but
 /// then initialize desired_threads=num_threads, stranding subsequent setMax as a no-op).
 const bool lazy_allocation = ConcurrencyControl::instance().getLazyAllocation();

A config reload between “allocate the slots” and “set desired_threads” would leave the query in a state where setMax is a no-op and the pipeline never grows. One read, one strategy, for the lifetime of the query.

Preemption and downscaling

Threads can also be taken away. Under the resource scheduler with preemption enabled, a thread’s CPU lease is renewed periodically, and the renewal can fail:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:539
 if (resources.isCPULeaseRenewNeeded()) // Check if preemption is enabled (see `cpu_slot_preemption` server setting)
 {
 try
 {
 // Preemption point. Renewal could block execution due to CPU overload.
 // It may trigger callbacks to tasks.preempt() and tasks.resume()
 if (!resources.renewCPULease())
 {
 tasks.downscale(resources.getCPULeaseSlotId());
 break; // Downscaling. Unable to renew the lease - thread should stop (but could be rerun later).
 }

The renewal is checked at most every millisecond:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:119
 bool isCPULeaseRenewNeeded()
 {
 if (!lease)
 return false;
 UInt64 now_ns = clock_gettime_ns();
 if (now_ns - last_renew_ns < 1000000) // 1 ms: do not renew too often
 return false;
 last_renew_ns = now_ns;
 return true;
 }

Downscaling is not “kill the thread” — it is “let this thread stop when it finishes its current task, and do not schedule more onto it”:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:334
void ExecutorTasks::downscale(size_t slot_id)
{
 std::lock_guard lock(mutex);

 if (slot_id >= slot_count.size() || slot_count[slot_id] == 0)
 return;
 --slot_count[slot_id];

 if (slot_id + 1 == use_threads)
 {
 // The highest slot was downscaled, find new highest
 for (size_t i = use_threads; i > 0; --i)
 {
 if (slot_count[i - 1] > 0)
 {
 use_threads = i;
 break;
 }
 }
 }
}

Preemption is a third thing: the slot is released without downscaling, so the pipeline’s width is unchanged but the thread is not running. If the preempted thread was holding a task, that task is pushed back to the queue so another thread can steal it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:356
void ExecutorTasks::preempt(size_t slot_id)
{
 std::unique_lock lock(mutex);
 --total_slots;

 /// We should make sure that preempted thread has no local task inside context.
 /// It is allowed to have tasks in `task_queue` or `fast_task_queue` because they can be stealed by other threads.
 auto & context = executor_contexts[slot_id];
 if (auto * task = context->popTask())
 {
 task_queue.push(task, slot_id);
 /// Wake up at least one thread to avoid deadlocks (all other threads maybe idle)
 tryWakeUpAnyOtherThreadWithTasks(*context, lock); // this releases the lock if it wakes up a thread
 }

The asymmetry is the point: a local task (held in the thread’s context) is invisible to other threads and must be handed back explicitly, whereas a task in the shared queue is stealable and needs no action.

The pulling mode: `executeUntil`

Not every query is executed to completion in one go. The HTTP protocol, for instance, wants to stream rows to the client as they become available, and the client may disconnect at any moment. That is what executeUntil() is for:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:261
bool PipelineExecutor::executeUntil(std::atomic_bool * yield_flag)
{
 const auto uncaught_exceptions = std::uncaught_exceptions();
 SCOPE_EXIT({
 if (tasks.isFinished() || std::uncaught_exceptions() > uncaught_exceptions)
 detachSpillables();
 });
 if (!is_execution_initialized)
 {
 initializeExecution(1, true);

 // Acquire slot until we are done
 single_thread_cpu_slot = cpu_slots->acquire();
 tasks.upscale(single_thread_cpu_slot->slot_id);
 chassert(single_thread_cpu_slot && "Unable to allocate cpu slot for the first thread, but we just allocated at least one slot");

 if (yield_flag && *yield_flag)
 return true;
 }

 executeStepImpl(0, WorkloadResources(single_thread_cpu_slot.get(), process_list_element), yield_flag);

 if (!tasks.isFinished())
 return true;

The yield_flag is the interesting parameter. It is checked at the bottom of the worker loop:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:576
 /// We have executed single processor. Check if we need to yield execution.
 if (yield_flag && *yield_flag)
 break;

So the executor runs one processor at a time and then checks the flag. If the flag says “enough for now”, it returns to the caller, which can hand the chunk to the client and then call executeUntil() again. This is the “pulling” model at the level of the whole pipeline: the caller pulls.

PullingPipelineExecutor::pull() is the caller:

 cpp
 
 
// src/Processors/Executors/PullingPipelineExecutor.cpp:49
bool PullingPipelineExecutor::pull(Chunk & chunk)
{
 if (!executor)
 {
 executor = createExecutor(pipeline.processors, pipeline.process_list_element);
 executor->setReadProgressCallback(pipeline.getReadProgressCallback());
 executor->setStepProfiler(pipeline.getStepProfiler());
 }

 /// Throws when the time limit is exceeded with `timeout_overflow_mode = 'throw'`. With 'break' the partial result
 /// is returned as a success: the execution is cancelled with `CancelledByTimeout`, and `executeUntil` below
 /// finalizes it - the pending read progress is reported and the format is finalized - before the end of the
 /// data is reported.
 if (pipeline.process_list_element && !pipeline.process_list_element->checkTimeLimitSoft())
 {
 executor->cancel(IProcessor::CancelReason::CancelledByTimeout);
 pipeline.process_list_element->checkTimeLimit();
 }

 if (!executor->executeUntil(&has_data_flag))
 return false;

 chunk = pulling_format->getChunk();
 return true;
}

has_data_flag is set by the output format when it has a chunk ready. So the loop is: run the pipeline until the format has produced something (or the pipeline is finished), then hand the chunk up.

Note that executeUntil runs the executor in single-threaded mode (initializeExecution(1, true)). The pulling executor is deliberately not parallel — it exists for cases like INSERT ... SELECT into a table, or formatting a result to a client that wants to consume it incrementally.

Trade-offs

What this buys you.

The pool grows with the work, not with the setting. A query that turns out to be narrow never pays for more than one or two threads, even if max_threads = 64. A query that turns out to be wide gets threads as fast as the scheduler will give them, without any startup latency. And the measurement is exact rather than heuristic: pushTasks counts the tasks it created and subtracts the threads that could absorb them.

Work stealing with per-thread affinity gives good cache behaviour and good load balance: the common case (a thread feeding itself) is a pointer swap with no locking, and the uncommon case (one thread has all the work) is handled by theft.

The separation of “what work exists” from “how many resources it gets” also means that the same executor serves queries with max_threads = 1 (no contention at all, the single thread takes everything locally) and queries with max_threads = 128.

What it costs.

The machinery is intricate, and the comments are full of post-mortems: the try_lock/pending_demand race, the lazy_allocation config reload, the max_scheduled_local_tasks starvation. Each of those is a bug that shipped and was fixed. The design is sound but the implementation is carrying the weight of a decade of production.

There is also a hard limit on the demand signal. pushTasks measures the tasks pushed by one processor at one moment. If a pipeline is wide but each individual push produces one task, the demand signal is one at a time, and the pool grows one thread per push rather than immediately. The initial_parallel computation in initializeExecution exists precisely to patch this for the first wave:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:722
 /// Initial queued parallelism never routes through `pushTasks`, so size setMax here to
 /// cover it. For multi-source pipelines (e.g. UNION ALL of N subqueries) this prevents
 /// the pipeline from being stuck single-threaded if the master's first task happens not
 /// to push more work. Only matters on the lazy path; the eager path already has the full
 /// ceiling. `initial_parallel` is the total target — the master thread is one of its
 /// consumers, no +1 needed.
 if (lazy_allocation && initial_parallel > 1)

The preemption path is the other sharp edge. A thread that cannot renew its lease stops, and the pipeline continues with fewer threads — but if every thread is preempted at once, the pipeline must not deadlock. The threads_queue.size() + 1 == total_slots check in tryGetTask is the mechanism that detects “the last worker is about to go idle” and calls finish():

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:158
 /// This thread has no tasks to do and is going to wait.
 /// Finish execution if this was the last active thread.
 chassert(task_queue.empty() && fast_task_queue.empty());
 if (threads_queue.size() + 1 == total_slots && async_task_queue.empty())
 {
 lock.unlock();
 finish();
 return;
 }

Getting this wrong in either direction is bad: finish too early and the query returns partial data; never finish and the query hangs. The chassert documents that the two conditions must hold together — if the queues are not empty, the pipeline is not finished and the thread must keep waiting.

Finally, there is the question of when to stop growing. desired_threads only ever goes up within a query. A pipeline that is wide at the start and narrow at the end keeps its threads; there is no downscaling based on demand, only downscaling due to preemption. For a long-running query over a table that becomes narrow, that is wasted capacity.

How ClickHouse Plans Distributed Queries with Cascades

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A single-node query plan has one obvious shape: read the table, filter, aggregate, sort. A distributed query does not. The same SELECT can be executed as a gather everything to the coordinator, or as a shuffle of both sides of a join followed by a local aggregation, or as a broadcast of a tiny dimension table to every node followed by a local join. These choices interact — the best join strategy depends on how the data will be aggregated afterwards, and the best aggregation strategy depends on how the join output is distributed — so the planner has to explore combinations and compare them.

ClickHouse’s traditional answer was a set of fixed heuristics. Its new answer, arriving in 26.8 as an experimental setting, is a real cost-based optimizer built on the Cascades framework: a memo of alternative plans, a set of rules that generate alternatives, and a cost model that prices them.

 cpp
 
 
// src/Core/Settings.cpp:11131
 DECLARE(Bool, enable_cascades_optimizer, false, R"(
Enable the Cascades cost-based optimizer for distributed query plans.
Takes effect only together with `make_distributed_plan = 1`: the setting alone does not change single-node query planning.
)", EXPERIMENTAL, \
 {"26.8", false, false, "New experimental setting."}) \

This article walks the implementation: the memo, the task-based search, the rules, the cost model, the enforcers, and the places where the implementation is deliberately conservative.

The problem the optimizer solves

The ARCHITECTURE.md that ships with the code states it plainly:

 text
 
 
## What Problem Does This Solve?

ClickHouse can execute a query on a single node using its standard pipeline: plan the
query, read from MergeTree, apply filters/expressions/joins/aggregations, return results.
But when data is spread across multiple nodes (shared storage like S3, or a multi-node
cluster), the optimizer must decide how to distribute the work:

- Should a join shuffle both sides by the join key, or broadcast the smaller table to
 all nodes?
- Should an aggregation happen on each node first (partial agg) and then merge results,
 or gather all data to one node and aggregate there?
- Should a table scan be split across N nodes (parallel read), or should each node read
 the full table (replicated read for small dimension tables)?
- Where should exchanges (network data transfers between nodes) be placed?

Note the third bullet: with shared storage (S3), a 25-row dimension table does not need to be broadcast at all — every node can just read it from S3 itself. That alternative, ReplicatedRead, is the kind of thing that only a cost-based search would find, because it is not a “join strategy” in the classical sense; it is a read strategy chosen because the cost of re-reading a tiny table is lower than the cost of shipping it.

The memo

The memo is the central data structure. It contains groups, each representing a set of logically equivalent subexpressions, and each group holds both the logical expressions (ways of computing the same thing) and the physical expressions (ways of computing it with a concrete algorithm and concrete properties).

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Memo.h:10
class Memo
{
public:
 explicit Memo(LoggerPtr log_)
 : log(log_)
 {}

 GroupId addGroup(GroupExpressionPtr group_expression);

 GroupPtr getGroup(GroupId group_id);
 GroupConstPtr getGroup(GroupId group_id) const;

 size_t getGroupCount() const { return groups_by_id.size(); }

The plan is ingested into the memo one group per step. A step with children becomes a group whose expression has one input link per child, and the input link carries the required properties of that child:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:179
 auto group_expression = std::make_shared<GroupExpression>(std::move(node.step));
 auto group_id = memo.addGroup(group_expression);
 for (auto * child_node : node.children)
 {
 auto [input_group_id, pending_props] = addGroup(*child_node);
 group_expression->inputs.push_back({input_group_id, pending_props});
 }

Each input link therefore carries the child’s group id and the properties the parent requires of it — which is where the “required properties” of the whole framework enter the picture.

Properties: distribution and sorting

A physical expression has two kinds of properties.

Distribution — how the data is spread over the nodes:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.h:17
struct DistributionDescription
{
 DistributionColumns columns; /// Columns by which data is distributed. E.g. for shuffle exchange or partitioned table scan.
 /// Type each key is cast to before hashing (parallel to `columns`; empty when the raw
 /// types agree). Aligns buckets across both sides of a shuffle join.
 Names hash_type_names;
 bool is_replicated = false; /// All data is replicated to all nodes, so any distribution is satisfied. E.g. for small tables that are broadcasted to all nodes.
 size_t node_count = 1; /// Number of nodes among which data is distributed. E.g. for a shuffle exchange or a partitioned read.

 bool operator==(const DistributionDescription & other) const = default;

 void dump(WriteBuffer & out) const;
 String dump() const;
};

Sorting — whether the output is ordered, and by what.

A parent requires properties of its child, and a child satisfies a requirement if it provides at least as much. The matching is not trivial: a sorted-by-(a) stream satisfies a requirement of “sorted by (a)”, and the required sort must be a prefix of the existing one:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.cpp:9
bool ExpressionProperties::isSortingSatisfiedBy(const SortDescription & required, const SortDescription & existing)
{
 /// Required sorting is satisfied by existing sorting if required is the prefix of existing
 if (required.size() > existing.size())
 return false;
 for (size_t i = 0; i < required.size(); ++i)
 {
 if (required[i] != existing[i])
 return false;
 }
 return true;
}

and the distribution match is positional — data shuffled by (b, a) does not colocate with data shuffled by (a, b):

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.cpp:39
 /// The partition hash chains the key columns in order and may cast them first, so the
 /// match must be positional and the hash types must agree: data shuffled by `(b, a)`
 /// or hashed at a different type does not colocate with data shuffled by `(a, b)`.
 if (required.hash_type_names != existing.hash_type_names)
 return false;

Sorting is a property, not an operator

One design decision is worth calling out, because it differs from classical Cascades. A plain ORDER BY (no LIMIT) is not an operator in the memo — it is a property that the parent requires of its input. The optimizer then produces the sort itself, if some child can provide it more cheaply:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:162
 /// Strip a limit-less SortingStep::Full - pure sorting is a physical property, not a logical
 /// one. Return the child's GroupId along with the sorting as required properties, so the
 /// caller can attach them to the input link of the parent group expression.
 /// A SortingStep with a limit is a top-N (row-reducing) operator, so it is kept as an
 /// operator instead; the limit is owned by that operator, never by the sorting property.
 const auto * sorting_step = typeid_cast<const SortingStep *>(node.step.get());
 if (sorting_step && sorting_step->getType() == SortingStep::Type::Full && sorting_step->getLimit() == 0)
 {
 chassert(node.children.size() == 1);
 auto [child_group_id, _] = addGroup(*node.children.front());
 ExpressionProperties stripped_props;
 stripped_props.sorting = sorting_step->getSortDescription();
 return {child_group_id, stripped_props};
 }

A LIMIT after the sort makes it a top-N, which reduces the row count and therefore is an operator with its own rules and its own cost.

The search: tasks on a stack

The optimizer is a LIFO stack of tasks. There are six task types, and a group moves through four stages.

Stage 1 — Explore. Transformation rules generate logically equivalent alternatives (join commutativity, two-stage aggregation).

Stage 2 — Implement. Implementation rules turn logical expressions into physical ones with concrete properties (local join, broadcast join, shuffle join; local read, parallel read, replicated read).

Stage 3 — Enforce. Enforcer rules bridge property gaps by inserting exchange steps.

Stage 4 — Done.

OptimizeGroupTask drives the stages:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:38
 if (!group->isExplored())
 {
 /// Explore the group, then re-run `OptimizeGroupTask`
 optimizer.pushTask(std::make_unique<OptimizeGroupTask>(group_id, required_properties));
 optimizer.pushTask(std::make_unique<ExploreGroupTask>(group_id));
 }
 else if (!group->isOptimizedFor(required_properties))
 {
 optimizer.pushTask(std::make_unique<OptimizeGroupTask>(group_id, required_properties));

 for (auto & expression : group->logical_expressions)
 optimizer.pushTask(std::make_unique<OptimizeExpressionTask>(expression, required_properties));

 group->setOptimizedFor(required_properties);
 }

The interesting part is the pruning decision at the top of the task, which is explicitly not a cost-based prune:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:20
 /// Skip this group only if it is already fully processed (explored + implemented +
 /// enforced) for these properties and has a satisfying plan - re-running would be a no-op.
 /// We deliberately do not prune just because a current best is within a finite cost budget:
 /// the budget is an upper bound, not a lower bound, so such pruning is unsound for
 /// optimality, and it can return before stage-3 enforcers add the distributed alternatives.
 {
 const auto & cost_config = optimizer.getMemo().getContext().cost_config;
 bool group_fully_processed = group->isExplored()
 && group->isOptimizedFor(required_properties)
 && group->isEnforcedFor(required_properties);
 if (group_fully_processed && group->getBestImplementation(required_properties, cost_config).expression)
 {
 group->setFullyDoneFor(required_properties);
 LOG_TEST(optimizer.log, "OptimizeGroupTask group #{}: already fully processed", group_id);
 return;
 }
 }

This is a really instructive piece of engineering. Classical Cascades prunes with branch-and-bound: once a subtree’s cost exceeds the best complete plan, it is dropped. ClickHouse’s implementation refuses to do that, because a subtree’s current best is an upper bound on its final cost — a cheaper enforcer-built alternative may still appear — and pruning on it would be unsound. Instead, the search is bounded by a task count.

Rule ordering

Every rule has a static promise (a priority). The applicable rules are sorted by promise and pushed on the LIFO stack, so the highest-promise rule runs first:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:169
 std::vector<std::pair<Promise, OptimizationRulePtr>> moves;
 for (const auto & rule : rules)
 {
 if (!expression->isApplied(*rule, required_properties) && rule->checkPattern(expression, required_properties, optimizer.getMemo()))
 moves.push_back({rule->getPromise(), rule});
 }

 /// Sort ascending so the LIFO task stack pops the highest-promise rule first.
 std::sort(moves.begin(), moves.end(), [](const auto & lhs, const auto & rhs) { return lhs.first < rhs.first; });

The promise does not change which plans are considered — every applicable rule is eventually applied — it only changes the order, which breaks ties among equal-cost alternatives (the first-found best wins).

The cost model

The cost has three dimensions, combined by configurable weights:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.h:39
struct CostConfig
{
 Float64 work_weight = 1.0; /// Parallelizable work (scans, expression eval, I/O).
 Float64 network_weight = 1.0; /// Per-byte network transfer.
 /// Single-threaded phases (gather/scatter funnels, merge cursors). The weight is the
 /// per-node thread count: work parallelizes across the threads of a node while a serial
 /// phase holds one thread, so one serial row costs about `threads` work rows (Brent's law:
 /// wall-clock ~ work / threads + serial path).
 Float64 sequential_weight = 32.0;
 /// Fixed per-exchange latency (connection setup, metadata), in sequential rows. The
 /// weighted value (~1e5 work units) keeps a plan over a small input local: distribution
 /// cannot pay for its setup there.
 Float64 exchange_fixed_overhead = 3000.0;

and the combination:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.h:74
 Float64 total(const CostConfig & config) const
 {
 /// If any component is infinite the plan is impossible; return infinity directly. Multiplying
 /// first would let a zero weight turn `inf * 0` into NaN, and a NaN cost compares as neither
 /// better nor worse, so an impossible plan could be picked as best.
 if (!std::isfinite(work) || !std::isfinite(network) || !std::isfinite(sequential))
 return std::numeric_limits<Float64>::infinity();

 return work * config.work_weight
 + network * config.network_weight
 + sequential * config.sequential_weight;
 }

The inf * 0 = NaN guard is the kind of detail that separates a cost model that works from one that occasionally picks an impossible plan.

The formulas

A hash join prices the probe, the output and the hash-table build:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:130
static Cost hashJoinCost(const CostInputs & inputs, bool is_broadcast)
{
 const auto & left_stats = inputStats(inputs, 0);
 const auto & right_stats = inputStats(inputs, 1);

 Cost cost;
 cost.work = (left_stats.estimated_row_count + inputs.output_stats.estimated_row_count) / inputs.parallelism;

 /// The build side is materialized fully on every node for a broadcast join (the nodes build
 /// concurrently) and 1/N per node otherwise. The build is work, not a sequential phase:
 /// `parallel_hash` shards the hash table across threads within a node.
 const Float64 build_rows_per_node = is_broadcast
 ? right_stats.estimated_row_count
 : right_stats.estimated_row_count / inputs.parallelism;
 /// Hash table materialization: memory allocation + cache pressure.
 cost.work += inputs.config.hash_table_build_factor * build_rows_per_node
 + build_rows_per_node * right_stats.estimated_bytes_per_row;
 return cost;
}

The exchange prices the network, and the distinction between a shuffle and a gather is where most of the distributed reasoning lives:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:240
static Cost exchangeCost(const CostInputs & inputs, const IQueryPlanStep & step)
{
 /// A sorted gather over a partial top-N transfers min(input_rows, L * node_count) rows,
 /// not the group's trimmed L; the caller resolves that override from the memo.
 const Float64 rows = inputs.exchange_rows_override.value_or(inputs.output_stats.estimated_row_count);
 Cost cost;
 /// The network term is wall-clock per node, like the work term. A shuffle (N->N) moves
 /// 1/N of the data per node, all nodes concurrently; without the division one node
 /// scanning the whole table is priced the same as the cluster shuffling it, and the
 /// optimizer keeps big scans on a single node.
 Float64 network_bytes = rows * inputs.output_stats.estimated_bytes_per_row;
 if (dynamic_cast<const ShuffleExchangeStep *>(&step))
 network_bytes /= std::max(1.0, inputs.node_count);
 cost.network += network_bytes;
 cost.sequential += inputs.config.exchange_fixed_overhead;
 /// Gather (N->1) and Scatter (1->N) funnel every row through a single node that
 /// sends or receives them sequentially, so their transfer stays undivided and each row
 /// pays the funnel cost. Without it a gather of a large input looks as cheap as a
 /// shuffle, so the optimizer distributes work (e.g. a sort) that should stay local.
 if (dynamic_cast<const GatherExchangeStep *>(&step) || dynamic_cast<const ScatterExchangeStep *>(&step))
 cost.sequential += inputs.config.funnel_sequential_cost_per_row * rows;
 return cost;
}

Three subtleties are worth spelling out:

A shuffle divides the network cost by the node count, because each node ships 1/N of the data and all nodes do it concurrently.

A broadcast does not divide, because every node ingests the whole payload:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:227
static Cost broadcastExchangeCost(const CostInputs & inputs)
{
 Cost cost;
 /// The network term is wall-clock per node, like the work term (which is divided by
 /// parallelism): every receiving node ingests the full payload, all of them concurrently,
 /// and the senders each ship their share of the copies in parallel. So one full payload
 /// is the per-node bound. A shuffle moves 1/N per node, so a broadcast still costs N times
 /// a shuffle of the same data.
 cost.network += inputs.output_stats.estimated_row_count * inputs.output_stats.estimated_bytes_per_row;
 cost.sequential += inputs.config.exchange_fixed_overhead;
 return cost;
}

Every exchange pays a fixed overhead in sequential units, which is what keeps a plan over a small input local: distributing 25 rows is not worth a network round trip.

A read is priced on its scan volume, not its output

This one is subtle and important. A filter that the primary key cannot prune leaves the whole granule selection to scan, so the output estimate badly understates the cost:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:150
/// One node reads the whole table.
static Cost fullReadCost(const CostInputs & inputs)
{
 /// Price the scan volume, not the output volume: a filter the primary key cannot prune
 /// leaves the whole granule selection to scan, and without this a replicated read of such
 /// a table would look orders of magnitude cheaper than the scan it does.
 if (inputs.output_stats.physical_read_bytes > 0)
 return Cost{.work = inputs.output_stats.physical_read_bytes};
 return Cost{.work = inputs.output_stats.estimated_row_count * inputs.output_stats.estimated_bytes_per_row};
}

Without this, ReplicatedRead — every node reading the whole table from S3 — would look free for a table whose filter prunes 99 % of the rows.

The cost model is query-tunable

Everything above is overridable with one JSON parameter:

 sql
 
SET param__internal_cascades_cost_config = '{"work_weight":1,"network_weight":1,"sequential_weight":32}';

and an invalid configuration rejects the query rather than silently falling back to the defaults.

The rules

Join implementations

One rule produces all the join alternatives:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/HashJoinImplementation.cpp:19
/// Produces all applicable hash join implementations in a single rule:
/// - Local join: both inputs gathered to one node (always applicable)
/// - Broadcast join: left partitioned any way, right replicated (only when node_count > 1)
/// - Partitioned (shuffle) join: both inputs shuffled by join keys (only when node_count > 1
/// and the join has equi-join predicates)
///
/// When node_count == 1 all three strategies produce the same plan, so only the local join
/// is emitted to avoid redundant identical alternatives in the memo.
class HashJoinImplementation : public IOptimizationRule

A shuffle join partitions both sides by the join key. The key columns are recorded positionally on both sides, together with the type they are hashed at:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/HashJoinImplementation.cpp:316
 auto add_key = [&](const JoinKeyPair & key)
 {
 left_dist.columns.push_back({key.left});
 right_dist.columns.push_back({key.right});
 if (needs_hash_cast)
 {
 const String & type_name = hash_type_for(key);
 left_dist.hash_type_names.push_back(type_name);
 right_dist.hash_type_names.push_back(type_name);
 }
 /// Advertise the output partitioning only on the preserved side(s).
 NameSet output_key;
 if (left_preserved)
 output_key.insert(key.left);
 if (right_preserved)
 output_key.insert(key.right);

The hash_type_names exist because if the two sides have different key types, they must be cast to a common type before hashing, or the two sides would land in different buckets and the join would produce nothing.

Two-stage aggregation

The transformation rule splits a single-phase aggregation into a partial and a merge:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:92
/// Logical transformation that splits a single-phase Agg into a two-phase plan:
/// PartialAgg(input) -> ShuffleExchange -> FinalMergeAgg
///
/// PartialAgg computes partial aggregate states locally on each node without finalization.
/// FinalMergeAgg receives pre-aggregated states and merges them into final results.
/// The exchange between the two is inserted by the `DistributionEnforcer` based on the
/// distribution requirements set by the implementation rules on FinalMergeAgg.
///
/// A non-final aggregation is split too when it promises bucket order to a consumer outside the
/// plan: the merge then stays non-final and restores the bucket order over the gathered partials,
/// which is the distributed alternative to the single-node partial `AggregationImplementation`
/// leaves for such a step.
class TwoStageAggregationTransformation : public IOptimizationRule

and the split clones the aggregation step, marks the clone as non-final, and creates a MergingAggregatedStep for the merge half:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:345
 /// Phase 1: partial aggregation - takes raw rows, outputs intermediate aggregate states.
 auto partial_step_ptr = cloneStepAs(*agg_step);
 auto * partial_step = partial_step_ptr.get();
 partial_step->setFinal(false);

The shuffle aggregation is only applicable in some cases, and the reasons are all correctness reasons:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:64
 /// See `addShuffleAggregation` for why each condition disables the shuffle strategy.
 bool isShuffleApplicable() const
 {
 return !agg_step.getParams().keys.empty()
 && !agg_step.isGroupingSets()
 && !agg_step.getParams().overflow_row
 && agg_step.getParams().max_rows_to_group_by == 0;
 }

no group keys → a global aggregation has nothing to shuffle by;
grouping sets → the keys are the union of the sets’ keys, so shuffling by the union splits the rows of one grouping-set group across nodes;
an overflow row → every node would emit its own;
max_rows_to_group_by → the limit is a global contract, but each node would enforce it independently.

Single-key shuffle alternatives

A nice touch: for a multi-key GROUP BY, the optimizer also considers shuffling by one of the keys, which is cheaper (fewer bytes in the exchange) and still correct:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:198
/// Single-key shuffle alternatives.
/// For aggregations with 2+ group-by keys, generate a shuffle alternative for each
/// individual key. Correctness: `GROUP BY (A, B)` with data shuffled by `A` is correct
/// because all rows with the same `(A, B)` have the same `A`, hence the same node.
void AggregationImplementation::StrategyEnumerator::addSingleKeyShuffleAggregations(const std::vector<size_t> & candidate_node_counts)

Enforcers: self-referential expressions

When a parent requires {1 node} and the cheapest child produces {4 nodes}, something has to bridge the gap. That something is an enforcer — an exchange step inserted by the optimizer.

Enforcers are the hairiest part of the implementation, because an enforcer’s input is the same group with relaxed requirements, which looks like a cycle:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/DistributionEnforcer.cpp:24
/// Produces self-referential enforcer expressions that bridge distribution gaps.
/// Each enforcer expression lives in the same group as the source expression; its
/// single input points back to the same group with normalized properties (node_count,
/// is_replicated, and sorting where needed) as the requirement - this lets the optimizer
/// pick the cheapest source with that distribution shape. The optimizer recursively
/// satisfies this self-referential input through the normal task mechanism, enabling
/// natural enforcer composition (e.g. a sort and a gather compose into gather-then-sort,
/// see `SortingEnforcer`, without
/// bundling steps).
class DistributionEnforcer : public IOptimizationRule

The construction is centralised:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/DistributionEnforcer.cpp:104
void DistributionEnforcer::EnforcerEnumerator::addEnforcer(QueryPlanStepPtr step, ExpressionProperties input_required, SortDescription output_sorting)
{
 ExpressionProperties output_properties;
 output_properties.distribution = required_properties.distribution;
 output_properties.sorting = std::move(output_sorting);
 auto enforcer_expr = makeEnforcerExpression(
 expression, std::move(step), std::move(input_required), std::move(output_properties), EnforcedProperty::Distribution);

 rule.addPhysicalToMemo(enforcer_expr, required_properties, memo, result);
}

Two invariants keep the self-reference acyclic. The first is that an enforcer carries an axis (Sorting or Distribution), and an enforcer that satisfies a wildcard axis only by over-providing on it is not eligible — a sorted enforcer cannot fill an empty-sort slot, a keyed exchange cannot fill an empty-columns slot:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:159
 if (input_is_self_referential && candidate->enforced_property != EnforcedProperty::None)
 {
 /// Empty sort requirement: reject a sorted enforcer (it over-provides on the sort axis).
 if (required_properties.sorting.empty() && !candidate->properties.sorting.empty())
 return;
 /// Empty distribution-columns requirement: reject a keyed exchange. A sorting enforcer
 /// carrying keyed columns is still needed to feed a sorted gather, so keep it eligible.
 if (candidate->enforced_property == EnforcedProperty::Distribution
 && required_properties.distribution.columns.empty()
 && (!candidate->properties.distribution.columns.empty()
 || !candidate->properties.distribution.hash_type_names.empty()))
 return;
 }

The second is that an enforcer must never evict a non-enforcer alternative from the best-implementation cache, because that alternative is the acyclic way out:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:78
 /// An enforcer "satisfies" a weaker non-enforcer only by over-providing (e.g. a sorted gather
 /// covers an empty-sort requirement). Dropping the base would leave a self-referential enforcer
 /// chain with no acyclic alternative, so an enforcer never suppresses or evicts a distinct base.
 auto enforcer_over_provides_for = [](const GroupExpressionPtr & enforcer_candidate, const GroupExpressionPtr & base_candidate)
 {
 return enforcer_candidate->enforced_property != EnforcedProperty::None
 && base_candidate->enforced_property == EnforcedProperty::None
 && !(enforcer_candidate->properties == base_candidate->properties);
 };

The fixed point

Enforcers compose: a sort enables a sorted gather, which enables a gather. The Stage 3 loop iterates until no new enforcer is produced:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:96
 /// Fixed-point loop: iterate over newly-added physical expressions until no
 /// new enforcers are produced. Each iteration may create expressions that
 /// enable further enforcers (e.g. a sort enables a sorted `GatherExchange`).
 size_t enforced_up_to = 0;
 bool new_enforcers_created = true;
 while (new_enforcers_created)
 {
 new_enforcers_created = false;

Keeping the best implementation

Each group keeps a Pareto frontier of implementations, bucketed by distribution shape so that a lookup is O(1) plus a short scan:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:73
void Group::updateBestImplementation(GroupExpressionPtr expression, const CostConfig & cost_config)
{
 UInt64 key = distributionKey(expression->properties.distribution);
 auto & bucket = best_implementations[key];

and the eviction rule is a domination test: an implementation is dropped when another one satisfies at least the same properties at a lower cost.

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:88
 /// Remove all known best expressions with higher cost and properties satisfied by the new expression.
 /// Only the matching distribution-shape bucket needs checking - `isSatisfiedBy` requires
 /// exact match on (node_count, is_replicated).
 for (auto best_it = bucket.begin(); best_it != bucket.end();)
 {
 if (expression->properties.isSatisfiedBy((*best_it)->properties) &&
 (*best_it)->cost->subtree_cost.total(cost_config) <= expression->cost->subtree_cost.total(cost_config) &&
 !enforcer_over_provides_for(*best_it, expression))
 {
 /// There is already a cheaper implementation that satisfies the same properties
 return;
 }

Failing closed

The search is bounded by a task count, not by branch-and-bound. And when the budget runs out, the query is rejected rather than planned from a partial memo:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:308
 /// Limit the time in terms of optimization tasks instead of wall clock time. This is done for stability of generated plans regardless of system load.
 /// Microsoft SQL Server's optimizer team describes this in Andy Pavlo's seminar: https://www.youtube.com/watch?v=pQe1LQJiXN0
 const size_t executed_tasks_limit = getCascadesTaskLimitParam(query_context, CascadesDefaults::DEFAULT_TASK_LIMIT);
 size_t executed_tasks_count = 0;
 for (; !tasks.empty() && executed_tasks_count < executed_tasks_limit; ++executed_tasks_count)
 {
 auto task = std::move(tasks.top());
 tasks.pop();
 task->execute(*this);
 }

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:321
 /// Fail closed if the search did not finish within the task budget: building a plan from a
 /// partial memo can yield a non-minimal plan or a confusing failure deep inside buildBestPlan.
 /// Surface a clear error instead and point at the knob to raise the limit.
 if (!tasks.empty())
 throw Exception(ErrorCodes::SUPPORT_IS_DISABLED,
 "Cascades optimizer did not finish within the task budget of {} tasks "
 "(root group #{}, required properties {}, {} groups in memo, {} tasks left, next: {}). "
 "The distributed Cascades optimizer is experimental; set enable_cascades_optimizer = 0 "
 "or simplify the query.",
 executed_tasks_limit, root_group_id, root_required_properties.dump(),
 memo.getGroupCount(), tasks.size(), tasks.top()->describe());

The default budget is 100 000 tasks:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/OptimizerDefaults.h:17
constexpr size_t DEFAULT_TASK_LIMIT = 100000;

Trying it

 sql
 
 
SET enable_cascades_optimizer = 1;
SET make_distributed_plan = 1;

SELECT n_name, SUM(o_totalprice) AS revenue
FROM orders
JOIN customer ON o_custkey = c_custkey
JOIN nation ON c_nationkey = n_nationkey
WHERE o_orderdate >= '1995-01-01' AND o_orderdate < '1995-04-01'
GROUP BY n_name
ORDER BY revenue DESC
SETTINGS send_logs_level = 'test';

The test log level prints the memo, every costed expression with its work/network/sequential breakdown, and the final plan — which is the intended way to understand why the optimizer chose what it chose.

Trade-offs

What it buys: plans that no fixed heuristic would produce — a broadcast of a tiny table instead of a shuffle of a huge one, a ReplicatedRead from shared storage instead of a network transfer, a two-phase aggregation instead of a gather, a sort per node instead of a sort of everything.

What it costs:

It is experimental. The setting is EXPERIMENTAL and off by default, and it only applies with make_distributed_plan = 1.
Join ordering is not part of it. This is the biggest difference from classical Cascades. ClickHouse fixes the join order first (with the existing dpsize/dphyp/greedy algorithms) and Cascades only optimises the distribution strategy. The published literature calls this the “split” design, and it loses the global optimum when the join order interacts with physical properties — which, in a distributed setting, it does.
The statistics are weak. Today they come from index analysis and from query-parameter hints; there is no cardinality estimator as sophisticated as the one the pre-Cascades join orderer uses. A cost model on bad statistics picks the wrong plan confidently.
Only two parallelism levels are explored: {1 node} and {N nodes}. Intermediate counts were never chosen on TPC-H, and exploring them would multiply the search space.
The search can be exponential, which is why there is a hard task budget and a fail-closed error. Classical Cascades would prune instead.
Some plan shapes are rejected outright — WITH TOTALS, ROLLUP, CUBE, extremes, PASTE JOIN, and reads that cannot be cloned across workers.

The most interesting thing about this code, though, is not the optimizer itself but the discipline around it: every place where the implementation deliberately deviates from the textbook — no branch-and-bound, a fail-closed budget, the enforcer-axis rules, the “never evict the acyclic fallback” rule — carries a comment explaining exactly why the textbook version would be wrong here. That is rare, and it is the part of the code most worth reading.

How ClickHouse Packs Short Strings Into Hash Table Keys

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Strings are the most common GROUP BY key in practice. They are also the worst case for a hash table: a std::string in a cell means one heap allocation per distinct key, and a std::string_view in a cell means the bytes must outlive the table, which for an aggregation means copying them into an arena.

ClickHouse’s answer is a hash table that is not a hash table. Instead of one table with a variable-size key, StringHashTable keeps five separate tables, one per key size class, and packs short keys into fixed-width integers. The result is a table where every cell is the same size, no key ever allocates, and the compiler can dispatch on the key length at compile time.

The file is src/Common/HashTable/StringHashTable.h, with the map specialisation in StringHashMap.h.

Four size classes and one oddity

The composite table is five sub-tables:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:234
template <typename SubMaps>
class StringHashTable : private boost::noncopyable
{
protected:
 static constexpr size_t NUM_MAPS = 5;
 // Map for storing empty string
 using T0 = typename SubMaps::T0;

 // Short strings are stored as numbers
 using T1 = typename SubMaps::T1;
 using T2 = typename SubMaps::T2;
 using T3 = typename SubMaps::T3;

 // Long strings are stored as std::string_view along with saved hash
 using Ts = typename SubMaps::Ts;
 using Self = StringHashTable;

and the storage:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:254
 T0 m0;
 T1 m1;
 T2 m2;
 T3 m3;
 Ts ms;

The classes are:

Table	Key type	Holds
`m0`	—	the empty string, in a dedicated slot
`m1`	`StringKey8` (`UInt64`)	1..8 bytes
`m2`	`StringKey16` (`UInt128`)	9..16 bytes
`m3`	`StringKey24`	17..24 bytes
`ms`	`std::string_view` + saved hash	25+ bytes, or short strings with a trailing zero

The key types are just integers:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:10
using StringKey8 = UInt64;
using StringKey16 = UInt128;
struct StringKey24
{
 UInt64 a;
 UInt64 b;
 UInt64 c;

 bool operator==(const StringKey24 rhs) const { return a == rhs.a && b == rhs.b && c == rhs.c; }
};

So a 20-byte string becomes three UInt64s and nothing else: no length field, no pointer, no allocation. The length is implicit — it is recovered from the position of the first zero byte.

The packing

The heart of the file is dispatchOnKeyClass, which classifies a key, packs it, hashes it, and calls a callback with the right sub-table. It is written for speed, and the comment says so:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:357
 // Dispatch is written in a way that maximizes the performance:
 // 1. Always memcpy 8 times bytes
 // 2. Use switch case extension to generate fast dispatching table
 // 3. Funcs are named callables that can be force_inlined
 //
 /// Note: the key bytes must be readable in 8-byte chunks (a `std::string` cannot be passed
 /// here, while e.g. `ColumnString::getDataAt` can), because the packing copies 8 bytes at a
 /// time.
 template <typename Resolver, typename KeyHolder, typename HashProvider, typename Func>
 static auto ALWAYS_INLINE dispatchOnKeyClass(Resolver && resolver, KeyHolder && key_holder, HashProvider && hash_of, Func && func)
 {
 const auto & x = keyHolderGetKey(key_holder);
 const size_t sz = x.size();
 if (sz == 0)
 {
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 0>{}, 0), VoidKey{}, 0);
 }

 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

 const char * p = x.data();
 // pending bits that needs to be shifted out
 const char s = (-sz & 7) * 8;

The switch ((sz - 1) >> 3) that follows turns the length into one of four cases, and the compiler turns that into a jump table or a chain of comparisons:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:394
 switch ((sz - 1) >> 3)
 {
 case 0: // 1..8 bytes
 {
 // first half page
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)
 {
 memcpy(&n[0], p, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] &= -1ULL >> s;
 else
 n[0] &= -1ULL << s;
 }
 else
 {
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[0], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] >>= s;
 else
 n[0] <<= s;
 }
 const size_t res = hash_of(k8);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 1>{}, res), k8, res);
 }

Three tricks are hiding in those twenty lines.

Trick one: the trailing-zero test. A fixed-size string key encodes the length implicitly, by the position of the first zero byte. That means a string whose last byte is zero cannot be represented: "ab\0" and "ab" would pack to the same 8 bytes. So such strings are pushed to the generic table before the switch:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:376
 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

Trick two: the mask/shift pair. s = (-sz & 7) * 8 is the number of padding bits needed to round the length up to a multiple of 8. In the first branch the 8 bytes are read forwards from the start of the string and the padding bits are cleared with a mask; in the second branch the 8 bytes are read backwards from the end of the string and the padding bits are shifted out.

Trick three: the “first half page” rule. Reading 8 bytes from a 3-byte string reads 5 bytes past its end. That is only safe if those bytes are mapped. The condition

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:399
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)

tests whether the address is 2048-byte aligned; if it is, at least 2048 bytes of the page are still ahead of it, so reading 8 bytes forward cannot fault. Otherwise the code reads backwards instead, ending exactly at the last byte of the string. The bytes before the string that this reads are then shifted out — the shift >>= s discards exactly the (-sz & 7) bytes that belong to the previous string or to the column’s offset array. So the garbage is read but never used.

The 9..16 and 17..24 byte cases use the same trick with two and three words:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:420
 case 1: // 9..16 bytes
 {
 memcpy(&n[0], p, 8);
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[1], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[1] >>= s;
 else
 n[1] <<= s;
 const size_t res = hash_of(k16);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 2>{}, res), k16, res);
 }

Note that the first word is always read forwards from p — for a 9-byte string that is inside the string — and the last word is read backwards from the end. The overlap between them is harmless: the bytes in the middle are read twice with the same values.

Unpacking

Going the other way, the length is recovered from the number of leading zero bytes:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:21
inline std::string_view ALWAYS_INLINE toStringView(const StringKey8 & n)
{
 chassert(n != 0);
 if constexpr (std::endian::native == std::endian::big)
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countr_zero(n) >> 3)};
 else
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countl_zero(n) >> 3)};
}

countl_zero(n) >> 3 is the number of whole zero bytes at the end of the word, so 8 - that is the string length. On a big-endian machine the zero bytes are at the other end, hence the countr_zero variant.

The empty-cell protocol, specialised

The “empty cell is a zeroed cell” rule from the generic table needs a suitable notion of “zero” for a packed string key. For StringKey16 the rule is “the last word is zero”:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:20
template <typename TMapped>
struct StringHashMapCell<StringKey16, TMapped> : public HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>
{
 using Base = HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>;
 using value_type = typename Base::value_type;
 using Base::Base;
 static constexpr bool need_zero_value_storage = false;
 bool isZero(const HashTableNoState & state) const { return isZero(this->value.first, state); }

 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey16 & key, const HashTableNoState &) { return key.items[1] == 0; }
 void setZero() { this->value.first.items[1] = 0; }

The reasoning is the same as in the generic table: a packed key with a zero last word would have to encode the length “16” with no room for a terminating zero byte, so such a key cannot exist — the encoding is injective on all valid keys, and the all-zero pattern is unreachable. That makes it safe to use as the empty marker. Same for StringKey24, where the marker is the third word:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:49
 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey24 & key, const HashTableNoState &)
 { return key.c == 0; }
 void setZero() { this->value.first.c = 0; }

The generic std::string_view cell cannot do this — a string_view is a pointer and a length, and a zeroed one is a null pointer with length 0, which is a legal key (the empty string). So the generic cell declares that it needs no zero storage and the empty string gets its own slot instead:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:134
template <typename Cell>
struct StringHashTableEmpty // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
 using Self = StringHashTableEmpty;

 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

The empty slot is not a hash table at all — it is a single cell with an emplace that ignores the key entirely:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:172
 template <typename KeyHolder>
 void ALWAYS_INLINE emplace(KeyHolder &&, LookupResult & it, bool & inserted, size_t = 0)
 {
 if (!hasZero())
 {
 setHasZero();
 inserted = true;
 }
 else
 inserted = false;
 it = zeroValue();
 }

Hashing

The hash function is chosen per class, and for the fixed-size classes it is the hardware CRC32C instruction:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:46
struct StringHashTableHash
{
#if defined(__SSE4_2__)
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key);
 return res;
 }
 size_t ALWAYS_INLINE operator()(StringKey16 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key.items[0]);
 res = _mm_crc32_u64(res, key.items[1]);
 return res;
 }

with an AArch64 CRC32 path, an s390x path, and CityHash as the portable fallback:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:114
#else
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 return CityHash_v1_0_2::CityHash64(reinterpret_cast<const char *>(&key), 8);
 }

CRC32 is a poor hash function in the abstract — the header of the generic hash file says so — but it is one instruction per 8 bytes with a three-cycle latency, and the table is small enough to fit in cache most of the time. It also has to be consistent with the packing: the hash is computed over the packed word, not over the original bytes, so "abc" and "abc\0..." cannot collide by accident — they hash different words.

The lookup result

The string table does not return a pointer to a cell; it returns a pointer to the mapped value, wrapped in a small type:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:214
template <typename Mapped>
struct StringHashTableLookupResult
{
 Mapped * mapped_ptr;
 StringHashTableLookupResult() {} /// NOLINT
 StringHashTableLookupResult(Mapped * mapped_ptr_) : mapped_ptr(mapped_ptr_) {} /// NOLINT
 StringHashTableLookupResult(std::nullptr_t) {} /// NOLINT
 const VoidKey getKey() const { return {}; } /// NOLINT
 auto & getMapped() { return *mapped_ptr; }
 auto & operator*() { return *this; }
 auto & operator*() const { return *this; }
 auto * operator->() { return this; }
 auto * operator->() const { return this; }
 explicit operator bool() const { return mapped_ptr; }

The reason is that the key lives in the cell and the caller usually only wants the aggregate state; but the key is still there, in a different sub-table with a different representation. Returning a pointer to it would mean returning a pointer to a StringKey8 or to a string_view, which have different types depending on the size class. So the key is erased from the interface: getKey() returns a VoidKey.

Reserving and growing

The string table grows differently from the generic one:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:207
template <size_t initial_size_degree = 8>
struct StringHashTableGrower : public HashTableGrowerWithPrecalculation<initial_size_degree>
{
 // Smooth growing for string maps
 void increaseSize() { this->increaseSizeDegree(1); }
};

The generic grower doubles the buffer on overflow; the string grower grows by a factor of two in the exponent, i.e. by one power of two — the same thing, but expressed as a single degree increment so it composes with the precalculated mask.

Reserving is divided evenly over the four real sub-maps, because the caller only knows the total number of keys:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:317
 explicit StringHashTable(size_t reserve_for_num_elements)
 : m1{reserve_for_num_elements / 4}
 , m2{reserve_for_num_elements / 4}
 , m3{reserve_for_num_elements / 4}
 , ms{reserve_for_num_elements / 4}
 {
 }

That guess is often wrong — a table of 20-byte strings uses m3 only, and m1, m2, ms reserve space they never use. The aggregator knows this and refuses to pre-reserve the string tables:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5214
 /// `StringHashTable::reserve` splits the hint evenly over its four size-class sub-maps,
 /// while a real key set concentrates in one of them, so it is not reserved.
 constexpr bool can_reserve = requires { dst.reserve(size_t{}); } && !requires { dst.emptyStringSlot(); };

The emptyStringSlot() method exists precisely so that this requires expression can tell a StringHashTable from an ordinary hash table:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:312
 /// The dedicated slot of the empty-string key, which the sub-table dispatch hashes to 0.
 T0 & emptyStringSlot() { return m0; }

Where it is used

The string map is one of the standard aggregation data variants:

 cpp
 
 
// src/Interpreters/AggregatedData.h:53
using AggregatedDataWithPackedStringKey = HashMap<PackedStringRef, AggregateDataPtr>;
using AggregatedDataWithShortStringKey = StringHashMap<AggregateDataPtr>;
using AggregatedDataWithStringKey = HashMapWithSavedHash<std::string_view, AggregateDataPtr>;

Three flavours of string key, three tables:

AggregatedDataWithPackedStringKey — a PackedStringRef, a pointer plus a length, with the payload inline for short strings;
AggregatedDataWithShortStringKey — the composite StringHashMap of this article;
AggregatedDataWithStringKey — a plain std::string_view map with a saved hash, where the key bytes live in an arena.

The StringHashMap variant is the interesting one, because it is the only one where the key does not live anywhere else: the string is in the table, as three machine words.

Trade-offs

What the packed string table buys:

No allocation per key. A 20-byte key is three UInt64s inside the cell. Nothing is copied into an arena, nothing is freed.
A uniform cell size. Every cell of m1 is 8 bytes of key plus the mapped value; every cell of m3 is 24 bytes plus the mapped value. The table can be scanned, and the compiler can generate good code for it.
A compile-time dispatch on length. dispatchOnKeyClass is a template over the size class, so each branch is specialised: the m1 path hashes one word, the m3 path hashes three.
A hash over the packed form. The hash is computed over exactly the bytes that are stored, so there is no chance of the stored key and the hashed key disagreeing.

What it costs:

The bytes of the key are duplicated. The cell holds the packed key, and if the caller needs the original string it has to be reconstructed with toStringView.
The trailing-zero case is a wart. A string ending in \0 — which happens with binary data, and with FixedString columns padded with zeros — cannot use the packed classes and falls back to the generic table, where the key is a string_view into the arena.
The reservation heuristic is bad. Reserving n/4 for each of the four sub-maps wastes memory whenever the keys concentrate in one class, which is the normal case.
The empty string needs its own slot, and the key type of the generic sub-table is not the same as the key type of the packed ones, so the composite has to be written as a template over a SubMaps bundle rather than as a plain HashTable.

The design is a good example of the general principle: when the key type has a size distribution, splitting the table by size class turns a variable-size-cell problem into a set of fixed-size-cell problems, and lets the compiler specialise each one.

How ClickHouse Merges Columns One at a Time: the Vertical Merge Algorithm

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every MergeTree table is a pile of immutable parts. Every part is a pile of immutable columns. Background merges take a contiguous range of parts and rewrite them as one bigger part. That sounds like a simple copy, and for a table with three narrow columns it is one.

Now consider a table with two hundred columns, one of which is a 40 MB String of raw JSON and the rest are UInt64 counters. The obvious merge implementation reads all two hundred columns of the source parts, merges the sorting key, and writes two hundred columns out. It has to hold, in memory at once, the merged key plus every payload column of the rows it is currently holding — which for a wide table means the peak memory of a merge is proportional to the sum of all column widths, not to the width of the widest column.

That is a bad trade. The payload columns are almost always much larger than the key, and they are almost always independent of the key: if you merge the keys first and separately, you can then reconstruct each payload column without ever holding two payload columns in memory at the same time.

ClickHouse implements exactly that. It is called the vertical merge algorithm, and it is one of the more elegant pieces of engineering in the MergeTree code base. The trick that makes it work is a one-byte-per-row side-channel file that records where each output row came from.

The problem, stated precisely

Suppose we merge three parts, each sorted by the key k:

 text
 
 
 part A (10 rows) part B (10 rows) part C (10 rows)
 k=1 a1=... k=1 b1=... k=2 c1=...
 k=1 a2=... k=2 b2=... k=2 c2=...
 k=3 a3=... k=2 b3=... k=5 c3=...
 ... ... ...

A horizontal merge reads all three parts’ k columns, merges them into one sorted stream, and then — because MergeTree guarantees that all rows with the same key are adjacent and, within a key, appear in source-part order — it can reconstruct every other column by a gather: walk the merged key stream, and for each row, take the next value of that column from the part the row came from.

The output order of the merged key is not the concatenation of the inputs; it is the k-way merge of them. So for each output row we need to know which input part it came from. That is the entire information content of the vertical merge’s side channel.

And it is exactly one small integer per row.

The horizontal stage: merging the key and writing the row sources

The vertical merge is not a separate code path that replaces the normal merge. It is a second stage that runs after the normal merge has merged the key. The normal merge is called the “horizontal” stage, and the code says so explicitly — the stage class is called ExecuteAndFinalizeHorizontalPart:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.h:63
/**
 * Overview of the merge algorithm
 *
 * Each merge is executed sequentially block by block.
 * The main idea is to make a merge not a subroutine which is executed
 * in a thread pool and may occupy a thread for a period of time,
 * but to make a merge a coroutine which can suspend the execution
 * in some points and then resume the execution from this point.
 *
 * A perfect point where to suspend the execution is after the work over a block is finished.
 * The task itself will be executed via BackgroundJobExecutor.
 *
 * The interface of the task is simple.
 * The main method is `execute()` which will return true, if the task wants to be executed again and false otherwise.
 *
 * With this kind of task we can give a merge a priority.
 * A priority is simple - the lower the size of the merge, the higher priority.
 * So, if ClickHouse wants to merge some really big parts into a bigger part,
 * then it will be executed for a long time, because the result of the merge is not really needed immediately.
 * It is better to merge small parts as soon as possible.
*/

Note the second half of that comment: the merge is a coroutine. It is resumed by the background pool between blocks, which is what makes it possible to give a small merge priority over a huge one without pinning a thread. The vertical stage is expressed in the same idiom: a list of sub-tasks, each of which returns true when it wants to be resumed.

The side channel is a real file, managed by a small class:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:257
/// Manages the "rows_sources" temporary file that is used during vertical merge.
class RowsSourcesTemporaryFile : public ITemporaryFileLookup
{
public:
 /// A logical name of the temporary file under which it will be known to the plan steps that use it.
 static constexpr auto FILE_ID = "rows_sources";

 explicit RowsSourcesTemporaryFile(TemporaryDataOnDiskScopePtr temporary_data_on_disk_)
 : temporary_data_on_disk(temporary_data_on_disk_->childScope({CurrentMetrics::TemporaryFilesForMerge}))
 {
 }

and it is deliberately re-openable, so that several columns can be gathered in parallel from independent file handles — getTemporaryFileForReading returns tmp_data_buffer->read(), a fresh reader each time, with the comment “Reopen the file for each read so that multiple reads can be performed in parallel and there is no need to seek to the beginning.”

That matters: the row-source stream is consumed once per column, and each column’s gather starts from the beginning of the file. If the file were a single sequential stream, gathering column #2 would have to seek back to offset 0 and re-read column #1’s worth of bytes. Re-opening gives every column its own independent read cursor.

The encoding: one byte per row

The row source is not a struct with a source number and a flag in two fields. It is a single byte with two bit-fields packed into it:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:15
/// Tiny struct, stores number of a Part from which current row was fetched, and insertion flag.
struct RowSourcePart
{
 UInt8 data = 0;

 RowSourcePart() = default;

 explicit RowSourcePart(size_t source_num, bool skip_flag = false)
 {
 static_assert(sizeof(*this) == 1, "Size of RowSourcePart is too big due to compiler settings");
 setSourceNum(source_num);
 setSkipFlag(skip_flag);
 }

 size_t getSourceNum() const { return data & MASK_NUMBER; }

 /// In CollapsingMergeTree case flag means "skip this rows"
 bool getSkipFlag() const { return (data & MASK_FLAG) != 0; }

 void setSourceNum(size_t source_num)
 {
 data = (data & MASK_FLAG) | (static_cast<UInt8>(source_num) & MASK_NUMBER);
 }

 void setSkipFlag(bool flag)
 {
 data = flag ? data | MASK_FLAG : data & ~MASK_FLAG;
 }

 static constexpr size_t MAX_PARTS = 0x7F;
 static constexpr UInt8 MASK_NUMBER = 0x7F;
 static constexpr UInt8 MASK_FLAG = 0x80;
};

The top bit is the “skip this row” flag (used when a lightweight delete mask removes rows during the merge), and the low seven bits are the source part number. Seven bits means at most 127 source parts per merge — and you can watch that limit propagate all the way into the merge-algorithm choice (section “When is it worth it?” below).

The static_assert is not decoration. If the compiler ever made this struct two bytes, the row-source file would double in size and every write to it would become a two-byte write; the assert turns a silent performance regression into a build failure.

Who writes the bytes

The horizontal stage’s merge transform writes one byte per output row as it emits it. For the Ordinary merging mode that is the plain k-way merge algorithm:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:444
void MergingSortedAlgorithm::insertRow(const SortCursorImpl & current)
{
 auto write_row_source = [&](bool skipped)
 {
 if (out_row_sources_buf)
 {
 RowSourcePart row_source(current.order, skipped);
 out_row_sources_buf->write(row_source.data);
 }
 };

 size_t current_row = current.getRow();

 if (const auto * mask = source_row_filter_masks[current.order])
 {
 if ((*mask)[current_row])
 merged_data.insertRow(current.all_columns, current_row, current.rows);

 write_row_source(!(*mask)[current_row]);
 }
 else
 {
 merged_data.insertRow(current.all_columns, current_row, current.rows);
 write_row_source(false);
 }
}

current.order is the index of the source stream the row came from. The merged algorithm already knows this — it is the whole reason it can merge — so emitting it costs one byte write per row, i.e. one byte per row of overhead on a merge that is about to rewrite gigabytes.

Note the ordering: merged_data.insertRow(...) happens first, then the row source is written. The write buffer is only non-null for a vertical merge — the merge step is constructed with the file name only when the vertical algorithm was chosen:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3654
 auto merge_step = std::make_unique<MergePartsStep>(
 merge_parts_query_plan.getCurrentHeader(),
 sort_description,
 partition_and_sorting_required_columns,
 global_ctx->merging_params,
 (is_vertical_merge ? RowsSourcesTemporaryFile::FILE_ID : ""), /// rows_sources' temporary file is used only for vertical merge
 filter_column_name,
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size],
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size_bytes],
 max_dynamic_subcolumns,
 is_vertical_merge,
 sorting_queue_strategy,
 cleanup,
 global_ctx->time_of_merge);

so a horizontal merge pays nothing for the side channel.

The vertical stage: gathering one column at a time

Now the second stage. It does not merge anything. It gathers. For each non-key column, in turn:

open a read stream for that column in every source part;
union those streams;
run a ColumnGathererTransform over them, which uses the row-source file to decide which source’s value belongs at each output position.

Here is the step that wires the gatherer into the plan:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2172
 createReadFromPartStep(
 MergeTreeSequentialSourceType::Merge,
 *plan_for_part,
 *global_ctx->data,
 global_ctx->storage_snapshot,
 RangesInDataPart(global_ctx->future_part->parts[part_num], nullptr, part_num, part_starting_offset),
 global_ctx->alter_conversions[part_num],
 global_ctx->merged_part_offsets,
 Names{column_name},
 global_ctx->input_rows_filtered,
 apply_deleted_mask,
 /*filter=*/ std::nullopt,
 ctx->read_with_direct_io,
 ctx->use_prefetch,
 global_ctx->context,
 getLogger("VerticalMergeStage"));

Names{column_name} — exactly one column is read per part. The read plan for a vertical merge is a column read plan, not a row read plan. Every source part is opened, and only the column of interest is decoded.

Then the per-part streams are unioned, and a ColumnGathererStep is added to the plan. The gatherer is a merging transform, so it plugs into the same IMergingTransform machinery as the key merge: its input is one column per source part, its output is one merged column, and the row-source file is handed to it as a ReadBuffer by name.

The gather loop

The heart of it is ColumnGathererStream::gather. It reads the row-source bytes and copies values out of the per-part source columns:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:131
template <typename Column>
void ColumnGathererStream::gather(Column & column_res)
{
 row_sources_buf.nextIfAtEnd();
 RowSourcePart * row_source_pos = reinterpret_cast<RowSourcePart *>(row_sources_buf.position());
 RowSourcePart * row_sources_end = reinterpret_cast<RowSourcePart *>(row_sources_buf.buffer().end());

 if (next_required_source == -1)
 {
 /// Start new column.
 /// Actually reserve works only for fixed size columns.
 /// So it's safe to ignore preferred size in bytes and call reserve for number of rows.
 size_t size_to_reserve = std::min(static_cast<size_t>(row_sources_end - row_source_pos), block_preferred_size_rows);
 column_res.reserve(size_to_reserve);
 }

 next_required_source = -1;

 /// We use do ... while here to ensure there will be at least one iteration of this loop.
 /// Because the column_res.byteSize() could be bigger than block_preferred_size_bytes already at this point.
 do
 {
 if (row_source_pos >= row_sources_end)
 break;

 RowSourcePart row_source = *row_source_pos;
 size_t source_num = row_source.getSourceNum();
 Source & source = sources[source_num];
 bool source_skip = row_source.getSkipFlag();

 if (source.pos >= source.size) /// Fetch new block from source_num part
 {
 next_required_source = source_num;
 return;
 }

 ++row_source_pos;

 /// Consecutive optimization. TODO: precompute lengths
 size_t len = 1;
 size_t max_len = std::min(static_cast<size_t>(row_sources_end - row_source_pos), source.size - source.pos); // interval should be in the same block

 while (len < max_len && row_source_pos->data == row_source.data)
 {
 ++len;
 ++row_source_pos;
 }

 row_sources_buf.position() = reinterpret_cast<char *>(row_source_pos);

 if (!source_skip)
 {
 /// Whole block could be produced via copying pointer from current block
 if (source.pos == 0 && source.size == len)
 {
 /// If current block already contains data, return it.
 /// Whole column from current source will be returned on next read() iteration.
 source_to_fully_copy = &source;
 return;
 }
 if (len == 1)
 column_res.insertFrom(*source.column, source.pos);
 else
 column_res.insertRangeFrom(*source.column, source.pos, len);
 }

 source.pos += len;
 } while (column_res.size() < block_preferred_size_rows && column_res.byteSize() < block_preferred_size_bytes);
}

Three optimizations are hiding in there.

Run-length coalescing. The inner while loop scans forward over consecutive row-source bytes that are identical. Since the byte encodes “which source part”, a run of identical bytes means “these N output rows came from the same source part, consecutively” — which, because the source part’s own column is read in order, means “these N output rows are exactly source part’s rows [pos, pos+N)”. So the whole run is copied with a single insertRangeFrom call instead of N single-element inserts. The comment /// Consecutive optimization. TODO: precompute lengths is honest about the remaining inefficiency: the run length is rediscovered for every column, N times.

Whole-column pointer steal. If the run covers the entire remaining source column (source.pos == 0 && source.size == len), the output column can simply be the source column — no copy at all. The transform returns the source ColumnPtr directly and sets source_to_fully_copy, and the merge() method hands it out:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.cpp:114
 else
 {
 res.addColumn(source_to_fully_copy->column);
 }

For a merge where one part dominates — the common case after a few rounds of merging, where you merge one big part with one small one — most columns of the big part are copied by pointer. The “merge” of such a column is essentially free.

Sparse output. The source_skip flag comes from the lightweight-delete mask: rows deleted by a DELETE are recorded in the row-source stream with the skip bit set, and the gatherer simply does not copy them out. The horizontal stage does the same filtering, so the two stages agree on which rows exist without either having to re-evaluate the delete predicate.

Writing the gathered column

The gathered column is written by MergedColumnOnlyOutputStream — the vertical stage’s counterpart of MergedBlockOutputStream, except that it writes exactly one column. Its constructor carries the invariant that makes the whole thing safe:

 cpp
 
 
// src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:36
 /// Granularity is never recomputed while writing only columns.
 MergeTreeWriterSettings writer_settings(

The horizontal stage has already decided where the granule boundaries are (it wrote the marks for the key columns), and the payload columns must use the same boundaries, or the part would be corrupt.

When is it worth it?

Vertical merging is not always a win. It has costs: an extra pass over the row-source file per column, an extra file in the temporary space, and a pipeline per column. So chooseMergeAlgorithm() gates it behind a set of conditions:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3760
MergeAlgorithm MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm() const
{
 const size_t total_rows_count = global_ctx->merge_list_element_ptr->total_rows_count;
 const size_t total_size_bytes_uncompressed = global_ctx->merge_list_element_ptr->total_size_bytes_uncompressed;
 const auto & merge_tree_settings = global_ctx->data_settings;

 if (global_ctx->deduplicate)
 return MergeAlgorithm::Horizontal;
 if ((*merge_tree_settings)[MergeTreeSetting::enable_vertical_merge_algorithm] == 0)
 return MergeAlgorithm::Horizontal;
 if (ctx->need_remove_expired_values)
 {
 /// `TTLTransform` stops reading after the first block when the rows TTL has expired for the whole part,
 /// while a vertical merge reads every key row to write `rows_sources`.
 if (global_ctx->future_part->merge_type == MergeType::TTLDrop && global_ctx->metadata_snapshot->hasRowsTTL())
 return MergeAlgorithm::Horizontal;
 if (!canVerticalTTLDelete(*global_ctx))
 return MergeAlgorithm::Horizontal;
 }
 if (global_ctx->future_part->part_format.part_type != MergeTreeDataPartType::Wide)
 return MergeAlgorithm::Horizontal;
 if (global_ctx->cleanup)
 return MergeAlgorithm::Horizontal;

and then the quantitative thresholds:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3793
 bool is_supported_storage =
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Ordinary ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Collapsing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Replacing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::VersionedCollapsing;

 bool enough_ordinary_cols = global_ctx->gathering_columns.size() >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_columns_to_activate];

 bool enough_total_rows = total_rows_count >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_rows_to_activate];

 bool enough_total_bytes = total_size_bytes_uncompressed >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_bytes_to_activate];

 bool no_parts_overflow = global_ctx->future_part->parts.size() <= RowSourcePart::MAX_PARTS;

 auto merge_alg = (is_supported_storage && enough_total_rows && enough_total_bytes && enough_ordinary_cols && no_parts_overflow) ?
 MergeAlgorithm::Vertical : MergeAlgorithm::Horizontal;

 return merge_alg;
}

Read those conditions as a description of when the vertical algorithm is the right shape. Wide parts only, because a Compact part stores all columns in one data file and you cannot read one column of it without reading the others. No deduplication, because deduplication compares whole rows and so needs all columns at once. Only Ordinary/Collapsing/Replacing/ VersionedCollapsing merging modes, because Summing/Aggregating/ Coalescing merge the values of equal keys and so have to merge the payload rather than gather it. At least 11 non-key columns (vertical_merge_algorithm_min_columns_to_activate) and at least 16·8192 rows (vertical_merge_algorithm_min_rows_to_activate) — below those, the extra machinery is not worth it. And at most 127 source parts, because RowSourcePart::MAX_PARTS is 0x7F and the one-byte encoding cannot express more.

The default thresholds live in MergeTreeSettings: enable_vertical_merge_algorithm = 1, vertical_merge_algorithm_min_rows_to_activate = 16 * 8192, vertical_merge_algorithm_min_bytes_to_activate = 0, and vertical_merge_algorithm_min_columns_to_activate = 11.

Note the framing of the setting description: “potentially at the cost of merge speed”. Vertical merging is a memory optimization, not a speed one. It trades IO locality (one column at a time, so each source file is opened and read once per column rather than once for all columns) for peak memory. For a table where one column dominates the size, that trade is usually right; for a table with many equally-sized columns, the re-reading can dominate.

The `Wide`-only restriction, and the escape hatch

The Wide restriction is the interesting one. A Compact part interleaves all columns of a row in a single .bin file, so “read only column X” is impossible — you would have to decode the whole file. The sources may still be Compact while the result is Wide, and that case is allowed by allow_vertical_merges_from_compact_to_wide_parts (default true):

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3784
 if (!(*merge_tree_settings)[MergeTreeSetting::allow_vertical_merges_from_compact_to_wide_parts])
 {
 for (const auto & part : global_ctx->future_part->parts)
 {
 if (!isWidePart(part))
 return MergeAlgorithm::Horizontal;
 }
 }

A Compact source is read once per column during the vertical stage, so its file gets decoded N times. That is expensive but it is bounded — and it buys the memory reduction for wide tables whose legacy parts happen to be Compact.

Progress accounting

A merge is a long-running background job, and system.merges shows a progress bar. For a vertical merge, “progress” spans two axes: first the key columns are merged (the horizontal stage), then each payload column is gathered (the vertical stage). The horizontal stage is weighted by the size of the key columns relative to the whole part; the vertical stage is weighted by the size of each individual column:

 cpp
 
 
// src/Storages/MergeTree/ColumnSizeEstimator.h:9
/* Allow to compute more accurate progress statistics */
class ColumnSizeEstimator
{
 using ColumnToSize = std::map<String, UInt64>;
 ColumnToSize map;
public:

 /// Stores approximate size of columns in bytes
 /// Exact values are not required since it used for relative values estimation (progress).
 size_t sum_total = 0;
 size_t sum_index_columns = 0;
 size_t sum_ordinary_columns = 0;

 ColumnSizeEstimator(ColumnToSize && map_, const NamesAndTypesList & key_columns, const NamesAndTypesList & ordinary_columns)
 : map(std::move(map_))
 {
 for (const auto & [name, _] : key_columns)
 if (!map.contains(name)) map[name] = 0;
 for (const auto & [name, _] : ordinary_columns)
 if (!map.contains(name)) map[name] = 0;

 for (const auto & [name, _] : key_columns)
 sum_index_columns += map.at(name);

 for (const auto & [name, _] : ordinary_columns)
 sum_ordinary_columns += map.at(name);

 sum_total = std::max(static_cast<decltype(sum_index_columns)>(1), sum_index_columns + sum_ordinary_columns);
 }

 Float64 columnWeight(const String & column) const
 {
 return static_cast<Float64>(map.at(column)) / static_cast<Float64>(sum_total);
 }

 Float64 keyColumnsWeight() const
 {
 return static_cast<Float64>(sum_index_columns) / static_cast<Float64>(sum_total);
 }
};

The weights are proportional to bytes, not rows, so a merge that is 90% one giant String column reports 90% progress for most of its duration and then jumps. That is the honest answer for a byte-oriented operation. The sizes come from the parts themselves (accumulateColumnSizes, a cheap per-part statistic), so the whole estimator costs one map construction per merge.

The horizontal stage’s contribution: the row count assertion

There is one more piece of rigor worth pointing out. After the horizontal stage finishes and the row-source file is closed, the code checks that the file contains exactly one byte per input row:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2055
 size_t rows_sources_count = ctx->rows_sources_temporary_file->finalizeWriting();

 /// In special case, when there is only one source part, and no rows were skipped, we may have
 /// skipped writing rows_sources file. Otherwise rows_sources_count must be equal to the total
 /// number of input rows.
 /// Note that only one byte index is written for each row, so number of rows is equals to the number of bytes written.
 if ((rows_sources_count > 0 || global_ctx->future_part->parts.size() > 1) && sum_input_rows_exact != rows_sources_count + input_rows_filtered)
 throw Exception(
 ErrorCodes::LOGICAL_ERROR,
 "Number of rows in source parts ({}) excluding filtered rows ({}) differs from number "
 "of bytes written to rows_sources file ({}). It is a bug.",
 sum_input_rows_exact, input_rows_filtered, rows_sources_count);

This is a cheap, total consistency check on the side channel. If the horizontal stage and the vertical stage ever disagree about which rows exist, the merge fails loudly rather than producing a part whose payload columns are misaligned with its key — which would be a silent data corruption. Note the subtlety in the condition: with exactly one source part and no filtered rows, the file may legitimately be empty (a single-source merge does not need to record where rows came from), so the check is skipped in that one case.

Prefetching the next column

Because the vertical stage reads one column at a time, and each column read is a separate pipeline over the source parts, the obvious optimization is to start reading column N+1 while column N is still being written. The code does exactly that, and only for remote filesystems, where the latency of the first read of a column is the dominant cost:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2076
 bool all_parts_on_remote_disks = std::ranges::all_of(global_ctx->future_part->parts, [](const auto & part) { return part->isStoredOnRemoteDisk(); });
 ctx->use_prefetch = all_parts_on_remote_disks && storage_settings[MergeTreeSetting::vertical_merge_remote_filesystem_prefetch];

 if (ctx->use_prefetch && ctx->it_name_and_type != global_ctx->gathering_columns.end())
 ctx->prepared_pipeline = createPipelineForReadingOneColumn(ctx->it_name_and_type->name);

The “prefetching” is simply that building the pipeline — which opens the parts and issues the first read requests — starts the IO. It is not a separate prefetch thread; it is eager pipeline construction, one column ahead (prepareVerticalMergeForOneColumn builds the next column’s pipeline while the current one runs).

The whole flow

 text
 
 
 source parts (Wide)
 ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
 │ k, a, b, c │ │ k, a, b, c │ │ k, a, b, c │
 └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
 │ │ │
 └────────────┬────┴─────────────────┘
 │ HORIZONTAL STAGE
 │ k-way merge of the key columns
 │ emits: merged k + rows_sources (1 byte/row)
 ▼
 ┌──────────────────────────────────────────┐
 │ rows_sources: 00 00 01 01 02 00 01 ... │
 └──────────────────────────────────────────┘
 │
 ┌─────────────────┼──────────────────┐
 ▼ ▼ ▼
 gather col a gather col b gather col c VERTICAL STAGE
 read a from read b from read c from one column at a time
 each part, each part, each part,
 copy by copy by copy by
 rows_sources rows_sources rows_sources
 │ │ │
 ▼ ▼ ▼
 └────────────┬────┴──────────────────┘
 ▼
 result part (Wide): k, a, b, c

What the hack buys you, and what it costs

It buys you peak memory proportional to the widest column, not the sum of the columns. For a table with a 40 MB String column and fifty UInt64 columns, a horizontal merge holds the payload of all the columns at once; a vertical merge holds one column at a time. That is the difference between a merge that OOMs and one that does not — and it is what makes the “wide part” format practical at all.

It costs you IO locality. Each column is a separate pass over the source parts, so a Compact source file is decoded once per column — which is why allow_vertical_merges_from_compact_to_wide_parts defaults to true but carries a warning in its description.

It costs you a temporary file. One byte per row of scratch space: 100 MB for a 100-million-row merge. Trivial compared to the data being merged, but not zero.

It costs you the ability to merge every kind of table. Summing, Aggregating and Coalescing merges have to combine the values of equal keys, which is a merge, not a gather. They stay horizontal, permanently.

It is not automatically faster. The setting description says it plainly: “potentially at the cost of merge speed”. The vertical algorithm optimizes memory, and only optimizes IO when a whole column happens to come from a single source part and can be stolen by pointer.

Seeing it in action

You can see which algorithm a merge used from the profile events:

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN (
 'MergeVerticalStageExecuteMilliseconds',
 'MergeHorizontalStageTotalMilliseconds',
 'MergeVerticalStageTotalMilliseconds'
);

MergeVerticalStageExecuteMilliseconds is only non-zero for tables where the vertical algorithm actually ran. If it is zero on your wide table, check enable_vertical_merge_algorithm, the part format (Compact parts cannot be vertically merged), and the vertical_merge_algorithm_min_* thresholds.

And to see the tables that would benefit:

 sql
 
 
SELECT
 database,
 table,
 formatReadableSize(sum(bytes_on_disk)) AS size,
 sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC
LIMIT 10;

A table with hundreds of columns and a large String payload is exactly the shape the vertical merge was built for. A table with ten narrow columns is not, and the thresholds will keep it horizontal.

How ClickHouse Marks Empty Hash Table Cells With Zeroed Memory

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every GROUP BY in ClickHouse is a hash table. The table has to hold millions of keys, be probed once per input row, and be resized while it is being filled — all while the query is competing for memory with everything else on the machine. A std::unordered_map is simply not competitive here: it allocates a node per element, keeps a pointer chain, and pays a pointer-chase for every collision.

So ClickHouse ships its own hash table in src/Common/HashTable/HashTable.h: a flat array of cells, open addressing, linear probing. The interesting part is not the probing — that is textbook — but the bookkeeping around emptiness. There is no “occupied” bit, no tombstone, no sentinel pointer, no version counter. A cell is empty when its key is all zero bytes, and the whole buffer starts out zeroed by the allocator.

That single decision shapes the entire file: the allocator, the cell layout, the way the zero key is handled, the way deletion works, and the reason the hash function has to be good.

The shape of the table

 cpp
 
 
// src/Common/HashTable/HashTable.h:518
template <typename Key, typename Cell, typename Hash, typename Grower, typename Allocator>
class HashTable : private boost::noncopyable,
 protected Hash,
 protected Allocator,
 protected Cell::State,
 public ZeroValueStorage<Cell::need_zero_value_storage, Cell> /// empty base optimization
{

The table is one contiguous buffer of Cells plus a Grower that knows the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:550
 size_t m_size = 0; /// Amount of elements
 Cell * buf; /// A piece of memory for all elements except the element with zero key.
 Grower grower;

Cell is a policy class. The simplest one stores nothing but a key:

 cpp
 
 
// src/Common/HashTable/HashTable.h:154
template <typename Key, typename Hash, typename TState = HashTableNoState>
struct HashTableCell
{
 using State = TState;

 using key_type = Key;
 using value_type = Key;
 using mapped_type = VoidMapped;

 Key key;

The map flavour stores a pair, and the variant that ClickHouse uses for aggregation additionally caches the hash so that a rehash does not have to recompute it:

 cpp
 
 
// src/Common/HashTable/HashMap.h:165
template <typename Key, typename TMapped, typename Hash, typename TState = HashTableNoState>
struct HashMapCellWithSavedHash : public HashMapCell<Key, TMapped, Hash, TState> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `saved_hash` is set by `setHash` immediately after placement construction on insert; on the hot aggregation/join path we must avoid the redundant store
{
 using Base = HashMapCell<Key, TMapped, Hash, TState>;

 size_t saved_hash;

Empty means zero

Here is the whole emptiness protocol:

 cpp
 
 
// src/Common/HashTable/HashTable.h:191
 /// Whether the key is zero. In the main buffer, cells with a zero key are considered empty.
 /// If zero keys can be inserted into the table, then the cell for the zero key is stored separately, not in the main buffer.
 /// Zero keys must be such that the zeroed-down piece of memory is a zero key.
 bool isZero(const State & state) const { return isZero(key, state); }
 static bool isZero(const Key & key, const State & /*state*/) { return ZeroTraits::check(key); }

 /// Set the key value to zero.
 void setZero() { ZeroTraits::set(key); }

and the generic part:

 cpp
 
 
// src/Common/HashTable/HashTable.h:89
namespace ZeroTraits
{

template <typename T>
bool check(const T x)
{
 return bitEquals(x, T{});
}

template <typename T>
void set(T & x) { x = T{}; }

}

bitEquals is bitwise, not ==, and that matters for floating point:

 cpp
 
 
// src/Common/HashTable/HashTable.h:77
template <typename T>
inline ALWAYS_INLINE bool bitEquals(T a, T b)
{
 if constexpr (is_floating_point<T>)
 /// Note that memcmp with constant size is a compiler builtin.
 return 0 == memcmp(&a, &b, sizeof(T)); /// NOLINT
 else
 return a == b;
}

The comment above it explains why bit equality is mandatory rather than merely nice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:71
/** Numbers are compared bitwise.
 * Complex types are compared by operator== as usual (this is important if there are gaps).
 *
 * This is needed if you use floats as keys. They are compared by bit equality.
 * Otherwise the invariants in hash table probing do not met when NaNs are present.
 */

NaN never equals itself, so a ==-based emptiness test could never recognise an empty float cell, and the probe loop would run off the end of the buffer.

The zeroed buffer comes from the allocator:

 cpp
 
 
// src/Common/HashTable/HashTableAllocator.h:8
/**
 * We are going to use the entire memory we allocated when resizing a hash
 * table, so it makes sense to pre-fault the pages so that page faults don't
 * interrupt the resize loop. Set the allocator parameter accordingly.
 */
using HashTableAllocator = Allocator<true /* clear_memory */, true /* populate */>;

clear_memory becomes calloc:

 cpp
 
 
// src/Common/Allocator.cpp:79
 if (alignment <= MALLOC_MIN_ALIGNMENT) [[likely]]
 {
 if constexpr (clear_memory)
 buf = __real_calloc(size, 1);
 else
 buf = __real_malloc(size);

So the invariant is: the bytes of a fresh buffer are zero, therefore a fresh cell is empty. Nothing more is needed to track occupancy. The diagram of the buffer is therefore very simple:

 text
 
 
 buf buf + bufSize()
 | |
 v v
 +--------+--------+--------+--------+--------+--------+--------+--------+
 | cell 0 | cell 1 | cell 2 | cell 3 | cell 4 | cell 5 | cell 6 | cell 7 |
 +--------+--------+--------+--------+--------+--------+--------+--------+
 key=A (zero) key=C (zero) key=B (zero) (zero)
 [used] [empty] [used] [empty] [used] [empty] [empty]

Note what is not there: no occupancy byte, no pointer, no version. A HashMapCell<UInt64, AggregateState*> is 16 bytes of pure payload; the empty/used distinction is free.

The zero key gets its own slot

There is one hole in the “zero means empty” scheme: 0 is a perfectly legal key. If it were stored in the buffer, it would be indistinguishable from an empty cell. So it is stored outside the buffer, in a ZeroValueStorage that the table inherits from:

 cpp
 
 
// src/Common/HashTable/HashTable.h:474
template <typename Cell>
struct ZeroValueStorage<true, Cell> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
private:
 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

 void setHasZero()
 {
 has_zero = true;
 new (zeroValue()) Cell();
 }

and the insert path routes the zero key to it:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1354
 bool ALWAYS_INLINE emplaceIfZero(const Key & x, LookupResult & it, bool & inserted, size_t hash_value)
 {
 /// If it is claimed that the zero key can not be inserted into the table.
 if constexpr (!Cell::need_zero_value_storage)
 return false;

 if (unlikely(Cell::isZero(x, *this)))
 {
 it = this->zeroValue();

 if (!this->hasZero())
 {
 ++m_size;
 this->setHasZero();
 this->zeroValue()->setHash(hash_value);
 inserted = true;
 }
 else
 inserted = false;

 return true;
 }

 return false;
 }

The zero key lives at a fixed address, so a lookup for it is a branch and a pointer, never a probe. The cost of the scheme is one bool plus one Cell of storage per table, and a slightly awkward iteration order (the zero cell is visited first, then the buffer).

This is not an exotic corner. 0 is a perfectly ordinary grouping key:

 sql
 
SELECT k, count() FROM t GROUP BY k;

with a k column full of zeros produces a group whose key is the zero key, and that group is the one stored in the separate slot.

Note also that the design is conditional: a cell type can declare that zero keys are impossible (need_zero_value_storage = false), and then ZeroValueStorage<false, Cell> degenerates to nothing at all. The string tables do exactly that — see the article on the string-specialised map.

Probing

The probe sequence is the simplest possible one: linear probing with a mask.

 cpp
 
 
// src/Common/HashTable/HashTable.h:232
 /// The size of the hash table in the cells.
 size_t bufSize() const { return 1ULL << size_degree; }

 size_t maxFill() const { return 1ULL << (size_degree - 1); }
 size_t mask() const { return bufSize() - 1; }

 /// From the hash value, get the cell number in the hash table.
 size_t place(size_t x) const { return x & mask(); }

 /// The next cell in the collision resolution chain.
 size_t next(size_t pos) const { ++pos; return pos & mask(); }

 /// Whether the hash table is sufficiently full. You need to increase the size of the hash table, or remove something unnecessary from it.
 bool overflow(size_t elems) const { return elems > maxFill(); }

The load factor is 0.5: maxFill() is half the buffer. And the probe loop itself:

 cpp
 
 
// src/Common/HashTable/HashTable.h:559
 size_t ALWAYS_INLINE findCell(const Key & x, size_t hash_value, size_t place_value) const
 {
 while (!buf[place_value].isZero(*this) && !buf[place_value].keyEquals(x, hash_value, *this))
 {
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }

 return place_value;
 }

Two things to note. First, isZero is checked before keyEquals: the loop stops at the first empty cell, because with linear probing an empty cell terminates the chain. Second, the comparison is a bitwise equality against the key, and for the HashMapCellWithSavedHash variant it is short-circuited by the cached hash.

The zero key is handled before the probe:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1617
 LookupResult ALWAYS_INLINE find(const Key & x)
 {
 if (Cell::isZero(x, *this))
 return this->hasZero() ? this->zeroValue() : nullptr;

 return findNonZeroCell(x, hash(x));
 }

Insertion

emplace computes the hash, routes the zero key, and otherwise finds a place:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1380
 template <typename KeyHolder>
 void ALWAYS_INLINE emplaceNonZeroImpl(size_t place_value, KeyHolder && key_holder,
 LookupResult & it, bool & inserted, size_t hash_value)
 {
 it = &buf[place_value];

 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);
 const auto & key = keyHolderGetKey(key_holder);

 new (&buf[place_value]) Cell(key, *this);
 buf[place_value].setHash(hash_value);
 inserted = true;
 ++m_size;

 if (unlikely(grower.overflow(m_size)))
 {
 try
 {
 resize();
 }
 catch (...)
 {
 /** If we have not resized successfully, then there will be problems.
 * There remains a key, but uninitialized mapped-value,
 * which, perhaps, can not even be called a destructor.
 */
 --m_size;
 buf[place_value].setZero();
 inserted = false;
 keyHolderDiscardKey(key_holder);
 throw;
 }

 // The hash table was rehashed, so we have to re-find the key.
 it = findNonZeroCell(key, hash_value);
 chassert(it);
 }
 }

Two details worth pausing on. The cell is placement-newed; the buffer is raw memory, and the destructor of a non-trivial cell is only run for cells that were actually constructed (destroyElements walks the table and calls the destructor only for non-zero cells).

And the resize happens after the insert, not before. That means the table can exceed its nominal capacity by one element, and it means the iterator has to be re-found afterwards, because the rehash moved the cell. The exception handler is careful to undo the insert (decrement the size, zero the cell) so that a failed allocation does not leave a half-built cell behind.

Growing: the wrap-around special case

The resize path is where the linear-probing invariants get hairy. When the buffer grows, every element has to be re-placed, and an element that was at the end of the old buffer may belong at the beginning of the new one because its collision chain wrapped around:

 cpp
 
 
// src/Common/HashTable/HashTable.h:837
 void resize(size_t for_num_elems = 0, size_t for_buf_size = 0)
 {
#ifdef DBMS_HASH_MAP_DEBUG_RESIZES
 Stopwatch watch;
#endif

 size_t old_size = grower.bufSize();

 /** In case of exception for the object to remain in the correct state,
 * changing the variable `grower` (which determines the buffer size of the hash table)
 * is postponed for a moment after a real buffer change.
 * The temporary variable `new_grower` is used to determine the new size.
 */
 Grower new_grower = grower;

and the re-placement loop:

 cpp
 
 
// src/Common/HashTable/HashTable.h:884
 size_t i = 0;
 for (; i < old_size; ++i)
 if (!buf[i].isZero(*this))
 reinsert(buf[i], buf[i].getHash(*this));

 /** There is also a special case:
 * if the element was to be at the end of the old buffer, [ x]
 * but is at the beginning because of the collision resolution chain, [o x]
 * then after resizing, it will first be out of place again, [ xo ]
 * and in order to transfer it where necessary,
 * after transferring all the elements from the old halves you need to [ o x ]
 * process tail from the collision resolution chain immediately after it [ o x ]
 */
 size_t new_size = grower.bufSize();
 for (; i < new_size && !buf[i].isZero(*this); ++i)
 reinsert(buf[i], buf[i].getHash(*this));

The picture in that comment is the whole problem: after the buffer is doubled, the element at the very end of the old buffer (x) is now in the middle of the new one, and the element that was pushed behind it (o) belongs before it. The second loop therefore keeps going past the end of the old buffer, as long as the cells it finds are non-empty, re-inserting them.

The re-insertion itself is a memcpy of a trivially-copyable cell into the new position:

 cpp
 
 
// src/Common/HashTable/HashTable.h:918
 void reinsert(Cell & x, size_t hash_value)
 {
 size_t place_value = grower.place(hash_value);

 /// If the element is in its place.
 if (&x == &buf[place_value])
 return;

 /// Compute a new location, taking into account the collision resolution chain.
 place_value = findCell(Cell::getKey(x.getValue()), hash_value, place_value);

 /// If the item remains in its place in the old collision resolution chain.
 if (!buf[place_value].isZero(*this))
 return;

 /// Copy to a new location and zero the old one.
 x.setHash(hash_value);
 memcpy(static_cast<void*>(&buf[place_value]), &x, sizeof(x)); /// NOLINT(bugprone-undefined-memory-manipulation)
 x.setZero();
 }

The grower doubles the size until max_size_degree (23) and then grows by one step at a time:

 cpp
 
 
// src/Common/HashTable/HashTable.h:247
 /// Increase the size of the hash table.
 void increaseSize()
 {
 size_degree += size_degree >= max_size_degree ? 1 : 2;
 }

Robin Hood: what changed recently

The most interesting recent addition to the file is a second probing discipline. The classic HashTableGrower keeps the load factor at 0.5; the new HashTableGrowerWithPrecalculationAndRobinHood keeps 0.5 until the table reaches 2^16 cells, and then switches to Robin Hood hashing with a 90% load factor — in place, without reallocating:

 cpp
 
 
// src/Common/HashTable/HashTable.h:336
/** Like `HashTableGrowerWithPrecalculation`, but adapts to large hash tables.
 *
 * For large hash tables, random memory access is the main bottleneck, not the computations.
 * When the buffer reaches 2^robin_hood_size_degree cells, the grower switches the hash table to the Robin Hood mode:
 * the max load factor is increased from 0.5 to robin_hood_max_load_factor_percent / 100 (the hash table becomes up to two times smaller),
 * and the elements inside collision resolution chains are ordered by their optimal positions,
 * so a lookup of a missing key can stop early instead of walking to the end of the chain.
 *
 * The memory layout is the same in both modes, and switching happens in place, without growing the buffer.
 * Hash tables check the mode only if the grower has the `isRobinHood` method (see `GrowerWithRobinHood`),
 * so there is no cost for other growers.
 *
 * Requirements to the cell: obtaining the hash value with `getHash` must be cheap (the hash is saved or cheap to compute),
 * and the hash table must not keep pointers to cells while inserting other elements,
 * because an insertion may move other elements along the collision resolution chain.
 */

The payoff is twofold. The table becomes up to twice as small (0.9 vs 0.5 load factor), which halves the number of cache misses in a random probe; and a failed lookup can stop early, because the chain is sorted by “distance from the ideal position”:

 cpp
 
 
// src/Common/HashTable/HashTable.h:607
 size_t ALWAYS_INLINE findCellRobinHood(const Key & x, size_t hash_value, size_t place_value, bool & found) const
 {
 for (size_t distance = 0;; ++distance)
 {
 const Cell & cell = buf[place_value];
 if (cell.isZero(*this))
 {
 found = false;
 return place_value;
 }
 if (cell.keyEquals(x, hash_value, *this))
 {
 found = true;
 return place_value;
 }
 /// The element at the optimal position of the key cannot be closer to its own optimal position, so do not compute its hash.
 if (distance && distanceFromOptimalPosition(cell, place_value) < distance)
 {
 found = false;
 return place_value;
 }
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }
 }

“Distance from the optimal position” is just the difference between where the key would go and where it is, measured modulo the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:597
 size_t ALWAYS_INLINE distanceFromOptimalPosition(const Cell & cell, size_t place_value) const
 {
 return grower.place(place_value - grower.place(cell.getHash(*this)));
 }

Robin Hood ordering also changes insertion: the new key may have to be placed before an existing one, which means shifting the rest of the chain to the right:

 cpp
 
 
// src/Common/HashTable/HashTable.h:664
 void NO_INLINE shiftCellsRight(size_t place_value)
 {
 size_t empty_place = findEmptyCell(place_value);

 if (empty_place < place_value)
 {
 /// The chain wraps around the end of the buffer.
 size_t last_place = grower.bufSize() - 1;
 memmove(static_cast<void *>(&buf[1]), static_cast<const void *>(&buf[0]), empty_place * sizeof(Cell));
 memcpy(static_cast<void *>(&buf[0]), static_cast<const void *>(&buf[last_place]), sizeof(Cell));
 empty_place = last_place;
 }

 memmove(static_cast<void *>(&buf[place_value + 1]), static_cast<const void *>(&buf[place_value]), (empty_place - place_value) * sizeof(Cell));
 buf[place_value].setZero();
 }

and the chains have to be re-sorted when the mode is switched on — by an insertion sort, since the chains are “already almost ordered” in practice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:681
 /** Reorder the elements in every collision resolution chain by their optimal positions, as required in the Robin Hood mode.
 * Used after the elements were placed by the ordinary linear probing: on resize, or when switching to the Robin Hood mode.
 * It is an insertion sort, which is fast for the chains that are already almost ordered.
 */

Deletion without tombstones

Deleting from an open-addressing table usually means writing a tombstone, so that the probe chain is not cut short. Tombstones accumulate, they need to be cleaned up, and they slow down every subsequent lookup.

ClickHouse does not use tombstones. Instead it moves the elements of the chain:

 cpp
 
 
// src/Common/HashTable/HashTable.h:731
 /// Remove the element at place_value without calling its destructor, moving the following elements of the collision resolution chain to keep them reachable.
 /// Not inlined to keep the code of inserts small.
 void eraseCellAt(size_t erased_key_position)
 {
 /// We need to guarantee loop termination because there will be empty position
 chassert(m_size < grower.bufSize());

 size_t next_position = erased_key_position;

 /**
 * During element deletion there is a possibility that the search will be broken for one
 * of the following elements, because this place erased_key_position is empty. We will check
 * next_element. Consider a sequence from (erased_key_position, next_element], if the
 * optimal_position of next_element falls into it, then removing erased_key_position
 * will not break search for next_element.
 * If optimal_position of the element does not fall into the sequence (erased_key_position, next_element]
 * then deleting a erased_key_position will break search for it, so we need to move next_element
 * to erased_key_position. Now we have empty place at next_element, so we apply the identical
 * procedure for it.
 * If an empty element is encountered then means that there is no more next elements for which we can
 * break the search so we can exit.
 *
 * In the Robin Hood mode, the elements of a chain are ordered by their optimal positions,
 * and this procedure moves them one cell to the left, keeping the order.
 */

The rule the comment spells out is the classic one: an element may stay where it is if its ideal position is not before the hole; otherwise it has to slide left into the hole, which moves the hole one step further right, and the procedure repeats:

 cpp
 
 
// src/Common/HashTable/HashTable.h:798
 /// Move the element to the freed place
 memcpy(static_cast<void *>(&buf[erased_key_position]), static_cast<void *>(&buf[next_position]), sizeof(Cell));

 /// Now we have another freed place
 erased_key_position = next_position;
 }

 buf[erased_key_position].setZero();
 --m_size;
 }

This costs a memcpy per displaced element but keeps the table free of garbage: the buffer is always exactly as full as m_size says, and a probe chain never walks over a tombstone.

The allocator: zeroed and pre-faulted

HashTableAllocator is Allocator<true /* clear_memory */, true /* populate */>. The second flag is the interesting one: it pre-faults the pages of a large allocation with MADV_POPULATE_WRITE, so that the resize loop does not stop for page faults:

 cpp
 
 
// src/Common/Allocator.cpp:59
void prefaultPages([[maybe_unused]] void * buf_, [[maybe_unused]] size_t len_)
{
#if defined(MADV_POPULATE_WRITE)
 if (len_ < POPULATE_THRESHOLD)
 return;

 if (!is_supported_by_kernel) [[unlikely]]
 return;

 auto [buf, len] = adjustToPageSize(buf_, len_, ::getPageSize());
 ::madvise(buf, len, MADV_POPULATE_WRITE);
#endif
}

 cpp
 
 
// src/Common/Allocator.cpp:132
/// Constant is chosen almost arbitrarily, what I observed is 128KB is too small, 1MB is almost indistinguishable from 64MB and 1GB is too large.
extern const size_t POPULATE_THRESHOLD = std::max(Int64{16 * 1024 * 1024}, ::getPageSize());

The rationale is spelled out in HashTableAllocator.h itself: the whole allocation will be used, so pre-faulting it costs nothing extra and removes a source of latency spikes from the middle of a rehash. The kernel support check is a version test (Linux 5.14+), not an EINVAL probe, because madvise returns EINVAL for many unrelated reasons.

Note that calloc and the “zero means empty” invariant fit together perfectly: calloc of a large block is free on Linux for big allocations (the kernel hands out zero pages), so the “clear the buffer” requirement that the emptiness protocol imposes costs nothing at all.

Prefetching

A hash table probe is a random memory access; at 90% load factor with Robin Hood ordering, a lookup of a missing key can touch several cache lines. ClickHouse tries to hide that latency by prefetching the cell a few iterations ahead. The helper measures its own iteration latency to decide how far ahead to look:

 cpp
 
 
// src/Common/HashTable/Prefetching.h:33
class PrefetchingHelper
{
public:
 size_t calcPrefetchLookAhead()
 {
 static constexpr auto assumed_load_latency_ns = 100;
 static constexpr auto just_coefficient = 4;
 static constexpr auto numerator = just_coefficient * assumed_load_latency_ns * iterations_to_measure;
 size_t elapsed_ns = watch.elapsedNanoseconds();
 if (unlikely(elapsed_ns == 0))
 return max_look_ahead_value;
 size_t look_ahead = (numerator + elapsed_ns - 1) / elapsed_ns;
 return std::clamp<size_t>(look_ahead, min_look_ahead_value, max_look_ahead_value);
 }

 static constexpr size_t getInitialLookAheadValue() { return min_look_ahead_value; }

 static constexpr size_t iterationsToMeasure() { return iterations_to_measure; }

private:
 static constexpr size_t iterations_to_measure = 100;
 static constexpr size_t min_look_ahead_value = 4;
 static constexpr size_t max_look_ahead_value = 32;

The look-ahead is thus between 4 and 32 cells, tuned to the machine: the formula is 4 * 100ns * 100 / elapsed, i.e. four times the assumed memory latency, expressed in iterations.

The prefetching iterator of the table uses it, and also prefetches the key bytes of the cell it is about to look at, not just the cell:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1021
 void prefetch()
 {
 static_assert(CouldPrefetchKey<cell_type>);
 auto * buf_end = container->buf + container->grower.bufSize();
 iter_count++;
 if (prefetch_ptr < buf_end && prefetched_count < prefetch_ahead) [[likely]]
 {
 if (iter_count == DB::PrefetchingHelper::iterationsToMeasure()) [[unlikely]]
 prefetch_ahead = prefetching.calcPrefetchLookAhead();
 auto n = prefetch_ahead - prefetched_count;
 cell_type * last_ptr = nullptr;
 for (size_t i = 0; i < n; ++i)
 {
 while (prefetch_ptr < buf_end && prefetch_ptr->isZero(*container))
 ++prefetch_ptr;

 if (prefetch_ptr < buf_end) [[likely]]
 {
 last_ptr = prefetch_ptr;
 prefetch_ptr++;
 prefetched_count++;
 }
 else
 break;
 }

 if (last_ptr) [[likely]]
 keyPrefetch(last_ptr->getKey());
 }
 }

keyPrefetch touches up to four cache lines of the key data:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:182
inline void keyPrefetch(const std::string_view key)
{
 const size_t cache_line_size = 64;
 // Prefetch up to 4 cache lines.
 const size_t max_prefetch = cache_line_size * 4;
 const size_t n = std::min(key.size(), max_prefetch);
 const char * ptr = key.data();
 const char * end = ptr + n;
 for (; ptr < end; ptr += cache_line_size)
 {
 __builtin_prefetch(ptr);
 }
}

Keys that live outside the table

Not every key can live inside the cell. A std::string_view key points at data owned by somebody else — often a column that will be gone by the time the aggregation is finalised. ClickHouse solves this with a key holder: a small policy object that knows how to make a temporary key persistent, and when to throw it away.

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:53
/**
 * Returns the key. Can return the temporary key initially.
 * After the call to keyHolderPersistKey(), must return the persistent key.
 */
template <typename Key>
inline Key & ALWAYS_INLINE keyHolderGetKey(Key && key) { return key; } /// NOLINT(bugprone-return-const-ref-from-parameter)

/**
 * Make the key persistent. keyHolderGetKey() must return the persistent key
 * after this call.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderPersistKey(Key &&) {}

/**
 * Discard the key. Calling keyHolderGetKey() is ill-defined after this.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderDiscardKey(Key &&) {}

The default is a no-op: for a key stored by value, there is nothing to do, and the branch is optimised away. The ArenaKeyHolder version copies the bytes into an Arena when the key turns out to be new:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:98
inline void ALWAYS_INLINE keyHolderPersistKey(DB::ArenaKeyHolder & holder)
{
 // Normally, our hash table shouldn't ask to persist a zero key,
 // but it can happened in the case of clearable hash table (ClearableHashSet, for example).
 // The clearable hash table doesn't use zero storage and
 // distinguishes empty keys by using cell version, not the value itself.
 // So, when an empty std::string_view is inserted in ClearableHashSet we'll get here key of zero size.
 holder.key = std::string_view{holder.pool.insert(holder.key.data(), holder.key.size()), holder.key.size()};
}

The elegance is in where the decision is made. The hash table knows whether the key it just found was new; the caller does not. So the table calls keyHolderDiscardKey when the key already existed (nothing to persist, and the arena allocation is rolled back) and keyHolderPersistKey when it did not:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1386
 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);

The hash function has to be good

Because the table uses linear probing and the low bits of the hash to pick the bucket, a poor hash function is catastrophic: the low bits of a timestamp-like key are nearly constant, and every key lands in the same few cells. The header says so bluntly:

 cpp
 
 
// src/Common/HashTable/Hash.h:15
/** Hash functions that are better than the trivial function std::hash.
 *
 * Example: when we do aggregation by the visitor ID, the performance increase is more than 5 times.
 * This is because of following reasons:
 * - in Metrica web analytics system, visitor identifier is an integer that has timestamp with seconds resolution in lower bits;
 * - in typical implementation of standard library, hash function for integers is trivial and just use lower bits;
 * - traffic is non-uniformly distributed across a day;
 * - we are using open-addressing linear probing hash tables that are most critical to hash function quality,
 * and trivial hash function gives disastrous results.
 */

The default integer hash is the MurmurHash finalizer:

 cpp
 
 
// src/Common/HashTable/Hash.h:26
/** Taken from MurmurHash. This is Murmur finalizer.
 * Faster than intHash32 when inserting into the hash table UInt64 -> UInt64, where the key is the visitor ID.
 */
inline UInt64 intHash64(UInt64 x)
{
 x ^= x >> 33;
 x *= 0xff51afd7ed558ccdULL;
 x ^= x >> 33;
 x *= 0xc4ceb9fe1a85ec53ULL;
 x ^= x >> 33;

 return x;
}

Five instructions and two multiplies turn “the low bits are constant” into “the low bits are random”. For string keys and general keys, the CRC32 hardware instruction is used where it exists — three cycles of latency and one per cycle of throughput, which is why the string tables use it even though CRC32 is a poor hash in the abstract:

 cpp
 
 
// src/Common/HashTable/Hash.h:40
/** CRC32C is not very high-quality as a hash function,
 * according to avalanche and bit independence tests (see SMHasher software), as well as a small number of bits,
 * but can behave well when used in hash tables,
 * due to high speed (latency 3 + 1 clock cycle, throughput 1 clock cycle).
 * Works only with SSE 4.2 support.
 */

Trade-offs

What the “zero means empty” scheme buys:

Zero bytes of metadata per cell. A HashMapCell<UInt64, T*> is 16 bytes, all of it payload. A scheme with an occupancy bit would need to steal a bit from the key or add a byte.
A free calloc. The kernel can hand out pre-zeroed pages; the emptiness protocol costs nothing at allocation time.
No tombstones, ever. Deletion moves cells instead of marking them, so the table never degrades.
A cheap scan. for (auto & cell : buf) if (!cell.isZero()) ... is the iteration order; no indirection, no linked list.

What it costs:

The key must be zero-comparable, and bit-comparable. Floating point keys are compared with memcmp, and NaN semantics have to be papered over.
The zero key is special. It lives in a separate slot, it is visited first during iteration, and its handling adds a branch to every insert and lookup.
The hash must be excellent, because linear probing amplifies clustering, and the low bits of the hash are used directly.
Rehashing is a memcpy storm. Growing the table means moving every cell, and the wrap-around case needs a second pass over the head of the new buffer.

And the recent Robin Hood addition changes the balance again: it trades a little CPU (reordering chains, occasionally shifting cells on insert) for a halving of the memory footprint at large sizes — which, for a table that is probed randomly, is exactly the right trade, because the cost is cache misses, not instructions.

How ClickHouse JIT-Compiles Scalar Expressions with LLVM

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every columnar database pays the same tax: a SELECT is not a loop over rows, it is a graph of actions, and every action dispatches through a virtual call per block. The block is the unit of work in ClickHouse (65 409 rows by default), so the dispatch overhead is paid 65 409 times less often than in a row-at-a-time engine — but for a narrow expression like a + b > c the arithmetic itself is a couple of instructions per row, and the interpreter’s per-node costs (a virtual dispatch, a column-type switch, a ColumnPtr copy) dominate.

ClickHouse’s answer is a JIT compiler that is on by default since 25.5:

 cpp
 
 
// src/Core/Settings.cpp:1459
 DECLARE(Bool, compile_expressions, true, R"(
Compile some scalar functions and operators to native code.
)", 0, \
 {"25.5", false, true, "We believe that the LLVM infrastructure behind the JIT compiler is stable enough to enable this setting by default."}) \

This article follows the whole path: from an ActionsDAG node, through a small intermediate representation (CompileDAG), into hand-built LLVM IR, through LLVM’s O3 pipeline, into an executable page, and finally into a tight loop that runs over raw column data.

The switch

The JIT is not free: compiling a module costs milliseconds. So ClickHouse only compiles an expression after it has been seen a few times. The threshold is a setting, default 3:

 cpp
 
 
// src/Core/Settings.cpp:1463
 DECLARE(UInt64, min_count_to_compile_expression, 3, R"(
Minimum count of executing same expression before it is get compiled.
)", 0) \

The counter is a global hash map keyed by a hash of the expression DAG, so the “same expression” test is structural, not textual:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:299
static FunctionBasePtr compile(
 const CompileDAG & dag,
 size_t min_count_to_compile_expression)
{
 static std::unordered_map<UInt128, UInt64, UInt128Hash> counter;
 static std::mutex mutex;

 auto hash_key = dag.hash();
 {
 std::lock_guard lock(mutex);
 if (counter[hash_key]++ < min_count_to_compile_expression)
 return nullptr;
 }

Note the ++ inside the comparison: the counter is incremented on every call, and the expression is compiled on the fourth sighting (with the default of 3). The counter never decays, so a hot expression stays hot forever.

The trigger point is the construction of ExpressionActions, which is where a DAG becomes an executable list of actions:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:74
#if USE_EMBEDDED_COMPILER
 if (settings.can_compile_expressions && settings.compile_expressions == CompileExpressions::yes)
 actions_dag.compileExpressions(settings.min_count_to_compile_expression, lazy_executed_nodes);
#endif

Note the ordering: short-circuit functions are analysed first and the set of “lazily executed” nodes is computed before compiling, because a node that is only evaluated under a condition must not be folded into a compiled straight-line expression.

CompileDAG: a tiny IR

ActionsDAG is the general-purpose graph ClickHouse uses for every expression: it carries column nodes, aliases, lambdas, arrays, and functions. It is too rich to translate to LLVM directly, so the JIT first flattens it into CompileDAG, which has exactly three kinds of node:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.h:36
 enum class CompileType : uint8_t
 {
 INPUT = 0,
 CONSTANT = 1,
 FUNCTION = 2,
 };

The flattening is a post-order DFS over the ActionsDAG that stops at anything it cannot compile. getCompilableDAG in ExpressionJIT.cpp walks the DAG with an explicit stack and produces, for each candidate node, a CompileDAG plus the list of children that became inputs:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:404
static CompileDAG getCompilableDAG(
 const ActionsDAG::Node * root,
 ActionsDAG::NodeRawConstPtrs & children,
 const std::unordered_set<const ActionsDAG::Node *> & lazy_executed_nodes)
{
 /// Extract CompileDAG from root actions dag node.

 CompileDAG dag;

 std::unordered_map<const ActionsDAG::Node *, size_t> visited_node_to_compile_dag_position;

A node is a leaf of the compiled fragment when it is a constant, when it is not compilable at all, or when its function declares that the argument does not participate in compilation:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:430
 bool is_compilable_constant = isCompilableConstant(*node);
 bool is_compilable_function = isCompilableFunction(*node, lazy_executed_nodes);

 if (!is_compilable_function || is_compilable_constant || frame.skip_compile)
 {
 CompileDAG::Node compile_node;
 compile_node.function = node->function_base;
 compile_node.result_type = node->result_type;

 if (is_compilable_constant)
 {
 compile_node.type = CompileDAG::CompileType::CONSTANT;
 compile_node.column = node->column;
 }
 else if (frame.skip_compile)
 {
 compile_node.type = CompileDAG::CompileType::CONSTANT;
 compile_node.column = node->column;
 compile_node.skip_compile = true;
 }
 else
 {
 compile_node.type = CompileDAG::CompileType::INPUT;
 children.emplace_back(node);
 }

The skip_compile flag is how “arguments that don’t participate” are handled: the argument is still passed to the function at runtime (the C++ function signature has all its arguments), but its value is not computed by the compiled code. This is what makes if(cond, f(x), g(x))-style functions compile at all: the lazily evaluated branch is a runtime argument, not a compiled computation.

Choosing what to compile

ActionsDAG::compileFunctions is the part that decides which nodes become compiled fragments. It is not “compile everything”: it computes, for each node, how many compilable descendants it has, and only compiles nodes that have at least one compilable child and whose parents are not themselves compilable (so a whole subtree is compiled once, at its root, not piece by piece):

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:594
 for (const auto & node : nodes)
 {
 auto & node_data = node_to_data[&node];
 bool node_is_valid_for_compilation = node_data.is_compilable_in_isolation && node_data.compilable_children_size > 0;

 for (const auto & child : node.children)
 node_to_data[child].all_parents_compilable &= node_is_valid_for_compilation;
 }

 for (const auto & output_node : outputs)
 {
 /// Force output nodes to compile
 node_to_data[output_node].all_parents_compilable = false;
 }

Then the fragments are sorted so that a parent is compiled before its children — otherwise the same subexpression would be compiled twice under two different names:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:626
 /** Sort nodes before compilation using their children size to avoid compiling subexpression before compile parent expression.
 * This is needed to avoid compiling expression more than once with different names because of compilation order.
 */
 ::sort(nodes_to_compile.begin(), nodes_to_compile.end(), [&](const Node * lhs, const Node * rhs)
 {
 return node_to_data[lhs].children_size > node_to_data[rhs].children_size;
 });

and the compiled node replaces the original in place:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:649
 node->type = ActionsDAG::ActionType::FUNCTION;
 node->function_base = fn;
 node->function = fn->prepare(arguments);
 node->children.swap(new_children);
 node->is_function_compiled = true;
 node->column = nullptr;

From CompileDAG to LLVM IR

CompileDAG::compile emits the IR for the fragment. For a function node it recursively materialises the argument values and asks the function itself to emit its own IR:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.cpp:45
 case CompileType::FUNCTION:
 {
 ValuesWithType temporary_values;
 temporary_values.reserve(node.arguments.size());
 for (auto argument_index : node.arguments)
 temporary_values.emplace_back(compiled_values[argument_index]);

 ValueWithType compiled_value{node.function->compile(builder, temporary_values), node.function->getResultType()};
 if (!node.result_type->equals(*node.function->getResultType()))
 compiled_values[compiled_values_index] = {nativeCast(b, compiled_value, node.result_type), node.result_type};
 else
 compiled_values[compiled_values_index] = std::move(compiled_value);

 break;
 }

Constants are baked into the module as native values, inputs are the loop’s loaded values, and functions call IFunction::compile.

The function contract

Each SQL function that wants to be JIT-compilable implements compileImpl, which receives the already-typed argument values as LLVM values and returns an LLVM value. A binary arithmetic function looks like this:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3821
 llvm::Value * compileImpl(llvm::IRBuilderBase & builder, const ValuesWithType & arguments, const DataTypePtr & result_type) const override
 {
 chassert(2 == arguments.size());

 auto denull_left_type = removeNullable(arguments[0].type);
 auto denull_right_type = removeNullable(arguments[1].type);
 llvm::Value * result = nullptr;

and ends with the actual operation, dispatched on the native type:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3846
 auto & b = static_cast<llvm::IRBuilder<> &>(builder);
 auto * lval = nativeCast(b, arguments[0], result_type);
 auto * rval = nativeCast(b, arguments[1], result_type);
 result = OpSpec::compile(b, lval, rval, is_signed_v<typename ResultDataType::FieldType>);
 return true;

NULLs: the ternary trick

Nullable(T) is represented in IR as a two-element struct { T value, UInt8 is_null }. Handling NULLs in compiled code is where most of the complexity lives, because ClickHouse’s default implementation for NULLs is “if any argument is NULL, the result is NULL” — which is not what and/or want (three-valued logic).

The framework’s answer is to compile the denulled function and then OR the null flags back in:

 cpp
 
 
// src/Functions/IFunction.cpp:1281
 auto denulled_arguments_types = removeNullables(arguments_types);
 if (useDefaultImplementationForNulls() && denulled_arguments_types)
 {
 auto & b = static_cast<llvm::IRBuilder<> &>(builder);

 ValuesWithType unwrapped_arguments;
 unwrapped_arguments.reserve(arguments.size());

 VectorWithMemoryTracking<llvm::Value *> is_null_values;

 auto skip_arguments = getArgumentsThatDontParticipateInCompilation(arguments_types);
 for (size_t i = 0; i < arguments.size(); ++i)
 {
 const auto & argument = arguments[i];
 llvm::Value * unwrapped_value = argument.value;

 bool skip_compile = (std::find(skip_arguments.begin(), skip_arguments.end(), i) != skip_arguments.end());
 if (argument.type->isNullable())
 {
 unwrapped_value = skip_compile ? nullptr : b.CreateExtractValue(argument.value, {0});
 is_null_values.emplace_back(skip_compile ? nullptr : b.CreateExtractValue(argument.value, {1}));
 }

 unwrapped_arguments.emplace_back(unwrapped_value, (*denulled_arguments_types)[i]);
 }

and then, at the end, the null flags are merged into the result:

 cpp
 
 
// src/Functions/IFunction.cpp:1317
 for (auto * is_null_value : is_null_values)
 {
 if (is_null_value)
 nullable_structure_result_null = b.CreateOr(nullable_structure_result_null, is_null_value);
 }

 return b.CreateInsertValue(nullable_structure_with_result_value, nullable_structure_result_null, {1});

Functions with special NULL semantics (the logical ones) opt out of this and do the work themselves, using an explicit three-valued encoding:

 cpp
 
 
// src/Functions/FunctionsLogical.h:267
 llvm::Value * compileImpl(llvm::IRBuilderBase & builder, const ValuesWithType & values, const DataTypePtr & result_type) const override
 {
 chassert(!values.empty());

 auto & b = static_cast<llvm::IRBuilder<> &>(builder);
 if constexpr (Impl::specialImplementationForNulls())
 {
 if (result_type->isNullable())
 {
 /// First we need to cast all values to ternary logic
 llvm::Value * ternary_result = nativeTernaryCast(b, values[0]);
 for (size_t i = 1; i < values.size(); ++i)
 {
 llvm::Value * casted_value = nativeTernaryCast(b, values[i]);
 ternary_result = Impl::ternaryApply(b, ternary_result, casted_value);
 }

 /// Then transform ternary logic to struct which represents nullable result
 llvm::Value * is_null = b.CreateICmpEQ(ternary_result, b.getInt8(Ternary::Null));
 llvm::Value * is_true = b.CreateICmpEQ(ternary_result, b.getInt8(Ternary::True));

This is a nice illustration of the general ClickHouse pattern: the interpreter path handles NULLs with a nested column representation and a mask, the compiled path handles them with a struct and a select — the semantics are the same, the representation is not.

The generated loop

The whole pipeline, from a plan node to a running loop:

 text
 
 
 ActionsDAG node CompileDAG (3 node kinds)
 (Function, Column, Alias, ...) INPUT column to read
 | CONSTANT baked-in literal
 | getCompilableDAG() FUNCTION nested call
 v
 CompileDAG --hash()--> SipHash-128 --min_count_to_compile_expression--> ?
 | (seen 4 times?)
 v compile()
 llvm::Module (IRBuilder: entry / loop / end blocks)
 |
 v O3 pipeline + RuntimeDyld
 executable page (PROT_READ|PROT_EXEC) <---- CompiledExpressionCache (weighted by code size)
 |
 v callJITFunction(rows, columns)
 for (i = 0; i < rows; ++i) { result[i] = expr(columns[0][i], columns[1][i]); }

The IR builder in compileFunction.cpp wraps the expression in a loop over the rows. The function’s ABI is void f(size_t rows, ColumnData * columns), where ColumnData is a { const char * data, const char * null_data, const char * offsets } triple — the raw pointers of the column, its null map, and (for strings) its offsets:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:88
static llvm::StructType * buildColumnDataStruct(llvm::IRBuilderBase & b)
{
 auto * char_ptr_type = b.getInt8Ty()->getPointerTo();
 return llvm::StructType::get(char_ptr_type, char_ptr_type, char_ptr_type);
}

The function itself is created with ExternalLinkage so that the linker can find it by name later, and the loop is a textbook phi-node counter:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:139
 /// Initialize loop

 auto * end = llvm::BasicBlock::Create(b.getContext(), "end", func);
 auto * loop = llvm::BasicBlock::Create(b.getContext(), "loop", func);
 b.CreateCondBr(b.CreateICmpEQ(rows_count_arg, llvm::ConstantInt::get(size_type, 0)), end, loop);

 b.SetInsertPoint(loop);

 /// Loop

 auto * counter_phi = b.CreatePHI(rows_count_arg->getType(), 2);
 counter_phi->addIncoming(llvm::ConstantInt::get(size_type, 0), entry);

Each argument is loaded from the column data with an in-bounds GEP on the row index:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:162
 auto * column_data_ptr = column.data_ptr;
 auto * column_element_value = b.CreateLoad(column.data_element_type, b.CreateInBoundsGEP(column.data_element_type, column_data_ptr, counter_phi));

 if (!type->isNullable())
 {
 arguments.emplace_back(column_element_value, type);
 continue;
 }

 auto * column_is_null_element_value = b.CreateLoad(b.getInt8Ty(), b.CreateInBoundsGEP(b.getInt8Ty(), column.null_data_ptr, counter_phi));
 auto * is_null = b.CreateICmpNE(column_is_null_element_value, b.getInt8(0));
 auto * nullable_unitialized = llvm::Constant::getNullValue(toNullableType(b, column.data_element_type));
 auto * nullable_value = b.CreateInsertValue(b.CreateInsertValue(nullable_unitialized, column_element_value, {0}), is_null, {1});
 arguments.emplace_back(nullable_value, type);

and the result is stored back into the output column, which is passed as the last element of the columns array:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:184
 auto * result_column_element_ptr = b.CreateInBoundsGEP(columns.back().data_element_type, columns.back().data_ptr, counter_phi);

 if (columns.back().null_data_ptr)
 {
 b.CreateStore(b.CreateExtractValue(result, {0}), result_column_element_ptr);
 auto * result_column_is_null_element_ptr = b.CreateInBoundsGEP(b.getInt8Ty(), columns.back().null_data_ptr, counter_phi);
 auto * is_result_column_element_null = b.CreateSelect(b.CreateExtractValue(result, {1}), b.getInt8(1), b.getInt8(0));
 b.CreateStore(is_result_column_element_null, result_column_is_null_element_ptr);
 }
 else
 {
 b.CreateStore(result, result_column_element_ptr);
 }

Running it

At runtime, LLVMExecutableFunction::executeImpl allocates an output column, gathers the raw pointers, and calls into the compiled code:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:99
 ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & result_type, size_t input_rows_count) const override
 {
 if (!canBeNativeType(*result_type))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "LLVMExecutableFunction unexpected result type in: {}", result_type->getName());

 auto result_column = result_type->createUninitializedColumnWithSize(input_rows_count);

 if (input_rows_count)
 {
 std::vector<ColumnData> columns(arguments.size() + 1);
 Columns columns_backup;

 for (size_t i = 0; i < arguments.size(); ++i)
 {
 auto column = arguments[i].column->convertToFullColumnIfConst();
 columns_backup.emplace_back(column);
 columns[i] = getColumnData(column.get());
 }

 columns[arguments.size()] = getColumnData(result_column.get());

 auto jit_compiled_function = compiled_function_holder->compiled_function.compiled_function;
 callJITFunction(jit_compiled_function, input_rows_count, columns.data());

getColumnData is where the layout knowledge lives — a String column is not a contiguous array of values, it is a chars buffer plus an offsets buffer, and the JIT comparator indexes both directly:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:63
 /// skip null key data for one nullable key optimization
 if (WhichDataType(column->getDataType()).isString())
 {
 /// For String columns, use base pointers to chars and offsets arrays directly.
 /// The JIT comparator uses absolute row indices into these arrays, so both
 /// pointers must be to the start of their respective arrays regardless of skip_rows.
 const auto * string_column = assert_cast<const ColumnString *>(column);
 result.data = reinterpret_cast<const char *>(string_column->getChars().data());
 result.offset_data = reinterpret_cast<const char *>(string_column->getOffsets().data());
 }

For a String the “element type” is therefore a {chars*, offsets*, index} struct, not a char*:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:94
static llvm::StructType * buildStringRefType(llvm::IRBuilderBase &b)
{
 auto * data_ptr_type = b.getInt8Ty()->getPointerTo();
 auto * offset_ptr_type = b.getInt8Ty()->getPointerTo();
 auto * index_type = b.getInt64Ty();
 return llvm::StructType::get(data_ptr_type, offset_ptr_type, index_type);
}

The JIT itself

CHJIT is a small, self-contained LLVM JIT: it uses RuntimeDyld to link the object file into memory it manages itself.

The whole compilation is serialised by a mutex, because LLVM’s context is shared:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:480
CHJIT::CompiledModule CHJIT::compileModule(std::function<void (llvm::Module &)> compile_function)
{
 std::lock_guard lock(jit_lock);

The module is then run through a full O3 pipeline with SLP vectorisation enabled:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:620
void CHJIT::runOptimizationPassesOnModule(llvm::Module & module) const
{
 llvm::LoopAnalysisManager lam;
 llvm::FunctionAnalysisManager fam;
 llvm::CGSCCAnalysisManager cgam;
 llvm::ModuleAnalysisManager mam;

 llvm::PipelineTuningOptions pto;
 pto.SLPVectorization = true;

 /// Pass the actual TargetMachine so that all optimization passes
 /// (including target-specific ones) have proper target information.
 llvm::PassBuilder pb(machine.get(), pto);

 pb.registerModuleAnalyses(mam);
 pb.registerCGSCCAnalyses(cgam);
 pb.registerFunctionAnalyses(fam);
 pb.registerLoopAnalyses(lam);
 pb.crossRegisterProxies(lam, fam, cgam, mam);

 llvm::ModulePassManager mpm = pb.buildPerModuleDefaultPipeline(llvm::OptimizationLevel::O3);
 mpm.run(module, mam);
}

The target machine is the host’s, with the host CPU and features, so the compiled code uses whatever vector instructions the machine has:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:644
std::unique_ptr<llvm::TargetMachine> CHJIT::getTargetMachine()
{
 static std::once_flag llvm_target_initialized;
 std::call_once(llvm_target_initialized, []()
 {
 llvm::InitializeNativeTarget();
 llvm::InitializeNativeTargetAsmPrinter();
 llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
 });

 std::string error;
 auto cpu = llvm::sys::getHostCPUName();
 auto triple = llvm::Triple(llvm::sys::getProcessTriple());

Memory management

JIT code needs executable pages. ClickHouse does not use mmap directly; it allocates page-aligned blocks with posix_memalign and flips their protection with mprotect once the linking is done:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:287
 void allocateNextPageBlock(size_t size)
 {
 size_t pages_to_allocate_size = ((size / page_size) + 1) * 2;
 size_t allocate_size = page_size * pages_to_allocate_size;

 void * buf = nullptr;
 int res = posix_memalign(&buf, page_size, allocate_size);

The double allocation (+1 then *2 pages) gives both slack for the alignment request and room to grow without a new mprotect dance. Three separate arenas exist — read-write data, read-only data, and executable code — and the memory manager marks them appropriately at the end:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:353
 bool finalizeMemory(std::string *) override
 {
 ro_page_arena.protect(PROT_READ);
 ex_page_arena.protect(PROT_READ | PROT_EXEC);
 return true;
 }

On ARM the code has to invalidate the instruction cache after mprotect, and some ARM CPUs fault if the page is not readable while the cache is being flushed — hence the special case:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:211
# if defined(__arm__) || defined(__aarch64__)
 /// Certain ARM implementations treat icache clear instruction as a memory read,
 /// and CPU segfaults on trying to clear cache on !PROT_READ page.
 /// Therefore we need to temporarily add PROT_READ for the sake of flushing the instruction caches.
 if (invalidate_cache && !(protection_flags & PROT_READ))
 {
 int res = mprotect(block.base(), block.blockSize(), protection_flags | PROT_READ);

Symbol resolution

The compiled code may call back into ClickHouse (e.g. memcpy, or the 128-bit integer helpers for Int128 arithmetic). A JITSymbolResolver maps those names to addresses, and the trick of storing mangled names removes a platform-specific lookup path:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:397
 void registerSymbol(const std::string & symbol_name, void * symbol)
 {
 std::string mangled_name;
 llvm::raw_string_ostream mangled_name_stream(mangled_name);
 llvm::Mangler::getNameWithPrefix(mangled_name_stream, symbol_name, layout);
 mangled_name_stream.flush();
 symbol_name_to_symbol_address[mangled_name] = symbol;
 }

The 128-bit helpers come in signed/unsigned pairs, and the comment explains why both halves must be registered:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:76
/// IMPORTANT: every libcall is registered as a signed/unsigned pair. LLVM picks the
/// signed or unsigned variant based on whether the IR uses `fptosi`/`sitofp`/`sdiv`/`srem`
/// or `fptoui`/`uitofp`/`udiv`/`urem`. Forgetting the unsigned half compiles fine but
/// fails the JIT link at run time with `Could not find symbol __fixunsdfti` (or similar).
/// Keep the lists below symmetric.

Caching the compiled code

Compiling the same expression twice is pure waste, so compiled modules are cached by the CompileDAG hash:

 cpp
 
 
// src/Interpreters/JIT/CompiledExpressionCache.h:37
class CompiledExpressionCache : public CacheBase<UInt128, CompiledExpressionCacheEntry, UInt128Hash, CompiledFunctionWeightFunction>
{
public:
 using Base = CacheBase<UInt128, CompiledExpressionCacheEntry, UInt128Hash, CompiledFunctionWeightFunction>;
 using Base::Base;
};

The cache is weight-aware: an entry’s weight is the number of bytes of machine code it occupies, so evicting a big module frees proportionally more memory.

The hash that keys the cache is a structural hash over the DAG:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.cpp:126
UInt128 CompileDAG::hash() const
{
 SipHash hash;
 for (const auto & node : nodes)
 {
 hash.update(node.type);

 const auto & result_type_name = node.result_type->getName();
 hash.update(result_type_name.size());
 hash.update(result_type_name);

 switch (node.type)
 {
 case CompileType::CONSTANT:
 {
 node.column->getDataColumn().updateHashWithValue(0, hash);
 break;
 }

so two syntactically different but semantically identical expressions hash the same, and the same expression with a different constant does not.

Beyond scalar expressions

The same machinery compiles two more things.

Aggregate functions. A whole aggregation is compiled as five entry points:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:544
CompiledAggregateFunctions compileAggregateFunctions(CHJIT & jit, const std::vector<AggregateFunctionWithOffset> & functions, std::string functions_dump_name)
{
 Stopwatch watch;

 std::string create_aggregate_states_functions_name = functions_dump_name + "_create";
 std::string add_aggregate_states_functions_name = functions_dump_name + "_add";
 std::string add_aggregate_states_functions_name_single_place = functions_dump_name + "_add_single_place";
 std::string merge_aggregate_states_functions_name = functions_dump_name + "_merge";
 std::string insert_aggregate_states_functions_name = functions_dump_name + "_insert";

_create builds the aggregate states, _add folds a row into them, _merge merges two states, _insert writes the final values into the result columns. The _add function is generated twice: once for the “many places” layout (a hash table of aggregate states) and once for the “single place” layout (one state per group):

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:556
 compileCreateAggregateStatesFunctions(module, functions, create_aggregate_states_functions_name);
 compileAddIntoAggregateStatesFunctions(module, functions, add_aggregate_states_functions_name, AddIntoAggregateStatesPlacesArgumentType::MultiplePlaces);
 compileAddIntoAggregateStatesFunctions(module, functions, add_aggregate_states_functions_name_single_place, AddIntoAggregateStatesPlacesArgumentType::SinglePlace);
 compileMergeAggregatesStates(module, functions, merge_aggregate_states_functions_name);
 compileInsertAggregatesIntoResultColumns(module, functions, insert_aggregate_states_functions_name);

Sort comparators. A multi-column sort key is compiled into a single comparator function that returns int8. The IR is a chain of basic blocks, one per sort column, joined by a phi node — a hand-written decision tree:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:742
 /** 1. If it is last condition block move to join block.
 * 2. If column elements are not equal move to join block.
 * 3. If column elements are equal move to next column condition.
 */
 if (i == columns_size - 1)
 b.CreateBr(comparator_join);
 else
 b.CreateCondBr(b.CreateICmpEQ(result, lhs_equals_rhs_result), comparator_steps_and_results[i + 1].first, comparator_join);

and the function’s return attribute is not a detail but a correctness requirement on Apple’s arm64 ABI:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:608
 /// The result is read back through an `int8_t (*)(...)` function pointer, so the i8 return
 /// must obey the C `int8_t` calling convention. This matters only on Darwin/Apple arm64:
 /// Apple's arm64 ABI makes the *callee* sign-extend sub-word return values and the caller
 /// relies on it (it emits no `sxtb` after the call). Without `signext` the JIT callee leaves
 /// the upper bits unset, so e.g. a -1 result is read as 255 and NULL ordering breaks.
 /// Linux arm64 (AAPCS64) is unaffected because there the *caller* sign-extends the return
 /// itself (`sxtb w0, w0`); x86-64 is likewise a no-op.
 comparator_func->addRetAttr(llvm::Attribute::SExt);

Trade-offs

What the JIT buys: the per-block dispatch, the per-node type switches and the column representation overhead disappear for the compiled fragment; the loop is a plain memory-bound loop that the CPU can auto-vectorise. For a filter like WHERE user_id = 42 AND ts > now() - INTERVAL 1 HOUR, this is the difference between a few hundred nanoseconds per block and a few microseconds.

What it costs:

Memory: every compiled module occupies executable pages, and the cache is weight-limited by machine-code size. A server with thousands of distinct hot expressions holds a lot of RX pages.
Compile latency: the first four executions of an expression pay for the compile; the O3 pipeline is not cheap (hundreds of microseconds to milliseconds). The min_count_to_compile_expression threshold exists exactly to amortise this.
A second implementation of every function’s semantics: compileImpl and executeImpl must agree. Bugs where the JIT path and the interpreter path differ are a real class of ClickHouse bugs — the Nullable handling above is where most of the subtlety lives.

Restricted type coverage: only “native” types (fixed-width integers, floats, Date, DateTime, and String/FixedString through their special reference structs) can be compiled. Decimal is explicitly excluded from the arithmetic family:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3840
 if constexpr (
 !std::is_same_v<ResultDataType, InvalidType> && !IsDataTypeDecimal<ResultDataType>
 && !IsDataTypeDecimal<LeftDataType> && !IsDataTypeDecimal<RightDataType> && OpSpec::compilable)

Locking: CHJIT::compileModule holds a global mutex, so the compile of one expression blocks the compile of another. The compiled code is fully parallel — it is only the compilation that is serialised.

And the escape hatch is one setting:

 sql
 
SELECT count() FROM events WHERE user_id = 42 SETTINGS compile_expressions = 0;

How ClickHouse Evaluates IN with Hash Sets

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

x IN (1, 2, 3) looks like a comparison. x IN (SELECT user_id FROM logins WHERE …) looks like a join. Under the hood both become the same thing: a hash set, built once, probed once per row of the left-hand side.

The interesting part is not “it is a hash table” — it is which hash table, because ClickHouse picks a different data structure for every shape of the right-hand side, and because the set can be large enough that it no longer fits in memory and has to be spilled to disk behind Bloom filters.

The set is a plan artifact, not a function argument

The IN operator is a normal SQL function whose second argument is not a value but a set object:

 cpp
 
 
// src/Functions/in.cpp:145
 ColumnPtr column_set_ptr = arguments[1].column;
 const ColumnSet * column_set = tryGetColumnSet(column_set_ptr);
 if (!column_set)
 throw Exception(ErrorCodes::ILLEGAL_COLUMN, "Second argument for function '{}' must be Set; found {}",
 getName(), column_set_ptr->getName());

 auto future_set = column_set->getData();
 if (!future_set)
 {
 if (dry_run)
 return ColumnUInt8::create(input_rows_count, static_cast<UInt8>(0));

 throw Exception(ErrorCodes::LOGICAL_ERROR, "No Set is passed as the second argument for function '{}'", getName());
 }

The set arrives as a ColumnSet holding a FutureSet — a promise, because the set may not be built yet when the function’s type is resolved. The set is built by a separate step in the pipeline, and the function then asks for it:

 cpp
 
 
// src/Functions/in.cpp:160
 auto set = future_set->get();
 if (!set)
 {
 if (dry_run)
 return ColumnUInt8::create(input_rows_count, static_cast<UInt8>(0));

 throw Exception(ErrorCodes::LOGICAL_ERROR, "Not-ready Set is passed as the second argument for function '{}'", getName());
 }

There are three kinds of FutureSet:

 cpp
 
 
// src/Interpreters/PreparedSets.h:66
class FutureSet
{
public:
 virtual ~FutureSet() = default;

 /// Returns set if set is ready (created and filled) or nullptr if not.
 virtual SetPtr get() const = 0;
 /// Returns set->getElementsTypes(), even if set is not created yet.
 virtual DataTypes getTypes() const = 0;
 /// If possible, return set with stored elements useful for PK analysis.
 virtual SetPtr buildOrderedSetInplace(const ContextPtr & context) = 0;

FutureSetFromTuple — the right-hand side is a literal list, IN (1, 2, 3). It is built during analysis, because the values are already known.
FutureSetFromSubquery — the right-hand side is a SELECT. It is built during execution, by a CreatingSetStep that runs before the steps that use it.
FutureSetFromStorage — the right-hand side is an ENGINE = Set table. The set already exists; the query only reads it.

The literal case is the interesting one for the “prepare once, use many” idea, because the same literal list may appear many times in a query:

 cpp
 
 
// src/Interpreters/ActionsVisitor.cpp:557
 auto set_key = right_arg->getTreeHash(/*ignore_aliases=*/ true);
 if (auto set = prepared_sets.findTuple(set_key, set_element_keys))
 return set; /// Already prepared.

 return prepared_sets.addFromTuple(set_key, right_arg_func, std::move(block), context->getSettingsRef());

The key is the tree hash of the AST of the right-hand side, so x IN (1,2,3) and y IN (1,2,3) share one set. The prepared sets live in a per-query container:

 cpp
 
 
// src/Interpreters/PreparedSets.h:301
 using SetsFromTuple = std::unordered_map<Hash, std::vector<FutureSetFromTuplePtr>, Hashing>;
 using SetsFromStorage = std::unordered_map<Hash, FutureSetFromStoragePtr, Hashing>;
 using SetsFromSubqueries = std::unordered_map<Hash, FutureSetFromSubqueryPtr, Hashing>;

The hash is 128 bits of CityHash, and the container hashes it down to 64 bits:

 cpp
 
 
// src/Interpreters/PreparedSets.h:278
struct FutureSetHashing
{
 UInt64 operator()(const FutureSet::Hash & key) const { return key.low64 ^ key.high64; }
};

The shape of the thing

 text
 
 
 SELECT ... WHERE x IN (SELECT id FROM t)
 |
 v
 CreatingSetStep ---> FutureSetFromSubquery
 | (built during execution, not analysis)
 v
 ActionsDAG: FunctionIn(x, ColumnSet) ---> Set::execute(key_columns)
 |
 chooseMethod(key_columns) -+--> key8 / key16 / key32 / key64
 | (one integer key: the key IS the slot)
 +--> keys32 / keys64 / keys128 / keys256
 | (fixed-width tuple: packed into one word)
 +--> key_string / key_fixed_string
 | (one string: chars arena + saved hash)
 +--> nullable_keys128 / nullable_keys256
 | (value + null flag packed together)
 +--> hashed
 (anything else: 128-bit hash only)

Choosing the data structure

This is the part that makes ClickHouse’s IN fast. The set’s key type is not “some value” — it is a tuple of columns with concrete types, and each shape deserves a different hash table.

SetVariants::chooseMethod inspects the key columns and picks one of thirteen implementations:

 cpp
 
 
// src/Interpreters/SetVariants.h:417
 #define APPLY_FOR_SET_VARIANTS(M) \
 M(key8) \
 M(key16) \
 M(key32) \
 M(key64) \
 M(key_string) \
 M(key_fixed_string) \
 M(keys32) \
 M(keys64) \
 M(keys128) \
 M(keys256) \
 M(nullable_keys128) \
 M(nullable_keys256) \
 M(hashed)

The decision tree, in order:

One nullable numeric key. The value and the NULL flag are packed into one 16-byte key, so the hash table needs no separate null handling:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:149
 if (has_nullable_key)
 {
 /// At least one key is nullable. Therefore we choose a method
 /// that takes into account this fact.
 if ((keys_size == 1) && (nested_key_columns[0]->isNumeric()))
 {
 /// We have exactly one key and it is nullable. We shall add it a tag
 /// which specifies whether its value is null or not.
 size_t size_of_field = nested_key_columns[0]->sizeOfValueIfFixed();
 if ((size_of_field == 1) || (size_of_field == 2) || (size_of_field == 4) || (size_of_field == 8))
 return Type::nullable_keys128;

One numeric key. A single UInt32 column gets a HashSet<UInt32> — the key is the hash-table key, no packing, no hashing beyond the CRC:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:180
 /// If there is one numeric key that fits into 64 bits
 if (keys_size == 1 && nested_key_columns[0]->isNumeric() && !nested_key_columns[0]->lowCardinality())
 {
 size_t size_of_field = nested_key_columns[0]->sizeOfValueIfFixed();
 if (size_of_field == 1)
 return Type::key8;
 if (size_of_field == 2)
 return Type::key16;
 if (size_of_field == 4)
 return Type::key32;
 if (size_of_field == 8)
 return Type::key64;

Several fixed-width keys. They are packed into 32/64/128/256 bits and the whole packed word is the key:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:199
 /// If the keys fit in N bits, we will use a hash table for N-bit-packed keys
 if (all_fixed && keys_bytes <= 4)
 return Type::keys32;
 if (all_fixed && keys_bytes <= 8)
 return Type::keys64;
 if (all_fixed && keys_bytes <= 16)
 return Type::keys128;
 if (all_fixed && keys_bytes <= 32)
 return Type::keys256;

One string key. A dedicated method with the string’s own arena:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:209
 /// If there is single string key, use hash table of it's values.
 if (keys_size == 1
 && (typeid_cast<const ColumnString *>(nested_key_columns[0])
 || (isColumnConst(*nested_key_columns[0]) && typeid_cast<const ColumnString *>(&assert_cast<const ColumnConst *>(nested_key_columns[0])->getDataColumn()))))
 return Type::key_string;

Anything else. The fallback hashes the whole tuple to 128 bits and stores only the hash — the original values are not kept:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:218
 /// Otherwise, will use set of cryptographic hashes of unambiguously serialized values.
 return Type::hashed;

The comment in the header explains the memory trade-off of that last choice:

 cpp
 
 
// src/Interpreters/SetVariants.h:357
 /** Unlike Aggregator, `concat` method is not used here.
 * This is done because `hashed` method, although slower, but in this case, uses less RAM.
 * since when you use it, the key values themselves are not stored.
 */

Presence-only vs counting

There are two flavours of the same set family. A plain IN only needs to know whether a key is present, so it uses a HashSet whose cell maps to VoidMapped. INTERSECT ALL / EXCEPT ALL need how many times a key occurs, so they use a HashMap with a UInt64 counter. The mapping is derived from the table rather than passed in, so a method stays “a set” when given a HashSet and becomes a multiset when given a HashMap:

 cpp
 
 
// src/Interpreters/SetVariants.h:30
/// The method structs below are shared between presence-only sets (a `HashSet`, whose cell reports
/// `VoidMapped`) and counting multisets (a `HashMap`, with a real mapped type, used by INTERSECT
/// ALL / EXCEPT ALL). The mapped type is derived from the table rather than passed explicitly, so a
/// method stays "a set" when given a `HashSet` and becomes a counting multiset when given a `HashMap`.
template <typename TData>
using SetMethodMapped = std::conditional_t<std::is_same_v<typename TData::mapped_type, VoidMapped>, void, typename TData::mapped_type>;

The choice of table type per key shape is visible in the variant declarations:

 cpp
 
 
// src/Interpreters/SetVariants.h:344
 std::unique_ptr<SetMethodOneNumber<UInt32, HashSet<UInt32, HashCRC32<UInt32>>>> key32;
 std::unique_ptr<SetMethodOneNumber<UInt64, HashSet<UInt64, HashCRC32<UInt64>>>> key64;
 std::unique_ptr<SetMethodString<HashSetWithSavedHash<std::string_view>>> key_string;
 std::unique_ptr<SetMethodFixedString<HashSetWithSavedHash<std::string_view>>> key_fixed_string;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt32, HashCRC32<UInt32>>>> keys32;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt64, HashCRC32<UInt64>>>> keys64;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt128, UInt128HashCRC32>>> keys128;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt256, UInt256HashCRC32>>> keys256;
 std::unique_ptr<SetMethodHashed<HashSet<UInt128, UInt128TrivialHash>>> hashed;

HashSetWithSavedHash is worth a note: for strings it stores the hash next to the string, so a probe that hits the same bucket does not have to re-hash the candidate key.

Building the set

Insertion goes through emplaceKey on the method’s state, one row at a time, with a twist: the same loop can build a deduplication filter, which records which rows were new. That filter is how the set keeps the original values around for primary-key analysis:

 cpp
 
 
// src/Interpreters/Set.cpp:370
 typename Method::State state(key_columns, key_sizes, nullptr);

 /// For all rows
 for (size_t i = 0; i < rows; ++i)
 {
 if constexpr (has_null_map)
 {
 if ((*null_map)[i])
 {
 if constexpr (build_filter)
 {
 (*out_filter)[i] = false;
 }
 continue;
 }
 }

 [[maybe_unused]] auto emplace_result = state.emplaceKey(method.data, i, variants.string_pool);

 if constexpr (build_filter)
 (*out_filter)[i] = emplace_result.isInserted();
 }

emplace_result.isInserted() is the deduplication: a row whose key was already in the set is not new, so it is filtered out of the “explicit set elements” — the copy of the values kept for index analysis.

Keeping the values is optional and bounded. When the set grows past max_elements_to_fill, the copy is dropped and the set keeps only the hashes:

 cpp
 
 
// src/Interpreters/Set.cpp:495
 bool inserted = insertFromColumnsImpl(columns, holder);
 if (inserted && fill_set_elements)
 {
 if (max_elements_to_fill && max_elements_to_fill < data->getTotalRowCount())
 {
 /// Drop filled elementes
 fill_set_elements = false;
 set_elements.clear();
 }
 else
 appendSetElements(holder);
 }

This is a graceful degradation: the set still answers IN correctly, it just can no longer be used to prune granules by primary key. The threshold is the setting use_index_for_in_with_subqueries_max_values.

Probing the set

The probe is the hot path: it runs once per row of the left-hand side.

 cpp
 
 
// src/Interpreters/Set.cpp:1002
 Arena pool;
 typename Method::State state(key_columns, key_sizes, nullptr);

 /// Clustered key columns (e.g. a primary key prefix) arrive in runs of equal consecutive
 /// rows. The consecutive-keys optimization in ColumnsHashing handles them inside `findKey`:
 /// the last-element cache compares the key with the previous row's before probing the hash
 /// table, and `HashMethodHashed` additionally compares the raw key bytes before even
 /// calculating the hash.
 for (size_t i = 0; i < rows; ++i)
 {
 if (has_null_map && (*null_map)[i])
 {
 vec_res[i] = negative;
 }
 else
 {
 auto find_result = state.findKey(method.data, i, pool);
 vec_res[i] = negative ^ find_result.isFound();
 }
 }

Two things are worth noticing.

The “last element cache”. When the input is sorted (or clustered by a primary key), consecutive rows often have the same key. The hashing state remembers the last key and its answer, so a repeat costs one comparison instead of a hash computation plus a probe. This optimisation is on for numeric keys:

 cpp
 
 
// src/Interpreters/SetVariants.h:44
template <typename FieldType, typename TData, bool use_cache = true> /// UInt8/16/32/64 for any types with corresponding bit width.
struct SetMethodOneNumber
{
 using Data = TData;
 using Key = typename Data::key_type;

 Data data;

 using State = ColumnsHashing::HashMethodOneNumber<typename Data::value_type,
 SetMethodMapped<Data>, FieldType, set_method_use_cache<Data, use_cache>>;

and off for strings, with a measurement in the comment:

 cpp
 
 
// src/Interpreters/SetVariants.h:62
/// For the case where there is one string key.
/// Unlike aggregation, the consecutive-keys cache stays disabled for string keys: a set probe's
/// per-row work is just hash+lookup, which for strings is cheaper than the extra bytewise
/// comparison the cache check performs on every miss. Measured 2026-08 on 20M-row `IN` probes:
/// enabling the cache made unclustered inputs 40% slower for short `String` keys and 24% slower
/// for `FixedString(36)` keys, while clustered inputs (runs of equal consecutive keys) improved
/// only 8-16%. In aggregation the same check is relatively cheap because the per-row work there
/// (aggregate-state updates) is much heavier.
template <typename TData>
struct SetMethodString
{
 using Data = TData;
 using Key = typename Data::key_type;

 Data data;

 using State = ColumnsHashing::HashMethodString<typename Data::value_type, SetMethodMapped<Data>, true, false>;

That is a rare and welcome thing to find in a codebase: an optimisation that was measured, found harmful for one type family, and turned off — with the numbers kept in the source.

The NULL rule. A row whose key contains a NULL is never in the set, so the answer is negative (0 for IN, 1 for NOT IN) without a lookup. The rationale is in the header:

 cpp
 
 
// src/Interpreters/Set.h:202
 /** How IN works with Nullable types.
 *
 * For simplicity reasons, all NULL values and any tuples with at least one NULL element are ignored in the Set.
 * And for left hand side values, that are NULLs or contain any NULLs, we return 0 (means that element is not in Set).
 *
 * If we want more standard compliant behaviour, we must return NULL
 * if lhs is NULL and set is not empty or if lhs is not in set, but set contains at least one NULL.
 * It is more complicated with tuples.
 * For example,
 * (1, NULL, 2) IN ((1, NULL, 3)) must return 0,
 * but (1, NULL, 2) IN ((1, 1111, 2)) must return NULL.
 *
 * We have not implemented such sophisticated behaviour.
 */

So NULL IN (1, 2, 3) is 0, not NULL. That is a deliberate deviation from SQL standard, documented in the source.

Type coercion on the probe

The left-hand side of the IN may have a different type from the set’s key — the set’s type wins, and the probe column is cast. The cast is cached, so it is not rebuilt for every block:

 cpp
 
 
// src/Interpreters/Set.h:249
 /// A cache for cast functions (if any) to avoid rebuilding cast functions
 /// for every call to `execute`
 mutable std::unique_ptr<InternalCastFunctionCache> cast_cache;

and when the value does not fit the key type, it is not an error — it is simply “not a member”:

 cpp
 
 
// src/Interpreters/Set.cpp:833
 if (can_convert_leniently)
 {
 /// `transform_null_in` makes NULL an ordinary matchable value, so the accurate-or-null
 /// cast above is not taken. A value the key type cannot represent is still simply not a
 /// member - which is what the literal `IN` list concludes, because it compares in a
 /// common supertype - but converting it strictly raised `CANNOT_CONVERT_TYPE` on rows
 /// the read path happened to deliver, so `v IN set_table` failed where `v IN (1)`
 /// returned 0.

Size limits and overflow

The set is subject to max_rows_in_set and max_bytes_in_set, checked after every inserted chunk:

 cpp
 
 
// src/Interpreters/Set.cpp:556
 bool within_limits = limits.check(data->getTotalRowCount(), data->getTotalByteCount(), "IN-set", ErrorCodes::SET_SIZE_LIMIT_EXCEEDED);
 if (!within_limits)
 is_truncated = true;

The overflow mode is set_overflow_mode: throw (the default) aborts the query, break truncates the set and returns the partial result — which is wrong for IN but is the only sane option for a GLOBAL IN over a huge table, where a partial set is still useful.

Spilling to disk

A set built from a subquery can be enormous: x IN (SELECT id FROM big_table) over a billion rows. ClickHouse 26.10 added spilling: the set moves to a temporary file, and the in-memory part shrinks to a directory plus a set of Bloom filters.

The disk representation is not the original values. Each key is converted into a compact disk key: numbers and FixedStrings of at most 32 bytes are packed into integers, and everything else becomes a 128-bit hash:

 cpp
 
 
// src/Interpreters/Set.h:85
 /// the keys that come first in the order of their disk keys rather than the first ones to arrive. Disk
 /// keys keep the order of the keys that the set packs into integers in memory, numbers and `FixedString`
 /// values of at most 32 bytes in total, with `NULL` after the values, so these are the smallest keys.
 /// Other keys, such as `String`, are stored as 128-bit hashes, whose order is arbitrary.

The keys are then sorted, so that a lookup can binary-search the directory and find the block that may contain the key.

The Bloom filters

Each block of the set gets its own Bloom filter, and all the filters share one array of bits:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:15
/// Bloom filters of the blocks of a set on disk, which the set keeps in memory. A lookup checks the filter of
/// the only block that can hold its key, and reads the block only when the filter lets the key pass.
///
/// The filters share one array of bits. Each block takes a range of it in proportion to its keys, and each
/// key sets a few bits in the range of its block. A single filter for all keys would take as much memory, but
/// a key would set bits anywhere in it, leading to writes at random places all over the array. For a large
/// set, the array is far larger than the cache, so each such write would miss the cache and the TLB, and
/// these misses would slow down writing the set, because the set fills the filters on the single thread that
/// writes its blocks. The range of a block is about as small as the block, so it stays in the cache while the
/// set writes the block. Lookups check the ranges in the order of the blocks, as they read the blocks.

That is a lovely piece of engineering: the filters are per block so that a write touches only a cache-resident range, not the whole (possibly huge) filter array.

The filter is sized by the number of distinct keys, which is estimated with a HyperLogLog counter while the set is being filled:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:29
 /// Counts the distinct keys of a set as they arrive, so that the bloom filters can be sized before the set
 /// writes its blocks. With 2^20 buckets, the estimate is within about 0.1% and the counter takes only 768 KiB.
 class DistinctKeyCounter
 {
 public:
 void add(std::span<const Key> keys)
 {
 for (const Key & key : keys)
 counter.insert(hashKey(key));
 }

and the bits-per-key is chosen to fit the memory budget, trading false positives for memory:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:46
 /// Creates the filters for about `expected_keys` keys. They take 8 bits per key, which lets about 3% of
 /// the keys that the set does not have pass. When that does not fit in `max_bytes` bytes with the ranges
 /// of the blocks, they take fewer bits per key, so that they fit: with 2 bits per key, about 40% of these
 /// keys pass, and with 1 bit per key, about 63%. Every block but the last holds at least
 /// `min_keys_per_block` keys, which bounds the number of ranges. Returns nothing when fewer than 1 bit
 /// per key fits, because such filters let so many keys pass that they do not pay for filling them.

The number of hash positions per key is capped at 3, with the reasoning spelled out:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:82
 /// A key sets the bits at most at this many positions in the range of its block. Each position costs a
 /// memory access when a lookup checks a key that the set has, while more positions let only slightly
 /// fewer other keys pass: with 8 bits per key, 3 positions let about 3.1% of them pass, and 6 positions,
 /// the most effective number, about 2.2%.
 static constexpr size_t max_positions_per_key = 3;

The lookup

A lookup against the disk set is a five-step algorithm: collect the probes, sort them, find the blocks, consult the filters, read the blocks.

 cpp
 
 
// src/Interpreters/DiskSet.cpp:121
 /// Step 1: Collect the probes: the rows whose keys are in the range of the set, except a row that repeats
 /// the key of the previous row, which takes that row's answer in step 5. So a run of equal keys, as in a
 /// sorted column, takes one probe.
 struct Probe
 {
 Key key;
 size_t row;
 };

 VectorWithMemoryTracking<Probe> probes;
 probes.reserve(keys.size());
 const Key & first_key = directory.getFirstKey(0);
 for (size_t row = 0; row < keys.size(); ++row)
 {
 const auto & key = keys[row];
 if ((row == 0 || key != keys[row - 1]) && first_key <= key && key <= last_key)
 probes.push_back({key, row});
 }

Note the range check first_key <= key && key <= last_key: keys outside the set’s key range cannot be in the set, and are answered without any I/O.

Then the probes are sorted, so that each block is read once and in order:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:143
 /// Step 2: Sort the probes by key. Probes of equal keys then come together, so each distinct key is
 /// looked up once, and the probes of each block come together in block order, so they share a read.
 /// Results go back to the original rows, so evaluating `IN` does not reorder the input.
 ::sort(probes.begin(), probes.end(), [](const Probe & lhs, const Probe & rhs) { return lhs.key < rhs.key; });

and the filters decide which blocks are actually read:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:174
 for (size_t begin = 0; begin < probes.size();)
 {
 size_t end = begin + 1;
 while (end < probes.size() && probes[end].key == probes[begin].key)
 ++end;

 block = directory.findBlock(probes[begin].key, block);
 if (!filters || filters->mayContain(block, probes[begin].key))
 {
 if (lookups.empty() || block != lookups.back().block)
 needed_blocks.push_back(block); /// Each block once, in ascending order
 lookups.push_back({block, begin, end});
 }
 else
 {
 for (size_t i = begin; i < end; ++i)
 skipped_rows += count_rows_of_probe(probes[i]);
 }
 begin = end;
 }

The number of rows the filters answered is a profile event, so the effectiveness of the filters is directly observable:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:195
 /// A skipped key counts once for each of its rows, as `ExternalSetLookupKeys` does, so that the two
 /// events give the share of the rows that the filters answer.
 ProfileEvents::increment(ProfileEvents::ExternalSetBloomFilterSkippedKeys, skipped_rows);

When does it spill?

The threshold is a memory budget, and the default is “half of the available memory”:

 cpp
 
 
// src/Core/Settings.cpp:3905
 DECLARE(Double, max_bytes_ratio_before_external_set, 0.5, R"(
Fraction of available server or user memory used to calculate the threshold for spilling the set of `IN`
with a subquery to disk, at the start of execution. For example, `0.5` uses half of the available memory.

The set spills while it is built and while it is used, and the two cases are handled differently. While it is built, the keys are handed to a DiskSetBuilder that sorts them into runs; while it is used, the keys are written out while the lookups keep reading the in-memory table, and the disk set replaces it at the end.

The bloom filter has its own budget:

 cpp
 
 
// src/Core/Settings.cpp:3917
 DECLARE(UInt64, max_bytes_for_external_set_bloom_filter, 128_MiB, R"(
Memory budget, in bytes, for the Bloom filter that each set of `IN` written to disk keeps in memory. The
filter lets lookups of values that are not in the set skip reading the disk. It does not speed up lookups
of values that are in the set.

Note the asymmetry in that sentence: the filter helps for misses, not for hits. A key that is in the set still costs a block read. That is the fundamental trade of a Bloom filter, and it is why the filter is only worth it when the selectivity is low.

Trade-offs

What it buys: IN over a subquery is a hash-table probe per row, with a data structure chosen to fit the key type exactly. No row-at-a-time interpretation, no generic comparison, and for the common cases (one integer key, one string key) the key is the hash-table key.

What it costs:

Memory. The set is proportional to the number of distinct values on the right-hand side. x IN (SELECT id FROM orders) with 100 M distinct ids is a 100 M-entry hash table. This is why the spilling machinery exists, and why max_rows_in_set exists.
Spilling is much slower than RAM. A probe against a disk set is a sort, a binary search, a Bloom filter check and possibly a block read. ClickHouse sorts the probe keys to make the reads sequential, which is a lot of work per block. The alternative — failing the query with SET_SIZE_LIMIT_EXCEEDED — is often better for the user.
The hash fallback loses information. For a heterogeneous key, the set stores a 128-bit hash of the tuple, not the tuple. Two different tuples with the same hash would be “equal”; the probability is negligible, but it is not zero.
NULL semantics deviate from the standard, as documented above.
The hashed method is slower per probe than the packed methods, because the key must be serialised and hashed on every probe rather than once at build time. For a set of strings, the string method stores the string and its hash; the hashed method stores only the hash.

The last point is the general lesson of the whole design: ClickHouse’s IN is fast not because hashing is fast, but because the shape of the data is known at analysis time and the implementation specialises on it — thirteen variants, each with its own hash function, its own packing, its own arena, and its own micro-optimisations, some of which are enabled and some of which are deliberately disabled after measurement.

How ClickHouse Estimates Distinct Counts by Thinning a Hash Table

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

uniq() is one of the most-used aggregate functions in ClickHouse, and one of the most surprising. It is not exact, and it is not a HyperLogLog: it is a hash table that throws data away on purpose, thinning itself as it fills up, and then multiplies back up by the thinning factor to estimate the cardinality.

The algorithm behind it is the BJKST algorithm (Bretag, Karp, Kutz, Schlossmann — “near-optimal algorithms for estimating the number of distinct elements in a stream of data”). The implementation is in src/AggregateFunctions/UniquesHashSet.h, and it is one of the more elegant pieces of code in the aggregation framework: a fixed-size table, a single integer of state, and a mathematically exact estimator of what the sample means.

The scheme

The header states the general recipe for approximate aggregation:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:27
/** Approximate calculation of anything, as usual, is constructed according to the following scheme:
 * - some data structure is used to calculate the value of X;
 * - Not all values are added to the data structure, but only selected ones (according to some selectivity criteria);
 * - after processing all elements, the data structure is in some state S;
 * - as an approximate value of X, the value calculated according to the maximum likelihood principle is returned:
 * at what real value X, the probability of finding the data structure in the obtained state S is maximal.
 */

and then the specific recipe:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:38
/** Very simple hash-set for approximate number of unique values.
 * Works like this:
 * - you can insert UInt64;
 * - before insertion, first the hash function UInt64 -> UInt32 is calculated;
 * - the original value is not saved (lost);
 * - further all operations are made with these hashes;
 * - hash table is constructed according to the scheme:
 * - open addressing (one buffer, position in buffer is calculated by taking remainder of division by its size);
 * - linear probing (if the cell already has a value, then the cell following it is taken, etc.);
 * - the missing value is zero-encoded; to remember presence of zero in set, separate variable of type bool is used;
 * - buffer growth by 2 times when filling more than 50%;
 * - if the set has more than UNIQUES_HASH_MAX_SIZE elements, then all the elements are removed from the set,
 * not divisible by 2, and then all elements that do not divide by 2 are not inserted into the set;
 * - if the situation repeats, then only elements dividing by 4, etc., are taken.
 * - the size() method returns an approximate number of elements that have been inserted into the set;
 * - there are methods for quick reading and writing in binary and text form.
 *
 * A 32-bit hash cannot distinguish more than a few billion elements: the estimate degrades quickly
 * beyond a billion, and above ~30 billion the correction formula even resulted in undefined behavior
 * (issue #6078). To keep the estimate accurate up to trillions, in addition to the main set a small
 * "wide" set of full 64-bit hashes (XOR-ed with a constant, see `wideHash`) is maintained (state version 1).
 * It is thinned out aggressively from the very beginning - only values with UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE
 * zero low-order bits enter it - so it stays empty or tiny for the states where the main set is accurate
 * (about 4 values per million distinct elements), and it is bounded by UNIQUES_HASH_SET_WIDE_MAX_SIZE
 * values thinned out further. Once the wide set has enough values for its own estimate to be reliable
 * (which happens at about two billion elements), size() switches to it.
 *
 * The wide set has to be maintained from the first insertion: the high bits of the hashes
 * of the already consumed elements cannot be recovered later. In particular, a state written
 * in the legacy format (state version 0) carries no wide set, and its elements were never
 * offered to one. Such a state is marked as having an incomplete wide sample: it drops its own
 * wide set, never starts a new one, and propagates the mark through merges, so that size()
 * of anything it is merged into falls back to the legacy 32-bit estimator. Estimating from
 * a wide set that covers only a part of the elements would undercount the rest entirely,
 * which is much worse than the degrading precision of the 32-bit estimate.
 */

That comment is the whole algorithm. Read it twice: the key insight is that the set does not grow without bound; it changes what it accepts.

The constants

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:75
/// The maximum degree of buffer size before the values are discarded
#define UNIQUES_HASH_MAX_SIZE_DEGREE 17

/// The maximum number of elements before the values are discarded
#define UNIQUES_HASH_MAX_SIZE (1ULL << (UNIQUES_HASH_MAX_SIZE_DEGREE - 1))

/** The number of least significant bits used for thinning. The remaining high-order bits are used to determine the position in the hash table.
 * (high-order bits are taken because the younger bits will be constant after dropping some of the values)
 */
#define UNIQUES_HASH_BITS_FOR_SKIP (32 - UNIQUES_HASH_MAX_SIZE_DEGREE)

/// Initial buffer size degree
#define UNIQUES_HASH_SET_INITIAL_SIZE_DEGREE 4

So the main set holds at most 2^16 = 65 536 hashes, in a buffer of 2^17 slots, and 15 low bits of the hash are reserved for the thinning mask. The remaining 17 high bits select the cell:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:195
 size_t place(HashValue x) const { return (x >> UNIQUES_HASH_BITS_FOR_SKIP) & mask(); }

 /// The value is divided by 2 ^ skip_degree
 bool good(HashValue hash) const { return hash == ((hash >> skip_degree) << skip_degree); }

good() is the thinning test: the low skip_degree bits of the hash must be zero. The high bits are used for the position, because the low bits are constant across all surviving elements — using them for the position would put every element in the same cell.

Thinning

The state is three integers plus a flag:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:155
 UInt32 m_size; /// Number of elements
 UInt8 size_degree{}; /// The size of the table as a power of 2
 UInt8 skip_degree; /// Skip elements not divisible by 2 ^ skip_degree
 bool has_zero; /// The hash table contains an element with a hash value of 0.

and the moment of truth is shrinkIfNeed:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:346
 /** If the hash table is full enough, then do resize.
 * If there are too many items, then throw half the pieces until they are small enough.
 */
 void shrinkIfNeed()
 {
 if (unlikely(m_size > max_fill()))
 {
 if (m_size > UNIQUES_HASH_MAX_SIZE)
 {
 while (m_size > UNIQUES_HASH_MAX_SIZE)
 {
 if (unlikely(skip_degree >= max_skip_degree))
 throw DB::Exception(DB::ErrorCodes::INCORRECT_DATA,
 "Cannot thin out UniquesHashSet: {} elements remain at the maximum skip degree of {}",
 static_cast<size_t>(m_size), static_cast<size_t>(max_skip_degree));

 ++skip_degree;
 rehash();
 }
 }
 else
 resize();
 }
 }

So: the set grows normally until it holds more than 65 536 elements. Then it does not grow — it thins. skip_degree goes up by one, and every element whose hash is not divisible by 2^skip_degree is deleted:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:200
 /// Delete all values whose hashes do not divide by 2 ^ skip_degree
 void rehash()
 {
 for (size_t i = 0; i < buf_size(); ++i)
 {
 if (buf[i])
 {
 if (!good(buf[i]))
 {
 buf[i] = 0;
 --m_size;
 }
 /** After removing the elements, there may have been room for items,
 * which were placed further than necessary, due to a collision.
 * You need to move them.
 */
 else if (i != place(buf[i]))
 {
 HashValue x = buf[i];
 buf[i] = 0;
 reinsertImpl(x);
 }
 }
 }

The picture is this:

 text
 
 
 distinct values arriving: A B C D E F G H ... (millions)

 skip_degree = 0 -> accept everything, table grows
 table full at 65536 elements, distinct count so far maybe 200 000
 skip_degree = 1 -> accept only hashes with bit0 == 0 (half of them)
 table full again at 65536, distinct count so far maybe 400 000
 skip_degree = 2 -> accept only hashes with bits1..0 == 0 (a quarter)
 ...
 skip_degree = k -> accept 1 in 2^k elements

Each thinning step halves the sample rate, and the sample size is kept roughly constant. The estimate multiplies the surviving count back up by 2^skip_degree.

The estimator

The estimator is not simply m_size * 2^skip_degree — there is a correction for the systematic error of 32-bit hash collisions, and a pseudo-random remainder so that the result does not visibly sit on a power of two:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:842
 if (0 == skip_degree)
 return m_size;

 size_t res = m_size * (1ULL << skip_degree);

 /** Pseudo-random remainder - in order to be not visible,
 * that the number is divided by the power of two.
 */
 res += (intHashCRC32(m_size) & ((1ULL << skip_degree) - 1));

 /** Correction of a systematic error due to collisions during hashing in UInt32.
 * `fixed_res(res)` formula
 * - with how many different elements of fixed_res,
 * when randomly scattered across 2^32 buckets,
 * filled buckets with average of res is obtained.
 */
 return correctFor32BitCollisions(res);

The correction is an analytic formula for the birthday problem. If res distinct elements are scattered over 2^32 buckets, the number of occupied buckets is less than res because of collisions; to recover the number of distinct elements from the number of occupied buckets, you invert that relation:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:548
 /** With how many different elements, when randomly scattered across 2^32 buckets,
 * the given number of non-empty buckets is obtained on average.
 * This is the correction of the systematic error due to collisions of the 32-bit hashes.
 */
 static size_t correctFor32BitCollisions(size_t res)
 {
 size_t p32 = 1ULL << 32;

 /** When the number of distinct elements is much larger than 2^32, almost all buckets are filled
 * and `res` may reach (or, due to the pseudo-random remainder, exceed) 2^32,
 * where the formula below is not defined (previously this resulted in undefined behavior
 * when casting infinity or NaN to an integer). Saturate the estimate instead:
 * 32 bits of a hash contain no information about larger cardinalities anyway
 * (and the wide set takes over long before this point).
 */
 if (res >= p32)
 res = p32 - 1;

 return static_cast<size_t>(round(static_cast<double>(p32) * (log(static_cast<double>(p32)) - log(static_cast<double>(p32 - res)))));
 }

Read the formula as: p32 * (ln p32 - ln(p32 - res)). For small res it is approximately res; for res near p32 it blows up, which is why the saturation guard exists — and that guard is the fix for issue #6078, where the cast of inf to an integer was undefined behaviour.

The wide set: fixing the 32-bit ceiling

The 32-bit hash has a hard ceiling. At a few billion distinct elements the correction formula is already straining, and above ~30 billion it was producing garbage. The fix, added in state version 1, is a second set that stores full 64-bit hashes — but only a tiny, heavily thinned sample of them.

The sample is defined by a skip degree of 18: only hashes with 18 zero low-order bits enter it, i.e. one in 262 144:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:89
/** Only hashes with this many zero low-order bits enter the wide set, so that it stays
 * negligibly small while the main set is accurate: one hash in 262144 passes.
 */
#define UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE 18

/// The maximum degree of the wide set buffer size before the values are thinned out further.
#define UNIQUES_HASH_SET_WIDE_MAX_SIZE_DEGREE 15

/// The maximum number of elements of the wide set: 16384 values take 256 KiB.
#define UNIQUES_HASH_SET_WIDE_MAX_SIZE (1ULL << (UNIQUES_HASH_SET_WIDE_MAX_SIZE_DEGREE - 1))

/** The estimate is taken from the wide set when it has at least this many values, i.e. starting
 * from about UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE * 2^UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE,
 * about two billion elements: at that point the statistical error of the wide estimate (about 1%)
 * meets the degrading precision of the 32-bit one.
 */
#define UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE 8192

So the wide set holds at most 16 384 hashes (256 KiB) and needs 8192 of them before it is trusted. Since only one in 262 144 elements enters it, 8192 elements means about 2.1 billion distinct values have been seen — exactly the point where the 32-bit estimator starts to degrade.

The wide set has one more twist: the hashes are XOR-ed with a constant before being stored.

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:518
 /** The wide set samples the full hashes XOR-ed with this constant rather than the hashes themselves.
 * The sample stays just as uniform, but the hash of a frequent value never becomes a candidate
 * systematically: the default hash of the integer 0 is 0, and zeros are frequent in real data -
 * without the offset every such row would be a candidate for the wide set and would cost
 * an extra pass over its batch (this doubled the time of `uniq` over such columns).
 * The low bits up to the maximum thinning of the main set are zero, so that every candidate
 * still passes good() of the main set (see `widePrefilterMask`), and the next ones are not,
 * so that the hash 0 is never a candidate.
 */
 static constexpr UInt64 wide_hash_offset = 0x9E3779B97F4A0000ULL;

The constant is the golden-ratio fraction used in many hash mixers (0x9E3779B97F4A0000 is 2^64 / φ). The problem it solves is delightfully concrete: the default hash of the integer 0 is 0, and the wide-set candidate test is “are the low 18 bits zero?”. The hash 0 passes that test — so every zero row in the data would enter the wide set, and the extra work doubled the runtime of uniq over columns full of zeros. XOR-ing with an offset whose low bits are non-zero removes that pathology while preserving uniformity.

The offset has to satisfy two constraints, and they are checked at compile time:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:528
 static_assert((wide_hash_offset & ((1ULL << max_skip_degree) - 1)) == 0);
 static_assert((wide_hash_offset & ((1ULL << UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE) - 1)) != 0);

The first says: the offset’s low bits are zero up to the maximum thinning degree of the main set, so that a wide candidate still passes the main set’s good() test (the wide set is a subset of the main sample). The second says: the offset’s low 18 bits are not all zero, so the hash 0 is never a wide candidate.

The wide set is allocated lazily, in a single block holding both the header and the table:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:373
 static WideSet * allocWideSet(UInt8 new_size_degree)
 {
 /// The header and the table are placed in a single allocation.
 size_t bytes = sizeof(WideSet) + (1ULL << new_size_degree) * sizeof(UInt64);
 WideSet * res = new (HashTableAllocator().alloc(bytes)) WideSet;
 res->size_degree = new_size_degree;
 res->buf = reinterpret_cast<UInt64 *>(reinterpret_cast<char *>(res) + sizeof(WideSet));
 return res;
 }

and it is only allocated when the first candidate shows up — which, for most data, never happens:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:469
 if (!wide)
 wide = allocWideSet(UNIQUES_HASH_SET_INITIAL_SIZE_DEGREE);

Which estimator is used

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:820
 size_t size() const
 {
 /** The wide set gives a reliable estimate once it has enough values;
 * below that the 32-bit main set is more precise. If the wide sample does not cover
 * all the elements (see `wide_incomplete`), it is not there at all and the legacy
 * estimator is used, because a partial wide estimate would silently lose the rest.
 */
 if (wide && wide->m_size >= UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE)
 {
 size_t res = static_cast<size_t>(wide->m_size) * (1ULL << wide->skip_degree);

 /** Pseudo-random remainder - in order to be not visible,
 * that the number is divided by the power of two.
 */
 res += (intHashCRC32(wide->m_size) & ((1ULL << wide->skip_degree) - 1));

 /** Collisions of 64-bit hashes are negligible: their systematic error would reach
 * just 0.003% at 10^15 distinct elements, far below the statistical error of the estimate.
 */
 return res;
 }

Note the last comment: at 10^15 distinct elements, the systematic error of 64-bit hash collisions would be 0.003%, which is far below the statistical error of the estimate itself. So the wide set is not exact either — it is exact enough.

The hot loop: batching

The insertion path is where the performance lives, because uniq is usually applied to millions of rows. The implementation does not insert one hash at a time; it computes eight hashes and eight target cells first, then inserts them:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:749
 size_t i = 0;
 while (i < size)
 {
 /// Check that we have enough capacity and enough values to insert in batch. If not, we will insert values one by one which
 /// allows us to check the shrink condition after each insert and avoid inserting too many values before shrinking.
 if ((max_fill() - m_size) >= insert_many_batch_size && ((size - i) >= insert_many_batch_size))
 {
 /// We read and transform multiple values at once which allows both the compiler and the CPU to better optimize the code.
 /// We calculate place() even for !good() hashes to maximize data independence and enable better out-of-order execution.
 /// The extra work is negligible compared to the instruction level parallelization benefits.
 std::array<HashValue, insert_many_batch_size> hash_value; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - filled by the loop below before read
 for (size_t j = 0; j < insert_many_batch_size; ++j)
 {
 hash_value[j] = hash(Transform(data[i + j]));
 }

with the batch size fixed at eight:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:665
 /// This value is arbitrary. The optimal value might depend on the CPU pipeline depth, cache line size (64B = 8 UInt64s),
 /// hash function quality, collision rate, instruction parallelism, etc.
 /// We choose a value that is big enough to provide sufficient instruction level parallelism but not too big to bloat the code size.
 static constexpr size_t insert_many_batch_size = 8;

Two details are worth noticing. place() is computed even for hashes that will be rejected by good(), on purpose: it keeps the loop free of data-dependent branches and lets the out-of-order engine overlap the work. And the wide-set candidates are collected into a small local buffer during the loop and inserted after it, because inserting them inside would force the compiler to reload the set’s fields from memory on every iteration:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:738
 /** The candidates for the wide set (see the comment at `widePrefilterMask`) are detected
 * behind the `good()` test, which every one of them passes and which already rejects all
 * but one value in 2^skip_degree, and they are only collected into a local buffer there.
 * They are inserted after the loop rather than inside it: a call inside would make
 * the compiler cautiously reload the fields of the set from memory all the time.
 * This loop is very sensitive to the code generation: the batch loops below must be
 * fully unrolled as they are without the wide set, otherwise `uniq` gets up to 50% slower.
 */

“otherwise uniq gets up to 50% slower” is a nice reminder that in an aggregation hot loop, the code layout is part of the algorithm.

What is hashed

The value that goes into the set is not the value itself but a 64-bit hash of it, of which only the low 32 bits feed the main set:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:637
 /// The full 64-bit hash: its low 32 bits feed the main set, and the whole value the wide set (see `wideHash`).
 static UInt64 fullHash(Value key) { return static_cast<UInt64>(Hash()(key)); }
 static HashValue hash(Value key) { return static_cast<HashValue>(fullHash(key)); }

and the per-type hashing is chosen in the aggregate function:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniq.h:273
template <typename T, typename ColumnType>
struct AggregateFunctionUniqTraits
{
 static ALWAYS_INLINE auto hash(T x)
 {
 if constexpr (std::is_same_v<T, std::string_view>)
 {
 return CityHash_v1_0_2::CityHash64(x.data(), x.size());
 }
 else if constexpr (std::is_same_v<T, IPv6>)
 {
 /// For compatibility reasons IPv6 is hashed as FixedString(16).
 return CityHash_v1_0_2::CityHash64(reinterpret_cast<const char *>(&x), sizeof(x));
 }
 else if constexpr (is_floating_point<T>)
 {
 return bit_cast<UInt64>(x);
 }
 else if constexpr (sizeof(T) <= sizeof(UInt64))
 {
 return x;
 }
 else
 {
 return DefaultHash64<T>(x);
 }
 }

Note that for integers the “hash” is the identity: an integer’s low bits are used directly, which is fine because the thinning test only looks at the low 15..18 bits and the position is taken from the high bits, which for an identity hash of a sequential integer is… well, for sequential integers the high bits are also nearly constant. That is precisely why the wide set XORs with an offset, and why the main set’s place() uses the high bits of the 32-bit hash — the 32-bit hash is intHash32, which mixes.

Indeed the set’s own hash function is the 32-bit integer hash:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:108
/** This hash function is not the most optimal, but UniquesHashSet states counted with it,
 * stored in many places on disks (in many companies), so it continues to be used.
 */
struct UniquesHashSetDefaultHash
{
 size_t operator() (UInt64 x) const
 {
 return intHash32<0>(x);
 }
};

The comment is a nice piece of engineering history: the hash is frozen because the serialized states of uniq are stored on disk in many companies’ systems, and changing the hash would invalidate them.

The state format

Because the set is serialised into aggregate-function states that travel between servers, the format is versioned. Version 1 adds a flags byte:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:963
 /** Two serialization formats exist.
 * The legacy format (state version 0): skip_degree, then the number of values, then the 32-bit values
 * of the main set.
 * The new format (state version 1): a flags byte (bit 0 - a wide set follows the main set,
 * bit 1 - the wide sample is incomplete, see `wide_incomplete`; the two are mutually exclusive),
 * then the main set in the legacy layout, then, if present, the wide set: its skip_degree,
 * the number of values, and the 64-bit values.
 */

and a legacy state read back is marked as having an incomplete wide sample:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:1048
 /** A non-empty state written in the legacy format carries elements that were never offered
 * to a wide set, so no wide estimate of this state (or of anything it is merged into)
 * can be trusted. An empty legacy state contributes nothing and does not spoil a merge.
 */
 if (use_legacy_format && m_size > 0)
 wide_incomplete = true;

That mark is sticky and propagates through merges, because the missing high bits of the already-consumed elements cannot be recovered. The alternative — estimating from a partially-covering wide sample — would silently undercount everything the sample missed, which is far worse than a merely imprecise estimate.

The whole family

uniq is one of several cardinality estimators in ClickHouse, and they make different trade-offs:

function	structure	exact?
`uniqExact`	a full hash set of the values	yes
`uniq`	the thinning set of this article	no, ~1-2% error
`uniqCombined`	small set → hash set → HyperLogLog, switching as it grows	no
`uniqTheta`	a datasketches theta sketch	no, mergeable

uniqCombined is the one that switches representation:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniqCombined.h:112
 // We want to migrate from |HashSet| to |HyperLogLogCounter| when the sizes in memory become almost equal.
 // The size per element in |HashSet| is sizeof(Key)*2 bytes, and the overall size of |HyperLogLogCounter| is 2^K * 6 bits.
 // For Key=UInt32 we can calculate: 2^X * 4 * 2 ≤ 2^(K-3) * 6 ⇒ X ≤ K-4.

and uniqExact keeps the two-level structure of the previous article in reserve, converting when it grows:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniq.h:154
 /// When creating, the hash table must be small.
 static constexpr size_t initial_size_degree = 4;
 using SingleLevelSet = HashSetWithStackMemory<Key, HashCRC32<Key>, initial_size_degree>;
 using TwoLevelSet = TwoLevelHashSet<Key, HashCRC32<Key>>;
 using Set = UniqExactSet<SingleLevelSet, TwoLevelSet>;

Note the “hash table must be small”: the initial buffer lives on the stack of the aggregate function state, so a uniqExact state that never grows past 2^4 slots allocates nothing on the heap at all.

A quick sanity check of the behaviour in SQL:

 sql
 
 
SELECT
 uniqExact(user_id) AS exact,
 uniq(user_id) AS approx,
 uniqCombined(user_id) AS combined
FROM events;

approx will differ from exact by a fraction of a percent on a few million distinct users, and by more on a few hundred million — that is the thinning, and it is the price of a bounded 64 KiB state.

Trade-offs

What the thinning set buys:

A bounded, tiny state. The main set is at most 2^17 × 4 bytes = 512 KiB, and the wide set at most 256 KiB. A uniq state never grows with the cardinality, which matters when the state is merged across a cluster or spilled to disk.
A mergeable state. Merging two uniq states is a union of hash sets plus a max(skip_degree), which is why uniq can be used with Distributed tables and AggregatingMergeTree.
A principled estimator. The thinning factor is known exactly, so the estimate is a maximum-likelihood estimate rather than a heuristic.
No allocation in the hot loop. The table is a flat array of UInt32, and the batch of eight hashes is computed with instruction-level parallelism.

What it costs:

Precision. The relative error grows with the cardinality, because the sample size stays constant while the multiplier grows. The wide set pushes the useful range from billions to trillions, but the error is still around 1% at the switch-over point.
A hash-quality dependency. The whole scheme assumes the hash is uniform; a bad hash breaks the thinning, because the surviving elements would not be a uniform sample.
Irreversible loss. Once thinned, the dropped hashes are gone. That is why the wide set has to be fed from the first insertion, and why a legacy state can never grow a wide sample.
A 32-bit ceiling without the wide set, which is exactly the bug that issue #6078 was, and why the state format has two versions.

The design is a good example of an approximation that is statistically principled rather than ad hoc: the state is small because the algorithm throws away information on purpose, and the estimate is exact given the information that was kept.

How ClickHouse Decides Which Parts to Merge: a Cost Model for Write Amplification

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A MergeTree table is a set of immutable parts. Every INSERT adds one. Background merges take contiguous runs of parts and rewrite them as one bigger part, which reduces the number of parts a SELECT has to open — and costs a full rewrite of the data involved.

That trade-off has no free lunch, and the two directions of it pull in opposite directions:

Merge eagerly and you keep the part count low, but you rewrite the same bytes over and over. A part that is merged into a bigger part, which is merged again, is written twice.
Merge lazily and write amplification is low, but the part count climbs, and every query has to open every part.

ClickHouse’s answer is a small, explicit cost model, and — unusually — the model is documented in the source, together with its own limitations. This is a tour of that model: the score function, the “base of the logarithm” knob, the sliding window that prevents merge backlogs from snowballing, and the background scheduler’s exponential backoff.

The design note

The most valuable thing about ClickHouse’s merge selector is that the authors wrote down what they were trying to do, and where they thought they were wrong. The header of SimpleMergeSelector.h is a design document:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:6
/**
We have a set of data parts that is dynamically changing - new data parts are added and there is background merging process.
Background merging process periodically selects continuous range of data parts to merge.

It tries to optimize the following metrics:
1. Write amplification: total amount of data written on disk (source data + merges) relative to the amount of source data.
It can be also considered as the total amount of work for merges.
2. The number of data parts in the set at the random moment of time (average + quantiles).

Also taking the following considerations:
1. Older data parts should be merged less frequently than newer data parts.
2. Larger data parts should be merged less frequently than smaller data parts.
3. If no new parts arrive, we should continue to merge existing data parts to eventually optimize the table.
4. Never allow too many parts, because it will slow down SELECT queries significantly.
5. Multiple background merges can run concurrently but not too many.

It is not possible to optimize both metrics, because they contradict to each other.
To lower the number of parts we can merge eagerly but write amplification will increase.
Then we need some balance between optimization of these two metrics.

But some optimizations may improve both metrics.

For example, we can look at the "merge tree" - the tree of data parts that were merged.
If the tree is perfectly balanced then its depth is proportional to the log(data size),
the total amount of work is proportional to data_size * log(data_size)
and the write amplification is proportional to log(data_size).
If it's not balanced (e.g. every new data part is always merged with existing data parts),
its depth is proportional to the data size, total amount of work is proportional to data_size^2.

Read that last paragraph carefully, because it is the key insight of the whole design. The merges form a tree: leaves are inserted parts, internal nodes are merge results. If the tree is balanced, its depth is log_base(total_size) and the total write amplification is proportional to that depth. If the tree is a degenerate chain — every new part merged immediately into the current biggest part — the depth is proportional to the data size, and the total write amplification is O(data_size). Every byte gets rewritten once per level.

So the selector’s job is to keep the merge tree balanced, and the knob that controls balance is the arity of the tree — how many parts are merged at once.

The base of the logarithm

The arity is called base, and the header explains how it is generalized to parts of unequal size:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:35
We can also control the "base of the logarithm" - you can look it as the number of data parts
that are merged at once (the tree "arity"). But as the data parts are of different size, we should generalize it:
calculate the ratio between total size of merged parts to the size of the largest part participated in merge.
For example, if we merge 4 parts of size 5, 3, 1, 1 - then "base" will be 2 (10 / 5).

Base of the logarithm (simply called `base` in `SimpleMergeSelector`) is the main knob to control the write amplification.
The more it is, the less is write amplification but we will have more data parts on average.

To fit all the considerations, we also adjust `base` depending on total parts count,
parts size and parts age, with linear interpolation (then `base` is not a constant but a function of multiple variables,
looking like a section of hyperplane).

So the “base” of a candidate merge is sum_size / max_size — the ratio of the total size of the range to the size of its largest part. Merging four parts of sizes 5, 3, 1, 1 gives a base of 10/5 = 2. Merging a huge part with a tiny one gives a base barely above 1, which is bad: that is the degenerate chain, and the selector is supposed to avoid it.

base is exposed as a MergeTree setting, default 5:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1090
 DECLARE(Float, merge_selector_base, 5.0, R"(Affects write amplification of
 assigned merges (expert level setting, don't change if you don't understand
 what it is doing). Works for Simple and StochasticSimple merge selectors
)", 0) \

The admissibility test: is this range allowed at all?

Before any candidate is scored, it has to pass an admissibility test. The test is allow(), and its central quantity is the lowered base — a base that is adjusted downward depending on how stale the parts are, how many parts the partition has, and how big the range is.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:241
 /// Map size to 0..1 using logarithmic scale
 /// Use log(1 + x) instead of log1p(x) because our sum_size is always integer.
 /// Also log1p seems to be slow and significantly affect performance of merges assignment.
 double size_normalized = mapPiecewiseLinearToUnit(log(1 + sum_size), min_size_to_lower_base_log, max_size_to_lower_base_log);
 /// Calculate boundaries for age
 double min_age_to_lower_base = interpolateLinear(static_cast<double>(settings.min_age_to_lower_base_at_min_size), static_cast<double>(settings.min_age_to_lower_base_at_max_size), size_normalized);
 double max_age_to_lower_base = interpolateLinear(static_cast<double>(settings.max_age_to_lower_base_at_min_size), static_cast<double>(settings.max_age_to_lower_base_at_max_size), size_normalized);
 /// Map age to 0..1
 double age_normalized = mapPiecewiseLinearToUnit(min_age, min_age_to_lower_base, max_age_to_lower_base);
 /// Map partition_size to 0..1
 double num_parts_normalized = mapPiecewiseLinearToUnit(partition_size, static_cast<double>(settings.min_parts_to_lower_base), static_cast<double>(settings.max_parts_to_lower_base));
 /// The ratio should be within [0, 1]
 double combined_ratio = std::min(1.0, age_normalized + num_parts_normalized);

 double lowered_base = interpolateLinear(settings.base, 2.0, combined_ratio);

The three inputs, and what each one does:

Input	Effect on `lowered_base`
`age_normalized` — how old the youngest part in the range is	Older parts are cheaper to merge (they are not going to be queried as freshly-inserted data), so the base is lowered and the merge happens sooner.
`num_parts_normalized` — how many parts the partition has	A partition with too many parts is a problem for queries, so the base is lowered and merges get more aggressive.
`size_normalized` — how big the range is	Big ranges need to be older before they are merged, because merging them is expensive.

The mapping is deliberately non-linear in size (logarithmic), and the comment explains why log(1+x) rather than log1p(x): log1p is slow enough to matter, and the selector runs on every selection round.

Then the actual test:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:271
 return (sum_size + static_cast<double>(size) * static_cast<double>(settings.size_fixed_cost_to_add)) / (max_size + static_cast<double>(settings.size_fixed_cost_to_add)) >= lowered_base;

Read it as: “is sum_size / max_size at least lowered_base?” — modulo the fixed cost of merging (size_fixed_cost_to_add, default 5 MB) added to both numerator and denominator, which models the fact that merging even two tiny parts costs a fixed amount of work.

Two escape hatches come before it:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:215
 if (settings.min_age_to_force_merge && min_age >= static_cast<double>(settings.min_age_to_force_merge))
 return true;

 if (settings.min_partition_age_to_force_merge && min_partition_age > 0)
 if (min_partition_age >= static_cast<double>(settings.min_partition_age_to_force_merge))
 return true;

If the parts are old enough (or the whole partition is old enough), the range is merged no matter what the base says. This is the safety valve: the cost model can be wrong, but a part that has been sitting there for a day must eventually be merged.

The score function

Now, given that several ranges are admissible, which one do you pick? This is the score() function, and it is where the “cost model” really lives:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:122
 static double score(double count, double sum_size, double sum_size_fixed_cost)
 {
 /** Consider we have two alternative ranges of data parts to merge.
 * Assume time to merge a range is proportional to sum size of its parts.
 *
 * Cost of query execution is proportional to total number of data parts in a moment of time.
 * Let define our target: to minimize average (in time) total number of data parts.
 *
 * Let calculate integral of total number of parts, if we are going to do merge of one or another range.
 * It must be lower, and thus we decide, what range is better to merge.
 *
 * The integral is lower iff the following formula is lower:
 *
 * sum_size / (count - 1)
 *
 * But we have some tunes to prefer longer ranges.
 */
 return (sum_size + sum_size_fixed_cost * count) / (count - 1.9);
 }

The derivation in the comment is a genuine (if informal) calculus argument. If you merge a range of count parts, the part count drops by count - 1 (you replace count parts with 1). The merge takes time proportional to sum_size. The “average number of parts over time” is therefore sum_size / (count - 1) — and lower is better, so the range with the smallest sum_size / (count - 1) is the one to merge.

The - 1.9 is a fudge factor. With - 1 the denominator vanishes for a two-part merge, giving an infinite score. The authors replaced - 1 with - 1.9, which is a softening of the singularity: merging two parts is penalized less than the pure model would suggest, and merging many parts is penalized more. The comment’s own summary is “we have some tunes to prefer longer ranges”.

size_fixed_cost_to_add (5 MB by default) enters the numerator as sum_size + fixed * count — every part in the range carries the fixed cost of being merged, so a range of ten tiny parts is much more expensive than its byte sum suggests.

The candidate enumeration walks every contiguous sub-range of the partition and scores each one:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:108
 double current_score = score(static_cast<double>(end - begin), static_cast<double>(sum_size), static_cast<double>(settings.size_fixed_cost_to_add));

 if (settings.enable_heuristic_to_align_parts
 && static_cast<double>(size_prev_at_left) > static_cast<double>(sum_size) * settings.heuristic_to_align_parts_min_ratio_of_sum_size_to_prev_part)
 {
 double difference = std::abs(log2(static_cast<double>(sum_size) / static_cast<double>(size_prev_at_left)));
 if (difference < settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two)
 current_score *= interpolateLinear(settings.heuristic_to_align_parts_max_score_adjustment, 1,
 difference / settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two);
 }

The “align parts” heuristic is a preference for ranges whose total size is roughly a power of two times the size of the part to their left. Why? Because a balanced merge tree wants each level to double the size. If the part to the left is 100 MB and the candidate range is 100 MB, the range “fits” the tree; if it is 30 MB or 300 MB, the resulting tree is lopsided and the next merge will have to rewrite more. The score is scaled down by up to 25% (heuristic_to_align_parts_max_score_adjustment = 0.75) when the sizes are close to a power of two apart.

The right-tail heuristic

There is one more adjustment, applied before scoring, and it is the most “heuristic” of them all:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:51
 const bool should_remove_small_parts_at_right = settings.enable_heuristic_to_remove_small_parts_at_right
 && (!settings.merge_selector_min_age_to_disable_right_tail_heuristic
 || min_age < settings.merge_selector_min_age_to_disable_right_tail_heuristic);
 if (should_remove_small_parts_at_right)
 {
 size_t size_delta = 0;
 size_t rows_delta = 0;
 while (end >= begin + 3 && static_cast<double>((end - 1)->size) < settings.heuristic_to_remove_small_parts_at_right_max_ratio * static_cast<double>(sum_size))
 {
 size_delta += (end - 1)->size;
 rows_delta += (end - 1)->rows;
 --end;
 }

 sum_size -= size_delta;
 sum_rows -= rows_delta;
 }

The idea: a range that ends with a couple of very small parts is a bad merge. Those small parts are probably new parts that just arrived, and folding them into a large merge means they will sit in the merge queue for a long time. Worse, it means the newly-arrived data is not queryable quickly. So the heuristic trims the right tail of the range — the small parts at the newest end — and scores what remains.

The default ratio is 1%: a part at the right end whose size is less than 1% of the range’s total is trimmed, as long as at least three parts remain. And there is an age-based escape hatch (merge_selector_min_age_to_disable_right_tail_heuristic): once the range is old enough, the tail is kept, because at that point staleness is worse than freshness.

The sliding window: anti-snowball protection

The single most valuable piece of engineering in the selector is this:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:307
 /// If the parts in the parts vector are sorted by block number,
 /// it may not be ideal to only select parts for merging from the first N ones.
 /// This is because if there are more than N parts in the partition,
 /// we will not be able to assign a merge for newly created parts.
 /// As a result, the total number of parts within the partition could
 /// grow uncontrollably, similar to a snowball effect.
 /// To address this we will try to assign a merge taking into consideration
 /// only last N parts.
 const size_t parts_threshold = settings.window_size;
 size_t begin = 0;
 if (parts_count >= parts_threshold)
 {
 if (settings.enable_stochastic_sliding)
 begin = calculateRangeWithStochasticSliding(parts_count, parts_threshold);
 else
 begin = parts_count - parts_threshold;
 }

window_size defaults to 1000. The comment describes the failure mode it prevents, and it is a real one:

If the parts in the parts vector are sorted by block number, it may not be ideal to only select parts for merging from the first N ones. This is because if there are more than N parts in the partition, we will not be able to assign a merge for newly created parts. As a result, the total number of parts within the partition could grow uncontrollably, similar to a snowball effect.

Think about it: if the selector always preferred the oldest parts, then in a partition with 10 000 parts it would always be merging parts 1..100, which are ancient and useless to merge, while parts 9000..10000 — the ones that arrived in the last minute — never get merged. The partition’s part count never goes down. The merge queue is always busy with work that does not help.

By only ever considering the last N parts, every merge consumes recent parts, so the recent parts get compacted and the partition’s part count actually falls. It is a one-line fix for a self-reinforcing failure mode, and it is worth more than most of the arithmetic around it.

The stochastic variant (StochasticSimpleMergeSelector) randomizes the window position, so that different replicas do not all merge the same parts:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:275
size_t calculateRangeWithStochasticSliding(size_t parts_count, size_t parts_threshold)
{
 auto mean = static_cast<double>(parts_count);
 std::normal_distribution<double> distribution{mean, mean / 4};

 size_t right_boundary = static_cast<size_t>(distribution(thread_local_rng));
 if (right_boundary > parts_count)
 right_boundary = 2 * parts_count - right_boundary;

 right_boundary = std::max(right_boundary, parts_threshold);

 return right_boundary - parts_threshold;
}

A normal distribution with mean parts_count and standard deviation parts_count / 4, folded back into range by mirroring at parts_count. The std::max(right_boundary, parts_threshold) at the end guarantees the window never shrinks below window_size — the anti-snowball guarantee is never traded away for randomization.

The blurry base

The StochasticSimpleMergeSelector adds one more trick: instead of a sharp threshold on the base, it uses a probability distribution around the lowered base.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:256
 if (settings.use_blurry_base)
 {
 double partition_fill_factor = std::max(0., 1 - partition_size / static_cast<double>(settings.parts_to_throw_insert));
 /// Scale factor controls when (relativelty to the number of parts in partition)
 /// do we activate our special algorithm.
 /// With standard parameters the logic kicks in starting from 80% empty factor.
 /// The division by 2 is due to the fact that for normal distribution nearly 95.4%
 /// of all observations fall within two standard deviations.
 double scaling_factor = std::pow(partition_fill_factor, settings.blurry_base_scale_factor) / 2;
 /// The base lower than 1 doesn't make sense, so we try to avoid it.
 std::normal_distribution<double> distribution{lowered_base, (lowered_base - 1) * scaling_factor};
 /// The threshold should be strictly bigger than 1, because we don't allow to merge the part with itself.
 lowered_base = std::min(distribution(thread_local_rng), std::max(1.01, lowered_base));
 }

The idea: instead of “merge if base ≥ 5.0”, it says “merge if base ≥ a randomly-drawn threshold around 5.0”. A merge whose base is 4.9 might happen this round and not the next. The result is that the distribution of merges is smoother — no sharp cliff at a particular base value — and, critically, the merge decisions of different replicas do not line up.

The width of the distribution grows with the emptiness of the partition: partition_fill_factor is 0 when the partition is full (parts_to_throw_insert parts) and approaches 1 when it is empty. blurry_base_scale_factor (default 42) makes the effect very nonlinear — the blur only kicks in when the partition is nearly empty, which is exactly when you want merges to be infrequent and gentle.

The final std::min(distribution(...), std::max(1.01, lowered_base)) is a careful clamp: the drawn threshold can be lower than the base (making the merge more eager) but never lower than 1.01, because a base ≤ 1 would mean merging a part with itself.

Picking the winner

The scored candidates go into a max-heap, and the best admissible, non-overlapping one wins:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:142
 std::optional<PartsRange> buildMergeRange(const MergeConstraint & constraint)
 {
 constexpr static auto range_compare = [](const ScoredRange & lhs, const ScoredRange & rhs)
 {
 /// If the ranges have the same score, use the range that is most likely to have the lower merge level -
 /// the right one, according to the block number sorting.
 return lhs.score > rhs.score || (lhs.score == rhs.score && lhs.range_begin < rhs.range_begin);
 };

(the tie-break is “the right one”, i.e. the newer parts — the opposite of what a naive reading of “oldest first” would suggest, and consistent with the sliding window.)

The disjoint_set.addRangeIfPossible call in the pop loop enforces that two concurrent merges never claim overlapping ranges of parts. The disjoint set is implemented as a std::set of range boundaries per independent range, with an upper_bound lookup:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/DisjointPartsRangesSet.cpp:113
bool DisjointPartsRangesSet::addRangeIfPossible(RangesIterator range_it, PartsIterator range_begin, PartsIterator range_end)
{
 chassert(sanityCheck(uncovered_ranges, range_it, range_begin, range_end));

 auto & sorted_ranges = disjoint[range_it.base()];
 PartsRangeBoundaries range_boundaries{std::move(range_begin), std::move(range_end)};

 if (!isDisjoint(range_boundaries, sorted_ranges))
 return false;

 sorted_ranges.insert(std::move(range_boundaries));
 return true;
}

From parts to a merge: the whole pipeline

The selector is one step of a longer pipeline. The entry point is MergeTreeDataMergerMutator::selectPartsToMerge:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:328
std::expected<MergeSelectorChoices, SelectMergeFailure> MergeTreeDataMergerMutator::selectPartsToMerge(
 const PartsCollectorPtr & parts_collector,
 const MergePredicatePtr & merge_predicate,
 const MergeSelectorApplier & selector,
 const std::optional<PartitionIdsHint> & partitions_hint)
{
 const auto context = data.getContext();
 const auto settings = data.getSettings();
 const auto metadata_snapshot = data.getInMemoryMetadataPtr(context, false);
 const auto storage_policy = data.getStoragePolicy();
 const time_t current_time = std::time(nullptr);
 const bool can_use_ttl_merges = !ttl_merges_blocker.isCancelled();
 LogSeriesLimiter series_log(log, 1, /*interval_s_=*/60 * 30);

 auto collected = collectAllPossibleRanges(parts_collector, metadata_snapshot, storage_policy, current_time, partitions_hint, series_log);

Three phases:

collect — find the ranges of parts that could be merged (same partition, compatible schema, not already covered by a running merge);
split by merge predicate — cut those ranges wherever a merge is forbidden (a part that overlaps a running merge, a part with a different schema version, and so on);
choose — run the selector’s cost model over what is left.

The LogSeriesLimiter is a nice touch: the trace log of this function is rate-limited to one message per 30 minutes, because the function is called on every selection round and can produce a lot of output.

The background scheduler’s backoff

Selection is cheap; the merge is not. The background scheduler that runs the selection loop uses an exponential backoff when there is nothing to do:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:62
void BackgroundJobsAssignee::postpone()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 no_work_done_count += 1;
 double random_addition = std::uniform_real_distribution<double>(0, sleep_settings.task_sleep_seconds_when_no_work_random_part)(rng);

 size_t next_time_to_execute = static_cast<size_t>(
 1000 * (data.getBiasBackoffSeconds() + std::min(
 sleep_settings.task_sleep_seconds_when_no_work_max,
 sleep_settings.thread_sleep_seconds_if_nothing_to_do * std::pow(sleep_settings.task_sleep_seconds_when_no_work_multiplier, no_work_done_count))
 + random_addition));

 holder->scheduleAfter(next_time_to_execute, false);
}

with the settings:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.h:18
struct BackgroundTaskSchedulingSettings
{
 double thread_sleep_seconds_random_part = 1.0;
 double thread_sleep_seconds_if_nothing_to_do = 0.1;
 double task_sleep_seconds_when_no_work_max = 600;
 /// For exponential backoff.
 double task_sleep_seconds_when_no_work_multiplier = 1.1;

 double task_sleep_seconds_when_no_work_random_part = 1.0;

So: start at 0.1 s, multiply by 1.1 per idle round, cap at 600 s, plus up to 1 s of random jitter. An idle table’s selection loop slows down to one attempt every ten minutes after about 130 idle rounds, and the jitter prevents a fleet of servers from synchronizing their wake-ups.

And when work does appear, the backoff is unwound geometrically rather than reset:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:48
void BackgroundJobsAssignee::trigger()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 /// Do not reset backoff factor if some task has appeared,
 /// but decrease it exponentially on every new task.
 no_work_done_count /= 2;
 /// We have background jobs, schedule task as soon as possible
 holder->schedule();
}

no_work_done_count /= 2 rather than = 0. Halving means that after one new task the sleep time is back to roughly its previous value, and it takes several rounds of activity to fully unwind. A table that receives one part per minute therefore does not flip between “poll every 0.1 s” and “poll every 600 s” — it settles at a rate proportional to the arrival rate. That is a genuinely nice piece of control theory for what looks like three lines of code.

The whole picture

 text
 
 
 parts in a partition (by block number)
 ┌────┬────┬────┬────┬────┬──────────────┬────┬────┐
 │ p1 │ p2 │ p3 │ p4 │ p5 │ ... │pn-1│ pn │
 └────┴────┴────┴────┴────┴──────────────┴────┴────┘
 ↑ ↑
 └──────── sliding window (last N=1000) ─────┘
 (only these are eligible)

 for each contiguous sub-range of the window:
 base = sum_size / max_size
 if base < lowered_base(age, partition_size, range_size):
 skip
 score = (sum_size + 5MB * count) / (count - 1.9)
 trim small parts off the right tail
 adjust score for power-of-two alignment with the part on the left

 pick the lowest-scoring admissible range that does not overlap
 an already-assigned merge

What this buys you, and what it costs

It buys you a merge tree that stays balanced, which is what keeps write amplification at O(log n) instead of O(n). The base knob is the lever, and the whole selector is a machine for keeping base near its target.

It buys you a system that self-corrects. When a partition fills up, num_parts_normalized rises, the base is lowered, and merges become more eager — without anybody tuning anything. When a partition sits idle, the opposite happens.

It buys you an anti-snowball guarantee. The sliding window is the difference between a table whose part count stabilizes and one that grows without bound.

It costs you a cost model that is admittedly approximate. The header comment says so:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:57
It's still unclear if this algorithm is good or optimal at all. It's unclear if this algorithm is using the optimal coefficients.

and it describes how the coefficients were tuned (an insert/merge simulator, replayed against production system.part_log data), and how the research found slightly better coefficients that were not used because the test data’s representativeness was in question.

It costs you a - 1.9. The score function’s singularity is papered over with a constant rather than derived. It is a fudge factor, honestly labelled as one.

It costs you a lot of tuning surface. merge_selector_base, merge_selector_window_size, parts_to_throw_insert, max_parts_to_merge_at_once, min_parts_to_merge_at_once, merge_selector_small_parts_*, the align-parts heuristic, the right-tail heuristic, the blurry base, the stochastic sliding window. Every one of these is a lever that was pulled for a reason, and the reason is usually written in a comment somewhere in SimpleMergeSelector.h.

Seeing it in action

The merge list shows what was selected and why:

 sql
 
 
SELECT
 database,
 table,
 elapsed,
 progress,
 num_parts,
 formatReadableSize(total_size_bytes_compressed) AS size,
 formatReadableSize(memory_usage) AS mem,
 result_part_name
FROM system.merges
ORDER BY elapsed DESC;

And the parts-per-partition histogram is the metric the selector is ultimately optimizing:

 sql
 
 
SELECT
 database,
 table,
 partition,
 count() AS parts,
 sum(rows) AS rows,
 formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active
GROUP BY database, table, partition
ORDER BY parts DESC
LIMIT 20;

A partition with hundreds of parts means the selector is not keeping up — usually because the merge pool is saturated, or because the parts are too large to merge under parts_to_throw_insert pressure, or because merge_selector_base is set too high for the workload.

And if you want to see the cost model at work, the trace log has it:

 sql
 
 
SET log_level = 'trace';
SELECT count() FROM hits WHERE URL LIKE '%very-selective-prefix%';

The MergeTreeDataMergerMutator::selectPartsToMerge trace messages show which ranges were considered and which one was chosen.

How ClickHouse Caches the Answer to 'Which Granules Can Match This Predicate?'

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Index analysis is not free. Every time a query runs, ClickHouse has to work out which granules of which parts could possibly contain a matching row. That work involves evaluating the predicate against the primary key index, against the minmax indexes, against the skip indexes, and — for the harder cases — running a recursive exclusion search over the marks.

Now consider the shape of a real workload: a monitoring dashboard that runs the same query every 30 seconds, a service that filters on tenant_id = 42 over and over, a batch job that re-reads yesterday’s data every hour. The predicate is the same every time. The data is not — but the data is also mostly not matching.

This is a textbook case for memoization, and ClickHouse implements it as the QueryConditionCache: a global LRU that remembers, for each (table UUID, part name, predicate hash) triple, a bitmap of the granules that the predicate emptied during the last scan.

What the cache actually stores

The class doc is unusually explicit about the semantics:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:26
/// An implementation of predicate caching a la https://doi.org/10.1145/3626246.3653395
///
/// Given the table, part name and a hash of a predicate as key, caches which marks definitely don't match the predicate and which marks may
/// match the predicate. This allows to skip the scan if the same predicate is evaluated on the same data again. Note that this doesn't work
/// the other way round: we can't tell if _all_ rows in the mark match the predicate.
///
/// Note: The cache may store more than the minimal number of matching marks. For example, assume a very selective predicate that matches
/// just a single row in a single mark. One would expect that the cache records just a single mark as potentially matching:
/// 000000010000000000000000000
/// But it is equally correct for the cache to store this.
/// 000001111111110000000000000
/// It is just less efficient for pruning (false positives).
class QueryConditionCache
{
public:
 /// False means none of the rows in the mark match the predicate. We can skip such marks.
 /// True means at least one row in the mark matches the predicate. We need to read such marks.
 using MatchingMarks = std::vector<bool>;

Two things to internalize.

It is one-directional. The cache records “this granule matched nothing”. It never records “this granule matched everything” — because that is not something a scan can establish. If a granule produced rows, the cache cannot tell whether all of its rows matched.

It is allowed to be imprecise, in one direction only. The bitmap’s true means “may match”. A granule that matched one row in a million is marked true — the whole granule has to be read. The cache never produces a false negative, which is the only thing that matters for correctness.

MatchingMarks is a std::vector<bool> — a bit-packed boolean array, one bit per granule. A part with 100 000 granules costs 12.5 KB per predicate.

The whole mechanism is controlled by use_query_condition_cache, which defaults to true.

The key: three components, hashed

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:45
private:
 /// A hash of the table id, part name and condition id.
 /// CityHash128 is enough to use for practical applications as the probability of collisions is very low.
 /// https://github.com/ClickHouse/ClickHouse/issues/9506
 using Key = UInt128;

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:237
QueryConditionCache::Key QueryConditionCache::makeKey(const UUID & table_id, const String & part_name, UInt64 condition_hash)
{
 SipHash hash;
 hash.update(table_id);
 hash.update(part_name);
 hash.update(condition_hash);
 return hash.get128();
}

The key is the part, so a cached verdict about one part never leaks into another. The part name is qualified by the table UUID, so two tables with the same part name (a very common thing in MergeTree — part names are derived from block ranges and partition values) do not collide.

There is a wrinkle for file-backed engines, where “part” is a file:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:93
 /// Compose the `part_name` component of a cache key for a file-backed table (e.g. `File`, `S3`,
 /// object storage). Uses the full path (not just the base name) so files that share a name in
 /// different directories do not collide, and folds in a content-version token so an in-place
 /// rewrite of the file yields a different key rather than a stale hit. The token is the ETag for
 /// remote objects, or a local identity (modification time + inode + size) for local files. For
 /// immutable files (e.g. data-lake data files) the path alone is a stable identity and the token
 /// may be left empty. The path and the token are separated by a NUL byte, which cannot occur in
 /// either, so the mapping is unambiguous.
 static String makeFilePartName(const String & path, std::string_view version_token);

“folds in a content-version token so an in-place rewrite of the file yields a different key rather than a stale hit” — for a file-backed table, the “part” can change under you, and a stale “this file matches nothing” verdict would be a correctness bug. The ETag (or mtime + inode + size) is folded in.

The safety property: everything starts as “may match”

This is the detail that makes the whole thing safe. When an entry is created, every granule is marked true — “may match” — so a fresh entry skips nothing:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:396
QueryConditionCache::Entry::Entry(size_t mark_count)
 : matching_marks(mark_count, true) /// by default, all marks potentially are potential matches, i.e. we can't skip them
{
}

and the header explains why that default is a correctness requirement, not a performance one:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:76
 /// (**) About error handling: There could be an exception after the i-th scan and cache entries could (theoretically) be left in a
 /// corrupt state. If we are not careful, future scans queries could then skip too many ranges. To prevent this, it is important to
 /// initialize all marks of each entry as non-matching. In case of an exception, future scans will then not skip them.

Consider the failure mode. A query scans part P with predicate Q. Partway through, the query is cancelled or throws. If the cache had been updated optimistically — “these granules are going to match nothing” — the cancellation would leave a verdict that skips granules the predicate would in fact have matched. The query would return wrong results, silently, forever, because the cache entry survives.

By initializing everything to “may match” and only ever clearing bits after the scan has actually proven the granule empty, a crash can only lose information (the cache knows less than it could), never gain it.

Writing: only clear bits, never set them

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:272
void QueryConditionCache::write(
 const UUID & table_id, const String & part_name, UInt64 condition_hash, const String & condition,
 const MarkRanges & mark_ranges, size_t marks_count, bool has_final_mark)
{
 if (table_id == UUIDHelpers::Nil)
 return; /// Issue #92863: Certain database engines provide no table UUIDs

 Key key = makeKey(table_id, part_name, condition_hash);

#if defined(DEBUG_OR_SANITIZER_BUILD)
 auto load_func = [&](){ return std::make_shared<Entry>(marks_count, table_id, part_name, condition_hash, condition); };
#else
 auto load_func = [&](){ return std::make_shared<Entry>(marks_count); };
#endif

 auto [entry, inserted] = cache.getOrSet(key, load_func);

getOrSet is atomic: two concurrent scans of the same part with the same predicate cannot create two entries; they share one.

Then the update, and note the direction of the operation:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:311
 {
 std::lock_guard lock(entry->mutex); /// (*)

 chassert(marks_count == entry->matching_marks.size());

 /// The input mark ranges are the areas which the scan can skip later on.
 for (const auto & mark_range : mark_ranges)
 std::fill(entry->matching_marks.begin() + mark_range.begin, entry->matching_marks.begin() + mark_range.end, false);

 if (has_final_mark)
 entry->matching_marks[marks_count - 1] = false;
 }

std::fill(..., false) — the entry is only ever narrowed. The mark ranges passed in are the ones the scan proved empty. Nothing is ever set back to true, because “this granule matched something” is not a fact the cache can rely on: a later scan of the same part with the same predicate might read a different set of marks (because another query’s cached verdict pruned some of them!), and those un-read marks must stay “may match”.

That last point is subtle and important: the cache is a cache of negative results only, and the negative results are monotone. Once a granule is known empty for a predicate, it stays known empty for as long as the entry lives. The entry is invalidated by the part’s identity changing (a new part has a new name), never by the predicate being re-evaluated.

There is a fast path that avoids taking the exclusive lock when there is nothing to do:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:289
 /// Try to avoid acquiring the RW lock below (*) by early-ing out. Matters for systems with lots of cores.
 {
 std::shared_lock shared_lock(entry->mutex); /// cheap

 bool need_not_update_marks = true;
 for (const auto & mark_range : mark_ranges)
 {
 /// If the bits are already in the desired state (false), we don't need to update them.
 need_not_update_marks = std::all_of(entry->matching_marks.begin() + mark_range.begin,
 entry->matching_marks.begin() + mark_range.end,
 [](auto b) { return b == false; });
 if (!need_not_update_marks)
 break;
 }

 /// Do we either have no final mark or final mark is already in the desired state?
 bool need_not_update_final_mark = !has_final_mark || entry->matching_marks[marks_count - 1] == false;

 if (need_not_update_marks && need_not_update_final_mark)
 return;
 }

Most writes are no-ops: the same query, run repeatedly, finds the same granules empty, and the bits are already clear. The shared lock is cheap and the exclusive lock is never taken.

Why the entry has its own mutex

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:71
 /// (*) You might wonder why Entry has its own mutex considering that CacheBase locks internally already. The reason is that
 /// ClickHouse scans ranges within the same part in parallel. The first scan creates and inserts a new Key + Entry into the cache,
 /// the 2nd ... Nth scans find the existing Key and update its Entry for the new ranges. This can only be done safely in a
 /// synchronized fashion.

The CacheBase lock protects the map — the mapping from key to entry. It does not protect the contents of the entry. And the contents are mutated concurrently: a query with 16 streams reads 16 disjoint mark ranges of the same part simultaneously, and each of them writes back the ranges it proved empty.

Without a per-entry lock, two threads could be doing std::fill(begin, mid, false) and std::fill(mid, end, false) simultaneously on the same std::vector<bool>. std::vector<bool> packs eight booleans into a byte, so the two fills could land in the same byte and lose one another’s writes — a lost update, which would mean a granule that was proven empty stays “may match”. Not a correctness bug (only a lost optimization), but a real one.

The SharedMutex lets the many readers of a cached verdict proceed in parallel while the rare writer excludes them.

Reading: turning a bitmap back into mark ranges

The consult side is in MergeTreeDataSelectExecutor::filterPartsByQueryConditionCache. After the cache lookup, the bitmap is converted back into MarkRanges — the surviving granules:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1944
 for (const auto & mark_range : part_with_ranges.ranges)
 {
 size_t begin = mark_range.begin;
 for (size_t mark_it = begin; mark_it < mark_range.end;)
 {
 if (!matching_marks[mark_it])
 {
 if (mark_it == begin)
 {
 /// mark_range.begin -> 0 0 0 1 x x x x. Need to skip starting zeros.
 ++stats.granules_dropped;
 ++begin;
 ++mark_it;
 }
 else
 {
 size_t end = mark_it;
 for (; end < mark_range.end && !matching_marks[end]; ++end)
 ;

 if (min_marks_for_seek && end != mark_range.end && end - mark_it <= min_marks_for_seek)
 {
 /// x x x 1 1 1 0 0 1 x x x. And gap is small enough to merge, skip gap.
 mark_it = end + 1;
 }
 else
 {
 /// Case1: x x x 1 1 1 0 0 1 x x x. Gap is too big to merge, do not merge
 /// Case2: x x x 1 1 1 0 0 0 0 -> mark_range.end. Reach the end of range, do not merge
 stats.granules_dropped += end - mark_it;
 ranges.emplace_back(begin, mark_it);
 begin = end;

 if (end == mark_range.end)
 break;

 mark_it = end + 1;
 }
 }
 }
 else
 ++mark_it;
 }

The comments in that loop are worth reading twice. 0 means “skip this granule”, 1 means “must read it”, x means “unknown, keep it”.

Leading zeros are skipped: 0 0 0 1 x x x x → the range starts at the first 1.
A gap of zeros between two ones is dropped — unless the gap is smaller than min_marks_for_seek, in which case reading through it is cheaper than seeking past it. That is the same seek heuristic used by the index analysis, and it is applied here for the same reason.

And if every granule of the part was eliminated, the part is dropped from the query entirely:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1994
 if (ranges.empty())
 it = parts_with_ranges.erase(it);
 else
 {
 part_with_ranges.ranges = std::move(ranges);
 ++it;
 }

A part that the cached verdict proves empty is not opened at all. No marks read, no decompressed, no filters evaluated.

The predicate hash is not just the predicate

The most delicate part of the design is the key. If two different predicates hashed to the same value, one query’s verdict would be applied to another query’s data — silently returning wrong results.

The base hash is the ActionsDAG hash of the filter expression, which covers the function names, the result types, the children and the constant values. But that is not enough. Consider:

 sql
 
SELECT * FROM t WHERE ts >= now() - INTERVAL 10 DAY;

The DAG hash changes every time the query runs (the constant changes), so this predicate never gets a cache hit. Fine — but there is a subtler problem in the other direction: a setting can change a function’s result without changing the DAG. formatDateTime with different flags produces different strings from the same DAG. least/greatest propagate or skip NULLs differently. h3ToGeo swaps its arguments.

So the cache key includes a salt computed from every setting that a function might capture at build time:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:80
UInt64 queryConditionCacheSettingsSalt(const Settings & settings)
{
 /// Registration rule: a setting belongs here when a function captures it at build time (in its constructor
 /// or `build`), it changes the value the function returns, and nothing about it reaches
 /// `ActionsDAG::Node::updateHash` (which sees only the function name, the result type name, the children and
 /// the constant values). A setting that changes the result *type* is already covered by the type name in the
 /// hash and does not need an entry.
 SipHash hash;

and the salt function goes on for a hundred lines, hashing the values of formatdatetime_f_prints_single_zero, formatdatetime_parsedatetime_m_is_month_name, function_locate_has_mysql_compatible_argument_order, least_greatest_legacy_null_behavior, h3togeo_lon_lat_result_order, short_circuit_function_evaluation, session_timezone, date_time_output_format, bool_true_representation, input_format_null_as_default, and dozens more.

The last one is the most interesting:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:219
 /// `session_timezone` is not read by any function when it is built: `DateLUT::instance()` looks it up in the
 /// query context every time a function needs the implicit time zone, which is what `toDateTime(s)`,
 /// `parseDateTime(s, format)` and their relatives do for a `String` argument without an explicit time zone
 /// (`KeyCondition` refuses to analyze exactly these for the same reason). The result type is the plain
 /// `DateTime` without a time zone in its name, so the same `s` parses to a different Unix timestamp in two
 /// sessions with an identical DAG hash.
 hash.update(settings[Setting::session_timezone].value);

Two sessions in different time zones produce the same DAG for toDateTime(s), but parse s differently. Without the salt, one session’s verdict would be served to the other. This is the kind of bug that is nearly impossible to find by testing and completely obvious once you have read the rule.

The time-condition trick

now() in a predicate is a special case, and it is handled with a derived condition rather than a salt.

The problem: WHERE ts >= now() - INTERVAL 10 DAY has a different DAG hash every second, so it never caches. But it is a very common predicate.

The solution: round the time constant onto a grid. On the write side, the condition is strengthened; on the read side, it is weakened:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/updateQueryConditionCache.cpp:107
 /// Non-deterministic conditions must not be cached under their own hash (issues #81506 and #84508).
 /// However, for conditions that are non-deterministic only because of folded current-time constants
 /// (e.g. `time >= now() - INTERVAL 10 DAY`), a deterministic condition can be derived by rounding
 /// the constants onto a time grid (issue #115504). Since this is the write side, the rounding must
 /// *strengthen* the condition: "the WHERE filter matched no rows of this granule" then implies
 /// "the derived condition matches no rows of this granule", so the entry stored under the derived
 /// hash is sound. The read side (filterPartsByQueryConditionCache) probes the weakened variant;
 /// the two coincide for grid-aligned constants and are one grid cell apart otherwise.

Read the logic carefully, because the asymmetry is the whole trick.

Write side (strengthen): the actual filter used now() - 10 days. The derived condition is a rounded version, e.g. now() - 10 days rounded up to the next grid point, which is a narrower condition. If the narrow condition matched no rows in this granule, then the wider (actual) condition certainly matched no rows either. So the entry (written under the derived hash) is sound.
Read side (weaken): the query now probes with the rounded-down version, which is a wider condition. “The wider condition matched no rows” implies “the narrower condition matched no rows” — so skipping is sound.

The two coincide for grid-aligned constants and are one grid cell apart otherwise. The grid factor is a setting (query_condition_cache_time_condition_grid_factor).

This is a genuinely elegant piece of reasoning: it takes a predicate that is not cacheable (its value changes every second) and makes it cacheable by proving that a slightly different, grid-aligned predicate implies the right thing in both directions.

The skip-index profile: another salt

There is one more key problem, and it is the subtlest of all.

Some cached verdicts come from skip index analysis — “the minmax index on this column says no row in this granule can match”. But skip index analysis depends on which indexes exist and are enabled. If a query runs with use_skip_indexes = 0, and another runs with them on, the verdicts differ.

So the skip-index verdicts are stored under a profiled key — the plain condition hash salted with the profile of the index set that produced the verdict:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1771
 /// Each condition is looked up under two keys (see the write sides):
 /// * the bare condition hash, holding row-level exclusions (FilterTransform,
 /// MergeTreeSelectProcessor, object storage) which are always sound; and
 /// * a hash salted with the effective skip-index profile that index analysis ran
 /// (getSkipIndexProfiledConditionHash), holding the skip-index-analysis exclusions
 /// written by ReadFromMergeTree.
 /// Skip-index exclusions can legitimately diverge from the row-level predicate (e.g. a text
 /// index with a preprocessor), so they must only be consulted by a query that ran the same set
 /// of indexes; salting the key with the running index set (and disjunction mode) guarantees a
 /// query that ran a different set (use_skip_indexes = 0, an index dropped/ignored, or a
 /// different disjunction mode) never reads them. The two verdicts are merged: a mark may be
 /// skipped iff either verdict says it does not match. This keeps the pure-QCC case
 /// (use_skip_indexes = 0 still reusing row-level entries) working while preventing the
 /// skip-index poisoning of issue #108519. TopK reads (which only get here when
 /// `use_query_condition_cache_for_top_k` is on, see the gate above) consult TopK-salted keys
 /// and also the `topk_reuse_predicate_only_hash` so plain `SELECT ... WHERE` entries can be
 /// reused; TopK-salted entries are not read otherwise.

“a mark may be skipped iff either verdict says it does not match” — the merge is an AND of the two “may match” bitmaps:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1881
 auto merge_opt_marks = [](QueryConditionCache::MatchingMarks & matching_marks, const std::optional<QueryConditionCache::MatchingMarks> & marks_opt)
 {
 if (!marks_opt)
 return;
 if (matching_marks.empty())
 matching_marks = *marks_opt;
 else if (matching_marks.size() == marks_opt->size())
 for (size_t i = 0; i < matching_marks.size(); ++i)
 matching_marks[i] = matching_marks[i] && (*marks_opt)[i];
 };

AND of the “may match” bits: a granule is skipped only if both verdicts agree it cannot match. Conservative in the right direction.

Where the “empty granule” fact comes from

The write side is the read pipeline itself. When the PREWHERE filter processes a granule and the filter removes every row of it, the read task records the granule as “unmatched”:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeReadTask.cpp:590
void MergeTreeReadTask::addPrewhereUnmatchedMarks(const MarkRanges & mark_ranges_)
{
 prewhere_unmatched_marks.insert(prewhere_unmatched_marks.end(), mark_ranges_.begin(), mark_ranges_.end());
}

and the write happens when the task is finished:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSelectProcessor.cpp:617
 if (reader_settings.use_query_condition_cache && task && prewhere_info
 && !task->readersChainCanSkipMarksBeforePrewhere(prewhere_filters_by_top_k_threshold)
 && !task->appliesMutationsBeforePrewhere()
 && !row_level_filter
 /// QueryConditionCache needs the concrete part's storage UUID; skip for borrowed parts.
 && task->getInfo().data_part_info->getDataPart())
 {

Every one of those four extra conditions is a soundness condition, and each has a comment explaining it. The interesting ones:

!task->readersChainCanSkipMarksBeforePrewhere(...) — if a skip index upstream of PREWHERE could have filtered marks before PREWHERE saw them, then the marks PREWHERE “emptied” are not the marks PREWHERE would have emptied, and attributing them to the PREWHERE predicate would be a lie. (Issue #104781.)
!task->appliesMutationsBeforePrewhere() — on-the-fly mutations filter rows ahead of PREWHERE, for the same reason.
!row_level_filter — a row-level security filter is prepended before PREWHERE, but the cache key covers only the query’s PREWHERE hash. A mark hidden by a row policy must not be attributed to the PREWHERE predicate, because a later query without that policy would then skip rows it should see.

That last one is a beautiful example of a cache whose key does not cover everything that influenced the result — the fix is to not write in that case, rather than to make the key bigger.

The whole flow

 text
 
 
 query with predicate Q
 │
 ▼
 hash(Q) + settings salt + [skip-index profile]
 │
 ├──► QueryConditionCache::read(table_uuid, part, hash)
 │ │
 │ └──► bitmap of "may match" granules
 │ │
 │ ▼
 │ surviving MarkRanges per part
 │ (parts with no surviving ranges are dropped entirely)
 │
 ▼
 index analysis (primary key, minmax, skip indexes)
 │
 ▼
 scan the surviving granules
 │
 ├── granules where the filter removed everything ──► cache.write(cleared bits)
 │
 └── rows ──► result

What this buys you, and what it costs

It buys you the elimination of repeated index analysis. The expensive part of “which granules can match” is not the bitmap lookup, it is the KeyCondition evaluation and the exclusion search. On a warm cache, that work is replaced by a hash lookup and a bit scan.

It buys you the elimination of reading granules that match nothing. This is the bigger win. The index analysis can only prove a granule is empty using the index; a predicate on a non-key column cannot be resolved from the index at all. But once a scan has observed that a granule contains no matching rows, that fact is cached, and the next run does not read it.

It costs you memory. One bit per granule per (table, part, predicate). The cache is a CacheBase LRU with a max_size_bytes, and the weight function accounts for the bit-packed vector plus a fixed overhead:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:229
size_t QueryConditionCache::EntryWeight::operator()(const Entry & entry) const
{
 size_t memory = sizeof(Key) + sizeof(Entry);
 /// Estimate the memory size of `std::vector<bool>` (it uses bit-packing internally)
 /// Round up to bytes.
 memory += (entry.matching_marks.capacity() + 7) / 8;
#if defined(DEBUG_OR_SANITIZER_BUILD)
 memory += entry.part_name.capacity() + entry.condition.capacity();
#endif
 return memory;
}

It costs you correctness vigilance. Everything in this design is about not serving a verdict that was computed under different circumstances: a different predicate, a different set of skip indexes, a different time zone, a different row-level policy, a TopK filter that elides granules. The salt functions, the profiles, the derived time conditions, and the !row_level_filter guard are all the same idea applied four times.

It gives up on one direction. The cache cannot record “this granule matches everything”, so a highly selective predicate that matches a few rows in every granule buys nothing. The cache is for predicates that are granule-level selective — which is most of them, since a granule is 10 MB of data.

Seeing it in action

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN ('QueryConditionCacheHits', 'QueryConditionCacheMisses');

A hit/miss ratio that is high and stable is the signature of a workload with repeated predicates over stable data. A ratio near zero means the predicates are always new — dashboards with per-request filters, for instance — and the cache is pure overhead.

The cache is also inspectable:

 sql
 
 
SELECT *
FROM system.query_condition_cache
LIMIT 10;

And to see it working, run the same query twice and compare the SelectedMarks and SelectedRanges counters in system.query_log:

 sql
 
 
SELECT
 query_id,
 ProfileEvents['SelectedMarks'] AS marks,
 ProfileEvents['SelectedRanges'] AS ranges,
 ProfileEvents['QueryConditionCacheHits'] AS qcc_hits
FROM system.query_log
WHERE query LIKE '%tenant_id = 42%'
ORDER BY event_time;

The second run should show dramatically fewer selected marks — and, if the predicate empties whole granules, the same query will not even open the part.

How ClickHouse Caches Query Results in Memory and on Disk

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A query result cache is the oldest trick in the database book: run the same query twice, serve the second one from memory. What makes it hard is not the storage but the key. Two queries that produce different results must not share an entry, and two queries that produce identical results should. Textual equality is hopeless — the same query can be spelled with different whitespace, different letter case, a different current database, or a different value of a setting that changes the answer.

ClickHouse’s answer is to key the cache on a hash of the parsed query, plus the settings that could change the answer, plus the identity of the user. This article walks the whole path: the key construction, the TTL cache, the writer’s many reasons to refuse an entry, the reader’s access checks, and the on-disk backend that stores entries in a filesystem cache.

Why the cache is hard to get right

The header of the cache class states the contract plainly:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:48
/// Maps queries to query results. Useful to avoid repeated query calculation.
///
/// The cache does not aim to be transactionally consistent (which is difficult to get right). For example, the cache is not invalidated
/// when data is inserted/deleted into/from tables referenced by queries in the cache. In such situations, incorrect results may be
/// returned. In order to still obtain sufficiently up-to-date query results, a expiry time (TTL) must be specified for each cache entry
/// after which it becomes stale and is ignored. Stale entries are removed opportunistically from the cache, they are only evicted when a
/// new entry is inserted and the cache has insufficient capacity.

So the correctness story is “TTL, not invalidation”. A result is valid for as long as its TTL says, and the entry is not invalidated when the underlying table changes. This is a deliberate trade: tracking table versions per entry would make every entry depend on the write path.

The key: a hash of the AST, not of the text

The key’s hashed part is an IASTHash — a 128-bit hash of the parsed query:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:59
 struct Key
 {
 /// ----------------------------------------------------
 /// The actual key (data which gets hashed):

 /// The hash of the query AST.
 /// Unlike the query string, the AST is agnostic to lower/upper case (SELECT vs. select).
 IASTHash ast_hash;

The AST is not hashed as-is, though. Two normalisations happen first.

Query-cache settings are removed. A query that says SETTINGS use_query_cache = true should hit the entry written by the same query without that clause:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:500
ASTPtr removeQueryResultCacheSettings(ASTPtr ast)
{
 RemoveQueryResultCacheSettingsMatcher::Data visitor_data;
 RemoveQueryResultCacheSettingsVisitor(visitor_data).visit(ast);

 return ast;
}

and the predicate that decides which settings are pruned is deliberately generous about output-format settings, which do not change the rows:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:413
bool settingDoesNotAffectQueryResultCache(const String & setting_name)
{
 return setting_name == "log_comment"
 /// As of today, the output format settings only affect the final output.
 /// However, it should be taken with caution - we should not use these settings in deterministic SQL functions.
 || setting_name.starts_with("output_format_")
 /// This setting is used to tune the server response, but does not affect the query behavior.
 /// An example why it should not affect query caching:
 /// - if you run a query as usual, and then run the same query with asking the server
 /// for Content-Disposition: attachment to download the result.
 || setting_name == "http_response_headers";
}

Planner-generated table aliases are removed for subqueries, so that two identical subqueries under different plans share one entry. The pattern is matched exactly, because a user identifier that happens to look like __table1 must not be rewritten:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:513
bool isPlannerGeneratedTableAlias(std::string_view name)
{
 constexpr std::string_view prefix = "__table";
 if (!name.starts_with(prefix))
 return false;
 auto suffix = name.substr(prefix.size());
 if (suffix.empty())
 return false;
 for (char c : suffix)
 {
 if (c < '0' || c > '9')
 return false;
 }
 return true;
}

What else goes into the hash

After the AST, the hash takes the current database and the changed settings:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:578
IASTHash calculateASTHash(ASTPtr ast, const String & current_database, const Settings & settings, const bool is_subquery, const bool pre_cleaned = false)
{
 if (!pre_cleaned)
 {
 ast = ast->clone();
 if (is_subquery)
 ast = removeTableAliases(ast);
 }
 ast = removeQueryResultCacheSettings(ast);

 /// Hash the AST, we must consider aliases (issue #56258)
 SipHash hash;
 ast->updateTreeHash(hash, /*ignore_aliases=*/ false);

 /// Also hash the database specified via SQL `USE db`, otherwise identifiers in same query (AST) may mean different columns in different
 /// tables (issue #64136)
 hash.update(current_database);

and then the settings, sorted, because Settings::changes() returns them in random order and a hash must be order-independent:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:596
 /// Finally, hash the (changed) settings as they might affect the query result (e.g. think of settings `additional_table_filters` and `limit`).
 /// Note: allChanged() returns the settings in random order. Also, update()-s of the composite hash must be done in deterministic order.
 /// Therefore, collect and sort the settings first, then hash them.
 auto changed_settings = settings.changes();
 std::vector<std::pair<String, String>> changed_settings_sorted; /// (name, value)
 for (const auto & change : changed_settings)
 {
 const String & name = change.name;
 if (!isSettingIgnoredInQueryResultCache(name)) /// see removeQueryResultCacheSettings() and isSubquerySpecificSetting() for why this is a good idea
 changed_settings_sorted.push_back({name, Settings::valueToStringUtil(change.name, change.value)});
 }

Only changed settings are hashed — a setting that was never touched cannot change the result relative to the default. But there is a subtlety for subqueries, because the planner forcibly sets extremes, max_result_bytes and max_result_rows for them, which would make a subquery’s key differ from the same query run at top level:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:608
 /// The Planner forcibly sets `extremes`, `max_result_bytes`, `max_result_rows` for subqueries, which makes them
 /// appear in `settings.changes()`. Non-subquery executions of the same AST typically don't have these in their
 /// changed settings. To ensure cache hits between subquery writes and subquery reads, normalize by always
 /// including the default values of these settings for subqueries.
 if (is_subquery)
 {
 if (!changed_settings.tryGet("extremes"))
 changed_settings_sorted.push_back({"extremes", settings[Setting::extremes].toString()});

The equality of two keys

Note what is not in the equality check. Two keys are equal if the AST hashes match and they are both subqueries or both not:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:715
bool QueryResultCache::Key::operator==(const Key & other) const
{
 return ast_hash == other.ast_hash && is_subquery == other.is_subquery;
}

The user id and the roles are not part of the key. They are checked at read time instead, which is why the cache can hold one entry per query and still enforce access control — see below.

The hasher is trivial, since the AST hash is already a good hash:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:720
size_t QueryResultCache::KeyHasher::operator()(const Key & key) const
{
 SipHash hash;
 hash.update(key.ast_hash.low64);
 hash.update(key.is_subquery);
 return hash.get64();
}

What may be cached at all

Not every query may be cached. A query with a non-deterministic function (now(), rand()) would return a different answer the second time. The check walks the AST and asks the function factory:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:105
 if (const auto func = FunctionFactory::instance().tryGet(function->name, data.context))
 {
 if (!func->isDeterministic())
 data.has_non_deterministic_functions = true;
 return;
 }

Two escape hatches are handled with care. The eval table function hides its real query inside an opaque string, so it cannot be inspected and must be assumed non-deterministic:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:96
 /// The `eval` table function hides its real query inside an opaque string argument, so the
 /// generated query cannot be inspected here. Treat it as non-deterministic (and, in
 /// HasSystemTablesMatcher, as touching a system table) to keep such queries out of the cache,
 /// e.g. `eval('SELECT now()')` must not be cached as if it were deterministic.
 if (function->name == "eval")
 {
 data.has_non_deterministic_functions = true;
 return;
 }

and SQL UDFs, whose determinism ClickHouse does not know:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:111
 if (const auto udf_sql = UserDefinedSQLFunctionFactory::instance().tryGet(function->name))
 {
 /// Driver-created executable functions are also persisted in the SQL-object storage,
 /// but their determinism is described by the generated executable UDF configuration checked below.
 if (!udf_sql->as<ASTCreateFunctionWithDriverQuery>())
 {
 /// ClickHouse currently doesn't know if SQL-based UDFs are deterministic or not. We must assume they are non-deterministic.
 data.has_non_deterministic_functions = true;
 return;
 }
 }

The obfuscate table function is subtler: it is deterministic if a seed is given, and the seed can be overridden by a SETTINGS clause on any enclosing query, so the check has to descend into the AST while tracking the effective settings:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:309
 if (const auto * function = node->as<ASTFunction>())
 if (function->name == "obfuscate" && context->getSettingsRef()[Setting::obfuscate_seed].value.empty())
 return true;

What the user wants to happen on a violation is a setting, with three values (throw, save, ignore):

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:371
 const QueryResultCacheNondeterministicFunctionHandling nondeterministic_function_handling
 = settings[Setting::query_cache_nondeterministic_function_handling];
 const QueryResultCacheSystemTableHandling system_table_handling = settings[Setting::query_cache_system_table_handling];

 if (ast_contains_nondeterministic_functions && nondeterministic_function_handling == QueryResultCacheNondeterministicFunctionHandling::Throw)
 throw Exception(ErrorCodes::QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS,
 "The query result was not cached because the query contains a non-deterministic function."
 " Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless, or omit caching");

So:

 sql
 
 
-- A non-deterministic query is refused by default:
SELECT * FROM events WHERE ts > now() - INTERVAL 1 HOUR;
-- ERROR: The query result was not cached because the query contains a non-deterministic function.

-- ... unless you insist:
SELECT * FROM events WHERE ts > now() - INTERVAL 1 HOUR
SETTINGS query_cache_nondeterministic_function_handling = 'save';

The cache itself

The cache is a TTL cache with per-user quotas and its own metrics:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1180
QueryResultCache::QueryResultCache(size_t max_size_in_bytes, size_t max_entries, size_t max_entry_size_in_bytes_, size_t max_entry_size_in_rows_)
 : cache(std::make_unique<TTLCachePolicy<Key, Entry, KeyHasher, EntryWeight, IsStale>>(
 CurrentMetrics::QueryCacheBytes, CurrentMetrics::QueryCacheEntries, std::make_unique<PerUserTTLCachePolicyUserQuota>()))
{
 updateConfiguration(max_size_in_bytes, max_entries, max_entry_size_in_bytes_, max_entry_size_in_rows_);
}

Staleness is a pure function of the key:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:738
bool QueryResultCache::IsStale::operator()(const Key & key) const
{
 return (key.expires_at < std::chrono::system_clock::now());
};

The eviction policy is worth spelling out, because it is not an LRU. When an insert does not fit, the cache first purges the stale entries — the ones whose TTL has passed — and if that is not enough, the insert is simply refused and the new result is dropped. The cache never evicts a fresh entry to make room for another fresh entry:

 cpp
 
 
// src/Common/TTLCachePolicy.h:237
 if (!sufficient_space_in_cache() || !sufficient_space_in_cache_for_user())
 {
 /// Remove stale entries
 for (auto it = cache.begin(); it != cache.end();)
 {
 if (is_stale_function(it->first))
 {
 size_t sz = weight_function(*it->second);
 if (it->first.user_id.has_value())
 Base::user_quotas->decreaseActual(*it->first.user_id, sz);
 it = cache.erase(it);
 size_in_bytes -= sz;
 }
 else
 ++it;
 }
 }

and the weight of an entry is the number of bytes its chunks actually allocate:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:728
size_t QueryResultCache::EntryWeight::operator()(const Entry & entry) const
{
 size_t res = 0;
 for (const auto & chunk : entry.chunks)
 res += chunk.allocatedBytes();
 res += entry.totals.has_value() ? entry.totals->allocatedBytes() : 0;
 res += entry.extremes.has_value() ? entry.extremes->allocatedBytes() : 0;
 return res;
}

Per-user quotas are pushed into the cache on every write, because there is no better hook for “the settings profile of this user changed”:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1220
 /// Update the per-user cache quotas with the values stored in the query context. This happens per query which writes into the query
 /// cache. Obviously, this is overkill but I could find the good place to hook into which is called when the settings profiles in
 /// users.xml change.
 /// user_id == std::nullopt is the internal user for which no quota can be configured
 if (key.user_id.has_value())
 cache.setQuotaForUser(*key.user_id, max_query_result_cache_size_in_bytes_quota, max_query_result_cache_entries_quota);

The two backends

 text
 
 
 query AST
 | clone + strip cache settings + strip planner aliases
 | hash(AST) + hash(current_database) + hash(changed settings)
 v
 QueryResultCache::Key { ast_hash, user_id, roles, is_shared, ttl, header }
 |
 +--> in-memory TTLCachePolicy <-- Chunks (optionally compressed)
 | | weight = allocated bytes, per-user quota
 | v
 | SourceFromChunks --> the query pipeline
 |
 +--> on-disk: FileCacheKey(salted hash) --> filesystem cache
 | FixedHeader (magic, version, sizes, ttl, build fingerprint, checksum)
 | AccessMetadata (is_shared, user, roles, tag)
 | CompressedWriteBuffer + NativeWriter (header + chunks + totals + extremes)
 v
 filesystem cache (evicted by the same rules as any other data)

Writing: many reasons to refuse

The writer buffers the result chunks as they stream by, and decides at the very end whether to keep them. That is deliberate — a query that dies half-way must not leave a partial entry:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:222
/// Buffers multiple partial query result chunks (buffer()) and eventually stores them as cache entry (finalizeWrite()).
///
/// Implementation note: Queries may throw exceptions during runtime, e.g. out-of-memory errors. In this case, no query result must be
/// written into the query result cache. Unfortunately, neither the Writer nor the special transform added on top of the query pipeline
/// which holds the Writer know whether they are destroyed because the query ended successfully or because of an exception (otherwise, we
/// could simply implement a check in their destructors). To handle exceptions correctly nevertheless, we do the actual insert in
/// finalizeWrite() as opposed to the Writer destructor. This function is then called only for successful queries in finish_callback() which
/// runs before the transform and the Writer are destroyed, whereas for unsuccessful queries we do nothing (the Writer is destroyed w/o
/// inserting anything).

The list of refusal reasons in finalizeWrite is the interesting part:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:862
 if (result_is_incomplete)
 {
 LOG_TRACE(logger, "Skipped insert because the query result is incomplete (a consumer stopped reading it before the end), query: {}", doubleQuoteString(key.query_string));
 return;
 }

 if (auto query_runtime = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now() - query_start_time); query_runtime < min_query_runtime)
 {
 LOG_TRACE(logger, "Skipped insert because the query is not expensive enough, query runtime: {} msec (minimum query runtime: {} msec), query: {}",
 query_runtime.count(), min_query_runtime.count(), doubleQuoteString(key.query_string));
 return;
 }

The second one is the classic “don’t cache the cheap queries” rule: caching a 1 ms query costs more than recomputing it. The setting is query_cache_min_query_duration.

Partial chunks are squashed into blocks of max_block_size rows before being stored, so that the cached result looks like a freshly produced one:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:889
 if (squash_partial_results)
 {
 /// Squash partial result chunks to chunks of size 'max_block_size' each. This costs some performance but provides a more natural
 /// compression of neither too small nor big blocks. Also, it will look like 'max_block_size' is respected when the query result is
 /// served later on from the query result cache.

And the entries are compressed, at the cost of a decompression on read:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:969
 if (key.is_compressed)
 {
 /// Compress result chunks. Reduces the space consumption of the cache but means reading from it will be slower due to decompression.

 Chunks compressed_chunks;

 for (const auto & chunk : query_result->chunks)
 {
 const Columns & columns = chunk.getColumns();
 Columns compressed_columns;
 for (const auto & column : columns)
 {
 auto compressed_column = column->compress(/*force_compression=*/false);
 compressed_columns.push_back(compressed_column);
 }
 Chunk compressed_chunk(compressed_columns, chunk.getNumRows());
 compressed_chunks.push_back(std::move(compressed_chunk));
 }
 query_result->chunks = std::move(compressed_chunks);
 }

column->compress(force_compression=false) is the interesting bit: a column is compressed only if it is worth it — a tiny column of random UInt64s costs more in CPU than it saves in RAM.

Finally the size limits are checked against the compressed weight:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:992
 size_t new_entry_size_in_bytes = QueryResultCache::EntryWeight()(*query_result);
 size_t new_entry_size_in_rows = count_rows_in_chunks(*query_result);

 if ((new_entry_size_in_bytes > max_entry_size_in_bytes) || (new_entry_size_in_rows > max_entry_size_in_rows))
 {
 LOG_TRACE(logger, "Skipped insert because the query result is too big, query result size: {} (maximum size: {}), query result size in rows: {} (maximum size: {}), query: {}",
 formatReadableSizeWithBinarySuffix(new_entry_size_in_bytes, 0), formatReadableSizeWithBinarySuffix(max_entry_size_in_bytes, 0), new_entry_size_in_rows, max_entry_size_in_rows, doubleQuoteString(key.query_string));
 return;
 }

 cache.set(key, query_result);

Reading: access control

The reader looks the key up and then checks whether the caller may have it. The user id and the current roles are stored in the key but are not part of the equality check, so the same entry can be found by a different user — and then rejected:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1067
 const auto & entry_key = entry->key;
 const auto & entry_mapped = entry->mapped;

 const bool is_same_user_id = ((!entry_key.user_id.has_value() && !key.user_id.has_value()) || (entry_key.user_id.has_value() && key.user_id.has_value() && *entry_key.user_id == *key.user_id));
 const bool is_same_current_user_roles = (entry_key.current_user_roles == key.current_user_roles);
 if (!entry_key.is_shared && (!is_same_user_id || !is_same_current_user_roles))
 {
 LOG_TRACE(logger, "Inaccessible query result found for query {}", doubleQuoteString(key.query_string));
 return;
 }

The rationale for checking the roles and not just the user is in the key’s comment:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:78
 /// The id and current roles of the user who executed the query.
 /// These members are necessary to ensure that a (non-shared, see below) entry can only be written and read by the same user with
 /// the same roles. Example attack scenarios:
 /// - after DROP USER, it must not be possible to create a new user with with the dropped user name and access the dropped user's
 /// query result cache entries
 /// - different roles of the same user may be tied to different row-level policies. It must not be possible to switch role and
 /// access another role's cache entries
 std::optional<UUID> user_id;
 std::vector<UUID> current_user_roles;

A stale entry is also refused:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1078
 if (QueryResultCache::IsStale()(entry_key))
 {
 LOG_TRACE(logger, "Stale query result found for query {}", doubleQuoteString(key.query_string));
 return;
 }

and the chunks are cloned (uncompressed case) or decompressed (compressed case) before being handed to a SourceFromChunks processor, which becomes the new head of the query pipeline.

The on-disk backend

The in-memory cache is bounded by RAM. The on-disk backend stores entries in a filesystem cache — the same infrastructure used for remote table data — which means the entries can be much larger, survive restarts, and (with a distributed filesystem cache) be shared between servers.

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.h:14
/// Stores query result cache entries in a preconfigured filesystem cache (an entry of the `filesystem_caches` section of the server
/// configuration), so that cached query results get more space than in memory, survive server restarts, and (with a distributed
/// filesystem cache) can be shared between servers. The filesystem cache is selected per query with setting
/// `query_cache_on_disk_cache_name`.
///
/// The entries are ordinary filesystem cache entries, indistinguishable from other data in the same filesystem cache: they are not
/// held from deletion, they are evicted by the same rules, and there are no specific limits on their total size or number other than
/// the limits of the filesystem cache itself. The keys are hashed with a salt so that they cannot intersect with other keys in the
/// filesystem cache. All metadata needed to validate an entry (TTL, access control) is serialized into the entry itself; no in-memory
/// state is kept.

The key

The on-disk key is a salted hash of the AST hash, plus the access context for non-shared entries:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:136
FileCacheKey makeFileCacheKey(const QueryResultCache::Key & key, bool is_shared)
{
 /// Salt the hash so that keys of the on-disk query result cache cannot intersect with other keys in the same filesystem cache
 /// (those are typically derived from storage paths).
 SipHash hash;
 hash.update(std::string_view("QueryResultCacheOnDisk"));
 hash.update(key.ast_hash.low64);
 hash.update(key.ast_hash.high64);
 hash.update(key.is_subquery);
 hash.update(is_shared);
 if (!is_shared)
 {
 hash.update(key.user_id.has_value());
 if (key.user_id)
 hash.update(key.user_id->toUnderType());
 /// The roles are compared as an ordered sequence, same as in the in-memory query result cache.
 hash.update(key.current_user_roles.size());
 for (const auto & role : key.current_user_roles)
 hash.update(role.toUnderType());
 }

Note that unlike the in-memory cache, the on-disk key does include the user: the file cache is shared storage, so one user’s entry must not shadow another’s.

A marker in the middle of the key lets the cache find its own entries without reading anything from disk:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:103
/// Bits 64..95 of every key of the on-disk query result cache hold this marker ("QRCC"), so that the entries can be told apart from
/// the other keys of the filesystem cache by the key alone, without reading anything from disk (see `clear`). The marker sits in the
/// middle of the key on purpose: the filesystem cache derives the directory of a key from its topmost bits and the metadata bucket
/// from its lowest bits, both of which must keep varying. The remaining 96 bits of hash are plenty to keep the entries apart. A key
/// of other data carries the marker by chance with probability 2^-32; such a key is still recognized as foreign by its content.
constexpr UInt32 KEY_MARKER = 0x51524343;

The entry format

The on-disk entry has a fixed-size header, then uncompressed access metadata, then the compressed payload:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:63
/// On-disk entry layout, version 6:
///
/// Fixed header (FIXED_HEADER_SIZE bytes):
/// char[8] magic "QRCache1"
/// UInt32 format version
/// UInt32 TCP protocol revision the Native blocks are serialized with
/// UInt64 total entry size in bytes, including the fixed header
/// UInt64 created_at, seconds since epoch
/// UInt64 expires_at, seconds since epoch
/// UInt64 fingerprint of the server build which wrote the entry (see `getServerBuildFingerprint`)
/// UInt128 SipHash-128 of the preceding fixed header bytes and of everything after the fixed header. The freshness
/// fields are authenticated as well: a corruption of `expires_at` or `total_size` alone would otherwise
/// silently change how long the entry is served, without ever failing the integrity check.
///
/// Access metadata (uncompressed):
/// UInt8 is_shared
/// UInt8 has_user_id, [UInt128 user_id]
/// varUInt number of user roles, [UInt128 role]...
/// String tag (setting `query_cache_tag`), so that `SYSTEM DROP QUERY CACHE TAG` can find the entry
///
/// Result payload (compressed with the codec from setting `query_cache_on_disk_codec`; the compression frames are
/// self-describing, so reading does not depend on the setting):
/// varUInt number of result chunks
/// Native header block (zero rows)
/// Chunk one per result chunk
/// UInt8 has_totals, [Chunk]
/// UInt8 has_extremes, [Chunk]

Two details are worth dwelling on.

The build fingerprint. An entry is only valid for the server build that wrote it, because a new build may compute a different answer for the same query:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:110
/// The entries survive restarts and can be shared between servers, but a result is only valid for the server build which computed
/// it: a newer or older build may compute a different result for the same query (fixes of functions or of the planner, changed
/// defaults), even when neither the TCP protocol revision nor the entry format changes. So every entry records the build which wrote
/// it, and an entry of another build is treated like an entry of an incompatible format: it is never served, and it is overwritten
/// on the next write.

The checksum covers the freshness fields. Otherwise a corrupted expires_at would silently change how long the entry lives.

The header is patched in place after the payload is serialised, and the checksum is computed over the header and the body:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:514
 unalignedStoreLittleEndian<UInt64>(data.data() + TOTAL_SIZE_OFFSET_IN_FIXED_HEADER, data.size());

 SipHash hash;
 hash.update(data.data(), CHECKSUM_OFFSET_IN_FIXED_HEADER);
 hash.update(data.data() + FIXED_HEADER_SIZE, data.size() - FIXED_HEADER_SIZE);
 UInt128 checksum = hash.get128();
 transformEndianness<std::endian::little>(checksum);
 memcpy(data.data() + CHECKSUM_OFFSET_IN_FIXED_HEADER, &checksum, sizeof(checksum));

Probing and writing

Writing is best-effort and never throws. Before writing, the existing entry is probed to see whether it is fresh, being written concurrently, or garbage:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:365
QueryResultCacheOnDisk::ProbeResult QueryResultCacheOnDisk::probeExistingEntry(const FileCacheKey & cache_key) const
{
 const auto & user_id = FileCache::getCommonOrigin().user_id;

 auto holder = file_cache->getDownloadedContiguousOrEmpty(cache_key, 0, FIXED_HEADER_SIZE, user_id);
 for (size_t attempt = 0; holder->empty(); ++attempt)
 {
 /// No readable header: either there is no entry, or another query is writing one (its segments are held and not
 /// downloaded yet), or only the tail of a partially evicted entry is left. The tail must be removed, otherwise no
 /// entry could be created for the key until it is evicted too.
 const auto segments = file_cache->tryGetFileSegmentInfos(cache_key, user_id);
 if (segments.empty())
 return ProbeResult::None;
 for (const auto & segment : segments)
 {
 if (segment.state != FileSegment::State::DOWNLOADED || segment.references > 1)
 return ProbeResult::InProgress;
 }

The write itself reserves the space for all the segments of the entry before writing any of them, so an entry is either complete or absent:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:529
 /// Create all file segments of the entry at once. This fails gracefully if another query created an entry for the same key in
 /// the meantime (then we simply skip the insert), and guarantees that we exclusively own all segments of the key while writing.
 /// A reader sees the incomplete entry as a miss because not all of its bytes are downloaded yet.
 auto holder = file_cache->trySet(cache_key, 0, data.size(), CreateFileSegmentSettings(FileSegmentKind::Regular), origin);
 if (!holder)
 {
 LOG_TRACE(logger, "Skipped insert into the on-disk query result cache because of a concurrent insert for query {}",
 doubleQuoteString(key.query_string));
 return;
 }

The reader side is symmetric: it downloads the whole entry, verifies the checksum, and only then decompresses it.

Using it

The cache is off by default; you turn it on per query or per session:

 xml
 
 
<!-- Server configuration: a 1 GiB in-memory query result cache. -->
<query_cache>
 <max_size_in_bytes>1073741824</max_size_in_bytes>
 <max_entries>1024</max_entries>
 <max_entry_size_in_bytes>1048576</max_entry_size_in_bytes>
 <max_entry_size_in_rows>30000000</max_entry_size_in_rows>
</query_cache>

 sql
 
 
SELECT count() FROM hits WHERE UserID = 42 SETTINGS use_query_cache = true;
-- the second execution is served from the cache

The per-user quota is a setting, not a config element, so it can differ per profile: query_cache_max_size_in_bytes and query_cache_max_entries.

The observable state lives in system tables:

 sql
 
 
SELECT query_id, length, hits, QueryCacheHits, QueryCacheMisses FROM system.query_log LIMIT 10;
SELECT query, result_size, tag, stale, shared, compressed, expires_at FROM system.query_result_cache LIMIT 10;

SYSTEM CLEAR QUERY CACHE clears everything, SYSTEM CLEAR QUERY CACHE TAG '<tag>' clears one tag’s worth.

Trade-offs

What it buys: a dashboard that runs the same heavy aggregate every 10 seconds stops paying for the aggregate, and the server’s CPU goes to the queries that are actually different.

What it costs:

Staleness is unbounded up to the TTL. The cache is not invalidated on INSERT. A TTL of 1 hour means an hour-old answer. This is the documented contract, not a bug, but it is the reason the cache is opt-in.
The key is a hash, so it inherits hash-collision risk. Two different queries with the same 128-bit AST hash would share an entry. In practice this does not happen; there is no secondary check of the query text.
Memory is the scarce resource. The in-memory cache holds result data, not plans, and the results are held compressed. A server with a 1 GiB cache can easily be asked for 10 GiB of results; the size limits (max_entry_size_in_bytes, max_entry_size_in_rows) exist to keep one huge result from evicting everything else.
Compression is a trade, not a win. The default query_cache_compress_entries makes reads slower and saves RAM; on a server that is memory-bound it is right, on one that is CPU-bound it is wrong.
The on-disk backend has its own failure modes: partially evicted entries, entries written by a different build, a filesystem cache that is not yet initialised. All of them are handled by treating the entry as a miss, and the probe-then-write dance means two concurrent writers cannot corrupt each other — one of them simply loses.

The final irony is that the cache’s hardest problem — the key — is solved by ClickHouse’s parser rather than by the cache: hashing the AST instead of the text means select and SELECT share an entry, and the planner’s internal alias noise does not leak into the key.

How ClickHouse Aggregates In Order and Stops Reading Early

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

SELECT count() FROM hits GROUP BY UserID is, conceptually, a hash table build: read every row, hash every key, update a counter. The hash table’s size is the number of distinct keys, which for a column with a million distinct values is a million entries — and for a column with a billion distinct values is a billion entries, and the query dies.

But if the table is sorted by UserID, that is a lot of wasted work. The rows arrive grouped: all the rows with UserID = 1 are contiguous, then all the rows with UserID = 2, and so on. You do not need a hash table at all. You need to count the run, emit it, count the next run, emit it.

That is the optimize_aggregation_in_order optimisation. It turns an unbounded-memory operation into a streaming one whose memory footprint is the size of a single group. And, in recent versions, it also enables an even more aggressive trick: pushing the LIMIT down into the aggregation so that the read stops early.

This is a tour of that machinery — the plan-level decision, the run-by-run aggregation loop, and the limit pushdown with its subtle “prefix boundary” rule.

When the optimisation applies

The decision is made in AggregatingStep::transformPipeline, and it is based on a single field: sort_description_for_merging. If that is non-empty, the input is already sorted on the group-by key and the in-order path is taken.

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:505
 /// optimize_aggregation_in_order
 if (!sort_description_for_merging.empty())
 {
 /// two-level aggregation is not supported anyway for in order aggregation.
 allow_to_use_two_level_group_by = false;

 /// It is incorrect for in order aggregation.
 params.stats_collecting_params.disable();

 /// Aggregation in order rebuilds the aggregation-method state for every run of equal
 /// order-key values, so the whole-block `prealloc_serialized` method would make it
 /// quadratic. Fall back to the plain `serialized` method (see `Params::aggregation_in_order`).
 params.aggregation_in_order = true;
 }

Three consequences fall out of this one branch, and each is a real limitation:

Two-level aggregation is disabled. The classic ClickHouse optimisation for GROUP BY with many distinct keys is two-level aggregation: hash the key, bucket by hash, aggregate each bucket in a separate hash table, and merge the buckets in parallel. That is exactly what you do not want when the data is already grouped — the whole point is to avoid a hash table entirely.

Statistics collection is disabled. The uniqCombined / uniqExact family of aggregate functions maintains auxiliary state to compute statistics that are only meaningful for the whole dataset. In-order aggregation finalises each group as it goes, so those statistics would be wrong.

The prealloc_serialized aggregation method is downgraded. This one deserves an explanation, because it is a performance pathology that the comment records:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:849
 /// See `Params::aggregation_in_order` and `method_chosen_for_in_order`: the `prealloc_serialized`
 /// method serializes the whole block's keys on state construction, which is pathological for the
 /// per-run in-order path, where a fresh state is constructed for every run of equal order-key
 /// values. There it uses the plain `serialized` method, whose construction is O(1) and which
 /// serializes keys lazily. The whole-block paths (including `mergeBlocks`) keep `method_chosen`.
 method_chosen_for_in_order = method_chosen;
 if (params.aggregation_in_order)
 {
 if (method_chosen_for_in_order == AggregatedDataVariants::Type::prealloc_serialized)
 method_chosen_for_in_order = AggregatedDataVariants::Type::serialized;

prealloc_serialized is an optimisation for the hash table case: when you create a state for a key, you serialise the key into a contiguous buffer right away, so the key data is contiguous and the hash table lookup is cheap. In the in-order case a fresh state is created for every run of equal keys — potentially millions of times for a million groups — and serialising the whole key each time would be quadratic in the key width. The lazy serialized variant amortises it.

The pipeline shape

When the in-order path is taken and there is more than one stream, the pipeline looks like this:

 text
 
 
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket

as the comment in AggregatingStep says:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:678
 /** The pipeline is the following:
 *
 * --> AggregatingInOrder --> MergingAggregatedBucket
 * --> AggregatingInOrder --> FinishAggregatingInOrder --> ResizeProcessor --> MergingAggregatedBucket
 * --> AggregatingInOrder --> MergingAggregatedBucket
 */

Each stream runs its own AggregatingInOrderTransform, each with its own ManyAggregatedData variant, and the partial results are merged later. The transform is constructed per stream:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:692
 return std::make_shared<AggregatingInOrderTransform>(
 header,
 transform_params,
 sort_description_for_merging,
 group_by_sort_description,
 max_block_size,
 aggregation_in_order_max_block_bytes / new_merge_threads,
 many_data,
 counter++,
 limit_hint,
 limit_hint_prefix_columns,

The counter++ is the stream index — each transform gets its own slot in the shared ManyAggregatedData, so the later merge knows which partial state belongs to which stream.

The core loop: aggregating one run at a time

AggregatingInOrderTransform::consume() is the heart of it. The key observation is that the input is sorted, so the rows belonging to one group form a contiguous interval, and the interval can be found with a binary search.

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:183
 size_t interval_index = 0;

 /// Will split block into segments with the same key
 while (key_end != rows)
 {
 /// Cancellation is only checked between work() calls, but one consume() over a chunk with many
 /// keys can run for a long time; check per key interval so a cancelled query stops promptly.
 if (isCancelled())
 {
 LOG_TEST(log, "Cancelled between key intervals");
 return;
 }

and the search itself:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:208
 /// Find the first position of new (not current) key in current chunk
 auto indices = collections::range(key_begin, rows);
 auto it = std::upper_bound(indices.begin(), indices.end(), cur_block_size - 1,
 [&](size_t lhs_row, size_t rhs_row)
 {
 return less(res_key_columns, key_columns, lhs_row, rhs_row, group_by_description);
 });

 key_end = (it == indices.end() ? rows : *it);

 /// Add data to aggr. state if interval is not empty. Empty when haven't found current key in new block.
 if (key_begin != key_end)
 {
 if (params->params.only_merge)
 {
 if (group_by_key)
 params->aggregator.mergeOnBlockSmall(variants, key_begin, key_end, aggregate_columns_data, key_columns_raw);
 else
 params->aggregator.mergeOnIntervalWithoutKey(variants, key_begin, key_end, aggregate_columns_data, is_cancelled);
 }
 else
 {
 if (group_by_key)
 params->aggregator.executeOnBlockSmall(variants, key_begin, key_end, key_columns_raw, aggregate_function_instructions.data());
 else
 params->aggregator.executeOnIntervalWithoutKey(variants, key_begin, key_end, aggregate_function_instructions.data());
 }
 }

res_key_columns holds the current key (one row), key_columns holds the whole chunk’s key column. upper_bound with the less comparator finds the first row whose key is strictly greater than the current key. Everything between key_begin and key_end belongs to the current group.

less() is the generic column-wise comparison:

 cpp
 
 
// src/Core/SortCursor.h:969
template <typename TLeftColumns, typename TRightColumns>
bool less(const TLeftColumns & lhs, const TRightColumns & rhs, size_t i, size_t j, const SortDescriptionWithPositions & descr)
{
 for (const auto & elem : descr)
 {
 size_t ind = elem.column_number;
 int res = elem.base.direction * lhs[ind]->compareAt(i, j, *rhs[ind], elem.base.nulls_direction);
 if (res < 0)
 return true;
 if (res > 0)
 return false;
 }

 return false;
}

So the loop is: binary-search the end of the current run, aggregate the run into the current state, finalise the state into the output columns, record the next key, repeat.

Two modes: with a key and without

group_by_key selects between two quite different implementations. The header comment on the aggregator methods explains the split:

 cpp
 
 
// src/Interpreters/Aggregator.h:553
 /// Used for optimize_aggregation_in_order:
 /// - No two-level aggregation
 /// - No external aggregation
 /// - No without_key support (it is implemented using executeOnIntervalWithoutKey())
 void executeOnBlockSmall(
 AggregatedDataVariants & result,
 size_t row_begin,
 size_t row_end,
 ColumnRawPtrs & key_columns,
 AggregateFunctionInstruction * aggregate_instructions) const;

 void executeOnIntervalWithoutKey(
 AggregatedDataVariants & data_variants,
 size_t row_begin,
 size_t row_end,
 AggregateFunctionInstruction * aggregate_instructions) const;

group_by_key is true when the GROUP BY key is a prefix of the sort description — i.e. the input is sorted by exactly the grouping key. In that case there is only ever one group at a time, so no hash table is needed at all: the aggregate state is a single object that is created, filled, and finalised per run.

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:147
 /// If we don't have a block we create it and fill with first key
 if (!cur_block_size)
 {
 res_key_columns.resize(params->params.keys_size);
 for (size_t i = 0; i < params->params.keys_size; ++i)
 res_key_columns[i] = res_header.safeGetByPosition(i).type->createColumn();

 if (group_by_key)
 params->aggregator.fillKeyColumnsWithSingleKey(key_columns, key_begin, res_key_columns);
 else
 params->aggregator.createStatesAndFillKeyColumnsWithSingleKey(variants, key_columns, key_begin, res_key_columns);

fillKeyColumnsWithSingleKey — the state is not created; the key is just copied into the output column. createStatesAndFillKeyColumnsWithSingleKey — a state is created in the hash table, because the grouping key is a prefix of the sort key but not the whole of it, so several distinct groups may share the same sort-key prefix and must be kept apart.

When the group is finished, the state is written out and the key advances:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:239
 /// We finalize last key aggregation state if a new key found.
 if (key_end != rows)
 {
 if (!group_by_key)
 params->aggregator.addSingleKeyToAggregateColumns(variants, res_aggregate_columns);

 /// Enough groups have been emitted and the new key starts a new value of the
 /// `ORDER BY` prefix, so the rest of the input cannot be needed (see `limit_prefix_columns`).
 /// With `group_by_key`, `cur_block_size` counts runs of the sorted key columns rather than
 /// groups; the groups accumulated for the current block are the entries of the hash table.
 size_t cur_block_groups = group_by_key ? variants.size() : cur_block_size;
 if (limit_hint && cur_block_groups + res_rows >= limit_hint && isLimitPrefixBoundary(key_columns, key_end))
 limit_reached = true;

And the “the rest of the chunk is not needed” case:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:253
 /// If max_block_size or limit_hint is reached we have to stop consuming and generate the block. Save the extra rows into new chunk.
 if (cur_block_size >= max_block_size || cur_block_bytes + current_memory_usage >= max_block_bytes || limit_reached)
 {
 if (group_by_key)
 group_by_chunk
 = params->aggregator.prepareChunkAndFillSingleLevel</* return_single_block */ true>(variants, /* final= */ false).chunk;
 cur_block_bytes += current_memory_usage;

 /// When limit is reached, don't save leftover rows — we're done.
 /// Set block_end_reached + need_generate to trigger generate(),
 /// which will produce the output and then set is_consume_finished.
 if (limit_reached)
 {
 block_end_reached = true;
 need_generate = true;
 variants.invalidate();
 }
 else
 {
 finalizeCurrentChunk(std::move(chunk), key_end);
 }

Note the difference between the two branches. Normally the leftover rows of the chunk (those after key_end) are saved — finalizeCurrentChunk cuts them off and keeps them for the next consume():

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:298
void AggregatingInOrderTransform::finalizeCurrentChunk(Chunk chunk, size_t key_end)
{
 size_t rows = chunk.getNumRows();
 Columns source_columns = chunk.detachColumns();

 for (auto & source_column : source_columns)
 source_column = source_column->cut(key_end, rows - key_end);

 current_chunk = Chunk(source_columns, rows - key_end);
 src_rows -= current_chunk.getNumRows();

 block_end_reached = true;
 need_generate = true;
 variants.invalidate();
}

But when the limit is reached, the leftovers are dropped: the query is over, and reading the rest of the chunk would be wasted work.

Pushing `LIMIT` into the aggregation

The limit pushdown is a plan-level optimisation, in src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp, and it is gated on the optimize_aggregation_in_order_limit setting, which defaults to true:

text

// src/Core/Settings.cpp:5163
 DECLARE(Bool, optimize_aggregation_in_order_limit, true, R"(
When enabled and aggregation in order is active, pushes LIMIT into the aggregation step to enable early termination after producing enough groups. This reduces the amount of data read when ORDER BY matches the GROUP BY key prefix. May reduce the value reported by `rows_before_limit_at_least`; use `exact_rows_before_limit` if exact counts are needed.
)", 0) \

The optimisation looks for the shape LIMIT → SORT → AGGREGATE:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:77
void optimizeLimitForAggregationInOrder(QueryPlan::Node & root)
{
 Stack stack;
 stack.push_back({.node = &root});

 while (!stack.empty())
 {
 auto & frame = stack.back();

 if (frame.next_child < frame.node->children.size())
 {
 stack.push_back({.node = frame.node->children[frame.next_child++]});
 continue;
 }

 auto * node = frame.node;
 stack.pop_back();

 auto * limit_step = typeid_cast<LimitStep *>(node->step.get());
 if (!limit_step || limit_step->withTies() || limit_step->alwaysReadTillEnd())
 continue;

 size_t limit = limit_step->getLimitForSorting();
 if (!limit)
 continue;

WITH TIES and always_read_till_end are excluded immediately — both change the meaning of the limit in ways the pushdown cannot honour.

Then it walks down through “transparent” steps (steps that neither change the row count nor the sort order) looking for the aggregation:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:117
 /// Find AggregatingStep below SortingStep, skipping transparent steps.
 /// Stop at TotalsHavingStep — HAVING may filter groups.
 while (current)
 {
 auto * aggregating_step = typeid_cast<AggregatingStep *>(current->step.get());
 if (aggregating_step)
 {
 /// overflow_row is set by WITH TOTALS.
 if (!aggregating_step->inOrder()
 || !aggregating_step->getFinal()
 || aggregating_step->getParams().overflow_row
 || aggregating_step->isGroupingSets())
 break;

and then the crucial check:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:131
 /// The sort description may be a strict prefix of the group-by sort description.
 /// Each in-order stream then stops only at a boundary of that prefix (see
 /// `AggregatingInOrderTransform`), so the groups it drops sort strictly after
 /// at least `limit` complete groups and the final sort can never pick a group
 /// with a partial aggregate value (issue #116849).
 const auto & sort_desc = sorting_step->getSortDescription();
 const auto & agg_sort_desc = aggregating_step->getGroupBySortDescription();
 if (sort_desc.empty() || !agg_sort_desc.hasPrefix(sort_desc))
 break;

 /// Use the smallest limit if multiple LimitSteps point to the same AggregatingStep.
 size_t current_hint = aggregating_step->getLimitHint();
 if (!current_hint || limit < current_hint)
 aggregating_step->setLimitHint(limit, sort_desc.size());
 break;

agg_sort_desc.hasPrefix(sort_desc) — the ORDER BY columns must be a prefix of the GROUP BY sort description. This is the condition that makes the pushdown safe, and the reasoning in the comment is worth unpacking because it is subtle.

The prefix boundary rule

The ORDER BY of the query may be a prefix of the GROUP BY key. Say:

 sql
 
 
SELECT a, b, count()
FROM t
GROUP BY a, b
ORDER BY a
LIMIT 10

Here the table is sorted by (a, b). The GROUP BY produces groups in (a, b) order. The ORDER BY a then sorts them by a — and since the groups already come out in (a, b) order, sorting by a is a stable reordering: groups with the same a keep their relative (a, b) order.

Now: the aggregation stream can stop early once it has produced 10 groups. But it must not stop in the middle of a run of groups that share the same a — because another stream might produce a group with the same a but a smaller b, and the final sort (by a only) could then pick that group over the partial one.

The rule is: stop only at a boundary of the ORDER BY prefix. That is what limit_prefix_columns counts and what isLimitPrefixBoundary() checks:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:85
bool AggregatingInOrderTransform::isLimitPrefixBoundary(const Columns & key_columns, size_t row) const
{
 for (size_t i = 0; i < limit_prefix_columns; ++i)
 {
 const auto & elem = group_by_description[i];
 size_t ind = elem.column_number;
 if (res_key_columns[ind]->compareAt(cur_block_size - 1, row, *key_columns[ind], elem.base.nulls_direction) != 0)
 return true;
 }
 return false;
}

It compares the last emitted group’s value of the prefix column with the next group’s value. If they differ, the run of equal prefix values has ended, and it is safe to stop: every group with that prefix value has been emitted.

Note the comment on the member:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.h:97
 /// The stream may stop early once it has emitted `limit_hint` groups, but only at a
 /// boundary of the first `limit_prefix_columns` columns of `group_by_description`, which
 /// are the columns the query is ordered by. Otherwise a group tied on those columns with
 /// the emitted ones could be cut off here while another stream emits its partial state,
 /// and the final sort could pick that partial group.
 size_t limit_hint = 0;
 size_t limit_prefix_columns = 0;
 bool limit_reached = false;

And the prefix count is clamped to the group-by description:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:80
 /// The `ORDER BY` may extend past the columns the input is sorted by; a boundary of the
 /// sorted columns alone is then a boundary of the `ORDER BY` prefix as well.
 limit_prefix_columns = std::min(limit_prefix_columns_, group_by_description.size());

This is the bug that issue #116849 was about, and the comment in the optimisation pass records it.

What it buys, concretely

 sql
 
 
SELECT user_id, count()
FROM hits
GROUP BY user_id
ORDER BY user_id
LIMIT 10

on a table sorted by user_id. Without the pushdown, the whole table is read, aggregated, sorted, and then 10 rows are kept. With it, the aggregation stops after the first 10 groups, and — crucially — the source stops too, because the transform closes its input:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:354
 /// When limit is reached, we set is_consume_finished in generate()
 /// and don't save leftover rows. Finish immediately after pushing.
 if (is_consume_finished)
 {
 output.finish();
 input.close();
 LOG_DEBUG(log, "Aggregated. {} to {} rows (from {})", src_rows, res_rows, formatReadableSizeWithBinarySuffix(src_bytes));
 return Status::Finished;
 }

input.close() propagates the stop all the way up to the storage layer. On a MergeTree, that means the remaining granules are never read.

The cost is stated in the setting’s description: rows_before_limit_at_least becomes an underestimate, because the rows that were never read are not counted. exact_rows_before_limit restores the exact count at the price of reading everything.

Trade-offs

What this buys you.

The in-order aggregation replaces a hash table whose size is the number of distinct keys with a state whose size is one group. For a table sorted by the grouping key, that is the difference between a query that fits in memory and one that does not.

It also composes with the parallel execution model: each stream aggregates its own runs independently, and the partial states are merged at the end by FinishAggregatingInOrderTransform and MergingAggregatedBucketTransform. The parallelism is preserved; only the memory behaviour changes.

The limit pushdown extends the idea from “less memory” to “less work”: the read itself stops. On a large MergeTree with a selective LIMIT, this is the difference between reading 10 granules and reading a thousand.

What it costs.

The optimisation is conditional on the data layout. If the table is not sorted by the grouping key, the plan does not use it — and if the table is sorted but the GROUP BY key is not a prefix of the sort key, it does not apply either. ClickHouse has to prove the sortedness, and it does so from the table’s primary key and the optimize_read_in_order machinery, so the whole optimisation is contingent on the optimiser’s knowledge of the physical layout.

The in-order path gives up several general-purpose optimisations: two-level aggregation, external aggregation, and statistics collection. For a query with a small number of distinct keys on a large table, the ordinary hash-table path with two-level aggregation might actually be faster, because it parallelises the hash table itself rather than relying on the input being pre-grouped. The optimize_aggregation_in_order setting defaults to false for exactly this reason.

The prealloc_serialized → serialized downgrade is a real regression in the hash-table case that the in-order case forced, paid for by every query that takes the in-order path.

The limit pushdown has a correctness-adjacent cost that the code is careful about but that is easy to get wrong: the prefix-boundary rule. Stop at the wrong place and the final sort may pick a group whose aggregate state was truncated — a wrong answer, not a slow one. That is why the rule is conservative (stop only at a prefix boundary, never in the middle of a run of equal prefix values) and why the plan-level check requires the ORDER BY to be a prefix of the group-by description.

And the pushdown changes the meaning of a system counter: rows_before_limit_at_least becomes “rows that were actually read”, not “rows that would have been read without the limit”. Any monitoring built on that counter has to know about the optimisation.

How PostgreSQL Stores a Row

Jay Hubbert — Tue, 20 Oct 2026 09:00:00 +0000

Every database gives you a row and every database lies a little about what that row is. MySQL gives you a record copied into a row buffer. SQLite gives you a record in a b-tree leaf. PostgreSQL gives you something stranger and more interesting: a 23-byte header followed by a payload, living at some byte offset on an 8 kB page, reached through a 4-byte indirection that lives in a completely different part of that same page.

That indirection is the reason PostgreSQL can move tuples around during VACUUM without invalidating a single index entry. The header is the reason an UPDATE can leave a trail of old versions behind. The slotted page is the reason a table’s rows are not laid out contiguously — and why reclaiming the space left behind by dead rows is a whole subsystem.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

A row is not a record

Let’s start with the container. PostgreSQL stores a table as an array of fixed-size pages — 8 kB by default, configurable at initdb time. The heap access method never stores anything that isn’t on a page, and it never stores a row that spans two pages. A row that doesn’t fit in 8 kB gets pushed out of line entirely (more on that later).

Each page is a slotted page. From src/include/storage/bufpage.h:

 c
 
 
/*
 * a postgres disk page is always a slotted page of the form:
 *
 * +----------------+---------------------------------+
 * | PageHeaderData | linp1 linp2 linp3 ... |
 * +-----------+----+---------------------------------+
 * | ... linpN |									 |
 * +-----------+--------------------------------------+
 * |		 ^ pd_lower							 |
 * |												 |
 * |			 v pd_upper							 |
 * +-------------+------------------------------------+
 * |			 | tupleN ... |
 * +-------------+------------------+-----------------+
 * |	 ... tuple3 tuple2 tuple1 | "special space" |
 * +--------------------------------+-----------------+
 *									^ pd_special
 */

Five parts, and they grow in opposite directions:

Region	Size on an 8 kB page	Grows
`PageHeaderData`	24 bytes	fixed
Line pointer array	4 bytes per row	upwards from `pd_lower`
Free space	whatever is left	—
Tuples	variable	downwards from `pd_upper`
Special space	0 for heap pages	fixed

The header is small enough to write out in full:

 c
 
 
typedef struct PageHeaderData
{
	PageXLogRecPtr pd_lsn;		/* LSN: next byte after last byte of xlog
								 * record for last change to this page */
	uint16		pd_checksum;	/* checksum */
	uint16		pd_flags;		/* flag bits, see below */
	LocationIndex pd_lower;		/* offset to start of free space */
	LocationIndex pd_upper;		/* offset to end of free space */
	LocationIndex pd_special;	/* offset to start of special space */
	uint16		pd_pagesize_version;
	TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */
	ItemIdData	pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */
} PageHeaderData;

pd_lower and pd_upper are the entire free-space bookkeeping. pd_lower marks the end of the line pointer array, pd_upper marks the start of the tuple area; the gap between them is what’s available. There is no free-list, no extent map, no per-page allocation structure. Free space is just an interval, and it is used from both ends.

A heap page has pd_special == BLCKSZ, because the “special space” is where an index access method stashes its own per-page state. A btree leaf page, for instance, ends with a BTPageOpaqueData holding the sibling links and the tree level. Heap pages have nothing to say for themselves.

Line pointers: the indirection that makes everything possible

The line pointer array is where the “row” abstraction starts to leak. Each entry is four bytes (src/include/storage/itemid.h):

 c
 
 
typedef struct ItemIdData
{
	unsigned	lp_off:15,		/* offset to tuple (from start of page) */
				lp_flags:2,		/* state of line pointer, see below */
				lp_len:15;		/* byte length of tuple */
} ItemIdData;

Fifteen bits of byte offset, fifteen bits of length, two bits of state. lp_flags has four values:

 c
 
 
#define LP_UNUSED		0		/* unused (should always have lp_len=0) */
#define LP_NORMAL		1		/* used (should always have lp_len>0) */
#define LP_REDIRECT		2		/* HOT redirect (should have lp_len=0) */
#define LP_DEAD			3		/* dead, may or may not have storage */

Note the convention, stated in the header comment: by convention lp_len == 0 in every line pointer that does not have storage, independently of its lp_flags state. A line pointer can be “in use” and still point at nothing. That is the whole trick.

So a row is addressed by a TID — a tuple identifier — which is a (block number, offset number) pair. The block number says which page; the offset number says which line pointer. The line pointer then says which byte range the tuple occupies. Three levels of addressing, and only the last one is a physical location.

This indirection is not incidental. It’s what lets VACUUM slide tuples around within a page, or move them between pages, and leave every index entry untouched — the index stores a TID, and the TID still resolves to the right line pointer, and the line pointer has been updated to point at the tuple’s new home.

Two details worth knowing:

The number of line pointers is derived, not stored. It is (pd_lower - SizeOfPageHeaderData) / sizeof(ItemIdData). There’s no counter; the array simply grows.
A line pointer’s lp_off is overloaded. For LP_NORMAL it’s a byte offset. For LP_REDIRECT it’s the offset number of another line pointer on the same page. Same field, completely different meaning.

The tuple header: 23 bytes

Now the header at the front of every heap tuple (src/include/access/htup_details.h):

 c
 
 
struct HeapTupleHeaderData
{
	union
	{
		HeapTupleFields t_heap;
		DatumTupleFields t_datum;
	}			t_choice;

	ItemPointerData t_ctid;		/* current TID of this or newer tuple (or a
								 * speculative insertion token) */

	/* Fields below here must match MinimalTupleData! */

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK2 2
	uint16		t_infomask2;	/* number of attributes + various flags */

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK 3
	uint16		t_infomask;		/* various flag bits, see below */

#define FIELDNO_HEAPTUPLEHEADERDATA_HOFF 4
	uint8		t_hoff;			/* sizeof header incl. bitmap, padding */

	/* ^ - 23 bytes - ^ */

#define FIELDNO_HEAPTUPLEHEADERDATA_BITS 5
	uint8		t_bits[FLEXIBLE_ARRAY_MEMBER];	/* bitmap of NULLs */

	/* MORE DATA FOLLOWS AT END OF STRUCT */
};

Twenty-three bytes. Here is what each byte is doing, on a 64-bit machine with MAXIMUM_ALIGNOF == 8:

Offset	Bytes	Field	Meaning
0	4	`t_xmin`	XID of the inserting transaction
4	4	`t_xmax`	XID of the deleting or locking transaction
8	4	`t_cid`	insert and/or delete command ID (overlays `t_xvac`)
12	4	`t_ctid.ip_blkid`	block number of this or a newer version
16	2	`t_ctid.ip_posid`	offset number of this or a newer version
18	2	`t_infomask2`	attribute count + HOT/heap-only flags
20	2	`t_infomask`	content and visibility flags
22	1	`t_hoff`	offset of the user data

ItemPointerData is six bytes, not eight, and it gets there by fiat:

 c
 
 
/* If compiler understands packed and aligned pragmas, use those */
#if defined(pg_attribute_packed) && defined(pg_attribute_aligned)
			pg_attribute_packed()
			pg_attribute_aligned(2)
#endif
ItemPointerData;

The comment above it explains why: “because there is an item pointer in each tuple header and index tuple header on disk, it’s very important not to waste space with structure padding bytes.” Two bytes saved here, on every row, forever.

Five virtual fields in three physical ones

The header stores five logical fields — Xmin, Cmin, Xmax, Cmax, Xvac — in three physical slots. From the header comment:

We store five “virtual” fields Xmin, Cmin, Xmax, Cmax, and Xvac in three physical fields. Xmin and Xmax are always really stored, but Cmin, Cmax and Xvac share a field. This works because we know that Cmin and Cmax are only interesting for the lifetime of the inserting and deleting transaction respectively.

Cmin and Cmax are command IDs — counters that order the statements within a single transaction. If the same transaction inserts and then deletes a row, both values have to survive. PostgreSQL solves this with a “combo CID”: a synthetic value that maps back to both, using local state in the originating backend. The HEAP_COMBOCID bit in t_infomask says “this field is a combo”.

Before 8.3, HeapTupleHeaderData had separate cmin and cmax fields. Removing one 4-byte field from every row in every table was worth the combo-CID machinery.

t_infomask

Sixteen bits, split into two groups. The low four describe the physical contents of the tuple:

 c
 
 
#define HEAP_HASNULL			0x0001	/* has null attribute(s) */
#define HEAP_HASVARWIDTH		0x0002	/* has variable-width attribute(s) */
#define HEAP_HASEXTERNAL		0x0004	/* has external stored attribute(s) */
#define HEAP_HASOID_OLD			0x0008	/* has an object-id field */

The upper twelve are about visibility — who can see this row, and what happened to it:

 c
 
 
#define HEAP_XMAX_KEYSHR_LOCK	0x0010	/* xmax is a key-shared locker */
#define HEAP_COMBOCID			0x0020	/* t_cid is a combo CID */
#define HEAP_XMAX_EXCL_LOCK		0x0040	/* xmax is exclusive locker */
#define HEAP_XMAX_LOCK_ONLY		0x0080	/* xmax, if valid, is only a locker */

#define HEAP_XMIN_COMMITTED		0x0100	/* t_xmin committed */
#define HEAP_XMIN_INVALID		0x0200	/* t_xmin invalid/aborted */
#define HEAP_XMIN_FROZEN		(HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID)
#define HEAP_XMAX_COMMITTED		0x0400	/* t_xmax committed */
#define HEAP_XMAX_INVALID		0x0800	/* t_xmax invalid/aborted */
#define HEAP_XMAX_IS_MULTI		0x1000	/* t_xmax is a MultiXactId */
#define HEAP_UPDATED			0x2000	/* this is UPDATEd version of row */
#define HEAP_MOVED_OFF			0x4000	/* moved to another place by pre-9.0
										 * VACUUM FULL; kept for binary
										 * upgrade support */
#define HEAP_MOVED_IN			0x8000	/* moved from another place by pre-9.0
										 * VACUUM FULL; kept for binary
										 * upgrade support */

HEAP_XMIN_COMMITTED and friends are hint bits. They cache a fact that could otherwise require looking up the transaction’s commit record in CLOG. Once a reader has established that the inserting transaction committed, it stamps the tuple and everyone afterwards gets the answer for free. The rules are conservative — a hint is only set when the answer is known-good — and the code is careful never to set a hint that would be wrong if the buffer is later written to disk:

 c
 
 
	if (TransactionIdIsValid(xid))
	{
		if (BufferIsPermanent(buffer))
		{
			/* NB: xid must be known committed here! */
			XLogRecPtr	commitLSN = TransactionIdGetCommitLSN(xid);

			if (XLogNeedsFlush(commitLSN) &&
				BufferGetLSNAtomic(buffer) < commitLSN)
			{
				/* not flushed and no LSN interlock, so don't set hint */
				return;
			}
		}
	}

Notice the subtlety: if the commit record isn’t on disk yet, the hint isn’t set, because the page might reach disk before the commit record does, and then a crash would leave a tuple claiming to be committed when it isn’t.

t_infomask2

 c
 
 
#define HEAP_NATTS_MASK			0x07FF	/* 11 bits for number of attributes */
/* bits 0x1800 are available */
#define HEAP_KEYS_UPDATED		0x2000	/* tuple was updated and key cols
										 * modified, or tuple deleted */
#define HEAP_HOT_UPDATED		0x4000	/* tuple was HOT-updated */
#define HEAP_ONLY_TUPLE			0x8000	/* this is heap-only tuple */

The low eleven bits hold the attribute count, which is why t_infomask2 is a uint16 and the number of columns is limited. The other three bits carry the HOT state machine, which we will come back to.

The null bitmap and the 23/24 trick

HEAP_HASNULL in t_infomask means a null bitmap is present, immediately after the fixed header, one bit per column:

 c
 
 
static inline int
BITMAPLEN(int NATTS)
{
	return (NATTS + 7) / 8;
}

The bitmap is inverted — a 1 bit means not null:

 c
 
 
/*
 * Check a tuple's null bitmap to determine whether the attribute is null.
 * Note that a 0 in the null bitmap indicates a null, while 1 indicates
 * non-null.
 */
static inline bool
att_isnull(int ATT, const uint8 *BITS)
{
	return !(BITS[ATT >> 3] & (1 << (ATT & 0x07)));
}

Attribute 0 is the low bit of byte 0. (This is the sort of detail that makes hand-parsing a heap page a mildly masochistic exercise. The pageinspect extension exists for a reason.)

Now, t_hoff is the offset from the start of the tuple to the first byte of user data, and it must be a multiple of MAXALIGN — 8 bytes on a 64-bit machine. The reason is that the data area must be 8-byte aligned so that fetchatt() can do a direct aligned load of an int8 or double column.

The calculation, in heap_form_tuple() (src/backend/access/common/heaptuple.c):

 c
 
	len = offsetof(HeapTupleHeaderData, t_bits);

	if (hasnull)
		len += BITMAPLEN(numberOfAttributes);

	hoff = len = MAXALIGN(len); /* align user data safely */

offsetof(HeapTupleHeaderData, t_bits) is 23. So:

No NULLs: len = 23 → MAXALIGN(23) = 24 → t_hoff = 24. Byte 23 is pure padding.
1–8 columns, some NULL: len = 23 + 1 = 24 → MAXALIGN(24) = 24 → t_hoff = 24. The bitmap lands exactly in the padding byte.
9–16 columns, some NULL: len = 23 + 2 = 25 → MAXALIGN(25) = 32 → t_hoff = 32. Seven bytes of padding.

So the header is a 23-byte struct that behaves like a 24-byte header. The one-byte discrepancy is deliberate: a 24-byte header is 8-byte aligned, so the data area inherits that alignment for free, and any 4-byte-aligned field inside it is also correctly aligned. Had the header been 24 bytes and the bitmap 1 byte, the data would start at 25 — unaligned.

There is a hard limit hiding here. t_hoff is a uint8, so it can hold at most 255. The comment in htup_details.h spells out the constraint:

The key limit on this value is that the size of the fixed overhead for a tuple, plus the size of the null-values bitmap (at 1 bit per column), plus MAXALIGN alignment, must fit into t_hoff which is uint8. On most machines the upper limit without making t_hoff wider would be a little over 1700.

Which is why MaxTupleAttributeNumber is 1664 and MaxHeapAttributeNumber is 1600. PostgreSQL’s real column limit is a storage-layout limit, not an arbitrary one.

The data area

After t_hoff come the column values, packed in attribute order. The rules come out of fill_val() in heaptuple.c:

NULL columns cost zero bytes. fill_val sets the bitmap bit and returns before writing any data. The space is genuinely not reserved.
By-value columns (int4, bool, oid, …) are aligned to their attalign and stored inline.
Fixed-length by-reference columns (numeric, uuid, …) are aligned and copied inline.
Short varlena (a text of 126 bytes or less) gets no alignment padding at all — it can sit at any byte offset.
Full 4-byte-header varlena is aligned, because the 4-byte length word must be naturally aligned.
cstring (attlen == -2) never needs alignment.

The interesting one is #4. Here is the code:

 c
 
 
		else if (VARATT_IS_SHORT(val))
		{
			/* no alignment for short varlenas */
			data_length = VARSIZE_SHORT(val);
			memcpy(data, val, data_length);
		}
		else if (att->attispackable && VARATT_CAN_MAKE_SHORT(val))
		{
			/* convert to short varlena -- no alignment */
			data_length = VARATT_CONVERTED_SHORT_SIZE(val);
			SET_VARSIZE_SHORT(data, data_length);
			memcpy(data + 1, VARDATA(val), data_length - 1);
		}
		else
		{
			/* full 4-byte header varlena */
			data = (char *) att_nominal_alignby(data, att->attalignby);
			data_length = VARSIZE(val);
			memcpy(data, val, data_length);
		}

A text value whose payload is 126 bytes or less is stored with a one-byte header and needs no padding. A payload of 127 bytes cannot be short — the limit is VARATT_SHORT_MAX (0x7F) — so it falls back to the 4-byte-header format, which is aligned, and the value suddenly costs several bytes more than the one below it. It is a real, if obscure, discontinuity in the space cost of text columns.

Variable-length values: the four header formats

Every variable-length value shares the varlena concept, but there are four on-disk encodings, distinguished by the low bits of the first header byte (src/include/varatt.h):

 c
 
 
typedef struct
{
	struct						/* Normal varlena (4-byte length) */
	{
		uint32		va_header;
		char		va_data[FLEXIBLE_ARRAY_MEMBER];
	}			va_4byte;
	struct						/* Compressed-in-line format */
	{
		uint32		va_header;
		uint32		va_tcinfo;	/* Original data size (excludes header) and
								 * compression method; see va_extinfo */
		char		va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
	}			va_compressed;
} varattrib_4b;

typedef struct
{
	uint8		va_header;
	char		va_data[FLEXIBLE_ARRAY_MEMBER]; /* Data begins here */
} varattrib_1b;

typedef struct
{
	uint8		va_header;		/* Always 0x80 or 0x01 */
	uint8		va_tag;			/* Type of datum */
	char		va_data[FLEXIBLE_ARRAY_MEMBER]; /* Type-specific data */
} varattrib_1b_e;

4-byte header — the general case. The header stores the length in its upper 30 bits and flags in the low bits (compressed / external).
1-byte header — the “short” format, for payloads up to VARATT_SHORT_MAX (0x7F = 127) bytes. One bit of the header is reserved as a flag; the other seven hold the length.
1-byte header + tag — the external (TOAST pointer) format. Also a one-byte header, so TOAST pointers cost no alignment.

TOAST: when a row doesn’t fit

A tuple can be at most MaxHeapTupleSize:

#define MaxHeapTupleSize (BLCKSZ - MAXALIGN(SizeOfPageHeaderData + sizeof(ItemIdData)))

With BLCKSZ 8192, a 24-byte page header and a 4-byte line pointer, that’s 8192 − 32 = 8160 bytes.

But TOAST kicks in much earlier. From src/include/access/heaptoast.h:

 c
 
 
/*
 * Find the maximum size of a tuple if there are to be N tuples per page.
 */
#define MaximumBytesPerTuple(tuplesPerPage) \
	MAXALIGN_DOWN((BLCKSZ - \
				 MAXALIGN(SizeOfPageHeaderData + (tuplesPerPage) * sizeof(ItemIdData))) \
				 / (tuplesPerPage))

#define TOAST_TUPLES_PER_PAGE	4

#define TOAST_TUPLE_THRESHOLD	MaximumBytesPerTuple(TOAST_TUPLES_PER_PAGE)

#define TOAST_TUPLE_TARGET		TOAST_TUPLE_THRESHOLD

TOAST_TUPLE_THRESHOLD works out to 2032 bytes: the largest tuple such that four of them fit on a page. If a tuple exceeds that, the toaster takes over.

The toaster’s strategy, in order:

Compress every compressible column, in place. Compression is tried per-column, and a column is only kept compressed if it actually got smaller. A text column that compresses to 40% of its size stays compressed inline, with a 4-byte header that records both the compressed length and the original length.
Move EXTENDED and EXTERNAL columns out of line into the table’s TOAST table, replacing them in the row with an 18-byte pointer.
As a last resort, move PLAIN and MAIN columns out of line too, up to TOAST_TUPLE_TARGET_MAIN (8160 bytes — the largest tuple that fits on a page at all). The user said “keep this inline if at all possible”, so this only happens when nothing else works.

The TOAST pointer itself is the varattrib_1b_e format: a one-byte header, a tag byte, and a payload:

 c
 
 
typedef struct varatt_external_oid
{
	int32		va_rawsize;		/* Original data size (includes header) */
	uint32		va_extinfo;		/* External saved size (without header) and
								 * additional flag bits */
	Oid			va_valueid;		/* Unique ID of value within TOAST table */
	Oid			va_toastrelid;	/* RelID of TOAST table containing it */
} varatt_external_oid;

The key design decision: the pointer identifies the value by (toast table, value id), not by the physical location of the row that contains it. That is what makes TOAST pointers relocatable — a row can be moved by VACUUM FULL, CLUSTER, or ALTER TABLE ... SET TABLESPACE without rewriting its out-of-line data.

The TOAST table

Every table with toastable columns gets a companion table named pg_toast_<oid>, created in src/backend/catalog/toasting.c:

 c
 
 
	tupdesc = CreateTemplateTupleDesc(3);
	TupleDescInitEntry(tupdesc, (AttrNumber) 1,
					 "chunk_id",
					 toast_chunkid_typid,
					 -1, 0);
	TupleDescInitEntry(tupdesc, (AttrNumber) 2,
					 "chunk_seq",
					 INT4OID,
					 -1, 0);
	TupleDescInitEntry(tupdesc, (AttrNumber) 3,
					 "chunk_data",
					 BYTEAOID,
					 -1, 0);

Three columns: an identifier for the value, a chunk sequence number, and the chunk payload. Large values are split across multiple rows.

The chunk size is bounded by what has to fit on a page:

 c
 
 
#define TOAST_OID_MAX_CHUNK_SIZE	\
	(EXTERN_TUPLE_MAX_SIZE -							\
	 MAXALIGN(SizeofHeapTupleHeader) -					\
	 sizeof(Oid) -										\
	 sizeof(int32) -									\
	 VARHDRSZ)

EXTERN_TUPLE_MAX_SIZE is MaximumBytesPerTuple(4) = 2032, so the chunk payload is 2032 − 24 − 4 − 4 − 4 = 1996 bytes for an oid value id (and 1992 for the oid8 variant).

One last detail that is easy to miss:

 c
 
 
	/*
	 * Ensure that the toast table doesn't itself get toasted, or we'll be
	 * toast :-(. This is essential for chunk_data because type bytea is
	 * toastable; hit the other two just to be sure.
	 */
	TupleDescAttr(tupdesc, 0)->attstorage = TYPSTORAGE_PLAIN;
	TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN;
	TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN;

The TOAST table is explicitly not toastable. A row cannot contain a pointer to a value that lives in a table that itself contains pointers to values.

UPDATE: the version chain

Now that we know what a tuple looks like, we can see what an UPDATE does. PostgreSQL never modifies a row in place (with one narrow exception for system catalogs). It writes a new version and links the old one to it.

The link is t_ctid. From the header comment:

A word about t_ctid: whenever a new tuple is stored on disk, its t_ctid is initialized with its own TID (location). If the tuple is ever updated, its t_ctid is changed to point to the replacement version of the tuple.

So t_ctid is either a self-reference (this is the newest version) or a pointer to a successor. The header comment also warns about the obvious hazard:

Beware however that VACUUM might erase the pointed-to (newer) tuple before erasing the pointing (older) tuple. Hence, when following a t_ctid link, it is necessary to check to see if the referenced slot is empty or contains an unrelated tuple. Check that the referenced tuple has XMIN equal to the referencing tuple’s XMAX to verify that it is actually the descendant version.

That XMIN/XMAX check is the chain-integrity test, and it appears everywhere — in the index scan, in the pruner, and in the row-locking code. It’s how a reader knows it hasn’t followed a link into a recycled slot.

The actual update, in heap_update() (src/backend/access/heap/heapam.c):

 c
 
 
	newtup->t_data->t_infomask &= ~(HEAP_XACT_MASK);
	newtup->t_data->t_infomask2 &= ~(HEAP2_XACT_MASK);
	HeapTupleHeaderSetXmin(newtup->t_data, xid);
	HeapTupleHeaderSetCmin(newtup->t_data, cid);
	newtup->t_data->t_infomask |= HEAP_UPDATED | infomask_new_tuple;
	newtup->t_data->t_infomask2 |= infomask2_new_tuple;
	HeapTupleHeaderSetXmax(newtup->t_data, xmax_new_tuple);

The new version gets the updater’s XID as its xmin. Then, at the very end of the operation, the old version is finalized:

 c
 
 
	/* Clear obsolete visibility flags, possibly set by ourselves above... */
	oldtup.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
	oldtup.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
	/* ... and store info about transaction updating this tuple */
	Assert(TransactionIdIsValid(xmax_old_tuple));
	HeapTupleHeaderSetXmax(oldtup.t_data, xmax_old_tuple);
	oldtup.t_data->t_infomask |= infomask_old_tuple;
	oldtup.t_data->t_infomask2 |= infomask2_old_tuple;
	HeapTupleHeaderSetCmax(oldtup.t_data, cid, iscombo);

	/* record address of new tuple in t_ctid of old one */
	oldtup.t_data->t_ctid = heaptup->t_self;

One line — oldtup.t_data->t_ctid = heaptup->t_self — is the entire mechanism of MVCC at the storage layer. The old version records where its successor lives, and the old version’s xmax records who did the update.

Notice also HEAP_XMAX_BITS, which is masked out before the new value is written:

 c
 
 
#define HEAP_XMAX_BITS (HEAP_XMAX_COMMITTED | HEAP_XMAX_INVALID | \
						HEAP_XMAX_IS_MULTI | HEAP_LOCK_MASK | HEAP_XMAX_LOCK_ONLY)

Everything about xmax is recomputed from scratch on every write, because the lock state can change independently of the row’s identity.

Visibility: the rule

Visibility is decided by HeapTupleSatisfiesMVCC(), and the rule is compact:

the tuple is visible iff xmin is committed-and-not-in-snapshot AND (xmax is invalid / lock-only / in-snapshot-and-not-committed)

The Cmin/Cmax comparison against the snapshot’s curcid is what distinguishes “my own earlier command updated this row” from “my own later command will update this row” — the TM_SelfModified versus TM_Invisible split. And one detail that is easy to miss:

 c
 
 
	if (tuple->t_infomask & HEAP_XMAX_COMMITTED)
	{
		if (HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask))
			return TM_Ok;
		if (!ItemPointerEquals(&htup->t_self, &tuple->t_ctid))
			return TM_Updated;	/* updated by other */
		else
			return TM_Deleted;	/* deleted by other */
	}

The same t_ctid field that carries the HOT chain also distinguishes UPDATE from DELETE. t_ctid != t_self means “this version was updated”; t_ctid == t_self means “this version was deleted”. One field, two questions, answered by the same comparison.

HOT: updating without touching the index

The obvious cost of an UPDATE is that it creates a new index entry for the new version. Heap-Only Tuples avoid that. From the manual:

To help reduce the overhead of updates, PostgreSQL has an optimization called heap-only tuples (HOT).

HOT applies when both of these hold:

The update does not modify any column referenced by an index (excluding summarizing indexes like BRIN).
There is sufficient free space on the page containing the old row.

Then the new version is written to the same page, no index entry is created, and the chain is walked from the root tuple via t_ctid. The flags make this possible:

 c
 
 
	if (use_hot_update)
	{
		/* Mark the old tuple as HOT-updated */
		HeapTupleSetHotUpdated(&oldtup);
		/* And mark the new tuple as heap-only */
		HeapTupleSetHeapOnly(heaptup);
		/* Mark the caller's copy too, in case different from heaptup */
		HeapTupleSetHeapOnly(newtup);
	}

HEAP_HOT_UPDATED marks the predecessor (“my successor is heap-only”); HEAP_ONLY_TUPLE marks the successor (“no index entry points at me”). The chain looks like this:

text

 btree index entry ──► (blk, offnum=1) root tuple
 t_ctid = (blk, offnum=2)
 t_infomask2 |= HEAP_HOT_UPDATED
 lp[2] ──► version 2
 t_ctid = (blk, offnum=3)
 t_infomask2 |= HEAP_HOT_UPDATED
 lp[3] ──► version 3
 t_ctid = (blk, offnum=3) (self — end)
 t_infomask2 |= HEAP_ONLY_TUPLE

The reader follows the chain in heap_hot_search_buffer() (src/backend/access/heap/heapam_indexscan.c):

 c
 
 
		/* Check to see if HOT chain continues past this tuple; if so fetch
		 * the next offnum and loop around.
		 */
		if (HeapTupleIsHotUpdated(heapTuple))
		{
			Assert(ItemPointerGetBlockNumber(&heapTuple->t_data->t_ctid) ==
				 blkno);
			offnum = ItemPointerGetOffsetNumber(&heapTuple->t_data->t_ctid);
			at_chain_start = false;
			prev_xmax = HeapTupleHeaderGetUpdateXid(heapTuple->t_data);
		}
		else
			break;				/* end of chain */

Note the assertion: a HOT chain never crosses a page boundary. The successor is always in the same block. That is why HOT requires free space on the same page — a chain that hopped pages would need a different mechanism to find the next version.

The chain-integrity check follows immediately after:

 c
 
 
		/*
		 * The xmin should match the previous xmax value, else chain is
		 * broken.
		 */
		if (TransactionIdIsValid(prev_xmax) &&
			!TransactionIdEquals(prev_xmax,
								 HeapTupleHeaderGetXmin(heapTuple->t_data)))
			break;

Pruning: reclaiming the space

Dead versions accumulate. When a row is updated ten times, there are ten tuples on the page, and nine of them are garbage once no transaction can see them.

Pruning happens opportunistically, during ordinary scans, via heap_page_prune_opt(). The trigger is fillfactor-aware:

 c
 
 
	minfree = RelationGetTargetPageFreeSpace(relation,
											 HEAP_DEFAULT_FILLFACTOR);
	minfree = Max(minfree, BLCKSZ / 10);

	if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)

and it needs a cleanup lock, not merely an exclusive lock, because pruning can destroy line pointers other backends may hold references to.

The chain surgery is in heap_prune_chain(). Its header comment is the clearest statement of the algorithm in the whole tree:

The root line pointer is redirected to the tuple immediately after the latest DEAD tuple. If all tuples in the chain are DEAD, the root line pointer is marked LP_DEAD.

Three outcomes:

Situation	Root line pointer	Intermediate line pointers
No dead member	unchanged	unchanged
All members dead	→ `LP_DEAD`	→ `LP_UNUSED`
Some dead prefix	→ `LP_REDIRECT`	→ `LP_UNUSED`

The important subtlety: the t_ctid links are not rewritten during pruning. The chain is repaired by replacing the root line pointer’s storage with a redirect, not by patching t_ctid:

 c
 
		ItemIdSetRedirect(fromlp, tooff);

with

 c
 
 
#define ItemIdSetRedirect(itemId, link) \
( \
	(itemId)->lp_flags = LP_REDIRECT, \
	(itemId)->lp_off = (link), \
	(itemId)->lp_len = 0 \
)

The root keeps its offset number, so index entries pointing at it stay valid, but loses its storage (lp_len = 0) and repurposes lp_off to hold the offset number of the target. The reader follows the redirect at the start of a chain:

 c
 
 
		/* check for unused, dead, or redirected items */
		if (!ItemIdIsNormal(lp))
		{
			/* We should only see a redirect at start of chain */
			if (ItemIdIsRedirected(lp) && at_chain_start)
			{
				/* Follow the redirect */
				offnum = ItemIdGetRedirect(lp);
				at_chain_start = false;
				continue;
			}
			/* else must be end of chain */
			break;
		}

Now, the freed tuple bytes are still sitting there in the middle of the page, unreachable. PageRepairFragmentation() moves the survivors downward to close the gap:

 c
 
 
		/* shift the target offset down by the length of this tuple */
		upper -= itemidptr->alignedlen;
		/* point the copy_head to the start of this tuple */
		copy_head = itemidptr->itemoff;

		/* update the line pointer to reference the new offset */
		lp->lp_off = upper;

Note that lp_off is rewritten for every moved tuple. The line pointers are the only thing that has to stay stable; the tuples can go anywhere.

Two side structures

The free space map

The FSM is a per-relation array of pages, each page holding a tree of free-space values per heap page. It’s consulted when a tuple needs to be placed and the current target page is full, to find a page with room.

The visibility map

The VM is a bitmap with two bits per heap page — all-visible and all-frozen:

 c
 
 
/* Number of bits for one heap page */
#define BITS_PER_HEAPBLOCK 2

/* Flags for bit map */
#define VISIBILITYMAP_ALL_VISIBLE	0x01
#define VISIBILITYMAP_ALL_FROZEN	0x02

The VM page is nearly all bitmap:

 c
 
 
/*
 * Size of the bitmap on each visibility map page, in bytes. There's no
 * extra headers, so the whole page minus the standard page header is
 * used for the bitmap.
 */
#define MAPSIZE (BLCKSZ - MAXALIGN(SizeOfPageHeaderData))

8168 bytes of bitmap per VM page, four heap pages per byte, so 32 672 heap pages per VM page. The VM fork is about 1/32672 the size of the heap fork.

When every tuple on a heap page is visible to everyone, that page’s all-visible bit is set, and readers can skip the page entirely. This is why SELECT count(*) on an untouched table is nearly instant.

Look at it yourself

The pageinspect extension exposes all of this. It lives in contrib/pageinspect/ and is not installed by default. The raw-page functions require superuser:

 sql
 
 
CREATE EXTENSION pageinspect;

-- the line pointer array, and the header of every tuple on the page
SELECT lp, lp_off, lp_len, lp_flags, lp_flags = 1 AS normal,
 t_xmin, t_xmax, t_ctid, t_hoff
FROM heap_page_items(get_raw_page('mytable', 0));

-- the decoded flag names
SELECT (heap_tuple_infomask_flags(t_infomask, t_infomask2)).*
FROM heap_page_items(get_raw_page('mytable', 0));

-- where does each column live inside the tuple
SELECT t_ctid, s.*
FROM heap_page_items(get_raw_page('mytable', 0)) h,
 LATERAL unnest(tuple_data_split('mytable'::regclass,
 h.t_data, h.t_infomask,
 h.t_infomask2, h.t_bits)) s;

heap_page_items gives you the line pointer array and the raw tuple headers; tuple_data_split gives you the byte range of every column within a tuple. These two together are enough to verify everything in this article by hand.

What it all means in practice

A few consequences fall out of the layout, and they’re worth having in your head when you’re reasoning about a slow table:

Row width is expensive in a way that row count is not. The 23-byte header is paid by every row, but the alignment rules mean a table with one wide text column can waste more space in padding than a table with many narrow columns. MaxHeapTuplesPerPage is 291 on an 8 kB page:

 c
 
 
#define MaxHeapTuplesPerPage	\
	((int) ((BLCKSZ - SizeOfPageHeaderData) / \
			(MAXALIGN(SizeofHeapTupleHeader) + sizeof(ItemIdData))))

(8192 − 24) / (24 + 4) = 291. A page of 291 minimal rows has 291 tuples of 24 bytes plus 291 line pointers of 4 bytes, in 8192 bytes.

HOT updates are dramatically cheaper than cold ones. If you can update a row without touching an indexed column, and there’s room on the page, PostgreSQL writes a new tuple and nothing else. No index churn, no dead-index-entry cleanup. This is why fillfactor matters: a lower fillfactor leaves room on the page for the new version, and the chance of a HOT update goes up.

The 4-byte line pointer is a per-row tax on every page. It is the price of the indirection that makes VACUUM, HOT, and page defragmentation possible. It’s also why MaxHeapTupleSize is 8160 and not 8168.

An UPDATE never reuses the old tuple’s bytes. It appends a new version and leaves the old one for VACUUM to reclaim. A table that is updated constantly will bloat until vacuum catches up, and the bloat is not fragmentation — it’s whole tuples.

The t_ctid field is the load-bearing structure. It answers three questions at once: where is the newest version of this row, was this version updated or deleted, and where does the next HOT chain member live. Six bytes, three jobs.

Internals on Opensource internals

How ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

The layout

Global vs local order

Reading a row

Building the offsets

Insertion keeps the two arrays in sync

The insert path

Serialization: compact discriminators

Dynamic: the open-ended case

The insert path

Adding a variant

Seeing it in SQL

Trade-offs

How ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

The shape of the thing

The index column grows its own type

The dictionary: ColumnUnique and its reverse index

Two reserved slots

The reverse index stores row numbers, not values

The index is built lazily, and the hash table is filled in one pass

Float NaNs are canonicalized

Sharing the dictionary between columns

But a shared dictionary is immutable

Inserting from a column with the same dictionary

On disk: one dictionary per part, versioned per granule

The version byte

The per-granule header

Writing: the dictionary accumulates, then overflows

Reading: the reader has to reconstruct the same state

The single-dictionary-for-part fast path

The overflow path on the read side

The whole thing in SQL

Trade-offs

How ClickHouse Splits GROUP BY into 256 Buckets for Parallel Merges

The structure

The sub-tables are ordinary tables

How a bucket grows

Resizing is amortised

When is the two-level table used?

The merge is the point

The string flavour

Offsets across buckets

Trade-offs

Further reading

How ClickHouse Sorts in Parallel: Partial Sort, Merge Sorted Streams and External Sort

The plan: what gets built

PartialSortingTransform: sort the chunk, throw away the rest

Updating the threshold

FinishSortingTransform: merging streams that are already sorted

MergeSortingTransform: accumulate, remerge, spill, merge

Consume: accumulate chunks

Serialize and Generate

The k-way merge

Trade-offs

How ClickHouse Skips Data: Granules, Marks, and Bit-Packed Indexes

Granules: sized in bytes, not rows

Marks: where in the file does each granule start?

Where do the marks come from at write time?

The mark cache

The primary key index: one key value per granule

The trimming trick

The primary index cache

Now the payoff: two different search algorithms

Algorithm 1: binary search (continuous range)

Algorithm 2: generic exclusion search (everything else)

The step budget

The seek heuristic

Putting it together

What this buys you, and what it costs

Seeing it in action

How ClickHouse Runs a Query Pipeline: Ports, Status Codes and the Executing Graph

The building block: IProcessor

The port: a mailbox with the flags in the pointer

prepare() and work()

Version counters: how a processor learns that a neighbour changed

The ExecutingGraph: nodes, edges, and who owns what

The graph can change shape at runtime

Trade-offs

How ClickHouse Runs a Query Pipeline on N Threads: Task Queues, Work Stealing and Growing the Pool

`PartialSortingTransform`: sort the chunk, throw away the rest

`FinishSortingTransform`: merging streams that are already sorted

`MergeSortingTransform`: accumulate, remerge, spill, merge

The building block: `IProcessor`

`prepare()` and `work()`

The `ExecutingGraph`: nodes, edges, and who owns what

Measuring demand: `pushTasks` returns a number

CPU slots: `max_threads` is a ceiling, not a promise

The pulling mode: `executeUntil`

The `Wide`-only restriction, and the escape hatch