Performance on Opensource internals

How ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A Variant column is a tagged union: each row holds a value of one of several types, and the column itself knows the full list of alternatives. Dynamic is the same idea with an open-ended list — the set of types is discovered from the data.

The obvious implementation is a struct-of-arrays: one sub-column per alternative, each holding only the rows that belong to it, plus a tag per row saying which sub-column to look in. That is what ClickHouse does. What makes the implementation interesting is not the tag — it is the set of awkward consequences that fall out of it:

The order of the alternatives must not matter (Variant(String, UInt64) and Variant(UInt64, String) are the same type), but the storage of a discriminator is a small integer, so the order does matter on disk.
The type must be extendable at runtime without rewriting the data, which means a column’s internal ordering may differ from the type’s ordering.
The number of alternatives must be bounded, but a Dynamic column cannot know its alternatives in advance, so it needs an escape hatch.
Reading a row must not require a virtual call and a branch on the tag.

Let’s look at how ClickHouse 26.9 solves all four.

The layout

The header comment of ColumnVariant is a complete specification, including a worked example:

 cpp
 
 
// src/Columns/ColumnVariant.h:12
/**
 * Column for storing Variant(...) type values.
 * Variant type represents a union of other data types.
 * For example, type Variant(T1, T2, ..., TN) means that each row of this type
 * has a value of either type T1 or T2 or ... or TN or none of them (NULL value)
 *
 * ColumnVariant stores:
 * - The discriminators column, which determines which variant is stored in each row.
 * - The offsets column, which determines the offset in the corresponding variant column in each row.
 * - The list of variant columns with only real values (so the sizes of variant columns can be different).
 * Discriminator is an index of a variant in the variants list, it also has special value called NULL_DISCRIMINATOR
 * that indicates that the value in the row is NULL.
 *
 * We want to be able to extend Variant column for free without rewriting the data, but as we don't care about the
 * order of variants during Variant creation (we want Variant(T1, T2) to be the same as Variant(T2, T1)), we support
 * some global order of nested types inside Variant during type creation, so after extension the order of variant types
 * (and so their discriminators) can change. For example: Variant(T1, T3) -> Variant(T1, T2, T3).
 * To avoid full rewrite of discriminators column on Variant extension, we differentiate local order of variants
 * inside a column and global order of variants created during type creation. So, ColumnVariant stores only local
 * discriminators and additionally stores the mapping between global and local discriminators.
 * So, when we need to extend Variant column with new variant, we can just append it to a list of variant columns
 * with new local discriminator and update mapping from global to local orders.
 *
 * Note that two instances of ColumnVariant can have different local orders, so we should always use global
 * discriminators during inter-column interactions.
 *
 * Let's take an example with type Variant(UInt32, String, Array(UInt32)):
 * During type creation we will sort types by their names and get the global order: Array(UInt32), String, UInt32.
 * So, type Array(UInt32) will have global discriminator 0, String - 1 and UInt32 - 2.
 * Let's say we have a column with local order (String, UInt32, Array(UInt32)) and values:
 * 'Hello', 42, NULL, 'World', 43, [1, 2, 3], NULL, 44
 *
 * Let's see how these values will be stored in ColumnVariant:
 *
 * local_to_global_discriminators: {0 : 1, 1 : 2, 2 : 0}
 * global_to_local_discriminators: {0 : 2, 1 : 0, 2 : 1}
 * local_discriminators offsets String UInt32 Array(UInt32)
 * 0 0 'Hello' 42 [1, 2, 3]
 * 1 0 'World' 43
 * NULL_DISCRIMINATOR 0 44
 * 0 1
 * 1 1
 * 2 0
 * NULL_DISCRIMINATOR 0
 * 1 2
 *
 */

That diagram is worth staring at, because it shows the two central ideas:

text

 row: 0 1 2 3 4 5 6 7
 local: 0 1 NULL 0 1 2 NULL 1
 offsets: 0 0 0 1 1 0 0 2

 String: 'Hello' 'World' (2 values)
 UInt32: 42 43 44 (3 values)
 Array: [1,2,3] (1 value)

The discriminator is one byte per row. It is a local index into the column’s own list of variant columns.
The offset is a row index within that particular variant column. Row 3 has local discriminator 0 (String) and offset 1, so its value is String[1] == 'World'. Row 5 has local discriminator 2 (Array) and offset 0, so its value is Array[0].

So per row you pay one byte of tag plus one Offset (UInt64) of position. The values themselves are stored densely, one after another, inside their own sub-column — a String sub-column holds only the strings, a UInt32 sub-column holds only the integers, and neither has holes.

The types involved:

 cpp
 
 
// src/Columns/ColumnVariant.h:62
 using Discriminator = UInt8;
 using Discriminators = PaddedPODArray<Discriminator>;
 using ColumnDiscriminators = ColumnVector<Discriminator>;
 using ColumnOffsets = ColumnVector<Offset>;

 static constexpr UInt8 NULL_DISCRIMINATOR = std::numeric_limits<Discriminator>::max(); /// 255
 static constexpr size_t MAX_NESTED_COLUMNS = std::numeric_limits<Discriminator>::max(); /// 255

NULL_DISCRIMINATOR is 255 — one past the largest legal alternative index. That means a column can hold at most 255 alternatives, and 255 is reserved for NULL. The value is not stored in any sub-column; it is only a tag.

Global vs local order

The tag stored in the column is a local discriminator. The global discriminator is the position in the type’s canonical ordering. The canonical ordering is by type name — that is how Variant(T1, T2) and Variant(T2, T1) become the same type:

 cpp
 
 
// src/DataTypes/DataTypeVariant.cpp:46
DataTypeVariant::DataTypeVariant(const DataTypes & variants_)
{
 /// Sort nested types by their full names and squash identical types.
 std::map<String, DataTypePtr> name_to_type;
 for (const auto & type : variants_)
 {
 checkAllowedInsideVariant(type);

 /// Don't use Nothing type as a variant.
 if (!isNothing(type))
 name_to_type[type->getName()] = type;
 }

 variants.reserve(name_to_type.size());
 for (const auto & [_, type] : name_to_type)
 variants.push_back(type);

 checkVariantsNotEmptyAndNotTooMany(variants);
}

A std::map keyed by the type name gives the sorted, de-duplicated order for free.

Now, why do we need two orders at all? Because a Dynamic column is created with one set of alternatives and then extended as new types arrive. When Variant(T1, T3) becomes Variant(T1, T2, T3), the canonical (global) order of T3 may change — T2 might sort before it. If the column stored global discriminators, every stored tag would have to be rewritten.

The fix is to keep the column’s local order stable and just maintain a mapping:

 cpp
 
 
// src/Columns/ColumnVariant.h:160
 Discriminator ALWAYS_INLINE globalDiscriminatorByLocal(Discriminator local_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return local_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : local_to_global_discriminators[local_discr];
 }

 Discriminator ALWAYS_INLINE localDiscriminatorByGlobal(Discriminator global_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return global_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : global_to_local_discriminators[global_discr];
 }

and the extension operation is a pure metadata change:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1904
void ColumnVariant::extend(const VectorWithMemoryTracking<Discriminator> & old_to_new_global_discriminators, VectorWithMemoryTracking<std::pair<MutableColumnPtr, Discriminator>> && new_variants_and_discriminators)
{
 /// Update global discriminators for current variants.
 for (Discriminator & global_discr : local_to_global_discriminators)
 global_discr = old_to_new_global_discriminators[global_discr];

 /// Add new variants.
 variants.reserve(variants.size() + new_variants_and_discriminators.size());
 local_to_global_discriminators.reserve(local_to_global_discriminators.size() + new_variants_and_discriminators.size());
 for (auto & new_variant_and_discriminator : new_variants_and_discriminators)
 {
 variants.emplace_back(std::move(new_variant_and_discriminator.first));
 local_to_global_discriminators.push_back(new_variant_and_discriminator.second);
 }

 /// Update global -> local discriminators matching.
 global_to_local_discriminators.resize(local_to_global_discriminators.size());
 for (Discriminator local_discr = 0; local_discr != local_to_global_discriminators.size(); ++local_discr)
 global_to_local_discriminators[local_to_global_discriminators[local_discr]] = local_discr;
}

Nothing in that function touches the discriminator column or the offsets column. The existing rows’ local tags stay valid; only the interpretation of the tags changes. That is the “extend Variant column for free without rewriting the data” promise, kept.

The invariant is that local discriminators are always dense: local discriminator k is the index of variants[k]. Global discriminators are sparse in the sense that a given column may not contain all of them.

Reading a row

 cpp
 
 
// src/Columns/ColumnVariant.cpp:415
Field ColumnVariant::operator[](size_t n) const
{
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return Null();
 return (*variants[discr])[offsetAt(n)];
}

Two loads and a branch. The branch on NULL_DISCRIMINATOR is unavoidable; what you want to avoid is a virtual call per row, and variants[discr] is an IColumn&, so operator[] on it is a virtual call. That is why the hot paths in ClickHouse do not use operator[] but instead reach into the concrete sub-column via getDataAt, and why ColumnVariant also exposes getVariantByLocalDiscriminator so that callers can hoist the type test out of the loop.

Note also that the offsets array is not a byte offset into a shared arena. It is an element index into the sub-column. That is what allows each sub-column to be an ordinary ClickHouse column with its own type-specific representation (a ColumnString with offsets+chars, a ColumnVector<UInt32>, a ColumnArray, …), and it is why byteSizeAt has to consult the sub-column:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1564
size_t ColumnVariant::byteSizeAt(size_t n) const
{
 size_t res = sizeof(Offset) + sizeof(Discriminator);
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return res;

 return res + variants[discr]->byteSizeAt(offsetAt(n));
}

So a Variant row costs 9 bytes of overhead (1 discriminator + 8 offset) plus the payload. The offsets are the expensive part, and that is deliberate: an offset into a typed sub-column is what lets the sub-column be stored in its native, compressible layout.

Building the offsets

The offsets column is derived from the discriminators. constructOffsetsFromDiscriminators computes it, and it has three special cases:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:69
void ColumnVariant::constructOffsetsFromDiscriminators()
{
 const ColumnDiscriminators * discriminators_concrete = typeid_cast<const ColumnDiscriminators *>(local_discriminators.get());
 Offsets & offsets_data = typeid_cast<ColumnOffsets *>(offsets.get())->getData();
 offsets_data.clear();
 offsets_data.reserve(discriminators_concrete->size());
 /// If we have only NULLs, offsets column will not contain any real offsets.
 if (hasOnlyNulls())
 {
 /// Offsets of NULL rows are never read for their value, but compression reads the whole buffer.
 offsets_data.resize_fill(discriminators_concrete->size(), 0);
 }
 /// If we have only one non empty variant and no NULLs,
 /// offsets column will contain just sequential offsets 0, 1, 2, ...
 else if (getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 for (size_t i = 0; i != discriminators_concrete->size(); ++i)
 offsets_data.push_back(i);
 }
 /// Otherwise we should iterate through discriminators and
 /// remember current offset for each variant column.
 else
 {
 VectorWithMemoryTracking<Offset> nested_offsets;
 nested_offsets.resize(variants.size());
 for (Discriminator discr : discriminators_concrete->getData())
 {
 if (discr == NULL_DISCRIMINATOR)
 {
 offsets_data.emplace_back();
 }
 else
 {
 offsets_data.push_back(nested_offsets[discr]);
 ++nested_offsets[discr];
 }
 }
 }
}

The general case is a running counter per variant — a “rank within the sub-column” computation. The special cases matter for the serializer: if every row is NULL, the offsets buffer is all zeros (and the comment notes that compression reads the whole buffer, so it cannot be empty); if there is exactly one non-empty variant, the offsets are simply 0, 1, 2, ... and the whole offsets column is unnecessary information.

Insertion keeps the two arrays in sync

The discriminator and the offsets column must always have the same length. Because an allocation failure in the middle of an insert could leave them inconsistent, the code is careful:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:737
void ColumnVariant::insertDefault()
{
 getLocalDiscriminators().push_back(NULL_DISCRIMINATOR);
 /// Keep local_discriminators and offsets in sync even if appending to offsets throws
 /// (e.g. on a memory limit), otherwise popBack would over-pop the shorter one.
 try
 {
 getOffsets().emplace_back();
 }
 catch (...)
 {
 getLocalDiscriminators().popBack();
 throw;
 }
}

That is exception-safety hygiene that you will see repeatedly in this code base: the two columns are a matched pair, and one of them can throw (memory allocation) while the other cannot.

The insert path

Inserting a value from another variant column is where the global/local mapping earns its keep:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:501
void ColumnVariant::insertFromImpl(const DB::IColumn & src_, size_t n, const VectorWithMemoryTracking<ColumnVariant::Discriminator> * global_discriminators_mapping)
{
 const size_t num_variants = variants.size();
 const ColumnVariant & src = assert_cast<const ColumnVariant &>(src_);

 if (!global_discriminators_mapping && src.variants.size() != num_variants)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot insert value of Variant type with different number of types");

 Discriminator src_global_discr = src.globalDiscriminatorAt(n);
 Discriminator global_discr = src_global_discr;
 if (global_discriminators_mapping && src_global_discr != NULL_DISCRIMINATOR)
 global_discr = (*global_discriminators_mapping)[src_global_discr];
 Discriminator local_discr = localDiscriminatorByGlobal(global_discr);
 getLocalDiscriminators().push_back(local_discr);
 if (local_discr == NULL_DISCRIMINATOR)
 {
 getOffsets().emplace_back();
 }
 else
 {
 getOffsets().push_back(variants[local_discr]->size());
 variants[local_discr]->insertFrom(src.getVariantByGlobalDiscriminator(src_global_discr), src.offsetAt(n));
 }
}

Read it as a three-step translation: source local tag → source global tag → destination local tag. The source’s local tag means nothing to the destination; the global tag is the portable currency; the destination’s local tag is what gets stored.

Serialization: compact discriminators

On disk, the discriminator stream has two modes:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.h:54
 struct DiscriminatorsSerializationMode
 {
 enum Value
 {
 BASIC = 0, /// Store the whole discriminators column.
 COMPACT = 1, /// Don't write discriminators in granule if all of them are the same.
 };

In BASIC mode the whole discriminator column is written. In COMPACT mode a granule whose discriminators are all the same writes one discriminator plus a marker byte instead of 65 536 of them. The writer has to detect those three situations (uniform non-empty, all NULL, mixed) and there is a fast path for each:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:343
 /// Write number of rows in this granule in compact mode.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 writeVarUInt(UInt64(limit), *discriminators_stream);

 /// If column has only one none empty discriminators and no NULLs we don't need to
 /// calculate limits for variants and use provided offset/limit.
 if (auto non_empty_local_discr = col.getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 auto non_empty_global_discr = col.globalDiscriminatorByLocal(*non_empty_local_discr);

 /// In compact mode write the format of the granule and single non-empty discriminator.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 {
 writeBinaryLittleEndian(UInt8(CompactDiscriminatorsGranuleFormat::COMPACT), *discriminators_stream);
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }
 /// For basic mode just serialize this discriminator limit times.
 else
 {
 for (size_t i = 0; i < limit; ++i)
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }

The “one non-empty variant” fast path also lets the value streams be serialized with the same offset/limit as the whole column, because in that case the sub-column has exactly the same number of rows as the variant column:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:370
 /// We can use the same offset/limit as for whole Variant column
 if (i == non_empty_global_discr)
 {
 const auto & variant_column = col.getVariantByGlobalDiscriminator(i);
 if (variant_column.size() < offset + limit)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Variant {} has less rows ({}) than expected rows to serialize ({})", variant_names[i], variant_column.size(), offset + limit);

 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(variant_column, offset, limit, settings, variant_state->variant_states[i]);
 }
 else
 {
 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(col.getVariantByGlobalDiscriminator(i), col.getVariantByGlobalDiscriminator(i).size(), 0, settings, variant_state->variant_states[i]);
 }

In the general case each sub-column is serialized with its own length, which is the whole point of the dense layout: the String sub-column is written as a String column with only the strings in it, and so it compresses as well as a plain string column would.

Dynamic: the open-ended case

Dynamic is a Variant whose alternative list is discovered at runtime. It is implemented as a ColumnVariant plus a VariantInfo record:

 cpp
 
 
// src/Columns/ColumnDynamic.h:15
/**
 * Column for storing Dynamic type values.
 * Dynamic column allows to insert and store values of any data types inside.
 * Inside it stores:
 * - Variant column with all inserted values of different types.
 * - Information about currently stored variants.
 *
 * When new values are inserted into Dynamic column, the internal Variant
 * type and column are extended if the inserted value has new type.
 * When the limit on number of dynamic types is exceeded, all values
 * with new types are inserted into special shared variant with type String
 * that contains values and their types in binary format.
 */
class ColumnDynamic final : public COWHelper<IColumnHelper<ColumnDynamic>, ColumnDynamic>
{
public:
 /// Maximum limit on dynamic types. We use ColumnVariant to store all the types,
 /// so the limit cannot be greater then ColumnVariant::MAX_NESTED_COLUMNS.
 /// We also always have reserved variant for shared variant.
 static constexpr size_t MAX_DYNAMIC_TYPES_LIMIT = ColumnVariant::MAX_NESTED_COLUMNS - 1;
 static constexpr const char * SHARED_VARIANT_TYPE_NAME = "SharedVariant";

The escape hatch is a shared variant: one of the alternatives is always a String column, and it holds the type name and the binary-serialized value of anything that did not fit into a real alternative.

It has to be a String with a custom name, because a plain String alternative might legitimately be in the same column:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:71
/// Shared variant will contain String values but we cannot use usual String type
/// because we can have regular variant with type String.
/// To solve it, we use String type with custom name for shared variant.
const DataTypePtr & ColumnDynamic::getSharedVariantDataType()
{
 static thread_local DataTypePtr type = DataTypeFactory::instance().getCustom("String", std::make_unique<DataTypeCustomDesc>(std::make_unique<DataTypeCustomFixedName>(getSharedVariantTypeName())));
 return type;
}

and a fresh Dynamic column starts life with only the shared variant:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:80
ColumnDynamic::ColumnDynamic(size_t max_dynamic_types_) : max_dynamic_types(max_dynamic_types_), global_max_dynamic_types(max_dynamic_types)
{
 /// Create Variant with shared variant.
 setVariantType(std::make_shared<DataTypeVariant>(DataTypes{getSharedVariantDataType()}));
}

The insert path

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:257
void ColumnDynamic::insert(const Field & x)
{
 if (x.isNull())
 {
 insertDefault();
 return;
 }

 auto & variant_col = getVariantColumn();
 auto shared_variant_discr = getSharedVariantDiscriminator();
 /// Check if we can insert field into existing variants and avoid Variant extension.
 for (ColumnVariant::Discriminator i = 0; i != variant_col.getNumVariants(); ++i)
 {
 if (i != shared_variant_discr && variant_col.getVariantByGlobalDiscriminator(i).tryInsert(x))
 {
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(i));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(i).size() - 1);
 return;
 }
 }

 /// If we cannot insert field into current variant column, extend it with new variant for this field from its type.
 /// Use LeastSupertypeOnError::Dynamic so that arrays with incompatible element types (e.g. ["text", {"k":1}])
 /// are typed as Array(Dynamic) rather than throwing NO_COMMON_TYPE. Dynamic can hold any element value.
 auto field_data_type = applyVisitor(FieldToDataType<LeastSupertypeOnError::Dynamic>(), x);
 auto field_data_type_name = field_data_type->getName();
 if (addNewVariant(field_data_type, field_data_type_name))
 {
 /// Insert this field into newly added variant.
 auto discr = variant_info.variant_name_to_discriminator[field_data_type_name];
 variant_col.getVariantByGlobalDiscriminator(discr).insert(x);
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(discr));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(discr).size() - 1);
 }
 else
 {
 /// We reached maximum number of variants and couldn't add new variant.
 /// In this case we add the value of this new variant into special shared variant.
 /// We store values in shared variant in binary form with binary encoded type.
 auto & shared_variant = getSharedVariant();
 auto & chars = shared_variant.getChars();
 {
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());
 }
 shared_variant.getOffsets().push_back(chars.size());
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(shared_variant_discr));
 variant_col.getOffsets().push_back(shared_variant.size() - 1);
 }
}

Three outcomes, in order:

The value fits an existing alternative — use it, no extension.
The value has a new type and there is room — extend the variant list and add a real alternative.
The value has a new type and the list is full — encode the type and the value into the shared variant’s character buffer.

Case 3 is the interesting one. The shared variant is a ColumnString, so writing into it means writing into its chars buffer directly. The type is encoded first, then the value:

 cpp
 
 
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());

and reading it back is the mirror image:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:323
void ColumnDynamic::get(size_t n, Field & res) const
{
 /// Object, Dynamic and Variant values nest into each other to a depth that comes from the data, not from the declared type.
 checkStackSize();

 const auto & variant_col = getVariantColumn();
 /// Check if value is not in shared variant.
 if (variant_col.globalDiscriminatorAt(n) != getSharedVariantDiscriminator())
 {
 variant_col.get(n, res);
 if (!res.isNull())
 res = convertFieldToType(res, *getTypeAt(n));
 return;
 }

 /// We should deserialize value from shared variant.
 const auto & shared_variant = getSharedVariant();
 auto value_data = shared_variant.getDataAt(variant_col.offsetAt(n));
 ReadBufferFromMemory buf(value_data);
 auto type = decodeDataType(buf);
 type->getDefaultSerialization()->deserializeBinary(res, buf, getFormatSettings());
}

Note checkStackSize(): because Dynamic can nest inside Dynamic inside Dynamic, the depth is a function of the data, and a malicious input could otherwise blow the stack. That is a real concern for a type whose shape is discovered at runtime, and it shows up in the code as an explicit guard.

Adding a variant

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:145
bool ColumnDynamic::addNewVariant(const DataTypePtr & new_variant, const String & new_variant_name)
{
 /// Check if we already have such variant.
 if (variant_info.variant_name_to_discriminator.contains(new_variant_name))
 return true;

 /// Check if we reached maximum number of variants.
 if (!canAddNewVariant())
 {
 /// Dynamic column should always have shared variant.
 if (!variant_info.variant_name_to_discriminator.contains(getSharedVariantTypeName()))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Maximum number of variants reached, but no shared variant exists");

 return false;
 }

 const DataTypes & current_variants = assert_cast<const DataTypeVariant &>(*variant_info.variant_type).getVariants();
 DataTypes all_variants = current_variants;
 all_variants.push_back(new_variant);
 auto new_variant_type = std::make_shared<DataTypeVariant>(all_variants);
 updateVariantInfoAndExpandVariantColumn(new_variant_type);
 return true;
}

The new type is appended to the type, and then the column is expanded to match. updateVariantInfoAndExpandVariantColumn is where ColumnVariant::extend gets called with the old-to-new global discriminator mapping.

Seeing it in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `payload` Dynamic
)
ENGINE = MergeTree
ORDER BY ts;

INSERT INTO events VALUES
 ('2026-01-01 00:00:00', 42),
 ('2026-01-01 00:00:01', 'hello'),
 ('2026-01-01 00:00:02', [1, 2, 3]),
 ('2026-01-01 00:00:03', NULL);

-- The type of each value is remembered.
SELECT payload, dynamicType(payload) FROM events;

The dynamicType function is the user-visible face of the discriminator: it reads the tag and maps it to the type name.

Trade-offs

What the design buys you.

Each alternative is stored in its native layout, so a String alternative compresses like a string column, an UInt64 alternative compresses like a UInt64 column, and the whole thing benefits from per-column codecs. A Dynamic column that only ever saw strings is physically almost identical to a LowCardinality(String) or a plain String column — one byte of tag per row plus the payloads.

The global/local split means adding a type to a Dynamic column is O(1) in the number of rows. Nothing is rewritten; only two small mapping arrays change length.

What it costs you.

Nine bytes per row of pure overhead (one discriminator, one UInt64 offset), versus zero for a plain column. For a Variant with two alternatives and short values that overhead can exceed the payload.

The offsets are the expensive part, and they are the price of the dense sub-column layout. A “one big union buffer with a tag” design would pay one byte per row but would force every alternative into a common representation, losing per-type compression and per-type codecs.

The escape hatch is a real cost. Once a Dynamic column overflows into the shared variant, those values become opaque byte strings with an embedded type name. Reading one back requires decoding the type and then running the right deserializer — a dynamic dispatch on data, with a checkStackSize() guard because the nesting depth is attacker-controlled. And the shared variant is a String column, so it compresses like text even when it holds binary payloads.

The complexity is concentrated in one place. The mapping from global to local discriminator is the only genuinely subtle piece, and it exists purely so that Variant(T1, T2) and Variant(T2, T1) can be the same type while keeping the on-disk tags stable across extensions. Everything else — the offsets, the compact serialization, the NULL discriminator — is a fairly direct consequence of “one sub-column per alternative, one tag per row”.

How ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every column-oriented database eventually meets the same data: a column of country codes, of status strings, of user agents, of enum-like labels. There are a hundred thousand rows and twelve distinct values, repeated over and over. Storing the value in every row is absurd; storing the position of the value in every row is not.

ClickHouse’s answer is a column type called LowCardinality(T), and the implementation is one of the more interesting pieces of the whole storage engine, because it has to solve three separate problems at once:

how to represent the dictionary and the index in memory,
how to make the same dictionary serve many different blocks, granules and parts, so that the dictionary is stored once and not once per block,
how to serialize that arrangement on disk so that a reader can rebuild it, including the case where the dictionary had to change halfway through.

This article walks through all three, quoting the actual source from ClickHouse 26.9 (04eac2ea3).

The shape of the thing

The header comment of ColumnLowCardinality states the design in four lines:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:17
/**
 * How data is stored (in a nutshell):
 * we have a dictionary @e reverse_index in ColumnUnique that holds pairs (DataType, UIntXX) and a column
 * with UIntXX holding actual data indices.
 * To obtain the value's index, call #getOrFindIndex.
 * To operate on the data (so called indices column), call #getIndexes.
 *
 * @note The indices column always contains the default value (empty std::string_view) with the first index.
 */

The class holds exactly two members:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:398
 Dictionary dictionary;
 ColumnIndex idx;

Dictionary is a thin wrapper around a ColumnUnique plus a shared flag:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:366
private:
 class Dictionary
 {
 public:
 Dictionary(const Dictionary & other) = default;
 explicit Dictionary(MutableColumnPtr && column_unique, bool is_shared);
 explicit Dictionary(ColumnPtr column_unique, bool is_shared);

 const WrappedPtr & getColumnUniquePtr() const { return column_unique; }
 WrappedPtr & getColumnUniquePtr() { return column_unique; }

 const IColumnUnique & getColumnUnique() const { return static_cast<const IColumnUnique &>(*column_unique); }
 IColumnUnique & getColumnUnique() { return static_cast<IColumnUnique &>(*column_unique); }

 /// Dictionary may be shared for several mutable columns.
 /// Immutable columns may have the same column unique, which isn't necessarily shared dictionary.
 void setShared(const ColumnPtr & column_unique_);
 bool isShared() const { return shared; }

 /// Create new dictionary with only keys that are mentioned in indexes.
 void compact(MutableColumnPtr & indexes);
 /// Create nullable dictionary with only keys that are mentioned in indexes.
 void compactToNullable(MutableColumnPtr & indexes);

 static MutableColumnPtr compact(const IColumnUnique & column_unique, MutableColumnPtr & indexes);
 static MutableColumnPtr compactToNullable(const IColumnUnique & column_unique, MutableColumnPtr & indexes);

 private:
 WrappedPtr column_unique;
 bool shared = false;
 };

So a LowCardinality column is:

text

 +---------------------------+
 row 0 -----> | dictionary[ idx[0] ] | idx is a ColumnVector<UInt8|UInt16|UInt32|UInt64>
 row 1 -----> | dictionary[ idx[1] ] | dictionary is a ColumnUnique<ColumnString> (etc.)
 row 2 -----> | dictionary[ idx[2] ] |
 ... +---------------------------+

The values are in the dictionary; the rows are indices. Getting row n is two loads: getIndexes().getUInt(n) then a lookup in the dictionary. That second load is a random access into a (hopefully tiny) array — and the whole point is that the dictionary array is tiny enough to stay in L1/L2 while the index array streams.

The index column grows its own type

The index column is not fixed-width. It starts as UInt8 and is widened to UInt16, UInt32, UInt64 as the dictionary outgrows the range. The widening logic lives in ColumnIndex:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:135
void ColumnIndex::expandType()
{
 auto expand = [&](auto type)
 {
 using CurIndexType = decltype(type);
 constexpr auto next_size = NumberTraits::nextSize(sizeof(CurIndexType));
 if (next_size == sizeof(CurIndexType))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't expand indexes type for ColumnLowCardinality from type: {}",
 demangle(typeid(CurIndexType).name()));

 using NewIndexType = typename NumberTraits::Construct<false, false, next_size>::Type;
 convertIndexes<NewIndexType>();
 };

 callForType(std::move(expand), size_of_type);
}

and the bound that decides when to widen:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:152
size_t ColumnIndex::getMaxIndexForCurrentType() const
{
 chassert(size_of_type == 1 || size_of_type == 2 || size_of_type == 4 || size_of_type == 8);
 /// Shifting by 64 is undefined, and this is called per inserted value, so no `callForType` here.
 if (size_of_type == sizeof(UInt64))
 return std::numeric_limits<UInt64>::max();
 return (1ULL << (8 * size_of_type)) - 1;
}

The per-value insert path is where the widening happens, and note the comment about avoiding a typeid_cast on the hot path:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:198
void ColumnIndex::insertIndex(size_t index)
{
 while (index > getMaxIndexForCurrentType())
 expandType();

 auto insert = [&]<typename CurIndexType>(CurIndexType /*type_value*/)
 {
 /// `size_of_type` tracks the type of `indexes` and `expandType` keeps the two in sync, so
 /// this is `static_cast` in release. `getIndexesData` would run a `typeid_cast` instead, and
 /// this is the per-value path.
 assert_cast<ColumnVector<CurIndexType> *>(indexes.get())->getData().push_back(static_cast<CurIndexType>(index));
 };

 callForType(std::move(insert), size_of_type);

 /// Same invariant, so re-deriving it from `indexes` is a debug-only check here.
 chassert(size_of_type == getSizeOfIndexType(*indexes, size_of_type));
}

callForType is a hand-rolled dispatch over the four possible widths — no virtual call, no switch on a runtime type id, just four instantiations of a lambda. This is the pattern used throughout ClickHouse’s hot column code: the virtual IColumn interface is for the cold, structural paths; the per-row paths are devirtualized by dispatching on the concrete type once and then running a monomorphic loop.

Reading an index back has the same shape, and is marked ALWAYS_INLINE because it is called from the tightest loops in the engine:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:326
 ALWAYS_INLINE size_t getIndexAt(size_t row) const
 {
 const IColumn * indexes = &getIndexes();

 switch (idx.getSizeOfIndexType())
 {
 case sizeof(UInt8): return assert_cast<const ColumnUInt8 *>(indexes)->getElement(row);
 case sizeof(UInt16): return assert_cast<const ColumnUInt16 *>(indexes)->getElement(row);
 case sizeof(UInt32): return assert_cast<const ColumnUInt32 *>(indexes)->getElement(row);
 case sizeof(UInt64): return assert_cast<const ColumnUInt64 *>(indexes)->getElement(row);
 default: throwUnexpectedLowCardinalityIndexType(idx.getSizeOfIndexType());
 }
 }

The dictionary: ColumnUnique and its reverse index

ColumnUnique is the dictionary. It is not a column type you can select; the comment says so:

 cpp
 
 
// src/Columns/ColumnUnique.h:36
/** Stores another column with unique values
 * and also an index that allows to find position by value.
 *
 * This column is not used on it's own but only as implementation detail of ColumnLowCardinality.
 */
template <typename ColumnType>
class ColumnUnique final : public COWHelper<IColumnUnique, ColumnUnique<ColumnType>>

Its members are the values plus the lookup structure:

 cpp
 
 
// src/Columns/ColumnUnique.h:203
private:
 IColumn::WrappedPtr column_holder;
 bool is_nullable;
 size_t size_of_value_if_fixed = 0;

 ReverseIndex<UInt64, ColumnType> reverse_index;

 /// For DataTypeNullable, stores null map.
 IColumn::WrappedPtr nested_null_mask;
 IColumn::WrappedPtr nested_column_nullable;

Two reserved slots

The first row of the dictionary is special: it is the default value of the type. If the dictionary is nullable, row 0 is the NULL and row 1 is the default value of the inner type:

 cpp
 
 
// src/Columns/ColumnUnique.h:74
 size_t getDefaultValueIndex() const override { return 0; }
 size_t getNullValueIndex() const override;
 size_t getNestedTypeDefaultValueIndex() const override { return is_nullable ? 1 : 0; }
 bool canContainNulls() const override { return is_nullable; }

and

 cpp
 
 
// src/Columns/ColumnUnique.h:370
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::getNullValueIndex() const
{
 if (!is_nullable)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ColumnUnique can't contain null values.");

 return 0;
}

The count of reserved slots is therefore one or two:

 cpp
 
 
// src/Columns/ColumnUnique.h:238
 static size_t numSpecialValues(bool is_nullable) { return is_nullable ? 2 : 1; }
 size_t numSpecialValues() const { return numSpecialValues(is_nullable); }

Those reserved slots are not in the reverse index, which is why the insert path has to special-case them by hand:

 cpp
 
 
// src/Columns/ColumnUnique.h:513
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsertData(const char * pos, size_t length)
{
 /// The reserved prefix slots are not in the reverse index, so the default value is matched here.
 /// Comparing the first byte first keeps the `memcmp` call out of this path for the values that
 /// are not the default, which is all of them in a typical dictionary.
 const size_t default_index = getNestedTypeDefaultValueIndex();
 const std::string_view default_value = getRawColumnPtr()->getDataAt(default_index);
 if (default_value.size() == length && (length == 0 || (default_value[0] == pos[0] && default_value == std::string_view(pos, length))))
 return default_index;

 auto insertion_point = reverse_index.insert({pos, length});

 updateNullMask();

 return insertion_point;
}

The comment is a nice piece of micro-optimisation reasoning: comparing the first byte before the whole value keeps the expensive memcmp off the path for the (overwhelmingly common) case where the value is not the default.

The reverse index stores row numbers, not values

ReverseIndex is the interesting part. A naive dictionary would be a HashMap<value, index>, i.e. the values would be stored twice: once in the column, once as hash-table keys. ReverseIndex inverts that: the hash table stores the row number in the dictionary column, and comparisons dereference back into the column:

 cpp
 
 
// src/Columns/ReverseIndex.h:96
 /// Special case when we want to compare with something not in index_column.
 /// When we compare something inside column default keyEquals checks only that row numbers are equal.
 bool keyEquals(std::string_view object, size_t hash_ [[maybe_unused]], const State & state) const
 {
 auto index = key;
 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return hash_ == (*state.saved_hash_column)[index] && object == state.index_column->getDataAt(index);
 else
 return object == state.index_column->getDataAt(index);
 }

The hash table therefore holds 8-byte integers, and the dictionary column holds the only copy of the data. For a LowCardinality(String) with 1000 distinct values that is the difference between a hash table with 1000 string keys (plus their hashes, plus their lengths) and a hash table with 1000 UInt64s.

For string columns the hash is expensive, so it is precomputed once and kept in a parallel ColumnUInt64:

 cpp
 
 
// src/Columns/ReverseIndex.h:43
template <typename ColumnType>
struct ReverseIndexHashTableState<ColumnType, /* with_saved_hash = */ true, /* has_base_index */ false>
{
 constexpr static bool with_saved_hash = true;
 constexpr static bool has_base_index = false;

 ColumnType * index_column;
 typename ColumnVector<UInt64>::Container * saved_hash_column;
};

and the hash function for a stored row reads the precomputed hash rather than re-hashing the string:

 cpp
 
 
// src/Columns/ReverseIndex.h:108
 size_t getHash(const Hash & hash) const
 {
 auto index = key;

 /// Hack. HashTable is Hash itself.
 const auto & state = static_cast<const State &>(static_cast<const HashTable &>(hash));

 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return (*state.saved_hash_column)[index];
 else
 {
 using ValueType = typename ColumnType::ValueType;
 ValueType value = unalignedLoad<ValueType>(state.index_column->getDataAt(index).data());
 return DefaultHash<ValueType>()(value);
 }
 }

The index is built lazily, and the hash table is filled in one pass

The reverse index is not built at construction time; it is built on the first insert, and the build is a single pass over the dictionary:

 cpp
 
 
// src/Columns/ReverseIndex.h:435
template <typename IndexType, typename ColumnType>
void ReverseIndex<IndexType, ColumnType>::buildIndex()
{
 if (index)
 return;

 if (!column)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ReverseIndex can't build index because index column wasn't set.");

 auto size = column->size();
 index = std::make_unique<IndexMapType>(size);

 if constexpr (use_saved_hash)
 saved_hash = calcHashes();

 auto & state = index->getState();
 state.index_column = column;
 state.base_index = base_index;
 if constexpr (use_saved_hash)
 state.saved_hash_column = &saved_hash->getData();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 for (auto row : collections::range(num_prefix_rows_to_skip, size))
 {
 UInt64 hash = 0;
 if constexpr (use_saved_hash)
 hash = saved_hash->getElement(row);
 else
 hash = getHash(column->getDataAt(row));

 index->reverseIndexEmplace(row + base_index, iterator, inserted, hash, column->getDataAt(row));

 if (!inserted)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Duplicating keys found in ReverseIndex.");
 }
}

Note num_prefix_rows_to_skip: the reserved NULL/default slots are skipped, because they are matched by hand in uniqueInsertData.

The insert path is the mirror image, and it is written to be branch-light for the two different column kinds:

 cpp
 
 
// src/Columns/ReverseIndex.h:490
template <typename IndexType, typename ColumnType>
UInt64 ReverseIndex<IndexType, ColumnType>::insert(std::string_view data)
{
 if (!index)
 buildIndex();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 auto hash = getHash(data);
 UInt64 num_rows = size();

 if constexpr (use_saved_hash)
 {
 auto & column_data = saved_hash->getData();
 if (column_data.size() <= num_rows)
 column_data.resize(num_rows + 1);
 column_data[num_rows] = hash;
 }
 else
 column->insertData(data.data(), data.size());

 index->reverseIndexEmplace(num_rows + base_index, iterator, inserted, hash, data);

 if constexpr (use_saved_hash)
 {
 if (inserted)
 column->insertData(data.data(), data.size());
 }
 else
 {
 if (!inserted)
 column->popBack(1);
 }

 return iterator->getValue();
}

For strings the hash is appended optimistically and the value is appended only if the key was new. For fixed-size values the value is appended optimistically and popped back if the key turned out to be a duplicate. Each kind pushes and pops the cheaper thing.

Float NaNs are canonicalized

A LowCardinality(Float64) dictionary must not contain two NaNs, because NaN == NaN is false but all NaNs must compare equal in the dictionary semantics. So NaN is normalized at insert time:

 cpp
 
 
// src/Columns/ColumnUnique.h:389
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsert(const Field & x)
{
 if (x.isNull())
 return getNullValueIndex();

 // NaN can contain different sign or mantissa bits, but we need to consider all NaNs equal.
 if constexpr (is_float_vector_v<ColumnType>)
 {
 if (isNaN(x.safeGet<typename ColumnType::ValueType>()))
 {
 auto nan = NaNOrZero<typename ColumnType::ValueType>();
 return uniqueInsertData(reinterpret_cast<char *>(&nan), sizeof(nan));
 }
 }

Here is where the design pays off. A LowCardinality column can point at a dictionary owned by another column. The shared flag on Dictionary records this, and byteSize() deliberately does not count a shared dictionary:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:200
 /// Don't count the dictionary size as it can be shared between different blocks.
 size_t byteSize() const override { return idx.getIndexes()->byteSize() + (isSharedDictionary() ? 0 : getDictionary().byteSize()); }

 size_t byteSizeAt(size_t n) const override { return getDictionary().byteSizeAt(getIndexes().getUInt(n)); }
 size_t allocatedBytes() const override { return idx.getIndexes()->allocatedBytes() + getDictionary().allocatedBytes(); }

Note the asymmetry: byteSize() (the logical size, used for statistics and for system.parts_columns) excludes the shared dictionary, but allocatedBytes() (the physical size, used for memory accounting) includes it. That distinction is what makes the accounting correct when a part has a hundred blocks all pointing at one dictionary.

Sharing also propagates through every operation that produces a new column from an old one — cut, filter, permute, index, replicate all hand the same dictionary pointer to the result:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:130
 ColumnPtr filter(const Filter & filt, ssize_t result_size_hint) const override
 {
 return ColumnLowCardinality::create(
 dictionary.getColumnUniquePtr(), getIndexes().filter(filt, result_size_hint), isSharedDictionary());
 }

But a shared dictionary is immutable

The moment you want to mutate a column whose dictionary is shared with somebody else, the dictionary has to be un-shared. This happens in compactIfSharedDictionary(), which is called at the top of every mutating insert:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:146
void ColumnLowCardinality::insert(const Field & x)
{
 compactIfSharedDictionary();
 idx.insertIndex(getDictionary().uniqueInsert(x));
}

bool ColumnLowCardinality::tryInsert(const Field & x)
{
 compactIfSharedDictionary();

 size_t index = 0;
 if (!dictionary.getColumnUnique().tryUniqueInsert(x, index))
 return false;

 idx.insertIndex(index);
 return true;
}

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:894
void ColumnLowCardinality::compactIfSharedDictionary()
{
 if (dictionary.isShared())
 compactInplace();
}

Compaction rebuilds a private dictionary containing only the keys that are actually referenced by this column’s indexes, and remaps the indexes accordingly:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:1002
MutableColumnPtr ColumnLowCardinality::Dictionary::compact(const IColumnUnique & unique, MutableColumnPtr & indexes)
{
 auto new_column_unique = unique.cloneEmpty();
 auto & new_unique = static_cast<IColumnUnique &>(*new_column_unique);

 auto unique_indexes = mapUniqueIndex(*indexes);
 auto sub_keys = unique.getNestedColumn()->index(*unique_indexes, 0);
 auto new_indexes = new_unique.uniqueInsertRangeFrom(*sub_keys, 0, sub_keys->size());

 indexes = IColumn::mutate(new_indexes->index(*indexes, 0));
 return new_column_unique;
}

mapUniqueIndex is the piece that does the renumbering. It is a counting-sort-like trick that works in O(n) with a flat array, and it has an overflow path for when the dictionary is shared and the indexes do not fit the “max index < number of rows” assumption:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:79
template <typename T>
MutableColumnPtr mapUniqueIndexImpl(PaddedPODArray<T> & index)
{
 if (index.empty())
 return ColumnVector<T>::create();

 auto size = index.size();

 T max_val = index[0];
 for (size_t i = 1; i < size; ++i)
 max_val = std::max(max_val, index[i]);

 /// May happen when dictionary is shared.
 if (max_val > size)
 return mapUniqueIndexImplRef(index);

 auto map_size = static_cast<UInt64>(max_val) + 1;
 PaddedPODArray<T> map(map_size, 0);
 T zero_pos_value = index[0];
 index[0] = 0;
 T cur_pos = 0;
 for (size_t i = 1; i < size; ++i)
 {
 T val = index[i];
 if (val != zero_pos_value && map[val] == 0)
 {
 ++cur_pos;
 map[val] = cur_pos;
 }

 index[i] = map[val];
 }

The “overflow” case (max_val > size) means the column references dictionary entries that belong to a bigger dictionary — exactly what happens when the dictionary is shared. Then a hash map is used instead of a flat array:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:53
template <typename T>
MutableColumnPtr mapUniqueIndexImplRef(PaddedPODArray<T> & index)
{
 PaddedPODArray<T> copy(index.cbegin(), index.cend());

 HashMap<T, T> hash_map;
 for (auto val : index)
 hash_map.insert({val, static_cast<T>(hash_map.size())});

 auto res_col = ColumnVector<T>::create();
 auto & data = res_col->getData();

 data.resize(hash_map.size());
 for (const auto & val : hash_map)
 data[val.getMapped()] = val.getKey();

 for (auto & ind : index)
 ind = hash_map[ind];

 for (size_t i = 0; i < index.size(); ++i)
 if (data[index[i]] != copy[i])
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected {}, but got {}", toString(data[index[i]]), toString(copy[i]));

 return res_col;
}

That last loop is a self-check: after the renumbering, the new index must still resolve to the same value. Cheap insurance against an off-by-one in a data structure that is otherwise very hard to test.

Inserting from a column with the same dictionary

Once dictionaries are shared, a very common operation becomes trivial: copying a row from another LowCardinality column that happens to use the same dictionary is just a copy of an index.

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:180
 size_t position = low_cardinality_src->getIndexes().getUInt(n);

 if (&low_cardinality_src->getDictionary() == &getDictionary())
 {
 /// Dictionary is shared with src column. Insert only index.
 idx.insertIndex(position);
 }
 else
 {
 compactIfSharedDictionary();
 const auto & nested = *low_cardinality_src->getDictionary().getNestedColumn();
 idx.insertIndex(getDictionary().uniqueInsertFrom(nested, position));
 }

The test is a pointer comparison of the dictionary addresses. If they are the same object, the index is valid as-is; otherwise the value has to be re-inserted into this column’s dictionary and the resulting index used.

On disk: one dictionary per part, versioned per granule

Now the hard part. In a MergeTree part, a LowCardinality column is written as two streams: DictionaryKeys and DictionaryIndexes.

The keys stream holds the dictionary; the indexes stream holds, per granule, a header and then the index data. The dictionary is not rewritten per granule — it accumulates until it hits a size limit, at which point it is flushed and a new one starts.

The version byte

The keys stream starts with a version:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:102
struct KeysSerializationVersion
{
 enum Value
 {
 /// Version is written at the start of <name.dict.bin>.
 /// Dictionary is written as number N and N keys after them.
 /// Dictionary can be shared for continuous range of granules, so some marks may point to the same position.
 /// Shared dictionary is stored in state and is read once.
 SharedDictionariesWithAdditionalKeys = 1,
 };

 Value value;

 static void checkVersion(UInt64 version)
 {
 if (version != SharedDictionariesWithAdditionalKeys)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Invalid version for SerializationLowCardinality key column.");
 }

 explicit KeysSerializationVersion(UInt64 version) : value(static_cast<Value>(version)) { checkVersion(version); }
};

The single legal version is called SharedDictionariesWithAdditionalKeys, and the comment tells you the whole story in three lines: the dictionary is written as a count followed by that many keys, it can be shared across a continuous range of granules, and the shared dictionary is kept in the serialization state and read once.

The per-granule header

Every granule’s index data begins with a header that encodes the index width plus three flags:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:124
/// Version is stored at the start of each granule. It's used to store indexes type and flags.
struct IndexesSerializationType
{
 using SerializationType = UInt64;
 /// Need to read dictionary if it wasn't.
 static constexpr SerializationType NeedGlobalDictionaryBit = 1u << 8u;
 /// Need to read additional keys. Additional keys are stored before indexes as value N and N keys after them.
 static constexpr SerializationType HasAdditionalKeysBit = 1u << 9u;
 /// Need to update dictionary. It means that previous granule has different dictionary.
 static constexpr SerializationType NeedUpdateDictionary = 1u << 10u;

 enum Type
 {
 TUInt8 = 0,
 TUInt16,
 TUInt32,
 TUInt64,
 };

 Type type;
 bool has_additional_keys;
 bool need_global_dictionary;
 bool need_update_dictionary;

 static constexpr SerializationType resetFlags(SerializationType type)
 {
 return type & (~(HasAdditionalKeysBit | NeedGlobalDictionaryBit | NeedUpdateDictionary));
 }

The three flags are the entire state machine:

NeedGlobalDictionaryBit — “there is a dictionary in the keys stream that I have not read yet”.
HasAdditionalKeysBit — “this granule carries keys that did not fit in the shared dictionary; they are stored inline, before the index data”.
NeedUpdateDictionary — “the dictionary changed since the previous granule”.

Serialization and deserialization are symmetric and trivial:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:162
 void serialize(WriteBuffer & buffer) const
 {
 SerializationType val = type;
 if (has_additional_keys)
 val |= HasAdditionalKeysBit;
 if (need_global_dictionary)
 val |= NeedGlobalDictionaryBit;
 if (need_update_dictionary)
 val |= NeedUpdateDictionary;
 writeBinaryLittleEndian(val, buffer);
 }

 void deserialize(ReadBuffer & buffer, const ISerialization::DeserializeBinaryBulkSettings & settings)
 {
 SerializationType val = 0;
 readBinaryLittleEndian(val, buffer);

 checkType(val);
 has_additional_keys = (val & HasAdditionalKeysBit) != 0;
 need_global_dictionary = (val & NeedGlobalDictionaryBit) != 0;
 need_update_dictionary = (val & NeedUpdateDictionary) != 0;
 type = static_cast<Type>(resetFlags(val));

 if (settings.native_format)
 {
 if (need_global_dictionary)
 throw Exception(ErrorCodes::INCORRECT_DATA,
 "LowCardinality indexes serialization type for Native format "
 "cannot use global dictionary");
 }
 }

The low four bits carry the index width; bits 8, 9, 10 are the flags. Bits 4–7 are unused — a hole left for future flags.

Writing: the dictionary accumulates, then overflows

The writer keeps the accumulating dictionary in the serialization state:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:232
struct SerializeStateLowCardinality : public ISerialization::SerializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 MutableColumnUniquePtr shared_dictionary;

 explicit SerializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}
};

The write path is where the interesting decisions live:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:561
 const ColumnLowCardinality & low_cardinality_column = typeid_cast<const ColumnLowCardinality &>(column);

 auto * low_cardinality_state = checkAndGetState<SerializeStateLowCardinality>(state);
 auto & global_dictionary = low_cardinality_state->shared_dictionary;
 KeysSerializationVersion::checkVersion(low_cardinality_state->key_version.value);

 size_t max_limit = column.size() - offset;
 limit = limit ? std::min(limit, max_limit) : max_limit;

 /// Do not write anything for empty column. (May happen while writing empty arrays.)
 if (limit == 0)
 return;

 bool need_update_dictionary = global_dictionary == nullptr;
 if (need_update_dictionary)
 global_dictionary = DataTypeLowCardinality::createColumnUnique(*dictionary_type);

 auto sub_column = low_cardinality_column.cutAndCompact(offset, limit);
 ColumnPtr positions = sub_column->getIndexesPtr();
 ColumnPtr keys = sub_column->getDictionary().getNestedColumn();

 if (settings.low_cardinality_max_dictionary_size)
 {
 /// Insert used_keys into global dictionary and update sub_index.
 auto indexes_with_overflow = global_dictionary->uniqueInsertRangeWithOverflow(*keys, 0, keys->size(),
 settings.low_cardinality_max_dictionary_size);

 if (global_dictionary->size() > settings.low_cardinality_max_dictionary_size)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Got dictionary with size {} but max dictionary size is {}",
 global_dictionary->size(), settings.low_cardinality_max_dictionary_size);

 positions = indexes_with_overflow.indexes->index(*positions, 0);
 keys = std::move(indexes_with_overflow.overflowed_keys);

 if (global_dictionary->size() < settings.low_cardinality_max_dictionary_size && !keys->empty())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Has additional keys, but dict size is {} which is less "
 "then max dictionary size ({})", global_dictionary->size(),
 settings.low_cardinality_max_dictionary_size);
 }

cutAndCompact is the first step: it takes the slice of the column being written, and produces a private dictionary containing only the keys that slice actually uses. That is what keeps the accumulated dictionary small — without it, every granule would re-insert the whole dictionary.

Then the keys are merged into the global dictionary, and anything that does not fit within low_cardinality_max_dictionary_size becomes “overflowed keys” — keys carried inline by that granule.

The flush happens when the accumulated dictionary reaches the limit:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:602
 bool need_additional_keys = !keys->empty();
 bool need_dictionary = settings.low_cardinality_max_dictionary_size != 0;
 bool need_write_dictionary = !settings.low_cardinality_use_single_dictionary_for_part
 && global_dictionary->size() >= settings.low_cardinality_max_dictionary_size;

 IndexesSerializationType index_version(*positions, need_additional_keys, need_dictionary, need_update_dictionary);
 index_version.serialize(*indexes_stream);

 if (need_write_dictionary)
 {
 const auto & nested_column = global_dictionary->getNestedNotNullableColumn();
 UInt64 num_keys = nested_column->size();
 writeBinaryLittleEndian(num_keys, *keys_stream);
 dict_inner_serialization->serializeBinaryBulk(*nested_column, *keys_stream, 0, num_keys);
 low_cardinality_state->shared_dictionary = nullptr;
 }

 if (need_additional_keys)
 {
 UInt64 num_keys = keys->size();
 writeBinaryLittleEndian(num_keys, *indexes_stream);
 dict_inner_serialization->serializeBinaryBulk(*keys, *indexes_stream, 0, num_keys);
 }

 UInt64 num_rows = positions->size();
 writeBinaryLittleEndian(num_rows, *indexes_stream);
 auto index_serialization = index_version.getDataType()->getDefaultSerialization();
 index_serialization->serializeBinaryBulk(*positions, *indexes_stream, 0, num_rows);

The default limit is 8192 keys:

 cpp
 
 
// src/Core/Settings.cpp:5045
 DECLARE(UInt64, low_cardinality_max_dictionary_size, 8192, R"(
Sets a maximum size in rows of a shared global dictionary for the [LowCardinality](/reference/data-types/lowcardinality) data type that can be written to a storage file system. This setting prevents issues with RAM in case of unlimited dictionary growth. All the data that can't be encoded due to maximum dictionary size limitation ClickHouse writes in an ordinary method.

Possible values:

- Any positive integer.
)", 0) \

So: with an 8192-key limit, a part whose column has 100 distinct values writes the dictionary once; a part with 100 000 distinct values writes the dictionary roughly every 8192 keys and starts a new one.

Note that need_update_dictionary is global_dictionary == nullptr — i.e. “this granule starts a new dictionary”. And the dictionary is flushed after the header was written, so the header says “read the dictionary” and the dictionary follows in the keys stream.

Reading: the reader has to reconstruct the same state

The read side keeps the shared dictionary in the deserialize state, along with a flag that survives granule skips:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:240
struct DeserializeStateLowCardinality : public ISerialization::DeserializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 ColumnUniquePtr global_dictionary;

 IndexesSerializationType index_type{};
 ColumnPtr additional_keys;
 ColumnPtr null_map;
 UInt64 num_pending_rows = 0;

 /// If dictionary should be updated.
 /// Can happen is some granules was skipped while reading from MergeTree.
 /// We should store this flag in State because
 /// in case of long block of empty arrays we may not need read dictionary at first reading.
 bool need_update_dictionary = false;

 /// True if this part has a single dictionary for the entire column. The prefix
 /// reader first checks `has_uniform_marks_callback` as a cheap prefilter, then
 /// verifies it by reading the first dictionary and checking that `DictionaryKeys`
 /// reached EOF. When true, non-continuous reads (granule skips) do not invalidate
 /// the cached dictionary, and the dictionary can be shared across all threads via `clone`.
 bool single_dictionary_for_part = false;

 explicit DeserializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}

 ISerialization::DeserializeBinaryBulkStatePtr cloneImpl(ISerialization::DeserializeBinaryBulkStateCloneContext &) const override
 {
 return std::make_shared<DeserializeStateLowCardinality>(*this);
 }
};

The need_update_dictionary comment is the interesting one: when reading is not continuous (i.e. some granules were skipped), the reader cannot assume that the dictionary it holds is the one the next granule refers to, so it must force a re-read. That flag lives in the state, not on the stack, precisely because a skipped granule may be discovered long after the state was created.

The main read loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:752
 if (!settings.continuous_reading)
 {
 low_cardinality_state->num_pending_rows = 0;

 /// Some granules were skipped; the next granule header may signal a dictionary
 /// update. For multi-dict parts this forces a re-read at the next boundary.
 low_cardinality_state->need_update_dictionary = true;
 }

 while (limit)
 {
 if (low_cardinality_state->num_pending_rows == 0)
 {
 if (indexes_stream->eof())
 break;

 auto & index_type = low_cardinality_state->index_type;
 auto & global_dictionary = low_cardinality_state->global_dictionary;

 index_type.deserialize(*indexes_stream, settings);

 bool need_update_dictionary =
 !global_dictionary || index_type.need_update_dictionary || low_cardinality_state->need_update_dictionary;
 if (index_type.need_global_dictionary && need_update_dictionary)
 {
 /// For single-dict parts the in-memory dictionary is valid for the entire part,
 /// so skip re-reading it even if the stream was seeked back to the beginning.
 if (!low_cardinality_state->single_dictionary_for_part || !global_dictionary)
 read_dictionary();
 low_cardinality_state->need_update_dictionary = false;
 }

 if (low_cardinality_state->index_type.has_additional_keys)
 read_additional_keys();
 else
 low_cardinality_state->additional_keys = nullptr;

 readBinaryLittleEndian(low_cardinality_state->num_pending_rows, *indexes_stream);
 }

 size_t num_rows_to_read = std::min<UInt64>(limit, low_cardinality_state->num_pending_rows);
 read_indexes(num_rows_to_read);
 limit -= num_rows_to_read;
 low_cardinality_state->num_pending_rows -= num_rows_to_read;
 }

num_pending_rows is what makes a granule’s worth of rows readable in chunks: the header is read once, then the reader consumes num_pending_rows rows from the index stream, and only when that counter hits zero does it look for the next header.

The single-dictionary-for-part fast path

There is a special case worth highlighting: a part where the column has exactly one dictionary for the whole part. The prefix reader detects it by reading the first dictionary and checking whether the DictionaryKeys stream reached EOF:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:348
 if (might_have_single_dictionary && !stream->eof())
 {
 /// For `Wide` parts that might have a single dictionary, the stream is
 /// now positioned right at the dictionary data (after the version).
 /// Read the first dictionary and check EOF afterwards. If this proves
 /// that there are no more dictionaries, all threads sharing this prefix
 /// state via `DeserializationPrefixesCache` get the dictionary through
 /// a `shared_ptr` clone instead of each reading it independently.

and

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:377
 if (stream->eof())
 {
 new_state->single_dictionary_for_part = true;
 new_state->global_dictionary = std::move(global_dictionary);
 }
 else
 {
 /// The mark heuristic can be fooled by several dictionaries written
 /// inside one mark interval. Reset the `DictionaryKeys` stream back
 /// to the normal post-prefix position and let data deserialization
 /// read dictionary updates from the stream.
 if (!settings.seek_to_start_callback)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot reset LowCardinality dictionary stream without seek_to_start_callback");

 settings.seek_to_start_callback(dictionary_keys_path);

 UInt64 repeated_keys_version = 0;
 readBinaryLittleEndian(repeated_keys_version, *stream);
 if (repeated_keys_version != keys_version)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Inconsistent version while resetting LowCardinality dictionary stream");
 }

This is a heuristic with a verification step. The cheap check (has_uniform_marks_callback) says “all marks point at the same position in the keys stream, so maybe there is only one dictionary”. The expensive check (actually reading the dictionary and seeing EOF) confirms it. If the heuristic lied, the stream is rewound and the normal path takes over. That combination — cheap prefilter, expensive confirmation, safe fallback — is a pattern you will see repeatedly in ClickHouse’s MergeTree reader.

When the fast path holds, cloneImpl copies the state (including the dictionary pointer) so every thread reading the part gets the same dictionary without re-reading it.

The overflow path on the read side

When a granule carries inline “additional keys”, the reader has to merge two dictionaries: the global one and the granule-local one. mapIndexWithAdditionalKeys does that with two flat maps — one for the global dictionary’s keys and one for the overflow keys:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:515
 /// Update column and return map with old indexes.
 /// Let N is the number of distinct values which are less than max_size;
 /// old_column - column before function call;
 /// new_column - column after function call:
 /// * if old_column[i] < max_size, than
 /// dictionary_map[new_column[i]] = old_column[i]
 /// * else
 /// additional_keys_map[new_column[i]] = old_column[i] - dict_size + N
 IndexMapsWithAdditionalKeys mapIndexWithAdditionalKeys(IColumn & column, size_t dict_size)

and then the keys are stitched back together:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:726
 else
 {
 auto maps = mapIndexWithAdditionalKeys(*indexes_column, global_dictionary->size());

 auto used_keys = IColumn::mutate(global_dictionary->getNestedColumn()->index(*maps.dictionary_map, 0));

 if (!maps.additional_keys_map->empty())
 {
 if (additional_keys == nullptr)
 throw Exception(ErrorCodes::INCORRECT_DATA, "No additional keys found.");

 auto used_add_keys = additional_keys->index(*maps.additional_keys_map, 0);

 if (dictionary_type->isNullable())
 {
 ColumnPtr null_map = ColumnUInt8::create(used_add_keys->size(), static_cast<UInt8>(0));
 used_add_keys = ColumnNullable::create(used_add_keys, null_map);
 }

 used_keys->insertRangeFrom(*used_add_keys, 0, used_add_keys->size());
 }

 low_cardinality_column.insertRangeFromDictionaryEncodedColumn(*used_keys, *indexes_column);
 }

The final call re-inserts the merged keys into the destination column’s own dictionary and remaps the indexes one last time.

The whole thing in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `user_id` UInt64,
 `kind` LowCardinality(String), -- 12 distinct values in 100M rows
 `country` LowCardinality(FixedString(2))
)
ENGINE = MergeTree
ORDER BY (ts, user_id);

 sql
 
 
-- The dictionary is shared across every block of the part, so the per-column
-- cost is one byte per row (UInt8 index) plus the dictionary once.
SELECT kind, count() FROM events GROUP BY kind;

The type-level guarantee is that LowCardinality is only worth it while the dictionary stays small; the documentation says so plainly:

text

// src/DataTypes/DataTypeLowCardinality.cpp:198
The efficiency of using `LowCardinality` data type depends on data diversity. If a dictionary contains less than 10,000 distinct values, then ClickHouse mostly shows higher efficiency of data reading and storing. If a dictionary contains more than 100,000 distinct values, then ClickHouse can perform worse in comparison with using ordinary data types.

Trade-offs

What the hack buys you.

A LowCardinality(String) column with 12 distinct values costs one byte per row plus a few hundred bytes of dictionary, instead of ~10 bytes per row of strings. It also means the scan is a byte scan, not a string scan: filters, GROUP BY, and ORDER BY all operate on UInt8 indexes, and the dictionary is touched only for the rows that survive the filter.

What it costs you.

Every value access is two dependent loads: read the index, then read the dictionary entry. That is a pointer chase that a plain String column does not have — although for a plain string column you also pay a pointer chase into the character arena, so the two are more comparable than they look.

The dictionary is a global per part, and the index stream has to carry a per-granule header with three flags and a row count, plus the possibility of inline overflow keys. A plain column has none of that. For a column with 100 000+ distinct values you have bought complexity and lost compression.

The sharing is also asymmetric: byteSize() excludes a shared dictionary while allocatedBytes() includes it. That is correct but subtle, and it means the two numbers in system.parts_columns can disagree in a way that looks like a bug until you know why.

The design decision that makes it work. The dictionary is not stored per block, and not per granule; it is stored per part, capped at 8192 keys, and flushed when it overflows. Everything else — the three header bits, the need_update_dictionary flag, the single_dictionary_for_part heuristic — exists to make that one decision survive partial reads, granule skips and multi-threaded readers.

How ClickHouse Sorts in Parallel: Partial Sort, Merge Sorted Streams and External Sort

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

ORDER BY is the operation every database is worst at. It is the one requirement that is inherently global: the first row of the result depends on every row of the input. Everything else — filtering, projection, aggregation — is local and streams beautifully. Sorting does not.

The classic answer is “sort in chunks, then merge the sorted chunks”. That is what every system does, but there are two quite different versions of it, and ClickHouse implements both, in two quite different classes.

One version assumes the input is already sorted on a prefix of the sort key (a MergeTree table read in primary-key order, or the output of a previous ORDER BY). Then the merge is cheap and the only work is stitching together runs that are already in order.

The other version assumes nothing: it sorts each chunk independently and then does a k-way merge. This is the general case, and it is where the memory pressure lives — a k-way merge of N sorted runs needs all N runs resident at once.

This is a tour of both, of the trick that lets ClickHouse throw away rows it knows are useless, and of the machinery that spills sorted runs to disk when memory runs out.

The plan: what gets built

SortingStep decides which of the two regimes applies. The difference is visible in the query:

 sql
 
 
-- Full sort: the input is in no particular order, so each stream is sorted
-- on its own and the sorted streams are merged.
SELECT * FROM hits ORDER BY UserID, EventTime LIMIT 100;

 sql
 
 
-- Finish sorting: the table is sorted by (a, b) and the query asks for
-- (a, b) too, so the merge only has to stitch runs together.
SELECT * FROM hits WHERE a = 1 ORDER BY a, b LIMIT 100;

The “finish sorting” case is when the input is already sorted on a prefix:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:381
void SortingStep::finishSorting(
 QueryPipelineBuilder & pipeline, const SortDescription & input_sort_desc, const SortDescription & result_sort_desc, const UInt64 limit_)
{
 pipeline.addSimpleTransform(
 [&](const SharedHeader & header, QueryPipelineBuilder::StreamType stream_type) -> ProcessorPtr
 {
 if (stream_type != QueryPipelineBuilder::StreamType::Main)
 return nullptr;

 return std::make_shared<PartialSortingTransform>(header, result_sort_desc, limit_, threshold_tracker);
 });

Two transforms per stream: a PartialSortingTransform that sorts each chunk as it arrives, and a FinishSortingTransform that merges the sorted chunks. The “full sort” case is the same first transform followed by a MergeSortingTransform:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:522
 if (!skip_partial_sort || limit_)
 {
 pipeline.addSimpleTransform(
 [&](const SharedHeader & header, QueryPipelineBuilder::StreamType stream_type) -> ProcessorPtr
 {
 if (stream_type != sorted_stream_type)
 return nullptr;

 return std::make_shared<PartialSortingTransform>(header, result_sort_desc, limit_, threshold_tracker);
 });

Note the condition !skip_partial_sort || limit_. When there is a LIMIT, the partial sort is always applied, even if the input is already sorted, because the partial sort is what makes the limit cheap — it discards everything but the top limit rows of each chunk, so the merge only ever sees limit rows per chunk instead of the whole chunk.

`PartialSortingTransform`: sort the chunk, throw away the rest

The transform itself is short, and the interesting part is the threshold filter at the top:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:104
void PartialSortingTransform::transform(Chunk & chunk)
{
 if (chunk.getNumRows())
 {
 // The following code works with Blocks and will lose the number of
 // rows when there are no columns. We shouldn't get such block, because
 // we have to sort by at least one column.
 chassert(chunk.getNumColumns());
 }

 if (read_rows)
 read_rows->add(chunk.getNumRows());

 auto block = getInputPort().getHeader().cloneWithColumns(chunk.detachColumns());

 /// Materialize sort key columns that are ColumnReplicated to avoid index indirection during comparison and sorting.
 for (const auto & col_desc : description)
 {
 auto & column_entry = block.getByName(col_desc.column_name);
 column_entry.column = convertToFullColumnIfReplicationNotUseful(column_entry.column);
 }

 /** If we've saved columns from previously blocks we could filter all rows from current block
 * which are unnecessary for sortBlock(...) because they obviously won't be in the top LIMIT rows.
 */
 if (!sort_description_threshold_columns.empty() && !threshold_tracker)
 {
 UInt64 rows_num = block.rows();
 auto block_columns = extractRawColumns(block, description_with_positions);

 size_t result_size_hint = getFilterMask(
 block_columns, sort_description_threshold_columns,
 description, rows_num, filter, rows_to_compare, compare_results);

 /// Everything was filtered. Skip whole chunk.
 if (result_size_hint == 0)
 return;

 if (result_size_hint < rows_num)
 {
 for (auto & column : block)
 column.column = column.column->filter(filter, result_size_hint);
 }
 }

 sortBlock(block, description, limit);

Two things happen here, and the order matters.

First, the threshold filter. sort_description_threshold_columns holds the sort key of the current N-th best row seen so far — the “cutoff”. Every row in the new chunk that compares greater than the cutoff provably cannot be in the final top-N, so it is dropped before the sort. The filter is computed with a vectorised column comparison:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:27
size_t getFilterMask(const ColumnRawPtrs & raw_block_columns, const Columns & threshold_columns,
 const SortDescription & description, size_t num_rows, IColumn::Filter & filter,
 PaddedPODArray<UInt64> & rows_to_compare, PaddedPODArray<Int8> & compare_results)
{
 filter.resize(num_rows);
 compare_results.resize(num_rows);

 if (description.size() == 1)
 {
 /// Fast path for single column
 raw_block_columns[0]->compareColumn(*threshold_columns[0], 0, nullptr, compare_results,
 description[0].direction, description[0].nulls_direction);
 }
 else
 {
 rows_to_compare.resize(num_rows);
 iota(rows_to_compare.data(), num_rows, UInt64(0));

 size_t size = description.size();
 for (size_t i = 0; i < size; ++i)
 {
 raw_block_columns[i]->compareColumn(*threshold_columns[i], 0, &rows_to_compare, compare_results,
 description[i].direction, description[i].nulls_direction);

 if (rows_to_compare.empty())
 break;
 }
 }

 size_t result_size_hint = 0;

 for (size_t i = 0; i < num_rows; ++i)
 {
 /// Leave only rows that are less then row from rhs.
 filter[i] = compare_results[i] < 0;
 result_size_hint += filter[i];
 }

 return result_size_hint;
}

compareColumn is the columnar comparison primitive: it compares a whole column against a single-row column and writes a per-row sign into compare_results. For a single sort column it is one vectorised pass; for multiple columns it narrows the candidate set on each column in turn (the rows_to_compare array is the survivor list, and it is emptied when no rows survive).

Second, sortBlock(block, description, limit). The limit argument makes the sort itself partial: ClickHouse sorts the chunk but only materialises the first limit rows of the permutation.

 cpp
 
 
// src/Interpreters/sortBlock.cpp:384
void sortBlock(Block & block, const SortDescription & description, UInt64 limit, IColumn::PermutationSortStability stability)
{
 IColumn::Permutation permutation;

#ifndef NDEBUG
 block.checkNumberOfRows();
#endif
 getBlockSortPermutationImpl(block, description, stability, limit, permutation);

#ifndef NDEBUG
 checkSortedWithPermutation(block, description, limit, permutation);
#endif

 if (permutation.empty())
 return;

 bool is_identity_permutation = isIdentityPermutation(permutation, limit);
 if (is_identity_permutation && limit == 0)
 return;

 size_t output_rows = limit ? std::min(static_cast<size_t>(limit), permutation.size()) : permutation.size();
 Columns columns = block.getColumns();
 transformColumnsWithSharedIndex(
 columns,
 [&](const ColumnPtr & col) { return is_identity_permutation ? col->cut(0, output_rows) : col->permute(permutation, limit); });
 block.setColumns(columns);
}

Note the output_rows clamp: with a limit of 10 and a chunk of 100 000 rows, the output chunk has 10 rows. The sort still had to consider all 100 000 rows (the top 10 of an unsorted chunk are not the first 10), but the permutation only materialises 10 entries and the columns are permuted with that.

So the memory cost of ORDER BY ... LIMIT 10 is proportional to the chunk size, not to the limit. Each chunk is sorted in full, truncated to 10 rows, and the merge only ever sees 10 rows per chunk.

Updating the threshold

After sorting, the transform updates the threshold to the current chunk’s limit-th row:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:151
 /// Check if we can use this block for optimization.
 if ((min_limit_for_partial_sort_optimization <= limit || threshold_tracker) && limit <= block.rows())
 {
 /** If we filtered more than limit rows from block take block last row.
 * Otherwise take last limit row.
 *
 * If current threshold value is empty, update current threshold value.
 * If min block value is less than current threshold value, update current threshold value.
 */
 size_t min_row_to_compare = limit - 1;
 auto raw_block_columns = extractRawColumns(block, description_with_positions);

 if (sort_description_threshold_columns.empty() ||
 compareWithThreshold(raw_block_columns, min_row_to_compare, sort_description_threshold_columns, description))
 {
 size_t raw_block_columns_size = raw_block_columns.size();
 Columns sort_description_threshold_columns_updated(raw_block_columns_size);

 for (size_t i = 0; i < raw_block_columns_size; ++i)
 {
 MutableColumnPtr sort_description_threshold_column_updated = raw_block_columns[i]->cloneEmpty();
 sort_description_threshold_column_updated->insertFrom(*raw_block_columns[i], min_row_to_compare);
 /// Without this, `assertTypeEquality` may fail when comparing a ColumnNullable against a ColumnReplicated (received from JOIN).
 sort_description_threshold_columns_updated[i]
 = removeSpecialRepresentations(sort_description_threshold_column_updated->getPtr());
 }

 sort_description_threshold_columns = std::move(sort_description_threshold_columns_updated);

The threshold is monotonically non-increasing (it only ever gets better), so the filter gets more aggressive over time. The min_limit_for_partial_sort_optimization guard exists because for very small limits the bookkeeping is not worth it.

There is a subtlety worth noticing: previous_row_chunk style stability is not preserved by this filter. When a row ties with the threshold, the filter keeps it (compare_results[i] < 0 is false for equal, so equal rows are… let me be careful). compare_results[i] < 0 keeps rows that are strictly less than the threshold. So a row equal to the threshold is dropped. That is fine for ORDER BY ... LIMIT without WITH TIES, and it is why LimitTransform has a completely separate mechanism for WITH TIES.

`FinishSortingTransform`: merging streams that are already sorted

When the input is already sorted on a prefix of the key, the merge does not need a heap at all — it just has to concatenate runs. The one hard part is handling a key that spans a chunk boundary.

 cpp
 
 
// src/Processors/Transforms/FinishSortingTransform.cpp:83
 /// Find the position of last already read key in current chunk.
 if (!chunks.empty())
 {
 size_t size = chunk.getNumRows();
 const auto & last_chunk = chunks.back();

 ssize_t low = -1;
 ssize_t high = size;
 while (high - low > 1)
 {
 ssize_t mid = (low + high) / 2;
 if (!less(last_chunk.getColumns(), chunk.getColumns(), last_chunk.getNumRows() - 1, mid, description_with_positions))
 low = mid;
 else
 high = mid;
 }

 size_t tail_pos = high;

 /// We need to save tail of chunk, because next chunk may starts with the same key as in tail
 /// and we should sort these rows in one portion.
 if (tail_pos != size)
 {
 auto source_columns = chunk.detachColumns();
 Columns tail_columns;

 for (auto & source_column : source_columns)
 {
 tail_columns.emplace_back(source_column->cut(tail_pos, size - tail_pos));
 source_column = source_column->cut(0, tail_pos);
 }

 chunks.emplace_back(std::move(source_columns), tail_pos);
 tail_chunk.setColumns(std::move(tail_columns), size - tail_pos);

 stage = Stage::Generate;
 return;
 }
 }

The binary search finds the first row of the new chunk whose key is greater than the last key of the previous chunk. Everything from there on has a key that might tie with the tail, so it is cut off into tail_chunk and held back. The tail is re-merged with the next chunk, so that rows with equal keys end up in one sorted run rather than being split across two and appearing in the wrong order.

The stage = Stage::Generate is the switch that makes this work: the transform has to emit what it has so far, then go back to consuming.

`MergeSortingTransform`: accumulate, remerge, spill, merge

The general case is MergeSortingTransform, and it is a state machine with three stages — Consume, Serialize, Generate — declared in the base class:

 cpp
 
 
// src/Processors/Transforms/SortingTransform.h:126
 enum class Stage : uint8_t
 {
 Consume = 0,
 Generate,
 Serialize,
 };

 Stage stage = Stage::Consume;

The base class prepare() is the dispatcher:

 cpp
 
 
// src/Processors/Transforms/SortingTransform.cpp:293
IProcessor::Status SortingTransform::prepare()
{
 if (stage == Stage::Serialize)
 {
 if (!processors.empty())
 return Status::UpdatePipeline;

 auto status = prepareSerialize();
 if (status != Status::Finished)
 return status;

 stage = Stage::Consume;
 }

 if (stage == Stage::Consume)
 {
 auto status = prepareConsume();
 if (status != Status::Finished)
 return status;

 stage = Stage::Generate;
 }

 /// stage == Stage::Generate

 if (!generated_prefix || !chunks.empty())
 return Status::Ready;

 if (!processors.empty())
 return Status::UpdatePipeline;

 return prepareGenerate();
}

The Status::UpdatePipeline returns are the graph-rewriting mechanism from the first article: when the sort decides it needs to spill to disk, it returns UpdatePipeline, the executor asks it for updatePipeline(), and the graph is rewired to insert a file sink and a file source.

Consume: accumulate chunks

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:94
void MergeSortingTransform::consume(Chunk chunk)
{
 /** Algorithm:
 * - read to memory blocks from source stream;
 * - if too many of them and if external sorting is enabled,
 * - merge all blocks to sorted stream and write it to temporary file;
 * - at the end, merge all sorted streams from temporary files and also from rest of blocks in memory.
 */

 /// If there were only const columns in sort description, then there is no need to sort.
 /// Return the chunk as is.
 if (description.empty())
 {
 generated_chunk = std::move(chunk);
 return;
 }

 removeConstColumns(chunk);
 compactReplicatedColumns(chunk);

 sum_rows_in_blocks += chunk.getNumRows();
 sum_bytes_in_blocks += chunk.allocatedBytes();
 chunks.push_back(std::move(chunk));

 /** If significant amount of data was accumulated, perform preliminary merging step.
 */
 if ((chunks.size() > 1
 && limit
 && limit * 2 < sum_rows_in_blocks /// 2 is just a guess.
 && remerge_is_useful
 && max_bytes_before_remerge
 && sum_bytes_in_blocks > max_bytes_before_remerge) || (threshold_tracker && (static_cast<double>(sum_rows_in_blocks) > static_cast<double>(limit) * 1.5)))
 {
 remerge();
 }

 /** If too many of them and if external sorting is enabled,
 * will merge blocks that we have in memory at this moment and write merged stream to temporary (compressed) file.
 * NOTE. It's possible to check free space in filesystem.
 */
 if (max_bytes_in_block_before_external_sort && sum_bytes_in_blocks > max_bytes_in_block_before_external_sort)
 {
 Int64 query_memory = getCurrentQueryMemoryUsage();
 if (!max_bytes_in_query_before_external_sort || query_memory > static_cast<Int64>(max_bytes_in_query_before_external_sort))
 {
 auto sink = prepareSpill();
 auto source = std::make_shared<BufferingFromFileSource>(sink->getPort().getSharedHeader(), sink->getHolder(), log);
 processors.emplace_back(source);
 processors.emplace_back(sink);
 stage = Stage::Serialize;
 }
 }
}

Three behaviours in one function.

Remerge. When there is a LIMIT and enough data has accumulated, the in-memory chunks are merged now rather than later. This is the classic top-K trick: after merging, only limit rows survive, so the memory footprint of the accumulator collapses from “all data so far” to “the top limit rows”. The condition limit * 2 < sum_rows_in_blocks means “we have at least twice as many rows as we need”, so the remerge is worth it.

Spill. When the accumulated bytes exceed max_bytes_in_block_before_external_sort (and the whole query’s memory exceeds max_bytes_in_query_before_external_sort), the accumulated chunks are merged and written to a temporary file. The transform enters the Serialize stage, and the new processors — the file sink and the file source — are queued for updatePipeline().

The limit on remerge. remerge_is_useful is a latch:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:333
void MergeSortingTransform::remerge()
{
 LOG_DEBUG(log, "Re-merging intermediate ORDER BY data ({} blocks with {} rows) to save memory consumption", chunks.size(), sum_rows_in_blocks);

 /// NOTE Maybe concat all blocks and partial sort will be faster than merge?
 MergeSorter remerge_sorter(std::make_shared<const Block>(header_without_constants), std::move(chunks), description, max_merged_block_size, limit);

 Chunks new_chunks;
 size_t new_sum_rows_in_blocks = 0;
 size_t new_sum_bytes_in_blocks = 0;

 while (auto chunk = remerge_sorter.read())
 {
 new_sum_rows_in_blocks += chunk.getNumRows();
 new_sum_bytes_in_blocks += chunk.allocatedBytes();
 new_chunks.emplace_back(std::move(chunk));
 }

 LOG_DEBUG(log, "Memory usage is lowered from {} to {}", ReadableSize(sum_bytes_in_blocks), ReadableSize(new_sum_bytes_in_blocks));

 /// If the memory consumption was not lowered enough - we will not perform remerge anymore.
 if (remerge_lowered_memory_bytes_ratio > 0.0 && (static_cast<double>(new_sum_bytes_in_blocks) * remerge_lowered_memory_bytes_ratio > static_cast<double>(sum_bytes_in_blocks)))
 {
 remerge_is_useful = false;
 LOG_DEBUG(log, "Re-merging is not useful (memory usage was not lowered by remerge_sort_lowered_memory_bytes_ratio={})", remerge_lowered_memory_bytes_ratio);
 }

The default ratio is 2.0 (remerge_sort_lowered_memory_bytes_ratio), so the remerge must halve the memory to be considered worth doing. If it does not, the flag is cleared and no further remerges happen — because a remerge that does not reduce memory is pure loss: you paid for a merge and got nothing.

The default threshold is max_bytes_before_remerge_sort = 1000000000, one gigabyte.

Serialize and Generate

Serialize is the stage where the accumulated chunks are drained into the file sink, one sorted run at a time:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:252
void MergeSortingTransform::serialize()
{
 current_chunk = merge_sorter->read();
 if (!current_chunk)
 merge_sorter.reset();
 /// A read of `MergeSorter::Mode::MergeUniqueChunks` that consumes only duplicates returns no rows. The
 /// run gets no block for it, and the next call reads on.
 else if (!current_chunk.hasRows())
 current_chunk.clear();
}

Generate is where the final merge happens. If nothing was spilled, the answer is a plain in-memory merge of the accumulated chunks:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:263
void MergeSortingTransform::generate()
{
 if (!generated_prefix)
 {
 if (temporary_files_num == 0)
 {
 merge_sorter = std::make_unique<MergeSorter>(
 std::make_shared<const Block>(header_without_constants),
 std::move(chunks),
 description,
 max_merged_block_size,
 limit,
 merge_mode);
 }
 else
 {
 ProfileEvents::increment(external_merge_event);
 LOG_INFO(log, "There are {} temporary sorted parts to merge", temporary_files_num);

 processors.emplace_back(
 std::make_shared<MergeSorterSource>(
 std::make_shared<const Block>(header_without_constants),
 std::move(chunks),
 description,
 max_merged_block_size,
 limit,
 merge_mode));
 }

 generated_prefix = true;
 }

If there were temporary files, a MergeSorterSource is added to the pipeline and the graph is rewired so that the spilled runs (read back from disk) and the in-memory tail are merged together by a MergingSortedTransform — which is a proper k-way merge with a heap.

The k-way merge

MergingSortedTransform uses MergingSortedAlgorithm, which keeps one SortCursor per input stream in a heap and pops the smallest key. The interesting optimisation is the “totally less” shortcut:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:547
template <typename TSortingHeap>
IMergingAlgorithm::Status MergingSortedAlgorithm::mergeImpl(TSortingHeap & queue)
{
 /// Take rows in required order and put them into `merged_data`, while the rows are no more than `max_block_size`
 while (queue.isValid())
 {
 if (merged_data.hasEnoughRows())
 return Status(merged_data.pull());

 auto current = queue.current();

 if (current.impl->isLast() && current_inputs[current.impl->order].skip_last_row)
 {
 /// Get the next block from the corresponding source, if there is one.
 queue.removeTop();
 return Status(current.impl->order);
 }

 if (current.impl->isFirst()
 && !current_inputs[current.impl->order].skip_last_row /// Ignore optimization if last row should be skipped.
 && (queue.size() == 1
 || (queue.size() >= 2 && current.totallyLessOrEquals(queue.nextChild()))))
 {
 /** This is special optimization if current cursor is totally less than next cursor.
 * We want to insert current cursor chunk directly in merged data.
 *
 * First if merged_data is not empty we need to flush it.
 * We will get into the same condition on next mergeBatch call.
 *
 * Then we can insert chunk directly in merged data.
 */

 if (merged_data.mergedRows() != 0)
 return Status(merged_data.pull());

 /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl)
 size_t source_num = current.impl->order;

 auto chunk_num_rows = current_inputs[source_num].chunk.getNumRows();

 UInt64 total_merged_rows_after_insertion = merged_data.mergedRows() + chunk_num_rows;
 bool limit_reached = limit && total_merged_rows_after_insertion >= limit;

 if (limit && total_merged_rows_after_insertion > limit)
 chunk_num_rows -= total_merged_rows_after_insertion - limit;

 insertChunk(source_num);

 /// We will get the next block from the corresponding source, if there is one.
 queue.removeTop();

 auto status = Status(merged_data.pull(), limit_reached);

 if (!limit_reached)
 status.required_source = source_num;

 return status;
 }

totallyLessOrEquals asks: “is the last key of this stream less than or equal to the first key of the next-smallest stream?” If yes, then the entire chunk of the current stream comes before everything in the other streams, and it can be copied wholesale instead of row by row.

That turns the merge from O(rows log streams) comparisons into a memcpy for the common case. When the data is nearly sorted — which it usually is, because each run came from an already-sorted source — most chunks take the fast path.

Trade-offs

What this buys you.

The LIMIT optimisation is the big one. ORDER BY x LIMIT 10 over a hundred million rows costs: sort each chunk (in memory, chunk by chunk), truncate to 10 rows, merge 10 rows per chunk. Memory is O(chunk size × threads), not O(dataset). And the threshold filter improves on that further: once the cutoff is tight, whole chunks are discarded without being sorted at all — if (result_size_hint == 0) return;.

The remerge trick generalises the same idea beyond LIMIT. Even without a limit, merging periodically keeps the accumulator’s memory bounded and converts a “merge N runs at the end” problem into a series of smaller merges, which is also better for the page cache.

The external sort path is honest about the fact that ORDER BY is a blocking operation: if the data does not fit, it goes to disk, in sorted runs, and the final merge is a k-way merge over those runs. The runs are written with the configured temporary-files codec (usually LZ4), so the disk traffic is compressed.

What it costs.

Sorting is still the slowest thing in the query, and the partial sort does not help the general case: without a LIMIT, every chunk must be fully sorted, and the merge must touch every row. sortBlock allocates a permutation of the chunk’s row indices — 8 bytes per row — and the column permutation that follows is a random-access pass over the column data, which is cache-hostile for wide rows.

The threshold filter has a cost of its own: a full vectorised comparison of the chunk against the cutoff, even when it rejects nothing. For a LIMIT query where the data is not sorted, the filter will usually reject nothing on the first chunk and progressively more later, so the average benefit depends heavily on the data.

The remerge is a gamble. It costs a merge, and it only pays off if the merge actually reduces memory by the configured ratio. The remerge_is_useful latch means the first unhelpful remerge disables all future ones — but the first one was already paid for. And the threshold for even attempting a remerge is one gigabyte of accumulated data, so the memory spike before the first remerge is real.

Finally, the tail-splitting logic in FinishSortingTransform is subtle and easy to get wrong. If a key spans a chunk boundary and the tail is not held back, the rows with equal keys are emitted in two separate sorted runs and the final order among them is wrong. The binary search that finds the split point is O(log n) per chunk, which is cheap, but the reason it exists is not obvious from the code — which is why the comment is there.

How ClickHouse Skips Data: Granules, Marks, and Bit-Packed Indexes

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A SELECT on a MergeTree table is, mechanically, a scan of a set of files. The only reason it is not a full scan is that the files are chopped into pieces, and each piece carries enough metadata for the engine to prove that the piece cannot contain a matching row.

ClickHouse calls those pieces granules, the metadata is called a mark, and the mechanism that ties them together is the primary key index — one key value per granule, held in memory.

This is the single most important performance idea in MergeTree, and it is built out of three small, very specific tricks:

Granules are sized in bytes, not rows, so a granule of fat rows is not a granule of a million rows.
Marks are bit-packed in memory — three bytes per mark for an integer column, instead of the sixteen bytes a naive {offset, offset_in_block} pair would take.
The primary key index is trimmed: if a key column is nearly constant within a part, the key columns after it are not kept in memory at all, because they cannot narrow anything down.

Granules: sized in bytes, not rows

The classic description of MergeTree granularity is “8192 rows”, and that is the default value of the index_granularity setting. But that is only the cap. The actual mechanism computes the number of rows per granule from the size of the data being written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:115
size_t computeIndexGranularity(
 size_t rows,
 size_t bytes_uncompressed,
 size_t index_granularity_bytes,
 size_t fixed_index_granularity_rows,
 bool blocks_are_granules,
 bool can_use_adaptive_index_granularity)
{
 size_t index_granularity_for_block = 0;

 if (!can_use_adaptive_index_granularity)
 {
 index_granularity_for_block = fixed_index_granularity_rows;
 }
 else
 {
 if (blocks_are_granules)
 {
 index_granularity_for_block = rows;
 }
 else if (index_granularity_bytes != 0 && bytes_uncompressed >= index_granularity_bytes)
 {
 /// index_granularity_bytes == 0 disables adaptive sizing upstream, so this branch is
 /// unreachable with a zero divisor; the explicit check makes that visible to static analysis.
 size_t granules_in_block = bytes_uncompressed / index_granularity_bytes;
 index_granularity_for_block = rows / granules_in_block;
 }
 else
 {
 size_t size_of_row_in_bytes = std::max(bytes_uncompressed / rows, 1UL);
 index_granularity_for_block = index_granularity_bytes / size_of_row_in_bytes;
 }
 }

 /// We should be less or equal than fixed index granularity.
 /// But if block size is a granule size then do not adjust it.
 /// Granularity greater than fixed granularity might come from compact part.
 if (!blocks_are_granules)
 index_granularity_for_block = std::min(fixed_index_granularity_rows, index_granularity_for_block);

 /// Very rare case when index granularity bytes less than single row.
 if (index_granularity_for_block == 0)
 index_granularity_for_block = 1;

 return index_granularity_for_block;
}

Read the interesting branch — the else at the bottom. If the block is smaller than index_granularity_bytes, then

text

 rows_per_granule = index_granularity_bytes / average_row_size

That is: keep the granule a constant number of bytes. The default index_granularity_bytes is 10 MB, so a granule is 10 MB of data whether its rows are 10 bytes wide or 10 KB wide.

Why does this matter? Because a granule is the unit of skipping. If granules were a fixed 8192 rows, then a table of 1 KB rows would have 8 MB granules (fine) but a table of 100 KB rows would have 800 MB granules (useless — you cannot skip anything smaller than 800 MB). Sizing in bytes makes the granule a predictable, bounded amount of IO, and therefore a predictable, bounded amount of waste when you have to read a granule that does contain matches.

The price is that the row count of a granule is no longer constant, so it has to be remembered. The adaptive granularity object stores partial sums:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularityAdaptive.h:9
/// Class that stores adaptive index granularity.
/// Inside it contains vector of partial sums of rows after mark:
/// |-----|---|----|----|
/// | 5 | 8 | 12 | 16 |
class MergeTreeIndexGranularityAdaptive final : public MergeTreeIndexGranularity

One size_t per mark, and the row count of any mark is a subtraction. The alternative — a constant number of rows per granule — is available as use_const_adaptive_granularity, and its setting description explains the trade-off exactly:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:2174
 DECLARE(Bool, use_const_adaptive_granularity, false, R"(
Non-const adaptive granularity (the default, `index_granularity_bytes` is not `0`) keeps
granules at a constant size in bytes, so their row count varies: for every
written block, rows per granule is `index_granularity_bytes` divided by the
average size of a row in that block, capped at `index_granularity`. Because it differs
from granule to granule, the row count of every granule in the part has to be kept in
memory, 8 bytes each, which on large tables adds up to tens of gigabytes.

“Tens of gigabytes” — that is the cost of the byte-granularity guarantee, and it is the reason the constant-row variant exists at all. On a table with 100 million parts × 100 marks each, 8 bytes per mark is 80 GB of pure bookkeeping.

The marks themselves are appended as the block is written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:162
void fillIndexGranularityForWidePart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 index_granularity.appendMark(index_granularity_for_block);
}

A Wide part gets exactly index_granularity_for_block rows per mark. A Compact part, where all columns share one file, has to be more careful:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:169
void fillIndexGranularityForCompactPart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 {
 size_t rows_left_in_block = rows_in_block - current_row;

 /// Try to extend last granule if block is large enough
 /// or it isn't first in granule (index_offset != 0).
 if (rows_left_in_block < index_granularity_for_block &&
 (rows_in_block >= index_granularity_for_block || index_offset != 0))
 {
 // If enough rows are left, create a new granule. Otherwise, extend previous granule.
 // So, real size of granule differs from index_granularity_for_block not more than 50%.
 if (rows_left_in_block * 2 >= index_granularity_for_block)
 index_granularity.appendMark(rows_left_in_block);
 else
 index_granularity.addRowsToLastMark(rows_left_in_block);
 }
 else
 {
 index_granularity.appendMark(index_granularity_for_block);
 }
 }
}

The comment “real size of granule differs from index_granularity_for_block not more than 50%” is the invariant: a short tail is either given its own granule (if it is at least half a granule) or merged into the previous one.

Marks: where in the file does each granule start?

A granule is a range of rows. To read it, you need to know where in the compressed .bin file its rows live. That is the mark:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:14
/** Mark is the position in the compressed file. The compressed file consists of adjacent compressed blocks.
 * Mark is a tuple - the offset in the file to the start of the compressed block, the offset in the decompressed block to the start of the data.
 */
struct MarkInCompressedFile
{
 size_t offset_in_compressed_file;
 size_t offset_in_decompressed_block;

Two numbers, both size_t, so sixteen bytes per mark in the naive layout. For a part with 100 marks that is nothing; for a table with 100 000 parts of 1 GB each (100 TB), 100 marks per part × 16 bytes × 100 000 parts = 160 MB of index — and that is only if every mark of every column is resident.

Which is why the in-memory representation is not a struct array:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:30
/**
 * In-memory representation of an array of marks.
 *
 * Uses an ad-hoc compression scheme that decreases memory usage while allowing
 * random access in O(1) time.
 * This is independent from the marks *file* format, which may be uncompressed
 * or use a different compression method.
 *
 * Typical memory usage:
 * * ~3 bytes/mark for integer columns
 * * ~5 bytes/mark for string columns
 * * ~0.3 bytes/mark for trivial marks in auxiliary dict files of LowCardinality columns
 */
class MarksInCompressedFile

Three bytes per mark. How?

The trick is that marks are not random. The comment in the header spells out the regularities the compression exploits:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:51
 /** We need to store a sequence of marks, each consisting of two 64-bit integers:
 * offset_in_compressed_file and offset_in_decompressed_block. We'll call them x and y for
 * convenience, since compression doesn't care what they mean. The compression exploits the
 * following regularities:
 * * y is usually zero.
 * * x usually increases steadily.
 * * Differences between x values in nearby marks usually fit in much fewer than 64 bits.
 *
 * We split the sequence of marks into blocks, each containing MARKS_PER_BLOCK marks.
 * (Not to be confused with data blocks.)
 * For each mark, we store the difference [value] - [min value in the block], for each of the
 * two values in the mark. Each block specifies the number of bits to use for these differences
 * for all marks in this block.
 * The smaller the blocks the fewer bits are required, but the bigger the relative overhead of
 * block headers.
 *
 * Packed marks and block headers all live in one contiguous array.
 */

So: 256 marks per block; per block, store the minimum x and y, and the number of bits needed to represent the spread within the block; then store every mark as a delta from the block minimum, in that many bits.

The block header computation:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:122
void MarksInCompressedFile::Builder::flushBlock(const MarkInCompressedFile * data, size_t count)
{
 chassert(count > 0 && count <= MARKS_PER_BLOCK);

 size_t block_idx = marks_flushed / MARKS_PER_BLOCK;
 BlockInfo & block = blocks[block_idx];
 block.bit_offset_in_packed_array = packed_bits;

 /// Compute block metadata: min values, bit widths.
 size_t max_x = 0;
 size_t max_y = 0;
 for (size_t i = 0; i < count; ++i)
 {
 block.min_x = std::min(block.min_x, data[i].offset_in_compressed_file);
 max_x = std::max(max_x, data[i].offset_in_compressed_file);
 block.min_y = std::min(block.min_y, data[i].offset_in_decompressed_block);
 max_y = std::max(max_y, data[i].offset_in_decompressed_block);
 block.trailing_zero_bits_in_y
 = std::min(block.trailing_zero_bits_in_y, static_cast<UInt8>(getTrailingZeroBits(data[i].offset_in_decompressed_block)));
 }

 block.bits_for_x = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits(max_x - block.min_x));
 block.bits_for_y
 = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits((max_y - block.min_y) >> block.trailing_zero_bits_in_y));

Two ideas in six lines.

Delta encoding per block. bits_for_x is the number of bits needed to represent the range of x values inside the block, not the magnitude of x itself. If a block of 256 marks spans 64 KB of compressed file, that is 19 bits per mark, not 64.

Trailing-zero stripping for y. y is the offset inside the decompressed block, and it is usually zero — a granule normally starts at the beginning of a compressed block. The code computes the number of trailing zero bits common to all the y values in the block and shifts them all right by that amount before packing. For a block where every y is zero, the shift is 63 and bits_for_y becomes zero: the y coordinate costs no bits at all.

The write path, showing both encodings:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:153
 /// Write bit-packed deltas.
 size_t bit_offset = packed_bits;
 for (size_t i = 0; i < count; ++i)
 {
 writeBitsPacked64(packed.data(), bit_offset, data[i].offset_in_compressed_file - block.min_x);
 writeBitsPacked64(
 packed.data(),
 bit_offset + block.bits_for_x,
 (data[i].offset_in_decompressed_block - block.min_y) >> block.trailing_zero_bits_in_y);
 bit_offset += block.bits_for_x + block.bits_for_y;
 }

And the read path, which has to be O(1) — a mark lookup happens for every granule the reader touches:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:47
std::tuple<const MarksInCompressedFile::BlockInfo *, size_t> MarksInCompressedFile::lookUpMark(size_t idx) const
{
 size_t block_idx = idx / MARKS_PER_BLOCK;
 const BlockInfo & block = blocks[block_idx];
 size_t offset = block.bit_offset_in_packed_array + (idx - block_idx * MARKS_PER_BLOCK) * (block.bits_for_x + block.bits_for_y);
 return {&block, offset};
}

Divide by 256 to find the block header, then a shift-and-mask to find the mark inside it. That is the whole random-access structure: one array division and one multiply. Compare that to a compressed list of marks, which would need a binary search.

Where do the marks come from at write time?

The mark is written when the compressed block is flushed, so it is simply the state of the hashing/compression stream at that moment:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:351
 MarkInCompressedFile mark{stream->plain_hashing.count(), stream->compressed_hashing.offset()};

plain_hashing.count() is the number of uncompressed bytes written so far — that is y, the offset inside the decompressed block. And compressed_hashing.offset() is the number of compressed bytes written so far — that is x, the offset of the compressed block in the file.

The mark cache

Marks are read from .mrk files and cached in a global LRU keyed by the file path:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeMarksLoader.cpp:313
 if (mark_cache)
 {
 auto key = MarkCache::hash(data_part_storage->getDiskName() + ":" + (fs::path(data_part_storage->getFullPath()) / mrk_path).string());

 if (save_marks_in_cache)
 {
 auto callback = [this] { return loadMarksImpl(); };
 loaded_marks = mark_cache->getOrSet(key, callback);
 }
 else
 {
 loaded_marks = mark_cache->getWithoutMetrics(key);
 if (!loaded_marks)
 loaded_marks = loadMarksImpl();
 }
 }

Note the two paths. getOrSet records hits and misses and inserts the loaded marks into the cache. getWithoutMetrics only looks: it is used when the marks are wanted for a merge or a mutation, which should not evict the query’s working set nor pollute the cache statistics. This is a small but real piece of cache hygiene — background work must not steal the cache from foreground queries.

The cache itself weights each entry by its compressed size plus a fixed overhead:

 cpp
 
 
// src/Storages/MarkCache.h:23
/// Estimate of number of bytes in cache for marks.
struct MarksWeightFunction
{
 /// We spent additional bytes on key in hashmap, linked lists, shared pointers, etc ...
 static constexpr size_t MARK_CACHE_OVERHEAD = 128;

 size_t operator()(const MarksInCompressedFile & marks) const
 {
 return marks.approximateMemoryUsage() + MARK_CACHE_OVERHEAD;
 }
};

The 128-byte constant is the per-entry overhead of the hash map node, the list node, the shared pointer, and the MarksInCompressedFile object itself. For a part with 100 marks, the compressed marks are ~300 bytes and the overhead is 128 — so the accounting is honest about the fact that the bookkeeping dominates for small parts.

The primary key index: one key value per granule

The marks tell you where the bytes are. The primary key index tells you what is in them. It is a separate file, primary.idx, containing the sorting key values of the first row of every granule:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1935
std::shared_ptr<IMergeTreeDataPart::Index> IMergeTreeDataPart::loadIndex() const
{
 auto component_guard = Coordination::setCurrentComponent("IMergeTreeDataPart::loadIndex");
 /// Memory for index must not be accounted as memory usage for query, because it belongs to a table.
 MemoryTrackerBlockerInThread temporarily_disable_memory_tracker;

and the loop that reads it is the whole format — marks_count rows of key_size values, serialized one after another:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1972
 FormatSettings format_settings;
 for (size_t i = 0; i < marks_count; ++i)
 {
 for (size_t j = 0; j < key_size; ++j)
 key_serializations[j]->deserializeBinary(*loaded_index[j], *index_file, format_settings);
 }

marks_count rows of key_size values, serialized one after another. The in-memory form is a set of ColumnPtrs — one column per key column, each holding marks_count values — which is exactly the layout a KeyCondition wants to evaluate a predicate against.

The trimming trick

Now the interesting part. The primary key may be (a, b, c). If a is almost constant within a part — say, a low-cardinality column — then keeping b and c in memory is a waste: a query filtering on b cannot be answered from the index anyway, because a does not narrow anything down. So the index is trimmed:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1895
void IMergeTreeDataPart::optimizeIndexColumns(size_t marks_count, Columns & index_columns) const
{
 if (marks_count == 0)
 {
 chassert(isEmpty());
 return;
 }

 /// Do not optimize index for patch parts because patch parts
 /// use manual index analysis which requires all index columns.
 if (info.isPatch())
 return;

 size_t key_size = index_columns.size();
 Float64 ratio_to_drop_suffix_columns = static_cast<double>((*storage.getSettings())[MergeTreeSetting::primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns]);

 /// Cut useless suffix columns, if necessary.
 if (key_size > 1 && ratio_to_drop_suffix_columns > 0 && ratio_to_drop_suffix_columns < 1)
 {
 for (size_t j = 0; j < key_size - 1; ++j)
 {
 size_t num_changes = 0;
 for (size_t i = 1; i < marks_count; ++i)
 {
 if (0 != index_columns[j]->compareAt(i, i - 1, *index_columns[j], 0))
 ++num_changes;
 }

 if (static_cast<Float64>(num_changes) / static_cast<Float64>(marks_count) >= ratio_to_drop_suffix_columns)
 {
 key_size = j + 1;
 index_columns.resize(key_size);
 break;
 }
 }

The rule: while the fraction of granules in which key column j changes is at least primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns (default 0.9), column j is informative, so keep it and move on. The first column that changes in fewer than 90% of granules is “nearly constant” within this part — the part is probably all one value of it — and then everything after it is dropped.

This is a per-part decision. A part that covers one day of data might have a constant country column and a varying user_id; the index keeps country and drops user_id. Another part, spanning many countries, keeps both. The consequence is that index analysis must know which columns are actually available — and the code accounts for it, treating the missing suffix columns as “constant over the whole universe”, which relaxes the analysis to a safe over-approximation.

The primary index cache

For very large tables, keeping every part’s primary index in memory is not possible, so it can be cached in an LRU instead:

 cpp
 
 
// src/Storages/MergeTree/PrimaryIndexCache.h:29
/** Cache of primary index for MergeTree tables.
 * Primary index is a list of columns from primary key
 * that store first row for each granule of data part.
 */
class PrimaryIndexCache : public CacheBase<UInt128, PrimaryIndex, UInt128TrivialHash, PrimaryIndexWeightFunction>

and the part consults it lazily:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:706
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::getIndex() const
{
 std::scoped_lock lock(index_mutex);

 if (index)
 return index;

 if (auto index_cache = storage.getPrimaryIndexCache())
 return loadIndexToCache(*index_cache);

 index = loadIndex();
 return index;
}

with the key being the disk name plus the part’s path, so that two parts with the same name on different disks do not collide:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:726
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::loadIndexToCache(PrimaryIndexCache & index_cache) const
{
 auto key = PrimaryIndexCache::hash(getDataPartStorage().getDiskName() + ":" + getRelativePathOfActivePart());
 auto callback = [this] { return loadIndex(); };
 return index_cache.getOrSet(key, callback);
}

Now the payoff: two different search algorithms

With marks and the primary index in hand, a query has to answer “which granules can contain a matching row?”. ClickHouse has two algorithms for this, and it picks between them based on the shape of the predicate.

The decision point is whether the predicate describes a continuous range of the sorting key:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2705
 bool key_condition_exact_range = !key_condition_useful || key_condition.matchesExactContinuousRange();
 bool part_offset_condition_exact_range = !part_offset_condition_useful || part_offset_condition->matchesExactContinuousRange();
 bool total_offset_condition_exact_range = !total_offset_condition_useful || total_offset_condition->matchesExactContinuousRange();
 const String & part_name = part->isProjectionPart() ? fmt::format("{}.{}", part->name, part->getParentPart()->name) : part->name;

 if (!key_condition_exact_range || !part_offset_condition_exact_range || !total_offset_condition_exact_range)
 {
 // Do exclusion search, where we drop ranges that do not match

Algorithm 1: binary search (continuous range)

If the predicate describes a single continuous interval of the sorting key — WHERE key >= 100 AND key < 200 — then the matching granules form a contiguous range in the index, and two binary searches find its boundaries:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2801
 /// Invariant: !check_in_range(part_range.begin..searched_left).can_be_true
 /// check_in_range(part_range.begin..searched_right).can_be_true
 /// (part_range.end + 1 is a sentinel out-of-bounds index, such that by definition
 /// check_in_range(x..part_range.end+1).can_be_true is true. The binary search will never
 /// actually call check_in_range with this out-of-bounds index.)
 /// If the condition is not true anywhere, we'll end up with searched_left == part_range.end.
 size_t searched_left = part_range.begin;
 size_t searched_right = part_range.end + 1;

 while (searched_left + 1 < searched_right)
 {
 const size_t middle = (searched_left + searched_right) / 2;
 MarkRange range(0, middle);
 if (check_in_range(range, BoolMask::consider_only_can_be_true).can_be_true)
 searched_right = middle;
 else
 searched_left = middle;
 ++steps;
 }
 result_range.begin = searched_left;
 LOG_TEST(log, "Found (LEFT) boundary mark: {}", searched_left);

check_in_range(0, middle) asks: “could any row in granules [0, middle) match?” The KeyCondition evaluates the predicate against the index values at the two ends of the range and the monotonicity of the key columns. Because the key is sorted, the answer is monotone in middle, so binary search is valid. O(log n) index checks for the whole part.

Algorithm 2: generic exclusion search (everything else)

Now consider WHERE key2 = 5 where key2 is the second column of the sorting key, or WHERE key = 5 OR key = 900000. The matching granules are not contiguous. A binary search cannot be used. ClickHouse falls back to an exclusion search: split the mark range recursively, drop the parts that cannot match.

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:61
void searchUnlimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 for (const auto & initial_range : initial_ranges)
 {
 std::vector<MarkRange> ranges_stack = {initial_range};
 while (!ranges_stack.empty())
 {
 MarkRange range = ranges_stack.back();
 ranges_stack.pop_back();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 appendWithMaxGap(result.ranges, range, search_settings.min_marks_for_seek);

 /// Unlike `ranges`, exact ranges must never absorb the gap between two accepted
 /// ranges: every mark of an exact range has to fully match the condition, while the
 /// skipped marks in between do not.
 if (collect_exact_ranges && exact)
 appendWithMaxGap(result.exact_ranges, range, 0);
 }
 else
 {
 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { ranges_stack.push_back(subrange); });
 }
 }
 }
}

This is a depth-first search over mark ranges. The check returns a BoolMask with two bits: can_be_true and can_be_false.

!can_be_true → the whole range cannot match → drop it.
!can_be_false → every row in the range matches → keep it whole.
otherwise → the range is a mixture → split it into merge_tree_coarse_index_granularity (default 8) subranges and recurse.

Splitting into 8 rather than 2 is deliberate: it is a breadth-ish compromise. Splitting in half is the classic binary split, but each split costs one check_in_range call, and each check is not free — it evaluates a KeyCondition against the index columns at the range boundaries. Eight subranges per step means the search converges in about log_8(n) steps while doing 8× the work per step.

The step budget

The unlimited search has no bound: a pathological predicate could make it check thousands of ranges. So there is a budget:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:119
void searchLimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 std::vector<MarkRange> heap_storage(initial_ranges.begin(), initial_ranges.end());
 std::priority_queue<MarkRange, std::vector<MarkRange>, LessBySize> queue(LessBySize{}, std::move(heap_storage));

 /// Ranges are accepted out of spatial order, so they are collected raw and merged after sorting.
 std::vector<MarkRange> accepted;
 std::vector<MarkRange> accepted_exact;

 while (!queue.empty())
 {
 MarkRange range = queue.top();
 queue.pop();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 accepted.push_back(range);
 if (collect_exact_ranges && exact)
 accepted_exact.push_back(range);
 continue;
 }

 if (result.num_steps + queue.size() > search_settings.max_steps)
 {
 result.reached_step_limit = true;
 accepted.push_back(range);
 continue;
 }

 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { queue.push(subrange); });
 }

Two things change relative to the unlimited version.

A priority queue, largest range first. The budget is spent where it buys the most: splitting a huge range is what removes the most granules, so the biggest range is examined first. The comparator is explicit:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:102
/// Orders the queue of the step-limited search: the largest range comes first.
struct LessBySize
{
 bool operator()(const MarkRange & lhs, const MarkRange & rhs) const
 {
 size_t lhs_size = lhs.end - lhs.begin;
 size_t rhs_size = rhs.end - rhs.begin;
 return std::tie(lhs_size, rhs.begin) < std::tie(rhs_size, lhs.begin);
 }
};

When the budget runs out, the range is accepted whole. This is the important bit: the search never returns an incorrect answer. When result.num_steps + queue.size() > max_steps, the range is pushed into accepted as-is — meaning “we do not know, so read it”. The result is a superset of the true matching granules, so the query is correct; it just reads more data than an unlimited search would have.

That is what the setting says:

 cpp
 
 
// src/Core/Settings.cpp:1936
 DECLARE(UInt64, merge_tree_generic_exclusion_search_max_steps, 0, R"(
When a filter cannot be evaluated as a single continuous range of the primary key, for example when it uses key columns other than the first one, ClickHouse runs an iterative generic exclusion search algorithm over the index marks. The same algorithm is used for the analysis of the text index. This setting limits the number of steps (index checks) the algorithm spends on each data part.

The budget is spent on the largest remaining mark ranges first. When it is exhausted, the ranges that were not fully analyzed are accepted as a whole, so the query stays correct but may read more granules than an unlimited search would select. A lower budget speeds up index analysis at the cost of reading more data. The limit is approximate rather than a strict cap on the analysis cost: the search can exceed it by roughly one round of splitting, and when the part is already divided into many ranges (for example, by the query condition cache), each of them is checked at least once regardless of the limit.

“The default (0) means unlimited steps.” So the default behaviour is exact and unbounded; the budget is an escape hatch for pathological cases.

The seek heuristic

One more knob, and it is about IO, not about correctness. When two accepted ranges are separated by a small gap of marks, it is cheaper to read the gap sequentially than to seek past it: appendWithMaxGap merges the two ranges when the gap is at most max_gap marks wide.

The gap threshold comes from merge_tree_min_rows_for_seek and merge_tree_min_bytes_for_seek, converted into marks by the granularity of the part. Both default to 0, i.e. “never bridge a gap” — but on a spinning disk or an object store with a high per-request cost, setting them turns “read 8 KB from here, seek, read 8 KB from there” into “read 40 KB contiguously”.

Note the asymmetry that the comment points out: exact_ranges are merged with max_gap = 0, always. An exact range is a range where every row matches; bridging a gap into it would claim that the skipped marks match too, which is false. So the gap-bridging optimization applies only to the “may match” ranges, never to the “definitely matches” ones.

Putting it together

 text
 
 
 part on disk
 ┌───────────────────────────────────────────────────────────────┐
 │ granule 0 granule 1 granule 2 granule 3 │
 │ rows 0..8191 rows 8192..16383 rows ... rows ... │
 └───────────────────────────────────────────────────────────────┘
 │ │ │ │
 ▼ ▼ ▼ ▼
 marks: (x0,y0) (x1,y1) (x2,y2) (x3,y3)
 bit-packed, ~3 bytes each, one per column file
 │
 ▼
 primary.idx: key values of the FIRST ROW of each granule
 │
 ▼
 KeyCondition.checkInRange(...)
 │
 ├── predicate is a continuous key range → binary search (2·log₂ n checks)
 │
 └── predicate is anything else → exclusion search (recursive, budgeted)
 │
 ▼
 MarkRanges: the granules that must be read

What this buys you, and what it costs

It buys you a scan that is proportional to the data you asked for, not to the data you stored. A SELECT with a range predicate on a sorted key reads roughly the granules that overlap the range, plus one granule of granularity on each side.

It buys you an index that fits in memory. Three bytes per mark, and the primary key index trimmed to the columns that can actually narrow things down. Without those two tricks, a 10 TB table would need tens of gigabytes of index.

It costs you a fixed granularity of waste. A predicate that matches a single row still reads a whole granule — 10 MB by default. This is the central trade of the whole design, and it is why index_granularity_bytes is a setting you may want to lower for a table with a very selective predicate and small rows.

It costs you an in-memory structure that has to be built and kept. The primary index is loaded per part (or cached), and the marks are loaded per column file that the query touches. Both are cached, and both have their own LRU, but the first query against a cold part pays for them.

It costs you complexity in the analysis. Two algorithms, a budget, a trimming heuristic that makes the analysis conservative in some cases, and the requirement that the analysis be sound (never skip a granule that could match) even when it is not complete.

Seeing it in action

The profile events tell you which algorithm ran:

 sql
 
 
SELECT event, value
FROM system.events
WHERE event IN (
 'IndexBinarySearchAlgorithm',
 'IndexGenericExclusionSearchAlgorithm',
 'IndexGenericExclusionSearchStepLimitReached',
 'FilteringMarksWithPrimaryKeyProcessedMarks',
 'FilteringMarksWithPrimaryKey'
);

FilteringMarksWithPrimaryKeyProcessedMarks is the number of marks examined; FilteringMarksWithPrimaryKey is the number of parts where the primary key was used at all. The ratio between the two, and the ratio of marks read to marks examined, is the “how selective is my index” metric.

And the parts themselves report the granularity they were written with:

 sql
 
 
SELECT
 name,
 rows,
 marks,
 round(rows / marks) AS rows_per_mark,
 formatReadableSize(bytes_on_disk) AS size
FROM system.parts
WHERE active AND rows > 0
ORDER BY marks DESC
LIMIT 10;

A rows_per_mark far below 8192 means the table has fat rows and the byte-based granularity kicked in. A rows_per_mark of exactly 8192 means use_const_adaptive_granularity is on, or index_granularity_bytes was 0 when the part was written.

How ClickHouse Plans Distributed Queries with Cascades

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A single-node query plan has one obvious shape: read the table, filter, aggregate, sort. A distributed query does not. The same SELECT can be executed as a gather everything to the coordinator, or as a shuffle of both sides of a join followed by a local aggregation, or as a broadcast of a tiny dimension table to every node followed by a local join. These choices interact — the best join strategy depends on how the data will be aggregated afterwards, and the best aggregation strategy depends on how the join output is distributed — so the planner has to explore combinations and compare them.

ClickHouse’s traditional answer was a set of fixed heuristics. Its new answer, arriving in 26.8 as an experimental setting, is a real cost-based optimizer built on the Cascades framework: a memo of alternative plans, a set of rules that generate alternatives, and a cost model that prices them.

 cpp
 
 
// src/Core/Settings.cpp:11131
 DECLARE(Bool, enable_cascades_optimizer, false, R"(
Enable the Cascades cost-based optimizer for distributed query plans.
Takes effect only together with `make_distributed_plan = 1`: the setting alone does not change single-node query planning.
)", EXPERIMENTAL, \
 {"26.8", false, false, "New experimental setting."}) \

This article walks the implementation: the memo, the task-based search, the rules, the cost model, the enforcers, and the places where the implementation is deliberately conservative.

The problem the optimizer solves

The ARCHITECTURE.md that ships with the code states it plainly:

 text
 
 
## What Problem Does This Solve?

ClickHouse can execute a query on a single node using its standard pipeline: plan the
query, read from MergeTree, apply filters/expressions/joins/aggregations, return results.
But when data is spread across multiple nodes (shared storage like S3, or a multi-node
cluster), the optimizer must decide how to distribute the work:

- Should a join shuffle both sides by the join key, or broadcast the smaller table to
 all nodes?
- Should an aggregation happen on each node first (partial agg) and then merge results,
 or gather all data to one node and aggregate there?
- Should a table scan be split across N nodes (parallel read), or should each node read
 the full table (replicated read for small dimension tables)?
- Where should exchanges (network data transfers between nodes) be placed?

Note the third bullet: with shared storage (S3), a 25-row dimension table does not need to be broadcast at all — every node can just read it from S3 itself. That alternative, ReplicatedRead, is the kind of thing that only a cost-based search would find, because it is not a “join strategy” in the classical sense; it is a read strategy chosen because the cost of re-reading a tiny table is lower than the cost of shipping it.

The memo

The memo is the central data structure. It contains groups, each representing a set of logically equivalent subexpressions, and each group holds both the logical expressions (ways of computing the same thing) and the physical expressions (ways of computing it with a concrete algorithm and concrete properties).

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Memo.h:10
class Memo
{
public:
 explicit Memo(LoggerPtr log_)
 : log(log_)
 {}

 GroupId addGroup(GroupExpressionPtr group_expression);

 GroupPtr getGroup(GroupId group_id);
 GroupConstPtr getGroup(GroupId group_id) const;

 size_t getGroupCount() const { return groups_by_id.size(); }

The plan is ingested into the memo one group per step. A step with children becomes a group whose expression has one input link per child, and the input link carries the required properties of that child:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:179
 auto group_expression = std::make_shared<GroupExpression>(std::move(node.step));
 auto group_id = memo.addGroup(group_expression);
 for (auto * child_node : node.children)
 {
 auto [input_group_id, pending_props] = addGroup(*child_node);
 group_expression->inputs.push_back({input_group_id, pending_props});
 }

Each input link therefore carries the child’s group id and the properties the parent requires of it — which is where the “required properties” of the whole framework enter the picture.

Properties: distribution and sorting

A physical expression has two kinds of properties.

Distribution — how the data is spread over the nodes:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.h:17
struct DistributionDescription
{
 DistributionColumns columns; /// Columns by which data is distributed. E.g. for shuffle exchange or partitioned table scan.
 /// Type each key is cast to before hashing (parallel to `columns`; empty when the raw
 /// types agree). Aligns buckets across both sides of a shuffle join.
 Names hash_type_names;
 bool is_replicated = false; /// All data is replicated to all nodes, so any distribution is satisfied. E.g. for small tables that are broadcasted to all nodes.
 size_t node_count = 1; /// Number of nodes among which data is distributed. E.g. for a shuffle exchange or a partitioned read.

 bool operator==(const DistributionDescription & other) const = default;

 void dump(WriteBuffer & out) const;
 String dump() const;
};

Sorting — whether the output is ordered, and by what.

A parent requires properties of its child, and a child satisfies a requirement if it provides at least as much. The matching is not trivial: a sorted-by-(a) stream satisfies a requirement of “sorted by (a)”, and the required sort must be a prefix of the existing one:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.cpp:9
bool ExpressionProperties::isSortingSatisfiedBy(const SortDescription & required, const SortDescription & existing)
{
 /// Required sorting is satisfied by existing sorting if required is the prefix of existing
 if (required.size() > existing.size())
 return false;
 for (size_t i = 0; i < required.size(); ++i)
 {
 if (required[i] != existing[i])
 return false;
 }
 return true;
}

and the distribution match is positional — data shuffled by (b, a) does not colocate with data shuffled by (a, b):

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.cpp:39
 /// The partition hash chains the key columns in order and may cast them first, so the
 /// match must be positional and the hash types must agree: data shuffled by `(b, a)`
 /// or hashed at a different type does not colocate with data shuffled by `(a, b)`.
 if (required.hash_type_names != existing.hash_type_names)
 return false;

Sorting is a property, not an operator

One design decision is worth calling out, because it differs from classical Cascades. A plain ORDER BY (no LIMIT) is not an operator in the memo — it is a property that the parent requires of its input. The optimizer then produces the sort itself, if some child can provide it more cheaply:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:162
 /// Strip a limit-less SortingStep::Full - pure sorting is a physical property, not a logical
 /// one. Return the child's GroupId along with the sorting as required properties, so the
 /// caller can attach them to the input link of the parent group expression.
 /// A SortingStep with a limit is a top-N (row-reducing) operator, so it is kept as an
 /// operator instead; the limit is owned by that operator, never by the sorting property.
 const auto * sorting_step = typeid_cast<const SortingStep *>(node.step.get());
 if (sorting_step && sorting_step->getType() == SortingStep::Type::Full && sorting_step->getLimit() == 0)
 {
 chassert(node.children.size() == 1);
 auto [child_group_id, _] = addGroup(*node.children.front());
 ExpressionProperties stripped_props;
 stripped_props.sorting = sorting_step->getSortDescription();
 return {child_group_id, stripped_props};
 }

A LIMIT after the sort makes it a top-N, which reduces the row count and therefore is an operator with its own rules and its own cost.

The search: tasks on a stack

The optimizer is a LIFO stack of tasks. There are six task types, and a group moves through four stages.

Stage 1 — Explore. Transformation rules generate logically equivalent alternatives (join commutativity, two-stage aggregation).

Stage 2 — Implement. Implementation rules turn logical expressions into physical ones with concrete properties (local join, broadcast join, shuffle join; local read, parallel read, replicated read).

Stage 3 — Enforce. Enforcer rules bridge property gaps by inserting exchange steps.

Stage 4 — Done.

OptimizeGroupTask drives the stages:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:38
 if (!group->isExplored())
 {
 /// Explore the group, then re-run `OptimizeGroupTask`
 optimizer.pushTask(std::make_unique<OptimizeGroupTask>(group_id, required_properties));
 optimizer.pushTask(std::make_unique<ExploreGroupTask>(group_id));
 }
 else if (!group->isOptimizedFor(required_properties))
 {
 optimizer.pushTask(std::make_unique<OptimizeGroupTask>(group_id, required_properties));

 for (auto & expression : group->logical_expressions)
 optimizer.pushTask(std::make_unique<OptimizeExpressionTask>(expression, required_properties));

 group->setOptimizedFor(required_properties);
 }

The interesting part is the pruning decision at the top of the task, which is explicitly not a cost-based prune:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:20
 /// Skip this group only if it is already fully processed (explored + implemented +
 /// enforced) for these properties and has a satisfying plan - re-running would be a no-op.
 /// We deliberately do not prune just because a current best is within a finite cost budget:
 /// the budget is an upper bound, not a lower bound, so such pruning is unsound for
 /// optimality, and it can return before stage-3 enforcers add the distributed alternatives.
 {
 const auto & cost_config = optimizer.getMemo().getContext().cost_config;
 bool group_fully_processed = group->isExplored()
 && group->isOptimizedFor(required_properties)
 && group->isEnforcedFor(required_properties);
 if (group_fully_processed && group->getBestImplementation(required_properties, cost_config).expression)
 {
 group->setFullyDoneFor(required_properties);
 LOG_TEST(optimizer.log, "OptimizeGroupTask group #{}: already fully processed", group_id);
 return;
 }
 }

This is a really instructive piece of engineering. Classical Cascades prunes with branch-and-bound: once a subtree’s cost exceeds the best complete plan, it is dropped. ClickHouse’s implementation refuses to do that, because a subtree’s current best is an upper bound on its final cost — a cheaper enforcer-built alternative may still appear — and pruning on it would be unsound. Instead, the search is bounded by a task count.

Rule ordering

Every rule has a static promise (a priority). The applicable rules are sorted by promise and pushed on the LIFO stack, so the highest-promise rule runs first:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:169
 std::vector<std::pair<Promise, OptimizationRulePtr>> moves;
 for (const auto & rule : rules)
 {
 if (!expression->isApplied(*rule, required_properties) && rule->checkPattern(expression, required_properties, optimizer.getMemo()))
 moves.push_back({rule->getPromise(), rule});
 }

 /// Sort ascending so the LIFO task stack pops the highest-promise rule first.
 std::sort(moves.begin(), moves.end(), [](const auto & lhs, const auto & rhs) { return lhs.first < rhs.first; });

The promise does not change which plans are considered — every applicable rule is eventually applied — it only changes the order, which breaks ties among equal-cost alternatives (the first-found best wins).

The cost model

The cost has three dimensions, combined by configurable weights:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.h:39
struct CostConfig
{
 Float64 work_weight = 1.0; /// Parallelizable work (scans, expression eval, I/O).
 Float64 network_weight = 1.0; /// Per-byte network transfer.
 /// Single-threaded phases (gather/scatter funnels, merge cursors). The weight is the
 /// per-node thread count: work parallelizes across the threads of a node while a serial
 /// phase holds one thread, so one serial row costs about `threads` work rows (Brent's law:
 /// wall-clock ~ work / threads + serial path).
 Float64 sequential_weight = 32.0;
 /// Fixed per-exchange latency (connection setup, metadata), in sequential rows. The
 /// weighted value (~1e5 work units) keeps a plan over a small input local: distribution
 /// cannot pay for its setup there.
 Float64 exchange_fixed_overhead = 3000.0;

and the combination:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.h:74
 Float64 total(const CostConfig & config) const
 {
 /// If any component is infinite the plan is impossible; return infinity directly. Multiplying
 /// first would let a zero weight turn `inf * 0` into NaN, and a NaN cost compares as neither
 /// better nor worse, so an impossible plan could be picked as best.
 if (!std::isfinite(work) || !std::isfinite(network) || !std::isfinite(sequential))
 return std::numeric_limits<Float64>::infinity();

 return work * config.work_weight
 + network * config.network_weight
 + sequential * config.sequential_weight;
 }

The inf * 0 = NaN guard is the kind of detail that separates a cost model that works from one that occasionally picks an impossible plan.

The formulas

A hash join prices the probe, the output and the hash-table build:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:130
static Cost hashJoinCost(const CostInputs & inputs, bool is_broadcast)
{
 const auto & left_stats = inputStats(inputs, 0);
 const auto & right_stats = inputStats(inputs, 1);

 Cost cost;
 cost.work = (left_stats.estimated_row_count + inputs.output_stats.estimated_row_count) / inputs.parallelism;

 /// The build side is materialized fully on every node for a broadcast join (the nodes build
 /// concurrently) and 1/N per node otherwise. The build is work, not a sequential phase:
 /// `parallel_hash` shards the hash table across threads within a node.
 const Float64 build_rows_per_node = is_broadcast
 ? right_stats.estimated_row_count
 : right_stats.estimated_row_count / inputs.parallelism;
 /// Hash table materialization: memory allocation + cache pressure.
 cost.work += inputs.config.hash_table_build_factor * build_rows_per_node
 + build_rows_per_node * right_stats.estimated_bytes_per_row;
 return cost;
}

The exchange prices the network, and the distinction between a shuffle and a gather is where most of the distributed reasoning lives:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:240
static Cost exchangeCost(const CostInputs & inputs, const IQueryPlanStep & step)
{
 /// A sorted gather over a partial top-N transfers min(input_rows, L * node_count) rows,
 /// not the group's trimmed L; the caller resolves that override from the memo.
 const Float64 rows = inputs.exchange_rows_override.value_or(inputs.output_stats.estimated_row_count);
 Cost cost;
 /// The network term is wall-clock per node, like the work term. A shuffle (N->N) moves
 /// 1/N of the data per node, all nodes concurrently; without the division one node
 /// scanning the whole table is priced the same as the cluster shuffling it, and the
 /// optimizer keeps big scans on a single node.
 Float64 network_bytes = rows * inputs.output_stats.estimated_bytes_per_row;
 if (dynamic_cast<const ShuffleExchangeStep *>(&step))
 network_bytes /= std::max(1.0, inputs.node_count);
 cost.network += network_bytes;
 cost.sequential += inputs.config.exchange_fixed_overhead;
 /// Gather (N->1) and Scatter (1->N) funnel every row through a single node that
 /// sends or receives them sequentially, so their transfer stays undivided and each row
 /// pays the funnel cost. Without it a gather of a large input looks as cheap as a
 /// shuffle, so the optimizer distributes work (e.g. a sort) that should stay local.
 if (dynamic_cast<const GatherExchangeStep *>(&step) || dynamic_cast<const ScatterExchangeStep *>(&step))
 cost.sequential += inputs.config.funnel_sequential_cost_per_row * rows;
 return cost;
}

Three subtleties are worth spelling out:

A shuffle divides the network cost by the node count, because each node ships 1/N of the data and all nodes do it concurrently.

A broadcast does not divide, because every node ingests the whole payload:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:227
static Cost broadcastExchangeCost(const CostInputs & inputs)
{
 Cost cost;
 /// The network term is wall-clock per node, like the work term (which is divided by
 /// parallelism): every receiving node ingests the full payload, all of them concurrently,
 /// and the senders each ship their share of the copies in parallel. So one full payload
 /// is the per-node bound. A shuffle moves 1/N per node, so a broadcast still costs N times
 /// a shuffle of the same data.
 cost.network += inputs.output_stats.estimated_row_count * inputs.output_stats.estimated_bytes_per_row;
 cost.sequential += inputs.config.exchange_fixed_overhead;
 return cost;
}

Every exchange pays a fixed overhead in sequential units, which is what keeps a plan over a small input local: distributing 25 rows is not worth a network round trip.

A read is priced on its scan volume, not its output

This one is subtle and important. A filter that the primary key cannot prune leaves the whole granule selection to scan, so the output estimate badly understates the cost:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:150
/// One node reads the whole table.
static Cost fullReadCost(const CostInputs & inputs)
{
 /// Price the scan volume, not the output volume: a filter the primary key cannot prune
 /// leaves the whole granule selection to scan, and without this a replicated read of such
 /// a table would look orders of magnitude cheaper than the scan it does.
 if (inputs.output_stats.physical_read_bytes > 0)
 return Cost{.work = inputs.output_stats.physical_read_bytes};
 return Cost{.work = inputs.output_stats.estimated_row_count * inputs.output_stats.estimated_bytes_per_row};
}

Without this, ReplicatedRead — every node reading the whole table from S3 — would look free for a table whose filter prunes 99 % of the rows.

The cost model is query-tunable

Everything above is overridable with one JSON parameter:

 sql
 
SET param__internal_cascades_cost_config = '{"work_weight":1,"network_weight":1,"sequential_weight":32}';

and an invalid configuration rejects the query rather than silently falling back to the defaults.

The rules

Join implementations

One rule produces all the join alternatives:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/HashJoinImplementation.cpp:19
/// Produces all applicable hash join implementations in a single rule:
/// - Local join: both inputs gathered to one node (always applicable)
/// - Broadcast join: left partitioned any way, right replicated (only when node_count > 1)
/// - Partitioned (shuffle) join: both inputs shuffled by join keys (only when node_count > 1
/// and the join has equi-join predicates)
///
/// When node_count == 1 all three strategies produce the same plan, so only the local join
/// is emitted to avoid redundant identical alternatives in the memo.
class HashJoinImplementation : public IOptimizationRule

A shuffle join partitions both sides by the join key. The key columns are recorded positionally on both sides, together with the type they are hashed at:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/HashJoinImplementation.cpp:316
 auto add_key = [&](const JoinKeyPair & key)
 {
 left_dist.columns.push_back({key.left});
 right_dist.columns.push_back({key.right});
 if (needs_hash_cast)
 {
 const String & type_name = hash_type_for(key);
 left_dist.hash_type_names.push_back(type_name);
 right_dist.hash_type_names.push_back(type_name);
 }
 /// Advertise the output partitioning only on the preserved side(s).
 NameSet output_key;
 if (left_preserved)
 output_key.insert(key.left);
 if (right_preserved)
 output_key.insert(key.right);

The hash_type_names exist because if the two sides have different key types, they must be cast to a common type before hashing, or the two sides would land in different buckets and the join would produce nothing.

Two-stage aggregation

The transformation rule splits a single-phase aggregation into a partial and a merge:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:92
/// Logical transformation that splits a single-phase Agg into a two-phase plan:
/// PartialAgg(input) -> ShuffleExchange -> FinalMergeAgg
///
/// PartialAgg computes partial aggregate states locally on each node without finalization.
/// FinalMergeAgg receives pre-aggregated states and merges them into final results.
/// The exchange between the two is inserted by the `DistributionEnforcer` based on the
/// distribution requirements set by the implementation rules on FinalMergeAgg.
///
/// A non-final aggregation is split too when it promises bucket order to a consumer outside the
/// plan: the merge then stays non-final and restores the bucket order over the gathered partials,
/// which is the distributed alternative to the single-node partial `AggregationImplementation`
/// leaves for such a step.
class TwoStageAggregationTransformation : public IOptimizationRule

and the split clones the aggregation step, marks the clone as non-final, and creates a MergingAggregatedStep for the merge half:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:345
 /// Phase 1: partial aggregation - takes raw rows, outputs intermediate aggregate states.
 auto partial_step_ptr = cloneStepAs(*agg_step);
 auto * partial_step = partial_step_ptr.get();
 partial_step->setFinal(false);

The shuffle aggregation is only applicable in some cases, and the reasons are all correctness reasons:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:64
 /// See `addShuffleAggregation` for why each condition disables the shuffle strategy.
 bool isShuffleApplicable() const
 {
 return !agg_step.getParams().keys.empty()
 && !agg_step.isGroupingSets()
 && !agg_step.getParams().overflow_row
 && agg_step.getParams().max_rows_to_group_by == 0;
 }

no group keys → a global aggregation has nothing to shuffle by;
grouping sets → the keys are the union of the sets’ keys, so shuffling by the union splits the rows of one grouping-set group across nodes;
an overflow row → every node would emit its own;
max_rows_to_group_by → the limit is a global contract, but each node would enforce it independently.

Single-key shuffle alternatives

A nice touch: for a multi-key GROUP BY, the optimizer also considers shuffling by one of the keys, which is cheaper (fewer bytes in the exchange) and still correct:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:198
/// Single-key shuffle alternatives.
/// For aggregations with 2+ group-by keys, generate a shuffle alternative for each
/// individual key. Correctness: `GROUP BY (A, B)` with data shuffled by `A` is correct
/// because all rows with the same `(A, B)` have the same `A`, hence the same node.
void AggregationImplementation::StrategyEnumerator::addSingleKeyShuffleAggregations(const std::vector<size_t> & candidate_node_counts)

Enforcers: self-referential expressions

When a parent requires {1 node} and the cheapest child produces {4 nodes}, something has to bridge the gap. That something is an enforcer — an exchange step inserted by the optimizer.

Enforcers are the hairiest part of the implementation, because an enforcer’s input is the same group with relaxed requirements, which looks like a cycle:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/DistributionEnforcer.cpp:24
/// Produces self-referential enforcer expressions that bridge distribution gaps.
/// Each enforcer expression lives in the same group as the source expression; its
/// single input points back to the same group with normalized properties (node_count,
/// is_replicated, and sorting where needed) as the requirement - this lets the optimizer
/// pick the cheapest source with that distribution shape. The optimizer recursively
/// satisfies this self-referential input through the normal task mechanism, enabling
/// natural enforcer composition (e.g. a sort and a gather compose into gather-then-sort,
/// see `SortingEnforcer`, without
/// bundling steps).
class DistributionEnforcer : public IOptimizationRule

The construction is centralised:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/DistributionEnforcer.cpp:104
void DistributionEnforcer::EnforcerEnumerator::addEnforcer(QueryPlanStepPtr step, ExpressionProperties input_required, SortDescription output_sorting)
{
 ExpressionProperties output_properties;
 output_properties.distribution = required_properties.distribution;
 output_properties.sorting = std::move(output_sorting);
 auto enforcer_expr = makeEnforcerExpression(
 expression, std::move(step), std::move(input_required), std::move(output_properties), EnforcedProperty::Distribution);

 rule.addPhysicalToMemo(enforcer_expr, required_properties, memo, result);
}

Two invariants keep the self-reference acyclic. The first is that an enforcer carries an axis (Sorting or Distribution), and an enforcer that satisfies a wildcard axis only by over-providing on it is not eligible — a sorted enforcer cannot fill an empty-sort slot, a keyed exchange cannot fill an empty-columns slot:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:159
 if (input_is_self_referential && candidate->enforced_property != EnforcedProperty::None)
 {
 /// Empty sort requirement: reject a sorted enforcer (it over-provides on the sort axis).
 if (required_properties.sorting.empty() && !candidate->properties.sorting.empty())
 return;
 /// Empty distribution-columns requirement: reject a keyed exchange. A sorting enforcer
 /// carrying keyed columns is still needed to feed a sorted gather, so keep it eligible.
 if (candidate->enforced_property == EnforcedProperty::Distribution
 && required_properties.distribution.columns.empty()
 && (!candidate->properties.distribution.columns.empty()
 || !candidate->properties.distribution.hash_type_names.empty()))
 return;
 }

The second is that an enforcer must never evict a non-enforcer alternative from the best-implementation cache, because that alternative is the acyclic way out:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:78
 /// An enforcer "satisfies" a weaker non-enforcer only by over-providing (e.g. a sorted gather
 /// covers an empty-sort requirement). Dropping the base would leave a self-referential enforcer
 /// chain with no acyclic alternative, so an enforcer never suppresses or evicts a distinct base.
 auto enforcer_over_provides_for = [](const GroupExpressionPtr & enforcer_candidate, const GroupExpressionPtr & base_candidate)
 {
 return enforcer_candidate->enforced_property != EnforcedProperty::None
 && base_candidate->enforced_property == EnforcedProperty::None
 && !(enforcer_candidate->properties == base_candidate->properties);
 };

The fixed point

Enforcers compose: a sort enables a sorted gather, which enables a gather. The Stage 3 loop iterates until no new enforcer is produced:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:96
 /// Fixed-point loop: iterate over newly-added physical expressions until no
 /// new enforcers are produced. Each iteration may create expressions that
 /// enable further enforcers (e.g. a sort enables a sorted `GatherExchange`).
 size_t enforced_up_to = 0;
 bool new_enforcers_created = true;
 while (new_enforcers_created)
 {
 new_enforcers_created = false;

Keeping the best implementation

Each group keeps a Pareto frontier of implementations, bucketed by distribution shape so that a lookup is O(1) plus a short scan:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:73
void Group::updateBestImplementation(GroupExpressionPtr expression, const CostConfig & cost_config)
{
 UInt64 key = distributionKey(expression->properties.distribution);
 auto & bucket = best_implementations[key];

and the eviction rule is a domination test: an implementation is dropped when another one satisfies at least the same properties at a lower cost.

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:88
 /// Remove all known best expressions with higher cost and properties satisfied by the new expression.
 /// Only the matching distribution-shape bucket needs checking - `isSatisfiedBy` requires
 /// exact match on (node_count, is_replicated).
 for (auto best_it = bucket.begin(); best_it != bucket.end();)
 {
 if (expression->properties.isSatisfiedBy((*best_it)->properties) &&
 (*best_it)->cost->subtree_cost.total(cost_config) <= expression->cost->subtree_cost.total(cost_config) &&
 !enforcer_over_provides_for(*best_it, expression))
 {
 /// There is already a cheaper implementation that satisfies the same properties
 return;
 }

Failing closed

The search is bounded by a task count, not by branch-and-bound. And when the budget runs out, the query is rejected rather than planned from a partial memo:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:308
 /// Limit the time in terms of optimization tasks instead of wall clock time. This is done for stability of generated plans regardless of system load.
 /// Microsoft SQL Server's optimizer team describes this in Andy Pavlo's seminar: https://www.youtube.com/watch?v=pQe1LQJiXN0
 const size_t executed_tasks_limit = getCascadesTaskLimitParam(query_context, CascadesDefaults::DEFAULT_TASK_LIMIT);
 size_t executed_tasks_count = 0;
 for (; !tasks.empty() && executed_tasks_count < executed_tasks_limit; ++executed_tasks_count)
 {
 auto task = std::move(tasks.top());
 tasks.pop();
 task->execute(*this);
 }

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:321
 /// Fail closed if the search did not finish within the task budget: building a plan from a
 /// partial memo can yield a non-minimal plan or a confusing failure deep inside buildBestPlan.
 /// Surface a clear error instead and point at the knob to raise the limit.
 if (!tasks.empty())
 throw Exception(ErrorCodes::SUPPORT_IS_DISABLED,
 "Cascades optimizer did not finish within the task budget of {} tasks "
 "(root group #{}, required properties {}, {} groups in memo, {} tasks left, next: {}). "
 "The distributed Cascades optimizer is experimental; set enable_cascades_optimizer = 0 "
 "or simplify the query.",
 executed_tasks_limit, root_group_id, root_required_properties.dump(),
 memo.getGroupCount(), tasks.size(), tasks.top()->describe());

The default budget is 100 000 tasks:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/OptimizerDefaults.h:17
constexpr size_t DEFAULT_TASK_LIMIT = 100000;

Trying it

 sql
 
 
SET enable_cascades_optimizer = 1;
SET make_distributed_plan = 1;

SELECT n_name, SUM(o_totalprice) AS revenue
FROM orders
JOIN customer ON o_custkey = c_custkey
JOIN nation ON c_nationkey = n_nationkey
WHERE o_orderdate >= '1995-01-01' AND o_orderdate < '1995-04-01'
GROUP BY n_name
ORDER BY revenue DESC
SETTINGS send_logs_level = 'test';

The test log level prints the memo, every costed expression with its work/network/sequential breakdown, and the final plan — which is the intended way to understand why the optimizer chose what it chose.

Trade-offs

What it buys: plans that no fixed heuristic would produce — a broadcast of a tiny table instead of a shuffle of a huge one, a ReplicatedRead from shared storage instead of a network transfer, a two-phase aggregation instead of a gather, a sort per node instead of a sort of everything.

What it costs:

It is experimental. The setting is EXPERIMENTAL and off by default, and it only applies with make_distributed_plan = 1.
Join ordering is not part of it. This is the biggest difference from classical Cascades. ClickHouse fixes the join order first (with the existing dpsize/dphyp/greedy algorithms) and Cascades only optimises the distribution strategy. The published literature calls this the “split” design, and it loses the global optimum when the join order interacts with physical properties — which, in a distributed setting, it does.
The statistics are weak. Today they come from index analysis and from query-parameter hints; there is no cardinality estimator as sophisticated as the one the pre-Cascades join orderer uses. A cost model on bad statistics picks the wrong plan confidently.
Only two parallelism levels are explored: {1 node} and {N nodes}. Intermediate counts were never chosen on TPC-H, and exploring them would multiply the search space.
The search can be exponential, which is why there is a hard task budget and a fail-closed error. Classical Cascades would prune instead.
Some plan shapes are rejected outright — WITH TOTALS, ROLLUP, CUBE, extremes, PASTE JOIN, and reads that cannot be cloned across workers.

The most interesting thing about this code, though, is not the optimizer itself but the discipline around it: every place where the implementation deliberately deviates from the textbook — no branch-and-bound, a fail-closed budget, the enforcer-axis rules, the “never evict the acyclic fallback” rule — carries a comment explaining exactly why the textbook version would be wrong here. That is rare, and it is the part of the code most worth reading.

How ClickHouse Packs Short Strings Into Hash Table Keys

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Strings are the most common GROUP BY key in practice. They are also the worst case for a hash table: a std::string in a cell means one heap allocation per distinct key, and a std::string_view in a cell means the bytes must outlive the table, which for an aggregation means copying them into an arena.

ClickHouse’s answer is a hash table that is not a hash table. Instead of one table with a variable-size key, StringHashTable keeps five separate tables, one per key size class, and packs short keys into fixed-width integers. The result is a table where every cell is the same size, no key ever allocates, and the compiler can dispatch on the key length at compile time.

The file is src/Common/HashTable/StringHashTable.h, with the map specialisation in StringHashMap.h.

Four size classes and one oddity

The composite table is five sub-tables:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:234
template <typename SubMaps>
class StringHashTable : private boost::noncopyable
{
protected:
 static constexpr size_t NUM_MAPS = 5;
 // Map for storing empty string
 using T0 = typename SubMaps::T0;

 // Short strings are stored as numbers
 using T1 = typename SubMaps::T1;
 using T2 = typename SubMaps::T2;
 using T3 = typename SubMaps::T3;

 // Long strings are stored as std::string_view along with saved hash
 using Ts = typename SubMaps::Ts;
 using Self = StringHashTable;

and the storage:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:254
 T0 m0;
 T1 m1;
 T2 m2;
 T3 m3;
 Ts ms;

The classes are:

Table	Key type	Holds
`m0`	—	the empty string, in a dedicated slot
`m1`	`StringKey8` (`UInt64`)	1..8 bytes
`m2`	`StringKey16` (`UInt128`)	9..16 bytes
`m3`	`StringKey24`	17..24 bytes
`ms`	`std::string_view` + saved hash	25+ bytes, or short strings with a trailing zero

The key types are just integers:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:10
using StringKey8 = UInt64;
using StringKey16 = UInt128;
struct StringKey24
{
 UInt64 a;
 UInt64 b;
 UInt64 c;

 bool operator==(const StringKey24 rhs) const { return a == rhs.a && b == rhs.b && c == rhs.c; }
};

So a 20-byte string becomes three UInt64s and nothing else: no length field, no pointer, no allocation. The length is implicit — it is recovered from the position of the first zero byte.

The packing

The heart of the file is dispatchOnKeyClass, which classifies a key, packs it, hashes it, and calls a callback with the right sub-table. It is written for speed, and the comment says so:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:357
 // Dispatch is written in a way that maximizes the performance:
 // 1. Always memcpy 8 times bytes
 // 2. Use switch case extension to generate fast dispatching table
 // 3. Funcs are named callables that can be force_inlined
 //
 /// Note: the key bytes must be readable in 8-byte chunks (a `std::string` cannot be passed
 /// here, while e.g. `ColumnString::getDataAt` can), because the packing copies 8 bytes at a
 /// time.
 template <typename Resolver, typename KeyHolder, typename HashProvider, typename Func>
 static auto ALWAYS_INLINE dispatchOnKeyClass(Resolver && resolver, KeyHolder && key_holder, HashProvider && hash_of, Func && func)
 {
 const auto & x = keyHolderGetKey(key_holder);
 const size_t sz = x.size();
 if (sz == 0)
 {
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 0>{}, 0), VoidKey{}, 0);
 }

 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

 const char * p = x.data();
 // pending bits that needs to be shifted out
 const char s = (-sz & 7) * 8;

The switch ((sz - 1) >> 3) that follows turns the length into one of four cases, and the compiler turns that into a jump table or a chain of comparisons:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:394
 switch ((sz - 1) >> 3)
 {
 case 0: // 1..8 bytes
 {
 // first half page
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)
 {
 memcpy(&n[0], p, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] &= -1ULL >> s;
 else
 n[0] &= -1ULL << s;
 }
 else
 {
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[0], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] >>= s;
 else
 n[0] <<= s;
 }
 const size_t res = hash_of(k8);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 1>{}, res), k8, res);
 }

Three tricks are hiding in those twenty lines.

Trick one: the trailing-zero test. A fixed-size string key encodes the length implicitly, by the position of the first zero byte. That means a string whose last byte is zero cannot be represented: "ab\0" and "ab" would pack to the same 8 bytes. So such strings are pushed to the generic table before the switch:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:376
 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

Trick two: the mask/shift pair. s = (-sz & 7) * 8 is the number of padding bits needed to round the length up to a multiple of 8. In the first branch the 8 bytes are read forwards from the start of the string and the padding bits are cleared with a mask; in the second branch the 8 bytes are read backwards from the end of the string and the padding bits are shifted out.

Trick three: the “first half page” rule. Reading 8 bytes from a 3-byte string reads 5 bytes past its end. That is only safe if those bytes are mapped. The condition

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:399
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)

tests whether the address is 2048-byte aligned; if it is, at least 2048 bytes of the page are still ahead of it, so reading 8 bytes forward cannot fault. Otherwise the code reads backwards instead, ending exactly at the last byte of the string. The bytes before the string that this reads are then shifted out — the shift >>= s discards exactly the (-sz & 7) bytes that belong to the previous string or to the column’s offset array. So the garbage is read but never used.

The 9..16 and 17..24 byte cases use the same trick with two and three words:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:420
 case 1: // 9..16 bytes
 {
 memcpy(&n[0], p, 8);
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[1], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[1] >>= s;
 else
 n[1] <<= s;
 const size_t res = hash_of(k16);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 2>{}, res), k16, res);
 }

Note that the first word is always read forwards from p — for a 9-byte string that is inside the string — and the last word is read backwards from the end. The overlap between them is harmless: the bytes in the middle are read twice with the same values.

Unpacking

Going the other way, the length is recovered from the number of leading zero bytes:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:21
inline std::string_view ALWAYS_INLINE toStringView(const StringKey8 & n)
{
 chassert(n != 0);
 if constexpr (std::endian::native == std::endian::big)
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countr_zero(n) >> 3)};
 else
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countl_zero(n) >> 3)};
}

countl_zero(n) >> 3 is the number of whole zero bytes at the end of the word, so 8 - that is the string length. On a big-endian machine the zero bytes are at the other end, hence the countr_zero variant.

The empty-cell protocol, specialised

The “empty cell is a zeroed cell” rule from the generic table needs a suitable notion of “zero” for a packed string key. For StringKey16 the rule is “the last word is zero”:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:20
template <typename TMapped>
struct StringHashMapCell<StringKey16, TMapped> : public HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>
{
 using Base = HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>;
 using value_type = typename Base::value_type;
 using Base::Base;
 static constexpr bool need_zero_value_storage = false;
 bool isZero(const HashTableNoState & state) const { return isZero(this->value.first, state); }

 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey16 & key, const HashTableNoState &) { return key.items[1] == 0; }
 void setZero() { this->value.first.items[1] = 0; }

The reasoning is the same as in the generic table: a packed key with a zero last word would have to encode the length “16” with no room for a terminating zero byte, so such a key cannot exist — the encoding is injective on all valid keys, and the all-zero pattern is unreachable. That makes it safe to use as the empty marker. Same for StringKey24, where the marker is the third word:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:49
 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey24 & key, const HashTableNoState &)
 { return key.c == 0; }
 void setZero() { this->value.first.c = 0; }

The generic std::string_view cell cannot do this — a string_view is a pointer and a length, and a zeroed one is a null pointer with length 0, which is a legal key (the empty string). So the generic cell declares that it needs no zero storage and the empty string gets its own slot instead:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:134
template <typename Cell>
struct StringHashTableEmpty // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
 using Self = StringHashTableEmpty;

 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

The empty slot is not a hash table at all — it is a single cell with an emplace that ignores the key entirely:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:172
 template <typename KeyHolder>
 void ALWAYS_INLINE emplace(KeyHolder &&, LookupResult & it, bool & inserted, size_t = 0)
 {
 if (!hasZero())
 {
 setHasZero();
 inserted = true;
 }
 else
 inserted = false;
 it = zeroValue();
 }

Hashing

The hash function is chosen per class, and for the fixed-size classes it is the hardware CRC32C instruction:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:46
struct StringHashTableHash
{
#if defined(__SSE4_2__)
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key);
 return res;
 }
 size_t ALWAYS_INLINE operator()(StringKey16 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key.items[0]);
 res = _mm_crc32_u64(res, key.items[1]);
 return res;
 }

with an AArch64 CRC32 path, an s390x path, and CityHash as the portable fallback:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:114
#else
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 return CityHash_v1_0_2::CityHash64(reinterpret_cast<const char *>(&key), 8);
 }

CRC32 is a poor hash function in the abstract — the header of the generic hash file says so — but it is one instruction per 8 bytes with a three-cycle latency, and the table is small enough to fit in cache most of the time. It also has to be consistent with the packing: the hash is computed over the packed word, not over the original bytes, so "abc" and "abc\0..." cannot collide by accident — they hash different words.

The lookup result

The string table does not return a pointer to a cell; it returns a pointer to the mapped value, wrapped in a small type:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:214
template <typename Mapped>
struct StringHashTableLookupResult
{
 Mapped * mapped_ptr;
 StringHashTableLookupResult() {} /// NOLINT
 StringHashTableLookupResult(Mapped * mapped_ptr_) : mapped_ptr(mapped_ptr_) {} /// NOLINT
 StringHashTableLookupResult(std::nullptr_t) {} /// NOLINT
 const VoidKey getKey() const { return {}; } /// NOLINT
 auto & getMapped() { return *mapped_ptr; }
 auto & operator*() { return *this; }
 auto & operator*() const { return *this; }
 auto * operator->() { return this; }
 auto * operator->() const { return this; }
 explicit operator bool() const { return mapped_ptr; }

The reason is that the key lives in the cell and the caller usually only wants the aggregate state; but the key is still there, in a different sub-table with a different representation. Returning a pointer to it would mean returning a pointer to a StringKey8 or to a string_view, which have different types depending on the size class. So the key is erased from the interface: getKey() returns a VoidKey.

Reserving and growing

The string table grows differently from the generic one:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:207
template <size_t initial_size_degree = 8>
struct StringHashTableGrower : public HashTableGrowerWithPrecalculation<initial_size_degree>
{
 // Smooth growing for string maps
 void increaseSize() { this->increaseSizeDegree(1); }
};

The generic grower doubles the buffer on overflow; the string grower grows by a factor of two in the exponent, i.e. by one power of two — the same thing, but expressed as a single degree increment so it composes with the precalculated mask.

Reserving is divided evenly over the four real sub-maps, because the caller only knows the total number of keys:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:317
 explicit StringHashTable(size_t reserve_for_num_elements)
 : m1{reserve_for_num_elements / 4}
 , m2{reserve_for_num_elements / 4}
 , m3{reserve_for_num_elements / 4}
 , ms{reserve_for_num_elements / 4}
 {
 }

That guess is often wrong — a table of 20-byte strings uses m3 only, and m1, m2, ms reserve space they never use. The aggregator knows this and refuses to pre-reserve the string tables:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5214
 /// `StringHashTable::reserve` splits the hint evenly over its four size-class sub-maps,
 /// while a real key set concentrates in one of them, so it is not reserved.
 constexpr bool can_reserve = requires { dst.reserve(size_t{}); } && !requires { dst.emptyStringSlot(); };

The emptyStringSlot() method exists precisely so that this requires expression can tell a StringHashTable from an ordinary hash table:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:312
 /// The dedicated slot of the empty-string key, which the sub-table dispatch hashes to 0.
 T0 & emptyStringSlot() { return m0; }

Where it is used

The string map is one of the standard aggregation data variants:

 cpp
 
 
// src/Interpreters/AggregatedData.h:53
using AggregatedDataWithPackedStringKey = HashMap<PackedStringRef, AggregateDataPtr>;
using AggregatedDataWithShortStringKey = StringHashMap<AggregateDataPtr>;
using AggregatedDataWithStringKey = HashMapWithSavedHash<std::string_view, AggregateDataPtr>;

Three flavours of string key, three tables:

AggregatedDataWithPackedStringKey — a PackedStringRef, a pointer plus a length, with the payload inline for short strings;
AggregatedDataWithShortStringKey — the composite StringHashMap of this article;
AggregatedDataWithStringKey — a plain std::string_view map with a saved hash, where the key bytes live in an arena.

The StringHashMap variant is the interesting one, because it is the only one where the key does not live anywhere else: the string is in the table, as three machine words.

Trade-offs

What the packed string table buys:

No allocation per key. A 20-byte key is three UInt64s inside the cell. Nothing is copied into an arena, nothing is freed.
A uniform cell size. Every cell of m1 is 8 bytes of key plus the mapped value; every cell of m3 is 24 bytes plus the mapped value. The table can be scanned, and the compiler can generate good code for it.
A compile-time dispatch on length. dispatchOnKeyClass is a template over the size class, so each branch is specialised: the m1 path hashes one word, the m3 path hashes three.
A hash over the packed form. The hash is computed over exactly the bytes that are stored, so there is no chance of the stored key and the hashed key disagreeing.

What it costs:

The bytes of the key are duplicated. The cell holds the packed key, and if the caller needs the original string it has to be reconstructed with toStringView.
The trailing-zero case is a wart. A string ending in \0 — which happens with binary data, and with FixedString columns padded with zeros — cannot use the packed classes and falls back to the generic table, where the key is a string_view into the arena.
The reservation heuristic is bad. Reserving n/4 for each of the four sub-maps wastes memory whenever the keys concentrate in one class, which is the normal case.
The empty string needs its own slot, and the key type of the generic sub-table is not the same as the key type of the packed ones, so the composite has to be written as a template over a SubMaps bundle rather than as a plain HashTable.

The design is a good example of the general principle: when the key type has a size distribution, splitting the table by size class turns a variable-size-cell problem into a set of fixed-size-cell problems, and lets the compiler specialise each one.

How ClickHouse Merges Columns One at a Time: the Vertical Merge Algorithm

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every MergeTree table is a pile of immutable parts. Every part is a pile of immutable columns. Background merges take a contiguous range of parts and rewrite them as one bigger part. That sounds like a simple copy, and for a table with three narrow columns it is one.

Now consider a table with two hundred columns, one of which is a 40 MB String of raw JSON and the rest are UInt64 counters. The obvious merge implementation reads all two hundred columns of the source parts, merges the sorting key, and writes two hundred columns out. It has to hold, in memory at once, the merged key plus every payload column of the rows it is currently holding — which for a wide table means the peak memory of a merge is proportional to the sum of all column widths, not to the width of the widest column.

That is a bad trade. The payload columns are almost always much larger than the key, and they are almost always independent of the key: if you merge the keys first and separately, you can then reconstruct each payload column without ever holding two payload columns in memory at the same time.

ClickHouse implements exactly that. It is called the vertical merge algorithm, and it is one of the more elegant pieces of engineering in the MergeTree code base. The trick that makes it work is a one-byte-per-row side-channel file that records where each output row came from.

The problem, stated precisely

Suppose we merge three parts, each sorted by the key k:

 text
 
 
 part A (10 rows) part B (10 rows) part C (10 rows)
 k=1 a1=... k=1 b1=... k=2 c1=...
 k=1 a2=... k=2 b2=... k=2 c2=...
 k=3 a3=... k=2 b3=... k=5 c3=...
 ... ... ...

A horizontal merge reads all three parts’ k columns, merges them into one sorted stream, and then — because MergeTree guarantees that all rows with the same key are adjacent and, within a key, appear in source-part order — it can reconstruct every other column by a gather: walk the merged key stream, and for each row, take the next value of that column from the part the row came from.

The output order of the merged key is not the concatenation of the inputs; it is the k-way merge of them. So for each output row we need to know which input part it came from. That is the entire information content of the vertical merge’s side channel.

And it is exactly one small integer per row.

The horizontal stage: merging the key and writing the row sources

The vertical merge is not a separate code path that replaces the normal merge. It is a second stage that runs after the normal merge has merged the key. The normal merge is called the “horizontal” stage, and the code says so explicitly — the stage class is called ExecuteAndFinalizeHorizontalPart:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.h:63
/**
 * Overview of the merge algorithm
 *
 * Each merge is executed sequentially block by block.
 * The main idea is to make a merge not a subroutine which is executed
 * in a thread pool and may occupy a thread for a period of time,
 * but to make a merge a coroutine which can suspend the execution
 * in some points and then resume the execution from this point.
 *
 * A perfect point where to suspend the execution is after the work over a block is finished.
 * The task itself will be executed via BackgroundJobExecutor.
 *
 * The interface of the task is simple.
 * The main method is `execute()` which will return true, if the task wants to be executed again and false otherwise.
 *
 * With this kind of task we can give a merge a priority.
 * A priority is simple - the lower the size of the merge, the higher priority.
 * So, if ClickHouse wants to merge some really big parts into a bigger part,
 * then it will be executed for a long time, because the result of the merge is not really needed immediately.
 * It is better to merge small parts as soon as possible.
*/

Note the second half of that comment: the merge is a coroutine. It is resumed by the background pool between blocks, which is what makes it possible to give a small merge priority over a huge one without pinning a thread. The vertical stage is expressed in the same idiom: a list of sub-tasks, each of which returns true when it wants to be resumed.

The side channel is a real file, managed by a small class:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:257
/// Manages the "rows_sources" temporary file that is used during vertical merge.
class RowsSourcesTemporaryFile : public ITemporaryFileLookup
{
public:
 /// A logical name of the temporary file under which it will be known to the plan steps that use it.
 static constexpr auto FILE_ID = "rows_sources";

 explicit RowsSourcesTemporaryFile(TemporaryDataOnDiskScopePtr temporary_data_on_disk_)
 : temporary_data_on_disk(temporary_data_on_disk_->childScope({CurrentMetrics::TemporaryFilesForMerge}))
 {
 }

and it is deliberately re-openable, so that several columns can be gathered in parallel from independent file handles — getTemporaryFileForReading returns tmp_data_buffer->read(), a fresh reader each time, with the comment “Reopen the file for each read so that multiple reads can be performed in parallel and there is no need to seek to the beginning.”

That matters: the row-source stream is consumed once per column, and each column’s gather starts from the beginning of the file. If the file were a single sequential stream, gathering column #2 would have to seek back to offset 0 and re-read column #1’s worth of bytes. Re-opening gives every column its own independent read cursor.

The encoding: one byte per row

The row source is not a struct with a source number and a flag in two fields. It is a single byte with two bit-fields packed into it:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:15
/// Tiny struct, stores number of a Part from which current row was fetched, and insertion flag.
struct RowSourcePart
{
 UInt8 data = 0;

 RowSourcePart() = default;

 explicit RowSourcePart(size_t source_num, bool skip_flag = false)
 {
 static_assert(sizeof(*this) == 1, "Size of RowSourcePart is too big due to compiler settings");
 setSourceNum(source_num);
 setSkipFlag(skip_flag);
 }

 size_t getSourceNum() const { return data & MASK_NUMBER; }

 /// In CollapsingMergeTree case flag means "skip this rows"
 bool getSkipFlag() const { return (data & MASK_FLAG) != 0; }

 void setSourceNum(size_t source_num)
 {
 data = (data & MASK_FLAG) | (static_cast<UInt8>(source_num) & MASK_NUMBER);
 }

 void setSkipFlag(bool flag)
 {
 data = flag ? data | MASK_FLAG : data & ~MASK_FLAG;
 }

 static constexpr size_t MAX_PARTS = 0x7F;
 static constexpr UInt8 MASK_NUMBER = 0x7F;
 static constexpr UInt8 MASK_FLAG = 0x80;
};

The top bit is the “skip this row” flag (used when a lightweight delete mask removes rows during the merge), and the low seven bits are the source part number. Seven bits means at most 127 source parts per merge — and you can watch that limit propagate all the way into the merge-algorithm choice (section “When is it worth it?” below).

The static_assert is not decoration. If the compiler ever made this struct two bytes, the row-source file would double in size and every write to it would become a two-byte write; the assert turns a silent performance regression into a build failure.

Who writes the bytes

The horizontal stage’s merge transform writes one byte per output row as it emits it. For the Ordinary merging mode that is the plain k-way merge algorithm:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:444
void MergingSortedAlgorithm::insertRow(const SortCursorImpl & current)
{
 auto write_row_source = [&](bool skipped)
 {
 if (out_row_sources_buf)
 {
 RowSourcePart row_source(current.order, skipped);
 out_row_sources_buf->write(row_source.data);
 }
 };

 size_t current_row = current.getRow();

 if (const auto * mask = source_row_filter_masks[current.order])
 {
 if ((*mask)[current_row])
 merged_data.insertRow(current.all_columns, current_row, current.rows);

 write_row_source(!(*mask)[current_row]);
 }
 else
 {
 merged_data.insertRow(current.all_columns, current_row, current.rows);
 write_row_source(false);
 }
}

current.order is the index of the source stream the row came from. The merged algorithm already knows this — it is the whole reason it can merge — so emitting it costs one byte write per row, i.e. one byte per row of overhead on a merge that is about to rewrite gigabytes.

Note the ordering: merged_data.insertRow(...) happens first, then the row source is written. The write buffer is only non-null for a vertical merge — the merge step is constructed with the file name only when the vertical algorithm was chosen:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3654
 auto merge_step = std::make_unique<MergePartsStep>(
 merge_parts_query_plan.getCurrentHeader(),
 sort_description,
 partition_and_sorting_required_columns,
 global_ctx->merging_params,
 (is_vertical_merge ? RowsSourcesTemporaryFile::FILE_ID : ""), /// rows_sources' temporary file is used only for vertical merge
 filter_column_name,
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size],
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size_bytes],
 max_dynamic_subcolumns,
 is_vertical_merge,
 sorting_queue_strategy,
 cleanup,
 global_ctx->time_of_merge);

so a horizontal merge pays nothing for the side channel.

The vertical stage: gathering one column at a time

Now the second stage. It does not merge anything. It gathers. For each non-key column, in turn:

open a read stream for that column in every source part;
union those streams;
run a ColumnGathererTransform over them, which uses the row-source file to decide which source’s value belongs at each output position.

Here is the step that wires the gatherer into the plan:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2172
 createReadFromPartStep(
 MergeTreeSequentialSourceType::Merge,
 *plan_for_part,
 *global_ctx->data,
 global_ctx->storage_snapshot,
 RangesInDataPart(global_ctx->future_part->parts[part_num], nullptr, part_num, part_starting_offset),
 global_ctx->alter_conversions[part_num],
 global_ctx->merged_part_offsets,
 Names{column_name},
 global_ctx->input_rows_filtered,
 apply_deleted_mask,
 /*filter=*/ std::nullopt,
 ctx->read_with_direct_io,
 ctx->use_prefetch,
 global_ctx->context,
 getLogger("VerticalMergeStage"));

Names{column_name} — exactly one column is read per part. The read plan for a vertical merge is a column read plan, not a row read plan. Every source part is opened, and only the column of interest is decoded.

Then the per-part streams are unioned, and a ColumnGathererStep is added to the plan. The gatherer is a merging transform, so it plugs into the same IMergingTransform machinery as the key merge: its input is one column per source part, its output is one merged column, and the row-source file is handed to it as a ReadBuffer by name.

The gather loop

The heart of it is ColumnGathererStream::gather. It reads the row-source bytes and copies values out of the per-part source columns:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:131
template <typename Column>
void ColumnGathererStream::gather(Column & column_res)
{
 row_sources_buf.nextIfAtEnd();
 RowSourcePart * row_source_pos = reinterpret_cast<RowSourcePart *>(row_sources_buf.position());
 RowSourcePart * row_sources_end = reinterpret_cast<RowSourcePart *>(row_sources_buf.buffer().end());

 if (next_required_source == -1)
 {
 /// Start new column.
 /// Actually reserve works only for fixed size columns.
 /// So it's safe to ignore preferred size in bytes and call reserve for number of rows.
 size_t size_to_reserve = std::min(static_cast<size_t>(row_sources_end - row_source_pos), block_preferred_size_rows);
 column_res.reserve(size_to_reserve);
 }

 next_required_source = -1;

 /// We use do ... while here to ensure there will be at least one iteration of this loop.
 /// Because the column_res.byteSize() could be bigger than block_preferred_size_bytes already at this point.
 do
 {
 if (row_source_pos >= row_sources_end)
 break;

 RowSourcePart row_source = *row_source_pos;
 size_t source_num = row_source.getSourceNum();
 Source & source = sources[source_num];
 bool source_skip = row_source.getSkipFlag();

 if (source.pos >= source.size) /// Fetch new block from source_num part
 {
 next_required_source = source_num;
 return;
 }

 ++row_source_pos;

 /// Consecutive optimization. TODO: precompute lengths
 size_t len = 1;
 size_t max_len = std::min(static_cast<size_t>(row_sources_end - row_source_pos), source.size - source.pos); // interval should be in the same block

 while (len < max_len && row_source_pos->data == row_source.data)
 {
 ++len;
 ++row_source_pos;
 }

 row_sources_buf.position() = reinterpret_cast<char *>(row_source_pos);

 if (!source_skip)
 {
 /// Whole block could be produced via copying pointer from current block
 if (source.pos == 0 && source.size == len)
 {
 /// If current block already contains data, return it.
 /// Whole column from current source will be returned on next read() iteration.
 source_to_fully_copy = &source;
 return;
 }
 if (len == 1)
 column_res.insertFrom(*source.column, source.pos);
 else
 column_res.insertRangeFrom(*source.column, source.pos, len);
 }

 source.pos += len;
 } while (column_res.size() < block_preferred_size_rows && column_res.byteSize() < block_preferred_size_bytes);
}

Three optimizations are hiding in there.

Run-length coalescing. The inner while loop scans forward over consecutive row-source bytes that are identical. Since the byte encodes “which source part”, a run of identical bytes means “these N output rows came from the same source part, consecutively” — which, because the source part’s own column is read in order, means “these N output rows are exactly source part’s rows [pos, pos+N)”. So the whole run is copied with a single insertRangeFrom call instead of N single-element inserts. The comment /// Consecutive optimization. TODO: precompute lengths is honest about the remaining inefficiency: the run length is rediscovered for every column, N times.

Whole-column pointer steal. If the run covers the entire remaining source column (source.pos == 0 && source.size == len), the output column can simply be the source column — no copy at all. The transform returns the source ColumnPtr directly and sets source_to_fully_copy, and the merge() method hands it out:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.cpp:114
 else
 {
 res.addColumn(source_to_fully_copy->column);
 }

For a merge where one part dominates — the common case after a few rounds of merging, where you merge one big part with one small one — most columns of the big part are copied by pointer. The “merge” of such a column is essentially free.

Sparse output. The source_skip flag comes from the lightweight-delete mask: rows deleted by a DELETE are recorded in the row-source stream with the skip bit set, and the gatherer simply does not copy them out. The horizontal stage does the same filtering, so the two stages agree on which rows exist without either having to re-evaluate the delete predicate.

Writing the gathered column

The gathered column is written by MergedColumnOnlyOutputStream — the vertical stage’s counterpart of MergedBlockOutputStream, except that it writes exactly one column. Its constructor carries the invariant that makes the whole thing safe:

 cpp
 
 
// src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:36
 /// Granularity is never recomputed while writing only columns.
 MergeTreeWriterSettings writer_settings(

The horizontal stage has already decided where the granule boundaries are (it wrote the marks for the key columns), and the payload columns must use the same boundaries, or the part would be corrupt.

When is it worth it?

Vertical merging is not always a win. It has costs: an extra pass over the row-source file per column, an extra file in the temporary space, and a pipeline per column. So chooseMergeAlgorithm() gates it behind a set of conditions:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3760
MergeAlgorithm MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm() const
{
 const size_t total_rows_count = global_ctx->merge_list_element_ptr->total_rows_count;
 const size_t total_size_bytes_uncompressed = global_ctx->merge_list_element_ptr->total_size_bytes_uncompressed;
 const auto & merge_tree_settings = global_ctx->data_settings;

 if (global_ctx->deduplicate)
 return MergeAlgorithm::Horizontal;
 if ((*merge_tree_settings)[MergeTreeSetting::enable_vertical_merge_algorithm] == 0)
 return MergeAlgorithm::Horizontal;
 if (ctx->need_remove_expired_values)
 {
 /// `TTLTransform` stops reading after the first block when the rows TTL has expired for the whole part,
 /// while a vertical merge reads every key row to write `rows_sources`.
 if (global_ctx->future_part->merge_type == MergeType::TTLDrop && global_ctx->metadata_snapshot->hasRowsTTL())
 return MergeAlgorithm::Horizontal;
 if (!canVerticalTTLDelete(*global_ctx))
 return MergeAlgorithm::Horizontal;
 }
 if (global_ctx->future_part->part_format.part_type != MergeTreeDataPartType::Wide)
 return MergeAlgorithm::Horizontal;
 if (global_ctx->cleanup)
 return MergeAlgorithm::Horizontal;

and then the quantitative thresholds:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3793
 bool is_supported_storage =
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Ordinary ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Collapsing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Replacing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::VersionedCollapsing;

 bool enough_ordinary_cols = global_ctx->gathering_columns.size() >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_columns_to_activate];

 bool enough_total_rows = total_rows_count >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_rows_to_activate];

 bool enough_total_bytes = total_size_bytes_uncompressed >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_bytes_to_activate];

 bool no_parts_overflow = global_ctx->future_part->parts.size() <= RowSourcePart::MAX_PARTS;

 auto merge_alg = (is_supported_storage && enough_total_rows && enough_total_bytes && enough_ordinary_cols && no_parts_overflow) ?
 MergeAlgorithm::Vertical : MergeAlgorithm::Horizontal;

 return merge_alg;
}

Read those conditions as a description of when the vertical algorithm is the right shape. Wide parts only, because a Compact part stores all columns in one data file and you cannot read one column of it without reading the others. No deduplication, because deduplication compares whole rows and so needs all columns at once. Only Ordinary/Collapsing/Replacing/ VersionedCollapsing merging modes, because Summing/Aggregating/ Coalescing merge the values of equal keys and so have to merge the payload rather than gather it. At least 11 non-key columns (vertical_merge_algorithm_min_columns_to_activate) and at least 16·8192 rows (vertical_merge_algorithm_min_rows_to_activate) — below those, the extra machinery is not worth it. And at most 127 source parts, because RowSourcePart::MAX_PARTS is 0x7F and the one-byte encoding cannot express more.

The default thresholds live in MergeTreeSettings: enable_vertical_merge_algorithm = 1, vertical_merge_algorithm_min_rows_to_activate = 16 * 8192, vertical_merge_algorithm_min_bytes_to_activate = 0, and vertical_merge_algorithm_min_columns_to_activate = 11.

Note the framing of the setting description: “potentially at the cost of merge speed”. Vertical merging is a memory optimization, not a speed one. It trades IO locality (one column at a time, so each source file is opened and read once per column rather than once for all columns) for peak memory. For a table where one column dominates the size, that trade is usually right; for a table with many equally-sized columns, the re-reading can dominate.

The `Wide`-only restriction, and the escape hatch

The Wide restriction is the interesting one. A Compact part interleaves all columns of a row in a single .bin file, so “read only column X” is impossible — you would have to decode the whole file. The sources may still be Compact while the result is Wide, and that case is allowed by allow_vertical_merges_from_compact_to_wide_parts (default true):

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3784
 if (!(*merge_tree_settings)[MergeTreeSetting::allow_vertical_merges_from_compact_to_wide_parts])
 {
 for (const auto & part : global_ctx->future_part->parts)
 {
 if (!isWidePart(part))
 return MergeAlgorithm::Horizontal;
 }
 }

A Compact source is read once per column during the vertical stage, so its file gets decoded N times. That is expensive but it is bounded — and it buys the memory reduction for wide tables whose legacy parts happen to be Compact.

Progress accounting

A merge is a long-running background job, and system.merges shows a progress bar. For a vertical merge, “progress” spans two axes: first the key columns are merged (the horizontal stage), then each payload column is gathered (the vertical stage). The horizontal stage is weighted by the size of the key columns relative to the whole part; the vertical stage is weighted by the size of each individual column:

 cpp
 
 
// src/Storages/MergeTree/ColumnSizeEstimator.h:9
/* Allow to compute more accurate progress statistics */
class ColumnSizeEstimator
{
 using ColumnToSize = std::map<String, UInt64>;
 ColumnToSize map;
public:

 /// Stores approximate size of columns in bytes
 /// Exact values are not required since it used for relative values estimation (progress).
 size_t sum_total = 0;
 size_t sum_index_columns = 0;
 size_t sum_ordinary_columns = 0;

 ColumnSizeEstimator(ColumnToSize && map_, const NamesAndTypesList & key_columns, const NamesAndTypesList & ordinary_columns)
 : map(std::move(map_))
 {
 for (const auto & [name, _] : key_columns)
 if (!map.contains(name)) map[name] = 0;
 for (const auto & [name, _] : ordinary_columns)
 if (!map.contains(name)) map[name] = 0;

 for (const auto & [name, _] : key_columns)
 sum_index_columns += map.at(name);

 for (const auto & [name, _] : ordinary_columns)
 sum_ordinary_columns += map.at(name);

 sum_total = std::max(static_cast<decltype(sum_index_columns)>(1), sum_index_columns + sum_ordinary_columns);
 }

 Float64 columnWeight(const String & column) const
 {
 return static_cast<Float64>(map.at(column)) / static_cast<Float64>(sum_total);
 }

 Float64 keyColumnsWeight() const
 {
 return static_cast<Float64>(sum_index_columns) / static_cast<Float64>(sum_total);
 }
};

The weights are proportional to bytes, not rows, so a merge that is 90% one giant String column reports 90% progress for most of its duration and then jumps. That is the honest answer for a byte-oriented operation. The sizes come from the parts themselves (accumulateColumnSizes, a cheap per-part statistic), so the whole estimator costs one map construction per merge.

The horizontal stage’s contribution: the row count assertion

There is one more piece of rigor worth pointing out. After the horizontal stage finishes and the row-source file is closed, the code checks that the file contains exactly one byte per input row:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2055
 size_t rows_sources_count = ctx->rows_sources_temporary_file->finalizeWriting();

 /// In special case, when there is only one source part, and no rows were skipped, we may have
 /// skipped writing rows_sources file. Otherwise rows_sources_count must be equal to the total
 /// number of input rows.
 /// Note that only one byte index is written for each row, so number of rows is equals to the number of bytes written.
 if ((rows_sources_count > 0 || global_ctx->future_part->parts.size() > 1) && sum_input_rows_exact != rows_sources_count + input_rows_filtered)
 throw Exception(
 ErrorCodes::LOGICAL_ERROR,
 "Number of rows in source parts ({}) excluding filtered rows ({}) differs from number "
 "of bytes written to rows_sources file ({}). It is a bug.",
 sum_input_rows_exact, input_rows_filtered, rows_sources_count);

This is a cheap, total consistency check on the side channel. If the horizontal stage and the vertical stage ever disagree about which rows exist, the merge fails loudly rather than producing a part whose payload columns are misaligned with its key — which would be a silent data corruption. Note the subtlety in the condition: with exactly one source part and no filtered rows, the file may legitimately be empty (a single-source merge does not need to record where rows came from), so the check is skipped in that one case.

Prefetching the next column

Because the vertical stage reads one column at a time, and each column read is a separate pipeline over the source parts, the obvious optimization is to start reading column N+1 while column N is still being written. The code does exactly that, and only for remote filesystems, where the latency of the first read of a column is the dominant cost:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2076
 bool all_parts_on_remote_disks = std::ranges::all_of(global_ctx->future_part->parts, [](const auto & part) { return part->isStoredOnRemoteDisk(); });
 ctx->use_prefetch = all_parts_on_remote_disks && storage_settings[MergeTreeSetting::vertical_merge_remote_filesystem_prefetch];

 if (ctx->use_prefetch && ctx->it_name_and_type != global_ctx->gathering_columns.end())
 ctx->prepared_pipeline = createPipelineForReadingOneColumn(ctx->it_name_and_type->name);

The “prefetching” is simply that building the pipeline — which opens the parts and issues the first read requests — starts the IO. It is not a separate prefetch thread; it is eager pipeline construction, one column ahead (prepareVerticalMergeForOneColumn builds the next column’s pipeline while the current one runs).

The whole flow

 text
 
 
 source parts (Wide)
 ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
 │ k, a, b, c │ │ k, a, b, c │ │ k, a, b, c │
 └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
 │ │ │
 └────────────┬────┴─────────────────┘
 │ HORIZONTAL STAGE
 │ k-way merge of the key columns
 │ emits: merged k + rows_sources (1 byte/row)
 ▼
 ┌──────────────────────────────────────────┐
 │ rows_sources: 00 00 01 01 02 00 01 ... │
 └──────────────────────────────────────────┘
 │
 ┌─────────────────┼──────────────────┐
 ▼ ▼ ▼
 gather col a gather col b gather col c VERTICAL STAGE
 read a from read b from read c from one column at a time
 each part, each part, each part,
 copy by copy by copy by
 rows_sources rows_sources rows_sources
 │ │ │
 ▼ ▼ ▼
 └────────────┬────┴──────────────────┘
 ▼
 result part (Wide): k, a, b, c

What the hack buys you, and what it costs

It buys you peak memory proportional to the widest column, not the sum of the columns. For a table with a 40 MB String column and fifty UInt64 columns, a horizontal merge holds the payload of all the columns at once; a vertical merge holds one column at a time. That is the difference between a merge that OOMs and one that does not — and it is what makes the “wide part” format practical at all.

It costs you IO locality. Each column is a separate pass over the source parts, so a Compact source file is decoded once per column — which is why allow_vertical_merges_from_compact_to_wide_parts defaults to true but carries a warning in its description.

It costs you a temporary file. One byte per row of scratch space: 100 MB for a 100-million-row merge. Trivial compared to the data being merged, but not zero.

It costs you the ability to merge every kind of table. Summing, Aggregating and Coalescing merges have to combine the values of equal keys, which is a merge, not a gather. They stay horizontal, permanently.

It is not automatically faster. The setting description says it plainly: “potentially at the cost of merge speed”. The vertical algorithm optimizes memory, and only optimizes IO when a whole column happens to come from a single source part and can be stolen by pointer.

Seeing it in action

You can see which algorithm a merge used from the profile events:

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN (
 'MergeVerticalStageExecuteMilliseconds',
 'MergeHorizontalStageTotalMilliseconds',
 'MergeVerticalStageTotalMilliseconds'
);

MergeVerticalStageExecuteMilliseconds is only non-zero for tables where the vertical algorithm actually ran. If it is zero on your wide table, check enable_vertical_merge_algorithm, the part format (Compact parts cannot be vertically merged), and the vertical_merge_algorithm_min_* thresholds.

And to see the tables that would benefit:

 sql
 
 
SELECT
 database,
 table,
 formatReadableSize(sum(bytes_on_disk)) AS size,
 sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC
LIMIT 10;

A table with hundreds of columns and a large String payload is exactly the shape the vertical merge was built for. A table with ten narrow columns is not, and the thresholds will keep it horizontal.

How ClickHouse Marks Empty Hash Table Cells With Zeroed Memory

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every GROUP BY in ClickHouse is a hash table. The table has to hold millions of keys, be probed once per input row, and be resized while it is being filled — all while the query is competing for memory with everything else on the machine. A std::unordered_map is simply not competitive here: it allocates a node per element, keeps a pointer chain, and pays a pointer-chase for every collision.

So ClickHouse ships its own hash table in src/Common/HashTable/HashTable.h: a flat array of cells, open addressing, linear probing. The interesting part is not the probing — that is textbook — but the bookkeeping around emptiness. There is no “occupied” bit, no tombstone, no sentinel pointer, no version counter. A cell is empty when its key is all zero bytes, and the whole buffer starts out zeroed by the allocator.

That single decision shapes the entire file: the allocator, the cell layout, the way the zero key is handled, the way deletion works, and the reason the hash function has to be good.

The shape of the table

 cpp
 
 
// src/Common/HashTable/HashTable.h:518
template <typename Key, typename Cell, typename Hash, typename Grower, typename Allocator>
class HashTable : private boost::noncopyable,
 protected Hash,
 protected Allocator,
 protected Cell::State,
 public ZeroValueStorage<Cell::need_zero_value_storage, Cell> /// empty base optimization
{

The table is one contiguous buffer of Cells plus a Grower that knows the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:550
 size_t m_size = 0; /// Amount of elements
 Cell * buf; /// A piece of memory for all elements except the element with zero key.
 Grower grower;

Cell is a policy class. The simplest one stores nothing but a key:

 cpp
 
 
// src/Common/HashTable/HashTable.h:154
template <typename Key, typename Hash, typename TState = HashTableNoState>
struct HashTableCell
{
 using State = TState;

 using key_type = Key;
 using value_type = Key;
 using mapped_type = VoidMapped;

 Key key;

The map flavour stores a pair, and the variant that ClickHouse uses for aggregation additionally caches the hash so that a rehash does not have to recompute it:

 cpp
 
 
// src/Common/HashTable/HashMap.h:165
template <typename Key, typename TMapped, typename Hash, typename TState = HashTableNoState>
struct HashMapCellWithSavedHash : public HashMapCell<Key, TMapped, Hash, TState> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `saved_hash` is set by `setHash` immediately after placement construction on insert; on the hot aggregation/join path we must avoid the redundant store
{
 using Base = HashMapCell<Key, TMapped, Hash, TState>;

 size_t saved_hash;

Empty means zero

Here is the whole emptiness protocol:

 cpp
 
 
// src/Common/HashTable/HashTable.h:191
 /// Whether the key is zero. In the main buffer, cells with a zero key are considered empty.
 /// If zero keys can be inserted into the table, then the cell for the zero key is stored separately, not in the main buffer.
 /// Zero keys must be such that the zeroed-down piece of memory is a zero key.
 bool isZero(const State & state) const { return isZero(key, state); }
 static bool isZero(const Key & key, const State & /*state*/) { return ZeroTraits::check(key); }

 /// Set the key value to zero.
 void setZero() { ZeroTraits::set(key); }

and the generic part:

 cpp
 
 
// src/Common/HashTable/HashTable.h:89
namespace ZeroTraits
{

template <typename T>
bool check(const T x)
{
 return bitEquals(x, T{});
}

template <typename T>
void set(T & x) { x = T{}; }

}

bitEquals is bitwise, not ==, and that matters for floating point:

 cpp
 
 
// src/Common/HashTable/HashTable.h:77
template <typename T>
inline ALWAYS_INLINE bool bitEquals(T a, T b)
{
 if constexpr (is_floating_point<T>)
 /// Note that memcmp with constant size is a compiler builtin.
 return 0 == memcmp(&a, &b, sizeof(T)); /// NOLINT
 else
 return a == b;
}

The comment above it explains why bit equality is mandatory rather than merely nice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:71
/** Numbers are compared bitwise.
 * Complex types are compared by operator== as usual (this is important if there are gaps).
 *
 * This is needed if you use floats as keys. They are compared by bit equality.
 * Otherwise the invariants in hash table probing do not met when NaNs are present.
 */

NaN never equals itself, so a ==-based emptiness test could never recognise an empty float cell, and the probe loop would run off the end of the buffer.

The zeroed buffer comes from the allocator:

 cpp
 
 
// src/Common/HashTable/HashTableAllocator.h:8
/**
 * We are going to use the entire memory we allocated when resizing a hash
 * table, so it makes sense to pre-fault the pages so that page faults don't
 * interrupt the resize loop. Set the allocator parameter accordingly.
 */
using HashTableAllocator = Allocator<true /* clear_memory */, true /* populate */>;

clear_memory becomes calloc:

 cpp
 
 
// src/Common/Allocator.cpp:79
 if (alignment <= MALLOC_MIN_ALIGNMENT) [[likely]]
 {
 if constexpr (clear_memory)
 buf = __real_calloc(size, 1);
 else
 buf = __real_malloc(size);

So the invariant is: the bytes of a fresh buffer are zero, therefore a fresh cell is empty. Nothing more is needed to track occupancy. The diagram of the buffer is therefore very simple:

 text
 
 
 buf buf + bufSize()
 | |
 v v
 +--------+--------+--------+--------+--------+--------+--------+--------+
 | cell 0 | cell 1 | cell 2 | cell 3 | cell 4 | cell 5 | cell 6 | cell 7 |
 +--------+--------+--------+--------+--------+--------+--------+--------+
 key=A (zero) key=C (zero) key=B (zero) (zero)
 [used] [empty] [used] [empty] [used] [empty] [empty]

Note what is not there: no occupancy byte, no pointer, no version. A HashMapCell<UInt64, AggregateState*> is 16 bytes of pure payload; the empty/used distinction is free.

The zero key gets its own slot

There is one hole in the “zero means empty” scheme: 0 is a perfectly legal key. If it were stored in the buffer, it would be indistinguishable from an empty cell. So it is stored outside the buffer, in a ZeroValueStorage that the table inherits from:

 cpp
 
 
// src/Common/HashTable/HashTable.h:474
template <typename Cell>
struct ZeroValueStorage<true, Cell> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
private:
 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

 void setHasZero()
 {
 has_zero = true;
 new (zeroValue()) Cell();
 }

and the insert path routes the zero key to it:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1354
 bool ALWAYS_INLINE emplaceIfZero(const Key & x, LookupResult & it, bool & inserted, size_t hash_value)
 {
 /// If it is claimed that the zero key can not be inserted into the table.
 if constexpr (!Cell::need_zero_value_storage)
 return false;

 if (unlikely(Cell::isZero(x, *this)))
 {
 it = this->zeroValue();

 if (!this->hasZero())
 {
 ++m_size;
 this->setHasZero();
 this->zeroValue()->setHash(hash_value);
 inserted = true;
 }
 else
 inserted = false;

 return true;
 }

 return false;
 }

The zero key lives at a fixed address, so a lookup for it is a branch and a pointer, never a probe. The cost of the scheme is one bool plus one Cell of storage per table, and a slightly awkward iteration order (the zero cell is visited first, then the buffer).

This is not an exotic corner. 0 is a perfectly ordinary grouping key:

 sql
 
SELECT k, count() FROM t GROUP BY k;

with a k column full of zeros produces a group whose key is the zero key, and that group is the one stored in the separate slot.

Note also that the design is conditional: a cell type can declare that zero keys are impossible (need_zero_value_storage = false), and then ZeroValueStorage<false, Cell> degenerates to nothing at all. The string tables do exactly that — see the article on the string-specialised map.

Probing

The probe sequence is the simplest possible one: linear probing with a mask.

 cpp
 
 
// src/Common/HashTable/HashTable.h:232
 /// The size of the hash table in the cells.
 size_t bufSize() const { return 1ULL << size_degree; }

 size_t maxFill() const { return 1ULL << (size_degree - 1); }
 size_t mask() const { return bufSize() - 1; }

 /// From the hash value, get the cell number in the hash table.
 size_t place(size_t x) const { return x & mask(); }

 /// The next cell in the collision resolution chain.
 size_t next(size_t pos) const { ++pos; return pos & mask(); }

 /// Whether the hash table is sufficiently full. You need to increase the size of the hash table, or remove something unnecessary from it.
 bool overflow(size_t elems) const { return elems > maxFill(); }

The load factor is 0.5: maxFill() is half the buffer. And the probe loop itself:

 cpp
 
 
// src/Common/HashTable/HashTable.h:559
 size_t ALWAYS_INLINE findCell(const Key & x, size_t hash_value, size_t place_value) const
 {
 while (!buf[place_value].isZero(*this) && !buf[place_value].keyEquals(x, hash_value, *this))
 {
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }

 return place_value;
 }

Two things to note. First, isZero is checked before keyEquals: the loop stops at the first empty cell, because with linear probing an empty cell terminates the chain. Second, the comparison is a bitwise equality against the key, and for the HashMapCellWithSavedHash variant it is short-circuited by the cached hash.

The zero key is handled before the probe:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1617
 LookupResult ALWAYS_INLINE find(const Key & x)
 {
 if (Cell::isZero(x, *this))
 return this->hasZero() ? this->zeroValue() : nullptr;

 return findNonZeroCell(x, hash(x));
 }

Insertion

emplace computes the hash, routes the zero key, and otherwise finds a place:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1380
 template <typename KeyHolder>
 void ALWAYS_INLINE emplaceNonZeroImpl(size_t place_value, KeyHolder && key_holder,
 LookupResult & it, bool & inserted, size_t hash_value)
 {
 it = &buf[place_value];

 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);
 const auto & key = keyHolderGetKey(key_holder);

 new (&buf[place_value]) Cell(key, *this);
 buf[place_value].setHash(hash_value);
 inserted = true;
 ++m_size;

 if (unlikely(grower.overflow(m_size)))
 {
 try
 {
 resize();
 }
 catch (...)
 {
 /** If we have not resized successfully, then there will be problems.
 * There remains a key, but uninitialized mapped-value,
 * which, perhaps, can not even be called a destructor.
 */
 --m_size;
 buf[place_value].setZero();
 inserted = false;
 keyHolderDiscardKey(key_holder);
 throw;
 }

 // The hash table was rehashed, so we have to re-find the key.
 it = findNonZeroCell(key, hash_value);
 chassert(it);
 }
 }

Two details worth pausing on. The cell is placement-newed; the buffer is raw memory, and the destructor of a non-trivial cell is only run for cells that were actually constructed (destroyElements walks the table and calls the destructor only for non-zero cells).

And the resize happens after the insert, not before. That means the table can exceed its nominal capacity by one element, and it means the iterator has to be re-found afterwards, because the rehash moved the cell. The exception handler is careful to undo the insert (decrement the size, zero the cell) so that a failed allocation does not leave a half-built cell behind.

Growing: the wrap-around special case

The resize path is where the linear-probing invariants get hairy. When the buffer grows, every element has to be re-placed, and an element that was at the end of the old buffer may belong at the beginning of the new one because its collision chain wrapped around:

 cpp
 
 
// src/Common/HashTable/HashTable.h:837
 void resize(size_t for_num_elems = 0, size_t for_buf_size = 0)
 {
#ifdef DBMS_HASH_MAP_DEBUG_RESIZES
 Stopwatch watch;
#endif

 size_t old_size = grower.bufSize();

 /** In case of exception for the object to remain in the correct state,
 * changing the variable `grower` (which determines the buffer size of the hash table)
 * is postponed for a moment after a real buffer change.
 * The temporary variable `new_grower` is used to determine the new size.
 */
 Grower new_grower = grower;

and the re-placement loop:

 cpp
 
 
// src/Common/HashTable/HashTable.h:884
 size_t i = 0;
 for (; i < old_size; ++i)
 if (!buf[i].isZero(*this))
 reinsert(buf[i], buf[i].getHash(*this));

 /** There is also a special case:
 * if the element was to be at the end of the old buffer, [ x]
 * but is at the beginning because of the collision resolution chain, [o x]
 * then after resizing, it will first be out of place again, [ xo ]
 * and in order to transfer it where necessary,
 * after transferring all the elements from the old halves you need to [ o x ]
 * process tail from the collision resolution chain immediately after it [ o x ]
 */
 size_t new_size = grower.bufSize();
 for (; i < new_size && !buf[i].isZero(*this); ++i)
 reinsert(buf[i], buf[i].getHash(*this));

The picture in that comment is the whole problem: after the buffer is doubled, the element at the very end of the old buffer (x) is now in the middle of the new one, and the element that was pushed behind it (o) belongs before it. The second loop therefore keeps going past the end of the old buffer, as long as the cells it finds are non-empty, re-inserting them.

The re-insertion itself is a memcpy of a trivially-copyable cell into the new position:

 cpp
 
 
// src/Common/HashTable/HashTable.h:918
 void reinsert(Cell & x, size_t hash_value)
 {
 size_t place_value = grower.place(hash_value);

 /// If the element is in its place.
 if (&x == &buf[place_value])
 return;

 /// Compute a new location, taking into account the collision resolution chain.
 place_value = findCell(Cell::getKey(x.getValue()), hash_value, place_value);

 /// If the item remains in its place in the old collision resolution chain.
 if (!buf[place_value].isZero(*this))
 return;

 /// Copy to a new location and zero the old one.
 x.setHash(hash_value);
 memcpy(static_cast<void*>(&buf[place_value]), &x, sizeof(x)); /// NOLINT(bugprone-undefined-memory-manipulation)
 x.setZero();
 }

The grower doubles the size until max_size_degree (23) and then grows by one step at a time:

 cpp
 
 
// src/Common/HashTable/HashTable.h:247
 /// Increase the size of the hash table.
 void increaseSize()
 {
 size_degree += size_degree >= max_size_degree ? 1 : 2;
 }

Robin Hood: what changed recently

The most interesting recent addition to the file is a second probing discipline. The classic HashTableGrower keeps the load factor at 0.5; the new HashTableGrowerWithPrecalculationAndRobinHood keeps 0.5 until the table reaches 2^16 cells, and then switches to Robin Hood hashing with a 90% load factor — in place, without reallocating:

 cpp
 
 
// src/Common/HashTable/HashTable.h:336
/** Like `HashTableGrowerWithPrecalculation`, but adapts to large hash tables.
 *
 * For large hash tables, random memory access is the main bottleneck, not the computations.
 * When the buffer reaches 2^robin_hood_size_degree cells, the grower switches the hash table to the Robin Hood mode:
 * the max load factor is increased from 0.5 to robin_hood_max_load_factor_percent / 100 (the hash table becomes up to two times smaller),
 * and the elements inside collision resolution chains are ordered by their optimal positions,
 * so a lookup of a missing key can stop early instead of walking to the end of the chain.
 *
 * The memory layout is the same in both modes, and switching happens in place, without growing the buffer.
 * Hash tables check the mode only if the grower has the `isRobinHood` method (see `GrowerWithRobinHood`),
 * so there is no cost for other growers.
 *
 * Requirements to the cell: obtaining the hash value with `getHash` must be cheap (the hash is saved or cheap to compute),
 * and the hash table must not keep pointers to cells while inserting other elements,
 * because an insertion may move other elements along the collision resolution chain.
 */

The payoff is twofold. The table becomes up to twice as small (0.9 vs 0.5 load factor), which halves the number of cache misses in a random probe; and a failed lookup can stop early, because the chain is sorted by “distance from the ideal position”:

 cpp
 
 
// src/Common/HashTable/HashTable.h:607
 size_t ALWAYS_INLINE findCellRobinHood(const Key & x, size_t hash_value, size_t place_value, bool & found) const
 {
 for (size_t distance = 0;; ++distance)
 {
 const Cell & cell = buf[place_value];
 if (cell.isZero(*this))
 {
 found = false;
 return place_value;
 }
 if (cell.keyEquals(x, hash_value, *this))
 {
 found = true;
 return place_value;
 }
 /// The element at the optimal position of the key cannot be closer to its own optimal position, so do not compute its hash.
 if (distance && distanceFromOptimalPosition(cell, place_value) < distance)
 {
 found = false;
 return place_value;
 }
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }
 }

“Distance from the optimal position” is just the difference between where the key would go and where it is, measured modulo the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:597
 size_t ALWAYS_INLINE distanceFromOptimalPosition(const Cell & cell, size_t place_value) const
 {
 return grower.place(place_value - grower.place(cell.getHash(*this)));
 }

Robin Hood ordering also changes insertion: the new key may have to be placed before an existing one, which means shifting the rest of the chain to the right:

 cpp
 
 
// src/Common/HashTable/HashTable.h:664
 void NO_INLINE shiftCellsRight(size_t place_value)
 {
 size_t empty_place = findEmptyCell(place_value);

 if (empty_place < place_value)
 {
 /// The chain wraps around the end of the buffer.
 size_t last_place = grower.bufSize() - 1;
 memmove(static_cast<void *>(&buf[1]), static_cast<const void *>(&buf[0]), empty_place * sizeof(Cell));
 memcpy(static_cast<void *>(&buf[0]), static_cast<const void *>(&buf[last_place]), sizeof(Cell));
 empty_place = last_place;
 }

 memmove(static_cast<void *>(&buf[place_value + 1]), static_cast<const void *>(&buf[place_value]), (empty_place - place_value) * sizeof(Cell));
 buf[place_value].setZero();
 }

and the chains have to be re-sorted when the mode is switched on — by an insertion sort, since the chains are “already almost ordered” in practice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:681
 /** Reorder the elements in every collision resolution chain by their optimal positions, as required in the Robin Hood mode.
 * Used after the elements were placed by the ordinary linear probing: on resize, or when switching to the Robin Hood mode.
 * It is an insertion sort, which is fast for the chains that are already almost ordered.
 */

Deletion without tombstones

Deleting from an open-addressing table usually means writing a tombstone, so that the probe chain is not cut short. Tombstones accumulate, they need to be cleaned up, and they slow down every subsequent lookup.

ClickHouse does not use tombstones. Instead it moves the elements of the chain:

 cpp
 
 
// src/Common/HashTable/HashTable.h:731
 /// Remove the element at place_value without calling its destructor, moving the following elements of the collision resolution chain to keep them reachable.
 /// Not inlined to keep the code of inserts small.
 void eraseCellAt(size_t erased_key_position)
 {
 /// We need to guarantee loop termination because there will be empty position
 chassert(m_size < grower.bufSize());

 size_t next_position = erased_key_position;

 /**
 * During element deletion there is a possibility that the search will be broken for one
 * of the following elements, because this place erased_key_position is empty. We will check
 * next_element. Consider a sequence from (erased_key_position, next_element], if the
 * optimal_position of next_element falls into it, then removing erased_key_position
 * will not break search for next_element.
 * If optimal_position of the element does not fall into the sequence (erased_key_position, next_element]
 * then deleting a erased_key_position will break search for it, so we need to move next_element
 * to erased_key_position. Now we have empty place at next_element, so we apply the identical
 * procedure for it.
 * If an empty element is encountered then means that there is no more next elements for which we can
 * break the search so we can exit.
 *
 * In the Robin Hood mode, the elements of a chain are ordered by their optimal positions,
 * and this procedure moves them one cell to the left, keeping the order.
 */

The rule the comment spells out is the classic one: an element may stay where it is if its ideal position is not before the hole; otherwise it has to slide left into the hole, which moves the hole one step further right, and the procedure repeats:

 cpp
 
 
// src/Common/HashTable/HashTable.h:798
 /// Move the element to the freed place
 memcpy(static_cast<void *>(&buf[erased_key_position]), static_cast<void *>(&buf[next_position]), sizeof(Cell));

 /// Now we have another freed place
 erased_key_position = next_position;
 }

 buf[erased_key_position].setZero();
 --m_size;
 }

This costs a memcpy per displaced element but keeps the table free of garbage: the buffer is always exactly as full as m_size says, and a probe chain never walks over a tombstone.

The allocator: zeroed and pre-faulted

HashTableAllocator is Allocator<true /* clear_memory */, true /* populate */>. The second flag is the interesting one: it pre-faults the pages of a large allocation with MADV_POPULATE_WRITE, so that the resize loop does not stop for page faults:

 cpp
 
 
// src/Common/Allocator.cpp:59
void prefaultPages([[maybe_unused]] void * buf_, [[maybe_unused]] size_t len_)
{
#if defined(MADV_POPULATE_WRITE)
 if (len_ < POPULATE_THRESHOLD)
 return;

 if (!is_supported_by_kernel) [[unlikely]]
 return;

 auto [buf, len] = adjustToPageSize(buf_, len_, ::getPageSize());
 ::madvise(buf, len, MADV_POPULATE_WRITE);
#endif
}

 cpp
 
 
// src/Common/Allocator.cpp:132
/// Constant is chosen almost arbitrarily, what I observed is 128KB is too small, 1MB is almost indistinguishable from 64MB and 1GB is too large.
extern const size_t POPULATE_THRESHOLD = std::max(Int64{16 * 1024 * 1024}, ::getPageSize());

The rationale is spelled out in HashTableAllocator.h itself: the whole allocation will be used, so pre-faulting it costs nothing extra and removes a source of latency spikes from the middle of a rehash. The kernel support check is a version test (Linux 5.14+), not an EINVAL probe, because madvise returns EINVAL for many unrelated reasons.

Note that calloc and the “zero means empty” invariant fit together perfectly: calloc of a large block is free on Linux for big allocations (the kernel hands out zero pages), so the “clear the buffer” requirement that the emptiness protocol imposes costs nothing at all.

Prefetching

A hash table probe is a random memory access; at 90% load factor with Robin Hood ordering, a lookup of a missing key can touch several cache lines. ClickHouse tries to hide that latency by prefetching the cell a few iterations ahead. The helper measures its own iteration latency to decide how far ahead to look:

 cpp
 
 
// src/Common/HashTable/Prefetching.h:33
class PrefetchingHelper
{
public:
 size_t calcPrefetchLookAhead()
 {
 static constexpr auto assumed_load_latency_ns = 100;
 static constexpr auto just_coefficient = 4;
 static constexpr auto numerator = just_coefficient * assumed_load_latency_ns * iterations_to_measure;
 size_t elapsed_ns = watch.elapsedNanoseconds();
 if (unlikely(elapsed_ns == 0))
 return max_look_ahead_value;
 size_t look_ahead = (numerator + elapsed_ns - 1) / elapsed_ns;
 return std::clamp<size_t>(look_ahead, min_look_ahead_value, max_look_ahead_value);
 }

 static constexpr size_t getInitialLookAheadValue() { return min_look_ahead_value; }

 static constexpr size_t iterationsToMeasure() { return iterations_to_measure; }

private:
 static constexpr size_t iterations_to_measure = 100;
 static constexpr size_t min_look_ahead_value = 4;
 static constexpr size_t max_look_ahead_value = 32;

The look-ahead is thus between 4 and 32 cells, tuned to the machine: the formula is 4 * 100ns * 100 / elapsed, i.e. four times the assumed memory latency, expressed in iterations.

The prefetching iterator of the table uses it, and also prefetches the key bytes of the cell it is about to look at, not just the cell:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1021
 void prefetch()
 {
 static_assert(CouldPrefetchKey<cell_type>);
 auto * buf_end = container->buf + container->grower.bufSize();
 iter_count++;
 if (prefetch_ptr < buf_end && prefetched_count < prefetch_ahead) [[likely]]
 {
 if (iter_count == DB::PrefetchingHelper::iterationsToMeasure()) [[unlikely]]
 prefetch_ahead = prefetching.calcPrefetchLookAhead();
 auto n = prefetch_ahead - prefetched_count;
 cell_type * last_ptr = nullptr;
 for (size_t i = 0; i < n; ++i)
 {
 while (prefetch_ptr < buf_end && prefetch_ptr->isZero(*container))
 ++prefetch_ptr;

 if (prefetch_ptr < buf_end) [[likely]]
 {
 last_ptr = prefetch_ptr;
 prefetch_ptr++;
 prefetched_count++;
 }
 else
 break;
 }

 if (last_ptr) [[likely]]
 keyPrefetch(last_ptr->getKey());
 }
 }

keyPrefetch touches up to four cache lines of the key data:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:182
inline void keyPrefetch(const std::string_view key)
{
 const size_t cache_line_size = 64;
 // Prefetch up to 4 cache lines.
 const size_t max_prefetch = cache_line_size * 4;
 const size_t n = std::min(key.size(), max_prefetch);
 const char * ptr = key.data();
 const char * end = ptr + n;
 for (; ptr < end; ptr += cache_line_size)
 {
 __builtin_prefetch(ptr);
 }
}

Keys that live outside the table

Not every key can live inside the cell. A std::string_view key points at data owned by somebody else — often a column that will be gone by the time the aggregation is finalised. ClickHouse solves this with a key holder: a small policy object that knows how to make a temporary key persistent, and when to throw it away.

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:53
/**
 * Returns the key. Can return the temporary key initially.
 * After the call to keyHolderPersistKey(), must return the persistent key.
 */
template <typename Key>
inline Key & ALWAYS_INLINE keyHolderGetKey(Key && key) { return key; } /// NOLINT(bugprone-return-const-ref-from-parameter)

/**
 * Make the key persistent. keyHolderGetKey() must return the persistent key
 * after this call.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderPersistKey(Key &&) {}

/**
 * Discard the key. Calling keyHolderGetKey() is ill-defined after this.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderDiscardKey(Key &&) {}

The default is a no-op: for a key stored by value, there is nothing to do, and the branch is optimised away. The ArenaKeyHolder version copies the bytes into an Arena when the key turns out to be new:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:98
inline void ALWAYS_INLINE keyHolderPersistKey(DB::ArenaKeyHolder & holder)
{
 // Normally, our hash table shouldn't ask to persist a zero key,
 // but it can happened in the case of clearable hash table (ClearableHashSet, for example).
 // The clearable hash table doesn't use zero storage and
 // distinguishes empty keys by using cell version, not the value itself.
 // So, when an empty std::string_view is inserted in ClearableHashSet we'll get here key of zero size.
 holder.key = std::string_view{holder.pool.insert(holder.key.data(), holder.key.size()), holder.key.size()};
}

The elegance is in where the decision is made. The hash table knows whether the key it just found was new; the caller does not. So the table calls keyHolderDiscardKey when the key already existed (nothing to persist, and the arena allocation is rolled back) and keyHolderPersistKey when it did not:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1386
 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);

The hash function has to be good

Because the table uses linear probing and the low bits of the hash to pick the bucket, a poor hash function is catastrophic: the low bits of a timestamp-like key are nearly constant, and every key lands in the same few cells. The header says so bluntly:

 cpp
 
 
// src/Common/HashTable/Hash.h:15
/** Hash functions that are better than the trivial function std::hash.
 *
 * Example: when we do aggregation by the visitor ID, the performance increase is more than 5 times.
 * This is because of following reasons:
 * - in Metrica web analytics system, visitor identifier is an integer that has timestamp with seconds resolution in lower bits;
 * - in typical implementation of standard library, hash function for integers is trivial and just use lower bits;
 * - traffic is non-uniformly distributed across a day;
 * - we are using open-addressing linear probing hash tables that are most critical to hash function quality,
 * and trivial hash function gives disastrous results.
 */

The default integer hash is the MurmurHash finalizer:

 cpp
 
 
// src/Common/HashTable/Hash.h:26
/** Taken from MurmurHash. This is Murmur finalizer.
 * Faster than intHash32 when inserting into the hash table UInt64 -> UInt64, where the key is the visitor ID.
 */
inline UInt64 intHash64(UInt64 x)
{
 x ^= x >> 33;
 x *= 0xff51afd7ed558ccdULL;
 x ^= x >> 33;
 x *= 0xc4ceb9fe1a85ec53ULL;
 x ^= x >> 33;

 return x;
}

Five instructions and two multiplies turn “the low bits are constant” into “the low bits are random”. For string keys and general keys, the CRC32 hardware instruction is used where it exists — three cycles of latency and one per cycle of throughput, which is why the string tables use it even though CRC32 is a poor hash in the abstract:

 cpp
 
 
// src/Common/HashTable/Hash.h:40
/** CRC32C is not very high-quality as a hash function,
 * according to avalanche and bit independence tests (see SMHasher software), as well as a small number of bits,
 * but can behave well when used in hash tables,
 * due to high speed (latency 3 + 1 clock cycle, throughput 1 clock cycle).
 * Works only with SSE 4.2 support.
 */

Trade-offs

What the “zero means empty” scheme buys:

Zero bytes of metadata per cell. A HashMapCell<UInt64, T*> is 16 bytes, all of it payload. A scheme with an occupancy bit would need to steal a bit from the key or add a byte.
A free calloc. The kernel can hand out pre-zeroed pages; the emptiness protocol costs nothing at allocation time.
No tombstones, ever. Deletion moves cells instead of marking them, so the table never degrades.
A cheap scan. for (auto & cell : buf) if (!cell.isZero()) ... is the iteration order; no indirection, no linked list.

What it costs:

The key must be zero-comparable, and bit-comparable. Floating point keys are compared with memcmp, and NaN semantics have to be papered over.
The zero key is special. It lives in a separate slot, it is visited first during iteration, and its handling adds a branch to every insert and lookup.
The hash must be excellent, because linear probing amplifies clustering, and the low bits of the hash are used directly.
Rehashing is a memcpy storm. Growing the table means moving every cell, and the wrap-around case needs a second pass over the head of the new buffer.

And the recent Robin Hood addition changes the balance again: it trades a little CPU (reordering chains, occasionally shifting cells on insert) for a halving of the memory footprint at large sizes — which, for a table that is probed randomly, is exactly the right trade, because the cost is cache misses, not instructions.

How ClickHouse JIT-Compiles Scalar Expressions with LLVM

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Every columnar database pays the same tax: a SELECT is not a loop over rows, it is a graph of actions, and every action dispatches through a virtual call per block. The block is the unit of work in ClickHouse (65 409 rows by default), so the dispatch overhead is paid 65 409 times less often than in a row-at-a-time engine — but for a narrow expression like a + b > c the arithmetic itself is a couple of instructions per row, and the interpreter’s per-node costs (a virtual dispatch, a column-type switch, a ColumnPtr copy) dominate.

ClickHouse’s answer is a JIT compiler that is on by default since 25.5:

 cpp
 
 
// src/Core/Settings.cpp:1459
 DECLARE(Bool, compile_expressions, true, R"(
Compile some scalar functions and operators to native code.
)", 0, \
 {"25.5", false, true, "We believe that the LLVM infrastructure behind the JIT compiler is stable enough to enable this setting by default."}) \

This article follows the whole path: from an ActionsDAG node, through a small intermediate representation (CompileDAG), into hand-built LLVM IR, through LLVM’s O3 pipeline, into an executable page, and finally into a tight loop that runs over raw column data.

The switch

The JIT is not free: compiling a module costs milliseconds. So ClickHouse only compiles an expression after it has been seen a few times. The threshold is a setting, default 3:

 cpp
 
 
// src/Core/Settings.cpp:1463
 DECLARE(UInt64, min_count_to_compile_expression, 3, R"(
Minimum count of executing same expression before it is get compiled.
)", 0) \

The counter is a global hash map keyed by a hash of the expression DAG, so the “same expression” test is structural, not textual:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:299
static FunctionBasePtr compile(
 const CompileDAG & dag,
 size_t min_count_to_compile_expression)
{
 static std::unordered_map<UInt128, UInt64, UInt128Hash> counter;
 static std::mutex mutex;

 auto hash_key = dag.hash();
 {
 std::lock_guard lock(mutex);
 if (counter[hash_key]++ < min_count_to_compile_expression)
 return nullptr;
 }

Note the ++ inside the comparison: the counter is incremented on every call, and the expression is compiled on the fourth sighting (with the default of 3). The counter never decays, so a hot expression stays hot forever.

The trigger point is the construction of ExpressionActions, which is where a DAG becomes an executable list of actions:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:74
#if USE_EMBEDDED_COMPILER
 if (settings.can_compile_expressions && settings.compile_expressions == CompileExpressions::yes)
 actions_dag.compileExpressions(settings.min_count_to_compile_expression, lazy_executed_nodes);
#endif

Note the ordering: short-circuit functions are analysed first and the set of “lazily executed” nodes is computed before compiling, because a node that is only evaluated under a condition must not be folded into a compiled straight-line expression.

CompileDAG: a tiny IR

ActionsDAG is the general-purpose graph ClickHouse uses for every expression: it carries column nodes, aliases, lambdas, arrays, and functions. It is too rich to translate to LLVM directly, so the JIT first flattens it into CompileDAG, which has exactly three kinds of node:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.h:36
 enum class CompileType : uint8_t
 {
 INPUT = 0,
 CONSTANT = 1,
 FUNCTION = 2,
 };

The flattening is a post-order DFS over the ActionsDAG that stops at anything it cannot compile. getCompilableDAG in ExpressionJIT.cpp walks the DAG with an explicit stack and produces, for each candidate node, a CompileDAG plus the list of children that became inputs:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:404
static CompileDAG getCompilableDAG(
 const ActionsDAG::Node * root,
 ActionsDAG::NodeRawConstPtrs & children,
 const std::unordered_set<const ActionsDAG::Node *> & lazy_executed_nodes)
{
 /// Extract CompileDAG from root actions dag node.

 CompileDAG dag;

 std::unordered_map<const ActionsDAG::Node *, size_t> visited_node_to_compile_dag_position;

A node is a leaf of the compiled fragment when it is a constant, when it is not compilable at all, or when its function declares that the argument does not participate in compilation:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:430
 bool is_compilable_constant = isCompilableConstant(*node);
 bool is_compilable_function = isCompilableFunction(*node, lazy_executed_nodes);

 if (!is_compilable_function || is_compilable_constant || frame.skip_compile)
 {
 CompileDAG::Node compile_node;
 compile_node.function = node->function_base;
 compile_node.result_type = node->result_type;

 if (is_compilable_constant)
 {
 compile_node.type = CompileDAG::CompileType::CONSTANT;
 compile_node.column = node->column;
 }
 else if (frame.skip_compile)
 {
 compile_node.type = CompileDAG::CompileType::CONSTANT;
 compile_node.column = node->column;
 compile_node.skip_compile = true;
 }
 else
 {
 compile_node.type = CompileDAG::CompileType::INPUT;
 children.emplace_back(node);
 }

The skip_compile flag is how “arguments that don’t participate” are handled: the argument is still passed to the function at runtime (the C++ function signature has all its arguments), but its value is not computed by the compiled code. This is what makes if(cond, f(x), g(x))-style functions compile at all: the lazily evaluated branch is a runtime argument, not a compiled computation.

Choosing what to compile

ActionsDAG::compileFunctions is the part that decides which nodes become compiled fragments. It is not “compile everything”: it computes, for each node, how many compilable descendants it has, and only compiles nodes that have at least one compilable child and whose parents are not themselves compilable (so a whole subtree is compiled once, at its root, not piece by piece):

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:594
 for (const auto & node : nodes)
 {
 auto & node_data = node_to_data[&node];
 bool node_is_valid_for_compilation = node_data.is_compilable_in_isolation && node_data.compilable_children_size > 0;

 for (const auto & child : node.children)
 node_to_data[child].all_parents_compilable &= node_is_valid_for_compilation;
 }

 for (const auto & output_node : outputs)
 {
 /// Force output nodes to compile
 node_to_data[output_node].all_parents_compilable = false;
 }

Then the fragments are sorted so that a parent is compiled before its children — otherwise the same subexpression would be compiled twice under two different names:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:626
 /** Sort nodes before compilation using their children size to avoid compiling subexpression before compile parent expression.
 * This is needed to avoid compiling expression more than once with different names because of compilation order.
 */
 ::sort(nodes_to_compile.begin(), nodes_to_compile.end(), [&](const Node * lhs, const Node * rhs)
 {
 return node_to_data[lhs].children_size > node_to_data[rhs].children_size;
 });

and the compiled node replaces the original in place:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:649
 node->type = ActionsDAG::ActionType::FUNCTION;
 node->function_base = fn;
 node->function = fn->prepare(arguments);
 node->children.swap(new_children);
 node->is_function_compiled = true;
 node->column = nullptr;

From CompileDAG to LLVM IR

CompileDAG::compile emits the IR for the fragment. For a function node it recursively materialises the argument values and asks the function itself to emit its own IR:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.cpp:45
 case CompileType::FUNCTION:
 {
 ValuesWithType temporary_values;
 temporary_values.reserve(node.arguments.size());
 for (auto argument_index : node.arguments)
 temporary_values.emplace_back(compiled_values[argument_index]);

 ValueWithType compiled_value{node.function->compile(builder, temporary_values), node.function->getResultType()};
 if (!node.result_type->equals(*node.function->getResultType()))
 compiled_values[compiled_values_index] = {nativeCast(b, compiled_value, node.result_type), node.result_type};
 else
 compiled_values[compiled_values_index] = std::move(compiled_value);

 break;
 }

Constants are baked into the module as native values, inputs are the loop’s loaded values, and functions call IFunction::compile.

The function contract

Each SQL function that wants to be JIT-compilable implements compileImpl, which receives the already-typed argument values as LLVM values and returns an LLVM value. A binary arithmetic function looks like this:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3821
 llvm::Value * compileImpl(llvm::IRBuilderBase & builder, const ValuesWithType & arguments, const DataTypePtr & result_type) const override
 {
 chassert(2 == arguments.size());

 auto denull_left_type = removeNullable(arguments[0].type);
 auto denull_right_type = removeNullable(arguments[1].type);
 llvm::Value * result = nullptr;

and ends with the actual operation, dispatched on the native type:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3846
 auto & b = static_cast<llvm::IRBuilder<> &>(builder);
 auto * lval = nativeCast(b, arguments[0], result_type);
 auto * rval = nativeCast(b, arguments[1], result_type);
 result = OpSpec::compile(b, lval, rval, is_signed_v<typename ResultDataType::FieldType>);
 return true;

NULLs: the ternary trick

Nullable(T) is represented in IR as a two-element struct { T value, UInt8 is_null }. Handling NULLs in compiled code is where most of the complexity lives, because ClickHouse’s default implementation for NULLs is “if any argument is NULL, the result is NULL” — which is not what and/or want (three-valued logic).

The framework’s answer is to compile the denulled function and then OR the null flags back in:

 cpp
 
 
// src/Functions/IFunction.cpp:1281
 auto denulled_arguments_types = removeNullables(arguments_types);
 if (useDefaultImplementationForNulls() && denulled_arguments_types)
 {
 auto & b = static_cast<llvm::IRBuilder<> &>(builder);

 ValuesWithType unwrapped_arguments;
 unwrapped_arguments.reserve(arguments.size());

 VectorWithMemoryTracking<llvm::Value *> is_null_values;

 auto skip_arguments = getArgumentsThatDontParticipateInCompilation(arguments_types);
 for (size_t i = 0; i < arguments.size(); ++i)
 {
 const auto & argument = arguments[i];
 llvm::Value * unwrapped_value = argument.value;

 bool skip_compile = (std::find(skip_arguments.begin(), skip_arguments.end(), i) != skip_arguments.end());
 if (argument.type->isNullable())
 {
 unwrapped_value = skip_compile ? nullptr : b.CreateExtractValue(argument.value, {0});
 is_null_values.emplace_back(skip_compile ? nullptr : b.CreateExtractValue(argument.value, {1}));
 }

 unwrapped_arguments.emplace_back(unwrapped_value, (*denulled_arguments_types)[i]);
 }

and then, at the end, the null flags are merged into the result:

 cpp
 
 
// src/Functions/IFunction.cpp:1317
 for (auto * is_null_value : is_null_values)
 {
 if (is_null_value)
 nullable_structure_result_null = b.CreateOr(nullable_structure_result_null, is_null_value);
 }

 return b.CreateInsertValue(nullable_structure_with_result_value, nullable_structure_result_null, {1});

Functions with special NULL semantics (the logical ones) opt out of this and do the work themselves, using an explicit three-valued encoding:

 cpp
 
 
// src/Functions/FunctionsLogical.h:267
 llvm::Value * compileImpl(llvm::IRBuilderBase & builder, const ValuesWithType & values, const DataTypePtr & result_type) const override
 {
 chassert(!values.empty());

 auto & b = static_cast<llvm::IRBuilder<> &>(builder);
 if constexpr (Impl::specialImplementationForNulls())
 {
 if (result_type->isNullable())
 {
 /// First we need to cast all values to ternary logic
 llvm::Value * ternary_result = nativeTernaryCast(b, values[0]);
 for (size_t i = 1; i < values.size(); ++i)
 {
 llvm::Value * casted_value = nativeTernaryCast(b, values[i]);
 ternary_result = Impl::ternaryApply(b, ternary_result, casted_value);
 }

 /// Then transform ternary logic to struct which represents nullable result
 llvm::Value * is_null = b.CreateICmpEQ(ternary_result, b.getInt8(Ternary::Null));
 llvm::Value * is_true = b.CreateICmpEQ(ternary_result, b.getInt8(Ternary::True));

This is a nice illustration of the general ClickHouse pattern: the interpreter path handles NULLs with a nested column representation and a mask, the compiled path handles them with a struct and a select — the semantics are the same, the representation is not.

The generated loop

The whole pipeline, from a plan node to a running loop:

 text
 
 
 ActionsDAG node CompileDAG (3 node kinds)
 (Function, Column, Alias, ...) INPUT column to read
 | CONSTANT baked-in literal
 | getCompilableDAG() FUNCTION nested call
 v
 CompileDAG --hash()--> SipHash-128 --min_count_to_compile_expression--> ?
 | (seen 4 times?)
 v compile()
 llvm::Module (IRBuilder: entry / loop / end blocks)
 |
 v O3 pipeline + RuntimeDyld
 executable page (PROT_READ|PROT_EXEC) <---- CompiledExpressionCache (weighted by code size)
 |
 v callJITFunction(rows, columns)
 for (i = 0; i < rows; ++i) { result[i] = expr(columns[0][i], columns[1][i]); }

The IR builder in compileFunction.cpp wraps the expression in a loop over the rows. The function’s ABI is void f(size_t rows, ColumnData * columns), where ColumnData is a { const char * data, const char * null_data, const char * offsets } triple — the raw pointers of the column, its null map, and (for strings) its offsets:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:88
static llvm::StructType * buildColumnDataStruct(llvm::IRBuilderBase & b)
{
 auto * char_ptr_type = b.getInt8Ty()->getPointerTo();
 return llvm::StructType::get(char_ptr_type, char_ptr_type, char_ptr_type);
}

The function itself is created with ExternalLinkage so that the linker can find it by name later, and the loop is a textbook phi-node counter:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:139
 /// Initialize loop

 auto * end = llvm::BasicBlock::Create(b.getContext(), "end", func);
 auto * loop = llvm::BasicBlock::Create(b.getContext(), "loop", func);
 b.CreateCondBr(b.CreateICmpEQ(rows_count_arg, llvm::ConstantInt::get(size_type, 0)), end, loop);

 b.SetInsertPoint(loop);

 /// Loop

 auto * counter_phi = b.CreatePHI(rows_count_arg->getType(), 2);
 counter_phi->addIncoming(llvm::ConstantInt::get(size_type, 0), entry);

Each argument is loaded from the column data with an in-bounds GEP on the row index:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:162
 auto * column_data_ptr = column.data_ptr;
 auto * column_element_value = b.CreateLoad(column.data_element_type, b.CreateInBoundsGEP(column.data_element_type, column_data_ptr, counter_phi));

 if (!type->isNullable())
 {
 arguments.emplace_back(column_element_value, type);
 continue;
 }

 auto * column_is_null_element_value = b.CreateLoad(b.getInt8Ty(), b.CreateInBoundsGEP(b.getInt8Ty(), column.null_data_ptr, counter_phi));
 auto * is_null = b.CreateICmpNE(column_is_null_element_value, b.getInt8(0));
 auto * nullable_unitialized = llvm::Constant::getNullValue(toNullableType(b, column.data_element_type));
 auto * nullable_value = b.CreateInsertValue(b.CreateInsertValue(nullable_unitialized, column_element_value, {0}), is_null, {1});
 arguments.emplace_back(nullable_value, type);

and the result is stored back into the output column, which is passed as the last element of the columns array:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:184
 auto * result_column_element_ptr = b.CreateInBoundsGEP(columns.back().data_element_type, columns.back().data_ptr, counter_phi);

 if (columns.back().null_data_ptr)
 {
 b.CreateStore(b.CreateExtractValue(result, {0}), result_column_element_ptr);
 auto * result_column_is_null_element_ptr = b.CreateInBoundsGEP(b.getInt8Ty(), columns.back().null_data_ptr, counter_phi);
 auto * is_result_column_element_null = b.CreateSelect(b.CreateExtractValue(result, {1}), b.getInt8(1), b.getInt8(0));
 b.CreateStore(is_result_column_element_null, result_column_is_null_element_ptr);
 }
 else
 {
 b.CreateStore(result, result_column_element_ptr);
 }

Running it

At runtime, LLVMExecutableFunction::executeImpl allocates an output column, gathers the raw pointers, and calls into the compiled code:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:99
 ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & result_type, size_t input_rows_count) const override
 {
 if (!canBeNativeType(*result_type))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "LLVMExecutableFunction unexpected result type in: {}", result_type->getName());

 auto result_column = result_type->createUninitializedColumnWithSize(input_rows_count);

 if (input_rows_count)
 {
 std::vector<ColumnData> columns(arguments.size() + 1);
 Columns columns_backup;

 for (size_t i = 0; i < arguments.size(); ++i)
 {
 auto column = arguments[i].column->convertToFullColumnIfConst();
 columns_backup.emplace_back(column);
 columns[i] = getColumnData(column.get());
 }

 columns[arguments.size()] = getColumnData(result_column.get());

 auto jit_compiled_function = compiled_function_holder->compiled_function.compiled_function;
 callJITFunction(jit_compiled_function, input_rows_count, columns.data());

getColumnData is where the layout knowledge lives — a String column is not a contiguous array of values, it is a chars buffer plus an offsets buffer, and the JIT comparator indexes both directly:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:63
 /// skip null key data for one nullable key optimization
 if (WhichDataType(column->getDataType()).isString())
 {
 /// For String columns, use base pointers to chars and offsets arrays directly.
 /// The JIT comparator uses absolute row indices into these arrays, so both
 /// pointers must be to the start of their respective arrays regardless of skip_rows.
 const auto * string_column = assert_cast<const ColumnString *>(column);
 result.data = reinterpret_cast<const char *>(string_column->getChars().data());
 result.offset_data = reinterpret_cast<const char *>(string_column->getOffsets().data());
 }

For a String the “element type” is therefore a {chars*, offsets*, index} struct, not a char*:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:94
static llvm::StructType * buildStringRefType(llvm::IRBuilderBase &b)
{
 auto * data_ptr_type = b.getInt8Ty()->getPointerTo();
 auto * offset_ptr_type = b.getInt8Ty()->getPointerTo();
 auto * index_type = b.getInt64Ty();
 return llvm::StructType::get(data_ptr_type, offset_ptr_type, index_type);
}

The JIT itself

CHJIT is a small, self-contained LLVM JIT: it uses RuntimeDyld to link the object file into memory it manages itself.

The whole compilation is serialised by a mutex, because LLVM’s context is shared:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:480
CHJIT::CompiledModule CHJIT::compileModule(std::function<void (llvm::Module &)> compile_function)
{
 std::lock_guard lock(jit_lock);

The module is then run through a full O3 pipeline with SLP vectorisation enabled:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:620
void CHJIT::runOptimizationPassesOnModule(llvm::Module & module) const
{
 llvm::LoopAnalysisManager lam;
 llvm::FunctionAnalysisManager fam;
 llvm::CGSCCAnalysisManager cgam;
 llvm::ModuleAnalysisManager mam;

 llvm::PipelineTuningOptions pto;
 pto.SLPVectorization = true;

 /// Pass the actual TargetMachine so that all optimization passes
 /// (including target-specific ones) have proper target information.
 llvm::PassBuilder pb(machine.get(), pto);

 pb.registerModuleAnalyses(mam);
 pb.registerCGSCCAnalyses(cgam);
 pb.registerFunctionAnalyses(fam);
 pb.registerLoopAnalyses(lam);
 pb.crossRegisterProxies(lam, fam, cgam, mam);

 llvm::ModulePassManager mpm = pb.buildPerModuleDefaultPipeline(llvm::OptimizationLevel::O3);
 mpm.run(module, mam);
}

The target machine is the host’s, with the host CPU and features, so the compiled code uses whatever vector instructions the machine has:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:644
std::unique_ptr<llvm::TargetMachine> CHJIT::getTargetMachine()
{
 static std::once_flag llvm_target_initialized;
 std::call_once(llvm_target_initialized, []()
 {
 llvm::InitializeNativeTarget();
 llvm::InitializeNativeTargetAsmPrinter();
 llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
 });

 std::string error;
 auto cpu = llvm::sys::getHostCPUName();
 auto triple = llvm::Triple(llvm::sys::getProcessTriple());

Memory management

JIT code needs executable pages. ClickHouse does not use mmap directly; it allocates page-aligned blocks with posix_memalign and flips their protection with mprotect once the linking is done:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:287
 void allocateNextPageBlock(size_t size)
 {
 size_t pages_to_allocate_size = ((size / page_size) + 1) * 2;
 size_t allocate_size = page_size * pages_to_allocate_size;

 void * buf = nullptr;
 int res = posix_memalign(&buf, page_size, allocate_size);

The double allocation (+1 then *2 pages) gives both slack for the alignment request and room to grow without a new mprotect dance. Three separate arenas exist — read-write data, read-only data, and executable code — and the memory manager marks them appropriately at the end:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:353
 bool finalizeMemory(std::string *) override
 {
 ro_page_arena.protect(PROT_READ);
 ex_page_arena.protect(PROT_READ | PROT_EXEC);
 return true;
 }

On ARM the code has to invalidate the instruction cache after mprotect, and some ARM CPUs fault if the page is not readable while the cache is being flushed — hence the special case:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:211
# if defined(__arm__) || defined(__aarch64__)
 /// Certain ARM implementations treat icache clear instruction as a memory read,
 /// and CPU segfaults on trying to clear cache on !PROT_READ page.
 /// Therefore we need to temporarily add PROT_READ for the sake of flushing the instruction caches.
 if (invalidate_cache && !(protection_flags & PROT_READ))
 {
 int res = mprotect(block.base(), block.blockSize(), protection_flags | PROT_READ);

Symbol resolution

The compiled code may call back into ClickHouse (e.g. memcpy, or the 128-bit integer helpers for Int128 arithmetic). A JITSymbolResolver maps those names to addresses, and the trick of storing mangled names removes a platform-specific lookup path:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:397
 void registerSymbol(const std::string & symbol_name, void * symbol)
 {
 std::string mangled_name;
 llvm::raw_string_ostream mangled_name_stream(mangled_name);
 llvm::Mangler::getNameWithPrefix(mangled_name_stream, symbol_name, layout);
 mangled_name_stream.flush();
 symbol_name_to_symbol_address[mangled_name] = symbol;
 }

The 128-bit helpers come in signed/unsigned pairs, and the comment explains why both halves must be registered:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:76
/// IMPORTANT: every libcall is registered as a signed/unsigned pair. LLVM picks the
/// signed or unsigned variant based on whether the IR uses `fptosi`/`sitofp`/`sdiv`/`srem`
/// or `fptoui`/`uitofp`/`udiv`/`urem`. Forgetting the unsigned half compiles fine but
/// fails the JIT link at run time with `Could not find symbol __fixunsdfti` (or similar).
/// Keep the lists below symmetric.

Caching the compiled code

Compiling the same expression twice is pure waste, so compiled modules are cached by the CompileDAG hash:

 cpp
 
 
// src/Interpreters/JIT/CompiledExpressionCache.h:37
class CompiledExpressionCache : public CacheBase<UInt128, CompiledExpressionCacheEntry, UInt128Hash, CompiledFunctionWeightFunction>
{
public:
 using Base = CacheBase<UInt128, CompiledExpressionCacheEntry, UInt128Hash, CompiledFunctionWeightFunction>;
 using Base::Base;
};

The cache is weight-aware: an entry’s weight is the number of bytes of machine code it occupies, so evicting a big module frees proportionally more memory.

The hash that keys the cache is a structural hash over the DAG:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.cpp:126
UInt128 CompileDAG::hash() const
{
 SipHash hash;
 for (const auto & node : nodes)
 {
 hash.update(node.type);

 const auto & result_type_name = node.result_type->getName();
 hash.update(result_type_name.size());
 hash.update(result_type_name);

 switch (node.type)
 {
 case CompileType::CONSTANT:
 {
 node.column->getDataColumn().updateHashWithValue(0, hash);
 break;
 }

so two syntactically different but semantically identical expressions hash the same, and the same expression with a different constant does not.

Beyond scalar expressions

The same machinery compiles two more things.

Aggregate functions. A whole aggregation is compiled as five entry points:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:544
CompiledAggregateFunctions compileAggregateFunctions(CHJIT & jit, const std::vector<AggregateFunctionWithOffset> & functions, std::string functions_dump_name)
{
 Stopwatch watch;

 std::string create_aggregate_states_functions_name = functions_dump_name + "_create";
 std::string add_aggregate_states_functions_name = functions_dump_name + "_add";
 std::string add_aggregate_states_functions_name_single_place = functions_dump_name + "_add_single_place";
 std::string merge_aggregate_states_functions_name = functions_dump_name + "_merge";
 std::string insert_aggregate_states_functions_name = functions_dump_name + "_insert";

_create builds the aggregate states, _add folds a row into them, _merge merges two states, _insert writes the final values into the result columns. The _add function is generated twice: once for the “many places” layout (a hash table of aggregate states) and once for the “single place” layout (one state per group):

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:556
 compileCreateAggregateStatesFunctions(module, functions, create_aggregate_states_functions_name);
 compileAddIntoAggregateStatesFunctions(module, functions, add_aggregate_states_functions_name, AddIntoAggregateStatesPlacesArgumentType::MultiplePlaces);
 compileAddIntoAggregateStatesFunctions(module, functions, add_aggregate_states_functions_name_single_place, AddIntoAggregateStatesPlacesArgumentType::SinglePlace);
 compileMergeAggregatesStates(module, functions, merge_aggregate_states_functions_name);
 compileInsertAggregatesIntoResultColumns(module, functions, insert_aggregate_states_functions_name);

Sort comparators. A multi-column sort key is compiled into a single comparator function that returns int8. The IR is a chain of basic blocks, one per sort column, joined by a phi node — a hand-written decision tree:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:742
 /** 1. If it is last condition block move to join block.
 * 2. If column elements are not equal move to join block.
 * 3. If column elements are equal move to next column condition.
 */
 if (i == columns_size - 1)
 b.CreateBr(comparator_join);
 else
 b.CreateCondBr(b.CreateICmpEQ(result, lhs_equals_rhs_result), comparator_steps_and_results[i + 1].first, comparator_join);

and the function’s return attribute is not a detail but a correctness requirement on Apple’s arm64 ABI:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:608
 /// The result is read back through an `int8_t (*)(...)` function pointer, so the i8 return
 /// must obey the C `int8_t` calling convention. This matters only on Darwin/Apple arm64:
 /// Apple's arm64 ABI makes the *callee* sign-extend sub-word return values and the caller
 /// relies on it (it emits no `sxtb` after the call). Without `signext` the JIT callee leaves
 /// the upper bits unset, so e.g. a -1 result is read as 255 and NULL ordering breaks.
 /// Linux arm64 (AAPCS64) is unaffected because there the *caller* sign-extends the return
 /// itself (`sxtb w0, w0`); x86-64 is likewise a no-op.
 comparator_func->addRetAttr(llvm::Attribute::SExt);

Trade-offs

What the JIT buys: the per-block dispatch, the per-node type switches and the column representation overhead disappear for the compiled fragment; the loop is a plain memory-bound loop that the CPU can auto-vectorise. For a filter like WHERE user_id = 42 AND ts > now() - INTERVAL 1 HOUR, this is the difference between a few hundred nanoseconds per block and a few microseconds.

What it costs:

Memory: every compiled module occupies executable pages, and the cache is weight-limited by machine-code size. A server with thousands of distinct hot expressions holds a lot of RX pages.
Compile latency: the first four executions of an expression pay for the compile; the O3 pipeline is not cheap (hundreds of microseconds to milliseconds). The min_count_to_compile_expression threshold exists exactly to amortise this.
A second implementation of every function’s semantics: compileImpl and executeImpl must agree. Bugs where the JIT path and the interpreter path differ are a real class of ClickHouse bugs — the Nullable handling above is where most of the subtlety lives.

Restricted type coverage: only “native” types (fixed-width integers, floats, Date, DateTime, and String/FixedString through their special reference structs) can be compiled. Decimal is explicitly excluded from the arithmetic family:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3840
 if constexpr (
 !std::is_same_v<ResultDataType, InvalidType> && !IsDataTypeDecimal<ResultDataType>
 && !IsDataTypeDecimal<LeftDataType> && !IsDataTypeDecimal<RightDataType> && OpSpec::compilable)

Locking: CHJIT::compileModule holds a global mutex, so the compile of one expression blocks the compile of another. The compiled code is fully parallel — it is only the compilation that is serialised.

And the escape hatch is one setting:

 sql
 
SELECT count() FROM events WHERE user_id = 42 SETTINGS compile_expressions = 0;

How ClickHouse Evaluates IN with Hash Sets

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

x IN (1, 2, 3) looks like a comparison. x IN (SELECT user_id FROM logins WHERE …) looks like a join. Under the hood both become the same thing: a hash set, built once, probed once per row of the left-hand side.

The interesting part is not “it is a hash table” — it is which hash table, because ClickHouse picks a different data structure for every shape of the right-hand side, and because the set can be large enough that it no longer fits in memory and has to be spilled to disk behind Bloom filters.

The set is a plan artifact, not a function argument

The IN operator is a normal SQL function whose second argument is not a value but a set object:

 cpp
 
 
// src/Functions/in.cpp:145
 ColumnPtr column_set_ptr = arguments[1].column;
 const ColumnSet * column_set = tryGetColumnSet(column_set_ptr);
 if (!column_set)
 throw Exception(ErrorCodes::ILLEGAL_COLUMN, "Second argument for function '{}' must be Set; found {}",
 getName(), column_set_ptr->getName());

 auto future_set = column_set->getData();
 if (!future_set)
 {
 if (dry_run)
 return ColumnUInt8::create(input_rows_count, static_cast<UInt8>(0));

 throw Exception(ErrorCodes::LOGICAL_ERROR, "No Set is passed as the second argument for function '{}'", getName());
 }

The set arrives as a ColumnSet holding a FutureSet — a promise, because the set may not be built yet when the function’s type is resolved. The set is built by a separate step in the pipeline, and the function then asks for it:

 cpp
 
 
// src/Functions/in.cpp:160
 auto set = future_set->get();
 if (!set)
 {
 if (dry_run)
 return ColumnUInt8::create(input_rows_count, static_cast<UInt8>(0));

 throw Exception(ErrorCodes::LOGICAL_ERROR, "Not-ready Set is passed as the second argument for function '{}'", getName());
 }

There are three kinds of FutureSet:

 cpp
 
 
// src/Interpreters/PreparedSets.h:66
class FutureSet
{
public:
 virtual ~FutureSet() = default;

 /// Returns set if set is ready (created and filled) or nullptr if not.
 virtual SetPtr get() const = 0;
 /// Returns set->getElementsTypes(), even if set is not created yet.
 virtual DataTypes getTypes() const = 0;
 /// If possible, return set with stored elements useful for PK analysis.
 virtual SetPtr buildOrderedSetInplace(const ContextPtr & context) = 0;

FutureSetFromTuple — the right-hand side is a literal list, IN (1, 2, 3). It is built during analysis, because the values are already known.
FutureSetFromSubquery — the right-hand side is a SELECT. It is built during execution, by a CreatingSetStep that runs before the steps that use it.
FutureSetFromStorage — the right-hand side is an ENGINE = Set table. The set already exists; the query only reads it.

The literal case is the interesting one for the “prepare once, use many” idea, because the same literal list may appear many times in a query:

 cpp
 
 
// src/Interpreters/ActionsVisitor.cpp:557
 auto set_key = right_arg->getTreeHash(/*ignore_aliases=*/ true);
 if (auto set = prepared_sets.findTuple(set_key, set_element_keys))
 return set; /// Already prepared.

 return prepared_sets.addFromTuple(set_key, right_arg_func, std::move(block), context->getSettingsRef());

The key is the tree hash of the AST of the right-hand side, so x IN (1,2,3) and y IN (1,2,3) share one set. The prepared sets live in a per-query container:

 cpp
 
 
// src/Interpreters/PreparedSets.h:301
 using SetsFromTuple = std::unordered_map<Hash, std::vector<FutureSetFromTuplePtr>, Hashing>;
 using SetsFromStorage = std::unordered_map<Hash, FutureSetFromStoragePtr, Hashing>;
 using SetsFromSubqueries = std::unordered_map<Hash, FutureSetFromSubqueryPtr, Hashing>;

The hash is 128 bits of CityHash, and the container hashes it down to 64 bits:

 cpp
 
 
// src/Interpreters/PreparedSets.h:278
struct FutureSetHashing
{
 UInt64 operator()(const FutureSet::Hash & key) const { return key.low64 ^ key.high64; }
};

The shape of the thing

 text
 
 
 SELECT ... WHERE x IN (SELECT id FROM t)
 |
 v
 CreatingSetStep ---> FutureSetFromSubquery
 | (built during execution, not analysis)
 v
 ActionsDAG: FunctionIn(x, ColumnSet) ---> Set::execute(key_columns)
 |
 chooseMethod(key_columns) -+--> key8 / key16 / key32 / key64
 | (one integer key: the key IS the slot)
 +--> keys32 / keys64 / keys128 / keys256
 | (fixed-width tuple: packed into one word)
 +--> key_string / key_fixed_string
 | (one string: chars arena + saved hash)
 +--> nullable_keys128 / nullable_keys256
 | (value + null flag packed together)
 +--> hashed
 (anything else: 128-bit hash only)

Choosing the data structure

This is the part that makes ClickHouse’s IN fast. The set’s key type is not “some value” — it is a tuple of columns with concrete types, and each shape deserves a different hash table.

SetVariants::chooseMethod inspects the key columns and picks one of thirteen implementations:

 cpp
 
 
// src/Interpreters/SetVariants.h:417
 #define APPLY_FOR_SET_VARIANTS(M) \
 M(key8) \
 M(key16) \
 M(key32) \
 M(key64) \
 M(key_string) \
 M(key_fixed_string) \
 M(keys32) \
 M(keys64) \
 M(keys128) \
 M(keys256) \
 M(nullable_keys128) \
 M(nullable_keys256) \
 M(hashed)

The decision tree, in order:

One nullable numeric key. The value and the NULL flag are packed into one 16-byte key, so the hash table needs no separate null handling:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:149
 if (has_nullable_key)
 {
 /// At least one key is nullable. Therefore we choose a method
 /// that takes into account this fact.
 if ((keys_size == 1) && (nested_key_columns[0]->isNumeric()))
 {
 /// We have exactly one key and it is nullable. We shall add it a tag
 /// which specifies whether its value is null or not.
 size_t size_of_field = nested_key_columns[0]->sizeOfValueIfFixed();
 if ((size_of_field == 1) || (size_of_field == 2) || (size_of_field == 4) || (size_of_field == 8))
 return Type::nullable_keys128;

One numeric key. A single UInt32 column gets a HashSet<UInt32> — the key is the hash-table key, no packing, no hashing beyond the CRC:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:180
 /// If there is one numeric key that fits into 64 bits
 if (keys_size == 1 && nested_key_columns[0]->isNumeric() && !nested_key_columns[0]->lowCardinality())
 {
 size_t size_of_field = nested_key_columns[0]->sizeOfValueIfFixed();
 if (size_of_field == 1)
 return Type::key8;
 if (size_of_field == 2)
 return Type::key16;
 if (size_of_field == 4)
 return Type::key32;
 if (size_of_field == 8)
 return Type::key64;

Several fixed-width keys. They are packed into 32/64/128/256 bits and the whole packed word is the key:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:199
 /// If the keys fit in N bits, we will use a hash table for N-bit-packed keys
 if (all_fixed && keys_bytes <= 4)
 return Type::keys32;
 if (all_fixed && keys_bytes <= 8)
 return Type::keys64;
 if (all_fixed && keys_bytes <= 16)
 return Type::keys128;
 if (all_fixed && keys_bytes <= 32)
 return Type::keys256;

One string key. A dedicated method with the string’s own arena:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:209
 /// If there is single string key, use hash table of it's values.
 if (keys_size == 1
 && (typeid_cast<const ColumnString *>(nested_key_columns[0])
 || (isColumnConst(*nested_key_columns[0]) && typeid_cast<const ColumnString *>(&assert_cast<const ColumnConst *>(nested_key_columns[0])->getDataColumn()))))
 return Type::key_string;

Anything else. The fallback hashes the whole tuple to 128 bits and stores only the hash — the original values are not kept:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:218
 /// Otherwise, will use set of cryptographic hashes of unambiguously serialized values.
 return Type::hashed;

The comment in the header explains the memory trade-off of that last choice:

 cpp
 
 
// src/Interpreters/SetVariants.h:357
 /** Unlike Aggregator, `concat` method is not used here.
 * This is done because `hashed` method, although slower, but in this case, uses less RAM.
 * since when you use it, the key values themselves are not stored.
 */

Presence-only vs counting

There are two flavours of the same set family. A plain IN only needs to know whether a key is present, so it uses a HashSet whose cell maps to VoidMapped. INTERSECT ALL / EXCEPT ALL need how many times a key occurs, so they use a HashMap with a UInt64 counter. The mapping is derived from the table rather than passed in, so a method stays “a set” when given a HashSet and becomes a multiset when given a HashMap:

 cpp
 
 
// src/Interpreters/SetVariants.h:30
/// The method structs below are shared between presence-only sets (a `HashSet`, whose cell reports
/// `VoidMapped`) and counting multisets (a `HashMap`, with a real mapped type, used by INTERSECT
/// ALL / EXCEPT ALL). The mapped type is derived from the table rather than passed explicitly, so a
/// method stays "a set" when given a `HashSet` and becomes a counting multiset when given a `HashMap`.
template <typename TData>
using SetMethodMapped = std::conditional_t<std::is_same_v<typename TData::mapped_type, VoidMapped>, void, typename TData::mapped_type>;

The choice of table type per key shape is visible in the variant declarations:

 cpp
 
 
// src/Interpreters/SetVariants.h:344
 std::unique_ptr<SetMethodOneNumber<UInt32, HashSet<UInt32, HashCRC32<UInt32>>>> key32;
 std::unique_ptr<SetMethodOneNumber<UInt64, HashSet<UInt64, HashCRC32<UInt64>>>> key64;
 std::unique_ptr<SetMethodString<HashSetWithSavedHash<std::string_view>>> key_string;
 std::unique_ptr<SetMethodFixedString<HashSetWithSavedHash<std::string_view>>> key_fixed_string;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt32, HashCRC32<UInt32>>>> keys32;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt64, HashCRC32<UInt64>>>> keys64;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt128, UInt128HashCRC32>>> keys128;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt256, UInt256HashCRC32>>> keys256;
 std::unique_ptr<SetMethodHashed<HashSet<UInt128, UInt128TrivialHash>>> hashed;

HashSetWithSavedHash is worth a note: for strings it stores the hash next to the string, so a probe that hits the same bucket does not have to re-hash the candidate key.

Building the set

Insertion goes through emplaceKey on the method’s state, one row at a time, with a twist: the same loop can build a deduplication filter, which records which rows were new. That filter is how the set keeps the original values around for primary-key analysis:

 cpp
 
 
// src/Interpreters/Set.cpp:370
 typename Method::State state(key_columns, key_sizes, nullptr);

 /// For all rows
 for (size_t i = 0; i < rows; ++i)
 {
 if constexpr (has_null_map)
 {
 if ((*null_map)[i])
 {
 if constexpr (build_filter)
 {
 (*out_filter)[i] = false;
 }
 continue;
 }
 }

 [[maybe_unused]] auto emplace_result = state.emplaceKey(method.data, i, variants.string_pool);

 if constexpr (build_filter)
 (*out_filter)[i] = emplace_result.isInserted();
 }

emplace_result.isInserted() is the deduplication: a row whose key was already in the set is not new, so it is filtered out of the “explicit set elements” — the copy of the values kept for index analysis.

Keeping the values is optional and bounded. When the set grows past max_elements_to_fill, the copy is dropped and the set keeps only the hashes:

 cpp
 
 
// src/Interpreters/Set.cpp:495
 bool inserted = insertFromColumnsImpl(columns, holder);
 if (inserted && fill_set_elements)
 {
 if (max_elements_to_fill && max_elements_to_fill < data->getTotalRowCount())
 {
 /// Drop filled elementes
 fill_set_elements = false;
 set_elements.clear();
 }
 else
 appendSetElements(holder);
 }

This is a graceful degradation: the set still answers IN correctly, it just can no longer be used to prune granules by primary key. The threshold is the setting use_index_for_in_with_subqueries_max_values.

Probing the set

The probe is the hot path: it runs once per row of the left-hand side.

 cpp
 
 
// src/Interpreters/Set.cpp:1002
 Arena pool;
 typename Method::State state(key_columns, key_sizes, nullptr);

 /// Clustered key columns (e.g. a primary key prefix) arrive in runs of equal consecutive
 /// rows. The consecutive-keys optimization in ColumnsHashing handles them inside `findKey`:
 /// the last-element cache compares the key with the previous row's before probing the hash
 /// table, and `HashMethodHashed` additionally compares the raw key bytes before even
 /// calculating the hash.
 for (size_t i = 0; i < rows; ++i)
 {
 if (has_null_map && (*null_map)[i])
 {
 vec_res[i] = negative;
 }
 else
 {
 auto find_result = state.findKey(method.data, i, pool);
 vec_res[i] = negative ^ find_result.isFound();
 }
 }

Two things are worth noticing.

The “last element cache”. When the input is sorted (or clustered by a primary key), consecutive rows often have the same key. The hashing state remembers the last key and its answer, so a repeat costs one comparison instead of a hash computation plus a probe. This optimisation is on for numeric keys:

 cpp
 
 
// src/Interpreters/SetVariants.h:44
template <typename FieldType, typename TData, bool use_cache = true> /// UInt8/16/32/64 for any types with corresponding bit width.
struct SetMethodOneNumber
{
 using Data = TData;
 using Key = typename Data::key_type;

 Data data;

 using State = ColumnsHashing::HashMethodOneNumber<typename Data::value_type,
 SetMethodMapped<Data>, FieldType, set_method_use_cache<Data, use_cache>>;

and off for strings, with a measurement in the comment:

 cpp
 
 
// src/Interpreters/SetVariants.h:62
/// For the case where there is one string key.
/// Unlike aggregation, the consecutive-keys cache stays disabled for string keys: a set probe's
/// per-row work is just hash+lookup, which for strings is cheaper than the extra bytewise
/// comparison the cache check performs on every miss. Measured 2026-08 on 20M-row `IN` probes:
/// enabling the cache made unclustered inputs 40% slower for short `String` keys and 24% slower
/// for `FixedString(36)` keys, while clustered inputs (runs of equal consecutive keys) improved
/// only 8-16%. In aggregation the same check is relatively cheap because the per-row work there
/// (aggregate-state updates) is much heavier.
template <typename TData>
struct SetMethodString
{
 using Data = TData;
 using Key = typename Data::key_type;

 Data data;

 using State = ColumnsHashing::HashMethodString<typename Data::value_type, SetMethodMapped<Data>, true, false>;

That is a rare and welcome thing to find in a codebase: an optimisation that was measured, found harmful for one type family, and turned off — with the numbers kept in the source.

The NULL rule. A row whose key contains a NULL is never in the set, so the answer is negative (0 for IN, 1 for NOT IN) without a lookup. The rationale is in the header:

 cpp
 
 
// src/Interpreters/Set.h:202
 /** How IN works with Nullable types.
 *
 * For simplicity reasons, all NULL values and any tuples with at least one NULL element are ignored in the Set.
 * And for left hand side values, that are NULLs or contain any NULLs, we return 0 (means that element is not in Set).
 *
 * If we want more standard compliant behaviour, we must return NULL
 * if lhs is NULL and set is not empty or if lhs is not in set, but set contains at least one NULL.
 * It is more complicated with tuples.
 * For example,
 * (1, NULL, 2) IN ((1, NULL, 3)) must return 0,
 * but (1, NULL, 2) IN ((1, 1111, 2)) must return NULL.
 *
 * We have not implemented such sophisticated behaviour.
 */

So NULL IN (1, 2, 3) is 0, not NULL. That is a deliberate deviation from SQL standard, documented in the source.

Type coercion on the probe

The left-hand side of the IN may have a different type from the set’s key — the set’s type wins, and the probe column is cast. The cast is cached, so it is not rebuilt for every block:

 cpp
 
 
// src/Interpreters/Set.h:249
 /// A cache for cast functions (if any) to avoid rebuilding cast functions
 /// for every call to `execute`
 mutable std::unique_ptr<InternalCastFunctionCache> cast_cache;

and when the value does not fit the key type, it is not an error — it is simply “not a member”:

 cpp
 
 
// src/Interpreters/Set.cpp:833
 if (can_convert_leniently)
 {
 /// `transform_null_in` makes NULL an ordinary matchable value, so the accurate-or-null
 /// cast above is not taken. A value the key type cannot represent is still simply not a
 /// member - which is what the literal `IN` list concludes, because it compares in a
 /// common supertype - but converting it strictly raised `CANNOT_CONVERT_TYPE` on rows
 /// the read path happened to deliver, so `v IN set_table` failed where `v IN (1)`
 /// returned 0.

Size limits and overflow

The set is subject to max_rows_in_set and max_bytes_in_set, checked after every inserted chunk:

 cpp
 
 
// src/Interpreters/Set.cpp:556
 bool within_limits = limits.check(data->getTotalRowCount(), data->getTotalByteCount(), "IN-set", ErrorCodes::SET_SIZE_LIMIT_EXCEEDED);
 if (!within_limits)
 is_truncated = true;

The overflow mode is set_overflow_mode: throw (the default) aborts the query, break truncates the set and returns the partial result — which is wrong for IN but is the only sane option for a GLOBAL IN over a huge table, where a partial set is still useful.

Spilling to disk

A set built from a subquery can be enormous: x IN (SELECT id FROM big_table) over a billion rows. ClickHouse 26.10 added spilling: the set moves to a temporary file, and the in-memory part shrinks to a directory plus a set of Bloom filters.

The disk representation is not the original values. Each key is converted into a compact disk key: numbers and FixedStrings of at most 32 bytes are packed into integers, and everything else becomes a 128-bit hash:

 cpp
 
 
// src/Interpreters/Set.h:85
 /// the keys that come first in the order of their disk keys rather than the first ones to arrive. Disk
 /// keys keep the order of the keys that the set packs into integers in memory, numbers and `FixedString`
 /// values of at most 32 bytes in total, with `NULL` after the values, so these are the smallest keys.
 /// Other keys, such as `String`, are stored as 128-bit hashes, whose order is arbitrary.

The keys are then sorted, so that a lookup can binary-search the directory and find the block that may contain the key.

The Bloom filters

Each block of the set gets its own Bloom filter, and all the filters share one array of bits:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:15
/// Bloom filters of the blocks of a set on disk, which the set keeps in memory. A lookup checks the filter of
/// the only block that can hold its key, and reads the block only when the filter lets the key pass.
///
/// The filters share one array of bits. Each block takes a range of it in proportion to its keys, and each
/// key sets a few bits in the range of its block. A single filter for all keys would take as much memory, but
/// a key would set bits anywhere in it, leading to writes at random places all over the array. For a large
/// set, the array is far larger than the cache, so each such write would miss the cache and the TLB, and
/// these misses would slow down writing the set, because the set fills the filters on the single thread that
/// writes its blocks. The range of a block is about as small as the block, so it stays in the cache while the
/// set writes the block. Lookups check the ranges in the order of the blocks, as they read the blocks.

That is a lovely piece of engineering: the filters are per block so that a write touches only a cache-resident range, not the whole (possibly huge) filter array.

The filter is sized by the number of distinct keys, which is estimated with a HyperLogLog counter while the set is being filled:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:29
 /// Counts the distinct keys of a set as they arrive, so that the bloom filters can be sized before the set
 /// writes its blocks. With 2^20 buckets, the estimate is within about 0.1% and the counter takes only 768 KiB.
 class DistinctKeyCounter
 {
 public:
 void add(std::span<const Key> keys)
 {
 for (const Key & key : keys)
 counter.insert(hashKey(key));
 }

and the bits-per-key is chosen to fit the memory budget, trading false positives for memory:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:46
 /// Creates the filters for about `expected_keys` keys. They take 8 bits per key, which lets about 3% of
 /// the keys that the set does not have pass. When that does not fit in `max_bytes` bytes with the ranges
 /// of the blocks, they take fewer bits per key, so that they fit: with 2 bits per key, about 40% of these
 /// keys pass, and with 1 bit per key, about 63%. Every block but the last holds at least
 /// `min_keys_per_block` keys, which bounds the number of ranges. Returns nothing when fewer than 1 bit
 /// per key fits, because such filters let so many keys pass that they do not pay for filling them.

The number of hash positions per key is capped at 3, with the reasoning spelled out:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:82
 /// A key sets the bits at most at this many positions in the range of its block. Each position costs a
 /// memory access when a lookup checks a key that the set has, while more positions let only slightly
 /// fewer other keys pass: with 8 bits per key, 3 positions let about 3.1% of them pass, and 6 positions,
 /// the most effective number, about 2.2%.
 static constexpr size_t max_positions_per_key = 3;

The lookup

A lookup against the disk set is a five-step algorithm: collect the probes, sort them, find the blocks, consult the filters, read the blocks.

 cpp
 
 
// src/Interpreters/DiskSet.cpp:121
 /// Step 1: Collect the probes: the rows whose keys are in the range of the set, except a row that repeats
 /// the key of the previous row, which takes that row's answer in step 5. So a run of equal keys, as in a
 /// sorted column, takes one probe.
 struct Probe
 {
 Key key;
 size_t row;
 };

 VectorWithMemoryTracking<Probe> probes;
 probes.reserve(keys.size());
 const Key & first_key = directory.getFirstKey(0);
 for (size_t row = 0; row < keys.size(); ++row)
 {
 const auto & key = keys[row];
 if ((row == 0 || key != keys[row - 1]) && first_key <= key && key <= last_key)
 probes.push_back({key, row});
 }

Note the range check first_key <= key && key <= last_key: keys outside the set’s key range cannot be in the set, and are answered without any I/O.

Then the probes are sorted, so that each block is read once and in order:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:143
 /// Step 2: Sort the probes by key. Probes of equal keys then come together, so each distinct key is
 /// looked up once, and the probes of each block come together in block order, so they share a read.
 /// Results go back to the original rows, so evaluating `IN` does not reorder the input.
 ::sort(probes.begin(), probes.end(), [](const Probe & lhs, const Probe & rhs) { return lhs.key < rhs.key; });

and the filters decide which blocks are actually read:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:174
 for (size_t begin = 0; begin < probes.size();)
 {
 size_t end = begin + 1;
 while (end < probes.size() && probes[end].key == probes[begin].key)
 ++end;

 block = directory.findBlock(probes[begin].key, block);
 if (!filters || filters->mayContain(block, probes[begin].key))
 {
 if (lookups.empty() || block != lookups.back().block)
 needed_blocks.push_back(block); /// Each block once, in ascending order
 lookups.push_back({block, begin, end});
 }
 else
 {
 for (size_t i = begin; i < end; ++i)
 skipped_rows += count_rows_of_probe(probes[i]);
 }
 begin = end;
 }

The number of rows the filters answered is a profile event, so the effectiveness of the filters is directly observable:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:195
 /// A skipped key counts once for each of its rows, as `ExternalSetLookupKeys` does, so that the two
 /// events give the share of the rows that the filters answer.
 ProfileEvents::increment(ProfileEvents::ExternalSetBloomFilterSkippedKeys, skipped_rows);

When does it spill?

The threshold is a memory budget, and the default is “half of the available memory”:

 cpp
 
 
// src/Core/Settings.cpp:3905
 DECLARE(Double, max_bytes_ratio_before_external_set, 0.5, R"(
Fraction of available server or user memory used to calculate the threshold for spilling the set of `IN`
with a subquery to disk, at the start of execution. For example, `0.5` uses half of the available memory.

The set spills while it is built and while it is used, and the two cases are handled differently. While it is built, the keys are handed to a DiskSetBuilder that sorts them into runs; while it is used, the keys are written out while the lookups keep reading the in-memory table, and the disk set replaces it at the end.

The bloom filter has its own budget:

 cpp
 
 
// src/Core/Settings.cpp:3917
 DECLARE(UInt64, max_bytes_for_external_set_bloom_filter, 128_MiB, R"(
Memory budget, in bytes, for the Bloom filter that each set of `IN` written to disk keeps in memory. The
filter lets lookups of values that are not in the set skip reading the disk. It does not speed up lookups
of values that are in the set.

Note the asymmetry in that sentence: the filter helps for misses, not for hits. A key that is in the set still costs a block read. That is the fundamental trade of a Bloom filter, and it is why the filter is only worth it when the selectivity is low.

Trade-offs

What it buys: IN over a subquery is a hash-table probe per row, with a data structure chosen to fit the key type exactly. No row-at-a-time interpretation, no generic comparison, and for the common cases (one integer key, one string key) the key is the hash-table key.

What it costs:

Memory. The set is proportional to the number of distinct values on the right-hand side. x IN (SELECT id FROM orders) with 100 M distinct ids is a 100 M-entry hash table. This is why the spilling machinery exists, and why max_rows_in_set exists.
Spilling is much slower than RAM. A probe against a disk set is a sort, a binary search, a Bloom filter check and possibly a block read. ClickHouse sorts the probe keys to make the reads sequential, which is a lot of work per block. The alternative — failing the query with SET_SIZE_LIMIT_EXCEEDED — is often better for the user.
The hash fallback loses information. For a heterogeneous key, the set stores a 128-bit hash of the tuple, not the tuple. Two different tuples with the same hash would be “equal”; the probability is negligible, but it is not zero.
NULL semantics deviate from the standard, as documented above.
The hashed method is slower per probe than the packed methods, because the key must be serialised and hashed on every probe rather than once at build time. For a set of strings, the string method stores the string and its hash; the hashed method stores only the hash.

The last point is the general lesson of the whole design: ClickHouse’s IN is fast not because hashing is fast, but because the shape of the data is known at analysis time and the implementation specialises on it — thirteen variants, each with its own hash function, its own packing, its own arena, and its own micro-optimisations, some of which are enabled and some of which are deliberately disabled after measurement.

How ClickHouse Decides Which Parts to Merge: a Cost Model for Write Amplification

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A MergeTree table is a set of immutable parts. Every INSERT adds one. Background merges take contiguous runs of parts and rewrite them as one bigger part, which reduces the number of parts a SELECT has to open — and costs a full rewrite of the data involved.

That trade-off has no free lunch, and the two directions of it pull in opposite directions:

Merge eagerly and you keep the part count low, but you rewrite the same bytes over and over. A part that is merged into a bigger part, which is merged again, is written twice.
Merge lazily and write amplification is low, but the part count climbs, and every query has to open every part.

ClickHouse’s answer is a small, explicit cost model, and — unusually — the model is documented in the source, together with its own limitations. This is a tour of that model: the score function, the “base of the logarithm” knob, the sliding window that prevents merge backlogs from snowballing, and the background scheduler’s exponential backoff.

The design note

The most valuable thing about ClickHouse’s merge selector is that the authors wrote down what they were trying to do, and where they thought they were wrong. The header of SimpleMergeSelector.h is a design document:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:6
/**
We have a set of data parts that is dynamically changing - new data parts are added and there is background merging process.
Background merging process periodically selects continuous range of data parts to merge.

It tries to optimize the following metrics:
1. Write amplification: total amount of data written on disk (source data + merges) relative to the amount of source data.
It can be also considered as the total amount of work for merges.
2. The number of data parts in the set at the random moment of time (average + quantiles).

Also taking the following considerations:
1. Older data parts should be merged less frequently than newer data parts.
2. Larger data parts should be merged less frequently than smaller data parts.
3. If no new parts arrive, we should continue to merge existing data parts to eventually optimize the table.
4. Never allow too many parts, because it will slow down SELECT queries significantly.
5. Multiple background merges can run concurrently but not too many.

It is not possible to optimize both metrics, because they contradict to each other.
To lower the number of parts we can merge eagerly but write amplification will increase.
Then we need some balance between optimization of these two metrics.

But some optimizations may improve both metrics.

For example, we can look at the "merge tree" - the tree of data parts that were merged.
If the tree is perfectly balanced then its depth is proportional to the log(data size),
the total amount of work is proportional to data_size * log(data_size)
and the write amplification is proportional to log(data_size).
If it's not balanced (e.g. every new data part is always merged with existing data parts),
its depth is proportional to the data size, total amount of work is proportional to data_size^2.

Read that last paragraph carefully, because it is the key insight of the whole design. The merges form a tree: leaves are inserted parts, internal nodes are merge results. If the tree is balanced, its depth is log_base(total_size) and the total write amplification is proportional to that depth. If the tree is a degenerate chain — every new part merged immediately into the current biggest part — the depth is proportional to the data size, and the total write amplification is O(data_size). Every byte gets rewritten once per level.

So the selector’s job is to keep the merge tree balanced, and the knob that controls balance is the arity of the tree — how many parts are merged at once.

The base of the logarithm

The arity is called base, and the header explains how it is generalized to parts of unequal size:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:35
We can also control the "base of the logarithm" - you can look it as the number of data parts
that are merged at once (the tree "arity"). But as the data parts are of different size, we should generalize it:
calculate the ratio between total size of merged parts to the size of the largest part participated in merge.
For example, if we merge 4 parts of size 5, 3, 1, 1 - then "base" will be 2 (10 / 5).

Base of the logarithm (simply called `base` in `SimpleMergeSelector`) is the main knob to control the write amplification.
The more it is, the less is write amplification but we will have more data parts on average.

To fit all the considerations, we also adjust `base` depending on total parts count,
parts size and parts age, with linear interpolation (then `base` is not a constant but a function of multiple variables,
looking like a section of hyperplane).

So the “base” of a candidate merge is sum_size / max_size — the ratio of the total size of the range to the size of its largest part. Merging four parts of sizes 5, 3, 1, 1 gives a base of 10/5 = 2. Merging a huge part with a tiny one gives a base barely above 1, which is bad: that is the degenerate chain, and the selector is supposed to avoid it.

base is exposed as a MergeTree setting, default 5:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1090
 DECLARE(Float, merge_selector_base, 5.0, R"(Affects write amplification of
 assigned merges (expert level setting, don't change if you don't understand
 what it is doing). Works for Simple and StochasticSimple merge selectors
)", 0) \

The admissibility test: is this range allowed at all?

Before any candidate is scored, it has to pass an admissibility test. The test is allow(), and its central quantity is the lowered base — a base that is adjusted downward depending on how stale the parts are, how many parts the partition has, and how big the range is.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:241
 /// Map size to 0..1 using logarithmic scale
 /// Use log(1 + x) instead of log1p(x) because our sum_size is always integer.
 /// Also log1p seems to be slow and significantly affect performance of merges assignment.
 double size_normalized = mapPiecewiseLinearToUnit(log(1 + sum_size), min_size_to_lower_base_log, max_size_to_lower_base_log);
 /// Calculate boundaries for age
 double min_age_to_lower_base = interpolateLinear(static_cast<double>(settings.min_age_to_lower_base_at_min_size), static_cast<double>(settings.min_age_to_lower_base_at_max_size), size_normalized);
 double max_age_to_lower_base = interpolateLinear(static_cast<double>(settings.max_age_to_lower_base_at_min_size), static_cast<double>(settings.max_age_to_lower_base_at_max_size), size_normalized);
 /// Map age to 0..1
 double age_normalized = mapPiecewiseLinearToUnit(min_age, min_age_to_lower_base, max_age_to_lower_base);
 /// Map partition_size to 0..1
 double num_parts_normalized = mapPiecewiseLinearToUnit(partition_size, static_cast<double>(settings.min_parts_to_lower_base), static_cast<double>(settings.max_parts_to_lower_base));
 /// The ratio should be within [0, 1]
 double combined_ratio = std::min(1.0, age_normalized + num_parts_normalized);

 double lowered_base = interpolateLinear(settings.base, 2.0, combined_ratio);

The three inputs, and what each one does:

Input	Effect on `lowered_base`
`age_normalized` — how old the youngest part in the range is	Older parts are cheaper to merge (they are not going to be queried as freshly-inserted data), so the base is lowered and the merge happens sooner.
`num_parts_normalized` — how many parts the partition has	A partition with too many parts is a problem for queries, so the base is lowered and merges get more aggressive.
`size_normalized` — how big the range is	Big ranges need to be older before they are merged, because merging them is expensive.

The mapping is deliberately non-linear in size (logarithmic), and the comment explains why log(1+x) rather than log1p(x): log1p is slow enough to matter, and the selector runs on every selection round.

Then the actual test:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:271
 return (sum_size + static_cast<double>(size) * static_cast<double>(settings.size_fixed_cost_to_add)) / (max_size + static_cast<double>(settings.size_fixed_cost_to_add)) >= lowered_base;

Read it as: “is sum_size / max_size at least lowered_base?” — modulo the fixed cost of merging (size_fixed_cost_to_add, default 5 MB) added to both numerator and denominator, which models the fact that merging even two tiny parts costs a fixed amount of work.

Two escape hatches come before it:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:215
 if (settings.min_age_to_force_merge && min_age >= static_cast<double>(settings.min_age_to_force_merge))
 return true;

 if (settings.min_partition_age_to_force_merge && min_partition_age > 0)
 if (min_partition_age >= static_cast<double>(settings.min_partition_age_to_force_merge))
 return true;

If the parts are old enough (or the whole partition is old enough), the range is merged no matter what the base says. This is the safety valve: the cost model can be wrong, but a part that has been sitting there for a day must eventually be merged.

The score function

Now, given that several ranges are admissible, which one do you pick? This is the score() function, and it is where the “cost model” really lives:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:122
 static double score(double count, double sum_size, double sum_size_fixed_cost)
 {
 /** Consider we have two alternative ranges of data parts to merge.
 * Assume time to merge a range is proportional to sum size of its parts.
 *
 * Cost of query execution is proportional to total number of data parts in a moment of time.
 * Let define our target: to minimize average (in time) total number of data parts.
 *
 * Let calculate integral of total number of parts, if we are going to do merge of one or another range.
 * It must be lower, and thus we decide, what range is better to merge.
 *
 * The integral is lower iff the following formula is lower:
 *
 * sum_size / (count - 1)
 *
 * But we have some tunes to prefer longer ranges.
 */
 return (sum_size + sum_size_fixed_cost * count) / (count - 1.9);
 }

The derivation in the comment is a genuine (if informal) calculus argument. If you merge a range of count parts, the part count drops by count - 1 (you replace count parts with 1). The merge takes time proportional to sum_size. The “average number of parts over time” is therefore sum_size / (count - 1) — and lower is better, so the range with the smallest sum_size / (count - 1) is the one to merge.

The - 1.9 is a fudge factor. With - 1 the denominator vanishes for a two-part merge, giving an infinite score. The authors replaced - 1 with - 1.9, which is a softening of the singularity: merging two parts is penalized less than the pure model would suggest, and merging many parts is penalized more. The comment’s own summary is “we have some tunes to prefer longer ranges”.

size_fixed_cost_to_add (5 MB by default) enters the numerator as sum_size + fixed * count — every part in the range carries the fixed cost of being merged, so a range of ten tiny parts is much more expensive than its byte sum suggests.

The candidate enumeration walks every contiguous sub-range of the partition and scores each one:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:108
 double current_score = score(static_cast<double>(end - begin), static_cast<double>(sum_size), static_cast<double>(settings.size_fixed_cost_to_add));

 if (settings.enable_heuristic_to_align_parts
 && static_cast<double>(size_prev_at_left) > static_cast<double>(sum_size) * settings.heuristic_to_align_parts_min_ratio_of_sum_size_to_prev_part)
 {
 double difference = std::abs(log2(static_cast<double>(sum_size) / static_cast<double>(size_prev_at_left)));
 if (difference < settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two)
 current_score *= interpolateLinear(settings.heuristic_to_align_parts_max_score_adjustment, 1,
 difference / settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two);
 }

The “align parts” heuristic is a preference for ranges whose total size is roughly a power of two times the size of the part to their left. Why? Because a balanced merge tree wants each level to double the size. If the part to the left is 100 MB and the candidate range is 100 MB, the range “fits” the tree; if it is 30 MB or 300 MB, the resulting tree is lopsided and the next merge will have to rewrite more. The score is scaled down by up to 25% (heuristic_to_align_parts_max_score_adjustment = 0.75) when the sizes are close to a power of two apart.

The right-tail heuristic

There is one more adjustment, applied before scoring, and it is the most “heuristic” of them all:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:51
 const bool should_remove_small_parts_at_right = settings.enable_heuristic_to_remove_small_parts_at_right
 && (!settings.merge_selector_min_age_to_disable_right_tail_heuristic
 || min_age < settings.merge_selector_min_age_to_disable_right_tail_heuristic);
 if (should_remove_small_parts_at_right)
 {
 size_t size_delta = 0;
 size_t rows_delta = 0;
 while (end >= begin + 3 && static_cast<double>((end - 1)->size) < settings.heuristic_to_remove_small_parts_at_right_max_ratio * static_cast<double>(sum_size))
 {
 size_delta += (end - 1)->size;
 rows_delta += (end - 1)->rows;
 --end;
 }

 sum_size -= size_delta;
 sum_rows -= rows_delta;
 }

The idea: a range that ends with a couple of very small parts is a bad merge. Those small parts are probably new parts that just arrived, and folding them into a large merge means they will sit in the merge queue for a long time. Worse, it means the newly-arrived data is not queryable quickly. So the heuristic trims the right tail of the range — the small parts at the newest end — and scores what remains.

The default ratio is 1%: a part at the right end whose size is less than 1% of the range’s total is trimmed, as long as at least three parts remain. And there is an age-based escape hatch (merge_selector_min_age_to_disable_right_tail_heuristic): once the range is old enough, the tail is kept, because at that point staleness is worse than freshness.

The sliding window: anti-snowball protection

The single most valuable piece of engineering in the selector is this:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:307
 /// If the parts in the parts vector are sorted by block number,
 /// it may not be ideal to only select parts for merging from the first N ones.
 /// This is because if there are more than N parts in the partition,
 /// we will not be able to assign a merge for newly created parts.
 /// As a result, the total number of parts within the partition could
 /// grow uncontrollably, similar to a snowball effect.
 /// To address this we will try to assign a merge taking into consideration
 /// only last N parts.
 const size_t parts_threshold = settings.window_size;
 size_t begin = 0;
 if (parts_count >= parts_threshold)
 {
 if (settings.enable_stochastic_sliding)
 begin = calculateRangeWithStochasticSliding(parts_count, parts_threshold);
 else
 begin = parts_count - parts_threshold;
 }

window_size defaults to 1000. The comment describes the failure mode it prevents, and it is a real one:

If the parts in the parts vector are sorted by block number, it may not be ideal to only select parts for merging from the first N ones. This is because if there are more than N parts in the partition, we will not be able to assign a merge for newly created parts. As a result, the total number of parts within the partition could grow uncontrollably, similar to a snowball effect.

Think about it: if the selector always preferred the oldest parts, then in a partition with 10 000 parts it would always be merging parts 1..100, which are ancient and useless to merge, while parts 9000..10000 — the ones that arrived in the last minute — never get merged. The partition’s part count never goes down. The merge queue is always busy with work that does not help.

By only ever considering the last N parts, every merge consumes recent parts, so the recent parts get compacted and the partition’s part count actually falls. It is a one-line fix for a self-reinforcing failure mode, and it is worth more than most of the arithmetic around it.

The stochastic variant (StochasticSimpleMergeSelector) randomizes the window position, so that different replicas do not all merge the same parts:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:275
size_t calculateRangeWithStochasticSliding(size_t parts_count, size_t parts_threshold)
{
 auto mean = static_cast<double>(parts_count);
 std::normal_distribution<double> distribution{mean, mean / 4};

 size_t right_boundary = static_cast<size_t>(distribution(thread_local_rng));
 if (right_boundary > parts_count)
 right_boundary = 2 * parts_count - right_boundary;

 right_boundary = std::max(right_boundary, parts_threshold);

 return right_boundary - parts_threshold;
}

A normal distribution with mean parts_count and standard deviation parts_count / 4, folded back into range by mirroring at parts_count. The std::max(right_boundary, parts_threshold) at the end guarantees the window never shrinks below window_size — the anti-snowball guarantee is never traded away for randomization.

The blurry base

The StochasticSimpleMergeSelector adds one more trick: instead of a sharp threshold on the base, it uses a probability distribution around the lowered base.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:256
 if (settings.use_blurry_base)
 {
 double partition_fill_factor = std::max(0., 1 - partition_size / static_cast<double>(settings.parts_to_throw_insert));
 /// Scale factor controls when (relativelty to the number of parts in partition)
 /// do we activate our special algorithm.
 /// With standard parameters the logic kicks in starting from 80% empty factor.
 /// The division by 2 is due to the fact that for normal distribution nearly 95.4%
 /// of all observations fall within two standard deviations.
 double scaling_factor = std::pow(partition_fill_factor, settings.blurry_base_scale_factor) / 2;
 /// The base lower than 1 doesn't make sense, so we try to avoid it.
 std::normal_distribution<double> distribution{lowered_base, (lowered_base - 1) * scaling_factor};
 /// The threshold should be strictly bigger than 1, because we don't allow to merge the part with itself.
 lowered_base = std::min(distribution(thread_local_rng), std::max(1.01, lowered_base));
 }

The idea: instead of “merge if base ≥ 5.0”, it says “merge if base ≥ a randomly-drawn threshold around 5.0”. A merge whose base is 4.9 might happen this round and not the next. The result is that the distribution of merges is smoother — no sharp cliff at a particular base value — and, critically, the merge decisions of different replicas do not line up.

The width of the distribution grows with the emptiness of the partition: partition_fill_factor is 0 when the partition is full (parts_to_throw_insert parts) and approaches 1 when it is empty. blurry_base_scale_factor (default 42) makes the effect very nonlinear — the blur only kicks in when the partition is nearly empty, which is exactly when you want merges to be infrequent and gentle.

The final std::min(distribution(...), std::max(1.01, lowered_base)) is a careful clamp: the drawn threshold can be lower than the base (making the merge more eager) but never lower than 1.01, because a base ≤ 1 would mean merging a part with itself.

Picking the winner

The scored candidates go into a max-heap, and the best admissible, non-overlapping one wins:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:142
 std::optional<PartsRange> buildMergeRange(const MergeConstraint & constraint)
 {
 constexpr static auto range_compare = [](const ScoredRange & lhs, const ScoredRange & rhs)
 {
 /// If the ranges have the same score, use the range that is most likely to have the lower merge level -
 /// the right one, according to the block number sorting.
 return lhs.score > rhs.score || (lhs.score == rhs.score && lhs.range_begin < rhs.range_begin);
 };

(the tie-break is “the right one”, i.e. the newer parts — the opposite of what a naive reading of “oldest first” would suggest, and consistent with the sliding window.)

The disjoint_set.addRangeIfPossible call in the pop loop enforces that two concurrent merges never claim overlapping ranges of parts. The disjoint set is implemented as a std::set of range boundaries per independent range, with an upper_bound lookup:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/DisjointPartsRangesSet.cpp:113
bool DisjointPartsRangesSet::addRangeIfPossible(RangesIterator range_it, PartsIterator range_begin, PartsIterator range_end)
{
 chassert(sanityCheck(uncovered_ranges, range_it, range_begin, range_end));

 auto & sorted_ranges = disjoint[range_it.base()];
 PartsRangeBoundaries range_boundaries{std::move(range_begin), std::move(range_end)};

 if (!isDisjoint(range_boundaries, sorted_ranges))
 return false;

 sorted_ranges.insert(std::move(range_boundaries));
 return true;
}

From parts to a merge: the whole pipeline

The selector is one step of a longer pipeline. The entry point is MergeTreeDataMergerMutator::selectPartsToMerge:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:328
std::expected<MergeSelectorChoices, SelectMergeFailure> MergeTreeDataMergerMutator::selectPartsToMerge(
 const PartsCollectorPtr & parts_collector,
 const MergePredicatePtr & merge_predicate,
 const MergeSelectorApplier & selector,
 const std::optional<PartitionIdsHint> & partitions_hint)
{
 const auto context = data.getContext();
 const auto settings = data.getSettings();
 const auto metadata_snapshot = data.getInMemoryMetadataPtr(context, false);
 const auto storage_policy = data.getStoragePolicy();
 const time_t current_time = std::time(nullptr);
 const bool can_use_ttl_merges = !ttl_merges_blocker.isCancelled();
 LogSeriesLimiter series_log(log, 1, /*interval_s_=*/60 * 30);

 auto collected = collectAllPossibleRanges(parts_collector, metadata_snapshot, storage_policy, current_time, partitions_hint, series_log);

Three phases:

collect — find the ranges of parts that could be merged (same partition, compatible schema, not already covered by a running merge);
split by merge predicate — cut those ranges wherever a merge is forbidden (a part that overlaps a running merge, a part with a different schema version, and so on);
choose — run the selector’s cost model over what is left.

The LogSeriesLimiter is a nice touch: the trace log of this function is rate-limited to one message per 30 minutes, because the function is called on every selection round and can produce a lot of output.

The background scheduler’s backoff

Selection is cheap; the merge is not. The background scheduler that runs the selection loop uses an exponential backoff when there is nothing to do:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:62
void BackgroundJobsAssignee::postpone()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 no_work_done_count += 1;
 double random_addition = std::uniform_real_distribution<double>(0, sleep_settings.task_sleep_seconds_when_no_work_random_part)(rng);

 size_t next_time_to_execute = static_cast<size_t>(
 1000 * (data.getBiasBackoffSeconds() + std::min(
 sleep_settings.task_sleep_seconds_when_no_work_max,
 sleep_settings.thread_sleep_seconds_if_nothing_to_do * std::pow(sleep_settings.task_sleep_seconds_when_no_work_multiplier, no_work_done_count))
 + random_addition));

 holder->scheduleAfter(next_time_to_execute, false);
}

with the settings:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.h:18
struct BackgroundTaskSchedulingSettings
{
 double thread_sleep_seconds_random_part = 1.0;
 double thread_sleep_seconds_if_nothing_to_do = 0.1;
 double task_sleep_seconds_when_no_work_max = 600;
 /// For exponential backoff.
 double task_sleep_seconds_when_no_work_multiplier = 1.1;

 double task_sleep_seconds_when_no_work_random_part = 1.0;

So: start at 0.1 s, multiply by 1.1 per idle round, cap at 600 s, plus up to 1 s of random jitter. An idle table’s selection loop slows down to one attempt every ten minutes after about 130 idle rounds, and the jitter prevents a fleet of servers from synchronizing their wake-ups.

And when work does appear, the backoff is unwound geometrically rather than reset:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:48
void BackgroundJobsAssignee::trigger()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 /// Do not reset backoff factor if some task has appeared,
 /// but decrease it exponentially on every new task.
 no_work_done_count /= 2;
 /// We have background jobs, schedule task as soon as possible
 holder->schedule();
}

no_work_done_count /= 2 rather than = 0. Halving means that after one new task the sleep time is back to roughly its previous value, and it takes several rounds of activity to fully unwind. A table that receives one part per minute therefore does not flip between “poll every 0.1 s” and “poll every 600 s” — it settles at a rate proportional to the arrival rate. That is a genuinely nice piece of control theory for what looks like three lines of code.

The whole picture

 text
 
 
 parts in a partition (by block number)
 ┌────┬────┬────┬────┬────┬──────────────┬────┬────┐
 │ p1 │ p2 │ p3 │ p4 │ p5 │ ... │pn-1│ pn │
 └────┴────┴────┴────┴────┴──────────────┴────┴────┘
 ↑ ↑
 └──────── sliding window (last N=1000) ─────┘
 (only these are eligible)

 for each contiguous sub-range of the window:
 base = sum_size / max_size
 if base < lowered_base(age, partition_size, range_size):
 skip
 score = (sum_size + 5MB * count) / (count - 1.9)
 trim small parts off the right tail
 adjust score for power-of-two alignment with the part on the left

 pick the lowest-scoring admissible range that does not overlap
 an already-assigned merge

What this buys you, and what it costs

It buys you a merge tree that stays balanced, which is what keeps write amplification at O(log n) instead of O(n). The base knob is the lever, and the whole selector is a machine for keeping base near its target.

It buys you a system that self-corrects. When a partition fills up, num_parts_normalized rises, the base is lowered, and merges become more eager — without anybody tuning anything. When a partition sits idle, the opposite happens.

It buys you an anti-snowball guarantee. The sliding window is the difference between a table whose part count stabilizes and one that grows without bound.

It costs you a cost model that is admittedly approximate. The header comment says so:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:57
It's still unclear if this algorithm is good or optimal at all. It's unclear if this algorithm is using the optimal coefficients.

and it describes how the coefficients were tuned (an insert/merge simulator, replayed against production system.part_log data), and how the research found slightly better coefficients that were not used because the test data’s representativeness was in question.

It costs you a - 1.9. The score function’s singularity is papered over with a constant rather than derived. It is a fudge factor, honestly labelled as one.

It costs you a lot of tuning surface. merge_selector_base, merge_selector_window_size, parts_to_throw_insert, max_parts_to_merge_at_once, min_parts_to_merge_at_once, merge_selector_small_parts_*, the align-parts heuristic, the right-tail heuristic, the blurry base, the stochastic sliding window. Every one of these is a lever that was pulled for a reason, and the reason is usually written in a comment somewhere in SimpleMergeSelector.h.

Seeing it in action

The merge list shows what was selected and why:

 sql
 
 
SELECT
 database,
 table,
 elapsed,
 progress,
 num_parts,
 formatReadableSize(total_size_bytes_compressed) AS size,
 formatReadableSize(memory_usage) AS mem,
 result_part_name
FROM system.merges
ORDER BY elapsed DESC;

And the parts-per-partition histogram is the metric the selector is ultimately optimizing:

 sql
 
 
SELECT
 database,
 table,
 partition,
 count() AS parts,
 sum(rows) AS rows,
 formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active
GROUP BY database, table, partition
ORDER BY parts DESC
LIMIT 20;

A partition with hundreds of parts means the selector is not keeping up — usually because the merge pool is saturated, or because the parts are too large to merge under parts_to_throw_insert pressure, or because merge_selector_base is set too high for the workload.

And if you want to see the cost model at work, the trace log has it:

 sql
 
 
SET log_level = 'trace';
SELECT count() FROM hits WHERE URL LIKE '%very-selective-prefix%';

The MergeTreeDataMergerMutator::selectPartsToMerge trace messages show which ranges were considered and which one was chosen.

How ClickHouse Caches the Answer to 'Which Granules Can Match This Predicate?'

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Index analysis is not free. Every time a query runs, ClickHouse has to work out which granules of which parts could possibly contain a matching row. That work involves evaluating the predicate against the primary key index, against the minmax indexes, against the skip indexes, and — for the harder cases — running a recursive exclusion search over the marks.

Now consider the shape of a real workload: a monitoring dashboard that runs the same query every 30 seconds, a service that filters on tenant_id = 42 over and over, a batch job that re-reads yesterday’s data every hour. The predicate is the same every time. The data is not — but the data is also mostly not matching.

This is a textbook case for memoization, and ClickHouse implements it as the QueryConditionCache: a global LRU that remembers, for each (table UUID, part name, predicate hash) triple, a bitmap of the granules that the predicate emptied during the last scan.

What the cache actually stores

The class doc is unusually explicit about the semantics:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:26
/// An implementation of predicate caching a la https://doi.org/10.1145/3626246.3653395
///
/// Given the table, part name and a hash of a predicate as key, caches which marks definitely don't match the predicate and which marks may
/// match the predicate. This allows to skip the scan if the same predicate is evaluated on the same data again. Note that this doesn't work
/// the other way round: we can't tell if _all_ rows in the mark match the predicate.
///
/// Note: The cache may store more than the minimal number of matching marks. For example, assume a very selective predicate that matches
/// just a single row in a single mark. One would expect that the cache records just a single mark as potentially matching:
/// 000000010000000000000000000
/// But it is equally correct for the cache to store this.
/// 000001111111110000000000000
/// It is just less efficient for pruning (false positives).
class QueryConditionCache
{
public:
 /// False means none of the rows in the mark match the predicate. We can skip such marks.
 /// True means at least one row in the mark matches the predicate. We need to read such marks.
 using MatchingMarks = std::vector<bool>;

Two things to internalize.

It is one-directional. The cache records “this granule matched nothing”. It never records “this granule matched everything” — because that is not something a scan can establish. If a granule produced rows, the cache cannot tell whether all of its rows matched.

It is allowed to be imprecise, in one direction only. The bitmap’s true means “may match”. A granule that matched one row in a million is marked true — the whole granule has to be read. The cache never produces a false negative, which is the only thing that matters for correctness.

MatchingMarks is a std::vector<bool> — a bit-packed boolean array, one bit per granule. A part with 100 000 granules costs 12.5 KB per predicate.

The whole mechanism is controlled by use_query_condition_cache, which defaults to true.

The key: three components, hashed

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:45
private:
 /// A hash of the table id, part name and condition id.
 /// CityHash128 is enough to use for practical applications as the probability of collisions is very low.
 /// https://github.com/ClickHouse/ClickHouse/issues/9506
 using Key = UInt128;

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:237
QueryConditionCache::Key QueryConditionCache::makeKey(const UUID & table_id, const String & part_name, UInt64 condition_hash)
{
 SipHash hash;
 hash.update(table_id);
 hash.update(part_name);
 hash.update(condition_hash);
 return hash.get128();
}

The key is the part, so a cached verdict about one part never leaks into another. The part name is qualified by the table UUID, so two tables with the same part name (a very common thing in MergeTree — part names are derived from block ranges and partition values) do not collide.

There is a wrinkle for file-backed engines, where “part” is a file:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:93
 /// Compose the `part_name` component of a cache key for a file-backed table (e.g. `File`, `S3`,
 /// object storage). Uses the full path (not just the base name) so files that share a name in
 /// different directories do not collide, and folds in a content-version token so an in-place
 /// rewrite of the file yields a different key rather than a stale hit. The token is the ETag for
 /// remote objects, or a local identity (modification time + inode + size) for local files. For
 /// immutable files (e.g. data-lake data files) the path alone is a stable identity and the token
 /// may be left empty. The path and the token are separated by a NUL byte, which cannot occur in
 /// either, so the mapping is unambiguous.
 static String makeFilePartName(const String & path, std::string_view version_token);

“folds in a content-version token so an in-place rewrite of the file yields a different key rather than a stale hit” — for a file-backed table, the “part” can change under you, and a stale “this file matches nothing” verdict would be a correctness bug. The ETag (or mtime + inode + size) is folded in.

The safety property: everything starts as “may match”

This is the detail that makes the whole thing safe. When an entry is created, every granule is marked true — “may match” — so a fresh entry skips nothing:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:396
QueryConditionCache::Entry::Entry(size_t mark_count)
 : matching_marks(mark_count, true) /// by default, all marks potentially are potential matches, i.e. we can't skip them
{
}

and the header explains why that default is a correctness requirement, not a performance one:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:76
 /// (**) About error handling: There could be an exception after the i-th scan and cache entries could (theoretically) be left in a
 /// corrupt state. If we are not careful, future scans queries could then skip too many ranges. To prevent this, it is important to
 /// initialize all marks of each entry as non-matching. In case of an exception, future scans will then not skip them.

Consider the failure mode. A query scans part P with predicate Q. Partway through, the query is cancelled or throws. If the cache had been updated optimistically — “these granules are going to match nothing” — the cancellation would leave a verdict that skips granules the predicate would in fact have matched. The query would return wrong results, silently, forever, because the cache entry survives.

By initializing everything to “may match” and only ever clearing bits after the scan has actually proven the granule empty, a crash can only lose information (the cache knows less than it could), never gain it.

Writing: only clear bits, never set them

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:272
void QueryConditionCache::write(
 const UUID & table_id, const String & part_name, UInt64 condition_hash, const String & condition,
 const MarkRanges & mark_ranges, size_t marks_count, bool has_final_mark)
{
 if (table_id == UUIDHelpers::Nil)
 return; /// Issue #92863: Certain database engines provide no table UUIDs

 Key key = makeKey(table_id, part_name, condition_hash);

#if defined(DEBUG_OR_SANITIZER_BUILD)
 auto load_func = [&](){ return std::make_shared<Entry>(marks_count, table_id, part_name, condition_hash, condition); };
#else
 auto load_func = [&](){ return std::make_shared<Entry>(marks_count); };
#endif

 auto [entry, inserted] = cache.getOrSet(key, load_func);

getOrSet is atomic: two concurrent scans of the same part with the same predicate cannot create two entries; they share one.

Then the update, and note the direction of the operation:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:311
 {
 std::lock_guard lock(entry->mutex); /// (*)

 chassert(marks_count == entry->matching_marks.size());

 /// The input mark ranges are the areas which the scan can skip later on.
 for (const auto & mark_range : mark_ranges)
 std::fill(entry->matching_marks.begin() + mark_range.begin, entry->matching_marks.begin() + mark_range.end, false);

 if (has_final_mark)
 entry->matching_marks[marks_count - 1] = false;
 }

std::fill(..., false) — the entry is only ever narrowed. The mark ranges passed in are the ones the scan proved empty. Nothing is ever set back to true, because “this granule matched something” is not a fact the cache can rely on: a later scan of the same part with the same predicate might read a different set of marks (because another query’s cached verdict pruned some of them!), and those un-read marks must stay “may match”.

That last point is subtle and important: the cache is a cache of negative results only, and the negative results are monotone. Once a granule is known empty for a predicate, it stays known empty for as long as the entry lives. The entry is invalidated by the part’s identity changing (a new part has a new name), never by the predicate being re-evaluated.

There is a fast path that avoids taking the exclusive lock when there is nothing to do:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:289
 /// Try to avoid acquiring the RW lock below (*) by early-ing out. Matters for systems with lots of cores.
 {
 std::shared_lock shared_lock(entry->mutex); /// cheap

 bool need_not_update_marks = true;
 for (const auto & mark_range : mark_ranges)
 {
 /// If the bits are already in the desired state (false), we don't need to update them.
 need_not_update_marks = std::all_of(entry->matching_marks.begin() + mark_range.begin,
 entry->matching_marks.begin() + mark_range.end,
 [](auto b) { return b == false; });
 if (!need_not_update_marks)
 break;
 }

 /// Do we either have no final mark or final mark is already in the desired state?
 bool need_not_update_final_mark = !has_final_mark || entry->matching_marks[marks_count - 1] == false;

 if (need_not_update_marks && need_not_update_final_mark)
 return;
 }

Most writes are no-ops: the same query, run repeatedly, finds the same granules empty, and the bits are already clear. The shared lock is cheap and the exclusive lock is never taken.

Why the entry has its own mutex

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:71
 /// (*) You might wonder why Entry has its own mutex considering that CacheBase locks internally already. The reason is that
 /// ClickHouse scans ranges within the same part in parallel. The first scan creates and inserts a new Key + Entry into the cache,
 /// the 2nd ... Nth scans find the existing Key and update its Entry for the new ranges. This can only be done safely in a
 /// synchronized fashion.

The CacheBase lock protects the map — the mapping from key to entry. It does not protect the contents of the entry. And the contents are mutated concurrently: a query with 16 streams reads 16 disjoint mark ranges of the same part simultaneously, and each of them writes back the ranges it proved empty.

Without a per-entry lock, two threads could be doing std::fill(begin, mid, false) and std::fill(mid, end, false) simultaneously on the same std::vector<bool>. std::vector<bool> packs eight booleans into a byte, so the two fills could land in the same byte and lose one another’s writes — a lost update, which would mean a granule that was proven empty stays “may match”. Not a correctness bug (only a lost optimization), but a real one.

The SharedMutex lets the many readers of a cached verdict proceed in parallel while the rare writer excludes them.

Reading: turning a bitmap back into mark ranges

The consult side is in MergeTreeDataSelectExecutor::filterPartsByQueryConditionCache. After the cache lookup, the bitmap is converted back into MarkRanges — the surviving granules:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1944
 for (const auto & mark_range : part_with_ranges.ranges)
 {
 size_t begin = mark_range.begin;
 for (size_t mark_it = begin; mark_it < mark_range.end;)
 {
 if (!matching_marks[mark_it])
 {
 if (mark_it == begin)
 {
 /// mark_range.begin -> 0 0 0 1 x x x x. Need to skip starting zeros.
 ++stats.granules_dropped;
 ++begin;
 ++mark_it;
 }
 else
 {
 size_t end = mark_it;
 for (; end < mark_range.end && !matching_marks[end]; ++end)
 ;

 if (min_marks_for_seek && end != mark_range.end && end - mark_it <= min_marks_for_seek)
 {
 /// x x x 1 1 1 0 0 1 x x x. And gap is small enough to merge, skip gap.
 mark_it = end + 1;
 }
 else
 {
 /// Case1: x x x 1 1 1 0 0 1 x x x. Gap is too big to merge, do not merge
 /// Case2: x x x 1 1 1 0 0 0 0 -> mark_range.end. Reach the end of range, do not merge
 stats.granules_dropped += end - mark_it;
 ranges.emplace_back(begin, mark_it);
 begin = end;

 if (end == mark_range.end)
 break;

 mark_it = end + 1;
 }
 }
 }
 else
 ++mark_it;
 }

The comments in that loop are worth reading twice. 0 means “skip this granule”, 1 means “must read it”, x means “unknown, keep it”.

Leading zeros are skipped: 0 0 0 1 x x x x → the range starts at the first 1.
A gap of zeros between two ones is dropped — unless the gap is smaller than min_marks_for_seek, in which case reading through it is cheaper than seeking past it. That is the same seek heuristic used by the index analysis, and it is applied here for the same reason.

And if every granule of the part was eliminated, the part is dropped from the query entirely:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1994
 if (ranges.empty())
 it = parts_with_ranges.erase(it);
 else
 {
 part_with_ranges.ranges = std::move(ranges);
 ++it;
 }

A part that the cached verdict proves empty is not opened at all. No marks read, no decompressed, no filters evaluated.

The predicate hash is not just the predicate

The most delicate part of the design is the key. If two different predicates hashed to the same value, one query’s verdict would be applied to another query’s data — silently returning wrong results.

The base hash is the ActionsDAG hash of the filter expression, which covers the function names, the result types, the children and the constant values. But that is not enough. Consider:

 sql
 
SELECT * FROM t WHERE ts >= now() - INTERVAL 10 DAY;

The DAG hash changes every time the query runs (the constant changes), so this predicate never gets a cache hit. Fine — but there is a subtler problem in the other direction: a setting can change a function’s result without changing the DAG. formatDateTime with different flags produces different strings from the same DAG. least/greatest propagate or skip NULLs differently. h3ToGeo swaps its arguments.

So the cache key includes a salt computed from every setting that a function might capture at build time:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:80
UInt64 queryConditionCacheSettingsSalt(const Settings & settings)
{
 /// Registration rule: a setting belongs here when a function captures it at build time (in its constructor
 /// or `build`), it changes the value the function returns, and nothing about it reaches
 /// `ActionsDAG::Node::updateHash` (which sees only the function name, the result type name, the children and
 /// the constant values). A setting that changes the result *type* is already covered by the type name in the
 /// hash and does not need an entry.
 SipHash hash;

and the salt function goes on for a hundred lines, hashing the values of formatdatetime_f_prints_single_zero, formatdatetime_parsedatetime_m_is_month_name, function_locate_has_mysql_compatible_argument_order, least_greatest_legacy_null_behavior, h3togeo_lon_lat_result_order, short_circuit_function_evaluation, session_timezone, date_time_output_format, bool_true_representation, input_format_null_as_default, and dozens more.

The last one is the most interesting:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:219
 /// `session_timezone` is not read by any function when it is built: `DateLUT::instance()` looks it up in the
 /// query context every time a function needs the implicit time zone, which is what `toDateTime(s)`,
 /// `parseDateTime(s, format)` and their relatives do for a `String` argument without an explicit time zone
 /// (`KeyCondition` refuses to analyze exactly these for the same reason). The result type is the plain
 /// `DateTime` without a time zone in its name, so the same `s` parses to a different Unix timestamp in two
 /// sessions with an identical DAG hash.
 hash.update(settings[Setting::session_timezone].value);

Two sessions in different time zones produce the same DAG for toDateTime(s), but parse s differently. Without the salt, one session’s verdict would be served to the other. This is the kind of bug that is nearly impossible to find by testing and completely obvious once you have read the rule.

The time-condition trick

now() in a predicate is a special case, and it is handled with a derived condition rather than a salt.

The problem: WHERE ts >= now() - INTERVAL 10 DAY has a different DAG hash every second, so it never caches. But it is a very common predicate.

The solution: round the time constant onto a grid. On the write side, the condition is strengthened; on the read side, it is weakened:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/updateQueryConditionCache.cpp:107
 /// Non-deterministic conditions must not be cached under their own hash (issues #81506 and #84508).
 /// However, for conditions that are non-deterministic only because of folded current-time constants
 /// (e.g. `time >= now() - INTERVAL 10 DAY`), a deterministic condition can be derived by rounding
 /// the constants onto a time grid (issue #115504). Since this is the write side, the rounding must
 /// *strengthen* the condition: "the WHERE filter matched no rows of this granule" then implies
 /// "the derived condition matches no rows of this granule", so the entry stored under the derived
 /// hash is sound. The read side (filterPartsByQueryConditionCache) probes the weakened variant;
 /// the two coincide for grid-aligned constants and are one grid cell apart otherwise.

Read the logic carefully, because the asymmetry is the whole trick.

Write side (strengthen): the actual filter used now() - 10 days. The derived condition is a rounded version, e.g. now() - 10 days rounded up to the next grid point, which is a narrower condition. If the narrow condition matched no rows in this granule, then the wider (actual) condition certainly matched no rows either. So the entry (written under the derived hash) is sound.
Read side (weaken): the query now probes with the rounded-down version, which is a wider condition. “The wider condition matched no rows” implies “the narrower condition matched no rows” — so skipping is sound.

The two coincide for grid-aligned constants and are one grid cell apart otherwise. The grid factor is a setting (query_condition_cache_time_condition_grid_factor).

This is a genuinely elegant piece of reasoning: it takes a predicate that is not cacheable (its value changes every second) and makes it cacheable by proving that a slightly different, grid-aligned predicate implies the right thing in both directions.

The skip-index profile: another salt

There is one more key problem, and it is the subtlest of all.

Some cached verdicts come from skip index analysis — “the minmax index on this column says no row in this granule can match”. But skip index analysis depends on which indexes exist and are enabled. If a query runs with use_skip_indexes = 0, and another runs with them on, the verdicts differ.

So the skip-index verdicts are stored under a profiled key — the plain condition hash salted with the profile of the index set that produced the verdict:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1771
 /// Each condition is looked up under two keys (see the write sides):
 /// * the bare condition hash, holding row-level exclusions (FilterTransform,
 /// MergeTreeSelectProcessor, object storage) which are always sound; and
 /// * a hash salted with the effective skip-index profile that index analysis ran
 /// (getSkipIndexProfiledConditionHash), holding the skip-index-analysis exclusions
 /// written by ReadFromMergeTree.
 /// Skip-index exclusions can legitimately diverge from the row-level predicate (e.g. a text
 /// index with a preprocessor), so they must only be consulted by a query that ran the same set
 /// of indexes; salting the key with the running index set (and disjunction mode) guarantees a
 /// query that ran a different set (use_skip_indexes = 0, an index dropped/ignored, or a
 /// different disjunction mode) never reads them. The two verdicts are merged: a mark may be
 /// skipped iff either verdict says it does not match. This keeps the pure-QCC case
 /// (use_skip_indexes = 0 still reusing row-level entries) working while preventing the
 /// skip-index poisoning of issue #108519. TopK reads (which only get here when
 /// `use_query_condition_cache_for_top_k` is on, see the gate above) consult TopK-salted keys
 /// and also the `topk_reuse_predicate_only_hash` so plain `SELECT ... WHERE` entries can be
 /// reused; TopK-salted entries are not read otherwise.

“a mark may be skipped iff either verdict says it does not match” — the merge is an AND of the two “may match” bitmaps:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1881
 auto merge_opt_marks = [](QueryConditionCache::MatchingMarks & matching_marks, const std::optional<QueryConditionCache::MatchingMarks> & marks_opt)
 {
 if (!marks_opt)
 return;
 if (matching_marks.empty())
 matching_marks = *marks_opt;
 else if (matching_marks.size() == marks_opt->size())
 for (size_t i = 0; i < matching_marks.size(); ++i)
 matching_marks[i] = matching_marks[i] && (*marks_opt)[i];
 };

AND of the “may match” bits: a granule is skipped only if both verdicts agree it cannot match. Conservative in the right direction.

Where the “empty granule” fact comes from

The write side is the read pipeline itself. When the PREWHERE filter processes a granule and the filter removes every row of it, the read task records the granule as “unmatched”:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeReadTask.cpp:590
void MergeTreeReadTask::addPrewhereUnmatchedMarks(const MarkRanges & mark_ranges_)
{
 prewhere_unmatched_marks.insert(prewhere_unmatched_marks.end(), mark_ranges_.begin(), mark_ranges_.end());
}

and the write happens when the task is finished:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSelectProcessor.cpp:617
 if (reader_settings.use_query_condition_cache && task && prewhere_info
 && !task->readersChainCanSkipMarksBeforePrewhere(prewhere_filters_by_top_k_threshold)
 && !task->appliesMutationsBeforePrewhere()
 && !row_level_filter
 /// QueryConditionCache needs the concrete part's storage UUID; skip for borrowed parts.
 && task->getInfo().data_part_info->getDataPart())
 {

Every one of those four extra conditions is a soundness condition, and each has a comment explaining it. The interesting ones:

!task->readersChainCanSkipMarksBeforePrewhere(...) — if a skip index upstream of PREWHERE could have filtered marks before PREWHERE saw them, then the marks PREWHERE “emptied” are not the marks PREWHERE would have emptied, and attributing them to the PREWHERE predicate would be a lie. (Issue #104781.)
!task->appliesMutationsBeforePrewhere() — on-the-fly mutations filter rows ahead of PREWHERE, for the same reason.
!row_level_filter — a row-level security filter is prepended before PREWHERE, but the cache key covers only the query’s PREWHERE hash. A mark hidden by a row policy must not be attributed to the PREWHERE predicate, because a later query without that policy would then skip rows it should see.

That last one is a beautiful example of a cache whose key does not cover everything that influenced the result — the fix is to not write in that case, rather than to make the key bigger.

The whole flow

 text
 
 
 query with predicate Q
 │
 ▼
 hash(Q) + settings salt + [skip-index profile]
 │
 ├──► QueryConditionCache::read(table_uuid, part, hash)
 │ │
 │ └──► bitmap of "may match" granules
 │ │
 │ ▼
 │ surviving MarkRanges per part
 │ (parts with no surviving ranges are dropped entirely)
 │
 ▼
 index analysis (primary key, minmax, skip indexes)
 │
 ▼
 scan the surviving granules
 │
 ├── granules where the filter removed everything ──► cache.write(cleared bits)
 │
 └── rows ──► result

What this buys you, and what it costs

It buys you the elimination of repeated index analysis. The expensive part of “which granules can match” is not the bitmap lookup, it is the KeyCondition evaluation and the exclusion search. On a warm cache, that work is replaced by a hash lookup and a bit scan.

It buys you the elimination of reading granules that match nothing. This is the bigger win. The index analysis can only prove a granule is empty using the index; a predicate on a non-key column cannot be resolved from the index at all. But once a scan has observed that a granule contains no matching rows, that fact is cached, and the next run does not read it.

It costs you memory. One bit per granule per (table, part, predicate). The cache is a CacheBase LRU with a max_size_bytes, and the weight function accounts for the bit-packed vector plus a fixed overhead:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:229
size_t QueryConditionCache::EntryWeight::operator()(const Entry & entry) const
{
 size_t memory = sizeof(Key) + sizeof(Entry);
 /// Estimate the memory size of `std::vector<bool>` (it uses bit-packing internally)
 /// Round up to bytes.
 memory += (entry.matching_marks.capacity() + 7) / 8;
#if defined(DEBUG_OR_SANITIZER_BUILD)
 memory += entry.part_name.capacity() + entry.condition.capacity();
#endif
 return memory;
}

It costs you correctness vigilance. Everything in this design is about not serving a verdict that was computed under different circumstances: a different predicate, a different set of skip indexes, a different time zone, a different row-level policy, a TopK filter that elides granules. The salt functions, the profiles, the derived time conditions, and the !row_level_filter guard are all the same idea applied four times.

It gives up on one direction. The cache cannot record “this granule matches everything”, so a highly selective predicate that matches a few rows in every granule buys nothing. The cache is for predicates that are granule-level selective — which is most of them, since a granule is 10 MB of data.

Seeing it in action

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN ('QueryConditionCacheHits', 'QueryConditionCacheMisses');

A hit/miss ratio that is high and stable is the signature of a workload with repeated predicates over stable data. A ratio near zero means the predicates are always new — dashboards with per-request filters, for instance — and the cache is pure overhead.

The cache is also inspectable:

 sql
 
 
SELECT *
FROM system.query_condition_cache
LIMIT 10;

And to see it working, run the same query twice and compare the SelectedMarks and SelectedRanges counters in system.query_log:

 sql
 
 
SELECT
 query_id,
 ProfileEvents['SelectedMarks'] AS marks,
 ProfileEvents['SelectedRanges'] AS ranges,
 ProfileEvents['QueryConditionCacheHits'] AS qcc_hits
FROM system.query_log
WHERE query LIKE '%tenant_id = 42%'
ORDER BY event_time;

The second run should show dramatically fewer selected marks — and, if the predicate empties whole granules, the same query will not even open the part.

How ClickHouse Caches Query Results in Memory and on Disk

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

A query result cache is the oldest trick in the database book: run the same query twice, serve the second one from memory. What makes it hard is not the storage but the key. Two queries that produce different results must not share an entry, and two queries that produce identical results should. Textual equality is hopeless — the same query can be spelled with different whitespace, different letter case, a different current database, or a different value of a setting that changes the answer.

ClickHouse’s answer is to key the cache on a hash of the parsed query, plus the settings that could change the answer, plus the identity of the user. This article walks the whole path: the key construction, the TTL cache, the writer’s many reasons to refuse an entry, the reader’s access checks, and the on-disk backend that stores entries in a filesystem cache.

Why the cache is hard to get right

The header of the cache class states the contract plainly:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:48
/// Maps queries to query results. Useful to avoid repeated query calculation.
///
/// The cache does not aim to be transactionally consistent (which is difficult to get right). For example, the cache is not invalidated
/// when data is inserted/deleted into/from tables referenced by queries in the cache. In such situations, incorrect results may be
/// returned. In order to still obtain sufficiently up-to-date query results, a expiry time (TTL) must be specified for each cache entry
/// after which it becomes stale and is ignored. Stale entries are removed opportunistically from the cache, they are only evicted when a
/// new entry is inserted and the cache has insufficient capacity.

So the correctness story is “TTL, not invalidation”. A result is valid for as long as its TTL says, and the entry is not invalidated when the underlying table changes. This is a deliberate trade: tracking table versions per entry would make every entry depend on the write path.

The key: a hash of the AST, not of the text

The key’s hashed part is an IASTHash — a 128-bit hash of the parsed query:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:59
 struct Key
 {
 /// ----------------------------------------------------
 /// The actual key (data which gets hashed):

 /// The hash of the query AST.
 /// Unlike the query string, the AST is agnostic to lower/upper case (SELECT vs. select).
 IASTHash ast_hash;

The AST is not hashed as-is, though. Two normalisations happen first.

Query-cache settings are removed. A query that says SETTINGS use_query_cache = true should hit the entry written by the same query without that clause:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:500
ASTPtr removeQueryResultCacheSettings(ASTPtr ast)
{
 RemoveQueryResultCacheSettingsMatcher::Data visitor_data;
 RemoveQueryResultCacheSettingsVisitor(visitor_data).visit(ast);

 return ast;
}

and the predicate that decides which settings are pruned is deliberately generous about output-format settings, which do not change the rows:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:413
bool settingDoesNotAffectQueryResultCache(const String & setting_name)
{
 return setting_name == "log_comment"
 /// As of today, the output format settings only affect the final output.
 /// However, it should be taken with caution - we should not use these settings in deterministic SQL functions.
 || setting_name.starts_with("output_format_")
 /// This setting is used to tune the server response, but does not affect the query behavior.
 /// An example why it should not affect query caching:
 /// - if you run a query as usual, and then run the same query with asking the server
 /// for Content-Disposition: attachment to download the result.
 || setting_name == "http_response_headers";
}

Planner-generated table aliases are removed for subqueries, so that two identical subqueries under different plans share one entry. The pattern is matched exactly, because a user identifier that happens to look like __table1 must not be rewritten:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:513
bool isPlannerGeneratedTableAlias(std::string_view name)
{
 constexpr std::string_view prefix = "__table";
 if (!name.starts_with(prefix))
 return false;
 auto suffix = name.substr(prefix.size());
 if (suffix.empty())
 return false;
 for (char c : suffix)
 {
 if (c < '0' || c > '9')
 return false;
 }
 return true;
}

What else goes into the hash

After the AST, the hash takes the current database and the changed settings:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:578
IASTHash calculateASTHash(ASTPtr ast, const String & current_database, const Settings & settings, const bool is_subquery, const bool pre_cleaned = false)
{
 if (!pre_cleaned)
 {
 ast = ast->clone();
 if (is_subquery)
 ast = removeTableAliases(ast);
 }
 ast = removeQueryResultCacheSettings(ast);

 /// Hash the AST, we must consider aliases (issue #56258)
 SipHash hash;
 ast->updateTreeHash(hash, /*ignore_aliases=*/ false);

 /// Also hash the database specified via SQL `USE db`, otherwise identifiers in same query (AST) may mean different columns in different
 /// tables (issue #64136)
 hash.update(current_database);

and then the settings, sorted, because Settings::changes() returns them in random order and a hash must be order-independent:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:596
 /// Finally, hash the (changed) settings as they might affect the query result (e.g. think of settings `additional_table_filters` and `limit`).
 /// Note: allChanged() returns the settings in random order. Also, update()-s of the composite hash must be done in deterministic order.
 /// Therefore, collect and sort the settings first, then hash them.
 auto changed_settings = settings.changes();
 std::vector<std::pair<String, String>> changed_settings_sorted; /// (name, value)
 for (const auto & change : changed_settings)
 {
 const String & name = change.name;
 if (!isSettingIgnoredInQueryResultCache(name)) /// see removeQueryResultCacheSettings() and isSubquerySpecificSetting() for why this is a good idea
 changed_settings_sorted.push_back({name, Settings::valueToStringUtil(change.name, change.value)});
 }

Only changed settings are hashed — a setting that was never touched cannot change the result relative to the default. But there is a subtlety for subqueries, because the planner forcibly sets extremes, max_result_bytes and max_result_rows for them, which would make a subquery’s key differ from the same query run at top level:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:608
 /// The Planner forcibly sets `extremes`, `max_result_bytes`, `max_result_rows` for subqueries, which makes them
 /// appear in `settings.changes()`. Non-subquery executions of the same AST typically don't have these in their
 /// changed settings. To ensure cache hits between subquery writes and subquery reads, normalize by always
 /// including the default values of these settings for subqueries.
 if (is_subquery)
 {
 if (!changed_settings.tryGet("extremes"))
 changed_settings_sorted.push_back({"extremes", settings[Setting::extremes].toString()});

The equality of two keys

Note what is not in the equality check. Two keys are equal if the AST hashes match and they are both subqueries or both not:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:715
bool QueryResultCache::Key::operator==(const Key & other) const
{
 return ast_hash == other.ast_hash && is_subquery == other.is_subquery;
}

The user id and the roles are not part of the key. They are checked at read time instead, which is why the cache can hold one entry per query and still enforce access control — see below.

The hasher is trivial, since the AST hash is already a good hash:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:720
size_t QueryResultCache::KeyHasher::operator()(const Key & key) const
{
 SipHash hash;
 hash.update(key.ast_hash.low64);
 hash.update(key.is_subquery);
 return hash.get64();
}

What may be cached at all

Not every query may be cached. A query with a non-deterministic function (now(), rand()) would return a different answer the second time. The check walks the AST and asks the function factory:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:105
 if (const auto func = FunctionFactory::instance().tryGet(function->name, data.context))
 {
 if (!func->isDeterministic())
 data.has_non_deterministic_functions = true;
 return;
 }

Two escape hatches are handled with care. The eval table function hides its real query inside an opaque string, so it cannot be inspected and must be assumed non-deterministic:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:96
 /// The `eval` table function hides its real query inside an opaque string argument, so the
 /// generated query cannot be inspected here. Treat it as non-deterministic (and, in
 /// HasSystemTablesMatcher, as touching a system table) to keep such queries out of the cache,
 /// e.g. `eval('SELECT now()')` must not be cached as if it were deterministic.
 if (function->name == "eval")
 {
 data.has_non_deterministic_functions = true;
 return;
 }

and SQL UDFs, whose determinism ClickHouse does not know:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:111
 if (const auto udf_sql = UserDefinedSQLFunctionFactory::instance().tryGet(function->name))
 {
 /// Driver-created executable functions are also persisted in the SQL-object storage,
 /// but their determinism is described by the generated executable UDF configuration checked below.
 if (!udf_sql->as<ASTCreateFunctionWithDriverQuery>())
 {
 /// ClickHouse currently doesn't know if SQL-based UDFs are deterministic or not. We must assume they are non-deterministic.
 data.has_non_deterministic_functions = true;
 return;
 }
 }

The obfuscate table function is subtler: it is deterministic if a seed is given, and the seed can be overridden by a SETTINGS clause on any enclosing query, so the check has to descend into the AST while tracking the effective settings:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:309
 if (const auto * function = node->as<ASTFunction>())
 if (function->name == "obfuscate" && context->getSettingsRef()[Setting::obfuscate_seed].value.empty())
 return true;

What the user wants to happen on a violation is a setting, with three values (throw, save, ignore):

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:371
 const QueryResultCacheNondeterministicFunctionHandling nondeterministic_function_handling
 = settings[Setting::query_cache_nondeterministic_function_handling];
 const QueryResultCacheSystemTableHandling system_table_handling = settings[Setting::query_cache_system_table_handling];

 if (ast_contains_nondeterministic_functions && nondeterministic_function_handling == QueryResultCacheNondeterministicFunctionHandling::Throw)
 throw Exception(ErrorCodes::QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS,
 "The query result was not cached because the query contains a non-deterministic function."
 " Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless, or omit caching");

So:

 sql
 
 
-- A non-deterministic query is refused by default:
SELECT * FROM events WHERE ts > now() - INTERVAL 1 HOUR;
-- ERROR: The query result was not cached because the query contains a non-deterministic function.

-- ... unless you insist:
SELECT * FROM events WHERE ts > now() - INTERVAL 1 HOUR
SETTINGS query_cache_nondeterministic_function_handling = 'save';

The cache itself

The cache is a TTL cache with per-user quotas and its own metrics:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1180
QueryResultCache::QueryResultCache(size_t max_size_in_bytes, size_t max_entries, size_t max_entry_size_in_bytes_, size_t max_entry_size_in_rows_)
 : cache(std::make_unique<TTLCachePolicy<Key, Entry, KeyHasher, EntryWeight, IsStale>>(
 CurrentMetrics::QueryCacheBytes, CurrentMetrics::QueryCacheEntries, std::make_unique<PerUserTTLCachePolicyUserQuota>()))
{
 updateConfiguration(max_size_in_bytes, max_entries, max_entry_size_in_bytes_, max_entry_size_in_rows_);
}

Staleness is a pure function of the key:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:738
bool QueryResultCache::IsStale::operator()(const Key & key) const
{
 return (key.expires_at < std::chrono::system_clock::now());
};

The eviction policy is worth spelling out, because it is not an LRU. When an insert does not fit, the cache first purges the stale entries — the ones whose TTL has passed — and if that is not enough, the insert is simply refused and the new result is dropped. The cache never evicts a fresh entry to make room for another fresh entry:

 cpp
 
 
// src/Common/TTLCachePolicy.h:237
 if (!sufficient_space_in_cache() || !sufficient_space_in_cache_for_user())
 {
 /// Remove stale entries
 for (auto it = cache.begin(); it != cache.end();)
 {
 if (is_stale_function(it->first))
 {
 size_t sz = weight_function(*it->second);
 if (it->first.user_id.has_value())
 Base::user_quotas->decreaseActual(*it->first.user_id, sz);
 it = cache.erase(it);
 size_in_bytes -= sz;
 }
 else
 ++it;
 }
 }

and the weight of an entry is the number of bytes its chunks actually allocate:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:728
size_t QueryResultCache::EntryWeight::operator()(const Entry & entry) const
{
 size_t res = 0;
 for (const auto & chunk : entry.chunks)
 res += chunk.allocatedBytes();
 res += entry.totals.has_value() ? entry.totals->allocatedBytes() : 0;
 res += entry.extremes.has_value() ? entry.extremes->allocatedBytes() : 0;
 return res;
}

Per-user quotas are pushed into the cache on every write, because there is no better hook for “the settings profile of this user changed”:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1220
 /// Update the per-user cache quotas with the values stored in the query context. This happens per query which writes into the query
 /// cache. Obviously, this is overkill but I could find the good place to hook into which is called when the settings profiles in
 /// users.xml change.
 /// user_id == std::nullopt is the internal user for which no quota can be configured
 if (key.user_id.has_value())
 cache.setQuotaForUser(*key.user_id, max_query_result_cache_size_in_bytes_quota, max_query_result_cache_entries_quota);

The two backends

 text
 
 
 query AST
 | clone + strip cache settings + strip planner aliases
 | hash(AST) + hash(current_database) + hash(changed settings)
 v
 QueryResultCache::Key { ast_hash, user_id, roles, is_shared, ttl, header }
 |
 +--> in-memory TTLCachePolicy <-- Chunks (optionally compressed)
 | | weight = allocated bytes, per-user quota
 | v
 | SourceFromChunks --> the query pipeline
 |
 +--> on-disk: FileCacheKey(salted hash) --> filesystem cache
 | FixedHeader (magic, version, sizes, ttl, build fingerprint, checksum)
 | AccessMetadata (is_shared, user, roles, tag)
 | CompressedWriteBuffer + NativeWriter (header + chunks + totals + extremes)
 v
 filesystem cache (evicted by the same rules as any other data)

Writing: many reasons to refuse

The writer buffers the result chunks as they stream by, and decides at the very end whether to keep them. That is deliberate — a query that dies half-way must not leave a partial entry:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:222
/// Buffers multiple partial query result chunks (buffer()) and eventually stores them as cache entry (finalizeWrite()).
///
/// Implementation note: Queries may throw exceptions during runtime, e.g. out-of-memory errors. In this case, no query result must be
/// written into the query result cache. Unfortunately, neither the Writer nor the special transform added on top of the query pipeline
/// which holds the Writer know whether they are destroyed because the query ended successfully or because of an exception (otherwise, we
/// could simply implement a check in their destructors). To handle exceptions correctly nevertheless, we do the actual insert in
/// finalizeWrite() as opposed to the Writer destructor. This function is then called only for successful queries in finish_callback() which
/// runs before the transform and the Writer are destroyed, whereas for unsuccessful queries we do nothing (the Writer is destroyed w/o
/// inserting anything).

The list of refusal reasons in finalizeWrite is the interesting part:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:862
 if (result_is_incomplete)
 {
 LOG_TRACE(logger, "Skipped insert because the query result is incomplete (a consumer stopped reading it before the end), query: {}", doubleQuoteString(key.query_string));
 return;
 }

 if (auto query_runtime = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now() - query_start_time); query_runtime < min_query_runtime)
 {
 LOG_TRACE(logger, "Skipped insert because the query is not expensive enough, query runtime: {} msec (minimum query runtime: {} msec), query: {}",
 query_runtime.count(), min_query_runtime.count(), doubleQuoteString(key.query_string));
 return;
 }

The second one is the classic “don’t cache the cheap queries” rule: caching a 1 ms query costs more than recomputing it. The setting is query_cache_min_query_duration.

Partial chunks are squashed into blocks of max_block_size rows before being stored, so that the cached result looks like a freshly produced one:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:889
 if (squash_partial_results)
 {
 /// Squash partial result chunks to chunks of size 'max_block_size' each. This costs some performance but provides a more natural
 /// compression of neither too small nor big blocks. Also, it will look like 'max_block_size' is respected when the query result is
 /// served later on from the query result cache.

And the entries are compressed, at the cost of a decompression on read:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:969
 if (key.is_compressed)
 {
 /// Compress result chunks. Reduces the space consumption of the cache but means reading from it will be slower due to decompression.

 Chunks compressed_chunks;

 for (const auto & chunk : query_result->chunks)
 {
 const Columns & columns = chunk.getColumns();
 Columns compressed_columns;
 for (const auto & column : columns)
 {
 auto compressed_column = column->compress(/*force_compression=*/false);
 compressed_columns.push_back(compressed_column);
 }
 Chunk compressed_chunk(compressed_columns, chunk.getNumRows());
 compressed_chunks.push_back(std::move(compressed_chunk));
 }
 query_result->chunks = std::move(compressed_chunks);
 }

column->compress(force_compression=false) is the interesting bit: a column is compressed only if it is worth it — a tiny column of random UInt64s costs more in CPU than it saves in RAM.

Finally the size limits are checked against the compressed weight:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:992
 size_t new_entry_size_in_bytes = QueryResultCache::EntryWeight()(*query_result);
 size_t new_entry_size_in_rows = count_rows_in_chunks(*query_result);

 if ((new_entry_size_in_bytes > max_entry_size_in_bytes) || (new_entry_size_in_rows > max_entry_size_in_rows))
 {
 LOG_TRACE(logger, "Skipped insert because the query result is too big, query result size: {} (maximum size: {}), query result size in rows: {} (maximum size: {}), query: {}",
 formatReadableSizeWithBinarySuffix(new_entry_size_in_bytes, 0), formatReadableSizeWithBinarySuffix(max_entry_size_in_bytes, 0), new_entry_size_in_rows, max_entry_size_in_rows, doubleQuoteString(key.query_string));
 return;
 }

 cache.set(key, query_result);

Reading: access control

The reader looks the key up and then checks whether the caller may have it. The user id and the current roles are stored in the key but are not part of the equality check, so the same entry can be found by a different user — and then rejected:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1067
 const auto & entry_key = entry->key;
 const auto & entry_mapped = entry->mapped;

 const bool is_same_user_id = ((!entry_key.user_id.has_value() && !key.user_id.has_value()) || (entry_key.user_id.has_value() && key.user_id.has_value() && *entry_key.user_id == *key.user_id));
 const bool is_same_current_user_roles = (entry_key.current_user_roles == key.current_user_roles);
 if (!entry_key.is_shared && (!is_same_user_id || !is_same_current_user_roles))
 {
 LOG_TRACE(logger, "Inaccessible query result found for query {}", doubleQuoteString(key.query_string));
 return;
 }

The rationale for checking the roles and not just the user is in the key’s comment:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:78
 /// The id and current roles of the user who executed the query.
 /// These members are necessary to ensure that a (non-shared, see below) entry can only be written and read by the same user with
 /// the same roles. Example attack scenarios:
 /// - after DROP USER, it must not be possible to create a new user with with the dropped user name and access the dropped user's
 /// query result cache entries
 /// - different roles of the same user may be tied to different row-level policies. It must not be possible to switch role and
 /// access another role's cache entries
 std::optional<UUID> user_id;
 std::vector<UUID> current_user_roles;

A stale entry is also refused:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1078
 if (QueryResultCache::IsStale()(entry_key))
 {
 LOG_TRACE(logger, "Stale query result found for query {}", doubleQuoteString(key.query_string));
 return;
 }

and the chunks are cloned (uncompressed case) or decompressed (compressed case) before being handed to a SourceFromChunks processor, which becomes the new head of the query pipeline.

The on-disk backend

The in-memory cache is bounded by RAM. The on-disk backend stores entries in a filesystem cache — the same infrastructure used for remote table data — which means the entries can be much larger, survive restarts, and (with a distributed filesystem cache) be shared between servers.

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.h:14
/// Stores query result cache entries in a preconfigured filesystem cache (an entry of the `filesystem_caches` section of the server
/// configuration), so that cached query results get more space than in memory, survive server restarts, and (with a distributed
/// filesystem cache) can be shared between servers. The filesystem cache is selected per query with setting
/// `query_cache_on_disk_cache_name`.
///
/// The entries are ordinary filesystem cache entries, indistinguishable from other data in the same filesystem cache: they are not
/// held from deletion, they are evicted by the same rules, and there are no specific limits on their total size or number other than
/// the limits of the filesystem cache itself. The keys are hashed with a salt so that they cannot intersect with other keys in the
/// filesystem cache. All metadata needed to validate an entry (TTL, access control) is serialized into the entry itself; no in-memory
/// state is kept.

The key

The on-disk key is a salted hash of the AST hash, plus the access context for non-shared entries:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:136
FileCacheKey makeFileCacheKey(const QueryResultCache::Key & key, bool is_shared)
{
 /// Salt the hash so that keys of the on-disk query result cache cannot intersect with other keys in the same filesystem cache
 /// (those are typically derived from storage paths).
 SipHash hash;
 hash.update(std::string_view("QueryResultCacheOnDisk"));
 hash.update(key.ast_hash.low64);
 hash.update(key.ast_hash.high64);
 hash.update(key.is_subquery);
 hash.update(is_shared);
 if (!is_shared)
 {
 hash.update(key.user_id.has_value());
 if (key.user_id)
 hash.update(key.user_id->toUnderType());
 /// The roles are compared as an ordered sequence, same as in the in-memory query result cache.
 hash.update(key.current_user_roles.size());
 for (const auto & role : key.current_user_roles)
 hash.update(role.toUnderType());
 }

Note that unlike the in-memory cache, the on-disk key does include the user: the file cache is shared storage, so one user’s entry must not shadow another’s.

A marker in the middle of the key lets the cache find its own entries without reading anything from disk:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:103
/// Bits 64..95 of every key of the on-disk query result cache hold this marker ("QRCC"), so that the entries can be told apart from
/// the other keys of the filesystem cache by the key alone, without reading anything from disk (see `clear`). The marker sits in the
/// middle of the key on purpose: the filesystem cache derives the directory of a key from its topmost bits and the metadata bucket
/// from its lowest bits, both of which must keep varying. The remaining 96 bits of hash are plenty to keep the entries apart. A key
/// of other data carries the marker by chance with probability 2^-32; such a key is still recognized as foreign by its content.
constexpr UInt32 KEY_MARKER = 0x51524343;

The entry format

The on-disk entry has a fixed-size header, then uncompressed access metadata, then the compressed payload:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:63
/// On-disk entry layout, version 6:
///
/// Fixed header (FIXED_HEADER_SIZE bytes):
/// char[8] magic "QRCache1"
/// UInt32 format version
/// UInt32 TCP protocol revision the Native blocks are serialized with
/// UInt64 total entry size in bytes, including the fixed header
/// UInt64 created_at, seconds since epoch
/// UInt64 expires_at, seconds since epoch
/// UInt64 fingerprint of the server build which wrote the entry (see `getServerBuildFingerprint`)
/// UInt128 SipHash-128 of the preceding fixed header bytes and of everything after the fixed header. The freshness
/// fields are authenticated as well: a corruption of `expires_at` or `total_size` alone would otherwise
/// silently change how long the entry is served, without ever failing the integrity check.
///
/// Access metadata (uncompressed):
/// UInt8 is_shared
/// UInt8 has_user_id, [UInt128 user_id]
/// varUInt number of user roles, [UInt128 role]...
/// String tag (setting `query_cache_tag`), so that `SYSTEM DROP QUERY CACHE TAG` can find the entry
///
/// Result payload (compressed with the codec from setting `query_cache_on_disk_codec`; the compression frames are
/// self-describing, so reading does not depend on the setting):
/// varUInt number of result chunks
/// Native header block (zero rows)
/// Chunk one per result chunk
/// UInt8 has_totals, [Chunk]
/// UInt8 has_extremes, [Chunk]

Two details are worth dwelling on.

The build fingerprint. An entry is only valid for the server build that wrote it, because a new build may compute a different answer for the same query:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:110
/// The entries survive restarts and can be shared between servers, but a result is only valid for the server build which computed
/// it: a newer or older build may compute a different result for the same query (fixes of functions or of the planner, changed
/// defaults), even when neither the TCP protocol revision nor the entry format changes. So every entry records the build which wrote
/// it, and an entry of another build is treated like an entry of an incompatible format: it is never served, and it is overwritten
/// on the next write.

The checksum covers the freshness fields. Otherwise a corrupted expires_at would silently change how long the entry lives.

The header is patched in place after the payload is serialised, and the checksum is computed over the header and the body:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:514
 unalignedStoreLittleEndian<UInt64>(data.data() + TOTAL_SIZE_OFFSET_IN_FIXED_HEADER, data.size());

 SipHash hash;
 hash.update(data.data(), CHECKSUM_OFFSET_IN_FIXED_HEADER);
 hash.update(data.data() + FIXED_HEADER_SIZE, data.size() - FIXED_HEADER_SIZE);
 UInt128 checksum = hash.get128();
 transformEndianness<std::endian::little>(checksum);
 memcpy(data.data() + CHECKSUM_OFFSET_IN_FIXED_HEADER, &checksum, sizeof(checksum));

Probing and writing

Writing is best-effort and never throws. Before writing, the existing entry is probed to see whether it is fresh, being written concurrently, or garbage:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:365
QueryResultCacheOnDisk::ProbeResult QueryResultCacheOnDisk::probeExistingEntry(const FileCacheKey & cache_key) const
{
 const auto & user_id = FileCache::getCommonOrigin().user_id;

 auto holder = file_cache->getDownloadedContiguousOrEmpty(cache_key, 0, FIXED_HEADER_SIZE, user_id);
 for (size_t attempt = 0; holder->empty(); ++attempt)
 {
 /// No readable header: either there is no entry, or another query is writing one (its segments are held and not
 /// downloaded yet), or only the tail of a partially evicted entry is left. The tail must be removed, otherwise no
 /// entry could be created for the key until it is evicted too.
 const auto segments = file_cache->tryGetFileSegmentInfos(cache_key, user_id);
 if (segments.empty())
 return ProbeResult::None;
 for (const auto & segment : segments)
 {
 if (segment.state != FileSegment::State::DOWNLOADED || segment.references > 1)
 return ProbeResult::InProgress;
 }

The write itself reserves the space for all the segments of the entry before writing any of them, so an entry is either complete or absent:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:529
 /// Create all file segments of the entry at once. This fails gracefully if another query created an entry for the same key in
 /// the meantime (then we simply skip the insert), and guarantees that we exclusively own all segments of the key while writing.
 /// A reader sees the incomplete entry as a miss because not all of its bytes are downloaded yet.
 auto holder = file_cache->trySet(cache_key, 0, data.size(), CreateFileSegmentSettings(FileSegmentKind::Regular), origin);
 if (!holder)
 {
 LOG_TRACE(logger, "Skipped insert into the on-disk query result cache because of a concurrent insert for query {}",
 doubleQuoteString(key.query_string));
 return;
 }

The reader side is symmetric: it downloads the whole entry, verifies the checksum, and only then decompresses it.

Using it

The cache is off by default; you turn it on per query or per session:

 xml
 
 
<!-- Server configuration: a 1 GiB in-memory query result cache. -->
<query_cache>
 <max_size_in_bytes>1073741824</max_size_in_bytes>
 <max_entries>1024</max_entries>
 <max_entry_size_in_bytes>1048576</max_entry_size_in_bytes>
 <max_entry_size_in_rows>30000000</max_entry_size_in_rows>
</query_cache>

 sql
 
 
SELECT count() FROM hits WHERE UserID = 42 SETTINGS use_query_cache = true;
-- the second execution is served from the cache

The per-user quota is a setting, not a config element, so it can differ per profile: query_cache_max_size_in_bytes and query_cache_max_entries.

The observable state lives in system tables:

 sql
 
 
SELECT query_id, length, hits, QueryCacheHits, QueryCacheMisses FROM system.query_log LIMIT 10;
SELECT query, result_size, tag, stale, shared, compressed, expires_at FROM system.query_result_cache LIMIT 10;

SYSTEM CLEAR QUERY CACHE clears everything, SYSTEM CLEAR QUERY CACHE TAG '<tag>' clears one tag’s worth.

Trade-offs

What it buys: a dashboard that runs the same heavy aggregate every 10 seconds stops paying for the aggregate, and the server’s CPU goes to the queries that are actually different.

What it costs:

Staleness is unbounded up to the TTL. The cache is not invalidated on INSERT. A TTL of 1 hour means an hour-old answer. This is the documented contract, not a bug, but it is the reason the cache is opt-in.
The key is a hash, so it inherits hash-collision risk. Two different queries with the same 128-bit AST hash would share an entry. In practice this does not happen; there is no secondary check of the query text.
Memory is the scarce resource. The in-memory cache holds result data, not plans, and the results are held compressed. A server with a 1 GiB cache can easily be asked for 10 GiB of results; the size limits (max_entry_size_in_bytes, max_entry_size_in_rows) exist to keep one huge result from evicting everything else.
Compression is a trade, not a win. The default query_cache_compress_entries makes reads slower and saves RAM; on a server that is memory-bound it is right, on one that is CPU-bound it is wrong.
The on-disk backend has its own failure modes: partially evicted entries, entries written by a different build, a filesystem cache that is not yet initialised. All of them are handled by treating the entry as a miss, and the probe-then-write dance means two concurrent writers cannot corrupt each other — one of them simply loses.

The final irony is that the cache’s hardest problem — the key — is solved by ClickHouse’s parser rather than by the cache: hashing the AST instead of the text means select and SELECT share an entry, and the planner’s internal alias noise does not leak into the key.

How ClickHouse Aggregates In Order and Stops Reading Early

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

SELECT count() FROM hits GROUP BY UserID is, conceptually, a hash table build: read every row, hash every key, update a counter. The hash table’s size is the number of distinct keys, which for a column with a million distinct values is a million entries — and for a column with a billion distinct values is a billion entries, and the query dies.

But if the table is sorted by UserID, that is a lot of wasted work. The rows arrive grouped: all the rows with UserID = 1 are contiguous, then all the rows with UserID = 2, and so on. You do not need a hash table at all. You need to count the run, emit it, count the next run, emit it.

That is the optimize_aggregation_in_order optimisation. It turns an unbounded-memory operation into a streaming one whose memory footprint is the size of a single group. And, in recent versions, it also enables an even more aggressive trick: pushing the LIMIT down into the aggregation so that the read stops early.

This is a tour of that machinery — the plan-level decision, the run-by-run aggregation loop, and the limit pushdown with its subtle “prefix boundary” rule.

When the optimisation applies

The decision is made in AggregatingStep::transformPipeline, and it is based on a single field: sort_description_for_merging. If that is non-empty, the input is already sorted on the group-by key and the in-order path is taken.

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:505
 /// optimize_aggregation_in_order
 if (!sort_description_for_merging.empty())
 {
 /// two-level aggregation is not supported anyway for in order aggregation.
 allow_to_use_two_level_group_by = false;

 /// It is incorrect for in order aggregation.
 params.stats_collecting_params.disable();

 /// Aggregation in order rebuilds the aggregation-method state for every run of equal
 /// order-key values, so the whole-block `prealloc_serialized` method would make it
 /// quadratic. Fall back to the plain `serialized` method (see `Params::aggregation_in_order`).
 params.aggregation_in_order = true;
 }

Three consequences fall out of this one branch, and each is a real limitation:

Two-level aggregation is disabled. The classic ClickHouse optimisation for GROUP BY with many distinct keys is two-level aggregation: hash the key, bucket by hash, aggregate each bucket in a separate hash table, and merge the buckets in parallel. That is exactly what you do not want when the data is already grouped — the whole point is to avoid a hash table entirely.

Statistics collection is disabled. The uniqCombined / uniqExact family of aggregate functions maintains auxiliary state to compute statistics that are only meaningful for the whole dataset. In-order aggregation finalises each group as it goes, so those statistics would be wrong.

The prealloc_serialized aggregation method is downgraded. This one deserves an explanation, because it is a performance pathology that the comment records:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:849
 /// See `Params::aggregation_in_order` and `method_chosen_for_in_order`: the `prealloc_serialized`
 /// method serializes the whole block's keys on state construction, which is pathological for the
 /// per-run in-order path, where a fresh state is constructed for every run of equal order-key
 /// values. There it uses the plain `serialized` method, whose construction is O(1) and which
 /// serializes keys lazily. The whole-block paths (including `mergeBlocks`) keep `method_chosen`.
 method_chosen_for_in_order = method_chosen;
 if (params.aggregation_in_order)
 {
 if (method_chosen_for_in_order == AggregatedDataVariants::Type::prealloc_serialized)
 method_chosen_for_in_order = AggregatedDataVariants::Type::serialized;

prealloc_serialized is an optimisation for the hash table case: when you create a state for a key, you serialise the key into a contiguous buffer right away, so the key data is contiguous and the hash table lookup is cheap. In the in-order case a fresh state is created for every run of equal keys — potentially millions of times for a million groups — and serialising the whole key each time would be quadratic in the key width. The lazy serialized variant amortises it.

The pipeline shape

When the in-order path is taken and there is more than one stream, the pipeline looks like this:

 text
 
 
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket

as the comment in AggregatingStep says:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:678
 /** The pipeline is the following:
 *
 * --> AggregatingInOrder --> MergingAggregatedBucket
 * --> AggregatingInOrder --> FinishAggregatingInOrder --> ResizeProcessor --> MergingAggregatedBucket
 * --> AggregatingInOrder --> MergingAggregatedBucket
 */

Each stream runs its own AggregatingInOrderTransform, each with its own ManyAggregatedData variant, and the partial results are merged later. The transform is constructed per stream:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:692
 return std::make_shared<AggregatingInOrderTransform>(
 header,
 transform_params,
 sort_description_for_merging,
 group_by_sort_description,
 max_block_size,
 aggregation_in_order_max_block_bytes / new_merge_threads,
 many_data,
 counter++,
 limit_hint,
 limit_hint_prefix_columns,

The counter++ is the stream index — each transform gets its own slot in the shared ManyAggregatedData, so the later merge knows which partial state belongs to which stream.

The core loop: aggregating one run at a time

AggregatingInOrderTransform::consume() is the heart of it. The key observation is that the input is sorted, so the rows belonging to one group form a contiguous interval, and the interval can be found with a binary search.

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:183
 size_t interval_index = 0;

 /// Will split block into segments with the same key
 while (key_end != rows)
 {
 /// Cancellation is only checked between work() calls, but one consume() over a chunk with many
 /// keys can run for a long time; check per key interval so a cancelled query stops promptly.
 if (isCancelled())
 {
 LOG_TEST(log, "Cancelled between key intervals");
 return;
 }

and the search itself:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:208
 /// Find the first position of new (not current) key in current chunk
 auto indices = collections::range(key_begin, rows);
 auto it = std::upper_bound(indices.begin(), indices.end(), cur_block_size - 1,
 [&](size_t lhs_row, size_t rhs_row)
 {
 return less(res_key_columns, key_columns, lhs_row, rhs_row, group_by_description);
 });

 key_end = (it == indices.end() ? rows : *it);

 /// Add data to aggr. state if interval is not empty. Empty when haven't found current key in new block.
 if (key_begin != key_end)
 {
 if (params->params.only_merge)
 {
 if (group_by_key)
 params->aggregator.mergeOnBlockSmall(variants, key_begin, key_end, aggregate_columns_data, key_columns_raw);
 else
 params->aggregator.mergeOnIntervalWithoutKey(variants, key_begin, key_end, aggregate_columns_data, is_cancelled);
 }
 else
 {
 if (group_by_key)
 params->aggregator.executeOnBlockSmall(variants, key_begin, key_end, key_columns_raw, aggregate_function_instructions.data());
 else
 params->aggregator.executeOnIntervalWithoutKey(variants, key_begin, key_end, aggregate_function_instructions.data());
 }
 }

res_key_columns holds the current key (one row), key_columns holds the whole chunk’s key column. upper_bound with the less comparator finds the first row whose key is strictly greater than the current key. Everything between key_begin and key_end belongs to the current group.

less() is the generic column-wise comparison:

 cpp
 
 
// src/Core/SortCursor.h:969
template <typename TLeftColumns, typename TRightColumns>
bool less(const TLeftColumns & lhs, const TRightColumns & rhs, size_t i, size_t j, const SortDescriptionWithPositions & descr)
{
 for (const auto & elem : descr)
 {
 size_t ind = elem.column_number;
 int res = elem.base.direction * lhs[ind]->compareAt(i, j, *rhs[ind], elem.base.nulls_direction);
 if (res < 0)
 return true;
 if (res > 0)
 return false;
 }

 return false;
}

So the loop is: binary-search the end of the current run, aggregate the run into the current state, finalise the state into the output columns, record the next key, repeat.

Two modes: with a key and without

group_by_key selects between two quite different implementations. The header comment on the aggregator methods explains the split:

 cpp
 
 
// src/Interpreters/Aggregator.h:553
 /// Used for optimize_aggregation_in_order:
 /// - No two-level aggregation
 /// - No external aggregation
 /// - No without_key support (it is implemented using executeOnIntervalWithoutKey())
 void executeOnBlockSmall(
 AggregatedDataVariants & result,
 size_t row_begin,
 size_t row_end,
 ColumnRawPtrs & key_columns,
 AggregateFunctionInstruction * aggregate_instructions) const;

 void executeOnIntervalWithoutKey(
 AggregatedDataVariants & data_variants,
 size_t row_begin,
 size_t row_end,
 AggregateFunctionInstruction * aggregate_instructions) const;

group_by_key is true when the GROUP BY key is a prefix of the sort description — i.e. the input is sorted by exactly the grouping key. In that case there is only ever one group at a time, so no hash table is needed at all: the aggregate state is a single object that is created, filled, and finalised per run.

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:147
 /// If we don't have a block we create it and fill with first key
 if (!cur_block_size)
 {
 res_key_columns.resize(params->params.keys_size);
 for (size_t i = 0; i < params->params.keys_size; ++i)
 res_key_columns[i] = res_header.safeGetByPosition(i).type->createColumn();

 if (group_by_key)
 params->aggregator.fillKeyColumnsWithSingleKey(key_columns, key_begin, res_key_columns);
 else
 params->aggregator.createStatesAndFillKeyColumnsWithSingleKey(variants, key_columns, key_begin, res_key_columns);

fillKeyColumnsWithSingleKey — the state is not created; the key is just copied into the output column. createStatesAndFillKeyColumnsWithSingleKey — a state is created in the hash table, because the grouping key is a prefix of the sort key but not the whole of it, so several distinct groups may share the same sort-key prefix and must be kept apart.

When the group is finished, the state is written out and the key advances:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:239
 /// We finalize last key aggregation state if a new key found.
 if (key_end != rows)
 {
 if (!group_by_key)
 params->aggregator.addSingleKeyToAggregateColumns(variants, res_aggregate_columns);

 /// Enough groups have been emitted and the new key starts a new value of the
 /// `ORDER BY` prefix, so the rest of the input cannot be needed (see `limit_prefix_columns`).
 /// With `group_by_key`, `cur_block_size` counts runs of the sorted key columns rather than
 /// groups; the groups accumulated for the current block are the entries of the hash table.
 size_t cur_block_groups = group_by_key ? variants.size() : cur_block_size;
 if (limit_hint && cur_block_groups + res_rows >= limit_hint && isLimitPrefixBoundary(key_columns, key_end))
 limit_reached = true;

And the “the rest of the chunk is not needed” case:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:253
 /// If max_block_size or limit_hint is reached we have to stop consuming and generate the block. Save the extra rows into new chunk.
 if (cur_block_size >= max_block_size || cur_block_bytes + current_memory_usage >= max_block_bytes || limit_reached)
 {
 if (group_by_key)
 group_by_chunk
 = params->aggregator.prepareChunkAndFillSingleLevel</* return_single_block */ true>(variants, /* final= */ false).chunk;
 cur_block_bytes += current_memory_usage;

 /// When limit is reached, don't save leftover rows — we're done.
 /// Set block_end_reached + need_generate to trigger generate(),
 /// which will produce the output and then set is_consume_finished.
 if (limit_reached)
 {
 block_end_reached = true;
 need_generate = true;
 variants.invalidate();
 }
 else
 {
 finalizeCurrentChunk(std::move(chunk), key_end);
 }

Note the difference between the two branches. Normally the leftover rows of the chunk (those after key_end) are saved — finalizeCurrentChunk cuts them off and keeps them for the next consume():

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:298
void AggregatingInOrderTransform::finalizeCurrentChunk(Chunk chunk, size_t key_end)
{
 size_t rows = chunk.getNumRows();
 Columns source_columns = chunk.detachColumns();

 for (auto & source_column : source_columns)
 source_column = source_column->cut(key_end, rows - key_end);

 current_chunk = Chunk(source_columns, rows - key_end);
 src_rows -= current_chunk.getNumRows();

 block_end_reached = true;
 need_generate = true;
 variants.invalidate();
}

But when the limit is reached, the leftovers are dropped: the query is over, and reading the rest of the chunk would be wasted work.

Pushing `LIMIT` into the aggregation

The limit pushdown is a plan-level optimisation, in src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp, and it is gated on the optimize_aggregation_in_order_limit setting, which defaults to true:

text

// src/Core/Settings.cpp:5163
 DECLARE(Bool, optimize_aggregation_in_order_limit, true, R"(
When enabled and aggregation in order is active, pushes LIMIT into the aggregation step to enable early termination after producing enough groups. This reduces the amount of data read when ORDER BY matches the GROUP BY key prefix. May reduce the value reported by `rows_before_limit_at_least`; use `exact_rows_before_limit` if exact counts are needed.
)", 0) \

The optimisation looks for the shape LIMIT → SORT → AGGREGATE:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:77
void optimizeLimitForAggregationInOrder(QueryPlan::Node & root)
{
 Stack stack;
 stack.push_back({.node = &root});

 while (!stack.empty())
 {
 auto & frame = stack.back();

 if (frame.next_child < frame.node->children.size())
 {
 stack.push_back({.node = frame.node->children[frame.next_child++]});
 continue;
 }

 auto * node = frame.node;
 stack.pop_back();

 auto * limit_step = typeid_cast<LimitStep *>(node->step.get());
 if (!limit_step || limit_step->withTies() || limit_step->alwaysReadTillEnd())
 continue;

 size_t limit = limit_step->getLimitForSorting();
 if (!limit)
 continue;

WITH TIES and always_read_till_end are excluded immediately — both change the meaning of the limit in ways the pushdown cannot honour.

Then it walks down through “transparent” steps (steps that neither change the row count nor the sort order) looking for the aggregation:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:117
 /// Find AggregatingStep below SortingStep, skipping transparent steps.
 /// Stop at TotalsHavingStep — HAVING may filter groups.
 while (current)
 {
 auto * aggregating_step = typeid_cast<AggregatingStep *>(current->step.get());
 if (aggregating_step)
 {
 /// overflow_row is set by WITH TOTALS.
 if (!aggregating_step->inOrder()
 || !aggregating_step->getFinal()
 || aggregating_step->getParams().overflow_row
 || aggregating_step->isGroupingSets())
 break;

and then the crucial check:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:131
 /// The sort description may be a strict prefix of the group-by sort description.
 /// Each in-order stream then stops only at a boundary of that prefix (see
 /// `AggregatingInOrderTransform`), so the groups it drops sort strictly after
 /// at least `limit` complete groups and the final sort can never pick a group
 /// with a partial aggregate value (issue #116849).
 const auto & sort_desc = sorting_step->getSortDescription();
 const auto & agg_sort_desc = aggregating_step->getGroupBySortDescription();
 if (sort_desc.empty() || !agg_sort_desc.hasPrefix(sort_desc))
 break;

 /// Use the smallest limit if multiple LimitSteps point to the same AggregatingStep.
 size_t current_hint = aggregating_step->getLimitHint();
 if (!current_hint || limit < current_hint)
 aggregating_step->setLimitHint(limit, sort_desc.size());
 break;

agg_sort_desc.hasPrefix(sort_desc) — the ORDER BY columns must be a prefix of the GROUP BY sort description. This is the condition that makes the pushdown safe, and the reasoning in the comment is worth unpacking because it is subtle.

The prefix boundary rule

The ORDER BY of the query may be a prefix of the GROUP BY key. Say:

 sql
 
 
SELECT a, b, count()
FROM t
GROUP BY a, b
ORDER BY a
LIMIT 10

Here the table is sorted by (a, b). The GROUP BY produces groups in (a, b) order. The ORDER BY a then sorts them by a — and since the groups already come out in (a, b) order, sorting by a is a stable reordering: groups with the same a keep their relative (a, b) order.

Now: the aggregation stream can stop early once it has produced 10 groups. But it must not stop in the middle of a run of groups that share the same a — because another stream might produce a group with the same a but a smaller b, and the final sort (by a only) could then pick that group over the partial one.

The rule is: stop only at a boundary of the ORDER BY prefix. That is what limit_prefix_columns counts and what isLimitPrefixBoundary() checks:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:85
bool AggregatingInOrderTransform::isLimitPrefixBoundary(const Columns & key_columns, size_t row) const
{
 for (size_t i = 0; i < limit_prefix_columns; ++i)
 {
 const auto & elem = group_by_description[i];
 size_t ind = elem.column_number;
 if (res_key_columns[ind]->compareAt(cur_block_size - 1, row, *key_columns[ind], elem.base.nulls_direction) != 0)
 return true;
 }
 return false;
}

It compares the last emitted group’s value of the prefix column with the next group’s value. If they differ, the run of equal prefix values has ended, and it is safe to stop: every group with that prefix value has been emitted.

Note the comment on the member:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.h:97
 /// The stream may stop early once it has emitted `limit_hint` groups, but only at a
 /// boundary of the first `limit_prefix_columns` columns of `group_by_description`, which
 /// are the columns the query is ordered by. Otherwise a group tied on those columns with
 /// the emitted ones could be cut off here while another stream emits its partial state,
 /// and the final sort could pick that partial group.
 size_t limit_hint = 0;
 size_t limit_prefix_columns = 0;
 bool limit_reached = false;

And the prefix count is clamped to the group-by description:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:80
 /// The `ORDER BY` may extend past the columns the input is sorted by; a boundary of the
 /// sorted columns alone is then a boundary of the `ORDER BY` prefix as well.
 limit_prefix_columns = std::min(limit_prefix_columns_, group_by_description.size());

This is the bug that issue #116849 was about, and the comment in the optimisation pass records it.

What it buys, concretely

 sql
 
 
SELECT user_id, count()
FROM hits
GROUP BY user_id
ORDER BY user_id
LIMIT 10

on a table sorted by user_id. Without the pushdown, the whole table is read, aggregated, sorted, and then 10 rows are kept. With it, the aggregation stops after the first 10 groups, and — crucially — the source stops too, because the transform closes its input:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:354
 /// When limit is reached, we set is_consume_finished in generate()
 /// and don't save leftover rows. Finish immediately after pushing.
 if (is_consume_finished)
 {
 output.finish();
 input.close();
 LOG_DEBUG(log, "Aggregated. {} to {} rows (from {})", src_rows, res_rows, formatReadableSizeWithBinarySuffix(src_bytes));
 return Status::Finished;
 }

input.close() propagates the stop all the way up to the storage layer. On a MergeTree, that means the remaining granules are never read.

The cost is stated in the setting’s description: rows_before_limit_at_least becomes an underestimate, because the rows that were never read are not counted. exact_rows_before_limit restores the exact count at the price of reading everything.

Trade-offs

What this buys you.

The in-order aggregation replaces a hash table whose size is the number of distinct keys with a state whose size is one group. For a table sorted by the grouping key, that is the difference between a query that fits in memory and one that does not.

It also composes with the parallel execution model: each stream aggregates its own runs independently, and the partial states are merged at the end by FinishAggregatingInOrderTransform and MergingAggregatedBucketTransform. The parallelism is preserved; only the memory behaviour changes.

The limit pushdown extends the idea from “less memory” to “less work”: the read itself stops. On a large MergeTree with a selective LIMIT, this is the difference between reading 10 granules and reading a thousand.

What it costs.

The optimisation is conditional on the data layout. If the table is not sorted by the grouping key, the plan does not use it — and if the table is sorted but the GROUP BY key is not a prefix of the sort key, it does not apply either. ClickHouse has to prove the sortedness, and it does so from the table’s primary key and the optimize_read_in_order machinery, so the whole optimisation is contingent on the optimiser’s knowledge of the physical layout.

The in-order path gives up several general-purpose optimisations: two-level aggregation, external aggregation, and statistics collection. For a query with a small number of distinct keys on a large table, the ordinary hash-table path with two-level aggregation might actually be faster, because it parallelises the hash table itself rather than relying on the input being pre-grouped. The optimize_aggregation_in_order setting defaults to false for exactly this reason.

The prealloc_serialized → serialized downgrade is a real regression in the hash-table case that the in-order case forced, paid for by every query that takes the in-order path.

The limit pushdown has a correctness-adjacent cost that the code is careful about but that is easy to get wrong: the prefix-boundary rule. Stop at the wrong place and the final sort may pick a group whose aggregate state was truncated — a wrong answer, not a slow one. That is why the rule is conservative (stop only at a prefix boundary, never in the middle of a run of equal prefix values) and why the plan-level check requires the ORDER BY to be a prefix of the group-by description.

And the pushdown changes the meaning of a system counter: rows_before_limit_at_least becomes “rows that were actually read”, not “rows that would have been read without the limit”. Any monitoring built on that counter has to know about the optimisation.

How tmux Parses Escape Sequences Fast

Jay Hubbert — Sun, 11 Oct 2026 10:00:00 +0000

The single hottest path in tmux is not the redraw — it is the parser. Every byte that any program in any pane produces goes through tmux’s own VT parser, and a yes or a cat of a large file will push megabytes through it per second.

This article is about how that parser is built: a range-compressed state machine with a one-entry transition cache, a binary search over the CSI command table, a character trie for the other direction (terminal key sequences), and a Unicode width cache.

The parser: a state machine with tables

input.c is about 3800 lines, and the core of it is a table-driven state machine in the style of a real VT parser. There are ten states: ground, escape enter, escape intermediate, CSI enter, CSI parameter, CSI intermediate, OSC, DCS, APC, and the string-consumption states.

Each state has a list of transitions, and each transition is a range of bytes:

 c
 
 
struct input_transition {
	int			first;
	int			last;

	const struct input_state	*state;
	int			(*handler)(struct input_ctx *ictx);
};

The reason for ranges rather than a 256-entry table per state is memory: a transition is {int, int, pointer, pointer} — 24 bytes on 64-bit — and a full table would be 6 KB per state. With ranges, a state might have five transitions.

The parser core:

 c
 
 
/* Parse data. */
static void
input_parse(struct input_ctx *ictx, const u_char *buf, size_t len)
{
	struct screen_write_ctx		*sctx = &ictx->ctx;
	const struct input_state	*state = NULL;
	const struct input_transition	*itr = NULL;
	size_t				 off = 0;

	/* Parse the input. */
	while (off < len) {
		ictx->ch = buf[off++];

		/* Find the transition. */
		if (ictx->state != state ||
		 itr == NULL ||
		 ictx->ch < itr->first ||
		 ictx->ch > itr->last) {
			itr = ictx->state->transitions;
			while (itr->first != -1 && itr->last != -1) {
				if (ictx->ch >= itr->first &&
				 ictx->ch <= itr->last)
					break;
				itr++;
			}
			if (itr->first == -1 || itr->last == -1) {
				/* No transition? Eh? */
				fatalx("no transition from state");
			}
		}
		state = ictx->state;

The trick is the itr cache. The state machine remembers the transition it used last time, and only rescans the table if:

the state changed, or
the character is outside the range of the cached transition.

For a run of printable ASCII in the ground state, the first character scans the table and every subsequent character hits the cached transition. The scan is a linear walk over ranges, which for the ground state is a handful of comparisons.

The collect-buffer optimisation

The comment above screen_write_collect_end() is a nice piece of engineering honesty:

 c
 
 
		/*
		 * Any state except print stops the current collection. This is
		 * an optimization to avoid checking if the attributes have
		 * changed for every character. It will stop unnecessarily for
		 * sequences that don't make a terminal change, but they should
		 * be the minority.
		 */
		if (itr->handler != input_print)
			screen_write_collect_end(sctx);

The idea: tmux batches consecutive printable characters into a “collect buffer” so that they can be written to the grid in one go. But collecting requires checking, for each character, whether the attributes have changed — which is the expensive part.

Instead of checking, the parser uses a proxy: if the transition is not the plain input_print handler, then the sequence is doing something (setting a colour, moving the cursor) and the collection is ended. That is a conservative over-approximation — a sequence that does not actually change attributes still ends the collection — but plain text is by far the common case, so the fast path stays fast.

Binary search over the CSI table

Once the parser reaches the CSI dispatch point, it has a final byte and a parameter string, and it needs to find the handler. The table is sorted and searched with bsearch():

 c
 
 
	entry = bsearch(ictx, input_csi_table, nitems(input_csi_table),
	 sizeof input_csi_table[0], input_table_compare);
	if (entry == NULL) {
		log_debug("%s: unknown '%c'", __func__, ictx->ch);
		return (0);
	}

with the comparator:

 c
 
 
static int
input_table_compare(const void *key, const void *value)
{
	const struct input_ctx		*ictx = key;
	const struct input_table_entry	*entry = value;
	if (ictx->ch != entry->ch)
		return (ictx->ch - entry->ch);
	return (strcmp(ictx->interm_buf, entry->interm));
}

The comparison is by final byte first, then by the intermediate string — which is how CSI ? ... h (private mode) is distinguished from CSI ... h (set mode). The same pattern is used for the ESC dispatch table.

Bounded buffers

The parser context has fixed-size scratch buffers for the small parts of a sequence:

 c
 
 
	u_char				interm_buf[4];
	size_t				interm_len;

	u_char				param_buf[64];
	size_t				param_len;

#define INPUT_BUF_START 32
	u_char			 *input_buf;
	size_t				input_len;
	size_t				input_space;
	enum input_end_type		input_end;

	struct input_param		param_list[24];
	u_int				param_list_len;

The intermediate string of an escape sequence is at most 4 bytes and the parameter string at most 64. A sequence that exceeds them is discarded, not buffered — an important denial-of-service defence, since a hostile program inside a pane could otherwise send an endless CSI 111111111... and make tmux allocate without bound.

The one growable buffer is the OSC/DCS payload, and it is bounded too:

 c
 
 
	while (ictx->input_len + 1 >= available) {
		available *= 2;
		if (available > input_buffer_size) {
			ictx->flags |= INPUT_DISCARD;
			return (0);
		}
	}

input_buffer_size defaults to INPUT_BUF_DEFAULT_SIZE (1 MB) and is configurable with the input-buffer-size option. When the buffer is discarded, the parser resets and waits for the next sequence — the rest of the garbage is dropped.

The buffer is also shrunk back when the parser returns to the ground state, so one huge OSC does not permanently inflate memory.

The other direction: terminal key sequences

The parser above handles bytes flowing out of a program. The reverse direction — bytes coming from the user’s terminal, which may be an escape sequence rather than a keystroke — is handled by tty-keys.c.

The difficulty is that the terminal sends a byte stream and tmux has to decide whether ESC [ A is the up arrow, or the Escape key followed by [ and A. The answer depends on timing and on what tmux asked the terminal.

tmux stores terminal key sequences in a character trie, where each node has up to three children:

 c
 
 
/* A key tree entry. */
struct tty_key {
	char		 ch;
	key_code	 key;

	struct tty_key	*left;
	struct tty_key	*right;

	struct tty_key	*next;
};

The lookup is a simple binary-tree walk with a “next” pointer for the sequence continuation:

 c
 
 
/* Find the next node. */
static struct tty_key *
tty_keys_find1(struct tty_key *tk, const char *buf, size_t len, size_t *size)
{
	/* If no data, no match. */
	if (len == 0)
		return (NULL);

	/* If the node is NULL, this is the end of the tree. No match. */
	if (tk == NULL)
		return (NULL);

	/* Pick the next in the sequence. */
	if (tk->ch == *buf) {
		/* Move forward in the string. */
		buf++; len--;
		(*size)++;

		/* At the end of the string, return the current node. */
		if (len == 0 || (tk->next == NULL && tk->key != KEYC_UNKNOWN))
			return (tk);

		/* Move into the next tree for the following character. */
		tk = tk->next;
	} else {
		if (*buf < tk->ch)
			tk = tk->left;
		else if (*buf > tk->ch)
			tk = tk->right;
	}

	/* Move to the next in the tree. */
	return (tty_keys_find1(tk, buf, len, size));
}

The left/right pointers are a binary search tree keyed on the byte; the next pointer continues the sequence to the next trie. So the structure is a tree of trees, and the lookup is O(log n) per byte with no allocation.

Terminal replies (device attributes, colour reports, kitty graphics queries) are matched before the general key tree, via a small table of function pointers:

 c
 
 
/* Terminal replies, in the order they are checked. */
static const struct {
	key_code	key;
	int	 (*fn)(struct tty *, const char *, size_t, size_t *, int);
} tty_keys_replies[] = {
	{ KEYC_REPORT_CLIPBOARD, tty_keys_clipboard },
	{ KEYC_REPORT_SYNC, tty_keys_sync },
	{ KEYC_REPORT_DA, tty_keys_device_attributes },
	{ KEYC_REPORT_DA2, tty_keys_device_attributes2 },
	{ KEYC_REPORT_XDA, tty_keys_extended_device_attributes },
	{ KEYC_REPORT_COLOURS, tty_keys_colours1 },
	{ KEYC_REPORT_PALETTE, tty_keys_palette },
	{ KEYC_REPORT_WINSZ, tty_keys_winsz },

That ordering matters: a device attributes reply is ESC [ ? 6 2 ; 1 6 c, which would otherwise be an unknown key sequence.

The escape timeout

The ambiguity between Escape and a sequence prefix is resolved with a timer, and tmux’s timer is adaptive:

 c
 
 
	/* Get the time period. */
	delay = options_get_number(global_options, "escape-time");
	if (delay == 0)
		delay = 1;
	if ((tty->flags & TTY_BRACKETPASTE) &&
	 tty_keys_partial_paste_end(buf, len)) {
		log_debug("%s: increasing delay (partial paste end)", c->name);
		if (delay < 500)
			delay = 500;
	}
	if (tty->flags & (TTY_WAITFG|TTY_WAITBG) ||
	 tty->flags & (TTY_OSC52QUERY|TTY_WINSIZEQUERY) ||
	 (tty->flags & TTY_ALL_REQUEST_FLAGS) != TTY_ALL_REQUEST_FLAGS ||
	 !TAILQ_EMPTY(&c->input_requests)) {
		log_debug("%s: increasing delay (active query)", c->name);
		if (delay < 500)
			delay = 500;
	}

and then, in two specific situations, the delay is raised to at least 500 ms:

when the buffer ends with a partial bracketed paste end sequence (ESC [ 201 ~), and
when tmux has an outstanding terminal query and is waiting for a reply.

The second case is the interesting one. If tmux has just asked the terminal for its device attributes and the user presses Escape, the reply and the keystroke arrive in the same stream. Waiting a little longer makes it much more likely that the reply is consumed as a reply rather than being misinterpreted as a Meta-prefixed key.

The exponential-ish retry is also handled: when the timer expires without a match, the buffer is re-parsed with “no longer ambiguous” semantics, so a lone Escape eventually arrives as the Escape key.

There is a matching problem on the reply side: an Escape immediately before a terminal reply must not be swallowed as Meta. That has its own regression test (regress/tty-keys-reply-escape.sh).

UTF-8 and the width cache

The last hot path is Unicode. Decoding UTF-8 incrementally is the easy part; the hard part is that wcwidth() is not cheap, and a screen full of emoji would call it thousands of times per second.

tmux keeps a cache of codepoint widths in a red-black tree:

 c
 
 
struct utf8_width_item {
	wchar_t				wc;
	u_int				width;
	int				allocated;

	RB_ENTRY(utf8_width_item)	entry;
};

seeded with a static table of a few hundred known cases:

 c
 
 
static struct utf8_width_item utf8_default_width_cache[] = {
	{ .wc = 0x0270A, .width = 2 },
	{ .wc = 0x0270B, .width = 2 },
	{ .wc = 0x1F1E6, .width = 1 },
	{ .wc = 0x1F1E7, .width = 1 },
	{ .wc = 0x1F1E8, .width = 1 },

The lookup happens before wcwidth() is called at all:

 c
 
 
/* Get width of Unicode character. */
static enum utf8_state
utf8_width(struct utf8_data *ud, int *width)
{
	struct utf8_width_item	*uw;
	wchar_t			 wc;

	if (utf8_towc(ud, &wc) != UTF8_DONE)
		return (UTF8_ERROR);
	uw = utf8_find_in_width_cache(wc);
	if (uw != NULL) {
		*width = uw->width;
		log_debug("cached width for %08X is %d", (u_int)wc, *width);
		return (UTF8_DONE);
	}
#ifdef HAVE_UTF8PROC
	*width = utf8proc_wcwidth(wc);

There are similar caches for the codepoint-to-utf8_data conversion (utf8_data_tree and utf8_index_tree), so a repeated character does not have to be re-decoded.

The width is also packed into the character word itself rather than kept in a separate structure:

 c
 
 
#define UTF8_GET_SIZE(wc) ...
#define UTF8_GET_WIDTH(wc) ...
#define UTF8_SET_SIZE(wc, sz) ...
#define UTF8_SET_WIDTH(wc, wd) ...

so the width is available without dereferencing anything.

Why the widths are so complicated

The width rules are a mess of Unicode special cases, and tmux handles several of them explicitly:

Combining marks (utf8-combined.c) — a combining character is rendered on top of the previous cell, and does not advance the cursor.
Variation selectors — VS16 (U+FE0F) requests the emoji presentation, which is usually double-width. tmux has an option variation-selector-always-wide because the correct behaviour depends on the outer terminal, which tmux cannot always detect.
Regional indicators (flag emoji) — a pair of regional indicator symbols is one flag, so they combine.
Emoji modifiers (skin tones) — combine with the preceding emoji.
Private use area — where Powerline glyphs live, which have “ambiguous” width. The bundled utf8proc shim says:

 c
 
 
	if (cat == UTF8PROC_CATEGORY_CO) {
		/* The private use category is where powerline and similar
		 * codepoints are stored, they have "ambiguous" width - use 1. */
		return (1);
	}

The lesson

There is no clever trick in the tmux parser. It is a state machine with a transition cache, a binary search, a trie, and a width cache — four small, boring optimisations, each of which removes work from the innermost loop.

What makes it worth writing down is the shape of the optimisation. Each one replaces a check with a cheaper check:

Expensive	Cheap replacement
256-entry transition table per state	ranges + a one-entry cache
linear scan of the CSI table	`bsearch()`
`wcwidth()` per character	red-black tree of known widths
linear scan of key sequences	binary search tree per node
attribute check per collected character	“is this the print handler?”

That is the whole story of tmux’s input performance: never do the expensive check when a cheaper one is sufficient, and make the cheap one the default path.

How PostgreSQL Speeds Up Hash Joins

Jay Hubbert — Sun, 11 Oct 2026 09:00:00 +0000

A hash join is the simplest join algorithm in the database: build a hash table from the inner relation, then probe it once per outer row. There is no sort, no merge, no nested loop over the inner side.

And yet it is one of the most performance-sensitive pieces of the executor, because it does the same tiny bit of work an enormous number of times: compute a hash value, look it up, compare the key. For a join between a million-row hash table and a ten-million-row outer relation, that is ten million hash computations and ten million bucket scans, and each one is measured in tens of nanoseconds. Everything else is noise.

So PostgreSQL has spent the last few years shaving that loop down. The strategy has been consistent: move the hashing out of the general-purpose expression interpreter and into specialised C functions, selected by pattern-matching the compiled expression’s step list at startup time.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

Where the time goes

A hash join has two sides and two kinds of work.

Building the hash table. Every inner row’s join key has to be hashed and the row inserted into a bucket chain. This is done once per inner row.

Probing. Every outer row’s join key has to be hashed, the resulting bucket found, and the bucket chain walked, comparing keys until a match is found. This is done once per outer row, and it is the side that dominates: a hash join between a small inner table and a huge outer relation does enormously more probing than building.

Both sides need the same hash function for a given join key, and PostgreSQL has to compute it from an arbitrary expression (the join key can be a.b + 1, or lower(name), or a plain column).

That expression is evaluated by the expression interpreter (src/backend/executor/execExprInterp.c), which compiles an expression tree into a flat list of steps and then runs them through a threaded dispatch loop. The interpreter is fast, but it has a fixed per-step overhead, and for a hash join the step list for “hash column 3” is tiny — four or five steps for what is, in machine code, about six instructions.

The opportunity is obvious: if you know the shape of the step list before you run it, you can write a function that does the whole thing without a dispatch loop at all.

Hashing as an expression

The first move was to make hashing an expression operation at all. Commit adf97c15623 (“Speed up Hash Join by making ExprStates support hashing”, August 2024) introduced a family of new opcodes and the ExecBuildHash32Expr() constructor. Its commit message explains the motivation:

This allows both faster hashing and also JIT compilation of these expressions. This is especially useful when hash joins have multiple join keys as the previous code called ExecEvalExpr on each hash join key individually and that was inefficient as tuple deformation would have only taken into account one key at a time, which could lead to walking the tuple once for each join key. With the new code, we’ll determine the maximum attribute required and deform the tuple to that point only once.

Some performance tests done with this change have shown up to a 20% performance increase of a query containing a Hash Join without JIT compilation and up to a 26% performance increase when JIT is enabled and optimization and inlining were performed by the JIT compiler.

The opcodes themselves, from src/include/executor/execExpr.h:

 c
 
 
	/* evaluation steps for hashing */
	EEOP_HASHDATUM_SET_INITVAL,
	EEOP_HASHDATUM_FIRST,
	EEOP_HASHDATUM_FIRST_STRICT,
	EEOP_HASHDATUM_NEXT32,
	EEOP_HASHDATUM_NEXT32_STRICT,

Five opcodes, with a clear division of labour:

SET_INITVAL seeds the hash with a caller-supplied 32-bit value.
FIRST hashes the first key and replaces the running value.
NEXT32 hashes the next key and combines it with the running value.
The _STRICT variants abort the whole computation when the key is NULL, instead of treating NULL as a zero hash.

The combination is a rotate-then-XOR, from the interpreter:

 c
 
 
		EEO_CASE(EEOP_HASHDATUM_NEXT32)
		{
			FunctionCallInfo fcinfo = op->d.hashdatum.fcinfo_data;
			uint32		existinghash;

			existinghash = DatumGetUInt32(op->d.hashdatum.iresult->value);
			/* combine successive hash values by rotating */
			existinghash = pg_rotate_left32(existinghash, 1);

			/* leave the hash value alone on NULL inputs */
			if (!fcinfo->args[0].isnull)
			{
				uint32		hashvalue;

				/* execute hash func and combine with previous hash value */
				hashvalue = DatumGetUInt32(op->d.hashdatum.fn_addr(fcinfo));
				existinghash = existinghash ^ hashvalue;
			}

			*op->resvalue = UInt32GetDatum(existinghash);
			*op->resnull = false;

			EEO_NEXT();
		}

And the FIRST variant, which has to cope with NULL by storing zero so that a subsequent NEXT32 still has something to combine with:

 c
 
 
		EEO_CASE(EEOP_HASHDATUM_FIRST)
		{
			FunctionCallInfo fcinfo = op->d.hashdatum.fcinfo_data;

			/*
			 * Save the Datum on non-null inputs, otherwise store 0 so that
			 * subsequent NEXT32 operations combine with an initialized value.
			 */
			if (!fcinfo->args[0].isnull)
				*op->resvalue = op->d.hashdatum.fn_addr(fcinfo);
			else
				*op->resvalue = (Datum) 0;

			*op->resnull = false;

			EEO_NEXT();
		}

Building the hash expression

ExecBuildHash32Expr() (src/backend/executor/execExpr.c) is the constructor. It takes the list of key expressions, the hash functions to use, and builds a step list:

 c
 
 
ExprState *
ExecBuildHash32Expr(TupleDesc desc, const TupleTableSlotOps *ops,
					const Oid *hashfunc_oids, const List *collations,
					const List *hash_exprs, const bool *opstrict,
					PlanState *parent, uint32 init_value)
{
	ExprState *state = makeNode(ExprState);
	ExprEvalStep scratch = {0};
	NullableDatum *iresult = NULL;
	List	 *adjust_jumps = NIL;
	ListCell *lc;
	ListCell *lc2;
	intptr_t	strict_opcode;
	intptr_t	opcode;
	int			num_exprs = list_length(hash_exprs);

	Assert(num_exprs == list_length(collations));

	state->parent = parent;

	/* Insert setup steps as needed. */
	ExecCreateExprSetupSteps(state, unconstify(Node *, (const Node *) hash_exprs));

	/*
	 * Make a place to store intermediate hash values between subsequent
	 * hashing of individual expressions. We only need this if there is more
	 * than one expression to hash or an initial value plus one expression.
	 */
	if ((int64) num_exprs + (init_value != 0) > 1)
		iresult = palloc_object(NullableDatum);

The iresult slot is where the running hash value lives between keys. Notice the comment in the header of the function:

If any hash_expr yields NULL and the corresponding hash operator is strict, the created ExprState will return NULL. (If the operator is not strict, we treat NULL values as having a hash value of zero. The hash functions themselves are always treated as strict.)

The two modes matter, and the difference is the subject of a bug we will come back to.

The hash join node wires it up in ExecInitHashJoin() (src/backend/executor/nodeHashjoin.c):

 c
 
 
		/*
		 * Build ExprStates to obtain hash values for either side of the join.
		 * Note: must build the ExprStates before ExecHashTableCreate() so we
		 * properly attribute any SubPlans that exist in the hash expressions
		 * to the correct PlanState.
		 */
		nkeys = list_length(node->hashoperators);

		outer_hashfuncid = palloc_array(Oid, nkeys);
		inner_hashfuncid = palloc_array(Oid, nkeys);
		hash_strict = palloc_array(bool, nkeys);
		...
		/*
		 * Build an ExprState to generate the hash value for the expressions
		 * on the outer side of the join.
		 */
		hjstate->hj_OuterHash =
			ExecBuildHash32Expr(hjstate->js.ps.ps_ResultTupleDesc,
								hjstate->js.ps.resultops,
								outer_hashfuncid,
								node->hashcollations,
								node->hashkeys,
								hash_strict,
								&hjstate->js.ps,
								0);

		/* As above, but for the inner side of the join */
		hashstate->hash_expr =
			ExecBuildHash32Expr(hashstate->ps.ps_ResultTupleDesc,
								hashstate->ps.resultops,
								inner_hashfuncid,
								node->hashcollations,
								hash->hashkeys,
								hash_strict,
								&hashstate->ps,
								0);

Two ExprStates, one per side, each evaluating a list of key expressions into a single 32-bit hash.

The inner side is evaluated when the hash table is built (src/backend/executor/nodeHash.c):

 c
 
 
		slot = ExecProcNode(outerNode);
		if (TupIsNull(slot))
			break;
		/* We have to compute the hash value */
		econtext->ecxt_outertuple = slot;

		ResetExprContext(econtext);

		hashdatum = ExecEvalExprSwitchContext(node->hash_expr, econtext,
											 &isnull);

		if (!isnull)
		{
			/* normal case with a non-null join key */
			uint32		hashvalue = DatumGetUInt32(hashdatum);
			int			bucketNumber;

			bucketNumber = ExecHashGetSkewBucket(hashtable, hashvalue);
			if (bucketNumber != INVALID_SKEW_BUCKET_NO)
			{
				/* It's a skew tuple, so put it into that hash table */
				ExecHashSkewTableInsert(hashtable, slot, hashvalue,
										bucketNumber);
			}
			else
			{
				/* Not subject to skew optimization, so insert normally */
				ExecHashTableInsert(hashtable, slot, hashvalue);
			}
			hashtable->totalTuples += 1;
		}

And the outer side, once per probe, in nodeHashjoin.c:

 c
 
 
			*hashvalue = DatumGetUInt32(ExecEvalExprSwitchContext(hjstate->hj_OuterHash,
																 econtext,
																 &isnull));

ExecEvalExprSwitchContext() is a thin wrapper that switches to the per-tuple memory context and then calls state->evalfunc():

 c
 
 
static inline Datum
ExecEvalExprSwitchContext(ExprState *state,
						 ExprContext *econtext,
						 bool *isNull)
{
	Datum		retDatum;
	MemoryContext oldContext;

	oldContext = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
	retDatum = state->evalfunc(state, econtext, isNull);
	MemoryContextSwitchTo(oldContext);
	return retDatum;
}

So state->evalfunc is the hook. Whatever it points at is what runs ten million times.

The interpreter

Before the optimisation, evalfunc always pointed at ExecInterpExpr(), the threaded interpreter. Its header comment explains the dispatch strategy:

This file provides either a “direct threaded” (for gcc, clang and compatible) or a “switch threaded” (for all compilers) implementation of expression evaluation. The former is amongst the fastest known methods of interpreting programs without resorting to assembly level work, or just-in-time compilation, but it requires support for computed gotos.

In direct threading, we use gcc’s label-as-values extension - also adopted by some other compilers - to replace ExprEvalStep->opcode with the address of the block implementing the instruction. Dispatch to the next instruction is done by a “computed goto”. This allows for better branch prediction (as the jumps are happening from different locations) and fewer jumps (as no preparatory jump to a common dispatch location is needed).

Direct threading is the classic “threaded code” technique: each opcode’s implementation ends with a jump to the next opcode’s implementation, so there is no central dispatch point at all. The interpreter is about as fast as an interpreter gets.

But it is still an interpreter. Every step pays for the dispatch.

And here is the key observation from ExecReadyInterpretedExpr():

For very simple instructions the overhead of the full interpreter “startup”, as minimal as it is, is noticeable. Therefore ExecReadyInterpretedExpr will choose to implement certain simple opcode patterns using special fast-path routines (ExecJust*).

Pattern-matching the step list

ExecReadyInterpretedExpr() (src/backend/executor/execExprInterp.c) runs once, when the expression is prepared. Its job is to look at the compiled step list and decide: “is this one of the shapes I know how to do without an interpreter?”

 c
 
 
	/*
	 * Select fast-path evalfuncs for very simple expressions. "Starting up"
	 * the full interpreter is a measurable overhead for these, and these
	 * patterns occur often enough to be worth optimizing.
	 */
	if (state->steps_len == 5)
	{
		ExprEvalOp	step0 = state->steps[0].opcode;
		ExprEvalOp	step1 = state->steps[1].opcode;
		ExprEvalOp	step2 = state->steps[2].opcode;
		ExprEvalOp	step3 = state->steps[3].opcode;

		if (step0 == EEOP_INNER_FETCHSOME &&
			step1 == EEOP_HASHDATUM_SET_INITVAL &&
			step2 == EEOP_INNER_VAR &&
			step3 == EEOP_HASHDATUM_NEXT32)
		{
			state->evalfunc_private = (void *) ExecJustHashInnerVarWithIV;
			return;
		}
	}
	else if (state->steps_len == 4)
	{
		ExprEvalOp	step0 = state->steps[0].opcode;
		ExprEvalOp	step1 = state->steps[1].opcode;
		ExprEvalOp	step2 = state->steps[2].opcode;

		if (step0 == EEOP_OUTER_FETCHSOME &&
			step1 == EEOP_OUTER_VAR &&
			step2 == EEOP_HASHDATUM_FIRST)
		{
			state->evalfunc_private = (void *) ExecJustHashOuterVar;
			return;
		}
		else if (step0 == EEOP_INNER_FETCHSOME &&
				 step1 == EEOP_INNER_VAR &&
				 step2 == EEOP_HASHDATUM_FIRST)
		{
			state->evalfunc_private = (void *) ExecJustHashInnerVar;
			return;
		}
		else if (step0 == EEOP_OUTER_FETCHSOME &&
				 step1 == EEOP_OUTER_VAR &&
				 step2 == EEOP_HASHDATUM_FIRST_STRICT)
		{
			state->evalfunc_private = (void *) ExecJustHashOuterVarStrict;
			return;
		}
	}
	else if (state->steps_len == 3)
	{
		...
		else if (step0 == EEOP_INNER_VAR &&
				 step1 == EEOP_HASHDATUM_FIRST)
		{
			state->evalfunc_private = (void *) ExecJustHashInnerVarVirt;
			return;
		}
		else if (step0 == EEOP_OUTER_VAR &&
				 step1 == EEOP_HASHDATUM_FIRST)
		{
			state->evalfunc_private = (void *) ExecJustHashOuterVarVirt;
			return;
		}
	}

This is the whole trick, and it is worth reading slowly.

It matches on steps_len first. The step count is a cheap necessary condition, and it buckets the cases: five steps, four steps, three steps, two steps.

It then matches on the exact opcode sequence. Not “contains a HASHDATUM somewhere” — the exact sequence, in order. EEOP_INNER_FETCHSOME, EEOP_INNER_VAR, EEOP_HASHDATUM_FIRST and nothing else.

Note what is not matched. There is no pattern for EEOP_INNER_FETCHSOME, EEOP_INNER_VAR, EEOP_HASHDATUM_NEXT32 — the non-strict “next key” case. There is no pattern with two key expressions. There is no pattern for a hash key that is an expression rather than a plain column. Every one of those falls through to the general interpreter.

The last step is always a terminator — EEOP_DONE_RETURN or EEOP_DONE_NO_RETURN, which is asserted at the top of ExecReadyInterpretedExpr(). That is why a four-opcode expression has steps_len == 5 (four real opcodes plus the terminator). The count includes the terminator.

The result is stored in evalfunc_private, not evalfunc. Look at the surrounding code:

 c
 
 
	/*
	 * First time through, check whether attribute matches Var. Might not be
	 * ok anymore, due to schema changes. We do that by setting up a callback
	 * that does checking on the first call, which then sets the evalfunc
	 * callback to the actual method of execution.
	 */
	state->evalfunc = ExecInterpExprStillValid;

The first call goes through a validity check (the schema might have changed since the plan was built), and only then does the real function get installed:

 c
 
 
Datum
ExecInterpExprStillValid(ExprState *state, ExprContext *econtext, bool *isNull)
{
	/*
	 * First time through, check whether attribute matches Var. Might not be
	 * ok anymore, due to schema changes.
	 */
	CheckExprStillValid(state, econtext);

	/* skip the check during further executions */
	state->evalfunc = (ExprStateEvalFunc) state->evalfunc_private;

	/* and actually execute */
	return state->evalfunc(state, econtext, isNull);
}

So evalfunc_private holds the chosen implementation, and evalfunc holds the trampoline that installs it. Elegant, and it means the one-time schema check is not paid on every call.

The specialised functions

Now the payoff. The four-step inner case:

 c
 
 
/* implementation of ExecJustHash(Inner|Outer)Var */
static pg_always_inline Datum
ExecJustHashVarImpl(ExprState *state, TupleTableSlot *slot, bool *isnull)
{
	ExprEvalStep *fetchop = &state->steps[0];
	ExprEvalStep *var = &state->steps[1];
	ExprEvalStep *hashop = &state->steps[2];
	FunctionCallInfo fcinfo = hashop->d.hashdatum.fcinfo_data;
	int			attnum = var->d.var.attnum;

	CheckOpSlotCompatibility(fetchop, slot);
	slot_getsomeattrs(slot, fetchop->d.fetch.last_var);

	fcinfo->args[0].value = slot->tts_values[attnum];
	fcinfo->args[0].isnull = slot->tts_isnull[attnum];

	*isnull = false;

	if (!fcinfo->args[0].isnull)
		return hashop->d.hashdatum.fn_addr(fcinfo);
	else
		return (Datum) 0;
}

That is the entire hash computation, for the common case of “hash one plain column of the inner relation”:

Deform the tuple as far as needed (slot_getsomeattrs).
Copy the attribute value and its NULL flag into the function call frame.
Call the hash function.
If the value was NULL, return 0 (matching EEOP_HASHDATUM_FIRST’s “store 0 on NULL” behaviour).

No dispatch loop. No EEO_NEXT(). No step array traversal. The compiler can see the whole thing.

The outer and inner wrappers are one-liners:

 c
 
 
/* implementation for hashing an outer Var */
static Datum
ExecJustHashOuterVar(ExprState *state, ExprContext *econtext, bool *isnull)
{
	return ExecJustHashVarImpl(state, econtext->ecxt_outertuple, isnull);
}

/* implementation for hashing an inner Var */
static Datum
ExecJustHashInnerVar(ExprState *state, ExprContext *econtext, bool *isnull)
{
	return ExecJustHashVarImpl(state, econtext->ecxt_innertuple, isnull);
}

The strict variant

The strict case is different, and the difference is exactly the bug discussed below. In strict mode a NULL key means “the whole hash is NULL” — the join key does not match anything:

 c
 
 
/*
 * implementation for hashing an outer Var. Returns NULL on NULL input.
 */
static Datum
ExecJustHashOuterVarStrict(ExprState *state, ExprContext *econtext,
						 bool *isnull)
{
	ExprEvalStep *fetchop = &state->steps[0];
	ExprEvalStep *var = &state->steps[1];
	ExprEvalStep *hashop = &state->steps[2];
	FunctionCallInfo fcinfo = hashop->d.hashdatum.fcinfo_data;
	int			attnum = var->d.var.attnum;

	CheckOpSlotCompatibility(fetchop, econtext->ecxt_outertuple);
	slot_getsomeattrs(econtext->ecxt_outertuple, fetchop->d.fetch.last_var);

	fcinfo->args[0].value = econtext->ecxt_outertuple->tts_values[attnum];
	fcinfo->args[0].isnull = econtext->ecxt_outertuple->tts_isnull[attnum];

	if (!fcinfo->args[0].isnull)
	{
		*isnull = false;
		return hashop->d.hashdatum.fn_addr(fcinfo);
	}
	else
	{
		/* return NULL on NULL input */
		*isnull = true;
		return (Datum) 0;
	}
}

The seeded variant

The five-step pattern is for a hash seeded with an initial value — used by parallel hash aggregation, which seeds the hash with a worker-specific IV:

 c
 
 
/*
 * implementation for hashing an inner Var, seeding with an initial value.
 */
static Datum
ExecJustHashInnerVarWithIV(ExprState *state, ExprContext *econtext,
						 bool *isnull)
{
	ExprEvalStep *fetchop = &state->steps[0];
	ExprEvalStep *setivop = &state->steps[1];
	ExprEvalStep *innervar = &state->steps[2];
	ExprEvalStep *hashop = &state->steps[3];
	FunctionCallInfo fcinfo = hashop->d.hashdatum.fcinfo_data;
	int			attnum = innervar->d.var.attnum;
	uint32		hashkey;

	CheckOpSlotCompatibility(fetchop, econtext->ecxt_innertuple);
	slot_getsomeattrs(econtext->ecxt_innertuple, fetchop->d.fetch.last_var);

	fcinfo->args[0].value = econtext->ecxt_innertuple->tts_values[attnum];
	fcinfo->args[0].isnull = econtext->ecxt_innertuple->tts_isnull[attnum];

	hashkey = DatumGetUInt32(setivop->d.hashdatum_initvalue.init_value);
	hashkey = pg_rotate_left32(hashkey, 1);

	if (!fcinfo->args[0].isnull)
	{
		uint32		hashvalue;

		hashvalue = DatumGetUInt32(hashop->d.hashdatum.fn_addr(fcinfo));
		hashkey = hashkey ^ hashvalue;
	}

	*isnull = false;
	return UInt32GetDatum(hashkey);
}

Note the step indices: steps[1] is the SET_INITVAL step and steps[2] is the INNER_VAR step. The initial value is stored in the step’s own payload and is fetched directly, not through the generic *op->resvalue indirection.

The virtual-slot variant

The three-step pattern has no FETCHSOME step at all. That absence is meaningful, and it is the reason the virtual-slot variant can skip slot_getsomeattrs() entirely:

 c
 
 
/* implementation for ExecJustHash(Inner|Outer)VarVirt */
static pg_always_inline Datum
ExecJustHashVarVirtImpl(ExprState *state, TupleTableSlot *slot, bool *isnull)
{
	ExprEvalStep *var = &state->steps[0];
	ExprEvalStep *hashop = &state->steps[1];
	FunctionCallInfo fcinfo = hashop->d.hashdatum.fcinfo_data;
	int			attnum = var->d.var.attnum;

	fcinfo->args[0].value = slot->tts_values[attnum];
	fcinfo->args[0].isnull = slot->tts_isnull[attnum];

	*isnull = false;

	if (!fcinfo->args[0].isnull)
		return hashop->d.hashdatum.fn_addr(fcinfo);
	else
		return (Datum) 0;
}

No slot_getsomeattrs. No CheckOpSlotCompatibility. It reads slot->tts_values[attnum] directly.

Why is that safe? Because a virtual slot has no on-disk tuple to deform. Its values are already materialised in the tts_values[] array. The interpreter knows this, and the code that emits the setup steps knows it too:

 c
 
 
	/* if the slot is known to always virtual we never need to deform */
	if (op->d.fetch.fixed && op->d.fetch.kind == &TTSOpsVirtual)
		return false;

ExecComputeSlotInfo() returns false — meaning “do not push a FETCHSOME step” — when the slot is known to be virtual. That is why the virtual case produces a three-step expression rather than a four-step one, and why the pattern matcher can tell them apart by step count alone.

The ExecJustVarVirtImpl() counterpart says it explicitly:

 c
 
 
	/*
	 * As it is guaranteed that a virtual slot is used, there never is a need
	 * to perform tuple deforming (nor would it be possible). Therefore
	 * execExpr.c has not emitted an EEOP_*_FETCHSOME step. Verify, as much as
	 * possible, that that determination was accurate.
	 */
	Assert(TTS_IS_VIRTUAL(slot));
	Assert(TTS_FIXED(slot));
	Assert(attnum >= 0 && attnum < slot->tts_nvalid);

For a heap tuple slot, slot_getsomeattrs() is not free — it is the tuple deformation machinery, which unpacks varlena headers, follows TOAST pointers, and so on. Skipping it when it is not needed is a real saving.

The payoff

Commit 50416cc4843 (“Speedup Hash Joins with dedicated functions for ExprState hashing”, December 2024) is the one that introduced the ExecJust*Hash* family. Its commit message:

Hashing of a single Var is a very common operation for ExprState to perform. Here we add dedicated ExecJust* functions which helps speed up Hash Joins by removing the interpretation overhead in ExecInterpExpr().

This change currently only affects Hash Joins on a single column. Hash Joins with multiple join keys or an expression still run through ExecInterpExpr().

Some testing has shown up to 10% query performance increases on recent AMD hardware and nearly 7% increase on an Apple M2 for a query performing a hash join with a large number of lookups on a small hash table.

Two things to note. First, the scope: single column only. Second, the shape of the workload that benefits: “a large number of lookups on a small hash table” — i.e. probe-dominated, which is exactly the case where per-probe overhead matters.

Hardcoding the NULL check

The same “specialise the common case” philosophy was applied to strict function calls. Commit d35d32d7112 (“Add special case fast-paths for strict functions”, March 2025):

Many STRICT function calls will have one or two arguments, in which case we can speed up checking for NULL input by avoiding setting up a loop over the arguments. This adds EEOP_FUNCEXPR_STRICT_1 and the corresponding EEOP_FUNCEXPR_STRICT_2 for functions with one and two arguments respectively.

The generic strict function step loops:

 c
 
 
		/* strict function call with more than two arguments */
		EEO_CASE(EEOP_FUNCEXPR_STRICT)
		{
			FunctionCallInfo fcinfo = op->d.func.fcinfo_data;
			NullableDatum *args = fcinfo->args;
			int			nargs = op->d.func.nargs;
			Datum		d;

			Assert(nargs > 2);

			/* strict function, so check for NULL args */
			for (int argno = 0; argno < nargs; argno++)
			{
				if (args[argno].isnull)
				{
					*op->resnull = true;
					goto strictfail;
				}
			}
			fcinfo->isnull = false;
			d = op->d.func.fn_addr(fcinfo);
			*op->resvalue = d;
			*op->resnull = fcinfo->isnull;

	strictfail:
			EEO_NEXT();
		}

A loop, with a goto, and a nargs that is not known at compile time. The one-argument version:

 c
 
 
		/* strict function call with one argument */
		EEO_CASE(EEOP_FUNCEXPR_STRICT_1)
		{
			FunctionCallInfo fcinfo = op->d.func.fcinfo_data;
			NullableDatum *args = fcinfo->args;

			Assert(op->d.func.nargs == 1);

			/* strict function, so check for NULL args */
			if (args[0].isnull)
				*op->resnull = true;
			else
			{
				Datum		d;

				fcinfo->isnull = false;
				d = op->d.func.fn_addr(fcinfo);
				*op->resvalue = d;
				*op->resnull = fcinfo->isnull;
			}

			EEO_NEXT();
		}

and the two-argument version:

 c
 
 
		/* strict function call with two arguments */
		EEO_CASE(EEOP_FUNCEXPR_STRICT_2)
		{
			FunctionCallInfo fcinfo = op->d.func.fcinfo_data;
			NullableDatum *args = fcinfo->args;

			Assert(op->d.func.nargs == 2);

			/* strict function, so check for NULL args */
			if (args[0].isnull || args[1].isnull)
				*op->resnull = true;
			else
			{
				Datum		d;

				fcinfo->isnull = false;
				d = op->d.func.fn_addr(fcinfo);
				*op->resvalue = d;
				*op->resnull = fcinfo->isnull;
			}

			EEO_NEXT();
		}

The same trick appears for aggregate transition functions’ strict input checks:

 c
 
 
		/* special case for just one argument */
		EEO_CASE(EEOP_AGG_STRICT_INPUT_CHECK_ARGS_1)
		{
			NullableDatum *args = op->d.agg_strict_input_check.args;
			PG_USED_FOR_ASSERTS_ONLY int nargs = op->d.agg_strict_input_check.nargs;

			Assert(nargs == 1);

			if (args[0].isnull)
				EEO_JUMP(op->d.agg_strict_input_check.jumpnull);
			EEO_NEXT();
		}

versus the general case, which loops over the arguments.

And the choice of opcode happens once, at expression-build time:

 c
 
 
	/* Insert appropriate opcode depending on strictness and stats level */
	if (pgstat_track_functions <= flinfo->fn_stats)
	{
		if (flinfo->fn_strict && nargs > 0)
		{
			/* Choose nargs optimized implementation if available. */
			if (nargs == 1)
				scratch->opcode = EEOP_FUNCEXPR_STRICT_1;
			else if (nargs == 2)
				scratch->opcode = EEOP_FUNCEXPR_STRICT_2;
			else
				scratch->opcode = EEOP_FUNCEXPR_STRICT;
		}
		else
			scratch->opcode = EEOP_FUNCEXPR;
	}

Note the pgstat_track_functions condition: if per-function statistics collection is enabled, the specialised opcodes are not used, because they bypass the FUSAGE variants that do the counting.

The NULL-key bug

Specialisation is where bugs live. Commit 60826a352d4 (“Fix Hash Join performance issue when hashing NULL values”, July 2026) fixed a real one. Its message is a model of a good bug report:

adf97c156 allowed expression evaluation to perform hashing, and subsequently 9ca67658d fixed a memory stomping bug in that commit that caused unrelated-to-hashing expression op steps to stomp on the intermediate hash value. The intermediate hash value needs to be maintained when hashing multiple hash keys. 9ca67658d didn’t quite get things right when in “strict” mode when it aborted hashing early after encountering a NULL hash key. What was meant to happen was that the expression returns NULL directly to indicate to the caller the value hashed to NULL. The problem was that any EEOP_HASHDATUM_FIRST_STRICT or EEOP_HASHDATUM_NEXT32_STRICT op step that didn’t belong to the final key to be hashed would have its op->resnull and op->resvalue pointing to the location to store the intermediate hash value. That’s correct for non-NULLs since we bit-rotate the intermediate value and continue hashing, but with the strict case, when we get a NULL key, we immediately jump to the “jumpdone” step. The problem is the jumpdone step expects the ExprState resnull and resvalue fields to be set (as they would be if we didn’t abort hashing early due to the NULL), but when we aborted early, the ExprState fields never got set. This would result in inserting records into the hash table that would never match to any join partner, which is a waste of CPU and memory.

Let me unpack that, because it is subtle and it is a nice illustration of why specialisation is dangerous.

There are two “output slots” in play:

op->resvalue / op->resnull — where this step writes its result. For a key that is not the last one, this points at the intermediate hash slot (iresult), because the running hash must be preserved for the next key.
state->resvalue / state->resnull — the final result of the whole expression.

For a multi-key hash, the first key’s HASHDATUM_FIRST_STRICT step writes to the intermediate slot, and only the last key’s step writes to state->resvalue.

Now, in strict mode, when a NULL key is found, the interpreter aborts immediately:

 c
 
 
		EEO_CASE(EEOP_HASHDATUM_FIRST_STRICT)
		{
			FunctionCallInfo fcinfo = op->d.hashdatum.fcinfo_data;

			if (fcinfo->args[0].isnull)
			{
				/*
				 * With strict we have the expression return NULL instead of
				 * ignoring NULL input values. We've nothing more to do after
				 * finding a NULL.
				 */
				state->resnull = true;
				state->resvalue = (Datum) 0;
				EEO_JUMP(op->d.hashdatum.jumpdone);
			}

(note: this is the fixed version; the bug was *op->resnull = true; *op->resvalue = (Datum) 0;).

The bug: when the NULL was found in the first key of a multi-key hash, the step wrote to *op->resnull / *op->resvalue — the intermediate slot — and then jumped to the “done” step. The “done” step reads state->resnull / state->resvalue. Those were never written, so they held whatever was left over from the previous evaluation.

Consequences: the caller saw a non-NULL hash value for a tuple whose join key was NULL. That tuple got inserted into the hash table, where it could never match anything — pure waste of memory and CPU, in the hash-join build phase.

The fix, in both _STRICT opcodes, is to write to the expression’s result fields directly:

 c
 
 
				state->resnull = true;
				state->resvalue = (Datum) 0;

instead of *op->resnull / *op->resvalue. One character of difference — op-> versus state-> — with a real performance consequence.

The commit also notes the blast radius:

Although Hash Agg and Hashed Subplans do use hashing from ExprStates, those were unaffected by this bug, as neither of those uses the STRICT op steps.

Which is precisely why it survived: hash aggregate and hashed subplans build their hash expressions with init_value == 0 and non-strict operators, so they never take the _STRICT path.

The reporter was a user: “Reported-by: Dan Stefura”.

The hash table side

Specialising the hash computation is only half of it. The probe side walks a bucket chain, and for each candidate it has to re-check the join qual. From ExecScanHashBucket() in nodeHash.c:

 c
 
 
	while (hashTuple != NULL)
	{
		if (hashTuple->hashvalue == hashvalue)
		{
			TupleTableSlot *inntuple;

			/* insert hashtable's tuple into exec slot so ExecQual sees it */
			inntuple = ExecStoreMinimalTuple(HJTUPLE_MINTUPLE(hashTuple),
											 hjstate->hj_HashTupleSlot,
											 false);	/* do not pfree */
			econtext->ecxt_innertuple = inntuple;

			if (ExecQualAndReset(hjclauses, econtext))
			{
				hjstate->hj_CurTuple = hashTuple;
				return true;
			}
		}

		hashTuple = hashTuple->next.unshared;
	}

Two things happen per candidate: a cheap 32-bit hash comparison, and only on a match, the full join qual (hjclauses) is evaluated. The 32-bit comparison is what makes the bucket scan cheap — a hash collision is expected roughly once in 2^32 rows, so the qual almost never runs.

ExecQualAndReset() resets the expression context after each evaluation, so the qual’s temporary memory does not accumulate across candidates.

Memoize: a hash join that remembers

The Memoize node (src/backend/executor/nodeMemoize.c) is a materialised cache in front of a subplan, and it is the other place where hash-join-style machinery shows up: a hash table keyed by the subplan’s parameters, with an LRU eviction policy.

The probe slot

The design is unusual: the hash and equality functions do not use the stored key at all. They read the probe slot instead, which is populated before every lookup:

 c
 
 
/*
 * MemoizeHash_hash
 *		Hash function for simplehash hashtable. 'key' is unused here as we
 *		require that all table lookups first populate the MemoizeState's
 *		probeslot with the key values to be looked up.
 */
static uint32
MemoizeHash_hash(struct memoize_hash *tb, const MemoizeKey *key)

The population function is prepare_probe_slot():

 c
 
 
static inline void
prepare_probe_slot(MemoizeState *mstate, MemoizeKey *key)
{
	TupleTableSlot *pslot = mstate->probeslot;
	TupleTableSlot *tslot = mstate->tableslot;
	int			numKeys = mstate->nkeys;

	ExecClearTuple(pslot);

	if (key == NULL)
	{
		ExprContext *econtext = mstate->ss.ps.ps_ExprContext;
		MemoryContext oldcontext;

		oldcontext = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);

		/* Set the probeslot's values based on the current parameter values */
		for (int i = 0; i < numKeys; i++)
			pslot->tts_values[i] = ExecEvalExpr(mstate->param_exprs[i],
												econtext,
												&pslot->tts_isnull[i]);

		MemoryContextSwitchTo(oldcontext);
	}
	else
	{
		/* Process the key's MinimalTuple and store the values in probeslot */
		ExecStoreMinimalTuple(key->params, tslot, false);
		slot_getallattrs(tslot);
		memcpy(pslot->tts_values, tslot->tts_values, sizeof(Datum) * numKeys);
		memcpy(pslot->tts_isnull, tslot->tts_isnull, sizeof(bool) * numKeys);
	}

	ExecStoreVirtualTuple(pslot);
}

Note the two paths. When looking up the current parameter values (key == NULL), the parameter expressions are evaluated directly into the probe slot’s arrays — no tuple copying at all. When re-hashing an entry that is being evicted, the stored key tuple is re-materialised into the probe slot so that the same hash function can be used.

And the probe slot is turned into a virtual tuple at the end:

 c
 
	ExecStoreVirtualTuple(pslot);

which is the same trick the hash join’s virtual-slot fast path relies on: no deformation needed, values already materialised.

The LRU eviction

The cache is bounded by mem_limit, and when it is exceeded, entries are evicted from the least-recently-used end of a doubly-linked list:

 c
 
 
/*
 * cache_reduce_memory
 *		Evict older and less recently used items from the cache in order to
 *		reduce the memory consumption back to something below the
 *		MemoizeState's mem_limit.
 */
static bool
cache_reduce_memory(MemoizeState *mstate, MemoizeKey *specialkey)
{
	...
	/* Start the eviction process starting at the head of the LRU list. */
	dlist_foreach_modify(iter, &mstate->lru_list)
	{
		MemoizeKey *key = dlist_container(MemoizeKey, lru_node, iter.cur);
		MemoizeEntry *entry;

		/*
		 * Populate the hash probe slot in preparation for looking up this LRU
		 * entry.
		 */
		prepare_probe_slot(mstate, key);

		/*
		 * Ideally the LRU list pointers would be stored in the entry itself
		 * rather than in the key. Unfortunately, we can't do that as the
		 * simplehash.h code may resize the table and allocate new memory for
		 * entries which would result in those pointers pointing to the old
		 * buckets. However, it's fine to use the key to store this as that's
		 * only referenced by a pointer in the entry, which of course follows
		 * the entry whenever the hash table is resized. Since we only have a
		 * pointer to the key here, we must perform a hash table lookup to
		 * find the entry that the key belongs to.
		 */
		entry = memoize_lookup(mstate->hashtable, NULL);
		...
		remove_cache_entry(mstate, entry);

		evictions++;

		/* Exit if we've freed enough memory */
		if (mstate->mem_used <= mstate->mem_limit)
			break;
	}

Note the comment about why the LRU list pointer lives in the key rather than the entry: simplehash.h may resize the table and reallocate its buckets, invalidating pointers stored inside the entries. The key is referenced by pointer from the entry, so it moves with the entry.

The eviction loop has to do a full hash-table lookup per evicted entry (just to find the entry that belongs to the key it is evicting) — the price of that design decision.

16 bytes per tuple

Commit 609f969f61b (“Reduce memory overheads for storing a Memoize tuple”, August 2026) is a nice piece of allocator-level thinking. Each cached tuple used to be a separately allocated structure plus a MinimalTuple. Now the “next tuple” pointer is stored in 8 extra bytes allocated with the tuple:

 c
 
 
/*
 * The number of extra bytes we request from ExecCopySlotMinimalTupleExtra to
 * allow storage of the pointer to the next cached tuple for a MemoizeEntry.
 */
#define MEMOIZE_NEXT_TUPLE_EXTRA_BYTES MAXALIGN(sizeof(MinimalTuple))

MAXALIGN(sizeof(MinimalTuple)) is 8 bytes on a 64-bit machine. And the pointer lives before the tuple:

 c
 
 
/*
 * Calculate the address of the "next" pointer from the MinimalTuple
 */
#define MemoizeNextTupleAddress(t) \
	((MinimalTuple *) ((char *) (t) - MEMOIZE_NEXT_TUPLE_EXTRA_BYTES))

ExecCopySlotMinimalTupleExtra() allocates the tuple with that much extra space, already zeroed:

 c
 
 
/*
 * ExecCopySlotMinimalTupleExtra - return MinimalTuple allocated in caller's
 * context, with extra bytes (maxaligned and zeroed) before the tuple for data
 * the caller wishes to store along with the tuple (without requiring the
 * caller to make an additional allocation).
 */
static inline MinimalTuple
ExecCopySlotMinimalTupleExtra(TupleTableSlot *slot, Size extra)
{
	return slot->tts_ops->copy_minimal_tuple(slot, extra);
}

And the cache store path:

 c
 
 
	/*
	 * Form a MinimalTuple with extra space to store a "next" pointer so that
	 * we can form a singly linked list of tuples belonging to this
	 * MemoizeEntry.
	 */
	mintuple = ExecCopySlotMinimalTupleExtra(slot,
											 MEMOIZE_NEXT_TUPLE_EXTRA_BYTES);

	/*
	 * No need to use MemoizeSetNextTuple to point the next tuple to NULL as
	 * ExecCopySlotMinimalTupleExtra zeros the extra bytes.
	 */

The commit message quantifies the win:

This uses 16 fewer bytes per tuple stored in each MemoizeEntry. 24 bytes can be saved by not doing palloc_object(MemoizeTuple), and instead we can store the pointer to the next tuple by requesting that ExecCopySlotMinimalTupleExtra() allocate an extra 8 bytes per MinimalTuple, a net saving of 16 bytes per cached tuple, which equates to 33% less memory in the most extreme case (assuming very few entries with many tuples and the narrowest possible MinimalTuple).

Saving memory in Memoize allows more entries to exist concurrently and reduces the chances that an entry will be evicted from the cache due to memory pressure. That increases the chances of getting cache hits.

The accounting macro follows:

 c
 
 
/* Helper macros for memory accounting */
#define EMPTY_ENTRY_MEMORY_BYTES(e)		(sizeof(MemoizeEntry) + \
										 sizeof(MemoizeKey) + \
										 (e)->key->params->t_len)
#define CACHE_TUPLE_BYTES(t)			((t)->t_len + \
										 MEMOIZE_NEXT_TUPLE_EXTRA_BYTES)

This is a micro-optimisation with a macro-level payoff: 16 bytes out of a typical tuple’s ~48 bytes of overhead is a third of the per-tuple memory, and memory per cached tuple is the cache’s capacity.

The JIT has its own copy

One more wrinkle. PostgreSQL’s JIT compiler (src/backend/jit/llvm/) compiles the same step list, but it is a completely separate implementation with its own switch over the same opcodes. So every change to the opcode set has to be made twice.

The strict-function specialisation had to be mirrored:

 c
 
 
			case EEOP_FUNCEXPR:
			case EEOP_FUNCEXPR_STRICT:
			case EEOP_FUNCEXPR_STRICT_1:
			case EEOP_FUNCEXPR_STRICT_2:
				{
					FunctionCallInfo fcinfo = op->d.func.fcinfo_data;
					LLVMValueRef v_fcinfo_isnull;
					LLVMValueRef v_retval;

					if (opcode == EEOP_FUNCEXPR_STRICT ||
						opcode == EEOP_FUNCEXPR_STRICT_1 ||
						opcode == EEOP_FUNCEXPR_STRICT_2)
					{

and the hashing opcodes likewise:

 c
 
 
			case EEOP_HASHDATUM_SET_INITVAL:
				{
					LLVMValueRef v_initvalue;

					v_initvalue = l_datum_const(op->d.hashdatum_initvalue.init_value);

					LLVMBuildStore(b, v_initvalue, v_resvaluep);
					LLVMBuildStore(b, l_sbool_const(0), v_resnullp);
					LLVMBuildBr(b, opblocks[opno + 1]);
					break;
				}

			case EEOP_HASHDATUM_FIRST:
			case EEOP_HASHDATUM_FIRST_STRICT:
			case EEOP_HASHDATUM_NEXT32:
			case EEOP_HASHDATUM_NEXT32_STRICT:

Note that the JIT has to be careful about when it rotates the running hash, and it differs deliberately between strict and non-strict mode:

 c
 
 
					/*
					 * When performing the next hash and not in strict mode we
					 * perform a rotation of the previously stored hash value
					 * before doing the NULL check. We want to do this even
					 * when we receive a NULL Datum to hash. In strict mode,
					 * we do this after the NULL check so as not to waste the
					 * effort of rotating the bits when we're going to throw
					 * away the hash value and return NULL.
					 */

This is the same subtlety that produced the NULL-key bug, and the JIT had to be written with the same care.

The JIT also changes which fast path applies. ExecReadyExpr() is the single entry point:

 c
 
 
static void
ExecReadyExpr(ExprState *state)
{
	if (jit_compile_expr(state))
		return;

	ExecReadyInterpretedExpr(state);
}

If the JIT compiled the expression, ExecReadyInterpretedExpr() is never called, so the ExecJust* fast paths are not installed at all when JIT is active. The JIT’s own code is the fast path in that case — which is why the adf97c15623 commit measured a 26% improvement with JIT enabled versus 20% without.

Caveats

Only very specific opcode shapes are specialised. A hash join on two columns, or on an expression like a + 1, or on a column with a collation-sensitive hash function, falls through to the general interpreter. The commit message is explicit: “This change currently only affects Hash Joins on a single column. Hash Joins with multiple join keys or an expression still run through ExecInterpExpr().”

The strict fast paths only cover 1 and 2 arguments. EEOP_FUNCEXPR_STRICT still loops. And the specialised opcodes are skipped entirely when pgstat_track_functions is enabled, because they bypass the statistics-collecting variants.

The JIT and the interpreter are separate implementations of the same specification. Every opcode change must be made twice, and the two can disagree — as they nearly did around the NULL-key bug.

Specialisation is where bugs live. The op->resnull versus state->resnull distinction is invisible at the level of “return NULL if the key is NULL”, but the difference between writing the intermediate slot and the final slot is the difference between a correct result and a hash table full of rows that can never match.

pfree is not the only thing that got faster. The hash join’s build side also got faster, because ExecBuildHash32Expr() deforms the tuple once to the highest needed attribute rather than once per key.

What it all means in practice

The pattern is “specialise the shape you actually have”. PostgreSQL’s expression evaluation is a general interpreter with a general dispatcher, and it is used for everything from qual evaluation to aggregate transitions to hash keys. Rather than making the interpreter faster in general, the strategy is to notice the handful of shapes that dominate real workloads and hand-write those.

Matching on step count first is a cheap filter that pays for itself. steps_len == 5 narrows the space before any opcode is examined, and the check happens exactly once per expression.

The specialised functions are not “clever”. They are straight-line code that a compiler can see through — no dispatch, no goto, no loop over nargs. The win comes from removing the interpreter’s per-step overhead, not from doing anything clever.

The virtual-slot trick is about knowing what you don’t have to do. A virtual slot has no tuple to deform, so the deformation step is not emitted, so the fast path does not perform it. Absence of a step is the signal.

Memory per cached entry is cache capacity. Saving 16 bytes per Memoize tuple is not a micro-optimisation; in a cache bounded by work_mem, it directly translates into more entries surviving, fewer evictions, and more cache hits.

A 10% win on a probe-dominated hash join is worth a lot. The commit measured “up to 10% query performance increases on recent AMD hardware” for a workload with many probes against a small hash table. That is the shape of workload where per-probe overhead is the whole cost.

How PostgreSQL Waits for Locks

Jay Hubbert — Tue, 06 Oct 2026 09:00:00 +0000

Every lock in PostgreSQL is a bet. When a backend wants a resource that somebody else has, it has to choose: burn CPU waiting, or go to sleep and let the kernel schedule somebody else. Both are wrong sometimes. Spinning wastes a core and burns power; sleeping costs a context switch, wakes up in the wrong place, and — worst of all — adds the victim’s release time to the waiter’s latency.

PostgreSQL’s answer is a ladder. It spins in user space with an architecture-specific pause instruction, then sleeps for a randomized, exponentially growing interval, and only then blocks in the kernel on a POSIX semaphore. Every rung of that ladder is a separate piece of code with a separate set of trade-offs, and the interesting part of the design is where the rungs are.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

The ladder, in one table

Layer	Object	Waits by	Kernel involved?
1	Spinlock (`slock_t`)	`lock; xchgb` in a loop, then `pg_usleep()`	Only after ~1000 sleeps
2	Atomic (`pg_atomic_uint32`)	Compare-and-swap loop	No
3	LWLock	Spin, then block on a POSIX semaphore	Yes
4	Latch	Sleep in `epoll_wait()`/`poll()`	Yes
5	Heavyweight lock	Block on the lock manager’s semaphore	Yes

The ladder is not uniform, and that is the point. A spinlock protects something so small that the only correct thing to do is spin. An LWLock protects something that might take milliseconds. A latch is not a lock at all — it is a one-way door that a backend waits on until somebody opens it.

Layer 1: the spinlock

The hardware primitives

A spinlock is one byte. The contract is spelled out in src/include/storage/s_lock.h:

int TAS(slock_t *lock) Atomic test-and-set instruction. Attempt to acquire the lock, but do not wait. Returns 0 if successful, nonzero if unable to acquire the lock.

int TAS_SPIN(slock_t *lock) Like TAS(), but this version is used when waiting for a lock previously found to be contended. By default, this is the same as TAS(), but on some architectures it’s better to poll a contended lock using an unlocked instruction and retry the atomic test-and-set only when it appears free.

On x86_64 the whole thing is one instruction pair:

 c
 
 
// src/include/storage/s_lock.h
static inline int
tas(volatile slock_t *lock)
{
	slock_t		_res = 1;

	__asm__ __volatile__(
	"	lock			\n"
	"	xchgb	%0,%1	\n"
:		"+q"(_res), "+m"(*lock)
:		/* no inputs */
:		"memory", "cc");
	return (int) _res;
}

lock xchg is a read-modify-write with the bus held, so it is atomic with respect to every other core. The old value comes back in %0, and the return value tells the caller whether it got the lock (old value zero) or found it taken.

The free test

The interesting part is TAS_SPIN, and on x86_64 it is a one-liner:

 c
 
 
// src/include/storage/s_lock.h
/*
 * On Intel EM64T, it's a win to use a non-locking test before the xchg proper,
 * but only when spinning.
 *
 * See also Implementing Scalable Atomic Locks for Multi-Core Intel(tm) EM64T
 * and IA32, by Michael Chynoweth and Mary R. Lee. As of this writing, it is
 * available at:
 * http://software.intel.com/en-us/articles/implementing-scalable-atomic-locks-for-multi-core-intel-em64t-and-ia32-architectures
 */
#define TAS_SPIN(lock) (*(lock) ? 1 : TAS(lock))

Read it as: if the lock byte is non-zero, it is taken, so return “failed” without touching the bus. Only if it looks free do we pay for the lock xchgb.

The reason this only happens while spinning is subtle. A lock-prefixed instruction on x86 is a full barrier: it drains the store buffer and serializes against other cores’ atomic operations. If you issue one for every failed attempt, you are asking the memory subsystem to arbitrate between every spinner on the machine — which is exactly the contention you are trying to avoid. A plain load is free (it can be satisfied from the local cache) and lets the spinner’s core keep running at full speed.

The “only when spinning” part is about the first acquisition. On the first attempt, the lock is very likely free, and the cost of the lock xchgb is paid anyway; doing a speculative plain load first would just add a branch and a potential cache miss on the fast path. And there is a correctness-adjacent reason too: the free test is a prediction. Between the load and the xchgb, another core could take the lock. That is fine — the xchgb is still the arbiter, and it will report the failure — but it means the free test can produce false “free” answers, and only the atomic instruction may decide.

The 32-bit x86 version takes the opposite approach and does the free test inside tas() itself:

 c
 
 
// src/include/storage/s_lock.h
static inline int
tas(volatile slock_t *lock)
{
	slock_t		_res = 1;

	/*
	 * Use a non-locking test before asserting the bus lock. Note that the
	 * extra test appears to be a small loss on some x86 platforms and a small
	 * win on others; it's by no means clear that we should keep it.
	 *
	 * When this was last tested, we didn't have separate TAS() and TAS_SPIN()
	 * macros. Nowadays it probably would be better to do a non-locking test
	 * in TAS_SPIN() but not in TAS(), like on x86_64, but no-one's done the
	 * testing to verify that. Without some empirical evidence, better to
	 * leave it alone.
	 */
	__asm__ __volatile__(
	"	cmpb	$0,%1		\n"
	"	jne		TAS%=_out	\n"
	"	lock				\n"
	"	xchgb	%0,%1		\n"
	"TAS%=_out: \n"
:		"+q"(_res), "+m"(*lock)
:		/* no inputs */
:		"memory", "cc");
	return (int) _res;
}

Note the comment’s honesty: “it’s by no means clear that we should keep it”. This is the kind of micro-optimization that is kept because it was measured once, long ago, and nobody has had the heart to re-measure it.

The pause instruction

Between attempts, the spinner has to do something. Doing nothing is bad: modern x86 cores issue speculatively and out of order, so a tight loop that keeps polling a memory location hammers the memory subsystem and the core’s pipeline. The answer is a hint instruction that tells the CPU “you are in a spin-wait loop”.

On x86 that is PAUSE, and PostgreSQL spells it the classic way:

 c
 
 
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * Adding a PAUSE in the spin delay loop is demonstrably a no-op on
	 * Opteron, but it may be of some use on EM64T, so we keep it.
	 */
	__asm__ __volatile__(
	" rep; nop			\n");
}

The 32-bit x86 version carries the full Intel explanation:

 c
 
 
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * This sequence is equivalent to the PAUSE instruction ("rep" is
	 * ignored by old IA32 processors if the following instruction is
	 * not a string operation); the IA-32 Architecture Software
	 * Developer's Manual, Vol. 3, Section 7.7.2 describes why using
	 * PAUSE in the inner loop of a spin lock is necessary for good
	 * performance:
	 *
	 * The PAUSE instruction improves the performance of IA-32
	 * processors supporting Hyper-Threading Technology when
	 * executing spin-wait loops and other routines where one
	 * thread is accessing a shared lock or semaphore in a tight
	 * polling loop. When executing a spin-wait loop, the
	 * processor can suffer a severe performance penalty when
	 * exiting the loop because it detects a possible memory order
	 * violation and flushes the core processor's pipeline. The
	 * PAUSE instruction provides a hint to the processor that the
	 * code sequence is a spin-wait loop. The processor uses this
	 * hint to avoid the memory order violation and prevent the
	 * pipeline flush. In addition, the PAUSE instruction
	 * de-pipelines the spin-wait loop to prevent it from
	 * consuming execution resources excessively.
	 */
	__asm__ __volatile__(
	" rep; nop			\n");
}

“De-pipelines the spin-wait loop to prevent it from consuming execution resources excessively” is the key phrase: PAUSE exists so that a spinning thread does not steal issue slots from the thread that is about to release the lock. On a hyper-threaded machine that matters enormously, because the spinner and the holder may share a physical core.

On ARM64 the answer is ISB:

 c
 
 
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * Using an ISB instruction to delay in spinlock loops appears beneficial
	 * on high-core-count ARM64 processors. It seems mostly a wash for smaller
	 * gear, and ISB doesn't exist at all on pre-v7 ARM chips.
	 */
	__asm__ __volatile__(
	" isb;				\n");
}

ISB flushes the instruction pipeline, which has the same effect as PAUSE: it stops the core from speculatively executing past the spin loop.

PowerPC is interesting because it does not define SPIN_DELAY() at all, so the generic fallback in s_lock.h applies:

 c
 
 
// src/include/storage/s_lock.h
#if !defined(SPIN_DELAY)
#define SPIN_DELAY()	((void) 0)
#endif		/* SPIN_DELAY */

Instead, PowerPC gets its memory ordering from the lwarx/stwcx. compare-and-swap loop, and uses lwsync as its barrier:

 c
 
 
// src/include/storage/s_lock.h
/*
 * NOTE: per the Enhanced PowerPC Architecture manual, v1.0 dated 7-May-2002,
 * an isync is a sufficient synchronization barrier after a lwarx/stwcx loop.
 * But if the spinlock is in ordinary memory, we can use lwsync instead for
 * better performance.
 */
static inline int
tas(volatile slock_t *lock)
{
	slock_t _t;
	int _res;

	__asm__ __volatile__(
"	lwarx %0,0,%3,1	\n"
"	cmpwi %0,0		\n"
"	bne TAS%=_fail	\n"
"	addi %0,%0,1		\n"
"	stwcx. %0,0,%3		\n"
"	beq TAS%=_ok	\n"
"TAS%=_fail: \n"
"	li %1,1		\n"
"	b TAS%=_out	\n"
"TAS%=_ok: \n"
"	lwsync				\n"
"	li %1,0		\n"
"TAS%=_out: \n"
:	"=&b"(_t), "=r"(_res), "+m"(*lock)
:	"r"(lock)
:	"memory", "cc");
	return _res;
}

/*
 * PowerPC S_UNLOCK is almost standard but requires a "sync" instruction.
 * But we can use lwsync instead for better performance.
 */
#define S_UNLOCK(lock)	\
do \
{ \
	__asm__ __volatile__ ("	lwsync \n" ::: "memory"); \
	*((volatile slock_t *) (lock)) = 0; \
} while (0)

lwarx is “load with reservation” and stwcx. is “store with conditional”, which together implement a compare-and-swap on PowerPC’s older architectures. The lwsync after a successful acquisition is the acquire barrier; the lwsync before the store in S_UNLOCK is the release barrier.

The memory ordering contract

The header is unusually explicit about what the compiler and the hardware are allowed to do:

It is the responsibility of these macros to make sure that the compiler does not re-order accesses to shared memory to precede the actual lock acquisition, or follow the lock release.

On platforms with weak memory ordering, the TAS(), TAS_SPIN(), and S_UNLOCK() macros must further include hardware-level memory fence instructions to prevent similar re-ordering at the hardware level. TAS() and TAS_SPIN() must guarantee that loads and stores issued after the macro are not executed until the lock has been obtained. Conversely, S_UNLOCK() must guarantee that loads and stores issued before the macro have been executed before the lock is released.

Two different barriers are needed, and the difference is why S_UNLOCK() is not just *(lock) = 0 everywhere:

 c
 
 
// src/include/storage/s_lock.h
/*
 * Default implementation of S_UNLOCK() for gcc/icc.
 *
 * Note that this implementation is unsafe for any platform that can reorder
 * a memory access (either load or store) after a following store. That
 * happens not to be possible on x86 and most legacy architectures (some are
 * single-processor!), but many modern systems have weaker memory ordering.
 * Those that do must define their own version of S_UNLOCK() rather than
 * relying on this one.
 */
#if !defined(S_UNLOCK)
#define S_UNLOCK(lock)	\
	do { __asm__ __volatile__("" : : : "memory"); *(lock) = 0; } while (0)
#endif		/* !defined(S_UNLOCK) */

The default is a compiler barrier only. On x86 that is enough, because the hardware is total-store-ordered: stores cannot be reordered with other stores, and the lock prefix on the acquisition side prevents the hardware from moving anything past it. On ARM or PowerPC you need an explicit instruction, which is why those architectures define their own S_UNLOCK.

The generic spin loop

Everything above is platform-specific. The portable part is in src/backend/storage/lmgr/s_lock.c:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
#ifdef USE_DEFAULT_S_LOCK
/*
 * s_lock(lock) - platform-independent portion of waiting for a spinlock.
 */
int
s_lock(volatile slock_t *lock, const char *file, int line, const char *func)
{
	SpinDelayStatus delayStatus;

	init_spin_delay(&delayStatus, file, line, func);

	while (TAS_SPIN(lock))
	{
		perform_spin_delay(&delayStatus);
	}

	finish_spin_delay(&delayStatus);

	return delayStatus.delays;
}
#endif

The fast path is one TAS_SPIN that succeeds. The slow path calls perform_spin_delay() in a loop until the lock is acquired, and the number of delays is returned so that the caller can report it (this is how a “stuck spinlock” panic gets a sensible message).

perform_spin_delay()

This is the function that actually implements the ladder:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * Wait while spinning on a contended spinlock.
 */
void
perform_spin_delay(SpinDelayStatus *status)
{
	/* CPU-specific delay each time through the loop */
	SPIN_DELAY();

	/* Block the process every spins_per_delay tries */
	if (++(status->spins) >= spins_per_delay)
	{
		if (++(status->delays) > NUM_DELAYS)
			s_lock_stuck(status->file, status->line, status->func);

		if (status->cur_delay == 0) /* first time to delay? */
			status->cur_delay = MIN_DELAY_USEC;

		/*
		 * Once we start sleeping, the overhead of reporting a wait event is
		 * justified. Actively spinning easily stands out in profilers, but
		 * sleeping with an exponential backoff is harder to spot...
		 *
		 * We might want to report something more granular at some point, but
		 * this is better than nothing.
		 */
		pgstat_report_wait_start(WAIT_EVENT_SPIN_DELAY);
		pg_usleep(status->cur_delay);
		pgstat_report_wait_end();

#if defined(S_LOCK_TEST)
		fprintf(stdout, "*");
		fflush(stdout);
#endif

		/* increase delay by a random fraction between 1X and 2X */
		status->cur_delay += (int) (status->cur_delay *
									pg_prng_double(&pg_global_prng_state) + 0.5);
		/* wrap back to minimum delay when max is exceeded */
		if (status->cur_delay > MAX_DELAY_USEC)
			status->cur_delay = MIN_DELAY_USEC;

		status->spins = 0;
	}
}

Three phases per iteration:

SPIN_DELAY() — the architecture-specific pause. Executed on every pass through the loop, not just every spins_per_delay tries.
Every spins_per_delay attempts, sleep for cur_delay microseconds via pg_usleep().
Multiply the delay by a random factor between 1× and 2×.

The constants:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
#define MIN_SPINS_PER_DELAY 10
#define MAX_SPINS_PER_DELAY 1000
#define NUM_DELAYS			1000
#define MIN_DELAY_USEC		1000L
#define MAX_DELAY_USEC		1000000L

So: spin up to spins_per_delay times (default 100), then sleep 1 ms, then sleep between 1 and 2 ms, then between 2 and 4 ms, and so on, capped at 1 second — and after 1000 delays, declare the spinlock stuck and PANIC.

Note the granularity of the backoff: cur_delay += (int) (cur_delay * pg_prng_double(...) + 0.5) multiplies by a random number in [1.0, 2.0), rounded. That is a jittered exponential backoff, and the jitter is the whole point.

Why the randomness matters

The file header explains the starvation problem:

Once we do decide to block, we use randomly increasing pg_usleep() delays. The first delay is 1 msec, then the delay randomly increases to about one second, after which we reset to 1 msec and start again. The idea here is that in the presence of heavy contention we need to increase the delay, else the spinlock holder may never get to run and release the lock. (Consider situation where spinlock holder has been nice’d down in priority by the scheduler — it will not get scheduled until all would-be acquirers are sleeping, so if we always use a 1-msec sleep, there is a real possibility of starvation.) But we can’t just clamp the delay to an upper bound, else it would take a long time to make a reasonable number of tries.

Two separate effects are at work:

Un-jittered backoff would synchronise the spinners. If every waiter sleeps for exactly 1 ms, they all wake up at exactly the same time, all collide, and all go back to sleep for exactly 1 ms. The lock holder gets one collision per millisecond instead of a smooth trickle of retries. Worse, the collisions mean the lock’s cache line ping-pongs between cores.
Un-jittered backoff starves the holder. On a machine with more runnable threads than cores, a spinner that wakes up every millisecond is runnable every millisecond. The holder, which needs the CPU to release the lock, gets scheduled less often the more spinners there are. The jitter breaks that pattern: spinners wake at uncorrelated times, so the holder gets a chance to run between them.

The pg_prng_double() call uses the backend’s global PRNG state, so the jitter differs between backends — two spinners on the same lock will not pick the same delay.

The adaptive spin count

How many times should we spin before sleeping? Too few and we pay a kernel round trip for a lock that would have been free in a microsecond; too many and we burn CPU on a lock that is never going to be released soon. PostgreSQL guesses, and then learns.

The guess starts at 100:

 c
 
 
// src/include/storage/s_lock.h
/* Support for dynamic adjustment of spins_per_delay */
#define DEFAULT_SPINS_PER_DELAY 100

The learning happens in two steps. First, locally, at the end of each contended acquisition:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * After acquiring a spinlock, update estimates about how long to loop.
 *
 * If we were able to acquire the lock without delaying, it's a good
 * indication we are in a multiprocessor. If we had to delay, it's a sign
 * (but not a sure thing) that we are in a uniprocessor. Hence, we
 * decrement spins_per_delay slowly when we had to delay, and increase it
 * rapidly when we didn't. It's expected that spins_per_delay will
 * converge to the minimum value on a uniprocessor and to the maximum
 * value on a multiprocessor.
 *
 * Note: spins_per_delay is local within our current process. We want to
 * average these observations across multiple backends, since it's
 * relatively rare for this function to even get entered, and so a single
 * backend might not live long enough to converge on a good value. That
 * is handled by the two routines below.
 */
void
finish_spin_delay(SpinDelayStatus *status)
{
	if (status->cur_delay == 0)
	{
		/* we never had to delay */
		if (spins_per_delay < MAX_SPINS_PER_DELAY)
			spins_per_delay = Min(spins_per_delay + 100, MAX_SPINS_PER_DELAY);
	}
	else
	{
		if (spins_per_delay > MIN_SPINS_PER_DELAY)
			spins_per_delay = Max(spins_per_delay - 1, MIN_SPINS_PER_DELAY);
	}
}

The asymmetry is the interesting part: +100 when we got the lock without sleeping, −1 when we had to sleep. The reasoning is that “we never had to delay” is a strong, cheap signal that spinning is worth it, whereas “we had to delay” is a weak signal that it is not — maybe the lock holder was descheduled, maybe it was just unlucky. So the estimate ratchets up quickly and down slowly.

Second, globally. The local value is folded into a shared estimate at backend exit:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * Update shared estimate of spins_per_delay during backend exit.
 *
 * NB: this has to be pretty fast as it is called while holding a spinlock
 */
int
update_spins_per_delay(int shared_spins_per_delay)
{
	/*
	 * We use an exponential moving average with a relatively slow adaption
	 * rate, so that noise in any one backend's result won't affect the shared
	 * value too much. As long as both inputs are within the allowed range,
	 * the result must be too, so we need not worry about clamping the result.
	 *
	 * We deliberately truncate rather than rounding; this is so that single
	 * adjustments inside a backend can affect the shared estimate (see the
	 * asymmetric adjustment rules above).
	 */
	return (shared_spins_per_delay * 15 + spins_per_delay) / 16;
}

(shared * 15 + local) / 16 is an exponential moving average with a weight of 1/16 on the new sample. So a single backend’s experience moves the shared estimate by at most 1/16 of the distance between the two values — and with the truncation, it moves it by at least one unit, which is why the comment mentions “we deliberately truncate rather than rounding”.

The shared value lives in ProcGlobal->spins_per_delay and is read into each backend’s local copy at startup:

 c
 
 
// src/backend/storage/lmgr/proc.c
	/* Update shared estimate of spins_per_delay */
	ProcGlobal->spins_per_delay = update_spins_per_delay(ProcGlobal->spins_per_delay);

	SpinLockRelease(&ProcGlobal->freeProcsLock);

So the estimate is a genuinely global, slowly-adapting, cross-backend number — and it converges, over time, to 1000 (spin hard) on a machine where locks are usually free, or to 10 (give up quickly) on a machine where they are not.

Stuck spinlock detection

If the lock is never released, the loop must terminate. After NUM_DELAYS (1000) sleeps, the backend panics:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * s_lock_stuck() - complain about a stuck spinlock
 */
static void
s_lock_stuck(const char *file, int line, const char *func)
{
	if (!func)
		func = "(unknown)";
#if defined(S_LOCK_TEST)
	fprintf(stderr,
			"\nStuck spinlock detected at %s, %s:%d.\n",
			func, file, line);
	exit(1);
#else
	elog(PANIC, "stuck spinlock detected at %s, %s:%d",
		 func, file, line);
#endif
}

The file header explains the timeout philosophy:

We time out and declare error after NUM_DELAYS delays (thus, exactly that many tries). With the given settings, this will usually take 2 or so minutes. It seems better to fix the total number of tries (and thus the probability of unintended failure) than to fix the total time spent.

Fixing the number of tries rather than the elapsed time means the failure probability does not depend on how loaded the machine is. And the message includes __FILE__, __LINE__ and __func__ of the spinlock’s acquisition site, which is the first thing you want to know when the postmaster takes a PANIC.

Note also that the spin delay is a wait event:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
		pgstat_report_wait_start(WAIT_EVENT_SPIN_DELAY);
		pg_usleep(status->cur_delay);
		pgstat_report_wait_end();

The event is called SpinDelay in the wait event catalogue (src/backend/utils/activity/wait_event_names.txt):

text

SPIN_DELAY	"Waiting while acquiring a contended spinlock."

So pg_stat_activity will show a backend in SpinDelay state — but only when it is sleeping, not when it is actively spinning. The comment in the code explains why the reporting is only done around the sleep: “Actively spinning easily stands out in profilers, but sleeping with an exponential backoff is harder to spot”. If you see a backend burning CPU with no wait event at all, that is a spinner.

Layer 2: the atomics

Spinlocks are the blunt instrument. PostgreSQL also has a proper compare-and-swap API, in src/include/port/atomics.h and the per-architecture headers under src/include/port/atomics/.

The x86 implementations

 c
 
 
// src/include/port/atomics/arch-x86.h
#define PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32
static inline bool
pg_atomic_compare_exchange_u32_impl(volatile pg_atomic_uint32 *ptr,
									uint32 *expected, uint32 newval)
{
	char	ret;

	/*
	 * Perform cmpxchg and use the zero flag which it implicitly sets when
	 * equal to measure the success.
	 */
	__asm__ __volatile__(
	"	lock				\n"
	"	cmpxchgl	%4,%5	\n"
	" setz		%2		\n"
:		"=a" (*expected), "=m"(ptr->value), "=q" (ret)
:		"a" (*expected), "r" (newval), "m"(ptr->value)
:		"memory", "cc");
	return (bool) ret;
}

#define PG_HAVE_ATOMIC_FETCH_ADD_U32
static inline uint32
pg_atomic_fetch_add_u32_impl(volatile pg_atomic_uint32 *ptr, int32 add_)
{
	uint32 res;
	__asm__ __volatile__(
	"	lock				\n"
	"	xaddl	%0,%1		\n"
:		"=q"(res), "=m"(ptr->value)
:		"0" (add_), "m"(ptr->value)
:		"memory", "cc");
	return res;
}

Two instructions, two ideas:

lock cmpxchgl compares the accumulator (%eax) with the destination; if they are equal, the destination is loaded with the source. The old destination value is written back into %eax on failure — which is where the “expected” value comes from. The lock prefix makes the whole thing atomic.
setz reads the zero flag that cmpxchg implicitly sets when the comparison succeeded, and turns it into a 0-or-1 byte. This is the trick the comment refers to: “Perform cmpxchg and use the zero flag which it implicitly sets when equal to measure the success.” Rather than branching (which would be a mispredict), the flag is materialized as a value with setcc.
lock xaddl is “exchange and add”: it atomically adds the source to the destination and returns the old value. One instruction, no loop, no retry.

Note the memory clobbers: "memory", "cc". The memory clobber tells the compiler that the asm may read or write arbitrary memory, so it cannot keep anything in a register across it; the cc clobber tells it that the condition codes were modified.

The API contract

 c
 
 
// src/include/port/atomics.h
/*
 * pg_atomic_compare_exchange_u32 - CAS operation
 *
 * Atomically compare the current value of ptr with *expected and store newval
 * iff ptr and *expected have the same value. The current value of *ptr will
 * always be stored in *expected.
 *
 * Return true if values have been exchanged, false otherwise.
 *
 * Full barrier semantics.
 */

That third sentence — “The current value of *ptr will always be stored in *expected” — is the reason the CAS-loop implementations of the other operations are so short.

The CAS loops

Not every operation has a dedicated instruction on every architecture, so generic.h builds them out of the CAS primitive:

 c
 
 
// src/include/port/atomics/generic.h
#if !defined(PG_HAVE_ATOMIC_FETCH_AND_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_FETCH_AND_U32
static inline uint32
pg_atomic_fetch_and_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 and_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, old & and_))
		/* skip */;
	return old;
}
#endif

#if !defined(PG_HAVE_ATOMIC_FETCH_OR_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_FETCH_OR_U32
static inline uint32
pg_atomic_fetch_or_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 or_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, old | or_))
		/* skip */;
	return old;
}
#endif

The neat part is that the retry needs no extra load. A failed cmpxchg writes the current value of *ptr back into *expected, which is the same variable the loop uses as its source of truth. So after a failure, old already holds the fresh value, and the next iteration computes old & and_ (or old | or_) from it. The comment /* ok if read is not atomic */ on the initial load acknowledges the one non-atomic read in the function: it is only a starting guess, and the CAS loop corrects it immediately.

The same pattern gives you fetch_sub, exchange, and add_fetch:

 c
 
 
// src/include/port/atomics/generic.h
#if !defined(PG_HAVE_ATOMIC_EXCHANGE_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_EXCHANGE_U32
static inline uint32
pg_atomic_exchange_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 xchg_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, xchg_))
		/* skip */;
	return old;
}
#endif

Memory barriers

 c
 
 
// src/include/port/atomics/arch-x86.h
/*
 * Both 32 and 64 bit x86 do not allow loads to be reordered with other loads,
 * or stores to be reordered with other stores, but a load can be performed
 * before a subsequent store.
 *
 * Technically, some x86-ish chips support uncached memory access and/or
 * special instructions that are weakly ordered. In those cases we'd need
 * the read and write barriers to be lfence and sfence. But since we don't
 * do those things, a compiler barrier should be enough.
 *
 * "lock; addl" has worked for longer than "mfence". It's also rumored to be
 * faster in many scenarios.
 */
#if defined(__GNUC__) || defined(__INTEL_COMPILER)
#if defined(__i386__)
#define pg_memory_barrier_impl()		\
	__asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory", "cc")
#elif defined(__x86_64__)
#define pg_memory_barrier_impl()		\
	__asm__ __volatile__ ("lock; addl $0,0(%%rsp)" : : : "memory", "cc")
#endif
#endif /* defined(__GNUC__) || defined(__INTEL_COMPILER) */

#define pg_read_barrier_impl()		pg_compiler_barrier_impl()
#define pg_write_barrier_impl()		pg_compiler_barrier_impl()

lock; addl $0, 0(%rsp) is the classic x86 full barrier: it adds zero to a stack location, but the lock prefix makes it an atomic read-modify-write, which is a full memory barrier. And on x86, a read or write barrier is just a compiler barrier, because the hardware’s memory model is strong enough that you never need more than that.

The 64-bit fallback

On a 32-bit platform without a 64-bit CAS instruction, fallback.h provides an emulated 64-bit atomic — and the emulation is, of course, a spinlock:

 c
 
 
// src/include/port/atomics/fallback.h
/*
 * fallback.h
 * Fallback for platforms without 64 bit atomics support. Slower
 * than native atomics support, but not unusably slow.
 */

#if !defined(PG_HAVE_ATOMIC_U64_SUPPORT)

#define PG_HAVE_ATOMIC_U64_SIMULATION

#define PG_HAVE_ATOMIC_U64_SUPPORT
typedef struct pg_atomic_uint64
{
	int			sema;
	volatile uint64 value;
} pg_atomic_uint64;

And the implementation, in src/backend/port/atomics.c:

 c
 
 
// src/backend/port/atomics.c
bool
pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
									uint64 *expected, uint64 newval)
{
	bool		ret;

	/*
	 * Do atomic op under a spinlock. It might look like we could just skip
	 * the cmpxchg if the lock isn't available, but that'd just emulate a
	 * 'weak' compare and swap. I.e. one that allows spurious failures. Since
	 * several algorithms rely on a strong variant and that is efficiently
	 * implementable on most major architectures let's emulate it here as
	 * well.
	 */
	SpinLockAcquire((slock_t *) &ptr->sema);

	/* perform compare/exchange logic */
	ret = ptr->value == *expected;
	*expected = ptr->value;
	if (ret)
		ptr->value = newval;

	/* and release lock */
	SpinLockRelease((slock_t *) &ptr->sema);

	return ret;
}

The comment is a nice piece of engineering reasoning: a lock-free implementation would be tempted to skip the compare-and-swap when the lock is contended, but that would be a weak CAS — one that may fail spuriously. Several algorithms in PostgreSQL depend on the strong variant, so the fallback implements the strong one, at the cost of spinning.

Note that this is a recursive layering: the 64-bit atomic’s implementation uses the 32-bit spinlock, which uses S_LOCK, which uses perform_spin_delay(). The whole ladder is recursive.

Layer 3: the LWLock

An LWLock is a shared-memory reader/writer lock. It is “lightweight” compared to the heavyweight lock manager in the sense that it has no owner, no lock table, no deadlock detection, and no interaction with pg_locks.

The state word

Everything about an LWLock’s state lives in one 32-bit word:

 c
 
 
// src/include/storage/lmgr/lwlock.h
typedef struct LWLock
{
	uint16		tranche;		/* tranche ID */
	pg_atomic_uint32 state;		/* state of exclusive/nonexclusive lockers */
	proclist_head waiters;		/* list of waiting PGPROCs */
#ifdef LOCK_DEBUG
	pg_atomic_uint32 nwaiters;	/* number of waiters */
	struct PGPROC *owner;		/* last exclusive owner of the lock */
#endif
} LWLock;

And the layout of that word:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
#define LW_FLAG_HAS_WAITERS			((uint32) 1 << 31)
#define LW_FLAG_WAKE_IN_PROGRESS	((uint32) 1 << 30)
#define LW_FLAG_LOCKED				((uint32) 1 << 29)
#define LW_FLAG_BITS				3
#define LW_FLAG_MASK				(((1<<LW_FLAG_BITS)-1)<<(32-LW_FLAG_BITS))

/* assumes MAX_BACKENDS is a (power of 2) - 1, checked below */
#define LW_VAL_EXCLUSIVE			(MAX_BACKENDS + 1)
#define LW_VAL_SHARED				1

/* already (power of 2)-1, i.e. suitable for a mask */
#define LW_SHARED_MASK				MAX_BACKENDS
#define LW_LOCK_MASK				(MAX_BACKENDS | LW_VAL_EXCLUSIVE)

Three flags in the top three bits, and the bottom bits hold the holder count:

Bits	Meaning
31	`LW_FLAG_HAS_WAITERS` — somebody is queued
30	`LW_FLAG_WAKE_IN_PROGRESS` — a wakeup is in flight
29	`LW_FLAG_LOCKED` — the wait list is locked
28–0	shared holder count, or `LW_VAL_EXCLUSIVE` for an exclusive holder

The scheme is described in the file header:

The basic idea is to have a single atomic variable ’lockcount’ instead of the formerly separate shared and exclusive counters and to use atomic operations to acquire the lock. That’s fairly easy to do for plain rw-spinlocks, but a lot harder for something like LWLocks that want to wait in the OS.

For lock acquisition we use an atomic compare-and-exchange on the lockcount variable. For exclusive lock we swap in a sentinel value (LW_VAL_EXCLUSIVE), for shared locks we count the number of holders.

To release the lock we use an atomic decrement to release the lock. If the new value is zero (we get that atomically), we know we can/have to release waiters.

LW_VAL_EXCLUSIVE is MAX_BACKENDS + 1, chosen so that it cannot be confused with a shared holder count — there can never be more than MAX_BACKENDS shared holders, so the two ranges do not overlap. The static assertions check exactly that:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
StaticAssertDecl(((MAX_BACKENDS + 1) & MAX_BACKENDS) == 0,
				 "MAX_BACKENDS + 1 needs to be a power of 2");

StaticAssertDecl((MAX_BACKENDS & LW_FLAG_MASK) == 0,
				 "MAX_BACKENDS and LW_FLAG_MASK overlap");

StaticAssertDecl((LW_VAL_EXCLUSIVE & LW_FLAG_MASK) == 0,
				 "LW_VAL_EXCLUSIVE and LW_FLAG_MASK overlap");

The attempt

 c
 
 
// src/backend/storage/lmgr/lwlock.c
/*
 * Internal function that tries to atomically acquire the lwlock in the passed
 * in mode.
 *
 * This function will not block waiting for a lock to become free - that's the
 * caller's job.
 *
 * Returns true if the lock isn't free and we need to wait.
 */
static bool
LWLockAttemptLock(LWLock *lock, LWLockMode mode)
{
	uint32		old_state;

	Assert(mode == LW_EXCLUSIVE || mode == LW_SHARED);

	/*
	 * Read once outside the loop, later iterations will get the newer value
	 * via compare & exchange.
	 */
	old_state = pg_atomic_read_u32(&lock->state);

	/* loop until we've determined whether we could acquire the lock or not */
	while (true)
	{
		uint32		desired_state;
		bool		lock_free;

		desired_state = old_state;

		if (mode == LW_EXCLUSIVE)
		{
			lock_free = (old_state & LW_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += LW_VAL_EXCLUSIVE;
		}
		else
		{
			lock_free = (old_state & LW_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += LW_VAL_SHARED;
		}

		/*
		 * Attempt to swap in the state we are expecting. If we didn't see
		 * lock to be free, that's just the old value. If we saw it as free,
		 * we'll attempt to mark it acquired. The reason that we always swap
		 * in the value is that this doubles as a memory barrier. We could try
		 * to be smarter and only swap in values if we saw the lock as free,
		 * but benchmark haven't shown it as beneficial so far.
		 *
		 * Retry if the value changed since we last looked at it.
		 */
		if (pg_atomic_compare_exchange_u32(&lock->state,
										 &old_state, desired_state))
		{
			if (lock_free)
			{
				/* Great! Got the lock. */
#ifdef LOCK_DEBUG
				if (mode == LW_EXCLUSIVE)
					lock->owner = MyProc;
#endif
				return false;
			}
			else
				return true;	/* somebody else has the lock */
		}
	}
	pg_unreachable();
}

The comment about the memory barrier is the interesting part:

The reason that we always swap in the value is that this doubles as a memory barrier. We could try to be smarter and only swap in values if we saw the lock as free, but benchmark haven’t shown it as beneficial so far.

On x86, lock cmpxchg is a full barrier whatever it does. So whether the CAS succeeds or fails, the CPU has executed a full fence. The code takes advantage of that unconditionally: it always issues the CAS, even in the case where it already knows the lock is held and it is going to have to wait anyway. The alternative — checking first and skipping the CAS — would save an atomic operation on the contended path, but benchmarks said it wasn’t worth it. (And on a weak-memory architecture, the unconditional CAS is what gives you the barrier for free; skipping it would require an explicit fence.)

Also note the loop structure: old_state is refreshed by the failed CAS (the “current value of *ptr will always be stored in *expected” contract), so the retry always operates on the freshest value.

The wait list lock is a bit in the same word

The LW_FLAG_LOCKED bit in the state word is a spinlock protecting the waiters list. This is a neat bit of economy: the wait list lock lives in the same cache line as the state word, so taking it does not cost an extra cache miss.

 c
 
 
// src/backend/storage/lmgr/lwlock.c
/*
 * Lock the LWLock's wait list against concurrent activity.
 *
 * NB: even though the wait list is locked, non-conflicting lock operations
 * may still happen concurrently.
 *
 * Time spent holding mutex should be short!
 */
static void
LWLockWaitListLock(LWLock *lock)
{
	uint32		old_state;
#ifdef LWLOCK_STATS
	lwlock_stats *lwstats;
	uint32		delays = 0;

	lwstats = get_lwlock_stats_entry(lock);
#endif

	while (true)
	{
		/*
		 * Always try once to acquire the lock directly, without setting up
		 * the spin-delay infrastructure. The work necessary for that shows up
		 * in profiles and is rarely necessary.
		 */
		old_state = pg_atomic_fetch_or_u32(&lock->state, LW_FLAG_LOCKED);
		if (likely(!(old_state & LW_FLAG_LOCKED)))
			break;				/* got lock */

		/* and then spin without atomic operations until lock is released */
		{
			SpinDelayStatus delayStatus;

			init_local_spin_delay(&delayStatus);

			while (old_state & LW_FLAG_LOCKED)
			{
				perform_spin_delay(&delayStatus);
				old_state = pg_atomic_read_u32(&lock->state);
			}
#ifdef LWLOCK_STATS
			delays += delayStatus.delays;
#endif
			finish_spin_delay(&delayStatus);
		}

		/*
		 * Retry. The lock might obviously already be re-acquired by the time
		 * we're attempting to get it again.
		 */
	}

#ifdef LWLOCK_STATS
	lwstats->spin_delay_count += delays;
#endif
}

Two things to notice:

“Always try once to acquire the lock directly, without setting up the spin-delay infrastructure.” The SpinDelayStatus struct has to be initialised, and init_local_spin_delay() writes five fields. For the overwhelmingly common case where the wait list is not contended, that setup is pure overhead — and the comment says it “shows up in profiles”. So the code tries the atomic fetch-or first and only sets up the spin machinery if that fails.
“spin without atomic operations until lock is released” — inside the spin loop, the state word is read with a plain atomic read (pg_atomic_read_u32), not a CAS loop. There is no attempt to take the lock during the spin; the spinner just watches the bit. Only when the bit clears does it retry the fetch-or. That avoids hammering the cache line with atomic RMWs.

Queue-self-then-retry

Now the interesting algorithmic part. The file header describes the two-phase approach:

The attentive reader might have noticed that naively doing the above has a glaring race condition: We try to lock using the atomic operations and notice that we have to wait. Unfortunately by the time we have finished queuing, the former locker very well might have already finished its work. That’s problematic because we’re now stuck waiting inside the OS.

To mitigate those races we use a two phased attempt at locking: Phase 1: Try to do it atomically, if we succeed, nice Phase 2: Add ourselves to the waitqueue of the lock Phase 3: Try to grab the lock again, if we succeed, remove ourselves from the queue Phase 4: Sleep till wake-up, goto Phase 1

And the implementation:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
	for (;;)
	{
		bool		mustwait;

		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = LWLockAttemptLock(lock, mode);

		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "immediately acquired lock");
			break;				/* got the lock */
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		LWLockQueueSelf(lock, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = LWLockAttemptLock(lock, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "acquired, undoing queue");

			LWLockDequeueSelf(lock);
			break;
		}

		/*
		 * Wait until awakened.
		 *
		 * It is possible that we get awakened for a reason other than being
		 * signaled by LWLockRelease. If so, loop back and wait again. Once
		 * we've gotten the LWLock, re-increment the sema by the number of
		 * additional signals received.
		 */
		LOG_LWDEBUG("LWLockAcquire", lock, "waiting");

		LWLockReportWaitStart(lock);
		...
		for (;;)
		{
			PGSemaphoreLock(proc->sem);
			if (proc->lwWaiting == LW_WS_NOT_WAITING)
				break;
			extraWaits++;
		}

		/* Retrying, allow LWLockRelease to release waiters again. */
		pg_atomic_fetch_and_u32(&lock->state, ~LW_FLAG_WAKE_IN_PROGRESS);

		...
		/* Now loop back and try to acquire lock again. */
		result = false;
	}

The critical insight is in the comment: queue first, then try again. If you tried the lock, failed, and then queued, the holder could have released the lock in the window between your failed attempt and your queueing — and nobody would ever wake you up, because the release had already happened. By queueing before the second attempt, you guarantee that any release that happens after your first failed attempt will see you on the wait list and wake you.

The queueing itself:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
static void
LWLockQueueSelf(LWLock *lock, LWLockMode mode)
{
	/*
	 * If we don't have a PGPROC structure, there's no way to wait. This
	 * should never occur, since MyProc should only be null during shared
	 * memory initialization.
	 */
	if (MyProc == NULL)
		elog(PANIC, "cannot wait without a PGPROC structure");

	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

	LWLockWaitListLock(lock);

	/* setting the flag is protected by the spinlock */
	pg_atomic_fetch_or_u32(&lock->state, LW_FLAG_HAS_WAITERS);

	MyProc->lwWaiting = LW_WS_WAITING;
	MyProc->lwWaitMode = mode;

	/* LW_WAIT_UNTIL_FREE waiters are always at the front of the queue */
	if (mode == LW_WAIT_UNTIL_FREE)
		proclist_push_head(&lock->waiters, MyProcNumber, lwWaitLink);
	else
		proclist_push_tail(&lock->waiters, MyProcNumber, lwWaitLink);

	/* Can release the mutex now */
	LWLockWaitListUnlock(lock);
	...
}

Note the panic on double-queueing: elog(PANIC, "queueing for lock while waiting on another one"). An LWLock is a leaf lock — you may hold at most one while waiting for another.

Sleeping: a POSIX semaphore, not a futex

The actual blocking is PGSemaphoreLock(proc->sem) — a POSIX semaphore stored in shared memory, one per PGPROC:

 c
 
 
// src/backend/port/posix_sema.c
/*
 * PGSemaphoreLock
 *
 * Lock a semaphore (decrement count), blocking if count would be < 0
 */
void
PGSemaphoreLock(PGSemaphore sema)
{
	int			errStatus;

	/* See notes in sysv_sema.c's implementation of PGSemaphoreLock. */
	do
	{
		errStatus = sem_wait(PG_SEM_REF(sema));
	} while (errStatus < 0 && errno == EINTR);

	if (errStatus < 0)
		elog(FATAL, "sem_wait failed: %m");
}

The semaphore is padded to a cache line to avoid false sharing:

 c
 
 
// src/backend/port/posix_sema.c
typedef union SemTPadded
{
	sem_t		pgsem;
	char		pad[PG_CACHE_LINE_SIZE];
} SemTPadded;

There is no futex here. PostgreSQL does not implement the Linux futex fast path, so every LWLock wait that reaches the kernel costs a real sem_wait syscall, and every wakeup costs a sem_post. The semaphore implementations are chosen at configure time:

unnamed POSIX semaphores (sem_init/sem_wait), the default where available;
named POSIX semaphores (sem_open);
SysV semaphores (semop) as a fallback.

Releasing and waking

The release path wakes waiters, and it uses the LW_FLAG_WAKE_IN_PROGRESS bit to make sure only one waker is in flight at a time:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
	/* unset required flags, and release lock, in one fell swoop */
	{
		uint32		old_state;
		uint32		desired_state;

		old_state = pg_atomic_read_u32(&lock->state);
		while (true)
		{
			desired_state = old_state;

			/* compute desired flags */

			if (new_wake_in_progress)
				desired_state |= LW_FLAG_WAKE_IN_PROGRESS;
			else
				desired_state &= ~LW_FLAG_WAKE_IN_PROGRESS;

			if (proclist_is_empty(&lock->waiters))
				desired_state &= ~LW_FLAG_HAS_WAITERS;

			desired_state &= ~LW_FLAG_LOCKED;	/* release lock */

			if (pg_atomic_compare_exchange_u32(&lock->state, &old_state,
											 desired_state))
				break;
		}
	}

	/* Awaken any waiters I removed from the queue. */
	proclist_foreach_modify(iter, &wakeup, lwWaitLink)
	{
		PGPROC	 *waiter = GetPGProcByNumber(iter.cur);

		LOG_LWDEBUG("LWLockRelease", lock, "release waiter");
		proclist_delete(&wakeup, iter.cur, lwWaitLink);

		/*
		 * Guarantee that lwWaiting being unset only becomes visible once the
		 * unlink from the link has completed. Otherwise the target backend
		 * could be woken up for other reason and enqueue for a new lock - if
		 * that happens before the list unlink happens, the list would end up
		 * being corrupted.
		 *
		 * The barrier pairs with the LWLockWaitListLock() when enqueuing for
		 * another lock.
		 */
		pg_write_barrier();
		waiter->lwWaiting = LW_WS_NOT_WAITING;
		PGSemaphoreUnlock(waiter->sem);
	}

Two details: the flags are updated with a single CAS so that the wait-list lock is released and the flags are cleared atomically together (otherwise a racing LWLockQueueSelf() could set LW_FLAG_HAS_WAITERS after the releaser checked the list and before it released the lock, and the flag would be stuck on forever); and the write barrier before clearing lwWaiting ensures that the unlink from the wait list is visible before the waiter can wake up and try to queue for another lock.

Deadlock detection: not here

LWLocks have no deadlock detection, and this is not an oversight — it is a design decision. An LWLock is a leaf lock: it protects a small piece of shared state, and the code that holds it does not wait on other locks while holding it. There is no wait-for graph to inspect.

Deadlock detection lives in the heavyweight lock manager instead, in src/backend/storage/lmgr/deadlock.c, which maintains a wait-for graph over the heavyweight locks:

DeadLockCheck – Checks for deadlocks for a given process

This code looks for deadlocks involving the given process. If any are found, we try to resolve them by rearranging the wait queues. If resolution is impossible, return DS_HARD_DEADLOCK — the process must then throw an error.

The practical consequence: if an LWLock deadlock happens, PostgreSQL does not detect it. The backends sit there until deadlock_timeout fires — but that timeout applies to lock waits that the lock manager knows about, not to arbitrary LWLock waits.

Layer 4: latches

A latch is not a lock. It is a flag in shared memory plus a way to wake up whoever is waiting on it. MyLatch is the one every backend has, and it is what WaitLatch() waits on.

SetLatch

 c
 
 
// src/backend/storage/ipc/latch.c
/*
 * Sets a latch and wakes up anyone waiting on it.
 *
 * This is cheap if the latch is already set, otherwise not so much.
 *
 * NB: when calling this in a signal handler, be sure to save and restore
 * errno around it. (That's standard practice in most signal handlers, of
 * course, but we used to omit it in handlers that only set a flag.)
 *
 * NB: this function is called from critical sections and signal handlers so
 * throwing an error is not a good idea.
 */
void
SetLatch(Latch *latch)
{
	...
	/*
	 * The memory barrier has to be placed here to ensure that any flag
	 * variables possibly changed by this process have been flushed to main
	 * memory, before we check/set is_set.
	 */
	pg_memory_barrier();

	/* Quick exit if already set */
	if (latch->is_set)
		return;

	latch->is_set = true;

	pg_memory_barrier();
	if (!latch->maybe_sleeping)
		return;

	...
	owner_pid = latch->owner_pid;
	if (owner_pid == 0)
		return;
	else if (owner_pid == MyProcPid)
		WakeupMyProc();
	else
		WakeupOtherProc(owner_pid);
	...
}

Four things, in order:

A full memory barrier before the test. Any flag variables the process has changed must be visible in main memory before we publish is_set. Otherwise a waiter could see is_set == true, wake up, and look at flags that still have stale values. This is the classic Dekker-style store-load ordering problem, and the barrier is what makes it safe.
“Quick exit if already set.” Setting an already-set latch is a no-op. This is the common case in a signal handler that fires repeatedly, and it means SetLatch() is cheap enough to call from anywhere.
A second memory barrier after setting is_set. This makes the store to is_set visible before we look at maybe_sleeping.
The maybe_sleeping test. This is the optimisation that matters: if nobody is waiting on the latch, do not signal anybody. Signalling is expensive (a syscall, or a write to the self-pipe), and the overwhelmingly common case for SetLatch() is “set the flag and carry on”. The maybe_sleeping flag is set by the waiter just before it blocks:
c
```
		// src/backend/storage/ipc/waiteventset.c
		if (set->latch && !set->latch->is_set)
		{
			/* about to sleep on a latch */
			set->latch->maybe_sleeping = true;
			pg_memory_barrier();
			/* and recheck */
		}

		if (set->latch && set->latch->is_set)
		{
			...
		}
```
The waiter sets maybe_sleeping, issues a barrier, and then re-checks is_set. That re-check is the other half of the protocol: if the latch was set in the window between the waiter’s check and its setting of maybe_sleeping, the re-check catches it and the waiter never sleeps at all. If the latch is set after the re-check, the setter will see maybe_sleeping == true and will signal. Between the two, no wakeup is lost.

Note the ordering of the two flags: the waiter publishes maybe_sleeping before the final check of is_set, and the setter publishes is_set before the final check of maybe_sleeping. That is the standard “store both, then check both” pattern, and it is why there are two barriers in SetLatch().

ResetLatch

 c
 
 
// src/backend/storage/ipc/latch.c
/*
 * Clear the latch. Calling WaitLatch after this will sleep, unless
 * the latch is set again before the WaitLatch call.
 */
void
ResetLatch(Latch *latch)
{
	/* Only the owner should reset the latch */
	Assert(latch->owner_pid == MyProcPid);
	Assert(latch->maybe_sleeping == false);

	latch->is_set = false;

	/*
	 * Ensure that the write to is_set gets flushed to main memory before we
	 * examine any flag variables. Otherwise a concurrent SetLatch might
	 * falsely conclude that it needn't signal us, even though we have missed
	 * seeing some flag updates that SetLatch was supposed to inform us of.
	 */
	pg_memory_barrier();
}

Symmetrically, the barrier after clearing is_set ensures that the clear is visible before the owner examines any flags it might use to decide what to do next. Without it, a concurrent SetLatch() could see the stale is_set == true, decide “already set, nothing to do”, and skip the signal — while the owner has already decided to go to sleep.

WaitLatch and the long-lived WaitEventSet

 c
 
 
// src/backend/storage/ipc/latch.c
/* A common WaitEventSet used to implement WaitLatch() */
static WaitEventSet *LatchWaitSet;

/* The positions of the latch and PM death events in LatchWaitSet */
#define LatchWaitSetLatchPos 0
#define LatchWaitSetPostmasterDeathPos 1

void
InitializeLatchWaitSet(void)
{
	...
	/* Set up the WaitEventSet used by WaitLatch(). */
	LatchWaitSet = CreateWaitEventSet(NULL, 2);
	latch_pos = AddWaitEventToSet(LatchWaitSet, WL_LATCH_SET, PGINVALID_SOCKET,
								 MyLatch, NULL);
	Assert(latch_pos == LatchWaitSetLatchPos);

	/*
	 * WaitLatch will modify this to WL_EXIT_ON_PM_DEATH or
	 * WL_POSTMASTER_DEATH on each call.
	 */
	if (IsUnderPostmaster)
	{
		latch_pos = AddWaitEventToSet(LatchWaitSet, WL_EXIT_ON_PM_DEATH,
									 PGINVALID_SOCKET, NULL, NULL);
		Assert(latch_pos == LatchWaitSetPostmasterDeathPos);
	}
}

One WaitEventSet, created once per backend, reused forever. Every WaitLatch() call does not create a new epoll set; it just modifies the existing one:

 c
 
 
// src/backend/storage/ipc/latch.c
int
WaitLatch(Latch *latch, int wakeEvents, long timeout,
		 uint32 wait_event_info)
{
	WaitEvent	event;

	/* Postmaster-managed callers must handle postmaster death somehow. */
	Assert(!IsUnderPostmaster ||
		 (wakeEvents & WL_EXIT_ON_PM_DEATH) ||
		 (wakeEvents & WL_POSTMASTER_DEATH));

	/*
	 * Some callers may have a latch other than MyLatch, or no latch at all,
	 * or want to handle postmaster death differently. It's cheap to assign
	 * those, so just do it every time.
	 */
	if (!(wakeEvents & WL_LATCH_SET))
		latch = NULL;
	ModifyWaitEvent(LatchWaitSet, LatchWaitSetLatchPos, WL_LATCH_SET, latch);

	if (IsUnderPostmaster)
		ModifyWaitEvent(LatchWaitSet, LatchWaitSetPostmasterDeathPos,
						(wakeEvents & (WL_EXIT_ON_PM_DEATH | WL_POSTMASTER_DEATH)),
						NULL);

	if (WaitEventSetWait(LatchWaitSet,
						 (wakeEvents & WL_TIMEOUT) ? timeout : -1,
						 &event, 1,
						 wait_event_info) == 0)
		return WL_TIMEOUT;
	else
		return event.events;
}

The comment “Some callers may have a latch other than MyLatch, or no latch at all, or want to handle postmaster death differently. It’s cheap to assign those, so just do it every time” is a nice micro-optimization philosophy: rather than maintaining several WaitEventSets and picking one, keep one and mutate it.

epoll and signalfd versus the classic self-pipe

waiteventset.c has three implementations of the same idea, and the file header explains the race each one has to solve:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
/*
 * The poll() implementation uses the so-called self-pipe trick to overcome the
 * race condition involved with poll() and setting a global flag in the signal
 * handler. When a latch is set and the current process is waiting for it, the
 * signal handler wakes up the poll() in WaitLatch by writing a byte to a pipe.
 * A signal by itself doesn't interrupt poll() on all platforms, and even on
 * platforms where it does, a signal that arrives just before the poll() call
 * does not prevent poll() from entering sleep. An incoming byte on a pipe
 * however reliably interrupts the sleep, and causes poll() to return
 * immediately even if the signal arrives before poll() begins.
 *
 * The epoll() implementation overcomes the race with a different technique: it
 * keeps SIGURG blocked and consumes from a signalfd() descriptor instead. We
 * don't need to register a signal handler or create our own self-pipe. We
 * assume that any system that has Linux epoll() also has Linux signalfd().
 *
 * The kqueue() implementation waits for SIGURG with EVFILT_SIGNAL.
 *
 * The Windows implementation uses Windows events that are inherited by all
 * postmaster child processes. There's no need for the self-pipe trick there.
 */

The choice is made at compile time:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
#if defined(WAIT_USE_EPOLL) || defined(WAIT_USE_POLL) || \
	defined(WAIT_USE_KQUEUE) || defined(WAIT_USE_WIN32)
/* don't overwrite manual choice */
#elif defined(HAVE_SYS_EPOLL_H)
#define WAIT_USE_EPOLL
#elif defined(HAVE_KQUEUE)
#define WAIT_USE_KQUEUE
#elif defined(HAVE_POLL)
#define WAIT_USE_POLL
#elif WIN32
#define WAIT_USE_WIN32
#else
#error "no wait set implementation available"
#endif

/*
 * By default, we use a self-pipe with poll() and a signalfd with epoll(), if
 * available. For testing the choice can also be manually specified.
 */
#if defined(WAIT_USE_POLL) || defined(WAIT_USE_EPOLL)
#if defined(WAIT_USE_SELF_PIPE) || defined(WAIT_USE_SIGNALFD)
/* don't overwrite manual choice */
#elif defined(WAIT_USE_EPOLL) && defined(HAVE_SYS_SIGNALFD_H)
#define WAIT_USE_SIGNALFD
#else
#define WAIT_USE_SELF_PIPE
#endif
#endif

So on Linux: epoll + signalfd. Elsewhere with poll(): poll + self-pipe.

The signalfd variant

 c
 
 
// src/backend/storage/ipc/waiteventset.c
#ifdef WAIT_USE_SIGNALFD
	sigset_t	signalfd_mask;

	...
	/* Block SIGURG, because we'll receive it through a signalfd. */
	sigaddset(&UnBlockSig, SIGURG);

	/* Set up the signalfd to receive SIGURG notifications. */
	sigemptyset(&signalfd_mask);
	sigaddset(&signalfd_mask, SIGURG);
	signal_fd = signalfd(-1, &signalfd_mask, SFD_NONBLOCK | SFD_CLOEXEC);
	if (signal_fd < 0)
		elog(FATAL, "signalfd() failed");
	ReserveExternalFD();
#endif

SIGURG is blocked in the process, so it never runs a handler; instead it becomes readable on the signalfd, and the fd is registered with epoll. When the latch is set, SetLatch() sends SIGURG to itself, the kernel marks the signalfd readable, and epoll_wait() returns.

 c
 
 
// src/backend/storage/ipc/waiteventset.c
	/* Sleep */
	rc = epoll_wait(set->epoll_fd, set->epoll_ret_events,
					Min(nevents, set->nevents_space), cur_timeout);

	/* Check return code */
	if (rc < 0)
	{
		/* EINTR is okay, otherwise complain */
		if (errno != EINTR)
		{
			waiting = false;
			ereport(ERROR,
					(errcode_for_socket_access(),
					 errmsg("%s() failed: %m",
							"epoll_wait")));
		}
		return 0;
	}

Note that epoll_wait is a level-triggered wait on the signalfd: the fd stays readable until the pending signals are consumed, which is what the drain() call does.

The self-pipe variant

The classic trick. The pipe is created with both ends non-blocking and close-on-exec:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
	/*
	 * Set up the self-pipe that allows a signal handler to wake up the
	 * poll()/epoll_wait() in WaitLatch. Make the write-end non-blocking, so
	 * that SetLatch won't block if the event has already been set many times
	 * filling the kernel buffer. Make the read-end non-blocking too, so that
	 * we can easily clear the pipe by reading until EAGAIN or EWOULDBLOCK.
	 * Also, make both FDs close-on-exec, since we surely do not want any
	 * child processes messing with them.
	 */
	if (pipe(pipefd) < 0)
		elog(FATAL, "pipe() failed: %m");
	if (fcntl(pipefd[0], F_SETFL, O_NONBLOCK) == -1)
		elog(FATAL, "fcntl(F_SETFL) failed on read-end of self-pipe: %m");
	if (fcntl(pipefd[1], F_SETFL, O_NONBLOCK) == -1)
		elog(FATAL, "fcntl(F_SETFL) failed on write-end of self-pipe: %m");
	if (fcntl(pipefd[0], F_SETFD, FD_CLOEXEC) == -1)
		elog(FATAL, "fcntl(F_SETFD) failed on read-end of self-pipe: %m");
	if (fcntl(pipefd[1], F_SETFD, FD_CLOEXEC) == -1)
		elog(FATAL, "fcntl(F_SETFD) failed on write-end of self-pipe: %m");

	selfpipe_readfd = pipefd[0];
	selfpipe_writefd = pipefd[1];
	selfpipe_owner_pid = MyProcPid;

And the signal handler:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
/*
 * SetLatch uses SIGURG to wake up the process waiting on the latch.
 *
 * Wake up WaitLatch, if we're waiting.
 */
static void
latch_sigurg_handler(SIGNAL_ARGS)
{
	if (waiting)
		sendSelfPipeByte();
}

/* Send one byte to the self-pipe, to wake up WaitLatch */
static void
sendSelfPipeByte(void)
{
	ssize_t		rc;
	char		dummy = 0;

retry:
	rc = write(selfpipe_writefd, &dummy, 1);
	if (rc < 0)
	{
		/* If interrupted by signal, just retry */
		if (errno == EINTR)
			goto retry;

		/*
		 * If the pipe is full, we don't need to retry, the data that's there
		 * already is enough to wake up WaitLatch.
		 */
		if (errno == EAGAIN || errno == EWOULDBLOCK)
			return;

		/*
		 * Oops, the write() failed for some other reason. We might be in a
		 * signal handler, so it's not safe to elog(). We have no choice but
		 * silently ignore the error.
		 */
		return;
	}
}

Every detail here is load-bearing:

One byte, not a message. The byte’s value is irrelevant; only its presence matters.
The write end is non-blocking, and EAGAIN is treated as success. If the pipe’s kernel buffer is full, there is already a byte pending, and a pending byte is exactly as good as a new one: the reader will wake up anyway. Retrying would be pointless (the buffer is full!) and blocking would be fatal (we might be in a signal handler, where blocking is not allowed).
EINTR is retried, because a signal can interrupt the write.
Any other error is silently ignored, because we may be in a signal handler and elog() is not safe there.

The waiting flag is what keeps the handler cheap when nobody is waiting:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
#ifndef WIN32
/* Are we currently in WaitLatch? The signal handler would like to know. */
static volatile sig_atomic_t waiting = false;
#endif

volatile sig_atomic_t is the only kind of variable a signal handler may safely touch.

And the drain:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
/*
 * Read all available data from self-pipe or signalfd.
 * ...
 */
static void
drain(void)
{
	char		buf[1024];
	ssize_t		rc;
	int			fd;

#ifdef WAIT_USE_SELF_PIPE
	fd = selfpipe_readfd;
#else
	fd = signal_fd;
#endif

	for (;;)
	{
		rc = read(fd, buf, sizeof(buf));
		if (rc < 0)
		{
			if (errno == EAGAIN || errno == EWOULDBLOCK)
				break;			/* the descriptor is empty */
			else if (errno == EINTR)
				continue;		/* retry */
			else
			{
				waiting = false;
				...
				elog(ERROR, "read() on self-pipe failed: %m");
			}
		}
		else if (rc == 0)
		{
			...
		}
		else if (rc < sizeof(buf))
		{
			/* we successfully drained the pipe; no need to read() again */
			break;
		}
		/* else buffer wasn't big enough, so read again */
	}
}

The same function drains a signalfd or a self-pipe, which is why the code is shared between the two variants.

The whole picture

text

contended lock
 │
 ├─ spinlock ──── TAS_SPIN (free test + lock xchgb)
 │ └── SPIN_DELAY(): rep; nop / isb
 │ └── pg_usleep(cur_delay), cur_delay *= rand(1,2)
 │ └── after NUM_DELAYS: PANIC "stuck spinlock"
 │
 ├─ LWLock ────── CAS on the state word
 │ └── queue self on the wait list, try again
 │ └── PGSemaphoreLock (sem_wait) — kernel
 │
 └─ Latch ─────── maybe_sleeping test
 └── epoll_wait (signalfd) or poll (self-pipe) — kernel

What it all means in practice

Spinning burns CPU, and it is visible. If a backend is spinning, pg_stat_activity will show it with no wait event, and top will show it at 100% of a core. If it has fallen asleep in the spin delay, the wait event is SpinDelay. Both are symptoms of the same thing: somebody is holding a lock that a hot path needs.

The adaptive spin count is deliberately coarse, and that is fine. The estimate moves by +100 or −1 per acquisition locally, and by 1/16 of the difference per backend exit globally. It is not trying to be precise; it is trying to be stable. A backend that spins 100 times when 10 would do wastes 90 cache-coherence round trips; a backend that spins 10 times when 100 would do pays a pg_usleep() syscall. The asymmetry of the local adjustment (+100/−1) says that erring on the side of spinning is preferred, because spinning does not involve the kernel.

The global estimate only updates when a backend exits. It lives in ProcGlobal->spins_per_delay and is folded in at ProcKill() time. A long-lived backend with a short-lived one can therefore be running with a stale estimate for a long time. This is deliberate: the adjustment is cheap and the noise would be worse than the staleness.

PostgreSQL does not use futexes. Every LWLock wait that reaches the kernel costs a sem_wait syscall, and every wakeup costs a sem_post. On a machine with high LWLock contention, that syscall traffic shows up in strace counts and in perf samples of sem_wait. If you are chasing microseconds of lock overhead, this is where they go.

LWLocks have no deadlock detection. If two LWLocks could deadlock each other, PostgreSQL would not notice and would not report it. The heavyweight lock manager (deadlock.c) does the deadlock detection, and it only knows about heavyweight locks. The practical rule: never acquire an LWLock while holding another one in a way that could cycle. The elog(PANIC, "queueing for lock while waiting on another one") in LWLockQueueSelf() enforces the weaker property that you cannot wait for an LWLock while already waiting for one.

The wait event tells you which rung you are on. pg_stat_activity distinguishes LWLock (waiting for an LWLock), SpinDelay (sleeping inside a spinlock wait), and the latch/socket wait events. If a query is “stuck” and you see LWLock: BufferMapping for a long time, you have a buffer contention problem; if you see SpinDelay, you have a spinlock problem, which usually means something else is wrong.

SetLatch() is cheap on purpose. The maybe_sleeping test means that setting a latch nobody is waiting on costs one memory barrier and a predictable branch. This matters because latches are set on hot paths — every CHECK_FOR_INTERRUPTS(), every lock acquisition that might need to be interrupted — and a syscall there would be unacceptable.

The self-pipe trick exists because signals are unreliable as wakeups. A signal that arrives just before poll() is entered does not prevent poll() from sleeping. A byte in a pipe does. This is why the pipe write end is non-blocking and why EAGAIN is a success: the wakeup mechanism has to be safe to call from a signal handler, where blocking is forbidden and where throwing an error would be fatal.

The 64-bit atomic fallback is a spinlock. On a 32-bit platform without 64-bit CAS, pg_atomic_uint64 carries an embedded spinlock and every 64-bit atomic operation is a spinlock acquisition. That is “not unusably slow” per the header comment, but it does mean that 64-bit atomics are a scalability cliff on such platforms.

Two orders of magnitude separate “spin” from “sleep”. A spinlock acquisition that succeeds costs tens of nanoseconds; one that has to sleep costs at least a microsecond, usually much more. Most performance problems attributed to “lock contention” are really problems with the duration the lock is held, not with the waiting mechanism. Shortening the critical section is worth far more than tuning the spin count.

Performance on Opensource internals

How ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

The layout

Global vs local order

Reading a row

Building the offsets

Insertion keeps the two arrays in sync

The insert path

Serialization: compact discriminators

Dynamic: the open-ended case

The insert path

Adding a variant

Seeing it in SQL

Trade-offs

How ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

The shape of the thing

The index column grows its own type

The dictionary: ColumnUnique and its reverse index

Two reserved slots

The reverse index stores row numbers, not values

The index is built lazily, and the hash table is filled in one pass

Float NaNs are canonicalized

Sharing the dictionary between columns

But a shared dictionary is immutable

Inserting from a column with the same dictionary

On disk: one dictionary per part, versioned per granule

The version byte

The per-granule header

Writing: the dictionary accumulates, then overflows

Reading: the reader has to reconstruct the same state

The single-dictionary-for-part fast path

The overflow path on the read side

The whole thing in SQL

Trade-offs

How ClickHouse Sorts in Parallel: Partial Sort, Merge Sorted Streams and External Sort

The plan: what gets built

PartialSortingTransform: sort the chunk, throw away the rest

Updating the threshold

FinishSortingTransform: merging streams that are already sorted

MergeSortingTransform: accumulate, remerge, spill, merge

Consume: accumulate chunks

Serialize and Generate

The k-way merge

Trade-offs

How ClickHouse Skips Data: Granules, Marks, and Bit-Packed Indexes

Granules: sized in bytes, not rows

Marks: where in the file does each granule start?

Where do the marks come from at write time?

The mark cache

The primary key index: one key value per granule

The trimming trick

The primary index cache

Now the payoff: two different search algorithms

Algorithm 1: binary search (continuous range)

Algorithm 2: generic exclusion search (everything else)

The step budget

The seek heuristic

Putting it together

What this buys you, and what it costs

Seeing it in action

How ClickHouse Plans Distributed Queries with Cascades

The problem the optimizer solves

The memo

Properties: distribution and sorting

Sorting is a property, not an operator

The search: tasks on a stack

Rule ordering

The cost model

The formulas

A read is priced on its scan volume, not its output

The cost model is query-tunable

The rules

Join implementations

Two-stage aggregation

Single-key shuffle alternatives

Enforcers: self-referential expressions

The fixed point

Keeping the best implementation

Failing closed

Trying it

`PartialSortingTransform`: sort the chunk, throw away the rest

`FinishSortingTransform`: merging streams that are already sorted

`MergeSortingTransform`: accumulate, remerge, spill, merge

The `Wide`-only restriction, and the escape hatch