Posts on Opensource internals

How ClickHouse's Sparse Column Stores Only the Rows That Differ

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Real columns are mostly default. A status column is 'active' for 98% of rows and '' for the rest. A retries column is 0 almost always. An amount column that is Nullable is NULL for most rows and non-null for a handful.

Storing all those zeros is paying for bytes you could have described in one sentence. ClickHouse has two answers to this, and they live in different layers:

ColumnNullable keeps a separate byte-per-row null map next to the value column. The values are still all there; the null map says which ones to ignore.
ColumnSparse keeps only the non-default values plus a sorted array of the row positions where they live. The default values are not stored at all — they are implied.

ColumnSparse is the more aggressive of the two, and it is the more interesting one, because the offsets array is not a list of positions but a run-length encoding of gaps, and the whole thing has to survive being filtered, permuted, replicated and merged.

This article is about ColumnSparse and its serializer, in ClickHouse 26.9 (04eac2ea3).

The layout

The header comment says it in four lines:

 cpp
 
 
// src/Columns/ColumnSparse.h:13
/** Column for spare representation.
 * It stores column with non-default values and column
 * with their sorted positions in original column. Column with
 * values contains also one default value at 0 position to make
 * implementation of execution of functions and sorting more convenient.
 */

and the members:

 cpp
 
 
// src/Columns/ColumnSparse.h:268
 /// Contains default value at 0 position.
 /// It's convenient, because it allows to execute, e.g functions or sorting,
 /// for this column without handling different cases.
 WrappedPtr values;

 /// Sorted offsets of non-default values in the full column.
 /// 'offsets[i]' corresponds to 'values[i + 1]'.
 WrappedPtr offsets;
 size_t _size; /// NOLINT

So the invariant is precise: offsets[i] is the row number of the value stored at values[i + 1]. values[0] is always the default value, and is never referenced by an offset.

A worked example. A UInt32 column of 10 rows where only rows 2, 5 and 9 are non-zero:

text

 row: 0 1 2 3 4 5 6 7 8 9
 value: 0 0 17 0 0 42 0 0 0 99

 values: [ 0, 17, 42, 99 ] <- values[0] is the default
 offsets: [ 2, 5, 9 ] <- offsets[i] -> values[i+1]
 _size: 10

To read row n you binary-search offsets for n:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:975
size_t ColumnSparse::getValueIndex(size_t n) const
{
 chassert(n < _size);

 const auto & offsets_data = getOffsetsData();
 const auto * it = std::lower_bound(offsets_data.begin(), offsets_data.end(), n);
 if (it == offsets_data.end() || *it != n)
 return 0;

 return it - offsets_data.begin() + 1;
}

getValueIndex returns 0 for a default row (meaning “use values[0]”) and i+1 otherwise. The header documents the cost:

 cpp
 
 
// src/Columns/ColumnSparse.h:186
 /// Return position of element in 'values' columns,
 /// that corresponds to n-th element of full column.
 /// O(log(offsets.size())) complexity,
 size_t getValueIndex(size_t n) const;

So random access to a sparse column is a binary search — O(log k) where k is the number of non-default values — plus one indirection into the values column. For a column with 2% non-default rows in a million-row block, that is 20 comparisons instead of a single array index. That is the price of the encoding, and it is why ColumnSparse is a storage optimization that the query layer tries to peel off before doing real work.

The constructor enforces the invariant

The invariants are checked at construction, not just assumed:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:35
ColumnSparse::ColumnSparse(MutableColumnPtr && values_, MutableColumnPtr && offsets_, size_t size_)
 : values(std::move(values_)), offsets(std::move(offsets_)), _size(size_)
{
 const ColumnUInt64 * offsets_concrete = typeid_cast<const ColumnUInt64 *>(offsets.get());

 if (!offsets_concrete)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "'offsets' column must be a ColumnUInt64, got: {}", offsets->getName());

 /// 'values' should contain one extra element: default value at 0 position.
 if (offsets->size() + 1 != values->size())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Values size ({}) is inconsistent with offsets size ({})", values->size(), offsets->size());

 if (_size < offsets->size())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Size of sparse column ({}) cannot be lower than number of non-default values ({})", _size, offsets->size());

 if (!offsets_concrete->empty() && _size <= offsets_concrete->getData().back())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Size of sparse column ({}) should be greater than last position of non-default value ({})",
 _size, offsets_concrete->getData().back());

#ifdef DEBUG_OR_SANITIZER_BUILD
 const auto & offsets_data = getOffsetsData();
 const auto * it = std::adjacent_find(offsets_data.begin(), offsets_data.end(), std::greater_equal<>());
 if (it != offsets_data.end())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Offsets of ColumnSparse must be strictly sorted");
#endif
}

Note the asymmetry: the size checks run in release builds, but the strictly sorted check is debug-only (#ifdef DEBUG_OR_SANITIZER_BUILD), because it is O(n) and a sortedness violation would already have produced garbage from the binary search.

Inserting

Insertion is where the “values[0] is the default” convention pays off. You insert the value into the values column and then decide whether to keep it:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:152
void ColumnSparse::insertSingleValue(const Inserter & inserter)
{
 inserter(*values);

 size_t last_idx = values->size() - 1;
 if (values->isDefaultAt(last_idx))
 values->popBack(1);
 else
 getOffsetsData().push_back(_size);

 ++_size;
}

That is the whole trick in six lines: append the value, and if it turned out to be the default, pop it back and do not record an offset. Otherwise record the row number. The values column therefore only ever contains non-default values (plus the sentinel at index 0), and the offsets array is strictly increasing by construction.

Note that isDefaultAt is the type’s notion of default — for UInt32 it is “equals zero”, for String it is “empty”, for Nullable it is “NULL”:

 cpp
 
 
// src/Columns/ColumnVector.h:292
 bool isDefaultAt(size_t n) const override
 {
 if constexpr (is_floating_point<T>)
 {
 /// For floating-point types, use bit_cast to compare raw bit patterns instead of
 /// arithmetic equality. IEEE 754 defines -0.0 == +0.0, so the arithmetic check
 /// would incorrectly treat -0.0 as the default value, losing the sign on
 /// deserialization. Comparing bits directly distinguishes the two: +0.0 is
 /// all-zero bits, while -0.0 has its sign bit set.
 ///
 /// std::conditional_t selects an unsigned integer type of the same size as T,
 /// satisfying the requirement of std::bit_cast that both types have equal size.
 /// Unsigned integers are chosen because their value equals their bit pattern,
 /// making the comparison to 0 unambiguous.
 using Bits = std::conditional_t<sizeof(T) == 2, UInt16,
 std::conditional_t<sizeof(T) == 4, UInt32, UInt64>>;
 return std::bit_cast<Bits>(data[n]) == 0;
 }
 else
 {
 return data[n] == T{};
 }
 }

That -0.0 handling is a nice detail: for floating-point columns the default check is a bit comparison, not an arithmetic one, so that -0.0 is not mistaken for 0.0 and its sign survives a round trip.

The iterator: amortized O(1) sequential access

The binary search is fine for random access, but a scan over a sparse column should not do a binary search per row. The Iterator class exists for that, and it maintains two cursors — the current row and the current offset:

 cpp
 
 
// src/Columns/ColumnSparse.h:206
 /// This class helps to iterate over all values in ColumnSparse.
 class Iterator
 {
 public:
 Iterator(const PaddedPODArray<UInt64> & offsets_, size_t size_, size_t current_offset_, size_t current_row_)
 : offsets(offsets_), offsets_size(offsets.size()), size(size_), current_offset(current_offset_), current_row(current_row_)
 {
 }

 bool ALWAYS_INLINE isDefault() const { return current_offset == offsets_size || current_row != offsets[current_offset]; }
 size_t ALWAYS_INLINE getValueIndex() const { return isDefault() ? 0 : current_offset + 1; }
 size_t ALWAYS_INLINE getCurrentRow() const { return current_row; }
 size_t ALWAYS_INLINE getCurrentOffset() const { return current_offset; }
 size_t ALWAYS_INLINE increaseCurrentRow() const { return ++current_row; }
 size_t ALWAYS_INLINE increaseCurrentOffset() const { return ++current_offset; }

 /// Moves the iterator forward to the given row, which must not be behind the current one.
 /// Amortized constant time when the visited rows are increasing.
 void ALWAYS_INLINE advanceToRow(size_t row)
 {
 while (current_offset < offsets_size && offsets[current_offset] < row)
 ++current_offset;
 current_row = row;
 }

advanceToRow walks the offsets array forward, never backward, so a monotonically increasing scan is O(rows + offsets) in total, not O(rows log offsets). This is the standard “galloping cursor” pattern.

On disk: the offsets become run-length groups

The in-memory representation stores each non-default row’s position explicitly. On disk, that is wasteful: a million rows with 2% non-default values would store 20 000 absolute positions, when what you really want to say is “200 default rows, then a value, then 300 default rows, then a value, …”.

That is what serializeOffsets does:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:70
void serializeOffsets(const IColumn::Offsets & offsets, WriteBuffer & ostr, size_t start, size_t end)
{
 size_t size = offsets.size();
 for (size_t i = 0; i < size; ++i)
 {
 size_t group_size = offsets[i] - start;
 writeVarUInt(group_size, ostr);
 start += group_size + 1;
 }

 size_t group_size = start < end ? end - start : 0;
 group_size |= END_OF_GRANULE_FLAG;
 writeVarUInt(group_size, ostr);
}

Read that carefully. For each non-default row it writes the number of default rows that precede it (offsets[i] - start), and then advances start past that value too (start += group_size + 1). At the end it writes the trailing run of default rows with a special flag bit set.

The flag is bit 62:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:43
{

/// 2^62, because VarInt supports only values < 2^63.
constexpr auto END_OF_GRANULE_FLAG = 1ULL << 62;

Because the offsets are VarInt-encoded, the values are small and the whole offsets stream for a 2%-sparse million-row column is a few kilobytes of VarInts, most of them one or two bytes long.

Decoding it

The decoder has to be resumable, because a read may stop in the middle of a group — the reader asked for limit rows and the group runs past it. The state carries the leftovers:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:48
struct DeserializeStateSparse : public ISerialization::DeserializeBinaryBulkState
{
 /// Number of default values, that remain from previous read.
 size_t num_trailing_defaults = 0;
 /// Do we have non-default value after @num_trailing_defaults?
 bool has_value_after_defaults = false;
 ISerialization::DeserializeBinaryBulkStatePtr nested;

 void reset()
 {
 num_trailing_defaults = 0;
 has_value_after_defaults = false;
 }

and the main loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:125
 size_t group_size = 0;
 while (!istr.eof())
 {
 readVarUInt(group_size, istr);

 bool end_of_granule = group_size & END_OF_GRANULE_FLAG;
 group_size &= ~END_OF_GRANULE_FLAG;

 size_t next_total_rows = total_rows + group_size;
 group_size += num_trailing_defaults;

 if (unlikely(next_total_rows >= limit))
 {
 /// If it was not last group in granule,
 /// we have to add current non-default value at further reads.
 state.num_trailing_defaults = next_total_rows - limit;
 state.has_value_after_defaults = !end_of_granule;
 return limit;
 }

 if (unlikely(end_of_granule))
 {
 has_value_after_defaults = false;
 num_trailing_defaults = group_size;
 }
 else
 {
 /// If we add value to column for first time in current read,
 /// start from column's current size, because it can have some defaults after last offset,
 /// otherwise just start from previous offset.
 /// After the first push `first` is false and `offsets` is non-empty, so the
 /// `!offsets.empty()` guard from the original is redundant and dropped here.
 size_t start_of_group = first ? start : offsets.back() + 1;
 offsets.push_back(start_of_group + group_size);
 first = false;

 num_trailing_defaults = 0;
 ++next_total_rows;
 }

 total_rows = next_total_rows;
 }

The two state variables handle the two ways a read can be cut short:

num_trailing_defaults — the read stopped in the middle of a run of default rows. Those defaults belong to the next read.
has_value_after_defaults — the read stopped exactly at a boundary where a non-default value is pending. The next read has to emit it.

Both are stored in the serialization state, not on the stack, precisely so that a partial read can be resumed later.

The decoder also reserves space for the offsets array using a prior guess:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:109
 /// Just try to guess number of offsets.
 offsets.reserve(offsets.size()
 + static_cast<size_t>(static_cast<double>(limit) * (1.0 - ColumnSparse::DEFAULT_RATIO_FOR_SPARSE_SERIALIZATION)));

using the constant

 cpp
 
 
// src/Columns/ColumnSparse.h:29
 static constexpr auto DEFAULT_ROWS_SEARCH_SAMPLE_RATIO = 0.1;
 static constexpr auto DEFAULT_RATIO_FOR_SPARSE_SERIALIZATION = 0.95;

So the guess is “5% of the rows are non-default”.

How the encoding gets chosen

The sparse encoding is not something you ask for per column. It is selected by the part writer based on the fraction of default values in the column, and it is recorded in the part’s metadata so the reader knows what to expect:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:415
ISerialization::KindStack SerializationInfo::chooseKindStack(const Data & data, const Settings & settings)
{
 ISerialization::KindStack kind_stack = {ISerialization::Kind::DEFAULT};
 double ratio = data.num_rows ? std::min(static_cast<double>(data.num_defaults) / static_cast<double>(data.num_rows), 1.0) : 0.0;
 if (ratio > settings.ratio_of_defaults_for_sparse)
 kind_stack.push_back(ISerialization::Kind::SPARSE);
 return kind_stack;
}

The threshold is the MergeTree table setting ratio_of_defaults_for_sparse_serialization, default 0.9375:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:97
 DECLARE(Float, ratio_of_defaults_for_sparse_serialization, 0.9375f, R"(
Minimal ratio of the number of _default_ values to the number of _all_ values
in a column. Setting this value causes the column to be stored using sparse
serializations.

The kind_stack is a stack of wrappers, not a single flag, because a column can be Sparse inside Replicated inside Sparse, and the order matters:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:266
/// The order in which the kinds wrap each other, innermost first: `ColumnSparse` can sit inside
/// `ColumnReplicated` but not the other way round (see `removeSpecialRepresentations`), and nothing
/// wraps a `ColumnBLOB`. Not the order of the enum, whose values are part of the Native format.

and the reconstruction is a loop over the stack:

 cpp
 
 
// src/DataTypes/IDataType.cpp:456
SerializationPtr IDataType::wrapSerializationBasedOnKindStack(SerializationPtr serialization, const ISerialization::KindStack & kind_stack, const SerializationInfoSettings & settings) const
{
 for (auto kind : kind_stack)
 {
 if (settings.canUseSparseSerialization(*this) && kind == ISerialization::Kind::SPARSE)
 serialization = SerializationSparse::create(serialization);
 else if (kind == ISerialization::Kind::DETACHED)
 serialization = SerializationDetached::create(serialization);
 else if (kind == ISerialization::Kind::REPLICATED)
 serialization = SerializationReplicated::create(serialization);
 }

 return serialization;
}

The kind stack is serialized into serialization.json in the part, so a reader knows to expect a sparse column without having to guess:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:235
void SerializationInfo::serializeKindStackBinary(WriteBuffer & out) const
{
 if (kind_stack == ISerialization::KindStack{ISerialization::Kind::DEFAULT})
 {
 writeBinary(static_cast<UInt8>(KindStackBinarySerializationType::DEFAULT), out);
 }
 else if (kind_stack == ISerialization::KindStack{ISerialization::Kind::DEFAULT, ISerialization::Kind::SPARSE})
 {
 writeBinary(static_cast<UInt8>(KindStackBinarySerializationType::SPARSE), out);
 }

The three substreams

A sparse column on disk is three substreams: the offsets, the elements, and (for a nullable nested type) a null map:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:259
void SerializationSparse::enumerateStreams(
 EnumerateStreamsSettings & settings, const StreamCallback & callback, const SubstreamData & data) const
{
 const auto * column_sparse = data.column ? typeid_cast<const ColumnSparse *>(data.column.get()) : nullptr;
 size_t column_size = column_sparse ? column_sparse->size() : 0;

 settings.path.push_back(Substream::SparseOffsets);
 auto offsets_data = SubstreamData(SerializationNumber<UInt64>::create())
 .withColumn(column_sparse ? column_sparse->getOffsetsPtr() : nullptr)
 .withSerializationInfo(data.serialization_info);

 settings.path.back().data = offsets_data;
 callback(settings.path);

 if (sparse_null_map)
 {
 const auto * type_nullable = data.type ? &assert_cast<const DataTypeNullable &>(*data.type) : nullptr;
 settings.path.back() = Substream::SparseNullMap;
 settings.path.back().creator = std::make_shared<NullMapSubcolumnCreator>(offsets_data.column, column_size);
 auto null_map_data = SubstreamData(sparse_null_map)
 .withType(type_nullable ? std::make_shared<DataTypeUInt8>() : nullptr)
 .withColumn(nullptr) /// subcolumn will be created in NullMapSubcolumnCreator based on offsets
 .withSerializationInfo(data.serialization_info);

 settings.path.back().data = null_map_data;
 callback(settings.path);
 }

 settings.path.back() = Substream::SparseElements;
 settings.path.back().creator = std::make_shared<SubcolumnCreator>(offsets_data.column, column_size);
 settings.path.back().data = data;

The null map of a sparse nullable column is not stored — it is derived from the offsets, because “the row is NULL” and “the row is default” are the same statement for a nullable column:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:248
ColumnPtr SerializationSparse::NullMapSubcolumnCreator::create(const ColumnPtr & /* prev */) const
{
 auto null_map_col = ColumnUInt8::create();
 auto & null_map_data = null_map_col->getData();
 null_map_data.resize_fill(size, static_cast<UInt8>(1));
 const auto & offsets_data = assert_cast<const ColumnUInt64 &>(*offsets).getData();
 for (UInt64 offset : offsets_data)
 null_map_data[offset] = 0;
 return null_map_col;
}

All ones, then punch a zero at each non-default row. That is the whole null map, reconstructed from the offsets array.

Writing

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:311
void SerializationSparse::serializeBinaryBulkWithMultipleStreams(
 const IColumn & column, size_t offset, size_t limit, SerializeBinaryBulkSettings & settings, SerializeBinaryBulkStatePtr & state) const
{
 size_t size = column.size();

 auto offsets_column = DataTypeNumber<IColumn::Offset>().createColumn();
 auto & offsets_data = assert_cast<ColumnVector<IColumn::Offset> &>(*offsets_column).getData();
 column.getIndicesOfNonDefaultRows(offsets_data, offset, limit);

 settings.path.push_back(Substream::SparseOffsets);
 if (auto * stream = settings.getter(settings.path))
 {
 size_t end = limit && offset + limit < size ? offset + limit : size;
 serializeOffsets(offsets_data, *stream, offset, end);
 }

 settings.path.back() = Substream::SparseElements;
 if (!offsets_data.empty())
 {
 if (const auto * column_sparse = typeid_cast<const ColumnSparse *>(&column))
 {
 const auto & values = column_sparse->getValuesColumn();
 size_t begin = column_sparse->getValueIndex(offsets_data[0]);
 size_t end = column_sparse->getValueIndex(offsets_data.back());
 nested->serializeBinaryBulkWithMultipleStreams(values, begin, end - begin + 1, settings, state);
 }
 else
 {
 auto values = column.index(*offsets_column, 0);
 nested->serializeBinaryBulkWithMultipleStreams(*values, 0, values->size(), settings, state);
 }
 }

Note the two cases in the elements branch. If the column is already a ColumnSparse in memory, the values sub-column is contiguous and only the relevant slice is written. If it is a plain column that merely has many defaults, the writer first materializes the non-default values with column.index(*offsets_column, 0).

The query layer wants to peel the sparse wrapper off

Because random access to a sparse column is a binary search, most query operations want the full column. ColumnSparse therefore provides convertToFullColumnIfSparse:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:147
ColumnPtr ColumnSparse::convertToFullColumnIfSparse() const
{
 return values->createWithOffsets(getOffsetsData(), *createColumnConst(values, 0), _size, /*shift=*/ 1);
}

createWithOffsets is the generic mechanism: it takes a values column and an offsets array and produces a full column where each value is repeated offsets[i] - offsets[i-1] times.

There is one place where the engine keeps the sparse representation: when a function has exactly one sparse argument and the function is deterministic, it can be executed on the values only, and the result re-wrapped. That saves decompressing the default rows entirely:

 cpp
 
 
// src/Functions/IFunction.cpp:795
 if (num_sparse_columns == 1 && num_full_columns == 0 && isCallDeterministicInScopeOfQuery(arguments))
 {
 auto & arg_with_sparse = columns_without_sparse[sparse_column_position];
 ColumnPtr sparse_offsets;
 {
 /// New scope to avoid possible mistakes on dangling reference.
 auto & column_sparse = assert_cast<const ColumnSparse &>(*arg_with_sparse.column);
 sparse_offsets = column_sparse.getOffsetsPtr();
 arg_with_sparse.column = column_sparse.getValuesPtr();
 }

 size_t values_size = arg_with_sparse.column->size();
 for (size_t i = 0; i < columns_without_sparse.size(); ++i)
 {
 if (i == sparse_column_position)
 continue;

 columns_without_sparse[i].column = columns_without_sparse[i].column->cloneResized(values_size);
 }

 auto res = executeWithoutSparseColumns(columns_without_sparse, result_type, values_size, dry_run);

 if (isColumnConst(*res))
 return res->cloneResized(input_rows_count);

 /// If default of sparse column is changed after execution of function, convert to full column.
 /// If there are any default in non-zero position after execution of function, convert to full column.
 /// Currently there is no easy way to rebuild sparse column with new offsets.
 if (!result_type->canBeInsideSparseColumns() || !res->isDefaultAt(0) || res->getNumberOfDefaultRows() != 1)
 {
 const auto & offsets_data = assert_cast<const ColumnVector<UInt64> &>(*sparse_offsets).getData();
 return res->createWithOffsets(offsets_data, *createColumnConst(res, 0), input_rows_count, /*shift=*/1);
 }

 return ColumnSparse::create(res, sparse_offsets, input_rows_count);
 }

The condition at the end is the interesting part: the result can only stay sparse if (a) the result type is allowed inside a sparse column, (b) the result’s default value is still the default (position 0 is the only default), and (c) there is exactly one default row. If the function changed the default value, or produced extra defaults in the middle, the sparse structure no longer matches and the column is expanded. The comment says so plainly: “Currently there is no easy way to rebuild sparse column with new offsets.”

That is the honest limitation of this design, stated in the source.

The other side of the coin: the null map

ColumnNullable takes the opposite approach: keep everything, add a byte per row saying which rows are NULL.

 cpp
 
 
// src/Columns/ColumnNullable.h:19
/// Class that specifies nullable columns. A nullable column represents
/// a column, which may have any type, provided with the possibility of
/// storing NULL values. For this purpose, a ColumNullable object stores
/// an ordinary column along with a special column, namely a byte map,
/// whose type is ColumnUInt8. The latter column indicates whether the
/// value of a given row is a NULL or not. Such a design is preferred
/// over a bitmap because columns are usually stored on disk as compressed
/// files. In this regard, using a bitmap instead of a byte map would
/// greatly complicate the implementation with little to no benefits.

The comment explains the design choice: a bitmap would be 8× smaller, but columns are stored compressed, and a byte map of mostly zeros compresses just as well as a bitmap while being trivially indexable. The null map is literally a ColumnUInt8:

 cpp
 
 
// src/Columns/ColumnNullable.h:16
using NullMap = ColumnUInt8::Container;
using ConstNullMapPtr = const NullMap *;

and it is a separate column, which means it is a separate compression stream, a separate substream on disk, and separately filterable:

 cpp
 
 
// src/DataTypes/Serializations/SerializationNullable.cpp:124
 if (!use_default_null_map)
 {
 /// First serialize null map.
 settings.path.push_back(Substream::NullMap);
 if (auto * stream = settings.getter(settings.path))
 SerializationNumber<UInt8>::create()->serializeBinaryBulk(col.getNullMapColumn(), *stream, offset, limit);
 settings.path.pop_back();
 }

The null map is written to its own substream, before the values.

The value column of a Nullable still has one entry per row — the value under a NULL is undefined, but it occupies a slot. insertData shows the convention:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:183
void ColumnNullable::insertData(const char * pos, size_t length)
{
 if (pos == nullptr)
 {
 getNestedColumn().insertDefault();
 getNullMapData().push_back(true);
 }
 else
 {
 getNestedColumn().insertData(pos, length);
 getNullMapData().push_back(false);
 }
}

A NULL still pushes a default into the value column, to keep the two arrays the same length. And the two must stay in sync even if the push throws:

 cpp
 
 
// src/Columns/ColumnNullable.h:97
 void insertDefault() override
 {
 getNestedColumn().insertDefault();
 /// Keep the nested column and the null map in sync even if appending to the
 /// null map throws (e.g. on a memory limit).
 try
 {
 getNullMapData().push_back(true);
 }
 catch (...)
 {
 getNestedColumn().popBack(1);
 throw;
 }
 }

The null map is also where the “combine two nullable columns” logic lives, and it has a subtlety worth reading:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:932
void ColumnNullable::applyNullMapImpl(const NullMap & map, size_t offset)
{
 NullMap & arr = getNullMapData();

 if (offset + map.size() != arr.size())
 throw Exception(
 ErrorCodes::LOGICAL_ERROR,
 "Null map of size {} at offset {} does not match ColumnNullable of size {}",
 map.size(), offset, arr.size());

 /// Any non-zero byte means NULL, so reduce the map before negating it: `negative ^ 2` would
 /// yield 3 and mark a row the map leaves present.
 for (size_t i = 0, size = map.size(); i < size; ++i)
 arr[offset + i] |= negative ^ !!map[i];
}

The null map is a byte map where any non-zero means NULL. When negating, the code has to normalize the incoming map to 0/1 first (!!map[i]), because the bitwise XOR against negative would otherwise produce values like 3 from a 2 — which would still read as “non-zero means NULL” for this map but would corrupt subsequent operations.

Seeing it in SQL

The setting is a MergeTree table setting:

 sql
 
 
CREATE TABLE my_sparse_table
(
 `id` UInt64,
 `s` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS ratio_of_defaults_for_sparse_serialization = 0.95;

INSERT INTO my_sparse_table
SELECT
 number,
 number % 20 = 0 ? toString(number): ''
FROM
 numbers(10000000);

and you can check what happened:

 sql
 
 
-- Which columns got the sparse encoding?
SELECT column, serialization_kind FROM system.parts_columns
WHERE table = 'my_sparse_table';

 sql
 
 
-- And how much did it save?
SELECT table, name, data_compressed_bytes, data_uncompressed_bytes FROM system.columns
WHERE table LIKE 'my_%_table';

The documentation in the setting itself shows the effect on a table where 95% of the s column is the empty string: the sparse version stores fewer bytes both compressed and uncompressed.

Trade-offs

What the sparse encoding buys you.

A column that is 95% default costs 5% of what it would otherwise cost, plus a small offsets array. And the offsets array is not 8 bytes per non-default row on disk — it is a VarInt run-length encoding of the gaps, so a column with long runs of defaults costs almost nothing per gap.

Better still, the sparse representation is propagated: a function applied to a sparse column is executed on the values only, so the CPU cost is proportional to the non-default rows too, not just the memory cost.

What it costs you.

Random access becomes a binary search. A scan is fine (the Iterator is amortized O(1)), but a WHERE row_id = 42 on a sparse column is O(log k) plus an indirection, where a plain column would be one array index. That is why the query layer tries to convert sparse columns to full ones as early as possible — convertToFullColumnIfSparse is called all over the MergeTree read path.

The structure is also fragile in one specific way: the invariant that “the only default is at position 0” must hold for the column to stay sparse. A function that changes the default value, or produces additional defaults in the middle of the column, forces a full expansion. The source says so without embarrassment.

Sparse vs Nullable. These are not competitors; they solve different problems. Nullable answers “is this row NULL?” and pays one byte per row plus the value slot. Sparse answers “what are the rows that are not default?” and pays only for those rows — but it only applies to default values, and defaultness is a property of the type, not of the data. A nullable column whose non-null values are all distinct is not sparse at all; it is a Nullable with a null map.

The design decision that makes it work. The offsets array is stored as gaps, not positions. That single choice turns “8 bytes per non-default row” into “one VarInt per run of default rows”, which for real data (long runs of defaults) is a factor of hundreds. Everything else — the resumable decoder state, the END_OF_GRANULE_FLAG, the derived null map — exists to make that choice work across granule boundaries and partial reads.

How ClickHouse Stores a Column: PODArray, the Typed Array, and the Padding Trick

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Underneath every ClickHouse data type there is exactly one data structure: a flat array of trivially-copyable elements. UInt64 is an array of UInt64. String is a flat character buffer plus an array of offsets. Nullable(UInt32) is an array of UInt32 plus an array of UInt8. There is no node, no pointer chasing, no per-row allocation.

That design is what makes ClickHouse fast, and it comes with a set of non-obvious constraints that shape everything above it:

The array must not be std::vector, because std::vector value-initializes and ClickHouse does not want to pay for zeroing bytes it is about to overwrite.
The array must be able to grow without copying more than amortized O(1) times — but the growth policy has to be chosen, not inherited.
SIMD code needs to be allowed to read and write past the end of the data.
Code that computes “the size of element i” from “the offset of element i” wants a -1 index that is guaranteed to be zero.

This article is about PODArray and ColumnVector, the layer where all of that comes together. Source is from ClickHouse 26.9 (04eac2ea3).

The container

PODArray is documented at the top of its own header:

 cpp
 
 
// src/Common/PODArray.h:33
/** A dynamic array for POD types.
 * Designed for a small number of large arrays (rather than a lot of small ones).
 * To be more precise - for use in ColumnVector.
 * It differs from std::vector in that it does not initialize the elements.
 *
 * Made noncopyable so that there are no accidental copies. You can copy the data using `assign` method.
 *
 * Only part of the std::vector interface is supported.
 *
 * The default constructor creates an empty object that does not allocate memory.
 * Then the memory is allocated at least initial_bytes bytes.
 *
 * If you insert elements with push_back, without making a `reserve`, then PODArray is about 2.5 times faster than std::vector.
 *
 * The template parameter `pad_right` - always allocate at the end of the array as many unused bytes.
 * Can be used to make optimistic reading, writing, copying with unaligned SIMD instructions.
 *
 * The template parameter `pad_left` - always allocate memory before 0th element of the array (rounded up to the whole number of elements)
 * and zero initialize -1th element. It allows to use -1th element that will have value 0.
 * This gives performance benefits when converting an array of offsets to array of sizes.

That comment is the whole design brief. Let the pieces land one at a time.

Noncopyable

 cpp
 
 
// src/Common/PODArray.h:99
template <size_t ELEMENT_SIZE, size_t initial_bytes, typename TAllocator, size_t pad_right_, size_t pad_left_>
class PODArrayBase : private boost::noncopyable, private TAllocator /// empty base optimization
{

The boost::noncopyable private base is not decoration. A Column in ClickHouse is copy-on-write: ColumnPtr is an immutable, refcounted pointer, and mutation happens by cloning first. If the underlying array were copyable, someone would eventually copy one by accident and silently double a column’s memory. Making the array noncopyable pushes every copy through the explicit assign path, where the intent is visible.

The private TAllocator is the other half of that line: the allocator is a base class, not a member, so the array’s object size does not grow with the allocator’s state. The comment in the header explains why:

 cpp
 
 
// src/Common/PODArray.h:58
 * Why Allocator is not passed through constructor, as it is done in C++ standard library?
 * Because sometimes we have many small objects, that share same allocator with same parameters,
 * and we must avoid larger object size due to storing the same parameters in each object.
 * This is required for states of aggregate functions.

Three pointers, not a size and a capacity

The array is three raw pointers:

 cpp
 
 
// src/Common/PODArray.h:119
 char * c_start = null; /// Does not include pad_left.
 char * c_end = null;
 char * c_end_of_storage = null; /// Does not include pad_right.

and the accessors are arithmetic on them:

 cpp
 
 
// src/Common/PODArray.h:227
public:
 bool empty() const { return c_end == c_start; }
 size_t size() const { return (c_end - c_start) / ELEMENT_SIZE; }
 size_t capacity() const { return (c_end_of_storage - c_start) / ELEMENT_SIZE; }

 /// This method is safe to use only for information about memory usage.
 size_t allocated_bytes() const { return c_end_of_storage - c_start + pad_right + pad_left; } /// NOLINT

 void clear() { c_end = c_start; }

size() is a subtraction and a division by a compile-time constant, which the compiler turns into a shift. There is no size_ field to keep in sync, which means there is no possibility of size_ disagreeing with the data.

The empty array points at static memory

A default-constructed PODArray must not allocate. But its c_start must still be a valid pointer, because data() may be called on it, and the padding trick needs memory before the first element. The solution is a static buffer:

 cpp
 
 
// src/Common/PODArray.h:66
static constexpr size_t empty_pod_array_size = 1024;
alignas(std::max_align_t) extern const char empty_pod_array[empty_pod_array_size];

 cpp
 
 
// src/Common/PODArray.cpp:34
/// Used for left padding of PODArray when empty
alignas(std::max_align_t) const char empty_pod_array[empty_pod_array_size]{};

1024 bytes, zero-initialized, max_align_t-aligned, and:

 cpp
 
 
// src/Common/PODArray.h:107
 /// Empty array will point to this static memory as padding and begin/end.
 static constexpr char * null = const_cast<char *>(empty_pod_array) + pad_left;

So an empty array’s c_start points 64 bytes into a static zeroed buffer, which means c_start - 1 is a valid zero byte. The empty array is not a special case in any of the arithmetic; it is a zero-length window into a shared static buffer.

That is also why the static_assert on the left padding exists:

 cpp
 
 
// src/Common/PODArray.h:110
protected:
 static_assert(pad_left <= empty_pod_array_size && "Left Padding exceeds empty_pod_array_size. Is the element size too large?");
 static_assert(pad_left % ELEMENT_SIZE == 0, "pad_left must be multiple of element alignment");

The padding trick

The two padding parameters are the most unusual part of the design.

 cpp
 
 
// src/Common/PODArray.h:103
 /// Round padding up to an whole number of elements to simplify arithmetic.
 static constexpr size_t pad_right = integerRoundUp(pad_right_, ELEMENT_SIZE);
 /// pad_left is also rounded up to 16 bytes to maintain alignment of allocated memory.
 static constexpr size_t pad_left = integerRoundUp(pad_left_, std::lcm(ELEMENT_SIZE, 16));

And the standard instantiation for columns uses 64 bytes on each side:

 cpp
 
 
// src/Common/PODArray_fwd.h:24
/** For columns. Padding is enough to read and write xmm-register at the address of the last element. */
template <typename T, size_t initial_bytes = 4096, typename TAllocator = Allocator<false>>
using PaddedPODArray = PODArray<T, initial_bytes, TAllocator, PADDING_FOR_SIMD - 1, PADDING_FOR_SIMD>;

with

 cpp
 
 
// src/Core/Defines.h:28
static constexpr auto PADDING_FOR_SIMD = 64;

So 63 bytes of right padding and 64 bytes of left padding, for every array in every column of every block. That sounds wasteful until you see what it buys.

Left padding: the `-1` element

Look at how the allocation sets it up:

 cpp
 
 
// src/Common/PODArray.h:128
 template <typename ... TAllocatorParams>
 void alloc(size_t bytes, TAllocatorParams &&... allocator_params)
 {
 char * allocated = reinterpret_cast<char *>(TAllocator::alloc(bytes, std::forward<TAllocatorParams>(allocator_params)...));

 c_start = allocated + pad_left;
 c_end = c_start;
 c_end_of_storage = allocated + bytes - pad_right;

 if (pad_left)
 memset(c_start - ELEMENT_SIZE, 0, ELEMENT_SIZE);
 }

One memset of exactly one element, immediately before the first element. The element at index -1 is guaranteed to be zero.

And the subscript operator is signed to allow it:

 cpp
 
 
// src/Common/PODArray.h:393
 /// The index is signed to access -1th element without pointer overflow.
 T & operator[] (ssize_t n)
 {
 /// <= size, because taking address of one element past memory range is Ok in C++ (expression like &arr[arr.size()] is perfectly valid).
 chassert((n >= (static_cast<ssize_t>(pad_left_) ? -1 : 0)) && (n <= static_cast<ssize_t>(this->size())));
 return t_start()[n];
 }

The -1 is only legal when there is left padding — the assertion encodes the precondition. Without left padding, n = -1 would be a genuine out-of-bounds read.

Now, who uses it? ColumnString:

 cpp
 
 
// src/Columns/ColumnString.h:43
 size_t ALWAYS_INLINE offsetAt(ssize_t i) const { return offsets[i - 1]; }

 /// Size of i-th element
 size_t ALWAYS_INLINE sizeAt(ssize_t i) const
 {
 chassert(offsets[i] >= offsets[i - 1]);
 return offsets[i] - offsets[i - 1];
 }

offsetAt(0) reads offsets[-1], which is the zeroed element. So the offsets array of a ColumnString is a classic “one past the end” offset array where the first offset is implicitly 0, and sizeAt(i) is the difference of two adjacent offsets.

The alternative would be to store offsets as “start of element i” with an explicit offsets[0] = 0 and a separate offsets[size] sentinel, and then sizeAt(i) = offsets[i+1] - offsets[i] — which needs one more slot and would require the array to have size()+1 entries, breaking the invariant that offsets.size() == column.size().

The source even flags the trick where it is used:

 cpp
 
 
// src/Columns/ColumnString.cpp:59
 const size_t src_offset = src_concrete.offsets[position - 1];
 const size_t src_buf_size
 = src_concrete.offsets[position] - src_concrete.offsets[position - 1]; /// -1th index is Ok, see PaddedPODArray.

The same trick appears in ColumnArray:

 cpp
 
 
// src/Columns/ColumnArray.h:242
 size_t ALWAYS_INLINE offsetAt(ssize_t i) const { return getOffsets()[i - 1]; }
 size_t ALWAYS_INLINE sizeAt(ssize_t i) const { return getOffsets()[i] - getOffsets()[i - 1]; }

Right padding: SIMD may write past the end

Right padding exists because ClickHouse’s SIMD filters do not write exactly the number of surviving elements. They write in whole vector-width chunks and then fix up the count:

 cpp
 
 
// src/Columns/ColumnVector.cpp:944
/// Writes the rows of the block at `data_pos` selected by `mask` to `res` and returns their number. Mixed blocks are compressed
/// with a table-driven byte shuffle. Up to `COMPRESS_MAX_OVERWRITE` bytes past the returned count may be written.
/// `res` may alias `data_pos` if it is not ahead of it (filtering in place): each store ends within the rows already loaded.

and the amount of possible overwrite is a compile-time constant:

 cpp
 
 
// src/Columns/ColumnVector.cpp:912
/// One shuffle handles 8 rows of 1-byte elements (a 256-entry table of 16 bytes would be too large for 16 rows),
/// otherwise the rows of a 16-byte vector (32-byte for `COMPRESS_DWORDS`).
template <size_t ELEMENT_WIDTH>
constexpr size_t COMPRESS_ROWS = ELEMENT_WIDTH == 1 ? 8 : (COMPRESS_DWORDS<ELEMENT_WIDTH> ? 32 : 16) / ELEMENT_WIDTH;

/// One shuffle stores a whole vector of `COMPRESS_ROWS` elements from the current count, which can be past all selected rows.
template <size_t ELEMENT_WIDTH>
constexpr size_t COMPRESS_MAX_OVERWRITE = COMPRESS_ROWS<ELEMENT_WIDTH> * ELEMENT_WIDTH;

The padding requirement is then a static assertion:

 cpp
 
 
// src/Columns/ColumnVector.cpp:1000
/// Filters whole blocks of `SIMD_ELEMENTS` rows into `res_data`.
template <typename T, typename Container, size_t SIMD_ELEMENTS>
void doFilterAlignedShuffle(const UInt8 *& filt_pos, const UInt8 *& filt_end_aligned, const T *& data_pos, Container & res_data)
{
 /// The writes of `compressBlock` past the selected rows land in the right padding, so an exact `result_size_hint` is kept.
 static_assert(Container::pad_right >= COMPRESS_MAX_OVERWRITE<sizeof(T)>);

This is the payoff of the padding design: the SIMD filter can be written without a single branch for the tail, because the buffer is guaranteed to have room for the overrun. The static_assert ties the guarantee to the code that depends on it, so the two cannot drift apart.

The same trick shows up in the small-string copy helper:

 cpp
 
 
// src/Common/PODArray.h:509
 /// Works under assumption, that it's possible to read up to 15 excessive bytes after `from_end` and this PODArray is padded.
 template <typename It1, typename It2, typename ... TAllocatorParams>
 void insertSmallAllowReadWriteOverflow15(It1 from_begin, It2 from_end, TAllocatorParams &&... allocator_params)
 {
 static_assert(pad_right_ >= PADDING_FOR_SIMD - 1);
 static_assert(sizeof(T) == sizeof(*from_begin));
 insertPrepare(from_begin, from_end, std::forward<TAllocatorParams>(allocator_params)...);
 size_t bytes_to_copy = PODArrayDetails::byte_size(from_end - from_begin, sizeof(T));
 memcpySmallAllowReadWriteOverflow15(this->c_end, reinterpret_cast<const void *>(&*from_begin), bytes_to_copy);
 this->c_end += bytes_to_copy;
 }

which copies 16 bytes at a time regardless of the actual size:

 cpp
 
 
// src/Common/memcpySmall.h:41
 inline void memcpySmallAllowReadWriteOverflow15Impl(char * __restrict dst, const char * __restrict src, ssize_t n)
 {
 __msan_unpoison_overflow_15(src, n);
 while (n > 0)
 {
 _mm_storeu_si128(reinterpret_cast<__m128i *>(dst),
 _mm_loadu_si128(reinterpret_cast<const __m128i *>(src)));

 dst += 16;
 src += 16;
 n -= 16;

 /// Avoid clang loop-idiom optimization, which transforms _mm_storeu_si128 to built-in memcpy
 __asm__ __volatile__("" : : : "memory");
 }
 }

Note the n -= 16 in a loop that runs until n <= 0: for a 5-byte copy it runs once and advances 16 bytes past the end. That is legal only because of the padding, and the static_assert above enforces it. The __msan_unpoison_overflow_15 call tells MemorySanitizer that the 15-byte overread is intentional, so sanitizers do not flag it.

The growth policy

There are three different growth behaviours in PODArray, and choosing between them is a deliberate decision at each call site.

`reserve`: round up to a power of two

 cpp
 
 
// src/Common/PODArray.h:237
 template <typename ... TAllocatorParams>
 ALWAYS_INLINE void reserve(size_t n, TAllocatorParams &&... allocator_params)
 {
 if (n > capacity())
 reallocPowerOfTwoElements(n, std::forward<TAllocatorParams>(allocator_params)...);
 }

which rounds the element count up to a power of two:

 cpp
 
 
// src/Common/PODArray.h:200
 template <typename ... TAllocatorParams>
 NO_INLINE void reallocPowerOfTwoElements(size_t n, TAllocatorParams &&... allocator_params)
 {
 realloc(roundUpToPowerOfTwoOrZero(PODArrayDetails::minimum_memory_for_elements(n, ELEMENT_SIZE, pad_left, pad_right)), std::forward<TAllocatorParams>(allocator_params)...);
 }

The rounding helper is a one-liner:

 cpp
 
 
// src/Common/BitHelpers.h:15
inline size_t roundUpToPowerOfTwoOrZero(size_t n)
{
 // if MSB is set, return n, to avoid return zero
 if (unlikely(n >= 0x8000000000000000ULL))
 return n;
 else if (n <= 1)
 return n;

and the comment explains the clzll/clzl choice, which is a nice example of a portability trap being closed with a comment rather than a bug report:

 cpp
 
 
// src/Common/BitHelpers.h:23
 /// `clzll`, not `clzl`: the argument is a 64-bit `size_t`, as the hardcoded 64 and the
 /// `0x8000000000000000ULL` above both assume, but `unsigned long` - what `clzl` takes - is
 /// only 64-bit wide on LP64 platforms. On an LLP64 one it is 32, and the argument would be
 /// truncated. `clzll` is at least 64 bits everywhere.

`reserve_exact`: exactly what was asked for

 cpp
 
 
// src/Common/PODArray.h:244
 template <typename ... TAllocatorParams>
 void reserve_exact(size_t n, TAllocatorParams &&... allocator_params) /// NOLINT
 {
 if (n > capacity())
 realloc(PODArrayDetails::minimum_memory_for_elements(n, ELEMENT_SIZE, pad_left, pad_right), std::forward<TAllocatorParams>(allocator_params)...);
 }

This is the one that surprises people. ColumnVector::reserve uses reserve_exact, not reserve:

 cpp
 
 
// src/Columns/ColumnVector.h:182
 void reserve(size_t n) override
 {
 data.reserve_exact(n);
 }

 size_t capacity() const override
 {
 return data.capacity();
 }

 void shrinkToFit() override
 {
 data.shrink_to_fit();
 }

Why would a column reserve exactly? Because a column’s final size is usually known in advance. When ClickHouse reads a MergeTree granule it knows how many rows the granule has; when it deserializes a Block from the wire it knows the row count. Rounding up to a power of two would waste up to 50% of the memory on every column in every block — and a block with 50 columns of 65 536 rows each would waste tens of megabytes.

The shrinkToFit override exists for the case where the size was not known in advance: after the block is fully built, the over-allocation is returned.

The same policy shows up in ColumnString:

 cpp
 
 
// src/Columns/ColumnString.cpp:636
void ColumnString::reserve(size_t n)
{
 offsets.reserve_exact(n);
}

and in ColumnFixedString, which reserves in characters:

 cpp
 
 
// src/Columns/ColumnFixedString.h:197
 void reserve(size_t size) override
 {
 chars.reserve_exact(n * size);
 }

`push_back`: double the allocation

The third policy is the incremental one:

 cpp
 
 
// src/Common/PODArray.h:442
 template <typename U, typename ... TAllocatorParams>
 void push_back(U && x, TAllocatorParams &&... allocator_params) /// NOLINT
 {
 if (unlikely(this->c_end + sizeof(T) > this->c_end_of_storage))
 this->reserveForNextSize(std::forward<TAllocatorParams>(allocator_params)...);
 new (reinterpret_cast<void*>(t_end())) T(std::forward<U>(x));
 this->c_end += sizeof(T);
 }

and the growth itself:

 cpp
 
 
// src/Common/PODArray.h:185
 template <typename ... TAllocatorParams>
 NO_INLINE void reserveForNextSize(TAllocatorParams &&... allocator_params)
 {
 if (empty())
 {
 // The allocated memory should be multiplication of ELEMENT_SIZE to hold the element, otherwise,
 // memory issue such as corruption could appear in edge case.
 realloc(std::max(integerRoundUp(initial_bytes, ELEMENT_SIZE),
 PODArrayDetails::minimum_memory_for_elements(1, ELEMENT_SIZE, pad_left, pad_right)),
 std::forward<TAllocatorParams>(allocator_params)...);
 }
 else
 realloc(allocated_bytes() * 2, std::forward<TAllocatorParams>(allocator_params)...);
 }

Three details worth noting:

The growth is allocated_bytes() * 2 — doubling the bytes, including the padding. So the element capacity grows by slightly more than 2× when the element is small relative to the 127 bytes of padding.
The first allocation is initial_bytes (4096 for column arrays), not one element. So a column that gets a handful of rows still allocates 4 KB — but only once, and initial_bytes is the minimum, so it is paid once per column, not per row.
reserveForNextSize is NO_INLINE. The comment above it explains why:

 cpp
 
 
// src/Common/PODArray.h:183
 /// NO_INLINE functions for slow paths of otherwise fast small inlinable functions.

The realloc path is cold; inlining it into push_back would bloat every caller.

The “shrink if less than half used” trick

One more policy lives in ColumnAggregateFunction, and it is a good illustration of the trade-off. When you filter a column, you know approximately how many rows will survive, so you can reserve exactly. But if the filter is much more selective than the hint, you are left with a huge buffer holding a few elements:

 cpp
 
 
// src/Columns/ColumnAggregateFunction.cpp:335
 auto res = createView();
 auto & res_data = res->data;

 if (result_size_hint)
 res_data.reserve_exact(result_size_hint > 0 ? result_size_hint : size);

 for (size_t i = 0; i < size; ++i)
 if (filter[i])
 res_data.push_back(data[i]);

 /// To save RAM in case of too strong filtering.
 if (res_data.size() * 2 < res_data.capacity())
 res_data = Container(res_data.cbegin(), res_data.cend());

The check size * 2 < capacity means “if less than half the buffer is used, copy it into a tight one”. The copy costs O(size) and is worth it when it frees more than half the buffer.

ColumnVector::filter does the same thing, but with a SIMD path that reserves based on the hint:

 cpp
 
 
// src/Columns/ColumnVector.cpp:1046
 auto res = this->create();
 Container & res_data = res->getData();

 if (result_size_hint)
 res_data.reserve_exact(result_size_hint > 0 ? result_size_hint : size);

`ColumnVector`: the typed array

With the container settled, ColumnVector is a thin wrapper:

 cpp
 
 
// src/Columns/ColumnVector.h:39
public:
 using ValueType = T;
 using Container = PaddedPODArray<ValueType>;

private:
 ColumnVector() = default;
 explicit ColumnVector(const size_t n) : data(n) {}
 ColumnVector(const size_t n, const ValueType x) : data(n, x) {}
 ColumnVector(const ColumnVector & src) : data(src.data.begin(), src.data.end()) {}
 ColumnVector(Container::const_iterator begin, Container::const_iterator end) : data(begin, end) { }

Note that the copy constructor of ColumnVector copies the data explicitly, element by element, rather than memcpy’ing the whole buffer. That looks wasteful, and it is — but ColumnVector is copy-on-write at the IColumn level, so the copy constructor is only reached when someone genuinely wants a copy, and the element-by-element copy lets the compiler vectorize it. PaddedPODArray is noncopyable, so this constructor is the only way to copy one, and it is not accidental.

The interesting part is what ColumnVector exposes to code that wants to avoid virtual dispatch:

 cpp
 
 
// src/Columns/ColumnVector.h:335
 /** More efficient methods of manipulation - to manipulate with data directly. */
 Container & getData()
 {
 return data;
 }

 const Container & getData() const
 {
 return data;
 }

 const T & getElement(size_t n) const
 {
 return data[n];
 }

 T & getElement(size_t n)
 {
 return data[n];
 }

protected:
 Container data;

getData() hands out the raw PaddedPODArray, so a caller that has already established that the column is a ColumnVector<UInt64> can read data.data()[i] directly with no virtual call, no bounds check, and no per-element function call.

This is the pattern that ClickHouse uses throughout: the IColumn interface is for structural operations (filter, permute, serialize), and the concrete class exposes its guts for the data operations. A typical aggregation function does one checkAndGetColumn<ColumnVector<UInt64>>() and then runs a monomorphic loop over getData().

`insertRangeFrom`: memcpy, not a loop

 cpp
 
 
// src/Columns/ColumnVector.cpp:746
template <typename T>
#if !defined(DEBUG_OR_SANITIZER_BUILD)
void ColumnVector<T>::insertRangeFrom(const IColumn & src, size_t start, size_t length)
#else
void ColumnVector<T>::doInsertRangeFrom(const IColumn & src, size_t start, size_t length)
#endif
{
 const ColumnVector & src_vec = assert_cast<const ColumnVector &>(src);

 if (start > src_vec.data.size() || length > src_vec.data.size() - start)
 throwInsertRangeFromOutOfBound("ColumnVector<T>", start, length, src_vec.data.size());

 size_t old_size = data.size();
 data.resize(old_size + length);
 memcpy(data.data() + old_size, &src_vec.data[start], length * sizeof(data[0]));
}

Two things here. First, data.resize(...) — and PODArray::resize does not initialize:

 cpp
 
 
// src/Common/PODArray.h:251
 template <typename ... TAllocatorParams>
 void resize(size_t n, TAllocatorParams &&... allocator_params)
 {
 reserve(n, std::forward<TAllocatorParams>(allocator_params)...);
 resize_assume_reserved(n);
 }

 cpp
 
 
// src/Common/PODArray.h:271
 void resize_assume_reserved(const size_t n) /// NOLINT
 {
 c_end = c_start + PODArrayDetails::byte_size(n, ELEMENT_SIZE);
 }

So resize just moves c_end. No zeroing, no element construction. The memcpy then fills the region. Compare with resize_fill:

 cpp
 
 
// src/Common/PODArray.h:419
 /// Same as resize, but zeroes new elements.
 void resize_fill(size_t n) /// NOLINT
 {
 size_t old_size = this->size();
 if (n > old_size)
 {
 this->reserve(n);
 memset(this->c_end, 0, PODArrayDetails::byte_size(n - old_size, sizeof(T)));
 }
 this->c_end = this->c_start + PODArrayDetails::byte_size(n, sizeof(T));
 }

ColumnVector uses resize_fill where the values matter:

 cpp
 
 
// src/Columns/ColumnVector.h:70
 void insertManyFrom(const IColumn & src, size_t position, size_t length) override
 {
 ValueType v = assert_cast<const Self &>(src).getData()[position];
 data.resize_fill(data.size() + length, v);
 }

Second, insertRangeFrom does not call push_back in a loop. It does one resize and one memcpy. That is only possible because the destination is a flat array of the same element type — the whole reason for the ColumnVector design.

`insertRawUninitialized`: writing into the column’s memory directly

There is an even more aggressive version of this, used by the Parquet reader and the hash-join builder. The column grows by n elements and hands back a pointer to the new region, so the caller can decompress straight into it:

 cpp
 
 
// src/Columns/ColumnVector.cpp:1528
template <typename T>
std::span<char> ColumnVector<T>::insertRawUninitialized(size_t count)
{
 size_t start = data.size();
 data.resize(start + count);
 return {reinterpret_cast<char *>(data.data() + start), count * sizeof(T)};
}

The interface says what it is for:

 cpp
 
 
// src/Columns/IColumn.h:928
 /// Appends n elements with unspecified values and returns a span pointing to their memory range.
 /// Can be used to decompress or deserialize data directly into the column.
 /// Supported only for simple column types like ColumnVector and ColumnFixedString.
 [[nodiscard]] virtual std::span<char> insertRawUninitialized(size_t count);

This is the “external buffer” trick in modern ClickHouse form: instead of decompressing into a temporary buffer and then copying into the column, the decoder writes into the column. One copy instead of two.

The Nullable case does it for both halves at once:

 cpp
 
 
// src/Interpreters/HashJoin/gatherJoinOutputColumns.cpp:407
 const std::span<char> null_span = dst.getNullMapColumn().insertRawUninitialized(count);
 const std::span<char> value_span = dst.getNestedColumn().insertRawUninitialized(count);

`ColumnString`: offsets plus a flat character buffer

ColumnString is the same idea, extended to variable-length data. Two arrays:

 cpp
 
 
// src/Columns/ColumnString.h:34
private:
 friend class COWHelper<IColumnHelper<ColumnString>, ColumnString>;

 /// Maps i'th position to offset to i+1'th element. Last offset maps to the end of all chars (is the size of all chars).
 Offsets offsets;

 /// Bytes of strings, placed contiguously. Note that strings are not zero-terminated and could contain zero bytes in the middle.
 Chars chars;

Note the comment on chars: strings are not zero-terminated. The length comes entirely from the offsets. And the offsets array holds size()+1 entries conceptually — the last one is the end of the character buffer — but because of the -1 trick, the stored array has only size() entries and the implicit offsets[-1] == 0 provides the first one.

The layout for three strings:

text

 offsets: [ 0, 5, 6, 11 ] (implicit offsets[-1] = 0)
 chars: [ 'H','e','l','l','o', ',', 'W','o','r','l','d' ]

 string 0 = chars[offsets[-1] .. offsets[0]) = chars[0..5) = "Hello"
 string 1 = chars[offsets[0] .. offsets[1]) = chars[5..6) = ","
 string 2 = chars[offsets[1] .. offsets[2]) = chars[6..11) = "World"

Reading a row is two array reads and a subtraction:

 cpp
 
 
// src/Columns/ColumnString.h:118
 std::string_view getDataAt(size_t n) const override
 {
 chassert(n < size());
 return std::string_view(reinterpret_cast<const char *>(&chars[offsetAt(n)]), sizeAt(n));
 }

No pointer to chase, no per-string allocation, no length prefix in the character buffer. The whole column is two contiguous allocations.

Bulk insertion

Copying a range of strings is one memcpy per string plus one pass over the offsets:

 cpp
 
 
// src/Columns/ColumnString.cpp:58
 const ColumnString & src_concrete = assert_cast<const ColumnString &>(src);
 const size_t src_offset = src_concrete.offsets[position - 1];
 const size_t src_buf_size
 = src_concrete.offsets[position] - src_concrete.offsets[position - 1]; /// -1th index is Ok, see PaddedPODArray.

 const size_t old_rows = offsets.size();
 const size_t new_rows = old_rows + length;
 /// Reserve offsets before changing chars to keep the column consistent if allocation fails.
 offsets.reserve(new_rows);

 const size_t old_size = chars.size();
 const size_t new_size = old_size + src_buf_size * length;
 chars.resize(new_size);

 const UInt8 * src_buf = &src_concrete.chars[src_offset];

 offsets.resize_assume_reserved(new_rows);

 for (size_t current_offset = old_size; current_offset < new_size; current_offset += src_buf_size)
 memcpySmallAllowReadWriteOverflow15(&chars[current_offset], src_buf, src_buf_size);

 for (size_t i = 0, current_offset = old_size + src_buf_size; i < length; ++i, current_offset += src_buf_size)
 offsets[old_rows + i] = current_offset;

Note the ordering, and the comment explaining it: offsets.reserve happens before chars is modified, so that if the allocation fails the column is still consistent. And chars.resize does not initialize — the memcpySmallAllowReadWriteOverflow15 calls fill it.

Note also that the offsets are written after the characters. That ordering means the offsets array is never in a state where it points past the end of the character buffer, even transiently.

`ColumnFixedString`: the inline variant

FixedString(N) is the degenerate case where every value has the same length, so no offsets array is needed at all — just a flat character buffer of N * size bytes:

 cpp
 
 
// src/Columns/ColumnFixedString.h:26
private:
 /// Bytes of strings, laid in succession. The strings are stored without a trailing zero byte.
 /** NOTE It is required that the offset and type of chars in the object be the same as that of `data in ColumnUInt8`.
 * Used in `packFixed` function (AggregationCommon.h)
 */
 Chars chars;
 /// The size of the rows.
 const size_t n;

The comment is a constraint on the representation: ColumnFixedString’s buffer must have exactly the same layout as a ColumnUInt8’s, so that generic code can reinterpret one as the other. That is what lets ColumnFixedSizeHelper do this:

 cpp
 
 
// src/Columns/ColumnFixedSizeHelper.h:31
 template <size_t ELEMENT_SIZE>
 const char * getRawDataBegin() const
 {
 return reinterpret_cast<const PODArrayBase<ELEMENT_SIZE, 4096, Allocator<false>, PADDING_FOR_SIMD - 1, PADDING_FOR_SIMD> *>(
 reinterpret_cast<const char *>(this) + sizeof(*this))
 ->raw_data();
 }

The class exists so that this downcast is technically legal:

 cpp
 
 
// src/Columns/ColumnFixedSizeHelper.h:11
/** Allows to access internal array of fixed-size column without cast to concrete type.
 * We will inherit ColumnVector and ColumnFixedString from this class instead of IColumn.
 * Assumes data layout of ColumnVector, ColumnFixedString and PODArray.
 *
 * Why it is needed?
 *
 * There are some algorithms that specialize on the size of data type but doesn't care about concrete type.
 * The same specialization may work for UInt64, Int64, Float64, FixedString(8), if it only does byte moving and hashing.
 * To avoid code bloat and compile time increase, we can use single template instantiation for these cases
 * and just static_cast pointer to some single column type (e. g. ColumnUInt64) assuming that all types have identical memory layout.
 *
 * But this static_cast (downcast to unrelated type) is illegal according to the C++ standard and UBSan warns about it.
 * To allow functional tests to work under UBSan we have to separate some base class that will present the memory layout in explicit way,
 * and we will do static_cast to this class.
 */

That is a lovely piece of engineering commentary: the constraint is documented, the workaround is a dedicated base class, and the reason is UBSan.

The n is fixed at construction and the buffer is sized in n-byte units:

 cpp
 
 
// src/Columns/ColumnFixedString.h:55
 size_t size() const override
 {
 return chars.size() / n;
 }

with the trailing-zero-byte question resolved by the serializer rather than the column: the in-memory buffer holds exactly n bytes per row with no terminator, and shorter strings are zero-padded on the way in:

 cpp
 
 
// src/Columns/ColumnFixedString.cpp:117
void ColumnFixedString::insertData(const char * pos, size_t length)
{
 if (length > n)
 throw Exception(ErrorCodes::TOO_LARGE_STRING_SIZE, "Too large string for FixedString column");

 size_t old_size = chars.size();
 chars.resize(old_size + n);
 memcpy(chars.data() + old_size, pos, length);
 memset(chars.data() + old_size + length, 0, n - length);
}

`ColumnNullable`: the null map as a separate column

Nullable(T) is a T column plus a ColumnUInt8 of the same length. The comment gives the reasoning:

 cpp
 
 
// src/Columns/ColumnNullable.h:19
/// Class that specifies nullable columns. A nullable column represents
/// a column, which may have any type, provided with the possibility of
/// storing NULL values. For this purpose, a ColumNullable object stores
/// an ordinary column along with a special column, namely a byte map,
/// whose type is ColumnUInt8. The latter column indicates whether the
/// value of a given row is a NULL or not. Such a design is preferred
/// over a bitmap because columns are usually stored on disk as compressed
/// files. In this regard, using a bitmap instead of a byte map would
/// greatly complicate the implementation with little to no benefits.

That is the whole design decision: a bitmap would be 8× smaller in RAM, but columns are stored compressed, and a byte map of mostly zeros compresses just as well as a bitmap while remaining a perfectly ordinary column that can be filtered, compressed and serialized by the generic machinery.

The two members are independent columns, each with its own lifecycle:

 cpp
 
 
// src/Columns/ColumnNullable.h:263
private:
 WrappedPtr nested_column;
 WrappedPtr null_map;

and reading a row is a lookup in each:

 cpp
 
 
// src/Columns/ColumnNullable.h:55
 size_t size() const final { return assert_cast<const ColumnUInt8 &>(*null_map).size(); }
 bool isNullAt(size_t n) const final { return assert_cast<const ColumnUInt8 &>(*null_map).getData()[n] != 0;}

Note that size() is defined by the null map, not the nested column. The null map is the authoritative row count.

There is a subtle point about what lives in the nested column under a NULL. The convention is that the value is undefined — it is present, but meaningless. The code makes that explicit in the one place where it matters:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:995
 if (isNullAt(0))
 {
 /// Value in main column, when null map is 1 is implementation defined. So, take any value.
 new_values = nested_column->createWithOffsets(offsets, *createColumnConstWithDefaultValue(nested_column), total_rows, shift);

So “nullable” is implemented as “a value column plus a parallel byte column”, and the value column is not branch-free — reading a nullable value always requires a branch on the null map.

Trade-offs

What the flat-array design buys you.

Every column is one or two contiguous allocations. Scans are sequential memory reads that the prefetcher handles perfectly. There is no per-row indirection, so a SELECT sum(x) FROM t on a UInt64 column is a linear sweep over memory at memory bandwidth.

The offsets+chars layout for strings means a string column is also a linear sweep: the character buffer is contiguous, and the offsets are a separate dense UInt64 array. This is also why ClickHouse’s string compression works so well — the LZ4 compressor sees a long run of human-readable bytes.

What it costs you.

The PaddedPODArray pays 127 bytes of padding per array. For a column of UInt8 with 1000 rows that is 12% overhead. For a column with a million rows it is nothing. The design assumes large columns, which is exactly what the header comment says: “Designed for a small number of large arrays (rather than a lot of small ones).”

The -1 element trick means offsets and offsets[-1] are both live data, and the invariant that offsets[-1] == 0 is maintained by a single memset at allocation time. It is checked nowhere. If a future change ever wrote to c_start - ELEMENT_SIZE, every string in the column would silently shift.

The reserve_exact policy for columns is a bet that the row count is known in advance. When it is not, the array over-allocates by up to 2×, and the ColumnAggregateFunction shrink trick exists to clean that up after the fact. The right fix is to know the size; the fallback is a copy.

The design decision that makes it work. PODArray refuses to initialize its elements, and then makes up for it with padding that guarantees the absence of initialization bugs: 63 bytes on the right so that SIMD can overrun, 64 bytes on the left with a zeroed element at index -1 so that offset arithmetic needs no special case. The result is a container where the fastest code paths — memcpy ranges, SIMD filters, unaligned vector copies — are also the simplest code paths, because the padding has already paid for the slack they need.

ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: A Variant column is a struct of arrays — one sub-column per alternative, one tag byte per row, one offset per row. The tag is a local index; a separate global↔local mapping lets the type grow at runtime without rewriting a single row.

A Variant column is a tagged union: each row holds a value of one of several types, and the column itself knows the full list of alternatives. Dynamic is the same idea with an open-ended list — the set of types is discovered from the data.

The obvious implementation is a struct-of-arrays: one sub-column per alternative, each holding only the rows that belong to it, plus a tag per row saying which sub-column to look in. That’s what ClickHouse does.

What makes the implementation interesting isn’t the tag. It’s the set of awkward consequences that fall out of it:

The order of the alternatives must not matter (Variant(String, UInt64) and Variant(UInt64, String) are the same type), but the storage of a discriminator is a small integer, so the order does matter on disk.
The type must be extendable at runtime without rewriting the data, which means a column’s internal ordering may differ from the type’s ordering.
The number of alternatives must be bounded, but a Dynamic column can’t know its alternatives in advance, so it needs an escape hatch.
Reading a row must not require a virtual call and a branch on the tag.

Let’s look at how ClickHouse 26.9 solves all four.

The layout

The header comment of ColumnVariant is a complete specification, worked example included:

 cpp
 
 
// src/Columns/ColumnVariant.h:12
/**
 * Column for storing Variant(...) type values.
 * Variant type represents a union of other data types.
 * For example, type Variant(T1, T2, ..., TN) means that each row of this type
 * has a value of either type T1 or T2 or ... or TN or none of them (NULL value)
 *
 * ColumnVariant stores:
 * - The discriminators column, which determines which variant is stored in each row.
 * - The offsets column, which determines the offset in the corresponding variant column in each row.
 * - The list of variant columns with only real values (so the sizes of variant columns can be different).
 * Discriminator is an index of a variant in the variants list, it also has special value called NULL_DISCRIMINATOR
 * that indicates that the value in the row is NULL.
 *
 * We want to be able to extend Variant column for free without rewriting the data, but as we don't care about the
 * order of variants during Variant creation (we want Variant(T1, T2) to be the same as Variant(T2, T1)), we support
 * some global order of nested types inside Variant during type creation, so after extension the order of variant types
 * (and so their discriminators) can change. For example: Variant(T1, T3) -> Variant(T1, T2, T3).
 * To avoid full rewrite of discriminators column on Variant extension, we differentiate local order of variants
 * inside a column and global order of variants created during type creation. So, ColumnVariant stores only local
 * discriminators and additionally stores the mapping between global and local discriminators.
 * So, when we need to extend Variant column with new variant, we can just append it to a list of variant columns
 * with new local discriminator and update mapping from global to local orders.
 *
 * Note that two instances of ColumnVariant can have different local orders, so we should always use global
 * discriminators during inter-column interactions.
 *
 * Let's take an example with type Variant(UInt32, String, Array(UInt32)):
 * During type creation we will sort types by their names and get the global order: Array(UInt32), String, UInt32.
 * So, type Array(UInt32) will have global discriminator 0, String - 1 and UInt32 - 2.
 * Let's say we have a column with local order (String, UInt32, Array(UInt32)) and values:
 * 'Hello', 42, NULL, 'World', 43, [1, 2, 3], NULL, 44
 *
 * Let's see how these values will be stored in ColumnVariant:
 *
 * local_to_global_discriminators: {0 : 1, 1 : 2, 2 : 0}
 * global_to_local_discriminators: {0 : 2, 1 : 0, 2 : 1}
 * local_discriminators offsets String UInt32 Array(UInt32))
 * 0 0 'Hello' 42 [1, 2, 3]
 * 1 0 'World' 43
 * NULL_DISCRIMINATOR 0 44
 * 0 1
 * 1 1
 * 2 0
 * NULL_DISCRIMINATOR 0
 * 1 2
 *
 */

That diagram is worth staring at, because it shows the two central ideas:

text

 row: 0 1 2 3 4 5 6 7
 local: 0 1 NULL 0 1 2 NULL 1
 offsets: 0 0 0 1 1 0 0 2

 String: 'Hello' 'World' (2 values)
 UInt32: 42 43 44 (3 values)
 Array: [1,2,3] (1 value)

The discriminator is one byte per row. It’s a local index into the column’s own list of variant columns.
The offset is a row index within that particular variant column. Row 3 has local discriminator 0 (String) and offset 1, so its value is String[1] == 'World'. Row 5 has local discriminator 2 (Array) and offset 0, so its value is Array[0].

So per row you pay one byte of tag plus one Offset (UInt64) of position. The values themselves are stored densely, one after another, inside their own sub-column — a String sub-column holds only the strings, a UInt32 sub-column holds only the integers, and neither has holes.

The types involved:

 cpp
 
 
// src/Columns/ColumnVariant.h:62
 using Discriminator = UInt8;
 using Discriminators = PaddedPODArray<Discriminator>;
 using ColumnDiscriminators = ColumnVector<Discriminator>;
 using ColumnOffsets = ColumnVector<Offset>;

 static constexpr UInt8 NULL_DISCRIMINATOR = std::numeric_limits<Discriminator>::max(); /// 255
 static constexpr size_t MAX_NESTED_COLUMNS = std::numeric_limits<Discriminator>::max(); /// 255

NULL_DISCRIMINATOR is 255 — one past the largest legal alternative index. That means a column can hold at most 255 alternatives, and 255 is reserved for NULL. The value is not stored in any sub-column; it’s only a tag.

Global vs local order

The tag stored in the column is a local discriminator. The global discriminator is the position in the type’s canonical ordering. The canonical ordering is by type name — that’s how Variant(T1, T2) and Variant(T2, T1) become the same type:

 cpp
 
 
// src/DataTypes/DataTypeVariant.cpp:46
DataTypeVariant::DataTypeVariant(const DataTypes & variants_)
{
 /// Sort nested types by their full names and squash identical types.
 std::map<String, DataTypePtr> name_to_type;
 for (const auto & type : variants_)
 {
 checkAllowedInsideVariant(type);

 /// Don't use Nothing type as a variant.
 if (!isNothing(type))
 name_to_type[type->getName()] = type;
 }

 variants.reserve(name_to_type.size());
 for (const auto & [_, type] : name_to_type)
 variants.push_back(type);

 checkVariantsNotEmptyAndNotTooMany(variants);
}

A std::map keyed by the type name gives the sorted, de-duplicated order for free.

Now, why do we need two orders at all? Because a Dynamic column is created with one set of alternatives and then extended as new types arrive. When Variant(T1, T3) becomes Variant(T1, T2, T3), the canonical (global) order of T3 may change — T2 might sort before it. If the column stored global discriminators, every stored tag would have to be rewritten.

The fix is to keep the column’s local order stable and just maintain a mapping:

 cpp
 
 
// src/Columns/ColumnVariant.h:160
 Discriminator ALWAYS_INLINE globalDiscriminatorByLocal(Discriminator local_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return local_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : local_to_global_discriminators[local_discr];
 }

 Discriminator ALWAYS_INLINE localDiscriminatorByGlobal(Discriminator global_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return global_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : global_to_local_discriminators[global_discr];
 }

and the extension operation is a pure metadata change:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1904
void ColumnVariant::extend(const VectorWithMemoryTracking<Discriminator> & old_to_new_global_discriminators, VectorWithMemoryTracking<std::pair<MutableColumnPtr, Discriminator>> && new_variants_and_discriminators)
{
 /// Update global discriminators for current variants.
 for (Discriminator & global_discr : local_to_global_discriminators)
 global_discr = old_to_new_global_discriminators[global_discr];

 /// Add new variants.
 variants.reserve(variants.size() + new_variants_and_discriminators.size());
 local_to_global_discriminators.reserve(local_to_global_discriminators.size() + new_variants_and_discriminators.size());
 for (auto & new_variant_and_discriminator : new_variants_and_discriminators)
 {
 variants.emplace_back(std::move(new_variant_and_discriminator.first));
 local_to_global_discriminators.push_back(new_variant_and_discriminator.second);
 }

 /// Update global -> local discriminators matching.
 global_to_local_discriminators.resize(local_to_global_discriminators.size());
 for (Discriminator local_discr = 0; local_discr != local_to_global_discriminators.size(); ++local_discr)
 global_to_local_discriminators[local_to_global_discriminators[local_discr]] = local_discr;
}

Nothing in that function touches the discriminator column or the offsets column. The existing rows’ local tags stay valid; only the interpretation of the tags changes. That’s the “extend Variant column for free without rewriting the data” promise, kept.

The invariant is that local discriminators are always dense: local discriminator k is the index of variants[k]. Global discriminators are sparse in the sense that a given column may not contain all of them.

Reading a row

 cpp
 
 
// src/Columns/ColumnVariant.cpp:415
Field ColumnVariant::operator[](size_t n) const
{
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return Null();
 return (*variants[discr])[offsetAt(n)];
}

Two loads and a branch. The branch on NULL_DISCRIMINATOR is unavoidable. What you want to avoid is a virtual call per row — and variants[discr] is an IColumn&, so operator[] on it is a virtual call.

That’s why the hot paths in ClickHouse don’t use operator[] but instead reach into the concrete sub-column via getDataAt, and why ColumnVariant also exposes getVariantByLocalDiscriminator so callers can hoist the type test out of the loop.

Note also that the offsets array is not a byte offset into a shared arena. It’s an element index into the sub-column. That’s what allows each sub-column to be an ordinary ClickHouse column with its own type-specific representation (a ColumnString with offsets+chars, a ColumnVector<UInt32>, a ColumnArray, …), and it’s why byteSizeAt has to consult the sub-column:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1564
size_t ColumnVariant::byteSizeAt(size_t n) const
{
 size_t res = sizeof(Offset) + sizeof(Discriminator);
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return res;

 return res + variants[discr]->byteSizeAt(offsetAt(n));
}

So a Variant row costs 9 bytes of overhead (1 discriminator + 8 offset) plus the payload. The offsets are the expensive part, and that’s deliberate: an offset into a typed sub-column is what lets the sub-column be stored in its native, compressible layout.

Building the offsets

The offsets column is derived from the discriminators. constructOffsetsFromDiscriminators computes it, and it has three special cases:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:69
void ColumnVariant::constructOffsetsFromDiscriminators()
{
 const ColumnDiscriminators * discriminators_concrete = typeid_cast<const ColumnDiscriminators *>(local_discriminators.get());
 Offsets & offsets_data = typeid_cast<ColumnOffsets *>(offsets.get())->getData();
 offsets_data.clear();
 offsets_data.reserve(discriminators_concrete->size());
 /// If we have only NULLs, offsets column will not contain any real offsets.
 if (hasOnlyNulls())
 {
 /// Offsets of NULL rows are never read for their value, but compression reads the whole buffer.
 offsets_data.resize_fill(discriminators_concrete->size(), 0);
 }
 /// If we have only one non empty variant and no NULLs,
 /// offsets column will contain just sequential offsets 0, 1, 2, ...
 else if (getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 for (size_t i = 0; i != discriminators_concrete->size(); ++i)
 offsets_data.push_back(i);
 }
 /// Otherwise we should iterate through discriminators and
 /// remember current offset for each variant column.
 else
 {
 VectorWithMemoryTracking<Offset> nested_offsets;
 nested_offsets.resize(variants.size());
 for (Discriminator discr : discriminators_concrete->getData())
 {
 if (discr == NULL_DISCRIMINATOR)
 {
 offsets_data.emplace_back();
 }
 else
 {
 offsets_data.push_back(nested_offsets[discr]);
 ++nested_offsets[discr];
 }
 }
 }
}

The general case is a running counter per variant — a “rank within the sub-column” computation.

The special cases matter for the serializer. If every row is NULL, the offsets buffer is all zeros (and the comment notes that compression reads the whole buffer, so it can’t be empty). If there’s exactly one non-empty variant, the offsets are simply 0, 1, 2, ... and the whole offsets column is unnecessary information.

Insertion keeps the two arrays in sync

The discriminator and the offsets column must always have the same length. Because an allocation failure in the middle of an insert could leave them inconsistent, the code is careful:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:737
void ColumnVariant::insertDefault()
{
 getLocalDiscriminators().push_back(NULL_DISCRIMINATOR);
 /// Keep local_discriminators and offsets in sync even if appending to offsets throws
 /// (e.g. on a memory limit), otherwise popBack would over-pop the shorter one.
 try
 {
 getOffsets().emplace_back();
 }
 catch (...)
 {
 getLocalDiscriminators().popBack();
 throw;
 }
}

That’s exception-safety hygiene you’ll see repeatedly in this code base: the two columns are a matched pair, and one of them can throw (memory allocation) while the other can’t.

The insert path

Inserting a value from another variant column is where the global/local mapping earns its keep:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:501
void ColumnVariant::insertFromImpl(const DB::IColumn & src_, size_t n, const VectorWithMemoryTracking<ColumnVariant::Discriminator> * global_discriminators_mapping)
{
 const size_t num_variants = variants.size();
 const ColumnVariant & src = assert_cast<const ColumnVariant &>(src_);

 if (!global_discriminators_mapping && src.variants.size() != num_variants)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot insert value of Variant type with different number of types");

 Discriminator src_global_discr = src.globalDiscriminatorAt(n);
 Discriminator global_discr = src_global_discr;
 if (global_discriminators_mapping && src_global_discr != NULL_DISCRIMINATOR)
 global_discr = (*global_discriminators_mapping)[src_global_discr];
 Discriminator local_discr = localDiscriminatorByGlobal(global_discr);
 getLocalDiscriminators().push_back(local_discr);
 if (local_discr == NULL_DISCRIMINATOR)
 {
 getOffsets().emplace_back();
 }
 else
 {
 getOffsets().push_back(variants[local_discr]->size());
 variants[local_discr]->insertFrom(src.getVariantByGlobalDiscriminator(src_global_discr), src.offsetAt(n));
 }
}

Read it as a three-step translation: source local tag → source global tag → destination local tag. The source’s local tag means nothing to the destination; the global tag is the portable currency; the destination’s local tag is what gets stored.

Serialization: compact discriminators

On disk, the discriminator stream has two modes:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.h:54
 struct DiscriminatorsSerializationMode
 {
 enum Value
 {
 BASIC = 0, /// Store the whole discriminators column.
 COMPACT = 1, /// Don't write discriminators in granule if all of them are the same.
 };

In BASIC mode the whole discriminator column is written. In COMPACT mode a granule whose discriminators are all the same writes one discriminator plus a marker byte instead of 65 536 of them. The writer has to detect those three situations (uniform non-empty, all NULL, mixed) and there’s a fast path for each:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:343
 /// Write number of rows in this granule in compact mode.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 writeVarUInt(UInt64(limit), *discriminators_stream);

 /// If column has only one none empty discriminators and no NULLs we don't need to
 /// calculate limits for variants and use provided offset/limit.
 if (auto non_empty_local_discr = col.getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 auto non_empty_global_discr = col.globalDiscriminatorByLocal(*non_empty_local_discr);

 /// In compact mode write the format of the granule and single non-empty discriminator.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 {
 writeBinaryLittleEndian(UInt8(CompactDiscriminatorsGranuleFormat::COMPACT), *discriminators_stream);
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }
 /// For basic mode just serialize this discriminator limit times.
 else
 {
 for (size_t i = 0; i < limit; ++i)
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }

The “one non-empty variant” fast path also lets the value streams be serialized with the same offset/limit as the whole column, because in that case the sub-column has exactly the same number of rows as the variant column:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:370
 /// We can use the same offset/limit as for whole Variant column
 if (i == non_empty_global_discr)
 {
 const auto & variant_column = col.getVariantByGlobalDiscriminator(i);
 if (variant_column.size() < offset + limit)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Variant {} has less rows ({}) than expected rows to serialize ({})", variant_names[i], variant_column.size(), offset + limit);

 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(variant_column, offset, limit, settings, variant_state->variant_states[i]);
 }
 else
 {
 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(col.getVariantByGlobalDiscriminator(i), col.getVariantByGlobalDiscriminator(i).size(), 0, settings, variant_state->variant_states[i]);
 }

In the general case each sub-column is serialized with its own length, which is the whole point of the dense layout: the String sub-column is written as a String column with only the strings in it, and so it compresses as well as a plain string column would.

Dynamic: the open-ended case

Dynamic is a Variant whose alternative list is discovered at runtime. It’s implemented as a ColumnVariant plus a VariantInfo record:

 cpp
 
 
// src/Columns/ColumnDynamic.h:15
/**
 * Column for storing Dynamic type values.
 * Dynamic column allows to insert and store values of any data types inside.
 * Inside it stores:
 * - Variant column with all inserted values of different types.
 * - Information about currently stored variants.
 *
 * When new values are inserted into Dynamic column, the internal Variant
 * type and column are extended if the inserted value has new type.
 * When the limit on number of dynamic types is exceeded, all values
 * with new types are inserted into special shared variant with type String
 * that contains values and their types in binary format.
 */
class ColumnDynamic final : public COWHelper<IColumnHelper<ColumnDynamic>, ColumnDynamic>
{
public:
 /// Maximum limit on dynamic types. We use ColumnVariant to store all the types,
 /// so the limit cannot be greater then ColumnVariant::MAX_NESTED_COLUMNS.
 /// We also always have reserved variant for shared variant.
 static constexpr size_t MAX_DYNAMIC_TYPES_LIMIT = ColumnVariant::MAX_NESTED_COLUMNS - 1;
 static constexpr const char * SHARED_VARIANT_TYPE_NAME = "SharedVariant";

The escape hatch is a shared variant: one of the alternatives is always a String column, and it holds the type name and the binary-serialized value of anything that didn’t fit into a real alternative.

It has to be a String with a custom name, because a plain String alternative might legitimately be in the same column:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:71
/// Shared variant will contain String values but we cannot use usual String type
/// because we can have regular variant with type String.
/// To solve it, we use String type with custom name for shared variant.
const DataTypePtr & ColumnDynamic::getSharedVariantDataType()
{
 static thread_local DataTypePtr type = DataTypeFactory::instance().getCustom("String", std::make_unique<DataTypeCustomDesc>(std::make_unique<DataTypeCustomFixedName>(getSharedVariantTypeName())));
 return type;
}

and a fresh Dynamic column starts life with only the shared variant:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:80
ColumnDynamic::ColumnDynamic(size_t max_dynamic_types_) : max_dynamic_types(max_dynamic_types_), global_max_dynamic_types(max_dynamic_types)
{
 /// Create Variant with shared variant.
 setVariantType(std::make_shared<DataTypeVariant>(DataTypes{getSharedVariantDataType()}));
}

The insert path

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:257
void ColumnDynamic::insert(const Field & x)
{
 if (x.isNull())
 {
 insertDefault();
 return;
 }

 auto & variant_col = getVariantColumn();
 auto shared_variant_discr = getSharedVariantDiscriminator();
 /// Check if we can insert field into existing variants and avoid Variant extension.
 for (ColumnVariant::Discriminator i = 0; i != variant_col.getNumVariants(); ++i)
 {
 if (i != shared_variant_discr && variant_col.getVariantByGlobalDiscriminator(i).tryInsert(x))
 {
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(i));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(i).size() - 1);
 return;
 }
 }

 /// If we cannot insert field into current variant column, extend it with new variant for this field from its type.
 /// Use LeastSupertypeOnError::Dynamic so that arrays with incompatible element types (e.g. ["text", {"k":1}])
 /// are typed as Array(Dynamic) rather than throwing NO_COMMON_TYPE. Dynamic can hold any element value.
 auto field_data_type = applyVisitor(FieldToDataType<LeastSupertypeOnError::Dynamic>(), x);
 auto field_data_type_name = field_data_type->getName();
 if (addNewVariant(field_data_type, field_data_type_name))
 {
 /// Insert this field into newly added variant.
 auto discr = variant_info.variant_name_to_discriminator[field_data_type_name];
 variant_col.getVariantByGlobalDiscriminator(discr).insert(x);
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(discr));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(discr).size() - 1);
 }
 else
 {
 /// We reached maximum number of variants and couldn't add new variant.
 /// In this case we add the value of this new variant into special shared variant.
 /// We store values in shared variant in binary form with binary encoded type.
 auto & shared_variant = getSharedVariant();
 auto & chars = shared_variant.getChars();
 {
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());
 }
 shared_variant.getOffsets().push_back(chars.size());
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(shared_variant_discr));
 variant_col.getOffsets().push_back(shared_variant.size() - 1);
 }
}

Three outcomes, in order:

The value fits an existing alternative — use it, no extension.
The value has a new type and there’s room — extend the variant list and add a real alternative.
The value has a new type and the list is full — encode the type and the value into the shared variant’s character buffer.

Case 3 is the interesting one. The shared variant is a ColumnString, so writing into it means writing into its chars buffer directly. The type is encoded first, then the value:

 cpp
 
 
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());

and reading it back is the mirror image:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:323
void ColumnDynamic::get(size_t n, Field & res) const
{
 /// Object, Dynamic and Variant values nest into each other to a depth that comes from the data, not from the declared type.
 checkStackSize();

 const auto & variant_col = getVariantColumn();
 /// Check if value is not in shared variant.
 if (variant_col.globalDiscriminatorAt(n) != getSharedVariantDiscriminator())
 {
 variant_col.get(n, res);
 if (!res.isNull())
 res = convertFieldToType(res, *getTypeAt(n));
 return;
 }

 /// We should deserialize value from shared variant.
 const auto & shared_variant = getSharedVariant();
 auto value_data = shared_variant.getDataAt(variant_col.offsetAt(n));
 ReadBufferFromMemory buf(value_data);
 auto type = decodeDataType(buf);
 type->getDefaultSerialization()->deserializeBinary(res, buf, getFormatSettings());
}

Note checkStackSize(). Because Dynamic can nest inside Dynamic inside Dynamic, the depth is a function of the data, and a malicious input could otherwise blow the stack. That’s a real concern for a type whose shape is discovered at runtime, and it shows up in the code as an explicit guard.

Adding a variant

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:145
bool ColumnDynamic::addNewVariant(const DataTypePtr & new_variant, const String & new_variant_name)
{
 /// Check if we already have such variant.
 if (variant_info.variant_name_to_discriminator.contains(new_variant_name))
 return true;

 /// Check if we reached maximum number of variants.
 if (!canAddNewVariant())
 {
 /// Dynamic column should always have shared variant.
 if (!variant_info.variant_name_to_discriminator.contains(getSharedVariantTypeName()))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Maximum number of variants reached, but no shared variant exists");

 return false;
 }

 const DataTypes & current_variants = assert_cast<const DataTypeVariant &>(*variant_info.variant_type).getVariants();
 DataTypes all_variants = current_variants;
 all_variants.push_back(new_variant);
 auto new_variant_type = std::make_shared<DataTypeVariant>(all_variants);
 updateVariantInfoAndExpandVariantColumn(new_variant_type);
 return true;
}

The new type is appended to the type, and then the column is expanded to match. updateVariantInfoAndExpandVariantColumn is where ColumnVariant::extend gets called with the old-to-new global discriminator mapping.

Seeing it in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `payload` Dynamic
)
ENGINE = MergeTree
ORDER BY ts;

INSERT INTO events VALUES
 ('2026-01-01 00:00:00', 42),
 ('2026-01-01 00:00:01', 'hello'),
 ('2026-01-01 00:00:02', [1, 2, 3]),
 ('2026-01-01 00:00:03', NULL);

-- The type of each value is remembered.
SELECT payload, dynamicType(payload) FROM events;

The dynamicType function is the user-visible face of the discriminator: it reads the tag and maps it to the type name.

Trade-offs

What the design buys you.

Each alternative is stored in its native layout, so a String alternative compresses like a string column, an UInt64 alternative compresses like a UInt64 column, and the whole thing benefits from per-column codecs. A Dynamic column that only ever saw strings is physically almost identical to a LowCardinality(String) or a plain String column — one byte of tag per row plus the payloads.

The global/local split means adding a type to a Dynamic column is O(1) in the number of rows. Nothing is rewritten; only two small mapping arrays change length.

What it costs you.

Nine bytes per row of pure overhead (one discriminator, one UInt64 offset), versus zero for a plain column. For a Variant with two alternatives and short values, that overhead can exceed the payload.

The offsets are the expensive part, and they’re the price of the dense sub-column layout. A “one big union buffer with a tag” design would pay one byte per row but would force every alternative into a common representation, losing per-type compression and per-type codecs.

The escape hatch is a real cost. Once a Dynamic column overflows into the shared variant, those values become opaque byte strings with an embedded type name. Reading one back requires decoding the type and then running the right deserializer — a dynamic dispatch on data, with a checkStackSize() guard because the nesting depth is attacker-controlled. And the shared variant is a String column, so it compresses like text even when it holds binary payloads.

The complexity is concentrated in one place. The mapping from global to local discriminator is the only genuinely subtle piece, and it exists purely so that Variant(T1, T2) and Variant(T2, T1) can be the same type while keeping the on-disk tags stable across extensions. Everything else — the offsets, the compact serialization, the NULL discriminator — is a fairly direct consequence of “one sub-column per alternative, one tag per row”.

Wrapping up

Worth internalizing: the local/global split is the trick that makes Dynamic viable. Store global tags and every extension rewrites the column. Store local tags and extension is a metadata edit.

What to go do about it: if you use Dynamic, watch how many distinct types actually show up in your data. Once a column overflows into the shared variant, you lose per-type compression and pay a runtime type dispatch on every read. If you know the shape of your data, declare a Variant with the alternatives you need — or better, a plain column — and you keep the per-type codecs and pay zero bytes of tag.

And if you’re benchmarking, remember the overhead is 9 bytes per row. For short values, that’s not a rounding error — it’s the whole payload.

ClickHouse's Query Result Cache: A Hash of the AST, Not the Text

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse caches query results by hashing the parsed query, not the text you typed, plus the settings and database that could change the answer. Everything else - TTLs instead of invalidation, per-user quotas, an optional on-disk tier - follows from that one decision.

Every database has this feature and most of them get it half-right. The idea is old and simple: run the same query twice, serve the second one from memory.

The hard part was never the storage. It was the key.

Textual equality is hopeless. The same query can be spelled with different whitespace, different letter case, a different current database, or a different value of a setting that changes the answer. Meanwhile two queries that look different on the surface may produce identical rows, and you would love them to share an entry.

ClickHouse’s answer: hash the parsed query. That decision ripples through the whole design - the key construction, the TTL cache, the writer’s long list of reasons to refuse an entry, the reader’s access checks, and the on-disk backend that parks entries in a filesystem cache.

Let’s walk it end to end.

Why caching results is hard to get right

The header of the cache class states the contract plainly:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:48
/// Maps queries to query results. Useful to avoid repeated query calculation.
///
/// The cache does not aim to be transactionally consistent (which is difficult to get right). For example, the cache is not invalidated
/// when data is inserted/deleted into/from tables referenced by queries in the cache. In such situations, incorrect results may be
/// returned. In order to still obtain sufficiently up-to-date query results, a expiry time (TTL) must be specified for each cache entry
/// after which it becomes stale and is ignored. Stale entries are removed opportunistically from the cache, they are only evicted when a
/// new entry is inserted and the cache has insufficient capacity.

Read that again, because it is the whole correctness story: TTL, not invalidation. A result is valid for as long as its TTL says, and the entry is not invalidated when the underlying table changes.

This is a deliberate trade. Tracking table versions per entry would make every cache entry depend on the write path - and the write path is exactly the hot, latency-sensitive thing you do not want to slow down for the read path’s convenience.

The key: a hash of the AST, not of the text

The hashed part of the key is an IASTHash - a 128-bit hash of the parsed query:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:59
 struct Key
 {
 /// ----------------------------------------------------
 /// The actual key (data which gets hashed):

 /// The hash of the query AST.
 /// Unlike the query string, the AST is agnostic to lower/upper case (SELECT vs. select).
 IASTHash ast_hash;

Notice the comment. SELECT and select produce the same AST, so they produce the same hash, so they share one cache entry. That is the whole trick, and it falls out for free.

The AST is not hashed as-is, though. Two normalisations happen first.

Query-cache settings are stripped. A query that says SETTINGS use_query_cache = true should hit the entry written by the same query without that clause:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:500
ASTPtr removeQueryResultCacheSettings(ASTPtr ast)
{
 RemoveQueryResultCacheSettingsMatcher::Data visitor_data;
 RemoveQueryResultCacheSettingsVisitor(visitor_data).visit(ast);

 return ast;
}

And the predicate that decides which settings get pruned is deliberately generous about output-format settings, since those change how the answer is rendered, not what it is:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:413
bool settingDoesNotAffectQueryResultCache(const String & setting_name)
{
 return setting_name == "log_comment"
 /// As of today, the output format settings only affect the final output.
 /// However, it should be taken with caution - we should not use these settings in deterministic SQL functions.
 || setting_name.starts_with("output_format_")
 /// This setting is used to tune the server response, but does not affect the query behavior.
 /// An example why it should not affect query caching:
 /// - if you run a query as usual, and then run the same query with asking the server
 /// for Content-Disposition: attachment to download the result.
 || setting_name == "http_response_headers";
}

That last comment is a great one. Ask the server for the same rows as a download and you still want the same cached rows. Output format is not part of the result.

Planner-generated table aliases are removed for subqueries, so two identical subqueries that ended up under different plans share one entry. The pattern is matched exactly, because a user identifier that happens to look like __table1 must not be rewritten:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:513
bool isPlannerGeneratedTableAlias(std::string_view name)
{
 constexpr std::string_view prefix = "__table";
 if (!name.starts_with(prefix))
 return false;
 auto suffix = name.substr(prefix.size());
 if (suffix.empty())
 return false;
 for (char c : suffix)
 {
 if (c < '0' || c > '9')
 return false;
 }
 return true;
}

Strictly __table plus digits. No regex looseness, no user input sneaking through. This is the kind of detail that bites you years later if you get it wrong, and it took about four lines to get right.

What else goes into the hash

After the AST, the hash takes the current database and the changed settings:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:578
IASTHash calculateASTHash(ASTPtr ast, const String & current_database, const Settings & settings, const bool is_subquery, const bool pre_cleaned = false)
{
 if (!pre_cleaned)
 {
 ast = ast->clone();
 if (is_subquery)
 ast = removeTableAliases(ast);
 }
 ast = removeQueryResultCacheSettings(ast);

 /// Hash the AST, we must consider aliases (issue #56258)
 SipHash hash;
 ast->updateTreeHash(hash, /*ignore_aliases=*/ false);

 /// Also hash the database specified via SQL `USE db`, otherwise identifiers in same query (AST) may mean different columns in different
 /// tables (issue #64136)
 hash.update(current_database);

The current_database line is worth pausing on. An unqualified column name means different things in different databases, so the database is part of the identity of the answer. Hashing the AST alone would have been wrong here.

Then the settings, sorted, because Settings::changes() returns them in random order and a hash must be order-independent:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:596
 /// Finally, hash the (changed) settings as they might affect the query result (e.g. think of settings `additional_table_filters` and `limit`).
 /// Note: allChanged() returns the settings in random order. Also, update()-s of the composite hash must be done in deterministic order.
 /// Therefore, collect and sort the settings first, then hash them.
 auto changed_settings = settings.changes();
 std::vector<std::pair<String, String>> changed_settings_sorted; /// (name, value)
 for (const auto & change : changed_settings)
 {
 const String & name = change.name;
 if (!isSettingIgnoredInQueryResultCache(name)) /// see removeQueryResultCacheSettings() and isSubquerySpecificSetting() for why this is a good idea
 changed_settings_sorted.push_back({name, Settings::valueToStringUtil(change.name, change.value)});
 }

Only changed settings are hashed - a setting nobody touched cannot change the result relative to the default.

But there is a subtlety for subqueries, because the planner forcibly sets extremes, max_result_bytes and max_result_rows for them. Left alone, a subquery’s key would differ from the same query run at top level, and the cache would never hit:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:608
 /// The Planner forcibly sets `extremes`, `max_result_bytes`, `max_result_rows` for subqueries, which makes them
 /// appear in `settings.changes()`. Non-subquery executions of the same AST typically don't have these in their
 /// changed settings. To ensure cache hits between subquery writes and subquery reads, normalize by always
 /// including the default values of these settings for subqueries.
 if (is_subquery)
 {
 if (!changed_settings.tryGet("extremes"))
 changed_settings_sorted.push_back({"extremes", settings[Setting::extremes].toString()});

The equality of two keys

Now note what is not in the equality check. Two keys are equal if the AST hashes match and they are both subqueries or both not:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:715
bool QueryResultCache::Key::operator==(const Key & other) const
{
 return ast_hash == other.ast_hash && is_subquery == other.is_subquery;
}

The user id and the roles are not part of the key. They get checked at read time instead, which is why one entry per query is enough to still enforce access control. More on that below.

The hasher is trivial, because the AST hash is already a good hash:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:720
size_t QueryResultCache::KeyHasher::operator()(const Key & key) const
{
 SipHash hash;
 hash.update(key.ast_hash.low64);
 hash.update(key.is_subquery);
 return hash.get64();
}

What may be cached at all

Not every query may be cached. A query with a non-deterministic function (now(), rand()) would return a different answer the second time. The check walks the AST and asks the function factory:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:105
 if (const auto func = FunctionFactory::instance().tryGet(function->name, data.context))
 {
 if (!func->isDeterministic())
 data.has_non_deterministic_functions = true;
 return;
 }

Two escape hatches get special care.

The eval table function hides its real query inside an opaque string, so ClickHouse cannot inspect it. What it cannot see, it must not trust:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:96
 /// The `eval` table function hides its real query inside an opaque string argument, so the
 /// generated query cannot be inspected here. Treat it as non-deterministic (and, in
 /// HasSystemTablesMatcher, as touching a system table) to keep such queries out of the cache,
 /// e.g. `eval('SELECT now()')` must not be cached as if it were deterministic.
 if (function->name == "eval")
 {
 data.has_non_deterministic_functions = true;
 return;
 }

And SQL UDFs, whose determinism ClickHouse simply does not know:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:111
 if (const auto udf_sql = UserDefinedSQLFunctionFactory::instance().tryGet(function->name))
 {
 /// Driver-created executable functions are also persisted in the SQL-object storage,
 /// but their determinism is described by the generated executable UDF configuration checked below.
 if (!udf_sql->as<ASTCreateFunctionWithDriverQuery>())
 {
 /// ClickHouse currently doesn't know if SQL-based UDFs are deterministic or not. We must assume they are non-deterministic.
 data.has_non_deterministic_functions = true;
 return;
 }
 }

The obfuscate table function is subtler still: it is deterministic if a seed is given, and the seed can be overridden by a SETTINGS clause on any enclosing query. So the check has to descend into the AST while tracking the effective settings:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:309
 if (const auto * function = node->as<ASTFunction>())
 if (function->name == "obfuscate" && context->getSettingsRef()[Setting::obfuscate_seed].value.empty())
 return true;

What the user wants to happen on a violation is a setting, with three values (throw, save, ignore):

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:371
 const QueryResultCacheNondeterministicFunctionHandling nondeterministic_function_handling
 = settings[Setting::query_cache_nondeterministic_function_handling];
 const QueryResultCacheSystemTableHandling system_table_handling = settings[Setting::query_cache_system_table_handling];

 if (ast_contains_nondeterministic_functions && nondeterministic_function_handling == QueryResultCacheNondeterministicFunctionHandling::Throw)
 throw Exception(ErrorCodes::QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS,
 "The query result was not cached because the query contains a non-deterministic function."
 " Use setting `query_cache_nondeterministic_function_handling = 'save'` or `= 'ignore'` to cache the query result regardless, or omit caching");

So:

 sql
 
 
-- A non-deterministic query is refused by default:
SELECT * FROM events WHERE ts > now() - INTERVAL 1 HOUR;
-- ERROR: The query result was not cached because the query contains a non-deterministic function.

-- ... unless you insist:
SELECT * FROM events WHERE ts > now() - INTERVAL 1 HOUR
SETTINGS query_cache_nondeterministic_function_handling = 'save';

Gotcha: the default is throw, and the error message arrives after the query has already run. You paid for the computation, then threw the result away.

The cache itself

The cache is a TTL cache with per-user quotas and its own metrics:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1180
QueryResultCache::QueryResultCache(size_t max_size_in_bytes, size_t max_entries, size_t max_entry_size_in_bytes_, size_t max_entry_size_in_rows_)
 : cache(std::make_unique<TTLCachePolicy<Key, Entry, KeyHasher, EntryWeight, IsStale>>(
 CurrentMetrics::QueryCacheBytes, CurrentMetrics::QueryCacheEntries, std::make_unique<PerUserTTLCachePolicyUserQuota>()))
{
 updateConfiguration(max_size_in_bytes, max_entries, max_entry_size_in_bytes_, max_entry_size_in_rows_);
}

Staleness is a pure function of the key:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:738
bool QueryResultCache::IsStale::operator()(const Key & key) const
{
 return (key.expires_at < std::chrono::system_clock::now());
};

The eviction policy is worth spelling out, because it is not an LRU. When an insert does not fit, the cache first purges the stale entries - the ones whose TTL has passed - and if that is still not enough, the insert is simply refused and the new result is dropped. The cache never evicts a fresh entry to make room for another fresh entry:

 cpp
 
 
// src/Common/TTLCachePolicy.h:237
 if (!sufficient_space_in_cache() || !sufficient_space_in_cache_for_user())
 {
 /// Remove stale entries
 for (auto it = cache.begin(); it != cache.end();)
 {
 if (is_stale_function(it->first))
 {
 size_t sz = weight_function(*it->second);
 if (it->first.user_id.has_value())
 Base::user_quotas->decreaseActual(*it->first.user_id, sz);
 it = cache.erase(it);
 size_in_bytes -= sz;
 }
 else
 ++it;
 }
 }

That’s a deliberate refusal to thrash. A cache that evicts live entries to admit fresh ones can end up with a 0% hit rate under pressure. This one keeps its hot set and simply refuses new work when full.

The weight of an entry is the number of bytes its chunks actually allocate:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:728
size_t QueryResultCache::EntryWeight::operator()(const Entry & entry) const
{
 size_t res = 0;
 for (const auto & chunk : entry.chunks)
 res += chunk.allocatedBytes();
 res += entry.totals.has_value() ? entry.totals->allocatedBytes() : 0;
 res += entry.extremes.has_value() ? entry.extremes->allocatedBytes() : 0;
 return res;
}

Per-user quotas are pushed into the cache on every write, because there is no better hook for “the settings profile of this user changed”:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1220
 /// Update the per-user cache quotas with the values stored in the query context. This happens per query which writes into the query
 /// cache. Obviously, this is overkill but I could find the good place to hook into which is called when the settings profiles in
 /// users.xml change.
 /// user_id == std::nullopt is the internal user for which no quota can be configured
 if (key.user_id.has_value())
 cache.setQuotaForUser(*key.user_id, max_query_result_cache_size_in_bytes_quota, max_query_result_cache_entries_quota);

The comment admits it: this is overkill. It works, but it is the kind of thing you notice on a Friday afternoon and want to fix. It never gets fixed.

The two backends

 text
 
 
 query AST
 | clone + strip cache settings + strip planner aliases
 | hash(AST) + hash(current_database) + hash(changed settings)
 v
 QueryResultCache::Key { ast_hash, user_id, roles, is_shared, ttl, header }
 |
 +--> in-memory TTLCachePolicy <-- Chunks (optionally compressed)
 | | weight = allocated bytes, per-user quota
 | v
 | SourceFromChunks --> the query pipeline
 |
 +--> on-disk: FileCacheKey(salted hash) --> filesystem cache
 | FixedHeader (magic, version, sizes, ttl, build fingerprint, checksum)
 | AccessMetadata (is_shared, user, roles, tag)
 | CompressedWriteBuffer + NativeWriter (header + chunks + totals + extremes)
 v
 filesystem cache (evicted by the same rules as any other data)

Writing: many reasons to refuse

The writer buffers the result chunks as they stream by and decides at the very end whether to keep them. That is deliberate - a query that dies half-way must not leave a partial entry:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:222
/// Buffers multiple partial query result chunks (buffer()) and eventually stores them as cache entry (finalizeWrite()).
///
/// Implementation note: Queries may throw exceptions during runtime, e.g. out-of-memory errors. In this case, no query result must be
/// written into the query result cache. Unfortunately, neither the Writer nor the special transform added on top of the query pipeline
/// which holds the Writer know whether they are destroyed because the query ended successfully or because of an exception (otherwise, we
/// could simply implement a check in their destructors). To handle exceptions correctly nevertheless, we do the actual insert in
/// finalizeWrite() as opposed to the Writer destructor. This function is then called only for successful queries in finish_callback() which
/// runs before the transform and the Writer are destroyed, whereas for unsuccessful queries we do nothing (the Writer is destroyed w/o
/// inserting anything).

Here’s the thing: the insert happens in finalizeWrite(), not in a destructor. Destructors cannot tell “the query finished” from “the query threw”, so the code routes the decision through a callback that only fires on success. That is a small piece of engineering taste that saves a lot of corrupted cache entries.

The list of refusal reasons in finalizeWrite is the interesting part:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:862
 if (result_is_incomplete)
 {
 LOG_TRACE(logger, "Skipped insert because the query result is incomplete (a consumer stopped reading it before the end), query: {}", doubleQuoteString(key.query_string));
 return;
 }

 if (auto query_runtime = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now() - query_start_time); query_runtime < min_query_runtime)
 {
 LOG_TRACE(logger, "Skipped insert because the query is not expensive enough, query runtime: {} msec (minimum query runtime: {} msec), query: {}",
 query_runtime.count(), min_query_runtime.count(), doubleQuoteString(key.query_string));
 return;
 }

The second one is the classic “don’t cache the cheap queries” rule. Caching a 1 ms query costs more than recomputing it. The setting is query_cache_min_query_duration.

Partial chunks are squashed into blocks of max_block_size rows before being stored, so the cached result looks like a freshly produced one:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:889
 if (squash_partial_results)
 {
 /// Squash partial result chunks to chunks of size 'max_block_size' each. This costs some performance but provides a more natural
 /// compression of neither too small nor big blocks. Also, it will look like 'max_block_size' is respected when the query result is
 /// served later on from the query result cache.

And the entries are compressed, at the cost of a decompression on read:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:969
 if (key.is_compressed)
 {
 /// Compress result chunks. Reduces the space consumption of the cache but means reading from it will be slower due to decompression.

 Chunks compressed_chunks;

 for (const auto & chunk : query_result->chunks)
 {
 const Columns & columns = chunk.getColumns();
 Columns compressed_columns;
 for (const auto & column : columns)
 {
 auto compressed_column = column->compress(/*force_compression=*/false);
 compressed_columns.push_back(compressed_column);
 }
 Chunk compressed_chunk(compressed_columns, chunk.getNumRows());
 compressed_chunks.push_back(std::move(compressed_chunk));
 }
 query_result->chunks = std::move(compressed_chunks);
 }

column->compress(force_compression=false) is the interesting bit: a column is compressed only if it is worth it. A tiny column of random UInt64s costs more CPU than it saves in RAM, so it is stored as-is.

Finally the size limits are checked against the compressed weight:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:992
 size_t new_entry_size_in_bytes = QueryResultCache::EntryWeight()(*query_result);
 size_t new_entry_size_in_rows = count_rows_in_chunks(*query_result);

 if ((new_entry_size_in_bytes > max_entry_size_in_bytes) || (new_entry_size_in_rows > max_entry_size_in_rows))
 {
 LOG_TRACE(logger, "Skipped insert because the query result is too big, query result size: {} (maximum size: {}), query result size in rows: {} (maximum size: {}), query: {}",
 formatReadableSizeWithBinarySuffix(new_entry_size_in_bytes, 0), formatReadableSizeWithBinarySuffix(max_entry_size_in_bytes, 0), new_entry_size_in_rows, max_entry_size_in_rows, doubleQuoteString(key.query_string));
 return;
 }

 cache.set(key, query_result);

Reading: access control

The reader looks the key up and then checks whether the caller may have it. The user id and the current roles live in the key but are not part of the equality check, so the same entry can be found by a different user - and then rejected:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1067
 const auto & entry_key = entry->key;
 const auto & entry_mapped = entry->mapped;

 const bool is_same_user_id = ((!entry_key.user_id.has_value() && !key.user_id.has_value()) || (entry_key.user_id.has_value() && key.user_id.has_value() && *entry_key.user_id == *key.user_id));
 const bool is_same_current_user_roles = (entry_key.current_user_roles == key.current_user_roles);
 if (!entry_key.is_shared && (!is_same_user_id || !is_same_current_user_roles))
 {
 LOG_TRACE(logger, "Inaccessible query result found for query {}", doubleQuoteString(key.query_string));
 return;
 }

The rationale for checking the roles and not just the user is in the key’s comment:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.h:78
 /// The id and current roles of the user who executed the query.
 /// These members are necessary to ensure that a (non-shared, see below) entry can only be written and read by the same user with
 /// the same roles. Example attack scenarios:
 /// - after DROP USER, it must not be possible to create a new user with with the dropped user name and access the dropped user's
 /// query result cache entries
 /// - different roles of the same user may be tied to different row-level policies. It must not be possible to switch role and
 /// access another role's cache entries
 std::optional<UUID> user_id;
 std::vector<UUID> current_user_roles;

Read that as two attack scenarios. First: drop a user, recreate someone with the same name, and try to read the old user’s cached rows. Second: one user, two roles, different row-level policies - switch role, keep the rows.

Storing roles in the key but not in the equality check is the clever part. The entry stays shareable by identity, and access control happens on the way out.

A stale entry is also refused:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCache.cpp:1078
 if (QueryResultCache::IsStale()(entry_key))
 {
 LOG_TRACE(logger, "Stale query result found for query {}", doubleQuoteString(key.query_string));
 return;
 }

And the chunks are cloned (uncompressed case) or decompressed (compressed case) before being handed to a SourceFromChunks processor, which becomes the new head of the query pipeline.

The on-disk backend

The in-memory cache is bounded by RAM. The on-disk backend stores entries in a filesystem cache - the same infrastructure used for remote table data - which means the entries can be much larger, survive restarts, and (with a distributed filesystem cache) be shared between servers.

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.h:14
/// Stores query result cache entries in a preconfigured filesystem cache (an entry of the `filesystem_caches` section of the server
/// configuration), so that cached query results get more space than in memory, survive server restarts, and (with a distributed
/// filesystem cache) can be shared between servers. The filesystem cache is selected per query with setting
/// `query_cache_on_disk_cache_name`.
///
/// The entries are ordinary filesystem cache entries, indistinguishable from other data in the same filesystem cache: they are not
/// held from deletion, they are evicted by the same rules, and there are no specific limits on their total size or number other than
/// the limits of the filesystem cache itself. The keys are hashed with a salt so that they cannot intersect with other keys in the
/// filesystem cache. All metadata needed to validate an entry (TTL, access control) is serialized into the entry itself; no in-memory
/// state is kept.

Reusing the filesystem cache is a lovely bit of leverage. You inherit its eviction, its checksumming, its remote-storage support, and its cluster plumbing, instead of inventing a parallel universe.

The key

The on-disk key is a salted hash of the AST hash, plus the access context for non-shared entries:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:136
FileCacheKey makeFileCacheKey(const QueryResultCache::Key & key, bool is_shared)
{
 /// Salt the hash so that keys of the on-disk query result cache cannot intersect with other keys in the same filesystem cache
 /// (those are typically derived from storage paths).
 SipHash hash;
 hash.update(std::string_view("QueryResultCacheOnDisk"));
 hash.update(key.ast_hash.low64);
 hash.update(key.ast_hash.high64);
 hash.update(key.is_subquery);
 hash.update(is_shared);
 if (!is_shared)
 {
 hash.update(key.user_id.has_value());
 if (key.user_id)
 hash.update(key.user_id->toUnderType());
 /// The roles are compared as an ordered sequence, same as in the in-memory query result cache.
 hash.update(key.current_user_roles.size());
 for (const auto & role : key.current_user_roles)
 hash.update(role.toUnderType());
 }

Note that unlike the in-memory cache, the on-disk key does include the user. The filesystem cache is shared storage, so one user’s entry must never shadow another’s - there is no “check on read” step in shared storage, so identity has to be baked into the key itself.

A marker in the middle of the key lets the cache find its own entries without reading anything from disk:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:103
/// Bits 64..95 of every key of the on-disk query result cache hold this marker ("QRCC"), so that the entries can be told apart from
/// the other keys of the filesystem cache by the key alone, without reading anything from disk (see `clear`). The marker sits in the
/// middle of the key on purpose: the filesystem cache derives the directory of a key from its topmost bits and the metadata bucket
/// from its lowest bits, both of which must keep varying. The remaining 96 bits of hash are plenty to keep the entries apart. A key
/// of other data carries the marker by chance with probability 2^-32; such a key is still recognized as foreign by its content.
constexpr UInt32 KEY_MARKER = 0x51524343;

That comment is a small masterpiece of constraint juggling. The marker cannot sit at the top (the directory is derived from the top bits) and cannot sit at the bottom (the metadata bucket is derived from the lowest bits), so it goes in the middle - and it admits the 2^-32 collision chance right there.

The entry format

The on-disk entry has a fixed-size header, then uncompressed access metadata, then the compressed payload:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:63
/// On-disk entry layout, version 6:
///
/// Fixed header (FIXED_HEADER_SIZE bytes):
/// char[8] magic "QRCache1"
/// UInt32 format version
/// UInt32 TCP protocol revision the Native blocks are serialized with
/// UInt64 total entry size in bytes, including the fixed header
/// UInt64 created_at, seconds since epoch
/// UInt64 expires_at, seconds since epoch
/// UInt64 fingerprint of the server build which wrote the entry (see `getServerBuildFingerprint`)
/// UInt128 SipHash-128 of the preceding fixed header bytes and of everything after the fixed header. The freshness
/// fields are authenticated as well: a corruption of `expires_at` or `total_size` alone would otherwise
/// silently change how long the entry is served, without ever failing the integrity check.
///
/// Access metadata (uncompressed):
/// UInt8 is_shared
/// UInt8 has_user_id, [UInt128 user_id]
/// varUInt number of user roles, [UInt128 role]...
/// String tag (setting `query_cache_tag`), so that `SYSTEM DROP QUERY CACHE TAG` can find the entry
///
/// Result payload (compressed with the codec from setting `query_cache_on_disk_codec`; the compression frames are
/// self-describing, so reading does not depend on the setting):
/// varUInt number of result chunks
/// Native header block (zero rows)
/// Chunk one per result chunk
/// UInt8 has_totals, [Chunk]
/// UInt8 has_extremes, [Chunk]

Two details are worth dwelling on.

The build fingerprint. An entry is only valid for the server build that wrote it, because a new build may compute a different answer for the same query:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:110
/// The entries survive restarts and can be shared between servers, but a result is only valid for the server build which computed
/// it: a newer or older build may compute a different result for the same query (fixes of functions or of the planner, changed
/// defaults), even when neither the TCP protocol revision nor the entry format changes. So every entry records the build which wrote
/// it, and an entry of another build is treated like an entry of an incompatible format: it is never served, and it is overwritten
/// on the next write.

The checksum covers the freshness fields. Otherwise a corrupted expires_at would silently change how long the entry lives - and a bit flip in the wrong byte would be invisible.

The header is patched in place after the payload is serialised, and the checksum is computed over the header and the body:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:514
 unalignedStoreLittleEndian<UInt64>(data.data() + TOTAL_SIZE_OFFSET_IN_FIXED_HEADER, data.size());

 SipHash hash;
 hash.update(data.data(), CHECKSUM_OFFSET_IN_FIXED_HEADER);
 hash.update(data.data() + FIXED_HEADER_SIZE, data.size() - FIXED_HEADER_SIZE);
 UInt128 checksum = hash.get128();
 transformEndianness<std::endian::little>(checksum);
 memcpy(data.data() + CHECKSUM_OFFSET_IN_FIXED_HEADER, &checksum, sizeof(checksum));

Probing and writing

Writing is best-effort and never throws. Before writing, the existing entry is probed to see whether it is fresh, being written concurrently, or garbage:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:365
QueryResultCacheOnDisk::ProbeResult QueryResultCacheOnDisk::probeExistingEntry(const FileCacheKey & cache_key) const
{
 const auto & user_id = FileCache::getCommonOrigin().user_id;

 auto holder = file_cache->getDownloadedContiguousOrEmpty(cache_key, 0, FIXED_HEADER_SIZE, user_id);
 for (size_t attempt = 0; holder->empty(); ++attempt)
 {
 /// No readable header: either there is no entry, or another query is writing one (its segments are held and not
 /// downloaded yet), or only the tail of a partially evicted entry is left. The tail must be removed, otherwise no
 /// entry could be created for the key until it is evicted too.
 const auto segments = file_cache->tryGetFileSegmentInfos(cache_key, user_id);
 if (segments.empty())
 return ProbeResult::None;
 for (const auto & segment : segments)
 {
 if (segment.state != FileSegment::State::DOWNLOADED || segment.references > 1)
 return ProbeResult::InProgress;
 }

The write itself reserves the space for all the segments of the entry before writing any of them, so an entry is either complete or absent:

 cpp
 
 
// src/Interpreters/Cache/QueryResultCacheOnDisk.cpp:529
 /// Create all file segments of the entry at once. This fails gracefully if another query created an entry for the same key in
 /// the meantime (then we simply skip the insert), and guarantees that we exclusively own all segments of the key while writing.
 /// A reader sees the incomplete entry as a miss because not all of its bytes are downloaded yet.
 auto holder = file_cache->trySet(cache_key, 0, data.size(), CreateFileSegmentSettings(FileSegmentKind::Regular), origin);
 if (!holder)
 {
 LOG_TRACE(logger, "Skipped insert into the on-disk query result cache because of a concurrent insert for query {}",
 doubleQuoteString(key.query_string));
 return;
 }

Two concurrent writers, one winner, no corruption. The loser just skips the insert. That is a much nicer failure mode than the usual write-write race.

The reader side is symmetric: it downloads the whole entry, verifies the checksum, and only then decompresses it.

Using it

The cache is off by default; you turn it on per query or per session:

 xml
 
 
<!-- Server configuration: a 1 GiB in-memory query result cache. -->
<query_cache>
 <max_size_in_bytes>1073741824</max_size_in_bytes>
 <max_entries>1024</max_entries>
 <max_entry_size_in_bytes>1048576</max_entry_size_in_bytes>
 <max_entry_size_in_rows>30000000</max_entry_size_in_rows>
</query_cache>

 sql
 
 
SELECT count() FROM hits WHERE UserID = 42 SETTINGS use_query_cache = true;
-- the second execution is served from the cache

The per-user quota is a setting, not a config element, so it can differ per profile: query_cache_max_size_in_bytes and query_cache_max_entries.

The observable state lives in system tables:

 sql
 
 
SELECT query_id, length, hits, QueryCacheHits, QueryCacheMisses FROM system.query_log LIMIT 10;
SELECT query, result_size, tag, stale, shared, compressed, expires_at FROM system.query_result_cache LIMIT 10;

SYSTEM CLEAR QUERY CACHE clears everything, SYSTEM CLEAR QUERY CACHE TAG '<tag>' clears one tag’s worth.

Trade-offs

What it buys: a dashboard that runs the same heavy aggregate every 10 seconds stops paying for the aggregate, and the server’s CPU goes to the queries that are actually different. If you have ever watched a Grafana dashboard redraw in lockstep with 40 open tabs asking for the same number, this is the fix.

What it costs:

Staleness is unbounded up to the TTL. The cache is not invalidated on INSERT. A TTL of 1 hour means an hour-old answer. This is the documented contract, not a bug, but it is the reason the cache is opt-in.
The key is a hash, so it inherits hash-collision risk. Two different queries with the same 128-bit AST hash would share an entry. In practice this does not happen; there is no secondary check of the query text.
Memory is the scarce resource. The in-memory cache holds result data, not plans, and the results are held compressed. A server with a 1 GiB cache can easily be asked for 10 GiB of results; the size limits (max_entry_size_in_bytes, max_entry_size_in_rows) exist to keep one huge result from evicting everything else.
Compression is a trade, not a win. The default query_cache_compress_entries makes reads slower and saves RAM; on a server that is memory-bound it is right, on one that is CPU-bound it is wrong.
The on-disk backend has its own failure modes: partially evicted entries, entries written by a different build, a filesystem cache that is not yet initialised. All of them are handled by treating the entry as a miss, and the probe-then-write dance means two concurrent writers cannot corrupt each other - one of them simply loses.

Wrapping up

The one thing to remember: the cache’s hardest problem - the key - is solved by ClickHouse’s parser rather than by the cache. Hashing the AST instead of the text means select and SELECT share an entry, and the planner’s internal alias noise never leaks into the key.

Worth internalizing: this is a cache that is allowed to be wrong, on purpose. TTLs instead of invalidation is a choice, and the price is an answer that can be up to a TTL old. Pick the TTL like you mean it.

What to go do about it: if you have a dashboard, an ORM that generates the same SQL every 30 seconds, or a report job that hammers the same aggregate - turn on use_query_cache for exactly those queries, set a short TTL, and check system.query_result_cache to see whether the hit rate justifies the memory.

ClickHouse's Query Pipeline: Ports, Status Codes and the Executing Graph

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse doesn’t call operators in a fixed order. Every processor reports a status — “I need data”, “my output is full”, “call my work()” — and the executor walks the graph, re-preparing only the neighbours that actually changed.

Every query engine eventually has to answer the same question: given a graph of operators — read these files, filter these rows, group them, sort them, write them out — in what order do you actually run them?

The textbook answer is the Volcano iterator model. Every operator exposes a next() that returns one tuple at a time, and the executor is a nest of while (child.next()) loops. It’s beautifully simple, and it’s also unbelievably slow: every tuple crosses every operator through a virtual call, and nobody ever gets to see a whole batch.

ClickHouse’s answer is the opposite extreme. Operators exchange blocks — tens of thousands of rows at a time — through ports, and the executor never calls operators in a fixed order. Instead every processor is asked, in effect, “given the state of your neighbours, what should happen next?”, and it answers with one of five status codes.

The executor is a scheduler that walks the graph, and the graph tells it where to go.

This is a tour of that machinery: the IProcessor interface, the one-slot mailbox that is a Port, the five status codes, the version counters that let a processor notice that a neighbour changed, and the ExecutingGraph that ties them together.

The building block: `IProcessor`

A processor is a node with zero or more input ports and zero or more output ports. The taxonomy in the header comment of src/Processors/IProcessor.h still describes every processor in the tree — sources, sinks, simple transformations, squashing transformations (DISTINCT, WHERE), accumulating transformations (ORDER BY, GROUP BY), and limiting transformations.

The crucial design decision is that there is no next(). A processor never asks its input for data; it declares whether it needs data, and the executor arranges for it to arrive. And a processor never decides what runs next — it returns a status, and the executor interprets it.

Here’s the entire status vocabulary:

 cpp
 
 
// src/Processors/IProcessor.h:150
 enum class Status : uint8_t
 {
 /// Processor needs some data at its inputs to proceed.
 /// You need to run another processor to generate required input and then call 'prepare' again.
 NeedData,

 /// Processor cannot proceed because output port is full or not isNeeded().
 /// You need to transfer data from output port to the input port of another processor and then call 'prepare' again.
 PortFull,

 /// All work is done (all data is processed or all output are closed), nothing more to do.
 Finished,

 /// You may call 'work' method and processor will do some work synchronously.
 Ready,

 /// You may call 'schedule' method and processor will return a descriptor.
 /// You need to poll this descriptor and call work() afterwards.
 Async,

 /// Processor wants to add new processors and/or remove finished neighbours.
 /// Update must be obtained by updatePipeline() call.
 UpdatePipeline,
 };

NeedData and PortFull are the two “I’m blocked” answers, and they’re the whole point. They tell the executor which neighbour has to run next: NeedData means “run my upstream”, PortFull means “run my downstream”. Ready means “I have work, call work() on me”. Async means “I have a file descriptor, poll it”. UpdatePipeline means “I changed the graph”.

The contract for prepare() is spelled out carefully, because it’s where most of the subtlety lives:

 cpp
 
 
// src/Processors/IProcessor.h:177
 /** Method 'prepare' is responsible for all cheap ("instantaneous": O(1) of data volume, no wait) calculations.
 *
 * It may access input and output ports,
 * indicate the need for work by another processor by returning NeedData or PortFull,
 * or indicate that processing has finished by returning `Finished`,
 * it may pull data from input ports and push data to output ports.
 *
 * The method is not thread-safe and must be called from a single thread in one moment of time,
 * even for different connected processors.
 *
 * Instead of all long work (CPU calculations or waiting) it should just prepare all required data and return Ready or Async.
 *
 * Thread safety and parallel execution:
 * - no methods (prepare, work, schedule) of single object can be executed in parallel;
 * - method 'work' can be executed in parallel for different objects, even for connected processors;
 * - method 'prepare' cannot be executed in parallel even for different objects,
 * if they are connected (including indirectly) to each other by their ports;
 */
 virtual Status prepare();

Read that last bullet twice. It’s the fundamental law of the model: work() may run in parallel with anything; prepare() may not run in parallel with anything it’s connected to.

The port connections define the happens-before edges of the whole execution, and the executor honours them by construction.

The port: a mailbox with the flags in the pointer

A port is where two processors meet. It’s shared state, and it must be readable and writable from two different threads (the producer and the consumer), so it’s an atomic pointer to a Data slot — plus three flag bits that say what the slot currently holds.

 cpp
 
 
// src/Processors/Port.h:75
 /// Flags for Port state.
 /// Will store them in least pointer bits.

 /// Port was set finished or closed.
 static constexpr std::uintptr_t IS_FINISHED = 1;
 /// Block is not needed right now, but may be will be needed later.
 /// This allows to pause calculations if we are not sure that we need more data.
 static constexpr std::uintptr_t IS_NEEDED = 2;
 /// Check if port has data.
 static constexpr std::uintptr_t HAS_DATA = 4;

 static constexpr std::uintptr_t FLAGS_MASK = IS_FINISHED | IS_NEEDED | HAS_DATA;
 static constexpr std::uintptr_t PTR_MASK = ~FLAGS_MASK;

Packing the flags into the low bits of the pointer itself is the trick. It means the entire port state — the slot pointer plus the three flags — is a single atomic word. So a push and a “set not needed” can be one CAS operation, and there’s never a window in which the flags disagree with the pointer.

The little DataPtr class exists to make that safe. The swap() method is the whole implementation of a push or a pull:

 cpp
 
 
// src/Processors/Port.h:117
 uintptr_t ALWAYS_INLINE swap(std::atomic<Data *> & value, std::uintptr_t flags, std::uintptr_t mask) /// NOLINT
 {
 Data * expected = nullptr;
 Data * desired = getPtr(flags | getUInt(data));

 while (!value.compare_exchange_weak(expected, desired))
 desired = getPtr((getUInt(expected) & FLAGS_MASK & (~mask)) | flags | getUInt(data));

 /// It's not very safe. In case of exception after exchange and before assignment we will get leak.
 /// Don't know how to make it better.
 data = getPtr(getUInt(expected) & PTR_MASK);

 return getUInt(expected) & FLAGS_MASK;
 }

The mask argument says which flags the caller intends to change and which ones it wants to preserve from whatever is currently in the word. The CAS loop re-computes desired from expected on every failure, so concurrent flag updates by the two sides are merged rather than lost.

push() and pull() are then thin wrappers:

 cpp
 
 
// src/Processors/Port.h:154
 void ALWAYS_INLINE push(DataPtr & data_, std::uintptr_t & flags)
 {
 flags = data_.swap(data, HAS_DATA, HAS_DATA);

 /// It's possible to push data into finished port. Will just ignore it.
 /// if (flags & IS_FINISHED)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to finished port.");

 /// It's possible to push data into port which is not needed now.
 /// if ((flags & IS_NEEDED) == 0)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which is not needed.");

 if (unlikely(flags & HAS_DATA))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which already has data");
 }

 void ALWAYS_INLINE pull(DataPtr & data_, std::uintptr_t & flags, bool set_not_needed = false)
 {
 uintptr_t mask = HAS_DATA;

 if (set_not_needed)
 mask |= IS_NEEDED;

 flags = data_.swap(data, 0, mask);

 /// It's ok to check because this flag can be changed only by pulling thread.
 if (unlikely((flags & IS_NEEDED) == 0) && !set_not_needed)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which is not needed");

 if (unlikely((flags & HAS_DATA) == 0))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which has no data");
 }

Note the semantics of IS_NEEDED. It’s not “the consumer wants data”; it’s “the consumer might want data later”.

A producer may push into a port that’s currently IS_NEEDED == 0 — the data simply sits in the slot — but it may not pull from a port that’s not needed, because pulling implies “give me the next thing” and the consumer has explicitly said it doesn’t want more right now.

canPush() is the producer’s gate:

 cpp
 
 
// src/Processors/Port.h:478
 bool ALWAYS_INLINE canPush() const
 {
 assumeConnected();
 auto flags = state->getFlags();
 return (flags & State::IS_NEEDED) && ((flags & State::HAS_DATA) == 0);
 }

Both conditions matter. If the consumer said “not needed”, the producer must wait; and if the slot already has data, the producer must wait, because the port holds exactly one chunk.

This “not needed” handshake is what makes the pull side of the model honest. The consumer says input.setNotNeeded() while it’s busy, the producer’s canPush() goes false, and the producer stops.

That’s how a LIMIT transform can stop a system.numbers_mt source from producing infinity. It closes the port.

 sql
 
 
-- The Limit transform closes its input port once the limit is reached,
-- so the source stops generating rows instead of the query reading forever.
SELECT * FROM system.numbers_mt WHERE number = 1000000 LIMIT 1;

The shape of the resulting graph is visible with EXPLAIN PIPELINE, which prints the processors and the ports the executor will walk:

 sql
 
 
EXPLAIN PIPELINE
SELECT number FROM numbers(10) WHERE number > 5 LIMIT 2;

`prepare()` and `work()`

The canonical implementation of the two-phase protocol is ISimpleTransform, the base class of nearly every operator in the tree (filters, expressions, DISTINCT, window functions, LIMIT BY…).

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:23
ISimpleTransform::Status ISimpleTransform::prepare()
{
 /// Check can output.

 if (output.isFinished())
 {
 input.close();
 onFinish();
 return Status::Finished;
 }

 if (!output.canPush())
 {
 input.setNotNeeded();
 return Status::PortFull;
 }

 /// Output if has data.
 if (has_output)
 {
 output.pushData(std::move(output_data));
 has_output = false;

 if (!no_more_data_needed)
 return Status::PortFull;

 }

 /// Stop if don't need more data.
 if (no_more_data_needed)
 {
 input.close();
 output.finish();
 return Status::Finished;
 }

 /// Check can input.
 if (!has_input)
 {
 if (input.isFinished())
 {
 output.finish();
 onFinish();
 return Status::Finished;
 }

 input.setNeeded();

 if (!input.hasData())
 return Status::NeedData;

 input_data = input.pullData(set_input_not_needed_after_read);
 has_input = true;

 if (input_data.exception)
 /// No more data needed. Exception will be thrown (or swallowed) later.
 input.setNotNeeded();
 }

 /// Now transform.
 return Status::Ready;
}

Every branch here is a decision about who runs next:

 text
 
 
 output.isFinished() -> input.close(); output.finish() -> Finished
 !output.canPush() -> input.setNotNeeded() -> PortFull
 has_output -> output.pushData() -> PortFull
 no_more_data_needed -> input.close(); output.finish() -> Finished
 input.isFinished() -> output.finish() -> Finished
 !input.hasData() -> input.setNeeded() -> NeedData
 otherwise -> pullData(); transform() -> Ready

Notice that prepare() does push: output.pushData(...) and input.pullData(...) both happen inside prepare(). The division of labour isn’t “prepare moves no data”, it’s “prepare does no expensive work”.

Pulling a chunk is O(1) — it’s a pointer swap. Actually computing on it is not, and that’s what work() is for:

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:86
void ISimpleTransform::work()
{
 if (input_data.exception)
 {
 transform(input_data.exception);

 output_data = std::move(input_data);
 has_input = false;
 has_output = true;
 return;
 }

 try
 {
 transform(input_data.chunk, output_data.chunk);
 }
 catch (DB::Exception &)
 {
 output_data.exception = std::current_exception();
 has_output = true;
 has_input = false;
 return;
 }

 has_input = !needInputData();

 if (!skip_empty_chunks || output_data.chunk)
 has_output = true;

 if (has_output && !output_data.chunk && !getOutputPort().getHeader().empty())
 /// Support invariant that chunks must have the same number of columns as header.
 output_data.chunk = Chunk(getOutputPort().getHeader().cloneEmpty().getColumns(), 0);
}

Three things are worth noting.

Exceptions are data. A failure in the upstream processor is pushed into the port as an exception_ptr alongside the chunk, not thrown across the thread boundary. work() picks it up, hands it to transform(), and pushes it on. The exception then travels downstream with the stream, which means the error surfaces in the right place and every processor upstream of the failure gets to clean up.

work() cannot block. That’s why the Status::Async case exists: a processor that needs to wait for a socket or a file descriptor returns Async from prepare() and hands out an fd from schedule().

The empty-chunk invariant. Every chunk that crosses a port must have exactly as many columns as the port’s header, even if it has zero rows. ISimpleTransform enforces it at the bottom of work() by synthesising an empty chunk with the right shape.

Version counters: how a processor learns that a neighbour changed

The executor must know which processors to wake up. Polling every processor in the graph after every chunk would be O(V) per chunk, which for a twenty-node graph and a million chunks is a lot of pointless work.

ClickHouse’s answer is a version counter per connection, with a twist: the counter lives in the UpdateInfo of the port, and it’s compared against the value the neighbour last saw.

 cpp
 
 
// src/Processors/Port.h:34
 struct UpdateInfo
 {
 using UpdateList = VectorWithMemoryTracking<void *>;

 UpdateList * update_list = nullptr;
 void * id = nullptr;
 UInt64 version = 0;
 UInt64 prev_version = 0;

 void inline ALWAYS_INLINE update()
 {
 if (version == prev_version && update_list)
 update_list->push_back(id);

 ++version;
 }

 void inline ALWAYS_INLINE trigger() { prev_version = version; }
 };

update() is called whenever the port’s state changes (a push, a pull, a setNeeded). It bumps version, and — this is the interesting part — if version == prev_version, meaning nobody has consumed the change yet, it appends the port’s identity to a list of “ports that changed”. trigger() marks the change as seen by setting prev_version = version.

So the protocol is: a port that changes while nobody was looking records itself on a list; the executor drains that list and re-prepares exactly those neighbours. If a port changes twice before the neighbour is re-prepared, the neighbour is only told once. It doesn’t need to know how many times it changed, only that it did.

This is the mechanism that replaced the old “wake up the neighbour immediately” design, and it’s why prepare() can be called on a subset of the graph instead of on all of it.

The `ExecutingGraph`: nodes, edges, and who owns what

The executor’s view of the pipeline is ExecutingGraph, in src/Processors/Executors/Runtime/V1/. It holds one Node per processor and two kinds of Edge per connection: a direct edge from an output port to the consuming processor’s input port, and a backward edge from an input port back to the producing processor’s output port.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:31
 /// Edge represents connection between OutputPort and InputPort.
 /// For every connection, two edges are created: direct and backward (it is specified by backward flag).
 struct Edge
 {
 Edge(Node * to_, bool backward_,
 InputPort * input_port_, OutputPort * output_port_,
 VectorWithMemoryTracking<void *> * update_list)
 : to(to_), backward(backward_)
 , input_port(input_port_), output_port(output_port_)
 {
 update_info.update_list = update_list;
 update_info.id = this;
 }

 /// Node this edge points to.
 /// It is the node with output_port for direct edge or the node with input_port for backward edge.
 Node * to = nullptr;
 bool backward;
 /// Ports of this connection. Both pointers are set regardless of direction:
 /// for a direct edge (out -> in) input_port belongs to `to`; for a back edge the reverse.
 InputPort * input_port = nullptr;
 OutputPort * output_port = nullptr;

 /// Edge version is increased when port's state is changed (e.g. when data is pushed). See Port.h for details.
 /// To compare version with prev_version we can decide if neighbour processor need to be prepared.
 Port::UpdateInfo update_info;
 };

The Node carries a status with a very specific ownership protocol:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:62
 /// Status for processor.
 /// Can be owning or not. Owning means that executor who set this status can change node's data and nobody else can.
 enum class ExecStatus : uint8_t
 {
 Idle, /// prepare returned NeedData or PortFull. Non-owning.
 Preparing, /// some executor is preparing processor, or processor is in task_queue. Owning.
 Executing, /// prepare returned Ready or Async and task is executing. Owning.
 Finished, /// prepare returned Finished. Non-owning.
 };

“Owning” is the key word. Two threads may never call prepare() on the same processor, and may never call prepare() on connected processors at the same time. The status is how the graph enforces that: a node in Preparing or Executing is owned by exactly one thread, and the per-node std::mutex status_mutex is taken when ownership is transferred.

The traversal itself is updateNode(). It starts from one processor and spreads along the edges, re-preparing whoever was affected:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:524
ExecutingGraph::UpdateNodeStatus ExecutingGraph::updateNode(IProcessor & initial, Queue & queue, Queue & async_queue, UpdateNodeBuffers & buffers)
{
 Processors delayed_destruction;
 auto & updated_edges = buffers.updated_edges;
 auto & updated_processors = buffers.updated_processors;
 auto & pending_expansion = buffers.pending_expansion;

 /// The buffers may be left non-empty by a previous call that exited early or threw.
 buffers.reset();

 std::shared_lock read_lock(nodes_mutex);

 /// Traversal starts from the node related to updated processor
 updated_processors.push_back(processors_map.at(&initial));

 while (!updated_processors.empty() || !updated_edges.empty())
 {
 std::optional<std::unique_lock<std::mutex>> stack_top_lock;

 if (updated_processors.empty())
 {
 auto * edge = updated_edges.front();
 updated_edges.pop_front();

 /// Here we have ownership on edge, but node can be concurrently accessed.

 auto & node = *edge->to;

 std::unique_lock lock(node.status_mutex);

 ExecutingGraph::ExecStatus status = node.status;

 if (status != ExecutingGraph::ExecStatus::Finished)
 {
 if (edge->backward)
 node.updated_output_ports.push_back(edge->output_port);
 else
 node.updated_input_ports.push_back(edge->input_port);

 if (status == ExecutingGraph::ExecStatus::Idle)
 {
 node.status = ExecutingGraph::ExecStatus::Preparing;
 updated_processors.push_front(edge->to);
 stack_top_lock = std::move(lock);
 }
 else
 edge->to->processor()->onUpdatePorts();
 }
 }

Two work lists are at work here, and the distinction is worth spelling out. updated_processors is a queue of nodes to call prepare() on. updated_edges is a queue of connections whose port state changed and whose far side therefore needs to know.

The loop interleaves them. An edge whose far side is Idle promotes that node to Preparing and puts it on the processor queue. If the far side is already Preparing or Executing, the edge is simply recorded in that node’s updated_input_ports / updated_output_ports list, and the node will pick it up on its next prepare().

That’s the “which processor am I at” trick in its modern form: there is no cursor. There’s no “current processor” that remembers where it was. Instead, each processor holds the set of ports that changed since it was last prepared, and the graph walks edges until it reaches a fixed point.

The core of the traversal is the prepare() call and the dispatch on its result:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:621
 switch (*node.last_processor_status)
 {
 case IProcessor::Status::NeedData:
 case IProcessor::Status::PortFull:
 {
 node.status = ExecutingGraph::ExecStatus::Idle;
 if (spill_state.memory_reservation)
 {
 /// A request may have arrived while this node was owned by normal execution.
 for (auto * spillable : processor.getSpillables())
 if (spill_state.scheduleSpill(node, spillable, queue))
 break;
 }
 break;
 }
 case IProcessor::Status::Finished:
 {
 node.status = ExecutingGraph::ExecStatus::Finished;
 if (last_status != IProcessor::Status::Finished)
 spill_state.unregisterSpillables(node);
 accountFinishedProcessorInGroup(*node.processor_iter);
 break;
 }
 case IProcessor::Status::Ready:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 queue.push(node.processor());
 break;
 }
 case IProcessor::Status::Async:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 async_queue.push(node.processor());
 break;
 }
 case IProcessor::Status::UpdatePipeline:
 {
 need_update_pipeline = true;
 pending_expansion.push_back(current);
 break;
 }
 }

Ready puts the processor on the work queue, to be picked up by a worker thread. NeedData/PortFull put the node back to Idle, meaning “somebody else must move first”. Finished retires the node for good. And then the edges that this processor disturbed get re-armed:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:666
 /// If you wonder why edges are pushed in reverse order,
 /// it is because updated_edges is a stack, and we prefer to get from stack
 /// input ports firstly, and then outputs, both in-order.
 ///
 /// Actually, there should be no difference in which order we process edges.
 /// However, some tests are sensitive to it (e.g. something like SELECT 1 UNION ALL 2).
 /// Let's not break this behaviour so far.

 for (auto it = node.post_updated_output_ports.rbegin(); it != node.post_updated_output_ports.rend(); ++it)
 {
 auto * edge = static_cast<ExecutingGraph::Edge *>(*it);
 updated_edges.push_front(edge);
 edge->update_info.trigger();
 }

edge->update_info.trigger() is the prev_version = version from Port.h, and it’s called after the edge is queued. The ordering matters: trigger() says “the neighbour has now seen this change”, and the queue entry is what makes it true.

The graph can change shape at runtime

The last status, UpdatePipeline, is unusual: a processor can, while running, decide that it needs more processors.

The classic case is external sort. MergeSortingTransform starts as a single-input, single-output processor, and when it decides to spill to disk it must insert a BufferingToFileSink and a BufferingFromFileSource into the middle of its own pipeline.

The mechanism is updatePipeline(), which returns a PipelineUpdate:

 cpp
 
 
// src/Processors/IProcessor.h:262
 struct PipelineUpdate
 {
 Processors to_add;
 Processors to_remove;
 };
 virtual PipelineUpdate updatePipeline();

The graph handles the insertion by rebuilding the edges of every node and scheduling the affected ones for re-preparation. The interesting part is the error handling, which encodes a hard-won lesson about allocation failures:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:324
 try
 {
 return updatePipelineImpl(stack, cur_node, update);
 }
 catch (...)
 {
 expansion_failed = true;
 /// The ports of the pipeline's processors point at the new processors, and the pipeline is read after the graph is gone.
 std::lock_guard guard(processors_mutex);
 for (auto it = update.to_add.begin(); it != update.to_add.end();)
 {
 auto next = std::next(it);
 if (std::ranges::find(*processors, *it) == processors->end())
 processors->splice(processors->end(), update.to_add, it);
 it = next;
 }
 throw;
 }

The processor has already connected its ports to the new processors before updatePipeline() is even called. So if the graph then fails to record them — say, addNode throws MEMORY_LIMIT_EXCEEDED — the ports point at processors the graph doesn’t know about, and the graph is permanently inconsistent.

The handler keeps the new processors alive (so the ports stay valid) and sets expansion_failed, which makes every later expansion bail out instead of walking into the inconsistency.

Trade-offs

What this buys you.

The model is fully data-parallel by construction. Any node of the graph may be scheduled on any thread, because the only shared state between processors is a port, and a port is a single atomic word. There’s no notion of “this operator belongs to this thread”, which is what lets ClickHouse run a SELECT over a MergeTree with 16 streams on 16 cores without any of the 16 streams knowing about the other 15.

The graph also reshapes itself. External sort, GLOBAL JOIN, subquery materialisation, and LIMIT pushdown all insert or remove processors while the query is running, and none of them require a separate “compile the plan” phase.

Finally, the split between prepare() (cheap, may not run concurrently with connected peers) and work() (expensive, may run concurrently with everything) is precisely the split that lets the executor overlap computation without any locking on the hot path.

What it costs.

prepare() cannot run in parallel with connected processors. For a wide graph — say a UNION ALL of twenty subqueries, each with a dozen operators — this means the scheduling phase is effectively serialised per connection. That’s why the pipeline shape matters so much: every extra operator in a chain adds a serialisation point between the parallel parts.

It’s also why Pipe::resize() has a comment about splitting the resize processor when multiple threads are allocated:

 cpp
 
 
// src/QueryPipeline/Pipe.cpp:781
 /// Performance bottleneck identified: Severe lock contention for ExecutingGraph::Node::status_mutex.
 /// Issues observed:
 /// 1. Increased latency of ExecutingGraph::updateNode, lengthening overall query latency.
 /// 2. Unnecessary CPU cycle consumption in native_queued_spin_lock_slowpath (kernel).
 /// 3. Decreased CPU utilization.
 ///
 /// Proposed solution: Split ResizeProcessor when multiple threads are allocated to execute a query.

The second cost is the per-node status_mutex. Every edge traversal takes it. On a hot path with many small chunks this is measurable, and the comment above is the record of someone measuring it.

The third cost is that the model is hard to reason about in the presence of prepare() side effects. ISimpleTransform::prepare() pushes data. LimitTransform::prepare() closes ports. SortingTransform::prepare() can return UpdatePipeline and rebuild the graph. There’s no single “execute this operator” entry point — which is exactly why ClickHouse’s test infrastructure is built around assembling a whole Pipe.

Wrapping up

The real win: ClickHouse’s pipeline doesn’t have a “current operator”. It has a set of statuses and a graph that knows how to resolve them. That single design choice buys data parallelism for free, because the only thing two processors share is an atomic word.

Two things to internalize:

If you’re debugging a stuck ClickHouse query, look at the graph, not the SQL. EXPLAIN PIPELINE shows you the processors and the ports; the status codes tell you who’s blocked and on whom.
The model is deliberately push-driven at the ports. IS_NEEDED and HAS_DATA are the only two things that matter, and every producer must respect them or you get “Cannot push block to port which already has data”.

So next time you add a custom operator, remember: your prepare() must be O(1), your work() must not block, and your exceptions must travel as data — not as thrown errors.

ClickHouse's Query Condition Cache: Skip Granules You Don't Need to Read

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse caches, per (table, part, predicate), a bitmap of granules that a filter proved empty. Repeated queries skip those granules entirely — no index analysis, no reading, no decompression. The cache only records negative results (“this granule matched nothing”), never positive ones, and it salts the key with settings, time zones, and skip-index profiles so a wrong verdict can never leak between queries.

Index analysis is not free. Every time a query runs, ClickHouse has to figure out which granules of which parts could possibly contain a matching row. That means evaluating the predicate against the primary key index, the minmax indexes, the skip indexes, and — in the hard cases — a recursive exclusion search over the marks.

Now think about a real workload: a monitoring dashboard that runs the same query every 30 seconds, a service filtering on tenant_id = 42 over and over, a batch job re-reading yesterday’s data hourly. The predicate is the same every time. The data isn’t — but the data is also mostly not matching.

That’s a textbook case for memoization, and ClickHouse implements it as the QueryConditionCache: a global LRU that remembers, for each (table UUID, part name, predicate hash) triple, a bitmap of the granules the predicate emptied during the last scan.

What the cache actually stores

The class doc is unusually explicit:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:26
/// An implementation of predicate caching a la https://doi.org/10.1145/3626246.3653395
///
/// Given the table, part name and a hash of a predicate as key, caches which marks definitely don't match the predicate and which marks may
/// match the predicate. This allows to skip the scan if the same predicate is evaluated on the same data again. Note that this doesn't work
/// the other way round: we can't tell if _all_ rows in the mark match the predicate.
class QueryConditionCache
{
public:
 /// False means none of the rows in the mark match the predicate. We can skip such marks.
 /// True means at least one row in the mark matches the predicate. We need to read such marks.
 using MatchingMarks = std::vector<bool>;

Two things to internalize:

It’s one-directional. The cache records “this granule matched nothing.” It never records “this granule matched everything” — because a scan can’t establish that. If a granule produced rows, the cache can’t tell whether all of them matched.

It’s allowed to be imprecise — in one direction only. The bitmap’s true means “may match.” A granule that matched one row in a million is marked true — the whole granule has to be read. The cache never produces a false negative, which is the only thing that matters for correctness.

MatchingMarks is a std::vector<bool> — a bit-packed boolean array, one bit per granule. A part with 100,000 granules costs 12.5 KB per predicate.

The whole mechanism is controlled by use_query_condition_cache, which defaults to true.

The key: three components, hashed

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:45
private:
 /// A hash of the table id, part name and condition id.
 using Key = UInt128;

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:237
QueryConditionCache::Key QueryConditionCache::makeKey(const UUID & table_id, const String & part_name, UInt64 condition_hash)
{
 SipHash hash;
 hash.update(table_id);
 hash.update(part_name);
 hash.update(condition_hash);
 return hash.get128();
}

The key is the part, so a cached verdict about one part never leaks into another. The part name is qualified by the table UUID, so two tables with the same part name (very common in MergeTree — part names come from block ranges and partition values) don’t collide.

There’s a wrinkle for file-backed engines, where “part” is a file:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.h:93
 static String makeFilePartName(const String & path, std::string_view version_token);

This folds in a content-version token (ETag for remote objects, mtime+inode+size for local files) so an in-place rewrite of the file yields a different key rather than a stale hit. For a file-backed table, the “part” can change under you, and a stale “this file matches nothing” verdict would be a correctness bug.

The safety property: everything starts as “may match”

This is the detail that makes the whole thing safe. When an entry is created, every granule is marked true — “may match” — so a fresh entry skips nothing:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:396
QueryConditionCache::Entry::Entry(size_t mark_count)
 : matching_marks(mark_count, true)
{
}

The header explains why that default is a correctness requirement, not a performance one: if a query is cancelled mid-scan and the cache had been updated optimistically (“these granules match nothing”), future scans would skip granules the predicate would in fact have matched — wrong results, silently, forever.

By initializing everything to “may match” and only ever clearing bits after the scan has proven the granule empty, a crash can only lose information (the cache knows less than it could), never gain it.

Writing: only clear bits, never set them

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:272
void QueryConditionCache::write(
 const UUID & table_id, const String & part_name, UInt64 condition_hash, const String & condition,
 const MarkRanges & mark_ranges, size_t marks_count, bool has_final_mark)
{
 if (table_id == UUIDHelpers::Nil)
 return;

 Key key = makeKey(table_id, part_name, condition_hash);

 auto load_func = [&](){ return std::make_shared<Entry>(marks_count); };

 auto [entry, inserted] = cache.getOrSet(key, load_func);

getOrSet is atomic: two concurrent scans of the same part with the same predicate can’t create two entries; they share one.

Then the update — note the direction:

 cpp
 
 
// src/Interpreters/Cache/QueryConditionCache.cpp:311
 {
 std::lock_guard lock(entry->mutex);

 for (const auto & mark_range : mark_ranges)
 std::fill(entry->matching_marks.begin() + mark_range.begin, entry->matching_marks.begin() + mark_range.end, false);

 if (has_final_mark)
 entry->matching_marks[marks_count - 1] = false;
 }

std::fill(..., false) — the entry is only ever narrowed. The mark ranges passed in are the ones the scan proved empty. Nothing is ever set back to true, because “this granule matched something” isn’t a fact the cache can rely on: a later scan of the same part with the same predicate might read a different set of marks (another query’s cached verdict pruned some!), and those un-read marks must stay “may match.”

That last point is subtle and important: the cache is a cache of negative results only, and the negative results are monotone. Once a granule is known empty for a predicate, it stays known empty for as long as the entry lives.

There’s also a fast path that avoids taking the exclusive lock when there’s nothing to do — most writes are no-ops because the same query run repeatedly finds the same granules empty and the bits are already clear.

Why the entry has its own mutex

The CacheBase lock protects the map — the mapping from key to entry. It does not protect the contents of the entry. And the contents are mutated concurrently: a query with 16 streams reads 16 disjoint mark ranges of the same part simultaneously, and each writes back the ranges it proved empty.

Without a per-entry lock, two threads could be doing std::fill(begin, mid, false) and std::fill(mid, end, false) simultaneously on the same std::vector<bool>. std::vector<bool> packs eight booleans into a byte, so the two fills could land in the same byte and lose one another’s writes — a lost update, which means a granule proven empty stays “may match.” Not a correctness bug (only a lost optimization), but a real one.

The SharedMutex lets the many readers of a cached verdict proceed in parallel while the rare writer excludes them.

Reading: turning a bitmap back into mark ranges

The consult side is in MergeTreeDataSelectExecutor::filterPartsByQueryConditionCache. After the cache lookup, the bitmap is converted back into MarkRanges — the surviving granules:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1944
 for (const auto & mark_range : part_with_ranges.ranges)
 {
 size_t begin = mark_range.begin;
 for (size_t mark_it = begin; mark_it < mark_range.end;)
 {
 if (!matching_marks[mark_it])
 {
 // ... collapse consecutive zeros into skip ranges ...
 }
 else
 ++mark_it;
 }
 }

0 means “skip this granule,” 1 means “must read it.” Leading zeros are skipped. A gap of zeros between two ones is dropped — unless the gap is smaller than min_marks_for_seek, in which case reading through it is cheaper than seeking past it.

And if every granule of the part was eliminated, the part is dropped from the query entirely:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1994
 if (ranges.empty())
 it = parts_with_ranges.erase(it);

A part the cached verdict proves empty is not opened at all. No marks read, no decompression, no filters evaluated.

The predicate hash is not just the predicate

The most delicate part of the design is the key. If two different predicates hashed to the same value, one query’s verdict would be applied to another query’s data — silently returning wrong results.

The base hash is the ActionsDAG hash of the filter expression, covering function names, result types, children, and constant values. But that’s not enough:

 sql
 
SELECT * FROM t WHERE ts >= now() - INTERVAL 10 DAY;

The DAG hash changes every time the query runs (the constant changes), so this predicate never gets a cache hit. Fine — but there’s a subtler problem in the other direction: a setting can change a function’s result without changing the DAG. formatDateTime with different flags produces different strings from the same DAG. least/greatest propagate NULLs differently. h3ToGeo swaps its arguments.

So the cache key includes a salt computed from every setting a function might capture at build time — settings like session_timezone, formatdatetime_f_prints_single_zero, least_greatest_legacy_null_behavior, h3togeo_lon_lat_result_order, and dozens more.

Two sessions in different time zones produce the same DAG for toDateTime(s), but parse s differently. Without the salt, one session’s verdict would be served to the other. This is the kind of bug that’s nearly impossible to find by testing and completely obvious once you’ve read the rule.

The time-condition trick

now() in a predicate is a special case, handled with a derived condition rather than a salt.

The problem: WHERE ts >= now() - INTERVAL 10 DAY has a different DAG hash every second, so it never caches. But it’s a very common predicate.

The solution: round the time constant onto a grid. On the write side, the condition is strengthened; on the read side, it’s weakened:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/updateQueryConditionCache.cpp:107
 /// Non-deterministic conditions must not be cached under their own hash.
 /// For conditions non-deterministic only because of folded current-time constants,
 /// a deterministic condition can be derived by rounding the constants onto a time grid.

Read the logic carefully, because the asymmetry is the whole trick:

Write side (strengthen): the actual filter used now() - 10 days. The derived condition is a rounded version, e.g. rounded up to the next grid point — a narrower condition. If the narrow condition matched no rows in this granule, the wider (actual) condition certainly matched no rows either. So the entry (written under the derived hash) is sound.
Read side (weaken): the query probes with the rounded-down version, which is wider. “The wider condition matched no rows” implies “the narrower condition matched no rows” — so skipping is sound.

The two coincide for grid-aligned constants and are one grid cell apart otherwise. The grid factor is a setting (query_condition_cache_time_condition_grid_factor).

This is genuinely elegant reasoning: it takes a predicate that’s not cacheable (its value changes every second) and makes it cacheable by proving that a slightly different, grid-aligned predicate implies the right thing in both directions.

The skip-index profile: another salt

Some cached verdicts come from skip index analysis — “the minmax index on this column says no row in this granule can match.” But skip index analysis depends on which indexes exist and are enabled. If a query runs with use_skip_indexes = 0 and another runs with them on, the verdicts differ.

So skip-index verdicts are stored under a profiled key — the condition hash salted with the profile of the index set that produced the verdict:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1771
 /// Each condition is looked up under two keys:
 /// * the bare condition hash, holding row-level exclusions (always sound);
 /// * a hash salted with the effective skip-index profile (skip-index exclusions).

A mark may be skipped iff either verdict says it doesn’t match — the merge is an AND of the two “may match” bitmaps:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1881
 matching_marks[i] = matching_marks[i] && (*marks_opt)[i];

AND of the “may match” bits: a granule is skipped only if both verdicts agree it cannot match. Conservative in the right direction.

Where the “empty granule” fact comes from

The write side is the read pipeline itself. When the PREWHERE filter processes a granule and the filter removes every row of it, the read task records the granule as “unmatched”:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeReadTask.cpp:590
void MergeTreeReadTask::addPrewhereUnmatchedMarks(const MarkRanges & mark_ranges_)
{
 prewhere_unmatched_marks.insert(prewhere_unmatched_marks.end(), mark_ranges_.begin(), mark_ranges_.end());
}

And the write happens when the task is finished:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSelectProcessor.cpp:617
 if (reader_settings.use_query_condition_cache && task && prewhere_info
 && !task->readersChainCanSkipMarksBeforePrewhere(prewhere_filters_by_top_k_threshold)
 && !task->appliesMutationsBeforePrewhere()
 && !row_level_filter
 && task->getInfo().data_part_info->getDataPart())

Every one of those four extra conditions is a soundness condition:

!task->readersChainCanSkipMarksBeforePrewhere(...) — if a skip index upstream of PREWHERE could have filtered marks before PREWHERE saw them, attributing them to the PREWHERE predicate would be a lie.
!task->appliesMutationsBeforePrewhere() — on-the-fly mutations filter rows ahead of PREWHERE, for the same reason.
!row_level_filter — a row-level security filter is prepended before PREWHERE, but the cache key covers only the query’s PREWHERE hash. A mark hidden by a row policy must not be attributed to the PREWHERE predicate.

That last one is a beautiful example of a cache whose key doesn’t cover everything that influenced the result — the fix is to not write in that case, rather than to make the key bigger.

The whole flow

 text
 
 
 query with predicate Q
 │
 ▼
 hash(Q) + settings salt + [skip-index profile]
 │
 ├──► QueryConditionCache::read(table_uuid, part, hash)
 │ │
 │ └──► bitmap of "may match" granules
 │ │
 │ ▼
 │ surviving MarkRanges per part
 │ (parts with no surviving ranges are dropped entirely)
 │
 ▼
 index analysis (primary key, minmax, skip indexes)
 │
 ▼
 scan the surviving granules
 │
 ├── granules where the filter removed everything ──► cache.write(cleared bits)
 │
 └── rows ──► result

What this buys you, and what it costs

It buys you the elimination of repeated index analysis. The expensive part of “which granules can match” isn’t the bitmap lookup — it’s the KeyCondition evaluation and the exclusion search. On a warm cache, that work is replaced by a hash lookup and a bit scan.

It buys you the elimination of reading granules that match nothing. The index analysis can only prove a granule is empty using the index; a predicate on a non-key column can’t be resolved from the index at all. But once a scan has observed that a granule contains no matching rows, that fact is cached, and the next run doesn’t read it.

It costs you memory. One bit per granule per (table, part, predicate). The cache is a CacheBase LRU with a max_size_bytes.

It costs you correctness vigilance. Everything in this design is about not serving a verdict computed under different circumstances: a different predicate, a different set of skip indexes, a different time zone, a different row-level policy. The salt functions, the profiles, the derived time conditions, and the !row_level_filter guard are all the same idea applied four times.

It gives up on one direction. The cache can’t record “this granule matches everything,” so a highly selective predicate matching a few rows in every granule buys nothing. The cache is for predicates that are granule-level selective — which is most of them, since a granule is 10 MB of data.

Seeing it in action

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN ('QueryConditionCacheHits', 'QueryConditionCacheMisses');

A high, stable hit/miss ratio is the signature of a workload with repeated predicates over stable data. A ratio near zero means the predicates are always new — dashboards with per-request filters, for instance — and the cache is pure overhead.

The cache is also inspectable:

 sql
 
 
SELECT *
FROM system.query_condition_cache
LIMIT 10;

And to see it working, run the same query twice and compare the SelectedMarks and SelectedRanges counters in system.query_log:

 sql
 
 
SELECT
 query_id,
 ProfileEvents['SelectedMarks'] AS marks,
 ProfileEvents['SelectedRanges'] AS ranges,
 ProfileEvents['QueryConditionCacheHits'] AS qcc_hits
FROM system.query_log
WHERE query LIKE '%tenant_id = 42%'
ORDER BY event_time;

The second run should show dramatically fewer selected marks — and, if the predicate empties whole granules, the same query won’t even open the part.

ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: LowCardinality(T) stores a dictionary of distinct values once and keeps a 1–8 byte index per row. The dictionary is shared across every block, granule and part, so twelve distinct values cost one byte per row instead of ten. It pays off below ~10 000 distinct values and hurts above ~100 000.

Every column-oriented database eventually meets the same data: a column of country codes, of status strings, of user agents, of enum-like labels. A hundred thousand rows, twelve distinct values, repeated over and over.

Storing the value in every row is absurd. Storing the position of the value in every row is not.

ClickHouse’s answer is a column type called LowCardinality(T), and the implementation is one of the more interesting pieces of the whole storage engine, because it has to solve three separate problems at once:

how to represent the dictionary and the index in memory,
how to make the same dictionary serve many different blocks, granules and parts, so the dictionary is stored once and not once per block,
how to serialize that arrangement on disk so a reader can rebuild it — including the case where the dictionary had to change halfway through.

Let’s take all three. Source quoted from ClickHouse 26.9 (04eac2ea3).

The shape of the thing

The header comment of ColumnLowCardinality states the design in four lines:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:17
/**
 * How data is stored (in a nutshell):
 * we have a dictionary @e reverse_index in ColumnUnique that holds pairs (DataType, UIntXX) and a column
 * with UIntXX holding actual data indices.
 * To obtain the value's index, call #getOrFindIndex.
 * To operate on the data (so called indices column), call #getIndexes.
 *
 * @note The indices column always contains the default value (empty std::string_view) with the first index.
 */

The class holds exactly two members:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:398
 Dictionary dictionary;
 ColumnIndex idx;

Dictionary is a thin wrapper around a ColumnUnique plus a shared flag:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:366
private:
 class Dictionary
 {
 public:
 Dictionary(const Dictionary & other) = default;
 explicit Dictionary(MutableColumnPtr && column_unique, bool is_shared);
 explicit Dictionary(ColumnPtr column_unique, bool is_shared);

 const WrappedPtr & getColumnUniquePtr() const { return column_unique; }
 WrappedPtr & getColumnUniquePtr() { return column_unique; }

 const IColumnUnique & getColumnUnique() const { return static_cast<const IColumnUnique &>(*column_unique); }
 IColumnUnique & getColumnUnique() { return static_cast<IColumnUnique &>(*column_unique); }

 /// Dictionary may be shared for several mutable columns.
 /// Immutable columns may have the same column unique, which isn't necessarily shared dictionary.
 void setShared(const ColumnPtr & column_unique_);
 bool isShared() const { return shared; }

 /// Create new dictionary with only keys that are mentioned in indexes.
 void compact(MutableColumnPtr & indexes);
 /// Create nullable dictionary with only keys that are mentioned in indexes.
 void compactToNullable(MutableColumnPtr & indexes);

 static MutableColumnPtr compact(const IColumnUnique & column_unique, MutableColumnPtr & indexes);
 static MutableColumnPtr compactToNullable(const IColumnUnique & column_unique, MutableColumnPtr & indexes);

 private:
 WrappedPtr column_unique;
 bool shared = false;
 };

So a LowCardinality column is:

text

 +---------------------------+
 row 0 -----> | dictionary[ idx[0] ] | idx is a ColumnVector<UInt8|UInt16|UInt32|UInt64>
 row 1 -----> | dictionary[ idx[1] ] | dictionary is a ColumnUnique<ColumnString> (etc.)
 row 2 -----> | dictionary[ idx[2] ] |
 ... +---------------------------+

The values are in the dictionary; the rows are indices. Getting row n is two loads: getIndexes().getUInt(n), then a lookup in the dictionary.

That second load is a random access into a (hopefully tiny) array — and the whole point is that the dictionary array is tiny enough to stay in L1/L2 while the index array streams past it.

The index column grows its own type

The index column is not fixed-width. It starts as UInt8 and widens to UInt16, UInt32, UInt64 as the dictionary outgrows the range. The widening logic lives in ColumnIndex:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:135
void ColumnIndex::expandType()
{
 auto expand = [&](auto type)
 {
 using CurIndexType = decltype(type);
 constexpr auto next_size = NumberTraits::nextSize(sizeof(CurIndexType));
 if (next_size == sizeof(CurIndexType))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't expand indexes type for ColumnLowCardinality from type: {}",
 demangle(typeid(CurIndexType).name()));

 using NewIndexType = typename NumberTraits::Construct<false, false, next_size>::Type;
 convertIndexes<NewIndexType>();
 };

 callForType(std::move(expand), size_of_type);
}

and the bound that decides when to widen:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:152
size_t ColumnIndex::getMaxIndexForCurrentType() const
{
 chassert(size_of_type == 1 || size_of_type == 2 || size_of_type == 4 || size_of_type == 8);
 /// Shifting by 64 is undefined, and this is called per inserted value, so no `callForType` here.
 if (size_of_type == sizeof(UInt64))
 return std::numeric_limits<UInt64>::max();
 return (1ULL << (8 * size_of_type)) - 1;
}

The per-value insert path is where the widening happens, and note the comment about avoiding a typeid_cast on the hot path:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:198
void ColumnIndex::insertIndex(size_t index)
{
 while (index > getMaxIndexForCurrentType())
 expandType();

 auto insert = [&]<typename CurIndexType>(CurIndexType /*type_value*/)
 {
 /// `size_of_type` tracks the type of `indexes` and `expandType` keeps the two in sync, so
 /// this is `static_cast` in release. `getIndexesData` would run a `typeid_cast` instead, and
 /// this is the per-value path.
 assert_cast<ColumnVector<CurIndexType> *>(indexes.get())->getData().push_back(static_cast<CurIndexType>(index));
 };

 callForType(std::move(insert), size_of_type);

 /// Same invariant, so re-deriving it from `indexes` is a debug-only check here.
 chassert(size_of_type == getSizeOfIndexType(*indexes, size_of_type));
}

callForType is a hand-rolled dispatch over the four possible widths — no virtual call, no switch on a runtime type id, just four instantiations of a lambda. This is the pattern used throughout ClickHouse’s hot column code: the virtual IColumn interface is for the cold, structural paths; the per-row paths are devirtualized by dispatching on the concrete type once and then running a monomorphic loop.

Reading an index back has the same shape, and is marked ALWAYS_INLINE because it is called from the tightest loops in the engine:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:326
 ALWAYS_INLINE size_t getIndexAt(size_t row) const
 {
 const IColumn * indexes = &getIndexes();

 switch (idx.getSizeOfIndexType())
 {
 case sizeof(UInt8): return assert_cast<const ColumnUInt8 *>(indexes)->getElement(row);
 case sizeof(UInt16): return assert_cast<const ColumnUInt16 *>(indexes)->getElement(row);
 case sizeof(UInt32): return assert_cast<const ColumnUInt32 *>(indexes)->getElement(row);
 case sizeof(UInt64): return assert_cast<const ColumnUInt64 *>(indexes)->getElement(row);
 default: throwUnexpectedLowCardinalityIndexType(idx.getSizeOfIndexType());
 }
 }

The dictionary: ColumnUnique and its reverse index

ColumnUnique is the dictionary. It is not a column type you can select; the comment says so:

 cpp
 
 
// src/Columns/ColumnUnique.h:36
/** Stores another column with unique values
 * and also an index that allows to find position by value.
 *
 * This column is not used on it's own but only as implementation detail of ColumnLowCardinality.
 */
template <typename ColumnType>
class ColumnUnique final : public COWHelper<IColumnUnique, ColumnUnique<ColumnType>>

Its members are the values plus the lookup structure:

 cpp
 
 
// src/Columns/ColumnUnique.h:203
private:
 IColumn::WrappedPtr column_holder;
 bool is_nullable;
 size_t size_of_value_if_fixed = 0;

 ReverseIndex<UInt64, ColumnType> reverse_index;

 /// For DataTypeNullable, stores null map.
 IColumn::WrappedPtr nested_null_mask;
 IColumn::WrappedPtr nested_column_nullable;

Two reserved slots

The first row of the dictionary is special: it is the default value of the type. If the dictionary is nullable, row 0 is the NULL and row 1 is the default value of the inner type:

 cpp
 
 
// src/Columns/ColumnUnique.h:74
 size_t getDefaultValueIndex() const override { return 0; }
 size_t getNullValueIndex() const override;
 size_t getNestedTypeDefaultValueIndex() const override { return is_nullable ? 1 : 0; }
 bool canContainNulls() const override { return is_nullable; }

and

 cpp
 
 
// src/Columns/ColumnUnique.h:370
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::getNullValueIndex() const
{
 if (!is_nullable)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ColumnUnique can't contain null values.");

 return 0;
}

The count of reserved slots is therefore one or two:

 cpp
 
 
// src/Columns/ColumnUnique.h:238
 static size_t numSpecialValues(bool is_nullable) { return is_nullable ? 2 : 1; }
 size_t numSpecialValues() const { return numSpecialValues(is_nullable); }

Those reserved slots are not in the reverse index, which is why the insert path has to special-case them by hand:

 cpp
 
 
// src/Columns/ColumnUnique.h:513
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsertData(const char * pos, size_t length)
{
 /// The reserved prefix slots are not in the reverse index, so the default value is matched here.
 /// Comparing the first byte first keeps the `memcmp` call out of this path for the values that
 /// are not the default, which is all of them in a typical dictionary.
 const size_t default_index = getNestedTypeDefaultValueIndex();
 const std::string_view default_value = getRawColumnPtr()->getDataAt(default_index);
 if (default_value.size() == length && (length == 0 || (default_value[0] == pos[0] && default_value == std::string_view(pos, length))))
 return default_index;

 auto insertion_point = reverse_index.insert({pos, length});

 updateNullMask();

 return insertion_point;
}

The comment is a nice piece of micro-optimisation reasoning: comparing the first byte before the whole value keeps the expensive memcmp off the path for the (overwhelmingly common) case where the value is not the default.

The reverse index stores row numbers, not values

ReverseIndex is the interesting part. A naive dictionary would be a HashMap<value, index> — the values stored twice, once in the column and once as hash-table keys. ReverseIndex inverts that: the hash table stores the row number in the dictionary column, and comparisons dereference back into the column:

 cpp
 
 
// src/Columns/ReverseIndex.h:94
 /// Special case when we want to compare with something not in index_column.
 /// When we compare something inside column default keyEquals checks only that row numbers are equal.
 bool keyEquals(std::string_view object, size_t hash_ [[maybe_unused]], const State & state) const
 {
 auto index = key;
 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return hash_ == (*state.saved_hash_column)[index] && object == state.index_column->getDataAt(index);
 else
 return object == state.index_column->getDataAt(index);
 }

The hash table therefore holds 8-byte integers, and the dictionary column holds the only copy of the data. For a LowCardinality(String) with 1000 distinct values, that is the difference between a hash table with 1000 string keys (plus their hashes, plus their lengths) and a hash table with 1000 UInt64s.

For string columns the hash is expensive, so it is precomputed once and kept in a parallel ColumnUInt64:

 cpp
 
 
// src/Columns/ReverseIndex.h:43
template <typename ColumnType>
struct ReverseIndexHashTableState<ColumnType, /* with_saved_hash = */ true, /* has_base_index */ false>
{
 constexpr static bool with_saved_hash = true;
 constexpr static bool has_base_index = false;

 ColumnType * index_column;
 typename ColumnVector<UInt64>::Container * saved_hash_column;
};

and the hash function for a stored row reads the precomputed hash rather than re-hashing the string:

 cpp
 
 
// src/Columns/ReverseIndex.h:108
 size_t getHash(const Hash & hash) const
 {
 auto index = key;

 /// Hack. HashTable is Hash itself.
 const auto & state = static_cast<const State &>(static_cast<const HashTable &>(hash));

 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return (*state.saved_hash_column)[index];
 else
 {
 using ValueType = typename ColumnType::ValueType;
 ValueType value = unalignedLoad<ValueType>(state.index_column->getDataAt(index).data());
 return DefaultHash<ValueType>()(value);
 }
 }

The index is built lazily, and the hash table is filled in one pass

The reverse index is not built at construction time; it is built on the first insert, and the build is a single pass over the dictionary:

 cpp
 
 
// src/Columns/ReverseIndex.h:435
template <typename IndexType, typename ColumnType>
void ReverseIndex<IndexType, ColumnType>::buildIndex()
{
 if (index)
 return;

 if (!column)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ReverseIndex can't build index because index column wasn't set.");

 auto size = column->size();
 index = std::make_unique<IndexMapType>(size);

 if constexpr (use_saved_hash)
 saved_hash = calcHashes();

 auto & state = index->getState();
 state.index_column = column;
 state.base_index = base_index;
 if constexpr (use_saved_hash)
 state.saved_hash_column = &saved_hash->getData();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 for (auto row : collections::range(num_prefix_rows_to_skip, size))
 {
 UInt64 hash = 0;
 if constexpr (use_saved_hash)
 hash = saved_hash->getElement(row);
 else
 hash = getHash(column->getDataAt(row));

 index->reverseIndexEmplace(row + base_index, iterator, inserted, hash, column->getDataAt(row));

 if (!inserted)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Duplicating keys found in ReverseIndex.");
 }
}

Note num_prefix_rows_to_skip: the reserved NULL/default slots are skipped, because they are matched by hand in uniqueInsertData.

The insert path is the mirror image, and it is written to be branch-light for the two different column kinds:

 cpp
 
 
// src/Columns/ReverseIndex.h:490
template <typename IndexType, typename ColumnType>
UInt64 ReverseIndex<IndexType, ColumnType>::insert(std::string_view data)
{
 if (!index)
 buildIndex();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 auto hash = getHash(data);
 UInt64 num_rows = size();

 if constexpr (use_saved_hash)
 {
 auto & column_data = saved_hash->getData();
 if (column_data.size() <= num_rows)
 column_data.resize(num_rows + 1);
 column_data[num_rows] = hash;
 }
 else
 column->insertData(data.data(), data.size());

 index->reverseIndexEmplace(num_rows + base_index, iterator, inserted, hash, data);

 if constexpr (use_saved_hash)
 {
 if (inserted)
 column->insertData(data.data(), data.size());
 }
 else
 {
 if (!inserted)
 column->popBack(1);
 }

 return iterator->getValue();
}

For strings the hash is appended optimistically and the value is appended only if the key was new. For fixed-size values the value is appended optimistically and popped back if the key turned out to be a duplicate. Each kind pushes and pops the cheaper thing.

Float NaNs are canonicalized

A LowCardinality(Float64) dictionary must not contain two NaNs, because NaN == NaN is false but all NaNs must compare equal in the dictionary semantics. So NaN is normalized at insert time:

 cpp
 
 
// src/Columns/ColumnUnique.h:389
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsert(const Field & x)
{
 if (x.isNull())
 return getNullValueIndex();

 // NaN can contain different sign or mantissa bits, but we need to consider all NaNs equal.
 if constexpr (is_float_vector_v<ColumnType>)
 {
 if (isNaN(x.safeGet<typename ColumnType::ValueType>()))
 {
 auto nan = NaNOrZero<typename ColumnType::ValueType>();
 return uniqueInsertData(reinterpret_cast<char *>(&nan), sizeof(nan));
 }
 }

Here is where the design pays off. A LowCardinality column can point at a dictionary owned by another column. The shared flag on Dictionary records this, and byteSize() deliberately does not count a shared dictionary:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:200
 /// Don't count the dictionary size as it can be shared between different blocks.
 size_t byteSize() const override { return idx.getIndexes()->byteSize() + (isSharedDictionary() ? 0 : getDictionary().byteSize()); }

 size_t byteSizeAt(size_t n) const override { return getDictionary().byteSizeAt(getIndexes().getUInt(n)); }
 size_t allocatedBytes() const override { return idx.getIndexes()->allocatedBytes() + getDictionary().allocatedBytes(); }

Note the asymmetry: byteSize() (the logical size, used for statistics and for system.parts_columns) excludes the shared dictionary, but allocatedBytes() (the physical size, used for memory accounting) includes it. That distinction is what makes the accounting correct when a part has a hundred blocks all pointing at one dictionary.

Sharing also propagates through every operation that produces a new column from an old one — cut, filter, permute, index, replicate all hand the same dictionary pointer to the result:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:130
 ColumnPtr filter(const Filter & filt, ssize_t result_size_hint) const override
 {
 return ColumnLowCardinality::create(
 dictionary.getColumnUniquePtr(), getIndexes().filter(filt, result_size_hint), isSharedDictionary());
 }

But a shared dictionary is immutable

The moment you want to mutate a column whose dictionary is shared with somebody else, the dictionary has to be un-shared. This happens in compactIfSharedDictionary(), which is called at the top of every mutating insert:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:146
void ColumnLowCardinality::insert(const Field & x)
{
 compactIfSharedDictionary();
 idx.insertIndex(getDictionary().uniqueInsert(x));
}

bool ColumnLowCardinality::tryInsert(const Field & x)
{
 compactIfSharedDictionary();

 size_t index = 0;
 if (!dictionary.getColumnUnique().tryUniqueInsert(x, index))
 return false;

 idx.insertIndex(index);
 return true;
}

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:894
void ColumnLowCardinality::compactIfSharedDictionary()
{
 if (dictionary.isShared())
 compactInplace();
}

Compaction rebuilds a private dictionary containing only the keys that are actually referenced by this column’s indexes, and remaps the indexes accordingly:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:1002
MutableColumnPtr ColumnLowCardinality::Dictionary::compact(const IColumnUnique & unique, MutableColumnPtr & indexes)
{
 auto new_column_unique = unique.cloneEmpty();
 auto & new_unique = static_cast<IColumnUnique &>(*new_column_unique);

 auto unique_indexes = mapUniqueIndex(*indexes);
 auto sub_keys = unique.getNestedColumn()->index(*unique_indexes, 0);
 auto new_indexes = new_unique.uniqueInsertRangeFrom(*sub_keys, 0, sub_keys->size());

 indexes = IColumn::mutate(new_indexes->index(*indexes, 0));
 return new_column_unique;
}

mapUniqueIndex is the piece that does the renumbering. It is a counting-sort-like trick that works in O(n) with a flat array, and it has an overflow path for when the dictionary is shared and the indexes do not fit the “max index < number of rows” assumption:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:79
template <typename T>
MutableColumnPtr mapUniqueIndexImpl(PaddedPODArray<T> & index)
{
 if (index.empty())
 return ColumnVector<T>::create();

 auto size = index.size();

 T max_val = index[0];
 for (size_t i = 1; i < size; ++i)
 max_val = std::max(max_val, index[i]);

 /// May happen when dictionary is shared.
 if (max_val > size)
 return mapUniqueIndexImplRef(index);

 auto map_size = static_cast<UInt64>(max_val) + 1;
 PaddedPODArray<T> map(map_size, 0);
 T zero_pos_value = index[0];
 index[0] = 0;
 T cur_pos = 0;
 for (size_t i = 1; i < size; ++i)
 {
 T val = index[i];
 if (val != zero_pos_value && map[val] == 0)
 {
 ++cur_pos;
 map[val] = cur_pos;
 }

 index[i] = map[val];
 }

The “overflow” case (max_val > size) means the column references dictionary entries that belong to a bigger dictionary — exactly what happens when the dictionary is shared. Then a hash map is used instead of a flat array:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:53
template <typename T>
MutableColumnPtr mapUniqueIndexImplRef(PaddedPODArray<T> & index)
{
 PaddedPODArray<T> copy(index.cbegin(), index.cend());

 HashMap<T, T> hash_map;
 for (auto val : index)
 hash_map.insert({val, static_cast<T>(hash_map.size())});

 auto res_col = ColumnVector<T>::create();
 auto & data = res_col->getData();

 data.resize(hash_map.size());
 for (const auto & val : hash_map)
 data[val.getMapped()] = val.getKey();

 for (auto & ind : index)
 ind = hash_map[ind];

 for (size_t i = 0; i < index.size(); ++i)
 if (data[index[i]] != copy[i])
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected {}, but got {}", toString(data[index[i]]), toString(copy[i]));

 return res_col;
}

That last loop is a self-check: after the renumbering, the new index must still resolve to the same value. Cheap insurance against an off-by-one in a data structure that is otherwise very hard to test.

Inserting from a column with the same dictionary

Once dictionaries are shared, a very common operation becomes trivial: copying a row from another LowCardinality column that happens to use the same dictionary is just a copy of an index.

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:180
 size_t position = low_cardinality_src->getIndexes().getUInt(n);

 if (&low_cardinality_src->getDictionary() == &getDictionary())
 {
 /// Dictionary is shared with src column. Insert only index.
 idx.insertIndex(position);
 }
 else
 {
 compactIfSharedDictionary();
 const auto & nested = *low_cardinality_src->getDictionary().getNestedColumn();
 idx.insertIndex(getDictionary().uniqueInsertFrom(nested, position));
 }

The test is a pointer comparison of the dictionary addresses. If they are the same object, the index is valid as-is; otherwise the value has to be re-inserted into this column’s dictionary and the resulting index used.

On disk: one dictionary per part, versioned per granule

Now the hard part. In a MergeTree part, a LowCardinality column is written as two streams: DictionaryKeys and DictionaryIndexes.

The keys stream holds the dictionary; the indexes stream holds, per granule, a header and then the index data. The dictionary is not rewritten per granule — it accumulates until it hits a size limit, at which point it is flushed and a new one starts.

The version byte

The keys stream starts with a version:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:102
struct KeysSerializationVersion
{
 enum Value
 {
 /// Version is written at the start of <name.dict.bin>.
 /// Dictionary is written as number N and N keys after them.
 /// Dictionary can be shared for continuous range of granules, so some marks may point to the same position.
 /// Shared dictionary is stored in state and is read once.
 SharedDictionariesWithAdditionalKeys = 1,
 };

 Value value;

 static void checkVersion(UInt64 version)
 {
 if (version != SharedDictionariesWithAdditionalKeys)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Invalid version for SerializationLowCardinality key column.");
 }

 explicit KeysSerializationVersion(UInt64 version) : value(static_cast<Value>(version)) { checkVersion(version); }
};

The single legal version is called SharedDictionariesWithAdditionalKeys, and the comment tells you the whole story in three lines: the dictionary is written as a count followed by that many keys, it can be shared across a continuous range of granules, and the shared dictionary is kept in the serialization state and read once.

The per-granule header

Every granule’s index data begins with a header that encodes the index width plus three flags:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:124
/// Version is stored at the start of each granule. It's used to store indexes type and flags.
struct IndexesSerializationType
{
 using SerializationType = UInt64;
 /// Need to read dictionary if it wasn't.
 static constexpr SerializationType NeedGlobalDictionaryBit = 1u << 8u;
 /// Need to read additional keys. Additional keys are stored before indexes as value N and N keys after them.
 static constexpr SerializationType HasAdditionalKeysBit = 1u << 9u;
 /// Need to update dictionary. It means that previous granule has different dictionary.
 static constexpr SerializationType NeedUpdateDictionary = 1u << 10u;

 enum Type
 {
 TUInt8 = 0,
 TUInt16,
 TUInt32,
 TUInt64,
 };

 Type type;
 bool has_additional_keys;
 bool need_global_dictionary;
 bool need_update_dictionary;

 static constexpr SerializationType resetFlags(SerializationType type)
 {
 return type & (~(HasAdditionalKeysBit | NeedGlobalDictionaryBit | NeedUpdateDictionary));
 }

The three flags are the entire state machine:

NeedGlobalDictionaryBit — “there is a dictionary in the keys stream that I have not read yet”.
HasAdditionalKeysBit — “this granule carries keys that did not fit in the shared dictionary; they are stored inline, before the index data”.
NeedUpdateDictionary — “the dictionary changed since the previous granule”.

Serialization and deserialization are symmetric and trivial:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:162
 void serialize(WriteBuffer & buffer) const
 {
 SerializationType val = type;
 if (has_additional_keys)
 val |= HasAdditionalKeysBit;
 if (need_global_dictionary)
 val |= NeedGlobalDictionaryBit;
 if (need_update_dictionary)
 val |= NeedUpdateDictionary;
 writeBinaryLittleEndian(val, buffer);
 }

 void deserialize(ReadBuffer & buffer, const ISerialization::DeserializeBinaryBulkSettings & settings)
 {
 SerializationType val = 0;
 readBinaryLittleEndian(val, buffer);

 checkType(val);
 has_additional_keys = (val & HasAdditionalKeysBit) != 0;
 need_global_dictionary = (val & NeedGlobalDictionaryBit) != 0;
 need_update_dictionary = (val & NeedUpdateDictionary) != 0;
 type = static_cast<Type>(resetFlags(val));

 if (settings.native_format)
 {
 if (need_global_dictionary)
 throw Exception(ErrorCodes::INCORRECT_DATA,
 "LowCardinality indexes serialization type for Native format "
 "cannot use global dictionary");
 }
 }

The low four bits carry the index width; bits 8, 9, 10 are the flags. Bits 4–7 are unused — a hole left for future flags.

Writing: the dictionary accumulates, then overflows

The writer keeps the accumulating dictionary in the serialization state:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:232
struct SerializeStateLowCardinality : public ISerialization::SerializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 MutableColumnUniquePtr shared_dictionary;

 explicit SerializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}
};

The write path is where the interesting decisions live:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:561
 const ColumnLowCardinality & low_cardinality_column = typeid_cast<const ColumnLowCardinality &>(column);

 auto * low_cardinality_state = checkAndGetState<SerializeStateLowCardinality>(state);
 auto & global_dictionary = low_cardinality_state->shared_dictionary;
 KeysSerializationVersion::checkVersion(low_cardinality_state->key_version.value);

 size_t max_limit = column.size() - offset;
 limit = limit ? std::min(limit, max_limit) : max_limit;

 /// Do not write anything for empty column. (May happen while writing empty arrays.)
 if (limit == 0)
 return;

 bool need_update_dictionary = global_dictionary == nullptr;
 if (need_update_dictionary)
 global_dictionary = DataTypeLowCardinality::createColumnUnique(*dictionary_type);

 auto sub_column = low_cardinality_column.cutAndCompact(offset, limit);
 ColumnPtr positions = sub_column->getIndexesPtr();
 ColumnPtr keys = sub_column->getDictionary().getNestedColumn();

 if (settings.low_cardinality_max_dictionary_size)
 {
 /// Insert used_keys into global dictionary and update sub_index.
 auto indexes_with_overflow = global_dictionary->uniqueInsertRangeWithOverflow(*keys, 0, keys->size(),
 settings.low_cardinality_max_dictionary_size);

 if (global_dictionary->size() > settings.low_cardinality_max_dictionary_size)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Got dictionary with size {} but max dictionary size is {}",
 global_dictionary->size(), settings.low_cardinality_max_dictionary_size);

 positions = indexes_with_overflow.indexes->index(*positions, 0);
 keys = std::move(indexes_with_overflow.overflowed_keys);

 if (global_dictionary->size() < settings.low_cardinality_max_dictionary_size && !keys->empty())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Has additional keys, but dict size is {} which is less "
 "then max dictionary size ({})", global_dictionary->size(),
 settings.low_cardinality_max_dictionary_size);
 }

cutAndCompact is the first step: it takes the slice of the column being written and produces a private dictionary containing only the keys that slice actually uses. That is what keeps the accumulated dictionary small — without it, every granule would re-insert the whole dictionary.

Then the keys are merged into the global dictionary, and anything that does not fit within low_cardinality_max_dictionary_size becomes “overflowed keys” — keys carried inline by that granule.

The flush happens when the accumulated dictionary reaches the limit:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:602
 bool need_additional_keys = !keys->empty();
 bool need_dictionary = settings.low_cardinality_max_dictionary_size != 0;
 bool need_write_dictionary = !settings.low_cardinality_use_single_dictionary_for_part
 && global_dictionary->size() >= settings.low_cardinality_max_dictionary_size;

 IndexesSerializationType index_version(*positions, need_additional_keys, need_dictionary, need_update_dictionary);
 index_version.serialize(*indexes_stream);

 if (need_write_dictionary)
 {
 const auto & nested_column = global_dictionary->getNestedNotNullableColumn();
 UInt64 num_keys = nested_column->size();
 writeBinaryLittleEndian(num_keys, *keys_stream);
 dict_inner_serialization->serializeBinaryBulk(*nested_column, *keys_stream, 0, num_keys);
 low_cardinality_state->shared_dictionary = nullptr;
 }

 if (need_additional_keys)
 {
 UInt64 num_keys = keys->size();
 writeBinaryLittleEndian(num_keys, *indexes_stream);
 dict_inner_serialization->serializeBinaryBulk(*keys, *indexes_stream, 0, num_keys);
 }

 UInt64 num_rows = positions->size();
 writeBinaryLittleEndian(num_rows, *indexes_stream);
 auto index_serialization = index_version.getDataType()->getDefaultSerialization();
 index_serialization->serializeBinaryBulk(*positions, *indexes_stream, 0, num_rows);

The default limit is 8192 keys:

 cpp
 
 
// src/Core/Settings.cpp:5045
 DECLARE(UInt64, low_cardinality_max_dictionary_size, 8192, R"(
Sets a maximum size in rows of a shared global dictionary for the [LowCardinality](/reference/data-types/lowcardinality) data type that can be written to a storage file system. This setting prevents issues with RAM in case of unlimited dictionary growth. All the data that can't be encoded due to maximum dictionary size limitation ClickHouse writes in an ordinary method.

Possible values:

- Any positive integer.
)", 0) \

So: with an 8192-key limit, a part whose column has 100 distinct values writes the dictionary once; a part with 100 000 distinct values writes the dictionary roughly every 8192 keys and starts a new one.

Note that need_update_dictionary is global_dictionary == nullptr — i.e. “this granule starts a new dictionary”. And the dictionary is flushed after the header was written, so the header says “read the dictionary” and the dictionary follows in the keys stream.

Reading: the reader has to reconstruct the same state

The read side keeps the shared dictionary in the deserialize state, along with a flag that survives granule skips:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:240
struct DeserializeStateLowCardinality : public ISerialization::DeserializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 ColumnUniquePtr global_dictionary;

 IndexesSerializationType index_type{};
 ColumnPtr additional_keys;
 ColumnPtr null_map;
 UInt64 num_pending_rows = 0;

 /// If dictionary should be updated.
 /// Can happen is some granules was skipped while reading from MergeTree.
 /// We should store this flag in State because
 /// in case of long block of empty arrays we may not need read dictionary at first reading.
 bool need_update_dictionary = false;

 /// True if this part has a single dictionary for the entire column. The prefix
 /// reader first checks `has_uniform_marks_callback` as a cheap prefilter, then
 /// verifies it by reading the first dictionary and checking that `DictionaryKeys`
 /// reached EOF. When true, non-continuous reads (granule skips) do not invalidate
 /// the cached dictionary, and the dictionary can be shared across all threads via `clone`.
 bool single_dictionary_for_part = false;

 explicit DeserializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}

 ISerialization::DeserializeBinaryBulkStatePtr cloneImpl(ISerialization::DeserializeBinaryBulkStateCloneContext &) const override
 {
 return std::make_shared<DeserializeStateLowCardinality>(*this);
 }
};

The need_update_dictionary comment is the interesting one: when reading is not continuous (i.e. some granules were skipped), the reader cannot assume that the dictionary it holds is the one the next granule refers to, so it must force a re-read. That flag lives in the state, not on the stack, precisely because a skipped granule may be discovered long after the state was created.

The main read loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:752
 if (!settings.continuous_reading)
 {
 low_cardinality_state->num_pending_rows = 0;

 /// Some granules were skipped; the next granule header may signal a dictionary
 /// update. For multi-dict parts this forces a re-read at the next boundary.
 low_cardinality_state->need_update_dictionary = true;
 }

 while (limit)
 {
 if (low_cardinality_state->num_pending_rows == 0)
 {
 if (indexes_stream->eof())
 break;

 auto & index_type = low_cardinality_state->index_type;
 auto & global_dictionary = low_cardinality_state->global_dictionary;

 index_type.deserialize(*indexes_stream, settings);

 bool need_update_dictionary =
 !global_dictionary || index_type.need_update_dictionary || low_cardinality_state->need_update_dictionary;
 if (index_type.need_global_dictionary && need_update_dictionary)
 {
 /// For single-dict parts the in-memory dictionary is valid for the entire part,
 /// so skip re-reading it even if the stream was seeked back to the beginning.
 if (!low_cardinality_state->single_dictionary_for_part || !global_dictionary)
 read_dictionary();
 low_cardinality_state->need_update_dictionary = false;
 }

 if (low_cardinality_state->index_type.has_additional_keys)
 read_additional_keys();
 else
 low_cardinality_state->additional_keys = nullptr;

 readBinaryLittleEndian(low_cardinality_state->num_pending_rows, *indexes_stream);
 }

 size_t num_rows_to_read = std::min<UInt64>(limit, low_cardinality_state->num_pending_rows);
 read_indexes(num_rows_to_read);
 limit -= num_rows_to_read;
 low_cardinality_state->num_pending_rows -= num_rows_to_read;
 }

num_pending_rows is what makes a granule’s worth of rows readable in chunks: the header is read once, then the reader consumes num_pending_rows rows from the index stream, and only when that counter hits zero does it look for the next header.

The single-dictionary-for-part fast path

There is a special case worth highlighting: a part where the column has exactly one dictionary for the whole part. The prefix reader detects it by reading the first dictionary and checking whether the DictionaryKeys stream reached EOF:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:348
 if (might_have_single_dictionary && !stream->eof())
 {
 /// For `Wide` parts that might have a single dictionary, the stream is
 /// now positioned right at the dictionary data (after the version).
 /// Read the first dictionary and check EOF afterwards. If this proves
 /// that there are no more dictionaries, all threads sharing this prefix
 /// state via `DeserializationPrefixesCache` get the dictionary through
 /// a `shared_ptr` clone instead of each reading it independently.

and

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:377
 if (stream->eof())
 {
 new_state->single_dictionary_for_part = true;
 new_state->global_dictionary = std::move(global_dictionary);
 }
 else
 {
 /// The mark heuristic can be fooled by several dictionaries written
 /// inside one mark interval. Reset the `DictionaryKeys` stream back
 /// to the normal post-prefix position and let data deserialization
 /// read dictionary updates from the stream.
 if (!settings.seek_to_start_callback)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot reset LowCardinality dictionary stream without seek_to_start_callback");

 settings.seek_to_start_callback(dictionary_keys_path);

 UInt64 repeated_keys_version = 0;
 readBinaryLittleEndian(repeated_keys_version, *stream);
 if (repeated_keys_version != keys_version)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Inconsistent version while resetting LowCardinality dictionary stream");
 }

This is a heuristic with a verification step. The cheap check (has_uniform_marks_callback) says “all marks point at the same position in the keys stream, so maybe there is only one dictionary”. The expensive check (actually reading the dictionary and seeing EOF) confirms it. If the heuristic lied, the stream is rewound and the normal path takes over.

That’s a pattern you will see repeatedly in ClickHouse’s MergeTree reader: cheap prefilter, expensive confirmation, safe fallback.

When the fast path holds, cloneImpl copies the state (including the dictionary pointer) so every thread reading the part gets the same dictionary without re-reading it.

The overflow path on the read side

When a granule carries inline “additional keys”, the reader has to merge two dictionaries: the global one and the granule-local one. mapIndexWithAdditionalKeys does that with two flat maps — one for the global dictionary’s keys and one for the overflow keys:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:515
 /// Update column and return map with old indexes.
 /// Let N is the number of distinct values which are less than max_size;
 /// old_column - column before function call;
 /// new_column - column after function call:
 /// * if old_column[i] < max_size, than
 /// dictionary_map[new_column[i]] = old_column[i]
 /// * else
 /// additional_keys_map[new_column[i]] = old_column[i] - dict_size + N
 IndexMapsWithAdditionalKeys mapIndexWithAdditionalKeys(IColumn & column, size_t dict_size)

and then the keys are stitched back together:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:726
 else
 {
 auto maps = mapIndexWithAdditionalKeys(*indexes_column, global_dictionary->size());

 auto used_keys = IColumn::mutate(global_dictionary->getNestedColumn()->index(*maps.dictionary_map, 0));

 if (!maps.additional_keys_map->empty())
 {
 if (additional_keys == nullptr)
 throw Exception(ErrorCodes::INCORRECT_DATA, "No additional keys found.");

 auto used_add_keys = additional_keys->index(*maps.additional_keys_map, 0);

 if (dictionary_type->isNullable())
 {
 ColumnPtr null_map = ColumnUInt8::create(used_add_keys->size(), static_cast<UInt8>(0));
 used_add_keys = ColumnNullable::create(used_add_keys, null_map);
 }

 used_keys->insertRangeFrom(*used_add_keys, 0, used_add_keys->size());
 }

 low_cardinality_column.insertRangeFromDictionaryEncodedColumn(*used_keys, *indexes_column);
 }

The final call re-inserts the merged keys into the destination column’s own dictionary and remaps the indexes one last time.

The whole thing in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `user_id` UInt64,
 `kind` LowCardinality(String), -- 12 distinct values in 100M rows
 `country` LowCardinality(FixedString(2))
)
ENGINE = MergeTree
ORDER BY (ts, user_id);

 sql
 
 
-- The dictionary is shared across every block of the part, so the per-column
-- cost is one byte per row (UInt8 index) plus the dictionary once.
SELECT kind, count() FROM events GROUP BY kind;

The type-level guarantee is that LowCardinality is only worth it while the dictionary stays small; the documentation says so plainly:

text

// src/DataTypes/DataTypeLowCardinality.cpp:198
The efficiency of using `LowCardinality` data type depends on data diversity. If a dictionary contains less than 10,000 distinct values, then ClickHouse mostly shows higher efficiency of data reading and storing. If a dictionary contains more than 100,000 distinct values, then ClickHouse can perform worse in comparison with using ordinary data types.

Trade-offs

What the hack buys you.

A LowCardinality(String) column with 12 distinct values costs one byte per row plus a few hundred bytes of dictionary, instead of ~10 bytes per row of strings. It also means the scan is a byte scan, not a string scan: filters, GROUP BY, and ORDER BY all operate on UInt8 indexes, and the dictionary is touched only for the rows that survive the filter.

What it costs you.

Every value access is two dependent loads: read the index, then read the dictionary entry. That is a pointer chase a plain String column does not have — although for a plain string column you also pay a pointer chase into the character arena, so the two are more comparable than they look.

The dictionary is a global per part, and the index stream has to carry a per-granule header with three flags and a row count, plus the possibility of inline overflow keys. A plain column has none of that. For a column with 100 000+ distinct values you have bought complexity and lost compression.

The sharing is also asymmetric: byteSize() excludes a shared dictionary while allocatedBytes() includes it. That is correct but subtle, and it means the two numbers in system.parts_columns can disagree in a way that looks like a bug until you know why.

The design decision that makes it work. The dictionary is not stored per block, and not per granule; it is stored per part, capped at 8192 keys, and flushed when it overflows. Everything else — the three header bits, the need_update_dictionary flag, the single_dictionary_for_part heuristic — exists to make that one decision survive partial reads, granule skips and multi-threaded readers.

Wrapping up

The one thing to remember: LowCardinality moves the repetition out of the rows and into a shared dictionary, and the whole design is about making that dictionary shared — across blocks, granules, parts, and threads.

The rule of thumb: below ~10 000 distinct values, use it. Above ~100 000, don’t. In between, measure.

What to go do about it: count the distinct values in your “status” or “country” columns (uniqExact), then wrap them. One byte per row instead of ten is not a rounding error on a hundred-million-row table.

ClickHouse Splits GROUP BY into 256 Buckets for Parallel Merges

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: A GROUP BY needs a serial merge at the end — and that’s where the memory high-water mark lives. ClickHouse splits the key space into 256 buckets by the top bits of the hash, so the merge becomes 256 independent, lock-free, parallel jobs.

A GROUP BY in ClickHouse is usually computed by several threads at once, each with its own hash table, and the tables are then merged.

Merging N hash tables of M entries each is not something you want to do on one thread. It’s a serial phase in an otherwise parallel query, and it’s the phase where the memory high-water mark is reached.

The way out is to make the merge divisible. If the keys are partitioned into a fixed number of buckets by a deterministic function of the key, then two tables can be merged bucket by bucket, and different buckets can be merged by different threads with no coordination at all: bucket 17 of table A can only collide with bucket 17 of table B.

That’s what TwoLevelHashTable in src/Common/HashTable/TwoLevelHashTable.h does, and it’s what the group_by_two_level_threshold setting controls.

The structure

The two-level table is literally an array of ordinary hash tables:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:9
/** Two-level hash table.
 * Represents 256 (or 1ULL << BITS_FOR_BUCKET) small hash tables (buckets of the first level).
 * To determine which one to use, one of the bytes of the hash function is taken.
 *
 * Usually works a little slower than a simple hash table.
 * However, it has advantages in some cases:
 * - if you need to merge two hash tables together, then you can easily parallelize it by buckets;
 * - delay during resizes is amortized, since the small hash tables will be resized separately;
 * - in theory, resizes are cache-local in a larger range of sizes.
 *
 * With `BITS_FOR_BUCKET = 0` there is a single bucket: routing folds to a constant.
 * Lookups and inserts compile down to what the single-level table does.
 *
 * Buckets share no state.
 * Threads may fill different buckets at the same time when every bucket is written under its own lock.
 */

The bucket index comes out of the hash:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:58
 static constexpr UInt32 NUM_BUCKETS = 1ULL << BITS_FOR_BUCKET;
 static constexpr UInt32 MAX_BUCKET = NUM_BUCKETS - 1;

 size_t hash(const Key & x) const { return Hash::operator()(x); }

 /// NOTE Bad for hash tables with more than 2^32 cells.
 static size_t getBucketFromHash(size_t hash_value) { return (hash_value >> (32 - BITS_FOR_BUCKET)) & MAX_BUCKET; }

BITS_FOR_BUCKET defaults to 8, so 256 buckets, and the bucket is taken from the high bytes of the 64-bit hash:

 text
 
 
 64-bit hash
 |<-------------------------->|
 | bucket | sub-table index |
 | 8 bits | (mask & (size-1)) |
 +-----------+---------------------+
 | 0..255 | depends on the bucket's own size |
 +-----------+---------------------+

 hash >> 24 & 255 -> which of the 256 sub-tables
 hash & (2^k - 1) -> which cell inside that sub-table

The choice of the high bits is not arbitrary, and it’s the subtle part of the whole design. The single-level table uses the low bits of the hash to index its cells:

 cpp
 
 
// src/Common/HashTable/HashTable.h:238
 /// From the hash value, get the cell number in the hash table.
 size_t place(size_t x) const { return x & mask(); }

If the bucket were also selected from the low bits, the two uses would collide. Every bucket would receive keys whose low bits are a fixed value, and the sub-table’s own probe would be confined to a single cell. The sub-tables would degenerate into 256 linked lists.

Taking the bucket from the top of the hash and the cell from the bottom keeps the two independent — which is exactly what a good hash function provides.

The default bucket count is 8 bits, and the comment “Bad for hash tables with more than 2^32 cells” is about the other direction: the shift hash_value >> (32 - BITS_FOR_BUCKET) throws away everything above bit 32, so the bucket selection only sees 32 bits of the hash. For a table with more than 2^32 cells, the top bits would carry real information and throwing them away would lose it.

The sub-tables are ordinary tables

Each bucket is a plain HashTable, so all the machinery from the previous article — zeroed cells, linear probing, Robin Hood ordering — applies inside a bucket. The two-level table only adds the routing:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:316
 template <typename KeyHolder>
 void ALWAYS_INLINE emplace(KeyHolder && key_holder, LookupResult & it, bool & inserted, size_t hash_value)
 {
 size_t buck = getBucketFromHash(hash_value);
 impls[buck].emplace(key_holder, it, inserted, hash_value);
 }

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:333
 LookupResult ALWAYS_INLINE find(Key x, size_t hash_value)
 {
 size_t buck = getBucketFromHash(hash_value);
 return impls[buck].find(x, hash_value);
 }

and the storage is a plain array of sub-tables:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:100
 Impl impls[NUM_BUCKETS];

 /// `bucket_cells_prefix[b]` is the number of cells in the buckets before `b`. See `computeBucketPrefix`.
 std::vector<size_t> bucket_cells_prefix;

Impl impls[NUM_BUCKETS] is 256 sub-tables by value, not by pointer. Each sub-table owns its buffer, so the whole structure is 256 small allocations plus the bookkeeping of each.

How a bucket grows

The buckets do grow, but they grow in a different shape from the single-level table. The grower used by the two-level table caps the quadratic-growth phase at 2^15 cells and switches to the Robin Hood mode above that:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:26
/// The buckets grow by 4 times while they are small, and switch to the Robin Hood mode when they become large,
/// see `HashTableGrowerWithPrecalculationAndRobinHood`.
template <size_t initial_size_degree = 8, size_t robin_hood_size_degree = 16>
using TwoLevelHashTableGrower = HashTableGrowerWithPrecalculationAndRobinHood<initial_size_degree, 15, robin_hood_size_degree>;

The template parameters are initial_size_degree = 8, max_size_degree_quadratic = 15, robin_hood_size_degree = 16. So:

up to 2^15 cells per bucket, the sub-table quadruples its size on overflow (a degree increment of two);
from 2^15 to 2^16 it doubles;
above 2^16 cells per bucket it switches to the Robin Hood mode with a 90% load factor, and from then on it keeps doubling like any other table — but at 90% load instead of 50%.

The effect of the switch is memory, not capacity. A bucket that grows large stops paying the factor-of-two penalty of the 0.5 load factor, because the same number of keys now needs half as many cells.

That’s also what makes the external-memory path possible: when the aggregation spills to disk, it spills bucket by bucket, and each bucket’s data is self-contained.

Resizing is amortised

A single-level table resizes by re-inserting every element. At the moment of the resize, the whole table has to be walked and every cell re-placed.

In a two-level table, each bucket resizes on its own, and since the buckets are filled roughly uniformly, the individual resizes happen at different times. The total work is the same, but it’s spread out, and — crucially — the pause is per-bucket rather than table-wide.

The comment in the header says it exactly:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:14
 * - delay during resizes is amortized, since the small hash tables will be resized separately;

When is the two-level table used?

The decision is a setting, and it’s a pair of settings now — one counting keys, one counting bytes:

 cpp
 
 
// src/Core/Settings.cpp:1500
 DECLARE(UInt64, group_by_two_level_threshold, 100000, R"(
From what number of keys, a two-level aggregation starts. 0 - the threshold is not set.
)", 0) \
 DECLARE(UInt64, group_by_two_level_threshold_bytes, 50000000, R"(
From what size of the aggregation state in bytes, a two-level aggregation begins to be used. 0 - the threshold is not set. Two-level aggregation is used when at least one of the thresholds is triggered.
)", 0) \

So by default a GROUP BY switches to the two-level structure once it holds 100 000 keys, or once its state grows to 50 MB — whichever happens first. Both thresholds can be forced from SQL:

 sql
 
 
SELECT user_id, count()
FROM events
GROUP BY user_id
SETTINGS group_by_two_level_threshold = 1000000,
 group_by_two_level_threshold_bytes = 50000000;

Setting either threshold to 0 disables that trigger — which is exactly what the pipeline does when a two-level aggregation would be useless.

The predicate that implements this lives in src/Interpreters/Aggregator.cpp:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:118
bool worthConvertToTwoLevel(
 size_t group_by_two_level_threshold, size_t result_size, size_t group_by_two_level_threshold_bytes, auto result_size_bytes)
{
 // params.group_by_two_level_threshold will be equal to 0 if we have only one thread to execute aggregation (refer to AggregatingStep::transformPipeline).
 return (group_by_two_level_threshold && result_size >= group_by_two_level_threshold)
 || (group_by_two_level_threshold_bytes && result_size_bytes >= static_cast<Int64>(group_by_two_level_threshold_bytes));
}

Note the && in each disjunct: a threshold of 0 disables that particular trigger, which is why the settings can be turned off individually.

The conversion itself happens during the aggregation, when a block has been added and the table is re-examined:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:2668
 bool worth_convert_to_two_level = worthConvertToTwoLevel(
 params.group_by_two_level_threshold, result_size, params.group_by_two_level_threshold_bytes, result_size_bytes);

 /** Converting to a two-level data structure.
 * It allows you to make, in the subsequent, an effective merge - either economical from memory or parallel.
 */
 if (result.isConvertibleToTwoLevel() && worth_convert_to_two_level)
 result.convertToTwoLevel();

and the conversion itself rebuilds the variant with the two-level type:

 cpp
 
 
// src/Interpreters/AggregatedDataVariants.cpp:263
void AggregatedDataVariants::convertToTwoLevel()
{
 ProfileEvents::increment(ProfileEvents::AggregationConvertedToTwoLevel);

 if (aggregator)
 LOG_TRACE(aggregator->log, "Converting aggregation data to two-level.");

 switch (type)
 {
#define M(NAME) \
 case Type::NAME: \
 NAME ## _two_level = std::make_unique<decltype(NAME ## _two_level)::element_type>(*(NAME)); \
 (NAME ## _two_level)->top_k_heap = std::move((NAME)->top_k_heap); \
 (NAME).reset(); \
 type = Type::NAME ## _two_level; \
 break;

 APPLY_FOR_VARIANTS_CONVERTIBLE_TO_TWO_LEVEL(M)

#undef M

 default:
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Wrong data variant passed.");
 }
}

The TwoLevelHashTable has a constructor that takes a single-level table and distributes it, which is what makes the conversion a one-liner:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:116
 template <typename Source>
 requires(!std::is_arithmetic_v<Source>)
 explicit TwoLevelHashTable(const Source & src)
 {
 typename Source::const_iterator it = src.begin();

 /// It is assumed that the zero key (stored separately) is first in iteration order.
 if (it != src.end() && it.getPtr()->isZero(src))
 {
 insert(it->getValue());
 ++it;
 }

 for (; it != src.end(); ++it)
 {
 const Cell * cell = it.getPtr();
 size_t hash_value = cell->getHash(src);
 size_t buck = getBucketFromHash(hash_value);
 impls[buck].insertUniqueNonZero(cell, hash_value);
 }
 }

The zero key is handled explicitly, because it lives outside the buffer and doesn’t follow the bucket rule — it goes to bucket 0.

The merge is the point

Now the reason the whole thing exists. The aggregation pipeline decides whether a two-level aggregation is even allowed:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:503
 bool allow_to_use_two_level_group_by = pipeline.getNumStreams() > 1 || params.max_bytes_before_external_group_by != 0;

Two conditions: either there’s more than one stream (so the merge is worth parallelising), or the query is allowed to spill to disk (so the bucket structure is needed to spill incrementally). If neither holds, the thresholds are zeroed out and the aggregation stays single-level:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:520
 if (!allow_to_use_two_level_group_by)
 {
 params.group_by_two_level_threshold = 0;
 params.group_by_two_level_threshold_bytes = 0;
 }

When the two-level table is used, the merge stage is a source that hands out one bucket at a time, and the pipeline runs several of them in parallel:

 cpp
 
 
// src/Processors/Transforms/AggregatingTransform.cpp:208
/// Worker which merges buckets for two-level aggregation.
/// Atomically increments bucket counter and returns merged result.
class ConvertingAggregatedToChunksWithMergingSource final : public ISource
{
public:
 static constexpr UInt32 NUM_BUCKETS = 256;

The worker takes the next bucket with an atomic counter and merges it across all the per-thread tables:

 cpp
 
 
// src/Processors/Transforms/AggregatingTransform.cpp:272
 Chunk generate() override
 {
 UInt32 bucket_num = shared_data->next_bucket_to_merge.fetch_add(1);

 if (bucket_num >= NUM_BUCKETS)
 {
 data.reset();
 return {};
 }

The atomic counter is the entire coordination mechanism. No lock, no work queue, no barrier. A worker that finishes bucket 3 immediately grabs bucket 4, and the workers end up balanced by the cost of the buckets they happen to pick up. Since the buckets partition the key space, no two workers ever touch the same key.

The actual merge of one bucket, in Aggregator::mergeBucketImpl, is “for each thread’s table, merge its bucket into the first thread’s bucket”:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5200
void NO_INLINE Aggregator::mergeBucketImpl(
 ManyAggregatedDataVariants & data, Int32 bucket, Arena * arena, std::atomic<bool> & is_cancelled) const
{
 /// We merge all aggregation results to the first.
 AggregatedDataVariantsPtr & res = data[0];

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5213
 auto & dst = getDataVariant<Method>(*res).data.impls[bucket];

Note that the bucket is an index into the impls[] array of the two-level table — the same array the routing used. That’s the whole contract: the bucket is a function of the key, therefore the bucket is a partition of the key space.

The same trick appears in the uniqExact aggregate function, which keeps a single-level set and converts it to a two-level one when it grows:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:386
 static bool worthConvertingToTwoLevel(size_t size) { return size > 100'000; }

and merges it bucket-wise in parallel:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:225
 auto thread_func = [&two_level_ptrs, &first_two_level, next_bucket_to_merge, &is_cancelled]()
 {
 while (true)
 {
 if (is_cancelled.load(std::memory_order_seq_cst))
 return;

 const auto bucket = next_bucket_to_merge->fetch_add(1);
 if (bucket >= NUM_BUCKETS)
 return;

 for (size_t j = 1; j < two_level_ptrs.size(); ++j)
 {
 if (is_cancelled.load(std::memory_order_seq_cst))
 return;

 first_two_level.impls[bucket].merge(two_level_ptrs[j]->impls[bucket]);
 }
 }
 };

with the same bounded-threads trick:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:246
 const size_t max_threads_to_enqueue = std::min<size_t>(thread_pool.getMaxThreads(), NUM_BUCKETS);

There’s no point in queueing more workers than there are buckets.

The string flavour

Strings get their own two-level table, because the string table isn’t a HashTable but a composite of five sub-tables (one per key size class). The routing therefore has to pick both the bucket and the size class:

 cpp
 
 
// src/Common/HashTable/TwoLevelStringHashTable.h:77
 /// The bucket-aware size-class dispatch: the shared implementation in `StringHashTable`
 /// classifies and packs the key and computes its hash; the resolver here picks the bucket
 /// from that hash and hands out the bucket's submap of the right class.
 template <typename Self, typename Func, typename KeyHolder>
 static auto ALWAYS_INLINE dispatch(Self & self, KeyHolder && key_holder, Func && func)
 {
 return Impl::dispatchOnKeyClass(
 [&](auto key_class, size_t hash_value) -> auto &
 { return self.impls[getBucketFromHash(hash_value)].template submapForClass<decltype(key_class)::value>(); },
 std::forward<KeyHolder>(key_holder),
 StringHashTableHash{},
 std::forward<Func>(func));
 }

256 buckets times 5 size classes is 1280 sub-tables. That’s a lot of bookkeeping for a table that usually isn’t huge, which is why the string tables are only converted to two-level when the aggregation really needs it.

Offsets across buckets

One thing the two-level table has to solve is numbering the cells globally. The aggregation code refers to cells by an offset into the whole table, and with 256 separate buffers there’s no single array to index into.

The solution is a prefix sum over the buckets:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:435
 /// Prefix sums that `offsetInternal` uses to number cells across all buckets.
 /// Call it after the last insert and before reading offsets, never while another thread reads them.
 void computeBucketPrefix()
 {
 bucket_cells_prefix.assign(NUM_BUCKETS, 0);
 size_t run = 0;
 for (UInt32 i = 0; i < NUM_BUCKETS; ++i)
 {
 bucket_cells_prefix[i] = run;
 run += impls[i].getBufferSizeInCells();
 }
 }

and the offset itself:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:448
 /// Number of the cell over all buckets. The zero cell gets 0. Any other cell gets its position in the
 /// concatenated bucket buffers plus one. So the number fits an array of `getBufferSizeInCells() + 1`.
 size_t offsetInternal(ConstLookupResult ptr) const
 {
 if constexpr (NUM_BUCKETS == 1)
 return impls[0].offsetInternal(ptr);

 const size_t buck = getBucketFromHash(ptr->getHash(*this));
 if (ptr->isZero(impls[buck]))
 return 0;

 chassert(!bucket_cells_prefix.empty(), "computeBucketPrefix must run before an offset is read");
 return bucket_cells_prefix[buck] + (ptr - impls[buck].buf) + 1;
 }

The “+1” is the zero key’s slot, which is numbered 0.

Trade-offs

What the two-level table buys:

A parallelisable merge. The merge of N per-thread tables becomes 256 independent merges with an atomic counter and no locks. The serial tail of a parallel GROUP BY mostly disappears.
Incremental spilling. Because a bucket is self-contained, the aggregation can write one bucket to disk and continue, and the merge can read buckets back one at a time.
Amortised resize pauses. The expensive rehash is per-bucket, so no single pause is as long.
A memory-efficient large bucket. The Robin Hood switch at 2^16 cells means a bucket that grows large halves its memory overhead per key, at the cost of a little CPU for the chain ordering.

What it costs:

One extra indirection on every probe. The single-level table computes hash & mask; the two-level table computes a shift, an array index, and then the mask. The header admits it: “Usually works a little slower than a simple hash table.”
256 allocations instead of one. Each sub-table has its own buffer, its own grower, its own size counter.
A weaker guarantee for large tables. The bucket is selected from 32 bits of the hash, so a table with more than 2^32 cells is explicitly out of scope.
Complexity at the boundaries. The zero key doesn’t belong to a bucket; the prefix sums must be recomputed after growth; the string variant has to route on two axes at once.

This is a good example of a structural optimisation. Rather than making the hash table faster, ClickHouse changes the shape of the data so that the expensive part of the query — the merge — can be decomposed into 256 embarrassingly parallel pieces.

Wrapping up

Here’s the thing: the whole design rests on one sentence. The bucket is a function of the key, so the bucket is a partition of the key space. Every other property — parallel merge, incremental spill, amortised resize — falls out of that.

What to go do about it: if you have a GROUP BY over a high-cardinality column that takes a long time to finish but not a long time to start, you’re probably hitting the serial merge. Check AggregationConvertedToTwoLevel in system.events — if it’s zero and your key count is above 100 000, the conversion isn’t happening, and that’s your smoking gun.

And if you want to tune it, group_by_two_level_threshold is the knob. Lower it and the merge gets parallel sooner; the cost is one extra indirection per probe on the smaller table.

ClickHouse Sorts in Parallel: Partial Sort, Merge Sorted Streams and External Sort

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse never sorts the whole dataset at once. It sorts each chunk, throws away rows that can’t make the top-N, merges the sorted streams, and spills runs to disk when memory runs out.

ORDER BY is the operation every database is worst at. It’s the one requirement that’s inherently global: the first row of the result depends on every row of the input.

Everything else — filtering, projection, aggregation — is local and streams beautifully. Sorting doesn’t.

The classic answer is “sort in chunks, then merge the sorted chunks”. Every system does it. But there are two quite different versions of that, and ClickHouse implements both, in two quite different classes.

One version assumes the input is already sorted on a prefix of the sort key — a MergeTree table read in primary-key order, or the output of a previous ORDER BY. Then the merge is cheap and the only work is stitching together runs that are already in order.

The other version assumes nothing. It sorts each chunk independently and then does a k-way merge. This is the general case, and it’s where the memory pressure lives — a k-way merge of N sorted runs needs all N runs resident at once.

This is a tour of both, of the trick that lets ClickHouse throw away rows it knows are useless, and of the machinery that spills sorted runs to disk when memory runs out.

The plan: what gets built

SortingStep decides which of the two regimes applies. You can see the difference in the query:

 sql
 
 
-- Full sort: the input is in no particular order, so each stream is sorted
-- on its own and the sorted streams are merged.
SELECT * FROM hits ORDER BY UserID, EventTime LIMIT 100;

 sql
 
 
-- Finish sorting: the table is sorted by (a, b) and the query asks for
-- (a, b) too, so the merge only has to stitch runs together.
SELECT * FROM hits WHERE a = 1 ORDER BY a, b LIMIT 100;

The “finish sorting” case is when the input is already sorted on a prefix:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:381
void SortingStep::finishSorting(
 QueryPipelineBuilder & pipeline, const SortDescription & input_sort_desc, const SortDescription & result_sort_desc, const UInt64 limit_)
{
 pipeline.addSimpleTransform(
 [&](const SharedHeader & header, QueryPipelineBuilder::StreamType stream_type) -> ProcessorPtr
 {
 if (stream_type != QueryPipelineBuilder::StreamType::Main)
 return nullptr;

 return std::make_shared<PartialSortingTransform>(header, result_sort_desc, limit_, threshold_tracker);
 });

Two transforms per stream: a PartialSortingTransform that sorts each chunk as it arrives, and a FinishSortingTransform that merges the sorted chunks. The “full sort” case is the same first transform followed by a MergeSortingTransform:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:522
 if (!skip_partial_sort || limit_)
 {
 pipeline.addSimpleTransform(
 [&](const SharedHeader & header, QueryPipelineBuilder::StreamType stream_type) -> ProcessorPtr
 {
 if (stream_type != sorted_stream_type)
 return nullptr;

 return std::make_shared<PartialSortingTransform>(header, result_sort_desc, limit_, threshold_tracker);
 });

Note the condition !skip_partial_sort || limit_. When there’s a LIMIT, the partial sort is always applied, even if the input is already sorted. That’s because the partial sort is what makes the limit cheap: it discards everything but the top limit rows of each chunk, so the merge only ever sees limit rows per chunk instead of the whole chunk.

`PartialSortingTransform`: sort the chunk, throw away the rest

The transform itself is short, and the interesting part is the threshold filter at the top:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:104
void PartialSortingTransform::transform(Chunk & chunk)
{
 if (chunk.getNumRows())
 {
 // The following code works with Blocks and will lose the number of
 // rows when there are no columns. We shouldn't get such block, because
 // we have to sort by at least one column.
 chassert(chunk.getNumColumns());
 }

 if (read_rows)
 read_rows->add(chunk.getNumRows());

 auto block = getInputPort().getHeader().cloneWithColumns(chunk.detachColumns());

 /// Materialize sort key columns that are ColumnReplicated to avoid index indirection during comparison and sorting.
 for (const auto & col_desc : description)
 {
 auto & column_entry = block.getByName(col_desc.column_name);
 column_entry.column = convertToFullColumnIfReplicationNotUseful(column_entry.column);
 }

 /** If we've saved columns from previously blocks we could filter all rows from current block
 * which are unnecessary for sortBlock(...) because they obviously won't be in the top LIMIT rows.
 */
 if (!sort_description_threshold_columns.empty() && !threshold_tracker)
 {
 UInt64 rows_num = block.rows();
 auto block_columns = extractRawColumns(block, description_with_positions);

 size_t result_size_hint = getFilterMask(
 block_columns, sort_description_threshold_columns,
 description, rows_num, filter, rows_to_compare, compare_results);

 /// Everything was filtered. Skip whole chunk.
 if (result_size_hint == 0)
 return;

 if (result_size_hint < rows_num)
 {
 for (auto & column : block)
 column.column = column.column->filter(filter, result_size_hint);
 }
 }

 sortBlock(block, description, limit);

Two things happen here, and the order matters.

First, the threshold filter. sort_description_threshold_columns holds the sort key of the current N-th best row seen so far — the “cutoff”. Every row in the new chunk that compares greater than the cutoff provably cannot be in the final top-N, so it’s dropped before the sort. The filter is computed with a vectorised column comparison:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:27
size_t getFilterMask(const ColumnRawPtrs & raw_block_columns, const Columns & threshold_columns,
 const SortDescription & description, size_t num_rows, IColumn::Filter & filter,
 PaddedPODArray<UInt64> & rows_to_compare, PaddedPODArray<Int8> & compare_results)
{
 filter.resize(num_rows);
 compare_results.resize(num_rows);

 if (description.size() == 1)
 {
 /// Fast path for single column
 raw_block_columns[0]->compareColumn(*threshold_columns[0], 0, nullptr, compare_results,
 description[0].direction, description[0].nulls_direction);
 }
 else
 {
 rows_to_compare.resize(num_rows);
 iota(rows_to_compare.data(), num_rows, UInt64(0));

 size_t size = description.size();
 for (size_t i = 0; i < size; ++i)
 {
 raw_block_columns[i]->compareColumn(*threshold_columns[i], 0, &rows_to_compare, compare_results,
 description[i].direction, description[i].nulls_direction);

 if (rows_to_compare.empty())
 break;
 }
 }

 size_t result_size_hint = 0;

 for (size_t i = 0; i < num_rows; ++i)
 {
 /// Leave only rows that are less then row from rhs.
 filter[i] = compare_results[i] < 0;
 result_size_hint += filter[i];
 }

 return result_size_hint;
}

compareColumn is the columnar comparison primitive: it compares a whole column against a single-row column and writes a per-row sign into compare_results. For a single sort column it’s one vectorised pass. For multiple columns it narrows the candidate set on each column in turn — the rows_to_compare array is the survivor list, and it’s emptied when no rows survive.

Second, sortBlock(block, description, limit). The limit argument makes the sort itself partial: ClickHouse sorts the chunk but only materialises the first limit rows of the permutation.

 cpp
 
 
// src/Interpreters/sortBlock.cpp:384
void sortBlock(Block & block, const SortDescription & description, UInt64 limit, IColumn::PermutationSortStability stability)
{
 IColumn::Permutation permutation;

#ifndef NDEBUG
 block.checkNumberOfRows();
#endif
 getBlockSortPermutationImpl(block, description, stability, limit, permutation);

#ifndef NDEBUG
 checkSortedWithPermutation(block, description, limit, permutation);
#endif

 if (permutation.empty())
 return;

 bool is_identity_permutation = isIdentityPermutation(permutation, limit);
 if (is_identity_permutation && limit == 0)
 return;

 size_t output_rows = limit ? std::min(static_cast<size_t>(limit), permutation.size()) : permutation.size();
 Columns columns = block.getColumns();
 transformColumnsWithSharedIndex(
 columns,
 [&](const ColumnPtr & col) { return is_identity_permutation ? col->cut(0, output_rows) : col->permute(permutation, limit); });
 block.setColumns(columns);
}

Note the output_rows clamp. With a limit of 10 and a chunk of 100 000 rows, the output chunk has 10 rows.

The sort still had to consider all 100 000 rows (the top 10 of an unsorted chunk aren’t the first 10), but the permutation only materialises 10 entries and the columns are permuted with that.

So the memory cost of ORDER BY ... LIMIT 10 is proportional to the chunk size, not to the limit. Each chunk is sorted in full, truncated to 10 rows, and the merge only ever sees 10 rows per chunk.

Updating the threshold

After sorting, the transform updates the threshold to the current chunk’s limit-th row:

 cpp
 
 
// src/Processors/Transforms/PartialSortingTransform.cpp:151
 /// Check if we can use this block for optimization.
 if ((min_limit_for_partial_sort_optimization <= limit || threshold_tracker) && limit <= block.rows())
 {
 /** If we filtered more than limit rows from block take block last row.
 * Otherwise take last limit row.
 *
 * If current threshold value is empty, update current threshold value.
 * If min block value is less than current threshold value, update current threshold value.
 */
 size_t min_row_to_compare = limit - 1;
 auto raw_block_columns = extractRawColumns(block, description_with_positions);

 if (sort_description_threshold_columns.empty() ||
 compareWithThreshold(raw_block_columns, min_row_to_compare, sort_description_threshold_columns, description))
 {
 size_t raw_block_columns_size = raw_block_columns.size();
 Columns sort_description_threshold_columns_updated(raw_block_columns_size);

 for (size_t i = 0; i < raw_block_columns_size; ++i)
 {
 MutableColumnPtr sort_description_threshold_column_updated = raw_block_columns[i]->cloneEmpty();
 sort_description_threshold_column_updated->insertFrom(*raw_block_columns[i], min_row_to_compare);
 /// Without this, `assertTypeEquality` may fail when comparing a ColumnNullable against a ColumnReplicated (received from JOIN).
 sort_description_threshold_columns_updated[i]
 = removeSpecialRepresentations(sort_description_threshold_column_updated->getPtr());
 }

 sort_description_threshold_columns = std::move(sort_description_threshold_columns_updated);

The threshold is monotonically non-increasing — it only ever gets better — so the filter gets more aggressive over time. The min_limit_for_partial_sort_optimization guard exists because for very small limits the bookkeeping isn’t worth it.

There’s a subtlety worth noticing: the filter is not stable. The comparison keeps rows that are strictly less than the threshold (compare_results[i] < 0), so a row equal to the threshold gets dropped. That’s fine for ORDER BY ... LIMIT without WITH TIES, and it’s why LimitTransform has a completely separate mechanism for WITH TIES.

`FinishSortingTransform`: merging streams that are already sorted

When the input is already sorted on a prefix of the key, the merge doesn’t need a heap at all — it just has to concatenate runs. The one hard part is handling a key that spans a chunk boundary.

 cpp
 
 
// src/Processors/Transforms/FinishSortingTransform.cpp:83
 /// Find the position of last already read key in current chunk.
 if (!chunks.empty())
 {
 size_t size = chunk.getNumRows();
 const auto & last_chunk = chunks.back();

 ssize_t low = -1;
 ssize_t high = size;
 while (high - low > 1)
 {
 ssize_t mid = (low + high) / 2;
 if (!less(last_chunk.getColumns(), chunk.getColumns(), last_chunk.getNumRows() - 1, mid, description_with_positions))
 low = mid;
 else
 high = mid;
 }

 size_t tail_pos = high;

 /// We need to save tail of chunk, because next chunk may starts with the same key as in tail
 /// and we should sort these rows in one portion.
 if (tail_pos != size)
 {
 auto source_columns = chunk.detachColumns();
 Columns tail_columns;

 for (auto & source_column : source_columns)
 {
 tail_columns.emplace_back(source_column->cut(tail_pos, size - tail_pos));
 source_column = source_column->cut(0, tail_pos);
 }

 chunks.emplace_back(std::move(source_columns), tail_pos);
 tail_chunk.setColumns(std::move(tail_columns), size - tail_pos);

 stage = Stage::Generate;
 return;
 }
 }

The binary search finds the first row of the new chunk whose key is greater than the last key of the previous chunk. Everything from there on has a key that might tie with the tail, so it’s cut off into tail_chunk and held back. The tail is re-merged with the next chunk, so that rows with equal keys end up in one sorted run rather than being split across two and appearing in the wrong order.

The stage = Stage::Generate is the switch that makes this work: the transform has to emit what it has so far, then go back to consuming.

`MergeSortingTransform`: accumulate, remerge, spill, merge

The general case is MergeSortingTransform, and it’s a state machine with three stages — Consume, Serialize, Generate — declared in the base class:

 cpp
 
 
// src/Processors/Transforms/SortingTransform.h:126
 enum class Stage : uint8_t
 {
 Consume = 0,
 Generate,
 Serialize,
 };

 Stage stage = Stage::Consume;

The base class prepare() is the dispatcher:

 cpp
 
 
// src/Processors/Transforms/SortingTransform.cpp:293
IProcessor::Status SortingTransform::prepare()
{
 if (stage == Stage::Serialize)
 {
 if (!processors.empty())
 return Status::UpdatePipeline;

 auto status = prepareSerialize();
 if (status != Status::Finished)
 return status;

 stage = Stage::Consume;
 }

 if (stage == Stage::Consume)
 {
 auto status = prepareConsume();
 if (status != Status::Finished)
 return status;

 stage = Stage::Generate;
 }

 /// stage == Stage::Generate

 if (!generated_prefix || !chunks.empty())
 return Status::Ready;

 if (!processors.empty())
 return Status::UpdatePipeline;

 return prepareGenerate();
}

The Status::UpdatePipeline returns are the graph-rewriting mechanism from the first article: when the sort decides it needs to spill to disk, it returns UpdatePipeline, the executor asks it for updatePipeline(), and the graph is rewired to insert a file sink and a file source.

Consume: accumulate chunks

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:94
void MergeSortingTransform::consume(Chunk chunk)
{
 /** Algorithm:
 * - read to memory blocks from source stream;
 * - if too many of them and if external sorting is enabled,
 * - merge all blocks to sorted stream and write it to temporary file;
 * - at the end, merge all sorted streams from temporary files and also from rest of blocks in memory.
 */

 /// If there were only const columns in sort description, then there is no need to sort.
 /// Return the chunk as is.
 if (description.empty())
 {
 generated_chunk = std::move(chunk);
 return;
 }

 removeConstColumns(chunk);
 compactReplicatedColumns(chunk);

 sum_rows_in_blocks += chunk.getNumRows();
 sum_bytes_in_blocks += chunk.allocatedBytes();
 chunks.push_back(std::move(chunk));

 /** If significant amount of data was accumulated, perform preliminary merging step.
 */
 if ((chunks.size() > 1
 && limit
 && limit * 2 < sum_rows_in_blocks /// 2 is just a guess.
 && remerge_is_useful
 && max_bytes_before_remerge
 && sum_bytes_in_blocks > max_bytes_before_remerge) || (threshold_tracker && (static_cast<double>(sum_rows_in_blocks) > static_cast<double>(limit) * 1.5)))
 {
 remerge();
 }

 /** If too many of them and if external sorting is enabled,
 * will merge blocks that we have in memory at this moment and write merged stream to temporary (compressed) file.
 * NOTE. It's possible to check free space in filesystem.
 */
 if (max_bytes_in_block_before_external_sort && sum_bytes_in_blocks > max_bytes_in_block_before_external_sort)
 {
 Int64 query_memory = getCurrentQueryMemoryUsage();
 if (!max_bytes_in_query_before_external_sort || query_memory > static_cast<Int64>(max_bytes_in_query_before_external_sort))
 {
 auto sink = prepareSpill();
 auto source = std::make_shared<BufferingFromFileSource>(sink->getPort().getSharedHeader(), sink->getHolder(), log);
 processors.emplace_back(source);
 processors.emplace_back(sink);
 stage = Stage::Serialize;
 }
 }
}

Three behaviours in one function.

Remerge. When there’s a LIMIT and enough data has accumulated, the in-memory chunks are merged now rather than later. This is the classic top-K trick: after merging, only limit rows survive, so the memory footprint of the accumulator collapses from “all data so far” to “the top limit rows”. The condition limit * 2 < sum_rows_in_blocks means “we have at least twice as many rows as we need”, so the remerge is worth it.

Spill. When the accumulated bytes exceed max_bytes_in_block_before_external_sort (and the whole query’s memory exceeds max_bytes_in_query_before_external_sort), the accumulated chunks are merged and written to a temporary file. The transform enters the Serialize stage, and the new processors — the file sink and the file source — are queued for updatePipeline().

The limit on remerge. remerge_is_useful is a latch:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:333
void MergeSortingTransform::remerge()
{
 LOG_DEBUG(log, "Re-merging intermediate ORDER BY data ({} blocks with {} rows) to save memory consumption", chunks.size(), sum_rows_in_blocks);

 /// NOTE Maybe concat all blocks and partial sort will be faster than merge?
 MergeSorter remerge_sorter(std::make_shared<const Block>(header_without_constants), std::move(chunks), description, max_merged_block_size, limit);

 Chunks new_chunks;
 size_t new_sum_rows_in_blocks = 0;
 size_t new_sum_bytes_in_blocks = 0;

 while (auto chunk = remerge_sorter.read())
 {
 new_sum_rows_in_blocks += chunk.getNumRows();
 new_sum_bytes_in_blocks += chunk.allocatedBytes();
 new_chunks.emplace_back(std::move(chunk));
 }

 LOG_DEBUG(log, "Memory usage is lowered from {} to {}", ReadableSize(sum_bytes_in_blocks), ReadableSize(new_sum_bytes_in_blocks));

 /// If the memory consumption was not lowered enough - we will not perform remerge anymore.
 if (remerge_lowered_memory_bytes_ratio > 0.0 && (static_cast<double>(new_sum_bytes_in_blocks) * remerge_lowered_memory_bytes_ratio > static_cast<double>(sum_bytes_in_blocks)))
 {
 remerge_is_useful = false;
 LOG_DEBUG(log, "Re-merging is not useful (memory usage was not lowered by remerge_sort_lowered_memory_bytes_ratio={})", remerge_lowered_memory_bytes_ratio);
 }

The default ratio is 2.0 (remerge_sort_lowered_memory_bytes_ratio), so the remerge must halve the memory to be considered worth doing. If it doesn’t, the flag is cleared and no further remerges happen — because a remerge that doesn’t reduce memory is pure loss: you paid for a merge and got nothing.

The default threshold is max_bytes_before_remerge_sort = 1000000000, one gigabyte.

Serialize and Generate

Serialize is the stage where the accumulated chunks are drained into the file sink, one sorted run at a time:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:252
void MergeSortingTransform::serialize()
{
 current_chunk = merge_sorter->read();
 if (!current_chunk)
 merge_sorter.reset();
 /// A read of `MergeSorter::Mode::MergeUniqueChunks` that consumes only duplicates returns no rows. The
 /// run gets no block for it, and the next call reads on.
 else if (!current_chunk.hasRows())
 current_chunk.clear();
}

Generate is where the final merge happens. If nothing was spilled, the answer is a plain in-memory merge of the accumulated chunks:

 cpp
 
 
// src/Processors/Transforms/MergeSortingTransform.cpp:263
void MergeSortingTransform::generate()
{
 if (!generated_prefix)
 {
 if (temporary_files_num == 0)
 {
 merge_sorter = std::make_unique<MergeSorter>(
 std::make_shared<const Block>(header_without_constants),
 std::move(chunks),
 description,
 max_merged_block_size,
 limit,
 merge_mode);
 }
 else
 {
 ProfileEvents::increment(external_merge_event);
 LOG_INFO(log, "There are {} temporary sorted parts to merge", temporary_files_num);

 processors.emplace_back(
 std::make_shared<MergeSorterSource>(
 std::make_shared<const Block>(header_without_constants),
 std::move(chunks),
 description,
 max_merged_block_size,
 limit,
 merge_mode));
 }

 generated_prefix = true;
 }

If there were temporary files, a MergeSorterSource is added to the pipeline and the graph is rewired so that the spilled runs (read back from disk) and the in-memory tail are merged together by a MergingSortedTransform — which is a proper k-way merge with a heap.

The k-way merge

MergingSortedTransform uses MergingSortedAlgorithm, which keeps one SortCursor per input stream in a heap and pops the smallest key. The interesting optimisation is the “totally less” shortcut:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:547
template <typename TSortingHeap>
IMergingAlgorithm::Status MergingSortedAlgorithm::mergeImpl(TSortingHeap & queue)
{
 /// Take rows in required order and put them into `merged_data`, while the rows are no more than `max_block_size`
 while (queue.isValid())
 {
 if (merged_data.hasEnoughRows())
 return Status(merged_data.pull());

 auto current = queue.current();

 if (current.impl->isLast() && current_inputs[current.impl->order].skip_last_row)
 {
 /// Get the next block from the corresponding source, if there is one.
 queue.removeTop();
 return Status(current.impl->order);
 }

 if (current.impl->isFirst()
 && !current_inputs[current.impl->order].skip_last_row /// Ignore optimization if last row should be skipped.
 && (queue.size() == 1
 || (queue.size() >= 2 && current.totallyLessOrEquals(queue.nextChild()))))
 {
 /** This is special optimization if current cursor is totally less than next cursor.
 * We want to insert current cursor chunk directly in merged data.
 *
 * First if merged_data is not empty we need to flush it.
 * We will get into the same condition on next mergeBatch call.
 *
 * Then we can insert chunk directly in merged data.
 */

 if (merged_data.mergedRows() != 0)
 return Status(merged_data.pull());

 /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl)
 size_t source_num = current.impl->order;

 auto chunk_num_rows = current_inputs[source_num].chunk.getNumRows();

 UInt64 total_merged_rows_after_insertion = merged_data.mergedRows() + chunk_num_rows;
 bool limit_reached = limit && total_merged_rows_after_insertion >= limit;

 if (limit && total_merged_rows_after_insertion > limit)
 chunk_num_rows -= total_merged_rows_after_insertion - limit;

 insertChunk(source_num);

 /// We will get the next block from the corresponding source, if there is one.
 queue.removeTop();

 auto status = Status(merged_data.pull(), limit_reached);

 if (!limit_reached)
 status.required_source = source_num;

 return status;
 }

totallyLessOrEquals asks: “is the last key of this stream less than or equal to the first key of the next-smallest stream?” If yes, then the entire chunk of the current stream comes before everything in the other streams, and it can be copied wholesale instead of row by row.

That turns the merge from O(rows log streams) comparisons into a memcpy for the common case. When the data is nearly sorted — which it usually is, because each run came from an already-sorted source — most chunks take the fast path.

Trade-offs

What this buys you.

The LIMIT optimisation is the big one. ORDER BY x LIMIT 10 over a hundred million rows costs: sort each chunk (in memory, chunk by chunk), truncate to 10 rows, merge 10 rows per chunk. Memory is O(chunk size × threads), not O(dataset). And the threshold filter improves on that further: once the cutoff is tight, whole chunks are discarded without being sorted at all — if (result_size_hint == 0) return;.

The remerge trick generalises the same idea beyond LIMIT. Even without a limit, merging periodically keeps the accumulator’s memory bounded and converts a “merge N runs at the end” problem into a series of smaller merges, which is also better for the page cache.

The external sort path is honest about the fact that ORDER BY is a blocking operation: if the data doesn’t fit, it goes to disk, in sorted runs, and the final merge is a k-way merge over those runs. The runs are written with the configured temporary-files codec (usually LZ4), so the disk traffic is compressed.

What it costs.

Sorting is still the slowest thing in the query, and the partial sort doesn’t help the general case. Without a LIMIT, every chunk must be fully sorted, and the merge must touch every row. sortBlock allocates a permutation of the chunk’s row indices — 8 bytes per row — and the column permutation that follows is a random-access pass over the column data, which is cache-hostile for wide rows.

The threshold filter has a cost of its own: a full vectorised comparison of the chunk against the cutoff, even when it rejects nothing. For a LIMIT query where the data isn’t sorted, the filter will usually reject nothing on the first chunk and progressively more later, so the average benefit depends heavily on the data.

The remerge is a gamble. It costs a merge, and it only pays off if the merge actually reduces memory by the configured ratio. The remerge_is_useful latch means the first unhelpful remerge disables all future ones — but the first one was already paid for. And the threshold for even attempting a remerge is one gigabyte of accumulated data, so the memory spike before the first remerge is real.

Finally, the tail-splitting logic in FinishSortingTransform is subtle and easy to get wrong. If a key spans a chunk boundary and the tail isn’t held back, the rows with equal keys are emitted in two separate sorted runs and the final order among them is wrong. The binary search that finds the split point is O(log n) per chunk, which is cheap — but the reason it exists is not obvious from the code, which is why the comment is there.

Wrapping up

The real win: the threshold filter. ORDER BY ... LIMIT N in ClickHouse isn’t “sort everything, take N”. It’s “sort a chunk, keep N, discard the rest, and tighten the cutoff so later chunks mostly die at the door”. The query gets faster the longer it runs.

What to do about it: if your ORDER BY ... LIMIT query is slow, check whether the sort key matches the table’s primary key. If it does, you get the “finish sorting” path — a merge of already-sorted runs, which is close to free. If it doesn’t, you’re paying for a full sort plus a threshold filter that has to reject everything by comparison.

And if you’re spilling to disk, watch the external_merge_event profile event and max_bytes_before_external_sort. That’s the knob that decides how much of your sort stays in RAM.

ClickHouse Skips Data: Granules, Marks, and Bit-Packed Indexes

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: A MergeTree scan isn’t a full scan. Parts are cut into granules, each granule gets a bit-packed mark, and the primary key index stores one key value per granule — so ClickHouse can prove a granule can’t match and skip it entirely.

A SELECT on a MergeTree table is, mechanically, a scan of a set of files. The only reason it isn’t a full scan is that the files are chopped into pieces, and each piece carries enough metadata for the engine to prove that the piece cannot contain a matching row.

ClickHouse calls those pieces granules, the metadata is called a mark, and the mechanism that ties them together is the primary key index — one key value per granule, held in memory.

This is the single most important performance idea in MergeTree, and it’s built out of three small, very specific tricks:

Granules are sized in bytes, not rows, so a granule of fat rows isn’t a granule of a million rows.
Marks are bit-packed in memory — three bytes per mark for an integer column, instead of the sixteen bytes a naive {offset, offset_in_block} pair would take.
The primary key index is trimmed: if a key column is nearly constant within a part, the key columns after it aren’t kept in memory at all, because they can’t narrow anything down.

Granules: sized in bytes, not rows

The classic description of MergeTree granularity is “8192 rows”, and that’s the default value of the index_granularity setting. But that’s only the cap. The actual mechanism computes the number of rows per granule from the size of the data being written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:115
size_t computeIndexGranularity(
 size_t rows,
 size_t bytes_uncompressed,
 size_t index_granularity_bytes,
 size_t fixed_index_granularity_rows,
 bool blocks_are_granules,
 bool can_use_adaptive_index_granularity)
{
 size_t index_granularity_for_block = 0;

 if (!can_use_adaptive_index_granularity)
 {
 index_granularity_for_block = fixed_index_granularity_rows;
 }
 else
 {
 if (blocks_are_granules)
 {
 index_granularity_for_block = rows;
 }
 else if (index_granularity_bytes != 0 && bytes_uncompressed >= index_granularity_bytes)
 {
 /// index_granularity_bytes == 0 disables adaptive sizing upstream, so this branch is
 /// unreachable with a zero divisor; the explicit check makes that visible to static analysis.
 size_t granules_in_block = bytes_uncompressed / index_granularity_bytes;
 index_granularity_for_block = rows / granules_in_block;
 }
 else
 {
 size_t size_of_row_in_bytes = std::max(bytes_uncompressed / rows, 1UL);
 index_granularity_for_block = index_granularity_bytes / size_of_row_in_bytes;
 }
 }

 /// We should be less or equal than fixed index granularity.
 /// But if block size is a granule size then do not adjust it.
 /// Granularity greater than fixed granularity might come from compact part.
 if (!blocks_are_granules)
 index_granularity_for_block = std::min(fixed_index_granularity_rows, index_granularity_for_block);

 /// Very rare case when index granularity bytes less than single row.
 if (index_granularity_for_block == 0)
 index_granularity_for_block = 1;

 return index_granularity_for_block;
}

Read the interesting branch — the else at the bottom. If the block is smaller than index_granularity_bytes, then

text

 rows_per_granule = index_granularity_bytes / average_row_size

The granule stays a constant number of bytes. The default index_granularity_bytes is 10 MB, so a granule is 10 MB of data whether its rows are 10 bytes wide or 10 KB wide.

Why does that matter? Because a granule is the unit of skipping. If granules were a fixed 8192 rows, then a table of 1 KB rows would have 8 MB granules (fine), but a table of 100 KB rows would have 800 MB granules (useless — you can’t skip anything smaller than 800 MB). Sizing in bytes makes the granule a predictable, bounded amount of IO, and therefore a predictable, bounded amount of waste when you have to read a granule that does contain matches.

The price is that the row count of a granule is no longer constant, so it has to be remembered. The adaptive granularity object stores partial sums:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularityAdaptive.h:9
/// Class that stores adaptive index granularity.
/// Inside it contains vector of partial sums of rows after mark:
/// |-----|---|----|----|
/// | 5 | 8 | 12 | 16 |
class MergeTreeIndexGranularityAdaptive final : public MergeTreeIndexGranularity

One size_t per mark, and the row count of any mark is a subtraction. The alternative — a constant number of rows per granule — is available as use_const_adaptive_granularity, and its setting description explains the trade-off exactly:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:2174
 DECLARE(Bool, use_const_adaptive_granularity, false, R"(
Non-const adaptive granularity (the default, `index_granularity_bytes` is not `0`) keeps
granules at a constant size in bytes, so their row count varies: for every
written block, rows per granule is `index_granularity_bytes` divided by the
average size of a row in that block, capped at `index_granularity`. Because it differs
from granule to granule, the row count of every granule in the part has to be kept in
memory, 8 bytes each, which on large tables adds up to tens of gigabytes.

“Tens of gigabytes” — that’s the cost of the byte-granularity guarantee, and it’s the reason the constant-row variant exists at all. On a table with 100 million parts × 100 marks each, 8 bytes per mark is 80 GB of pure bookkeeping.

The marks themselves are appended as the block is written:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:162
void fillIndexGranularityForWidePart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 index_granularity.appendMark(index_granularity_for_block);
}

A Wide part gets exactly index_granularity_for_block rows per mark. A Compact part, where all columns share one file, has to be more careful:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeIndexGranularity.cpp:169
void fillIndexGranularityForCompactPart(
 MergeTreeIndexGranularity & index_granularity, size_t index_offset, size_t index_granularity_for_block, size_t rows_in_block)
{
 for (size_t current_row = index_offset; current_row < rows_in_block; current_row += index_granularity_for_block)
 {
 size_t rows_left_in_block = rows_in_block - current_row;

 /// Try to extend last granule if block is large enough
 /// or it isn't first in granule (index_offset != 0).
 if (rows_left_in_block < index_granularity_for_block &&
 (rows_in_block >= index_granularity_for_block || index_offset != 0))
 {
 // If enough rows are left, create a new granule. Otherwise, extend previous granule.
 // So, real size of granule differs from index_granularity_for_block not more than 50%.
 if (rows_left_in_block * 2 >= index_granularity_for_block)
 index_granularity.appendMark(rows_left_in_block);
 else
 index_granularity.addRowsToLastMark(rows_left_in_block);
 }
 else
 {
 index_granularity.appendMark(index_granularity_for_block);
 }
 }
}

The comment “real size of granule differs from index_granularity_for_block not more than 50%” is the invariant: a short tail is either given its own granule (if it’s at least half a granule) or merged into the previous one.

Marks: where in the file does each granule start?

A granule is a range of rows. To read it, you need to know where in the compressed .bin file its rows live. That’s the mark:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:14
/** Mark is the position in the compressed file. The compressed file consists of adjacent compressed blocks.
 * Mark is a tuple - the offset in the file to the start of the compressed block, the offset in the decompressed block to the start of the data.
 */
struct MarkInCompressedFile
{
 size_t offset_in_compressed_file;
 size_t offset_in_decompressed_block;

Two numbers, both size_t, so sixteen bytes per mark in the naive layout. For a part with 100 marks that’s nothing. For a table with 100 000 parts of 1 GB each (100 TB), 100 marks per part × 16 bytes × 100 000 parts = 160 MB of index — and that’s only if every mark of every column is resident.

Which is why the in-memory representation isn’t a struct array:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:30
/**
 * In-memory representation of an array of marks.
 *
 * Uses an ad-hoc compression scheme that decreases memory usage while allowing
 * random access in O(1) time.
 * This is independent from the marks *file* format, which may be uncompressed
 * or use a different compression method.
 *
 * Typical memory usage:
 * * ~3 bytes/mark for integer columns
 * * ~5 bytes/mark for string columns
 * * ~0.3 bytes/mark for trivial marks in auxiliary dict files of LowCardinality columns
 */
class MarksInCompressedFile

Three bytes per mark. How?

The trick is that marks are not random. The comment in the header spells out the regularities the compression exploits:

 cpp
 
 
// src/Formats/MarkInCompressedFile.h:51
 /** We need to store a sequence of marks, each consisting of two 64-bit integers:
 * offset_in_compressed_file and offset_in_decompressed_block. We'll call them x and y for
 * convenience, since compression doesn't care what they mean. The compression exploits the
 * following regularities:
 * * y is usually zero.
 * * x usually increases steadily.
 * * Differences between x values in nearby marks usually fit in much fewer than 64 bits.
 *
 * We split the sequence of marks into blocks, each containing MARKS_PER_BLOCK marks.
 * (Not to be confused with data blocks.)
 * For each mark, we store the difference [value] - [min value in the block], for each of the
 * two values in the mark. Each block specifies the number of bits to use for these differences
 * for all marks in this block.
 * The smaller the blocks the fewer bits are required, but the bigger the relative overhead of
 * block headers.
 *
 * Packed marks and block headers all live in one contiguous array.
 */

So: 256 marks per block. Per block, store the minimum x and y and the number of bits needed to represent the spread within the block. Then store every mark as a delta from the block minimum, in that many bits.

The block header computation:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:122
void MarksInCompressedFile::Builder::flushBlock(const MarkInCompressedFile * data, size_t count)
{
 chassert(count > 0 && count <= MARKS_PER_BLOCK);

 size_t block_idx = marks_flushed / MARKS_PER_BLOCK;
 BlockInfo & block = blocks[block_idx];
 block.bit_offset_in_packed_array = packed_bits;

 /// Compute block metadata: min values, bit widths.
 size_t max_x = 0;
 size_t max_y = 0;
 for (size_t i = 0; i < count; ++i)
 {
 block.min_x = std::min(block.min_x, data[i].offset_in_compressed_file);
 max_x = std::max(max_x, data[i].offset_in_compressed_file);
 block.min_y = std::min(block.min_y, data[i].offset_in_decompressed_block);
 max_y = std::max(max_y, data[i].offset_in_decompressed_block);
 block.trailing_zero_bits_in_y
 = std::min(block.trailing_zero_bits_in_y, static_cast<UInt8>(getTrailingZeroBits(data[i].offset_in_decompressed_block)));
 }

 block.bits_for_x = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits(max_x - block.min_x));
 block.bits_for_y
 = static_cast<UInt8>(sizeof(size_t) * 8 - getLeadingZeroBits((max_y - block.min_y) >> block.trailing_zero_bits_in_y));

Two ideas in six lines.

Delta encoding per block. bits_for_x is the number of bits needed to represent the range of x values inside the block, not the magnitude of x itself. If a block of 256 marks spans 64 KB of compressed file, that’s 19 bits per mark, not 64.

Trailing-zero stripping for y. y is the offset inside the decompressed block, and it’s usually zero — a granule normally starts at the beginning of a compressed block. The code computes the number of trailing zero bits common to all the y values in the block and shifts them all right by that amount before packing. For a block where every y is zero, the shift is 63 and bits_for_y becomes zero: the y coordinate costs no bits at all.

The write path, showing both encodings:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:153
 /// Write bit-packed deltas.
 size_t bit_offset = packed_bits;
 for (size_t i = 0; i < count; ++i)
 {
 writeBitsPacked64(packed.data(), bit_offset, data[i].offset_in_compressed_file - block.min_x);
 writeBitsPacked64(
 packed.data(),
 bit_offset + block.bits_for_x,
 (data[i].offset_in_decompressed_block - block.min_y) >> block.trailing_zero_bits_in_y);
 bit_offset += block.bits_for_x + block.bits_for_y;
 }

And the read path, which has to be O(1) — a mark lookup happens for every granule the reader touches:

 cpp
 
 
// src/Formats/MarkInCompressedFile.cpp:47
std::tuple<const MarksInCompressedFile::BlockInfo *, size_t> MarksInCompressedFile::lookUpMark(size_t idx) const
{
 size_t block_idx = idx / MARKS_PER_BLOCK;
 const BlockInfo & block = blocks[block_idx];
 size_t offset = block.bit_offset_in_packed_array + (idx - block_idx * MARKS_PER_BLOCK) * (block.bits_for_x + block.bits_for_y);
 return {&block, offset};
}

Divide by 256 to find the block header, then a shift-and-mask to find the mark inside it. That’s the whole random-access structure: one array division and one multiply. Compare that to a compressed list of marks, which would need a binary search.

Where do the marks come from at write time?

The mark is written when the compressed block is flushed, so it’s simply the state of the hashing/compression stream at that moment:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp:351
 MarkInCompressedFile mark{stream->plain_hashing.count(), stream->compressed_hashing.offset()};

plain_hashing.count() is the number of uncompressed bytes written so far — that’s y, the offset inside the decompressed block. And compressed_hashing.offset() is the number of compressed bytes written so far — that’s x, the offset of the compressed block in the file.

The mark cache

Marks are read from .mrk files and cached in a global LRU keyed by the file path:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeMarksLoader.cpp:313
 if (mark_cache)
 {
 auto key = MarkCache::hash(data_part_storage->getDiskName() + ":" + (fs::path(data_part_storage->getFullPath()) / mrk_path).string());

 if (save_marks_in_cache)
 {
 auto callback = [this] { return loadMarksImpl(); };
 loaded_marks = mark_cache->getOrSet(key, callback);
 }
 else
 {
 loaded_marks = mark_cache->getWithoutMetrics(key);
 if (!loaded_marks)
 loaded_marks = loadMarksImpl();
 }
 }

Note the two paths. getOrSet records hits and misses and inserts the loaded marks into the cache. getWithoutMetrics only looks: it’s used when the marks are wanted for a merge or a mutation, which shouldn’t evict the query’s working set or pollute the cache statistics.

That’s a small but real piece of cache hygiene — background work must not steal the cache from foreground queries.

The cache itself weights each entry by its compressed size plus a fixed overhead:

 cpp
 
 
// src/Storages/MergeTree/MarkCache.h:23
/// Estimate of number of bytes in cache for marks.
struct MarksWeightFunction
{
 /// We spent additional bytes on key in hashmap, linked lists, shared pointers, etc ...
 static constexpr size_t MARK_CACHE_OVERHEAD = 128;

 size_t operator()(const MarksInCompressedFile & marks) const
 {
 return marks.approximateMemoryUsage() + MARK_CACHE_OVERHEAD;
 }
};

The 128-byte constant is the per-entry overhead of the hash map node, the list node, the shared pointer, and the MarksInCompressedFile object itself. For a part with 100 marks, the compressed marks are ~300 bytes and the overhead is 128 — so the accounting is honest about the fact that the bookkeeping dominates for small parts.

The primary key index: one key value per granule

The marks tell you where the bytes are. The primary key index tells you what’s in them. It’s a separate file, primary.idx, containing the sorting key values of the first row of every granule:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1935
std::shared_ptr<IMergeTreeDataPart::Index> IMergeTreeDataPart::loadIndex() const
{
 auto component_guard = Coordination::setCurrentComponent("IMergeTreeDataPart::loadIndex");
 /// Memory for index must not be accounted as memory usage for query, because it belongs to a table.
 MemoryTrackerBlockerInThread temporarily_disable_memory_tracker;

and the loop that reads it is the whole format — marks_count rows of key_size values, serialized one after another:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1972
 FormatSettings format_settings;
 for (size_t i = 0; i < marks_count; ++i)
 {
 for (size_t j = 0; j < key_size; ++j)
 key_serializations[j]->deserializeBinary(*loaded_index[j], *index_file, format_settings);
 }

marks_count rows of key_size values, serialized one after another. The in-memory form is a set of ColumnPtrs — one column per key column, each holding marks_count values — which is exactly the layout a KeyCondition wants to evaluate a predicate against.

The trimming trick

Now the interesting part. The primary key may be (a, b, c). If a is almost constant within a part — say, a low-cardinality column — then keeping b and c in memory is a waste. A query filtering on b can’t be answered from the index anyway, because a doesn’t narrow anything down. So the index is trimmed:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1895
void IMergeTreeDataPart::optimizeIndexColumns(size_t marks_count, Columns & index_columns) const
{
 if (marks_count == 0)
 {
 chassert(isEmpty());
 return;
 }

 /// Do not optimize index for patch parts because patch parts
 /// use manual index analysis which requires all index columns.
 if (info.isPatch())
 return;

 size_t key_size = index_columns.size();
 Float64 ratio_to_drop_suffix_columns = static_cast<double>((*storage.getSettings())[MergeTreeSetting::primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns]);

 /// Cut useless suffix columns, if necessary.
 if (key_size > 1 && ratio_to_drop_suffix_columns > 0 && ratio_to_drop_suffix_columns < 1)
 {
 for (size_t j = 0; j < key_size - 1; ++j)
 {
 size_t num_changes = 0;
 for (size_t i = 1; i < marks_count; ++i)
 {
 if (0 != index_columns[j]->compareAt(i, i - 1, *index_columns[j], 0))
 ++num_changes;
 }

 if (static_cast<Float64>(num_changes) / static_cast<Float64>(marks_count) >= ratio_to_drop_suffix_columns)
 {
 key_size = j + 1;
 index_columns.resize(key_size);
 break;
 }
 }

The rule: while the fraction of granules in which key column j changes is at least primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns (default 0.9), column j is informative, so keep it and move on. The first column that changes in fewer than 90% of granules is “nearly constant” within this part — the part is probably all one value of it — and then everything after it is dropped.

This is a per-part decision. A part that covers one day of data might have a constant country column and a varying user_id; the index keeps country and drops user_id. Another part, spanning many countries, keeps both.

The consequence is that index analysis must know which columns are actually available — and the code accounts for it, treating the missing suffix columns as “constant over the whole universe”, which relaxes the analysis to a safe over-approximation.

The primary index cache

For very large tables, keeping every part’s primary index in memory isn’t possible, so it can be cached in an LRU instead:

 cpp
 
 
// src/Storages/MergeTree/PrimaryIndexCache.h:29
/** Cache of primary index for MergeTree tables.
 * Primary index is a list of columns from primary key
 * that store first row for each granule of data part.
 */
class PrimaryIndexCache : public CacheBase<UInt128, PrimaryIndex, UInt128TrivialHash, PrimaryIndexWeightFunction>

and the part consults it lazily:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:706
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::getIndex() const
{
 std::scoped_lock lock(index_mutex);

 if (index)
 return index;

 if (auto index_cache = storage.getPrimaryIndexCache())
 return loadIndexToCache(*index_cache);

 index = loadIndex();
 return index;
}

with the key being the disk name plus the part’s path, so that two parts with the same name on different disks don’t collide:

 cpp
 
 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:726
IMergeTreeDataPart::IndexPtr IMergeTreeDataPart::loadIndexToCache(PrimaryIndexCache & index_cache) const
{
 auto key = PrimaryIndexCache::hash(getDataPartStorage().getDiskName() + ":" + getRelativePathOfActivePart());
 auto callback = [this] { return loadIndex(); };
 return index_cache.getOrSet(key, callback);
}

Now the payoff: two different search algorithms

With marks and the primary index in hand, a query has to answer “which granules can contain a matching row?”. ClickHouse has two algorithms for this, and it picks between them based on the shape of the predicate.

The decision point is whether the predicate describes a continuous range of the sorting key:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2705
 bool key_condition_exact_range = !key_condition_useful || key_condition.matchesExactContinuousRange();
 bool part_offset_condition_exact_range = !part_offset_condition_useful || part_offset_condition->matchesExactContinuousRange();
 bool total_offset_condition_exact_range = !total_offset_condition_useful || total_offset_condition->matchesExactContinuousRange();
 const String & part_name = part->isProjectionPart() ? fmt::format("{}.{}", part->name, part->getParentPart()->name) : part->name;

 if (!key_condition_exact_range || !part_offset_condition_exact_range || !total_offset_condition_exact_range)
 {
 // Do exclusion search, where we drop ranges that do not match

Algorithm 1: binary search (continuous range)

If the predicate describes a single continuous interval of the sorting key — WHERE key >= 100 AND key < 200 — then the matching granules form a contiguous range in the index, and two binary searches find its boundaries:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:2801
 /// Invariant: !check_in_range(part_range.begin..searched_left).can_be_true
 /// check_in_range(part_range.begin..searched_right).can_be_true
 /// (part_range.end + 1 is a sentinel out-of-bounds index, such that by definition
 /// check_in_range(x..part_range.end+1).can_be_true is true. The binary search will never
 /// actually call check_in_range with this out-of-bounds index.)
 /// If the condition is not true anywhere, we'll end up with searched_left == part_range.end.
 size_t searched_left = part_range.begin;
 size_t searched_right = part_range.end + 1;

 while (searched_left + 1 < searched_right)
 {
 const size_t middle = (searched_left + searched_right) / 2;
 MarkRange range(0, middle);
 if (check_in_range(range, BoolMask::consider_only_can_be_true).can_be_true)
 searched_right = middle;
 else
 searched_left = middle;
 ++steps;
 }
 result_range.begin = searched_left;
 LOG_TEST(log, "Found (LEFT) boundary mark: {}", searched_left);

check_in_range(0, middle) asks: “could any row in granules [0, middle) match?” The KeyCondition evaluates the predicate against the index values at the two ends of the range and the monotonicity of the key columns. Because the key is sorted, the answer is monotone in middle, so binary search is valid. O(log n) index checks for the whole part.

Algorithm 2: generic exclusion search (everything else)

Now consider WHERE key2 = 5 where key2 is the second column of the sorting key, or WHERE key = 5 OR key = 900000. The matching granules are not contiguous, so a binary search can’t be used.

ClickHouse falls back to an exclusion search: split the mark range recursively, drop the parts that cannot match.

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:61
void searchUnlimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 for (const auto & initial_range : initial_ranges)
 {
 std::vector<MarkRange> ranges_stack = {initial_range};
 while (!ranges_stack.empty())
 {
 MarkRange range = ranges_stack.back();
 ranges_stack.pop_back();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 appendWithMaxGap(result.ranges, range, search_settings.min_marks_for_seek);

 /// Unlike `ranges`, exact ranges must never absorb the gap between two accepted
 /// ranges: every mark of an exact range has to fully match the condition, while the
 /// skipped marks in between do not.
 if (collect_exact_ranges && exact)
 appendWithMaxGap(result.exact_ranges, range, 0);
 }
 else
 {
 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { ranges_stack.push_back(subrange); });
 }
 }
 }
}

This is a depth-first search over mark ranges. The check returns a BoolMask with two bits: can_be_true and can_be_false.

!can_be_true → the whole range cannot match → drop it.
!can_be_false → every row in the range matches → keep it whole.
otherwise → the range is a mixture → split it into merge_tree_coarse_index_granularity (default 8) subranges and recurse.

Splitting into 8 rather than 2 is deliberate. It’s a breadth-ish compromise: splitting in half is the classic binary split, but each split costs one check_in_range call, and each check isn’t free — it evaluates a KeyCondition against the index columns at the range boundaries. Eight subranges per step means the search converges in about log_8(n) steps while doing 8× the work per step.

The step budget

The unlimited search has no bound. A pathological predicate could make it check thousands of ranges, so there’s a budget:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:119
void searchLimited(
 const MarkRanges & initial_ranges,
 const MarkRangeCheck & check_in_range,
 const GenericExclusionSearchSettings & search_settings,
 bool collect_exact_ranges,
 GenericExclusionSearchResult & result)
{
 std::vector<MarkRange> heap_storage(initial_ranges.begin(), initial_ranges.end());
 std::priority_queue<MarkRange, std::vector<MarkRange>, LessBySize> queue(LessBySize{}, std::move(heap_storage));

 /// Ranges are accepted out of spatial order, so they are collected raw and merged after sorting.
 std::vector<MarkRange> accepted;
 std::vector<MarkRange> accepted_exact;

 while (!queue.empty())
 {
 MarkRange range = queue.top();
 queue.pop();

 ++result.num_steps;

 BoolMask mask = check_in_range(range);

 if (!mask.can_be_true)
 continue;

 bool exact = !mask.can_be_false;
 if (exact || range.end == range.begin + 1)
 {
 accepted.push_back(range);
 if (collect_exact_ranges && exact)
 accepted_exact.push_back(range);
 continue;
 }

 if (result.num_steps + queue.size() > search_settings.max_steps)
 {
 result.reached_step_limit = true;
 accepted.push_back(range);
 continue;
 }

 splitRange(range, search_settings.coarse_index_granularity, [&](MarkRange subrange) { queue.push(subrange); });
 }

Two things change relative to the unlimited version.

A priority queue, largest range first. The budget is spent where it buys the most: splitting a huge range is what removes the most granules, so the biggest range is examined first. The comparator is explicit:

 cpp
 
 
// src/Storages/MergeTree/GenericExclusionSearch.cpp:102
/// Orders the queue of the step-limited search: the largest range comes first.
struct LessBySize
{
 bool operator()(const MarkRange & lhs, const MarkRange & rhs) const
 {
 size_t lhs_size = lhs.end - lhs.begin;
 size_t rhs_size = rhs.end - rhs.begin;
 return std::tie(lhs_size, rhs.begin) < std::tie(rhs_size, lhs.begin);
 }
};

When the budget runs out, the range is accepted whole. This is the important bit: the search never returns an incorrect answer. When result.num_steps + queue.size() > max_steps, the range is pushed into accepted as-is — meaning “we don’t know, so read it”. The result is a superset of the true matching granules, so the query is correct; it just reads more data than an unlimited search would have.

That’s what the setting says:

 cpp
 
 
// src/Core/Settings.cpp:1936
 DECLARE(UInt64, merge_tree_generic_exclusion_search_max_steps, 0, R"(
When a filter cannot be evaluated as a single continuous range of the primary key, for example when it uses key columns other than the first one, ClickHouse runs an iterative generic exclusion search algorithm over the index marks. The same algorithm is used for the analysis of the text index. This setting limits the number of steps (index checks) the algorithm spends on each data part.

The budget is spent on the largest remaining mark ranges first. When it is exhausted, the ranges that were not fully analyzed are accepted as a whole, so the query stays correct but may read more granules than an unlimited search would select. A lower budget speeds up index analysis at the cost of reading more data. The limit is approximate rather than a strict cap on the analysis cost: the search can exceed it by roughly one round of splitting, and when the part is already divided into many ranges (for example, by the query condition cache), each of them is checked at least once regardless of the limit.

The default of 0 means unlimited steps. So the default behaviour is exact and unbounded; the budget is an escape hatch for pathological cases.

The seek heuristic

One more knob, and it’s about IO, not about correctness. When two accepted ranges are separated by a small gap of marks, it’s cheaper to read the gap sequentially than to seek past it. appendWithMaxGap merges the two ranges when the gap is at most max_gap marks wide.

The gap threshold comes from merge_tree_min_rows_for_seek and merge_tree_min_bytes_for_seek, converted into marks by the granularity of the part. Both default to 0, i.e. “never bridge a gap” — but on a spinning disk or an object store with a high per-request cost, setting them turns “read 8 KB from here, seek, read 8 KB from there” into “read 40 KB contiguously”.

Note the asymmetry that the comment points out: exact_ranges are merged with max_gap = 0, always. An exact range is a range where every row matches, and bridging a gap into it would claim that the skipped marks match too — which is false. So the gap-bridging optimization applies only to the “may match” ranges, never to the “definitely matches” ones.

Putting it together

 text
 
 
 part on disk
 ┌───────────────────────────────────────────────────────────────┐
 │ granule 0 granule 1 granule 2 granule 3 │
 │ rows 0..8191 rows 8192..16383 rows ... rows ... │
 └───────────────────────────────────────────────────────────────┘
 │ │ │ │
 ▼ ▼ ▼ ▼
 marks: (x0,y0) (x1,y1) (x2,y2) (x3,y3)
 bit-packed, ~3 bytes each, one per column file
 │
 ▼
 primary.idx: key values of the FIRST ROW of each granule
 │
 ▼
 KeyCondition.checkInRange(...)
 │
 ├── predicate is a continuous key range → binary search (2·log₂ n checks)
 │
 └── predicate is anything else → exclusion search (recursive, budgeted)
 │
 ▼
 MarkRanges: the granules that must be read

What this buys you, and what it costs

It buys you a scan that’s proportional to the data you asked for, not to the data you stored. A SELECT with a range predicate on a sorted key reads roughly the granules that overlap the range, plus one granule of granularity on each side.

It buys you an index that fits in memory. Three bytes per mark, and the primary key index trimmed to the columns that can actually narrow things down. Without those two tricks, a 10 TB table would need tens of gigabytes of index.

It costs you a fixed granularity of waste. A predicate that matches a single row still reads a whole granule — 10 MB by default. This is the central trade of the whole design, and it’s why index_granularity_bytes is a setting you may want to lower for a table with a very selective predicate and small rows.

It costs you an in-memory structure that has to be built and kept. The primary index is loaded per part (or cached), and the marks are loaded per column file that the query touches. Both are cached, and both have their own LRU, but the first query against a cold part pays for them.

It costs you complexity in the analysis. Two algorithms, a budget, a trimming heuristic that makes the analysis conservative in some cases, and the requirement that the analysis be sound (never skip a granule that could match) even when it isn’t complete.

Seeing it in action

The profile events tell you which algorithm ran:

 sql
 
 
SELECT event, value
FROM system.events
WHERE event IN (
 'IndexBinarySearchAlgorithm',
 'IndexGenericExclusionSearchAlgorithm',
 'IndexGenericExclusionSearchStepLimitReached',
 'FilteringMarksWithPrimaryKeyProcessedMarks',
 'FilteringMarksWithPrimaryKey'
);

FilteringMarksWithPrimaryKeyProcessedMarks is the number of marks examined. FilteringMarksWithPrimaryKey is the number of parts where the primary key was used at all. The ratio between the two, and the ratio of marks read to marks examined, is your “how selective is my index” metric.

And the parts themselves report the granularity they were written with:

 sql
 
 
SELECT
 name,
 rows,
 marks,
 round(rows / marks) AS rows_per_mark,
 formatReadableSize(bytes_on_disk) AS size
FROM system.parts
WHERE active AND rows > 0
ORDER BY marks DESC
LIMIT 10;

A rows_per_mark far below 8192 means the table has fat rows and the byte-based granularity kicked in. A rows_per_mark of exactly 8192 means use_const_adaptive_granularity is on, or index_granularity_bytes was 0 when the part was written.

Wrapping up

The one thing to remember: a MergeTree granule is the unit of skipping, and everything else — marks, the primary index, the two search algorithms — exists to answer one question about it: can this granule possibly match?

What to go do about it: if a query on your table isn’t skipping granules, look at the shape of the predicate before you touch anything else. A predicate on the first sorting-key column with a continuous range gets the fast binary search. A predicate on a later column, or an OR across values, gets the recursive exclusion search — and there you should be watching IndexGenericExclusionSearchStepLimitReached.

If you see that event firing, the search ran out of budget and fell back to reading more granules than it needed to. Either raise merge_tree_generic_exclusion_search_max_steps, or — better — add a projected column and query that instead.

ClickHouse Runs a Query Pipeline on N Threads: Task Queues, Work Stealing and a Pool That Grows on Demand

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse doesn’t hand you max_threads threads. It starts with one, measures how much parallel work the pipeline actually creates, and grows the pool only when idle threads genuinely can’t absorb it — then shrinks again if the server preempts you.

A query plan in ClickHouse is not one pipeline. It’s a stack of pipelines: read from storage, filter, project, aggregate, sort, format. Each stage produces many parallel streams, and consecutive stages are connected by ResizeProcessors that shuffle chunks between them.

Running that means answering three questions at once:

Which processor should run next? (the graph — see the previous article on ExecutingGraph)
Which thread should run it?
How many threads should exist at all?

The third question is the interesting one. A query that reads one small table does not want sixteen threads. A query that scans a hundred partitions does. And a query that has been running for thirty seconds may need fewer threads than it started with, because the server is under load from other queries.

ClickHouse’s answer, in src/Processors/Executors/Runtime/V1/, is a system that starts with one thread, measures demand as the pipeline runs, and grows the pool only when there is genuinely parallel work that idle threads cannot absorb.

Two kinds of parallelism

Before the machinery, let’s be precise about what “parallel” means here, because ClickHouse does two different things and they get conflated a lot.

Data parallelism is the obvious one: SELECT count() FROM t reads from N partitions, each partition is its own stream, and N streams are processed at once. This is a property of the plan — the number of streams is fixed when the plan is built.

Pipeline parallelism is the subtler one. Even in a pipeline with a single stream, the stage reading from disk isn’t blocked while the stage formatting the output is busy. A chunk is handed from one processor to the next, and while the consumer works on chunk k, the producer is already building chunk k+1.

The executor supports both, and the interesting bit is how it decides how many threads to use:

 sql
 
 
-- Sixteen streams of data parallelism; the number of threads that actually
-- run them is decided at execution time, not by the plan.
SELECT count() FROM numbers_mt;

 sql
 
 
-- The ceiling. The pool starts at one thread and grows only if there is
-- parallel work that idle threads cannot absorb.
SETTINGS max_threads = 16;
SELECT count() FROM numbers_mt;

Note that the two are not the same thing. A pipeline with 16 streams needs 16 threads to be fully parallel, but it will still make progress — just less of it — on one thread.

The shape of the executor

PipelineExecutor owns three things:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.h:49
private:
 /// Outlives graph registrations, including when construction fails.
 QueryStatusPtr process_list_element;
 ExecutingGraphPtr graph;

 ExecutorTasks tasks;

 /// Concurrency control related
 SlotAllocationPtr cpu_slots;
 AcquiredSlotPtr single_thread_cpu_slot; // cpu slot for single-thread mode to work using executeUntil()
 std::unique_ptr<ThreadPool> pool;
 std::mutex spawn_mutex;

graph is the processor graph (the subject of the previous article). tasks is the scheduler: the queues, the per-thread contexts, the counters. cpu_slots and pool are the resources: how many threads are allowed to exist, and the threads themselves.

The entry point is execute(), which is unremarkable apart from its comment about output formats:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:231
void PipelineExecutor::execute(size_t num_threads, bool concurrency_control)
{
 /// `QueryStatus` may release its reservation before this executor is destroyed.
 SCOPE_EXIT({ detachSpillables(); });
 if (process_list_element && !process_list_element->checkTimeLimit())
 cancel(IProcessor::CancelReason::CancelledByTimeout);

 num_threads = std::max<size_t>(num_threads, 1);

and executeImpl() is where the threads actually appear:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:783
void PipelineExecutor::executeImpl(size_t num_threads, bool concurrency_control)
{
 initializeExecution(num_threads, concurrency_control);

 try
 {
 if (num_threads > 1)
 {
 {
 std::lock_guard lock(spawn_mutex);
 // Start at least one thread, could block to acquire the first CPU slot
 spawnThreads(cpu_slots->acquire());
 }
 tasks.processAsyncTasks();
 pool->wait();
 }
 else
 {
 auto slot = cpu_slots->acquire();
 tasks.upscale(slot->slot_id);
 executeSingleThread(slot->slot_id, WorkloadResources(slot.get(), process_list_element));
 }
 }

Notice pool->wait(). The master thread does not execute pipeline work in the multi-threaded case. It spawns worker threads and then parks in processAsyncTasks(), which is the epoll loop for processors that returned Status::Async.

The master thread is a scheduler, not a worker — and that distinction matters for deadlock avoidance, which is why the master is never downscaled:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:591
 // The first thread is called master thread.
 // It is NOT the thread that handles async tasks (unless query has max_threads=1).
 // Master thread is different from other threads due to special role in scheduling:
 // 1. During query start, master thread spawns worker threads, so starting it fast is important.
 // 2. It should never be downscaled to avoid query deadlock (0 threads to process tasks), although it can be preempted.
 // 3. When using ConcurrencyControl, master thread is granted immediately and may not request CPU slot (see 'fair_round_robin').
 // 4. When using resource scheduler, master thread may use different resource link than worker threads.
 const auto master_threads = 1uz;

Task queues with work stealing

The scheduler’s central data structure is TaskQueue, and it’s per-thread-first:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/TasksQueue.h:17
/// A set of task queues for multithreaded processing.
/// Every threads has its dedicated queue and uses it preferably.
/// When there are no tasks left in dedicated queue it steals tasks from other threads
template <typename Task>
class TaskQueue
{
public:
 void init(size_t num_threads) { queues.resize(num_threads); }

 /// Push a task into thread's dedicated queue
 void push(Task * task, size_t thread_num)
 {
 queues[thread_num].push(task);
 use_queues = std::max(use_queues, thread_num + 1);
 ++num_tasks;
 }

 /// Returns thread number to pop task from.
 /// First it check dedicated queue, and only if it is empty, it steal from other threads
 size_t getAnyThreadWithTasks(size_t from_thread)
 {
 if (num_tasks == 0)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");

 if (from_thread >= use_queues)
 from_thread = 0;

 for (size_t i = 0, max_iterations = use_queues; i < max_iterations; ++i)
 {
 if (!queues[from_thread].empty())
 return from_thread;

 ++from_thread;
 if (from_thread == use_queues)
 {
 use_queues--; // lazy update: less expensive than updating on every pop
 from_thread = 0;
 }
 }

 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");
 }

Each thread has a private FIFO. A thread with work prefers its own queue; only when it’s empty does it walk the other threads’ queues looking for something to steal.

use_queues is a lazily-maintained upper bound on how many queues are non-empty, so the scan is O(number of active threads) and not O(total threads). That “lazy update” comment is doing real work — updating the bound on every pop would cost more than it saves.

The per-thread affinity matters for cache reasons: a task that runs on thread 3 tends to produce data that thread 3 will consume next, and the data stays in that core’s L1/L2.

There are three queues, not one:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.h:43
 /// Queue with pointers to tasks. Each thread will concurrently read from it until finished flag is set.
 /// Stores processors need to be prepared. Preparing status is already set for them.
 TaskQueue<IProcessor> task_queue;

 /// Async tasks should be processed with higher priority, but also require task stealing logic.
 /// So we have a separate queue specifically for them.
 TaskQueue<IProcessor> fast_task_queue;
 std::atomic_bool has_fast_tasks = false; // Required only to enable local task optimization
 std::atomic_bool spill_requested = false;

 /// Queue which stores tasks where processors returned Async status after prepare.
 /// If multiple threads are used, main thread will wait for async tasks.
 /// For single thread, will wait for async tasks only when task_queue is empty.
 PollingQueue async_task_queue;

fast_task_queue holds processors that came back from an asynchronous operation — a socket that became readable, a file that finished being written. They get priority because they’re latency-sensitive: the data is already there and the client is waiting.

async_task_queue is different in kind. It’s not a task queue at all, it’s an epoll wrapper that owns the file descriptors.

The local-task optimization

Here’s the single most important performance trick in the executor, and it’s small.

When a processor finishes, the next processor to run is often known immediately — it’s the downstream consumer of the chunk that was just produced. Rather than pushing it onto a shared queue (where another thread would have to wake up, take the mutex, and pop it), the executor hands it directly to the thread that’s already running.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:192
size_t ExecutorTasks::pushTasks(Queue & queue, Queue & async_queue, ExecutionThreadContext & context)
{
 /// Take local task from queue if has one.
 if (!queue.empty() && !has_fast_tasks.load(std::memory_order_relaxed)
 && context.num_scheduled_local_tasks < ExecutionThreadContext::max_scheduled_local_tasks)
 {
 ++context.num_scheduled_local_tasks;
 context.setTask(queue.front());
 queue.pop();
 }
 else
 {
 context.num_scheduled_local_tasks = 0;
 context.setTask(nullptr);
 }

and the constant that bounds it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutionThreadContext.h:55
 /// There is a performance optimization that schedules a task to the current thread, avoiding global task queue.
 /// Optimization decreases contention on global task queue but may cause starvation.
 /// See 01104_distributed_numbers_test.sql
 /// This constant tells us that we should skip the optimization
 /// if it was applied more than `max_scheduled_local_tasks` in a row.
 constexpr static size_t max_scheduled_local_tasks = 128;

128 in a row. The comment names the failure mode: starvation. If one thread keeps grabbing work locally while other threads sit idle with nothing to do, the pipeline runs on one core while the other max_threads - 1 cores idle.

The counter resets as soon as the local optimization is skipped, so a thread that has been running alone for 128 tasks will put the next one on the global queue and let somebody else take it.

The referenced test, 01104_distributed_numbers_test.sql, is a good illustration of why this matters: SELECT * FROM numbers(1000000000) with distributed_product_mode and a LIMIT — the limit transform stops the source, and if the source isn’t rebalanced the other threads never wake up.

The has_fast_tasks check in the condition is the second half of the guard: if there’s an asynchronous task waiting, it takes priority over a local task, because asynchronous tasks are latency-critical.

Measuring demand: `pushTasks` returns a number

Now the interesting part. pushTasks doesn’t just enqueue work — it measures how much parallel work was created, and returns it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:208
 if (!queue.empty() || !async_queue.empty())
 {
 std::unique_lock lock(mutex);

 /// Measure the parallelism of the push: how many threads could run the new tasks
 /// concurrently. Used by `PipelineExecutor` to size the slot-allocation ceiling via
 /// `ISlotAllocation::setMax` on the pipeline's `cpu_slots` handle.
 const size_t pushed = queue.size() + async_queue.size();

and then, at the end:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:232
 /// Capture state under the lock — tryWakeUp may unlock it internally to wake a
 /// thread, after which `total_slots` could change.
 const size_t idle_before = threads_queue.size();
 const size_t total_slots_now = total_slots;
 const size_t num_threads_now = num_threads; // const after init but read for clarity

 /// Wake up at least one thread that will wake up other threads if required
 tryWakeUpAnyOtherThreadWithTasks(context, lock);

 /// Return the number of additional pipeline threads the caller should target: pushed
 /// tasks that idle threads can't cover, clamped to remaining pipeline-width headroom
 /// (`num_threads - total_slots`). `PipelineExecutor` adds this delta to the
 /// `cpu_slots->setMax(...)` target so the slot allocator grows in step with demand.
 const size_t coverable = std::min(pushed, idle_before);
 const size_t uncovered = pushed - coverable;
 const size_t headroom = (num_threads_now > total_slots_now) ? (num_threads_now - total_slots_now) : 0;
 return std::min(uncovered, headroom);
 }
 return 0; // No new tasks -- no need for new threads

The arithmetic is: pushed tasks were created. Idle threads can absorb that many without being woken up. The remainder is genuine demand for new threads — clamped to the pipeline’s own width ceiling (num_threads, from max_threads), because the plan has a fixed number of streams and there’s no point running more threads than there is parallelism.

This is the demand signal. It’s computed from two observable facts — how many tasks were pushed, how many threads are parked — and nothing else. No profiling, no heuristics, no timers.

The executor then accumulates that demand and acts on it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:492
 /// Upscaling: `pushTasks` tells us how many additional threads would usefully
 /// parallelize the tasks it just enqueued. Publish our spawn_count into the shared
 /// `pending_demand` counter; the thread that actually holds `spawn_mutex` drains it
 /// so no demand is lost when multiple threads push concurrently.
 if (pool && spawn_count > 0)
 pending_demand.fetch_add(spawn_count, std::memory_order_relaxed);

 // Drain pending_demand under `spawn_mutex`. A thread that fails `try_lock`
 // must not exit while `pending_demand > 0` -- otherwise its add could race
 // with the owner's final `exchange(0)` and be left stranded after the owner
 // unlocks. The outer loop holds the thread until either it drains the demand
 // itself or observes the counter return to zero (meaning the owner caught it).
 while (pool && pending_demand.load(std::memory_order_acquire) > 0)
 {
 if (!spawn_mutex.try_lock())
 {
 std::this_thread::yield();
 continue;
 }
 try
 {
 std::lock_guard lock(spawn_mutex, std::adopt_lock);
 while (true)
 {
 size_t accumulated = pending_demand.exchange(0, std::memory_order_acq_rel);
 if (accumulated == 0)
 break;
 size_t target = std::min<size_t>(max_pipeline_threads, desired_threads + accumulated);
 if (target > desired_threads)
 {
 desired_threads = target;
 cpu_slots->setMax(target);
 }
 }
 spawnThreads({});
 }

This is small and rather careful concurrent code, and the comments are essentially a post-mortem.

The pending_demand counter is atomic so that threads which fail to take spawn_mutex don’t lose their demand. But a naive fetch_add would race with the owner’s exchange(0): thread A adds 1, the owner exchanges and gets 0, the owner unlocks, and thread A’s 1 sits in the counter forever.

The outer while loop is the fix. A thread that can’t get the mutex keeps spinning (yielding) until either it drains the demand itself or sees the counter back at zero, which means the owner already caught it.

The target computation is the second piece of policy: demand is accumulated into desired_threads, capped at max_pipeline_threads (the plan’s width), and applied by raising the slot allocator’s ceiling with cpu_slots->setMax(target).

spawnThreads then converts slots into actual OS threads:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:743
void PipelineExecutor::spawnThreads(AcquiredSlotPtr slot)
{
 while (cpu_slots)
 {
 if (!slot)
 slot = cpu_slots->tryAcquire();
 if (!slot)
 return;

 size_t thread_num = slot->slot_id;

 /// Count of threads in use should be updated for proper finish() condition.
 /// `upscale` returns the remaining headroom for spawning. Even when >0 we spawn
 /// one thread per iteration and re-enter `upscale` after the next `tryAcquire`.
 const size_t remaining_capacity = tasks.upscale(thread_num);

 /// Start new thread
 pool->scheduleOrThrowOnError([this, thread_num, thread_group = CurrentThread::getGroup(), my_slot = std::move(slot)]
 {
 ThreadGroupSwitcher switcher(thread_group, ThreadName::QUERY_ASYNC_EXECUTOR);

 try
 {
 executeSingleThread(thread_num, WorkloadResources(my_slot.get(), process_list_element));
 }
 catch (...)
 {
 /// In case of exception from executor itself, stop other threads.
 finish();
 tasks.getThreadContext(thread_num).setException(std::current_exception());
 }
 });

The tryAcquire is the gate. The slot allocator may refuse to hand out a slot if the server is out of CPU budget. If it refuses, no thread is spawned and the pipeline simply runs with fewer threads.

The thread count is not max_threads; it’s max_threads subject to what the scheduler is willing to give you.

CPU slots: `max_threads` is a ceiling, not a promise

The slot allocation is where the concurrency-control policy lives. There are four cases, and the comment enumerates them:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:605
 // There are a few possible modes of concurrency control.
 // 1) If we have `CREATE RESOURCE cpu (MASTER THREAD, WORKER THREAD)`:
 // all thread should be allocated through scheduler using the same resource link
 // 2) If we have `CREATE RESOURCE cpu (WORKER THREAD)`:
 // master thread uses "free" noncompeting CPU slot, the other threads are allocated through resource scheduler
 // 3) If we have two resources `CREATE RESOURCE master_cpu (MASTER THREAD)` and `CREATE RESOURCE worker_cpu (WORKER THREAD)`:
 // all thread should be allocated through scheduler using different resource links
 // 4) If we have no cpu-related resources:
 // the ConcurrencyControl class is used instead of resource scheduler

In the default case (case 4) the allocation is deliberately lazy:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:672
 else
 {
 /// Allocate CPU slots from concurrency control with guaranteed master thread slot.
 /// Default path (lazy_allocation=true): start at max=master_threads and grow on
 /// demand via setMax from the upscaling block. Zero-waste per idle pipeline.
 /// Rollback (lazy_allocation=false): pre-branch eager allocation — request the
 /// full ceiling upfront and never call setMax.
 SlotCount initial_max = lazy_allocation ? master_threads : num_threads;
 return ConcurrencyControl::instance().allocate(master_threads, initial_max);
 }

“Zero-waste per idle pipeline” is the point. A server running a thousand concurrent queries, each of which is a small scan, doesn’t want a thousand times max_threads slots reserved. It wants one slot each, growing if the query turns out to be wide.

The lazy flag is read exactly once, and the reason is a real bug:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:696
 /// Read the flag ONCE. A config reload mid-initialization would otherwise split a
 /// single query between lazy and eager strategies (e.g. allocate lazy with max=1 but
 /// then initialize desired_threads=num_threads, stranding subsequent setMax as a no-op).
 const bool lazy_allocation = ConcurrencyControl::instance().getLazyAllocation();

A config reload between “allocate the slots” and “set desired_threads” would leave the query in a state where setMax is a no-op and the pipeline never grows. One read, one strategy, for the lifetime of the query.

Preemption and downscaling

Threads can also be taken away. Under the resource scheduler with preemption enabled, a thread’s CPU lease is renewed periodically, and the renewal can fail:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:539
 if (resources.isCPULeaseRenewNeeded()) // Check if preemption is enabled (see `cpu_slot_preemption` server setting)
 {
 try
 {
 // Preemption point. Renewal could block execution due to CPU overload.
 // It may trigger callbacks to tasks.preempt() and tasks.resume()
 if (!resources.renewCPULease())
 {
 tasks.downscale(resources.getCPULeaseSlotId());
 break; // Downscaling. Unable to renew the lease - thread should stop (but could be rerun later).
 }

The renewal is checked at most every millisecond:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:119
 bool isCPULeaseRenewNeeded()
 {
 if (!lease)
 return false;
 UInt64 now_ns = clock_gettime_ns();
 if (now_ns - last_renew_ns < 1000000) // 1 ms: do not renew too often
 return false;
 last_renew_ns = now_ns;
 return true;
 }

Downscaling is not “kill the thread”. It’s “let this thread stop when it finishes its current task, and don’t schedule more onto it”:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:334
void ExecutorTasks::downscale(size_t slot_id)
{
 std::lock_guard lock(mutex);

 if (slot_id >= slot_count.size() || slot_count[slot_id] == 0)
 return;
 --slot_count[slot_id];

 if (slot_id + 1 == use_threads)
 {
 // The highest slot was downscaled, find new highest
 for (size_t i = use_threads; i > 0; --i)
 {
 if (slot_count[i - 1] > 0)
 {
 use_threads = i;
 break;
 }
 }
 }
}

Preemption is a third thing: the slot is released without downscaling, so the pipeline’s width is unchanged but the thread isn’t running. If the preempted thread was holding a task, that task is pushed back to the queue so another thread can steal it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:356
void ExecutorTasks::preempt(size_t slot_id)
{
 std::unique_lock lock(mutex);
 --total_slots;

 /// We should make sure that preempted thread has no local task inside context.
 /// It is allowed to have tasks in `task_queue` or `fast_task_queue` because they can be stealed by other threads.
 auto & context = executor_contexts[slot_id];
 if (auto * task = context->popTask())
 {
 task_queue.push(task, slot_id);
 /// Wake up at least one thread to avoid deadlocks (all other threads maybe idle)
 tryWakeUpAnyOtherThreadWithTasks(*context, lock); // this releases the lock if it wakes up a thread
 }

The asymmetry is the point. A local task (held in the thread’s context) is invisible to other threads and must be handed back explicitly, whereas a task in the shared queue is stealable and needs no action.

The pulling mode: `executeUntil`

Not every query is executed to completion in one go. The HTTP protocol, for instance, wants to stream rows to the client as they become available, and the client may disconnect at any moment. That’s what executeUntil() is for:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:261
bool PipelineExecutor::executeUntil(std::atomic_bool * yield_flag)
{
 const auto uncaught_exceptions = std::uncaught_exceptions();
 SCOPE_EXIT({
 if (tasks.isFinished() || std::uncaught_exceptions() > uncaught_exceptions)
 detachSpillables();
 });
 if (!is_execution_initialized)
 {
 initializeExecution(1, true);

 // Acquire slot until we are done
 single_thread_cpu_slot = cpu_slots->acquire();
 tasks.upscale(single_thread_cpu_slot->slot_id);
 chassert(single_thread_cpu_slot && "Unable to allocate cpu slot for the first thread, but we just allocated at least one slot");

 if (yield_flag && *yield_flag)
 return true;
 }

 executeStepImpl(0, WorkloadResources(single_thread_cpu_slot.get(), process_list_element), yield_flag);

 if (!tasks.isFinished())
 return true;

The yield_flag is the interesting parameter. It’s checked at the bottom of the worker loop:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:576
 /// We have executed single processor. Check if we need to yield execution.
 if (yield_flag && *yield_flag)
 break;

So the executor runs one processor at a time and then checks the flag. If the flag says “enough for now”, it returns to the caller, which can hand the chunk to the client and then call executeUntil() again. This is the “pulling” model at the level of the whole pipeline: the caller pulls.

PullingPipelineExecutor::pull() is the caller:

 cpp
 
 
// src/Processors/Executors/PullingPipelineExecutor.cpp:49
bool PullingPipelineExecutor::pull(Chunk & chunk)
{
 if (!executor)
 {
 executor = createExecutor(pipeline.processors, pipeline.process_list_element);
 executor->setReadProgressCallback(pipeline.getReadProgressCallback());
 executor->setStepProfiler(pipeline.getStepProfiler());
 }

 /// Throws when the time limit is exceeded with `timeout_overflow_mode = 'throw'`. With 'break' the partial result
 /// is returned as a success: the execution is cancelled with `CancelledByTimeout`, and `executeUntil` below
 /// finalizes it - the pending read progress is reported and the format is finalized - before the end of the
 /// data is reported.
 if (pipeline.process_list_element && !pipeline.process_list_element->checkTimeLimitSoft())
 {
 executor->cancel(IProcessor::CancelReason::CancelledByTimeout);
 pipeline.process_list_element->checkTimeLimit();
 }

 if (!executor->executeUntil(&has_data_flag))
 return false;

 chunk = pulling_format->getChunk();
 return true;
}

has_data_flag is set by the output format when it has a chunk ready. So the loop is: run the pipeline until the format has produced something (or the pipeline is finished), then hand the chunk up.

Note that executeUntil runs the executor in single-threaded mode (initializeExecution(1, true)). The pulling executor is deliberately not parallel — it exists for cases like INSERT ... SELECT into a table, or formatting a result to a client that wants to consume it incrementally.

Trade-offs

What this buys you.

The pool grows with the work, not with the setting. A query that turns out to be narrow never pays for more than one or two threads, even if max_threads = 64. A query that turns out to be wide gets threads as fast as the scheduler will give them, without any startup latency. And the measurement is exact rather than heuristic: pushTasks counts the tasks it created and subtracts the threads that could absorb them.

Work stealing with per-thread affinity gives good cache behaviour and good load balance. The common case (a thread feeding itself) is a pointer swap with no locking; the uncommon case (one thread has all the work) is handled by theft.

The separation of “what work exists” from “how many resources it gets” also means the same executor serves queries with max_threads = 1 (no contention at all, the single thread takes everything locally) and queries with max_threads = 128.

What it costs.

The machinery is intricate, and the comments are full of post-mortems: the try_lock/pending_demand race, the lazy_allocation config reload, the max_scheduled_local_tasks starvation. Each of those is a bug that shipped and was fixed. The design is sound, but the implementation is carrying the weight of a decade of production.

There’s also a hard limit on the demand signal. pushTasks measures the tasks pushed by one processor at one moment. If a pipeline is wide but each individual push produces one task, the demand signal is one at a time, and the pool grows one thread per push rather than immediately. The initial_parallel computation in initializeExecution exists precisely to patch this for the first wave:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:722
 /// Initial queued parallelism never routes through `pushTasks`, so size setMax here to
 /// cover it. For multi-source pipelines (e.g. UNION ALL of N subqueries) this prevents
 /// the pipeline from being stuck single-threaded if the master's first task happens not
 /// to push more work. Only matters on the lazy path; the eager path already has the full
 /// ceiling. `initial_parallel` is the total target — the master thread is one of its
 /// consumers, no +1 needed.
 if (lazy_allocation && initial_parallel > 1)

The preemption path is the other sharp edge. A thread that can’t renew its lease stops, and the pipeline continues with fewer threads — but if every thread is preempted at once, the pipeline must not deadlock. The threads_queue.size() + 1 == total_slots check in tryGetTask is the mechanism that detects “the last worker is about to go idle” and calls finish():

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:158
 /// This thread has no tasks to do and is going to wait.
 /// Finish execution if this was the last active thread.
 chassert(task_queue.empty() && fast_task_queue.empty());
 if (threads_queue.size() + 1 == total_slots && async_task_queue.empty())
 {
 lock.unlock();
 finish();
 return;
 }

Getting this wrong in either direction is bad: finish too early and the query returns partial data, never finish and the query hangs. The chassert documents that the two conditions must hold together — if the queues are not empty, the pipeline is not finished and the thread must keep waiting.

Finally, there’s the question of when to stop growing. desired_threads only ever goes up within a query. A pipeline that’s wide at the start and narrow at the end keeps its threads; there’s no downscaling based on demand, only downscaling due to preemption. For a long-running query over a table that becomes narrow, that’s wasted capacity.

Wrapping up

The real win: ClickHouse doesn’t guess how many threads your query needs. It measures. pushTasks returns a number — how many new tasks idle threads can’t cover — and the pool grows by exactly that much, capped by the plan’s own width.

Two things to internalize:

max_threads is a ceiling, not a schedule. If you see a query using 2 threads on a 64-core box, that’s the demand signal working, not a bug.
The server can take threads back from you. Preemption and downscaling are first-class, and your query is expected to keep going on fewer threads.

So when you profile a slow query, don’t count threads first. Count streams, and ask whether the pipeline had enough independent work to push into idle threads in the first place.

ClickHouse Plans Distributed Queries with Cascades: A Real Cost-Based Optimizer

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse’s new Cascades optimizer treats a distributed query as a search problem. It generates alternative plans in a memo, prices each one with a work/network/sequential cost model, and ships the cheapest — but it refuses to guess, and fails loudly when it can’t decide.

A single-node query plan has one obvious shape: read the table, filter, aggregate, sort. A distributed query does not.

The same SELECT can be executed as a gather-everything-to-the-coordinator, or as a shuffle of both sides of a join followed by a local aggregation, or as a broadcast of a tiny dimension table to every node followed by a local join. And these choices interact — the best join strategy depends on how the data will be aggregated afterwards, and the best aggregation strategy depends on how the join output is distributed.

So the planner has to explore combinations and compare them.

ClickHouse’s traditional answer was a set of fixed heuristics. Its new answer, arriving in 26.8 as an experimental setting, is a real cost-based optimizer built on the Cascades framework: a memo of alternative plans, a set of rules that generate alternatives, and a cost model that prices them.

 cpp
 
 
// src/Core/Settings.cpp:11131
 DECLARE(Bool, enable_cascades_optimizer, false, R"(
Enable the Cascades cost-based optimizer for distributed query plans.
Takes effect only together with `make_distributed_plan = 1`: the setting alone does not change single-node query planning.
)", EXPERIMENTAL, \
 {"26.8", false, false, "New experimental setting."}) \

This article walks the implementation: the memo, the task-based search, the rules, the cost model, the enforcers, and the places where the implementation is deliberately conservative.

The problem the optimizer solves

The ARCHITECTURE.md that ships with the code states it plainly:

 text
 
 
## What Problem Does This Solve?

ClickHouse can execute a query on a single node using its standard pipeline: plan the
query, read from MergeTree, apply filters/expressions/joins/aggregations, return results.
But when data is spread across multiple nodes (shared storage like S3, or a multi-node
cluster), the optimizer must decide how to distribute the work:

- Should a join shuffle both sides by the join key, or broadcast the smaller table to
 all nodes?
- Should an aggregation happen on each node first (partial agg) and then merge results,
 or gather all data to one node and aggregate there?
- Should a table scan be split across N nodes (parallel read), or should each node read
 the full table (replicated read for small dimension tables)?
- Where should exchanges (network data transfers between nodes) be placed?

Pay attention to the third bullet. With shared storage (S3), a 25-row dimension table doesn’t need to be broadcast at all — every node can just read it from S3 itself.

That alternative, ReplicatedRead, is exactly the kind of thing only a cost-based search would find, because it isn’t a “join strategy” in the classical sense. It’s a read strategy, chosen because the cost of re-reading a tiny table is lower than the cost of shipping it.

The memo

The memo is the central data structure. It contains groups, each representing a set of logically equivalent subexpressions. Each group holds both the logical expressions (ways of computing the same thing) and the physical expressions (ways of computing it with a concrete algorithm and concrete properties).

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Memo.h:10
class Memo
{
public:
 explicit Memo(LoggerPtr log_)
 : log(log_)
 {}

 GroupId addGroup(GroupExpressionPtr group_expression);

 GroupPtr getGroup(GroupId group_id);
 GroupConstPtr getGroup(GroupId group_id) const;

 size_t getGroupCount() const { return groups_by_id.size(); }

The plan is ingested into the memo one group per step. A step with children becomes a group whose expression has one input link per child, and the input link carries the required properties of that child:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:179
 auto group_expression = std::make_shared<GroupExpression>(std::move(node.step));
 auto group_id = memo.addGroup(group_expression);
 for (auto * child_node : node.children)
 {
 auto [input_group_id, pending_props] = addGroup(*child_node);
 group_expression->inputs.push_back({input_group_id, pending_props});
 }

So each input link carries the child’s group id and the properties the parent requires of it. That’s where the “required properties” of the whole framework enter the picture.

Properties: distribution and sorting

A physical expression has two kinds of properties.

Distribution — how the data is spread over the nodes:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.h:17
struct DistributionDescription
{
 DistributionColumns columns; /// Columns by which data is distributed. E.g. for shuffle exchange or partitioned table scan.
 /// Type each key is cast to before hashing (parallel to `columns`; empty when the raw
 /// types agree). Aligns buckets across both sides of a shuffle join.
 Names hash_type_names;
 bool is_replicated = false; /// All data is replicated to all nodes, so any distribution is satisfied. E.g. for small tables that are broadcasted to all nodes.
 size_t node_count = 1; /// Number of nodes among which data is distributed. E.g. for a shuffle exchange or a partitioned read.

 bool operator==(const DistributionDescription & other) const = default;

 void dump(WriteBuffer & out) const;
 String dump() const;
};

Sorting — whether the output is ordered, and by what.

A parent requires properties of its child, and a child satisfies a requirement if it provides at least as much. The matching isn’t trivial: a sorted-by-(a) stream satisfies a requirement of “sorted by (a)”, and the required sort must be a prefix of the existing one:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.cpp:9
bool ExpressionProperties::isSortingSatisfiedBy(const SortDescription & required, const SortDescription & existing)
{
 /// Required sorting is satisfied by existing sorting if required is the prefix of existing
 if (required.size() > existing.size())
 return false;
 for (size_t i = 0; i < required.size(); ++i)
 {
 if (required[i] != existing[i])
 return false;
 }
 return true;
}

and the distribution match is positional — data shuffled by (b, a) does not colocate with data shuffled by (a, b):

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Properties.cpp:39
 /// The partition hash chains the key columns in order and may cast them first, so the
 /// match must be positional and the hash types must agree: data shuffled by `(b, a)`
 /// or hashed at a different type does not colocate with data shuffled by `(a, b)`.
 if (required.hash_type_names != existing.hash_type_names)
 return false;

Sorting is a property, not an operator

One design decision deserves a callout, because it differs from classical Cascades.

A plain ORDER BY (no LIMIT) is not an operator in the memo — it’s a property that the parent requires of its input. The optimizer then produces the sort itself, if some child can provide it more cheaply:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:162
 /// Strip a limit-less SortingStep::Full - pure sorting is a physical property, not a logical
 /// one. Return the child's GroupId along with the sorting as required properties, so the
 /// caller can attach them to the input link of the parent group expression.
 /// A SortingStep with a limit is a top-N (row-reducing) operator, so it is kept as an
 /// operator instead; the limit is owned by that operator, never by the sorting property.
 const auto * sorting_step = typeid_cast<const SortingStep *>(node.step.get());
 if (sorting_step && sorting_step->getType() == SortingStep::Type::Full && sorting_step->getLimit() == 0)
 {
 chassert(node.children.size() == 1);
 auto [child_group_id, _] = addGroup(*node.children.front());
 ExpressionProperties stripped_props;
 stripped_props.sorting = sorting_step->getSortDescription();
 return {child_group_id, stripped_props};
 }

And a LIMIT after the sort makes it a top-N, which reduces the row count and therefore is an operator, with its own rules and its own cost.

The search: tasks on a stack

The optimizer is a LIFO stack of tasks. There are six task types, and a group moves through four stages.

Stage 1 — Explore. Transformation rules generate logically equivalent alternatives (join commutativity, two-stage aggregation).

Stage 2 — Implement. Implementation rules turn logical expressions into physical ones with concrete properties (local join, broadcast join, shuffle join; local read, parallel read, replicated read).

Stage 3 — Enforce. Enforcer rules bridge property gaps by inserting exchange steps.

Stage 4 — Done.

OptimizeGroupTask drives the stages:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:38
 if (!group->isExplored())
 {
 /// Explore the group, then re-run `OptimizeGroupTask`
 optimizer.pushTask(std::make_unique<OptimizeGroupTask>(group_id, required_properties));
 optimizer.pushTask(std::make_unique<ExploreGroupTask>(group_id));
 }
 else if (!group->isOptimizedFor(required_properties))
 {
 optimizer.pushTask(std::make_unique<OptimizeGroupTask>(group_id, required_properties));

 for (auto & expression : group->logical_expressions)
 optimizer.pushTask(std::make_unique<OptimizeExpressionTask>(expression, required_properties));

 group->setOptimizedFor(required_properties);
 }

The interesting part is the pruning decision at the top of the task, which is explicitly not a cost-based prune:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:20
 /// Skip this group only if it is already fully processed (explored + implemented +
 /// enforced) for these properties and has a satisfying plan - re-running would be a no-op.
 /// We deliberately do not prune just because a current best is within a finite cost budget:
 /// the budget is an upper bound, not a lower bound, so such pruning is unsound for
 /// optimality, and it can return before stage-3 enforcers add the distributed alternatives.
 {
 const auto & cost_config = optimizer.getMemo().getContext().cost_config;
 bool group_fully_processed = group->isExplored()
 && group->isOptimizedFor(required_properties)
 && group->isEnforcedFor(required_properties);
 if (group_fully_processed && group->getBestImplementation(required_properties, cost_config).expression)
 {
 group->setFullyDoneFor(required_properties);
 LOG_TEST(optimizer.log, "OptimizeGroupTask group #{}: already fully processed", group_id);
 return;
 }
 }

This is the part people get wrong when they read this code.

Classical Cascades prunes with branch-and-bound: once a subtree’s cost exceeds the best complete plan, it gets dropped. ClickHouse’s implementation refuses to do that, because a subtree’s current best is an upper bound on its final cost — a cheaper enforcer-built alternative may still appear — and pruning on it would be unsound.

Instead, the search is bounded by a task count.

Rule ordering

Every rule has a static promise (a priority). The applicable rules are sorted by promise and pushed onto the LIFO stack, so the highest-promise rule runs first:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:169
 std::vector<std::pair<Promise, OptimizationRulePtr>> moves;
 for (const auto & rule : rules)
 {
 if (!expression->isApplied(*rule, required_properties) && rule->checkPattern(expression, required_properties, optimizer.getMemo()))
 moves.push_back({rule->getPromise(), rule});
 }

 /// Sort ascending so the LIFO task stack pops the highest-promise rule first.
 std::sort(moves.begin(), moves.end(), [](const auto & lhs, const auto & rhs) { return lhs.first < rhs.first; });

The promise doesn’t change which plans are considered — every applicable rule is eventually applied. It only changes the order, which breaks ties among equal-cost alternatives (the first-found best wins).

The cost model

The cost has three dimensions, combined by configurable weights:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.h:39
struct CostConfig
{
 Float64 work_weight = 1.0; /// Parallelizable work (scans, expression eval, I/O).
 Float64 network_weight = 1.0; /// Per-byte network transfer.
 /// Single-threaded phases (gather/scatter funnels, merge cursors). The weight is the
 /// per-node thread count: work parallelizes across the threads of a node while a serial
 /// phase holds one thread, so one serial row costs about `threads` work rows (Brent's law:
 /// wall-clock ~ work / threads + serial path).
 Float64 sequential_weight = 32.0;
 /// Fixed per-exchange latency (connection setup, metadata), in sequential rows. The
 /// weighted value (~1e5 work units) keeps a plan over a small input local: distribution
 /// cannot pay for its setup there.
 Float64 exchange_fixed_overhead = 3000.0;

and the combination:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.h:74
 Float64 total(const CostConfig & config) const
 {
 /// If any component is infinite the plan is impossible; return infinity directly. Multiplying
 /// first would let a zero weight turn `inf * 0` into NaN, and a NaN cost compares as neither
 /// better nor worse, so an impossible plan could be picked as best.
 if (!std::isfinite(work) || !std::isfinite(network) || !std::isfinite(sequential))
 return std::numeric_limits<Float64>::infinity();

 return work * config.work_weight
 + network * config.network_weight
 + sequential * config.sequential_weight;
 }

The inf * 0 = NaN guard is the kind of detail that separates a cost model that works from one that occasionally picks an impossible plan.

The formulas

A hash join prices the probe, the output and the hash-table build:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:130
static Cost hashJoinCost(const CostInputs & inputs, bool is_broadcast)
{
 const auto & left_stats = inputStats(inputs, 0);
 const auto & right_stats = inputStats(inputs, 1);

 Cost cost;
 cost.work = (left_stats.estimated_row_count + inputs.output_stats.estimated_row_count) / inputs.parallelism;

 /// The build side is materialized fully on every node for a broadcast join (the nodes build
 /// concurrently) and 1/N per node otherwise. The build is work, not a sequential phase:
 /// `parallel_hash` shards the hash table across threads within a node.
 const Float64 build_rows_per_node = is_broadcast
 ? right_stats.estimated_row_count
 : right_stats.estimated_row_count / inputs.parallelism;
 /// Hash table materialization: memory allocation + cache pressure.
 cost.work += inputs.config.hash_table_build_factor * build_rows_per_node
 + build_rows_per_node * right_stats.estimated_bytes_per_row;
 return cost;
}

The exchange prices the network, and the distinction between a shuffle and a gather is where most of the distributed reasoning lives:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:240
static Cost exchangeCost(const CostInputs & inputs, const IQueryPlanStep & step)
{
 /// A sorted gather over a partial top-N transfers min(input_rows, L * node_count) rows,
 /// not the group's trimmed L; the caller resolves that override from the memo.
 const Float64 rows = inputs.exchange_rows_override.value_or(inputs.output_stats.estimated_row_count);
 Cost cost;
 /// The network term is wall-clock per node, like the work term. A shuffle (N->N) moves
 /// 1/N of the data per node, all nodes concurrently; without the division one node
 /// scanning the whole table is priced the same as the cluster shuffling it, and the
 /// optimizer keeps big scans on a single node.
 Float64 network_bytes = rows * inputs.output_stats.estimated_bytes_per_row;
 if (dynamic_cast<const ShuffleExchangeStep *>(&step))
 network_bytes /= std::max(1.0, inputs.node_count);
 cost.network += network_bytes;
 cost.sequential += inputs.config.exchange_fixed_overhead;
 /// Gather (N->1) and Scatter (1->N) funnel every row through a single node that
 /// sends or receives them sequentially, so their transfer stays undivided and each row
 /// pays the funnel cost. Without it a gather of a large input looks as cheap as a
 /// shuffle, so the optimizer distributes work (e.g. a sort) that should stay local.
 if (dynamic_cast<const GatherExchangeStep *>(&step) || dynamic_cast<const ScatterExchangeStep *>(&step))
 cost.sequential += inputs.config.funnel_sequential_cost_per_row * rows;
 return cost;
}

Three subtleties are worth spelling out:

A shuffle divides the network cost by the node count, because each node ships 1/N of the data and all nodes do it concurrently.

A broadcast does not divide, because every node ingests the whole payload:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:227
static Cost broadcastExchangeCost(const CostInputs & inputs)
{
 Cost cost;
 /// The network term is wall-clock per node, like the work term (which is divided by
 /// parallelism): every receiving node ingests the full payload, all of them concurrently,
 /// and the senders each ship their share of the copies in parallel. So one full payload
 /// is the per-node bound. A shuffle moves 1/N per node, so a broadcast still costs N times
 /// a shuffle of the same data.
 cost.network += inputs.output_stats.estimated_row_count * inputs.output_stats.estimated_bytes_per_row;
 cost.sequential += inputs.config.exchange_fixed_overhead;
 return cost;
}

Every exchange pays a fixed overhead in sequential units, which is what keeps a plan over a small input local: distributing 25 rows is not worth a network round trip.

A read is priced on its scan volume, not its output

This one is subtle and important. A filter that the primary key can’t prune leaves the whole granule selection to scan, so the output estimate badly understates the cost:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Cost.cpp:150
/// One node reads the whole table.
static Cost fullReadCost(const CostInputs & inputs)
{
 /// Price the scan volume, not the output volume: a filter the primary key cannot prune
 /// leaves the whole granule selection to scan, and without this a replicated read of such
 /// a table would look orders of magnitude cheaper than the scan it does.
 if (inputs.output_stats.physical_read_bytes > 0)
 return Cost{.work = inputs.output_stats.physical_read_bytes};
 return Cost{.work = inputs.output_stats.estimated_row_count * inputs.output_stats.estimated_bytes_per_row};
}

Without this, ReplicatedRead — every node reading the whole table from S3 — would look free for a table whose filter prunes 99 % of the rows.

The cost model is query-tunable

Everything above is overridable with one JSON parameter:

 sql
 
SET param__internal_cascades_cost_config = '{"work_weight":1,"network_weight":1,"sequential_weight":32}';

and an invalid configuration rejects the query rather than silently falling back to the defaults.

The rules

Join implementations

One rule produces all the join alternatives:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/HashJoinImplementation.cpp:19
/// Produces all applicable hash join implementations in a single rule:
/// - Local join: both inputs gathered to one node (always applicable)
/// - Broadcast join: left partitioned any way, right replicated (only when node_count > 1)
/// - Partitioned (shuffle) join: both inputs shuffled by join keys (only when node_count > 1
/// and the join has equi-join predicates)
///
/// When node_count == 1 all three strategies produce the same plan, so only the local join
/// is emitted to avoid redundant identical alternatives in the memo.
class HashJoinImplementation : public IOptimizationRule

A shuffle join partitions both sides by the join key. The key columns are recorded positionally on both sides, together with the type they’re hashed at:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/HashJoinImplementation.cpp:316
 auto add_key = [&](const JoinKeyPair & key)
 {
 left_dist.columns.push_back({key.left});
 right_dist.columns.push_back({key.right});
 if (needs_hash_cast)
 {
 const String & type_name = hash_type_for(key);
 left_dist.hash_type_names.push_back(type_name);
 right_dist.hash_type_names.push_back(type_name);
 }
 /// Advertise the output partitioning only on the preserved side(s).
 NameSet output_key;
 if (left_preserved)
 output_key.insert(key.left);
 if (right_preserved)
 output_key.insert(key.right);

The hash_type_names exist because if the two sides have different key types, they must be cast to a common type before hashing. Otherwise the two sides land in different buckets and the join produces nothing.

Two-stage aggregation

The transformation rule splits a single-phase aggregation into a partial and a merge:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:92
/// Logical transformation that splits a single-phase Agg into a two-phase plan:
/// PartialAgg(input) -> ShuffleExchange -> FinalMergeAgg
///
/// PartialAgg computes partial aggregate states locally on each node without finalization.
/// FinalMergeAgg receives pre-aggregated states and merges them into final results.
/// The exchange between the two is inserted by the `DistributionEnforcer` based on the
/// distribution requirements set by the implementation rules on FinalMergeAgg.
///
/// A non-final aggregation is split too when it promises bucket order to a consumer outside the
/// plan: the merge then stays non-final and restores the bucket order over the gathered partials,
/// which is the distributed alternative to the single-node partial `AggregationImplementation`
/// leaves for such a step.
class TwoStageAggregationTransformation : public IOptimizationRule

and the split clones the aggregation step, marks the clone as non-final, and creates a MergingAggregatedStep for the merge half:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:345
 /// Phase 1: partial aggregation - takes raw rows, outputs intermediate aggregate states.
 auto partial_step_ptr = cloneStepAs(*agg_step);
 auto * partial_step = partial_step_ptr.get();
 partial_step->setFinal(false);

The shuffle aggregation is only applicable in some cases, and every one of those reasons is a correctness reason:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:64
 /// See `addShuffleAggregation` for why each condition disables the shuffle strategy.
 bool isShuffleApplicable() const
 {
 return !agg_step.getParams().keys.empty()
 && !agg_step.isGroupingSets()
 && !agg_step.getParams().overflow_row
 && agg_step.getParams().max_rows_to_group_by == 0;
 }

no group keys → a global aggregation has nothing to shuffle by;
grouping sets → the keys are the union of the sets’ keys, so shuffling by the union splits the rows of one grouping-set group across nodes;
an overflow row → every node would emit its own;
max_rows_to_group_by → the limit is a global contract, but each node would enforce it independently.

Single-key shuffle alternatives

A nice touch: for a multi-key GROUP BY, the optimizer also considers shuffling by one of the keys, which is cheaper (fewer bytes in the exchange) and still correct:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/AggregationImplementations.cpp:198
/// Single-key shuffle alternatives.
/// For aggregations with 2+ group-by keys, generate a shuffle alternative for each
/// individual key. Correctness: `GROUP BY (A, B)` with data shuffled by `A` is correct
/// because all rows with the same `(A, B)` have the same `A`, hence the same node.
void AggregationImplementation::StrategyEnumerator::addSingleKeyShuffleAggregations(const std::vector<size_t> & candidate_node_counts)

Enforcers: self-referential expressions

When a parent requires {1 node} and the cheapest child produces {4 nodes}, something has to bridge the gap. That something is an enforcer — an exchange step inserted by the optimizer.

Enforcers are the hairiest part of the implementation, because an enforcer’s input is the same group with relaxed requirements. That looks like a cycle:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/DistributionEnforcer.cpp:24
/// Produces self-referential enforcer expressions that bridge distribution gaps.
/// Each enforcer expression lives in the same group as the source expression; its
/// single input points back to the same group with normalized properties (node_count,
/// is_replicated, and sorting where needed) as the requirement - this lets the optimizer
/// pick the cheapest source with that distribution shape. The optimizer recursively
/// satisfies this self-referential input through the normal task mechanism, enabling
/// natural enforcer composition (e.g. a sort and a gather compose into gather-then-sort,
/// see `SortingEnforcer`, without
/// bundling steps).
class DistributionEnforcer : public IOptimizationRule

The construction is centralised:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Rules/DistributionEnforcer.cpp:104
void DistributionEnforcer::EnforcerEnumerator::addEnforcer(QueryPlanStepPtr step, ExpressionProperties input_required, SortDescription output_sorting)
{
 ExpressionProperties output_properties;
 output_properties.distribution = required_properties.distribution;
 output_properties.sorting = std::move(output_sorting);
 auto enforcer_expr = makeEnforcerExpression(
 expression, std::move(step), std::move(input_required), std::move(output_properties), EnforcedProperty::Distribution);

 rule.addPhysicalToMemo(enforcer_expr, required_properties, memo, result);
}

Two invariants keep the self-reference acyclic.

The first is that an enforcer carries an axis (Sorting or Distribution), and an enforcer that satisfies a wildcard axis only by over-providing on it isn’t eligible. A sorted enforcer can’t fill an empty-sort slot; a keyed exchange can’t fill an empty-columns slot:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:159
 if (input_is_self_referential && candidate->enforced_property != EnforcedProperty::None)
 {
 /// Empty sort requirement: reject a sorted enforcer (it over-provides on the sort axis).
 if (required_properties.sorting.empty() && !candidate->properties.sorting.empty())
 return;
 /// Empty distribution-columns requirement: reject a keyed exchange. A sorting enforcer
 /// carrying keyed columns is still needed to feed a sorted gather, so keep it eligible.
 if (candidate->enforced_property == EnforcedProperty::Distribution
 && required_properties.distribution.columns.empty()
 && (!candidate->properties.distribution.columns.empty()
 || !candidate->properties.distribution.hash_type_names.empty()))
 return;
 }

The second is that an enforcer must never evict a non-enforcer alternative from the best-implementation cache, because that alternative is the acyclic way out:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:78
 /// An enforcer "satisfies" a weaker non-enforcer only by over-providing (e.g. a sorted gather
 /// covers an empty-sort requirement). Dropping the base would leave a self-referential enforcer
 /// chain with no acyclic alternative, so an enforcer never suppresses or evicts a distinct base.
 auto enforcer_over_provides_for = [](const GroupExpressionPtr & enforcer_candidate, const GroupExpressionPtr & base_candidate)
 {
 return enforcer_candidate->enforced_property != EnforcedProperty::None
 && base_candidate->enforced_property == EnforcedProperty::None
 && !(enforcer_candidate->properties == base_candidate->properties);
 };

The fixed point

Enforcers compose: a sort enables a sorted gather, which enables a gather. The Stage 3 loop iterates until no new enforcer is produced:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Task.cpp:96
 /// Fixed-point loop: iterate over newly-added physical expressions until no
 /// new enforcers are produced. Each iteration may create expressions that
 /// enable further enforcers (e.g. a sort enables a sorted `GatherExchange`).
 size_t enforced_up_to = 0;
 bool new_enforcers_created = true;
 while (new_enforcers_created)
 {
 new_enforcers_created = false;

Keeping the best implementation

Each group keeps a Pareto frontier of implementations, bucketed by distribution shape so that a lookup is O(1) plus a short scan:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:73
void Group::updateBestImplementation(GroupExpressionPtr expression, const CostConfig & cost_config)
{
 UInt64 key = distributionKey(expression->properties.distribution);
 auto & bucket = best_implementations[key];

and the eviction rule is a domination test: an implementation is dropped when another one satisfies at least the same properties at a lower cost.

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Group.cpp:88
 /// Remove all known best expressions with higher cost and properties satisfied by the new expression.
 /// Only the matching distribution-shape bucket needs checking - `isSatisfiedBy` requires
 /// exact match on (node_count, is_replicated).
 for (auto best_it = bucket.begin(); best_it != bucket.end();)
 {
 if (expression->properties.isSatisfiedBy((*best_it)->properties) &&
 (*best_it)->cost->subtree_cost.total(cost_config) <= expression->cost->subtree_cost.total(cost_config) &&
 !enforcer_over_provides_for(*best_it, expression))
 {
 /// There is already a cheaper implementation that satisfies the same properties
 return;
 }

Failing closed

The search is bounded by a task count, not by branch-and-bound. And when the budget runs out, the query is rejected rather than planned from a partial memo:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:308
 /// Limit the time in terms of optimization tasks instead of wall clock time. This is done for stability of generated plans regardless of system load.
 /// Microsoft SQL Server's optimizer team describes this in Andy Pavlo's seminar: https://www.youtube.com/watch?v=pQe1LQJiXN0
 const size_t executed_tasks_limit = getCascadesTaskLimitParam(query_context, CascadesDefaults::DEFAULT_TASK_LIMIT);
 size_t executed_tasks_count = 0;
 for (; !tasks.empty() && executed_tasks_count < executed_tasks_limit; ++executed_tasks_count)
 {
 auto task = std::move(tasks.top());
 tasks.pop();
 task->execute(*this);
 }

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/Optimizer.cpp:321
 /// Fail closed if the search did not finish within the task budget: building a plan from a
 /// partial memo can yield a non-minimal plan or a confusing failure deep inside buildBestPlan.
 /// Surface a clear error instead and point at the knob to raise the limit.
 if (!tasks.empty())
 throw Exception(ErrorCodes::SUPPORT_IS_DISABLED,
 "Cascades optimizer did not finish within the task budget of {} tasks "
 "(root group #{}, required properties {}, {} groups in memo, {} tasks left, next: {}). "
 "The distributed Cascades optimizer is experimental; set enable_cascades_optimizer = 0 "
 "or simplify the query.",
 executed_tasks_limit, root_group_id, root_required_properties.dump(),
 memo.getGroupCount(), tasks.size(), tasks.top()->describe());

The default budget is 100 000 tasks:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/Cascades/OptimizerDefaults.h:17
constexpr size_t DEFAULT_TASK_LIMIT = 100000;

Trying it

 sql
 
 
SET enable_cascades_optimizer = 1;
SET make_distributed_plan = 1;

SELECT n_name, SUM(o_totalprice) AS revenue
FROM orders
JOIN customer ON o_custkey = c_custkey
JOIN nation ON c_nationkey = n_nationkey
WHERE o_orderdate >= '1995-01-01' AND o_orderdate < '1995-04-01'
GROUP BY n_name
ORDER BY revenue DESC
SETTINGS send_logs_level = 'test';

The test log level prints the memo, every costed expression with its work/network/sequential breakdown, and the final plan — which is the intended way to understand why the optimizer chose what it chose.

Trade-offs

What it buys: plans that no fixed heuristic would produce. A broadcast of a tiny table instead of a shuffle of a huge one. A ReplicatedRead from shared storage instead of a network transfer. A two-phase aggregation instead of a gather. A sort per node instead of a sort of everything.

What it costs:

It is experimental. The setting is EXPERIMENTAL and off by default, and it only applies with make_distributed_plan = 1.
Join ordering is not part of it. This is the biggest difference from classical Cascades. ClickHouse fixes the join order first (with the existing dpsize/dphyp/greedy algorithms) and Cascades only optimises the distribution strategy. The published literature calls this the “split” design, and it loses the global optimum when the join order interacts with physical properties — which, in a distributed setting, it does.
The statistics are weak. Today they come from index analysis and from query-parameter hints. There’s no cardinality estimator as sophisticated as the one the pre-Cascades join orderer uses. A cost model on bad statistics picks the wrong plan confidently.
Only two parallelism levels are explored: {1 node} and {N nodes}. Intermediate counts were never chosen on TPC-H, and exploring them would multiply the search space.
The search can be exponential, which is why there’s a hard task budget and a fail-closed error. Classical Cascades would prune instead.
Some plan shapes are rejected outright — WITH TOTALS, ROLLUP, CUBE, extremes, PASTE JOIN, and reads that cannot be cloned across workers.

Wrapping up

Here’s the thing: the most interesting part of this code isn’t the optimizer. It’s the discipline around it.

Every place where the implementation deliberately deviates from the textbook — no branch-and-bound, a fail-closed budget, the enforcer-axis rules, the “never evict the acyclic fallback” rule — carries a comment explaining exactly why the textbook version would be wrong here. That’s rare, and it’s the part of the code most worth reading.

What to do about it: if you’re running a distributed ClickHouse cluster, turn enable_cascades_optimizer = 1 on a staging cluster, set send_logs_level = 'test', and read the memo for one of your real queries. You’ll see immediately whether your statistics are good enough for it to trust itself.

And if you hit the task budget, that’s not a bug — that’s the optimizer telling you it couldn’t afford to think. Fix the query shape, or raise the limit.

ClickHouse Packs Short Strings Into Hash Table Keys: No Allocation Per Key

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse never stores a std::string in a hash table cell. It splits keys into size classes and packs strings of up to 24 bytes into plain integers, so your GROUP BY allocates nothing per key.

Strings are the most common GROUP BY key you’ll ever write. They’re also the worst case for a hash table.

Put a std::string in a cell and you get one heap allocation per distinct key. Put a std::string_view in a cell and the bytes have to outlive the table — which for an aggregation means copying them into an arena first.

ClickHouse’s answer is a hash table that is not a hash table. Instead of one table with a variable-size key, StringHashTable keeps five separate tables, one per key size class, and packs short keys into fixed-width integers.

The payoff: every cell is the same size, no key ever allocates, and the compiler can dispatch on key length at compile time.

The whole thing lives in src/Common/HashTable/StringHashTable.h, with the map specialisation in StringHashMap.h.

Five tables, one per size class

The composite table is five sub-tables:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:234
template <typename SubMaps>
class StringHashTable : private boost::noncopyable
{
protected:
 static constexpr size_t NUM_MAPS = 5;
 // Map for storing empty string
 using T0 = typename SubMaps::T0;

 // Short strings are stored as numbers
 using T1 = typename SubMaps::T1;
 using T2 = typename SubMaps::T2;
 using T3 = typename SubMaps::T3;

 // Long strings are stored as std::string_view along with saved hash
 using Ts = typename SubMaps::Ts;
 using Self = StringHashTable;

and the storage:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:254
 T0 m0;
 T1 m1;
 T2 m2;
 T3 m3;
 Ts ms;

Here’s what lives where:

Table	Key type	Holds
`m0`	—	the empty string, in a dedicated slot
`m1`	`StringKey8` (`UInt64`)	1..8 bytes
`m2`	`StringKey16` (`UInt128`)	9..16 bytes
`m3`	`StringKey24`	17..24 bytes
`ms`	`std::string_view` + saved hash	25+ bytes, or short strings with a trailing zero

And the key types are just integers:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:10
using StringKey8 = UInt64;
using StringKey16 = UInt128;
struct StringKey24
{
 UInt64 a;
 UInt64 b;
 UInt64 c;

 bool operator==(const StringKey24 rhs) const { return a == rhs.a && b == rhs.b && c == rhs.c; }
};

So a 20-byte string becomes three UInt64s and nothing else. No length field. No pointer. No allocation.

The length is implicit — it’s recovered from the position of the first zero byte.

The packing: how a string becomes a number

The heart of the file is dispatchOnKeyClass. It classifies a key, packs it, hashes it, and calls a callback with the right sub-table. It’s written for speed, and the comment says so out loud:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:357
 // Dispatch is written in a way that maximizes the performance:
 // 1. Always memcpy 8 times bytes
 // 2. Use switch case extension to generate fast dispatching table
 // 3. Funcs are named callables that can be force_inlined
 //
 /// Note: the key bytes must be readable in 8-byte chunks (a `std::string` cannot be passed
 /// here, while e.g. `ColumnString::getDataAt` can), because the packing copies 8 bytes at a
 /// time.
 template <typename Resolver, typename KeyHolder, typename HashProvider, typename Func>
 static auto ALWAYS_INLINE dispatchOnKeyClass(Resolver && resolver, KeyHolder && key_holder, HashProvider && hash_of, Func && func)
 {
 const auto & x = keyHolderGetKey(key_holder);
 const size_t sz = x.size();
 if (sz == 0)
 {
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 0>{}, 0), VoidKey{}, 0);
 }

 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

 const char * p = x.data();
 // pending bits that needs to be shifted out
 const char s = (-sz & 7) * 8;

The switch ((sz - 1) >> 3) that follows turns the length into one of four cases, and the compiler turns that into a jump table or a chain of comparisons:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:394
 switch ((sz - 1) >> 3)
 {
 case 0: // 1..8 bytes
 {
 // first half page
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)
 {
 memcpy(&n[0], p, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] &= -1ULL >> s;
 else
 n[0] &= -1ULL << s;
 }
 else
 {
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[0], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] >>= s;
 else
 n[0] <<= s;
 }
 const size_t res = hash_of(k8);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 1>{}, res), k8, res);
 }

Three tricks are hiding in those twenty lines. This is the part people get wrong the first time they read it.

Trick one: the trailing-zero test. A fixed-size string key encodes its length implicitly, by the position of the first zero byte. That means a string whose last byte is zero cannot be represented at all: "ab\0" and "ab" would pack to the same 8 bytes. So such strings get pushed to the generic table before the switch:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:376
 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

Trick two: the mask/shift pair. s = (-sz & 7) * 8 is the number of padding bits you need to round the length up to a multiple of 8. In the first branch, the 8 bytes are read forwards from the start of the string and the padding bits are cleared with a mask. In the second branch, the 8 bytes are read backwards from the end of the string and the padding bits are shifted out.

Trick three: the “first half page” rule. Reading 8 bytes from a 3-byte string reads 5 bytes past its end. That’s only safe if those bytes are mapped. The condition

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:399
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)

tests whether the address is 2048-byte aligned. If it is, at least 2048 bytes of the page are still ahead of it, so reading 8 bytes forward cannot fault. Otherwise the code reads backwards instead, ending exactly at the last byte of the string.

The bytes before the string that this reads are then shifted out — the >>= s discards exactly the (-sz & 7) bytes that belong to the previous string or to the column’s offset array. So the garbage is read but never used.

The 9..16 and 17..24 byte cases use the same trick with two and three words:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:420
 case 1: // 9..16 bytes
 {
 memcpy(&n[0], p, 8);
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[1], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[1] >>= s;
 else
 n[1] <<= s;
 const size_t res = hash_of(k16);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 2>{}, res), k16, res);
 }

Notice that the first word is always read forwards from p — for a 9-byte string that’s inside the string — while the last word is read backwards from the end. The overlap between them is harmless: the bytes in the middle get read twice with the same values.

Unpacking: getting the string back

Going the other way, the length is recovered from the number of leading zero bytes:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:21
inline std::string_view ALWAYS_INLINE toStringView(const StringKey8 & n)
{
 chassert(n != 0);
 if constexpr (std::endian::native == std::endian::big)
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countr_zero(n) >> 3)};
 else
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countl_zero(n) >> 3)};
}

countl_zero(n) >> 3 is the number of whole zero bytes at the end of the word, so 8 - that is the string length. On a big-endian machine the zero bytes sit at the other end, hence the countr_zero variant.

The empty-cell rule, specialised per size class

The “empty cell is a zeroed cell” rule from the generic table needs a sensible notion of “zero” for a packed string key. For StringKey16 the rule is “the last word is zero”:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:20
template <typename TMapped>
struct StringHashMapCell<StringKey16, TMapped> : public HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>
{
 using Base = HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>;
 using value_type = typename Base::value_type;
 using Base::Base;
 static constexpr bool need_zero_value_storage = false;
 bool isZero(const HashTableNoState & state) const { return isZero(this->value.first, state); }

 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey16 & key, const HashTableNoState &) { return key.items[1] == 0; }
 void setZero() { this->value.first.items[1] = 0; }

The reasoning is the same as in the generic table. A packed key with a zero last word would have to encode the length “16” with no room for a terminating zero byte, so such a key cannot exist. The encoding is injective on all valid keys, and the all-zero pattern is unreachable. That makes it safe to use as the empty marker.

Same for StringKey24, where the marker is the third word:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:49
 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey24 & key, const HashTableNoState &)
 { return key.c == 0; }
 void setZero() { this->value.first.c = 0; }

The generic std::string_view cell can’t do this. A string_view is a pointer and a length, and a zeroed one is a null pointer with length 0 — which is a legal key, namely the empty string. So the generic cell declares that it needs no zero storage, and the empty string gets its own slot instead:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:134
template <typename Cell>
struct StringHashTableEmpty // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
 using Self = StringHashTableEmpty;

 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

And that empty slot isn’t a hash table at all — it’s a single cell with an emplace that ignores the key entirely:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:172
 template <typename KeyHolder>
 void ALWAYS_INLINE emplace(KeyHolder &&, LookupResult & it, bool & inserted, size_t = 0)
 {
 if (!hasZero())
 {
 setHasZero();
 inserted = true;
 }
 else
 inserted = false;
 it = zeroValue();
 }

Hashing: CRC32C where it can, CityHash where it can’t

The hash function is chosen per class, and for the fixed-size classes it’s the hardware CRC32C instruction:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:46
struct StringHashTableHash
{
#if defined(__SSE4_2__)
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key);
 return res;
 }
 size_t ALWAYS_INLINE operator()(StringKey16 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key.items[0]);
 res = _mm_crc32_u64(res, key.items[1]);
 return res;
 }

with an AArch64 CRC32 path, an s390x path, and CityHash as the portable fallback:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:114
#else
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 return CityHash_v1_0_2::CityHash64(reinterpret_cast<const char *>(&key), 8);
 }

CRC32 is a poor hash function in the abstract — the header of the generic hash file says so — but it’s one instruction per 8 bytes with a three-cycle latency, and the table is small enough to fit in cache most of the time.

There’s another requirement, and it’s the one that actually constrains the design: the hash has to be consistent with the packing. The hash is computed over the packed word, not over the original bytes, so "abc" and "abc\0..." can’t collide by accident. They hash different words.

The lookup result: why you never see the key

The string table doesn’t return a pointer to a cell. It returns a pointer to the mapped value, wrapped in a small type:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:214
template <typename Mapped>
struct StringHashTableLookupResult
{
 Mapped * mapped_ptr;
 StringHashTableLookupResult() {} /// NOLINT
 StringHashTableLookupResult(Mapped * mapped_ptr_) : mapped_ptr(mapped_ptr_) {} /// NOLINT
 StringHashTableLookupResult(std::nullptr_t) {} /// NOLINT
 const VoidKey getKey() const { return {}; } /// NOLINT
 auto & getMapped() { return *mapped_ptr; }
 auto & operator*() { return *this; }
 auto & operator*() const { return *this; }
 auto * operator->() { return this; }
 auto * operator->() const { return this; }
 explicit operator bool() const { return mapped_ptr; }

The reason is that the key lives in the cell, and the caller usually only wants the aggregate state. But the key is still there — just in a different sub-table with a different representation. Returning a pointer to it would mean returning a pointer to a StringKey8 or to a string_view, which have different types depending on the size class.

So the key is erased from the interface: getKey() returns a VoidKey.

Reserving and growing: where the heuristic leaks

The string table grows differently from the generic one:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:207
template <size_t initial_size_degree = 8>
struct StringHashTableGrower : public HashTableGrowerWithPrecalculation<initial_size_degree>
{
 // Smooth growing for string maps
 void increaseSize() { this->increaseSizeDegree(1); }
};

The generic grower doubles the buffer on overflow. The string grower grows by a factor of two in the exponent, i.e. by one power of two — the same thing, but expressed as a single degree increment so it composes with the precalculated mask.

Reserving is divided evenly over the four real sub-maps, because the caller only knows the total number of keys:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:317
 explicit StringHashTable(size_t reserve_for_num_elements)
 : m1{reserve_for_num_elements / 4}
 , m2{reserve_for_num_elements / 4}
 , m3{reserve_for_num_elements / 4}
 , ms{reserve_for_num_elements / 4}
 {
 }

That guess is often wrong. A table of 20-byte strings uses m3 only, and m1, m2, ms reserve space they will never touch.

The aggregator knows this, and refuses to pre-reserve the string tables:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5214
 /// `StringHashTable::reserve` splits the hint evenly over its four size-class sub-maps,
 /// while a real key set concentrates in one of them, so it is not reserved.
 constexpr bool can_reserve = requires { dst.reserve(size_t{}); } && !requires { dst.emptyStringSlot(); };

And the emptyStringSlot() method exists precisely so that this requires expression can tell a StringHashTable from an ordinary hash table:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:312
 /// The dedicated slot of the empty-string key, which the sub-table dispatch hashes to 0.
 T0 & emptyStringSlot() { return m0; }

Where it’s used: three string-key aggregation variants

The string map is one of the standard aggregation data variants:

 cpp
 
 
// src/Interpreters/AggregatedData.h:53
using AggregatedDataWithPackedStringKey = HashMap<PackedStringRef, AggregateDataPtr>;
using AggregatedDataWithShortStringKey = StringHashMap<AggregateDataPtr>;
using AggregatedDataWithStringKey = HashMapWithSavedHash<std::string_view, AggregateDataPtr>;

Three flavours of string key, three tables:

AggregatedDataWithPackedStringKey — a PackedStringRef, a pointer plus a length, with the payload inline for short strings;
AggregatedDataWithShortStringKey — the composite StringHashMap of this article;
AggregatedDataWithStringKey — a plain std::string_view map with a saved hash, where the key bytes live in an arena.

The StringHashMap variant is the interesting one, because it’s the only one where the key doesn’t live anywhere else. The string is in the table, as three machine words.

Trade-offs: what you win and what you pay

What the packed string table buys you:

No allocation per key. A 20-byte key is three UInt64s inside the cell. Nothing is copied into an arena, nothing is freed.
A uniform cell size. Every cell of m1 is 8 bytes of key plus the mapped value; every cell of m3 is 24 bytes plus the mapped value. The table can be scanned, and the compiler can generate good code for it.
A compile-time dispatch on length. dispatchOnKeyClass is a template over the size class, so each branch is specialised: the m1 path hashes one word, the m3 path hashes three.
A hash over the packed form. The hash is computed over exactly the bytes that are stored, so the stored key and the hashed key can never disagree.

What it costs you:

The bytes of the key are duplicated. The cell holds the packed key, and if the caller needs the original string it has to be reconstructed with toStringView.
The trailing-zero case is a wart. A string ending in \0 — which happens with binary data, and with FixedString columns padded with zeros — can’t use the packed classes and falls back to the generic table, where the key is a string_view into the arena.
The reservation heuristic is bad. Reserving n/4 for each of the four sub-maps wastes memory whenever the keys concentrate in one class, which is the normal case.
The empty string needs its own slot, and the key type of the generic sub-table isn’t the same as the key type of the packed ones, so the composite has to be written as a template over a SubMaps bundle rather than as a plain HashTable.

This is a good example of a general principle: when the key type has a size distribution, splitting the table by size class turns a variable-size-cell problem into a set of fixed-size-cell problems, and lets the compiler specialise each one.

Wrapping up

Worth internalizing: a GROUP BY over strings is a memory-bound benchmark before it’s a CPU-bound one. ClickHouse’s whole trick here is to make the key be the data — no arena, no pointer chasing, no allocator traffic.

So if you’re tuning an aggregation and it feels allocation-heavy, check what your key type maps to. Short, non-zero-terminated strings hit the fast path. FixedString padded with zeros and raw binary blobs do not — and that’s a silent cliff, not an error message.

Next time you’re profiling, watch for memcpy traffic coming out of the aggregator rather than out of the storage layer. That’s the signature of this code path not doing what you expect.

ClickHouse Merges Columns One at a Time: The Vertical Merge Algorithm

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse’s vertical merge splits a merge into two stages: merge the key and record where each output row came from (one byte per row), then gather each payload column separately. Peak memory drops from “all columns at once” to “one column at a time” — which is the difference between a merge that OOMs and one that doesn’t.

Every MergeTree table is a pile of immutable parts. Every part is a pile of immutable columns. Background merges take a contiguous range of parts and rewrite them as one bigger part. That sounds like a simple copy, and for a table with three narrow columns it is one.

Now consider a table with two hundred columns, one of which is a 40 MB String of raw JSON and the rest are UInt64 counters.

The obvious merge implementation reads all two hundred columns of the source parts, merges the sorting key, and writes two hundred columns out. It has to hold, in memory at once, the merged key plus every payload column of the rows it is currently holding — which for a wide table means the peak memory of a merge is proportional to the sum of all column widths, not to the width of the widest column.

That is a bad trade. The payload columns are almost always much larger than the key, and they are almost always independent of the key: merge the keys first and separately, and you can then reconstruct each payload column without ever holding two payload columns in memory at the same time.

ClickHouse implements exactly that. It is called the vertical merge algorithm, and it is one of the more elegant pieces of engineering in the MergeTree code base. The trick that makes it work is a one-byte-per-row side-channel file that records where each output row came from.

The problem, stated precisely

Suppose we merge three parts, each sorted by the key k:

 text
 
 
 part A (10 rows) part B (10 rows) part C (10 rows)
 k=1 a1=... k=1 b1=... k=2 c1=...
 k=1 a2=... k=2 b2=... k=2 c2=...
 k=3 a3=... k=2 b3=... k=5 c3=...
 ... ... ...

A horizontal merge reads all three parts’ k columns, merges them into one sorted stream, and then — because MergeTree guarantees that all rows with the same key are adjacent and, within a key, appear in source-part order — it can reconstruct every other column by a gather: walk the merged key stream, and for each row, take the next value of that column from the part the row came from.

The output order of the merged key is not the concatenation of the inputs; it is the k-way merge of them. So for each output row we need to know which input part it came from.

That is the entire information content of the vertical merge’s side channel.

And it is exactly one small integer per row.

The horizontal stage: merging the key and writing the row sources

The vertical merge is not a separate code path that replaces the normal merge. It is a second stage that runs after the normal merge has merged the key. The normal merge is called the “horizontal” stage, and the code says so explicitly — the stage class is called ExecuteAndFinalizeHorizontalPart:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.h:63
/**
 * Overview of the merge algorithm
 *
 * Each merge is executed sequentially block by block.
 * The main idea is to make a merge not a subroutine which is executed
 * in a thread pool and may occupy a thread for a period of time,
 * but to make a merge a coroutine which can suspend the execution
 * in some points and then resume the execution from this point.
 *
 * A perfect point where to suspend the execution is after the work over a block is finished.
 * The task itself will be executed via BackgroundJobExecutor.
 *
 * The interface of the task is simple.
 * The main method is `execute()` which will return true, if the task wants to be executed again and false otherwise.
 *
 * With this kind of task we can give a merge a priority.
 * A priority is simple - the lower the size of the merge, the higher priority.
 * So, if ClickHouse wants to merge some really big parts into a bigger part,
 * then it will be executed for a long time, because the result of the merge is not really needed immediately.
 * It is better to merge small parts as soon as possible.
*/

Note the second half of that comment: the merge is a coroutine. It is resumed by the background pool between blocks, which is what makes it possible to give a small merge priority over a huge one without pinning a thread. The vertical stage is expressed in the same idiom: a list of sub-tasks, each of which returns true when it wants to be resumed.

The side channel is a real file, managed by a small class:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:257
/// Manages the "rows_sources" temporary file that is used during vertical merge.
class RowsSourcesTemporaryFile : public ITemporaryFileLookup
{
public:
 /// A logical name of the temporary file under which it will be known to the plan steps that use it.
 static constexpr auto FILE_ID = "rows_sources";

 explicit RowsSourcesTemporaryFile(TemporaryDataOnDiskScopePtr temporary_data_on_disk_)
 : temporary_data_on_disk(temporary_data_on_disk_->childScope({CurrentMetrics::TemporaryFilesForMerge}))
 {
 }

and it is deliberately re-openable, so that several columns can be gathered in parallel from independent file handles — getTemporaryFileForReading returns tmp_data_buffer->read(), a fresh reader each time, with the comment “Reopen the file for each read so that multiple reads can be performed in parallel and there is no need to seek to the beginning.”

That matters: the row-source stream is consumed once per column, and each column’s gather starts from the beginning of the file. If the file were a single sequential stream, gathering column #2 would have to seek back to offset 0 and re-read column #1’s worth of bytes. Re-opening gives every column its own independent read cursor.

The encoding: one byte per row

The row source is not a struct with a source number and a flag in two fields. It is a single byte with two bit-fields packed into it:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:15
/// Tiny struct, stores number of a Part from which current row was fetched, and insertion flag.
struct RowSourcePart
{
 UInt8 data = 0;

 RowSourcePart() = default;

 explicit RowSourcePart(size_t source_num, bool skip_flag = false)
 {
 static_assert(sizeof(*this) == 1, "Size of RowSourcePart is too big due to compiler settings");
 setSourceNum(source_num);
 setSkipFlag(skip_flag);
 }

 size_t getSourceNum() const { return data & MASK_NUMBER; }

 /// In CollapsingMergeTree case flag means "skip this rows"
 bool getSkipFlag() const { return (data & MASK_FLAG) != 0; }

 void setSourceNum(size_t source_num)
 {
 data = (data & MASK_FLAG) | (static_cast<UInt8>(source_num) & MASK_NUMBER);
 }

 void setSkipFlag(bool flag)
 {
 data = flag ? data | MASK_FLAG : data & ~MASK_FLAG;
 }

 static constexpr size_t MAX_PARTS = 0x7F;
 static constexpr UInt8 MASK_NUMBER = 0x7F;
 static constexpr UInt8 MASK_FLAG = 0x80;
};

The top bit is the “skip this row” flag (used when a lightweight delete mask removes rows during the merge), and the low seven bits are the source part number. Seven bits means at most 127 source parts per merge — and you can watch that limit propagate all the way into the merge-algorithm choice (see “When is it worth it?” below).

The static_assert is not decoration. If the compiler ever made this struct two bytes, the row-source file would double in size and every write to it would become a two-byte write. The assert turns a silent performance regression into a build failure.

Who writes the bytes

The horizontal stage’s merge transform writes one byte per output row as it emits it. For the Ordinary merging mode that is the plain k-way merge algorithm:

 cpp
 
 
// src/Processors/Merges/Algorithms/MergingSortedAlgorithm.cpp:444
void MergingSortedAlgorithm::insertRow(const SortCursorImpl & current)
{
 auto write_row_source = [&](bool skipped)
 {
 if (out_row_sources_buf)
 {
 RowSourcePart row_source(current.order, skipped);
 out_row_sources_buf->write(row_source.data);
 }
 };

 size_t current_row = current.getRow();

 if (const auto * mask = source_row_filter_masks[current.order])
 {
 if ((*mask)[current_row])
 merged_data.insertRow(current.all_columns, current_row, current.rows);

 write_row_source(!(*mask)[current_row]);
 }
 else
 {
 merged_data.insertRow(current.all_columns, current_row, current.rows);
 write_row_source(false);
 }
}

current.order is the index of the source stream the row came from. The merged algorithm already knows this — it is the whole reason it can merge — so emitting it costs one byte write per row. One byte per row of overhead on a merge that is about to rewrite gigabytes.

Note the ordering: merged_data.insertRow(...) happens first, then the row source is written. The write buffer is only non-null for a vertical merge — the merge step is constructed with the file name only when the vertical algorithm was chosen:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3654
 auto merge_step = std::make_unique<MergePartsStep>(
 merge_parts_query_plan.getCurrentHeader(),
 sort_description,
 partition_and_sorting_required_columns,
 global_ctx->merging_params,
 (is_vertical_merge ? RowsSourcesTemporaryFile::FILE_ID : ""), /// rows_sources' temporary file is used only for vertical merge
 filter_column_name,
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size],
 (*merge_tree_settings)[MergeTreeSetting::merge_max_block_size_bytes],
 max_dynamic_subcolumns,
 is_vertical_merge,
 sorting_queue_strategy,
 cleanup,
 global_ctx->time_of_merge);

so a horizontal merge pays nothing for the side channel.

The vertical stage: gathering one column at a time

Now the second stage. It does not merge anything. It gathers. For each non-key column, in turn:

open a read stream for that column in every source part;
union those streams;
run a ColumnGathererTransform over them, which uses the row-source file to decide which source’s value belongs at each output position.

Here is the step that wires the gatherer into the plan:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2172
 createReadFromPartStep(
 MergeTreeSequentialSourceType::Merge,
 *plan_for_part,
 *global_ctx->data,
 global_ctx->storage_snapshot,
 RangesInDataPart(global_ctx->future_part->parts[part_num], nullptr, part_num, part_starting_offset),
 global_ctx->alter_conversions[part_num],
 global_ctx->merged_part_offsets,
 Names{column_name},
 global_ctx->input_rows_filtered,
 apply_deleted_mask,
 /*filter=*/ std::nullopt,
 ctx->read_with_direct_io,
 ctx->use_prefetch,
 global_ctx->context,
 getLogger("VerticalMergeStage"));

Names{column_name} — exactly one column is read per part. The read plan for a vertical merge is a column read plan, not a row read plan. Every source part is opened, and only the column of interest is decoded.

Then the per-part streams are unioned, and a ColumnGathererStep is added to the plan. The gatherer is a merging transform, so it plugs into the same IMergingTransform machinery as the key merge: its input is one column per source part, its output is one merged column, and the row-source file is handed to it as a ReadBuffer by name.

The gather loop

The heart of it is ColumnGathererStream::gather. It reads the row-source bytes and copies values out of the per-part source columns:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.h:131
template <typename Column>
void ColumnGathererStream::gather(Column & column_res)
{
 row_sources_buf.nextIfAtEnd();
 RowSourcePart * row_source_pos = reinterpret_cast<RowSourcePart *>(row_sources_buf.position());
 RowSourcePart * row_sources_end = reinterpret_cast<RowSourcePart *>(row_sources_buf.buffer().end());

 if (next_required_source == -1)
 {
 /// Start new column.
 /// Actually reserve works only for fixed size columns.
 /// So it's safe to ignore preferred size in bytes and call reserve for number of rows.
 size_t size_to_reserve = std::min(static_cast<size_t>(row_sources_end - row_source_pos), block_preferred_size_rows);
 column_res.reserve(size_to_reserve);
 }

 next_required_source = -1;

 /// We use do ... while here to ensure there will be at least one iteration of this loop.
 /// Because the column_res.byteSize() could be bigger than block_preferred_size_bytes already at this point.
 do
 {
 if (row_source_pos >= row_sources_end)
 break;

 RowSourcePart row_source = *row_source_pos;
 size_t source_num = row_source.getSourceNum();
 Source & source = sources[source_num];
 bool source_skip = row_source.getSkipFlag();

 if (source.pos >= source.size) /// Fetch new block from source_num part
 {
 next_required_source = source_num;
 return;
 }

 ++row_source_pos;

 /// Consecutive optimization. TODO: precompute lengths
 size_t len = 1;
 size_t max_len = std::min(static_cast<size_t>(row_sources_end - row_source_pos), source.size - source.pos); // interval should be in the same block

 while (len < max_len && row_source_pos->data == row_source.data)
 {
 ++len;
 ++row_source_pos;
 }

 row_sources_buf.position() = reinterpret_cast<char *>(row_source_pos);

 if (!source_skip)
 {
 /// Whole block could be produced via copying pointer from current block
 if (source.pos == 0 && source.size == len)
 {
 /// If current block already contains data, return it.
 /// Whole column from current source will be returned on next read() iteration.
 source_to_fully_copy = &source;
 return;
 }
 if (len == 1)
 column_res.insertFrom(*source.column, source.pos);
 else
 column_res.insertRangeFrom(*source.column, source.pos, len);
 }

 source.pos += len;
 } while (column_res.size() < block_preferred_size_rows && column_res.byteSize() < block_preferred_size_bytes);
}

Three optimizations are hiding in there.

Run-length coalescing. The inner while loop scans forward over consecutive row-source bytes that are identical. Since the byte encodes “which source part”, a run of identical bytes means “these N output rows came from the same source part, consecutively” — which, because the source part’s own column is read in order, means “these N output rows are exactly source part’s rows [pos, pos+N)”. So the whole run is copied with a single insertRangeFrom call instead of N single-element inserts. The comment /// Consecutive optimization. TODO: precompute lengths is honest about the remaining inefficiency: the run length is rediscovered for every column, N times.

Whole-column pointer steal. If the run covers the entire remaining source column (source.pos == 0 && source.size == len), the output column can simply be the source column — no copy at all. The transform returns the source ColumnPtr directly and sets source_to_fully_copy, and the merge() method hands it out:

 cpp
 
 
// src/Processors/Transforms/ColumnGathererTransform.cpp:114
 else
 {
 res.addColumn(source_to_fully_copy->column);
 }

For a merge where one part dominates — the common case after a few rounds of merging, where you merge one big part with one small one — most columns of the big part are copied by pointer. The “merge” of such a column is essentially free.

Sparse output. The source_skip flag comes from the lightweight-delete mask: rows deleted by a DELETE are recorded in the row-source stream with the skip bit set, and the gatherer simply does not copy them out. The horizontal stage does the same filtering, so the two stages agree on which rows exist without either having to re-evaluate the delete predicate.

Writing the gathered column

The gathered column is written by MergedColumnOnlyOutputStream — the vertical stage’s counterpart of MergedBlockOutputStream, except that it writes exactly one column. Its constructor carries the invariant that makes the whole thing safe:

 cpp
 
 
// src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:36
 /// Granularity is never recomputed while writing only columns.
 MergeTreeWriterSettings writer_settings(

The horizontal stage has already decided where the granule boundaries are (it wrote the marks for the key columns), and the payload columns must use the same boundaries, or the part would be corrupt.

When is it worth it?

Vertical merging is not always a win. It has costs: an extra pass over the row-source file per column, an extra file in the temporary space, and a pipeline per column. So chooseMergeAlgorithm() gates it behind a set of conditions:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3760
MergeAlgorithm MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm() const
{
 const size_t total_rows_count = global_ctx->merge_list_element_ptr->total_rows_count;
 const size_t total_size_bytes_uncompressed = global_ctx->merge_list_element_ptr->total_size_bytes_uncompressed;
 const auto & merge_tree_settings = global_ctx->data_settings;

 if (global_ctx->deduplicate)
 return MergeAlgorithm::Horizontal;
 if ((*merge_tree_settings)[MergeTreeSetting::enable_vertical_merge_algorithm] == 0)
 return MergeAlgorithm::Horizontal;
 if (ctx->need_remove_expired_values)
 {
 /// `TTLTransform` stops reading after the first block when the rows TTL has expired for the whole part,
 /// while a vertical merge reads every key row to write `rows_sources`.
 if (global_ctx->future_part->merge_type == MergeType::TTLDrop && global_ctx->metadata_snapshot->hasRowsTTL())
 return MergeAlgorithm::Horizontal;
 if (!canVerticalTTLDelete(*global_ctx))
 return MergeAlgorithm::Horizontal;
 }
 if (global_ctx->future_part->part_format.part_type != MergeTreeDataPartType::Wide)
 return MergeAlgorithm::Horizontal;
 if (global_ctx->cleanup)
 return MergeAlgorithm::Horizontal;

and then the quantitative thresholds:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3793
 bool is_supported_storage =
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Ordinary ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Collapsing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::Replacing ||
 global_ctx->merging_params.mode == MergeTreeData::MergingParams::VersionedCollapsing;

 bool enough_ordinary_cols = global_ctx->gathering_columns.size() >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_columns_to_activate];

 bool enough_total_rows = total_rows_count >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_rows_to_activate];

 bool enough_total_bytes = total_size_bytes_uncompressed >= (*merge_tree_settings)[MergeTreeSetting::vertical_merge_algorithm_min_bytes_to_activate];

 bool no_parts_overflow = global_ctx->future_part->parts.size() <= RowSourcePart::MAX_PARTS;

 auto merge_alg = (is_supported_storage && enough_total_rows && enough_total_bytes && enough_ordinary_cols && no_parts_overflow) ?
 MergeAlgorithm::Vertical : MergeAlgorithm::Horizontal;

 return merge_alg;
}

Read those conditions as a description of when the vertical algorithm is the right shape.

Wide parts only, because a Compact part stores all columns in one data file and you cannot read one column of it without reading the others. No deduplication, because deduplication compares whole rows and so needs all columns at once. Only Ordinary/Collapsing/Replacing/VersionedCollapsing merging modes, because Summing/Aggregating/Coalescing merge the values of equal keys and so have to merge the payload rather than gather it. At least 11 non-key columns (vertical_merge_algorithm_min_columns_to_activate) and at least 16·8192 rows (vertical_merge_algorithm_min_rows_to_activate) — below those, the extra machinery is not worth it. And at most 127 source parts, because RowSourcePart::MAX_PARTS is 0x7F and the one-byte encoding cannot express more.

The default thresholds live in MergeTreeSettings: enable_vertical_merge_algorithm = 1, vertical_merge_algorithm_min_rows_to_activate = 16 * 8192, vertical_merge_algorithm_min_bytes_to_activate = 0, and vertical_merge_algorithm_min_columns_to_activate = 11.

Note the framing of the setting description: “potentially at the cost of merge speed”. Vertical merging is a memory optimization, not a speed one. It trades IO locality (one column at a time, so each source file is opened and read once per column rather than once for all columns) for peak memory. For a table where one column dominates the size, that trade is usually right; for a table with many equally-sized columns, the re-reading can dominate.

The `Wide`-only restriction, and the escape hatch

The Wide restriction is the interesting one. A Compact part interleaves all columns of a row in a single .bin file, so “read only column X” is impossible — you would have to decode the whole file. The sources may still be Compact while the result is Wide, and that case is allowed by allow_vertical_merges_from_compact_to_wide_parts (default true):

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:3784
 if (!(*merge_tree_settings)[MergeTreeSetting::allow_vertical_merges_from_compact_to_wide_parts])
 {
 for (const auto & part : global_ctx->future_part->parts)
 {
 if (!isWidePart(part))
 return MergeAlgorithm::Horizontal;
 }
 }

A Compact source is read once per column during the vertical stage, so its file gets decoded N times. That is expensive but it is bounded — and it buys the memory reduction for wide tables whose legacy parts happen to be Compact.

Progress accounting

A merge is a long-running background job, and system.merges shows a progress bar. For a vertical merge, “progress” spans two axes: first the key columns are merged (the horizontal stage), then each payload column is gathered (the vertical stage). The horizontal stage is weighted by the size of the key columns relative to the whole part; the vertical stage is weighted by the size of each individual column:

 cpp
 
 
// src/Storages/MergeTree/ColumnSizeEstimator.h:9
/* Allow to compute more accurate progress statistics */
class ColumnSizeEstimator
{
 using ColumnToSize = std::map<String, UInt64>;
 ColumnToSize map;
public:

 /// Stores approximate size of columns in bytes
 /// Exact values are not required since it used for relative values estimation (progress).
 size_t sum_total = 0;
 size_t sum_index_columns = 0;
 size_t sum_ordinary_columns = 0;

 ColumnSizeEstimator(ColumnToSize && map_, const NamesAndTypesList & key_columns, const NamesAndTypesList & ordinary_columns)
 : map(std::move(map_))
 {
 for (const auto & [name, _] : key_columns)
 if (!map.contains(name)) map[name] = 0;
 for (const auto & [name, _] : ordinary_columns)
 if (!map.contains(name)) map[name] = 0;

 for (const auto & [name, _] : key_columns)
 sum_index_columns += map.at(name);

 for (const auto & [name, _] : ordinary_columns)
 sum_ordinary_columns += map.at(name);

 sum_total = std::max(static_cast<decltype(sum_index_columns)>(1), sum_index_columns + sum_ordinary_columns);
 }

 Float64 columnWeight(const String & column) const
 {
 return static_cast<Float64>(map.at(column)) / static_cast<Float64>(sum_total);
 }

 Float64 keyColumnsWeight() const
 {
 return static_cast<Float64>(sum_index_columns) / static_cast<Float64>(sum_total);
 }
};

The weights are proportional to bytes, not rows, so a merge that is 90% one giant String column reports 90% progress for most of its duration and then jumps. That’s the honest answer for a byte-oriented operation. The sizes come from the parts themselves (accumulateColumnSizes, a cheap per-part statistic), so the whole estimator costs one map construction per merge.

The horizontal stage’s contribution: the row count assertion

There is one more piece of rigor worth pointing out. After the horizontal stage finishes and the row-source file is closed, the code checks that the file contains exactly one byte per input row:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2055
 size_t rows_sources_count = ctx->rows_sources_temporary_file->finalizeWriting();

 /// In special case, when there is only one source part, and no rows were skipped, we may have
 /// skipped writing rows_sources file. Otherwise rows_sources_count must be equal to the total
 /// number of input rows.
 /// Note that only one byte index is written for each row, so number of rows is equals to the number of bytes written.
 if ((rows_sources_count > 0 || global_ctx->future_part->parts.size() > 1) && sum_input_rows_exact != rows_sources_count + input_rows_filtered)
 throw Exception(
 ErrorCodes::LOGICAL_ERROR,
 "Number of rows in source parts ({}) excluding filtered rows ({}) differs from number "
 "of bytes written to rows_sources file ({}). It is a bug.",
 sum_input_rows_exact, input_rows_filtered, rows_sources_count);

This is a cheap, total consistency check on the side channel. If the horizontal stage and the vertical stage ever disagree about which rows exist, the merge fails loudly rather than producing a part whose payload columns are misaligned with its key — which would be a silent data corruption.

Note the subtlety in the condition: with exactly one source part and no filtered rows, the file may legitimately be empty (a single-source merge does not need to record where rows came from), so the check is skipped in that one case.

Prefetching the next column

Because the vertical stage reads one column at a time, and each column read is a separate pipeline over the source parts, the obvious optimization is to start reading column N+1 while column N is still being written. The code does exactly that, and only for remote filesystems, where the latency of the first read of a column is the dominant cost:

 cpp
 
 
// src/Storages/MergeTree/MergeTask.cpp:2076
 bool all_parts_on_remote_disks = std::ranges::all_of(global_ctx->future_part->parts, [](const auto & part) { return part->isStoredOnRemoteDisk(); });
 ctx->use_prefetch = all_parts_on_remote_disks && storage_settings[MergeTreeSetting::vertical_merge_remote_filesystem_prefetch];

 if (ctx->use_prefetch && ctx->it_name_and_type != global_ctx->gathering_columns.end())
 ctx->prepared_pipeline = createPipelineForReadingOneColumn(ctx->it_name_and_type->name);

The “prefetching” is simply that building the pipeline — which opens the parts and issues the first read requests — starts the IO. It is not a separate prefetch thread; it is eager pipeline construction, one column ahead (prepareVerticalMergeForOneColumn builds the next column’s pipeline while the current one runs).

The whole flow

 text
 
 
 source parts (Wide)
 ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
 │ k, a, b, c │ │ k, a, b, c │ │ k, a, b, c │
 └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
 │ │ │
 └────────────┬────┴─────────────────┘
 │ HORIZONTAL STAGE
 │ k-way merge of the key columns
 │ emits: merged k + rows_sources (1 byte/row)
 ▼
 ┌──────────────────────────────────────────┐
 │ rows_sources: 00 00 01 01 02 00 01 ... │
 └──────────────────────────────────────────┘
 │
 ┌─────────────────┼──────────────────┐
 ▼ ▼ ▼
 gather col a gather col b gather col c VERTICAL STAGE
 read a from read b from read c from one column at a time
 each part, each part, each part,
 copy by copy by copy by
 rows_sources rows_sources rows_sources
 │ │ │
 ▼ ▼ ▼
 └────────────┬────┴──────────────────┘
 ▼
 result part (Wide): k, a, b, c

What the hack buys you, and what it costs

It buys you peak memory proportional to the widest column, not the sum of the columns. For a table with a 40 MB String column and fifty UInt64 columns, a horizontal merge holds the payload of all the columns at once; a vertical merge holds one column at a time. That is the difference between a merge that OOMs and one that does not — and it is what makes the “wide part” format practical at all.

It costs you IO locality. Each column is a separate pass over the source parts, so a Compact source file is decoded once per column — which is why allow_vertical_merges_from_compact_to_wide_parts defaults to true but carries a warning in its description.

It costs you a temporary file. One byte per row of scratch space: 100 MB for a 100-million-row merge. Trivial compared to the data being merged, but not zero.

It costs you the ability to merge every kind of table. Summing, Aggregating and Coalescing merges have to combine the values of equal keys, which is a merge, not a gather. They stay horizontal, permanently.

It is not automatically faster. The setting description says it plainly: “potentially at the cost of merge speed”. The vertical algorithm optimizes memory, and only optimizes IO when a whole column happens to come from a single source part and can be stolen by pointer.

Seeing it in action

You can see which algorithm a merge used from the profile events:

 sql
 
 
SELECT
 event,
 value
FROM system.events
WHERE event IN (
 'MergeVerticalStageExecuteMilliseconds',
 'MergeHorizontalStageTotalMilliseconds',
 'MergeVerticalStageTotalMilliseconds'
);

MergeVerticalStageExecuteMilliseconds is only non-zero for tables where the vertical algorithm actually ran. If it is zero on your wide table, check enable_vertical_merge_algorithm, the part format (Compact parts cannot be vertically merged), and the vertical_merge_algorithm_min_* thresholds.

And to see the tables that would benefit:

 sql
 
 
SELECT
 database,
 table,
 formatReadableSize(sum(bytes_on_disk)) AS size,
 sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC
LIMIT 10;

A table with hundreds of columns and a large String payload is exactly the shape the vertical merge was built for. A table with ten narrow columns is not, and the thresholds will keep it horizontal.

Wrapping up

The one thing to remember: the vertical merge is a memory optimization, not a speed one. It trades IO locality for peak memory, and it wins big exactly when one column dominates your table’s width.

The part people get wrong: the whole thing hinges on a one-byte-per-row side channel. If you ever see a merge fail with “It is a bug”, that assertion is the row-count check, and it means the two stages disagreed about which rows exist.

What to go do about it: count your columns. Under 11, or under 16·8192 rows, the vertical algorithm will never kick in — and that is deliberate, not a bug.

ClickHouse Marks Empty Hash Table Cells With Zeroed Memory

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse’s hash table has no occupancy bit, no tombstone, and no sentinel pointer. A cell is empty when its key is all zero bytes, and the buffer starts zeroed because the allocator uses calloc. That one decision ripples through the cell layout, the zero-key special case, deletion, and even why the hash function has to be good.

Every GROUP BY in ClickHouse is a hash table. The table has to hold millions of keys, be probed once per input row, and be resized while it is being filled — all while the query is competing for memory with everything else on the machine.

std::unordered_map is simply not competitive here: it allocates a node per element, keeps a pointer chain, and pays a pointer chase for every collision.

So ClickHouse ships its own hash table in src/Common/HashTable/HashTable.h: a flat array of cells, open addressing, linear probing. The probing is textbook. The interesting part is the bookkeeping around emptiness.

There is no “occupied” bit. No tombstone. No sentinel pointer. No version counter. A cell is empty when its key is all zero bytes, and the whole buffer starts out zeroed by the allocator.

That single decision shapes the entire file: the allocator, the cell layout, the way the zero key is handled, the way deletion works, and the reason the hash function has to be good.

The shape of the table

 cpp
 
 
// src/Common/HashTable/HashTable.h:518
template <typename Key, typename Cell, typename Hash, typename Grower, typename Allocator>
class HashTable : private boost::noncopyable,
 protected Hash,
 protected Allocator,
 protected Cell::State,
 public ZeroValueStorage<Cell::need_zero_value_storage, Cell> /// empty base optimization
{

The table is one contiguous buffer of Cells plus a Grower that knows the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:550
 size_t m_size = 0; /// Amount of elements
 Cell * buf; /// A piece of memory for all elements except the element with zero key.
 Grower grower;

Cell is a policy class. The simplest one stores nothing but a key:

 cpp
 
 
// src/Common/HashTable/HashTable.h:154
template <typename Key, typename Hash, typename TState = HashTableNoState>
struct HashTableCell
{
 using State = TState;

 using key_type = Key;
 using value_type = Key;
 using mapped_type = VoidMapped;

 Key key;

The map flavour stores a pair, and the variant that ClickHouse uses for aggregation additionally caches the hash so that a rehash does not have to recompute it:

 cpp
 
 
// src/Common/HashTable/HashMap.h:165
template <typename Key, typename TMapped, typename Hash, typename TState = HashTableNoState>
struct HashMapCellWithSavedHash : public HashMapCell<Key, TMapped, Hash, TState> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `saved_hash` is set by `setHash` immediately after placement construction on insert; on the hot aggregation/join path we must avoid the redundant store
{
 using Base = HashMapCell<Key, TMapped, Hash, TState>;

 size_t saved_hash;

Empty means zero

Here is the whole emptiness protocol:

 cpp
 
 
// src/Common/HashTable/HashTable.h:191
 /// Whether the key is zero. In the main buffer, cells with a zero key are considered empty.
 /// If zero keys can be inserted into the table, then the cell for the zero key is stored separately, not in the main buffer.
 /// Zero keys must be such that the zeroed-down piece of memory is a zero key.
 bool isZero(const State & state) const { return isZero(key, state); }
 static bool isZero(const Key & key, const State & /*state*/) { return ZeroTraits::check(key); }

 /// Set the key value to zero.
 void setZero() { ZeroTraits::set(key); }

and the generic part:

 cpp
 
 
// src/Common/HashTable/HashTable.h:89
namespace ZeroTraits
{

template <typename T>
bool check(const T x)
{
 return bitEquals(x, T{});
}

template <typename T>
void set(T & x) { x = T{}; }

}

bitEquals is bitwise, not ==, and that matters for floating point:

 cpp
 
 
// src/Common/HashTable/HashTable.h:77
template <typename T>
inline ALWAYS_INLINE bool bitEquals(T a, T b)
{
 if constexpr (is_floating_point<T>)
 /// Note that memcmp with constant size is a compiler builtin.
 return 0 == memcmp(&a, &b, sizeof(T)); /// NOLINT
 else
 return a == b;
}

The comment above it explains why bit equality is mandatory rather than merely nice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:71
/** Numbers are compared bitwise.
 * Complex types are compared by operator== as usual (this is important if there are gaps).
 *
 * This is needed if you use floats as keys. They are compared by bit equality.
 * Otherwise the invariants in hash table probing do not met when NaNs are present.
 */

NaN never equals itself, so a ==-based emptiness test could never recognise an empty float cell — and the probe loop would run right off the end of the buffer.

The zeroed buffer comes from the allocator:

 cpp
 
 
// src/Common/HashTable/HashTableAllocator.h:8
/**
 * We are going to use the entire memory we allocated when resizing a hash
 * table, so it makes sense to pre-fault the pages so that page faults don't
 * interrupt the resize loop. Set the allocator parameter accordingly.
 */
using HashTableAllocator = Allocator<true /* clear_memory */, true /* populate */>;

clear_memory becomes calloc:

 cpp
 
 
// src/Common/Allocator.cpp:79
 if (alignment <= MALLOC_MIN_ALIGNMENT) [[likely]]
 {
 if constexpr (clear_memory)
 buf = __real_calloc(size, 1);
 else
 buf = __real_malloc(size);

So the invariant is: the bytes of a fresh buffer are zero, therefore a fresh cell is empty. Nothing more is needed to track occupancy. The diagram of the buffer is therefore very simple:

 text
 
 
 buf buf + bufSize()
 | |
 v v
 +--------+--------+--------+--------+--------+--------+--------+--------+
 | cell 0 | cell 1 | cell 2 | cell 3 | cell 4 | cell 5 | cell 6 | cell 7 |
 +--------+--------+--------+--------+--------+--------+--------+--------+
 key=A (zero) key=C (zero) key=B (zero) (zero)
 [used] [empty] [used] [empty] [used] [empty] [empty]

Note what is not there: no occupancy byte, no pointer, no version. A HashMapCell<UInt64, AggregateState*> is 16 bytes of pure payload. The empty/used distinction is free.

The zero key gets its own slot

There is one hole in the “zero means empty” scheme: 0 is a perfectly legal key. If it were stored in the buffer, it would be indistinguishable from an empty cell. So it is stored outside the buffer, in a ZeroValueStorage that the table inherits from:

 cpp
 
 
// src/Common/HashTable/HashTable.h:474
template <typename Cell>
struct ZeroValueStorage<true, Cell> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
private:
 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

 void setHasZero()
 {
 has_zero = true;
 new (zeroValue()) Cell();
 }

and the insert path routes the zero key to it:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1354
 bool ALWAYS_INLINE emplaceIfZero(const Key & x, LookupResult & it, bool & inserted, size_t hash_value)
 {
 /// If it is claimed that the zero key can not be inserted into the table.
 if constexpr (!Cell::need_zero_value_storage)
 return false;

 if (unlikely(Cell::isZero(x, *this)))
 {
 it = this->zeroValue();

 if (!this->hasZero())
 {
 ++m_size;
 this->setHasZero();
 this->zeroValue()->setHash(hash_value);
 inserted = true;
 }
 else
 inserted = false;

 return true;
 }

 return false;
 }

The zero key lives at a fixed address, so a lookup for it is a branch and a pointer, never a probe. The cost of the scheme is one bool plus one Cell of storage per table, and a slightly awkward iteration order (the zero cell is visited first, then the buffer).

And this is not an exotic corner. 0 is a perfectly ordinary grouping key:

 sql
 
SELECT k, count() FROM t GROUP BY k;

with a k column full of zeros produces a group whose key is the zero key, and that group is the one stored in the separate slot.

Note also that the design is conditional: a cell type can declare that zero keys are impossible (need_zero_value_storage = false), and then ZeroValueStorage<false, Cell> degenerates to nothing at all. The string tables do exactly that — see the article on the string-specialised map.

Probing

The probe sequence is the simplest possible one: linear probing with a mask.

 cpp
 
 
// src/Common/HashTable/HashTable.h:232
 /// The size of the hash table in the cells.
 size_t bufSize() const { return 1ULL << size_degree; }

 size_t maxFill() const { return 1ULL << (size_degree - 1); }
 size_t mask() const { return bufSize() - 1; }

 /// From the hash value, get the cell number in the hash table.
 size_t place(size_t x) const { return x & mask(); }

 /// The next cell in the collision resolution chain.
 size_t next(size_t pos) const { ++pos; return pos & mask(); }

 /// Whether the hash table is sufficiently full. You need to increase the size of the hash table, or remove something unnecessary from it.
 bool overflow(size_t elems) const { return elems > maxFill(); }

The load factor is 0.5: maxFill() is half the buffer. And the probe loop itself:

 cpp
 
 
// src/Common/HashTable/HashTable.h:559
 size_t ALWAYS_INLINE findCell(const Key & x, size_t hash_value, size_t place_value) const
 {
 while (!buf[place_value].isZero(*this) && !buf[place_value].keyEquals(x, hash_value, *this))
 {
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }

 return place_value;
 }

Two things to note. First, isZero is checked before keyEquals: the loop stops at the first empty cell, because with linear probing an empty cell terminates the chain. Second, the comparison is a bitwise equality against the key, and for the HashMapCellWithSavedHash variant it is short-circuited by the cached hash.

The zero key is handled before the probe:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1617
 LookupResult ALWAYS_INLINE find(const Key & x)
 {
 if (Cell::isZero(x, *this))
 return this->hasZero() ? this->zeroValue() : nullptr;

 return findNonZeroCell(x, hash(x));
 }

Insertion

emplace computes the hash, routes the zero key, and otherwise finds a place:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1380
 template <typename KeyHolder>
 void ALWAYS_INLINE emplaceNonZeroImpl(size_t place_value, KeyHolder && key_holder,
 LookupResult & it, bool & inserted, size_t hash_value)
 {
 it = &buf[place_value];

 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);
 const auto & key = keyHolderGetKey(key_holder);

 new (&buf[place_value]) Cell(key, *this);
 buf[place_value].setHash(hash_value);
 inserted = true;
 ++m_size;

 if (unlikely(grower.overflow(m_size)))
 {
 try
 {
 resize();
 }
 catch (...)
 {
 /** If we have not resized successfully, then there will be problems.
 * There remains a key, but uninitialized mapped-value,
 * which, perhaps, can not even be called a destructor.
 */
 --m_size;
 buf[place_value].setZero();
 inserted = false;
 keyHolderDiscardKey(key_holder);
 throw;
 }

 // The hash table was rehashed, so we have to re-find the key.
 it = findNonZeroCell(key, hash_value);
 chassert(it);
 }
 }

Two details worth pausing on. The cell is placement-newed; the buffer is raw memory, and the destructor of a non-trivial cell is only run for cells that were actually constructed (destroyElements walks the table and calls the destructor only for non-zero cells).

And the resize happens after the insert, not before. That means the table can exceed its nominal capacity by one element, and it means the iterator has to be re-found afterwards, because the rehash moved the cell. The exception handler is careful to undo the insert (decrement the size, zero the cell) so that a failed allocation does not leave a half-built cell behind.

Growing: the wrap-around special case

The resize path is where the linear-probing invariants get hairy. When the buffer grows, every element has to be re-placed, and an element that was at the end of the old buffer may belong at the beginning of the new one because its collision chain wrapped around:

 cpp
 
 
// src/Common/HashTable/HashTable.h:837
 void resize(size_t for_num_elems = 0, size_t for_buf_size = 0)
 {
#ifdef DBMS_HASH_MAP_DEBUG_RESIZES
 Stopwatch watch;
#endif

 size_t old_size = grower.bufSize();

 /** In case of exception for the object to remain in the correct state,
 * changing the variable `grower` (which determines the buffer size of the hash table)
 * is postponed for a moment after a real buffer change.
 * The temporary variable `new_grower` is used to determine the new size.
 */
 Grower new_grower = grower;

and the re-placement loop:

 cpp
 
 
// src/Common/HashTable/HashTable.h:884
 size_t i = 0;
 for (; i < old_size; ++i)
 if (!buf[i].isZero(*this))
 reinsert(buf[i], buf[i].getHash(*this));

 /** There is also a special case:
 * if the element was to be at the end of the old buffer, [ x]
 * but is at the beginning because of the collision resolution chain, [o x]
 * then after resizing, it will first be out of place again, [ xo ]
 * and in order to transfer it where necessary,
 * after transferring all the elements from the old halves you need to [ o x ]
 * process tail from the collision resolution chain immediately after it [ o x ]
 */
 size_t new_size = grower.bufSize();
 for (; i < new_size && !buf[i].isZero(*this); ++i)
 reinsert(buf[i], buf[i].getHash(*this));

The picture in that comment is the whole problem: after the buffer is doubled, the element at the very end of the old buffer (x) is now in the middle of the new one, and the element that was pushed behind it (o) belongs before it. The second loop therefore keeps going past the end of the old buffer, as long as the cells it finds are non-empty, re-inserting them.

The re-insertion itself is a memcpy of a trivially-copyable cell into the new position:

 cpp
 
 
// src/Common/HashTable/HashTable.h:918
 void reinsert(Cell & x, size_t hash_value)
 {
 size_t place_value = grower.place(hash_value);

 /// If the element is in its place.
 if (&x == &buf[place_value])
 return;

 /// Compute a new location, taking into account the collision resolution chain.
 place_value = findCell(Cell::getKey(x.getValue()), hash_value, place_value);

 /// If the item remains in its place in the old collision resolution chain.
 if (!buf[place_value].isZero(*this))
 return;

 /// Copy to a new location and zero the old one.
 x.setHash(hash_value);
 memcpy(static_cast<void*>(&buf[place_value]), &x, sizeof(x)); /// NOLINT(bugprone-undefined-memory-manipulation)
 x.setZero();
 }

The grower doubles the size until max_size_degree (23) and then grows by one step at a time:

 cpp
 
 
// src/Common/HashTable/HashTable.h:247
 /// Increase the size of the hash table.
 void increaseSize()
 {
 size_degree += size_degree >= max_size_degree ? 1 : 2;
 }

Robin Hood: what changed recently

The most interesting recent addition to the file is a second probing discipline. The classic HashTableGrower keeps the load factor at 0.5; the new HashTableGrowerWithPrecalculationAndRobinHood keeps 0.5 until the table reaches 2^16 cells, and then switches to Robin Hood hashing with a 90% load factor — in place, without reallocating:

 cpp
 
 
// src/Common/HashTable/HashTable.h:336
/** Like `HashTableGrowerWithPrecalculation`, but adapts to large hash tables.
 *
 * For large hash tables, random memory access is the main bottleneck, not the computations.
 * When the buffer reaches 2^robin_hood_size_degree cells, the grower switches the hash table to the Robin Hood mode:
 * the max load factor is increased from 0.5 to robin_hood_max_load_factor_percent / 100 (the hash table becomes up to two times smaller),
 * and the elements inside collision resolution chains are ordered by their optimal positions,
 * so a lookup of a missing key can stop early instead of walking to the end of the chain.
 *
 * The memory layout is the same in both modes, and switching happens in place, without growing the buffer.
 * Hash tables check the mode only if the grower has the `isRobinHood` method (see `GrowerWithRobinHood`),
 * so there is no cost for other growers.
 *
 * Requirements to the cell: obtaining the hash value with `getHash` must be cheap (the hash is saved or cheap to compute),
 * and the hash table must not keep pointers to cells while inserting other elements,
 * because an insertion may move other elements along the collision resolution chain.
 */

The payoff is twofold. The table becomes up to twice as small (0.9 vs 0.5 load factor), which halves the number of cache misses in a random probe; and a failed lookup can stop early, because the chain is sorted by “distance from the ideal position”:

 cpp
 
 
// src/Common/HashTable/HashTable.h:607
 size_t ALWAYS_INLINE findCellRobinHood(const Key & x, size_t hash_value, size_t place_value, bool & found) const
 {
 for (size_t distance = 0;; ++distance)
 {
 const Cell & cell = buf[place_value];
 if (cell.isZero(*this))
 {
 found = false;
 return place_value;
 }
 if (cell.keyEquals(x, hash_value, *this))
 {
 found = true;
 return place_value;
 }
 /// The element at the optimal position of the key cannot be closer to its own optimal position, so do not compute its hash.
 if (distance && distanceFromOptimalPosition(cell, place_value) < distance)
 {
 found = false;
 return place_value;
 }
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }
 }

“Distance from the optimal position” is just the difference between where the key would go and where it is, measured modulo the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:597
 size_t ALWAYS_INLINE distanceFromOptimalPosition(const Cell & cell, size_t place_value) const
 {
 return grower.place(place_value - grower.place(cell.getHash(*this)));
 }

Robin Hood ordering also changes insertion: the new key may have to be placed before an existing one, which means shifting the rest of the chain to the right:

 cpp
 
 
// src/Common/HashTable/HashTable.h:664
 void NO_INLINE shiftCellsRight(size_t place_value)
 {
 size_t empty_place = findEmptyCell(place_value);

 if (empty_place < place_value)
 {
 /// The chain wraps around the end of the buffer.
 size_t last_place = grower.bufSize() - 1;
 memmove(static_cast<void *>(&buf[1]), static_cast<const void *>(&buf[0]), empty_place * sizeof(Cell));
 memcpy(static_cast<void *>(&buf[0]), static_cast<const void *>(&buf[last_place]), sizeof(Cell));
 empty_place = last_place;
 }

 memmove(static_cast<void *>(&buf[place_value + 1]), static_cast<const void *>(&buf[place_value]), (empty_place - place_value) * sizeof(Cell));
 buf[place_value].setZero();
 }

and the chains have to be re-sorted when the mode is switched on — by an insertion sort, since the chains are “already almost ordered” in practice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:681
 /** Reorder the elements in every collision resolution chain by their optimal positions, as required in the Robin Hood mode.
 * Used after the elements were placed by the ordinary linear probing: on resize, or when switching to the Robin Hood mode.
 * It is an insertion sort, which is fast for the chains that are already almost ordered.
 */

Deletion without tombstones

Deleting from an open-addressing table usually means writing a tombstone, so that the probe chain is not cut short. Tombstones accumulate, they need to be cleaned up, and they slow down every subsequent lookup.

ClickHouse does not use tombstones. Instead it moves the elements of the chain:

 cpp
 
 
// src/Common/HashTable/HashTable.h:731
 /// Remove the element at place_value without calling its destructor, moving the following elements of the collision resolution chain to keep them reachable.
 /// Not inlined to keep the code of inserts small.
 void eraseCellAt(size_t erased_key_position)
 {
 /// We need to guarantee loop termination because there will be empty position
 chassert(m_size < grower.bufSize());

 size_t next_position = erased_key_position;

 /**
 * During element deletion there is a possibility that the search will be broken for one
 * of the following elements, because this place erased_key_position is empty. We will check
 * next_element. Consider a sequence from (erased_key_position, next_element], if the
 * optimal_position of next_element falls into it, then removing erased_key_position
 * will not break search for next_element.
 * If optimal_position of the element does not fall into the sequence (erased_key_position, next_element]
 * then deleting a erased_key_position will break search for it, so we need to move next_element
 * to erased_key_position. Now we have empty place at next_element, so we apply the identical
 * procedure for it.
 * If an empty element is encountered then means that there is no more next elements for which we can
 * break the search so we can exit.
 *
 * In the Robin Hood mode, the elements of a chain are ordered by their optimal positions,
 * and this procedure moves them one cell to the left, keeping the order.
 */

The rule the comment spells out is the classic one: an element may stay where it is if its ideal position is not before the hole; otherwise it has to slide left into the hole, which moves the hole one step further right, and the procedure repeats:

 cpp
 
 
// src/Common/HashTable/HashTable.h:798
 /// Move the element to the freed place
 memcpy(static_cast<void *>(&buf[erased_key_position]), static_cast<void *>(&buf[next_position]), sizeof(Cell));

 /// Now we have another freed place
 erased_key_position = next_position;
 }

 buf[erased_key_position].setZero();
 --m_size;
 }

This costs a memcpy per displaced element but keeps the table free of garbage: the buffer is always exactly as full as m_size says, and a probe chain never walks over a tombstone.

The allocator: zeroed and pre-faulted

HashTableAllocator is Allocator<true /* clear_memory */, true /* populate */>. The second flag is the interesting one: it pre-faults the pages of a large allocation with MADV_POPULATE_WRITE, so that the resize loop does not stop for page faults:

 cpp
 
 
// src/Common/Allocator.cpp:59
void prefaultPages([[maybe_unused]] void * buf_, [[maybe_unused]] size_t len_)
{
#if defined(MADV_POPULATE_WRITE)
 if (len_ < POPULATE_THRESHOLD)
 return;

 if (!is_supported_by_kernel) [[unlikely]]
 return;

 auto [buf, len] = adjustToPageSize(buf_, len_, ::getPageSize());
 ::madvise(buf, len, MADV_POPULATE_WRITE);
#endif
}

 cpp
 
 
// src/Common/Allocator.cpp:132
/// Constant is chosen almost arbitrarily, what I observed is 128KB is too small, 1MB is almost indistinguishable from 64MB and 1GB is too large.
extern const size_t POPULATE_THRESHOLD = std::max(Int64{16 * 1024 * 1024}, ::getPageSize());

The rationale is spelled out in HashTableAllocator.h itself: the whole allocation will be used, so pre-faulting it costs nothing extra and removes a source of latency spikes from the middle of a rehash. The kernel support check is a version test (Linux 5.14+), not an EINVAL probe, because madvise returns EINVAL for many unrelated reasons.

And note how calloc and the “zero means empty” invariant fit together perfectly: calloc of a large block is free on Linux for big allocations (the kernel hands out zero pages), so the “clear the buffer” requirement that the emptiness protocol imposes costs nothing at all.

Prefetching

A hash table probe is a random memory access; at 90% load factor with Robin Hood ordering, a lookup of a missing key can touch several cache lines. ClickHouse tries to hide that latency by prefetching the cell a few iterations ahead. The helper measures its own iteration latency to decide how far ahead to look:

 cpp
 
 
// src/Common/HashTable/Prefetching.h:33
class PrefetchingHelper
{
public:
 size_t calcPrefetchLookAhead()
 {
 static constexpr auto assumed_load_latency_ns = 100;
 static constexpr auto just_coefficient = 4;
 static constexpr auto numerator = just_coefficient * assumed_load_latency_ns * iterations_to_measure;
 size_t elapsed_ns = watch.elapsedNanoseconds();
 if (unlikely(elapsed_ns == 0))
 return max_look_ahead_value;
 size_t look_ahead = (numerator + elapsed_ns - 1) / elapsed_ns;
 return std::clamp<size_t>(look_ahead, min_look_ahead_value, max_look_ahead_value);
 }

 static constexpr size_t getInitialLookAheadValue() { return min_look_ahead_value; }

 static constexpr size_t iterationsToMeasure() { return iterations_to_measure; }

private:
 static constexpr size_t iterations_to_measure = 100;
 static constexpr size_t min_look_ahead_value = 4;
 static constexpr size_t max_look_ahead_value = 32;

The look-ahead is thus between 4 and 32 cells, tuned to the machine: the formula is 4 * 100ns * 100 / elapsed, i.e. four times the assumed memory latency, expressed in iterations.

The prefetching iterator of the table uses it, and also prefetches the key bytes of the cell it is about to look at, not just the cell:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1021
 void prefetch()
 {
 static_assert(CouldPrefetchKey<cell_type>);
 auto * buf_end = container->buf + container->grower.bufSize();
 iter_count++;
 if (prefetch_ptr < buf_end && prefetched_count < prefetch_ahead) [[likely]]
 {
 if (iter_count == DB::PrefetchingHelper::iterationsToMeasure()) [[unlikely]]
 prefetch_ahead = prefetching.calcPrefetchLookAhead();
 auto n = prefetch_ahead - prefetched_count;
 cell_type * last_ptr = nullptr;
 for (size_t i = 0; i < n; ++i)
 {
 while (prefetch_ptr < buf_end && prefetch_ptr->isZero(*container))
 ++prefetch_ptr;

 if (prefetch_ptr < buf_end) [[likely]]
 {
 last_ptr = prefetch_ptr;
 prefetch_ptr++;
 prefetched_count++;
 }
 else
 break;
 }

 if (last_ptr) [[likely]]
 keyPrefetch(last_ptr->getKey());
 }
 }

keyPrefetch touches up to four cache lines of the key data:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:182
inline void keyPrefetch(const std::string_view key)
{
 const size_t cache_line_size = 64;
 // Prefetch up to 4 cache lines.
 const size_t max_prefetch = cache_line_size * 4;
 const size_t n = std::min(key.size(), max_prefetch);
 const char * ptr = key.data();
 const char * end = ptr + n;
 for (; ptr < end; ptr += cache_line_size)
 {
 __builtin_prefetch(ptr);
 }
}

Keys that live outside the table

Not every key can live inside the cell. A std::string_view key points at data owned by somebody else — often a column that will be gone by the time the aggregation is finalised. ClickHouse solves this with a key holder: a small policy object that knows how to make a temporary key persistent, and when to throw it away.

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:53
/**
 * Returns the key. Can return the temporary key initially.
 * After the call to keyHolderPersistKey(), must return the persistent key.
 */
template <typename Key>
inline Key & ALWAYS_INLINE keyHolderGetKey(Key && key) { return key; } /// NOLINT(bugprone-return-const-ref-from-parameter)

/**
 * Make the key persistent. keyHolderGetKey() must return the persistent key
 * after this call.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderPersistKey(Key &&) {}

/**
 * Discard the key. Calling keyHolderGetKey() is ill-defined after this.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderDiscardKey(Key &&) {}

The default is a no-op: for a key stored by value, there is nothing to do, and the branch is optimised away. The ArenaKeyHolder version copies the bytes into an Arena when the key turns out to be new:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:98
inline void ALWAYS_INLINE keyHolderPersistKey(DB::ArenaKeyHolder & holder)
{
 // Normally, our hash table shouldn't ask to persist a zero key,
 // but it can happened in the case of clearable hash table (ClearableHashSet, for example).
 // The clearable hash table doesn't use zero storage and
 // distinguishes empty keys by using cell version, not the value itself.
 // So, when an empty std::string_view is inserted in ClearableHashSet we'll get here key of zero size.
 holder.key = std::string_view{holder.pool.insert(holder.key.data(), holder.key.size()), holder.key.size()};
}

The elegance is in where the decision is made. The hash table knows whether the key it just found was new; the caller does not. So the table calls keyHolderDiscardKey when the key already existed (nothing to persist, and the arena allocation is rolled back) and keyHolderPersistKey when it did not:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1386
 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);

The hash function has to be good

Because the table uses linear probing and the low bits of the hash to pick the bucket, a poor hash function is catastrophic: the low bits of a timestamp-like key are nearly constant, and every key lands in the same few cells. The header says so bluntly:

 cpp
 
 
// src/Common/HashTable/Hash.h:15
/** Hash functions that are better than the trivial function std::hash.
 *
 * Example: when we do aggregation by the visitor ID, the performance increase is more than 5 times.
 * This is because of following reasons:
 * - in Metrica web analytics system, visitor identifier is an integer that has timestamp with seconds resolution in lower bits;
 * - in typical implementation of standard library, hash function for integers is trivial and just use lower bits;
 * - traffic is non-uniformly distributed across a day;
 * - we are using open-addressing linear probing hash tables that are most critical to hash function quality,
 * and trivial hash function gives disastrous results.
 */

The default integer hash is the MurmurHash finalizer:

 cpp
 
 
// src/Common/HashTable/Hash.h:26
/** Taken from MurmurHash. This is Murmur finalizer.
 * Faster than intHash32 when inserting into the hash table UInt64 -> UInt64, where the key is the visitor ID.
 */
inline UInt64 intHash64(UInt64 x)
{
 x ^= x >> 33;
 x *= 0xff51afd7ed558ccdULL;
 x ^= x >> 33;
 x *= 0xc4ceb9fe1a85ec53ULL;
 x ^= x >> 33;

 return x;
}

Five instructions and two multiplies turn “the low bits are constant” into “the low bits are random”. For string keys and general keys, the CRC32 hardware instruction is used where it exists — three cycles of latency and one per cycle of throughput, which is why the string tables use it even though CRC32 is a poor hash in the abstract:

 cpp
 
 
// src/Common/HashTable/Hash.h:40
/** CRC32C is not very high-quality as a hash function,
 * according to avalanche and bit independence tests (see SMHasher software), as well as a small number of bits,
 * but can behave well when used in hash tables,
 * due to high speed (latency 3 + 1 clock cycle, throughput 1 clock cycle).
 * Works only with SSE 4.2 support.
 */

Trade-offs

What the “zero means empty” scheme buys:

Zero bytes of metadata per cell. A HashMapCell<UInt64, T*> is 16 bytes, all of it payload. A scheme with an occupancy bit would need to steal a bit from the key or add a byte.
A free calloc. The kernel can hand out pre-zeroed pages; the emptiness protocol costs nothing at allocation time.
No tombstones, ever. Deletion moves cells instead of marking them, so the table never degrades.
A cheap scan. for (auto & cell : buf) if (!cell.isZero()) ... is the iteration order; no indirection, no linked list.

What it costs:

The key must be zero-comparable, and bit-comparable. Floating point keys are compared with memcmp, and NaN semantics have to be papered over.
The zero key is special. It lives in a separate slot, it is visited first during iteration, and its handling adds a branch to every insert and lookup.
The hash must be excellent, because linear probing amplifies clustering, and the low bits of the hash are used directly.
Rehashing is a memcpy storm. Growing the table means moving every cell, and the wrap-around case needs a second pass over the head of the new buffer.

And the recent Robin Hood addition changes the balance again: it trades a little CPU (reordering chains, occasionally shifting cells on insert) for a halving of the memory footprint at large sizes — which, for a table that is probed randomly, is exactly the right trade, because the cost is cache misses, not instructions.

Wrapping up

The one thing to remember: “empty means all-zero bytes” buys you a cell layout that is 100% payload, with no occupancy bit to steal from your key.

The part people get wrong: this is not free. The zero key becomes a special case, float keys need bitwise comparison, and the hash function has to be genuinely good — a five-line Murmur finalizer stands between you and a 5× slower aggregation.

What to go do about it: if you ever profile a GROUP BY and see the cost dominated by cache misses on a huge hash table, remember that the Robin Hood grower exists precisely for that case: it halves the footprint at large sizes.

ClickHouse JIT-Compiles Scalar Expressions with LLVM

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse compiles hot scalar expressions to native code with LLVM. The path is: ActionsDAG → a three-node CompileDAG → hand-built IR → LLVM’s O3 pipeline → an RX page → a tight loop over raw column pointers. It is on by default since 25.5.

Every columnar database pays the same tax: a SELECT is not a loop over rows, it is a graph of actions, and every action dispatches through a virtual call per block.

ClickHouse’s block is 65 409 rows by default, so that dispatch overhead is paid 65 409 times less often than in a row-at-a-time engine. But for a narrow expression like a + b > c, the arithmetic itself is a couple of instructions per row — and the interpreter’s per-node costs (a virtual dispatch, a column-type switch, a ColumnPtr copy) dominate.

ClickHouse’s answer is a JIT compiler that is on by default since 25.5:

 cpp
 
 
// src/Core/Settings.cpp:1459
 DECLARE(Bool, compile_expressions, true, R"(
Compile some scalar functions and operators to native code.
)", 0, \
 {"25.5", false, true, "We believe that the LLVM infrastructure behind the JIT compiler is stable enough to enable this setting by default."}) \

This article follows the whole path: from an ActionsDAG node, through a small intermediate representation (CompileDAG), into hand-built LLVM IR, through LLVM’s O3 pipeline, into an executable page, and finally into a tight loop that runs over raw column data.

The switch

The JIT is not free: compiling a module costs milliseconds. So ClickHouse only compiles an expression after it has been seen a few times. The threshold is a setting, default 3:

 cpp
 
 
// src/Core/Settings.cpp:1463
 DECLARE(UInt64, min_count_to_compile_expression, 3, R"(
Minimum count of executing same expression before it is get compiled.
)", 0) \

The counter is a global hash map keyed by a hash of the expression DAG, so the “same expression” test is structural, not textual:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:299
static FunctionBasePtr compile(
 const CompileDAG & dag,
 size_t min_count_to_compile_expression)
{
 static std::unordered_map<UInt128, UInt64, UInt128Hash> counter;
 static std::mutex mutex;

 auto hash_key = dag.hash();
 {
 std::lock_guard lock(mutex);
 if (counter[hash_key]++ < min_count_to_compile_expression)
 return nullptr;
 }

Note the ++ inside the comparison: the counter is incremented on every call, and the expression is compiled on the fourth sighting (with the default of 3). The counter never decays, so a hot expression stays hot forever.

The trigger point is the construction of ExpressionActions, which is where a DAG becomes an executable list of actions:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:74
#if USE_EMBEDDED_COMPILER
 if (settings.can_compile_expressions && settings.compile_expressions == CompileExpressions::yes)
 actions_dag.compileExpressions(settings.min_count_to_compile_expression, lazy_executed_nodes);
#endif

Note the ordering: short-circuit functions are analysed first and the set of “lazily executed” nodes is computed before compiling, because a node that is only evaluated under a condition must not be folded into a compiled straight-line expression.

CompileDAG: a tiny IR

ActionsDAG is the general-purpose graph ClickHouse uses for every expression: it carries column nodes, aliases, lambdas, arrays, and functions. It is too rich to translate to LLVM directly, so the JIT first flattens it into CompileDAG, which has exactly three kinds of node:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.h:36
 enum class CompileType : uint8_t
 {
 INPUT = 0,
 CONSTANT = 1,
 FUNCTION = 2,
 };

The flattening is a post-order DFS over the ActionsDAG that stops at anything it cannot compile. getCompilableDAG in ExpressionJIT.cpp walks the DAG with an explicit stack and produces, for each candidate node, a CompileDAG plus the list of children that became inputs:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:404
static CompileDAG getCompilableDAG(
 const ActionsDAG::Node * root,
 ActionsDAG::NodeRawConstPtrs & children,
 const std::unordered_set<const ActionsDAG::Node *> & lazy_executed_nodes)
{
 /// Extract CompileDAG from root actions dag node.

 CompileDAG dag;

 std::unordered_map<const ActionsDAG::Node *, size_t> visited_node_to_compile_dag_position;

A node is a leaf of the compiled fragment when it is a constant, when it is not compilable at all, or when its function declares that the argument does not participate in compilation:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:430
 bool is_compilable_constant = isCompilableConstant(*node);
 bool is_compilable_function = isCompilableFunction(*node, lazy_executed_nodes);

 if (!is_compilable_function || is_compilable_constant || frame.skip_compile)
 {
 CompileDAG::Node compile_node;
 compile_node.function = node->function_base;
 compile_node.result_type = node->result_type;

 if (is_compilable_constant)
 {
 compile_node.type = CompileDAG::CompileType::CONSTANT;
 compile_node.column = node->column;
 }
 else if (frame.skip_compile)
 {
 compile_node.type = CompileDAG::CompileType::CONSTANT;
 compile_node.column = node->column;
 compile_node.skip_compile = true;
 }
 else
 {
 compile_node.type = CompileDAG::CompileType::INPUT;
 children.emplace_back(node);
 }

The skip_compile flag is how “arguments that don’t participate” get handled: the argument is still passed to the function at runtime (the C++ function signature has all its arguments), but its value is not computed by the compiled code. This is what makes if(cond, f(x), g(x))-style functions compile at all: the lazily evaluated branch is a runtime argument, not a compiled computation.

Choosing what to compile

ActionsDAG::compileFunctions is the part that decides which nodes become compiled fragments. It is not “compile everything”: it computes, for each node, how many compilable descendants it has, and only compiles nodes that have at least one compilable child and whose parents are not themselves compilable (so a whole subtree is compiled once, at its root, not piece by piece):

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:594
 for (const auto & node : nodes)
 {
 auto & node_data = node_to_data[&node];
 bool node_is_valid_for_compilation = node_data.is_compilable_in_isolation && node_data.compilable_children_size > 0;

 for (const auto & child : node.children)
 node_to_data[child].all_parents_compilable &= node_is_valid_for_compilation;
 }

 for (const auto & output_node : outputs)
 {
 /// Force output nodes to compile
 node_to_data[output_node].all_parents_compilable = false;
 }

Then the fragments are sorted so that a parent is compiled before its children — otherwise the same subexpression would be compiled twice under two different names:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:626
 /** Sort nodes before compilation using their children size to avoid compiling subexpression before compile parent expression.
 * This is needed to avoid compiling expression more than once with different names because of compilation order.
 */
 ::sort(nodes_to_compile.begin(), nodes_to_compile.end(), [&](const Node * lhs, const Node * rhs)
 {
 return node_to_data[lhs].children_size > node_to_data[rhs].children_size;
 });

and the compiled node replaces the original in place:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:649
 node->type = ActionsDAG::ActionType::FUNCTION;
 node->function_base = fn;
 node->function = fn->prepare(arguments);
 node->children.swap(new_children);
 node->is_function_compiled = true;
 node->column = nullptr;

From CompileDAG to LLVM IR

CompileDAG::compile emits the IR for the fragment. For a function node it recursively materialises the argument values and asks the function itself to emit its own IR:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.cpp:45
 case CompileType::FUNCTION:
 {
 ValuesWithType temporary_values;
 temporary_values.reserve(node.arguments.size());
 for (auto argument_index : node.arguments)
 temporary_values.emplace_back(compiled_values[argument_index]);

 ValueWithType compiled_value{node.function->compile(builder, temporary_values), node.function->getResultType()};
 if (!node.result_type->equals(*node.function->getResultType()))
 compiled_values[compiled_values_index] = {nativeCast(b, compiled_value, node.result_type), node.result_type};
 else
 compiled_values[compiled_values_index] = std::move(compiled_value);

 break;
 }

Constants are baked into the module as native values, inputs are the loop’s loaded values, and functions call IFunction::compile.

The function contract

Each SQL function that wants to be JIT-compilable implements compileImpl, which receives the already-typed argument values as LLVM values and returns an LLVM value. A binary arithmetic function looks like this:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3821
 llvm::Value * compileImpl(llvm::IRBuilderBase & builder, const ValuesWithType & arguments, const DataTypePtr & result_type) const override
 {
 chassert(2 == arguments.size());

 auto denull_left_type = removeNullable(arguments[0].type);
 auto denull_right_type = removeNullable(arguments[1].type);
 llvm::Value * result = nullptr;

and ends with the actual operation, dispatched on the native type:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3846
 auto & b = static_cast<llvm::IRBuilder<> &>(builder);
 auto * lval = nativeCast(b, arguments[0], result_type);
 auto * rval = nativeCast(b, arguments[1], result_type);
 result = OpSpec::compile(b, lval, rval, is_signed_v<typename ResultDataType::FieldType>);
 return true;

NULLs: the ternary trick

Nullable(T) is represented in IR as a two-element struct { T value, UInt8 is_null }. Handling NULLs in compiled code is where most of the complexity lives, because ClickHouse’s default implementation for NULLs is “if any argument is NULL, the result is NULL” — which is not what and/or want (three-valued logic).

The framework’s answer is to compile the denulled function and then OR the null flags back in:

 cpp
 
 
// src/Functions/IFunction.cpp:1281
 auto denulled_arguments_types = removeNullables(arguments_types);
 if (useDefaultImplementationForNulls() && denulled_arguments_types)
 {
 auto & b = static_cast<llvm::IRBuilder<> &>(builder);

 ValuesWithType unwrapped_arguments;
 unwrapped_arguments.reserve(arguments.size());

 VectorWithMemoryTracking<llvm::Value *> is_null_values;

 auto skip_arguments = getArgumentsThatDontParticipateInCompilation(arguments_types);
 for (size_t i = 0; i < arguments.size(); ++i)
 {
 const auto & argument = arguments[i];
 llvm::Value * unwrapped_value = argument.value;

 bool skip_compile = (std::find(skip_arguments.begin(), skip_arguments.end(), i) != skip_arguments.end());
 if (argument.type->isNullable())
 {
 unwrapped_value = skip_compile ? nullptr : b.CreateExtractValue(argument.value, {0});
 is_null_values.emplace_back(skip_compile ? nullptr : b.CreateExtractValue(argument.value, {1}));
 }

 unwrapped_arguments.emplace_back(unwrapped_value, (*denulled_arguments_types)[i]);
 }

and then, at the end, the null flags are merged into the result:

 cpp
 
 
// src/Functions/IFunction.cpp:1317
 for (auto * is_null_value : is_null_values)
 {
 if (is_null_value)
 nullable_structure_result_null = b.CreateOr(nullable_structure_result_null, is_null_value);
 }

 return b.CreateInsertValue(nullable_structure_with_result_value, nullable_structure_result_null, {1});

Functions with special NULL semantics (the logical ones) opt out of this and do the work themselves, using an explicit three-valued encoding:

 cpp
 
 
// src/Functions/FunctionsLogical.h:267
 llvm::Value * compileImpl(llvm::IRBuilderBase & builder, const ValuesWithType & values, const DataTypePtr & result_type) const override
 {
 chassert(!values.empty());

 auto & b = static_cast<llvm::IRBuilder<> &>(builder);
 if constexpr (Impl::specialImplementationForNulls())
 {
 if (result_type->isNullable())
 {
 /// First we need to cast all values to ternary logic
 llvm::Value * ternary_result = nativeTernaryCast(b, values[0]);
 for (size_t i = 1; i < values.size(); ++i)
 {
 llvm::Value * casted_value = nativeTernaryCast(b, values[i]);
 ternary_result = Impl::ternaryApply(b, ternary_result, casted_value);
 }

 /// Then transform ternary logic to struct which represents nullable result
 llvm::Value * is_null = b.CreateICmpEQ(ternary_result, b.getInt8(Ternary::Null));
 llvm::Value * is_true = b.CreateICmpEQ(ternary_result, b.getInt8(Ternary::True));

This is a nice illustration of the general ClickHouse pattern: the interpreter path handles NULLs with a nested column representation and a mask, the compiled path handles them with a struct and a select — the semantics are the same, the representation is not.

The generated loop

The whole pipeline, from a plan node to a running loop:

 text
 
 
 ActionsDAG node CompileDAG (3 node kinds)
 (Function, Column, Alias, ...) INPUT column to read
 | CONSTANT baked-in literal
 | getCompilableDAG() FUNCTION nested call
 v
 CompileDAG --hash()--> SipHash-128 --min_count_to_compile_expression--> ?
 | (seen 4 times?)
 v compile()
 llvm::Module (IRBuilder: entry / loop / end blocks)
 |
 v O3 pipeline + RuntimeDyld
 executable page (PROT_READ|PROT_EXEC) <---- CompiledExpressionCache (weighted by code size)
 |
 v callJITFunction(rows, columns)
 for (i = 0; i < rows; ++i) { result[i] = expr(columns[0][i], columns[1][i]); }

The IR builder in compileFunction.cpp wraps the expression in a loop over the rows. The function’s ABI is void f(size_t rows, ColumnData * columns), where ColumnData is a { const char * data, const char * null_data, const char * offsets } triple — the raw pointers of the column, its null map, and (for strings) its offsets:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:88
static llvm::StructType * buildColumnDataStruct(llvm::IRBuilderBase & b)
{
 auto * char_ptr_type = b.getInt8Ty()->getPointerTo();
 return llvm::StructType::get(char_ptr_type, char_ptr_type, char_ptr_type);
}

The function itself is created with ExternalLinkage so the linker can find it by name later, and the loop is a textbook phi-node counter:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:139
 /// Initialize loop

 auto * end = llvm::BasicBlock::Create(b.getContext(), "end", func);
 auto * loop = llvm::BasicBlock::Create(b.getContext(), "loop", func);
 b.CreateCondBr(b.CreateICmpEQ(rows_count_arg, llvm::ConstantInt::get(size_type, 0)), end, loop);

 b.SetInsertPoint(loop);

 /// Loop

 auto * counter_phi = b.CreatePHI(rows_count_arg->getType(), 2);
 counter_phi->addIncoming(llvm::ConstantInt::get(size_type, 0), entry);

Each argument is loaded from the column data with an in-bounds GEP on the row index:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:162
 auto * column_data_ptr = column.data_ptr;
 auto * column_element_value = b.CreateLoad(column.data_element_type, b.CreateInBoundsGEP(column.data_element_type, column_data_ptr, counter_phi));

 if (!type->isNullable())
 {
 arguments.emplace_back(column_element_value, type);
 continue;
 }

 auto * column_is_null_element_value = b.CreateLoad(b.getInt8Ty(), b.CreateInBoundsGEP(b.getInt8Ty(), column.null_data_ptr, counter_phi));
 auto * is_null = b.CreateICmpNE(column_is_null_element_value, b.getInt8(0));
 auto * nullable_unitialized = llvm::Constant::getNullValue(toNullableType(b, column.data_element_type));
 auto * nullable_value = b.CreateInsertValue(b.CreateInsertValue(nullable_unitialized, column_element_value, {0}), is_null, {1});
 arguments.emplace_back(nullable_value, type);

and the result is stored back into the output column, which is passed as the last element of the columns array:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:184
 auto * result_column_element_ptr = b.CreateInBoundsGEP(columns.back().data_element_type, columns.back().data_ptr, counter_phi);

 if (columns.back().null_data_ptr)
 {
 b.CreateStore(b.CreateExtractValue(result, {0}), result_column_element_ptr);
 auto * result_column_is_null_element_ptr = b.CreateInBoundsGEP(b.getInt8Ty(), columns.back().null_data_ptr, counter_phi);
 auto * is_result_column_element_null = b.CreateSelect(b.CreateExtractValue(result, {1}), b.getInt8(1), b.getInt8(0));
 b.CreateStore(is_result_column_element_null, result_column_is_null_element_ptr);
 }
 else
 {
 b.CreateStore(result, result_column_element_ptr);
 }

Running it

At runtime, LLVMExecutableFunction::executeImpl allocates an output column, gathers the raw pointers, and calls into the compiled code:

 cpp
 
 
// src/Interpreters/ExpressionJIT.cpp:99
 ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & result_type, size_t input_rows_count) const override
 {
 if (!canBeNativeType(*result_type))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "LLVMExecutableFunction unexpected result type in: {}", result_type->getName());

 auto result_column = result_type->createUninitializedColumnWithSize(input_rows_count);

 if (input_rows_count)
 {
 std::vector<ColumnData> columns(arguments.size() + 1);
 Columns columns_backup;

 for (size_t i = 0; i < arguments.size(); ++i)
 {
 auto column = arguments[i].column->convertToFullColumnIfConst();
 columns_backup.emplace_back(column);
 columns[i] = getColumnData(column.get());
 }

 columns[arguments.size()] = getColumnData(result_column.get());

 auto jit_compiled_function = compiled_function_holder->compiled_function.compiled_function;
 callJITFunction(jit_compiled_function, input_rows_count, columns.data());

getColumnData is where the layout knowledge lives — a String column is not a contiguous array of values, it is a chars buffer plus an offsets buffer, and the JIT comparator indexes both directly:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:63
 /// skip null key data for one nullable key optimization
 if (WhichDataType(column->getDataType()).isString())
 {
 /// For String columns, use base pointers to chars and offsets arrays directly.
 /// The JIT comparator uses absolute row indices into these arrays, so both
 /// pointers must be to the start of their respective arrays regardless of skip_rows.
 const auto * string_column = assert_cast<const ColumnString *>(column);
 result.data = reinterpret_cast<const char *>(string_column->getChars().data());
 result.offset_data = reinterpret_cast<const char *>(string_column->getOffsets().data());
 }

For a String the “element type” is therefore a {chars*, offsets*, index} struct, not a char*:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:94
static llvm::StructType * buildStringRefType(llvm::IRBuilderBase &b)
{
 auto * data_ptr_type = b.getInt8Ty()->getPointerTo();
 auto * offset_ptr_type = b.getInt8Ty()->getPointerTo();
 auto * index_type = b.getInt64Ty();
 return llvm::StructType::get(data_ptr_type, offset_ptr_type, index_type);
}

The JIT itself

CHJIT is a small, self-contained LLVM JIT: it uses RuntimeDyld to link the object file into memory it manages itself.

The whole compilation is serialised by a mutex, because LLVM’s context is shared:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:480
CHJIT::CompiledModule CHJIT::compileModule(std::function<void (llvm::Module &)> compile_function)
{
 std::lock_guard lock(jit_lock);

The module is then run through a full O3 pipeline with SLP vectorisation enabled:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:620
void CHJIT::runOptimizationPassesOnModule(llvm::Module & module) const
{
 llvm::LoopAnalysisManager lam;
 llvm::FunctionAnalysisManager fam;
 llvm::CGSCCAnalysisManager cgam;
 llvm::ModuleAnalysisManager mam;

 llvm::PipelineTuningOptions pto;
 pto.SLPVectorization = true;

 /// Pass the actual TargetMachine so that all optimization passes
 /// (including target-specific ones) have proper target information.
 llvm::PassBuilder pb(machine.get(), pto);

 pb.registerModuleAnalyses(mam);
 pb.registerCGSCCAnalyses(cgam);
 pb.registerFunctionAnalyses(fam);
 pb.registerLoopAnalyses(lam);
 pb.crossRegisterProxies(lam, fam, cgam, mam);

 llvm::ModulePassManager mpm = pb.buildPerModuleDefaultPipeline(llvm::OptimizationLevel::O3);
 mpm.run(module, mam);
}

The target machine is the host’s, with the host CPU and features, so the compiled code uses whatever vector instructions the machine has:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:644
std::unique_ptr<llvm::TargetMachine> CHJIT::getTargetMachine()
{
 static std::once_flag llvm_target_initialized;
 std::call_once(llvm_target_initialized, []()
 {
 llvm::InitializeNativeTarget();
 llvm::InitializeNativeTargetAsmPrinter();
 llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
 });

 std::string error;
 auto cpu = llvm::sys::getHostCPUName();
 auto triple = llvm::Triple(llvm::sys::getProcessTriple());

Memory management

JIT code needs executable pages. ClickHouse does not use mmap directly; it allocates page-aligned blocks with posix_memalign and flips their protection with mprotect once the linking is done:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:287
 void allocateNextPageBlock(size_t size)
 {
 size_t pages_to_allocate_size = ((size / page_size) + 1) * 2;
 size_t allocate_size = page_size * pages_to_allocate_size;

 void * buf = nullptr;
 int res = posix_memalign(&buf, page_size, allocate_size);

The double allocation (+1 then *2 pages) gives both slack for the alignment request and room to grow without a new mprotect dance. Three separate arenas exist — read-write data, read-only data, and executable code — and the memory manager marks them appropriately at the end:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:353
 bool finalizeMemory(std::string *) override
 {
 ro_page_arena.protect(PROT_READ);
 ex_page_arena.protect(PROT_READ | PROT_EXEC);
 return true;
 }

On ARM the code has to invalidate the instruction cache after mprotect, and some ARM CPUs fault if the page is not readable while the cache is being flushed — hence the special case:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:211
# if defined(__arm__) || defined(__aarch64__)
 /// Certain ARM implementations treat icache clear instruction as a memory read,
 /// and CPU segfaults on trying to clear cache on !PROT_READ page.
 /// Therefore we need to temporarily add PROT_READ for the sake of flushing the instruction caches.
 if (invalidate_cache && !(protection_flags & PROT_READ))
 {
 int res = mprotect(block.base(), block.blockSize(), protection_flags | PROT_READ);

Symbol resolution

The compiled code may call back into ClickHouse (e.g. memcpy, or the 128-bit integer helpers for Int128 arithmetic). A JITSymbolResolver maps those names to addresses, and the trick of storing mangled names removes a platform-specific lookup path:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:397
 void registerSymbol(const std::string & symbol_name, void * symbol)
 {
 std::string mangled_name;
 llvm::raw_string_ostream mangled_name_stream(mangled_name);
 llvm::Mangler::getNameWithPrefix(mangled_name_stream, symbol_name, layout);
 mangled_name_stream.flush();
 symbol_name_to_symbol_address[mangled_name] = symbol;
 }

The 128-bit helpers come in signed/unsigned pairs, and the comment explains why both halves must be registered:

 cpp
 
 
// src/Interpreters/JIT/CHJIT.cpp:76
/// IMPORTANT: every libcall is registered as a signed/unsigned pair. LLVM picks the
/// signed or unsigned variant based on whether the IR uses `fptosi`/`sitofp`/`sdiv`/`srem`
/// or `fptoui`/`uitofp`/`udiv`/`urem`. Forgetting the unsigned half compiles fine but
/// fails the JIT link at run time with `Could not find symbol __fixunsdfti` (or similar).
/// Keep the lists below symmetric.

Caching the compiled code

Compiling the same expression twice is pure waste, so compiled modules are cached by the CompileDAG hash:

 cpp
 
 
// src/Interpreters/JIT/CompiledExpressionCache.h:37
class CompiledExpressionCache : public CacheBase<UInt128, CompiledExpressionCacheEntry, UInt128Hash, CompiledFunctionWeightFunction>
{
public:
 using Base = CacheBase<UInt128, CompiledExpressionCacheEntry, UInt128Hash, CompiledFunctionWeightFunction>;
 using Base::Base;
};

The cache is weight-aware: an entry’s weight is the number of bytes of machine code it occupies, so evicting a big module frees proportionally more memory.

The hash that keys the cache is a structural hash over the DAG:

 cpp
 
 
// src/Interpreters/JIT/CompileDAG.cpp:126
UInt128 CompileDAG::hash() const
{
 SipHash hash;
 for (const auto & node : nodes)
 {
 hash.update(node.type);

 const auto & result_type_name = node.result_type->getName();
 hash.update(result_type_name.size());
 hash.update(result_type_name);

 switch (node.type)
 {
 case CompileType::CONSTANT:
 {
 node.column->getDataColumn().updateHashWithValue(0, hash);
 break;
 }

so two syntactically different but semantically identical expressions hash the same, and the same expression with a different constant does not.

Beyond scalar expressions

The same machinery compiles two more things.

Aggregate functions. A whole aggregation is compiled as five entry points:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:544
CompiledAggregateFunctions compileAggregateFunctions(CHJIT & jit, const std::vector<AggregateFunctionWithOffset> & functions, std::string functions_dump_name)
{
 Stopwatch watch;

 std::string create_aggregate_states_functions_name = functions_dump_name + "_create";
 std::string add_aggregate_states_functions_name = functions_dump_name + "_add";
 std::string add_aggregate_states_functions_name_single_place = functions_dump_name + "_add_single_place";
 std::string merge_aggregate_states_functions_name = functions_dump_name + "_merge";
 std::string insert_aggregate_states_functions_name = functions_dump_name + "_insert";

_create builds the aggregate states, _add folds a row into them, _merge merges two states, _insert writes the final values into the result columns. The _add function is generated twice: once for the “many places” layout (a hash table of aggregate states) and once for the “single place” layout (one state per group):

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:556
 compileCreateAggregateStatesFunctions(module, functions, create_aggregate_states_functions_name);
 compileAddIntoAggregateStatesFunctions(module, functions, add_aggregate_states_functions_name, AddIntoAggregateStatesPlacesArgumentType::MultiplePlaces);
 compileAddIntoAggregateStatesFunctions(module, functions, add_aggregate_states_functions_name_single_place, AddIntoAggregateStatesPlacesArgumentType::SinglePlace);
 compileMergeAggregatesStates(module, functions, merge_aggregate_states_functions_name);
 compileInsertAggregatesIntoResultColumns(module, functions, insert_aggregate_states_functions_name);

Sort comparators. A multi-column sort key is compiled into a single comparator function that returns int8. The IR is a chain of basic blocks, one per sort column, joined by a phi node — a hand-written decision tree:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:742
 /** 1. If it is last condition block move to join block.
 * 2. If column elements are not equal move to join block.
 * 3. If column elements are equal move to next column condition.
 */
 if (i == columns_size - 1)
 b.CreateBr(comparator_join);
 else
 b.CreateCondBr(b.CreateICmpEQ(result, lhs_equals_rhs_result), comparator_steps_and_results[i + 1].first, comparator_join);

and the function’s return attribute is not a detail but a correctness requirement on Apple’s arm64 ABI:

 cpp
 
 
// src/Interpreters/JIT/compileFunction.cpp:608
 /// The result is read back through an `int8_t (*)(...)` function pointer, so the i8 return
 /// must obey the C `int8_t` calling convention. This matters only on Darwin/Apple arm64:
 /// Apple's arm64 ABI makes the *callee* sign-extend sub-word return values and the caller
 /// relies on it (it emits no `sxtb` after the call). Without `signext` the JIT callee leaves
 /// the upper bits unset, so e.g. a -1 result is read as 255 and NULL ordering breaks.
 /// Linux arm64 (AAPCS64) is unaffected because there the *caller* sign-extends the return
 /// itself (`sxtb w0, w0`); x86-64 is likewise a no-op.
 comparator_func->addRetAttr(llvm::Attribute::SExt);

Trade-offs

What the JIT buys: the per-block dispatch, the per-node type switches and the column representation overhead disappear for the compiled fragment; the loop is a plain memory-bound loop that the CPU can auto-vectorise. For a filter like WHERE user_id = 42 AND ts > now() - INTERVAL 1 HOUR, this is the difference between a few hundred nanoseconds per block and a few microseconds.

What it costs:

Memory: every compiled module occupies executable pages, and the cache is weight-limited by machine-code size. A server with thousands of distinct hot expressions holds a lot of RX pages.
Compile latency: the first four executions of an expression pay for the compile; the O3 pipeline is not cheap (hundreds of microseconds to milliseconds). The min_count_to_compile_expression threshold exists exactly to amortise this.
A second implementation of every function’s semantics: compileImpl and executeImpl must agree. Bugs where the JIT path and the interpreter path differ are a real class of ClickHouse bugs — the Nullable handling above is where most of the subtlety lives.

Restricted type coverage: only “native” types (fixed-width integers, floats, Date, DateTime, and String/FixedString through their special reference structs) can be compiled. Decimal is explicitly excluded from the arithmetic family:

 cpp
 
 
// src/Functions/FunctionBinaryArithmetic.h:3840
 if constexpr (
 !std::is_same_v<ResultDataType, InvalidType> && !IsDataTypeDecimal<ResultDataType>
 && !IsDataTypeDecimal<LeftDataType> && !IsDataTypeDecimal<RightDataType> && OpSpec::compilable)

Locking: CHJIT::compileModule holds a global mutex, so the compile of one expression blocks the compile of another. The compiled code is fully parallel — it is only the compilation that is serialised.

And the escape hatch is one setting:

 sql
 
SELECT count() FROM events WHERE user_id = 42 SETTINGS compile_expressions = 0;

Wrapping up

The one thing to remember: the JIT removes interpretation overhead, not arithmetic. If your expression is already dominated by memory bandwidth, the JIT will not save you. If it is dominated by per-node dispatch, it will save a lot.

The part people get wrong: the first four executions of an expression are interpreted. A query that runs once, or a low-traffic tenant, gets no benefit at all from the JIT — you pay the compile latency and never amortise it.

What to go do about it: compare compile_expressions = 0 against the default on your hottest filter queries and look at the wall clock. If the difference is small, the interpreter was never your bottleneck and the real fix is a different query shape.

ClickHouse Evaluates IN with Hash Sets

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: IN is not a comparison, it is a hash set. ClickHouse picks one of thirteen specialisations based on the shape of the right-hand side, packs integer keys into the hash table’s own key, and — if the set grows past half your memory — spills it to disk behind per-block Bloom filters.

x IN (1, 2, 3) looks like a comparison. x IN (SELECT user_id FROM logins WHERE …) looks like a join. Under the hood, both become the same thing: a hash set, built once, probed once per row of the left-hand side.

The interesting part isn’t “it is a hash table” — it’s which hash table. ClickHouse picks a different data structure for every shape of the right-hand side, and the set can get big enough that it no longer fits in memory and has to spill to disk behind Bloom filters.

Let’s walk it.

The set is a plan artifact, not a function argument

The IN operator is a normal SQL function whose second argument is not a value but a set object:

 cpp
 
 
// src/Functions/in.cpp:145
 ColumnPtr column_set_ptr = arguments[1].column;
 const ColumnSet * column_set = tryGetColumnSet(column_set_ptr);
 if (!column_set)
 throw Exception(ErrorCodes::ILLEGAL_COLUMN, "Second argument for function '{}' must be Set; found {}",
 getName(), column_set_ptr->getName());

 auto future_set = column_set->getData();
 if (!future_set)
 {
 if (dry_run)
 return ColumnUInt8::create(input_rows_count, static_cast<UInt8>(0));

 throw Exception(ErrorCodes::LOGICAL_ERROR, "No Set is passed as the second argument for function '{}'", getName());
 }

The set arrives as a ColumnSet holding a FutureSet — a promise, because the set may not be built yet when the function’s type is resolved. A separate step in the pipeline builds it, and the function then asks for it:

 cpp
 
 
// src/Functions/in.cpp:160
 auto set = future_set->get();
 if (!set)
 {
 if (dry_run)
 return ColumnUInt8::create(input_rows_count, static_cast<UInt8>(0));

 throw Exception(ErrorCodes::LOGICAL_ERROR, "Not-ready Set is passed as the second argument for function '{}'", getName());
 }

There are three kinds of FutureSet:

 cpp
 
 
// src/Interpreters/PreparedSets.h:66
class FutureSet
{
public:
 virtual ~FutureSet() = default;

 /// Returns set if set is ready (created and filled) or nullptr if not.
 virtual SetPtr get() const = 0;
 /// Returns set->getElementsTypes(), even if set is not created yet.
 virtual DataTypes getTypes() const = 0;
 /// If possible, return set with stored elements useful for PK analysis.
 virtual SetPtr buildOrderedSetInplace(const ContextPtr & context) = 0;

FutureSetFromTuple — the right-hand side is a literal list, IN (1, 2, 3). It is built during analysis, because the values are already known.
FutureSetFromSubquery — the right-hand side is a SELECT. It is built during execution, by a CreatingSetStep that runs before the steps that use it.
FutureSetFromStorage — the right-hand side is an ENGINE = Set table. The set already exists; the query only reads it.

The literal case is where the “prepare once, use many” idea shows up, because the same literal list may appear many times in a query:

 cpp
 
 
// src/Interpreters/ActionsVisitor.cpp:557
 auto set_key = right_arg->getTreeHash(/*ignore_aliases=*/ true);
 if (auto set = prepared_sets.findTuple(set_key, set_element_keys))
 return set; /// Already prepared.

 return prepared_sets.addFromTuple(set_key, right_arg_func, std::move(block), context->getSettingsRef());

The key is the tree hash of the AST of the right-hand side, so x IN (1,2,3) and y IN (1,2,3) share one set. The prepared sets live in a per-query container:

 cpp
 
 
// src/Interpreters/PreparedSets.h:301
 using SetsFromTuple = std::unordered_map<Hash, std::vector<FutureSetFromTuplePtr>, Hashing>;
 using SetsFromStorage = std::unordered_map<Hash, FutureSetFromStoragePtr, Hashing>;
 using SetsFromSubqueries = std::unordered_map<Hash, FutureSetFromSubqueryPtr, Hashing>;

The hash is 128 bits of CityHash, and the container hashes it down to 64 bits:

 cpp
 
 
// src/Interpreters/PreparedSets.h:278
struct FutureSetHashing
{
 UInt64 operator()(const FutureSet::Hash & key) const { return key.low64 ^ key.high64; }
};

The shape of the thing

 text
 
 
 SELECT ... WHERE x IN (SELECT id FROM t)
 |
 v
 CreatingSetStep ---> FutureSetFromSubquery
 | (built during execution, not analysis)
 v
 ActionsDAG: FunctionIn(x, ColumnSet) ---> Set::execute(key_columns)
 |
 chooseMethod(key_columns) -+--> key8 / key16 / key32 / key64
 | (one integer key: the key IS the slot)
 +--> keys32 / keys64 / keys128 / keys256
 | (fixed-width tuple: packed into one word)
 +--> key_string / key_fixed_string
 | (one string: chars arena + saved hash)
 +--> nullable_keys128 / nullable_keys256
 | (value + null flag packed together)
 +--> hashed
 (anything else: 128-bit hash only)

Choosing the data structure

This is the part that makes ClickHouse’s IN fast. The set’s key type is not “some value” — it is a tuple of columns with concrete types, and each shape deserves a different hash table.

SetVariants::chooseMethod inspects the key columns and picks one of thirteen implementations:

 cpp
 
 
// src/Interpreters/SetVariants.h:417
 #define APPLY_FOR_SET_VARIANTS(M) \
 M(key8) \
 M(key16) \
 M(key32) \
 M(key64) \
 M(key_string) \
 M(key_fixed_string) \
 M(keys32) \
 M(keys64) \
 M(keys128) \
 M(keys256) \
 M(nullable_keys128) \
 M(nullable_keys256) \
 M(hashed)

The decision tree, in order:

One nullable numeric key. The value and the NULL flag are packed into one 16-byte key, so the hash table needs no separate null handling:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:149
 if (has_nullable_key)
 {
 /// At least one key is nullable. Therefore we choose a method
 /// that takes into account this fact.
 if ((keys_size == 1) && (nested_key_columns[0]->isNumeric()))
 {
 /// We have exactly one key and it is nullable. We shall add it a tag
 /// which specifies whether its value is null or not.
 size_t size_of_field = nested_key_columns[0]->sizeOfValueIfFixed();
 if ((size_of_field == 1) || (size_of_field == 2) || (size_of_field == 4) || (size_of_field == 8))
 return Type::nullable_keys128;

One numeric key. A single UInt32 column gets a HashSet<UInt32> — the key is the hash-table key, no packing, no hashing beyond the CRC:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:180
 /// If there is one numeric key that fits into 64 bits
 if (keys_size == 1 && nested_key_columns[0]->isNumeric() && !nested_key_columns[0]->lowCardinality())
 {
 size_t size_of_field = nested_key_columns[0]->sizeOfValueIfFixed();
 if (size_of_field == 1)
 return Type::key8;
 if (size_of_field == 2)
 return Type::key16;
 if (size_of_field == 4)
 return Type::key32;
 if (size_of_field == 8)
 return Type::key64;

Several fixed-width keys. They are packed into 32/64/128/256 bits and the whole packed word is the key:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:199
 /// If the keys fit in N bits, we will use a hash table for N-bit-packed keys
 if (all_fixed && keys_bytes <= 4)
 return Type::keys32;
 if (all_fixed && keys_bytes <= 8)
 return Type::keys64;
 if (all_fixed && keys_bytes <= 16)
 return Type::keys128;
 if (all_fixed && keys_bytes <= 32)
 return Type::keys256;

One string key. A dedicated method with the string’s own arena:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:209
 /// If there is single string key, use hash table of it's values.
 if (keys_size == 1
 && (typeid_cast<const ColumnString *>(nested_key_columns[0])
 || (isColumnConst(*nested_key_columns[0]) && typeid_cast<const ColumnString *>(&assert_cast<const ColumnConst *>(nested_key_columns[0])->getDataColumn())))
 return Type::key_string;

Anything else. The fallback hashes the whole tuple to 128 bits and stores only the hash — the original values are not kept:

 cpp
 
 
// src/Interpreters/SetVariants.cpp:218
 /// Otherwise, will use set of cryptographic hashes of unambiguously serialized values.
 return Type::hashed;

The comment in the header explains the memory trade-off of that last choice:

 cpp
 
 
// src/Interpreters/SetVariants.h:357
 /** Unlike Aggregator, `concat` method is not used here.
 * This is done because `hashed` method, although slower, but in this case, uses less RAM.
 * since when you use it, the key values themselves are not stored.
 */

Presence-only vs counting

There are two flavours of the same set family. A plain IN only needs to know whether a key is present, so it uses a HashSet whose cell maps to VoidMapped. INTERSECT ALL / EXCEPT ALL need how many times a key occurs, so they use a HashMap with a UInt64 counter. The mapping is derived from the table rather than passed in, so a method stays “a set” when given a HashSet and becomes a multiset when given a HashMap:

 cpp
 
 
// src/Interpreters/SetVariants.h:30
/// The method structs below are shared between presence-only sets (a `HashSet`, whose cell reports
/// `VoidMapped`) and counting multisets (a `HashMap`, with a real mapped type, used by INTERSECT
/// ALL / EXCEPT ALL). The mapped type is derived from the table rather than passed explicitly, so a
/// method stays "a set" when given a `HashSet` and becomes a counting multiset when given a `HashMap`.
template <typename TData>
using SetMethodMapped = std::conditional_t<std::is_same_v<typename TData::mapped_type, VoidMapped>, void, typename TData::mapped_type>;

The choice of table type per key shape is visible in the variant declarations:

 cpp
 
 
// src/Interpreters/SetVariants.h:344
 std::unique_ptr<SetMethodOneNumber<UInt32, HashSet<UInt32, HashCRC32<UInt32>>>> key32;
 std::unique_ptr<SetMethodOneNumber<UInt64, HashSet<UInt64, HashCRC32<UInt64>>>> key64;
 std::unique_ptr<SetMethodString<HashSetWithSavedHash<std::string_view>>> key_string;
 std::unique_ptr<SetMethodFixedString<HashSetWithSavedHash<std::string_view>>> key_fixed_string;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt32, HashCRC32<UInt32>>>> keys32;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt64, HashCRC32<UInt64>>>> keys64;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt128, UInt128HashCRC32>>> keys128;
 std::unique_ptr<SetMethodKeysFixed<HashSet<UInt256, UInt256HashCRC32>>> keys256;
 std::unique_ptr<SetMethodHashed<HashSet<UInt128, UInt128TrivialHash>>> hashed;

HashSetWithSavedHash is worth a note: for strings it stores the hash next to the string, so a probe that hits the same bucket doesn’t have to re-hash the candidate key.

Building the set

Insertion goes through emplaceKey on the method’s state, one row at a time, with a twist: the same loop can build a deduplication filter, which records which rows were new. That filter is how the set keeps the original values around for primary-key analysis:

 cpp
 
 
// src/Interpreters/Set.cpp:370
 typename Method::State state(key_columns, key_sizes, nullptr);

 /// For all rows
 for (size_t i = 0; i < rows; ++i)
 {
 if constexpr (has_null_map)
 {
 if ((*null_map)[i])
 {
 if constexpr (build_filter)
 {
 (*out_filter)[i] = false;
 }
 continue;
 }
 }

 [[maybe_unused]] auto emplace_result = state.emplaceKey(method.data, i, variants.string_pool);

 if constexpr (build_filter)
 (*out_filter)[i] = emplace_result.isInserted();
 }

emplace_result.isInserted() is the deduplication: a row whose key was already in the set is not new, so it gets filtered out of the “explicit set elements” — the copy of the values kept for index analysis.

Keeping the values is optional and bounded. When the set grows past max_elements_to_fill, the copy is dropped and the set keeps only the hashes:

 cpp
 
 
// src/Interpreters/Set.cpp:495
 bool inserted = insertFromColumnsImpl(columns, holder);
 if (inserted && fill_set_elements)
 {
 if (max_elements_to_fill && max_elements_to_fill < data->getTotalRowCount())
 {
 /// Drop filled elementes
 fill_set_elements = false;
 set_elements.clear();
 }
 else
 appendSetElements(holder);
 }

That’s a graceful degradation: the set still answers IN correctly, it just can no longer be used to prune granules by primary key. The threshold is the setting use_index_for_in_with_subqueries_max_values.

Probing the set

The probe is the hot path: it runs once per row of the left-hand side.

 cpp
 
 
// src/Interpreters/Set.cpp:1002
 Arena pool;
 typename Method::State state(key_columns, key_sizes, nullptr);

 /// Clustered key columns (e.g. a primary key prefix) arrive in runs of equal consecutive
 /// rows. The consecutive-keys optimization in ColumnsHashing handles them inside `findKey`:
 /// the last-element cache compares the key with the previous row's before probing the hash
 /// table, and `HashMethodHashed` additionally compares the raw key bytes before even
 /// calculating the hash.
 for (size_t i = 0; i < rows; ++i)
 {
 if (has_null_map && (*null_map)[i])
 {
 vec_res[i] = negative;
 }
 else
 {
 auto find_result = state.findKey(method.data, i, pool);
 vec_res[i] = negative ^ find_result.isFound();
 }
 }

Two things worth noticing.

The “last element cache”. When the input is sorted (or clustered by a primary key), consecutive rows often have the same key. The hashing state remembers the last key and its answer, so a repeat costs one comparison instead of a hash computation plus a probe. This optimisation is on for numeric keys:

 cpp
 
 
// src/Interpreters/SetVariants.h:44
template <typename FieldType, typename TData, bool use_cache = true> /// UInt8/16/32/64 for any types with corresponding bit width.
struct SetMethodOneNumber
{
 using Data = TData;
 using Key = typename Data::key_type;

 Data data;

 using State = ColumnsHashing::HashMethodOneNumber<typename Data::value_type,
 SetMethodMapped<Data>, FieldType, set_method_use_cache<Data, use_cache>>;

and off for strings, with a measurement in the comment:

 cpp
 
 
// src/Interpreters/SetVariants.h:62
/// For the case where there is one string key.
/// Unlike aggregation, the consecutive-keys cache stays disabled for string keys: a set probe's
/// per-row work is just hash+lookup, which for strings is cheaper than the extra bytewise
/// comparison the cache check performs on every miss. Measured 2026-08 on 20M-row `IN` probes:
/// enabling the cache made unclustered inputs 40% slower for short `String` keys and 24% slower
/// for `FixedString(36)` keys, while clustered inputs (runs of equal consecutive keys) improved
/// only 8-16%. In aggregation the same check is relatively cheap because the per-row work there
/// (aggregate-state updates) is much heavier.
template <typename TData>
struct SetMethodString
{
 using Data = TData;
 using Key = typename Data::key_type;

 Data data;

 using State = ColumnsHashing::HashMethodString<typename Data::value_type, SetMethodMapped<Data>, true, false>;

That is a rare and welcome thing to find in a codebase: an optimisation that was measured, found harmful for one type family, and turned off — with the numbers kept in the source. Most codebases just leave the “optimisation” in and wonder why strings are slower than integers.

The NULL rule. A row whose key contains a NULL is never in the set, so the answer is negative (0 for IN, 1 for NOT IN) without a lookup. The rationale is in the header:

 cpp
 
 
// src/Interpreters/Set.h:202
 /** How IN works with Nullable types.
 *
 * For simplicity reasons, all NULL values and any tuples with at least one NULL element are ignored in the Set.
 * And for left hand side values, that are NULLs or contain any NULLs, we return 0 (means that element is not in Set).
 *
 * If we want more standard compliant behaviour, we must return NULL
 * if lhs is NULL and set is not empty or if lhs is not in set, but set contains at least one NULL.
 * It is more complicated with tuples.
 * For example,
 * (1, NULL, 2) IN ((1, NULL, 3)) must return 0,
 * but (1, NULL, 2) IN ((1, 1111, 2)) must return NULL.
 *
 * We have not implemented such sophisticated behaviour.
 */

So NULL IN (1, 2, 3) is 0, not NULL. A deliberate deviation from the SQL standard, documented in the source. Gotcha if you are porting queries from a stricter engine.

Type coercion on the probe

The left-hand side of the IN may have a different type from the set’s key — the set’s type wins, and the probe column is cast. The cast is cached, so it isn’t rebuilt for every block:

 cpp
 
 
// src/Interpreters/Set.h:249
 /// A cache for cast functions (if any) to avoid rebuilding cast functions
 /// for every call to `execute`
 mutable std::unique_ptr<InternalCastFunctionCache> cast_cache;

and when the value does not fit the key type, it is not an error — it is simply “not a member”:

 cpp
 
 
// src/Interpreters/Set.cpp:833
 if (can_convert_leniently)
 {
 /// `transform_null_in` makes NULL an ordinary matchable value, so the accurate-or-null
 /// cast above is not taken. A value the key type cannot represent is still simply not a
 /// member - which is what the literal `IN` list concludes, because it compares in a
 /// common supertype - but converting it strictly raised `CANNOT_CONVERT_TYPE` on rows
 /// the read path happened to deliver, so `v IN set_table` failed where `v IN (1)`
 /// returned 0.

Size limits and overflow

The set is subject to max_rows_in_set and max_bytes_in_set, checked after every inserted chunk:

 cpp
 
 
// src/Interpreters/Set.cpp:556
 bool within_limits = limits.check(data->getTotalRowCount(), data->getTotalByteCount(), "IN-set", ErrorCodes::SET_SIZE_LIMIT_EXCEEDED);
 if (!within_limits)
 is_truncated = true;

The overflow mode is set_overflow_mode: throw (the default) aborts the query, break truncates the set and returns the partial result — which is wrong for IN, but is the only sane option for a GLOBAL IN over a huge table, where a partial set is still useful.

Spilling to disk

A set built from a subquery can be enormous: x IN (SELECT id FROM big_table) over a billion rows. ClickHouse 26.10 added spilling: the set moves to a temporary file, and the in-memory part shrinks to a directory plus a set of Bloom filters.

The disk representation is not the original values. Each key is converted into a compact disk key: numbers and FixedStrings of at most 32 bytes are packed into integers, and everything else becomes a 128-bit hash:

 cpp
 
 
// src/Interpreters/Set.h:85
 /// the keys that come first in the order of their disk keys rather than the first ones to arrive. Disk
 /// keys keep the order of the keys that the set packs into integers in memory, numbers and `FixedString`
 /// values of at most 32 bytes in total, with `NULL` after the values, so these are the smallest keys.
 /// Other keys, such as `String`, are stored as 128-bit hashes, whose order is arbitrary.

The keys are then sorted, so a lookup can binary-search the directory and find the block that may contain the key.

The Bloom filters

Each block of the set gets its own Bloom filter, and all the filters share one array of bits:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:15
/// Bloom filters of the blocks of a set on disk, which the set keeps in memory. A lookup checks the filter of
/// the only block that can hold its key, and reads the block only when the filter lets the key pass.
///
/// The filters share one array of bits. Each block takes a range of it in proportion to its keys, and each
/// key sets a few bits in the range of its block. A single filter for all keys would take as much memory, but
/// a key would set bits anywhere in it, leading to writes at random places all over the array. For a large
/// set, the array is far larger than the cache, so each such write would miss the cache and the TLB, and
/// these misses would slow down writing the set, because the set fills the filters on the single thread that
/// writes its blocks. The range of a block is about as small as the block, so it stays in the cache while the
/// set writes the block. Lookups check the ranges in the order of the blocks, as they read the blocks.

That’s a lovely piece of engineering: the filters are per block so a write touches only a cache-resident range, not the whole (possibly huge) filter array. Think of it as giving each block its own little padlock instead of one giant lock everyone fights over.

The filter is sized by the number of distinct keys, which is estimated with a HyperLogLog counter while the set is being filled:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:29
 /// Counts the distinct keys of a set as they arrive, so that the bloom filters can be sized before the set
 /// writes its blocks. With 2^20 buckets, the estimate is within about 0.1% and the counter takes only 768 KiB.
 class DistinctKeyCounter
 {
 public:
 void add(std::span<const Key> keys)
 {
 for (const Key & key : keys)
 counter.insert(hashKey(key));
 }

and the bits-per-key is chosen to fit the memory budget, trading false positives for memory:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:46
 /// Creates the filters for about `expected_keys` keys. They take 8 bits per key, which lets about 3% of
 /// the keys that the set does not have pass. When that does not fit in `max_bytes` bytes with the ranges
 /// of the blocks, they take fewer bits per key, so that they fit: with 2 bits per key, about 40% of these
 /// keys pass, and with 1 bit per key, about 63%. Every block but the last holds at least
 /// `min_keys_per_block` keys, which bounds the number of ranges. Returns nothing when fewer than 1 bit
 /// per key fits, because such filters let so many keys pass that they do not pay for filling them.

The number of hash positions per key is capped at 3, with the reasoning spelled out:

 cpp
 
 
// src/Interpreters/DiskSetBlockFilters.h:82
 /// A key sets the bits at most at this many positions in the range of its block. Each position costs a
 /// memory access when a lookup checks a key that the set has, while more positions let only slightly
 /// fewer other keys pass: with 8 bits per key, 3 positions let about 3.1% of them pass, and 6 positions,
 /// the most effective number, about 2.2%.
 static constexpr size_t max_positions_per_key = 3;

The lookup

A lookup against the disk set is a five-step algorithm: collect the probes, sort them, find the blocks, consult the filters, read the blocks.

 cpp
 
 
// src/Interpreters/DiskSet.cpp:121
 /// Step 1: Collect the probes: the rows whose keys are in the range of the set, except a row that repeats
 /// the key of the previous row, which takes that row's answer in step 5. So a run of equal keys, as in a
 /// sorted column, takes one probe.
 struct Probe
 {
 Key key;
 size_t row;
 };

 VectorWithMemoryTracking<Probe> probes;
 probes.reserve(keys.size());
 const Key & first_key = directory.getFirstKey(0);
 for (size_t row = 0; row < keys.size(); ++row)
 {
 const auto & key = keys[row];
 if ((row == 0 || key != keys[row - 1]) && first_key <= key && key <= last_key)
 probes.push_back({key, row});
 }

Note the range check first_key <= key && key <= last_key: keys outside the set’s key range cannot be in the set, and are answered without any I/O.

Then the probes are sorted, so that each block is read once and in order:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:143
 /// Step 2: Sort the probes by key. Probes of equal keys then come together, so each distinct key is
 /// looked up once, and the probes of each block come together in block order, so they share a read.
 /// Results go back to the original rows, so evaluating `IN` does not reorder the input.
 ::sort(probes.begin(), probes.end(), [](const Probe & lhs, const Probe & rhs) { return lhs.key < rhs.key; });

and the filters decide which blocks are actually read:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:174
 for (size_t begin = 0; begin < probes.size();)
 {
 size_t end = begin + 1;
 while (end < probes.size() && probes[end].key == probes[begin].key)
 ++end;

 block = directory.findBlock(probes[begin].key, block);
 if (!filters || filters->mayContain(block, probes[begin].key))
 {
 if (lookups.empty() || block != lookups.back().block)
 needed_blocks.push_back(block); /// Each block once, in ascending order
 lookups.push_back({block, begin, end});
 }
 else
 {
 for (size_t i = begin; i < end; ++i)
 skipped_rows += count_rows_of_probe(probes[i]);
 }
 begin = end;
 }

The number of rows the filters answered is a profile event, so the effectiveness of the filters is directly observable:

 cpp
 
 
// src/Interpreters/DiskSet.cpp:195
 /// A skipped key counts once for each of its rows, as `ExternalSetLookupKeys` does, so that the two
 /// events give the share of the rows that the filters answer.
 ProfileEvents::increment(ProfileEvents::ExternalSetBloomFilterSkippedKeys, skipped_rows);

When does it spill?

The threshold is a memory budget, and the default is “half of the available memory”:

 cpp
 
 
// src/Core/Settings.cpp:3905
 DECLARE(Double, max_bytes_ratio_before_external_set, 0.5, R"(
Fraction of available server or user memory used to calculate the threshold for spilling the set of `IN`
with a subquery to disk, at the start of execution. For example, `0.5` uses half of the available memory.

The set spills while it is built and while it is used, and the two cases are handled differently. While it is built, the keys are handed to a DiskSetBuilder that sorts them into runs; while it is used, the keys are written out while the lookups keep reading the in-memory table, and the disk set replaces it at the end.

The bloom filter has its own budget:

 cpp
 
 
// src/Core/Settings.cpp:3917
 DECLARE(UInt64, max_bytes_for_external_set_bloom_filter, 128_MiB, R"(
Memory budget, in bytes, for the Bloom filter that each set of `IN` written to disk keeps in memory. The
filter lets lookups of values that are not in the set skip reading the disk. It does not speed up lookups
of values that are in the set.

Note the asymmetry in that sentence: the filter helps for misses, not for hits. A key that is in the set still costs a block read. That’s the fundamental trade of a Bloom filter, and it’s why the filter is only worth it when the selectivity is low.

Trade-offs

What it buys: IN over a subquery is a hash-table probe per row, with a data structure chosen to fit the key type exactly. No row-at-a-time interpretation, no generic comparison, and for the common cases (one integer key, one string key) the key is the hash-table key.

What it costs:

Memory. The set is proportional to the number of distinct values on the right-hand side. x IN (SELECT id FROM orders) with 100 M distinct ids is a 100 M-entry hash table. This is why the spilling machinery exists, and why max_rows_in_set exists.
Spilling is much slower than RAM. A probe against a disk set is a sort, a binary search, a Bloom filter check and possibly a block read. ClickHouse sorts the probe keys to make the reads sequential, which is a lot of work per block. The alternative — failing the query with SET_SIZE_LIMIT_EXCEEDED — is often better for the user.
The hash fallback loses information. For a heterogeneous key, the set stores a 128-bit hash of the tuple, not the tuple. Two different tuples with the same hash would be “equal”; the probability is negligible, but it is not zero.
NULL semantics deviate from the standard, as documented above.
The hashed method is slower per probe than the packed methods, because the key must be serialised and hashed on every probe rather than once at build time. For a set of strings, the string method stores the string and its hash; the hashed method stores only the hash.

That last point is the general lesson of the whole design: ClickHouse’s IN is fast not because hashing is fast, but because the shape of the data is known at analysis time and the implementation specialises on it — thirteen variants, each with its own hash function, its own packing, its own arena, and its own micro-optimisations, some of which are enabled and some of which are deliberately disabled after measurement.

Wrapping up

The one thing to remember: IN is a hash set, and the set’s data structure is chosen from the type of the right-hand side at analysis time. That is why IN (1,2,3) on an integer column is essentially free.

What to go do about it: if you have a query with IN (SELECT ...) over a big table, check whether the right-hand side has an index or a small enough cardinality. If neither, the hash set is going to be the bottleneck — and spilling it to disk makes it worse, not better.

The part people get wrong: a Bloom filter in front of a disk set only helps with misses. If your IN mostly hits, the filter is pure overhead.

ClickHouse Estimates Distinct Counts by Thinning a Hash Table

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: uniq() is a hash table that deliberately throws data away. When it fills up, it halves its sample rate, keeps the survivors, and multiplies the count back up by the thinning factor. The state stays at a few hundred KiB no matter how many distinct values you feed it.

uniq() is one of the most-used aggregate functions in ClickHouse, and one of the most surprising. It is not exact. It is not a HyperLogLog. It is a hash table that throws data away on purpose, thinning itself as it fills up, and then multiplies back up by the thinning factor to estimate the cardinality.

The algorithm behind it is BJKST (Bretag, Karp, Kutz, Schlossmann — “near-optimal algorithms for estimating the number of distinct elements in a stream of data”). The implementation lives in src/AggregateFunctions/UniquesHashSet.h, and it is one of the more elegant pieces of code in the aggregation framework: a fixed-size table, a single integer of state, and a mathematically exact estimator of what the sample means.

The scheme

The header states the general recipe for approximate aggregation:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:27
/** Approximate calculation of anything, as usual, is constructed according to the following scheme:
 * - some data structure is used to calculate the value of X;
 * - Not all values are added to the data structure, but only selected ones (according to some selectivity criteria);
 * - after processing all elements, the data structure is in some state S;
 * - as an approximate value of X, the value calculated according to the maximum likelihood principle is returned:
 * at what real value X, the probability of finding the data structure in the obtained state S is maximal.
 */

And then the specific recipe:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:38
/** Very simple hash-set for approximate number of unique values.
 * Works like this:
 * - you can insert UInt64;
 * - before insertion, first the hash function UInt64 -> UInt32 is calculated;
 * - the original value is not saved (lost);
 * - further all operations are made with these hashes;
 * - hash table is constructed according to the scheme:
 * - open addressing (one buffer, position in buffer is calculated by taking remainder of division by its size);
 * - linear probing (if the cell already has a value, then the cell following it is taken, etc.);
 * - the missing value is zero-encoded; to remember presence of zero in set, separate variable of type bool is used;
 * - buffer growth by 2 times when filling more than 50%;
 * - if the set has more than UNIQUES_HASH_MAX_SIZE elements, then all the elements are removed from the set,
 * not divisible by 2, and then all elements that do not divide by 2 are not inserted into the set;
 * - if the situation repeats, then only elements dividing by 4, etc., are taken.
 * - the size() method returns an approximate number of elements that have been inserted into the set;
 * - there are methods for quick reading and writing in binary and text form.
 *
 * A 32-bit hash cannot distinguish more than a few billion elements: the estimate degrades quickly
 * beyond a billion, and above ~30 billion the correction formula even resulted in undefined behavior
 * (issue #6078). To keep the estimate accurate up to trillions, in addition to the main set a small
 * "wide" set of full 64-bit hashes (XOR-ed with a constant, see `wideHash`) is maintained (state version 1).
 * It is thinned out aggressively from the very beginning - only values with UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE
 * zero low-order bits enter it - so it stays empty or tiny for the states where the main set is accurate
 * (about 4 values per million distinct elements), and it is bounded by UNIQUES_HASH_SET_WIDE_MAX_SIZE
 * values thinned out further. Once the wide set has enough values for its own estimate to be reliable
 * (which happens at about two billion elements), size() switches to it.
 *
 * The wide set has to be maintained from the first insertion: the high bits of the hashes
 * of the already consumed elements cannot be recovered later. In particular, a state written
 * in the legacy format (state version 0) carries no wide set, and its elements were never
 * offered to one. Such a state is marked as having an incomplete wide sample: it drops its own
 * wide set, never starts a new one, and propagates the mark through merges, so that size()
 * of anything it is merged into falls back to the legacy 32-bit estimator. Estimating from
 * a wide set that covers only a part of the elements would undercount the rest entirely,
 * which is much worse than the degrading precision of the 32-bit estimate.
 */

That comment is the whole algorithm. Read it twice: the key insight is that the set does not grow without bound — it changes what it accepts.

The constants

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:75
/// The maximum degree of buffer size before the values are discarded
#define UNIQUES_HASH_MAX_SIZE_DEGREE 17

/// The maximum number of elements before the values are discarded
#define UNIQUES_HASH_MAX_SIZE (1ULL << (UNIQUES_HASH_MAX_SIZE_DEGREE - 1))

/** The number of least significant bits used for thinning. The remaining high-order bits are used to determine the position in the hash table.
 * (high-order bits are taken because the younger bits will be constant after dropping some of the values)
 */
#define UNIQUES_HASH_BITS_FOR_SKIP (32 - UNIQUES_HASH_MAX_SIZE_DEGREE)

/// Initial buffer size degree
#define UNIQUES_HASH_SET_INITIAL_SIZE_DEGREE 4

So the main set holds at most 2^16 = 65 536 hashes in a buffer of 2^17 slots, and 15 low bits of the hash are reserved for the thinning mask. The remaining 17 high bits select the cell:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:195
 size_t place(HashValue x) const { return (x >> UNIQUES_HASH_BITS_FOR_SKIP) & mask(); }

 /// The value is divided by 2 ^ skip_degree
 bool good(HashValue hash) const { return hash == ((hash >> skip_degree) << skip_degree); }

good() is the thinning test: the low skip_degree bits of the hash must be zero. The high bits pick the position, because the low bits are constant across all surviving elements — using them for the position would pile every element into the same cell.

Thinning

The state is three integers plus a flag:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:155
 UInt32 m_size; /// Number of elements
 UInt8 size_degree{}; /// The size of the table as a power of 2
 UInt8 skip_degree; /// Skip elements not divisible by 2 ^ skip_degree
 bool has_zero; /// The hash table contains an element with a hash value of 0.

And the moment of truth is shrinkIfNeed:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:346
 /** If the hash table is full enough, then do resize.
 * If there are too many items, then throw half the pieces until they are small enough.
 */
 void shrinkIfNeed()
 {
 if (unlikely(m_size > max_fill()))
 {
 if (m_size > UNIQUES_HASH_MAX_SIZE)
 {
 while (m_size > UNIQUES_HASH_MAX_SIZE)
 {
 if (unlikely(skip_degree >= max_skip_degree))
 throw DB::Exception(DB::ErrorCodes::INCORRECT_DATA,
 "Cannot thin out UniquesHashSet: {} elements remain at the maximum skip degree of {}",
 static_cast<size_t>(m_size), static_cast<size_t>(max_skip_degree));

 ++skip_degree;
 rehash();
 }
 }
 else
 resize();
 }
 }

So: the set grows normally until it holds more than 65 536 elements. Then it does not grow — it thins. skip_degree goes up by one, and every element whose hash is not divisible by 2^skip_degree is deleted:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:200
 /// Delete all values whose hashes do not divide by 2 ^ skip_degree
 void rehash()
 {
 for (size_t i = 0; i < buf_size(); ++i)
 {
 if (buf[i])
 {
 if (!good(buf[i]))
 {
 buf[i] = 0;
 --m_size;
 }
 /** After removing the elements, there may have been room for items,
 * which were placed further than necessary, due to a collision.
 * You need to move them.
 */
 else if (i != place(buf[i]))
 {
 HashValue x = buf[i];
 buf[i] = 0;
 reinsertImpl(x);
 }
 }
 }

The picture is this:

 text
 
 
 distinct values arriving: A B C D E F G H ... (millions)

 skip_degree = 0 -> accept everything, table grows
 table full at 65536 elements, distinct count so far maybe 200 000
 skip_degree = 1 -> accept only hashes with bit0 == 0 (half of them)
 table full again at 65536, distinct count so far maybe 400 000
 skip_degree = 2 -> accept only hashes with bits1..0 == 0 (a quarter)
 ...
 skip_degree = k -> accept 1 in 2^k elements

Each thinning step halves the sample rate, and the sample size stays roughly constant. The estimate multiplies the surviving count back up by 2^skip_degree.

Here’s the thing: the table does not need to know how many distinct values it has seen. It only needs to know how aggressively it is currently sampling. The count falls out of that at the end.

The estimator

The estimator is not simply m_size * 2^skip_degree — there is a correction for the systematic error of 32-bit hash collisions, and a pseudo-random remainder so that the result does not visibly sit on a power of two:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:842
 if (0 == skip_degree)
 return m_size;

 size_t res = m_size * (1ULL << skip_degree);

 /** Pseudo-random remainder - in order to be not visible,
 * that the number is divided by the power of two.
 */
 res += (intHashCRC32(m_size) & ((1ULL << skip_degree) - 1));

 /** Correction of a systematic error due to collisions during hashing in UInt32.
 * `fixed_res(res)` formula
 * - with how many different elements of fixed_res,
 * when randomly scattered across 2^32 buckets,
 * filled buckets with average of res is obtained.
 */
 return correctFor32BitCollisions(res);

The pseudo-random remainder is a lovely touch of honesty. Without it, every estimate would be a multiple of a power of two, and users would (correctly!) assume the result was rounded. It only costs one CRC and one mask.

The collision correction is an analytic formula for the birthday problem. If res distinct elements are scattered over 2^32 buckets, the number of occupied buckets is less than res because of collisions; to recover the number of distinct elements from the number of occupied buckets, you invert that relation:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:548
 /** With how many different elements, when randomly scattered across 2^32 buckets,
 * the given number of non-empty buckets is obtained on average.
 * This is the correction of the systematic error due to collisions of the 32-bit hashes.
 */
 static size_t correctFor32BitCollisions(size_t res)
 {
 size_t p32 = 1ULL << 32;

 /** When the number of distinct elements is much larger than 2^32, almost all buckets are filled
 * and `res` may reach (or, due to the pseudo-random remainder, exceed) 2^32,
 * where the formula below is not defined (previously this resulted in undefined behavior
 * when casting infinity or NaN to an integer). Saturate the estimate instead:
 * 32 bits of a hash contain no information about larger cardinalities anyway
 * (and the wide set takes over long before this point).
 */
 if (res >= p32)
 res = p32 - 1;

 return static_cast<size_t>(round(static_cast<double>(p32) * (log(static_cast<double>(p32)) - log(static_cast<double>(p32 - res)))));
 }

Read the formula as p32 * (ln p32 - ln(p32 - res)). For small res it is approximately res; for res near p32 it blows up, which is why the saturation guard exists — and that guard is the fix for issue #6078, where the cast of inf to an integer was undefined behaviour.

The wide set: fixing the 32-bit ceiling

The 32-bit hash has a hard ceiling. At a few billion distinct elements the correction formula is already straining, and above ~30 billion it was producing garbage. The fix, added in state version 1, is a second set that stores full 64-bit hashes — but only a tiny, heavily thinned sample of them.

The sample is defined by a skip degree of 18: only hashes with 18 zero low-order bits enter it, i.e. one in 262 144:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:89
/** Only hashes with this many zero low-order bits enter the wide set, so that it stays
 * negligibly small while the main set is accurate: one hash in 262144 passes.
 */
#define UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE 18

/// The maximum degree of the wide set buffer size before the values are thinned out further.
#define UNIQUES_HASH_SET_WIDE_MAX_SIZE_DEGREE 15

/// The maximum number of elements of the wide set: 16384 values take 256 KiB.
#define UNIQUES_HASH_SET_WIDE_MAX_SIZE (1ULL << (UNIQUES_HASH_SET_WIDE_MAX_SIZE_DEGREE - 1))

/** The estimate is taken from the wide set when it has at least this many values, i.e. starting
 * from about UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE * 2^UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE,
 * about two billion elements: at that point the statistical error of the wide estimate (about 1%)
 * meets the degrading precision of the 32-bit one.
 */
#define UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE 8192

So the wide set holds at most 16 384 hashes (256 KiB) and needs 8192 of them before it is trusted. Since only one in 262 144 elements enters it, 8192 elements means about 2.1 billion distinct values have been seen — exactly the point where the 32-bit estimator starts to degrade.

The wide set has one more twist: the hashes are XOR-ed with a constant before being stored.

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:518
 /** The wide set samples the full hashes XOR-ed with this constant rather than the hashes themselves.
 * The sample stays just as uniform, but the hash of a frequent value never becomes a candidate
 * systematically: the default hash of the integer 0 is 0, and zeros are frequent in real data -
 * without the offset every such row would be a candidate for the wide set and would cost
 * an extra pass over its batch (this doubled the time of `uniq` over such columns).
 * The low bits up to the maximum thinning of the main set are zero, so that every candidate
 * still passes good() of the main set (see `widePrefilterMask`), and the next ones are not,
 * so that the hash 0 is never a candidate.
 */
 static constexpr UInt64 wide_hash_offset = 0x9E3779B97F4A0000ULL;

The constant is the golden-ratio fraction used in many hash mixers (0x9E3779B97F4A0000 is 2^64 / φ). The problem it solves is delightfully concrete: the default hash of the integer 0 is 0, and the wide-set candidate test is “are the low 18 bits zero?”. The hash 0 passes that test — so every zero row in the data would enter the wide set, and the extra work doubled the runtime of uniq over columns full of zeros.

XOR-ing with an offset whose low bits are non-zero removes that pathology while preserving uniformity.

The offset has to satisfy two constraints, and they are checked at compile time:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:528
 static_assert((wide_hash_offset & ((1ULL << max_skip_degree) - 1)) == 0);
 static_assert((wide_hash_offset & ((1ULL << UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE) - 1)) != 0);

The first says: the offset’s low bits are zero up to the maximum thinning degree of the main set, so that a wide candidate still passes the main set’s good() test — the wide set is a subset of the main sample. The second says: the offset’s low 18 bits are not all zero, so the hash 0 is never a wide candidate.

Two constraints, two static_asserts. This is the good stuff: an invariant expressed as a compile-time proof instead of a comment.

The wide set is allocated lazily, in a single block holding both the header and the table:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:373
 static WideSet * allocWideSet(UInt8 new_size_degree)
 {
 /// The header and the table are placed in a single allocation.
 size_t bytes = sizeof(WideSet) + (1ULL << new_size_degree) * sizeof(UInt64);
 WideSet * res = new (HashTableAllocator().alloc(bytes)) WideSet;
 res->size_degree = new_size_degree;
 res->buf = reinterpret_cast<UInt64 *>(reinterpret_cast<char *>(res) + sizeof(WideSet));
 return res;
 }

and it is only allocated when the first candidate shows up — which, for most data, never happens:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:469
 if (!wide)
 wide = allocWideSet(UNIQUES_HASH_SET_INITIAL_SIZE_DEGREE);

Which estimator is used

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:820
 size_t size() const
 {
 /** The wide set gives a reliable estimate once it has enough values;
 * below that the 32-bit main set is more precise. If the wide sample does not cover
 * all the elements (see `wide_incomplete`), it is not there at all and the legacy
 * estimator is used, because a partial wide estimate would silently lose the rest.
 */
 if (wide && wide->m_size >= UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE)
 {
 size_t res = static_cast<size_t>(wide->m_size) * (1ULL << wide->skip_degree);

 /** Pseudo-random remainder - in order to be not visible,
 * that the number is divided by the power of two.
 */
 res += (intHashCRC32(wide->m_size) & ((1ULL << wide->skip_degree) - 1));

 /** Collisions of 64-bit hashes are negligible: their systematic error would reach
 * just 0.003% at 10^15 distinct elements, far below the statistical error of the estimate.
 */
 return res;
 }

Note the last comment: at 10^15 distinct elements, the systematic error of 64-bit hash collisions would be 0.003%, far below the statistical error of the estimate itself. So the wide set is not exact either — it is exact enough.

The hot loop: batching

The insertion path is where the performance lives, because uniq is usually applied to millions of rows. The implementation does not insert one hash at a time; it computes eight hashes and eight target cells first, then inserts them:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:749
 size_t i = 0;
 while (i < size)
 {
 /// Check that we have enough capacity and enough values to insert in batch. If not, we will insert values one by one which
 /// allows us to check the shrink condition after each insert and avoid inserting too many values before shrinking.
 if ((max_fill() - m_size) >= insert_many_batch_size && ((size - i) >= insert_many_batch_size))
 {
 /// We read and transform multiple values at once which allows both the compiler and the CPU to better optimize the code.
 /// We calculate place() even for !good() hashes to maximize data independence and enable better out-of-order execution.
 /// The extra work is negligible compared to the instruction level parallelization benefits.
 std::array<HashValue, insert_many_batch_size> hash_value; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - filled by the loop below before read
 for (size_t j = 0; j < insert_many_batch_size; ++j)
 {
 hash_value[j] = hash(Transform(data[i + j]));
 }

with the batch size fixed at eight:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:665
 /// This value is arbitrary. The optimal value might depend on the CPU pipeline depth, cache line size (64B = 8 UInt64s),
 /// hash function quality, collision rate, instruction parallelism, etc.
 /// We choose a value that is big enough to provide sufficient instruction level parallelism but not too big to bloat the code size.
 static constexpr size_t insert_many_batch_size = 8;

Two details worth noticing. place() is computed even for hashes that will be rejected by good(), on purpose: it keeps the loop free of data-dependent branches and lets the out-of-order engine overlap the work. And the wide-set candidates are collected into a small local buffer during the loop and inserted after it, because inserting them inside would force the compiler to reload the set’s fields from memory on every iteration:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:738
 /** The candidates for the wide set (see the comment at `widePrefilterMask`) are detected
 * behind the `good()` test, which every one of them passes and which already rejects all
 * but one value in 2^skip_degree, and they are only collected into a local buffer there.
 * They are inserted after the loop rather than inside it: a call inside would make
 * the compiler cautiously reload the fields of the set from memory all the time.
 * This loop is very sensitive to the code generation: the batch loops below must be
 * fully unrolled as they are without the wide set, otherwise `uniq` gets up to 50% slower.
 */

“otherwise uniq gets up to 50% slower” is a great reminder that in an aggregation hot loop, the code layout is part of the algorithm.

What is hashed

The value that goes into the set is not the value itself but a 64-bit hash of it, of which only the low 32 bits feed the main set:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:637
 /// The full 64-bit hash: its low 32 bits feed the main set, and the whole value the wide set (see `wideHash`).
 static UInt64 fullHash(Value key) { return static_cast<UInt64>(Hash()(key)); }
 static HashValue hash(Value key) { return static_cast<HashValue>(fullHash(key)); }

and the per-type hashing is chosen in the aggregate function:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniq.h:273
template <typename T, typename ColumnType>
struct AggregateFunctionUniqTraits
{
 static ALWAYS_INLINE auto hash(T x)
 {
 if constexpr (std::is_same_v<T, std::string_view>)
 {
 return CityHash_v1_0_2::CityHash64(x.data(), x.size());
 }
 else if constexpr (std::is_same_v<T, IPv6>)
 {
 /// For compatibility reasons IPv6 is hashed as FixedString(16).
 return CityHash_v1_0_2::CityHash64(reinterpret_cast<const char *>(&x), sizeof(x));
 }
 else if constexpr (is_floating_point<T>)
 {
 return bit_cast<UInt64>(x);
 }
 else if constexpr (sizeof(T) <= sizeof(UInt64))
 {
 return x;
 }
 else
 {
 return DefaultHash64<T>(x);
 }
 }

Note that for integers the “hash” is the identity: an integer’s low bits are used directly, which is fine because the thinning test only looks at the low 15..18 bits and the position comes from the high bits. For sequential integers the high bits are nearly constant too — which is precisely why the wide set XORs with an offset, and why the main set’s place() uses the high bits of the 32-bit hash, which is intHash32 and mixes.

Indeed the set’s own hash function is the 32-bit integer hash:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:108
/** This hash function is not the most optimal, but UniquesHashSet states counted with it,
 * stored in many places on disks (in many companies), so it continues to be used.
 */
struct UniquesHashSetDefaultHash
{
 size_t operator() (UInt64 x) const
 {
 return intHash32<0>(x);
 }
};

That comment is a nice piece of engineering history: the hash is frozen because the serialized states of uniq are stored on disk in many companies’ systems, and changing the hash would invalidate all of them.

The state format

Because the set is serialised into aggregate-function states that travel between servers, the format is versioned. Version 1 adds a flags byte:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:963
 /** Two serialization formats exist.
 * The legacy format (state version 0): skip_degree, then the number of values, then the 32-bit values
 * of the main set.
 * The new format (state version 1): a flags byte (bit 0 - a wide set follows the main set,
 * bit 1 - the wide sample is incomplete, see `wide_incomplete`; the two are mutually exclusive),
 * then the main set in the legacy layout, then, if present, the wide set: its skip_degree,
 * the number of values, and the 64-bit values.
 */

and a legacy state read back is marked as having an incomplete wide sample:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:1048
 /** A non-empty state written in the legacy format carries elements that were never offered
 * to a wide set, so no wide estimate of this state (or of anything it is merged into)
 * can be trusted. An empty legacy state contributes nothing and does not spoil a merge.
 */
 if (use_legacy_format && m_size > 0)
 wide_incomplete = true;

That mark is sticky and propagates through merges, because the missing high bits of the already-consumed elements cannot be recovered. The alternative — estimating from a partially-covering wide sample — would silently undercount everything the sample missed, which is far worse than a merely imprecise estimate.

The whole family

uniq is one of several cardinality estimators in ClickHouse, and they make different trade-offs:

function	structure	exact?
`uniqExact`	a full hash set of the values	yes
`uniq`	the thinning set of this article	no, ~1-2% error
`uniqCombined`	small set → hash set → HyperLogLog, switching as it grows	no
`uniqTheta`	a datasketches theta sketch	no, mergeable

uniqCombined is the one that switches representation:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniqCombined.h:112
 // We want to migrate from |HashSet| to |HyperLogLogCounter| when the sizes in memory become almost equal.
 // The size per element in |HashSet| is sizeof(Key)*2 bytes, and the overall size of |HyperLogLogCounter| is 2^K * 6 bits.
 // For Key=UInt32 we can calculate: 2^X * 4 * 2 ≤ 2^(K-3) * 6 ⇒ X ≤ K-4.

and uniqExact keeps the two-level structure of the previous article in reserve, converting when it grows:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniq.h:154
 /// When creating, the hash table must be small.
 static constexpr size_t initial_size_degree = 4;
 using SingleLevelSet = HashSetWithStackMemory<Key, HashCRC32<Key>, initial_size_degree>;
 using TwoLevelSet = TwoLevelHashSet<Key, HashCRC32<Key>>;
 using Set = UniqExactSet<SingleLevelSet, TwoLevelSet>;

Note the “hash table must be small”: the initial buffer lives on the stack of the aggregate function state, so a uniqExact state that never grows past 2^4 slots allocates nothing on the heap at all.

A quick sanity check of the behaviour in SQL:

 sql
 
 
SELECT
 uniqExact(user_id) AS exact,
 uniq(user_id) AS approx,
 uniqCombined(user_id) AS combined
FROM events;

approx will differ from exact by a fraction of a percent on a few million distinct users, and by more on a few hundred million — that is the thinning, and it is the price of a bounded 64 KiB state.

Trade-offs

What the thinning set buys:

A bounded, tiny state. The main set is at most 2^17 × 4 bytes = 512 KiB, and the wide set at most 256 KiB. A uniq state never grows with the cardinality, which matters when the state is merged across a cluster or spilled to disk.
A mergeable state. Merging two uniq states is a union of hash sets plus a max(skip_degree), which is why uniq works with Distributed tables and AggregatingMergeTree.
A principled estimator. The thinning factor is known exactly, so the estimate is a maximum-likelihood estimate rather than a heuristic.
No allocation in the hot loop. The table is a flat array of UInt32, and the batch of eight hashes is computed with instruction-level parallelism.

What it costs:

Precision. The relative error grows with the cardinality, because the sample size stays constant while the multiplier grows. The wide set pushes the useful range from billions to trillions, but the error is still around 1% at the switch-over point.
A hash-quality dependency. The whole scheme assumes the hash is uniform; a bad hash breaks the thinning, because the surviving elements would not be a uniform sample.
Irreversible loss. Once thinned, the dropped hashes are gone. That is why the wide set has to be fed from the first insertion, and why a legacy state can never grow a wide sample.
A 32-bit ceiling without the wide set, which is exactly the bug that issue #6078 was, and why the state format has two versions.

This is a good example of an approximation that is statistically principled rather than ad hoc: the state is small because the algorithm throws away information on purpose, and the estimate is exact given the information that was kept.

Wrapping up

The one thing to remember: uniq() does not estimate by approximating — it estimates by sampling, and it knows the sample rate exactly. That is why the state never grows and why the estimate is a maximum-likelihood one rather than a guess.

The part people get wrong: the state is not just “a hash set”. It is a hash set plus a thinning factor, plus a second sparse 64-bit sample that takes over past ~2 billion distinct values. If you are estimating cardinalities in the trillions, you are living in the wide set’s world.

What to go do about it: compare uniq against uniqExact on your own data before you trust it. If the two agree within a percent, uniq is fine and cheaper. If they do not, and you need exactness, you know now what you are paying.

ClickHouse Decides Which Parts to Merge: A Cost Model for Write Amplification

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse picks merge candidates with a tiny, explicit cost model: score each contiguous range as sum_size / (count - 1.9), pick the cheapest admissible one, and only ever look at the newest 1000 parts so backlogs cannot snowball. The knobs that control it are merge_selector_base and merge_selector_window_size.

A MergeTree table is a set of immutable parts. Every INSERT adds one. Background merges take contiguous runs of parts and rewrite them as one bigger part, which cuts the number of parts a SELECT has to open — and costs a full rewrite of the data involved.

That trade-off has no free lunch, and the two directions pull against each other:

Merge eagerly and the part count stays low, but you rewrite the same bytes over and over. A part merged into a bigger part, which is merged again, is written twice.
Merge lazily and write amplification is low, but the part count climbs, and every query has to open every part.

ClickHouse’s answer is a small, explicit cost model. And — unusually — the model is documented in the source, together with its own limitations. Which means you can read the reasoning, not just guess at it.

Let’s take the score function, the “base of the logarithm” knob, the sliding window that stops merge backlogs from snowballing, and the background scheduler’s exponential backoff.

The design note

The most valuable thing about ClickHouse’s merge selector is that the authors wrote down what they were trying to do — and where they thought they were wrong. The header of SimpleMergeSelector.h is a design document:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:6
/**
We have a set of data parts that is dynamically changing - new data parts are added and there is background merging process.
Background merging process periodically selects continuous range of data parts to merge.

It tries to optimize the following metrics:
1. Write amplification: total amount of data written on disk (source data + merges) relative to the amount of source data.
It can be also considered as the total amount of work for merges.
2. The number of data parts in the set at the random moment of time (average + quantiles).

Also taking the following considerations:
1. Older data parts should be merged less frequently than newer data parts.
2. Larger data parts should be merged less frequently than smaller data parts.
3. If no new parts arrive, we should continue to merge existing data parts to eventually optimize the table.
4. Never allow too many parts, because it will slow down SELECT queries significantly.
5. Multiple background merges can run concurrently but not too many.

It is not possible to optimize both metrics, because they contradict to each other.
To lower the number of parts we can merge eagerly but write amplification will increase.
Then we need some balance between optimization of these two metrics.

But some optimizations may improve both metrics.

For example, we can look at the "merge tree" - the tree of data parts that were merged.
If the tree is perfectly balanced then its depth is proportional to the log(data size),
the total amount of work is proportional to data_size * log(data_size)
and the write amplification is proportional to log(data_size).
If it's not balanced (e.g. every new data part is always merged with existing data parts),
its depth is proportional to the data size, total amount of work is proportional to data_size^2.
*/

That last paragraph is the key insight of the whole design, so read it twice.

The merges form a tree: leaves are inserted parts, internal nodes are merge results. If the tree is balanced, its depth is log_base(total_size) and total write amplification is proportional to that depth. If the tree degenerates into a chain — every new part merged immediately into the current biggest part — the depth is proportional to the data size, and total write amplification is O(data_size). Every byte gets rewritten once per level.

So the selector’s real job is keeping the merge tree balanced. And the knob that controls balance is the arity of the tree: how many parts get merged at once.

The base of the logarithm

That arity is called base, and the header explains how it generalizes to parts of unequal size:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:35
We can also control the "base of the logarithm" - you can look it as the number of data parts
that are merged at once (the tree "arity"). But as the data parts are of different size, we should generalize it:
calculate the ratio between total size of merged parts to the size of the largest part participated in merge.
For example, if we merge 4 parts of size 5, 3, 1, 1 - then "base" will be 2 (10 / 5).

Base of the logarithm (simply called `base` in `SimpleMergeSelector`) is the main knob to control the write amplification.
The more it is, the less is write amplification but we will have more data parts on average.

To fit all the considerations, we also adjust `base` depending on total parts count,
parts size and parts age, with linear interpolation (then `base` is not a constant but a function of multiple variables,
looking like a section of hyperplane).

So the base of a candidate merge is sum_size / max_size: total size of the range divided by the size of its largest part. Merge four parts of sizes 5, 3, 1, 1 and you get a base of 10/5 = 2.

Now merge a huge part with a tiny one. Base is barely above 1 — and that is bad. That is the degenerate chain, the exact shape the selector exists to avoid.

base is exposed as a MergeTree setting, default 5:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1090
 DECLARE(Float, merge_selector_base, 5.0, R"(Affects write amplification of
 assigned merges (expert level setting, don't change if you don't understand
 what it is doing). Works for Simple and StochasticSimple merge selectors
)", 0) \

The admissibility test: is this range allowed at all?

Before any candidate is scored, it has to pass an admissibility test. The test is allow(), and its central quantity is the lowered base — a base adjusted downward depending on how stale the parts are, how many parts the partition has, and how big the range is.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:241
 /// Map size to 0..1 using logarithmic scale
 /// Use log(1 + x) instead of log1p(x) because our sum_size is always integer.
 /// Also log1p seems to be slow and significantly affect performance of merges assignment.
 double size_normalized = mapPiecewiseLinearToUnit(log(1 + sum_size), min_size_to_lower_base_log, max_size_to_lower_base_log);
 /// Calculate boundaries for age
 double min_age_to_lower_base = interpolateLinear(static_cast<double>(settings.min_age_to_lower_base_at_min_size), static_cast<double>(settings.min_age_to_lower_base_at_max_size), size_normalized);
 double max_age_to_lower_base = interpolateLinear(static_cast<double>(settings.max_age_to_lower_base_at_min_size), static_cast<double>(settings.max_age_to_lower_base_at_max_size), size_normalized);
 /// Map age to 0..1
 double age_normalized = mapPiecewiseLinearToUnit(min_age, min_age_to_lower_base, max_age_to_lower_base);
 /// Map partition_size to 0..1
 double num_parts_normalized = mapPiecewiseLinearToUnit(partition_size, static_cast<double>(settings.min_parts_to_lower_base), static_cast<double>(settings.max_parts_to_lower_base));
 /// The ratio should be within [0, 1]
 double combined_ratio = std::min(1.0, age_normalized + num_parts_normalized);

 double lowered_base = interpolateLinear(settings.base, 2.0, combined_ratio);

Three inputs, three effects:

Input	Effect on `lowered_base`
`age_normalized` — how old the youngest part in the range is	Older parts are cheaper to merge (they are not going to be queried as freshly-inserted data), so the base is lowered and the merge happens sooner.
`num_parts_normalized` — how many parts the partition has	A partition with too many parts is a problem for queries, so the base is lowered and merges get more aggressive.
`size_normalized` — how big the range is	Big ranges need to be older before they are merged, because merging them is expensive.

The mapping is deliberately non-linear in size (logarithmic), and the comment explains why log(1+x) rather than log1p(x): log1p is slow enough to matter, and the selector runs on every selection round.

Then the actual test:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:271
 return (sum_size + static_cast<double>(size) * static_cast<double>(settings.size_fixed_cost_to_add)) / (max_size + static_cast<double>(settings.size_fixed_cost_to_add)) >= lowered_base;

Read it as: “is sum_size / max_size at least lowered_base?” — modulo the fixed cost of merging (size_fixed_cost_to_add, default 5 MB) added to both numerator and denominator, which models the fact that merging even two tiny parts costs a fixed amount of work.

Two escape hatches come before it:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:215
 if (settings.min_age_to_force_merge && min_age >= static_cast<double>(settings.min_age_to_force_merge))
 return true;

 if (settings.min_partition_age_to_force_merge && min_partition_age > 0)
 if (min_partition_age >= static_cast<double>(settings.min_partition_age_to_force_merge))
 return true;

If the parts are old enough — or the whole partition is old enough — the range is merged no matter what the base says. This is the safety valve: the cost model can be wrong, but a part that has been sitting there for a day must eventually be merged.

The score function

Now, given that several ranges are admissible, which one do you pick? This is the score() function, and it is where the “cost model” really lives:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:122
 static double score(double count, double sum_size, double sum_size_fixed_cost)
 {
 /** Consider we have two alternative ranges of data parts to merge.
 * Assume time to merge a range is proportional to sum size of its parts.
 *
 * Cost of query execution is proportional to total number of data parts in a moment of time.
 * Let define our target: to minimize average (in time) total number of data parts.
 *
 * Let calculate integral of total number of parts, if we are going to do merge of one or another range.
 * It must be lower, and thus we decide, what range is better to merge.
 *
 * The integral is lower iff the following formula is lower:
 *
 * sum_size / (count - 1)
 *
 * But we have some tunes to prefer longer ranges.
 */
 return (sum_size + sum_size_fixed_cost * count) / (count - 1.9);
 }

That comment is a genuine, if informal, calculus argument. Merge a range of count parts and the part count drops by count - 1 (you replace count parts with one). The merge itself takes time proportional to sum_size. So the “average number of parts over time” is sum_size / (count - 1) — and lower is better. The range with the smallest score is the one to merge.

Now the - 1.9. With a plain - 1 the denominator vanishes for a two-part merge, giving an infinite score. The authors replaced - 1 with - 1.9, which softens the singularity: merging two parts is penalized less than the pure model would suggest, and merging many parts is penalized more. The comment’s own summary is “we have some tunes to prefer longer ranges”.

size_fixed_cost_to_add (5 MB by default) enters the numerator as sum_size + fixed * count — every part in the range carries the fixed cost of being merged, so a range of ten tiny parts is far more expensive than its byte sum suggests.

The candidate enumeration walks every contiguous sub-range of the partition and scores each one:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:108
 double current_score = score(static_cast<double>(end - begin), static_cast<double>(sum_size), static_cast<double>(settings.size_fixed_cost_to_add));

 if (settings.enable_heuristic_to_align_parts
 && static_cast<double>(size_prev_at_left) > static_cast<double>(sum_size) * settings.heuristic_to_align_parts_min_ratio_of_sum_size_to_prev_part)
 {
 double difference = std::abs(log2(static_cast<double>(sum_size) / static_cast<double>(size_prev_at_left)));
 if (difference < settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two)
 current_score *= interpolateLinear(settings.heuristic_to_align_parts_max_score_adjustment, 1,
 difference / settings.heuristic_to_align_parts_max_absolute_difference_in_powers_of_two);
 }

The “align parts” heuristic prefers ranges whose total size is roughly a power of two times the size of the part to their left. Why? Because a balanced merge tree wants each level to double the size. If the part on the left is 100 MB and the candidate range is 100 MB, the range “fits” the tree; at 30 MB or 300 MB the resulting tree is lopsided and the next merge has to rewrite more. The score is scaled down by up to 25% (heuristic_to_align_parts_max_score_adjustment = 0.75) when the sizes sit close to a power of two apart.

The right-tail heuristic

One more adjustment, applied before scoring, and it is the most “heuristic” of them all:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:51
 const bool should_remove_small_parts_at_right = settings.enable_heuristic_to_remove_small_parts_at_right
 && (!settings.merge_selector_min_age_to_disable_right_tail_heuristic
 || min_age < settings.merge_selector_min_age_to_disable_right_tail_heuristic);
 if (should_remove_small_parts_at_right)
 {
 size_t size_delta = 0;
 size_t rows_delta = 0;
 while (end >= begin + 3 && static_cast<double>((end - 1)->size) < settings.heuristic_to_remove_small_parts_at_right_max_ratio * static_cast<double>(sum_size))
 {
 size_delta += (end - 1)->size;
 rows_delta += (end - 1)->rows;
 --end;
 }

 sum_size -= size_delta;
 sum_rows -= rows_delta;
 }

The idea: a range that ends with a couple of very small parts is a bad merge. Those small parts are probably new parts that just arrived, and folding them into a large merge means they sit in the merge queue for a long time. Worse, it means freshly-arrived data is not queryable quickly. So the heuristic trims the right tail of the range — the small parts at the newest end — and scores what remains.

The default ratio is 1%: a part at the right end smaller than 1% of the range’s total gets trimmed, as long as at least three parts remain. And there is an age-based escape hatch (merge_selector_min_age_to_disable_right_tail_heuristic): once the range is old enough the tail is kept, because at that point staleness is worse than freshness.

The sliding window: anti-snowball protection

The single most valuable piece of engineering in the selector is this:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:307
 /// If the parts in the parts vector are sorted by block number,
 /// it may not be ideal to only select parts for merging from the first N ones.
 /// This is because if there are more than N parts in the partition,
 /// we will not be able to assign a merge for newly created parts.
 /// As a result, the total number of parts within the partition could
 /// grow uncontrollably, similar to a snowball effect.
 /// To address this we will try to assign a merge taking into consideration
 /// only last N parts.
 const size_t parts_threshold = settings.window_size;
 size_t begin = 0;
 if (parts_count >= parts_threshold)
 {
 if (settings.enable_stochastic_sliding)
 begin = calculateRangeWithStochasticSliding(parts_count, parts_threshold);
 else
 begin = parts_count - parts_threshold;
 }

window_size defaults to 1000. The comment describes the failure mode it prevents, and it is a real one:

If the parts in the parts vector are sorted by block number, it may not be ideal to only select parts for merging from the first N ones. This is because if there are more than N parts in the partition, we will not be able to assign a merge for newly created parts. As a result, the total number of parts within the partition could grow uncontrollably, similar to a snowball effect.

Picture it: if the selector always preferred the oldest parts, then in a partition with 10 000 parts it would forever be merging parts 1..100 — ancient, useless candidates — while parts 9000..10000, the ones that arrived in the last minute, never get merged. The part count never goes down. The merge queue is permanently busy with work that does not help.

By only ever considering the last N parts, every merge consumes recent parts, so recent parts get compacted and the partition’s part count actually falls. One line of code, and it kills a self-reinforcing failure mode.

The stochastic variant (StochasticSimpleMergeSelector) randomizes the window position, so different replicas do not all merge the same parts:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:275
size_t calculateRangeWithStochasticSliding(size_t parts_count, size_t parts_threshold)
{
 auto mean = static_cast<double>(parts_count);
 std::normal_distribution<double> distribution{mean, mean / 4};

 size_t right_boundary = static_cast<size_t>(distribution(thread_local_rng));
 if (right_boundary > parts_count)
 right_boundary = 2 * parts_count - right_boundary;

 right_boundary = std::max(right_boundary, parts_threshold);

 return right_boundary - parts_threshold;
}

A normal distribution with mean parts_count and standard deviation parts_count / 4, folded back into range by mirroring at parts_count. And note the std::max(right_boundary, parts_threshold) at the end: the window never shrinks below window_size. The anti-snowball guarantee is never traded away for randomization.

The blurry base

StochasticSimpleMergeSelector adds one more trick: instead of a sharp threshold on the base, it uses a probability distribution around the lowered base.

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:256
 if (settings.use_blurry_base)
 {
 double partition_fill_factor = std::max(0., 1 - partition_size / static_cast<double>(settings.parts_to_throw_insert));
 /// Scale factor controls when (relativelty to the number of parts in partition)
 /// do we activate our special algorithm.
 /// With standard parameters the logic kicks in starting from 80% empty factor.
 /// The division by 2 is due to the fact that for normal distribution nearly 95.4%
 /// of all observations fall within two standard deviations.
 double scaling_factor = std::pow(partition_fill_factor, settings.blurry_base_scale_factor) / 2;
 /// The base lower than 1 doesn't make sense, so we try to avoid it.
 std::normal_distribution<double> distribution{lowered_base, (lowered_base - 1) * scaling_factor};
 /// The threshold should be strictly bigger than 1, because we don't allow to merge the part with itself.
 lowered_base = std::min(distribution(thread_local_rng), std::max(1.01, lowered_base));
 }

Instead of “merge if base ≥ 5.0”, it says “merge if base ≥ a randomly-drawn threshold around 5.0”. A merge whose base is 4.9 might happen this round and not the next. The result: the distribution of merges is smoother, no sharp cliff at a particular base value, and — critically — the merge decisions of different replicas do not line up.

The width of the distribution grows with the emptiness of the partition: partition_fill_factor is 0 when the partition is full (parts_to_throw_insert parts) and approaches 1 when it is empty. blurry_base_scale_factor (default 42) makes the effect very nonlinear, so the blur only kicks in when the partition is nearly empty — exactly when you want merges to be infrequent and gentle.

The final std::min(distribution(...), std::max(1.01, lowered_base)) is a careful clamp: the drawn threshold can be lower than the base, making the merge more eager, but never lower than 1.01, because a base ≤ 1 would mean merging a part with itself.

Picking the winner

The scored candidates go into a max-heap, and the best admissible, non-overlapping one wins:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.cpp:142
 std::optional<PartsRange> buildMergeRange(const MergeConstraint & constraint)
 {
 constexpr static auto range_compare = [](const ScoredRange & lhs, const ScoredRange & rhs)
 {
 /// If the ranges have the same score, use the range that is most likely to have the lower merge level -
 /// the right one, according to the block number sorting.
 return lhs.score > rhs.score || (lhs.score == rhs.score && lhs.range_begin < rhs.range_begin);
 };

Note the tie-break: “the right one”, i.e. the newer parts. That is the opposite of what a naive reading of “oldest first” would suggest, and it is consistent with the sliding window.

The disjoint_set.addRangeIfPossible call in the pop loop enforces that two concurrent merges never claim overlapping ranges of parts. The disjoint set is implemented as a std::set of range boundaries per independent range, with an upper_bound lookup:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/DisjointPartsRangesSet.cpp:113
bool DisjointPartsRangesSet::addRangeIfPossible(RangesIterator range_it, PartsIterator range_begin, PartsIterator range_end)
{
 chassert(sanityCheck(uncovered_ranges, range_it, range_begin, range_end));

 auto & sorted_ranges = disjoint[range_it.base()];
 PartsRangeBoundaries range_boundaries{std::move(range_begin), std::move(range_end)};

 if (!isDisjoint(range_boundaries, sorted_ranges))
 return false;

 sorted_ranges.insert(std::move(range_boundaries));
 return true;
}

From parts to a merge: the whole pipeline

The selector is one step of a longer pipeline. The entry point is MergeTreeDataMergerMutator::selectPartsToMerge:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:328
std::expected<MergeSelectorChoices, SelectMergeFailure> MergeTreeDataMergerMutator::selectPartsToMerge(
 const PartsCollectorPtr & parts_collector,
 const MergePredicatePtr & merge_predicate,
 const MergeSelectorApplier & selector,
 const std::optional<PartitionIdsHint> & partitions_hint)
{
 const auto context = data.getContext();
 const auto settings = data.getSettings();
 const auto metadata_snapshot = data.getInMemoryMetadataPtr(context, false);
 const auto storage_policy = data.getStoragePolicy();
 const time_t current_time = std::time(nullptr);
 const bool can_use_ttl_merges = !ttl_merges_blocker.isCancelled();
 LogSeriesLimiter series_log(log, 1, /*interval_s_=*/60 * 30);

 auto collected = collectAllPossibleRanges(parts_collector, metadata_snapshot, storage_policy, current_time, partitions_hint, series_log);

Three phases:

collect — find the ranges of parts that could be merged (same partition, compatible schema, not already covered by a running merge);
split by merge predicate — cut those ranges wherever a merge is forbidden (a part that overlaps a running merge, a part with a different schema version, and so on);
choose — run the selector’s cost model over what is left.

The LogSeriesLimiter is a nice touch: the trace log of this function is rate-limited to one message per 30 minutes, because the function runs on every selection round and can produce a lot of output.

The background scheduler’s backoff

Selection is cheap; the merge is not. The background scheduler that runs the selection loop uses an exponential backoff when there is nothing to do:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:62
void BackgroundJobsAssignee::postpone()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 no_work_done_count += 1;
 double random_addition = std::uniform_real_distribution<double>(0, sleep_settings.task_sleep_seconds_when_no_work_random_part)(rng);

 size_t next_time_to_execute = static_cast<size_t>(
 1000 * (data.getBiasBackoffSeconds() + std::min(
 sleep_settings.task_sleep_seconds_when_no_work_max,
 sleep_settings.thread_sleep_seconds_if_nothing_to_do * std::pow(sleep_settings.task_sleep_seconds_when_no_work_multiplier, no_work_done_count))
 + random_addition));

 holder->scheduleAfter(next_time_to_execute, false);
}

with the settings:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.h:18
struct BackgroundTaskSchedulingSettings
{
 double thread_sleep_seconds_random_part = 1.0;
 double thread_sleep_seconds_if_nothing_to_do = 0.1;
 double task_sleep_seconds_when_no_work_max = 600;
 /// For exponential backoff.
 double task_sleep_seconds_when_no_work_multiplier = 1.1;

 double task_sleep_seconds_when_no_work_random_part = 1.0;

So: start at 0.1 s, multiply by 1.1 per idle round, cap at 600 s, plus up to 1 s of random jitter. An idle table’s selection loop slows to one attempt every ten minutes after about 130 idle rounds, and the jitter stops a fleet of servers from synchronizing their wake-ups.

And when work does appear, the backoff is unwound geometrically rather than reset:

 cpp
 
 
// src/Storages/MergeTree/BackgroundJobsAssignee.cpp:48
void BackgroundJobsAssignee::trigger()
{
 std::lock_guard lock(holder_mutex);

 if (!holder)
 return;

 /// Do not reset backoff factor if some task has appeared,
 /// but decrease it exponentially on every new task.
 no_work_done_count /= 2;
 /// We have background jobs, schedule task as soon as possible
 holder->schedule();
}

no_work_done_count /= 2 rather than = 0. Halving means that after one new task the sleep time is back to roughly its previous value, and it takes several rounds of activity to fully unwind. A table that receives one part per minute therefore does not flip between “poll every 0.1 s” and “poll every 600 s” — it settles at a rate proportional to the arrival rate.

Here’s the thing: that is genuinely nice control theory hiding inside what looks like three lines of code.

The whole picture

 text
 
 
 parts in a partition (by block number)
 ┌────┬────┬────┬────┬────┬──────────────┬────┬────┐
 │ p1 │ p2 │ p3 │ p4 │ p5 │ ... │pn-1│ pn │
 └────┴────┴────┴────┴────┴──────────────┴────┴────┘
 ↑ ↑
 └──────── sliding window (last N=1000) ─────┘
 (only these are eligible)

 for each contiguous sub-range of the window:
 base = sum_size / max_size
 if base < lowered_base(age, partition_size, range_size):
 skip
 score = (sum_size + 5MB * count) / (count - 1.9)
 trim small parts off the right tail
 adjust score for power-of-two alignment with the part on the left

 pick the lowest-scoring admissible range that does not overlap
 an already-assigned merge

What this buys you, and what it costs

It buys you a merge tree that stays balanced, which is what keeps write amplification at O(log n) instead of O(n). The base knob is the lever, and the whole selector is a machine for keeping base near its target.

It buys you a system that self-corrects. When a partition fills up, num_parts_normalized rises, the base is lowered, and merges become more eager — without anybody tuning anything. When a partition sits idle, the opposite happens.

It buys you an anti-snowball guarantee. The sliding window is the difference between a table whose part count stabilizes and one that grows without bound.

It costs you a cost model that is admittedly approximate. The header comment says so:

 cpp
 
 
// src/Storages/MergeTree/Compaction/MergeSelectors/SimpleMergeSelector.h:57
It's still unclear if this algorithm is good or optimal at all. It's unclear if this algorithm is using the optimal coefficients.

and it describes how the coefficients were tuned (an insert/merge simulator, replayed against production system.part_log data), and how the research found slightly better coefficients that were not used because the test data’s representativeness was in question.

It costs you a - 1.9. The score function’s singularity is papered over with a constant rather than derived. It is a fudge factor, honestly labelled as one.

It costs you a lot of tuning surface. merge_selector_base, merge_selector_window_size, parts_to_throw_insert, max_parts_to_merge_at_once, min_parts_to_merge_at_once, merge_selector_small_parts_*, the align-parts heuristic, the right-tail heuristic, the blurry base, the stochastic sliding window. Every one of these is a lever that was pulled for a reason, and the reason is usually written in a comment somewhere in SimpleMergeSelector.h.

Seeing it in action

The merge list shows what was selected and why:

 sql
 
 
SELECT
 database,
 table,
 elapsed,
 progress,
 num_parts,
 formatReadableSize(total_size_bytes_compressed) AS size,
 formatReadableSize(memory_usage) AS mem,
 result_part_name
FROM system.merges
ORDER BY elapsed DESC;

And the parts-per-partition histogram is the metric the selector is ultimately optimizing:

 sql
 
 
SELECT
 database,
 table,
 partition,
 count() AS parts,
 sum(rows) AS rows,
 formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active
GROUP BY database, table, partition
ORDER BY parts DESC
LIMIT 20;

A partition with hundreds of parts means the selector is not keeping up — usually because the merge pool is saturated, or because the parts are too large to merge under parts_to_throw_insert pressure, or because merge_selector_base is set too high for the workload.

And if you want to watch the cost model work, the trace log has it:

 sql
 
 
SET log_level = 'trace';
SELECT count() FROM hits WHERE URL LIKE '%very-selective-prefix%';

The MergeTreeDataMergerMutator::selectPartsToMerge trace messages show which ranges were considered and which one was chosen.

Wrapping up

The one thing to remember: the merge tree is a tree, and a balanced tree costs O(log n) to write while a chain costs O(n). merge_selector_base is the knob that decides which one you get.

The part people get wrong: the sliding window is not an optimization, it is a correctness guard. Without it, a busy partition can keep a permanent merge queue full of work that never reduces the part count.

What to go do about it: run the parts-per-partition query against your biggest tables. If a partition is crawling toward parts_to_throw_insert, the merge pool is the bottleneck, not the selector — and no amount of knob twiddling will fix that.

Posts on Opensource internals

How ClickHouse's Sparse Column Stores Only the Rows That Differ

The layout

The constructor enforces the invariant

Inserting

The iterator: amortized O(1) sequential access

On disk: the offsets become run-length groups

Decoding it

How the encoding gets chosen

The three substreams

Writing

The query layer wants to peel the sparse wrapper off

The other side of the coin: the null map

Seeing it in SQL

Trade-offs

How ClickHouse Stores a Column: PODArray, the Typed Array, and the Padding Trick

The container

Noncopyable

Three pointers, not a size and a capacity

The empty array points at static memory

The padding trick

Left padding: the -1 element

Right padding: SIMD may write past the end

The growth policy

reserve: round up to a power of two

reserve_exact: exactly what was asked for

push_back: double the allocation

The “shrink if less than half used” trick

ColumnVector: the typed array

insertRangeFrom: memcpy, not a loop

insertRawUninitialized: writing into the column’s memory directly

ColumnString: offsets plus a flat character buffer

Bulk insertion

ColumnFixedString: the inline variant

ColumnNullable: the null map as a separate column

Trade-offs

ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

The layout

Global vs local order

Reading a row

Building the offsets

Insertion keeps the two arrays in sync

The insert path

Serialization: compact discriminators

Dynamic: the open-ended case

The insert path

Adding a variant

Seeing it in SQL

Trade-offs

Wrapping up

ClickHouse's Query Result Cache: A Hash of the AST, Not the Text

Why caching results is hard to get right

The key: a hash of the AST, not of the text

What else goes into the hash

The equality of two keys

What may be cached at all

The cache itself

The two backends

Writing: many reasons to refuse

Reading: access control

The on-disk backend

The key

The entry format

Probing and writing

Using it

Trade-offs

Wrapping up

ClickHouse's Query Pipeline: Ports, Status Codes and the Executing Graph

The building block: IProcessor

The port: a mailbox with the flags in the pointer

prepare() and work()

Version counters: how a processor learns that a neighbour changed

The ExecutingGraph: nodes, edges, and who owns what

The graph can change shape at runtime

Trade-offs

Wrapping up

ClickHouse's Query Condition Cache: Skip Granules You Don't Need to Read

What the cache actually stores

The key: three components, hashed

The safety property: everything starts as “may match”

Left padding: the `-1` element

`reserve`: round up to a power of two

`reserve_exact`: exactly what was asked for

`push_back`: double the allocation

`ColumnVector`: the typed array

`insertRangeFrom`: memcpy, not a loop

`insertRawUninitialized`: writing into the column’s memory directly

`ColumnString`: offsets plus a flat character buffer

`ColumnFixedString`: the inline variant

`ColumnNullable`: the null map as a separate column

The building block: `IProcessor`

`prepare()` and `work()`

The `ExecutingGraph`: nodes, edges, and who owns what

`PartialSortingTransform`: sort the chunk, throw away the rest

`FinishSortingTransform`: merging streams that are already sorted

`MergeSortingTransform`: accumulate, remerge, spill, merge