Aggregation on Opensource internals

ClickHouse Splits GROUP BY into 256 Buckets for Parallel Merges

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: A GROUP BY needs a serial merge at the end — and that’s where the memory high-water mark lives. ClickHouse splits the key space into 256 buckets by the top bits of the hash, so the merge becomes 256 independent, lock-free, parallel jobs.

A GROUP BY in ClickHouse is usually computed by several threads at once, each with its own hash table, and the tables are then merged.

Merging N hash tables of M entries each is not something you want to do on one thread. It’s a serial phase in an otherwise parallel query, and it’s the phase where the memory high-water mark is reached.

The way out is to make the merge divisible. If the keys are partitioned into a fixed number of buckets by a deterministic function of the key, then two tables can be merged bucket by bucket, and different buckets can be merged by different threads with no coordination at all: bucket 17 of table A can only collide with bucket 17 of table B.

That’s what TwoLevelHashTable in src/Common/HashTable/TwoLevelHashTable.h does, and it’s what the group_by_two_level_threshold setting controls.

The structure

The two-level table is literally an array of ordinary hash tables:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:9
/** Two-level hash table.
 * Represents 256 (or 1ULL << BITS_FOR_BUCKET) small hash tables (buckets of the first level).
 * To determine which one to use, one of the bytes of the hash function is taken.
 *
 * Usually works a little slower than a simple hash table.
 * However, it has advantages in some cases:
 * - if you need to merge two hash tables together, then you can easily parallelize it by buckets;
 * - delay during resizes is amortized, since the small hash tables will be resized separately;
 * - in theory, resizes are cache-local in a larger range of sizes.
 *
 * With `BITS_FOR_BUCKET = 0` there is a single bucket: routing folds to a constant.
 * Lookups and inserts compile down to what the single-level table does.
 *
 * Buckets share no state.
 * Threads may fill different buckets at the same time when every bucket is written under its own lock.
 */

The bucket index comes out of the hash:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:58
 static constexpr UInt32 NUM_BUCKETS = 1ULL << BITS_FOR_BUCKET;
 static constexpr UInt32 MAX_BUCKET = NUM_BUCKETS - 1;

 size_t hash(const Key & x) const { return Hash::operator()(x); }

 /// NOTE Bad for hash tables with more than 2^32 cells.
 static size_t getBucketFromHash(size_t hash_value) { return (hash_value >> (32 - BITS_FOR_BUCKET)) & MAX_BUCKET; }

BITS_FOR_BUCKET defaults to 8, so 256 buckets, and the bucket is taken from the high bytes of the 64-bit hash:

 text
 
 
 64-bit hash
 |<-------------------------->|
 | bucket | sub-table index |
 | 8 bits | (mask & (size-1)) |
 +-----------+---------------------+
 | 0..255 | depends on the bucket's own size |
 +-----------+---------------------+

 hash >> 24 & 255 -> which of the 256 sub-tables
 hash & (2^k - 1) -> which cell inside that sub-table

The choice of the high bits is not arbitrary, and it’s the subtle part of the whole design. The single-level table uses the low bits of the hash to index its cells:

 cpp
 
 
// src/Common/HashTable/HashTable.h:238
 /// From the hash value, get the cell number in the hash table.
 size_t place(size_t x) const { return x & mask(); }

If the bucket were also selected from the low bits, the two uses would collide. Every bucket would receive keys whose low bits are a fixed value, and the sub-table’s own probe would be confined to a single cell. The sub-tables would degenerate into 256 linked lists.

Taking the bucket from the top of the hash and the cell from the bottom keeps the two independent — which is exactly what a good hash function provides.

The default bucket count is 8 bits, and the comment “Bad for hash tables with more than 2^32 cells” is about the other direction: the shift hash_value >> (32 - BITS_FOR_BUCKET) throws away everything above bit 32, so the bucket selection only sees 32 bits of the hash. For a table with more than 2^32 cells, the top bits would carry real information and throwing them away would lose it.

The sub-tables are ordinary tables

Each bucket is a plain HashTable, so all the machinery from the previous article — zeroed cells, linear probing, Robin Hood ordering — applies inside a bucket. The two-level table only adds the routing:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:316
 template <typename KeyHolder>
 void ALWAYS_INLINE emplace(KeyHolder && key_holder, LookupResult & it, bool & inserted, size_t hash_value)
 {
 size_t buck = getBucketFromHash(hash_value);
 impls[buck].emplace(key_holder, it, inserted, hash_value);
 }

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:333
 LookupResult ALWAYS_INLINE find(Key x, size_t hash_value)
 {
 size_t buck = getBucketFromHash(hash_value);
 return impls[buck].find(x, hash_value);
 }

and the storage is a plain array of sub-tables:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:100
 Impl impls[NUM_BUCKETS];

 /// `bucket_cells_prefix[b]` is the number of cells in the buckets before `b`. See `computeBucketPrefix`.
 std::vector<size_t> bucket_cells_prefix;

Impl impls[NUM_BUCKETS] is 256 sub-tables by value, not by pointer. Each sub-table owns its buffer, so the whole structure is 256 small allocations plus the bookkeeping of each.

How a bucket grows

The buckets do grow, but they grow in a different shape from the single-level table. The grower used by the two-level table caps the quadratic-growth phase at 2^15 cells and switches to the Robin Hood mode above that:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:26
/// The buckets grow by 4 times while they are small, and switch to the Robin Hood mode when they become large,
/// see `HashTableGrowerWithPrecalculationAndRobinHood`.
template <size_t initial_size_degree = 8, size_t robin_hood_size_degree = 16>
using TwoLevelHashTableGrower = HashTableGrowerWithPrecalculationAndRobinHood<initial_size_degree, 15, robin_hood_size_degree>;

The template parameters are initial_size_degree = 8, max_size_degree_quadratic = 15, robin_hood_size_degree = 16. So:

up to 2^15 cells per bucket, the sub-table quadruples its size on overflow (a degree increment of two);
from 2^15 to 2^16 it doubles;
above 2^16 cells per bucket it switches to the Robin Hood mode with a 90% load factor, and from then on it keeps doubling like any other table — but at 90% load instead of 50%.

The effect of the switch is memory, not capacity. A bucket that grows large stops paying the factor-of-two penalty of the 0.5 load factor, because the same number of keys now needs half as many cells.

That’s also what makes the external-memory path possible: when the aggregation spills to disk, it spills bucket by bucket, and each bucket’s data is self-contained.

Resizing is amortised

A single-level table resizes by re-inserting every element. At the moment of the resize, the whole table has to be walked and every cell re-placed.

In a two-level table, each bucket resizes on its own, and since the buckets are filled roughly uniformly, the individual resizes happen at different times. The total work is the same, but it’s spread out, and — crucially — the pause is per-bucket rather than table-wide.

The comment in the header says it exactly:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:14
 * - delay during resizes is amortized, since the small hash tables will be resized separately;

When is the two-level table used?

The decision is a setting, and it’s a pair of settings now — one counting keys, one counting bytes:

 cpp
 
 
// src/Core/Settings.cpp:1500
 DECLARE(UInt64, group_by_two_level_threshold, 100000, R"(
From what number of keys, a two-level aggregation starts. 0 - the threshold is not set.
)", 0) \
 DECLARE(UInt64, group_by_two_level_threshold_bytes, 50000000, R"(
From what size of the aggregation state in bytes, a two-level aggregation begins to be used. 0 - the threshold is not set. Two-level aggregation is used when at least one of the thresholds is triggered.
)", 0) \

So by default a GROUP BY switches to the two-level structure once it holds 100 000 keys, or once its state grows to 50 MB — whichever happens first. Both thresholds can be forced from SQL:

 sql
 
 
SELECT user_id, count()
FROM events
GROUP BY user_id
SETTINGS group_by_two_level_threshold = 1000000,
 group_by_two_level_threshold_bytes = 50000000;

Setting either threshold to 0 disables that trigger — which is exactly what the pipeline does when a two-level aggregation would be useless.

The predicate that implements this lives in src/Interpreters/Aggregator.cpp:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:118
bool worthConvertToTwoLevel(
 size_t group_by_two_level_threshold, size_t result_size, size_t group_by_two_level_threshold_bytes, auto result_size_bytes)
{
 // params.group_by_two_level_threshold will be equal to 0 if we have only one thread to execute aggregation (refer to AggregatingStep::transformPipeline).
 return (group_by_two_level_threshold && result_size >= group_by_two_level_threshold)
 || (group_by_two_level_threshold_bytes && result_size_bytes >= static_cast<Int64>(group_by_two_level_threshold_bytes));
}

Note the && in each disjunct: a threshold of 0 disables that particular trigger, which is why the settings can be turned off individually.

The conversion itself happens during the aggregation, when a block has been added and the table is re-examined:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:2668
 bool worth_convert_to_two_level = worthConvertToTwoLevel(
 params.group_by_two_level_threshold, result_size, params.group_by_two_level_threshold_bytes, result_size_bytes);

 /** Converting to a two-level data structure.
 * It allows you to make, in the subsequent, an effective merge - either economical from memory or parallel.
 */
 if (result.isConvertibleToTwoLevel() && worth_convert_to_two_level)
 result.convertToTwoLevel();

and the conversion itself rebuilds the variant with the two-level type:

 cpp
 
 
// src/Interpreters/AggregatedDataVariants.cpp:263
void AggregatedDataVariants::convertToTwoLevel()
{
 ProfileEvents::increment(ProfileEvents::AggregationConvertedToTwoLevel);

 if (aggregator)
 LOG_TRACE(aggregator->log, "Converting aggregation data to two-level.");

 switch (type)
 {
#define M(NAME) \
 case Type::NAME: \
 NAME ## _two_level = std::make_unique<decltype(NAME ## _two_level)::element_type>(*(NAME)); \
 (NAME ## _two_level)->top_k_heap = std::move((NAME)->top_k_heap); \
 (NAME).reset(); \
 type = Type::NAME ## _two_level; \
 break;

 APPLY_FOR_VARIANTS_CONVERTIBLE_TO_TWO_LEVEL(M)

#undef M

 default:
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Wrong data variant passed.");
 }
}

The TwoLevelHashTable has a constructor that takes a single-level table and distributes it, which is what makes the conversion a one-liner:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:116
 template <typename Source>
 requires(!std::is_arithmetic_v<Source>)
 explicit TwoLevelHashTable(const Source & src)
 {
 typename Source::const_iterator it = src.begin();

 /// It is assumed that the zero key (stored separately) is first in iteration order.
 if (it != src.end() && it.getPtr()->isZero(src))
 {
 insert(it->getValue());
 ++it;
 }

 for (; it != src.end(); ++it)
 {
 const Cell * cell = it.getPtr();
 size_t hash_value = cell->getHash(src);
 size_t buck = getBucketFromHash(hash_value);
 impls[buck].insertUniqueNonZero(cell, hash_value);
 }
 }

The zero key is handled explicitly, because it lives outside the buffer and doesn’t follow the bucket rule — it goes to bucket 0.

The merge is the point

Now the reason the whole thing exists. The aggregation pipeline decides whether a two-level aggregation is even allowed:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:503
 bool allow_to_use_two_level_group_by = pipeline.getNumStreams() > 1 || params.max_bytes_before_external_group_by != 0;

Two conditions: either there’s more than one stream (so the merge is worth parallelising), or the query is allowed to spill to disk (so the bucket structure is needed to spill incrementally). If neither holds, the thresholds are zeroed out and the aggregation stays single-level:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:520
 if (!allow_to_use_two_level_group_by)
 {
 params.group_by_two_level_threshold = 0;
 params.group_by_two_level_threshold_bytes = 0;
 }

When the two-level table is used, the merge stage is a source that hands out one bucket at a time, and the pipeline runs several of them in parallel:

 cpp
 
 
// src/Processors/Transforms/AggregatingTransform.cpp:208
/// Worker which merges buckets for two-level aggregation.
/// Atomically increments bucket counter and returns merged result.
class ConvertingAggregatedToChunksWithMergingSource final : public ISource
{
public:
 static constexpr UInt32 NUM_BUCKETS = 256;

The worker takes the next bucket with an atomic counter and merges it across all the per-thread tables:

 cpp
 
 
// src/Processors/Transforms/AggregatingTransform.cpp:272
 Chunk generate() override
 {
 UInt32 bucket_num = shared_data->next_bucket_to_merge.fetch_add(1);

 if (bucket_num >= NUM_BUCKETS)
 {
 data.reset();
 return {};
 }

The atomic counter is the entire coordination mechanism. No lock, no work queue, no barrier. A worker that finishes bucket 3 immediately grabs bucket 4, and the workers end up balanced by the cost of the buckets they happen to pick up. Since the buckets partition the key space, no two workers ever touch the same key.

The actual merge of one bucket, in Aggregator::mergeBucketImpl, is “for each thread’s table, merge its bucket into the first thread’s bucket”:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5200
void NO_INLINE Aggregator::mergeBucketImpl(
 ManyAggregatedDataVariants & data, Int32 bucket, Arena * arena, std::atomic<bool> & is_cancelled) const
{
 /// We merge all aggregation results to the first.
 AggregatedDataVariantsPtr & res = data[0];

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5213
 auto & dst = getDataVariant<Method>(*res).data.impls[bucket];

Note that the bucket is an index into the impls[] array of the two-level table — the same array the routing used. That’s the whole contract: the bucket is a function of the key, therefore the bucket is a partition of the key space.

The same trick appears in the uniqExact aggregate function, which keeps a single-level set and converts it to a two-level one when it grows:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:386
 static bool worthConvertingToTwoLevel(size_t size) { return size > 100'000; }

and merges it bucket-wise in parallel:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:225
 auto thread_func = [&two_level_ptrs, &first_two_level, next_bucket_to_merge, &is_cancelled]()
 {
 while (true)
 {
 if (is_cancelled.load(std::memory_order_seq_cst))
 return;

 const auto bucket = next_bucket_to_merge->fetch_add(1);
 if (bucket >= NUM_BUCKETS)
 return;

 for (size_t j = 1; j < two_level_ptrs.size(); ++j)
 {
 if (is_cancelled.load(std::memory_order_seq_cst))
 return;

 first_two_level.impls[bucket].merge(two_level_ptrs[j]->impls[bucket]);
 }
 }
 };

with the same bounded-threads trick:

 cpp
 
 
// src/AggregateFunctions/UniqExactSet.h:246
 const size_t max_threads_to_enqueue = std::min<size_t>(thread_pool.getMaxThreads(), NUM_BUCKETS);

There’s no point in queueing more workers than there are buckets.

The string flavour

Strings get their own two-level table, because the string table isn’t a HashTable but a composite of five sub-tables (one per key size class). The routing therefore has to pick both the bucket and the size class:

 cpp
 
 
// src/Common/HashTable/TwoLevelStringHashTable.h:77
 /// The bucket-aware size-class dispatch: the shared implementation in `StringHashTable`
 /// classifies and packs the key and computes its hash; the resolver here picks the bucket
 /// from that hash and hands out the bucket's submap of the right class.
 template <typename Self, typename Func, typename KeyHolder>
 static auto ALWAYS_INLINE dispatch(Self & self, KeyHolder && key_holder, Func && func)
 {
 return Impl::dispatchOnKeyClass(
 [&](auto key_class, size_t hash_value) -> auto &
 { return self.impls[getBucketFromHash(hash_value)].template submapForClass<decltype(key_class)::value>(); },
 std::forward<KeyHolder>(key_holder),
 StringHashTableHash{},
 std::forward<Func>(func));
 }

256 buckets times 5 size classes is 1280 sub-tables. That’s a lot of bookkeeping for a table that usually isn’t huge, which is why the string tables are only converted to two-level when the aggregation really needs it.

Offsets across buckets

One thing the two-level table has to solve is numbering the cells globally. The aggregation code refers to cells by an offset into the whole table, and with 256 separate buffers there’s no single array to index into.

The solution is a prefix sum over the buckets:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:435
 /// Prefix sums that `offsetInternal` uses to number cells across all buckets.
 /// Call it after the last insert and before reading offsets, never while another thread reads them.
 void computeBucketPrefix()
 {
 bucket_cells_prefix.assign(NUM_BUCKETS, 0);
 size_t run = 0;
 for (UInt32 i = 0; i < NUM_BUCKETS; ++i)
 {
 bucket_cells_prefix[i] = run;
 run += impls[i].getBufferSizeInCells();
 }
 }

and the offset itself:

 cpp
 
 
// src/Common/HashTable/TwoLevelHashTable.h:448
 /// Number of the cell over all buckets. The zero cell gets 0. Any other cell gets its position in the
 /// concatenated bucket buffers plus one. So the number fits an array of `getBufferSizeInCells() + 1`.
 size_t offsetInternal(ConstLookupResult ptr) const
 {
 if constexpr (NUM_BUCKETS == 1)
 return impls[0].offsetInternal(ptr);

 const size_t buck = getBucketFromHash(ptr->getHash(*this));
 if (ptr->isZero(impls[buck]))
 return 0;

 chassert(!bucket_cells_prefix.empty(), "computeBucketPrefix must run before an offset is read");
 return bucket_cells_prefix[buck] + (ptr - impls[buck].buf) + 1;
 }

The “+1” is the zero key’s slot, which is numbered 0.

Trade-offs

What the two-level table buys:

A parallelisable merge. The merge of N per-thread tables becomes 256 independent merges with an atomic counter and no locks. The serial tail of a parallel GROUP BY mostly disappears.
Incremental spilling. Because a bucket is self-contained, the aggregation can write one bucket to disk and continue, and the merge can read buckets back one at a time.
Amortised resize pauses. The expensive rehash is per-bucket, so no single pause is as long.
A memory-efficient large bucket. The Robin Hood switch at 2^16 cells means a bucket that grows large halves its memory overhead per key, at the cost of a little CPU for the chain ordering.

What it costs:

One extra indirection on every probe. The single-level table computes hash & mask; the two-level table computes a shift, an array index, and then the mask. The header admits it: “Usually works a little slower than a simple hash table.”
256 allocations instead of one. Each sub-table has its own buffer, its own grower, its own size counter.
A weaker guarantee for large tables. The bucket is selected from 32 bits of the hash, so a table with more than 2^32 cells is explicitly out of scope.
Complexity at the boundaries. The zero key doesn’t belong to a bucket; the prefix sums must be recomputed after growth; the string variant has to route on two axes at once.

This is a good example of a structural optimisation. Rather than making the hash table faster, ClickHouse changes the shape of the data so that the expensive part of the query — the merge — can be decomposed into 256 embarrassingly parallel pieces.

Wrapping up

Here’s the thing: the whole design rests on one sentence. The bucket is a function of the key, so the bucket is a partition of the key space. Every other property — parallel merge, incremental spill, amortised resize — falls out of that.

What to go do about it: if you have a GROUP BY over a high-cardinality column that takes a long time to finish but not a long time to start, you’re probably hitting the serial merge. Check AggregationConvertedToTwoLevel in system.events — if it’s zero and your key count is above 100 000, the conversion isn’t happening, and that’s your smoking gun.

And if you want to tune it, group_by_two_level_threshold is the knob. Lower it and the merge gets parallel sooner; the cost is one extra indirection per probe on the smaller table.

ClickHouse Estimates Distinct Counts by Thinning a Hash Table

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: uniq() is a hash table that deliberately throws data away. When it fills up, it halves its sample rate, keeps the survivors, and multiplies the count back up by the thinning factor. The state stays at a few hundred KiB no matter how many distinct values you feed it.

uniq() is one of the most-used aggregate functions in ClickHouse, and one of the most surprising. It is not exact. It is not a HyperLogLog. It is a hash table that throws data away on purpose, thinning itself as it fills up, and then multiplies back up by the thinning factor to estimate the cardinality.

The algorithm behind it is BJKST (Bretag, Karp, Kutz, Schlossmann — “near-optimal algorithms for estimating the number of distinct elements in a stream of data”). The implementation lives in src/AggregateFunctions/UniquesHashSet.h, and it is one of the more elegant pieces of code in the aggregation framework: a fixed-size table, a single integer of state, and a mathematically exact estimator of what the sample means.

The scheme

The header states the general recipe for approximate aggregation:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:27
/** Approximate calculation of anything, as usual, is constructed according to the following scheme:
 * - some data structure is used to calculate the value of X;
 * - Not all values are added to the data structure, but only selected ones (according to some selectivity criteria);
 * - after processing all elements, the data structure is in some state S;
 * - as an approximate value of X, the value calculated according to the maximum likelihood principle is returned:
 * at what real value X, the probability of finding the data structure in the obtained state S is maximal.
 */

And then the specific recipe:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:38
/** Very simple hash-set for approximate number of unique values.
 * Works like this:
 * - you can insert UInt64;
 * - before insertion, first the hash function UInt64 -> UInt32 is calculated;
 * - the original value is not saved (lost);
 * - further all operations are made with these hashes;
 * - hash table is constructed according to the scheme:
 * - open addressing (one buffer, position in buffer is calculated by taking remainder of division by its size);
 * - linear probing (if the cell already has a value, then the cell following it is taken, etc.);
 * - the missing value is zero-encoded; to remember presence of zero in set, separate variable of type bool is used;
 * - buffer growth by 2 times when filling more than 50%;
 * - if the set has more than UNIQUES_HASH_MAX_SIZE elements, then all the elements are removed from the set,
 * not divisible by 2, and then all elements that do not divide by 2 are not inserted into the set;
 * - if the situation repeats, then only elements dividing by 4, etc., are taken.
 * - the size() method returns an approximate number of elements that have been inserted into the set;
 * - there are methods for quick reading and writing in binary and text form.
 *
 * A 32-bit hash cannot distinguish more than a few billion elements: the estimate degrades quickly
 * beyond a billion, and above ~30 billion the correction formula even resulted in undefined behavior
 * (issue #6078). To keep the estimate accurate up to trillions, in addition to the main set a small
 * "wide" set of full 64-bit hashes (XOR-ed with a constant, see `wideHash`) is maintained (state version 1).
 * It is thinned out aggressively from the very beginning - only values with UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE
 * zero low-order bits enter it - so it stays empty or tiny for the states where the main set is accurate
 * (about 4 values per million distinct elements), and it is bounded by UNIQUES_HASH_SET_WIDE_MAX_SIZE
 * values thinned out further. Once the wide set has enough values for its own estimate to be reliable
 * (which happens at about two billion elements), size() switches to it.
 *
 * The wide set has to be maintained from the first insertion: the high bits of the hashes
 * of the already consumed elements cannot be recovered later. In particular, a state written
 * in the legacy format (state version 0) carries no wide set, and its elements were never
 * offered to one. Such a state is marked as having an incomplete wide sample: it drops its own
 * wide set, never starts a new one, and propagates the mark through merges, so that size()
 * of anything it is merged into falls back to the legacy 32-bit estimator. Estimating from
 * a wide set that covers only a part of the elements would undercount the rest entirely,
 * which is much worse than the degrading precision of the 32-bit estimate.
 */

That comment is the whole algorithm. Read it twice: the key insight is that the set does not grow without bound — it changes what it accepts.

The constants

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:75
/// The maximum degree of buffer size before the values are discarded
#define UNIQUES_HASH_MAX_SIZE_DEGREE 17

/// The maximum number of elements before the values are discarded
#define UNIQUES_HASH_MAX_SIZE (1ULL << (UNIQUES_HASH_MAX_SIZE_DEGREE - 1))

/** The number of least significant bits used for thinning. The remaining high-order bits are used to determine the position in the hash table.
 * (high-order bits are taken because the younger bits will be constant after dropping some of the values)
 */
#define UNIQUES_HASH_BITS_FOR_SKIP (32 - UNIQUES_HASH_MAX_SIZE_DEGREE)

/// Initial buffer size degree
#define UNIQUES_HASH_SET_INITIAL_SIZE_DEGREE 4

So the main set holds at most 2^16 = 65 536 hashes in a buffer of 2^17 slots, and 15 low bits of the hash are reserved for the thinning mask. The remaining 17 high bits select the cell:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:195
 size_t place(HashValue x) const { return (x >> UNIQUES_HASH_BITS_FOR_SKIP) & mask(); }

 /// The value is divided by 2 ^ skip_degree
 bool good(HashValue hash) const { return hash == ((hash >> skip_degree) << skip_degree); }

good() is the thinning test: the low skip_degree bits of the hash must be zero. The high bits pick the position, because the low bits are constant across all surviving elements — using them for the position would pile every element into the same cell.

Thinning

The state is three integers plus a flag:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:155
 UInt32 m_size; /// Number of elements
 UInt8 size_degree{}; /// The size of the table as a power of 2
 UInt8 skip_degree; /// Skip elements not divisible by 2 ^ skip_degree
 bool has_zero; /// The hash table contains an element with a hash value of 0.

And the moment of truth is shrinkIfNeed:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:346
 /** If the hash table is full enough, then do resize.
 * If there are too many items, then throw half the pieces until they are small enough.
 */
 void shrinkIfNeed()
 {
 if (unlikely(m_size > max_fill()))
 {
 if (m_size > UNIQUES_HASH_MAX_SIZE)
 {
 while (m_size > UNIQUES_HASH_MAX_SIZE)
 {
 if (unlikely(skip_degree >= max_skip_degree))
 throw DB::Exception(DB::ErrorCodes::INCORRECT_DATA,
 "Cannot thin out UniquesHashSet: {} elements remain at the maximum skip degree of {}",
 static_cast<size_t>(m_size), static_cast<size_t>(max_skip_degree));

 ++skip_degree;
 rehash();
 }
 }
 else
 resize();
 }
 }

So: the set grows normally until it holds more than 65 536 elements. Then it does not grow — it thins. skip_degree goes up by one, and every element whose hash is not divisible by 2^skip_degree is deleted:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:200
 /// Delete all values whose hashes do not divide by 2 ^ skip_degree
 void rehash()
 {
 for (size_t i = 0; i < buf_size(); ++i)
 {
 if (buf[i])
 {
 if (!good(buf[i]))
 {
 buf[i] = 0;
 --m_size;
 }
 /** After removing the elements, there may have been room for items,
 * which were placed further than necessary, due to a collision.
 * You need to move them.
 */
 else if (i != place(buf[i]))
 {
 HashValue x = buf[i];
 buf[i] = 0;
 reinsertImpl(x);
 }
 }
 }

The picture is this:

 text
 
 
 distinct values arriving: A B C D E F G H ... (millions)

 skip_degree = 0 -> accept everything, table grows
 table full at 65536 elements, distinct count so far maybe 200 000
 skip_degree = 1 -> accept only hashes with bit0 == 0 (half of them)
 table full again at 65536, distinct count so far maybe 400 000
 skip_degree = 2 -> accept only hashes with bits1..0 == 0 (a quarter)
 ...
 skip_degree = k -> accept 1 in 2^k elements

Each thinning step halves the sample rate, and the sample size stays roughly constant. The estimate multiplies the surviving count back up by 2^skip_degree.

Here’s the thing: the table does not need to know how many distinct values it has seen. It only needs to know how aggressively it is currently sampling. The count falls out of that at the end.

The estimator

The estimator is not simply m_size * 2^skip_degree — there is a correction for the systematic error of 32-bit hash collisions, and a pseudo-random remainder so that the result does not visibly sit on a power of two:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:842
 if (0 == skip_degree)
 return m_size;

 size_t res = m_size * (1ULL << skip_degree);

 /** Pseudo-random remainder - in order to be not visible,
 * that the number is divided by the power of two.
 */
 res += (intHashCRC32(m_size) & ((1ULL << skip_degree) - 1));

 /** Correction of a systematic error due to collisions during hashing in UInt32.
 * `fixed_res(res)` formula
 * - with how many different elements of fixed_res,
 * when randomly scattered across 2^32 buckets,
 * filled buckets with average of res is obtained.
 */
 return correctFor32BitCollisions(res);

The pseudo-random remainder is a lovely touch of honesty. Without it, every estimate would be a multiple of a power of two, and users would (correctly!) assume the result was rounded. It only costs one CRC and one mask.

The collision correction is an analytic formula for the birthday problem. If res distinct elements are scattered over 2^32 buckets, the number of occupied buckets is less than res because of collisions; to recover the number of distinct elements from the number of occupied buckets, you invert that relation:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:548
 /** With how many different elements, when randomly scattered across 2^32 buckets,
 * the given number of non-empty buckets is obtained on average.
 * This is the correction of the systematic error due to collisions of the 32-bit hashes.
 */
 static size_t correctFor32BitCollisions(size_t res)
 {
 size_t p32 = 1ULL << 32;

 /** When the number of distinct elements is much larger than 2^32, almost all buckets are filled
 * and `res` may reach (or, due to the pseudo-random remainder, exceed) 2^32,
 * where the formula below is not defined (previously this resulted in undefined behavior
 * when casting infinity or NaN to an integer). Saturate the estimate instead:
 * 32 bits of a hash contain no information about larger cardinalities anyway
 * (and the wide set takes over long before this point).
 */
 if (res >= p32)
 res = p32 - 1;

 return static_cast<size_t>(round(static_cast<double>(p32) * (log(static_cast<double>(p32)) - log(static_cast<double>(p32 - res)))));
 }

Read the formula as p32 * (ln p32 - ln(p32 - res)). For small res it is approximately res; for res near p32 it blows up, which is why the saturation guard exists — and that guard is the fix for issue #6078, where the cast of inf to an integer was undefined behaviour.

The wide set: fixing the 32-bit ceiling

The 32-bit hash has a hard ceiling. At a few billion distinct elements the correction formula is already straining, and above ~30 billion it was producing garbage. The fix, added in state version 1, is a second set that stores full 64-bit hashes — but only a tiny, heavily thinned sample of them.

The sample is defined by a skip degree of 18: only hashes with 18 zero low-order bits enter it, i.e. one in 262 144:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:89
/** Only hashes with this many zero low-order bits enter the wide set, so that it stays
 * negligibly small while the main set is accurate: one hash in 262144 passes.
 */
#define UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE 18

/// The maximum degree of the wide set buffer size before the values are thinned out further.
#define UNIQUES_HASH_SET_WIDE_MAX_SIZE_DEGREE 15

/// The maximum number of elements of the wide set: 16384 values take 256 KiB.
#define UNIQUES_HASH_SET_WIDE_MAX_SIZE (1ULL << (UNIQUES_HASH_SET_WIDE_MAX_SIZE_DEGREE - 1))

/** The estimate is taken from the wide set when it has at least this many values, i.e. starting
 * from about UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE * 2^UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE,
 * about two billion elements: at that point the statistical error of the wide estimate (about 1%)
 * meets the degrading precision of the 32-bit one.
 */
#define UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE 8192

So the wide set holds at most 16 384 hashes (256 KiB) and needs 8192 of them before it is trusted. Since only one in 262 144 elements enters it, 8192 elements means about 2.1 billion distinct values have been seen — exactly the point where the 32-bit estimator starts to degrade.

The wide set has one more twist: the hashes are XOR-ed with a constant before being stored.

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:518
 /** The wide set samples the full hashes XOR-ed with this constant rather than the hashes themselves.
 * The sample stays just as uniform, but the hash of a frequent value never becomes a candidate
 * systematically: the default hash of the integer 0 is 0, and zeros are frequent in real data -
 * without the offset every such row would be a candidate for the wide set and would cost
 * an extra pass over its batch (this doubled the time of `uniq` over such columns).
 * The low bits up to the maximum thinning of the main set are zero, so that every candidate
 * still passes good() of the main set (see `widePrefilterMask`), and the next ones are not,
 * so that the hash 0 is never a candidate.
 */
 static constexpr UInt64 wide_hash_offset = 0x9E3779B97F4A0000ULL;

The constant is the golden-ratio fraction used in many hash mixers (0x9E3779B97F4A0000 is 2^64 / φ). The problem it solves is delightfully concrete: the default hash of the integer 0 is 0, and the wide-set candidate test is “are the low 18 bits zero?”. The hash 0 passes that test — so every zero row in the data would enter the wide set, and the extra work doubled the runtime of uniq over columns full of zeros.

XOR-ing with an offset whose low bits are non-zero removes that pathology while preserving uniformity.

The offset has to satisfy two constraints, and they are checked at compile time:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:528
 static_assert((wide_hash_offset & ((1ULL << max_skip_degree) - 1)) == 0);
 static_assert((wide_hash_offset & ((1ULL << UNIQUES_HASH_SET_WIDE_INITIAL_SKIP_DEGREE) - 1)) != 0);

The first says: the offset’s low bits are zero up to the maximum thinning degree of the main set, so that a wide candidate still passes the main set’s good() test — the wide set is a subset of the main sample. The second says: the offset’s low 18 bits are not all zero, so the hash 0 is never a wide candidate.

Two constraints, two static_asserts. This is the good stuff: an invariant expressed as a compile-time proof instead of a comment.

The wide set is allocated lazily, in a single block holding both the header and the table:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:373
 static WideSet * allocWideSet(UInt8 new_size_degree)
 {
 /// The header and the table are placed in a single allocation.
 size_t bytes = sizeof(WideSet) + (1ULL << new_size_degree) * sizeof(UInt64);
 WideSet * res = new (HashTableAllocator().alloc(bytes)) WideSet;
 res->size_degree = new_size_degree;
 res->buf = reinterpret_cast<UInt64 *>(reinterpret_cast<char *>(res) + sizeof(WideSet));
 return res;
 }

and it is only allocated when the first candidate shows up — which, for most data, never happens:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:469
 if (!wide)
 wide = allocWideSet(UNIQUES_HASH_SET_INITIAL_SIZE_DEGREE);

Which estimator is used

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:820
 size_t size() const
 {
 /** The wide set gives a reliable estimate once it has enough values;
 * below that the 32-bit main set is more precise. If the wide sample does not cover
 * all the elements (see `wide_incomplete`), it is not there at all and the legacy
 * estimator is used, because a partial wide estimate would silently lose the rest.
 */
 if (wide && wide->m_size >= UNIQUES_HASH_SET_WIDE_MIN_SIZE_FOR_ESTIMATE)
 {
 size_t res = static_cast<size_t>(wide->m_size) * (1ULL << wide->skip_degree);

 /** Pseudo-random remainder - in order to be not visible,
 * that the number is divided by the power of two.
 */
 res += (intHashCRC32(wide->m_size) & ((1ULL << wide->skip_degree) - 1));

 /** Collisions of 64-bit hashes are negligible: their systematic error would reach
 * just 0.003% at 10^15 distinct elements, far below the statistical error of the estimate.
 */
 return res;
 }

Note the last comment: at 10^15 distinct elements, the systematic error of 64-bit hash collisions would be 0.003%, far below the statistical error of the estimate itself. So the wide set is not exact either — it is exact enough.

The hot loop: batching

The insertion path is where the performance lives, because uniq is usually applied to millions of rows. The implementation does not insert one hash at a time; it computes eight hashes and eight target cells first, then inserts them:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:749
 size_t i = 0;
 while (i < size)
 {
 /// Check that we have enough capacity and enough values to insert in batch. If not, we will insert values one by one which
 /// allows us to check the shrink condition after each insert and avoid inserting too many values before shrinking.
 if ((max_fill() - m_size) >= insert_many_batch_size && ((size - i) >= insert_many_batch_size))
 {
 /// We read and transform multiple values at once which allows both the compiler and the CPU to better optimize the code.
 /// We calculate place() even for !good() hashes to maximize data independence and enable better out-of-order execution.
 /// The extra work is negligible compared to the instruction level parallelization benefits.
 std::array<HashValue, insert_many_batch_size> hash_value; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - filled by the loop below before read
 for (size_t j = 0; j < insert_many_batch_size; ++j)
 {
 hash_value[j] = hash(Transform(data[i + j]));
 }

with the batch size fixed at eight:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:665
 /// This value is arbitrary. The optimal value might depend on the CPU pipeline depth, cache line size (64B = 8 UInt64s),
 /// hash function quality, collision rate, instruction parallelism, etc.
 /// We choose a value that is big enough to provide sufficient instruction level parallelism but not too big to bloat the code size.
 static constexpr size_t insert_many_batch_size = 8;

Two details worth noticing. place() is computed even for hashes that will be rejected by good(), on purpose: it keeps the loop free of data-dependent branches and lets the out-of-order engine overlap the work. And the wide-set candidates are collected into a small local buffer during the loop and inserted after it, because inserting them inside would force the compiler to reload the set’s fields from memory on every iteration:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:738
 /** The candidates for the wide set (see the comment at `widePrefilterMask`) are detected
 * behind the `good()` test, which every one of them passes and which already rejects all
 * but one value in 2^skip_degree, and they are only collected into a local buffer there.
 * They are inserted after the loop rather than inside it: a call inside would make
 * the compiler cautiously reload the fields of the set from memory all the time.
 * This loop is very sensitive to the code generation: the batch loops below must be
 * fully unrolled as they are without the wide set, otherwise `uniq` gets up to 50% slower.
 */

“otherwise uniq gets up to 50% slower” is a great reminder that in an aggregation hot loop, the code layout is part of the algorithm.

What is hashed

The value that goes into the set is not the value itself but a 64-bit hash of it, of which only the low 32 bits feed the main set:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:637
 /// The full 64-bit hash: its low 32 bits feed the main set, and the whole value the wide set (see `wideHash`).
 static UInt64 fullHash(Value key) { return static_cast<UInt64>(Hash()(key)); }
 static HashValue hash(Value key) { return static_cast<HashValue>(fullHash(key)); }

and the per-type hashing is chosen in the aggregate function:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniq.h:273
template <typename T, typename ColumnType>
struct AggregateFunctionUniqTraits
{
 static ALWAYS_INLINE auto hash(T x)
 {
 if constexpr (std::is_same_v<T, std::string_view>)
 {
 return CityHash_v1_0_2::CityHash64(x.data(), x.size());
 }
 else if constexpr (std::is_same_v<T, IPv6>)
 {
 /// For compatibility reasons IPv6 is hashed as FixedString(16).
 return CityHash_v1_0_2::CityHash64(reinterpret_cast<const char *>(&x), sizeof(x));
 }
 else if constexpr (is_floating_point<T>)
 {
 return bit_cast<UInt64>(x);
 }
 else if constexpr (sizeof(T) <= sizeof(UInt64))
 {
 return x;
 }
 else
 {
 return DefaultHash64<T>(x);
 }
 }

Note that for integers the “hash” is the identity: an integer’s low bits are used directly, which is fine because the thinning test only looks at the low 15..18 bits and the position comes from the high bits. For sequential integers the high bits are nearly constant too — which is precisely why the wide set XORs with an offset, and why the main set’s place() uses the high bits of the 32-bit hash, which is intHash32 and mixes.

Indeed the set’s own hash function is the 32-bit integer hash:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:108
/** This hash function is not the most optimal, but UniquesHashSet states counted with it,
 * stored in many places on disks (in many companies), so it continues to be used.
 */
struct UniquesHashSetDefaultHash
{
 size_t operator() (UInt64 x) const
 {
 return intHash32<0>(x);
 }
};

That comment is a nice piece of engineering history: the hash is frozen because the serialized states of uniq are stored on disk in many companies’ systems, and changing the hash would invalidate all of them.

The state format

Because the set is serialised into aggregate-function states that travel between servers, the format is versioned. Version 1 adds a flags byte:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:963
 /** Two serialization formats exist.
 * The legacy format (state version 0): skip_degree, then the number of values, then the 32-bit values
 * of the main set.
 * The new format (state version 1): a flags byte (bit 0 - a wide set follows the main set,
 * bit 1 - the wide sample is incomplete, see `wide_incomplete`; the two are mutually exclusive),
 * then the main set in the legacy layout, then, if present, the wide set: its skip_degree,
 * the number of values, and the 64-bit values.
 */

and a legacy state read back is marked as having an incomplete wide sample:

 cpp
 
 
// src/AggregateFunctions/UniquesHashSet.h:1048
 /** A non-empty state written in the legacy format carries elements that were never offered
 * to a wide set, so no wide estimate of this state (or of anything it is merged into)
 * can be trusted. An empty legacy state contributes nothing and does not spoil a merge.
 */
 if (use_legacy_format && m_size > 0)
 wide_incomplete = true;

That mark is sticky and propagates through merges, because the missing high bits of the already-consumed elements cannot be recovered. The alternative — estimating from a partially-covering wide sample — would silently undercount everything the sample missed, which is far worse than a merely imprecise estimate.

The whole family

uniq is one of several cardinality estimators in ClickHouse, and they make different trade-offs:

function	structure	exact?
`uniqExact`	a full hash set of the values	yes
`uniq`	the thinning set of this article	no, ~1-2% error
`uniqCombined`	small set → hash set → HyperLogLog, switching as it grows	no
`uniqTheta`	a datasketches theta sketch	no, mergeable

uniqCombined is the one that switches representation:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniqCombined.h:112
 // We want to migrate from |HashSet| to |HyperLogLogCounter| when the sizes in memory become almost equal.
 // The size per element in |HashSet| is sizeof(Key)*2 bytes, and the overall size of |HyperLogLogCounter| is 2^K * 6 bits.
 // For Key=UInt32 we can calculate: 2^X * 4 * 2 ≤ 2^(K-3) * 6 ⇒ X ≤ K-4.

and uniqExact keeps the two-level structure of the previous article in reserve, converting when it grows:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionUniq.h:154
 /// When creating, the hash table must be small.
 static constexpr size_t initial_size_degree = 4;
 using SingleLevelSet = HashSetWithStackMemory<Key, HashCRC32<Key>, initial_size_degree>;
 using TwoLevelSet = TwoLevelHashSet<Key, HashCRC32<Key>>;
 using Set = UniqExactSet<SingleLevelSet, TwoLevelSet>;

Note the “hash table must be small”: the initial buffer lives on the stack of the aggregate function state, so a uniqExact state that never grows past 2^4 slots allocates nothing on the heap at all.

A quick sanity check of the behaviour in SQL:

 sql
 
 
SELECT
 uniqExact(user_id) AS exact,
 uniq(user_id) AS approx,
 uniqCombined(user_id) AS combined
FROM events;

approx will differ from exact by a fraction of a percent on a few million distinct users, and by more on a few hundred million — that is the thinning, and it is the price of a bounded 64 KiB state.

Trade-offs

What the thinning set buys:

A bounded, tiny state. The main set is at most 2^17 × 4 bytes = 512 KiB, and the wide set at most 256 KiB. A uniq state never grows with the cardinality, which matters when the state is merged across a cluster or spilled to disk.
A mergeable state. Merging two uniq states is a union of hash sets plus a max(skip_degree), which is why uniq works with Distributed tables and AggregatingMergeTree.
A principled estimator. The thinning factor is known exactly, so the estimate is a maximum-likelihood estimate rather than a heuristic.
No allocation in the hot loop. The table is a flat array of UInt32, and the batch of eight hashes is computed with instruction-level parallelism.

What it costs:

Precision. The relative error grows with the cardinality, because the sample size stays constant while the multiplier grows. The wide set pushes the useful range from billions to trillions, but the error is still around 1% at the switch-over point.
A hash-quality dependency. The whole scheme assumes the hash is uniform; a bad hash breaks the thinning, because the surviving elements would not be a uniform sample.
Irreversible loss. Once thinned, the dropped hashes are gone. That is why the wide set has to be fed from the first insertion, and why a legacy state can never grow a wide sample.
A 32-bit ceiling without the wide set, which is exactly the bug that issue #6078 was, and why the state format has two versions.

This is a good example of an approximation that is statistically principled rather than ad hoc: the state is small because the algorithm throws away information on purpose, and the estimate is exact given the information that was kept.

Wrapping up

The one thing to remember: uniq() does not estimate by approximating — it estimates by sampling, and it knows the sample rate exactly. That is why the state never grows and why the estimate is a maximum-likelihood one rather than a guess.

The part people get wrong: the state is not just “a hash set”. It is a hash set plus a thinning factor, plus a second sparse 64-bit sample that takes over past ~2 billion distinct values. If you are estimating cardinalities in the trillions, you are living in the wide set’s world.

What to go do about it: compare uniq against uniqExact on your own data before you trust it. If the two agree within a percent, uniq is fine and cheaper. If they do not, and you need exactness, you know now what you are paying.

ClickHouse Aggregates In Order and Stops Reading Early

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: If your table is sorted by the GROUP BY key, ClickHouse can skip the hash table entirely: it counts each contiguous run of equal keys and emits it. With optimize_aggregation_in_order_limit, it also pushes LIMIT into the aggregation so the read itself stops early.

SELECT count() FROM hits GROUP BY UserID is, conceptually, a hash table build: read every row, hash every key, update a counter. The hash table’s size is the number of distinct keys — a million entries for a column with a million distinct values, a billion entries for a billion distinct values, and then the query dies.

But if the table is sorted by UserID, that is a lot of wasted work. The rows arrive grouped: all the rows with UserID = 1 are contiguous, then all the rows with UserID = 2, and so on.

You don’t need a hash table at all. You need to count the run, emit it, count the next run, emit it.

That is the optimize_aggregation_in_order optimisation. It turns an unbounded-memory operation into a streaming one whose memory footprint is the size of a single group. And in recent versions it also enables an even more aggressive trick: pushing the LIMIT down into the aggregation so the read stops early.

Let’s walk the machinery: the plan-level decision, the run-by-run aggregation loop, and the limit pushdown with its subtle “prefix boundary” rule.

When the optimisation applies

The decision is made in AggregatingStep::transformPipeline, and it is based on a single field: sort_description_for_merging. If that is non-empty, the input is already sorted on the group-by key and the in-order path is taken.

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:505
 /// optimize_aggregation_in_order
 if (!sort_description_for_merging.empty())
 {
 /// two-level aggregation is not supported anyway for in order aggregation.
 allow_to_use_two_level_group_by = false;

 /// It is incorrect for in order aggregation.
 params.stats_collecting_params.disable();

 /// Aggregation in order rebuilds the aggregation-method state for every run of equal
 /// order-key values, so the whole-block `prealloc_serialized` method would make it
 /// quadratic. Fall back to the plain `serialized` method (see `Params::aggregation_in_order`).
 params.aggregation_in_order = true;
 }

Three consequences fall out of this one branch, and each is a real limitation.

Two-level aggregation is disabled. The classic ClickHouse optimisation for GROUP BY with many distinct keys is two-level aggregation: hash the key, bucket by hash, aggregate each bucket in a separate hash table, and merge the buckets in parallel. That is exactly what you do not want when the data is already grouped — the whole point is to avoid a hash table entirely.

Statistics collection is disabled. The uniqCombined / uniqExact family of aggregate functions maintains auxiliary state to compute statistics that are only meaningful for the whole dataset. In-order aggregation finalises each group as it goes, so those statistics would be wrong.

The prealloc_serialized aggregation method is downgraded. This one deserves an explanation, because it is a performance pathology that the comment records:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:849
 /// See `Params::aggregation_in_order` and `method_chosen_for_in_order`: the `prealloc_serialized`
 /// method serializes the whole block's keys on state construction, which is pathological for the
 /// per-run in-order path, where a fresh state is constructed for every run of equal order-key
 /// values. There it uses the plain `serialized` method, whose construction is O(1) and which
 /// serializes keys lazily. The whole-block paths (including `mergeBlocks`) keep `method_chosen`.
 method_chosen_for_in_order = method_chosen;
 if (params.aggregation_in_order)
 {
 if (method_chosen_for_in_order == AggregatedDataVariants::Type::prealloc_serialized)
 method_chosen_for_in_order = AggregatedDataVariants::Type::serialized;

prealloc_serialized is an optimisation for the hash table case: when you create a state for a key, you serialise the key into a contiguous buffer right away, so the key data is contiguous and the hash table lookup is cheap. In the in-order case a fresh state is created for every run of equal keys — potentially millions of times for a million groups — and serialising the whole key each time would be quadratic in the key width. The lazy serialized variant amortises it.

The pipeline shape

When the in-order path is taken and there is more than one stream, the pipeline looks like this:

 text
 
 
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket
 --> AggregatingInOrder --> FinishAggregatingInOrder --> Resize --> MergingAggregatedBucket

as the comment in AggregatingStep says:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:678
 /** The pipeline is the following:
 *
 * --> AggregatingInOrder --> MergingAggregatedBucket
 * --> AggregatingInOrder --> FinishAggregatingInOrder --> ResizeProcessor --> MergingAggregatedBucket
 * --> AggregatingInOrder --> MergingAggregatedBucket
 */

Each stream runs its own AggregatingInOrderTransform, each with its own ManyAggregatedData variant, and the partial results are merged later. The transform is constructed per stream:

 cpp
 
 
// src/Processors/QueryPlan/AggregatingStep.cpp:692
 return std::make_shared<AggregatingInOrderTransform>(
 header,
 transform_params,
 sort_description_for_merging,
 group_by_sort_description,
 max_block_size,
 aggregation_in_order_max_block_bytes / new_merge_threads,
 many_data,
 counter++,
 limit_hint,
 limit_hint_prefix_columns,

The counter++ is the stream index — each transform gets its own slot in the shared ManyAggregatedData, so the later merge knows which partial state belongs to which stream.

The core loop: aggregating one run at a time

AggregatingInOrderTransform::consume() is the heart of it. The key observation is that the input is sorted, so the rows belonging to one group form a contiguous interval, and the interval can be found with a binary search.

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:183
 size_t interval_index = 0;

 /// Will split block into segments with the same key
 while (key_end != rows)
 {
 /// Cancellation is only checked between work() calls, but one consume() over a chunk with many
 /// keys can run for a long time; check per key interval so a cancelled query stops promptly.
 if (isCancelled())
 {
 LOG_TEST(log, "Cancelled between key intervals");
 return;
 }

and the search itself:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:208
 /// Find the first position of new (not current) key in current chunk
 auto indices = collections::range(key_begin, rows);
 auto it = std::upper_bound(indices.begin(), indices.end(), cur_block_size - 1,
 [&](size_t lhs_row, size_t rhs_row)
 {
 return less(res_key_columns, key_columns, lhs_row, rhs_row, group_by_description);
 });

 key_end = (it == indices.end() ? rows : *it);

 /// Add data to aggr. state if interval is not empty. Empty when haven't found current key in new block.
 if (key_begin != key_end)
 {
 if (params->params.only_merge)
 {
 if (group_by_key)
 params->aggregator.mergeOnBlockSmall(variants, key_begin, key_end, aggregate_columns_data, key_columns_raw);
 else
 params->aggregator.mergeOnIntervalWithoutKey(variants, key_begin, key_end, aggregate_columns_data, is_cancelled);
 }
 else
 {
 if (group_by_key)
 params->aggregator.executeOnBlockSmall(variants, key_begin, key_end, key_columns_raw, aggregate_function_instructions.data());
 else
 params->aggregator.executeOnIntervalWithoutKey(variants, key_begin, key_end, aggregate_function_instructions.data());
 }
 }

res_key_columns holds the current key (one row), key_columns holds the whole chunk’s key column. upper_bound with the less comparator finds the first row whose key is strictly greater than the current key. Everything between key_begin and key_end belongs to the current group.

less() is the generic column-wise comparison:

 cpp
 
 
// src/Core/SortCursor.h:969
template <typename TLeftColumns, typename TRightColumns>
bool less(const TLeftColumns & lhs, const TRightColumns & rhs, size_t i, size_t j, const SortDescriptionWithPositions & descr)
{
 for (const auto & elem : descr)
 {
 size_t ind = elem.column_number;
 int res = elem.base.direction * lhs[ind]->compareAt(i, j, *rhs[ind], elem.base.nulls_direction);
 if (res < 0)
 return true;
 if (res > 0)
 return false;
 }

 return false;
}

So the loop is: binary-search the end of the current run, aggregate the run into the current state, finalise the state into the output columns, record the next key, repeat.

Two modes: with a key and without

group_by_key selects between two quite different implementations. The header comment on the aggregator methods explains the split:

 cpp
 
 
// src/Interpreters/Aggregator.h:553
 /// Used for optimize_aggregation_in_order:
 /// - No two-level aggregation
 /// - No external aggregation
 /// - No without_key support (it is implemented using executeOnIntervalWithoutKey())
 void executeOnBlockSmall(
 AggregatedDataVariants & result,
 size_t row_begin,
 size_t row_end,
 ColumnRawPtrs & key_columns,
 AggregateFunctionInstruction * aggregate_instructions) const;

 void executeOnIntervalWithoutKey(
 AggregatedDataVariants & data_variants,
 size_t row_begin,
 size_t row_end,
 AggregateFunctionInstruction * aggregate_instructions) const;

group_by_key is true when the GROUP BY key is a prefix of the sort description — i.e. the input is sorted by exactly the grouping key. In that case there is only ever one group at a time, so no hash table is needed at all: the aggregate state is a single object that is created, filled, and finalised per run.

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:147
 /// If we don't have a block we create it and fill with first key
 if (!cur_block_size)
 {
 res_key_columns.resize(params->params.keys_size);
 for (size_t i = 0; i < params->params.keys_size; ++i)
 res_key_columns[i] = res_header.safeGetByPosition(i).type->createColumn();

 if (group_by_key)
 params->aggregator.fillKeyColumnsWithSingleKey(key_columns, key_begin, res_key_columns);
 else
 params->aggregator.createStatesAndFillKeyColumnsWithSingleKey(variants, key_columns, key_begin, res_key_columns);

fillKeyColumnsWithSingleKey — the state is not created; the key is just copied into the output column. createStatesAndFillKeyColumnsWithSingleKey — a state is created in the hash table, because the grouping key is a prefix of the sort key but not the whole of it, so several distinct groups may share the same sort-key prefix and must be kept apart.

When the group is finished, the state is written out and the key advances:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:239
 /// We finalize last key aggregation state if a new key found.
 if (key_end != rows)
 {
 if (!group_by_key)
 params->aggregator.addSingleKeyToAggregateColumns(variants, res_aggregate_columns);

 /// Enough groups have been emitted and the new key starts a new value of the
 /// `ORDER BY` prefix, so the rest of the input cannot be needed (see `limit_prefix_columns`).
 /// With `group_by_key`, `cur_block_size` counts runs of the sorted key columns rather than
 /// groups; the groups accumulated for the current block are the entries of the hash table.
 size_t cur_block_groups = group_by_key ? variants.size() : cur_block_size;
 if (limit_hint && cur_block_groups + res_rows >= limit_hint && isLimitPrefixBoundary(key_columns, key_end))
 limit_reached = true;

And the “the rest of the chunk is not needed” case:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:253
 /// If max_block_size or limit_hint is reached we have to stop consuming and generate the block. Save the extra rows into new chunk.
 if (cur_block_size >= max_block_size || cur_block_bytes + current_memory_usage >= max_block_bytes || limit_reached)
 {
 if (group_by_key)
 group_by_chunk
 = params->aggregator.prepareChunkAndFillSingleLevel</* return_single_block */ true>(variants, /* final= */ false).chunk;
 cur_block_bytes += current_memory_usage;

 /// When limit is reached, don't save leftover rows — we're done.
 /// Set block_end_reached + need_generate to trigger generate(),
 /// which will produce the output and then set is_consume_finished.
 if (limit_reached)
 {
 block_end_reached = true;
 need_generate = true;
 variants.invalidate();
 }
 else
 {
 finalizeCurrentChunk(std::move(chunk), key_end);
 }

Note the difference between the two branches. Normally the leftover rows of the chunk (those after key_end) are saved — finalizeCurrentChunk cuts them off and keeps them for the next consume():

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:298
void AggregatingInOrderTransform::finalizeCurrentChunk(Chunk chunk, size_t key_end)
{
 size_t rows = chunk.getNumRows();
 Columns source_columns = chunk.detachColumns();

 for (auto & source_column : source_columns)
 source_column = source_column->cut(key_end, rows - key_end);

 current_chunk = Chunk(source_columns, rows - key_end);
 src_rows -= current_chunk.getNumRows();

 block_end_reached = true;
 need_generate = true;
 variants.invalidate();
}

But when the limit is reached, the leftovers are dropped: the query is over, and reading the rest of the chunk would be wasted work.

Pushing `LIMIT` into the aggregation

The limit pushdown is a plan-level optimisation, in src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp, and it is gated on the optimize_aggregation_in_order_limit setting, which defaults to true:

text

// src/Core/Settings.cpp:5163
 DECLARE(Bool, optimize_aggregation_in_order_limit, true, R"(
When enabled and aggregation in order is active, pushes LIMIT into the aggregation step to enable early termination after producing enough groups. This reduces the amount of data read when ORDER BY matches the GROUP BY key prefix. May reduce the value reported by `rows_before_limit_at_least`; use `exact_rows_before_limit` if exact counts are needed.
)", 0) \

The optimisation looks for the shape LIMIT → SORT → AGGREGATE:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:77
void optimizeLimitForAggregationInOrder(QueryPlan::Node & root)
{
 Stack stack;
 stack.push_back({.node = &root});

 while (!stack.empty())
 {
 auto & frame = stack.back();

 if (frame.next_child < frame.node->children.size())
 {
 stack.push_back({.node = frame.node->children[frame.next_child++]});
 continue;
 }

 auto * node = frame.node;
 stack.pop_back();

 auto * limit_step = typeid_cast<LimitStep *>(node->step.get());
 if (!limit_step || limit_step->withTies() || limit_step->alwaysReadTillEnd())
 continue;

 size_t limit = limit_step->getLimitForSorting();
 if (!limit)
 continue;

WITH TIES and always_read_till_end are excluded immediately — both change the meaning of the limit in ways the pushdown cannot honour.

Then it walks down through “transparent” steps (steps that neither change the row count nor the sort order) looking for the aggregation:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:117
 /// Find AggregatingStep below SortingStep, skipping transparent steps.
 /// Stop at TotalsHavingStep — HAVING may filter groups.
 while (current)
 {
 auto * aggregating_step = typeid_cast<AggregatingStep *>(current->step.get());
 if (aggregating_step)
 {
 /// overflow_row is set by WITH TOTALS.
 if (!aggregating_step->inOrder()
 || !aggregating_step->getFinal()
 || aggregating_step->getParams().overflow_row
 || aggregating_step->isGroupingSets())
 break;

and then the crucial check:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeLimitForAggregationInOrder.cpp:131
 /// The sort description may be a strict prefix of the group-by sort description.
 /// Each in-order stream then stops only at a boundary of that prefix (see
 /// `AggregatingInOrderTransform`), so the groups it drops sort strictly after
 /// at least `limit` complete groups and the final sort can never pick a group
 /// with a partial aggregate value (issue #116849).
 const auto & sort_desc = sorting_step->getSortDescription();
 const auto & agg_sort_desc = aggregating_step->getGroupBySortDescription();
 if (sort_desc.empty() || !agg_sort_desc.hasPrefix(sort_desc))
 break;

 /// Use the smallest limit if multiple LimitSteps point to the same AggregatingStep.
 size_t current_hint = aggregating_step->getLimitHint();
 if (!current_hint || limit < current_hint)
 aggregating_step->setLimitHint(limit, sort_desc.size());
 break;

agg_sort_desc.hasPrefix(sort_desc) — the ORDER BY columns must be a prefix of the GROUP BY sort description. This is the condition that makes the pushdown safe, and the reasoning in the comment is worth unpacking because it is subtle.

The prefix boundary rule

The ORDER BY of the query may be a prefix of the GROUP BY key. Say:

 sql
 
 
SELECT a, b, count()
FROM t
GROUP BY a, b
ORDER BY a
LIMIT 10

Here the table is sorted by (a, b). The GROUP BY produces groups in (a, b) order. The ORDER BY a then sorts them by a — and since the groups already come out in (a, b) order, sorting by a is a stable reordering: groups with the same a keep their relative (a, b) order.

Now: the aggregation stream can stop early once it has produced 10 groups. But it must not stop in the middle of a run of groups that share the same a — because another stream might produce a group with the same a but a smaller b, and the final sort (by a only) could then pick that group over the partial one.

The rule is: stop only at a boundary of the ORDER BY prefix. That is what limit_prefix_columns counts and what isLimitPrefixBoundary() checks:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:85
bool AggregatingInOrderTransform::isLimitPrefixBoundary(const Columns & key_columns, size_t row) const
{
 for (size_t i = 0; i < limit_prefix_columns; ++i)
 {
 const auto & elem = group_by_description[i];
 size_t ind = elem.column_number;
 if (res_key_columns[ind]->compareAt(cur_block_size - 1, row, *key_columns[ind], elem.base.nulls_direction) != 0)
 return true;
 }
 return false;
}

It compares the last emitted group’s value of the prefix column with the next group’s value. If they differ, the run of equal prefix values has ended, and it is safe to stop: every group with that prefix value has been emitted.

Note the comment on the member:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.h:97
 /// The stream may stop early once it has emitted `limit_hint` groups, but only at a
 /// boundary of the first `limit_prefix_columns` columns of `group_by_description`, which
 /// are the columns the query is ordered by. Otherwise a group tied on those columns with
 /// the emitted ones could be cut off here while another stream emits its partial state,
 /// and the final sort could pick that partial group.
 size_t limit_hint = 0;
 size_t limit_prefix_columns = 0;
 bool limit_reached = false;

And the prefix count is clamped to the group-by description:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:80
 /// The `ORDER BY` may extend past the columns the input is sorted by; a boundary of the
 /// sorted columns alone is then a boundary of the `ORDER BY` prefix as well.
 limit_prefix_columns = std::min(limit_prefix_columns_, group_by_description.size());

This is the bug that issue #116849 was about, and the comment in the optimisation pass records it.

What it buys, concretely

 sql
 
 
SELECT user_id, count()
FROM hits
GROUP BY user_id
ORDER BY user_id
LIMIT 10

on a table sorted by user_id. Without the pushdown, the whole table is read, aggregated, sorted, and then 10 rows are kept. With it, the aggregation stops after the first 10 groups, and — crucially — the source stops too, because the transform closes its input:

 cpp
 
 
// src/Processors/Transforms/AggregatingInOrderTransform.cpp:354
 /// When limit is reached, we set is_consume_finished in generate()
 /// and don't save leftover rows. Finish immediately after pushing.
 if (is_consume_finished)
 {
 output.finish();
 input.close();
 LOG_DEBUG(log, "Aggregated. {} to {} rows (from {})", src_rows, res_rows, formatReadableSizeWithBinarySuffix(src_bytes));
 return Status::Finished;
 }

input.close() propagates the stop all the way up to the storage layer. On a MergeTree, that means the remaining granules are never read.

The cost is stated in the setting’s description: rows_before_limit_at_least becomes an underestimate, because the rows that were never read are not counted. exact_rows_before_limit restores the exact count at the price of reading everything.

Trade-offs

What this buys you.

The in-order aggregation replaces a hash table whose size is the number of distinct keys with a state whose size is one group. For a table sorted by the grouping key, that is the difference between a query that fits in memory and one that does not.

It also composes with the parallel execution model: each stream aggregates its own runs independently, and the partial states are merged at the end by FinishAggregatingInOrderTransform and MergingAggregatedBucketTransform. The parallelism is preserved; only the memory behaviour changes.

The limit pushdown extends the idea from “less memory” to “less work”: the read itself stops. On a large MergeTree with a selective LIMIT, this is the difference between reading 10 granules and reading a thousand.

What it costs.

The optimisation is conditional on the data layout. If the table is not sorted by the grouping key, the plan does not use it — and if the table is sorted but the GROUP BY key is not a prefix of the sort key, it does not apply either. ClickHouse has to prove the sortedness, and it does so from the table’s primary key and the optimize_read_in_order machinery, so the whole optimisation is contingent on the optimiser’s knowledge of the physical layout.

The in-order path gives up several general-purpose optimisations: two-level aggregation, external aggregation, and statistics collection. For a query with a small number of distinct keys on a large table, the ordinary hash-table path with two-level aggregation might actually be faster, because it parallelises the hash table itself rather than relying on the input being pre-grouped. The optimize_aggregation_in_order setting defaults to false for exactly this reason.

The prealloc_serialized → serialized downgrade is a real regression in the hash-table case that the in-order case forced, paid for by every query that takes the in-order path.

The limit pushdown has a correctness-adjacent cost that the code is careful about but that is easy to get wrong: the prefix-boundary rule. Stop at the wrong place and the final sort may pick a group whose aggregate state was truncated — a wrong answer, not a slow one. That is why the rule is conservative (stop only at a prefix boundary, never in the middle of a run of equal prefix values) and why the plan-level check requires the ORDER BY to be a prefix of the group-by description.

And the pushdown changes the meaning of a system counter: rows_before_limit_at_least becomes “rows that were actually read”, not “rows that would have been read without the limit”. Any monitoring built on that counter has to know about the optimisation.

Wrapping up

The one thing to remember: sorting your table by the GROUP BY key is not just about skipping data — it changes the shape of the aggregation. A hash table the size of your cardinality becomes a state the size of one group.

The part people get wrong: the LIMIT pushdown only fires when ORDER BY is a prefix of the GROUP BY key. If it isn’t, you get the memory benefit but not the early-termination benefit — and the read still scans the whole table.

What to go do about it: check whether your table’s ORDER BY matches your most common GROUP BY. If it doesn’t, you are leaving one of the largest optimisations in the engine on the table.

Aggregation on Opensource internals

ClickHouse Splits GROUP BY into 256 Buckets for Parallel Merges

The structure

The sub-tables are ordinary tables

How a bucket grows

Resizing is amortised

When is the two-level table used?

The merge is the point

The string flavour

Offsets across buckets

Trade-offs

Further reading

Wrapping up

ClickHouse Estimates Distinct Counts by Thinning a Hash Table

The scheme

The constants

Thinning

The estimator

The wide set: fixing the 32-bit ceiling

Which estimator is used

The hot loop: batching

What is hashed

The state format

The whole family

Trade-offs

Further reading

Wrapping up

ClickHouse Aggregates In Order and Stops Reading Early

When the optimisation applies

The pipeline shape

The core loop: aggregating one run at a time

Two modes: with a key and without

Pushing `LIMIT` into the aggregation

The prefix boundary rule

What it buys, concretely

Trade-offs

Wrapping up

Aggregation on Opensource internals

ClickHouse Splits GROUP BY into 256 Buckets for Parallel Merges

The structure

The sub-tables are ordinary tables

How a bucket grows

Resizing is amortised

When is the two-level table used?

The merge is the point

The string flavour

Offsets across buckets

Trade-offs

Further reading

Wrapping up

ClickHouse Estimates Distinct Counts by Thinning a Hash Table

The scheme

The constants

Thinning

The estimator

The wide set: fixing the 32-bit ceiling

Which estimator is used

The hot loop: batching

What is hashed

The state format

The whole family

Trade-offs

Further reading

Wrapping up

ClickHouse Aggregates In Order and Stops Reading Early

When the optimisation applies

The pipeline shape

The core loop: aggregating one run at a time

Two modes: with a key and without

Pushing LIMIT into the aggregation

The prefix boundary rule

What it buys, concretely

Trade-offs

Wrapping up

Pushing `LIMIT` into the aggregation