Streaming on Opensource internals

ClickHouse Estimates topK with a Space-Saving Counter and an Error Bound

Jay Hubbert — Sat, 24 Oct 2026 09:00:00 +0000

TL;DR: topK cannot keep every value it has seen — that would be groupArray. It keeps a bounded number of candidates, each with a count and an error bound. When a new value arrives and the table is full, the evicted level is tracked in an alpha_map: a hash-bucketed array of approximate minimum counts, so admission is O(1) instead of a scan for the minimum. The counts are exact while the number of distinct values fits in the reserved space, and approximate-with-an-error-bound after that.

SELECT topK(10)(user_agent) FROM hits sounds like it wants an exact answer: the ten most common user agents. But an exact answer requires counting every distinct user agent, and there might be ten million of them. The whole point of a topK aggregate is to run in constant memory over an unbounded stream.

So the algorithm has to make a promise weaker than “exact”. The promise ClickHouse makes is the classic one from the streaming literature: every reported count is an over-estimate, and the over-estimate is bounded.

The algorithm, named

The implementation is not a house invention. The file says so:

 cpp
 
 
// src/Common/SpaceSaving.h:16
/*
 * Implementation of the Filtered Space-Saving for TopK streaming analysis.
 * http://www.l2f.inesc-id.pt/~fmmb/wiki/uploads/Work/misnis.ref0a.pdf
 * It implements suggested reduce-and-combine algorithm from Parallel Space Saving:
 * https://arxiv.org/pdf/1401.0702.pdf
 */

and so does the function’s documentation:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:632
 FunctionDocumentation::Description description_topK = R"(
Returns an array of the approximately most frequent values in the specified column. The resulting array is sorted in descending order of approximate frequency of values (not by the values themselves).

Implements the [Filtered Space-Saving](https://doi.org/10.1016/j.ins.2010.08.024) algorithm for analyzing TopK, based on the reduce-and-combine algorithm from [Parallel Space Saving](https://doi.org/10.1016/j.ins.2015.09.003).

This function does not provide a guaranteed result. In certain situations, errors might occur and it might return frequent values that aren't the most frequent values.

Three phrases carry the whole design: Filtered Space-Saving (only the candidates are kept, not the whole stream), Parallel Space-Saving (the merge step is designed to combine two partial tables, so shards can be merged), and approximately most frequent.

The counter: a count and an error

The unit of storage is a Counter, and it has four fields:

 cpp
 
 
// src/Common/SpaceSaving.h:100
 struct Counter
 {
 Counter() = default;

 explicit Counter(const TKey & k, UInt64 c = 0, UInt64 e = 0, size_t h = 0)
 : key(k)
 , hash(h)
 , count(c)
 , error(e)
 {
 }

 cpp
 
 
// src/Common/SpaceSaving.h:142
 TKey key{};
 size_t hash{};
 UInt64 count{};
 UInt64 error{};
 };

count is the estimated count and error is how much that estimate might be off. The intent of the pair is:

text

count(key) - error(key) <= true_count(key) <= count(key)

so count - error is a lower bound on the truth and count is an upper bound. That is why the comparison operator below subtracts the error before comparing: a counter that was inflated by a large eviction should not out-rank a counter that is merely accurate.

The error is not a heuristic fudge. In textbook Space-Saving it is the minimum count among all counters in the table at the moment the key was admitted: the new key’s count is set to min + 1 and its error to min, because the true count of a brand-new key is at most min + 1 (if it had occurred more often, the minimum counter would not have been the minimum). Every counter therefore carries the same error value at insertion time, and errors do not grow afterwards.

The comparison operator encodes the same idea, and it is the only place the error is consulted:

 cpp
 
 
// src/Common/SpaceSaving.h:136
 // greater() taking error into account
 bool operator> (const Counter & b) const
 {
 return ((count - error) > (b.count - b.error)) || ((count - error) == (b.count - b.error) && count > b.count);
 }

count - error is the lower bound of the true count. So the ordering compares lower bounds first (a counter that might be much bigger wins) and breaks ties on the raw count. This matters: without the error term, a counter that was inflated by a large eviction would always look like the winner, and the algorithm would systematically prefer stale entries.

The hash field is a third thing: the hash of the key, stored in the counter so it does not have to be recomputed. The counter is looked up in a hash map keyed by the value, but the map cell only stores the index into a flat vector of counters:

 cpp
 
 
// src/Common/SpaceSaving.h:453
 ALWAYS_INLINE Counter * findCounter(const TKey & key, size_t hash)
 {
 auto it = counter_map.find(key, hash);
 if (!it)
 return nullptr;

 return &counter_list[it->getMapped()];
 }

So the layout is a hash map from key to index, plus a flat vector of counters. The vector is what gets sorted and truncated; the map is what gets probed.

Insertion: three cases

The insert path is short and each branch is a different case of the algorithm:

 cpp
 
 
// src/Common/SpaceSaving.h:183
 void insert(const TKey & key, UInt64 increment = 1, UInt64 error = 0)
 {
 // Increase weight of a key that already exists
 auto hash = counter_map.hash(key);

 if (auto * counter = findCounter(key, hash))
 {
 counter->count += increment;
 counter->error += error;
 return;
 }

 // Key doesn't exist, but can fit in the top K
 if (unlikely(counter_list.size() < capacity()))
 {
 push(arena.emplace(key), increment, error, hash);
 return;
 }

 const UInt64 alpha_mask = alpha_map.size() - 1;
 auto & alpha = alpha_map[hash & alpha_mask];
 push(arena.emplace(key), alpha + increment, alpha + error, hash);
 }

Case 1: the key is already known. Bump the count. The error stays where it is — the bound only depends on what was evicted at admission time.

Case 2: the key is new and there is room. Insert it with count = increment, error = 0. It is exact: we have seen it exactly once and nothing has been evicted yet. This is why topK gives exact counts when the number of distinct values fits in the reserved space.

Case 3: the key is new and the table is full. Here is the whole trick. The new counter is given:

text

count = alpha + increment
error = alpha

(error on the right-hand side is the error argument of insert, which is zero for the plain and weighted topK.) Note that count - error comes out as exactly increment — the new counter’s lower bound on the truth is just “I have seen this once”.

alpha is the per-bucket estimate of the level at which counters have been evicted. In textbook Space-Saving, when the table is full you evict the minimum counter (call its count m) and replace it with the new key at count m + 1, recording an error of m: the new key’s true count is somewhere in [1, m+1], so m+1 over-counts by at most m.

The difference is that the minimum is not tracked exactly. alpha comes from a hash-bucketed array and is local to the bucket rather than global. Textbook Space-Saving would have to maintain the exact global minimum, which means a heap or a linear scan on every insertion; the bucketised estimate is O(1).

The price is that the error bound is no longer derived from the true global minimum. That is why the function’s own documentation says This function does not provide a guaranteed result rather than promising a specific error bound, and why the 'counts' output includes the error term at all: the user is told how much slack there is rather than being promised a guarantee.

That is the “Filtered” in Filtered Space-Saving.

The alpha map: an O(1) approximate minimum

The alpha_map is an array of UInt64, indexed by hash & mask. Each entry accumulates the eviction level of the counters that hashed into that bucket and were later dropped — see the truncation code below.

Its size is chosen to be a power of two, so the index is a mask rather than a modulo:

 cpp
 
 
// src/Common/SpaceSaving.h:85
 // Suggested constants in the paper "Finding top-k elements in data streams", chap 6. equation (24)
 // Round to nearest power of 2 for cheaper binning without modulo
 constexpr UInt64 nextAlphaSize(UInt64 x)
 {
 constexpr UInt64 alpha_map_elements_per_counter = 6;
 return 1ULL << (sizeof(UInt64) * 8 - std::countl_zero(x * alpha_map_elements_per_counter));
 }

alpha_map_elements_per_counter = 6 is the paper’s suggested constant: six alpha slots per retained counter. nextAlphaSize rounds 6 * capacity up to the next power of two.

And there is a hard cap on the alpha values, so that a pathological merge cannot overflow the counter:

 cpp
 
 
// src/Common/SpaceSaving.h:93
 /// Limit the max alpha value to avoid overflow with merges or topKWeighted with huge weights
 /// Better to have a less precise value than an overflow (which will lead to incorrect results)
 static constexpr UInt64 MAX_ALPHA_VALUE = UInt64{UINT32_MAX};

Better to have a less precise value than an overflow — the comment is a good summary of the whole philosophy of approximate algorithms: degrade the precision, never the correctness of the arithmetic.

Truncation: the eviction

The table does not evict on every insert. It grows to twice the requested capacity and then truncates in one batch:

 cpp
 
 
// src/Common/SpaceSaving.h:170
 void resize(size_t new_capacity)
 {
 if (requested_capacity != new_capacity)
 {
 chassert(empty());
 alpha_map.resize(nextAlphaSize(new_capacity), 0);
 requested_capacity = new_capacity;
 /// If the requested capacity is really small we would end up sorting and truncating too often
 target_capacity = std::max(size_t{64}, requested_capacity * 2);
 counter_list.reserve(target_capacity);
 }
 }

Two numbers: requested_capacity (the k the user asked for) and target_capacity (the high-water mark, max(64, 2k)). The comment explains the 64 floor: with a tiny k you would truncate on nearly every insert, and each truncation is O(n).

The truncation itself:

 cpp
 
 
// src/Common/SpaceSaving.h:410
 ALWAYS_INLINE void truncateIfNeeded(bool force_counter_map_rebuild)
 {
 if (unlikely(counter_list.size() >= target_capacity))
 {
 ::nth_element(
 counter_list.begin(),
 counter_list.begin() + capacity(),
 counter_list.end(),
 [](const auto & l, const auto & r) { return l > r; });

 const UInt64 alpha_mask = alpha_map.size() - 1;
 for (size_t i = requested_capacity; i < counter_list.size(); ++i)
 {
 size_t pos = counter_list[i].hash & alpha_mask;
 alpha_map[pos] = std::min(alpha_map[pos] + counter_list[i].count - counter_list[i].error, MAX_ALPHA_VALUE);
 arena.free(counter_list[i].key);
 }

 counter_list.resize(requested_capacity);
 force_counter_map_rebuild = true;
 }

 if (force_counter_map_rebuild)
 {
 /// Rebuild the counter map
 counter_map.clear();
 for (size_t i = 0; i < counter_list.size(); ++i)
 counter_map.insertIfNotPresent(counter_list[i].key, counter_list[i].hash, i);
 }
 }

nth_element is the right tool: it is O(n) on average and only partially sorts, putting the capacity() best counters at the front. The rest are evicted.

Now the crucial line, the one that closes the loop:

 cpp
 
alpha_map[pos] = std::min(alpha_map[pos] + counter_list[i].count - counter_list[i].error, MAX_ALPHA_VALUE);

count - error is the evicted counter’s lower bound on its true count. That lower bound is added into the alpha map of its hash bucket, so the bucket’s “eviction level” rises. The next new key that hashes into that bucket is admitted with alpha as its error and alpha + increment as its count — the same shape as the textbook rule, but with a bucketed estimate of the minimum in place of the exact one.

arena.free(counter_list[i].key) releases the key’s storage. For String keys that memory lives in a freelist arena; for POD keys free is a no-op:

 cpp
 
 
// src/Common/SpaceSaving.h:50
/*
 * Specialized storage for std::string_view with a freelist arena.
 * Keys of this type that are retained on insertion must be serialized into local storage,
 * otherwise the reference would be invalid after the processed block is released.
 */
template <>
struct SpaceSavingArena<std::string_view>
{
 std::string_view emplace(std::string_view key)
 {
 if (!key.data())
 return key;

 return copyStringInArena(arena, key);
 }

 void free(std::string_view key)
 {
 if (key.data())
 arena.free(const_cast<char *>(key.data()), key.size());
 }

private:
 ArenaWithFreeLists arena;
};

The if (!key.data()) guard is a small but important detail: a string_view over a String column is only valid while the block is alive, so a retained key has to be copied into the arena. A key that was never retained (the null view) must not be freed.

The force_counter_map_rebuild flag exists because nth_element reorders the vector, and the hash map stores indices into that vector. After a truncation the indices are wrong, so the map has to be rebuilt from scratch. That is O(k) per truncation, amortised over k inserts — cheap, but it is a real cost of the “map to index, vector of counters” layout.

The `push` helper and a stack-protector detail

 cpp
 
 
// src/Common/SpaceSaving.h:399
protected:
 /// Fields by value: a `Counter` temporary would escape into this `NO_INLINE` call and give the
 /// per-row `insert` a `-fstack-protector-strong` canary.
 NO_INLINE void push(TKey key, UInt64 count, UInt64 error, size_t hash)
 {
 size_t pos = counter_list.size();
 counter_map.insertIfNotPresent(key, hash, pos);
 counter_list.push_back(Counter{key, count, error, hash});
 truncateIfNeeded(false);
 }

The comment is a compiler-specific micro-optimisation with a surprising side effect. push is NO_INLINE — deliberately, so the per-row insert stays small. But if the Counter temporary were constructed inside push and passed by reference, its address would escape into a non-inlined function, and the compiler would have to protect the stack frame of insert with a canary — on every row. Passing the four fields by value keeps the temporary’s address inside push, so the canary stays where it belongs.

This is the kind of detail that only exists because somebody profiled it.

Getting the answer out

At finalization, the counters are sorted by the same operator> that the truncation used:

 cpp
 
 
// src/Common/SpaceSaving.h:275
 std::vector<Counter> topK(size_t k) const
 {
 std::vector<Counter> new_list;
 new_list.reserve(counter_list.size());
 for (auto & counter : counter_list)
 new_list.push_back(counter);

 size_t return_size = std::min(new_list.size(), k);
 ::partial_sort(
 new_list.begin(), new_list.begin() + return_size, new_list.end(), [](const auto & l, const auto & r) { return l > r; });

 new_list.resize(return_size);

 return new_list;
 }

partial_sort — O(n log k), and it only sorts the first k. The copy into new_list is because topK is const and the state must not be mutated: insertResultInto may be called more than once in the runningAccumulate / window-function paths, and the state has to stay valid.

The aggregate function then turns the counters into a column, optionally with the counts and the error bounds:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:260
 void insertResultInto(AggregateDataPtr __restrict place, IColumn & to, Arena *) const override
 {
 ColumnArray & arr_to = assert_cast<ColumnArray &>(to);
 ColumnArray::Offsets & offsets_to = arr_to.getOffsets();

 const typename State::Set & set = this->data(place).value;
 auto result_vec = set.topK(threshold);
 size_t size = result_vec.size();

 offsets_to.push_back(offsets_to.back() + size);

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:273
 if (include_counts)
 {
 auto & column_tuple = assert_cast<ColumnTuple &>(data_to);

 auto & column_key = assert_cast<ColumnVector<T> &>(column_tuple.getColumn(0)).getData();
 auto & column_count = assert_cast<ColumnVector<UInt64> &>(column_tuple.getColumn(1)).getData();
 auto & column_error = assert_cast<ColumnVector<UInt64> &>(column_tuple.getColumn(2)).getData();

The result type with counts is a Tuple(item, count, error) — the error bound is exposed to the user, not just used internally:

 sql
 
 
SELECT topKWeighted(2, 10, 'counts')(k, w)
FROM VALUES('k Char, w UInt64', ('y', 1), ('y', 1), ('x', 5), ('y', 1), ('z', 10));

 text
 
 
┌─topKWeighted(2, 10, 'counts')(k, w)─┐
│ [('z',10,0),('x',5,0)] │
└─────────────────────────────────────┘

The trailing 0 is the error bound — zero, because with only five rows nothing was ever evicted.

The `reserved` parameter: when is it exact?

The user’s k is not the same as the memory the algorithm uses. The parameter reserved is how many candidates the table can hold:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:556
 UInt64 threshold = 10; /// default values
 UInt64 load_factor = 3;
 bool include_counts = is_approx_top_k;
 UInt64 reserved = threshold * load_factor;

reserved defaults to k * 3. The parameter is validated against a hard ceiling:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:605
 if (reserved > DB::TOP_K_MAX_SIZE || load_factor > DB::TOP_K_MAX_SIZE || threshold > DB::TOP_K_MAX_SIZE)
 throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND,
 "Too large parameter(s) for aggregate function '{}' (maximum is {})", name, toString(TOP_K_MAX_SIZE));

with

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:43
inline constexpr UInt64 TOP_K_MAX_SIZE = 0xFFFFFF;

16 777 215 — a bit over 16 million candidates.

The semantics follow from the insertion logic. The table holds up to reserved counters. Eviction only starts once counter_list.size() >= capacity(), and capacity() is requested_capacity — which is set from reserved:

 cpp
 
 
// src/Common/SpaceSaving.h:163
 size_t capacity() const { return requested_capacity; }

So:

If the number of distinct values is less than reserved, nothing is ever evicted, every error is zero, and the counts are exact.
If it exceeds reserved, eviction begins and the counts become over-estimates with a bounded error.

That is exactly what the documentation says:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:750
 FunctionDocumentation::Parameters parameters_approx_top_k = {
 {"N", "The number of elements to return. Default value: `10`. Maximum value of `N = 65536`.", {"UInt64"}},
 {"reserved", "Optional. Defines how many cells reserved for values. If `uniq(column) > reserved`, the result will be approximate. Default value: `N * 3`.", {"UInt64"}},
 };

If uniq(column) > reserved, the result will be approximate. One line, and it tells the user exactly when they can trust the answer.

The topK function has the same knob, but named load_factor:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:581
 load_factor = applyVisitor(FieldVisitorConvertToNumber<UInt64>(), params[1]);

 if (load_factor < 1)
 throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND,
 "Too small parameter 'load_factor' for aggregate function '{}' (got {}, minimum is 1)", name, load_factor);

and it is a multiplier on N, not an absolute cell count:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:695
 FunctionDocumentation::Parameters parameters_topKWeighted = {
 {"N", "The number of elements to return. Default value: 10.", {"UInt64"}},
 {"load_factor", "Optional. Defines, how many cells reserved for values. If `uniq(column) > N * load_factor`, result of topK function will be approximate. Default value: 3.", {"UInt64"}},
 {"counts", "Optional. Defines whether the result should contain an approximate count and error value.", {"Bool"}}
 };

So topK(10, 100)(x) means “give me 10, but keep 10 × 100 = 1000 candidates around” — a hundredfold memory blow-up relative to k, and a correspondingly tighter error bound, because evictions only start once 1000 distinct values have been seen.

Merging: the Parallel Space-Saving reduce-and-combine

The hard part of any top-k algorithm is merging two partial results, because a value that was in the top-k of one shard may not be in the top-k of the other. Plain Space-Saving is not mergeable: the error bounds of the two sides are not comparable.

The Parallel Space-Saving paper’s trick is to make the merge account for the fact that a counter from the other side suffers from the first side’s lack of precision. ClickHouse implements it as a two-sweep in-place update:

 cpp
 
 
// src/Common/SpaceSaving.h:207
 /*
 * Parallel Space Saving reduction and combine step from:
 * https://arxiv.org/pdf/1401.0702.pdf
 */
 void merge(const Self & rhs)
 {
 if (rhs.empty())
 return;

 if (empty())
 {
 *this = rhs;
 return;
 }

 UInt64 m1 = 0;
 UInt64 m2 = 0;

 if (size() >= capacity())
 {
 m1 = *std::ranges::max_element(alpha_map);
 }

 if (rhs.counter_list.size() >= rhs.capacity())
 {
 m2 = *std::ranges::max_element(rhs.alpha_map);
 }

m1 and m2 are the maximum alpha values of the two sides — that is, the approximate minimum counts of the two tables, if both were full. They are the “resolution” of the two sides.

Then:

 cpp
 
 
// src/Common/SpaceSaving.h:235
 /*
 * Updated algorithm to mutate current table in place
 * without mutating rhs table or creating new one
 * in the first step we expect that no elements overlap
 * and in the second sweep we correct the error if they do.
 */
 if (m2 > 0)
 {
 for (auto & counter : counter_list)
 {
 counter.count += m2;
 counter.error += m2;
 }
 }

 for (const auto & counter : rhs.counter_list)
 {
 size_t hash = counter.hash;
 if (auto * current = findCounter(counter.key, hash))
 {
 // Subtract m2 previously added, guaranteed not negative
 current->count += (counter.count - m2);
 current->error += (counter.error - m2);
 }
 else
 {
 // Counters not monitored in S1
 counter_list.push_back(Counter{arena.emplace(counter.key), counter.count + m1, counter.error + m1, hash});
 }
 }

Sweep 1. Every counter of the left table gets += m2 on both count and error. Since count and error both grow by m2, count - error — the lower bound — is unchanged. The upper bound grows, because the right side’s counters may have been inflated by evictions we did not see. So the left table’s counters now carry the right table’s uncertainty.

Sweep 2. The right table’s counters are folded in. For a key that already exists on the left, the m2 added in sweep 1 is subtracted back (the comment says Subtract m2 previously added, guaranteed not negative), and the right side’s own count and error are added. For a key that does not exist on the left, it is admitted with count + m1 and error + m1 — the left side’s uncertainty is charged against it.

The alpha maps are then merged element-wise:

 cpp
 
 
// src/Common/SpaceSaving.h:266
 if (alpha_map.size() == rhs.alpha_map.size())
 {
 for (size_t i = 0; i < alpha_map.size(); i++)
 alpha_map[i] = std::min(alpha_map[i] + rhs.alpha_map[i], MAX_ALPHA_VALUE);
 }

 truncateIfNeeded(true);

Note the size check: the two alpha maps are only merged if they are the same size, which they are if both were resized with the same capacity. The MAX_ALPHA_VALUE clamp is the same “degrade precision, never overflow” rule as before.

The final truncateIfNeeded(true) brings the merged table back down to capacity. The true forces the counter map rebuild, because the merge pushed new counters into the vector.

What the aggregate function adds

AggregateFunctionTopK itself is thin. The state is the SpaceSaving set, and the function is a template over the key type:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:45
template <typename T>
struct AggregateFunctionTopKData
{
 using Set = SpaceSaving<T, HashCRC32<T>>;

 Set value;
};

The addBatchSinglePlace override is the interesting part — it is where the per-row insert loop lives, and it handles the -If condition column:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:138
 void addBatchSinglePlace(
 size_t row_begin, size_t row_end, AggregateDataPtr __restrict place, const IColumn ** columns, Arena *, ssize_t if_argument_pos)
 const override
 {
 auto & set = this->data(place).value;
 ensureCapacity(set);

 auto & data = assert_cast<const ColumnVector<T> &>(*columns[0]).getData();

 if (if_argument_pos >= 0)
 {
 const auto & flags = assert_cast<const ColumnUInt8 &>(*columns[if_argument_pos]).getData();
 for (size_t i = row_begin; i < row_end; ++i)
 {
 if (flags[i])
 {
 if constexpr (is_weighted)
 set.insert(data[i], columns[1]->getUInt(i));
 else
 set.insert(data[i]);
 }
 }
 }

and the capacity has to be re-established on every block, because the state may have been deserialized:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:132
 void ensureCapacity(AggregateFunctionTopKData<T>::Set & set) const
 {
 if (unlikely(set.capacity() != reserved))
 set.resize(reserved);
 }

resize asserts that the set is empty, so this only fires for a fresh state. The unlikely branch is the deserialization path.

The trade-offs

What this buys you.

Constant memory over an unbounded stream. The state size is bounded by reserved counters regardless of how many distinct values arrive.
A real guarantee, not a vibe. Every count is an over-estimate with a known bound, and the bound is exposed to the user as the third element of the 'counts' tuple.
Exactness when it is affordable. If uniq(column) <= reserved, every error is zero and the answer is exactly right. The approximation degrades gracefully as the cardinality grows past the reservation.
Mergeable. The Parallel Space-Saving reduce-and-combine step means topK can be computed per-shard and merged, which is what makes it usable in a distributed query and in AggregatingMergeTree.
Cheap eviction. O(1) amortised per insert, thanks to the alpha map, instead of the O(n) minimum scan of textbook Space-Saving.

What it costs you.

Over-counting. The reported counts are upper bounds. A value that appears 3 times may be reported as 100 if the table was churning. The ranking is usually right; the magnitudes may not be.
A heuristic minimum. The alpha map buckets by hash, so alpha is a per-bucket estimate, not the global minimum. The error bound is therefore valid but looser than exact Space-Saving would give. The paper’s suggested 6 alpha slots per counter is the knob that trades memory for tightness.
Amortised but real truncation cost. Each truncation is an O(n) nth_element plus an O(k) hash map rebuild, and it happens every ~k inserts. For very small k the max(64, 2k) floor is what keeps this from dominating.
Order dependence. The result depends on the order in which values arrive, which is why the function is registered as order-dependent:

 cpp
 
 
// src/AggregateFunctions/AggregateFunctionTopK.cpp:630
 AggregateFunctionProperties properties = { .returns_default_when_only_null = false, .is_order_dependent = true };

The same input in a different order can give a different answer. This is inherent to streaming sketches, and it is the price of not storing everything.

The string-key copy. A String key must be copied into the arena when retained, so topK over strings is not free.

The essential idea is that the error bound is not metadata added after the fact — it is a first-class part of the counter, maintained through every insertion, every eviction, and every merge. That is what turns “we approximate” into “we approximate, and here is by how much”.

How the ClickHouse Kafka Engine Commits Offsets Only After the Block Is Written

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: A Kafka consumer that commits its offset before the data is durable loses data; one that commits after duplicates data. ClickHouse picks the second: the background task builds a block, pushes it through the materialized views, and only when the pipeline has finished does it call commit(). If the block is cancelled halfway, the consumer rewinds to the start of that block, so the block is replayed instead of lost. The cost is a real hole — the code says so in an exception message.

Kafka is a log with a cursor, and every consumer of a log has to answer the same question: when do I tell the broker I have read this? Answer it too early and a crash between the commit and the write loses data; answer it too late and a crash between the write and the commit duplicates data. Neither is “exactly once”, and the difference between them is where the correctness bug lives.

ClickHouse’s Kafka engine answers it as late as it can: the offset commit happens after the rows have been written into the materialized views. This article follows that ordering through the code, and then through the places where it leaks.

The shape of the engine

A Kafka table in ClickHouse is not a table you query — it is a stream source with materialized views attached:

 sql
 
 
CREATE TABLE events_queue (id UInt64, payload String)
ENGINE = Kafka
SETTINGS kafka_broker_list = 'kafka:9092',
 kafka_topic_list = 'events',
 kafka_group_name = 'clickhouse',
 kafka_format = 'JSONEachRow';

CREATE MATERIALIZED VIEW events_mv TO events AS SELECT * FROM events_queue;

The engine’s job is to read messages, parse them with a format, and push the resulting blocks into the views. It never stores anything itself. The background work is a scheduled task, one per consumer (or one for all of them, unless kafka_thread_per_consumer is set):

 cpp
 
 
// src/Storages/Kafka/StorageKafka.cpp:217
 auto task = getContext()->getMessageBrokerSchedulePool()->createTask(getStorageID(), log->name(), [this, i]{ threadFunc(i); });

threadFunc is the outer loop: it checks that the dependent views are ready, claims a “cycle” (so that two tasks do not consume the same topic concurrently), and then streams until it stalls or hits the work-duration limit. The work itself is in streamToViews, and this is where the ordering lives.

The push: write first, commit after

streamToViews builds an INSERT pipeline into the dependent views, runs it to completion, and only then commits the offsets:

 cpp
 
 
// src/Storages/Kafka/StorageKafka.cpp:757
 std::atomic_size_t rows = 0;
 {
 block_io.pipeline.complete(std::move(pipe));

 // we need to read all consumers in parallel (sequential read may lead to situation
 // when some of consumers are not used, and will break some Kafka consumer invariants)
 block_io.pipeline.setNumThreads(stream_count);
 block_io.pipeline.setConcurrencyControl(kafka_context->getSettingsRef()[Setting::use_concurrency_control]);

 block_io.pipeline.setProgressCallback([&](const Progress & progress) { rows += progress.read_rows.load(); });
 CompletedPipelineExecutor executor(block_io.pipeline);
 executor.execute();
 }

 bool some_stream_is_stalled = false;
 for (auto & source : sources)
 {
 some_stream_is_stalled = some_stream_is_stalled || source->isStalled();
 source->commit();
 }

Read the two halves again. executor.execute() runs the whole pipeline: Kafka source → format parser → INSERT into the materialized views → the view’s own storage engine (usually a MergeTree, which writes a part). When it returns, the data is in the destination table. source->commit() then tells Kafka “I have read up to offset N”.

If the process dies during executor.execute(), the offsets were never committed, so the messages are re-read after restart, and the destination table does not have them. No data loss. If the process dies between executor.execute() and source->commit(), the destination table has the rows and Kafka does not know it: the messages will be re-read and re-written. Duplicates.

That window is unavoidable without a transaction spanning Kafka and ClickHouse, and the code says so out loud — the exception message when the commit fails is one of the best pieces of honest error reporting in the codebase:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:257
 if (!committed)
 {
 // TODO: insert atomicity / transactions is needed here (possibility to rollback, on 2 phase commits)
 ProfileEvents::increment(ProfileEvents::KafkaCommitFailures);
 throw Exception(ErrorCodes::CANNOT_COMMIT_OFFSET,
 "All commit attempts failed. Last block was already written to target table(s), "
 "but was not committed to Kafka.");
 }

“Last block was already written to target table(s), but was not committed to Kafka.” That is the exactly-once hole, named precisely. The TODO says what would close it: a two-phase commit.

Note also that the commit is retried five times before giving up, because a Kafka commit can fail for boring transient reasons — “broker may reject commit if during offsets.commit.timeout.ms (5000 by default), there were not enough replicas available for the __consumer_offsets topic”. One error code is treated as success: RD_KAFKA_RESP_ERR__NO_OFFSET means “there was nothing to commit”, so it short-circuits the retry loop. Everything else is retried, and when the retries run out:

The block boundary

The commit is per block, not per message. The size of a block is decided by the source, and the source’s loop has four exit conditions:

 cpp
 
 
// src/Storages/Kafka/KafkaSource.cpp:323
 if (!consumer->hasMorePolledMessages()
 && (total_rows >= max_block_size || !checkTimeLimit() || failed_poll_attempts >= MAX_FAILED_POLL_ATTEMPTS))
 {
 break;
 }

total_rows >= max_block_size — the block is full.
!checkTimeLimit() — the flush interval elapsed. The default is stream_flush_interval_ms = 7500 ms, or kafka_flush_interval_ms if it is set.
failed_poll_attempts >= MAX_FAILED_POLL_ATTEMPTS — the topic is empty and the poll has failed 10 times in a row (“with default poll timeout (500ms) it will give about 5 sec delay for doing 10 retries when selecting from empty topic”).
!consumer->hasMorePolledMessages() — the polled batch is exhausted, so there is nothing more to read without another poll.

The block size is max_insert_block_size / num_consumers by default, so that N consumers each produce a block of 1/N of the insert size (or kafka_max_block_size if it is set explicitly). A bigger block means fewer commits and fewer parts; a smaller block means lower latency for the first rows. This is the knob.

The block start: how to replay

Here is the part I find genuinely clever. When a block is aborted — the table is detached, the query is cancelled, the view is not ready — the engine must not lose the messages it already consumed but did not commit. It must also not re-consume messages it already committed.

The trick: the consumer records where the current block started, per topic-partition, before it consumes anything:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:586
void KafkaConsumer::storeLastReadMessageOffset()
{
 if (!isStalled())
 {
 trackCurrentBlockStart(*(current - 1));
 consumer->store_offset(*(current - 1));
 ++offsets_stored;
 }
}

void KafkaConsumer::trackCurrentBlockStart(const cppkafka::Message & message)
{
 for (const auto & tp : block_start_offsets)
 if (tp.get_partition() == message.get_partition() && tp.get_topic() == message.get_topic())
 return;
 block_start_offsets.emplace_back(message.get_topic(), message.get_partition(), message.get_offset());
}

storeLastReadMessageOffset() is called after every message that produced rows. It stores the current offset (the one to commit later) and records the first offset seen for this topic-partition since the last commit — trackCurrentBlockStart is idempotent per partition, so only the first one is kept.

And when the block is cancelled, the source rewinds to those recorded starting points:

 cpp
 
 
// src/Storages/Kafka/KafkaSource.cpp:330
 if (cancelled || storage.isConsumeCancelRequested(cancel_epoch))
 {
 try
 {
 consumer->rewindToLastCommitted();
 }
 catch (const cppkafka::HandleException & e)
 {
 LOG_WARNING(log, "Failed to rewind to last committed offset on abort: {}", e.what());
 consumer->cleanBlockStartOffsets();
 consumer->markDirty();
 }
 return {};
 }

and the rewind assigns the consumer back to the block’s start offsets and commits them, so the rewind is durable:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:411
 /// Rewind to where the in-flight block started. Re-commit the block partitions
 size_t max_retries = 5;
 while (max_retries > 0)
 {
 try
 {
 auto positions = consumer->get_offsets_committed(consumer->get_assignment());
 for (auto & tp : positions)
 for (const auto & start : block_start_offsets)
 if (tp.get_partition() == start.get_partition() && tp.get_topic() == start.get_topic())
 tp.set_offset(start.get_offset());

 consumer->assign(positions);
 consumer->commit(block_start_offsets);
 LOG_TRACE(log, "Aborted in-flight block, rewound to block start: {}", block_start_offsets);
 cleanBlockStartOffsets();
 return;
 }

So the failure taxonomy is:

 text
 
 
 block in flight (offsets stored, not committed)
 |
 | crash / cancel / view not ready
 v
 rewind to block_start_offsets -> the block is re-read from the broker
 |
 | the block is re-consumed and re-written
 v
 duplicates in the destination (the previous attempt may have
 partially or fully written before the failure)

The rewind is what makes the engine not lose data. The duplicates are the price, paid in the destination table — which is why the standard advice is to use a deduplicating engine (ReplacingMergeTree, CollapsingMergeTree) or to make the INSERT idempotent. If there are no recorded block starts (because kafka_commit_every_batch committed mid-block), the rewind is a plain “go back to the last committed offset”.

The other mode: commit every batch

The default is to commit once per block, after the write. The setting kafka_commit_every_batch (default false) changes the trade-off: when it is on, the consumer commits each polled batch as it is consumed, before the block is written:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:547
 if (hasMorePolledMessages())
 return getNextMessage();

 if (intermediate_commit)
 commit();

 doPoll();

The intermediate_commit happens right before a new poll, i.e. at the boundary between batches. This is the at-most-once mode: if the server dies after the commit but before the write, the messages are gone.

So the two settings give the two ends of the classic trade-off:

setting	commit point	failure mode
`kafka_commit_every_batch = false` (default)	after the block is written	duplicates on crash
`kafka_commit_every_batch = true`	after each polled batch	loss on crash

The default picks duplicates: in a stream-processing pipeline duplicates are usually recoverable (dedup downstream) and losses are not.

Reading from a Kafka table directly

A SELECT from a Kafka table is a different path, and a strange one: the table is read destructively, consuming the messages.

 cpp
 
 
// src/Storages/Kafka/StorageKafka.cpp:146
 for (size_t i = 0; i < kafka_storage.num_consumers; ++i)
 {
 /// Use block size of 1, otherwise LIMIT won't work properly as it will buffer excess messages in the last block
 /// TODO: probably that leads to awful performance.
 /// FIXME: seems that doesn't help with extra reading and committing unprocessed messages.
 pipes.emplace_back(std::make_shared<KafkaSource>(
 kafka_storage,
 storage_snapshot,
 modified_context,
 column_names,
 kafka_storage.log,
 1,
 (*kafka_storage.kafka_settings)[KafkaSetting::kafka_commit_on_select]));
 }

Note the block size of 1: with a larger block, LIMIT 5 would have to read and buffer a whole block of messages to return five rows, and the extra messages would be committed anyway. With a block size of 1 the LIMIT works, at the cost of one pipeline setup per row.

And the commit is done in the suffix of the source — KafkaSource::generate() commits only when the source is exhausted and the whole result has been consumed by the client — which is why the setting is called kafka_commit_on_select and defaults to false. With it off, a SELECT from a Kafka table consumes the messages but never commits their offsets, so the group will read them again later; with it on, the offsets are committed when the select finishes, and the messages are gone for good.

The consumer pool

One more piece of machinery that is easy to miss: the Kafka consumers are pooled. The engine creates num_consumers consumer objects up front, and a SELECT or a streaming task borrows one with popConsumer(timeout).

The pool exists because creating a cppkafka::Consumer and subscribing is expensive (it involves a group join and a rebalance). popConsumer() has three cases, in order: a free consumer exists, so return it; no consumer is free but a slot has been closed, so create a new one under the lock (cheap, because creating the client does not subscribe yet); otherwise wait up to the timeout for one to be released.

Unused consumers are closed by a cleanup thread after kafka_consumers_pool_ttl_ms (60 seconds by default), so a burst of SELECTs does not leave a hundred subscribed consumers hanging around.

The modern version: an RAII offset guard

The engine that ships alongside the old one — StorageKafka2 — solves the same problem with a different tool: a guard object whose destructor rolls the offset back.

 cpp
 
 
// src/Storages/Kafka/KeeperHandlingConsumer.h:60
 class OffsetGuard
 {
 public:
 OffsetGuard(KeeperHandlingConsumer & consumer_, int64_t new_offset_);
 OffsetGuard(OffsetGuard && other) noexcept;
 ~OffsetGuard();

 void commit();
 private:
 bool needs_rollback{true};
 KeeperHandlingConsumer * consumer;
 int64_t new_offset;
 };

and its destructor and commit() are the whole mechanism:

 cpp
 
 
// src/Storages/Kafka/KeeperHandlingConsumer.cpp:97
KeeperHandlingConsumer::OffsetGuard::~OffsetGuard()
{
 if (consumer && needs_rollback)
 consumer->rollbackToCommittedOffsetsNoThrow();
}

void KeeperHandlingConsumer::OffsetGuard::commit()
{
 chassert(needs_rollback && consumer);
 consumer->saveCommittedOffset(new_offset);
 needs_rollback = false;
}

The guard is created when the messages are polled and rolls the offset back unless someone explicitly calls commit() on it. The source takes the guard only after it knows it will not throw — “the cancel may have arrived while pollConsumer was polling. Drop the block by not taking its offset guard” — and the destructor of the source releases it: “if commit was not called, the OffsetGuard destructor rolls back”.

This is the same “commit after the write” ordering as the old engine, but expressed as a scope: the offset advances if and only if the block survived to the end of the scope. Every early return, every exception, every cancellation rolls it back automatically. In this version the offsets live in ClickHouse Keeper rather than in Kafka itself, so the engine can be replicated.

Trade-offs

What the ordering buys: a crash never loses a message that ClickHouse had not yet written. The failure mode is duplicates, which a downstream ReplacingMergeTree or a deduplicating materialized view can absorb.

What it costs:

There is no exactly-once. The window between “the block is in the MergeTree” and “the offset is committed” is a window in which a crash duplicates. The exception message names the missing piece: “insert atomicity / transactions is needed here”.
The commit is a synchronous round trip to the broker, in the middle of the streaming loop. A broker that cannot accept the commit (no replicas for __consumer_offsets) stalls the whole stream.
kafka_commit_every_batch = true trades duplicates for loss — the right choice when a message’s content makes an INSERT idempotent anyway, and the wrong choice when it does not.
The rewind is per-partition and per-block, and the “rewind” is itself a commit, so an aborted block costs an extra broker round trip.
A SELECT from a Kafka table is destructive and, with a block size of 1, slow. It is a debugging tool, not a query interface.

Wrapping up

The one thing to remember: the commit is after the write — a deliberate choice of “duplicates on crash” over “data loss on crash”, enforced by the structure of the code (executor.execute() then source->commit()) rather than by a comment.

The part people get wrong: a crash between the write and the commit duplicates the block, and nothing in ClickHouse cleans that up for you. If your pipeline cannot tolerate duplicates, the destination engine has to be a deduplicating one.

What to go do about it: watch KafkaCommitFailures in system.events. It should be zero. If it is not, the error message in the server log is the exact sentence quoted above.