Compression on Opensource internals

How ClickHouse's Sparse Column Stores Only the Rows That Differ

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Real columns are mostly default. A status column is 'active' for 98% of rows and '' for the rest. A retries column is 0 almost always. An amount column that is Nullable is NULL for most rows and non-null for a handful.

Storing all those zeros is paying for bytes you could have described in one sentence. ClickHouse has two answers to this, and they live in different layers:

ColumnNullable keeps a separate byte-per-row null map next to the value column. The values are still all there; the null map says which ones to ignore.
ColumnSparse keeps only the non-default values plus a sorted array of the row positions where they live. The default values are not stored at all — they are implied.

ColumnSparse is the more aggressive of the two, and it is the more interesting one, because the offsets array is not a list of positions but a run-length encoding of gaps, and the whole thing has to survive being filtered, permuted, replicated and merged.

This article is about ColumnSparse and its serializer, in ClickHouse 26.9 (04eac2ea3).

The layout

The header comment says it in four lines:

 cpp
 
 
// src/Columns/ColumnSparse.h:13
/** Column for spare representation.
 * It stores column with non-default values and column
 * with their sorted positions in original column. Column with
 * values contains also one default value at 0 position to make
 * implementation of execution of functions and sorting more convenient.
 */

and the members:

 cpp
 
 
// src/Columns/ColumnSparse.h:268
 /// Contains default value at 0 position.
 /// It's convenient, because it allows to execute, e.g functions or sorting,
 /// for this column without handling different cases.
 WrappedPtr values;

 /// Sorted offsets of non-default values in the full column.
 /// 'offsets[i]' corresponds to 'values[i + 1]'.
 WrappedPtr offsets;
 size_t _size; /// NOLINT

So the invariant is precise: offsets[i] is the row number of the value stored at values[i + 1]. values[0] is always the default value, and is never referenced by an offset.

A worked example. A UInt32 column of 10 rows where only rows 2, 5 and 9 are non-zero:

text

 row: 0 1 2 3 4 5 6 7 8 9
 value: 0 0 17 0 0 42 0 0 0 99

 values: [ 0, 17, 42, 99 ] <- values[0] is the default
 offsets: [ 2, 5, 9 ] <- offsets[i] -> values[i+1]
 _size: 10

To read row n you binary-search offsets for n:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:975
size_t ColumnSparse::getValueIndex(size_t n) const
{
 chassert(n < _size);

 const auto & offsets_data = getOffsetsData();
 const auto * it = std::lower_bound(offsets_data.begin(), offsets_data.end(), n);
 if (it == offsets_data.end() || *it != n)
 return 0;

 return it - offsets_data.begin() + 1;
}

getValueIndex returns 0 for a default row (meaning “use values[0]”) and i+1 otherwise. The header documents the cost:

 cpp
 
 
// src/Columns/ColumnSparse.h:186
 /// Return position of element in 'values' columns,
 /// that corresponds to n-th element of full column.
 /// O(log(offsets.size())) complexity,
 size_t getValueIndex(size_t n) const;

So random access to a sparse column is a binary search — O(log k) where k is the number of non-default values — plus one indirection into the values column. For a column with 2% non-default rows in a million-row block, that is 20 comparisons instead of a single array index. That is the price of the encoding, and it is why ColumnSparse is a storage optimization that the query layer tries to peel off before doing real work.

The constructor enforces the invariant

The invariants are checked at construction, not just assumed:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:35
ColumnSparse::ColumnSparse(MutableColumnPtr && values_, MutableColumnPtr && offsets_, size_t size_)
 : values(std::move(values_)), offsets(std::move(offsets_)), _size(size_)
{
 const ColumnUInt64 * offsets_concrete = typeid_cast<const ColumnUInt64 *>(offsets.get());

 if (!offsets_concrete)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "'offsets' column must be a ColumnUInt64, got: {}", offsets->getName());

 /// 'values' should contain one extra element: default value at 0 position.
 if (offsets->size() + 1 != values->size())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Values size ({}) is inconsistent with offsets size ({})", values->size(), offsets->size());

 if (_size < offsets->size())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Size of sparse column ({}) cannot be lower than number of non-default values ({})", _size, offsets->size());

 if (!offsets_concrete->empty() && _size <= offsets_concrete->getData().back())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Size of sparse column ({}) should be greater than last position of non-default value ({})",
 _size, offsets_concrete->getData().back());

#ifdef DEBUG_OR_SANITIZER_BUILD
 const auto & offsets_data = getOffsetsData();
 const auto * it = std::adjacent_find(offsets_data.begin(), offsets_data.end(), std::greater_equal<>());
 if (it != offsets_data.end())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Offsets of ColumnSparse must be strictly sorted");
#endif
}

Note the asymmetry: the size checks run in release builds, but the strictly sorted check is debug-only (#ifdef DEBUG_OR_SANITIZER_BUILD), because it is O(n) and a sortedness violation would already have produced garbage from the binary search.

Inserting

Insertion is where the “values[0] is the default” convention pays off. You insert the value into the values column and then decide whether to keep it:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:152
void ColumnSparse::insertSingleValue(const Inserter & inserter)
{
 inserter(*values);

 size_t last_idx = values->size() - 1;
 if (values->isDefaultAt(last_idx))
 values->popBack(1);
 else
 getOffsetsData().push_back(_size);

 ++_size;
}

That is the whole trick in six lines: append the value, and if it turned out to be the default, pop it back and do not record an offset. Otherwise record the row number. The values column therefore only ever contains non-default values (plus the sentinel at index 0), and the offsets array is strictly increasing by construction.

Note that isDefaultAt is the type’s notion of default — for UInt32 it is “equals zero”, for String it is “empty”, for Nullable it is “NULL”:

 cpp
 
 
// src/Columns/ColumnVector.h:292
 bool isDefaultAt(size_t n) const override
 {
 if constexpr (is_floating_point<T>)
 {
 /// For floating-point types, use bit_cast to compare raw bit patterns instead of
 /// arithmetic equality. IEEE 754 defines -0.0 == +0.0, so the arithmetic check
 /// would incorrectly treat -0.0 as the default value, losing the sign on
 /// deserialization. Comparing bits directly distinguishes the two: +0.0 is
 /// all-zero bits, while -0.0 has its sign bit set.
 ///
 /// std::conditional_t selects an unsigned integer type of the same size as T,
 /// satisfying the requirement of std::bit_cast that both types have equal size.
 /// Unsigned integers are chosen because their value equals their bit pattern,
 /// making the comparison to 0 unambiguous.
 using Bits = std::conditional_t<sizeof(T) == 2, UInt16,
 std::conditional_t<sizeof(T) == 4, UInt32, UInt64>>;
 return std::bit_cast<Bits>(data[n]) == 0;
 }
 else
 {
 return data[n] == T{};
 }
 }

That -0.0 handling is a nice detail: for floating-point columns the default check is a bit comparison, not an arithmetic one, so that -0.0 is not mistaken for 0.0 and its sign survives a round trip.

The iterator: amortized O(1) sequential access

The binary search is fine for random access, but a scan over a sparse column should not do a binary search per row. The Iterator class exists for that, and it maintains two cursors — the current row and the current offset:

 cpp
 
 
// src/Columns/ColumnSparse.h:206
 /// This class helps to iterate over all values in ColumnSparse.
 class Iterator
 {
 public:
 Iterator(const PaddedPODArray<UInt64> & offsets_, size_t size_, size_t current_offset_, size_t current_row_)
 : offsets(offsets_), offsets_size(offsets.size()), size(size_), current_offset(current_offset_), current_row(current_row_)
 {
 }

 bool ALWAYS_INLINE isDefault() const { return current_offset == offsets_size || current_row != offsets[current_offset]; }
 size_t ALWAYS_INLINE getValueIndex() const { return isDefault() ? 0 : current_offset + 1; }
 size_t ALWAYS_INLINE getCurrentRow() const { return current_row; }
 size_t ALWAYS_INLINE getCurrentOffset() const { return current_offset; }
 size_t ALWAYS_INLINE increaseCurrentRow() const { return ++current_row; }
 size_t ALWAYS_INLINE increaseCurrentOffset() const { return ++current_offset; }

 /// Moves the iterator forward to the given row, which must not be behind the current one.
 /// Amortized constant time when the visited rows are increasing.
 void ALWAYS_INLINE advanceToRow(size_t row)
 {
 while (current_offset < offsets_size && offsets[current_offset] < row)
 ++current_offset;
 current_row = row;
 }

advanceToRow walks the offsets array forward, never backward, so a monotonically increasing scan is O(rows + offsets) in total, not O(rows log offsets). This is the standard “galloping cursor” pattern.

On disk: the offsets become run-length groups

The in-memory representation stores each non-default row’s position explicitly. On disk, that is wasteful: a million rows with 2% non-default values would store 20 000 absolute positions, when what you really want to say is “200 default rows, then a value, then 300 default rows, then a value, …”.

That is what serializeOffsets does:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:70
void serializeOffsets(const IColumn::Offsets & offsets, WriteBuffer & ostr, size_t start, size_t end)
{
 size_t size = offsets.size();
 for (size_t i = 0; i < size; ++i)
 {
 size_t group_size = offsets[i] - start;
 writeVarUInt(group_size, ostr);
 start += group_size + 1;
 }

 size_t group_size = start < end ? end - start : 0;
 group_size |= END_OF_GRANULE_FLAG;
 writeVarUInt(group_size, ostr);
}

Read that carefully. For each non-default row it writes the number of default rows that precede it (offsets[i] - start), and then advances start past that value too (start += group_size + 1). At the end it writes the trailing run of default rows with a special flag bit set.

The flag is bit 62:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:43
{

/// 2^62, because VarInt supports only values < 2^63.
constexpr auto END_OF_GRANULE_FLAG = 1ULL << 62;

Because the offsets are VarInt-encoded, the values are small and the whole offsets stream for a 2%-sparse million-row column is a few kilobytes of VarInts, most of them one or two bytes long.

Decoding it

The decoder has to be resumable, because a read may stop in the middle of a group — the reader asked for limit rows and the group runs past it. The state carries the leftovers:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:48
struct DeserializeStateSparse : public ISerialization::DeserializeBinaryBulkState
{
 /// Number of default values, that remain from previous read.
 size_t num_trailing_defaults = 0;
 /// Do we have non-default value after @num_trailing_defaults?
 bool has_value_after_defaults = false;
 ISerialization::DeserializeBinaryBulkStatePtr nested;

 void reset()
 {
 num_trailing_defaults = 0;
 has_value_after_defaults = false;
 }

and the main loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:125
 size_t group_size = 0;
 while (!istr.eof())
 {
 readVarUInt(group_size, istr);

 bool end_of_granule = group_size & END_OF_GRANULE_FLAG;
 group_size &= ~END_OF_GRANULE_FLAG;

 size_t next_total_rows = total_rows + group_size;
 group_size += num_trailing_defaults;

 if (unlikely(next_total_rows >= limit))
 {
 /// If it was not last group in granule,
 /// we have to add current non-default value at further reads.
 state.num_trailing_defaults = next_total_rows - limit;
 state.has_value_after_defaults = !end_of_granule;
 return limit;
 }

 if (unlikely(end_of_granule))
 {
 has_value_after_defaults = false;
 num_trailing_defaults = group_size;
 }
 else
 {
 /// If we add value to column for first time in current read,
 /// start from column's current size, because it can have some defaults after last offset,
 /// otherwise just start from previous offset.
 /// After the first push `first` is false and `offsets` is non-empty, so the
 /// `!offsets.empty()` guard from the original is redundant and dropped here.
 size_t start_of_group = first ? start : offsets.back() + 1;
 offsets.push_back(start_of_group + group_size);
 first = false;

 num_trailing_defaults = 0;
 ++next_total_rows;
 }

 total_rows = next_total_rows;
 }

The two state variables handle the two ways a read can be cut short:

num_trailing_defaults — the read stopped in the middle of a run of default rows. Those defaults belong to the next read.
has_value_after_defaults — the read stopped exactly at a boundary where a non-default value is pending. The next read has to emit it.

Both are stored in the serialization state, not on the stack, precisely so that a partial read can be resumed later.

The decoder also reserves space for the offsets array using a prior guess:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:109
 /// Just try to guess number of offsets.
 offsets.reserve(offsets.size()
 + static_cast<size_t>(static_cast<double>(limit) * (1.0 - ColumnSparse::DEFAULT_RATIO_FOR_SPARSE_SERIALIZATION)));

using the constant

 cpp
 
 
// src/Columns/ColumnSparse.h:29
 static constexpr auto DEFAULT_ROWS_SEARCH_SAMPLE_RATIO = 0.1;
 static constexpr auto DEFAULT_RATIO_FOR_SPARSE_SERIALIZATION = 0.95;

So the guess is “5% of the rows are non-default”.

How the encoding gets chosen

The sparse encoding is not something you ask for per column. It is selected by the part writer based on the fraction of default values in the column, and it is recorded in the part’s metadata so the reader knows what to expect:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:415
ISerialization::KindStack SerializationInfo::chooseKindStack(const Data & data, const Settings & settings)
{
 ISerialization::KindStack kind_stack = {ISerialization::Kind::DEFAULT};
 double ratio = data.num_rows ? std::min(static_cast<double>(data.num_defaults) / static_cast<double>(data.num_rows), 1.0) : 0.0;
 if (ratio > settings.ratio_of_defaults_for_sparse)
 kind_stack.push_back(ISerialization::Kind::SPARSE);
 return kind_stack;
}

The threshold is the MergeTree table setting ratio_of_defaults_for_sparse_serialization, default 0.9375:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:97
 DECLARE(Float, ratio_of_defaults_for_sparse_serialization, 0.9375f, R"(
Minimal ratio of the number of _default_ values to the number of _all_ values
in a column. Setting this value causes the column to be stored using sparse
serializations.

The kind_stack is a stack of wrappers, not a single flag, because a column can be Sparse inside Replicated inside Sparse, and the order matters:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:266
/// The order in which the kinds wrap each other, innermost first: `ColumnSparse` can sit inside
/// `ColumnReplicated` but not the other way round (see `removeSpecialRepresentations`), and nothing
/// wraps a `ColumnBLOB`. Not the order of the enum, whose values are part of the Native format.

and the reconstruction is a loop over the stack:

 cpp
 
 
// src/DataTypes/IDataType.cpp:456
SerializationPtr IDataType::wrapSerializationBasedOnKindStack(SerializationPtr serialization, const ISerialization::KindStack & kind_stack, const SerializationInfoSettings & settings) const
{
 for (auto kind : kind_stack)
 {
 if (settings.canUseSparseSerialization(*this) && kind == ISerialization::Kind::SPARSE)
 serialization = SerializationSparse::create(serialization);
 else if (kind == ISerialization::Kind::DETACHED)
 serialization = SerializationDetached::create(serialization);
 else if (kind == ISerialization::Kind::REPLICATED)
 serialization = SerializationReplicated::create(serialization);
 }

 return serialization;
}

The kind stack is serialized into serialization.json in the part, so a reader knows to expect a sparse column without having to guess:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:235
void SerializationInfo::serializeKindStackBinary(WriteBuffer & out) const
{
 if (kind_stack == ISerialization::KindStack{ISerialization::Kind::DEFAULT})
 {
 writeBinary(static_cast<UInt8>(KindStackBinarySerializationType::DEFAULT), out);
 }
 else if (kind_stack == ISerialization::KindStack{ISerialization::Kind::DEFAULT, ISerialization::Kind::SPARSE})
 {
 writeBinary(static_cast<UInt8>(KindStackBinarySerializationType::SPARSE), out);
 }

The three substreams

A sparse column on disk is three substreams: the offsets, the elements, and (for a nullable nested type) a null map:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:259
void SerializationSparse::enumerateStreams(
 EnumerateStreamsSettings & settings, const StreamCallback & callback, const SubstreamData & data) const
{
 const auto * column_sparse = data.column ? typeid_cast<const ColumnSparse *>(data.column.get()) : nullptr;
 size_t column_size = column_sparse ? column_sparse->size() : 0;

 settings.path.push_back(Substream::SparseOffsets);
 auto offsets_data = SubstreamData(SerializationNumber<UInt64>::create())
 .withColumn(column_sparse ? column_sparse->getOffsetsPtr() : nullptr)
 .withSerializationInfo(data.serialization_info);

 settings.path.back().data = offsets_data;
 callback(settings.path);

 if (sparse_null_map)
 {
 const auto * type_nullable = data.type ? &assert_cast<const DataTypeNullable &>(*data.type) : nullptr;
 settings.path.back() = Substream::SparseNullMap;
 settings.path.back().creator = std::make_shared<NullMapSubcolumnCreator>(offsets_data.column, column_size);
 auto null_map_data = SubstreamData(sparse_null_map)
 .withType(type_nullable ? std::make_shared<DataTypeUInt8>() : nullptr)
 .withColumn(nullptr) /// subcolumn will be created in NullMapSubcolumnCreator based on offsets
 .withSerializationInfo(data.serialization_info);

 settings.path.back().data = null_map_data;
 callback(settings.path);
 }

 settings.path.back() = Substream::SparseElements;
 settings.path.back().creator = std::make_shared<SubcolumnCreator>(offsets_data.column, column_size);
 settings.path.back().data = data;

The null map of a sparse nullable column is not stored — it is derived from the offsets, because “the row is NULL” and “the row is default” are the same statement for a nullable column:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:248
ColumnPtr SerializationSparse::NullMapSubcolumnCreator::create(const ColumnPtr & /* prev */) const
{
 auto null_map_col = ColumnUInt8::create();
 auto & null_map_data = null_map_col->getData();
 null_map_data.resize_fill(size, static_cast<UInt8>(1));
 const auto & offsets_data = assert_cast<const ColumnUInt64 &>(*offsets).getData();
 for (UInt64 offset : offsets_data)
 null_map_data[offset] = 0;
 return null_map_col;
}

All ones, then punch a zero at each non-default row. That is the whole null map, reconstructed from the offsets array.

Writing

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:311
void SerializationSparse::serializeBinaryBulkWithMultipleStreams(
 const IColumn & column, size_t offset, size_t limit, SerializeBinaryBulkSettings & settings, SerializeBinaryBulkStatePtr & state) const
{
 size_t size = column.size();

 auto offsets_column = DataTypeNumber<IColumn::Offset>().createColumn();
 auto & offsets_data = assert_cast<ColumnVector<IColumn::Offset> &>(*offsets_column).getData();
 column.getIndicesOfNonDefaultRows(offsets_data, offset, limit);

 settings.path.push_back(Substream::SparseOffsets);
 if (auto * stream = settings.getter(settings.path))
 {
 size_t end = limit && offset + limit < size ? offset + limit : size;
 serializeOffsets(offsets_data, *stream, offset, end);
 }

 settings.path.back() = Substream::SparseElements;
 if (!offsets_data.empty())
 {
 if (const auto * column_sparse = typeid_cast<const ColumnSparse *>(&column))
 {
 const auto & values = column_sparse->getValuesColumn();
 size_t begin = column_sparse->getValueIndex(offsets_data[0]);
 size_t end = column_sparse->getValueIndex(offsets_data.back());
 nested->serializeBinaryBulkWithMultipleStreams(values, begin, end - begin + 1, settings, state);
 }
 else
 {
 auto values = column.index(*offsets_column, 0);
 nested->serializeBinaryBulkWithMultipleStreams(*values, 0, values->size(), settings, state);
 }
 }

Note the two cases in the elements branch. If the column is already a ColumnSparse in memory, the values sub-column is contiguous and only the relevant slice is written. If it is a plain column that merely has many defaults, the writer first materializes the non-default values with column.index(*offsets_column, 0).

The query layer wants to peel the sparse wrapper off

Because random access to a sparse column is a binary search, most query operations want the full column. ColumnSparse therefore provides convertToFullColumnIfSparse:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:147
ColumnPtr ColumnSparse::convertToFullColumnIfSparse() const
{
 return values->createWithOffsets(getOffsetsData(), *createColumnConst(values, 0), _size, /*shift=*/ 1);
}

createWithOffsets is the generic mechanism: it takes a values column and an offsets array and produces a full column where each value is repeated offsets[i] - offsets[i-1] times.

There is one place where the engine keeps the sparse representation: when a function has exactly one sparse argument and the function is deterministic, it can be executed on the values only, and the result re-wrapped. That saves decompressing the default rows entirely:

 cpp
 
 
// src/Functions/IFunction.cpp:795
 if (num_sparse_columns == 1 && num_full_columns == 0 && isCallDeterministicInScopeOfQuery(arguments))
 {
 auto & arg_with_sparse = columns_without_sparse[sparse_column_position];
 ColumnPtr sparse_offsets;
 {
 /// New scope to avoid possible mistakes on dangling reference.
 auto & column_sparse = assert_cast<const ColumnSparse &>(*arg_with_sparse.column);
 sparse_offsets = column_sparse.getOffsetsPtr();
 arg_with_sparse.column = column_sparse.getValuesPtr();
 }

 size_t values_size = arg_with_sparse.column->size();
 for (size_t i = 0; i < columns_without_sparse.size(); ++i)
 {
 if (i == sparse_column_position)
 continue;

 columns_without_sparse[i].column = columns_without_sparse[i].column->cloneResized(values_size);
 }

 auto res = executeWithoutSparseColumns(columns_without_sparse, result_type, values_size, dry_run);

 if (isColumnConst(*res))
 return res->cloneResized(input_rows_count);

 /// If default of sparse column is changed after execution of function, convert to full column.
 /// If there are any default in non-zero position after execution of function, convert to full column.
 /// Currently there is no easy way to rebuild sparse column with new offsets.
 if (!result_type->canBeInsideSparseColumns() || !res->isDefaultAt(0) || res->getNumberOfDefaultRows() != 1)
 {
 const auto & offsets_data = assert_cast<const ColumnVector<UInt64> &>(*sparse_offsets).getData();
 return res->createWithOffsets(offsets_data, *createColumnConst(res, 0), input_rows_count, /*shift=*/1);
 }

 return ColumnSparse::create(res, sparse_offsets, input_rows_count);
 }

The condition at the end is the interesting part: the result can only stay sparse if (a) the result type is allowed inside a sparse column, (b) the result’s default value is still the default (position 0 is the only default), and (c) there is exactly one default row. If the function changed the default value, or produced extra defaults in the middle, the sparse structure no longer matches and the column is expanded. The comment says so plainly: “Currently there is no easy way to rebuild sparse column with new offsets.”

That is the honest limitation of this design, stated in the source.

The other side of the coin: the null map

ColumnNullable takes the opposite approach: keep everything, add a byte per row saying which rows are NULL.

 cpp
 
 
// src/Columns/ColumnNullable.h:19
/// Class that specifies nullable columns. A nullable column represents
/// a column, which may have any type, provided with the possibility of
/// storing NULL values. For this purpose, a ColumNullable object stores
/// an ordinary column along with a special column, namely a byte map,
/// whose type is ColumnUInt8. The latter column indicates whether the
/// value of a given row is a NULL or not. Such a design is preferred
/// over a bitmap because columns are usually stored on disk as compressed
/// files. In this regard, using a bitmap instead of a byte map would
/// greatly complicate the implementation with little to no benefits.

The comment explains the design choice: a bitmap would be 8× smaller, but columns are stored compressed, and a byte map of mostly zeros compresses just as well as a bitmap while being trivially indexable. The null map is literally a ColumnUInt8:

 cpp
 
 
// src/Columns/ColumnNullable.h:16
using NullMap = ColumnUInt8::Container;
using ConstNullMapPtr = const NullMap *;

and it is a separate column, which means it is a separate compression stream, a separate substream on disk, and separately filterable:

 cpp
 
 
// src/DataTypes/Serializations/SerializationNullable.cpp:124
 if (!use_default_null_map)
 {
 /// First serialize null map.
 settings.path.push_back(Substream::NullMap);
 if (auto * stream = settings.getter(settings.path))
 SerializationNumber<UInt8>::create()->serializeBinaryBulk(col.getNullMapColumn(), *stream, offset, limit);
 settings.path.pop_back();
 }

The null map is written to its own substream, before the values.

The value column of a Nullable still has one entry per row — the value under a NULL is undefined, but it occupies a slot. insertData shows the convention:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:183
void ColumnNullable::insertData(const char * pos, size_t length)
{
 if (pos == nullptr)
 {
 getNestedColumn().insertDefault();
 getNullMapData().push_back(true);
 }
 else
 {
 getNestedColumn().insertData(pos, length);
 getNullMapData().push_back(false);
 }
}

A NULL still pushes a default into the value column, to keep the two arrays the same length. And the two must stay in sync even if the push throws:

 cpp
 
 
// src/Columns/ColumnNullable.h:97
 void insertDefault() override
 {
 getNestedColumn().insertDefault();
 /// Keep the nested column and the null map in sync even if appending to the
 /// null map throws (e.g. on a memory limit).
 try
 {
 getNullMapData().push_back(true);
 }
 catch (...)
 {
 getNestedColumn().popBack(1);
 throw;
 }
 }

The null map is also where the “combine two nullable columns” logic lives, and it has a subtlety worth reading:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:932
void ColumnNullable::applyNullMapImpl(const NullMap & map, size_t offset)
{
 NullMap & arr = getNullMapData();

 if (offset + map.size() != arr.size())
 throw Exception(
 ErrorCodes::LOGICAL_ERROR,
 "Null map of size {} at offset {} does not match ColumnNullable of size {}",
 map.size(), offset, arr.size());

 /// Any non-zero byte means NULL, so reduce the map before negating it: `negative ^ 2` would
 /// yield 3 and mark a row the map leaves present.
 for (size_t i = 0, size = map.size(); i < size; ++i)
 arr[offset + i] |= negative ^ !!map[i];
}

The null map is a byte map where any non-zero means NULL. When negating, the code has to normalize the incoming map to 0/1 first (!!map[i]), because the bitwise XOR against negative would otherwise produce values like 3 from a 2 — which would still read as “non-zero means NULL” for this map but would corrupt subsequent operations.

Seeing it in SQL

The setting is a MergeTree table setting:

 sql
 
 
CREATE TABLE my_sparse_table
(
 `id` UInt64,
 `s` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS ratio_of_defaults_for_sparse_serialization = 0.95;

INSERT INTO my_sparse_table
SELECT
 number,
 number % 20 = 0 ? toString(number): ''
FROM
 numbers(10000000);

and you can check what happened:

 sql
 
 
-- Which columns got the sparse encoding?
SELECT column, serialization_kind FROM system.parts_columns
WHERE table = 'my_sparse_table';

 sql
 
 
-- And how much did it save?
SELECT table, name, data_compressed_bytes, data_uncompressed_bytes FROM system.columns
WHERE table LIKE 'my_%_table';

The documentation in the setting itself shows the effect on a table where 95% of the s column is the empty string: the sparse version stores fewer bytes both compressed and uncompressed.

Trade-offs

What the sparse encoding buys you.

A column that is 95% default costs 5% of what it would otherwise cost, plus a small offsets array. And the offsets array is not 8 bytes per non-default row on disk — it is a VarInt run-length encoding of the gaps, so a column with long runs of defaults costs almost nothing per gap.

Better still, the sparse representation is propagated: a function applied to a sparse column is executed on the values only, so the CPU cost is proportional to the non-default rows too, not just the memory cost.

What it costs you.

Random access becomes a binary search. A scan is fine (the Iterator is amortized O(1)), but a WHERE row_id = 42 on a sparse column is O(log k) plus an indirection, where a plain column would be one array index. That is why the query layer tries to convert sparse columns to full ones as early as possible — convertToFullColumnIfSparse is called all over the MergeTree read path.

The structure is also fragile in one specific way: the invariant that “the only default is at position 0” must hold for the column to stay sparse. A function that changes the default value, or produces additional defaults in the middle of the column, forces a full expansion. The source says so without embarrassment.

Sparse vs Nullable. These are not competitors; they solve different problems. Nullable answers “is this row NULL?” and pays one byte per row plus the value slot. Sparse answers “what are the rows that are not default?” and pays only for those rows — but it only applies to default values, and defaultness is a property of the type, not of the data. A nullable column whose non-null values are all distinct is not sparse at all; it is a Nullable with a null map.

The design decision that makes it work. The offsets array is stored as gaps, not positions. That single choice turns “8 bytes per non-default row” into “one VarInt per run of default rows”, which for real data (long runs of defaults) is a factor of hundreds. Everything else — the resumable decoder state, the END_OF_GRANULE_FLAG, the derived null map — exists to make that choice work across granule boundaries and partial reads.

ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: LowCardinality(T) stores a dictionary of distinct values once and keeps a 1–8 byte index per row. The dictionary is shared across every block, granule and part, so twelve distinct values cost one byte per row instead of ten. It pays off below ~10 000 distinct values and hurts above ~100 000.

Every column-oriented database eventually meets the same data: a column of country codes, of status strings, of user agents, of enum-like labels. A hundred thousand rows, twelve distinct values, repeated over and over.

Storing the value in every row is absurd. Storing the position of the value in every row is not.

ClickHouse’s answer is a column type called LowCardinality(T), and the implementation is one of the more interesting pieces of the whole storage engine, because it has to solve three separate problems at once:

how to represent the dictionary and the index in memory,
how to make the same dictionary serve many different blocks, granules and parts, so the dictionary is stored once and not once per block,
how to serialize that arrangement on disk so a reader can rebuild it — including the case where the dictionary had to change halfway through.

Let’s take all three. Source quoted from ClickHouse 26.9 (04eac2ea3).

The shape of the thing

The header comment of ColumnLowCardinality states the design in four lines:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:17
/**
 * How data is stored (in a nutshell):
 * we have a dictionary @e reverse_index in ColumnUnique that holds pairs (DataType, UIntXX) and a column
 * with UIntXX holding actual data indices.
 * To obtain the value's index, call #getOrFindIndex.
 * To operate on the data (so called indices column), call #getIndexes.
 *
 * @note The indices column always contains the default value (empty std::string_view) with the first index.
 */

The class holds exactly two members:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:398
 Dictionary dictionary;
 ColumnIndex idx;

Dictionary is a thin wrapper around a ColumnUnique plus a shared flag:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:366
private:
 class Dictionary
 {
 public:
 Dictionary(const Dictionary & other) = default;
 explicit Dictionary(MutableColumnPtr && column_unique, bool is_shared);
 explicit Dictionary(ColumnPtr column_unique, bool is_shared);

 const WrappedPtr & getColumnUniquePtr() const { return column_unique; }
 WrappedPtr & getColumnUniquePtr() { return column_unique; }

 const IColumnUnique & getColumnUnique() const { return static_cast<const IColumnUnique &>(*column_unique); }
 IColumnUnique & getColumnUnique() { return static_cast<IColumnUnique &>(*column_unique); }

 /// Dictionary may be shared for several mutable columns.
 /// Immutable columns may have the same column unique, which isn't necessarily shared dictionary.
 void setShared(const ColumnPtr & column_unique_);
 bool isShared() const { return shared; }

 /// Create new dictionary with only keys that are mentioned in indexes.
 void compact(MutableColumnPtr & indexes);
 /// Create nullable dictionary with only keys that are mentioned in indexes.
 void compactToNullable(MutableColumnPtr & indexes);

 static MutableColumnPtr compact(const IColumnUnique & column_unique, MutableColumnPtr & indexes);
 static MutableColumnPtr compactToNullable(const IColumnUnique & column_unique, MutableColumnPtr & indexes);

 private:
 WrappedPtr column_unique;
 bool shared = false;
 };

So a LowCardinality column is:

text

 +---------------------------+
 row 0 -----> | dictionary[ idx[0] ] | idx is a ColumnVector<UInt8|UInt16|UInt32|UInt64>
 row 1 -----> | dictionary[ idx[1] ] | dictionary is a ColumnUnique<ColumnString> (etc.)
 row 2 -----> | dictionary[ idx[2] ] |
 ... +---------------------------+

The values are in the dictionary; the rows are indices. Getting row n is two loads: getIndexes().getUInt(n), then a lookup in the dictionary.

That second load is a random access into a (hopefully tiny) array — and the whole point is that the dictionary array is tiny enough to stay in L1/L2 while the index array streams past it.

The index column grows its own type

The index column is not fixed-width. It starts as UInt8 and widens to UInt16, UInt32, UInt64 as the dictionary outgrows the range. The widening logic lives in ColumnIndex:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:135
void ColumnIndex::expandType()
{
 auto expand = [&](auto type)
 {
 using CurIndexType = decltype(type);
 constexpr auto next_size = NumberTraits::nextSize(sizeof(CurIndexType));
 if (next_size == sizeof(CurIndexType))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't expand indexes type for ColumnLowCardinality from type: {}",
 demangle(typeid(CurIndexType).name()));

 using NewIndexType = typename NumberTraits::Construct<false, false, next_size>::Type;
 convertIndexes<NewIndexType>();
 };

 callForType(std::move(expand), size_of_type);
}

and the bound that decides when to widen:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:152
size_t ColumnIndex::getMaxIndexForCurrentType() const
{
 chassert(size_of_type == 1 || size_of_type == 2 || size_of_type == 4 || size_of_type == 8);
 /// Shifting by 64 is undefined, and this is called per inserted value, so no `callForType` here.
 if (size_of_type == sizeof(UInt64))
 return std::numeric_limits<UInt64>::max();
 return (1ULL << (8 * size_of_type)) - 1;
}

The per-value insert path is where the widening happens, and note the comment about avoiding a typeid_cast on the hot path:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:198
void ColumnIndex::insertIndex(size_t index)
{
 while (index > getMaxIndexForCurrentType())
 expandType();

 auto insert = [&]<typename CurIndexType>(CurIndexType /*type_value*/)
 {
 /// `size_of_type` tracks the type of `indexes` and `expandType` keeps the two in sync, so
 /// this is `static_cast` in release. `getIndexesData` would run a `typeid_cast` instead, and
 /// this is the per-value path.
 assert_cast<ColumnVector<CurIndexType> *>(indexes.get())->getData().push_back(static_cast<CurIndexType>(index));
 };

 callForType(std::move(insert), size_of_type);

 /// Same invariant, so re-deriving it from `indexes` is a debug-only check here.
 chassert(size_of_type == getSizeOfIndexType(*indexes, size_of_type));
}

callForType is a hand-rolled dispatch over the four possible widths — no virtual call, no switch on a runtime type id, just four instantiations of a lambda. This is the pattern used throughout ClickHouse’s hot column code: the virtual IColumn interface is for the cold, structural paths; the per-row paths are devirtualized by dispatching on the concrete type once and then running a monomorphic loop.

Reading an index back has the same shape, and is marked ALWAYS_INLINE because it is called from the tightest loops in the engine:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:326
 ALWAYS_INLINE size_t getIndexAt(size_t row) const
 {
 const IColumn * indexes = &getIndexes();

 switch (idx.getSizeOfIndexType())
 {
 case sizeof(UInt8): return assert_cast<const ColumnUInt8 *>(indexes)->getElement(row);
 case sizeof(UInt16): return assert_cast<const ColumnUInt16 *>(indexes)->getElement(row);
 case sizeof(UInt32): return assert_cast<const ColumnUInt32 *>(indexes)->getElement(row);
 case sizeof(UInt64): return assert_cast<const ColumnUInt64 *>(indexes)->getElement(row);
 default: throwUnexpectedLowCardinalityIndexType(idx.getSizeOfIndexType());
 }
 }

The dictionary: ColumnUnique and its reverse index

ColumnUnique is the dictionary. It is not a column type you can select; the comment says so:

 cpp
 
 
// src/Columns/ColumnUnique.h:36
/** Stores another column with unique values
 * and also an index that allows to find position by value.
 *
 * This column is not used on it's own but only as implementation detail of ColumnLowCardinality.
 */
template <typename ColumnType>
class ColumnUnique final : public COWHelper<IColumnUnique, ColumnUnique<ColumnType>>

Its members are the values plus the lookup structure:

 cpp
 
 
// src/Columns/ColumnUnique.h:203
private:
 IColumn::WrappedPtr column_holder;
 bool is_nullable;
 size_t size_of_value_if_fixed = 0;

 ReverseIndex<UInt64, ColumnType> reverse_index;

 /// For DataTypeNullable, stores null map.
 IColumn::WrappedPtr nested_null_mask;
 IColumn::WrappedPtr nested_column_nullable;

Two reserved slots

The first row of the dictionary is special: it is the default value of the type. If the dictionary is nullable, row 0 is the NULL and row 1 is the default value of the inner type:

 cpp
 
 
// src/Columns/ColumnUnique.h:74
 size_t getDefaultValueIndex() const override { return 0; }
 size_t getNullValueIndex() const override;
 size_t getNestedTypeDefaultValueIndex() const override { return is_nullable ? 1 : 0; }
 bool canContainNulls() const override { return is_nullable; }

and

 cpp
 
 
// src/Columns/ColumnUnique.h:370
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::getNullValueIndex() const
{
 if (!is_nullable)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ColumnUnique can't contain null values.");

 return 0;
}

The count of reserved slots is therefore one or two:

 cpp
 
 
// src/Columns/ColumnUnique.h:238
 static size_t numSpecialValues(bool is_nullable) { return is_nullable ? 2 : 1; }
 size_t numSpecialValues() const { return numSpecialValues(is_nullable); }

Those reserved slots are not in the reverse index, which is why the insert path has to special-case them by hand:

 cpp
 
 
// src/Columns/ColumnUnique.h:513
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsertData(const char * pos, size_t length)
{
 /// The reserved prefix slots are not in the reverse index, so the default value is matched here.
 /// Comparing the first byte first keeps the `memcmp` call out of this path for the values that
 /// are not the default, which is all of them in a typical dictionary.
 const size_t default_index = getNestedTypeDefaultValueIndex();
 const std::string_view default_value = getRawColumnPtr()->getDataAt(default_index);
 if (default_value.size() == length && (length == 0 || (default_value[0] == pos[0] && default_value == std::string_view(pos, length))))
 return default_index;

 auto insertion_point = reverse_index.insert({pos, length});

 updateNullMask();

 return insertion_point;
}

The comment is a nice piece of micro-optimisation reasoning: comparing the first byte before the whole value keeps the expensive memcmp off the path for the (overwhelmingly common) case where the value is not the default.

The reverse index stores row numbers, not values

ReverseIndex is the interesting part. A naive dictionary would be a HashMap<value, index> — the values stored twice, once in the column and once as hash-table keys. ReverseIndex inverts that: the hash table stores the row number in the dictionary column, and comparisons dereference back into the column:

 cpp
 
 
// src/Columns/ReverseIndex.h:94
 /// Special case when we want to compare with something not in index_column.
 /// When we compare something inside column default keyEquals checks only that row numbers are equal.
 bool keyEquals(std::string_view object, size_t hash_ [[maybe_unused]], const State & state) const
 {
 auto index = key;
 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return hash_ == (*state.saved_hash_column)[index] && object == state.index_column->getDataAt(index);
 else
 return object == state.index_column->getDataAt(index);
 }

The hash table therefore holds 8-byte integers, and the dictionary column holds the only copy of the data. For a LowCardinality(String) with 1000 distinct values, that is the difference between a hash table with 1000 string keys (plus their hashes, plus their lengths) and a hash table with 1000 UInt64s.

For string columns the hash is expensive, so it is precomputed once and kept in a parallel ColumnUInt64:

 cpp
 
 
// src/Columns/ReverseIndex.h:43
template <typename ColumnType>
struct ReverseIndexHashTableState<ColumnType, /* with_saved_hash = */ true, /* has_base_index */ false>
{
 constexpr static bool with_saved_hash = true;
 constexpr static bool has_base_index = false;

 ColumnType * index_column;
 typename ColumnVector<UInt64>::Container * saved_hash_column;
};

and the hash function for a stored row reads the precomputed hash rather than re-hashing the string:

 cpp
 
 
// src/Columns/ReverseIndex.h:108
 size_t getHash(const Hash & hash) const
 {
 auto index = key;

 /// Hack. HashTable is Hash itself.
 const auto & state = static_cast<const State &>(static_cast<const HashTable &>(hash));

 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return (*state.saved_hash_column)[index];
 else
 {
 using ValueType = typename ColumnType::ValueType;
 ValueType value = unalignedLoad<ValueType>(state.index_column->getDataAt(index).data());
 return DefaultHash<ValueType>()(value);
 }
 }

The index is built lazily, and the hash table is filled in one pass

The reverse index is not built at construction time; it is built on the first insert, and the build is a single pass over the dictionary:

 cpp
 
 
// src/Columns/ReverseIndex.h:435
template <typename IndexType, typename ColumnType>
void ReverseIndex<IndexType, ColumnType>::buildIndex()
{
 if (index)
 return;

 if (!column)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ReverseIndex can't build index because index column wasn't set.");

 auto size = column->size();
 index = std::make_unique<IndexMapType>(size);

 if constexpr (use_saved_hash)
 saved_hash = calcHashes();

 auto & state = index->getState();
 state.index_column = column;
 state.base_index = base_index;
 if constexpr (use_saved_hash)
 state.saved_hash_column = &saved_hash->getData();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 for (auto row : collections::range(num_prefix_rows_to_skip, size))
 {
 UInt64 hash = 0;
 if constexpr (use_saved_hash)
 hash = saved_hash->getElement(row);
 else
 hash = getHash(column->getDataAt(row));

 index->reverseIndexEmplace(row + base_index, iterator, inserted, hash, column->getDataAt(row));

 if (!inserted)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Duplicating keys found in ReverseIndex.");
 }
}

Note num_prefix_rows_to_skip: the reserved NULL/default slots are skipped, because they are matched by hand in uniqueInsertData.

The insert path is the mirror image, and it is written to be branch-light for the two different column kinds:

 cpp
 
 
// src/Columns/ReverseIndex.h:490
template <typename IndexType, typename ColumnType>
UInt64 ReverseIndex<IndexType, ColumnType>::insert(std::string_view data)
{
 if (!index)
 buildIndex();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 auto hash = getHash(data);
 UInt64 num_rows = size();

 if constexpr (use_saved_hash)
 {
 auto & column_data = saved_hash->getData();
 if (column_data.size() <= num_rows)
 column_data.resize(num_rows + 1);
 column_data[num_rows] = hash;
 }
 else
 column->insertData(data.data(), data.size());

 index->reverseIndexEmplace(num_rows + base_index, iterator, inserted, hash, data);

 if constexpr (use_saved_hash)
 {
 if (inserted)
 column->insertData(data.data(), data.size());
 }
 else
 {
 if (!inserted)
 column->popBack(1);
 }

 return iterator->getValue();
}

For strings the hash is appended optimistically and the value is appended only if the key was new. For fixed-size values the value is appended optimistically and popped back if the key turned out to be a duplicate. Each kind pushes and pops the cheaper thing.

Float NaNs are canonicalized

A LowCardinality(Float64) dictionary must not contain two NaNs, because NaN == NaN is false but all NaNs must compare equal in the dictionary semantics. So NaN is normalized at insert time:

 cpp
 
 
// src/Columns/ColumnUnique.h:389
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsert(const Field & x)
{
 if (x.isNull())
 return getNullValueIndex();

 // NaN can contain different sign or mantissa bits, but we need to consider all NaNs equal.
 if constexpr (is_float_vector_v<ColumnType>)
 {
 if (isNaN(x.safeGet<typename ColumnType::ValueType>()))
 {
 auto nan = NaNOrZero<typename ColumnType::ValueType>();
 return uniqueInsertData(reinterpret_cast<char *>(&nan), sizeof(nan));
 }
 }

Here is where the design pays off. A LowCardinality column can point at a dictionary owned by another column. The shared flag on Dictionary records this, and byteSize() deliberately does not count a shared dictionary:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:200
 /// Don't count the dictionary size as it can be shared between different blocks.
 size_t byteSize() const override { return idx.getIndexes()->byteSize() + (isSharedDictionary() ? 0 : getDictionary().byteSize()); }

 size_t byteSizeAt(size_t n) const override { return getDictionary().byteSizeAt(getIndexes().getUInt(n)); }
 size_t allocatedBytes() const override { return idx.getIndexes()->allocatedBytes() + getDictionary().allocatedBytes(); }

Note the asymmetry: byteSize() (the logical size, used for statistics and for system.parts_columns) excludes the shared dictionary, but allocatedBytes() (the physical size, used for memory accounting) includes it. That distinction is what makes the accounting correct when a part has a hundred blocks all pointing at one dictionary.

Sharing also propagates through every operation that produces a new column from an old one — cut, filter, permute, index, replicate all hand the same dictionary pointer to the result:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:130
 ColumnPtr filter(const Filter & filt, ssize_t result_size_hint) const override
 {
 return ColumnLowCardinality::create(
 dictionary.getColumnUniquePtr(), getIndexes().filter(filt, result_size_hint), isSharedDictionary());
 }

But a shared dictionary is immutable

The moment you want to mutate a column whose dictionary is shared with somebody else, the dictionary has to be un-shared. This happens in compactIfSharedDictionary(), which is called at the top of every mutating insert:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:146
void ColumnLowCardinality::insert(const Field & x)
{
 compactIfSharedDictionary();
 idx.insertIndex(getDictionary().uniqueInsert(x));
}

bool ColumnLowCardinality::tryInsert(const Field & x)
{
 compactIfSharedDictionary();

 size_t index = 0;
 if (!dictionary.getColumnUnique().tryUniqueInsert(x, index))
 return false;

 idx.insertIndex(index);
 return true;
}

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:894
void ColumnLowCardinality::compactIfSharedDictionary()
{
 if (dictionary.isShared())
 compactInplace();
}

Compaction rebuilds a private dictionary containing only the keys that are actually referenced by this column’s indexes, and remaps the indexes accordingly:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:1002
MutableColumnPtr ColumnLowCardinality::Dictionary::compact(const IColumnUnique & unique, MutableColumnPtr & indexes)
{
 auto new_column_unique = unique.cloneEmpty();
 auto & new_unique = static_cast<IColumnUnique &>(*new_column_unique);

 auto unique_indexes = mapUniqueIndex(*indexes);
 auto sub_keys = unique.getNestedColumn()->index(*unique_indexes, 0);
 auto new_indexes = new_unique.uniqueInsertRangeFrom(*sub_keys, 0, sub_keys->size());

 indexes = IColumn::mutate(new_indexes->index(*indexes, 0));
 return new_column_unique;
}

mapUniqueIndex is the piece that does the renumbering. It is a counting-sort-like trick that works in O(n) with a flat array, and it has an overflow path for when the dictionary is shared and the indexes do not fit the “max index < number of rows” assumption:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:79
template <typename T>
MutableColumnPtr mapUniqueIndexImpl(PaddedPODArray<T> & index)
{
 if (index.empty())
 return ColumnVector<T>::create();

 auto size = index.size();

 T max_val = index[0];
 for (size_t i = 1; i < size; ++i)
 max_val = std::max(max_val, index[i]);

 /// May happen when dictionary is shared.
 if (max_val > size)
 return mapUniqueIndexImplRef(index);

 auto map_size = static_cast<UInt64>(max_val) + 1;
 PaddedPODArray<T> map(map_size, 0);
 T zero_pos_value = index[0];
 index[0] = 0;
 T cur_pos = 0;
 for (size_t i = 1; i < size; ++i)
 {
 T val = index[i];
 if (val != zero_pos_value && map[val] == 0)
 {
 ++cur_pos;
 map[val] = cur_pos;
 }

 index[i] = map[val];
 }

The “overflow” case (max_val > size) means the column references dictionary entries that belong to a bigger dictionary — exactly what happens when the dictionary is shared. Then a hash map is used instead of a flat array:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:53
template <typename T>
MutableColumnPtr mapUniqueIndexImplRef(PaddedPODArray<T> & index)
{
 PaddedPODArray<T> copy(index.cbegin(), index.cend());

 HashMap<T, T> hash_map;
 for (auto val : index)
 hash_map.insert({val, static_cast<T>(hash_map.size())});

 auto res_col = ColumnVector<T>::create();
 auto & data = res_col->getData();

 data.resize(hash_map.size());
 for (const auto & val : hash_map)
 data[val.getMapped()] = val.getKey();

 for (auto & ind : index)
 ind = hash_map[ind];

 for (size_t i = 0; i < index.size(); ++i)
 if (data[index[i]] != copy[i])
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected {}, but got {}", toString(data[index[i]]), toString(copy[i]));

 return res_col;
}

That last loop is a self-check: after the renumbering, the new index must still resolve to the same value. Cheap insurance against an off-by-one in a data structure that is otherwise very hard to test.

Inserting from a column with the same dictionary

Once dictionaries are shared, a very common operation becomes trivial: copying a row from another LowCardinality column that happens to use the same dictionary is just a copy of an index.

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:180
 size_t position = low_cardinality_src->getIndexes().getUInt(n);

 if (&low_cardinality_src->getDictionary() == &getDictionary())
 {
 /// Dictionary is shared with src column. Insert only index.
 idx.insertIndex(position);
 }
 else
 {
 compactIfSharedDictionary();
 const auto & nested = *low_cardinality_src->getDictionary().getNestedColumn();
 idx.insertIndex(getDictionary().uniqueInsertFrom(nested, position));
 }

The test is a pointer comparison of the dictionary addresses. If they are the same object, the index is valid as-is; otherwise the value has to be re-inserted into this column’s dictionary and the resulting index used.

On disk: one dictionary per part, versioned per granule

Now the hard part. In a MergeTree part, a LowCardinality column is written as two streams: DictionaryKeys and DictionaryIndexes.

The keys stream holds the dictionary; the indexes stream holds, per granule, a header and then the index data. The dictionary is not rewritten per granule — it accumulates until it hits a size limit, at which point it is flushed and a new one starts.

The version byte

The keys stream starts with a version:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:102
struct KeysSerializationVersion
{
 enum Value
 {
 /// Version is written at the start of <name.dict.bin>.
 /// Dictionary is written as number N and N keys after them.
 /// Dictionary can be shared for continuous range of granules, so some marks may point to the same position.
 /// Shared dictionary is stored in state and is read once.
 SharedDictionariesWithAdditionalKeys = 1,
 };

 Value value;

 static void checkVersion(UInt64 version)
 {
 if (version != SharedDictionariesWithAdditionalKeys)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Invalid version for SerializationLowCardinality key column.");
 }

 explicit KeysSerializationVersion(UInt64 version) : value(static_cast<Value>(version)) { checkVersion(version); }
};

The single legal version is called SharedDictionariesWithAdditionalKeys, and the comment tells you the whole story in three lines: the dictionary is written as a count followed by that many keys, it can be shared across a continuous range of granules, and the shared dictionary is kept in the serialization state and read once.

The per-granule header

Every granule’s index data begins with a header that encodes the index width plus three flags:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:124
/// Version is stored at the start of each granule. It's used to store indexes type and flags.
struct IndexesSerializationType
{
 using SerializationType = UInt64;
 /// Need to read dictionary if it wasn't.
 static constexpr SerializationType NeedGlobalDictionaryBit = 1u << 8u;
 /// Need to read additional keys. Additional keys are stored before indexes as value N and N keys after them.
 static constexpr SerializationType HasAdditionalKeysBit = 1u << 9u;
 /// Need to update dictionary. It means that previous granule has different dictionary.
 static constexpr SerializationType NeedUpdateDictionary = 1u << 10u;

 enum Type
 {
 TUInt8 = 0,
 TUInt16,
 TUInt32,
 TUInt64,
 };

 Type type;
 bool has_additional_keys;
 bool need_global_dictionary;
 bool need_update_dictionary;

 static constexpr SerializationType resetFlags(SerializationType type)
 {
 return type & (~(HasAdditionalKeysBit | NeedGlobalDictionaryBit | NeedUpdateDictionary));
 }

The three flags are the entire state machine:

NeedGlobalDictionaryBit — “there is a dictionary in the keys stream that I have not read yet”.
HasAdditionalKeysBit — “this granule carries keys that did not fit in the shared dictionary; they are stored inline, before the index data”.
NeedUpdateDictionary — “the dictionary changed since the previous granule”.

Serialization and deserialization are symmetric and trivial:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:162
 void serialize(WriteBuffer & buffer) const
 {
 SerializationType val = type;
 if (has_additional_keys)
 val |= HasAdditionalKeysBit;
 if (need_global_dictionary)
 val |= NeedGlobalDictionaryBit;
 if (need_update_dictionary)
 val |= NeedUpdateDictionary;
 writeBinaryLittleEndian(val, buffer);
 }

 void deserialize(ReadBuffer & buffer, const ISerialization::DeserializeBinaryBulkSettings & settings)
 {
 SerializationType val = 0;
 readBinaryLittleEndian(val, buffer);

 checkType(val);
 has_additional_keys = (val & HasAdditionalKeysBit) != 0;
 need_global_dictionary = (val & NeedGlobalDictionaryBit) != 0;
 need_update_dictionary = (val & NeedUpdateDictionary) != 0;
 type = static_cast<Type>(resetFlags(val));

 if (settings.native_format)
 {
 if (need_global_dictionary)
 throw Exception(ErrorCodes::INCORRECT_DATA,
 "LowCardinality indexes serialization type for Native format "
 "cannot use global dictionary");
 }
 }

The low four bits carry the index width; bits 8, 9, 10 are the flags. Bits 4–7 are unused — a hole left for future flags.

Writing: the dictionary accumulates, then overflows

The writer keeps the accumulating dictionary in the serialization state:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:232
struct SerializeStateLowCardinality : public ISerialization::SerializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 MutableColumnUniquePtr shared_dictionary;

 explicit SerializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}
};

The write path is where the interesting decisions live:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:561
 const ColumnLowCardinality & low_cardinality_column = typeid_cast<const ColumnLowCardinality &>(column);

 auto * low_cardinality_state = checkAndGetState<SerializeStateLowCardinality>(state);
 auto & global_dictionary = low_cardinality_state->shared_dictionary;
 KeysSerializationVersion::checkVersion(low_cardinality_state->key_version.value);

 size_t max_limit = column.size() - offset;
 limit = limit ? std::min(limit, max_limit) : max_limit;

 /// Do not write anything for empty column. (May happen while writing empty arrays.)
 if (limit == 0)
 return;

 bool need_update_dictionary = global_dictionary == nullptr;
 if (need_update_dictionary)
 global_dictionary = DataTypeLowCardinality::createColumnUnique(*dictionary_type);

 auto sub_column = low_cardinality_column.cutAndCompact(offset, limit);
 ColumnPtr positions = sub_column->getIndexesPtr();
 ColumnPtr keys = sub_column->getDictionary().getNestedColumn();

 if (settings.low_cardinality_max_dictionary_size)
 {
 /// Insert used_keys into global dictionary and update sub_index.
 auto indexes_with_overflow = global_dictionary->uniqueInsertRangeWithOverflow(*keys, 0, keys->size(),
 settings.low_cardinality_max_dictionary_size);

 if (global_dictionary->size() > settings.low_cardinality_max_dictionary_size)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Got dictionary with size {} but max dictionary size is {}",
 global_dictionary->size(), settings.low_cardinality_max_dictionary_size);

 positions = indexes_with_overflow.indexes->index(*positions, 0);
 keys = std::move(indexes_with_overflow.overflowed_keys);

 if (global_dictionary->size() < settings.low_cardinality_max_dictionary_size && !keys->empty())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Has additional keys, but dict size is {} which is less "
 "then max dictionary size ({})", global_dictionary->size(),
 settings.low_cardinality_max_dictionary_size);
 }

cutAndCompact is the first step: it takes the slice of the column being written and produces a private dictionary containing only the keys that slice actually uses. That is what keeps the accumulated dictionary small — without it, every granule would re-insert the whole dictionary.

Then the keys are merged into the global dictionary, and anything that does not fit within low_cardinality_max_dictionary_size becomes “overflowed keys” — keys carried inline by that granule.

The flush happens when the accumulated dictionary reaches the limit:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:602
 bool need_additional_keys = !keys->empty();
 bool need_dictionary = settings.low_cardinality_max_dictionary_size != 0;
 bool need_write_dictionary = !settings.low_cardinality_use_single_dictionary_for_part
 && global_dictionary->size() >= settings.low_cardinality_max_dictionary_size;

 IndexesSerializationType index_version(*positions, need_additional_keys, need_dictionary, need_update_dictionary);
 index_version.serialize(*indexes_stream);

 if (need_write_dictionary)
 {
 const auto & nested_column = global_dictionary->getNestedNotNullableColumn();
 UInt64 num_keys = nested_column->size();
 writeBinaryLittleEndian(num_keys, *keys_stream);
 dict_inner_serialization->serializeBinaryBulk(*nested_column, *keys_stream, 0, num_keys);
 low_cardinality_state->shared_dictionary = nullptr;
 }

 if (need_additional_keys)
 {
 UInt64 num_keys = keys->size();
 writeBinaryLittleEndian(num_keys, *indexes_stream);
 dict_inner_serialization->serializeBinaryBulk(*keys, *indexes_stream, 0, num_keys);
 }

 UInt64 num_rows = positions->size();
 writeBinaryLittleEndian(num_rows, *indexes_stream);
 auto index_serialization = index_version.getDataType()->getDefaultSerialization();
 index_serialization->serializeBinaryBulk(*positions, *indexes_stream, 0, num_rows);

The default limit is 8192 keys:

 cpp
 
 
// src/Core/Settings.cpp:5045
 DECLARE(UInt64, low_cardinality_max_dictionary_size, 8192, R"(
Sets a maximum size in rows of a shared global dictionary for the [LowCardinality](/reference/data-types/lowcardinality) data type that can be written to a storage file system. This setting prevents issues with RAM in case of unlimited dictionary growth. All the data that can't be encoded due to maximum dictionary size limitation ClickHouse writes in an ordinary method.

Possible values:

- Any positive integer.
)", 0) \

So: with an 8192-key limit, a part whose column has 100 distinct values writes the dictionary once; a part with 100 000 distinct values writes the dictionary roughly every 8192 keys and starts a new one.

Note that need_update_dictionary is global_dictionary == nullptr — i.e. “this granule starts a new dictionary”. And the dictionary is flushed after the header was written, so the header says “read the dictionary” and the dictionary follows in the keys stream.

Reading: the reader has to reconstruct the same state

The read side keeps the shared dictionary in the deserialize state, along with a flag that survives granule skips:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:240
struct DeserializeStateLowCardinality : public ISerialization::DeserializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 ColumnUniquePtr global_dictionary;

 IndexesSerializationType index_type{};
 ColumnPtr additional_keys;
 ColumnPtr null_map;
 UInt64 num_pending_rows = 0;

 /// If dictionary should be updated.
 /// Can happen is some granules was skipped while reading from MergeTree.
 /// We should store this flag in State because
 /// in case of long block of empty arrays we may not need read dictionary at first reading.
 bool need_update_dictionary = false;

 /// True if this part has a single dictionary for the entire column. The prefix
 /// reader first checks `has_uniform_marks_callback` as a cheap prefilter, then
 /// verifies it by reading the first dictionary and checking that `DictionaryKeys`
 /// reached EOF. When true, non-continuous reads (granule skips) do not invalidate
 /// the cached dictionary, and the dictionary can be shared across all threads via `clone`.
 bool single_dictionary_for_part = false;

 explicit DeserializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}

 ISerialization::DeserializeBinaryBulkStatePtr cloneImpl(ISerialization::DeserializeBinaryBulkStateCloneContext &) const override
 {
 return std::make_shared<DeserializeStateLowCardinality>(*this);
 }
};

The need_update_dictionary comment is the interesting one: when reading is not continuous (i.e. some granules were skipped), the reader cannot assume that the dictionary it holds is the one the next granule refers to, so it must force a re-read. That flag lives in the state, not on the stack, precisely because a skipped granule may be discovered long after the state was created.

The main read loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:752
 if (!settings.continuous_reading)
 {
 low_cardinality_state->num_pending_rows = 0;

 /// Some granules were skipped; the next granule header may signal a dictionary
 /// update. For multi-dict parts this forces a re-read at the next boundary.
 low_cardinality_state->need_update_dictionary = true;
 }

 while (limit)
 {
 if (low_cardinality_state->num_pending_rows == 0)
 {
 if (indexes_stream->eof())
 break;

 auto & index_type = low_cardinality_state->index_type;
 auto & global_dictionary = low_cardinality_state->global_dictionary;

 index_type.deserialize(*indexes_stream, settings);

 bool need_update_dictionary =
 !global_dictionary || index_type.need_update_dictionary || low_cardinality_state->need_update_dictionary;
 if (index_type.need_global_dictionary && need_update_dictionary)
 {
 /// For single-dict parts the in-memory dictionary is valid for the entire part,
 /// so skip re-reading it even if the stream was seeked back to the beginning.
 if (!low_cardinality_state->single_dictionary_for_part || !global_dictionary)
 read_dictionary();
 low_cardinality_state->need_update_dictionary = false;
 }

 if (low_cardinality_state->index_type.has_additional_keys)
 read_additional_keys();
 else
 low_cardinality_state->additional_keys = nullptr;

 readBinaryLittleEndian(low_cardinality_state->num_pending_rows, *indexes_stream);
 }

 size_t num_rows_to_read = std::min<UInt64>(limit, low_cardinality_state->num_pending_rows);
 read_indexes(num_rows_to_read);
 limit -= num_rows_to_read;
 low_cardinality_state->num_pending_rows -= num_rows_to_read;
 }

num_pending_rows is what makes a granule’s worth of rows readable in chunks: the header is read once, then the reader consumes num_pending_rows rows from the index stream, and only when that counter hits zero does it look for the next header.

The single-dictionary-for-part fast path

There is a special case worth highlighting: a part where the column has exactly one dictionary for the whole part. The prefix reader detects it by reading the first dictionary and checking whether the DictionaryKeys stream reached EOF:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:348
 if (might_have_single_dictionary && !stream->eof())
 {
 /// For `Wide` parts that might have a single dictionary, the stream is
 /// now positioned right at the dictionary data (after the version).
 /// Read the first dictionary and check EOF afterwards. If this proves
 /// that there are no more dictionaries, all threads sharing this prefix
 /// state via `DeserializationPrefixesCache` get the dictionary through
 /// a `shared_ptr` clone instead of each reading it independently.

and

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:377
 if (stream->eof())
 {
 new_state->single_dictionary_for_part = true;
 new_state->global_dictionary = std::move(global_dictionary);
 }
 else
 {
 /// The mark heuristic can be fooled by several dictionaries written
 /// inside one mark interval. Reset the `DictionaryKeys` stream back
 /// to the normal post-prefix position and let data deserialization
 /// read dictionary updates from the stream.
 if (!settings.seek_to_start_callback)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot reset LowCardinality dictionary stream without seek_to_start_callback");

 settings.seek_to_start_callback(dictionary_keys_path);

 UInt64 repeated_keys_version = 0;
 readBinaryLittleEndian(repeated_keys_version, *stream);
 if (repeated_keys_version != keys_version)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Inconsistent version while resetting LowCardinality dictionary stream");
 }

This is a heuristic with a verification step. The cheap check (has_uniform_marks_callback) says “all marks point at the same position in the keys stream, so maybe there is only one dictionary”. The expensive check (actually reading the dictionary and seeing EOF) confirms it. If the heuristic lied, the stream is rewound and the normal path takes over.

That’s a pattern you will see repeatedly in ClickHouse’s MergeTree reader: cheap prefilter, expensive confirmation, safe fallback.

When the fast path holds, cloneImpl copies the state (including the dictionary pointer) so every thread reading the part gets the same dictionary without re-reading it.

The overflow path on the read side

When a granule carries inline “additional keys”, the reader has to merge two dictionaries: the global one and the granule-local one. mapIndexWithAdditionalKeys does that with two flat maps — one for the global dictionary’s keys and one for the overflow keys:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:515
 /// Update column and return map with old indexes.
 /// Let N is the number of distinct values which are less than max_size;
 /// old_column - column before function call;
 /// new_column - column after function call:
 /// * if old_column[i] < max_size, than
 /// dictionary_map[new_column[i]] = old_column[i]
 /// * else
 /// additional_keys_map[new_column[i]] = old_column[i] - dict_size + N
 IndexMapsWithAdditionalKeys mapIndexWithAdditionalKeys(IColumn & column, size_t dict_size)

and then the keys are stitched back together:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:726
 else
 {
 auto maps = mapIndexWithAdditionalKeys(*indexes_column, global_dictionary->size());

 auto used_keys = IColumn::mutate(global_dictionary->getNestedColumn()->index(*maps.dictionary_map, 0));

 if (!maps.additional_keys_map->empty())
 {
 if (additional_keys == nullptr)
 throw Exception(ErrorCodes::INCORRECT_DATA, "No additional keys found.");

 auto used_add_keys = additional_keys->index(*maps.additional_keys_map, 0);

 if (dictionary_type->isNullable())
 {
 ColumnPtr null_map = ColumnUInt8::create(used_add_keys->size(), static_cast<UInt8>(0));
 used_add_keys = ColumnNullable::create(used_add_keys, null_map);
 }

 used_keys->insertRangeFrom(*used_add_keys, 0, used_add_keys->size());
 }

 low_cardinality_column.insertRangeFromDictionaryEncodedColumn(*used_keys, *indexes_column);
 }

The final call re-inserts the merged keys into the destination column’s own dictionary and remaps the indexes one last time.

The whole thing in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `user_id` UInt64,
 `kind` LowCardinality(String), -- 12 distinct values in 100M rows
 `country` LowCardinality(FixedString(2))
)
ENGINE = MergeTree
ORDER BY (ts, user_id);

 sql
 
 
-- The dictionary is shared across every block of the part, so the per-column
-- cost is one byte per row (UInt8 index) plus the dictionary once.
SELECT kind, count() FROM events GROUP BY kind;

The type-level guarantee is that LowCardinality is only worth it while the dictionary stays small; the documentation says so plainly:

text

// src/DataTypes/DataTypeLowCardinality.cpp:198
The efficiency of using `LowCardinality` data type depends on data diversity. If a dictionary contains less than 10,000 distinct values, then ClickHouse mostly shows higher efficiency of data reading and storing. If a dictionary contains more than 100,000 distinct values, then ClickHouse can perform worse in comparison with using ordinary data types.

Trade-offs

What the hack buys you.

A LowCardinality(String) column with 12 distinct values costs one byte per row plus a few hundred bytes of dictionary, instead of ~10 bytes per row of strings. It also means the scan is a byte scan, not a string scan: filters, GROUP BY, and ORDER BY all operate on UInt8 indexes, and the dictionary is touched only for the rows that survive the filter.

What it costs you.

Every value access is two dependent loads: read the index, then read the dictionary entry. That is a pointer chase a plain String column does not have — although for a plain string column you also pay a pointer chase into the character arena, so the two are more comparable than they look.

The dictionary is a global per part, and the index stream has to carry a per-granule header with three flags and a row count, plus the possibility of inline overflow keys. A plain column has none of that. For a column with 100 000+ distinct values you have bought complexity and lost compression.

The sharing is also asymmetric: byteSize() excludes a shared dictionary while allocatedBytes() includes it. That is correct but subtle, and it means the two numbers in system.parts_columns can disagree in a way that looks like a bug until you know why.

The design decision that makes it work. The dictionary is not stored per block, and not per granule; it is stored per part, capped at 8192 keys, and flushed when it overflows. Everything else — the three header bits, the need_update_dictionary flag, the single_dictionary_for_part heuristic — exists to make that one decision survive partial reads, granule skips and multi-threaded readers.

Wrapping up

The one thing to remember: LowCardinality moves the repetition out of the rows and into a shared dictionary, and the whole design is about making that dictionary shared — across blocks, granules, parts, and threads.

The rule of thumb: below ~10 000 distinct values, use it. Above ~100 000, don’t. In between, measure.

What to go do about it: count the distinct values in your “status” or “country” columns (uniqExact), then wrap them. One byte per row instead of ten is not a rounding error on a hundred-million-row table.

Compression on Opensource internals

How ClickHouse's Sparse Column Stores Only the Rows That Differ

The layout

The constructor enforces the invariant

Inserting

The iterator: amortized O(1) sequential access

On disk: the offsets become run-length groups

Decoding it

How the encoding gets chosen

The three substreams

Writing

The query layer wants to peel the sparse wrapper off

The other side of the coin: the null map

Seeing it in SQL

Trade-offs

ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

The shape of the thing

The index column grows its own type

The dictionary: ColumnUnique and its reverse index

Two reserved slots

The reverse index stores row numbers, not values

The index is built lazily, and the hash table is filled in one pass

Float NaNs are canonicalized

Sharing the dictionary between columns

But a shared dictionary is immutable

Inserting from a column with the same dictionary

On disk: one dictionary per part, versioned per granule

The version byte

The per-granule header

Writing: the dictionary accumulates, then overflows

Reading: the reader has to reconstruct the same state

The single-dictionary-for-part fast path

The overflow path on the read side

The whole thing in SQL

Trade-offs

Wrapping up