Memory on Opensource internals

How ClickHouse's Sparse Column Stores Only the Rows That Differ

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Real columns are mostly default. A status column is 'active' for 98% of rows and '' for the rest. A retries column is 0 almost always. An amount column that is Nullable is NULL for most rows and non-null for a handful.

Storing all those zeros is paying for bytes you could have described in one sentence. ClickHouse has two answers to this, and they live in different layers:

ColumnNullable keeps a separate byte-per-row null map next to the value column. The values are still all there; the null map says which ones to ignore.
ColumnSparse keeps only the non-default values plus a sorted array of the row positions where they live. The default values are not stored at all — they are implied.

ColumnSparse is the more aggressive of the two, and it is the more interesting one, because the offsets array is not a list of positions but a run-length encoding of gaps, and the whole thing has to survive being filtered, permuted, replicated and merged.

This article is about ColumnSparse and its serializer, in ClickHouse 26.9 (04eac2ea3).

The layout

The header comment says it in four lines:

 cpp
 
 
// src/Columns/ColumnSparse.h:13
/** Column for spare representation.
 * It stores column with non-default values and column
 * with their sorted positions in original column. Column with
 * values contains also one default value at 0 position to make
 * implementation of execution of functions and sorting more convenient.
 */

and the members:

 cpp
 
 
// src/Columns/ColumnSparse.h:268
 /// Contains default value at 0 position.
 /// It's convenient, because it allows to execute, e.g functions or sorting,
 /// for this column without handling different cases.
 WrappedPtr values;

 /// Sorted offsets of non-default values in the full column.
 /// 'offsets[i]' corresponds to 'values[i + 1]'.
 WrappedPtr offsets;
 size_t _size; /// NOLINT

So the invariant is precise: offsets[i] is the row number of the value stored at values[i + 1]. values[0] is always the default value, and is never referenced by an offset.

A worked example. A UInt32 column of 10 rows where only rows 2, 5 and 9 are non-zero:

text

 row: 0 1 2 3 4 5 6 7 8 9
 value: 0 0 17 0 0 42 0 0 0 99

 values: [ 0, 17, 42, 99 ] <- values[0] is the default
 offsets: [ 2, 5, 9 ] <- offsets[i] -> values[i+1]
 _size: 10

To read row n you binary-search offsets for n:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:975
size_t ColumnSparse::getValueIndex(size_t n) const
{
 chassert(n < _size);

 const auto & offsets_data = getOffsetsData();
 const auto * it = std::lower_bound(offsets_data.begin(), offsets_data.end(), n);
 if (it == offsets_data.end() || *it != n)
 return 0;

 return it - offsets_data.begin() + 1;
}

getValueIndex returns 0 for a default row (meaning “use values[0]”) and i+1 otherwise. The header documents the cost:

 cpp
 
 
// src/Columns/ColumnSparse.h:186
 /// Return position of element in 'values' columns,
 /// that corresponds to n-th element of full column.
 /// O(log(offsets.size())) complexity,
 size_t getValueIndex(size_t n) const;

So random access to a sparse column is a binary search — O(log k) where k is the number of non-default values — plus one indirection into the values column. For a column with 2% non-default rows in a million-row block, that is 20 comparisons instead of a single array index. That is the price of the encoding, and it is why ColumnSparse is a storage optimization that the query layer tries to peel off before doing real work.

The constructor enforces the invariant

The invariants are checked at construction, not just assumed:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:35
ColumnSparse::ColumnSparse(MutableColumnPtr && values_, MutableColumnPtr && offsets_, size_t size_)
 : values(std::move(values_)), offsets(std::move(offsets_)), _size(size_)
{
 const ColumnUInt64 * offsets_concrete = typeid_cast<const ColumnUInt64 *>(offsets.get());

 if (!offsets_concrete)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "'offsets' column must be a ColumnUInt64, got: {}", offsets->getName());

 /// 'values' should contain one extra element: default value at 0 position.
 if (offsets->size() + 1 != values->size())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Values size ({}) is inconsistent with offsets size ({})", values->size(), offsets->size());

 if (_size < offsets->size())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Size of sparse column ({}) cannot be lower than number of non-default values ({})", _size, offsets->size());

 if (!offsets_concrete->empty() && _size <= offsets_concrete->getData().back())
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Size of sparse column ({}) should be greater than last position of non-default value ({})",
 _size, offsets_concrete->getData().back());

#ifdef DEBUG_OR_SANITIZER_BUILD
 const auto & offsets_data = getOffsetsData();
 const auto * it = std::adjacent_find(offsets_data.begin(), offsets_data.end(), std::greater_equal<>());
 if (it != offsets_data.end())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Offsets of ColumnSparse must be strictly sorted");
#endif
}

Note the asymmetry: the size checks run in release builds, but the strictly sorted check is debug-only (#ifdef DEBUG_OR_SANITIZER_BUILD), because it is O(n) and a sortedness violation would already have produced garbage from the binary search.

Inserting

Insertion is where the “values[0] is the default” convention pays off. You insert the value into the values column and then decide whether to keep it:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:152
void ColumnSparse::insertSingleValue(const Inserter & inserter)
{
 inserter(*values);

 size_t last_idx = values->size() - 1;
 if (values->isDefaultAt(last_idx))
 values->popBack(1);
 else
 getOffsetsData().push_back(_size);

 ++_size;
}

That is the whole trick in six lines: append the value, and if it turned out to be the default, pop it back and do not record an offset. Otherwise record the row number. The values column therefore only ever contains non-default values (plus the sentinel at index 0), and the offsets array is strictly increasing by construction.

Note that isDefaultAt is the type’s notion of default — for UInt32 it is “equals zero”, for String it is “empty”, for Nullable it is “NULL”:

 cpp
 
 
// src/Columns/ColumnVector.h:292
 bool isDefaultAt(size_t n) const override
 {
 if constexpr (is_floating_point<T>)
 {
 /// For floating-point types, use bit_cast to compare raw bit patterns instead of
 /// arithmetic equality. IEEE 754 defines -0.0 == +0.0, so the arithmetic check
 /// would incorrectly treat -0.0 as the default value, losing the sign on
 /// deserialization. Comparing bits directly distinguishes the two: +0.0 is
 /// all-zero bits, while -0.0 has its sign bit set.
 ///
 /// std::conditional_t selects an unsigned integer type of the same size as T,
 /// satisfying the requirement of std::bit_cast that both types have equal size.
 /// Unsigned integers are chosen because their value equals their bit pattern,
 /// making the comparison to 0 unambiguous.
 using Bits = std::conditional_t<sizeof(T) == 2, UInt16,
 std::conditional_t<sizeof(T) == 4, UInt32, UInt64>>;
 return std::bit_cast<Bits>(data[n]) == 0;
 }
 else
 {
 return data[n] == T{};
 }
 }

That -0.0 handling is a nice detail: for floating-point columns the default check is a bit comparison, not an arithmetic one, so that -0.0 is not mistaken for 0.0 and its sign survives a round trip.

The iterator: amortized O(1) sequential access

The binary search is fine for random access, but a scan over a sparse column should not do a binary search per row. The Iterator class exists for that, and it maintains two cursors — the current row and the current offset:

 cpp
 
 
// src/Columns/ColumnSparse.h:206
 /// This class helps to iterate over all values in ColumnSparse.
 class Iterator
 {
 public:
 Iterator(const PaddedPODArray<UInt64> & offsets_, size_t size_, size_t current_offset_, size_t current_row_)
 : offsets(offsets_), offsets_size(offsets.size()), size(size_), current_offset(current_offset_), current_row(current_row_)
 {
 }

 bool ALWAYS_INLINE isDefault() const { return current_offset == offsets_size || current_row != offsets[current_offset]; }
 size_t ALWAYS_INLINE getValueIndex() const { return isDefault() ? 0 : current_offset + 1; }
 size_t ALWAYS_INLINE getCurrentRow() const { return current_row; }
 size_t ALWAYS_INLINE getCurrentOffset() const { return current_offset; }
 size_t ALWAYS_INLINE increaseCurrentRow() const { return ++current_row; }
 size_t ALWAYS_INLINE increaseCurrentOffset() const { return ++current_offset; }

 /// Moves the iterator forward to the given row, which must not be behind the current one.
 /// Amortized constant time when the visited rows are increasing.
 void ALWAYS_INLINE advanceToRow(size_t row)
 {
 while (current_offset < offsets_size && offsets[current_offset] < row)
 ++current_offset;
 current_row = row;
 }

advanceToRow walks the offsets array forward, never backward, so a monotonically increasing scan is O(rows + offsets) in total, not O(rows log offsets). This is the standard “galloping cursor” pattern.

On disk: the offsets become run-length groups

The in-memory representation stores each non-default row’s position explicitly. On disk, that is wasteful: a million rows with 2% non-default values would store 20 000 absolute positions, when what you really want to say is “200 default rows, then a value, then 300 default rows, then a value, …”.

That is what serializeOffsets does:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:70
void serializeOffsets(const IColumn::Offsets & offsets, WriteBuffer & ostr, size_t start, size_t end)
{
 size_t size = offsets.size();
 for (size_t i = 0; i < size; ++i)
 {
 size_t group_size = offsets[i] - start;
 writeVarUInt(group_size, ostr);
 start += group_size + 1;
 }

 size_t group_size = start < end ? end - start : 0;
 group_size |= END_OF_GRANULE_FLAG;
 writeVarUInt(group_size, ostr);
}

Read that carefully. For each non-default row it writes the number of default rows that precede it (offsets[i] - start), and then advances start past that value too (start += group_size + 1). At the end it writes the trailing run of default rows with a special flag bit set.

The flag is bit 62:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:43
{

/// 2^62, because VarInt supports only values < 2^63.
constexpr auto END_OF_GRANULE_FLAG = 1ULL << 62;

Because the offsets are VarInt-encoded, the values are small and the whole offsets stream for a 2%-sparse million-row column is a few kilobytes of VarInts, most of them one or two bytes long.

Decoding it

The decoder has to be resumable, because a read may stop in the middle of a group — the reader asked for limit rows and the group runs past it. The state carries the leftovers:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:48
struct DeserializeStateSparse : public ISerialization::DeserializeBinaryBulkState
{
 /// Number of default values, that remain from previous read.
 size_t num_trailing_defaults = 0;
 /// Do we have non-default value after @num_trailing_defaults?
 bool has_value_after_defaults = false;
 ISerialization::DeserializeBinaryBulkStatePtr nested;

 void reset()
 {
 num_trailing_defaults = 0;
 has_value_after_defaults = false;
 }

and the main loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:125
 size_t group_size = 0;
 while (!istr.eof())
 {
 readVarUInt(group_size, istr);

 bool end_of_granule = group_size & END_OF_GRANULE_FLAG;
 group_size &= ~END_OF_GRANULE_FLAG;

 size_t next_total_rows = total_rows + group_size;
 group_size += num_trailing_defaults;

 if (unlikely(next_total_rows >= limit))
 {
 /// If it was not last group in granule,
 /// we have to add current non-default value at further reads.
 state.num_trailing_defaults = next_total_rows - limit;
 state.has_value_after_defaults = !end_of_granule;
 return limit;
 }

 if (unlikely(end_of_granule))
 {
 has_value_after_defaults = false;
 num_trailing_defaults = group_size;
 }
 else
 {
 /// If we add value to column for first time in current read,
 /// start from column's current size, because it can have some defaults after last offset,
 /// otherwise just start from previous offset.
 /// After the first push `first` is false and `offsets` is non-empty, so the
 /// `!offsets.empty()` guard from the original is redundant and dropped here.
 size_t start_of_group = first ? start : offsets.back() + 1;
 offsets.push_back(start_of_group + group_size);
 first = false;

 num_trailing_defaults = 0;
 ++next_total_rows;
 }

 total_rows = next_total_rows;
 }

The two state variables handle the two ways a read can be cut short:

num_trailing_defaults — the read stopped in the middle of a run of default rows. Those defaults belong to the next read.
has_value_after_defaults — the read stopped exactly at a boundary where a non-default value is pending. The next read has to emit it.

Both are stored in the serialization state, not on the stack, precisely so that a partial read can be resumed later.

The decoder also reserves space for the offsets array using a prior guess:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:109
 /// Just try to guess number of offsets.
 offsets.reserve(offsets.size()
 + static_cast<size_t>(static_cast<double>(limit) * (1.0 - ColumnSparse::DEFAULT_RATIO_FOR_SPARSE_SERIALIZATION)));

using the constant

 cpp
 
 
// src/Columns/ColumnSparse.h:29
 static constexpr auto DEFAULT_ROWS_SEARCH_SAMPLE_RATIO = 0.1;
 static constexpr auto DEFAULT_RATIO_FOR_SPARSE_SERIALIZATION = 0.95;

So the guess is “5% of the rows are non-default”.

How the encoding gets chosen

The sparse encoding is not something you ask for per column. It is selected by the part writer based on the fraction of default values in the column, and it is recorded in the part’s metadata so the reader knows what to expect:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:415
ISerialization::KindStack SerializationInfo::chooseKindStack(const Data & data, const Settings & settings)
{
 ISerialization::KindStack kind_stack = {ISerialization::Kind::DEFAULT};
 double ratio = data.num_rows ? std::min(static_cast<double>(data.num_defaults) / static_cast<double>(data.num_rows), 1.0) : 0.0;
 if (ratio > settings.ratio_of_defaults_for_sparse)
 kind_stack.push_back(ISerialization::Kind::SPARSE);
 return kind_stack;
}

The threshold is the MergeTree table setting ratio_of_defaults_for_sparse_serialization, default 0.9375:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:97
 DECLARE(Float, ratio_of_defaults_for_sparse_serialization, 0.9375f, R"(
Minimal ratio of the number of _default_ values to the number of _all_ values
in a column. Setting this value causes the column to be stored using sparse
serializations.

The kind_stack is a stack of wrappers, not a single flag, because a column can be Sparse inside Replicated inside Sparse, and the order matters:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:266
/// The order in which the kinds wrap each other, innermost first: `ColumnSparse` can sit inside
/// `ColumnReplicated` but not the other way round (see `removeSpecialRepresentations`), and nothing
/// wraps a `ColumnBLOB`. Not the order of the enum, whose values are part of the Native format.

and the reconstruction is a loop over the stack:

 cpp
 
 
// src/DataTypes/IDataType.cpp:456
SerializationPtr IDataType::wrapSerializationBasedOnKindStack(SerializationPtr serialization, const ISerialization::KindStack & kind_stack, const SerializationInfoSettings & settings) const
{
 for (auto kind : kind_stack)
 {
 if (settings.canUseSparseSerialization(*this) && kind == ISerialization::Kind::SPARSE)
 serialization = SerializationSparse::create(serialization);
 else if (kind == ISerialization::Kind::DETACHED)
 serialization = SerializationDetached::create(serialization);
 else if (kind == ISerialization::Kind::REPLICATED)
 serialization = SerializationReplicated::create(serialization);
 }

 return serialization;
}

The kind stack is serialized into serialization.json in the part, so a reader knows to expect a sparse column without having to guess:

 cpp
 
 
// src/DataTypes/Serializations/SerializationInfo.cpp:235
void SerializationInfo::serializeKindStackBinary(WriteBuffer & out) const
{
 if (kind_stack == ISerialization::KindStack{ISerialization::Kind::DEFAULT})
 {
 writeBinary(static_cast<UInt8>(KindStackBinarySerializationType::DEFAULT), out);
 }
 else if (kind_stack == ISerialization::KindStack{ISerialization::Kind::DEFAULT, ISerialization::Kind::SPARSE})
 {
 writeBinary(static_cast<UInt8>(KindStackBinarySerializationType::SPARSE), out);
 }

The three substreams

A sparse column on disk is three substreams: the offsets, the elements, and (for a nullable nested type) a null map:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:259
void SerializationSparse::enumerateStreams(
 EnumerateStreamsSettings & settings, const StreamCallback & callback, const SubstreamData & data) const
{
 const auto * column_sparse = data.column ? typeid_cast<const ColumnSparse *>(data.column.get()) : nullptr;
 size_t column_size = column_sparse ? column_sparse->size() : 0;

 settings.path.push_back(Substream::SparseOffsets);
 auto offsets_data = SubstreamData(SerializationNumber<UInt64>::create())
 .withColumn(column_sparse ? column_sparse->getOffsetsPtr() : nullptr)
 .withSerializationInfo(data.serialization_info);

 settings.path.back().data = offsets_data;
 callback(settings.path);

 if (sparse_null_map)
 {
 const auto * type_nullable = data.type ? &assert_cast<const DataTypeNullable &>(*data.type) : nullptr;
 settings.path.back() = Substream::SparseNullMap;
 settings.path.back().creator = std::make_shared<NullMapSubcolumnCreator>(offsets_data.column, column_size);
 auto null_map_data = SubstreamData(sparse_null_map)
 .withType(type_nullable ? std::make_shared<DataTypeUInt8>() : nullptr)
 .withColumn(nullptr) /// subcolumn will be created in NullMapSubcolumnCreator based on offsets
 .withSerializationInfo(data.serialization_info);

 settings.path.back().data = null_map_data;
 callback(settings.path);
 }

 settings.path.back() = Substream::SparseElements;
 settings.path.back().creator = std::make_shared<SubcolumnCreator>(offsets_data.column, column_size);
 settings.path.back().data = data;

The null map of a sparse nullable column is not stored — it is derived from the offsets, because “the row is NULL” and “the row is default” are the same statement for a nullable column:

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:248
ColumnPtr SerializationSparse::NullMapSubcolumnCreator::create(const ColumnPtr & /* prev */) const
{
 auto null_map_col = ColumnUInt8::create();
 auto & null_map_data = null_map_col->getData();
 null_map_data.resize_fill(size, static_cast<UInt8>(1));
 const auto & offsets_data = assert_cast<const ColumnUInt64 &>(*offsets).getData();
 for (UInt64 offset : offsets_data)
 null_map_data[offset] = 0;
 return null_map_col;
}

All ones, then punch a zero at each non-default row. That is the whole null map, reconstructed from the offsets array.

Writing

 cpp
 
 
// src/DataTypes/Serializations/SerializationSparse.cpp:311
void SerializationSparse::serializeBinaryBulkWithMultipleStreams(
 const IColumn & column, size_t offset, size_t limit, SerializeBinaryBulkSettings & settings, SerializeBinaryBulkStatePtr & state) const
{
 size_t size = column.size();

 auto offsets_column = DataTypeNumber<IColumn::Offset>().createColumn();
 auto & offsets_data = assert_cast<ColumnVector<IColumn::Offset> &>(*offsets_column).getData();
 column.getIndicesOfNonDefaultRows(offsets_data, offset, limit);

 settings.path.push_back(Substream::SparseOffsets);
 if (auto * stream = settings.getter(settings.path))
 {
 size_t end = limit && offset + limit < size ? offset + limit : size;
 serializeOffsets(offsets_data, *stream, offset, end);
 }

 settings.path.back() = Substream::SparseElements;
 if (!offsets_data.empty())
 {
 if (const auto * column_sparse = typeid_cast<const ColumnSparse *>(&column))
 {
 const auto & values = column_sparse->getValuesColumn();
 size_t begin = column_sparse->getValueIndex(offsets_data[0]);
 size_t end = column_sparse->getValueIndex(offsets_data.back());
 nested->serializeBinaryBulkWithMultipleStreams(values, begin, end - begin + 1, settings, state);
 }
 else
 {
 auto values = column.index(*offsets_column, 0);
 nested->serializeBinaryBulkWithMultipleStreams(*values, 0, values->size(), settings, state);
 }
 }

Note the two cases in the elements branch. If the column is already a ColumnSparse in memory, the values sub-column is contiguous and only the relevant slice is written. If it is a plain column that merely has many defaults, the writer first materializes the non-default values with column.index(*offsets_column, 0).

The query layer wants to peel the sparse wrapper off

Because random access to a sparse column is a binary search, most query operations want the full column. ColumnSparse therefore provides convertToFullColumnIfSparse:

 cpp
 
 
// src/Columns/ColumnSparse.cpp:147
ColumnPtr ColumnSparse::convertToFullColumnIfSparse() const
{
 return values->createWithOffsets(getOffsetsData(), *createColumnConst(values, 0), _size, /*shift=*/ 1);
}

createWithOffsets is the generic mechanism: it takes a values column and an offsets array and produces a full column where each value is repeated offsets[i] - offsets[i-1] times.

There is one place where the engine keeps the sparse representation: when a function has exactly one sparse argument and the function is deterministic, it can be executed on the values only, and the result re-wrapped. That saves decompressing the default rows entirely:

 cpp
 
 
// src/Functions/IFunction.cpp:795
 if (num_sparse_columns == 1 && num_full_columns == 0 && isCallDeterministicInScopeOfQuery(arguments))
 {
 auto & arg_with_sparse = columns_without_sparse[sparse_column_position];
 ColumnPtr sparse_offsets;
 {
 /// New scope to avoid possible mistakes on dangling reference.
 auto & column_sparse = assert_cast<const ColumnSparse &>(*arg_with_sparse.column);
 sparse_offsets = column_sparse.getOffsetsPtr();
 arg_with_sparse.column = column_sparse.getValuesPtr();
 }

 size_t values_size = arg_with_sparse.column->size();
 for (size_t i = 0; i < columns_without_sparse.size(); ++i)
 {
 if (i == sparse_column_position)
 continue;

 columns_without_sparse[i].column = columns_without_sparse[i].column->cloneResized(values_size);
 }

 auto res = executeWithoutSparseColumns(columns_without_sparse, result_type, values_size, dry_run);

 if (isColumnConst(*res))
 return res->cloneResized(input_rows_count);

 /// If default of sparse column is changed after execution of function, convert to full column.
 /// If there are any default in non-zero position after execution of function, convert to full column.
 /// Currently there is no easy way to rebuild sparse column with new offsets.
 if (!result_type->canBeInsideSparseColumns() || !res->isDefaultAt(0) || res->getNumberOfDefaultRows() != 1)
 {
 const auto & offsets_data = assert_cast<const ColumnVector<UInt64> &>(*sparse_offsets).getData();
 return res->createWithOffsets(offsets_data, *createColumnConst(res, 0), input_rows_count, /*shift=*/1);
 }

 return ColumnSparse::create(res, sparse_offsets, input_rows_count);
 }

The condition at the end is the interesting part: the result can only stay sparse if (a) the result type is allowed inside a sparse column, (b) the result’s default value is still the default (position 0 is the only default), and (c) there is exactly one default row. If the function changed the default value, or produced extra defaults in the middle, the sparse structure no longer matches and the column is expanded. The comment says so plainly: “Currently there is no easy way to rebuild sparse column with new offsets.”

That is the honest limitation of this design, stated in the source.

The other side of the coin: the null map

ColumnNullable takes the opposite approach: keep everything, add a byte per row saying which rows are NULL.

 cpp
 
 
// src/Columns/ColumnNullable.h:19
/// Class that specifies nullable columns. A nullable column represents
/// a column, which may have any type, provided with the possibility of
/// storing NULL values. For this purpose, a ColumNullable object stores
/// an ordinary column along with a special column, namely a byte map,
/// whose type is ColumnUInt8. The latter column indicates whether the
/// value of a given row is a NULL or not. Such a design is preferred
/// over a bitmap because columns are usually stored on disk as compressed
/// files. In this regard, using a bitmap instead of a byte map would
/// greatly complicate the implementation with little to no benefits.

The comment explains the design choice: a bitmap would be 8× smaller, but columns are stored compressed, and a byte map of mostly zeros compresses just as well as a bitmap while being trivially indexable. The null map is literally a ColumnUInt8:

 cpp
 
 
// src/Columns/ColumnNullable.h:16
using NullMap = ColumnUInt8::Container;
using ConstNullMapPtr = const NullMap *;

and it is a separate column, which means it is a separate compression stream, a separate substream on disk, and separately filterable:

 cpp
 
 
// src/DataTypes/Serializations/SerializationNullable.cpp:124
 if (!use_default_null_map)
 {
 /// First serialize null map.
 settings.path.push_back(Substream::NullMap);
 if (auto * stream = settings.getter(settings.path))
 SerializationNumber<UInt8>::create()->serializeBinaryBulk(col.getNullMapColumn(), *stream, offset, limit);
 settings.path.pop_back();
 }

The null map is written to its own substream, before the values.

The value column of a Nullable still has one entry per row — the value under a NULL is undefined, but it occupies a slot. insertData shows the convention:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:183
void ColumnNullable::insertData(const char * pos, size_t length)
{
 if (pos == nullptr)
 {
 getNestedColumn().insertDefault();
 getNullMapData().push_back(true);
 }
 else
 {
 getNestedColumn().insertData(pos, length);
 getNullMapData().push_back(false);
 }
}

A NULL still pushes a default into the value column, to keep the two arrays the same length. And the two must stay in sync even if the push throws:

 cpp
 
 
// src/Columns/ColumnNullable.h:97
 void insertDefault() override
 {
 getNestedColumn().insertDefault();
 /// Keep the nested column and the null map in sync even if appending to the
 /// null map throws (e.g. on a memory limit).
 try
 {
 getNullMapData().push_back(true);
 }
 catch (...)
 {
 getNestedColumn().popBack(1);
 throw;
 }
 }

The null map is also where the “combine two nullable columns” logic lives, and it has a subtlety worth reading:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:932
void ColumnNullable::applyNullMapImpl(const NullMap & map, size_t offset)
{
 NullMap & arr = getNullMapData();

 if (offset + map.size() != arr.size())
 throw Exception(
 ErrorCodes::LOGICAL_ERROR,
 "Null map of size {} at offset {} does not match ColumnNullable of size {}",
 map.size(), offset, arr.size());

 /// Any non-zero byte means NULL, so reduce the map before negating it: `negative ^ 2` would
 /// yield 3 and mark a row the map leaves present.
 for (size_t i = 0, size = map.size(); i < size; ++i)
 arr[offset + i] |= negative ^ !!map[i];
}

The null map is a byte map where any non-zero means NULL. When negating, the code has to normalize the incoming map to 0/1 first (!!map[i]), because the bitwise XOR against negative would otherwise produce values like 3 from a 2 — which would still read as “non-zero means NULL” for this map but would corrupt subsequent operations.

Seeing it in SQL

The setting is a MergeTree table setting:

 sql
 
 
CREATE TABLE my_sparse_table
(
 `id` UInt64,
 `s` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS ratio_of_defaults_for_sparse_serialization = 0.95;

INSERT INTO my_sparse_table
SELECT
 number,
 number % 20 = 0 ? toString(number): ''
FROM
 numbers(10000000);

and you can check what happened:

 sql
 
 
-- Which columns got the sparse encoding?
SELECT column, serialization_kind FROM system.parts_columns
WHERE table = 'my_sparse_table';

 sql
 
 
-- And how much did it save?
SELECT table, name, data_compressed_bytes, data_uncompressed_bytes FROM system.columns
WHERE table LIKE 'my_%_table';

The documentation in the setting itself shows the effect on a table where 95% of the s column is the empty string: the sparse version stores fewer bytes both compressed and uncompressed.

Trade-offs

What the sparse encoding buys you.

A column that is 95% default costs 5% of what it would otherwise cost, plus a small offsets array. And the offsets array is not 8 bytes per non-default row on disk — it is a VarInt run-length encoding of the gaps, so a column with long runs of defaults costs almost nothing per gap.

Better still, the sparse representation is propagated: a function applied to a sparse column is executed on the values only, so the CPU cost is proportional to the non-default rows too, not just the memory cost.

What it costs you.

Random access becomes a binary search. A scan is fine (the Iterator is amortized O(1)), but a WHERE row_id = 42 on a sparse column is O(log k) plus an indirection, where a plain column would be one array index. That is why the query layer tries to convert sparse columns to full ones as early as possible — convertToFullColumnIfSparse is called all over the MergeTree read path.

The structure is also fragile in one specific way: the invariant that “the only default is at position 0” must hold for the column to stay sparse. A function that changes the default value, or produces additional defaults in the middle of the column, forces a full expansion. The source says so without embarrassment.

Sparse vs Nullable. These are not competitors; they solve different problems. Nullable answers “is this row NULL?” and pays one byte per row plus the value slot. Sparse answers “what are the rows that are not default?” and pays only for those rows — but it only applies to default values, and defaultness is a property of the type, not of the data. A nullable column whose non-null values are all distinct is not sparse at all; it is a Nullable with a null map.

The design decision that makes it work. The offsets array is stored as gaps, not positions. That single choice turns “8 bytes per non-default row” into “one VarInt per run of default rows”, which for real data (long runs of defaults) is a factor of hundreds. Everything else — the resumable decoder state, the END_OF_GRANULE_FLAG, the derived null map — exists to make that choice work across granule boundaries and partial reads.

How ClickHouse Stores a Column: PODArray, the Typed Array, and the Padding Trick

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

Underneath every ClickHouse data type there is exactly one data structure: a flat array of trivially-copyable elements. UInt64 is an array of UInt64. String is a flat character buffer plus an array of offsets. Nullable(UInt32) is an array of UInt32 plus an array of UInt8. There is no node, no pointer chasing, no per-row allocation.

That design is what makes ClickHouse fast, and it comes with a set of non-obvious constraints that shape everything above it:

The array must not be std::vector, because std::vector value-initializes and ClickHouse does not want to pay for zeroing bytes it is about to overwrite.
The array must be able to grow without copying more than amortized O(1) times — but the growth policy has to be chosen, not inherited.
SIMD code needs to be allowed to read and write past the end of the data.
Code that computes “the size of element i” from “the offset of element i” wants a -1 index that is guaranteed to be zero.

This article is about PODArray and ColumnVector, the layer where all of that comes together. Source is from ClickHouse 26.9 (04eac2ea3).

The container

PODArray is documented at the top of its own header:

 cpp
 
 
// src/Common/PODArray.h:33
/** A dynamic array for POD types.
 * Designed for a small number of large arrays (rather than a lot of small ones).
 * To be more precise - for use in ColumnVector.
 * It differs from std::vector in that it does not initialize the elements.
 *
 * Made noncopyable so that there are no accidental copies. You can copy the data using `assign` method.
 *
 * Only part of the std::vector interface is supported.
 *
 * The default constructor creates an empty object that does not allocate memory.
 * Then the memory is allocated at least initial_bytes bytes.
 *
 * If you insert elements with push_back, without making a `reserve`, then PODArray is about 2.5 times faster than std::vector.
 *
 * The template parameter `pad_right` - always allocate at the end of the array as many unused bytes.
 * Can be used to make optimistic reading, writing, copying with unaligned SIMD instructions.
 *
 * The template parameter `pad_left` - always allocate memory before 0th element of the array (rounded up to the whole number of elements)
 * and zero initialize -1th element. It allows to use -1th element that will have value 0.
 * This gives performance benefits when converting an array of offsets to array of sizes.

That comment is the whole design brief. Let the pieces land one at a time.

Noncopyable

 cpp
 
 
// src/Common/PODArray.h:99
template <size_t ELEMENT_SIZE, size_t initial_bytes, typename TAllocator, size_t pad_right_, size_t pad_left_>
class PODArrayBase : private boost::noncopyable, private TAllocator /// empty base optimization
{

The boost::noncopyable private base is not decoration. A Column in ClickHouse is copy-on-write: ColumnPtr is an immutable, refcounted pointer, and mutation happens by cloning first. If the underlying array were copyable, someone would eventually copy one by accident and silently double a column’s memory. Making the array noncopyable pushes every copy through the explicit assign path, where the intent is visible.

The private TAllocator is the other half of that line: the allocator is a base class, not a member, so the array’s object size does not grow with the allocator’s state. The comment in the header explains why:

 cpp
 
 
// src/Common/PODArray.h:58
 * Why Allocator is not passed through constructor, as it is done in C++ standard library?
 * Because sometimes we have many small objects, that share same allocator with same parameters,
 * and we must avoid larger object size due to storing the same parameters in each object.
 * This is required for states of aggregate functions.

Three pointers, not a size and a capacity

The array is three raw pointers:

 cpp
 
 
// src/Common/PODArray.h:119
 char * c_start = null; /// Does not include pad_left.
 char * c_end = null;
 char * c_end_of_storage = null; /// Does not include pad_right.

and the accessors are arithmetic on them:

 cpp
 
 
// src/Common/PODArray.h:227
public:
 bool empty() const { return c_end == c_start; }
 size_t size() const { return (c_end - c_start) / ELEMENT_SIZE; }
 size_t capacity() const { return (c_end_of_storage - c_start) / ELEMENT_SIZE; }

 /// This method is safe to use only for information about memory usage.
 size_t allocated_bytes() const { return c_end_of_storage - c_start + pad_right + pad_left; } /// NOLINT

 void clear() { c_end = c_start; }

size() is a subtraction and a division by a compile-time constant, which the compiler turns into a shift. There is no size_ field to keep in sync, which means there is no possibility of size_ disagreeing with the data.

The empty array points at static memory

A default-constructed PODArray must not allocate. But its c_start must still be a valid pointer, because data() may be called on it, and the padding trick needs memory before the first element. The solution is a static buffer:

 cpp
 
 
// src/Common/PODArray.h:66
static constexpr size_t empty_pod_array_size = 1024;
alignas(std::max_align_t) extern const char empty_pod_array[empty_pod_array_size];

 cpp
 
 
// src/Common/PODArray.cpp:34
/// Used for left padding of PODArray when empty
alignas(std::max_align_t) const char empty_pod_array[empty_pod_array_size]{};

1024 bytes, zero-initialized, max_align_t-aligned, and:

 cpp
 
 
// src/Common/PODArray.h:107
 /// Empty array will point to this static memory as padding and begin/end.
 static constexpr char * null = const_cast<char *>(empty_pod_array) + pad_left;

So an empty array’s c_start points 64 bytes into a static zeroed buffer, which means c_start - 1 is a valid zero byte. The empty array is not a special case in any of the arithmetic; it is a zero-length window into a shared static buffer.

That is also why the static_assert on the left padding exists:

 cpp
 
 
// src/Common/PODArray.h:110
protected:
 static_assert(pad_left <= empty_pod_array_size && "Left Padding exceeds empty_pod_array_size. Is the element size too large?");
 static_assert(pad_left % ELEMENT_SIZE == 0, "pad_left must be multiple of element alignment");

The padding trick

The two padding parameters are the most unusual part of the design.

 cpp
 
 
// src/Common/PODArray.h:103
 /// Round padding up to an whole number of elements to simplify arithmetic.
 static constexpr size_t pad_right = integerRoundUp(pad_right_, ELEMENT_SIZE);
 /// pad_left is also rounded up to 16 bytes to maintain alignment of allocated memory.
 static constexpr size_t pad_left = integerRoundUp(pad_left_, std::lcm(ELEMENT_SIZE, 16));

And the standard instantiation for columns uses 64 bytes on each side:

 cpp
 
 
// src/Common/PODArray_fwd.h:24
/** For columns. Padding is enough to read and write xmm-register at the address of the last element. */
template <typename T, size_t initial_bytes = 4096, typename TAllocator = Allocator<false>>
using PaddedPODArray = PODArray<T, initial_bytes, TAllocator, PADDING_FOR_SIMD - 1, PADDING_FOR_SIMD>;

with

 cpp
 
 
// src/Core/Defines.h:28
static constexpr auto PADDING_FOR_SIMD = 64;

So 63 bytes of right padding and 64 bytes of left padding, for every array in every column of every block. That sounds wasteful until you see what it buys.

Left padding: the `-1` element

Look at how the allocation sets it up:

 cpp
 
 
// src/Common/PODArray.h:128
 template <typename ... TAllocatorParams>
 void alloc(size_t bytes, TAllocatorParams &&... allocator_params)
 {
 char * allocated = reinterpret_cast<char *>(TAllocator::alloc(bytes, std::forward<TAllocatorParams>(allocator_params)...));

 c_start = allocated + pad_left;
 c_end = c_start;
 c_end_of_storage = allocated + bytes - pad_right;

 if (pad_left)
 memset(c_start - ELEMENT_SIZE, 0, ELEMENT_SIZE);
 }

One memset of exactly one element, immediately before the first element. The element at index -1 is guaranteed to be zero.

And the subscript operator is signed to allow it:

 cpp
 
 
// src/Common/PODArray.h:393
 /// The index is signed to access -1th element without pointer overflow.
 T & operator[] (ssize_t n)
 {
 /// <= size, because taking address of one element past memory range is Ok in C++ (expression like &arr[arr.size()] is perfectly valid).
 chassert((n >= (static_cast<ssize_t>(pad_left_) ? -1 : 0)) && (n <= static_cast<ssize_t>(this->size())));
 return t_start()[n];
 }

The -1 is only legal when there is left padding — the assertion encodes the precondition. Without left padding, n = -1 would be a genuine out-of-bounds read.

Now, who uses it? ColumnString:

 cpp
 
 
// src/Columns/ColumnString.h:43
 size_t ALWAYS_INLINE offsetAt(ssize_t i) const { return offsets[i - 1]; }

 /// Size of i-th element
 size_t ALWAYS_INLINE sizeAt(ssize_t i) const
 {
 chassert(offsets[i] >= offsets[i - 1]);
 return offsets[i] - offsets[i - 1];
 }

offsetAt(0) reads offsets[-1], which is the zeroed element. So the offsets array of a ColumnString is a classic “one past the end” offset array where the first offset is implicitly 0, and sizeAt(i) is the difference of two adjacent offsets.

The alternative would be to store offsets as “start of element i” with an explicit offsets[0] = 0 and a separate offsets[size] sentinel, and then sizeAt(i) = offsets[i+1] - offsets[i] — which needs one more slot and would require the array to have size()+1 entries, breaking the invariant that offsets.size() == column.size().

The source even flags the trick where it is used:

 cpp
 
 
// src/Columns/ColumnString.cpp:59
 const size_t src_offset = src_concrete.offsets[position - 1];
 const size_t src_buf_size
 = src_concrete.offsets[position] - src_concrete.offsets[position - 1]; /// -1th index is Ok, see PaddedPODArray.

The same trick appears in ColumnArray:

 cpp
 
 
// src/Columns/ColumnArray.h:242
 size_t ALWAYS_INLINE offsetAt(ssize_t i) const { return getOffsets()[i - 1]; }
 size_t ALWAYS_INLINE sizeAt(ssize_t i) const { return getOffsets()[i] - getOffsets()[i - 1]; }

Right padding: SIMD may write past the end

Right padding exists because ClickHouse’s SIMD filters do not write exactly the number of surviving elements. They write in whole vector-width chunks and then fix up the count:

 cpp
 
 
// src/Columns/ColumnVector.cpp:944
/// Writes the rows of the block at `data_pos` selected by `mask` to `res` and returns their number. Mixed blocks are compressed
/// with a table-driven byte shuffle. Up to `COMPRESS_MAX_OVERWRITE` bytes past the returned count may be written.
/// `res` may alias `data_pos` if it is not ahead of it (filtering in place): each store ends within the rows already loaded.

and the amount of possible overwrite is a compile-time constant:

 cpp
 
 
// src/Columns/ColumnVector.cpp:912
/// One shuffle handles 8 rows of 1-byte elements (a 256-entry table of 16 bytes would be too large for 16 rows),
/// otherwise the rows of a 16-byte vector (32-byte for `COMPRESS_DWORDS`).
template <size_t ELEMENT_WIDTH>
constexpr size_t COMPRESS_ROWS = ELEMENT_WIDTH == 1 ? 8 : (COMPRESS_DWORDS<ELEMENT_WIDTH> ? 32 : 16) / ELEMENT_WIDTH;

/// One shuffle stores a whole vector of `COMPRESS_ROWS` elements from the current count, which can be past all selected rows.
template <size_t ELEMENT_WIDTH>
constexpr size_t COMPRESS_MAX_OVERWRITE = COMPRESS_ROWS<ELEMENT_WIDTH> * ELEMENT_WIDTH;

The padding requirement is then a static assertion:

 cpp
 
 
// src/Columns/ColumnVector.cpp:1000
/// Filters whole blocks of `SIMD_ELEMENTS` rows into `res_data`.
template <typename T, typename Container, size_t SIMD_ELEMENTS>
void doFilterAlignedShuffle(const UInt8 *& filt_pos, const UInt8 *& filt_end_aligned, const T *& data_pos, Container & res_data)
{
 /// The writes of `compressBlock` past the selected rows land in the right padding, so an exact `result_size_hint` is kept.
 static_assert(Container::pad_right >= COMPRESS_MAX_OVERWRITE<sizeof(T)>);

This is the payoff of the padding design: the SIMD filter can be written without a single branch for the tail, because the buffer is guaranteed to have room for the overrun. The static_assert ties the guarantee to the code that depends on it, so the two cannot drift apart.

The same trick shows up in the small-string copy helper:

 cpp
 
 
// src/Common/PODArray.h:509
 /// Works under assumption, that it's possible to read up to 15 excessive bytes after `from_end` and this PODArray is padded.
 template <typename It1, typename It2, typename ... TAllocatorParams>
 void insertSmallAllowReadWriteOverflow15(It1 from_begin, It2 from_end, TAllocatorParams &&... allocator_params)
 {
 static_assert(pad_right_ >= PADDING_FOR_SIMD - 1);
 static_assert(sizeof(T) == sizeof(*from_begin));
 insertPrepare(from_begin, from_end, std::forward<TAllocatorParams>(allocator_params)...);
 size_t bytes_to_copy = PODArrayDetails::byte_size(from_end - from_begin, sizeof(T));
 memcpySmallAllowReadWriteOverflow15(this->c_end, reinterpret_cast<const void *>(&*from_begin), bytes_to_copy);
 this->c_end += bytes_to_copy;
 }

which copies 16 bytes at a time regardless of the actual size:

 cpp
 
 
// src/Common/memcpySmall.h:41
 inline void memcpySmallAllowReadWriteOverflow15Impl(char * __restrict dst, const char * __restrict src, ssize_t n)
 {
 __msan_unpoison_overflow_15(src, n);
 while (n > 0)
 {
 _mm_storeu_si128(reinterpret_cast<__m128i *>(dst),
 _mm_loadu_si128(reinterpret_cast<const __m128i *>(src)));

 dst += 16;
 src += 16;
 n -= 16;

 /// Avoid clang loop-idiom optimization, which transforms _mm_storeu_si128 to built-in memcpy
 __asm__ __volatile__("" : : : "memory");
 }
 }

Note the n -= 16 in a loop that runs until n <= 0: for a 5-byte copy it runs once and advances 16 bytes past the end. That is legal only because of the padding, and the static_assert above enforces it. The __msan_unpoison_overflow_15 call tells MemorySanitizer that the 15-byte overread is intentional, so sanitizers do not flag it.

The growth policy

There are three different growth behaviours in PODArray, and choosing between them is a deliberate decision at each call site.

`reserve`: round up to a power of two

 cpp
 
 
// src/Common/PODArray.h:237
 template <typename ... TAllocatorParams>
 ALWAYS_INLINE void reserve(size_t n, TAllocatorParams &&... allocator_params)
 {
 if (n > capacity())
 reallocPowerOfTwoElements(n, std::forward<TAllocatorParams>(allocator_params)...);
 }

which rounds the element count up to a power of two:

 cpp
 
 
// src/Common/PODArray.h:200
 template <typename ... TAllocatorParams>
 NO_INLINE void reallocPowerOfTwoElements(size_t n, TAllocatorParams &&... allocator_params)
 {
 realloc(roundUpToPowerOfTwoOrZero(PODArrayDetails::minimum_memory_for_elements(n, ELEMENT_SIZE, pad_left, pad_right)), std::forward<TAllocatorParams>(allocator_params)...);
 }

The rounding helper is a one-liner:

 cpp
 
 
// src/Common/BitHelpers.h:15
inline size_t roundUpToPowerOfTwoOrZero(size_t n)
{
 // if MSB is set, return n, to avoid return zero
 if (unlikely(n >= 0x8000000000000000ULL))
 return n;
 else if (n <= 1)
 return n;

and the comment explains the clzll/clzl choice, which is a nice example of a portability trap being closed with a comment rather than a bug report:

 cpp
 
 
// src/Common/BitHelpers.h:23
 /// `clzll`, not `clzl`: the argument is a 64-bit `size_t`, as the hardcoded 64 and the
 /// `0x8000000000000000ULL` above both assume, but `unsigned long` - what `clzl` takes - is
 /// only 64-bit wide on LP64 platforms. On an LLP64 one it is 32, and the argument would be
 /// truncated. `clzll` is at least 64 bits everywhere.

`reserve_exact`: exactly what was asked for

 cpp
 
 
// src/Common/PODArray.h:244
 template <typename ... TAllocatorParams>
 void reserve_exact(size_t n, TAllocatorParams &&... allocator_params) /// NOLINT
 {
 if (n > capacity())
 realloc(PODArrayDetails::minimum_memory_for_elements(n, ELEMENT_SIZE, pad_left, pad_right), std::forward<TAllocatorParams>(allocator_params)...);
 }

This is the one that surprises people. ColumnVector::reserve uses reserve_exact, not reserve:

 cpp
 
 
// src/Columns/ColumnVector.h:182
 void reserve(size_t n) override
 {
 data.reserve_exact(n);
 }

 size_t capacity() const override
 {
 return data.capacity();
 }

 void shrinkToFit() override
 {
 data.shrink_to_fit();
 }

Why would a column reserve exactly? Because a column’s final size is usually known in advance. When ClickHouse reads a MergeTree granule it knows how many rows the granule has; when it deserializes a Block from the wire it knows the row count. Rounding up to a power of two would waste up to 50% of the memory on every column in every block — and a block with 50 columns of 65 536 rows each would waste tens of megabytes.

The shrinkToFit override exists for the case where the size was not known in advance: after the block is fully built, the over-allocation is returned.

The same policy shows up in ColumnString:

 cpp
 
 
// src/Columns/ColumnString.cpp:636
void ColumnString::reserve(size_t n)
{
 offsets.reserve_exact(n);
}

and in ColumnFixedString, which reserves in characters:

 cpp
 
 
// src/Columns/ColumnFixedString.h:197
 void reserve(size_t size) override
 {
 chars.reserve_exact(n * size);
 }

`push_back`: double the allocation

The third policy is the incremental one:

 cpp
 
 
// src/Common/PODArray.h:442
 template <typename U, typename ... TAllocatorParams>
 void push_back(U && x, TAllocatorParams &&... allocator_params) /// NOLINT
 {
 if (unlikely(this->c_end + sizeof(T) > this->c_end_of_storage))
 this->reserveForNextSize(std::forward<TAllocatorParams>(allocator_params)...);
 new (reinterpret_cast<void*>(t_end())) T(std::forward<U>(x));
 this->c_end += sizeof(T);
 }

and the growth itself:

 cpp
 
 
// src/Common/PODArray.h:185
 template <typename ... TAllocatorParams>
 NO_INLINE void reserveForNextSize(TAllocatorParams &&... allocator_params)
 {
 if (empty())
 {
 // The allocated memory should be multiplication of ELEMENT_SIZE to hold the element, otherwise,
 // memory issue such as corruption could appear in edge case.
 realloc(std::max(integerRoundUp(initial_bytes, ELEMENT_SIZE),
 PODArrayDetails::minimum_memory_for_elements(1, ELEMENT_SIZE, pad_left, pad_right)),
 std::forward<TAllocatorParams>(allocator_params)...);
 }
 else
 realloc(allocated_bytes() * 2, std::forward<TAllocatorParams>(allocator_params)...);
 }

Three details worth noting:

The growth is allocated_bytes() * 2 — doubling the bytes, including the padding. So the element capacity grows by slightly more than 2× when the element is small relative to the 127 bytes of padding.
The first allocation is initial_bytes (4096 for column arrays), not one element. So a column that gets a handful of rows still allocates 4 KB — but only once, and initial_bytes is the minimum, so it is paid once per column, not per row.
reserveForNextSize is NO_INLINE. The comment above it explains why:

 cpp
 
 
// src/Common/PODArray.h:183
 /// NO_INLINE functions for slow paths of otherwise fast small inlinable functions.

The realloc path is cold; inlining it into push_back would bloat every caller.

The “shrink if less than half used” trick

One more policy lives in ColumnAggregateFunction, and it is a good illustration of the trade-off. When you filter a column, you know approximately how many rows will survive, so you can reserve exactly. But if the filter is much more selective than the hint, you are left with a huge buffer holding a few elements:

 cpp
 
 
// src/Columns/ColumnAggregateFunction.cpp:335
 auto res = createView();
 auto & res_data = res->data;

 if (result_size_hint)
 res_data.reserve_exact(result_size_hint > 0 ? result_size_hint : size);

 for (size_t i = 0; i < size; ++i)
 if (filter[i])
 res_data.push_back(data[i]);

 /// To save RAM in case of too strong filtering.
 if (res_data.size() * 2 < res_data.capacity())
 res_data = Container(res_data.cbegin(), res_data.cend());

The check size * 2 < capacity means “if less than half the buffer is used, copy it into a tight one”. The copy costs O(size) and is worth it when it frees more than half the buffer.

ColumnVector::filter does the same thing, but with a SIMD path that reserves based on the hint:

 cpp
 
 
// src/Columns/ColumnVector.cpp:1046
 auto res = this->create();
 Container & res_data = res->getData();

 if (result_size_hint)
 res_data.reserve_exact(result_size_hint > 0 ? result_size_hint : size);

`ColumnVector`: the typed array

With the container settled, ColumnVector is a thin wrapper:

 cpp
 
 
// src/Columns/ColumnVector.h:39
public:
 using ValueType = T;
 using Container = PaddedPODArray<ValueType>;

private:
 ColumnVector() = default;
 explicit ColumnVector(const size_t n) : data(n) {}
 ColumnVector(const size_t n, const ValueType x) : data(n, x) {}
 ColumnVector(const ColumnVector & src) : data(src.data.begin(), src.data.end()) {}
 ColumnVector(Container::const_iterator begin, Container::const_iterator end) : data(begin, end) { }

Note that the copy constructor of ColumnVector copies the data explicitly, element by element, rather than memcpy’ing the whole buffer. That looks wasteful, and it is — but ColumnVector is copy-on-write at the IColumn level, so the copy constructor is only reached when someone genuinely wants a copy, and the element-by-element copy lets the compiler vectorize it. PaddedPODArray is noncopyable, so this constructor is the only way to copy one, and it is not accidental.

The interesting part is what ColumnVector exposes to code that wants to avoid virtual dispatch:

 cpp
 
 
// src/Columns/ColumnVector.h:335
 /** More efficient methods of manipulation - to manipulate with data directly. */
 Container & getData()
 {
 return data;
 }

 const Container & getData() const
 {
 return data;
 }

 const T & getElement(size_t n) const
 {
 return data[n];
 }

 T & getElement(size_t n)
 {
 return data[n];
 }

protected:
 Container data;

getData() hands out the raw PaddedPODArray, so a caller that has already established that the column is a ColumnVector<UInt64> can read data.data()[i] directly with no virtual call, no bounds check, and no per-element function call.

This is the pattern that ClickHouse uses throughout: the IColumn interface is for structural operations (filter, permute, serialize), and the concrete class exposes its guts for the data operations. A typical aggregation function does one checkAndGetColumn<ColumnVector<UInt64>>() and then runs a monomorphic loop over getData().

`insertRangeFrom`: memcpy, not a loop

 cpp
 
 
// src/Columns/ColumnVector.cpp:746
template <typename T>
#if !defined(DEBUG_OR_SANITIZER_BUILD)
void ColumnVector<T>::insertRangeFrom(const IColumn & src, size_t start, size_t length)
#else
void ColumnVector<T>::doInsertRangeFrom(const IColumn & src, size_t start, size_t length)
#endif
{
 const ColumnVector & src_vec = assert_cast<const ColumnVector &>(src);

 if (start > src_vec.data.size() || length > src_vec.data.size() - start)
 throwInsertRangeFromOutOfBound("ColumnVector<T>", start, length, src_vec.data.size());

 size_t old_size = data.size();
 data.resize(old_size + length);
 memcpy(data.data() + old_size, &src_vec.data[start], length * sizeof(data[0]));
}

Two things here. First, data.resize(...) — and PODArray::resize does not initialize:

 cpp
 
 
// src/Common/PODArray.h:251
 template <typename ... TAllocatorParams>
 void resize(size_t n, TAllocatorParams &&... allocator_params)
 {
 reserve(n, std::forward<TAllocatorParams>(allocator_params)...);
 resize_assume_reserved(n);
 }

 cpp
 
 
// src/Common/PODArray.h:271
 void resize_assume_reserved(const size_t n) /// NOLINT
 {
 c_end = c_start + PODArrayDetails::byte_size(n, ELEMENT_SIZE);
 }

So resize just moves c_end. No zeroing, no element construction. The memcpy then fills the region. Compare with resize_fill:

 cpp
 
 
// src/Common/PODArray.h:419
 /// Same as resize, but zeroes new elements.
 void resize_fill(size_t n) /// NOLINT
 {
 size_t old_size = this->size();
 if (n > old_size)
 {
 this->reserve(n);
 memset(this->c_end, 0, PODArrayDetails::byte_size(n - old_size, sizeof(T)));
 }
 this->c_end = this->c_start + PODArrayDetails::byte_size(n, sizeof(T));
 }

ColumnVector uses resize_fill where the values matter:

 cpp
 
 
// src/Columns/ColumnVector.h:70
 void insertManyFrom(const IColumn & src, size_t position, size_t length) override
 {
 ValueType v = assert_cast<const Self &>(src).getData()[position];
 data.resize_fill(data.size() + length, v);
 }

Second, insertRangeFrom does not call push_back in a loop. It does one resize and one memcpy. That is only possible because the destination is a flat array of the same element type — the whole reason for the ColumnVector design.

`insertRawUninitialized`: writing into the column’s memory directly

There is an even more aggressive version of this, used by the Parquet reader and the hash-join builder. The column grows by n elements and hands back a pointer to the new region, so the caller can decompress straight into it:

 cpp
 
 
// src/Columns/ColumnVector.cpp:1528
template <typename T>
std::span<char> ColumnVector<T>::insertRawUninitialized(size_t count)
{
 size_t start = data.size();
 data.resize(start + count);
 return {reinterpret_cast<char *>(data.data() + start), count * sizeof(T)};
}

The interface says what it is for:

 cpp
 
 
// src/Columns/IColumn.h:928
 /// Appends n elements with unspecified values and returns a span pointing to their memory range.
 /// Can be used to decompress or deserialize data directly into the column.
 /// Supported only for simple column types like ColumnVector and ColumnFixedString.
 [[nodiscard]] virtual std::span<char> insertRawUninitialized(size_t count);

This is the “external buffer” trick in modern ClickHouse form: instead of decompressing into a temporary buffer and then copying into the column, the decoder writes into the column. One copy instead of two.

The Nullable case does it for both halves at once:

 cpp
 
 
// src/Interpreters/HashJoin/gatherJoinOutputColumns.cpp:407
 const std::span<char> null_span = dst.getNullMapColumn().insertRawUninitialized(count);
 const std::span<char> value_span = dst.getNestedColumn().insertRawUninitialized(count);

`ColumnString`: offsets plus a flat character buffer

ColumnString is the same idea, extended to variable-length data. Two arrays:

 cpp
 
 
// src/Columns/ColumnString.h:34
private:
 friend class COWHelper<IColumnHelper<ColumnString>, ColumnString>;

 /// Maps i'th position to offset to i+1'th element. Last offset maps to the end of all chars (is the size of all chars).
 Offsets offsets;

 /// Bytes of strings, placed contiguously. Note that strings are not zero-terminated and could contain zero bytes in the middle.
 Chars chars;

Note the comment on chars: strings are not zero-terminated. The length comes entirely from the offsets. And the offsets array holds size()+1 entries conceptually — the last one is the end of the character buffer — but because of the -1 trick, the stored array has only size() entries and the implicit offsets[-1] == 0 provides the first one.

The layout for three strings:

text

 offsets: [ 0, 5, 6, 11 ] (implicit offsets[-1] = 0)
 chars: [ 'H','e','l','l','o', ',', 'W','o','r','l','d' ]

 string 0 = chars[offsets[-1] .. offsets[0]) = chars[0..5) = "Hello"
 string 1 = chars[offsets[0] .. offsets[1]) = chars[5..6) = ","
 string 2 = chars[offsets[1] .. offsets[2]) = chars[6..11) = "World"

Reading a row is two array reads and a subtraction:

 cpp
 
 
// src/Columns/ColumnString.h:118
 std::string_view getDataAt(size_t n) const override
 {
 chassert(n < size());
 return std::string_view(reinterpret_cast<const char *>(&chars[offsetAt(n)]), sizeAt(n));
 }

No pointer to chase, no per-string allocation, no length prefix in the character buffer. The whole column is two contiguous allocations.

Bulk insertion

Copying a range of strings is one memcpy per string plus one pass over the offsets:

 cpp
 
 
// src/Columns/ColumnString.cpp:58
 const ColumnString & src_concrete = assert_cast<const ColumnString &>(src);
 const size_t src_offset = src_concrete.offsets[position - 1];
 const size_t src_buf_size
 = src_concrete.offsets[position] - src_concrete.offsets[position - 1]; /// -1th index is Ok, see PaddedPODArray.

 const size_t old_rows = offsets.size();
 const size_t new_rows = old_rows + length;
 /// Reserve offsets before changing chars to keep the column consistent if allocation fails.
 offsets.reserve(new_rows);

 const size_t old_size = chars.size();
 const size_t new_size = old_size + src_buf_size * length;
 chars.resize(new_size);

 const UInt8 * src_buf = &src_concrete.chars[src_offset];

 offsets.resize_assume_reserved(new_rows);

 for (size_t current_offset = old_size; current_offset < new_size; current_offset += src_buf_size)
 memcpySmallAllowReadWriteOverflow15(&chars[current_offset], src_buf, src_buf_size);

 for (size_t i = 0, current_offset = old_size + src_buf_size; i < length; ++i, current_offset += src_buf_size)
 offsets[old_rows + i] = current_offset;

Note the ordering, and the comment explaining it: offsets.reserve happens before chars is modified, so that if the allocation fails the column is still consistent. And chars.resize does not initialize — the memcpySmallAllowReadWriteOverflow15 calls fill it.

Note also that the offsets are written after the characters. That ordering means the offsets array is never in a state where it points past the end of the character buffer, even transiently.

`ColumnFixedString`: the inline variant

FixedString(N) is the degenerate case where every value has the same length, so no offsets array is needed at all — just a flat character buffer of N * size bytes:

 cpp
 
 
// src/Columns/ColumnFixedString.h:26
private:
 /// Bytes of strings, laid in succession. The strings are stored without a trailing zero byte.
 /** NOTE It is required that the offset and type of chars in the object be the same as that of `data in ColumnUInt8`.
 * Used in `packFixed` function (AggregationCommon.h)
 */
 Chars chars;
 /// The size of the rows.
 const size_t n;

The comment is a constraint on the representation: ColumnFixedString’s buffer must have exactly the same layout as a ColumnUInt8’s, so that generic code can reinterpret one as the other. That is what lets ColumnFixedSizeHelper do this:

 cpp
 
 
// src/Columns/ColumnFixedSizeHelper.h:31
 template <size_t ELEMENT_SIZE>
 const char * getRawDataBegin() const
 {
 return reinterpret_cast<const PODArrayBase<ELEMENT_SIZE, 4096, Allocator<false>, PADDING_FOR_SIMD - 1, PADDING_FOR_SIMD> *>(
 reinterpret_cast<const char *>(this) + sizeof(*this))
 ->raw_data();
 }

The class exists so that this downcast is technically legal:

 cpp
 
 
// src/Columns/ColumnFixedSizeHelper.h:11
/** Allows to access internal array of fixed-size column without cast to concrete type.
 * We will inherit ColumnVector and ColumnFixedString from this class instead of IColumn.
 * Assumes data layout of ColumnVector, ColumnFixedString and PODArray.
 *
 * Why it is needed?
 *
 * There are some algorithms that specialize on the size of data type but doesn't care about concrete type.
 * The same specialization may work for UInt64, Int64, Float64, FixedString(8), if it only does byte moving and hashing.
 * To avoid code bloat and compile time increase, we can use single template instantiation for these cases
 * and just static_cast pointer to some single column type (e. g. ColumnUInt64) assuming that all types have identical memory layout.
 *
 * But this static_cast (downcast to unrelated type) is illegal according to the C++ standard and UBSan warns about it.
 * To allow functional tests to work under UBSan we have to separate some base class that will present the memory layout in explicit way,
 * and we will do static_cast to this class.
 */

That is a lovely piece of engineering commentary: the constraint is documented, the workaround is a dedicated base class, and the reason is UBSan.

The n is fixed at construction and the buffer is sized in n-byte units:

 cpp
 
 
// src/Columns/ColumnFixedString.h:55
 size_t size() const override
 {
 return chars.size() / n;
 }

with the trailing-zero-byte question resolved by the serializer rather than the column: the in-memory buffer holds exactly n bytes per row with no terminator, and shorter strings are zero-padded on the way in:

 cpp
 
 
// src/Columns/ColumnFixedString.cpp:117
void ColumnFixedString::insertData(const char * pos, size_t length)
{
 if (length > n)
 throw Exception(ErrorCodes::TOO_LARGE_STRING_SIZE, "Too large string for FixedString column");

 size_t old_size = chars.size();
 chars.resize(old_size + n);
 memcpy(chars.data() + old_size, pos, length);
 memset(chars.data() + old_size + length, 0, n - length);
}

`ColumnNullable`: the null map as a separate column

Nullable(T) is a T column plus a ColumnUInt8 of the same length. The comment gives the reasoning:

 cpp
 
 
// src/Columns/ColumnNullable.h:19
/// Class that specifies nullable columns. A nullable column represents
/// a column, which may have any type, provided with the possibility of
/// storing NULL values. For this purpose, a ColumNullable object stores
/// an ordinary column along with a special column, namely a byte map,
/// whose type is ColumnUInt8. The latter column indicates whether the
/// value of a given row is a NULL or not. Such a design is preferred
/// over a bitmap because columns are usually stored on disk as compressed
/// files. In this regard, using a bitmap instead of a byte map would
/// greatly complicate the implementation with little to no benefits.

That is the whole design decision: a bitmap would be 8× smaller in RAM, but columns are stored compressed, and a byte map of mostly zeros compresses just as well as a bitmap while remaining a perfectly ordinary column that can be filtered, compressed and serialized by the generic machinery.

The two members are independent columns, each with its own lifecycle:

 cpp
 
 
// src/Columns/ColumnNullable.h:263
private:
 WrappedPtr nested_column;
 WrappedPtr null_map;

and reading a row is a lookup in each:

 cpp
 
 
// src/Columns/ColumnNullable.h:55
 size_t size() const final { return assert_cast<const ColumnUInt8 &>(*null_map).size(); }
 bool isNullAt(size_t n) const final { return assert_cast<const ColumnUInt8 &>(*null_map).getData()[n] != 0;}

Note that size() is defined by the null map, not the nested column. The null map is the authoritative row count.

There is a subtle point about what lives in the nested column under a NULL. The convention is that the value is undefined — it is present, but meaningless. The code makes that explicit in the one place where it matters:

 cpp
 
 
// src/Columns/ColumnNullable.cpp:995
 if (isNullAt(0))
 {
 /// Value in main column, when null map is 1 is implementation defined. So, take any value.
 new_values = nested_column->createWithOffsets(offsets, *createColumnConstWithDefaultValue(nested_column), total_rows, shift);

So “nullable” is implemented as “a value column plus a parallel byte column”, and the value column is not branch-free — reading a nullable value always requires a branch on the null map.

Trade-offs

What the flat-array design buys you.

Every column is one or two contiguous allocations. Scans are sequential memory reads that the prefetcher handles perfectly. There is no per-row indirection, so a SELECT sum(x) FROM t on a UInt64 column is a linear sweep over memory at memory bandwidth.

The offsets+chars layout for strings means a string column is also a linear sweep: the character buffer is contiguous, and the offsets are a separate dense UInt64 array. This is also why ClickHouse’s string compression works so well — the LZ4 compressor sees a long run of human-readable bytes.

What it costs you.

The PaddedPODArray pays 127 bytes of padding per array. For a column of UInt8 with 1000 rows that is 12% overhead. For a column with a million rows it is nothing. The design assumes large columns, which is exactly what the header comment says: “Designed for a small number of large arrays (rather than a lot of small ones).”

The -1 element trick means offsets and offsets[-1] are both live data, and the invariant that offsets[-1] == 0 is maintained by a single memset at allocation time. It is checked nowhere. If a future change ever wrote to c_start - ELEMENT_SIZE, every string in the column would silently shift.

The reserve_exact policy for columns is a bet that the row count is known in advance. When it is not, the array over-allocates by up to 2×, and the ColumnAggregateFunction shrink trick exists to clean that up after the fact. The right fix is to know the size; the fallback is a copy.

The design decision that makes it work. PODArray refuses to initialize its elements, and then makes up for it with padding that guarantees the absence of initialization bugs: 63 bytes on the right so that SIMD can overrun, 64 bytes on the left with a zeroed element at index -1 so that offset arithmetic needs no special case. The result is a container where the fastest code paths — memcpy ranges, SIMD filters, unaligned vector copies — are also the simplest code paths, because the padding has already paid for the slack they need.

ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: A Variant column is a struct of arrays — one sub-column per alternative, one tag byte per row, one offset per row. The tag is a local index; a separate global↔local mapping lets the type grow at runtime without rewriting a single row.

A Variant column is a tagged union: each row holds a value of one of several types, and the column itself knows the full list of alternatives. Dynamic is the same idea with an open-ended list — the set of types is discovered from the data.

The obvious implementation is a struct-of-arrays: one sub-column per alternative, each holding only the rows that belong to it, plus a tag per row saying which sub-column to look in. That’s what ClickHouse does.

What makes the implementation interesting isn’t the tag. It’s the set of awkward consequences that fall out of it:

The order of the alternatives must not matter (Variant(String, UInt64) and Variant(UInt64, String) are the same type), but the storage of a discriminator is a small integer, so the order does matter on disk.
The type must be extendable at runtime without rewriting the data, which means a column’s internal ordering may differ from the type’s ordering.
The number of alternatives must be bounded, but a Dynamic column can’t know its alternatives in advance, so it needs an escape hatch.
Reading a row must not require a virtual call and a branch on the tag.

Let’s look at how ClickHouse 26.9 solves all four.

The layout

The header comment of ColumnVariant is a complete specification, worked example included:

 cpp
 
 
// src/Columns/ColumnVariant.h:12
/**
 * Column for storing Variant(...) type values.
 * Variant type represents a union of other data types.
 * For example, type Variant(T1, T2, ..., TN) means that each row of this type
 * has a value of either type T1 or T2 or ... or TN or none of them (NULL value)
 *
 * ColumnVariant stores:
 * - The discriminators column, which determines which variant is stored in each row.
 * - The offsets column, which determines the offset in the corresponding variant column in each row.
 * - The list of variant columns with only real values (so the sizes of variant columns can be different).
 * Discriminator is an index of a variant in the variants list, it also has special value called NULL_DISCRIMINATOR
 * that indicates that the value in the row is NULL.
 *
 * We want to be able to extend Variant column for free without rewriting the data, but as we don't care about the
 * order of variants during Variant creation (we want Variant(T1, T2) to be the same as Variant(T2, T1)), we support
 * some global order of nested types inside Variant during type creation, so after extension the order of variant types
 * (and so their discriminators) can change. For example: Variant(T1, T3) -> Variant(T1, T2, T3).
 * To avoid full rewrite of discriminators column on Variant extension, we differentiate local order of variants
 * inside a column and global order of variants created during type creation. So, ColumnVariant stores only local
 * discriminators and additionally stores the mapping between global and local discriminators.
 * So, when we need to extend Variant column with new variant, we can just append it to a list of variant columns
 * with new local discriminator and update mapping from global to local orders.
 *
 * Note that two instances of ColumnVariant can have different local orders, so we should always use global
 * discriminators during inter-column interactions.
 *
 * Let's take an example with type Variant(UInt32, String, Array(UInt32)):
 * During type creation we will sort types by their names and get the global order: Array(UInt32), String, UInt32.
 * So, type Array(UInt32) will have global discriminator 0, String - 1 and UInt32 - 2.
 * Let's say we have a column with local order (String, UInt32, Array(UInt32)) and values:
 * 'Hello', 42, NULL, 'World', 43, [1, 2, 3], NULL, 44
 *
 * Let's see how these values will be stored in ColumnVariant:
 *
 * local_to_global_discriminators: {0 : 1, 1 : 2, 2 : 0}
 * global_to_local_discriminators: {0 : 2, 1 : 0, 2 : 1}
 * local_discriminators offsets String UInt32 Array(UInt32))
 * 0 0 'Hello' 42 [1, 2, 3]
 * 1 0 'World' 43
 * NULL_DISCRIMINATOR 0 44
 * 0 1
 * 1 1
 * 2 0
 * NULL_DISCRIMINATOR 0
 * 1 2
 *
 */

That diagram is worth staring at, because it shows the two central ideas:

text

 row: 0 1 2 3 4 5 6 7
 local: 0 1 NULL 0 1 2 NULL 1
 offsets: 0 0 0 1 1 0 0 2

 String: 'Hello' 'World' (2 values)
 UInt32: 42 43 44 (3 values)
 Array: [1,2,3] (1 value)

The discriminator is one byte per row. It’s a local index into the column’s own list of variant columns.
The offset is a row index within that particular variant column. Row 3 has local discriminator 0 (String) and offset 1, so its value is String[1] == 'World'. Row 5 has local discriminator 2 (Array) and offset 0, so its value is Array[0].

So per row you pay one byte of tag plus one Offset (UInt64) of position. The values themselves are stored densely, one after another, inside their own sub-column — a String sub-column holds only the strings, a UInt32 sub-column holds only the integers, and neither has holes.

The types involved:

 cpp
 
 
// src/Columns/ColumnVariant.h:62
 using Discriminator = UInt8;
 using Discriminators = PaddedPODArray<Discriminator>;
 using ColumnDiscriminators = ColumnVector<Discriminator>;
 using ColumnOffsets = ColumnVector<Offset>;

 static constexpr UInt8 NULL_DISCRIMINATOR = std::numeric_limits<Discriminator>::max(); /// 255
 static constexpr size_t MAX_NESTED_COLUMNS = std::numeric_limits<Discriminator>::max(); /// 255

NULL_DISCRIMINATOR is 255 — one past the largest legal alternative index. That means a column can hold at most 255 alternatives, and 255 is reserved for NULL. The value is not stored in any sub-column; it’s only a tag.

Global vs local order

The tag stored in the column is a local discriminator. The global discriminator is the position in the type’s canonical ordering. The canonical ordering is by type name — that’s how Variant(T1, T2) and Variant(T2, T1) become the same type:

 cpp
 
 
// src/DataTypes/DataTypeVariant.cpp:46
DataTypeVariant::DataTypeVariant(const DataTypes & variants_)
{
 /// Sort nested types by their full names and squash identical types.
 std::map<String, DataTypePtr> name_to_type;
 for (const auto & type : variants_)
 {
 checkAllowedInsideVariant(type);

 /// Don't use Nothing type as a variant.
 if (!isNothing(type))
 name_to_type[type->getName()] = type;
 }

 variants.reserve(name_to_type.size());
 for (const auto & [_, type] : name_to_type)
 variants.push_back(type);

 checkVariantsNotEmptyAndNotTooMany(variants);
}

A std::map keyed by the type name gives the sorted, de-duplicated order for free.

Now, why do we need two orders at all? Because a Dynamic column is created with one set of alternatives and then extended as new types arrive. When Variant(T1, T3) becomes Variant(T1, T2, T3), the canonical (global) order of T3 may change — T2 might sort before it. If the column stored global discriminators, every stored tag would have to be rewritten.

The fix is to keep the column’s local order stable and just maintain a mapping:

 cpp
 
 
// src/Columns/ColumnVariant.h:160
 Discriminator ALWAYS_INLINE globalDiscriminatorByLocal(Discriminator local_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return local_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : local_to_global_discriminators[local_discr];
 }

 Discriminator ALWAYS_INLINE localDiscriminatorByGlobal(Discriminator global_discr) const
 {
 /// NULL_DISCRIMINATOR is always the same in local and global orders.
 return global_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : global_to_local_discriminators[global_discr];
 }

and the extension operation is a pure metadata change:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1904
void ColumnVariant::extend(const VectorWithMemoryTracking<Discriminator> & old_to_new_global_discriminators, VectorWithMemoryTracking<std::pair<MutableColumnPtr, Discriminator>> && new_variants_and_discriminators)
{
 /// Update global discriminators for current variants.
 for (Discriminator & global_discr : local_to_global_discriminators)
 global_discr = old_to_new_global_discriminators[global_discr];

 /// Add new variants.
 variants.reserve(variants.size() + new_variants_and_discriminators.size());
 local_to_global_discriminators.reserve(local_to_global_discriminators.size() + new_variants_and_discriminators.size());
 for (auto & new_variant_and_discriminator : new_variants_and_discriminators)
 {
 variants.emplace_back(std::move(new_variant_and_discriminator.first));
 local_to_global_discriminators.push_back(new_variant_and_discriminator.second);
 }

 /// Update global -> local discriminators matching.
 global_to_local_discriminators.resize(local_to_global_discriminators.size());
 for (Discriminator local_discr = 0; local_discr != local_to_global_discriminators.size(); ++local_discr)
 global_to_local_discriminators[local_to_global_discriminators[local_discr]] = local_discr;
}

Nothing in that function touches the discriminator column or the offsets column. The existing rows’ local tags stay valid; only the interpretation of the tags changes. That’s the “extend Variant column for free without rewriting the data” promise, kept.

The invariant is that local discriminators are always dense: local discriminator k is the index of variants[k]. Global discriminators are sparse in the sense that a given column may not contain all of them.

Reading a row

 cpp
 
 
// src/Columns/ColumnVariant.cpp:415
Field ColumnVariant::operator[](size_t n) const
{
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return Null();
 return (*variants[discr])[offsetAt(n)];
}

Two loads and a branch. The branch on NULL_DISCRIMINATOR is unavoidable. What you want to avoid is a virtual call per row — and variants[discr] is an IColumn&, so operator[] on it is a virtual call.

That’s why the hot paths in ClickHouse don’t use operator[] but instead reach into the concrete sub-column via getDataAt, and why ColumnVariant also exposes getVariantByLocalDiscriminator so callers can hoist the type test out of the loop.

Note also that the offsets array is not a byte offset into a shared arena. It’s an element index into the sub-column. That’s what allows each sub-column to be an ordinary ClickHouse column with its own type-specific representation (a ColumnString with offsets+chars, a ColumnVector<UInt32>, a ColumnArray, …), and it’s why byteSizeAt has to consult the sub-column:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:1564
size_t ColumnVariant::byteSizeAt(size_t n) const
{
 size_t res = sizeof(Offset) + sizeof(Discriminator);
 Discriminator discr = localDiscriminatorAt(n);
 if (discr == NULL_DISCRIMINATOR)
 return res;

 return res + variants[discr]->byteSizeAt(offsetAt(n));
}

So a Variant row costs 9 bytes of overhead (1 discriminator + 8 offset) plus the payload. The offsets are the expensive part, and that’s deliberate: an offset into a typed sub-column is what lets the sub-column be stored in its native, compressible layout.

Building the offsets

The offsets column is derived from the discriminators. constructOffsetsFromDiscriminators computes it, and it has three special cases:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:69
void ColumnVariant::constructOffsetsFromDiscriminators()
{
 const ColumnDiscriminators * discriminators_concrete = typeid_cast<const ColumnDiscriminators *>(local_discriminators.get());
 Offsets & offsets_data = typeid_cast<ColumnOffsets *>(offsets.get())->getData();
 offsets_data.clear();
 offsets_data.reserve(discriminators_concrete->size());
 /// If we have only NULLs, offsets column will not contain any real offsets.
 if (hasOnlyNulls())
 {
 /// Offsets of NULL rows are never read for their value, but compression reads the whole buffer.
 offsets_data.resize_fill(discriminators_concrete->size(), 0);
 }
 /// If we have only one non empty variant and no NULLs,
 /// offsets column will contain just sequential offsets 0, 1, 2, ...
 else if (getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 for (size_t i = 0; i != discriminators_concrete->size(); ++i)
 offsets_data.push_back(i);
 }
 /// Otherwise we should iterate through discriminators and
 /// remember current offset for each variant column.
 else
 {
 VectorWithMemoryTracking<Offset> nested_offsets;
 nested_offsets.resize(variants.size());
 for (Discriminator discr : discriminators_concrete->getData())
 {
 if (discr == NULL_DISCRIMINATOR)
 {
 offsets_data.emplace_back();
 }
 else
 {
 offsets_data.push_back(nested_offsets[discr]);
 ++nested_offsets[discr];
 }
 }
 }
}

The general case is a running counter per variant — a “rank within the sub-column” computation.

The special cases matter for the serializer. If every row is NULL, the offsets buffer is all zeros (and the comment notes that compression reads the whole buffer, so it can’t be empty). If there’s exactly one non-empty variant, the offsets are simply 0, 1, 2, ... and the whole offsets column is unnecessary information.

Insertion keeps the two arrays in sync

The discriminator and the offsets column must always have the same length. Because an allocation failure in the middle of an insert could leave them inconsistent, the code is careful:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:737
void ColumnVariant::insertDefault()
{
 getLocalDiscriminators().push_back(NULL_DISCRIMINATOR);
 /// Keep local_discriminators and offsets in sync even if appending to offsets throws
 /// (e.g. on a memory limit), otherwise popBack would over-pop the shorter one.
 try
 {
 getOffsets().emplace_back();
 }
 catch (...)
 {
 getLocalDiscriminators().popBack();
 throw;
 }
}

That’s exception-safety hygiene you’ll see repeatedly in this code base: the two columns are a matched pair, and one of them can throw (memory allocation) while the other can’t.

The insert path

Inserting a value from another variant column is where the global/local mapping earns its keep:

 cpp
 
 
// src/Columns/ColumnVariant.cpp:501
void ColumnVariant::insertFromImpl(const DB::IColumn & src_, size_t n, const VectorWithMemoryTracking<ColumnVariant::Discriminator> * global_discriminators_mapping)
{
 const size_t num_variants = variants.size();
 const ColumnVariant & src = assert_cast<const ColumnVariant &>(src_);

 if (!global_discriminators_mapping && src.variants.size() != num_variants)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot insert value of Variant type with different number of types");

 Discriminator src_global_discr = src.globalDiscriminatorAt(n);
 Discriminator global_discr = src_global_discr;
 if (global_discriminators_mapping && src_global_discr != NULL_DISCRIMINATOR)
 global_discr = (*global_discriminators_mapping)[src_global_discr];
 Discriminator local_discr = localDiscriminatorByGlobal(global_discr);
 getLocalDiscriminators().push_back(local_discr);
 if (local_discr == NULL_DISCRIMINATOR)
 {
 getOffsets().emplace_back();
 }
 else
 {
 getOffsets().push_back(variants[local_discr]->size());
 variants[local_discr]->insertFrom(src.getVariantByGlobalDiscriminator(src_global_discr), src.offsetAt(n));
 }
}

Read it as a three-step translation: source local tag → source global tag → destination local tag. The source’s local tag means nothing to the destination; the global tag is the portable currency; the destination’s local tag is what gets stored.

Serialization: compact discriminators

On disk, the discriminator stream has two modes:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.h:54
 struct DiscriminatorsSerializationMode
 {
 enum Value
 {
 BASIC = 0, /// Store the whole discriminators column.
 COMPACT = 1, /// Don't write discriminators in granule if all of them are the same.
 };

In BASIC mode the whole discriminator column is written. In COMPACT mode a granule whose discriminators are all the same writes one discriminator plus a marker byte instead of 65 536 of them. The writer has to detect those three situations (uniform non-empty, all NULL, mixed) and there’s a fast path for each:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:343
 /// Write number of rows in this granule in compact mode.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 writeVarUInt(UInt64(limit), *discriminators_stream);

 /// If column has only one none empty discriminators and no NULLs we don't need to
 /// calculate limits for variants and use provided offset/limit.
 if (auto non_empty_local_discr = col.getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
 {
 auto non_empty_global_discr = col.globalDiscriminatorByLocal(*non_empty_local_discr);

 /// In compact mode write the format of the granule and single non-empty discriminator.
 if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
 {
 writeBinaryLittleEndian(UInt8(CompactDiscriminatorsGranuleFormat::COMPACT), *discriminators_stream);
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }
 /// For basic mode just serialize this discriminator limit times.
 else
 {
 for (size_t i = 0; i < limit; ++i)
 writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
 }

The “one non-empty variant” fast path also lets the value streams be serialized with the same offset/limit as the whole column, because in that case the sub-column has exactly the same number of rows as the variant column:

 cpp
 
 
// src/DataTypes/Serializations/SerializationVariant.cpp:370
 /// We can use the same offset/limit as for whole Variant column
 if (i == non_empty_global_discr)
 {
 const auto & variant_column = col.getVariantByGlobalDiscriminator(i);
 if (variant_column.size() < offset + limit)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Variant {} has less rows ({}) than expected rows to serialize ({})", variant_names[i], variant_column.size(), offset + limit);

 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(variant_column, offset, limit, settings, variant_state->variant_states[i]);
 }
 else
 {
 variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(col.getVariantByGlobalDiscriminator(i), col.getVariantByGlobalDiscriminator(i).size(), 0, settings, variant_state->variant_states[i]);
 }

In the general case each sub-column is serialized with its own length, which is the whole point of the dense layout: the String sub-column is written as a String column with only the strings in it, and so it compresses as well as a plain string column would.

Dynamic: the open-ended case

Dynamic is a Variant whose alternative list is discovered at runtime. It’s implemented as a ColumnVariant plus a VariantInfo record:

 cpp
 
 
// src/Columns/ColumnDynamic.h:15
/**
 * Column for storing Dynamic type values.
 * Dynamic column allows to insert and store values of any data types inside.
 * Inside it stores:
 * - Variant column with all inserted values of different types.
 * - Information about currently stored variants.
 *
 * When new values are inserted into Dynamic column, the internal Variant
 * type and column are extended if the inserted value has new type.
 * When the limit on number of dynamic types is exceeded, all values
 * with new types are inserted into special shared variant with type String
 * that contains values and their types in binary format.
 */
class ColumnDynamic final : public COWHelper<IColumnHelper<ColumnDynamic>, ColumnDynamic>
{
public:
 /// Maximum limit on dynamic types. We use ColumnVariant to store all the types,
 /// so the limit cannot be greater then ColumnVariant::MAX_NESTED_COLUMNS.
 /// We also always have reserved variant for shared variant.
 static constexpr size_t MAX_DYNAMIC_TYPES_LIMIT = ColumnVariant::MAX_NESTED_COLUMNS - 1;
 static constexpr const char * SHARED_VARIANT_TYPE_NAME = "SharedVariant";

The escape hatch is a shared variant: one of the alternatives is always a String column, and it holds the type name and the binary-serialized value of anything that didn’t fit into a real alternative.

It has to be a String with a custom name, because a plain String alternative might legitimately be in the same column:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:71
/// Shared variant will contain String values but we cannot use usual String type
/// because we can have regular variant with type String.
/// To solve it, we use String type with custom name for shared variant.
const DataTypePtr & ColumnDynamic::getSharedVariantDataType()
{
 static thread_local DataTypePtr type = DataTypeFactory::instance().getCustom("String", std::make_unique<DataTypeCustomDesc>(std::make_unique<DataTypeCustomFixedName>(getSharedVariantTypeName())));
 return type;
}

and a fresh Dynamic column starts life with only the shared variant:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:80
ColumnDynamic::ColumnDynamic(size_t max_dynamic_types_) : max_dynamic_types(max_dynamic_types_), global_max_dynamic_types(max_dynamic_types)
{
 /// Create Variant with shared variant.
 setVariantType(std::make_shared<DataTypeVariant>(DataTypes{getSharedVariantDataType()}));
}

The insert path

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:257
void ColumnDynamic::insert(const Field & x)
{
 if (x.isNull())
 {
 insertDefault();
 return;
 }

 auto & variant_col = getVariantColumn();
 auto shared_variant_discr = getSharedVariantDiscriminator();
 /// Check if we can insert field into existing variants and avoid Variant extension.
 for (ColumnVariant::Discriminator i = 0; i != variant_col.getNumVariants(); ++i)
 {
 if (i != shared_variant_discr && variant_col.getVariantByGlobalDiscriminator(i).tryInsert(x))
 {
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(i));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(i).size() - 1);
 return;
 }
 }

 /// If we cannot insert field into current variant column, extend it with new variant for this field from its type.
 /// Use LeastSupertypeOnError::Dynamic so that arrays with incompatible element types (e.g. ["text", {"k":1}])
 /// are typed as Array(Dynamic) rather than throwing NO_COMMON_TYPE. Dynamic can hold any element value.
 auto field_data_type = applyVisitor(FieldToDataType<LeastSupertypeOnError::Dynamic>(), x);
 auto field_data_type_name = field_data_type->getName();
 if (addNewVariant(field_data_type, field_data_type_name))
 {
 /// Insert this field into newly added variant.
 auto discr = variant_info.variant_name_to_discriminator[field_data_type_name];
 variant_col.getVariantByGlobalDiscriminator(discr).insert(x);
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(discr));
 variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(discr).size() - 1);
 }
 else
 {
 /// We reached maximum number of variants and couldn't add new variant.
 /// In this case we add the value of this new variant into special shared variant.
 /// We store values in shared variant in binary form with binary encoded type.
 auto & shared_variant = getSharedVariant();
 auto & chars = shared_variant.getChars();
 {
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());
 }
 shared_variant.getOffsets().push_back(chars.size());
 variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(shared_variant_discr));
 variant_col.getOffsets().push_back(shared_variant.size() - 1);
 }
}

Three outcomes, in order:

The value fits an existing alternative — use it, no extension.
The value has a new type and there’s room — extend the variant list and add a real alternative.
The value has a new type and the list is full — encode the type and the value into the shared variant’s character buffer.

Case 3 is the interesting one. The shared variant is a ColumnString, so writing into it means writing into its chars buffer directly. The type is encoded first, then the value:

 cpp
 
 
 WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
 encodeDataType(field_data_type, value_buf);
 getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());

and reading it back is the mirror image:

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:323
void ColumnDynamic::get(size_t n, Field & res) const
{
 /// Object, Dynamic and Variant values nest into each other to a depth that comes from the data, not from the declared type.
 checkStackSize();

 const auto & variant_col = getVariantColumn();
 /// Check if value is not in shared variant.
 if (variant_col.globalDiscriminatorAt(n) != getSharedVariantDiscriminator())
 {
 variant_col.get(n, res);
 if (!res.isNull())
 res = convertFieldToType(res, *getTypeAt(n));
 return;
 }

 /// We should deserialize value from shared variant.
 const auto & shared_variant = getSharedVariant();
 auto value_data = shared_variant.getDataAt(variant_col.offsetAt(n));
 ReadBufferFromMemory buf(value_data);
 auto type = decodeDataType(buf);
 type->getDefaultSerialization()->deserializeBinary(res, buf, getFormatSettings());
}

Note checkStackSize(). Because Dynamic can nest inside Dynamic inside Dynamic, the depth is a function of the data, and a malicious input could otherwise blow the stack. That’s a real concern for a type whose shape is discovered at runtime, and it shows up in the code as an explicit guard.

Adding a variant

 cpp
 
 
// src/Columns/ColumnDynamic.cpp:145
bool ColumnDynamic::addNewVariant(const DataTypePtr & new_variant, const String & new_variant_name)
{
 /// Check if we already have such variant.
 if (variant_info.variant_name_to_discriminator.contains(new_variant_name))
 return true;

 /// Check if we reached maximum number of variants.
 if (!canAddNewVariant())
 {
 /// Dynamic column should always have shared variant.
 if (!variant_info.variant_name_to_discriminator.contains(getSharedVariantTypeName()))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Maximum number of variants reached, but no shared variant exists");

 return false;
 }

 const DataTypes & current_variants = assert_cast<const DataTypeVariant &>(*variant_info.variant_type).getVariants();
 DataTypes all_variants = current_variants;
 all_variants.push_back(new_variant);
 auto new_variant_type = std::make_shared<DataTypeVariant>(all_variants);
 updateVariantInfoAndExpandVariantColumn(new_variant_type);
 return true;
}

The new type is appended to the type, and then the column is expanded to match. updateVariantInfoAndExpandVariantColumn is where ColumnVariant::extend gets called with the old-to-new global discriminator mapping.

Seeing it in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `payload` Dynamic
)
ENGINE = MergeTree
ORDER BY ts;

INSERT INTO events VALUES
 ('2026-01-01 00:00:00', 42),
 ('2026-01-01 00:00:01', 'hello'),
 ('2026-01-01 00:00:02', [1, 2, 3]),
 ('2026-01-01 00:00:03', NULL);

-- The type of each value is remembered.
SELECT payload, dynamicType(payload) FROM events;

The dynamicType function is the user-visible face of the discriminator: it reads the tag and maps it to the type name.

Trade-offs

What the design buys you.

Each alternative is stored in its native layout, so a String alternative compresses like a string column, an UInt64 alternative compresses like a UInt64 column, and the whole thing benefits from per-column codecs. A Dynamic column that only ever saw strings is physically almost identical to a LowCardinality(String) or a plain String column — one byte of tag per row plus the payloads.

The global/local split means adding a type to a Dynamic column is O(1) in the number of rows. Nothing is rewritten; only two small mapping arrays change length.

What it costs you.

Nine bytes per row of pure overhead (one discriminator, one UInt64 offset), versus zero for a plain column. For a Variant with two alternatives and short values, that overhead can exceed the payload.

The offsets are the expensive part, and they’re the price of the dense sub-column layout. A “one big union buffer with a tag” design would pay one byte per row but would force every alternative into a common representation, losing per-type compression and per-type codecs.

The escape hatch is a real cost. Once a Dynamic column overflows into the shared variant, those values become opaque byte strings with an embedded type name. Reading one back requires decoding the type and then running the right deserializer — a dynamic dispatch on data, with a checkStackSize() guard because the nesting depth is attacker-controlled. And the shared variant is a String column, so it compresses like text even when it holds binary payloads.

The complexity is concentrated in one place. The mapping from global to local discriminator is the only genuinely subtle piece, and it exists purely so that Variant(T1, T2) and Variant(T2, T1) can be the same type while keeping the on-disk tags stable across extensions. Everything else — the offsets, the compact serialization, the NULL discriminator — is a fairly direct consequence of “one sub-column per alternative, one tag per row”.

Wrapping up

Worth internalizing: the local/global split is the trick that makes Dynamic viable. Store global tags and every extension rewrites the column. Store local tags and extension is a metadata edit.

What to go do about it: if you use Dynamic, watch how many distinct types actually show up in your data. Once a column overflows into the shared variant, you lose per-type compression and pay a runtime type dispatch on every read. If you know the shape of your data, declare a Variant with the alternatives you need — or better, a plain column — and you keep the per-type codecs and pay zero bytes of tag.

And if you’re benchmarking, remember the overhead is 9 bytes per row. For short values, that’s not a rounding error — it’s the whole payload.

ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: LowCardinality(T) stores a dictionary of distinct values once and keeps a 1–8 byte index per row. The dictionary is shared across every block, granule and part, so twelve distinct values cost one byte per row instead of ten. It pays off below ~10 000 distinct values and hurts above ~100 000.

Every column-oriented database eventually meets the same data: a column of country codes, of status strings, of user agents, of enum-like labels. A hundred thousand rows, twelve distinct values, repeated over and over.

Storing the value in every row is absurd. Storing the position of the value in every row is not.

ClickHouse’s answer is a column type called LowCardinality(T), and the implementation is one of the more interesting pieces of the whole storage engine, because it has to solve three separate problems at once:

how to represent the dictionary and the index in memory,
how to make the same dictionary serve many different blocks, granules and parts, so the dictionary is stored once and not once per block,
how to serialize that arrangement on disk so a reader can rebuild it — including the case where the dictionary had to change halfway through.

Let’s take all three. Source quoted from ClickHouse 26.9 (04eac2ea3).

The shape of the thing

The header comment of ColumnLowCardinality states the design in four lines:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:17
/**
 * How data is stored (in a nutshell):
 * we have a dictionary @e reverse_index in ColumnUnique that holds pairs (DataType, UIntXX) and a column
 * with UIntXX holding actual data indices.
 * To obtain the value's index, call #getOrFindIndex.
 * To operate on the data (so called indices column), call #getIndexes.
 *
 * @note The indices column always contains the default value (empty std::string_view) with the first index.
 */

The class holds exactly two members:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:398
 Dictionary dictionary;
 ColumnIndex idx;

Dictionary is a thin wrapper around a ColumnUnique plus a shared flag:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:366
private:
 class Dictionary
 {
 public:
 Dictionary(const Dictionary & other) = default;
 explicit Dictionary(MutableColumnPtr && column_unique, bool is_shared);
 explicit Dictionary(ColumnPtr column_unique, bool is_shared);

 const WrappedPtr & getColumnUniquePtr() const { return column_unique; }
 WrappedPtr & getColumnUniquePtr() { return column_unique; }

 const IColumnUnique & getColumnUnique() const { return static_cast<const IColumnUnique &>(*column_unique); }
 IColumnUnique & getColumnUnique() { return static_cast<IColumnUnique &>(*column_unique); }

 /// Dictionary may be shared for several mutable columns.
 /// Immutable columns may have the same column unique, which isn't necessarily shared dictionary.
 void setShared(const ColumnPtr & column_unique_);
 bool isShared() const { return shared; }

 /// Create new dictionary with only keys that are mentioned in indexes.
 void compact(MutableColumnPtr & indexes);
 /// Create nullable dictionary with only keys that are mentioned in indexes.
 void compactToNullable(MutableColumnPtr & indexes);

 static MutableColumnPtr compact(const IColumnUnique & column_unique, MutableColumnPtr & indexes);
 static MutableColumnPtr compactToNullable(const IColumnUnique & column_unique, MutableColumnPtr & indexes);

 private:
 WrappedPtr column_unique;
 bool shared = false;
 };

So a LowCardinality column is:

text

 +---------------------------+
 row 0 -----> | dictionary[ idx[0] ] | idx is a ColumnVector<UInt8|UInt16|UInt32|UInt64>
 row 1 -----> | dictionary[ idx[1] ] | dictionary is a ColumnUnique<ColumnString> (etc.)
 row 2 -----> | dictionary[ idx[2] ] |
 ... +---------------------------+

The values are in the dictionary; the rows are indices. Getting row n is two loads: getIndexes().getUInt(n), then a lookup in the dictionary.

That second load is a random access into a (hopefully tiny) array — and the whole point is that the dictionary array is tiny enough to stay in L1/L2 while the index array streams past it.

The index column grows its own type

The index column is not fixed-width. It starts as UInt8 and widens to UInt16, UInt32, UInt64 as the dictionary outgrows the range. The widening logic lives in ColumnIndex:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:135
void ColumnIndex::expandType()
{
 auto expand = [&](auto type)
 {
 using CurIndexType = decltype(type);
 constexpr auto next_size = NumberTraits::nextSize(sizeof(CurIndexType));
 if (next_size == sizeof(CurIndexType))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Can't expand indexes type for ColumnLowCardinality from type: {}",
 demangle(typeid(CurIndexType).name()));

 using NewIndexType = typename NumberTraits::Construct<false, false, next_size>::Type;
 convertIndexes<NewIndexType>();
 };

 callForType(std::move(expand), size_of_type);
}

and the bound that decides when to widen:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:152
size_t ColumnIndex::getMaxIndexForCurrentType() const
{
 chassert(size_of_type == 1 || size_of_type == 2 || size_of_type == 4 || size_of_type == 8);
 /// Shifting by 64 is undefined, and this is called per inserted value, so no `callForType` here.
 if (size_of_type == sizeof(UInt64))
 return std::numeric_limits<UInt64>::max();
 return (1ULL << (8 * size_of_type)) - 1;
}

The per-value insert path is where the widening happens, and note the comment about avoiding a typeid_cast on the hot path:

 cpp
 
 
// src/Columns/ColumnIndex.cpp:198
void ColumnIndex::insertIndex(size_t index)
{
 while (index > getMaxIndexForCurrentType())
 expandType();

 auto insert = [&]<typename CurIndexType>(CurIndexType /*type_value*/)
 {
 /// `size_of_type` tracks the type of `indexes` and `expandType` keeps the two in sync, so
 /// this is `static_cast` in release. `getIndexesData` would run a `typeid_cast` instead, and
 /// this is the per-value path.
 assert_cast<ColumnVector<CurIndexType> *>(indexes.get())->getData().push_back(static_cast<CurIndexType>(index));
 };

 callForType(std::move(insert), size_of_type);

 /// Same invariant, so re-deriving it from `indexes` is a debug-only check here.
 chassert(size_of_type == getSizeOfIndexType(*indexes, size_of_type));
}

callForType is a hand-rolled dispatch over the four possible widths — no virtual call, no switch on a runtime type id, just four instantiations of a lambda. This is the pattern used throughout ClickHouse’s hot column code: the virtual IColumn interface is for the cold, structural paths; the per-row paths are devirtualized by dispatching on the concrete type once and then running a monomorphic loop.

Reading an index back has the same shape, and is marked ALWAYS_INLINE because it is called from the tightest loops in the engine:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:326
 ALWAYS_INLINE size_t getIndexAt(size_t row) const
 {
 const IColumn * indexes = &getIndexes();

 switch (idx.getSizeOfIndexType())
 {
 case sizeof(UInt8): return assert_cast<const ColumnUInt8 *>(indexes)->getElement(row);
 case sizeof(UInt16): return assert_cast<const ColumnUInt16 *>(indexes)->getElement(row);
 case sizeof(UInt32): return assert_cast<const ColumnUInt32 *>(indexes)->getElement(row);
 case sizeof(UInt64): return assert_cast<const ColumnUInt64 *>(indexes)->getElement(row);
 default: throwUnexpectedLowCardinalityIndexType(idx.getSizeOfIndexType());
 }
 }

The dictionary: ColumnUnique and its reverse index

ColumnUnique is the dictionary. It is not a column type you can select; the comment says so:

 cpp
 
 
// src/Columns/ColumnUnique.h:36
/** Stores another column with unique values
 * and also an index that allows to find position by value.
 *
 * This column is not used on it's own but only as implementation detail of ColumnLowCardinality.
 */
template <typename ColumnType>
class ColumnUnique final : public COWHelper<IColumnUnique, ColumnUnique<ColumnType>>

Its members are the values plus the lookup structure:

 cpp
 
 
// src/Columns/ColumnUnique.h:203
private:
 IColumn::WrappedPtr column_holder;
 bool is_nullable;
 size_t size_of_value_if_fixed = 0;

 ReverseIndex<UInt64, ColumnType> reverse_index;

 /// For DataTypeNullable, stores null map.
 IColumn::WrappedPtr nested_null_mask;
 IColumn::WrappedPtr nested_column_nullable;

Two reserved slots

The first row of the dictionary is special: it is the default value of the type. If the dictionary is nullable, row 0 is the NULL and row 1 is the default value of the inner type:

 cpp
 
 
// src/Columns/ColumnUnique.h:74
 size_t getDefaultValueIndex() const override { return 0; }
 size_t getNullValueIndex() const override;
 size_t getNestedTypeDefaultValueIndex() const override { return is_nullable ? 1 : 0; }
 bool canContainNulls() const override { return is_nullable; }

and

 cpp
 
 
// src/Columns/ColumnUnique.h:370
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::getNullValueIndex() const
{
 if (!is_nullable)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ColumnUnique can't contain null values.");

 return 0;
}

The count of reserved slots is therefore one or two:

 cpp
 
 
// src/Columns/ColumnUnique.h:238
 static size_t numSpecialValues(bool is_nullable) { return is_nullable ? 2 : 1; }
 size_t numSpecialValues() const { return numSpecialValues(is_nullable); }

Those reserved slots are not in the reverse index, which is why the insert path has to special-case them by hand:

 cpp
 
 
// src/Columns/ColumnUnique.h:513
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsertData(const char * pos, size_t length)
{
 /// The reserved prefix slots are not in the reverse index, so the default value is matched here.
 /// Comparing the first byte first keeps the `memcmp` call out of this path for the values that
 /// are not the default, which is all of them in a typical dictionary.
 const size_t default_index = getNestedTypeDefaultValueIndex();
 const std::string_view default_value = getRawColumnPtr()->getDataAt(default_index);
 if (default_value.size() == length && (length == 0 || (default_value[0] == pos[0] && default_value == std::string_view(pos, length))))
 return default_index;

 auto insertion_point = reverse_index.insert({pos, length});

 updateNullMask();

 return insertion_point;
}

The comment is a nice piece of micro-optimisation reasoning: comparing the first byte before the whole value keeps the expensive memcmp off the path for the (overwhelmingly common) case where the value is not the default.

The reverse index stores row numbers, not values

ReverseIndex is the interesting part. A naive dictionary would be a HashMap<value, index> — the values stored twice, once in the column and once as hash-table keys. ReverseIndex inverts that: the hash table stores the row number in the dictionary column, and comparisons dereference back into the column:

 cpp
 
 
// src/Columns/ReverseIndex.h:94
 /// Special case when we want to compare with something not in index_column.
 /// When we compare something inside column default keyEquals checks only that row numbers are equal.
 bool keyEquals(std::string_view object, size_t hash_ [[maybe_unused]], const State & state) const
 {
 auto index = key;
 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return hash_ == (*state.saved_hash_column)[index] && object == state.index_column->getDataAt(index);
 else
 return object == state.index_column->getDataAt(index);
 }

The hash table therefore holds 8-byte integers, and the dictionary column holds the only copy of the data. For a LowCardinality(String) with 1000 distinct values, that is the difference between a hash table with 1000 string keys (plus their hashes, plus their lengths) and a hash table with 1000 UInt64s.

For string columns the hash is expensive, so it is precomputed once and kept in a parallel ColumnUInt64:

 cpp
 
 
// src/Columns/ReverseIndex.h:43
template <typename ColumnType>
struct ReverseIndexHashTableState<ColumnType, /* with_saved_hash = */ true, /* has_base_index */ false>
{
 constexpr static bool with_saved_hash = true;
 constexpr static bool has_base_index = false;

 ColumnType * index_column;
 typename ColumnVector<UInt64>::Container * saved_hash_column;
};

and the hash function for a stored row reads the precomputed hash rather than re-hashing the string:

 cpp
 
 
// src/Columns/ReverseIndex.h:108
 size_t getHash(const Hash & hash) const
 {
 auto index = key;

 /// Hack. HashTable is Hash itself.
 const auto & state = static_cast<const State &>(static_cast<const HashTable &>(hash));

 if constexpr (has_base_index)
 index -= state.base_index;

 if constexpr (string_hash)
 return (*state.saved_hash_column)[index];
 else
 {
 using ValueType = typename ColumnType::ValueType;
 ValueType value = unalignedLoad<ValueType>(state.index_column->getDataAt(index).data());
 return DefaultHash<ValueType>()(value);
 }
 }

The index is built lazily, and the hash table is filled in one pass

The reverse index is not built at construction time; it is built on the first insert, and the build is a single pass over the dictionary:

 cpp
 
 
// src/Columns/ReverseIndex.h:435
template <typename IndexType, typename ColumnType>
void ReverseIndex<IndexType, ColumnType>::buildIndex()
{
 if (index)
 return;

 if (!column)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "ReverseIndex can't build index because index column wasn't set.");

 auto size = column->size();
 index = std::make_unique<IndexMapType>(size);

 if constexpr (use_saved_hash)
 saved_hash = calcHashes();

 auto & state = index->getState();
 state.index_column = column;
 state.base_index = base_index;
 if constexpr (use_saved_hash)
 state.saved_hash_column = &saved_hash->getData();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 for (auto row : collections::range(num_prefix_rows_to_skip, size))
 {
 UInt64 hash = 0;
 if constexpr (use_saved_hash)
 hash = saved_hash->getElement(row);
 else
 hash = getHash(column->getDataAt(row));

 index->reverseIndexEmplace(row + base_index, iterator, inserted, hash, column->getDataAt(row));

 if (!inserted)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Duplicating keys found in ReverseIndex.");
 }
}

Note num_prefix_rows_to_skip: the reserved NULL/default slots are skipped, because they are matched by hand in uniqueInsertData.

The insert path is the mirror image, and it is written to be branch-light for the two different column kinds:

 cpp
 
 
// src/Columns/ReverseIndex.h:490
template <typename IndexType, typename ColumnType>
UInt64 ReverseIndex<IndexType, ColumnType>::insert(std::string_view data)
{
 if (!index)
 buildIndex();

 using IteratorType = typename IndexMapType::iterator;
 IteratorType iterator;
 bool inserted = false;

 auto hash = getHash(data);
 UInt64 num_rows = size();

 if constexpr (use_saved_hash)
 {
 auto & column_data = saved_hash->getData();
 if (column_data.size() <= num_rows)
 column_data.resize(num_rows + 1);
 column_data[num_rows] = hash;
 }
 else
 column->insertData(data.data(), data.size());

 index->reverseIndexEmplace(num_rows + base_index, iterator, inserted, hash, data);

 if constexpr (use_saved_hash)
 {
 if (inserted)
 column->insertData(data.data(), data.size());
 }
 else
 {
 if (!inserted)
 column->popBack(1);
 }

 return iterator->getValue();
}

For strings the hash is appended optimistically and the value is appended only if the key was new. For fixed-size values the value is appended optimistically and popped back if the key turned out to be a duplicate. Each kind pushes and pops the cheaper thing.

Float NaNs are canonicalized

A LowCardinality(Float64) dictionary must not contain two NaNs, because NaN == NaN is false but all NaNs must compare equal in the dictionary semantics. So NaN is normalized at insert time:

 cpp
 
 
// src/Columns/ColumnUnique.h:389
template <typename ColumnType>
size_t ColumnUnique<ColumnType>::uniqueInsert(const Field & x)
{
 if (x.isNull())
 return getNullValueIndex();

 // NaN can contain different sign or mantissa bits, but we need to consider all NaNs equal.
 if constexpr (is_float_vector_v<ColumnType>)
 {
 if (isNaN(x.safeGet<typename ColumnType::ValueType>()))
 {
 auto nan = NaNOrZero<typename ColumnType::ValueType>();
 return uniqueInsertData(reinterpret_cast<char *>(&nan), sizeof(nan));
 }
 }

Here is where the design pays off. A LowCardinality column can point at a dictionary owned by another column. The shared flag on Dictionary records this, and byteSize() deliberately does not count a shared dictionary:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:200
 /// Don't count the dictionary size as it can be shared between different blocks.
 size_t byteSize() const override { return idx.getIndexes()->byteSize() + (isSharedDictionary() ? 0 : getDictionary().byteSize()); }

 size_t byteSizeAt(size_t n) const override { return getDictionary().byteSizeAt(getIndexes().getUInt(n)); }
 size_t allocatedBytes() const override { return idx.getIndexes()->allocatedBytes() + getDictionary().allocatedBytes(); }

Note the asymmetry: byteSize() (the logical size, used for statistics and for system.parts_columns) excludes the shared dictionary, but allocatedBytes() (the physical size, used for memory accounting) includes it. That distinction is what makes the accounting correct when a part has a hundred blocks all pointing at one dictionary.

Sharing also propagates through every operation that produces a new column from an old one — cut, filter, permute, index, replicate all hand the same dictionary pointer to the result:

 cpp
 
 
// src/Columns/ColumnLowCardinality.h:130
 ColumnPtr filter(const Filter & filt, ssize_t result_size_hint) const override
 {
 return ColumnLowCardinality::create(
 dictionary.getColumnUniquePtr(), getIndexes().filter(filt, result_size_hint), isSharedDictionary());
 }

But a shared dictionary is immutable

The moment you want to mutate a column whose dictionary is shared with somebody else, the dictionary has to be un-shared. This happens in compactIfSharedDictionary(), which is called at the top of every mutating insert:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:146
void ColumnLowCardinality::insert(const Field & x)
{
 compactIfSharedDictionary();
 idx.insertIndex(getDictionary().uniqueInsert(x));
}

bool ColumnLowCardinality::tryInsert(const Field & x)
{
 compactIfSharedDictionary();

 size_t index = 0;
 if (!dictionary.getColumnUnique().tryUniqueInsert(x, index))
 return false;

 idx.insertIndex(index);
 return true;
}

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:894
void ColumnLowCardinality::compactIfSharedDictionary()
{
 if (dictionary.isShared())
 compactInplace();
}

Compaction rebuilds a private dictionary containing only the keys that are actually referenced by this column’s indexes, and remaps the indexes accordingly:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:1002
MutableColumnPtr ColumnLowCardinality::Dictionary::compact(const IColumnUnique & unique, MutableColumnPtr & indexes)
{
 auto new_column_unique = unique.cloneEmpty();
 auto & new_unique = static_cast<IColumnUnique &>(*new_column_unique);

 auto unique_indexes = mapUniqueIndex(*indexes);
 auto sub_keys = unique.getNestedColumn()->index(*unique_indexes, 0);
 auto new_indexes = new_unique.uniqueInsertRangeFrom(*sub_keys, 0, sub_keys->size());

 indexes = IColumn::mutate(new_indexes->index(*indexes, 0));
 return new_column_unique;
}

mapUniqueIndex is the piece that does the renumbering. It is a counting-sort-like trick that works in O(n) with a flat array, and it has an overflow path for when the dictionary is shared and the indexes do not fit the “max index < number of rows” assumption:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:79
template <typename T>
MutableColumnPtr mapUniqueIndexImpl(PaddedPODArray<T> & index)
{
 if (index.empty())
 return ColumnVector<T>::create();

 auto size = index.size();

 T max_val = index[0];
 for (size_t i = 1; i < size; ++i)
 max_val = std::max(max_val, index[i]);

 /// May happen when dictionary is shared.
 if (max_val > size)
 return mapUniqueIndexImplRef(index);

 auto map_size = static_cast<UInt64>(max_val) + 1;
 PaddedPODArray<T> map(map_size, 0);
 T zero_pos_value = index[0];
 index[0] = 0;
 T cur_pos = 0;
 for (size_t i = 1; i < size; ++i)
 {
 T val = index[i];
 if (val != zero_pos_value && map[val] == 0)
 {
 ++cur_pos;
 map[val] = cur_pos;
 }

 index[i] = map[val];
 }

The “overflow” case (max_val > size) means the column references dictionary entries that belong to a bigger dictionary — exactly what happens when the dictionary is shared. Then a hash map is used instead of a flat array:

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:53
template <typename T>
MutableColumnPtr mapUniqueIndexImplRef(PaddedPODArray<T> & index)
{
 PaddedPODArray<T> copy(index.cbegin(), index.cend());

 HashMap<T, T> hash_map;
 for (auto val : index)
 hash_map.insert({val, static_cast<T>(hash_map.size())});

 auto res_col = ColumnVector<T>::create();
 auto & data = res_col->getData();

 data.resize(hash_map.size());
 for (const auto & val : hash_map)
 data[val.getMapped()] = val.getKey();

 for (auto & ind : index)
 ind = hash_map[ind];

 for (size_t i = 0; i < index.size(); ++i)
 if (data[index[i]] != copy[i])
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected {}, but got {}", toString(data[index[i]]), toString(copy[i]));

 return res_col;
}

That last loop is a self-check: after the renumbering, the new index must still resolve to the same value. Cheap insurance against an off-by-one in a data structure that is otherwise very hard to test.

Inserting from a column with the same dictionary

Once dictionaries are shared, a very common operation becomes trivial: copying a row from another LowCardinality column that happens to use the same dictionary is just a copy of an index.

 cpp
 
 
// src/Columns/ColumnLowCardinality.cpp:180
 size_t position = low_cardinality_src->getIndexes().getUInt(n);

 if (&low_cardinality_src->getDictionary() == &getDictionary())
 {
 /// Dictionary is shared with src column. Insert only index.
 idx.insertIndex(position);
 }
 else
 {
 compactIfSharedDictionary();
 const auto & nested = *low_cardinality_src->getDictionary().getNestedColumn();
 idx.insertIndex(getDictionary().uniqueInsertFrom(nested, position));
 }

The test is a pointer comparison of the dictionary addresses. If they are the same object, the index is valid as-is; otherwise the value has to be re-inserted into this column’s dictionary and the resulting index used.

On disk: one dictionary per part, versioned per granule

Now the hard part. In a MergeTree part, a LowCardinality column is written as two streams: DictionaryKeys and DictionaryIndexes.

The keys stream holds the dictionary; the indexes stream holds, per granule, a header and then the index data. The dictionary is not rewritten per granule — it accumulates until it hits a size limit, at which point it is flushed and a new one starts.

The version byte

The keys stream starts with a version:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:102
struct KeysSerializationVersion
{
 enum Value
 {
 /// Version is written at the start of <name.dict.bin>.
 /// Dictionary is written as number N and N keys after them.
 /// Dictionary can be shared for continuous range of granules, so some marks may point to the same position.
 /// Shared dictionary is stored in state and is read once.
 SharedDictionariesWithAdditionalKeys = 1,
 };

 Value value;

 static void checkVersion(UInt64 version)
 {
 if (version != SharedDictionariesWithAdditionalKeys)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Invalid version for SerializationLowCardinality key column.");
 }

 explicit KeysSerializationVersion(UInt64 version) : value(static_cast<Value>(version)) { checkVersion(version); }
};

The single legal version is called SharedDictionariesWithAdditionalKeys, and the comment tells you the whole story in three lines: the dictionary is written as a count followed by that many keys, it can be shared across a continuous range of granules, and the shared dictionary is kept in the serialization state and read once.

The per-granule header

Every granule’s index data begins with a header that encodes the index width plus three flags:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:124
/// Version is stored at the start of each granule. It's used to store indexes type and flags.
struct IndexesSerializationType
{
 using SerializationType = UInt64;
 /// Need to read dictionary if it wasn't.
 static constexpr SerializationType NeedGlobalDictionaryBit = 1u << 8u;
 /// Need to read additional keys. Additional keys are stored before indexes as value N and N keys after them.
 static constexpr SerializationType HasAdditionalKeysBit = 1u << 9u;
 /// Need to update dictionary. It means that previous granule has different dictionary.
 static constexpr SerializationType NeedUpdateDictionary = 1u << 10u;

 enum Type
 {
 TUInt8 = 0,
 TUInt16,
 TUInt32,
 TUInt64,
 };

 Type type;
 bool has_additional_keys;
 bool need_global_dictionary;
 bool need_update_dictionary;

 static constexpr SerializationType resetFlags(SerializationType type)
 {
 return type & (~(HasAdditionalKeysBit | NeedGlobalDictionaryBit | NeedUpdateDictionary));
 }

The three flags are the entire state machine:

NeedGlobalDictionaryBit — “there is a dictionary in the keys stream that I have not read yet”.
HasAdditionalKeysBit — “this granule carries keys that did not fit in the shared dictionary; they are stored inline, before the index data”.
NeedUpdateDictionary — “the dictionary changed since the previous granule”.

Serialization and deserialization are symmetric and trivial:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:162
 void serialize(WriteBuffer & buffer) const
 {
 SerializationType val = type;
 if (has_additional_keys)
 val |= HasAdditionalKeysBit;
 if (need_global_dictionary)
 val |= NeedGlobalDictionaryBit;
 if (need_update_dictionary)
 val |= NeedUpdateDictionary;
 writeBinaryLittleEndian(val, buffer);
 }

 void deserialize(ReadBuffer & buffer, const ISerialization::DeserializeBinaryBulkSettings & settings)
 {
 SerializationType val = 0;
 readBinaryLittleEndian(val, buffer);

 checkType(val);
 has_additional_keys = (val & HasAdditionalKeysBit) != 0;
 need_global_dictionary = (val & NeedGlobalDictionaryBit) != 0;
 need_update_dictionary = (val & NeedUpdateDictionary) != 0;
 type = static_cast<Type>(resetFlags(val));

 if (settings.native_format)
 {
 if (need_global_dictionary)
 throw Exception(ErrorCodes::INCORRECT_DATA,
 "LowCardinality indexes serialization type for Native format "
 "cannot use global dictionary");
 }
 }

The low four bits carry the index width; bits 8, 9, 10 are the flags. Bits 4–7 are unused — a hole left for future flags.

Writing: the dictionary accumulates, then overflows

The writer keeps the accumulating dictionary in the serialization state:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:232
struct SerializeStateLowCardinality : public ISerialization::SerializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 MutableColumnUniquePtr shared_dictionary;

 explicit SerializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}
};

The write path is where the interesting decisions live:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:561
 const ColumnLowCardinality & low_cardinality_column = typeid_cast<const ColumnLowCardinality &>(column);

 auto * low_cardinality_state = checkAndGetState<SerializeStateLowCardinality>(state);
 auto & global_dictionary = low_cardinality_state->shared_dictionary;
 KeysSerializationVersion::checkVersion(low_cardinality_state->key_version.value);

 size_t max_limit = column.size() - offset;
 limit = limit ? std::min(limit, max_limit) : max_limit;

 /// Do not write anything for empty column. (May happen while writing empty arrays.)
 if (limit == 0)
 return;

 bool need_update_dictionary = global_dictionary == nullptr;
 if (need_update_dictionary)
 global_dictionary = DataTypeLowCardinality::createColumnUnique(*dictionary_type);

 auto sub_column = low_cardinality_column.cutAndCompact(offset, limit);
 ColumnPtr positions = sub_column->getIndexesPtr();
 ColumnPtr keys = sub_column->getDictionary().getNestedColumn();

 if (settings.low_cardinality_max_dictionary_size)
 {
 /// Insert used_keys into global dictionary and update sub_index.
 auto indexes_with_overflow = global_dictionary->uniqueInsertRangeWithOverflow(*keys, 0, keys->size(),
 settings.low_cardinality_max_dictionary_size);

 if (global_dictionary->size() > settings.low_cardinality_max_dictionary_size)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Got dictionary with size {} but max dictionary size is {}",
 global_dictionary->size(), settings.low_cardinality_max_dictionary_size);

 positions = indexes_with_overflow.indexes->index(*positions, 0);
 keys = std::move(indexes_with_overflow.overflowed_keys);

 if (global_dictionary->size() < settings.low_cardinality_max_dictionary_size && !keys->empty())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Has additional keys, but dict size is {} which is less "
 "then max dictionary size ({})", global_dictionary->size(),
 settings.low_cardinality_max_dictionary_size);
 }

cutAndCompact is the first step: it takes the slice of the column being written and produces a private dictionary containing only the keys that slice actually uses. That is what keeps the accumulated dictionary small — without it, every granule would re-insert the whole dictionary.

Then the keys are merged into the global dictionary, and anything that does not fit within low_cardinality_max_dictionary_size becomes “overflowed keys” — keys carried inline by that granule.

The flush happens when the accumulated dictionary reaches the limit:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:602
 bool need_additional_keys = !keys->empty();
 bool need_dictionary = settings.low_cardinality_max_dictionary_size != 0;
 bool need_write_dictionary = !settings.low_cardinality_use_single_dictionary_for_part
 && global_dictionary->size() >= settings.low_cardinality_max_dictionary_size;

 IndexesSerializationType index_version(*positions, need_additional_keys, need_dictionary, need_update_dictionary);
 index_version.serialize(*indexes_stream);

 if (need_write_dictionary)
 {
 const auto & nested_column = global_dictionary->getNestedNotNullableColumn();
 UInt64 num_keys = nested_column->size();
 writeBinaryLittleEndian(num_keys, *keys_stream);
 dict_inner_serialization->serializeBinaryBulk(*nested_column, *keys_stream, 0, num_keys);
 low_cardinality_state->shared_dictionary = nullptr;
 }

 if (need_additional_keys)
 {
 UInt64 num_keys = keys->size();
 writeBinaryLittleEndian(num_keys, *indexes_stream);
 dict_inner_serialization->serializeBinaryBulk(*keys, *indexes_stream, 0, num_keys);
 }

 UInt64 num_rows = positions->size();
 writeBinaryLittleEndian(num_rows, *indexes_stream);
 auto index_serialization = index_version.getDataType()->getDefaultSerialization();
 index_serialization->serializeBinaryBulk(*positions, *indexes_stream, 0, num_rows);

The default limit is 8192 keys:

 cpp
 
 
// src/Core/Settings.cpp:5045
 DECLARE(UInt64, low_cardinality_max_dictionary_size, 8192, R"(
Sets a maximum size in rows of a shared global dictionary for the [LowCardinality](/reference/data-types/lowcardinality) data type that can be written to a storage file system. This setting prevents issues with RAM in case of unlimited dictionary growth. All the data that can't be encoded due to maximum dictionary size limitation ClickHouse writes in an ordinary method.

Possible values:

- Any positive integer.
)", 0) \

So: with an 8192-key limit, a part whose column has 100 distinct values writes the dictionary once; a part with 100 000 distinct values writes the dictionary roughly every 8192 keys and starts a new one.

Note that need_update_dictionary is global_dictionary == nullptr — i.e. “this granule starts a new dictionary”. And the dictionary is flushed after the header was written, so the header says “read the dictionary” and the dictionary follows in the keys stream.

Reading: the reader has to reconstruct the same state

The read side keeps the shared dictionary in the deserialize state, along with a flag that survives granule skips:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:240
struct DeserializeStateLowCardinality : public ISerialization::DeserializeBinaryBulkState
{
 KeysSerializationVersion key_version;
 ColumnUniquePtr global_dictionary;

 IndexesSerializationType index_type{};
 ColumnPtr additional_keys;
 ColumnPtr null_map;
 UInt64 num_pending_rows = 0;

 /// If dictionary should be updated.
 /// Can happen is some granules was skipped while reading from MergeTree.
 /// We should store this flag in State because
 /// in case of long block of empty arrays we may not need read dictionary at first reading.
 bool need_update_dictionary = false;

 /// True if this part has a single dictionary for the entire column. The prefix
 /// reader first checks `has_uniform_marks_callback` as a cheap prefilter, then
 /// verifies it by reading the first dictionary and checking that `DictionaryKeys`
 /// reached EOF. When true, non-continuous reads (granule skips) do not invalidate
 /// the cached dictionary, and the dictionary can be shared across all threads via `clone`.
 bool single_dictionary_for_part = false;

 explicit DeserializeStateLowCardinality(UInt64 key_version_) : key_version(key_version_) {}

 ISerialization::DeserializeBinaryBulkStatePtr cloneImpl(ISerialization::DeserializeBinaryBulkStateCloneContext &) const override
 {
 return std::make_shared<DeserializeStateLowCardinality>(*this);
 }
};

The need_update_dictionary comment is the interesting one: when reading is not continuous (i.e. some granules were skipped), the reader cannot assume that the dictionary it holds is the one the next granule refers to, so it must force a re-read. That flag lives in the state, not on the stack, precisely because a skipped granule may be discovered long after the state was created.

The main read loop:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:752
 if (!settings.continuous_reading)
 {
 low_cardinality_state->num_pending_rows = 0;

 /// Some granules were skipped; the next granule header may signal a dictionary
 /// update. For multi-dict parts this forces a re-read at the next boundary.
 low_cardinality_state->need_update_dictionary = true;
 }

 while (limit)
 {
 if (low_cardinality_state->num_pending_rows == 0)
 {
 if (indexes_stream->eof())
 break;

 auto & index_type = low_cardinality_state->index_type;
 auto & global_dictionary = low_cardinality_state->global_dictionary;

 index_type.deserialize(*indexes_stream, settings);

 bool need_update_dictionary =
 !global_dictionary || index_type.need_update_dictionary || low_cardinality_state->need_update_dictionary;
 if (index_type.need_global_dictionary && need_update_dictionary)
 {
 /// For single-dict parts the in-memory dictionary is valid for the entire part,
 /// so skip re-reading it even if the stream was seeked back to the beginning.
 if (!low_cardinality_state->single_dictionary_for_part || !global_dictionary)
 read_dictionary();
 low_cardinality_state->need_update_dictionary = false;
 }

 if (low_cardinality_state->index_type.has_additional_keys)
 read_additional_keys();
 else
 low_cardinality_state->additional_keys = nullptr;

 readBinaryLittleEndian(low_cardinality_state->num_pending_rows, *indexes_stream);
 }

 size_t num_rows_to_read = std::min<UInt64>(limit, low_cardinality_state->num_pending_rows);
 read_indexes(num_rows_to_read);
 limit -= num_rows_to_read;
 low_cardinality_state->num_pending_rows -= num_rows_to_read;
 }

num_pending_rows is what makes a granule’s worth of rows readable in chunks: the header is read once, then the reader consumes num_pending_rows rows from the index stream, and only when that counter hits zero does it look for the next header.

The single-dictionary-for-part fast path

There is a special case worth highlighting: a part where the column has exactly one dictionary for the whole part. The prefix reader detects it by reading the first dictionary and checking whether the DictionaryKeys stream reached EOF:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:348
 if (might_have_single_dictionary && !stream->eof())
 {
 /// For `Wide` parts that might have a single dictionary, the stream is
 /// now positioned right at the dictionary data (after the version).
 /// Read the first dictionary and check EOF afterwards. If this proves
 /// that there are no more dictionaries, all threads sharing this prefix
 /// state via `DeserializationPrefixesCache` get the dictionary through
 /// a `shared_ptr` clone instead of each reading it independently.

and

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:377
 if (stream->eof())
 {
 new_state->single_dictionary_for_part = true;
 new_state->global_dictionary = std::move(global_dictionary);
 }
 else
 {
 /// The mark heuristic can be fooled by several dictionaries written
 /// inside one mark interval. Reset the `DictionaryKeys` stream back
 /// to the normal post-prefix position and let data deserialization
 /// read dictionary updates from the stream.
 if (!settings.seek_to_start_callback)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot reset LowCardinality dictionary stream without seek_to_start_callback");

 settings.seek_to_start_callback(dictionary_keys_path);

 UInt64 repeated_keys_version = 0;
 readBinaryLittleEndian(repeated_keys_version, *stream);
 if (repeated_keys_version != keys_version)
 throw Exception(ErrorCodes::INCORRECT_DATA, "Inconsistent version while resetting LowCardinality dictionary stream");
 }

This is a heuristic with a verification step. The cheap check (has_uniform_marks_callback) says “all marks point at the same position in the keys stream, so maybe there is only one dictionary”. The expensive check (actually reading the dictionary and seeing EOF) confirms it. If the heuristic lied, the stream is rewound and the normal path takes over.

That’s a pattern you will see repeatedly in ClickHouse’s MergeTree reader: cheap prefilter, expensive confirmation, safe fallback.

When the fast path holds, cloneImpl copies the state (including the dictionary pointer) so every thread reading the part gets the same dictionary without re-reading it.

The overflow path on the read side

When a granule carries inline “additional keys”, the reader has to merge two dictionaries: the global one and the granule-local one. mapIndexWithAdditionalKeys does that with two flat maps — one for the global dictionary’s keys and one for the overflow keys:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:515
 /// Update column and return map with old indexes.
 /// Let N is the number of distinct values which are less than max_size;
 /// old_column - column before function call;
 /// new_column - column after function call:
 /// * if old_column[i] < max_size, than
 /// dictionary_map[new_column[i]] = old_column[i]
 /// * else
 /// additional_keys_map[new_column[i]] = old_column[i] - dict_size + N
 IndexMapsWithAdditionalKeys mapIndexWithAdditionalKeys(IColumn & column, size_t dict_size)

and then the keys are stitched back together:

 cpp
 
 
// src/DataTypes/Serializations/SerializationLowCardinality.cpp:726
 else
 {
 auto maps = mapIndexWithAdditionalKeys(*indexes_column, global_dictionary->size());

 auto used_keys = IColumn::mutate(global_dictionary->getNestedColumn()->index(*maps.dictionary_map, 0));

 if (!maps.additional_keys_map->empty())
 {
 if (additional_keys == nullptr)
 throw Exception(ErrorCodes::INCORRECT_DATA, "No additional keys found.");

 auto used_add_keys = additional_keys->index(*maps.additional_keys_map, 0);

 if (dictionary_type->isNullable())
 {
 ColumnPtr null_map = ColumnUInt8::create(used_add_keys->size(), static_cast<UInt8>(0));
 used_add_keys = ColumnNullable::create(used_add_keys, null_map);
 }

 used_keys->insertRangeFrom(*used_add_keys, 0, used_add_keys->size());
 }

 low_cardinality_column.insertRangeFromDictionaryEncodedColumn(*used_keys, *indexes_column);
 }

The final call re-inserts the merged keys into the destination column’s own dictionary and remaps the indexes one last time.

The whole thing in SQL

 sql
 
 
CREATE TABLE events
(
 `ts` DateTime,
 `user_id` UInt64,
 `kind` LowCardinality(String), -- 12 distinct values in 100M rows
 `country` LowCardinality(FixedString(2))
)
ENGINE = MergeTree
ORDER BY (ts, user_id);

 sql
 
 
-- The dictionary is shared across every block of the part, so the per-column
-- cost is one byte per row (UInt8 index) plus the dictionary once.
SELECT kind, count() FROM events GROUP BY kind;

The type-level guarantee is that LowCardinality is only worth it while the dictionary stays small; the documentation says so plainly:

text

// src/DataTypes/DataTypeLowCardinality.cpp:198
The efficiency of using `LowCardinality` data type depends on data diversity. If a dictionary contains less than 10,000 distinct values, then ClickHouse mostly shows higher efficiency of data reading and storing. If a dictionary contains more than 100,000 distinct values, then ClickHouse can perform worse in comparison with using ordinary data types.

Trade-offs

What the hack buys you.

A LowCardinality(String) column with 12 distinct values costs one byte per row plus a few hundred bytes of dictionary, instead of ~10 bytes per row of strings. It also means the scan is a byte scan, not a string scan: filters, GROUP BY, and ORDER BY all operate on UInt8 indexes, and the dictionary is touched only for the rows that survive the filter.

What it costs you.

Every value access is two dependent loads: read the index, then read the dictionary entry. That is a pointer chase a plain String column does not have — although for a plain string column you also pay a pointer chase into the character arena, so the two are more comparable than they look.

The dictionary is a global per part, and the index stream has to carry a per-granule header with three flags and a row count, plus the possibility of inline overflow keys. A plain column has none of that. For a column with 100 000+ distinct values you have bought complexity and lost compression.

The sharing is also asymmetric: byteSize() excludes a shared dictionary while allocatedBytes() includes it. That is correct but subtle, and it means the two numbers in system.parts_columns can disagree in a way that looks like a bug until you know why.

The design decision that makes it work. The dictionary is not stored per block, and not per granule; it is stored per part, capped at 8192 keys, and flushed when it overflows. Everything else — the three header bits, the need_update_dictionary flag, the single_dictionary_for_part heuristic — exists to make that one decision survive partial reads, granule skips and multi-threaded readers.

Wrapping up

The one thing to remember: LowCardinality moves the repetition out of the rows and into a shared dictionary, and the whole design is about making that dictionary shared — across blocks, granules, parts, and threads.

The rule of thumb: below ~10 000 distinct values, use it. Above ~100 000, don’t. In between, measure.

What to go do about it: count the distinct values in your “status” or “country” columns (uniqExact), then wrap them. One byte per row instead of ten is not a rounding error on a hundred-million-row table.

ClickHouse Marks Empty Hash Table Cells With Zeroed Memory

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse’s hash table has no occupancy bit, no tombstone, and no sentinel pointer. A cell is empty when its key is all zero bytes, and the buffer starts zeroed because the allocator uses calloc. That one decision ripples through the cell layout, the zero-key special case, deletion, and even why the hash function has to be good.

Every GROUP BY in ClickHouse is a hash table. The table has to hold millions of keys, be probed once per input row, and be resized while it is being filled — all while the query is competing for memory with everything else on the machine.

std::unordered_map is simply not competitive here: it allocates a node per element, keeps a pointer chain, and pays a pointer chase for every collision.

So ClickHouse ships its own hash table in src/Common/HashTable/HashTable.h: a flat array of cells, open addressing, linear probing. The probing is textbook. The interesting part is the bookkeeping around emptiness.

There is no “occupied” bit. No tombstone. No sentinel pointer. No version counter. A cell is empty when its key is all zero bytes, and the whole buffer starts out zeroed by the allocator.

That single decision shapes the entire file: the allocator, the cell layout, the way the zero key is handled, the way deletion works, and the reason the hash function has to be good.

The shape of the table

 cpp
 
 
// src/Common/HashTable/HashTable.h:518
template <typename Key, typename Cell, typename Hash, typename Grower, typename Allocator>
class HashTable : private boost::noncopyable,
 protected Hash,
 protected Allocator,
 protected Cell::State,
 public ZeroValueStorage<Cell::need_zero_value_storage, Cell> /// empty base optimization
{

The table is one contiguous buffer of Cells plus a Grower that knows the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:550
 size_t m_size = 0; /// Amount of elements
 Cell * buf; /// A piece of memory for all elements except the element with zero key.
 Grower grower;

Cell is a policy class. The simplest one stores nothing but a key:

 cpp
 
 
// src/Common/HashTable/HashTable.h:154
template <typename Key, typename Hash, typename TState = HashTableNoState>
struct HashTableCell
{
 using State = TState;

 using key_type = Key;
 using value_type = Key;
 using mapped_type = VoidMapped;

 Key key;

The map flavour stores a pair, and the variant that ClickHouse uses for aggregation additionally caches the hash so that a rehash does not have to recompute it:

 cpp
 
 
// src/Common/HashTable/HashMap.h:165
template <typename Key, typename TMapped, typename Hash, typename TState = HashTableNoState>
struct HashMapCellWithSavedHash : public HashMapCell<Key, TMapped, Hash, TState> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `saved_hash` is set by `setHash` immediately after placement construction on insert; on the hot aggregation/join path we must avoid the redundant store
{
 using Base = HashMapCell<Key, TMapped, Hash, TState>;

 size_t saved_hash;

Empty means zero

Here is the whole emptiness protocol:

 cpp
 
 
// src/Common/HashTable/HashTable.h:191
 /// Whether the key is zero. In the main buffer, cells with a zero key are considered empty.
 /// If zero keys can be inserted into the table, then the cell for the zero key is stored separately, not in the main buffer.
 /// Zero keys must be such that the zeroed-down piece of memory is a zero key.
 bool isZero(const State & state) const { return isZero(key, state); }
 static bool isZero(const Key & key, const State & /*state*/) { return ZeroTraits::check(key); }

 /// Set the key value to zero.
 void setZero() { ZeroTraits::set(key); }

and the generic part:

 cpp
 
 
// src/Common/HashTable/HashTable.h:89
namespace ZeroTraits
{

template <typename T>
bool check(const T x)
{
 return bitEquals(x, T{});
}

template <typename T>
void set(T & x) { x = T{}; }

}

bitEquals is bitwise, not ==, and that matters for floating point:

 cpp
 
 
// src/Common/HashTable/HashTable.h:77
template <typename T>
inline ALWAYS_INLINE bool bitEquals(T a, T b)
{
 if constexpr (is_floating_point<T>)
 /// Note that memcmp with constant size is a compiler builtin.
 return 0 == memcmp(&a, &b, sizeof(T)); /// NOLINT
 else
 return a == b;
}

The comment above it explains why bit equality is mandatory rather than merely nice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:71
/** Numbers are compared bitwise.
 * Complex types are compared by operator== as usual (this is important if there are gaps).
 *
 * This is needed if you use floats as keys. They are compared by bit equality.
 * Otherwise the invariants in hash table probing do not met when NaNs are present.
 */

NaN never equals itself, so a ==-based emptiness test could never recognise an empty float cell — and the probe loop would run right off the end of the buffer.

The zeroed buffer comes from the allocator:

 cpp
 
 
// src/Common/HashTable/HashTableAllocator.h:8
/**
 * We are going to use the entire memory we allocated when resizing a hash
 * table, so it makes sense to pre-fault the pages so that page faults don't
 * interrupt the resize loop. Set the allocator parameter accordingly.
 */
using HashTableAllocator = Allocator<true /* clear_memory */, true /* populate */>;

clear_memory becomes calloc:

 cpp
 
 
// src/Common/Allocator.cpp:79
 if (alignment <= MALLOC_MIN_ALIGNMENT) [[likely]]
 {
 if constexpr (clear_memory)
 buf = __real_calloc(size, 1);
 else
 buf = __real_malloc(size);

So the invariant is: the bytes of a fresh buffer are zero, therefore a fresh cell is empty. Nothing more is needed to track occupancy. The diagram of the buffer is therefore very simple:

 text
 
 
 buf buf + bufSize()
 | |
 v v
 +--------+--------+--------+--------+--------+--------+--------+--------+
 | cell 0 | cell 1 | cell 2 | cell 3 | cell 4 | cell 5 | cell 6 | cell 7 |
 +--------+--------+--------+--------+--------+--------+--------+--------+
 key=A (zero) key=C (zero) key=B (zero) (zero)
 [used] [empty] [used] [empty] [used] [empty] [empty]

Note what is not there: no occupancy byte, no pointer, no version. A HashMapCell<UInt64, AggregateState*> is 16 bytes of pure payload. The empty/used distinction is free.

The zero key gets its own slot

There is one hole in the “zero means empty” scheme: 0 is a perfectly legal key. If it were stored in the buffer, it would be indistinguishable from an empty cell. So it is stored outside the buffer, in a ZeroValueStorage that the table inherits from:

 cpp
 
 
// src/Common/HashTable/HashTable.h:474
template <typename Cell>
struct ZeroValueStorage<true, Cell> // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
private:
 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

 void setHasZero()
 {
 has_zero = true;
 new (zeroValue()) Cell();
 }

and the insert path routes the zero key to it:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1354
 bool ALWAYS_INLINE emplaceIfZero(const Key & x, LookupResult & it, bool & inserted, size_t hash_value)
 {
 /// If it is claimed that the zero key can not be inserted into the table.
 if constexpr (!Cell::need_zero_value_storage)
 return false;

 if (unlikely(Cell::isZero(x, *this)))
 {
 it = this->zeroValue();

 if (!this->hasZero())
 {
 ++m_size;
 this->setHasZero();
 this->zeroValue()->setHash(hash_value);
 inserted = true;
 }
 else
 inserted = false;

 return true;
 }

 return false;
 }

The zero key lives at a fixed address, so a lookup for it is a branch and a pointer, never a probe. The cost of the scheme is one bool plus one Cell of storage per table, and a slightly awkward iteration order (the zero cell is visited first, then the buffer).

And this is not an exotic corner. 0 is a perfectly ordinary grouping key:

 sql
 
SELECT k, count() FROM t GROUP BY k;

with a k column full of zeros produces a group whose key is the zero key, and that group is the one stored in the separate slot.

Note also that the design is conditional: a cell type can declare that zero keys are impossible (need_zero_value_storage = false), and then ZeroValueStorage<false, Cell> degenerates to nothing at all. The string tables do exactly that — see the article on the string-specialised map.

Probing

The probe sequence is the simplest possible one: linear probing with a mask.

 cpp
 
 
// src/Common/HashTable/HashTable.h:232
 /// The size of the hash table in the cells.
 size_t bufSize() const { return 1ULL << size_degree; }

 size_t maxFill() const { return 1ULL << (size_degree - 1); }
 size_t mask() const { return bufSize() - 1; }

 /// From the hash value, get the cell number in the hash table.
 size_t place(size_t x) const { return x & mask(); }

 /// The next cell in the collision resolution chain.
 size_t next(size_t pos) const { ++pos; return pos & mask(); }

 /// Whether the hash table is sufficiently full. You need to increase the size of the hash table, or remove something unnecessary from it.
 bool overflow(size_t elems) const { return elems > maxFill(); }

The load factor is 0.5: maxFill() is half the buffer. And the probe loop itself:

 cpp
 
 
// src/Common/HashTable/HashTable.h:559
 size_t ALWAYS_INLINE findCell(const Key & x, size_t hash_value, size_t place_value) const
 {
 while (!buf[place_value].isZero(*this) && !buf[place_value].keyEquals(x, hash_value, *this))
 {
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }

 return place_value;
 }

Two things to note. First, isZero is checked before keyEquals: the loop stops at the first empty cell, because with linear probing an empty cell terminates the chain. Second, the comparison is a bitwise equality against the key, and for the HashMapCellWithSavedHash variant it is short-circuited by the cached hash.

The zero key is handled before the probe:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1617
 LookupResult ALWAYS_INLINE find(const Key & x)
 {
 if (Cell::isZero(x, *this))
 return this->hasZero() ? this->zeroValue() : nullptr;

 return findNonZeroCell(x, hash(x));
 }

Insertion

emplace computes the hash, routes the zero key, and otherwise finds a place:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1380
 template <typename KeyHolder>
 void ALWAYS_INLINE emplaceNonZeroImpl(size_t place_value, KeyHolder && key_holder,
 LookupResult & it, bool & inserted, size_t hash_value)
 {
 it = &buf[place_value];

 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);
 const auto & key = keyHolderGetKey(key_holder);

 new (&buf[place_value]) Cell(key, *this);
 buf[place_value].setHash(hash_value);
 inserted = true;
 ++m_size;

 if (unlikely(grower.overflow(m_size)))
 {
 try
 {
 resize();
 }
 catch (...)
 {
 /** If we have not resized successfully, then there will be problems.
 * There remains a key, but uninitialized mapped-value,
 * which, perhaps, can not even be called a destructor.
 */
 --m_size;
 buf[place_value].setZero();
 inserted = false;
 keyHolderDiscardKey(key_holder);
 throw;
 }

 // The hash table was rehashed, so we have to re-find the key.
 it = findNonZeroCell(key, hash_value);
 chassert(it);
 }
 }

Two details worth pausing on. The cell is placement-newed; the buffer is raw memory, and the destructor of a non-trivial cell is only run for cells that were actually constructed (destroyElements walks the table and calls the destructor only for non-zero cells).

And the resize happens after the insert, not before. That means the table can exceed its nominal capacity by one element, and it means the iterator has to be re-found afterwards, because the rehash moved the cell. The exception handler is careful to undo the insert (decrement the size, zero the cell) so that a failed allocation does not leave a half-built cell behind.

Growing: the wrap-around special case

The resize path is where the linear-probing invariants get hairy. When the buffer grows, every element has to be re-placed, and an element that was at the end of the old buffer may belong at the beginning of the new one because its collision chain wrapped around:

 cpp
 
 
// src/Common/HashTable/HashTable.h:837
 void resize(size_t for_num_elems = 0, size_t for_buf_size = 0)
 {
#ifdef DBMS_HASH_MAP_DEBUG_RESIZES
 Stopwatch watch;
#endif

 size_t old_size = grower.bufSize();

 /** In case of exception for the object to remain in the correct state,
 * changing the variable `grower` (which determines the buffer size of the hash table)
 * is postponed for a moment after a real buffer change.
 * The temporary variable `new_grower` is used to determine the new size.
 */
 Grower new_grower = grower;

and the re-placement loop:

 cpp
 
 
// src/Common/HashTable/HashTable.h:884
 size_t i = 0;
 for (; i < old_size; ++i)
 if (!buf[i].isZero(*this))
 reinsert(buf[i], buf[i].getHash(*this));

 /** There is also a special case:
 * if the element was to be at the end of the old buffer, [ x]
 * but is at the beginning because of the collision resolution chain, [o x]
 * then after resizing, it will first be out of place again, [ xo ]
 * and in order to transfer it where necessary,
 * after transferring all the elements from the old halves you need to [ o x ]
 * process tail from the collision resolution chain immediately after it [ o x ]
 */
 size_t new_size = grower.bufSize();
 for (; i < new_size && !buf[i].isZero(*this); ++i)
 reinsert(buf[i], buf[i].getHash(*this));

The picture in that comment is the whole problem: after the buffer is doubled, the element at the very end of the old buffer (x) is now in the middle of the new one, and the element that was pushed behind it (o) belongs before it. The second loop therefore keeps going past the end of the old buffer, as long as the cells it finds are non-empty, re-inserting them.

The re-insertion itself is a memcpy of a trivially-copyable cell into the new position:

 cpp
 
 
// src/Common/HashTable/HashTable.h:918
 void reinsert(Cell & x, size_t hash_value)
 {
 size_t place_value = grower.place(hash_value);

 /// If the element is in its place.
 if (&x == &buf[place_value])
 return;

 /// Compute a new location, taking into account the collision resolution chain.
 place_value = findCell(Cell::getKey(x.getValue()), hash_value, place_value);

 /// If the item remains in its place in the old collision resolution chain.
 if (!buf[place_value].isZero(*this))
 return;

 /// Copy to a new location and zero the old one.
 x.setHash(hash_value);
 memcpy(static_cast<void*>(&buf[place_value]), &x, sizeof(x)); /// NOLINT(bugprone-undefined-memory-manipulation)
 x.setZero();
 }

The grower doubles the size until max_size_degree (23) and then grows by one step at a time:

 cpp
 
 
// src/Common/HashTable/HashTable.h:247
 /// Increase the size of the hash table.
 void increaseSize()
 {
 size_degree += size_degree >= max_size_degree ? 1 : 2;
 }

Robin Hood: what changed recently

The most interesting recent addition to the file is a second probing discipline. The classic HashTableGrower keeps the load factor at 0.5; the new HashTableGrowerWithPrecalculationAndRobinHood keeps 0.5 until the table reaches 2^16 cells, and then switches to Robin Hood hashing with a 90% load factor — in place, without reallocating:

 cpp
 
 
// src/Common/HashTable/HashTable.h:336
/** Like `HashTableGrowerWithPrecalculation`, but adapts to large hash tables.
 *
 * For large hash tables, random memory access is the main bottleneck, not the computations.
 * When the buffer reaches 2^robin_hood_size_degree cells, the grower switches the hash table to the Robin Hood mode:
 * the max load factor is increased from 0.5 to robin_hood_max_load_factor_percent / 100 (the hash table becomes up to two times smaller),
 * and the elements inside collision resolution chains are ordered by their optimal positions,
 * so a lookup of a missing key can stop early instead of walking to the end of the chain.
 *
 * The memory layout is the same in both modes, and switching happens in place, without growing the buffer.
 * Hash tables check the mode only if the grower has the `isRobinHood` method (see `GrowerWithRobinHood`),
 * so there is no cost for other growers.
 *
 * Requirements to the cell: obtaining the hash value with `getHash` must be cheap (the hash is saved or cheap to compute),
 * and the hash table must not keep pointers to cells while inserting other elements,
 * because an insertion may move other elements along the collision resolution chain.
 */

The payoff is twofold. The table becomes up to twice as small (0.9 vs 0.5 load factor), which halves the number of cache misses in a random probe; and a failed lookup can stop early, because the chain is sorted by “distance from the ideal position”:

 cpp
 
 
// src/Common/HashTable/HashTable.h:607
 size_t ALWAYS_INLINE findCellRobinHood(const Key & x, size_t hash_value, size_t place_value, bool & found) const
 {
 for (size_t distance = 0;; ++distance)
 {
 const Cell & cell = buf[place_value];
 if (cell.isZero(*this))
 {
 found = false;
 return place_value;
 }
 if (cell.keyEquals(x, hash_value, *this))
 {
 found = true;
 return place_value;
 }
 /// The element at the optimal position of the key cannot be closer to its own optimal position, so do not compute its hash.
 if (distance && distanceFromOptimalPosition(cell, place_value) < distance)
 {
 found = false;
 return place_value;
 }
 place_value = grower.next(place_value);
#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
 ++collisions;
#endif
 }
 }

“Distance from the optimal position” is just the difference between where the key would go and where it is, measured modulo the buffer size:

 cpp
 
 
// src/Common/HashTable/HashTable.h:597
 size_t ALWAYS_INLINE distanceFromOptimalPosition(const Cell & cell, size_t place_value) const
 {
 return grower.place(place_value - grower.place(cell.getHash(*this)));
 }

Robin Hood ordering also changes insertion: the new key may have to be placed before an existing one, which means shifting the rest of the chain to the right:

 cpp
 
 
// src/Common/HashTable/HashTable.h:664
 void NO_INLINE shiftCellsRight(size_t place_value)
 {
 size_t empty_place = findEmptyCell(place_value);

 if (empty_place < place_value)
 {
 /// The chain wraps around the end of the buffer.
 size_t last_place = grower.bufSize() - 1;
 memmove(static_cast<void *>(&buf[1]), static_cast<const void *>(&buf[0]), empty_place * sizeof(Cell));
 memcpy(static_cast<void *>(&buf[0]), static_cast<const void *>(&buf[last_place]), sizeof(Cell));
 empty_place = last_place;
 }

 memmove(static_cast<void *>(&buf[place_value + 1]), static_cast<const void *>(&buf[place_value]), (empty_place - place_value) * sizeof(Cell));
 buf[place_value].setZero();
 }

and the chains have to be re-sorted when the mode is switched on — by an insertion sort, since the chains are “already almost ordered” in practice:

 cpp
 
 
// src/Common/HashTable/HashTable.h:681
 /** Reorder the elements in every collision resolution chain by their optimal positions, as required in the Robin Hood mode.
 * Used after the elements were placed by the ordinary linear probing: on resize, or when switching to the Robin Hood mode.
 * It is an insertion sort, which is fast for the chains that are already almost ordered.
 */

Deletion without tombstones

Deleting from an open-addressing table usually means writing a tombstone, so that the probe chain is not cut short. Tombstones accumulate, they need to be cleaned up, and they slow down every subsequent lookup.

ClickHouse does not use tombstones. Instead it moves the elements of the chain:

 cpp
 
 
// src/Common/HashTable/HashTable.h:731
 /// Remove the element at place_value without calling its destructor, moving the following elements of the collision resolution chain to keep them reachable.
 /// Not inlined to keep the code of inserts small.
 void eraseCellAt(size_t erased_key_position)
 {
 /// We need to guarantee loop termination because there will be empty position
 chassert(m_size < grower.bufSize());

 size_t next_position = erased_key_position;

 /**
 * During element deletion there is a possibility that the search will be broken for one
 * of the following elements, because this place erased_key_position is empty. We will check
 * next_element. Consider a sequence from (erased_key_position, next_element], if the
 * optimal_position of next_element falls into it, then removing erased_key_position
 * will not break search for next_element.
 * If optimal_position of the element does not fall into the sequence (erased_key_position, next_element]
 * then deleting a erased_key_position will break search for it, so we need to move next_element
 * to erased_key_position. Now we have empty place at next_element, so we apply the identical
 * procedure for it.
 * If an empty element is encountered then means that there is no more next elements for which we can
 * break the search so we can exit.
 *
 * In the Robin Hood mode, the elements of a chain are ordered by their optimal positions,
 * and this procedure moves them one cell to the left, keeping the order.
 */

The rule the comment spells out is the classic one: an element may stay where it is if its ideal position is not before the hole; otherwise it has to slide left into the hole, which moves the hole one step further right, and the procedure repeats:

 cpp
 
 
// src/Common/HashTable/HashTable.h:798
 /// Move the element to the freed place
 memcpy(static_cast<void *>(&buf[erased_key_position]), static_cast<void *>(&buf[next_position]), sizeof(Cell));

 /// Now we have another freed place
 erased_key_position = next_position;
 }

 buf[erased_key_position].setZero();
 --m_size;
 }

This costs a memcpy per displaced element but keeps the table free of garbage: the buffer is always exactly as full as m_size says, and a probe chain never walks over a tombstone.

The allocator: zeroed and pre-faulted

HashTableAllocator is Allocator<true /* clear_memory */, true /* populate */>. The second flag is the interesting one: it pre-faults the pages of a large allocation with MADV_POPULATE_WRITE, so that the resize loop does not stop for page faults:

 cpp
 
 
// src/Common/Allocator.cpp:59
void prefaultPages([[maybe_unused]] void * buf_, [[maybe_unused]] size_t len_)
{
#if defined(MADV_POPULATE_WRITE)
 if (len_ < POPULATE_THRESHOLD)
 return;

 if (!is_supported_by_kernel) [[unlikely]]
 return;

 auto [buf, len] = adjustToPageSize(buf_, len_, ::getPageSize());
 ::madvise(buf, len, MADV_POPULATE_WRITE);
#endif
}

 cpp
 
 
// src/Common/Allocator.cpp:132
/// Constant is chosen almost arbitrarily, what I observed is 128KB is too small, 1MB is almost indistinguishable from 64MB and 1GB is too large.
extern const size_t POPULATE_THRESHOLD = std::max(Int64{16 * 1024 * 1024}, ::getPageSize());

The rationale is spelled out in HashTableAllocator.h itself: the whole allocation will be used, so pre-faulting it costs nothing extra and removes a source of latency spikes from the middle of a rehash. The kernel support check is a version test (Linux 5.14+), not an EINVAL probe, because madvise returns EINVAL for many unrelated reasons.

And note how calloc and the “zero means empty” invariant fit together perfectly: calloc of a large block is free on Linux for big allocations (the kernel hands out zero pages), so the “clear the buffer” requirement that the emptiness protocol imposes costs nothing at all.

Prefetching

A hash table probe is a random memory access; at 90% load factor with Robin Hood ordering, a lookup of a missing key can touch several cache lines. ClickHouse tries to hide that latency by prefetching the cell a few iterations ahead. The helper measures its own iteration latency to decide how far ahead to look:

 cpp
 
 
// src/Common/HashTable/Prefetching.h:33
class PrefetchingHelper
{
public:
 size_t calcPrefetchLookAhead()
 {
 static constexpr auto assumed_load_latency_ns = 100;
 static constexpr auto just_coefficient = 4;
 static constexpr auto numerator = just_coefficient * assumed_load_latency_ns * iterations_to_measure;
 size_t elapsed_ns = watch.elapsedNanoseconds();
 if (unlikely(elapsed_ns == 0))
 return max_look_ahead_value;
 size_t look_ahead = (numerator + elapsed_ns - 1) / elapsed_ns;
 return std::clamp<size_t>(look_ahead, min_look_ahead_value, max_look_ahead_value);
 }

 static constexpr size_t getInitialLookAheadValue() { return min_look_ahead_value; }

 static constexpr size_t iterationsToMeasure() { return iterations_to_measure; }

private:
 static constexpr size_t iterations_to_measure = 100;
 static constexpr size_t min_look_ahead_value = 4;
 static constexpr size_t max_look_ahead_value = 32;

The look-ahead is thus between 4 and 32 cells, tuned to the machine: the formula is 4 * 100ns * 100 / elapsed, i.e. four times the assumed memory latency, expressed in iterations.

The prefetching iterator of the table uses it, and also prefetches the key bytes of the cell it is about to look at, not just the cell:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1021
 void prefetch()
 {
 static_assert(CouldPrefetchKey<cell_type>);
 auto * buf_end = container->buf + container->grower.bufSize();
 iter_count++;
 if (prefetch_ptr < buf_end && prefetched_count < prefetch_ahead) [[likely]]
 {
 if (iter_count == DB::PrefetchingHelper::iterationsToMeasure()) [[unlikely]]
 prefetch_ahead = prefetching.calcPrefetchLookAhead();
 auto n = prefetch_ahead - prefetched_count;
 cell_type * last_ptr = nullptr;
 for (size_t i = 0; i < n; ++i)
 {
 while (prefetch_ptr < buf_end && prefetch_ptr->isZero(*container))
 ++prefetch_ptr;

 if (prefetch_ptr < buf_end) [[likely]]
 {
 last_ptr = prefetch_ptr;
 prefetch_ptr++;
 prefetched_count++;
 }
 else
 break;
 }

 if (last_ptr) [[likely]]
 keyPrefetch(last_ptr->getKey());
 }
 }

keyPrefetch touches up to four cache lines of the key data:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:182
inline void keyPrefetch(const std::string_view key)
{
 const size_t cache_line_size = 64;
 // Prefetch up to 4 cache lines.
 const size_t max_prefetch = cache_line_size * 4;
 const size_t n = std::min(key.size(), max_prefetch);
 const char * ptr = key.data();
 const char * end = ptr + n;
 for (; ptr < end; ptr += cache_line_size)
 {
 __builtin_prefetch(ptr);
 }
}

Keys that live outside the table

Not every key can live inside the cell. A std::string_view key points at data owned by somebody else — often a column that will be gone by the time the aggregation is finalised. ClickHouse solves this with a key holder: a small policy object that knows how to make a temporary key persistent, and when to throw it away.

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:53
/**
 * Returns the key. Can return the temporary key initially.
 * After the call to keyHolderPersistKey(), must return the persistent key.
 */
template <typename Key>
inline Key & ALWAYS_INLINE keyHolderGetKey(Key && key) { return key; } /// NOLINT(bugprone-return-const-ref-from-parameter)

/**
 * Make the key persistent. keyHolderGetKey() must return the persistent key
 * after this call.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderPersistKey(Key &&) {}

/**
 * Discard the key. Calling keyHolderGetKey() is ill-defined after this.
 */
template <typename Key>
inline void ALWAYS_INLINE keyHolderDiscardKey(Key &&) {}

The default is a no-op: for a key stored by value, there is nothing to do, and the branch is optimised away. The ArenaKeyHolder version copies the bytes into an Arena when the key turns out to be new:

 cpp
 
 
// src/Common/HashTable/HashTableKeyHolder.h:98
inline void ALWAYS_INLINE keyHolderPersistKey(DB::ArenaKeyHolder & holder)
{
 // Normally, our hash table shouldn't ask to persist a zero key,
 // but it can happened in the case of clearable hash table (ClearableHashSet, for example).
 // The clearable hash table doesn't use zero storage and
 // distinguishes empty keys by using cell version, not the value itself.
 // So, when an empty std::string_view is inserted in ClearableHashSet we'll get here key of zero size.
 holder.key = std::string_view{holder.pool.insert(holder.key.data(), holder.key.size()), holder.key.size()};
}

The elegance is in where the decision is made. The hash table knows whether the key it just found was new; the caller does not. So the table calls keyHolderDiscardKey when the key already existed (nothing to persist, and the arena allocation is rolled back) and keyHolderPersistKey when it did not:

 cpp
 
 
// src/Common/HashTable/HashTable.h:1386
 if (!buf[place_value].isZero(*this))
 {
 keyHolderDiscardKey(key_holder);
 inserted = false;
 return;
 }

 keyHolderPersistKey(key_holder);

The hash function has to be good

Because the table uses linear probing and the low bits of the hash to pick the bucket, a poor hash function is catastrophic: the low bits of a timestamp-like key are nearly constant, and every key lands in the same few cells. The header says so bluntly:

 cpp
 
 
// src/Common/HashTable/Hash.h:15
/** Hash functions that are better than the trivial function std::hash.
 *
 * Example: when we do aggregation by the visitor ID, the performance increase is more than 5 times.
 * This is because of following reasons:
 * - in Metrica web analytics system, visitor identifier is an integer that has timestamp with seconds resolution in lower bits;
 * - in typical implementation of standard library, hash function for integers is trivial and just use lower bits;
 * - traffic is non-uniformly distributed across a day;
 * - we are using open-addressing linear probing hash tables that are most critical to hash function quality,
 * and trivial hash function gives disastrous results.
 */

The default integer hash is the MurmurHash finalizer:

 cpp
 
 
// src/Common/HashTable/Hash.h:26
/** Taken from MurmurHash. This is Murmur finalizer.
 * Faster than intHash32 when inserting into the hash table UInt64 -> UInt64, where the key is the visitor ID.
 */
inline UInt64 intHash64(UInt64 x)
{
 x ^= x >> 33;
 x *= 0xff51afd7ed558ccdULL;
 x ^= x >> 33;
 x *= 0xc4ceb9fe1a85ec53ULL;
 x ^= x >> 33;

 return x;
}

Five instructions and two multiplies turn “the low bits are constant” into “the low bits are random”. For string keys and general keys, the CRC32 hardware instruction is used where it exists — three cycles of latency and one per cycle of throughput, which is why the string tables use it even though CRC32 is a poor hash in the abstract:

 cpp
 
 
// src/Common/HashTable/Hash.h:40
/** CRC32C is not very high-quality as a hash function,
 * according to avalanche and bit independence tests (see SMHasher software), as well as a small number of bits,
 * but can behave well when used in hash tables,
 * due to high speed (latency 3 + 1 clock cycle, throughput 1 clock cycle).
 * Works only with SSE 4.2 support.
 */

Trade-offs

What the “zero means empty” scheme buys:

Zero bytes of metadata per cell. A HashMapCell<UInt64, T*> is 16 bytes, all of it payload. A scheme with an occupancy bit would need to steal a bit from the key or add a byte.
A free calloc. The kernel can hand out pre-zeroed pages; the emptiness protocol costs nothing at allocation time.
No tombstones, ever. Deletion moves cells instead of marking them, so the table never degrades.
A cheap scan. for (auto & cell : buf) if (!cell.isZero()) ... is the iteration order; no indirection, no linked list.

What it costs:

The key must be zero-comparable, and bit-comparable. Floating point keys are compared with memcmp, and NaN semantics have to be papered over.
The zero key is special. It lives in a separate slot, it is visited first during iteration, and its handling adds a branch to every insert and lookup.
The hash must be excellent, because linear probing amplifies clustering, and the low bits of the hash are used directly.
Rehashing is a memcpy storm. Growing the table means moving every cell, and the wrap-around case needs a second pass over the head of the new buffer.

And the recent Robin Hood addition changes the balance again: it trades a little CPU (reordering chains, occasionally shifting cells on insert) for a halving of the memory footprint at large sizes — which, for a table that is probed randomly, is exactly the right trade, because the cost is cache misses, not instructions.

Wrapping up

The one thing to remember: “empty means all-zero bytes” buys you a cell layout that is 100% payload, with no occupancy bit to steal from your key.

The part people get wrong: this is not free. The zero key becomes a special case, float keys need bitwise comparison, and the hash function has to be genuinely good — a five-line Murmur finalizer stands between you and a 5× slower aggregation.

What to go do about it: if you ever profile a GROUP BY and see the cost dominated by cache misses on a huge hash table, remember that the Robin Hood grower exists precisely for that case: it halves the footprint at large sizes.

PostgreSQL Allocates Memory Fast

Jay Hubbert — Sat, 19 Sep 2026 09:00:00 +0000

Every C program that wants to be fast needs to be fast at allocating memory. PostgreSQL allocates memory in the tightest loop imaginable: a hash join probes its hash table once per outer row, an aggregate re-evaluates its transition function once per input row, and a sort copies tuples into a buffer. In a query that touches ten million rows, that is tens of millions of allocations, and every one of them has to be cheap.

PostgreSQL’s answer is palloc(), and palloc() is emphatically not malloc(). It does not call malloc() per allocation. It does not keep a general-purpose heap. It quantises every request to a power of two, keeps one free list per size class, carves the tail of an exhausted block into free chunks rather than throwing it away, and recycles whole memory context objects so that a per-tuple context never has to go back to malloc() at all.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

The idea: lifetimes, not ownership

The foundational idea is not an allocator technique at all. It is that memory has a lifetime, and the lifetime should decide who frees it.

PostgreSQL’s src/backend/utils/mmgr/README states it plainly:

The main advantage of memory contexts over plain use of malloc/free is that the entire contents of a memory context can be freed easily, without having to request freeing of each individual chunk within it. This is both faster and more reliable than per-chunk bookkeeping.

So instead of tracking every individual allocation, PostgreSQL builds a tree of memory contexts, each with its own lifetime, and frees a whole subtree in one operation. A per-tuple context is reset after every tuple. A per-query context is reset after every query. A per-transaction context is reset after every transaction. And the allocator inside each context is free to make whatever local trade-offs it likes, because the context boundary, not the individual free, is what actually matters for memory return.

The tree is a plain first-child/next-sibling structure in src/include/nodes/memnodes.h:

 c
 
 
typedef struct MemoryContextData
{
	pg_node_attr(abstract)		/* there are no nodes of this type */

	NodeTag		type;			/* identifies exact kind of context */
	/* these two fields are placed here to minimize alignment wastage: */
	bool		isReset;		/* T = no space allocated since last reset */
	bool		allowInCritSection; /* allow palloc in critical section */
	Size		mem_allocated;	/* track memory allocated for this context */
	const MemoryContextMethods *methods;	/* virtual function table */
	MemoryContext parent;		/* NULL if no parent (toplevel context) */
	MemoryContext firstchild;	/* head of linked list of children */
	MemoryContext prevchild;	/* previous child of same parent */
	MemoryContext nextchild;	/* next child of same parent */
	const char *name;			/* context name */
	const char *ident;			/* context ID if any */
	MemoryContextCallback *reset_cbs;	/* list of reset/delete callbacks */
} MemoryContextData;

Note the methods pointer: a memory context is a virtual object. The same tree can be rooted at an AllocSetContext, a SlabContext, a BumpContext or a GenerationContext, and each has completely different allocation behaviour while presenting the same API.

One indirection, but a deliberate one

palloc() does not call malloc(). It calls through the context’s method table. From src/backend/utils/mmgr/mcxt.c:

 c
 
 
/*
 * Call the given function in the MemoryContextMethods for the memory context
 * type that 'pointer' belongs to.
 */
#define MCXT_METHOD(pointer, method) \
	mcxt_methods[GetMemoryChunkMethodID(pointer)].method

And palloc() itself:

 c
 
 
void *
palloc(Size size)
{
	/* duplicates MemoryContextAlloc to avoid increased overhead */
	void	 *ret;
	MemoryContext context = CurrentMemoryContext;

	Assert(MemoryContextIsValid(context));
	AssertNotInCriticalSection(context);

	context->isReset = false;

	/*
	 * For efficiency reasons, we purposefully offload the handling of
	 * allocation failures to the MemoryContextMethods implementation as this
	 * allows these checks to be performed only when an actual malloc needs to
	 * be done to request more memory from the OS. Additionally, not having
	 * to execute any instructions after this call allows the compiler to use
	 * the sibling call optimization. If you're considering adding code after
	 * this call, consider making it the responsibility of the 'alloc'
	 * function instead.
	 */
	ret = context->methods->alloc(context, size, 0);
	/* We expect OOM to be handled by the alloc function */
	Assert(ret != NULL);
	VALGRIND_MEMPOOL_ALLOC(context, ret, size);

	return ret;
}

Three things in that comment deserve attention.

First, the out-of-memory check lives in the allocator, not here. The generic wrapper does not test for NULL and call ereport(ERROR). It cannot: an allocation that is satisfied from an existing free list never touches the operating system, so there is no failure to report. Only when the allocator actually has to malloc() more memory from the OS can it fail — and that is where the check belongs:

 c
 
 
void *
MemoryContextAllocationFailure(MemoryContext context, Size size, int flags)
{
	if ((flags & MCXT_ALLOC_NO_OOM) == 0)
	{
		if (TopMemoryContext)
			MemoryContextStats(TopMemoryContext);
		ereport(ERROR,
				(errcode(ERRCODE_OUT_OF_MEMORY),
				 errmsg("out of memory"),
				 errdetail("Failed on request of size %zu in memory context \"%s\".",
						 size, context->name)));
	}
	return NULL;
}

Second, the tail call. Because the comment says “not having to execute any instructions after this call”, palloc() compiles to something close to a jump: load context->methods->alloc, load size, call it, return the result. The compiler is free to turn the final return ret; into a tail jump into AllocSetAlloc(), avoiding a stack frame in the wrapper entirely. Add an instruction after the call and that optimisation evaporates. This is why the comment ends with a warning to future contributors: “If you’re considering adding code after this call, consider making it the responsibility of the ‘alloc’ function instead.”

Third, palloc0 does not call palloc. It is a copy of the same function with the MemSetAligned bolted on, for the same reason — /* duplicates MemoryContextAlloc to avoid increased overhead */. The extra call would cost a stack frame on one of the hottest paths in the server.

The MemoryContextAlloc() variant, which takes an explicit context rather than using CurrentMemoryContext, carries the identical comment and the identical shape:

 c
 
 
void *
MemoryContextAlloc(MemoryContext context, Size size)
{
	void	 *ret;

	Assert(MemoryContextIsValid(context));
	AssertNotInCriticalSection(context);

	context->isReset = false;

	/*
	 * For efficiency reasons, we purposefully offload the handling of
	 * allocation failures to the MemoryContextMethods implementation as this
	 * allows these checks to be performed only when an actual malloc needs to
	 * be done to request more memory from the OS. Additionally, not having
	 * to execute any instructions after this call allows the compiler to use
	 * the sibling call optimization. If you're considering adding code after
	 * this call, consider making it the responsibility of the 'alloc'
	 * function instead.
	 */
	ret = context->methods->alloc(context, size, 0);

	VALGRIND_MEMPOOL_ALLOC(context, ret, size);

	return ret;
}

How `pfree` knows where to send the pointer

The method-table dispatch works in both directions. pfree() does not take a context argument — a chunk can be freed from any context — so it has to find the right methods from the pointer alone. The trick is an 8-byte header stored immediately before every chunk.

From src/include/utils/memutils_memorychunk.h:

 c
 
 
typedef struct MemoryChunk
{
#ifdef MEMORY_CONTEXT_CHECKING
	Size		requested_size;
#endif

	/* bitfield for storing details about the chunk */
	uint64		hdrmask;		/* must be last */
} MemoryChunk;

In a normal build MemoryChunk is exactly 8 bytes. The file’s header comment describes how those 64 bits are carved up:

The MemoryChunk contains a uint64 field named ‘hdrmask’. This field is used to encode 4 separate pieces of information. Starting with the least significant bits of ‘hdrmask’, the bit space is reserved as follows:

4-bits to indicate the MemoryContextMethodID as defined by MEMORY_CONTEXT_METHODID_MASK

1-bit to denote an “external” chunk (see below)

30-bits reserved for the MemoryContext to use for anything it requires. Most MemoryContexts likely want to store the size of the chunk here.

30-bits for the number of bytes that must be subtracted from the chunk to obtain the address of the block that the chunk is stored on.

And the trick that makes it add up to 64 bits:

If you’re paying close attention, you’ll notice this adds up to 65 bits rather than 64 bits. This is because the highest-order bit of #3 is the same bit as the lowest-order bit of #4. We can do this as we insist that the chunk and block pointers are both MAXALIGNed, therefore the relative offset between those will always be a MAXALIGNed value which means the lowest order bit is always 0.

So one 64-bit word carries the method ID (4 bits, to dispatch pfree), an “external” flag, 30 bits of context-specific payload, and a 30-bit offset back to the owning block. Reading it back:

 c
 
 
static inline void
MemoryChunkSetHdrMask(MemoryChunk *chunk, void *block,
					 Size value, MemoryContextMethodID methodid)
{
	Size		blockoffset = (char *) chunk - (char *) block;

	Assert((char *) chunk >= (char *) block);
	Assert((blockoffset & MEMORYCHUNK_BLOCKOFFSET_MASK) == blockoffset);
	Assert(value <= MEMORYCHUNK_MAX_VALUE);
	Assert((int) methodid <= MEMORY_CONTEXT_METHODID_MASK);

	chunk->hdrmask = (((uint64) blockoffset) << MEMORYCHUNK_BLOCKOFFSET_BASEBIT) |
		(((uint64) value) << MEMORYCHUNK_VALUE_BASEBIT) |
		methodid;
}

GetMemoryChunkMethodID() in mcxt.c reads the low 4 bits, and pfree() becomes a one-line indirect call:

 c
 
 
void
pfree(void *pointer)
{
#ifdef USE_VALGRIND
	MemoryContext context = GetMemoryChunkContext(pointer);
#endif

	MCXT_METHOD(pointer, free_p) (pointer);

	VALGRIND_MEMPOOL_FREE(context, pointer);
}

The 8-byte header is the price of admission. Every allocation pays it. For an 8-byte allocation, that means the usable chunk is 8 bytes and the header is 8 bytes — a 100% overhead. This is precisely why the other allocator types exist.

The AllocSet: power-of-two size classes

aset.c is the default allocator, and it is where all the interesting tricks live. The file’s own header comment describes the design:

 c
 
 
/*--------------------
 * Chunk freelist k holds chunks of size 1 << (k + ALLOC_MINBITS),
 * for k = 0 .. ALLOCSET_NUM_FREELISTS-1.
 *
 * Note that all chunks in the freelists have power-of-2 sizes. This
 * improves recyclability: we may waste some space, but the wasted space
 * should stay pretty constant as requests are made and released.
 *
 * A request too large for the last freelist is handled by allocating a
 * dedicated block from malloc(). The block still has a block header and
 * chunk header, but when the chunk is freed we'll return the whole block
 * to malloc(), not put it on our freelists.
 *
 * CAUTION: ALLOC_MINBITS must be large enough so that
 * 1<<ALLOC_MINBITS is at least MAXALIGN,
 * or we may fail to align the smallest chunks adequately.
 * 8-byte alignment is enough on all currently known machines. This 8-byte
 * minimum also allows us to store a pointer to the next freelist item within
 * the chunk of memory itself.
 *
 * With the current parameters, request sizes up to 8K are treated as chunks,
 * larger requests go into dedicated blocks. Change ALLOCSET_NUM_FREELISTS
 * to adjust the boundary point; and adjust ALLOCSET_SEPARATE_THRESHOLD in
 * memutils.h to agree.
 *--------------------
 */

#define ALLOC_MINBITS		3	/* smallest chunk size is 8 bytes */
#define ALLOCSET_NUM_FREELISTS	11
#define ALLOC_CHUNK_LIMIT	(1 << (ALLOCSET_NUM_FREELISTS-1+ALLOC_MINBITS))
/* Size of largest chunk that we use a fixed size for */
#define ALLOC_CHUNK_FRACTION	4
/* We allow chunks to be at most 1/4 of maxBlockSize (less overhead) */

Eleven free lists. The chunk sizes are:

Free list index	Chunk size	Notes
0	8 bytes	`1 << 3`; also the minimum chunk
1	16 bytes
2	32 bytes
3	64 bytes
4	128 bytes
5	256 bytes
6	512 bytes
7	1 kB
8	2 kB
9	4 kB
10	8 kB	`ALLOC_CHUNK_LIMIT`
—	> 8 kB	dedicated `malloc()` block

A request of 100 bytes lands in free list 4 and gets a 128-byte chunk. A request of 129 bytes lands in free list 5 and gets 256 bytes. The waste is bounded by a factor of two and, crucially, constant — which is the whole point. A 100-byte allocation freed and re-requested always gets the same 128-byte chunk, forever, because it goes back on free list 4.

The context itself:

 c
 
 
typedef struct AllocSetContext
{
	MemoryContextData header;	/* Standard memory-context fields */
	/* Info about storage allocated in this context: */
	AllocBlock	blocks;			/* head of list of blocks in this set */
	MemoryChunk *freelist[ALLOCSET_NUM_FREELISTS];	/* free chunk lists */
	/* Allocation parameters for this context: */
	uint32		initBlockSize;	/* initial block size */
	uint32		maxBlockSize;	/* maximum block size */
	uint32		nextBlockSize;	/* next block size to allocate */
	uint32		allocChunkLimit;	/* effective chunk size limit */
	/* freelist this context could be put in, or -1 if not a candidate: */
	int			freeListIndex;	/* index in context_freelists[], or -1 */
} AllocSetContext;

Eleven pointers, plus four sizing parameters, plus one integer for the context recycling pool. That is the entire state of an AllocSet.

And the block:

 c
 
 
typedef struct AllocBlockData
{
	AllocSet	aset;			/* aset that owns this block */
	AllocBlock	prev;			/* prev block in aset's blocks list, if any */
	AllocBlock	next;			/* next block in aset's blocks list, if any */
	char	 *freeptr;		/* start of free space in this block */
	char	 *endptr;			/* end of space in this block */
}			AllocBlockData;

Blocks are obtained from malloc() and carved into chunks. freeptr walks forward as the active block fills up.

Computing the free list index

Given a request size, the allocator has to decide which of the eleven free lists to use. That is AllocSetFreeIndex():

 c
 
 
static inline int
AllocSetFreeIndex(Size size)
{
	int			idx;

	if (size > (1 << ALLOC_MINBITS))
	{
		/*----------
		 * At this point we must compute ceil(log2(size >> ALLOC_MINBITS)).
		 * This is the same as
		 *		pg_leftmost_one_pos32((size - 1) >> ALLOC_MINBITS) + 1
		 * or equivalently
		 *		pg_leftmost_one_pos32(size - 1) - ALLOC_MINBITS + 1
		 *
		 * However, for platforms without intrinsic support, we duplicate the
		 * logic here, allowing an additional optimization. It's reasonable
		 * to assume that ALLOC_CHUNK_LIMIT fits in 16 bits, so we can unroll
		 * the byte-at-a-time loop in pg_leftmost_one_pos32 and just handle
		 * the last two bytes.
		 *
		 * Yes, this function is enough of a hot-spot to make it worth this
		 * much trouble.
		 *----------
		 */
#ifdef HAVE_BITSCAN_REVERSE
		idx = pg_leftmost_one_pos32((uint32) size - 1) - ALLOC_MINBITS + 1;
#else
		uint32		t,
					tsize;

		/* Statically assert that we only have a 16-bit input value. */
		StaticAssertDecl(ALLOC_CHUNK_LIMIT < (1 << 16),
						 "ALLOC_CHUNK_LIMIT must be less than 64kB");

		tsize = size - 1;
		t = tsize >> 8;
		idx = t ? pg_leftmost_one_pos[t] + 8 : pg_leftmost_one_pos[tsize];
		idx -= ALLOC_MINBITS - 1;
#endif

		Assert(idx < ALLOCSET_NUM_FREELISTS);
	}
	else
		idx = 0;

	return idx;
}

“Yes, this function is enough of a hot-spot to make it worth this much trouble.” That is a real PostgreSQL comment, and it explains the shape of the code. pg_leftmost_one_pos32() is the count-leading-zeros primitive:

 c
 
 
static inline int
pg_leftmost_one_pos32(uint32 word)
{
#ifdef HAVE__BUILTIN_CLZ
	Assert(word != 0);

	return 31 - __builtin_clz(word);

On x86 this compiles to a single bsr instruction (or lzcnt). One instruction to convert “how big is this request?” into “which free list do I use?”. The portable fallback uses a 256-entry byte table and only looks at the top two bytes of the value, which is why the code carries the StaticAssertDecl about ALLOC_CHUNK_LIMIT fitting in 16 bits.

The index is stored in the chunk header

Here is the part that makes pfree() cheap. When a chunk is freed, the allocator has to know which free list to put it back on. It could recompute it from the chunk’s size — but it doesn’t need to, because the chunk header has 30 spare bits.

Recall the MemoryChunk layout: bits 5–34 are a 30-bit “value” field that the context may use for whatever it likes. aset.c uses it to store the free list index:

 c
 
 
	/* store the free list index in the value field */
	MemoryChunkSetHdrMask(chunk, block, fidx, MCTX_ASET_ID);

and AllocSetFree() reads it straight back out:

 c
 
 
		fidx = MemoryChunkGetValue(chunk);
		Assert(FreeListIdxIsValid(fidx));
		link = GetFreeListLink(chunk);

No clz, no lookup table, no search. One shift and one mask. The comment in aset.c makes the rationale explicit:

Note that all chunks in the freelists have power-of-2 sizes. This improves recyclability: we may waste some space, but the wasted space should stay pretty constant as requests are made and released.

The free list index is the size class, so storing the index is exactly equivalent to storing the size — but it is 4 bits instead of 32, and it needs no arithmetic to recover.

The free list link lives inside the chunk

A free list needs a next pointer per chunk. Where does that go? PostgreSQL puts it in the chunk’s own user data:

 c
 
 
/*
 * Obtain a AllocFreeListLink for the given chunk. Allocation sizes are
 * always at least sizeof(AllocFreeListLink), so we reuse the pointer's memory
 * itself to store the freelist link.
 */
#define GetFreeListLink(chkptr) \
	(AllocFreeListLink *) ((char *) (chkptr) + ALLOC_CHUNKHDRSZ)

This is exactly why ALLOC_MINBITS is 3 — the smallest chunk is 8 bytes, which is exactly enough to hold a pointer. When the chunk is live, that space is the caller’s data. When the chunk is free, it is the list link. No separate array of freelist nodes, no extra allocation.

The allocation fast path

AllocSetAlloc() is the whole story for a typical small allocation:

 c
 
 
void *
AllocSetAlloc(MemoryContext context, Size size, int flags)
{
	AllocSet	set = (AllocSet) context;
	AllocBlock	block;
	MemoryChunk *chunk;
	int			fidx;
	Size		chunk_size;
	Size		availspace;

	Assert(AllocSetIsValid(set));

	/* due to the keeper block set->blocks should never be NULL */
	Assert(set->blocks != NULL);

	/*
	 * If requested size exceeds maximum for chunks we hand the request off to
	 * AllocSetAllocLarge().
	 */
	if (size > set->allocChunkLimit)
		return AllocSetAllocLarge(context, size, flags);

	/*
	 * Request is small enough to be treated as a chunk. Look in the
	 * corresponding free list to see if there is a free chunk we could reuse.
	 * If one is found, remove it from the free list, make it again a member
	 * of the alloc set and return its data address.
	 *
	 * Note that we don't attempt to ensure there's space for the sentinel
	 * byte here. We expect a large proportion of allocations to be for sizes
	 * which are already a power of 2. If we were to always make space for a
	 * sentinel byte in MEMORY_CONTEXT_CHECKING builds, then we'd end up
	 * doubling the memory requirements for such allocations.
	 */
	fidx = AllocSetFreeIndex(size);
	chunk = set->freelist[fidx];
	if (chunk != NULL)
	{
		AllocFreeListLink *link = GetFreeListLink(chunk);

		/* Allow access to the chunk header. */
		VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOC_CHUNKHDRSZ);

		Assert(fidx == MemoryChunkGetValue(chunk));

		/* pop this chunk off the freelist */
		VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreeListLink));
		set->freelist[fidx] = link->next;
		VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreeListLink));
		...
		return MemoryChunkGetPointer(chunk);
	}

	/*
	 * Choose the actual chunk size to allocate.
	 */
	chunk_size = GetChunkSizeFromFreeListIdx(fidx);
	Assert(chunk_size >= size);

	block = set->blocks;
	availspace = block->endptr - block->freeptr;

	/*
	 * If there is enough room in the active allocation block, we will put the
	 * chunk into that block. Else must start a new one.
	 */
	if (unlikely(availspace < (chunk_size + ALLOC_CHUNKHDRSZ)))
		return AllocSetAllocFromNewBlock(context, size, flags, fidx);

	/* There's enough space on the current block, so allocate from that */
	return AllocSetAllocChunkFromBlock(context, block, size, chunk_size, fidx);
}

The order of operations is the whole design:

Too big? Hand off to AllocSetAllocLarge(), which does a dedicated malloc().
Free list non-empty? Pop it. This is the common case in a long-running loop, and it costs: one clz, one array load, one pointer swap. No malloc, no free, no copying.
Room in the active block? Bump freeptr and write the header.
Otherwise carve the tail (below) and start a new block.

Notice the Assert(fidx == MemoryChunkGetValue(chunk)); — the assertion that the stored index really does match the list the chunk came off. That is the invariant that makes pfree O(1) with respect to size classes.

Carving the tail of a dying block

Step 4 is where the interesting work happens. When the active block cannot satisfy the request, PostgreSQL does not throw away the remaining space in it. It carves the remainder into free chunks, one per size class, and puts them on the free lists.

From AllocSetAllocFromNewBlock():

 c
 
 
	/*
	 * The existing active (top) block does not have enough room for the
	 * requested allocation, but it might still have a useful amount of space
	 * in it. Once we push it down in the block list, we'll never try to
	 * allocate more space from it. So, before we do that, carve up its free
	 * space into chunks that we can put on the set's freelists.
	 *
	 * Because we can only get here when there's less than ALLOC_CHUNK_LIMIT
	 * left in the block, this loop cannot iterate more than
	 * ALLOCSET_NUM_FREELISTS-1 times.
	 */
	while (availspace >= ((1 << ALLOC_MINBITS) + ALLOC_CHUNKHDRSZ))
	{
		AllocFreeListLink *link;
		MemoryChunk *chunk;
		Size		availchunk = availspace - ALLOC_CHUNKHDRSZ;
		int			a_fidx = AllocSetFreeIndex(availchunk);

		/*
		 * In most cases, we'll get back the index of the next larger freelist
		 * than the one we need to put this chunk on. The exception is when
		 * availchunk is exactly a power of 2.
		 */
		if (availchunk != GetChunkSizeFromFreeListIdx(a_fidx))
		{
			a_fidx--;
			Assert(a_fidx >= 0);
			availchunk = GetChunkSizeFromFreeListIdx(a_fidx);
		}

		chunk = (MemoryChunk *) (block->freeptr);

		/* Prepare to initialize the chunk header. */
		VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
		block->freeptr += (availchunk + ALLOC_CHUNKHDRSZ);
		availspace -= (availchunk + ALLOC_CHUNKHDRSZ);

		/* store the freelist index in the value field */
		MemoryChunkSetHdrMask(chunk, block, a_fidx, MCTX_ASET_ID);
#ifdef MEMORY_CONTEXT_CHECKING
		chunk->requested_size = InvalidAllocSize;	/* mark it free */
#endif
		/* push this chunk onto the free list */
		link = GetFreeListLink(chunk);

		VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreeListLink));
		link->next = set->freelist[a_fidx];
		VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreeListLink));

		set->freelist[a_fidx] = chunk;
	}

Read that loop carefully. Suppose the block has 5000 bytes left. It takes the largest chunk that fits into 5000 bytes: 4096 bytes, on free list 9, leaving 904. Then 512 bytes (free list 6), leaving 392. Then 256 (free list 5), leaving 136. Then 64 (free list 3), leaving 72. Then 32 (free list 2), leaving 0. Loop ends.

So one dying block yields five reusable chunks in five different size classes, and nothing at all is wasted — not even the 8-byte chunk headers, which are counted against the available space as the loop runs. Compare that to the naive behaviour of throwing the tail away: those 5000 bytes would have been returned to malloc() (or worse, kept and never reused), and the next 4096-byte request would have had to malloc() again.

Then the block size doubles:

 c
 
 
	/*
	 * The first such block has size initBlockSize, and we double the space in
	 * each succeeding block, but not more than maxBlockSize.
	 */
	blksize = set->nextBlockSize;
	set->nextBlockSize <<= 1;
	if (set->nextBlockSize > set->maxBlockSize)
		set->nextBlockSize = set->maxBlockSize;

So a context starts with an 8 kB block (ALLOCSET_DEFAULT_INITSIZE), then 16 kB, then 32 kB, up to maxBlockSize (8 MB by default). The rationale is in the header comment:

The first block allocated for an allocset has size initBlockSize. Each time we have to allocate another block, we double the block size (if possible, and without exceeding maxBlockSize), so as to reduce the bookkeeping load on malloc().

Note also the retry loop for huge blocks:

 c
 
 
	/*
	 * We could be asking for pretty big blocks here, so cope if malloc fails.
	 * But give up if there's less than 1 MB or so available...
	 */
	while (block == NULL && blksize > 1024 * 1024)
	{
		blksize >>= 1;
		if (blksize < required_size)
			break;
		block = (AllocBlock) malloc(blksize);
	}

Asking for 64 MB and getting a failure is survivable; asking for 64 MB when the machine has 40 MB free is not, so PostgreSQL halves the request and tries again down to 1 MB.

Chunk limits are per-context, not global

The 8 kB ALLOC_CHUNK_LIMIT is the global maximum, but each context computes its own effective limit:

 c
 
 
static inline uint32
AllocSetComputeChunkLimit(Size maxBlockSize)
{
	uint32		allocChunkLimit;

	/*
	 * Compute the allocation chunk size limit for this context. It can't be
	 * more than ALLOC_CHUNK_LIMIT because of the fixed number of freelists.
	 * If maxBlockSize is small then requests exceeding the maxBlockSize, or
	 * even a significant fraction of it, should be treated as large chunks
	 * too. For the typical case of maxBlockSize a power of 2, the chunk size
	 * limit will be at most 1/8th maxBlockSize, so that given a stream of
	 * requests that are all the maximum chunk size we will waste at most
	 * 1/8th of the allocated space.
	 ...
	 */
	allocChunkLimit = ALLOC_CHUNK_LIMIT;
	while ((Size) (allocChunkLimit + ALLOC_CHUNKHDRSZ) >
		 (Size) ((maxBlockSize - ALLOC_BLOCKHDRSZ) / ALLOC_CHUNK_FRACTION))
		allocChunkLimit >>= 1;

	return allocChunkLimit;
}

A context with maxBlockSize of 8 kB (the ALLOCSET_SMALL_SIZES preset) gets a chunk limit of 1 kB or so, so that a stream of maximum-size requests cannot waste more than a quarter of the block. Small contexts therefore do not get 8 kB chunks they can never fill.

`repalloc`: sometimes it is free

Growing a chunk usually means copying. But not always. AllocSetRealloc() first checks whether the existing chunk is already big enough:

 c
 
 
	/*
	 * Chunk sizes are aligned to power of 2 in AllocSetAlloc(). Maybe the
	 * allocated area already is >= the new size. (In particular, we will
	 * fall out here if the requested size is a decrease.)
	 */
	if (oldchksize >= size)
	{
		...
		/* Disallow access to the chunk header. */
		VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);

		return pointer;
	}

oldchksize is GetChunkSizeFromFreeListIdx(fidx) — the chunk size, not the requested size. So a 40-byte allocation lives in a 64-byte chunk; repallocing it to 60 bytes finds 64 >= 60 and returns the same pointer, with no copy at all. The extra 24 bytes were already there, reserved by the power-of-two quantisation.

This is a free side effect of the quantisation scheme. The waste is already paid; handing it back to the caller costs nothing.

Growing past the chunk size does require a copy, and the comment is candid about why the obvious optimisation was rejected:

 c
 
 
		/*
		 * Enlarge-a-small-chunk case. We just do this by brute force, ie,
		 * allocate a new chunk and copy the data. Since we know the existing
		 * data isn't huge, this won't involve any great memcpy expense, so
		 * it's not worth being smarter. (At one time we tried to avoid
		 * memcpy when it was possible to enlarge the chunk in-place, but that
		 * turns out to misbehave unpleasantly for repeated cycles of
		 * palloc/repalloc/pfree: the eventually freed chunks go into the
		 * wrong freelist for the next initial palloc request, and so we leak
		 * memory indefinitely. See pgsql-hackers archives for 2007-08-11.)
		 */

“the eventually freed chunks go into the wrong freelist” — enlarging in place would change the chunk’s size class, and when the chunk is later freed it would go onto a different free list than the one it came off, which permanently removes a chunk from the list it was supposed to return to. A slow leak, per cycle, forever. Good lesson.

Internal fragmentation: the trade-off

Quantising to powers of two is not free. A 100-byte request occupies 128 bytes. Here is the full picture, ignoring the 8-byte chunk header:

Request size	Chunk size	Wasted bytes	Overhead
1	8	7	700%
9	16	7	78%
17	32	15	88%
33	64	31	94%
65	128	63	97%
129	256	127	98%
257	512	255	99%
513	1 kB	511	100%
1025	2 kB	1023	100%
2049	4 kB	2047	100%
4097	8 kB	4095	100%
8193	dedicated block	0	header only

The “overhead” column is worst-case (a request one byte above a power of two). Averaged over a uniformly distributed range of request sizes, a request wastes roughly a third of its chunk — and, crucially, the waste is predictable, which matters more than the average, because predictable overhead can be budgeted for.

The counter-argument is in the header comment:

Note that all chunks in the freelists have power-of-2 sizes. This improves recyclability: we may waste some space, but the wasted space should stay pretty constant as requests are made and released.

A general-purpose allocator that returns exactly the requested size wastes nothing on paper, but then a 100-byte free chunk can only satisfy a 100-byte request, and the space in a 700-byte gap can satisfy nothing. Quantisation guarantees that free space is always usable, at the cost of always being a bit larger than needed.

The other side of the ledger: requests above 8 kB bypass the whole scheme and get their own malloc() block, so large allocations (index tuples, TOAST chunks, hash table buckets) pay no quantisation waste at all.

Recycling whole contexts

Now the second half of the story. Contexts themselves are allocated with malloc() and freed with free(). If a query creates a per-tuple context and destroys it per tuple, that is a malloc/free pair per tuple — exactly the thrashing the context system was supposed to avoid.

PostgreSQL’s answer is a pool of dead contexts. The design note is in aset.c:

 c
 
 
/*
 * Rather than repeatedly creating and deleting memory contexts, we keep some
 * freed contexts in freelists so that we can hand them out again with little
 * work. Before putting a context in a freelist, we reset it so that it has
 * only its initial malloc chunk and no others. To be a candidate for a
 * freelist, a context must have the same minContextSize/initBlockSize as
 * other contexts in the list; but its maxBlockSize is irrelevant since that
 * doesn't affect the size of the initial chunk.
 *
 * We currently provide one freelist for ALLOCSET_DEFAULT_SIZES contexts
 * and one for ALLOCSET_SMALL_SIZES contexts; the latter works for
 * ALLOCSET_START_SMALL_SIZES too, since only the maxBlockSize differs.
 *
 * Ordinarily, we re-use freelist contexts in last-in-first-out order, in
 * hopes of improving locality of reference. But if there get to be too
 * many contexts in the list, we'd prefer to drop the most-recently-created
 * contexts in hopes of keeping the process memory map compact.
 * We approximate that by simply deleting all existing entries when the list
 * overflows, on the assumption that queries that allocate a lot of contexts
 * will probably free them in more or less reverse order of allocation.
 *
 * Contexts in a freelist are chained via their nextchild pointers.
 */
#define MAX_FREE_CONTEXTS 100	/* arbitrary limit on freelist length */

“Contexts in a freelist are chained via their nextchild pointers.” That is a wonderful piece of field-hacking: a deleted context is no longer in the context tree, so its nextchild field (which links it to its siblings) is dead weight. It is reused as the free-list link.

The pool itself:

 c
 
 
typedef struct AllocSetFreeList
{
	int			num_free;		/* current list length */
	AllocSetContext *first_free;	/* list header */
} AllocSetFreeList;

/* context_freelists[0] is for default params, [1] for small params */
static AllocSetFreeList context_freelists[2] =
{
	{
		0, NULL
	},
	{
		0, NULL
	}
};

Two pools: one for “ordinary” contexts and one for “small” contexts, matching the two size presets in memutils.h:

 c
 
 
#define ALLOCSET_DEFAULT_MINSIZE 0
#define ALLOCSET_DEFAULT_INITSIZE (8 * 1024)
#define ALLOCSET_DEFAULT_MAXSIZE (8 * 1024 * 1024)
#define ALLOCSET_DEFAULT_SIZES \
	ALLOCSET_DEFAULT_MINSIZE, ALLOCSET_DEFAULT_INITSIZE, ALLOCSET_DEFAULT_MAXSIZE

#define ALLOCSET_SMALL_MINSIZE	 0
#define ALLOCSET_SMALL_INITSIZE (1 * 1024)
#define ALLOCSET_SMALL_MAXSIZE (8 * 1024)
#define ALLOCSET_SMALL_SIZES \
	ALLOCSET_SMALL_MINSIZE, ALLOCSET_SMALL_INITSIZE, ALLOCSET_SMALL_MAXSIZE

Deleting a context puts it on the pool

AllocSetDelete() is where the recycling happens:

 c
 
 
	/*
	 * If the context is a candidate for a freelist, put it into that freelist
	 * instead of destroying it.
	 */
	if (set->freeListIndex >= 0)
	{
		AllocSetFreeList *freelist = &context_freelists[set->freeListIndex];

		/*
		 * Reset the context, if it needs it, so that we aren't hanging on to
		 * more than the initial malloc chunk.
		 */
		if (!context->isReset)
			MemoryContextResetOnly(context);

		/*
		 * If the freelist is full, just discard what's already in it. See
		 * comments with context_freelists[].
		 */
		if (freelist->num_free >= MAX_FREE_CONTEXTS)
		{
			while (freelist->first_free != NULL)
			{
				AllocSetContext *oldset = freelist->first_free;

				freelist->first_free = (AllocSetContext *) oldset->header.nextchild;
				freelist->num_free--;

				/* Destroy the context's vpool --- see notes below */
				VALGRIND_DESTROY_MEMPOOL(oldset);

				/* All that remains is to free the header/initial block */
				free(oldset);
			}
			Assert(freelist->num_free == 0);
		}

		/* Now add the just-deleted context to the freelist. */
		set->header.nextchild = (MemoryContext) freelist->first_free;
		freelist->first_free = set;
		freelist->num_free++;

		return;
	}

Push at the head → LIFO. The comment explains why: “in hopes of improving locality of reference”. The most recently freed context was also the most recently allocated, so it is the one whose memory is hottest in the cache.

The pool is capped at 100 contexts. When it overflows, the entire pool is thrown away rather than just the oldest entry — because, as the comment says, “queries that allocate a lot of contexts will probably free them in more or less reverse order of allocation”, so dropping everything keeps the process memory map compact.

Creating a context takes one from the pool

The other half is AllocSetContextCreateInternal():

 c
 
 
	/*
	 * Check whether the parameters match either available freelist. We do
	 * not need to demand a match of maxBlockSize.
	 */
	if (minContextSize == ALLOCSET_DEFAULT_MINSIZE &&
		initBlockSize == ALLOCSET_DEFAULT_INITSIZE)
		freeListIndex = 0;
	else if (minContextSize == ALLOCSET_SMALL_MINSIZE &&
			 initBlockSize == ALLOCSET_SMALL_INITSIZE)
		freeListIndex = 1;
	else
		freeListIndex = -1;

	/*
	 * If a suitable freelist entry exists, just recycle that context.
	 */
	if (freeListIndex >= 0)
	{
		AllocSetFreeList *freelist = &context_freelists[freeListIndex];

		if (freelist->first_free != NULL)
		{
			/* Remove entry from freelist */
			set = freelist->first_free;
			freelist->first_free = (AllocSet) set->header.nextchild;
			freelist->num_free--;

			/* nextBlockSize should have been reset in AllocSetReset() */
			Assert(set->nextBlockSize == set->initBlockSize);

			if (set->maxBlockSize != maxBlockSize)
			{
				/*
				 * Update its maxBlockSize and allocChunkLimit; everything
				 * else should be OK.
				 */
				set->maxBlockSize = maxBlockSize;
				set->allocChunkLimit = AllocSetComputeChunkLimit(maxBlockSize);
			}

			/* Reinitialize its header, installing correct name and parent */
			MemoryContextCreate((MemoryContext) set,
								T_AllocSetContext,
								MCTX_ASET_ID,
								parent,
								name);

			((MemoryContext) set)->mem_allocated =
				KeeperBlock(set)->endptr - ((char *) set);

			return (MemoryContext) set;
		}
	}

No malloc. The context object is reused verbatim, only its header is re-initialised and its maxBlockSize adjusted (the block size limit is the only parameter that does not affect the initial chunk’s size, so a context can be recycled across different maxBlockSize values).

The result: a per-tuple context costs zero system allocations once the pool is warm. Create, use, destroy, repeat, and the whole cycle is pointer arithmetic.

Reset versus delete

Two operations, and the difference matters:

MemoryContextReset() — free everything allocated in the context (and its children), but keep the context object itself. The context remains usable.
MemoryContextDelete() — free the context object too. The name is gone.

For AllocSet, the two differ in one crucial detail: the keeper block.

 c
 
 
/*
 * AllocSetReset
 *		Frees all memory which is allocated in the given set.
 *
 * Actually, this routine has some discretion about what to do.
 * It should mark all allocated chunks freed, but it need not necessarily
 * give back all the resources the set owns. Our actual implementation is
 * that we give back all but the "keeper" block (which we must keep, since
 * it shares a malloc chunk with the context header). In this way, we don't
 * thrash malloc() when a context is repeatedly reset after small allocations,
 * which is typical behavior for per-tuple contexts.
 */

The “keeper block” is the region of memory immediately following the AllocSetContext struct, in the same malloc() chunk:

 c
 
 
/* Obtain the keeper block for an allocation set */
#define KeeperBlock(set) \
	((AllocBlock) (((char *) set) + MAXALIGN(sizeof(AllocSetContext))))

/* Check if the block is the keeper block of the given allocation set */
#define IsKeeperBlock(set, block) ((block) == (KeeperBlock(set)))

So the very first malloc() a context makes contains both the context header and its first block. AllocSetReset() keeps that block, wipes its contents, and free()s everything else:

 c
 
 
	block = set->blocks;

	/* New blocks list will be just the keeper block */
	set->blocks = KeeperBlock(set);

	while (block != NULL)
	{
		AllocBlock	next = block->next;

		if (IsKeeperBlock(set, block))
		{
			/* Reset the block, but don't return it to malloc */
			char	 *datastart = ((char *) block) + ALLOC_BLOCKHDRSZ;
			...
			block->freeptr = datastart;
			block->prev = NULL;
			block->next = NULL;
		}
		else
		{
			/* Normal case, release the block */
			context->mem_allocated -= block->endptr - ((char *) block);
			...
			free(block);
		}
		block = next;
	}

So a per-tuple context that allocates 200 bytes and is reset per tuple does exactly one malloc() (at creation) and zero free()s (until it is deleted). Compare with malloc/free per tuple, which would be two system calls per tuple.

AllocSetDelete() is the version that gives everything back, and it is also the version that puts the context on the recycling pool.

The other allocators

aset.c is the general-purpose one, but PostgreSQL ships four context types, because different allocation patterns want different trade-offs. The method table in mcxt.c lists them all:

 c
 
 
static const MemoryContextMethods mcxt_methods[] = {
	/* aset.c */
	[MCTX_ASET_ID].alloc = AllocSetAlloc,
	...
	/* generation.c */
	[MCTX_GENERATION_ID].alloc = GenerationAlloc,
	...
	/* slab.c */
	[MCTX_SLAB_ID].alloc = SlabAlloc,
	...
	/* alignedalloc.c */
	[MCTX_ALIGNED_REDIRECT_ID].alloc = NULL,	/* not required */
	...
	/* bump.c */
	[MCTX_BUMP_ID].alloc = BumpAlloc,
	...

Bump: no chunk header at all

The bump.c header comment states the design goal:

Bump is a MemoryContext implementation designed for memory usages which require allocating a large number of chunks, none of which ever need to be pfree’d or realloc’d. Chunks allocated by this context have no chunk header and operations which ordinarily require looking at the chunk header cannot be performed. For example, pfree, realloc, GetMemoryChunkSpace and GetMemoryChunkContext are all not possible with bump allocated chunks. The only way to release memory allocated by this context type is to reset or delete the context.

And the size of the header is literally zero:

 c
 
 
/* No chunk header unless built with MEMORY_CONTEXT_CHECKING */
#ifdef MEMORY_CONTEXT_CHECKING
#define Bump_CHUNKHDRSZ	sizeof(MemoryChunk)
#else
#define Bump_CHUNKHDRSZ	0
#endif

No 8-byte header per allocation. The allocation is just a bump of a pointer:

 c
 
 
	/* point the freeptr beyond this chunk */
	block->freeptr += (Bump_CHUNKHDRSZ + chunk_size);
	Assert(block->freeptr <= block->endptr);

The cost is that the operations that need the header are refused outright:

 c
 
 
/*
 * BumpFree
 *		Unsupported.
 */
void
BumpFree(void *pointer)
{
	elog(ERROR, "%s is not supported by the bump memory allocator", "pfree");
}

Same for BumpRealloc, BumpGetChunkContext and BumpGetChunkSpace. The MEMORY_CONTEXT_CHECKING build adds the header back precisely so that these mistakes are caught rather than silently corrupting memory.

Where is Bump used? Wherever there are “a large number of short-lived chunks where performance matters” — hash aggregation, tuplesort, set operations, nodeSubplan, the TID store. The header comment explains the shape of the win:

Because bump allocated chunks don’t have a chunk header, it can fit more chunks on each block. This means we can do more with less memory and fewer cache lines.

Slab: one size, and the free list lives in the chunk

slab.c is for the opposite extreme: many objects of exactly the same size. The chunk size is fixed at context creation time:

 c
 
 
MemoryContext
SlabContextCreate(MemoryContext parent,
				 const char *name,
				 Size blockSize,
				 Size chunkSize)

and enforced at allocation time:

 c
 
 
	/*
	 * Make sure we only allow correct request size. This doubles as the
	 * MemoryContextCheckSize check.
	 */
	if (unlikely(size != slab->chunkSize))
		SlabAllocInvalidSize(context, size);

The free list is stored inside the free chunk, exactly as in aset.c — but with an important difference. In aset.c the chunk’s user data holds the link; in slab.c the link is written over the start of the chunk’s data area, and the chunk size is forced to be at least a pointer:

 c
 
 
	/*
	 * Ensure there's space to store the pointer to the next free chunk
	 * in the memory of the (otherwise) unused allocation.
	 */
	if (chunkSize < sizeof(MemoryChunk *))
		chunkSize = sizeof(MemoryChunk *);

 c
 
 
	/* push this chunk onto the head of the block's free list */
	*(MemoryChunk **) pointer = block->freehead;
	block->freehead = chunk;

The interesting policy in slab.c is “fullest block first”. The header comment explains the motivation:

Slab can also help reduce memory fragmentation in cases where longer-lived chunks remain stored on blocks while most of the other chunks have already been pfree’d. We give priority to putting new allocations into the “fullest” block. This help avoid having too many sparsely used blocks around and allows blocks to more easily become completely unused which allows them to be eventually free’d.

The implementation is a small array of lists, partitioned by how many free chunks a block has:

 c
 
 
/*
 * The number of partitions to divide the blocklist into based their number
 * of free chunks. There must be at least 2.
 */
#define SLAB_BLOCKLIST_COUNT 3

with a deliberately clever index computation:

 c
 
 
static inline int32
SlabBlocklistIndex(SlabContext *slab, int nfree)
{
	int32		index;
	int32		blocklist_shift = slab->blocklist_shift;

	Assert(nfree >= 0 && nfree <= slab->chunksPerBlock);

	/*
	 * Determine the blocklist index based on the number of free chunks. We
	 * must ensure that 0 free chunks is dedicated to index 0. Everything
	 * else must be >= 1 and < SLAB_BLOCKLIST_COUNT.
	 *
	 * To make this as efficient as possible, we exploit some two's complement
	 * arithmetic where we reverse the sign before bit shifting. This results
	 * in an nfree of 0 using index 0 and anything non-zero staying non-zero.
	 * This is exploiting 0 and -0 being the same in two's complement. When
	 * we're done, we just need to flip the sign back over again for a
	 * positive index.
	 */
	index = -((-nfree) >> blocklist_shift);

	if (nfree == 0)
		Assert(index == 0);
	else
		Assert(index >= 1 && index < SLAB_BLOCKLIST_COUNT);

	return index;
}

-((-nfree) >> shift) — for nfree == 0 this is -0 == 0, and for anything else it is a positive number. Two’s complement does the work.

The high-water-mark trick

slab.c has one more elegant trick. When a fresh block is allocated, all of its chunks are free — but writing out a free list for every chunk would mean touching the entire block at creation time. Instead, slab.c keeps a pointer to the next never-used chunk and a count of how many there are:

 c
 
 
typedef struct SlabBlock
{
	SlabContext *slab;			/* owning context */
	int32		nfree;			/* number of chunks on free + unused chunks */
	int32		nunused;		/* number of unused chunks */
	MemoryChunk *freehead;		/* pointer to the first free chunk */
	MemoryChunk *unused;		/* pointer to the next unused chunk */
	dlist_node	node;			/* doubly-linked list for blocklist[] */
} SlabBlock;

The header comment explains it:

When we allocate a new block, technically all chunks are free, however, to avoid having to write out the entire block to set the linked list for the free chunks for every chunk in the block, we instead store a pointer to the next “unused” chunk on the block and keep track of how many of these unused chunks there are. When a new block is malloc’d, all chunks are unused. The unused pointer starts with the first chunk on the block and as chunks are allocated, the unused pointer is incremented. As chunks are pfree’d, the unused pointer never goes backwards. The unused pointer can be thought of as a high watermark for the maximum number of chunks in the block which have been in use concurrently. When a chunk is pfree’d the chunk is put onto the head of the free list and the unused pointer is not changed. We only consume more unused chunks if we run out of free chunks on the free list. This method effectively gives priority to using previously used chunks over previously unused chunks, which should perform better due to CPU caching effects.

A high-water mark for concurrency. Freed chunks are reused before virgin space is touched, so the hot parts of the block stay hot in cache.

Generation: for FIFO lifetimes

The fourth allocator, generation.c, is for the case where chunks are freed roughly in the order they were allocated:

This memory context is based on the assumption that the chunks are freed roughly in the same order as they were allocated (FIFO), or in groups with similar lifespan (generations - hence the name of the context).

Instead of a global free list, each block tracks allocated and freed chunks, and an empty block is recycled for future allocations before it is handed back to malloc().

The temporary context

There is no src/backend/utils/mmgr/temp.c in this tree — the four allocator types are aset.c, slab.c, generation.c and bump.c, and the file listing in src/backend/utils/mmgr/ confirms it. The “temporary context” of PostgreSQL’s design is instead a policy, described in the README:

When creating a prepared statement, the parse and plan trees will be built in a temporary context that’s a child of MessageContext (so that it will go away automatically upon error). On success, the finished plan is copied to the prepared statement’s private context, and the temp context is released; this allows planner temporary space to be recovered before execution begins.

The word “temporary” here describes the lifetime, not the allocator type. It is an ordinary AllocSetContext created with AllocSetContextCreate() and destroyed with MemoryContextDelete() once the plan has been copied out. What makes it “temporary” is that it is a child of MessageContext and is destroyed — not reset — once the planner is finished with it.

The executor’s per-tuple contexts work the same way, and they are where the context recycling machinery pays for itself most visibly.

Caveats

Memory is not returned to the OS promptly. pfree() on a chunk smaller than 8 kB does not call free(); it pushes the chunk onto a free list and moves on. The memory stays in the process. Only a context reset or delete, or the release of a dedicated large block, actually hands memory back — and even then, whether the OS reclaims it is malloc’s business, not PostgreSQL’s. A backend that allocated 1 GB of short-lived chunks will keep that 1 GB resident until the contexts holding it are reset.

Fragmentation is real, but bounded. Within a context, the free lists keep chunks segregated by size class, so a freed 100-byte chunk cannot satisfy a 1000-byte request. That is deliberate (recyclability over density), but it means a context whose allocation size distribution changes over time will accumulate idle chunks in the wrong classes.

Reset is not delete. MemoryContextReset() keeps the context object and its keeper block; MemoryContextDelete() frees everything and (for an AllocSet with poolable parameters) puts the context object itself on the recycling pool. Choosing reset where you meant delete leaves the context alive; choosing delete where you meant reset churns the pool.

pfree on a bump chunk is an error, not a no-op. The bump allocator has no chunk header, so it cannot find the freelist to return the chunk to. BumpFree() raises elog(ERROR, "pfree is not supported by the bump memory allocator"). If you are using a bump context, you must let the whole context go.

repalloc on a slab chunk is only allowed at the same size. A slab context allocates one fixed size, and SlabRealloc() errors out for any other size:

 c
 
 
	/* can't do actual realloc with slab, but let's try to be gentle */
	if (size == slab->chunkSize)
		return pointer;

	elog(ERROR, "slab allocator does not support realloc()");

The 8-byte header is real overhead. For an AllocSet, every allocation pays 8 bytes of header plus up to 2× quantisation waste. That is why Slab, Bump and Generation exist, and why MemoryContextAllocAligned() has to allocate extra bytes to build a “redirection” chunk for the alignment.

What it all means in practice

palloc is a size-class allocator, not a heap. If you are porting C code that assumes malloc semantics — variable sizes, exact accounting, memory returned promptly — some of that will not hold.

The 8 kB threshold is a cliff, not a slope. Requests of 8192 bytes or less are quantised chunks in a shared block; requests of 8193 bytes or more get their own malloc(). That is why ALLOCSET_SEPARATE_THRESHOLD exists in memutils.h — “tuplesort/tuplestore need to know it” — so they can account for the difference.

repalloc is often free, and that is not a coincidence. The power-of-two quantisation already reserved the space. If you are writing code that grows a buffer, repalloc within the same size class costs nothing.

Free-list churn is the price of the free list. The pfree path reads the index out of the chunk header and pushes the chunk onto a list; it does not coalesce with neighbours and does not return memory. The payoff is that the next palloc of the same size class is a single array load and a pointer swap.

Contexts are the unit of memory lifetime, and recycling them is what makes per-tuple contexts viable. The difference between “allocate a per-tuple context with malloc/free” and “reuse one from the pool” is two system calls per tuple versus none.

Do not allocate in the wrong context. The CurrentMemoryContext pointer is the single biggest source of memory leaks in PostgreSQL extensions. The README is blunt about it: “Only in very circumscribed code should it ever point at a context having greater than transaction lifespan, since doing so risks permanent memory leaks.”

tmux Stores Grid Cells in Five Bytes

Jay Hubbert — Wed, 22 Jul 2026 08:00:00 +0000

TL;DR: tmux splits a grid cell into two representations: a fat 32-byte in-memory form and a packed 5-byte stored form. The union makes the choice free — the same five bytes are either attributes, colours and a character, or an offset into a side table for the rare cells that need RGB, wide characters or hyperlinks.

A terminal grid cell isn’t small. In the “in-memory” representation it holds a UTF-8 character (up to four bytes), a foreground colour, a background colour, an underline colour, a set of attribute bits, a hyperlink id and some flags. That’s comfortably 32 bytes per cell.

Multiply by 80x24 and you get 61 KB per screen. Multiply by 10,000 lines of scrollback and 200 panes, and it’s gigabytes.

tmux’s answer is a split representation: the common case is packed into five bytes, and the rare case lives in a side table.

The two representations

The in-memory form (tmux.h) is the “fat” one:

 c
 
 
/* Grid cell data. */
struct grid_cell {
	struct utf8_data	data;
	u_short			attr;
	u_char			flags;
	int			fg;
	int			bg;
	int			us;
	u_int			link;
};

struct utf8_data alone is six bytes, and the three int colours are twelve. So the fat cell is around 32 bytes.

The stored form is the packed one:

 c
 
 
/* Grid cell entry. */
struct grid_cell_entry {
	union {
		u_int		offset;
		struct {
			u_char	attr;
			u_char	fg;
			u_char	bg;
			u_char	data;
		} data;
	};
	u_char			flags;
} __packed;

The union is the trick. The same five bytes are either:

data.attr (8 bits of attributes), data.fg (8-bit colour), data.bg (8-bit colour), data.data (the character), plus flags — the packed form, or
offset — an index into the line’s array of grid_extd_entry — the extended form.

__packed matters: without it the compiler would pad the struct to eight bytes and the whole optimisation would evaporate.

The extended form:

 c
 
 
/* Grid extended cell entry. */
struct grid_extd_entry {
	utf8_char		data;
	u_short			attr;
	u_char			flags;
	int			fg;
	int			bg;
	int			us;
	u_int			link;
} __packed;

has room for everything: RGB colours, wide characters, more than eight attribute bits, hyperlinks.

Deciding which form to use

The decision is made by grid_need_extended_cell():

 c
 
 
/* Check if a cell should be an extended cell. */
static int
grid_need_extended_cell(const struct grid_cell_entry *gce,
 const struct grid_cell *gc)
{
	if (gce->flags & GRID_FLAG_EXTENDED)
		return (1);
	if (gc->attr > 0xff)
		return (1);
	if (gc->data.size > 1 || gc->data.width > 1)
		return (1);
	if ((gc->fg & (COLOUR_FLAG_RGB|COLOUR_FLAG_THEME)) ||
	 (gc->bg & (COLOUR_FLAG_RGB|COLOUR_FLAG_THEME)))
		return (1);
	if (gc->us != 8) /* only supports 256 or RGB */
		return (1);
	if (gc->link != 0)
		return (1);
	if (gc->flags & GRID_FLAG_TAB)
		return (1);
	return (0);
}

So a cell is extended if it has:

more than eight attribute bits,
a multi-byte or double-width character,
an RGB or theme colour,
an underline colour other than the default,
a hyperlink,
or is a tab.

Everything else — plain ASCII, 8-bit colours, standard attributes — is packed.

For a typical terminal session that’s nearly every cell. The 256-colour palette fits in eight bits, and the eight standard attribute bits (bold, underline, reverse, and so on) fit in eight bits.

Note the first line: once a cell is extended, it stays extended. That simplifies the code enormously — you never have to convert back — at the cost of not being able to compact a cell that has become simple again.

256 colours in eight bits

The colour packing is slightly more subtle than “eight bits of colour”:

 c
 
 
/* Store cell in entry. */
static void
grid_store_cell(struct grid_cell_entry *gce, const struct grid_cell *gc,
 u_char c)
{
	gce->flags = (gc->flags & ~GRID_FLAG_CLEARED);

	gce->data.fg = gc->fg & 0xff;
	if (gc->fg & COLOUR_FLAG_256)
		gce->flags |= GRID_FLAG_FG256;

	gce->data.bg = gc->bg & 0xff;
	if (gc->bg & COLOUR_FLAG_256)
		gce->flags |= GRID_FLAG_BG256;

	gce->data.attr = gc->attr;
	gce->data.data = c;
}

The colour value is 0-255 in the byte, and the flags record whether it came from the 256-colour palette or the 16-colour one, because “bright black” means different things in the two palettes. The RGB flag lives in COLOUR_FLAG_RGB, which is above 8 bits and therefore forces the extended form.

Lazy line growth

Lines aren’t allocated at full width up front. grid_expand_line() grows them in geometric steps:

 c
 
 
/* Expand line to fit to cell. */
static void
grid_expand_line(struct grid *gd, u_int py, u_int sx, u_int bg)
{
	struct grid_line	*gl;
	u_int			 xx;

	gl = &gd->linedata[py];
	if (sx <= gl->cellsize)
		return;

	if (sx < gd->sx / 4)
		sx = gd->sx / 4;
	else if (sx < gd->sx / 2)
		sx = gd->sx / 2;
	else if (gd->sx > sx)
		sx = gd->sx;

	gl->celldata = xreallocarray(gl->celldata, sx,
	 sizeof *gl->celldata);
	...
	gl->cellsize = sx;
}

The sizes are 1/4 width, 1/2 width, full width. So a program that prints one character on a fresh line allocates a quarter of a line, not a whole one, and a program that prints a few characters doesn’t repeatedly realloc.

The grid_line struct tracks cellused and cellsize separately, so the trailing cells of a mostly-empty line don’t need to be stored at all.

This was a deliberate change; the CHANGES file records:

“Create every line as zero length and only expand it as data is written, rather than creating at full size immediately.”

Not leaking extended cells

Extended cells are allocated from a per-line array. When a cell is overwritten with something simple, the extended entry it used would normally be leaked — the line’s extended array would grow forever as a program repaints the same line with different colours.

grid_clear_cell() reuses the existing slot:

 c
 
 
/* Copy default into a cell. */
static void
grid_clear_cell(struct grid *gd, u_int px, u_int py, u_int bg, int moved)
{
	struct grid_line	*gl = &gd->linedata[py];
	struct grid_cell_entry	*gce = &gl->celldata[px];
	struct grid_extd_entry	*gee;
	u_int			 old_offset = gce->offset;
	int			 had_extd = (gce->flags & GRID_FLAG_EXTENDED);

	memcpy(gce, &grid_cleared_entry, sizeof *gce);
	if (!moved && had_extd && old_offset < gl->extdsize) {
		gce->flags |= GRID_FLAG_EXTENDED;
		gce->offset = old_offset;
		gee = grid_extended_cell(gl, gce, &grid_cleared_cell);
		if (bg != 8)
			gee->bg = bg;
	} else if (bg != 8) {
		...

Note old_offset being saved and restored: the cell keeps its extended form and its slot in the extended array, it just gets reset to the cleared state. The commit message for this was:

“Reuse extended entry when clearing RGB cell, to prevent memory growth.”

And the extended array itself is compacted when the line shrinks:

 c
 
 
/* Free up unused extended cells. */
static void
grid_compact_line(struct grid_line *gl)
{
	int			 new_extdsize = 0;
	...
	for (px = 0; px < gl->cellsize; px++) {
		gce = &gl->celldata[px];
		if (gce->flags & GRID_FLAG_EXTENDED)
			new_extdsize++;
	}
	...
	idx = 0;
	for (px = 0; px < gl->cellsize; px++) {
		gce = &gl->celldata[px];
		if (gce->flags & GRID_FLAG_EXTENDED) {
			gee = &gl->extddata[gce->offset];
			memcpy(&new_extddata[idx], gee, sizeof *gee);
			gce->offset = idx++;
		}
	}

	free(gl->extddata);
	gl->extddata = new_extddata;
	gl->extdsize = new_extdsize;
}

This is a mark-and-sweep compaction: count the live extended cells, allocate a new array of exactly that size, copy them over, and fix up the offsets. The old array is freed. The cell’s offset changes, which is why this can’t be done while the grid is being read by anything else.

History trimming

Scrollback is the other place memory grows without bound. When the history hits the limit, tmux frees the oldest 10% rather than one line at a time:

 c
 
 
/*
 * Collect lines from the history if at the limit. Free the top (oldest) 10%
 * and shift up.
 */
void
grid_collect_history(struct grid *gd, int all)

and the shift itself is a memmove() of the line array:

 c
 
 
void
grid_trim_history(struct grid *gd, u_int ny)
{
	u_int	remaining;

	grid_free_lines(gd, 0, ny);
	remaining = gd->hsize + gd->sy - ny;
	memmove(&gd->linedata[0], &gd->linedata[ny],
	 remaining * (sizeof *gd->linedata));
	memset(&gd->linedata[remaining], 0, ny * (sizeof *gd->linedata));
}

Trimming one line at a time would be O(n) per line of output, which is O(n²) over a full history. Trimming 10% at a time is O(n) per 10% of the history.

The CHANGES file dates this to 2007:

“Rather than shifting up one line at a time once the history is full, shift by 10% of the history each time. This is faster.”

The allocation helpers

All of this depends on allocation being cheap and safe. xmalloc.c provides xmalloc, xcalloc, xreallocarray, xrecallocarray and xmemdup, all of which abort the process on failure rather than returning NULL:

 c
 
 
void *
xcalloc(size_t nmemb, size_t size)
{
	void *ptr;

	if (size == 0 || nmemb == 0)
		fatalx("xcalloc: zero size");
	if (SIZE_MAX / nmemb < size)
		fatalx("xcalloc: nmemb * size > SIZE_MAX");
	ptr = calloc(nmemb, size);
	if (ptr == NULL)
		fatal("xcalloc: allocating %zu bytes", size * nmemb);
	return (ptr);
}

The overflow check is the interesting part: nmemb * size is computed in calloc() too, but the check has to happen first, in a form that can’t itself overflow.

Because allocation never returns NULL, no hot path has to check for it, and the xreallocarray() variants make the “array of n things” case safe by default.

The numbers

For an 80x24 pane of plain text:

Representation	Bytes per cell	Total
Fat `grid_cell`	~32	61 KB
Packed `grid_cell_entry`	5	9.6 KB

With 10,000 lines of scrollback per pane and 20 panes, the difference is about 1 GB versus 160 MB.

Wrapping up

The general technique here — a packed common case and an out-of-line rare case — is the same one used by Python’s compact dicts, PostgreSQL’s varlena headers, and the Linux kernel’s struct file_operations.

The key insight is that the in-memory representation used during manipulation and the stored representation used for retention don’t have to be the same.

tmux gets to have both, because the fat form is only materialised one cell at a time, on the stack, while parsing and drawing. The grid — the thing that lives for hours and grows without bound — is the packed one.

CPython Reuses Objects with Free Lists: Recycling Dead Objects for Free

Jay Hubbert — Sat, 14 Mar 2026 09:00:00 +0000

TL;DR: CPython keeps small per-type LIFO caches of dead objects. When a list, tuple, dict, float, or int dies, it’s pushed onto its type’s freelist; the next request for that type pops a ready-made shell and skips the allocator entirely.

Allocation isn’t free, even with a good allocator. CPython’s hot paths create and destroy the same kinds of objects over and over — a list while building a comprehension, a tuple for a function call’s argument pack, a dict for a namespace, a float for an arithmetic result.

Rather than handing every one of those back to the general allocator and asking for a fresh block next time, CPython keeps small per-type caches called freelists and recycles the recently dead objects directly. Let’s trace that machinery through the 3.15 source.

The idea in one paragraph

A freelist is an intrusive LIFO stack of dead objects.

When an object of a recyclable type dies, CPython tries to push it onto its type’s freelist instead of freeing it. The next time that type is needed, CPython pops a ready-made shell, reinitializes its first word, and skips allocation entirely.

The stack is bounded — a few dozen to a few thousand entries — so memory can’t grow without limit. It’s also emptied during garbage collection, so recycled shells don’t keep arenas alive.

The core machinery

Two headers own the whole scheme:

 c
 
 
// Include/internal/pycore_freelist.h
#define _Py_FREELIST_FREE(NAME, op, freefunc) \
 _PyFreeList_Free(&_Py_freelists_GET()->NAME, _PyObject_CAST(op), \
 Py_ ## NAME ## _MAXFREELIST, freefunc)

#define _Py_FREELIST_PUSH(NAME, op, limit) \
 _PyFreeList_Push(&_Py_freelists_GET()->NAME, _PyObject_CAST(op), limit)

#define _Py_FREELIST_POP(TYPE, NAME) \
 _Py_CAST(TYPE*, _PyFreeList_Pop(&_Py_freelists_GET()->NAME))

#define _Py_FREELIST_POP_MEM(NAME) \
 _PyFreeList_PopMem(&_Py_freelists_GET()->NAME)

 c
 
 
// Include/internal/pycore_freelist_state.h
struct _Py_freelist {
 // Entries are linked together using the first word of the object.
 // For PyObjects, this overlaps with ob_refcnt (GIL build) or ob_tid
 // (free-threaded build).
 void *freelist;
 // -1 means the freelist is permanently disabled.
 Py_ssize_t size;
};

The push and pop helpers are tiny and inline:

 c
 
 
// Include/internal/pycore_freelist.h
static inline int
_PyFreeList_Push(struct _Py_freelist *fl, void *obj, Py_ssize_t maxsize)
{
 if (fl->size < maxsize && fl->size >= 0) {
 FT_ATOMIC_STORE_PTR_RELAXED(*(void **)obj, fl->freelist);
 fl->freelist = obj;
 fl->size++;
 OBJECT_STAT_INC(to_freelist);
 return 1;
 }
 return 0;
}

static inline PyObject *
_PyFreeList_Pop(struct _Py_freelist *fl)
{
 PyObject *op = _PyFreeList_PopNoStats(fl);
 if (op != NULL) {
 OBJECT_STAT_INC(from_freelist);
 _Py_NewReference(op); // resets refcount / ownership / GC bits
 }
 return op;
}

Three C tricks are already visible here:

Intrusive linked list. There are no separate list nodes. While an object sits on a freelist, its first word is overwritten with the next pointer. For a PyObject that word is ob_refcnt (or ob_tid in the free-threaded build), which is dead anyway because the object has been deallocated. On reuse, _Py_NewReference reinitializes that word.
LIFO. Recently freed objects are the hottest in cache, so reusing them first is good for locality.
Bounded. size < maxsize is the entire guard. When a freelist is full, the push fails and the object is really freed — so a tight loop can’t pin unbounded memory.

_Py_FREELIST_POP_MEM is the variant for plain C structs that aren’t PyObjects (for example PyDictKeysObject): it pops a block but doesn’t call _Py_NewReference.

What gets recycled, and how much

The limits live in Include/internal/pycore_freelist_state.h:

 c
 
 
#define PyTuple_MAXSAVESIZE 20
#define Py_tuple_MAXFREELIST 2000
#define Py_lists_MAXFREELIST 80
#define Py_list_iters_MAXFREELIST 10
#define Py_tuple_iters_MAXFREELIST 10
#define Py_dicts_MAXFREELIST 80
#define Py_dictkeys_MAXFREELIST 80
#define Py_floats_MAXFREELIST 100
#define Py_complexes_MAXFREELIST 100
#define Py_ints_MAXFREELIST 100
#define Py_slices_MAXFREELIST 1
#define Py_ranges_MAXFREELIST 6
#define Py_range_iters_MAXFREELIST 6
#define Py_pymethodobjects_MAXFREELIST 20

The container types are where the savings are largest, so they get the biggest limits. Tuples are special: instead of one freelist there are twenty, one for each size from 1 to 20.

list

 c
 
 
// Objects/listobject.c
PyListObject *op = _Py_FREELIST_POP(PyListObject, lists);
if (op == NULL) {
 op = PyObject_GC_New(PyListObject, &PyList_Type);
 ...
}
...
// dealloc
if (PyList_CheckExact(op)) {
 _Py_FREELIST_FREE(lists, op, PyObject_GC_Del);
}

Only an exact list (not a subclass) is recycled, and the items array is freed before the shell is pushed — so a recycled list is always empty. Limit: 80.

tuple

 c
 
 
// Objects/tupleobject.c
Py_ssize_t index = size - 1;
if (index < PyTuple_MAXSAVESIZE) {
 PyTupleObject *op = _Py_FREELIST_POP(PyTupleObject, tuples[index]);
 ...
}

 c
 
 
// Objects/tupleobject.c
static inline int
maybe_freelist_push(PyTupleObject *op)
{
 if (!Py_IS_TYPE(op, &PyTuple_Type)) {
 return 0;
 }
 Py_ssize_t index = Py_SIZE(op) - 1;
 if (index < PyTuple_MAXSAVESIZE) {
 return _Py_FREELIST_PUSH(tuples[index], op, Py_tuple_MAXFREELIST);
 }
 return 0;
}

Twenty buckets by size, 2000 entries each. The empty tuple is an immortal singleton and never gets freelist-recycled. Contents are decref’d before the shell is pushed.

dict

 c
 
 
// Objects/dictobject.c
PyDictObject *mp = _Py_FREELIST_POP(PyDictObject, dicts);
...
// keys: only minimum-size unicode keys
if (log2_size == PyDict_LOG_MINSIZE && unicode) {
 dk = _Py_FREELIST_POP_MEM(dictkeys);
}

 c
 
 
// Objects/dictobject.c
if (Py_IS_TYPE(mp, &PyDict_Type)) {
 _Py_FREELIST_FREE(dicts, mp, Py_TYPE(mp)->tp_free);
}
...
if (DK_LOG_SIZE(keys) == PyDict_LOG_MINSIZE &&
 keys->dk_kind == DICT_KEYS_UNICODE) {
 _Py_FREELIST_FREE(dictkeys, keys, PyMem_Free);
}

The dict object freelist holds 80 shells. The keys freelist holds 80 too, but only for minimum-size Unicode-key tables; larger or shared keys are freed normally. PyFrozenDict is not recycled.

float and int

 c
 
 
// Objects/floatobject.c
PyFloatObject *op = _Py_FREELIST_POP(PyFloatObject, floats);
if (op == NULL) {
 op = PyObject_Malloc(sizeof(PyFloatObject));
 ...
 _PyObject_Init((PyObject*)op, &PyFloat_Type);
}
op->ob_fval = fval;

 c
 
 
// Objects/longobject.c
if (ndigits == 1) {
 result = (PyLongObject *)_Py_FREELIST_POP(PyLongObject, ints);
}

 c
 
 
// Objects/longobject.c
void _PyLong_ExactDealloc(PyObject *self) {
 assert(PyLong_CheckExact(self));
 if (_PyLong_IsSmallInt((PyLongObject *)self)) {
 _Py_SetImmortal(self);
 return;
 }
 if (_PyLong_IsCompact((PyLongObject *)self)) {
 _Py_FREELIST_FREE(ints, self, PyObject_Free);
 return;
 }
 PyObject_Free(self);
}

Floats get 100 slots. Ints get 100, but only for compact, exact, non-small single-digit ints; small ints are immortal and multi-digit ints are freed outright. The small-int cache is a separate, larger mechanism — it gets its own article.

Frames are the exception

CPython 3.15 does not have a PyFrameObject freelist. frame_dealloc ends with PyObject_GC_Del(f).

Instead, interpreter frames are allocated from a per-thread chunked stack, with a single cached chunk reused:

 c
 
 
// Python/pystate.c
_PyInterpreterFrame *_PyThreadState_PushFrame(PyThreadState *tstate, size_t size) {
 if (_PyThreadState_HasStackSpace(tstate, (int)size)) {
 _PyInterpreterFrame *res = (_PyInterpreterFrame *)tstate->datastack_top;
 tstate->datastack_top += size;
 return res;
 }
 return (_PyInterpreterFrame *)push_chunk(tstate, (int)size);
}

So frames dodge per-object allocation by being slices of a reusable arena. Different trick, same purpose.

Thread safety

Where the freelist state lives depends on the build:

 c
 
 
// Include/internal/pycore_freelist.h
static inline struct _Py_freelists *
_Py_freelists_GET(void)
{
#ifdef Py_GIL_DISABLED
 PyThreadState *tstate = _PyThreadState_GET();
 return &((_PyThreadStateImpl*)tstate)->freelists; // per-thread
#else
 PyInterpreterState *interp = _PyInterpreterState_GET();
 return &interp->object_state.freelists; // per-interpreter
#endif
}

GIL build: freelists are per-interpreter and protected by the GIL. Plain loads and stores.
Free-threaded build: freelists are per-thread, so there’s no cross-thread sharing and no locks. Object ownership is encoded in ob_tid; _Py_NewReference sets it on reuse. Push uses FT_ATOMIC_STORE_PTR_RELAXED to publish the link.

Clearing differs accordingly: one interpreter freelist in the GIL build, versus walking every thread state in the free-threaded build. Pushes also use the relaxed atomic store in both builds, so link publication stays consistent with the free-threaded memory model.

Finalization and garbage collection

 c
 
 
// Objects/object.c
static void clear_freelist(struct _Py_freelist *freelist, int is_finalization,
 freefunc dofree) {
 void *ptr;
 while ((ptr = _PyFreeList_PopNoStats(freelist)) != NULL) {
 dofree(ptr);
 }
 assert(freelist->size == 0 || freelist->size == -1);
 if (is_finalization) {
 freelist->size = -1; // disable
 }
}

Setting size = -1 permanently disables a freelist. That matters during shutdown, so a recycled shell is never reused after its type’s deallocator has been torn down.

Objects sitting on freelists are untracked by the GC and empty, so they hold no references. During the highest-generation collection, all freelists are cleared (_PyGC_ClearAllFreeLists) so recycled shells don’t keep pymalloc arenas — and therefore memory — alive longer than necessary.

Why this is fast

The whole scheme is one idea repeated: reuse the first word of a dead object as the next pointer of an intrusive LIFO stack, bounded by a small per-type limit, and reinitialize the object with _Py_NewReference on reuse.

No separate list nodes. No hashing. No locking on the GIL path.

For types created and destroyed millions of times per second — tuples for calls, floats for arithmetic, ints for indexing — skipping the allocator on most operations is a large, measurable win. It’s the reason CPython’s object creation is as cheap as it is.

CPython's Pymalloc: A Fast Allocator for Small Objects

Jay Hubbert — Tue, 11 Nov 2025 09:00:00 +0000

TL;DR: Every small object CPython creates — a 56-byte str, a 56-byte int, a 64-byte list — comes from pymalloc, a segregated allocator built on arenas, pools, and size classes. The fast path is one array lookup, one comparison, and a free-list pop. No per-block size metadata, no system call, no scan.

Most Python objects are small. A 56-byte str, a 56-byte int, a 64-byte list: the interpreter allocates and frees these millions of times per second.

Going to the system allocator for each one would be slow, and it would fragment memory badly. CPython instead uses pymalloc, a segregated allocator tuned for objects up to 512 bytes. This article traces its three-level design and the handful of C tricks that keep the fast path short.

When pymalloc is used

pymalloc serves requests up to SMALL_REQUEST_THRESHOLD (512 bytes). Anything larger goes straight to the raw allocator.

It is the default obj and mem allocator in GIL builds. Free-threaded builds use mimalloc instead (see below), because pymalloc has no per-thread caches and leans on the GIL for safety.

The three levels

text

arena (1 MiB; 2 MiB with hugepages; 256 KiB on 32-bit)
 └── pool (16 KiB on 64-bit; 4 KiB == page otherwise)
 └── block (8..512 bytes, 16-byte aligned on 64-bit)

The numbers behind that diagram:

ARENA_SIZE = 1 << ARENA_BITS (20, so 1 MiB).
POOL_SIZE = 1 << POOL_BITS (14, so 16 KiB on 64-bit).
SMALL_REQUEST_THRESHOLD = 512.
NB_SMALL_SIZE_CLASSES = 512 / ALIGNMENT = 32 (16-byte spacing).
Size class index: (nbytes - 1) >> ALIGNMENT_SHIFT (0..31).
Block size: INDEX2SIZE(i) = (i + 1) << ALIGNMENT_SHIFT.

Think of it as a sorting office: an arena is the building, a pool is a room, and a room only ever holds one kind of envelope. An arena is carved into pools, a pool is carved into blocks of a single size class, and every block in that pool is the same size.

That segregation is the whole point. Because a pool never mixes sizes, there’s no per-block size metadata to store and look up on free.

The structures that make it work

 c
 
 
// Include/internal/pycore_obmalloc.h
struct pool_header {
 union { pymem_block *_padding; uint count; } ref; // allocated block count
 pymem_block *freeblock; // singly-linked free list head
 struct pool_header *nextpool, *prevpool; // circular used-pools list
 uint arenaindex;
 uint szidx; // size class
 uint nextoffset; // offset to next virgin block
 uint maxnextoffset;
};

struct arena_object {
 /* pool_address is the carving cursor */
 ...
 uint nfreepools, ntotalpools;
 struct pool_header *freepools; // singly-linked empty pools
 ...
};

ref.count is simply the number of allocated blocks in the pool.

freeblock is a singly-linked free list threaded through the dead blocks themselves. Same intrusive-list trick CPython uses for its freelists, but here it’s happening inside a pool: the memory of a freed block holds the pointer to the next free block.

The usedpools fudge

_obmalloc_state holds an array usedpools with one slot per size class. Here’s the trick: the array is indexed as if each slot were a pool whose nextpool/prevpool point at itself, forming an empty circular list head:

 c
 
 
// Include/internal/pycore_obmalloc_init.h
/* usedpools[i] is really a pool_header* whose nextpool/prevpool are itself */

So the allocation path never has to special-case “no pools yet”. The empty list simply looks empty.

 c
 
 
// Objects/obmalloc.c
uint size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT;
poolp pool = usedpools[size + size];
if (LIKELY(pool != pool->nextpool)) { // a used pool exists
 ++pool->ref.count;
 bp = pool->freeblock;
 if (UNLIKELY((pool->freeblock = *(pymem_block**)bp) == NULL)
 pymalloc_pool_extend(pool, size); // carve next virgin block
}
else
 bp = allocate_from_new_pool(state, size); // empty pool / new arena

Read that again and you’ll see the whole allocator: one array lookup, one comparison, one free-list pop. No locks on the GIL path, no size metadata, no search.

Block carving is lazy

A pool is not fully initialized when it’s handed out. Only the first two blocks are set up; nextoffset and maxnextoffset track the still-virgin region.

Blocks get carved on demand as the free list empties. A pool becomes “full” implicitly when the free list is empty and nextoffset > maxnextoffset, at which point it gets unlinked from the used list.

On free, the block is prepended to freeblock. If the pool had been full, it is re-linked to the front of the used list (LRU-ish reuse, so hot sizes stay hot). If ref.count drops to zero, the pool moves to the arena’s freepools list and the arena bookkeeping is updated.

Picking an arena without a linear scan

With many arenas, searching for one with a free pool would be O(n). CPython keeps usable_arenas sorted ascending by nfreepools and uses nfp2lasta[] “search fingers” so the lookup stays O(1) instead of linear (bpo-37029).

When an arena becomes entirely free, it goes back to the OS (_PyObject_Arena.free), its radix-tree entry is cleared, and the arena_object is recycled into unused_arena_objects.

One detail worth internalizing: one wholly-free arena is deliberately retained rather than returned immediately, to avoid thrashing (bpo-37257). Handing the memory back and immediately re-requesting it is a classic allocator disaster.

The radix tree answers “is this pointer mine?” in O(1)

On free, pymalloc must decide whether a pointer belongs to it or to the raw allocator. Guessing wrong is a memory-corruption bug, not a slow path.

On 64-bit builds a three-level radix tree (arena_map_is_used) answers that in O(depth) — roughly 3 levels — without scanning the arena array at all. On 32-bit builds a single level suffices. Without the radix tree, the code reads pool->arenaindex once (volatile) and checks allarenas[ai].address.

The radix tree also handles arenas that straddle an “ideal” boundary via arena_coverage_t { tail_hi, tail_lo }, and its top/mid arrays live in BSS and are demand-paged.

Address checks and debugging

 c
 
 
static inline int pymalloc_free(OMState *state, void *ctx, void *p) {
 poolp pool = POOL_ADDR(p);
 if (UNLIKELY(!address_in_range(state, p, pool))) return 0; // not ours
 assert(pool->ref.count > 0);
 *(pymem_block**)p = pool->freeblock;
 pool->freeblock = (pymem_block*)p;
 pool->ref.count--;
 if (LIKELY(pool->ref.count != 0)) return 1;
 insert_to_freepool(state, pool); // may return arena to OS
 return 1;
}

Notice the shape: check ownership, push onto the free list, decrement the count, and only when the pool empties do you do anything expensive.

Debug mode wraps every allocation with padding words: [size][id+forbidden][data][forbidden tail][serial]. _PyMem_DebugCheckAddress validates the forbidden bytes and API id; a mismatch dumps the address and fatally errors.

PYTHONMALLOCSTATS=1 triggers pymalloc_print_stats (or mimalloc stats), also reachable from Python via sys._debugmallocstats().

Free-threaded builds: mimalloc plus QSBR

pymalloc leans on the GIL, so a Py_GIL_DISABLED build needs WITH_MIMALLOC and uses mimalloc for the mem and obj domains.

Mimalloc brings per-thread heaps and caches. CPython layers Python-specific pieces on top:

Separate heaps for GC objects (_Py_MIMALLOC_HEAP_GC, _GC_PRE) so the GC header/pre-header layout is honored.
Object realloc copies the PyObject header atomically, word by word, before memcpy-ing the rest.
QSBR delayed free: freed pointers and refcount decrements are queued in _mem_work_chunk (254 items, 4096 bytes) with a QSBR goal and processed from the eval breaker when _Py_qsbr_poll succeeds. Limits: QSBR_DEFERRED_LIMIT 127, QSBR_FREE_MEM_LIMIT 1 MiB, QSBR_PAGE_MEM_LIMIT 80 KiB. This is what prevents use-after-free when another thread is still reading a refcount.
Threads donate segments with live blocks on exit; other threads claim and reuse them.

Gotcha: the free-threaded allocator story is genuinely different from the GIL story. If you’re reasoning about allocator behavior, check whether your build is Py_GIL_DISABLED first.

Interpreters and huge pages

The main interpreter (and interpreters with Py_RTFLAGS_USE_MAIN_OBMALLOC) share obmalloc_state_main in BSS (~256 KiB, mostly the radix top array). Subinterpreters with their own state get a heap-allocated state.
Huge pages: with PYMALLOC_USE_HUGEPAGES and --with-pymalloc-hugepages, the arena size becomes 2 MiB via MAP_HUGETLB (Linux) or VirtualAlloc (Windows), reducing TLB misses.

Wrapping up: why pymalloc is fast

The fast path is a size-class lookup, one comparison against a self-looping list head, and a free-list pop. No size stored per block. No global lock beyond the GIL. No system call.

Segregation by size class means a pool either fits your request exactly or gets skipped entirely. The radix tree makes frees O(1) with respect to the number of arenas.

For the millions of small objects Python creates per second, that is the difference between a usable interpreter and a slow one.

The one thing to remember: pymalloc’s speed comes from not asking questions. It never has to work out how big a block is, whether it owns a pointer, or which pool to search — all of that is answered by structure instead of by computation.

If you want to see it for yourself, set PYTHONMALLOCSTATS=1 and watch the arena and pool counts move as you churn through a list of a million small tuples.

Memory on Opensource internals

How ClickHouse's Sparse Column Stores Only the Rows That Differ

The layout

The constructor enforces the invariant

Inserting

The iterator: amortized O(1) sequential access

On disk: the offsets become run-length groups

Decoding it

How the encoding gets chosen

The three substreams

Writing

The query layer wants to peel the sparse wrapper off

The other side of the coin: the null map

Seeing it in SQL

Trade-offs

How ClickHouse Stores a Column: PODArray, the Typed Array, and the Padding Trick

The container

Noncopyable

Three pointers, not a size and a capacity

The empty array points at static memory

The padding trick

Left padding: the -1 element

Right padding: SIMD may write past the end

The growth policy

reserve: round up to a power of two

reserve_exact: exactly what was asked for

push_back: double the allocation

The “shrink if less than half used” trick

ColumnVector: the typed array

insertRangeFrom: memcpy, not a loop

insertRawUninitialized: writing into the column’s memory directly

ColumnString: offsets plus a flat character buffer

Bulk insertion

ColumnFixedString: the inline variant

ColumnNullable: the null map as a separate column

Trade-offs

ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

The layout

Global vs local order

Reading a row

Building the offsets

Insertion keeps the two arrays in sync

The insert path

Serialization: compact discriminators

Dynamic: the open-ended case

The insert path

Adding a variant

Seeing it in SQL

Trade-offs

Wrapping up

ClickHouse's LowCardinality Column Shares One Dictionary Across Millions of Rows

The shape of the thing

The index column grows its own type

The dictionary: ColumnUnique and its reverse index

Two reserved slots

The reverse index stores row numbers, not values

The index is built lazily, and the hash table is filled in one pass

Float NaNs are canonicalized

Sharing the dictionary between columns

But a shared dictionary is immutable

Inserting from a column with the same dictionary

On disk: one dictionary per part, versioned per granule

The version byte

The per-granule header

Writing: the dictionary accumulates, then overflows

Reading: the reader has to reconstruct the same state

The single-dictionary-for-part fast path

The overflow path on the read side

The whole thing in SQL

Trade-offs

Wrapping up

ClickHouse Marks Empty Hash Table Cells With Zeroed Memory

The shape of the table

Empty means zero

The zero key gets its own slot

Probing

Insertion

Growing: the wrap-around special case

Robin Hood: what changed recently

Deletion without tombstones

Left padding: the `-1` element

`reserve`: round up to a power of two

`reserve_exact`: exactly what was asked for

`push_back`: double the allocation

`ColumnVector`: the typed array

`insertRangeFrom`: memcpy, not a loop

`insertRawUninitialized`: writing into the column’s memory directly

`ColumnString`: offsets plus a flat character buffer

`ColumnFixedString`: the inline variant

`ColumnNullable`: the null map as a separate column

How `pfree` knows where to send the pointer

`repalloc`: sometimes it is free