Opensource internals

ClickHouse's Variant Column Uses a Discriminator Byte and a Shared Fallback

19 min read Jay Hubbert

ColumnVariant stores one discriminator byte plus one offset per row next to one sub-column per alternative, and ColumnDynamic falls back to a shared String variant.

TL;DR: A Variant column is a struct of arrays — one sub-column per alternative, one tag byte per row, one offset per row. The tag is a local index; a separate global↔local mapping lets the type grow at runtime without rewriting a single row.

A Variant column is a tagged union: each row holds a value of one of several types, and the column itself knows the full list of alternatives. Dynamic is the same idea with an open-ended list — the set of types is discovered from the data.

The obvious implementation is a struct-of-arrays: one sub-column per alternative, each holding only the rows that belong to it, plus a tag per row saying which sub-column to look in. That’s what ClickHouse does.

What makes the implementation interesting isn’t the tag. It’s the set of awkward consequences that fall out of it:

  • The order of the alternatives must not matter (Variant(String, UInt64) and Variant(UInt64, String) are the same type), but the storage of a discriminator is a small integer, so the order does matter on disk.
  • The type must be extendable at runtime without rewriting the data, which means a column’s internal ordering may differ from the type’s ordering.
  • The number of alternatives must be bounded, but a Dynamic column can’t know its alternatives in advance, so it needs an escape hatch.
  • Reading a row must not require a virtual call and a branch on the tag.

Let’s look at how ClickHouse 26.9 solves all four.

The layout

The header comment of ColumnVariant is a complete specification, worked example included:

cpp 
// src/Columns/ColumnVariant.h:12
/**
 * Column for storing Variant(...) type values.
 * Variant type represents a union of other data types.
 * For example, type Variant(T1, T2, ..., TN) means that each row of this type
 * has a value of either type T1 or T2 or ... or TN or none of them (NULL value)
 *
 * ColumnVariant stores:
 *   - The discriminators column, which determines which variant is stored in each row.
 *   - The offsets column, which determines the offset in the corresponding variant column in each row.
 *   - The list of variant columns with only real values (so the sizes of variant columns can be different).
 * Discriminator is an index of a variant in the variants list, it also has special value called NULL_DISCRIMINATOR
 * that indicates that the value in the row is NULL.
 *
 * We want to be able to extend Variant column for free without rewriting the data, but as we don't care about the
 * order of variants during Variant creation (we want Variant(T1, T2) to be the same as Variant(T2, T1)), we support
 * some global order of nested types inside Variant during type creation, so after extension the order of variant types
 * (and so their discriminators) can change. For example: Variant(T1, T3) -> Variant(T1, T2, T3).
 * To avoid full rewrite of discriminators column on Variant extension, we differentiate local order of variants
 * inside a column and global order of variants created during type creation. So, ColumnVariant stores only local
 * discriminators and additionally stores the mapping between global and local discriminators.
 * So, when we need to extend Variant column with new variant, we can just append it to a list of variant columns
 * with new local discriminator and update mapping from global to local orders.
 *
 * Note that two instances of ColumnVariant can have different local orders, so we should always use global
 * discriminators during inter-column interactions.
 *
 * Let's take an example with type Variant(UInt32, String, Array(UInt32)):
 * During type creation we will sort types by their names and get the global order: Array(UInt32), String, UInt32.
 * So, type Array(UInt32) will have global discriminator 0, String - 1 and UInt32 - 2.
 * Let's say we have a column with local order (String, UInt32, Array(UInt32)) and values:
 * 'Hello', 42, NULL, 'World', 43, [1, 2, 3], NULL, 44
 *
 * Let's see how these values will be stored in ColumnVariant:
 *
 * local_to_global_discriminators: {0 : 1, 1 : 2, 2 : 0}
 * global_to_local_discriminators: {0 : 2, 1 : 0, 2 : 1}
 * local_discriminators    offsets    String    UInt32    Array(UInt32))
 *         0                  0       'Hello'     42        [1, 2, 3]
 *         1                  0       'World'     43
 *  NULL_DISCRIMINATOR        0                   44
 *         0                  1
 *         1                  1
 *         2                  0
 *  NULL_DISCRIMINATOR        0
 *         1                  2
 *
 */

That diagram is worth staring at, because it shows the two central ideas:

text 
   row:        0      1      2       3      4      5        6      7
   local:      0      1    NULL     0      1      2      NULL     1
   offsets:    0      0      0       1      1      0        0      2

   String:   'Hello' 'World'                       (2 values)
   UInt32:                42        43              44  (3 values)
   Array:                            [1,2,3]              (1 value)
  1. The discriminator is one byte per row. It’s a local index into the column’s own list of variant columns.
  2. The offset is a row index within that particular variant column. Row 3 has local discriminator 0 (String) and offset 1, so its value is String[1] == 'World'. Row 5 has local discriminator 2 (Array) and offset 0, so its value is Array[0].

So per row you pay one byte of tag plus one Offset (UInt64) of position. The values themselves are stored densely, one after another, inside their own sub-column — a String sub-column holds only the strings, a UInt32 sub-column holds only the integers, and neither has holes.

The types involved:

cpp 
// src/Columns/ColumnVariant.h:62
    using Discriminator = UInt8;
    using Discriminators = PaddedPODArray<Discriminator>;
    using ColumnDiscriminators = ColumnVector<Discriminator>;
    using ColumnOffsets = ColumnVector<Offset>;

    static constexpr UInt8 NULL_DISCRIMINATOR = std::numeric_limits<Discriminator>::max(); /// 255
    static constexpr size_t MAX_NESTED_COLUMNS = std::numeric_limits<Discriminator>::max(); /// 255

NULL_DISCRIMINATOR is 255 — one past the largest legal alternative index. That means a column can hold at most 255 alternatives, and 255 is reserved for NULL. The value is not stored in any sub-column; it’s only a tag.

Global vs local order

The tag stored in the column is a local discriminator. The global discriminator is the position in the type’s canonical ordering. The canonical ordering is by type name — that’s how Variant(T1, T2) and Variant(T2, T1) become the same type:

cpp 
// src/DataTypes/DataTypeVariant.cpp:46
DataTypeVariant::DataTypeVariant(const DataTypes & variants_)
{
    /// Sort nested types by their full names and squash identical types.
    std::map<String, DataTypePtr> name_to_type;
    for (const auto & type : variants_)
    {
        checkAllowedInsideVariant(type);

        /// Don't use Nothing type as a variant.
        if (!isNothing(type))
            name_to_type[type->getName()] = type;
    }

    variants.reserve(name_to_type.size());
    for (const auto & [_, type] : name_to_type)
        variants.push_back(type);

    checkVariantsNotEmptyAndNotTooMany(variants);
}

A std::map keyed by the type name gives the sorted, de-duplicated order for free.

Now, why do we need two orders at all? Because a Dynamic column is created with one set of alternatives and then extended as new types arrive. When Variant(T1, T3) becomes Variant(T1, T2, T3), the canonical (global) order of T3 may change — T2 might sort before it. If the column stored global discriminators, every stored tag would have to be rewritten.

The fix is to keep the column’s local order stable and just maintain a mapping:

cpp 
// src/Columns/ColumnVariant.h:160
    Discriminator ALWAYS_INLINE globalDiscriminatorByLocal(Discriminator local_discr) const
    {
        /// NULL_DISCRIMINATOR is always the same in local and global orders.
        return local_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : local_to_global_discriminators[local_discr];
    }

    Discriminator ALWAYS_INLINE localDiscriminatorByGlobal(Discriminator global_discr) const
    {
        /// NULL_DISCRIMINATOR is always the same in local and global orders.
        return global_discr == NULL_DISCRIMINATOR ? NULL_DISCRIMINATOR : global_to_local_discriminators[global_discr];
    }

and the extension operation is a pure metadata change:

cpp 
// src/Columns/ColumnVariant.cpp:1904
void ColumnVariant::extend(const VectorWithMemoryTracking<Discriminator> & old_to_new_global_discriminators, VectorWithMemoryTracking<std::pair<MutableColumnPtr, Discriminator>> && new_variants_and_discriminators)
{
    /// Update global discriminators for current variants.
    for (Discriminator & global_discr : local_to_global_discriminators)
        global_discr = old_to_new_global_discriminators[global_discr];

    /// Add new variants.
    variants.reserve(variants.size() + new_variants_and_discriminators.size());
    local_to_global_discriminators.reserve(local_to_global_discriminators.size() + new_variants_and_discriminators.size());
    for (auto & new_variant_and_discriminator : new_variants_and_discriminators)
    {
        variants.emplace_back(std::move(new_variant_and_discriminator.first));
        local_to_global_discriminators.push_back(new_variant_and_discriminator.second);
    }

    /// Update global -> local discriminators matching.
    global_to_local_discriminators.resize(local_to_global_discriminators.size());
    for (Discriminator local_discr = 0; local_discr != local_to_global_discriminators.size(); ++local_discr)
        global_to_local_discriminators[local_to_global_discriminators[local_discr]] = local_discr;
}

Nothing in that function touches the discriminator column or the offsets column. The existing rows’ local tags stay valid; only the interpretation of the tags changes. That’s the “extend Variant column for free without rewriting the data” promise, kept.

The invariant is that local discriminators are always dense: local discriminator k is the index of variants[k]. Global discriminators are sparse in the sense that a given column may not contain all of them.

Reading a row

cpp 
// src/Columns/ColumnVariant.cpp:415
Field ColumnVariant::operator[](size_t n) const
{
    Discriminator discr = localDiscriminatorAt(n);
    if (discr == NULL_DISCRIMINATOR)
        return Null();
    return (*variants[discr])[offsetAt(n)];
}

Two loads and a branch. The branch on NULL_DISCRIMINATOR is unavoidable. What you want to avoid is a virtual call per row — and variants[discr] is an IColumn&, so operator[] on it is a virtual call.

That’s why the hot paths in ClickHouse don’t use operator[] but instead reach into the concrete sub-column via getDataAt, and why ColumnVariant also exposes getVariantByLocalDiscriminator so callers can hoist the type test out of the loop.

Note also that the offsets array is not a byte offset into a shared arena. It’s an element index into the sub-column. That’s what allows each sub-column to be an ordinary ClickHouse column with its own type-specific representation (a ColumnString with offsets+chars, a ColumnVector<UInt32>, a ColumnArray, …), and it’s why byteSizeAt has to consult the sub-column:

cpp 
// src/Columns/ColumnVariant.cpp:1564
size_t ColumnVariant::byteSizeAt(size_t n) const
{
    size_t res = sizeof(Offset) + sizeof(Discriminator);
    Discriminator discr = localDiscriminatorAt(n);
    if (discr == NULL_DISCRIMINATOR)
        return res;

    return res + variants[discr]->byteSizeAt(offsetAt(n));
}

So a Variant row costs 9 bytes of overhead (1 discriminator + 8 offset) plus the payload. The offsets are the expensive part, and that’s deliberate: an offset into a typed sub-column is what lets the sub-column be stored in its native, compressible layout.

Building the offsets

The offsets column is derived from the discriminators. constructOffsetsFromDiscriminators computes it, and it has three special cases:

cpp 
// src/Columns/ColumnVariant.cpp:69
void ColumnVariant::constructOffsetsFromDiscriminators()
{
    const ColumnDiscriminators * discriminators_concrete = typeid_cast<const ColumnDiscriminators *>(local_discriminators.get());
    Offsets & offsets_data = typeid_cast<ColumnOffsets *>(offsets.get())->getData();
    offsets_data.clear();
    offsets_data.reserve(discriminators_concrete->size());
    /// If we have only NULLs, offsets column will not contain any real offsets.
    if (hasOnlyNulls())
    {
        /// Offsets of NULL rows are never read for their value, but compression reads the whole buffer.
        offsets_data.resize_fill(discriminators_concrete->size(), 0);
    }
    /// If we have only one non empty variant and no NULLs,
    /// offsets column will contain just sequential offsets 0, 1, 2, ...
    else if (getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
    {
        for (size_t i = 0; i != discriminators_concrete->size(); ++i)
            offsets_data.push_back(i);
    }
    /// Otherwise we should iterate through discriminators and
    /// remember current offset for each variant column.
    else
    {
        VectorWithMemoryTracking<Offset> nested_offsets;
        nested_offsets.resize(variants.size());
        for (Discriminator discr : discriminators_concrete->getData())
        {
            if (discr == NULL_DISCRIMINATOR)
            {
                offsets_data.emplace_back();
            }
            else
            {
                offsets_data.push_back(nested_offsets[discr]);
                ++nested_offsets[discr];
            }
        }
    }
}

The general case is a running counter per variant — a “rank within the sub-column” computation.

The special cases matter for the serializer. If every row is NULL, the offsets buffer is all zeros (and the comment notes that compression reads the whole buffer, so it can’t be empty). If there’s exactly one non-empty variant, the offsets are simply 0, 1, 2, ... and the whole offsets column is unnecessary information.

Insertion keeps the two arrays in sync

The discriminator and the offsets column must always have the same length. Because an allocation failure in the middle of an insert could leave them inconsistent, the code is careful:

cpp 
// src/Columns/ColumnVariant.cpp:737
void ColumnVariant::insertDefault()
{
    getLocalDiscriminators().push_back(NULL_DISCRIMINATOR);
    /// Keep local_discriminators and offsets in sync even if appending to offsets throws
    /// (e.g. on a memory limit), otherwise popBack would over-pop the shorter one.
    try
    {
        getOffsets().emplace_back();
    }
    catch (...)
    {
        getLocalDiscriminators().popBack();
        throw;
    }
}

That’s exception-safety hygiene you’ll see repeatedly in this code base: the two columns are a matched pair, and one of them can throw (memory allocation) while the other can’t.

The insert path

Inserting a value from another variant column is where the global/local mapping earns its keep:

cpp 
// src/Columns/ColumnVariant.cpp:501
void ColumnVariant::insertFromImpl(const DB::IColumn & src_, size_t n, const VectorWithMemoryTracking<ColumnVariant::Discriminator> * global_discriminators_mapping)
{
    const size_t num_variants = variants.size();
    const ColumnVariant & src = assert_cast<const ColumnVariant &>(src_);

    if (!global_discriminators_mapping && src.variants.size() != num_variants)
        throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot insert value of Variant type with different number of types");

    Discriminator src_global_discr = src.globalDiscriminatorAt(n);
    Discriminator global_discr = src_global_discr;
    if (global_discriminators_mapping && src_global_discr != NULL_DISCRIMINATOR)
        global_discr = (*global_discriminators_mapping)[src_global_discr];
    Discriminator local_discr = localDiscriminatorByGlobal(global_discr);
    getLocalDiscriminators().push_back(local_discr);
    if (local_discr == NULL_DISCRIMINATOR)
    {
        getOffsets().emplace_back();
    }
    else
    {
        getOffsets().push_back(variants[local_discr]->size());
        variants[local_discr]->insertFrom(src.getVariantByGlobalDiscriminator(src_global_discr), src.offsetAt(n));
    }
}

Read it as a three-step translation: source local tag → source global tag → destination local tag. The source’s local tag means nothing to the destination; the global tag is the portable currency; the destination’s local tag is what gets stored.

Serialization: compact discriminators

On disk, the discriminator stream has two modes:

cpp 
// src/DataTypes/Serializations/SerializationVariant.h:54
    struct DiscriminatorsSerializationMode
    {
        enum Value
        {
            BASIC = 0,    /// Store the whole discriminators column.
            COMPACT = 1,  /// Don't write discriminators in granule if all of them are the same.
        };

In BASIC mode the whole discriminator column is written. In COMPACT mode a granule whose discriminators are all the same writes one discriminator plus a marker byte instead of 65 536 of them. The writer has to detect those three situations (uniform non-empty, all NULL, mixed) and there’s a fast path for each:

cpp 
// src/DataTypes/Serializations/SerializationVariant.cpp:343
    /// Write number of rows in this granule in compact mode.
    if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
        writeVarUInt(UInt64(limit), *discriminators_stream);

    /// If column has only one none empty discriminators and no NULLs we don't need to
    /// calculate limits for variants and use provided offset/limit.
    if (auto non_empty_local_discr = col.getLocalDiscriminatorOfOneNoneEmptyVariantNoNulls())
    {
        auto non_empty_global_discr = col.globalDiscriminatorByLocal(*non_empty_local_discr);

        /// In compact mode write the format of the granule and single non-empty discriminator.
        if (variant_state->discriminators_mode.value == DiscriminatorsSerializationMode::COMPACT)
        {
            writeBinaryLittleEndian(UInt8(CompactDiscriminatorsGranuleFormat::COMPACT), *discriminators_stream);
            writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
        }
        /// For basic mode just serialize this discriminator limit times.
        else
        {
            for (size_t i = 0; i < limit; ++i)
                writeBinaryLittleEndian(non_empty_global_discr, *discriminators_stream);
        }

The “one non-empty variant” fast path also lets the value streams be serialized with the same offset/limit as the whole column, because in that case the sub-column has exactly the same number of rows as the variant column:

cpp 
// src/DataTypes/Serializations/SerializationVariant.cpp:370
            /// We can use the same offset/limit as for whole Variant column
            if (i == non_empty_global_discr)
            {
                const auto & variant_column = col.getVariantByGlobalDiscriminator(i);
                if (variant_column.size() < offset + limit)
                    throw Exception(ErrorCodes::LOGICAL_ERROR, "Variant {} has less rows ({}) than expected rows to serialize ({})", variant_names[i], variant_column.size(), offset + limit);

                variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(variant_column, offset, limit, settings, variant_state->variant_states[i]);
            }
            else
            {
                variant_serializations[i]->serializeBinaryBulkWithMultipleStreams(col.getVariantByGlobalDiscriminator(i), col.getVariantByGlobalDiscriminator(i).size(), 0, settings, variant_state->variant_states[i]);
            }

In the general case each sub-column is serialized with its own length, which is the whole point of the dense layout: the String sub-column is written as a String column with only the strings in it, and so it compresses as well as a plain string column would.

Dynamic: the open-ended case

Dynamic is a Variant whose alternative list is discovered at runtime. It’s implemented as a ColumnVariant plus a VariantInfo record:

cpp 
// src/Columns/ColumnDynamic.h:15
/**
 * Column for storing Dynamic type values.
 * Dynamic column allows to insert and store values of any data types inside.
 * Inside it stores:
 *   - Variant column with all inserted values of different types.
 *   - Information about currently stored variants.
 *
 * When new values are inserted into Dynamic column, the internal Variant
 * type and column are extended if the inserted value has new type.
 * When the limit on number of dynamic types is exceeded, all values
 * with new types are inserted into special shared variant with type String
 * that contains values and their types in binary format.
 */
class ColumnDynamic final : public COWHelper<IColumnHelper<ColumnDynamic>, ColumnDynamic>
{
public:
    /// Maximum limit on dynamic types. We use ColumnVariant to store all the types,
    /// so the limit cannot be greater then ColumnVariant::MAX_NESTED_COLUMNS.
    /// We also always have reserved variant for shared variant.
    static constexpr size_t MAX_DYNAMIC_TYPES_LIMIT = ColumnVariant::MAX_NESTED_COLUMNS - 1;
    static constexpr const char * SHARED_VARIANT_TYPE_NAME = "SharedVariant";

The escape hatch is a shared variant: one of the alternatives is always a String column, and it holds the type name and the binary-serialized value of anything that didn’t fit into a real alternative.

It has to be a String with a custom name, because a plain String alternative might legitimately be in the same column:

cpp 
// src/Columns/ColumnDynamic.cpp:71
/// Shared variant will contain String values but we cannot use usual String type
/// because we can have regular variant with type String.
/// To solve it, we use String type with custom name for shared variant.
const DataTypePtr & ColumnDynamic::getSharedVariantDataType()
{
    static thread_local DataTypePtr type = DataTypeFactory::instance().getCustom("String", std::make_unique<DataTypeCustomDesc>(std::make_unique<DataTypeCustomFixedName>(getSharedVariantTypeName())));
    return type;
}

and a fresh Dynamic column starts life with only the shared variant:

cpp 
// src/Columns/ColumnDynamic.cpp:80
ColumnDynamic::ColumnDynamic(size_t max_dynamic_types_) : max_dynamic_types(max_dynamic_types_), global_max_dynamic_types(max_dynamic_types)
{
    /// Create Variant with shared variant.
    setVariantType(std::make_shared<DataTypeVariant>(DataTypes{getSharedVariantDataType()}));
}

The insert path

cpp 
// src/Columns/ColumnDynamic.cpp:257
void ColumnDynamic::insert(const Field & x)
{
    if (x.isNull())
    {
        insertDefault();
        return;
    }

    auto & variant_col = getVariantColumn();
    auto shared_variant_discr = getSharedVariantDiscriminator();
    /// Check if we can insert field into existing variants and avoid Variant extension.
    for (ColumnVariant::Discriminator i = 0; i != variant_col.getNumVariants(); ++i)
    {
        if (i != shared_variant_discr && variant_col.getVariantByGlobalDiscriminator(i).tryInsert(x))
        {
            variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(i));
            variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(i).size() - 1);
            return;
        }
    }

    /// If we cannot insert field into current variant column, extend it with new variant for this field from its type.
    /// Use LeastSupertypeOnError::Dynamic so that arrays with incompatible element types (e.g. ["text", {"k":1}])
    /// are typed as Array(Dynamic) rather than throwing NO_COMMON_TYPE. Dynamic can hold any element value.
    auto field_data_type = applyVisitor(FieldToDataType<LeastSupertypeOnError::Dynamic>(), x);
    auto field_data_type_name = field_data_type->getName();
    if (addNewVariant(field_data_type, field_data_type_name))
    {
        /// Insert this field into newly added variant.
        auto discr = variant_info.variant_name_to_discriminator[field_data_type_name];
        variant_col.getVariantByGlobalDiscriminator(discr).insert(x);
        variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(discr));
        variant_col.getOffsets().push_back(variant_col.getVariantByGlobalDiscriminator(discr).size() - 1);
    }
    else
    {
        /// We reached maximum number of variants and couldn't add new variant.
        /// In this case we add the value of this new variant into special shared variant.
        /// We store values in shared variant in binary form with binary encoded type.
        auto & shared_variant = getSharedVariant();
        auto & chars = shared_variant.getChars();
        {
            WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
            encodeDataType(field_data_type, value_buf);
            getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());
        }
        shared_variant.getOffsets().push_back(chars.size());
        variant_col.getLocalDiscriminators().push_back(variant_col.localDiscriminatorByGlobal(shared_variant_discr));
        variant_col.getOffsets().push_back(shared_variant.size() - 1);
    }
}

Three outcomes, in order:

  1. The value fits an existing alternative — use it, no extension.
  2. The value has a new type and there’s room — extend the variant list and add a real alternative.
  3. The value has a new type and the list is full — encode the type and the value into the shared variant’s character buffer.

Case 3 is the interesting one. The shared variant is a ColumnString, so writing into it means writing into its chars buffer directly. The type is encoded first, then the value:

cpp 
            WriteBufferFromVector<ColumnString::Chars> value_buf(chars, AppendModeTag());
            encodeDataType(field_data_type, value_buf);
            getVariantSerialization(field_data_type, field_data_type_name)->serializeBinary(x, value_buf, getFormatSettings());

and reading it back is the mirror image:

cpp 
// src/Columns/ColumnDynamic.cpp:323
void ColumnDynamic::get(size_t n, Field & res) const
{
    /// Object, Dynamic and Variant values nest into each other to a depth that comes from the data, not from the declared type.
    checkStackSize();

    const auto & variant_col = getVariantColumn();
    /// Check if value is not in shared variant.
    if (variant_col.globalDiscriminatorAt(n) != getSharedVariantDiscriminator())
    {
        variant_col.get(n, res);
        if (!res.isNull())
            res = convertFieldToType(res, *getTypeAt(n));
        return;
    }

    /// We should deserialize value from shared variant.
    const auto & shared_variant = getSharedVariant();
    auto value_data = shared_variant.getDataAt(variant_col.offsetAt(n));
    ReadBufferFromMemory buf(value_data);
    auto type = decodeDataType(buf);
    type->getDefaultSerialization()->deserializeBinary(res, buf, getFormatSettings());
}

Note checkStackSize(). Because Dynamic can nest inside Dynamic inside Dynamic, the depth is a function of the data, and a malicious input could otherwise blow the stack. That’s a real concern for a type whose shape is discovered at runtime, and it shows up in the code as an explicit guard.

Adding a variant

cpp 
// src/Columns/ColumnDynamic.cpp:145
bool ColumnDynamic::addNewVariant(const DataTypePtr & new_variant, const String & new_variant_name)
{
    /// Check if we already have such variant.
    if (variant_info.variant_name_to_discriminator.contains(new_variant_name))
        return true;

    /// Check if we reached maximum number of variants.
    if (!canAddNewVariant())
    {
        /// Dynamic column should always have shared variant.
        if (!variant_info.variant_name_to_discriminator.contains(getSharedVariantTypeName()))
            throw Exception(ErrorCodes::LOGICAL_ERROR, "Maximum number of variants reached, but no shared variant exists");

        return false;
    }

    const DataTypes & current_variants = assert_cast<const DataTypeVariant &>(*variant_info.variant_type).getVariants();
    DataTypes all_variants = current_variants;
    all_variants.push_back(new_variant);
    auto new_variant_type = std::make_shared<DataTypeVariant>(all_variants);
    updateVariantInfoAndExpandVariantColumn(new_variant_type);
    return true;
}

The new type is appended to the type, and then the column is expanded to match. updateVariantInfoAndExpandVariantColumn is where ColumnVariant::extend gets called with the old-to-new global discriminator mapping.

Seeing it in SQL

sql 
CREATE TABLE events
(
    `ts`      DateTime,
    `payload` Dynamic
)
ENGINE = MergeTree
ORDER BY ts;

INSERT INTO events VALUES
    ('2026-01-01 00:00:00', 42),
    ('2026-01-01 00:00:01', 'hello'),
    ('2026-01-01 00:00:02', [1, 2, 3]),
    ('2026-01-01 00:00:03', NULL);

-- The type of each value is remembered.
SELECT payload, dynamicType(payload) FROM events;

The dynamicType function is the user-visible face of the discriminator: it reads the tag and maps it to the type name.

Trade-offs

What the design buys you.

Each alternative is stored in its native layout, so a String alternative compresses like a string column, an UInt64 alternative compresses like a UInt64 column, and the whole thing benefits from per-column codecs. A Dynamic column that only ever saw strings is physically almost identical to a LowCardinality(String) or a plain String column — one byte of tag per row plus the payloads.

The global/local split means adding a type to a Dynamic column is O(1) in the number of rows. Nothing is rewritten; only two small mapping arrays change length.

What it costs you.

Nine bytes per row of pure overhead (one discriminator, one UInt64 offset), versus zero for a plain column. For a Variant with two alternatives and short values, that overhead can exceed the payload.

The offsets are the expensive part, and they’re the price of the dense sub-column layout. A “one big union buffer with a tag” design would pay one byte per row but would force every alternative into a common representation, losing per-type compression and per-type codecs.

The escape hatch is a real cost. Once a Dynamic column overflows into the shared variant, those values become opaque byte strings with an embedded type name. Reading one back requires decoding the type and then running the right deserializer — a dynamic dispatch on data, with a checkStackSize() guard because the nesting depth is attacker-controlled. And the shared variant is a String column, so it compresses like text even when it holds binary payloads.

The complexity is concentrated in one place. The mapping from global to local discriminator is the only genuinely subtle piece, and it exists purely so that Variant(T1, T2) and Variant(T2, T1) can be the same type while keeping the on-disk tags stable across extensions. Everything else — the offsets, the compact serialization, the NULL discriminator — is a fairly direct consequence of “one sub-column per alternative, one tag per row”.

Wrapping up

Worth internalizing: the local/global split is the trick that makes Dynamic viable. Store global tags and every extension rewrites the column. Store local tags and extension is a metadata edit.

What to go do about it: if you use Dynamic, watch how many distinct types actually show up in your data. Once a column overflows into the shared variant, you lose per-type compression and pay a runtime type dispatch on every read. If you know the shape of your data, declare a Variant with the alternatives you need — or better, a plain column — and you keep the per-type codecs and pay zero bytes of tag.

And if you’re benchmarking, remember the overhead is 9 bytes per row. For short values, that’s not a rounding error — it’s the whole payload.

ClickhousePerformanceInternalsMemoryTypes