How ClickHouse Writes the Native Format, Column at a Time

TL;DR: A Native-format block is: a column count, a row count, then for each column a name, a type, a “serialization kind” byte, and the column’s bytes. The column’s data is written whole, by the type’s own serialiser, into a single buffer — so UInt64 values are memcpy’d out of a ColumnVector64, and a Nullable column writes its null mask and then its values, with no per-row work at all.

The single most expensive thing a database does wrong is take data it already has in columnar form and turn it into rows, so that it can turn it back into columns on the other side. That transposition is what RowBinary does, and what CSV does, and it is why importing a CSV into ClickHouse costs more CPU than the query you then run on it.

The Native format exists to avoid the round trip entirely. Its own documentation says so:

cpp
// src/Processors/Formats/Impl/NativeFormat.cpp:145
The `Native` format is ClickHouse's most efficient format because it is truly "columnar" 
in that it does not convert columns to rows.  

In this format data is written and read by [blocks](/resources/develop-contribute/introduction/architecture#block) in a binary format. 
For each block, the number of rows, number of columns, column names and types, and parts of columns in the block are recorded one after another. 

This article is about how that is actually implemented: the block layout, the “one buffer per column” trick that makes the bulk serialisers fast, the serialization-kind negotiation that lets the reader know what it is about to read, and the index that lets a reader skip columns it does not want.

The block layout

Here is a real Native block — two columns, three rows — dumped as bytes:

cpp
// src/Processors/Formats/Impl/NativeFormat.cpp:182
const data = new Uint8Array([
  // --- Block Header ---
  0x02,                   // 2 columns
  0x03,                   // 3 rows
  // -- Column 1 Header --
  0x06,                   // LEB128 - column name 'number' has 6 bytes
  0x6e, 0x75, 0x6d,       
  0x62, 0x65, 0x72,       // column name: 'number'
  0x06,                   // LEB128 - column type 'UInt64' has 6 bytes
  0x55, 0x49, 0x6e,
  0x74, 0x36, 0x34,       // 'UInt64'
  0x00, 0x00, 0x00, 0x00, 
  0x00, 0x00, 0x00, 0x00, // 0 as UInt64
  0x01, 0x00, 0x00, 0x00, 
  0x00, 0x00, 0x00, 0x00, // 1 as UInt64
  0x02, 0x00, 0x00, 0x00, 
  0x00, 0x00, 0x00, 0x00, // 2 as UInt64
  0x03,                   // LEB128 - column name 'str' has 3 bytes
  0x73, 0x74, 0x72,       // column name: 'str'
  0x06,                   // LEB128 - column type 'String' has 6 bytes
  0x53, 0x74, 0x72, 
  0x69, 0x6e, 0x67,       // 'String'
  0x01,                   // LEB128 - the string has 1 byte

Three observations:

The header is columns then rows, both LEB128 varints.
Column 1’s data comes before column 2’s header. The layout is column-major, not row-major: name, type, data, name, type, data.
The data of a UInt64 column is 8 bytes per value, in order, with no framing at all. The row count in the header is the only thing that says how many values there are.

That third point is the whole point. A ColumnVector64 is an array of UInt64 in memory. Writing it is a memcpy. Reading it is a memcpy into a freshly reserved ColumnVector64. There is no loop over rows anywhere.

The writer

The whole block is written by one function:

cpp
// src/Formats/NativeWriter.cpp:130
size_t NativeWriter::write(const Block & block)
{
    size_t written_before = ostr.count();

    /// Additional information about the block.
    if (client_revision > 0)
        block.info.write(ostr, client_revision);

    block.checkNumberOfRows();

    /// Dimensions
    size_t columns = block.columns();
    size_t rows = block.rows();

    writeVarUInt(columns, ostr);
    writeVarUInt(rows, ostr);

then a loop over columns, and inside it, name, type, serialization info, data:

cpp
// src/Formats/NativeWriter.cpp:184
        /// Name
        writeStringBinary(column.name, ostr);

cpp
// src/Formats/NativeWriter.cpp:202
        /// Type
        if (format_settings && format_settings->native.encode_types_in_binary_format)
        {
            encodeDataType(column.type, ostr);
        }
        else
        {
            String type_name = column.type->getName();

            /// For compatibility, we will not send explicit timezone parameter in DateTime data type
            ///  to older clients, that cannot understand it.
            if (client_revision < DBMS_MIN_REVISION_WITH_TIME_ZONE_PARAMETER_IN_DATETIME_DATA_TYPE
                && startsWith(type_name, "DateTime("))
                type_name = "DateTime";

            writeStringBinary(type_name, ostr);
        }

Two options for the type: a name ("UInt64", "Nullable(UInt64)") or a binary encoding of the type, toggled by output_format_native_encode_types_in_binary_format. The name is the default because it is human-readable and because the reader can resolve it through the DataTypeFactory. The binary encoding exists because the name is ambiguous for types with parameters — Decimal(10,2) and Decimal(38,10) both have parameters that the name encodes as text.

sql
-- Write the type of every column as a binary encoding instead of a name.
SELECT * FROM numbers(3) SETTINGS output_format_native_encode_types_in_binary_format = 1;

-- Read it back.
SELECT * FROM file('block.bin', Native) SETTINGS input_format_native_decode_types_in_binary_format = 1;

And then the serialization kind:

cpp
// src/Formats/NativeWriter.cpp:220
        /// Serialization. Dynamic, if client supports it.
        SerializationPtr serialization;
        {
            SerializationInfoPtr info;
            std::tie(serialization, info, column.column) = getSerializationAndColumn(client_revision, column);
            if (info)
            {
                writeBinary(static_cast<UInt8>(info->hasCustomSerialization()), ostr);
                if (info->hasCustomSerialization())
                    info->serialializeKindStackBinary(ostr);
            }
        }

        /// Data
        if (rows)    /// Zero items of data is always represented as zero number of bytes.
            writeData(*serialization, column.column, ostr, format_settings, 0, 0, client_revision);

That single byte plus an optional kind stack is the interesting part, and it is covered below.

One buffer per column

How does the column’s data actually get written? Not by NativeWriter — by the type’s serialiser, which knows the layout of that type. The handoff is a callback:

cpp
// src/DataTypes/Serializations/ISerialization.h:471
    using OutputStreamGetter = std::function<WriteBuffer*(const SubstreamPath &)>;

The serialiser pushes a substream path onto a stack and asks the getter for a buffer to write that substream into. In a MergeTree part, different substreams of the same column go to different files (the null map of a Nullable column is a separate .null.bin file); in the Native format, they all go into the same buffer:

cpp
// src/Formats/NativeWriter.cpp:60
/*static*/ void NativeWriter::writeData(
    const ISerialization & serialization,
    const ColumnPtr & column,
    WriteBuffer & ostr,
    const std::optional<FormatSettings> & format_settings,
    UInt64 offset,
    UInt64 limit,
    UInt64 client_revision)
{
    /** If there are columns-constants - then we materialize them.
      * (Since the data type does not know how to serialize / deserialize constants.)
      * The same for compressed columns in-memory.
      */
    ColumnPtr full_column = column->convertToFullColumnIfConst()->decompress();

    ISerialization::SerializeBinaryBulkSettings settings;
    settings.getter = [&ostr](ISerialization::SubstreamPath) -> WriteBuffer * { return &ostr; };
    settings.position_independent_encoding = false;
    settings.low_cardinality_max_dictionary_size = 0;
    settings.native_format = true;

Look at that getter: it ignores the path entirely and returns the same WriteBuffer for every substream. So all of a column’s substreams are concatenated into one contiguous byte run.

Here is what that looks like for a Nullable, which has two substreams — the null map and the nested values:

cpp
// src/DataTypes/Serializations/SerializationNullable.cpp:114
void SerializationNullable::serializeBinaryBulkWithMultipleStreams(
    const IColumn & column,
    size_t offset,
    size_t limit,
    SerializeBinaryBulkSettings & settings,
    SerializeBinaryBulkStatePtr & state) const
{
    const ColumnNullable & col = assert_cast<const ColumnNullable &>(column);
    col.checkConsistency();

    if (!use_default_null_map)
    {
        /// First serialize null map.
        settings.path.push_back(Substream::NullMap);
        if (auto * stream = settings.getter(settings.path))
            SerializationNumber<UInt8>::create()->serializeBinaryBulk(col.getNullMapColumn(), *stream, offset, limit);
        settings.path.pop_back();
    }

    /// Then serialize contents of elements.
    settings.path.push_back(Substream::NullableElements);
    nested->serializeBinaryBulkWithMultipleStreams(col.getNestedColumn(), offset, limit, settings, state);
    settings.path.pop_back();
}

The null map is a ColumnUInt8 — one byte per row — and it is written by SerializationNumber<UInt8>, which is a bulk memcpy of the column’s data. So a Nullable(UInt64) column is: N bytes of mask, then 8N bytes of values. No row loop, no branch per row.

The same machinery produces the Array layout: a stream of UInt64 offsets, then a stream of all the nested elements concatenated. The Variant type writes a discriminator byte per row, then one bulk stream per alternative containing only the rows that belong to it.

Why `position_independent_encoding = false`

That flag matters. “Position-independent encoding” means the serialiser may choose an encoding that depends on where the data ends up (offsets relative to the block start, for example). It is turned off for the Native format because the format must be byte-identical regardless of which buffer the data lands in — the same block written to a file and sent over the wire has to be the same bytes.

The serialization kind

A column’s type does not uniquely determine its byte layout. UInt64 can be written as a plain array, or as a sparse array (mostly zeros, stored as a list of non-default positions), or with a custom serialisation chosen by the MergeTree settings. The reader has to know which one the writer used, and it cannot infer it.

ClickHouse’s answer is a kind stack — a list of serialisation kinds, one per level of nesting:

cpp
// src/DataTypes/Serializations/SerializationInfo.h:64
    virtual bool hasCustomSerialization() const { return kind_stack.size() > 1; }

If the stack has one element — the default serialisation — nothing extra is written. If it has more, the writer emits a 1 byte followed by the stack:

cpp
// src/Formats/NativeWriter.cpp:225
            if (info)
            {
                writeBinary(static_cast<UInt8>(info->hasCustomSerialization()), ostr);
                if (info->hasCustomSerialization())
                    info->serialializeKindStackBinary(ostr);
            }

So the wire format is: one byte, usually zero. A column with a custom serialisation carries its recipe inline. There is no negotiation round trip, no “capability” exchange — the writer simply describes what it did, and the reader either understands the description or refuses the data.

The reader mirrors it:

cpp
// src/Formats/NativeReader.cpp:222
        if (server_revision >= DBMS_MIN_REVISION_WITH_CUSTOM_SERIALIZATION)
        {
            /// NativeReader must enable all supported serializations (e.g. nullable sparse) here. Since it operates on
            /// in-memory state, it should be able to handle all possible serialization variants.
            auto info = column.type->createSerializationInfo(SerializationInfoSettings::enableAllSupportedSerializations(
                server_revision >= DBMS_MIN_REVISION_WITH_STRING_WITH_SIZE_STREAM_SERIALIZATION));

            UInt8 has_custom = 0;
            readBinary(has_custom, istr);
            if (has_custom)
                info->deserializeFromKindsBinary(istr, allowed_kinds);

            serialization = column.type->getSerialization(*info);
            auto new_column = column.type->createColumn(*serialization);
            new_column->reserve(rows_to_reserve);
            read_column = std::move(new_column);
        }

Note allowed_kinds: the reader passes a set of serialisation kinds it is willing to accept, and deserializeFromKindsBinary rejects anything outside it. A kind that the reader cannot handle is invalid data, not a silent misparse. That matters for a format that travels over the network between different server versions.

The reader

The read side is the mirror image, and it is worth looking at closely because of one detail: how much it trusts the header.

cpp
// src/Formats/NativeReader.cpp:165
    if (!use_index)
    {
        readVarUInt(columns, istr);
        readVarUInt(rows, istr);

        if (columns > DEFAULT_NATIVE_BINARY_MAX_NUM_COLUMNS)
            throw Exception(ErrorCodes::TOO_LARGE_ARRAY_SIZE, "Suspiciously many columns in Native format: {}", columns);
        if (rows > DEFAULT_NATIVE_BINARY_MAX_NUM_ROWS)
            throw Exception(ErrorCodes::TOO_LARGE_ARRAY_SIZE, "Suspiciously many rows in Native format: {}", rows);
    }

The row count comes from the stream, so it is attacker-controlled. The bounds check above stops the absurd cases, but there is a second, subtler guard:

cpp
// src/Formats/NativeReader.cpp:184
    /// `rows` comes from the block header, and the limit it is checked against is deliberately
    /// generous, so it must not be used to preallocate the columns: a header declaring a huge row
    /// count would reserve that much per column before a single byte of column data is read.
    /// Reserving is only an optimization here - deserialization appends to the column anyway - so
    /// bound it by a plausible block size and let the column grow past that on its own.
    const size_t rows_to_reserve = std::min<size_t>(rows, DEFAULT_INSERT_BLOCK_SIZE);

A malicious header claiming 4 billion rows would make a naive reader reserve() 32 GB per column before reading a single byte. The fix is to reserve only a plausible block size and let the column grow on its own — the reserve is an optimisation, not a contract.

Then, for each column, the reader reads the name, the type, and the data:

cpp
// src/Formats/NativeReader.cpp:199
        ColumnWithTypeAndName column;

        /// Name
        readBinary(column.name, istr);

        /// Type
        String type_name;
        if (format_settings && format_settings->native.decode_types_in_binary_format)
        {
            column.type = decodeDataType(istr, format_settings->binary.max_binary_type_complexity);
            type_name = column.type->getName();
        }
        else
        {
            readBinary(type_name, istr);
            column.type = data_type_factory.get(type_name);
        }

and finally the data:

cpp
// src/Formats/NativeReader.cpp:255
        /// If no rows, nothing to read.
        if (rows)
        {
            const auto * format = format_settings ? &*format_settings : nullptr;
            NameAndTypePair name_and_type = {column.name, column.type};
            readData(*serialization, *read_column, istr, format, rows, &name_and_type, &avg_value_size_hints);
        }

The header match

The Native format is self-describing — the stream says what its columns are — but it is also often read into a known schema. When a header is supplied, the reader matches by name, and then has to reconcile the types:

cpp
// src/Formats/NativeReader.cpp:275
                if (!header_column.type->equals(*column.type))
                {
                    /// In the event of the same aggregate function but of a different variant (e.g. Window vs Aggregate),
                    /// we should try to convert and read, since the difference in `Window` vs `Aggregate` is not a
                    /// user-facing type difference but rather an internal implementation detail.
                    /// This can happen when external sort spills blocks to disk: the header carries the Window variant from the query plan,
                    /// but `NativeReader` deserializes the type name and resolves it via `AggregateFunctionFactory`, which always produces the
                    /// Aggregation variant.
                    const auto * header_agg_type = typeid_cast<const DataTypeAggregateFunction *>(header_column.type.get());
                    bool convertible_agg_variant = header_agg_type && header_agg_type->equalsIgnoringVariant(*column.type);

                    if ((format_settings && format_settings->native.allow_types_conversion) || convertible_agg_variant)
                    {

So: equal types → use them; convertible types (including the Window/Aggregate variant case, which is an implementation detail rather than a user-facing difference) → cast the column; otherwise → fall back to a low-cardinality conversion, which is the historical “old client, no LowCardinality type” path.

And columns in the stream that are not in the header can be skipped — the reader simply does not insert them:

cpp
// src/Formats/NativeReader.cpp:319
            else
            {
                if (format_settings && !format_settings->skip_unknown_fields)
                    throw Exception(ErrorCodes::INCORRECT_DATA, "Unknown column with name {} found while reading data in Native format", column.name);
                use_in_result = false;
            }

while columns in the header that were not in the stream are filled with defaults, and the missing-ness is recorded:

cpp
// src/Formats/NativeReader.cpp:344
    if (rows && !header.empty())
    {
        /// Allow to skip columns. We will fill them with default values later.
        Block tmp_res;

        for (size_t column_i = 0; column_i != header.columns(); ++column_i)
        {
            auto & col = header.getByPosition(column_i);
            if (res.has(col.name))
            {
                tmp_res.insert(res.getByName(col.name));
            }
            else
            {
                tmp_res.insert({col.type->createColumn()->cloneResized(rows), col.type, col.name});
                if (block_missing_values)
                    block_missing_values->setBits(column_i, rows);
            }
        }

block_missing_values is a bitmap of which columns were absent — it travels with the block so that downstream transforms can tell “this column was missing” from “this column was NULL”.

The index: skipping columns

The Native format can carry a separate index that says where each column lives:

cpp
// src/Formats/IndexForNativeFormat.h:12
/** The Native format can contain a separately located index,
  *  which allows you to understand where what column is located,
  *  and skip unnecessary columns.
  */

Each entry is a mark — the position of the column in the compressed file:

cpp
// src/Formats/MarkInCompressedFile.h:14
/** Mark is the position in the compressed file. The compressed file consists of adjacent compressed blocks.
  * Mark is a tuple - the offset in the file to the start of the compressed block, the offset in the decompressed block to the start of the data.
  */
struct MarkInCompressedFile
{
    size_t offset_in_compressed_file;
    size_t offset_in_decompressed_block;

The writer emits one mark per column, and to do that it has to flush the compressed block at every column boundary:

cpp
// src/Formats/NativeWriter.cpp:162
    for (size_t i = 0; i < columns; ++i)
    {
        /// For the index.
        MarkInCompressedFile mark{0, 0};

        if (index)
        {
            ostr_concrete->next();  /// Finish compressed block.
            mark.offset_in_compressed_file = initial_size_of_file + ostr_concrete->getCompressedBytes();
            mark.offset_in_decompressed_block = ostr_concrete->getRemainingBytes();
        }

and the reader uses the mark to seek, skipping every column it does not want:

cpp
// src/Formats/NativeReader.cpp:191
    for (size_t i = 0; i < columns; ++i)
    {
        if (use_index)
        {
            /// If the current position is what is required, the real seek does not occur.
            istr_concrete->seek(index_column_it->location.offset_in_compressed_file, index_column_it->location.offset_in_decompressed_block);
        }

Note the comment: “If the current position is what is required, the real seek does not occur.” Reading column 0 when you only want column 1 still costs a readBinary of the name and the type and then a seek — you cannot skip within a column’s bytes, only between columns.

Trade-offs

What the format buys:

No transposition. A UInt64 column is copied, not iterated. This is the difference between an import that costs 5% of the file size in CPU and one that costs 200%.
Bulk, not scalar. Every serialiser has a “bulk” path that operates on whole columns. SerializationNumber<UInt64> writes the column’s memory directly.
Self-describing but also schema-aware. The stream says what it contains, and a reader with a schema can additionally skip columns and reconcile types.
A one-byte negotiation. The serialisation kind is described inline, so there is no handshake and no version dance for the common case.

What it costs:

It is not a stable interchange format. The type names round-trip through DataTypeFactory, so a rename or a parameter change breaks compatibility. The binary type encoding exists to mitigate this but is not the default. ClickHouse’s own documentation is blunt about it: “It might not be practical to work with this format yourself.”
Version negotiation is threaded through everything. client_revision appears in the writer, the reader, the serialisation info, the aggregate function state versions, the DateTime timezone parameter. A format that has to survive a decade of client versions accumulates conditionals.
Column-at-a-time hurts some things. If the consumer wants rows — say, an application reading the Native format to emit JSON — it now has to transpose, which is exactly what the format was designed to avoid. The format optimises for the case where the consumer is also columnar, which in practice means another ClickHouse.
The index costs a flush per column. Emitting a mark per column forces the compressed stream to be cut at every column boundary, which costs compression ratio. It is opt-in.
Type resolution is a string lookup. data_type_factory.get(type_name) is a hash lookup per column per block; for a 10-column block read a million times, that is not free — but it is amortised over 65 409 rows.

The design is a good illustration of a general principle: a format’s cost is dominated by the mismatch between its layout and the consumer’s. ClickHouse stores columns; the Native format speaks columns; the transposition never happens.