Opensource internals

ClickHouse Stores a Part's Metadata in Plain Text Files Beside the Data

21 min read Jay Hubbert

A MergeTree part is a directory of column files plus a handful of human-readable text sidecars - columns.txt, checksums.txt, count.txt, uuid.txt - each written atomically and re-read at load time.

TL;DR: A MergeTree part is a directory. Inside it, the data is a pile of .bin/.mrk files, and the metadata is a pile of small text files: columns.txt, columns_substreams.txt, checksums.txt, count.txt, uuid.txt, metadata_version.txt, serialization.json. They are written atomically (temp file → fsync → rename), parsed back at load time, and then shared between parts in RAM so that a thousand parts cost one copy.

Every columnar database keeps some description of its data files next to the data. The interesting question is not whether — it is what format, and when it is read.

ClickHouse has a choice here that most engines do not: it could store the schema of a part in a binary blob, parsed with a hand-written deserializer, the way it stores column data. Or it could store it as text.

It stores it as text. And the consequences of that decision run through the entire MergeTree implementation.

What a part directory actually contains

The list of files that belong to a part’s metadata — as opposed to its data — is written down once, in one place, because several subsystems need to agree on it (the replication protocol, the backup code, the orphan-part sweeper):

cpp 
// src/Storages/MergeTree/IMergeTreeDataPart.h:851
    inline static const Strings COMMON_METADATA_FILES =
    {
        "uuid.txt",
        "checksums.txt",
        "columns.txt",
        "columns_substreams.txt",
        "count.txt",
        "metadata_version.txt",
        "default_compression_codec.txt",
        "serialization.json",
        "partition.dat",
        "ttl.txt",
    };

Ten names. Six of them are plain text (uuid.txt, columns.txt, columns_substreams.txt, count.txt, metadata_version.txt, default_compression_codec.txt), one is JSON (serialization.json), and three are binary (checksums.txt, partition.dat, ttl.txt). A typical part directory looks like this:

text 
store/3a8/3a83f2c1-.../all_1_7_0/
├── columns.txt                  <- the physical column list: names + types
├── columns_substreams.txt       <- which substreams each column was written as
├── checksums.txt                <- size + SipHash-128 of every file
├── count.txt                    <- the row count, as a decimal integer
├── uuid.txt                     <- the part's UUID
├── metadata_version.txt         <- the table's metadata version at write time
├── default_compression_codec.txt<- "LZ4" (or whatever the default was)
├── serialization.json           <- per-column serialization kinds (Sparse etc.)
├── partition.dat                <- the partition tuple, binary
├── ttl.txt                      <- the TTL expressions and their min/max
├── primary.idx                  <- data
├── primary.mrk                  <- data
├── event_date.mrk2              <- data
├── event_date.bin               <- data
├── ...

Only the first ten are metadata. The rest is data.

columns.txt: the physical column list, and nothing else

Here is the whole writer:

cpp 
// src/Core/NamesAndTypes.cpp:115
void NamesAndTypesList::writeText(WriteBuffer & buf) const
{
    writeString("columns format version: 1\n", buf);
    DB::writeText(size(), buf);
    writeString(" columns:\n", buf);
    for (const auto & it : *this)
    {
        writeBackQuotedString(it.name, buf);
        writeChar(' ', buf);
        writeString(it.type->getName(), buf);
        writeChar('\n', buf);
    }
}

And the whole reader:

cpp 
// src/Core/NamesAndTypes.cpp:90
void NamesAndTypesList::readText(ReadBuffer & buf, bool check_eof)
{
    const DataTypeFactory & data_type_factory = DataTypeFactory::instance();

    assertString("columns format version: 1\n", buf);
    size_t count = 0;
    DB::readText(count, buf);
    assertString(" columns:\n", buf);

    String column_name;
    String type_name;
    for (size_t i = 0; i < count; ++i)
    {
        readBackQuotedStringWithSQLStyle(column_name, buf);
        assertChar(' ', buf);
        readString(type_name, buf);
        assertChar('\n', buf);

        emplace_back(column_name, data_type_factory.get(type_name));
    }

    if (check_eof)
        assertEOF(buf);
}

There is no length prefix, no offset table, no varint. A file on disk looks like this:

text 
columns format version: 1
5 columns:
`event_date` Date
`event_time` DateTime
`user_id` UInt64
`url` String
`referer` String

Which is worth pausing on, because it is a deliberate narrowing of what a part records. It is tempting to imagine columns.txt as a snapshot of the table’s schema: the column names, the types, the compression codec, the DEFAULT/MATERIALIZED kind, the default expression, the TTL, the comment. It is not. A part records only the physical columns it actually stores, and their types. Everything else — the codec, the default expression, the column TTL, the comment — lives in the table’s metadata (metadata/<db>.<table>.sql), not in the part.

That is a real design decision, and it is the reason ALTER TABLE ... MODIFY COLUMN does not have to rewrite parts. A part’s columns.txt is a statement about bytes on disk, not about schema. When you add a column, old parts simply do not have it, and the reader substitutes defaults. When you change a codec, old parts keep the codec they were written with, and the codec pipeline in the data file’s frames says so.

This is also why system.parts_columns reports default_kind and default_expression from the table metadata, not from the part:

cpp 
// src/Storages/System/StorageSystemPartsColumns.cpp:110
    auto metadata_snapshot = info.storage->getInMemoryMetadataPtr(context, false);
    size_t metadata_column_number = 0;
    for (const auto & column : metadata_snapshot->getColumns())
    {
        /// The prepass alone can take a long time on a table with many columns.
        /// A partially filled `columns_info` would report wrong defaults, so give up the whole storage instead.
        ++metadata_column_number;
        slowDownSystemPartsMetadataEnumeration(info.table, metadata_column_number);
        if (query_status && metadata_column_number % COLUMNS_CANCELLATION_CHECK_PERIOD == 0 && !query_status->checkTimeLimit())
            return;

        ColumnInfo column_info;
        if (column.default_desc.expression)
        {
            column_info.default_kind = toString(column.default_desc.kind);
            column_info.default_expression = column.default_desc.expression->formatForLogging();
        }

        columns_info[column.name] = column_info;
    }

Note the prepass: the defaults are computed once per table, not once per part per column. The part only contributes its column list; the table contributes the semantics. So the two views of a column’s metadata look like this:

sql 
-- The table's metadata: the semantic definition of the column.
SELECT name, type, default_kind, default_expression, compression_codec
FROM system.columns
WHERE database = 'default' AND table = 'events';

-- The part's metadata: the physical bytes that exist on disk.
SELECT part_name, column, type, column_bytes_on_disk
FROM system.parts_columns
WHERE database = 'default' AND table = 'events' AND active;

The first query is about the schema; the second is about the storage. A column added yesterday appears in the first query and in the second query only for parts created since.

checksums.txt: a compressed varint table

checksums.txt is the metadata file that is not text you would want to cat. It is written in format version 4, which is a compressed binary encoding:

cpp 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:247
void MergeTreeDataPartChecksums::write(WriteBuffer & to) const
{
    writeString("checksums format version: 4\n", to);

    CompressedWriteBuffer out{to, CompressionCodecFactory::instance().getDefaultCodec(), 1 << 16};

    writeVarUInt(files.size(), out);

    for (const auto & [name, sum] : files)
    {
        writeStringBinary(name, out);
        writeVarUInt(sum.file_size, out);
        writeBinaryLittleEndian(sum.file_hash, out);
        writeBinaryLittleEndian(sum.is_compressed, out);

        if (sum.is_compressed)
        {
            writeVarUInt(sum.uncompressed_size, out);
            writeBinaryLittleEndian(sum.uncompressed_hash, out);
        }
    }

    out.finalize();
}

So the file is: a text version banner, then a compressed stream of (name, size, hash) triples. The older formats (v2) were human-readable; v3 switched to binary; v4 added the compression. The reader is symmetric:

cpp 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:160
bool MergeTreeDataPartChecksums::read(ReadBuffer & in)
{
    files.clear();

    assertString("checksums format version: ", in);
    size_t format_version = 0;
    readText(format_version, in);
    assertChar('\n', in);

    read(in, format_version);
    return true;
}

and the version dispatch is explicit, because a part written by a 2019 server must still load on a 2026 one:

cpp 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:143
bool MergeTreeDataPartChecksums::read(ReadBuffer & in, size_t format_version)
{
    switch (format_version)
    {
        case 1:
            return false;
        case 2:
            return readV2(in);
        case 3:
            return readV3(in);
        case 4:
            return readV4(in);
        default:
            throw Exception(ErrorCodes::UNKNOWN_FORMAT, "Bad checksums format version: {}", DB::toString(format_version));
    }
}

readV2 is the old text format, still supported:

cpp 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:173
bool MergeTreeDataPartChecksums::readV2(ReadBuffer & in)
{
    size_t count = 0;

    readText(count, in);
    assertString(" files:\n", in);

    for (size_t i = 0; i < count; ++i)
    {
        String name;
        Checksum sum;

        readString(name, in);
        assertFileNameIsRelativeAndContained(name);
        assertString("\n\tsize: ", in);
        readText(sum.file_size, in);
        assertString("\n\thash: ", in);
        readText(sum.file_hash.low64, in);
        assertString(" ", in);
        readText(sum.file_hash.high64, in);
        assertString("\n\tcompressed: ", in);
        readText(sum.is_compressed, in);

Notice the assertFileNameIsRelativeAndContained(name) call. File names in checksums.txt are used verbatim to build paths inside the part directory (when checking sizes, when removing the part), so a hostile or corrupted checksums.txt must not be able to escape it:

cpp 
// src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp:120
void assertFileNameIsRelativeAndContained(const String & name)
{
    if (name.empty())
        throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART, "Empty file name in checksums of data part");

    if (name.starts_with('/'))
        throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART, "Absolute file name '{}' in checksums of data part", name);

    /// A NUL byte would be kept inside a single path component here, but the local disk layer
    /// passes the joined path to C APIs which truncate at the first NUL, so "..\0/x" would act as "..".
    if (name.contains('\0'))
        throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART, "File name '{}' in checksums of data part contains a NUL byte", name);

    for (const auto & component : std::filesystem::path(name))
    {
        if (component == "." || component == "..")
            throw Exception(ErrorCodes::UNEXPECTED_FILE_IN_DATA_PART,
                "File name '{}' in checksums of data part contains '{}' path component", name, component.string());
    }
}

This is a nice illustration of what “the metadata is just a file on disk” costs you: the file is trusted input from the point of view of the storage layer, and every consumer of it has to validate it. The .. check, the NUL check, the absolute-path check — three lines of paranoia that exist only because the format is a text file that anyone with write access to the data directory can edit.

The other sidecars

columns_substreams.txt records, for every column, the list of substreams it was written as (a LowCardinality(String) column, for instance, is written as three substreams: the dictionary, the dictionary size, and the index):

cpp 
// src/Storages/MergeTree/ColumnsSubstreams.cpp:208
void ColumnsSubstreams::writeText(WriteBuffer & buf) const
{
    writeString("columns substreams version: 1\n", buf);
    DB::writeText(columns_substreams.size(), buf);
    writeString(" columns:\n", buf);
    for (const auto & entry : columns_substreams)
    {
        DB::writeText(entry->substreams.size(), buf);
        writeString(" substreams for column ", buf);
        writeBackQuotedString(entry->column, buf);
        writeCString(":\n", buf);
        for (const auto & substream : entry->substreams)
        {
            writeChar('\t', buf);
            writeString(substream, buf);
            writeChar('\n', buf);
        }
    }
}

count.txt is the row count, and it is astonishingly small — a single decimal integer:

cpp 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:410
    write_hashed_file("count.txt", [&](auto & buffer)
    {
        writeIntText(rows_count, buffer);
    });

uuid.txt is the part’s UUID, written only if one was assigned:

cpp 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:360
    if (!new_part->isProjectionPart())
    {
        if (new_part->uuid != UUIDHelpers::Nil)
        {
            write_hashed_file(IMergeTreeDataPart::UUID_FILE_NAME, [&](auto & buffer)
            {
                writeUUIDText(new_part->uuid, buffer);
            });
        }

Note write_hashed_file vs write_plain_file. count.txt, ttl.txt, serialization.json and uuid.txt are hashed — they appear in checksums.txt and are therefore covered by the part’s integrity check. columns.txt, columns_substreams.txt, metadata_version.txt, default_compression_codec.txt and checksums.txt itself are plain — they are not in checksums.txt, because checksums.txt cannot checksum itself, and because columns.txt must remain readable even when the checksums are lost (see below).

The distinction matters. Here is the full set of writes at part finalization:

cpp 
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:450
    write_plain_file("columns.txt", [&](auto & buffer)
    {
        new_part->getColumns().writeText(buffer);
    });

    /// Merge columns substreams from current writer and additional columns substreams
    /// from other writers (that could be used during vertical merge).
    /// If there are no additional columns substreams we still need to call merge
    /// so we will keep only columns that are present in new_part->getColumns().
    /// It may happen that new_part->getColumns() has less columns then columns substreams
    /// from writer because of expired TTL.
    auto columns_substreams = ColumnsSubstreams::merge(
        writer->getColumnsSubstreams(),
        gathered_data.columns_substreams,
        new_part->getColumns().getNames());

    if (!columns_substreams.empty())
    {
        write_plain_file(IMergeTreeDataPart::COLUMNS_SUBSTREAMS_FILE_NAME, [&](auto & buffer)
        {
            columns_substreams.writeText(buffer);
        });

        new_part->setColumnsSubstreams(columns_substreams);
    }

Writing them: atomicity is not optional

A part directory is created, filled, and then renamed into place — the visibility trick that another article covers. But the metadata files themselves are written after that rename, in some cases, because they describe a part that is already live. So they cannot rely on an enclosing rename for atomicity, and they get their own:

cpp 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2280
void IMergeTreeDataPart::writeMetadata(const String & filename, const WriteSettings & settings, Writer && writer)
{
    auto & data_part_storage = getDataPartStorage();

    /// Rewrite the metadata file atomically and durably (write to a temp file, fsync, rename),
    /// because it goes into an already-committed part directory with no enclosing rename to rely
    /// on. An interrupted in-place write could otherwise leave an empty checksums.txt, which
    /// bricks the whole part. Mirrors VersionMetadataOnDisk::storeInfoToDataPartStorage.
    /// The temp name carries the thread id so two concurrent repairs of the same part (from
    /// separate CHECK TABLE queries) do not rename each other's half-written file. It keeps the
    /// .tmp suffix so checkDataPart skips it if a crash leaves it behind.
    const String tmp_filename = fmt::format("{}.{}.tmp", filename, getThreadId());

    try
    {
        auto out = data_part_storage.writeFile(tmp_filename, 4096, settings);
        writer(*out);
        out->finalize();
        out->sync();

        SyncGuardPtr sync_guard;
        if ((*storage.getSettings())[MergeTreeSetting::fsync_part_directory])
            sync_guard = data_part_storage.getDirectorySyncGuard();
        data_part_storage.replaceFile(tmp_filename, filename);
    }
    catch (...)
    {
        try
        {
            data_part_storage.removeFileIfExists(tmp_filename);
        }
        catch (...)
        {
            tryLogCurrentException(storage.log, __PRETTY_FUNCTION__);
        }
        throw;
    }
}

Temp file named <file>.<thread_id>.tmp, written, finalize()d, sync()ed (a real fsync), and only then rename()d over the target. The thread id is in the temp name so that two concurrent CHECK TABLE queries repairing the same part do not rename each other’s half-written file. The .tmp suffix is kept so that checkDataPart skips it if a crash leaves it behind.

Two details worth calling out:

  1. finalize() before sync(). For a compressed output stream, finalize() flushes the compressed frame and writes the checksum footer. Calling sync() first would sync a half-written compressed stream.
  2. The directory sync is conditional. fsync_part_directory is a setting; without it, the rename is durable but the directory entry might not survive a power loss.

Reading them back: the load path

When a part is attached — at server startup, or on an ATTACH — the metadata is read once and cached in the part object. loadColumnsChecksumsIndexes is the entry point, and its ordering is not arbitrary:

cpp 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:1628
void IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool require_columns_checksums, bool check_consistency, bool load_metadata_version)
{
    /// Memory should not be limited during ATTACH TABLE query.
    /// This is already true at the server startup but must be also ensured for manual table ATTACH.
    /// Motivation: memory for index is shared between queries - not belong to the query itself.
    MemoryTrackerBlockerInThread temporarily_disable_memory_tracker;

    /// Long-lived per-part metadata (columns substreams, checksums, index granularity, patch part
    /// index, primary index, per-column sizes, partition / minmax index, TTL infos, projections)
    /// is routed into the dedicated parts arena by the inner block below. Deliberately kept OUT
    /// of that arena:
    ///   - `loadColumns`: its file read and text/JSON parsing are short-lived scratch; the
    ///     persistent columns/serializations it produces are arena-scoped inside `setColumns`.
    ///   - `checkConsistency`: pure file-existence/size verification, allocates nothing persistent.
    ///   - `loadDefaultCompressionCodec`: a tiny per-part codec pointer.
    /// These paths churn many short-lived allocations; keeping them in the default per-CPU arenas
    /// avoids serializing that churn on the single arena's locks under many concurrent merges.

    try
    {
        if (!isStoredOnReadonlyDisk())
            loadUUID();

        loadColumns(require_columns_checksums, load_metadata_version);

        bool has_broken_projections = false;
        {
            ScopedJemallocThreadArena mergetree_arena_scope(JemallocMergeTreeArena::getArenaIndex());

            loadColumnsSubstreams();
            loadInvalidatedSystemColumns();
            loadChecksums(require_columns_checksums);
            loadIndexGranularity();

            /// Load `source_parts.dat` before the primary index: a v2 patch's rebuilt metadata takes
            /// the sort-key columns from it, and the index cannot be deserialized without them.
            loadPatchPartIndex();

            /// It's important to load index after index granularity.
            if (!(*storage.getSettings())[MergeTreeSetting::primary_key_lazy_load])
                index = loadIndex();

            loadRowsCount(); /// Must be called after loadIndexGranularity() as it uses the value of `index_granularity`.

Read that comment: loadRowsCount() must come after loadIndexGranularity() because the row count is derived from the number of marks times the granularity when the part is in the “adaptive granularity” format. The dependency chain of the metadata files is real, and it is documented at every link.

loadColumns is where the interesting recovery logic lives. It handles the case where columns.txt is missing or empty:

cpp 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2887
void IMergeTreeDataPart::loadColumns(bool require, bool load_metadata_version)
{
    String path = fs::path(getDataPartStorage().getRelativePath()) / "columns.txt";

    NamesAndTypesList loaded_columns;
    bool is_readonly_storage = getDataPartStorage().isReadonly();

    /** A power loss can leave a file that was never `fsync`ed at length zero while the rest of the part
      * survives - the file's inode is persisted, its data block is not - and neither `columns.txt` nor
      * `metadata_version.txt` is covered by the part checksums, so nothing else notices that one of them
      * is gone. Both have a safe path for being absent: the column list is regenerated from the table
      * metadata, the version falls back to the table's. An empty file carries exactly as much as an
      * absent one, so take the same path for it instead of failing to parse it and detaching the whole
      * part - with all of its rows - as broken.
      */
    auto read_non_empty_file_if_exists = [this](const String & file_name)
    {
        auto in = readFileIfExists(file_name);
        if (in && in->eof())
            return std::unique_ptr<ReadBuffer>{};
        return in;
    };

    auto columns_file = readFileIfExists("columns.txt");
    bool columns_file_is_empty = columns_file && columns_file->eof();

    if (columns_file && !columns_file_is_empty)
    {
        loaded_columns.readText(*columns_file);

        for (auto & column : loaded_columns)
            setVersionToAggregateFunctions(column.type, true);

        if (!info.isPatch())
            attachQuantizeSerializations(loaded_columns, getMetadataSnapshot()->getColumns());
    }

This is the payoff of the “no schema in the part” decision. Because columns.txt only records which physical columns are stored, and because the part’s column list can be reconstructed from the table’s metadata cross-checked against the substreams file, an empty columns.txt is not a fatal error. The part rebuilds its own column list:

cpp 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2923
    /// If there is no column.txt or it's empty, try to recover it from the metadata.
    else
    {
        /// We can get list of columns only from columns.txt in compact parts.
        if (part_type == Type::Compact || info.isPatch())
            throw Exception(ErrorCodes::NO_FILE_IN_DATA_PART, "No columns.txt in part {}, expected path {} on disk {}",
                name, path, getDataPartStorage().getDiskName());

        auto metadata_snapshot = getMetadataSnapshot();

        /// No (or empty) columns.txt: rebuild it.

        /// Load columns substreams, so we can check if specific column exists in the data part via getFirstFileNameForColumn correctly.
        loadColumnsSubstreams(/*validate_against_loaded_columns=*/false);

and then verifies the reconstruction against columns_substreams.txt, which is a second, independent record of the same fact:

cpp 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2987
        /// loadColumnsSubstreams() requires exactly this list once loadColumns() returns; checking it
        /// here keeps the incomplete list off disk.
        auto recorded_columns = getColumnsSubstreams().getColumnNames();
        if (!recorded_columns.empty() && recorded_columns != loaded_columns.getNames())
            throw Exception(ErrorCodes::NO_FILE_IN_DATA_PART,
                "Cannot rebuild columns.txt of part {}: it stores columns [{}], the list rebuilt from the table metadata is [{}]",
                name, fmt::join(recorded_columns, ", "), fmt::join(loaded_columns.getNames(), ", "));

        if (!is_readonly_storage)
            writeColumns(loaded_columns, {});
    }

Two metadata files recording the same fact, cross-checked against each other, is how a torn write is turned into a recoverable situation rather than a detached part. The rebuilt list is written back to disk, so the next load takes the fast path.

The same trick applies to checksums.txt. If it is missing or empty, the part is re-checksummed from the data files:

cpp 
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2566
void IMergeTreeDataPart::loadChecksums(bool require)
{
    /// Arena-agnostic: the real part-load callers (`loadColumnsChecksumsIndexes`, the projection load)
    /// already run inside the parts-arena scope, while transient checksum-only probes must stay on the
    /// default arenas. So the caller picks the arena rather than pinning it here.
    auto buf = readFileIfExists("checksums.txt");

    /// An existing but empty checksums.txt is treated like an absent one (recalculated from the
    /// data files), not as fatal: a zero-byte file left by an interrupted repair would otherwise
    /// brick an intact part, which is strictly worse than the missing-file state that self-heals.
    if (buf && !buf->eof())
    {
        if (checksums.read(*buf))
        {
            assertEOF(*buf);
            bytes_on_disk = checksums.getTotalSizeOnDisk();
            bytes_uncompressed_on_disk = checksums.getTotalSizeUncompressedOnDisk();
        }
        else
            bytes_on_disk = getDataPartStorage().calculateTotalSizeOnDisk();
    }
    else
    {
        if (require)
            throw Exception(ErrorCodes::NO_FILE_IN_DATA_PART, "No checksums.txt in part {}", name);

        /// If the checksums file is not present (or empty), calculate the checksums and write
        /// them to disk. Check the data while we are at it.
        LOG_WARNING(storage.log, "Checksums for part {} not found or empty. Will calculate them from data on disk.", name);

require is require_part_metadata, set by StorageReplicatedMergeTree: a replicated part with missing metadata must not be guessed at, because the replica can re-fetch it from a healthy peer. A local MergeTree part has nowhere to re-fetch from, so it self-heals.

The metadata is read once — then it is shared

Here is the part that surprises people. When a part is loaded, its column list is turned into a NamesAndTypesList, a name→position map, a ColumnsDescription, a serialization map, and a substream list. That is a lot of memory, and it is identical for every part of a table with the same schema. A thousand parts would have a thousand copies.

ClickHouse does not do that. The bundle is interned once per table:

cpp 
// src/Storages/MergeTree/SharedPartColumns.h:21
/// Schema-derived metadata shared by all data parts of a table that store the same set of columns.
///
/// Each active data part used to carry its own copy of this metadata (`columns`, the name-to-position
/// map, the `ColumnsDescription`s, the `serializations` map and `columns_substreams`), all built in
/// `IMergeTreeDataPart::setColumns` and immutable afterwards. All of it scales with the number of
/// columns and is identical across parts with the same stored schema, so for tables with many parts
/// (and especially wide tables) the per-part copies dominated the server RSS. Instead, parts now hold
/// a `shared_ptr` to one immutable `SharedPartColumns` bundle interned in a per-`MergeTreeData` cache
/// keyed by the stored column list (see `MergeTreeData::getSharedPartColumnsForColumns`).

The interning key is not just the column list, though — NamesAndTypesList equality is not enough, because it compares types with IDataType::equals, which ignores custom names at any depth:

cpp 
// src/Storages/MergeTree/SharedPartColumns.h:129
    /// What makes two column lists interchangeable for a data part, and therefore the interning key of a
    /// bundle: the names of the columns, their full type names and the identities of the custom
    /// serializations attached to them from outside the type. `NamesAndTypesList` equality is not enough:
    /// it compares types with `IDataType::equals`, which ignores custom names at any depth
    /// (`Nullable(Bool)` against `Nullable(UInt8)`) although a part persists the name it was declared
    /// with, and no type name shows a serialization attached by a codec (`Quantized` attaches one).
    static String describeColumns(const NamesAndTypesList & columns);

And the serialization half of the bundle is interned in a second-level cache, because two parts can agree on their column list but disagree on whether a column was written sparsely:

cpp 
// src/Storages/MergeTree/SharedPartColumns.h:39
/// A serialization kind is an `ISerialization::Kind` (`Default`, `Sparse`, `Detached`, `Replicated`),
/// chosen per column when a part is written and stored in its `serialization.json`. Kinds stack
/// (`Detached` over `Sparse` over `Default`) and every subcolumn of a nested type has its own stack, so
/// a column's serialization is identified by a whole recursive kind stack.

That is the reason serialization.json exists at all. If every part always wrote every column densely, the reader could reconstruct the serialization from the type alone and the file would be unnecessary. The file exists because the choice is made per part at write time, and it must be recoverable.

The whole picture

text 
        write time                              read time (attach)
        ---------                              -----------------

  MergeTreeDataWriter                   loadColumnsChecksumsIndexes()
        |                                       |
        |  rows, columns, types                 |  loadUUID()        <- uuid.txt
        v                                       |  loadColumns()     <- columns.txt
  MergedBlockOutputStream                     |  loadColumnsSubstreams()
   finalizePartOnDisk()                       |  loadChecksums()   <- checksums.txt
        |                                      |  loadIndexGranularity()
        |  write count.txt       (hashed)       |  loadIndex()       <- primary.idx
        |  write ttl.txt         (hashed)       |  loadRowsCount()   <- count.txt
        |  write uuid.txt        (hashed)       |  loadPartitionAndMinMaxIndex()
        |  write serialization.json (hashed)    |  loadTTLInfos()    <- ttl.txt
        |  write columns.txt     (plain)        |  loadDefaultCompressionCodec()
        |  write columns_substreams.txt (plain) |  loadProjections()
        |  write metadata_version.txt  (plain)  |
        |  write default_compression_codec.txt |
        |  write checksums.txt   (plain)        v
        |                                  setColumns()
        |                                       |
        |                                       v
        |                            SharedPartColumns (interned, shared
        |                            by every part with the same schema)
        v
   part directory renamed into place

Trade-offs

What the text format buys you:

  • A part is inspectable with cat and grep. When a merge goes wrong and you want to know what a part actually contains, head columns.txt is a complete answer. There is no tool to download and no format to decode.
  • Forward compatibility is explicit. Every file starts with a version banner (columns format version: 1, checksums format version: 4, columns substreams version: 1). Adding a field is a new version, and the reader dispatches on it. There is no “unknown trailing bytes” hazard.
  • Partial corruption is recoverable. Because the metadata is small, redundant and cross-checked, a torn write to one file can be repaired from the others. A monolithic binary header has no such redundancy: one bad byte loses the whole part.
  • Cheap to write. columns.txt for a 500-column table is about 15 KB of text. It is written with a plain writeFile and no compression stage — compressing 15 KB would cost more in CPU than it saves in I/O.

What it costs:

  • Parsing at load time. Every part load parses columns.txt into DataType objects via the DataTypeFactory, and checksums.txt into a hash map. For a server with 10 000 parts, that is 10 000 parses of the same 15 KB. This is exactly why the parsed result is interned — the parse happens once per distinct schema, and the rest is a pointer copy.
  • The trust boundary. As assertFileNameIsRelativeAndContained shows, a text file that names other files is an input to path construction and has to be validated. A single binary blob that is length-prefixed and checksummed has fewer ways to be hostile.
  • Space. Text is bigger than binary. A part with 500 columns spends roughly 15 KB on columns.txt (about 30 bytes a line) and a few KB on columns_substreams.txt. For a table with 10 000 tiny parts, that is hundreds of MB of metadata — which is why columns.txt and columns_substreams.txt are not hashed into checksums.txt and why the in-memory representation is shared.
  • Self-consistency has to be maintained by hand. columns.txt and columns_substreams.txt both record the column list. The loader cross-checks them explicitly, and there is a comment explaining why. That redundancy is a feature (it enables recovery) but it is also an invariant that a future change can break.

Wrapping up

The one thing to remember: a MergeTree part’s metadata is a handful of small text files next to the data, and it is reconstructed rather than trusted wherever possible. The schema of a part is not stored in the part; only the list of physical columns is. Everything else is derived.

The part people get wrong: columns.txt does not contain the codec, the default expression, or the column TTL. Those live in the table metadata. If you cat columns.txt and expect to see CODEC(ZSTD(3)), you will not find it — and system.parts_columns will not show it either.

What to go do about it: go look at a part on disk.

sql 
SELECT name, path FROM system.parts WHERE active AND table = 'your_table' LIMIT 1;

then cat the columns.txt in that directory. It is four lines long, it is the truth about what is stored, and it is the first thing to check when a query returns a column you did not expect.

ClickhouseInternalsMergetreeStoragePerformance