Opensource internals

How the ClickHouse Log Engine Stores Rows in Append-Only Files with a Marks Index

13 min read Jay Hubbert

The Log engine writes one .bin file per column stream plus a __marks.mrk index of (rows, offset) pairs, so a read splits across threads by seeking to a mark, and a crash is undone by truncating files.

TL;DR: Log is an append-only table with no parts and no merges. Every column gets its own compressed .bin file, and a __marks.mrk file records, for every inserted block, the row count and the byte offset in each data file. That index is what lets a SELECT be split across threads: each thread seeks to its mark and reads a fixed row range. Crash safety comes from a separate sizes.json that records the committed length of every file — on restart, anything longer is truncated back.

MergeTree is a very heavy engine. It is the right answer for data you will keep for a year and query a billion times. It is the wrong answer for a scratch table: the log of a background job, a staging table for a migration, a table that a test suite creates and drops fifty times a minute.

The Log engine family is the light alternative: no parts, no merging, no primary index. Just append to a file and read it back. The interesting engineering is not the append — it is the three problems that append-only storage creates: how do you read a file that is being appended to from several threads at once, how do you survive a crash in the middle of a write, and how do you read one column without reading all of them. The answers are a marks file, a size ledger, and one file per column.

Two engines, one class

First, the surprise: Log and TinyLog are the same class. The only difference is a boolean.

cpp 
// src/Storages/StorageLog.h:21
/** Implements Log - a simple table engine without support of indices.
  * The data is stored in a compressed form.
  *
  * Also implements TinyLog - a table engine that is suitable for small chunks of the log.
  * It differs from Log in the absence of mark files.
  */

and the boolean is decided by the engine name:

cpp 
// src/Storages/StorageLog.cpp:762
    , use_marks_file(engine_name == "Log")

Both are registered with the same factory lambda, which passes the engine name straight through to the constructor.

The engine’s own documentation is blunt about what TinyLog gives up: “Unlike the Log engine, TinyLog does not use mark files. This reduces complexity but also limits performance optimizations for larger datasets.” A TinyLog table is read in a single stream and is expected to hold up to about a million rows.

The file layout

Three kinds of files live in the table’s directory:

text 
store/log/table_name/
    ├── col_a.bin        <- compressed data for column a
    ├── col_b.bin        <- compressed data for column b
    ├── col_b.size0.bin  <- (for Array/Nested) the offsets substream
    ├── __marks.mrk      <- the index: (rows, offset) per block, per file
    └── sizes.json       <- the committed size of every file above

The names come from two macros — DBMS_STORAGE_LOG_DATA_FILE_EXTENSION (".bin") and DBMS_STORAGE_LOG_MARKS_FILE_NAME ("__marks.mrk").

The per-column file naming is not a hand-written convention; it comes from the serialization itself. ClickHouse’s ISerialization can split a column into several “substreams” — an Array(UInt8) is two substreams (the elements and the offsets), a LowCardinality(String) is two (the dictionary and the index), a Nullable adds a null map. The Log engine gives each substream its own file:

cpp 
// src/Storages/StorageLog.cpp:828
        String data_file_name = ISerialization::getFileNameForStream(column, substream_path, {});
        if (!data_files_by_names.contains(data_file_name))
        {
            DataFile & data_file = data_files.emplace_back();
            data_file.name = data_file_name;
            data_file.path = table_path + data_file_name + DBMS_STORAGE_LOG_DATA_FILE_EXTENSION;
            data_file.index = num_data_files++;
            data_files_by_names.emplace(data_file_name, nullptr);
        }

Two consequences follow. Reading one column does not read the others: SELECT a FROM t opens col_a.bin and nothing else — the engine is columnar, like MergeTree, but without parts. And each column can have its own compression codec, because getCodecOrDefault looks the column up in the table’s metadata and falls back to the server default.

The in-memory representation of that layout is small: a DataFile with an index, a name, a path, and its own vector of marks.

The marks file

This is the index. A Mark is a row count and a byte offset:

cpp 
// src/Storages/StorageLog.h:130
    /** Offsets to some row number in a file for column in table.
      * They are needed so that you can read the data in several threads.
      */
    struct Mark
    {
        size_t rows;   /// How many rows are before this offset including the block at this offset.
        size_t offset; /// The offset in compressed file.

        void write(WriteBuffer & out) const;
        void read(ReadBuffer & in);
    };
    using Marks = std::vector<Mark>;

Note the wording of rows: “how many rows are before this offset including the block at this offset”. The marks form a cumulative row count, not a per-block count: a mark is a position — “after this mark, rows rows have been written, and the next byte of data is at offset”. That is the same idea as a MergeTree mark, but a thousand times simpler — a MergeTree mark points into a compressed granule and carries a whole primary-key structure, while a Log mark is 16 bytes of size_t.

On disk, a mark is two little-endian size_ts (rows then offset), and the file is a flat array of them, interleaved by data file. The layout is “for each mark index, for each data file, one mark”:

cpp 
// src/Storages/StorageLog.cpp:865
    size_t num_marks = 0;
    if (disk->existsFile(marks_file_path))
    {
        size_t file_size = disk->getFileSize(marks_file_path);
        if (file_size % (num_data_files * sizeof(Mark)) != 0)
            throw Exception(ErrorCodes::SIZES_OF_MARKS_FILES_ARE_INCONSISTENT, "Size of marks file is inconsistent");

        num_marks = file_size / (num_data_files * sizeof(Mark));

        for (auto & data_file : data_files)
            data_file.marks.resize(num_marks);

The consistency check is the interesting bit: the marks file size must be an exact multiple of num_data_files * sizeof(Mark). If it is not, the file is torn — a partial write from a crash — and the engine refuses to start rather than reading garbage. The file is a pure array, not a header plus a payload, and the number of marks is derived from its size. That is what makes it cheap to append to: no header to rewrite, no index to update, just more bytes at the end.

And the write side appends only the new marks:

cpp 
// src/Storages/StorageLog.cpp:911
    size_t start = num_marks_saved;
    auto marks_stream = disk->writeFile(marks_file_path, 4096, WriteMode::Append);

    for (size_t i = start; i != num_marks; ++i)
    {
        for (const auto & data_file : data_files)
        {
            const auto & mark = data_file.marks[i];
            mark.write(*marks_stream);
        }
    }

start = num_marks_saved — only the new marks get written, into a file opened in WriteMode::Append; the previously saved prefix is never rewritten. An append-only index over an append-only data file: the whole design is coherent.

Where the marks come from

Now the write path, and the actual mechanism: for each inserted block, a mark is recorded per data file with the row count and the byte offset at which that block starts.

cpp 
// src/Storages/StorageLog.cpp:669
    if (storage.use_marks_file)
    {
        serialization->enumerateStreams([&] (const ISerialization::SubstreamPath & path)
        {
            String data_file_name = ISerialization::getFileNameForStream(name_and_type, path, {});
            const auto & stream = streams.at(data_file_name);
            if (stream.written)
                return;

            auto & data_file = *storage.data_files_by_names.at(data_file_name);
            auto & marks = data_file.marks;
            size_t prev_num_rows = marks.empty() ? 0 : marks.back().rows;
            auto & mark = marks.emplace_back();
            mark.rows = prev_num_rows + column.size();
            mark.offset = stream.plain_offset + stream.plain->count();
        });
    }

mark.rows = prev_num_rows + column.size() derives the cumulative row count from the previous mark, which is why the marks vector must be read in order and why the first column’s marks define the table’s row count. mark.offset = stream.plain_offset + stream.plain->count() is the number of bytes already written to the file when the block starts — plain_offset is the file length when the stream was opened, and plain->count() is how much this INSERT has written so far. The offset is in the plain (uncompressed) file, which is what a seek() needs.

One subtlety is easy to miss: for an Array column the elements file and the offsets file have different row counts. The engine handles this by convention — the first data file carries the real row count:

cpp 
// src/Storages/StorageLog.cpp:723
    /// So for Array data type, first stream is array sizes; and number of array sizes is the number of arrays.
    /// Thus we assume we can always get the real number of rows from the first column.
    constexpr size_t INDEX_WITH_REAL_ROW_COUNT = 0;

The constant INDEX_WITH_REAL_ROW_COUNT = 0 is used in three places: to get the total row count, to decide whether the table is empty, and to split the read across streams.

Reading: splitting the table across threads

Here is where the marks earn their keep. createReadingPipe divides the mark list into num_streams contiguous slices and gives each stream the offsets of its first mark:

cpp 
// src/Storages/StorageLog.cpp:1102
    for (size_t stream = 0; stream < num_streams; ++stream)
    {
        if (use_marks_file)
        {
            size_t mark_begin = stream * num_marks / num_streams;
            size_t mark_end = (stream + 1) * num_marks / num_streams;
            size_t start_row = mark_begin ? marks_with_real_row_count[mark_begin - 1].rows : 0;
            size_t end_row = mark_end ? marks_with_real_row_count[mark_end - 1].rows : 0;
            row_limit = end_row - start_row;
            for (const auto & data_file : data_files)
                offsets[data_file.index] = data_file.marks[mark_begin].offset;
        }

Read that carefully: the row range of a stream comes from the cumulative row counts of the marks at its boundaries, and its starting byte offset in each data file is the offset of its first mark. So a stream is (offset, row_limit) per file — a byte range and a row count.

The number of streams is capped by the number of marks: max_streams = use_marks_file ? num_marks : 1. For TinyLog, use_marks_file is false, so max_streams is 1: a single stream that reads the files to their ends. That is the entire implementation difference between the two engines.

And the source seeks to the offset and stops at the snapshotted file size:

cpp 
// src/Storages/StorageLog.cpp:156
            /// Limit reads to the file size that was snapshotted under the read lock.
            /// The file may have grown since (due to concurrent inserts after lock release),
            /// but we must not read beyond the snapshotted range.
            plain->setReadUntilPosition(file_size);

            if (offset)
                plain->seek(offset, SEEK_SET);

The file sizes were snapshotted under the read lock, so a concurrent INSERT that grows the file after the lock is released does not confuse the reader. This is also why the read lock can be released before the data is actually read — “no need to hold lock while reading because we read fixed range of data that does not change while appending more data”.

Appending only ever writes past the end of the data the reader is looking at, so those bytes are immutable. The lock is needed for a consistent snapshot of the marks and the file sizes, and nothing else.

There is one more thing the read path does that is worth copying: it tolerates a mixed-codec file. The data file is append-only, and a server upgrade can change the default codec, so one file can contain LZ4 blocks written before the upgrade and ZSTD blocks written after:

cpp 
// src/Storages/StorageLog.cpp:164
            /// `allow_different_codecs = true`: the data file is append-only, so blocks written by
            /// different inserts may use different codecs - in particular after a server upgrade that
            /// changes the default compression codec (e.g. `LZ4` -> `ZSTD`). Each compressed block is
            /// self-describing (the codec method byte is in its header), so a mixed-codec stream is valid.

Each compressed block carries its own method byte, so the reader does not need to know which codec wrote which block.

Crash safety: the size ledger

Now the hard part. An append-only file plus an append-only index, and the process dies halfway through an INSERT. What is on disk?

The answer is a third file: sizes.json, managed by FileChecker, which records the committed length of every file. The write path commits the data files and the marks file first, and only then records their sizes — saveMarksAndFileSizes() wraps the two in a try/catch that restores the saved-mark count if the size recording fails, because the marks file is itself one of the files whose size is recorded, and “how many marks did I save” is only meaningful if the recorded sizes agree with it.

The sizes are recorded as a unit: FileChecker::save() writes the JSON to tmp_sizes.json, fsyncs it, and renames it over sizes.json. So sizes.json is replaced atomically, and it always describes a consistent set of file lengths.

Now the recovery. On ATTACH, the constructor calls file_checker.repair(), which truncates every file that is longer than its recorded size and throws if a file is shorter:

cpp 
// src/Common/FileChecker.cpp:148
void FileChecker::repair()
{
    for (const auto & name_size : map)
    {
        const String & name = name_size.first;
        size_t expected_size = name_size.second;
        String path = parentPath(files_info_path) + name;
        bool exists = fileReallyExists(path);
        auto real_size = exists ? getRealFileSize(path) : 0;  /// No race condition assuming no one else is working with these files.

        if (real_size < expected_size)
            throw Exception(ErrorCodes::UNEXPECTED_END_OF_FILE, "Size of {} is less than expected. Size is {} but should be {}.",
                path, real_size, expected_size);

        if (real_size > expected_size)
        {
            LOG_WARNING(log, "Will truncate file {} that has size {} to size {}", path, real_size, expected_size);
            disk->truncateFile(path, expected_size);
        }
    }
}

This is the whole crash-safety story in nine lines:

text 
   INSERT of block B:
       append B to col_a.bin, col_b.bin ...        (files grow)
       append marks for B to __marks.mrk            (file grows)
       write sizes.json = {current lengths}         (atomic rename)

   crash at any point:

       crash before the rename:
           sizes.json still holds the OLD lengths
           -> repair() truncates the data files back to the old lengths
           -> the half-written block is gone, the table is consistent

       crash after the rename:
           sizes.json holds the NEW lengths
           -> repair() finds nothing to truncate
           -> the block is fully there

The same thing happens without a crash, when an INSERT fails mid-way: the LogSink destructor rolls the write back.

cpp 
// src/Storages/StorageLog.cpp:456
                /// Rollback partial writes.

                /// No more writing.
                for (auto & [_, stream] : streams)
                {
                    stream.cancel();
                }
                streams.clear();

                /// Truncate files to the older sizes.
                storage.file_checker.repair();

                /// Remove excessive marks.
                storage.removeUnsavedMarks(lock);

Three steps: cancel the write buffers, truncate the files back to the recorded sizes, and drop the marks that were never committed. The engine does not attempt a partial write; it undoes it.

Trade-offs

What the Log engine buys: a table with no parts, no merges, no background work, and a read path that is a seek plus a decompress. It is the fastest thing in ClickHouse for “append a block, read it back”, and it is a legitimate destination for a materialized view.

What it costs:

  • One file per column substream. A table with 20 columns has 20+ files, each with its own open file descriptor during a read. This is why TinyLog exists (no marks, one stream).
  • No compaction, no deduplication, no ordering. Rows are stored in INSERT order. A Log table that is written twice has the rows twice.
  • The marks file grows forever. One mark per inserted block per data file, never rewritten, never compacted. A table with a million single-row INSERTs has a million marks, all read into memory on the first read.
  • Log is read-only-ish. No UPDATE, no DELETE, no partition pruning, no primary index. It is a log, in the literal sense.

Wrapping up

The one thing to remember: the Log engine is three files and a 16-byte-per-block index, and the index is what makes it usable. Without __marks.mrk you would have one reader and no parallelism; with it, the read splits into min(num_streams, num_marks) independent seeks.

The part people get wrong: the data files and the marks file are not the commit point. The commit point is sizes.json, and everything else is rolled back to it. Copy a Log table’s directory by hand and copy sizes.json too — or the engine will truncate your data on the next attach.

What to go do about it: create a Log table, insert a few million rows, and ls the data directory. You will find one .bin per column (per serialization substream, so an Array column gets two), a __marks.mrk that is 16 * num_blocks * num_data_files bytes, and a sizes.json with one size per file.

ClickhouseInternalsStorage enginesLog