Storage-Engines on Opensource internals

How the ClickHouse Log Engine Stores Rows in Append-Only Files with a Marks Index

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: Log is an append-only table with no parts and no merges. Every column gets its own compressed .bin file, and a __marks.mrk file records, for every inserted block, the row count and the byte offset in each data file. That index is what lets a SELECT be split across threads: each thread seeks to its mark and reads a fixed row range. Crash safety comes from a separate sizes.json that records the committed length of every file — on restart, anything longer is truncated back.

MergeTree is a very heavy engine. It is the right answer for data you will keep for a year and query a billion times. It is the wrong answer for a scratch table: the log of a background job, a staging table for a migration, a table that a test suite creates and drops fifty times a minute.

The Log engine family is the light alternative: no parts, no merging, no primary index. Just append to a file and read it back. The interesting engineering is not the append — it is the three problems that append-only storage creates: how do you read a file that is being appended to from several threads at once, how do you survive a crash in the middle of a write, and how do you read one column without reading all of them. The answers are a marks file, a size ledger, and one file per column.

Two engines, one class

First, the surprise: Log and TinyLog are the same class. The only difference is a boolean.

 cpp
 
 
// src/Storages/StorageLog.h:21
/** Implements Log - a simple table engine without support of indices.
 * The data is stored in a compressed form.
 *
 * Also implements TinyLog - a table engine that is suitable for small chunks of the log.
 * It differs from Log in the absence of mark files.
 */

and the boolean is decided by the engine name:

 cpp
 
 
// src/Storages/StorageLog.cpp:762
 , use_marks_file(engine_name == "Log")

Both are registered with the same factory lambda, which passes the engine name straight through to the constructor.

The engine’s own documentation is blunt about what TinyLog gives up: “Unlike the Log engine, TinyLog does not use mark files. This reduces complexity but also limits performance optimizations for larger datasets.” A TinyLog table is read in a single stream and is expected to hold up to about a million rows.

The file layout

Three kinds of files live in the table’s directory:

 text
 
 
store/log/table_name/
 ├── col_a.bin <- compressed data for column a
 ├── col_b.bin <- compressed data for column b
 ├── col_b.size0.bin <- (for Array/Nested) the offsets substream
 ├── __marks.mrk <- the index: (rows, offset) per block, per file
 └── sizes.json <- the committed size of every file above

The names come from two macros — DBMS_STORAGE_LOG_DATA_FILE_EXTENSION (".bin") and DBMS_STORAGE_LOG_MARKS_FILE_NAME ("__marks.mrk").

The per-column file naming is not a hand-written convention; it comes from the serialization itself. ClickHouse’s ISerialization can split a column into several “substreams” — an Array(UInt8) is two substreams (the elements and the offsets), a LowCardinality(String) is two (the dictionary and the index), a Nullable adds a null map. The Log engine gives each substream its own file:

 cpp
 
 
// src/Storages/StorageLog.cpp:828
 String data_file_name = ISerialization::getFileNameForStream(column, substream_path, {});
 if (!data_files_by_names.contains(data_file_name))
 {
 DataFile & data_file = data_files.emplace_back();
 data_file.name = data_file_name;
 data_file.path = table_path + data_file_name + DBMS_STORAGE_LOG_DATA_FILE_EXTENSION;
 data_file.index = num_data_files++;
 data_files_by_names.emplace(data_file_name, nullptr);
 }

Two consequences follow. Reading one column does not read the others: SELECT a FROM t opens col_a.bin and nothing else — the engine is columnar, like MergeTree, but without parts. And each column can have its own compression codec, because getCodecOrDefault looks the column up in the table’s metadata and falls back to the server default.

The in-memory representation of that layout is small: a DataFile with an index, a name, a path, and its own vector of marks.

The marks file

This is the index. A Mark is a row count and a byte offset:

 cpp
 
 
// src/Storages/StorageLog.h:130
 /** Offsets to some row number in a file for column in table.
 * They are needed so that you can read the data in several threads.
 */
 struct Mark
 {
 size_t rows; /// How many rows are before this offset including the block at this offset.
 size_t offset; /// The offset in compressed file.

 void write(WriteBuffer & out) const;
 void read(ReadBuffer & in);
 };
 using Marks = std::vector<Mark>;

Note the wording of rows: “how many rows are before this offset including the block at this offset”. The marks form a cumulative row count, not a per-block count: a mark is a position — “after this mark, rows rows have been written, and the next byte of data is at offset”. That is the same idea as a MergeTree mark, but a thousand times simpler — a MergeTree mark points into a compressed granule and carries a whole primary-key structure, while a Log mark is 16 bytes of size_t.

On disk, a mark is two little-endian size_ts (rows then offset), and the file is a flat array of them, interleaved by data file. The layout is “for each mark index, for each data file, one mark”:

 cpp
 
 
// src/Storages/StorageLog.cpp:865
 size_t num_marks = 0;
 if (disk->existsFile(marks_file_path))
 {
 size_t file_size = disk->getFileSize(marks_file_path);
 if (file_size % (num_data_files * sizeof(Mark)) != 0)
 throw Exception(ErrorCodes::SIZES_OF_MARKS_FILES_ARE_INCONSISTENT, "Size of marks file is inconsistent");

 num_marks = file_size / (num_data_files * sizeof(Mark));

 for (auto & data_file : data_files)
 data_file.marks.resize(num_marks);

The consistency check is the interesting bit: the marks file size must be an exact multiple of num_data_files * sizeof(Mark). If it is not, the file is torn — a partial write from a crash — and the engine refuses to start rather than reading garbage. The file is a pure array, not a header plus a payload, and the number of marks is derived from its size. That is what makes it cheap to append to: no header to rewrite, no index to update, just more bytes at the end.

And the write side appends only the new marks:

 cpp
 
 
// src/Storages/StorageLog.cpp:911
 size_t start = num_marks_saved;
 auto marks_stream = disk->writeFile(marks_file_path, 4096, WriteMode::Append);

 for (size_t i = start; i != num_marks; ++i)
 {
 for (const auto & data_file : data_files)
 {
 const auto & mark = data_file.marks[i];
 mark.write(*marks_stream);
 }
 }

start = num_marks_saved — only the new marks get written, into a file opened in WriteMode::Append; the previously saved prefix is never rewritten. An append-only index over an append-only data file: the whole design is coherent.

Where the marks come from

Now the write path, and the actual mechanism: for each inserted block, a mark is recorded per data file with the row count and the byte offset at which that block starts.

 cpp
 
 
// src/Storages/StorageLog.cpp:669
 if (storage.use_marks_file)
 {
 serialization->enumerateStreams([&] (const ISerialization::SubstreamPath & path)
 {
 String data_file_name = ISerialization::getFileNameForStream(name_and_type, path, {});
 const auto & stream = streams.at(data_file_name);
 if (stream.written)
 return;

 auto & data_file = *storage.data_files_by_names.at(data_file_name);
 auto & marks = data_file.marks;
 size_t prev_num_rows = marks.empty() ? 0 : marks.back().rows;
 auto & mark = marks.emplace_back();
 mark.rows = prev_num_rows + column.size();
 mark.offset = stream.plain_offset + stream.plain->count();
 });
 }

mark.rows = prev_num_rows + column.size() derives the cumulative row count from the previous mark, which is why the marks vector must be read in order and why the first column’s marks define the table’s row count. mark.offset = stream.plain_offset + stream.plain->count() is the number of bytes already written to the file when the block starts — plain_offset is the file length when the stream was opened, and plain->count() is how much this INSERT has written so far. The offset is in the plain (uncompressed) file, which is what a seek() needs.

One subtlety is easy to miss: for an Array column the elements file and the offsets file have different row counts. The engine handles this by convention — the first data file carries the real row count:

 cpp
 
 
// src/Storages/StorageLog.cpp:723
 /// So for Array data type, first stream is array sizes; and number of array sizes is the number of arrays.
 /// Thus we assume we can always get the real number of rows from the first column.
 constexpr size_t INDEX_WITH_REAL_ROW_COUNT = 0;

The constant INDEX_WITH_REAL_ROW_COUNT = 0 is used in three places: to get the total row count, to decide whether the table is empty, and to split the read across streams.

Reading: splitting the table across threads

Here is where the marks earn their keep. createReadingPipe divides the mark list into num_streams contiguous slices and gives each stream the offsets of its first mark:

 cpp
 
 
// src/Storages/StorageLog.cpp:1102
 for (size_t stream = 0; stream < num_streams; ++stream)
 {
 if (use_marks_file)
 {
 size_t mark_begin = stream * num_marks / num_streams;
 size_t mark_end = (stream + 1) * num_marks / num_streams;
 size_t start_row = mark_begin ? marks_with_real_row_count[mark_begin - 1].rows : 0;
 size_t end_row = mark_end ? marks_with_real_row_count[mark_end - 1].rows : 0;
 row_limit = end_row - start_row;
 for (const auto & data_file : data_files)
 offsets[data_file.index] = data_file.marks[mark_begin].offset;
 }

Read that carefully: the row range of a stream comes from the cumulative row counts of the marks at its boundaries, and its starting byte offset in each data file is the offset of its first mark. So a stream is (offset, row_limit) per file — a byte range and a row count.

The number of streams is capped by the number of marks: max_streams = use_marks_file ? num_marks : 1. For TinyLog, use_marks_file is false, so max_streams is 1: a single stream that reads the files to their ends. That is the entire implementation difference between the two engines.

And the source seeks to the offset and stops at the snapshotted file size:

 cpp
 
 
// src/Storages/StorageLog.cpp:156
 /// Limit reads to the file size that was snapshotted under the read lock.
 /// The file may have grown since (due to concurrent inserts after lock release),
 /// but we must not read beyond the snapshotted range.
 plain->setReadUntilPosition(file_size);

 if (offset)
 plain->seek(offset, SEEK_SET);

The file sizes were snapshotted under the read lock, so a concurrent INSERT that grows the file after the lock is released does not confuse the reader. This is also why the read lock can be released before the data is actually read — “no need to hold lock while reading because we read fixed range of data that does not change while appending more data”.

Appending only ever writes past the end of the data the reader is looking at, so those bytes are immutable. The lock is needed for a consistent snapshot of the marks and the file sizes, and nothing else.

There is one more thing the read path does that is worth copying: it tolerates a mixed-codec file. The data file is append-only, and a server upgrade can change the default codec, so one file can contain LZ4 blocks written before the upgrade and ZSTD blocks written after:

 cpp
 
 
// src/Storages/StorageLog.cpp:164
 /// `allow_different_codecs = true`: the data file is append-only, so blocks written by
 /// different inserts may use different codecs - in particular after a server upgrade that
 /// changes the default compression codec (e.g. `LZ4` -> `ZSTD`). Each compressed block is
 /// self-describing (the codec method byte is in its header), so a mixed-codec stream is valid.

Each compressed block carries its own method byte, so the reader does not need to know which codec wrote which block.

Crash safety: the size ledger

Now the hard part. An append-only file plus an append-only index, and the process dies halfway through an INSERT. What is on disk?

The answer is a third file: sizes.json, managed by FileChecker, which records the committed length of every file. The write path commits the data files and the marks file first, and only then records their sizes — saveMarksAndFileSizes() wraps the two in a try/catch that restores the saved-mark count if the size recording fails, because the marks file is itself one of the files whose size is recorded, and “how many marks did I save” is only meaningful if the recorded sizes agree with it.

The sizes are recorded as a unit: FileChecker::save() writes the JSON to tmp_sizes.json, fsyncs it, and renames it over sizes.json. So sizes.json is replaced atomically, and it always describes a consistent set of file lengths.

Now the recovery. On ATTACH, the constructor calls file_checker.repair(), which truncates every file that is longer than its recorded size and throws if a file is shorter:

 cpp
 
 
// src/Common/FileChecker.cpp:148
void FileChecker::repair()
{
 for (const auto & name_size : map)
 {
 const String & name = name_size.first;
 size_t expected_size = name_size.second;
 String path = parentPath(files_info_path) + name;
 bool exists = fileReallyExists(path);
 auto real_size = exists ? getRealFileSize(path) : 0; /// No race condition assuming no one else is working with these files.

 if (real_size < expected_size)
 throw Exception(ErrorCodes::UNEXPECTED_END_OF_FILE, "Size of {} is less than expected. Size is {} but should be {}.",
 path, real_size, expected_size);

 if (real_size > expected_size)
 {
 LOG_WARNING(log, "Will truncate file {} that has size {} to size {}", path, real_size, expected_size);
 disk->truncateFile(path, expected_size);
 }
 }
}

This is the whole crash-safety story in nine lines:

 text
 
 
 INSERT of block B:
 append B to col_a.bin, col_b.bin ... (files grow)
 append marks for B to __marks.mrk (file grows)
 write sizes.json = {current lengths} (atomic rename)

 crash at any point:

 crash before the rename:
 sizes.json still holds the OLD lengths
 -> repair() truncates the data files back to the old lengths
 -> the half-written block is gone, the table is consistent

 crash after the rename:
 sizes.json holds the NEW lengths
 -> repair() finds nothing to truncate
 -> the block is fully there

The same thing happens without a crash, when an INSERT fails mid-way: the LogSink destructor rolls the write back.

 cpp
 
 
// src/Storages/StorageLog.cpp:456
 /// Rollback partial writes.

 /// No more writing.
 for (auto & [_, stream] : streams)
 {
 stream.cancel();
 }
 streams.clear();

 /// Truncate files to the older sizes.
 storage.file_checker.repair();

 /// Remove excessive marks.
 storage.removeUnsavedMarks(lock);

Three steps: cancel the write buffers, truncate the files back to the recorded sizes, and drop the marks that were never committed. The engine does not attempt a partial write; it undoes it.

Trade-offs

What the Log engine buys: a table with no parts, no merges, no background work, and a read path that is a seek plus a decompress. It is the fastest thing in ClickHouse for “append a block, read it back”, and it is a legitimate destination for a materialized view.

What it costs:

One file per column substream. A table with 20 columns has 20+ files, each with its own open file descriptor during a read. This is why TinyLog exists (no marks, one stream).
No compaction, no deduplication, no ordering. Rows are stored in INSERT order. A Log table that is written twice has the rows twice.
The marks file grows forever. One mark per inserted block per data file, never rewritten, never compacted. A table with a million single-row INSERTs has a million marks, all read into memory on the first read.
Log is read-only-ish. No UPDATE, no DELETE, no partition pruning, no primary index. It is a log, in the literal sense.

Wrapping up

The one thing to remember: the Log engine is three files and a 16-byte-per-block index, and the index is what makes it usable. Without __marks.mrk you would have one reader and no parallelism; with it, the read splits into min(num_streams, num_marks) independent seeks.

The part people get wrong: the data files and the marks file are not the commit point. The commit point is sizes.json, and everything else is rolled back to it. Copy a Log table’s directory by hand and copy sizes.json too — or the engine will truncate your data on the next attach.

What to go do about it: create a Log table, insert a few million rows, and ls the data directory. You will find one .bin per column (per serialization substream, so an Array column gets two), a __marks.mrk that is 16 * num_blocks * num_data_files bytes, and a sizes.json with one size per file.

How the ClickHouse Kafka Engine Commits Offsets Only After the Block Is Written

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: A Kafka consumer that commits its offset before the data is durable loses data; one that commits after duplicates data. ClickHouse picks the second: the background task builds a block, pushes it through the materialized views, and only when the pipeline has finished does it call commit(). If the block is cancelled halfway, the consumer rewinds to the start of that block, so the block is replayed instead of lost. The cost is a real hole — the code says so in an exception message.

Kafka is a log with a cursor, and every consumer of a log has to answer the same question: when do I tell the broker I have read this? Answer it too early and a crash between the commit and the write loses data; answer it too late and a crash between the write and the commit duplicates data. Neither is “exactly once”, and the difference between them is where the correctness bug lives.

ClickHouse’s Kafka engine answers it as late as it can: the offset commit happens after the rows have been written into the materialized views. This article follows that ordering through the code, and then through the places where it leaks.

The shape of the engine

A Kafka table in ClickHouse is not a table you query — it is a stream source with materialized views attached:

 sql
 
 
CREATE TABLE events_queue (id UInt64, payload String)
ENGINE = Kafka
SETTINGS kafka_broker_list = 'kafka:9092',
 kafka_topic_list = 'events',
 kafka_group_name = 'clickhouse',
 kafka_format = 'JSONEachRow';

CREATE MATERIALIZED VIEW events_mv TO events AS SELECT * FROM events_queue;

The engine’s job is to read messages, parse them with a format, and push the resulting blocks into the views. It never stores anything itself. The background work is a scheduled task, one per consumer (or one for all of them, unless kafka_thread_per_consumer is set):

 cpp
 
 
// src/Storages/Kafka/StorageKafka.cpp:217
 auto task = getContext()->getMessageBrokerSchedulePool()->createTask(getStorageID(), log->name(), [this, i]{ threadFunc(i); });

threadFunc is the outer loop: it checks that the dependent views are ready, claims a “cycle” (so that two tasks do not consume the same topic concurrently), and then streams until it stalls or hits the work-duration limit. The work itself is in streamToViews, and this is where the ordering lives.

The push: write first, commit after

streamToViews builds an INSERT pipeline into the dependent views, runs it to completion, and only then commits the offsets:

 cpp
 
 
// src/Storages/Kafka/StorageKafka.cpp:757
 std::atomic_size_t rows = 0;
 {
 block_io.pipeline.complete(std::move(pipe));

 // we need to read all consumers in parallel (sequential read may lead to situation
 // when some of consumers are not used, and will break some Kafka consumer invariants)
 block_io.pipeline.setNumThreads(stream_count);
 block_io.pipeline.setConcurrencyControl(kafka_context->getSettingsRef()[Setting::use_concurrency_control]);

 block_io.pipeline.setProgressCallback([&](const Progress & progress) { rows += progress.read_rows.load(); });
 CompletedPipelineExecutor executor(block_io.pipeline);
 executor.execute();
 }

 bool some_stream_is_stalled = false;
 for (auto & source : sources)
 {
 some_stream_is_stalled = some_stream_is_stalled || source->isStalled();
 source->commit();
 }

Read the two halves again. executor.execute() runs the whole pipeline: Kafka source → format parser → INSERT into the materialized views → the view’s own storage engine (usually a MergeTree, which writes a part). When it returns, the data is in the destination table. source->commit() then tells Kafka “I have read up to offset N”.

If the process dies during executor.execute(), the offsets were never committed, so the messages are re-read after restart, and the destination table does not have them. No data loss. If the process dies between executor.execute() and source->commit(), the destination table has the rows and Kafka does not know it: the messages will be re-read and re-written. Duplicates.

That window is unavoidable without a transaction spanning Kafka and ClickHouse, and the code says so out loud — the exception message when the commit fails is one of the best pieces of honest error reporting in the codebase:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:257
 if (!committed)
 {
 // TODO: insert atomicity / transactions is needed here (possibility to rollback, on 2 phase commits)
 ProfileEvents::increment(ProfileEvents::KafkaCommitFailures);
 throw Exception(ErrorCodes::CANNOT_COMMIT_OFFSET,
 "All commit attempts failed. Last block was already written to target table(s), "
 "but was not committed to Kafka.");
 }

“Last block was already written to target table(s), but was not committed to Kafka.” That is the exactly-once hole, named precisely. The TODO says what would close it: a two-phase commit.

Note also that the commit is retried five times before giving up, because a Kafka commit can fail for boring transient reasons — “broker may reject commit if during offsets.commit.timeout.ms (5000 by default), there were not enough replicas available for the __consumer_offsets topic”. One error code is treated as success: RD_KAFKA_RESP_ERR__NO_OFFSET means “there was nothing to commit”, so it short-circuits the retry loop. Everything else is retried, and when the retries run out:

The block boundary

The commit is per block, not per message. The size of a block is decided by the source, and the source’s loop has four exit conditions:

 cpp
 
 
// src/Storages/Kafka/KafkaSource.cpp:323
 if (!consumer->hasMorePolledMessages()
 && (total_rows >= max_block_size || !checkTimeLimit() || failed_poll_attempts >= MAX_FAILED_POLL_ATTEMPTS))
 {
 break;
 }

total_rows >= max_block_size — the block is full.
!checkTimeLimit() — the flush interval elapsed. The default is stream_flush_interval_ms = 7500 ms, or kafka_flush_interval_ms if it is set.
failed_poll_attempts >= MAX_FAILED_POLL_ATTEMPTS — the topic is empty and the poll has failed 10 times in a row (“with default poll timeout (500ms) it will give about 5 sec delay for doing 10 retries when selecting from empty topic”).
!consumer->hasMorePolledMessages() — the polled batch is exhausted, so there is nothing more to read without another poll.

The block size is max_insert_block_size / num_consumers by default, so that N consumers each produce a block of 1/N of the insert size (or kafka_max_block_size if it is set explicitly). A bigger block means fewer commits and fewer parts; a smaller block means lower latency for the first rows. This is the knob.

The block start: how to replay

Here is the part I find genuinely clever. When a block is aborted — the table is detached, the query is cancelled, the view is not ready — the engine must not lose the messages it already consumed but did not commit. It must also not re-consume messages it already committed.

The trick: the consumer records where the current block started, per topic-partition, before it consumes anything:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:586
void KafkaConsumer::storeLastReadMessageOffset()
{
 if (!isStalled())
 {
 trackCurrentBlockStart(*(current - 1));
 consumer->store_offset(*(current - 1));
 ++offsets_stored;
 }
}

void KafkaConsumer::trackCurrentBlockStart(const cppkafka::Message & message)
{
 for (const auto & tp : block_start_offsets)
 if (tp.get_partition() == message.get_partition() && tp.get_topic() == message.get_topic())
 return;
 block_start_offsets.emplace_back(message.get_topic(), message.get_partition(), message.get_offset());
}

storeLastReadMessageOffset() is called after every message that produced rows. It stores the current offset (the one to commit later) and records the first offset seen for this topic-partition since the last commit — trackCurrentBlockStart is idempotent per partition, so only the first one is kept.

And when the block is cancelled, the source rewinds to those recorded starting points:

 cpp
 
 
// src/Storages/Kafka/KafkaSource.cpp:330
 if (cancelled || storage.isConsumeCancelRequested(cancel_epoch))
 {
 try
 {
 consumer->rewindToLastCommitted();
 }
 catch (const cppkafka::HandleException & e)
 {
 LOG_WARNING(log, "Failed to rewind to last committed offset on abort: {}", e.what());
 consumer->cleanBlockStartOffsets();
 consumer->markDirty();
 }
 return {};
 }

and the rewind assigns the consumer back to the block’s start offsets and commits them, so the rewind is durable:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:411
 /// Rewind to where the in-flight block started. Re-commit the block partitions
 size_t max_retries = 5;
 while (max_retries > 0)
 {
 try
 {
 auto positions = consumer->get_offsets_committed(consumer->get_assignment());
 for (auto & tp : positions)
 for (const auto & start : block_start_offsets)
 if (tp.get_partition() == start.get_partition() && tp.get_topic() == start.get_topic())
 tp.set_offset(start.get_offset());

 consumer->assign(positions);
 consumer->commit(block_start_offsets);
 LOG_TRACE(log, "Aborted in-flight block, rewound to block start: {}", block_start_offsets);
 cleanBlockStartOffsets();
 return;
 }

So the failure taxonomy is:

 text
 
 
 block in flight (offsets stored, not committed)
 |
 | crash / cancel / view not ready
 v
 rewind to block_start_offsets -> the block is re-read from the broker
 |
 | the block is re-consumed and re-written
 v
 duplicates in the destination (the previous attempt may have
 partially or fully written before the failure)

The rewind is what makes the engine not lose data. The duplicates are the price, paid in the destination table — which is why the standard advice is to use a deduplicating engine (ReplacingMergeTree, CollapsingMergeTree) or to make the INSERT idempotent. If there are no recorded block starts (because kafka_commit_every_batch committed mid-block), the rewind is a plain “go back to the last committed offset”.

The other mode: commit every batch

The default is to commit once per block, after the write. The setting kafka_commit_every_batch (default false) changes the trade-off: when it is on, the consumer commits each polled batch as it is consumed, before the block is written:

 cpp
 
 
// src/Storages/Kafka/KafkaConsumer.cpp:547
 if (hasMorePolledMessages())
 return getNextMessage();

 if (intermediate_commit)
 commit();

 doPoll();

The intermediate_commit happens right before a new poll, i.e. at the boundary between batches. This is the at-most-once mode: if the server dies after the commit but before the write, the messages are gone.

So the two settings give the two ends of the classic trade-off:

setting	commit point	failure mode
`kafka_commit_every_batch = false` (default)	after the block is written	duplicates on crash
`kafka_commit_every_batch = true`	after each polled batch	loss on crash

The default picks duplicates: in a stream-processing pipeline duplicates are usually recoverable (dedup downstream) and losses are not.

Reading from a Kafka table directly

A SELECT from a Kafka table is a different path, and a strange one: the table is read destructively, consuming the messages.

 cpp
 
 
// src/Storages/Kafka/StorageKafka.cpp:146
 for (size_t i = 0; i < kafka_storage.num_consumers; ++i)
 {
 /// Use block size of 1, otherwise LIMIT won't work properly as it will buffer excess messages in the last block
 /// TODO: probably that leads to awful performance.
 /// FIXME: seems that doesn't help with extra reading and committing unprocessed messages.
 pipes.emplace_back(std::make_shared<KafkaSource>(
 kafka_storage,
 storage_snapshot,
 modified_context,
 column_names,
 kafka_storage.log,
 1,
 (*kafka_storage.kafka_settings)[KafkaSetting::kafka_commit_on_select]));
 }

Note the block size of 1: with a larger block, LIMIT 5 would have to read and buffer a whole block of messages to return five rows, and the extra messages would be committed anyway. With a block size of 1 the LIMIT works, at the cost of one pipeline setup per row.

And the commit is done in the suffix of the source — KafkaSource::generate() commits only when the source is exhausted and the whole result has been consumed by the client — which is why the setting is called kafka_commit_on_select and defaults to false. With it off, a SELECT from a Kafka table consumes the messages but never commits their offsets, so the group will read them again later; with it on, the offsets are committed when the select finishes, and the messages are gone for good.

The consumer pool

One more piece of machinery that is easy to miss: the Kafka consumers are pooled. The engine creates num_consumers consumer objects up front, and a SELECT or a streaming task borrows one with popConsumer(timeout).

The pool exists because creating a cppkafka::Consumer and subscribing is expensive (it involves a group join and a rebalance). popConsumer() has three cases, in order: a free consumer exists, so return it; no consumer is free but a slot has been closed, so create a new one under the lock (cheap, because creating the client does not subscribe yet); otherwise wait up to the timeout for one to be released.

Unused consumers are closed by a cleanup thread after kafka_consumers_pool_ttl_ms (60 seconds by default), so a burst of SELECTs does not leave a hundred subscribed consumers hanging around.

The modern version: an RAII offset guard

The engine that ships alongside the old one — StorageKafka2 — solves the same problem with a different tool: a guard object whose destructor rolls the offset back.

 cpp
 
 
// src/Storages/Kafka/KeeperHandlingConsumer.h:60
 class OffsetGuard
 {
 public:
 OffsetGuard(KeeperHandlingConsumer & consumer_, int64_t new_offset_);
 OffsetGuard(OffsetGuard && other) noexcept;
 ~OffsetGuard();

 void commit();
 private:
 bool needs_rollback{true};
 KeeperHandlingConsumer * consumer;
 int64_t new_offset;
 };

and its destructor and commit() are the whole mechanism:

 cpp
 
 
// src/Storages/Kafka/KeeperHandlingConsumer.cpp:97
KeeperHandlingConsumer::OffsetGuard::~OffsetGuard()
{
 if (consumer && needs_rollback)
 consumer->rollbackToCommittedOffsetsNoThrow();
}

void KeeperHandlingConsumer::OffsetGuard::commit()
{
 chassert(needs_rollback && consumer);
 consumer->saveCommittedOffset(new_offset);
 needs_rollback = false;
}

The guard is created when the messages are polled and rolls the offset back unless someone explicitly calls commit() on it. The source takes the guard only after it knows it will not throw — “the cancel may have arrived while pollConsumer was polling. Drop the block by not taking its offset guard” — and the destructor of the source releases it: “if commit was not called, the OffsetGuard destructor rolls back”.

This is the same “commit after the write” ordering as the old engine, but expressed as a scope: the offset advances if and only if the block survived to the end of the scope. Every early return, every exception, every cancellation rolls it back automatically. In this version the offsets live in ClickHouse Keeper rather than in Kafka itself, so the engine can be replicated.

Trade-offs

What the ordering buys: a crash never loses a message that ClickHouse had not yet written. The failure mode is duplicates, which a downstream ReplacingMergeTree or a deduplicating materialized view can absorb.

What it costs:

There is no exactly-once. The window between “the block is in the MergeTree” and “the offset is committed” is a window in which a crash duplicates. The exception message names the missing piece: “insert atomicity / transactions is needed here”.
The commit is a synchronous round trip to the broker, in the middle of the streaming loop. A broker that cannot accept the commit (no replicas for __consumer_offsets) stalls the whole stream.
kafka_commit_every_batch = true trades duplicates for loss — the right choice when a message’s content makes an INSERT idempotent anyway, and the wrong choice when it does not.
The rewind is per-partition and per-block, and the “rewind” is itself a commit, so an aborted block costs an extra broker round trip.
A SELECT from a Kafka table is destructive and, with a block size of 1, slow. It is a debugging tool, not a query interface.

Wrapping up

The one thing to remember: the commit is after the write — a deliberate choice of “duplicates on crash” over “data loss on crash”, enforced by the structure of the code (executor.execute() then source->commit()) rather than by a comment.

The part people get wrong: a crash between the write and the commit duplicates the block, and nothing in ClickHouse cleans that up for you. If your pipeline cannot tolerate duplicates, the destination engine has to be a deduplicating one.

What to go do about it: watch KafkaCommitFailures in system.events. It should be zero. If it is not, the error message in the server log is the exact sentence quoted above.

How the ClickHouse Buffer Engine Holds Writes in RAM and Flushes Them in the Background

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: Buffer keeps its rows in RAM, spread over num_layers independent shards, each shard guarded by its own mutex. A write either goes straight to the destination (if the block is already bigger than the max_* thresholds) or is appended to the least busy shard. A background task wakes up when the soonest threshold is due and flushes each shard by swapping its block out first — so if the destination is down, the block is swapped back in and retried later instead of being lost.

The classic ClickHouse deployment pattern is “a fleet of application servers inserts into ClickHouse”. Each server, in a burst, inserts a few hundred rows. A MergeTree does not care how many rows an INSERT has; it cares that each INSERT produces a part, and a part is a set of files that must be merged later. Ten thousand servers inserting one row a second produce ten thousand tiny parts a second, and the merge queue falls over.

The naive fix is to batch in the application. The ClickHouse-native fix is the Buffer engine: a table that lives in RAM, absorbs small INSERTs, and periodically writes them to a real table in big chunks. The interesting part is the flush policy — three families of thresholds, a scheduler that computes its own next wake-up time, and a failure mode that must be survivable.

The shape of the thing

The class comment in the header is the design document. The buffer is a set of num_shards blocks, and the rule for a busy shard is “try the next block in a round-robin fashion, and so no more than num_shards times (then wait for lock)”:

 cpp
 
 
// src/Storages/StorageBuffer.h:28
 * When using a block, it is locked by some mutex. If during write the corresponding block is already occupied
 * - try to lock the next block in a round-robin fashion, and so no more than `num_shards` times (then wait for lock).
 * Thresholds are checked on insertion, and, periodically, in the background thread (to implement time thresholds).
 * Thresholds act independently for each shard. Each shard can be flushed independently of the others.
 * If a block is inserted into the table, which itself exceeds the max-thresholds, it is written directly to the subordinate table without buffering.

Five facts fall out of that comment: the buffer is a set of shards, each with its own mutex; thresholds are per shard; a block bigger than the max thresholds bypasses the buffer entirely; there are three threshold families (min, max, flush); and the data is not durable — “with a rough restart of the server, the data is lost”.

A shard is remarkably small:

 cpp
 
 
// src/Storages/StorageBuffer.h:155
 struct Buffer
 {
 time_t first_write_time = 0;
 Block data;

plus an int32_t metadata_version, which exists because an ALTER changes the column set and blocks written before and after it cannot be concatenated. The chosen policy is “skip stale buffers rather than convert them” — a data-visibility bug in exchange for not writing a schema migrator.

The engine arguments decode like this:

 sql
 
 
CREATE TABLE merge.hits_buffer AS merge.hits
ENGINE = Buffer(merge, hits, 1, 10, 100, 10000, 1000000, 10000000, 100000000);

 text
 
 
Buffer(merge, hits,
 num_layers = 1, -- independent shards
 min_time = 10, max_time = 100, -- seconds since the first write
 min_rows = 10000, max_rows = 1000000, -- rows in the shard
 min_bytes = 10000000, max_bytes = 100000000) -- bytes in the shard

num_layers multiplies the RAM ceiling: with 16 layers and max_bytes = 100 000 000, the worst case is 1.6 GB. This is the single most common misconfiguration of the Buffer engine.

The write path

BufferSink::consume is where the policy lives. First, the bypass.

 cpp
 
 
// src/Storages/StorageBuffer.cpp:897
 /// If the block already exceeds the maximum limit, then we skip the buffer.
 if (rows > storage.max_thresholds.rows || bytes > storage.max_thresholds.bytes)
 {
 if (destination)
 {
 LOG_DEBUG(storage.log, "Writing block with {} rows, {} bytes directly.", rows, bytes);
 storage.writeBlockToDestination(block, destination);
 }
 return;
 }

This is what makes Buffer safe to point at a bulk loader: a 10-million-row INSERT does not sit in RAM waiting for a threshold that will never be reached. The cost is that the destination sees one huge INSERT.

Then the shard selection. The obvious implementation is “hash the thread id to a shard”; the code instead tries every shard with a non-blocking lock and picks the one with the fewest rows:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:908
 /// We distribute the load on the shards by the stream number.
 const auto start_shard_num = getThreadId() % storage.num_shards;

 /// We loop through the buffers, trying to lock mutex. No more than one lap.
 auto shard_num = start_shard_num;

 StorageBuffer::Buffer * least_busy_buffer = nullptr;
 std::unique_lock<std::mutex> least_busy_lock;
 size_t least_busy_shard_rows = 0;

 for (size_t try_no = 0; try_no < storage.num_shards; ++try_no)
 {
 std::unique_lock lock(storage.buffers[shard_num].tryLock());

 if (lock.owns_lock())
 {
 size_t num_rows = storage.buffers[shard_num].data.rows();
 if (!least_busy_buffer || num_rows < least_busy_shard_rows)
 {
 least_busy_buffer = &storage.buffers[shard_num];
 least_busy_lock = std::move(lock);
 least_busy_shard_rows = num_rows;
 }
 }

 shard_num = (shard_num + 1) % storage.num_shards;
 }

Three properties fall out of this loop: the starting shard is thread-dependent, so two simultaneous writers begin their scan at different shards; the lock is held for the whole scan, so the “least busy” measurement is not a race — the writer that picks a shard also owns it; and the scan is exactly one lap.

If every shard is locked (all of them are being flushed at once), the writer blocks on its own starting shard, appends, unlocks, and calls storage.reschedule(0). Note that last call: the background task does not run on a fixed timer — it is re-armed by the writes themselves, and the “0” means “schedule me as soon as possible if a threshold is due”.

The append itself takes column checkpoints first, because appending a block to a block is a per-column insertRangeFrom, and a MEMORY_LIMIT_EXCEEDED in the middle of column 7 of 10 would leave a corrupt buffer. On failure, each column is rolled back to its checkpoint. The block’s columns are also sorted before being appended, so that concatenating two buffered blocks is a cheap insertRangeFrom rather than a column-by-column reshuffle.

The threshold algebra

Here is the part people get wrong when they read the docs. There are three families of thresholds, and they mean different things.

 cpp
 
 
// src/Storages/StorageBuffer.cpp:1152
bool StorageBuffer::checkThresholdsImpl(bool direct, size_t rows, size_t bytes, time_t time_passed) const
{
 if (time_passed > min_thresholds.time && rows > min_thresholds.rows && bytes > min_thresholds.bytes)
 {
 ProfileEvents::increment(ProfileEvents::StorageBufferPassedAllMinThresholds);
 return true;
 }

 if (time_passed > max_thresholds.time)
 {
 ProfileEvents::increment(ProfileEvents::StorageBufferPassedTimeMaxThreshold);
 return true;
 }

 if (rows > max_thresholds.rows)
 {
 ProfileEvents::increment(ProfileEvents::StorageBufferPassedRowsMaxThreshold);
 return true;
 }

 if (bytes > max_thresholds.bytes)
 {
 ProfileEvents::increment(ProfileEvents::StorageBufferPassedBytesMaxThreshold);
 return true;
 }

Read the boolean structure:

min_* is an AND of all three. Time, rows and bytes must all be exceeded: the “the buffer is full” condition, which only fires when the shard is genuinely full on every axis.
max_* is an OR of any of the three. Time, rows or bytes being exceeded is enough: the “I cannot let this grow any further” safety valve.
flush_* is checked only for non-direct flushes (the !direct guard, plus “the threshold is non-zero”, so an omitted flush_* never fires). This is the “background worker, wake up” condition, and it exists so that the INSERT path does not block on a destination write.

That last distinction is why flush_* exists. With max_rows = 1e6 and flush_rows = 500e3, the shard is only forced to flush at 1e6 rows, but the background worker will opportunistically flush at 500e3 rows — and because that happens on another thread, the INSERT does not pay the latency of the destination write.

The direct flag is true on the insert path, where the flush threshold is deliberately ignored:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:958
 if (storage.checkThresholds(buffer, /* direct= */true, current_time, sorted_block.rows(), sorted_block.bytes()) ||
 buffer.metadata_version != metadata_version)

and false on the background worker:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:1245
 if (!checkThresholdsImpl(/* direct= */false, rows, bytes, time_passed))
 return false;

One more subtlety: checkThresholds is called before the new block is appended, with the incoming row and byte counts added in. The check is a prediction — “would the shard exceed the threshold if I add these rows?” — not an observation, and the threshold can still be exceeded: if max_rows = 1 000 000 and the buffer already had 500 000 rows, adding 800 000 rows leaves 1 300 000 in the buffer. The engine does not split the incoming block to respect the threshold; it accepts the overshoot and then flushes.

The background worker

The background flush is a BackgroundSchedulePool task:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:196
 flush_handle = bg_pool->createTask(getStorageID(), log->name() + "/Bg", [this]{ backgroundFlush(); });

Its body: flush everything, swallow any exception, re-arm itself.

 cpp
 
 
// src/Storages/StorageBuffer.cpp:1383
void StorageBuffer::backgroundFlush()
{
 try
 {
 LOG_DEBUG(log, "Running background flush of buffers.");
 flushAllBuffers(true);
 }
 catch (...)
 {
 tryLogCurrentException(__PRETTY_FUNCTION__);
 }

The catch (...) is the “the destination is down” behaviour. What happens to the data is in flushBuffer, not here.

Computing the next wake-up

The scheduler does not poll every second: it computes the earliest moment at which any threshold could possibly fire, and sleeps until then:

The loop over the shards uses tryLock() rather than a blocking lock — “to avoid waiting for other layers flushing to be finished, since the buffer table may push to a Distributed table, that may take too much time” — and collects two things: the oldest first_write_time among the non-empty shards, and the total row count.

The oldest non-empty shard sets the clock: time_passed = now - min_first_write_time is how long the oldest data has been waiting, and the three candidate wake-up times are “threshold minus time already passed”:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:1434
 time_t current_time = time(nullptr);
 time_t time_passed = current_time - min_first_write_time;

 /// checkThresholdsImpl() uses strict comparison (> not >=), so we need to add offset to original time values
 static constexpr time_t THRESHOLD_COMPARISON_OFFSET = 1;

 /// For minimal threshold min_delay is ignored, since otherwise it will be triggered too frequently, once it is reached
 /// (while the Buffer cannot be flushed due to other min thresholds).
 size_t min = std::max<ssize_t>(min_thresholds.time + THRESHOLD_COMPARISON_OFFSET - time_passed, 1);
 size_t max = std::max<ssize_t>(max_thresholds.time + THRESHOLD_COMPARISON_OFFSET - time_passed, min_delay);
 size_t reschedule_sec = 0;
 if (flush_thresholds.time)
 {
 size_t flush = std::max<ssize_t>(flush_thresholds.time + THRESHOLD_COMPARISON_OFFSET - time_passed, min_delay);
 reschedule_sec = std::min({min, max, flush});
 }

Three things to notice:

The +1 offset exists because the comparison in checkThresholdsImpl is strict (> not >=). Without it, a 100-second threshold would be considered reached at 99 seconds, and the worker would wake up one second early, every time.
The min threshold deliberately ignores min_delay, because the min_* condition is an AND: waking up when only the time part of it is satisfied accomplishes nothing.
The wake-up is the minimum of the three candidates, so whichever threshold fires first wins.

And if there is nothing buffered at all, no reschedule happens at all: an idle Buffer table costs zero scheduler wake-ups, and the task is only re-armed by an INSERT.

The failure mode: the destination is down

This is the interesting part: a two-line trick with a large consequence.

flushBuffer does not write in place. It swaps the buffer’s block out into a local variable, and only then writes it:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:1249
 buffer.data.swap(block_to_write);
 buffer.first_write_time = 0;

swap is O(1) — it exchanges the contents of two Blocks. After the swap the shard is empty and the data lives in the local variable. The shard stays locked for the duration of the write, which the code justifies:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:1270
 /** For simplicity, buffer is locked during write.
 * We could unlock buffer temporary, but it would lead to too many difficulties:
 * - data, that is written, will not be visible for SELECTs;
 * - new data could be appended to buffer, and in case of exception, we must merge it with old data, that has not been written;
 * - this could lead to infinite memory growth.
 */

And if the write throws — the destination table is gone, the disk is full, the network is down — the block is swapped back into the shard and the exception is rethrown:

 cpp
 
 
// src/Storages/StorageBuffer.cpp:1282
 catch (...)
 {
 ProfileEvents::increment(ProfileEvents::StorageBufferErrorOnFlush);

 /// Return the block to its place in the buffer.

 CurrentMetrics::add(CurrentMetrics::StorageBufferRows, block_to_write.rows());
 CurrentMetrics::add(CurrentMetrics::StorageBufferBytes, block_to_write.bytes());

 buffer.data.swap(block_to_write);

 if (!buffer.first_write_time)
 buffer.first_write_time = current_time;

 /// After a while, the next write attempt will happen.
 throw;
 }

The first_write_time is restored too, so the age of the data — which is what the time thresholds measure — survives the failed attempt. The data is not lost and the next background tick will try again.

Combined with the catch (...) in backgroundFlush, the behaviour is:

 text
 
 
 INSERT -> shard buffer (RAM)
 | thresholds: min (AND) | max (OR) | flush (background only)
 v
 backgroundFlush() --(exception swallowed, rescheduled)
 v
 flushBuffer(): swap out --> write to destination
 ^ |
 +--- swap back -- (exception)

One caveat that the code itself flags: the swap-back assumes the destination write failed before writing anything. If the destination wrote part of the block and then failed, the data is written twice — once in the destination, once in the buffer. There is no transaction spanning the two, which is why the engine’s documentation warns that “it is not possible to have a reliable ’exactly once’ write to replicated tables.”

Trade-offs

What the trick buys: a MergeTree is fed large parts instead of a hundred thousand tiny ones, and the INSERT path never blocks on the destination. The threshold algebra gives three separate knobs — “flush when full”, “never let this grow past that”, “and do it in the background so my INSERT stays fast”.

What it costs:

The data is not durable. It is in RAM, in a process, and “with a rough restart of the server, the data is lost”. A Buffer table is a performance device in front of a durable table, not a queue; the engine’s documentation now recommends asynchronous inserts instead.
The order is destroyed. Rows arrive in the destination in whatever order the shards flushed in, and in whatever block sizes the shards happened to have. This breaks CollapsingMergeTree (hence the documentation’s advice to use num_layers = 1) and it breaks deduplication on replicated tables.
A failed flush can duplicate. The swap-back is only safe if the destination failed atomically; there is no 2-phase commit between the buffer and the destination.
The thresholds are per shard. max_bytes = 100 000 000 with 16 layers is 1.6 GB, not 100 MB. And because min_* is an AND of three axes, a shard that never reaches the row count is only ever flushed by a max_* or flush_* threshold, or by the server shutting down.

Wrapping up

The one thing to remember: the Buffer engine is a RAM buffer with a three-tier threshold algebra — min_* (all of time, rows, bytes), max_* (any of them), flush_* (background only) — evaluated per shard.

The part people get wrong: it is not durable and it is not ordered. If you need at-least-once delivery from a queue, use Kafka; if you need to avoid tiny parts, use Buffer or asynchronous inserts.

What to go do about it: insert one row at a time into a Buffer table and watch which threshold is actually doing the work in your deployment.

 sql
 
 
SELECT event, count
FROM system.events
WHERE event LIKE 'StorageBuffer%'
ORDER BY event;