Insert on Opensource internals

How ClickHouse Maps an INSERT onto the Table's Schema

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: An INSERT is resolved into a sample block — a Block with the right columns and types — by looking each requested column up by name in the table’s insertable columns. That sample block is both the parser’s schema and the pipeline’s header. The pipeline then counts rows, stamps deduplication info, squashes blocks to the target size, and fans the single input stream out to one sink chain per destination.

An INSERT statement is the one query where the shape of the data is not known until the last moment. The client says INSERT INTO t VALUES (1, 'a'); the server has to work out what 1 and 'a' mean.

ClickHouse solves this with a single object: the sample block. It is a Block — a list of (name, type, empty column) — that plays three roles at once:

it is the schema the input format parses against,
it is the header of the insert pipeline,
it is the target the conversion step casts the data into.

Everything else in the insert path is a consequence of that one idea.

The problem in three shapes

There are three ways an insert can name its columns, and they resolve differently.

No column list. INSERT INTO t VALUES (1, 'a'). The client is saying “the values are in the table’s column order”. The target is the table’s insertable columns — ordinary columns plus EPHEMERAL ones, but not MATERIALIZED and not ALIAS.

An explicit column list. INSERT INTO t (b, a) VALUES ('a', 1). The client is naming columns, and the order is theirs, not the table’s. This is the case that needs a name lookup.

A SELECT as the source. INSERT INTO t SELECT .... Here the source produces a fixed shape, and the mapping between the source’s columns and the table’s is positional — because a SELECT has no column names to speak of at the point where the pipeline is built.

The engine is responsible for the third case (the storage knows how to convert), but the first case is a table-schema question: which columns of t are insertable at all?

Which columns are insertable

The answer is in ColumnsDescription:

 cpp
 
 
// src/Storages/ColumnsDescription.cpp:615
NamesAndTypesList ColumnsDescription::getInsertable() const
{
 NamesAndTypesList ret;
 for (const auto & col : columns)
 if (col.default_desc.kind == ColumnDefaultKind::Default || col.default_desc.kind == ColumnDefaultKind::Ephemeral)
 ret.emplace_back(col.name, col.type);
 return ret;
}

So: DEFAULT columns and EPHEMERAL columns are insertable; MATERIALIZED columns are computed by the engine and cannot be written; ALIAS columns do not exist physically at all.

The sample block is built from that list:

 cpp
 
 
// src/Storages/StorageInMemoryMetadata.cpp:538
Block StorageInMemoryMetadata::getSampleBlockInsertable() const
{
 Block res;

 for (const auto & column : getColumns().getInsertable())
 res.insert({column.type->createColumn(), column.type, column.name});

 return res;
}

Note column.type->createColumn() — the sample block holds empty columns of the right type. It is a schema, not data. (Compare getSampleBlockNonMaterialized(), which is the same thing for the “ordinary” columns — used by the streaming storages that need the physical shape.)

The sample block: by name

When the insert names its columns, the lookup happens by name:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:239
Block InterpreterInsertQuery::getSampleBlock(
 const Names & names,
 const StoragePtr & table,
 const StorageMetadataPtr & metadata_snapshot,
 bool allow_virtuals,
 bool allow_materialized)
{
 std::vector<size_t> missing_positions;
 Block table_sample_insertable = metadata_snapshot->getSampleBlockInsertable();

 ColumnsWithTypeAndName res{names.size()};
 std::unordered_set<String> inserted_names;

 for (size_t i = 0; i < names.size(); i++)
 {
 const auto & current_name = names[i];
 if (!inserted_names.insert(current_name).second)
 throw Exception(
 ErrorCodes::DUPLICATE_COLUMN,
 "Column {} in table {} specified more than once",
 current_name,
 table->getStorageID().getNameForLogs());

 const ColumnWithTypeAndName * insertable_col = table_sample_insertable.findByName(current_name);
 if (!insertable_col)
 missing_positions.emplace_back(i);
 else
 res[i] = *insertable_col;
 }

Three things happen here that are worth naming.

The result is in the client’s order, not the table’s. res[i] is the column the client asked for at position i. The pipeline header is therefore INSERT INTO t (b, a) order, and the sink is the one that has to reorder. This matters for the INSERT ... SELECT case, where the conversion is positional instead.

Duplicate names are rejected. inserted_names is a set, and a second occurrence of a name throws DUPLICATE_COLUMN. Without that check, INSERT INTO t (a, a) would silently take the second a’s type.

A name that is not insertable is not immediately an error. It goes into missing_positions and is resolved in a second pass:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:269
 if (!missing_positions.empty())
 {
 Block table_sample_physical = metadata_snapshot->getSampleBlock();
 Block table_sample_virtuals;
 if (allow_virtuals)
 table_sample_virtuals = metadata_snapshot->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::All);

 /// Columns are not ordinary or ephemeral
 for (auto pos : missing_positions)
 {
 const auto & current_name = names[pos];

 if (table_sample_physical.has(current_name))
 {
 /// Column is materialized
 if (!allow_materialized)
 throw Exception(ErrorCodes::ILLEGAL_COLUMN, "Cannot insert column {}, because it is MATERIALIZED column", current_name);
 res[pos] = table_sample_physical.getByName(current_name);
 }
 else if (table_sample_virtuals.has(current_name))
 {
 res[pos] = table_sample_virtuals.getByName(current_name);
 }
 else
 {
 /// The table does not have a column with that name
 throw Exception(
 ErrorCodes::NO_SUCH_COLUMN_IN_TABLE,
 "No such column {} in table {}",
 current_name,
 table->getStorageID().getNameForLogs());
 }
 }
 }

So the resolution order is: insertable columns first, then materialized columns (if the caller allows it), then virtual columns, then an error. The allow_materialized flag is the difference between INSERT INTO t (mat_col) VALUES (...) working and failing — and it is controlled by the insert_allow_materialized_columns setting.

The two-pass structure is not an optimisation; it is a precedence rule. If a table has a column that is both insertable and (say) virtual, the insertable one wins. And the error messages are deliberately distinct: “No such column X in table Y” versus “Cannot insert column X, because it is MATERIALIZED column” — a user who typos a column name gets a different message than a user who tries to write a computed column.

From the AST to the names

Before the lookup, the column list from the query has to be normalised. The client may write qualified names, and the table may have column transformers (MATERIALIZED expr computed from other columns, ALIAS):

 cpp
 
 
// src/Interpreters/processColumnTransformers.cpp:26
 TablesWithColumns tables_with_columns;
 {
 auto table_expr = make_intrusive<ASTTableExpression>();
 table_expr->database_and_table_name = make_intrusive<ASTTableIdentifier>(storage_id);
 table_expr->children.push_back(table_expr->database_and_table_name);
 tables_with_columns.emplace_back(DatabaseAndTableWithAlias(*table_expr, current_database), names_and_types);
 }

 tables_with_columns[0].addHiddenColumns(columns.getMaterialized());
 tables_with_columns[0].addHiddenColumns(columns.getAliases());
 tables_with_columns[0].addHiddenColumns(virtual_columns);

The materialized and alias columns are added as hidden columns, so a qualified name like t.alias_col resolves to the right thing even though the column is not insertable.

The `INSERT ... SELECT` case: positional, not by name

When the source is a SELECT, the mapping is by position, because the SELECT’s output has no user-supplied names at that point:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:428
 auto actions_dag = ActionsDAG::makeConvertingActions(
 pipeline.getHeader().getColumnsWithTypeAndName(),
 query_sample_block.getColumnsWithTypeAndName(),
 ActionsDAG::MatchColumnsMode::Position,
 context_);
 auto actions = std::make_shared<ExpressionActions>(std::move(actions_dag), ExpressionActionsSettings(context_, CompileExpressions::yes));

 pipeline.addSimpleTransform([&](const SharedHeader & in_header) -> ProcessorPtr
 {
 return std::make_shared<ExpressionTransform>(in_header, actions);
 });

 return query_sample_block;

MatchColumnsMode::Position versus MatchColumnsMode::Name — the same function, two policies. The by-name mode is what the async insert queue uses to normalise pre-parsed blocks onto the header it computed:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1199
 auto converting_dag = ActionsDAG::makeConvertingActions(
 block.getColumnsWithTypeAndName(),
 header.getColumnsWithTypeAndName(),
 ActionsDAG::MatchColumnsMode::Name,
 context);

Before the conversion, there is one subtlety: insert_null_as_default. A Nullable source column can be inserted into a non-nullable target column, with the NULLs replaced by the column’s default. The trick is to widen the target to Nullable for the conversion, and let the defaults step narrow it back:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:398
 /// Allow to insert Nullable into non-Nullable columns, NULL values will be added as defaults values.
 if (context_->getSettingsRef()[Setting::insert_null_as_default])
 {
 const auto & input_columns = pipeline.getHeader().getColumnsWithTypeAndName();
 const auto & query_columns = query_sample_block.getColumnsWithTypeAndName();
 const auto & output_columns = metadata_snapshot->getColumns();

 if (input_columns.size() == query_columns.size())
 {
 for (size_t col_idx = 0; col_idx < query_columns.size(); ++col_idx)
 {
 /// Change query sample block columns to Nullable to allow inserting nullable columns, where NULL values will be substituted with
 /// default column values (in AddingDefaultsTransform), so all values will be cast correctly.
 if (isNullableOrLowCardinalityNullable(input_columns[col_idx].type)
 && !isNullableOrLowCardinalityNullable(query_columns[col_idx].type)
 && !isVariant(query_columns[col_idx].type)
 && !isDynamic(query_columns[col_idx].type)
 && output_columns.has(query_columns[col_idx].name))
 {
 query_sample_block.setColumn(
 col_idx,
 ColumnWithTypeAndName(
 makeNullableOrLowCardinalityNullable(query_columns[col_idx].column),
 makeNullableOrLowCardinalityNullable(query_columns[col_idx].type),
 query_columns[col_idx].name));
 }
 }
 }
 }

The Variant and Dynamic exclusions are there because those types have their own null representation and cannot be blanket-wrapped.

The pipeline

Now the sample block becomes a pipeline. The interesting part is the shape: a single input stream, a single head, and then a fan-out to one sink chain per destination.

The head is built in buildPushPipelineFromDependencies:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1149
 /// Shrink over-allocated columns produced by parsing (e.g. String columns grown power-of-two) to
 /// fit, right after the source where the chunk is uniquely owned, to reduce peak memory usage.
 if (static_cast<double>(settings[Setting::shrink_over_allocated_columns_min_waste_ratio]) > 1.0)
 add_head_transform(std::make_shared<ShrinkColumnsTransform>(
 insert_header,
 static_cast<double>(settings[Setting::shrink_over_allocated_columns_min_waste_ratio]),
 settings[Setting::shrink_over_allocated_columns_min_waste_bytes]));

 {
 auto counting = std::make_shared<CountingTransform>(insert_header, context_->getQuota(), context_->getNormalizedQueryHash());
 counting->setProcessListElement(context_->getProcessListElement());
 counting->setProgressCallback(context_->getProgressCallback());
 add_head_transform(std::move(counting));
 }

Three transforms, in this order, and the order is a memory optimisation:

Shrink — the parser allocates String columns with power-of-two growth, so a 1 KB string may live in a 2 KB buffer. Shrinking right after the source, where the chunk is uniquely owned, halves the peak memory of a wide insert. The condition > 1.0 means “only shrink if the waste ratio is worth the copy”.
Count — the quota accounting. The counter is attached to the normalized query hash, so WRITTEN_ROWS quotas can be per-query-shape.
Dedup stamp — the deduplication info.

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1164
 if (!squash_with_strict_limits)
 add_head_transform(std::make_shared<AddDeduplicationInfoTransform>(
 insert_dependencies,
 insert_dependencies->getRootViewID(),
 settings[Setting::insert_deduplication_token].value,
 insert_header));

The AddDeduplicationInfoTransform stamps each chunk with the source block number — the ordinal of the block within this insert. That number is part of the deduplication token, which is what makes a retried insert recognisable as a retry rather than as new data.

Then the squashing:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1171
 if (should_squash)
 {
 bool table_prefers_large_blocks = table->prefersLargeBlocks();
 size_t min_block_size_bytes = table_prefers_large_blocks ? settings[Setting::min_insert_block_size_bytes] : 0ULL;
 /// On low-memory systems, cap squashing block size to avoid accumulating too much data.
 if (auto memory_limit = total_memory_tracker.getHardLimit(); memory_limit > 0)
 min_block_size_bytes = std::min<size_t>(min_block_size_bytes, static_cast<size_t>(static_cast<double>(memory_limit) * 0.9) / 8);
 add_head_transform(std::make_shared<PlanSquashingTransform>(
 insert_header,
 table_prefers_large_blocks ? settings[Setting::min_insert_block_size_rows] : settings[Setting::max_block_size],
 min_block_size_bytes,
 settings[Setting::max_insert_block_size],
 settings[Setting::max_insert_block_size_bytes],
 squash_with_strict_limits));
 }

The squashing transform is a plan, not a simple transform: it accumulates chunks and emits them when they reach the target size. The target comes from the table’s preference — a MergeTree wants big blocks (min_insert_block_size_rows), a Memory table does not.

The 0.9 * memory_limit / 8 cap is a nice touch: the squasher’s buffer is allowed to grow to at most 90 % of the server’s memory limit divided by 8 (roughly, eight concurrent squashing buffers).

The fan-out

After the head, the single input stream is distributed to the sink chains:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1203
 /// Distribute the single input stream across the parallel insert streams.
 std::vector<OutputPort *> stream_outputs;
 if (sink_stream_size > 1)
 {
 auto resize = std::make_shared<ResizeProcessor>(head_output->getSharedHeader(), 1, sink_stream_size);
 connect(*head_output, resize->getInputs().front());
 for (auto & output : resize->getOutputs())
 stream_outputs.push_back(&output);
 processors->emplace_back(std::move(resize));
 }
 else
 {
 stream_outputs.push_back(head_output);
 }

A ResizeProcessor with one input and N outputs is a round-robin distributor. This is how a single INSERT is written by several threads in parallel.

The number of sink streams is not simply max_insert_threads. It is a carefully-computed function of the deduplication mode, the view graph, the quorum mode and the storage topology:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1064
 const bool dedup_single_stream = !async_insert
 && ((per_branch_dedup_ids && source_deduplicates)
 || forwarded_dependent_mv_dedup_hazard
 || forwards_to_separate_context
 || hidden_views_forward_to_separate_context);

and the reason is spelled out in the long comment above it — the source block number is stamped globally in the head, but a fan-out would let two branches produce blocks with the same number, and the deduplication token would then collide, and the second block would be silently dropped as a duplicate. That is a data-loss bug, so the planner fails closed and keeps such inserts single-stream.

The quorum case has its own reason:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1070
 /// A non-parallel quorum insert (`insert_quorum >= 2` or `'auto'`, with `insert_quorum_parallel = 0`)
 /// permits a single in-flight quorum part per table: every `ReplicatedMergeTreeSink` checks in
 /// `onStart` that the quorum of all previous writes is already satisfied (`checkQuorumPrecondition`)
 /// and throws `UNSATISFIED_QUORUM_FOR_PREVIOUS_WRITE` otherwise. With a write fan-out every branch
 /// runs its own sink - including branches that receive no data - so sibling sinks of the same
 /// `INSERT` race against the not-yet-satisfied quorum node of the part committed by the branch that
 /// got the data. Keep such inserts single-stream.

and the final stream count:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1080
 const size_t insert_threads
 = (async_insert || dedup_single_stream || serial_hidden_views || sequential_quorum_insert) ? 1 : max_insert_threads;

Async inserts are always single-stream — the async queue does its own batching, so a second level of parallelism here would just complicate the flush.

The views: `InsertDependenciesBuilder`

The destination of an insert is rarely one table. There may be materialized views, each with its own target table. InsertDependenciesBuilder walks that graph and produces one sink chain per leaf:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1118
 auto sink_chains = insert_dependencies->createChainWithDependenciesForAllStreams();
 const size_t sink_stream_size = insert_dependencies->getSinkStreamSize();
 chassert(sink_chains.size() == sink_stream_size);
 chassert(sink_stream_size >= 1);

and each chain is terminated with an EmptySink and connected to its share of the fan-out:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1220
 /// Connect each parallel stream to its sink chain and terminate it with an empty sink.
 QueryPlanResourceHolder resources;
 size_t stream_index = 0;
 for (auto & sink_chain : sink_chains)
 {
 connect(*stream_outputs[stream_index], sink_chain.getInputPort());
 ++stream_index;

 auto sink = std::make_shared<EmptySink>(sink_chain.getOutputSharedHeader());
 connect(sink_chain.getOutputPort(), sink->getPort());

The chains are chains, not sinks: each one may contain several processors (the view’s own ExpressionTransforms, its own AddingDefaultsTransform, and so on) before the actual storage sink.

There is one more subtlety: with use_strict_insert_block_limits, the dedup stamp has to happen per branch, after the fan-out, not in the head:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1187
 /// Prepend the per-stream transforms to each sink chain. `addSource` prepends, so the
 /// resulting top-to-bottom order matches the previous single-stream pipeline:
 /// ApplySquashing -> AddDeduplicationInfo (strict) -> sink.
 for (auto & sink_chain : sink_chains)
 {
 if (squash_with_strict_limits)
 sink_chain.addSource(std::make_shared<AddDeduplicationInfoTransform>(
 insert_dependencies,
 insert_dependencies->getRootViewID(),
 settings[Setting::insert_deduplication_token].value,
 sink_chain.getInputSharedHeader()));

 if (should_squash)
 sink_chain.addSource(std::make_shared<ApplySquashingTransform>(sink_chain.getInputSharedHeader()));
 }

Strict limits mean the block boundaries are part of the deduplication semantics, so the stamp must be applied to the squashed block, in the branch that writes it.

`execute()`: the routing

InterpreterInsertQuery::execute is the entry point, and it does the access checks, takes the lock, and chooses between the two pipeline builders:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1492
 table->updateExternalDynamicMetadataIfExists(context);
 auto metadata_snapshot = table->getInMemoryMetadataPtr(context, false);
 auto query_sample_block = getSampleBlock(query, table, metadata_snapshot, context, no_destination, allow_materialized);

The sample block is computed before the access check, because the access check is column-aware: the user may have GRANT SELECT (a, b) but not SELECT (c), and the insert’s column list says which columns are being written:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1501
 if (!query.table_function && !skip_target_insert_access_check)
 context->checkAccess(AccessType::INSERT, query.table_id, query_sample_block.getNames());

and the storage’s own guard is checked too, because it may be a different kind of check (the source access of a URL database, say):

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1504
 /// Access the storage itself guards the write with (e.g. the source access of a table of a
 /// `URL` database). It is also checked when the sink is created, but that happens in a
 /// background flush for asynchronous inserts, so the check has to be repeated here.
 if (!query.table_function)
 table->checkInsertIsAllowed(context);

The routing itself:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1517
 BlockIO res;
 if (query.select && !query.async_insert_flush)
 {

query.select means “this insert has a SELECT source”. The !query.async_insert_flush guard is the other direction: when the async queue flushes, it builds a fresh InterpreterInsertQuery over the original query — which still has its SELECT — but at that point the data has already been materialised into a block, so the SELECT must be ignored.

If there is no SELECT, the plain insert pipeline is built:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1593
 else
 {
 res.pipeline = buildInsertPipeline(query, table);
 }

and the source is attached at the end:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1094
 if (query.hasInlinedData() && !async_insert)
 {
 auto format = getInputFormatFromASTInsertQuery(query_ptr, true, *query_sample_block, context, nullptr);

 if (settings[Setting::enable_parsing_to_custom_serialization])
 format->setSerializationHints(table->getSerializationHints());

 auto pipe = getSourceFromInputFormat(query_ptr, std::move(format), context, nullptr);
 pipeline.complete(std::move(pipe));
 }

Note that the format is asked for the sample block — so the parser’s column list and types come from the resolved schema, not from the client’s text. That is the whole point of the sample block: the client sends VALUES or CSV rows, and the server knows what they mean.

The lock

One more thing execute() does that is easy to miss: it takes a share lock on the table and holds it for the lifetime of the pipeline, not just for the build:

 cpp
 
 
// src/Interpreters/InterpreterInsertQuery.cpp:1600
 /// Keep the share lock until the pipeline finishes (not just while it is being built), so that
 /// the dependent-view discovery and the commit of the inserted data are indivisible with respect
 /// to an exclusive lock on the table. The atomic `CREATE MATERIALIZED VIEW ... POPULATE` relies
 /// on this: under its brief exclusive lock on the source, any concurrent `INSERT` has either
 /// already committed (and is covered by the pinned snapshot) or has not yet discovered the
 /// dependent views (and will see the newly registered view).
 /// Empty when the async insert queue transform took the lock over: it commits through a separate
 /// flush that takes its own lock, and the route is refused when any dependent view exists.
 if (table_lock)
 {
 QueryPlanResourceHolder insert_resources;
 insert_resources.table_locks.emplace_back(std::move(table_lock));
 res.pipeline.addResources(std::move(insert_resources));
 }

The invariant is: discovering the dependent views and committing the data must be atomic with respect to a CREATE MATERIALIZED VIEW. Otherwise a concurrent INSERT could commit before the view is registered, and the rows would never reach the view. The lock is stored in the pipeline’s resource holder so that it is released exactly when the pipeline is done — the resource holder is destroyed with the pipeline, not with the interpreter.

The whole picture

 text
 
 
 INSERT INTO t (b, a) VALUES (...)
 |
 v
 getSampleBlock(names)
 insertable columns -> by name -> [b, a] (client's order!)
 else materialized / virtual / error
 |
 v
 sample block = Block([b:String, a:UInt64])
 |
 v
 +---------- pipeline --------------------------------+
 | ShrinkColumns -> Counting -> AddDeduplicationInfo |
 | -> PlanSquashing |
 | | |
 | ResizeProcessor (1 -> N) |
 | / \ |
 | chain 0 chain 1 (one per view leaf)|
 | [ApplySquashing] [ApplySquashing] |
 | [AddDedupInfo ] |
 | | | |
 | MergeTreeSink MV target sink |
 +---------------------------------------------------+
 |
 v
 one part per squash

Trade-offs

What the sample block buys.

One place to resolve the schema. The parser, the conversion step, the pipeline header and the access check all read the same resolved block. A column that is UInt64 in the table is UInt64 everywhere.
By-name resolution is a hash lookup, not a scan, and it produces a specific error for each failure mode.
The client never has to know the table’s column order — but it can still use it, by omitting the column list.

What it costs.

A conversion step is always present for INSERT ... SELECT, even when the types already match. makeConvertingActions in Position mode will produce an empty DAG if everything lines up, and the ExpressionTransform becomes a no-op copy — but it is still a processor in the pipeline, with a chunk copy.
The sample block is a copy of the schema on every insert. For a table with 1000 columns, that is 1000 empty columns allocated per insert. This is why the metadata snapshot is taken once and reused.
The by-name lookup is only for the explicit column list. The positional case (INSERT INTO t VALUES) skips it entirely and just takes the insertable columns in order.

The bottom line

The insert path is a schema resolution followed by a fixed pipeline shape. The schema resolution produces a sample block; the pipeline shape is “count, stamp, squash, fan out, write”.

The subtle parts are all about not losing rows:

the fan-out is disabled whenever the deduplication numbering would collide between branches;
the fan-out is disabled for a non-parallel quorum insert, because sibling sinks would race against the quorum node;
the fan-out is disabled when a dependent view is hidden behind an Alias and parallel_view_processing = 0;
the share lock is held until the pipeline is done, so that a concurrent CREATE MATERIALIZED VIEW cannot slip between the discovery and the commit.

Every one of those is a case where the obvious implementation would silently drop data.

The thing people get wrong

INSERT INTO t (a, b) is not the same as INSERT INTO t (b, a) when the table’s column order differs, and it is not the same as INSERT INTO t VALUES at all:

 sql
 
 
CREATE TABLE t (a UInt64, b String) ENGINE = MergeTree ORDER BY a;
INSERT INTO t VALUES (1, 'x'); -- a = 1, b = 'x' (table order)
INSERT INTO t (b, a) VALUES ('x', 1);-- b = 'x', a = 1 (named, client's order)
INSERT INTO t (b) VALUES ('x'); -- a gets its DEFAULT

and the third case is the one people get wrong: the omitted column is filled by AddingDefaultsTransform, not by leaving a gap. The sample block only contains the columns the client named, so the pipeline’s header is [b:String], and the defaults step is what widens it to the table’s full insertable shape before the sink sees it.

How ClickHouse Batches Async Inserts in a Per-Key Queue

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: Every async insert is hashed into a UInt128 key that covers the query AST, the user’s identity, the format and every non-default setting. Inserts with the same key land in the same buffer, one buffer per key per shard. The buffer is flushed when it is big enough, holds enough queries, or its (adaptive) deadline expires — and the flush is done by a background thread while the client blocks on a std::future.

Writing to a MergeTree table is expensive per statement: a part has to be created, its metadata written, its checksums computed, and (for a replicated table) a ZooKeeper transaction has to allocate a block number. A client that sends 10 000 single-row INSERT statements pays that cost 10 000 times.

The obvious fix — “batch client-side, send one big INSERT” — puts the responsibility on the client, which every client gets wrong at least once, and which the INSERT ... SELECT path cannot use at all.

ClickHouse’s answer is to move the batching into the server and make it transparent: the client still sends one statement per batch of rows, the server buffers it, and the server decides when to turn N statements into one part.

The problem is that “one batch” is not a well-defined thing

Before you can buffer inserts, you have to decide which inserts may share a buffer. Two inserts into the same table with the same format obviously can. But what about two inserts into the same table with different SETTINGS? One with insert_deduplication_token = 'a', another with insert_deduplication_token = 'b'? Or one from alice and one from bob?

The answer has to be: nothing but inserts that are semantically indistinguishable may share a buffer. And that set is large — it is a function of the query text, the user, the roles, the credentials, the settings, the format, and even the physical representation of the data.

The second problem is that a buffer is not just a std::vector<Block>. It has to:

know when to flush,
be visible to a background thread,
carry a deadline that a background thread can watch,
and let the client learn, later, whether its rows actually landed — and how many.

The key: a hash of everything that matters

AsynchronousInsertQueue keys each buffer by an InsertQuery object, and that object computes a UInt128 hash of everything that must not be shared:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:162
 SipHash siphash;

 siphash.update(data_kind);
 query->updateTreeHash(siphash, /*ignore_aliases=*/ true);

 if (user_id)
 {
 siphash.update(*user_id);
 for (const auto & current_role : current_roles)
 {
 siphash.update(current_role);
 }
 }

The query is hashed structurally, through the AST (updateTreeHash), not textually. Two INSERT INTO t VALUES (1) statements with different whitespace produce the same key; two inserts with different column lists do not.

Then the credential limit and the credential expiry are folded in, because a deferred flush must not run with more rights than the session had, and must not run at all if the credential expired while the data sat in the buffer:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:176
 /// Fold the credential grant limit into the key so inserts made under different limits are never
 /// coalesced into one flush (the queue is keyed by `hash` alone). A null limit (the common,
 /// unrestricted case) contributes nothing, so it does not change the hash of existing keys.
 /// A non-null limit is hashed by its textual form even when it is semantically empty (a deny-all
 /// token), so it never collides with the null case; the leading length prefix keeps it unambiguous.
 if (authentication_grants)
 {
 /// Use the precise serialization (never `toString`): under `enable_read_write_grants = 0` the
 /// backward-compatibility widening collapses distinct source limits such as `READ ON FILE` and
 /// `WRITE ON FILE` into one, which would let a read-only token piggyback into a write token's bucket.
 const auto grants_str = authentication_grants->toStringPrecise();
 siphash.update(grants_str.size());
 siphash.update(grants_str);
 }

and the identity strings are length-prefixed, because a SipHash::update(String) streams raw bytes and the queue is keyed by the hash alone:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:198
 /// Length-prefix each field: update(String) streams only bytes and the queue is keyed
 /// by hash alone, so otherwise "a"/"a"/"aaa" and "aa"/"aa"/"a" would collide.
 for (const String & identity_field : {current_user, initial_user, authenticated_user, quota_key})
 {
 siphash.update(identity_field.size());
 siphash.update(identity_field);
 }

Finally, the settings. Every setting that differs from the default makes a different buffer, because a setting is semantics — max_insert_block_size changes the block shape, async_insert_deduplicate changes whether deduplication happens at all:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:206
 setting_changes = settings->changes();
 for (auto it = setting_changes.begin(); it != setting_changes.end();)
 {
 if (settings_to_skip.contains(it->name))
 {
 it = setting_changes.erase(it);
 }
 else
 {
 siphash.update(it->name);
 applyVisitor(FieldVisitorHash(siphash), it->value);
 ++it;
 }
 }

 hash = siphash.get128();

Two settings are explicitly skipped, and the comment says why:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:127
static const NameSet settings_to_skip
{
 /// We don't consider this setting because it is only for deduplication,
 /// which means we can put two inserts with different tokens in the same block safely.
 "insert_deduplication_token",
 "log_comment",
};

insert_deduplication_token is the interesting one. It looks like semantics — it names a deduplication token — but the token travels with the data, not with the buffer, so two inserts with different tokens can safely share a buffer. The token is carried per entry and re-attached at flush time (see below).

The queue: a multimap of deadlines plus a hash index

The data structure is a two-part affair. The primary container is ordered by time, because the background worker needs to find the earliest deadline:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.h:270
 struct Container
 {
 InsertQuery key;
 InsertDataPtr data;
 };

 /// Ordered container
 /// Key is a timestamp of the first insert into batch.
 /// Used to detect for how long the batch is active, so we can dump it by timer.
 /// Must be a multimap: two queries with different keys may theoretically compute the same deadline.
 using Queue = std::multimap<std::chrono::steady_clock::time_point, Container>;
 using QueueIterator = Queue::iterator;
 using QueueIteratorByKey = std::unordered_map<UInt128, QueueIterator>;

The comment about multimap is a real subtlety: two different keys can legitimately compute the same deadline (they both hashed to different buckets but the adaptive timeout converged them), and a std::map would silently drop one of the buffers.

The secondary container is a UInt128 → iterator index, because the producer needs to find an existing buffer for its key in O(1) and append to it. The queue is also sharded, so that two producers never contend on the same mutex:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.h:286
 struct QueueShard
 {
 mutable std::mutex mutex;
 mutable std::condition_variable are_tasks_available;
 /// Counts batches removed by producers or the deadline worker, including those
 /// still waiting for pool admission. Released when the batch is destroyed.
 std::atomic<size_t> in_flight_flushes{0};

 Queue queue TSA_GUARDED_BY(mutex);
 QueueIteratorByKey iterators TSA_GUARDED_BY(mutex);

 OptionalTimePoint last_insert_time TSA_GUARDED_BY(mutex);
 std::chrono::milliseconds busy_timeout_ms TSA_GUARDED_BY(mutex) {};
 };

The shard is picked by the key hash, so the same key always goes to the same shard and two keys with the same hash share a shard:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:736
 size_t shard_num = static_cast<size_t>(key.hash % pool_size);
 auto & shard = queue_shards[shard_num];
 const auto flush_time_points = flush_time_history_per_queue_shard[shard_num].getRecentTimePoints();
 {
 std::lock_guard lock(shard.mutex);

and this is where the buffer is found or created:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:760
 if (inserted)
 {
 try
 {
 it->second = shard.queue.emplace(now + timeout_ms, Container{key, std::make_unique<InsertData>(timeout_ms)});
 }
 catch (...)
 {
 shard.iterators.erase(it);
 throw;
 }
 }

it->second is the deadline of the buffer. Note that the deadline is computed at creation time and stored in the multimap’s key, so the deadline worker does not have to look at the buffer to know whether it is expired — it just looks at queue.begin().

The buffer: a list of entries, each with a promise

The buffer is a list of entries. Each entry is one client’s insert: the raw bytes (or a pre-parsed block), the query id, the dedup token, the format, and a std::promise that the flush will fulfil:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.h:191
 struct InsertData
 {
 struct Entry
 {
 public:
 DataChunk chunk;
 const String query_id;
 const String async_dedup_token;
 const String format;
 /// Keeps the queued bytes charged to the user that pushed them until the flush frees them.
 const std::unique_ptr<MemoryTracker> queued_data_tracker;
 const std::chrono::time_point<std::chrono::system_clock> create_time;
 NameToNameMap query_parameters;

 Entry(
 DataChunk && chunk_,
 String && query_id_,
 const String & async_dedup_token_,
 const String & format_,
 std::unique_ptr<MemoryTracker> queued_data_tracker_);

 void resetChunk();
 void finish(ResultProgress result = {});
 void finish(std::exception_ptr exception_);

 std::future<ResultProgress> getFuture() { return promise.get_future(); }
 bool isFinished() const { return finished; }

 private:
 std::promise<ResultProgress> promise;
 std::atomic_bool finished = false;
 };

The promise is the whole client-facing contract. The producer takes the future before releasing the shard lock, and returns it to the client:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:803
 data->size_in_bytes += entry_data_size;
 progress_future = entry->getFuture();

The finished flag is an atomic_bool with an exchange, which makes finish() idempotent — the flush path and the shutdown path can both try to finish the same entry:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:327
void AsynchronousInsertQueue::InsertData::Entry::finish(ResultProgress result)
{
 if (finished.exchange(true))
 return;

 resetChunk();
 promise.set_value(result);
}

There is a second, subtler reason for the atomic_bool: the buffer’s destructor also finishes the entries, so an entry that was already flushed must not have its promise set twice.

When to flush: three triggers

A buffer is flushed when any of three conditions holds.

Size. The buffer accumulated at least async_insert_max_data_size bytes (10 MiB by default):

 cpp
 
 
// src/Core/Settings.cpp:8526
 DECLARE(UInt64, async_insert_max_data_size, 10485760, R"(
Maximum size in bytes of unparsed data collected per query before being inserted

Cloud default value: `104857600` (100 MiB).
)", 0, \
 {"24.2", 1000000, 10485760, "The previous value appeared to be too small."}) \

Query count. The buffer holds at least async_insert_max_query_number (450) entries — but only when deduplication is on, because otherwise merging the rows into one block would change the deduplication semantics:

 cpp
 
 
// src/Core/Settings.cpp:8532
 DECLARE(UInt64, async_insert_max_query_number, 450, R"(
Maximum number of insert queries before being inserted.
Only takes effect if setting [`async_insert_deduplicate`](#async_insert_deduplicate) is 1.
)", 0) \

Deadline. The buffer’s deadline has passed.

The three are evaluated together, and if any fires, the buffer is removed from the queue and handed to the thread pool:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:811
 bool has_enough_bytes = data->size_in_bytes >= (*key.settings)[Setting::async_insert_max_data_size];
 bool has_enough_queries
 = data->entries.size() >= (*key.settings)[Setting::async_insert_max_query_number] && isDeduplicationEnabledForInsert(*key.settings);

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:828
 if (!flush_stopped && (has_enough_bytes || has_enough_queries || max_busy_timeout_exceeded()))
 {
 LOG_TRACE(log, "Scheduling async insert processing job because {}",
 has_enough_bytes ? "enough bytes accumulated" :
 has_enough_queries ? "enough queries accumulated" :
 "maximum busy wait timeout exceeded");
 data->timeout_ms = Milliseconds::zero();
 data->trackFlush(shard.in_flight_flushes);
 data_to_process = std::move(data);

 NOEXCEPT_SCOPE({
 shard.iterators.erase(it);
 shard.queue.erase(queue_it);
 });
 }

Note the data_to_process variable: the flush is scheduled after the shard lock is released, so a slow pool admission does not block other producers appending to other buffers in the same shard.

The adaptive deadline

The deadline is not a constant. It adapts to the insert rate, per shard:

 cpp
 
 
// src/Core/Settings.cpp:8540
 DECLARE(Bool, async_insert_use_adaptive_busy_timeout, true, R"(
If it is set to true, use adaptive busy timeout for asynchronous inserts
)", 0, \
 {"24.2", false, true, "Use adaptive asynchronous insert timeout"}) \
 DECLARE(Milliseconds, async_insert_busy_timeout_min_ms, 50, R"(
If auto-adjusting is enabled through async_insert_use_adaptive_busy_timeout, minimum time to wait before dumping collected data per query since the first data appeared. It also serves as the initial value for the adaptive algorithm
)", 0, \
 {"24.2", 50, 50, "The minimum value of the asynchronous insert timeout in milliseconds; it also serves as the initial value, which may be increased later by the adaptive algorithm"}) \
 DECLARE_WITH_ALIAS(Milliseconds, async_insert_busy_timeout_max_ms, 200, R"(
Maximum time to wait before dumping collected data per query since the first data appeared.

Cloud default value: `1000` (1s).
)", 0, async_insert_busy_timeout_ms, \
 {"24.2", 200, 200, "The minimum value of the asynchronous insert timeout in milliseconds; async_insert_busy_timeout_ms is aliased to async_insert_busy_timeout_max_ms"}) \

The default window is therefore 50 ms to 200 ms, with a 20 % growth and 20 % decay per step. The rule is simple: frequent inserts grow the window, idle inserts shrink it.

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:896
 const auto decreased_timeout_ms = std::min(
 std::chrono::duration_cast<Milliseconds>(shard.busy_timeout_ms / (1.0 + decrease_rate)), shard.busy_timeout_ms - Milliseconds(1));

 /// Increase the timeout for frequent inserts.
 if (last_insert_time + min_ms > now)
 {
 auto timeout_ms = std::max(
 std::chrono::duration_cast<Milliseconds>(shard.busy_timeout_ms * (1.0 + increase_rate)),
 shard.busy_timeout_ms + Milliseconds(1));

 return normalize(timeout_ms);
 }

The “frequent” test is last_insert_time + min_ms > now: an insert arrived less than min_ms (50 ms) after the previous one, so the producer is hot and the buffer is worth keeping open longer. The decay test additionally requires that the last two flushes were also far apart, so the timeout only shrinks when the shard is genuinely idle:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:908
 /// Decrease the timeout if inserts are not frequent,
 /// that is, if the time since the last insert and the difference between the last two queue flushes were both
 /// long enough (exceeding the adjusted timeout).
 /// This ensures the timeout value converges to the minimum over time for non-frequent inserts.
 if (last_insert_time + decreased_timeout_ms < now && t1 + decreased_timeout_ms < t2)
 return normalize(decreased_timeout_ms);

Note that the timeout is per shard, not per key. That is a deliberate simplification: with 16 shards and hundreds of keys, a per-key timeout would mean hundreds of independent timers, and the “thundering herd” problem would be solved by the shard’s own mutex anyway. The per-shard value is what all the keys in that shard share.

The deadline worker

One thread per shard watches the deadlines:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1111
void AsynchronousInsertQueue::processBatchDeadlines(size_t shard_num) TSA_NO_THREAD_SAFETY_ANALYSIS
{
 auto & shard = queue_shards[shard_num];

 while (!shutdown)
 {
 std::vector<Container> entries_to_flush;
 {
 std::unique_lock lock(shard.mutex);

 const auto rel_time
 = std::min(shard.busy_timeout_ms, saturatedMilliseconds(getContext()->getSettingsRef()[Setting::async_insert_poll_timeout_ms].totalMilliseconds()));
 shard.are_tasks_available.wait_for(
 lock,
 rel_time,
 [&shard, this] TSA_NO_THREAD_SAFETY_ANALYSIS
 {
 if (shutdown)
 return true;

 if (!shard.queue.empty() && shard.queue.begin()->first < std::chrono::steady_clock::now())
 return true;

 return false;
 });

The wait is a predicate wait, not a plain sleep: the worker wakes up early if the front of the queue has expired, and it is also woken explicitly by a producer (shard.are_tasks_available.notify_one()) when a new buffer is created. The poll interval is bounded by async_insert_poll_timeout_ms, which defaults to 10 ms:

 cpp
 
 
// src/Core/Settings.cpp:8536
 DECLARE(Milliseconds, async_insert_poll_timeout_ms, 10, R"(
Timeout for polling data from asynchronous insert queue
)", 0, \
 {"24.2", 10, 10, "Timeout in milliseconds for polling data from asynchronous insert queue"}) \

so the worst case extra delay is the poll interval, not the window. The window is the deadline; the poll interval is how often the worker checks it. So a buffer with a 200 ms deadline is flushed between 200 ms and 210 ms after it was created.

Then the worker pops all expired buffers (the queue is ordered by deadline, so it is a prefix) and schedules them:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1145
 while (true)
 {
 if (shard.queue.empty() || shard.queue.begin()->first > now)
 break;

 auto it = shard.queue.begin();

 NOEXCEPT_SCOPE({
 /// The only exception that is possible here is MEMORY_LIMIT_EXCEEDED, by blocking them it is highly unlikely that we will fail here.
 /// Besides it is not possible to write exception safe code here even with proper rollback, since the object can be moved out already (we can do copy, but this is more costly)
 MemoryTrackerBlockerInThread lock_memory_tracker;

 shard.iterators.erase(it->second.key.hash);

 it->second.data->trackFlush(shard.in_flight_flushes);
 entries_to_flush.emplace_back(std::move(it->second));

 shard.queue.erase(it);
 });
 }
 }

 for (auto & entry : entries_to_flush)
 scheduleDataProcessingJob(entry.key, std::move(entry.data), getContext(), shard_num);

The flush: a whole new query

The flush is not a “write the buffer to the table” function. It builds a complete new query — a new context, a new query id, a new process list entry, a new pipeline — and runs it. That is the only way to get all the downstream machinery (materialized views, quotas, query_log) to behave exactly as if the client had sent one big insert.

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1261
 /// Declared before `insert_context` so it's freed before the detach below reparents the tracker to
 /// `total_memory_tracker`; it accumulates access info charged to the inserting user while attached.
 DB::QueryScope query_scope;

 auto insert_context = Context::createCopy(global_context);
 insert_context->makeQueryContext();

The context is a copy of the global context, not of the client’s context — because the client’s context is gone by the time the flush runs. The identity has to be replayed by hand:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1275
 /// Access rights must be checked for the user who executed the initial INSERT query.
 if (key.user_id)
 {
 /// Replay the whole originating identity in one call: the external (pushed) roles, the
 /// credential grant limit and its expiry are restored together with the user, so a limited
 /// credential does not regain the full user's rights and a pushed-role session does not fail
 /// role revalidation.
 insert_context->setUser(*key.user_id, key.external_roles, key.authentication_grants, key.authentication_valid_until);
 /// `current_roles` are the session's *effective* current roles, which already include the
 /// external roles restored above. Re-apply them without the grant check: external roles are not
 /// locally granted, so a checked re-apply would throw `SET_NON_GRANTED_ROLE`; the locally
 /// granted current roles are kept and the external ones come from `setUser`.
 insert_context->setCurrentRoles(key.current_roles, /*check_grants=*/ false);
 }

and the settings have to be restored too, because the flush is a different query with its own settings:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1300
 insert_context->setSettings(*key.settings);

The credential expiry is re-checked at flush time, which is a genuinely nice property — the check the session would have done per query is done once, at the moment the data is actually written:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1237
 /// Fail closed if the authentication method that queued this insert has expired between enqueue
 /// and flush. The synchronous path re-checks per query in `Session::checkIfUserIsStillValid`; the
 /// deferred flush has no session, so without this a token could enqueue data just before expiry
 /// and have the server flush it afterwards. The per-method expiry is part of the batching key, so
 /// every entry in this batch shares it and they all fail closed together. 0 means no expiry.
 if (key.authentication_valid_until != 0)
 {
 const time_t now = std::chrono::system_clock::to_time_t(std::chrono::system_clock::now());
 if (now > key.authentication_valid_until)
 throw Exception(ErrorCodes::USER_EXPIRED, "Authentication method used to submit the deferred insert has expired");
 }

Then the interpreter is built and run, exactly as for a synchronous insert:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1443
 try
 {
 interpreter = std::make_unique<InterpreterInsertQuery>(
 key.query,
 insert_context,
 (*key.settings)[Setting::insert_allow_materialized_columns],
 false,
 false,
 true);

 pipeline = interpreter->execute().pipeline;
 chassert(pipeline.pushing());

The rows: parse everything, insert once

The flush then parses every entry into a single chunk and pushes that one chunk through the pipeline. This is where the per-entry dedup token is re-attached — the token travels with the data, not with the buffer:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1618
 StreamingFormatExecutor executor(
 header,
 format,
 std::move(on_error),
 data->size_in_bytes,
 data->entries.size(),
 std::move(adding_defaults_transform),
 [query_status = insert_context->getProcessListElement()] { return query_status && query_status->isKilled(); });

 auto deduplication_info = DeduplicationInfo::create(/*async_insert=*/true);

 for (const auto & entry : data->entries)
 {
 current_entry = entry;

 const auto * bytes = entry->chunk.asString();
 if (!bytes)
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Expected entry with data kind Parsed. Got: {}", entry->chunk.getDataKind());

 auto buffer = std::make_unique<ReadBufferFromString>(*bytes);
 executor.setQueryParameters(entry->query_parameters);

 size_t num_bytes = bytes->size();
 size_t num_rows = executor.execute(*buffer, num_bytes);

 total_rows += num_rows;

 /// it is ok if total_rows is 0 here or async_dedup_token is empty
 deduplication_info->setUserToken(entry->async_dedup_token, num_rows);

 add_to_async_insert_log(entry, current_exception, num_rows, num_bytes);
 current_exception.clear();
 entry->resetChunk();
 }

A StreamingFormatExecutor is used rather than a per-entry InputFormat: the executor keeps the parser state alive across entries, so 450 CSV payloads are parsed as one continuous stream. The on_error callback is what makes a single malformed entry fail only itself:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1605
 auto on_error = [&](const MutableColumns & result_columns, const ColumnCheckpoints & checkpoints, Exception & e)
 {
 current_exception = e.displayText();
 LOG_ERROR(logger, "Failed parsing for insert query id {}. {}",
 current_entry->query_id, current_exception);

 for (size_t i = 0; i < result_columns.size(); ++i)
 result_columns[i]->rollback(*checkpoints[i]);

 current_entry->finish(std::current_exception());
 return 0;
 };

rollback(*checkpoints[i]) unwinds the partially-parsed columns so the bad row does not poison the good rows that follow it in the same buffer. The bad entry’s promise is fulfilled with the exception, so its client gets an error, while the other 449 clients get their rows.

Note the two data kinds. If the client sent a pre-parsed block (the native protocol path), the data is already a Block and the entries are simply concatenated after a header conversion:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:1690
 if (!isCompatibleHeader(block_to_insert, header))
 convertBlockToHeader(block_to_insert, header, context_);

The client side: two different contracts

There are two entry points into the queue, and they answer the client differently.

Inline data (INSERT INTO t VALUES (...) over HTTP, or the data part of a native-protocol query) goes through executeQuery:

 cpp
 
 
// src/Interpreters/executeQuery.cpp:3123
 auto result = queue->pushQueryWithInlinedData(out_ast, context);

 if (result.status == AsynchronousInsertQueue::PushResult::OK)
 {
 // Increment InsertQuery for async insert with inline data
 ProfileEvents::increment(ProfileEvents::InsertQuery);

 if (settings[Setting::wait_for_async_insert])
 {
 auto timeout = saturatedMilliseconds(settings[Setting::wait_for_async_insert_timeout].totalMilliseconds()).count();
 auto source = std::make_shared<WaitForAsyncInsertSource>(
 std::move(result.future),
 timeout,
 context->getProcessListElement(),
 context->getProgressCallback(),
 /* report_read_progress */ true);
 res.pipeline = QueryPipeline(Pipe(std::move(source)));
 res.pipeline.complete(std::make_shared<NullOutputFormat>(std::make_shared<const Block>(Block())));
 }

Block data (the native protocol, where the client sends Blocks) goes through TCPHandler:

 cpp
 
 
// src/Server/TCPHandler.cpp:1595
 if (insert_queue && async_insert_enabled && !insert_query.select)
 {
 auto result = processAsyncInsertQuery(state, *insert_queue);
 if (result.status == AsynchronousInsertQueue::PushResult::OK)
 {
 /// Reset pipeline because it may hold write lock for some storages.
 state.io.resetPipeline(/*cancel=*/true);
 if (settings[Setting::wait_for_async_insert])
 {
 auto timeout = saturatedMilliseconds(settings[Setting::wait_for_async_insert_timeout].totalMilliseconds());
 size_t timeout_ms = timeout.count();
 auto wait_status = result.future.wait_for(timeout);

 if (wait_status == std::future_status::deferred)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Got future in deferred state");

 if (wait_status == std::future_status::timeout)
 throw Exception(ErrorCodes::TIMEOUT_EXCEEDED, "Wait for async insert timeout ({} ms) exceeded", timeout_ms);

 auto progress_result = result.future.get();

The two paths differ in how they wait (a source processor that participates in the query pipeline vs. a blocking future::wait_for), but the semantics are the same.

wait_for_async_insert defaults to true, so the default behaviour is “the client is told the truth”:

 cpp
 
 
// src/Core/Settings.cpp:8520
 DECLARE(Bool, wait_for_async_insert, true, R"(
If true wait for processing of asynchronous insertion.
)", 0) \

With wait_for_async_insert = 0, the query returns as soon as the data is in the buffer, and the guarantee is only “the server has taken responsibility for these bytes”. The query_log documentation spells out the difference:

 cpp
 
 
// src/Interpreters/QueryLog.cpp:80
 {"written_rows", std::make_shared<DataTypeUInt64>(), "The number of rows written by the query, including any rows written by downstream inserts triggered by the pipeline, such as attached materialized views. For a synchronous insert these downstream rows are recorded on the `query_kind` = `Insert` entry; for an asynchronous insert they are recorded on the `query_kind` = `AsyncInsertFlush` entry, while the client-facing `Insert` entry records only the rows accepted from the client, and only when `wait_for_async_insert = 1` makes it wait for that flush result; with `wait_for_async_insert = 0` the client-facing `Insert` entry returns before the flush and this field stays 0. For queries that do not write rows, it is 0."},

So there are two query_log rows for one async insert: the client-facing Insert row and the server-side AsyncInsertFlush row.

The escape hatch: too much data

If a single insert is larger than async_insert_max_data_size, buffering it would be silly — the server would hold 100 MiB of client data in RAM and then write it. So the producer reads at most the limit, and if there is more left over, it hands the rest back to the caller, which falls back to a synchronous insert:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:638
 if (!read_buf->eof())
 {
 /// Concat read buffer with already extracted from insert
 /// query data and with the rest data from insert query.
 ConcatReadBuffer::Buffers buffers;
 buffers.emplace_back(std::make_unique<ReadBufferFromOwnString>(bytes));
 buffers.emplace_back(std::move(read_buf));

 return PushResult
 {
 .status = PushResult::TOO_MUCH_DATA,
 .future = {},
 .insert_data_buffer = std::make_unique<ConcatReadBuffer>(std::move(buffers)),
 };
 }

The PushResult::TOO_MUCH_DATA status is handled identically in both entry points, and the “rest” of the data is not lost: the caller re-attaches it to the query and runs the whole thing synchronously.

Things that are refused

Some combinations simply do not make sense, and the queue rejects them at the door rather than at flush time.

Quorum inserts. A non-parallel quorum insert allows exactly one in-flight quorum part per table, which a flush (which writes the data of several queries at once) cannot honour:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:686
 auto quorum_is_enabled = settings[Setting::insert_quorum].valueOr(0) > 1 || settings[Setting::insert_quorum].is_auto;
 if (quorum_is_enabled && !settings[Setting::insert_quorum_parallel])
 throw Exception(
 ErrorCodes::UNSUPPORTED_PARAMETER,
 "Async inserts with quorum only make sense with enabled insert_quorum_parallel setting, either disable quorum "
 "or set insert_quorum_parallel=1 or do not use async inserts");

Transactions. Async inserts inside a transaction are rejected, because the flush happens after the transaction would have committed (or rolled back).

INSERT ... SELECT with views. The INSERT ... SELECT route to the queue is only taken when the destination has no dependent materialized views, because a diverted block leaves the views’ sink chain built but unfed.

Trade-offs

What it buys. One part per 450 inserts instead of one part per insert. The part is the expensive object: metadata, checksums, ZooKeeper transaction, merges. Amortising it over hundreds of statements is a 10-100× reduction in per-statement overhead. The number of round trips does not change — the client still sends one statement per batch — but the number of parts the server has to create, merge and later compact does.

What it costs.

Latency. The data is invisible to SELECT until the flush. With the default 50-200 ms adaptive window, a freshly inserted row may not be queryable for up to ~200 ms. This is the classic “read-your-writes” problem, and it is why wait_for_async_insert defaults to true — the client waits for the flush, so the client knows the data is visible, but other clients still have to wait for the window.

Memory. The buffer holds up to async_insert_max_data_size bytes per key per shard. With many distinct keys (many clients, many settings) this multiplies. The MemoryTracker per entry keeps the charge on the user that pushed the data, not on the server, so a runaway client is stopped by its own quota:

 cpp
 
 
// src/Interpreters/AsynchronousInsertQueue.cpp:310
void AsynchronousInsertQueue::InsertData::Entry::resetChunk()
{
 /// Released against the user that pushed the data, never the flush, whose user is whoever's insert it
 /// happens to be flushing. The parameters were charged the same way and go with it.
 if (queued_data_tracker)
 {
 MemoryTrackerSwitcher switcher(queued_data_tracker.get());
 chunk = {};
 query_parameters.clear();
 return;
 }

Error attribution. A batch of 450 inserts is one part. If the part cannot be written, all 450 clients get the error — which is correct, but it means a single bad row in one entry can fail the whole batch if the parse error is not isolated. The on_error rollback above is what prevents that.
The key is a hash. Two inserts whose keys differ only in a way the hash cannot see would be merged. The hash covers the AST structurally, the user, the roles, the credential limit and expiry, the identity strings, and every non-default setting — that is a lot of coverage, but it is coverage by construction, not by proof.

The bottom line

The async insert queue is a content-addressed batcher. The key is a UInt128 SipHash over everything that could make two inserts behave differently; the value is a list of entries, each carrying its own promise. Three independent triggers — bytes, entry count, deadline — decide when a buffer is handed to a thread pool, and the flush replays the whole query machinery (context, identity, settings, process list, query_log) so that a deferred insert is indistinguishable from a synchronous one, except for its query id.

The most interesting design decision is not the batching but the replay: the flush does not write blocks to a table, it runs a query. That is what makes materialized views, quotas, row policies and query_log keep working on the deferred path — and it is also why the queue has to carry so much identity in its key.

The thing people get wrong

async_insert = 1 does not mean “faster”. It means “the server will batch this for you, and the data will not be visible for a short while”. If your workload needs read-your-writes, the answer is wait_for_async_insert = 1 (turn the waiting on) or async_insert = 0 (turn the batching off) — not wait_for_async_insert = 0, which just means “I do not want to know whether it worked”.

And if you are seeing rows appear in batches of 450 rather than batches of 10 000, check async_insert_max_data_size (10 MiB) and async_insert_busy_timeout_max_ms (200 ms): one of the two is your throughput limit, and the other is your latency limit.

How ClickHouse Deduplicates Retried Inserts with a Content Hash

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: Every inserted block is hashed column-by-column into a UInt128. The hash, plus the partition id, forms a “deduplication token” that is recorded in a bounded window — in ZooKeeper for replicated tables, in a local log file otherwise. An insert whose token is already in the window is recognised as a retry and silently skipped.

The problem: networks retry

INSERT INTO t VALUES (...) is a request over a network. Networks drop packets, proxies time out, and load balancers retry. A client that gets no response does not know whether the server applied the insert or not, so it sends it again.

For an INSERT that appends rows, that means the data is written twice.

ClickHouse’s answer is to make the insert idempotent: if the exact same bytes have already been inserted recently, do not insert them again. The question is how to recognise “the exact same bytes” cheaply, and how to bound the memory of the recognition.

What gets hashed

The hash is over the data, not over a client-supplied identifier. The DeduplicationInfo chunk-info object carries a list of tokens; a token is either a user-supplied string or a computed content hash.

The content hash is computed column-major over the token’s row range:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:402
void DeduplicationInfo::calculateDataHashes() const
{
 std::vector<size_t> pending;
 for (size_t offset = 0; offset < tokens.size(); ++offset)
 if (tokens[offset].by_user.empty() && !tokens[offset].data_hash_batch.has_value())
 pending.push_back(offset);

 if (pending.empty())
 return;

 /// Counted per token, not per pass: pre-warming keeps this at O(tokens); the per-partition
 /// cold-clone path drives it to O(partitions*tokens) for tokens that span several partitions.
 ProfileEvents::increment(ProfileEvents::DuplicationDataHashComputations, pending.size());

 chassert(original_block && original_block->rows() == getRows());

 /// The hash must not depend on the column's representation, so strip every representation wrapper
 /// (`ColumnConst`, `ColumnReplicated`, `ColumnSparse`) before hashing. A wrapper hashes through the
 /// generic per-row loop, whose byte stream differs from the dense column's range overload for every
 /// variable-size type - the same rows would then produce two different block hashes and a retried
 /// insert would not deduplicate.
 /// `convertToFullIfWrapped` strips the whole column tree rather than just the root, because a
 /// composite column hashes by delegating to its children (`ColumnTuple::updateHashWithValueRange`
 /// calls each element's), so a wrapper anywhere in the tree would reach the generic loop. Every
 /// wrapper observed here so far sits at the root, and it costs nothing to be exhaustive: with no
 /// wrapper anywhere the call returns the column itself.
 /// `LowCardinality` is deliberately not removed: it is a semantic type rather than a representation,
 /// so it cannot differ between two attempts at the same insert.
 /// Column-major so only one column is materialized at a time, instead of a dense copy of the whole block.
 std::vector<SipHash> hashes(pending.size());
 for (const auto & col : original_block->getColumns())
 {
 auto dense = col->convertToFullIfWrapped();
 for (size_t i = 0; i < pending.size(); ++i)
 dense->updateHashWithValueRange(getTokenBegin(pending[i]), getTokenEnd(pending[i]), hashes[i]);
 }

Three details in that comment are worth their own paragraphs.

The hash must not depend on the column’s representation. A ColumnConst (a column where every row is the same value) hashes through a generic per-row loop, while a dense column hashes through a range overload. The two produce different byte streams for the same logical values. If a retry arrives with the values in a different physical representation — say, a constant-folded column versus a materialized one — the hashes would differ and the retry would not be deduplicated. convertToFullIfWrapped() normalizes the representation away.

LowCardinality is deliberately not stripped, because it is a semantic type, not a representation: the same logical values always have the same LowCardinality encoding.

The hash is column-major. Only one column is materialized at a time, rather than a dense copy of the whole block. For a block of 65536 rows and 100 columns, the row-major alternative would need a 6.5-million-element buffer; the column-major version needs one column’s worth.

The hash is per token, not per block. A block can carry several tokens if it was assembled from several sources — see below. The pending list collects the tokens that do not yet have a hash, and the whole column is walked once for all of them.

The result is a UInt128 per token — 128 bits of SipHash output. The choice of 128 bits matters: the token is a dictionary key in ZooKeeper, and a 64-bit hash would make accidental collisions a real concern at scale.

The unified token

The content hash is only half the story. The final token is a SipHash over a string that embeds the content hash plus a set of extra tokens:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:456
DeduplicationHash DeduplicationInfo::getBlockUnifiedHash(size_t offset, const std::string & partition_id) const
{
 // when there is no user token, compute the full column-wise hash of the data
 // this hash is used for deduplication of sync and async inserts in a unified manner

 auto & token = tokens[offset];

 std::string extension;
 if (!token.by_user.empty())
 {
 extension = "user-token-" + token.by_user;
 }
 else
 {
 auto data_hash = getDataHash(offset);
 extension = fmt::format("{}_{}", data_hash.items[0], data_hash.items[1]);
 }

 // for other token sources addition parts are appended
 for (const auto & extra : token.extra_tokens)
 {
 if (is_async_insert && extra.type == TokenDefinition::Extra::SOURCE_NUMBER)
 {
 // do not include source number for async inserts, they are not relevant as data hash is used or user token
 // a token describes only the data in one block
 extension.append(":");
 extension.append(TokenDefinition::Extra::asSourceNumber(0).toString());
 continue;
 }

 extension.append(":");
 extension.append(extra.toString());
 }

 SipHash hash;
 hash.update(extension.data(), extension.size());
 return DeduplicationHash::createUnifiedHash(hash.get128(), partition_id);
}

The “extra tokens” are the provenance of the block: which source it came from, and which materialized view it passed through. They are what keeps two different blocks with identical content from colliding.

The Extra type enumerates them:

 cpp
 
 
// src/Interpreters/InsertDeduplication.h:208
 struct Extra
 {
 enum Type
 {
 SOURCE_ID,
 SOURCE_NUMBER,
 VIEW_ID,
 VIEW_NUMBER,
 };

 Type type;

 using Range = std::pair<size_t, size_t>;
 std::variant<Range, StorageIDMaybeEmpty> value_variant;

SOURCE_ID is the root table the insert started at, SOURCE_NUMBER is the block’s ordinal within it, VIEW_ID/VIEW_NUMBER identify a materialized view in the chain. When an insert feeds a materialized view, the view’s data is a projection of the source data, and two different source blocks can legitimately produce identical view rows. Without the view id in the token, the second one would be wrongly deduplicated.

The SOURCE_NUMBER special case for async inserts is the flip side: an async insert’s blocks are batched and reordered, so the block ordinal is not stable across retries, and including it would break deduplication entirely. The comment says so.

The final token string, and the block id derived from it:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:69
std::string DeduplicationHash::getBlockId() const
{
 return fmt::format("{}_{}_{}", partition_id, hash.items[0], hash.items[1]);
}

So the block id is partition_<hash_high>_<hash_low>. The partition id is part of the key, which is why the same rows inserted into two different partitions are not deduplicated against each other.

And the ZooKeeper path, which reveals the three generations of this feature that are still on disk:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:75
std::string DeduplicationHash::getPath(const std::string & storage_path) const
{
 std::string hashes_directory = [&] ()
 {
 switch (hash_type)
 {
 case HashType::SYNC:
 return "blocks";
 case HashType::ASYNC:
 return "async_blocks";
 case HashType::UNIFIED:
 return "deduplication_hashes";
 }
 }();
 return (std::filesystem::path(storage_path) / hashes_directory / getBlockId()).string();
}

blocks/ is the old sync-insert directory, async_blocks/ the old async one, and deduplication_hashes/ the current unified one. The cleanup thread still drains the legacy directory during a rolling upgrade:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:76
 // Old replicas (pre-unified-hash) still write the legacy async-insert ids to /async_blocks
 // during a rolling upgrade; the current leader keeps draining that directory with the legacy
 // window settings until the minimum supported version no longer uses it.

The ZooKeeper check

The dedup check for a replicated table is a check-not-exists request folded into the same ZooKeeper multi-op that allocates the block number. It is not a separate round trip:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:71
 /// Check for duplicates in advance, to avoid superfluous block numbers allocation
 Coordination::Requests ops;

 for (const auto & single_dedup_path : deduplication_paths)
 zkutil::addCheckNotExistsRequest(ops, *zookeeper_, single_dedup_path);

 auto deduplication_path_ops_size = ops.size();

 ops.emplace_back(zkutil::makeCreateRequest(path_prefix_, holder_path, zkutil::CreateMode::EphemeralSequential));
 Coordination::Responses responses;
 Coordination::Error e = zookeeper_->tryMulti(ops, responses);
 if (e == Coordination::Error::ZNODEEXISTS)
 {
 auto failed_idx = zkutil::getFailedOpIndex(Coordination::Error::ZNODEEXISTS, responses);

 if (failed_idx < deduplication_path_ops_size)
 {
 const String & failed_op_path = ops[failed_idx]->getPath();
 LOG_DEBUG(
 getLogger("createEphemeralLockInZooKeeper"),
 "Deduplication path already exists: deduplication_path={}",
 failed_op_path);
 return EphemeralLockInZooKeeper{"", nullptr, "", 0, failed_op_path};
 }
 }

The conflict_path in the returned lock is how the sink learns which hash collided — the failed op’s path carries the block id.

This is a genuinely elegant bit of engineering. The dedup check and the block-number allocation are one ZooKeeper transaction, so:

There is no TOCTOU window between “check for duplicate” and “allocate”.
A duplicate is detected before a block number is consumed.
The whole thing costs one round trip, not two.

The comment on the sink side confirms the cost:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:934
 auto block_number_lock = storage.allocateBlockNumber(part->info.getPartitionId(), zookeeper, block_id_pathes, "", block_data); /// 1 RTT

The window

The dedup history is bounded. For replicated tables, the bound is two settings — a count and a time:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1545
 DECLARE(UInt64, replicated_deduplication_window, 10000, R"(
The number of most recently inserted blocks for which ClickHouse Keeper stores
hash sums to check for duplicates.

and

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1566
 DECLARE(UInt64, replicated_deduplication_window_seconds, 60 * 60 /* one hour */, R"(
The number of seconds after which the hash sums of the inserted blocks are
removed from ClickHouse Keeper.

The cleanup thread enforces both bounds — the count and the age:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:383
 /// Use ZooKeeper's first node (last according to time) timestamp as "current" time.
 Int64 current_time = timed_blocks.front().ctime;
 Int64 time_threshold = std::max(
 static_cast<Int64>(0),
 current_time - static_cast<Int64>(1000 * window_seconds));

 /// Virtual node, all nodes that are "greater" than this one will be deleted
 NodeWithStat block_threshold{{}, time_threshold, 0, 0};

 size_t current_deduplication_window = std::min<size_t>(timed_blocks.size(), window_size);
 auto first_outdated_block_fixed_threshold = timed_blocks.begin() + current_deduplication_window;
 auto first_outdated_block_time_threshold = std::upper_bound(
 timed_blocks.begin(), timed_blocks.end(), block_threshold, NodeWithStat::greaterByTime);
 auto first_outdated_block = std::min(first_outdated_block_fixed_threshold, first_outdated_block_time_threshold);

A node is deleted if it is either beyond the count window or older than the time window. The time is measured against the newest node’s ctime, not the wall clock:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1579
The time is relative to the time of the most recent record, not to the wall
time. If it's the only record it will be stored forever.

That is deliberate. If the server’s clock jumps, or if the table is idle, a wall-clock-based expiry would either delete everything or keep everything. Measuring from the newest record makes the window self-relative.

The deletions are issued asynchronously and in parallel:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:407
 zkutil::AsyncResponses<Coordination::RemoveResponse> try_remove_futures;
 for (auto it = first_outdated_block; it != timed_blocks.end(); ++it)
 {
 String path = zookeeper_path + "/" + blocks_dir_name + "/" + it->node;
 try_remove_futures.emplace_back(path, zookeeper.asyncTryRemove(path, it->version));
 }

The non-replicated case: a local log

A non-replicated MergeTree has no ZooKeeper, so the dedup history lives in a file on the table’s own disk. The setting is off by default:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:763
 DECLARE(UInt64, non_replicated_deduplication_window, 0, R"(
The number of the most recently inserted blocks in the non-replicated
[MergeTree](/reference/engines/table-engines/mergetree-family/mergetree) table
for which hash sums are stored to check for duplicates.

and the mechanism is described in the setting’s own text:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:772
A deduplication mechanism is used, similar to replicated tables (see
[replicated_deduplication_window](#replicated_deduplication_window) setting): the deduplication
hash sum covers the whole inserted block. The hash sums are written to
a local file on a disk rather than to ClickHouse Keeper.

The log’s on-disk format is deliberately human-readable — tab-separated records of operation, part name and block id:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:126
/// Fixed-size log for deduplication in non-replicated MergeTree.
/// Stores records on disk for zero-level parts in human-readable format:
/// operation part_name partition_id_check_sum
/// 1 88_18_18_0 88_10619499460461868496_9553701830997749308
/// 2 77_14_14_0 77_15147918179036854170_6725063583757244937
/// 2 77_15_15_0 77_14977227047908934259_8047656067364802772
/// 1 77_20_20_0 77_15147918179036854170_6725063583757244937
/// Also stores them in memory in hash table with limited size.

The in-memory index is a hash map with a FIFO eviction policy, implemented as a list plus an index into it:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:29
template <typename V>
class LimitedOrderedHashMap
{
private:
 struct ListNode
 {
 std::string key;
 V value;
 };
 using Queue = std::list<ListNode>;
 using IndexMap = std::unordered_map<std::string_view, typename Queue::iterator, StringViewHash>;

 Queue queue;
 IndexMap map;
 size_t max_size;

The std::list gives O(1) FIFO eviction from the front; the unordered_map<string_view, iterator> gives O(1) lookup. The map stores string_views into the list nodes’ keys, so no key is copied twice.

The eviction is what makes the window a window:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:91
 bool insert(const std::string & key, const V & value)
 {
 auto it = map.find(key);
 if (it != map.end())
 return false;

 if (size() == max_size)
 {
 map.erase(queue.front().key);
 queue.pop_front();
 }

 ListNode elem{key, value};
 auto itr = queue.insert(queue.end(), elem);
 map.emplace(itr->key, itr);
 return true;
 }

When the map is full, the oldest entry is dropped. A retried insert is only deduplicated if the original is still in the window — which is exactly the semantics you want: a retry happens within seconds of the original, not hours later.

The log files themselves rotate, with a rotation interval of twice the window:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:94
 , rotate_interval(deduplication_window_ * 2) /// actually it doesn't matter

and the rotation is driven by the number of entries in the current file:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:255
void MergeTreeDeduplicationLog::rotateAndDropIfNeeded()
{
 /// If we don't have logs at all or already have enough records in current
 /// For the disk that doesn't support writing with append, we can't append logs to the last file.
 if (existing_logs.empty() || existing_logs[current_log_number].entries_count >= rotate_interval || !disk_supports_writing_with_append)
 {
 rotate();
 dropOutdatedLogs();
 }
}

Old logs are dropped by walking from the newest backwards and accumulating entry counts until the window is covered:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:211
void MergeTreeDeduplicationLog::dropOutdatedLogs()
{
 size_t current_sum = 0;
 size_t remove_from_value = 0;
 /// Go from end to the beginning
 for (auto itr = existing_logs.rbegin(); itr != existing_logs.rend(); ++itr)
 {
 /// Never drop the current active log — it may still be open for writing
 if (itr->first == current_log_number)
 {
 current_sum += itr->second.entries_count;
 continue;
 }

 if (current_sum >= deduplication_window)
 {
 /// We have more logs than required, all older files (excluding current) can be dropped
 remove_from_value = itr->first;
 break;
 }

The comment “actually it doesn’t matter” on the rotation interval is a nice piece of honesty — the interval only affects how many files exist on disk, not the semantics.

The retry loop

When a conflict is detected, the sink does not fail. It filters out the duplicate rows and retries with what is left:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:353
 ProfileEvents::increment(ProfileEvents::DuplicatedInsertedBlocks, conflicts.size());

 auto result = partition.deduplication_info->deduplicateBlock(
 conflicts,
 partition.block_with_partition.partition_id,
 context);

deduplicateBlock finds the offsets of the colliding tokens and filters them out of the block. If everything collided, the part is simply not committed:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:382
 if (result.filtered_block->rows() == 0)
 {
 LOG_DEBUG(
 storage.log,
 "All rows are deduplicated for part with block IDs: {}, skipping the part commit.",
 fmt::join(conflicts, ", "));

and the part log records the deduplication as an execution status, so system.part_log shows what happened:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:393
 PartLog::addNewPart(
 storage.getContext(),
 PartLog::PartLogEntry(partition.temp_part->part, partition.elapsed_ns, counters_snapshot),
 getDeduplicationBlockIds(deduplication_hashes),
 ExecutionStatus(ErrorCodes::INSERT_WAS_DEDUPLICATED, "The part was deduplicated"));

The filter itself is careful about a boundary case. If a partial collision happens, the colliding rows have to be sliced out of the block — but the block may have been re-anchored by an intermediate view, in which case the offsets no longer describe it:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:303
 chassert(original_block && !original_block->empty() && original_block->rows() > 0);

 auto & block = *original_block;

 /// A partial collision requires slicing the collided tokens' rows out of the block, which is
 /// only possible while the offsets still describe the block. Behind an `Alias` hop over a
 /// row-count-changing view the block is re-anchored to the view-output chunks and there is no
 /// mapping from the tokens' source rows to the block's rows anymore. Refuse loudly instead of
 /// reading out of the block's bounds; full-block deduplication is handled above.
 if (block.rows() != getRows())
 throw Exception(
 ErrorCodes::NOT_IMPLEMENTED,
 "Cannot filter {} of {} deduplicated blocks out of the insert: the deduplication info describes {} rows, "
 "but the block has {} rows because a materialized view with a row-count-changing inner query "
 "was processed before a table with the `Alias` engine. Only the whole insert can be deduplicated on this path. "
 "Debug: {}",

Failing loudly is the right call. Silently reading out of bounds would be a memory-safety bug; silently skipping the dedup would insert duplicates.

Self-deduplication: the same block twice

There is a second, subtler case: the same insert containing the same data twice. A client that retries a batch of inserts with insert_deduplication_token unset, or a materialized view that produces identical rows, can end up with duplicate content inside a single block.

deduplicateSelf handles that, and it operates on the token level:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:217
std::set<size_t> DeduplicationInfo::filterSelf(const String & partition_id) const
{
 if (getCount() <= 1)
 return {};

 std::set<size_t> fitered_offsets;
 /// fitered_offsets will contain all but first offsets for each block id
 /// so that only first occurrence of each block id will remain
 for (const auto & [_, block_offsets] : buildOffsetsMapImpl(partition_id))
 {
 if (block_offsets.size() > 1)
 fitered_offsets.insert(block_offsets.begin() + 1, block_offsets.end());
 }

 return fitered_offsets;
}

Group the tokens by their hash; within each group, keep the first and drop the rest. The grouping is done on the sorted hash values:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:496
std::vector<std::pair<UInt128, std::vector<size_t>>> DeduplicationInfo::buildOffsetsMapImpl(const std::string & partition_id) const
{
 /// (hash, offset) pairs sorted by hash, then offset; runs of equal hashes are the groups.
 std::vector<std::pair<UInt128, size_t>> sorted;
 sorted.reserve(offsets.size());
 for (size_t offset = 0; offset < offsets.size(); ++offset)
 sorted.emplace_back(getBlockUnifiedHash(offset, partition_id).hash, offset);
 std::sort(sorted.begin(), sorted.end());

Note that this recomputes the unified hash for every token — including the extra tokens, so the grouping is by the final token, not the raw content hash. Two tokens with the same content but different provenance are not duplicates.

The user token

A pipeline that writes its own idempotency key can bypass content hashing entirely, by supplying the key as a session setting:

 sql
 
 
INSERT INTO events SETTINGS insert_deduplication_token = 'batch-2026-10-22-0001'
VALUES (1, 'click'), (2, 'view');

The same token on a later insert means “this is a duplicate of the earlier one”; a different token means “this is new data, even if the rows are identical”.

The user can bypass content hashing entirely with the insert_deduplication_token setting:

 cpp
 
 
// src/Core/Settings.cpp:9135
 DECLARE(String, insert_deduplication_token, "", R"(
The setting allows a user to provide own deduplication semantic in MergeTree/ReplicatedMergeTree
For example, by providing a unique value for the setting in each INSERT statement,
user can avoid the same inserted data being deduplicated.

When the token is set, it replaces the content hash in the extension string ("user-token-" + token instead of the data hash). The semantics are therefore inverted: with a token, the same token means duplicate; without one, the same data means duplicate.

The setting’s documentation spells out the partition subtlety:

 cpp
 
 
// src/Core/Settings.cpp:9150
<Note>
`insert_deduplication_token` is tracked per partition, so multiple partitions written by one insert can carry the same token. Without a token, the default content checksum is computed over the whole inserted block, so an insert is deduplicated only when its entire data matches a previous insert (a retry), not when a single partition's rows happen to coincide with a different insert.
</Note>

When is deduplication on?

The whole thing is controlled by deduplicate_insert, with a backwards-compatible split between sync and async:

 cpp
 
 
// src/Core/DeduplicateInsert.cpp:26
bool isDeduplicationEnabledForInsert(bool is_async_insert, const Settings & settings)
{
 switch (settings[Setting::deduplicate_insert].value)
 {
 case DeduplicateInsertMode::BACKWARD_COMPATIBLE_CHOICE:
 return is_async_insert ? settings[Setting::async_insert_deduplicate] : settings[Setting::insert_deduplicate];
 case DeduplicateInsertMode::ENABLE:
 return true;
 case DeduplicateInsertMode::DISABLE:
 return false;
 }
}

For INSERT ... SELECT, there is an extra wrinkle: the SELECT might not be stable, meaning two executions return different data, and content-hash deduplication would then be meaningless. The mode ENABLE_WHEN_POSSIBLE handles it:

 cpp
 
 
// src/Core/DeduplicateInsert.cpp:73
 case DeduplicateInsertSelectMode::ENABLE_WHEN_POSSIBLE:
 {
 if (logger && !select_query_sorted && !has_user_token && isDeduplicationEnabledForInsert(false, settings))
 LOG_INFO(logger, "INSERT SELECT deduplication is disabled because SELECT is not stable");

 return (select_query_sorted || has_user_token) && isDeduplicationEnabledForInsert(false, settings);
 }

A stable SELECT is one with ORDER BY ALL and a single-stream pipeline — or, as the setting’s documentation puts it, a non-empty insert_deduplication_token is “an equivalent substitute for stability”.

Trade-offs

Space. The dedup window costs one ZooKeeper znode per inserted block (or one line in a local log). 10000 blocks is 10000 znodes, and the cleanup thread deletes them as they age out.

CPU. The content hash is a SipHash pass over every column of every inserted block. For a wide table that is not free — but it is one pass over data that is about to be written anyway, and it is only computed when deduplication is enabled and no user token was supplied.

Correctness of the window. Deduplication is only guaranteed within the window. An insert retried after the original has aged out is inserted twice. This is why the window is expressed in both a count and a time: the count covers a burst, the time covers a slow trickle.

The content hash is a heuristic. Two different blocks with the same content hash collide, and the second is dropped. With 128 bits of SipHash that is not a practical concern, but it is why the implementation bothers with UInt128 rather than UInt64.

The aliasing problem is a real limitation. Behind an Alias hop over a row-count-changing view, a partial collision cannot be resolved and the insert fails with NOT_IMPLEMENTED rather than corrupting anything. That is a deliberate trade: correctness over convenience.

The bottom line

Block-level insert deduplication is a content-addressed cache of recent inserts, with the cache keys being UInt128 SipHashes over a column-major hash of the block’s data, salted with the partition id and the block’s provenance so that unrelated identical content does not collide.

The interesting engineering is not the hash — it is the integration. The check-not-exists is folded into the block-number allocation’s ZooKeeper transaction, so the whole thing is one round trip with no TOCTOU window. The hash is computed lazily and cached, so the cost is paid once per block even when it spans several partitions. The representation wrappers are stripped before hashing, so a retry that arrives with a different physical layout still deduplicates. And the window is bounded by both a count and a self-relative age, so a clock jump cannot cause the history to evaporate or to persist forever.

The cost is a hash pass over the inserted data, one znode per block, and a guarantee that only holds inside the window. In exchange, a client that retries an insert after a network hiccup does not double its data — which, for a pipeline that writes to ClickHouse from a queue, is worth a great deal.

Insert on Opensource internals

How ClickHouse Maps an INSERT onto the Table's Schema

The problem in three shapes

Which columns are insertable

The sample block: by name

From the AST to the names

The INSERT ... SELECT case: positional, not by name

The pipeline

The fan-out

The views: InsertDependenciesBuilder

execute(): the routing

The lock

The whole picture

Trade-offs

The bottom line

The thing people get wrong

How ClickHouse Batches Async Inserts in a Per-Key Queue

The problem is that “one batch” is not a well-defined thing

The key: a hash of everything that matters

The queue: a multimap of deadlines plus a hash index

The buffer: a list of entries, each with a promise

When to flush: three triggers

The adaptive deadline

The deadline worker

The flush: a whole new query

The rows: parse everything, insert once

The client side: two different contracts

The escape hatch: too much data

Things that are refused

Trade-offs

The bottom line

The thing people get wrong

How ClickHouse Deduplicates Retried Inserts with a Content Hash

The problem: networks retry

What gets hashed

The unified token

The ZooKeeper check

The window

The non-replicated case: a local log

The retry loop

Self-deduplication: the same block twice

The user token

When is deduplication on?

Trade-offs

The bottom line

The `INSERT ... SELECT` case: positional, not by name

The views: `InsertDependenciesBuilder`

`execute()`: the routing