Replication on Opensource internals

How ClickHouse Replicates MergeTree: A ZooKeeper Log and a Hash Deduplication

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: ReplicatedMergeTree is a log in ZooKeeper plus a set of workers that execute the log’s entries. The log is a directory of sequential znodes holding text; each replica copies the log into its own queue and drains it out of order. The interesting part is the insert path: the block number is allocated by an ephemeral-sequential znode, and in the same atomic multi the insert checks whether the block’s content hash already exists. If it does, the duplicate rows are removed and the part is rewritten.

Replication is a distributed systems problem wearing a database’s clothes. Three replicas of a table must end up with the same data. The network partitions. A replica goes down mid-merge. The client retries an insert because it did not see the acknowledgement.

The requirements that fall out of this:

A total order. The replicas must agree on what happened, not just on the final state — otherwise a merge on one replica and a drop on another produce two different tables.
Idempotence. Any operation may be executed twice, and the second execution must be a no-op.
Convergence without a coordinator. No replica may be the one that decides what the others do.

ClickHouse’s answer to (1) is a ZooKeeper log. Its answer to (2) is a content hash. Its answer to (3) is that every replica reads the same log and decides independently what it can execute right now.

The log

The log is a directory of sequential znodes under /log. Each znode is named log-0000000001, log-0000000002, … and its content is a text blob describing one operation:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.h:27
 enum Type
 {
 EMPTY, /// Not used.
 GET_PART, /// Get the part from another replica.
 ATTACH_PART, /// Attach the part, possibly from our own replica (if found in /detached folder).
 /// You may think of it as a GET_PART with some optimizations as they're nearly identical.
 MERGE_PARTS, /// Merge the parts.
 DROP_RANGE, /// Delete the parts in the specified partition in the specified number range.
 CLEAR_COLUMN, /// NOTE: Deprecated. Drop specific column from specified partition.
 CLEAR_INDEX, /// NOTE: Deprecated. Drop specific index from specified partition.
 REPLACE_RANGE, /// Drop certain range of parts and replace them by new ones
 MUTATE_PART, /// Apply one or several mutations to the part.
 ALTER_METADATA, /// Apply alter modification according to global /metadata and /columns paths
 SYNC_PINNED_PART_UUIDS, /// Synchronization point for ensuring that all replicas have up to date in-memory state.
 CLONE_PART_FROM_SHARD, /// Clone part from another shard.
 DROP_PART, /// NOTE: Virtual (has the same (de)serialization format as DROP_RANGE). Deletes the specified part.
 };

The log entry is human-readable, and it is versioned:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.cpp:77
 out << "format version: " << format_version << "\n"
 << "create_time: " << LocalDateTime(create_time ? create_time : time(nullptr), DateLUT::serverTimezoneInstance()) << "\n"
 << "source replica: " << source_replica << '\n'
 << "block_id: " << escape << block_hashes_string << '\n';

with a version enum that grows as features are added:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.cpp:26
enum FormatVersion : UInt8
{
 FORMAT_WITH_CREATE_TIME = 2,
 FORMAT_WITH_BLOCK_ID = 3,
 FORMAT_WITH_DEDUPLICATE = 4,
 FORMAT_WITH_UUID = 5,
 FORMAT_WITH_DEDUPLICATE_BY_COLUMNS = 6,
 FORMAT_WITH_LOG_ENTRY_ID = 7,

 FORMAT_LAST = 8,
};

This matters because the log is shared between replicas of different versions during a rolling upgrade. An old replica must be able to read a new replica’s entries (at least the subset it understands), and a new replica must be able to read an old replica’s entries. A text format with an explicit version number, where unknown trailing fields are skipped, is far easier to evolve than a binary one.

Copying the log into a queue

A replica does not execute the shared log directly. It copies the entries into its own queue, under /replicas/<name>/queue/, and remembers how far it has read with a log_pointer. The copy is a PersistentSequential create, so the queue entries are numbered in the order they were copied:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:938
 ops.emplace_back(zkutil::makeCreateRequest(
 fs::path(replica_path) / "queue/queue-", res.data, zkutil::CreateMode::PersistentSequential));

and the pointer is advanced in the same multi, so the copy and the pointer move are atomic:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:953
 ops.emplace_back(zkutil::makeSetRequest(
 fs::path(replica_path) / "log_pointer", toString(last_entry_index + 1), -1));

Why copy at all? Because a replica needs a private queue: entries it has already executed are removed from it, entries it cannot execute yet (because they conflict with a running merge) stay in it, and the shared log is append-only for everyone. The queue is the replica’s private “to-do list”, derived from the shared “history”.

The copy happens in batches, and the batch size grows exponentially so that a replica that has been offline for a long time catches up with a handful of round trips instead of thousands:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:903
 /// Increase the batch size exponentially, so it will saturate to MAX_MULTI_OPS.
 if (current_multi_batch_size < MAX_MULTI_OPS)
 current_multi_batch_size = std::min<size_t>(MAX_MULTI_OPS, current_multi_batch_size * 2);

Executing out of order

Here is the part that surprises people. The queue is a queue, but the execution order is not the queue order:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.h:96
 /** The queue of what you need to do on this line to catch up. It is taken from ZooKeeper (/replicas/me/queue/).
 * In ZK records in chronological order. Here they are executed in parallel and reorder after entry execution.
 * Order of execution is not "queue" at all. Look at selectEntryToProcess.
 */
 Queue queue;

The selection walks the queue and takes the first entry that is currently executable:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:2202
 for (auto it = queue.begin(); it != queue.end(); ++it)
 {
 if ((*it)->currently_executing)
 continue;

 if (shouldExecuteLogEntry(**it, (*it)->postpone_reason, merger_mutator, data, committing_blocks, lock))
 {
 entry = *it;
 /// We gave a chance for the entry, move it to the tail of the queue, after that
 /// we move it to the end of the queue.
 queue.splice(queue.end(), queue, it);
 break;
 }

 ++(*it)->num_postponed;
 (*it)->last_postpone_time = time(nullptr);
 }

Two details are worth pausing on.

The entry is moved to the tail when it is selected. So a failing entry goes to the back of the line and does not block the entries behind it. This is the difference between “one bad merge stalls the whole replica” and “one bad merge is retried while the rest of the queue drains”.

“Currently executable” is a real predicate. An entry may be postponed because:

it failed recently, and there is an exponential backoff on it;
it produces a part that a currently executing entry also produces (isCoveredByFuturePartsImpl);
it intersects a DROP_RANGE/REPLACE_RANGE intent;
it produces a part that is going to be dropped anyway — “Optimization: it does not really make sense to generate parts that are going to be dropped anyway”;
the fetch pool is saturated.

The invariant that makes all of this safe is spelled out in the header:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.h:90
 /// A set of parts that should be on this replica according to the queue entries that have been done
 /// up to this point. The invariant holds: `virtual_parts` = `current_parts` + `queue`.
 /// Note: it can be different from the actual set of parts because the replica can decide to fetch
 /// a bigger part instead of the part mentioned in the log entry.
 ActiveDataPartSet current_parts;

virtual_parts is what the replica will have once the queue drains; it is computed, not observed. That is what lets the merge-assignment logic reason about the future without talking to anybody.

Which pool

Not all entries are equal. Fetching a part is I/O-bound and small; a merge is CPU- and disk-bound and large. So the entry type picks the thread pool:

 cpp
 
 
// src/Storages/StorageReplicatedMergeTree.cpp:4364
 /// Depending on entry type execute in fetches (small) pool or big merge_mutate pool
 if (job_type == LogEntry::GET_PART || job_type == LogEntry::ATTACH_PART)
 {
 assignee.scheduleFetchTask(std::make_shared<ExecutableLambdaAdapter>(
 [this, selected_entry] () mutable
 {
 return processQueueEntry(selected_entry);
 }, common_assignee_trigger, getStorageID()));
 return true;
 }
 if (job_type == LogEntry::MERGE_PARTS)
 {
 auto task = std::make_shared<MergeFromLogEntryTask>(selected_entry, *this, common_assignee_trigger);
 assignee.scheduleMergeMutateTask(task);
 return true;
 }

A replica that is catching up after being offline for an hour has hundreds of GET_PART entries and a handful of MERGE_PARTS entries; putting them in separate pools means the merges do not starve the fetches (or the other way round).

When an entry completes, it is removed from the RAM queue and then from ZooKeeper:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:788
 /// Remove the job from the queue in the RAM.
 /// You can not just refer to a pre-saved iterator, because someone else might be able to delete the task.
 /// Why do we view the queue from the end?
 /// - because the task for execution first is moved to the end of the queue, so that in case of failure it remains at the end.
 for (Queue::iterator it = queue.end(); it != queue.begin();)

The iteration is from the end precisely because of the “move to the tail” rule above.

The insert path: block number and deduplication in one round trip

Now the heart of it. A client inserts a block. The replica must:

give the resulting part a name;
know that this block has not been inserted before;
do both atomically with respect to other replicas doing the same thing.

The block number

Every MergeTree part has a block number — the number of the last block of rows in it — and in a replicated table that number has to be allocated by a coordinator, because two replicas inserting concurrently must not pick the same number.

ClickHouse allocates it with an ephemeral sequential znode:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:53
EphemeralLockInZooKeeper createEphemeralLockInZooKeeper(
 const String & path_prefix_,
 const String & temp_path,
 const ZooKeeperWithFaultInjectionPtr & zookeeper_,
 const std::vector<String> & deduplication_paths,
 const std::optional<String> & znode_data)
{
 String path;

 if (deduplication_paths.empty())
 {
 String holder_path = znode_data.value_or(temp_path + "/" + EphemeralLockInZooKeeper::LEGACY_LOCK_OTHER);
 path = zookeeper_->create(path_prefix_, holder_path, zkutil::CreateMode::EphemeralSequential);
 }

ZooKeeper’s sequential create appends a 10-digit counter to the path prefix. The counter is the block number:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:106
 const UInt64 number = parseSequentialNodeNumber(path, path_prefix_.size());
 warnIfBlockNumberIsHigh(number, path);
 CurrentMetrics::max(CurrentMetrics::MaxAllocatedEphemeralLockSequentialNumber, number);
 return EphemeralLockInZooKeeper{path_prefix_, zookeeper_, path, number};

This is a lovely piece of engineering: the block number is allocated by the same operation that creates the lock, in one round trip, with no read- modify-write. The node is ephemeral, so if the session dies the number is released — but the number is not reused, because the sequential counter is monotone regardless.

The path is per partition:

 cpp
 
 
// src/Storages/StorageReplicatedMergeTree.cpp:7987
 auto lock = createEphemeralLockInZooKeeper(
 fs::path(partition_path) / "block-", fs::path(zookeeper_table_path) / "temp", zookeeper, zookeeper_block_id_paths, znode_data);

so /block_numbers/<partition>/block-0000000123 is the lock for block number 123 of that partition. The lock is deleted as part of the commit multi.

The deduplication check rides along

The second thing the insert needs is a duplicate check. The deduplication id is a hash of the block’s content, not the block number:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:69
std::string DeduplicationHash::getBlockId() const
{
 return fmt::format("{}_{}_{}", partition_id, hash.items[0], hash.items[1]);
}

and it lives in a znode whose content is the name of the part that holds that data:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:75
std::string DeduplicationHash::getPath(const std::string & storage_path) const
{
 std::string hashes_directory = [&] ()
 {
 switch (hash_type)
 {
 case HashType::SYNC:
 return "blocks";
 case HashType::ASYNC:
 return "async_blocks";
 case HashType::UNIFIED:
 return "deduplication_hashes";
 }
 }();
 return (std::filesystem::path(storage_path) / hashes_directory / getBlockId()).string();
}

The three directories are a compatibility story: blocks is the old sync-insert location, async_blocks the old async-insert one, and deduplication_hashes the unified one that both now share.

Now the trick. The duplicate check and the lock creation are performed in a single multi:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:67
 else
 {
 String holder_path = znode_data.value_or(temp_path + "/" + EphemeralLockInZooKeeper::LEGACY_LOCK_INSERT);

 /// Check for duplicates in advance, to avoid superfluous block numbers allocation
 Coordination::Requests ops;

 for (const auto & single_dedup_path : deduplication_paths)
 zkutil::addCheckNotExistsRequest(ops, *zookeeper_, single_dedup_path);

 auto deduplication_path_ops_size = ops.size();

 ops.emplace_back(zkutil::makeCreateRequest(path_prefix_, holder_path, zkutil::CreateMode::EphemeralSequential));
 Coordination::Responses responses;
 Coordination::Error e = zookeeper_->tryMulti(ops, responses);
 if (e == Coordination::Error::ZNODEEXISTS)
 {
 auto failed_idx = zkutil::getFailedOpIndex(Coordination::Error::ZNODEEXISTS, responses);

 if (failed_idx < deduplication_path_ops_size)
 {
 const String & failed_op_path = ops[failed_idx]->getPath();
 LOG_DEBUG(
 getLogger("createEphemeralLockInZooKeeper"),
 "Deduplication path already exists: deduplication_path={}",
 failed_op_path);
 return EphemeralLockInZooKeeper{"", nullptr, "", 0, failed_op_path};
 }
 }

Read that carefully, because it is the whole trick in one function:

The multi is [check-not-exists(dedup_path_1), ..., create(block-)].
If the dedup path does not exist, the create succeeds and the block number is allocated — atomically, in one round trip.
If the dedup path does exist, the multi fails with ZNODEEXISTS, the failed op index tells you which dedup path conflicted, and the function returns a lock with isLocked() == false and a conflict_path.

The caller then knows, without any extra round trip, that this exact content has already been inserted:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:934
 auto block_number_lock = storage.allocateBlockNumber(part->info.getPartitionId(), zookeeper, block_id_pathes, "", block_data); /// 1 RTT

 ThreadFuzzer::maybeInjectSleep();

 {
 std::filesystem::path conflict_path = block_number_lock.getConflictPath();
 if (!conflict_path.empty())
 {
 LOG_DEBUG(log, "Cannot get lock, the conflict path is {}", conflict_path);
 auto conflicted_hash_it = std::find_if(
 deduplication_hashes.begin(),
 deduplication_hashes.end(),
 [&](const DeduplicationHash & hash)
 {
 return hash.getBlockId() == conflict_path.filename().string();
 });
 chassert(conflicted_hash_it != deduplication_hashes.end());
 retry_context.conflict_deduplication_hashes.push_back(*conflicted_hash_it);

 return CommitRetryContext::RESOLVE_CONFLICTS;
 }
 }

The comment /// 1 RTT on the allocateBlockNumber call is the author telling you that this whole thing — check for duplicates and allocate the block number — costs exactly one round trip to ZooKeeper. That is the entire point. A naive implementation would be: read the dedup znode (1 RTT), decide, create the block number (1 RTT), check again (1 RTT). Three round trips, and a race window between the check and the create.

Resolving the conflict

When the multi fails with a conflict, the replica needs to know which part holds the duplicate data — that is what the dedup znode’s content is for. It reads the conflicting znode:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:767
 auto response = zookeeper->tryGet(getDeduplicationPaths(storage.zookeeper_path, retry_context.conflict_deduplication_hashes));
 for (size_t i = 0; i < retry_context.conflict_deduplication_hashes.size(); ++i)
 {
 auto & deduplication_hash = retry_context.conflict_deduplication_hashes[i];
 const auto & resp = response[i];

and then the rows that duplicate the existing part are removed from the block, and the part is rewritten with what is left:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:512
 auto result = partition.deduplication_info->deduplicateBlock(
 getDeduplicationBlockIds(conflicts),
 partition.block_with_partition.partition_id,
 context);

If everything was a duplicate, the part is dropped and the insert reports that it was deduplicated:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:538
 if (partition.block_with_partition.block->rows() == 0)
 {
 // Whole block was deduplicated

 if (!is_async_insert)
 {
 chassert(conflicts.size() == 1);
 auto block_id = conflicts.front().getBlockId();

and the retry loop goes back to the top and re-commits the remaining rows with a fresh block number.

Note what this means: the deduplication is not “skip the whole insert”. It is “remove the rows that are already there and insert the rest”. A block that partially overlaps with an existing part is inserted partially. That is why the dedup id is a content hash and not the block number: the block number is allocated per partition and is meaningless as a duplicate detector, while the content hash is exactly the question “have I seen these rows before?”.

The whole insert, end to end

 text
 
 
 INSERT (block B into partition P)
 |
 | hash the content of B -> dedup id = "<partition>_<hash>"
 v
 multi( [check-not-exists(/deduplication_hashes/<id>),
 create(/block_numbers/P/block-0000000123, EPHEMERAL_SEQUENTIAL)] )
 |
 +--- ZNODEEXISTS --> the block is a duplicate
 | read the znode -> the part that holds it
 | drop the duplicate rows, rewrite the part, retry
 |
 +--- ZOK --> block number = 123
 part->info.min_block = max_block = 123
 part name = <partition>_<min_block>_<max_block>_<level>
 |
 v
 multi( [create(/log/log-...), delete(block-number lock),
 create(quorum znode), create(shared data lock), ...] )
 |
 v
 the log entry "get <part name>" is now the history;
 every replica will fetch the part from whoever has it

The quorum

For a replicated table, “the insert succeeded” means more than “the part is on disk here”. The insert waits for a quorum of replicas to have the part:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:789
 /** If we need a quorum - create a node in which the quorum is monitored.
 * (If such a node already exists, then someone has managed to make another quorum record at the same time,
 * but for it the quorum has not yet been reached.
 * You can not do the next quorum record at this time.)
 */
 if (isQuorumEnabled())
 {
 ReplicatedMergeTreeQuorumEntry quorum_entry;
 quorum_entry.part_name = part->name;
 quorum_entry.required_number_of_replicas = getQuorumSize(total_replicas_count);
 quorum_entry.replicas.insert(storage.replica_name);

The quorum znode is a classic “watch and count” pattern: each replica that receives the part adds itself to the znode’s content, and when the count reaches the required number the znode is deleted, which wakes up the waiters.

Looking at it

 sql
 
 
-- The replication log, as text.
SELECT * FROM system.zookeeper_log WHERE path LIKE '/clickhouse/tables/%/log/%' LIMIT 5;

-- The per-replica queue.
SELECT replica_name, type, new_part_name, num_tries, num_postponed, last_exception
FROM system.replication_queue
ORDER BY create_time;

-- The deduplication znodes, with the part name they point at.
SELECT name, length(value) FROM system.zookeeper
WHERE path = '/clickhouse/tables/{shard}/default/tbl/deduplication_hashes';

Trade-offs

What the log buys you:

A total order that every replica can read, without a coordinator.
Recoverability. A replica that was down for an hour catches up by reading the log; it does not need to be told what happened.
Human-readable history. The log entries are text, with an explicit format version, which is why a rolling upgrade with mixed versions works and why you can debug a replication problem with system.zookeeper.

What it costs:

ZooKeeper is in the write path. Every insert is at least one round trip to ZooKeeper (the dedup + block-number multi) plus one more (the commit multi that creates the log entry). On a cross-availability-zone deployment, that is milliseconds.
The log grows forever unless something truncates it. The queue is trimmed per replica, but the shared log is the history.
The dedup window is finite. The deduplication znodes are only kept for the last replicated_deduplication_window blocks (10000 by default), so a retry that arrives much later is not deduplicated. This is a correctness-relevant limit, not an optimization.

What the block-number lock buys you:

One round trip for both the duplicate check and the number allocation.
No read-modify-write. The number comes from ZooKeeper’s sequential counter, so two concurrent inserts on two replicas cannot get the same number, and a crashed session cannot leave a gap that is reused.

What it costs:

The block number is not the dedup id. It is tempting to think “the part is named after the block number, so the block number is the duplicate detector”, but that is wrong: the block number is allocated per partition and says nothing about content. The dedup id is a content hash, and it is the hash that makes a retried insert a no-op.
A duplicate is not free. The conflict path is resolved by removing the duplicate rows from the block and rewriting the part, which means a partially-overlapping insert is written twice: once (partially) rejected, once rewritten with the remainder.
The dedup znodes are garbage that has to be collected. They are deleted lazily, when the parts they point at are dropped, and the window setting bounds how far back they go.

The part people get wrong: ReplicatedMergeTree is not exactly-once by itself. It is at-least-once delivery plus a finite-window content-hash deduplication. A client that retries an insert more than replicated_deduplication_window blocks later will duplicate the rows. If you need a stronger guarantee, the dedup id has to come from the application — the insert_deduplication_token setting lets the client supply its own id, which is then used instead of the content hash and is stable across retries forever.

How ClickHouse Deduplicates Retried Inserts with a Content Hash

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: Every inserted block is hashed column-by-column into a UInt128. The hash, plus the partition id, forms a “deduplication token” that is recorded in a bounded window — in ZooKeeper for replicated tables, in a local log file otherwise. An insert whose token is already in the window is recognised as a retry and silently skipped.

The problem: networks retry

INSERT INTO t VALUES (...) is a request over a network. Networks drop packets, proxies time out, and load balancers retry. A client that gets no response does not know whether the server applied the insert or not, so it sends it again.

For an INSERT that appends rows, that means the data is written twice.

ClickHouse’s answer is to make the insert idempotent: if the exact same bytes have already been inserted recently, do not insert them again. The question is how to recognise “the exact same bytes” cheaply, and how to bound the memory of the recognition.

What gets hashed

The hash is over the data, not over a client-supplied identifier. The DeduplicationInfo chunk-info object carries a list of tokens; a token is either a user-supplied string or a computed content hash.

The content hash is computed column-major over the token’s row range:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:402
void DeduplicationInfo::calculateDataHashes() const
{
 std::vector<size_t> pending;
 for (size_t offset = 0; offset < tokens.size(); ++offset)
 if (tokens[offset].by_user.empty() && !tokens[offset].data_hash_batch.has_value())
 pending.push_back(offset);

 if (pending.empty())
 return;

 /// Counted per token, not per pass: pre-warming keeps this at O(tokens); the per-partition
 /// cold-clone path drives it to O(partitions*tokens) for tokens that span several partitions.
 ProfileEvents::increment(ProfileEvents::DuplicationDataHashComputations, pending.size());

 chassert(original_block && original_block->rows() == getRows());

 /// The hash must not depend on the column's representation, so strip every representation wrapper
 /// (`ColumnConst`, `ColumnReplicated`, `ColumnSparse`) before hashing. A wrapper hashes through the
 /// generic per-row loop, whose byte stream differs from the dense column's range overload for every
 /// variable-size type - the same rows would then produce two different block hashes and a retried
 /// insert would not deduplicate.
 /// `convertToFullIfWrapped` strips the whole column tree rather than just the root, because a
 /// composite column hashes by delegating to its children (`ColumnTuple::updateHashWithValueRange`
 /// calls each element's), so a wrapper anywhere in the tree would reach the generic loop. Every
 /// wrapper observed here so far sits at the root, and it costs nothing to be exhaustive: with no
 /// wrapper anywhere the call returns the column itself.
 /// `LowCardinality` is deliberately not removed: it is a semantic type rather than a representation,
 /// so it cannot differ between two attempts at the same insert.
 /// Column-major so only one column is materialized at a time, instead of a dense copy of the whole block.
 std::vector<SipHash> hashes(pending.size());
 for (const auto & col : original_block->getColumns())
 {
 auto dense = col->convertToFullIfWrapped();
 for (size_t i = 0; i < pending.size(); ++i)
 dense->updateHashWithValueRange(getTokenBegin(pending[i]), getTokenEnd(pending[i]), hashes[i]);
 }

Three details in that comment are worth their own paragraphs.

The hash must not depend on the column’s representation. A ColumnConst (a column where every row is the same value) hashes through a generic per-row loop, while a dense column hashes through a range overload. The two produce different byte streams for the same logical values. If a retry arrives with the values in a different physical representation — say, a constant-folded column versus a materialized one — the hashes would differ and the retry would not be deduplicated. convertToFullIfWrapped() normalizes the representation away.

LowCardinality is deliberately not stripped, because it is a semantic type, not a representation: the same logical values always have the same LowCardinality encoding.

The hash is column-major. Only one column is materialized at a time, rather than a dense copy of the whole block. For a block of 65536 rows and 100 columns, the row-major alternative would need a 6.5-million-element buffer; the column-major version needs one column’s worth.

The hash is per token, not per block. A block can carry several tokens if it was assembled from several sources — see below. The pending list collects the tokens that do not yet have a hash, and the whole column is walked once for all of them.

The result is a UInt128 per token — 128 bits of SipHash output. The choice of 128 bits matters: the token is a dictionary key in ZooKeeper, and a 64-bit hash would make accidental collisions a real concern at scale.

The unified token

The content hash is only half the story. The final token is a SipHash over a string that embeds the content hash plus a set of extra tokens:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:456
DeduplicationHash DeduplicationInfo::getBlockUnifiedHash(size_t offset, const std::string & partition_id) const
{
 // when there is no user token, compute the full column-wise hash of the data
 // this hash is used for deduplication of sync and async inserts in a unified manner

 auto & token = tokens[offset];

 std::string extension;
 if (!token.by_user.empty())
 {
 extension = "user-token-" + token.by_user;
 }
 else
 {
 auto data_hash = getDataHash(offset);
 extension = fmt::format("{}_{}", data_hash.items[0], data_hash.items[1]);
 }

 // for other token sources addition parts are appended
 for (const auto & extra : token.extra_tokens)
 {
 if (is_async_insert && extra.type == TokenDefinition::Extra::SOURCE_NUMBER)
 {
 // do not include source number for async inserts, they are not relevant as data hash is used or user token
 // a token describes only the data in one block
 extension.append(":");
 extension.append(TokenDefinition::Extra::asSourceNumber(0).toString());
 continue;
 }

 extension.append(":");
 extension.append(extra.toString());
 }

 SipHash hash;
 hash.update(extension.data(), extension.size());
 return DeduplicationHash::createUnifiedHash(hash.get128(), partition_id);
}

The “extra tokens” are the provenance of the block: which source it came from, and which materialized view it passed through. They are what keeps two different blocks with identical content from colliding.

The Extra type enumerates them:

 cpp
 
 
// src/Interpreters/InsertDeduplication.h:208
 struct Extra
 {
 enum Type
 {
 SOURCE_ID,
 SOURCE_NUMBER,
 VIEW_ID,
 VIEW_NUMBER,
 };

 Type type;

 using Range = std::pair<size_t, size_t>;
 std::variant<Range, StorageIDMaybeEmpty> value_variant;

SOURCE_ID is the root table the insert started at, SOURCE_NUMBER is the block’s ordinal within it, VIEW_ID/VIEW_NUMBER identify a materialized view in the chain. When an insert feeds a materialized view, the view’s data is a projection of the source data, and two different source blocks can legitimately produce identical view rows. Without the view id in the token, the second one would be wrongly deduplicated.

The SOURCE_NUMBER special case for async inserts is the flip side: an async insert’s blocks are batched and reordered, so the block ordinal is not stable across retries, and including it would break deduplication entirely. The comment says so.

The final token string, and the block id derived from it:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:69
std::string DeduplicationHash::getBlockId() const
{
 return fmt::format("{}_{}_{}", partition_id, hash.items[0], hash.items[1]);
}

So the block id is partition_<hash_high>_<hash_low>. The partition id is part of the key, which is why the same rows inserted into two different partitions are not deduplicated against each other.

And the ZooKeeper path, which reveals the three generations of this feature that are still on disk:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:75
std::string DeduplicationHash::getPath(const std::string & storage_path) const
{
 std::string hashes_directory = [&] ()
 {
 switch (hash_type)
 {
 case HashType::SYNC:
 return "blocks";
 case HashType::ASYNC:
 return "async_blocks";
 case HashType::UNIFIED:
 return "deduplication_hashes";
 }
 }();
 return (std::filesystem::path(storage_path) / hashes_directory / getBlockId()).string();
}

blocks/ is the old sync-insert directory, async_blocks/ the old async one, and deduplication_hashes/ the current unified one. The cleanup thread still drains the legacy directory during a rolling upgrade:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:76
 // Old replicas (pre-unified-hash) still write the legacy async-insert ids to /async_blocks
 // during a rolling upgrade; the current leader keeps draining that directory with the legacy
 // window settings until the minimum supported version no longer uses it.

The ZooKeeper check

The dedup check for a replicated table is a check-not-exists request folded into the same ZooKeeper multi-op that allocates the block number. It is not a separate round trip:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:71
 /// Check for duplicates in advance, to avoid superfluous block numbers allocation
 Coordination::Requests ops;

 for (const auto & single_dedup_path : deduplication_paths)
 zkutil::addCheckNotExistsRequest(ops, *zookeeper_, single_dedup_path);

 auto deduplication_path_ops_size = ops.size();

 ops.emplace_back(zkutil::makeCreateRequest(path_prefix_, holder_path, zkutil::CreateMode::EphemeralSequential));
 Coordination::Responses responses;
 Coordination::Error e = zookeeper_->tryMulti(ops, responses);
 if (e == Coordination::Error::ZNODEEXISTS)
 {
 auto failed_idx = zkutil::getFailedOpIndex(Coordination::Error::ZNODEEXISTS, responses);

 if (failed_idx < deduplication_path_ops_size)
 {
 const String & failed_op_path = ops[failed_idx]->getPath();
 LOG_DEBUG(
 getLogger("createEphemeralLockInZooKeeper"),
 "Deduplication path already exists: deduplication_path={}",
 failed_op_path);
 return EphemeralLockInZooKeeper{"", nullptr, "", 0, failed_op_path};
 }
 }

The conflict_path in the returned lock is how the sink learns which hash collided — the failed op’s path carries the block id.

This is a genuinely elegant bit of engineering. The dedup check and the block-number allocation are one ZooKeeper transaction, so:

There is no TOCTOU window between “check for duplicate” and “allocate”.
A duplicate is detected before a block number is consumed.
The whole thing costs one round trip, not two.

The comment on the sink side confirms the cost:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:934
 auto block_number_lock = storage.allocateBlockNumber(part->info.getPartitionId(), zookeeper, block_id_pathes, "", block_data); /// 1 RTT

The window

The dedup history is bounded. For replicated tables, the bound is two settings — a count and a time:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1545
 DECLARE(UInt64, replicated_deduplication_window, 10000, R"(
The number of most recently inserted blocks for which ClickHouse Keeper stores
hash sums to check for duplicates.

and

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1566
 DECLARE(UInt64, replicated_deduplication_window_seconds, 60 * 60 /* one hour */, R"(
The number of seconds after which the hash sums of the inserted blocks are
removed from ClickHouse Keeper.

The cleanup thread enforces both bounds — the count and the age:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:383
 /// Use ZooKeeper's first node (last according to time) timestamp as "current" time.
 Int64 current_time = timed_blocks.front().ctime;
 Int64 time_threshold = std::max(
 static_cast<Int64>(0),
 current_time - static_cast<Int64>(1000 * window_seconds));

 /// Virtual node, all nodes that are "greater" than this one will be deleted
 NodeWithStat block_threshold{{}, time_threshold, 0, 0};

 size_t current_deduplication_window = std::min<size_t>(timed_blocks.size(), window_size);
 auto first_outdated_block_fixed_threshold = timed_blocks.begin() + current_deduplication_window;
 auto first_outdated_block_time_threshold = std::upper_bound(
 timed_blocks.begin(), timed_blocks.end(), block_threshold, NodeWithStat::greaterByTime);
 auto first_outdated_block = std::min(first_outdated_block_fixed_threshold, first_outdated_block_time_threshold);

A node is deleted if it is either beyond the count window or older than the time window. The time is measured against the newest node’s ctime, not the wall clock:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1579
The time is relative to the time of the most recent record, not to the wall
time. If it's the only record it will be stored forever.

That is deliberate. If the server’s clock jumps, or if the table is idle, a wall-clock-based expiry would either delete everything or keep everything. Measuring from the newest record makes the window self-relative.

The deletions are issued asynchronously and in parallel:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:407
 zkutil::AsyncResponses<Coordination::RemoveResponse> try_remove_futures;
 for (auto it = first_outdated_block; it != timed_blocks.end(); ++it)
 {
 String path = zookeeper_path + "/" + blocks_dir_name + "/" + it->node;
 try_remove_futures.emplace_back(path, zookeeper.asyncTryRemove(path, it->version));
 }

The non-replicated case: a local log

A non-replicated MergeTree has no ZooKeeper, so the dedup history lives in a file on the table’s own disk. The setting is off by default:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:763
 DECLARE(UInt64, non_replicated_deduplication_window, 0, R"(
The number of the most recently inserted blocks in the non-replicated
[MergeTree](/reference/engines/table-engines/mergetree-family/mergetree) table
for which hash sums are stored to check for duplicates.

and the mechanism is described in the setting’s own text:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:772
A deduplication mechanism is used, similar to replicated tables (see
[replicated_deduplication_window](#replicated_deduplication_window) setting): the deduplication
hash sum covers the whole inserted block. The hash sums are written to
a local file on a disk rather than to ClickHouse Keeper.

The log’s on-disk format is deliberately human-readable — tab-separated records of operation, part name and block id:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:126
/// Fixed-size log for deduplication in non-replicated MergeTree.
/// Stores records on disk for zero-level parts in human-readable format:
/// operation part_name partition_id_check_sum
/// 1 88_18_18_0 88_10619499460461868496_9553701830997749308
/// 2 77_14_14_0 77_15147918179036854170_6725063583757244937
/// 2 77_15_15_0 77_14977227047908934259_8047656067364802772
/// 1 77_20_20_0 77_15147918179036854170_6725063583757244937
/// Also stores them in memory in hash table with limited size.

The in-memory index is a hash map with a FIFO eviction policy, implemented as a list plus an index into it:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:29
template <typename V>
class LimitedOrderedHashMap
{
private:
 struct ListNode
 {
 std::string key;
 V value;
 };
 using Queue = std::list<ListNode>;
 using IndexMap = std::unordered_map<std::string_view, typename Queue::iterator, StringViewHash>;

 Queue queue;
 IndexMap map;
 size_t max_size;

The std::list gives O(1) FIFO eviction from the front; the unordered_map<string_view, iterator> gives O(1) lookup. The map stores string_views into the list nodes’ keys, so no key is copied twice.

The eviction is what makes the window a window:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:91
 bool insert(const std::string & key, const V & value)
 {
 auto it = map.find(key);
 if (it != map.end())
 return false;

 if (size() == max_size)
 {
 map.erase(queue.front().key);
 queue.pop_front();
 }

 ListNode elem{key, value};
 auto itr = queue.insert(queue.end(), elem);
 map.emplace(itr->key, itr);
 return true;
 }

When the map is full, the oldest entry is dropped. A retried insert is only deduplicated if the original is still in the window — which is exactly the semantics you want: a retry happens within seconds of the original, not hours later.

The log files themselves rotate, with a rotation interval of twice the window:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:94
 , rotate_interval(deduplication_window_ * 2) /// actually it doesn't matter

and the rotation is driven by the number of entries in the current file:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:255
void MergeTreeDeduplicationLog::rotateAndDropIfNeeded()
{
 /// If we don't have logs at all or already have enough records in current
 /// For the disk that doesn't support writing with append, we can't append logs to the last file.
 if (existing_logs.empty() || existing_logs[current_log_number].entries_count >= rotate_interval || !disk_supports_writing_with_append)
 {
 rotate();
 dropOutdatedLogs();
 }
}

Old logs are dropped by walking from the newest backwards and accumulating entry counts until the window is covered:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:211
void MergeTreeDeduplicationLog::dropOutdatedLogs()
{
 size_t current_sum = 0;
 size_t remove_from_value = 0;
 /// Go from end to the beginning
 for (auto itr = existing_logs.rbegin(); itr != existing_logs.rend(); ++itr)
 {
 /// Never drop the current active log — it may still be open for writing
 if (itr->first == current_log_number)
 {
 current_sum += itr->second.entries_count;
 continue;
 }

 if (current_sum >= deduplication_window)
 {
 /// We have more logs than required, all older files (excluding current) can be dropped
 remove_from_value = itr->first;
 break;
 }

The comment “actually it doesn’t matter” on the rotation interval is a nice piece of honesty — the interval only affects how many files exist on disk, not the semantics.

The retry loop

When a conflict is detected, the sink does not fail. It filters out the duplicate rows and retries with what is left:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:353
 ProfileEvents::increment(ProfileEvents::DuplicatedInsertedBlocks, conflicts.size());

 auto result = partition.deduplication_info->deduplicateBlock(
 conflicts,
 partition.block_with_partition.partition_id,
 context);

deduplicateBlock finds the offsets of the colliding tokens and filters them out of the block. If everything collided, the part is simply not committed:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:382
 if (result.filtered_block->rows() == 0)
 {
 LOG_DEBUG(
 storage.log,
 "All rows are deduplicated for part with block IDs: {}, skipping the part commit.",
 fmt::join(conflicts, ", "));

and the part log records the deduplication as an execution status, so system.part_log shows what happened:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:393
 PartLog::addNewPart(
 storage.getContext(),
 PartLog::PartLogEntry(partition.temp_part->part, partition.elapsed_ns, counters_snapshot),
 getDeduplicationBlockIds(deduplication_hashes),
 ExecutionStatus(ErrorCodes::INSERT_WAS_DEDUPLICATED, "The part was deduplicated"));

The filter itself is careful about a boundary case. If a partial collision happens, the colliding rows have to be sliced out of the block — but the block may have been re-anchored by an intermediate view, in which case the offsets no longer describe it:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:303
 chassert(original_block && !original_block->empty() && original_block->rows() > 0);

 auto & block = *original_block;

 /// A partial collision requires slicing the collided tokens' rows out of the block, which is
 /// only possible while the offsets still describe the block. Behind an `Alias` hop over a
 /// row-count-changing view the block is re-anchored to the view-output chunks and there is no
 /// mapping from the tokens' source rows to the block's rows anymore. Refuse loudly instead of
 /// reading out of the block's bounds; full-block deduplication is handled above.
 if (block.rows() != getRows())
 throw Exception(
 ErrorCodes::NOT_IMPLEMENTED,
 "Cannot filter {} of {} deduplicated blocks out of the insert: the deduplication info describes {} rows, "
 "but the block has {} rows because a materialized view with a row-count-changing inner query "
 "was processed before a table with the `Alias` engine. Only the whole insert can be deduplicated on this path. "
 "Debug: {}",

Failing loudly is the right call. Silently reading out of bounds would be a memory-safety bug; silently skipping the dedup would insert duplicates.

Self-deduplication: the same block twice

There is a second, subtler case: the same insert containing the same data twice. A client that retries a batch of inserts with insert_deduplication_token unset, or a materialized view that produces identical rows, can end up with duplicate content inside a single block.

deduplicateSelf handles that, and it operates on the token level:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:217
std::set<size_t> DeduplicationInfo::filterSelf(const String & partition_id) const
{
 if (getCount() <= 1)
 return {};

 std::set<size_t> fitered_offsets;
 /// fitered_offsets will contain all but first offsets for each block id
 /// so that only first occurrence of each block id will remain
 for (const auto & [_, block_offsets] : buildOffsetsMapImpl(partition_id))
 {
 if (block_offsets.size() > 1)
 fitered_offsets.insert(block_offsets.begin() + 1, block_offsets.end());
 }

 return fitered_offsets;
}

Group the tokens by their hash; within each group, keep the first and drop the rest. The grouping is done on the sorted hash values:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:496
std::vector<std::pair<UInt128, std::vector<size_t>>> DeduplicationInfo::buildOffsetsMapImpl(const std::string & partition_id) const
{
 /// (hash, offset) pairs sorted by hash, then offset; runs of equal hashes are the groups.
 std::vector<std::pair<UInt128, size_t>> sorted;
 sorted.reserve(offsets.size());
 for (size_t offset = 0; offset < offsets.size(); ++offset)
 sorted.emplace_back(getBlockUnifiedHash(offset, partition_id).hash, offset);
 std::sort(sorted.begin(), sorted.end());

Note that this recomputes the unified hash for every token — including the extra tokens, so the grouping is by the final token, not the raw content hash. Two tokens with the same content but different provenance are not duplicates.

The user token

A pipeline that writes its own idempotency key can bypass content hashing entirely, by supplying the key as a session setting:

 sql
 
 
INSERT INTO events SETTINGS insert_deduplication_token = 'batch-2026-10-22-0001'
VALUES (1, 'click'), (2, 'view');

The same token on a later insert means “this is a duplicate of the earlier one”; a different token means “this is new data, even if the rows are identical”.

The user can bypass content hashing entirely with the insert_deduplication_token setting:

 cpp
 
 
// src/Core/Settings.cpp:9135
 DECLARE(String, insert_deduplication_token, "", R"(
The setting allows a user to provide own deduplication semantic in MergeTree/ReplicatedMergeTree
For example, by providing a unique value for the setting in each INSERT statement,
user can avoid the same inserted data being deduplicated.

When the token is set, it replaces the content hash in the extension string ("user-token-" + token instead of the data hash). The semantics are therefore inverted: with a token, the same token means duplicate; without one, the same data means duplicate.

The setting’s documentation spells out the partition subtlety:

 cpp
 
 
// src/Core/Settings.cpp:9150
<Note>
`insert_deduplication_token` is tracked per partition, so multiple partitions written by one insert can carry the same token. Without a token, the default content checksum is computed over the whole inserted block, so an insert is deduplicated only when its entire data matches a previous insert (a retry), not when a single partition's rows happen to coincide with a different insert.
</Note>

When is deduplication on?

The whole thing is controlled by deduplicate_insert, with a backwards-compatible split between sync and async:

 cpp
 
 
// src/Core/DeduplicateInsert.cpp:26
bool isDeduplicationEnabledForInsert(bool is_async_insert, const Settings & settings)
{
 switch (settings[Setting::deduplicate_insert].value)
 {
 case DeduplicateInsertMode::BACKWARD_COMPATIBLE_CHOICE:
 return is_async_insert ? settings[Setting::async_insert_deduplicate] : settings[Setting::insert_deduplicate];
 case DeduplicateInsertMode::ENABLE:
 return true;
 case DeduplicateInsertMode::DISABLE:
 return false;
 }
}

For INSERT ... SELECT, there is an extra wrinkle: the SELECT might not be stable, meaning two executions return different data, and content-hash deduplication would then be meaningless. The mode ENABLE_WHEN_POSSIBLE handles it:

 cpp
 
 
// src/Core/DeduplicateInsert.cpp:73
 case DeduplicateInsertSelectMode::ENABLE_WHEN_POSSIBLE:
 {
 if (logger && !select_query_sorted && !has_user_token && isDeduplicationEnabledForInsert(false, settings))
 LOG_INFO(logger, "INSERT SELECT deduplication is disabled because SELECT is not stable");

 return (select_query_sorted || has_user_token) && isDeduplicationEnabledForInsert(false, settings);
 }

A stable SELECT is one with ORDER BY ALL and a single-stream pipeline — or, as the setting’s documentation puts it, a non-empty insert_deduplication_token is “an equivalent substitute for stability”.

Trade-offs

Space. The dedup window costs one ZooKeeper znode per inserted block (or one line in a local log). 10000 blocks is 10000 znodes, and the cleanup thread deletes them as they age out.

CPU. The content hash is a SipHash pass over every column of every inserted block. For a wide table that is not free — but it is one pass over data that is about to be written anyway, and it is only computed when deduplication is enabled and no user token was supplied.

Correctness of the window. Deduplication is only guaranteed within the window. An insert retried after the original has aged out is inserted twice. This is why the window is expressed in both a count and a time: the count covers a burst, the time covers a slow trickle.

The content hash is a heuristic. Two different blocks with the same content hash collide, and the second is dropped. With 128 bits of SipHash that is not a practical concern, but it is why the implementation bothers with UInt128 rather than UInt64.

The aliasing problem is a real limitation. Behind an Alias hop over a row-count-changing view, a partial collision cannot be resolved and the insert fails with NOT_IMPLEMENTED rather than corrupting anything. That is a deliberate trade: correctness over convenience.

The bottom line

Block-level insert deduplication is a content-addressed cache of recent inserts, with the cache keys being UInt128 SipHashes over a column-major hash of the block’s data, salted with the partition id and the block’s provenance so that unrelated identical content does not collide.

The interesting engineering is not the hash — it is the integration. The check-not-exists is folded into the block-number allocation’s ZooKeeper transaction, so the whole thing is one round trip with no TOCTOU window. The hash is computed lazily and cached, so the cost is paid once per block even when it spans several partitions. The representation wrappers are stripped before hashing, so a retry that arrives with a different physical layout still deduplicates. And the window is bounded by both a count and a self-relative age, so a clock jump cannot cause the history to evaporate or to persist forever.

The cost is a hash pass over the inserted data, one znode per block, and a guarantee that only holds inside the window. In exchange, a client that retries an insert after a network hiccup does not double its data — which, for a pipeline that writes to ClickHouse from a queue, is worth a great deal.