How ClickHouse Deduplicates Retried Inserts with a Content Hash

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: Every inserted block is hashed column-by-column into a UInt128. The hash, plus the partition id, forms a “deduplication token” that is recorded in a bounded window — in ZooKeeper for replicated tables, in a local log file otherwise. An insert whose token is already in the window is recognised as a retry and silently skipped.

The problem: networks retry

INSERT INTO t VALUES (...) is a request over a network. Networks drop packets, proxies time out, and load balancers retry. A client that gets no response does not know whether the server applied the insert or not, so it sends it again.

For an INSERT that appends rows, that means the data is written twice.

ClickHouse’s answer is to make the insert idempotent: if the exact same bytes have already been inserted recently, do not insert them again. The question is how to recognise “the exact same bytes” cheaply, and how to bound the memory of the recognition.

What gets hashed

The hash is over the data, not over a client-supplied identifier. The DeduplicationInfo chunk-info object carries a list of tokens; a token is either a user-supplied string or a computed content hash.

The content hash is computed column-major over the token’s row range:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:402
void DeduplicationInfo::calculateDataHashes() const
{
 std::vector<size_t> pending;
 for (size_t offset = 0; offset < tokens.size(); ++offset)
 if (tokens[offset].by_user.empty() && !tokens[offset].data_hash_batch.has_value())
 pending.push_back(offset);

 if (pending.empty())
 return;

 /// Counted per token, not per pass: pre-warming keeps this at O(tokens); the per-partition
 /// cold-clone path drives it to O(partitions*tokens) for tokens that span several partitions.
 ProfileEvents::increment(ProfileEvents::DuplicationDataHashComputations, pending.size());

 chassert(original_block && original_block->rows() == getRows());

 /// The hash must not depend on the column's representation, so strip every representation wrapper
 /// (`ColumnConst`, `ColumnReplicated`, `ColumnSparse`) before hashing. A wrapper hashes through the
 /// generic per-row loop, whose byte stream differs from the dense column's range overload for every
 /// variable-size type - the same rows would then produce two different block hashes and a retried
 /// insert would not deduplicate.
 /// `convertToFullIfWrapped` strips the whole column tree rather than just the root, because a
 /// composite column hashes by delegating to its children (`ColumnTuple::updateHashWithValueRange`
 /// calls each element's), so a wrapper anywhere in the tree would reach the generic loop. Every
 /// wrapper observed here so far sits at the root, and it costs nothing to be exhaustive: with no
 /// wrapper anywhere the call returns the column itself.
 /// `LowCardinality` is deliberately not removed: it is a semantic type rather than a representation,
 /// so it cannot differ between two attempts at the same insert.
 /// Column-major so only one column is materialized at a time, instead of a dense copy of the whole block.
 std::vector<SipHash> hashes(pending.size());
 for (const auto & col : original_block->getColumns())
 {
 auto dense = col->convertToFullIfWrapped();
 for (size_t i = 0; i < pending.size(); ++i)
 dense->updateHashWithValueRange(getTokenBegin(pending[i]), getTokenEnd(pending[i]), hashes[i]);
 }

Three details in that comment are worth their own paragraphs.

The hash must not depend on the column’s representation. A ColumnConst (a column where every row is the same value) hashes through a generic per-row loop, while a dense column hashes through a range overload. The two produce different byte streams for the same logical values. If a retry arrives with the values in a different physical representation — say, a constant-folded column versus a materialized one — the hashes would differ and the retry would not be deduplicated. convertToFullIfWrapped() normalizes the representation away.

LowCardinality is deliberately not stripped, because it is a semantic type, not a representation: the same logical values always have the same LowCardinality encoding.

The hash is column-major. Only one column is materialized at a time, rather than a dense copy of the whole block. For a block of 65536 rows and 100 columns, the row-major alternative would need a 6.5-million-element buffer; the column-major version needs one column’s worth.

The hash is per token, not per block. A block can carry several tokens if it was assembled from several sources — see below. The pending list collects the tokens that do not yet have a hash, and the whole column is walked once for all of them.

The result is a UInt128 per token — 128 bits of SipHash output. The choice of 128 bits matters: the token is a dictionary key in ZooKeeper, and a 64-bit hash would make accidental collisions a real concern at scale.

The unified token

The content hash is only half the story. The final token is a SipHash over a string that embeds the content hash plus a set of extra tokens:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:456
DeduplicationHash DeduplicationInfo::getBlockUnifiedHash(size_t offset, const std::string & partition_id) const
{
 // when there is no user token, compute the full column-wise hash of the data
 // this hash is used for deduplication of sync and async inserts in a unified manner

 auto & token = tokens[offset];

 std::string extension;
 if (!token.by_user.empty())
 {
 extension = "user-token-" + token.by_user;
 }
 else
 {
 auto data_hash = getDataHash(offset);
 extension = fmt::format("{}_{}", data_hash.items[0], data_hash.items[1]);
 }

 // for other token sources addition parts are appended
 for (const auto & extra : token.extra_tokens)
 {
 if (is_async_insert && extra.type == TokenDefinition::Extra::SOURCE_NUMBER)
 {
 // do not include source number for async inserts, they are not relevant as data hash is used or user token
 // a token describes only the data in one block
 extension.append(":");
 extension.append(TokenDefinition::Extra::asSourceNumber(0).toString());
 continue;
 }

 extension.append(":");
 extension.append(extra.toString());
 }

 SipHash hash;
 hash.update(extension.data(), extension.size());
 return DeduplicationHash::createUnifiedHash(hash.get128(), partition_id);
}

The “extra tokens” are the provenance of the block: which source it came from, and which materialized view it passed through. They are what keeps two different blocks with identical content from colliding.

The Extra type enumerates them:

 cpp
 
 
// src/Interpreters/InsertDeduplication.h:208
 struct Extra
 {
 enum Type
 {
 SOURCE_ID,
 SOURCE_NUMBER,
 VIEW_ID,
 VIEW_NUMBER,
 };

 Type type;

 using Range = std::pair<size_t, size_t>;
 std::variant<Range, StorageIDMaybeEmpty> value_variant;

SOURCE_ID is the root table the insert started at, SOURCE_NUMBER is the block’s ordinal within it, VIEW_ID/VIEW_NUMBER identify a materialized view in the chain. When an insert feeds a materialized view, the view’s data is a projection of the source data, and two different source blocks can legitimately produce identical view rows. Without the view id in the token, the second one would be wrongly deduplicated.

The SOURCE_NUMBER special case for async inserts is the flip side: an async insert’s blocks are batched and reordered, so the block ordinal is not stable across retries, and including it would break deduplication entirely. The comment says so.

The final token string, and the block id derived from it:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:69
std::string DeduplicationHash::getBlockId() const
{
 return fmt::format("{}_{}_{}", partition_id, hash.items[0], hash.items[1]);
}

So the block id is partition_<hash_high>_<hash_low>. The partition id is part of the key, which is why the same rows inserted into two different partitions are not deduplicated against each other.

And the ZooKeeper path, which reveals the three generations of this feature that are still on disk:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:75
std::string DeduplicationHash::getPath(const std::string & storage_path) const
{
 std::string hashes_directory = [&] ()
 {
 switch (hash_type)
 {
 case HashType::SYNC:
 return "blocks";
 case HashType::ASYNC:
 return "async_blocks";
 case HashType::UNIFIED:
 return "deduplication_hashes";
 }
 }();
 return (std::filesystem::path(storage_path) / hashes_directory / getBlockId()).string();
}

blocks/ is the old sync-insert directory, async_blocks/ the old async one, and deduplication_hashes/ the current unified one. The cleanup thread still drains the legacy directory during a rolling upgrade:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:76
 // Old replicas (pre-unified-hash) still write the legacy async-insert ids to /async_blocks
 // during a rolling upgrade; the current leader keeps draining that directory with the legacy
 // window settings until the minimum supported version no longer uses it.

The ZooKeeper check

The dedup check for a replicated table is a check-not-exists request folded into the same ZooKeeper multi-op that allocates the block number. It is not a separate round trip:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:71
 /// Check for duplicates in advance, to avoid superfluous block numbers allocation
 Coordination::Requests ops;

 for (const auto & single_dedup_path : deduplication_paths)
 zkutil::addCheckNotExistsRequest(ops, *zookeeper_, single_dedup_path);

 auto deduplication_path_ops_size = ops.size();

 ops.emplace_back(zkutil::makeCreateRequest(path_prefix_, holder_path, zkutil::CreateMode::EphemeralSequential));
 Coordination::Responses responses;
 Coordination::Error e = zookeeper_->tryMulti(ops, responses);
 if (e == Coordination::Error::ZNODEEXISTS)
 {
 auto failed_idx = zkutil::getFailedOpIndex(Coordination::Error::ZNODEEXISTS, responses);

 if (failed_idx < deduplication_path_ops_size)
 {
 const String & failed_op_path = ops[failed_idx]->getPath();
 LOG_DEBUG(
 getLogger("createEphemeralLockInZooKeeper"),
 "Deduplication path already exists: deduplication_path={}",
 failed_op_path);
 return EphemeralLockInZooKeeper{"", nullptr, "", 0, failed_op_path};
 }
 }

The conflict_path in the returned lock is how the sink learns which hash collided — the failed op’s path carries the block id.

This is a genuinely elegant bit of engineering. The dedup check and the block-number allocation are one ZooKeeper transaction, so:

There is no TOCTOU window between “check for duplicate” and “allocate”.
A duplicate is detected before a block number is consumed.
The whole thing costs one round trip, not two.

The comment on the sink side confirms the cost:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:934
 auto block_number_lock = storage.allocateBlockNumber(part->info.getPartitionId(), zookeeper, block_id_pathes, "", block_data); /// 1 RTT

The window

The dedup history is bounded. For replicated tables, the bound is two settings — a count and a time:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1545
 DECLARE(UInt64, replicated_deduplication_window, 10000, R"(
The number of most recently inserted blocks for which ClickHouse Keeper stores
hash sums to check for duplicates.

and

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1566
 DECLARE(UInt64, replicated_deduplication_window_seconds, 60 * 60 /* one hour */, R"(
The number of seconds after which the hash sums of the inserted blocks are
removed from ClickHouse Keeper.

The cleanup thread enforces both bounds — the count and the age:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:383
 /// Use ZooKeeper's first node (last according to time) timestamp as "current" time.
 Int64 current_time = timed_blocks.front().ctime;
 Int64 time_threshold = std::max(
 static_cast<Int64>(0),
 current_time - static_cast<Int64>(1000 * window_seconds));

 /// Virtual node, all nodes that are "greater" than this one will be deleted
 NodeWithStat block_threshold{{}, time_threshold, 0, 0};

 size_t current_deduplication_window = std::min<size_t>(timed_blocks.size(), window_size);
 auto first_outdated_block_fixed_threshold = timed_blocks.begin() + current_deduplication_window;
 auto first_outdated_block_time_threshold = std::upper_bound(
 timed_blocks.begin(), timed_blocks.end(), block_threshold, NodeWithStat::greaterByTime);
 auto first_outdated_block = std::min(first_outdated_block_fixed_threshold, first_outdated_block_time_threshold);

A node is deleted if it is either beyond the count window or older than the time window. The time is measured against the newest node’s ctime, not the wall clock:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:1579
The time is relative to the time of the most recent record, not to the wall
time. If it's the only record it will be stored forever.

That is deliberate. If the server’s clock jumps, or if the table is idle, a wall-clock-based expiry would either delete everything or keep everything. Measuring from the newest record makes the window self-relative.

The deletions are issued asynchronously and in parallel:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp:407
 zkutil::AsyncResponses<Coordination::RemoveResponse> try_remove_futures;
 for (auto it = first_outdated_block; it != timed_blocks.end(); ++it)
 {
 String path = zookeeper_path + "/" + blocks_dir_name + "/" + it->node;
 try_remove_futures.emplace_back(path, zookeeper.asyncTryRemove(path, it->version));
 }

The non-replicated case: a local log

A non-replicated MergeTree has no ZooKeeper, so the dedup history lives in a file on the table’s own disk. The setting is off by default:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:763
 DECLARE(UInt64, non_replicated_deduplication_window, 0, R"(
The number of the most recently inserted blocks in the non-replicated
[MergeTree](/reference/engines/table-engines/mergetree-family/mergetree) table
for which hash sums are stored to check for duplicates.

and the mechanism is described in the setting’s own text:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSettings.cpp:772
A deduplication mechanism is used, similar to replicated tables (see
[replicated_deduplication_window](#replicated_deduplication_window) setting): the deduplication
hash sum covers the whole inserted block. The hash sums are written to
a local file on a disk rather than to ClickHouse Keeper.

The log’s on-disk format is deliberately human-readable — tab-separated records of operation, part name and block id:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:126
/// Fixed-size log for deduplication in non-replicated MergeTree.
/// Stores records on disk for zero-level parts in human-readable format:
/// operation part_name partition_id_check_sum
/// 1 88_18_18_0 88_10619499460461868496_9553701830997749308
/// 2 77_14_14_0 77_15147918179036854170_6725063583757244937
/// 2 77_15_15_0 77_14977227047908934259_8047656067364802772
/// 1 77_20_20_0 77_15147918179036854170_6725063583757244937
/// Also stores them in memory in hash table with limited size.

The in-memory index is a hash map with a FIFO eviction policy, implemented as a list plus an index into it:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:29
template <typename V>
class LimitedOrderedHashMap
{
private:
 struct ListNode
 {
 std::string key;
 V value;
 };
 using Queue = std::list<ListNode>;
 using IndexMap = std::unordered_map<std::string_view, typename Queue::iterator, StringViewHash>;

 Queue queue;
 IndexMap map;
 size_t max_size;

The std::list gives O(1) FIFO eviction from the front; the unordered_map<string_view, iterator> gives O(1) lookup. The map stores string_views into the list nodes’ keys, so no key is copied twice.

The eviction is what makes the window a window:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.h:91
 bool insert(const std::string & key, const V & value)
 {
 auto it = map.find(key);
 if (it != map.end())
 return false;

 if (size() == max_size)
 {
 map.erase(queue.front().key);
 queue.pop_front();
 }

 ListNode elem{key, value};
 auto itr = queue.insert(queue.end(), elem);
 map.emplace(itr->key, itr);
 return true;
 }

When the map is full, the oldest entry is dropped. A retried insert is only deduplicated if the original is still in the window — which is exactly the semantics you want: a retry happens within seconds of the original, not hours later.

The log files themselves rotate, with a rotation interval of twice the window:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:94
 , rotate_interval(deduplication_window_ * 2) /// actually it doesn't matter

and the rotation is driven by the number of entries in the current file:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:255
void MergeTreeDeduplicationLog::rotateAndDropIfNeeded()
{
 /// If we don't have logs at all or already have enough records in current
 /// For the disk that doesn't support writing with append, we can't append logs to the last file.
 if (existing_logs.empty() || existing_logs[current_log_number].entries_count >= rotate_interval || !disk_supports_writing_with_append)
 {
 rotate();
 dropOutdatedLogs();
 }
}

Old logs are dropped by walking from the newest backwards and accumulating entry counts until the window is covered:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeDeduplicationLog.cpp:211
void MergeTreeDeduplicationLog::dropOutdatedLogs()
{
 size_t current_sum = 0;
 size_t remove_from_value = 0;
 /// Go from end to the beginning
 for (auto itr = existing_logs.rbegin(); itr != existing_logs.rend(); ++itr)
 {
 /// Never drop the current active log — it may still be open for writing
 if (itr->first == current_log_number)
 {
 current_sum += itr->second.entries_count;
 continue;
 }

 if (current_sum >= deduplication_window)
 {
 /// We have more logs than required, all older files (excluding current) can be dropped
 remove_from_value = itr->first;
 break;
 }

The comment “actually it doesn’t matter” on the rotation interval is a nice piece of honesty — the interval only affects how many files exist on disk, not the semantics.

The retry loop

When a conflict is detected, the sink does not fail. It filters out the duplicate rows and retries with what is left:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:353
 ProfileEvents::increment(ProfileEvents::DuplicatedInsertedBlocks, conflicts.size());

 auto result = partition.deduplication_info->deduplicateBlock(
 conflicts,
 partition.block_with_partition.partition_id,
 context);

deduplicateBlock finds the offsets of the colliding tokens and filters them out of the block. If everything collided, the part is simply not committed:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:382
 if (result.filtered_block->rows() == 0)
 {
 LOG_DEBUG(
 storage.log,
 "All rows are deduplicated for part with block IDs: {}, skipping the part commit.",
 fmt::join(conflicts, ", "));

and the part log records the deduplication as an execution status, so system.part_log shows what happened:

 cpp
 
 
// src/Storages/MergeTree/MergeTreeSink.cpp:393
 PartLog::addNewPart(
 storage.getContext(),
 PartLog::PartLogEntry(partition.temp_part->part, partition.elapsed_ns, counters_snapshot),
 getDeduplicationBlockIds(deduplication_hashes),
 ExecutionStatus(ErrorCodes::INSERT_WAS_DEDUPLICATED, "The part was deduplicated"));

The filter itself is careful about a boundary case. If a partial collision happens, the colliding rows have to be sliced out of the block — but the block may have been re-anchored by an intermediate view, in which case the offsets no longer describe it:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:303
 chassert(original_block && !original_block->empty() && original_block->rows() > 0);

 auto & block = *original_block;

 /// A partial collision requires slicing the collided tokens' rows out of the block, which is
 /// only possible while the offsets still describe the block. Behind an `Alias` hop over a
 /// row-count-changing view the block is re-anchored to the view-output chunks and there is no
 /// mapping from the tokens' source rows to the block's rows anymore. Refuse loudly instead of
 /// reading out of the block's bounds; full-block deduplication is handled above.
 if (block.rows() != getRows())
 throw Exception(
 ErrorCodes::NOT_IMPLEMENTED,
 "Cannot filter {} of {} deduplicated blocks out of the insert: the deduplication info describes {} rows, "
 "but the block has {} rows because a materialized view with a row-count-changing inner query "
 "was processed before a table with the `Alias` engine. Only the whole insert can be deduplicated on this path. "
 "Debug: {}",

Failing loudly is the right call. Silently reading out of bounds would be a memory-safety bug; silently skipping the dedup would insert duplicates.

Self-deduplication: the same block twice

There is a second, subtler case: the same insert containing the same data twice. A client that retries a batch of inserts with insert_deduplication_token unset, or a materialized view that produces identical rows, can end up with duplicate content inside a single block.

deduplicateSelf handles that, and it operates on the token level:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:217
std::set<size_t> DeduplicationInfo::filterSelf(const String & partition_id) const
{
 if (getCount() <= 1)
 return {};

 std::set<size_t> fitered_offsets;
 /// fitered_offsets will contain all but first offsets for each block id
 /// so that only first occurrence of each block id will remain
 for (const auto & [_, block_offsets] : buildOffsetsMapImpl(partition_id))
 {
 if (block_offsets.size() > 1)
 fitered_offsets.insert(block_offsets.begin() + 1, block_offsets.end());
 }

 return fitered_offsets;
}

Group the tokens by their hash; within each group, keep the first and drop the rest. The grouping is done on the sorted hash values:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:496
std::vector<std::pair<UInt128, std::vector<size_t>>> DeduplicationInfo::buildOffsetsMapImpl(const std::string & partition_id) const
{
 /// (hash, offset) pairs sorted by hash, then offset; runs of equal hashes are the groups.
 std::vector<std::pair<UInt128, size_t>> sorted;
 sorted.reserve(offsets.size());
 for (size_t offset = 0; offset < offsets.size(); ++offset)
 sorted.emplace_back(getBlockUnifiedHash(offset, partition_id).hash, offset);
 std::sort(sorted.begin(), sorted.end());

Note that this recomputes the unified hash for every token — including the extra tokens, so the grouping is by the final token, not the raw content hash. Two tokens with the same content but different provenance are not duplicates.

The user token

A pipeline that writes its own idempotency key can bypass content hashing entirely, by supplying the key as a session setting:

 sql
 
 
INSERT INTO events SETTINGS insert_deduplication_token = 'batch-2026-10-22-0001'
VALUES (1, 'click'), (2, 'view');

The same token on a later insert means “this is a duplicate of the earlier one”; a different token means “this is new data, even if the rows are identical”.

The user can bypass content hashing entirely with the insert_deduplication_token setting:

 cpp
 
 
// src/Core/Settings.cpp:9135
 DECLARE(String, insert_deduplication_token, "", R"(
The setting allows a user to provide own deduplication semantic in MergeTree/ReplicatedMergeTree
For example, by providing a unique value for the setting in each INSERT statement,
user can avoid the same inserted data being deduplicated.

When the token is set, it replaces the content hash in the extension string ("user-token-" + token instead of the data hash). The semantics are therefore inverted: with a token, the same token means duplicate; without one, the same data means duplicate.

The setting’s documentation spells out the partition subtlety:

 cpp
 
 
// src/Core/Settings.cpp:9150
<Note>
`insert_deduplication_token` is tracked per partition, so multiple partitions written by one insert can carry the same token. Without a token, the default content checksum is computed over the whole inserted block, so an insert is deduplicated only when its entire data matches a previous insert (a retry), not when a single partition's rows happen to coincide with a different insert.
</Note>

When is deduplication on?

The whole thing is controlled by deduplicate_insert, with a backwards-compatible split between sync and async:

 cpp
 
 
// src/Core/DeduplicateInsert.cpp:26
bool isDeduplicationEnabledForInsert(bool is_async_insert, const Settings & settings)
{
 switch (settings[Setting::deduplicate_insert].value)
 {
 case DeduplicateInsertMode::BACKWARD_COMPATIBLE_CHOICE:
 return is_async_insert ? settings[Setting::async_insert_deduplicate] : settings[Setting::insert_deduplicate];
 case DeduplicateInsertMode::ENABLE:
 return true;
 case DeduplicateInsertMode::DISABLE:
 return false;
 }
}

For INSERT ... SELECT, there is an extra wrinkle: the SELECT might not be stable, meaning two executions return different data, and content-hash deduplication would then be meaningless. The mode ENABLE_WHEN_POSSIBLE handles it:

 cpp
 
 
// src/Core/DeduplicateInsert.cpp:73
 case DeduplicateInsertSelectMode::ENABLE_WHEN_POSSIBLE:
 {
 if (logger && !select_query_sorted && !has_user_token && isDeduplicationEnabledForInsert(false, settings))
 LOG_INFO(logger, "INSERT SELECT deduplication is disabled because SELECT is not stable");

 return (select_query_sorted || has_user_token) && isDeduplicationEnabledForInsert(false, settings);
 }

A stable SELECT is one with ORDER BY ALL and a single-stream pipeline — or, as the setting’s documentation puts it, a non-empty insert_deduplication_token is “an equivalent substitute for stability”.

Trade-offs

Space. The dedup window costs one ZooKeeper znode per inserted block (or one line in a local log). 10000 blocks is 10000 znodes, and the cleanup thread deletes them as they age out.

CPU. The content hash is a SipHash pass over every column of every inserted block. For a wide table that is not free — but it is one pass over data that is about to be written anyway, and it is only computed when deduplication is enabled and no user token was supplied.

Correctness of the window. Deduplication is only guaranteed within the window. An insert retried after the original has aged out is inserted twice. This is why the window is expressed in both a count and a time: the count covers a burst, the time covers a slow trickle.

The content hash is a heuristic. Two different blocks with the same content hash collide, and the second is dropped. With 128 bits of SipHash that is not a practical concern, but it is why the implementation bothers with UInt128 rather than UInt64.

The aliasing problem is a real limitation. Behind an Alias hop over a row-count-changing view, a partial collision cannot be resolved and the insert fails with NOT_IMPLEMENTED rather than corrupting anything. That is a deliberate trade: correctness over convenience.

The bottom line

Block-level insert deduplication is a content-addressed cache of recent inserts, with the cache keys being UInt128 SipHashes over a column-major hash of the block’s data, salted with the partition id and the block’s provenance so that unrelated identical content does not collide.

The interesting engineering is not the hash — it is the integration. The check-not-exists is folded into the block-number allocation’s ZooKeeper transaction, so the whole thing is one round trip with no TOCTOU window. The hash is computed lazily and cached, so the cost is paid once per block even when it spans several partitions. The representation wrappers are stripped before hashing, so a retry that arrives with a different physical layout still deduplicates. And the window is bounded by both a count and a self-relative age, so a clock jump cannot cause the history to evaporate or to persist forever.

The cost is a hash pass over the inserted data, one znode per block, and a guarantee that only holds inside the window. In exchange, a client that retries an insert after a network hiccup does not double its data — which, for a pipeline that writes to ClickHouse from a queue, is worth a great deal.

Deduplication on Opensource internals