Zookeeper on Opensource internals

ClickHouse Runs Distributed DDL Through a ZooKeeper Task Queue

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: ON CLUSTER does not open N connections and fire N copies of the query. It writes one ZooKeeper node — query-0000000123 — into a shared queue directory, and every host in the cluster watches that directory. Each host that is in the query’s host list runs the DDL and creates <entry>/finished/<host_id> with its exit code. The initiator then just lists <entry>/finished — one getChildren — and reads the status of every host that finished.

The obvious way to implement CREATE TABLE t ON CLUSTER c is to open a connection to each host in the cluster, send it the query, and collect the answers. That is what a lot of databases do, and it has an obvious problem: the initiator is a single point of failure, and it holds N connections open for the duration.

ClickHouse takes the opposite approach. The initiator writes the query into a shared queue in ZooKeeper and then gets out of the way. Every host pulls from the queue itself. This means the initiator can die and the DDL still happens; it means the work is spread over the cluster rather than funnelled through one server; and it means the “which hosts have finished” bookkeeping is just a directory listing.

The mechanism is worth looking at closely, because the interesting parts are all in the details of how ZooKeeper is used.

The queue

The DDL queue is a directory in ZooKeeper. Its path comes from the server setting distributed_ddl_path, which defaults to /clickhouse/task_queue/ddl/:

 cpp
 
 
// src/Core/ServerSettings.cpp:2102
 DECLARE(String, distributed_ddl_path, "/clickhouse/task_queue/ddl/", R"(the path in Keeper for the `<task_queue>` for DDL queries)", 0, "distributed_ddl.path") \
 DECLARE(String, distributed_ddl_replicas_path, "/clickhouse/task_queue/replicas/", R"(the path in Keeper for the `<task_queue>` for replicas)", 0, "distributed_ddl.replicas_path") \

The worker is started once per server, after all the tables are loaded:

 cpp
 
 
// programs/server/Server.cpp:3787
 if (has_zookeeper && config().has("distributed_ddl"))
 {
 /// DDL worker should be started after all tables were loaded
 String ddl_queue_path = server_settings[ServerSetting::distributed_ddl_path];
 String ddl_replicas_path = server_settings[ServerSetting::distributed_ddl_replicas_path];
 int pool_size = server_settings[ServerSetting::distributed_ddl_pool_size];
 if (pool_size < 1)
 throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND, "distributed_ddl.pool_size should be greater then 0");
 global_context->setDDLWorker(
 std::make_unique<DDLWorker>(
 pool_size,
 ddl_queue_path,
 ddl_replicas_path,
 global_context,
 &config(),
 "distributed_ddl",
 "default",
 "DDLWorker",
 &CurrentMetrics::MaxDDLEntryID,
 &CurrentMetrics::MaxPushedDDLEntryID),
 joinTasks(load_system_metadata_tasks, load_metadata_tasks));
 }

distributed_ddl.pool_size defaults to 1, i.e. one ON CLUSTER query at a time per server. The DDL queue is a serial queue.

The entry node

To enqueue a query, the initiator creates a sequential node. ZooKeeper appends a zero-padded counter to the name, and the counter is what makes the queue ordered:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:1198
String DDLWorker::enqueueQueryAttempt(DDLLogEntry & entry)
{
 auto zookeeper = getZooKeeperFromContext();

 String query_path_prefix = fs::path(queue_dir) / "query-";
 zookeeper->createAncestors(query_path_prefix);

 NameSet host_ids;
 for (const HostID & host : entry.hosts)
 host_ids.emplace(host.toString());
 createReplicaDirs(zookeeper, host_ids);

 String node_path = zookeeper->create(query_path_prefix, entry.toString(), zkutil::CreateMode::PersistentSequential);

The node’s payload is the serialized DDLLogEntry — the query text, the list of hosts that should run it, the initiator’s identity, the settings, and a version number.

The versioned entry format

The entry format is explicitly versioned, because a rolling upgrade means old and new servers read the same queue:

 cpp
 
 
// src/Interpreters/DDLTask.h:73
struct DDLLogEntry
{
 static constexpr const UInt64 OLDEST_VERSION = 1;
 static constexpr const UInt64 SETTINGS_IN_ZK_VERSION = 2;
 static constexpr const UInt64 NORMALIZE_CREATE_ON_INITIATOR_VERSION = 3;
 static constexpr const UInt64 OPENTELEMETRY_ENABLED_VERSION = 4;
 static constexpr const UInt64 PRESERVE_INITIAL_QUERY_ID_VERSION = 5;
 static constexpr const UInt64 BACKUP_RESTORE_FLAG_IN_ZK_VERSION = 6;
 static constexpr const UInt64 PARENT_TABLE_UUID_VERSION = 7;
 static constexpr const UInt64 INITIATOR_USER_VERSION = 8;
 /// Add new version here

 /// Remember to update the value below once new version is added
 static constexpr const UInt64 DDL_ENTRY_FORMAT_MAX_VERSION = 8;

 UInt64 version = 1;
 String query;
 std::vector<HostID> hosts;
 String initiator; // optional
 std::optional<SettingsChanges> settings;
 OpenTelemetry::TracingContext tracing_context;
 String initial_query_id;
 bool is_backup_restore = false;
 /// If present, this entry should be executed only if table with this uuid exists.
 /// Only for DatabaseReplicated.
 std::optional<UUID> parent_table_uuid;

 String initiator_user;
 Strings initiator_user_roles;

And the writer only emits the fields the version supports:

 cpp
 
 
// src/Interpreters/DDLTask.cpp:158
String DDLLogEntry::toString() const
{
 WriteBufferFromOwnString wb;

 wb << "version: " << version << "\n";
 wb << "query: " << escape << query << "\n";

 bool write_hosts = version == OLDEST_VERSION || !hosts.empty();
 if (write_hosts)
 {
 Strings host_id_strings(hosts.size());
 std::transform(hosts.begin(), hosts.end(), host_id_strings.begin(), HostID::applyToString);
 wb << "hosts: " << host_id_strings << "\n";
 }

 wb << "initiator: " << initiator << "\n";

 bool write_settings = SETTINGS_IN_ZK_VERSION <= version && settings && !settings->empty();
 if (write_settings)
 {
 ASTSetQuery ast;
 ast.is_standalone = false;
 ast.changes = *settings;
 wb << "settings: " << ast.formatWithSecretsOneLine() << "\n";
 }

The user-facing knob for this is distributed_ddl_entry_format_version (default 5). A cluster running a mix of old and new servers can pin it down so that every server can parse what every other server writes.

The name padding trick

Sequential ZooKeeper node names are query- followed by a 10-digit zero-padded counter. That padding is not cosmetic — it is what makes the string order equal the numeric order:

 cpp
 
 
// src/Common/ZooKeeper/ZooKeeper.cpp:2241
String getSequentialNodeName(const String & prefix, UInt64 number)
{
 /// NOTE Sequential counter in ZooKeeper is Int32.
 chassert(number < std::numeric_limits<Int32>::max());
 constexpr size_t seq_node_digits = 10;
 String num_str = std::to_string(number);
 String name = prefix + String(seq_node_digits - num_str.size(), '0') + num_str;
 return name;
}

Without the padding, query-9 would sort after query-10 as a string, and every range scan over the queue would be wrong. With it, the queue can be scanned with plain std::upper_bound / std::lower_bound on the node names, with no numeric parsing:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:435
 auto begin_node = queue_nodes.begin();
 if (first_failed_task_name)
 {
 /// If we had failed tasks, then we should start from the first failed task.
 chassert(reinitialized, fmt::format("Stale first_failed_task_name={}", *first_failed_task_name));
 begin_node = std::lower_bound(queue_nodes.begin(), queue_nodes.end(), first_failed_task_name);
 }
 else
 {
 /// We had no failed tasks. Let's just choose the maximum entry we have previously seen.
 String last_task_name;
 if (!current_tasks.empty())
 last_task_name = current_tasks.back()->entry_name;
 if (last_skipped_entry_name && last_task_name < *last_skipped_entry_name)
 last_task_name = *last_skipped_entry_name;
 begin_node = std::upper_bound(queue_nodes.begin(), queue_nodes.end(), last_task_name);
 }

std::upper_bound on the string names finds “the entry after the last one I processed”. The whole resumption logic after a ZooKeeper reconnect is built on this.

The padding has a limit, and the codebase is honest about it elsewhere:

 cpp
 
 
// src/Interpreters/TransactionPayloads.cpp:232
 /// Keeper pads to a minimum of ten digits and its counter is `int64_t`, so names grow past
 /// 10^10 and stop being order-comparable as strings. `zkutil::getSequentialNodeName` assumes
 /// Int32 and underflows its padding there.
 const String num = std::to_string(csn);
 return num.size() >= 10 ? "csn-" + num : "csn-" + String(10 - num.size(), '0') + num;

The host list

The entry names the hosts that should execute the query. The initiator computes it from the cluster definition:

 cpp
 
 
// src/Interpreters/executeDDLQueryOnCluster.cpp:141
 /// Enumerate hosts which will be used to send query.
 auto addresses = cluster->filterAddressesByShardOrReplica(params.only_shard_num, params.only_replica_num);
 if (addresses.empty())
 throw Exception(ErrorCodes::LOGICAL_ERROR, "No hosts defined to execute distributed DDL query");

 std::vector<HostID> hosts;
 hosts.reserve(addresses.size());
 for (const auto * address : addresses)
 hosts.emplace_back(*address);

Note that the query is not rewritten per host. The same query text goes into the entry, and each host rewrites it locally (its own cluster definition, its own default database). That is what DDLTask::setClusterInfo and formatRewrittenQuery are for.

Before the entry is created, the initiator does its own access check — but against all the databases the shard might use, because each host will resolve the unqualified table name against its own default database:

 cpp
 
 
// src/Interpreters/executeDDLQueryOnCluster.cpp:153
 AccessRightsElements access_to_check = params.access_to_check;
 bool need_replace_current_database = std::any_of(
 access_to_check.begin(),
 access_to_check.end(),
 [](const AccessRightsElement & elem) { return elem.isEmptyDatabase(); })
 || needsDefaultDatabaseForBareDictionaryOnCluster(query_ptr);

The per-host result directory

This is the trick I like most. Each host that runs the query creates a node named after itself under <entry>/finished/:

 cpp
 
 
// src/Interpreters/DDLTask.h:155
 String getActiveNodePath() const { return fs::path(entry_path) / "active" / host_id_str; }
 String getFinishedNodePath() const { return fs::path(entry_path) / "finished" / host_id_str; }
 String getShardNodePath() const { return fs::path(entry_path) / "shards" / getShardID(); }
 String getSyncedNodePath() const { return fs::path(entry_path) / "synced" / host_id_str; }

Because the finished node’s name is the host ID, the initiator does not need to know in advance which hosts will finish, does not need to poll each one individually, and does not need a counter. It just lists the directory.

The status source does exactly that — two getChildren calls, one for finished and one for active:

 cpp
 
 
// src/Interpreters/DistributedQueryStatusSource.cpp:220
 {
 auto retries_ctl = ZooKeeperRetriesControl("executeDistributedQueryOnCluster", getLogger(getName()), getRetriesInfo());
 retries_ctl.retryLoop(
 [&]()
 {
 auto zookeeper = context->getDefaultOrAuxiliaryZooKeeper(zookeeper_name);
 Strings paths = getNodesToWait();
 auto res = zookeeper->tryGetChildren(paths);
 for (size_t i = 0; i < res.size(); ++i)
 if (res[i].error != Coordination::Error::ZOK && res[i].error != Coordination::Error::ZNONODE)
 throw Coordination::Exception::fromPath(res[i].error, paths[i]);

 if (res[0].error == Coordination::Error::ZNONODE)
 node_exists = zookeeper->exists(node_path);
 else
 node_exists = true;
 tmp_hosts = res[0].names;
 tmp_active_hosts = res[1].names;

 if (only_running_hosts)
 offline_hosts = getOfflineHosts(waiting_hosts, zookeeper);
 });
 }

Two directories, two round trips, and the answer is “these hosts are done, these hosts are currently running”. Compare that with the alternative: N get calls, one per host, each of which can fail independently.

The finished/<host> node’s value is the serialized ExecutionStatus — the exit code and the error message:

 cpp
 
 
// src/Interpreters/DDLOnClusterQueryStatusSource.cpp:39
ExecutionStatus DDLOnClusterQueryStatusSource::checkStatus(const String & host_id)
{
 fs::path status_path = fs::path(node_path) / "finished" / host_id;
 return getExecutionStatus(status_path);
}

and the status is fetched with a tryGet that distinguishes “node absent” from “node present but corrupt”:

 cpp
 
 
// src/Interpreters/DistributedQueryStatusSource.cpp:138
ExecutionStatus DistributedQueryStatusSource::getExecutionStatus(const fs::path & status_path, bool * node_exists)
{
 ExecutionStatus status(-1, "Cannot obtain error message");

 String status_data;
 bool finished_exists = false;

 auto retries_ctl = ZooKeeperRetriesControl("executeDDLQueryOnCluster", getLogger("DDLQueryStatusSource"), getRetriesInfo());
 retries_ctl.retryLoop([&]() { finished_exists = context->getDefaultOrAuxiliaryZooKeeper(zookeeper_name)->tryGet(status_path, status_data); });
 if (node_exists)
 *node_exists = finished_exists;
 /// tryDeserializeText is atomic: a present-but-corrupt payload leaves the (-1) sentinel intact, so a
 /// caller pairing node_exists with the sentinel can tell an absent node from a corrupt present one.
 if (finished_exists)
 status.tryDeserializeText(status_data);

 return status;
}

The `active` node: a liveness signal

The active/<host> node is ephemeral. It exists exactly while the host is running the query, and it disappears when the host’s ZooKeeper session ends. That gives the initiator a cheap way to distinguish “slow” from “dead”:

 cpp
 
 
// src/Interpreters/DistributedQueryStatusSource.cpp:77
NameSet DistributedQueryStatusSource::getOfflineHosts(const NameSet & hosts_to_wait, const ZooKeeperPtr & zookeeper)
{
 Strings paths;
 Strings hosts_array;
 for (const auto & host : hosts_to_wait)
 {
 hosts_array.push_back(host);
 paths.push_back(fs::path(replicas_path) / host / "active");
 }

 NameSet offline;
 auto res = zookeeper->tryGet(paths);
 for (size_t i = 0; i < res.size(); ++i)
 {
 if (res[i].error == Coordination::Error::ZNONODE)
 offline.insert(hosts_array[i]);

 if (res[i].data.ends_with(DatabaseReplicated::REPLICA_UNSYNCED_MARKER))
 {
 LOG_TRACE(log, "Replica {} is not fully synced after recovery, considering it as offline", hosts_array[i]);
 offline.insert(hosts_array[i]);
 }
 }

Note the REPLICA_UNSYNCED_MARKER case: a replica that came back after a crash but has not finished syncing is alive but not usable, and the initiator treats it as offline.

The active node under the replicas directory (/clickhouse/task_queue/replicas/<host>/active) is a different thing — it marks the replica as a live participant in the cluster at all, and it is also ephemeral, with the server’s UUID as the payload so that a stale node from a previous incarnation can be removed safely.

The worker’s side

The worker watches the queue directory. This is a real ZooKeeper watch, not polling — the worker sleeps until a watch fires:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:410
 Strings queue_nodes = zookeeper->getChildrenWatch(
 queue_dir,
 &queue_node_stat,
 Coordination::WatchCallbackPtrOrEventPtr{queue_updated_event, ProfileEvents::ZooKeeperWatchTriggeredDistributedDDL});
 size_t size_before_filtering = queue_nodes.size();
 filterAndSortQueueNodes(queue_nodes);

with the filter dropping anything that is not a queue entry:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:324
static void filterAndSortQueueNodes(Strings & all_nodes)
{
 std::erase_if(all_nodes, [] (const String & s) { return !startsWith(s, "query-"); });
 ::sort(all_nodes.begin(), all_nodes.end());
}

When the worker picks up an entry, it first claims it with an ephemeral active node whose payload is a canary — the server’s UUID:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:693
 String active_node_path = task.getActiveNodePath();
 String finished_node_path = task.getFinishedNodePath();

 /// Step 1: Create ephemeral node in active/ status dir.
 /// It allows other hosts to understand that task is currently executing (useful for system.distributed_ddl_queue)
 /// and protects from concurrent deletion or the task.

 /// It will tryRemove(...) on exception
 auto active_node = zkutil::EphemeralNodeHolder::existing(active_node_path, *zookeeper);

 /// Try fast path
 const String canary_value = Field(ServerUUID::get()).dump();
 auto create_active_res = zookeeper->tryCreate(active_node_path, canary_value, zkutil::CreateMode::Ephemeral);
 if (create_active_res != Coordination::Error::ZOK)
 {
 if (create_active_res != Coordination::Error::ZNONODE && create_active_res != Coordination::Error::ZNODEEXISTS)
 {
 chassert(Coordination::isHardwareError(create_active_res));
 throw Coordination::Exception::fromPath(create_active_res, active_node_path);
 }

The canary exists to make cleanup safe: on a reconnect the worker might find its own previous ephemeral node still there, and it must delete only that one and not somebody else’s. Hence deleteEphemeralNodeIfContentMatches.

Then the DDL is executed, and the result is written in the same ZooKeeper transaction that removes the active node and creates the finished node:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:749
 /// If table and database engine supports it, they will execute task.ops by their own in a single transaction
 /// with other zk operations (such as appending something to ReplicatedMergeTree log, or
 /// updating metadata in Replicated database), so we make create request for finished_node_path with status "0",
 /// which means that query executed successfully.
 task.ops.emplace_back(zkutil::makeRemoveRequest(active_node_path, -1));
 task.ops.emplace_back(zkutil::makeCreateRequest(finished_node_path, ExecutionStatus(0).serializeText(), zkutil::CreateMode::Persistent));

and the multi is issued at the end:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:813
 /// Step 3: Create node in finished/ status dir and write execution status.
 /// FIXME: if server fails right here, the task will be executed twice. We need WAL here.
 /// NOTE: If ZooKeeper connection is lost here, we will try again to write query status.
 /// NOTE: If both table and database are replicated, task is executed in single ZK transaction.

 bool status_written = task.ops.empty();
 if (!status_written)
 {
 zookeeper->multi(task.ops);
 task.ops.clear();
 }

The FIXME is honest: if the server dies between “execute the DDL” and “write the finished node”, the DDL will run again on the next start. The code acknowledges the gap rather than pretending it does not exist.

Exactly-once, sort of

For a replicated database, the DDL must run on one replica of each shard, not on all of them — running ALTER TABLE ... ADD COLUMN three times is not harmless. The worker handles this with a ZooKeeper lock per shard plus an executed marker node:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:879
 String shard_path = task.getShardNodePath();
 String is_executed_path = fs::path(shard_path) / "executed";
 String tries_to_execute_path = fs::path(shard_path) / "tries_to_execute";
 String max_tries_exceeded_path = fs::path(shard_path) / "max_tries_exceeded";
 chassert(shard_path.starts_with(String(fs::path(task.entry_path) / "shards" / "")));
 zookeeper->createIfNotExists(fs::path(task.entry_path) / "shards", "");
 zookeeper->createIfNotExists(shard_path, "");

 /// Leader replica creates is_executed_path node on successful query execution.
 /// We will remove create_shard_flag from zk operations list, if current replica is just waiting for leader to execute the query.
 auto create_shard_flag = zkutil::makeCreateRequest(is_executed_path, task.host_id_str, zkutil::CreateMode::Persistent);

Every replica of the shard races for the lock; whoever gets it executes the query and creates shards/<shard>/executed. The others see the node and skip. There is a retry counter (MAX_TRIES_TO_EXECUTE = 3) so that a leader that dies mid-execution does not permanently block the shard, and a one-hour ceiling (MAX_EXECUTION_TIMEOUT_SEC = 3600).

There is a subtlety in the lock acquisition that is worth reading, because “in replicated merge tree we can have multiple leaders” is a real thing:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:942
 if (execute_on_single_replica_lock->tryLock())
 {
 /// In replicated merge tree we can have multiple leaders. So we can
 /// be "leader" and took lock, but another "leader" replica may have
 /// already executed this task.
 if (zookeeper->tryGet(is_executed_path, executed_by))
 {
 LOG_DEBUG(log, "Task {} has already been executed by replica ({}) of the same shard.", task.entry_name, executed_by);
 executed_by_other_leader = true;
 if (auto op = task.getOpToUpdateLogPointer())
 task.ops.push_back(op);
 break;
 }

Taking the lock is not enough — you must also check the executed marker, because a previous leader may have run the query and died before releasing the lock.

Collecting the result

The initiator polls with a linear backoff, capped at one second:

 cpp
 
 
// src/Interpreters/DistributedQueryStatusSource.cpp:213
 sleepForMilliseconds(std::min<size_t>(1000, 50 * try_number));

so the first check is after 50 ms, then 100 ms, 150 ms, … up to 1 s. The try_number counter increments per iteration, so a fast DDL is noticed quickly and a slow one is not polled aggressively.

The timeout is distributed_ddl_task_timeout, default 180 seconds:

 cpp
 
 
// src/Core/Settings.cpp:3237
 DECLARE(Int64, distributed_ddl_task_timeout, 180, R"(
Sets timeout for DDL query responses from all hosts in cluster. If a DDL request has not been performed on all hosts, a response will contain a timeout error and a request will be executed in an async mode. Negative value means infinite.

Possible values:

- Positive integer.
- 0 — Async mode.
- Negative integer — infinite timeout.
)", 0) \

When the timeout hits, the behaviour depends on distributed_ddl_output_mode (default throw):

 cpp
 
 
// src/Interpreters/DDLOnClusterQueryStatusSource.cpp:79
 constexpr auto msg_format = "Distributed DDL task {} is not finished on {} of {} hosts "
 "({} of them are currently executing the task, {} are inactive). "
 "They are going to execute the query in background. Was waiting for {:.3f} seconds{}";

Note the wording: “They are going to execute the query in background.” A timeout does not cancel the DDL. The workers keep going; the initiator just stops waiting. That is a deliberate design choice — the DDL is a durable queue entry, and abandoning it would leave the cluster in an inconsistent state.

Retries, and the “zoo” comment

Every ZooKeeper interaction in the DDL path is wrapped in a retry controller with exponential backoff. The defaults are 5 retries, 100 ms initial backoff, 5000 ms maximum:

 cpp
 
 
// src/Interpreters/DistributedQueryStatusSource.cpp:157
ZooKeeperRetriesInfo DistributedQueryStatusSource::getRetriesInfo() const
{
 const auto & config_ref = Context::getGlobalContextInstance()->getConfigRef();
 return ZooKeeperRetriesInfo(
 config_ref.getInt("distributed_ddl_keeper_max_retries", 5),
 config_ref.getInt("distributed_ddl_keeper_initial_backoff_ms", 100),
 config_ref.getInt("distributed_ddl_keeper_max_backoff_ms", 5000),
 context->getProcessListElement());
}

And the retry loop only retries hardware errors:

 cpp
 
 
// src/Common/ZooKeeper/ZooKeeperRetries.h:81
 catch (const zkutil::KeeperException & e)
 {
 iteration_cleanup();

 if (!Coordination::isHardwareError(e.code))
 throw;

 setKeeperError(std::current_exception(), e.code, e.message());
 }

The comment in scheduleTasks about reconnection is the best summary of what distributed DDL actually is:

 cpp
 
 
// src/Interpreters/DDLWorker.cpp:365
 else if (task->was_executed)
 {
 /// Connection was lost on attempt to write status. Will retry.
 bool status_written = zookeeper->exists(task->getFinishedNodePath());
 /// You might think that the following condition is redundant, because status_written implies completely_processed.
 /// But it's wrong. It's possible that (!task->completely_processed && status_written)
 /// if ZooKeeper successfully received and processed our request
 /// but we lost connection while waiting for the response.
 /// Yeah, distributed systems is a zoo.
 if (status_written)
 {
 /// TODO We cannot guarantee that query was actually executed synchronously if connection was lost.
 /// Let's simple create synced/ node for now, but it would be better to pass UNFINISHED status to initiator
 /// or wait for query to actually finish (requires https://github.com/ClickHouse/ClickHouse/issues/23513)
 task->createSyncedNodeIfNeed(zookeeper);
 task->completely_processed = true;
 }

Trade-offs

What the design buys:

The initiator is not in the data path. If it dies, the DDL still runs.
One round trip per poll, not N. Listing finished/ and active/ is two getChildren calls regardless of cluster size.
No central broker. ZooKeeper is the queue; there is nothing else to deploy.
The result set is derived, not transported. Each host publishes its own exit code; the initiator assembles the report from the filesystem metaphor.
Resumable. After a reconnect, the worker rescans the queue using the padded names and picks up where it left off.

What it costs:

At-least-once, not exactly-once. The FIXME in processTask is explicit: a crash between execution and status-writing re-runs the DDL. For replicated databases the shards/<n>/executed marker and the retry counter mitigate this, but the mitigation is best-effort.
The queue is serial. distributed_ddl.pool_size defaults to 1, so a slow ALTER blocks every subsequent ON CLUSTER query on that host.
Polling, on the initiator side. The worker uses a watch; the initiator uses a poll with a 1-second cap. For a 100-host cluster finishing in 10 ms, the initiator still waits ~50 ms.
The host list is in the entry. Every host reads the whole entry to find out whether it is in the host list, so a 1000-host cluster writes a 1000-element list into every entry.
The finished directory grows. Every entry keeps its per-host result nodes until the queue cleaner removes the entry.

Wrapping up

The one thing to remember: the result of a distributed DDL is a directory listing, not a set of RPC replies. Each host writes a node named after itself; the initiator reads the directory. That single inversion is what makes the initiator stateless with respect to the workers.

The part people get wrong: a distributed_ddl_task_timeout does not cancel anything. It only stops the initiator from waiting. The DDL is in a durable queue and will be executed whether or not anybody is watching.

What to go do about it: if ON CLUSTER feels slow, look at system.distributed_ddl_queue. It shows the queue’s entries and their state, and the gap between “created” and “finished” tells you whether you are waiting on the queue depth (pool size 1) or on a genuinely slow host.

ClickHouse Keeper Carries a Tree Digest in Every Raft Log Entry

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: Raft replicates a log, and assumes every replica applies that log to the same state. That assumption is load-bearing and almost never true in practice — a different library version, a compiler difference, or one non-deterministic line of code and the replicas silently diverge. clickhouse-keeper closes the gap by putting a digest of its entire node tree inside the log entry itself. Every replica recomputes the digest after applying the entry and compares it with the leader’s. Mismatch is fatal, and the process aborts rather than serving divergent data.

ClickHouse ships its own ZooKeeper-compatible server, clickhouse-keeper. It speaks the ZooKeeper wire protocol, so the ordinary ClickHouse ZooKeeper client talks to it unchanged — but the consensus underneath is NuRaft, a Raft implementation in C++, not the original Java one.

That choice brings with it Raft’s central promise and Raft’s central hole.

The hole in Raft

Raft’s correctness argument goes like this:

The leader appends a log entry.
The leader replicates that entry to a quorum of followers.
The leader commits it once a quorum has it.
Every replica applies committed entries in order.

Every replica therefore has the same log. From that, Raft concludes every replica has the same state. That conclusion is only valid if applying a log entry is a deterministic function of (state, entry). In practice:

A std::unordered_map iterated in a different order on a different binary.
A floating-point operation compiled differently with different optimisation flags.
A time() call, a random number, or a locale-dependent comparison.
A different version of a library on one node.

None of these break Raft. They break the state machine. The log is still identical everywhere; the trees are not. And ZooKeeper’s contract is extremely tight — two clients connected to two different replicas must see the same tree, or the whole thing is broken in ways that are very hard to debug.

So ClickHouse added a check. And the check is the interesting part.

The idea: put the expected state in the log

The naive way to detect divergence is to compare the replicas’ trees periodically — but that is O(tree size) and requires coordination. The cleverer way is to make the leader publish its expectation as part of the data it is already replicating.

That is exactly what Keeper does. When the leader is about to append an entry to the log, it:

Applies the request to its own in-memory tree (the “preprocess” step).
Computes the digest of the resulting tree.
Writes (zxid, digest_version, digest_value) into the log entry’s header.

 cpp
 
 
// src/Coordination/KeeperServer.cpp:1289
 case nuraft::cb_func::PreAppendLogLeader:
 {
 // we are relying on the fact that request are being processed under a mutex
 // and not a RW lock
 auto & entry = *static_cast<LogEntryPtr *>(param->ctx);

 chassert(entry->get_val_type() == nuraft::app_log);
 auto next_zxid = state_machine->getNextZxid();

 auto entry_buf = entry->get_buf_ptr();

 KeeperStateMachine::ZooKeeperLogSerializationVersion serialization_version = {};
 size_t request_end_position = 0;
 auto request_for_session = state_machine->parseRequest(*entry_buf, /*final=*/false, &serialization_version, &request_end_position);
 request_for_session->zxid = next_zxid;
 auto digest_after_preprocessing = state_machine->preprocess(*request_for_session, /*lock_mutex=*/ true);
 if (!digest_after_preprocessing)
 return nuraft::cb_func::ReturnCode::ReturnNull;

 request_for_session->digest = *digest_after_preprocessing;

and then, at the end of the entry buffer:

 cpp
 
 
// src/Coordination/KeeperServer.cpp:1336
 WriteBufferFromPointer write_buf(buffer_start, write_buffer_header_size);

 if (serialization_version < KeeperStateMachine::ZooKeeperLogSerializationVersion::WITH_TIME)
 writeIntBinary(request_for_session->time, write_buf);

 writeIntBinary(request_for_session->zxid, write_buf);
 writeIntBinary(static_cast<uint8_t>(request_for_session->digest->version), write_buf);
 if (request_for_session->digest->version != KeeperDigestVersion::NO_DIGEST)
 writeIntBinary(request_for_session->digest->value, write_buf);

 write_buf.finalize();

The digest is now part of the replicated log. Every follower receives it for free — no extra round trip, no extra message type, no coordinator.

Note the version byte. The digest algorithm has changed over the years (new node fields were added to it), so the version travels with the value. A follower that does not understand the version knows it cannot compare, and the check degrades to “skip”.

 cpp
 
 
// src/Coordination/KeeperCommon.h:38
enum class KeeperDigestVersion : uint8_t
{
 NO_DIGEST = 0,
 V1 = 1,
 V2 = 2, // added system nodes that modify the digest on startup so digest from V0 is invalid
 V3 = 3, // fixed bug with casting, removed duplicate czxid usage
 V4 = 4, // 0 is not a valid digest value
 V5 = 5 // added TTL fields (destroy_time and ttl) to the node digest
};

The comment on V3 — “fixed bug with casting” — is a nice reminder that this is a hash of a hash input, and any change to the input invalidates every previously computed digest.

What the digest actually is

The digest is not a Merkle tree. It is much simpler: a SipHash-64 over each node’s content, summed over the whole tree.

 cpp
 
 
// src/Coordination/KeeperCommon.cpp:153
/// When this function is updated, update KEEPER_CURRENT_DIGEST_VERSION!!
uint64_t KeeperNodeStats::calculateDigest(std::string_view path, std::string_view data) const
{
 /// Must match calculateDigest in KeeperStorage.cpp (KEEPER_CURRENT_DIGEST_VERSION).
 SipHash hash;

 hash.update(path);
 if (!data.empty())
 hash.update(data);

 hash.update(czxid);
 hash.update(mzxid);
 hash.update(getCTime());
 hash.update(mtime);
 hash.update(version);
 hash.update(cversion);
 hash.update(aversion);
 hash.update(getEphemeralOwner()); // covers EPHEMERAL and CONTAINER flags
 hash.update(getNumChildren());
 hash.update(pzxid);

 hash.update(isTTL());
 if (isTTL())
 hash.update(getTTL());

 /// TODO: Hash seq num (or replace getEphemeralOwner(), getCTime(), getTTL() above with plain ephemeral_or_seq_num_or_ttl and ctime_and_flags).

 uint64_t digest = hash.get64();

 /// 0 means no calculated digest, it's not a valid digest value.
 if (digest == 0)
 digest = 1;

 return digest;
}

Every field that distinguishes one node from another is in the hash: the path, the data, the creation and modification zxids, the version counters, the ephemeral owner, the child count, the parent zxid, and the TTL.

The TODO is honest — the sequential number is not hashed, and the comment explains why that is a known hole: getEphemeralOwner(), getCTime() and getTTL() are overloaded fields, so the same integer can mean different things for different node kinds.

And the “0 is not a valid digest value” guard is the kind of detail that saves you at 3am: 0 is used as the sentinel for “no digest”, so the real digest must never be zero.

Why a sum?

Because the digest has to be maintained incrementally. Every node’s hash can be cached, and the tree-wide digest is the sum of the cached hashes. A create adds one node’s hash; a delete subtracts it:

 cpp
 
 
// src/Coordination/KeeperMemNodesStorage.cpp:807
void KeeperMemNodesStorage::updateNodesDigest(uint64_t & current_digest, uint64_t for_zxid) const
{
 if (!keeper_context->digestEnabled())
 return;

 auto nodes_it = uncommitted_zxid_to_nodes.find(for_zxid);
 if (nodes_it == uncommitted_zxid_to_nodes.end())
 return;

 for (const auto node_it : nodes_it->second)
 {
 const auto & [path, uncommitted_node] = *node_it;
 if (uncommitted_node.node)
 {
 uncommitted_node.node->invalidateDigestCache();
 current_digest += uncommitted_node.node->getDigest(path);
 }
 }
}

and on commit:

 cpp
 
 
// src/Coordination/KeeperStorage.cpp:655
 if (!delta.path.empty())
 nodes_storage->commitDelta(delta, digest);

 if (result != Coordination::Error::ZOK)
 {
 /// Only ErrorDelta can produce error on commit.
 /// ErrorDelta should always be emitted alone (by `preprocess` on rejected request).
 /// If there could be any Deltas after an ErrorDelta, we'd have to do either commit
 /// or rollback on them here.
 chassert(std::distance(deltas.begin(), deltas.end()) == 1);

 return result;
 }
 }

 nodes_digest += digest_change;

The sum is commutative, which is exactly what you want: the order in which nodes are added does not matter, only the set. A Merkle tree would have given a stronger guarantee (it would localise which node diverged), but the sum is O(changed nodes) instead of O(tree).

The check

The check happens twice per entry: once after the preprocess (the speculative apply) and once after the commit (the real apply).

 cpp
 
 
// src/Coordination/KeeperStateMachine.cpp:1093
 if (keeper_context->digestEnabled() && request_for_session.digest)
 assertDigest(
 *request_for_session.digest,
 digest_after_preprocessing,
 *request_for_session.request,
 request_for_session.log_idx,
 request_for_session.session_id,
 false);

and:

 cpp
 
 
// src/Coordination/KeeperStateMachine.cpp:1260
 if (keeper_context->digestEnabled() && request_for_session->digest)
 assertDigest(
 *request_for_session->digest,
 storage->getNodesDigest(true, /*lock_transaction_mutex=*/true),
 *request_for_session->request,
 request_for_session->log_idx,
 request_for_session->session_id,
 true);

The preprocess check catches a divergence before the entry is committed — the follower refuses to advance at all. The commit check catches a divergence introduced by the commit path itself.

And the failure mode is deliberately violent:

 cpp
 
 
// src/Coordination/KeeperStateMachine.cpp:764
void assertDigest(
 const KeeperDigest & expected,
 const KeeperDigest & actual,
 const Coordination::ZooKeeperRequest & request,
 uint64_t log_idx,
 uint64_t session_id,
 bool committing)
{
 if (!KeeperStorage::checkDigest(expected, actual))
 {
 LOG_FATAL(
 getLogger("KeeperStateMachine"),
 "Digest for nodes is not matching after {} request of type '{}' at log index {} for session {}.\nExpected digest - {}, actual digest - {} "
 "(digest {}). Keeper will terminate to avoid inconsistencies.\nExtra information about the request:\n{}",
 committing ? "committing" : "preprocessing",
 request.getOpNum(),
 log_idx,
 session_id,
 expected.value,
 actual.value,
 expected.version,
 request.toString());
 std::terminate();
 }
}

std::terminate(), not an exception, not a degraded mode. The reasoning is that a Keeper that has diverged is dangerous: it will happily answer get requests with a tree that is different from its peers’, and the resulting client-side confusion is far worse than an outage. So the process dies, loudly, with the offending request printed, and a human comes and looks at it.

The digest is off by default in the server config:

 cpp
 
 
// src/Coordination/KeeperContext.cpp:111
 digest_enabled = config.getBool("keeper_server.digest_enabled", false);
 digest_enabled_on_commit = config.getBool("keeper_server.digest_enabled_on_commit", false);

digest_enabled_on_commit is a separate, more expensive mode: it recomputes the digest from scratch on every commit rather than maintaining it incrementally, catching bugs in the incremental maintenance itself.

Snapshots

The digest is the divergence detector; snapshots are how a replica catches up without replaying the entire log.

A Keeper snapshot is a serialization of the entire tree — nodes, data, ACLs, sessions — plus the metadata needed to resume:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.cpp:243
void KeeperStorageSnapshot::serialize(const KeeperStorageSnapshot & snapshot, WriteBuffer & out, KeeperContextPtr keeper_context)
{
 writeBinary(static_cast<uint8_t>(snapshot.version), out);
 serializeSnapshotMetadata(snapshot.snapshot_meta, out);

 if (snapshot.version >= SnapshotVersion::V5)
 {
 writeBinary(snapshot.zxid, out);
 if (keeper_context->digestEnabled())
 {
 writeBinary(static_cast<uint8_t>(KEEPER_CURRENT_DIGEST_VERSION), out);
 writeBinary(snapshot.nodes_digest, out);
 }
 else
 writeBinary(static_cast<uint8_t>(KeeperDigestVersion::NO_DIGEST), out);
 }

The snapshot carries the digest too (since V5), so a replica that loads a snapshot can check that the snapshot’s tree matches what the sender claimed.

Determinism

Snapshots must be byte-identical across replicas, or two replicas that snapshot at the same log index would produce different files and the transfer machinery would have no way to compare them. Two comments in the serializer say so explicitly:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.cpp:262
 /// Better to sort before serialization, otherwise snapshots can be different on different replicas
 std::vector<std::pair<ACLId, Coordination::ACLs>> sorted_acl_map(snapshot.acl_map.begin(), snapshot.acl_map.end());
 ::sort(sorted_acl_map.begin(), sorted_acl_map.end());

and, for sessions:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.cpp:306
 /// Session must be saved in a sorted order,
 /// otherwise snapshots will be different
 std::vector<std::pair<int64_t, int64_t>> sorted_session_and_timeout(
 snapshot.session_and_timeout.begin(), snapshot.session_and_timeout.end());
 ::sort(sorted_session_and_timeout.begin(), sorted_session_and_timeout.end());

The node tree itself is written in path order (the storage iterates it sorted), so that part is deterministic for free.

The format is versioned, with a version per feature:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.h:34
enum SnapshotVersion : uint8_t
{
 V0 = 0,
 V1 = 1, /// with ACL map
 V2 = 2, /// with 64 bit buffer header
 V3 = 3, /// compress snapshots with ZSTD codec
 V4 = 4, /// add Node size to snapshots
 V5 = 5, /// add ZXID and digest to snapshots
 V6 = 6, /// remove is_sequential, per node size, data length
 V7 = 7, /// acl_id narrowed from uint64_t to uint32_t, seq_num widened from int32_t to int64_t
 V8 = 8, /// add destroy_time and ttl for TTL nodes
 V9 = 9, /// container nodes (ephemeralOwner = INT64_MIN sentinel)
};

static constexpr auto MAX_SUPPORTED_SNAPSHOT_VERSION = SnapshotVersion::V9;

A newer Keeper writes V9; an older one that cannot understand V9 will refuse rather than misparse:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.cpp:1281
 uint8_t version = 0;
 readBinary(version, *in);
 if (version > static_cast<uint8_t>(MAX_SUPPORTED_SNAPSHOT_VERSION))
 throw Exception(ErrorCodes::UNKNOWN_FORMAT_VERSION, "Unsupported snapshot version {}", version);

Compression

Snapshots are compressed. Two formats are supported — the legacy custom LZ4 frame and Zstandard — and the reader detects which one it has by sniffing the magic number:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.cpp:759
bool KeeperSnapshotManager::isZstdCompressed(nuraft::ptr<nuraft::buffer> buffer)
{
 static constexpr unsigned char ZSTD_COMPRESSED_MAGIC[4] = {0x28, 0xB5, 0x2F, 0xFD};

 ReadBufferFromNuraftBuffer reader(buffer);
 unsigned char magic_from_buffer[4]{};
 reader.readStrict(reinterpret_cast<char *>(&magic_from_buffer), sizeof(magic_from_buffer));
 buffer->pos(0);
 return memcmp(magic_from_buffer, ZSTD_COMPRESSED_MAGIC, 4) == 0;
}

0x28 0xB5 0x2F 0xFD is the Zstandard frame magic. This matters because the setting compress_snapshots_with_zstd_format (default true) can be flipped on a live cluster, and old snapshots must still be readable.

On-disk layout

Snapshot files are named after the log index they cover, plus a random component so that two concurrent writes of the same index cannot collide:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.cpp:70
 std::string getSnapshotFileName(uint64_t up_to_log_idx, bool compress_zstd, uint64_t recovery_generation = 0)
 {
 /// Unique-from-birth name avoids collisions between concurrent same-index writes.
 auto base = recovery_generation
 ? fmt::format("snapshot_{}_recovered_{}_{:016x}.bin", up_to_log_idx, recovery_generation, thread_local_rng())
 : fmt::format("snapshot_{}_{:016x}.bin", up_to_log_idx, thread_local_rng());
 if (compress_zstd)
 base += ".zstd";
 return base;
 }

A partially-written snapshot is detected by a tmp_ marker file. Both the marker and the real file are removed if the marker is found on startup:

 cpp
 
 
// src/Coordination/KeeperSnapshotManager.cpp:435
 std::vector<std::string> snapshot_files;
 for (auto it = disk->iterateDirectory(""); it->isValid(); it->next())
 {
 if (it->name().starts_with(tmp_keeper_file_prefix))
 {
 incomplete_files.emplace(it->name().substr(tmp_keeper_file_prefix.size()), it->path());
 continue;
 }

 if (it->name().starts_with("snapshot_") && !clean_incomplete_file(it->path()))
 snapshot_files.push_back(it->path());
 }

The number of snapshots kept on disk is snapshots_to_keep, default 3.

Snapshot identity

Raft’s snapshot installation is a delicate operation: the follower must replace its entire state with the leader’s, and the leader must know which snapshot it is installing. The identity of a snapshot is (last_log_idx, last_log_term):

 cpp
 
 
// src/Coordination/KeeperStateMachine.cpp:101
/// Snapshot identity is (last_log_idx, last_log_term); Raft log matching guarantees one per committed prefix.
static bool sameSnapshotIdentity(const nuraft::snapshot & lhs, const nuraft::snapshot & rhs)

The term matters: two snapshots at the same log index but different terms are different snapshots (the leader may have overwritten its own log entry). NuRaft’s own snapshot object only holds Raft metadata, so ClickHouse clones it to keep the richer metadata separately:

 cpp
 
 
// src/Coordination/KeeperStateMachine.cpp:94
/// nuraft::snapshot holds only Raft metadata (last_log_idx, last_log_term, size, cluster_config).
static nuraft::ptr<nuraft::snapshot> cloneSnapshotMeta(nuraft::snapshot & s)

and the apply_snapshot path fails closed if the snapshot it is asked to apply is not the one it was told to expect:

 cpp
 
 
// src/Coordination/KeeperStateMachine.cpp:1334
 if (s.get_last_log_idx() < pending_snapshot_to_apply->get_last_log_idx())
 {
 /// A newer install's save displaced this pending context (NuRaft drops its lock
 /// between save and apply). Skip-as-success is sound only when local commits
 /// already cover the snapshot; otherwise fail closed — NuRaft exits the process
 /// and `init` recovers from the displacing install, which is fully saved on disk.
 const auto last_committed = keeper_context->lastCommittedIndex();
 if (last_committed >= s.get_last_log_idx())

The log itself

Underneath all of this is the changelog: Keeper’s own Raft log storage, separate from the tree.

 cpp
 
 
// src/Coordination/KeeperLogStore.cpp:60
uint64_t KeeperLogStore::append(nuraft::ptr<nuraft::log_entry> & entry)
{
 ProfiledExclusiveLock lock(changelog_lock, ProfileEvents::KeeperChangelogLockWaitMicroseconds);
 uint64_t idx = changelog.getNextEntryIndex();
 changelog.appendEntry(idx, entry);
 return idx;
}

The changelog is compressed and rotated:

 cpp
 
 
// src/Coordination/Changelog.h:193
struct LogFileSettings
{
 bool force_sync = true;
 bool compress_logs = true;
 uint64_t rotate_interval = 100000;
 uint64_t max_size = 0;
 uint64_t overallocate_size = 0;
 uint64_t latest_logs_cache_size_threshold = 0;
 uint64_t latest_logs_cache_entry_count_threshold = 0;
 /// 0 = automatically use the number of CPU cores (resolved by Changelog's constructor).
 uint64_t startup_read_max_streams = 0;
 uint64_t startup_read_buffer_size = 8 * 1024 * 1024;
};

rotate_interval = 100000 entries per file, force_sync = true so an acknowledged write survives power loss, and up to 8 MiB of read buffer per startup read stream.

The log is compacted once a snapshot covers it — compact(last_log_index) drops everything below the snapshot point — but a configurable number of entries is kept:

 cpp
 
 
// src/Coordination/CoordinationSettings.cpp:41
 DECLARE(UInt64, reserved_log_items, 100000, "How many log items to store (don't remove during compaction)", 0) \

Trade-offs

What the design buys:

Divergence is detected, not discovered later. The check runs on every entry, so a divergence is caught at the moment it happens, with the offending request printed.
Zero extra round trips. The digest rides along in the log entry that was being replicated anyway.
Deterministic snapshots. Sorted ACLs and sessions mean two replicas that snapshot at the same index produce byte-identical files.
Fail-stop rather than fail-silent. A diverged Keeper kills itself.

What it costs:

The digest is off by default. keeper_server.digest_enabled defaults to false. You have to opt in.
A sum, not a Merkle tree. The digest tells you that the trees differ, not where. Debugging a divergence means bisecting the log.
The digest is not cryptographically strong. SipHash-64 is a non-cryptographic hash; it is fast and good against accidental divergence, which is the threat model, but it is not a defence against a malicious replica.
The sequential number is not hashed. The TODO in calculateDigest documents this: a node’s seq_num is excluded, so two trees that differ only in sequential numbering would hash the same.
Terminating is drastic. A false positive (a bug in the digest maintenance, or a version mismatch during a rolling upgrade) kills a production node. That is why the version byte and the “no digest” value exist, and why the digest is opt-in.
Snapshots are O(tree). A million-node tree means a million-node snapshot, written to disk and potentially shipped over the network to a lagging follower.

Wrapping up

The one thing to remember: Keeper’s divergence check is a digest embedded in the replicated log. Raft replicates the log; by putting the expected state inside the log, the expected state gets replicated too, for free.

The part people get wrong: the digest is not on by default. If you run a multi-node Keeper and want to know that your replicas agree, you have to set keeper_server.digest_enabled = true yourself. And when you do, a mismatch kills the process — that is the design, not a bug.

What to go do about it: turn on keeper_server.digest_enabled on a staging Keeper cluster and watch KeeperCommits in system.events. If the cluster is healthy you will see commits proceed; if a node’s binary differs from its peers’, the first divergent entry will abort it, and the LOG_FATAL message will name the request, the log index, and both digests.

ClickHouse Coordinates a Cluster-Wide Backup with ZooKeeper

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: BACKUP ON CLUSTER is not a fan-out of independent backups — it is one logical backup with a shared manifest. Each host publishes a file info record in ZooKeeper, every host reads all of them, sorts them by name, and assigns each file a global data file index. Who writes which file is decided by a create on /writing_files/<index> — a ZooKeeper node creation is the mutex. Between phases, every host parks at a named stage and the group does not move on until every host has arrived.

Backing up a cluster looks like a solved thing until you try to solve it. The naive implementation is: send BACKUP to every host, let each one dump its own local data, call it a day. That works until you want the result to be a backup rather than N backups — one manifest, one file namespace, restorable as a unit.

The trouble is that a MergeTree cluster is not N independent databases. A replicated table has the same logical data on every replica; a Replicated database replicates its DDL history; system.users on one host is not the same as on another. If each host writes its own files, you get N copies of everything and no way to tell which copy is the real one.

ClickHouse’s answer is to make the backup global. There is exactly one manifest, exactly one file namespace, and a rule that decides who writes what. That rule is implemented with ZooKeeper primitives, and it is much cleverer than it first looks.

The shape of the thing

First, the trick that makes the whole thing possible: BACKUP ON CLUSTER reuses the distributed DDL machinery to ship the query. There is no separate “backup cluster protocol”.

 cpp
 
 
// src/Backups/BackupsWorker.cpp:1255
void BackupsWorker::sendQueryToOtherHosts(const ASTBackupQuery & backup_or_restore_query, const ClusterPtr & cluster,
 size_t only_shard_num, size_t only_replica_num, ContextMutablePtr context, const AccessRightsElements & access_to_check,
 const ZooKeeperRetriesInfo & retries_info) const
{
 chassert(cluster);

 DDLQueryOnClusterParams params;
 params.cluster = cluster;
 params.only_shard_num = only_shard_num;
 params.only_replica_num = only_replica_num;
 params.access_to_check = access_to_check;
 params.retries_info = retries_info;

 context->setSetting("distributed_ddl_task_timeout", Field{0});
 context->setSetting("distributed_ddl_output_mode", Field{"never_throw"});

 // executeDDLQueryOnCluster() will return without waiting for completion
 executeDDLQueryOnCluster(backup_or_restore_query.clone(), context, params);

The initiator does not run the backup itself. It ships the query, sets distributed_ddl_task_timeout to zero so it does not wait, and then watches ZooKeeper for the result of the internal per-host operations. That is why the backup coordinator has two roles: the initiator has an empty host name (kInitiator), every other host has its own host_id.

 cpp
 
 
// src/Backups/BackupsWorker.cpp:1286
 bool is_internal_backup = backup_settings.internal;

 String root_zk_path = context->getConfigRef().getString("backups.zookeeper_path", "/clickhouse/backups");
 auto get_zookeeper = [global_context = context->getGlobalContext()] { return global_context->getZooKeeper(); };
 auto keeper_settings = BackupKeeperSettings(context);

 auto all_hosts = BackupSettings::Util::filterHostIDs(
 backup_settings.cluster_host_ids, backup_settings.shard_num, backup_settings.replica_num);
 all_hosts.emplace_back(BackupCoordinationOnCluster::kInitiator);

 String current_host = is_internal_backup ? backup_settings.host_id : String{BackupCoordinationOnCluster::kInitiator};

Note all_hosts.emplace_back(BackupCoordinationOnCluster::kInitiator): the initiator is in the list as a pseudo-host, even though it does no work. That matters for the barrier — other hosts wait on the initiator’s stage node too.

The ZooKeeper layout

Everything for one backup lives under one subtree. The paths are constructed once, in the constructor, and every node name is derived from them:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:126
 , zookeeper_path(zookeeper_path_)
 , root_zookeeper_path(zookeeper_path.parent_path().parent_path())
 , operation_zookeeper_path(zookeeper_path.parent_path())
 , operation_node_name(zookeeper_path.parent_path().filename())
 , start_node_path(zookeeper_path / ("started|" + current_host))
 , finish_node_path(zookeeper_path / ("finished|" + current_host))
 , initiator_start_node_path(zookeeper_path / ("started|" + String{kInitiator}))
 , num_hosts_node_path(zookeeper_path / "num_hosts")
 , error_node_path(zookeeper_path / "error")
 , alive_node_path(zookeeper_path / ("alive|" + current_host))
 , alive_tracker_node_path(fs::path{root_zookeeper_path} / "alive_tracker")

With backups.zookeeper_path = /clickhouse/backups and a backup UUID:

 text
 
 
/clickhouse/backups/
├── alive_tracker (persistent; "someone is alive" version counter)
└── backup-<uuid>/
 ├── stage/
 │ ├── started|<host> (persistent; host has begun)
 │ ├── started| (persistent; the initiator)
 │ ├── finished|<host> (persistent; host is done)
 │ ├── alive|<host> (EPHEMERAL; host is alive)
 │ ├── num_hosts (persistent; counter)
 │ ├── error (persistent; someone failed)
 │ ├── current|<host>|gathering metadata (0)
 │ ├── current|<host>|writing backup (0)
 │ └── current|<host>|completed (0)
 ├── file_infos/<host>/000000 (persistent; the file manifest, chunked)
 ├── writing_files/17 (persistent; "host #17 owns data file 17")
 ├── repl_part_names/… repl_mutations/… repl_access/…
 ├── repl_sql_objects/… repl_workload_entities/…
 └── keeper_map_tables/… rocksdb_tables/…

Three things to notice:

alive|<host> is ephemeral — it disappears automatically when the host’s ZooKeeper session ends. That is the liveness signal, and it is why a crashed host does not deadlock the whole backup.
current|<host>|<stage> encodes both the stage and a small result payload. For most stages the result is empty; for metadata gathering it is "consistent" or "need_another_attempt". The stage barrier is thus a general barrier that also carries a vote.
file_infos/<host> holds the per-host manifest, chunked because ZooKeeper nodes have a size limit.

Phase 0: claiming the operation

Before any work happens, a host must announce itself and check that it is allowed to start at all. This happens in one ZooKeeper multi request, which matters: the whole claim must be atomic, or two hosts could both believe they own the backup.

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:304
 Coordination::Requests requests;
 requests.reserve(6);

 size_t operation_node_pos = static_cast<size_t>(-1);
 if (!zookeeper->exists(operation_zookeeper_path))
 {
 operation_node_pos = requests.size();
 requests.emplace_back(zkutil::makeCreateRequest(operation_zookeeper_path, "", zkutil::CreateMode::Persistent));
 }

 size_t zookeeper_path_pos = static_cast<size_t>(-1);
 if (!zookeeper->exists(zookeeper_path))
 {
 zookeeper_path_pos = requests.size();
 requests.emplace_back(zkutil::makeCreateRequest(zookeeper_path, "", zkutil::CreateMode::Persistent));
 }

 size_t num_hosts_node_pos = requests.size();
 if (num_hosts)
 requests.emplace_back(zkutil::makeSetRequest(num_hosts_node_path, toString(*num_hosts + 1), num_hosts_version));
 else
 requests.emplace_back(zkutil::makeCreateRequest(num_hosts_node_path, "1", zkutil::CreateMode::Persistent));

 size_t alive_tracker_node_pos = requests.size();
 requests.emplace_back(zkutil::makeSetRequest(alive_tracker_node_path, "", alive_tracker_version));

 requests.emplace_back(zkutil::makeCreateRequest(start_node_path, std::to_string(kCurrentVersion), zkutil::CreateMode::Persistent));
 requests.emplace_back(zkutil::makeCreateRequest(alive_node_path, "", zkutil::CreateMode::Ephemeral));

 Coordination::Responses responses;
 auto code = zookeeper->tryMulti(requests, responses);

Six operations, one transaction: create the operation node, create the stage node, bump the num_hosts counter with a versioned set, bump the alive_tracker version, create my started|<host> marker, create my ephemeral alive|<host> node.

The two sets are versioned deliberately. The version acts as an optimistic-concurrency token, and a ZBADVERSION on either means “somebody else changed the world under me, start over”:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:372
 else if ((num_hosts_node_pos < responses.size()) && num_hosts &&
 (responses[num_hosts_node_pos]->error == Coordination::Error::ZBADVERSION))
 {
 show_error_before_next_attempt(fmt::format("The version of node {} changed", num_hosts_node_path));
 num_hosts.reset(); /// needs to reread 'num_hosts' again
 }
 else if ((alive_tracker_node_pos < responses.size()) &&
 (responses[alive_tracker_node_pos]->error == Coordination::Error::ZBADVERSION))
 {
 show_error_before_next_attempt(fmt::format("The version of node {} changed", alive_tracker_node_path));
 check_concurrency = true; /// needs to recheck for concurrency again
 }

Read that second branch carefully: a failed alive_tracker bump forces a re-check of concurrency. That is the whole concurrency protocol in one line. The alive_tracker version is a cheap “has anything changed underneath me” flag; if it moved, the world may now contain another running backup.

The concurrency check

allow_concurrent_backups defaults to true, but the check exists for when it is turned off. “Another operation is running” is defined precisely as: another backup-<uuid> directory exists whose stage/ contains a node starting with alive.

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:408
 for (const String & found_operation : found_operations)
 {
 if (found_operation.starts_with(is_restore ? "restore-" : "backup-") && (found_operation != operation_node_name))
 {
 Strings stages;
 code = zookeeper->tryGetChildren(fs::path{root_zookeeper_path} / found_operation / "stage", stages);

 if (!((code == Coordination::Error::ZOK) || (code == Coordination::Error::ZNONODE)))
 throw zkutil::KeeperException::fromPath(code, fs::path{root_zookeeper_path} / found_operation / "stage");

 if (code == Coordination::Error::ZOK)
 {
 for (const String & stage : stages)
 {
 if (stage.starts_with("alive"))
 BackupConcurrencyCheck::throwConcurrentOperationNotAllowed(is_restore);
 }
 }
 }
 }

Because alive nodes are ephemeral, a dead host’s backup does not block the next one.

There is also a purely local concurrency check, in the server process rather than in ZooKeeper:

 cpp
 
 
// src/Backups/BackupConcurrencyCheck.cpp:44
 else
 {
 size_t num_local_backups = counters.local_backups;
 size_t num_on_cluster_backups = counters.on_cluster_backups.size();
 if (on_cluster)
 {
 if (!counters.on_cluster_backups.contains(zookeeper_path))
 ++num_on_cluster_backups;
 }
 else
 {
 ++num_local_backups;
 }
 found_concurrent_operation = (num_local_backups + num_on_cluster_backups > 1);
 }

The counters are reference-counted per ZooKeeper path, incremented in the constructor and decremented in the destructor — a scoped lock. Keying by path means two BACKUP ON CLUSTER operations against different clusters do not collide, but two against the same path do.

Phase 1: the metadata consensus loop

Here is the part that is genuinely unusual. Before copying any data, every host independently collects the list of databases and tables it will back up. If a table is renamed mid-collection, one host may see the old name and another the new one, and the resulting backup would be garbage.

ClickHouse’s answer is not a lock — it is a convergent retry loop with exponential backoff and a global agreement check. The loop is documented in a comment that reads like a design document:

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:237
void BackupEntriesCollector::gatherMetadataAndCheckConsistency()
{
 /// With the default values the metadata collecting works in the following way:
 /// 1) it tries to collect the metadata for the first time; then
 /// 2) it tries to collect it again, and compares the results from the first and the second collecting; if they match, it's done; otherwise
 /// 3) it sleeps 100 millisecond and tries to collect again, then compares the results from the second and the third collecting; if they match, it's done; otherwise
 /// 4) it tries to collect again, then compares the results from the third and the fourth collecting; if they match, it's done; otherwise
 /// 5) it sleeps 200 milliseconds and tries to collect again, then compares the results from the fourth and the fifth collecting; if they match, it's done;
 /// ...
 /// and so on, the sleep time is doubled each time until it reaches 5000 milliseconds.
 /// And such attempts will be continued until 600000 milliseconds pass.

The numbers come from the server config: backups.attempts_to_collect_metadata_before_sleep (default 2), backups.min_sleep_before_next_attempt_to_collect_metadata (default 100 ms), backups.max_sleep_before_next_attempt_to_collect_metadata (default 5000 ms), backups.collect_metadata_timeout (default 600000 ms). The backoff is a clean power-of-two ramp, capped at 2^10:

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:91
 /// How long we should sleep after finding an inconsistency error.
 std::chrono::milliseconds getSleepTimeAfterInconsistencyError(int attempt_no, unsigned int attempts_before_sleep, std::chrono::milliseconds min_sleep, std::chrono::milliseconds max_sleep)
 {
 attempts_before_sleep = std::max(attempts_before_sleep, 1U);
 if (((attempt_no + 1) % attempts_before_sleep) != 0)
 return std::chrono::milliseconds::zero(); /// no sleep

 int sleep_counter = attempt_no / attempts_before_sleep;
 std::chrono::milliseconds sleep_time = intExp2(std::min(sleep_counter, 10)) * min_sleep;
 return std::min(sleep_time, max_sleep);
 }

Two kinds of consistency are checked. The local one: collect twice in a row and compare. A table renamed between the two passes shows up as a mismatch.

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:312
 if (!compare_collected_metadata)
 return;

 /// We have to check consistency of collected information to protect from the case when some table or database is
 /// renamed during this collecting making the collected information invalid.
 String mismatch_description;
 if (!compareWithPrevious(mismatch_description))
 {
 /// Usually two passes is minimum.
 /// (Because we need to compare with table names from the previous pass to be sure we are not going to miss anything).
 if (attempt_no >= 1)
 inconsistency_error = Exception{ErrorCodes::INCONSISTENT_METADATA_FOR_BACKUP, "{}", mismatch_description};
 need_another_attempt = true;
 }

And the global one: every host publishes its verdict as the stage result and waits for everybody else’s verdict. The stage barrier does double duty as a consensus mechanism:

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:340
 /// We've gathered metadata, cool! But we have to check that other hosts coped with that too.
 auto all_hosts_results = setStage(next_stage, need_another_attempt ? "need_another_attempt" : "consistent");
 size_t count = std::min(all_hosts.size(), all_hosts_results.size());

 for (size_t i = 0; i != count; ++i)
 {
 const auto & other_host = all_hosts[i];
 const auto & other_host_result = all_hosts_results[i];
 if ((other_host_result != "consistent") && (other_host_result != "need_another_attempt"))
 {
 LOG_WARNING(log, "Found inconsistency on host {}: {}", other_host, other_host_result);
 inconsistency_error = Exception{ErrorCodes::INCONSISTENT_METADATA_FOR_BACKUP, "Found inconsistency on host {}: {}", other_host, other_host_result};
 need_another_attempt = true;
 return;
 }
 }

The stage name is parameterised by the attempt number — "gathering metadata (0)", "gathering metadata (1)", and so on — because the barrier is per attempt. Every host must reach attempt N before anyone proceeds to attempt N+1.

Note the subtlety: a host that failed to collect metadata publishes its error text as the stage result and returns immediately. The other hosts, on reading that result, see a string that is neither "consistent" nor "need_another_attempt", and they treat it as a failure and start their own next attempt.

The stage barrier itself

The barrier is a condition variable fed by a background polling thread. There is no watch on the stage nodes — the implementation polls.

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:504
 try
 {
 /// Recreate the 'alive' node if necessary and read a new state from ZooKeeper.
 auto holder = with_retries.createRetriesControlHolder("BackupCoordinationStageSync::watchingThread");
 auto & zookeeper = holder.faulty_zookeeper;
 with_retries.renewZooKeeper(zookeeper);

 if (should_stop())
 return;

 /// Recreate the 'alive' node if it was removed.
 createAliveNode(zookeeper);

 /// Reads the current state from nodes in ZooKeeper.
 readCurrentState(zookeeper);
 }

The alive node must be recreated on every session loss — an ephemeral node is tied to the session, so a reconnect means a new session means the node is gone. The recreation is a two-op multi (create the node, bump the alive_tracker version), which is what makes the tracker version a reliable “something happened” flag:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:547
void BackupCoordinationStageSync::createAliveNode(Coordination::ZooKeeperWithFaultInjection::Ptr zookeeper)
{
 if (zookeeper->exists(alive_node_path))
 return;

 Coordination::Requests requests;
 requests.emplace_back(zkutil::makeCreateRequest(alive_node_path, "", zkutil::CreateMode::Ephemeral));
 requests.emplace_back(zkutil::makeSetRequest(alive_tracker_node_path, "", -1));
 zookeeper->multi(requests);

 LOG_INFO(log, "The alive node was recreated for {}", current_host_desc);
}

The polling period is backups.sync_period_ms, default 5000 ms. So the worst-case latency for a stage barrier is five seconds. That is a deliberate trade: polling is simple and survives any kind of ZooKeeper weirdness, where a watch would need careful re-arming.

Publishing a stage is a plain create of a persistent node whose name encodes the stage, with the result as its payload:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:869
void BackupCoordinationStageSync::createStageNode(const String & stage, const String & stage_result, Coordination::ZooKeeperWithFaultInjection::Ptr zookeeper)
{
 if (isErrorSet())
 rethrowSetError();

 String serialized_error;
 if (zookeeper->tryGet(error_node_path, serialized_error))
 {
 auto [exception, host] = parseErrorNode(serialized_error);
 if (exception)
 {
 std::lock_guard lock{mutex};
 state.addErrorInfo(exception, host);
 std::rethrow_exception(exception);
 }
 }

 auto code = zookeeper->tryCreate(getStageNodePath(stage), stage_result, zkutil::CreateMode::Persistent);
 if (code == Coordination::Error::ZOK)
 {
 std::lock_guard lock{mutex};
 state.hosts.at(current_host).stages[stage] = stage_result;
 return;
 }

 if (code == Coordination::Error::ZNODEEXISTS)
 {
 String another_result = zookeeper->get(getStageNodePath(stage));
 std::lock_guard lock{mutex};
 state.hosts.at(current_host).stages[stage] = another_result;
 return;
 }

 throw zkutil::KeeperException::fromPath(code, getStageNodePath(stage));
}

ZNODEEXISTS is treated as success — the host already published this stage (possibly in a previous attempt that appeared to fail). Idempotence by tolerance.

And the waiting side:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:945
 const HostInfo & host_info = it->second;
 auto stage_it = host_info.stages.find(stage_to_wait);
 if (stage_it != host_info.stages.end())
 {
 results[i] = stage_it->second;
 continue;
 }

 if (host_info.finished)
 {
 if (stage_to_wait == BackupCoordinationStage::FINALIZING_TABLES)
 {
 /// This is a newly added stage. For compatibility with older server versions,
 /// allow other replicas to skip this stage. This doesn't break anything: this stage
 /// unpauses refreshable materialized views, but older server versions don't pause
 /// them in the first place.
 results[i] = "";
 continue;
 }
 throw Exception(ErrorCodes::FAILED_TO_SYNC_BACKUP_OR_RESTORE,
 "{} finished without coming to stage {}", getHostDesc(host), stage_to_wait);
 }

The FINALIZING_TABLES special case is a lovely piece of pragmatism: a stage was added to the protocol later, and an older server that has already finished will never reach it. Rather than deadlock the cluster, the barrier treats “finished” as “passed every stage” for that one stage only.

The stage list

The stage names are constants in one header (src/Backups/BackupCoordinationStage.h), which makes the protocol readable at a glance:

 cpp
 
 
// src/Backups/BackupCoordinationStage.h:9
namespace BackupCoordinationStage
{
 /// Finding all tables and databases which we're going to put to the backup and collecting their metadata.
 constexpr const char * GATHERING_METADATA = "gathering metadata";

 String formatGatheringMetadata(int attempt_no);

 /// Making temporary hard links and prepare backup entries.
 constexpr const char * EXTRACTING_DATA_FROM_TABLES = "extracting data from tables";

 /// Running special tasks for replicated tables which can also prepare some backup entries.
 constexpr const char * RUNNING_POST_TASKS = "running post-tasks";

 /// Building information about all files which will be written to a backup.
 constexpr const char * BUILDING_FILE_INFOS = "building file infos";

 /// Writing backup entries to the backup and removing temporary hard links.
 constexpr const char * WRITING_BACKUP = "writing backup";

 /// Finding databases and tables in the backup which we're going to restore.
 constexpr const char * FINDING_TABLES_IN_BACKUP = "finding tables in backup";

The backup half is: gathering metadata → extracting data → running post-tasks → building file infos → writing backup → completed. The restore half is separate (finding tables → checking access rights → creating databases → creating tables → inserting data → finalizing tables → completed). BackupCoordinationStageSync is constructed with an is_restore flag and serves both protocols.

Phase 2: the file manifest — the heart of it

Each host has its own set of files (its own MergeTree parts, its own system tables, its own access files). The backup needs one list.

Each host computes a BackupFileInfo for each of its files — name, size, checksum, plus the incremental-backup fields — and publishes them to ZooKeeper:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:945
void BackupCoordinationOnCluster::addFileInfos(BackupFileInfos && file_infos_)
{
 {
 std::lock_guard lock{file_infos_mutex};
 if (file_infos)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "addFileInfos() must not be called after preparing");
 }

 auto component_guard = Coordination::setCurrentComponent("BackupCoordinationOnCluster::addFileInfos");
 /// Serialize `file_infos_` and write it to ZooKeeper's nodes.
 String file_infos_str = FileInfos::serialize(file_infos_);
 serializeToMultipleZooKeeperNodes(zookeeper_path + "/file_infos/" + current_host, file_infos_str, "addFileInfos");
}

The serialized form is compact and deliberately omits the fields that are derived rather than observed:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:106
 static String serialize(const BackupFileInfos & file_infos_)
 {
 WriteBufferFromOwnString out;
 writeBinary(file_infos_.size(), out);
 for (const auto & info : file_infos_)
 {
 writeBinary(info.file_name, out);
 writeBinary(info.size, out);
 writeBinary(info.checksum, out);
 writeBinary(info.base_size, out);
 writeBinary(info.base_checksum, out);
 writeBinary(info.encrypted_by_disk, out);
 writeBinary(info.reference_target, out);
 /// We don't store `info.data_file_name` and `info.data_file_index` because they're determined automalically
 /// after reading file infos for all the hosts (see the class BackupCoordinationFileInfos).
 }
 return out.str();
 }

The payload can be large — a backup of a big cluster has millions of files — so it is chopped into chunks of at most backup_restore_keeper_value_max_size bytes (default 1 MiB) and stored as numbered child nodes:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:308
 size_t max_part_size = keeper_settings.value_max_size;
 if (!max_part_size)
 max_part_size = value.size();

 size_t num_parts = (value.size() + max_part_size - 1) / max_part_size; /// round up

 for (size_t i = 0; i != num_parts; ++i)
 {
 size_t begin = i * max_part_size;
 size_t end = std::min(begin + max_part_size, value.size());
 String part = value.substr(begin, end - begin);
 String part_path = fmt::format("{}/{:06}", path, i);

The {:06} zero-padding makes the lexicographic order of the child names equal to the numeric order of the parts — so reading them back is just getChildren + sort.

Then every host reads everyone’s file infos and computes the global plan:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:975
void BackupCoordinationOnCluster::prepareFileInfos() const
{
 if (file_infos)
 return;

 file_infos.emplace(BackupCoordinationFileInfos::Config{plain_backup, data_file_name_gen, data_file_name_prefix_length});

 Strings hosts_with_file_infos;
 {
 auto holder = with_retries.createRetriesControlHolder("prepareFileInfos::get_hosts");
 holder.retries_ctl.retryLoop(
 [&, &zk = holder.faulty_zookeeper]()
 {
 with_retries.renewZooKeeper(zk);
 hosts_with_file_infos = zk->getChildren(zookeeper_path + "/file_infos");
 });
 }

 for (const String & host : hosts_with_file_infos)
 {
 String file_infos_str = deserializeFromMultipleZooKeeperNodes(zookeeper_path + "/file_infos/" + host, "prepareFileInfos");
 auto deserialized_file_infos = FileInfos::deserialize(file_infos_str).file_infos;
 file_infos->addFileInfos(std::move(deserialized_file_infos), host);
 }
}

Phase 3: assigning a global name and index to every file

BackupCoordinationFileInfos::prepare() is where the global plan is computed. Every host runs this code over the same input and must get the same answer — which is why the sort key is the file name and nothing else.

 cpp
 
 
// src/Backups/BackupCoordinationFileInfos.cpp:67
 /// Make a list of all file infos from all hosts.
 size_t total_num_infos = 0;
 for (const auto & [_, infos] : file_infos)
 total_num_infos += infos.size();

 file_infos_for_all_hosts.reserve(total_num_infos);
 for (auto & [_, infos] : file_infos)
 for (auto & info : infos)
 file_infos_for_all_hosts.emplace_back(&info);

 /// Sort the list of all file infos by file name (file names must be unique).
 std::sort(file_infos_for_all_hosts.begin(), file_infos_for_all_hosts.end(), BackupFileInfo::LessByFileName{});

 auto adjacent_it = std::adjacent_find(file_infos_for_all_hosts.begin(), file_infos_for_all_hosts.end(), BackupFileInfo::EqualByFileName{});
 if (adjacent_it != file_infos_for_all_hosts.end())
 {
 throw Exception(
 ErrorCodes::BACKUP_ENTRY_ALREADY_EXISTS, "Entry {} added multiple times to backup", quoteString((*adjacent_it)->file_name));
 }

The file names are unique because each host stores its data under shards/<n>/replicas/<m>/ in the backup. Two hosts cannot produce the same name, and a duplicate is a hard error rather than a silent overwrite.

Now the interesting branch — the non-plain (deduplicating) path:

 cpp
 
 
// src/Backups/BackupCoordinationFileInfos.cpp:162
 /// For non-plain backups files with the same size and checksum are stored only once,
 /// in order to find those files we'll use this map.
 std::map<SizeAndChecksum, size_t> data_file_index_by_checksum;

 for (size_t i = 0; i != file_infos_for_all_hosts.size(); ++i)
 {
 auto & info = *(file_infos_for_all_hosts[i]);

 if (!info.reference_target.empty())
 {
 if (!try_resolve_reference(info))
 unresolved_references.push_back(&info);

 continue;
 }

 if (info.size == info.base_size)
 {
 /// A file is either empty or can be get from the base backup as a whole.
 info.data_file_name.clear();
 info.data_file_index = static_cast<size_t>(-1);
 }
 else
 {
 SizeAndChecksum size_and_checksum{info.size, info.checksum};
 auto [it, inserted] = data_file_index_by_checksum.emplace(size_and_checksum, i);
 if (inserted)
 {
 /// Found a new file.
 info.data_file_name = getBackupDataFileName(info, config.data_file_name_generator, config.data_file_name_prefix_length);
 info.data_file_index = i;
 ++num_files;
 total_size_of_files += info.size - info.base_size;
 }
 else
 {
 /// Found a file with the same size and checksum as some file before, reuse old `data_file_index` and `data_file_name`.
 info.data_file_index = it->second;
 info.data_file_name = file_infos_for_all_hosts[it->second]->data_file_name;
 }
 }

 file_name_to_info.emplace(info.file_name, &info);
 }

Three things happen here:

data_file_index is simply the position in the globally sorted list. That is the coordination key: index 17 is the same file on every host.
Files with identical (size, checksum) collapse to one physical data file. On a replicated cluster, the same part exists on every replica with the same checksum, so a 3-replica cluster stores one copy of the data and three manifest entries pointing at it.
total_size_of_files accumulates info.size - info.base_size, i.e. only the bytes that are new relative to the incremental base.

Phase 4: electing the writer — ZooKeeper `create` as a mutex

Now the question: host A has file 17, host B has file 23, host C has file 17 too (same content). Somebody has to write the physical file for index 17. Exactly one host must do it.

The answer is the simplest possible use of ZooKeeper: create is atomic, and fails with ZNODEEXISTS if the node is there.

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:1001
bool BackupCoordinationOnCluster::startWritingFile(size_t data_file_index)
{
 {
 /// Check if this host is already writing this file.
 std::lock_guard lock{writing_files_mutex};
 if (writing_files.contains(data_file_index))
 return false;
 }
 auto component_guard = Coordination::setCurrentComponent("BackupCoordinationOnCluster::startWritingFile");

 /// Store in Zookeeper that this host is the only host which is allowed to write this file.
 bool host_is_assigned = false;
 String full_path = zookeeper_path + "/writing_files/" + std::to_string(data_file_index);
 String host_index_str = std::to_string(current_host_index);

 auto holder = with_retries.createRetriesControlHolder("startWritingFile");
 holder.retries_ctl.retryLoop(
 [&, &zk = holder.faulty_zookeeper]()
 {
 with_retries.renewZooKeeper(zk);
 auto code = zk->tryCreate(full_path, host_index_str, zkutil::CreateMode::Persistent);

 if (code == Coordination::Error::ZOK)
 host_is_assigned = true; /// If we've just created this ZooKeeper's node, this host is assigned.
 else if (code == Coordination::Error::ZNODEEXISTS)
 host_is_assigned = (zk->get(full_path) == host_index_str); /// The previous retry could write this ZooKeeper's node and then fail.
 else
 throw zkutil::KeeperException::fromPath(code, full_path);
 });

 if (!host_is_assigned)
 return false; /// Other host is writing this file.

 {
 /// Check if this host is already writing this file,
 /// and if it's not, mark that this host is writing this file.
 /// We have to check that again because we were accessing ZooKeeper with the mutex unlocked.
 std::lock_guard lock{writing_files_mutex};
 return writing_files.emplace(data_file_index).second; /// Return false if this host is already writing this file.
 }
}

The ZNODEEXISTS branch is the interesting one. The node’s value is the index of the host that won the election. If the value equals my index, I won — even though I did not create the node. That covers the case where a retry after a network hiccup created the node successfully but the response was lost. Idempotent election without a separate “am I the owner?” round trip.

And the caller simply skips files it lost:

 cpp
 
 
// src/Backups/BackupImpl.cpp:1741
 if (!coordination->startWritingFile(info.data_file_index))
 {
 LOG_TRACE(log, "Writing backup for file {} from {}: skipped, data file #{} is already being written", info.data_file_name, src_file_desc, info.data_file_index);
 return;
 }

So on a 3-replica cluster with identical parts, exactly one replica uploads each distinct file and the other two skip it. The result is a backup that contains the data once.

 text
 
 
 host A host B host C
 | | |
 |-- file_infos/A ------> /file_infos/ <--- file_infos/B --|
 | | |
 | all three read all three manifests, sort by name |
 | all three compute the same data_file_index map |
 | | |
 | create /writing_files/17 -> ZOK (A wins) |
 | | create -> ZNODEEXISTS |
 | | value != my index |
 | | -> skip file 17 |
 | | |
 | write file 17 | write file 23 | write file 41

Retries: everything is wrapped

Every ZooKeeper operation in the backup path goes through a retry controller with exponential backoff. The default is deliberately generous, because a backup that fails halfway because of a 30-second Keeper blip is worse than one that takes longer:

 cpp
 
 
// src/Core/Settings.cpp:4848
 DECLARE(UInt64, backup_restore_keeper_max_retries, 1000, R"(
Max retries for [Zoo]Keeper operations in the middle of a BACKUP or RESTORE operation.
Should be big enough so the whole operation won't fail because of a temporary [Zoo]Keeper failure.
)", 0, \

The backoff is a plain doubling with a cap:

 cpp
 
 
// src/Common/ZooKeeper/ZooKeeperRetries.h:232
 /// retries
 logLastError("will retry due to error");
 sleepForMilliseconds(current_backoff_ms);
 current_backoff_ms = std::min(current_backoff_ms * 2, retries_info.max_backoff_ms);

with backup_restore_keeper_retry_initial_backoff_ms = 100 and backup_restore_keeper_retry_max_backoff_ms = 5000. Critically, only hardware errors are retried:

 cpp
 
 
// src/Common/ZooKeeper/ZooKeeperRetries.h:81
 catch (const zkutil::KeeperException & e)
 {
 iteration_cleanup();

 if (!Coordination::isHardwareError(e.code))
 throw;

 setKeeperError(std::current_exception(), e.code, e.message());
 }

A ZNODEEXISTS or a ZBADVERSION is a semantic answer, not a failure — retrying it would be wrong. A connection reset is a hardware error and gets retried.

Trade-offs

What the design buys:

One manifest, one file namespace. A BACKUP ON CLUSTER is restorable as a unit; RESTORE ON CLUSTER puts the data back on the right shards and replicas.
No data duplication. The (size, checksum) dedup means a 3-replica cluster stores the part once.
No coordination server. The only shared state is ZooKeeper, and the only ZooKeeper operation used for the critical election is create.
Crash tolerance. Every phase is idempotent. A host that dies leaves ephemeral nodes behind; the survivors notice within sync_period_ms and either retry or abort.

What it costs:

Polling latency. backups.sync_period_ms defaults to 5000 ms, so each stage barrier costs up to five seconds of wall clock even when everything is ready. A backup with six stages pays this repeatedly.
ZooKeeper is a hard dependency for the whole operation. It is not in the data path — the actual bytes go straight to S3 — but it is in the control path, and a Keeper outage stalls every backup in the cluster.
The manifest is O(files) in ZooKeeper. A backup of millions of parts writes millions of file-info records into Keeper, in 1 MiB chunks. This is the single biggest scaling constraint on BACKUP ON CLUSTER.
The metadata consensus loop is a heuristic, not a proof. Two consecutive identical reads on every host, plus a cross-host vote, gives strong practical safety — but there is no fencing token, so in principle a rename that happens to straddle the window identically on every host could slip through. The INCONSISTENT_METADATA_FOR_BACKUP error and the 600-second timeout are the safety nets.

Wrapping up

The one thing to remember: a cluster-wide backup is a global file namespace, and the global namespace is produced by sorting every host’s file list by name. The sort is the whole trick — it gives every file a cluster-unique index without any central allocation, because sorting is commutative and deterministic.

The part people get wrong: the deduplication is by (size, checksum), not by path. Two hosts with the same file at different paths in the backup share one physical object. That is why getNumFiles() and getNumEntries() differ, and why the backup’s on-disk size is much smaller than the sum of the replicas’ data.

What to go do about it: if your BACKUP ON CLUSTER is slow, look at the log lines Setting stage: ... and Hosts [...] reached stage .... The gap between them is barrier latency — and with the default 5-second sync period, that gap is almost certainly the problem, not the data transfer.

How ClickHouse Replicates MergeTree: A ZooKeeper Log and a Hash Deduplication

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: ReplicatedMergeTree is a log in ZooKeeper plus a set of workers that execute the log’s entries. The log is a directory of sequential znodes holding text; each replica copies the log into its own queue and drains it out of order. The interesting part is the insert path: the block number is allocated by an ephemeral-sequential znode, and in the same atomic multi the insert checks whether the block’s content hash already exists. If it does, the duplicate rows are removed and the part is rewritten.

Replication is a distributed systems problem wearing a database’s clothes. Three replicas of a table must end up with the same data. The network partitions. A replica goes down mid-merge. The client retries an insert because it did not see the acknowledgement.

The requirements that fall out of this:

A total order. The replicas must agree on what happened, not just on the final state — otherwise a merge on one replica and a drop on another produce two different tables.
Idempotence. Any operation may be executed twice, and the second execution must be a no-op.
Convergence without a coordinator. No replica may be the one that decides what the others do.

ClickHouse’s answer to (1) is a ZooKeeper log. Its answer to (2) is a content hash. Its answer to (3) is that every replica reads the same log and decides independently what it can execute right now.

The log

The log is a directory of sequential znodes under /log. Each znode is named log-0000000001, log-0000000002, … and its content is a text blob describing one operation:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.h:27
 enum Type
 {
 EMPTY, /// Not used.
 GET_PART, /// Get the part from another replica.
 ATTACH_PART, /// Attach the part, possibly from our own replica (if found in /detached folder).
 /// You may think of it as a GET_PART with some optimizations as they're nearly identical.
 MERGE_PARTS, /// Merge the parts.
 DROP_RANGE, /// Delete the parts in the specified partition in the specified number range.
 CLEAR_COLUMN, /// NOTE: Deprecated. Drop specific column from specified partition.
 CLEAR_INDEX, /// NOTE: Deprecated. Drop specific index from specified partition.
 REPLACE_RANGE, /// Drop certain range of parts and replace them by new ones
 MUTATE_PART, /// Apply one or several mutations to the part.
 ALTER_METADATA, /// Apply alter modification according to global /metadata and /columns paths
 SYNC_PINNED_PART_UUIDS, /// Synchronization point for ensuring that all replicas have up to date in-memory state.
 CLONE_PART_FROM_SHARD, /// Clone part from another shard.
 DROP_PART, /// NOTE: Virtual (has the same (de)serialization format as DROP_RANGE). Deletes the specified part.
 };

The log entry is human-readable, and it is versioned:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.cpp:77
 out << "format version: " << format_version << "\n"
 << "create_time: " << LocalDateTime(create_time ? create_time : time(nullptr), DateLUT::serverTimezoneInstance()) << "\n"
 << "source replica: " << source_replica << '\n'
 << "block_id: " << escape << block_hashes_string << '\n';

with a version enum that grows as features are added:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.cpp:26
enum FormatVersion : UInt8
{
 FORMAT_WITH_CREATE_TIME = 2,
 FORMAT_WITH_BLOCK_ID = 3,
 FORMAT_WITH_DEDUPLICATE = 4,
 FORMAT_WITH_UUID = 5,
 FORMAT_WITH_DEDUPLICATE_BY_COLUMNS = 6,
 FORMAT_WITH_LOG_ENTRY_ID = 7,

 FORMAT_LAST = 8,
};

This matters because the log is shared between replicas of different versions during a rolling upgrade. An old replica must be able to read a new replica’s entries (at least the subset it understands), and a new replica must be able to read an old replica’s entries. A text format with an explicit version number, where unknown trailing fields are skipped, is far easier to evolve than a binary one.

Copying the log into a queue

A replica does not execute the shared log directly. It copies the entries into its own queue, under /replicas/<name>/queue/, and remembers how far it has read with a log_pointer. The copy is a PersistentSequential create, so the queue entries are numbered in the order they were copied:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:938
 ops.emplace_back(zkutil::makeCreateRequest(
 fs::path(replica_path) / "queue/queue-", res.data, zkutil::CreateMode::PersistentSequential));

and the pointer is advanced in the same multi, so the copy and the pointer move are atomic:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:953
 ops.emplace_back(zkutil::makeSetRequest(
 fs::path(replica_path) / "log_pointer", toString(last_entry_index + 1), -1));

Why copy at all? Because a replica needs a private queue: entries it has already executed are removed from it, entries it cannot execute yet (because they conflict with a running merge) stay in it, and the shared log is append-only for everyone. The queue is the replica’s private “to-do list”, derived from the shared “history”.

The copy happens in batches, and the batch size grows exponentially so that a replica that has been offline for a long time catches up with a handful of round trips instead of thousands:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:903
 /// Increase the batch size exponentially, so it will saturate to MAX_MULTI_OPS.
 if (current_multi_batch_size < MAX_MULTI_OPS)
 current_multi_batch_size = std::min<size_t>(MAX_MULTI_OPS, current_multi_batch_size * 2);

Executing out of order

Here is the part that surprises people. The queue is a queue, but the execution order is not the queue order:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.h:96
 /** The queue of what you need to do on this line to catch up. It is taken from ZooKeeper (/replicas/me/queue/).
 * In ZK records in chronological order. Here they are executed in parallel and reorder after entry execution.
 * Order of execution is not "queue" at all. Look at selectEntryToProcess.
 */
 Queue queue;

The selection walks the queue and takes the first entry that is currently executable:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:2202
 for (auto it = queue.begin(); it != queue.end(); ++it)
 {
 if ((*it)->currently_executing)
 continue;

 if (shouldExecuteLogEntry(**it, (*it)->postpone_reason, merger_mutator, data, committing_blocks, lock))
 {
 entry = *it;
 /// We gave a chance for the entry, move it to the tail of the queue, after that
 /// we move it to the end of the queue.
 queue.splice(queue.end(), queue, it);
 break;
 }

 ++(*it)->num_postponed;
 (*it)->last_postpone_time = time(nullptr);
 }

Two details are worth pausing on.

The entry is moved to the tail when it is selected. So a failing entry goes to the back of the line and does not block the entries behind it. This is the difference between “one bad merge stalls the whole replica” and “one bad merge is retried while the rest of the queue drains”.

“Currently executable” is a real predicate. An entry may be postponed because:

it failed recently, and there is an exponential backoff on it;
it produces a part that a currently executing entry also produces (isCoveredByFuturePartsImpl);
it intersects a DROP_RANGE/REPLACE_RANGE intent;
it produces a part that is going to be dropped anyway — “Optimization: it does not really make sense to generate parts that are going to be dropped anyway”;
the fetch pool is saturated.

The invariant that makes all of this safe is spelled out in the header:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.h:90
 /// A set of parts that should be on this replica according to the queue entries that have been done
 /// up to this point. The invariant holds: `virtual_parts` = `current_parts` + `queue`.
 /// Note: it can be different from the actual set of parts because the replica can decide to fetch
 /// a bigger part instead of the part mentioned in the log entry.
 ActiveDataPartSet current_parts;

virtual_parts is what the replica will have once the queue drains; it is computed, not observed. That is what lets the merge-assignment logic reason about the future without talking to anybody.

Which pool

Not all entries are equal. Fetching a part is I/O-bound and small; a merge is CPU- and disk-bound and large. So the entry type picks the thread pool:

 cpp
 
 
// src/Storages/StorageReplicatedMergeTree.cpp:4364
 /// Depending on entry type execute in fetches (small) pool or big merge_mutate pool
 if (job_type == LogEntry::GET_PART || job_type == LogEntry::ATTACH_PART)
 {
 assignee.scheduleFetchTask(std::make_shared<ExecutableLambdaAdapter>(
 [this, selected_entry] () mutable
 {
 return processQueueEntry(selected_entry);
 }, common_assignee_trigger, getStorageID()));
 return true;
 }
 if (job_type == LogEntry::MERGE_PARTS)
 {
 auto task = std::make_shared<MergeFromLogEntryTask>(selected_entry, *this, common_assignee_trigger);
 assignee.scheduleMergeMutateTask(task);
 return true;
 }

A replica that is catching up after being offline for an hour has hundreds of GET_PART entries and a handful of MERGE_PARTS entries; putting them in separate pools means the merges do not starve the fetches (or the other way round).

When an entry completes, it is removed from the RAM queue and then from ZooKeeper:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:788
 /// Remove the job from the queue in the RAM.
 /// You can not just refer to a pre-saved iterator, because someone else might be able to delete the task.
 /// Why do we view the queue from the end?
 /// - because the task for execution first is moved to the end of the queue, so that in case of failure it remains at the end.
 for (Queue::iterator it = queue.end(); it != queue.begin();)

The iteration is from the end precisely because of the “move to the tail” rule above.

The insert path: block number and deduplication in one round trip

Now the heart of it. A client inserts a block. The replica must:

give the resulting part a name;
know that this block has not been inserted before;
do both atomically with respect to other replicas doing the same thing.

The block number

Every MergeTree part has a block number — the number of the last block of rows in it — and in a replicated table that number has to be allocated by a coordinator, because two replicas inserting concurrently must not pick the same number.

ClickHouse allocates it with an ephemeral sequential znode:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:53
EphemeralLockInZooKeeper createEphemeralLockInZooKeeper(
 const String & path_prefix_,
 const String & temp_path,
 const ZooKeeperWithFaultInjectionPtr & zookeeper_,
 const std::vector<String> & deduplication_paths,
 const std::optional<String> & znode_data)
{
 String path;

 if (deduplication_paths.empty())
 {
 String holder_path = znode_data.value_or(temp_path + "/" + EphemeralLockInZooKeeper::LEGACY_LOCK_OTHER);
 path = zookeeper_->create(path_prefix_, holder_path, zkutil::CreateMode::EphemeralSequential);
 }

ZooKeeper’s sequential create appends a 10-digit counter to the path prefix. The counter is the block number:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:106
 const UInt64 number = parseSequentialNodeNumber(path, path_prefix_.size());
 warnIfBlockNumberIsHigh(number, path);
 CurrentMetrics::max(CurrentMetrics::MaxAllocatedEphemeralLockSequentialNumber, number);
 return EphemeralLockInZooKeeper{path_prefix_, zookeeper_, path, number};

This is a lovely piece of engineering: the block number is allocated by the same operation that creates the lock, in one round trip, with no read- modify-write. The node is ephemeral, so if the session dies the number is released — but the number is not reused, because the sequential counter is monotone regardless.

The path is per partition:

 cpp
 
 
// src/Storages/StorageReplicatedMergeTree.cpp:7987
 auto lock = createEphemeralLockInZooKeeper(
 fs::path(partition_path) / "block-", fs::path(zookeeper_table_path) / "temp", zookeeper, zookeeper_block_id_paths, znode_data);

so /block_numbers/<partition>/block-0000000123 is the lock for block number 123 of that partition. The lock is deleted as part of the commit multi.

The deduplication check rides along

The second thing the insert needs is a duplicate check. The deduplication id is a hash of the block’s content, not the block number:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:69
std::string DeduplicationHash::getBlockId() const
{
 return fmt::format("{}_{}_{}", partition_id, hash.items[0], hash.items[1]);
}

and it lives in a znode whose content is the name of the part that holds that data:

 cpp
 
 
// src/Interpreters/InsertDeduplication.cpp:75
std::string DeduplicationHash::getPath(const std::string & storage_path) const
{
 std::string hashes_directory = [&] ()
 {
 switch (hash_type)
 {
 case HashType::SYNC:
 return "blocks";
 case HashType::ASYNC:
 return "async_blocks";
 case HashType::UNIFIED:
 return "deduplication_hashes";
 }
 }();
 return (std::filesystem::path(storage_path) / hashes_directory / getBlockId()).string();
}

The three directories are a compatibility story: blocks is the old sync-insert location, async_blocks the old async-insert one, and deduplication_hashes the unified one that both now share.

Now the trick. The duplicate check and the lock creation are performed in a single multi:

 cpp
 
 
// src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp:67
 else
 {
 String holder_path = znode_data.value_or(temp_path + "/" + EphemeralLockInZooKeeper::LEGACY_LOCK_INSERT);

 /// Check for duplicates in advance, to avoid superfluous block numbers allocation
 Coordination::Requests ops;

 for (const auto & single_dedup_path : deduplication_paths)
 zkutil::addCheckNotExistsRequest(ops, *zookeeper_, single_dedup_path);

 auto deduplication_path_ops_size = ops.size();

 ops.emplace_back(zkutil::makeCreateRequest(path_prefix_, holder_path, zkutil::CreateMode::EphemeralSequential));
 Coordination::Responses responses;
 Coordination::Error e = zookeeper_->tryMulti(ops, responses);
 if (e == Coordination::Error::ZNODEEXISTS)
 {
 auto failed_idx = zkutil::getFailedOpIndex(Coordination::Error::ZNODEEXISTS, responses);

 if (failed_idx < deduplication_path_ops_size)
 {
 const String & failed_op_path = ops[failed_idx]->getPath();
 LOG_DEBUG(
 getLogger("createEphemeralLockInZooKeeper"),
 "Deduplication path already exists: deduplication_path={}",
 failed_op_path);
 return EphemeralLockInZooKeeper{"", nullptr, "", 0, failed_op_path};
 }
 }

Read that carefully, because it is the whole trick in one function:

The multi is [check-not-exists(dedup_path_1), ..., create(block-)].
If the dedup path does not exist, the create succeeds and the block number is allocated — atomically, in one round trip.
If the dedup path does exist, the multi fails with ZNODEEXISTS, the failed op index tells you which dedup path conflicted, and the function returns a lock with isLocked() == false and a conflict_path.

The caller then knows, without any extra round trip, that this exact content has already been inserted:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:934
 auto block_number_lock = storage.allocateBlockNumber(part->info.getPartitionId(), zookeeper, block_id_pathes, "", block_data); /// 1 RTT

 ThreadFuzzer::maybeInjectSleep();

 {
 std::filesystem::path conflict_path = block_number_lock.getConflictPath();
 if (!conflict_path.empty())
 {
 LOG_DEBUG(log, "Cannot get lock, the conflict path is {}", conflict_path);
 auto conflicted_hash_it = std::find_if(
 deduplication_hashes.begin(),
 deduplication_hashes.end(),
 [&](const DeduplicationHash & hash)
 {
 return hash.getBlockId() == conflict_path.filename().string();
 });
 chassert(conflicted_hash_it != deduplication_hashes.end());
 retry_context.conflict_deduplication_hashes.push_back(*conflicted_hash_it);

 return CommitRetryContext::RESOLVE_CONFLICTS;
 }
 }

The comment /// 1 RTT on the allocateBlockNumber call is the author telling you that this whole thing — check for duplicates and allocate the block number — costs exactly one round trip to ZooKeeper. That is the entire point. A naive implementation would be: read the dedup znode (1 RTT), decide, create the block number (1 RTT), check again (1 RTT). Three round trips, and a race window between the check and the create.

Resolving the conflict

When the multi fails with a conflict, the replica needs to know which part holds the duplicate data — that is what the dedup znode’s content is for. It reads the conflicting znode:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:767
 auto response = zookeeper->tryGet(getDeduplicationPaths(storage.zookeeper_path, retry_context.conflict_deduplication_hashes));
 for (size_t i = 0; i < retry_context.conflict_deduplication_hashes.size(); ++i)
 {
 auto & deduplication_hash = retry_context.conflict_deduplication_hashes[i];
 const auto & resp = response[i];

and then the rows that duplicate the existing part are removed from the block, and the part is rewritten with what is left:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:512
 auto result = partition.deduplication_info->deduplicateBlock(
 getDeduplicationBlockIds(conflicts),
 partition.block_with_partition.partition_id,
 context);

If everything was a duplicate, the part is dropped and the insert reports that it was deduplicated:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:538
 if (partition.block_with_partition.block->rows() == 0)
 {
 // Whole block was deduplicated

 if (!is_async_insert)
 {
 chassert(conflicts.size() == 1);
 auto block_id = conflicts.front().getBlockId();

and the retry loop goes back to the top and re-commits the remaining rows with a fresh block number.

Note what this means: the deduplication is not “skip the whole insert”. It is “remove the rows that are already there and insert the rest”. A block that partially overlaps with an existing part is inserted partially. That is why the dedup id is a content hash and not the block number: the block number is allocated per partition and is meaningless as a duplicate detector, while the content hash is exactly the question “have I seen these rows before?”.

The whole insert, end to end

 text
 
 
 INSERT (block B into partition P)
 |
 | hash the content of B -> dedup id = "<partition>_<hash>"
 v
 multi( [check-not-exists(/deduplication_hashes/<id>),
 create(/block_numbers/P/block-0000000123, EPHEMERAL_SEQUENTIAL)] )
 |
 +--- ZNODEEXISTS --> the block is a duplicate
 | read the znode -> the part that holds it
 | drop the duplicate rows, rewrite the part, retry
 |
 +--- ZOK --> block number = 123
 part->info.min_block = max_block = 123
 part name = <partition>_<min_block>_<max_block>_<level>
 |
 v
 multi( [create(/log/log-...), delete(block-number lock),
 create(quorum znode), create(shared data lock), ...] )
 |
 v
 the log entry "get <part name>" is now the history;
 every replica will fetch the part from whoever has it

The quorum

For a replicated table, “the insert succeeded” means more than “the part is on disk here”. The insert waits for a quorum of replicas to have the part:

 cpp
 
 
// src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:789
 /** If we need a quorum - create a node in which the quorum is monitored.
 * (If such a node already exists, then someone has managed to make another quorum record at the same time,
 * but for it the quorum has not yet been reached.
 * You can not do the next quorum record at this time.)
 */
 if (isQuorumEnabled())
 {
 ReplicatedMergeTreeQuorumEntry quorum_entry;
 quorum_entry.part_name = part->name;
 quorum_entry.required_number_of_replicas = getQuorumSize(total_replicas_count);
 quorum_entry.replicas.insert(storage.replica_name);

The quorum znode is a classic “watch and count” pattern: each replica that receives the part adds itself to the znode’s content, and when the count reaches the required number the znode is deleted, which wakes up the waiters.

Looking at it

 sql
 
 
-- The replication log, as text.
SELECT * FROM system.zookeeper_log WHERE path LIKE '/clickhouse/tables/%/log/%' LIMIT 5;

-- The per-replica queue.
SELECT replica_name, type, new_part_name, num_tries, num_postponed, last_exception
FROM system.replication_queue
ORDER BY create_time;

-- The deduplication znodes, with the part name they point at.
SELECT name, length(value) FROM system.zookeeper
WHERE path = '/clickhouse/tables/{shard}/default/tbl/deduplication_hashes';

Trade-offs

What the log buys you:

A total order that every replica can read, without a coordinator.
Recoverability. A replica that was down for an hour catches up by reading the log; it does not need to be told what happened.
Human-readable history. The log entries are text, with an explicit format version, which is why a rolling upgrade with mixed versions works and why you can debug a replication problem with system.zookeeper.

What it costs:

ZooKeeper is in the write path. Every insert is at least one round trip to ZooKeeper (the dedup + block-number multi) plus one more (the commit multi that creates the log entry). On a cross-availability-zone deployment, that is milliseconds.
The log grows forever unless something truncates it. The queue is trimmed per replica, but the shared log is the history.
The dedup window is finite. The deduplication znodes are only kept for the last replicated_deduplication_window blocks (10000 by default), so a retry that arrives much later is not deduplicated. This is a correctness-relevant limit, not an optimization.

What the block-number lock buys you:

One round trip for both the duplicate check and the number allocation.
No read-modify-write. The number comes from ZooKeeper’s sequential counter, so two concurrent inserts on two replicas cannot get the same number, and a crashed session cannot leave a gap that is reused.

What it costs:

The block number is not the dedup id. It is tempting to think “the part is named after the block number, so the block number is the duplicate detector”, but that is wrong: the block number is allocated per partition and says nothing about content. The dedup id is a content hash, and it is the hash that makes a retried insert a no-op.
A duplicate is not free. The conflict path is resolved by removing the duplicate rows from the block and rewriting the part, which means a partially-overlapping insert is written twice: once (partially) rejected, once rewritten with the remainder.
The dedup znodes are garbage that has to be collected. They are deleted lazily, when the parts they point at are dropped, and the window setting bounds how far back they go.

The part people get wrong: ReplicatedMergeTree is not exactly-once by itself. It is at-least-once delivery plus a finite-window content-hash deduplication. A client that retries an insert more than replicated_deduplication_window blocks later will duplicate the rows. If you need a stronger guarantee, the dedup id has to come from the application — the insert_deduplication_token setting lets the client supply its own id, which is then used instead of the content hash and is stable across retries forever.

Zookeeper on Opensource internals

ClickHouse Runs Distributed DDL Through a ZooKeeper Task Queue

The queue

The entry node

The versioned entry format

The name padding trick

The host list

The per-host result directory

The active node: a liveness signal

The worker’s side

Exactly-once, sort of

Collecting the result

Retries, and the “zoo” comment

Trade-offs

Wrapping up

ClickHouse Keeper Carries a Tree Digest in Every Raft Log Entry

The hole in Raft

The idea: put the expected state in the log

What the digest actually is

Why a sum?

The check

Snapshots

Determinism

Compression

On-disk layout

Snapshot identity

The log itself

Trade-offs

Wrapping up

ClickHouse Coordinates a Cluster-Wide Backup with ZooKeeper

The shape of the thing

The ZooKeeper layout

Phase 0: claiming the operation

The concurrency check

Phase 1: the metadata consensus loop

The stage barrier itself

The stage list

Phase 2: the file manifest — the heart of it

Phase 3: assigning a global name and index to every file

Phase 4: electing the writer — ZooKeeper create as a mutex

Retries: everything is wrapped

Trade-offs

Wrapping up

How ClickHouse Replicates MergeTree: A ZooKeeper Log and a Hash Deduplication

The log

Copying the log into a queue

Executing out of order

Which pool

The insert path: block number and deduplication in one round trip

The block number

The deduplication check rides along

Resolving the conflict

The whole insert, end to end

The quorum

Looking at it

Trade-offs

The `active` node: a liveness signal

Phase 4: electing the writer — ZooKeeper `create` as a mutex