Backups on Opensource internals

ClickHouse Incremental Backups Store Only the Changed Tail of a File

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: BACKUP ... SETTINGS BASE_BACKUP = ... does not do block-level or chunk-level diffing. It does something much simpler and much more robust: for every file it asks “is the base backup’s copy of this file a prefix of the current copy?” — verified by hashing the first base_size bytes and comparing against the base’s checksum. If yes, the backup records base_size and uploads only the remaining size - base_size bytes. At restore time the two halves are concatenated back into one file.

Take a database with 10 TB of data. Back it up: fine, once. Now back it up again tomorrow, after 10 GB of new data arrived. The obvious implementation uploads 10 TB again. That is not a bug, it is what “backup” means to most systems — a backup is a self-contained snapshot.

But 10 TB of re-upload per day is the kind of cost that makes people stop backing up. So ClickHouse has incremental backups, and the interesting thing about them is how little they actually do.

The naive incremental backup, and why it isn’t used

The textbook approach is a diff algorithm: rsync-style rolling checksum, FastCDC content-defined chunking, or a Merkle tree per part. Split both versions of the file into blocks, find the blocks that match, and upload only the unmatched ones.

ClickHouse does not do this. It does something cruder:

 text
 
 
base: [AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA|]
new: [AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA|BBBBBBBBBB]
 ^ base_size = 30

backup stores: "BBBBBBBB" (10 bytes) + base_size = 30
restore produces: base[0..30) ++ backup[0..10) == new

The base backup’s file must be a prefix of the new file. Not “similar”, not “mostly the same” — literally the first N bytes are identical, and the backup stores the remaining size - N bytes.

That sounds restrictive. And for MergeTree data parts it is: a part is immutable, so a changed part is a completely different file and gets stored in full. But most of what a backup contains is not data parts. It is metadata, columns.txt files, system.users, access files, DDL scripts — and many of those are append-only: a log file that grows, a SQL file that gets a statement appended. For those, the prefix trick is exactly right.

Where the base backup comes from

The base is named in the SETTINGS clause, using the same syntax as the backup destination:

 sql
 
 
BACKUP TABLE events TO Disk('s3', 'backups/daily-2026-10-23')
SETTINGS BASE_BACKUP = Disk('s3', 'backups/full-2026-10-01');

Which is parsed as a regular setting:

 cpp
 
 
// src/Backups/BackupSettings.h:23
 /// Base backup, if it's set an incremental backup will be built. That means only differences made after the base backup will be put
 /// into a new backup.
 std::optional<BackupInfo> base_backup_info;

The base backup is opened lazily and — this is a nice touch — in a reduced mode when the backup is being written. Writing an incremental backup only needs the base’s (size, checksum) list, not its full file metadata, and for a base with millions of files that difference is gigabytes:

 cpp
 
 
// src/Backups/BackupImpl.cpp:433
 BackupFactory::CreateParams base_params = params.getCreateParamsForBaseBackup(std::move(effective_base_backup_info), archive_params.password);
 /// A backup being written only looks up the sizes and checksums of the files of its base, to skip the files the base
 /// already has. For a base of millions of files, the rest of the information about its files takes gigabytes.
 base_params.for_file_lookups_only = (open_mode == OpenMode::WRITE);
 base_backup = BackupFactory::instance().createBackup(base_params);

And the lookup itself is a binary search over a sorted vector of (size, checksum) pairs:

 cpp
 
 
// src/Backups/BackupImpl.cpp:1351
bool BackupImpl::fileExists(const SizeAndChecksum & size_and_checksum) const
{
 if (open_mode == OpenMode::WRITE)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "The backup file should not be opened for writing. Something is wrong internally");

 std::lock_guard lock{mutex};
 if (params.for_file_lookups_only)
 return std::ranges::binary_search(sizes_and_checksums, size_and_checksum);
 return file_infos.contains(size_and_checksum);
}

The (size, checksum) vector is kept deduplicated, because a backup is full of files with identical content (columns.txt of every part of a table is byte-identical):

 cpp
 
 
// src/Backups/BackupImpl.cpp:825
 /// Many files of a backup have the same size and checksum (e.g. `columns.txt` of the parts of a table), so the keys
 /// are deduplicated while they are read, whenever their number doubles since the previous time: this keeps the vector
 /// at about twice the number of distinct keys, however many duplicates there are.
 static constexpr size_t min_num_sizes_and_checksums_to_deduplicate = 65536;
 size_t num_sizes_and_checksums_after_deduplication = 0;
 auto deduplicate_sizes_and_checksums = [&]() TSA_NO_THREAD_SAFETY_ANALYSIS
 {
 std::ranges::sort(sizes_and_checksums);
 sizes_and_checksums.erase(std::ranges::unique(sizes_and_checksums).begin(), sizes_and_checksums.end());
 num_sizes_and_checksums_after_deduplication = sizes_and_checksums.size();
 };

65536 is the floor: below that many entries, deduplication is not worth the sort.

Deciding what to store

Now the core of it. For every file, buildFileInfoForBackupEntry asks three questions, in order.

Question 1: is this file smaller than, or the same size as, the base’s copy?

 cpp
 
 
// src/Backups/BackupFileInfo.cpp:42
 CheckBackupResult checkBaseBackupForFile(const SizeAndChecksum & base_backup_info, const BackupFileInfo & new_entry_info)
 {
 /// We cannot reuse base backup because our file is smaller
 /// than file stored in previous backup
 if ((new_entry_info.size < base_backup_info.first) || !base_backup_info.first)
 return CheckBackupResult::HasNothing;

 if (base_backup_info.first == new_entry_info.size)
 return CheckBackupResult::HasFull;

 return CheckBackupResult::HasPrefix;

 }

Three possible answers: HasNothing (store the whole file), HasFull (the file is unchanged, or at least the same size — the checksum decides later), HasPrefix (the base is shorter, so maybe it’s a prefix).

Question 2: is the base’s copy actually a byte-exact prefix of the new one? This is where the hashing happens:

 cpp
 
 
// src/Backups/BackupFileInfo.cpp:155
 /// File with the same name but smaller size exist in previous backup
 if (check_base == CheckBackupResult::HasPrefix)
 {
 auto checksums = calculateNewEntryChecksumsIfNeeded(backup_entry, base_backup_file_info->first, read_settings);
 info.checksum = checksums.full_checksum;

 /// We have prefix of this file in backup with the same checksum.
 /// In ClickHouse this can happen for StorageLog for example,
 /// or for .sql files that store DDL if only last part of the query was changed.
 if (checksums.prefix_checksum == base_backup_file_info->second)
 {
 LOG_TRACE(log, "Found prefix of file {} in the base backup, will write rest of the file to current backup", adjusted_path);
 info.base_size = base_backup_file_info->first;
 info.base_checksum = base_backup_file_info->second;
 }
 else
 {
 LOG_TRACE(log, "Prefix of file {} doesn't match the file in the base backup", adjusted_path);
 }
 }

The comparison is prefix_checksum == base_backup_file_info->second. The right-hand side is the base backup’s checksum for that file; the left-hand side is the checksum of the first base_size bytes of the current file. If they are equal, the base’s bytes are a prefix of the current bytes.

The comment names the two real-world cases: StorageLog (a table that only ever gets appended to) and .sql files that store DDL history (a file that gets a statement appended).

Question 3: what if the file was renamed? The prefix check above is keyed on the path. If a file moved, the path lookup fails — but the content may still be there under a different name:

 cpp
 
 
// src/Backups/BackupFileInfo.cpp:206
 /// We don't have info about this file_name (sic!) in base backup,
 /// however file could be renamed, so we will check one more time using size and checksum
 if (base_backup && base_backup->fileExists(std::pair{info.size, info.checksum}))
 {
 LOG_TRACE(log, "Found a file in the base backup with the same size and checksum as {}", adjusted_path);
 info.base_size = info.size;
 info.base_checksum = info.checksum;
 }

Note info.base_size = info.size: the whole file comes from the base, and nothing at all is written for it. That is why the metadata distinguishes “base covers the whole file” from “base covers a prefix”.

Why partial checksums are only allowed for append-only files

There is a subtlety in the checksum computation that is easy to miss and expensive to get wrong. The prefix checksum is a hash of the first base_size bytes. But hashing a prefix of a file only makes sense if that prefix is stable — i.e. if the file’s earlier bytes cannot change under you.

ClickHouse encodes exactly that distinction:

 cpp
 
 
// src/Backups/BackupEntryWithChecksumCalculation.h:23
 virtual bool isPartialChecksumAllowed() const = 0;

 cpp
 
 
// src/Backups/BackupEntryFromImmutableFile.h:42
 bool isPartialChecksumAllowed() const override { return false; }

 cpp
 
 
// src/Backups/BackupEntryFromAppendOnlyFile.h:41
 bool isPartialChecksumAllowed() const override { return true; }

So MergeTree data parts (immutable) never get a partial checksum; log tables and DDL files (append-only) do. If a partial checksum is not allowed, the computation refuses to produce one:

 cpp
 
 
// src/Backups/BackupEntryWithChecksumCalculation.cpp:226
 /// Calculates a hash of the file contents at offsets [0..limit_).
 auto calculate_hash = [&](UInt64 limit_) -> std::optional<UInt128>
 {
 if ((limit_ != size) && !isPartialChecksumAllowed())
 return {};

and, symmetrically, refuses to accept a file that grew while it was being hashed:

 cpp
 
 
// src/Backups/BackupEntryWithChecksumCalculation.cpp:250
 if ((limit_ == size) && !isPartialChecksumAllowed() && !hashing_read_buffer->eof())
 {
 /// We reached offset `size` which is expected to be the end of the file but the file surprisingly didn't end there.
 /// Perhaps it was appended after we got its size for backup but that's not allowed for immutable files
 /// (function `isPartialChecksumAllowed()` returns false for immutable files).
 throw Exception(ErrorCodes::EXPECTED_END_OF_FILE,
 "Size of immutable file {} increased unexpectedly while calculating its checksum",
 getFilePath());
 }

This is the correctness backbone of the whole scheme. If a file is not append-only, its prefix might change between the moment the backup computed the prefix checksum and the moment it wrote the suffix — and the restored file would be a chimera of two different versions of the same file. The isPartialChecksumAllowed() flag makes that impossible by construction.

There is one more wrinkle: the prefix checksum must be computed before the full checksum, because computing both in one pass over the file is cheaper than two passes:

 cpp
 
 
// src/Backups/BackupFileInfo.cpp:67
 ChecksumsForNewEntry calculateNewEntryChecksumsIfNeeded(const BackupEntryPtr & entry, UInt64 prefix_size, const ReadSettings & read_settings)
 {
 ChecksumsForNewEntry res;
 /// The partial checksum should be calculated before the full checksum to enable optimization in BackupEntryWithChecksumCalculation.
 res.prefix_checksum = entry->getPartialChecksum(prefix_size, read_settings);
 res.full_checksum = entry->getChecksum(read_settings);
 return res;
 }

and getPartialChecksum remembers the full checksum it computed along the way, so the second pass is free:

 cpp
 
 
// src/Backups/BackupEntryWithChecksumCalculation.cpp:55
std::optional<UInt128> BackupEntryWithChecksumCalculation::getPartialChecksum(UInt64 limit, const ReadSettings & read_settings) const
{
 UInt64 size = getSize();
 if (limit >= size)
 return getChecksum(read_settings);

 bool has_calculated_full_checksum = false;

 {
 std::lock_guard lock{mutex};
 has_calculated_full_checksum = calculated_checksum.has_value();
 }

 if (has_calculated_full_checksum)
 return calculateChecksum(limit, read_settings);

 auto [partial_checksum, full_checksum] = calculateOneOrTwoChecksums(limit, size, read_settings);
 chassert(full_checksum);

 {
 std::lock_guard lock{mutex};
 calculated_checksum = full_checksum.value();
 }

 return partial_checksum;
}

Writing only the suffix

Once the plan is known, the write side is trivial: skip the first base_size bytes.

 cpp
 
 
// src/Backups/BackupImpl.cpp:1759
 if (src_disk && from_immutable_file)
 {
 LOG_TRACE(log, "Writing backup for file {} from {} (disk {}): data file #{}", info.data_file_name, src_file_desc, src_disk->getName(), info.data_file_index);
 writer->copyFileFromDisk(info.data_file_name, src_disk, src_file_path, info.encrypted_by_disk, info.base_size, info.size - info.base_size);
 }
 else
 {
 LOG_TRACE(log, "Writing backup for file {} from {}: data file #{}", info.data_file_name, src_file_desc, info.data_file_index);
 auto create_read_buffer = [entry, read_settings = writer->getReadSettings()] { return entry->getReadBuffer(read_settings); };
 writer->copyDataToFile(info.data_file_name, create_read_buffer, info.base_size, info.size - info.base_size);
 }

And copyDataToFile does the seek:

 cpp
 
 
// src/Backups/BackupIO_Default.cpp:189
void BackupWriterDefault::copyDataToFile(const String & path_in_backup, const CreateReadBufferFunction & create_read_buffer, UInt64 start_pos, UInt64 length)
{
 auto read_buffer = create_read_buffer();

 if (start_pos)
 read_buffer->seek(start_pos, SEEK_SET);

 auto write_buffer = writeFile(path_in_backup);

 copyData(*read_buffer, *write_buffer, length);
 write_buffer->finalize();
}

Note the parameter names: start_pos and length. The “delta” is a byte range. There is no diff algorithm anywhere — the whole incremental backup machinery reduces to “upload bytes [base_size, size)”.

The accounting follows: the backup’s reported size is only the delta.

 cpp
 
 
// src/Backups/BackupImpl.cpp:1784
 {
 std::lock_guard lock{mutex};
 ++num_entries;
 size_of_entries += info.size - info.base_size;
 uncompressed_size += info.size - info.base_size;
 }

which is why getTotalSize() (the logical size of all files) and getSizeOfEntries() (the bytes actually stored) differ for an incremental backup.

The manifest

The manifest (.backup, an XML file inside the archive) records the base relationship per file:

 cpp
 
 
// src/Backups/BackupImpl.cpp:649
 if (info.size)
 {
 *out << "<checksum>" << hexChecksum(info.checksum) << "</checksum>";
 if (info.base_size)
 {
 *out << "<use_base>true</use_base>";
 if (info.base_size != info.size)
 {
 *out << "<base_size>" << info.base_size << "</base_size>";
 *out << "<base_checksum>" << hexChecksum(info.base_checksum) << "</base_checksum>";
 }
 }
 if (!info.data_file_name.empty() && (info.data_file_name != info.file_name))
 *out << "<data_file>" << xml << xml_string("data_file", info.data_file_name) << "</data_file>";
 if (info.encrypted_by_disk)
 *out << "<encrypted_by_disk>true</encrypted_by_disk>";
 }

Three tags, and the third one is optional by design: when base_size == size, the whole file comes from the base and the manifest does not repeat the size — it is implied by <use_base>true</use_base> and the file’s own <size>.

The read side mirrors it:

 cpp
 
 
// src/Backups/BackupImpl.cpp:870
 bool use_base = get_bool("use_base", false);
 auto base_size_it = f.find("base_size");
 info.base_size = (base_size_it != f.end()) ? to_uint64(base_size_it->second, "base_size") : (use_base ? info.size : 0);
 if (info.base_size)
 use_base = true;

 if (info.base_size > info.size)
 {
 throw Exception(
 ErrorCodes::BACKUP_DAMAGED,
 "Backup {}: Base size must not be greater than the size of entry {}",
 backup_name_for_logging,
 quoteString(info.file_name));
 }

 if (use_base)
 {
 if (info.base_size == info.size)
 info.base_checksum = info.checksum;
 else
 info.base_checksum = unhexChecksum(req("base_checksum"));
 }

Note the validation: base_size > size is a BACKUP_DAMAGED error. A corrupted manifest cannot make the restore read past the end of a file.

Restoring: stitching the halves back together

Here is the payoff. When the restore reads a file, it may need two sources: the base backup’s prefix and this backup’s suffix.

 cpp
 
 
// src/Backups/BackupImpl.cpp:1461
 if (info.size > info.base_size)
 {
 /// Make `read_buffer` if there is data for this backup entry in this backup.
 if (use_archive)
 read_buffer = archive_reader->readFile(info.data_file_name, /*throw_on_not_found=*/true);
 else
 read_buffer = reader->readFile(info.data_file_name);
 }

 if (info.base_size)
 {
 /// Make `base_read_buffer` if there is data for this backup entry in the base backup.
 auto base = getBaseBackup();
 if (!base)
 {
 throw Exception(
 ErrorCodes::NO_BASE_BACKUP,
 "Backup {}: Entry {} is marked to be read from a base backup, but there is no base backup specified",
 backup_name_for_logging, formatSizeAndChecksum(size_and_checksum));
 }

 if (!base->fileExists(std::pair(info.base_size, info.base_checksum)))
 {
 throw Exception(
 ErrorCodes::WRONG_BASE_BACKUP,
 "Backup {}: Entry {} is marked to be read from a base backup, but doesn't exist there",
 backup_name_for_logging, formatSizeAndChecksum(size_and_checksum));
 }

 base_read_buffer = base->readFile(info.file_name, std::pair{info.base_size, info.base_checksum});
 }

Two important details. First, the base is opened by the same name — the file is looked up in the base under the current file’s name, not under a recorded path. The base backup is expected to have the same layout.

Second, the base lookup is by (base_size, base_checksum), not by name. If the base backup no longer contains that exact content — because it was replaced, or because the wrong base was specified — the restore fails loudly with WRONG_BASE_BACKUP instead of silently producing a corrupt file.

And the final stitch:

 cpp
 
 
// src/Backups/BackupImpl.cpp:1500
 if (!info.base_size)
 {
 /// Data comes completely from this backup, the base backup isn't used.
 return read_buffer;
 }
 if (info.size == info.base_size)
 {
 /// Data comes completely from the base backup (nothing comes from this backup).
 return base_read_buffer;
 }

 /// The beginning of the data comes from the base backup,
 /// and the ending comes from this backup.
 return std::make_unique<ConcatReadBufferFromFile>(
 info.data_file_name, std::move(base_read_buffer), info.base_size, std::move(read_buffer), info.size - info.base_size);

ConcatReadBufferFromFile is a ReadBufferFromFileBase that presents two buffers as one contiguous file. It is a real seekable buffer, not a one-shot stream — seek() walks the list of buffers to find the right one:

 cpp
 
 
// src/IO/ConcatReadBufferFromFile.cpp:141
 off_t change_position = new_position - current_position;
 if ((working_buffer.begin() <= pos + change_position) && (pos + change_position <= working_buffer.end()))
 {
 /// Position is still inside the same working buffer.
 pos += change_position;
 chassert(pos >= working_buffer.begin());
 chassert(pos <= working_buffer.end());
 return new_position;
 }

 while (new_position < static_cast<off_t>(current_start_pos))
 current_start_pos -= buffers[--current].size;

 while (new_position >= static_cast<off_t>(current_start_pos + buffers[current].size))
 current_start_pos += buffers[current++].size;

 buffers[current].in->seek(new_position - current_start_pos, SEEK_SET);

That matters because a restore does not simply stream a file to a disk — it may read parts of it, checksum it, and copy it. The concatenated buffer has to behave like a normal file.

The disk-copy fast path

When the file needs no suffix (nothing new) the copy is forwarded to the base backup directly, so the base’s own sync and encryption handling are honoured:

 cpp
 
 
// src/Backups/BackupImpl.cpp:1667
 else if (info.size && (info.size == info.base_size))
 {
 /// Data comes completely from the base backup (nothing comes from this backup). The base
 /// backup is itself a BackupImpl that honours `sync` and can read its own encrypted-by-disk
 /// entries, so forward the copy (and the `sync` request) there. Going through the generic
 /// branch below instead would read the base via the public readFile(), which always requests
 /// unencrypted data and would fail on an encrypted entry (CANNOT_RESTORE_TO_NONENCRYPTED_DISK).
 getBaseBackup()->copyFileToDisk(std::pair{info.base_size, info.base_checksum}, destination_disk, destination_path, write_mode, sync);
 file_copied = true;
 }

The prefix trick is only one of the two ways an incremental backup saves space. The other is deduplication by content, and it applies within a single backup as well as across backups: two files with the same (size, checksum) are stored once.

That is handled in the coordination layer, not the file layer:

 cpp
 
 
// src/Backups/BackupCoordinationFileInfos.cpp:186
 SizeAndChecksum size_and_checksum{info.size, info.checksum};
 auto [it, inserted] = data_file_index_by_checksum.emplace(size_and_checksum, i);
 if (inserted)
 {
 /// Found a new file.
 info.data_file_name = getBackupDataFileName(info, config.data_file_name_generator, config.data_file_name_prefix_length);
 info.data_file_index = i;
 ++num_files;
 total_size_of_files += info.size - info.base_size;
 }
 else
 {
 /// Found a file with the same size and checksum as some file before, reuse old `data_file_index` and `data_file_name`.
 info.data_file_index = it->second;
 info.data_file_name = file_infos_for_all_hosts[it->second]->data_file_name;
 }

A file whose size equals its base_size gets no data file at all:

 cpp
 
 
// src/Backups/BackupCoordinationFileInfos.cpp:178
 if (info.size == info.base_size)
 {
 /// A file is either empty or can be get from the base backup as a whole.
 info.data_file_name.clear();
 info.data_file_index = static_cast<size_t>(-1);
 }

An empty data_file_name means “there is no physical file for this entry”. That is also how empty files are handled — they have size == 0 and hence base_size == 0, so they fall into the same branch.

Trade-offs

What the design buys:

Dramatically smaller incremental backups for append-only workloads. A StorageLog table or a DDL-history file that grew by 1 KB stores 1 KB.
No diff algorithm, no chunk index, no content-defined chunking. The restore path is a seek and a concat; there is no library to get wrong.
Content-addressed safety. The prefix is verified by hash, not assumed. A base that does not match produces WRONG_BASE_BACKUP, not a corrupt restore.
The base is opened in a reduced mode. Looking up millions of files in the base costs a sorted (size, checksum) vector, not a full metadata parse.

What it costs:

The prefix must be byte-exact. A file that was rewritten in place — even if 99.9% of it is unchanged — is stored in full. There is no content-defined chunking to rescue the middle of the file.
The backup must read the file to compute the checksum. Even when the file is later skipped entirely, the checksum pass has already read it. This is the dominant cost of an incremental backup on a local disk: you pay a full read of every file to decide how much of it to write.
Immutable files never get the prefix treatment. A MergeTree part that was merged or rewritten is a new file, stored in full. So the incremental win is concentrated in the metadata, not in the data.
The restore needs the base. An incremental backup is not self-contained; losing the base means losing the backup. The manifest names the base, and RESTORE will open it, but the operational burden is on the user.
The base must keep the same file names. The restore looks the file up in the base under the current name. Rename a file between the base and the incremental backup and the prefix is lost (the file gets stored in full — safe, but wasteful).

Wrapping up

The one thing to remember: incremental backups in ClickHouse are not diffs. They are byte-range references. A backup entry is either “the whole file” or “bytes [N, size) of this file, plus bytes [0, N) from the base”.

The part people get wrong: the “prefix” is not a heuristic — it is verified with a hash, and the hash is only computed for files that are guaranteed append-only. isPartialChecksumAllowed() is the guard rail that makes the scheme safe, and it is the first thing to look at if you ever see an incremental backup produce a corrupt file.

What to go do about it: run a BACKUP ... SETTINGS BASE_BACKUP = ... twice in a row over a table with a StorageLog and compare the total_size and compressed_size columns of system.backups for the two runs. If the second run’s compressed_size is a small fraction of the first, the prefix path is working. If the two are about the same, your data is all in immutable MergeTree parts, and the prefix trick has nothing to work with — at which point you are paying the checksum read for nothing and should think about whether the base backup is worth it.

ClickHouse Coordinates a Cluster-Wide Backup with ZooKeeper

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: BACKUP ON CLUSTER is not a fan-out of independent backups — it is one logical backup with a shared manifest. Each host publishes a file info record in ZooKeeper, every host reads all of them, sorts them by name, and assigns each file a global data file index. Who writes which file is decided by a create on /writing_files/<index> — a ZooKeeper node creation is the mutex. Between phases, every host parks at a named stage and the group does not move on until every host has arrived.

Backing up a cluster looks like a solved thing until you try to solve it. The naive implementation is: send BACKUP to every host, let each one dump its own local data, call it a day. That works until you want the result to be a backup rather than N backups — one manifest, one file namespace, restorable as a unit.

The trouble is that a MergeTree cluster is not N independent databases. A replicated table has the same logical data on every replica; a Replicated database replicates its DDL history; system.users on one host is not the same as on another. If each host writes its own files, you get N copies of everything and no way to tell which copy is the real one.

ClickHouse’s answer is to make the backup global. There is exactly one manifest, exactly one file namespace, and a rule that decides who writes what. That rule is implemented with ZooKeeper primitives, and it is much cleverer than it first looks.

The shape of the thing

First, the trick that makes the whole thing possible: BACKUP ON CLUSTER reuses the distributed DDL machinery to ship the query. There is no separate “backup cluster protocol”.

 cpp
 
 
// src/Backups/BackupsWorker.cpp:1255
void BackupsWorker::sendQueryToOtherHosts(const ASTBackupQuery & backup_or_restore_query, const ClusterPtr & cluster,
 size_t only_shard_num, size_t only_replica_num, ContextMutablePtr context, const AccessRightsElements & access_to_check,
 const ZooKeeperRetriesInfo & retries_info) const
{
 chassert(cluster);

 DDLQueryOnClusterParams params;
 params.cluster = cluster;
 params.only_shard_num = only_shard_num;
 params.only_replica_num = only_replica_num;
 params.access_to_check = access_to_check;
 params.retries_info = retries_info;

 context->setSetting("distributed_ddl_task_timeout", Field{0});
 context->setSetting("distributed_ddl_output_mode", Field{"never_throw"});

 // executeDDLQueryOnCluster() will return without waiting for completion
 executeDDLQueryOnCluster(backup_or_restore_query.clone(), context, params);

The initiator does not run the backup itself. It ships the query, sets distributed_ddl_task_timeout to zero so it does not wait, and then watches ZooKeeper for the result of the internal per-host operations. That is why the backup coordinator has two roles: the initiator has an empty host name (kInitiator), every other host has its own host_id.

 cpp
 
 
// src/Backups/BackupsWorker.cpp:1286
 bool is_internal_backup = backup_settings.internal;

 String root_zk_path = context->getConfigRef().getString("backups.zookeeper_path", "/clickhouse/backups");
 auto get_zookeeper = [global_context = context->getGlobalContext()] { return global_context->getZooKeeper(); };
 auto keeper_settings = BackupKeeperSettings(context);

 auto all_hosts = BackupSettings::Util::filterHostIDs(
 backup_settings.cluster_host_ids, backup_settings.shard_num, backup_settings.replica_num);
 all_hosts.emplace_back(BackupCoordinationOnCluster::kInitiator);

 String current_host = is_internal_backup ? backup_settings.host_id : String{BackupCoordinationOnCluster::kInitiator};

Note all_hosts.emplace_back(BackupCoordinationOnCluster::kInitiator): the initiator is in the list as a pseudo-host, even though it does no work. That matters for the barrier — other hosts wait on the initiator’s stage node too.

The ZooKeeper layout

Everything for one backup lives under one subtree. The paths are constructed once, in the constructor, and every node name is derived from them:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:126
 , zookeeper_path(zookeeper_path_)
 , root_zookeeper_path(zookeeper_path.parent_path().parent_path())
 , operation_zookeeper_path(zookeeper_path.parent_path())
 , operation_node_name(zookeeper_path.parent_path().filename())
 , start_node_path(zookeeper_path / ("started|" + current_host))
 , finish_node_path(zookeeper_path / ("finished|" + current_host))
 , initiator_start_node_path(zookeeper_path / ("started|" + String{kInitiator}))
 , num_hosts_node_path(zookeeper_path / "num_hosts")
 , error_node_path(zookeeper_path / "error")
 , alive_node_path(zookeeper_path / ("alive|" + current_host))
 , alive_tracker_node_path(fs::path{root_zookeeper_path} / "alive_tracker")

With backups.zookeeper_path = /clickhouse/backups and a backup UUID:

 text
 
 
/clickhouse/backups/
├── alive_tracker (persistent; "someone is alive" version counter)
└── backup-<uuid>/
 ├── stage/
 │ ├── started|<host> (persistent; host has begun)
 │ ├── started| (persistent; the initiator)
 │ ├── finished|<host> (persistent; host is done)
 │ ├── alive|<host> (EPHEMERAL; host is alive)
 │ ├── num_hosts (persistent; counter)
 │ ├── error (persistent; someone failed)
 │ ├── current|<host>|gathering metadata (0)
 │ ├── current|<host>|writing backup (0)
 │ └── current|<host>|completed (0)
 ├── file_infos/<host>/000000 (persistent; the file manifest, chunked)
 ├── writing_files/17 (persistent; "host #17 owns data file 17")
 ├── repl_part_names/… repl_mutations/… repl_access/…
 ├── repl_sql_objects/… repl_workload_entities/…
 └── keeper_map_tables/… rocksdb_tables/…

Three things to notice:

alive|<host> is ephemeral — it disappears automatically when the host’s ZooKeeper session ends. That is the liveness signal, and it is why a crashed host does not deadlock the whole backup.
current|<host>|<stage> encodes both the stage and a small result payload. For most stages the result is empty; for metadata gathering it is "consistent" or "need_another_attempt". The stage barrier is thus a general barrier that also carries a vote.
file_infos/<host> holds the per-host manifest, chunked because ZooKeeper nodes have a size limit.

Phase 0: claiming the operation

Before any work happens, a host must announce itself and check that it is allowed to start at all. This happens in one ZooKeeper multi request, which matters: the whole claim must be atomic, or two hosts could both believe they own the backup.

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:304
 Coordination::Requests requests;
 requests.reserve(6);

 size_t operation_node_pos = static_cast<size_t>(-1);
 if (!zookeeper->exists(operation_zookeeper_path))
 {
 operation_node_pos = requests.size();
 requests.emplace_back(zkutil::makeCreateRequest(operation_zookeeper_path, "", zkutil::CreateMode::Persistent));
 }

 size_t zookeeper_path_pos = static_cast<size_t>(-1);
 if (!zookeeper->exists(zookeeper_path))
 {
 zookeeper_path_pos = requests.size();
 requests.emplace_back(zkutil::makeCreateRequest(zookeeper_path, "", zkutil::CreateMode::Persistent));
 }

 size_t num_hosts_node_pos = requests.size();
 if (num_hosts)
 requests.emplace_back(zkutil::makeSetRequest(num_hosts_node_path, toString(*num_hosts + 1), num_hosts_version));
 else
 requests.emplace_back(zkutil::makeCreateRequest(num_hosts_node_path, "1", zkutil::CreateMode::Persistent));

 size_t alive_tracker_node_pos = requests.size();
 requests.emplace_back(zkutil::makeSetRequest(alive_tracker_node_path, "", alive_tracker_version));

 requests.emplace_back(zkutil::makeCreateRequest(start_node_path, std::to_string(kCurrentVersion), zkutil::CreateMode::Persistent));
 requests.emplace_back(zkutil::makeCreateRequest(alive_node_path, "", zkutil::CreateMode::Ephemeral));

 Coordination::Responses responses;
 auto code = zookeeper->tryMulti(requests, responses);

Six operations, one transaction: create the operation node, create the stage node, bump the num_hosts counter with a versioned set, bump the alive_tracker version, create my started|<host> marker, create my ephemeral alive|<host> node.

The two sets are versioned deliberately. The version acts as an optimistic-concurrency token, and a ZBADVERSION on either means “somebody else changed the world under me, start over”:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:372
 else if ((num_hosts_node_pos < responses.size()) && num_hosts &&
 (responses[num_hosts_node_pos]->error == Coordination::Error::ZBADVERSION))
 {
 show_error_before_next_attempt(fmt::format("The version of node {} changed", num_hosts_node_path));
 num_hosts.reset(); /// needs to reread 'num_hosts' again
 }
 else if ((alive_tracker_node_pos < responses.size()) &&
 (responses[alive_tracker_node_pos]->error == Coordination::Error::ZBADVERSION))
 {
 show_error_before_next_attempt(fmt::format("The version of node {} changed", alive_tracker_node_path));
 check_concurrency = true; /// needs to recheck for concurrency again
 }

Read that second branch carefully: a failed alive_tracker bump forces a re-check of concurrency. That is the whole concurrency protocol in one line. The alive_tracker version is a cheap “has anything changed underneath me” flag; if it moved, the world may now contain another running backup.

The concurrency check

allow_concurrent_backups defaults to true, but the check exists for when it is turned off. “Another operation is running” is defined precisely as: another backup-<uuid> directory exists whose stage/ contains a node starting with alive.

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:408
 for (const String & found_operation : found_operations)
 {
 if (found_operation.starts_with(is_restore ? "restore-" : "backup-") && (found_operation != operation_node_name))
 {
 Strings stages;
 code = zookeeper->tryGetChildren(fs::path{root_zookeeper_path} / found_operation / "stage", stages);

 if (!((code == Coordination::Error::ZOK) || (code == Coordination::Error::ZNONODE)))
 throw zkutil::KeeperException::fromPath(code, fs::path{root_zookeeper_path} / found_operation / "stage");

 if (code == Coordination::Error::ZOK)
 {
 for (const String & stage : stages)
 {
 if (stage.starts_with("alive"))
 BackupConcurrencyCheck::throwConcurrentOperationNotAllowed(is_restore);
 }
 }
 }
 }

Because alive nodes are ephemeral, a dead host’s backup does not block the next one.

There is also a purely local concurrency check, in the server process rather than in ZooKeeper:

 cpp
 
 
// src/Backups/BackupConcurrencyCheck.cpp:44
 else
 {
 size_t num_local_backups = counters.local_backups;
 size_t num_on_cluster_backups = counters.on_cluster_backups.size();
 if (on_cluster)
 {
 if (!counters.on_cluster_backups.contains(zookeeper_path))
 ++num_on_cluster_backups;
 }
 else
 {
 ++num_local_backups;
 }
 found_concurrent_operation = (num_local_backups + num_on_cluster_backups > 1);
 }

The counters are reference-counted per ZooKeeper path, incremented in the constructor and decremented in the destructor — a scoped lock. Keying by path means two BACKUP ON CLUSTER operations against different clusters do not collide, but two against the same path do.

Phase 1: the metadata consensus loop

Here is the part that is genuinely unusual. Before copying any data, every host independently collects the list of databases and tables it will back up. If a table is renamed mid-collection, one host may see the old name and another the new one, and the resulting backup would be garbage.

ClickHouse’s answer is not a lock — it is a convergent retry loop with exponential backoff and a global agreement check. The loop is documented in a comment that reads like a design document:

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:237
void BackupEntriesCollector::gatherMetadataAndCheckConsistency()
{
 /// With the default values the metadata collecting works in the following way:
 /// 1) it tries to collect the metadata for the first time; then
 /// 2) it tries to collect it again, and compares the results from the first and the second collecting; if they match, it's done; otherwise
 /// 3) it sleeps 100 millisecond and tries to collect again, then compares the results from the second and the third collecting; if they match, it's done; otherwise
 /// 4) it tries to collect again, then compares the results from the third and the fourth collecting; if they match, it's done; otherwise
 /// 5) it sleeps 200 milliseconds and tries to collect again, then compares the results from the fourth and the fifth collecting; if they match, it's done;
 /// ...
 /// and so on, the sleep time is doubled each time until it reaches 5000 milliseconds.
 /// And such attempts will be continued until 600000 milliseconds pass.

The numbers come from the server config: backups.attempts_to_collect_metadata_before_sleep (default 2), backups.min_sleep_before_next_attempt_to_collect_metadata (default 100 ms), backups.max_sleep_before_next_attempt_to_collect_metadata (default 5000 ms), backups.collect_metadata_timeout (default 600000 ms). The backoff is a clean power-of-two ramp, capped at 2^10:

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:91
 /// How long we should sleep after finding an inconsistency error.
 std::chrono::milliseconds getSleepTimeAfterInconsistencyError(int attempt_no, unsigned int attempts_before_sleep, std::chrono::milliseconds min_sleep, std::chrono::milliseconds max_sleep)
 {
 attempts_before_sleep = std::max(attempts_before_sleep, 1U);
 if (((attempt_no + 1) % attempts_before_sleep) != 0)
 return std::chrono::milliseconds::zero(); /// no sleep

 int sleep_counter = attempt_no / attempts_before_sleep;
 std::chrono::milliseconds sleep_time = intExp2(std::min(sleep_counter, 10)) * min_sleep;
 return std::min(sleep_time, max_sleep);
 }

Two kinds of consistency are checked. The local one: collect twice in a row and compare. A table renamed between the two passes shows up as a mismatch.

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:312
 if (!compare_collected_metadata)
 return;

 /// We have to check consistency of collected information to protect from the case when some table or database is
 /// renamed during this collecting making the collected information invalid.
 String mismatch_description;
 if (!compareWithPrevious(mismatch_description))
 {
 /// Usually two passes is minimum.
 /// (Because we need to compare with table names from the previous pass to be sure we are not going to miss anything).
 if (attempt_no >= 1)
 inconsistency_error = Exception{ErrorCodes::INCONSISTENT_METADATA_FOR_BACKUP, "{}", mismatch_description};
 need_another_attempt = true;
 }

And the global one: every host publishes its verdict as the stage result and waits for everybody else’s verdict. The stage barrier does double duty as a consensus mechanism:

 cpp
 
 
// src/Backups/BackupEntriesCollector.cpp:340
 /// We've gathered metadata, cool! But we have to check that other hosts coped with that too.
 auto all_hosts_results = setStage(next_stage, need_another_attempt ? "need_another_attempt" : "consistent");
 size_t count = std::min(all_hosts.size(), all_hosts_results.size());

 for (size_t i = 0; i != count; ++i)
 {
 const auto & other_host = all_hosts[i];
 const auto & other_host_result = all_hosts_results[i];
 if ((other_host_result != "consistent") && (other_host_result != "need_another_attempt"))
 {
 LOG_WARNING(log, "Found inconsistency on host {}: {}", other_host, other_host_result);
 inconsistency_error = Exception{ErrorCodes::INCONSISTENT_METADATA_FOR_BACKUP, "Found inconsistency on host {}: {}", other_host, other_host_result};
 need_another_attempt = true;
 return;
 }
 }

The stage name is parameterised by the attempt number — "gathering metadata (0)", "gathering metadata (1)", and so on — because the barrier is per attempt. Every host must reach attempt N before anyone proceeds to attempt N+1.

Note the subtlety: a host that failed to collect metadata publishes its error text as the stage result and returns immediately. The other hosts, on reading that result, see a string that is neither "consistent" nor "need_another_attempt", and they treat it as a failure and start their own next attempt.

The stage barrier itself

The barrier is a condition variable fed by a background polling thread. There is no watch on the stage nodes — the implementation polls.

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:504
 try
 {
 /// Recreate the 'alive' node if necessary and read a new state from ZooKeeper.
 auto holder = with_retries.createRetriesControlHolder("BackupCoordinationStageSync::watchingThread");
 auto & zookeeper = holder.faulty_zookeeper;
 with_retries.renewZooKeeper(zookeeper);

 if (should_stop())
 return;

 /// Recreate the 'alive' node if it was removed.
 createAliveNode(zookeeper);

 /// Reads the current state from nodes in ZooKeeper.
 readCurrentState(zookeeper);
 }

The alive node must be recreated on every session loss — an ephemeral node is tied to the session, so a reconnect means a new session means the node is gone. The recreation is a two-op multi (create the node, bump the alive_tracker version), which is what makes the tracker version a reliable “something happened” flag:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:547
void BackupCoordinationStageSync::createAliveNode(Coordination::ZooKeeperWithFaultInjection::Ptr zookeeper)
{
 if (zookeeper->exists(alive_node_path))
 return;

 Coordination::Requests requests;
 requests.emplace_back(zkutil::makeCreateRequest(alive_node_path, "", zkutil::CreateMode::Ephemeral));
 requests.emplace_back(zkutil::makeSetRequest(alive_tracker_node_path, "", -1));
 zookeeper->multi(requests);

 LOG_INFO(log, "The alive node was recreated for {}", current_host_desc);
}

The polling period is backups.sync_period_ms, default 5000 ms. So the worst-case latency for a stage barrier is five seconds. That is a deliberate trade: polling is simple and survives any kind of ZooKeeper weirdness, where a watch would need careful re-arming.

Publishing a stage is a plain create of a persistent node whose name encodes the stage, with the result as its payload:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:869
void BackupCoordinationStageSync::createStageNode(const String & stage, const String & stage_result, Coordination::ZooKeeperWithFaultInjection::Ptr zookeeper)
{
 if (isErrorSet())
 rethrowSetError();

 String serialized_error;
 if (zookeeper->tryGet(error_node_path, serialized_error))
 {
 auto [exception, host] = parseErrorNode(serialized_error);
 if (exception)
 {
 std::lock_guard lock{mutex};
 state.addErrorInfo(exception, host);
 std::rethrow_exception(exception);
 }
 }

 auto code = zookeeper->tryCreate(getStageNodePath(stage), stage_result, zkutil::CreateMode::Persistent);
 if (code == Coordination::Error::ZOK)
 {
 std::lock_guard lock{mutex};
 state.hosts.at(current_host).stages[stage] = stage_result;
 return;
 }

 if (code == Coordination::Error::ZNODEEXISTS)
 {
 String another_result = zookeeper->get(getStageNodePath(stage));
 std::lock_guard lock{mutex};
 state.hosts.at(current_host).stages[stage] = another_result;
 return;
 }

 throw zkutil::KeeperException::fromPath(code, getStageNodePath(stage));
}

ZNODEEXISTS is treated as success — the host already published this stage (possibly in a previous attempt that appeared to fail). Idempotence by tolerance.

And the waiting side:

 cpp
 
 
// src/Backups/BackupCoordinationStageSync.cpp:945
 const HostInfo & host_info = it->second;
 auto stage_it = host_info.stages.find(stage_to_wait);
 if (stage_it != host_info.stages.end())
 {
 results[i] = stage_it->second;
 continue;
 }

 if (host_info.finished)
 {
 if (stage_to_wait == BackupCoordinationStage::FINALIZING_TABLES)
 {
 /// This is a newly added stage. For compatibility with older server versions,
 /// allow other replicas to skip this stage. This doesn't break anything: this stage
 /// unpauses refreshable materialized views, but older server versions don't pause
 /// them in the first place.
 results[i] = "";
 continue;
 }
 throw Exception(ErrorCodes::FAILED_TO_SYNC_BACKUP_OR_RESTORE,
 "{} finished without coming to stage {}", getHostDesc(host), stage_to_wait);
 }

The FINALIZING_TABLES special case is a lovely piece of pragmatism: a stage was added to the protocol later, and an older server that has already finished will never reach it. Rather than deadlock the cluster, the barrier treats “finished” as “passed every stage” for that one stage only.

The stage list

The stage names are constants in one header (src/Backups/BackupCoordinationStage.h), which makes the protocol readable at a glance:

 cpp
 
 
// src/Backups/BackupCoordinationStage.h:9
namespace BackupCoordinationStage
{
 /// Finding all tables and databases which we're going to put to the backup and collecting their metadata.
 constexpr const char * GATHERING_METADATA = "gathering metadata";

 String formatGatheringMetadata(int attempt_no);

 /// Making temporary hard links and prepare backup entries.
 constexpr const char * EXTRACTING_DATA_FROM_TABLES = "extracting data from tables";

 /// Running special tasks for replicated tables which can also prepare some backup entries.
 constexpr const char * RUNNING_POST_TASKS = "running post-tasks";

 /// Building information about all files which will be written to a backup.
 constexpr const char * BUILDING_FILE_INFOS = "building file infos";

 /// Writing backup entries to the backup and removing temporary hard links.
 constexpr const char * WRITING_BACKUP = "writing backup";

 /// Finding databases and tables in the backup which we're going to restore.
 constexpr const char * FINDING_TABLES_IN_BACKUP = "finding tables in backup";

The backup half is: gathering metadata → extracting data → running post-tasks → building file infos → writing backup → completed. The restore half is separate (finding tables → checking access rights → creating databases → creating tables → inserting data → finalizing tables → completed). BackupCoordinationStageSync is constructed with an is_restore flag and serves both protocols.

Phase 2: the file manifest — the heart of it

Each host has its own set of files (its own MergeTree parts, its own system tables, its own access files). The backup needs one list.

Each host computes a BackupFileInfo for each of its files — name, size, checksum, plus the incremental-backup fields — and publishes them to ZooKeeper:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:945
void BackupCoordinationOnCluster::addFileInfos(BackupFileInfos && file_infos_)
{
 {
 std::lock_guard lock{file_infos_mutex};
 if (file_infos)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "addFileInfos() must not be called after preparing");
 }

 auto component_guard = Coordination::setCurrentComponent("BackupCoordinationOnCluster::addFileInfos");
 /// Serialize `file_infos_` and write it to ZooKeeper's nodes.
 String file_infos_str = FileInfos::serialize(file_infos_);
 serializeToMultipleZooKeeperNodes(zookeeper_path + "/file_infos/" + current_host, file_infos_str, "addFileInfos");
}

The serialized form is compact and deliberately omits the fields that are derived rather than observed:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:106
 static String serialize(const BackupFileInfos & file_infos_)
 {
 WriteBufferFromOwnString out;
 writeBinary(file_infos_.size(), out);
 for (const auto & info : file_infos_)
 {
 writeBinary(info.file_name, out);
 writeBinary(info.size, out);
 writeBinary(info.checksum, out);
 writeBinary(info.base_size, out);
 writeBinary(info.base_checksum, out);
 writeBinary(info.encrypted_by_disk, out);
 writeBinary(info.reference_target, out);
 /// We don't store `info.data_file_name` and `info.data_file_index` because they're determined automalically
 /// after reading file infos for all the hosts (see the class BackupCoordinationFileInfos).
 }
 return out.str();
 }

The payload can be large — a backup of a big cluster has millions of files — so it is chopped into chunks of at most backup_restore_keeper_value_max_size bytes (default 1 MiB) and stored as numbered child nodes:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:308
 size_t max_part_size = keeper_settings.value_max_size;
 if (!max_part_size)
 max_part_size = value.size();

 size_t num_parts = (value.size() + max_part_size - 1) / max_part_size; /// round up

 for (size_t i = 0; i != num_parts; ++i)
 {
 size_t begin = i * max_part_size;
 size_t end = std::min(begin + max_part_size, value.size());
 String part = value.substr(begin, end - begin);
 String part_path = fmt::format("{}/{:06}", path, i);

The {:06} zero-padding makes the lexicographic order of the child names equal to the numeric order of the parts — so reading them back is just getChildren + sort.

Then every host reads everyone’s file infos and computes the global plan:

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:975
void BackupCoordinationOnCluster::prepareFileInfos() const
{
 if (file_infos)
 return;

 file_infos.emplace(BackupCoordinationFileInfos::Config{plain_backup, data_file_name_gen, data_file_name_prefix_length});

 Strings hosts_with_file_infos;
 {
 auto holder = with_retries.createRetriesControlHolder("prepareFileInfos::get_hosts");
 holder.retries_ctl.retryLoop(
 [&, &zk = holder.faulty_zookeeper]()
 {
 with_retries.renewZooKeeper(zk);
 hosts_with_file_infos = zk->getChildren(zookeeper_path + "/file_infos");
 });
 }

 for (const String & host : hosts_with_file_infos)
 {
 String file_infos_str = deserializeFromMultipleZooKeeperNodes(zookeeper_path + "/file_infos/" + host, "prepareFileInfos");
 auto deserialized_file_infos = FileInfos::deserialize(file_infos_str).file_infos;
 file_infos->addFileInfos(std::move(deserialized_file_infos), host);
 }
}

Phase 3: assigning a global name and index to every file

BackupCoordinationFileInfos::prepare() is where the global plan is computed. Every host runs this code over the same input and must get the same answer — which is why the sort key is the file name and nothing else.

 cpp
 
 
// src/Backups/BackupCoordinationFileInfos.cpp:67
 /// Make a list of all file infos from all hosts.
 size_t total_num_infos = 0;
 for (const auto & [_, infos] : file_infos)
 total_num_infos += infos.size();

 file_infos_for_all_hosts.reserve(total_num_infos);
 for (auto & [_, infos] : file_infos)
 for (auto & info : infos)
 file_infos_for_all_hosts.emplace_back(&info);

 /// Sort the list of all file infos by file name (file names must be unique).
 std::sort(file_infos_for_all_hosts.begin(), file_infos_for_all_hosts.end(), BackupFileInfo::LessByFileName{});

 auto adjacent_it = std::adjacent_find(file_infos_for_all_hosts.begin(), file_infos_for_all_hosts.end(), BackupFileInfo::EqualByFileName{});
 if (adjacent_it != file_infos_for_all_hosts.end())
 {
 throw Exception(
 ErrorCodes::BACKUP_ENTRY_ALREADY_EXISTS, "Entry {} added multiple times to backup", quoteString((*adjacent_it)->file_name));
 }

The file names are unique because each host stores its data under shards/<n>/replicas/<m>/ in the backup. Two hosts cannot produce the same name, and a duplicate is a hard error rather than a silent overwrite.

Now the interesting branch — the non-plain (deduplicating) path:

 cpp
 
 
// src/Backups/BackupCoordinationFileInfos.cpp:162
 /// For non-plain backups files with the same size and checksum are stored only once,
 /// in order to find those files we'll use this map.
 std::map<SizeAndChecksum, size_t> data_file_index_by_checksum;

 for (size_t i = 0; i != file_infos_for_all_hosts.size(); ++i)
 {
 auto & info = *(file_infos_for_all_hosts[i]);

 if (!info.reference_target.empty())
 {
 if (!try_resolve_reference(info))
 unresolved_references.push_back(&info);

 continue;
 }

 if (info.size == info.base_size)
 {
 /// A file is either empty or can be get from the base backup as a whole.
 info.data_file_name.clear();
 info.data_file_index = static_cast<size_t>(-1);
 }
 else
 {
 SizeAndChecksum size_and_checksum{info.size, info.checksum};
 auto [it, inserted] = data_file_index_by_checksum.emplace(size_and_checksum, i);
 if (inserted)
 {
 /// Found a new file.
 info.data_file_name = getBackupDataFileName(info, config.data_file_name_generator, config.data_file_name_prefix_length);
 info.data_file_index = i;
 ++num_files;
 total_size_of_files += info.size - info.base_size;
 }
 else
 {
 /// Found a file with the same size and checksum as some file before, reuse old `data_file_index` and `data_file_name`.
 info.data_file_index = it->second;
 info.data_file_name = file_infos_for_all_hosts[it->second]->data_file_name;
 }
 }

 file_name_to_info.emplace(info.file_name, &info);
 }

Three things happen here:

data_file_index is simply the position in the globally sorted list. That is the coordination key: index 17 is the same file on every host.
Files with identical (size, checksum) collapse to one physical data file. On a replicated cluster, the same part exists on every replica with the same checksum, so a 3-replica cluster stores one copy of the data and three manifest entries pointing at it.
total_size_of_files accumulates info.size - info.base_size, i.e. only the bytes that are new relative to the incremental base.

Phase 4: electing the writer — ZooKeeper `create` as a mutex

Now the question: host A has file 17, host B has file 23, host C has file 17 too (same content). Somebody has to write the physical file for index 17. Exactly one host must do it.

The answer is the simplest possible use of ZooKeeper: create is atomic, and fails with ZNODEEXISTS if the node is there.

 cpp
 
 
// src/Backups/BackupCoordinationOnCluster.cpp:1001
bool BackupCoordinationOnCluster::startWritingFile(size_t data_file_index)
{
 {
 /// Check if this host is already writing this file.
 std::lock_guard lock{writing_files_mutex};
 if (writing_files.contains(data_file_index))
 return false;
 }
 auto component_guard = Coordination::setCurrentComponent("BackupCoordinationOnCluster::startWritingFile");

 /// Store in Zookeeper that this host is the only host which is allowed to write this file.
 bool host_is_assigned = false;
 String full_path = zookeeper_path + "/writing_files/" + std::to_string(data_file_index);
 String host_index_str = std::to_string(current_host_index);

 auto holder = with_retries.createRetriesControlHolder("startWritingFile");
 holder.retries_ctl.retryLoop(
 [&, &zk = holder.faulty_zookeeper]()
 {
 with_retries.renewZooKeeper(zk);
 auto code = zk->tryCreate(full_path, host_index_str, zkutil::CreateMode::Persistent);

 if (code == Coordination::Error::ZOK)
 host_is_assigned = true; /// If we've just created this ZooKeeper's node, this host is assigned.
 else if (code == Coordination::Error::ZNODEEXISTS)
 host_is_assigned = (zk->get(full_path) == host_index_str); /// The previous retry could write this ZooKeeper's node and then fail.
 else
 throw zkutil::KeeperException::fromPath(code, full_path);
 });

 if (!host_is_assigned)
 return false; /// Other host is writing this file.

 {
 /// Check if this host is already writing this file,
 /// and if it's not, mark that this host is writing this file.
 /// We have to check that again because we were accessing ZooKeeper with the mutex unlocked.
 std::lock_guard lock{writing_files_mutex};
 return writing_files.emplace(data_file_index).second; /// Return false if this host is already writing this file.
 }
}

The ZNODEEXISTS branch is the interesting one. The node’s value is the index of the host that won the election. If the value equals my index, I won — even though I did not create the node. That covers the case where a retry after a network hiccup created the node successfully but the response was lost. Idempotent election without a separate “am I the owner?” round trip.

And the caller simply skips files it lost:

 cpp
 
 
// src/Backups/BackupImpl.cpp:1741
 if (!coordination->startWritingFile(info.data_file_index))
 {
 LOG_TRACE(log, "Writing backup for file {} from {}: skipped, data file #{} is already being written", info.data_file_name, src_file_desc, info.data_file_index);
 return;
 }

So on a 3-replica cluster with identical parts, exactly one replica uploads each distinct file and the other two skip it. The result is a backup that contains the data once.

 text
 
 
 host A host B host C
 | | |
 |-- file_infos/A ------> /file_infos/ <--- file_infos/B --|
 | | |
 | all three read all three manifests, sort by name |
 | all three compute the same data_file_index map |
 | | |
 | create /writing_files/17 -> ZOK (A wins) |
 | | create -> ZNODEEXISTS |
 | | value != my index |
 | | -> skip file 17 |
 | | |
 | write file 17 | write file 23 | write file 41

Retries: everything is wrapped

Every ZooKeeper operation in the backup path goes through a retry controller with exponential backoff. The default is deliberately generous, because a backup that fails halfway because of a 30-second Keeper blip is worse than one that takes longer:

 cpp
 
 
// src/Core/Settings.cpp:4848
 DECLARE(UInt64, backup_restore_keeper_max_retries, 1000, R"(
Max retries for [Zoo]Keeper operations in the middle of a BACKUP or RESTORE operation.
Should be big enough so the whole operation won't fail because of a temporary [Zoo]Keeper failure.
)", 0, \

The backoff is a plain doubling with a cap:

 cpp
 
 
// src/Common/ZooKeeper/ZooKeeperRetries.h:232
 /// retries
 logLastError("will retry due to error");
 sleepForMilliseconds(current_backoff_ms);
 current_backoff_ms = std::min(current_backoff_ms * 2, retries_info.max_backoff_ms);

with backup_restore_keeper_retry_initial_backoff_ms = 100 and backup_restore_keeper_retry_max_backoff_ms = 5000. Critically, only hardware errors are retried:

 cpp
 
 
// src/Common/ZooKeeper/ZooKeeperRetries.h:81
 catch (const zkutil::KeeperException & e)
 {
 iteration_cleanup();

 if (!Coordination::isHardwareError(e.code))
 throw;

 setKeeperError(std::current_exception(), e.code, e.message());
 }

A ZNODEEXISTS or a ZBADVERSION is a semantic answer, not a failure — retrying it would be wrong. A connection reset is a hardware error and gets retried.

Trade-offs

What the design buys:

One manifest, one file namespace. A BACKUP ON CLUSTER is restorable as a unit; RESTORE ON CLUSTER puts the data back on the right shards and replicas.
No data duplication. The (size, checksum) dedup means a 3-replica cluster stores the part once.
No coordination server. The only shared state is ZooKeeper, and the only ZooKeeper operation used for the critical election is create.
Crash tolerance. Every phase is idempotent. A host that dies leaves ephemeral nodes behind; the survivors notice within sync_period_ms and either retry or abort.

What it costs:

Polling latency. backups.sync_period_ms defaults to 5000 ms, so each stage barrier costs up to five seconds of wall clock even when everything is ready. A backup with six stages pays this repeatedly.
ZooKeeper is a hard dependency for the whole operation. It is not in the data path — the actual bytes go straight to S3 — but it is in the control path, and a Keeper outage stalls every backup in the cluster.
The manifest is O(files) in ZooKeeper. A backup of millions of parts writes millions of file-info records into Keeper, in 1 MiB chunks. This is the single biggest scaling constraint on BACKUP ON CLUSTER.
The metadata consensus loop is a heuristic, not a proof. Two consecutive identical reads on every host, plus a cross-host vote, gives strong practical safety — but there is no fencing token, so in principle a rename that happens to straddle the window identically on every host could slip through. The INCONSISTENT_METADATA_FOR_BACKUP error and the 600-second timeout are the safety nets.

Wrapping up

The one thing to remember: a cluster-wide backup is a global file namespace, and the global namespace is produced by sorting every host’s file list by name. The sort is the whole trick — it gives every file a cluster-unique index without any central allocation, because sorting is commutative and deterministic.

The part people get wrong: the deduplication is by (size, checksum), not by path. Two hosts with the same file at different paths in the backup share one physical object. That is why getNumFiles() and getNumEntries() differ, and why the backup’s on-disk size is much smaller than the sum of the replicas’ data.

What to go do about it: if your BACKUP ON CLUSTER is slow, look at the log lines Setting stage: ... and Hosts [...] reached stage .... The gap between them is barrier latency — and with the default 5-second sync period, that gap is almost certainly the problem, not the data transfer.

Backups on Opensource internals

ClickHouse Incremental Backups Store Only the Changed Tail of a File

The naive incremental backup, and why it isn’t used

Where the base backup comes from

Deciding what to store

Why partial checksums are only allowed for append-only files

Writing only the suffix

The manifest

Restoring: stitching the halves back together

The disk-copy fast path

The second kind of sharing: identical files

Trade-offs

Wrapping up

ClickHouse Coordinates a Cluster-Wide Backup with ZooKeeper

The shape of the thing

The ZooKeeper layout

Phase 0: claiming the operation

The concurrency check

Phase 1: the metadata consensus loop

The stage barrier itself

The stage list

Phase 2: the file manifest — the heart of it

Phase 3: assigning a global name and index to every file

Phase 4: electing the writer — ZooKeeper create as a mutex

Retries: everything is wrapped

Trade-offs

Wrapping up

Phase 4: electing the writer — ZooKeeper `create` as a mutex