ClickHouse Makes a MergeTree Part Visible with One Directory Rename

TL;DR: A part is written into tmp_insert_<part_name>/, one file per column, each through a stack of WriteBuffer decorators ending in a 1 MiB file buffer. When every byte is on disk, the whole directory is renamed to its final name with one rename(2) — the atomic step that makes the part visible. The fsyncs are ordered child-before-parent so the rename can never point at a not-yet-durable directory.

An INSERT into a MergeTree table does not “add rows to a table”. It materialises a directory containing a dozen files, and then makes that directory visible atomically. Everything hard about MergeTree’s durability model follows from that framing.

The naive implementation writes all_1_1_1_0/ file by file and updates the part list when it is done. That breaks in two ways. A reader that sees the directory mid-write reads a truncated .bin file. A crash between the last write() and the metadata update leaves a directory that looks like a part but is not in the part list.

ClickHouse’s answer has three parts: write under a temporary name, make the name change the atomic commit point, and fsync in an order that makes the rename durable.

text
   INSERT                        commit                     readers
     |                             |                          |
     v                             v                          v
  tmp_insert_all_1_1_1_0/  --rename-->  all_1_1_1_0/  ------>  visible
   (part is written here)      (atomic)   (the commit point)

The write buffer stack

A part is not one file. In the Wide part format it is one file per column, plus marks, plus metadata, plus checksums. Each is written through a stack of WriteBuffer decorators.

The bottom of the stack is a file buffer, 1 MiB by default:

cpp
// src/Core/Defines.h:23
static constexpr auto DBMS_DEFAULT_BUFFER_SIZE = 1048576ULL;

which is also the default of the disk-level write API:

cpp
// src/Disks/IDisk.h:271
    /// Open the file for write and return WriteBufferFromFileBase object.
    virtual std::unique_ptr<WriteBufferFromFileBase> writeFile( /// NOLINT
        const String & path,
        size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE,
        WriteMode mode = WriteMode::Rewrite,
        const WriteSettings & settings = {}) = 0;

The WriteBuffer protocol is small. write() has a fast path that copies into the buffer if there is room:

cpp
// src/IO/WriteBuffer.h:95
    void write(const char * from, size_t n)
    {
        /// `closed` is checked first: after `finalize` some buffers reset their storage to null (`set(nullptr, 0)`),
        /// and the pointer arithmetic in `available` must not be evaluated then.
        /// The comparison is unsigned as in `available`: for a constant `n` it is a single `cmp`.
        if (!closed && n <= available()) [[likely]]
        {
            copyBytes(pos, from, n);
            pos += n;
            return;
        }
        writeSlow(from, n);
    }

and next() is the flush, which calls the subclass’s nextImpl() and then resets the cursor to the beginning of the buffer:

cpp
// src/IO/WriteBuffer.h:54
        try
        {
            nextImpl();
            ++flush_count;
        }
        catch (CurrentBufferExhausted &)
        {
            pos = working_buffer.begin();
            bytes += bytes_in_buffer;
            throw;
        }
        catch (...)
        {
            /** If the nextImpl() call was unsuccessful, move the cursor to the beginning,
              * so that later (for example, when the stack was expanded) there was no second attempt to write data.
              */
            pos = working_buffer.begin();
            bytes += bytes_in_buffer;

            cancel();

            throw;
        }

        bytes += bytes_in_buffer;
        pos = working_buffer.begin() + nextimpl_working_buffer_offset;
        nextimpl_working_buffer_offset = 0;
    }

Two details matter. First, a failed flush cancels the whole buffer stack — an error in the innermost layer must not be retried by an outer layer. Second, bytes counts the bytes handed to the subclass, not the bytes that reached the disk; that distinction is why the hashing buffer can report a checksum for bytes that are still in the OS buffer.

At the bottom, the file buffer turns buffer-full into a write(2):

cpp
// src/IO/WriteBufferFromFileDescriptor.cpp:52
    size_t bytes_written = 0;
    while (bytes_written != offset())
    {
        ProfileEvents::increment(ProfileEvents::WriteBufferFromFileDescriptorWrite);

        ssize_t res = 0;
        {
            CurrentMetrics::Increment metric_increment{CurrentMetrics::Write};
            res = ::write(fd, working_buffer.begin() + bytes_written, offset() - bytes_written);
        }

Note the loop: a short write is not an error, it is a retry. The same loop shape appears in every layer of the stack, which is why one decorator works for files, sockets and object storage alike.

Hashing without a second pass

The interesting decorator is HashingWriteBuffer. A part’s checksums are computed over the bytes as they are compressed, on the way out:

cpp
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:340
    auto write_hashed_file = [&](const auto & filename, auto && writer)
    {
        auto out = new_part->getDataPartStorage().writeFile(filename, 4096, writer_settings.query_write_settings);
        HashingWriteBuffer out_hashing(*out);
        writer(out_hashing);
        out_hashing.finalize();
        checksums.files[filename].file_size = out_hashing.count();
        checksums.files[filename].file_hash = out_hashing.getHash();
        out->preFinalize();
        written_files.emplace_back(std::move(out));
    };

The stack here is HashingWriteBuffer → WriteBufferFromFileBase, and the checksum is taken from the hashing layer after it is finalized, so the hash covers exactly the bytes that were written — no re-read, no second pass over the compressed data. (Note that the part writer asks for a 4 KiB file buffer, not the 1 MiB default: the hash and compression layers above it do the buffering, and a part is many small files.)

Note out->preFinalize(). This is the hook that makes object storage bearable — it is documented in WriteBuffer as “similar to prefetch”:

cpp
// src/IO/WriteBuffer.h:123
    /// The idea is similar to prefetch. In case if all data is written, we can flush the buffer
    /// and start sending data asynchronously. It may improve writing performance in case you have
    /// multiple files to finalize. Mainly, for blob storage, finalization has high latency,
    /// and calling preFinalize in a loop may parallelize it.
    virtual void preFinalize() { next(); }

A part has many files; each file’s finalization on S3 is a separate PUT with its own latency. preFinalize in a loop lets those PUTs overlap instead of serialising them.

Finally, all files are finalized and then synced in parallel:

cpp
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:160
void MergedBlockOutputStream::Finalizer::Impl::finish()
{
    writer.finish(sync);

    /// Finalize all files first (writes any pending bytes to the OS),
    /// then sync them in parallel — fsync of independent files can run concurrently
    /// and is a major contributor to part finalization latency when many small files are involved.
    for (auto & file : written_files)
        file->finalize();

    if (sync)
    {
        std::vector<WriteBufferFromFileBase *> files_to_sync;
        files_to_sync.reserve(written_files.size());
        for (auto & file : written_files)
            files_to_sync.push_back(file.get());
        parallelSyncFiles(files_to_sync);
    }

Finalize-all-then-sync-all, rather than finalize-and-sync-each: fsync of independent files can run concurrently, and with a wide part that is a dozen files, the difference is the difference between one fsync latency and a dozen serial ones.

The temporary directory

The part directory does not get its final name until the data is in. The writer builds a name with a tmp_insert_ prefix:

cpp
// src/Storages/MergeTree/MergeTreeDataWriter.cpp:855
    std::string temp_prefix = "tmp_insert_";
    const auto & temp_postfix = data.getPostfixForTempInsertName();
    if (!temp_postfix.empty())
        temp_prefix += temp_postfix + "_";

    std::string part_dir = temp_prefix + part_name;

and then claims that name, so that two concurrent inserts that compute the same part name cannot fight over the same directory. MergeTreeData::claimTemporaryPartDirectory takes the claim (and, if a leftover may exist, reclaims the stale directory), and the claim itself is a name in a set, released by a scope guard:

cpp
// src/Storages/MergeTree/MergeTreeData.cpp:4539
scope_guard MergeTreeData::getTemporaryPartDirectoryHolder(const String & part_dir_name) const
{
    temporary_parts.add(part_dir_name);
    return [this, part_dir_name]() { temporary_parts.remove(part_dir_name); };
}

and it also makes leftover cleanup safe. If a previous run crashed after creating tmp_insert_all_1_1_1_0/, the next insert with the same part name reclaims it — but only because it owns the name:

cpp
// src/Storages/MergeTree/MergeTreeData.cpp:4561
    /// Only temporary names may be auto-reclaimed. Elsewhere (e.g. "detached/<dir>" during ATTACH) the
    /// directory contents are the payload, and reclaiming would destroy user data.
    if (!startsWith(part_dir_name, "tmp") || part_dir_name.contains('/'))
        throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot reclaim {}: not a temporary part directory name", part_dir_name);

The startsWith("tmp") check is a safety interlock: only a directory whose name provably looks temporary may be deleted automatically. A directory named detached/xyz — whose contents are user data being attached — can never be reclaimed.

The part builder is then told that the directory is guaranteed clean (PartDirIntent::CreateFresh, whose comment says “Directory is guaranteed clean by the claim”), so it does not have to probe the disk to find out.

The rename is the commit point

When the data is written, the part is renamed into place. The code says why outright:

cpp
// src/Storages/MergeTree/MergeTreeData.cpp:7699
    /// All checks are passed. Now we can rename the part on disk.
    /// So, we maintain invariant: if a non-temporary part in filesystem then it is in data_parts
    preparePartForCommit(part, out_transaction, lock, /* need_rename= */ true, rename_in_transaction);

and the rename itself:

cpp
// src/Storages/MergeTree/MergeTreeData.cpp:7601
    if (need_rename && !rename_in_transaction)
        part->renameTo(part->name, true);

renameTo is a directory rename; the destination is computed from the final relative path and handed to the part storage:

cpp
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:3187
    auto old_projection_root_path = getDataPartStorage().getRelativePath();
    auto to = fs::path(relative_path) / new_relative_path;

    getDataPartStorage().rename(to.parent_path(), to.filename(), storage.log.load(), remove_new_dir_if_exists, fsync_dir);

which ends in DiskLocal::moveDirectory, i.e. one rename(2):

cpp
// src/Disks/DiskLocal.cpp:346
void DiskLocal::moveDirectory(const String & from_path, const String & to_path)
{
    fs::rename(fs::path(disk_path) / from_path, fs::path(disk_path) / to_path);
}

rename(2) within a filesystem is atomic: a concurrent reader listing the table directory sees either tmp_insert_all_1_1_1_0 or all_1_1_1_0, never both and never a half-renamed mixture. There is no window in which the part directory exists under its final name but is incomplete — because it was complete under the temporary name all along.

The same trick is used elsewhere for a slightly different reason: when removing a file, ClickHouse renames it to a random 32-character name first and deletes that (RemoveRecursiveOperation::execute calls disk.moveFile(path, path_to) with getRandomASCIIString(32)). Renaming to a random name makes the removal atomic from the reader’s point of view: a reader that had opened the old path keeps its inode, and no new reader can open the path at all.

fsync ordering: child before parent

A rename is only durable if the contents it points at are durable, and if the directory entry of the rename is durable. The order matters:

cpp
// src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:785
        /// Only after moveDirectory() since before the directory does not exist.
        if (fsync_part_dir)
        {
            /// Sync child before parent so a parent dentry never points at a not-yet-synced
            /// child: the moved dir, then the parent(s) holding the renamed dentry.
            { SyncGuardPtr to_sync_guard = volume->getDisk()->getDirectorySyncGuard(to); }

            /// parent_path() twice: step past the trailing slash on `to`/`from`, then to the container.
            const String to_parent = fs::path(to).parent_path().parent_path().string();
            const String from_parent = fs::path(from).parent_path().parent_path().string();

            if (!to_parent.empty())
                { SyncGuardPtr to_parent_sync_guard = volume->getDisk()->getDirectorySyncGuard(to_parent); }

            if (!from_parent.empty() && from_parent != to_parent)
                { SyncGuardPtr from_parent_sync_guard = volume->getDisk()->getDirectorySyncGuard(from_parent); }
        }
    });

The comment is the whole lesson: sync child before parent, so a parent dentry never points at a not-yet-synced child. If you fsync the parent first and crash, the parent may durably reference a directory whose contents are not on disk — an empty or truncated part that looks valid.

The sync of the moved directory happens after moveDirectory(), and the comment says why: before the rename the directory did not exist at to, so there was nothing to sync there.

The invariant, and what happens when it breaks

The commit sequence establishes a clean invariant:

if a non-temporary part exists in the filesystem, then it is in data_parts.

The reverse is not true, and that asymmetry is deliberate: a part can be in data_parts (in memory) but not yet on disk, because the metadata commit and the data write are separate steps. The dangerous direction is the one the invariant forbids.

The check is asserted in the code: a part whose directory still has a tmp_ or tmp-fetch_ prefix must be registered in temporary_parts (MergeTreeData::preparePartForCommit asserts exactly that), otherwise it is a leftover from a crash and may be cleaned up.

On startup, the server reconciles reality with the invariant: tmp_* directories are leftovers and get removed; directories that look like real parts but are not in the part list get attached or detached, depending on how complete they are.

Moving a part between disks

The same rename trick is the last step of moving a part to another disk, which is how a storage policy rebalances tiers. The mover clones the part into the destination’s moving/ directory, and only then swaps:

cpp
// src/Storages/MergeTree/MergeTreePartsMover.cpp:364
    renameClonedPart(*cloned_part.part);

    /// If server goes down here we will get two copy of the part with the same name on different disks.
    /// And on the next ClickHouse startup during loading parts the first copy (in the order of defining disks
    /// in the storage policy) will be loaded as Active, the second one will be loaded as Outdated and removed as duplicate.
    /// See MergeTreeData::loadDataParts().
    data->swapActivePart(cloned_part.part, part_lock);

The comment is honest about the failure mode: a crash between the clone and the swap leaves two copies of the part, and the startup loader resolves it by loading the first and marking the second outdated. The move is not atomic across disks — it cannot be — but it is idempotent under recovery, which is the property that actually matters.

Trade-offs

What the design buys:

No half-visible parts. A reader either sees the whole part or none of it, because visibility is a single directory rename.
Crash-safe cleanup. Anything still named tmp_* after a restart is garbage by construction, and is reclaimable.
No second pass for checksums. The hash is computed on the way out, inside the write stack.
Parallel durability. Files are fsynced concurrently, so a wide part costs one fsync latency rather than N.

What it costs:

A directory rename per part. On object storage this is a metadata operation rather than a filesystem rename, but it is still a round trip.
Temporary directory churn. Every insert creates and renames a directory; on a slow-metadata filesystem this is measurable.
fsync is expensive and configurable. fsync_part_directory and the table-level fsync_after_insert trade durability for latency, and turning them off means a power loss can lose the last inserted part even though the server said the insert succeeded.
The invariant is checked, not enforced. A part can be in memory but not on disk; recovery has to reconcile.

Wrapping up

The one thing to remember: the rename is the transaction. Everything before it is preparation, and everything after it is bookkeeping.

The part people get wrong: a successful INSERT does not mean the data is on the platter. With fsync_after_insert = 0, “on the platter” means “in the page cache”. A power loss can lose an insert that the server already acknowledged.

What to go do about it: if you are chasing insert latency, look at fsync_after_insert and at how many files your part format produces. A Wide part with 50 columns is 50 files, 50 fsyncs (parallel, but still) and 50 times the metadata churn of a Compact part.