How ClickHouse Spreads Parts Across Disks: the Storage Policy

TL;DR: A storage policy is a list of volumes, each holding one or more disks. Writing a part calls reserve() down that ladder: the policy tries volumes in priority order, a volume tries its disks (round-robin or least-used), and a disk does the actual accounting. A volume can refuse to hold parts larger than max_data_part_size, which is how a 100 GiB part skips the NVMe tier and lands on the cheap one.

A MergeTree table has to answer two questions continuously: where does the next part go? and when does an old part move to cheaper storage? Answering them well means the hot data is on fast local disks and the cold data is on cheap object storage, without filling a disk to 100%.

The naive design — “put every part on the disk with the most free space” — fails in both directions. It ignores the fact that the tiers have different costs, and it makes placement depend on a moment-to-moment measurement, so the same part lands on different disks depending on when it was written.

ClickHouse’s design is a three-level ladder with an explicit priority order, and the interesting decisions are all about refusing work rather than accepting it.

The three levels

text
   StoragePolicy  "hot_to_cold"
       |
       +-- Volume "hot"          volume_priority = 1
       |     +-- Disk "nvme1"
       |     +-- Disk "nvme2"
       |
       +-- Volume "cold"         volume_priority = 2
             +-- Disk "s3"

A policy is an ordered list of volumes. It is what a table names in SETTINGS storage_policy = '...'.
A volume is a set of disks that are interchangeable for a given part — a JBOD. Its volume_priority says how early it is tried.
A disk is where bytes actually land: a local directory, an S3 bucket, an encrypted wrapper, a cache layer.

The policy’s volumes are sorted by priority, and the priority values are validated to be a dense range:

cpp
// src/Disks/StoragePolicy.cpp:85
    if (!volume_priorities.empty())
    {
        /// Check that priority values cover the range from 1 to N (lowest explicit priority)
        if (*volume_priorities.begin() != 1 || *volume_priorities.rbegin() != volume_priorities.size())
            throw Exception(
                ErrorCodes::INVALID_CONFIG_PARAMETER,
                "volume_priority values must cover the range from 1 to N (lowest priority specified) without gaps");

        ::stableSort(
            volumes.begin(), volumes.end(),
            [](const VolumePtr a, const VolumePtr b) { return a->volume_priority < b->volume_priority; });
    }

Two rules hide here. Priorities must be unique (checked when the volumes are built), and they must form a gapless range 1..N — the priority is used as an index (“start looking for space at volume index k”), so a gap would make that index meaningless.

Reserving space: the ladder

Writing a part does not write to a disk directly. It reserves space first, and the reservation is what makes concurrent inserts safe: two inserts that both see “10 GiB free” must not both plan to write 10 GiB.

The top of the ladder is the policy, and it walks volumes in priority order:

cpp
// src/Disks/StoragePolicy.cpp:269
ReservationPtrOrError StoragePolicy::reserve(UInt64 bytes, size_t min_volume_index) const
{
    bool all_volumes_read_only = true;
    for (size_t i = min_volume_index; i < volumes.size(); ++i)
    {
        const auto & volume = volumes[i];
        if (volume->isReadOnly())
            continue;
        all_volumes_read_only = false;
        auto reservation = volume->reserve(bytes);
        if (reservation)
            return reservation;
    }

    if (all_volumes_read_only)
        return std::unexpected(
            ReservationError(ErrorCodes::READONLY, "Could not reserve {} because all disk volumes are readonly", ReadableSize(bytes)));

    LOG_TRACE(log, "Could not reserve {} from volume index {}, total volumes {}", ReadableSize(bytes), min_volume_index, volumes.size());
    return std::unexpected(ReservationError(ErrorCodes::NOT_ENOUGH_SPACE, "Cannot reserve {}, not enough space", ReadableSize(bytes)));
}

Note min_volume_index: the caller can say “start at volume 2”. That is what the part-mover uses to push data down the ladder — see below.

The middle of the ladder is the volume, and this is where the interesting refusal lives:

cpp
// src/Disks/VolumeJBOD.cpp:127
ReservationPtr VolumeJBOD::reserveImpl(UInt64 bytes, const std::optional<ReservationConstraints> & constraints)
{
    /// This volume can not store data which size is greater than `max_data_part_size`
    /// to ensure that parts of size greater than that go to another volume(s).
    if (max_data_part_size != 0 && bytes > max_data_part_size)
        return {};

That single line is the tiering mechanism. A volume configured with max_data_part_size_bytes = 10737418240 (10 GiB) will refuse any reservation for a bigger part, so the policy falls through to the next volume. Small parts land on fast disks; the occasional huge part (a big OPTIMIZE, a large mutation result) lands on the cheap tier.

The volume then picks a disk, by one of two algorithms:

cpp
// src/Disks/VolumeJBOD.cpp:144
    switch (load_balancing)
    {
        case VolumeLoadBalancing::ROUND_ROBIN:
        {
            size_t disks_num = disks.size();
            for (size_t i = 0; i < disks_num; ++i)
            {
                size_t start_from = last_used.fetch_add(1u, std::memory_order_acq_rel);
                size_t index = start_from % disks_num;

                ReservationPtr reservation = try_reserve(disks[index]);
                if (reservation)
                    return reservation;
            }
            return {};
        }

Round-robin is a single atomic counter — no lock, no filesystem query. It spreads parts evenly regardless of how full the disks are, which is what you want when the disks are identical.

LEAST_USED is the alternative, and it is deliberately cached:

cpp
// src/Disks/VolumeJBOD.cpp:160
        case VolumeLoadBalancing::LEAST_USED:
        {
            std::lock_guard lock(mutex);

            ReservationPtr reservation;
            for (size_t i = 0; i < disks.size() && !reservation; ++i)
            {
                if (i == 0 && (!least_used_ttl_ms || least_used_update_watch.elapsedMilliseconds() >= least_used_ttl_ms))
                {
                    disks_by_size = LeastUsedDisksQueue(disks.begin(), disks.end());
                    least_used_update_watch.restart();

                    DiskWithSize disk = disks_by_size.top();
                    reservation = try_reserve(disk.disk);
                }

The ordering is by free space, and read-only disks always sort last:

cpp
// src/Disks/VolumeJBOD.h:89
        bool operator<(const DiskWithSize & rhs) const
        {
            if (disk->isReadOnly())
                return true;
            if (rhs.disk->isReadOnly())
                return false;
            return free_size < rhs.free_size;
        }

The TTL (60 seconds by default) is the important part:

cpp
// src/Disks/VolumeJBOD.cpp:81
    least_used_ttl_ms = config.getUInt64(config_prefix + ".least_used_ttl_ms", 60'000);

Querying free space means a statvfs on every disk, on every insert. The cache re-uses the answer for a minute and then refreshes it. In between, the “free space” of a disk is updated by the reservations themselves:

cpp
// src/Disks/VolumeJBOD.h:98
        ReservationPtr reserve(UInt64 bytes)
        {
            ReservationPtr reservation = disk->reserve(bytes);
            if (!reservation)
                return {};

            /// Not just subtract bytes, but update the value,
            /// since some reservations may be done directly via IDisk, or not by ClickHouse.
            free_size = reservation->getUnreservedSpace();
            return reservation;
        }

getUnreservedSpace() is the disk’s own accounting, so the cached number tracks ClickHouse’s own reservations exactly, and drifts only from reservations made outside ClickHouse — which is why the periodic refresh exists.

The bottom of the ladder: what a reservation actually is

At the disk level, a reservation is bytes subtracted from a counter, under a mutex:

cpp
// src/Disks/DiskLocal.cpp:202
std::optional<UInt64> DiskLocal::tryReserve(UInt64 bytes, const std::optional<ReservationConstraints> & constraints)
{
    std::lock_guard lock(DiskLocal::reservation_mutex);

    auto available_space = getAvailableSpace();

    UInt64 unreserved_space = available_space
        ? *available_space - std::min(*available_space, reserved_bytes)
        : std::numeric_limits<UInt64>::max();

and the commit is the counter increment:

cpp
// src/Disks/DiskLocal.cpp:274
        ++reservation_count;
        reserved_bytes += bytes;
        return {unreserved_space - bytes};

The reservation object is a token: it holds the disk’s bytes until it is destroyed, and the storage policy hands the token to the part, which holds it for the lifetime of the part. That is how the space is held while the part is being written — and it is why the free space reported by system.disks is lower than the free space reported by df.

ReservationConstraints adds a second, policy-level condition: reserve only if enough space would remain afterwards.

cpp
// src/Disks/DiskLocal.cpp:223
            UInt64 free_bytes_after = unreserved_space - bytes;

            /// Check min_bytes constraint
            if (constraints->min_bytes > 0 && free_bytes_after < constraints->min_bytes)
            {
                LOG_TRACE(logger, "Could not reserve {} ({} bytes) on disk {}. Free space after reservation {} bytes ({}) would be less than min_bytes {} bytes ({})",
                    ReadableSize(bytes), bytes, backQuote(name), free_bytes_after, ReadableSize(free_bytes_after), constraints->min_bytes, ReadableSize(constraints->min_bytes));
                return {};
            }

This is how ClickHouse refuses to fill a disk completely. The insert path turns it into a user-visible error:

cpp
// src/Storages/MergeTree/MergeTreeDataWriter.cpp:1005
    if (!is_system_database && (min_bytes_to_perform_insert > 0 || min_ratio_to_perform_insert > 0.0f))
    {
        ReservationConstraints constraints(min_bytes_to_perform_insert, min_ratio_to_perform_insert);

        /// Try to reserve space on volume with constraints
        reservation = volume->reserve(expected_size, constraints);

        if (!reservation)
        {
            throw Exception(
                ErrorCodes::NOT_ENOUGH_SPACE,
                "Could not perform insert. "
                "None of the disks in volume '{}' have enough free space to meet the configured threshold. "

so the insert fails before writing anything, rather than filling the disk to zero and failing later.

TTL rules beat policy rules

Before the policy ladder is consulted, the TTL move rules get a chance. If a table has TTL ts + INTERVAL 30 DAY TO VOLUME 'cold', and the inserted data is already expired, the part goes straight to the cold volume:

cpp
// src/Storages/MergeTree/MergeTreeData.cpp:12185
    auto move_ttl_entry = selectTTLDescriptionForTTLInfos(metadata_snapshot->getMoveTTLs(), ttl_infos.moves_ttl, time_of_move, true);

    if (move_ttl_entry)
    {
        LOG_TRACE(log, "Trying to reserve {} to apply a TTL rule. Will try to reserve in the destination", ReadableSize(expected_size));
        SpacePtr destination_ptr = getDestinationForMoveTTL(*move_ttl_entry);
        bool perform_ttl_move_on_insert = is_insert && destination_ptr && shouldPerformTTLMoveOnInsert(destination_ptr);

and if the destination cannot take it, the insert falls back to the policy ladder. Note the perform_ttl_move_on_insert flag: a volume can opt out of having expired data inserted into it, in which case the data is written normally and moved later by the TTL merge.

Moving parts down the ladder

Placement alone is not enough: data written to the fast tier today must be moved to the cold tier tomorrow, when the TTL says so. The mover picks parts to move by asking the policy for space starting after the source disk’s volume:

cpp
// src/Storages/MergeTree/MergeTreePartsMover.cpp:196
    for (auto && move : need_to_move)
    {
        auto min_volume_index = policy->getVolumeIndexByDiskName(move.first->getName()) + 1;
        for (auto && part : move.second.getAccumulatedParts())
        {
            auto reservation = policy->reserve(part->getBytesOnDisk(), min_volume_index).value_or(nullptr);
            if (!reservation)
            {
                /// Next parts to move from this disk has greater size and same min volume index.
                /// There are no space for them.
                /// But it can be possible to move data from other disks.
                break;
            }

min_volume_index = source volume index + 1 means: never move data to the volume it is already in, or to a faster one. Data flows in one direction only — down the priority ladder — which keeps the mover from thrashing.

The move itself is a clone plus an atomic rename, as covered in the article on part publication. What matters here is that it is not atomic across disks, and that recovery is defined to handle that: a crash mid-move leaves two copies, and the loader keeps one and marks the other outdated.

You can also move parts by hand:

sql
-- Move every part of partition 2024_05 to the 's3' disk of the 'cold' volume.
ALTER TABLE events MOVE PARTITION 2024_05 TO DISK 's3';

-- Or to a whole volume.
ALTER TABLE events MOVE PARTITION 2024_05 TO VOLUME 'cold';

Inspecting the policy

The policy is fully introspectable:

sql
SELECT
    policy_name,
    volume_name,
    volume_priority,
    disks,
    max_data_part_size,
    load_balancing,
    perform_ttl_move_on_insert
FROM system.storage_policies
ORDER BY policy_name, volume_priority;

and system.disks shows, per disk, the free space, the total space and the keep_free_space floor configured for it.

The `move_factor`, and what it does now

Every policy has a move_factor, described in the system table as “when the amount of available space gets lower than this factor, data automatically starts to move on the next volume if any”:

cpp
// src/Disks/StoragePolicy.cpp:113
    const double default_move_factor = volumes.size() > 1 ? 0.1 : 0.0;
    move_factor = config.getDouble(config_prefix + ".move_factor", default_move_factor);

It is worth being precise about this: in the current code the factor is reported (system.storage_policies.move_factor) but no longer drives an automatic background move. Data moves down the tiers because of TTL rules, or because you ask it to. The default of 0.1 for a multi-volume policy and 0.0 for a single-volume one is a leftover from the era when the factor was the trigger for automatic rebalancing.

Trade-offs

What the design buys:

Deterministic placement. A part’s volume is decided by the priority ladder and the size of the part, not by a transient free-space reading.
Big parts skip the fast tier. max_data_part_size is a one-line refusal that buys a whole tiering policy.
Reservations, not hope. Concurrent inserts cannot overcommit a disk, and the “keep N bytes free” constraint makes the refusal explicit and early.
One-direction data flow. The mover refuses to move data up the ladder.

What it costs:

Free space is read at the wrong granularity. least_used mode caches statvfs results for up to a minute, so placement can be slightly wrong. Round-robin avoids the question by ignoring free space entirely.
Reservations leak into the numbers. system.disks.free_space and df will disagree, and a long merge can hold a reservation for a long time.
The tier boundary is a cliff, not a slope. A part of 11 GiB goes entirely to the cold tier because the hot tier’s limit is 10 GiB; there is no “mostly hot” placement.
Moving is expensive and not atomic across disks. A move is a full copy plus a rename, and a crash in the middle leaves duplicates to reconcile.

Wrapping up

The one thing to remember: the placement rule is “first volume that accepts the reservation”, and a volume accepts or refuses based on the size of the part and the free space on its disks. Everything else — round-robin, least-used, TTL rules — is a detail of how a volume picks among its own disks.

The part people get wrong: a multi-volume policy does not spread data by usage. It spreads it by part size and by TTL. A table with two volumes and no TTL rule keeps all its new parts on volume 1 forever.

What to go do about it: look at system.storage_policies and check that max_data_part_size on your fast volume is smaller than your typical part size but larger than your typical merge result — otherwise big merges will silently land on your slow disks.