ClickHouse Caches Remote Objects as Local File Segments

TL;DR: A remote object is not cached as a file — it is chopped into 32 MiB segments, each of which becomes a local file named <offset>_<size>. The size lives in the name so that a restart can rebuild the cache index with a directory walk and no stat calls, and the eviction order lives only in memory, in an std::list used as an LRU queue.

Reading from object storage is a latency problem, not a bandwidth problem. A SELECT that touches one column of one part issues a handful of ranged GETs, each of which costs a TLS handshake or a round trip, and then waits. If the answer to “make it fast” is “download the file”, the file is usually 100× larger than the bytes actually wanted, and the first reader pays for all of it.

The obvious cache design — a map from path to a local copy of the whole file — cannot serve a partial hit (the reader wants bytes 40 MiB–41 MiB of a 3 GiB object), and cannot bound the damage of caching one enormous object nobody is going to read again.

ClickHouse’s answer is to make the unit of caching a segment: a fixed-size slice of a remote object, stored as its own local file, tracked individually by size, evicted individually by an LRU.

The unit of caching: a segment on a grid

The cache does not store arbitrary ranges. A range request is cut onto a grid whose cell size is max_file_segment_size, 32 MiB by default:

cpp
// src/Interpreters/FileCache/FileCache_fwd.h:8
static constexpr int FILECACHE_DEFAULT_MAX_FILE_SEGMENT_SIZE = 32 * 1024 * 1024; /// 32Mi
static constexpr int FILECACHE_DEFAULT_FILE_SEGMENT_ALIGNMENT = 4 * 1024 * 1024; /// 4Mi
static constexpr int FILECACHE_DEFAULT_MAX_RESERVE_GRANULARITY = 4 * 1024 * 1024; /// 4Mi. 0 disables reserve-ahead.
static constexpr int FILECACHE_DEFAULT_MAX_FILE_SEGMENT_SIZE_WITH_BACKGROUND_DOWLOAD = 4 * 1024 * 1024; /// 4Mi

The cutting function is FileCache::splitRange. It has two sizes, and the difference matters:

cpp
// src/Interpreters/FileCache/FileCache.cpp:774
    std::vector<FileSegment::Range> ranges;

    size_t current_pos = offset;
    size_t end_pos_non_included = offset + size;
    size_t remaining_size = aligned_size;

    const size_t max_size = max_file_segment_size.load();
    while (current_pos < end_pos_non_included)
    {
        auto current_file_segment_size = std::min(remaining_size, max_size);
        ranges.emplace_back(current_pos, current_pos + current_file_segment_size - 1);

        remaining_size -= current_file_segment_size;
        current_pos += current_file_segment_size;
    }

    return ranges;

size is what the caller asked for; aligned_size is the same request snapped up to the 4 MiB boundary_alignment. The loop walks the aligned extent but stops at the requested end, so the last cell can overhang the request — that overhang is the alignment, left as one segment rather than split in two.

The consequence: two readers asking for overlapping ranges of the same object almost always land on the same segment, so the second one hits.

There is also a hard “do not cache this at all” threshold, expressed as a segment state rather than a policy decision at the call site:

cpp
// src/Interpreters/FileCache/FileCache.cpp:650
    if (!ignore_bypass_threshold && bypass_cache_threshold && range.size() > bypass_cache_threshold)
    {
        auto file_segment = std::make_shared<FileSegment>(
            locked_key.getKey(), range.left, range.size(), FileSegment::State::DETACHED);
        return { file_segment };
    }

with the threshold at 256 MiB:

cpp
// src/Interpreters/FileCache/FileCache_fwd.h:17
static constexpr size_t FILECACHE_BYPASS_THRESHOLD = 256 * 1024 * 1024;

A DETACHED segment is a placeholder that holds no bytes, is not owned by the cache, and never changes state. This is how “cache nothing” is implemented: not as an error, but as a real segment that says “not me”.

The on-disk layout

A cached object is a directory; its segments are files in it:

text
  <cache_root>/Data/<first 3 chars of key>/<key>/<offset>_<size>
                            |            |      |
                            |            |      +--- one file per segment
                            |            +---------- one directory per cached object
                            +----------------------- fan-out, so no dir holds a million entries

The path is built from the key, with its first three characters used as a one-level fan-out:

cpp
// src/Interpreters/FileCache/Metadata.cpp:303
    const auto key_str = key.toString();
    const auto key_type_prefix = getKeyTypePrefix(origin.segment_type);
    if (write_cache_per_user_directory)
    {

and the non-per-user form:

cpp
// src/Interpreters/FileCache/Metadata.cpp:313
    return fs::path(path) / key_type_prefix / key_str.substr(0, 3) / key_str;

The key type prefix splits the cache into Data and System subtrees (the General type has no prefix), so a runaway query over system tables cannot evict your data.

Inside the key directory, each segment is one file whose name is the offset — plus the size, once the segment is complete:

cpp
// src/Interpreters/FileCache/Metadata.cpp:282
        case FileSegmentKind::Regular:
            /// `<offset>_<size>` once the segment is fully downloaded, just `<offset>` while
            /// it is still being written (final size not yet known).
            if (size.has_value())
                return std::to_string(offset) + "_" + std::to_string(*size);
            return std::to_string(offset);

So a partially downloaded segment is 1048576 and the same segment once complete is 1048576_33554432 — the size lives in the filename.

Why the size is in the filename

The cache’s in-memory index is not persisted: on startup ClickHouse rebuilds it by walking the cache directory. That walk has to learn each segment’s size, and the naïve way — stat() every file — is one syscall per segment, on a cache that may hold a hundred thousand segments.

Encoding the size in the name turns the walk into pure string parsing:

cpp
// src/Interpreters/FileCache/FileCache.cpp:2555
            UInt64 size_value = 0;
            if (parsed && tryParse<UInt64>(size_value, suffix))
                size_from_name = size_value;
            else
                parsed = false;

with a fallback to stat() for legacy files that predate the convention, and for files left behind by a download that was interrupted before the rename.

The rename that switches a segment from <offset> to <offset>_<size> happens once, when the download completes:

cpp
// src/Interpreters/FileCache/FileSegment.cpp:804
    /// it keeps the legacy name (the loader falls back to a `stat`). Doing it here, before publishing
    /// the `DOWNLOADED` state, also keeps `getPath` (which depends on `size_in_filename`) consistent
    /// with the file's actual on-disk name for the assertions below.
    renameToIncludeSizeInNameUnlocked(lock);

and the rename itself is the interesting part, because it also has to invalidate a file descriptor cache entry:

cpp
// src/Interpreters/FileCache/FileSegment.cpp:846
    try
    {
        fs::rename(old_path, new_path);
        size_in_filename = true;

cpp
// src/Interpreters/FileCache/FileSegment.cpp:858
        /// only prevents future opens from reusing the pre-rename descriptor. Mirror the flag handling in
        /// `removeFileSegmentImpl`: the file may have been opened with or without `O_DIRECT`.
        const int flags = getFlagsForLocalRead();
        OpenedFileCache::instance().remove(old_path, flags);
        OpenedFileCache::instance().remove(old_path, flags | O_DIRECT);
    }

This is the classic “rename invalidates a path-keyed cache” bug. A reader that opened the segment while it was still named <offset> left an entry in OpenedFileCache keyed by the old path; when the segment is later removed, only the current name is invalidated, so the stale entry would survive — and a future segment created at the same key and offset is named <offset> again while it downloads, so opening it could reuse the descriptor of a segment that has since been removed and re-created. Existing readers keep their own descriptors (the inode is unchanged across the rename), so only future opens are at risk.

Because rename(2) is atomic, a crash in the middle leaves either the old or the new name — both of which the loader parses correctly.

The in-memory LRU

The eviction order is not on disk at all. It lives in the priority queue, an std::list with the lowest priority at the head:

cpp
// src/Interpreters/FileCache/LRUFileCachePriority.h:17
/// Based on the LRU algorithm implementation, the record with the lowest priority is stored at
/// the head of the queue, and the record with the highest priority is stored at the tail.
class LRUFileCachePriority : public IFileCachePriority

cpp
// src/Interpreters/FileCache/LRUFileCachePriority.h:181
    using LRUQueue = std::list<EntryPtr>;

A hit does not move the entry; it moves the eviction cursor past it. The promotion of a segment on a hit is a single splice to the tail:

cpp
// src/Interpreters/FileCache/LRUFileCachePriority.cpp:762
    auto it = dynamic_cast<const LRUFileCachePriority::LRUIterator &>(iterator).get();
    moveEvictionPosIfEqual(it, lock);
    queue.splice(queue.end(), queue, it);
    return true;

Note that this is best-effort: it takes a tryWriteLock, and if the lock is contended the promotion is simply skipped:

cpp
// src/Interpreters/FileCache/LRUFileCachePriority.cpp:753
    assertNotSealed("tryIncreasePriority");
    /// Best-effort promotion: skip rather than block if the (per-priority) guard is contended.
    auto lock = getPriorityGuard().tryWriteLock();
    if (!lock.owns_lock())
        return false;

A cache hit must never block on a cache-internal lock — that turns a latency optimisation into a latency amplifier.

Because the LRU order is not persisted, every segment loaded at startup has the same priority, so ClickHouse shuffles the queue once, after loading:

cpp
// src/Interpreters/FileCache/FileCache.cpp:2168
    loadMetadataImpl();

    /// Shuffle file_segment_metadatas to have random order in LRUQueue
    /// as at startup all file_segment_metadatas have the same priority.
    main_priority->shuffle();

cpp
// src/Interpreters/FileCache/LRUFileCachePriority.cpp:899
    std::vector<LRUQueue::iterator> its;
    its.reserve(queue.size());
    for (auto it = queue.begin(); it != queue.end(); ++it)
        its.push_back(it);
    pcg64 generator(randomSeed());
    std::shuffle(its.begin(), its.end(), generator);
    for (auto & it : its)
        queue.splice(queue.end(), queue, it);

Without the shuffle, the eviction order after a restart would be “whatever the directory walk produced”, which is roughly creation order — and the oldest segments are exactly the ones that should go first, but also exactly the ones that a restart is most likely to have in the directory listing.

Reserving space without taking the lock every time

To write a segment into the cache, you must first reserve space for it in the cache’s accounting, and reserving may require evicting other segments. That eviction is expensive and needs the priority lock.

ClickHouse amortises it: a reader or writer reserves ahead of what it actually needs, doubling the reservation granularity each time, up to a cap:

cpp
// src/Interpreters/FileCache/FileSegment.h:35
    size_t getReserveSize(size_t size_to_reserve, size_t segment_reserve_limit, size_t max_reserve_granularity)
    {
        const size_t result = std::max(size_to_reserve, std::min({granularity, max_reserve_granularity, segment_reserve_limit}));
        granularity = std::min(max_reserve_granularity, std::max(granularity, size_to_reserve) * 2);
        return result;
    }

    void reset() { granularity = 0; }

    size_t granularity = 0;
};

The first reservation is exact; the second asks for twice as much; the third for four times as much, capped at reserve_granularity (4 MiB by default). The result is that a reader streaming a 32 MiB segment takes the cache lock a handful of times instead of once per block — and the unused part of an over-reservation is given back when the segment completes.

The cap matters: without it, one long download would reserve the whole segment on the first call and hold the difference for the entire transfer.

Residency as a walk over segments

The modern disk tier computes residency in one place: DiskCacheProvider::resolve opens one cache transaction with cache->getOrSet(...) — existing segments come back whole, virgin territory is cut by the cache’s own splitRange grid — and then walks the segments that intersect the request:

cpp
// src/IO/DiskCacheProvider.cpp:635
    while (!got_holder->empty())
    {
        /// One holder per segment; a partial segment's hit reader and miss writer SHARE it, so the
        /// segment completes once when the last of them drops.
        auto seg_holder = got_holder->popHolder();
        const FileSegment & segment = seg_holder->front();
        const auto & seg_range = segment.range();
        const size_t seg_left = seg_range.left;
        const size_t seg_end = seg_range.right + 1;

        /// A DETACHED placeholder holds no bytes and cannot take a downloader; serve it from source.
        if (segment.isDetached())
        {
            emit_uncacheable_miss(seg_left, seg_end);
            continue;
        }

        const size_t committed_end = segmentCommittedEnd(segment);

        if (committed_end > seg_left)
            emit_hit(seg_holder, seg_left, committed_end);
        if (committed_end < seg_end)
            emit_miss(seg_holder, seg_left, seg_end);
    }

The walk emits a hit over the committed prefix of a segment and a miss over its uncommitted tail, both bound to the same segment holder. That is what makes a partially-downloaded segment work: the reader takes the bytes already on disk while a writer fills the rest, and both agree on when the segment is complete. The state machine behind it is segmentCommittedEnd:

cpp
// src/IO/DiskCacheProvider.cpp:120
size_t segmentCommittedEnd(const FileSegment & segment)
{
    return segment.state() == FileSegmentState::DOWNLOADED
        ? segment.range().right + 1
        : segment.getCurrentWriteOffset();
}

Bypass mode: reading without disturbing the cache

There is a second mode, in which the cache is consulted but never modified. This is what background merges use by default: a merge reads every byte of every part it touches, and populating the cache with those bytes would evict everything the queries want. In this mode the provider only calls cache->get(...) — existing segments become hits, gaps and uncommitted tails become writer-less misses, and nothing is created, reserved or evicted.

The setting that turns merges into cache-populating reads is a table-level MergeTree setting:

sql
-- By default a background merge reads through the cache only for ranges that
-- are already cached; the rest is fetched straight from object storage.
-- This setting makes merges populate the cache as well.
ALTER TABLE events MODIFY SETTING force_read_through_cache_for_merges = 1;

The trade-off is explicit: force_read_through_cache_for_merges = 1 warms the cache but consumes capacity that interactive queries would have used.

You can watch the result:

sql
SELECT
    segment_type,
    size,
    cache_hits,
    downloaded_size
FROM system.filesystem_cache
ORDER BY cache_hits DESC
LIMIT 10;

Trade-offs

What the design buys:

Partial hits. A reader that wants 1 MiB of a 3 GiB object downloads one segment, not the object.
Bounded blast radius. One huge object cannot evict the whole cache; at most a few of its segments can.
Restart without a metadata file. The index is rebuilt by walking the directory; no separate database can disagree with the filesystem.
No stat storm. Sizes are in the names, so a 100k-segment cache loads with one readdir per directory.

What it costs:

Space amplification. The 4 MiB alignment means a 100 KiB read can pull a 4 MiB segment; for scattered tiny reads that is 40× the traffic asked for.
32 MiB granularity is coarse for small objects. An object smaller than one segment is one segment, and it is evicted as a unit.
The rename is best-effort. If it fails, that segment pays a stat on every restart, until it is evicted and re-downloaded.
The LRU order is lost on restart. The shuffle is a mitigation, not a solution: after a restart the cache has no memory of which segments were hot.
Two notions of “size”. The LRU accounts for reserved size, but for aligned on-disk size when use_real_disk_size is on, because a 4 KiB segment occupies a 4 KiB filesystem block. The cache makes that disagreement harmless rather than preventing it.

Wrapping up

The one thing to remember: the file cache is a segment cache, not a file cache. Everything else follows from that one decision.

The part people get wrong: the cache is not populated on write. An INSERT into a table on an object-storage disk does not warm it; only reads do, and only for the ranges they actually read.

What to go do about it: compare system.filesystem_cache against your access pattern. If most objects are read in full exactly once, the cache is pure overhead — set max_size small.