Data-Structures on Opensource internals

ClickHouse's LRU Cache Keeps Its Order in a Linked List

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: The cache is two containers: an unordered_map from key to cell, and a std::list of keys in recency order. Each cell stores an iterator into that list. A lookup is splice (move the node to the back), eviction is pop_front. No timestamps, no heap, no re-sorting — and the weight of every entry is tracked so the cache evicts by bytes, not by count.

Every cache in every database has the same skeleton: a map from key to value, and a rule for what to throw away when the map gets too big. The rule is the interesting part, because the obvious implementations are all bad in one of two ways.

The obvious LRU implementation stores a timestamp with each entry and, on eviction, scans the whole map for the oldest timestamp. That is O(n) per eviction, on the critical path of a request that just missed the cache. The obvious clock implementation stores a “referenced” bit and sweeps a hand over the map. That is O(1) amortised, but it needs the map to be traversable in a stable order, which a hash map is not.

ClickHouse’s answer is the classic one, and the classic one is worth studying because the details are unusually carefully done. Two containers:

 text
 
 
 unordered_map<Key, Cell> std::list<Key>
 ┌───────────────────────┐ ┌──────────────────────────┐
 │ Key ──► Cell │ │ [A] [B] [C] [D] [E] │
 │ │ │ │ ↑ │
 │ └───────────┼──iterator───►│ front = least recent │
 │ │ │ back = most recent │
 └───────────────────────┘ └──────────────────────────┘

The map gives you the value and a handle; the handle points at the node’s position in the list; the list is the eviction order. A hit is a pointer swap; an eviction is a pop from the front.

This article walks through the whole thing: the policy interface, the cell layout, the splice, the byte-weighted eviction loop, the metrics, and the thread-safe wrapper on top, including the cache-stampede protection that getOrSet() provides.

The policy interface

The cache machinery is split in two. A policy knows how to pick a victim and how to order the entries. A wrapper (CacheBase) adds a mutex, hit/miss counters, and the getOrSet dance. The policy is the interesting part, so it starts with a small abstract interface.

 cpp
 
 
// src/Common/ICachePolicy.h:22
template <typename TKey, typename TMapped, typename HashFunction = std::hash<TKey>, typename WeightFunction = EqualWeightFunction<TMapped>>
class ICachePolicy
{
public:
 using Key = TKey;
 using Mapped = TMapped;
 using MappedPtr = std::shared_ptr<Mapped>;
 using OnRemoveEntryFunction = std::function<void(size_t, const MappedPtr &)>; /// For per-item callback

 struct KeyMapped
 {
 Key key;
 MappedPtr mapped;
 };

Two things in that header are worth pausing on.

The value is a shared_ptr. Not a unique_ptr, not a value. This is deliberate: the cache hands out a reference to the stored value, and the caller may hold on to it for as long as it likes. If the cache evicts the entry, the value stays alive as long as somebody is using it. std::shared_ptr<Mapped> in the interface is the whole reason.

The default weight function counts entries, not bytes. If you do not supply a weight function, every entry weighs exactly one:

 cpp
 
 
// src/Common/ICachePolicy.h:13
template <typename T>
struct EqualWeightFunction
{
 size_t operator()(const T &) const
 {
 return 1;
 }
};

So “max_size_in_bytes” with the default weight function really means “max number of entries”. Every real cache in the tree supplies a weight function instead — the mark cache weighs a file’s marks, the query result cache weighs the bytes a result actually allocated.

There is also an interface subtlety that matters a lot in practice: contains() is explicitly not an access.

 cpp
 
 
// src/Common/ICachePolicy.h:55
 /// Like `get(key) != nullptr`, but doesn't consider it an "access".
 /// E.g. doesn't move it to the front of LRU list.
 /// Not allowed to allocate memory (see PageCache::contains).
 virtual bool contains(const Key & key) const = 0;

A cache that answers “is this key resident?” must not, as a side effect, reorder its eviction order. If it did, a monitoring loop that polls system.mark_cache would silently reshuffle the LRU list under the workload’s feet. And it must not allocate, because some callers use it on a path where allocation is not allowed.

The cell: the whole trick in nine words

Here is the entire data structure:

 cpp
 
 
// src/Common/LRUCachePolicy.h:203
private:
 using LRUQueue = std::list<Key>;
 using LRUQueueIterator = typename LRUQueue::iterator;

 LRUQueue queue;

 struct Cell
 {
 MappedPtr value;
 size_t size{};
 LRUQueueIterator queue_iterator;
 };

 using Cells = std::unordered_map<Key, Cell, HashFunction>;

 Cells cells;

That is it. The entire “recency ordering” mechanism is the third field of Cell: an iterator into the linked list.

The design decision that makes it work is that the list stores keys, not cells. The list node is therefore just a Key (plus two list pointers), and the map holds the cell which points back into the list. The alternative — a list of cells, with the map storing pointers to cells — would be equivalent in asymptotics but would allocate the cell on the heap and require the map to store pointers, making the map’s values unstable under rehashing.

Storing the key in the list has one visible cost: the key is stored twice, once in the map and once in the list node. For a cache with a 16-byte key, that is 16 bytes of pure overhead per entry. ClickHouse accepts that trade because it buys a very cheap hit path.

Note also that size is cached in the cell rather than recomputed. The WeightFunction is documented as a pure function of the value:

 cpp
 
 
// src/Common/LRUCachePolicy.h:11
/// Cache policy LRU evicts entries which are not used for a long time. Also see cache policy SLRU for reference.
/// WeightFunction is a functor that takes Mapped as a parameter and returns "weight" (approximate size) of that value.
/// Cache starts to evict entries when their total weight exceeds max_size_in_bytes.
/// Value weight should not change after insertion.

“Value weight should not change after insertion” is a contract on the user of the cache. Because the weight is snapshotted at insert time, a value that grows afterwards will be mis-accounted. That is why the query result cache weighs an entry by the bytes its chunks actually allocated, not by some estimate that might drift.

The hit path: one splice

Here is get() in full:

 cpp
 
 
// src/Common/LRUCachePolicy.h:123
 MappedPtr get(const Key & key) override
 {
 auto it = cells.find(key);
 if (it == cells.end())
 return {};

 Cell & cell = it->second;

 /// Move the key to the end of the queue. The iterator remains valid.
 queue.splice(queue.end(), queue, cell.queue_iterator);

 return cell.value;
 }

Three lines of work:

cells.find(key) — a hash lookup.
queue.splice(queue.end(), queue, cell.queue_iterator) — move the node representing this key to the back of the list.
Return the shared_ptr, bumping its refcount.

The splice is the interesting call. std::list::splice relinks nodes without allocating, copying, or moving any elements. It is O(1). The comment “The iterator remains valid” is the load-bearing part: after the splice, the cell’s stored queue_iterator still points at the (now moved) node, so a subsequent splice on the same cell works.

The list’s invariant is therefore:

 text
 
 
 front ............................................... back
 [oldest access] ... [newest access]

 get(k) ==> move k's node to the back
 set(k) ==> move k's node to the back
 evict ==> pop the front

Front is the least-recently-used entry; back is the most recent. Eviction pops the front.

Note what get() does not do: it does not touch cell.size, does not recompute the weight, and does not update any metric. A hit is a hash lookup and a pointer relink. That is the entire cost of an LRU hit.

The insert path

set() is a little longer because it has to handle the case where the key is already resident (an update rather than an insert):

 cpp
 
 
// src/Common/LRUCachePolicy.h:156
 void set(const Key & key, const MappedPtr & mapped) override
 {
 const size_t old_size_in_bytes = current_size_in_bytes;
 const size_t old_size = cells.size();

 auto [it, inserted] = cells.emplace(std::piecewise_construct,
 std::forward_as_tuple(key),
 std::forward_as_tuple());

 Cell & cell = it->second;

 if (inserted)
 {
 try
 {
 cell.queue_iterator = queue.insert(queue.end(), key);
 }
 catch (...)
 {
 cells.erase(it);
 throw;
 }
 }
 else
 {
 current_size_in_bytes -= cell.size;
 queue.splice(queue.end(), queue, cell.queue_iterator);
 }

 cell.value = mapped;
 cell.size = cell.value ? weight_function(*cell.value) : 0;
 current_size_in_bytes += cell.size;

 CurrentMetrics::add(current_size_in_bytes_metric, static_cast<Int64>(current_size_in_bytes) - old_size_in_bytes);
 CurrentMetrics::add(count_metric, static_cast<Int64>(cells.size()) - old_size);

 removeOverflow();
 }

Several details are worth calling out.

piecewise_construct with a tuple. This is the standard way to insert into an unordered_map without constructing a Cell you are about to throw away. emplace with piecewise_construct constructs the key and the mapped value in place in the node, so an update does not allocate a temporary Cell and then move it.

The try/catch around queue.insert. If the list insertion throws (bad alloc), the map entry is erased so the two containers stay consistent. If the list insert succeeded but the map insert had thrown, you would have a list node with no cell pointing at it — an entry that could never be evicted, and a permanent leak of its weight. The invariant is “the map and the list agree”, and this is where it is defended.

The update path subtracts the old weight. current_size_in_bytes -= cell.size before the new weight is computed. The new weight may be completely different (the value was replaced), and the total has to reflect that. Only after the new weight is known is the old one removed and the new one added — the two lines are separated by the weight_function call so that the subtraction never double-counts.

weight_function(*cell.value) is called on the dereferenced pointer. The dereference is deliberate: the weight function takes the value, not the pointer, and the null check cell.value ? ... : 0 guards the case where the policy is asked to store a null pointer.

Metrics are updated by delta, not by absolute value. CurrentMetrics::add takes the difference between the new and old totals, and this pattern is used everywhere in ClickHouse’s metrics code: the metric is a global counter shared by every instance of a cache, so each instance adds its own delta.

Eviction is triggered at the end of the insert, not lazily at the next lookup. The cache is always (just) within its limit, never over.

The eviction loop

 cpp
 
 
// src/Common/LRUCachePolicy.h:231
 void removeOverflow()
 {
 const size_t old_size_in_bytes = current_size_in_bytes;
 const size_t old_size = cells.size();

 size_t current_weight_lost = 0;
 size_t queue_size = cells.size();

 while ((current_size_in_bytes > max_size_in_bytes || (max_count != 0 && queue_size > max_count)) && (queue_size > 0))
 {
 const Key & key = queue.front();

 auto it = cells.find(key);
 if (it == cells.end())
 std::terminate(); // Queue became inconsistent

 const auto & cell = it->second;

 current_size_in_bytes -= cell.size;
 current_weight_lost += cell.size;
 /// Update cache-specific metrics.
 if (on_remove_entry_function)
 on_remove_entry_function(cell.size, cell.value);

 cells.erase(it);
 queue.pop_front();
 --queue_size;
 }

 if (current_size_in_bytes > (1ull << 63))
 std::terminate(); // Queue became inconsistent

 CurrentMetrics::sub(current_size_in_bytes_metric, old_size_in_bytes - current_size_in_bytes);
 CurrentMetrics::sub(count_metric, old_size - cells.size());
 }

Note the eviction condition: bytes over the limit OR count over the limit. max_count == 0 means “no count limit”, which is why the check is max_count != 0 && queue_size > max_count. Most ClickHouse caches pass max_count = 0 and are bounded purely by bytes, because the byte cost of a cached object varies by orders of magnitude (one mark file versus one query result).

The eviction loop is a while, not an if: a single insert can push a cache over its limit by more than one entry’s weight, and the loop evicts until it fits again.

Two std::terminate() calls guard the invariant. If the list’s front key is not in the map, the two containers have diverged and continuing would corrupt the cache; terminating loudly is the right response to a bug that should be impossible. The second one — current_size_in_bytes > (1ull << 63) — catches weight accounting underflow: if the accounting ever went negative it would wrap around to a huge positive number and the cache would stop evicting entirely.

The on_remove_entry_function callback is where per-entry eviction accounting happens. The mark cache uses it to count evicted files and evicted marks:

 cpp
 
 
// src/Storages/MarkCache.h:88
 void onEntryRemoval(const size_t weight_loss, const MappedPtr & mapped_ptr) override
 {
 /// File is the key of MarkCache, each removal means eviction of 1 file from the cache.
 ProfileEvents::increment(ProfileEvents::MarkCacheEvictedFiles);
 ProfileEvents::increment(ProfileEvents::MarkCacheEvictedBytes, weight_loss);

 const auto * marks_in_compressed_file = static_cast<const MarksInCompressedFile *>(mapped_ptr.get());
 ProfileEvents::increment(ProfileEvents::MarkCacheEvictedMarks, marks_in_compressed_file->getNumberOfMarks());
 }

That is a genuinely useful design: the policy knows when it evicts and what it evicted, but not what an eviction means. The subclass decides. And it means the eviction is observable from SQL:

 sql
 
 
SELECT sum(value) AS evicted_files
FROM system.events
WHERE event LIKE 'MarkCacheEvicted%';

The thread-safe wrapper

The policy is not thread-safe — it has no lock at all. Thread safety comes from CacheBase, which wraps a policy in a std::mutex:

 cpp
 
 
// src/Common/CacheBase.h:103
 MappedPtr get(const Key & key)
 {
 std::lock_guard lock(mutex);
 auto res = cache_policy->get(key);
 if (res)
 ++hits;
 else
 ++misses;
 return res;
 }

One mutex for the whole cache. That is a coarse lock, and it is the right choice here: the critical section is a hash lookup and a list splice, both nanoseconds, and a fine-grained lock would cost more in bookkeeping than the operation it protects. The values themselves are shared_ptrs and are not protected — a caller holding a pointer to an evicted value can keep using it safely, because the value’s lifetime is managed by the refcount, not by the cache’s mutex.

CacheBase also selects the policy by name, which is how the server setting mark_cache_policy reaches the cache:

 cpp
 
 
// src/Common/CacheBase.h:63
 /// Use this ctor if the user should be able to configure the cache policy and cache sizes via settings. Supports only general-purpose policies LRU, SLRU and SIEVE.
 explicit CacheBase(
 std::string_view cache_policy_name,
 CurrentMetrics::Metric size_in_bytes_metric,
 CurrentMetrics::Metric count_metric,
 size_t max_size_in_bytes,
 size_t max_count,
 double size_ratio)
 {
 auto on_remove_entry_function = makeOnRemoveEntryFunction();

 if (cache_policy_name.empty())
 {
 cache_policy_name = DEFAULT_CACHE_POLICY;
 }

 if (cache_policy_name == "LRU")
 {
 using LRUPolicy = LRUCachePolicy<TKey, TMapped, HashFunction, WeightFunction>;
 cache_policy = std::make_unique<LRUPolicy>(size_in_bytes_metric, count_metric, max_size_in_bytes, max_count, on_remove_entry_function);
 }
 else if (cache_policy_name == "SLRU")
 {
 using SLRUPolicy = SLRUCachePolicy<TKey, TMapped, HashFunction, WeightFunction>;
 cache_policy = std::make_unique<SLRUPolicy>(size_in_bytes_metric, count_metric, max_size_in_bytes, max_count, size_ratio, on_remove_entry_function);
 }
 else if (cache_policy_name == "SIEVE")
 {
 using SIEVEPolicy = SIEVECachePolicy<TKey, TMapped, HashFunction, WeightFunction>;
 cache_policy = std::make_unique<SIEVEPolicy>(size_in_bytes_metric, count_metric, max_size_in_bytes, max_count, on_remove_entry_function);
 }
 else
 throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown cache policy name: {}", cache_policy_name);
 }

The default is not LRU:

 cpp
 
 
// src/Common/CacheBase.h:45
 static constexpr auto NO_MAX_COUNT = 0uz;
 static constexpr auto DEFAULT_SIZE_RATIO = 0.5l;
 static constexpr auto DEFAULT_CACHE_POLICY = "SLRU";

SLRU, with a protected-queue ratio of 0.5 — half the cache is reserved for entries that have been hit at least twice. That is a whole separate article, but the reason to mention it here is that LRU is the exception, not the rule. Every default in the tree is SLRU:

 cpp
 
 
// src/Core/Defines.h:106
/// Default maximum (total and entry) sizes and policies of various caches
static constexpr auto DEFAULT_UNCOMPRESSED_CACHE_POLICY = "SLRU";
static constexpr auto DEFAULT_UNCOMPRESSED_CACHE_MAX_SIZE = 0_MiB;
static constexpr auto DEFAULT_UNCOMPRESSED_CACHE_SIZE_RATIO = 0.5l;
static constexpr auto DEFAULT_MARK_CACHE_POLICY = "SLRU";
static constexpr auto DEFAULT_MARK_CACHE_MAX_SIZE = 5_GiB;
static constexpr auto DEFAULT_MARK_CACHE_SIZE_RATIO = 0.5l;

You can see it from SQL, because the policy names are server settings:

 sql
 
 
SELECT name, value, default
FROM system.server_settings
WHERE name IN ('mark_cache_policy', 'mark_cache_size', 'mark_cache_size_ratio');

 text
 
 
┌─name──────────────────┬─value─────────┬─default─────────┐
│ mark_cache_policy │ SLRU │ SLRU │
│ mark_cache_size │ 5368709120 │ 5368709120 │
│ mark_cache_size_ratio│ 0.5 │ 0.5 │
└──────────────────────┴───────────────┴─────────────────┘

(mark_cache_size is a UInt64, so it prints as a byte count: 5368709120 is 5 GiB, the default from DEFAULT_MARK_CACHE_MAX_SIZE.)

And you can change it:

 sql
 
SYSTEM RELOAD CONFIG;

with mark_cache_policy set to LRU, SLRU or SIEVE in the server config. The setting’s own documentation spells out the three choices:

 cpp
 
 
// src/Core/ServerSettings.cpp:653
 DECLARE(String, mark_cache_policy, DEFAULT_MARK_CACHE_POLICY, R"(Mark cache eviction policy name. Possible values: `LRU`, `SLRU`, `SIEVE`.)", 0) \

Cache stampede: the insert token

There is one more thing CacheBase does that is worth its own section, because it is the most subtle code in the file.

Consider a cache in front of an expensive resource — a DNS resolver, a mark file on disk, an S3 object. Ten threads all miss at the same moment. Without protection, all ten call the loader, all ten do the expensive work, and all ten try to insert. That is a cache stampede, and it is exactly what happens when a popular key expires.

CacheBase::getOrSet() prevents it with a per-key insert token:

 cpp
 
 
// src/Common/CacheBase.h:185
 template <typename LoadFunc>
 std::pair<MappedPtr, bool> getOrSet(const Key & key, LoadFunc && load_func)
 {
 InsertTokenHolder token_holder;
 {
 std::lock_guard cache_lock(mutex);
 auto val = cache_policy->get(key);
 if (val)
 {
 ++hits;
 return std::make_pair(val, false);
 }

 auto & token = insert_tokens[key];
 if (!token)
 token = std::make_shared<InsertToken>(*this);

 token_holder.acquire(&key, token, cache_lock);
 }

 InsertToken * token = token_holder.token.get();

 std::lock_guard token_lock(token->mutex);

 token_holder.cleaned_up = token->cleaned_up;

 if (token->value)
 {
 /// Another thread already produced the value while we waited for token->mutex.
 ++hits;
 return std::make_pair(token->value, false);
 }

 ++misses;
 token->value = load_func();

 std::lock_guard cache_lock(mutex);

 /// Insert the new value only if the token is still in present in insert_tokens.
 /// (The token may be absent because of a concurrent clear() call).
 bool result = false;
 auto token_it = insert_tokens.find(key);
 if (token_it != insert_tokens.end() && token_it->second.get() == token)
 {
 cache_policy->set(key, token->value);
 result = true;
 }

 if (!token->cleaned_up)
 token_holder.cleanup(token_lock, cache_lock);

 return std::make_pair(token->value, result);
 }

The protocol, in order:

Under the cache mutex, look up the key. Hit? Return it.
Still under the cache mutex, find or create an InsertToken for this key. Because this happens under the mutex, exactly one token exists per key.
Release the cache mutex. The loader will be called without the cache lock held — this is essential, because the loader might itself touch the cache, or might block for a long time (a network read), and holding the cache lock through it would serialise the entire cache.
Take the token’s own mutex. This is a different lock, per key. All threads that missed on this key are now queued on this one lock.
Check token->value. If another thread already loaded it while we were waiting, return that value. We counted a hit, not a miss.
Otherwise call load_func(). Because we hold the token’s mutex, exactly one thread runs the loader for this key.
Re-take the cache mutex and insert, but only if our token is still the one registered for this key. A concurrent clear() may have wiped insert_tokens, in which case we do not insert (the value would be orphaned in a cleared cache).

The token itself:

 cpp
 
 
// src/Common/CacheBase.h:374
 /// Represents pending insertion attempt.
 struct InsertToken
 {
 explicit InsertToken(CacheBase & cache_) : cache(cache_) {}

 std::mutex mutex;
 bool cleaned_up TSA_GUARDED_BY(mutex) = false;
 MappedPtr value TSA_GUARDED_BY(mutex);

 CacheBase & cache;
 size_t refcount = 0; /// Protected by the cache mutex
 };

And the holder that makes sure the token gets removed from the map:

 cpp
 
 
// src/Common/CacheBase.h:389
 /// This class is responsible for removing used insert tokens from the insert_tokens map.
 /// Among several concurrent threads the first successful one is responsible for removal. But if they all
 /// fail, then the last one is responsible.
 struct InsertTokenHolder
 {
 const Key * key = nullptr;
 std::shared_ptr<InsertToken> token;
 bool cleaned_up = false;

 InsertTokenHolder() = default;

 void acquire(const Key * key_, const std::shared_ptr<InsertToken> & token_, std::lock_guard<std::mutex> & /* cache_lock */)
 TSA_NO_THREAD_SAFETY_ANALYSIS // disabled only because we can't reference the parent-level cache mutex from here
 {
 key = key_;
 token = token_;
 ++token->refcount;
 }

The refcount is the interesting bit. Suppose ten threads miss on key K. They all get the same token, all increment its refcount to 10. One of them runs the loader, inserts the value, and erases the token from insert_tokens. The other nine are waiting on the token’s mutex; when they wake up they see token->value is set and return it. Then each of them, as its InsertTokenHolder goes out of scope, decrements the refcount. Only the last one (refcount reaches zero) removes the token from the map.

Wait — the first thread already erased it. So why the refcount? Because of the failure path: if load_func() throws, the exception propagates out of getOrSet before the cleanup code runs. The InsertTokenHolder’s destructor is the backstop:

 cpp
 
 
// src/Common/CacheBase.h:416
 ~InsertTokenHolder()
 {
 if (!token)
 return;

 if (cleaned_up)
 return;

 std::lock_guard token_lock(token->mutex);

 if (token->cleaned_up)
 return;

 std::lock_guard cache_lock(token->cache.mutex);

 --token->refcount;
 if (token->refcount == 0)
 cleanup(token_lock, cache_lock);
 }

The destructor runs on the exception path. It takes the token lock (so it does not race with a thread that is currently loading), then the cache lock, and only removes the token if it was the last reference. If ten threads all had the loader throw, the token survives until the tenth one unwinds, and the next thread that misses on K gets a fresh token and retries the load. The comment says it exactly: “Among several concurrent threads the first successful one is responsible for removal. But if they all fail, then the last one is responsible.”

So the guarantees are:

At most one thread runs the loader per key, at a time.
A failed load does not poison the cache — the next caller retries.
The token map does not leak entries.

The price is a per-key std::mutex allocation on the miss path and a fair amount of subtle lock ordering. Note the lock order is always token mutex → cache mutex, never the reverse. The cache mutex is released before the token mutex is taken, so there is no path that holds the cache mutex while waiting for a token mutex. That is what makes the ordering deadlock-free.

Callers use it like this:

 cpp
 
 
// src/Common/DNSResolver.cpp:321
 auto [result, _ ] = impl->cache_host.getOrSet(host, [&host, this]() {return std::make_shared<DNSResolver::CacheEntry>(getResolvedIPAddressesWithFiltering(host), std::chrono::system_clock::now());});

The loader runs outside the cache lock, which is why the page cache can release a memory-tracker blocker before loading:

 cpp
 
 
// src/Common/PageCache.cpp:103
 std::tie(result, miss) = shard.getOrSet(key_hash, [&]() -> MappedPtr
 {
 /// At this point CacheBase is not holding the mutex, so it's ok to let MemoryTracker
 /// call autoResize.
 blocker.reset();

Trade-offs

What the linked list buys you. An O(1) hit that is also an O(1) recency update, with no allocation, no timestamp comparison, and no separate eviction structure to keep in sync. The cost of the hit is one pointer relink — a few nanoseconds — which is why the design tolerates doing it on every single lookup, including on paths that only wanted to check residency.

What it costs. The key is stored twice (map + list node), so a cache with a large key has a noticeable per-entry overhead. The whole cache is behind one mutex, so a cache used by many threads is a contention point — but the critical section is tiny.

Why the weight is snapshotted. It makes eviction O(1) per victim instead of O(size of value). The price is a contract: values must not change size after insertion. In practice ClickHouse works around this by weighing entries by their allocated size and by treating the cached object as immutable.

Why contains is not get. Because a residency check must not be an access. This is a small thing that shows up in real code: the page cache’s contains is used on paths where reordering the LRU list would be actively harmful.

Why SLRU is the default and LRU is not. Because a scan is the common case, not the exception. A query that touches ten thousand mark files will, under plain LRU, evict every mark file that the other queries were relying on. LRU is optimal for a workload where the access pattern is stationary; it is catastrophic for a workload with one dominant sequential scan. That is the next article.

Wrapping up

The one thing to remember: the cache is not “a map with timestamps”. It is a map of cells, and each cell holds an iterator into a linked list of keys. The recency order is not a property you compute; it is a property you maintain with one splice.

The part people get wrong: the weight is captured at insert time. If you put a value into the cache and then grow it, the cache’s byte accounting is wrong from that moment on, and it will evict the wrong things.

What to go do about it: look at SELECT metric, value FROM system.metrics WHERE metric LIKE '%Cache%' and compare the cache’s byte usage to its configured size. If a cache is chronically at its limit and its hit rate is low, that is usually a scan problem, not an LRU problem — and the fix is the policy setting, not the implementation.

ClickHouse Recycles Small Objects Through Per-Size Free Lists

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: A plain arena can allocate but not free. ArenaWithFreeLists adds fourteen free lists — one per power-of-two size class from 8 to 65536 bytes — and stores the “next free block” pointer inside the freed block itself, using a union. Recycling a 20-byte string is: find its size class, unlink the head of the list, done. No malloc, no free, no metadata.

Memory management in a database is a fight between two contradictory requirements. You want to allocate millions of tiny objects quickly — strings, keys, small arrays — without paying the general-purpose allocator’s price on every one. And you want to give some of them back when they become garbage, without the bookkeeping that makes a general-purpose allocator general-purpose.

ClickHouse’s answer is a pair of classes. Arena is the classic bump allocator: allocate fast, never free, release everything at once. ArenaWithFreeLists is the interesting one: it adds per-size-class free lists so that individual allocations can be recycled.

The trick that makes it work is in the details, and the details are where all the good stuff is.

The problem: an arena cannot free

Arena is a list of memory chunks with a bump pointer:

 cpp
 
 
// src/Common/Arena.h:23
/** Memory pool to append something. For example, short strings.
 * Usage scenario:
 * - put lot of strings inside pool, keep their addresses;
 * - addresses remain valid during lifetime of pool;
 * - at destruction of pool, all memory is freed;
 * - memory is allocated and freed by large MemoryChunks;
 * - freeing parts of data is not possible (but look at ArenaWithFreeLists if you need);
 */
class Arena : private boost::noncopyable

That comment is a complete specification, including the limitation. The allocation path is a pointer bump:

 cpp
 
 
// src/Common/Arena.h:213
 /// Get piece of memory, without alignment.
 /// Note: we expect it will return a non-nullptr even if the size is zero.
 char * alloc(size_t size)
 {
 used_bytes += size;
 if (unlikely(head.empty() || size > head.remaining()))
 addMemoryChunk(size);

 char * res = head.pos;
 head.pos += size;
 ASAN_UNPOISON_MEMORY_REGION(res, size + pad_right);
 return res;
 }

Two things to notice.

The pad_right padding. pad_right is PADDING_FOR_SIMD - 1, which is 63 bytes:

 cpp
 
 
// src/Common/Arena.h:34
 static constexpr size_t pad_right = PADDING_FOR_SIMD - 1;

Every allocation is followed by 63 bytes of untouched (and ASAN-poisoned) padding. That exists so that SIMD code can read 64 bytes starting at any allocation boundary without reading out of bounds. ClickHouse’s string functions and hash functions read in 64-byte blocks, and without the padding every SIMD loop would need a tail-handling branch. This is a real, measurable trade: 63 bytes of slack per allocation, in exchange for branch-free SIMD.

The used_bytes accounting. The arena tracks two numbers: allocated_bytes (how much it took from the OS) and used_bytes (how much it handed out). The difference is the slack.

The chunk list. Chunks are never reused or returned individually:

 cpp
 
 
// src/Common/Arena.h:36
 /// Contiguous MemoryChunk of memory and pointer to free space inside it. Member of single-linked list.
 struct alignas(16) MemoryChunk : private Allocator<false> /// empty base optimization
 {
 char * begin = nullptr;
 char * pos = nullptr;
 char * end = nullptr; /// does not include padding.

 std::unique_ptr<MemoryChunk> prev;

Note alignas(16) and the Allocator<false> base — the false is clear_memory_, meaning the arena does not zero the memory it hands out. Arena memory is assumed to be written before it is read.

The growth policy is where the arena gets interesting, because the naive version of “double the chunk size” has a pathological case:

 cpp
 
 
// src/Common/Arena.h:103
 /// Uses exponential buffer growth below `linear_growth_threshold` and linear growth above it
 /// to limit unused capacity.
 template <typename Size>
 Size nextSize(Size min_next_size) const
 {
 Size size_after_grow = 0;

 if (head.empty())
 {
 size_after_grow = std::max<Size>(min_next_size, initial_size);
 }
 else if (head.size() < linear_growth_threshold)
 {
 size_after_grow = std::max(min_next_size, Size(head.size()) * growth_factor);
 }
 else
 {
 /// With `allocContinue`, appending small amounts can repeatedly copy the accumulated data
 /// into a new `MemoryChunk`. Growing by similarly small amounts would make the number of
 /// copies proportional to the serialized size, resulting in quadratic work. Rounding the
 /// next size up to `linear_growth_threshold` makes these copies less frequent.
 size_after_grow = ((min_next_size + linear_growth_threshold - 1)
 / linear_growth_threshold) * linear_growth_threshold;
 }

 chassert(size_after_grow >= min_next_size);
 return roundUpToPageSize(size_after_grow, page_size);
 }

Exponential growth (doubling, growth_factor = 2) up to linear_growth_threshold (128 MiB by default), and linear growth above it. The reason is allocContinue, which appends to the last allocation and sometimes has to copy the whole thing into a new chunk. Under exponential growth, repeatedly appending small amounts would copy the accumulated data O(log n) times, which is fine — but under a regime where each copy is proportional to the total size, the copies become quadratic. Rounding the next chunk size up to a multiple of the threshold makes the copies rare.

Both sizes are finally rounded up to a page boundary, because the underlying allocator wants page-aligned memory.

The free-list arena

Now the interesting class. ArenaWithFreeLists wraps an Arena and adds free lists:

 cpp
 
 
// src/Common/ArenaWithFreeLists.h:16
/** Unlike Arena, allows you to release (for later re-use)
 * previously allocated (not necessarily just recently) chunks of memory.
 * For this, the requested size is rounded up to the power of two
 * (or up to 8, if less, or using memory allocation outside Arena if the size is greater than 65536).
 * When freeing memory, for each size (14 options in all: 8, 16 ... 65536),
 * a single-linked list of free blocks is kept track.
 * When allocating, we take the head of the list of free blocks,
 * or, if the list is empty - allocate a new block using Arena.
 */
class ArenaWithFreeLists : private Allocator<false>, private boost::noncopyable
{
private:
 /// If the block is free, then the pointer to the next free block is stored at its beginning, or nullptr, if there are no more free blocks.
 /// If the block is used, then some data is stored in it.
 union Block
 {
 Block * next;
 char data[0];
 };

 /// The maximum size of a piece of memory that is allocated with Arena. Otherwise, we use Allocator directly.
 static constexpr size_t max_fixed_block_size = 65536;

 /// Get the index in the freelist array for the specified size.
 static size_t findFreeListIndex(const size_t size)
 {
 return size <= 8 ? 2 : bitScanReverse(size - 1);
 }

 /// Arena is used to allocate blocks that are not too large.
 Arena pool;

 /// Lists of free blocks. Each element points to the head of the corresponding list, or is nullptr.
 /// The first two elements are not used, but are intended to simplify arithmetic.
 Block * free_lists[16] {};

Four ideas in thirty lines.

1. The union: the free-list link lives in the dead block

 cpp
 
 
// src/Common/ArenaWithFreeLists.h:30
 union Block
 {
 Block * next;
 char data[0];
 };

A freed block’s first eight bytes are reinterpreted as a pointer to the next free block. The block’s own payload is dead — nothing will read it again — so its storage is free to hold the list link.

This is the reason the whole scheme is cheap. A conventional free list needs a separate node per block (extra allocation, extra memory) or a side table. Here, the free block is the node. The free list is a singly linked list threaded through the memory it manages.

char data[0] is a zero-length array, a C89-ism that GCC and Clang accept as “a pointer to where the payload would start”. The union guarantees that data and next alias the same address, so free_block_ptr->data is the payload pointer and free_block_ptr->next is the list link.

2. Size classes: fourteen buckets

free_lists[16] is an array of 16 list heads. The comment says “The first two elements are not used, but are intended to simplify arithmetic.” Indices 2 through 15 are the fourteen real size classes: 8, 16, 32, …, 65536 bytes.

The mapping from a requested size to a class is:

 cpp
 
 
// src/Common/ArenaWithFreeLists.h:40
 static size_t findFreeListIndex(const size_t size)
 {
 return size <= 8 ? 2 : bitScanReverse(size - 1);
 }

bitScanReverse(x) is the “find the highest set bit” operation:

 cpp
 
 
// src/Common/BitHelpers.h:66
/** Returns log2 of number, rounded down.
 * Compiles to single 'bsr' instruction on x86.
 * For zero argument, result is unspecified.
 */
template <typename T>
inline uint32_t bitScanReverse(T x)
{
 return (std::max<size_t>(sizeof(T), sizeof(unsigned int))) * 8 - 1 - getLeadingZeroBitsUnsafe(x);
}

So findFreeListIndex(size) is floor(log2(size - 1)) for size > 8. Let’s check it against the size classes:

requested size	`size - 1`	`bitScanReverse`	index	block size
8	—	—	2	8
9	8	3	3	16
16	15	3	3	16
17	16	4	4	32
65536	65535	15	15	65536

The size - 1 is the classic “round up to a power of two” trick: for size = 16, size - 1 = 15 has its highest bit at position 3, so the index is 3, and the block is 1 << (3 + 1) = 16 bytes. For size = 17, size - 1 = 16 has its highest bit at position 4, index 4, block 1 << 5 = 32 bytes.

The index is one less than the log of the block size because index 2 is special-cased to mean “8 bytes” — the size <= 8 branch — and 8 is 2^3, so index 3 would be 16 bytes if the arithmetic were 1 << index. With 1 << (index + 1): index 2 → 8, index 3 → 16, index 15 → 65536. Correct.

The cost of rounding up is internal fragmentation: a 17-byte request occupies 32 bytes. For a dictionary with millions of short keys, that is the price of being able to recycle.

3. The freelist array is a fixed-size array on the stack

Block * free_lists[16] {} — 128 bytes, value-initialised to null. No allocation, no hash map, no indirection. The size class of a request is computed with one instruction (bsr), and the resulting list head is at a fixed offset in an array. That is about as cheap as a size class lookup can get.

4. Big allocations bypass the arena entirely

 cpp
 
 
// src/Common/ArenaWithFreeLists.h:37
 static constexpr size_t max_fixed_block_size = 65536;

Requests above 64 KiB go straight to Allocator<false>::alloc — the general purpose allocator — and are freed back to it. The arena only handles the small stuff, where the recycling machinery pays for itself. A 64 KiB block is big enough that the malloc overhead is negligible relative to the block size.

The allocation path

 cpp
 
 
// src/Common/ArenaWithFreeLists.h:60
 char * alloc(const size_t size)
 {
 if (size > max_fixed_block_size)
 return static_cast<char *>(Allocator<false>::alloc(size));

 /// find list of required size
 const auto list_idx = findFreeListIndex(size);

 /// If there is a free block.
 if (auto & free_block_ptr = free_lists[list_idx])
 {
 /// Let's take it. And change the head of the list to the next
 /// item in the list. We poisoned the free block before putting
 /// it into the free list, so we have to unpoison it before
 /// reading anything.
 ASAN_UNPOISON_MEMORY_REGION(free_block_ptr,
 std::max(size, sizeof(Block)));

 auto * const res = free_block_ptr->data;
 free_block_ptr = free_block_ptr->next;
 return res;
 }

 /// no block of corresponding size, allocate a new one
 return pool.alloc(1ULL << (list_idx + 1));
 }

Read it as four steps.

Too big? Hand it to the general allocator.
Find the class. One bsr.
Is there a free block? If the list head is non-null:
- Unpoison the memory (see below).
- res = free_block_ptr->data — the payload pointer.
- free_block_ptr = free_block_ptr->next — pop the head. The list is threaded through the block, so popping it means reading the next pointer out of the block we are about to hand out.
No free block? Ask the arena for a new one, of exactly the class size: pool.alloc(1ULL << (list_idx + 1)).

Step 3’s third line is the elegant bit: because the list link is stored in the block, popping the head and getting the payload pointer are the same memory read. There is no separate list node to free.

Note the size passed to pool.alloc is the rounded size, so the arena hands out a block that exactly matches the class. The arena’s own padding still applies (the pool is an Arena, whose alloc adds pad_right).

The free path

 cpp
 
 
// src/Common/ArenaWithFreeLists.h:87
 void free(char * ptr, const size_t size)
 {
 if (size > max_fixed_block_size)
 {
 Allocator<false>::free(ptr, size);
 return;
 }

 /// find list of required size
 const auto list_idx = findFreeListIndex(size);

 /// Insert the released block into the head of the list.
 auto & free_block_ptr = free_lists[list_idx];
 auto * const old_head = free_block_ptr;
 free_block_ptr = reinterpret_cast<Block *>(ptr);
 free_block_ptr->next = old_head;

 /// The requested size may be less than the size of the block, but
 /// we still want to poison the entire block.
 /// Strictly speaking, the free blocks must be unpoisoned in
 /// destructor, to support an underlying allocator that doesn't
 /// integrate with asan. We don't do that, and rely on the fact that
 /// our underlying allocator is Arena, which does have asan integration.
 ASAN_POISON_MEMORY_REGION(ptr, 1ULL << (list_idx + 1));
 }

The free path is the mirror image: compute the class, write the old head into the block’s next field, and make the block the new head.

Two subtleties:

The caller must pass the size. free(ptr, size) needs the size to recompute the class. If the caller passes a different size than it allocated with, the block goes into the wrong list and will later be handed out for a size it does not fit. Every caller in the tree passes the same size it allocated with.

The ASAN poisoning is over the whole block, not the requested size. The comment explains why: the block in the free list is bigger than the request (the rounded-up size), and the unused tail must also be poisoned, or ASAN would not catch a use-after-free that reads into the tail.

And the poisoning creates an obligation on the allocation path: you cannot read the next pointer out of a poisoned block, because ASAN would report a read of poisoned memory. That is why alloc unpoisons before reading free_block_ptr->next:

 cpp
 
 
// src/Common/ArenaWithFreeLists.h:75
 ASAN_UNPOISON_MEMORY_REGION(free_block_ptr,
 std::max(size, sizeof(Block)));

std::max(size, sizeof(Block)) — unpoison at least size bytes (the caller’s payload) and at least sizeof(Block) (8 bytes, so the next pointer can be read). The two constraints are different: the payload must be unpoisoned for the caller to write to, and the link must be unpoisoned for us to read.

The comment about the destructor admits a real limitation: strictly, the free blocks should be unpoisoned when the arena is destroyed, so that an underlying allocator without ASAN integration does not receive poisoned memory. The code relies on the underlying Arena having ASAN integration, which it does because Arena::alloc unpoisons on allocation and Arena’s destructor hands the chunk back to Allocator, which… let me check.

Actually, this matters: ArenaWithFreeLists inherits privately from Allocator<false>, and the Arena member’s destructor frees the chunks back to Allocator<false>::free, which calls __real_free. If the chunks contain poisoned free-list blocks, ASAN would complain when the memory is freed. Let me look at how the arena handles this.

 cpp
 
 
// src/Common/Arena.h:68
 ~MemoryChunk()
 {
 if (empty())
 return;

 /// We must unpoison the memory before returning to the allocator,
 /// because the allocator might not have asan integration, and the
 /// memory would stay poisoned forever. If the allocator supports
 /// asan, it will correctly poison the memory by itself.
 ASAN_UNPOISON_MEMORY_REGION(begin, size());

 Allocator<false>::free(begin, size());
 }

There it is — the arena’s chunk destructor unpoisons the whole chunk before handing it back. That is the guarantee the free-list arena relies on, and the comment in ArenaWithFreeLists::free is pointing exactly at it.

Who uses it

The interesting users are the ones that need to replace keys, not just accumulate them.

The cache dictionary

CacheDictionary keeps a fixed-size array of cells, each holding a key and its attributes. When an entry is updated with a different key, the old key’s string has to be released:

 cpp
 
 
// src/Dictionaries/CacheDictionaryStorage.h:447
 if (cell.key != key)
 {
 if constexpr (std::is_same_v<KeyType, std::string_view>)
 {
 char * data = const_cast<char *>(cell.key.data());
 arena.free(data, cell.key.size());
 cell.key = copyStringInArena(arena, key);
 }
 else
 cell.key = key;
 }

arena.free then copyStringInArena — the recycle-then-allocate pair. The string view is const, so the const_cast is needed to hand the pointer back to the arena. The cell.key.size() is the size the string was originally allocated with, which is what makes the size class lookup work.

Space-Saving, which powers `topK`

The topK aggregate function keeps the K most frequent elements and needs to free the strings of the ones it drops:

 cpp
 
 
// src/Common/SpaceSaving.h:420
 const UInt64 alpha_mask = alpha_map.size() - 1;
 for (size_t i = requested_capacity; i < counter_list.size(); ++i)
 {
 size_t pos = counter_list[i].hash & alpha_mask;
 alpha_map[pos] = std::min(alpha_map[pos] + counter_list[i].count - counter_list[i].error, MAX_ALPHA_VALUE);
 arena.free(counter_list[i].key);
 }

And SpaceSaving has a whole arena adapter for the string_view specialisation:

 cpp
 
 
// src/Common/SpaceSaving.h:50
/*
 * Specialized storage for std::string_view with a freelist arena.
 * Keys of this type that are retained on insertion must be serialized into local storage,
 * otherwise the reference would be invalid after the processed block is released.
 */
template <>
struct SpaceSavingArena<std::string_view>
{
 std::string_view emplace(std::string_view key)
 {
 if (!key.data())
 return key;

 return copyStringInArena(arena, key);
 }

 void free(std::string_view key)
 {
 if (key.data())
 arena.free(const_cast<char *>(key.data()), key.size());
 }

private:
 ArenaWithFreeLists arena;
};

Note the comment: “Keys of this type that are retained on insertion must be serialized into local storage, otherwise the reference would be invalid after the processed block is released.” The arena’s job is to make the string_view stable for as long as the aggregate state lives, and the free list makes the replacement cheap.

The generic helper that does the copying:

 cpp
 
 
// src/Common/ArenaUtils.h:10
template <typename Arena>
inline std::string_view copyStringInArena(Arena & arena, std::string_view value)
{
 size_t value_size = value.size();
 char * place_for_key = arena.alloc(value_size);
 memcpy(reinterpret_cast<void *>(place_for_key), reinterpret_cast<const void *>(value.data()), value_size);
 return std::string_view{place_for_key, value_size};
}

It is a template over the arena type, so it works with both Arena and ArenaWithFreeLists. The std::string_view{place_for_key, value_size} constructor takes the pointer and the length explicitly, so the string is not NUL-terminated — which is fine for a view, and saves a byte.

Seeing it from SQL

The topK aggregate function is the most visible user:

 sql
 
 
SELECT topK(3)(AirlineID) AS res
FROM VALUES('AirlineID UInt32', (19393), (19393), (19393), (19393), (19790), (19790), (19790), (19805), (19805), (20304));

 text
 
 
┌─res─────────────────┐
│ [19393,19790,19805] │
└─────────────────────┘

If the same query runs repeatedly with changing keys, the free list is what keeps the aggregate state’s memory flat instead of growing. And you can watch the state grow with:

 sql
 
 
SELECT
 query,
 formatReadableSize(memory_usage) AS state_size
FROM system.query_log
WHERE type = 'QueryFinish' AND memory_usage > 100000000
ORDER BY memory_usage DESC
LIMIT 10;

The point is not that topK uses an arena — the point is that the arena’s memory is recycled, so a long-running topK over a stream of changing keys does not accumulate garbage.

Trade-offs

What the free list buys you. Recycling a small object costs a class computation and a pointer write, instead of a free() call into the general allocator (which typically means a lock, a free-list walk, and a coalescing step). For the dictionary workload — millions of short keys, replaced individually — that difference is the whole point.

What it costs. Internal fragmentation: a 17-byte request occupies 32 bytes. The class list only has fourteen buckets, so the worst case is nearly 2x wasted. And the size must be tracked by the caller, since free needs it.

What the union buys you. The free list costs zero extra memory and zero extra allocation. There is no side table, no per-block header, no node to allocate. The price is that the block’s contents are destroyed on free — which is exactly what you want, since they were garbage anyway.

What the union costs you. You cannot read the first eight bytes of a free block (they hold the link), and you cannot distinguish a free block from a live one by looking at it. ASAN needs to be told which is which, which is why the poisoning is deliberate and the unpoisoning is explicit. A bug that frees the same pointer twice will silently corrupt the free list rather than crashing — the second free would write a new next pointer over the first one, and the list would then contain a cycle or a duplicate.

The part people get wrong. The free must be given the same size that was allocated. Passing a different size puts the block in the wrong class and the corruption is silent and delayed: the block is handed out later for a size it may not fit, and the overflow is a heap buffer overrun in somebody else’s code.

What to go do about it. If you are writing a ClickHouse-side component that holds many short-lived strings and has to replace them individually, ArenaWithFreeLists is the right tool. If you only ever append and never free, plain Arena is simpler and has no fragmentation. If the objects are large, neither is worth it — use the general allocator.

Wrapping up

The one thing to remember: the free-list link lives inside the freed block. That is why there is no per-block metadata, no side table, and no allocation on the free path. It is also why the block’s contents are unrecoverable after a free.

The part people get wrong: the size class is derived from the size you pass to free, not from anything stored in the block. A mismatch is silent corruption.

What to go do about it: look at where ArenaWithFreeLists is used in the tree — the cache dictionary and SpaceSaving — and notice that both use it in the same shape: allocate a string, store a view to it, and free the view when the entry is replaced. If your component has that shape, this class is already written for you.

Data-Structures on Opensource internals

ClickHouse's LRU Cache Keeps Its Order in a Linked List

The policy interface

The cell: the whole trick in nine words

The hit path: one splice

The insert path

The eviction loop

The thread-safe wrapper

Cache stampede: the insert token

Trade-offs

Wrapping up

ClickHouse Recycles Small Objects Through Per-Size Free Lists

The problem: an arena cannot free

The free-list arena

1. The union: the free-list link lives in the dead block

2. Size classes: fourteen buckets

3. The freelist array is a fixed-size array on the stack

4. Big allocations bypass the arena entirely

The allocation path

The free path

Who uses it

The cache dictionary

Space-Saving, which powers topK

Seeing it from SQL

Trade-offs

Wrapping up

Space-Saving, which powers `topK`