ClickHouse Batches Memory Accounting Per Thread

TL;DR: Charging a shared counter on every malloc is a cache-line ping-pong across every core. So ClickHouse keeps a per-thread balance of untracked bytes, and only when that balance exceeds 4 MiB (or the per-CPU budget runs out) does it flush the whole batch to the tracker hierarchy in a single charge. The cost: the tracked total can lag reality by up to 4 MiB per thread.

Every allocation in ClickHouse goes through a chain of four memory trackers, and each one does an atomic read-modify-write on a shared counter. Multiply that by four levels and by the number of allocations a query makes — a columnar engine allocates a lot, because every column is a separate buffer and every hash table growth is a reallocation — and you get the classic scalability wall: the counter is in a cache line, every core wants to write it, and the line ping-pongs between cores at the rate of the allocation rate.

This is the same problem that malloc has with its own arena metadata, and the same answer: don’t share, batch locally, flush rarely.

The thread-local pointer

First, the plumbing. ClickHouse does not look up “the current tracker” through a registry or a map; it reads a single pointer out of thread-local storage. But it is not a plain thread_local — it is a fiber-local:

cpp
// src/Common/CurrentThread.h:39
/**
 * We use **constinit** here to tell the compiler the current_thread variable is initialized.
 * If we didn't help the compiler, then it would most likely add a check before every use of the variable to initialize it if needed.
 * Instead it will trust that we are doing the right thing (and we do initialize it to nullptr) and emit more optimal code.
 * This is noticeable in functions like CurrentMemoryTracker::free and CurrentMemoryTracker::allocImpl
 * See also:
 * - https://en.cppreference.com/w/cpp/language/constinit
 * - https://github.com/ClickHouse/ClickHouse/pull/40078
 */
extern constinit FiberLocal<ThreadStatus *, FiberLocalSlot::CURRENT_THREAD> current_thread;

The constinit matters: without it the compiler would emit a guard variable check on every read of current_thread, and this is read on every allocation and every free. The comment says so explicitly.

FiberLocal is ClickHouse’s own thread-local storage abstraction, and it is not a wrapper around thread_local — on x86-64 it is a hand-written segment-relative load:

cpp
// src/Common/FiberLocal.h:172
    template <size_t slot>
    static uintptr_t load() noexcept
    {
        static_assert(slot < slot_count);
        uintptr_t value = 0;
#if defined(__x86_64__) && defined(__ELF__)
        __asm__ __volatile__(
            "movq %%fs:FiberLocalStorageThreadStorage@tpoff+%c1, %0"
            : "=r"(value)
            : "i"(slot * sizeof(void *))
            : "memory");

One movq from the FS segment, no function call, no TLS lookup helper. The slots are an array of words, and the whole array is swapped on a fiber switch:

cpp
// src/Common/FiberLocal.h:224
    alignas(DB::CH_CACHE_LINE_SIZE) std::array<uintptr_t, slot_count> slots = fiberLocalSlotDefaults();
    static_assert(
        sizeof(slots) <= 2 * DB::CH_CACHE_LINE_SIZE,
        "One slot is one word, and a context switch swaps the whole arena: keep it within two cache lines");

The tracker itself lives in the ThreadStatus object hanging off that pointer, together with the local balance:

cpp
// src/Common/ThreadStatus.h:209
    MemoryTracker memory_tracker{VariableContext::Thread};
    /// Small amount of untracked memory (per thread atomic-less counter)
    UntrackedMemoryCounter untracked_memory;
    /// MemoryTrackerBlockerInThread state corresponding to untracked_memory.
    VariableContext untracked_memory_blocker_level = VariableContext::Max;
    /// Each thread could new/delete memory in range of (-untracked_memory_limit, untracked_memory_limit) without access to common counters.
    Int64 untracked_memory_limit = 4 * 1024 * 1024;
    /// Per-CPU untracked memory
    PerCPUMemoryThreadState per_cpu_untracked_memory;

Four things to notice:

untracked_memory is an UntrackedMemoryCounter — “per thread atomic-less counter”.
The limit is 4 MiB, and it is signed: the range is (-limit, +limit), so the counter tracks both deferred allocations and deferred frees.
untracked_memory_blocker_level remembers which blocker level the deferred bytes belong to.
There is a per-CPU state too, which is the second-level batching.

The atomic-less counter

The counter is deliberately not an atomic read-modify-write, because only one thread ever writes it:

cpp
// src/Common/UntrackedMemoryRegistry.h:14
class ThreadLocalCounter
{
public:
    ThreadLocalCounter() = default;

    ThreadLocalCounter(const ThreadLocalCounter &) = delete;
    ThreadLocalCounter & operator=(ThreadLocalCounter &) = delete;
    ThreadLocalCounter(ThreadLocalCounter &&) = delete;
    ThreadLocalCounter & operator=(ThreadLocalCounter &&) = delete;

    ALWAYS_INLINE inline Int64 load() const noexcept { return value.load(std::memory_order_relaxed); }
    ALWAYS_INLINE inline void store(Int64 v) noexcept { value.store(v, std::memory_order_relaxed); }

    ALWAYS_INLINE inline Int64 add(Int64 delta) noexcept
    {
        /// Not a real atomic RMW for performance considerations.
        /// All writes (add, store) come from a single thread, so this is fine.
        const Int64 new_value = value.load(std::memory_order_relaxed) + delta;
        value.store(new_value, std::memory_order_relaxed);
        return new_value;
    }

private:
    std::atomic<Int64> value{0};
};

It is still an std::atomic<Int64> — so that a reader on another thread (e.g. the OOM message summing all threads) does not see a torn value — but the write path is a plain load, add and store. That is the entire point of the class: the write is thread-local, so it does not need to be atomic, but it is read cross-thread, so it must not be torn.

The counter is also registered in a global intrusive list, so the server can sum all of them:

cpp
// src/Common/UntrackedMemoryRegistry.h:49
class UntrackedMemoryRegistry
{
public:
    static UntrackedMemoryRegistry & instance();

    void add(UntrackedMemoryCounter * counter);
    void remove(UntrackedMemoryCounter * counter);

    Int64 sum() const;

private:
    UntrackedMemoryRegistry() = default;

    mutable std::mutex mutex;
    boost::intrusive::list<UntrackedMemoryCounter> counters;
};

That sum() is what produces the Untracked memory across all threads: line in the server-level out-of-memory message. It is the one number that tells you whether the server’s tracked total is honest.

The allocation path

Now the actual batching. CurrentMemoryTracker::allocImpl is where the decision is made. First, the tracker is found:

cpp
// src/Common/CurrentMemoryTracker.cpp:34
MemoryTracker * getMemoryTracker()
{
    if (auto * thread_memory_tracker = DB::CurrentThread::getMemoryTracker())
        return thread_memory_tracker;

    /// total_memory_tracker can be used before MainThreadStatus is initialized,
    /// but only after its own initialization and before teardown.
    if (DB::MainThreadStatus::initialized() || isTotalMemoryTrackerInitialized())
        return &total_memory_tracker;

    return nullptr;
}

Then the batch is updated, and only flushed if it has grown past the limit:

cpp
// src/Common/CurrentMemoryTracker.cpp:83
        DB::PerCPUMemoryThreadState previous_per_cpu = cur_thread->per_cpu_untracked_memory;
        Int64 new_untracked_memory = cur_thread->untracked_memory.add(size);
        Int64 previous_untracked_memory = new_untracked_memory - size;

        /// Flush when the per-thread cap is hit, or when the per-CPU budget cannot cover the deferral.
        if (new_untracked_memory > cur_thread->untracked_memory_limit
            || !DB::per_cpu_memory.sync(new_untracked_memory, cur_thread->per_cpu_untracked_memory))
        {
            DB::per_cpu_memory.release(cur_thread->per_cpu_untracked_memory);

            cur_thread->untracked_memory.store(0);

            try
            {
                /// We cannot return the AllocationTrace from here, since its sample_probability was calculated on the (batched) flushed size, which may not match the original allocation size.
                if (new_untracked_memory > 0)
                    std::ignore = memory_tracker->allocImpl(new_untracked_memory, enforce_memory_limit, /*query_tracker=*/ nullptr, /*_sample_probability=*/ 0.0);
                else
                    std::ignore = memory_tracker->free(-new_untracked_memory, /*_sample_probability=*/ 0.0);
            }
            catch (...)
            {
                cur_thread->untracked_memory.add(previous_untracked_memory);
                DB::per_cpu_memory.rollback(cur_thread->per_cpu_untracked_memory, previous_per_cpu);
                throw;
            }
        }

Read it carefully — it is the whole mechanism in one block.

The fast path is the first two lines: untracked_memory.add(size). That is a load, an add, a store, on a word that lives in this thread’s own cache lines. No atomic RMW, no cache-line transfer, no contention with other cores. That is the entire optimization.

The flush condition is new_untracked_memory > untracked_memory_limit — the per-thread cap — or the per-CPU budget saying “no more deferral”.

The flush itself is a single memory_tracker->allocImpl(new_untracked_memory, ...). One call, one charge, one walk up the four-level chain — for 4 MiB worth of allocations. That is a 4 MiB amortisation factor on the entire tracker machinery.

The catch restores the previous balance if the flush throws. This is the case where the server limit is hit: the flush throws MEMORY_LIMIT_EXCEEDED, and the thread has to put its 4 MiB back so that a later flush does not double-count it. The previous_untracked_memory was captured on the way in precisely so this restore is exact.

The _sample_probability of 0.0 in the flush call is a comment in disguise. The AllocationTrace returned by allocImpl carries a sampling probability computed for the batched size, but the caller wanted a trace for the original allocation. Returning the batch’s trace would attribute a stack trace to the wrong allocation. So the flush discards its trace, and the caller computes its own:

cpp
// src/Common/CurrentMemoryTracker.cpp:111
        return AllocationTrace(cur_thread->getEffectiveSampleProbability(size));

The blocker level

One subtlety: the deferred bytes belong to a particular blocker level. If the blocker’s level changes while bytes are pending, the pending bytes have to be flushed first, because they were accumulated under the old level’s rules:

cpp
// src/Common/CurrentMemoryTracker.cpp:71
        /// Make sure we do memory tracker calls with the correct level in MemoryTrackerBlockerInThread.
        /// E.g. suppose allocImpl is called twice: first for 2 MB with blocker set to
        /// VariableContext::User, then for 3 MB with no blocker. This should increase the
        /// Global memory tracker by 5 MB and the User memory tracker by 3 MB. So we can't group
        /// these two calls into one memory_tracker->allocImpl call.
        VariableContext blocker_level = MemoryTrackerBlockerInThread::getLevel();
        if (blocker_level != cur_thread->untracked_memory_blocker_level)
        {
            cur_thread->flushUntrackedMemory();
        }
        cur_thread->untracked_memory_blocker_level = blocker_level;

The example in the comment is the clearest possible statement of why batching is not always safe: 2 MB under a User-level blocker plus 3 MB without a blocker must not become one 5 MB charge, because the first 2 MB should not be charged to the user and the second 3 MB should be. So the level change forces a flush and the two charges stay separate.

The free path

The free side is symmetric, and has the same structure:

cpp
// src/Common/CurrentMemoryTracker.cpp:170
        Int64 new_untracked_memory = cur_thread->untracked_memory.add(-size);

        /// Flush when the per-thread cap is hit, or when the per-CPU budget cannot cover the deferral.
        if (new_untracked_memory < -cur_thread->untracked_memory_limit
            || !DB::per_cpu_memory.sync(new_untracked_memory, cur_thread->per_cpu_untracked_memory))
        {
            DB::per_cpu_memory.release(cur_thread->per_cpu_untracked_memory);

            cur_thread->untracked_memory.store(0);
            /// We cannot return the AllocationTrace from here, since its sample_probability was calculated on the (batched) flushed size, which may not match the original allocation size.
            if (new_untracked_memory > 0)
                std::ignore = memory_tracker->allocImpl(new_untracked_memory, /*enforce_memory_limit=*/ false, /*query_tracker=*/ nullptr, /*_sample_probability=*/ 0.0);
            else
                std::ignore = memory_tracker->free(-new_untracked_memory, /*_sample_probability=*/ 0.0);
        }

Two differences from the allocation path:

The condition is < -limit, not > +limit. A negative balance means deferred frees — memory that was allocated (and charged) elsewhere and is being freed here, so the balance can go negative.
The flush uses enforce_memory_limit = false for a positive balance. A deferred free must never be the thing that trips the memory limit: the bytes are already accounted for, and refusing the free would leave the tracker permanently over-charged. Freeing is always allowed.

The per-CPU budget

The per-thread cap alone is not enough. max_untracked_memory is 4 MiB per thread, and a server with 1 000 threads can therefore hide 4 GB of untracked memory. The per-CPU budget is the second line of defence:

cpp
// src/Common/PerCPUMemory.h:26
/// Per-CPU bound on the deferred (un-flushed) untracked memory the global MemoryTracker cannot yet
/// see, capping it in both directions: deferred allocations (-> OOM) and deferred frees (-> the
/// tracker over-reports usage).
///
/// Each CPU has two one-sided counters: `allocated` (>= 0) and `freed` (<= 0). A thread's
/// untracked_memory contributes to exactly one of them, by sign, batched so the shared counter is
/// touched only after drifting by `buffer`. When the occupied side leaves [-capacity, capacity]
/// the thread must flush, bounding the deferred bytes of every thread on the CPU, parked included.
///
/// Two separate counters, not one signed net, because a signed net would let a deferred free cancel
/// a deferred allocation: the allocation passes admission, then the free flushes on its own (its
/// thread exits or hits its per-thread cap) and the allocation is left untracked. Each thread can
/// leave up to max_untracked_memory untracked that way, i.e. nthreads * max_untracked_memory total —
/// the ~40 GiB overcommit this exists to remove.
///
/// Each counter is the sum of all threads' contributions to it (PerCPUMemoryThreadState::contributed):
/// every add is matched by a later subtract on flush, or a move to the other side/CPU on a sign
/// change or migration, so it equals the real deferred bytes on that side and the cap stays accurate.
///
/// The capacity check races with concurrent updates, so a counter may briefly exceed capacity —
/// harmless, since it only gates when bytes flush into the MemoryTracker, which enforces the real limit.
///
/// Bound per CPU: 0 <= allocated <= capacity, -capacity <= freed <= 0 (+ in-flight), so deferred
/// memory is ~ ncpu * max_per_cpu_untracked_memory + nthreads * per_cpu_untracked_memory_thread_buffer.
class PerCPUMemory

Read that comment twice. The “two separate counters, not one signed net” paragraph is the whole reason for the design, and the failure it prevents is subtle:

A thread allocates 3 MiB (deferred, not yet flushed) and then frees it (also deferred). With a single signed counter, the two cancel out and the counter returns to zero.
But the free was deferred against a block that was allocated by someone else — or, more simply, the free flushes on its own (the thread exits, or hits its per-thread cap in the negative direction) while the allocation is still sitting in the buffer.
The allocation is then left untracked forever: it passed admission, but nothing ever flushes it.

With one counter per side, the allocation sits on the allocated side and the free on the freed side, and each is accounted independently.

The constants:

cpp
// src/Common/PerCPUMemory.h:52
public:
    static constexpr Int64 DEFAULT_BUDGET = 8 * 1024 * 1024;
    static constexpr Int64 DEFAULT_THREAD_BUFFER = 32 * 1024;
    /// max_per_cpu_untracked_memory == 0 maps here: no side ever leaves the range, disabling the
    /// per-CPU bound (only the per-thread cap applies).
    static constexpr Int64 UNLIMITED_BUDGET = std::numeric_limits<Int64>::max() / 2;

8 MiB per CPU, 32 KiB of per-thread slack. The slack is the price of not touching the shared per-CPU counter on every single allocation:

cpp
// src/Common/PerCPUMemory.h:83
    [[nodiscard]] ALWAYS_INLINE bool sync(Int64 untracked_memory, PerCPUMemoryThreadState & state)
    {
        const Int64 buffer_now = buffer.load(std::memory_order_relaxed);
        if (likely(std::abs(untracked_memory - state.contributed) < buffer_now))
            return true;

        return publish(untracked_memory, state);
    }

state.contributed is what this thread last published to its CPU’s counter. If the thread’s own balance has not drifted more than buffer (32 KiB) from what it already published, nothing needs to happen — the shared counter already accounts for this thread. Only a drift of more than 32 KiB forces an update of the shared counter.

Publishing

cpp
// src/Common/PerCPUMemory.h:92
    [[nodiscard]] ALWAYS_INLINE bool publish(Int64 untracked_memory, PerCPUMemoryThreadState & state)
    {
        const int cpu = sched_getcpu();
        if (unlikely(static_cast<unsigned>(cpu) >= static_cast<unsigned>(cpu_count)))
            return false;

        Int64 occupied = 0;
        /// Fast path only for a genuine same-side delta. untracked_memory != 0 diverts a now-balanced
        /// thread to the slow branch: it holds nothing and must always be admitted, but the delta path
        /// would test the shared counter and could force a flush under other threads' pressure.
        const bool same_side = cpu == state.contributed_on_cpu
            && untracked_memory != 0
            && (untracked_memory > 0) == (state.contributed > 0);
        if (likely(same_side))
        {
            const Int64 delta = untracked_memory - state.contributed;
            std::atomic<Int64> & side = untracked_memory > 0 ? slots[cpu].allocated : slots[cpu].freed;
            occupied = side.fetch_add(delta, std::memory_order_relaxed) + delta;
            state.contributed = untracked_memory;
        }
        else
        {
            /// Migrated or changed sign: add to the new CPU/side before removing from the old, so
            /// the bytes are never charged to no CPU (a brief double-charge only tightens the bound).
            if (untracked_memory > 0)
                occupied = slots[cpu].allocated.fetch_add(untracked_memory, std::memory_order_relaxed) + untracked_memory;
            else if (untracked_memory < 0)
                occupied = slots[cpu].freed.fetch_add(untracked_memory, std::memory_order_relaxed) + untracked_memory;
            release(state);
            state.contributed = untracked_memory;
            state.contributed_on_cpu = cpu;
        }

        /// allocated >= 0 and freed <= 0, so one symmetric check covers both sides.
        const Int64 capacity_now = capacity.load(std::memory_order_relaxed);
        return -capacity_now <= occupied && occupied <= capacity_now;
    }

Three cases, all of them real:

Same CPU, same side: the fast path. One fetch_add of the delta.
Migrated to another CPU, or the sign flipped: the slow path. The bytes are added to the new side before being removed from the old, so they are never charged to no CPU. A brief double-charge is acceptable — it only tightens the bound, never loosens it.
The CPU id is out of range: return false, which means “flush”. This is the safe direction: the caller flushes to the real tracker.

Note sched_getcpu(). This is Linux-only; on other platforms the whole PerCPUMemory class is a stub whose sync always returns true, which disables the per-CPU bound and leaves only the per-thread cap.

Rolling back

If the flush throws, the per-CPU accounting has to be undone:

cpp
// src/Common/PerCPUMemory.h:143
    /// Re-add the pre-sync() snapshot after a failed flush; unconditional (unlike the gated
    /// sync) because the bytes are still live. May transiently exceed capacity; the next sync
    /// corrects it.
    ALWAYS_INLINE void rollback(PerCPUMemoryThreadState & state, const PerCPUMemoryThreadState & saved)
    {
        release(state);
        if (saved.contributed_on_cpu >= 0)
        {
            if (saved.contributed > 0)
                slots[saved.contributed_on_cpu].allocated.fetch_add(saved.contributed, std::memory_order_relaxed);
            else if (saved.contributed < 0)
                slots[saved.contributed_on_cpu].freed.fetch_add(saved.contributed, std::memory_order_relaxed);
        }
        state = saved;
    }

The asymmetry between sync (gated by the buffer) and rollback (unconditional) is deliberate and commented: the bytes are still live, so the counter must reflect them whether or not the buffer was crossed.

Flushing explicitly

Two situations force a flush regardless of the limits. One is a change of blocker level (shown above). The other is the thread finishing or changing its group:

cpp
// src/Common/ThreadStatus.cpp:263
void ThreadStatus::flushUntrackedMemory()
{
    /// The deferred bytes our contribution accounted for are about to be tracked, so remove it.
    per_cpu_memory.release(per_cpu_untracked_memory);

    Int64 current_untracked_memory = untracked_memory.load();
    if (current_untracked_memory == 0)
        return;

    MemoryTrackerBlockerInThread blocker(untracked_memory_blocker_level);
    untracked_memory.store(0);
    memory_tracker.adjustWithUntrackedMemory(current_untracked_memory);
}

The order is important: the per-CPU contribution is released first (because those bytes are about to be tracked for real), then the balance is zeroed, and only then is the tracker adjusted. adjustWithUntrackedMemory is a thin wrapper that picks the right direction:

cpp
// src/Common/MemoryTracker.cpp:669
void MemoryTracker::adjustWithUntrackedMemory(Int64 untracked_memory)
{
    if (untracked_memory > 0)
        std::ignore = allocImpl(untracked_memory, /*enforce_memory_limit*/ false);
    else
        std::ignore = free(-untracked_memory);
}

enforce_memory_limit = false again: the bytes were already allocated, and a flush must never be the thing that refuses them.

The limit is a session setting

The 4 MiB default is max_untracked_memory, and it is per-query:

cpp
// src/Core/Settings.cpp:4656
    DECLARE(UInt64, max_untracked_memory, (4 * 1024 * 1024), R"(
    Small allocations and deallocations are grouped in thread local variable and tracked or profiled only when an amount (in absolute value) becomes larger than the specified value. If the value is higher than 'memory_profiler_step' it will be effectively lowered to 'memory_profiler_step'.
)", 0) \

and it is lowered further if the memory profiler’s step is smaller, so that profiling granularity is preserved:

cpp
// src/Interpreters/ThreadStatusExt.cpp:394
    untracked_memory_limit = settings[Setting::max_untracked_memory];
    if (settings[Setting::memory_profiler_step] && settings[Setting::memory_profiler_step] < static_cast<UInt64>(untracked_memory_limit))
        untracked_memory_limit = settings[Setting::memory_profiler_step];

The per-CPU bounds are server-wide:

cpp
// src/Core/ServerSettings.cpp:432
    DECLARE(UInt64, max_per_cpu_untracked_memory, (8 * 1024 * 1024), R"(
    Upper bound, in bytes, on the untracked memory all threads running on one CPU may hold at once before it is flushed to the memory tracker. While `max_untracked_memory` bounds a single thread, this bounds the per-CPU total, so many threads cannot multiply their per-thread allowance into a large server-wide overcommit. The total untracked memory is therefore bounded by roughly `number_of_cpus * max_per_cpu_untracked_memory`. A value of `0` disables the per-CPU bound (only the per-thread `max_untracked_memory` applies). Linux only.
)", 0) \
    DECLARE(UInt64, per_cpu_untracked_memory_thread_buffer, (32 * 1024), R"(
    Amount of untracked memory, in bytes, each thread may hold without touching the shared per-CPU budget. It amortizes the cost of the per-CPU bookkeeping for small allocations and is the slack added on top of `max_per_cpu_untracked_memory * number_of_cpus` in the worst case. A value of `0` removes the slack: every allocation updates the shared per-CPU counter (most precise, but more contention). Linux only.
)", 0) \

The worst case is spelled out: number_of_cpus * 8 MiB + nthreads * 32 KiB. On a 64-core box with 1 000 threads, that is 512 MiB + 32 MiB ≈ 544 MiB of memory that the server’s tracked total does not know about at any given instant.

What it costs

The tracked total lags. Up to 4 MiB per thread and up to 8 MiB per CPU can be unaccounted for at any instant. system.processes.memory_usage and system.query_log.memory_usage are therefore lower bounds on what is actually held, not exact figures.

A freed block can be untracked before its allocation is. This is the freed_beyond_held case, and it is documented in the tracker header:

cpp
// src/Common/MemoryTracker.h:154
#ifdef DEBUG_OR_SANITIZER_BUILD
    std::atomic_bool drift_expected = false;
    /// A starting point, lowered as flagged sites get fixed.
    static constexpr Int64 drift_warn_threshold = 1024 * 1024;
    /// What a query root freed beyond what it held, which went straight to the server total (see `free`). For example,
    /// thread A of a query allocates a 2 MiB block that stays in its untracked memory, below `max_untracked_memory`, and
    /// thread B frees the block and flushes: the query holds less than 2 MiB, so the rest goes here. When A flushes later,
    /// the query is charged for a block that is already freed, which shows as drift at the end of the query but is not.
    std::atomic<Int64> freed_beyond_held = 0;
#endif

The drift is bounded and it is settled when the query ends:

cpp
// src/Common/MemoryTracker.cpp:719
Int64 MemoryTracker::unexpectedDrift() const
{
#ifdef DEBUG_OR_SANITIZER_BUILD
    auto * loaded_parent = parent.load(std::memory_order_relaxed);
    if (!loaded_parent || loaded_parent->level != VariableContext::User || drift_expected.load(std::memory_order_relaxed))
        return 0;
    Int64 drift = amount.load(std::memory_order_relaxed);
    /// Threads batch what they allocate and free (`max_untracked_memory`), so a free can arrive before the allocation
    /// it matches. It goes to the server total, and the allocation, arriving later, stays here without being drift.
    if (drift > 0)
        drift = std::max<Int64>(drift - freed_beyond_held.load(std::memory_order_relaxed), 0);
    return std::abs(drift) > drift_warn_threshold ? drift : 0;
#else
    return 0;
#endif
}

This is debug-build-only, and the threshold is 1 MiB — a smaller drift is noise.

The flush can throw. If the server limit is reached exactly when a thread flushes its 4 MiB, the flush throws MEMORY_LIMIT_EXCEEDED from inside an allocation that was going to succeed. This is the correct behaviour — the server really is out of memory — but it means the exception can surface at a confusing place.

sched_getcpu() per publish. The per-CPU mechanism needs the CPU id, which is a vDSO call on Linux but not free. The 32 KiB buffer exists precisely to make that call rare.

Measuring it

ClickHouse ships a benchmark for exactly this path:

cpp
// src/Common/examples/memory_tracker_benchmark.cpp:1
/// Benchmark of CurrentMemoryTracker::{alloc,free} under a 4-level nested
/// MemoryTracker hierarchy (Global -> User -> Process -> Thread).
///
/// Usage:
///   clickhouse-examples memory_tracker_benchmark
///       [--threads N] [--ops N] [--size N]
///       [--mode pair|alloc-only|free-only|new-delete]

The pair mode measures the tracker alone; the new-delete mode adds jemalloc on top, and the difference between them is the allocator’s share:

cpp
// src/Common/examples/memory_tracker_benchmark.cpp:180
        case Mode::NewDelete:
            for (Int64 i = 0; i < ops; ++i)
            {
                /// Goes through operator new -> Memory::trackMemory ->
                /// CurrentMemoryTracker::allocNoThrow -> jemalloc;
                /// delete[] mirrors with Memory::untrackMemory + jefree.
                /// Use volatile asm to prevent the compiler from eliding the
                /// pair, but skip touching the memory so we measure the
                /// allocator + tracker cost, not page-fault cost.
                char * ptr = new char[static_cast<size_t>(size)];
                clobber(ptr);
                delete[] ptr;
                ++done;
            }

Note that the benchmark constructs a ThreadStatus on the worker’s stack, so that current_thread is non-null and the per-thread and per-CPU paths are exercised:

cpp
// src/Common/examples/memory_tracker_benchmark.cpp:141
    /// Construct ThreadStatus on the worker stack so `current_thread` is
    /// non-null and `CurrentMemoryTracker` exercises the per-thread +
    /// per-CPU code paths. Re-parent the thread tracker to the process
    /// (query) level so the chain is Thread -> Process -> User -> Global.
    ThreadStatus thread_status;
    CurrentThread::get().memory_tracker.setParent(process_tracker);

What to go do about it

The one thing to remember: the memory numbers you read are a lower bound, and the gap is bounded and known. If you need exact accounting — for a profiler, or for a strict budget — set max_untracked_memory to 0:

sql
SELECT max_untracked_memory FROM system.settings WHERE name = 'max_untracked_memory';

sql
SELECT count()
FROM events
GROUP BY user_id
SETTINGS max_untracked_memory = 0;

That turns every allocation into an individual charge up the four-level chain, which is exactly what you want for accuracy and exactly what you do not want for throughput.

The part people get wrong: setting max_untracked_memory = 0 to “get accurate numbers” is a performance cliff, not a tuning knob. The tracker chain is on the hot path of every allocation, and the batching is what keeps it from dominating query time. Measure before you do it.

What to go do about it: if you are chasing a discrepancy between system.processes.memory_usage and the RSS of the process, the answer is usually in the server-level out-of-memory message’s Untracked memory across all threads figure, not in the per-query number.