ClickHouse's Thread Pool Wakes the Newest Idle Thread and Counts Events Per CPU

TL;DR: The pool is one boost::heap::priority_queue behind one mutex, but the wake-up is not notify_all on a shared condition variable — it is a per-thread CV plus an intrusive LIFO stack, so exactly one thread is woken and it is the one that ran most recently (better cache locality, no thundering herd). Every one of those transitions is counted, and the counters themselves are sharded per CPU on cache-line-padded rows.

Every server needs a thread pool, and the textbook one is a queue plus a condition variable:

text
   producers                        consumers
       |                                ^
       v                                |
  +---------+   push    +---------+  pop |
  |  queue  | --------> |  mutex  | -----+
  +---------+           +---------+
                              |  ^
                              v  | notify_all
                     +---------------+
                     | N-1 sleeping |
                     |   threads    |
                     +---------------+

It is simple, and it works. It also has two well-known pathologies, both of which a database hits hard:

The thundering herd. notify_all wakes every sleeping thread. On a pool with 10 000 threads and a queue that receives one small job, 9999 threads wake, contend for the mutex, find the queue empty, and go back to sleep. That is 9999 wakeups and 9999 context switches to do one unit of work.
The cache-line ping-pong. Whatever the workers were doing, they were probably doing it near each other in memory. Waking all of them scatters the working set across all of them.

ClickHouse’s pool is not a work-stealing pool (there is no per-thread deque, no stealing). It is something else: a shared, stable priority queue with a LIFO wake-up order and per-thread condition variables, plus a set of per-CPU counters that record every interesting transition.

This article walks the queue, the idle stack, the shutdown semantics, and then the counter machinery — because “how do you count every thread-pool event without paying for it” turns out to be the more interesting half.

The queue: a stable priority queue, not a deque

The pool’s job container is a single heap:

cpp
// src/Common/ThreadPool.h:238
    boost::heap::priority_queue<JobWithPriority,boost::heap::stable<true>> jobs;

boost::heap::stable<true> means the heap preserves insertion order among equal-priority elements. That matters: without it, two jobs with the same priority could be reordered arbitrarily, and ClickHouse relies on FIFO ordering inside a priority class for things like replication ordering.

The element type carries a fair amount of state:

cpp
// src/Common/ThreadPool.cpp:230
class JobWithPriority
{
public:
    using Job = std::function<void()>;

    Job job;
    Priority priority;
    CurrentMetrics::Increment metric_increment;
    ScopedDecrement available_threads_decrement;

    DB::OpenTelemetry::TracingContextOnThread thread_trace_context;

    /// Propagated like the tracing context: a job scheduled under `QueryCancellationBlocker` runs under one.
    bool query_cancellation_blocked = false;

    /// Call stacks of all jobs' schedulings leading to this one
    std::vector<FramePointers> frame_pointers;
    bool enable_job_stack_trace = false;
    Stopwatch job_create_time;

    // Deleted copy constructor and copy assignment operator
    JobWithPriority(const JobWithPriority&) = delete;
    JobWithPriority& operator=(const JobWithPriority&) = delete;

    // Move constructor and move assignment operator
    JobWithPriority(JobWithPriority&&) noexcept = default;
    JobWithPriority& operator=(JobWithPriority&&) noexcept = default;

    JobWithPriority(
        Job job_, Priority priority_, CurrentMetrics::Metric metric,
        const DB::OpenTelemetry::TracingContextOnThread & thread_trace_context_, bool query_cancellation_blocked_,
        bool capture_frame_pointers, ScopedDecrement available_threads_decrement_)
        : job(job_), priority(priority_), metric_increment(metric),
        available_threads_decrement(std::move(available_threads_decrement_)),
        thread_trace_context(thread_trace_context_), query_cancellation_blocked(query_cancellation_blocked_),
        enable_job_stack_trace(capture_frame_pointers)
    {
        if (!capture_frame_pointers)
            return;
        /// Save all previous jobs call stacks and append with current
        frame_pointers = DB::Exception::getThreadFramePointers();
        frame_pointers.push_back(StackTrace().getFramePointers());
    }

    bool operator<(const JobWithPriority & rhs) const
    {
        return priority > rhs.priority; // Reversed for `priority_queue` max-heap to yield minimum value (i.e. highest priority) first
    }

    UInt64 elapsedMicroseconds() const
    {
        return job_create_time.elapsedMicroseconds();
    }
};

Three things to note.

The priority inversion is deliberate. operator< is reversed, because boost::heap::priority_queue is a max-heap but ClickHouse wants the smallest priority value to run first:

cpp
// src/Common/Priority.h:4
/// Common type for priority values.
/// Separate type (rather than `Int64` is used just to avoid implicit conversion errors and to default-initialize
struct Priority
{
    using Value = Int64;
    Value value = 0; /// Note that lower value means higher priority.
    constexpr operator Value() const { return value; } /// NOLINT
};

The Stopwatch in the element is a queue-latency timer. The worker computes how long the job waited in the queue and reports it as a profile event — that is how you distinguish “the pool is saturated” from “the job is slow”.

The tracing context and the cancellation blocker travel with the job. thread_trace_context is the OpenTelemetry context of the scheduler, so the span parent is correct even though the job runs on a different thread. query_cancellation_blocked does the same for query cancellation: a job scheduled under a QueryCancellationBlocker inherits it.

The idle stack: the interesting part

The pool does not use one shared condition variable. Each worker thread owns one:

cpp
// src/Common/ThreadPool.h:91
        /// Per-thread condition variable for LIFO idle scheduling.
        /// Each idle thread waits on its own CV, so the scheduler can wake the
        /// LIFO-selected thread directly without disturbing other idle threads.
        /// This avoids the thundering herd that a shared CV with `notify_all` creates.
        std::condition_variable cv;

        /// LIFO idle thread scheduling flag.
        /// When a new job is scheduled, the most recently idle thread is popped
        /// from the LIFO stack, this flag is set to true, and only that thread's
        /// CV is notified. The worker wait predicate also re-checks the real
        /// pool state (queued jobs, shutdown, excess threads) as a safety net
        /// against missed wake-ups, so a worker that wakes either directly or
        /// through a spurious CV signal will still pick up pending work.
        bool idle_wakeup_flag = false;

        /// Intrusive links in the parent pool idle stack. They avoid allocations
        /// in the worker idle path, which runs under `DENY_ALLOCATIONS_IN_SCOPE`.
        ThreadFromThreadPool * idle_prev = nullptr;
        ThreadFromThreadPool * idle_next = nullptr;
        bool in_idle_stack = false;

And the pool keeps an intrusive doubly-linked stack of the idle threads:

cpp
// src/Common/ThreadPool.h:243
    /// Intrusive LIFO stack of idle threads for wake-up scheduling.
    /// When a new job is scheduled, the most recently idle thread is woken first.
    /// This concentrates work on fewer OS threads, improving CPU cache locality
    /// and reducing memory fragmentation from allocator per-thread caches (e.g. jemalloc tcache).
    /// See https://github.com/ClickHouse/ClickHouse/issues/10818
    ThreadFromThreadPool * idle_thread_head = nullptr; /// Oldest idle thread.
    ThreadFromThreadPool * idle_thread_tail = nullptr; /// Most recently idle thread.
    size_t idle_thread_count = 0;

Note “intrusive”: the links live inside the thread object, so pushing and popping the stack allocates nothing. That matters because the idle path runs with allocations forbidden:

cpp
// src/Common/ThreadPool.cpp:986
    DENY_ALLOCATIONS_IN_SCOPE;

The stack operations are six short functions:

cpp
// src/Common/ThreadPool.cpp:824
typename ThreadPoolImpl<Thread>::ThreadFromThreadPool * ThreadPoolImpl<Thread>::popNewestIdleThreadNoLock()
{
    ThreadFromThreadPool * thread = idle_thread_tail;
    if (thread)
        removeIdleThreadNoLock(thread);
    return thread;
}

template <typename Thread>
typename ThreadPoolImpl<Thread>::ThreadFromThreadPool * ThreadPoolImpl<Thread>::popOldestIdleThreadNoLock()
{
    ThreadFromThreadPool * thread = idle_thread_head;
    if (thread)
        removeIdleThreadNoLock(thread);
    return thread;
}

template <typename Thread>
void ThreadPoolImpl<Thread>::wakeIdleThreadNoLock(ThreadFromThreadPool * thread)
{
    thread->idle_wakeup_flag = true;
    thread->cv.notify_one();
}

template <typename Thread>
void ThreadPoolImpl<Thread>::wakeUpAllIdleThreadsNoLock()
{
    while (ThreadFromThreadPool * thread = popOldestIdleThreadNoLock())
        wakeIdleThreadNoLock(thread);
}

So there are two ends: the tail is the most recently idle thread (popped for new work — LIFO), and the head is the oldest idle thread (popped first when the pool needs to shrink, or on shutdown).

The worker loop

The worker waits on its own CV, but with a predicate that re-checks the pool state as a safety net:

cpp
// src/Common/ThreadPool.cpp:1067
            while (parent_pool.jobs.empty()
                && !parent_pool.finished
                && parent_pool.threads.size() <= std::min(parent_pool.max_threads, parent_pool.scheduled_jobs + parent_pool.max_free_threads))
            {
                idle_wakeup_flag = false;
                parent_pool.pushIdleThreadNoLock(this);
                cv.wait(lock, [this]
                {
                    return idle_wakeup_flag
                        || !parent_pool.jobs.empty()
                        || parent_pool.finished
                        || parent_pool.threads.size() > std::min(parent_pool.max_threads, parent_pool.scheduled_jobs + parent_pool.max_free_threads);
                });

                parent_pool.removeIdleThreadNoLock(this);
            }

That comment above it is a war story worth quoting, because it explains why the predicate is not simply idle_wakeup_flag:

cpp
// src/Common/ThreadPool.cpp:1045
            /// LIFO idle thread scheduling: link this thread into the intrusive
            /// idle stack and wait on its own per-thread CV until selected. The
            /// scheduler pops the most recently idle thread from the stack, sets
            /// its `idle_wakeup_flag`, and notifies only that thread's CV.
            ///
            /// The wait predicate also re-checks the real pool state. This is a
            /// required safety net: a previous attempt to drop it
            /// (commit `9322e66151ea`) deadlocked the
            /// `SchedulerWorkloadResourceManager.DropNotEmptyQueueLong` unit test
            /// under TSan. Even though every notifier
            /// (`popNewestIdleThreadNoLock`, `wakeUpAllIdleThreadsNoLock`,
            /// `wakeUpExcessIdleThreadsNoLock`) sets `idle_wakeup_flag` before
            /// notifying the per-thread CV, there are interleavings where a
            /// queued job becomes visible to this worker without a paired
            /// notification. Re-checking the pool state in the predicate lets a
            /// notified or spuriously-woken worker pick up such jobs (or observe
            /// shutdown / a reduced thread limit) instead of sleeping forever.

The three conditions are: I was explicitly selected (the fast path), or there is work (the safety net), or the pool is shutting down / shrinking.

Then the job is popped:

cpp
// src/Common/ThreadPool.cpp:1096
            /// boost::priority_queue does not provide interface for getting non-const reference to an element
            /// to prevent us from modifying its priority. We have to use const_cast to force move semantics on JobWithPriority.
            job_data = std::move(const_cast<JobWithPriority &>(parent_pool.jobs.top()));
            parent_pool.jobs.pop();

            ProfileEvents::increment(
                std::is_same_v<Thread, GlobalThreadType> ? ProfileEvents::GlobalThreadPoolJobWaitTimeMicroseconds : ProfileEvents::LocalThreadPoolJobWaitTimeMicroseconds,
                job_data->elapsedMicroseconds());

And on the scheduling side, exactly one thread is woken:

cpp
// src/Common/ThreadPool.cpp:575
        /// Select the most recently idle thread (LIFO order) and wake only that one
        /// via its per-thread CV. Concentrating work on fewer threads improves cache
        /// locality and lets excess threads exit naturally, freeing their allocator
        /// caches. Notifying a single thread (instead of `notify_all` on a shared CV)
        /// avoids a thundering herd where every idle thread wakes, contends for the
        /// mutex, and goes back to sleep on every job schedule.
        ///
        /// Notify while still holding `mutex`: once `idle_wakeup_flag` is set, a
        /// spurious wakeup is enough for the worker to run and remove itself from
        /// the pool, so the raw `ThreadFromThreadPool` pointer must not be used
        /// after the lock is released.
        if (ThreadFromThreadPool * thread_to_wake = popNewestIdleThreadNoLock())
            wakeIdleThreadNoLock(thread_to_wake);

The last paragraph is a subtle lifetime point: the woken thread may remove itself from the pool (see below), so the pointer must not be touched after the lock is dropped.

Why LIFO?

The rationale is in the header comment: concentrating work on fewer OS threads improves CPU cache locality. Concretely:

The thread that just finished a job has its caches warm — the data it was working on is still in L1/L2. Handing it the next job of the same query is a hit.
The allocator’s per-thread cache (jemalloc tcache) is warm on that thread, so allocations do not have to go to the arena.
Other threads stay idle, and idle threads exit (see wakeUpExcessIdleThreadsNoLock), releasing their stacks and their tcache arenas. A pool of 10 000 threads that only ever uses 32 of them should converge to 32 resident threads, not 10 000.

The downside is real: LIFO is unfair. A thread at the tail of the idle stack may starve while the head thread keeps getting work. In practice this does not matter because the pool has no per-thread affinity requirement — jobs are anonymous.

Shutdown semantics

finish() is not stop accepting work and drain. The queued jobs are discarded:

cpp
// src/Common/ThreadPool.h:171
    /// Stop accepting new jobs: further scheduling produces CANNOT_SCHEDULE_TASK, and the jobs that are
    /// already queued but have not started yet are discarded instead of being run.
    /// Jobs that are already running are neither interrupted nor waited for, call 'wait' to join them.
    void finish();

The implementation is a single negative sentinel:

cpp
// src/Common/ThreadPool.cpp:716
template <typename Thread>
void ThreadPoolImpl<Thread>::finishNoLock(const std::lock_guard<std::mutex> &)
{
    finished = true;

    /// scheduleImpl doesn't check for `finished` outside the critical section,
    /// so we set remaining_pool_capacity to a large negative value
    /// (e.g., -MAX_THEORETICAL_THREAD_COUNT) to signal that no new threads are needed.
    /// This effectively prevents any new threads from being started during shutdown.
    remaining_pool_capacity.store(-MAX_THEORETICAL_THREAD_COUNT, std::memory_order_relaxed);

    /// Wake up all idle threads so they can see it and exit gracefully.
    wakeUpAllIdleThreadsNoLock();
    job_finished.notify_all();
}

The remaining_pool_capacity trick is worth spelling out, because it shows how the pool avoids taking the lock on the fast path. The pool keeps two atomics:

cpp
// src/Common/ThreadPool.h:220
    // Originally equals to max_threads, but changes dynamically.
    // Decrements with every new thread started, increments when it finishes.
    // If positive, then more threads can be started.
    // When it comes to zero, it means that max_threads threads have already been started.
    // it can be below zero when the threadpool is shutting down
    std::atomic<int64_t> remaining_pool_capacity;

    // Increments every time a new thread joins the thread pool or a job finishes.
    // Decrements every time a task is scheduled.
    // If positive, it means that there are more threads than jobs (and some are idle).
    // If zero, it means that every thread has a job.
    // If negative, it means that we have more jobs than threads.
    std::atomic<int64_t> available_threads;

scheduleImpl tries to create a new thread before taking the lock, using a CAS on remaining_pool_capacity as the token:

cpp
// src/Common/ThreadPool.cpp:434
    std::unique_ptr<ThreadFromThreadPool> new_thread;
    // Load the current capacity
    int64_t capacity = remaining_pool_capacity.load(std::memory_order_relaxed);
    int64_t currently_available_threads = available_threads.load(std::memory_order_relaxed);

    while (currently_available_threads <= 0 && capacity > 0)
    {
        if (remaining_pool_capacity.compare_exchange_weak(capacity, capacity - 1, std::memory_order_relaxed))
        {
            try
            {
                new_thread = std::make_unique<ThreadFromThreadPool>(*this);
                break; // Exit the loop once a thread is successfully created.
            }
            catch (...)
            {
                // Failed to create the thread, restore capacity
                remaining_pool_capacity.fetch_add(1, std::memory_order_relaxed);
                std::lock_guard lock(mutex); // needed to change first_exception.
                return on_error(fmt::format("failed to start the thread: {}", DB::getCurrentExceptionMessage(true)));
            }
        }
        // capacity gets reloaded by (unsuccessful) compare_exchange_weak
        currently_available_threads = available_threads.load(std::memory_order_relaxed);
    }

Because finishNoLock sets the capacity to a huge negative number, the capacity > 0 test fails immediately and no thread is created — no lock, no race, no extra check.

Counting every transition

The pool is instrumented at six points, and the counters are the only way to know what a pool is actually doing. GlobalThreadPoolLockWaitMicroseconds is incremented at three places — the scheduler’s lock acquisition, the worker’s lock acquisition, and wait():

cpp
// src/Common/ThreadPool.cpp:460
    {
        Stopwatch watch;
        std::unique_lock lock(mutex);
        ProfileEvents::increment(
            std::is_same_v<Thread, GlobalThreadType> ? ProfileEvents::GlobalThreadPoolLockWaitMicroseconds : ProfileEvents::LocalThreadPoolLockWaitMicroseconds,
            watch.elapsedMicroseconds());

GlobalThreadPoolJobs counts scheduled jobs, GlobalThreadPoolJobWaitTimeMicroseconds the queue latency, GlobalThreadPoolExpansions/Shrinks the resizes, GlobalThreadPoolThreadCreationMicroseconds the time spent waiting for a freshly created worker to actually start running:

cpp
// src/Common/ThreadPool.cpp:896
    Stopwatch watch2;

    thread = Thread(&ThreadFromThreadPool::worker, this);

    ProfileEvents::increment(
        std::is_same_v<Thread, GlobalThreadType> ? ProfileEvents::GlobalThreadPoolThreadCreationMicroseconds : ProfileEvents::LocalThreadPoolThreadCreationMicroseconds,
        watch2.elapsedMicroseconds());
    ProfileEvents::increment(
        std::is_same_v<Thread, GlobalThreadType> ? ProfileEvents::GlobalThreadPoolExpansions : ProfileEvents::LocalThreadPoolExpansions);

Every one of these is a fetch_add on a shared counter, called from arbitrarily many threads. Which brings us to the question the title promises: how do you do that without the cache line bouncing between cores?

You can see all of it from SQL:

sql
SELECT event, value
FROM system.events
WHERE event ILIKE '%ThreadPool%'
ORDER BY event;

-- event                                     -- value
-- GlobalThreadPoolJobs                      -- 1283741
-- GlobalThreadPoolLockWaitMicroseconds      -- 91382
-- GlobalThreadPoolJobWaitTimeMicroseconds   -- 4471290
-- GlobalThreadPoolExpansions                -- 12
-- GlobalThreadPoolShrinks                   -- 3
-- GlobalThreadPoolThreadCreationMicroseconds-- 84022

GlobalThreadPoolJobWaitTimeMicroseconds / GlobalThreadPoolJobs is the average time a job spent in the queue. If that number climbs while GlobalThreadPoolLockWaitMicroseconds stays flat, the pool is saturated, not contended — a distinction you cannot make without these counters.

The counters: sharded per CPU

ClickHouse’s ProfileEvents used to be a single flat array of std::atomic, one counter per event. Every increment from every thread hit the same cache line for a given event. With GlobalThreadPoolJobs incremented once per scheduled job across 10 000 threads, that is a guaranteed ping-pong.

The fix is per-CPU sharding. The layout is computed at compile time:

cpp
// src/Common/ProfileEvents.cpp:1918
/// Row stride padded so each per-CPU row ends on a cache-line boundary. Without this the last
/// few events of one row share a cache line with the first events of the next row, causing
/// false sharing across CPUs at row boundaries.
constexpr size_t counts_per_cache_line = DB::CH_CACHE_LINE_SIZE / sizeof(Count);
static_assert((counts_per_cache_line & (counts_per_cache_line - 1)) == 0);
constexpr size_t per_cpu_stride = (static_cast<size_t>(END) + counts_per_cache_line - 1) & ~(counts_per_cache_line - 1);

/// Cell count for a layout: `cpus` padded rows, or a compact single row of raw events.
ALWAYS_INLINE inline size_t cellCount(uint32_t cpus)
{
    return cpus ? static_cast<size_t>(cpus) * per_cpu_stride : static_cast<size_t>(END);
}

/// Cells are plain `Count` accessed via `std::atomic_ref`; over a suitably aligned object it is
/// lock-free and generates the same code as a `std::atomic` member (required_alignment == 8).
ALWAYS_INLINE inline std::atomic_ref<Count> cell(Count * counters, size_t cpu, Event event)
{
    return std::atomic_ref<Count>(counters[cpu * per_cpu_stride + event]);
}

So the array is cpus × per_cpu_stride cells, where per_cpu_stride is the event count rounded up to a whole number of cache lines. Cell (cpu, event) lives at cpu * per_cpu_stride + event. Two different CPUs writing two different events are guaranteed to be on different cache lines.

The increment is then a relaxed atomic add into the current CPU’s row:

cpp
// src/Common/ProfileEvents.cpp:2080
void Counters::fetchAdd(Event event, Count amount, int32_t cpu)
{
    const uint32_t rows = cpus.load(std::memory_order_relaxed);
    if (rows)
    {
        /// `cpu` may be >= rows if a CPU above `MAX_CPUS` is online, and -1 on error. In both
        /// cases, fall back to row 0 — still atomic, still correct, just with less cache locality.
        const size_t row = (cpu >= 0 && static_cast<uint32_t>(cpu) < rows) ? static_cast<size_t>(cpu) : 0;
        cell(counters, row, event).fetch_add(amount, std::memory_order_relaxed);
    }
    else
        cell(counters, 0, event).fetch_add(amount, std::memory_order_relaxed);
}

relaxed ordering, because a counter’s value is not synchronisation for anything. And the CPU id comes from a TLS read, not a syscall:

cpp
// src/Common/PerCPU.h:39
ALWAYS_INLINE inline Int32 getCurrentCPU()
{
#if defined(OS_LINUX)
    /// TLS read via glibc rseq on modern kernels (see glibc-compatibility/musl/sched_getcpu.c).
    return sched_getcpu();

Reading the counters sums the rows:

cpp
// src/Common/ProfileEvents.cpp:2045
Count Counters::load(Event event) const
{
    const uint32_t rows = cpus.load(std::memory_order_relaxed);
    if (!rows)
        return cell(counters, 0, event).load(std::memory_order_relaxed);
    Count sum = 0;
    for (uint32_t s = 0; s < rows; ++s)
        sum += cell(counters, s, event).load(std::memory_order_relaxed);
    return sum;
}

The `atomic_ref` trick

Look closely at the cell type. It is not std::atomic<Count>. It is a plain Count accessed through std::atomic_ref. The comment explains why:

cpp
// src/Common/ProfileEvents.cpp:1943
/// Per-CPU storage for `global_counters`, cache-line aligned. `Count` is trivially default-
/// constructible, so this static array is a guaranteed zero-init BSS with no dynamic initializer:
/// valid before any dynamic initializer can touch `global_counters` (which points here), and only
/// touched rows fault in. An `atomic` element is not trivially constructible and would reintroduce
/// dynamic initialization for an array this large — hence plain `Count` cells + `atomic_ref`.
static_assert(std::is_trivially_default_constructible_v<Count>);
alignas(DB::CH_CACHE_LINE_SIZE) static Count global_counters_storage[PerCPU::MAX_CPUS * per_cpu_stride];

The array is 1024 CPUs × per_cpu_stride UInt64s — with a few thousand events that is tens of megabytes. If the elements were std::atomic<UInt64>, the array would need a dynamic initializer (atomic’s constructor is not trivially default-constructible in the required sense), which would mean a static-initialization-order problem: some other translation unit’s dynamic initializer could increment a counter before this one ran. By keeping the storage plain and wrapping accesses in std::atomic_ref, the array stays in BSS, is guaranteed zero, and is valid before any dynamic initialization.

The layout flips from single-row to per-CPU at static-init time:

cpp
// src/Common/ProfileEvents.cpp:1966
/// Switches `global_counters` to per-CPU layout during dynamic static init. Only `cpus` is
/// mutated — storage stays put, so both pre- and post-flip readers see a valid view of the same
/// BSS array (pre-flip increments land in row 0, which post-flip sums include). The store is
/// relaxed-atomic because a thread spawned by an earlier TU's dynamic initializer may already be
/// incrementing counters while this runs.
struct ProfileEventsPerCPUInitializer
{
    ProfileEventsPerCPUInitializer()
    {
        const uint32_t cpus = PerCPU::getNumCPUs();
        global_counters.cpus.store(cpus, std::memory_order_relaxed);
        user_counters_cpus.store(cpus, std::memory_order_relaxed);
    }
};
static const ProfileEventsPerCPUInitializer profile_events_per_cpu_initializer;

“pre-flip increments land in row 0, which post-flip sums include” — that is the elegant part. The layout change is not a reset; the single row simply becomes “row 0” of a multi-row array, and the sum includes it.

The trace-log fast path

There is one more thing increment() does, and it is a nice example of paying for the slow thing only when the slow thing is wanted:

cpp
// src/Common/ProfileEvents.cpp:2319
void Counters::increment(Event event, Count amount)
{
    Counters * current = this;
    bool send_to_trace_log = false;
    const int32_t cpu = PerCPU::getCurrentCPU();

    do
    {
        current->fetchAdd(event, amount, cpu);
        /// Small optimization for quite a hot path.
        /// Load with relaxed as it almost always returns null.
        /// If non-null, add an acquire fence.
        if (current->should_trace_array.load(std::memory_order_relaxed))
        {
            if (auto * trace_arr = current->should_trace_array.load(std::memory_order_acquire))
                send_to_trace_log |= trace_arr[event].load(std::memory_order_relaxed);
        }
        send_to_trace_log |= current->trace_all_profile_events.load(std::memory_order_relaxed);

        current = current->parent.load(std::memory_order_acquire);
    } while (current != nullptr);

    if (unlikely(send_to_trace_log))
        DB::TraceSender::send(DB::TraceType::ProfileEvent, StackTrace(), {.event = event, .increment = amount});
}

The common case is one relaxed load of a null pointer. The expensive path (StackTrace() — a full symbolised backtrace!) is behind an unlikely branch that is only taken when the user has explicitly asked for a specific event to be traced.

Also note the parent chain. A Counters object has a parent pointer, and an increment walks it. There are three levels: Thread, User, Global. That is how a query’s own ProfileEvents get attributed to the user and then to the server, with one fetch_add per level.

What “zero overhead” really means

The claim is not that counting is free — it is that it costs one uncontended fetch_add on a cache line nobody else is touching. Compare:

text
  shared counter (one cache line)        per-CPU counters (one line each)
  ---------------------------------      ---------------------------------
    core 0: LOCK XADD [L]  <---+          core 0: LOCK XADD [L0]
    core 1: LOCK XADD [L]  <---+--+       core 1: LOCK XADD [L1]
    core 2: LOCK XADD [L]  <---+  |       core 2: LOCK XADD [L2]
         |          |         |  |            |        |        |
         +----------+---------+  |            +--------+--------+
              ping-pong: the line                             no sharing:
              leaves the core on every store                 each core owns
                                                            its own line

The LOCK XADD is still there — atomics are atomics — but the line is in the core’s own cache in M state, so the store never leaves the core. The contention disappears; the instruction count does not.

Trade-offs

The LIFO wake-up:

Wins: no thundering herd; cache locality; idle threads exit; a notify_one instead of notify_all.
Costs: unfairness (a thread at the tail can starve); the pool must maintain an intrusive list; the wake-up path is more code than notify_all; and wait() must not wake idle threads, or LIFO ordering is destroyed — which is why it doesn’t:

cpp
// src/Common/ThreadPool.cpp:678
    /// Do NOT wake idle threads here. The LIFO scheduling guarantees that every
    /// queued job has an associated wake-up (either a notified thread popped from
    /// the idle stack, or a newly-created thread that will see the job in its
    /// first worker-loop iteration, or a busy thread that will see the queued
    /// job when it finishes its current one). Waking all idle threads would
    /// destroy LIFO ordering — the whole point of this pool design.
    job_finished.wait(lock, [this] { return scheduled_jobs == 0; });

The per-CPU counters:

Wins: counting scales across cores; the array stays in BSS; the trace path is opt-in.
Costs: reading a counter is now O(cpus) instead of O(1) (a few hundred cache-line reads — negligible for a system.query_log insert, but it is why ProfileEvents snapshots are not free); the storage is cpus × events × 8 bytes of address space (mostly untouched, thanks to lazy faulting); and sched_getcpu() is called on every increment, which is a TLS read on Linux with rseq but a real instruction sequence on other platforms.

The priority queue:

Wins: jobs can be prioritised (the AsyncLoader uses this to keep background merges from starving foreground work).
Costs: a heap is O(log n) on push and pop where a deque would be O(1); and because the queue is shared, it serialises all producers behind one mutex. For a pool whose whole point is to be the place where

all the contention goes, that is the right trade — but it is a trade.