ClickHouse's Token-Bucket Throttler Sleeps Without Burning a CPU Slot

TL;DR: A Throttler is a token bucket with a one-second burst. Every call to throttle(amount) refills the bucket by rate × elapsed, subtracts amount, and — if the bucket went negative — sleeps for exactly the deficit divided by the rate. The sleep is capped by max_block_ns, and where the thread holds a CPU lease the lease is parked for the duration, so the wait does not eat into the concurrency budget.

Rate limiting is one of the oldest problems in servers. The textbook answer is a token bucket: tokens accumulate at a fixed rate, each operation consumes some, and when you run out you wait. That handles bursts gracefully — a client can spend a second’s worth of tokens instantly and then settle into the steady-state rate.

But a token bucket is only half the problem. The other half is what the waiting costs you. A thread that sleeps still holds a CPU slot in every system that counts threads, still holds a thread stack, and still shows up in the server’s thread count. In ClickHouse, that matters more than in most servers, because ClickHouse has a CPU-slot budget that it enforces strictly (see the concurrency-control article). A query blocked on a slow disk would otherwise hold a slot it is not using.

ClickHouse’s Throttler solves both halves: it is a token bucket, and it parks the caller’s CPU lease while it sleeps.

The class

cpp
// src/Common/Throttler.h:13
/** Allows you to limit the speed of something (in tokens per second) using sleep.
  * Implemented using Token Bucket Throttling algorithm.
  * Also allows you to set a limit on the maximum number of tokens. If exceeded, an exception will be thrown.
  */
class Throttler : public IThrottler
{
public:
    static const size_t default_burst_seconds = 1;

One second of burst by default. A throttler with a 100 MB/s limit starts with 100 MB of tokens, so the first read of a 10 MB block goes through instantly even on a cold cache.

The state is small and mostly private:

cpp
// src/Common/Throttler.h:93
private:
    void throttleImpl(size_t amount, size_t & count_value, double & tokens_value);
    void throttleImpl(size_t amount, size_t & count_value, double & tokens_value, size_t & max_speed_value);

    size_t count{0};
    size_t max_speed TSA_GUARDED_BY(mutex){0}; /// in tokens per second.
    size_t max_burst TSA_GUARDED_BY(mutex){0}; /// in tokens.
    const UInt64 limit{0}; /// 0 - not limited.
    const char * limit_exceeded_exception_message = nullptr;
    mutable std::mutex mutex;
    std::atomic<UInt64> block_count{0}; // Number of threads currently under throttling
    double tokens{0}; /// Amount of tokens available in token bucket. Updated in `throttle` method.
    UInt64 prev_ns{0}; /// Previous `throttle` call time (in nanoseconds).

    /// Used to implement a hierarchy of throttlers
    ThrottlerPtr parent;

    /// See `setLimitsBlockDeviceBandwidth`.
    bool limits_block_device_bandwidth = false;

    /// Event to increment when throttler uses tokens
    ProfileEvents::Event event_amount{ProfileEvents::end()};

    /// Event to increment when throttler sleeps
    ProfileEvents::Event event_sleep_us{ProfileEvents::end()};
};

Note what is not atomic: tokens is a plain double guarded by the mutex. That is deliberate — the bucket must be updated as a unit (refill and subtract together), so there is nothing to be gained from a lock-free version.

Note also parent. A throttler can have a parent throttler, and throttle() chains into it. That is how a query-level limit is layered on top of a server-wide limit without either knowing about the other.

Construction

cpp
// src/Common/Throttler.cpp:38
Throttler::Throttler(size_t max_speed_, const ThrottlerPtr & parent_,
        ProfileEvents::Event event_amount_,
        ProfileEvents::Event event_sleep_us_)
    : max_speed(max_speed_)
    , max_burst(max_speed_ * default_burst_seconds)
    , limit_exceeded_exception_message("")
    , tokens(static_cast<double>(max_burst))
    , parent(parent_)
    , event_amount(event_amount_)
    , event_sleep_us(event_sleep_us_)
{}

The bucket starts full. tokens = max_burst, i.e. one second’s worth. This is the burst allowance: a query can read a big block immediately and only then settle into the configured rate.

The refill

The core arithmetic is six lines:

cpp
// src/Common/Throttler.cpp:144
void Throttler::throttleImpl(size_t amount, size_t & count_value, double & tokens_value, size_t & max_speed_value)
{
    std::lock_guard lock(mutex);
    auto now = clock_gettime_ns_adjusted(prev_ns);
    if (max_speed)
    {
        max_speed_value = max_speed;
        double delta_seconds = prev_ns ? static_cast<double>(now - prev_ns) / NS : 0;
        tokens = std::min<double>(
            tokens + static_cast<double>(max_speed) * delta_seconds - static_cast<double>(amount), static_cast<double>(max_burst));
    }
    count += amount;
    count_value = count;
    tokens_value = tokens;
    prev_ns = now;
}

Read it as: tokens = min(burst, tokens + rate × elapsed - amount).

Two details:

The refill is capped at max_burst. If a throttler is idle for a minute, the bucket does not accumulate a minute’s worth of tokens — it stays at one second’s worth. This is the difference between a token bucket and a leaky bucket, and it is what makes the burst bounded.

The clock is clock_gettime_ns_adjusted. That function is not a plain clock_gettime; it is a small wrapper that refuses to go backwards:

cpp
// src/Common/Stopwatch.h:29
/// Takes previously returned value and returns it again if time stepped back for some reason.
///
/// You should use this if OS does not support CLOCK_MONOTONIC_RAW
inline UInt64 clock_gettime_ns_adjusted(UInt64 prev_time, clockid_t clock_type = STOPWATCH_DEFAULT_CLOCK)
{
#ifdef CLOCK_MONOTONIC_RAW
    if (likely(clock_type == CLOCK_MONOTONIC_RAW))
        return clock_gettime_ns(clock_type);
#endif

    UInt64 current_time = clock_gettime_ns(clock_type);
    if (likely(prev_time <= current_time))
        return current_time;

    /// Something probably went completely wrong if time stepped back for more than 1 second.
    chassert(prev_time - current_time <= 1000000000ULL);
    return prev_time;
}

A throttler is a rate limiter, and a rate limiter is only correct if time moves forward. If NTP steps the clock backwards by an hour, delta_seconds goes negative and the bucket would be drained by a large negative refill — inverting the throttle. The wrapper makes the delta non-negative by clamping to the previous value.

(CLOCK_MONOTONIC_RAW is immune to NTP entirely, which is why it is the preferred clock where it exists — see the top of the same file. The adjusted wrapper is the fallback for platforms without it.)

The throttle

cpp
// src/Common/Throttler.cpp:79
bool Throttler::throttle(size_t amount, size_t max_block_ns)
{
    // Values obtained under lock to be checked after release
    size_t count_value = 0;
    double tokens_value = 0.0;
    size_t max_speed_value = 0;
    throttleImpl(amount, count_value, tokens_value, max_speed_value);
    if (event_amount != ProfileEvents::end())
        ProfileEvents::increment(event_amount, amount);

    if (limit && count_value > limit)
        throw Exception::createDeprecated(limit_exceeded_exception_message + std::string(" Maximum: ") + toString(limit), ErrorCodes::LIMIT_EXCEEDED);

    // Wait unless there is positive amount of tokens - throttling
    bool block = max_block_ns > 0 && max_speed_value > 0 && tokens_value < 0;
    if (block)
    {
        block_count.fetch_add(1, std::memory_order_relaxed);
        SCOPE_EXIT({
            block_count.fetch_sub(1, std::memory_order_relaxed);
        });

        auto & profile_events = CurrentThread::getProfileEvents();
        auto timer = profile_events.timer(ProfileEvents::ThrottlerSleepMicroseconds);
        std::optional<ProfileEvents::Timer> timer2;
        if (event_sleep_us != ProfileEvents::end())
            timer2.emplace(profile_events.timer(event_sleep_us));

        // Calculate how long to sleep
        double block_ns_double = -tokens_value / static_cast<double>(max_speed_value) * NS;
        chassert(block_ns_double >= 0.0);

        // Clamp to be safe and avoid any UB
        UInt64 block_ns = block_ns_double >= static_cast<double>(std::numeric_limits<UInt64>::max())
            ? std::numeric_limits<UInt64>::max()
            : static_cast<UInt64>(block_ns_double);

        // Note that throwing exception from the following blocking call is safe. It is important for query cancellation.
        sleep(std::min<UInt64>(max_block_ns, block_ns));
    }

    bool parent_block = false;
    if (parent)
        parent_block = parent->throttle(amount, max_block_ns);

    return block || parent_block;
}

Walk through it:

Charge the tokens. throttleImpl refills and subtracts.
Check the hard limit. count is the lifetime total. If it exceeds limit, throw LIMIT_EXCEEDED — this is how “this user has downloaded 100 GB today” is enforced, as opposed to the speed limit.
Decide whether to block. Three conditions must hold: max_block_ns > 0 (the caller allows blocking at all), max_speed_value > 0 (there is a rate limit), and tokens_value < 0 (the bucket is in debt).
Compute the sleep. -tokens_value / max_speed seconds. That is exactly the time needed for the bucket to refill back to zero.
Cap the sleep at max_block_ns. This is the crucial bit for cancellation: the caller says “you may sleep at most this long”, so a KILL QUERY is noticed within max_block_ns.
Chain to the parent, and report whether anyone blocked.

Note the return value: block || parent_block. The caller uses it to know whether it actually stalled — e.g. HTTPRequestThrottler uses the same idea for HTTP request counts.

The comment about throwing from a blocking call

cpp
// src/Common/Throttler.cpp:116
        // Note that throwing exception from the following blocking call is safe. It is important for query cancellation.
        sleep(std::min<UInt64>(max_block_ns, block_ns));

The safety here is an accounting property, not a signal-handling property. The tokens have already been spent by the time the sleep starts (the refill and the subtraction happen in throttleImpl, before the sleep). So if the sleep throws — and it can, sleepForNanoseconds throws std::system_error if clock_gettime fails — the bucket is left in a consistent state and the exception propagates to the caller as a genuine query error rather than as a corrupted rate limiter.

The sleep itself is an absolute-deadline sleep, which matters more than it looks: sleepForNanoseconds in base/base/sleep.cpp loops on clock_nanosleep(clock_type, TIMER_ABSTIME, &finish_time, nullptr) until the absolute deadline passes. With TIMER_ABSTIME, a signal does not shorten the sleep — the loop simply resumes with the same deadline. That is deliberate, because ClickHouse’s query profiler delivers SIGUSR1/SIGUSR2 to worker threads periodically, and a relative sleep would be cut short every time. The file’s own comment says so:

Sleep with nanoseconds precision. Tolerant to signal interruptions
In case query profiler is turned on, all threads spawned for query execution are repeatedly interrupted by signals from timer. Functions for relative sleep (sleep(3), nanosleep(2), etc.) have problems in this setup and man page for nanosleep(2) suggests using absolute deadlines, for instance clock_nanosleep(2).

So a throttled read waits out its full deficit regardless of what the profiler is doing.

`max_block_ns`: a bound nobody uses by default

The Poco interface this was inherited from documents the contract: “It never blocks longer than max_block_ns nanoseconds”, with an unlimited_block_ns = static_cast<size_t>(-1) constant, and the one-argument overload that uses it:

cpp
// src/Common/IThrottler.h:38
    bool throttleOSPageCacheRead(size_t amount)
    {
        return throttleOSPageCacheRead(amount, unlimited_block_ns);
    }

The same one-argument shape is what almost every call site uses — read_settings.remote_throttler->throttle(bytes_read) in the S3, HDFS, Azure and file-descriptor read buffers. So in practice a throttled read sleeps for exactly as long as its deficit requires, and the max_block_ns cap is a knob that exists for callers that care.

Parking the CPU lease

Here is the part that ties the throttler to the concurrency machinery:

cpp
// src/Common/Throttler.cpp:65
void Throttler::sleep(UInt64 nanoseconds)
{
#if USE_SILK
    if (Silk::isInsideFiber())
    {
        silk::FiberScheduler::sleep(nanoseconds);
        return;
    }
#endif
    /// Park the CPU lease while sleeping
    CPULeaseParkGuard park_guard;
    sleepForNanoseconds(nanoseconds);
}

Two different worlds:

If the thread is running inside a Silk fiber (ClickHouse’s fiber scheduler), the fiber is suspended — the underlying thread is released and can run other fibers. No CPU is consumed at all.
Otherwise, the OS thread sleeps, and the CPU lease is parked so that the slot is not counted against the concurrency limit during the sleep.

The guard is a thread-local, nesting-safe RAII:

cpp
// src/Common/Scheduler/CurrentCPULease.cpp:24
CPULeaseParkGuard::CPULeaseParkGuard()
    : lease(current_cpu_lease)
    , parked(false)
{
    if (lease)
    {
        // Only the outermost guard parks, and it unparks only if `park` succeeded
        if (cpu_park_depth == 0)
            parked = lease->park();
        ++cpu_park_depth;
    }
}

CPULeaseParkGuard::~CPULeaseParkGuard()
{
    if (lease)
    {
        --cpu_park_depth;
        if (parked)
            lease->unpark();
    }
}

The lease itself lives in a thread-local:

cpp
// src/Common/Scheduler/CurrentCPULease.cpp:8
    /// The CPU lease of the currently-running executor worker thread, or nullptr.
    thread_local ISlotLease * current_cpu_lease = nullptr;
    /// Number of active CPULeaseParkGuard scopes on this thread (for nesting).
    thread_local size_t cpu_park_depth = 0;

And the interface it drives:

cpp
// src/Common/ISlotControl.h:74
    /// Park the slot during a non-CPU wait, so it does not count against the CPU concurrency limit.
    /// Never blocks. Returns true if parked; then `unpark` must be called once.
    [[nodiscard]] virtual bool park() { return false; }

    /// Take the slot back after a successful `park`. Borrows a slot and never blocks.
    virtual void unpark() {}

“Never blocks” and “borrows a slot” are the key words: parking does not give the slot away to another query, it just stops counting it. So a query blocked on a network read keeps its place in the queue but does not consume concurrency budget.

That is what makes a throttler usable inside a slot-limited server. Without it, a query throttled to 1 MB/s would hold 16 CPU slots while sleeping, and the server would appear to be at its concurrency limit when it is actually idle.

Hierarchies and combinations

There are two ways to compose throttlers, and they mean different things.

Hierarchy (parent): a chain where every level applies. The query throttler charges the server throttler too:

cpp
// src/Interpreters/Context.cpp:6423
ThrottlerPtr Context::getRemoteReadThrottler(std::optional<UInt64> bandwidth) const
{
    ThrottlerPtr throttler = getServerWideRemoteReadThrottler();

    /// User-level throttler (`max_network_bandwidth_for_user` / `max_network_bandwidth_for_all_users`).
    if (auto process_list_element = getProcessListElementSafe())
        addThrottler(throttler, process_list_element->getUserNetworkThrottler());

    /// This mutex cannot be upgraded, so the shared lock is released before the exclusive one below.
    if (!bandwidth)
    {
        SharedLockGuard settings_lock(mutex);
        bandwidth = getSettingsRef()[Setting::max_remote_read_network_bandwidth];
    }

    if (*bandwidth)
    {
        std::lock_guard lock(mutex);
        if (!remote_read_query_throttler)
            remote_read_query_throttler = std::make_shared<Throttler>(*bandwidth, throttler, ProfileEvents::QueryRemoteReadThrottlerBytes, ProfileEvents::QueryRemoteReadThrottlerSleepMicroseconds);
        throttler = remote_read_query_throttler;
    }
    return throttler;
}

Combination (ThrottlerArray): unrelated limits that must all be satisfied, but where a single one of them is enough to describe the bottleneck. The backup worker uses this, because a backup is bounded by both its own bandwidth limit and the server’s remote bandwidth limit:

cpp
// src/Common/ThrottlerArray.h:107
/// Adds a new throttler to the existing one, combining them if necessary.
/// Note that it does not modify new throttler and does not modify hierarchy of existing throttlers.
inline void addThrottler(ThrottlerPtr & throttler, const ThrottlerPtr & new_throttler)
{
    if (!throttler)
        throttler = new_throttler; // Replace empty throttler with new one
    else if (new_throttler)
        throttler = std::make_shared<ThrottlerArray<2>>(throttler, new_throttler); // Combine existing and new throttler
}

and ThrottlerArray::throttle charges every element:

cpp
// src/Common/ThrottlerArray.h:27
    /// Use `amount` tokens on all throttlers
    bool throttle(size_t amount, size_t max_block_ns) override
    {
        bool result = false;
        for (const auto & throttler : throttlers)
        {
            bool blocked = throttler->throttle(amount, max_block_ns);
            result = result || blocked;
        }
        return result;
    }

Note that this can double-sleep: if both throttlers are in debt, the first sleeps for its deficit and the second sleeps for its own. That is intentional — each throttler independently guarantees its own rate.

Page-cache reads: the interesting exception

A throttler that exists to limit disk bandwidth should not count reads that never touched the disk. ClickHouse can detect those, and the interface has a dedicated method for it:

cpp
// src/Common/IThrottler.h:29
    /// Charge `amount` tokens for the data that was copied from the OS page cache,
    /// i.e. without performing any block device I/O.
    /// Throttlers that exist to limit the bandwidth of a block device ignore such reads,
    /// all the others account them as usual.
    virtual bool throttleOSPageCacheRead(size_t amount, size_t max_block_ns)
    {
        return throttle(amount, max_block_ns);
    }

The default implementation charges them like any other read. The disk-bandwidth throttler overrides it to ignore them:

cpp
// src/Common/Throttler.cpp:127
bool Throttler::throttleOSPageCacheRead(size_t amount, size_t max_block_ns)
{
    if (!limits_block_device_bandwidth)
        return throttle(amount, max_block_ns);

    /// This throttler does not account the read, but a parent still can.
    if (parent)
        return parent->throttleOSPageCacheRead(amount, max_block_ns);
    return false;
}

The limits_block_device_bandwidth flag is set by setLimitsBlockDeviceBandwidth(), and it is what ignoresOSPageCacheReads() reports:

cpp
// src/Common/Throttler.h:59
    bool ignoresOSPageCacheReads() const override
    {
        return (limits_block_device_bandwidth && getMaxSpeed() > 0) || (parent && parent->ignoresOSPageCacheReads());
    }

The caller then only pays for the detection when a throttler that cares is actually present:

cpp
// src/IO/ReadBufferFromFileDescriptor.cpp:145
void ReadBufferFromFileDescriptor::enableOSPageCacheReadsDetection(int flags)
{
    /// `RWF_NOWAIT` is ignored for `O_DIRECT` (see the comment in `ThreadPoolReader::submit`),
    /// so such a read could reach the device while looking like it was served from the page cache.
    /// The `O_DIRECT` check comes before `preadNoWaitUnavailableReason` for the same reason as there.
    /// The detection costs an extra system call for the data that is not in the page cache,
    /// which is why it is used only when there is a throttler that ignores such reads
    /// (e.g. not for merges, mutations or backups that are throttled only by their own bandwidth limits).
    bool enable = use_pread
        && throttler
        && throttler->ignoresOSPageCacheReads()
        && (flags == -1 || !(flags & O_DIRECT))
        && preadNoWaitUnavailableReason().empty();
    detect_os_page_cache_reads.store(enable, std::memory_order_relaxed);
}

and then the actual read path branches on the detection result:

cpp
// src/IO/ReadBufferFromFileDescriptor.cpp:105
        if (res > 0)
        {
            bytes_read += res;
            if (throttler)
            {
                /// See the comment in `AsynchronousReadBufferFromFileDescriptor::nextImpl`.
                if (from_os_page_cache)
                {
                    ProfileEvents::increment(ProfileEvents::ReadBufferFromFileDescriptorPageCacheHitBytes, res);
                    throttler->throttleOSPageCacheRead(res);
                }
                else
                    throttler->throttle(res);
            }
        }

This is a nice example of a policy decision (“a disk bandwidth limit should not punish a warm cache”) implemented as a three-way branch: the default implementation charges, the specialised one ignores, and the detection cost is only paid when it matters.

The settings that reach it

cpp
// src/Core/Settings.cpp:1032
    DECLARE(UInt64, max_local_read_bandwidth, 0, R"(
The maximum speed of local reads in bytes per second.

The limit applies to the data read from the block devices: reads that are served from the OS page cache are not accounted for, as long as the read method can detect them (which is the case for the default `local_filesystem_read_method = 'pread_threadpool'`, for `pread`, and for the reads of the filesystem cache files).
)", 0) \
    DECLARE(UInt64, max_local_write_bandwidth, 0, R"(
The maximum speed of local writes in bytes per second.
)", 0) \

Zero means unlimited — and, importantly, an unlimited throttler never blocks, which is why the code has an ignoresOSPageCacheReads() shortcut rather than always detecting page-cache hits.

A query-level limit looks like this:

sql
SELECT *
FROM s3('https://bucket.s3.amazonaws.com/prefix/*.parquet')
SETTINGS max_remote_read_network_bandwidth = 1000000;  -- 1 MB/s

and the server-wide equivalents are max_local_read_bandwidth, max_local_write_bandwidth, max_remote_read_network_bandwidth and max_remote_write_network_bandwidth. The counters to watch afterwards:

sql
SELECT event, value
FROM system.events
WHERE event ILIKE '%Throttler%' OR event ILIKE 'QueryRemote%Throttler%'
ORDER BY event;

-- event                                          -- value
-- ThrottlerSleepMicroseconds                     -- 881230
-- QueryRemoteReadThrottlerBytes                  -- 41200334
-- QueryRemoteReadThrottlerSleepMicroseconds      -- 771200

ThrottlerSleepMicroseconds is the total time spent sleeping in throttlers across all queries; the per-scope counters say which limit caused it.

Trade-offs

The token bucket:

Wins: bounded bursts (one second) so a cold query is not stalled at the start; the rate is a true long-run average; the deficit-based sleep is exact rather than a fixed quantum.
Costs: the bucket is a double under a mutex, so every throttled read takes a lock. For a 4 KB read at 100 MB/s that is a lock acquisition per 4 KB — which is why the throttler is called with the amount of a whole read rather than per byte.

Sleeping instead of signalling:

Wins: no queue, no wake-up ordering, no notification storm. The throttled thread simply sleeps and then proceeds.
Costs: a sleeping thread holds a thread stack and, without the CPU-lease parking, a concurrency slot. It also means the first thread to be throttled sets the pace for everyone behind it, rather than the throttle being a shared queue that everyone advances through.

The max_block_ns cap:

Wins: the interface guarantees the caller that a throttled operation never blocks longer than it asked to be blocked.
Costs: in practice almost every call site uses the unlimited variant, so a badly configured throttle can park a query thread for an arbitrarily long time. The cap exists; it is just not used.

Parking the CPU lease:

Wins: a throttled query does not consume concurrency budget, so the server’s CPU-slot accounting reflects real CPU usage.
Costs: parking is only implemented for leases (the workload-resource scheduler), not for plain ConcurrencyControl slots — ISlotLease::park() has a default no-op body, so under plain concurrency control the thread keeps its slot while it sleeps. The distinction is between “this thread holds a resource lease that may be preempted” and “this thread holds a slot in a counting scheme”.