ClickHouse Turns an Out-of-Memory Into an Exception

TL;DR: ClickHouse does not kill the process when a query runs out of memory. It refuses the allocation, throws DB::Exception with code MEMORY_LIMIT_EXCEEDED (code 241), unwinds the query, and keeps serving. Before it gives up it tries three things: shrink the page cache, shrink the columns cache, and ask the overcommit tracker to kill a different query.

Ask a C++ programmer what should happen when a process runs out of memory and they will say “throw std::bad_alloc”. Ask them what should happen when one query out of two hundred runs out of memory, on a server where the other one hundred and ninety-nine are perfectly healthy, and the answer gets complicated fast.

There are two naive answers, and ClickHouse rejects both:

Kill the process. One query’s GROUP BY on an unbounded key kills every other query, the replication queue, and the HTTP endpoint. Unacceptable in a multi-tenant server.
Let the allocation succeed and rely on the OOM killer. The Linux OOM killer picks a victim by badness heuristics, which has nothing to do with which query in ClickHouse actually caused the problem. Unacceptable, and ClickHouse runs in containers where the cgroup limit, not the host’s RAM, is the real ceiling.

The design that ClickHouse actually implements is: the allocation is refused, the refusal is an exception, and the exception unwinds only the offending query.

The refusal happens before the allocation

The first thing to understand is where the check happens. It is not after the memory is obtained — it is before the allocator is even called.

cpp
// src/Common/Allocator.cpp:73
template <bool clear_memory, bool populate>
void * allocImpl(size_t size, size_t alignment)
{
    auto trace = CurrentMemoryTracker::alloc(size);

    void * buf = nullptr;
    if (alignment <= MALLOC_MIN_ALIGNMENT) [[likely]]
    {
        if constexpr (clear_memory)
            buf = __real_calloc(size, 1);
        else
            buf = __real_malloc(size);

        if (nullptr == buf) [[unlikely]]
        {
            [[maybe_unused]] auto rollback_trace = CurrentMemoryTracker::free(size);
            throw DB::ErrnoException(
                DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY, "Allocator: Cannot malloc {}.", ReadableSize(static_cast<double>(size)));
        }

CurrentMemoryTracker::alloc(size) is the first statement. If it throws, __real_malloc is never called and the process never touches a new page. The CANNOT_ALLOCATE_MEMORY throw further down is a different failure entirely: the limit was fine, the OS simply refused.

Note also the rollback on the failure path. The tracker was already charged for size, so if malloc fails, the charge is undone before the exception leaves:

cpp
// src/Common/Allocator.cpp:98
            [[maybe_unused]] auto rollback_trace = CurrentMemoryTracker::free(size);

This “charge first, undo on failure” discipline is repeated at every level of the tracker chain, and the whole MemoryTracker test suite exists to pin it down.

Two different failure codes

There are two error codes in play, and confusing them leads to bad tuning.

Situation	Code	Meaning
`max_memory_usage` / `max_memory_usage_for_user` / `max_server_memory_usage` exceeded	`MEMORY_LIMIT_EXCEEDED` (241)	ClickHouse refused the allocation on purpose
The OS refused the allocation	`CANNOT_ALLOCATE_MEMORY` (173)	The limit was fine; the machine could not give the memory

MEMORY_LIMIT_EXCEEDED is a policy decision and is recoverable: the query is cancelled, other queries are unaffected, and the server keeps running. CANNOT_ALLOCATE_MEMORY means the kernel said no, which usually means the cgroup limit or the host’s swap situation is the real problem.

The message

The global-level message is much richer than the query-level one, because the global level is the only place that knows about RSS, the page cache, and the overcommit decision:

cpp
// src/Common/MemoryTracker.cpp:529
                if (level == VariableContext::Global)
                {
                    const Int64 untracked = DB::UntrackedMemoryRegistry::instance().sum();
                    throw DB::Exception(
                        DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
                        "{}{} exceeded: "
                        "would use {} (attempt to allocate chunk of {}), "
                        "current RSS: {}"
                        "{}, " /// page cache
                        "maximum: {}."
                        "{} Untracked memory across all threads: {}.",
                        description ? description : "",
                        description ? " memory limit" : "Memory limit",
                        formatReadableSizeWithBinarySuffix(will_be),
                        formatReadableSizeWithBinarySuffix(size),
                        formatReadableSizeWithBinarySuffix(rss.load(std::memory_order_relaxed)),
                        page_cache_ptr ? ", userspace page cache " + formatReadableSizeWithBinarySuffix(page_cache_ptr->sizeInBytes()) : "",
                        formatReadableSizeWithBinarySuffix(current_hard_limit),
                        overcommit_result_ignore ? "" : fmt::format(" OvercommitTracker decision: {}.", toDescription(overcommit_result)),
                        formatReadableSizeWithBinarySuffix(untracked));
                }
                else
                {
                    throw DB::Exception(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
                        "{}{} exceeded: would use {} (attempt to allocate chunk of {}), maximum: {}.{}",
                        description ? description : "",
                        description ? " memory limit" : "Memory limit",
                        formatReadableSizeWithBinarySuffix(will_be),
                        formatReadableSizeWithBinarySuffix(size),
                        formatReadableSizeWithBinarySuffix(current_hard_limit),
                        overcommit_result_ignore ? "" : fmt::format(" OvercommitTracker decision: {}.", toDescription(overcommit_result)));
                }

Reading a server-level MEMORY_LIMIT_EXCEEDED:

current RSS is the process’s real resident set, as corrected by the background memory worker.
userspace page cache is how much of that RSS is ClickHouse’s own page cache — memory the server could give back.
Untracked memory across all threads is the total of the per-thread deferred buffers. This is the number that tells you whether the limit was hit because of real usage or because a lot of threads had small amounts pending.

The query-level message has no RSS and no untracked total, because at that level the tracker does not know them.

The three-stage escalation

Before throwing, ClickHouse tries to not throw. The escalation ladder lives in the will_be > current_hard_limit branch:

cpp
// src/Common/MemoryTracker.cpp:481
        /// Try to evict from userspace page cache or kill a lower-priority query to free up memory.
        /// These operations are relatively slow, so we do them only if memoryTrackerCanThrow;
        /// otherwise all allocations may get slow when above memory limit.
        if (memoryTrackerCanThrow(level, false) && enforce_memory_limit)
        {
            OvercommitResult overcommit_result = OvercommitResult::NONE;

            /// Try to shrink the userspace page cache.
            DB::PageCache * page_cache_ptr = nullptr;
            if (level == VariableContext::Global)
                page_cache_ptr = page_cache.load(std::memory_order_relaxed);
            if (page_cache_ptr)
            {
                ProfileEvents::increment(ProfileEvents::PageCacheOvercommitResize);
                if (page_cache_ptr->autoResize(std::max(will_be, will_be_rss), current_hard_limit))
                    overcommit_result = OvercommitResult::MEMORY_FREED;
            }

            /// Then the columns cache, which also gives its memory back on demand.
            DB::IMemoryReleasableCache * releasable_cache_ptr = nullptr;
            if (overcommit_result == OvercommitResult::NONE && level == VariableContext::Global
                && (releasable_cache_ptr = DB::getMemoryReleasableCache()))
            {
                if (releasable_cache_ptr->autoResize(std::max(will_be, will_be_rss), current_hard_limit))
                    overcommit_result = OvercommitResult::MEMORY_FREED;
            }

            /// If that wasn't enough, try to stop some query.
            if (overcommit_result == OvercommitResult::NONE)
            {
                if (auto * overcommit_tracker_ptr = overcommit_tracker.load(std::memory_order_relaxed); overcommit_tracker_ptr && query_tracker != nullptr)
                    overcommit_result = overcommit_tracker_ptr->needToStopQuery(query_tracker, size);
            }

Three rungs, cheapest first:

Shrink the page cache. ClickHouse’s own userspace cache of file contents is memory it can return to the OS without hurting any query.
Shrink the columns cache. Same idea for cached columnar data.
Kill a different query. The overcommit tracker picks a victim.

If any of the three frees enough memory, the allocation proceeds and no exception is thrown. That is the MEMORY_FREED result.

This is why the query_tracker pointer is threaded down the tracker chain: the global level has no idea which query is asking, and the overcommit tracker needs to know the victim in order to compare it against the requester.

Picking the victim

The victim is chosen by ratio, not by size. The user-level tracker picks the query with the highest amount / soft_limit:

cpp
// src/Common/OvercommitTracker.cpp:182
void UserOvercommitTracker::pickQueryToExcludeImpl()
{
    MemoryTracker * query_tracker = nullptr;
    OvercommitRatio current_ratio{0, 0};
    // At this moment query list must be read only.
    // This is guaranteed by locking global_mutex in OvercommitTracker::needToStopQuery.
    auto & queries = user_process_list->queries;
    for (auto const & query : queries)
    {
        if (query.second->isKilled())
            continue;

        auto * memory_tracker = query.second->getMemoryTracker();
        if (!memory_tracker)
            continue;

        auto ratio = memory_tracker->getOvercommitRatio();
        if (ratio.soft_limit != 0 && current_ratio < ratio)
        {
            query_tracker = memory_tracker;
            current_ratio   = ratio;
        }
    }
    picked_tracker = query_tracker;
}

The global tracker uses the user’s soft limit as the denominator, so it compares queries across users:

cpp
// src/Common/OvercommitTracker.cpp:213
void GlobalOvercommitTracker::pickQueryToExcludeImpl()
{
    MemoryTracker * query_tracker = nullptr;
    OvercommitRatio current_ratio{0, 0};
    // At this moment query list must be read only.
    // This is guaranteed by locking global_mutex in OvercommitTracker::needToStopQuery.
    for (auto const & query : process_list->processes)
    {
        if (query->isKilled())
            continue;

        Int64 user_soft_limit = 0;
        if (auto const * user_process_list = query->getUserProcessList())
            user_soft_limit = user_process_list->user_memory_tracker.getSoftLimit();
        if (user_soft_limit == 0)
            continue;

        auto * memory_tracker = query->getMemoryTracker();
        if (!memory_tracker)
            continue;
        auto ratio = memory_tracker->getOvercommitRatio(user_soft_limit);
        if (current_ratio < ratio)
        {
            query_tracker = memory_tracker;
            current_ratio   = ratio;
        }
    }
    picked_tracker = query_tracker;
}

The ratio is the key idea: a query holding 900 MiB against a 1 GiB soft limit (90%) is a better victim than a query holding 20 GiB against a 100 GiB limit (20%), even though it holds less memory. The first query is close to the edge; the second is not.

The wait

Once a victim is picked, the requesting thread does not immediately give up. It waits for the victim to die:

cpp
// src/Common/OvercommitTracker.cpp:78
    if (picked_tracker == tracker)
    {
        // Query of the provided as an argument memory tracker was chosen.
        // It may happen even when current state is RUNNING, because
        // ThreadStatus::~ThreadStatus may call MemoryTracker::alloc.
        cancellation_state = QueryCancellationState::RUNNING;
        return OvercommitResult::SELECTED;
    }

    allow_release = true;

    required_memory += amount;
    auto wait_start_time = std::chrono::steady_clock::now();
    auto released = [this, id]()
    {
        return id < id_to_release || cancellation_state == QueryCancellationState::NONE;
    };
    bool satisfied = false;
    for (auto remaining = max_wait_time; remaining > ZERO_MICROSEC;)
    {
        const auto slice = std::min(remaining, MAX_WAIT_SLICE);
        if (cv.wait_for(lk, slice, released))
        {
            satisfied = true;
            break;
        }
        remaining -= slice;
    }

The wait is bounded by memory_usage_overcommit_max_wait_microseconds, set on the query’s tracker. The MAX_WAIT_SLICE of 24 hours exists because wait_for computes its deadline in nanoseconds and a nanosecond clock cannot represent a deadline that far away:

cpp
// src/Common/OvercommitTracker.cpp:25
/// `wait_for` adds its duration to `steady_clock::now` in nanoseconds, which cannot represent a
/// far-future deadline, so a wait longer than one slice is taken in slices and re-armed.
constexpr std::chrono::microseconds MAX_WAIT_SLICE = std::chrono::hours(24);

The four possible outcomes are:

Result	Meaning
`MEMORY_FREED`	The victim freed enough; the allocation proceeds
`SELECTED`	The requesting query is the victim; it should die
`TIMEOUTED`	The victim did not free enough in time
`NOT_ENOUGH_FREED`	The victim died but did not free enough
`DISABLED`	Overcommit is off (`max_wait_time == 0`)

DISABLED is the default outcome when the soft limit is zero, which is why memory_overcommit_ratio_denominator has a non-zero default of 1 GiB.

Not throwing at all: the lock

There is one situation where the tracker must not throw: while an exception is already in flight. Throwing a second exception from a destructor during unwinding is std::terminate, and a server that terminates because a query ran out of memory is exactly the outcome this design exists to prevent.

cpp
// src/Common/MemoryTracker.cpp:58
bool inline memoryTrackerCanThrow(VariableContext level, bool fault_injection)
{
    return !LockMemoryExceptionInThread::isBlocked(level, fault_injection) && !std::uncaught_exceptions();
}

Two guards in one expression:

LockMemoryExceptionInThread::isBlocked is an explicit opt-out, and it is scoped by level so a destructor can block only the Global level:

cpp
// src/Common/LockMemoryExceptionInThread.h:40
    static bool isBlocked(VariableContext current_level, bool fault_injection)
    {
        return counter > 0 && current_level >= level && (!fault_injection || block_fault_injections);
    }

std::uncaught_exceptions() is the implicit opt-out. The header says so:

cpp
// src/Common/LockMemoryExceptionInThread.h:16
/// NOTE: MEMORY_LIMIT_EXCEEDED Exception implicitly blocked if
/// stack unwinding is currently in progress in this thread (to avoid
/// std::terminate()), so you don't need to use it in this case explicitly.

When the tracker cannot throw, it ignores the limit and accounts the allocation anyway:

cpp
// src/Common/MemoryTracker.cpp:568
        else
        {
            memory_limit_exceeded_ignored = true;
        }

and the ignored allocation is counted, because silently dropping it would make the accounting wrong:

cpp
// src/Common/MemoryTracker.cpp:651
    if (memory_limit_exceeded_ignored || !enforce_memory_limit)
        incrementAllocationWithoutCheck(size);

The incrementAllocationWithoutCheck helper even emits a trace for large ignored allocations, because “we went over the limit and nobody noticed” is exactly the kind of thing you want to see in trace_log:

cpp
// src/Common/MemoryTracker.cpp:282
    /// The choice is arbitrary (maybe we should decrease it)
    constexpr Int64 threshold = 16 * 1024 * 1024;

    if (size > threshold)
    {
        DB::TraceSender::send(DB::TraceType::MemoryAllocatedWithoutCheck, StackTrace(), DB::TraceSender::Extras{
            .size = size,
            .memory_blocked_context = MemoryTrackerBlockerInThread::getLevel(),
        });
    }

The recursion guard

There is a nasty chicken-and-egg problem in throwing the exception. The DB::Exception constructor formats a message, which allocates a std::string, which calls operator new, which calls the memory tracker, which is in the middle of throwing. The fix is a blocker scope around the throw:

cpp
// src/Common/MemoryTracker.cpp:522
                /// Prevent recursion. Exception::ctor -> std::string -> new[] -> MemoryTracker::alloc
                MemoryTrackerBlockerInThread untrack_lock(VariableContext::Global);
                ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
                if (level == VariableContext::Global)
                    ProfileEvents::increment(ProfileEvents::GlobalMemoryLimitExceeded);

The same guard appears before every other throw in the file, including the fault-injection path and the peak-update path.

The blocker works by level, and the levels are ordered:

cpp
// src/Common/MemoryTracker.h:39
    /// level_ - block in level and above
    explicit MemoryTrackerBlockerInThread(VariableContext level_ = VariableContext::User);

with the test being a simple >=:

cpp
// src/Common/MemoryTrackerBlockerInThread.h:39
    static bool isBlocked(VariableContext current_level)
    {
        return current_level >= level;
    }

So MemoryTrackerBlockerInThread(VariableContext::Global) blocks everything, and MemoryTrackerBlockerInThread(VariableContext::User) blocks the user, process and thread levels but lets the global level still count.

Rolling back the charge

When the tracker refuses the allocation, it has already added the bytes to its own counter (the check happens after the fetch_add). The rollback is the decrementLocalUsage call:

cpp
// src/Common/MemoryTracker.cpp:593
Int64 MemoryTracker::decrementLocalUsage(Int64 size) noexcept
{
    Int64 accounted_size = size;
    if (level == VariableContext::Global)
    {
        amount.fetch_sub(accounted_size, std::memory_order_relaxed);
        rss.fetch_sub(accounted_size, std::memory_order_relaxed);
    }
    else if (level == VariableContext::Thread)
    {
        /// Could become negative if memory allocated in this thread is freed in another one
        amount.fetch_sub(accounted_size, std::memory_order_relaxed);
    }
    else
    {
        /// A query may free memory it never held (e.g. a cache eviction): take only what is there, in one
        /// compare-exchange so that concurrent frees cannot together remove more than is there.
        Int64 current = amount.load(std::memory_order_relaxed);
        do
        {
            accounted_size = std::min(size, std::max<Int64>(current, 0));
        }
        while (!amount.compare_exchange_weak(current, current - accounted_size, std::memory_order_relaxed));
    }

    return accounted_size;
}

Note the three cases: the global level subtracts from both amount and rss; the thread level subtracts unconditionally (and may go negative, which the comment admits); the user and process levels saturate at zero.

Because the parent chain unwinds through a catch (...) that also calls decrementLocalUsage, a refusal at the global level rolls back every level that already took the charge. The unit test states this as a property:

cpp
// src/Common/tests/gtest_memory_tracker.cpp:198
TEST(MemoryTracker, RepeatedParentLimitFailuresDoNotAccumulate)
{
    MemoryTrackerHierarchy hierarchy;
    hierarchy.user.setHardLimit(LIMIT);

    runInThread(hierarchy, [&](MemoryTracker &)
    {
        for (size_t attempt = 0; attempt < 100; ++attempt)
        {
            expectMemoryLimitExceeded(OVER_LIMIT);
            expectUsage(hierarchy, 0);
        }
    });
}

One hundred refusals, and the accounting is still exactly zero. Without the rollback, one refusal would leave a permanent residue equal to the refused size — and a server that refuses the same 1 GB allocation a thousand times would believe it is using a terabyte.

The peak is not updated by a refused allocation

A refused allocation must not move the peak counter, because the peak is “the most memory this scope actually held”. The check for the peak happens only after the allocation survives:

cpp
// src/Common/MemoryTracker.cpp:621
void MemoryTracker::commitAllocation(Int64 size, Int64 will_be, bool memory_limit_exceeded_ignored, bool enforce_memory_limit) noexcept
{
    /// Only reached once the allocation survived every limit check, so a refused or reverted add is
    /// never reported here.
    if (level == VariableContext::Global)

and the peak update itself is guarded by the “ignored” flag:

cpp
// src/Common/MemoryTracker.cpp:647
    bool peak_updated = false;
    if (!memory_limit_exceeded_ignored)
        peak_updated = updatePeak(will_be, enforce_memory_limit);

Two tests pin this down — ParentLimitFailureDoesNotUpdatePeaks and ParentLimitFailureDoesNotUpdateMetrics — because a peak that counted refused allocations would make peak_memory_usage in system.query_log meaningless.

Fault injection

ClickHouse can be told to fail allocations on purpose, which is how MEMORY_LIMIT_EXCEEDED handling gets tested without actually exhausting memory:

cpp
// src/Common/MemoryTracker.cpp:399
    double current_fault_probability = fault_probability.load(std::memory_order_relaxed);
    std::bernoulli_distribution fault(current_fault_probability);
    if (unlikely(current_fault_probability > 0.0 && fault(thread_local_rng)))
    {
        if (memoryTrackerCanThrow(level, true) && enforce_memory_limit)
        {
            decrementLocalUsage(size);

            /// Prevent recursion. Exception::ctor -> std::string -> new[] -> MemoryTracker::alloc
            MemoryTrackerBlockerInThread untrack_lock(VariableContext::Global);

            ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
            if (level == VariableContext::Global)
                ProfileEvents::increment(ProfileEvents::GlobalMemoryLimitExceeded);
            const auto * description = description_ptr.load(std::memory_order_relaxed);
            throw DB::Exception(
                DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
                "{}{}: fault injected. Would use {} (attempt to allocate chunk of {}), maximum: {}",

The probability is capped:

cpp
// src/Common/MemoryTracker.h:267
    void setFaultProbability(double value)
    {
        /// Cap to 0.5 to avoid infinite loops where every allocation fails
        /// and operations that retry on memory errors can never make progress.
        fault_probability.store(std::min(value, 0.5), std::memory_order_relaxed);
    }

A cap of 0.5 is not arbitrary. A retry loop that allocates a buffer, fails, frees it and tries again would spin forever if every allocation failed with probability 1. With probability 0.5, a retry succeeds on average on the second attempt.

There is also a manual version, injectFault, which throws at a chosen point rather than at a random allocation:

cpp
// src/Common/MemoryTracker.cpp:247
void MemoryTracker::injectFault() const
{
    if (!memoryTrackerCanThrow(level, true))
    {
        LOG_WARNING(getLogger("MemoryTracker"),
                    "Cannot inject fault at specific point. Uncaught exceptions: {}, stack trace:\n{}",
                    std::uncaught_exceptions(), StackTrace().toString());
        return;
    }

    /// Prevent recursion. Exception::ctor -> std::string -> new[] -> MemoryTracker::alloc
    MemoryTrackerBlockerInThread untrack_lock(VariableContext::Global);

    ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
    if (level == VariableContext::Global)
        ProfileEvents::increment(ProfileEvents::GlobalMemoryLimitExceeded);
    const auto * description = description_ptr.load(std::memory_order_relaxed);
    throw DB::Exception(
        DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
        "{}{}: fault injected (at specific point)",
        description ? description : "",
        description ? " memory tracker" : "Memory tracker");
}

What happens to the query afterwards

The exception unwinds through the query’s execution stack. ClickHouse’s handler for it lives where all the other query errors are handled, and it records the failure in system.query_log:

sql
SELECT
    query_id,
    exception_code,
    exception,
    memory_usage,
    peak_memory_usage
FROM system.query_log
WHERE exception_code = 241
ORDER BY event_time DESC
LIMIT 20;

Two things are worth noting about the numbers in that row:

memory_usage is the memory the query holds when it died, not the memory it tried to allocate. The refused allocation was rolled back.
peak_memory_usage is the peak of what it actually held. The refused allocation never moved the peak.

If the query was the one the overcommit tracker selected as the victim, the message carries the extra fragment:

text

Memory limit exceeded: would use 20.00 GiB (attempt to allocate chunk of 1.00 GiB), maximum: 19.00 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.

which comes from the toDescription helper:

cpp
// src/Common/MemoryTracker.cpp:92
    case OvercommitResult::SELECTED:
        return "Query was selected to stop by OvercommitTracker";

Note the asymmetry in the two cases: the requesting query survives. It blocked inside needToStopQuery until the victim died, and then it retried the allocation. The victim is the one whose system.query_log row carries the MEMORY_LIMIT_EXCEEDED.

How the victim actually dies

The overcommit tracker does not cancel the victim directly. It marks it, and the victim kills itself the next time it allocates:

cpp
// src/Common/OvercommitTracker.cpp:78
    if (picked_tracker == tracker)
    {
        // Query of the provided as an argument memory tracker was chosen.
        // It may happen even when current state is RUNNING, because
        // ThreadStatus::~ThreadStatus may call MemoryTracker::alloc.
        cancellation_state = QueryCancellationState::RUNNING;
        return OvercommitResult::SELECTED;
    }

The SELECTED result is not MEMORY_FREED, so the requester’s allocImpl takes the throw path. The victim is the query with the worst ratio, i.e. the one closest to its own limit, so it hits the limit next and finds itself already picked.

When the victim dies, the requester is woken up:

cpp
// src/Common/OvercommitTracker.cpp:151
void OvercommitTracker::onQueryStop(MemoryTracker * tracker)
{
    DENY_ALLOCATIONS_IN_SCOPE;

    {
        std::shared_lock read_lock(overcommit_m);
        if (picked_tracker != tracker)
            return;
    }

    std::lock_guard lk(overcommit_m);
    if (picked_tracker == tracker)
    {
        reset();
        cv.notify_all();
    }
}

and reset() puts the state machine back to NONE, which is the predicate the waiting thread is blocked on. The waiting thread is also released directly when the victim frees memory rather than dying:

cpp
// src/Common/MemoryTracker.cpp:678
Int64 MemoryTracker::adjustLocally(Int64 size)
{
    Int64 removed = decrementLocalUsage(size);
    if (removed < 0)
        updatePeak(amount.load(std::memory_order_relaxed), /*log_memory_usage*/ false);
    else if (removed > 0)
    {
        if (auto * overcommit_tracker_ptr = overcommit_tracker.load(std::memory_order_relaxed))
            overcommit_tracker_ptr->tryContinueQueryExecutionAfterFree(removed);
    }

So the victim does not have to die for the requester to proceed — freeing memory is enough.

What to go do about it

The one thing to remember: MEMORY_LIMIT_EXCEEDED is a refusal, not a crash. The server is fine. Your query is not.

The part people get wrong: raising max_memory_usage is the wrong fix when the message is a server-level one. If the message mentions current RSS, the server as a whole is the constraint, and the right move is to shed load, tune the page cache, or reduce concurrency — not to give one query more room.

The other part people get wrong: the exception is thrown from the allocation site, not from the query’s entry point. The stack trace in system.query_log therefore points at whatever code happened to ask for the memory — a hash table growing, a std::string being built — not at the SELECT that caused it. If you want the offending allocation, that is what min_allocation_size_to_log_stack_trace is for.

sql
SELECT value
FROM system.server_settings
WHERE name = 'min_allocation_size_to_log_stack_trace';

It needs a running trace_log, and the source is blunt about what happens without one:

cpp
// src/Common/MemoryTracker.cpp:298
        if (value && !DB::TraceSender::isCollecting())
        {
            static std::once_flag warned;
            std::call_once(warned, []
            {
                LOG_WARNING(
                    getLogger("MemoryTracker"),
                    "min_allocation_size_to_log_stack_trace is ignored because the trace collector is not "
                    "running (it requires `trace_log` to be configured).");
            });
            value = 0;
        }

And the trace itself is budgeted, because unwinding a stack on every large allocation is not free:

cpp
// src/Common/MemoryTracker.cpp:156
/// Symbolizing in TraceCollector is expensive and the trigger repeats, so traces are budgeted.
static constexpr UInt64 max_large_allocations_traced = 10;

Ten traces per TraceCollector lifetime. It is enough to find the culprit and not enough to melt the server.

The setting’s own documentation describes the failure mode it exists for, and it is nastier than “the server is using too much memory”:

cpp
// src/Core/ServerSettings.cpp:447
    DECLARE(UInt64, min_allocation_size_to_log_stack_trace, 0, R"(
Minimum size, in bytes, of a single charge to the global (server-wide) memory tracker for which a stack trace is captured, written to the server log at `Warning` level and inserted into [`system.trace_log`](/operations/system-tables/trace_log) with trace type `MemoryLargeAllocation`.

The size compared against the threshold, and reported, is what one tracker call charges, which is not necessarily one allocation: a thread defers its small allocations and flushes them as a single charge, so the reported size can exceed the allocation at the top of the reported stack by up to `max_untracked_memory`. Keeping the threshold well above `max_untracked_memory` keeps that difference immaterial.

This is a diagnostic for a global tracked total that has grown far beyond the process's real memory usage. In that state the server refuses every allocation, down to zero-byte ones, while using a fraction of its limit, and ordinary telemetry cannot attribute the step: allocations charged under a `MemoryTrackerBlockerInThread` skip the limit check and the traces that accompany it, and the `system.trace_log` inserts that would carry the rest fail once the server is wedged. The server log keeps being written, so the stack trace reaches it.

The key sentence: “the server refuses every allocation, down to zero-byte ones, while using a fraction of its limit”. A server whose tracked total has drifted far above its real RSS will refuse every allocation while sitting at 10% of its configured limit. That is a bug in the accounting, not a memory problem, and no amount of raising the limit helps — the limit is already enforced, it is just enforced against a number that is wrong.