Query-Execution on Opensource internals

ClickHouse's Query Pipeline: Ports, Status Codes and the Executing Graph

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse doesn’t call operators in a fixed order. Every processor reports a status — “I need data”, “my output is full”, “call my work()” — and the executor walks the graph, re-preparing only the neighbours that actually changed.

Every query engine eventually has to answer the same question: given a graph of operators — read these files, filter these rows, group them, sort them, write them out — in what order do you actually run them?

The textbook answer is the Volcano iterator model. Every operator exposes a next() that returns one tuple at a time, and the executor is a nest of while (child.next()) loops. It’s beautifully simple, and it’s also unbelievably slow: every tuple crosses every operator through a virtual call, and nobody ever gets to see a whole batch.

ClickHouse’s answer is the opposite extreme. Operators exchange blocks — tens of thousands of rows at a time — through ports, and the executor never calls operators in a fixed order. Instead every processor is asked, in effect, “given the state of your neighbours, what should happen next?”, and it answers with one of five status codes.

The executor is a scheduler that walks the graph, and the graph tells it where to go.

This is a tour of that machinery: the IProcessor interface, the one-slot mailbox that is a Port, the five status codes, the version counters that let a processor notice that a neighbour changed, and the ExecutingGraph that ties them together.

The building block: `IProcessor`

A processor is a node with zero or more input ports and zero or more output ports. The taxonomy in the header comment of src/Processors/IProcessor.h still describes every processor in the tree — sources, sinks, simple transformations, squashing transformations (DISTINCT, WHERE), accumulating transformations (ORDER BY, GROUP BY), and limiting transformations.

The crucial design decision is that there is no next(). A processor never asks its input for data; it declares whether it needs data, and the executor arranges for it to arrive. And a processor never decides what runs next — it returns a status, and the executor interprets it.

Here’s the entire status vocabulary:

 cpp
 
 
// src/Processors/IProcessor.h:150
 enum class Status : uint8_t
 {
 /// Processor needs some data at its inputs to proceed.
 /// You need to run another processor to generate required input and then call 'prepare' again.
 NeedData,

 /// Processor cannot proceed because output port is full or not isNeeded().
 /// You need to transfer data from output port to the input port of another processor and then call 'prepare' again.
 PortFull,

 /// All work is done (all data is processed or all output are closed), nothing more to do.
 Finished,

 /// You may call 'work' method and processor will do some work synchronously.
 Ready,

 /// You may call 'schedule' method and processor will return a descriptor.
 /// You need to poll this descriptor and call work() afterwards.
 Async,

 /// Processor wants to add new processors and/or remove finished neighbours.
 /// Update must be obtained by updatePipeline() call.
 UpdatePipeline,
 };

NeedData and PortFull are the two “I’m blocked” answers, and they’re the whole point. They tell the executor which neighbour has to run next: NeedData means “run my upstream”, PortFull means “run my downstream”. Ready means “I have work, call work() on me”. Async means “I have a file descriptor, poll it”. UpdatePipeline means “I changed the graph”.

The contract for prepare() is spelled out carefully, because it’s where most of the subtlety lives:

 cpp
 
 
// src/Processors/IProcessor.h:177
 /** Method 'prepare' is responsible for all cheap ("instantaneous": O(1) of data volume, no wait) calculations.
 *
 * It may access input and output ports,
 * indicate the need for work by another processor by returning NeedData or PortFull,
 * or indicate that processing has finished by returning `Finished`,
 * it may pull data from input ports and push data to output ports.
 *
 * The method is not thread-safe and must be called from a single thread in one moment of time,
 * even for different connected processors.
 *
 * Instead of all long work (CPU calculations or waiting) it should just prepare all required data and return Ready or Async.
 *
 * Thread safety and parallel execution:
 * - no methods (prepare, work, schedule) of single object can be executed in parallel;
 * - method 'work' can be executed in parallel for different objects, even for connected processors;
 * - method 'prepare' cannot be executed in parallel even for different objects,
 * if they are connected (including indirectly) to each other by their ports;
 */
 virtual Status prepare();

Read that last bullet twice. It’s the fundamental law of the model: work() may run in parallel with anything; prepare() may not run in parallel with anything it’s connected to.

The port connections define the happens-before edges of the whole execution, and the executor honours them by construction.

The port: a mailbox with the flags in the pointer

A port is where two processors meet. It’s shared state, and it must be readable and writable from two different threads (the producer and the consumer), so it’s an atomic pointer to a Data slot — plus three flag bits that say what the slot currently holds.

 cpp
 
 
// src/Processors/Port.h:75
 /// Flags for Port state.
 /// Will store them in least pointer bits.

 /// Port was set finished or closed.
 static constexpr std::uintptr_t IS_FINISHED = 1;
 /// Block is not needed right now, but may be will be needed later.
 /// This allows to pause calculations if we are not sure that we need more data.
 static constexpr std::uintptr_t IS_NEEDED = 2;
 /// Check if port has data.
 static constexpr std::uintptr_t HAS_DATA = 4;

 static constexpr std::uintptr_t FLAGS_MASK = IS_FINISHED | IS_NEEDED | HAS_DATA;
 static constexpr std::uintptr_t PTR_MASK = ~FLAGS_MASK;

Packing the flags into the low bits of the pointer itself is the trick. It means the entire port state — the slot pointer plus the three flags — is a single atomic word. So a push and a “set not needed” can be one CAS operation, and there’s never a window in which the flags disagree with the pointer.

The little DataPtr class exists to make that safe. The swap() method is the whole implementation of a push or a pull:

 cpp
 
 
// src/Processors/Port.h:117
 uintptr_t ALWAYS_INLINE swap(std::atomic<Data *> & value, std::uintptr_t flags, std::uintptr_t mask) /// NOLINT
 {
 Data * expected = nullptr;
 Data * desired = getPtr(flags | getUInt(data));

 while (!value.compare_exchange_weak(expected, desired))
 desired = getPtr((getUInt(expected) & FLAGS_MASK & (~mask)) | flags | getUInt(data));

 /// It's not very safe. In case of exception after exchange and before assignment we will get leak.
 /// Don't know how to make it better.
 data = getPtr(getUInt(expected) & PTR_MASK);

 return getUInt(expected) & FLAGS_MASK;
 }

The mask argument says which flags the caller intends to change and which ones it wants to preserve from whatever is currently in the word. The CAS loop re-computes desired from expected on every failure, so concurrent flag updates by the two sides are merged rather than lost.

push() and pull() are then thin wrappers:

 cpp
 
 
// src/Processors/Port.h:154
 void ALWAYS_INLINE push(DataPtr & data_, std::uintptr_t & flags)
 {
 flags = data_.swap(data, HAS_DATA, HAS_DATA);

 /// It's possible to push data into finished port. Will just ignore it.
 /// if (flags & IS_FINISHED)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to finished port.");

 /// It's possible to push data into port which is not needed now.
 /// if ((flags & IS_NEEDED) == 0)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which is not needed.");

 if (unlikely(flags & HAS_DATA))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which already has data");
 }

 void ALWAYS_INLINE pull(DataPtr & data_, std::uintptr_t & flags, bool set_not_needed = false)
 {
 uintptr_t mask = HAS_DATA;

 if (set_not_needed)
 mask |= IS_NEEDED;

 flags = data_.swap(data, 0, mask);

 /// It's ok to check because this flag can be changed only by pulling thread.
 if (unlikely((flags & IS_NEEDED) == 0) && !set_not_needed)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which is not needed");

 if (unlikely((flags & HAS_DATA) == 0))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which has no data");
 }

Note the semantics of IS_NEEDED. It’s not “the consumer wants data”; it’s “the consumer might want data later”.

A producer may push into a port that’s currently IS_NEEDED == 0 — the data simply sits in the slot — but it may not pull from a port that’s not needed, because pulling implies “give me the next thing” and the consumer has explicitly said it doesn’t want more right now.

canPush() is the producer’s gate:

 cpp
 
 
// src/Processors/Port.h:478
 bool ALWAYS_INLINE canPush() const
 {
 assumeConnected();
 auto flags = state->getFlags();
 return (flags & State::IS_NEEDED) && ((flags & State::HAS_DATA) == 0);
 }

Both conditions matter. If the consumer said “not needed”, the producer must wait; and if the slot already has data, the producer must wait, because the port holds exactly one chunk.

This “not needed” handshake is what makes the pull side of the model honest. The consumer says input.setNotNeeded() while it’s busy, the producer’s canPush() goes false, and the producer stops.

That’s how a LIMIT transform can stop a system.numbers_mt source from producing infinity. It closes the port.

 sql
 
 
-- The Limit transform closes its input port once the limit is reached,
-- so the source stops generating rows instead of the query reading forever.
SELECT * FROM system.numbers_mt WHERE number = 1000000 LIMIT 1;

The shape of the resulting graph is visible with EXPLAIN PIPELINE, which prints the processors and the ports the executor will walk:

 sql
 
 
EXPLAIN PIPELINE
SELECT number FROM numbers(10) WHERE number > 5 LIMIT 2;

`prepare()` and `work()`

The canonical implementation of the two-phase protocol is ISimpleTransform, the base class of nearly every operator in the tree (filters, expressions, DISTINCT, window functions, LIMIT BY…).

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:23
ISimpleTransform::Status ISimpleTransform::prepare()
{
 /// Check can output.

 if (output.isFinished())
 {
 input.close();
 onFinish();
 return Status::Finished;
 }

 if (!output.canPush())
 {
 input.setNotNeeded();
 return Status::PortFull;
 }

 /// Output if has data.
 if (has_output)
 {
 output.pushData(std::move(output_data));
 has_output = false;

 if (!no_more_data_needed)
 return Status::PortFull;

 }

 /// Stop if don't need more data.
 if (no_more_data_needed)
 {
 input.close();
 output.finish();
 return Status::Finished;
 }

 /// Check can input.
 if (!has_input)
 {
 if (input.isFinished())
 {
 output.finish();
 onFinish();
 return Status::Finished;
 }

 input.setNeeded();

 if (!input.hasData())
 return Status::NeedData;

 input_data = input.pullData(set_input_not_needed_after_read);
 has_input = true;

 if (input_data.exception)
 /// No more data needed. Exception will be thrown (or swallowed) later.
 input.setNotNeeded();
 }

 /// Now transform.
 return Status::Ready;
}

Every branch here is a decision about who runs next:

 text
 
 
 output.isFinished() -> input.close(); output.finish() -> Finished
 !output.canPush() -> input.setNotNeeded() -> PortFull
 has_output -> output.pushData() -> PortFull
 no_more_data_needed -> input.close(); output.finish() -> Finished
 input.isFinished() -> output.finish() -> Finished
 !input.hasData() -> input.setNeeded() -> NeedData
 otherwise -> pullData(); transform() -> Ready

Notice that prepare() does push: output.pushData(...) and input.pullData(...) both happen inside prepare(). The division of labour isn’t “prepare moves no data”, it’s “prepare does no expensive work”.

Pulling a chunk is O(1) — it’s a pointer swap. Actually computing on it is not, and that’s what work() is for:

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:86
void ISimpleTransform::work()
{
 if (input_data.exception)
 {
 transform(input_data.exception);

 output_data = std::move(input_data);
 has_input = false;
 has_output = true;
 return;
 }

 try
 {
 transform(input_data.chunk, output_data.chunk);
 }
 catch (DB::Exception &)
 {
 output_data.exception = std::current_exception();
 has_output = true;
 has_input = false;
 return;
 }

 has_input = !needInputData();

 if (!skip_empty_chunks || output_data.chunk)
 has_output = true;

 if (has_output && !output_data.chunk && !getOutputPort().getHeader().empty())
 /// Support invariant that chunks must have the same number of columns as header.
 output_data.chunk = Chunk(getOutputPort().getHeader().cloneEmpty().getColumns(), 0);
}

Three things are worth noting.

Exceptions are data. A failure in the upstream processor is pushed into the port as an exception_ptr alongside the chunk, not thrown across the thread boundary. work() picks it up, hands it to transform(), and pushes it on. The exception then travels downstream with the stream, which means the error surfaces in the right place and every processor upstream of the failure gets to clean up.

work() cannot block. That’s why the Status::Async case exists: a processor that needs to wait for a socket or a file descriptor returns Async from prepare() and hands out an fd from schedule().

The empty-chunk invariant. Every chunk that crosses a port must have exactly as many columns as the port’s header, even if it has zero rows. ISimpleTransform enforces it at the bottom of work() by synthesising an empty chunk with the right shape.

Version counters: how a processor learns that a neighbour changed

The executor must know which processors to wake up. Polling every processor in the graph after every chunk would be O(V) per chunk, which for a twenty-node graph and a million chunks is a lot of pointless work.

ClickHouse’s answer is a version counter per connection, with a twist: the counter lives in the UpdateInfo of the port, and it’s compared against the value the neighbour last saw.

 cpp
 
 
// src/Processors/Port.h:34
 struct UpdateInfo
 {
 using UpdateList = VectorWithMemoryTracking<void *>;

 UpdateList * update_list = nullptr;
 void * id = nullptr;
 UInt64 version = 0;
 UInt64 prev_version = 0;

 void inline ALWAYS_INLINE update()
 {
 if (version == prev_version && update_list)
 update_list->push_back(id);

 ++version;
 }

 void inline ALWAYS_INLINE trigger() { prev_version = version; }
 };

update() is called whenever the port’s state changes (a push, a pull, a setNeeded). It bumps version, and — this is the interesting part — if version == prev_version, meaning nobody has consumed the change yet, it appends the port’s identity to a list of “ports that changed”. trigger() marks the change as seen by setting prev_version = version.

So the protocol is: a port that changes while nobody was looking records itself on a list; the executor drains that list and re-prepares exactly those neighbours. If a port changes twice before the neighbour is re-prepared, the neighbour is only told once. It doesn’t need to know how many times it changed, only that it did.

This is the mechanism that replaced the old “wake up the neighbour immediately” design, and it’s why prepare() can be called on a subset of the graph instead of on all of it.

The `ExecutingGraph`: nodes, edges, and who owns what

The executor’s view of the pipeline is ExecutingGraph, in src/Processors/Executors/Runtime/V1/. It holds one Node per processor and two kinds of Edge per connection: a direct edge from an output port to the consuming processor’s input port, and a backward edge from an input port back to the producing processor’s output port.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:31
 /// Edge represents connection between OutputPort and InputPort.
 /// For every connection, two edges are created: direct and backward (it is specified by backward flag).
 struct Edge
 {
 Edge(Node * to_, bool backward_,
 InputPort * input_port_, OutputPort * output_port_,
 VectorWithMemoryTracking<void *> * update_list)
 : to(to_), backward(backward_)
 , input_port(input_port_), output_port(output_port_)
 {
 update_info.update_list = update_list;
 update_info.id = this;
 }

 /// Node this edge points to.
 /// It is the node with output_port for direct edge or the node with input_port for backward edge.
 Node * to = nullptr;
 bool backward;
 /// Ports of this connection. Both pointers are set regardless of direction:
 /// for a direct edge (out -> in) input_port belongs to `to`; for a back edge the reverse.
 InputPort * input_port = nullptr;
 OutputPort * output_port = nullptr;

 /// Edge version is increased when port's state is changed (e.g. when data is pushed). See Port.h for details.
 /// To compare version with prev_version we can decide if neighbour processor need to be prepared.
 Port::UpdateInfo update_info;
 };

The Node carries a status with a very specific ownership protocol:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:62
 /// Status for processor.
 /// Can be owning or not. Owning means that executor who set this status can change node's data and nobody else can.
 enum class ExecStatus : uint8_t
 {
 Idle, /// prepare returned NeedData or PortFull. Non-owning.
 Preparing, /// some executor is preparing processor, or processor is in task_queue. Owning.
 Executing, /// prepare returned Ready or Async and task is executing. Owning.
 Finished, /// prepare returned Finished. Non-owning.
 };

“Owning” is the key word. Two threads may never call prepare() on the same processor, and may never call prepare() on connected processors at the same time. The status is how the graph enforces that: a node in Preparing or Executing is owned by exactly one thread, and the per-node std::mutex status_mutex is taken when ownership is transferred.

The traversal itself is updateNode(). It starts from one processor and spreads along the edges, re-preparing whoever was affected:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:524
ExecutingGraph::UpdateNodeStatus ExecutingGraph::updateNode(IProcessor & initial, Queue & queue, Queue & async_queue, UpdateNodeBuffers & buffers)
{
 Processors delayed_destruction;
 auto & updated_edges = buffers.updated_edges;
 auto & updated_processors = buffers.updated_processors;
 auto & pending_expansion = buffers.pending_expansion;

 /// The buffers may be left non-empty by a previous call that exited early or threw.
 buffers.reset();

 std::shared_lock read_lock(nodes_mutex);

 /// Traversal starts from the node related to updated processor
 updated_processors.push_back(processors_map.at(&initial));

 while (!updated_processors.empty() || !updated_edges.empty())
 {
 std::optional<std::unique_lock<std::mutex>> stack_top_lock;

 if (updated_processors.empty())
 {
 auto * edge = updated_edges.front();
 updated_edges.pop_front();

 /// Here we have ownership on edge, but node can be concurrently accessed.

 auto & node = *edge->to;

 std::unique_lock lock(node.status_mutex);

 ExecutingGraph::ExecStatus status = node.status;

 if (status != ExecutingGraph::ExecStatus::Finished)
 {
 if (edge->backward)
 node.updated_output_ports.push_back(edge->output_port);
 else
 node.updated_input_ports.push_back(edge->input_port);

 if (status == ExecutingGraph::ExecStatus::Idle)
 {
 node.status = ExecutingGraph::ExecStatus::Preparing;
 updated_processors.push_front(edge->to);
 stack_top_lock = std::move(lock);
 }
 else
 edge->to->processor()->onUpdatePorts();
 }
 }

Two work lists are at work here, and the distinction is worth spelling out. updated_processors is a queue of nodes to call prepare() on. updated_edges is a queue of connections whose port state changed and whose far side therefore needs to know.

The loop interleaves them. An edge whose far side is Idle promotes that node to Preparing and puts it on the processor queue. If the far side is already Preparing or Executing, the edge is simply recorded in that node’s updated_input_ports / updated_output_ports list, and the node will pick it up on its next prepare().

That’s the “which processor am I at” trick in its modern form: there is no cursor. There’s no “current processor” that remembers where it was. Instead, each processor holds the set of ports that changed since it was last prepared, and the graph walks edges until it reaches a fixed point.

The core of the traversal is the prepare() call and the dispatch on its result:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:621
 switch (*node.last_processor_status)
 {
 case IProcessor::Status::NeedData:
 case IProcessor::Status::PortFull:
 {
 node.status = ExecutingGraph::ExecStatus::Idle;
 if (spill_state.memory_reservation)
 {
 /// A request may have arrived while this node was owned by normal execution.
 for (auto * spillable : processor.getSpillables())
 if (spill_state.scheduleSpill(node, spillable, queue))
 break;
 }
 break;
 }
 case IProcessor::Status::Finished:
 {
 node.status = ExecutingGraph::ExecStatus::Finished;
 if (last_status != IProcessor::Status::Finished)
 spill_state.unregisterSpillables(node);
 accountFinishedProcessorInGroup(*node.processor_iter);
 break;
 }
 case IProcessor::Status::Ready:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 queue.push(node.processor());
 break;
 }
 case IProcessor::Status::Async:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 async_queue.push(node.processor());
 break;
 }
 case IProcessor::Status::UpdatePipeline:
 {
 need_update_pipeline = true;
 pending_expansion.push_back(current);
 break;
 }
 }

Ready puts the processor on the work queue, to be picked up by a worker thread. NeedData/PortFull put the node back to Idle, meaning “somebody else must move first”. Finished retires the node for good. And then the edges that this processor disturbed get re-armed:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:666
 /// If you wonder why edges are pushed in reverse order,
 /// it is because updated_edges is a stack, and we prefer to get from stack
 /// input ports firstly, and then outputs, both in-order.
 ///
 /// Actually, there should be no difference in which order we process edges.
 /// However, some tests are sensitive to it (e.g. something like SELECT 1 UNION ALL 2).
 /// Let's not break this behaviour so far.

 for (auto it = node.post_updated_output_ports.rbegin(); it != node.post_updated_output_ports.rend(); ++it)
 {
 auto * edge = static_cast<ExecutingGraph::Edge *>(*it);
 updated_edges.push_front(edge);
 edge->update_info.trigger();
 }

edge->update_info.trigger() is the prev_version = version from Port.h, and it’s called after the edge is queued. The ordering matters: trigger() says “the neighbour has now seen this change”, and the queue entry is what makes it true.

The graph can change shape at runtime

The last status, UpdatePipeline, is unusual: a processor can, while running, decide that it needs more processors.

The classic case is external sort. MergeSortingTransform starts as a single-input, single-output processor, and when it decides to spill to disk it must insert a BufferingToFileSink and a BufferingFromFileSource into the middle of its own pipeline.

The mechanism is updatePipeline(), which returns a PipelineUpdate:

 cpp
 
 
// src/Processors/IProcessor.h:262
 struct PipelineUpdate
 {
 Processors to_add;
 Processors to_remove;
 };
 virtual PipelineUpdate updatePipeline();

The graph handles the insertion by rebuilding the edges of every node and scheduling the affected ones for re-preparation. The interesting part is the error handling, which encodes a hard-won lesson about allocation failures:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:324
 try
 {
 return updatePipelineImpl(stack, cur_node, update);
 }
 catch (...)
 {
 expansion_failed = true;
 /// The ports of the pipeline's processors point at the new processors, and the pipeline is read after the graph is gone.
 std::lock_guard guard(processors_mutex);
 for (auto it = update.to_add.begin(); it != update.to_add.end();)
 {
 auto next = std::next(it);
 if (std::ranges::find(*processors, *it) == processors->end())
 processors->splice(processors->end(), update.to_add, it);
 it = next;
 }
 throw;
 }

The processor has already connected its ports to the new processors before updatePipeline() is even called. So if the graph then fails to record them — say, addNode throws MEMORY_LIMIT_EXCEEDED — the ports point at processors the graph doesn’t know about, and the graph is permanently inconsistent.

The handler keeps the new processors alive (so the ports stay valid) and sets expansion_failed, which makes every later expansion bail out instead of walking into the inconsistency.

Trade-offs

What this buys you.

The model is fully data-parallel by construction. Any node of the graph may be scheduled on any thread, because the only shared state between processors is a port, and a port is a single atomic word. There’s no notion of “this operator belongs to this thread”, which is what lets ClickHouse run a SELECT over a MergeTree with 16 streams on 16 cores without any of the 16 streams knowing about the other 15.

The graph also reshapes itself. External sort, GLOBAL JOIN, subquery materialisation, and LIMIT pushdown all insert or remove processors while the query is running, and none of them require a separate “compile the plan” phase.

Finally, the split between prepare() (cheap, may not run concurrently with connected peers) and work() (expensive, may run concurrently with everything) is precisely the split that lets the executor overlap computation without any locking on the hot path.

What it costs.

prepare() cannot run in parallel with connected processors. For a wide graph — say a UNION ALL of twenty subqueries, each with a dozen operators — this means the scheduling phase is effectively serialised per connection. That’s why the pipeline shape matters so much: every extra operator in a chain adds a serialisation point between the parallel parts.

It’s also why Pipe::resize() has a comment about splitting the resize processor when multiple threads are allocated:

 cpp
 
 
// src/QueryPipeline/Pipe.cpp:781
 /// Performance bottleneck identified: Severe lock contention for ExecutingGraph::Node::status_mutex.
 /// Issues observed:
 /// 1. Increased latency of ExecutingGraph::updateNode, lengthening overall query latency.
 /// 2. Unnecessary CPU cycle consumption in native_queued_spin_lock_slowpath (kernel).
 /// 3. Decreased CPU utilization.
 ///
 /// Proposed solution: Split ResizeProcessor when multiple threads are allocated to execute a query.

The second cost is the per-node status_mutex. Every edge traversal takes it. On a hot path with many small chunks this is measurable, and the comment above is the record of someone measuring it.

The third cost is that the model is hard to reason about in the presence of prepare() side effects. ISimpleTransform::prepare() pushes data. LimitTransform::prepare() closes ports. SortingTransform::prepare() can return UpdatePipeline and rebuild the graph. There’s no single “execute this operator” entry point — which is exactly why ClickHouse’s test infrastructure is built around assembling a whole Pipe.

Wrapping up

The real win: ClickHouse’s pipeline doesn’t have a “current operator”. It has a set of statuses and a graph that knows how to resolve them. That single design choice buys data parallelism for free, because the only thing two processors share is an atomic word.

Two things to internalize:

If you’re debugging a stuck ClickHouse query, look at the graph, not the SQL. EXPLAIN PIPELINE shows you the processors and the ports; the status codes tell you who’s blocked and on whom.
The model is deliberately push-driven at the ports. IS_NEEDED and HAS_DATA are the only two things that matter, and every producer must respect them or you get “Cannot push block to port which already has data”.

So next time you add a custom operator, remember: your prepare() must be O(1), your work() must not block, and your exceptions must travel as data — not as thrown errors.

ClickHouse Runs a Query Pipeline on N Threads: Task Queues, Work Stealing and a Pool That Grows on Demand

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse doesn’t hand you max_threads threads. It starts with one, measures how much parallel work the pipeline actually creates, and grows the pool only when idle threads genuinely can’t absorb it — then shrinks again if the server preempts you.

A query plan in ClickHouse is not one pipeline. It’s a stack of pipelines: read from storage, filter, project, aggregate, sort, format. Each stage produces many parallel streams, and consecutive stages are connected by ResizeProcessors that shuffle chunks between them.

Running that means answering three questions at once:

Which processor should run next? (the graph — see the previous article on ExecutingGraph)
Which thread should run it?
How many threads should exist at all?

The third question is the interesting one. A query that reads one small table does not want sixteen threads. A query that scans a hundred partitions does. And a query that has been running for thirty seconds may need fewer threads than it started with, because the server is under load from other queries.

ClickHouse’s answer, in src/Processors/Executors/Runtime/V1/, is a system that starts with one thread, measures demand as the pipeline runs, and grows the pool only when there is genuinely parallel work that idle threads cannot absorb.

Two kinds of parallelism

Before the machinery, let’s be precise about what “parallel” means here, because ClickHouse does two different things and they get conflated a lot.

Data parallelism is the obvious one: SELECT count() FROM t reads from N partitions, each partition is its own stream, and N streams are processed at once. This is a property of the plan — the number of streams is fixed when the plan is built.

Pipeline parallelism is the subtler one. Even in a pipeline with a single stream, the stage reading from disk isn’t blocked while the stage formatting the output is busy. A chunk is handed from one processor to the next, and while the consumer works on chunk k, the producer is already building chunk k+1.

The executor supports both, and the interesting bit is how it decides how many threads to use:

 sql
 
 
-- Sixteen streams of data parallelism; the number of threads that actually
-- run them is decided at execution time, not by the plan.
SELECT count() FROM numbers_mt;

 sql
 
 
-- The ceiling. The pool starts at one thread and grows only if there is
-- parallel work that idle threads cannot absorb.
SETTINGS max_threads = 16;
SELECT count() FROM numbers_mt;

Note that the two are not the same thing. A pipeline with 16 streams needs 16 threads to be fully parallel, but it will still make progress — just less of it — on one thread.

The shape of the executor

PipelineExecutor owns three things:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.h:49
private:
 /// Outlives graph registrations, including when construction fails.
 QueryStatusPtr process_list_element;
 ExecutingGraphPtr graph;

 ExecutorTasks tasks;

 /// Concurrency control related
 SlotAllocationPtr cpu_slots;
 AcquiredSlotPtr single_thread_cpu_slot; // cpu slot for single-thread mode to work using executeUntil()
 std::unique_ptr<ThreadPool> pool;
 std::mutex spawn_mutex;

graph is the processor graph (the subject of the previous article). tasks is the scheduler: the queues, the per-thread contexts, the counters. cpu_slots and pool are the resources: how many threads are allowed to exist, and the threads themselves.

The entry point is execute(), which is unremarkable apart from its comment about output formats:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:231
void PipelineExecutor::execute(size_t num_threads, bool concurrency_control)
{
 /// `QueryStatus` may release its reservation before this executor is destroyed.
 SCOPE_EXIT({ detachSpillables(); });
 if (process_list_element && !process_list_element->checkTimeLimit())
 cancel(IProcessor::CancelReason::CancelledByTimeout);

 num_threads = std::max<size_t>(num_threads, 1);

and executeImpl() is where the threads actually appear:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:783
void PipelineExecutor::executeImpl(size_t num_threads, bool concurrency_control)
{
 initializeExecution(num_threads, concurrency_control);

 try
 {
 if (num_threads > 1)
 {
 {
 std::lock_guard lock(spawn_mutex);
 // Start at least one thread, could block to acquire the first CPU slot
 spawnThreads(cpu_slots->acquire());
 }
 tasks.processAsyncTasks();
 pool->wait();
 }
 else
 {
 auto slot = cpu_slots->acquire();
 tasks.upscale(slot->slot_id);
 executeSingleThread(slot->slot_id, WorkloadResources(slot.get(), process_list_element));
 }
 }

Notice pool->wait(). The master thread does not execute pipeline work in the multi-threaded case. It spawns worker threads and then parks in processAsyncTasks(), which is the epoll loop for processors that returned Status::Async.

The master thread is a scheduler, not a worker — and that distinction matters for deadlock avoidance, which is why the master is never downscaled:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:591
 // The first thread is called master thread.
 // It is NOT the thread that handles async tasks (unless query has max_threads=1).
 // Master thread is different from other threads due to special role in scheduling:
 // 1. During query start, master thread spawns worker threads, so starting it fast is important.
 // 2. It should never be downscaled to avoid query deadlock (0 threads to process tasks), although it can be preempted.
 // 3. When using ConcurrencyControl, master thread is granted immediately and may not request CPU slot (see 'fair_round_robin').
 // 4. When using resource scheduler, master thread may use different resource link than worker threads.
 const auto master_threads = 1uz;

Task queues with work stealing

The scheduler’s central data structure is TaskQueue, and it’s per-thread-first:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/TasksQueue.h:17
/// A set of task queues for multithreaded processing.
/// Every threads has its dedicated queue and uses it preferably.
/// When there are no tasks left in dedicated queue it steals tasks from other threads
template <typename Task>
class TaskQueue
{
public:
 void init(size_t num_threads) { queues.resize(num_threads); }

 /// Push a task into thread's dedicated queue
 void push(Task * task, size_t thread_num)
 {
 queues[thread_num].push(task);
 use_queues = std::max(use_queues, thread_num + 1);
 ++num_tasks;
 }

 /// Returns thread number to pop task from.
 /// First it check dedicated queue, and only if it is empty, it steal from other threads
 size_t getAnyThreadWithTasks(size_t from_thread)
 {
 if (num_tasks == 0)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");

 if (from_thread >= use_queues)
 from_thread = 0;

 for (size_t i = 0, max_iterations = use_queues; i < max_iterations; ++i)
 {
 if (!queues[from_thread].empty())
 return from_thread;

 ++from_thread;
 if (from_thread == use_queues)
 {
 use_queues--; // lazy update: less expensive than updating on every pop
 from_thread = 0;
 }
 }

 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");
 }

Each thread has a private FIFO. A thread with work prefers its own queue; only when it’s empty does it walk the other threads’ queues looking for something to steal.

use_queues is a lazily-maintained upper bound on how many queues are non-empty, so the scan is O(number of active threads) and not O(total threads). That “lazy update” comment is doing real work — updating the bound on every pop would cost more than it saves.

The per-thread affinity matters for cache reasons: a task that runs on thread 3 tends to produce data that thread 3 will consume next, and the data stays in that core’s L1/L2.

There are three queues, not one:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.h:43
 /// Queue with pointers to tasks. Each thread will concurrently read from it until finished flag is set.
 /// Stores processors need to be prepared. Preparing status is already set for them.
 TaskQueue<IProcessor> task_queue;

 /// Async tasks should be processed with higher priority, but also require task stealing logic.
 /// So we have a separate queue specifically for them.
 TaskQueue<IProcessor> fast_task_queue;
 std::atomic_bool has_fast_tasks = false; // Required only to enable local task optimization
 std::atomic_bool spill_requested = false;

 /// Queue which stores tasks where processors returned Async status after prepare.
 /// If multiple threads are used, main thread will wait for async tasks.
 /// For single thread, will wait for async tasks only when task_queue is empty.
 PollingQueue async_task_queue;

fast_task_queue holds processors that came back from an asynchronous operation — a socket that became readable, a file that finished being written. They get priority because they’re latency-sensitive: the data is already there and the client is waiting.

async_task_queue is different in kind. It’s not a task queue at all, it’s an epoll wrapper that owns the file descriptors.

The local-task optimization

Here’s the single most important performance trick in the executor, and it’s small.

When a processor finishes, the next processor to run is often known immediately — it’s the downstream consumer of the chunk that was just produced. Rather than pushing it onto a shared queue (where another thread would have to wake up, take the mutex, and pop it), the executor hands it directly to the thread that’s already running.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:192
size_t ExecutorTasks::pushTasks(Queue & queue, Queue & async_queue, ExecutionThreadContext & context)
{
 /// Take local task from queue if has one.
 if (!queue.empty() && !has_fast_tasks.load(std::memory_order_relaxed)
 && context.num_scheduled_local_tasks < ExecutionThreadContext::max_scheduled_local_tasks)
 {
 ++context.num_scheduled_local_tasks;
 context.setTask(queue.front());
 queue.pop();
 }
 else
 {
 context.num_scheduled_local_tasks = 0;
 context.setTask(nullptr);
 }

and the constant that bounds it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutionThreadContext.h:55
 /// There is a performance optimization that schedules a task to the current thread, avoiding global task queue.
 /// Optimization decreases contention on global task queue but may cause starvation.
 /// See 01104_distributed_numbers_test.sql
 /// This constant tells us that we should skip the optimization
 /// if it was applied more than `max_scheduled_local_tasks` in a row.
 constexpr static size_t max_scheduled_local_tasks = 128;

128 in a row. The comment names the failure mode: starvation. If one thread keeps grabbing work locally while other threads sit idle with nothing to do, the pipeline runs on one core while the other max_threads - 1 cores idle.

The counter resets as soon as the local optimization is skipped, so a thread that has been running alone for 128 tasks will put the next one on the global queue and let somebody else take it.

The referenced test, 01104_distributed_numbers_test.sql, is a good illustration of why this matters: SELECT * FROM numbers(1000000000) with distributed_product_mode and a LIMIT — the limit transform stops the source, and if the source isn’t rebalanced the other threads never wake up.

The has_fast_tasks check in the condition is the second half of the guard: if there’s an asynchronous task waiting, it takes priority over a local task, because asynchronous tasks are latency-critical.

Measuring demand: `pushTasks` returns a number

Now the interesting part. pushTasks doesn’t just enqueue work — it measures how much parallel work was created, and returns it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:208
 if (!queue.empty() || !async_queue.empty())
 {
 std::unique_lock lock(mutex);

 /// Measure the parallelism of the push: how many threads could run the new tasks
 /// concurrently. Used by `PipelineExecutor` to size the slot-allocation ceiling via
 /// `ISlotAllocation::setMax` on the pipeline's `cpu_slots` handle.
 const size_t pushed = queue.size() + async_queue.size();

and then, at the end:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:232
 /// Capture state under the lock — tryWakeUp may unlock it internally to wake a
 /// thread, after which `total_slots` could change.
 const size_t idle_before = threads_queue.size();
 const size_t total_slots_now = total_slots;
 const size_t num_threads_now = num_threads; // const after init but read for clarity

 /// Wake up at least one thread that will wake up other threads if required
 tryWakeUpAnyOtherThreadWithTasks(context, lock);

 /// Return the number of additional pipeline threads the caller should target: pushed
 /// tasks that idle threads can't cover, clamped to remaining pipeline-width headroom
 /// (`num_threads - total_slots`). `PipelineExecutor` adds this delta to the
 /// `cpu_slots->setMax(...)` target so the slot allocator grows in step with demand.
 const size_t coverable = std::min(pushed, idle_before);
 const size_t uncovered = pushed - coverable;
 const size_t headroom = (num_threads_now > total_slots_now) ? (num_threads_now - total_slots_now) : 0;
 return std::min(uncovered, headroom);
 }
 return 0; // No new tasks -- no need for new threads

The arithmetic is: pushed tasks were created. Idle threads can absorb that many without being woken up. The remainder is genuine demand for new threads — clamped to the pipeline’s own width ceiling (num_threads, from max_threads), because the plan has a fixed number of streams and there’s no point running more threads than there is parallelism.

This is the demand signal. It’s computed from two observable facts — how many tasks were pushed, how many threads are parked — and nothing else. No profiling, no heuristics, no timers.

The executor then accumulates that demand and acts on it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:492
 /// Upscaling: `pushTasks` tells us how many additional threads would usefully
 /// parallelize the tasks it just enqueued. Publish our spawn_count into the shared
 /// `pending_demand` counter; the thread that actually holds `spawn_mutex` drains it
 /// so no demand is lost when multiple threads push concurrently.
 if (pool && spawn_count > 0)
 pending_demand.fetch_add(spawn_count, std::memory_order_relaxed);

 // Drain pending_demand under `spawn_mutex`. A thread that fails `try_lock`
 // must not exit while `pending_demand > 0` -- otherwise its add could race
 // with the owner's final `exchange(0)` and be left stranded after the owner
 // unlocks. The outer loop holds the thread until either it drains the demand
 // itself or observes the counter return to zero (meaning the owner caught it).
 while (pool && pending_demand.load(std::memory_order_acquire) > 0)
 {
 if (!spawn_mutex.try_lock())
 {
 std::this_thread::yield();
 continue;
 }
 try
 {
 std::lock_guard lock(spawn_mutex, std::adopt_lock);
 while (true)
 {
 size_t accumulated = pending_demand.exchange(0, std::memory_order_acq_rel);
 if (accumulated == 0)
 break;
 size_t target = std::min<size_t>(max_pipeline_threads, desired_threads + accumulated);
 if (target > desired_threads)
 {
 desired_threads = target;
 cpu_slots->setMax(target);
 }
 }
 spawnThreads({});
 }

This is small and rather careful concurrent code, and the comments are essentially a post-mortem.

The pending_demand counter is atomic so that threads which fail to take spawn_mutex don’t lose their demand. But a naive fetch_add would race with the owner’s exchange(0): thread A adds 1, the owner exchanges and gets 0, the owner unlocks, and thread A’s 1 sits in the counter forever.

The outer while loop is the fix. A thread that can’t get the mutex keeps spinning (yielding) until either it drains the demand itself or sees the counter back at zero, which means the owner already caught it.

The target computation is the second piece of policy: demand is accumulated into desired_threads, capped at max_pipeline_threads (the plan’s width), and applied by raising the slot allocator’s ceiling with cpu_slots->setMax(target).

spawnThreads then converts slots into actual OS threads:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:743
void PipelineExecutor::spawnThreads(AcquiredSlotPtr slot)
{
 while (cpu_slots)
 {
 if (!slot)
 slot = cpu_slots->tryAcquire();
 if (!slot)
 return;

 size_t thread_num = slot->slot_id;

 /// Count of threads in use should be updated for proper finish() condition.
 /// `upscale` returns the remaining headroom for spawning. Even when >0 we spawn
 /// one thread per iteration and re-enter `upscale` after the next `tryAcquire`.
 const size_t remaining_capacity = tasks.upscale(thread_num);

 /// Start new thread
 pool->scheduleOrThrowOnError([this, thread_num, thread_group = CurrentThread::getGroup(), my_slot = std::move(slot)]
 {
 ThreadGroupSwitcher switcher(thread_group, ThreadName::QUERY_ASYNC_EXECUTOR);

 try
 {
 executeSingleThread(thread_num, WorkloadResources(my_slot.get(), process_list_element));
 }
 catch (...)
 {
 /// In case of exception from executor itself, stop other threads.
 finish();
 tasks.getThreadContext(thread_num).setException(std::current_exception());
 }
 });

The tryAcquire is the gate. The slot allocator may refuse to hand out a slot if the server is out of CPU budget. If it refuses, no thread is spawned and the pipeline simply runs with fewer threads.

The thread count is not max_threads; it’s max_threads subject to what the scheduler is willing to give you.

CPU slots: `max_threads` is a ceiling, not a promise

The slot allocation is where the concurrency-control policy lives. There are four cases, and the comment enumerates them:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:605
 // There are a few possible modes of concurrency control.
 // 1) If we have `CREATE RESOURCE cpu (MASTER THREAD, WORKER THREAD)`:
 // all thread should be allocated through scheduler using the same resource link
 // 2) If we have `CREATE RESOURCE cpu (WORKER THREAD)`:
 // master thread uses "free" noncompeting CPU slot, the other threads are allocated through resource scheduler
 // 3) If we have two resources `CREATE RESOURCE master_cpu (MASTER THREAD)` and `CREATE RESOURCE worker_cpu (WORKER THREAD)`:
 // all thread should be allocated through scheduler using different resource links
 // 4) If we have no cpu-related resources:
 // the ConcurrencyControl class is used instead of resource scheduler

In the default case (case 4) the allocation is deliberately lazy:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:672
 else
 {
 /// Allocate CPU slots from concurrency control with guaranteed master thread slot.
 /// Default path (lazy_allocation=true): start at max=master_threads and grow on
 /// demand via setMax from the upscaling block. Zero-waste per idle pipeline.
 /// Rollback (lazy_allocation=false): pre-branch eager allocation — request the
 /// full ceiling upfront and never call setMax.
 SlotCount initial_max = lazy_allocation ? master_threads : num_threads;
 return ConcurrencyControl::instance().allocate(master_threads, initial_max);
 }

“Zero-waste per idle pipeline” is the point. A server running a thousand concurrent queries, each of which is a small scan, doesn’t want a thousand times max_threads slots reserved. It wants one slot each, growing if the query turns out to be wide.

The lazy flag is read exactly once, and the reason is a real bug:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:696
 /// Read the flag ONCE. A config reload mid-initialization would otherwise split a
 /// single query between lazy and eager strategies (e.g. allocate lazy with max=1 but
 /// then initialize desired_threads=num_threads, stranding subsequent setMax as a no-op).
 const bool lazy_allocation = ConcurrencyControl::instance().getLazyAllocation();

A config reload between “allocate the slots” and “set desired_threads” would leave the query in a state where setMax is a no-op and the pipeline never grows. One read, one strategy, for the lifetime of the query.

Preemption and downscaling

Threads can also be taken away. Under the resource scheduler with preemption enabled, a thread’s CPU lease is renewed periodically, and the renewal can fail:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:539
 if (resources.isCPULeaseRenewNeeded()) // Check if preemption is enabled (see `cpu_slot_preemption` server setting)
 {
 try
 {
 // Preemption point. Renewal could block execution due to CPU overload.
 // It may trigger callbacks to tasks.preempt() and tasks.resume()
 if (!resources.renewCPULease())
 {
 tasks.downscale(resources.getCPULeaseSlotId());
 break; // Downscaling. Unable to renew the lease - thread should stop (but could be rerun later).
 }

The renewal is checked at most every millisecond:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:119
 bool isCPULeaseRenewNeeded()
 {
 if (!lease)
 return false;
 UInt64 now_ns = clock_gettime_ns();
 if (now_ns - last_renew_ns < 1000000) // 1 ms: do not renew too often
 return false;
 last_renew_ns = now_ns;
 return true;
 }

Downscaling is not “kill the thread”. It’s “let this thread stop when it finishes its current task, and don’t schedule more onto it”:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:334
void ExecutorTasks::downscale(size_t slot_id)
{
 std::lock_guard lock(mutex);

 if (slot_id >= slot_count.size() || slot_count[slot_id] == 0)
 return;
 --slot_count[slot_id];

 if (slot_id + 1 == use_threads)
 {
 // The highest slot was downscaled, find new highest
 for (size_t i = use_threads; i > 0; --i)
 {
 if (slot_count[i - 1] > 0)
 {
 use_threads = i;
 break;
 }
 }
 }
}

Preemption is a third thing: the slot is released without downscaling, so the pipeline’s width is unchanged but the thread isn’t running. If the preempted thread was holding a task, that task is pushed back to the queue so another thread can steal it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:356
void ExecutorTasks::preempt(size_t slot_id)
{
 std::unique_lock lock(mutex);
 --total_slots;

 /// We should make sure that preempted thread has no local task inside context.
 /// It is allowed to have tasks in `task_queue` or `fast_task_queue` because they can be stealed by other threads.
 auto & context = executor_contexts[slot_id];
 if (auto * task = context->popTask())
 {
 task_queue.push(task, slot_id);
 /// Wake up at least one thread to avoid deadlocks (all other threads maybe idle)
 tryWakeUpAnyOtherThreadWithTasks(*context, lock); // this releases the lock if it wakes up a thread
 }

The asymmetry is the point. A local task (held in the thread’s context) is invisible to other threads and must be handed back explicitly, whereas a task in the shared queue is stealable and needs no action.

The pulling mode: `executeUntil`

Not every query is executed to completion in one go. The HTTP protocol, for instance, wants to stream rows to the client as they become available, and the client may disconnect at any moment. That’s what executeUntil() is for:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:261
bool PipelineExecutor::executeUntil(std::atomic_bool * yield_flag)
{
 const auto uncaught_exceptions = std::uncaught_exceptions();
 SCOPE_EXIT({
 if (tasks.isFinished() || std::uncaught_exceptions() > uncaught_exceptions)
 detachSpillables();
 });
 if (!is_execution_initialized)
 {
 initializeExecution(1, true);

 // Acquire slot until we are done
 single_thread_cpu_slot = cpu_slots->acquire();
 tasks.upscale(single_thread_cpu_slot->slot_id);
 chassert(single_thread_cpu_slot && "Unable to allocate cpu slot for the first thread, but we just allocated at least one slot");

 if (yield_flag && *yield_flag)
 return true;
 }

 executeStepImpl(0, WorkloadResources(single_thread_cpu_slot.get(), process_list_element), yield_flag);

 if (!tasks.isFinished())
 return true;

The yield_flag is the interesting parameter. It’s checked at the bottom of the worker loop:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:576
 /// We have executed single processor. Check if we need to yield execution.
 if (yield_flag && *yield_flag)
 break;

So the executor runs one processor at a time and then checks the flag. If the flag says “enough for now”, it returns to the caller, which can hand the chunk to the client and then call executeUntil() again. This is the “pulling” model at the level of the whole pipeline: the caller pulls.

PullingPipelineExecutor::pull() is the caller:

 cpp
 
 
// src/Processors/Executors/PullingPipelineExecutor.cpp:49
bool PullingPipelineExecutor::pull(Chunk & chunk)
{
 if (!executor)
 {
 executor = createExecutor(pipeline.processors, pipeline.process_list_element);
 executor->setReadProgressCallback(pipeline.getReadProgressCallback());
 executor->setStepProfiler(pipeline.getStepProfiler());
 }

 /// Throws when the time limit is exceeded with `timeout_overflow_mode = 'throw'`. With 'break' the partial result
 /// is returned as a success: the execution is cancelled with `CancelledByTimeout`, and `executeUntil` below
 /// finalizes it - the pending read progress is reported and the format is finalized - before the end of the
 /// data is reported.
 if (pipeline.process_list_element && !pipeline.process_list_element->checkTimeLimitSoft())
 {
 executor->cancel(IProcessor::CancelReason::CancelledByTimeout);
 pipeline.process_list_element->checkTimeLimit();
 }

 if (!executor->executeUntil(&has_data_flag))
 return false;

 chunk = pulling_format->getChunk();
 return true;
}

has_data_flag is set by the output format when it has a chunk ready. So the loop is: run the pipeline until the format has produced something (or the pipeline is finished), then hand the chunk up.

Note that executeUntil runs the executor in single-threaded mode (initializeExecution(1, true)). The pulling executor is deliberately not parallel — it exists for cases like INSERT ... SELECT into a table, or formatting a result to a client that wants to consume it incrementally.

Trade-offs

What this buys you.

The pool grows with the work, not with the setting. A query that turns out to be narrow never pays for more than one or two threads, even if max_threads = 64. A query that turns out to be wide gets threads as fast as the scheduler will give them, without any startup latency. And the measurement is exact rather than heuristic: pushTasks counts the tasks it created and subtracts the threads that could absorb them.

Work stealing with per-thread affinity gives good cache behaviour and good load balance. The common case (a thread feeding itself) is a pointer swap with no locking; the uncommon case (one thread has all the work) is handled by theft.

The separation of “what work exists” from “how many resources it gets” also means the same executor serves queries with max_threads = 1 (no contention at all, the single thread takes everything locally) and queries with max_threads = 128.

What it costs.

The machinery is intricate, and the comments are full of post-mortems: the try_lock/pending_demand race, the lazy_allocation config reload, the max_scheduled_local_tasks starvation. Each of those is a bug that shipped and was fixed. The design is sound, but the implementation is carrying the weight of a decade of production.

There’s also a hard limit on the demand signal. pushTasks measures the tasks pushed by one processor at one moment. If a pipeline is wide but each individual push produces one task, the demand signal is one at a time, and the pool grows one thread per push rather than immediately. The initial_parallel computation in initializeExecution exists precisely to patch this for the first wave:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:722
 /// Initial queued parallelism never routes through `pushTasks`, so size setMax here to
 /// cover it. For multi-source pipelines (e.g. UNION ALL of N subqueries) this prevents
 /// the pipeline from being stuck single-threaded if the master's first task happens not
 /// to push more work. Only matters on the lazy path; the eager path already has the full
 /// ceiling. `initial_parallel` is the total target — the master thread is one of its
 /// consumers, no +1 needed.
 if (lazy_allocation && initial_parallel > 1)

The preemption path is the other sharp edge. A thread that can’t renew its lease stops, and the pipeline continues with fewer threads — but if every thread is preempted at once, the pipeline must not deadlock. The threads_queue.size() + 1 == total_slots check in tryGetTask is the mechanism that detects “the last worker is about to go idle” and calls finish():

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:158
 /// This thread has no tasks to do and is going to wait.
 /// Finish execution if this was the last active thread.
 chassert(task_queue.empty() && fast_task_queue.empty());
 if (threads_queue.size() + 1 == total_slots && async_task_queue.empty())
 {
 lock.unlock();
 finish();
 return;
 }

Getting this wrong in either direction is bad: finish too early and the query returns partial data, never finish and the query hangs. The chassert documents that the two conditions must hold together — if the queues are not empty, the pipeline is not finished and the thread must keep waiting.

Finally, there’s the question of when to stop growing. desired_threads only ever goes up within a query. A pipeline that’s wide at the start and narrow at the end keeps its threads; there’s no downscaling based on demand, only downscaling due to preemption. For a long-running query over a table that becomes narrow, that’s wasted capacity.

Wrapping up

The real win: ClickHouse doesn’t guess how many threads your query needs. It measures. pushTasks returns a number — how many new tasks idle threads can’t cover — and the pool grows by exactly that much, capped by the plan’s own width.

Two things to internalize:

max_threads is a ceiling, not a schedule. If you see a query using 2 threads on a 64-core box, that’s the demand signal working, not a bug.
The server can take threads back from you. Preemption and downscaling are first-class, and your query is expected to keep going on fewer threads.

So when you profile a slow query, don’t count threads first. Count streams, and ask whether the pipeline had enough independent work to push into idle threads in the first place.

Query-Execution on Opensource internals

ClickHouse's Query Pipeline: Ports, Status Codes and the Executing Graph

The building block: IProcessor

The port: a mailbox with the flags in the pointer

prepare() and work()

Version counters: how a processor learns that a neighbour changed

The ExecutingGraph: nodes, edges, and who owns what

The graph can change shape at runtime

Trade-offs

Wrapping up

ClickHouse Runs a Query Pipeline on N Threads: Task Queues, Work Stealing and a Pool That Grows on Demand

Two kinds of parallelism

The shape of the executor

Task queues with work stealing

The local-task optimization

Measuring demand: pushTasks returns a number

CPU slots: max_threads is a ceiling, not a promise

Preemption and downscaling

The pulling mode: executeUntil

Trade-offs

Wrapping up

The building block: `IProcessor`

`prepare()` and `work()`

The `ExecutingGraph`: nodes, edges, and who owns what

Measuring demand: `pushTasks` returns a number

CPU slots: `max_threads` is a ceiling, not a promise

The pulling mode: `executeUntil`