Concurrency on Opensource internals

ClickHouse's Query Pipeline: Ports, Status Codes and the Executing Graph

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse doesn’t call operators in a fixed order. Every processor reports a status — “I need data”, “my output is full”, “call my work()” — and the executor walks the graph, re-preparing only the neighbours that actually changed.

Every query engine eventually has to answer the same question: given a graph of operators — read these files, filter these rows, group them, sort them, write them out — in what order do you actually run them?

The textbook answer is the Volcano iterator model. Every operator exposes a next() that returns one tuple at a time, and the executor is a nest of while (child.next()) loops. It’s beautifully simple, and it’s also unbelievably slow: every tuple crosses every operator through a virtual call, and nobody ever gets to see a whole batch.

ClickHouse’s answer is the opposite extreme. Operators exchange blocks — tens of thousands of rows at a time — through ports, and the executor never calls operators in a fixed order. Instead every processor is asked, in effect, “given the state of your neighbours, what should happen next?”, and it answers with one of five status codes.

The executor is a scheduler that walks the graph, and the graph tells it where to go.

This is a tour of that machinery: the IProcessor interface, the one-slot mailbox that is a Port, the five status codes, the version counters that let a processor notice that a neighbour changed, and the ExecutingGraph that ties them together.

The building block: `IProcessor`

A processor is a node with zero or more input ports and zero or more output ports. The taxonomy in the header comment of src/Processors/IProcessor.h still describes every processor in the tree — sources, sinks, simple transformations, squashing transformations (DISTINCT, WHERE), accumulating transformations (ORDER BY, GROUP BY), and limiting transformations.

The crucial design decision is that there is no next(). A processor never asks its input for data; it declares whether it needs data, and the executor arranges for it to arrive. And a processor never decides what runs next — it returns a status, and the executor interprets it.

Here’s the entire status vocabulary:

 cpp
 
 
// src/Processors/IProcessor.h:150
 enum class Status : uint8_t
 {
 /// Processor needs some data at its inputs to proceed.
 /// You need to run another processor to generate required input and then call 'prepare' again.
 NeedData,

 /// Processor cannot proceed because output port is full or not isNeeded().
 /// You need to transfer data from output port to the input port of another processor and then call 'prepare' again.
 PortFull,

 /// All work is done (all data is processed or all output are closed), nothing more to do.
 Finished,

 /// You may call 'work' method and processor will do some work synchronously.
 Ready,

 /// You may call 'schedule' method and processor will return a descriptor.
 /// You need to poll this descriptor and call work() afterwards.
 Async,

 /// Processor wants to add new processors and/or remove finished neighbours.
 /// Update must be obtained by updatePipeline() call.
 UpdatePipeline,
 };

NeedData and PortFull are the two “I’m blocked” answers, and they’re the whole point. They tell the executor which neighbour has to run next: NeedData means “run my upstream”, PortFull means “run my downstream”. Ready means “I have work, call work() on me”. Async means “I have a file descriptor, poll it”. UpdatePipeline means “I changed the graph”.

The contract for prepare() is spelled out carefully, because it’s where most of the subtlety lives:

 cpp
 
 
// src/Processors/IProcessor.h:177
 /** Method 'prepare' is responsible for all cheap ("instantaneous": O(1) of data volume, no wait) calculations.
 *
 * It may access input and output ports,
 * indicate the need for work by another processor by returning NeedData or PortFull,
 * or indicate that processing has finished by returning `Finished`,
 * it may pull data from input ports and push data to output ports.
 *
 * The method is not thread-safe and must be called from a single thread in one moment of time,
 * even for different connected processors.
 *
 * Instead of all long work (CPU calculations or waiting) it should just prepare all required data and return Ready or Async.
 *
 * Thread safety and parallel execution:
 * - no methods (prepare, work, schedule) of single object can be executed in parallel;
 * - method 'work' can be executed in parallel for different objects, even for connected processors;
 * - method 'prepare' cannot be executed in parallel even for different objects,
 * if they are connected (including indirectly) to each other by their ports;
 */
 virtual Status prepare();

Read that last bullet twice. It’s the fundamental law of the model: work() may run in parallel with anything; prepare() may not run in parallel with anything it’s connected to.

The port connections define the happens-before edges of the whole execution, and the executor honours them by construction.

The port: a mailbox with the flags in the pointer

A port is where two processors meet. It’s shared state, and it must be readable and writable from two different threads (the producer and the consumer), so it’s an atomic pointer to a Data slot — plus three flag bits that say what the slot currently holds.

 cpp
 
 
// src/Processors/Port.h:75
 /// Flags for Port state.
 /// Will store them in least pointer bits.

 /// Port was set finished or closed.
 static constexpr std::uintptr_t IS_FINISHED = 1;
 /// Block is not needed right now, but may be will be needed later.
 /// This allows to pause calculations if we are not sure that we need more data.
 static constexpr std::uintptr_t IS_NEEDED = 2;
 /// Check if port has data.
 static constexpr std::uintptr_t HAS_DATA = 4;

 static constexpr std::uintptr_t FLAGS_MASK = IS_FINISHED | IS_NEEDED | HAS_DATA;
 static constexpr std::uintptr_t PTR_MASK = ~FLAGS_MASK;

Packing the flags into the low bits of the pointer itself is the trick. It means the entire port state — the slot pointer plus the three flags — is a single atomic word. So a push and a “set not needed” can be one CAS operation, and there’s never a window in which the flags disagree with the pointer.

The little DataPtr class exists to make that safe. The swap() method is the whole implementation of a push or a pull:

 cpp
 
 
// src/Processors/Port.h:117
 uintptr_t ALWAYS_INLINE swap(std::atomic<Data *> & value, std::uintptr_t flags, std::uintptr_t mask) /// NOLINT
 {
 Data * expected = nullptr;
 Data * desired = getPtr(flags | getUInt(data));

 while (!value.compare_exchange_weak(expected, desired))
 desired = getPtr((getUInt(expected) & FLAGS_MASK & (~mask)) | flags | getUInt(data));

 /// It's not very safe. In case of exception after exchange and before assignment we will get leak.
 /// Don't know how to make it better.
 data = getPtr(getUInt(expected) & PTR_MASK);

 return getUInt(expected) & FLAGS_MASK;
 }

The mask argument says which flags the caller intends to change and which ones it wants to preserve from whatever is currently in the word. The CAS loop re-computes desired from expected on every failure, so concurrent flag updates by the two sides are merged rather than lost.

push() and pull() are then thin wrappers:

 cpp
 
 
// src/Processors/Port.h:154
 void ALWAYS_INLINE push(DataPtr & data_, std::uintptr_t & flags)
 {
 flags = data_.swap(data, HAS_DATA, HAS_DATA);

 /// It's possible to push data into finished port. Will just ignore it.
 /// if (flags & IS_FINISHED)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to finished port.");

 /// It's possible to push data into port which is not needed now.
 /// if ((flags & IS_NEEDED) == 0)
 /// throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which is not needed.");

 if (unlikely(flags & HAS_DATA))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot push block to port which already has data");
 }

 void ALWAYS_INLINE pull(DataPtr & data_, std::uintptr_t & flags, bool set_not_needed = false)
 {
 uintptr_t mask = HAS_DATA;

 if (set_not_needed)
 mask |= IS_NEEDED;

 flags = data_.swap(data, 0, mask);

 /// It's ok to check because this flag can be changed only by pulling thread.
 if (unlikely((flags & IS_NEEDED) == 0) && !set_not_needed)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which is not needed");

 if (unlikely((flags & HAS_DATA) == 0))
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot pull block from port which has no data");
 }

Note the semantics of IS_NEEDED. It’s not “the consumer wants data”; it’s “the consumer might want data later”.

A producer may push into a port that’s currently IS_NEEDED == 0 — the data simply sits in the slot — but it may not pull from a port that’s not needed, because pulling implies “give me the next thing” and the consumer has explicitly said it doesn’t want more right now.

canPush() is the producer’s gate:

 cpp
 
 
// src/Processors/Port.h:478
 bool ALWAYS_INLINE canPush() const
 {
 assumeConnected();
 auto flags = state->getFlags();
 return (flags & State::IS_NEEDED) && ((flags & State::HAS_DATA) == 0);
 }

Both conditions matter. If the consumer said “not needed”, the producer must wait; and if the slot already has data, the producer must wait, because the port holds exactly one chunk.

This “not needed” handshake is what makes the pull side of the model honest. The consumer says input.setNotNeeded() while it’s busy, the producer’s canPush() goes false, and the producer stops.

That’s how a LIMIT transform can stop a system.numbers_mt source from producing infinity. It closes the port.

 sql
 
 
-- The Limit transform closes its input port once the limit is reached,
-- so the source stops generating rows instead of the query reading forever.
SELECT * FROM system.numbers_mt WHERE number = 1000000 LIMIT 1;

The shape of the resulting graph is visible with EXPLAIN PIPELINE, which prints the processors and the ports the executor will walk:

 sql
 
 
EXPLAIN PIPELINE
SELECT number FROM numbers(10) WHERE number > 5 LIMIT 2;

`prepare()` and `work()`

The canonical implementation of the two-phase protocol is ISimpleTransform, the base class of nearly every operator in the tree (filters, expressions, DISTINCT, window functions, LIMIT BY…).

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:23
ISimpleTransform::Status ISimpleTransform::prepare()
{
 /// Check can output.

 if (output.isFinished())
 {
 input.close();
 onFinish();
 return Status::Finished;
 }

 if (!output.canPush())
 {
 input.setNotNeeded();
 return Status::PortFull;
 }

 /// Output if has data.
 if (has_output)
 {
 output.pushData(std::move(output_data));
 has_output = false;

 if (!no_more_data_needed)
 return Status::PortFull;

 }

 /// Stop if don't need more data.
 if (no_more_data_needed)
 {
 input.close();
 output.finish();
 return Status::Finished;
 }

 /// Check can input.
 if (!has_input)
 {
 if (input.isFinished())
 {
 output.finish();
 onFinish();
 return Status::Finished;
 }

 input.setNeeded();

 if (!input.hasData())
 return Status::NeedData;

 input_data = input.pullData(set_input_not_needed_after_read);
 has_input = true;

 if (input_data.exception)
 /// No more data needed. Exception will be thrown (or swallowed) later.
 input.setNotNeeded();
 }

 /// Now transform.
 return Status::Ready;
}

Every branch here is a decision about who runs next:

 text
 
 
 output.isFinished() -> input.close(); output.finish() -> Finished
 !output.canPush() -> input.setNotNeeded() -> PortFull
 has_output -> output.pushData() -> PortFull
 no_more_data_needed -> input.close(); output.finish() -> Finished
 input.isFinished() -> output.finish() -> Finished
 !input.hasData() -> input.setNeeded() -> NeedData
 otherwise -> pullData(); transform() -> Ready

Notice that prepare() does push: output.pushData(...) and input.pullData(...) both happen inside prepare(). The division of labour isn’t “prepare moves no data”, it’s “prepare does no expensive work”.

Pulling a chunk is O(1) — it’s a pointer swap. Actually computing on it is not, and that’s what work() is for:

 cpp
 
 
// src/Processors/ISimpleTransform.cpp:86
void ISimpleTransform::work()
{
 if (input_data.exception)
 {
 transform(input_data.exception);

 output_data = std::move(input_data);
 has_input = false;
 has_output = true;
 return;
 }

 try
 {
 transform(input_data.chunk, output_data.chunk);
 }
 catch (DB::Exception &)
 {
 output_data.exception = std::current_exception();
 has_output = true;
 has_input = false;
 return;
 }

 has_input = !needInputData();

 if (!skip_empty_chunks || output_data.chunk)
 has_output = true;

 if (has_output && !output_data.chunk && !getOutputPort().getHeader().empty())
 /// Support invariant that chunks must have the same number of columns as header.
 output_data.chunk = Chunk(getOutputPort().getHeader().cloneEmpty().getColumns(), 0);
}

Three things are worth noting.

Exceptions are data. A failure in the upstream processor is pushed into the port as an exception_ptr alongside the chunk, not thrown across the thread boundary. work() picks it up, hands it to transform(), and pushes it on. The exception then travels downstream with the stream, which means the error surfaces in the right place and every processor upstream of the failure gets to clean up.

work() cannot block. That’s why the Status::Async case exists: a processor that needs to wait for a socket or a file descriptor returns Async from prepare() and hands out an fd from schedule().

The empty-chunk invariant. Every chunk that crosses a port must have exactly as many columns as the port’s header, even if it has zero rows. ISimpleTransform enforces it at the bottom of work() by synthesising an empty chunk with the right shape.

Version counters: how a processor learns that a neighbour changed

The executor must know which processors to wake up. Polling every processor in the graph after every chunk would be O(V) per chunk, which for a twenty-node graph and a million chunks is a lot of pointless work.

ClickHouse’s answer is a version counter per connection, with a twist: the counter lives in the UpdateInfo of the port, and it’s compared against the value the neighbour last saw.

 cpp
 
 
// src/Processors/Port.h:34
 struct UpdateInfo
 {
 using UpdateList = VectorWithMemoryTracking<void *>;

 UpdateList * update_list = nullptr;
 void * id = nullptr;
 UInt64 version = 0;
 UInt64 prev_version = 0;

 void inline ALWAYS_INLINE update()
 {
 if (version == prev_version && update_list)
 update_list->push_back(id);

 ++version;
 }

 void inline ALWAYS_INLINE trigger() { prev_version = version; }
 };

update() is called whenever the port’s state changes (a push, a pull, a setNeeded). It bumps version, and — this is the interesting part — if version == prev_version, meaning nobody has consumed the change yet, it appends the port’s identity to a list of “ports that changed”. trigger() marks the change as seen by setting prev_version = version.

So the protocol is: a port that changes while nobody was looking records itself on a list; the executor drains that list and re-prepares exactly those neighbours. If a port changes twice before the neighbour is re-prepared, the neighbour is only told once. It doesn’t need to know how many times it changed, only that it did.

This is the mechanism that replaced the old “wake up the neighbour immediately” design, and it’s why prepare() can be called on a subset of the graph instead of on all of it.

The `ExecutingGraph`: nodes, edges, and who owns what

The executor’s view of the pipeline is ExecutingGraph, in src/Processors/Executors/Runtime/V1/. It holds one Node per processor and two kinds of Edge per connection: a direct edge from an output port to the consuming processor’s input port, and a backward edge from an input port back to the producing processor’s output port.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:31
 /// Edge represents connection between OutputPort and InputPort.
 /// For every connection, two edges are created: direct and backward (it is specified by backward flag).
 struct Edge
 {
 Edge(Node * to_, bool backward_,
 InputPort * input_port_, OutputPort * output_port_,
 VectorWithMemoryTracking<void *> * update_list)
 : to(to_), backward(backward_)
 , input_port(input_port_), output_port(output_port_)
 {
 update_info.update_list = update_list;
 update_info.id = this;
 }

 /// Node this edge points to.
 /// It is the node with output_port for direct edge or the node with input_port for backward edge.
 Node * to = nullptr;
 bool backward;
 /// Ports of this connection. Both pointers are set regardless of direction:
 /// for a direct edge (out -> in) input_port belongs to `to`; for a back edge the reverse.
 InputPort * input_port = nullptr;
 OutputPort * output_port = nullptr;

 /// Edge version is increased when port's state is changed (e.g. when data is pushed). See Port.h for details.
 /// To compare version with prev_version we can decide if neighbour processor need to be prepared.
 Port::UpdateInfo update_info;
 };

The Node carries a status with a very specific ownership protocol:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.h:62
 /// Status for processor.
 /// Can be owning or not. Owning means that executor who set this status can change node's data and nobody else can.
 enum class ExecStatus : uint8_t
 {
 Idle, /// prepare returned NeedData or PortFull. Non-owning.
 Preparing, /// some executor is preparing processor, or processor is in task_queue. Owning.
 Executing, /// prepare returned Ready or Async and task is executing. Owning.
 Finished, /// prepare returned Finished. Non-owning.
 };

“Owning” is the key word. Two threads may never call prepare() on the same processor, and may never call prepare() on connected processors at the same time. The status is how the graph enforces that: a node in Preparing or Executing is owned by exactly one thread, and the per-node std::mutex status_mutex is taken when ownership is transferred.

The traversal itself is updateNode(). It starts from one processor and spreads along the edges, re-preparing whoever was affected:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:524
ExecutingGraph::UpdateNodeStatus ExecutingGraph::updateNode(IProcessor & initial, Queue & queue, Queue & async_queue, UpdateNodeBuffers & buffers)
{
 Processors delayed_destruction;
 auto & updated_edges = buffers.updated_edges;
 auto & updated_processors = buffers.updated_processors;
 auto & pending_expansion = buffers.pending_expansion;

 /// The buffers may be left non-empty by a previous call that exited early or threw.
 buffers.reset();

 std::shared_lock read_lock(nodes_mutex);

 /// Traversal starts from the node related to updated processor
 updated_processors.push_back(processors_map.at(&initial));

 while (!updated_processors.empty() || !updated_edges.empty())
 {
 std::optional<std::unique_lock<std::mutex>> stack_top_lock;

 if (updated_processors.empty())
 {
 auto * edge = updated_edges.front();
 updated_edges.pop_front();

 /// Here we have ownership on edge, but node can be concurrently accessed.

 auto & node = *edge->to;

 std::unique_lock lock(node.status_mutex);

 ExecutingGraph::ExecStatus status = node.status;

 if (status != ExecutingGraph::ExecStatus::Finished)
 {
 if (edge->backward)
 node.updated_output_ports.push_back(edge->output_port);
 else
 node.updated_input_ports.push_back(edge->input_port);

 if (status == ExecutingGraph::ExecStatus::Idle)
 {
 node.status = ExecutingGraph::ExecStatus::Preparing;
 updated_processors.push_front(edge->to);
 stack_top_lock = std::move(lock);
 }
 else
 edge->to->processor()->onUpdatePorts();
 }
 }

Two work lists are at work here, and the distinction is worth spelling out. updated_processors is a queue of nodes to call prepare() on. updated_edges is a queue of connections whose port state changed and whose far side therefore needs to know.

The loop interleaves them. An edge whose far side is Idle promotes that node to Preparing and puts it on the processor queue. If the far side is already Preparing or Executing, the edge is simply recorded in that node’s updated_input_ports / updated_output_ports list, and the node will pick it up on its next prepare().

That’s the “which processor am I at” trick in its modern form: there is no cursor. There’s no “current processor” that remembers where it was. Instead, each processor holds the set of ports that changed since it was last prepared, and the graph walks edges until it reaches a fixed point.

The core of the traversal is the prepare() call and the dispatch on its result:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:621
 switch (*node.last_processor_status)
 {
 case IProcessor::Status::NeedData:
 case IProcessor::Status::PortFull:
 {
 node.status = ExecutingGraph::ExecStatus::Idle;
 if (spill_state.memory_reservation)
 {
 /// A request may have arrived while this node was owned by normal execution.
 for (auto * spillable : processor.getSpillables())
 if (spill_state.scheduleSpill(node, spillable, queue))
 break;
 }
 break;
 }
 case IProcessor::Status::Finished:
 {
 node.status = ExecutingGraph::ExecStatus::Finished;
 if (last_status != IProcessor::Status::Finished)
 spill_state.unregisterSpillables(node);
 accountFinishedProcessorInGroup(*node.processor_iter);
 break;
 }
 case IProcessor::Status::Ready:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 queue.push(node.processor());
 break;
 }
 case IProcessor::Status::Async:
 {
 node.status = ExecutingGraph::ExecStatus::Executing;
 async_queue.push(node.processor());
 break;
 }
 case IProcessor::Status::UpdatePipeline:
 {
 need_update_pipeline = true;
 pending_expansion.push_back(current);
 break;
 }
 }

Ready puts the processor on the work queue, to be picked up by a worker thread. NeedData/PortFull put the node back to Idle, meaning “somebody else must move first”. Finished retires the node for good. And then the edges that this processor disturbed get re-armed:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:666
 /// If you wonder why edges are pushed in reverse order,
 /// it is because updated_edges is a stack, and we prefer to get from stack
 /// input ports firstly, and then outputs, both in-order.
 ///
 /// Actually, there should be no difference in which order we process edges.
 /// However, some tests are sensitive to it (e.g. something like SELECT 1 UNION ALL 2).
 /// Let's not break this behaviour so far.

 for (auto it = node.post_updated_output_ports.rbegin(); it != node.post_updated_output_ports.rend(); ++it)
 {
 auto * edge = static_cast<ExecutingGraph::Edge *>(*it);
 updated_edges.push_front(edge);
 edge->update_info.trigger();
 }

edge->update_info.trigger() is the prev_version = version from Port.h, and it’s called after the edge is queued. The ordering matters: trigger() says “the neighbour has now seen this change”, and the queue entry is what makes it true.

The graph can change shape at runtime

The last status, UpdatePipeline, is unusual: a processor can, while running, decide that it needs more processors.

The classic case is external sort. MergeSortingTransform starts as a single-input, single-output processor, and when it decides to spill to disk it must insert a BufferingToFileSink and a BufferingFromFileSource into the middle of its own pipeline.

The mechanism is updatePipeline(), which returns a PipelineUpdate:

 cpp
 
 
// src/Processors/IProcessor.h:262
 struct PipelineUpdate
 {
 Processors to_add;
 Processors to_remove;
 };
 virtual PipelineUpdate updatePipeline();

The graph handles the insertion by rebuilding the edges of every node and scheduling the affected ones for re-preparation. The interesting part is the error handling, which encodes a hard-won lesson about allocation failures:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutingGraph.cpp:324
 try
 {
 return updatePipelineImpl(stack, cur_node, update);
 }
 catch (...)
 {
 expansion_failed = true;
 /// The ports of the pipeline's processors point at the new processors, and the pipeline is read after the graph is gone.
 std::lock_guard guard(processors_mutex);
 for (auto it = update.to_add.begin(); it != update.to_add.end();)
 {
 auto next = std::next(it);
 if (std::ranges::find(*processors, *it) == processors->end())
 processors->splice(processors->end(), update.to_add, it);
 it = next;
 }
 throw;
 }

The processor has already connected its ports to the new processors before updatePipeline() is even called. So if the graph then fails to record them — say, addNode throws MEMORY_LIMIT_EXCEEDED — the ports point at processors the graph doesn’t know about, and the graph is permanently inconsistent.

The handler keeps the new processors alive (so the ports stay valid) and sets expansion_failed, which makes every later expansion bail out instead of walking into the inconsistency.

Trade-offs

What this buys you.

The model is fully data-parallel by construction. Any node of the graph may be scheduled on any thread, because the only shared state between processors is a port, and a port is a single atomic word. There’s no notion of “this operator belongs to this thread”, which is what lets ClickHouse run a SELECT over a MergeTree with 16 streams on 16 cores without any of the 16 streams knowing about the other 15.

The graph also reshapes itself. External sort, GLOBAL JOIN, subquery materialisation, and LIMIT pushdown all insert or remove processors while the query is running, and none of them require a separate “compile the plan” phase.

Finally, the split between prepare() (cheap, may not run concurrently with connected peers) and work() (expensive, may run concurrently with everything) is precisely the split that lets the executor overlap computation without any locking on the hot path.

What it costs.

prepare() cannot run in parallel with connected processors. For a wide graph — say a UNION ALL of twenty subqueries, each with a dozen operators — this means the scheduling phase is effectively serialised per connection. That’s why the pipeline shape matters so much: every extra operator in a chain adds a serialisation point between the parallel parts.

It’s also why Pipe::resize() has a comment about splitting the resize processor when multiple threads are allocated:

 cpp
 
 
// src/QueryPipeline/Pipe.cpp:781
 /// Performance bottleneck identified: Severe lock contention for ExecutingGraph::Node::status_mutex.
 /// Issues observed:
 /// 1. Increased latency of ExecutingGraph::updateNode, lengthening overall query latency.
 /// 2. Unnecessary CPU cycle consumption in native_queued_spin_lock_slowpath (kernel).
 /// 3. Decreased CPU utilization.
 ///
 /// Proposed solution: Split ResizeProcessor when multiple threads are allocated to execute a query.

The second cost is the per-node status_mutex. Every edge traversal takes it. On a hot path with many small chunks this is measurable, and the comment above is the record of someone measuring it.

The third cost is that the model is hard to reason about in the presence of prepare() side effects. ISimpleTransform::prepare() pushes data. LimitTransform::prepare() closes ports. SortingTransform::prepare() can return UpdatePipeline and rebuild the graph. There’s no single “execute this operator” entry point — which is exactly why ClickHouse’s test infrastructure is built around assembling a whole Pipe.

Wrapping up

The real win: ClickHouse’s pipeline doesn’t have a “current operator”. It has a set of statuses and a graph that knows how to resolve them. That single design choice buys data parallelism for free, because the only thing two processors share is an atomic word.

Two things to internalize:

If you’re debugging a stuck ClickHouse query, look at the graph, not the SQL. EXPLAIN PIPELINE shows you the processors and the ports; the status codes tell you who’s blocked and on whom.
The model is deliberately push-driven at the ports. IS_NEEDED and HAS_DATA are the only two things that matter, and every producer must respect them or you get “Cannot push block to port which already has data”.

So next time you add a custom operator, remember: your prepare() must be O(1), your work() must not block, and your exceptions must travel as data — not as thrown errors.

ClickHouse Runs a Query Pipeline on N Threads: Task Queues, Work Stealing and a Pool That Grows on Demand

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse doesn’t hand you max_threads threads. It starts with one, measures how much parallel work the pipeline actually creates, and grows the pool only when idle threads genuinely can’t absorb it — then shrinks again if the server preempts you.

A query plan in ClickHouse is not one pipeline. It’s a stack of pipelines: read from storage, filter, project, aggregate, sort, format. Each stage produces many parallel streams, and consecutive stages are connected by ResizeProcessors that shuffle chunks between them.

Running that means answering three questions at once:

Which processor should run next? (the graph — see the previous article on ExecutingGraph)
Which thread should run it?
How many threads should exist at all?

The third question is the interesting one. A query that reads one small table does not want sixteen threads. A query that scans a hundred partitions does. And a query that has been running for thirty seconds may need fewer threads than it started with, because the server is under load from other queries.

ClickHouse’s answer, in src/Processors/Executors/Runtime/V1/, is a system that starts with one thread, measures demand as the pipeline runs, and grows the pool only when there is genuinely parallel work that idle threads cannot absorb.

Two kinds of parallelism

Before the machinery, let’s be precise about what “parallel” means here, because ClickHouse does two different things and they get conflated a lot.

Data parallelism is the obvious one: SELECT count() FROM t reads from N partitions, each partition is its own stream, and N streams are processed at once. This is a property of the plan — the number of streams is fixed when the plan is built.

Pipeline parallelism is the subtler one. Even in a pipeline with a single stream, the stage reading from disk isn’t blocked while the stage formatting the output is busy. A chunk is handed from one processor to the next, and while the consumer works on chunk k, the producer is already building chunk k+1.

The executor supports both, and the interesting bit is how it decides how many threads to use:

 sql
 
 
-- Sixteen streams of data parallelism; the number of threads that actually
-- run them is decided at execution time, not by the plan.
SELECT count() FROM numbers_mt;

 sql
 
 
-- The ceiling. The pool starts at one thread and grows only if there is
-- parallel work that idle threads cannot absorb.
SETTINGS max_threads = 16;
SELECT count() FROM numbers_mt;

Note that the two are not the same thing. A pipeline with 16 streams needs 16 threads to be fully parallel, but it will still make progress — just less of it — on one thread.

The shape of the executor

PipelineExecutor owns three things:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.h:49
private:
 /// Outlives graph registrations, including when construction fails.
 QueryStatusPtr process_list_element;
 ExecutingGraphPtr graph;

 ExecutorTasks tasks;

 /// Concurrency control related
 SlotAllocationPtr cpu_slots;
 AcquiredSlotPtr single_thread_cpu_slot; // cpu slot for single-thread mode to work using executeUntil()
 std::unique_ptr<ThreadPool> pool;
 std::mutex spawn_mutex;

graph is the processor graph (the subject of the previous article). tasks is the scheduler: the queues, the per-thread contexts, the counters. cpu_slots and pool are the resources: how many threads are allowed to exist, and the threads themselves.

The entry point is execute(), which is unremarkable apart from its comment about output formats:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:231
void PipelineExecutor::execute(size_t num_threads, bool concurrency_control)
{
 /// `QueryStatus` may release its reservation before this executor is destroyed.
 SCOPE_EXIT({ detachSpillables(); });
 if (process_list_element && !process_list_element->checkTimeLimit())
 cancel(IProcessor::CancelReason::CancelledByTimeout);

 num_threads = std::max<size_t>(num_threads, 1);

and executeImpl() is where the threads actually appear:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:783
void PipelineExecutor::executeImpl(size_t num_threads, bool concurrency_control)
{
 initializeExecution(num_threads, concurrency_control);

 try
 {
 if (num_threads > 1)
 {
 {
 std::lock_guard lock(spawn_mutex);
 // Start at least one thread, could block to acquire the first CPU slot
 spawnThreads(cpu_slots->acquire());
 }
 tasks.processAsyncTasks();
 pool->wait();
 }
 else
 {
 auto slot = cpu_slots->acquire();
 tasks.upscale(slot->slot_id);
 executeSingleThread(slot->slot_id, WorkloadResources(slot.get(), process_list_element));
 }
 }

Notice pool->wait(). The master thread does not execute pipeline work in the multi-threaded case. It spawns worker threads and then parks in processAsyncTasks(), which is the epoll loop for processors that returned Status::Async.

The master thread is a scheduler, not a worker — and that distinction matters for deadlock avoidance, which is why the master is never downscaled:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:591
 // The first thread is called master thread.
 // It is NOT the thread that handles async tasks (unless query has max_threads=1).
 // Master thread is different from other threads due to special role in scheduling:
 // 1. During query start, master thread spawns worker threads, so starting it fast is important.
 // 2. It should never be downscaled to avoid query deadlock (0 threads to process tasks), although it can be preempted.
 // 3. When using ConcurrencyControl, master thread is granted immediately and may not request CPU slot (see 'fair_round_robin').
 // 4. When using resource scheduler, master thread may use different resource link than worker threads.
 const auto master_threads = 1uz;

Task queues with work stealing

The scheduler’s central data structure is TaskQueue, and it’s per-thread-first:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/TasksQueue.h:17
/// A set of task queues for multithreaded processing.
/// Every threads has its dedicated queue and uses it preferably.
/// When there are no tasks left in dedicated queue it steals tasks from other threads
template <typename Task>
class TaskQueue
{
public:
 void init(size_t num_threads) { queues.resize(num_threads); }

 /// Push a task into thread's dedicated queue
 void push(Task * task, size_t thread_num)
 {
 queues[thread_num].push(task);
 use_queues = std::max(use_queues, thread_num + 1);
 ++num_tasks;
 }

 /// Returns thread number to pop task from.
 /// First it check dedicated queue, and only if it is empty, it steal from other threads
 size_t getAnyThreadWithTasks(size_t from_thread)
 {
 if (num_tasks == 0)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");

 if (from_thread >= use_queues)
 from_thread = 0;

 for (size_t i = 0, max_iterations = use_queues; i < max_iterations; ++i)
 {
 if (!queues[from_thread].empty())
 return from_thread;

 ++from_thread;
 if (from_thread == use_queues)
 {
 use_queues--; // lazy update: less expensive than updating on every pop
 from_thread = 0;
 }
 }

 throw Exception(ErrorCodes::LOGICAL_ERROR, "TaskQueue is empty");
 }

Each thread has a private FIFO. A thread with work prefers its own queue; only when it’s empty does it walk the other threads’ queues looking for something to steal.

use_queues is a lazily-maintained upper bound on how many queues are non-empty, so the scan is O(number of active threads) and not O(total threads). That “lazy update” comment is doing real work — updating the bound on every pop would cost more than it saves.

The per-thread affinity matters for cache reasons: a task that runs on thread 3 tends to produce data that thread 3 will consume next, and the data stays in that core’s L1/L2.

There are three queues, not one:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.h:43
 /// Queue with pointers to tasks. Each thread will concurrently read from it until finished flag is set.
 /// Stores processors need to be prepared. Preparing status is already set for them.
 TaskQueue<IProcessor> task_queue;

 /// Async tasks should be processed with higher priority, but also require task stealing logic.
 /// So we have a separate queue specifically for them.
 TaskQueue<IProcessor> fast_task_queue;
 std::atomic_bool has_fast_tasks = false; // Required only to enable local task optimization
 std::atomic_bool spill_requested = false;

 /// Queue which stores tasks where processors returned Async status after prepare.
 /// If multiple threads are used, main thread will wait for async tasks.
 /// For single thread, will wait for async tasks only when task_queue is empty.
 PollingQueue async_task_queue;

fast_task_queue holds processors that came back from an asynchronous operation — a socket that became readable, a file that finished being written. They get priority because they’re latency-sensitive: the data is already there and the client is waiting.

async_task_queue is different in kind. It’s not a task queue at all, it’s an epoll wrapper that owns the file descriptors.

The local-task optimization

Here’s the single most important performance trick in the executor, and it’s small.

When a processor finishes, the next processor to run is often known immediately — it’s the downstream consumer of the chunk that was just produced. Rather than pushing it onto a shared queue (where another thread would have to wake up, take the mutex, and pop it), the executor hands it directly to the thread that’s already running.

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:192
size_t ExecutorTasks::pushTasks(Queue & queue, Queue & async_queue, ExecutionThreadContext & context)
{
 /// Take local task from queue if has one.
 if (!queue.empty() && !has_fast_tasks.load(std::memory_order_relaxed)
 && context.num_scheduled_local_tasks < ExecutionThreadContext::max_scheduled_local_tasks)
 {
 ++context.num_scheduled_local_tasks;
 context.setTask(queue.front());
 queue.pop();
 }
 else
 {
 context.num_scheduled_local_tasks = 0;
 context.setTask(nullptr);
 }

and the constant that bounds it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutionThreadContext.h:55
 /// There is a performance optimization that schedules a task to the current thread, avoiding global task queue.
 /// Optimization decreases contention on global task queue but may cause starvation.
 /// See 01104_distributed_numbers_test.sql
 /// This constant tells us that we should skip the optimization
 /// if it was applied more than `max_scheduled_local_tasks` in a row.
 constexpr static size_t max_scheduled_local_tasks = 128;

128 in a row. The comment names the failure mode: starvation. If one thread keeps grabbing work locally while other threads sit idle with nothing to do, the pipeline runs on one core while the other max_threads - 1 cores idle.

The counter resets as soon as the local optimization is skipped, so a thread that has been running alone for 128 tasks will put the next one on the global queue and let somebody else take it.

The referenced test, 01104_distributed_numbers_test.sql, is a good illustration of why this matters: SELECT * FROM numbers(1000000000) with distributed_product_mode and a LIMIT — the limit transform stops the source, and if the source isn’t rebalanced the other threads never wake up.

The has_fast_tasks check in the condition is the second half of the guard: if there’s an asynchronous task waiting, it takes priority over a local task, because asynchronous tasks are latency-critical.

Measuring demand: `pushTasks` returns a number

Now the interesting part. pushTasks doesn’t just enqueue work — it measures how much parallel work was created, and returns it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:208
 if (!queue.empty() || !async_queue.empty())
 {
 std::unique_lock lock(mutex);

 /// Measure the parallelism of the push: how many threads could run the new tasks
 /// concurrently. Used by `PipelineExecutor` to size the slot-allocation ceiling via
 /// `ISlotAllocation::setMax` on the pipeline's `cpu_slots` handle.
 const size_t pushed = queue.size() + async_queue.size();

and then, at the end:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:232
 /// Capture state under the lock — tryWakeUp may unlock it internally to wake a
 /// thread, after which `total_slots` could change.
 const size_t idle_before = threads_queue.size();
 const size_t total_slots_now = total_slots;
 const size_t num_threads_now = num_threads; // const after init but read for clarity

 /// Wake up at least one thread that will wake up other threads if required
 tryWakeUpAnyOtherThreadWithTasks(context, lock);

 /// Return the number of additional pipeline threads the caller should target: pushed
 /// tasks that idle threads can't cover, clamped to remaining pipeline-width headroom
 /// (`num_threads - total_slots`). `PipelineExecutor` adds this delta to the
 /// `cpu_slots->setMax(...)` target so the slot allocator grows in step with demand.
 const size_t coverable = std::min(pushed, idle_before);
 const size_t uncovered = pushed - coverable;
 const size_t headroom = (num_threads_now > total_slots_now) ? (num_threads_now - total_slots_now) : 0;
 return std::min(uncovered, headroom);
 }
 return 0; // No new tasks -- no need for new threads

The arithmetic is: pushed tasks were created. Idle threads can absorb that many without being woken up. The remainder is genuine demand for new threads — clamped to the pipeline’s own width ceiling (num_threads, from max_threads), because the plan has a fixed number of streams and there’s no point running more threads than there is parallelism.

This is the demand signal. It’s computed from two observable facts — how many tasks were pushed, how many threads are parked — and nothing else. No profiling, no heuristics, no timers.

The executor then accumulates that demand and acts on it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:492
 /// Upscaling: `pushTasks` tells us how many additional threads would usefully
 /// parallelize the tasks it just enqueued. Publish our spawn_count into the shared
 /// `pending_demand` counter; the thread that actually holds `spawn_mutex` drains it
 /// so no demand is lost when multiple threads push concurrently.
 if (pool && spawn_count > 0)
 pending_demand.fetch_add(spawn_count, std::memory_order_relaxed);

 // Drain pending_demand under `spawn_mutex`. A thread that fails `try_lock`
 // must not exit while `pending_demand > 0` -- otherwise its add could race
 // with the owner's final `exchange(0)` and be left stranded after the owner
 // unlocks. The outer loop holds the thread until either it drains the demand
 // itself or observes the counter return to zero (meaning the owner caught it).
 while (pool && pending_demand.load(std::memory_order_acquire) > 0)
 {
 if (!spawn_mutex.try_lock())
 {
 std::this_thread::yield();
 continue;
 }
 try
 {
 std::lock_guard lock(spawn_mutex, std::adopt_lock);
 while (true)
 {
 size_t accumulated = pending_demand.exchange(0, std::memory_order_acq_rel);
 if (accumulated == 0)
 break;
 size_t target = std::min<size_t>(max_pipeline_threads, desired_threads + accumulated);
 if (target > desired_threads)
 {
 desired_threads = target;
 cpu_slots->setMax(target);
 }
 }
 spawnThreads({});
 }

This is small and rather careful concurrent code, and the comments are essentially a post-mortem.

The pending_demand counter is atomic so that threads which fail to take spawn_mutex don’t lose their demand. But a naive fetch_add would race with the owner’s exchange(0): thread A adds 1, the owner exchanges and gets 0, the owner unlocks, and thread A’s 1 sits in the counter forever.

The outer while loop is the fix. A thread that can’t get the mutex keeps spinning (yielding) until either it drains the demand itself or sees the counter back at zero, which means the owner already caught it.

The target computation is the second piece of policy: demand is accumulated into desired_threads, capped at max_pipeline_threads (the plan’s width), and applied by raising the slot allocator’s ceiling with cpu_slots->setMax(target).

spawnThreads then converts slots into actual OS threads:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:743
void PipelineExecutor::spawnThreads(AcquiredSlotPtr slot)
{
 while (cpu_slots)
 {
 if (!slot)
 slot = cpu_slots->tryAcquire();
 if (!slot)
 return;

 size_t thread_num = slot->slot_id;

 /// Count of threads in use should be updated for proper finish() condition.
 /// `upscale` returns the remaining headroom for spawning. Even when >0 we spawn
 /// one thread per iteration and re-enter `upscale` after the next `tryAcquire`.
 const size_t remaining_capacity = tasks.upscale(thread_num);

 /// Start new thread
 pool->scheduleOrThrowOnError([this, thread_num, thread_group = CurrentThread::getGroup(), my_slot = std::move(slot)]
 {
 ThreadGroupSwitcher switcher(thread_group, ThreadName::QUERY_ASYNC_EXECUTOR);

 try
 {
 executeSingleThread(thread_num, WorkloadResources(my_slot.get(), process_list_element));
 }
 catch (...)
 {
 /// In case of exception from executor itself, stop other threads.
 finish();
 tasks.getThreadContext(thread_num).setException(std::current_exception());
 }
 });

The tryAcquire is the gate. The slot allocator may refuse to hand out a slot if the server is out of CPU budget. If it refuses, no thread is spawned and the pipeline simply runs with fewer threads.

The thread count is not max_threads; it’s max_threads subject to what the scheduler is willing to give you.

CPU slots: `max_threads` is a ceiling, not a promise

The slot allocation is where the concurrency-control policy lives. There are four cases, and the comment enumerates them:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:605
 // There are a few possible modes of concurrency control.
 // 1) If we have `CREATE RESOURCE cpu (MASTER THREAD, WORKER THREAD)`:
 // all thread should be allocated through scheduler using the same resource link
 // 2) If we have `CREATE RESOURCE cpu (WORKER THREAD)`:
 // master thread uses "free" noncompeting CPU slot, the other threads are allocated through resource scheduler
 // 3) If we have two resources `CREATE RESOURCE master_cpu (MASTER THREAD)` and `CREATE RESOURCE worker_cpu (WORKER THREAD)`:
 // all thread should be allocated through scheduler using different resource links
 // 4) If we have no cpu-related resources:
 // the ConcurrencyControl class is used instead of resource scheduler

In the default case (case 4) the allocation is deliberately lazy:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:672
 else
 {
 /// Allocate CPU slots from concurrency control with guaranteed master thread slot.
 /// Default path (lazy_allocation=true): start at max=master_threads and grow on
 /// demand via setMax from the upscaling block. Zero-waste per idle pipeline.
 /// Rollback (lazy_allocation=false): pre-branch eager allocation — request the
 /// full ceiling upfront and never call setMax.
 SlotCount initial_max = lazy_allocation ? master_threads : num_threads;
 return ConcurrencyControl::instance().allocate(master_threads, initial_max);
 }

“Zero-waste per idle pipeline” is the point. A server running a thousand concurrent queries, each of which is a small scan, doesn’t want a thousand times max_threads slots reserved. It wants one slot each, growing if the query turns out to be wide.

The lazy flag is read exactly once, and the reason is a real bug:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:696
 /// Read the flag ONCE. A config reload mid-initialization would otherwise split a
 /// single query between lazy and eager strategies (e.g. allocate lazy with max=1 but
 /// then initialize desired_threads=num_threads, stranding subsequent setMax as a no-op).
 const bool lazy_allocation = ConcurrencyControl::instance().getLazyAllocation();

A config reload between “allocate the slots” and “set desired_threads” would leave the query in a state where setMax is a no-op and the pipeline never grows. One read, one strategy, for the lifetime of the query.

Preemption and downscaling

Threads can also be taken away. Under the resource scheduler with preemption enabled, a thread’s CPU lease is renewed periodically, and the renewal can fail:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:539
 if (resources.isCPULeaseRenewNeeded()) // Check if preemption is enabled (see `cpu_slot_preemption` server setting)
 {
 try
 {
 // Preemption point. Renewal could block execution due to CPU overload.
 // It may trigger callbacks to tasks.preempt() and tasks.resume()
 if (!resources.renewCPULease())
 {
 tasks.downscale(resources.getCPULeaseSlotId());
 break; // Downscaling. Unable to renew the lease - thread should stop (but could be rerun later).
 }

The renewal is checked at most every millisecond:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:119
 bool isCPULeaseRenewNeeded()
 {
 if (!lease)
 return false;
 UInt64 now_ns = clock_gettime_ns();
 if (now_ns - last_renew_ns < 1000000) // 1 ms: do not renew too often
 return false;
 last_renew_ns = now_ns;
 return true;
 }

Downscaling is not “kill the thread”. It’s “let this thread stop when it finishes its current task, and don’t schedule more onto it”:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:334
void ExecutorTasks::downscale(size_t slot_id)
{
 std::lock_guard lock(mutex);

 if (slot_id >= slot_count.size() || slot_count[slot_id] == 0)
 return;
 --slot_count[slot_id];

 if (slot_id + 1 == use_threads)
 {
 // The highest slot was downscaled, find new highest
 for (size_t i = use_threads; i > 0; --i)
 {
 if (slot_count[i - 1] > 0)
 {
 use_threads = i;
 break;
 }
 }
 }
}

Preemption is a third thing: the slot is released without downscaling, so the pipeline’s width is unchanged but the thread isn’t running. If the preempted thread was holding a task, that task is pushed back to the queue so another thread can steal it:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:356
void ExecutorTasks::preempt(size_t slot_id)
{
 std::unique_lock lock(mutex);
 --total_slots;

 /// We should make sure that preempted thread has no local task inside context.
 /// It is allowed to have tasks in `task_queue` or `fast_task_queue` because they can be stealed by other threads.
 auto & context = executor_contexts[slot_id];
 if (auto * task = context->popTask())
 {
 task_queue.push(task, slot_id);
 /// Wake up at least one thread to avoid deadlocks (all other threads maybe idle)
 tryWakeUpAnyOtherThreadWithTasks(*context, lock); // this releases the lock if it wakes up a thread
 }

The asymmetry is the point. A local task (held in the thread’s context) is invisible to other threads and must be handed back explicitly, whereas a task in the shared queue is stealable and needs no action.

The pulling mode: `executeUntil`

Not every query is executed to completion in one go. The HTTP protocol, for instance, wants to stream rows to the client as they become available, and the client may disconnect at any moment. That’s what executeUntil() is for:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:261
bool PipelineExecutor::executeUntil(std::atomic_bool * yield_flag)
{
 const auto uncaught_exceptions = std::uncaught_exceptions();
 SCOPE_EXIT({
 if (tasks.isFinished() || std::uncaught_exceptions() > uncaught_exceptions)
 detachSpillables();
 });
 if (!is_execution_initialized)
 {
 initializeExecution(1, true);

 // Acquire slot until we are done
 single_thread_cpu_slot = cpu_slots->acquire();
 tasks.upscale(single_thread_cpu_slot->slot_id);
 chassert(single_thread_cpu_slot && "Unable to allocate cpu slot for the first thread, but we just allocated at least one slot");

 if (yield_flag && *yield_flag)
 return true;
 }

 executeStepImpl(0, WorkloadResources(single_thread_cpu_slot.get(), process_list_element), yield_flag);

 if (!tasks.isFinished())
 return true;

The yield_flag is the interesting parameter. It’s checked at the bottom of the worker loop:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:576
 /// We have executed single processor. Check if we need to yield execution.
 if (yield_flag && *yield_flag)
 break;

So the executor runs one processor at a time and then checks the flag. If the flag says “enough for now”, it returns to the caller, which can hand the chunk to the client and then call executeUntil() again. This is the “pulling” model at the level of the whole pipeline: the caller pulls.

PullingPipelineExecutor::pull() is the caller:

 cpp
 
 
// src/Processors/Executors/PullingPipelineExecutor.cpp:49
bool PullingPipelineExecutor::pull(Chunk & chunk)
{
 if (!executor)
 {
 executor = createExecutor(pipeline.processors, pipeline.process_list_element);
 executor->setReadProgressCallback(pipeline.getReadProgressCallback());
 executor->setStepProfiler(pipeline.getStepProfiler());
 }

 /// Throws when the time limit is exceeded with `timeout_overflow_mode = 'throw'`. With 'break' the partial result
 /// is returned as a success: the execution is cancelled with `CancelledByTimeout`, and `executeUntil` below
 /// finalizes it - the pending read progress is reported and the format is finalized - before the end of the
 /// data is reported.
 if (pipeline.process_list_element && !pipeline.process_list_element->checkTimeLimitSoft())
 {
 executor->cancel(IProcessor::CancelReason::CancelledByTimeout);
 pipeline.process_list_element->checkTimeLimit();
 }

 if (!executor->executeUntil(&has_data_flag))
 return false;

 chunk = pulling_format->getChunk();
 return true;
}

has_data_flag is set by the output format when it has a chunk ready. So the loop is: run the pipeline until the format has produced something (or the pipeline is finished), then hand the chunk up.

Note that executeUntil runs the executor in single-threaded mode (initializeExecution(1, true)). The pulling executor is deliberately not parallel — it exists for cases like INSERT ... SELECT into a table, or formatting a result to a client that wants to consume it incrementally.

Trade-offs

What this buys you.

The pool grows with the work, not with the setting. A query that turns out to be narrow never pays for more than one or two threads, even if max_threads = 64. A query that turns out to be wide gets threads as fast as the scheduler will give them, without any startup latency. And the measurement is exact rather than heuristic: pushTasks counts the tasks it created and subtracts the threads that could absorb them.

Work stealing with per-thread affinity gives good cache behaviour and good load balance. The common case (a thread feeding itself) is a pointer swap with no locking; the uncommon case (one thread has all the work) is handled by theft.

The separation of “what work exists” from “how many resources it gets” also means the same executor serves queries with max_threads = 1 (no contention at all, the single thread takes everything locally) and queries with max_threads = 128.

What it costs.

The machinery is intricate, and the comments are full of post-mortems: the try_lock/pending_demand race, the lazy_allocation config reload, the max_scheduled_local_tasks starvation. Each of those is a bug that shipped and was fixed. The design is sound, but the implementation is carrying the weight of a decade of production.

There’s also a hard limit on the demand signal. pushTasks measures the tasks pushed by one processor at one moment. If a pipeline is wide but each individual push produces one task, the demand signal is one at a time, and the pool grows one thread per push rather than immediately. The initial_parallel computation in initializeExecution exists precisely to patch this for the first wave:

 cpp
 
 
// src/Processors/Executors/Runtime/V1/PipelineExecutor.cpp:722
 /// Initial queued parallelism never routes through `pushTasks`, so size setMax here to
 /// cover it. For multi-source pipelines (e.g. UNION ALL of N subqueries) this prevents
 /// the pipeline from being stuck single-threaded if the master's first task happens not
 /// to push more work. Only matters on the lazy path; the eager path already has the full
 /// ceiling. `initial_parallel` is the total target — the master thread is one of its
 /// consumers, no +1 needed.
 if (lazy_allocation && initial_parallel > 1)

The preemption path is the other sharp edge. A thread that can’t renew its lease stops, and the pipeline continues with fewer threads — but if every thread is preempted at once, the pipeline must not deadlock. The threads_queue.size() + 1 == total_slots check in tryGetTask is the mechanism that detects “the last worker is about to go idle” and calls finish():

 cpp
 
 
// src/Processors/Executors/Runtime/V1/ExecutorTasks.cpp:158
 /// This thread has no tasks to do and is going to wait.
 /// Finish execution if this was the last active thread.
 chassert(task_queue.empty() && fast_task_queue.empty());
 if (threads_queue.size() + 1 == total_slots && async_task_queue.empty())
 {
 lock.unlock();
 finish();
 return;
 }

Getting this wrong in either direction is bad: finish too early and the query returns partial data, never finish and the query hangs. The chassert documents that the two conditions must hold together — if the queues are not empty, the pipeline is not finished and the thread must keep waiting.

Finally, there’s the question of when to stop growing. desired_threads only ever goes up within a query. A pipeline that’s wide at the start and narrow at the end keeps its threads; there’s no downscaling based on demand, only downscaling due to preemption. For a long-running query over a table that becomes narrow, that’s wasted capacity.

Wrapping up

The real win: ClickHouse doesn’t guess how many threads your query needs. It measures. pushTasks returns a number — how many new tasks idle threads can’t cover — and the pool grows by exactly that much, capped by the plan’s own width.

Two things to internalize:

max_threads is a ceiling, not a schedule. If you see a query using 2 threads on a 64-core box, that’s the demand signal working, not a bug.
The server can take threads back from you. Preemption and downscaling are first-class, and your query is expected to keep going on fewer threads.

So when you profile a slow query, don’t count threads first. Count streams, and ask whether the pipeline had enough independent work to push into idle threads in the first place.

PostgreSQL Waits for Locks

Jay Hubbert — Tue, 06 Oct 2026 09:00:00 +0000

Every lock in PostgreSQL is a bet. When a backend wants a resource that somebody else has, it has to choose: burn CPU waiting, or go to sleep and let the kernel schedule somebody else. Both are wrong sometimes. Spinning wastes a core and burns power; sleeping costs a context switch, wakes up in the wrong place, and — worst of all — adds the victim’s release time to the waiter’s latency.

PostgreSQL’s answer is a ladder. It spins in user space with an architecture-specific pause instruction, then sleeps for a randomized, exponentially growing interval, and only then blocks in the kernel on a POSIX semaphore. Every rung of that ladder is a separate piece of code with a separate set of trade-offs, and the interesting part of the design is where the rungs are.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

The ladder, in one table

Layer	Object	Waits by	Kernel involved?
1	Spinlock (`slock_t`)	`lock; xchgb` in a loop, then `pg_usleep()`	Only after ~1000 sleeps
2	Atomic (`pg_atomic_uint32`)	Compare-and-swap loop	No
3	LWLock	Spin, then block on a POSIX semaphore	Yes
4	Latch	Sleep in `epoll_wait()`/`poll()`	Yes
5	Heavyweight lock	Block on the lock manager’s semaphore	Yes

The ladder is not uniform, and that is the point. A spinlock protects something so small that the only correct thing to do is spin. An LWLock protects something that might take milliseconds. A latch is not a lock at all — it is a one-way door that a backend waits on until somebody opens it.

Layer 1: the spinlock

The hardware primitives

A spinlock is one byte. The contract is spelled out in src/include/storage/s_lock.h:

int TAS(slock_t *lock) Atomic test-and-set instruction. Attempt to acquire the lock, but do not wait. Returns 0 if successful, nonzero if unable to acquire the lock.

int TAS_SPIN(slock_t *lock) Like TAS(), but this version is used when waiting for a lock previously found to be contended. By default, this is the same as TAS(), but on some architectures it’s better to poll a contended lock using an unlocked instruction and retry the atomic test-and-set only when it appears free.

On x86_64 the whole thing is one instruction pair:

 c
 
 
// src/include/storage/s_lock.h
static inline int
tas(volatile slock_t *lock)
{
	slock_t		_res = 1;

	__asm__ __volatile__(
	"	lock			\n"
	"	xchgb	%0,%1	\n"
:		"+q"(_res), "+m"(*lock)
:		/* no inputs */
:		"memory", "cc");
	return (int) _res;
}

lock xchg is a read-modify-write with the bus held, so it is atomic with respect to every other core. The old value comes back in %0, and the return value tells the caller whether it got the lock (old value zero) or found it taken.

The free test

The interesting part is TAS_SPIN, and on x86_64 it is a one-liner:

 c
 
 
// src/include/storage/s_lock.h
/*
 * On Intel EM64T, it's a win to use a non-locking test before the xchg proper,
 * but only when spinning.
 *
 * See also Implementing Scalable Atomic Locks for Multi-Core Intel(tm) EM64T
 * and IA32, by Michael Chynoweth and Mary R. Lee. As of this writing, it is
 * available at:
 * http://software.intel.com/en-us/articles/implementing-scalable-atomic-locks-for-multi-core-intel-em64t-and-ia32-architectures
 */
#define TAS_SPIN(lock) (*(lock) ? 1 : TAS(lock))

Read it as: if the lock byte is non-zero, it is taken, so return “failed” without touching the bus. Only if it looks free do we pay for the lock xchgb.

The reason this only happens while spinning is subtle. A lock-prefixed instruction on x86 is a full barrier: it drains the store buffer and serializes against other cores’ atomic operations. If you issue one for every failed attempt, you are asking the memory subsystem to arbitrate between every spinner on the machine — which is exactly the contention you are trying to avoid. A plain load is free (it can be satisfied from the local cache) and lets the spinner’s core keep running at full speed.

The “only when spinning” part is about the first acquisition. On the first attempt, the lock is very likely free, and the cost of the lock xchgb is paid anyway; doing a speculative plain load first would just add a branch and a potential cache miss on the fast path. And there is a correctness-adjacent reason too: the free test is a prediction. Between the load and the xchgb, another core could take the lock. That is fine — the xchgb is still the arbiter, and it will report the failure — but it means the free test can produce false “free” answers, and only the atomic instruction may decide.

The 32-bit x86 version takes the opposite approach and does the free test inside tas() itself:

 c
 
 
// src/include/storage/s_lock.h
static inline int
tas(volatile slock_t *lock)
{
	slock_t		_res = 1;

	/*
	 * Use a non-locking test before asserting the bus lock. Note that the
	 * extra test appears to be a small loss on some x86 platforms and a small
	 * win on others; it's by no means clear that we should keep it.
	 *
	 * When this was last tested, we didn't have separate TAS() and TAS_SPIN()
	 * macros. Nowadays it probably would be better to do a non-locking test
	 * in TAS_SPIN() but not in TAS(), like on x86_64, but no-one's done the
	 * testing to verify that. Without some empirical evidence, better to
	 * leave it alone.
	 */
	__asm__ __volatile__(
	"	cmpb	$0,%1		\n"
	"	jne		TAS%=_out	\n"
	"	lock				\n"
	"	xchgb	%0,%1		\n"
	"TAS%=_out: \n"
:		"+q"(_res), "+m"(*lock)
:		/* no inputs */
:		"memory", "cc");
	return (int) _res;
}

Note the comment’s honesty: “it’s by no means clear that we should keep it”. This is the kind of micro-optimization that is kept because it was measured once, long ago, and nobody has had the heart to re-measure it.

The pause instruction

Between attempts, the spinner has to do something. Doing nothing is bad: modern x86 cores issue speculatively and out of order, so a tight loop that keeps polling a memory location hammers the memory subsystem and the core’s pipeline. The answer is a hint instruction that tells the CPU “you are in a spin-wait loop”.

On x86 that is PAUSE, and PostgreSQL spells it the classic way:

 c
 
 
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * Adding a PAUSE in the spin delay loop is demonstrably a no-op on
	 * Opteron, but it may be of some use on EM64T, so we keep it.
	 */
	__asm__ __volatile__(
	" rep; nop			\n");
}

The 32-bit x86 version carries the full Intel explanation:

 c
 
 
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * This sequence is equivalent to the PAUSE instruction ("rep" is
	 * ignored by old IA32 processors if the following instruction is
	 * not a string operation); the IA-32 Architecture Software
	 * Developer's Manual, Vol. 3, Section 7.7.2 describes why using
	 * PAUSE in the inner loop of a spin lock is necessary for good
	 * performance:
	 *
	 * The PAUSE instruction improves the performance of IA-32
	 * processors supporting Hyper-Threading Technology when
	 * executing spin-wait loops and other routines where one
	 * thread is accessing a shared lock or semaphore in a tight
	 * polling loop. When executing a spin-wait loop, the
	 * processor can suffer a severe performance penalty when
	 * exiting the loop because it detects a possible memory order
	 * violation and flushes the core processor's pipeline. The
	 * PAUSE instruction provides a hint to the processor that the
	 * code sequence is a spin-wait loop. The processor uses this
	 * hint to avoid the memory order violation and prevent the
	 * pipeline flush. In addition, the PAUSE instruction
	 * de-pipelines the spin-wait loop to prevent it from
	 * consuming execution resources excessively.
	 */
	__asm__ __volatile__(
	" rep; nop			\n");
}

“De-pipelines the spin-wait loop to prevent it from consuming execution resources excessively” is the key phrase: PAUSE exists so that a spinning thread does not steal issue slots from the thread that is about to release the lock. On a hyper-threaded machine that matters enormously, because the spinner and the holder may share a physical core.

On ARM64 the answer is ISB:

 c
 
 
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * Using an ISB instruction to delay in spinlock loops appears beneficial
	 * on high-core-count ARM64 processors. It seems mostly a wash for smaller
	 * gear, and ISB doesn't exist at all on pre-v7 ARM chips.
	 */
	__asm__ __volatile__(
	" isb;				\n");
}

ISB flushes the instruction pipeline, which has the same effect as PAUSE: it stops the core from speculatively executing past the spin loop.

PowerPC is interesting because it does not define SPIN_DELAY() at all, so the generic fallback in s_lock.h applies:

 c
 
 
// src/include/storage/s_lock.h
#if !defined(SPIN_DELAY)
#define SPIN_DELAY()	((void) 0)
#endif		/* SPIN_DELAY */

Instead, PowerPC gets its memory ordering from the lwarx/stwcx. compare-and-swap loop, and uses lwsync as its barrier:

 c
 
 
// src/include/storage/s_lock.h
/*
 * NOTE: per the Enhanced PowerPC Architecture manual, v1.0 dated 7-May-2002,
 * an isync is a sufficient synchronization barrier after a lwarx/stwcx loop.
 * But if the spinlock is in ordinary memory, we can use lwsync instead for
 * better performance.
 */
static inline int
tas(volatile slock_t *lock)
{
	slock_t _t;
	int _res;

	__asm__ __volatile__(
"	lwarx %0,0,%3,1	\n"
"	cmpwi %0,0		\n"
"	bne TAS%=_fail	\n"
"	addi %0,%0,1		\n"
"	stwcx. %0,0,%3		\n"
"	beq TAS%=_ok	\n"
"TAS%=_fail: \n"
"	li %1,1		\n"
"	b TAS%=_out	\n"
"TAS%=_ok: \n"
"	lwsync				\n"
"	li %1,0		\n"
"TAS%=_out: \n"
:	"=&b"(_t), "=r"(_res), "+m"(*lock)
:	"r"(lock)
:	"memory", "cc");
	return _res;
}

/*
 * PowerPC S_UNLOCK is almost standard but requires a "sync" instruction.
 * But we can use lwsync instead for better performance.
 */
#define S_UNLOCK(lock)	\
do \
{ \
	__asm__ __volatile__ ("	lwsync \n" ::: "memory"); \
	*((volatile slock_t *) (lock)) = 0; \
} while (0)

lwarx is “load with reservation” and stwcx. is “store with conditional”, which together implement a compare-and-swap on PowerPC’s older architectures. The lwsync after a successful acquisition is the acquire barrier; the lwsync before the store in S_UNLOCK is the release barrier.

The memory ordering contract

The header is unusually explicit about what the compiler and the hardware are allowed to do:

It is the responsibility of these macros to make sure that the compiler does not re-order accesses to shared memory to precede the actual lock acquisition, or follow the lock release.

On platforms with weak memory ordering, the TAS(), TAS_SPIN(), and S_UNLOCK() macros must further include hardware-level memory fence instructions to prevent similar re-ordering at the hardware level. TAS() and TAS_SPIN() must guarantee that loads and stores issued after the macro are not executed until the lock has been obtained. Conversely, S_UNLOCK() must guarantee that loads and stores issued before the macro have been executed before the lock is released.

Two different barriers are needed, and the difference is why S_UNLOCK() is not just *(lock) = 0 everywhere:

 c
 
 
// src/include/storage/s_lock.h
/*
 * Default implementation of S_UNLOCK() for gcc/icc.
 *
 * Note that this implementation is unsafe for any platform that can reorder
 * a memory access (either load or store) after a following store. That
 * happens not to be possible on x86 and most legacy architectures (some are
 * single-processor!), but many modern systems have weaker memory ordering.
 * Those that do must define their own version of S_UNLOCK() rather than
 * relying on this one.
 */
#if !defined(S_UNLOCK)
#define S_UNLOCK(lock)	\
	do { __asm__ __volatile__("" : : : "memory"); *(lock) = 0; } while (0)
#endif		/* !defined(S_UNLOCK) */

The default is a compiler barrier only. On x86 that is enough, because the hardware is total-store-ordered: stores cannot be reordered with other stores, and the lock prefix on the acquisition side prevents the hardware from moving anything past it. On ARM or PowerPC you need an explicit instruction, which is why those architectures define their own S_UNLOCK.

The generic spin loop

Everything above is platform-specific. The portable part is in src/backend/storage/lmgr/s_lock.c:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
#ifdef USE_DEFAULT_S_LOCK
/*
 * s_lock(lock) - platform-independent portion of waiting for a spinlock.
 */
int
s_lock(volatile slock_t *lock, const char *file, int line, const char *func)
{
	SpinDelayStatus delayStatus;

	init_spin_delay(&delayStatus, file, line, func);

	while (TAS_SPIN(lock))
	{
		perform_spin_delay(&delayStatus);
	}

	finish_spin_delay(&delayStatus);

	return delayStatus.delays;
}
#endif

The fast path is one TAS_SPIN that succeeds. The slow path calls perform_spin_delay() in a loop until the lock is acquired, and the number of delays is returned so that the caller can report it (this is how a “stuck spinlock” panic gets a sensible message).

perform_spin_delay()

This is the function that actually implements the ladder:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * Wait while spinning on a contended spinlock.
 */
void
perform_spin_delay(SpinDelayStatus *status)
{
	/* CPU-specific delay each time through the loop */
	SPIN_DELAY();

	/* Block the process every spins_per_delay tries */
	if (++(status->spins) >= spins_per_delay)
	{
		if (++(status->delays) > NUM_DELAYS)
			s_lock_stuck(status->file, status->line, status->func);

		if (status->cur_delay == 0) /* first time to delay? */
			status->cur_delay = MIN_DELAY_USEC;

		/*
		 * Once we start sleeping, the overhead of reporting a wait event is
		 * justified. Actively spinning easily stands out in profilers, but
		 * sleeping with an exponential backoff is harder to spot...
		 *
		 * We might want to report something more granular at some point, but
		 * this is better than nothing.
		 */
		pgstat_report_wait_start(WAIT_EVENT_SPIN_DELAY);
		pg_usleep(status->cur_delay);
		pgstat_report_wait_end();

#if defined(S_LOCK_TEST)
		fprintf(stdout, "*");
		fflush(stdout);
#endif

		/* increase delay by a random fraction between 1X and 2X */
		status->cur_delay += (int) (status->cur_delay *
									pg_prng_double(&pg_global_prng_state) + 0.5);
		/* wrap back to minimum delay when max is exceeded */
		if (status->cur_delay > MAX_DELAY_USEC)
			status->cur_delay = MIN_DELAY_USEC;

		status->spins = 0;
	}
}

Three phases per iteration:

SPIN_DELAY() — the architecture-specific pause. Executed on every pass through the loop, not just every spins_per_delay tries.
Every spins_per_delay attempts, sleep for cur_delay microseconds via pg_usleep().
Multiply the delay by a random factor between 1× and 2×.

The constants:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
#define MIN_SPINS_PER_DELAY 10
#define MAX_SPINS_PER_DELAY 1000
#define NUM_DELAYS			1000
#define MIN_DELAY_USEC		1000L
#define MAX_DELAY_USEC		1000000L

So: spin up to spins_per_delay times (default 100), then sleep 1 ms, then sleep between 1 and 2 ms, then between 2 and 4 ms, and so on, capped at 1 second — and after 1000 delays, declare the spinlock stuck and PANIC.

Note the granularity of the backoff: cur_delay += (int) (cur_delay * pg_prng_double(...) + 0.5) multiplies by a random number in [1.0, 2.0), rounded. That is a jittered exponential backoff, and the jitter is the whole point.

Why the randomness matters

The file header explains the starvation problem:

Once we do decide to block, we use randomly increasing pg_usleep() delays. The first delay is 1 msec, then the delay randomly increases to about one second, after which we reset to 1 msec and start again. The idea here is that in the presence of heavy contention we need to increase the delay, else the spinlock holder may never get to run and release the lock. (Consider situation where spinlock holder has been nice’d down in priority by the scheduler — it will not get scheduled until all would-be acquirers are sleeping, so if we always use a 1-msec sleep, there is a real possibility of starvation.) But we can’t just clamp the delay to an upper bound, else it would take a long time to make a reasonable number of tries.

Two separate effects are at work:

Un-jittered backoff would synchronise the spinners. If every waiter sleeps for exactly 1 ms, they all wake up at exactly the same time, all collide, and all go back to sleep for exactly 1 ms. The lock holder gets one collision per millisecond instead of a smooth trickle of retries. Worse, the collisions mean the lock’s cache line ping-pongs between cores.
Un-jittered backoff starves the holder. On a machine with more runnable threads than cores, a spinner that wakes up every millisecond is runnable every millisecond. The holder, which needs the CPU to release the lock, gets scheduled less often the more spinners there are. The jitter breaks that pattern: spinners wake at uncorrelated times, so the holder gets a chance to run between them.

The pg_prng_double() call uses the backend’s global PRNG state, so the jitter differs between backends — two spinners on the same lock will not pick the same delay.

The adaptive spin count

How many times should we spin before sleeping? Too few and we pay a kernel round trip for a lock that would have been free in a microsecond; too many and we burn CPU on a lock that is never going to be released soon. PostgreSQL guesses, and then learns.

The guess starts at 100:

 c
 
 
// src/include/storage/s_lock.h
/* Support for dynamic adjustment of spins_per_delay */
#define DEFAULT_SPINS_PER_DELAY 100

The learning happens in two steps. First, locally, at the end of each contended acquisition:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * After acquiring a spinlock, update estimates about how long to loop.
 *
 * If we were able to acquire the lock without delaying, it's a good
 * indication we are in a multiprocessor. If we had to delay, it's a sign
 * (but not a sure thing) that we are in a uniprocessor. Hence, we
 * decrement spins_per_delay slowly when we had to delay, and increase it
 * rapidly when we didn't. It's expected that spins_per_delay will
 * converge to the minimum value on a uniprocessor and to the maximum
 * value on a multiprocessor.
 *
 * Note: spins_per_delay is local within our current process. We want to
 * average these observations across multiple backends, since it's
 * relatively rare for this function to even get entered, and so a single
 * backend might not live long enough to converge on a good value. That
 * is handled by the two routines below.
 */
void
finish_spin_delay(SpinDelayStatus *status)
{
	if (status->cur_delay == 0)
	{
		/* we never had to delay */
		if (spins_per_delay < MAX_SPINS_PER_DELAY)
			spins_per_delay = Min(spins_per_delay + 100, MAX_SPINS_PER_DELAY);
	}
	else
	{
		if (spins_per_delay > MIN_SPINS_PER_DELAY)
			spins_per_delay = Max(spins_per_delay - 1, MIN_SPINS_PER_DELAY);
	}
}

The asymmetry is the interesting part: +100 when we got the lock without sleeping, −1 when we had to sleep. The reasoning is that “we never had to delay” is a strong, cheap signal that spinning is worth it, whereas “we had to delay” is a weak signal that it is not — maybe the lock holder was descheduled, maybe it was just unlucky. So the estimate ratchets up quickly and down slowly.

Second, globally. The local value is folded into a shared estimate at backend exit:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * Update shared estimate of spins_per_delay during backend exit.
 *
 * NB: this has to be pretty fast as it is called while holding a spinlock
 */
int
update_spins_per_delay(int shared_spins_per_delay)
{
	/*
	 * We use an exponential moving average with a relatively slow adaption
	 * rate, so that noise in any one backend's result won't affect the shared
	 * value too much. As long as both inputs are within the allowed range,
	 * the result must be too, so we need not worry about clamping the result.
	 *
	 * We deliberately truncate rather than rounding; this is so that single
	 * adjustments inside a backend can affect the shared estimate (see the
	 * asymmetric adjustment rules above).
	 */
	return (shared_spins_per_delay * 15 + spins_per_delay) / 16;
}

(shared * 15 + local) / 16 is an exponential moving average with a weight of 1/16 on the new sample. So a single backend’s experience moves the shared estimate by at most 1/16 of the distance between the two values — and with the truncation, it moves it by at least one unit, which is why the comment mentions “we deliberately truncate rather than rounding”.

The shared value lives in ProcGlobal->spins_per_delay and is read into each backend’s local copy at startup:

 c
 
 
// src/backend/storage/lmgr/proc.c
	/* Update shared estimate of spins_per_delay */
	ProcGlobal->spins_per_delay = update_spins_per_delay(ProcGlobal->spins_per_delay);

	SpinLockRelease(&ProcGlobal->freeProcsLock);

So the estimate is a genuinely global, slowly-adapting, cross-backend number — and it converges, over time, to 1000 (spin hard) on a machine where locks are usually free, or to 10 (give up quickly) on a machine where they are not.

Stuck spinlock detection

If the lock is never released, the loop must terminate. After NUM_DELAYS (1000) sleeps, the backend panics:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
/*
 * s_lock_stuck() - complain about a stuck spinlock
 */
static void
s_lock_stuck(const char *file, int line, const char *func)
{
	if (!func)
		func = "(unknown)";
#if defined(S_LOCK_TEST)
	fprintf(stderr,
			"\nStuck spinlock detected at %s, %s:%d.\n",
			func, file, line);
	exit(1);
#else
	elog(PANIC, "stuck spinlock detected at %s, %s:%d",
		 func, file, line);
#endif
}

The file header explains the timeout philosophy:

We time out and declare error after NUM_DELAYS delays (thus, exactly that many tries). With the given settings, this will usually take 2 or so minutes. It seems better to fix the total number of tries (and thus the probability of unintended failure) than to fix the total time spent.

Fixing the number of tries rather than the elapsed time means the failure probability does not depend on how loaded the machine is. And the message includes __FILE__, __LINE__ and __func__ of the spinlock’s acquisition site, which is the first thing you want to know when the postmaster takes a PANIC.

Note also that the spin delay is a wait event:

 c
 
 
// src/backend/storage/lmgr/s_lock.c
		pgstat_report_wait_start(WAIT_EVENT_SPIN_DELAY);
		pg_usleep(status->cur_delay);
		pgstat_report_wait_end();

The event is called SpinDelay in the wait event catalogue (src/backend/utils/activity/wait_event_names.txt):

text

SPIN_DELAY	"Waiting while acquiring a contended spinlock."

So pg_stat_activity will show a backend in SpinDelay state — but only when it is sleeping, not when it is actively spinning. The comment in the code explains why the reporting is only done around the sleep: “Actively spinning easily stands out in profilers, but sleeping with an exponential backoff is harder to spot”. If you see a backend burning CPU with no wait event at all, that is a spinner.

Layer 2: the atomics

Spinlocks are the blunt instrument. PostgreSQL also has a proper compare-and-swap API, in src/include/port/atomics.h and the per-architecture headers under src/include/port/atomics/.

The x86 implementations

 c
 
 
// src/include/port/atomics/arch-x86.h
#define PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32
static inline bool
pg_atomic_compare_exchange_u32_impl(volatile pg_atomic_uint32 *ptr,
									uint32 *expected, uint32 newval)
{
	char	ret;

	/*
	 * Perform cmpxchg and use the zero flag which it implicitly sets when
	 * equal to measure the success.
	 */
	__asm__ __volatile__(
	"	lock				\n"
	"	cmpxchgl	%4,%5	\n"
	" setz		%2		\n"
:		"=a" (*expected), "=m"(ptr->value), "=q" (ret)
:		"a" (*expected), "r" (newval), "m"(ptr->value)
:		"memory", "cc");
	return (bool) ret;
}

#define PG_HAVE_ATOMIC_FETCH_ADD_U32
static inline uint32
pg_atomic_fetch_add_u32_impl(volatile pg_atomic_uint32 *ptr, int32 add_)
{
	uint32 res;
	__asm__ __volatile__(
	"	lock				\n"
	"	xaddl	%0,%1		\n"
:		"=q"(res), "=m"(ptr->value)
:		"0" (add_), "m"(ptr->value)
:		"memory", "cc");
	return res;
}

Two instructions, two ideas:

lock cmpxchgl compares the accumulator (%eax) with the destination; if they are equal, the destination is loaded with the source. The old destination value is written back into %eax on failure — which is where the “expected” value comes from. The lock prefix makes the whole thing atomic.
setz reads the zero flag that cmpxchg implicitly sets when the comparison succeeded, and turns it into a 0-or-1 byte. This is the trick the comment refers to: “Perform cmpxchg and use the zero flag which it implicitly sets when equal to measure the success.” Rather than branching (which would be a mispredict), the flag is materialized as a value with setcc.
lock xaddl is “exchange and add”: it atomically adds the source to the destination and returns the old value. One instruction, no loop, no retry.

Note the memory clobbers: "memory", "cc". The memory clobber tells the compiler that the asm may read or write arbitrary memory, so it cannot keep anything in a register across it; the cc clobber tells it that the condition codes were modified.

The API contract

 c
 
 
// src/include/port/atomics.h
/*
 * pg_atomic_compare_exchange_u32 - CAS operation
 *
 * Atomically compare the current value of ptr with *expected and store newval
 * iff ptr and *expected have the same value. The current value of *ptr will
 * always be stored in *expected.
 *
 * Return true if values have been exchanged, false otherwise.
 *
 * Full barrier semantics.
 */

That third sentence — “The current value of *ptr will always be stored in *expected” — is the reason the CAS-loop implementations of the other operations are so short.

The CAS loops

Not every operation has a dedicated instruction on every architecture, so generic.h builds them out of the CAS primitive:

 c
 
 
// src/include/port/atomics/generic.h
#if !defined(PG_HAVE_ATOMIC_FETCH_AND_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_FETCH_AND_U32
static inline uint32
pg_atomic_fetch_and_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 and_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, old & and_))
		/* skip */;
	return old;
}
#endif

#if !defined(PG_HAVE_ATOMIC_FETCH_OR_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_FETCH_OR_U32
static inline uint32
pg_atomic_fetch_or_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 or_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, old | or_))
		/* skip */;
	return old;
}
#endif

The neat part is that the retry needs no extra load. A failed cmpxchg writes the current value of *ptr back into *expected, which is the same variable the loop uses as its source of truth. So after a failure, old already holds the fresh value, and the next iteration computes old & and_ (or old | or_) from it. The comment /* ok if read is not atomic */ on the initial load acknowledges the one non-atomic read in the function: it is only a starting guess, and the CAS loop corrects it immediately.

The same pattern gives you fetch_sub, exchange, and add_fetch:

 c
 
 
// src/include/port/atomics/generic.h
#if !defined(PG_HAVE_ATOMIC_EXCHANGE_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_EXCHANGE_U32
static inline uint32
pg_atomic_exchange_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 xchg_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, xchg_))
		/* skip */;
	return old;
}
#endif

Memory barriers

 c
 
 
// src/include/port/atomics/arch-x86.h
/*
 * Both 32 and 64 bit x86 do not allow loads to be reordered with other loads,
 * or stores to be reordered with other stores, but a load can be performed
 * before a subsequent store.
 *
 * Technically, some x86-ish chips support uncached memory access and/or
 * special instructions that are weakly ordered. In those cases we'd need
 * the read and write barriers to be lfence and sfence. But since we don't
 * do those things, a compiler barrier should be enough.
 *
 * "lock; addl" has worked for longer than "mfence". It's also rumored to be
 * faster in many scenarios.
 */
#if defined(__GNUC__) || defined(__INTEL_COMPILER)
#if defined(__i386__)
#define pg_memory_barrier_impl()		\
	__asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory", "cc")
#elif defined(__x86_64__)
#define pg_memory_barrier_impl()		\
	__asm__ __volatile__ ("lock; addl $0,0(%%rsp)" : : : "memory", "cc")
#endif
#endif /* defined(__GNUC__) || defined(__INTEL_COMPILER) */

#define pg_read_barrier_impl()		pg_compiler_barrier_impl()
#define pg_write_barrier_impl()		pg_compiler_barrier_impl()

lock; addl $0, 0(%rsp) is the classic x86 full barrier: it adds zero to a stack location, but the lock prefix makes it an atomic read-modify-write, which is a full memory barrier. And on x86, a read or write barrier is just a compiler barrier, because the hardware’s memory model is strong enough that you never need more than that.

The 64-bit fallback

On a 32-bit platform without a 64-bit CAS instruction, fallback.h provides an emulated 64-bit atomic — and the emulation is, of course, a spinlock:

 c
 
 
// src/include/port/atomics/fallback.h
/*
 * fallback.h
 * Fallback for platforms without 64 bit atomics support. Slower
 * than native atomics support, but not unusably slow.
 */

#if !defined(PG_HAVE_ATOMIC_U64_SUPPORT)

#define PG_HAVE_ATOMIC_U64_SIMULATION

#define PG_HAVE_ATOMIC_U64_SUPPORT
typedef struct pg_atomic_uint64
{
	int			sema;
	volatile uint64 value;
} pg_atomic_uint64;

And the implementation, in src/backend/port/atomics.c:

 c
 
 
// src/backend/port/atomics.c
bool
pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
									uint64 *expected, uint64 newval)
{
	bool		ret;

	/*
	 * Do atomic op under a spinlock. It might look like we could just skip
	 * the cmpxchg if the lock isn't available, but that'd just emulate a
	 * 'weak' compare and swap. I.e. one that allows spurious failures. Since
	 * several algorithms rely on a strong variant and that is efficiently
	 * implementable on most major architectures let's emulate it here as
	 * well.
	 */
	SpinLockAcquire((slock_t *) &ptr->sema);

	/* perform compare/exchange logic */
	ret = ptr->value == *expected;
	*expected = ptr->value;
	if (ret)
		ptr->value = newval;

	/* and release lock */
	SpinLockRelease((slock_t *) &ptr->sema);

	return ret;
}

The comment is a nice piece of engineering reasoning: a lock-free implementation would be tempted to skip the compare-and-swap when the lock is contended, but that would be a weak CAS — one that may fail spuriously. Several algorithms in PostgreSQL depend on the strong variant, so the fallback implements the strong one, at the cost of spinning.

Note that this is a recursive layering: the 64-bit atomic’s implementation uses the 32-bit spinlock, which uses S_LOCK, which uses perform_spin_delay(). The whole ladder is recursive.

Layer 3: the LWLock

An LWLock is a shared-memory reader/writer lock. It is “lightweight” compared to the heavyweight lock manager in the sense that it has no owner, no lock table, no deadlock detection, and no interaction with pg_locks.

The state word

Everything about an LWLock’s state lives in one 32-bit word:

 c
 
 
// src/include/storage/lmgr/lwlock.h
typedef struct LWLock
{
	uint16		tranche;		/* tranche ID */
	pg_atomic_uint32 state;		/* state of exclusive/nonexclusive lockers */
	proclist_head waiters;		/* list of waiting PGPROCs */
#ifdef LOCK_DEBUG
	pg_atomic_uint32 nwaiters;	/* number of waiters */
	struct PGPROC *owner;		/* last exclusive owner of the lock */
#endif
} LWLock;

And the layout of that word:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
#define LW_FLAG_HAS_WAITERS			((uint32) 1 << 31)
#define LW_FLAG_WAKE_IN_PROGRESS	((uint32) 1 << 30)
#define LW_FLAG_LOCKED				((uint32) 1 << 29)
#define LW_FLAG_BITS				3
#define LW_FLAG_MASK				(((1<<LW_FLAG_BITS)-1)<<(32-LW_FLAG_BITS))

/* assumes MAX_BACKENDS is a (power of 2) - 1, checked below */
#define LW_VAL_EXCLUSIVE			(MAX_BACKENDS + 1)
#define LW_VAL_SHARED				1

/* already (power of 2)-1, i.e. suitable for a mask */
#define LW_SHARED_MASK				MAX_BACKENDS
#define LW_LOCK_MASK				(MAX_BACKENDS | LW_VAL_EXCLUSIVE)

Three flags in the top three bits, and the bottom bits hold the holder count:

Bits	Meaning
31	`LW_FLAG_HAS_WAITERS` — somebody is queued
30	`LW_FLAG_WAKE_IN_PROGRESS` — a wakeup is in flight
29	`LW_FLAG_LOCKED` — the wait list is locked
28–0	shared holder count, or `LW_VAL_EXCLUSIVE` for an exclusive holder

The scheme is described in the file header:

The basic idea is to have a single atomic variable ’lockcount’ instead of the formerly separate shared and exclusive counters and to use atomic operations to acquire the lock. That’s fairly easy to do for plain rw-spinlocks, but a lot harder for something like LWLocks that want to wait in the OS.

For lock acquisition we use an atomic compare-and-exchange on the lockcount variable. For exclusive lock we swap in a sentinel value (LW_VAL_EXCLUSIVE), for shared locks we count the number of holders.

To release the lock we use an atomic decrement to release the lock. If the new value is zero (we get that atomically), we know we can/have to release waiters.

LW_VAL_EXCLUSIVE is MAX_BACKENDS + 1, chosen so that it cannot be confused with a shared holder count — there can never be more than MAX_BACKENDS shared holders, so the two ranges do not overlap. The static assertions check exactly that:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
StaticAssertDecl(((MAX_BACKENDS + 1) & MAX_BACKENDS) == 0,
				 "MAX_BACKENDS + 1 needs to be a power of 2");

StaticAssertDecl((MAX_BACKENDS & LW_FLAG_MASK) == 0,
				 "MAX_BACKENDS and LW_FLAG_MASK overlap");

StaticAssertDecl((LW_VAL_EXCLUSIVE & LW_FLAG_MASK) == 0,
				 "LW_VAL_EXCLUSIVE and LW_FLAG_MASK overlap");

The attempt

 c
 
 
// src/backend/storage/lmgr/lwlock.c
/*
 * Internal function that tries to atomically acquire the lwlock in the passed
 * in mode.
 *
 * This function will not block waiting for a lock to become free - that's the
 * caller's job.
 *
 * Returns true if the lock isn't free and we need to wait.
 */
static bool
LWLockAttemptLock(LWLock *lock, LWLockMode mode)
{
	uint32		old_state;

	Assert(mode == LW_EXCLUSIVE || mode == LW_SHARED);

	/*
	 * Read once outside the loop, later iterations will get the newer value
	 * via compare & exchange.
	 */
	old_state = pg_atomic_read_u32(&lock->state);

	/* loop until we've determined whether we could acquire the lock or not */
	while (true)
	{
		uint32		desired_state;
		bool		lock_free;

		desired_state = old_state;

		if (mode == LW_EXCLUSIVE)
		{
			lock_free = (old_state & LW_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += LW_VAL_EXCLUSIVE;
		}
		else
		{
			lock_free = (old_state & LW_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += LW_VAL_SHARED;
		}

		/*
		 * Attempt to swap in the state we are expecting. If we didn't see
		 * lock to be free, that's just the old value. If we saw it as free,
		 * we'll attempt to mark it acquired. The reason that we always swap
		 * in the value is that this doubles as a memory barrier. We could try
		 * to be smarter and only swap in values if we saw the lock as free,
		 * but benchmark haven't shown it as beneficial so far.
		 *
		 * Retry if the value changed since we last looked at it.
		 */
		if (pg_atomic_compare_exchange_u32(&lock->state,
										 &old_state, desired_state))
		{
			if (lock_free)
			{
				/* Great! Got the lock. */
#ifdef LOCK_DEBUG
				if (mode == LW_EXCLUSIVE)
					lock->owner = MyProc;
#endif
				return false;
			}
			else
				return true;	/* somebody else has the lock */
		}
	}
	pg_unreachable();
}

The comment about the memory barrier is the interesting part:

The reason that we always swap in the value is that this doubles as a memory barrier. We could try to be smarter and only swap in values if we saw the lock as free, but benchmark haven’t shown it as beneficial so far.

On x86, lock cmpxchg is a full barrier whatever it does. So whether the CAS succeeds or fails, the CPU has executed a full fence. The code takes advantage of that unconditionally: it always issues the CAS, even in the case where it already knows the lock is held and it is going to have to wait anyway. The alternative — checking first and skipping the CAS — would save an atomic operation on the contended path, but benchmarks said it wasn’t worth it. (And on a weak-memory architecture, the unconditional CAS is what gives you the barrier for free; skipping it would require an explicit fence.)

Also note the loop structure: old_state is refreshed by the failed CAS (the “current value of *ptr will always be stored in *expected” contract), so the retry always operates on the freshest value.

The wait list lock is a bit in the same word

The LW_FLAG_LOCKED bit in the state word is a spinlock protecting the waiters list. This is a neat bit of economy: the wait list lock lives in the same cache line as the state word, so taking it does not cost an extra cache miss.

 c
 
 
// src/backend/storage/lmgr/lwlock.c
/*
 * Lock the LWLock's wait list against concurrent activity.
 *
 * NB: even though the wait list is locked, non-conflicting lock operations
 * may still happen concurrently.
 *
 * Time spent holding mutex should be short!
 */
static void
LWLockWaitListLock(LWLock *lock)
{
	uint32		old_state;
#ifdef LWLOCK_STATS
	lwlock_stats *lwstats;
	uint32		delays = 0;

	lwstats = get_lwlock_stats_entry(lock);
#endif

	while (true)
	{
		/*
		 * Always try once to acquire the lock directly, without setting up
		 * the spin-delay infrastructure. The work necessary for that shows up
		 * in profiles and is rarely necessary.
		 */
		old_state = pg_atomic_fetch_or_u32(&lock->state, LW_FLAG_LOCKED);
		if (likely(!(old_state & LW_FLAG_LOCKED)))
			break;				/* got lock */

		/* and then spin without atomic operations until lock is released */
		{
			SpinDelayStatus delayStatus;

			init_local_spin_delay(&delayStatus);

			while (old_state & LW_FLAG_LOCKED)
			{
				perform_spin_delay(&delayStatus);
				old_state = pg_atomic_read_u32(&lock->state);
			}
#ifdef LWLOCK_STATS
			delays += delayStatus.delays;
#endif
			finish_spin_delay(&delayStatus);
		}

		/*
		 * Retry. The lock might obviously already be re-acquired by the time
		 * we're attempting to get it again.
		 */
	}

#ifdef LWLOCK_STATS
	lwstats->spin_delay_count += delays;
#endif
}

Two things to notice:

“Always try once to acquire the lock directly, without setting up the spin-delay infrastructure.” The SpinDelayStatus struct has to be initialised, and init_local_spin_delay() writes five fields. For the overwhelmingly common case where the wait list is not contended, that setup is pure overhead — and the comment says it “shows up in profiles”. So the code tries the atomic fetch-or first and only sets up the spin machinery if that fails.
“spin without atomic operations until lock is released” — inside the spin loop, the state word is read with a plain atomic read (pg_atomic_read_u32), not a CAS loop. There is no attempt to take the lock during the spin; the spinner just watches the bit. Only when the bit clears does it retry the fetch-or. That avoids hammering the cache line with atomic RMWs.

Queue-self-then-retry

Now the interesting algorithmic part. The file header describes the two-phase approach:

The attentive reader might have noticed that naively doing the above has a glaring race condition: We try to lock using the atomic operations and notice that we have to wait. Unfortunately by the time we have finished queuing, the former locker very well might have already finished its work. That’s problematic because we’re now stuck waiting inside the OS.

To mitigate those races we use a two phased attempt at locking: Phase 1: Try to do it atomically, if we succeed, nice Phase 2: Add ourselves to the waitqueue of the lock Phase 3: Try to grab the lock again, if we succeed, remove ourselves from the queue Phase 4: Sleep till wake-up, goto Phase 1

And the implementation:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
	for (;;)
	{
		bool		mustwait;

		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = LWLockAttemptLock(lock, mode);

		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "immediately acquired lock");
			break;				/* got the lock */
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		LWLockQueueSelf(lock, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = LWLockAttemptLock(lock, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "acquired, undoing queue");

			LWLockDequeueSelf(lock);
			break;
		}

		/*
		 * Wait until awakened.
		 *
		 * It is possible that we get awakened for a reason other than being
		 * signaled by LWLockRelease. If so, loop back and wait again. Once
		 * we've gotten the LWLock, re-increment the sema by the number of
		 * additional signals received.
		 */
		LOG_LWDEBUG("LWLockAcquire", lock, "waiting");

		LWLockReportWaitStart(lock);
		...
		for (;;)
		{
			PGSemaphoreLock(proc->sem);
			if (proc->lwWaiting == LW_WS_NOT_WAITING)
				break;
			extraWaits++;
		}

		/* Retrying, allow LWLockRelease to release waiters again. */
		pg_atomic_fetch_and_u32(&lock->state, ~LW_FLAG_WAKE_IN_PROGRESS);

		...
		/* Now loop back and try to acquire lock again. */
		result = false;
	}

The critical insight is in the comment: queue first, then try again. If you tried the lock, failed, and then queued, the holder could have released the lock in the window between your failed attempt and your queueing — and nobody would ever wake you up, because the release had already happened. By queueing before the second attempt, you guarantee that any release that happens after your first failed attempt will see you on the wait list and wake you.

The queueing itself:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
static void
LWLockQueueSelf(LWLock *lock, LWLockMode mode)
{
	/*
	 * If we don't have a PGPROC structure, there's no way to wait. This
	 * should never occur, since MyProc should only be null during shared
	 * memory initialization.
	 */
	if (MyProc == NULL)
		elog(PANIC, "cannot wait without a PGPROC structure");

	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

	LWLockWaitListLock(lock);

	/* setting the flag is protected by the spinlock */
	pg_atomic_fetch_or_u32(&lock->state, LW_FLAG_HAS_WAITERS);

	MyProc->lwWaiting = LW_WS_WAITING;
	MyProc->lwWaitMode = mode;

	/* LW_WAIT_UNTIL_FREE waiters are always at the front of the queue */
	if (mode == LW_WAIT_UNTIL_FREE)
		proclist_push_head(&lock->waiters, MyProcNumber, lwWaitLink);
	else
		proclist_push_tail(&lock->waiters, MyProcNumber, lwWaitLink);

	/* Can release the mutex now */
	LWLockWaitListUnlock(lock);
	...
}

Note the panic on double-queueing: elog(PANIC, "queueing for lock while waiting on another one"). An LWLock is a leaf lock — you may hold at most one while waiting for another.

Sleeping: a POSIX semaphore, not a futex

The actual blocking is PGSemaphoreLock(proc->sem) — a POSIX semaphore stored in shared memory, one per PGPROC:

 c
 
 
// src/backend/port/posix_sema.c
/*
 * PGSemaphoreLock
 *
 * Lock a semaphore (decrement count), blocking if count would be < 0
 */
void
PGSemaphoreLock(PGSemaphore sema)
{
	int			errStatus;

	/* See notes in sysv_sema.c's implementation of PGSemaphoreLock. */
	do
	{
		errStatus = sem_wait(PG_SEM_REF(sema));
	} while (errStatus < 0 && errno == EINTR);

	if (errStatus < 0)
		elog(FATAL, "sem_wait failed: %m");
}

The semaphore is padded to a cache line to avoid false sharing:

 c
 
 
// src/backend/port/posix_sema.c
typedef union SemTPadded
{
	sem_t		pgsem;
	char		pad[PG_CACHE_LINE_SIZE];
} SemTPadded;

There is no futex here. PostgreSQL does not implement the Linux futex fast path, so every LWLock wait that reaches the kernel costs a real sem_wait syscall, and every wakeup costs a sem_post. The semaphore implementations are chosen at configure time:

unnamed POSIX semaphores (sem_init/sem_wait), the default where available;
named POSIX semaphores (sem_open);
SysV semaphores (semop) as a fallback.

Releasing and waking

The release path wakes waiters, and it uses the LW_FLAG_WAKE_IN_PROGRESS bit to make sure only one waker is in flight at a time:

 c
 
 
// src/backend/storage/lmgr/lwlock.c
	/* unset required flags, and release lock, in one fell swoop */
	{
		uint32		old_state;
		uint32		desired_state;

		old_state = pg_atomic_read_u32(&lock->state);
		while (true)
		{
			desired_state = old_state;

			/* compute desired flags */

			if (new_wake_in_progress)
				desired_state |= LW_FLAG_WAKE_IN_PROGRESS;
			else
				desired_state &= ~LW_FLAG_WAKE_IN_PROGRESS;

			if (proclist_is_empty(&lock->waiters))
				desired_state &= ~LW_FLAG_HAS_WAITERS;

			desired_state &= ~LW_FLAG_LOCKED;	/* release lock */

			if (pg_atomic_compare_exchange_u32(&lock->state, &old_state,
											 desired_state))
				break;
		}
	}

	/* Awaken any waiters I removed from the queue. */
	proclist_foreach_modify(iter, &wakeup, lwWaitLink)
	{
		PGPROC	 *waiter = GetPGProcByNumber(iter.cur);

		LOG_LWDEBUG("LWLockRelease", lock, "release waiter");
		proclist_delete(&wakeup, iter.cur, lwWaitLink);

		/*
		 * Guarantee that lwWaiting being unset only becomes visible once the
		 * unlink from the link has completed. Otherwise the target backend
		 * could be woken up for other reason and enqueue for a new lock - if
		 * that happens before the list unlink happens, the list would end up
		 * being corrupted.
		 *
		 * The barrier pairs with the LWLockWaitListLock() when enqueuing for
		 * another lock.
		 */
		pg_write_barrier();
		waiter->lwWaiting = LW_WS_NOT_WAITING;
		PGSemaphoreUnlock(waiter->sem);
	}

Two details: the flags are updated with a single CAS so that the wait-list lock is released and the flags are cleared atomically together (otherwise a racing LWLockQueueSelf() could set LW_FLAG_HAS_WAITERS after the releaser checked the list and before it released the lock, and the flag would be stuck on forever); and the write barrier before clearing lwWaiting ensures that the unlink from the wait list is visible before the waiter can wake up and try to queue for another lock.

Deadlock detection: not here

LWLocks have no deadlock detection, and this is not an oversight — it is a design decision. An LWLock is a leaf lock: it protects a small piece of shared state, and the code that holds it does not wait on other locks while holding it. There is no wait-for graph to inspect.

Deadlock detection lives in the heavyweight lock manager instead, in src/backend/storage/lmgr/deadlock.c, which maintains a wait-for graph over the heavyweight locks:

DeadLockCheck – Checks for deadlocks for a given process

This code looks for deadlocks involving the given process. If any are found, we try to resolve them by rearranging the wait queues. If resolution is impossible, return DS_HARD_DEADLOCK — the process must then throw an error.

The practical consequence: if an LWLock deadlock happens, PostgreSQL does not detect it. The backends sit there until deadlock_timeout fires — but that timeout applies to lock waits that the lock manager knows about, not to arbitrary LWLock waits.

Layer 4: latches

A latch is not a lock. It is a flag in shared memory plus a way to wake up whoever is waiting on it. MyLatch is the one every backend has, and it is what WaitLatch() waits on.

SetLatch

 c
 
 
// src/backend/storage/ipc/latch.c
/*
 * Sets a latch and wakes up anyone waiting on it.
 *
 * This is cheap if the latch is already set, otherwise not so much.
 *
 * NB: when calling this in a signal handler, be sure to save and restore
 * errno around it. (That's standard practice in most signal handlers, of
 * course, but we used to omit it in handlers that only set a flag.)
 *
 * NB: this function is called from critical sections and signal handlers so
 * throwing an error is not a good idea.
 */
void
SetLatch(Latch *latch)
{
	...
	/*
	 * The memory barrier has to be placed here to ensure that any flag
	 * variables possibly changed by this process have been flushed to main
	 * memory, before we check/set is_set.
	 */
	pg_memory_barrier();

	/* Quick exit if already set */
	if (latch->is_set)
		return;

	latch->is_set = true;

	pg_memory_barrier();
	if (!latch->maybe_sleeping)
		return;

	...
	owner_pid = latch->owner_pid;
	if (owner_pid == 0)
		return;
	else if (owner_pid == MyProcPid)
		WakeupMyProc();
	else
		WakeupOtherProc(owner_pid);
	...
}

Four things, in order:

A full memory barrier before the test. Any flag variables the process has changed must be visible in main memory before we publish is_set. Otherwise a waiter could see is_set == true, wake up, and look at flags that still have stale values. This is the classic Dekker-style store-load ordering problem, and the barrier is what makes it safe.
“Quick exit if already set.” Setting an already-set latch is a no-op. This is the common case in a signal handler that fires repeatedly, and it means SetLatch() is cheap enough to call from anywhere.
A second memory barrier after setting is_set. This makes the store to is_set visible before we look at maybe_sleeping.
The maybe_sleeping test. This is the optimisation that matters: if nobody is waiting on the latch, do not signal anybody. Signalling is expensive (a syscall, or a write to the self-pipe), and the overwhelmingly common case for SetLatch() is “set the flag and carry on”. The maybe_sleeping flag is set by the waiter just before it blocks:
c
```
		// src/backend/storage/ipc/waiteventset.c
		if (set->latch && !set->latch->is_set)
		{
			/* about to sleep on a latch */
			set->latch->maybe_sleeping = true;
			pg_memory_barrier();
			/* and recheck */
		}

		if (set->latch && set->latch->is_set)
		{
			...
		}
```
The waiter sets maybe_sleeping, issues a barrier, and then re-checks is_set. That re-check is the other half of the protocol: if the latch was set in the window between the waiter’s check and its setting of maybe_sleeping, the re-check catches it and the waiter never sleeps at all. If the latch is set after the re-check, the setter will see maybe_sleeping == true and will signal. Between the two, no wakeup is lost.

Note the ordering of the two flags: the waiter publishes maybe_sleeping before the final check of is_set, and the setter publishes is_set before the final check of maybe_sleeping. That is the standard “store both, then check both” pattern, and it is why there are two barriers in SetLatch().

ResetLatch

 c
 
 
// src/backend/storage/ipc/latch.c
/*
 * Clear the latch. Calling WaitLatch after this will sleep, unless
 * the latch is set again before the WaitLatch call.
 */
void
ResetLatch(Latch *latch)
{
	/* Only the owner should reset the latch */
	Assert(latch->owner_pid == MyProcPid);
	Assert(latch->maybe_sleeping == false);

	latch->is_set = false;

	/*
	 * Ensure that the write to is_set gets flushed to main memory before we
	 * examine any flag variables. Otherwise a concurrent SetLatch might
	 * falsely conclude that it needn't signal us, even though we have missed
	 * seeing some flag updates that SetLatch was supposed to inform us of.
	 */
	pg_memory_barrier();
}

Symmetrically, the barrier after clearing is_set ensures that the clear is visible before the owner examines any flags it might use to decide what to do next. Without it, a concurrent SetLatch() could see the stale is_set == true, decide “already set, nothing to do”, and skip the signal — while the owner has already decided to go to sleep.

WaitLatch and the long-lived WaitEventSet

 c
 
 
// src/backend/storage/ipc/latch.c
/* A common WaitEventSet used to implement WaitLatch() */
static WaitEventSet *LatchWaitSet;

/* The positions of the latch and PM death events in LatchWaitSet */
#define LatchWaitSetLatchPos 0
#define LatchWaitSetPostmasterDeathPos 1

void
InitializeLatchWaitSet(void)
{
	...
	/* Set up the WaitEventSet used by WaitLatch(). */
	LatchWaitSet = CreateWaitEventSet(NULL, 2);
	latch_pos = AddWaitEventToSet(LatchWaitSet, WL_LATCH_SET, PGINVALID_SOCKET,
								 MyLatch, NULL);
	Assert(latch_pos == LatchWaitSetLatchPos);

	/*
	 * WaitLatch will modify this to WL_EXIT_ON_PM_DEATH or
	 * WL_POSTMASTER_DEATH on each call.
	 */
	if (IsUnderPostmaster)
	{
		latch_pos = AddWaitEventToSet(LatchWaitSet, WL_EXIT_ON_PM_DEATH,
									 PGINVALID_SOCKET, NULL, NULL);
		Assert(latch_pos == LatchWaitSetPostmasterDeathPos);
	}
}

One WaitEventSet, created once per backend, reused forever. Every WaitLatch() call does not create a new epoll set; it just modifies the existing one:

 c
 
 
// src/backend/storage/ipc/latch.c
int
WaitLatch(Latch *latch, int wakeEvents, long timeout,
		 uint32 wait_event_info)
{
	WaitEvent	event;

	/* Postmaster-managed callers must handle postmaster death somehow. */
	Assert(!IsUnderPostmaster ||
		 (wakeEvents & WL_EXIT_ON_PM_DEATH) ||
		 (wakeEvents & WL_POSTMASTER_DEATH));

	/*
	 * Some callers may have a latch other than MyLatch, or no latch at all,
	 * or want to handle postmaster death differently. It's cheap to assign
	 * those, so just do it every time.
	 */
	if (!(wakeEvents & WL_LATCH_SET))
		latch = NULL;
	ModifyWaitEvent(LatchWaitSet, LatchWaitSetLatchPos, WL_LATCH_SET, latch);

	if (IsUnderPostmaster)
		ModifyWaitEvent(LatchWaitSet, LatchWaitSetPostmasterDeathPos,
						(wakeEvents & (WL_EXIT_ON_PM_DEATH | WL_POSTMASTER_DEATH)),
						NULL);

	if (WaitEventSetWait(LatchWaitSet,
						 (wakeEvents & WL_TIMEOUT) ? timeout : -1,
						 &event, 1,
						 wait_event_info) == 0)
		return WL_TIMEOUT;
	else
		return event.events;
}

The comment “Some callers may have a latch other than MyLatch, or no latch at all, or want to handle postmaster death differently. It’s cheap to assign those, so just do it every time” is a nice micro-optimization philosophy: rather than maintaining several WaitEventSets and picking one, keep one and mutate it.

epoll and signalfd versus the classic self-pipe

waiteventset.c has three implementations of the same idea, and the file header explains the race each one has to solve:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
/*
 * The poll() implementation uses the so-called self-pipe trick to overcome the
 * race condition involved with poll() and setting a global flag in the signal
 * handler. When a latch is set and the current process is waiting for it, the
 * signal handler wakes up the poll() in WaitLatch by writing a byte to a pipe.
 * A signal by itself doesn't interrupt poll() on all platforms, and even on
 * platforms where it does, a signal that arrives just before the poll() call
 * does not prevent poll() from entering sleep. An incoming byte on a pipe
 * however reliably interrupts the sleep, and causes poll() to return
 * immediately even if the signal arrives before poll() begins.
 *
 * The epoll() implementation overcomes the race with a different technique: it
 * keeps SIGURG blocked and consumes from a signalfd() descriptor instead. We
 * don't need to register a signal handler or create our own self-pipe. We
 * assume that any system that has Linux epoll() also has Linux signalfd().
 *
 * The kqueue() implementation waits for SIGURG with EVFILT_SIGNAL.
 *
 * The Windows implementation uses Windows events that are inherited by all
 * postmaster child processes. There's no need for the self-pipe trick there.
 */

The choice is made at compile time:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
#if defined(WAIT_USE_EPOLL) || defined(WAIT_USE_POLL) || \
	defined(WAIT_USE_KQUEUE) || defined(WAIT_USE_WIN32)
/* don't overwrite manual choice */
#elif defined(HAVE_SYS_EPOLL_H)
#define WAIT_USE_EPOLL
#elif defined(HAVE_KQUEUE)
#define WAIT_USE_KQUEUE
#elif defined(HAVE_POLL)
#define WAIT_USE_POLL
#elif WIN32
#define WAIT_USE_WIN32
#else
#error "no wait set implementation available"
#endif

/*
 * By default, we use a self-pipe with poll() and a signalfd with epoll(), if
 * available. For testing the choice can also be manually specified.
 */
#if defined(WAIT_USE_POLL) || defined(WAIT_USE_EPOLL)
#if defined(WAIT_USE_SELF_PIPE) || defined(WAIT_USE_SIGNALFD)
/* don't overwrite manual choice */
#elif defined(WAIT_USE_EPOLL) && defined(HAVE_SYS_SIGNALFD_H)
#define WAIT_USE_SIGNALFD
#else
#define WAIT_USE_SELF_PIPE
#endif
#endif

So on Linux: epoll + signalfd. Elsewhere with poll(): poll + self-pipe.

The signalfd variant

 c
 
 
// src/backend/storage/ipc/waiteventset.c
#ifdef WAIT_USE_SIGNALFD
	sigset_t	signalfd_mask;

	...
	/* Block SIGURG, because we'll receive it through a signalfd. */
	sigaddset(&UnBlockSig, SIGURG);

	/* Set up the signalfd to receive SIGURG notifications. */
	sigemptyset(&signalfd_mask);
	sigaddset(&signalfd_mask, SIGURG);
	signal_fd = signalfd(-1, &signalfd_mask, SFD_NONBLOCK | SFD_CLOEXEC);
	if (signal_fd < 0)
		elog(FATAL, "signalfd() failed");
	ReserveExternalFD();
#endif

SIGURG is blocked in the process, so it never runs a handler; instead it becomes readable on the signalfd, and the fd is registered with epoll. When the latch is set, SetLatch() sends SIGURG to itself, the kernel marks the signalfd readable, and epoll_wait() returns.

 c
 
 
// src/backend/storage/ipc/waiteventset.c
	/* Sleep */
	rc = epoll_wait(set->epoll_fd, set->epoll_ret_events,
					Min(nevents, set->nevents_space), cur_timeout);

	/* Check return code */
	if (rc < 0)
	{
		/* EINTR is okay, otherwise complain */
		if (errno != EINTR)
		{
			waiting = false;
			ereport(ERROR,
					(errcode_for_socket_access(),
					 errmsg("%s() failed: %m",
							"epoll_wait")));
		}
		return 0;
	}

Note that epoll_wait is a level-triggered wait on the signalfd: the fd stays readable until the pending signals are consumed, which is what the drain() call does.

The self-pipe variant

The classic trick. The pipe is created with both ends non-blocking and close-on-exec:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
	/*
	 * Set up the self-pipe that allows a signal handler to wake up the
	 * poll()/epoll_wait() in WaitLatch. Make the write-end non-blocking, so
	 * that SetLatch won't block if the event has already been set many times
	 * filling the kernel buffer. Make the read-end non-blocking too, so that
	 * we can easily clear the pipe by reading until EAGAIN or EWOULDBLOCK.
	 * Also, make both FDs close-on-exec, since we surely do not want any
	 * child processes messing with them.
	 */
	if (pipe(pipefd) < 0)
		elog(FATAL, "pipe() failed: %m");
	if (fcntl(pipefd[0], F_SETFL, O_NONBLOCK) == -1)
		elog(FATAL, "fcntl(F_SETFL) failed on read-end of self-pipe: %m");
	if (fcntl(pipefd[1], F_SETFL, O_NONBLOCK) == -1)
		elog(FATAL, "fcntl(F_SETFL) failed on write-end of self-pipe: %m");
	if (fcntl(pipefd[0], F_SETFD, FD_CLOEXEC) == -1)
		elog(FATAL, "fcntl(F_SETFD) failed on read-end of self-pipe: %m");
	if (fcntl(pipefd[1], F_SETFD, FD_CLOEXEC) == -1)
		elog(FATAL, "fcntl(F_SETFD) failed on write-end of self-pipe: %m");

	selfpipe_readfd = pipefd[0];
	selfpipe_writefd = pipefd[1];
	selfpipe_owner_pid = MyProcPid;

And the signal handler:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
/*
 * SetLatch uses SIGURG to wake up the process waiting on the latch.
 *
 * Wake up WaitLatch, if we're waiting.
 */
static void
latch_sigurg_handler(SIGNAL_ARGS)
{
	if (waiting)
		sendSelfPipeByte();
}

/* Send one byte to the self-pipe, to wake up WaitLatch */
static void
sendSelfPipeByte(void)
{
	ssize_t		rc;
	char		dummy = 0;

retry:
	rc = write(selfpipe_writefd, &dummy, 1);
	if (rc < 0)
	{
		/* If interrupted by signal, just retry */
		if (errno == EINTR)
			goto retry;

		/*
		 * If the pipe is full, we don't need to retry, the data that's there
		 * already is enough to wake up WaitLatch.
		 */
		if (errno == EAGAIN || errno == EWOULDBLOCK)
			return;

		/*
		 * Oops, the write() failed for some other reason. We might be in a
		 * signal handler, so it's not safe to elog(). We have no choice but
		 * silently ignore the error.
		 */
		return;
	}
}

Every detail here is load-bearing:

One byte, not a message. The byte’s value is irrelevant; only its presence matters.
The write end is non-blocking, and EAGAIN is treated as success. If the pipe’s kernel buffer is full, there is already a byte pending, and a pending byte is exactly as good as a new one: the reader will wake up anyway. Retrying would be pointless (the buffer is full!) and blocking would be fatal (we might be in a signal handler, where blocking is not allowed).
EINTR is retried, because a signal can interrupt the write.
Any other error is silently ignored, because we may be in a signal handler and elog() is not safe there.

The waiting flag is what keeps the handler cheap when nobody is waiting:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
#ifndef WIN32
/* Are we currently in WaitLatch? The signal handler would like to know. */
static volatile sig_atomic_t waiting = false;
#endif

volatile sig_atomic_t is the only kind of variable a signal handler may safely touch.

And the drain:

 c
 
 
// src/backend/storage/ipc/waiteventset.c
/*
 * Read all available data from self-pipe or signalfd.
 * ...
 */
static void
drain(void)
{
	char		buf[1024];
	ssize_t		rc;
	int			fd;

#ifdef WAIT_USE_SELF_PIPE
	fd = selfpipe_readfd;
#else
	fd = signal_fd;
#endif

	for (;;)
	{
		rc = read(fd, buf, sizeof(buf));
		if (rc < 0)
		{
			if (errno == EAGAIN || errno == EWOULDBLOCK)
				break;			/* the descriptor is empty */
			else if (errno == EINTR)
				continue;		/* retry */
			else
			{
				waiting = false;
				...
				elog(ERROR, "read() on self-pipe failed: %m");
			}
		}
		else if (rc == 0)
		{
			...
		}
		else if (rc < sizeof(buf))
		{
			/* we successfully drained the pipe; no need to read() again */
			break;
		}
		/* else buffer wasn't big enough, so read again */
	}
}

The same function drains a signalfd or a self-pipe, which is why the code is shared between the two variants.

The whole picture

text

contended lock
 │
 ├─ spinlock ──── TAS_SPIN (free test + lock xchgb)
 │ └── SPIN_DELAY(): rep; nop / isb
 │ └── pg_usleep(cur_delay), cur_delay *= rand(1,2)
 │ └── after NUM_DELAYS: PANIC "stuck spinlock"
 │
 ├─ LWLock ────── CAS on the state word
 │ └── queue self on the wait list, try again
 │ └── PGSemaphoreLock (sem_wait) — kernel
 │
 └─ Latch ─────── maybe_sleeping test
 └── epoll_wait (signalfd) or poll (self-pipe) — kernel

What it all means in practice

Spinning burns CPU, and it is visible. If a backend is spinning, pg_stat_activity will show it with no wait event, and top will show it at 100% of a core. If it has fallen asleep in the spin delay, the wait event is SpinDelay. Both are symptoms of the same thing: somebody is holding a lock that a hot path needs.

The adaptive spin count is deliberately coarse, and that is fine. The estimate moves by +100 or −1 per acquisition locally, and by 1/16 of the difference per backend exit globally. It is not trying to be precise; it is trying to be stable. A backend that spins 100 times when 10 would do wastes 90 cache-coherence round trips; a backend that spins 10 times when 100 would do pays a pg_usleep() syscall. The asymmetry of the local adjustment (+100/−1) says that erring on the side of spinning is preferred, because spinning does not involve the kernel.

The global estimate only updates when a backend exits. It lives in ProcGlobal->spins_per_delay and is folded in at ProcKill() time. A long-lived backend with a short-lived one can therefore be running with a stale estimate for a long time. This is deliberate: the adjustment is cheap and the noise would be worse than the staleness.

PostgreSQL does not use futexes. Every LWLock wait that reaches the kernel costs a sem_wait syscall, and every wakeup costs a sem_post. On a machine with high LWLock contention, that syscall traffic shows up in strace counts and in perf samples of sem_wait. If you are chasing microseconds of lock overhead, this is where they go.

LWLocks have no deadlock detection. If two LWLocks could deadlock each other, PostgreSQL would not notice and would not report it. The heavyweight lock manager (deadlock.c) does the deadlock detection, and it only knows about heavyweight locks. The practical rule: never acquire an LWLock while holding another one in a way that could cycle. The elog(PANIC, "queueing for lock while waiting on another one") in LWLockQueueSelf() enforces the weaker property that you cannot wait for an LWLock while already waiting for one.

The wait event tells you which rung you are on. pg_stat_activity distinguishes LWLock (waiting for an LWLock), SpinDelay (sleeping inside a spinlock wait), and the latch/socket wait events. If a query is “stuck” and you see LWLock: BufferMapping for a long time, you have a buffer contention problem; if you see SpinDelay, you have a spinlock problem, which usually means something else is wrong.

SetLatch() is cheap on purpose. The maybe_sleeping test means that setting a latch nobody is waiting on costs one memory barrier and a predictable branch. This matters because latches are set on hot paths — every CHECK_FOR_INTERRUPTS(), every lock acquisition that might need to be interrupted — and a syscall there would be unacceptable.

The self-pipe trick exists because signals are unreliable as wakeups. A signal that arrives just before poll() is entered does not prevent poll() from sleeping. A byte in a pipe does. This is why the pipe write end is non-blocking and why EAGAIN is a success: the wakeup mechanism has to be safe to call from a signal handler, where blocking is forbidden and where throwing an error would be fatal.

The 64-bit atomic fallback is a spinlock. On a 32-bit platform without 64-bit CAS, pg_atomic_uint64 carries an embedded spinlock and every 64-bit atomic operation is a spinlock acquisition. That is “not unusably slow” per the header comment, but it does mean that 64-bit atomics are a scalability cliff on such platforms.

Two orders of magnitude separate “spin” from “sleep”. A spinlock acquisition that succeeds costs tens of nanoseconds; one that has to sleep costs at least a microsecond, usually much more. Most performance problems attributed to “lock contention” are really problems with the duration the lock is held, not with the waiting mechanism. Shortening the critical section is worth far more than tuning the spin count.

PostgreSQL Locks Buffers Without LWLocks

Jay Hubbert — Fri, 02 Oct 2026 09:00:00 +0000

Every time PostgreSQL reads a page it does two things: it pins the shared buffer that holds the page, and it takes a content lock on that buffer so that nobody rewrites the bytes underneath it while it is being read.

The second of those used to be an LWLock — a general-purpose, one-word-at-a-time lightweight lock living in a global array in shared memory. PostgreSQL 20 changed that. Buffer content locks are now implemented entirely inside a single 64-bit atomic word that already existed in the buffer descriptor, with its own three-level lock hierarchy, its own wait queue, and its own wakeup logic.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

What a content lock protects

Let’s be precise about what is being locked, because PostgreSQL has two different “buffer locks” and they are easy to confuse.

The buffer header lock protects the descriptor — the metadata in BufferDesc: the tag, the validity flags, the I/O state. It is a very short critical section, taken for a handful of instructions at a time.

The buffer content lock protects the page data — the 8 kB of bytes in the buffer. It is held for as long as the caller is looking at the page. This is the one that matters for correctness: two backends cannot be reading the same page while one of them is rewriting it.

The header comment in src/include/storage/buf_internals.h is explicit about the division:

The content of buffers is protected via the buffer content lock, implemented as part of the buffer state. Note that the buffer header lock is not used to control access to the data in the buffer! We used to use an LWLock to implement the content lock, but having a dedicated implementation of content locks allows us to implement some otherwise hard things (e.g. race-freely checking if AIO is in progress before locking a buffer exclusively) and enables otherwise impossible optimizations (e.g. unlocking and unpinning a buffer in one atomic operation).

That comment is the entire design rationale in one paragraph, and it is worth reading twice. The dedicated implementation exists to make two specific things possible: racing safely against asynchronous I/O, and fusing lock release with unpinning.

The old world: an LWLock per buffer

Before the change, BufferDesc had a lock in it:

 c
 
 
	/*
	 * Backend of pin-count waiter. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	int			wait_backend_pgprocno;

	PgAioWaitRef io_wref;		/* set iff AIO is in progress */
	LWLock		content_lock;	/* to lock access to buffer contents */
} BufferDesc;

and the accessor was a one-liner:

 c
 
 
static inline LWLock *
BufferDescriptorGetContentLock(const BufferDesc *bdesc)
{
	return (LWLock *) (&bdesc->content_lock);
}

An LWLock is a 16-byte structure with a uint32 state, a pg_atomic_uint32 nwaiters, a proclist_head waiters, and a bool is_shared. Taking one is a CAS on the state word; if that fails, you get pushed onto the wait queue in the same structure. Releasing it is another CAS plus, possibly, a walk of the wait queue.

The problem is not the algorithm — the LWLock algorithm is good. The problem is that there are now two independent atomic words per buffer: the LWLock’s state, and the BufferDesc.state word that holds the refcount and the flags. Anything that needs to reason about both at once has to do two separate atomic operations, with a window in between.

Concretely, the old code path for a shared content lock was:

 c
 
	LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);

and for an exclusive one:

 c
 
	LWLockAcquire(BufferDescriptorGetContentLock(desc), LW_EXCLUSIVE);

That is a full generic lock acquisition: read the lock’s state, decide whether it conflicts, CAS, and — if it conflicts — walk into lwlock.c’s queueing machinery.

The new world: one 64-bit word

The change is commit fcb9c977aa5, “bufmgr: Implement buffer content locks independently of lwlocks”, by Andres Freund, January 2026. The commit message lays out the four reasons:

Hint bits are currently set while holding only a share lock. This leads to having to copy pages while they are being written out if checksums are enabled, which is not cheap. We would like to add AIO writes, however once many buffers can be written out at the same time, it gets a lot more expensive to copy them, particularly because that copy needs to reside in shared buffers (for worker mode to have access to the buffer).

In addition, modifying buffers while they are being written out can cause issues with unbuffered/direct-IO, as some filesystems (like btrfs) do not like that, due to filesystem internal checksums getting corrupted.

The solution to this is to require a new share-exclusive lock-level to set hint bits and to write out buffers, making those operations mutually exclusive. We could introduce such a lock-level into the generic lwlock implementation, however it does not look like there would be other users, and it does add some overhead into important code paths.

For AIO writes we need to be able to race-freely check whether a buffer is undergoing IO and whether an exclusive lock on the page can be acquired. That is rather hard to do efficiently when the buffer state and the lock state are separate atomic variables. This is a major hindrance to allowing writes to be done asynchronously.

Buffer locks are by far the most frequently taken locks. Optimizing them specifically for their use case is worth the effort. E.g. by merging content locks into buffer locks we will be able to release a buffer lock and pin in one atomic operation.

There are more complicated optimizations, like long-lived “super pinned & locked” pages, that cannot realistically be implemented with the generic lwlock implementation.

Point 3 is the headline: buffer content locks are, by a wide margin, the most frequently taken lock in the system. Every ReadBuffer() pins and locks; every index scan does it once per page; every page inspection does it again.

The layout

The state word was already 64 bits wide before this change, but only half of it was used. The old header comment read:

 c
 
 
/*
 * Buffer state is a single 64-bit variable where following data is combined.
 *
 * State of the buffer itself (in order):
 * - 18 bits refcount
 * - 4 bits usage count
 * - 10 bits of flags
 *
 * Combining these values allows to perform some operations without locking
 * the buffer header, by modifying them together with a CAS loop.
 *
 * NB: A future commit will use a significant portion of the remaining bits to
 * implement buffer locking as part of the state variable.
 *
 * The definition of buffer state components is below.
 */
#define BUF_REFCOUNT_BITS 18
#define BUF_USAGECOUNT_BITS 4
#define BUF_FLAG_BITS 10

StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32,
				 "parts of buffer state space need to equal 32");

The new one (src/include/storage/buf_internals.h):

 c
 
 
/*
 * Buffer state is a single 64-bit variable where following data is combined.
 *
 * State of the buffer itself (in order):
 * - 18 bits refcount
 * - 4 bits usage count
 * - 12 bits of flags
 * - 18 bits share-lock count
 * - 1 bit share-exclusive locked
 * - 1 bit exclusive locked
 *
 * Combining these values allows to perform some operations without locking
 * the buffer header, by modifying them together with a CAS loop.
 *
 * The definition of buffer state components is below.
 */
#define BUF_REFCOUNT_BITS 18
#define BUF_USAGECOUNT_BITS 4
#define BUF_FLAG_BITS 12
#define BUF_LOCK_BITS (18+2)

StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS + BUF_LOCK_BITS <= 64,
				 "parts of buffer state space need to be <= 64");

So the word went from 32 bits used to 54 bits used, and the invariant changed from “exactly 32” to “at most 64”. The two extra flag bits are the two new lock-related flags; the 20 new lock bits are the lock state.

Here is the whole layout, on a 64-bit machine, with MAX_BACKENDS_BITS == 18:

Bits	Width	Field	Meaning
0–17	18	refcount	number of pins on this buffer
18–21	4	usage count	clock-sweep second-chance counter
22–33	12	flags	`BM_*` bits, see below
34–51	18	share-lock count	number of backends holding a share lock
52	1	share-exclusive	one backend holds a share-exclusive lock
53	1	exclusive	one backend holds an exclusive lock
54–63	10	unused

The shift constants:

 c
 
 
/* refcount related definitions */
#define BUF_REFCOUNT_ONE 1
#define BUF_REFCOUNT_MASK \
	((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1)

/* usage count related definitions */
#define BUF_USAGECOUNT_SHIFT \
	BUF_REFCOUNT_BITS
#define BUF_USAGECOUNT_MASK \
	(((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT))
#define BUF_USAGECOUNT_ONE \
	(UINT64CONST(1) << BUF_REFCOUNT_BITS)

/* flags related definitions */
#define BUF_FLAG_SHIFT \
	(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)
#define BUF_FLAG_MASK \
	(((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT)

and the lock bits:

 c
 
 
/* lock state related definitions */
#define BM_LOCK_SHIFT \
	(BUF_FLAG_SHIFT + BUF_FLAG_BITS)
#define BM_LOCK_VAL_SHARED \
	(UINT64CONST(1) << (BM_LOCK_SHIFT))
#define BM_LOCK_VAL_SHARE_EXCLUSIVE \
	(UINT64CONST(1) << (BM_LOCK_SHIFT + MAX_BACKENDS_BITS))
#define BM_LOCK_VAL_EXCLUSIVE \
	(UINT64CONST(1) << (BM_LOCK_SHIFT + MAX_BACKENDS_BITS + 1))
#define BM_LOCK_MASK \
	((((uint64) MAX_BACKENDS) << BM_LOCK_SHIFT) | BM_LOCK_VAL_SHARE_EXCLUSIVE | BM_LOCK_VAL_EXCLUSIVE)

MAX_BACKENDS_BITS comes from src/include/storage/procnumber.h:

 c
 
 
#define MAX_BACKENDS_BITS		18
#define MAX_BACKENDS			((1U << MAX_BACKENDS_BITS)-1)

Eighteen bits, which is exactly the number of lock bits available for the share-lock count. That is not a coincidence: it means the share-lock count can never exceed the number of possible backends, and the BM_LOCK_MASK expression (((uint64) MAX_BACKENDS) << BM_LOCK_SHIFT) covers exactly bits 34–51 — the whole share-lock field.

Two static assertions tie this together:

 c
 
 
StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS,
				 "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS");
StaticAssertDecl(MAX_BACKENDS_BITS <= (BUF_LOCK_BITS - 2),
				 "MAX_BACKENDS_BITS needs to be <= BUF_LOCK_BITS - 2");

The subtle part is that the “share lock” field is not a single bit — it is a count, stored in the low bits of the lock field. BM_LOCK_VAL_SHARED is the value 1 << 34, and acquiring a share lock adds it to the state word:

 c
 
 
		else
		{
			lock_free = (old_state & BM_LOCK_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARED;
		}

So desired_state += BM_LOCK_VAL_SHARED is really desired_state += 1 in the count field. The BM_LOCK_MASK then detects “is anybody holding any share lock” by testing all 18 count bits at once.

This is why the exclusive acquisition test is a mask against the whole lock field rather than a single bit:

 c
 
 
		if (mode == BUFFER_LOCK_EXCLUSIVE)
		{
			lock_free = (old_state & BM_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_EXCLUSIVE;
		}

BM_LOCK_MASK covers the share-lock count and both exclusive bits, so “the lock is free” means “no share lockers, no share-exclusive holder, no exclusive holder”.

The flag bits

The flags live in the middle of the word, and two of them are new:

 c
 
 
/* buffer header is locked */
#define BM_LOCKED					BUF_DEFINE_FLAG( 0)
/* data needs writing */
#define BM_DIRTY					BUF_DEFINE_FLAG( 1)
/* data is valid */
#define BM_VALID					BUF_DEFINE_FLAG( 2)
/* tag is assigned */
#define BM_TAG_VALID				BUF_DEFINE_FLAG( 3)
/* read or write in progress */
#define BM_IO_IN_PROGRESS			BUF_DEFINE_FLAG( 4)
/* previous I/O failed */
#define BM_IO_ERROR					BUF_DEFINE_FLAG( 5)
/* flag bit 6 is not used anymore */
/* have waiter for sole pin */
#define BM_PIN_COUNT_WAITER			BUF_DEFINE_FLAG( 7)
/* must write for checkpoint */
#define BM_CHECKPOINT_NEEDED		BUF_DEFINE_FLAG( 8)
/* permanent buffer (not unlogged, or init fork) */
#define BM_PERMANENT				BUF_DEFINE_FLAG( 9)
/* content lock has waiters */
#define BM_LOCK_HAS_WAITERS			BUF_DEFINE_FLAG(10)
/* waiter for content lock has been signalled but not yet run */
#define BM_LOCK_WAKE_IN_PROGRESS	BUF_DEFINE_FLAG(11)

BM_LOCK_HAS_WAITERS and BM_LOCK_WAKE_IN_PROGRESS are the content-lock equivalents of LW_FLAG_HAS_WAITERS and LW_FLAG_WAKE_IN_PROGRESS. They are in the flag field rather than the lock field because they are advisory hints, not part of the lock’s conflict state — they must not affect whether a lock can be acquired.

The descriptor

The BufferDesc itself changed shape. The embedded LWLock content_lock is gone; in its place is a wait list:

 c
 
 
typedef struct BufferDesc
{
	/*
	 * ID of page contained in buffer. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	BufferTag	tag;

	/*
	 * Buffer's index number (from 0). The field never changes after
	 * initialization, so does not need locking.
	 */
	int			buf_id;

	/*
	 * State of the buffer, containing flags, refcount and usagecount. See
	 * BUF_* and BM_* defines at the top of this file.
	 */
	pg_atomic_uint64 state;

	/*
	 * Backend of pin-count waiter. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	int			wait_backend_pgprocno;

	PgAioWaitRef io_wref;		/* set iff AIO is in progress */

	/*
	 * List of PGPROCs waiting for the buffer content lock. Protected by the
	 * buffer header spinlock.
	 */
	proclist_head lock_waiters;
} BufferDesc;

Note the proclist_head lock_waiters — a per-buffer intrusive wait list, analogous to the one inside an LWLock, but only 8 bytes instead of a whole LWLock’s worth of state.

Acquiring a content lock

The heart of it is BufferLockAttempt() in src/backend/storage/buffer/bufmgr.c. This is the analogue of LWLockAttemptLock():

 c
 
 
static inline bool
BufferLockAttempt(BufferDesc *buf_hdr, BufferLockMode mode)
{
	uint64		old_state;

	/*
	 * Read once outside the loop, later iterations will get the newer value
	 * via compare & exchange.
	 */
	old_state = pg_atomic_read_u64(&buf_hdr->state);

	/* loop until we've determined whether we could acquire the lock or not */
	while (true)
	{
		uint64		desired_state;
		bool		lock_free;

		desired_state = old_state;

		if (mode == BUFFER_LOCK_EXCLUSIVE)
		{
			lock_free = (old_state & BM_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_EXCLUSIVE;
		}
		else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			lock_free = (old_state & (BM_LOCK_VAL_EXCLUSIVE | BM_LOCK_VAL_SHARE_EXCLUSIVE)) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARE_EXCLUSIVE;
		}
		else
		{
			lock_free = (old_state & BM_LOCK_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARED;
		}

		/*
		 * Attempt to swap in the state we are expecting. If we didn't see
		 * lock to be free, that's just the old value. If we saw it as free,
		 * we'll attempt to mark it acquired. The reason that we always swap
		 * in the value is that this doubles as a memory barrier. We could try
		 * to be smarter and only swap in values if we saw the lock as free,
		 * but benchmark haven't shown it as beneficial so far.
		 *
		 * Retry if the value changed since we last looked at it.
		 */
		if (likely(pg_atomic_compare_exchange_u64(&buf_hdr->state,
												 &old_state, desired_state)))
		{
			if (lock_free)
			{
				/* Great! Got the lock. */
				return false;
			}
			else
				return true;	/* somebody else has the lock */
		}
	}

	pg_unreachable();
}

Read the three conflict rules off that:

Mode	Acquired if	Adds
`BUFFER_LOCK_SHARE`	no exclusive holder	`+1` to the share count
`BUFFER_LOCK_SHARE_EXCLUSIVE`	no exclusive and no share-exclusive holder	sets bit 52
`BUFFER_LOCK_EXCLUSIVE`	nothing held at all	sets bit 53

That is exactly the semantics the commit message describes: “An additional lock-level share-exclusive has been added. This lock-level conflicts with exclusive locks and itself, but not share locks.”

Two things are worth noting about the implementation.

First, the CAS always happens, even when the lock is known to be held. The comment explains why: the CAS doubles as a memory barrier, and the benchmark showed that conditionally skipping it did not help.

Second, the conflict test is a read of a shared word, not a read of a private lock structure. Because the lock state lives in the same word as the refcount and the flags, a concurrent pin or unpin is visible to the lock decision for free.

The queue-self-then-retry algorithm

If the first attempt fails, we cannot simply go to sleep: the lock might have been released in the meantime, and we would sleep on a lock nobody is going to signal. The classic solution — and the one lwlock.c uses — is to enqueue yourself and then try again. BufferLockAcquire() does exactly that:

 c
 
 
static inline void
BufferLockAcquire(Buffer buffer, BufferDesc *buf_hdr, BufferLockMode mode)
{
	PrivateRefCountEntry *entry;
	int			extraWaits = 0;

	/*
	 * Get reference to the refcount entry before we hold the lock, it seems
	 * better to do before holding the lock.
	 */
	entry = GetPrivateRefCountEntry(buffer, true);

	/*
	 * We better not already hold a lock on the buffer.
	 */
	Assert(entry->data.lockmode == BUFFER_LOCK_UNLOCK);

	/*
	 * Lock out cancel/die interrupts until we exit the code section protected
	 * by the content lock. This ensures that interrupts will not interfere
	 * with manipulations of data structures in shared memory.
	 */
	HOLD_INTERRUPTS();

	for (;;)
	{
		uint32		wait_event = 0; /* initialized to avoid compiler warning */
		bool		mustwait;

		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = BufferLockAttempt(buf_hdr, mode);

		if (likely(!mustwait))
		{
			break;
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		BufferLockQueueSelf(buf_hdr, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = BufferLockAttempt(buf_hdr, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			BufferLockDequeueSelf(buf_hdr);
			break;
		}

		switch (mode)
		{
			case BUFFER_LOCK_EXCLUSIVE:
				wait_event = WAIT_EVENT_BUFFER_EXCLUSIVE;
				break;
			case BUFFER_LOCK_SHARE_EXCLUSIVE:
				wait_event = WAIT_EVENT_BUFFER_SHARE_EXCLUSIVE;
				break;
			case BUFFER_LOCK_SHARE:
				wait_event = WAIT_EVENT_BUFFER_SHARED;
				break;
			case BUFFER_LOCK_UNLOCK:
				pg_unreachable();

		}
		pgstat_report_wait_start(wait_event);

		/*
		 * Wait until awakened.
		 *
		 * It is possible that we get awakened for a reason other than being
		 * signaled by BufferLockWakeup(). If so, loop back and wait again.
		 * Once we've gotten the lock, re-increment the sema by the number of
		 * additional signals received.
		 */
		for (;;)
		{
			PGSemaphoreLock(MyProc->sem);
			if (MyProc->lwWaiting == LW_WS_NOT_WAITING)
				break;
			extraWaits++;
		}

		pgstat_report_wait_end();

		/* Retrying, allow BufferLockReleaseSub to release waiters again. */
		pg_atomic_fetch_and_u64(&buf_hdr->state, ~BM_LOCK_WAKE_IN_PROGRESS);
	}

	/* Remember that we now hold this lock */
	entry->data.lockmode = mode;

	/*
	 * Fix the process wait semaphore's count for any absorbed wakeups.
	 */
	while (unlikely(extraWaits-- > 0))
		PGSemaphoreUnlock(MyProc->sem);
}

Compare this to LWLockAcquire() in src/backend/storage/lmgr/lwlock.c — the structure is identical, down to the comments:

 c
 
 
		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = LWLockAttemptLock(lock, mode);

		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "immediately acquired lock");
			break;				/* got the lock */
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		LWLockQueueSelf(lock, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = LWLockAttemptLock(lock, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "acquired, undoing queue");

			LWLockDequeueSelf(lock);
			break;
		}

The commit message is frank about this: “This commit unfortunately introduces some code that is very similar to the code in lwlock.c, however the code is not equivalent enough to easily merge it.”

One wart the commit message calls out explicitly: the new code still uses PGPROC->lwWaiting and PGPROC->lwWaitMode, rather than inventing a parallel set of fields:

 c
 
 
	/*
	 * These are currently used both for lwlocks and buffer content locks,
	 * which is acceptable, although not pretty, because a backend can't wait
	 * for both types of locks at the same time.
	 */
	MyProc->lwWaiting = LW_WS_WAITING;
	MyProc->lwWaitMode = mode;

	proclist_push_tail(&buf_hdr->lock_waiters, MyProcNumber, lwWaitLink);

A backend can only be blocked on one lock at a time, so reusing the fields is safe — but the fields are now lying about their name.

The public API

The lock modes are an enum in src/include/storage/bufmgr.h:

 c
 
 
/*
 * Buffer content lock modes (mode argument for LockBuffer())
 */
typedef enum BufferLockMode
{
	BUFFER_LOCK_UNLOCK,

	/*
	 * A share lock conflicts with exclusive locks.
	 */
	BUFFER_LOCK_SHARE,

	/*
	 * A share-exclusive lock conflicts with itself and exclusive locks.
	 */
	BUFFER_LOCK_SHARE_EXCLUSIVE,

	/*
	 * An exclusive lock conflicts with every other lock type.
	 */
	BUFFER_LOCK_EXCLUSIVE,
} BufferLockMode;

and there is a wrapper that keeps the mode argument a compile-time constant at the call site, because a runtime switch generated worse code:

 c
 
 
/*
 * Handling BUFFER_LOCK_UNLOCK in bufmgr.c leads to sufficiently worse branch
 * prediction to impact performance. Therefore handle that switch here, where
 * most of the time `mode` will be a constant and thus can be optimized out by
 * the compiler.
 */
static inline void
LockBuffer(Buffer buffer, BufferLockMode mode)
{
	if (mode == BUFFER_LOCK_UNLOCK)
		UnlockBuffer(buffer);
	else
		LockBufferInternal(buffer, mode);
}

LockBufferInternal() does the same trick one level down:

 c
 
 
	/*
	 * Test the most frequent lock modes first. While a switch (mode) would be
	 * nice, at least gcc generates considerably worse code for it.
	 *
	 * Call BufferLockAcquire() with a constant argument for mode, to generate
	 * more efficient code for the different lock modes.
	 */
	if (mode == BUFFER_LOCK_SHARE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_SHARE);
	else if (mode == BUFFER_LOCK_EXCLUSIVE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_EXCLUSIVE);
	else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_SHARE_EXCLUSIVE);
	else
		elog(ERROR, "unrecognized buffer lock mode: %d", mode);

Because BufferLockAcquire() is static inline and is called with a constant mode three different times, the compiler generates three specialised copies, each with the conflict test folded to a constant. A share lock’s conflict test becomes “is the exclusive bit set?” and nothing else.

Lock ownership tracking

Where lwlock.c keeps a global array of held locks per backend, the buffer content lock reuses the buffer manager’s existing private refcount hash:

 c
 
 
/*
 * This is separated out from PrivateRefCountEntry to allow for copying all
 * the data members via struct assignment.
 */
typedef struct PrivateRefCountData
{
	/*
	 * How many times has the buffer been pinned by this backend.
	 */
	int32		refcount;

	/*
	 * Is the buffer locked by this backend? BUFFER_LOCK_UNLOCK indicates that
	 * the buffer is not locked.
	 */
	BufferLockMode lockmode;
} PrivateRefCountData;

This is elegant: one hash lookup answers both “how many pins does this backend hold on this buffer?” and “in what mode is it locked?”. And it means error recovery needs no new machinery — the resource owner’s buffer cleanup already releases the content locks, so there is no LWLockReleaseAll() equivalent to write.

Releasing a content lock

The release path is split into three pieces, so that the “subtract the lock from the state word” step can be fused with other atomic operations:

 c
 
 
/*
 * Release a previously acquired buffer content lock.
 */
static void
BufferLockUnlock(Buffer buffer, BufferDesc *buf_hdr)
{
	BufferLockMode mode;
	uint64		oldstate;
	uint64		sub;

	mode = BufferLockDisownInternal(buffer, buf_hdr);

	/*
	 * Release my hold on lock, after that it can immediately be acquired by
	 * others, even if we still have to wakeup other waiters.
	 */
	sub = BufferLockReleaseSub(mode);

	oldstate = pg_atomic_sub_fetch_u64(&buf_hdr->state, sub);

	BufferLockProcessRelease(buf_hdr, mode, oldstate);

	/*
	 * Now okay to allow cancel/die interrupts.
	 */
	RESUME_INTERRUPTS();
}

BufferLockReleaseSub() computes what to subtract:

 c
 
 
static inline uint64
BufferLockReleaseSub(BufferLockMode mode)
{
	/*
	 * Turns out that a switch() leads to gcc to generate sufficiently worse code
	 * for this to show up in profiles...
	 */
	if (mode == BUFFER_LOCK_EXCLUSIVE)
		return BM_LOCK_VAL_EXCLUSIVE;
	else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		return BM_LOCK_VAL_SHARE_EXCLUSIVE;
	else
	{
		Assert(mode == BUFFER_LOCK_SHARE);
		return BM_LOCK_VAL_SHARED;
	}

	return 0;					/* keep compiler quiet */
}

Note that releasing a share lock subtracts BM_LOCK_VAL_SHARED — the count goes down, which is why it is a counter and not a bit.

The wakeup decision:

 c
 
 
static void
BufferLockProcessRelease(BufferDesc *buf_hdr, BufferLockMode mode, uint64 lockstate)
{
	bool		check_waiters = false;
	bool		wake_exclusive = false;

	/* nobody else can have that kind of lock */
	Assert(!(lockstate & BM_LOCK_VAL_EXCLUSIVE));

	/*
	 * If we're still waiting for backends to get scheduled, don't wake them
	 * up again. Otherwise check if we need to look through the waitqueue to
	 * wake other backends.
	 */
	if ((lockstate & BM_LOCK_HAS_WAITERS) &&
		!(lockstate & BM_LOCK_WAKE_IN_PROGRESS))
	{
		if ((lockstate & BM_LOCK_MASK) == 0)
		{
			/*
			 * We released a lock and the lock was, in that moment, free. We
			 * therefore can wake waiters for any kind of lock.
			 */
			check_waiters = true;
			wake_exclusive = true;
		}
		else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			/*
			 * We released the lock, but another backend still holds a lock. We
			 * can't have released an exclusive lock, as there couldn't
			 * have been other lock holders. If we released a share lock, no
			 * waiters need to be woken up, as there must be other share
			 * lockers. However, if we held a share-exclusive lock, another
			 * backend now could acquire a share-exclusive lock.
			 */
			check_waiters = true;
			wake_exclusive = false;
		}
	}

	/*
	 * As waking up waiters requires the spinlock to be acquired, only do so
	 * if necessary.
	 */
	if (check_waiters)
		BufferLockWakeup(buf_hdr, wake_exclusive);
}

The three cases:

The lock is now completely free (lockstate & BM_LOCK_MASK == 0). Wake everybody — share, share-exclusive and exclusive waiters.
We released a share lock and others still hold locks. Nothing to do: there must still be other share lockers, so a share waiter can proceed.
We released a share-exclusive lock and others still hold locks. A share-exclusive waiter can now proceed (share-exclusive does not conflict with share), but an exclusive waiter cannot.

BufferLockWakeup() then walks the per-buffer wait list. This is where the three-level conflict matrix shows up:

 c
 
 
	proclist_foreach_modify(iter, &buf_hdr->lock_waiters, lwWaitLink)
	{
		PGPROC	 *waiter = GetPGProcByNumber(iter.cur);

		/*
		 * Already woke up a conflicting lock, so skip over this wait list
		 * entry.
		 */
		if (!wake_exclusive && waiter->lwWaitMode == BUFFER_LOCK_EXCLUSIVE)
			continue;
		if (!wake_share_exclusive && waiter->lwWaitMode == BUFFER_LOCK_SHARE_EXCLUSIVE)
			continue;

		proclist_delete(&buf_hdr->lock_waiters, iter.cur, lwWaitLink);
		proclist_push_tail(&wakeup, iter.cur, lwWaitLink);

		/*
		 * Prevent additional wakeups until retryer gets to run. Backends that
		 * are just waiting for the lock to become free don't retry
		 * automatically.
		 */
		new_wake_in_progress = true;

		/*
		 * Signal that the process isn't on the wait list anymore. This allows
		 * BufferLockDequeueSelf() to remove itself from the waitlist with a
		 * proclist_delete(), rather than having to check if it has been
		 * removed from the list.
		 */
		Assert(waiter->lwWaiting == LW_WS_WAITING);
		waiter->lwWaiting = LW_WS_PENDING_WAKEUP;

		/*
		 * Don't wakeup further waiters after waking a conflicting waiter.
		 */
		if (waiter->lwWaitMode == BUFFER_LOCK_SHARE)
		{
			/*
			 * Share locks conflict with exclusive locks.
			 */
			wake_exclusive = false;
		}
		else if (waiter->lwWaitMode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			/*
			 * Share-exclusive locks conflict with share-exclusive and
			 * exclusive locks.
			 */
			wake_exclusive = false;
			wake_share_exclusive = false;
		}
		else if (waiter->lwWaitMode == BUFFER_LOCK_EXCLUSIVE)
		{
			/*
			 * Exclusive locks conflict with all other locks, there's no point
			 * in waking up anybody else.
			 */
			break;
		}
	}

The classic FIFO-ish behaviour: wake share waiters, and stop at the first exclusive waiter, because everybody behind it conflicts with it anyway.

The wait list itself is protected by the buffer header spinlock, which is what BufferLockQueueSelf() does:

 c
 
 
static void
BufferLockQueueSelf(BufferDesc *buf_hdr, BufferLockMode mode)
{
	/*
	 * If we don't have a PGPROC structure, there's no way to wait. This
	 * should never occur, since MyProc should only be null during shared
	 * memory initialization.
	 */
	if (MyProc == NULL)
		elog(PANIC, "cannot wait without a PGPROC structure");

	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

	LockBufHdr(buf_hdr);

	/* setting the flag is protected by the spinlock */
	pg_atomic_fetch_or_u64(&buf_hdr->state, BM_LOCK_HAS_WAITERS);
	...
	proclist_push_tail(&buf_hdr->lock_waiters, MyProcNumber, lwWaitLink);

	/* Can release the mutex now */
	UnlockBufHdr(buf_hdr);
}

The header lock is not an LWLock either

A pleasant side effect of the change is that the buffer header lock is now also not a real spinlock with a slock_t. It is a bit in the state word, and LockBufHdr() is a single fetch_or:

 c
 
 
/*
 * Lock buffer header - set BM_LOCKED in buffer state.
 */
uint64
LockBufHdr(BufferDesc *desc)
{
	uint64		old_buf_state;

	Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc)));

	while (true)
	{
		/*
		 * Always try once to acquire the lock directly, without setting up
		 * the spin-delay infrastructure. The work necessary for that shows up
		 * in profiles and is rarely necessary.
		 */
		old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED);
		if (likely(!(old_buf_state & BM_LOCKED)))
			break;				/* got lock */

		/* and then spin without atomic operations until lock is released */
		{
			SpinDelayStatus delayStatus;

			init_local_spin_delay(&delayStatus);

			while (old_buf_state & BM_LOCKED)
			{
				perform_spin_delay(&delayStatus);
				old_buf_state = pg_atomic_read_u64(&desc->state);
			}
			finish_spin_delay(&delayStatus);
		}

		/*
		 * Retry. The lock might obviously already be re-acquired by the time
		 * we're attempting to get it again.
		 */
	}

	return old_buf_state | BM_LOCKED;
}

That comment — “Always try once to acquire the lock directly, without setting up the spin-delay infrastructure. The work necessary for that shows up in profiles and is rarely necessary.” — is a micro-optimisation with a real history behind it. Setting up SpinDelayStatus costs enough that it showed up in profiles when the lock was uncontended (which it almost always is).

The uncontended path is now: one lock or on a cache line that is already hot, one branch. The contended path spins on a plain load, no atomic instructions at all, until the bit clears.

Releasing is symmetric and equally cheap:

 c
 
 
/*
 * Unlock the buffer header.
 *
 * This can only be used if the caller did not modify BufferDesc.state. To
 * set/unset flag bits or change the refcount use UnlockBufHdrExt().
 */
static inline void
UnlockBufHdr(BufferDesc *desc)
{
	Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED);

	pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED);
}

And the more general version, which sets flags and adjusts the refcount in the same CAS — this is the “release a buffer lock and pin in one atomic operation” that the commit message promised, in its simpler form:

 c
 
 
/*
 * Unlock the buffer header, while atomically unsetting the ones in
 * unset_bits, adding the flags in set_bits, and changing the refcount by
 * refcount_change. If a flag is both cleared and added, it will end up being
 * set.
 *
 * Note that this approach would not trivially work for usagecount, since we
 * need to cap the usagecount at BM_MAX_USAGE_COUNT.
 */
static inline uint64
UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state,
				uint64 set_bits, uint64 unset_bits,
				int refcount_change)
{
	for (;;)
	{
		uint64		buf_state = old_buf_state;

		Assert(buf_state & BM_LOCKED);

		/*
		 * Set bits after clearing bits, so that a cleared and re-added flag
		 * survives.
		 */
		buf_state &= ~unset_bits;
		buf_state |= set_bits;
		buf_state &= ~BM_LOCKED;

		if (refcount_change != 0)
			buf_state += BUF_REFCOUNT_ONE * refcount_change;

		if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state,
										 buf_state))
		{
			return old_buf_state;
		}
	}
}

Note the comment about the usage count: it cannot be handled this way, because the usage count is capped at BM_MAX_USAGE_COUNT and a blind addition would overflow the cap.

The counterpart — waiting for the header lock to clear — is a pg_noinline spin loop:

 c
 
 
/*
 * Wait until the BM_LOCKED flag isn't set anymore and return the buffer's
 * state at that point.
 *
 * Obviously the buffer could be locked by the time the value is returned, so
 * this is primarily useful in CAS style loops.
 */
pg_noinline uint64
WaitBufHdrUnlocked(BufferDesc *buf)
{
	SpinDelayStatus delayStatus;
	uint64		buf_state;

	init_local_spin_delay(&delayStatus);

	buf_state = pg_atomic_read_u64(&buf->state);

	while (buf_state & BM_LOCKED)
	{
		perform_spin_delay(&delayStatus);
		buf_state = pg_atomic_read_u64(&buf->state);
	}

	finish_spin_delay(&delayStatus);

	return buf_state;
}

pg_noinline is deliberate: this is the slow path, and inlining it into every caller would bloat the instruction cache for no benefit.

The reason the third lock level exists at all is a correctness issue with hint bits.

PostgreSQL caches visibility facts (Xmin committed, Xmax invalid, …) in the tuple’s t_infomask. Those are called hint bits. Setting them modifies the page. If the page is being written out at the same time, the page checksum computed over the bytes that go to disk does not match the bytes that were in memory when the checksum was computed — so the buffer manager had to copy the page before writing it, which is expensive.

With a third lock level, hint-bit setting and write-out can be made mutually exclusive. Both need share-exclusive, and share-exclusive conflicts with itself.

Writing the buffer takes share-exclusive:

 c
 
 
/*
 * Convenience wrapper around FlushBuffer() that locks/unlocks the buffer
 * before/after calling FlushBuffer().
 */
static void
FlushUnlockedBuffer(BufferDesc *buf, SMgrRelation reln,
					IOObject io_object, IOContext io_context)
{
	Buffer		buffer = BufferDescriptorGetBuffer(buf);

	BufferLockAcquire(buffer, buf, BUFFER_LOCK_SHARE_EXCLUSIVE);
	FlushBuffer(buf, reln, io_object, io_context);
	BufferLockUnlock(buffer, buf);
}

Setting hint bits takes share-exclusive too — but if the backend only holds a share lock, it has to upgrade first, and the upgrade can fail:

 c
 
 
/*
 * Helper for BufferBeginSetHintBits() and BufferSetHintBits16().
 *
 * This checks if the current lock mode already suffices to allow hint bits
 * being set and, if not, whether the current lock can be upgraded.
 *
 * Updates *lockstate when returning true.
 */
static inline bool
SharedBufferBeginSetHintBits(Buffer buffer, BufferDesc *buf_hdr, uint64 *lockstate)
{
	uint64		old_state;
	PrivateRefCountEntry *ref;
	BufferLockMode mode;

	ref = GetPrivateRefCountEntry(buffer, true);

	if (ref == NULL)
		elog(ERROR, "buffer is not pinned");

	mode = ref->data.lockmode;
	if (mode == BUFFER_LOCK_UNLOCK)
		elog(ERROR, "buffer is not locked");

	/* we're done if we are already holding a sufficient lock level */
	if (mode == BUFFER_LOCK_EXCLUSIVE || mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
	{
		*lockstate = pg_atomic_read_u64(&buf_hdr->state);
		return true;
	}

	/*
	 * We are only holding a share lock right now, try to upgrade it to
	 * SHARE_EXCLUSIVE.
	 */
	Assert(mode == BUFFER_LOCK_SHARE);

	old_state = pg_atomic_read_u64(&buf_hdr->state);
	while (true)
	{
		uint64		desired_state;

		desired_state = old_state;

		/*
		 * Can't upgrade if somebody else holds the lock in exclusive or
		 * share-exclusive mode.
		 */
		if (unlikely((old_state & (BM_LOCK_VAL_EXCLUSIVE | BM_LOCK_VAL_SHARE_EXCLUSIVE)) != 0))
		{
			return false;
		}

		/* currently held lock state */
		desired_state -= BM_LOCK_VAL_SHARED;

		/* new lock level */
		desired_state += BM_LOCK_VAL_SHARE_EXCLUSIVE;

		if (likely(pg_atomic_compare_exchange_u64(&buf_hdr->state,
												 &old_state, desired_state)))
		{
			ref->data.lockmode = BUFFER_LOCK_SHARE_EXCLUSIVE;
			*lockstate = desired_state;

			return true;
		}
	}
}

Note that the upgrade is a non-blocking CAS. If it fails — because somebody else holds share-exclusive or exclusive — the backend simply does not set the hint bits this time round. That is fine: hint bits are an optimisation, and the whole point is to avoid blocking a reader on a writer.

The comment on BufferBeginSetHintBits() spells out the trade-off:

 c
 
 
/*
 * Try to acquire the right to set hint bits on the buffer.
 *
 * To be allowed to set hint bits, this backend needs to hold either a
 * share-exclusive or an exclusive lock. In case this backend only holds a
 * share lock, this function will try to upgrade the lock to
 * share-exclusive. The caller is only allowed to set hint bits if true is
 * returned.
 *
 * Once BufferBeginSetHintBits() has returned true, hint bits may be set
 * without further calls to BufferBeginSetHintBits(), until the buffer is
 * unlocked.
 *
 *
 * Requiring a share-exclusive lock to set hint bits prevents setting hint
 * bits on buffers that are currently being written out, which could corrupt
 * the checksum on the page. Flushing buffers also requires a share-exclusive
 * lock.
 *
 * Due to a lock >= share-exclusive being required to set hint bits, only one
 * backend can set hint bits at a time. Allowing multiple backends to set
 * hint bits would require more complicated locking: For setting hint bits
 * we'd need to store the count of backends currently setting hint bits, for
 * I/O we would need another lock-level conflicting with the hint-setting
 * lock-level. Given that the share-exclusive lock for setting hint bits is
 * only held for a short time, that backends often would just set the same
 * hint bits and that the cost of occasionally not setting hint bits in hotly
 * accessed pages is fairly low, this seems like an acceptable tradeoff.
 */

The “only one backend at a time” is a direct consequence of the share-lock field being 18 bits wide but the share-exclusive level being a single bit. There is room in the word for a second share-exclusive counter; there just is not one.

The lock-free clock sweep

The other place the new atomicity pays off is the buffer replacement strategy. src/backend/storage/buffer/freelist.c implements the clock sweep, and the clock hand is a plain atomic counter:

 c
 
 
typedef struct
{
	/* Spinlock: protects the values below */
	slock_t		buffer_strategy_lock;

	/*
	 * clock-sweep hand: index of next buffer to consider grabbing. Note that
	 * this isn't a concrete buffer - we only ever increase the value. So, to
	 * get an actual buffer, it needs to be used modulo NBuffers.
	 */
	pg_atomic_uint32 nextVictimBuffer;

	/*
	 * Statistics. These counters should be wide enough that they can't
	 * overflow during a single bgwriter cycle.
	 */
	uint32		completePasses; /* Complete cycles of the clock-sweep */
	pg_atomic_uint32 numBufferAllocs;	/* Buffers allocated since last reset */

	/*
	 * Bgworker process to be notified upon activity or -1 if none. See
	 * StrategyNotifyBgWriter.
	 */
	int			bgwprocno;
} BufferStrategyControl;

The hand moves with a single fetch_add:

 c
 
 
static inline uint32
ClockSweepTick(void)
{
	uint32		victim;

	/*
	 * Atomically move hand ahead one buffer - if there's several processes
	 * doing this, this can lead to buffers being returned slightly out of
	 * apparent order.
	 */
	victim =
		pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);

	if (victim >= NBuffers)
	{
		uint32		originalVictim = victim;

		/* always wrap what we look up in BufferDescriptors */
		victim = victim % NBuffers;

		/*
		 * If we're the one that just caused a wraparound, force
		 * completePasses to be incremented while holding the spinlock. We
		 * need the spinlock so StrategySyncStart() can return a consistent
		 * value consisting of nextVictimBuffer and completePasses.
		 */
		if (victim == 0)
		{
			uint32		expected;
			uint32		wrapped;
			bool		success = false;

			expected = originalVictim + 1;

			while (!success)
			{
				/*
				 * Acquire the spinlock while increasing completePasses. That
				 * allows other readers to read nextVictimBuffer and
				 * completePasses in a consistent manner which is required for
				 * StrategySyncStart(). In theory delaying the increment
				 * could lead to an overflow of nextVictimBuffers, but that's
				 * highly unlikely and wouldn't be particularly harmful.
				 */
				SpinLockAcquire(&StrategyControl->buffer_strategy_lock);

				wrapped = expected % NBuffers;

				success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
														 &expected, wrapped);
				if (success)
					StrategyControl->completePasses++;
				SpinLockRelease(&StrategyControl->buffer_strategy_lock);
			}
		}
	}
	return victim;
}

There is exactly one spinlock left in the whole file, and it is held only for the wraparound bookkeeping. Advancing the hand is a single atomic add — no lock, ever.

The interesting part is StrategyGetBuffer(), which selects a victim and pins it without ever taking the buffer header spinlock:

 c
 
 
	/* Use the "clock sweep" algorithm to find a free buffer */
	trycounter = NBuffers;
	for (;;)
	{
		uint64		old_buf_state;
		uint64		local_buf_state;

		buf = GetBufferDescriptor(ClockSweepTick());

		/*
		 * Check whether the buffer can be used and pin it if so. Do this
		 * using a CAS loop, to avoid having to lock the buffer header.
		 */
		old_buf_state = pg_atomic_read_u64(&buf->state);
		for (;;)
		{
			local_buf_state = old_buf_state;

			/*
			 * If the buffer is pinned or has a nonzero usage_count, we cannot
			 * use it; decrement the usage_count (unless pinned) and keep
			 * scanning.
			 */

			if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0)
			{
				if (--trycounter == 0)
				{
					/*
					 * We've scanned all the buffers without making any state
					 * changes, so all the buffers are pinned (or were when we
					 * looked at them). We could hope that someone will free
					 * one eventually, but it's probably better to fail than
					 * to risk getting stuck in an infinite loop.
					 */
					elog(ERROR, "no unpinned buffers available");
				}
				break;
			}

			/* See equivalent code in PinBuffer() */
			if (unlikely(local_buf_state & BM_LOCKED))
			{
				old_buf_state = WaitBufHdrUnlocked(buf);
				continue;
			}

			if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0)
			{
				local_buf_state -= BUF_USAGECOUNT_ONE;

				if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
												 local_buf_state))
				{
					trycounter = NBuffers;
					break;
				}
			}
			else
			{
				/* pin the buffer if the CAS succeeds */
				local_buf_state += BUF_REFCOUNT_ONE;

				if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
												 local_buf_state))
				{
					/* Found a usable buffer */
					if (strategy != NULL)
						AddBufferToRing(strategy, buf);
					*buf_state = local_buf_state;

					TrackNewBufferPin(BufferDescriptorGetBuffer(buf));

					return buf;
				}
			}
		}
	}

Read the two CAS cases:

Usage count is non-zero. The buffer is not pinned, but it has been used recently, so it gets a second chance: decrement the usage count and move on. The CAS is needed because the usage count and the refcount live in the same word, and we must not clobber a concurrent refcount change.
Usage count is zero and refcount is zero. The buffer is free. Add one to the refcount — that is the pin — and take it.

The pin and the “is this buffer still free?” decision are the same CAS. Compare that with the old code, which had to take the buffer header spinlock to increment the refcount and then release it. Here, the only time the header lock is involved at all is when BM_LOCKED happens to be set, in which case the code waits for it to clear and retries:

 c
 
 
			/* See equivalent code in PinBuffer() */
			if (unlikely(local_buf_state & BM_LOCKED))
			{
				old_buf_state = WaitBufHdrUnlocked(buf);
				continue;
			}

The second chance, quantified

The usage count is four bits wide, capped at 5:

 c
 
 
/*
 * The maximum allowed value of usage_count represents a tradeoff between
 * accuracy and speed of the clock-sweep buffer management algorithm. A
 * large value (comparable to NBuffers) would approximate LRU semantics.
 * But it can take as many as BM_MAX_USAGE_COUNT+1 complete cycles of the
 * clock-sweep hand to find a free buffer, so in practice we don't want the
 * value to be very large.
 */
#define BM_MAX_USAGE_COUNT	5

StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS),
				 "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits");

Five is a compromise, and the comment says so: a large value would approximate LRU, but the clock hand might need BM_MAX_USAGE_COUNT + 1 full passes over the buffer array to find a free buffer. With shared_buffers = 128 GB and 8 kB pages that is 16 million buffers, and a value of 5 means at most 6 passes.

The value is bumped on pin, capped at the maximum:

 c
 
 
			/* increase refcount */
			buf_state += BUF_REFCOUNT_ONE;

			if (strategy == NULL)
			{
				/* Default case: increase usagecount unless already max. */
				if (BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
					buf_state += BUF_USAGECOUNT_ONE;
			}

This is CLOCK in the textbook sense: the hand gives each buffer a bounded number of second chances, which makes the eviction policy an approximation of LRU rather than a strict LRU.

Caveats

This is a big, invasive change

The commit touches the buffer manager, the buffer replacement strategy, the resource owner machinery, the wait-event taxonomy, and the index access methods. The diff is large, and the new code deliberately duplicates a meaningful chunk of lwlock.c.

The commit message is explicit about the cost:

This commit unfortunately introduces some code that is very similar to the code in lwlock.c, however the code is not equivalent enough to easily merge it. The future wins that this commit makes possible seem worth the cost.

And about the wart:

It’s worth calling out one wart in this commit: Despite content locks not being lwlocks anymore, they continue to use PGPROC->lw* - that seemed better than duplicating the relevant infrastructure.

The wait event taxonomy changed

Content locks are no longer reported as LWLock waits. There are three new wait events in the Buffer class:

text

BUFFER_SHARED	"Waiting to acquire a shared lock on a buffer."
BUFFER_SHARE_EXCLUSIVE	"Waiting to acquire a share exclusive lock on a buffer."
BUFFER_EXCLUSIVE	"Waiting to acquire an exclusive lock on a buffer."

The old BufferContent LWLock tranche is gone, and the BUFFERPIN wait event class was renamed to BUFFER in commit 6c5c393b740 to make room for them. Any monitoring that watched for LWLock waits on the buffer content tranche needs updating.

A backend can only hold one lock at a time

The implementation does not track multiple lock ownerships of the same buffer within one backend. BufferLockConditional() documents the consequence:

 c
 
 
/*
 * Acquire the content lock for the buffer, but only if we don't have to wait.
 *
 * It is allowed to try to conditionally acquire a lock on a buffer that this
 * backend has already locked, but the lock acquisition will always fail, even
 * if the new lock acquisition does not conflict with an already held lock
 * (e.g. two share locks). This is because we currently do not have space to
 * track multiple lock ownerships of the same buffer within one backend. That
 * is ok for the current uses of BufferLockConditional().
 */

So a backend that already holds a share lock on a buffer cannot conditionally upgrade it; BufferLockConditional() returns false. The non-conditional upgrade path (SharedBufferBeginSetHintBits()) does its own CAS and works fine.

The 18-bit share-lock field holds at most 262 143 concurrent share lockers. That is exactly MAX_BACKENDS, so it cannot overflow in practice — but it also means the design assumes at most one share lock per backend per buffer, which is what the private refcount hash enforces.

Deadlock semantics are essentially unchanged

The queueing algorithm is the LWLock algorithm, so the deadlock properties are the same: the wait list is FIFO-ish, exclusive waiters block later waiters, and there is no deadlock detector — PostgreSQL relies on the lock ordering conventions of the code that takes the locks.

The one structural difference is that content locks and LWLocks are now different lock types. A backend that holds a buffer content lock and then tries to take an LWLock (or vice versa) could in principle deadlock in a way the old design could not, since the old content lock was an LWLock and the global lock-ordering discipline applied uniformly. The commit’s answer to this is the PGPROC->lwWaiting check:

 c
 
 
	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

A backend cannot block on a buffer content lock while already blocked on something else — that case is a PANIC, not a deadlock. Which is a stronger guarantee than a deadlock detector, but only because it is enforced by the PGPROC state.

Platforms without 64-bit atomics

The whole design rests on pg_atomic_uint64. On platforms that lack native 64-bit atomic support, PostgreSQL emulates them with a spinlock (src/include/port/atomics/fallback.h):

 c
 
 
#if !defined(PG_HAVE_ATOMIC_U64_SUPPORT)

#define PG_HAVE_ATOMIC_U64_SIMULATION

#define PG_HAVE_ATOMIC_U64_SUPPORT
typedef struct pg_atomic_uint64
{
	int			sema;
	volatile uint64 value;
} pg_atomic_uint64;

#define PG_HAVE_ATOMIC_INIT_U64
extern void pg_atomic_init_u64_impl(volatile pg_atomic_uint64 *ptr, uint64 val_);

#define PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U64
extern bool pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
												uint64 *expected, uint64 newval);

#define PG_HAVE_ATOMIC_FETCH_ADD_U64
extern uint64 pg_atomic_fetch_add_u64_impl(volatile pg_atomic_uint64 *ptr, int64 add_);

#endif /* PG_HAVE_ATOMIC_U64_SUPPORT */

and the implementation in src/backend/port/atomics.c:

 c
 
 
bool
pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
									uint64 *expected, uint64 newval)
{
	bool		ret;

	/*
	 * Do atomic op under a spinlock. It might look like we could just skip
	 * the cmpxchg if the lock isn't available, but that'd just emulate a
	 * 'weak' compare and swap. I.e. one that allows spurious failures. Since
	 * several algorithms rely on a strong variant and that is efficiently
	 * implementable on most major architectures let's emulate it here as
	 * well.
	 */
	SpinLockAcquire((slock_t *) &ptr->sema);

	/* perform compare/exchange logic */
	ret = ptr->value == *expected;
	*expected = ptr->value;
	if (ret)
		ptr->value = newval;

	/* and release lock */
	SpinLockRelease((slock_t *) &ptr->sema);

	return ret;
}

The other operations are built on top of the CAS, in src/include/port/atomics/generic.h:

 c
 
 
#if !defined(PG_HAVE_ATOMIC_FETCH_OR_U64) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U64)
#define PG_HAVE_ATOMIC_FETCH_OR_U64
static inline uint64
pg_atomic_fetch_or_u64_impl(volatile pg_atomic_uint64 *ptr, uint64 or_)
{
	uint64 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u64_impl(ptr, &old, old | or_))
		/* skip */;
	return old;
}
#endif

So on such a platform, LockBufHdr() — a single fetch_or — becomes a spinlock acquire and a CAS loop. The logical result is identical; the performance is not. The whole point of the change is to make the common path cheap, and on a fallback platform it is not cheap at all.

This matters more for the content lock than for the header lock. The header lock was already a BM_LOCKED bit in the state word before this change — commit dac328c8a68, “bufmgr: Change BufferDesc.state to be a 64-bit atomic”, had just widened that word from 32 to 64 bits — but it was a bit on 32-bit atomics, which have native support on every platform PostgreSQL builds on. The content lock, likewise, was an LWLock, whose own state word is a uint32. Moving both into a 64-bit word turns a native lock acquisition into a spinlock-guarded CAS on a platform without 64-bit atomics.

On every mainstream platform — x86-64, ARM64, POWER, s390x — 64-bit atomics are native and the fallback is not used.

The state word is now a contention point

Before the change, the refcount lived in the state word and the lock lived in the LWLock. Now they live together. That is the whole point — it lets the lock decision see the refcount — but it also means that unrelated operations on the same buffer now touch the same cache line. A buffer that is being pinned and unpinned rapidly is a buffer whose state word is hot.

The BufferDesc is padded to a cache line to keep it from sharing with its neighbours:

 c
 
 
/*
 * Concurrent access to buffer headers has proven to be more efficient if
 * they're cache line aligned. So we force the start of the BufferDescriptors
 * array to be on a cache line boundary and force the elements to be cache
 * line sized.
 *
 * XXX: As this is primarily matters in highly concurrent workloads which
 * probably all are 64bit these days, and the space wastage would be a bit
 * more noticeable on 32bit systems, we don't force the stride to be cache
 * line sized on those. If somebody does actual performance testing, we can
 * reevaluate.
 *
 * Note that local buffer descriptors aren't forced to be aligned - as there's
 * no concurrent access to those it's unlikely to be beneficial.
 *
 * We use a 64-byte cache line size here, because that's the most common
 * size. Making it bigger would be a waste of memory. Even if running on a
 * platform with either 32 or 128 byte line sizes, it's good to align to
 * boundaries and avoid false sharing.
 */
#define BUFFERDESC_PAD_TO_SIZE	(SIZEOF_VOID_P == 8 ? 64 : 1)

typedef union BufferDescPadded
{
	BufferDesc	bufferdesc;
	char		pad[BUFFERDESC_PAD_TO_SIZE];
} BufferDescPadded;

Note the XXX comment: on 32-bit systems the padding is disabled, because the waste would be noticeable. The header comment in the struct is explicit about the constraint:

Be careful to avoid increasing the size of the struct when adding or reordering members. Keeping it below 64 bytes (the most common CPU cache line size) is fairly important for performance.

Removing the embedded LWLock (16 bytes) and replacing it with an 8-byte proclist_head is what keeps the struct within the budget.

Local buffers are a different story

Local buffers (temp tables) use a completely separate mechanism, and the content lock is a no-op for them:

 c
 
 
	Assert(BufferIsPinned(buffer));
	if (BufferIsLocal(buffer))
		return;					/* local buffers need no lock */

Because a local buffer belongs to exactly one backend, there is nothing to lock against.

What it all means in practice

The buffer content lock is now the cheapest lock in PostgreSQL. In the uncontended case it is a single compare-and-swap on a word that is already in the cache. There is no lock structure to walk, no separate state word to coherency-check, no LWLock array to index.

The lock state and the pin state are now the same word. This is the structural change. It means:

Deciding whether a buffer is free to evict and pinning it are one atomic operation (StrategyGetBuffer()).
Checking whether I/O is in progress and taking an exclusive lock are atomic with respect to each other — which is what makes asynchronous writes possible.
Releasing a lock and dropping a pin can be fused into one CAS.

Hint-bit setting is now correct with respect to write-out. Before, a reader could set a hint bit on a page while the checkpointer was writing that same page out, forcing an expensive page copy. Now the two are mutually exclusive via the share-exclusive level, and the copy can go away. That matters more as PostgreSQL moves towards AIO, where many buffers are in flight at once.

The lock-free clock sweep is the quiet win. StrategyGetBuffer() no longer takes the buffer header spinlock to pin a victim. On a busy system with a large shared_buffers, the clock hand’s path through the buffer array no longer serialises on a lock.

Watch the wait events. If you have dashboards that count LWLock waits on BufferContent, they are now dead. The replacements are Buffer/Shared, Buffer/ShareExclusive and Buffer/Exclusive.

PostgreSQL Locks Tables Fast

Jay Hubbert — Thu, 06 Aug 2026 09:00:00 +0000

Ask PostgreSQL how many locks it takes and the honest answer is “it depends, and the interesting part is which lock you’re talking about”. The src/backend/storage/lmgr/README file opens with four kinds of interprocess lock — spinlocks, lightweight locks, regular (heavyweight) locks, and predicate locks — and the heavyweight one is itself split in two: a shared hash table, and a per-backend fast path.

Every SELECT on a table takes an AccessShareLock. Every INSERT takes a RowExclusiveLock. Every ALTER TABLE takes an AccessExclusiveLock and then waits for every other session to go away. If all of that went through one lock table protected by one mutex, a thousand concurrent readers would be a thousand processes fighting over one cache line, and the lock manager would be the thing that limits your database’s concurrency.

It isn’t, because most of those locks never touch the lock table at all.

There are really two managers here:

	Heavyweight lock manager	Fast path
Where it lives	shared hash table	per-backend slice of `PGPROC`
Protected by	an LWLock, 16-way partitioned	`proc->fpInfoLock`
Lock modes	all 8	3 (weak relation locks)
Object types	everything	relations in the current database
Deadlock detection	yes	no (but also no waiting)
Cost	hash lookup + LWLock	a few instructions

The fast path exists for the overwhelmingly common case: a lock that nobody will ever wait for. If you already hold AccessShareLock on a table and you ask for AccessShareLock again, or if you take RowExclusiveLock on a table nobody is altering, you do not need a heavyweight lock. You need a counter.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

The lock tag: sixteen bytes that name anything

Everything in the lock manager is keyed by a LOCKTAG. It lives in src/include/storage/locktag.h (it used to live in lock.h, and was split out), and it is designed with malice aforethought:

 c
 
 
/*
 * The LOCKTAG struct is defined with malice aforethought to fit into 16
 * bytes with no padding. Note that this would need adjustment if we were
 * to widen Oid, BlockNumber, or TransactionId to more than 32 bits.
 *
 * We include lockmethodid in the locktag so that a single hash table in
 * shared memory can store locks of different lockmethods.
 */
typedef struct LOCKTAG
{
	uint32		locktag_field1; /* a 32-bit ID field */
	uint32		locktag_field2; /* a 32-bit ID field */
	uint32		locktag_field3; /* a 32-bit ID field */
	uint16		locktag_field4; /* a 16-bit ID field */
	uint8		locktag_type;	/* see enum LockTagType */
	uint8		locktag_lockmethodid;	/* lockmethod indicator */
} LOCKTAG;

Four bytes, four bytes, four bytes, two bytes, one byte, one byte: 16 bytes, no padding. The “one field per byte” framing in the field names is deliberately generic — the struct doesn’t know what it’s identifying, only how many bits each component needs. The interpretation is fixed by locktag_type:

 c
 
 
typedef enum LockTagType
{
	LOCKTAG_RELATION,			/* whole relation */
	LOCKTAG_RELATION_EXTEND,	/* the right to extend a relation */
	LOCKTAG_DATABASE_FROZEN_IDS,	/* pg_database.datfrozenxid */
	LOCKTAG_PAGE,				/* one page of a relation */
	LOCKTAG_TUPLE,				/* one physical tuple */
	LOCKTAG_TRANSACTION,		/* transaction (for waiting for xact done) */
	LOCKTAG_VIRTUALTRANSACTION, /* virtual transaction (ditto) */
	LOCKTAG_SPECULATIVE_TOKEN,	/* speculative insertion Xid and token */
	LOCKTAG_OBJECT,				/* non-relation database object */
	LOCKTAG_USERLOCK,			/* reserved for old contrib/userlock code */
	LOCKTAG_ADVISORY,			/* advisory user locks */
	LOCKTAG_APPLY_TRANSACTION,	/* transaction being applied on a logical
								 * replication subscriber */
} LockTagType;

Twelve kinds of lockable object. LOCKTAG_RELATION is the one you care about most of the time; LOCKTAG_TUPLE is the one that sounds scary but isn’t — tuple locks are a serialization fiction (the “locked tuple” flag on the tuple header), not entries in the lock table.

The comment about lockmethodid is the other half of the design. There are two lock methods — DEFAULT_LOCKMETHOD and USER_LOCKMETHOD — and they have identical mode semantics today but are conceptually separate. By putting the method id in the tag, one shared hash table can hold both, and the code can look up the right semantics table from the tag alone.

Filling in the tag

Because the fields are generic, there are macros for every kind of tag:

 c
 
 
/* ID info for a relation is DB OID + REL OID; DB OID = 0 if shared */
#define SET_LOCKTAG_RELATION(locktag,dboid,reloid) \
	((locktag).locktag_field1 = (dboid), \
	 (locktag).locktag_field2 = (reloid), \
	 (locktag).locktag_field3 = 0, \
	 (locktag).locktag_field4 = 0, \
	 (locktag).locktag_type = LOCKTAG_RELATION, \
	 (locktag).locktag_lockmethodid = DEFAULT_LOCKMETHOD)

Note the first field: the database OID is part of every relation tag. That is what makes a lock on mytable in db1 and a lock on mytable in db2 different locks, without any special-casing in the hash table.

SET_LOCKTAG_ADVISORY() shows the other pattern — the fields are just whatever the caller wants them to be:

 c
 
 
#define SET_LOCKTAG_ADVISORY(locktag,id1,id2,id3,id4) \
	((locktag).locktag_field1 = (id1), \
	 (locktag).locktag_field2 = (id2), \
	 (locktag).locktag_field3 = (id3), \
	 (locktag).locktag_field4 = (id4), \
	 (locktag).locktag_type = LOCKTAG_ADVISORY, \
	 (locktag).locktag_lockmethodid = USER_LOCKMETHOD)

and src/backend/utils/adt/lockfuncs.c uses it for pg_advisory_lock(), with a comment describing the packing:

 c
 
 
/*
 * Functions for manipulating advisory locks
 *
 * We make use of the locktag fields as follows:
 *
 *	field1: MyDatabaseId ... ensures locks are local to each database
 *	field2: first of 2 int4 keys, or high-order half of an int8 key
 *	field3: second of 2 int4 keys, or low-order half of an int8 key
 *	field4: 1 if using an int8 key, 2 if using 2 int4 keys
 */
#define SET_LOCKTAG_INT64(tag, key64) \
	SET_LOCKTAG_ADVISORY(tag, \
						 MyDatabaseId, \
						 (uint32) ((key64) >> 32), \
						 (uint32) (key64), \
						 1)

This is the SQL surface of the lock manager. lockfuncs.c is a thin layer: pg_lock_status() (the pg_locks view), pg_blocking_pids(), and the advisory-lock functions. The machinery itself is all in lock.c.

The heavyweight lock table

The shared lock table is two hash tables, both partitioned.

The first is keyed by LOCKTAG and holds one LOCK struct per lockable object that currently has locks on it (src/include/storage/lock.h):

 c
 
 
typedef struct LOCK
{
	/* hash key */
	LOCKTAG		tag;			/* unique identifier of lockable object */

	/* data */
	LOCKMASK	grantMask;		/* bitmask for lock types already granted */
	LOCKMASK	waitMask;		/* bitmask for lock types awaited */
	dlist_head	procLocks;		/* list of PROCLOCK objects assoc. with lock */
	dclist_head waitProcs;		/* list of PGPROC objects waiting on lock */
	int			requested[MAX_LOCKMODES];	/* counts of requested locks */
	int			nRequested;		/* total of requested[] array */
	int			granted[MAX_LOCKMODES]; /* counts of granted locks */
	int			nGranted;		/* total of granted locks */
} LOCK;

The LOCK object is the aggregation of all holders and waiters for one object. grantMask is a bitmask of which modes are held by somebody; waitMask is which modes somebody is waiting for. Those two bitmasks are what make the common case cheap: a conflict check is conflictTab[lockmode] & lock->grantMask, one AND.

The second table is keyed by PROCLOCKTAG — the pair (LOCK, PGPROC) — and holds the per-holder detail:

 c
 
 
typedef struct PROCLOCKTAG
{
	/* NB: we assume this struct contains no padding! */
	LOCK	 *myLock;			/* link to per-lockable-object information */
	PGPROC	 *myProc;			/* link to PGPROC of owning backend */
} PROCLOCKTAG;

typedef struct PROCLOCK
{
	/* tag */
	PROCLOCKTAG tag;			/* unique identifier of proclock object */

	/* data */
	PGPROC	 *groupLeader;	/* proc's lock group leader, or proc itself */
	LOCKMASK	holdMask;		/* bitmask for lock types currently held */
	LOCKMASK	releaseMask;	/* bitmask for lock types to be released */
	dlist_node	lockLink;		/* list link in LOCK's list of proclocks */
	dlist_node	procLink;		/* list link in PGPROC's list of proclocks */
} PROCLOCK;

So: one LOCK per object, one PROCLOCK per (object, backend). A table locked by fifty sessions has one LOCK and fifty PROCLOCKs.

Partitioning

The two hash tables are created with a partition count:

 c
 
 
	max_table_size = NLOCKENTS();

	/*
	 * Hash table for LOCK structs. This stores per-locked-object
	 * information.
	 */
	ShmemRequestHash(.name = "LOCK hash",
					 .nelems = max_table_size,
					 .ptr = &LockMethodLockHash,
					 .hash_info.keysize = sizeof(LOCKTAG),
					 .hash_info.entrysize = sizeof(LOCK),
					 .hash_info.num_partitions = NUM_LOCK_PARTITIONS,
					 .hash_flags = HASH_ELEM | HASH_BLOBS | HASH_PARTITION,
	);

	/* Assume an average of 2 holders per lock */
	max_table_size *= 2;

	ShmemRequestHash(.name = "PROCLOCK hash",
					 .nelems = max_table_size,
					 .ptr = &LockMethodProcLockHash,
					 .hash_info.keysize = sizeof(PROCLOCKTAG),
					 .hash_info.entrysize = sizeof(PROCLOCK),
					 .hash_info.hash = proclock_hash,
					 .hash_info.num_partitions = NUM_LOCK_PARTITIONS,
					 .hash_flags = HASH_ELEM | HASH_FUNCTION | HASH_PARTITION,
	);

and NUM_LOCK_PARTITIONS is 16:

 c
 
 
/* Number of partitions the shared lock tables are divided into */
#define LOG2_NUM_LOCK_PARTITIONS 4
#define NUM_LOCK_PARTITIONS (1 << LOG2_NUM_LOCK_PARTITIONS)

Each partition has its own LWLock, and the partition is chosen by the hash code:

 c
 
 
/*
 * The lockmgr's shared hash tables are partitioned to reduce contention.
 * To determine which partition a given locktag belongs to, compute the tag's
 * hash code with LockTagHashCode(), then apply one of these macros.
 * NB: NUM_LOCK_PARTITIONS must be a power of 2!
 */
#define LockHashPartition(hashcode) \
	((hashcode) % NUM_LOCK_PARTITIONS)
#define LockHashPartitionLock(hashcode) \
	(&MainLWLockArray[LOCK_MANAGER_LWLOCK_OFFSET + \
		LockHashPartition(hashcode)].lock)

So: lock contention is spread over sixteen independent LWLocks. Two sessions locking unrelated tables have a 15-in-16 chance of not contending on the partition lock at all. This is the mechanism behind “PostgreSQL scales well to many concurrent sessions” for lock-heavy workloads.

The hash code itself comes from the hash table’s own hash function:

 c
 
 
/*
 * Compute the hash code associated with a LOCKTAG.
 *
 * To avoid unnecessary recomputations of the hash code, we try to do this
 * just once per function, and then pass it around as needed. Aside from
 * passing the hashcode to hash_search_with_hash_value(), we can extract
 * the lock partition number from the hashcode.
 */
uint32
LockTagHashCode(const LOCKTAG *locktag)
{
	return get_hash_value(LockMethodLockHash, locktag);
}

The PROCLOCK table uses a different hash function, proclock_hash(), and the reason is a lovely bit of engineering: the proclock table must use the same partition locks as the LOCK table.

 c
 
 
/*
 * Compute the hash code associated with a PROCLOCKTAG.
 *
 * Because we want to use just one set of partition locks for both the
 * LOCK and PROCLOCK hash tables, we have to make sure that PROCLOCKs
 * fall into the same partition number as their associated LOCKs.
 */

Both tables are looked up while holding a single partition lock, so one LWLockAcquire covers both.

The per-backend local lock table

Every backend also keeps a local, non-shared hash table of the locks it holds. src/include/storage/lock.h explains why:

 c
 
 
/*
 * Each backend also maintains a local hash table with information about each
 * lock it is currently interested in. In particular the local table counts
 * the number of times that lock has been acquired. This allows multiple
 * requests for the same lock to be executed without additional accesses to
 * shared memory. We also track the number of lock acquisitions per
 * ResourceOwner, so that we can release just those locks belonging to a
 * particular ResourceOwner.
 *
 * When holding a lock taken "normally", the lock and proclock fields always
 * point to the associated objects in shared memory. However, if we acquired
 * the lock via the fast-path mechanism, the lock and proclock fields are set
 * to NULL, since there probably aren't any such objects in shared memory.
 * (If the lock later gets promoted to normal representation, we may eventually
 * update our locallock's lock/proclock fields after finding the shared
 * objects.)
 *
 * Caution: a locallock object can be left over from a failed lock acquisition
 * attempt. In this case its lock/proclock fields are untrustworthy, since
 * the shared lock object is neither held nor awaited, and hence is available
 * to be reclaimed. If nLocks > 0 then these pointers must either be valid or
 * NULL, but when nLocks == 0 they should be considered garbage.
 */

The struct:

 c
 
 
typedef struct LOCALLOCK
{
	/* tag */
	LOCALLOCKTAG tag;			/* unique identifier of locallock entry */

	/* data */
	uint32		hashcode;		/* copy of LOCKTAG's hash value */
	LOCK	 *lock;			/* associated LOCK object, if any */
	PROCLOCK *proclock;		/* associated PROCLOCK object, if any */
	int64		nLocks;			/* total number of times lock is held */
	int			numLockOwners;	/* # of relevant ResourceOwners */
	int			maxLockOwners;	/* allocated size of array */
	LOCALLOCKOWNER *lockOwners; /* dynamically resizable array */
	bool		holdsStrongLockCount;	/* bumped FastPathStrongRelationLocks */
	bool		lockCleared;	/* we read all sinval msgs for lock */
} LOCALLOCK;

Note that the local table is keyed by tag and mode:

 c
 
 
typedef struct LOCALLOCKTAG
{
	LOCKTAG		lock;			/* identifies the lockable object */
	LOCKMODE	mode;			/* lock mode for this table entry */
} LOCALLOCKTAG;

which is how a backend can hold AccessShareLock and RowExclusiveLock on the same table as two separate entries with separate counts.

It’s created in InitLockManagerAccess():

 c
 
 
	info.keysize = sizeof(LOCALLOCKTAG);
	info.entrysize = sizeof(LOCALLOCK);

	LockMethodLocalHash = hash_create("LOCALLOCK hash",
									 16,
									 &info,
									 HASH_ELEM | HASH_BLOBS);

The payoff is in LockAcquireExtended(): the very first thing it does after finding the locallock is check whether the lock is already held, and if so, bump a counter and return without touching shared memory at all:

 c
 
 
	/*
	 * If we already hold the lock, we can just increase the count locally.
	 *
	 * If lockCleared is already set, caller need not worry about absorbing
	 * sinval messages related to the lock's object.
	 */
	if (locallock->nLocks > 0)
	{
		GrantLockLocal(locallock, owner);
		if (locallock->lockCleared)
			return LOCKACQUIRE_ALREADY_CLEAR;
		else
			return LOCKACQUIRE_ALREADY_HELD;
	}

This is the single most important optimization in the whole lock manager for read-heavy workloads. A query that opens a cursor and reads a thousand rows takes AccessShareLock once, but the executor re-acquires the relation lock for every row it fetches in some code paths; with the local table, each re-acquisition is a local hash lookup and an integer increment.

The fast path

Now the fun part.

Where fast-path locks live

Fast-path locks are not in the lock table at all. They live in the backend’s own PGPROC struct, in shared memory, in two arrays:

 c
 
 
	/*-- recording fast-path locks taken by this backend. --*/
	LWLock		fpInfoLock;		/* protects per-backend fast-path state */
	uint64	 *fpLockBits;		/* lock modes held for each fast-path slot */
	Oid		 *fpRelId;		/* slots for rel oids */

fpRelId[f] is the relation OID in slot f; fpLockBits[f] is a bitmask of which lock modes are held on it. The arrays are indexed by group and slot:

 c
 
 
#define		FP_LOCK_GROUPS_PER_BACKEND_MAX	1024
#define		FP_LOCK_SLOTS_PER_GROUP		16	/* don't change */
#define		FastPathLockSlotsPerBackend() \
	(FP_LOCK_SLOTS_PER_GROUP * FastPathLockGroupsPerBackend)

and the group for a relation is a cheap hash:

 c
 
 
/*
 * Macros to calculate the fast-path group and index for a relation.
 *
 * The formula is a simple hash function, designed to spread the OIDs a bit,
 * so that even contiguous values end up in different groups. In most cases
 * there will be gaps anyway, but the multiplication should help a bit.
 *
 * The selected constant (49157) is a prime not too close to 2^k, and it's
 * small enough to not cause overflows (in 64-bit).
 *
 * We can assume that FastPathLockGroupsPerBackend is a power-of-two per
 * InitializeFastPathLocks().
 */
#define FAST_PATH_REL_GROUP(rel) \
	(((uint64) (rel) * 49157) & (FastPathLockGroupsPerBackend - 1))

So with the default configuration there are 8 groups of 16 slots each: 128 fast-path relation slots per backend, any of which can hold any of the three weak lock modes.

The bit packing is dense. Each slot uses 3 bits of the 64-bit word, and there are 3 usable lock modes (plus the offset):

 c
 
 
/* Macros for manipulating proc->fpLockBits */
#define FAST_PATH_BITS_PER_SLOT			3
#define FAST_PATH_LOCKNUMBER_OFFSET		1
#define FAST_PATH_MASK					((1 << FAST_PATH_BITS_PER_SLOT) - 1)

Which locks are eligible

 c
 
 
/*
 * The fast-path lock mechanism is concerned only with relation locks on
 * unshared relations by backends bound to a database. The fast-path
 * mechanism exists mostly to accelerate acquisition and release of locks
 * that rarely conflict. Because ShareUpdateExclusiveLock is
 * self-conflicting, it can't use the fast-path mechanism; but it also does
 * not conflict with any of the locks that do, so we can ignore it completely.
 */
#define EligibleForRelationFastPath(locktag, mode) \
	((locktag)->locktag_lockmethodid == DEFAULT_LOCKMETHOD && \
	(locktag)->locktag_type == LOCKTAG_RELATION && \
	(locktag)->locktag_field1 == MyDatabaseId && \
	MyDatabaseId != InvalidOid && \
	(mode) < ShareUpdateExclusiveLock)
#define ConflictsWithRelationFastPath(locktag, mode) \
	((locktag)->locktag_lockmethodid == DEFAULT_LOCKMETHOD && \
	(locktag)->locktag_type == LOCKTAG_RELATION && \
	(locktag)->locktag_field1 != InvalidOid && \
	(mode) > ShareUpdateExclusiveLock)

Read the second macro carefully: it fires for modes strictly greater than ShareUpdateExclusiveLock, i.e. ShareLock, ShareRowExclusiveLock, ExclusiveLock, AccessExclusiveLock. Those are the modes that conflict with the fast-path modes, and so they are the modes that must disable the fast path.

ShareUpdateExclusiveLock itself is excluded from both: it’s self-conflicting, so it can’t go on the fast path, but it doesn’t conflict with anything that can, so it doesn’t have to disable it either. Neat.

Note also that the fast path requires locktag_field1 == MyDatabaseId — locks on shared relations (locktag_field1 == InvalidOid, e.g. pg_class catalogs) are never eligible, because they’re visible across databases and a fast-path slot in one backend’s PGPROC says nothing about another database’s sessions.

Granting a fast-path lock

The grant path is in LockAcquireExtended():

 c
 
 
	if (EligibleForRelationFastPath(locktag, lockmode))
	{
		if (FastPathLocalUseCounts[FAST_PATH_REL_GROUP(locktag->locktag_field2)] <
			FP_LOCK_SLOTS_PER_GROUP)
		{
			uint32		fasthashcode = FastPathStrongLockHashPartition(hashcode);
			bool		acquired;

			/*
			 * LWLockAcquire acts as a memory sequencing point, so it's safe
			 * to assume that any strong locker whose increment to
			 * FastPathStrongRelationLocks becomes visible after we test it
			 * has yet to begin to transfer fast-path locks.
			 */
			LWLockAcquire(&MyProc->fpInfoLock, LW_EXCLUSIVE);
			if (pg_atomic_read_u32(&FastPathStrongRelationLocks[fasthashcode]) != 0)
				acquired = false;
			else
				acquired = FastPathGrantRelationLock(locktag->locktag_field2,
													 lockmode);
			LWLockRelease(&MyProc->fpInfoLock);
			if (acquired)
			{
				/*
				 * The locallock might contain stale pointers to some old
				 * shared objects; we MUST reset these to null before
				 * considering the lock to be acquired via fast-path.
				 */
				locallock->lock = NULL;
				locallock->proclock = NULL;
				GrantLockLocal(locallock, owner);
				return LOCKACQUIRE_OK;
			}
		}
		else
		{
			/*
			 * Increment the lock statistics counter if lock could not be
			 * acquired via the fast-path.
			 */
			pgstat_count_lock_fastpath_exceeded(locallock->tag.lock.locktag_type);
		}
	}

Notice what was not taken: the partition LWLock. The only lock acquired is MyProc->fpInfoLock, which protects this backend’s own fast-path state. Nobody else touches those slots — unless someone requests a strong lock, at which point they have to come and get them.

FastPathGrantRelationLock() is the actual slot allocation:

 c
 
 
static bool
FastPathGrantRelationLock(Oid relid, LOCKMODE lockmode)
{
	uint32		i;
	uint32		unused_slot = FastPathLockSlotsPerBackend();

	/* fast-path group the lock belongs to */
	uint32		group = FAST_PATH_REL_GROUP(relid);

	/* Scan for existing entry for this relid, remembering empty slot. */
	for (i = 0; i < FP_LOCK_SLOTS_PER_GROUP; i++)
	{
		/* index into the whole per-backend array */
		uint32		f = FAST_PATH_SLOT(group, i);

		if (FAST_PATH_GET_BITS(MyProc, f) == 0)
			unused_slot = f;
		else if (MyProc->fpRelId[f] == relid)
		{
			Assert(!FAST_PATH_CHECK_LOCKMODE(MyProc, f, lockmode));
			FAST_PATH_SET_LOCKMODE(MyProc, f, lockmode);
			return true;
		}
	}

	/* If no existing entry, use any empty slot. */
	if (unused_slot < FastPathLockSlotsPerBackend())
	{
		MyProc->fpRelId[unused_slot] = relid;
		FAST_PATH_SET_LOCKMODE(MyProc, unused_slot, lockmode);
		++FastPathLocalUseCounts[group];
		return true;
	}

	/* No existing entry, and no empty slot. */
	return false;
}

A scan of 16 slots within one group, and — crucially — a scan of one group, not of the whole array. The hash function is what keeps this O(1): you only look at the 16 slots that could possibly hold this relation. This is a very different shape from the heavyweight path, where you must take a partition lock and do a hash table lookup.

Note that FastPathGrantRelationLock returning false (no free slot) is not an error: the caller falls through to the heavyweight path. The fast path is an optimization, and running out of slots just means you take the slow path.

Releasing a fast-path lock

Symmetrically, in LockRelease():

 c
 
 
	/* Attempt fast release of any lock eligible for the fast path. */
	if (EligibleForRelationFastPath(locktag, lockmode) &&
		FastPathLocalUseCounts[FAST_PATH_REL_GROUP(locktag->locktag_field2)] > 0)
	{
		bool		released;

		/*
		 * We might not find the lock here, even if we originally entered it
		 * here. Another backend may have moved it to the main table.
		 */
		LWLockAcquire(&MyProc->fpInfoLock, LW_EXCLUSIVE);
		released = FastPathUnGrantRelationLock(locktag->locktag_field2,
											 lockmode);
		LWLockRelease(&MyProc->fpInfoLock);
		if (released)
		{
			RemoveLocalLock(locallock);
			return true;
		}
	}

That comment — “Another backend may have moved it to the main table” — is the whole story of the next section.

Upgrading from fast path to heavyweight

Here is the subtle part. Suppose session A holds a fast-path AccessShareLock on t, and session B wants AccessExclusiveLock on t. B’s request conflicts with the fast-path mode, so B must:

disable the fast path for that relation, so nobody takes a fast-path lock that B can’t see; and
transfer any existing fast-path locks on that relation into the heavyweight table, so that B’s conflict check can see them.

Step 1 is a counter array:

 c
 
 
/*
 * To make the fast-path lock mechanism work, we must have some way of
 * preventing the use of the fast-path when a conflicting lock might be
 * present. We partition* the locktag space into FAST_PATH_STRONG_LOCK_HASH_PARTITIONS,
 * and maintain an integer count of the number of "strong" lockers
 * in each partition. When any "strong" lockers are present (which is
 * hopefully not very often), the fast-path mechanism can't be used, and we
 * must fall back to the slower method of pushing matching locks directly
 * into the main lock tables.
 *
 * The deadlock detector does not know anything about the fast path mechanism,
 * so any locks that might be involved in a deadlock must be transferred from
 * the fast-path queues to the main lock table.
 */

#define FAST_PATH_STRONG_LOCK_HASH_BITS			10
#define FAST_PATH_STRONG_LOCK_HASH_PARTITIONS \
	(1 << FAST_PATH_STRONG_LOCK_HASH_BITS)
#define FastPathStrongLockHashPartition(hashcode) \
	((hashcode) % FAST_PATH_STRONG_LOCK_HASH_PARTITIONS)

static pg_atomic_uint32 *FastPathStrongRelationLocks;

1024 partitions of a counter array. The counter is checked by every fast-path acquisition, and it is the mechanism that makes the whole thing safe.

Step 2 is FastPathTransferRelationLocks(), which walks every backend:

 c
 
 
static bool
FastPathTransferRelationLocks(LockMethod lockMethodTable, const LOCKTAG *locktag,
							 uint32 hashcode)
{
	LWLock	 *partitionLock = LockHashPartitionLock(hashcode);
	Oid			relid = locktag->locktag_field2;
	uint32		i;

	/* fast-path group the lock belongs to */
	uint32		group = FAST_PATH_REL_GROUP(relid);

	/*
	 * Every PGPROC that can potentially hold a fast-path lock is present in
	 * ProcGlobal->allProcs. Prepared transactions are not, but any
	 * outstanding fast-path locks held by prepared transactions are
	 * transferred to the main lock table.
	 */
	for (i = 0; i < ProcGlobal->allProcCount; i++)
	{
		PGPROC	 *proc = GetPGProcByNumber(i);
		uint32		j;

		LWLockAcquire(&proc->fpInfoLock, LW_EXCLUSIVE);
		...
		for (j = 0; j < FP_LOCK_SLOTS_PER_GROUP; j++)
		{
			uint32		lockmode;

			/* index into the whole per-backend array */
			uint32		f = FAST_PATH_SLOT(group, j);

			/* Look for an allocated slot matching the given relid. */
			if (relid != proc->fpRelId[f] || FAST_PATH_GET_BITS(proc, f) == 0)
				continue;

			/* Find or create lock object. */
			LWLockAcquire(partitionLock, LW_EXCLUSIVE);
			for (lockmode = FAST_PATH_LOCKNUMBER_OFFSET;
				 lockmode < FAST_PATH_LOCKNUMBER_OFFSET + FAST_PATH_BITS_PER_SLOT;
				 ++lockmode)
			{
				PROCLOCK *proclock;

				if (!FAST_PATH_CHECK_LOCKMODE(proc, f, lockmode))
					continue;
				proclock = SetupLockInTable(lockMethodTable, proc, locktag,
											hashcode, lockmode);
				...
				GrantLock(proclock->tag.myLock, proclock, lockmode);
				FAST_PATH_CLEAR_LOCKMODE(proc, f, lockmode);
			}
			LWLockRelease(partitionLock);
			...
		}
		LWLockRelease(&proc->fpInfoLock);
	}
	return true;
}

Yes: to take a strong lock on a table, you walk every backend in the cluster. That is O(MaxBackends), and it is why the fast path is a bet that strong locks are rare.

But notice the early exit:

 c
 
 
		/*
		 * If the target backend isn't referencing the same database as the
		 * lock, then we needn't examine the individual relation IDs at all;
		 * none of them can be relevant.
		 *
		 * proc->databaseId is set at backend startup time and never changes
		 * thereafter, so it might be safe to perform this test before
		 * acquiring &proc->fpInfoLock. ...
		 *
		 * Also skip groups without any registered fast-path locks.
		 */
		if (proc->databaseId != locktag->locktag_field1 ||
			proc->fpLockBits[group] == 0)
		{
			LWLockRelease(&proc->fpInfoLock);
			continue;
		}

Two skips: backends in other databases (their locks can never be on this relation), and backends with nothing in this group. In practice the second skip kills almost all of the work.

And when the transferring backend is the same one that holds the fast-path lock, there’s a third path. If you hold a fast-path AccessShareLock and then ask for RowExclusiveLock on the same table, your new request is not fast-path eligible (well, RowExclusiveLock is weak, so it would take another fast-path slot)… but if you ask for AccessExclusiveLock, your own fast-path lock has to be visible to the heavyweight machinery, and the transfer loop handles that — it walks all backends including you.

The strong-lock counter

The FastPathStrongRelationLocks array is incremented by BeginStrongLockAcquire():

 c
 
 
/*
 * BeginStrongLockAcquire - inhibit use of fastpath for a given LOCALLOCK,
 * and arrange for error cleanup if it fails
 */
static void
BeginStrongLockAcquire(LOCALLOCK *locallock, uint32 fasthashcode)
{
	Assert(StrongLockInProgress == NULL);
	Assert(locallock->holdsStrongLockCount == false);

	pg_atomic_fetch_add_u32(&FastPathStrongRelationLocks[fasthashcode], 1);
	locallock->holdsStrongLockCount = true;
	StrongLockInProgress = locallock;
}

and — importantly — it is not decremented when the acquisition finishes. FinishStrongLockAcquire() only clears the cleanup hook:

 c
 
 
/*
 * FinishStrongLockAcquire - cancel pending cleanup for a strong lock
 * acquisition once it's no longer needed
 */
static void
FinishStrongLockAcquire(void)
{
	StrongLockInProgress = NULL;
}

The decrement happens when the locallock entry is finally removed — that is, when the strong lock is released:

 c
 
 
	if (locallock->holdsStrongLockCount)
	{
		uint32		fasthashcode;

		fasthashcode = FastPathStrongLockHashPartition(locallock->hashcode);

		Assert(pg_atomic_read_u32(&FastPathStrongRelationLocks[fasthashcode]) > 0);
		pg_atomic_fetch_sub_u32(&FastPathStrongRelationLocks[fasthashcode], 1);
		locallock->holdsStrongLockCount = false;
	}

So the semantics are: while a strong lock is held on a relation, the fast path is disabled for that hash partition. That is exactly right — a strong lock conflicts with the weak modes, so while it is outstanding, nobody may take a weak lock that the strong locker would fail to see. Every fast-path acquisition in that partition now fails the atomic check and falls back to the heavyweight table.

This is a partitioned disable, not a global one. A strong lock on one table only disables the fast path for the hash partition that table falls into.

Conflicts

The conflict table

Every lock method has a table of which modes conflict with which:

 c
 
 
static const LOCKMASK LockConflicts[] = {
	0,

	/* AccessShareLock */
	LOCKBIT_ON(AccessExclusiveLock),

	/* RowShareLock */
	LOCKBIT_ON(ExclusiveLock) | LOCKBIT_ON(AccessExclusiveLock),

	/* RowExclusiveLock */
	LOCKBIT_ON(ShareLock) | LOCKBIT_ON(ShareRowExclusiveLock) |
	LOCKBIT_ON(ExclusiveLock) | LOCKBIT_ON(AccessExclusiveLock),
	...

and the check is one AND against the aggregate mask:

 c
 
 
	/*
	 * first check for global conflicts: If no locks conflict with my request,
	 * then I get the lock.
	 *
	 * Checking for conflict: lock->grantMask represents the types of
	 * currently held locks. conflictTable[lockmode] has a bit set for each
	 * type of lock that conflicts with request. Bitwise compare tells if
	 * there is a conflict.
	 */
	if (!(conflictMask & lock->grantMask))
	{
		PROCLOCK_PRINT("LockCheckConflicts: no conflict", proclock);
		return false;
	}

If that’s non-zero, there’s potentially a conflict, and the code has to look at individual proclocks, because some of the conflicting locks might be held by me:

 c
 
 
	/*
	 * Rats. Something conflicts. But it could still be my own lock, or a
	 * lock held by another member of my locking group. First, figure out how
	 * many conflicts remain after subtracting out any locks I hold myself.
	 */
	myLocks = proclock->holdMask;
	for (i = 1; i <= numLockModes; i++)
	{
		if ((conflictMask & LOCKBIT_ON(i)) == 0)
		{
			conflictsRemaining[i] = 0;
			continue;
		}
		conflictsRemaining[i] = lock->granted[i];
		if (myLocks & LOCKBIT_ON(i))
			--conflictsRemaining[i];
		totalConflictsRemaining += conflictsRemaining[i];
	}

	/* If no conflicts remain, we get the lock. */
	if (totalConflictsRemaining == 0)
	{
		PROCLOCK_PRINT("LockCheckConflicts: resolved (simple)", proclock);
		return false;
	}

The granted[] array — a count per mode — is what makes this work without walking the proclock list. You know how many AccessExclusiveLocks are held; you subtract the ones you hold yourself; if the count is zero for every conflicting mode, you can have the lock.

Lock groups

If group locking is in play, you also have to subtract the locks held by other members of your group:

 c
 
 
	/*
	 * Locks held in conflicting modes by members of our own lock group are
	 * not real conflicts; we can subtract those out and see if we still have
	 * a conflict. This is O(N) in the number of processes holding or
	 * awaiting locks on this object. We could improve that by making the
	 * shared memory state more complex (and larger) but it doesn't seem worth
	 * it.
	 */
	dlist_foreach(proclock_iter, &lock->procLocks)
	{
		PROCLOCK *otherproclock =
			dlist_container(PROCLOCK, lockLink, proclock_iter.cur);

		if (proclock != otherproclock &&
			proclock->groupLeader == otherproclock->groupLeader &&
			(otherproclock->holdMask & conflictMask) != 0)
		{
			...
		}
	}

The groupLeader field on PROCLOCK is computed once, at creation:

 c
 
 
		/*
		 * It might seem unsafe to access proclock->groupLeader without a
		 * lock, but it's not really. Either we are initializing a proclock
		 * on our own behalf, in which case our group leader isn't changing
		 * because the group leader for a process can only ever be changed by
		 * the process itself; or else we are transferring a fast-path lock to
		 * the main lock table, in which case that process can't change its
		 * lock group leader without first releasing all of its locks (and in
		 * particular the one we are currently transferring).
		 */
		proclock->groupLeader = proc->lockGroupLeader != NULL ?
			proc->lockGroupLeader : proc;

A group is a leader PGPROC plus members. BecomeLockGroupLeader() creates one:

 c
 
 
/*
 * BecomeLockGroupLeader - designate process as lock group leader
 *
 * Once this function has returned, other processes can join the lock group
 * by calling BecomeLockGroupMember.
 */
void
BecomeLockGroupLeader(void)
{
	LWLock	 *leader_lwlock;

	/* If we already did it, we don't need to do it again. */
	if (MyProc->lockGroupLeader == MyProc)
		return;

	/* We had better not be a follower. */
	Assert(MyProc->lockGroupLeader == NULL);

	/* Create single-member group, containing only ourselves. */
	leader_lwlock = LockHashPartitionLockByProc(MyProc);
	LWLockAcquire(leader_lwlock, LW_EXCLUSIVE);
	MyProc->lockGroupLeader = MyProc;
	dlist_push_head(&MyProc->lockGroupMembers, &MyProc->lockGroupLink);
	LWLockRelease(leader_lwlock);
}

The group leader’s own partition lock protects the group’s membership fields:

 c
 
 
/*
 * The deadlock detector needs to be able to access lockGroupLeader and
 * related fields in the PGPROC, so we arrange for those fields to be protected
 * by one of the lock hash partition locks. Since the deadlock detector
 * acquires all such locks anyway, this makes it safe for it to access these
 * fields without doing anything extra. To avoid contention as much as
 * possible, we map different PGPROCs to different partition locks. The lock
 * used for a given lock group is determined by the group leader's pgprocno.
 */
#define LockHashPartitionLockByProc(leader_pgproc) \
	LockHashPartitionLock(GetNumberFromPGProc(leader_pgproc))

Groups exist so that a set of processes can hold conflicting locks on the same object without deadlocking each other — used by parallel query workers and by REPACK. The point is that a group is treated as one participant in the conflict and deadlock algorithms.

GetLockConflicts

Some callers (logical replication conflict resolution, for instance) need to know who is blocking them, not just that someone is. That’s GetLockConflicts():

 c
 
 
/*
 * GetLockConflicts
 *		Get an array of VirtualTransactionIds of xacts currently holding locks
 *		that would conflict with the specified lock/lockmode.
 *		xacts merely awaiting such a lock are NOT reported.
 *
 * The result array is palloc'd and is terminated with an invalid VXID.
 * *countp, if not null, is updated to the number of items set.
 *
 * Of course, the result could be out of date by the time it's returned, so
 * use of this function has to be thought about carefully. Similarly, a
 * PGPROC with no "lxid" will be considered non-conflicting regardless of any
 * lock it holds. ...
 */

The interesting part is that it has to check both the heavyweight table and every backend’s fast-path array:

 c
 
 
	/*
	 * Fast path locks might not have been entered in the primary lock table.
	 * If the lock we're dealing with could conflict with such a lock, we must
	 * examine each backend's fast-path array for conflicts.
	 */
	if (ConflictsWithRelationFastPath(locktag, lockmode))
	{
		Oid			relid = locktag->locktag_field2;
		...
		for (uint32 i = 0; i < ProcGlobal->allProcCount; i++)
		{
			PGPROC	 *proc = GetPGProcByNumber(i);
			...
			for (j = 0; j < FP_LOCK_SLOTS_PER_GROUP; j++)
			{
				...
				/* Look for an allocated slot matching the given relid. */
				if (relid != proc->fpRelId[f])
					continue;
				lockmask = FAST_PATH_GET_BITS(proc, f);
				if (!lockmask)
					continue;
				lockmask <<= FAST_PATH_LOCKNUMBER_OFFSET;

				/*
				 * There can only be one entry per relation, so if we found it
				 * and it doesn't conflict, we can skip the rest of the slots.
				 */
				if ((lockmask & conflictMask) == 0)
					break;

				/* Conflict! */
				GET_VXID_FROM_PGPROC(vxid, *proc);

				if (VirtualTransactionIdIsValid(vxid))
					vxids[count++] = vxid;
				/* else, xact already committed or aborted */
				break;
			}
			...
		}
	}

Note that it does not hold the partition lock while scanning the fast-path arrays — it takes each backend’s fpInfoLock in shared mode. The comment acknowledges the resulting fuzziness:

This is all a bit fuzzy, because new locks could be taken after we’ve visited a particular partition, but the callers had better be prepared to deal with that anyway, since the locks could equally well be taken between the time we return the value and the time the caller does something with it.

The wait queue

When a conflict is real, the waiter joins the lock’s queue.

JoinWaitQueue() (in src/backend/storage/lmgr/proc.c) is where the priority-queue logic lives:

 c
 
 
/*
 * JoinWaitQueue -- join the wait queue on the specified lock
 *
 * It's not actually guaranteed that we need to wait when this function is
 * called, because it could be that when we try to find a position at which
 * to insert ourself into the wait queue, we discover that we must be inserted
 * ahead of everyone who wants a lock that conflict with ours. In that case,
 * we get the lock immediately. Because of this, it's sensible for this function
 * to have a dontWait argument, despite the name.
 ...
 * NOTES: The process queue is now a priority queue for locking.
 */

The queue is a doubly-linked list with a “conflict mask ahead of me” accumulator. The interesting part is the reordering rule:

 c
 
 
	/*
	 * Determine where to add myself in the wait queue.
	 *
	 * Normally I should go at the end of the queue. However, if I already
	 * hold locks that conflict with the request of any previous waiter, put
	 * myself in the queue just in front of the first such waiter. This is not
	 * a necessary step, since deadlock detection would move me to before that
	 * waiter anyway; but it's relatively cheap to detect such a conflict
	 * immediately, and avoid delaying till deadlock timeout.
	 */

So the queue is not FIFO; it’s ordered by conflict relationships. A backend that would block an earlier waiter jumps ahead of it. This avoids a whole class of deadlocks that would otherwise only be detected by the (slow, one-second-granularity) deadlock timeout.

The insertion itself:

 c
 
 
	/*
	 * Insert self into queue, at the position determined above.
	 */
	if (insert_before)
		dclist_insert_before(waitQueue, &insert_before->waitLink, &MyProc->waitLink);
	else
		dclist_push_tail(waitQueue, &MyProc->waitLink);

	lock->waitMask |= LOCKBIT_ON(lockmode);

then the waiter sleeps in ProcSleep(), which is where the deadlock detector’s timer lives:

 c
 
 
/*
 * ProcSleep -- put process to sleep waiting on lock
 *
 * This must be called when JoinWaitQueue() returns PROC_WAIT_STATUS_WAITING.
 * Returns after the lock has been granted, or if a deadlock is detected. Can
 * also bail out with ereport(ERROR), if some other error condition, or a
 * timeout or cancellation is triggered.
 */

When a lock is released, ProcLockWakeup() walks the queue and wakes everyone who can now proceed:

 c
 
 
/*
 * ProcLockWakeup -- routine for waking up processes when a lock is
 *		released (or a prior waiter is aborted). Scan all waiters
 *	for lock, waken any that are no longer blocked.
 *
 * The appropriate lock partition lock must be held by caller.
 */
void
ProcLockWakeup(LockMethod lockMethodTable, LOCK *lock)
{
	dclist_head *waitQueue = &lock->waitProcs;
	LOCKMASK	aheadRequests = 0;
	dlist_mutable_iter miter;

	if (dclist_is_empty(waitQueue))
		return;

	dlist_foreach_modify(miter, waitQueue)
	{
		PGPROC	 *proc = dlist_container(PGPROC, waitLink, miter.cur);
		LOCKMODE	lockmode = proc->waitLockMode;

		/*
		 * Waken if (a) doesn't conflict with requests of earlier waiters, and
		 * (b) doesn't conflict with already-held locks.
		 */
		if ((lockMethodTable->conflictTab[lockmode] & aheadRequests) == 0 &&
			!LockCheckConflicts(lockMethodTable, lockmode, lock,
								proc->waitProcLock))
		{
			/* OK to waken */
			GrantLock(lock, proc->waitProcLock, lockmode);
			/* removes proc from the lock's waiting process queue */
			ProcWakeup(proc, PROC_WAIT_STATUS_OK);
		}
		else
		{
			/*
			 * Lock conflicts: Don't wake, but remember requested mode for
			 * later checks.
			 */
			aheadRequests |= LOCKBIT_ON(lockmode);
		}
	}
}

The aheadRequests accumulator is the queue’s ordering invariant, and it’s the same trick as in JoinWaitQueue(): a waiter is only woken if it conflicts with nothing ahead of it in the queue.

Deadlock detection

The deadlock detector lives in src/backend/storage/lmgr/deadlock.c and runs when deadlock_timeout (default 1 second) expires while waiting. It takes all the partition locks:

 c
 
 
	/*
	 * Acquire exclusive lock on the entire shared lock data structures. Must
	 * grab LWLocks in partition-number order to avoid LWLock deadlock.
	 *
	 * Note that the deadlock check interrupt had better not be enabled
	 * anywhere that this process itself holds lock partition locks, else this
	 * will wait forever. Also note that LWLockAcquire creates a critical
	 * section, so that this routine cannot be interrupted by cancel/die
	 * interrupts.
	 */
	for (i = 0; i < NUM_LOCK_PARTITIONS; i++)
		LWLockAcquire(LockHashPartitionLockByIndex(i), LW_EXCLUSIVE);

which is why the deadlock check is a heavyweight, rare operation and not something you want on the hot path.

Because the deadlock detector knows nothing about the fast path (as the comment in lock.c says explicitly: “The deadlock detector does not know anything about the fast path mechanism, so any locks that might be involved in a deadlock must be transferred from the fast-path queues to the main lock table”), the fast path is safe: a lock that might be involved in a deadlock is guaranteed to be in the heavyweight table by the time the detector looks.

Sizing the lock table

The shared lock table is sized from max_locks_per_transaction:

 c
 
 
#define NLOCKENTS() \
	mul_size(max_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))

and the GUC:

text

{ name => 'max_locks_per_transaction', type => 'int', context => 'PGC_POSTMASTER', group => 'LOCK_MANAGEMENT',
 short_desc => 'Sets the maximum number of locks per transaction.',
 long_desc => 'The shared lock table is sized on the assumption that at most "max_locks_per_transaction" objects per server process or prepared transaction will need to be locked at any one time.',
 variable => 'max_locks_per_xact',
 boot_val => '128',
 min => '10',
 max => 'INT_MAX',
},

So the table holds 128 × (MaxBackends + max_prepared_xacts) LOCK entries, and twice that many PROCLOCK entries. Running out produces:

 c
 
 
			if (reportMemoryError)
				ereport(ERROR,
						(errcode(ERRCODE_OUT_OF_MEMORY),
						 errmsg("out of shared memory"),
						 errhint("You might need to increase \"%s\".", "max_locks_per_transaction")));

The same GUC also sizes the fast path:

 c
 
 
void
InitializeFastPathLocks(void)
{
	/* Should be initialized only once. */
	Assert(FastPathLockGroupsPerBackend == 0);

	/*
	 * Based on the max_locks_per_transaction GUC, as that's a good indicator
	 * of the expected number of locks, figure out the value for
	 * FastPathLockGroupsPerBackend. This must be a power-of-two. We cap the
	 * value at FP_LOCK_GROUPS_PER_BACKEND_MAX and insist the value is at
	 * least 1.
	 *
	 * The default max_locks_per_transaction = 128 means 8 groups by default.
	 */
	FastPathLockGroupsPerBackend =
		Max(Min(pg_nextpower2_32(max_locks_per_xact) / FP_LOCK_SLOTS_PER_GROUP,
				FP_LOCK_GROUPS_PER_BACKEND_MAX), 1);

	/* Validate we did get a power-of-two */
	Assert(FastPathLockGroupsPerBackend ==
		 pg_nextpower2_32(FastPathLockGroupsPerBackend));
}

128 / 16 = 8 groups, 8 × 16 = 128 slots per backend. Raising max_locks_per_transaction gives both a bigger heavyweight table and more fast-path slots — the two are sized from the same knob because they’re sized from the same assumption.

Caveats

The fast path is only for relation locks in the current database. Shared relations (locktag_field1 == InvalidOid), objects, advisory locks, tuple locks, transaction locks — none of them are fast-path eligible.

The fast path is only for the three weak modes. AccessShareLock, RowShareLock, RowExclusiveLock. ShareUpdateExclusiveLock is excluded because it self-conflicts; everything stronger conflicts with the fast-path modes and therefore disables the fast path.

The fast path has a per-backend slot limit. With the default configuration that’s 128 relation slots. A session touching more than that many relations in one transaction falls back to the heavyweight table, and the fallback is counted:

 c
 
			pgstat_count_lock_fastpath_exceeded(locallock->tag.lock.locktag_type);

which is surfaced in the pg_stat_lock view, one row per lock tag type, with a fastpath_exceeded column.

Taking a strong lock walks every backend. FastPathTransferRelationLocks() iterates over ProcGlobal->allProcCount. On a cluster with hundreds of backends, a stream of ALTER TABLEs is measurably more expensive than the per-lock cost suggests.

Lock upgrades are heavyweight. Because the local lock table is keyed by (tag, mode), asking for a different mode on the same object is a different entry. If both modes are weak, the second one just sets another bit in the same fast-path slot; if the second mode is strong, the whole fast-path machinery for that relation is torn down and migrated into the heavyweight table. SELECT then UPDATE on the same table therefore takes two locks, not one.

Deadlock detection is a timeout, not a trap. deadlock_timeout defaults to 1 second. A deadlock is detected when the waiter notices the timeout and runs the detector, which then takes all sixteen partition locks. Deadlock detection is therefore not instantaneous, and it is not free.

max_locks_per_transaction is a startup parameter. It’s PGC_POSTMASTER, so changing it requires a restart, and it sizes shared memory, so raising it costs shared memory proportional to the number of backends.

Locks are not released on error mid-acquisition. Well — they are, but the locallock cleanup is subtle: the comment in lock.h warns that “a locallock object can be left over from a failed lock acquisition attempt”, in which case its lock/proclock pointers are garbage if nLocks == 0.

What it all means in practice

Most lock contention is imaginary. If you look at pg_locks and see thousands of AccessShareLock rows, that is not thousands of contended locks — those are just holders, and most of them are on the fast path. pg_locks shows both worlds, and the fastpath column tells you which is which:

 c
 
 
		TupleDescInitEntry(tupdesc, (AttrNumber) 15, "fastpath",
						 BOOLOID, -1, 0);

A row with fastpath = true was never in the heavyweight table at all.

The heavyweight table’s partitioning is why PostgreSQL scales. Sixteen independently-locked hash partitions. Two sessions locking different tables rarely touch the same LWLock. This is invisible in the API and completely visible in the throughput numbers.

The local lock table is why repeated locking is free. A backend that already holds a lock on an object bumps a counter instead of touching shared memory. The cost of “the executor re-locks the relation for every tuple” is a local hash lookup.

The fast path is a bet that strong locks are rare. The bet is paid out in full whenever a LOCK TABLE ... IN ACCESS EXCLUSIVE MODE or an ALTER TABLE shows up: the fast path is disabled for that hash partition and every backend’s fast-path locks on that relation are migrated into the heavyweight table. If your workload is dominated by DDL, the fast path buys you nothing and costs you the transfer.

max_locks_per_transaction is a shared-memory sizing knob, not a performance knob. Raising it gives you more heavyweight slots and more fast-path slots. It doesn’t make locking faster; it makes “out of shared memory” go away. If you’re seeing that error, the real question is why one transaction needs more than 128 locks — usually a partition of a table being attached, or a prepared transaction holding locks forever.

Deadlocks are a queue-ordering problem, not just a timeout problem. The wait queue is a priority queue keyed on conflict relationships, and JoinWaitQueue() deliberately jumps ahead of waiters you would block. This is why PostgreSQL rarely reports a deadlock that “shouldn’t” happen.

The fast path is invisible in the API and visible in the source. The whole mechanism is about not taking a lock. Every function that looks like it should be simple — FastPathGrantRelationLock, FastPathUnGrantRelationLock — is small, and the complexity is concentrated in the interaction between the two worlds: the strong-lock counter, the transfer loop, and the locallock->lock == NULL convention that means “this lock is on the fast path”.

Concurrency on Opensource internals

ClickHouse's Query Pipeline: Ports, Status Codes and the Executing Graph

The building block: IProcessor

The port: a mailbox with the flags in the pointer

prepare() and work()

Version counters: how a processor learns that a neighbour changed

The ExecutingGraph: nodes, edges, and who owns what

The graph can change shape at runtime

Trade-offs

Wrapping up

ClickHouse Runs a Query Pipeline on N Threads: Task Queues, Work Stealing and a Pool That Grows on Demand

Two kinds of parallelism

The shape of the executor

Task queues with work stealing

The local-task optimization

Measuring demand: pushTasks returns a number

CPU slots: max_threads is a ceiling, not a promise

Preemption and downscaling

The pulling mode: executeUntil

Trade-offs

Wrapping up

PostgreSQL Waits for Locks

The ladder, in one table

Layer 1: the spinlock

The hardware primitives

The free test

The pause instruction

The memory ordering contract

The generic spin loop

perform_spin_delay()

Why the randomness matters

The adaptive spin count

Stuck spinlock detection

Layer 2: the atomics

The x86 implementations

The API contract

The CAS loops

Memory barriers

The 64-bit fallback

Layer 3: the LWLock

The state word

The attempt

The wait list lock is a bit in the same word

Queue-self-then-retry

Sleeping: a POSIX semaphore, not a futex

Releasing and waking

Deadlock detection: not here

Layer 4: latches

SetLatch

ResetLatch

WaitLatch and the long-lived WaitEventSet

epoll and signalfd versus the classic self-pipe

The signalfd variant

The self-pipe variant

The whole picture

What it all means in practice

Further reading

PostgreSQL Locks Buffers Without LWLocks

What a content lock protects

The old world: an LWLock per buffer

The new world: one 64-bit word

The layout

The share-lock count is a counter, not a bit

The flag bits

The descriptor

Acquiring a content lock

The queue-self-then-retry algorithm

The public API

Lock ownership tracking

Releasing a content lock

The header lock is not an LWLock either

The share-exclusive level: hint bits and write-out

The lock-free clock sweep

The second chance, quantified

Caveats

This is a big, invasive change

The wait event taxonomy changed

A backend can only hold one lock at a time

The share-lock count is bounded by MAX_BACKENDS

Deadlock semantics are essentially unchanged

The building block: `IProcessor`

`prepare()` and `work()`

The `ExecutingGraph`: nodes, edges, and who owns what

Measuring demand: `pushTasks` returns a number

CPU slots: `max_threads` is a ceiling, not a promise

The pulling mode: `executeUntil`