How PostgreSQL Waits for Locks

Every lock in PostgreSQL is a bet. When a backend wants a resource that somebody else has, it has to choose: burn CPU waiting, or go to sleep and let the kernel schedule somebody else. Both are wrong sometimes. Spinning wastes a core and burns power; sleeping costs a context switch, wakes up in the wrong place, and — worst of all — adds the victim’s release time to the waiter’s latency.

PostgreSQL’s answer is a ladder. It spins in user space with an architecture-specific pause instruction, then sleeps for a randomized, exponentially growing interval, and only then blocks in the kernel on a POSIX semaphore. Every rung of that ladder is a separate piece of code with a separate set of trade-offs, and the interesting part of the design is where the rungs are.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

The ladder, in one table

Layer	Object	Waits by	Kernel involved?
1	Spinlock (`slock_t`)	`lock; xchgb` in a loop, then `pg_usleep()`	Only after ~1000 sleeps
2	Atomic (`pg_atomic_uint32`)	Compare-and-swap loop	No
3	LWLock	Spin, then block on a POSIX semaphore	Yes
4	Latch	Sleep in `epoll_wait()`/`poll()`	Yes
5	Heavyweight lock	Block on the lock manager’s semaphore	Yes

The ladder is not uniform, and that is the point. A spinlock protects something so small that the only correct thing to do is spin. An LWLock protects something that might take milliseconds. A latch is not a lock at all — it is a one-way door that a backend waits on until somebody opens it.

Layer 1: the spinlock

The hardware primitives

A spinlock is one byte. The contract is spelled out in src/include/storage/s_lock.h:

int TAS(slock_t *lock) Atomic test-and-set instruction. Attempt to acquire the lock, but do not wait. Returns 0 if successful, nonzero if unable to acquire the lock.
int TAS_SPIN(slock_t *lock) Like TAS(), but this version is used when waiting for a lock previously found to be contended. By default, this is the same as TAS(), but on some architectures it’s better to poll a contended lock using an unlocked instruction and retry the atomic test-and-set only when it appears free.

On x86_64 the whole thing is one instruction pair:

c
// src/include/storage/s_lock.h
static inline int
tas(volatile slock_t *lock)
{
	slock_t		_res = 1;

	__asm__ __volatile__(
	"	lock			\n"
	"	xchgb	%0,%1	\n"
:		"+q"(_res), "+m"(*lock)
:		/* no inputs */
:		"memory", "cc");
	return (int) _res;
}

lock xchg is a read-modify-write with the bus held, so it is atomic with respect to every other core. The old value comes back in %0, and the return value tells the caller whether it got the lock (old value zero) or found it taken.

The free test

The interesting part is TAS_SPIN, and on x86_64 it is a one-liner:

c
// src/include/storage/s_lock.h
/*
 * On Intel EM64T, it's a win to use a non-locking test before the xchg proper,
 * but only when spinning.
 *
 * See also Implementing Scalable Atomic Locks for Multi-Core Intel(tm) EM64T
 * and IA32, by Michael Chynoweth and Mary R. Lee. As of this writing, it is
 * available at:
 * http://software.intel.com/en-us/articles/implementing-scalable-atomic-locks-for-multi-core-intel-em64t-and-ia32-architectures
 */
#define TAS_SPIN(lock)    (*(lock) ? 1 : TAS(lock))

Read it as: if the lock byte is non-zero, it is taken, so return “failed” without touching the bus. Only if it looks free do we pay for the lock xchgb.

The reason this only happens while spinning is subtle. A lock-prefixed instruction on x86 is a full barrier: it drains the store buffer and serializes against other cores’ atomic operations. If you issue one for every failed attempt, you are asking the memory subsystem to arbitrate between every spinner on the machine — which is exactly the contention you are trying to avoid. A plain load is free (it can be satisfied from the local cache) and lets the spinner’s core keep running at full speed.

The “only when spinning” part is about the first acquisition. On the first attempt, the lock is very likely free, and the cost of the lock xchgb is paid anyway; doing a speculative plain load first would just add a branch and a potential cache miss on the fast path. And there is a correctness-adjacent reason too: the free test is a prediction. Between the load and the xchgb, another core could take the lock. That is fine — the xchgb is still the arbiter, and it will report the failure — but it means the free test can produce false “free” answers, and only the atomic instruction may decide.

The 32-bit x86 version takes the opposite approach and does the free test inside tas() itself:

c
// src/include/storage/s_lock.h
static inline int
tas(volatile slock_t *lock)
{
	slock_t		_res = 1;

	/*
	 * Use a non-locking test before asserting the bus lock.  Note that the
	 * extra test appears to be a small loss on some x86 platforms and a small
	 * win on others; it's by no means clear that we should keep it.
	 *
	 * When this was last tested, we didn't have separate TAS() and TAS_SPIN()
	 * macros.  Nowadays it probably would be better to do a non-locking test
	 * in TAS_SPIN() but not in TAS(), like on x86_64, but no-one's done the
	 * testing to verify that.  Without some empirical evidence, better to
	 * leave it alone.
	 */
	__asm__ __volatile__(
	"	cmpb	$0,%1		\n"
	"	jne		TAS%=_out	\n"
	"	lock				\n"
	"	xchgb	%0,%1		\n"
	"TAS%=_out: \n"
:		"+q"(_res), "+m"(*lock)
:		/* no inputs */
:		"memory", "cc");
	return (int) _res;
}

Note the comment’s honesty: “it’s by no means clear that we should keep it”. This is the kind of micro-optimization that is kept because it was measured once, long ago, and nobody has had the heart to re-measure it.

The pause instruction

Between attempts, the spinner has to do something. Doing nothing is bad: modern x86 cores issue speculatively and out of order, so a tight loop that keeps polling a memory location hammers the memory subsystem and the core’s pipeline. The answer is a hint instruction that tells the CPU “you are in a spin-wait loop”.

On x86 that is PAUSE, and PostgreSQL spells it the classic way:

c
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * Adding a PAUSE in the spin delay loop is demonstrably a no-op on
	 * Opteron, but it may be of some use on EM64T, so we keep it.
	 */
	__asm__ __volatile__(
	" rep; nop			\n");
}

The 32-bit x86 version carries the full Intel explanation:

c
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * This sequence is equivalent to the PAUSE instruction ("rep" is
	 * ignored by old IA32 processors if the following instruction is
	 * not a string operation); the IA-32 Architecture Software
	 * Developer's Manual, Vol. 3, Section 7.7.2 describes why using
	 * PAUSE in the inner loop of a spin lock is necessary for good
	 * performance:
	 *
	 *     The PAUSE instruction improves the performance of IA-32
	 *     processors supporting Hyper-Threading Technology when
	 *     executing spin-wait loops and other routines where one
	 *     thread is accessing a shared lock or semaphore in a tight
	 *     polling loop. When executing a spin-wait loop, the
	 *     processor can suffer a severe performance penalty when
	 *     exiting the loop because it detects a possible memory order
	 *     violation and flushes the core processor's pipeline. The
	 *     PAUSE instruction provides a hint to the processor that the
	 *     code sequence is a spin-wait loop. The processor uses this
	 *     hint to avoid the memory order violation and prevent the
	 *     pipeline flush. In addition, the PAUSE instruction
	 *     de-pipelines the spin-wait loop to prevent it from
	 *     consuming execution resources excessively.
	 */
	__asm__ __volatile__(
	" rep; nop			\n");
}

“De-pipelines the spin-wait loop to prevent it from consuming execution resources excessively” is the key phrase: PAUSE exists so that a spinning thread does not steal issue slots from the thread that is about to release the lock. On a hyper-threaded machine that matters enormously, because the spinner and the holder may share a physical core.

On ARM64 the answer is ISB:

c
// src/include/storage/s_lock.h
static inline void
spin_delay(void)
{
	/*
	 * Using an ISB instruction to delay in spinlock loops appears beneficial
	 * on high-core-count ARM64 processors.  It seems mostly a wash for smaller
	 * gear, and ISB doesn't exist at all on pre-v7 ARM chips.
	 */
	__asm__ __volatile__(
	" isb;				\n");
}

ISB flushes the instruction pipeline, which has the same effect as PAUSE: it stops the core from speculatively executing past the spin loop.

PowerPC is interesting because it does not define SPIN_DELAY() at all, so the generic fallback in s_lock.h applies:

c
// src/include/storage/s_lock.h
#if !defined(SPIN_DELAY)
#define SPIN_DELAY()	((void) 0)
#endif		/* SPIN_DELAY */

Instead, PowerPC gets its memory ordering from the lwarx/stwcx. compare-and-swap loop, and uses lwsync as its barrier:

c
// src/include/storage/s_lock.h
/*
 * NOTE: per the Enhanced PowerPC Architecture manual, v1.0 dated 7-May-2002,
 * an isync is a sufficient synchronization barrier after a lwarx/stwcx loop.
 * But if the spinlock is in ordinary memory, we can use lwsync instead for
 * better performance.
 */
static inline int
tas(volatile slock_t *lock)
{
	slock_t _t;
	int _res;

	__asm__ __volatile__(
"	lwarx   %0,0,%3,1	\n"
"	cmpwi   %0,0		\n"
"	bne     TAS%=_fail	\n"
"	addi    %0,%0,1		\n"
"	stwcx.  %0,0,%3		\n"
"	beq     TAS%=_ok	\n"
"TAS%=_fail: \n"
"	li      %1,1		\n"
"	b       TAS%=_out	\n"
"TAS%=_ok: \n"
"	lwsync				\n"
"	li      %1,0		\n"
"TAS%=_out: \n"
:	"=&b"(_t), "=r"(_res), "+m"(*lock)
:	"r"(lock)
:	"memory", "cc");
	return _res;
}

/*
 * PowerPC S_UNLOCK is almost standard but requires a "sync" instruction.
 * But we can use lwsync instead for better performance.
 */
#define S_UNLOCK(lock)	\
do \
{ \
	__asm__ __volatile__ ("	lwsync \n" ::: "memory"); \
	*((volatile slock_t *) (lock)) = 0; \
} while (0)

lwarx is “load with reservation” and stwcx. is “store with conditional”, which together implement a compare-and-swap on PowerPC’s older architectures. The lwsync after a successful acquisition is the acquire barrier; the lwsync before the store in S_UNLOCK is the release barrier.

The memory ordering contract

The header is unusually explicit about what the compiler and the hardware are allowed to do:

It is the responsibility of these macros to make sure that the compiler does not re-order accesses to shared memory to precede the actual lock acquisition, or follow the lock release.
On platforms with weak memory ordering, the TAS(), TAS_SPIN(), and S_UNLOCK() macros must further include hardware-level memory fence instructions to prevent similar re-ordering at the hardware level. TAS() and TAS_SPIN() must guarantee that loads and stores issued after the macro are not executed until the lock has been obtained. Conversely, S_UNLOCK() must guarantee that loads and stores issued before the macro have been executed before the lock is released.

Two different barriers are needed, and the difference is why S_UNLOCK() is not just *(lock) = 0 everywhere:

c
// src/include/storage/s_lock.h
/*
 * Default implementation of S_UNLOCK() for gcc/icc.
 *
 * Note that this implementation is unsafe for any platform that can reorder
 * a memory access (either load or store) after a following store.  That
 * happens not to be possible on x86 and most legacy architectures (some are
 * single-processor!), but many modern systems have weaker memory ordering.
 * Those that do must define their own version of S_UNLOCK() rather than
 * relying on this one.
 */
#if !defined(S_UNLOCK)
#define S_UNLOCK(lock)	\
	do { __asm__ __volatile__("" : : : "memory");  *(lock) = 0; } while (0)
#endif		/* !defined(S_UNLOCK) */

The default is a compiler barrier only. On x86 that is enough, because the hardware is total-store-ordered: stores cannot be reordered with other stores, and the lock prefix on the acquisition side prevents the hardware from moving anything past it. On ARM or PowerPC you need an explicit instruction, which is why those architectures define their own S_UNLOCK.

The generic spin loop

Everything above is platform-specific. The portable part is in src/backend/storage/lmgr/s_lock.c:

c
// src/backend/storage/lmgr/s_lock.c
#ifdef USE_DEFAULT_S_LOCK
/*
 * s_lock(lock) - platform-independent portion of waiting for a spinlock.
 */
int
s_lock(volatile slock_t *lock, const char *file, int line, const char *func)
{
	SpinDelayStatus delayStatus;

	init_spin_delay(&delayStatus, file, line, func);

	while (TAS_SPIN(lock))
	{
		perform_spin_delay(&delayStatus);
	}

	finish_spin_delay(&delayStatus);

	return delayStatus.delays;
}
#endif

The fast path is one TAS_SPIN that succeeds. The slow path calls perform_spin_delay() in a loop until the lock is acquired, and the number of delays is returned so that the caller can report it (this is how a “stuck spinlock” panic gets a sensible message).

perform_spin_delay()

This is the function that actually implements the ladder:

c
// src/backend/storage/lmgr/s_lock.c
/*
 * Wait while spinning on a contended spinlock.
 */
void
perform_spin_delay(SpinDelayStatus *status)
{
	/* CPU-specific delay each time through the loop */
	SPIN_DELAY();

	/* Block the process every spins_per_delay tries */
	if (++(status->spins) >= spins_per_delay)
	{
		if (++(status->delays) > NUM_DELAYS)
			s_lock_stuck(status->file, status->line, status->func);

		if (status->cur_delay == 0) /* first time to delay? */
			status->cur_delay = MIN_DELAY_USEC;

		/*
		 * Once we start sleeping, the overhead of reporting a wait event is
		 * justified. Actively spinning easily stands out in profilers, but
		 * sleeping with an exponential backoff is harder to spot...
		 *
		 * We might want to report something more granular at some point, but
		 * this is better than nothing.
		 */
		pgstat_report_wait_start(WAIT_EVENT_SPIN_DELAY);
		pg_usleep(status->cur_delay);
		pgstat_report_wait_end();

#if defined(S_LOCK_TEST)
		fprintf(stdout, "*");
		fflush(stdout);
#endif

		/* increase delay by a random fraction between 1X and 2X */
		status->cur_delay += (int) (status->cur_delay *
									pg_prng_double(&pg_global_prng_state) + 0.5);
		/* wrap back to minimum delay when max is exceeded */
		if (status->cur_delay > MAX_DELAY_USEC)
			status->cur_delay = MIN_DELAY_USEC;

		status->spins = 0;
	}
}

Three phases per iteration:

SPIN_DELAY() — the architecture-specific pause. Executed on every pass through the loop, not just every spins_per_delay tries.
Every spins_per_delay attempts, sleep for cur_delay microseconds via pg_usleep().
Multiply the delay by a random factor between 1× and 2×.

The constants:

c
// src/backend/storage/lmgr/s_lock.c
#define MIN_SPINS_PER_DELAY 10
#define MAX_SPINS_PER_DELAY 1000
#define NUM_DELAYS			1000
#define MIN_DELAY_USEC		1000L
#define MAX_DELAY_USEC		1000000L

So: spin up to spins_per_delay times (default 100), then sleep 1 ms, then sleep between 1 and 2 ms, then between 2 and 4 ms, and so on, capped at 1 second — and after 1000 delays, declare the spinlock stuck and PANIC.

Note the granularity of the backoff: cur_delay += (int) (cur_delay * pg_prng_double(...) + 0.5) multiplies by a random number in [1.0, 2.0), rounded. That is a jittered exponential backoff, and the jitter is the whole point.

Why the randomness matters

The file header explains the starvation problem:

Once we do decide to block, we use randomly increasing pg_usleep() delays. The first delay is 1 msec, then the delay randomly increases to about one second, after which we reset to 1 msec and start again. The idea here is that in the presence of heavy contention we need to increase the delay, else the spinlock holder may never get to run and release the lock. (Consider situation where spinlock holder has been nice’d down in priority by the scheduler — it will not get scheduled until all would-be acquirers are sleeping, so if we always use a 1-msec sleep, there is a real possibility of starvation.) But we can’t just clamp the delay to an upper bound, else it would take a long time to make a reasonable number of tries.

Two separate effects are at work:

Un-jittered backoff would synchronise the spinners. If every waiter sleeps for exactly 1 ms, they all wake up at exactly the same time, all collide, and all go back to sleep for exactly 1 ms. The lock holder gets one collision per millisecond instead of a smooth trickle of retries. Worse, the collisions mean the lock’s cache line ping-pongs between cores.
Un-jittered backoff starves the holder. On a machine with more runnable threads than cores, a spinner that wakes up every millisecond is runnable every millisecond. The holder, which needs the CPU to release the lock, gets scheduled less often the more spinners there are. The jitter breaks that pattern: spinners wake at uncorrelated times, so the holder gets a chance to run between them.

The pg_prng_double() call uses the backend’s global PRNG state, so the jitter differs between backends — two spinners on the same lock will not pick the same delay.

The adaptive spin count

How many times should we spin before sleeping? Too few and we pay a kernel round trip for a lock that would have been free in a microsecond; too many and we burn CPU on a lock that is never going to be released soon. PostgreSQL guesses, and then learns.

The guess starts at 100:

c
// src/include/storage/s_lock.h
/* Support for dynamic adjustment of spins_per_delay */
#define DEFAULT_SPINS_PER_DELAY  100

The learning happens in two steps. First, locally, at the end of each contended acquisition:

c
// src/backend/storage/lmgr/s_lock.c
/*
 * After acquiring a spinlock, update estimates about how long to loop.
 *
 * If we were able to acquire the lock without delaying, it's a good
 * indication we are in a multiprocessor.  If we had to delay, it's a sign
 * (but not a sure thing) that we are in a uniprocessor. Hence, we
 * decrement spins_per_delay slowly when we had to delay, and increase it
 * rapidly when we didn't.  It's expected that spins_per_delay will
 * converge to the minimum value on a uniprocessor and to the maximum
 * value on a multiprocessor.
 *
 * Note: spins_per_delay is local within our current process. We want to
 * average these observations across multiple backends, since it's
 * relatively rare for this function to even get entered, and so a single
 * backend might not live long enough to converge on a good value.  That
 * is handled by the two routines below.
 */
void
finish_spin_delay(SpinDelayStatus *status)
{
	if (status->cur_delay == 0)
	{
		/* we never had to delay */
		if (spins_per_delay < MAX_SPINS_PER_DELAY)
			spins_per_delay = Min(spins_per_delay + 100, MAX_SPINS_PER_DELAY);
	}
	else
	{
		if (spins_per_delay > MIN_SPINS_PER_DELAY)
			spins_per_delay = Max(spins_per_delay - 1, MIN_SPINS_PER_DELAY);
	}
}

The asymmetry is the interesting part: +100 when we got the lock without sleeping, −1 when we had to sleep. The reasoning is that “we never had to delay” is a strong, cheap signal that spinning is worth it, whereas “we had to delay” is a weak signal that it is not — maybe the lock holder was descheduled, maybe it was just unlucky. So the estimate ratchets up quickly and down slowly.

Second, globally. The local value is folded into a shared estimate at backend exit:

c
// src/backend/storage/lmgr/s_lock.c
/*
 * Update shared estimate of spins_per_delay during backend exit.
 *
 * NB: this has to be pretty fast as it is called while holding a spinlock
 */
int
update_spins_per_delay(int shared_spins_per_delay)
{
	/*
	 * We use an exponential moving average with a relatively slow adaption
	 * rate, so that noise in any one backend's result won't affect the shared
	 * value too much.  As long as both inputs are within the allowed range,
	 * the result must be too, so we need not worry about clamping the result.
	 *
	 * We deliberately truncate rather than rounding; this is so that single
	 * adjustments inside a backend can affect the shared estimate (see the
	 * asymmetric adjustment rules above).
	 */
	return (shared_spins_per_delay * 15 + spins_per_delay) / 16;
}

(shared * 15 + local) / 16 is an exponential moving average with a weight of 1/16 on the new sample. So a single backend’s experience moves the shared estimate by at most 1/16 of the distance between the two values — and with the truncation, it moves it by at least one unit, which is why the comment mentions “we deliberately truncate rather than rounding”.

The shared value lives in ProcGlobal->spins_per_delay and is read into each backend’s local copy at startup:

c
// src/backend/storage/lmgr/proc.c
	/* Update shared estimate of spins_per_delay */
	ProcGlobal->spins_per_delay = update_spins_per_delay(ProcGlobal->spins_per_delay);

	SpinLockRelease(&ProcGlobal->freeProcsLock);

So the estimate is a genuinely global, slowly-adapting, cross-backend number — and it converges, over time, to 1000 (spin hard) on a machine where locks are usually free, or to 10 (give up quickly) on a machine where they are not.

Stuck spinlock detection

If the lock is never released, the loop must terminate. After NUM_DELAYS (1000) sleeps, the backend panics:

c
// src/backend/storage/lmgr/s_lock.c
/*
 * s_lock_stuck() - complain about a stuck spinlock
 */
static void
s_lock_stuck(const char *file, int line, const char *func)
{
	if (!func)
		func = "(unknown)";
#if defined(S_LOCK_TEST)
	fprintf(stderr,
			"\nStuck spinlock detected at %s, %s:%d.\n",
			func, file, line);
	exit(1);
#else
	elog(PANIC, "stuck spinlock detected at %s, %s:%d",
		 func, file, line);
#endif
}

The file header explains the timeout philosophy:

We time out and declare error after NUM_DELAYS delays (thus, exactly that many tries). With the given settings, this will usually take 2 or so minutes. It seems better to fix the total number of tries (and thus the probability of unintended failure) than to fix the total time spent.

Fixing the number of tries rather than the elapsed time means the failure probability does not depend on how loaded the machine is. And the message includes __FILE__, __LINE__ and __func__ of the spinlock’s acquisition site, which is the first thing you want to know when the postmaster takes a PANIC.

Note also that the spin delay is a wait event:

c
// src/backend/storage/lmgr/s_lock.c
		pgstat_report_wait_start(WAIT_EVENT_SPIN_DELAY);
		pg_usleep(status->cur_delay);
		pgstat_report_wait_end();

The event is called SpinDelay in the wait event catalogue (src/backend/utils/activity/wait_event_names.txt):

text

SPIN_DELAY	"Waiting while acquiring a contended spinlock."

So pg_stat_activity will show a backend in SpinDelay state — but only when it is sleeping, not when it is actively spinning. The comment in the code explains why the reporting is only done around the sleep: “Actively spinning easily stands out in profilers, but sleeping with an exponential backoff is harder to spot”. If you see a backend burning CPU with no wait event at all, that is a spinner.

Layer 2: the atomics

Spinlocks are the blunt instrument. PostgreSQL also has a proper compare-and-swap API, in src/include/port/atomics.h and the per-architecture headers under src/include/port/atomics/.

The x86 implementations

c
// src/include/port/atomics/arch-x86.h
#define PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32
static inline bool
pg_atomic_compare_exchange_u32_impl(volatile pg_atomic_uint32 *ptr,
									uint32 *expected, uint32 newval)
{
	char	ret;

	/*
	 * Perform cmpxchg and use the zero flag which it implicitly sets when
	 * equal to measure the success.
	 */
	__asm__ __volatile__(
	"	lock				\n"
	"	cmpxchgl	%4,%5	\n"
	"   setz		%2		\n"
:		"=a" (*expected), "=m"(ptr->value), "=q" (ret)
:		"a" (*expected), "r" (newval), "m"(ptr->value)
:		"memory", "cc");
	return (bool) ret;
}

#define PG_HAVE_ATOMIC_FETCH_ADD_U32
static inline uint32
pg_atomic_fetch_add_u32_impl(volatile pg_atomic_uint32 *ptr, int32 add_)
{
	uint32 res;
	__asm__ __volatile__(
	"	lock				\n"
	"	xaddl	%0,%1		\n"
:		"=q"(res), "=m"(ptr->value)
:		"0" (add_), "m"(ptr->value)
:		"memory", "cc");
	return res;
}

Two instructions, two ideas:

lock cmpxchgl compares the accumulator (%eax) with the destination; if they are equal, the destination is loaded with the source. The old destination value is written back into %eax on failure — which is where the “expected” value comes from. The lock prefix makes the whole thing atomic.
setz reads the zero flag that cmpxchg implicitly sets when the comparison succeeded, and turns it into a 0-or-1 byte. This is the trick the comment refers to: “Perform cmpxchg and use the zero flag which it implicitly sets when equal to measure the success.” Rather than branching (which would be a mispredict), the flag is materialized as a value with setcc.
lock xaddl is “exchange and add”: it atomically adds the source to the destination and returns the old value. One instruction, no loop, no retry.

Note the memory clobbers: "memory", "cc". The memory clobber tells the compiler that the asm may read or write arbitrary memory, so it cannot keep anything in a register across it; the cc clobber tells it that the condition codes were modified.

The API contract

c
// src/include/port/atomics.h
/*
 * pg_atomic_compare_exchange_u32 - CAS operation
 *
 * Atomically compare the current value of ptr with *expected and store newval
 * iff ptr and *expected have the same value. The current value of *ptr will
 * always be stored in *expected.
 *
 * Return true if values have been exchanged, false otherwise.
 *
 * Full barrier semantics.
 */

That third sentence — “The current value of *ptr will always be stored in *expected” — is the reason the CAS-loop implementations of the other operations are so short.

The CAS loops

Not every operation has a dedicated instruction on every architecture, so generic.h builds them out of the CAS primitive:

c
// src/include/port/atomics/generic.h
#if !defined(PG_HAVE_ATOMIC_FETCH_AND_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_FETCH_AND_U32
static inline uint32
pg_atomic_fetch_and_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 and_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, old & and_))
		/* skip */;
	return old;
}
#endif

#if !defined(PG_HAVE_ATOMIC_FETCH_OR_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_FETCH_OR_U32
static inline uint32
pg_atomic_fetch_or_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 or_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, old | or_))
		/* skip */;
	return old;
}
#endif

The neat part is that the retry needs no extra load. A failed cmpxchg writes the current value of *ptr back into *expected, which is the same variable the loop uses as its source of truth. So after a failure, old already holds the fresh value, and the next iteration computes old & and_ (or old | or_) from it. The comment /* ok if read is not atomic */ on the initial load acknowledges the one non-atomic read in the function: it is only a starting guess, and the CAS loop corrects it immediately.

The same pattern gives you fetch_sub, exchange, and add_fetch:

c
// src/include/port/atomics/generic.h
#if !defined(PG_HAVE_ATOMIC_EXCHANGE_U32) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U32)
#define PG_HAVE_ATOMIC_EXCHANGE_U32
static inline uint32
pg_atomic_exchange_u32_impl(volatile pg_atomic_uint32 *ptr, uint32 xchg_)
{
	uint32 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u32_impl(ptr, &old, xchg_))
		/* skip */;
	return old;
}
#endif

Memory barriers

c
// src/include/port/atomics/arch-x86.h
/*
 * Both 32 and 64 bit x86 do not allow loads to be reordered with other loads,
 * or stores to be reordered with other stores, but a load can be performed
 * before a subsequent store.
 *
 * Technically, some x86-ish chips support uncached memory access and/or
 * special instructions that are weakly ordered.  In those cases we'd need
 * the read and write barriers to be lfence and sfence.  But since we don't
 * do those things, a compiler barrier should be enough.
 *
 * "lock; addl" has worked for longer than "mfence". It's also rumored to be
 * faster in many scenarios.
 */
#if defined(__GNUC__) || defined(__INTEL_COMPILER)
#if defined(__i386__)
#define pg_memory_barrier_impl()		\
	__asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory", "cc")
#elif defined(__x86_64__)
#define pg_memory_barrier_impl()		\
	__asm__ __volatile__ ("lock; addl $0,0(%%rsp)" : : : "memory", "cc")
#endif
#endif /* defined(__GNUC__) || defined(__INTEL_COMPILER) */

#define pg_read_barrier_impl()		pg_compiler_barrier_impl()
#define pg_write_barrier_impl()		pg_compiler_barrier_impl()

lock; addl $0, 0(%rsp) is the classic x86 full barrier: it adds zero to a stack location, but the lock prefix makes it an atomic read-modify-write, which is a full memory barrier. And on x86, a read or write barrier is just a compiler barrier, because the hardware’s memory model is strong enough that you never need more than that.

The 64-bit fallback

On a 32-bit platform without a 64-bit CAS instruction, fallback.h provides an emulated 64-bit atomic — and the emulation is, of course, a spinlock:

c
// src/include/port/atomics/fallback.h
/*
 * fallback.h
 *    Fallback for platforms without 64 bit atomics support. Slower
 *    than native atomics support, but not unusably slow.
 */

#if !defined(PG_HAVE_ATOMIC_U64_SUPPORT)

#define PG_HAVE_ATOMIC_U64_SIMULATION

#define PG_HAVE_ATOMIC_U64_SUPPORT
typedef struct pg_atomic_uint64
{
	int			sema;
	volatile uint64 value;
} pg_atomic_uint64;

And the implementation, in src/backend/port/atomics.c:

c
// src/backend/port/atomics.c
bool
pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
									uint64 *expected, uint64 newval)
{
	bool		ret;

	/*
	 * Do atomic op under a spinlock. It might look like we could just skip
	 * the cmpxchg if the lock isn't available, but that'd just emulate a
	 * 'weak' compare and swap. I.e. one that allows spurious failures. Since
	 * several algorithms rely on a strong variant and that is efficiently
	 * implementable on most major architectures let's emulate it here as
	 * well.
	 */
	SpinLockAcquire((slock_t *) &ptr->sema);

	/* perform compare/exchange logic */
	ret = ptr->value == *expected;
	*expected = ptr->value;
	if (ret)
		ptr->value = newval;

	/* and release lock */
	SpinLockRelease((slock_t *) &ptr->sema);

	return ret;
}

The comment is a nice piece of engineering reasoning: a lock-free implementation would be tempted to skip the compare-and-swap when the lock is contended, but that would be a weak CAS — one that may fail spuriously. Several algorithms in PostgreSQL depend on the strong variant, so the fallback implements the strong one, at the cost of spinning.

Note that this is a recursive layering: the 64-bit atomic’s implementation uses the 32-bit spinlock, which uses S_LOCK, which uses perform_spin_delay(). The whole ladder is recursive.

Layer 3: the LWLock

An LWLock is a shared-memory reader/writer lock. It is “lightweight” compared to the heavyweight lock manager in the sense that it has no owner, no lock table, no deadlock detection, and no interaction with pg_locks.

The state word

Everything about an LWLock’s state lives in one 32-bit word:

c
// src/include/storage/lmgr/lwlock.h
typedef struct LWLock
{
	uint16		tranche;		/* tranche ID */
	pg_atomic_uint32 state;		/* state of exclusive/nonexclusive lockers */
	proclist_head waiters;		/* list of waiting PGPROCs */
#ifdef LOCK_DEBUG
	pg_atomic_uint32 nwaiters;	/* number of waiters */
	struct PGPROC *owner;		/* last exclusive owner of the lock */
#endif
} LWLock;

And the layout of that word:

c
// src/backend/storage/lmgr/lwlock.c
#define LW_FLAG_HAS_WAITERS			((uint32) 1 << 31)
#define LW_FLAG_WAKE_IN_PROGRESS	((uint32) 1 << 30)
#define LW_FLAG_LOCKED				((uint32) 1 << 29)
#define LW_FLAG_BITS				3
#define LW_FLAG_MASK				(((1<<LW_FLAG_BITS)-1)<<(32-LW_FLAG_BITS))

/* assumes MAX_BACKENDS is a (power of 2) - 1, checked below */
#define LW_VAL_EXCLUSIVE			(MAX_BACKENDS + 1)
#define LW_VAL_SHARED				1

/* already (power of 2)-1, i.e. suitable for a mask */
#define LW_SHARED_MASK				MAX_BACKENDS
#define LW_LOCK_MASK				(MAX_BACKENDS | LW_VAL_EXCLUSIVE)

Three flags in the top three bits, and the bottom bits hold the holder count:

Bits	Meaning
31	`LW_FLAG_HAS_WAITERS` — somebody is queued
30	`LW_FLAG_WAKE_IN_PROGRESS` — a wakeup is in flight
29	`LW_FLAG_LOCKED` — the wait list is locked
28–0	shared holder count, or `LW_VAL_EXCLUSIVE` for an exclusive holder

The scheme is described in the file header:

The basic idea is to have a single atomic variable ’lockcount’ instead of the formerly separate shared and exclusive counters and to use atomic operations to acquire the lock. That’s fairly easy to do for plain rw-spinlocks, but a lot harder for something like LWLocks that want to wait in the OS.
For lock acquisition we use an atomic compare-and-exchange on the lockcount variable. For exclusive lock we swap in a sentinel value (LW_VAL_EXCLUSIVE), for shared locks we count the number of holders.
To release the lock we use an atomic decrement to release the lock. If the new value is zero (we get that atomically), we know we can/have to release waiters.

LW_VAL_EXCLUSIVE is MAX_BACKENDS + 1, chosen so that it cannot be confused with a shared holder count — there can never be more than MAX_BACKENDS shared holders, so the two ranges do not overlap. The static assertions check exactly that:

c
// src/backend/storage/lmgr/lwlock.c
StaticAssertDecl(((MAX_BACKENDS + 1) & MAX_BACKENDS) == 0,
				 "MAX_BACKENDS + 1 needs to be a power of 2");

StaticAssertDecl((MAX_BACKENDS & LW_FLAG_MASK) == 0,
				 "MAX_BACKENDS and LW_FLAG_MASK overlap");

StaticAssertDecl((LW_VAL_EXCLUSIVE & LW_FLAG_MASK) == 0,
				 "LW_VAL_EXCLUSIVE and LW_FLAG_MASK overlap");

The attempt

c
// src/backend/storage/lmgr/lwlock.c
/*
 * Internal function that tries to atomically acquire the lwlock in the passed
 * in mode.
 *
 * This function will not block waiting for a lock to become free - that's the
 * caller's job.
 *
 * Returns true if the lock isn't free and we need to wait.
 */
static bool
LWLockAttemptLock(LWLock *lock, LWLockMode mode)
{
	uint32		old_state;

	Assert(mode == LW_EXCLUSIVE || mode == LW_SHARED);

	/*
	 * Read once outside the loop, later iterations will get the newer value
	 * via compare & exchange.
	 */
	old_state = pg_atomic_read_u32(&lock->state);

	/* loop until we've determined whether we could acquire the lock or not */
	while (true)
	{
		uint32		desired_state;
		bool		lock_free;

		desired_state = old_state;

		if (mode == LW_EXCLUSIVE)
		{
			lock_free = (old_state & LW_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += LW_VAL_EXCLUSIVE;
		}
		else
		{
			lock_free = (old_state & LW_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += LW_VAL_SHARED;
		}

		/*
		 * Attempt to swap in the state we are expecting. If we didn't see
		 * lock to be free, that's just the old value. If we saw it as free,
		 * we'll attempt to mark it acquired. The reason that we always swap
		 * in the value is that this doubles as a memory barrier. We could try
		 * to be smarter and only swap in values if we saw the lock as free,
		 * but benchmark haven't shown it as beneficial so far.
		 *
		 * Retry if the value changed since we last looked at it.
		 */
		if (pg_atomic_compare_exchange_u32(&lock->state,
										   &old_state, desired_state))
		{
			if (lock_free)
			{
				/* Great! Got the lock. */
#ifdef LOCK_DEBUG
				if (mode == LW_EXCLUSIVE)
					lock->owner = MyProc;
#endif
				return false;
			}
			else
				return true;	/* somebody else has the lock */
		}
	}
	pg_unreachable();
}

The comment about the memory barrier is the interesting part:

The reason that we always swap in the value is that this doubles as a memory barrier. We could try to be smarter and only swap in values if we saw the lock as free, but benchmark haven’t shown it as beneficial so far.

On x86, lock cmpxchg is a full barrier whatever it does. So whether the CAS succeeds or fails, the CPU has executed a full fence. The code takes advantage of that unconditionally: it always issues the CAS, even in the case where it already knows the lock is held and it is going to have to wait anyway. The alternative — checking first and skipping the CAS — would save an atomic operation on the contended path, but benchmarks said it wasn’t worth it. (And on a weak-memory architecture, the unconditional CAS is what gives you the barrier for free; skipping it would require an explicit fence.)

Also note the loop structure: old_state is refreshed by the failed CAS (the “current value of *ptr will always be stored in *expected” contract), so the retry always operates on the freshest value.

The wait list lock is a bit in the same word

The LW_FLAG_LOCKED bit in the state word is a spinlock protecting the waiters list. This is a neat bit of economy: the wait list lock lives in the same cache line as the state word, so taking it does not cost an extra cache miss.

c
// src/backend/storage/lmgr/lwlock.c
/*
 * Lock the LWLock's wait list against concurrent activity.
 *
 * NB: even though the wait list is locked, non-conflicting lock operations
 * may still happen concurrently.
 *
 * Time spent holding mutex should be short!
 */
static void
LWLockWaitListLock(LWLock *lock)
{
	uint32		old_state;
#ifdef LWLOCK_STATS
	lwlock_stats *lwstats;
	uint32		delays = 0;

	lwstats = get_lwlock_stats_entry(lock);
#endif

	while (true)
	{
		/*
		 * Always try once to acquire the lock directly, without setting up
		 * the spin-delay infrastructure. The work necessary for that shows up
		 * in profiles and is rarely necessary.
		 */
		old_state = pg_atomic_fetch_or_u32(&lock->state, LW_FLAG_LOCKED);
		if (likely(!(old_state & LW_FLAG_LOCKED)))
			break;				/* got lock */

		/* and then spin without atomic operations until lock is released */
		{
			SpinDelayStatus delayStatus;

			init_local_spin_delay(&delayStatus);

			while (old_state & LW_FLAG_LOCKED)
			{
				perform_spin_delay(&delayStatus);
				old_state = pg_atomic_read_u32(&lock->state);
			}
#ifdef LWLOCK_STATS
			delays += delayStatus.delays;
#endif
			finish_spin_delay(&delayStatus);
		}

		/*
		 * Retry. The lock might obviously already be re-acquired by the time
		 * we're attempting to get it again.
		 */
	}

#ifdef LWLOCK_STATS
	lwstats->spin_delay_count += delays;
#endif
}

Two things to notice:

“Always try once to acquire the lock directly, without setting up the spin-delay infrastructure.” The SpinDelayStatus struct has to be initialised, and init_local_spin_delay() writes five fields. For the overwhelmingly common case where the wait list is not contended, that setup is pure overhead — and the comment says it “shows up in profiles”. So the code tries the atomic fetch-or first and only sets up the spin machinery if that fails.
“spin without atomic operations until lock is released” — inside the spin loop, the state word is read with a plain atomic read (pg_atomic_read_u32), not a CAS loop. There is no attempt to take the lock during the spin; the spinner just watches the bit. Only when the bit clears does it retry the fetch-or. That avoids hammering the cache line with atomic RMWs.

Queue-self-then-retry

Now the interesting algorithmic part. The file header describes the two-phase approach:

The attentive reader might have noticed that naively doing the above has a glaring race condition: We try to lock using the atomic operations and notice that we have to wait. Unfortunately by the time we have finished queuing, the former locker very well might have already finished its work. That’s problematic because we’re now stuck waiting inside the OS.
To mitigate those races we use a two phased attempt at locking: Phase 1: Try to do it atomically, if we succeed, nice Phase 2: Add ourselves to the waitqueue of the lock Phase 3: Try to grab the lock again, if we succeed, remove ourselves from the queue Phase 4: Sleep till wake-up, goto Phase 1

And the implementation:

c
// src/backend/storage/lmgr/lwlock.c
	for (;;)
	{
		bool		mustwait;

		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = LWLockAttemptLock(lock, mode);

		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "immediately acquired lock");
			break;				/* got the lock */
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		LWLockQueueSelf(lock, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = LWLockAttemptLock(lock, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "acquired, undoing queue");

			LWLockDequeueSelf(lock);
			break;
		}

		/*
		 * Wait until awakened.
		 *
		 * It is possible that we get awakened for a reason other than being
		 * signaled by LWLockRelease.  If so, loop back and wait again.  Once
		 * we've gotten the LWLock, re-increment the sema by the number of
		 * additional signals received.
		 */
		LOG_LWDEBUG("LWLockAcquire", lock, "waiting");

		LWLockReportWaitStart(lock);
		...
		for (;;)
		{
			PGSemaphoreLock(proc->sem);
			if (proc->lwWaiting == LW_WS_NOT_WAITING)
				break;
			extraWaits++;
		}

		/* Retrying, allow LWLockRelease to release waiters again. */
		pg_atomic_fetch_and_u32(&lock->state, ~LW_FLAG_WAKE_IN_PROGRESS);

		...
		/* Now loop back and try to acquire lock again. */
		result = false;
	}

The critical insight is in the comment: queue first, then try again. If you tried the lock, failed, and then queued, the holder could have released the lock in the window between your failed attempt and your queueing — and nobody would ever wake you up, because the release had already happened. By queueing before the second attempt, you guarantee that any release that happens after your first failed attempt will see you on the wait list and wake you.

The queueing itself:

c
// src/backend/storage/lmgr/lwlock.c
static void
LWLockQueueSelf(LWLock *lock, LWLockMode mode)
{
	/*
	 * If we don't have a PGPROC structure, there's no way to wait. This
	 * should never occur, since MyProc should only be null during shared
	 * memory initialization.
	 */
	if (MyProc == NULL)
		elog(PANIC, "cannot wait without a PGPROC structure");

	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

	LWLockWaitListLock(lock);

	/* setting the flag is protected by the spinlock */
	pg_atomic_fetch_or_u32(&lock->state, LW_FLAG_HAS_WAITERS);

	MyProc->lwWaiting = LW_WS_WAITING;
	MyProc->lwWaitMode = mode;

	/* LW_WAIT_UNTIL_FREE waiters are always at the front of the queue */
	if (mode == LW_WAIT_UNTIL_FREE)
		proclist_push_head(&lock->waiters, MyProcNumber, lwWaitLink);
	else
		proclist_push_tail(&lock->waiters, MyProcNumber, lwWaitLink);

	/* Can release the mutex now */
	LWLockWaitListUnlock(lock);
	...
}

Note the panic on double-queueing: elog(PANIC, "queueing for lock while waiting on another one"). An LWLock is a leaf lock — you may hold at most one while waiting for another.

Sleeping: a POSIX semaphore, not a futex

The actual blocking is PGSemaphoreLock(proc->sem) — a POSIX semaphore stored in shared memory, one per PGPROC:

c
// src/backend/port/posix_sema.c
/*
 * PGSemaphoreLock
 *
 * Lock a semaphore (decrement count), blocking if count would be < 0
 */
void
PGSemaphoreLock(PGSemaphore sema)
{
	int			errStatus;

	/* See notes in sysv_sema.c's implementation of PGSemaphoreLock. */
	do
	{
		errStatus = sem_wait(PG_SEM_REF(sema));
	} while (errStatus < 0 && errno == EINTR);

	if (errStatus < 0)
		elog(FATAL, "sem_wait failed: %m");
}

The semaphore is padded to a cache line to avoid false sharing:

c
// src/backend/port/posix_sema.c
typedef union SemTPadded
{
	sem_t		pgsem;
	char		pad[PG_CACHE_LINE_SIZE];
} SemTPadded;

There is no futex here. PostgreSQL does not implement the Linux futex fast path, so every LWLock wait that reaches the kernel costs a real sem_wait syscall, and every wakeup costs a sem_post. The semaphore implementations are chosen at configure time:

unnamed POSIX semaphores (sem_init/sem_wait), the default where available;
named POSIX semaphores (sem_open);
SysV semaphores (semop) as a fallback.

Releasing and waking

The release path wakes waiters, and it uses the LW_FLAG_WAKE_IN_PROGRESS bit to make sure only one waker is in flight at a time:

c
// src/backend/storage/lmgr/lwlock.c
	/* unset required flags, and release lock, in one fell swoop */
	{
		uint32		old_state;
		uint32		desired_state;

		old_state = pg_atomic_read_u32(&lock->state);
		while (true)
		{
			desired_state = old_state;

			/* compute desired flags */

			if (new_wake_in_progress)
				desired_state |= LW_FLAG_WAKE_IN_PROGRESS;
			else
				desired_state &= ~LW_FLAG_WAKE_IN_PROGRESS;

			if (proclist_is_empty(&lock->waiters))
				desired_state &= ~LW_FLAG_HAS_WAITERS;

			desired_state &= ~LW_FLAG_LOCKED;	/* release lock */

			if (pg_atomic_compare_exchange_u32(&lock->state, &old_state,
											   desired_state))
				break;
		}
	}

	/* Awaken any waiters I removed from the queue. */
	proclist_foreach_modify(iter, &wakeup, lwWaitLink)
	{
		PGPROC	   *waiter = GetPGProcByNumber(iter.cur);

		LOG_LWDEBUG("LWLockRelease", lock, "release waiter");
		proclist_delete(&wakeup, iter.cur, lwWaitLink);

		/*
		 * Guarantee that lwWaiting being unset only becomes visible once the
		 * unlink from the link has completed. Otherwise the target backend
		 * could be woken up for other reason and enqueue for a new lock - if
		 * that happens before the list unlink happens, the list would end up
		 * being corrupted.
		 *
		 * The barrier pairs with the LWLockWaitListLock() when enqueuing for
		 * another lock.
		 */
		pg_write_barrier();
		waiter->lwWaiting = LW_WS_NOT_WAITING;
		PGSemaphoreUnlock(waiter->sem);
	}

Two details: the flags are updated with a single CAS so that the wait-list lock is released and the flags are cleared atomically together (otherwise a racing LWLockQueueSelf() could set LW_FLAG_HAS_WAITERS after the releaser checked the list and before it released the lock, and the flag would be stuck on forever); and the write barrier before clearing lwWaiting ensures that the unlink from the wait list is visible before the waiter can wake up and try to queue for another lock.

Deadlock detection: not here

LWLocks have no deadlock detection, and this is not an oversight — it is a design decision. An LWLock is a leaf lock: it protects a small piece of shared state, and the code that holds it does not wait on other locks while holding it. There is no wait-for graph to inspect.

Deadlock detection lives in the heavyweight lock manager instead, in src/backend/storage/lmgr/deadlock.c, which maintains a wait-for graph over the heavyweight locks:

DeadLockCheck – Checks for deadlocks for a given process
This code looks for deadlocks involving the given process. If any are found, we try to resolve them by rearranging the wait queues. If resolution is impossible, return DS_HARD_DEADLOCK — the process must then throw an error.

The practical consequence: if an LWLock deadlock happens, PostgreSQL does not detect it. The backends sit there until deadlock_timeout fires — but that timeout applies to lock waits that the lock manager knows about, not to arbitrary LWLock waits.

Layer 4: latches

A latch is not a lock. It is a flag in shared memory plus a way to wake up whoever is waiting on it. MyLatch is the one every backend has, and it is what WaitLatch() waits on.

SetLatch

c
// src/backend/storage/ipc/latch.c
/*
 * Sets a latch and wakes up anyone waiting on it.
 *
 * This is cheap if the latch is already set, otherwise not so much.
 *
 * NB: when calling this in a signal handler, be sure to save and restore
 * errno around it.  (That's standard practice in most signal handlers, of
 * course, but we used to omit it in handlers that only set a flag.)
 *
 * NB: this function is called from critical sections and signal handlers so
 * throwing an error is not a good idea.
 */
void
SetLatch(Latch *latch)
{
	...
	/*
	 * The memory barrier has to be placed here to ensure that any flag
	 * variables possibly changed by this process have been flushed to main
	 * memory, before we check/set is_set.
	 */
	pg_memory_barrier();

	/* Quick exit if already set */
	if (latch->is_set)
		return;

	latch->is_set = true;

	pg_memory_barrier();
	if (!latch->maybe_sleeping)
		return;

	...
	owner_pid = latch->owner_pid;
	if (owner_pid == 0)
		return;
	else if (owner_pid == MyProcPid)
		WakeupMyProc();
	else
		WakeupOtherProc(owner_pid);
	...
}

Four things, in order:

A full memory barrier before the test. Any flag variables the process has changed must be visible in main memory before we publish is_set. Otherwise a waiter could see is_set == true, wake up, and look at flags that still have stale values. This is the classic Dekker-style store-load ordering problem, and the barrier is what makes it safe.
“Quick exit if already set.” Setting an already-set latch is a no-op. This is the common case in a signal handler that fires repeatedly, and it means SetLatch() is cheap enough to call from anywhere.
A second memory barrier after setting is_set. This makes the store to is_set visible before we look at maybe_sleeping.
The maybe_sleeping test. This is the optimisation that matters: if nobody is waiting on the latch, do not signal anybody. Signalling is expensive (a syscall, or a write to the self-pipe), and the overwhelmingly common case for SetLatch() is “set the flag and carry on”. The maybe_sleeping flag is set by the waiter just before it blocks:
c
```
		// src/backend/storage/ipc/waiteventset.c
		if (set->latch && !set->latch->is_set)
		{
			/* about to sleep on a latch */
			set->latch->maybe_sleeping = true;
			pg_memory_barrier();
			/* and recheck */
		}

		if (set->latch && set->latch->is_set)
		{
			...
		}
```
The waiter sets maybe_sleeping, issues a barrier, and then re-checks is_set. That re-check is the other half of the protocol: if the latch was set in the window between the waiter’s check and its setting of maybe_sleeping, the re-check catches it and the waiter never sleeps at all. If the latch is set after the re-check, the setter will see maybe_sleeping == true and will signal. Between the two, no wakeup is lost.
Note the ordering of the two flags: the waiter publishes maybe_sleeping before the final check of is_set, and the setter publishes is_set before the final check of maybe_sleeping. That is the standard “store both, then check both” pattern, and it is why there are two barriers in SetLatch().

ResetLatch

c
// src/backend/storage/ipc/latch.c
/*
 * Clear the latch. Calling WaitLatch after this will sleep, unless
 * the latch is set again before the WaitLatch call.
 */
void
ResetLatch(Latch *latch)
{
	/* Only the owner should reset the latch */
	Assert(latch->owner_pid == MyProcPid);
	Assert(latch->maybe_sleeping == false);

	latch->is_set = false;

	/*
	 * Ensure that the write to is_set gets flushed to main memory before we
	 * examine any flag variables.  Otherwise a concurrent SetLatch might
	 * falsely conclude that it needn't signal us, even though we have missed
	 * seeing some flag updates that SetLatch was supposed to inform us of.
	 */
	pg_memory_barrier();
}

Symmetrically, the barrier after clearing is_set ensures that the clear is visible before the owner examines any flags it might use to decide what to do next. Without it, a concurrent SetLatch() could see the stale is_set == true, decide “already set, nothing to do”, and skip the signal — while the owner has already decided to go to sleep.

WaitLatch and the long-lived WaitEventSet

c
// src/backend/storage/ipc/latch.c
/* A common WaitEventSet used to implement WaitLatch() */
static WaitEventSet *LatchWaitSet;

/* The positions of the latch and PM death events in LatchWaitSet */
#define LatchWaitSetLatchPos 0
#define LatchWaitSetPostmasterDeathPos 1

void
InitializeLatchWaitSet(void)
{
	...
	/* Set up the WaitEventSet used by WaitLatch(). */
	LatchWaitSet = CreateWaitEventSet(NULL, 2);
	latch_pos = AddWaitEventToSet(LatchWaitSet, WL_LATCH_SET, PGINVALID_SOCKET,
								  MyLatch, NULL);
	Assert(latch_pos == LatchWaitSetLatchPos);

	/*
	 * WaitLatch will modify this to WL_EXIT_ON_PM_DEATH or
	 * WL_POSTMASTER_DEATH on each call.
	 */
	if (IsUnderPostmaster)
	{
		latch_pos = AddWaitEventToSet(LatchWaitSet, WL_EXIT_ON_PM_DEATH,
									  PGINVALID_SOCKET, NULL, NULL);
		Assert(latch_pos == LatchWaitSetPostmasterDeathPos);
	}
}

One WaitEventSet, created once per backend, reused forever. Every WaitLatch() call does not create a new epoll set; it just modifies the existing one:

c
// src/backend/storage/ipc/latch.c
int
WaitLatch(Latch *latch, int wakeEvents, long timeout,
		  uint32 wait_event_info)
{
	WaitEvent	event;

	/* Postmaster-managed callers must handle postmaster death somehow. */
	Assert(!IsUnderPostmaster ||
		   (wakeEvents & WL_EXIT_ON_PM_DEATH) ||
		   (wakeEvents & WL_POSTMASTER_DEATH));

	/*
	 * Some callers may have a latch other than MyLatch, or no latch at all,
	 * or want to handle postmaster death differently.  It's cheap to assign
	 * those, so just do it every time.
	 */
	if (!(wakeEvents & WL_LATCH_SET))
		latch = NULL;
	ModifyWaitEvent(LatchWaitSet, LatchWaitSetLatchPos, WL_LATCH_SET, latch);

	if (IsUnderPostmaster)
		ModifyWaitEvent(LatchWaitSet, LatchWaitSetPostmasterDeathPos,
						(wakeEvents & (WL_EXIT_ON_PM_DEATH | WL_POSTMASTER_DEATH)),
						NULL);

	if (WaitEventSetWait(LatchWaitSet,
						 (wakeEvents & WL_TIMEOUT) ? timeout : -1,
						 &event, 1,
						 wait_event_info) == 0)
		return WL_TIMEOUT;
	else
		return event.events;
}

The comment “Some callers may have a latch other than MyLatch, or no latch at all, or want to handle postmaster death differently. It’s cheap to assign those, so just do it every time” is a nice micro-optimization philosophy: rather than maintaining several WaitEventSets and picking one, keep one and mutate it.

epoll and signalfd versus the classic self-pipe

waiteventset.c has three implementations of the same idea, and the file header explains the race each one has to solve:

c
// src/backend/storage/ipc/waiteventset.c
/*
 * The poll() implementation uses the so-called self-pipe trick to overcome the
 * race condition involved with poll() and setting a global flag in the signal
 * handler. When a latch is set and the current process is waiting for it, the
 * signal handler wakes up the poll() in WaitLatch by writing a byte to a pipe.
 * A signal by itself doesn't interrupt poll() on all platforms, and even on
 * platforms where it does, a signal that arrives just before the poll() call
 * does not prevent poll() from entering sleep. An incoming byte on a pipe
 * however reliably interrupts the sleep, and causes poll() to return
 * immediately even if the signal arrives before poll() begins.
 *
 * The epoll() implementation overcomes the race with a different technique: it
 * keeps SIGURG blocked and consumes from a signalfd() descriptor instead.  We
 * don't need to register a signal handler or create our own self-pipe.  We
 * assume that any system that has Linux epoll() also has Linux signalfd().
 *
 * The kqueue() implementation waits for SIGURG with EVFILT_SIGNAL.
 *
 * The Windows implementation uses Windows events that are inherited by all
 * postmaster child processes. There's no need for the self-pipe trick there.
 */

The choice is made at compile time:

c
// src/backend/storage/ipc/waiteventset.c
#if defined(WAIT_USE_EPOLL) || defined(WAIT_USE_POLL) || \
	defined(WAIT_USE_KQUEUE) || defined(WAIT_USE_WIN32)
/* don't overwrite manual choice */
#elif defined(HAVE_SYS_EPOLL_H)
#define WAIT_USE_EPOLL
#elif defined(HAVE_KQUEUE)
#define WAIT_USE_KQUEUE
#elif defined(HAVE_POLL)
#define WAIT_USE_POLL
#elif WIN32
#define WAIT_USE_WIN32
#else
#error "no wait set implementation available"
#endif

/*
 * By default, we use a self-pipe with poll() and a signalfd with epoll(), if
 * available.  For testing the choice can also be manually specified.
 */
#if defined(WAIT_USE_POLL) || defined(WAIT_USE_EPOLL)
#if defined(WAIT_USE_SELF_PIPE) || defined(WAIT_USE_SIGNALFD)
/* don't overwrite manual choice */
#elif defined(WAIT_USE_EPOLL) && defined(HAVE_SYS_SIGNALFD_H)
#define WAIT_USE_SIGNALFD
#else
#define WAIT_USE_SELF_PIPE
#endif
#endif

So on Linux: epoll + signalfd. Elsewhere with poll(): poll + self-pipe.

The signalfd variant

c
// src/backend/storage/ipc/waiteventset.c
#ifdef WAIT_USE_SIGNALFD
	sigset_t	signalfd_mask;

	...
	/* Block SIGURG, because we'll receive it through a signalfd. */
	sigaddset(&UnBlockSig, SIGURG);

	/* Set up the signalfd to receive SIGURG notifications. */
	sigemptyset(&signalfd_mask);
	sigaddset(&signalfd_mask, SIGURG);
	signal_fd = signalfd(-1, &signalfd_mask, SFD_NONBLOCK | SFD_CLOEXEC);
	if (signal_fd < 0)
		elog(FATAL, "signalfd() failed");
	ReserveExternalFD();
#endif

SIGURG is blocked in the process, so it never runs a handler; instead it becomes readable on the signalfd, and the fd is registered with epoll. When the latch is set, SetLatch() sends SIGURG to itself, the kernel marks the signalfd readable, and epoll_wait() returns.

c
// src/backend/storage/ipc/waiteventset.c
	/* Sleep */
	rc = epoll_wait(set->epoll_fd, set->epoll_ret_events,
					Min(nevents, set->nevents_space), cur_timeout);

	/* Check return code */
	if (rc < 0)
	{
		/* EINTR is okay, otherwise complain */
		if (errno != EINTR)
		{
			waiting = false;
			ereport(ERROR,
					(errcode_for_socket_access(),
					 errmsg("%s() failed: %m",
							"epoll_wait")));
		}
		return 0;
	}

Note that epoll_wait is a level-triggered wait on the signalfd: the fd stays readable until the pending signals are consumed, which is what the drain() call does.

The self-pipe variant

The classic trick. The pipe is created with both ends non-blocking and close-on-exec:

c
// src/backend/storage/ipc/waiteventset.c
	/*
	 * Set up the self-pipe that allows a signal handler to wake up the
	 * poll()/epoll_wait() in WaitLatch. Make the write-end non-blocking, so
	 * that SetLatch won't block if the event has already been set many times
	 * filling the kernel buffer. Make the read-end non-blocking too, so that
	 * we can easily clear the pipe by reading until EAGAIN or EWOULDBLOCK.
	 * Also, make both FDs close-on-exec, since we surely do not want any
	 * child processes messing with them.
	 */
	if (pipe(pipefd) < 0)
		elog(FATAL, "pipe() failed: %m");
	if (fcntl(pipefd[0], F_SETFL, O_NONBLOCK) == -1)
		elog(FATAL, "fcntl(F_SETFL) failed on read-end of self-pipe: %m");
	if (fcntl(pipefd[1], F_SETFL, O_NONBLOCK) == -1)
		elog(FATAL, "fcntl(F_SETFL) failed on write-end of self-pipe: %m");
	if (fcntl(pipefd[0], F_SETFD, FD_CLOEXEC) == -1)
		elog(FATAL, "fcntl(F_SETFD) failed on read-end of self-pipe: %m");
	if (fcntl(pipefd[1], F_SETFD, FD_CLOEXEC) == -1)
		elog(FATAL, "fcntl(F_SETFD) failed on write-end of self-pipe: %m");

	selfpipe_readfd = pipefd[0];
	selfpipe_writefd = pipefd[1];
	selfpipe_owner_pid = MyProcPid;

And the signal handler:

c
// src/backend/storage/ipc/waiteventset.c
/*
 * SetLatch uses SIGURG to wake up the process waiting on the latch.
 *
 * Wake up WaitLatch, if we're waiting.
 */
static void
latch_sigurg_handler(SIGNAL_ARGS)
{
	if (waiting)
		sendSelfPipeByte();
}

/* Send one byte to the self-pipe, to wake up WaitLatch */
static void
sendSelfPipeByte(void)
{
	ssize_t		rc;
	char		dummy = 0;

retry:
	rc = write(selfpipe_writefd, &dummy, 1);
	if (rc < 0)
	{
		/* If interrupted by signal, just retry */
		if (errno == EINTR)
			goto retry;

		/*
		 * If the pipe is full, we don't need to retry, the data that's there
		 * already is enough to wake up WaitLatch.
		 */
		if (errno == EAGAIN || errno == EWOULDBLOCK)
			return;

		/*
		 * Oops, the write() failed for some other reason. We might be in a
		 * signal handler, so it's not safe to elog(). We have no choice but
		 * silently ignore the error.
		 */
		return;
	}
}

Every detail here is load-bearing:

One byte, not a message. The byte’s value is irrelevant; only its presence matters.
The write end is non-blocking, and EAGAIN is treated as success. If the pipe’s kernel buffer is full, there is already a byte pending, and a pending byte is exactly as good as a new one: the reader will wake up anyway. Retrying would be pointless (the buffer is full!) and blocking would be fatal (we might be in a signal handler, where blocking is not allowed).
EINTR is retried, because a signal can interrupt the write.
Any other error is silently ignored, because we may be in a signal handler and elog() is not safe there.

The waiting flag is what keeps the handler cheap when nobody is waiting:

c
// src/backend/storage/ipc/waiteventset.c
#ifndef WIN32
/* Are we currently in WaitLatch? The signal handler would like to know. */
static volatile sig_atomic_t waiting = false;
#endif

volatile sig_atomic_t is the only kind of variable a signal handler may safely touch.

And the drain:

c
// src/backend/storage/ipc/waiteventset.c
/*
 * Read all available data from self-pipe or signalfd.
 * ...
 */
static void
drain(void)
{
	char		buf[1024];
	ssize_t		rc;
	int			fd;

#ifdef WAIT_USE_SELF_PIPE
	fd = selfpipe_readfd;
#else
	fd = signal_fd;
#endif

	for (;;)
	{
		rc = read(fd, buf, sizeof(buf));
		if (rc < 0)
		{
			if (errno == EAGAIN || errno == EWOULDBLOCK)
				break;			/* the descriptor is empty */
			else if (errno == EINTR)
				continue;		/* retry */
			else
			{
				waiting = false;
				...
				elog(ERROR, "read() on self-pipe failed: %m");
			}
		}
		else if (rc == 0)
		{
			...
		}
		else if (rc < sizeof(buf))
		{
			/* we successfully drained the pipe; no need to read() again */
			break;
		}
		/* else buffer wasn't big enough, so read again */
	}
}

The same function drains a signalfd or a self-pipe, which is why the code is shared between the two variants.

The whole picture

text

contended lock
     │
     ├─ spinlock ──── TAS_SPIN (free test + lock xchgb)
     │                 └── SPIN_DELAY(): rep; nop / isb
     │                      └── pg_usleep(cur_delay), cur_delay *= rand(1,2)
     │                           └── after NUM_DELAYS: PANIC "stuck spinlock"
     │
     ├─ LWLock ────── CAS on the state word
     │                 └── queue self on the wait list, try again
     │                      └── PGSemaphoreLock (sem_wait) — kernel
     │
     └─ Latch ─────── maybe_sleeping test
                       └── epoll_wait (signalfd) or poll (self-pipe) — kernel

What it all means in practice

Spinning burns CPU, and it is visible. If a backend is spinning, pg_stat_activity will show it with no wait event, and top will show it at 100% of a core. If it has fallen asleep in the spin delay, the wait event is SpinDelay. Both are symptoms of the same thing: somebody is holding a lock that a hot path needs.

The adaptive spin count is deliberately coarse, and that is fine. The estimate moves by +100 or −1 per acquisition locally, and by 1/16 of the difference per backend exit globally. It is not trying to be precise; it is trying to be stable. A backend that spins 100 times when 10 would do wastes 90 cache-coherence round trips; a backend that spins 10 times when 100 would do pays a pg_usleep() syscall. The asymmetry of the local adjustment (+100/−1) says that erring on the side of spinning is preferred, because spinning does not involve the kernel.

The global estimate only updates when a backend exits. It lives in ProcGlobal->spins_per_delay and is folded in at ProcKill() time. A long-lived backend with a short-lived one can therefore be running with a stale estimate for a long time. This is deliberate: the adjustment is cheap and the noise would be worse than the staleness.

PostgreSQL does not use futexes. Every LWLock wait that reaches the kernel costs a sem_wait syscall, and every wakeup costs a sem_post. On a machine with high LWLock contention, that syscall traffic shows up in strace counts and in perf samples of sem_wait. If you are chasing microseconds of lock overhead, this is where they go.

LWLocks have no deadlock detection. If two LWLocks could deadlock each other, PostgreSQL would not notice and would not report it. The heavyweight lock manager (deadlock.c) does the deadlock detection, and it only knows about heavyweight locks. The practical rule: never acquire an LWLock while holding another one in a way that could cycle. The elog(PANIC, "queueing for lock while waiting on another one") in LWLockQueueSelf() enforces the weaker property that you cannot wait for an LWLock while already waiting for one.

The wait event tells you which rung you are on. pg_stat_activity distinguishes LWLock (waiting for an LWLock), SpinDelay (sleeping inside a spinlock wait), and the latch/socket wait events. If a query is “stuck” and you see LWLock: BufferMapping for a long time, you have a buffer contention problem; if you see SpinDelay, you have a spinlock problem, which usually means something else is wrong.

SetLatch() is cheap on purpose. The maybe_sleeping test means that setting a latch nobody is waiting on costs one memory barrier and a predictable branch. This matters because latches are set on hot paths — every CHECK_FOR_INTERRUPTS(), every lock acquisition that might need to be interrupted — and a syscall there would be unacceptable.

The self-pipe trick exists because signals are unreliable as wakeups. A signal that arrives just before poll() is entered does not prevent poll() from sleeping. A byte in a pipe does. This is why the pipe write end is non-blocking and why EAGAIN is a success: the wakeup mechanism has to be safe to call from a signal handler, where blocking is forbidden and where throwing an error would be fatal.

The 64-bit atomic fallback is a spinlock. On a 32-bit platform without 64-bit CAS, pg_atomic_uint64 carries an embedded spinlock and every 64-bit atomic operation is a spinlock acquisition. That is “not unusably slow” per the header comment, but it does mean that 64-bit atomics are a scalability cliff on such platforms.

Two orders of magnitude separate “spin” from “sleep”. A spinlock acquisition that succeeds costs tens of nanoseconds; one that has to sleep costs at least a microsecond, usually much more. Most performance problems attributed to “lock contention” are really problems with the duration the lock is held, not with the waiting mechanism. Shortening the critical section is worth far more than tuning the spin count.

The ladder, in one table

Layer 1: the spinlock

The hardware primitives

The free test

The pause instruction

The memory ordering contract

The generic spin loop

perform_spin_delay()

Why the randomness matters

The adaptive spin count

Stuck spinlock detection

Layer 2: the atomics

The x86 implementations

The API contract

The CAS loops

Memory barriers

The 64-bit fallback

Layer 3: the LWLock

The state word

The attempt

The wait list lock is a bit in the same word

Queue-self-then-retry

Sleeping: a POSIX semaphore, not a futex

Releasing and waking

Deadlock detection: not here

Layer 4: latches

SetLatch

ResetLatch

WaitLatch and the long-lived WaitEventSet

epoll and signalfd versus the classic self-pipe

The signalfd variant

The self-pipe variant

The whole picture

What it all means in practice

Further reading