How PostgreSQL Locks Buffers Without LWLocks

Every time PostgreSQL reads a page it does two things: it pins the shared buffer that holds the page, and it takes a content lock on that buffer so that nobody rewrites the bytes underneath it while it is being read.

The second of those used to be an LWLock — a general-purpose, one-word-at-a-time lightweight lock living in a global array in shared memory. PostgreSQL 20 changed that. Buffer content locks are now implemented entirely inside a single 64-bit atomic word that already existed in the buffer descriptor, with its own three-level lock hierarchy, its own wait queue, and its own wakeup logic.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

What a content lock protects

Let’s be precise about what is being locked, because PostgreSQL has two different “buffer locks” and they are easy to confuse.

The buffer header lock protects the descriptor — the metadata in BufferDesc: the tag, the validity flags, the I/O state. It is a very short critical section, taken for a handful of instructions at a time.

The buffer content lock protects the page data — the 8 kB of bytes in the buffer. It is held for as long as the caller is looking at the page. This is the one that matters for correctness: two backends cannot be reading the same page while one of them is rewriting it.

The header comment in src/include/storage/buf_internals.h is explicit about the division:

The content of buffers is protected via the buffer content lock, implemented as part of the buffer state. Note that the buffer header lock is not used to control access to the data in the buffer! We used to use an LWLock to implement the content lock, but having a dedicated implementation of content locks allows us to implement some otherwise hard things (e.g. race-freely checking if AIO is in progress before locking a buffer exclusively) and enables otherwise impossible optimizations (e.g. unlocking and unpinning a buffer in one atomic operation).

That comment is the entire design rationale in one paragraph, and it is worth reading twice. The dedicated implementation exists to make two specific things possible: racing safely against asynchronous I/O, and fusing lock release with unpinning.

The old world: an LWLock per buffer

Before the change, BufferDesc had a lock in it:

c
	/*
	 * Backend of pin-count waiter. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	int			wait_backend_pgprocno;

	PgAioWaitRef io_wref;		/* set iff AIO is in progress */
	LWLock		content_lock;	/* to lock access to buffer contents */
} BufferDesc;

and the accessor was a one-liner:

c
static inline LWLock *
BufferDescriptorGetContentLock(const BufferDesc *bdesc)
{
	return (LWLock *) (&bdesc->content_lock);
}

An LWLock is a 16-byte structure with a uint32 state, a pg_atomic_uint32 nwaiters, a proclist_head waiters, and a bool is_shared. Taking one is a CAS on the state word; if that fails, you get pushed onto the wait queue in the same structure. Releasing it is another CAS plus, possibly, a walk of the wait queue.

The problem is not the algorithm — the LWLock algorithm is good. The problem is that there are now two independent atomic words per buffer: the LWLock’s state, and the BufferDesc.state word that holds the refcount and the flags. Anything that needs to reason about both at once has to do two separate atomic operations, with a window in between.

Concretely, the old code path for a shared content lock was:

c
	LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);

and for an exclusive one:

c
	LWLockAcquire(BufferDescriptorGetContentLock(desc), LW_EXCLUSIVE);

That is a full generic lock acquisition: read the lock’s state, decide whether it conflicts, CAS, and — if it conflicts — walk into lwlock.c’s queueing machinery.

The new world: one 64-bit word

The change is commit fcb9c977aa5, “bufmgr: Implement buffer content locks independently of lwlocks”, by Andres Freund, January 2026. The commit message lays out the four reasons:

Hint bits are currently set while holding only a share lock. This leads to having to copy pages while they are being written out if checksums are enabled, which is not cheap. We would like to add AIO writes, however once many buffers can be written out at the same time, it gets a lot more expensive to copy them, particularly because that copy needs to reside in shared buffers (for worker mode to have access to the buffer).
In addition, modifying buffers while they are being written out can cause issues with unbuffered/direct-IO, as some filesystems (like btrfs) do not like that, due to filesystem internal checksums getting corrupted.
The solution to this is to require a new share-exclusive lock-level to set hint bits and to write out buffers, making those operations mutually exclusive. We could introduce such a lock-level into the generic lwlock implementation, however it does not look like there would be other users, and it does add some overhead into important code paths.
For AIO writes we need to be able to race-freely check whether a buffer is undergoing IO and whether an exclusive lock on the page can be acquired. That is rather hard to do efficiently when the buffer state and the lock state are separate atomic variables. This is a major hindrance to allowing writes to be done asynchronously.
Buffer locks are by far the most frequently taken locks. Optimizing them specifically for their use case is worth the effort. E.g. by merging content locks into buffer locks we will be able to release a buffer lock and pin in one atomic operation.
There are more complicated optimizations, like long-lived “super pinned & locked” pages, that cannot realistically be implemented with the generic lwlock implementation.

Point 3 is the headline: buffer content locks are, by a wide margin, the most frequently taken lock in the system. Every ReadBuffer() pins and locks; every index scan does it once per page; every page inspection does it again.

The layout

The state word was already 64 bits wide before this change, but only half of it was used. The old header comment read:

c
/*
 * Buffer state is a single 64-bit variable where following data is combined.
 *
 * State of the buffer itself (in order):
 * - 18 bits refcount
 * - 4 bits usage count
 * - 10 bits of flags
 *
 * Combining these values allows to perform some operations without locking
 * the buffer header, by modifying them together with a CAS loop.
 *
 * NB: A future commit will use a significant portion of the remaining bits to
 * implement buffer locking as part of the state variable.
 *
 * The definition of buffer state components is below.
 */
#define BUF_REFCOUNT_BITS 18
#define BUF_USAGECOUNT_BITS 4
#define BUF_FLAG_BITS 10

StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32,
				 "parts of buffer state space need to equal 32");

The new one (src/include/storage/buf_internals.h):

c
/*
 * Buffer state is a single 64-bit variable where following data is combined.
 *
 * State of the buffer itself (in order):
 * - 18 bits refcount
 * - 4 bits usage count
 * - 12 bits of flags
 * - 18 bits share-lock count
 * - 1 bit share-exclusive locked
 * - 1 bit exclusive locked
 *
 * Combining these values allows to perform some operations without locking
 * the buffer header, by modifying them together with a CAS loop.
 *
 * The definition of buffer state components is below.
 */
#define BUF_REFCOUNT_BITS 18
#define BUF_USAGECOUNT_BITS 4
#define BUF_FLAG_BITS 12
#define BUF_LOCK_BITS (18+2)

StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS + BUF_LOCK_BITS <= 64,
				 "parts of buffer state space need to be <= 64");

So the word went from 32 bits used to 54 bits used, and the invariant changed from “exactly 32” to “at most 64”. The two extra flag bits are the two new lock-related flags; the 20 new lock bits are the lock state.

Here is the whole layout, on a 64-bit machine, with MAX_BACKENDS_BITS == 18:

Bits	Width	Field	Meaning
0–17	18	refcount	number of pins on this buffer
18–21	4	usage count	clock-sweep second-chance counter
22–33	12	flags	`BM_*` bits, see below
34–51	18	share-lock count	number of backends holding a share lock
52	1	share-exclusive	one backend holds a share-exclusive lock
53	1	exclusive	one backend holds an exclusive lock
54–63	10	unused

The shift constants:

c
/* refcount related definitions */
#define BUF_REFCOUNT_ONE 1
#define BUF_REFCOUNT_MASK \
	((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1)

/* usage count related definitions */
#define BUF_USAGECOUNT_SHIFT \
	BUF_REFCOUNT_BITS
#define BUF_USAGECOUNT_MASK \
	(((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT))
#define BUF_USAGECOUNT_ONE \
	(UINT64CONST(1) << BUF_REFCOUNT_BITS)

/* flags related definitions */
#define BUF_FLAG_SHIFT \
	(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)
#define BUF_FLAG_MASK \
	(((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT)

and the lock bits:

c
/* lock state related definitions */
#define BM_LOCK_SHIFT \
	(BUF_FLAG_SHIFT + BUF_FLAG_BITS)
#define BM_LOCK_VAL_SHARED \
	(UINT64CONST(1) << (BM_LOCK_SHIFT))
#define BM_LOCK_VAL_SHARE_EXCLUSIVE \
	(UINT64CONST(1) << (BM_LOCK_SHIFT + MAX_BACKENDS_BITS))
#define BM_LOCK_VAL_EXCLUSIVE \
	(UINT64CONST(1) << (BM_LOCK_SHIFT + MAX_BACKENDS_BITS + 1))
#define BM_LOCK_MASK \
	((((uint64) MAX_BACKENDS) << BM_LOCK_SHIFT) | BM_LOCK_VAL_SHARE_EXCLUSIVE | BM_LOCK_VAL_EXCLUSIVE)

MAX_BACKENDS_BITS comes from src/include/storage/procnumber.h:

c
#define MAX_BACKENDS_BITS		18
#define MAX_BACKENDS			((1U << MAX_BACKENDS_BITS)-1)

Eighteen bits, which is exactly the number of lock bits available for the share-lock count. That is not a coincidence: it means the share-lock count can never exceed the number of possible backends, and the BM_LOCK_MASK expression (((uint64) MAX_BACKENDS) << BM_LOCK_SHIFT) covers exactly bits 34–51 — the whole share-lock field.

Two static assertions tie this together:

c
StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS,
				 "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS");
StaticAssertDecl(MAX_BACKENDS_BITS <= (BUF_LOCK_BITS - 2),
				 "MAX_BACKENDS_BITS needs to be <= BUF_LOCK_BITS - 2");

The subtle part is that the “share lock” field is not a single bit — it is a count, stored in the low bits of the lock field. BM_LOCK_VAL_SHARED is the value 1 << 34, and acquiring a share lock adds it to the state word:

c
		else
		{
			lock_free = (old_state & BM_LOCK_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARED;
		}

So desired_state += BM_LOCK_VAL_SHARED is really desired_state += 1 in the count field. The BM_LOCK_MASK then detects “is anybody holding any share lock” by testing all 18 count bits at once.

This is why the exclusive acquisition test is a mask against the whole lock field rather than a single bit:

c
		if (mode == BUFFER_LOCK_EXCLUSIVE)
		{
			lock_free = (old_state & BM_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_EXCLUSIVE;
		}

BM_LOCK_MASK covers the share-lock count and both exclusive bits, so “the lock is free” means “no share lockers, no share-exclusive holder, no exclusive holder”.

The flag bits

The flags live in the middle of the word, and two of them are new:

c
/* buffer header is locked */
#define BM_LOCKED					BUF_DEFINE_FLAG( 0)
/* data needs writing */
#define BM_DIRTY					BUF_DEFINE_FLAG( 1)
/* data is valid */
#define BM_VALID					BUF_DEFINE_FLAG( 2)
/* tag is assigned */
#define BM_TAG_VALID				BUF_DEFINE_FLAG( 3)
/* read or write in progress */
#define BM_IO_IN_PROGRESS			BUF_DEFINE_FLAG( 4)
/* previous I/O failed */
#define BM_IO_ERROR					BUF_DEFINE_FLAG( 5)
/* flag bit 6 is not used anymore */
/* have waiter for sole pin */
#define BM_PIN_COUNT_WAITER			BUF_DEFINE_FLAG( 7)
/* must write for checkpoint */
#define BM_CHECKPOINT_NEEDED		BUF_DEFINE_FLAG( 8)
/* permanent buffer (not unlogged, or init fork) */
#define BM_PERMANENT				BUF_DEFINE_FLAG( 9)
/* content lock has waiters */
#define BM_LOCK_HAS_WAITERS			BUF_DEFINE_FLAG(10)
/* waiter for content lock has been signalled but not yet run */
#define BM_LOCK_WAKE_IN_PROGRESS	BUF_DEFINE_FLAG(11)

BM_LOCK_HAS_WAITERS and BM_LOCK_WAKE_IN_PROGRESS are the content-lock equivalents of LW_FLAG_HAS_WAITERS and LW_FLAG_WAKE_IN_PROGRESS. They are in the flag field rather than the lock field because they are advisory hints, not part of the lock’s conflict state — they must not affect whether a lock can be acquired.

The descriptor

The BufferDesc itself changed shape. The embedded LWLock content_lock is gone; in its place is a wait list:

c
typedef struct BufferDesc
{
	/*
	 * ID of page contained in buffer. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	BufferTag	tag;

	/*
	 * Buffer's index number (from 0). The field never changes after
	 * initialization, so does not need locking.
	 */
	int			buf_id;

	/*
	 * State of the buffer, containing flags, refcount and usagecount. See
	 * BUF_* and BM_* defines at the top of this file.
	 */
	pg_atomic_uint64 state;

	/*
	 * Backend of pin-count waiter. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	int			wait_backend_pgprocno;

	PgAioWaitRef io_wref;		/* set iff AIO is in progress */

	/*
	 * List of PGPROCs waiting for the buffer content lock. Protected by the
	 * buffer header spinlock.
	 */
	proclist_head lock_waiters;
} BufferDesc;

Note the proclist_head lock_waiters — a per-buffer intrusive wait list, analogous to the one inside an LWLock, but only 8 bytes instead of a whole LWLock’s worth of state.

Acquiring a content lock

The heart of it is BufferLockAttempt() in src/backend/storage/buffer/bufmgr.c. This is the analogue of LWLockAttemptLock():

c
static inline bool
BufferLockAttempt(BufferDesc *buf_hdr, BufferLockMode mode)
{
	uint64		old_state;

	/*
	 * Read once outside the loop, later iterations will get the newer value
	 * via compare & exchange.
	 */
	old_state = pg_atomic_read_u64(&buf_hdr->state);

	/* loop until we've determined whether we could acquire the lock or not */
	while (true)
	{
		uint64		desired_state;
		bool		lock_free;

		desired_state = old_state;

		if (mode == BUFFER_LOCK_EXCLUSIVE)
		{
			lock_free = (old_state & BM_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_EXCLUSIVE;
		}
		else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			lock_free = (old_state & (BM_LOCK_VAL_EXCLUSIVE | BM_LOCK_VAL_SHARE_EXCLUSIVE)) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARE_EXCLUSIVE;
		}
		else
		{
			lock_free = (old_state & BM_LOCK_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARED;
		}

		/*
		 * Attempt to swap in the state we are expecting. If we didn't see
		 * lock to be free, that's just the old value. If we saw it as free,
		 * we'll attempt to mark it acquired. The reason that we always swap
		 * in the value is that this doubles as a memory barrier. We could try
		 * to be smarter and only swap in values if we saw the lock as free,
		 * but benchmark haven't shown it as beneficial so far.
		 *
		 * Retry if the value changed since we last looked at it.
		 */
		if (likely(pg_atomic_compare_exchange_u64(&buf_hdr->state,
												  &old_state, desired_state)))
		{
			if (lock_free)
			{
				/* Great! Got the lock. */
				return false;
			}
			else
				return true;	/* somebody else has the lock */
		}
	}

	pg_unreachable();
}

Read the three conflict rules off that:

Mode	Acquired if	Adds
`BUFFER_LOCK_SHARE`	no exclusive holder	`+1` to the share count
`BUFFER_LOCK_SHARE_EXCLUSIVE`	no exclusive and no share-exclusive holder	sets bit 52
`BUFFER_LOCK_EXCLUSIVE`	nothing held at all	sets bit 53

That is exactly the semantics the commit message describes: “An additional lock-level share-exclusive has been added. This lock-level conflicts with exclusive locks and itself, but not share locks.”

Two things are worth noting about the implementation.

First, the CAS always happens, even when the lock is known to be held. The comment explains why: the CAS doubles as a memory barrier, and the benchmark showed that conditionally skipping it did not help.

Second, the conflict test is a read of a shared word, not a read of a private lock structure. Because the lock state lives in the same word as the refcount and the flags, a concurrent pin or unpin is visible to the lock decision for free.

The queue-self-then-retry algorithm

If the first attempt fails, we cannot simply go to sleep: the lock might have been released in the meantime, and we would sleep on a lock nobody is going to signal. The classic solution — and the one lwlock.c uses — is to enqueue yourself and then try again. BufferLockAcquire() does exactly that:

c
static inline void
BufferLockAcquire(Buffer buffer, BufferDesc *buf_hdr, BufferLockMode mode)
{
	PrivateRefCountEntry *entry;
	int			extraWaits = 0;

	/*
	 * Get reference to the refcount entry before we hold the lock, it seems
	 * better to do before holding the lock.
	 */
	entry = GetPrivateRefCountEntry(buffer, true);

	/*
	 * We better not already hold a lock on the buffer.
	 */
	Assert(entry->data.lockmode == BUFFER_LOCK_UNLOCK);

	/*
	 * Lock out cancel/die interrupts until we exit the code section protected
	 * by the content lock.  This ensures that interrupts will not interfere
	 * with manipulations of data structures in shared memory.
	 */
	HOLD_INTERRUPTS();

	for (;;)
	{
		uint32		wait_event = 0; /* initialized to avoid compiler warning */
		bool		mustwait;

		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = BufferLockAttempt(buf_hdr, mode);

		if (likely(!mustwait))
		{
			break;
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		BufferLockQueueSelf(buf_hdr, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = BufferLockAttempt(buf_hdr, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			BufferLockDequeueSelf(buf_hdr);
			break;
		}

		switch (mode)
		{
			case BUFFER_LOCK_EXCLUSIVE:
				wait_event = WAIT_EVENT_BUFFER_EXCLUSIVE;
				break;
			case BUFFER_LOCK_SHARE_EXCLUSIVE:
				wait_event = WAIT_EVENT_BUFFER_SHARE_EXCLUSIVE;
				break;
			case BUFFER_LOCK_SHARE:
				wait_event = WAIT_EVENT_BUFFER_SHARED;
				break;
			case BUFFER_LOCK_UNLOCK:
				pg_unreachable();

		}
		pgstat_report_wait_start(wait_event);

		/*
		 * Wait until awakened.
		 *
		 * It is possible that we get awakened for a reason other than being
		 * signaled by BufferLockWakeup().  If so, loop back and wait again.
		 * Once we've gotten the lock, re-increment the sema by the number of
		 * additional signals received.
		 */
		for (;;)
		{
			PGSemaphoreLock(MyProc->sem);
			if (MyProc->lwWaiting == LW_WS_NOT_WAITING)
				break;
			extraWaits++;
		}

		pgstat_report_wait_end();

		/* Retrying, allow BufferLockReleaseSub to release waiters again. */
		pg_atomic_fetch_and_u64(&buf_hdr->state, ~BM_LOCK_WAKE_IN_PROGRESS);
	}

	/* Remember that we now hold this lock */
	entry->data.lockmode = mode;

	/*
	 * Fix the process wait semaphore's count for any absorbed wakeups.
	 */
	while (unlikely(extraWaits-- > 0))
		PGSemaphoreUnlock(MyProc->sem);
}

Compare this to LWLockAcquire() in src/backend/storage/lmgr/lwlock.c — the structure is identical, down to the comments:

c
		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = LWLockAttemptLock(lock, mode);

		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "immediately acquired lock");
			break;				/* got the lock */
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		LWLockQueueSelf(lock, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = LWLockAttemptLock(lock, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "acquired, undoing queue");

			LWLockDequeueSelf(lock);
			break;
		}

The commit message is frank about this: “This commit unfortunately introduces some code that is very similar to the code in lwlock.c, however the code is not equivalent enough to easily merge it.”

One wart the commit message calls out explicitly: the new code still uses PGPROC->lwWaiting and PGPROC->lwWaitMode, rather than inventing a parallel set of fields:

c
	/*
	 * These are currently used both for lwlocks and buffer content locks,
	 * which is acceptable, although not pretty, because a backend can't wait
	 * for both types of locks at the same time.
	 */
	MyProc->lwWaiting = LW_WS_WAITING;
	MyProc->lwWaitMode = mode;

	proclist_push_tail(&buf_hdr->lock_waiters, MyProcNumber, lwWaitLink);

A backend can only be blocked on one lock at a time, so reusing the fields is safe — but the fields are now lying about their name.

The public API

The lock modes are an enum in src/include/storage/bufmgr.h:

c
/*
 * Buffer content lock modes (mode argument for LockBuffer())
 */
typedef enum BufferLockMode
{
	BUFFER_LOCK_UNLOCK,

	/*
	 * A share lock conflicts with exclusive locks.
	 */
	BUFFER_LOCK_SHARE,

	/*
	 * A share-exclusive lock conflicts with itself and exclusive locks.
	 */
	BUFFER_LOCK_SHARE_EXCLUSIVE,

	/*
	 * An exclusive lock conflicts with every other lock type.
	 */
	BUFFER_LOCK_EXCLUSIVE,
} BufferLockMode;

and there is a wrapper that keeps the mode argument a compile-time constant at the call site, because a runtime switch generated worse code:

c
/*
 * Handling BUFFER_LOCK_UNLOCK in bufmgr.c leads to sufficiently worse branch
 * prediction to impact performance. Therefore handle that switch here, where
 * most of the time `mode` will be a constant and thus can be optimized out by
 * the compiler.
 */
static inline void
LockBuffer(Buffer buffer, BufferLockMode mode)
{
	if (mode == BUFFER_LOCK_UNLOCK)
		UnlockBuffer(buffer);
	else
		LockBufferInternal(buffer, mode);
}

LockBufferInternal() does the same trick one level down:

c
	/*
	 * Test the most frequent lock modes first. While a switch (mode) would be
	 * nice, at least gcc generates considerably worse code for it.
	 *
	 * Call BufferLockAcquire() with a constant argument for mode, to generate
	 * more efficient code for the different lock modes.
	 */
	if (mode == BUFFER_LOCK_SHARE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_SHARE);
	else if (mode == BUFFER_LOCK_EXCLUSIVE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_EXCLUSIVE);
	else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_SHARE_EXCLUSIVE);
	else
		elog(ERROR, "unrecognized buffer lock mode: %d", mode);

Because BufferLockAcquire() is static inline and is called with a constant mode three different times, the compiler generates three specialised copies, each with the conflict test folded to a constant. A share lock’s conflict test becomes “is the exclusive bit set?” and nothing else.

Lock ownership tracking

Where lwlock.c keeps a global array of held locks per backend, the buffer content lock reuses the buffer manager’s existing private refcount hash:

c
/*
 * This is separated out from PrivateRefCountEntry to allow for copying all
 * the data members via struct assignment.
 */
typedef struct PrivateRefCountData
{
	/*
	 * How many times has the buffer been pinned by this backend.
	 */
	int32		refcount;

	/*
	 * Is the buffer locked by this backend? BUFFER_LOCK_UNLOCK indicates that
	 * the buffer is not locked.
	 */
	BufferLockMode lockmode;
} PrivateRefCountData;

This is elegant: one hash lookup answers both “how many pins does this backend hold on this buffer?” and “in what mode is it locked?”. And it means error recovery needs no new machinery — the resource owner’s buffer cleanup already releases the content locks, so there is no LWLockReleaseAll() equivalent to write.

Releasing a content lock

The release path is split into three pieces, so that the “subtract the lock from the state word” step can be fused with other atomic operations:

c
/*
 * Release a previously acquired buffer content lock.
 */
static void
BufferLockUnlock(Buffer buffer, BufferDesc *buf_hdr)
{
	BufferLockMode mode;
	uint64		oldstate;
	uint64		sub;

	mode = BufferLockDisownInternal(buffer, buf_hdr);

	/*
	 * Release my hold on lock, after that it can immediately be acquired by
	 * others, even if we still have to wakeup other waiters.
	 */
	sub = BufferLockReleaseSub(mode);

	oldstate = pg_atomic_sub_fetch_u64(&buf_hdr->state, sub);

	BufferLockProcessRelease(buf_hdr, mode, oldstate);

	/*
	 * Now okay to allow cancel/die interrupts.
	 */
	RESUME_INTERRUPTS();
}

BufferLockReleaseSub() computes what to subtract:

c
static inline uint64
BufferLockReleaseSub(BufferLockMode mode)
{
	/*
	 * Turns out that a switch() leads to gcc to generate sufficiently worse code
	 * for this to show up in profiles...
	 */
	if (mode == BUFFER_LOCK_EXCLUSIVE)
		return BM_LOCK_VAL_EXCLUSIVE;
	else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		return BM_LOCK_VAL_SHARE_EXCLUSIVE;
	else
	{
		Assert(mode == BUFFER_LOCK_SHARE);
		return BM_LOCK_VAL_SHARED;
	}

	return 0;					/* keep compiler quiet */
}

Note that releasing a share lock subtracts BM_LOCK_VAL_SHARED — the count goes down, which is why it is a counter and not a bit.

The wakeup decision:

c
static void
BufferLockProcessRelease(BufferDesc *buf_hdr, BufferLockMode mode, uint64 lockstate)
{
	bool		check_waiters = false;
	bool		wake_exclusive = false;

	/* nobody else can have that kind of lock */
	Assert(!(lockstate & BM_LOCK_VAL_EXCLUSIVE));

	/*
	 * If we're still waiting for backends to get scheduled, don't wake them
	 * up again. Otherwise check if we need to look through the waitqueue to
	 * wake other backends.
	 */
	if ((lockstate & BM_LOCK_HAS_WAITERS) &&
		!(lockstate & BM_LOCK_WAKE_IN_PROGRESS))
	{
		if ((lockstate & BM_LOCK_MASK) == 0)
		{
			/*
			 * We released a lock and the lock was, in that moment, free. We
			 * therefore can wake waiters for any kind of lock.
			 */
			check_waiters = true;
			wake_exclusive = true;
		}
		else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			/*
			 * We released the lock, but another backend still holds a lock. We
			 * can't have released an exclusive lock, as there couldn't
			 * have been other lock holders. If we released a share lock, no
			 * waiters need to be woken up, as there must be other share
			 * lockers. However, if we held a share-exclusive lock, another
			 * backend now could acquire a share-exclusive lock.
			 */
			check_waiters = true;
			wake_exclusive = false;
		}
	}

	/*
	 * As waking up waiters requires the spinlock to be acquired, only do so
	 * if necessary.
	 */
	if (check_waiters)
		BufferLockWakeup(buf_hdr, wake_exclusive);
}

The three cases:

The lock is now completely free (lockstate & BM_LOCK_MASK == 0). Wake everybody — share, share-exclusive and exclusive waiters.
We released a share lock and others still hold locks. Nothing to do: there must still be other share lockers, so a share waiter can proceed.
We released a share-exclusive lock and others still hold locks. A share-exclusive waiter can now proceed (share-exclusive does not conflict with share), but an exclusive waiter cannot.

BufferLockWakeup() then walks the per-buffer wait list. This is where the three-level conflict matrix shows up:

c
	proclist_foreach_modify(iter, &buf_hdr->lock_waiters, lwWaitLink)
	{
		PGPROC	   *waiter = GetPGProcByNumber(iter.cur);

		/*
		 * Already woke up a conflicting lock, so skip over this wait list
		 * entry.
		 */
		if (!wake_exclusive && waiter->lwWaitMode == BUFFER_LOCK_EXCLUSIVE)
			continue;
		if (!wake_share_exclusive && waiter->lwWaitMode == BUFFER_LOCK_SHARE_EXCLUSIVE)
			continue;

		proclist_delete(&buf_hdr->lock_waiters, iter.cur, lwWaitLink);
		proclist_push_tail(&wakeup, iter.cur, lwWaitLink);

		/*
		 * Prevent additional wakeups until retryer gets to run. Backends that
		 * are just waiting for the lock to become free don't retry
		 * automatically.
		 */
		new_wake_in_progress = true;

		/*
		 * Signal that the process isn't on the wait list anymore. This allows
		 * BufferLockDequeueSelf() to remove itself from the waitlist with a
		 * proclist_delete(), rather than having to check if it has been
		 * removed from the list.
		 */
		Assert(waiter->lwWaiting == LW_WS_WAITING);
		waiter->lwWaiting = LW_WS_PENDING_WAKEUP;

		/*
		 * Don't wakeup further waiters after waking a conflicting waiter.
		 */
		if (waiter->lwWaitMode == BUFFER_LOCK_SHARE)
		{
			/*
			 * Share locks conflict with exclusive locks.
			 */
			wake_exclusive = false;
		}
		else if (waiter->lwWaitMode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			/*
			 * Share-exclusive locks conflict with share-exclusive and
			 * exclusive locks.
			 */
			wake_exclusive = false;
			wake_share_exclusive = false;
		}
		else if (waiter->lwWaitMode == BUFFER_LOCK_EXCLUSIVE)
		{
			/*
			 * Exclusive locks conflict with all other locks, there's no point
			 * in waking up anybody else.
			 */
			break;
		}
	}

The classic FIFO-ish behaviour: wake share waiters, and stop at the first exclusive waiter, because everybody behind it conflicts with it anyway.

The wait list itself is protected by the buffer header spinlock, which is what BufferLockQueueSelf() does:

c
static void
BufferLockQueueSelf(BufferDesc *buf_hdr, BufferLockMode mode)
{
	/*
	 * If we don't have a PGPROC structure, there's no way to wait. This
	 * should never occur, since MyProc should only be null during shared
	 * memory initialization.
	 */
	if (MyProc == NULL)
		elog(PANIC, "cannot wait without a PGPROC structure");

	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

	LockBufHdr(buf_hdr);

	/* setting the flag is protected by the spinlock */
	pg_atomic_fetch_or_u64(&buf_hdr->state, BM_LOCK_HAS_WAITERS);
	...
	proclist_push_tail(&buf_hdr->lock_waiters, MyProcNumber, lwWaitLink);

	/* Can release the mutex now */
	UnlockBufHdr(buf_hdr);
}

The header lock is not an LWLock either

A pleasant side effect of the change is that the buffer header lock is now also not a real spinlock with a slock_t. It is a bit in the state word, and LockBufHdr() is a single fetch_or:

c
/*
 * Lock buffer header - set BM_LOCKED in buffer state.
 */
uint64
LockBufHdr(BufferDesc *desc)
{
	uint64		old_buf_state;

	Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc)));

	while (true)
	{
		/*
		 * Always try once to acquire the lock directly, without setting up
		 * the spin-delay infrastructure. The work necessary for that shows up
		 * in profiles and is rarely necessary.
		 */
		old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED);
		if (likely(!(old_buf_state & BM_LOCKED)))
			break;				/* got lock */

		/* and then spin without atomic operations until lock is released */
		{
			SpinDelayStatus delayStatus;

			init_local_spin_delay(&delayStatus);

			while (old_buf_state & BM_LOCKED)
			{
				perform_spin_delay(&delayStatus);
				old_buf_state = pg_atomic_read_u64(&desc->state);
			}
			finish_spin_delay(&delayStatus);
		}

		/*
		 * Retry. The lock might obviously already be re-acquired by the time
		 * we're attempting to get it again.
		 */
	}

	return old_buf_state | BM_LOCKED;
}

That comment — “Always try once to acquire the lock directly, without setting up the spin-delay infrastructure. The work necessary for that shows up in profiles and is rarely necessary.” — is a micro-optimisation with a real history behind it. Setting up SpinDelayStatus costs enough that it showed up in profiles when the lock was uncontended (which it almost always is).

The uncontended path is now: one lock or on a cache line that is already hot, one branch. The contended path spins on a plain load, no atomic instructions at all, until the bit clears.

Releasing is symmetric and equally cheap:

c
/*
 * Unlock the buffer header.
 *
 * This can only be used if the caller did not modify BufferDesc.state. To
 * set/unset flag bits or change the refcount use UnlockBufHdrExt().
 */
static inline void
UnlockBufHdr(BufferDesc *desc)
{
	Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED);

	pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED);
}

And the more general version, which sets flags and adjusts the refcount in the same CAS — this is the “release a buffer lock and pin in one atomic operation” that the commit message promised, in its simpler form:

c
/*
 * Unlock the buffer header, while atomically unsetting the ones in
 * unset_bits, adding the flags in set_bits, and changing the refcount by
 * refcount_change. If a flag is both cleared and added, it will end up being
 * set.
 *
 * Note that this approach would not trivially work for usagecount, since we
 * need to cap the usagecount at BM_MAX_USAGE_COUNT.
 */
static inline uint64
UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state,
				uint64 set_bits, uint64 unset_bits,
				int refcount_change)
{
	for (;;)
	{
		uint64		buf_state = old_buf_state;

		Assert(buf_state & BM_LOCKED);

		/*
		 * Set bits after clearing bits, so that a cleared and re-added flag
		 * survives.
		 */
		buf_state &= ~unset_bits;
		buf_state |= set_bits;
		buf_state &= ~BM_LOCKED;

		if (refcount_change != 0)
			buf_state += BUF_REFCOUNT_ONE * refcount_change;

		if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state,
										   buf_state))
		{
			return old_buf_state;
		}
	}
}

Note the comment about the usage count: it cannot be handled this way, because the usage count is capped at BM_MAX_USAGE_COUNT and a blind addition would overflow the cap.

The counterpart — waiting for the header lock to clear — is a pg_noinline spin loop:

c
/*
 * Wait until the BM_LOCKED flag isn't set anymore and return the buffer's
 * state at that point.
 *
 * Obviously the buffer could be locked by the time the value is returned, so
 * this is primarily useful in CAS style loops.
 */
pg_noinline uint64
WaitBufHdrUnlocked(BufferDesc *buf)
{
	SpinDelayStatus delayStatus;
	uint64		buf_state;

	init_local_spin_delay(&delayStatus);

	buf_state = pg_atomic_read_u64(&buf->state);

	while (buf_state & BM_LOCKED)
	{
		perform_spin_delay(&delayStatus);
		buf_state = pg_atomic_read_u64(&buf->state);
	}

	finish_spin_delay(&delayStatus);

	return buf_state;
}

pg_noinline is deliberate: this is the slow path, and inlining it into every caller would bloat the instruction cache for no benefit.

The reason the third lock level exists at all is a correctness issue with hint bits.

PostgreSQL caches visibility facts (Xmin committed, Xmax invalid, …) in the tuple’s t_infomask. Those are called hint bits. Setting them modifies the page. If the page is being written out at the same time, the page checksum computed over the bytes that go to disk does not match the bytes that were in memory when the checksum was computed — so the buffer manager had to copy the page before writing it, which is expensive.

With a third lock level, hint-bit setting and write-out can be made mutually exclusive. Both need share-exclusive, and share-exclusive conflicts with itself.

Writing the buffer takes share-exclusive:

c
/*
 * Convenience wrapper around FlushBuffer() that locks/unlocks the buffer
 * before/after calling FlushBuffer().
 */
static void
FlushUnlockedBuffer(BufferDesc *buf, SMgrRelation reln,
					IOObject io_object, IOContext io_context)
{
	Buffer		buffer = BufferDescriptorGetBuffer(buf);

	BufferLockAcquire(buffer, buf, BUFFER_LOCK_SHARE_EXCLUSIVE);
	FlushBuffer(buf, reln, io_object, io_context);
	BufferLockUnlock(buffer, buf);
}

Setting hint bits takes share-exclusive too — but if the backend only holds a share lock, it has to upgrade first, and the upgrade can fail:

c
/*
 * Helper for BufferBeginSetHintBits() and BufferSetHintBits16().
 *
 * This checks if the current lock mode already suffices to allow hint bits
 * being set and, if not, whether the current lock can be upgraded.
 *
 * Updates *lockstate when returning true.
 */
static inline bool
SharedBufferBeginSetHintBits(Buffer buffer, BufferDesc *buf_hdr, uint64 *lockstate)
{
	uint64		old_state;
	PrivateRefCountEntry *ref;
	BufferLockMode mode;

	ref = GetPrivateRefCountEntry(buffer, true);

	if (ref == NULL)
		elog(ERROR, "buffer is not pinned");

	mode = ref->data.lockmode;
	if (mode == BUFFER_LOCK_UNLOCK)
		elog(ERROR, "buffer is not locked");

	/* we're done if we are already holding a sufficient lock level */
	if (mode == BUFFER_LOCK_EXCLUSIVE || mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
	{
		*lockstate = pg_atomic_read_u64(&buf_hdr->state);
		return true;
	}

	/*
	 * We are only holding a share lock right now, try to upgrade it to
	 * SHARE_EXCLUSIVE.
	 */
	Assert(mode == BUFFER_LOCK_SHARE);

	old_state = pg_atomic_read_u64(&buf_hdr->state);
	while (true)
	{
		uint64		desired_state;

		desired_state = old_state;

		/*
		 * Can't upgrade if somebody else holds the lock in exclusive or
		 * share-exclusive mode.
		 */
		if (unlikely((old_state & (BM_LOCK_VAL_EXCLUSIVE | BM_LOCK_VAL_SHARE_EXCLUSIVE)) != 0))
		{
			return false;
		}

		/* currently held lock state */
		desired_state -= BM_LOCK_VAL_SHARED;

		/* new lock level */
		desired_state += BM_LOCK_VAL_SHARE_EXCLUSIVE;

		if (likely(pg_atomic_compare_exchange_u64(&buf_hdr->state,
												  &old_state, desired_state)))
		{
			ref->data.lockmode = BUFFER_LOCK_SHARE_EXCLUSIVE;
			*lockstate = desired_state;

			return true;
		}
	}
}

Note that the upgrade is a non-blocking CAS. If it fails — because somebody else holds share-exclusive or exclusive — the backend simply does not set the hint bits this time round. That is fine: hint bits are an optimisation, and the whole point is to avoid blocking a reader on a writer.

The comment on BufferBeginSetHintBits() spells out the trade-off:

c
/*
 * Try to acquire the right to set hint bits on the buffer.
 *
 * To be allowed to set hint bits, this backend needs to hold either a
 * share-exclusive or an exclusive lock. In case this backend only holds a
 * share lock, this function will try to upgrade the lock to
 * share-exclusive. The caller is only allowed to set hint bits if true is
 * returned.
 *
 * Once BufferBeginSetHintBits() has returned true, hint bits may be set
 * without further calls to BufferBeginSetHintBits(), until the buffer is
 * unlocked.
 *
 *
 * Requiring a share-exclusive lock to set hint bits prevents setting hint
 * bits on buffers that are currently being written out, which could corrupt
 * the checksum on the page. Flushing buffers also requires a share-exclusive
 * lock.
 *
 * Due to a lock >= share-exclusive being required to set hint bits, only one
 * backend can set hint bits at a time. Allowing multiple backends to set
 * hint bits would require more complicated locking: For setting hint bits
 * we'd need to store the count of backends currently setting hint bits, for
 * I/O we would need another lock-level conflicting with the hint-setting
 * lock-level. Given that the share-exclusive lock for setting hint bits is
 * only held for a short time, that backends often would just set the same
 * hint bits and that the cost of occasionally not setting hint bits in hotly
 * accessed pages is fairly low, this seems like an acceptable tradeoff.
 */

The “only one backend at a time” is a direct consequence of the share-lock field being 18 bits wide but the share-exclusive level being a single bit. There is room in the word for a second share-exclusive counter; there just is not one.

The lock-free clock sweep

The other place the new atomicity pays off is the buffer replacement strategy. src/backend/storage/buffer/freelist.c implements the clock sweep, and the clock hand is a plain atomic counter:

c
typedef struct
{
	/* Spinlock: protects the values below */
	slock_t		buffer_strategy_lock;

	/*
	 * clock-sweep hand: index of next buffer to consider grabbing. Note that
	 * this isn't a concrete buffer - we only ever increase the value. So, to
	 * get an actual buffer, it needs to be used modulo NBuffers.
	 */
	pg_atomic_uint32 nextVictimBuffer;

	/*
	 * Statistics.  These counters should be wide enough that they can't
	 * overflow during a single bgwriter cycle.
	 */
	uint32		completePasses; /* Complete cycles of the clock-sweep */
	pg_atomic_uint32 numBufferAllocs;	/* Buffers allocated since last reset */

	/*
	 * Bgworker process to be notified upon activity or -1 if none. See
	 * StrategyNotifyBgWriter.
	 */
	int			bgwprocno;
} BufferStrategyControl;

The hand moves with a single fetch_add:

c
static inline uint32
ClockSweepTick(void)
{
	uint32		victim;

	/*
	 * Atomically move hand ahead one buffer - if there's several processes
	 * doing this, this can lead to buffers being returned slightly out of
	 * apparent order.
	 */
	victim =
		pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);

	if (victim >= NBuffers)
	{
		uint32		originalVictim = victim;

		/* always wrap what we look up in BufferDescriptors */
		victim = victim % NBuffers;

		/*
		 * If we're the one that just caused a wraparound, force
		 * completePasses to be incremented while holding the spinlock. We
		 * need the spinlock so StrategySyncStart() can return a consistent
		 * value consisting of nextVictimBuffer and completePasses.
		 */
		if (victim == 0)
		{
			uint32		expected;
			uint32		wrapped;
			bool		success = false;

			expected = originalVictim + 1;

			while (!success)
			{
				/*
				 * Acquire the spinlock while increasing completePasses. That
				 * allows other readers to read nextVictimBuffer and
				 * completePasses in a consistent manner which is required for
				 * StrategySyncStart().  In theory delaying the increment
				 * could lead to an overflow of nextVictimBuffers, but that's
				 * highly unlikely and wouldn't be particularly harmful.
				 */
				SpinLockAcquire(&StrategyControl->buffer_strategy_lock);

				wrapped = expected % NBuffers;

				success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
														 &expected, wrapped);
				if (success)
					StrategyControl->completePasses++;
				SpinLockRelease(&StrategyControl->buffer_strategy_lock);
			}
		}
	}
	return victim;
}

There is exactly one spinlock left in the whole file, and it is held only for the wraparound bookkeeping. Advancing the hand is a single atomic add — no lock, ever.

The interesting part is StrategyGetBuffer(), which selects a victim and pins it without ever taking the buffer header spinlock:

c
	/* Use the "clock sweep" algorithm to find a free buffer */
	trycounter = NBuffers;
	for (;;)
	{
		uint64		old_buf_state;
		uint64		local_buf_state;

		buf = GetBufferDescriptor(ClockSweepTick());

		/*
		 * Check whether the buffer can be used and pin it if so. Do this
		 * using a CAS loop, to avoid having to lock the buffer header.
		 */
		old_buf_state = pg_atomic_read_u64(&buf->state);
		for (;;)
		{
			local_buf_state = old_buf_state;

			/*
			 * If the buffer is pinned or has a nonzero usage_count, we cannot
			 * use it; decrement the usage_count (unless pinned) and keep
			 * scanning.
			 */

			if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0)
			{
				if (--trycounter == 0)
				{
					/*
					 * We've scanned all the buffers without making any state
					 * changes, so all the buffers are pinned (or were when we
					 * looked at them). We could hope that someone will free
					 * one eventually, but it's probably better to fail than
					 * to risk getting stuck in an infinite loop.
					 */
					elog(ERROR, "no unpinned buffers available");
				}
				break;
			}

			/* See equivalent code in PinBuffer() */
			if (unlikely(local_buf_state & BM_LOCKED))
			{
				old_buf_state = WaitBufHdrUnlocked(buf);
				continue;
			}

			if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0)
			{
				local_buf_state -= BUF_USAGECOUNT_ONE;

				if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
												   local_buf_state))
				{
					trycounter = NBuffers;
					break;
				}
			}
			else
			{
				/* pin the buffer if the CAS succeeds */
				local_buf_state += BUF_REFCOUNT_ONE;

				if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
												   local_buf_state))
				{
					/* Found a usable buffer */
					if (strategy != NULL)
						AddBufferToRing(strategy, buf);
					*buf_state = local_buf_state;

					TrackNewBufferPin(BufferDescriptorGetBuffer(buf));

					return buf;
				}
			}
		}
	}

Read the two CAS cases:

Usage count is non-zero. The buffer is not pinned, but it has been used recently, so it gets a second chance: decrement the usage count and move on. The CAS is needed because the usage count and the refcount live in the same word, and we must not clobber a concurrent refcount change.
Usage count is zero and refcount is zero. The buffer is free. Add one to the refcount — that is the pin — and take it.

The pin and the “is this buffer still free?” decision are the same CAS. Compare that with the old code, which had to take the buffer header spinlock to increment the refcount and then release it. Here, the only time the header lock is involved at all is when BM_LOCKED happens to be set, in which case the code waits for it to clear and retries:

c
			/* See equivalent code in PinBuffer() */
			if (unlikely(local_buf_state & BM_LOCKED))
			{
				old_buf_state = WaitBufHdrUnlocked(buf);
				continue;
			}

The second chance, quantified

The usage count is four bits wide, capped at 5:

c
/*
 * The maximum allowed value of usage_count represents a tradeoff between
 * accuracy and speed of the clock-sweep buffer management algorithm.  A
 * large value (comparable to NBuffers) would approximate LRU semantics.
 * But it can take as many as BM_MAX_USAGE_COUNT+1 complete cycles of the
 * clock-sweep hand to find a free buffer, so in practice we don't want the
 * value to be very large.
 */
#define BM_MAX_USAGE_COUNT	5

StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS),
				 "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits");

Five is a compromise, and the comment says so: a large value would approximate LRU, but the clock hand might need BM_MAX_USAGE_COUNT + 1 full passes over the buffer array to find a free buffer. With shared_buffers = 128 GB and 8 kB pages that is 16 million buffers, and a value of 5 means at most 6 passes.

The value is bumped on pin, capped at the maximum:

c
			/* increase refcount */
			buf_state += BUF_REFCOUNT_ONE;

			if (strategy == NULL)
			{
				/* Default case: increase usagecount unless already max. */
				if (BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
					buf_state += BUF_USAGECOUNT_ONE;
			}

This is CLOCK in the textbook sense: the hand gives each buffer a bounded number of second chances, which makes the eviction policy an approximation of LRU rather than a strict LRU.

Caveats

This is a big, invasive change

The commit touches the buffer manager, the buffer replacement strategy, the resource owner machinery, the wait-event taxonomy, and the index access methods. The diff is large, and the new code deliberately duplicates a meaningful chunk of lwlock.c.

The commit message is explicit about the cost:

This commit unfortunately introduces some code that is very similar to the code in lwlock.c, however the code is not equivalent enough to easily merge it. The future wins that this commit makes possible seem worth the cost.

And about the wart:

It’s worth calling out one wart in this commit: Despite content locks not being lwlocks anymore, they continue to use PGPROC->lw* - that seemed better than duplicating the relevant infrastructure.

The wait event taxonomy changed

Content locks are no longer reported as LWLock waits. There are three new wait events in the Buffer class:

text

BUFFER_SHARED	"Waiting to acquire a shared lock on a buffer."
BUFFER_SHARE_EXCLUSIVE	"Waiting to acquire a share exclusive lock on a buffer."
BUFFER_EXCLUSIVE	"Waiting to acquire an exclusive lock on a buffer."

The old BufferContent LWLock tranche is gone, and the BUFFERPIN wait event class was renamed to BUFFER in commit 6c5c393b740 to make room for them. Any monitoring that watched for LWLock waits on the buffer content tranche needs updating.

A backend can only hold one lock at a time

The implementation does not track multiple lock ownerships of the same buffer within one backend. BufferLockConditional() documents the consequence:

c
/*
 * Acquire the content lock for the buffer, but only if we don't have to wait.
 *
 * It is allowed to try to conditionally acquire a lock on a buffer that this
 * backend has already locked, but the lock acquisition will always fail, even
 * if the new lock acquisition does not conflict with an already held lock
 * (e.g. two share locks). This is because we currently do not have space to
 * track multiple lock ownerships of the same buffer within one backend.  That
 * is ok for the current uses of BufferLockConditional().
 */

So a backend that already holds a share lock on a buffer cannot conditionally upgrade it; BufferLockConditional() returns false. The non-conditional upgrade path (SharedBufferBeginSetHintBits()) does its own CAS and works fine.

The 18-bit share-lock field holds at most 262 143 concurrent share lockers. That is exactly MAX_BACKENDS, so it cannot overflow in practice — but it also means the design assumes at most one share lock per backend per buffer, which is what the private refcount hash enforces.

Deadlock semantics are essentially unchanged

The queueing algorithm is the LWLock algorithm, so the deadlock properties are the same: the wait list is FIFO-ish, exclusive waiters block later waiters, and there is no deadlock detector — PostgreSQL relies on the lock ordering conventions of the code that takes the locks.

The one structural difference is that content locks and LWLocks are now different lock types. A backend that holds a buffer content lock and then tries to take an LWLock (or vice versa) could in principle deadlock in a way the old design could not, since the old content lock was an LWLock and the global lock-ordering discipline applied uniformly. The commit’s answer to this is the PGPROC->lwWaiting check:

c
	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

A backend cannot block on a buffer content lock while already blocked on something else — that case is a PANIC, not a deadlock. Which is a stronger guarantee than a deadlock detector, but only because it is enforced by the PGPROC state.

Platforms without 64-bit atomics

The whole design rests on pg_atomic_uint64. On platforms that lack native 64-bit atomic support, PostgreSQL emulates them with a spinlock (src/include/port/atomics/fallback.h):

c
#if !defined(PG_HAVE_ATOMIC_U64_SUPPORT)

#define PG_HAVE_ATOMIC_U64_SIMULATION

#define PG_HAVE_ATOMIC_U64_SUPPORT
typedef struct pg_atomic_uint64
{
	int			sema;
	volatile uint64 value;
} pg_atomic_uint64;

#define PG_HAVE_ATOMIC_INIT_U64
extern void pg_atomic_init_u64_impl(volatile pg_atomic_uint64 *ptr, uint64 val_);

#define PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U64
extern bool pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
												uint64 *expected, uint64 newval);

#define PG_HAVE_ATOMIC_FETCH_ADD_U64
extern uint64 pg_atomic_fetch_add_u64_impl(volatile pg_atomic_uint64 *ptr, int64 add_);

#endif /* PG_HAVE_ATOMIC_U64_SUPPORT */

and the implementation in src/backend/port/atomics.c:

c
bool
pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
									uint64 *expected, uint64 newval)
{
	bool		ret;

	/*
	 * Do atomic op under a spinlock. It might look like we could just skip
	 * the cmpxchg if the lock isn't available, but that'd just emulate a
	 * 'weak' compare and swap. I.e. one that allows spurious failures. Since
	 * several algorithms rely on a strong variant and that is efficiently
	 * implementable on most major architectures let's emulate it here as
	 * well.
	 */
	SpinLockAcquire((slock_t *) &ptr->sema);

	/* perform compare/exchange logic */
	ret = ptr->value == *expected;
	*expected = ptr->value;
	if (ret)
		ptr->value = newval;

	/* and release lock */
	SpinLockRelease((slock_t *) &ptr->sema);

	return ret;
}

The other operations are built on top of the CAS, in src/include/port/atomics/generic.h:

c
#if !defined(PG_HAVE_ATOMIC_FETCH_OR_U64) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U64)
#define PG_HAVE_ATOMIC_FETCH_OR_U64
static inline uint64
pg_atomic_fetch_or_u64_impl(volatile pg_atomic_uint64 *ptr, uint64 or_)
{
	uint64 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u64_impl(ptr, &old, old | or_))
		/* skip */;
	return old;
}
#endif

So on such a platform, LockBufHdr() — a single fetch_or — becomes a spinlock acquire and a CAS loop. The logical result is identical; the performance is not. The whole point of the change is to make the common path cheap, and on a fallback platform it is not cheap at all.

This matters more for the content lock than for the header lock. The header lock was already a BM_LOCKED bit in the state word before this change — commit dac328c8a68, “bufmgr: Change BufferDesc.state to be a 64-bit atomic”, had just widened that word from 32 to 64 bits — but it was a bit on 32-bit atomics, which have native support on every platform PostgreSQL builds on. The content lock, likewise, was an LWLock, whose own state word is a uint32. Moving both into a 64-bit word turns a native lock acquisition into a spinlock-guarded CAS on a platform without 64-bit atomics.

On every mainstream platform — x86-64, ARM64, POWER, s390x — 64-bit atomics are native and the fallback is not used.

The state word is now a contention point

Before the change, the refcount lived in the state word and the lock lived in the LWLock. Now they live together. That is the whole point — it lets the lock decision see the refcount — but it also means that unrelated operations on the same buffer now touch the same cache line. A buffer that is being pinned and unpinned rapidly is a buffer whose state word is hot.

The BufferDesc is padded to a cache line to keep it from sharing with its neighbours:

c
/*
 * Concurrent access to buffer headers has proven to be more efficient if
 * they're cache line aligned. So we force the start of the BufferDescriptors
 * array to be on a cache line boundary and force the elements to be cache
 * line sized.
 *
 * XXX: As this is primarily matters in highly concurrent workloads which
 * probably all are 64bit these days, and the space wastage would be a bit
 * more noticeable on 32bit systems, we don't force the stride to be cache
 * line sized on those. If somebody does actual performance testing, we can
 * reevaluate.
 *
 * Note that local buffer descriptors aren't forced to be aligned - as there's
 * no concurrent access to those it's unlikely to be beneficial.
 *
 * We use a 64-byte cache line size here, because that's the most common
 * size. Making it bigger would be a waste of memory. Even if running on a
 * platform with either 32 or 128 byte line sizes, it's good to align to
 * boundaries and avoid false sharing.
 */
#define BUFFERDESC_PAD_TO_SIZE	(SIZEOF_VOID_P == 8 ? 64 : 1)

typedef union BufferDescPadded
{
	BufferDesc	bufferdesc;
	char		pad[BUFFERDESC_PAD_TO_SIZE];
} BufferDescPadded;

Note the XXX comment: on 32-bit systems the padding is disabled, because the waste would be noticeable. The header comment in the struct is explicit about the constraint:

Be careful to avoid increasing the size of the struct when adding or reordering members. Keeping it below 64 bytes (the most common CPU cache line size) is fairly important for performance.

Removing the embedded LWLock (16 bytes) and replacing it with an 8-byte proclist_head is what keeps the struct within the budget.

Local buffers are a different story

Local buffers (temp tables) use a completely separate mechanism, and the content lock is a no-op for them:

c
	Assert(BufferIsPinned(buffer));
	if (BufferIsLocal(buffer))
		return;					/* local buffers need no lock */

Because a local buffer belongs to exactly one backend, there is nothing to lock against.

What it all means in practice

The buffer content lock is now the cheapest lock in PostgreSQL. In the uncontended case it is a single compare-and-swap on a word that is already in the cache. There is no lock structure to walk, no separate state word to coherency-check, no LWLock array to index.

The lock state and the pin state are now the same word. This is the structural change. It means:

Deciding whether a buffer is free to evict and pinning it are one atomic operation (StrategyGetBuffer()).
Checking whether I/O is in progress and taking an exclusive lock are atomic with respect to each other — which is what makes asynchronous writes possible.
Releasing a lock and dropping a pin can be fused into one CAS.

Hint-bit setting is now correct with respect to write-out. Before, a reader could set a hint bit on a page while the checkpointer was writing that same page out, forcing an expensive page copy. Now the two are mutually exclusive via the share-exclusive level, and the copy can go away. That matters more as PostgreSQL moves towards AIO, where many buffers are in flight at once.

The lock-free clock sweep is the quiet win. StrategyGetBuffer() no longer takes the buffer header spinlock to pin a victim. On a busy system with a large shared_buffers, the clock hand’s path through the buffer array no longer serialises on a lock.

Watch the wait events. If you have dashboards that count LWLock waits on BufferContent, they are now dead. The replacements are Buffer/Shared, Buffer/ShareExclusive and Buffer/Exclusive.