Storage on Opensource internals

PostgreSQL Stores a Row

Jay Hubbert — Tue, 20 Oct 2026 09:00:00 +0000

Every database gives you a row and every database lies a little about what that row is. MySQL gives you a record copied into a row buffer. SQLite gives you a record in a b-tree leaf. PostgreSQL gives you something stranger and more interesting: a 23-byte header followed by a payload, living at some byte offset on an 8 kB page, reached through a 4-byte indirection that lives in a completely different part of that same page.

That indirection is the reason PostgreSQL can move tuples around during VACUUM without invalidating a single index entry. The header is the reason an UPDATE can leave a trail of old versions behind. The slotted page is the reason a table’s rows are not laid out contiguously — and why reclaiming the space left behind by dead rows is a whole subsystem.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

A row is not a record

Let’s start with the container. PostgreSQL stores a table as an array of fixed-size pages — 8 kB by default, configurable at initdb time. The heap access method never stores anything that isn’t on a page, and it never stores a row that spans two pages. A row that doesn’t fit in 8 kB gets pushed out of line entirely (more on that later).

Each page is a slotted page. From src/include/storage/bufpage.h:

 c
 
 
/*
 * a postgres disk page is always a slotted page of the form:
 *
 * +----------------+---------------------------------+
 * | PageHeaderData | linp1 linp2 linp3 ... |
 * +-----------+----+---------------------------------+
 * | ... linpN |									 |
 * +-----------+--------------------------------------+
 * |		 ^ pd_lower							 |
 * |												 |
 * |			 v pd_upper							 |
 * +-------------+------------------------------------+
 * |			 | tupleN ... |
 * +-------------+------------------+-----------------+
 * |	 ... tuple3 tuple2 tuple1 | "special space" |
 * +--------------------------------+-----------------+
 *									^ pd_special
 */

Five parts, and they grow in opposite directions:

Region	Size on an 8 kB page	Grows
`PageHeaderData`	24 bytes	fixed
Line pointer array	4 bytes per row	upwards from `pd_lower`
Free space	whatever is left	—
Tuples	variable	downwards from `pd_upper`
Special space	0 for heap pages	fixed

The header is small enough to write out in full:

 c
 
 
typedef struct PageHeaderData
{
	PageXLogRecPtr pd_lsn;		/* LSN: next byte after last byte of xlog
								 * record for last change to this page */
	uint16		pd_checksum;	/* checksum */
	uint16		pd_flags;		/* flag bits, see below */
	LocationIndex pd_lower;		/* offset to start of free space */
	LocationIndex pd_upper;		/* offset to end of free space */
	LocationIndex pd_special;	/* offset to start of special space */
	uint16		pd_pagesize_version;
	TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */
	ItemIdData	pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */
} PageHeaderData;

pd_lower and pd_upper are the entire free-space bookkeeping. pd_lower marks the end of the line pointer array, pd_upper marks the start of the tuple area; the gap between them is what’s available. There is no free-list, no extent map, no per-page allocation structure. Free space is just an interval, and it is used from both ends.

A heap page has pd_special == BLCKSZ, because the “special space” is where an index access method stashes its own per-page state. A btree leaf page, for instance, ends with a BTPageOpaqueData holding the sibling links and the tree level. Heap pages have nothing to say for themselves.

Line pointers: the indirection that makes everything possible

The line pointer array is where the “row” abstraction starts to leak. Each entry is four bytes (src/include/storage/itemid.h):

 c
 
 
typedef struct ItemIdData
{
	unsigned	lp_off:15,		/* offset to tuple (from start of page) */
				lp_flags:2,		/* state of line pointer, see below */
				lp_len:15;		/* byte length of tuple */
} ItemIdData;

Fifteen bits of byte offset, fifteen bits of length, two bits of state. lp_flags has four values:

 c
 
 
#define LP_UNUSED		0		/* unused (should always have lp_len=0) */
#define LP_NORMAL		1		/* used (should always have lp_len>0) */
#define LP_REDIRECT		2		/* HOT redirect (should have lp_len=0) */
#define LP_DEAD			3		/* dead, may or may not have storage */

Note the convention, stated in the header comment: by convention lp_len == 0 in every line pointer that does not have storage, independently of its lp_flags state. A line pointer can be “in use” and still point at nothing. That is the whole trick.

So a row is addressed by a TID — a tuple identifier — which is a (block number, offset number) pair. The block number says which page; the offset number says which line pointer. The line pointer then says which byte range the tuple occupies. Three levels of addressing, and only the last one is a physical location.

This indirection is not incidental. It’s what lets VACUUM slide tuples around within a page, or move them between pages, and leave every index entry untouched — the index stores a TID, and the TID still resolves to the right line pointer, and the line pointer has been updated to point at the tuple’s new home.

Two details worth knowing:

The number of line pointers is derived, not stored. It is (pd_lower - SizeOfPageHeaderData) / sizeof(ItemIdData). There’s no counter; the array simply grows.
A line pointer’s lp_off is overloaded. For LP_NORMAL it’s a byte offset. For LP_REDIRECT it’s the offset number of another line pointer on the same page. Same field, completely different meaning.

The tuple header: 23 bytes

Now the header at the front of every heap tuple (src/include/access/htup_details.h):

 c
 
 
struct HeapTupleHeaderData
{
	union
	{
		HeapTupleFields t_heap;
		DatumTupleFields t_datum;
	}			t_choice;

	ItemPointerData t_ctid;		/* current TID of this or newer tuple (or a
								 * speculative insertion token) */

	/* Fields below here must match MinimalTupleData! */

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK2 2
	uint16		t_infomask2;	/* number of attributes + various flags */

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK 3
	uint16		t_infomask;		/* various flag bits, see below */

#define FIELDNO_HEAPTUPLEHEADERDATA_HOFF 4
	uint8		t_hoff;			/* sizeof header incl. bitmap, padding */

	/* ^ - 23 bytes - ^ */

#define FIELDNO_HEAPTUPLEHEADERDATA_BITS 5
	uint8		t_bits[FLEXIBLE_ARRAY_MEMBER];	/* bitmap of NULLs */

	/* MORE DATA FOLLOWS AT END OF STRUCT */
};

Twenty-three bytes. Here is what each byte is doing, on a 64-bit machine with MAXIMUM_ALIGNOF == 8:

Offset	Bytes	Field	Meaning
0	4	`t_xmin`	XID of the inserting transaction
4	4	`t_xmax`	XID of the deleting or locking transaction
8	4	`t_cid`	insert and/or delete command ID (overlays `t_xvac`)
12	4	`t_ctid.ip_blkid`	block number of this or a newer version
16	2	`t_ctid.ip_posid`	offset number of this or a newer version
18	2	`t_infomask2`	attribute count + HOT/heap-only flags
20	2	`t_infomask`	content and visibility flags
22	1	`t_hoff`	offset of the user data

ItemPointerData is six bytes, not eight, and it gets there by fiat:

 c
 
 
/* If compiler understands packed and aligned pragmas, use those */
#if defined(pg_attribute_packed) && defined(pg_attribute_aligned)
			pg_attribute_packed()
			pg_attribute_aligned(2)
#endif
ItemPointerData;

The comment above it explains why: “because there is an item pointer in each tuple header and index tuple header on disk, it’s very important not to waste space with structure padding bytes.” Two bytes saved here, on every row, forever.

Five virtual fields in three physical ones

The header stores five logical fields — Xmin, Cmin, Xmax, Cmax, Xvac — in three physical slots. From the header comment:

We store five “virtual” fields Xmin, Cmin, Xmax, Cmax, and Xvac in three physical fields. Xmin and Xmax are always really stored, but Cmin, Cmax and Xvac share a field. This works because we know that Cmin and Cmax are only interesting for the lifetime of the inserting and deleting transaction respectively.

Cmin and Cmax are command IDs — counters that order the statements within a single transaction. If the same transaction inserts and then deletes a row, both values have to survive. PostgreSQL solves this with a “combo CID”: a synthetic value that maps back to both, using local state in the originating backend. The HEAP_COMBOCID bit in t_infomask says “this field is a combo”.

Before 8.3, HeapTupleHeaderData had separate cmin and cmax fields. Removing one 4-byte field from every row in every table was worth the combo-CID machinery.

t_infomask

Sixteen bits, split into two groups. The low four describe the physical contents of the tuple:

 c
 
 
#define HEAP_HASNULL			0x0001	/* has null attribute(s) */
#define HEAP_HASVARWIDTH		0x0002	/* has variable-width attribute(s) */
#define HEAP_HASEXTERNAL		0x0004	/* has external stored attribute(s) */
#define HEAP_HASOID_OLD			0x0008	/* has an object-id field */

The upper twelve are about visibility — who can see this row, and what happened to it:

 c
 
 
#define HEAP_XMAX_KEYSHR_LOCK	0x0010	/* xmax is a key-shared locker */
#define HEAP_COMBOCID			0x0020	/* t_cid is a combo CID */
#define HEAP_XMAX_EXCL_LOCK		0x0040	/* xmax is exclusive locker */
#define HEAP_XMAX_LOCK_ONLY		0x0080	/* xmax, if valid, is only a locker */

#define HEAP_XMIN_COMMITTED		0x0100	/* t_xmin committed */
#define HEAP_XMIN_INVALID		0x0200	/* t_xmin invalid/aborted */
#define HEAP_XMIN_FROZEN		(HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID)
#define HEAP_XMAX_COMMITTED		0x0400	/* t_xmax committed */
#define HEAP_XMAX_INVALID		0x0800	/* t_xmax invalid/aborted */
#define HEAP_XMAX_IS_MULTI		0x1000	/* t_xmax is a MultiXactId */
#define HEAP_UPDATED			0x2000	/* this is UPDATEd version of row */
#define HEAP_MOVED_OFF			0x4000	/* moved to another place by pre-9.0
										 * VACUUM FULL; kept for binary
										 * upgrade support */
#define HEAP_MOVED_IN			0x8000	/* moved from another place by pre-9.0
										 * VACUUM FULL; kept for binary
										 * upgrade support */

HEAP_XMIN_COMMITTED and friends are hint bits. They cache a fact that could otherwise require looking up the transaction’s commit record in CLOG. Once a reader has established that the inserting transaction committed, it stamps the tuple and everyone afterwards gets the answer for free. The rules are conservative — a hint is only set when the answer is known-good — and the code is careful never to set a hint that would be wrong if the buffer is later written to disk:

 c
 
 
	if (TransactionIdIsValid(xid))
	{
		if (BufferIsPermanent(buffer))
		{
			/* NB: xid must be known committed here! */
			XLogRecPtr	commitLSN = TransactionIdGetCommitLSN(xid);

			if (XLogNeedsFlush(commitLSN) &&
				BufferGetLSNAtomic(buffer) < commitLSN)
			{
				/* not flushed and no LSN interlock, so don't set hint */
				return;
			}
		}
	}

Notice the subtlety: if the commit record isn’t on disk yet, the hint isn’t set, because the page might reach disk before the commit record does, and then a crash would leave a tuple claiming to be committed when it isn’t.

t_infomask2

 c
 
 
#define HEAP_NATTS_MASK			0x07FF	/* 11 bits for number of attributes */
/* bits 0x1800 are available */
#define HEAP_KEYS_UPDATED		0x2000	/* tuple was updated and key cols
										 * modified, or tuple deleted */
#define HEAP_HOT_UPDATED		0x4000	/* tuple was HOT-updated */
#define HEAP_ONLY_TUPLE			0x8000	/* this is heap-only tuple */

The low eleven bits hold the attribute count, which is why t_infomask2 is a uint16 and the number of columns is limited. The other three bits carry the HOT state machine, which we will come back to.

The null bitmap and the 23/24 trick

HEAP_HASNULL in t_infomask means a null bitmap is present, immediately after the fixed header, one bit per column:

 c
 
 
static inline int
BITMAPLEN(int NATTS)
{
	return (NATTS + 7) / 8;
}

The bitmap is inverted — a 1 bit means not null:

 c
 
 
/*
 * Check a tuple's null bitmap to determine whether the attribute is null.
 * Note that a 0 in the null bitmap indicates a null, while 1 indicates
 * non-null.
 */
static inline bool
att_isnull(int ATT, const uint8 *BITS)
{
	return !(BITS[ATT >> 3] & (1 << (ATT & 0x07)));
}

Attribute 0 is the low bit of byte 0. (This is the sort of detail that makes hand-parsing a heap page a mildly masochistic exercise. The pageinspect extension exists for a reason.)

Now, t_hoff is the offset from the start of the tuple to the first byte of user data, and it must be a multiple of MAXALIGN — 8 bytes on a 64-bit machine. The reason is that the data area must be 8-byte aligned so that fetchatt() can do a direct aligned load of an int8 or double column.

The calculation, in heap_form_tuple() (src/backend/access/common/heaptuple.c):

 c
 
	len = offsetof(HeapTupleHeaderData, t_bits);

	if (hasnull)
		len += BITMAPLEN(numberOfAttributes);

	hoff = len = MAXALIGN(len); /* align user data safely */

offsetof(HeapTupleHeaderData, t_bits) is 23. So:

No NULLs: len = 23 → MAXALIGN(23) = 24 → t_hoff = 24. Byte 23 is pure padding.
1–8 columns, some NULL: len = 23 + 1 = 24 → MAXALIGN(24) = 24 → t_hoff = 24. The bitmap lands exactly in the padding byte.
9–16 columns, some NULL: len = 23 + 2 = 25 → MAXALIGN(25) = 32 → t_hoff = 32. Seven bytes of padding.

So the header is a 23-byte struct that behaves like a 24-byte header. The one-byte discrepancy is deliberate: a 24-byte header is 8-byte aligned, so the data area inherits that alignment for free, and any 4-byte-aligned field inside it is also correctly aligned. Had the header been 24 bytes and the bitmap 1 byte, the data would start at 25 — unaligned.

There is a hard limit hiding here. t_hoff is a uint8, so it can hold at most 255. The comment in htup_details.h spells out the constraint:

The key limit on this value is that the size of the fixed overhead for a tuple, plus the size of the null-values bitmap (at 1 bit per column), plus MAXALIGN alignment, must fit into t_hoff which is uint8. On most machines the upper limit without making t_hoff wider would be a little over 1700.

Which is why MaxTupleAttributeNumber is 1664 and MaxHeapAttributeNumber is 1600. PostgreSQL’s real column limit is a storage-layout limit, not an arbitrary one.

The data area

After t_hoff come the column values, packed in attribute order. The rules come out of fill_val() in heaptuple.c:

NULL columns cost zero bytes. fill_val sets the bitmap bit and returns before writing any data. The space is genuinely not reserved.
By-value columns (int4, bool, oid, …) are aligned to their attalign and stored inline.
Fixed-length by-reference columns (numeric, uuid, …) are aligned and copied inline.
Short varlena (a text of 126 bytes or less) gets no alignment padding at all — it can sit at any byte offset.
Full 4-byte-header varlena is aligned, because the 4-byte length word must be naturally aligned.
cstring (attlen == -2) never needs alignment.

The interesting one is #4. Here is the code:

 c
 
 
		else if (VARATT_IS_SHORT(val))
		{
			/* no alignment for short varlenas */
			data_length = VARSIZE_SHORT(val);
			memcpy(data, val, data_length);
		}
		else if (att->attispackable && VARATT_CAN_MAKE_SHORT(val))
		{
			/* convert to short varlena -- no alignment */
			data_length = VARATT_CONVERTED_SHORT_SIZE(val);
			SET_VARSIZE_SHORT(data, data_length);
			memcpy(data + 1, VARDATA(val), data_length - 1);
		}
		else
		{
			/* full 4-byte header varlena */
			data = (char *) att_nominal_alignby(data, att->attalignby);
			data_length = VARSIZE(val);
			memcpy(data, val, data_length);
		}

A text value whose payload is 126 bytes or less is stored with a one-byte header and needs no padding. A payload of 127 bytes cannot be short — the limit is VARATT_SHORT_MAX (0x7F) — so it falls back to the 4-byte-header format, which is aligned, and the value suddenly costs several bytes more than the one below it. It is a real, if obscure, discontinuity in the space cost of text columns.

Variable-length values: the four header formats

Every variable-length value shares the varlena concept, but there are four on-disk encodings, distinguished by the low bits of the first header byte (src/include/varatt.h):

 c
 
 
typedef struct
{
	struct						/* Normal varlena (4-byte length) */
	{
		uint32		va_header;
		char		va_data[FLEXIBLE_ARRAY_MEMBER];
	}			va_4byte;
	struct						/* Compressed-in-line format */
	{
		uint32		va_header;
		uint32		va_tcinfo;	/* Original data size (excludes header) and
								 * compression method; see va_extinfo */
		char		va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
	}			va_compressed;
} varattrib_4b;

typedef struct
{
	uint8		va_header;
	char		va_data[FLEXIBLE_ARRAY_MEMBER]; /* Data begins here */
} varattrib_1b;

typedef struct
{
	uint8		va_header;		/* Always 0x80 or 0x01 */
	uint8		va_tag;			/* Type of datum */
	char		va_data[FLEXIBLE_ARRAY_MEMBER]; /* Type-specific data */
} varattrib_1b_e;

4-byte header — the general case. The header stores the length in its upper 30 bits and flags in the low bits (compressed / external).
1-byte header — the “short” format, for payloads up to VARATT_SHORT_MAX (0x7F = 127) bytes. One bit of the header is reserved as a flag; the other seven hold the length.
1-byte header + tag — the external (TOAST pointer) format. Also a one-byte header, so TOAST pointers cost no alignment.

TOAST: when a row doesn’t fit

A tuple can be at most MaxHeapTupleSize:

#define MaxHeapTupleSize (BLCKSZ - MAXALIGN(SizeOfPageHeaderData + sizeof(ItemIdData)))

With BLCKSZ 8192, a 24-byte page header and a 4-byte line pointer, that’s 8192 − 32 = 8160 bytes.

But TOAST kicks in much earlier. From src/include/access/heaptoast.h:

 c
 
 
/*
 * Find the maximum size of a tuple if there are to be N tuples per page.
 */
#define MaximumBytesPerTuple(tuplesPerPage) \
	MAXALIGN_DOWN((BLCKSZ - \
				 MAXALIGN(SizeOfPageHeaderData + (tuplesPerPage) * sizeof(ItemIdData))) \
				 / (tuplesPerPage))

#define TOAST_TUPLES_PER_PAGE	4

#define TOAST_TUPLE_THRESHOLD	MaximumBytesPerTuple(TOAST_TUPLES_PER_PAGE)

#define TOAST_TUPLE_TARGET		TOAST_TUPLE_THRESHOLD

TOAST_TUPLE_THRESHOLD works out to 2032 bytes: the largest tuple such that four of them fit on a page. If a tuple exceeds that, the toaster takes over.

The toaster’s strategy, in order:

Compress every compressible column, in place. Compression is tried per-column, and a column is only kept compressed if it actually got smaller. A text column that compresses to 40% of its size stays compressed inline, with a 4-byte header that records both the compressed length and the original length.
Move EXTENDED and EXTERNAL columns out of line into the table’s TOAST table, replacing them in the row with an 18-byte pointer.
As a last resort, move PLAIN and MAIN columns out of line too, up to TOAST_TUPLE_TARGET_MAIN (8160 bytes — the largest tuple that fits on a page at all). The user said “keep this inline if at all possible”, so this only happens when nothing else works.

The TOAST pointer itself is the varattrib_1b_e format: a one-byte header, a tag byte, and a payload:

 c
 
 
typedef struct varatt_external_oid
{
	int32		va_rawsize;		/* Original data size (includes header) */
	uint32		va_extinfo;		/* External saved size (without header) and
								 * additional flag bits */
	Oid			va_valueid;		/* Unique ID of value within TOAST table */
	Oid			va_toastrelid;	/* RelID of TOAST table containing it */
} varatt_external_oid;

The key design decision: the pointer identifies the value by (toast table, value id), not by the physical location of the row that contains it. That is what makes TOAST pointers relocatable — a row can be moved by VACUUM FULL, CLUSTER, or ALTER TABLE ... SET TABLESPACE without rewriting its out-of-line data.

The TOAST table

Every table with toastable columns gets a companion table named pg_toast_<oid>, created in src/backend/catalog/toasting.c:

 c
 
 
	tupdesc = CreateTemplateTupleDesc(3);
	TupleDescInitEntry(tupdesc, (AttrNumber) 1,
					 "chunk_id",
					 toast_chunkid_typid,
					 -1, 0);
	TupleDescInitEntry(tupdesc, (AttrNumber) 2,
					 "chunk_seq",
					 INT4OID,
					 -1, 0);
	TupleDescInitEntry(tupdesc, (AttrNumber) 3,
					 "chunk_data",
					 BYTEAOID,
					 -1, 0);

Three columns: an identifier for the value, a chunk sequence number, and the chunk payload. Large values are split across multiple rows.

The chunk size is bounded by what has to fit on a page:

 c
 
 
#define TOAST_OID_MAX_CHUNK_SIZE	\
	(EXTERN_TUPLE_MAX_SIZE -							\
	 MAXALIGN(SizeofHeapTupleHeader) -					\
	 sizeof(Oid) -										\
	 sizeof(int32) -									\
	 VARHDRSZ)

EXTERN_TUPLE_MAX_SIZE is MaximumBytesPerTuple(4) = 2032, so the chunk payload is 2032 − 24 − 4 − 4 − 4 = 1996 bytes for an oid value id (and 1992 for the oid8 variant).

One last detail that is easy to miss:

 c
 
 
	/*
	 * Ensure that the toast table doesn't itself get toasted, or we'll be
	 * toast :-(. This is essential for chunk_data because type bytea is
	 * toastable; hit the other two just to be sure.
	 */
	TupleDescAttr(tupdesc, 0)->attstorage = TYPSTORAGE_PLAIN;
	TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN;
	TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN;

The TOAST table is explicitly not toastable. A row cannot contain a pointer to a value that lives in a table that itself contains pointers to values.

UPDATE: the version chain

Now that we know what a tuple looks like, we can see what an UPDATE does. PostgreSQL never modifies a row in place (with one narrow exception for system catalogs). It writes a new version and links the old one to it.

The link is t_ctid. From the header comment:

A word about t_ctid: whenever a new tuple is stored on disk, its t_ctid is initialized with its own TID (location). If the tuple is ever updated, its t_ctid is changed to point to the replacement version of the tuple.

So t_ctid is either a self-reference (this is the newest version) or a pointer to a successor. The header comment also warns about the obvious hazard:

Beware however that VACUUM might erase the pointed-to (newer) tuple before erasing the pointing (older) tuple. Hence, when following a t_ctid link, it is necessary to check to see if the referenced slot is empty or contains an unrelated tuple. Check that the referenced tuple has XMIN equal to the referencing tuple’s XMAX to verify that it is actually the descendant version.

That XMIN/XMAX check is the chain-integrity test, and it appears everywhere — in the index scan, in the pruner, and in the row-locking code. It’s how a reader knows it hasn’t followed a link into a recycled slot.

The actual update, in heap_update() (src/backend/access/heap/heapam.c):

 c
 
 
	newtup->t_data->t_infomask &= ~(HEAP_XACT_MASK);
	newtup->t_data->t_infomask2 &= ~(HEAP2_XACT_MASK);
	HeapTupleHeaderSetXmin(newtup->t_data, xid);
	HeapTupleHeaderSetCmin(newtup->t_data, cid);
	newtup->t_data->t_infomask |= HEAP_UPDATED | infomask_new_tuple;
	newtup->t_data->t_infomask2 |= infomask2_new_tuple;
	HeapTupleHeaderSetXmax(newtup->t_data, xmax_new_tuple);

The new version gets the updater’s XID as its xmin. Then, at the very end of the operation, the old version is finalized:

 c
 
 
	/* Clear obsolete visibility flags, possibly set by ourselves above... */
	oldtup.t_data->t_infomask &= ~(HEAP_XMAX_BITS | HEAP_MOVED);
	oldtup.t_data->t_infomask2 &= ~HEAP_KEYS_UPDATED;
	/* ... and store info about transaction updating this tuple */
	Assert(TransactionIdIsValid(xmax_old_tuple));
	HeapTupleHeaderSetXmax(oldtup.t_data, xmax_old_tuple);
	oldtup.t_data->t_infomask |= infomask_old_tuple;
	oldtup.t_data->t_infomask2 |= infomask2_old_tuple;
	HeapTupleHeaderSetCmax(oldtup.t_data, cid, iscombo);

	/* record address of new tuple in t_ctid of old one */
	oldtup.t_data->t_ctid = heaptup->t_self;

One line — oldtup.t_data->t_ctid = heaptup->t_self — is the entire mechanism of MVCC at the storage layer. The old version records where its successor lives, and the old version’s xmax records who did the update.

Notice also HEAP_XMAX_BITS, which is masked out before the new value is written:

 c
 
 
#define HEAP_XMAX_BITS (HEAP_XMAX_COMMITTED | HEAP_XMAX_INVALID | \
						HEAP_XMAX_IS_MULTI | HEAP_LOCK_MASK | HEAP_XMAX_LOCK_ONLY)

Everything about xmax is recomputed from scratch on every write, because the lock state can change independently of the row’s identity.

Visibility: the rule

Visibility is decided by HeapTupleSatisfiesMVCC(), and the rule is compact:

the tuple is visible iff xmin is committed-and-not-in-snapshot AND (xmax is invalid / lock-only / in-snapshot-and-not-committed)

The Cmin/Cmax comparison against the snapshot’s curcid is what distinguishes “my own earlier command updated this row” from “my own later command will update this row” — the TM_SelfModified versus TM_Invisible split. And one detail that is easy to miss:

 c
 
 
	if (tuple->t_infomask & HEAP_XMAX_COMMITTED)
	{
		if (HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask))
			return TM_Ok;
		if (!ItemPointerEquals(&htup->t_self, &tuple->t_ctid))
			return TM_Updated;	/* updated by other */
		else
			return TM_Deleted;	/* deleted by other */
	}

The same t_ctid field that carries the HOT chain also distinguishes UPDATE from DELETE. t_ctid != t_self means “this version was updated”; t_ctid == t_self means “this version was deleted”. One field, two questions, answered by the same comparison.

HOT: updating without touching the index

The obvious cost of an UPDATE is that it creates a new index entry for the new version. Heap-Only Tuples avoid that. From the manual:

To help reduce the overhead of updates, PostgreSQL has an optimization called heap-only tuples (HOT).

HOT applies when both of these hold:

The update does not modify any column referenced by an index (excluding summarizing indexes like BRIN).
There is sufficient free space on the page containing the old row.

Then the new version is written to the same page, no index entry is created, and the chain is walked from the root tuple via t_ctid. The flags make this possible:

 c
 
 
	if (use_hot_update)
	{
		/* Mark the old tuple as HOT-updated */
		HeapTupleSetHotUpdated(&oldtup);
		/* And mark the new tuple as heap-only */
		HeapTupleSetHeapOnly(heaptup);
		/* Mark the caller's copy too, in case different from heaptup */
		HeapTupleSetHeapOnly(newtup);
	}

HEAP_HOT_UPDATED marks the predecessor (“my successor is heap-only”); HEAP_ONLY_TUPLE marks the successor (“no index entry points at me”). The chain looks like this:

text

 btree index entry ──► (blk, offnum=1) root tuple
 t_ctid = (blk, offnum=2)
 t_infomask2 |= HEAP_HOT_UPDATED
 lp[2] ──► version 2
 t_ctid = (blk, offnum=3)
 t_infomask2 |= HEAP_HOT_UPDATED
 lp[3] ──► version 3
 t_ctid = (blk, offnum=3) (self — end)
 t_infomask2 |= HEAP_ONLY_TUPLE

The reader follows the chain in heap_hot_search_buffer() (src/backend/access/heap/heapam_indexscan.c):

 c
 
 
		/* Check to see if HOT chain continues past this tuple; if so fetch
		 * the next offnum and loop around.
		 */
		if (HeapTupleIsHotUpdated(heapTuple))
		{
			Assert(ItemPointerGetBlockNumber(&heapTuple->t_data->t_ctid) ==
				 blkno);
			offnum = ItemPointerGetOffsetNumber(&heapTuple->t_data->t_ctid);
			at_chain_start = false;
			prev_xmax = HeapTupleHeaderGetUpdateXid(heapTuple->t_data);
		}
		else
			break;				/* end of chain */

Note the assertion: a HOT chain never crosses a page boundary. The successor is always in the same block. That is why HOT requires free space on the same page — a chain that hopped pages would need a different mechanism to find the next version.

The chain-integrity check follows immediately after:

 c
 
 
		/*
		 * The xmin should match the previous xmax value, else chain is
		 * broken.
		 */
		if (TransactionIdIsValid(prev_xmax) &&
			!TransactionIdEquals(prev_xmax,
								 HeapTupleHeaderGetXmin(heapTuple->t_data)))
			break;

Pruning: reclaiming the space

Dead versions accumulate. When a row is updated ten times, there are ten tuples on the page, and nine of them are garbage once no transaction can see them.

Pruning happens opportunistically, during ordinary scans, via heap_page_prune_opt(). The trigger is fillfactor-aware:

 c
 
 
	minfree = RelationGetTargetPageFreeSpace(relation,
											 HEAP_DEFAULT_FILLFACTOR);
	minfree = Max(minfree, BLCKSZ / 10);

	if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)

and it needs a cleanup lock, not merely an exclusive lock, because pruning can destroy line pointers other backends may hold references to.

The chain surgery is in heap_prune_chain(). Its header comment is the clearest statement of the algorithm in the whole tree:

The root line pointer is redirected to the tuple immediately after the latest DEAD tuple. If all tuples in the chain are DEAD, the root line pointer is marked LP_DEAD.

Three outcomes:

Situation	Root line pointer	Intermediate line pointers
No dead member	unchanged	unchanged
All members dead	→ `LP_DEAD`	→ `LP_UNUSED`
Some dead prefix	→ `LP_REDIRECT`	→ `LP_UNUSED`

The important subtlety: the t_ctid links are not rewritten during pruning. The chain is repaired by replacing the root line pointer’s storage with a redirect, not by patching t_ctid:

 c
 
		ItemIdSetRedirect(fromlp, tooff);

with

 c
 
 
#define ItemIdSetRedirect(itemId, link) \
( \
	(itemId)->lp_flags = LP_REDIRECT, \
	(itemId)->lp_off = (link), \
	(itemId)->lp_len = 0 \
)

The root keeps its offset number, so index entries pointing at it stay valid, but loses its storage (lp_len = 0) and repurposes lp_off to hold the offset number of the target. The reader follows the redirect at the start of a chain:

 c
 
 
		/* check for unused, dead, or redirected items */
		if (!ItemIdIsNormal(lp))
		{
			/* We should only see a redirect at start of chain */
			if (ItemIdIsRedirected(lp) && at_chain_start)
			{
				/* Follow the redirect */
				offnum = ItemIdGetRedirect(lp);
				at_chain_start = false;
				continue;
			}
			/* else must be end of chain */
			break;
		}

Now, the freed tuple bytes are still sitting there in the middle of the page, unreachable. PageRepairFragmentation() moves the survivors downward to close the gap:

 c
 
 
		/* shift the target offset down by the length of this tuple */
		upper -= itemidptr->alignedlen;
		/* point the copy_head to the start of this tuple */
		copy_head = itemidptr->itemoff;

		/* update the line pointer to reference the new offset */
		lp->lp_off = upper;

Note that lp_off is rewritten for every moved tuple. The line pointers are the only thing that has to stay stable; the tuples can go anywhere.

Two side structures

The free space map

The FSM is a per-relation array of pages, each page holding a tree of free-space values per heap page. It’s consulted when a tuple needs to be placed and the current target page is full, to find a page with room.

The visibility map

The VM is a bitmap with two bits per heap page — all-visible and all-frozen:

 c
 
 
/* Number of bits for one heap page */
#define BITS_PER_HEAPBLOCK 2

/* Flags for bit map */
#define VISIBILITYMAP_ALL_VISIBLE	0x01
#define VISIBILITYMAP_ALL_FROZEN	0x02

The VM page is nearly all bitmap:

 c
 
 
/*
 * Size of the bitmap on each visibility map page, in bytes. There's no
 * extra headers, so the whole page minus the standard page header is
 * used for the bitmap.
 */
#define MAPSIZE (BLCKSZ - MAXALIGN(SizeOfPageHeaderData))

8168 bytes of bitmap per VM page, four heap pages per byte, so 32 672 heap pages per VM page. The VM fork is about 1/32672 the size of the heap fork.

When every tuple on a heap page is visible to everyone, that page’s all-visible bit is set, and readers can skip the page entirely. This is why SELECT count(*) on an untouched table is nearly instant.

Look at it yourself

The pageinspect extension exposes all of this. It lives in contrib/pageinspect/ and is not installed by default. The raw-page functions require superuser:

 sql
 
 
CREATE EXTENSION pageinspect;

-- the line pointer array, and the header of every tuple on the page
SELECT lp, lp_off, lp_len, lp_flags, lp_flags = 1 AS normal,
 t_xmin, t_xmax, t_ctid, t_hoff
FROM heap_page_items(get_raw_page('mytable', 0));

-- the decoded flag names
SELECT (heap_tuple_infomask_flags(t_infomask, t_infomask2)).*
FROM heap_page_items(get_raw_page('mytable', 0));

-- where does each column live inside the tuple
SELECT t_ctid, s.*
FROM heap_page_items(get_raw_page('mytable', 0)) h,
 LATERAL unnest(tuple_data_split('mytable'::regclass,
 h.t_data, h.t_infomask,
 h.t_infomask2, h.t_bits)) s;

heap_page_items gives you the line pointer array and the raw tuple headers; tuple_data_split gives you the byte range of every column within a tuple. These two together are enough to verify everything in this article by hand.

What it all means in practice

A few consequences fall out of the layout, and they’re worth having in your head when you’re reasoning about a slow table:

Row width is expensive in a way that row count is not. The 23-byte header is paid by every row, but the alignment rules mean a table with one wide text column can waste more space in padding than a table with many narrow columns. MaxHeapTuplesPerPage is 291 on an 8 kB page:

 c
 
 
#define MaxHeapTuplesPerPage	\
	((int) ((BLCKSZ - SizeOfPageHeaderData) / \
			(MAXALIGN(SizeofHeapTupleHeader) + sizeof(ItemIdData))))

(8192 − 24) / (24 + 4) = 291. A page of 291 minimal rows has 291 tuples of 24 bytes plus 291 line pointers of 4 bytes, in 8192 bytes.

HOT updates are dramatically cheaper than cold ones. If you can update a row without touching an indexed column, and there’s room on the page, PostgreSQL writes a new tuple and nothing else. No index churn, no dead-index-entry cleanup. This is why fillfactor matters: a lower fillfactor leaves room on the page for the new version, and the chance of a HOT update goes up.

The 4-byte line pointer is a per-row tax on every page. It is the price of the indirection that makes VACUUM, HOT, and page defragmentation possible. It’s also why MaxHeapTupleSize is 8160 and not 8168.

An UPDATE never reuses the old tuple’s bytes. It appends a new version and leaves the old one for VACUUM to reclaim. A table that is updated constantly will bloat until vacuum catches up, and the bloat is not fragmentation — it’s whole tuples.

The t_ctid field is the load-bearing structure. It answers three questions at once: where is the newest version of this row, was this version updated or deleted, and where does the next HOT chain member live. Six bytes, three jobs.

PostgreSQL Locks Buffers Without LWLocks

Jay Hubbert — Fri, 02 Oct 2026 09:00:00 +0000

Every time PostgreSQL reads a page it does two things: it pins the shared buffer that holds the page, and it takes a content lock on that buffer so that nobody rewrites the bytes underneath it while it is being read.

The second of those used to be an LWLock — a general-purpose, one-word-at-a-time lightweight lock living in a global array in shared memory. PostgreSQL 20 changed that. Buffer content locks are now implemented entirely inside a single 64-bit atomic word that already existed in the buffer descriptor, with its own three-level lock hierarchy, its own wait queue, and its own wakeup logic.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

What a content lock protects

Let’s be precise about what is being locked, because PostgreSQL has two different “buffer locks” and they are easy to confuse.

The buffer header lock protects the descriptor — the metadata in BufferDesc: the tag, the validity flags, the I/O state. It is a very short critical section, taken for a handful of instructions at a time.

The buffer content lock protects the page data — the 8 kB of bytes in the buffer. It is held for as long as the caller is looking at the page. This is the one that matters for correctness: two backends cannot be reading the same page while one of them is rewriting it.

The header comment in src/include/storage/buf_internals.h is explicit about the division:

The content of buffers is protected via the buffer content lock, implemented as part of the buffer state. Note that the buffer header lock is not used to control access to the data in the buffer! We used to use an LWLock to implement the content lock, but having a dedicated implementation of content locks allows us to implement some otherwise hard things (e.g. race-freely checking if AIO is in progress before locking a buffer exclusively) and enables otherwise impossible optimizations (e.g. unlocking and unpinning a buffer in one atomic operation).

That comment is the entire design rationale in one paragraph, and it is worth reading twice. The dedicated implementation exists to make two specific things possible: racing safely against asynchronous I/O, and fusing lock release with unpinning.

The old world: an LWLock per buffer

Before the change, BufferDesc had a lock in it:

 c
 
 
	/*
	 * Backend of pin-count waiter. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	int			wait_backend_pgprocno;

	PgAioWaitRef io_wref;		/* set iff AIO is in progress */
	LWLock		content_lock;	/* to lock access to buffer contents */
} BufferDesc;

and the accessor was a one-liner:

 c
 
 
static inline LWLock *
BufferDescriptorGetContentLock(const BufferDesc *bdesc)
{
	return (LWLock *) (&bdesc->content_lock);
}

An LWLock is a 16-byte structure with a uint32 state, a pg_atomic_uint32 nwaiters, a proclist_head waiters, and a bool is_shared. Taking one is a CAS on the state word; if that fails, you get pushed onto the wait queue in the same structure. Releasing it is another CAS plus, possibly, a walk of the wait queue.

The problem is not the algorithm — the LWLock algorithm is good. The problem is that there are now two independent atomic words per buffer: the LWLock’s state, and the BufferDesc.state word that holds the refcount and the flags. Anything that needs to reason about both at once has to do two separate atomic operations, with a window in between.

Concretely, the old code path for a shared content lock was:

 c
 
	LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);

and for an exclusive one:

 c
 
	LWLockAcquire(BufferDescriptorGetContentLock(desc), LW_EXCLUSIVE);

That is a full generic lock acquisition: read the lock’s state, decide whether it conflicts, CAS, and — if it conflicts — walk into lwlock.c’s queueing machinery.

The new world: one 64-bit word

The change is commit fcb9c977aa5, “bufmgr: Implement buffer content locks independently of lwlocks”, by Andres Freund, January 2026. The commit message lays out the four reasons:

Hint bits are currently set while holding only a share lock. This leads to having to copy pages while they are being written out if checksums are enabled, which is not cheap. We would like to add AIO writes, however once many buffers can be written out at the same time, it gets a lot more expensive to copy them, particularly because that copy needs to reside in shared buffers (for worker mode to have access to the buffer).

In addition, modifying buffers while they are being written out can cause issues with unbuffered/direct-IO, as some filesystems (like btrfs) do not like that, due to filesystem internal checksums getting corrupted.

The solution to this is to require a new share-exclusive lock-level to set hint bits and to write out buffers, making those operations mutually exclusive. We could introduce such a lock-level into the generic lwlock implementation, however it does not look like there would be other users, and it does add some overhead into important code paths.

For AIO writes we need to be able to race-freely check whether a buffer is undergoing IO and whether an exclusive lock on the page can be acquired. That is rather hard to do efficiently when the buffer state and the lock state are separate atomic variables. This is a major hindrance to allowing writes to be done asynchronously.

Buffer locks are by far the most frequently taken locks. Optimizing them specifically for their use case is worth the effort. E.g. by merging content locks into buffer locks we will be able to release a buffer lock and pin in one atomic operation.

There are more complicated optimizations, like long-lived “super pinned & locked” pages, that cannot realistically be implemented with the generic lwlock implementation.

Point 3 is the headline: buffer content locks are, by a wide margin, the most frequently taken lock in the system. Every ReadBuffer() pins and locks; every index scan does it once per page; every page inspection does it again.

The layout

The state word was already 64 bits wide before this change, but only half of it was used. The old header comment read:

 c
 
 
/*
 * Buffer state is a single 64-bit variable where following data is combined.
 *
 * State of the buffer itself (in order):
 * - 18 bits refcount
 * - 4 bits usage count
 * - 10 bits of flags
 *
 * Combining these values allows to perform some operations without locking
 * the buffer header, by modifying them together with a CAS loop.
 *
 * NB: A future commit will use a significant portion of the remaining bits to
 * implement buffer locking as part of the state variable.
 *
 * The definition of buffer state components is below.
 */
#define BUF_REFCOUNT_BITS 18
#define BUF_USAGECOUNT_BITS 4
#define BUF_FLAG_BITS 10

StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32,
				 "parts of buffer state space need to equal 32");

The new one (src/include/storage/buf_internals.h):

 c
 
 
/*
 * Buffer state is a single 64-bit variable where following data is combined.
 *
 * State of the buffer itself (in order):
 * - 18 bits refcount
 * - 4 bits usage count
 * - 12 bits of flags
 * - 18 bits share-lock count
 * - 1 bit share-exclusive locked
 * - 1 bit exclusive locked
 *
 * Combining these values allows to perform some operations without locking
 * the buffer header, by modifying them together with a CAS loop.
 *
 * The definition of buffer state components is below.
 */
#define BUF_REFCOUNT_BITS 18
#define BUF_USAGECOUNT_BITS 4
#define BUF_FLAG_BITS 12
#define BUF_LOCK_BITS (18+2)

StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS + BUF_LOCK_BITS <= 64,
				 "parts of buffer state space need to be <= 64");

So the word went from 32 bits used to 54 bits used, and the invariant changed from “exactly 32” to “at most 64”. The two extra flag bits are the two new lock-related flags; the 20 new lock bits are the lock state.

Here is the whole layout, on a 64-bit machine, with MAX_BACKENDS_BITS == 18:

Bits	Width	Field	Meaning
0–17	18	refcount	number of pins on this buffer
18–21	4	usage count	clock-sweep second-chance counter
22–33	12	flags	`BM_*` bits, see below
34–51	18	share-lock count	number of backends holding a share lock
52	1	share-exclusive	one backend holds a share-exclusive lock
53	1	exclusive	one backend holds an exclusive lock
54–63	10	unused

The shift constants:

 c
 
 
/* refcount related definitions */
#define BUF_REFCOUNT_ONE 1
#define BUF_REFCOUNT_MASK \
	((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1)

/* usage count related definitions */
#define BUF_USAGECOUNT_SHIFT \
	BUF_REFCOUNT_BITS
#define BUF_USAGECOUNT_MASK \
	(((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT))
#define BUF_USAGECOUNT_ONE \
	(UINT64CONST(1) << BUF_REFCOUNT_BITS)

/* flags related definitions */
#define BUF_FLAG_SHIFT \
	(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)
#define BUF_FLAG_MASK \
	(((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT)

and the lock bits:

 c
 
 
/* lock state related definitions */
#define BM_LOCK_SHIFT \
	(BUF_FLAG_SHIFT + BUF_FLAG_BITS)
#define BM_LOCK_VAL_SHARED \
	(UINT64CONST(1) << (BM_LOCK_SHIFT))
#define BM_LOCK_VAL_SHARE_EXCLUSIVE \
	(UINT64CONST(1) << (BM_LOCK_SHIFT + MAX_BACKENDS_BITS))
#define BM_LOCK_VAL_EXCLUSIVE \
	(UINT64CONST(1) << (BM_LOCK_SHIFT + MAX_BACKENDS_BITS + 1))
#define BM_LOCK_MASK \
	((((uint64) MAX_BACKENDS) << BM_LOCK_SHIFT) | BM_LOCK_VAL_SHARE_EXCLUSIVE | BM_LOCK_VAL_EXCLUSIVE)

MAX_BACKENDS_BITS comes from src/include/storage/procnumber.h:

 c
 
 
#define MAX_BACKENDS_BITS		18
#define MAX_BACKENDS			((1U << MAX_BACKENDS_BITS)-1)

Eighteen bits, which is exactly the number of lock bits available for the share-lock count. That is not a coincidence: it means the share-lock count can never exceed the number of possible backends, and the BM_LOCK_MASK expression (((uint64) MAX_BACKENDS) << BM_LOCK_SHIFT) covers exactly bits 34–51 — the whole share-lock field.

Two static assertions tie this together:

 c
 
 
StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS,
				 "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS");
StaticAssertDecl(MAX_BACKENDS_BITS <= (BUF_LOCK_BITS - 2),
				 "MAX_BACKENDS_BITS needs to be <= BUF_LOCK_BITS - 2");

The subtle part is that the “share lock” field is not a single bit — it is a count, stored in the low bits of the lock field. BM_LOCK_VAL_SHARED is the value 1 << 34, and acquiring a share lock adds it to the state word:

 c
 
 
		else
		{
			lock_free = (old_state & BM_LOCK_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARED;
		}

So desired_state += BM_LOCK_VAL_SHARED is really desired_state += 1 in the count field. The BM_LOCK_MASK then detects “is anybody holding any share lock” by testing all 18 count bits at once.

This is why the exclusive acquisition test is a mask against the whole lock field rather than a single bit:

 c
 
 
		if (mode == BUFFER_LOCK_EXCLUSIVE)
		{
			lock_free = (old_state & BM_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_EXCLUSIVE;
		}

BM_LOCK_MASK covers the share-lock count and both exclusive bits, so “the lock is free” means “no share lockers, no share-exclusive holder, no exclusive holder”.

The flag bits

The flags live in the middle of the word, and two of them are new:

 c
 
 
/* buffer header is locked */
#define BM_LOCKED					BUF_DEFINE_FLAG( 0)
/* data needs writing */
#define BM_DIRTY					BUF_DEFINE_FLAG( 1)
/* data is valid */
#define BM_VALID					BUF_DEFINE_FLAG( 2)
/* tag is assigned */
#define BM_TAG_VALID				BUF_DEFINE_FLAG( 3)
/* read or write in progress */
#define BM_IO_IN_PROGRESS			BUF_DEFINE_FLAG( 4)
/* previous I/O failed */
#define BM_IO_ERROR					BUF_DEFINE_FLAG( 5)
/* flag bit 6 is not used anymore */
/* have waiter for sole pin */
#define BM_PIN_COUNT_WAITER			BUF_DEFINE_FLAG( 7)
/* must write for checkpoint */
#define BM_CHECKPOINT_NEEDED		BUF_DEFINE_FLAG( 8)
/* permanent buffer (not unlogged, or init fork) */
#define BM_PERMANENT				BUF_DEFINE_FLAG( 9)
/* content lock has waiters */
#define BM_LOCK_HAS_WAITERS			BUF_DEFINE_FLAG(10)
/* waiter for content lock has been signalled but not yet run */
#define BM_LOCK_WAKE_IN_PROGRESS	BUF_DEFINE_FLAG(11)

BM_LOCK_HAS_WAITERS and BM_LOCK_WAKE_IN_PROGRESS are the content-lock equivalents of LW_FLAG_HAS_WAITERS and LW_FLAG_WAKE_IN_PROGRESS. They are in the flag field rather than the lock field because they are advisory hints, not part of the lock’s conflict state — they must not affect whether a lock can be acquired.

The descriptor

The BufferDesc itself changed shape. The embedded LWLock content_lock is gone; in its place is a wait list:

 c
 
 
typedef struct BufferDesc
{
	/*
	 * ID of page contained in buffer. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	BufferTag	tag;

	/*
	 * Buffer's index number (from 0). The field never changes after
	 * initialization, so does not need locking.
	 */
	int			buf_id;

	/*
	 * State of the buffer, containing flags, refcount and usagecount. See
	 * BUF_* and BM_* defines at the top of this file.
	 */
	pg_atomic_uint64 state;

	/*
	 * Backend of pin-count waiter. The buffer header spinlock needs to be
	 * held to modify this field.
	 */
	int			wait_backend_pgprocno;

	PgAioWaitRef io_wref;		/* set iff AIO is in progress */

	/*
	 * List of PGPROCs waiting for the buffer content lock. Protected by the
	 * buffer header spinlock.
	 */
	proclist_head lock_waiters;
} BufferDesc;

Note the proclist_head lock_waiters — a per-buffer intrusive wait list, analogous to the one inside an LWLock, but only 8 bytes instead of a whole LWLock’s worth of state.

Acquiring a content lock

The heart of it is BufferLockAttempt() in src/backend/storage/buffer/bufmgr.c. This is the analogue of LWLockAttemptLock():

 c
 
 
static inline bool
BufferLockAttempt(BufferDesc *buf_hdr, BufferLockMode mode)
{
	uint64		old_state;

	/*
	 * Read once outside the loop, later iterations will get the newer value
	 * via compare & exchange.
	 */
	old_state = pg_atomic_read_u64(&buf_hdr->state);

	/* loop until we've determined whether we could acquire the lock or not */
	while (true)
	{
		uint64		desired_state;
		bool		lock_free;

		desired_state = old_state;

		if (mode == BUFFER_LOCK_EXCLUSIVE)
		{
			lock_free = (old_state & BM_LOCK_MASK) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_EXCLUSIVE;
		}
		else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			lock_free = (old_state & (BM_LOCK_VAL_EXCLUSIVE | BM_LOCK_VAL_SHARE_EXCLUSIVE)) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARE_EXCLUSIVE;
		}
		else
		{
			lock_free = (old_state & BM_LOCK_VAL_EXCLUSIVE) == 0;
			if (lock_free)
				desired_state += BM_LOCK_VAL_SHARED;
		}

		/*
		 * Attempt to swap in the state we are expecting. If we didn't see
		 * lock to be free, that's just the old value. If we saw it as free,
		 * we'll attempt to mark it acquired. The reason that we always swap
		 * in the value is that this doubles as a memory barrier. We could try
		 * to be smarter and only swap in values if we saw the lock as free,
		 * but benchmark haven't shown it as beneficial so far.
		 *
		 * Retry if the value changed since we last looked at it.
		 */
		if (likely(pg_atomic_compare_exchange_u64(&buf_hdr->state,
												 &old_state, desired_state)))
		{
			if (lock_free)
			{
				/* Great! Got the lock. */
				return false;
			}
			else
				return true;	/* somebody else has the lock */
		}
	}

	pg_unreachable();
}

Read the three conflict rules off that:

Mode	Acquired if	Adds
`BUFFER_LOCK_SHARE`	no exclusive holder	`+1` to the share count
`BUFFER_LOCK_SHARE_EXCLUSIVE`	no exclusive and no share-exclusive holder	sets bit 52
`BUFFER_LOCK_EXCLUSIVE`	nothing held at all	sets bit 53

That is exactly the semantics the commit message describes: “An additional lock-level share-exclusive has been added. This lock-level conflicts with exclusive locks and itself, but not share locks.”

Two things are worth noting about the implementation.

First, the CAS always happens, even when the lock is known to be held. The comment explains why: the CAS doubles as a memory barrier, and the benchmark showed that conditionally skipping it did not help.

Second, the conflict test is a read of a shared word, not a read of a private lock structure. Because the lock state lives in the same word as the refcount and the flags, a concurrent pin or unpin is visible to the lock decision for free.

The queue-self-then-retry algorithm

If the first attempt fails, we cannot simply go to sleep: the lock might have been released in the meantime, and we would sleep on a lock nobody is going to signal. The classic solution — and the one lwlock.c uses — is to enqueue yourself and then try again. BufferLockAcquire() does exactly that:

 c
 
 
static inline void
BufferLockAcquire(Buffer buffer, BufferDesc *buf_hdr, BufferLockMode mode)
{
	PrivateRefCountEntry *entry;
	int			extraWaits = 0;

	/*
	 * Get reference to the refcount entry before we hold the lock, it seems
	 * better to do before holding the lock.
	 */
	entry = GetPrivateRefCountEntry(buffer, true);

	/*
	 * We better not already hold a lock on the buffer.
	 */
	Assert(entry->data.lockmode == BUFFER_LOCK_UNLOCK);

	/*
	 * Lock out cancel/die interrupts until we exit the code section protected
	 * by the content lock. This ensures that interrupts will not interfere
	 * with manipulations of data structures in shared memory.
	 */
	HOLD_INTERRUPTS();

	for (;;)
	{
		uint32		wait_event = 0; /* initialized to avoid compiler warning */
		bool		mustwait;

		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = BufferLockAttempt(buf_hdr, mode);

		if (likely(!mustwait))
		{
			break;
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		BufferLockQueueSelf(buf_hdr, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = BufferLockAttempt(buf_hdr, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			BufferLockDequeueSelf(buf_hdr);
			break;
		}

		switch (mode)
		{
			case BUFFER_LOCK_EXCLUSIVE:
				wait_event = WAIT_EVENT_BUFFER_EXCLUSIVE;
				break;
			case BUFFER_LOCK_SHARE_EXCLUSIVE:
				wait_event = WAIT_EVENT_BUFFER_SHARE_EXCLUSIVE;
				break;
			case BUFFER_LOCK_SHARE:
				wait_event = WAIT_EVENT_BUFFER_SHARED;
				break;
			case BUFFER_LOCK_UNLOCK:
				pg_unreachable();

		}
		pgstat_report_wait_start(wait_event);

		/*
		 * Wait until awakened.
		 *
		 * It is possible that we get awakened for a reason other than being
		 * signaled by BufferLockWakeup(). If so, loop back and wait again.
		 * Once we've gotten the lock, re-increment the sema by the number of
		 * additional signals received.
		 */
		for (;;)
		{
			PGSemaphoreLock(MyProc->sem);
			if (MyProc->lwWaiting == LW_WS_NOT_WAITING)
				break;
			extraWaits++;
		}

		pgstat_report_wait_end();

		/* Retrying, allow BufferLockReleaseSub to release waiters again. */
		pg_atomic_fetch_and_u64(&buf_hdr->state, ~BM_LOCK_WAKE_IN_PROGRESS);
	}

	/* Remember that we now hold this lock */
	entry->data.lockmode = mode;

	/*
	 * Fix the process wait semaphore's count for any absorbed wakeups.
	 */
	while (unlikely(extraWaits-- > 0))
		PGSemaphoreUnlock(MyProc->sem);
}

Compare this to LWLockAcquire() in src/backend/storage/lmgr/lwlock.c — the structure is identical, down to the comments:

 c
 
 
		/*
		 * Try to grab the lock the first time, we're not in the waitqueue
		 * yet/anymore.
		 */
		mustwait = LWLockAttemptLock(lock, mode);

		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "immediately acquired lock");
			break;				/* got the lock */
		}

		/*
		 * Ok, at this point we couldn't grab the lock on the first try. We
		 * cannot simply queue ourselves to the end of the list and wait to be
		 * woken up because by now the lock could long have been released.
		 * Instead add us to the queue and try to grab the lock again. If we
		 * succeed we need to revert the queuing and be happy, otherwise we
		 * recheck the lock. If we still couldn't grab it, we know that the
		 * other locker will see our queue entries when releasing since they
		 * existed before we checked for the lock.
		 */

		/* add to the queue */
		LWLockQueueSelf(lock, mode);

		/* we're now guaranteed to be woken up if necessary */
		mustwait = LWLockAttemptLock(lock, mode);

		/* ok, grabbed the lock the second time round, need to undo queueing */
		if (!mustwait)
		{
			LOG_LWDEBUG("LWLockAcquire", lock, "acquired, undoing queue");

			LWLockDequeueSelf(lock);
			break;
		}

The commit message is frank about this: “This commit unfortunately introduces some code that is very similar to the code in lwlock.c, however the code is not equivalent enough to easily merge it.”

One wart the commit message calls out explicitly: the new code still uses PGPROC->lwWaiting and PGPROC->lwWaitMode, rather than inventing a parallel set of fields:

 c
 
 
	/*
	 * These are currently used both for lwlocks and buffer content locks,
	 * which is acceptable, although not pretty, because a backend can't wait
	 * for both types of locks at the same time.
	 */
	MyProc->lwWaiting = LW_WS_WAITING;
	MyProc->lwWaitMode = mode;

	proclist_push_tail(&buf_hdr->lock_waiters, MyProcNumber, lwWaitLink);

A backend can only be blocked on one lock at a time, so reusing the fields is safe — but the fields are now lying about their name.

The public API

The lock modes are an enum in src/include/storage/bufmgr.h:

 c
 
 
/*
 * Buffer content lock modes (mode argument for LockBuffer())
 */
typedef enum BufferLockMode
{
	BUFFER_LOCK_UNLOCK,

	/*
	 * A share lock conflicts with exclusive locks.
	 */
	BUFFER_LOCK_SHARE,

	/*
	 * A share-exclusive lock conflicts with itself and exclusive locks.
	 */
	BUFFER_LOCK_SHARE_EXCLUSIVE,

	/*
	 * An exclusive lock conflicts with every other lock type.
	 */
	BUFFER_LOCK_EXCLUSIVE,
} BufferLockMode;

and there is a wrapper that keeps the mode argument a compile-time constant at the call site, because a runtime switch generated worse code:

 c
 
 
/*
 * Handling BUFFER_LOCK_UNLOCK in bufmgr.c leads to sufficiently worse branch
 * prediction to impact performance. Therefore handle that switch here, where
 * most of the time `mode` will be a constant and thus can be optimized out by
 * the compiler.
 */
static inline void
LockBuffer(Buffer buffer, BufferLockMode mode)
{
	if (mode == BUFFER_LOCK_UNLOCK)
		UnlockBuffer(buffer);
	else
		LockBufferInternal(buffer, mode);
}

LockBufferInternal() does the same trick one level down:

 c
 
 
	/*
	 * Test the most frequent lock modes first. While a switch (mode) would be
	 * nice, at least gcc generates considerably worse code for it.
	 *
	 * Call BufferLockAcquire() with a constant argument for mode, to generate
	 * more efficient code for the different lock modes.
	 */
	if (mode == BUFFER_LOCK_SHARE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_SHARE);
	else if (mode == BUFFER_LOCK_EXCLUSIVE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_EXCLUSIVE);
	else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		BufferLockAcquire(buffer, buf_hdr, BUFFER_LOCK_SHARE_EXCLUSIVE);
	else
		elog(ERROR, "unrecognized buffer lock mode: %d", mode);

Because BufferLockAcquire() is static inline and is called with a constant mode three different times, the compiler generates three specialised copies, each with the conflict test folded to a constant. A share lock’s conflict test becomes “is the exclusive bit set?” and nothing else.

Lock ownership tracking

Where lwlock.c keeps a global array of held locks per backend, the buffer content lock reuses the buffer manager’s existing private refcount hash:

 c
 
 
/*
 * This is separated out from PrivateRefCountEntry to allow for copying all
 * the data members via struct assignment.
 */
typedef struct PrivateRefCountData
{
	/*
	 * How many times has the buffer been pinned by this backend.
	 */
	int32		refcount;

	/*
	 * Is the buffer locked by this backend? BUFFER_LOCK_UNLOCK indicates that
	 * the buffer is not locked.
	 */
	BufferLockMode lockmode;
} PrivateRefCountData;

This is elegant: one hash lookup answers both “how many pins does this backend hold on this buffer?” and “in what mode is it locked?”. And it means error recovery needs no new machinery — the resource owner’s buffer cleanup already releases the content locks, so there is no LWLockReleaseAll() equivalent to write.

Releasing a content lock

The release path is split into three pieces, so that the “subtract the lock from the state word” step can be fused with other atomic operations:

 c
 
 
/*
 * Release a previously acquired buffer content lock.
 */
static void
BufferLockUnlock(Buffer buffer, BufferDesc *buf_hdr)
{
	BufferLockMode mode;
	uint64		oldstate;
	uint64		sub;

	mode = BufferLockDisownInternal(buffer, buf_hdr);

	/*
	 * Release my hold on lock, after that it can immediately be acquired by
	 * others, even if we still have to wakeup other waiters.
	 */
	sub = BufferLockReleaseSub(mode);

	oldstate = pg_atomic_sub_fetch_u64(&buf_hdr->state, sub);

	BufferLockProcessRelease(buf_hdr, mode, oldstate);

	/*
	 * Now okay to allow cancel/die interrupts.
	 */
	RESUME_INTERRUPTS();
}

BufferLockReleaseSub() computes what to subtract:

 c
 
 
static inline uint64
BufferLockReleaseSub(BufferLockMode mode)
{
	/*
	 * Turns out that a switch() leads to gcc to generate sufficiently worse code
	 * for this to show up in profiles...
	 */
	if (mode == BUFFER_LOCK_EXCLUSIVE)
		return BM_LOCK_VAL_EXCLUSIVE;
	else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		return BM_LOCK_VAL_SHARE_EXCLUSIVE;
	else
	{
		Assert(mode == BUFFER_LOCK_SHARE);
		return BM_LOCK_VAL_SHARED;
	}

	return 0;					/* keep compiler quiet */
}

Note that releasing a share lock subtracts BM_LOCK_VAL_SHARED — the count goes down, which is why it is a counter and not a bit.

The wakeup decision:

 c
 
 
static void
BufferLockProcessRelease(BufferDesc *buf_hdr, BufferLockMode mode, uint64 lockstate)
{
	bool		check_waiters = false;
	bool		wake_exclusive = false;

	/* nobody else can have that kind of lock */
	Assert(!(lockstate & BM_LOCK_VAL_EXCLUSIVE));

	/*
	 * If we're still waiting for backends to get scheduled, don't wake them
	 * up again. Otherwise check if we need to look through the waitqueue to
	 * wake other backends.
	 */
	if ((lockstate & BM_LOCK_HAS_WAITERS) &&
		!(lockstate & BM_LOCK_WAKE_IN_PROGRESS))
	{
		if ((lockstate & BM_LOCK_MASK) == 0)
		{
			/*
			 * We released a lock and the lock was, in that moment, free. We
			 * therefore can wake waiters for any kind of lock.
			 */
			check_waiters = true;
			wake_exclusive = true;
		}
		else if (mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			/*
			 * We released the lock, but another backend still holds a lock. We
			 * can't have released an exclusive lock, as there couldn't
			 * have been other lock holders. If we released a share lock, no
			 * waiters need to be woken up, as there must be other share
			 * lockers. However, if we held a share-exclusive lock, another
			 * backend now could acquire a share-exclusive lock.
			 */
			check_waiters = true;
			wake_exclusive = false;
		}
	}

	/*
	 * As waking up waiters requires the spinlock to be acquired, only do so
	 * if necessary.
	 */
	if (check_waiters)
		BufferLockWakeup(buf_hdr, wake_exclusive);
}

The three cases:

The lock is now completely free (lockstate & BM_LOCK_MASK == 0). Wake everybody — share, share-exclusive and exclusive waiters.
We released a share lock and others still hold locks. Nothing to do: there must still be other share lockers, so a share waiter can proceed.
We released a share-exclusive lock and others still hold locks. A share-exclusive waiter can now proceed (share-exclusive does not conflict with share), but an exclusive waiter cannot.

BufferLockWakeup() then walks the per-buffer wait list. This is where the three-level conflict matrix shows up:

 c
 
 
	proclist_foreach_modify(iter, &buf_hdr->lock_waiters, lwWaitLink)
	{
		PGPROC	 *waiter = GetPGProcByNumber(iter.cur);

		/*
		 * Already woke up a conflicting lock, so skip over this wait list
		 * entry.
		 */
		if (!wake_exclusive && waiter->lwWaitMode == BUFFER_LOCK_EXCLUSIVE)
			continue;
		if (!wake_share_exclusive && waiter->lwWaitMode == BUFFER_LOCK_SHARE_EXCLUSIVE)
			continue;

		proclist_delete(&buf_hdr->lock_waiters, iter.cur, lwWaitLink);
		proclist_push_tail(&wakeup, iter.cur, lwWaitLink);

		/*
		 * Prevent additional wakeups until retryer gets to run. Backends that
		 * are just waiting for the lock to become free don't retry
		 * automatically.
		 */
		new_wake_in_progress = true;

		/*
		 * Signal that the process isn't on the wait list anymore. This allows
		 * BufferLockDequeueSelf() to remove itself from the waitlist with a
		 * proclist_delete(), rather than having to check if it has been
		 * removed from the list.
		 */
		Assert(waiter->lwWaiting == LW_WS_WAITING);
		waiter->lwWaiting = LW_WS_PENDING_WAKEUP;

		/*
		 * Don't wakeup further waiters after waking a conflicting waiter.
		 */
		if (waiter->lwWaitMode == BUFFER_LOCK_SHARE)
		{
			/*
			 * Share locks conflict with exclusive locks.
			 */
			wake_exclusive = false;
		}
		else if (waiter->lwWaitMode == BUFFER_LOCK_SHARE_EXCLUSIVE)
		{
			/*
			 * Share-exclusive locks conflict with share-exclusive and
			 * exclusive locks.
			 */
			wake_exclusive = false;
			wake_share_exclusive = false;
		}
		else if (waiter->lwWaitMode == BUFFER_LOCK_EXCLUSIVE)
		{
			/*
			 * Exclusive locks conflict with all other locks, there's no point
			 * in waking up anybody else.
			 */
			break;
		}
	}

The classic FIFO-ish behaviour: wake share waiters, and stop at the first exclusive waiter, because everybody behind it conflicts with it anyway.

The wait list itself is protected by the buffer header spinlock, which is what BufferLockQueueSelf() does:

 c
 
 
static void
BufferLockQueueSelf(BufferDesc *buf_hdr, BufferLockMode mode)
{
	/*
	 * If we don't have a PGPROC structure, there's no way to wait. This
	 * should never occur, since MyProc should only be null during shared
	 * memory initialization.
	 */
	if (MyProc == NULL)
		elog(PANIC, "cannot wait without a PGPROC structure");

	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

	LockBufHdr(buf_hdr);

	/* setting the flag is protected by the spinlock */
	pg_atomic_fetch_or_u64(&buf_hdr->state, BM_LOCK_HAS_WAITERS);
	...
	proclist_push_tail(&buf_hdr->lock_waiters, MyProcNumber, lwWaitLink);

	/* Can release the mutex now */
	UnlockBufHdr(buf_hdr);
}

The header lock is not an LWLock either

A pleasant side effect of the change is that the buffer header lock is now also not a real spinlock with a slock_t. It is a bit in the state word, and LockBufHdr() is a single fetch_or:

 c
 
 
/*
 * Lock buffer header - set BM_LOCKED in buffer state.
 */
uint64
LockBufHdr(BufferDesc *desc)
{
	uint64		old_buf_state;

	Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc)));

	while (true)
	{
		/*
		 * Always try once to acquire the lock directly, without setting up
		 * the spin-delay infrastructure. The work necessary for that shows up
		 * in profiles and is rarely necessary.
		 */
		old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED);
		if (likely(!(old_buf_state & BM_LOCKED)))
			break;				/* got lock */

		/* and then spin without atomic operations until lock is released */
		{
			SpinDelayStatus delayStatus;

			init_local_spin_delay(&delayStatus);

			while (old_buf_state & BM_LOCKED)
			{
				perform_spin_delay(&delayStatus);
				old_buf_state = pg_atomic_read_u64(&desc->state);
			}
			finish_spin_delay(&delayStatus);
		}

		/*
		 * Retry. The lock might obviously already be re-acquired by the time
		 * we're attempting to get it again.
		 */
	}

	return old_buf_state | BM_LOCKED;
}

That comment — “Always try once to acquire the lock directly, without setting up the spin-delay infrastructure. The work necessary for that shows up in profiles and is rarely necessary.” — is a micro-optimisation with a real history behind it. Setting up SpinDelayStatus costs enough that it showed up in profiles when the lock was uncontended (which it almost always is).

The uncontended path is now: one lock or on a cache line that is already hot, one branch. The contended path spins on a plain load, no atomic instructions at all, until the bit clears.

Releasing is symmetric and equally cheap:

 c
 
 
/*
 * Unlock the buffer header.
 *
 * This can only be used if the caller did not modify BufferDesc.state. To
 * set/unset flag bits or change the refcount use UnlockBufHdrExt().
 */
static inline void
UnlockBufHdr(BufferDesc *desc)
{
	Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED);

	pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED);
}

And the more general version, which sets flags and adjusts the refcount in the same CAS — this is the “release a buffer lock and pin in one atomic operation” that the commit message promised, in its simpler form:

 c
 
 
/*
 * Unlock the buffer header, while atomically unsetting the ones in
 * unset_bits, adding the flags in set_bits, and changing the refcount by
 * refcount_change. If a flag is both cleared and added, it will end up being
 * set.
 *
 * Note that this approach would not trivially work for usagecount, since we
 * need to cap the usagecount at BM_MAX_USAGE_COUNT.
 */
static inline uint64
UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state,
				uint64 set_bits, uint64 unset_bits,
				int refcount_change)
{
	for (;;)
	{
		uint64		buf_state = old_buf_state;

		Assert(buf_state & BM_LOCKED);

		/*
		 * Set bits after clearing bits, so that a cleared and re-added flag
		 * survives.
		 */
		buf_state &= ~unset_bits;
		buf_state |= set_bits;
		buf_state &= ~BM_LOCKED;

		if (refcount_change != 0)
			buf_state += BUF_REFCOUNT_ONE * refcount_change;

		if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state,
										 buf_state))
		{
			return old_buf_state;
		}
	}
}

Note the comment about the usage count: it cannot be handled this way, because the usage count is capped at BM_MAX_USAGE_COUNT and a blind addition would overflow the cap.

The counterpart — waiting for the header lock to clear — is a pg_noinline spin loop:

 c
 
 
/*
 * Wait until the BM_LOCKED flag isn't set anymore and return the buffer's
 * state at that point.
 *
 * Obviously the buffer could be locked by the time the value is returned, so
 * this is primarily useful in CAS style loops.
 */
pg_noinline uint64
WaitBufHdrUnlocked(BufferDesc *buf)
{
	SpinDelayStatus delayStatus;
	uint64		buf_state;

	init_local_spin_delay(&delayStatus);

	buf_state = pg_atomic_read_u64(&buf->state);

	while (buf_state & BM_LOCKED)
	{
		perform_spin_delay(&delayStatus);
		buf_state = pg_atomic_read_u64(&buf->state);
	}

	finish_spin_delay(&delayStatus);

	return buf_state;
}

pg_noinline is deliberate: this is the slow path, and inlining it into every caller would bloat the instruction cache for no benefit.

The reason the third lock level exists at all is a correctness issue with hint bits.

PostgreSQL caches visibility facts (Xmin committed, Xmax invalid, …) in the tuple’s t_infomask. Those are called hint bits. Setting them modifies the page. If the page is being written out at the same time, the page checksum computed over the bytes that go to disk does not match the bytes that were in memory when the checksum was computed — so the buffer manager had to copy the page before writing it, which is expensive.

With a third lock level, hint-bit setting and write-out can be made mutually exclusive. Both need share-exclusive, and share-exclusive conflicts with itself.

Writing the buffer takes share-exclusive:

 c
 
 
/*
 * Convenience wrapper around FlushBuffer() that locks/unlocks the buffer
 * before/after calling FlushBuffer().
 */
static void
FlushUnlockedBuffer(BufferDesc *buf, SMgrRelation reln,
					IOObject io_object, IOContext io_context)
{
	Buffer		buffer = BufferDescriptorGetBuffer(buf);

	BufferLockAcquire(buffer, buf, BUFFER_LOCK_SHARE_EXCLUSIVE);
	FlushBuffer(buf, reln, io_object, io_context);
	BufferLockUnlock(buffer, buf);
}

Setting hint bits takes share-exclusive too — but if the backend only holds a share lock, it has to upgrade first, and the upgrade can fail:

 c
 
 
/*
 * Helper for BufferBeginSetHintBits() and BufferSetHintBits16().
 *
 * This checks if the current lock mode already suffices to allow hint bits
 * being set and, if not, whether the current lock can be upgraded.
 *
 * Updates *lockstate when returning true.
 */
static inline bool
SharedBufferBeginSetHintBits(Buffer buffer, BufferDesc *buf_hdr, uint64 *lockstate)
{
	uint64		old_state;
	PrivateRefCountEntry *ref;
	BufferLockMode mode;

	ref = GetPrivateRefCountEntry(buffer, true);

	if (ref == NULL)
		elog(ERROR, "buffer is not pinned");

	mode = ref->data.lockmode;
	if (mode == BUFFER_LOCK_UNLOCK)
		elog(ERROR, "buffer is not locked");

	/* we're done if we are already holding a sufficient lock level */
	if (mode == BUFFER_LOCK_EXCLUSIVE || mode == BUFFER_LOCK_SHARE_EXCLUSIVE)
	{
		*lockstate = pg_atomic_read_u64(&buf_hdr->state);
		return true;
	}

	/*
	 * We are only holding a share lock right now, try to upgrade it to
	 * SHARE_EXCLUSIVE.
	 */
	Assert(mode == BUFFER_LOCK_SHARE);

	old_state = pg_atomic_read_u64(&buf_hdr->state);
	while (true)
	{
		uint64		desired_state;

		desired_state = old_state;

		/*
		 * Can't upgrade if somebody else holds the lock in exclusive or
		 * share-exclusive mode.
		 */
		if (unlikely((old_state & (BM_LOCK_VAL_EXCLUSIVE | BM_LOCK_VAL_SHARE_EXCLUSIVE)) != 0))
		{
			return false;
		}

		/* currently held lock state */
		desired_state -= BM_LOCK_VAL_SHARED;

		/* new lock level */
		desired_state += BM_LOCK_VAL_SHARE_EXCLUSIVE;

		if (likely(pg_atomic_compare_exchange_u64(&buf_hdr->state,
												 &old_state, desired_state)))
		{
			ref->data.lockmode = BUFFER_LOCK_SHARE_EXCLUSIVE;
			*lockstate = desired_state;

			return true;
		}
	}
}

Note that the upgrade is a non-blocking CAS. If it fails — because somebody else holds share-exclusive or exclusive — the backend simply does not set the hint bits this time round. That is fine: hint bits are an optimisation, and the whole point is to avoid blocking a reader on a writer.

The comment on BufferBeginSetHintBits() spells out the trade-off:

 c
 
 
/*
 * Try to acquire the right to set hint bits on the buffer.
 *
 * To be allowed to set hint bits, this backend needs to hold either a
 * share-exclusive or an exclusive lock. In case this backend only holds a
 * share lock, this function will try to upgrade the lock to
 * share-exclusive. The caller is only allowed to set hint bits if true is
 * returned.
 *
 * Once BufferBeginSetHintBits() has returned true, hint bits may be set
 * without further calls to BufferBeginSetHintBits(), until the buffer is
 * unlocked.
 *
 *
 * Requiring a share-exclusive lock to set hint bits prevents setting hint
 * bits on buffers that are currently being written out, which could corrupt
 * the checksum on the page. Flushing buffers also requires a share-exclusive
 * lock.
 *
 * Due to a lock >= share-exclusive being required to set hint bits, only one
 * backend can set hint bits at a time. Allowing multiple backends to set
 * hint bits would require more complicated locking: For setting hint bits
 * we'd need to store the count of backends currently setting hint bits, for
 * I/O we would need another lock-level conflicting with the hint-setting
 * lock-level. Given that the share-exclusive lock for setting hint bits is
 * only held for a short time, that backends often would just set the same
 * hint bits and that the cost of occasionally not setting hint bits in hotly
 * accessed pages is fairly low, this seems like an acceptable tradeoff.
 */

The “only one backend at a time” is a direct consequence of the share-lock field being 18 bits wide but the share-exclusive level being a single bit. There is room in the word for a second share-exclusive counter; there just is not one.

The lock-free clock sweep

The other place the new atomicity pays off is the buffer replacement strategy. src/backend/storage/buffer/freelist.c implements the clock sweep, and the clock hand is a plain atomic counter:

 c
 
 
typedef struct
{
	/* Spinlock: protects the values below */
	slock_t		buffer_strategy_lock;

	/*
	 * clock-sweep hand: index of next buffer to consider grabbing. Note that
	 * this isn't a concrete buffer - we only ever increase the value. So, to
	 * get an actual buffer, it needs to be used modulo NBuffers.
	 */
	pg_atomic_uint32 nextVictimBuffer;

	/*
	 * Statistics. These counters should be wide enough that they can't
	 * overflow during a single bgwriter cycle.
	 */
	uint32		completePasses; /* Complete cycles of the clock-sweep */
	pg_atomic_uint32 numBufferAllocs;	/* Buffers allocated since last reset */

	/*
	 * Bgworker process to be notified upon activity or -1 if none. See
	 * StrategyNotifyBgWriter.
	 */
	int			bgwprocno;
} BufferStrategyControl;

The hand moves with a single fetch_add:

 c
 
 
static inline uint32
ClockSweepTick(void)
{
	uint32		victim;

	/*
	 * Atomically move hand ahead one buffer - if there's several processes
	 * doing this, this can lead to buffers being returned slightly out of
	 * apparent order.
	 */
	victim =
		pg_atomic_fetch_add_u32(&StrategyControl->nextVictimBuffer, 1);

	if (victim >= NBuffers)
	{
		uint32		originalVictim = victim;

		/* always wrap what we look up in BufferDescriptors */
		victim = victim % NBuffers;

		/*
		 * If we're the one that just caused a wraparound, force
		 * completePasses to be incremented while holding the spinlock. We
		 * need the spinlock so StrategySyncStart() can return a consistent
		 * value consisting of nextVictimBuffer and completePasses.
		 */
		if (victim == 0)
		{
			uint32		expected;
			uint32		wrapped;
			bool		success = false;

			expected = originalVictim + 1;

			while (!success)
			{
				/*
				 * Acquire the spinlock while increasing completePasses. That
				 * allows other readers to read nextVictimBuffer and
				 * completePasses in a consistent manner which is required for
				 * StrategySyncStart(). In theory delaying the increment
				 * could lead to an overflow of nextVictimBuffers, but that's
				 * highly unlikely and wouldn't be particularly harmful.
				 */
				SpinLockAcquire(&StrategyControl->buffer_strategy_lock);

				wrapped = expected % NBuffers;

				success = pg_atomic_compare_exchange_u32(&StrategyControl->nextVictimBuffer,
														 &expected, wrapped);
				if (success)
					StrategyControl->completePasses++;
				SpinLockRelease(&StrategyControl->buffer_strategy_lock);
			}
		}
	}
	return victim;
}

There is exactly one spinlock left in the whole file, and it is held only for the wraparound bookkeeping. Advancing the hand is a single atomic add — no lock, ever.

The interesting part is StrategyGetBuffer(), which selects a victim and pins it without ever taking the buffer header spinlock:

 c
 
 
	/* Use the "clock sweep" algorithm to find a free buffer */
	trycounter = NBuffers;
	for (;;)
	{
		uint64		old_buf_state;
		uint64		local_buf_state;

		buf = GetBufferDescriptor(ClockSweepTick());

		/*
		 * Check whether the buffer can be used and pin it if so. Do this
		 * using a CAS loop, to avoid having to lock the buffer header.
		 */
		old_buf_state = pg_atomic_read_u64(&buf->state);
		for (;;)
		{
			local_buf_state = old_buf_state;

			/*
			 * If the buffer is pinned or has a nonzero usage_count, we cannot
			 * use it; decrement the usage_count (unless pinned) and keep
			 * scanning.
			 */

			if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0)
			{
				if (--trycounter == 0)
				{
					/*
					 * We've scanned all the buffers without making any state
					 * changes, so all the buffers are pinned (or were when we
					 * looked at them). We could hope that someone will free
					 * one eventually, but it's probably better to fail than
					 * to risk getting stuck in an infinite loop.
					 */
					elog(ERROR, "no unpinned buffers available");
				}
				break;
			}

			/* See equivalent code in PinBuffer() */
			if (unlikely(local_buf_state & BM_LOCKED))
			{
				old_buf_state = WaitBufHdrUnlocked(buf);
				continue;
			}

			if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0)
			{
				local_buf_state -= BUF_USAGECOUNT_ONE;

				if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
												 local_buf_state))
				{
					trycounter = NBuffers;
					break;
				}
			}
			else
			{
				/* pin the buffer if the CAS succeeds */
				local_buf_state += BUF_REFCOUNT_ONE;

				if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state,
												 local_buf_state))
				{
					/* Found a usable buffer */
					if (strategy != NULL)
						AddBufferToRing(strategy, buf);
					*buf_state = local_buf_state;

					TrackNewBufferPin(BufferDescriptorGetBuffer(buf));

					return buf;
				}
			}
		}
	}

Read the two CAS cases:

Usage count is non-zero. The buffer is not pinned, but it has been used recently, so it gets a second chance: decrement the usage count and move on. The CAS is needed because the usage count and the refcount live in the same word, and we must not clobber a concurrent refcount change.
Usage count is zero and refcount is zero. The buffer is free. Add one to the refcount — that is the pin — and take it.

The pin and the “is this buffer still free?” decision are the same CAS. Compare that with the old code, which had to take the buffer header spinlock to increment the refcount and then release it. Here, the only time the header lock is involved at all is when BM_LOCKED happens to be set, in which case the code waits for it to clear and retries:

 c
 
 
			/* See equivalent code in PinBuffer() */
			if (unlikely(local_buf_state & BM_LOCKED))
			{
				old_buf_state = WaitBufHdrUnlocked(buf);
				continue;
			}

The second chance, quantified

The usage count is four bits wide, capped at 5:

 c
 
 
/*
 * The maximum allowed value of usage_count represents a tradeoff between
 * accuracy and speed of the clock-sweep buffer management algorithm. A
 * large value (comparable to NBuffers) would approximate LRU semantics.
 * But it can take as many as BM_MAX_USAGE_COUNT+1 complete cycles of the
 * clock-sweep hand to find a free buffer, so in practice we don't want the
 * value to be very large.
 */
#define BM_MAX_USAGE_COUNT	5

StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS),
				 "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits");

Five is a compromise, and the comment says so: a large value would approximate LRU, but the clock hand might need BM_MAX_USAGE_COUNT + 1 full passes over the buffer array to find a free buffer. With shared_buffers = 128 GB and 8 kB pages that is 16 million buffers, and a value of 5 means at most 6 passes.

The value is bumped on pin, capped at the maximum:

 c
 
 
			/* increase refcount */
			buf_state += BUF_REFCOUNT_ONE;

			if (strategy == NULL)
			{
				/* Default case: increase usagecount unless already max. */
				if (BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
					buf_state += BUF_USAGECOUNT_ONE;
			}

This is CLOCK in the textbook sense: the hand gives each buffer a bounded number of second chances, which makes the eviction policy an approximation of LRU rather than a strict LRU.

Caveats

This is a big, invasive change

The commit touches the buffer manager, the buffer replacement strategy, the resource owner machinery, the wait-event taxonomy, and the index access methods. The diff is large, and the new code deliberately duplicates a meaningful chunk of lwlock.c.

The commit message is explicit about the cost:

This commit unfortunately introduces some code that is very similar to the code in lwlock.c, however the code is not equivalent enough to easily merge it. The future wins that this commit makes possible seem worth the cost.

And about the wart:

It’s worth calling out one wart in this commit: Despite content locks not being lwlocks anymore, they continue to use PGPROC->lw* - that seemed better than duplicating the relevant infrastructure.

The wait event taxonomy changed

Content locks are no longer reported as LWLock waits. There are three new wait events in the Buffer class:

text

BUFFER_SHARED	"Waiting to acquire a shared lock on a buffer."
BUFFER_SHARE_EXCLUSIVE	"Waiting to acquire a share exclusive lock on a buffer."
BUFFER_EXCLUSIVE	"Waiting to acquire an exclusive lock on a buffer."

The old BufferContent LWLock tranche is gone, and the BUFFERPIN wait event class was renamed to BUFFER in commit 6c5c393b740 to make room for them. Any monitoring that watched for LWLock waits on the buffer content tranche needs updating.

A backend can only hold one lock at a time

The implementation does not track multiple lock ownerships of the same buffer within one backend. BufferLockConditional() documents the consequence:

 c
 
 
/*
 * Acquire the content lock for the buffer, but only if we don't have to wait.
 *
 * It is allowed to try to conditionally acquire a lock on a buffer that this
 * backend has already locked, but the lock acquisition will always fail, even
 * if the new lock acquisition does not conflict with an already held lock
 * (e.g. two share locks). This is because we currently do not have space to
 * track multiple lock ownerships of the same buffer within one backend. That
 * is ok for the current uses of BufferLockConditional().
 */

So a backend that already holds a share lock on a buffer cannot conditionally upgrade it; BufferLockConditional() returns false. The non-conditional upgrade path (SharedBufferBeginSetHintBits()) does its own CAS and works fine.

The 18-bit share-lock field holds at most 262 143 concurrent share lockers. That is exactly MAX_BACKENDS, so it cannot overflow in practice — but it also means the design assumes at most one share lock per backend per buffer, which is what the private refcount hash enforces.

Deadlock semantics are essentially unchanged

The queueing algorithm is the LWLock algorithm, so the deadlock properties are the same: the wait list is FIFO-ish, exclusive waiters block later waiters, and there is no deadlock detector — PostgreSQL relies on the lock ordering conventions of the code that takes the locks.

The one structural difference is that content locks and LWLocks are now different lock types. A backend that holds a buffer content lock and then tries to take an LWLock (or vice versa) could in principle deadlock in a way the old design could not, since the old content lock was an LWLock and the global lock-ordering discipline applied uniformly. The commit’s answer to this is the PGPROC->lwWaiting check:

 c
 
 
	if (MyProc->lwWaiting != LW_WS_NOT_WAITING)
		elog(PANIC, "queueing for lock while waiting on another one");

A backend cannot block on a buffer content lock while already blocked on something else — that case is a PANIC, not a deadlock. Which is a stronger guarantee than a deadlock detector, but only because it is enforced by the PGPROC state.

Platforms without 64-bit atomics

The whole design rests on pg_atomic_uint64. On platforms that lack native 64-bit atomic support, PostgreSQL emulates them with a spinlock (src/include/port/atomics/fallback.h):

 c
 
 
#if !defined(PG_HAVE_ATOMIC_U64_SUPPORT)

#define PG_HAVE_ATOMIC_U64_SIMULATION

#define PG_HAVE_ATOMIC_U64_SUPPORT
typedef struct pg_atomic_uint64
{
	int			sema;
	volatile uint64 value;
} pg_atomic_uint64;

#define PG_HAVE_ATOMIC_INIT_U64
extern void pg_atomic_init_u64_impl(volatile pg_atomic_uint64 *ptr, uint64 val_);

#define PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U64
extern bool pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
												uint64 *expected, uint64 newval);

#define PG_HAVE_ATOMIC_FETCH_ADD_U64
extern uint64 pg_atomic_fetch_add_u64_impl(volatile pg_atomic_uint64 *ptr, int64 add_);

#endif /* PG_HAVE_ATOMIC_U64_SUPPORT */

and the implementation in src/backend/port/atomics.c:

 c
 
 
bool
pg_atomic_compare_exchange_u64_impl(volatile pg_atomic_uint64 *ptr,
									uint64 *expected, uint64 newval)
{
	bool		ret;

	/*
	 * Do atomic op under a spinlock. It might look like we could just skip
	 * the cmpxchg if the lock isn't available, but that'd just emulate a
	 * 'weak' compare and swap. I.e. one that allows spurious failures. Since
	 * several algorithms rely on a strong variant and that is efficiently
	 * implementable on most major architectures let's emulate it here as
	 * well.
	 */
	SpinLockAcquire((slock_t *) &ptr->sema);

	/* perform compare/exchange logic */
	ret = ptr->value == *expected;
	*expected = ptr->value;
	if (ret)
		ptr->value = newval;

	/* and release lock */
	SpinLockRelease((slock_t *) &ptr->sema);

	return ret;
}

The other operations are built on top of the CAS, in src/include/port/atomics/generic.h:

 c
 
 
#if !defined(PG_HAVE_ATOMIC_FETCH_OR_U64) && defined(PG_HAVE_ATOMIC_COMPARE_EXCHANGE_U64)
#define PG_HAVE_ATOMIC_FETCH_OR_U64
static inline uint64
pg_atomic_fetch_or_u64_impl(volatile pg_atomic_uint64 *ptr, uint64 or_)
{
	uint64 old;
	old = ptr->value;			/* ok if read is not atomic */
	while (!pg_atomic_compare_exchange_u64_impl(ptr, &old, old | or_))
		/* skip */;
	return old;
}
#endif

So on such a platform, LockBufHdr() — a single fetch_or — becomes a spinlock acquire and a CAS loop. The logical result is identical; the performance is not. The whole point of the change is to make the common path cheap, and on a fallback platform it is not cheap at all.

This matters more for the content lock than for the header lock. The header lock was already a BM_LOCKED bit in the state word before this change — commit dac328c8a68, “bufmgr: Change BufferDesc.state to be a 64-bit atomic”, had just widened that word from 32 to 64 bits — but it was a bit on 32-bit atomics, which have native support on every platform PostgreSQL builds on. The content lock, likewise, was an LWLock, whose own state word is a uint32. Moving both into a 64-bit word turns a native lock acquisition into a spinlock-guarded CAS on a platform without 64-bit atomics.

On every mainstream platform — x86-64, ARM64, POWER, s390x — 64-bit atomics are native and the fallback is not used.

The state word is now a contention point

Before the change, the refcount lived in the state word and the lock lived in the LWLock. Now they live together. That is the whole point — it lets the lock decision see the refcount — but it also means that unrelated operations on the same buffer now touch the same cache line. A buffer that is being pinned and unpinned rapidly is a buffer whose state word is hot.

The BufferDesc is padded to a cache line to keep it from sharing with its neighbours:

 c
 
 
/*
 * Concurrent access to buffer headers has proven to be more efficient if
 * they're cache line aligned. So we force the start of the BufferDescriptors
 * array to be on a cache line boundary and force the elements to be cache
 * line sized.
 *
 * XXX: As this is primarily matters in highly concurrent workloads which
 * probably all are 64bit these days, and the space wastage would be a bit
 * more noticeable on 32bit systems, we don't force the stride to be cache
 * line sized on those. If somebody does actual performance testing, we can
 * reevaluate.
 *
 * Note that local buffer descriptors aren't forced to be aligned - as there's
 * no concurrent access to those it's unlikely to be beneficial.
 *
 * We use a 64-byte cache line size here, because that's the most common
 * size. Making it bigger would be a waste of memory. Even if running on a
 * platform with either 32 or 128 byte line sizes, it's good to align to
 * boundaries and avoid false sharing.
 */
#define BUFFERDESC_PAD_TO_SIZE	(SIZEOF_VOID_P == 8 ? 64 : 1)

typedef union BufferDescPadded
{
	BufferDesc	bufferdesc;
	char		pad[BUFFERDESC_PAD_TO_SIZE];
} BufferDescPadded;

Note the XXX comment: on 32-bit systems the padding is disabled, because the waste would be noticeable. The header comment in the struct is explicit about the constraint:

Be careful to avoid increasing the size of the struct when adding or reordering members. Keeping it below 64 bytes (the most common CPU cache line size) is fairly important for performance.

Removing the embedded LWLock (16 bytes) and replacing it with an 8-byte proclist_head is what keeps the struct within the budget.

Local buffers are a different story

Local buffers (temp tables) use a completely separate mechanism, and the content lock is a no-op for them:

 c
 
 
	Assert(BufferIsPinned(buffer));
	if (BufferIsLocal(buffer))
		return;					/* local buffers need no lock */

Because a local buffer belongs to exactly one backend, there is nothing to lock against.

What it all means in practice

The buffer content lock is now the cheapest lock in PostgreSQL. In the uncontended case it is a single compare-and-swap on a word that is already in the cache. There is no lock structure to walk, no separate state word to coherency-check, no LWLock array to index.

The lock state and the pin state are now the same word. This is the structural change. It means:

Deciding whether a buffer is free to evict and pinning it are one atomic operation (StrategyGetBuffer()).
Checking whether I/O is in progress and taking an exclusive lock are atomic with respect to each other — which is what makes asynchronous writes possible.
Releasing a lock and dropping a pin can be fused into one CAS.

Hint-bit setting is now correct with respect to write-out. Before, a reader could set a hint bit on a page while the checkpointer was writing that same page out, forcing an expensive page copy. Now the two are mutually exclusive via the share-exclusive level, and the copy can go away. That matters more as PostgreSQL moves towards AIO, where many buffers are in flight at once.

The lock-free clock sweep is the quiet win. StrategyGetBuffer() no longer takes the buffer header spinlock to pin a victim. On a busy system with a large shared_buffers, the clock hand’s path through the buffer array no longer serialises on a lock.

Watch the wait events. If you have dashboards that count LWLock waits on BufferContent, they are now dead. The replacements are Buffer/Shared, Buffer/ShareExclusive and Buffer/Exclusive.

PostgreSQL Computes Page Checksums Fast

Jay Hubbert — Sat, 05 Sep 2026 09:00:00 +0000

Every page PostgreSQL writes to disk gets a checksum: two bytes in the page header that let the server notice, on the next read, that the page it just read is not the page it wrote.

That sounds trivial. It is not, and the reason is a nice illustration of how a deceptively simple requirement turns into a performance problem. The checksum is computed over 8 kB of memory, on every page write and every page read, and the whole point of the feature is that it must be cheap enough to always leave on. A checksum that costs five percent of your I/O is not a checksum; it is a tax you eventually turn off.

So PostgreSQL does something quite clever: it computes thirty-two checksums at once, over a thirty-two-column view of the page, so that the latency of the one expensive instruction in the algorithm — the multiplication — is hidden behind the others. Thirty-two is not an arbitrary number. It is the number that fits in the register file.

This is the story of that design, in version 20devel (commit e75fcc9867f).

Where the checksum lives

Two bytes, in the page header:

 c
 
 
// src/include/storage/bufpage.h
typedef struct PageHeaderData
{
	/* XXX LSN is member of *any* block, not only page-organized ones */
	PageXLogRecPtr pd_lsn;		/* LSN: next byte after last byte of xlog
								 * record for last change to this page */
	uint16		pd_checksum;	/* checksum */
	uint16		pd_flags;		/* flag bits, see below */
	LocationIndex pd_lower;		/* offset to start of free space */
	LocationIndex pd_upper;		/* offset to end of free space */
	LocationIndex pd_special;	/* offset to start of special space */
	uint16		pd_pagesize_version;
	TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */
	ItemIdData	pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */
} PageHeaderData;

The header comment above the struct explains the semantics, and they are subtler than they look:

pd_checksum stores the page checksum, if it has been set for this page; zero is a valid value for a checksum. If a checksum is not in use then we leave the field unset. This will typically mean the field is zero though non-zero values may also be present if databases have been pg_upgraded from releases prior to 9.3, when the same byte offset was used to store the current timelineid when the page was last updated. Note that there is no indication on a page as to whether the checksum is valid or not, a deliberate design choice which avoids the problem of relying on the page contents to decide whether to verify it. Hence there are no flag bits relating to checksums.

That last sentence is the design decision that makes the whole feature tractable: a page does not say “I have a checksum”. Whether a checksum is expected is a property of the cluster, not of the page. That means the verification code never has to guess, and it means enabling checksums on an existing cluster is a cluster-wide decision recorded in pg_control.

The states live in src/include/storage/checksum.h:

 c
 
 
// src/include/storage/checksum.h
/*
 * Checksum state 0 is used for when data checksums are disabled (OFF).
 * PG_DATA_CHECKSUM_INPROGRESS_{ON|OFF} defines that data checksums are either
 * currently being enabled or disabled, and PG_DATA_CHECKSUM_VERSION defines
 * that data checksums are enabled. The ChecksumStateType is stored in
 * pg_control so changing requires a catversion bump, and the values cannot
 * be reordered. New states must be added at the end.
 */
typedef enum ChecksumStateType
{
	PG_DATA_CHECKSUM_OFF = 0,
	PG_DATA_CHECKSUM_VERSION = 1,
	PG_DATA_CHECKSUM_INPROGRESS_OFF = 2,
	PG_DATA_CHECKSUM_INPROGRESS_ON = 3,
} ChecksumStateType;

Note the two “in progress” states. pg_checksums can be run offline to turn checksums on or off for a stopped cluster, but turning them on while the server is running has to be WAL-logged and crash-safe, so there is a state where checksums are written but not yet verified, and a state where they are verified but not yet written. Both matter for the “is a checksum expected here?” question.

The two runtime predicates are in src/backend/access/transam/xlog.c:

 c
 
 
// src/backend/access/transam/xlog.c
/*
 * DataChecksumsNeedWrite
 *		Returns whether data checksums must be written or not
 *
 * Returns true if data checksums are enabled, or are in the process of being
 * enabled. During "inprogress-on" and "inprogress-off" states checksums must
 * be written even though they are not verified (see datachecksum_state.c for
 * a longer discussion).
 *
 * This function is intended for callsites which are about to write a data page
 * to storage, and need to know whether to re-calculate the checksum for the
 * page header. Calling this function must be performed as close to the write
 * operation as possible to keep the critical section short.
 */
bool
DataChecksumsNeedWrite(void)
{
	return (LocalDataChecksumState == PG_DATA_CHECKSUM_VERSION ||
			LocalDataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_ON ||
			LocalDataChecksumState == PG_DATA_CHECKSUM_INPROGRESS_OFF);
}

 c
 
 
// src/backend/access/transam/xlog.c
/*
 * DataChecksumsNeedVerify
 *		Returns whether data checksums must be verified or not
 *
 * Data checksums are only verified if they are fully enabled in the cluster.
 * During the "inprogress-on" and "inprogress-off" states they are only
 * updated, not verified (see datachecksum_state.c for a longer discussion).
 *
 * This function is intended for callsites which have read data and are about
 * to perform checksum validation based on the result of this. Calling this
 * function must be performed as close to the validation call as possible to
 * keep the critical section short. This is in order to protect against time of
 * check/time of use situations around data checksum validation.
 */
bool
DataChecksumsNeedVerify(void)
{
	return (LocalDataChecksumState == PG_DATA_CHECKSUM_VERSION);
}

Note the asymmetry: DataChecksumsNeedWrite() is true in three states (the “on” state and both “in progress” states), while DataChecksumsNeedVerify() is true only in the fully-enabled state. You must not verify a checksum that was never written.

The algorithm: FNV-1a, with a twist

The core of the algorithm is documented in one of the best comments in the PostgreSQL tree. From src/include/storage/checksum_impl.h:

 c
 
 
// src/include/storage/checksum_impl.h
/*
 * The algorithm used to checksum pages is chosen for very fast calculation.
 * Workloads where the database working set fits into OS file cache but not
 * into shared buffers can read in pages at a very fast pace and the checksum
 * algorithm itself can become the largest bottleneck.
 *
 * The checksum algorithm itself is based on the FNV-1a hash (FNV is shorthand
 * for Fowler/Noll/Vo). The primitive of a plain FNV-1a hash folds in data 1
 * byte at a time according to the formula:
 *
 *	 hash = (hash ^ value) * FNV_PRIME
 *
 * FNV-1a algorithm is described at http://www.isthe.com/chongo/tech/comp/fnv/
 *
 * PostgreSQL doesn't use FNV-1a hash directly because it has bad mixing of
 * high bits - high order bits in input data only affect high order bits in
 * output data. To resolve this we xor in the value prior to multiplication
 * shifted right by 17 bits. The number 17 was chosen because it doesn't
 * have common denominator with set bit positions in FNV_PRIME and empirically
 * provides the fastest mixing for high order bits of final iterations quickly
 * avalanche into lower positions. For performance reasons we choose to combine
 * 4 bytes at a time. The actual hash formula used as the basis is:
 *
 *	 hash = (hash ^ value) * FNV_PRIME ^ ((hash ^ value) >> 17)
 *
 * The main bottleneck in this calculation is the multiplication latency. To
 * hide the latency and to make use of SIMD parallelism multiple hash values
 * are calculated in parallel. The page is treated as a 32 column two
 * dimensional array of 32 bit values. Each column is aggregated separately
 * into a partial checksum. Each partial checksum uses a different initial
 * value (offset basis in FNV terminology). The initial values actually used
 * were chosen randomly, as the values themselves don't matter as much as that
 * they are different and don't match anything in real data. After initializing
 * partial checksums each value in the column is aggregated according to the
 * above formula. Finally two more iterations of the formula are performed with
 * value 0 to mix the bits of the last value added.
 *
 * The partial checksums are then folded together using xor to form a single
 * 32-bit checksum. The caller can safely reduce the value to 16 bits
 * using modulo 2^16-1. That will cause a very slight bias towards lower
 * values but this is not significant for the performance of the
 * checksum.

The base is FNV-1a: a one-byte-at-a-time hash with the formula

text

 hash = (hash ^ value) * FNV_PRIME

PostgreSQL does not use it unmodified, for the reason given: plain FNV-1a mixes badly — high-order bits of the input only affect high-order bits of the output, which is exactly the kind of weakness a checksum cannot afford when the input is a page full of mostly-zero bytes.

The fix is to fold in the value shifted as well:

text

 hash = (hash ^ value) * FNV_PRIME ^ ((hash ^ value) >> 17)

The shift by 17 is not arbitrary; the comment explains that 17 “doesn’t have common denominator with set bit positions in FNV_PRIME and empirically provides the fastest mixing for high order bits”.

And then the crucial decision:

For performance reasons we choose to combine 4 bytes at a time.

So the unit of work is a 32-bit word, not a byte. The formula is implemented as a macro:

 c
 
 
// src/include/storage/checksum_impl.h
/*
 * Calculate one round of the checksum.
 */
#define CHECKSUM_COMP(checksum, value) \
do { \
	uint32 __tmp = (checksum) ^ (value); \
	(checksum) = __tmp * FNV_PRIME ^ (__tmp >> 17); \
} while (0)

__tmp is the xor of the running hash and the new word. The hash becomes __tmp * FNV_PRIME, xored with __tmp >> 17. Note that the xor with the shifted value happens after the multiplication — the shift of the pre-multiplication value, which is the same as __tmp >> 17 because the multiplication is applied to __tmp in the first term only.

Why thirty-two hashes

Here is the paragraph that explains the whole design:

The main bottleneck in this calculation is the multiplication latency. To hide the latency and to make use of SIMD parallelism multiple hash values are calculated in parallel. The page is treated as a 32 column two dimensional array of 32 bit values. Each column is aggregated separately into a partial checksum. Each partial checksum uses a different initial value (offset basis in FNV terminology).

An integer multiply on a modern x86 has a latency of about three cycles and a throughput of one per cycle. The critical path in the checksum loop is: multiply (3 cycles) → xor → multiply → xor … If you have only one hash in flight, you are latency-bound and the multiply unit is idle most of the time. If you have several independent hashes in flight, they fill the gap: the CPU issues all the multiplies, then all the xors, and the latency of one is covered by the throughput of the others.

So the page is viewed as a matrix, and the columns are the parallel hashes:

text

 page bytes, viewed as uint32 words

 col 0: w[0] w[32] w[64] ... w[2016] ──▶ hash 0
 col 1: w[1] w[33] w[65] ... w[2017] ──▶ hash 1
 col 2: w[2] w[34] w[66] ... w[2018] ──▶ hash 2
 ...
 col 31: w[31] w[63] w[95] ... w[2047] ──▶ hash 31

Each column is a contiguous run of words in the page, strided by 32 words. The 32 hashes march through their columns in lockstep, one word per row, and each row’s 32 multiplies are independent.

The implementation of that view is a union:

 c
 
 
// src/include/storage/checksum_impl.h
#include "storage/bufpage.h"

/* number of checksums to calculate in parallel */
#define N_SUMS 32
/* prime multiplier of FNV-1a hash */
#define FNV_PRIME 16777619

/* Use a union so that this code is valid under strict aliasing */
typedef union
{
	PageHeaderData phdr;
	uint32		data[BLCKSZ / (sizeof(uint32) * N_SUMS)][N_SUMS];
} PGChecksummablePage;

/*
 * Base offsets to initialize each of the parallel FNV hashes into a
 * different initial state.
 */
static const uint32 checksumBaseOffsets[N_SUMS] = {
	0x5B1F36E9, 0xB8525960, 0x02AB50AA, 0x1DE66D2A,
	0x79FF467A, 0x9BB9F8A3, 0x217E7CD2, 0x83E13D2C,
	0xF8D4474F, 0xE39EB970, 0x42C6AE16, 0x993216FA,
	0x7B093B5D, 0x98DAFF3C, 0xF718902A, 0x0B1C9CDB,
	0xE58F764B, 0x187636BC, 0x5D7B3BB1, 0xE73DE7DE,
	0x92BEC979, 0xCCA6C0B2, 0x304A0979, 0x85AA43D4,
	0x783125BB, 0x6CA8EAA2, 0xE407EAC6, 0x4B5CFC3E,
	0x9FBF8C76, 0x15CA20BE, 0xF2CA9FD3, 0x959BD756
};

Two things are hiding in there.

The union is not incidental — it is a strict-aliasing workaround. The comment says so: “Use a union so that this code is valid under strict aliasing”. The page is a char *; the code wants to read it as an array of uint32. If you did that with a cast and a pointer, the compiler would be entitled to assume that the uint32 accesses never alias the char accesses and could reorder or elide them — technically undefined behavior, and the kind that produces a checksum that is correct on one compiler and wrong on another. Putting the two views in a union is the standard C way of saying “these are the same bytes, and I know what I’m doing”. (PostgreSQL also compiles with -fno-strict-aliasing in some configurations, but the union is the portable answer, and it is what the code does.)

The array shape is the algorithm. data[BLCKSZ / (sizeof(uint32) * N_SUMS)][N_SUMS] is a 64-by-32 array of uint32 for an 8 kB page: 8192 / (4 * 32) = 64 rows, 32 columns. The first index is the row (which step along a column) and the second is the column (which hash). That is why page->data[i][j] in the main loop means “row i, column j” and why the inner loop over j is the vectorizable one.

The initial values for the 32 hashes are constants:

 c
 
 
// src/include/storage/checksum_impl.h
/*
 * Base offsets to initialize each of the parallel FNV hashes into a
 * different initial state.
 */
static const uint32 checksumBaseOffsets[N_SUMS] = {
	0x5B1F36E9, 0xB8525960, 0x02AB50AA, 0x1DE66D2A,
	0x79FF467A, 0x9BB9F8A3, 0x217E7CD2, 0x83E13D2C,
	0xF8D4474F, 0xE39EB970, 0x42C6AE16, 0x993216FA,
	0x7B093B5D, 0x98DAFF3C, 0xF718902A, 0x0B1C9CDB,
	0xE58F764B, 0x187636BC, 0x5D7B3BB1, 0xE73DE7DE,
	0x92BEC979, 0xCCA6C0B2, 0x304A0979, 0x85AA43D4,
	0x783125BB, 0x6CA8EAA2, 0xE407EAC6, 0x4B5CFC3E,
	0x9FBF8C76, 0x15CA20BE, 0xF2CA9FD3, 0x959BD756
};

and the comment above them says exactly what they are for:

Base offsets to initialize each of the parallel FNV hashes into a different initial state.

The header comment adds a note that is easy to miss and quite philosophical:

Each partial checksum uses a different initial value (offset basis in FNV terminology). The initial values actually used were chosen randomly, as the values themselves don’t matter as much as that they are different and don’t match anything in real data.

So the constants are not magic numbers with a story; they are just 32 different random 32-bit numbers, chosen so that no two hashes start in the same state.

The core loop

The actual algorithm lives in a separate header, checksum_block_internal.h, which is #included — as a body, not as a function — into whichever implementation is being compiled. That is the trick that lets the same algorithm be compiled four ways (see below) without any duplication.

 c
 
 
// src/include/storage/checksum_block_internal.h
/* there is deliberately not an #ifndef CHECKSUM_BLOCK_INTERNAL_H here */

uint32		sums[N_SUMS];
uint32		result = 0;
uint32		i,
			j;

/* ensure that the size is compatible with the algorithm */
Assert(sizeof(PGChecksummablePage) == BLCKSZ);

/* initialize partial checksums to their corresponding offsets */
memcpy(sums, checksumBaseOffsets, sizeof(checksumBaseOffsets));

/* main checksum calculation */
for (i = 0; i < (uint32) (BLCKSZ / (sizeof(uint32) * N_SUMS)); i++)
	for (j = 0; j < N_SUMS; j++)
		CHECKSUM_COMP(sums[j], page->data[i][j]);

/* finally add in two rounds of zeroes for additional mixing */
for (i = 0; i < 2; i++)
	for (j = 0; j < N_SUMS; j++)
		CHECKSUM_COMP(sums[j], 0);

/* xor fold partial checksums together */
for (i = 0; i < N_SUMS; i++)
	result ^= sums[i];

return result;

Read it as four phases.

Phase 1: initialize. The 32 partial checksums start at their base offsets.

Phase 2: the main loop. for i in 64 rows, for j in 32 columns: CHECKSUM_COMP(sums[j], page->data[i][j]). The inner loop over j is the vectorizable one: 32 independent multiply-xor chains, one per column. The compiler can put eight of them in a 256-bit AVX2 register (or four in a 128-bit SSE2 register) and issue a vector multiply.

Phase 3: two rounds of zeroes. CHECKSUM_COMP(sums[j], 0) twice. The header comment explains why:

Finally two more iterations of the formula are performed with value 0 to mix the bits of the last value added.

Without this, the last word added to each column would have a weaker effect on the final result than earlier words. Two extra rounds of the same mixing operation push the influence of the last real word further down the mix.

Phase 4: the fold. All 32 partial checksums are xored together into a single 32-bit value:

The partial checksums are then folded together using xor to form a single 32-bit checksum.

Note this is an XOR, not an addition. That matters: XOR is associative, commutative, and order-independent, so the fold is exact and there is no carry propagation between columns. It also means the fold is itself embarrassingly parallel — though at 32 elements, that hardly matters.

From 32 bits to 16

The fold produces a 32-bit value. The page header has sixteen bits. The final mapping is in pg_checksum_page():

 c
 
 
// src/include/storage/checksum_impl.h
/*
 * Compute the checksum for a Postgres page.
 *
 * The page must be adequately aligned (at least on a 4-byte boundary).
 * Beware also that the checksum field of the page is transiently zeroed.
 *
 * The checksum includes the block number (to detect the case where a page is
 * somehow moved to a different location), the page header (excluding the
 * checksum itself), and the page data.
 */
uint16
pg_checksum_page(char *page, BlockNumber blkno)
{
	PGChecksummablePage *cpage = (PGChecksummablePage *) page;
	uint16		save_checksum;
	uint32		checksum;

	/* We only calculate the checksum for properly-initialized pages */
	Assert(!PageIsNew((Page) page));

	/*
	 * Save pd_checksum and temporarily set it to zero, so that the checksum
	 * calculation isn't affected by the old checksum stored on the page.
	 * Restore it after, because actually updating the checksum is NOT part of
	 * the API of this function.
	 */
	save_checksum = cpage->phdr.pd_checksum;
	cpage->phdr.pd_checksum = 0;
	checksum = pg_checksum_block(cpage);
	cpage->phdr.pd_checksum = save_checksum;

	/* Mix in the block number to detect transposed pages */
	checksum ^= blkno;

	/*
	 * Reduce to a uint16 (to fit in the pd_checksum field) with an offset of
	 * one. That avoids checksums of zero, which seems like a good idea.
	 */
	return (uint16) ((checksum % 65535) + 1);
}

Several things are going on here.

The checksum is not self-referential. The header comment says it, and the code does it:

 c
 
 
	save_checksum = cpage->phdr.pd_checksum;
	cpage->phdr.pd_checksum = 0;
	checksum = pg_checksum_block(cpage);
	cpage->phdr.pd_checksum = save_checksum;

The old checksum is saved, the field is zeroed, the checksum is computed over the page including the (now zero) checksum field, and the old value is restored — because, as the comment notes, “actually updating the checksum is NOT part of the API of this function”. The caller does that.

Zeroing the field rather than excluding it from the hash is simpler, and it means the checksum covers the whole 8 kB without a special case for two bytes.

The block number is mixed in afterwards:

 c
 
 
	/* Mix in the block number to detect transposed pages */
	checksum ^= blkno;

A page copied to the wrong position in the file — a “transposed page” — would otherwise have a perfectly valid checksum, because its contents are intact. Xoring in the block number means such a page fails verification.

The reduction to sixteen bits avoids zero:

 c
 
 
	/*
	 * Reduce to a uint16 (to fit in the pd_checksum field) with an offset of
	 * one. That avoids checksums of zero, which seems like a good idea.
	 */
	return (uint16) ((checksum % 65535) + 1);

Modulo 65535 rather than 65536 keeps the result in 1..65535, so the stored value is never zero. And the header comment is upfront about the consequence:

The caller can safely reduce the value to 16 bits using modulo 2^16-1. That will cause a very slight bias towards lower values but this is not significant for the performance of the checksum.

The bias is real: 0xFFFF can never occur, and 0x0001 occurs slightly more often than its neighbors. For a corruption-detection checksum, that is irrelevant.

Where it is called from

Two places, and they are not symmetric.

Writing goes through PageSetChecksum():

 c
 
 
// src/backend/storage/page/bufpage.c
 * If the page is in shared buffers, it needs to be locked in at least
 * share-exclusive mode.
 *
 * If checksums are disabled, or if the page is not initialized, just
 * return. Otherwise compute and set the checksum.
 *
 * In the past this needed to be done on a copy of the page, due to the
 * possibility of e.g., hint bits being set concurrently. However, this is not
 * necessary anymore as hint bits won't be set while IO is going on.
 */
void
PageSetChecksum(Page page, BlockNumber blkno)
{
	HOLD_INTERRUPTS();
	/* If we don't need a checksum, just return */
	if (PageIsNew(page) || !DataChecksumsNeedWrite())
	{
		RESUME_INTERRUPTS();
		return;
	}

	((PageHeader) page)->pd_checksum = pg_checksum_page(page, blkno);
	RESUME_INTERRUPTS();
}

Note the HOLD_INTERRUPTS() / RESUME_INTERRUPTS() bracket, and note the comment above the function:

In the past this needed to be done on a copy of the page, due to the possibility of e.g., hint bits being set concurrently. However, this is not necessary anymore as hint bits won’t be set while IO is going on.

Reading goes through PageIsVerified():

 c
 
 
// src/backend/storage/page/bufpage.c
bool
PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_failure_p)
{
	const PageHeaderData *p = (const PageHeaderData *) page;
	size_t	 *pagebytes;
	bool		checksum_failure = false;
	bool		header_sane = false;
	uint16		checksum = 0;

	if (checksum_failure_p)
		*checksum_failure_p = false;

	/*
	 * Don't verify page data unless the page passes basic non-zero test
	 */
	if (!PageIsNew(page))
	{
		/*
		 * There shouldn't be any check for interrupt calls happening in this
		 * codepath, but just to be on the safe side we hold interrupts since
		 * if they did happen the data checksum state could change during
		 * verifying checksums, which could lead to incorrect verification
		 * results.
		 */
		HOLD_INTERRUPTS();
		if (DataChecksumsNeedVerify())
		{
			checksum = pg_checksum_page(page, blkno);

			if (checksum != p->pd_checksum)
			{
				checksum_failure = true;
				if (checksum_failure_p)
					*checksum_failure_p = true;
			}
		}
		RESUME_INTERRUPTS();

		/*
		 * The following checks don't prove the header is correct, only that
		 * it looks sane enough to allow into the buffer pool. Later usage of
		 * the block can still reveal problems, which is why we offer the
		 * checksum option.
		 */
		if ((p->pd_flags & ~PD_VALID_FLAG_BITS) == 0 &&
			p->pd_lower <= p->pd_upper &&
			p->pd_upper <= p->pd_special &&
			p->pd_special <= BLCKSZ &&
			p->pd_special == MAXALIGN(p->pd_special))
			header_sane = true;

		if (header_sane && !checksum_failure)
			return true;
	}

Note the order of the checks. The checksum is verified first, and then the header is checked for sanity — but the sanity check is explicitly not treated as proof:

 c
 
 
		/*
		 * The following checks don't prove the header is correct, only that
		 * it looks sane enough to allow into the buffer pool. Later usage of
		 * the block can still reveal problems, which is why we offer the
		 * checksum option.
		 */

And the all-zero page case is handled separately, because a zeroed page is a legitimate thing to find (a backend extended a relation and crashed before writing any WAL about it).

Runtime dispatch: the AVX2 path

Now the part that makes this a performance story rather than an algorithm story.

The checksum algorithm is a few dozen lines, but it is compiled into several implementations, and the right one is chosen at runtime.

 c
 
 
// src/backend/storage/page/checksum.c
#include "postgres.h"

#include "port/pg_cpu.h"
#include "storage/checksum.h"
/*
 * The actual code is in storage/checksum_impl.h. This is done so that
 * external programs can incorporate the checksum code by #include'ing
 * that file from the exported Postgres headers. (Compare our legacy
 * CRC code in pg_crc.h.)
 * The PG_CHECKSUM_INTERNAL symbol allows core to use hardware-specific
 * coding without affecting external programs.
 */
#define PG_CHECKSUM_INTERNAL
#include "storage/checksum_impl.h"	/* IWYU pragma: keep */


static uint32
pg_checksum_block_fallback(const PGChecksummablePage *page)
{
#include "storage/checksum_block_internal.h"
}

/*
 * AVX2-optimized block checksum algorithm.
 */
#ifdef USE_AVX2_WITH_RUNTIME_CHECK
pg_attribute_target("avx2")
static uint32
pg_checksum_block_avx2(const PGChecksummablePage *page)
{
#include "storage/checksum_block_internal.h"
}
#endif							/* USE_AVX2_WITH_RUNTIME_CHECK */

/*
 * Choose the best available checksum implementation.
 */
static uint32
pg_checksum_choose(const PGChecksummablePage *page)
{
	pg_checksum_block = pg_checksum_block_fallback;

#ifdef USE_AVX2_WITH_RUNTIME_CHECK
	if (x86_feature_available(PG_AVX2))
		pg_checksum_block = pg_checksum_block_avx2;
#endif

	return pg_checksum_block(page);
}

static uint32 (*pg_checksum_block) (const PGChecksummablePage *page) = pg_checksum_choose;

Four things are going on.

The function pointer. pg_checksum_block is a function pointer, initialized to pg_checksum_choose. The first call to pg_checksum_page goes through the pointer, and pg_checksum_choose replaces the pointer with the best available implementation before doing the work. Every subsequent call is a direct-ish indirect call to the chosen implementation. The AVX2 commit message explains the tradeoff:

Similar to other hardware-specific paths, we set a function pointer on first use. We don’t bother to avoid this on platforms without AVX2 since the overhead of indirect calls doesn’t matter for multi-kilobyte inputs.

That is the right call. The input is 8 kB; an indirect call is worth about a nanosecond. The AVX2 win is worth hundreds.

The same source, four ways. The pg_checksum_block_* functions do not contain the algorithm; they #include it:

 c
 
#include "storage/checksum_block_internal.h"

So the algorithm is textually included in the fallback and in the AVX2 variant. There is exactly one copy of the algorithm, and it is compiled twice with different compiler flags. That is the whole reason for the awkward “include a body” idiom.

The AVX2 variant uses a function attribute. pg_attribute_target("avx2") expands to __attribute__((target("avx2"))), which tells the compiler “compile this function for AVX2, even though the rest of the program was not compiled for AVX2”. The attribute is applied to a single function, so the rest of the binary remains runnable on a pre-AVX2 CPU. The macro itself:

 c
 
 
// src/include/c.h
 * pg_attribute_target allows specifying different target options that the
 * function should be compiled with (e.g., for using special CPU instructions).
 * Note that there still needs to be a configure-time check to verify that a
 * specific target is understood by the compiler.
 */
#if __has_attribute (target)
#define pg_attribute_target(...) __attribute__((target(__VA_ARGS__)))
#else
#define pg_attribute_target(...)
#endif

Note the comment’s warning: “Note that there still needs to be a configure-time check to verify that a specific target is understood by the compiler.” That is exactly what PGAC_AVX2_SUPPORT() in configure.ac is for.

Whether the compiler supports it is decided at configure time:

 c
 
 
// configure.ac
# Check for AVX2 target support
#
if test x"$host_cpu" = x"x86_64"; then
 PGAC_AVX2_SUPPORT()
 if test x"$pgac_avx2_support" = x"yes"; then
 AC_DEFINE(USE_AVX2_WITH_RUNTIME_CHECK, 1, [Define to 1 to use AVX2 instructions with a runtime check.])
 fi
fi

and the runtime check uses the CPUID-based feature detection in src/include/port/pg_cpu.h:

 c
 
 
// src/include/port/pg_cpu.h
#if defined(__x86_64__) || defined(__i386__)

typedef enum X86FeatureId
{
	/* Have we run feature detection? */
	INIT_PG_X86,

	/* scalar registers and 128-bit XMM registers */
	PG_SSE4_2,
	PG_POPCNT,

	/* 256-bit YMM registers */
	PG_AVX2,

	/* 512-bit ZMM registers */
	PG_AVX512_BW,
	PG_AVX512_VL,
	PG_AVX512_VPCLMULQDQ,
	PG_AVX512_VPOPCNTDQ,

	/* identification */
	PG_HYPERVISOR,

	/* Time-Stamp Counter (TSC) flags */
	PG_RDTSCP,
	PG_TSC_INVARIANT,

x86_feature_available(PG_AVX2) triggers set_x86_features() on first use and then reads a cached array.

The fallback relies on the compiler. For a build without AVX2 — or on a non-x86 platform — the fallback implementation is compiled with -funroll-loops -ftree-vectorize, applied to this one object file:

 make
 
 
# src/backend/storage/page/Makefile
# Provide special optimization flags for checksum.c
checksum.o: CFLAGS += ${CFLAGS_UNROLL_LOOPS} ${CFLAGS_VECTORIZE}

The header comment explains why those two flags are the whole trick:

Vectorization of the algorithm works best with a 32bit x 32bit -> 32bit vector integer multiplication instruction, Examples include x86 AVX2 extensions (vpmulld) and ARM NEON (vmul.i32). Without that, vectorization is still possible if the compiler can turn multiplication by FNV_PRIME into a sequence of vectorized shifts and adds. For simplicity we rely on the compiler to do the vectorization for us. For GCC and clang the flags -funroll-loops -ftree-vectorize are enough to achieve vectorization.

So: on AVX2 hardware you get real vector multiply (vpmulld), eight columns per instruction. Elsewhere you get the compiler’s best attempt at expressing the constant multiply x * 16777619 as shifts and adds, which is what SSE2 has to do — SSE2 has no 32-bit vector multiply at all.

Why thirty-two, exactly?

The header comment is explicit:

The parallelism number 32 was chosen based on the fact that it is the largest state that fits into architecturally visible x86 SSE registers while leaving some free registers for intermediate values. For processors with 256-bit vector registers this leaves some performance on the table.

So: 32 partial hashes = 32 × 32 bits = 1024 bits of state. In 128-bit XMM registers that is eight registers — half of x86-64’s sixteen XMM registers, leaving the other half free for the loop’s intermediate values. With AVX2’s 256-bit YMM registers you could hold twice as much state, but you cannot: the number 32 is baked into the algorithm.

And it cannot be unbaked:

The number chosen is a fixed part of the algorithm because changing the parallelism changes the checksum result.

Caveats

N_SUMS is part of the on-disk format

This is the big one. N_SUMS is 32, the base offsets are a fixed table, and the fold is a fixed XOR. The resulting 16-bit value is stored in pd_checksum on every page of every relation in the cluster. If you changed N_SUMS to 16, or reordered the base offsets, or changed the prime, every existing page in every existing cluster would fail verification. The checksum algorithm is not an implementation detail; it is a storage format.

This is why the algorithm lives in checksum_impl.h with the comment:

This file exists for the benefit of external programs that may wish to check Postgres page checksums. They can #include this to get the code referenced by storage/checksum.h.

External tools — pg_checksums, the pageinspect extension, backup and restore tooling, third-party extensions — need to compute the same value, so the algorithm is published as a header that anyone can include. The PG_CHECKSUM_INTERNAL symbol then splits it:

 c
 
 
// src/include/storage/checksum_impl.h
/*
 * Block checksum algorithm. The page must be adequately aligned
 * (at least on 4-byte boundary).
 */
#ifdef PG_CHECKSUM_INTERNAL
/* definitions in src/backend/storage/page/checksum.c */
static uint32 (*pg_checksum_block) (const PGChecksummablePage *page);

#else
/* static definition for external programs */
static uint32
pg_checksum_block(const PGChecksummablePage *page)
{
#include "storage/checksum_block_internal.h"
}

#endif

With PG_CHECKSUM_INTERNAL defined (as in src/backend/storage/page/checksum.c), the core gets the runtime-dispatched function pointer. Without it, an external program gets a plain static function and never has to know that dispatch exists.

What the checksum does and does not cover

It covers the whole page: the page header (with the checksum field zeroed), the line pointer array, the tuple data, and the special space. It does not cover the block number — the block number is mixed in afterwards, by XOR.

It cannot detect all corruption. Sixteen bits means a false-negative rate of roughly 1 in 65535 for a random corruption; the header comment quotes the measured figure:

Tests of 1-4 uncorrelated bit errors, stray 0 and 0xFF bytes, overwriting page from random position to end with 0 bytes, and overwriting random segments of page with 0x00, 0xFF and random data all show optimal 2e-16 false positive rate within margin of error.

(That “false positive rate” is the rate of undetected corruption — the terminology in the comment is from the perspective of “the checksum says OK when the data is bad”.)

And it is not a substitute for WAL. A page checksum detects that this page is corrupt; it cannot tell you what the page should have contained, and it cannot undo the damage. Recovery from corruption still requires a restore from backup or a pg_checksums-style surgical repair.

The cost is real

Every dirty page written to disk pays for the checksum, and every page read from disk pays for the verification. For a write-heavy workload on slow storage, that is measurable. That is why checksums are opt-in at initdb time (initdb --data-checksums), and why the pg_checksums utility exists to turn them on or off offline for an existing cluster.

The AVX2 commit message names the case where the cost hurts most:

This matters most when using io_uring since in that case the checksum computation is not done in parallel by IO workers.

Which is to say: the checksum is computed on the path to disk, and anything that adds work to that path shows up.

The “in progress” states are subtle

A cluster in the PG_DATA_CHECKSUM_INPROGRESS_ON state writes checksums but does not verify them; a cluster in PG_DATA_CHECKSUM_INPROGRESS_OFF state does neither. Getting this wrong means either failing to notice corruption or failing to read your own data, so the state machine is stored in pg_control and the transitions are WAL-logged.

Sixteen bits is not a lot

The stored value is a 16-bit checksum, and the reduction is not uniform: checksum % 65535 + 1 can produce 1..65535, so 0x0000 is impossible and the distribution is very slightly skewed. It is fine for detecting “this page is garbage”, and it is not fine for anything stronger. Do not confuse it with a cryptographic digest.

Look at it yourself

The pg_checksums utility is the supported way to inspect and repair checksums on a stopped cluster:

pg_checksums --check -D /var/lib/postgresql/16/main
pg_checksums --enable -D /var/lib/postgresql/16/main

pg_controldata shows the cluster’s checksum state:

pg_controldata /var/lib/postgresql/16/main | grep -i checksum

And to watch the algorithm run, look at the assembly of the fallback implementation — the pmulld/vpmulld in the AVX2 object file is the whole point:

objdump -d src/backend/storage/page/checksum.o | less

What it all means in practice

The checksum algorithm is a latency problem, not a throughput problem. The single expensive operation is a 32-bit multiply, and multiply latency — not bandwidth — is what limits a dependent chain of hashes. The fix is not a faster multiply; it is thirty-two independent chains so the CPU’s out-of-order engine can fill the pipeline. That is a general-purpose optimization that applies to any hash-like construction.

The page is treated as a matrix, and the matrix shape is the algorithm. Thirty-two columns, sixty-four rows. The inner loop walks across columns, and that is the loop that vectorizes. It is a staggered layout — column j reads words j, j+32, j+64 — so it is not contiguous, but it is perfectly regular, which is what the compiler and the vectorizer need.

Register pressure is a design constraint, not an afterthought. The number 32 came from “the largest state that fits into architecturally visible x86 SSE registers while leaving some free registers for intermediate values”. AVX2 could double it. It cannot, because the constant is part of the on-disk format. This is the general shape of optimizing a persistent format: the moment you define it, your implementation choices freeze.

Runtime dispatch with a function pointer is the pragmatic way to get AVX2 without breaking old CPUs. Compile one function with __attribute__((target("avx2"))), detect AVX2 with CPUID, and swap the pointer on first use. The overhead is one indirect call per 8 kB page, which is nothing.

The correctness argument rests on the compiler. The fallback implementation is not hand-written SIMD; it is the same C code with -funroll-loops -ftree-vectorize, trusting GCC or clang to vectorize the constant multiplication into shifts and adds. The result must be bit-identical to the AVX2 path, and it is, because both are the same source text.

PostgreSQL Hashes Lookups

Jay Hubbert — Wed, 19 Aug 2026 09:00:00 +0000

Every hash table in PostgreSQL is one of two things, and the difference between them explains a surprising amount of PostgreSQL’s behaviour. One is dynahash: a chained hash table that lives in shared memory, that never moves an entry once it is placed, and that grows by adding exactly one bucket at a time. The other is simplehash: a table generated from a C template, using open addressing with robin-hood insertion, that resizes its entire array in one memcpy-flavoured operation and never has to rehash a single key.

They are not interchangeable, and they are not supposed to be. The header of dynahash says so, in the source, in the first fifty lines (src/backend/utils/hash/dynahash.c):

 c
 
 
 * Compared to simplehash, dynahash has the following benefits:
 *
 * - It supports partitioning, which is useful for shared memory access using
 * locks.
 * - Shared memory hashes are allocated in a fixed size area at startup and
 * are discoverable by name from other processes.
 * - Because entries don't need to be moved in the case of hash conflicts,
 * dynahash has better performance for large entries.
 * - Guarantees stable pointers to entries.

Four benefits, and every one of them is a constraint that a general-purpose open-addressing table cannot satisfy. Partitioning. Fixed address. Stable pointers. Large entries. That is the trade: dynahash is slower per lookup and much more awkward to use, but it can be shared between processes at a fixed address, locked a partition at a time, and hold pointers to entries that remain valid for the lifetime of the table.

This is a walk through how both of them work, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

The shape of a dynahash table

A dynahash table has two halves. The header (HASHHDR) holds everything that can change at runtime; the control block (HTAB) holds the function pointers and the local, per-backend copy of a few fields.

 c
 
 
struct HASHHDR
{
	/*
	 * The freelist can become a point of contention in high-concurrency hash
	 * tables, so we use an array of freelists, each with its own mutex and
	 * nentries count, instead of just a single one. Although the freelists
	 * normally operate independently, we will scavenge entries from freelists
	 * other than a hashcode's default freelist when necessary.
	 *
	 * If the hash table is not partitioned, only freeList[0] is used and its
	 * spinlock is not used at all; callers' locking is assumed sufficient.
	 */
	FreeListData freeList[NUM_FREELISTS];

	/* These fields can change, but not in a partitioned table */
	/* Also, dsize can't change in a shared table, even if unpartitioned */
	int64		dsize;			/* directory size */
	int64		nsegs;			/* number of allocated segments (<= dsize) */
	uint32		max_bucket;		/* ID of maximum bucket in use */
	uint32		high_mask;		/* mask to modulo into entire table */
	uint32		low_mask;		/* mask to modulo into lower half of table */

	/* These fields are fixed at hashtable creation */
	Size		keysize;		/* hash key length in bytes */
	Size		entrysize;		/* total user element size in bytes */
	int64		num_partitions; /* # partitions (must be power of 2), or 0 */
	int64		max_dsize;		/* 'dsize' limit if directory is fixed size */
	int			nelem_alloc;	/* number of entries to allocate at once */
	bool		isfixed;		/* if true, don't enlarge */

	/* Current directory. In shared tables, this doesn't change */
	HASHSEGMENT *dir;

#ifdef HASH_STATISTICS

	/*
	 * Count statistics here. NB: stats code doesn't bother with mutex, so
	 * counts could be corrupted a bit in a partitioned table.
	 */
	uint64		accesses;
	uint64		collisions;
	uint64		expansions;
#endif
};

Two masks, one directory, and a max_bucket. That is the whole geometry.

The constants that define it are few and small:

 c
 
 
#define HASH_SEGSIZE			 256
#define HASH_SEGSIZE_SHIFT	 8	/* must be log2(HASH_SEGSIZE) */
#define DEF_DIRSIZE			 256

/* Number of freelists to be used for a partitioned hash table. */
#define NUM_FREELISTS			32

Constant	Value	Meaning
`HASH_SEGSIZE`	256	bucket headers per segment
`HASH_SEGSIZE_SHIFT`	8	`log2(HASH_SEGSIZE)`, used to split a bucket number into segment index and slot
`DEF_DIRSIZE`	256	minimum number of directory entries
`NUM_FREELISTS`	32	freelists in a partitioned table

The directory of segments

The buckets are not one flat array. They are arranged in a two-level structure: a directory of pointers to segments, each segment being an array of 256 bucket headers. The header comment says why:

 c
 
 
/*
 * A hash table has a top-level "directory", each of whose entries points to a
 * "segment" of HASH_SEGSIZE bucket headers. The maximum number of hash
 * buckets is thus dsize * HASH_SEGSIZE (but dsize may be expansible). Of
 * course, the number of records in the table can be larger, but we don't want
 * a whole lot of records per bucket or performance goes down.
 *
 * In a hash table allocated in shared memory, the directory cannot be
 * expanded because it must stay at a fixed address. The directory size is
 * chosen at creation based on the initial number of elements, so even though
 * we support allocating more elements later, performance will suffer if the
 * table grows much beyond the initial size. (Currently, shared memory hash
 * tables are only created by ShmemRequestHash()/ShmemInitHash() though, which
 * doesn't support growing at all.)
 */

The indirection through a directory exists so that a table can grow without moving the bucket headers themselves. A segment is allocated once and never moved; adding capacity means adding a pointer to the directory. And since the directory of a shared-memory table has to stay at a fixed address, dsize there is frozen:

 c
 
 
	/* SHM hash tables have a fixed directory. */
	if (hashp->isshared)
		hctl->max_dsize = hctl->dsize;

Turning a bucket number into a bucket header is two shifts and a mask, done in hash_initial_lookup():

 c
 
 
static inline uint32
hash_initial_lookup(HTAB *hashp, uint32 hashvalue, HASHBUCKET **bucketptr)
{
	HASHHDR *hctl = hashp->hctl;
	HASHSEGMENT segp;
	int64		segment_num;
	int64		segment_ndx;
	uint32		bucket;

	bucket = calc_bucket(hctl, hashvalue);

	segment_num = bucket >> HASH_SEGSIZE_SHIFT;
	segment_ndx = MOD(bucket, HASH_SEGSIZE);

	segp = hashp->dir[segment_num];

	if (segp == NULL)
		hash_corrupted(hashp);

	*bucketptr = &segp[segment_ndx];
	return bucket;
}

And MOD is the fast path:

 c
 
 
/*
 * Fast MOD arithmetic, assuming that y is a power of 2 !
 */
#define MOD(x,y)			 ((x) & ((y)-1))

Two masks and a power of two

The bucket count is always a power of two, which is what lets the whole thing use masks instead of divisions. init_htab() sets it up:

 c
 
 
	/*
	 * Allocate space for the next greater power of two number of buckets,
	 * assuming a desired maximum load factor of 1.
	 */
	nbuckets = next_pow2_int(nelem);

	/*
	 * In a partitioned table, nbuckets must be at least equal to
	 * num_partitions; were it less, keys with apparently different partition
	 * numbers would map to the same bucket, breaking partition independence.
	 * (Normally nbuckets will be much bigger; this is just a safety check.)
	 */
	while (nbuckets < hctl->num_partitions)
		nbuckets <<= 1;

	hctl->max_bucket = hctl->low_mask = nbuckets - 1;
	hctl->high_mask = (nbuckets << 1) - 1;

Two masks, not one. low_mask covers the lower half of the current table size and high_mask covers twice that. The bucket function is a two-step reduction:

 c
 
 
/* Convert a hash value to a bucket number */
static inline uint32
calc_bucket(HASHHDR *hctl, uint32 hash_val)
{
	uint32		bucket;

	bucket = hash_val & hctl->high_mask;
	if (bucket > hctl->max_bucket)
		bucket = bucket & hctl->low_mask;

	return bucket;
}

Read it as: mask down to the current hash space; if the result is beyond the buckets that actually exist, fold it back into the lower half. So:

`nbuckets` (initial)	`low_mask`	`high_mask`	buckets in use initially
8	7	15	0–7
16	15	31	0–15
32	31	63	0–31
64	63	127	0–63
128	127	255	0–127

The two-mask scheme is what makes incremental growth possible. A table of 8 buckets has a hash space of 16 (high_mask = 15). Growing it means creating bucket 8, then 9, then 10, and so on, up to bucket 15 — and only when the table reaches 16 buckets in use do the masks double, at which point the hash space becomes 32.

The important consequence, and the reason for the whole exercise, is that growing the table never requires rehashing every key. In a conventional open-addressing table, doubling the array means recomputing every key’s bucket index, because the modulus changed. Here the mask changes, but the entries are not redistributed by rehashing — only one old bucket is split, and only the entries that land in the new bucket move.

The partition trick

Note the shape of calc_bucket(): it uses the low-order bits of the hash value to select the bucket. That is deliberate, and the file header spells out the consequence:

 c
 
 
 * This lets any subset of the hash buckets be treated as a separately
 * lockable partition. We expect callers to use the low-order bits of a
 * lookup key's hash value as a partition number --- this will work because
 * of the way calc_bucket() maps hash values to bucket numbers.

Because the bucket is a function of the low bits of the hash, so is the partition. If partition p is defined as hashcode % p, then every key in partition p lands in a bucket that only ever holds keys from partition p. The buckets belonging to one partition are therefore an independent subtable, and can be locked on their own.

The entry layout

An entry in a dynahash table is a small header followed by the caller’s data, with the key at the start of the caller’s data. The header (src/include/utils/hsearch.h):

 c
 
 
/*
 * HASHELEMENT is the private part of a hashtable entry. The caller's data
 * follows the HASHELEMENT structure (on a MAXALIGN'd boundary). The hash key
 * is expected to be at the start of the caller's hash entry data structure.
 */
typedef struct HASHELEMENT
{
	struct HASHELEMENT *link;	/* link to next entry in same bucket */
	uint32		hashvalue;		/* hash function result for this entry */
} HASHELEMENT;

Two fields: a pointer to the next entry in the same bucket, and the hash value. The key follows, at a MAXALIGN’d boundary:

 c
 
 
/*
 * Key (also entry) part of a HASHELEMENT
 */
#define ELEMENTKEY(helem) (((char *)(helem)) + MAXALIGN(sizeof(HASHELEMENT)))

/*
 * Obtain element pointer given pointer to key
 */
#define ELEMENT_FROM_KEY(key) \
	((HASHELEMENT *) (((char *) (key)) - MAXALIGN(sizeof(HASHELEMENT))))

Note that HASHELEMENT carries the hash value with the entry. That is not decoration: it lets the chain walk reject a non-matching entry with a single 32-bit comparison instead of a memcmp of the whole key. Here is the entire chain-walking loop from hash_search_with_hash_value():

 c
 
 
	/*
	 * Follow collision chain looking for matching key
	 */
	match = hashp->match;		/* save one fetch in inner loop */
	keysize = hashp->keysize;	/* ditto */

	while (currBucket != NULL)
	{
		if (currBucket->hashvalue == hashvalue &&
			match(ELEMENTKEY(currBucket), keyPtr, keysize) == 0)
			break;
		prevBucketPtr = &(currBucket->link);
		currBucket = *prevBucketPtr;
#ifdef HASH_STATISTICS
		hctl->collisions++;
#endif
	}

The hash value is compared first, and only on a match is the full key comparison performed. On a 32-bit hash, a false positive requires the colliding pair to have identical hash values, which is 1 in 2^32 — so the memcmp is almost never executed for a non-matching entry.

Growth, one bucket at a time

Now the interesting part. Here is expand_table() in full (src/backend/utils/hash/dynahash.c):

 c
 
 
/*
 * Expand the table by adding one more hash bucket.
 */
static bool
expand_table(HTAB *hashp)
{
	HASHHDR *hctl = hashp->hctl;
	HASHSEGMENT old_seg,
				new_seg;
	int64		old_bucket,
				new_bucket;
	int64		new_segnum,
				new_segndx;
	int64		old_segnum,
				old_segndx;
	HASHBUCKET *oldlink,
			 *newlink;
	HASHBUCKET	currElement,
				nextElement;

	Assert(!hctl->isfixed);
	Assert(!IS_PARTITIONED(hctl));

#ifdef HASH_STATISTICS
	hctl->expansions++;
#endif

	new_bucket = hctl->max_bucket + 1;
	new_segnum = new_bucket >> HASH_SEGSIZE_SHIFT;
	new_segndx = MOD(new_bucket, HASH_SEGSIZE);

	if (new_segnum >= hctl->nsegs)
	{
		/* Allocate new segment if necessary -- could fail if dir full */
		if (new_segnum >= hctl->dsize)
			if (!dir_realloc(hashp))
				return false;
		if (!(hashp->dir[new_segnum] = seg_alloc(hashp)))
			return false;
		hctl->nsegs++;
	}

	/* OK, we created a new bucket */
	hctl->max_bucket++;

	/*
	 * *Before* changing masks, find old bucket corresponding to same hash
	 * values; values in that bucket may need to be relocated to new bucket.
	 * Note that new_bucket is certainly larger than low_mask at this point,
	 * so we can skip the first step of the regular hash mask calc.
	 */
	old_bucket = (new_bucket & hctl->low_mask);

	/*
	 * If we crossed a power of 2, readjust masks.
	 */
	if ((uint32) new_bucket > hctl->high_mask)
	{
		hctl->low_mask = hctl->high_mask;
		hctl->high_mask = (uint32) new_bucket | hctl->low_mask;
	}

	/*
	 * Relocate records to the new bucket. NOTE: because of the way the hash
	 * masking is done in calc_bucket, only one old bucket can need to be
	 * split at this point. With a different way of reducing the hash value,
	 * that might not be true!
	 */
	old_segnum = old_bucket >> HASH_SEGSIZE_SHIFT;
	old_segndx = MOD(old_bucket, HASH_SEGSIZE);

	old_seg = hashp->dir[old_segnum];
	new_seg = hashp->dir[new_segnum];

	oldlink = &old_seg[old_segndx];
	newlink = &new_seg[new_segndx];

	for (currElement = *oldlink;
		 currElement != NULL;
		 currElement = nextElement)
	{
		nextElement = currElement->link;
		if ((int64) calc_bucket(hctl, currElement->hashvalue) == old_bucket)
		{
			*oldlink = currElement;
			oldlink = &currElement->link;
		}
		else
		{
			*newlink = currElement;
			newlink = &currElement->link;
		}
	}
	/* don't forget to terminate the rebuilt hash chains... */
	*oldlink = NULL;
	*newlink = NULL;

	return true;
}

Step by step:

new_bucket = hctl->max_bucket + 1. One bucket. Not two, not a doubling.
If the new bucket falls in a segment that doesn’t exist yet, allocate a new 256-entry segment and add a pointer to it in the directory. If the directory itself is full, double the directory (non-shared tables only).
hctl->max_bucket++. Now calc_bucket() can produce new_bucket as a result.
Compute the one old bucket that will be split: old_bucket = new_bucket & low_mask.
If we have just crossed a power of two, adjust the masks: low_mask = high_mask, high_mask = new_bucket | low_mask.
Walk the chain of old_bucket once. Each entry is re-tested with calc_bucket() under the new masks; entries that still hash to old_bucket stay, the rest are pushed onto the new bucket’s chain.

No entry is ever moved in memory. The HASHELEMENT structs stay exactly where they were allocated; only the link pointers are rewritten. That is the “Guarantees stable pointers to entries” promise in the file header, and it is why dynahash can be used for things where a caller holds a pointer to an entry for weeks.

Why can only one bucket be split? Because of how calc_bucket() reduces the hash. When max_bucket is one below a power of two, the hash space is 2 * nbuckets and buckets 0..nbuckets-1 are “full” in the sense that every hash value in the lower half and the upper half maps into them. When you add the first bucket of the new half of the hash space, only the entries whose upper-half hash bits happen to land on it need to move — and all of those live in exactly one old bucket, because the old bucket for a given upper-half value is value & low_mask.

Note the comment on the split itself:

NOTE: because of the way the hash masking is done in calc_bucket, only one old bucket can need to be split at this point. With a different way of reducing the hash value, that might not be true!

That is a load-bearing comment. The whole “grow by one bucket” scheme depends on the low-bit reduction. Change the reduction and you get an O(n) rehash.

When expansion happens

Expansion is triggered from hash_search_with_hash_value(), before the lookup, and only on an insert:

 c
 
 
	/*
	 * If inserting, check if it is time to split a bucket.
	 *
	 * NOTE: failure to expand table is not a fatal error, it just means we
	 * have to run at higher fill factor than we wanted. However, if we're
	 * using the palloc allocator then it will throw error anyway on
	 * out-of-memory, so we must do this before modifying the table.
	 */
	if (action == HASH_ENTER || action == HASH_ENTER_NULL)
	{
		/*
		 * Can't split if running in partitioned mode, nor if frozen, nor if
		 * table is the subject of any active hash_seq_search scans.
		 */
		if (hctl->freeList[0].nentries > (int64) hctl->max_bucket &&
			!hctl->isfixed && !IS_PARTITIONED(hctl) && !hashp->frozen &&
			!has_seq_scans(hashp))
			(void) expand_table(hashp);
	}

Four separate conditions block the split, and each of them is a design decision worth understanding:

hctl->isfixed — the table has a hard size limit (HASH_FIXED_SIZE).
IS_PARTITIONED(hctl) — see below.
hashp->frozen — the table has been frozen with hash_freeze().
has_seq_scans(hashp) — see below.

And the trigger condition itself is a load factor of 1: grow when nentries > max_bucket, i.e. when the number of live entries exceeds the number of buckets. At that point the chain lengths start to become interesting.

The failure mode is graceful: expand_table() can fail (out of memory, directory full) and the table simply runs at a higher fill factor.

Allocating entries in chunks

Entries are not allocated one at a time. They are allocated in blocks, and the block size is chosen by choose_nelem_alloc():

 c
 
 
/*
 * Given the user-specified entry size, choose nelem_alloc, ie, how many
 * elements to add to the hash table when we need more.
 */
static int
choose_nelem_alloc(Size entrysize)
{
	int			nelem_alloc;
	Size		elementSize;
	Size		allocSize;

	/* Each element has a HASHELEMENT header plus user data. */
	/* NB: this had better match element_alloc() */
	elementSize = MAXALIGN(sizeof(HASHELEMENT)) + MAXALIGN(entrysize);

	/*
	 * The idea here is to choose nelem_alloc at least 32, but round up so
	 * that the allocation request will be a power of 2 or just less. This
	 * makes little difference for hash tables in shared memory, but for hash
	 * tables managed by palloc, the allocation request will be rounded up to
	 * a power of 2 anyway. If we fail to take this into account, we'll waste
	 * as much as half the allocated space.
	 */
	allocSize = 32 * 4;			/* assume elementSize at least 8 */
	do
	{
		allocSize <<= 1;
		nelem_alloc = allocSize / elementSize;
	} while (nelem_alloc < 32);

	return nelem_alloc;
}

At least 32 elements per allocation, rounded so that the request size is close to a power of two. The reasoning is about palloc’s allocator: if you ask for 100 bytes, palloc gives you a 128-byte chunk, and you have wasted 28 bytes per block. Asking for a size that is itself near a power of two wastes at most half.

element_alloc() carves a block into a chain of elements and threads it onto the freelist:

 c
 
 
/*
 * allocate some new elements and link them into the indicated free list
 */
static bool
element_alloc(HTAB *hashp, int nelem, int freelist_idx)
{
	HASHHDR *hctl = hashp->hctl;
	Size		elementSize;
	Size		requestSize;
	char	 *allocedBlock;
	HASHELEMENT *firstElement;
	HASHELEMENT *tmpElement;
	HASHELEMENT *prevElement;
	int			i;

	if (hctl->isfixed)
		return false;

	/* Each element has a HASHELEMENT header plus user data. */
	elementSize = MAXALIGN(sizeof(HASHELEMENT)) + MAXALIGN(hctl->entrysize);

	requestSize = nelem * elementSize;
	...
	/* prepare to link all the new entries into the freelist */
	prevElement = NULL;
	tmpElement = firstElement;
	for (i = 0; i < nelem; i++)
	{
		tmpElement->link = prevElement;
		prevElement = tmpElement;
		tmpElement = (HASHELEMENT *) (((char *) tmpElement) + elementSize);
	}

	/* if partitioned, must lock to touch freeList */
	if (IS_PARTITIONED(hctl))
		SpinLockAcquire(&hctl->freeList[freelist_idx].mutex);

	/* freelist could be nonempty if two backends did this concurrently */
	firstElement->link = hctl->freeList[freelist_idx].freeList;
	hctl->freeList[freelist_idx].freeList = prevElement;

	if (IS_PARTITIONED(hctl))
		SpinLockRelease(&hctl->freeList[freelist_idx].mutex);

	return true;
}

The elements in a block are contiguous, elementSize apart. The freelist threads them together by reusing the link field of each HASHELEMENT — a free entry’s link points at the next free entry rather than at the next entry in its bucket. One pointer, two uses, exactly like an arena allocator’s free list.

The 32 freelists

The FreeListData struct is small and, crucially, laid out so that the three things a partitioned table must protect do not share a cache line with another freelist’s:

 c
 
 
/*
 * Per-freelist data.
 *
 * In a partitioned hash table, each freelist is associated with a specific
 * set of hashcodes, as determined by the FREELIST_IDX() macro below.
 * nentries tracks the number of live hashtable entries having those hashcodes
 * (NOT the number of entries in the freelist, as you might expect).
 *
 * The coverage of a freelist might be more or less than one partition, so it
 * needs its own lock rather than relying on caller locking. Relying on that
 * wouldn't work even if the coverage was the same, because of the occasional
 * need to "borrow" entries from another freelist; see get_hash_entry().
 *
 * Using an array of FreeListData instead of separate arrays of mutexes,
 * nentries and freeLists helps to reduce sharing of cache lines between
 * different mutexes.
 */
typedef struct
{
	slock_t		mutex;			/* spinlock for this freelist */
	int64		nentries;		/* number of entries in associated buckets */
	HASHELEMENT *freeList;		/* chain of free elements */
} FreeListData;

Three fields: a spinlock, a live-entry count, and the free chain. The comment about “separate arrays of mutexes, nentries and freeLists” is the whole point: if you had mutexes[32], nentries[32] and freeLists[32] as three parallel arrays, then mutexes[i] and mutexes[i+1] might share a cache line, and every spinlock acquisition on freelist i would invalidate the line holding freelist i+1’s lock. One struct per freelist means each freelist’s hot data is contiguous, and adjacent freelists are unlikely to land in the same line.

Which freelist an entry belongs to is a function of its hash code:

 c
 
#define IS_PARTITIONED(hctl) ((hctl)->num_partitions != 0)

#define FREELIST_IDX(hctl, hashcode) \
	(IS_PARTITIONED(hctl) ? (hashcode) % NUM_FREELISTS : 0)

hashcode % 32 — the low five bits, as you would expect. And note the asymmetry: nentries counts live entries with those hash codes, not free entries. That is what makes the load-factor check in hash_search() work: hctl->freeList[0].nentries is the total number of entries in the table (for an unpartitioned table, where freelist 0 is the only one used).

Borrowing from another freelist

Here is the interesting part of get_hash_entry() — what happens when your own freelist is empty:

 c
 
 
		/*
		 * No free elements in this freelist. In a partitioned table, there
		 * might be entries in other freelists, but to reduce contention we
		 * prefer to first try to get another chunk of buckets from the main
		 * shmem allocator. If that fails, though, we *MUST* root through all
		 * the other freelists before giving up. There are multiple callers
		 * that assume that they can allocate every element in the initially
		 * requested table size, or that deleting an element guarantees they
		 * can insert a new element, even if shared memory is entirely full.
		 * Failing because the needed element is in a different freelist is
		 * not acceptable.
		 */
		if (!element_alloc(hashp, hctl->nelem_alloc, freelist_idx))
		{
			int			borrow_from_idx;

			if (!IS_PARTITIONED(hctl))
				return NULL;	/* out of memory */

			/* try to borrow element from another freelist */
			borrow_from_idx = freelist_idx;
			for (;;)
			{
				borrow_from_idx = (borrow_from_idx + 1) % NUM_FREELISTS;
				if (borrow_from_idx == freelist_idx)
					break;		/* examined all freelists, fail */

				SpinLockAcquire(&(hctl->freeList[borrow_from_idx].mutex));
				newElement = hctl->freeList[borrow_from_idx].freeList;

				if (newElement != NULL)
				{
					hctl->freeList[borrow_from_idx].freeList = newElement->link;
					SpinLockRelease(&(hctl->freeList[borrow_from_idx].mutex));

					/* careful: count the new element in its proper freelist */
					SpinLockAcquire(&hctl->freeList[freelist_idx].mutex);
					hctl->freeList[freelist_idx].nentries++;
					SpinLockRelease(&hctl->freeList[freelist_idx].mutex);

					return newElement;
				}

				SpinLockRelease(&(hctl->freeList[borrow_from_idx].mutex));
			}

			/* no elements available to borrow either, so out of memory */
			return NULL;
		}

Three levels of fallback:

Take an element from your freelist.
Allocate a fresh chunk of elements.
Borrow an element from another freelist, walking all 32 of them.

The borrowing is not an optimisation, it is a correctness requirement. The comment says why: callers assume that “deleting an element guarantees they can insert a new element, even if shared memory is entirely full”. If freelist 5 held the only free element in the table and a backend keyed into freelist 7 refused to take it, the insert would fail with “out of shared memory” even though memory was available. So the borrow is mandatory.

The one subtlety is the nentries bookkeeping. The borrowed element is counted in the borrowing freelist’s nentries, even though it physically came from the other one — because nentries counts live entries by hash code, and the new entry’s hash code belongs to the borrowing freelist. The comment flags it: /* careful: count the new element in its proper freelist */.

Partitioned tables

A partitioned table is one created with the HASH_PARTITION flag:

 c
 
#define HASH_PARTITION	0x0001	/* Hashtable is used w/partitioned locking */

From the file header:

To use a hash table in partitioned mode, the HASH_PARTITION flag must be given to hash_create. This prevents any attempt to split buckets on-the-fly. Therefore, each hash bucket chain operates independently, and no fields of the hash header change after init except nentries and freeList. (A partitioned table uses multiple copies of those fields, guarded by spinlocks, for additional concurrency.)

A partitioned table never splits buckets. And the rule is enforced at three levels:

The HASH_PARTITION flag is checked at creation, and the number of partitions must be a power of two:

 c
 
 
	if (flags & HASH_PARTITION)
	{
		/* Doesn't make sense to partition a local hash table */
		Assert(flags & HASH_SHARED_MEM);

		/*
		 * The number of partitions had better be a power of 2. Also, it must
		 * be less than INT_MAX (see init_htab()), so call the int version of
		 * next_pow2.
		 */
		Assert(info->num_partitions == next_pow2_int(info->num_partitions));

		hctl->num_partitions = info->num_partitions;
	}

The split itself is refused:

 c
 
 
	Assert(!hctl->isfixed);
	Assert(!IS_PARTITIONED(hctl));

And the trigger is skipped:

 c
 
 
		if (hctl->freeList[0].nentries > (int64) hctl->max_bucket &&
			!hctl->isfixed && !IS_PARTITIONED(hctl) && !hashp->frozen &&
			!has_seq_scans(hashp))
			(void) expand_table(hashp);

Why? Because a bucket split would move entries between buckets, and if two buckets belong to different partitions, the split would move an entry from one partition into another — while holding the lock on only one of them. That would be a data race on the chain pointer. The only safe thing to do for a partitioned table is to make the table big enough up front that the chains stay short.

The upshot: a partitioned table cannot grow. Its bucket count is fixed at creation, determined by nelem. If you underestimate nelem, the table keeps working but its chains get longer and longer, and lookups get slower and slower, with no error and no warning. This is why the shared-memory hash API insists on a good estimate:

 c
 
 
 * Note: for a shared-memory hashtable, nelem needs to be a pretty good
 * estimate, since we can't expand the table on the fly. But an unshared
 * hashtable can be expanded on-the-fly, so it's better for nelem to be
 * on the small side and let the table grow if it's exceeded. An overly
 * large nelem will penalize hash_seq_search speed without buying much.

Sequential scans and the split hazard

There is one more reason a table might refuse to split: an active hash_seq_search() scan. The mechanism is documented at length:

 c
 
 
/*
 * We track active hash_seq_search scans here. The need for this mechanism
 * comes from the fact that a scan will get confused if a bucket split occurs
 * while it's in progress: it might visit entries twice, or even miss some
 * entirely (if it's partway through the same bucket that splits). Hence
 * we want to inhibit bucket splits if there are any active scans on the
 * table being inserted into. This is a fairly rare case in current usage,
 * so just postponing the split until the next insertion seems sufficient.
 *
 * Given present usages of the function, only a few scans are likely to be
 * open concurrently; so a finite-size stack of open scans seems sufficient,
 * and we don't worry that linear search is too slow. Note that we do
 * allow multiple scans of the same hashtable to be open concurrently.
 *
 * This mechanism can support concurrent scan and insertion in a shared
 * hashtable if it's the same backend doing both. It would fail otherwise,
 * but locking reasons seem to preclude any such scenario anyway, so we don't
 * worry.
 *
 * This arrangement is reasonably robust if a transient hashtable is deleted
 * without notifying us. The absolute worst case is we might inhibit splits
 * in another table created later at exactly the same address. We will give
 * a warning at transaction end for reference leaks, so any bugs leading to
 * lack of notification should be easy to catch.
 */

#define MAX_SEQ_SCANS 100

static HTAB *seq_scan_tables[MAX_SEQ_SCANS];	/* tables being scanned */
static int	seq_scan_level[MAX_SEQ_SCANS];	/* subtransaction nest level */
static int	num_seq_scans = 0;

A scan holds a cursor into one bucket’s chain. If that bucket is split underneath it, the cursor now points into a chain whose tail has been rewired — the scan could visit entries twice or skip them entirely. So the split is postponed until the next insertion, by which time the scan is probably finished.

The registry is a fixed-size stack of 100 entries. Exceed it and you get an error:

 c
 
 
	if (num_seq_scans >= MAX_SEQ_SCANS)
		elog(ERROR, "too many active hash_seq_search scans, cannot start one on \"%s\"",
			 hashp->tabname);

And leaked scans are reported at transaction end:

 c
 
 
	if (isCommit)
	{
		int			i;

		for (i = 0; i < num_seq_scans; i++)
		{
			elog(WARNING, "leaked hash_seq_search scan for hash table %p",
				 seq_scan_tables[i]);
		}
	}

There is a related escape hatch: hash_freeze() prevents all further insertions, which means no more splits, which means no more need to register scans:

 c
 
 
/*
 * hash_freeze
 *			Freeze a hashtable against future insertions (deletions are
 *			still allowed)
 *
 * The reason for doing this is that by preventing any more bucket splits,
 * we no longer need to worry about registering hash_seq_search scans,
 * and thus caller need not be careful about ensuring hash_seq_term gets
 * called at the right times.
 */

HASH_REMOVE returns a dangling pointer

One API wart deserves its own section, because it is a genuine footgun. The documentation of hash_search() says:

 c
 
 
 * Return value is a pointer to the element found/entered/removed if any,
 * or NULL if no match was found. (NB: in the case of the REMOVE action,
 * the result is a dangling pointer that shouldn't be dereferenced!)

And the implementation says the same thing in more detail:

 c
 
 
				/*
				 * better hope the caller is synchronizing access to this
				 * element, because someone else is going to reuse it the next
				 * time something is added to the table
				 */
				return ELEMENTKEY(currBucket);

What actually happens on a remove is:

 c
 
 
				/* remove record from hash bucket's chain. */
				*prevBucketPtr = currBucket->link;

				/* add the record to the appropriate freelist. */
				currBucket->link = hctl->freeList[freelist_idx].freeList;
				hctl->freeList[freelist_idx].freeList = currBucket;

The entry is unlinked from its bucket and pushed onto the freelist, all under the same link pointer. From that instant the memory belongs to the freelist, and the next insert will take it and overwrite it. The returned pointer is useful for exactly one thing: identifying which entry was removed, if the caller needs to clean up a side table keyed by entry address. Dereferencing it is undefined behaviour, and in a partitioned table another backend may have already reused it.

Choosing the hash function

hash_create() selects the hash function, the comparison function, the key-copying function and the allocator, based on flags. The default selection:

 c
 
 
	/*
	 * Select the appropriate hash function (see comments at head of file).
	 */
	if (flags & HASH_FUNCTION)
	{
		Assert(!(flags & (HASH_BLOBS | HASH_STRINGS)));
		hashp->hash = info->hash;
	}
	else if (flags & HASH_BLOBS)
	{
		Assert(!(flags & HASH_STRINGS));
		/* We can optimize hashing for common key sizes */
		if (info->keysize == sizeof(uint32))
			hashp->hash = uint32_hash;
		else
			hashp->hash = tag_hash;
	}
	else
	{
		/*
		 * string_hash used to be considered the default hash method, and in a
		 * non-assert build it effectively still is. But we now consider it
		 * an assertion error to not say HASH_STRINGS explicitly. To help
		 * catch mistaken usage of HASH_STRINGS, we also insist on a
		 * reasonably long string length: if the keysize is only 4 or 8 bytes,
		 * it's almost certainly an integer or pointer not a string.
		 */
		Assert(flags & HASH_STRINGS);
		Assert(info->keysize > 8);

		hashp->hash = string_hash;
	}

The comparison function follows from the hash function:

 c
 
 
	/*
	 * If you don't specify a match function, it defaults to string_compare if
	 * you used string_hash, and to memcmp otherwise.
	 *
	 * Note: explicitly specifying string_hash is deprecated, because this
	 * might not work for callers in loadable modules on some platforms due to
	 * referencing a trampoline instead of the string_hash function proper.
	 * Specify HASH_STRINGS instead.
	 */
	if (flags & HASH_COMPARE)
		hashp->match = info->match;
	else if (hashp->hash == string_hash)
		hashp->match = (HashCompareFunc) string_compare;
	else
		hashp->match = memcmp;

So the choice is essentially:

Flags	Hash	Compare	Copy
`HASH_STRINGS`	`string_hash`	`string_compare`	`strlcpy`
`HASH_BLOBS`, keysize == 4	`uint32_hash`	`memcmp`	`memcpy`
`HASH_BLOBS`, other keysize	`tag_hash`	`memcmp`	`memcpy`
`HASH_FUNCTION`	yours	`memcmp`	`memcpy`

The hash functions themselves

The default hash for binary keys is Bob Jenkins’ 2006 hash, adapted for PostgreSQL. The file header explains the design decisions:

 c
 
 
/*
 * This hash function was written by Bob Jenkins
 * (bob_jenkins@burtleburtle.net), and superficially adapted
 * for PostgreSQL by Neil Conway. For more information on this
 * hash function, see http://burtleburtle.net/bob/hash/doobs.html,
 * or Bob's article in Dr. Dobb's Journal, Sept. 1997.
 *
 * In the current code, we have adopted Bob's 2006 update of his hash
 * function to fetch the data a word at a time when it is suitably aligned.
 * This makes for a useful speedup, at the cost of having to maintain
 * four code paths (aligned vs unaligned, and little-endian vs big-endian).
 * It also uses two separate mixing functions mix() and final(), instead
 * of a slower multi-purpose function.
 */

mix() and final()

 c
 
 
/*----------
 * mix -- mix 3 32-bit values reversibly.
 *
 * This is reversible, so any information in (a,b,c) before mix() is
 * still in (a,b,c) after mix().
 ...
 * This does not achieve avalanche. There are input bits of (a,b,c)
 * that fail to affect some output bits of (a,b,c), especially of a. The
 * most thoroughly mixed value is c, but it doesn't really even achieve
 * avalanche in c.
 *----------
 */
#define mix(a,b,c) \
{ \
 a -= c; a ^= rot(c, 4);	c += b; \
 b -= a; b ^= rot(a, 6);	a += c; \
 c -= b; c ^= rot(b, 8);	b += a; \
 a -= c; a ^= rot(c,16);	c += b; \
 b -= a; b ^= rot(a,19);	a += c; \
 c -= b; c ^= rot(b, 4);	b += a; \
}

 c
 
 
/*----------
 * final -- final mixing of 3 32-bit values (a,b,c) into c
 ...
 * The use of separate functions for mix() and final() allow for a
 * substantial performance increase since final() does not need to
 * do well in reverse, but is does need to affect all output bits.
 * mix(), on the other hand, does not need to affect all output
 * bits (affecting 32 bits is enough). The original hash function had
 * a single mixing operation that had to satisfy both sets of requirements
 * and was slower as a result.
 *----------
 */
#define final(a,b,c) \
{ \
 c ^= b; c -= rot(b,14); \
 a ^= c; a -= rot(c,11); \
 b ^= a; b -= rot(a,25); \
 c ^= b; c -= rot(b,16); \
 a ^= c; a -= rot(c, 4); \
 b ^= a; b -= rot(a,14); \
 c ^= b; c -= rot(b,24); \
}

The split exists because the two functions have different requirements. mix() is applied once per 12-byte chunk during the main loop, and it only needs to be reversible and to affect 32 bits of output; reversibility is what lets the same function be used for both directions, and it does not need full avalanche. final() is applied exactly once, at the end, and needs to affect all output bits but does not need to be reversible. Doing both jobs in one function means doing both jobs’ work every time, on every chunk.

Note that final() produces the answer in c; b is “a little less well mixed”, which is why hash_bytes_extended() combines them:

 c
 
 
	/* report the result */
	return ((uint64) b << 32) | c;

The aligned fast path

hash_bytes() is the workhorse:

 c
 
 
/*
 * hash_bytes() -- hash a variable-length key into a 32-bit value
 *		k		: the key (the unaligned variable-length array of bytes)
 *		len		: the length of the key, counting by bytes
 *
 * Returns a uint32 value. Every bit of the key affects every bit of
 * the return value. Every 1-bit and 2-bit delta achieves avalanche.
 * About 6*len+35 instructions. The best hash table sizes are powers
 * of 2. There is no need to do mod a prime (mod is sooo slow!).
 * If you need less than 32 bits, use a bitmask.
 *
 * This procedure must never throw elog(ERROR); the ResourceOwner code
 * relies on this not to fail.
 *
 * Note: we could easily change this function to return a 64-bit hash value
 * by using the final values of both b and c. b is perhaps a little less
 * well mixed than c, however.
 */
uint32
hash_bytes(const unsigned char *k, int keylen)

The body has two nearly-identical halves, selected by alignment:

 c
 
 
	/* If the source pointer is word-aligned, we use word-wide fetches */
	if (((uintptr_t) k & UINT32_ALIGN_MASK) == 0)
	{
		/* Code path for aligned source data */
		const uint32 *ka = (const uint32 *) k;

		/* handle most of the key */
		while (len >= 12)
		{
			a += ka[0];
			b += ka[1];
			c += ka[2];
			mix(a, b, c);
			ka += 3;
			len -= 12;
		}

versus the unaligned path, which reassembles the words byte by byte:

 c
 
 
		/* handle most of the key */
		while (len >= 12)
		{
#ifdef WORDS_BIGENDIAN
			a += (k[3] + ((uint32) k[2] << 8) + ((uint32) k[1] << 16) + ((uint32) k[0] << 24));
			b += (k[7] + ((uint32) k[6] << 8) + ((uint32) k[5] << 16) + ((uint32) k[4] << 24));
			c += (k[11] + ((uint32) k[10] << 8) + ((uint32) k[9] << 16) + ((uint32) k[8] << 24));
#else							/* !WORDS_BIGENDIAN */
			a += (k[0] + ((uint32) k[1] << 8) + ((uint32) k[2] << 16) + ((uint32) k[3] << 24));
			b += (k[4] + ((uint32) k[5] << 8) + ((uint32) k[6] << 16) + ((uint32) k[7] << 24));
			c += (k[8] + ((uint32) k[9] << 8) + ((uint32) k[10] << 16) + ((uint32) k[11] << 24));
#endif							/* WORDS_BIGENDIAN */
			mix(a, b, c);
			k += 12;
			len -= 12;
		}

Four code paths total: aligned/unaligned times little/big endian. That is the price the file header mentions, and the payoff is word-wide loads instead of byte-at-a-time assembly.

The tail — the last 0 to 11 bytes — is handled by a switch fallthrough that assembles the remainder, with the comment that appears in all four variants:

 c
 
 
			case 8:
				/* the lowest byte of c is reserved for the length */
				b += ka[1];
				a += ka[0];
				break;

The initial state mixes the length into the seed so that keys of different lengths hash differently:

 c
 
 
	/* Set up the internal state */
	len = keylen;
	a = b = c = 0x9e3779b9 + len + 3923095;

0x9e3779b9 is the golden-ratio constant from TEA, used all over the place as a “nothing up my sleeve” seed.

The 32-bit specialisation

 c
 
 
/*
 * hash_bytes_uint32() -- hash a 32-bit value to a 32-bit value
 *
 * This has the same result as
 *		hash_bytes(&k, sizeof(uint32))
 * but is faster and doesn't force the caller to store k into memory.
 */
uint32
hash_bytes_uint32(uint32 k)
{
	uint32		a,
				b,
				c;

	a = b = c = 0x9e3779b9 + (uint32) sizeof(uint32) + 3923095;
	a += k;

	final(a, b, c);

	/* report the result */
	return c;
}

No mix() at all — the whole key fits in one chunk, so the main loop never runs. It produces the same value as hash_bytes() would, which matters, because it lets dynahash switch between the two implementations based on key size without changing the hash of anything.

The three dynahash wrappers

 c
 
 
/*
 * string_hash: hash function for keys that are NUL-terminated strings.
 *
 * NOTE: this is the default hash function if none is specified.
 */
uint32
string_hash(const void *key, Size keysize)
{
	/*
	 * If the string exceeds keysize-1 bytes, we want to hash only that many,
	 * because when it is copied into the hash table it will be truncated at
	 * that length.
	 */
	Size		s_len = strlen((const char *) key);

	s_len = Min(s_len, keysize - 1);
	return hash_bytes((const unsigned char *) key, (int) s_len);
}

/*
 * tag_hash: hash function for fixed-size tag values
 */
uint32
tag_hash(const void *key, Size keysize)
{
	return hash_bytes((const unsigned char *) key, (int) keysize);
}

/*
 * uint32_hash: hash function for keys that are uint32 or int32
 *
 * (tag_hash works for this case too, but is slower)
 */
uint32
uint32_hash(const void *key, Size keysize)
{
	Assert(keysize == sizeof(uint32));
	return hash_bytes_uint32(*((const uint32 *) key));
}

string_hash is the only one that does anything clever: it truncates to keysize - 1 bytes because that is all that will survive the strlcpy that copies the key into the table. Two strings that agree on the first keysize - 1 bytes hash the same — and compare equal — because the stored copy is truncated.

simplehash: the templated open-addressing table

Now the other table. simplehash is not a runtime-configurable hash table; it is a code generator. You #define a handful of macros and #include the header, and it emits a complete, type-specialised hash table with your prefix on every symbol.

The header states the case for it:

 c
 
 
/*
 * simplehash.h
 *
 *	 When included this file generates a "templated" (by way of macros)
 *	 open-addressing hash table implementation specialized to user-defined
 *	 types.
 *
 *	 It's probably not worthwhile to generate such a specialized implementation
 *	 for hash tables that aren't performance or space sensitive.
 *
 *	 Compared to dynahash, simplehash has the following benefits:
 *
 *	 - Due to the "templated" code generation has known structure sizes and no
 *	 indirect function calls (which show up substantially in dynahash
 *	 profiles). These features considerably increase speed for small
 *	 entries.
 *	 - Open addressing has better CPU cache behavior than dynahash's chained
 *	 hashtables.
 *	 - The generated interface is type-safe and easier to use than dynahash,
 *	 though at the cost of more complex setup.
 *	 - Allocates memory in a MemoryContext or another allocator with a
 *	 malloc/free style interface (which isn't easily usable in a shared
 *	 memory context)
 *	 - Does not require the overhead of a separate memory context.
 */

“no indirect function calls (which show up substantially in dynahash profiles)” — that is the headline. In dynahash, every comparison goes through a function pointer (hashp->match), and every hash through hashp->hash. In simplehash, SH_EQUAL and SH_HASH_KEY are macros expanded into the generated code, so the compiler inlines them and knows the exact sizes of everything.

The generated type is tiny:

 c
 
 
typedef struct SH_TYPE
{
	/*
	 * Size of data / bucket array, 64 bits to handle UINT32_MAX sized hash
	 * tables. Note that the maximum number of elements is lower
	 * (SH_MAX_FILLFACTOR)
	 */
	uint64		size;

	/* how many elements have valid contents */
	uint32		members;

	/* mask for bucket and size calculations, based on size */
	uint32		sizemask;

	/* boundary after which to grow hashtable */
	uint32		grow_threshold;

	/* hash buckets */
	SH_ELEMENT_TYPE *data;

#ifndef SH_RAW_ALLOCATOR
	/* memory context to use for allocations */
	MemoryContext ctx;
#endif

	/* user defined data, useful for callbacks */
	void	 *private_data;
}			SH_TYPE;

One array of elements. The “buckets” are the array itself, and an entry is either in use or empty:

 c
 
 
typedef enum SH_STATUS
{
	SH_STATUS_EMPTY = 0x00,
	SH_STATUS_IN_USE = 0x01
} SH_STATUS;

The element type must have a status member of that type. That is the only requirement on the caller’s struct.

Sizing: power of two and a fill factor

 c
 
 
/* normal fillfactor, unless already close to maximum */
#ifndef SH_FILLFACTOR
#define SH_FILLFACTOR (0.9)
#endif
/* increase fillfactor if we otherwise would error out */
#define SH_MAX_FILLFACTOR (0.98)

A 0.9 fill factor. Open addressing degrades badly near 1.0, so the table grows at 90% full. The table size is always a power of two:

 c
 
 
static inline uint64
SH_COMPUTE_SIZE(uint64 newsize)
{
	uint64		size;

	/* supporting zero sized hashes would complicate matters */
	size = Max(newsize, 2);

	/* round up size to the next power of 2, that's how bucketing works */
	size = pg_nextpower2_64(size);
	Assert(size <= SH_MAX_SIZE);
	...
	return size;
}

And the mask:

 c
 
 
static inline void
SH_UPDATE_PARAMETERS(SH_TYPE * tb, uint64 newsize)
{
	uint64		size = SH_COMPUTE_SIZE(newsize);

	/* now set size */
	tb->size = size;
	tb->sizemask = (uint32) (size - 1);

	/*
	 * Compute the next threshold at which we need to grow the hash table
	 * again.
	 */
	if (tb->size == SH_MAX_SIZE)
		tb->grow_threshold = ((double) tb->size) * SH_MAX_FILLFACTOR;
	else
		tb->grow_threshold = ((double) tb->size) * SH_FILLFACTOR;
}

The bucket for a hash is hash & sizemask:

 c
 
 
/* return the optimal bucket for the hash */
static inline uint32
SH_INITIAL_BUCKET(SH_TYPE * tb, uint32 hash)
{
	return hash & tb->sizemask;
}

Probing is linear, with wraparound:

 c
 
 
/* return next bucket after the current, handling wraparound */
static inline uint32
SH_NEXT(SH_TYPE * tb, uint32 curelem, uint32 startelem)
{
	curelem = (curelem + 1) & tb->sizemask;

	Assert(curelem != startelem);

	return curelem;
}

/* return bucket before the current, handling wraparound */
static inline uint32
SH_PREV(SH_TYPE * tb, uint32 curelem, uint32 startelem)
{
	curelem = (curelem - 1) & tb->sizemask;

	Assert(curelem != startelem);

	return curelem;
}

And the distance from the optimal position, which is the quantity the robin-hood algorithm compares:

 c
 
 
/* return distance between bucket and its optimal position */
static inline uint32
SH_DISTANCE_FROM_OPTIMAL(SH_TYPE * tb, uint32 optimal, uint32 bucket)
{
	if (optimal <= bucket)
		return bucket - optimal;
	else
		return (tb->size + bucket) - optimal;
}

Robin hood insertion

The design comment describes the algorithm:

 c
 
 
 * Hash table design:
 *
 *	 The hash table design chosen is a variant of linear open-addressing. The
 *	 reason for doing so is that linear addressing is CPU cache & pipeline
 *	 friendly. The biggest disadvantage of simple linear addressing schemes
 *	 are highly variable lookup times due to clustering, and deletions
 *	 leaving a lot of tombstones around. To address these issues a variant
 *	 of "robin hood" hashing is employed. Robin hood hashing optimizes
 *	 chaining lengths by moving elements close to their optimal bucket
 *	 ("rich" elements), out of the way if a to-be-inserted element is further
 *	 away from its optimal position (i.e. it's "poor"). While that can make
 *	 insertions slower, the average lookup performance is a lot better, and
 *	 higher fill factors can be used in a still performant manner. To avoid
 *	 tombstones - which normally solve the issue that a deleted node's
 *	 presence is relevant to determine whether a lookup needs to continue
 *	 looking or is done - buckets following a deleted element are shifted
 *	 backwards, unless they're empty or already at their optimal position.

The insertion loop. This is the heart of it:

 c
 
 
	/* perform insert, start bucket search at optimal location */
	data = tb->data;
	startelem = SH_INITIAL_BUCKET(tb, hash);
	curelem = startelem;
	while (true)
	{
		uint32		curdist;
		uint32		curhash;
		uint32		curoptimal;
		SH_ELEMENT_TYPE *entry = &data[curelem];

		/* any empty bucket can directly be used */
		if (entry->status == SH_STATUS_EMPTY)
		{
			tb->members++;
			entry->SH_KEY = key;
#ifdef SH_STORE_HASH
			SH_GET_HASH(tb, entry) = hash;
#endif
			entry->status = SH_STATUS_IN_USE;
			*found = false;
			return entry;
		}

		/*
		 * If the bucket is not empty, we either found a match (in which case
		 * we're done), or we have to decide whether to skip over or move the
		 * colliding entry. When the colliding element's distance to its
		 * optimal position is smaller than the to-be-inserted entry's, we
		 * shift the colliding entry (and its followers) forward by one.
		 */

		if (SH_COMPARE_KEYS(tb, hash, key, entry))
		{
			Assert(entry->status == SH_STATUS_IN_USE);
			*found = true;
			return entry;
		}

		curhash = SH_ENTRY_HASH(tb, entry);
		curoptimal = SH_INITIAL_BUCKET(tb, curhash);
		curdist = SH_DISTANCE_FROM_OPTIMAL(tb, curoptimal, curelem);

		if (insertdist > curdist)
		{
			SH_ELEMENT_TYPE *lastentry = entry;
			uint32		emptyelem = curelem;
			uint32		moveelem;
			int32		emptydist = 0;

			/* find next empty bucket */
			while (true)
			{
				SH_ELEMENT_TYPE *emptyentry;

				emptyelem = SH_NEXT(tb, emptyelem, startelem);
				emptyentry = &data[emptyelem];

				if (emptyentry->status == SH_STATUS_EMPTY)
				{
					lastentry = emptyentry;
					break;
				}
				...
			}

			/* shift forward, starting at last occupied element */

			/*
			 * TODO: This could be optimized to be one memcpy in many cases,
			 * excepting wrapping around at the end of ->data. Hasn't shown up
			 * in profiles so far though.
			 */
			moveelem = emptyelem;
			while (moveelem != curelem)
			{
				SH_ELEMENT_TYPE *moveentry;

				moveelem = SH_PREV(tb, moveelem, startelem);
				moveentry = &data[moveelem];

				memcpy(lastentry, moveentry, sizeof(SH_ELEMENT_TYPE));
				lastentry = moveentry;
			}

			/* and fill the now empty spot */
			tb->members++;

			entry->SH_KEY = key;
#ifdef SH_STORE_HASH
			SH_GET_HASH(tb, entry) = hash;
#endif
			entry->status = SH_STATUS_IN_USE;
			*found = false;
			return entry;
		}

		curelem = SH_NEXT(tb, curelem, startelem);
		insertdist++;
		...
	}

The rule, in one line: if the entry already sitting in this bucket is closer to its own optimal position than we are, we steal its bucket and push it forward. The displaced entry is “rich” (it got lucky, it is close to home) and we are “poor” (we have been pushed far from home). The rich entry takes our place; we continue looking. This keeps the displacement distribution tight: the invariant is roughly that displacement is non-increasing as you probe forward, so lookups have predictable, bounded lengths.

The shift itself walks backwards from the empty slot, copying each element one slot forward, until it reaches the bucket we are stealing.

The compare is a macro, and if SH_STORE_HASH is defined, the hash is compared first:

 c
 
 
#ifdef SH_STORE_HASH
#define SH_COMPARE_KEYS(tb, ahash, akey, b) (ahash == SH_GET_HASH(tb, b) && SH_EQUAL(tb, b->SH_KEY, akey))
#else
#define SH_COMPARE_KEYS(tb, ahash, akey, b) (SH_EQUAL(tb, b->SH_KEY, akey))
#endif

Same trick as dynahash: compare the stored hash before calling the user’s equality function.

Two extra growth heuristics keep a badly-behaved table from degenerating. If a probe run gets too long, the table grows even if it is below its threshold:

 c
 
 
		if (unlikely(insertdist > SH_GROW_MAX_DIB) &&
			((double) tb->members / tb->size) >= SH_GROW_MIN_FILLFACTOR)
		{
			tb->grow_threshold = 0;
			goto restart;
		}

 c
 
 
#ifndef SH_GROW_MAX_DIB
#define SH_GROW_MAX_DIB 25
#endif
/* grow if more than elements to move when inserting */
#ifndef SH_GROW_MAX_MOVE
#define SH_GROW_MAX_MOVE 150
#endif
#ifndef SH_GROW_MIN_FILLFACTOR
/* but do not grow due to SH_GROW_MAX_* if below */
#define SH_GROW_MIN_FILLFACTOR 0.1
#endif

The comment explains the scenario the move-count heuristic guards against:

 c
 
 
				/*
				 * To avoid negative consequences from overly imbalanced
				 * hashtables, grow the hashtable if collisions would require
				 * us to move a lot of entries. The most likely cause of such
				 * imbalance is filling a (currently) small table, from a
				 * currently big one, in hash-table order. Don't grow if the
				 * hashtable would be too empty, to prevent quick space
				 * explosion for some weird edge cases.
				 */

SH_GROW_MIN_FILLFACTOR (0.1) is the “don’t grow if the table would be too empty” guard — otherwise a table with three elements in it could decide to double, repeatedly, for no reason.

Backward-shift deletion, and no tombstones

Open-addressing tables usually need tombstones: a deleted slot has to be marked as “something was here” so that a lookup probe does not stop early at an empty slot that used to be full. Tombstones accumulate, and degrade probe lengths until the table is rebuilt.

simplehash avoids them by shifting the following entries backwards:

 c
 
 
		if (entry->status == SH_STATUS_IN_USE &&
			SH_COMPARE_KEYS(tb, hash, key, entry))
		{
			SH_ELEMENT_TYPE *lastentry = entry;

			tb->members--;

			/*
			 * Backward shift following elements till either an empty element
			 * or an element at its optimal position is encountered.
			 *
			 * While that sounds expensive, the average chain length is short,
			 * and deletions would otherwise require tombstones.
			 */
			while (true)
			{
				SH_ELEMENT_TYPE *curentry;
				uint32		curhash;
				uint32		curoptimal;

				curelem = SH_NEXT(tb, curelem, startelem);
				curentry = &tb->data[curelem];

				if (curentry->status != SH_STATUS_IN_USE)
				{
					lastentry->status = SH_STATUS_EMPTY;
					break;
				}

				curhash = SH_ENTRY_HASH(tb, curentry);
				curoptimal = SH_INITIAL_BUCKET(tb, curhash);

				/* current is at optimal position, done */
				if (curoptimal == curelem)
				{
					lastentry->status = SH_STATUS_EMPTY;
					break;
				}

				/* shift */
				memcpy(lastentry, curentry, sizeof(SH_ELEMENT_TYPE));

				lastentry = curentry;
			}

			return true;
		}

The rule: after deleting an entry, pull the following entries backwards until you hit either an empty slot (mark the vacated slot empty) or an entry that is already at its optimal position (it cannot be pulled back any further without making its own displacement worse, so leave it and mark the vacated slot empty instead).

The SH_DELETE_ITEM() variant does the same thing but starting from a pointer to the entry:

 c
 
 
SH_SCOPE void
SH_DELETE_ITEM(SH_TYPE * tb, SH_ELEMENT_TYPE * entry)
{
	SH_ELEMENT_TYPE *lastentry = entry;
	uint32		hash = SH_ENTRY_HASH(tb, entry);
	uint32		startelem = SH_INITIAL_BUCKET(tb, hash);
	uint32		curelem;

	/* Calculate the index of 'entry' */
	curelem = entry - &tb->data[0];

	tb->members--;
	...
}

Note the pointer arithmetic: curelem = entry - &tb->data[0] recovers the bucket index from the pointer, which works because the entries are the buckets.

The resize trick

This is the best trick in the file. When the table grows, every entry has to move to a new bucket index, because the mask changed. The naive way is to rehash every key and reinsert it. simplehash does not rehash a single key, and the comment explains how:

 c
 
 
	/*
	 * Copy entries from the old data to newdata. We theoretically could use
	 * SH_INSERT here, to avoid code duplication, but that's more general than
	 * we need. We neither want tb->members increased, nor do we need to do
	 * deal with deleted elements, nor do we need to compare keys. So a
	 * special-cased implementation is lot faster. As resizing can be time
	 * consuming and frequent, that's worthwhile to optimize.
	 *
	 * To be able to simply move entries over, we have to start not at the
	 * first bucket (i.e olddata[0]), but find the first bucket that's either
	 * empty, or is occupied by an entry at its optimal position. Such a
	 * bucket has to exist in any table with a load factor under 1, as not all
	 * buckets are occupied, i.e. there always has to be an empty bucket. By
	 * starting at such a bucket we can move the entries to the larger table,
	 * without having to deal with conflicts.
	 */

	/* search for the first element in the hash that's not wrapped around */
	for (i = 0; i < oldsize; i++)
	{
		SH_ELEMENT_TYPE *oldentry = &olddata[i];
		uint32		hash;
		uint32		optimal;

		if (oldentry->status != SH_STATUS_IN_USE)
		{
			startelem = i;
			break;
		}

		hash = SH_ENTRY_HASH(tb, oldentry);
		optimal = SH_INITIAL_BUCKET(tb, hash);

		if (optimal == i)
		{
			startelem = i;
			break;
		}
	}

The argument: if you start copying at a bucket that is empty or holds an entry at its optimal position, then every entry from there onward can be placed in the new table by direct computation — the new bucket index is hash & new_mask, and since the new mask is larger, an entry that was at its optimal position in the old table is still at its optimal position in the new one, and no entry from that point onward can wrap around into the already-copied region.

And then the copy loop, which computes the destination directly:

 c
 
 
	/* and copy all elements in the old table */
	copyelem = startelem;
	for (i = 0; i < oldsize; i++)
	{
		SH_ELEMENT_TYPE *oldentry = &olddata[copyelem];

		if (oldentry->status == SH_STATUS_IN_USE)
		{
			uint32		hash;
			uint32		startelem2;
			uint32		curelem;
			SH_ELEMENT_TYPE *newentry;

			hash = SH_ENTRY_HASH(tb, oldentry);
			startelem2 = SH_INITIAL_BUCKET(tb, hash);
			curelem = startelem2;

			/* find empty element to put data into */
			while (true)
			{
				newentry = &newdata[curelem];

				if (newentry->status == SH_STATUS_EMPTY)
				{
					break;
				}

				curelem = SH_NEXT(tb, curelem, startelem2);
			}

			/* copy entry to new slot */
			memcpy(newentry, oldentry, sizeof(SH_ELEMENT_TYPE));
		}

		/* can't use SH_NEXT here, would use new size */
		copyelem++;
		if (copyelem >= oldsize)
		{
			copyelem = 0;
		}
	}

Note the subtlety: SH_INITIAL_BUCKET(tb, hash) is called with the already-updated tb->sizemask, so it computes the new bucket index. And the SH_NEXT in the inner loop is a belt-and-braces fallback — in the normal case, the first slot examined is already empty.

And there is no rehashing: SH_ENTRY_HASH reads the stored hash if SH_STORE_HASH is defined:

 c
 
 
static inline uint32
SH_ENTRY_HASH(SH_TYPE * tb, SH_ELEMENT_TYPE * entry)
{
#ifdef SH_STORE_HASH
	return SH_GET_HASH(tb, entry);
#else
	return SH_HASH_KEY(tb, entry->SH_KEY);
#endif
}

So with SH_STORE_HASH — which is strongly recommended — the resize is “copy the entry to hash & new_mask” and nothing else. With it undefined, the hash is recomputed per entry, which is still not a rehash in the “recompute bucket from scratch and reinsert with collision resolution” sense, but it is more work.

Iterating backwards

The iterator goes backwards, and the reason is the backward-shift deletion:

 c
 
 
/*
 * Initialize iterator.
 */
SH_SCOPE void
SH_START_ITERATE(SH_TYPE * tb, SH_ITERATOR * iter)
{
	uint64		startelem = PG_UINT64_MAX;

	/*
	 * Search for the first empty element. As deletions during iterations are
	 * supported, we want to start/end at an element that cannot be affected
	 * by elements being shifted.
	 */
	for (uint32 i = 0; i < tb->size; i++)
	{
		SH_ELEMENT_TYPE *entry = &tb->data[i];

		if (entry->status != SH_STATUS_IN_USE)
		{
			startelem = i;
			break;
		}
	}

	/* we should have found an empty element */
	Assert(startelem < SH_MAX_SIZE);

	/*
	 * Iterate backwards, that allows the current element to be deleted, even
	 * if there are backward shifts
	 */
	iter->cur = startelem;
	iter->end = iter->cur;
	iter->done = false;
}

 c
 
 
SH_SCOPE	SH_ELEMENT_TYPE *
SH_ITERATE(SH_TYPE * tb, SH_ITERATOR * iter)
{
	/* validate sanity of the given iterator */
	Assert(iter->cur < tb->size);
	Assert(iter->end < tb->size);

	while (!iter->done)
	{
		SH_ELEMENT_TYPE *elem;

		elem = &tb->data[iter->cur];

		/* next element in backward direction */
		iter->cur = (iter->cur - 1) & tb->sizemask;

		if ((iter->cur & tb->sizemask) == (iter->end & tb->sizemask))
			iter->done = true;
		if (elem->status == SH_STATUS_IN_USE)
		{
			return elem;
		}
	}

	return NULL;
}

The logic: if you iterate forwards and delete the current entry, the backward shift moves the next entries into the slot you just vacated, and your forward cursor will re-visit them. If you iterate backwards, the backward shift moves entries away from the cursor, and the element you just returned has already been handled. Starting at an empty slot and finishing at that same slot means the iteration never wraps over itself.

The SH_START_ITERATE_AT() variant lets a caller resume from an arbitrary bucket, for partial iteration:

 c
 
 
/*
 * Initialize iterator to a specific bucket. That's really only useful for
 * cases where callers are partially iterating over the hashspace, and that
 * iteration deletes and inserts elements based on visited entries. Doing that
 * repeatedly could lead to an unbalanced keyspace when always starting at the
 * same position.
 */

A real user: the shared buffer table

The shared buffer lookup table is the most performance-critical hash table in PostgreSQL: every buffer read and every buffer pin goes through it. Here is how it is set up (src/backend/storage/buffer/buf_table.c):

 c
 
 
/* entry for buffer lookup hashtable */
typedef struct
{
	BufferTag	key;			/* Tag of a disk page */
	int			id;				/* Associated buffer ID */
} BufferLookupEnt;

 c
 
 
void
BufTableShmemRequest(void *arg)
{
	int			size;

	/*
	 * Request the shared buffer lookup hashtable.
	 *
	 * Since we can't tolerate running out of lookup table entries, we must be
	 * sure to specify an adequate table size here. The maximum steady-state
	 * usage is of course NBuffers entries, but BufferAlloc() tries to insert
	 * a new entry before deleting the old. In principle this could be
	 * happening in each partition concurrently, so we could need as many as
	 * NBuffers + NUM_BUFFER_PARTITIONS entries.
	 */
	size = NBuffers + NUM_BUFFER_PARTITIONS;

	ShmemRequestHash(.name = "Shared Buffer Lookup Table",
					 .nelems = size,
					 .ptr = &SharedBufHash,
					 .hash_info.keysize = sizeof(BufferTag),
					 .hash_info.entrysize = sizeof(BufferLookupEnt),
					 .hash_info.num_partitions = NUM_BUFFER_PARTITIONS,
					 .hash_flags = HASH_ELEM | HASH_BLOBS | HASH_PARTITION | HASH_FIXED_SIZE,
		);
}

Look at the flags: HASH_PARTITION | HASH_FIXED_SIZE. Partitioned, so never splits. Fixed size, so it cannot grow. The nelem estimate is therefore a hard commitment, and the comment explains the fudge factor.

HASH_BLOBS with a keysize that is not 4 means tag_hash — the generic byte hash — is used.

Computing the hash once

The one piece of genuine cleverness in this file:

 c
 
 
/*
 * BufTableHashCode
 *		Compute the hash code associated with a BufferTag
 *
 * This must be passed to the lookup/insert/delete routines along with the
 * tag. We do it like this because the callers need to know the hash code
 * in order to determine which buffer partition to lock, and we don't want
 * to do the hash computation twice (hash_any is a bit slow).
 */
uint32
BufTableHashCode(BufferTag *tagPtr)
{
	return get_hash_value(SharedBufHash, tagPtr);
}

The caller needs the hash code before it can take the right lock — the partition is selected by the hash code — and then it needs the same hash code to do the lookup. Computing it twice would be wasteful, so the hash is computed once by the caller and passed down to hash_search_with_hash_value():

 c
 
 
int
BufTableLookup(BufferTag *tagPtr, uint32 hashcode)
{
	BufferLookupEnt *result;

	result = (BufferLookupEnt *)
		hash_search_with_hash_value(SharedBufHash,
									tagPtr,
									hashcode,
									HASH_FIND,
									NULL);

	if (!result)
		return -1;

	return result->id;
}

And the partition selection uses the low bits, as dynahash’s design promised (src/include/storage/buf_internals.h):

 c
 
 
/*
 * The shared buffer mapping table is partitioned to reduce contention.
 * To determine which partition lock a given tag requires, compute the tag's
 * hash code with BufTableHashCode(), then apply BufMappingPartitionLock().
 * NB: NUM_BUFFER_PARTITIONS must be a power of 2!
 */
static inline uint32
BufTableHashPartition(uint32 hashcode)
{
	return hashcode % NUM_BUFFER_PARTITIONS;
}

static inline LWLock *
BufMappingPartitionLock(uint32 hashcode)
{
	return &MainLWLockArray[BUFFER_MAPPING_LWLOCK_OFFSET +
							BufTableHashPartition(hashcode)].lock;
}

With NUM_BUFFER_PARTITIONS defined in src/include/storage/lwlock.h:

#define NUM_BUFFER_PARTITIONS 128

So: 128 LWLocks, one per partition, and the partition is chosen by the low 7 bits of the hash code. And because calc_bucket() uses the low bits of the hash to pick the bucket, all the keys in one partition land in a disjoint set of buckets from all the other partitions. The buckets of partition 3 are exclusively locked by partition lock 3. That is the entire trick, and it is why the buffer mapping table scales to hundreds of concurrent backends.

Caveats

A few things to keep in mind.

The hash value is 32 bits. Every hash in dynahash is a uint32. The stored hash is compared before the key, so a false positive means the full key comparison runs — but a false negative is impossible, because the stored hash is computed from the same key. So the only cost of a 32-bit hash is a slightly higher rate of full-key comparisons on collision. With 2^32 keys, expect a birthday-paradox collision somewhere around 77 000 distinct keys hashing to the same bucket. That is fine for a chain of a few entries; it is not fine if the hash is systematically bad.

A bad hash function is catastrophic and silent. hash_bytes() promises that “Every bit of the key affects every bit of the return value”. If you supply your own hash function via HASH_FUNCTION and it does not have that property, the table will degrade into a linked list and nothing will say so.

HASH_REMOVE returns a dangling pointer. Covered above. Do not dereference it.

Partitioned tables cannot grow. Their bucket count is fixed at creation. If you under-estimate nelem, you get long chains and no warning.

Shared hash tables cannot grow at all, directory or buckets:

 c
 
 
	/*
	 * Hash tables allocated in shared memory have a fixed directory and have
	 * all elements allocated upfront. We don't support growing because we'd
	 * need to grow the underlying shmem region with it.
	 *
	 * The shared memory allocator must be specified too.
	 */
	infoP->alloc = ShmemHashAlloc;
	infoP->alloc_arg = NULL;
	hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_FIXED_SIZE;

The two tables have different pointer guarantees. dynahash guarantees that an entry’s address never changes. simplehash does not: SH_GROW() reallocates the whole array, and SH_DELETE() moves entries backwards. Anything holding a SH_ELEMENT_TYPE * across an insert or a delete is holding a pointer that may have just been invalidated.

simplehash’s SH_START_ITERATE is not a snapshot. The documentation is blunt:

 c
 
 
 * During iteration the current entry in the hash table may be deleted,
 * without leading to elements being skipped or returned twice. Additionally
 * the rest of the table may be modified (i.e. there can be insertions or
 * deletions), but if so, there's neither a guarantee that all nodes are
 * visited at least once, nor a guarantee that a node is visited at most once.

hash_seq_search on dynahash has the same caveat, in stronger form: “deleting any other element while the scan is in progress is UNDEFINED (it might be the one that curIndex is pointing at!)”.

What it all means in practice

If you are writing an extension, pick the right table. If the table lives in shared memory, needs partitioned locking, needs stable pointers to entries, or holds large entries, use dynahash — there is nothing else that does those things. If the table is local, small, and lookup-bound, use simplehash: the templating removes the indirect calls that dominate dynahash profiles for small entries.

Sizing a dynahash table is a commitment, not a hint. The nelem you pass determines the bucket count forever (for shared and partitioned tables). Under-estimate and you get silently degrading chains. Over-estimate and you waste shared memory and make hash_seq_search slower, because the scanner walks every bucket.

The growth algorithm is why dynahash never needs to rehash. This is not a detail — it is the reason the whole “grow one bucket at a time” scheme exists. The two-mask reduction in calc_bucket() is what makes a single old bucket splittable, and the low-bit reduction is what makes partitions independent. Both properties fall out of the same 32-bit masking trick.

Hash collisions cost you a memcmp, not a lookup. Because the hash value is stored alongside the entry and compared first, a colliding key costs one extra 32-bit comparison plus one full key comparison. It does not cost a rescan of the chain.

The shared buffer table is the template for anything you build in shared memory. Compute the hash once, use the low bits to pick the partition lock, use the same hash value for the lookup, and size the table for the worst case rather than the average case.

Deletion is where the two designs diverge most. dynahash puts freed entries on a freelist and never touches them again (which is why the pointer dangles). simplehash shifts entries backwards so that no tombstones are needed (which is why pointers move). If your workload is insert-heavy, simplehash wins. If it is delete-heavy and you need stable pointers, dynahash wins.

PostgreSQL Defers and Speeds Up Tuple Deformation

Jay Hubbert — Tue, 28 Jul 2026 09:00:00 +0000

A heap tuple is a packed byte string with no field boundaries in it. Column 3 might be at byte 27, column 4 at byte 31, column 5 — if it is a text — at byte 35. Getting to column 40 means knowing where all 39 attributes before it ended.

For most of PostgreSQL’s life, that meant walking. The deformer would start at the beginning of the tuple, add up the length of attribute 1, align to attribute 2’s boundary, add its length, check the NULL bitmap, and so on. The cost of reading the last column of a wide table was proportional to the number of columns before it, and the whole tuple was walked again for every single column.

PostgreSQL 20 changes this in two ways: the byte offset of every fixed-width attribute is computed once per TupleDesc and cached, and the NULL bitmap is processed eight bits at a time instead of one.

This is a walk through how it all works, based on reading the PostgreSQL source (version 20devel, commit e75fcc9867f).

Why deformation is worth optimising

Deformation is the step between “a tuple is on disk” and “a query has a value”. It happens for every row, for every column that the query actually wants. For an OLAP query over a wide table, it can easily be the dominant cost of the whole plan.

The commit that started this work is c456e391138, “Optimize tuple deformation”, by David Rowley:

For some workloads, tuple deformation can be the most CPU intensive part of processing the query. Some testing with 16 columns on a table where the first column was variable length showed around a 10% increase in transactions per second for an OLAP type query performing aggregation on the 16th column. However, in certain cases, the increases were much higher, up to ~25% on one AMD Zen4 machine.

(That quote is from the earlier commit 5983a4cffc3, “Introduce CompactAttribute array in TupleDesc, take 2”, which is the foundation this one builds on. More on it below.)

The problem with walking

Here is the classic shape of the problem. A tuple’s data area is laid out in attribute order, with each attribute aligned to its type’s requirement:

int4 needs 4-byte alignment and occupies 4 bytes.
int8 needs 8-byte alignment and occupies 8 bytes.
text is a varlena: either a 1-byte header (payload ≤ 126 bytes, no alignment) or a 4-byte header (aligned).

To find attribute N you need the cumulative length of attributes 1..N-1, including the padding bytes inserted for alignment. And NULL attributes occupy zero bytes, so you have to consult the NULL bitmap to know whether to add a length at all.

So the deformer had to do, for each attribute:

 c
 
 
	if (att_isnull(attnum - 1, tup->t_data->t_bits))
	{
		*isnull = true;
		return (Datum) 0;
	}
	else
		return nocachegetattr(tup, attnum, tupleDesc);

One bitmap bit test per attribute, plus a full re-walk of the tuple for every column fetched. (That shape survives today, but only for tuples that have NULLs and whose target attribute has no cached offset — see fastgetattr() below.)

The TupleDesc is where the knowledge lives

The fix is to move the per-tuple work into the per-TupleDesc work. A TupleDesc describes the shape of a tuple — the columns, their types, their alignment, their nullability — and that shape does not change from row to row. So anything that can be computed once per shape should be.

src/include/access/tupdesc.h holds the result:

 c
 
 
/*
 * firstNonCachedOffsetAttr stores the index into the compact_attrs array for
 * the first attribute that we don't have a known attcacheoff for.
 *
 * firstNonGuaranteedAttr stores the index to into the compact_attrs array for
 * the first attribute that is either NULLable, missing, or !attbyval. This
 * can be used in locations as a guarantee that attributes before this will
 * always exist in tuples. The !attbyval part isn't required for this, but
 * including this allows various tuple deforming routines to forego any checks
 * for !attbyval.
 *
 * Once a TupleDesc has been populated, before it is used for any purpose,
 * TupleDescFinalize() must be called on it.
 */
typedef struct TupleDescData
{
	int			natts;			/* number of attributes in the tuple */
	Oid			tdtypeid;		/* composite type ID for tuple type */
	int32		tdtypmod;		/* typmod for tuple type */
	int			tdrefcount;		/* reference count, or -1 if not counting */
	int			firstNonCachedOffsetAttr;	/* index of first compact_attrs
											 * element without an attcacheoff */
	int			firstNonGuaranteedAttr; /* index of the first nullable,
										 * missing, dropped, or !attbyval
										 * compact_attrs element. */
	TupleConstr *constr;		/* constraints, or NULL if none */
	/* compact_attrs[N] is the compact metadata of Attribute Number N+1 */
	CompactAttribute compact_attrs[FLEXIBLE_ARRAY_MEMBER];
}			TupleDescData;

Two integers and a per-attribute attcacheoff field. That is the entire cache.

CompactAttribute

The per-attribute metadata is stored in a parallel array of CompactAttribute structs, which is a cut-down version of FormData_pg_attribute:

 c
 
 
/*
 * CompactAttribute
 *		Cut-down version of FormData_pg_attribute for faster access for tasks
 *		such as tuple deformation. The fields of this struct are populated
 *		using the populate_compact_attribute() function, which must be called
 *		directly after the FormData_pg_attribute struct is populated or
 *		altered in any way.
 *
 * Currently, this struct is 8 bytes. Any code changes which enlarge this
 * struct should be considered very carefully.
 *
 * Code which must access a TupleDesc's attribute data should always make use
 * the fields of this struct when required fields are available here. It's
 * more efficient to access the memory in CompactAttribute due to it being a
 * more compact representation of FormData_pg_attribute and also because
 * accessing the FormData_pg_attribute requires an additional calculations to
 * obtain the base address of the array within the TupleDesc.
 */
typedef struct CompactAttribute
{
	int16		attcacheoff;	/* fixed offset into tuple, if known, or -1 */
	int16		attlen;			/* attr len in bytes or -1 = varlen, -2 =
								 * cstring */
	bool		attbyval;		/* as FormData_pg_attribute.attbyval */
	uint8		attalignby;		/* alignment requirement in bytes */
	bool		attispackable:1;	/* FormData_pg_attribute.attstorage !=
									 * TYPSTORAGE_PLAIN */
	bool		atthasmissing:1;	/* as FormData_pg_attribute.atthasmissing */
	bool		attisdropped:1; /* as FormData_pg_attribute.attisdropped */
	bool		attgenerated:1; /* FormData_pg_attribute.attgenerated != '\0' */
	char		attnullability; /* status of not-null constraint, see below */
} CompactAttribute;

Eight bytes per column. The commit that introduced this array put it at 16 bytes; a later commit (d8a859d22b1, “Reduce size of CompactAttribute struct to 8 bytes”) halved it.

Compare that to FormData_pg_attribute, which is 104 bytes per column. The compact array is 13 times smaller, which means a 40-column table’s descriptor metadata fits in 320 bytes instead of 4160 — and, more importantly, the deformation loop touches one or two cache lines instead of seven.

The attcacheoff field is the cache itself. It is an int16, and -1 means “I don’t know where this attribute is”.

The accessor is trivial, but it has an assert-enabled sanity check:

 c
 
 
/*
 * Accessor for the i'th CompactAttribute element of tupdesc.
 */
static inline CompactAttribute *
TupleDescCompactAttr(TupleDesc tupdesc, int i)
{
	CompactAttribute *cattr = &tupdesc->compact_attrs[i];

#ifdef USE_ASSERT_CHECKING

	/* Check that the CompactAttribute is correctly populated */
	verify_compact_attribute(tupdesc, i);
#endif

	return cattr;
}

TupleDescFinalize: computing the cache once

The cache is built by TupleDescFinalize(), in src/backend/access/common/tupdesc.c:

 c
 
 
/*
 * TupleDescFinalize
 *		Finalize the given TupleDesc. This must be called after the
 *		attributes arrays have been populated or adjusted by any code.
 *
 * Must be called after populate_compact_attribute() and before
 * BlessTupleDesc().
 */
void
TupleDescFinalize(TupleDesc tupdesc)
{
	int			firstNonCachedOffsetAttr = 0;
	int			firstNonGuaranteedAttr = tupdesc->natts;
	int			off = 0;

	for (int i = 0; i < tupdesc->natts; i++)
	{
		CompactAttribute *cattr = TupleDescCompactAttr(tupdesc, i);
		Form_pg_attribute attr = TupleDescAttr(tupdesc, i);

		/*
		 * Find the highest attnum which is guaranteed to exist in all tuples
		 * in the table. We currently only pay attention to byval attributes
		 * to allow additional optimizations during tuple deformation.
		 */
		if (firstNonGuaranteedAttr == tupdesc->natts &&
			(cattr->attnullability != ATTNULLABLE_VALID || !cattr->attbyval ||
			 cattr->atthasmissing || cattr->attisdropped ||
			 cattr->attlen <= 0 ||
			 attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL))
			firstNonGuaranteedAttr = i;

		/*
		 * Don't cache offsets beyond fixed-width attributes. Virtual
		 * generated attributes are stored as NULLs in the tuple, so we don't
		 * cache offsets beyond these.
		 */
		if (cattr->attlen <= 0 ||
			attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
			break;

		off = att_nominal_alignby(off, cattr->attalignby);

		/*
		 * attcacheoff is an int16, so don't try to cache any offsets larger
		 * than will fit in that type. Any attributes which are offset more
		 * than 2^15 are likely due to variable-length attributes. Since we
		 * don't cache offsets for or beyond variable-length attributes, using
		 * an int16 rather than an int32 here is unlikely to cost us anything.
		 */
		if (off > PG_INT16_MAX)
			break;

		cattr->attcacheoff = (int16) off;

		off += cattr->attlen;
		firstNonCachedOffsetAttr = i + 1;
	}

	tupdesc->firstNonCachedOffsetAttr = firstNonCachedOffsetAttr;
	tupdesc->firstNonGuaranteedAttr = firstNonGuaranteedAttr;
}

This is the whole algorithm. It walks the attributes once, tracking a running offset, and stops at the first attribute it cannot handle.

Why it stops at the first varlena

The loop breaks when cattr->attlen <= 0. A negative attlen means variable-length (-1) or cstring (-2). The reason is obvious once you think about it: a varlena’s size is not known from the descriptor — it is stored in the tuple. So the offset of the attribute after a varlena cannot be computed from the descriptor alone.

That means attcacheoff is only defined for the fixed-width prefix of a tuple’s columns. If your table is id int, name text, age int, then id has attcacheoff == 0 and name and age have attcacheoff == -1. The walk has to start over at the varlena.

firstNonCachedOffsetAttr records where the fixed-width prefix ends, so the deformer knows how far it can trust the cache.

Why the `PG_INT16_MAX` limit barely matters

 c
 
 
		if (off > PG_INT16_MAX)
			break;

attcacheoff is an int16, so the maximum representable offset is 32767. The comment argues that this is nearly free:

Any attributes which are offset more than 2^15 are likely due to variable-length attributes. Since we don’t cache offsets for or beyond variable-length attributes, using an int16 rather than an int32 here is unlikely to cost us anything.

The reasoning: to reach offset 32767 you would need 32 kB of fixed-width attributes. But a tuple’s data area is at most MaxHeapTupleSize = 8160 bytes. So a fixed-width prefix can never get near 32 kB anyway. The limit is belt-and-braces.

The guaranteed-attribute computation

The same loop computes firstNonGuaranteedAttr: the index of the first attribute that is not guaranteed to exist in every tuple of the relation.

An attribute is guaranteed if:

it has a valid NOT NULL constraint (ATTNULLABLE_VALID, not ATTNULLABLE_UNKNOWN — see below),
it is attbyval (passes its value by value, not by reference),
it has no missing value (atthasmissing),
it is not dropped,
its attlen > 0 (fixed width),
it is not a virtual generated column.

The attbyval restriction is not strictly necessary for the guarantee, but the comment explains the extra benefit:

We currently only pay attention to byval attributes to allow additional optimizations during tuple deformation.

and, from the header:

The !attbyval part isn’t required for this, but including this allows various tuple deforming routines to forego any checks for !attbyval.

By-restriction values would require emitting a pointer-fetching branch in the hot loop even though the fast path never needs it. Restricting the guarantee to by-value attributes keeps the guaranteed-attribute loop a straight-line sequence of aligned loads.

The nullability values are:

 c
 
 
/* Valid values for CompactAttribute->attnullability */
#define	ATTNULLABLE_UNRESTRICTED 'f'	/* No constraint exists */
#define	ATTNULLABLE_UNKNOWN		'u' /* constraint exists, validity unknown */
#define	ATTNULLABLE_VALID		'v' /* valid constraint exists */
#define	ATTNULLABLE_INVALID		'i' /* constraint exists, marked invalid */

and they are populated in populate_compact_attribute_internal():

 c
 
 
	/*
	 * Assign nullability status for this column. Assuming that a constraint
	 * exists, at this point we don't know if a not-null constraint is valid,
	 * so we assign UNKNOWN unless the table is a catalog, in which case we
	 * know it's valid.
	 */
	dst->attnullability = !src->attnotnull ? ATTNULLABLE_UNRESTRICTED :
		IsCatalogRelationOid(src->attrelid) ? ATTNULLABLE_VALID :
		ATTNULLABLE_UNKNOWN;

Note the UNKNOWN state: a user table can have NOT NULL declared but not yet validated (or validated and then marked invalid), so the descriptor cannot assume the constraint holds until the constraint is confirmed valid. Catalogs, on the other hand, are trusted.

And the default attcacheoff is “unknown”:

 c
 
 
static inline void
populate_compact_attribute_internal(Form_pg_attribute src,
									CompactAttribute *dst)
{
	memset(dst, 0, sizeof(CompactAttribute));

	dst->attcacheoff = -1;
	dst->attlen = src->attlen;
	...
}

The fast path: fastgetattr

src/include/access/htup_details.h has the entry point:

 c
 
 
/*
 *	fastgetattr
 *		Fetch a user attribute's value as a Datum (might be either a
 *		value, or a pointer into the data area of the tuple).
 *
 *		This must not be used when a system attribute might be requested.
 *		Furthermore, the passed attnum MUST be valid. Use heap_getattr()
 *		instead, if in doubt.
 *
 *		This gets called many times, so we macro the cacheable and NULL
 *		lookups, and call nocachegetattr() for the rest.
 */
static inline Datum
fastgetattr(HeapTuple tup, int attnum, TupleDesc tupleDesc, bool *isnull)
{
	Assert(attnum > 0);

	*isnull = false;
	if (HeapTupleNoNulls(tup))
	{
		CompactAttribute *att;

		att = TupleDescCompactAttr(tupleDesc, attnum - 1);
		if (att->attcacheoff >= 0)
			return fetchatt(att, (char *) tup->t_data + tup->t_data->t_hoff +
							att->attcacheoff);
		else
			return nocachegetattr(tup, attnum, tupleDesc);
	}
	else
	{
		if (att_isnull(attnum - 1, tup->t_data->t_bits))
		{
			*isnull = true;
			return (Datum) 0;
		}
		else
			return nocachegetattr(tup, attnum, tupleDesc);
	}
}

This is the payoff. The whole function is:

If the tuple has no NULLs at all (HeapTupleNoNulls), skip the bitmap entirely.
Look up attcacheoff for the requested attribute.
If it is non-negative, the value is at t_data + t_hoff + attcacheoff — one addition, one load.

Fetching column 40 of a table whose first 40 columns are fixed-width really does cost one addition.

And the “no NULLs” test is a single bit test on the tuple header:

 c
 
 
static inline bool
HeapTupleHasNulls(const HeapTupleData *tuple)
{
	return (tuple->t_data->t_infomask & HEAP_HASNULL) != 0;
}

static inline bool
HeapTupleNoNulls(const HeapTupleData *tuple)
{
	return !HeapTupleHasNulls(tuple);
}

A tuple with no NULL attributes does not even have a NULL bitmap — the HEAP_HASNULL flag in t_infomask is not set, and the space for the bitmap was never reserved in t_hoff. So the whole bitmap is skipped, and there are no per-attribute NULL checks to make.

The value extraction itself:

 c
 
 
/*
 * Given an attbyval and an attlen from either a Form_pg_attribute or
 * CompactAttribute and a pointer into a tuple's data area, return the
 * correct value or pointer.
 *
 * We return a Datum value in all cases. If attbyval is false, we return
 * the same pointer into the tuple data area that we're passed. Otherwise, we
 * return the correct number of bytes fetched from the data area and extended
 * to Datum form.
 *
 * Note that T must already be properly aligned for this to work correctly.
 */
#define fetchatt(A,T) fetch_att(T, (A)->attbyval, (A)->attlen)

 c
 
 
static inline Datum
fetch_att(const void *T, bool attbyval, int attlen)
{
	if (attbyval)
	{
		switch (attlen)
		{
			case sizeof(char):
				return CharGetDatum(*((const char *) T));
			case sizeof(int16):
				return Int16GetDatum(*((const int16 *) T));
			case sizeof(int32):
				return Int32GetDatum(*((const int32 *) T));
			case sizeof(int64):
				return Int64GetDatum(*((const int64 *) T));
			default:
				elog(ERROR, "unsupported byval length: %d", attlen);
				return 0;
		}
	}
	else
		return PointerGetDatum(T);
}

Because attcacheoff was computed with att_nominal_alignby(), the pointer is correctly aligned for the type — which is why the comment about T being properly aligned is safe to ignore.

The slow path: nocachegetattr

When the attribute is past the fixed-width prefix, or the tuple has NULLs, the work falls back to nocachegetattr() in src/backend/access/common/heaptuple.c:

 c
 
 
Datum
nocachegetattr(HeapTuple tup,
			 int attnum,
			 TupleDesc tupleDesc)
{
	CompactAttribute *cattr;
	HeapTupleHeader td = tup->t_data;
	char	 *tp;				/* ptr to data part of tuple */
	uint8	 *bp = td->t_bits;	/* ptr to null bitmap in tuple */
	int			off;			/* current offset within data */
	int			startAttr;
	int			firstNullAttr;
	int			i;
	bool		hasnulls = HeapTupleHasNulls(tup);

	/* Did someone forget to call TupleDescFinalize()? */
	Assert(tupleDesc->firstNonCachedOffsetAttr >= 0);

	attnum--;

	/*
	 * To minimize the number of attributes we need to look at, start walking
	 * the tuple at the attribute with the highest attcacheoff prior to attnum
	 * or the first NULL attribute prior to attnum, whichever comes first.
	 */
	if (hasnulls)
		firstNullAttr = first_null_attr(bp, attnum);
	else
		firstNullAttr = attnum;

	if (tupleDesc->firstNonCachedOffsetAttr > 0 && firstNullAttr > 0)
	{
		/*
		 * Try to start with the highest attribute with an attcacheoff that's
		 * prior to the one we're looking for, or with the attribute prior to
		 * the first NULL attribute, if there is one.
		 */
		startAttr = Min(tupleDesc->firstNonCachedOffsetAttr - 1, firstNullAttr - 1);
		off = TupleDescCompactAttr(tupleDesc, startAttr)->attcacheoff;
	}
	else
	{
		/* Otherwise, start at the beginning... */
		startAttr = 0;
		off = 0;
	}

	tp = (char *) td + td->t_hoff;

	/*
	 * Calculate 'off' up to the first NULL attr. We use two cheaper loops
	 * when the tuple has no variable-width columns. When variable-width
	 * columns exists, we use att_addlength_pointer() to move the offset
	 * beyond the current attribute.
	 */
	if (!HeapTupleHasVarWidth(tup))
	{
		for (i = startAttr; i < firstNullAttr; i++)
		{
			cattr = TupleDescCompactAttr(tupleDesc, i);

			off = att_nominal_alignby(off, cattr->attalignby);
			off += cattr->attlen;
		}

		for (; i < attnum; i++)
		{
			if (att_isnull(i, bp))
				continue;

			cattr = TupleDescCompactAttr(tupleDesc, i);

			off = att_nominal_alignby(off, cattr->attalignby);
			off += cattr->attlen;
		}
	}
	else
	{
		for (i = startAttr; i < firstNullAttr; i++)
		{
			int			attlen;

			cattr = TupleDescCompactAttr(tupleDesc, i);
			attlen = cattr->attlen;
			off = att_pointer_alignby(off,
									 cattr->attalignby,
									 attlen,
									 tp + off);
			off = att_addlength_pointer(off, attlen, tp + off);
		}

		for (; i < attnum; i++)
		{
			int			attlen;

			if (att_isnull(i, bp))
				continue;

			cattr = TupleDescCompactAttr(tupleDesc, i);
			attlen = cattr->attlen;
			off = att_pointer_alignby(off, cattr->attalignby, attlen,
									 tp + off);
			off = att_addlength_pointer(off, attlen, tp + off);
		}
	}

	cattr = TupleDescCompactAttr(tupleDesc, attnum);
	off = att_pointer_alignby(off,
							 cattr->attalignby,
							 cattr->attlen,
							 tp + off);

	return fetchatt(cattr, tp + off);
}

Three optimisations live in here.

Restart from the highest cached offset

startAttr is Min(firstNonCachedOffsetAttr - 1, firstNullAttr - 1). The deformer does not start at attribute 0; it starts at the last attribute whose offset is cached and which is before the target, and takes its attcacheoff as the starting point.

So for a table with 10 fixed-width columns followed by a text column, if you ask for column 12, the walk starts at column 9’s cached offset and adds only columns 10, 11 and 12’s alignment and length. The 9 preceding columns are free.

The two-loop split

Each of the two loops above is split into two halves:

Before the first NULL: every attribute is non-NULL, so no bitmap check is needed at all.
From the first NULL onwards: each attribute is checked with att_isnull(i, bp).

That is the point of first_null_attr(): it tells you where the NULL bitmap stops being all-ones, so the prefix needs no checks.

The two halves are further split by whether the tuple has variable-width columns, because the two cases need different arithmetic:

Fixed-width only (!HeapTupleHasVarWidth): the offset advances by att_nominal_alignby(off, attalignby) plus attlen. No memory access needed to know the size.
Has varlena: the size is in the tuple, so you must read the tuple data to find it — att_pointer_alignby() (which knows that a short varlena needs no alignment) and att_addlength_pointer().

first_null_attr: finding the first NULL byte-wise

first_null_attr() lives in src/include/access/tupmacs.h:

 c
 
 
/*
 * first_null_attr
 *		Inspect a NULL bitmap from a tuple and return the 0-based attnum of the
 *		first NULL attribute. Returns natts if no NULLs were found.
 *
 * This is coded to expect that 'bits' contains at least one 0 bit somewhere
 * in the array, but not necessarily < natts. Note that natts may be passed
 * as a value lower than the number of bits physically stored in the tuple's
 * NULL bitmap, in which case we may not find a NULL and return natts.
 *
 * The reason we require at least one 0 bit somewhere in the NULL bitmap is
 * that the for loop that checks 0xFF bytes would loop to the last byte in
 * the array if all bytes were 0xFF, and the subsequent code that finds the
 * right-most 0 bit would access the first byte beyond the bitmap. Provided
 * we find a 0 bit before then, that won't happen. Since tuples which have no
 * NULLs don't have a NULL bitmap, this function won't get called for that
 * case.
 */
static inline int
first_null_attr(const uint8 *bits, int natts)
{
	int			nattByte = natts >> 3;
	int			bytenum;
	int			res;

#ifdef USE_ASSERT_CHECKING
	int			firstnull_check = natts;

	/* Do it the slow way and check we get the same answer. */
	for (int i = 0; i < natts; i++)
	{
		if (att_isnull(i, bits))
		{
			firstnull_check = i;
			break;
		}
	}
#endif

	/* Process all bytes up to just before the byte for the natts attribute */
	for (bytenum = 0; bytenum < nattByte; bytenum++)
	{
		/* break if there's any NULL attrs (a 0 bit) */
		if (bits[bytenum] != 0xFF)
			break;
	}

	/*
	 * Look for the highest 0-bit in the 'bytenum' element. To do this, we
	 * promote the uint8 to uint32 before performing the bitwise NOT and
	 * looking for the first 1-bit. This works even when the byte is 0xFF, as
	 * the bitwise NOT of 0xFF in 32 bits is 0xFFFFFF00, in which case
	 * pg_rightmost_one_pos32() will return 8. We may end up with a value
	 * higher than natts here, but we'll fix that with the Min() below.
	 */
	res = bytenum << 3;
	res += pg_rightmost_one_pos32(~((uint32) bits[bytenum]));

	/*
	 * Since we did no masking to mask out bits beyond the natts'th bit, we
	 * may have found a bit higher than natts, so we must cap res to natts
	 */
	res = Min(res, natts);

	/* Ensure we got the same answer as the att_isnull() loop got */
	Assert(res == firstnull_check);

	return res;
}

The bitmap is scanned one byte at a time, looking for the first byte that is not 0xFF. Eight attributes are eliminated per comparison instead of one. Then the answer is extracted from that single byte with pg_rightmost_one_pos32().

Note the USE_ASSERT_CHECKING block: in assert builds, the function also does the slow bit-by-bit scan and asserts the two agree. That is the kind of self-checking that makes a bit-twiddling optimisation safe to merge.

The SWAR trick: populate_isnull_array

The other place the NULL bitmap is processed is where the executor needs the NULL flags as an array of bools rather than as bits. That is populate_isnull_array() in src/include/access/tupmacs.h:

 c
 
 
/*
 * populate_isnull_array
 *		Transform a tuple's null bitmap into a boolean array.
 *
 * Caller must ensure that the isnull array is sized so it contains
 * at least as many elements as there are bits in the 'bits' array.
 * Callers should be aware that isnull is populated 8 elements at a time,
 * effectively as if natts is rounded up to the next multiple of 8.
 */
static inline void
populate_isnull_array(const uint8 *bits, int natts, bool *isnull)
{
	int			nbytes = (natts + 7) >> 3;

	/*
	 * Multiplying the inverted NULL bitmap byte by this value results in the
	 * lowest bit in each byte being set the same as each bit of the inverted
	 * byte. We perform this as 2 32-bit operations rather than a single
	 * 64-bit operation as multiplying by the required value to do this in
	 * 64-bits would result in overflowing a uint64 in some cases.
	 *
	 * XXX if we ever require BMI2 (-march=x86-64-v3), then this could be done
	 * more efficiently on most X86-64 CPUs with the PDEP instruction. Beware
	 * that some chips (e.g. AMD's Zen2) are horribly inefficient at PDEP.
	 */
#define SPREAD_BITS_MULTIPLIER_32 0x204081U

	for (int i = 0; i < nbytes; i++, isnull += 8)
	{
		uint64		isnull_8;
		uint8		nullbyte = ~bits[i];

		/* Convert the lower 4 bits of NULL bitmap word into a 64 bit int */
		isnull_8 = (nullbyte & 0xf) * SPREAD_BITS_MULTIPLIER_32;

		/*
		 * Convert the upper 4 bits of NULL bitmap word into a 64 bit int,
		 * shift into the upper 32 bit and bitwise-OR with the result of the
		 * lower 4 bits.
		 */
		isnull_8 |= ((uint64) ((nullbyte >> 4) * SPREAD_BITS_MULTIPLIER_32)) << 32;

		/* Mask out all other bits apart from the lowest bit of each byte. */
		isnull_8 &= UINT64CONST(0x0101010101010101);

#ifdef WORDS_BIGENDIAN

		/*
		 * Fix byte order on big-endian machines before copying to the array.
		 */
		isnull_8 = pg_bswap64(isnull_8);
#endif
		memcpy(isnull, &isnull_8, sizeof(uint64));
	}
}

This is a SWAR — SIMD Within A Register — bit-spreading trick. The goal: turn one byte of bitmap (8 bits) into 8 bytes of bool (each 0 or 1), i.e. a 64-bit word where each byte is 0x00 or 0x01.

The magic constant is 0x204081. Multiply a 4-bit value n (0..15) by 0x204081 and the result has n’s bits spread out, one into each of four byte positions, in the low bit of each byte. The mask 0x0101010101010101 then keeps only those low bits.

Because the bitmap is inverted (1 means not-null), the code inverts the byte first:

 c
 
		uint8		nullbyte = ~bits[i];

So a NULL attribute becomes a 1 bit in nullbyte, and the multiplication turns that into a true in the output array.

The whole operation is: one invert, two 32-bit multiplies, one shift, one OR, one AND, one 64-bit store. Eight attributes at a time, with no branches.

The comment about PDEP is a nice piece of engineering history:

XXX if we ever require BMI2 (-march=x86-64-v3), then this could be done more efficiently on most X86-64 CPUs with the PDEP instruction. Beware that some chips (e.g. AMD’s Zen2) are horribly inefficient at PDEP.

BMI2’s PDEP (“parallel deposit”) does exactly this bit-spreading in one instruction. But on AMD Zen 2 it is microcoded and slow — slower than the multiply trick — so the portable code stays.

Note also the big-endian fixup: pg_bswap64() reverses the byte order so that the array comes out in attribute order on a big-endian machine.

And note the contract: the caller must ensure the isnull array is sized for the bitmap rounded up to a multiple of 8. The executor’s slot allocation says so explicitly:

 c
 
 
	slot->tts_values = (Datum *)
		MemoryContextAlloc(slot->tts_mcxt, tupdesc->natts * sizeof(Datum));

	/*
	 * We round the size of tts_isnull up to the next highest multiple of 8.
	 * This is needed as populate_isnull_array() operates on 8 elements at a
	 * time when converting a tuple's NULL bitmap into a boolean array.
	 */
	slot->tts_isnull = (bool *)
		MemoryContextAlloc(slot->tts_mcxt, TYPEALIGN(8, tupdesc->natts * sizeof(bool)));

The formalisation of that requirement was part of the “Optimize tuple deformation” commit:

This results in much more optimal code when compared to determining the NULLness via att_isnull(). 8 isnull elements are processed at once using this method, which means we need to round the tts_isnull array size up to the next 8 bytes. The palloc code does this anyway, but the round-up needed to be formalized so as not to overwrite the sentinel byte in MEMORY_CONTEXT_CHECKING builds.

The executor: slot_deform_heap_tuple

The hot path in the executor is not fastgetattr() but slot_deform_heap_tuple() in src/backend/executor/execTuples.c, which deforms all attributes up to reqnatts in one go, rather than one at a time. It is an incremental version of heap_deform_tuple():

 c
 
 
/*
 * slot_deform_heap_tuple
 *		Given a TupleTableSlot, extract data from the slot's physical tuple
 *		into its Datum/isnull arrays. Data is extracted up through the
 *		reqnatts'th column. If there are insufficient attributes in the given
 *		tuple, then slot_getmissingattrs() is called to populate the
 *		remainder. If reqnatts is above the number of attributes in the
 *		slot's TupleDesc, an error is raised.
 *
 *		This is essentially an incremental version of heap_deform_tuple:
 *		on each call we extract attributes up to the one needed, without
 *		re-computing information about previously extracted attributes.
 *		slot->tts_nvalid is the number of attributes already extracted.
 *
 * This is marked as always inline, so the different offp for different types
 * of slots gets optimized away.
 *
 * support_cstring should be passed as a const to allow the compiler only
 * emit code during inlining for cstring deforming when it is required.
 * cstrings can exist in MinimalTuples, but not in HeapTuples.
 */
static pg_always_inline void
slot_deform_heap_tuple(TupleTableSlot *slot, HeapTuple tuple, uint32 *offp,
					 int reqnatts, bool support_cstring)

The structure of the function is a sequence of loops, each handling a different regime of attributes:

Regime	Loop	Cost per attribute
Guaranteed (NOT NULL, byval, fixed width)	1st	one load at a cached offset
Cached offset, before first NULL	2nd	one load at a cached offset
No cached offset, before first NULL	3rd	align, fetch, advance
Anything after the first NULL	4th	check isnull[], then align, fetch, advance

Here is the first loop — the guaranteed-attribute fast path:

 c
 
 
	firstNonGuaranteedAttr = Min(reqnatts, slot->tts_first_nonguaranteed);

	firstNonCacheOffsetAttr = tupleDesc->firstNonCachedOffsetAttr;

	if (HeapTupleHasNulls(tuple))
	{
		natts = HeapTupleHeaderGetNatts(tup);
		tp = (char *) tup + MAXALIGN(offsetof(HeapTupleHeaderData, t_bits) +
									 BITMAPLEN(natts));

		natts = Min(natts, reqnatts);
		if (natts > firstNonGuaranteedAttr)
		{
			uint8	 *bp = tup->t_bits;

			/* Find the first NULL attr */
			firstNullAttr = first_null_attr(bp, natts);

			/*
			 * And populate the isnull array for all attributes being fetched
			 * from the tuple.
			 */
			populate_isnull_array(bp, natts, isnull);
		}
		else
		{
			/* Otherwise all required columns are guaranteed to exist */
			firstNullAttr = natts;

			/*
			 * Check TupleDescFinalize() didn't get confused when setting
			 * firstNonGuaranteedAttr. There should never be a NULL in a
			 * guaranteed column.
			 */
			Assert(first_null_attr(tup->t_bits, natts) >= firstNullAttr);
		}
	}

The natts > firstNonGuaranteedAttr test is the interesting one. If every column the query asked for is guaranteed to exist, the code does not even need to read the tuple’s attribute count — which means it does not need to touch the tuple header at all, and it does not need to populate the isnull array. The Assert verifies that a guaranteed column really is not NULL.

The guaranteed-attribute loop:

 c
 
 
	if (attnum < firstNonGuaranteedAttr)
	{
		int			attlen;

		do
		{
			isnull[attnum] = false;
			cattr = &cattrs[attnum];
			attlen = cattr->attlen;

			/* We don't expect any non-byval types */
			pg_assume(attlen > 0);
			Assert(cattr->attbyval == true);

			off = cattr->attcacheoff;
			values[attnum] = fetch_att_noerr(tp + off, true, attlen);
			attnum++;
		} while (attnum < firstNonGuaranteedAttr);

		off += attlen;

		if (attnum == reqnatts)
			goto done;
	}

No NULL checks (they are guaranteed non-NULL), no alignment arithmetic (the offset is cached), no attbyval branch (guaranteed by-value). Just a load and a store.

Then the cached-offset loop, which is the same shape but does have to handle by-reference attributes:

 c
 
 
	/* We can use attcacheoff up until the first NULL */
	firstNonCacheOffsetAttr = Min(firstNonCacheOffsetAttr, firstNullAttr);

	/*
	 * Handle the portion of the tuple that we have cached the offset for up
	 * to the first NULL attribute. The offset is effectively fixed for
	 * these, so we can use the CompactAttribute's attcacheoff.
	 */
	if (attnum < firstNonCacheOffsetAttr)
	{
		int			attlen;

		do
		{
			isnull[attnum] = false;
			cattr = &cattrs[attnum];
			attlen = cattr->attlen;
			off = cattr->attcacheoff;
			values[attnum] = fetch_att_noerr(tp + off,
											 cattr->attbyval,
											 attlen);
			attnum++;
		} while (attnum < firstNonCacheOffsetAttr);

		/*
		 * Point the offset after the end of the last attribute with a cached
		 * offset. We expect the final cached offset attribute to have a
		 * fixed width, so just add the attlen to the attcacheoff
		 */
		Assert(attlen > 0);
		off += attlen;
	}

And then the two variable-width loops:

 c
 
 
	/*
	 * Handle any portion of the tuple that doesn't have a fixed offset up
	 * until the first NULL attribute. This loop only differs from the one
	 * after it by the NULL checks.
	 */
	for (; attnum < firstNullAttr; attnum++)
	{
		int			attlen;

		isnull[attnum] = false;
		cattr = &cattrs[attnum];
		attlen = cattr->attlen;

		/*
		 * Only emit the cstring-related code in align_fetch_then_add() when
		 * cstring support is needed. We assume support_cstring will be
		 * passed as a const to allow the compiler to eliminate this branch.
		 */
		if (!support_cstring)
			pg_assume(attlen > 0 || attlen == -1);

		/* align 'off', fetch the datum, and increment off beyond the datum */
		values[attnum] = align_fetch_then_add(tp,
											 &off,
											 cattr->attbyval,
											 attlen,
											 cattr->attalignby);
	}

	/*
	 * Now handle any remaining attributes in the tuple up to the requested
	 * attnum. This time, include NULL checks as we're now at the first NULL
	 * attribute.
	 */
	for (; attnum < natts; attnum++)
	{
		int			attlen;

		if (isnull[attnum])
		{
			values[attnum] = (Datum) 0;
			continue;
		}

		cattr = &cattrs[attnum];
		attlen = cattr->attlen;

		/* As above, only emit cstring code when needed. */
		if (!support_cstring)
			pg_assume(attlen > 0 || attlen == -1);

		/* align 'off', fetch the datum, and increment off beyond the datum */
		values[attnum] = align_fetch_then_add(tp,
											 &off,
											 cattr->attbyval,
											 attlen,
											 cattr->attalignby);
	}

The isnull[] array is consulted in the last loop — but it is a bool array that was populated eight bytes at a time by the SWAR trick, so the check is a byte load rather than a bit extraction.

align_fetch_then_add() is the workhorse that does align, fetch, and advance in one go:

 c
 
 
static inline Datum
align_fetch_then_add(const char *tupptr, uint32 *off, bool attbyval, int attlen,
					 uint8 attalignby)
{
	Datum		res;

	if (attlen > 0)
	{
		const char *offset_ptr;

		*off = TYPEALIGN(attalignby, *off);
		offset_ptr = tupptr + *off;
		*off += attlen;
		if (attbyval)
		{
			switch (attlen)
			{
				case sizeof(char):
					return CharGetDatum(*((const char *) offset_ptr));
				case sizeof(int16):
					return Int16GetDatum(*((const int16 *) offset_ptr));
				case sizeof(int32):
					return Int32GetDatum(*((const int32 *) offset_ptr));
				default:

					/*
					 * populate_compact_attribute_internal() should have
					 * checked
					 */
					Assert(attlen == sizeof(int64));
					return Int64GetDatum(*((const int64 *) offset_ptr));
			}
		}
		return PointerGetDatum(offset_ptr);
	}
	else if (attlen == -1)
	{
		if (!VARATT_IS_SHORT(tupptr + *off))
			*off = TYPEALIGN(attalignby, *off);

		res = PointerGetDatum(tupptr + *off);
		*off += VARSIZE_ANY(DatumGetPointer(res));
		return res;
	}
	else
	{
		Assert(attlen == -2);
		*off = TYPEALIGN(attalignby, *off);
		res = PointerGetDatum(tupptr + *off);
		*off += strlen(tupptr + *off) + 1;
		return res;
	}
}

Note the varlena case: VARATT_IS_SHORT() is checked before aligning, because a short varlena (1-byte header) needs no alignment at all. That is the same rule that governs tuple formation, so the deform path mirrors the form path exactly.

The bulk version

heap_deform_tuple() in src/backend/access/common/heaptuple.c is the non-incremental version, used when all attributes are wanted at once. Its header comment explains when it beats a loop over heap_getattr():

 c
 
 
 *		Note that for pass-by-reference datatypes, the pointer placed
 *		in the Datum will point into the given tuple.
 *
 *		When all or most of a tuple's fields need to be extracted,
 *		this routine will be significantly quicker than a loop around
 *		heap_getattr; the loop will become O(N^2) as soon as any
 *		noncacheable attribute offsets are involved.

The O(N²) is the point: fetching each attribute separately re-walks the prefix each time.

The opt-in: TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS

The guaranteed-attribute optimisation is not always safe. Some code deforms tuples before the NOT NULL constraints have been checked — for instance, when the executor is building a tuple that will be inserted, or when it is reading a row that might violate the constraint. In those cases “guaranteed to exist” is a lie.

So it is opt-in, via a slot flag:

 c
 
 
/*
 * true = slot's formed tuple guaranteed to not have NULLs in NOT NULLable
 * columns.
 */
#define			TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS		(1 << 3)
#define TTS_OBEYS_NOT_NULL_CONSTRAINTS(slot) \
	(((slot)->tts_flags & TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS) != 0)

and the slot caches the value:

 c
 
 
	int			tts_first_nonguaranteed;	/* The value from the TupleDesc's
											 * firstNonGuaranteedAttr, or 0
											 * when tts_flags does not contain
											 * TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS */

The flag is only set by the scan nodes, which read tuples that are known to satisfy the relation’s constraints:

src/backend/executor/nodeSeqscan.c
src/backend/executor/nodeIndexscan.c
src/backend/executor/nodeBitmapHeapscan.c
src/backend/executor/nodeTidscan.c
src/backend/executor/nodeTidrangescan.c
src/backend/executor/nodeSamplescan.c

and the slot initialisation decides whether to use it:

 c
 
 
		/*
		 * Precalculate the maximum guaranteed attribute that has to exist in
		 * every tuple which gets deformed into this slot. When the
		 * TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS flag is enabled, we simply take
		 * the pre-calculated value from the tupleDesc, otherwise the
		 * optimization is disabled, and we set the value to 0.
		 */
		if ((flags & TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS) != 0)
			slot->tts_first_nonguaranteed = tupleDesc->firstNonGuaranteedAttr;
		else
			slot->tts_first_nonguaranteed = 0;

The commit message explains the whole design:

Some locations in the code deform tuples that have yet to go through NOT NULL constraint validation. We’re unable to perform the guaranteed attribute optimization when that’s the case. This optimization is opt-in via the TupleTableSlot using the TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS flag.

The JIT path: sibling call optimisation

There is one more piece of the story, and it is about the JIT. The JIT compiler cannot inline arbitrary C, so it calls back into the executor through a small set of exported functions. One of them is slot_getsomeattrs_int():

 c
 
 
/*
 * slot_getsomeattrs_int
 *		external function to call getsomeattrs() for use in JIT
 */
void
slot_getsomeattrs_int(TupleTableSlot *slot, int attnum)
{
	/* Check for caller errors */
	Assert(slot->tts_nvalid < attnum);	/* checked in slot_getsomeattrs */
	Assert(attnum > 0);

	/* Fetch as many attributes as possible from the underlying tuple. */
	slot->tts_ops->getsomeattrs(slot, attnum);

	/*
	 * Avoid putting new code here as that would prevent the compiler from
	 * using the sibling call optimization for the above function.
	 */
}

The commit that produced this is 4deecb52aff, “Allow sibling call optimization in slot_getsomeattrs_int()”, by David Rowley:

This changes the TupleTableSlotOps contract to make it so the getsomeattrs() function is in charge of calling slot_getmissingattrs().

Since this removes all code from slot_getsomeattrs_int() aside from the getsomeattrs() call itself, we may as well adjust slot_getsomeattrs() so that it calls getsomeattrs() directly. We leave slot_getsomeattrs_int() intact as this is still called from the JIT code.

A sibling call is a tail call: the callee reuses the caller’s stack frame instead of pushing a new one. It matters here because the function being called is slot->tts_ops->getsomeattrs, which is a large, branch-heavy function. Without a tail call, the JIT’s call site would have to allocate a stack frame proportional to that function’s needs, and the frame could not be reused for the next call.

The whole point is visible in the comment: “Avoid putting new code here as that would prevent the compiler from using the sibling call optimization for the above function.” One statement, one tail call, no frame.

The contract change moved the slot_getmissingattrs() call into the getsomeattrs() implementation, so that the exported wrapper contains nothing but the call.

Caveats

TupleDescFinalize() must be called

This is the big one. The cache is not computed lazily; it is computed by an explicit call, and every code path that mutates a TupleDesc must call it again.

The header is unambiguous:

Once a TupleDesc has been populated, before it is used for any purpose, TupleDescFinalize() must be called on it.

and the code asserts it in several places:

 c
 
 
	/* Did someone forget to call TupleDescFinalize()? */
	Assert(tupleDesc->firstNonCachedOffsetAttr >= 0);

The trick is that firstNonCachedOffsetAttr is initialised to -1 in CreateTemplateTupleDesc():

 c
 
 
	/* This will be set to the correct value by TupleDescFinalize() */
	desc->firstNonCachedOffsetAttr = -1;
	desc->firstNonGuaranteedAttr = -1;

so “never finalized” is detectable by a negative value. BlessTupleDesc() checks it too:

 c
 
 
/*
 * BlessTupleDesc - make a completed tuple descriptor useful for SRFs
 *
 * Rowtype Datums returned by a function must contain valid type information.
 * This happens "for free" if the tupdesc came from a relcache entry, but
 * not if we have manufactured a tupdesc for a transient RECORD datatype.
 * In that case we have to notify typcache.c of the existence of the type.
 *
 * TupleDescFinalize() must be called on the TupleDesc before calling this
 * function.
 */
TupleDesc
BlessTupleDesc(TupleDesc tupdesc)
{
	/* Did someone forget to call TupleDescFinalize()? */
	Assert(tupdesc->firstNonCachedOffsetAttr >= 0);
	...
}

This was made explicit in a separate commit, 503620311e3, “Add all required calls to TupleDescFinalize()”, whose message says:

As of this commit all TupleDescs must have TupleDescFinalize() called on them once the TupleDesc is set up and before BlessTupleDesc() is called. … The follow-up commit which properly implements TupleDescFinalize() will cause any code which forgets to do this to fail in assert-enabled builds in BlessTupleDesc(). It may still be worth mentioning this change in the release notes so that extension authors update their code.

Extensions that build their own TupleDesc must call TupleDescFinalize(). This is an API change.

The cache is invalidated by any change to the descriptor

If anything modifies the FormData_pg_attribute array — adding a column, dropping one, changing a type — the cached offsets are stale and TupleDescFinalize() must be called again. The rule for the compact array is stated in the header:

Any code making changes manually to and fields in the FormData_pg_attribute array must subsequently call populate_compact_attribute() to flush the changes out to the corresponding ‘compact_attrs’ element.

and populate_compact_attribute() resets attcacheoff to -1, so a stale cache is not silently wrong — it becomes “unknown”, and the deformer falls back to walking. But firstNonCachedOffsetAttr and firstNonGuaranteedAttr would also need recomputing, which is TupleDescFinalize()’s job.

For catalogs, the mechanism is the relcache: ALTER TABLE invalidates the relcache entry, the relation’s TupleDesc is rebuilt from pg_attribute, and TupleDescFinalize() is called on it:

 c
 
	TupleDescFinalize(relation->rd_att);

That appears in src/backend/utils/cache/relcache.c in several places, and in src/backend/commands/tablecmds.c for the descriptor being built during ALTER TABLE.

Virtual generated columns break the offset cache

TupleDescFinalize() stops at a virtual generated column:

 c
 
 
		if (cattr->attlen <= 0 ||
			attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
			break;

because virtual generated columns are not stored in the tuple at all — they are computed on read. The offset cache would be wrong past that point.

There is a whole commit about this: 89eafad297a, “Fix tuple deforming with virtual generated columns”.

The guaranteed-attribute path is opt-in

Already covered. If the slot does not have TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS, tts_first_nonguaranteed is 0 and the whole first loop is skipped. The optimisation is only available on scan nodes that read committed tuples.

The `isnull` array must be rounded up

populate_isnull_array() writes 8 bytes at a time and the caller must have allocated the array accordingly. Getting this wrong would be a buffer overrun, so the requirement is spelled out in the function’s comment and formalised in the commit message. In MEMORY_CONTEXT_CHECKING builds, the palloc sentinel byte would catch an overflow.

`attcacheoff` is only a hint about a specific tuple layout

The cached offsets assume the tuple was formed by heap_form_tuple() with the same descriptor. That is true for tuples read from a relation. It is not true for tuples formed some other way — for example, a tuple formed for an index (IndexTuple), which has a different layout, or a tuple deformed from a different relation.

This is why index tuples have their own accessors (src/include/access/itup.h) and their own nocache_index_getattr(), even though the code is nearly identical:

 c
 
 
 *		This only gets called from fastgetattr(), in cases where we
 *		can't use the attcacheoff and the value is not null.
 *
 *		NOTE: if you need to change this code, see also heap_deform_tuple.
 *		Also see nocache_index_getattr, which is the same code for index
 *		tuples.

Index tuples store the same attributes but in an index-specific order and with index-specific types, so their descriptors have their own offsets.

The SWAR trick writes past the end of the attribute range

populate_isnull_array() processes whole bytes, so it may write isnull entries for attributes beyond natts. That is why the array must be padded to a multiple of 8 — and why heap_deform_tuple() does not use it:

 c
 
 
		/*
		 * XXX: it'd be nice to use populate_isnull_array() here, but that
		 * requires that the isnull array's size is rounded up to the next
		 * multiple of 8. Doing that would require adjusting many locations
		 * that allocate the array.
		 */

The bulk heap_deform_tuple() still uses per-attribute att_isnull() checks, because its isnull array is sized exactly natts and there are “many locations that allocate the array” to fix.

What it all means in practice

The width of your table’s fixed-width prefix is the thing that matters. attcacheoff is defined for every column up to the first text, bytea, numeric, or other varlena column. A table with 40 columns where the first is text has no cached offsets at all; a table with 40 columns where the first is int has 39 of them.

A table of fixed-width columns is dramatically cheaper to read than it used to be. The cost of reading column N is O(1) instead of O(N).

NULL-heavy tables got a different win. The bitmap is now scanned a byte at a time to find the first NULL, and the isnull array is built with a SWAR multiply rather than 40 bit tests. The prefix before the first NULL is deformed with no NULL checks at all.

NOT NULL constraints are now a performance feature. A column declared NOT NULL (and validated) that is pass-by-value and fixed-width is guaranteed to exist, and the deformer can skip the NULL bitmap, the attribute count, and the by-value/by-reference branch entirely.

Wide tables with a text column early on are still slow. The cache stops at the first varlena, and everything past it is walked. If you have a table with a nullable text column at position 2 and 40 columns after it, the offsets for columns 3..40 are not cached.

Storage on Opensource internals

PostgreSQL Stores a Row

A row is not a record

Line pointers: the indirection that makes everything possible

The tuple header: 23 bytes

Five virtual fields in three physical ones

t_infomask

t_infomask2

The null bitmap and the 23/24 trick

The data area

Variable-length values: the four header formats

TOAST: when a row doesn’t fit

The TOAST table

UPDATE: the version chain

Visibility: the rule

HOT: updating without touching the index

Pruning: reclaiming the space

Two side structures

The free space map

The visibility map

Look at it yourself

What it all means in practice

PostgreSQL Locks Buffers Without LWLocks

What a content lock protects

The old world: an LWLock per buffer

The new world: one 64-bit word

The layout

The share-lock count is a counter, not a bit

The flag bits

The descriptor

Acquiring a content lock

The queue-self-then-retry algorithm

The public API

Lock ownership tracking

Releasing a content lock

The header lock is not an LWLock either

The share-exclusive level: hint bits and write-out

The lock-free clock sweep

The second chance, quantified

Caveats

This is a big, invasive change

The wait event taxonomy changed

A backend can only hold one lock at a time

The share-lock count is bounded by MAX_BACKENDS

Deadlock semantics are essentially unchanged

Platforms without 64-bit atomics

The state word is now a contention point

Local buffers are a different story

What it all means in practice

Further reading

PostgreSQL Computes Page Checksums Fast

Where the checksum lives

The algorithm: FNV-1a, with a twist

Why thirty-two hashes

The core loop

From 32 bits to 16

Where it is called from

Runtime dispatch: the AVX2 path

Why thirty-two, exactly?

Caveats

N_SUMS is part of the on-disk format

What the checksum does and does not cover

The cost is real

The “in progress” states are subtle

Sixteen bits is not a lot

Look at it yourself

What it all means in practice

Further reading

PostgreSQL Hashes Lookups

The shape of a dynahash table

The directory of segments

Two masks and a power of two

The partition trick

The entry layout

Growth, one bucket at a time

When expansion happens

Allocating entries in chunks

The 32 freelists

Borrowing from another freelist

Partitioned tables

Why the `PG_INT16_MAX` limit barely matters

The `isnull` array must be rounded up

`attcacheoff` is only a hint about a specific tuple layout