PostgreSQL's json and jsonb Implementation in C

Most databases hand you a JSON type and hide the bytes. PostgreSQL hands you two, and they are almost nothing alike. json is the honest one: it is the exact text you typed, validated and then stored as a text varlena. jsonb is the interesting one: it is a binary tree of tagged nodes, laid out so that you can walk into it without parsing, and compressed by TOAST when it gets large.

The surprise is that the two types share almost the same catalog entry, and the jsonb format is not “a struct you can memcpy”. It is a varlena whose payload is a sequence of 4-byte JEntry headers followed by the nodes’ data — a layout chosen so that random access stays O(1) while the whole thing still compresses.

This is a walk through both, based on reading the PostgreSQL source (commit e75fcc9867f).

Two types, nearly identical catalog rows

From src/include/catalog/pg_type.dat:

c
{ oid => '114', array_type_oid => '199', descr => 'JSON stored as text',
  typname => 'json', typlen => '-1', typbyval => 'f', typcategory => 'U',
  typinput => 'json_in', typoutput => 'json_out', typreceive => 'json_recv',
  typsend => 'json_send', typalign => 'i', typstorage => 'x' },

{ oid => '3802', array_type_oid => '3807', descr => 'Binary JSON',
  typname => 'jsonb', typlen => '-1', typbyval => 'f', typcategory => 'U',
  typsubscript => 'jsonb_subscript_handler', typinput => 'jsonb_in',
  typoutput => 'jsonb_out', typreceive => 'jsonb_recv', typsend => 'jsonb_send',
  typalign => 'i', typstorage => 'x' },

Both are varlena (typlen = -1, typbyval = f), 4-byte aligned (typalign = 'i'), and “extended” (typstorage = 'x' — compressible and TOASTable). The only structural difference in the catalog is typsubscript => 'jsonb_subscript_handler', a newer addition that lets you subscript a jsonb directly. Everything else about the difference lives in the C files.

json: text in disguise

json_in in src/backend/utils/adt/json.c is eleven lines, and the comment says it all:

c
Datum
json_in(PG_FUNCTION_ARGS)
{
	char	   *json = PG_GETARG_CSTRING(0);
	text	   *result = cstring_to_text(json);
	JsonLexContext lex;

	/* validate it */
	makeJsonLexContext(&lex, result, false);
	if (!pg_parse_json_or_errsave(&lex, &nullSemAction, fcinfo->context))
		PG_RETURN_NULL();

	/* Internal representation is the same as text */
	PG_RETURN_TEXT_P(result);
}

It lexes the input only to prove it is well-formed JSON, then stores the original text verbatim. json_out is the inverse: TextDatumGetCString(), no normalization at all. The consequences are the whole point of the type:

Whitespace is preserved. ' {"a":1} '::json comes back with the spaces.
Key order is preserved. '{"b":1,"a":2}'::json keeps b before a.
Duplicate keys are preserved. '{"a":1,"a":2}'::json keeps both.

json_recv/json_send do the same thing over the binary protocol, just with a pq_sendtext wrapper. There is no parser hook, no tree, no hash. json is a validated text.

jsonb: the on-disk format

jsonb is where the work is. The header file is src/include/utils/jsonb.h, and the I/O lives in src/backend/utils/adt/jsonb.c with the builder in src/backend/utils/adt/jsonb_util.c.

The top-level datum is a Jsonb, which is a varlena plus a JsonbContainer root:

c
/* The top-level on-disk format for a jsonb datum. */
typedef struct
{
	int32		vl_len_;		/* varlena header (do not touch directly!) */
	JsonbContainer root;
} Jsonb;

A container is a uint32 header followed by a flexible array of JEntry:

c
typedef struct JsonbContainer
{
	uint32		header;			/* number of elements or key/value pairs, and
								 * flags */
	JEntry		children[FLEXIBLE_ARRAY_MEMBER];
} JsonbContainer;

The header packs a 28-bit count (JB_CMASK) with three flag bits: JB_FSCALAR (0x10000000), JB_FOBJECT (0x20000000), and JB_FARRAY (0x40000000). A container is either an object or an array, and a top-level scalar is faked by wrapping it in a one-element array with JB_FSCALAR set.

Now the JEntry. It is a single uint32 that describes one child node:

c
typedef uint32 JEntry;

#define JENTRY_OFFLENMASK		0x0FFFFFFF
#define JENTRY_TYPEMASK			0x70000000
#define JENTRY_HAS_OFF			0x80000000

#define JENTRY_ISSTRING			0x00000000
#define JENTRY_ISNUMERIC		0x10000000
#define JENTRY_ISBOOL_FALSE		0x20000000
#define JENTRY_ISBOOL_TRUE		0x30000000
#define JENTRY_ISNULL			0x40000000
#define JENTRY_ISCONTAINER		0x50000000	/* array or object */

The low 28 bits are either a length or an offset. The next three bits are the type. The high bit says which of the two the low bits mean.

The root node is the exception. It has no parent container to hold its JEntry, so no header is stored for it — the JsonbContainer header’s JB_FOBJECT / JB_FARRAY bits are how you tell what it is. The comment in jsonb.h spells this out: “the root node is an exception; it has no parent array or object that could hold its JEntry.”

For a small object like {"a":"b"} the bytes look like this:

text

varlena header        (4 bytes: total size)
root header           uint32 = 1 | JB_FOBJECT   -> 0x20000001
JEntry[0]  key "a"    0x00000001   (string, length 1)
JEntry[1]  value "b"  0x00000001   (string, length 1)
data                  'a' 'b'

Note that the JEntrys and the variable-length data are not interleaved. The container holds all its children’s JEntrys first, then the data. That is what makes it possible to scan the header array without touching the payload.

The offset-or-length compromise

The 28-bit field can hold a length or an offset. Why both? The header comment explains the design tension directly:

In the initial design each JEntry always stored an offset, but this resulted in JEntry arrays with horrible compressibility properties, so that TOAST compression of a JSONB did not work well. Storing only lengths would greatly improve compressibility, but it makes random access into large arrays expensive (O(N) not O(1)).

The resolution is JB_OFFSET_STRIDE:

#define JB_OFFSET_STRIDE		32

Every 32nd JEntry stores an absolute end-offset (JENTRY_HAS_OFF set); the other 31 store lengths. getJsonbOffset() walks backwards to the nearest offset-bearing entry and sums the lengths in between, so any node is reachable in at most 32 steps — O(1) no matter how large the container, while the resulting array stays repetitive enough for TOAST’s compression to bite. The macro is documented as a creation-time heuristic only: readers must honor the HAS_OFF bits rather than assuming a fixed stride, so the heuristic can change without breaking on-disk compatibility.

Building a jsonb: parse, then flatten

Input goes through jsonb_from_cstring in jsonb.c. It wires up a JsonSemAction — a set of parser callbacks — and hands it to the generic JSON parser:

c
	sem.object_start = jsonb_in_object_start;
	sem.array_start = jsonb_in_array_start;
	sem.object_end = jsonb_in_object_end;
	sem.array_end = jsonb_in_array_end;
	sem.scalar = jsonb_in_scalar;
	sem.object_field_start = jsonb_in_object_field_start;

	if (!pg_parse_json_or_errsave(&lex, &sem, escontext))
		return (Datum) 0;

	PG_RETURN_POINTER(JsonbValueToJsonb(state.result));

Those callbacks build an in-memory JsonbValue tree (the union-typed, deserialized representation declared in jsonb.h). JsonbValueToJsonb then flattens it into the varlena. For a scalar, it wraps the value in a raw-scalar pseudo-array so the root always has a container header.

The flattening is convertToJsonb() and its helpers in jsonb_util.c. It reserves room for the varlena header, then serializes recursively:

c
static Jsonb *
convertToJsonb(JsonbValue *val)
{
	StringInfoData buffer;
	...
	initStringInfo(&buffer);
	reserveFromBuffer(&buffer, VARHDRSZ);
	convertJsonbValue(&buffer, &jentry, val, 0);
	...
	res = (Jsonb *) buffer.data;
	SET_VARSIZE(res, buffer.len);
	return res;
}

convertJsonbArray aligns to 4 bytes, writes nElems | JB_FARRAY, reserves sizeof(JEntry) * nElems, recurses over each element, and converts every JB_OFFSET_STRIDE‘th entry from a length into an offset:

c
	containerhead = nElems | JB_FARRAY;
	appendToBuffer(buffer, &containerhead, sizeof(uint32));
	jentry_offset = reserveFromBuffer(buffer, sizeof(JEntry) * nElems);
	...
		if ((i % JB_OFFSET_STRIDE) == 0)
			meta = (meta & JENTRY_TYPEMASK) | totallen | JENTRY_HAS_OFF;

convertJsonbObject does the same but writes all the keys first, then all the values — two JEntrys per pair. That is not an accident: the header comment notes that keys sorted and grouped first “keeps the keys compact in memory, making a search for a particular key more cache-friendly.”

The scalars are cheap. convertJsonbScalar handles each type:

jbvNull → *header = JENTRY_ISNULL and no data bytes at all.
jbvString → raw bytes plus the length in the header.
jbvNumeric → pad to 4-byte alignment, then the Postgres numeric varlena; the header is JENTRY_ISNUMERIC | (padlen + numlen).
jbvBool → JENTRY_ISBOOL_TRUE or JENTRY_ISBOOL_FALSE, no data.

Storing numbers as numeric is the reason '1e100'::jsonb comes back as the full 101-digit expansion instead of a float. There is no IEEE-754 anywhere in the format.

Object keys: sorted, deduped, last-wins

Before an object is serialized, uniqueifyJsonbObject sorts its pairs with lengthCompareJsonbPair and removes duplicates:

c
	if (nPairs > 1)
		qsort_arg(pairs, nPairs, sizeof(JsonbPair),
				  lengthCompareJsonbPair, &hasNonUniq);

The comparator is length-wise, not lexical — lengthCompareJsonbString compares lengths first and only memcmps on a tie. For keys of different lengths the shorter one sorts first, which is why {"bb":1,"a":2} prints with a before bb. When two keys are equal, the pair with the higher order field sorts first, and the dedupe loop keeps the first, so the last key observed wins. jsonb_in passes unique_keys = false, so '{"a":1,"a":2}'::jsonb silently becomes {"a": 2}.

Reading it back

Output is jsonb_out → JsonbToCString → JsonbToCStringWorker. It does not walk the JEntry array directly; it uses a JsonbIterator, which yields a stream of tokens (WJB_BEGIN_OBJECT, WJB_KEY, WJB_VALUE, WJB_ELEM, …). The worker assembles the text from those tokens. That iterator is also the reason jsonb is pleasant to manipulate in C: jsonb_set, jsonb_insert, jsonb_path_query and friends all consume the same token stream rather than re-deriving the layout.

Look at it yourself

Everything above is observable from SQL:

sql
-- json keeps the bytes you gave it
SELECT '  {"b":1,"a":2}  '::json;   -- '  {"b":1,"a":2}  '

-- jsonb re-serializes: keys sorted length-wise, no whitespace
SELECT '{"b":1,"a":2}'::jsonb;      -- {"a": 2, "b": 1}

-- duplicate keys: last one wins, silently
SELECT '{"a":1,"a":2}'::jsonb;      -- {"a": 2}

-- numbers are exact numeric, not float
SELECT '1e100'::jsonb;              -- 10000...000

-- you can watch the type identity
SELECT pg_typeof('{"a":1}'::jsonb), pg_typeof('{"a":1}'::json);

If you want to see the actual JEntry bytes, jsonb is a varlena, so a pageinspect raw dump of a heap tuple will show the JEntry array and the string payload sitting right after the container header.

What it all means

json is not parsed into anything. It is a validated text. If you round-trip it, you get your whitespace, your key order, and your duplicate keys back. If you want a canonical value, use jsonb.
jsonb trades input text for a searchable tree. It costs a parse and a flatten on the way in, and it gives you sorted keys, deduplicated objects, and exact numeric storage.
The JEntry layout is a compression-aware random-access structure. The 28-bit length/offset split plus JB_OFFSET_STRIDE is the specific compromise that keeps both TOAST compression and O(1) child lookup working.
Numbers stay exact. Because jsonb embeds Postgres numeric varlenas, there is no float rounding and no 1e100 surprise.
The root is special on purpose. No JEntry for the root means the container header has to carry the object/array/scalar distinction — and it does, in three flag bits.