Duckdb on Opensource internals

The Validity Mask Is Optional: A Null Pointer Means No NULLs

Jay Hubbert — Thu, 22 Oct 2026 10:40:00 +0000

TL;DR: Every DuckDB vector carries a NULL bitmap — but the bitmap is lazily allocated. A null validity_mask pointer is the encoding of “this column has no NULLs”. RowIsValid short-circuits to true without touching memory, SetValid is a no-op, and only SetInvalid pays for an allocation.

The payoff is not just memory. The aggregate hash table asks CannotHaveNull() once, at construction, and if the group columns can never be NULL it swaps the row-match predicate from COMPARE_NOT_DISTINCT_FROM to plain COMPARE_EQUAL — changing the inner loop of every probe in the table.

The cheapest possible representation of “no NULLs”

Consider two tables with identical data:

 sql
 
 
CREATE TABLE a (id INTEGER NOT NULL, val INTEGER);
CREATE TABLE b (id INTEGER, val INTEGER);

Column a.id can never be NULL. Column b.id might be. As far as the values are concerned, the two are identical when b.id happens to have no NULLs in it.

The obvious design is to always carry a bitmap: one bit per row, 1 meaning “not null”. For a 2048-row vector of int64_t that is 16 KiB of data plus 256 bytes of bitmap — a 1.5% overhead, paid whether or not anyone needs it.

DuckDB’s design is better: the absence of the bitmap is the information. A ValidityMask is a uint64_t* plus a shared_ptr to the allocation. If the pointer is null, the mask says “all rows are valid” and there is nothing to store.

The type

 cpp
 
 
// src/include/duckdb/common/types/validity_mask.hpp:64
	static constexpr const idx_t BITS_PER_VALUE = ValidityBuffer::BITS_PER_VALUE;
	static constexpr const idx_t STANDARD_ENTRY_COUNT = (STANDARD_VECTOR_SIZE + (BITS_PER_VALUE - 1)) / BITS_PER_VALUE;
	static constexpr const idx_t STANDARD_MASK_SIZE = STANDARD_ENTRY_COUNT * sizeof(V);

validity_t is uint64_t, so BITS_PER_VALUE is 64. A standard vector of 2048 rows needs exactly 32 words — 256 bytes, four cache lines. The (BITS_PER_VALUE - 1) in the ceiling division is there for the ragged tail: a vector of, say, 100 rows needs two words, not one.

And the two predicates that the rest of the engine branches on:

 cpp
 
 
// src/include/duckdb/common/types/validity_mask.hpp:82
	inline bool CannotHaveNull() const {
		return !validity_mask;
	}
	inline bool CanHaveNull() const {
		return validity_mask;
	}

CannotHaveNull() is the interesting one. It is a null-pointer check, not a scan. It is O(1), branch-predictable, and it is the answer to “does this column need a NULL check at all?”.

Reading: the short circuit

 cpp
 
 
// src/include/duckdb/common/types/validity_mask.hpp:197
	//! Returns true if a row is valid (i.e. not null), false otherwise
	inline bool RowIsValid(idx_t row_idx) const {
#ifdef DEBUG
		if (row_idx >= capacity) {
			throw InternalException("ValidityMask::RowIsValid - row_idx %d is out-of-range for mask with capacity %llu",
			 row_idx, capacity);
		}
#endif
		if (!validity_mask) {
			return true;
		}
		return RowIsValidUnsafe(row_idx);
	}

Two branches. In a release build the first is gone entirely, and the second is if (!validity_mask) return true;.

This is the pattern to internalise: the check for “is there a mask?” is hoisted out of the per-row loop, but it is written once, in one place, and every caller gets it for free. A kernel that calls RowIsValid per row does not know whether it is paying for a mask read or not — but the branch predictor will, after the first few rows, nail the !validity_mask case completely, because it is the same answer every time.

Compare that with the alternative: allocate the mask, fill it with all-ones, and read a bit per row. Same answer, but 256 bytes of memory traffic per vector and one extra load in the dependency chain.

Writing: only one direction costs anything

Writing is asymmetric, and deliberately so.

Marking a row valid is free, because the default state is already “valid”:

 cpp
 
 
// src/include/duckdb/common/types/validity_mask.hpp:219
	//! Marks the entry at the specified row index as valid (i.e. not-null)
	inline void SetValid(idx_t row_idx) {
#ifdef DEBUG
		if (row_idx >= capacity) {
			throw InternalException("ValidityMask::SetValid - row_idx %d is out-of-range for mask with capacity %llu",
			 row_idx, capacity);
		}
#endif
		if (!validity_mask) {
			// if CannotHaveNull() we don't need to do anything
			// the row is already valid
			return;
		}
		SetValidUnsafe(row_idx);
	}

The comment says it plainly: “if CannotHaveNull() we don’t need to do anything / the row is already valid”.

Marking a row invalid is the only operation that allocates:

 cpp
 
 
// src/include/duckdb/common/types/validity_mask.hpp:248
	//! Marks the entry at the specified row index as invalid (i.e. null)
	inline void SetInvalid(idx_t row_idx) {
#ifdef DEBUG
		if (row_idx >= capacity) {
			throw InternalException("ValidityMask::SetInvalid - row_idx %d is out-of-range for mask with capacity %llu",
			 row_idx, capacity);
		}
#endif
		if (!validity_mask) {
			Initialize(capacity);
		}
		SetInvalidUnsafe(row_idx);
	}

Initialize(capacity) allocates the mask and — crucially — fills it with all-ones first, so that the other 2047 rows are correctly marked valid. One allocation, and only when a NULL actually shows up.

So the lifecycle is: start with no mask, stay with no mask until someone writes a NULL, allocate once at that moment, and keep the mask thereafter. A column that is NOT NULL never allocates a mask at all.

And you can always go back:

 cpp
 
 
// src/include/duckdb/common/types/validity_mask.hpp:144
	inline void Reset(idx_t target_count_p = STANDARD_VECTOR_SIZE) {
		validity_mask = nullptr;
		validity_data.reset();
		capacity = target_count_p;
	}

Reset frees the mask and sets the pointer back to null. “This column has no NULLs” is a state you can return to.

Where it pays off: the aggregate hash table

Here is where the trick stops being a memory optimisation and becomes a query optimisation.

DuckDB’s hash aggregate stores groups in a TupleDataLayout — a columnar layout with its own validity bytes per row. When it builds the table, it has to decide how to compare a probe row against a stored row. There are two candidate semantics:

COMPARE_EQUAL — plain equality. NULL = NULL is false, and a NULL never equals anything.
COMPARE_NOT_DISTINCT_FROM — SQL’s IS NOT DISTINCT FROM. NULL = NULL is true.

If the group columns can contain NULLs, you must use the second: GROUP BY x puts all the NULLs into one group, and a probe row with x = NULL has to find that group.

But if the group columns can never contain NULLs, the two semantics coincide, and you can use the much cheaper plain equality. DuckDB decides this once, at table construction time:

 cpp
 
 
// src/execution/aggregate_hashtable.cpp:86
	// Predicates
	const auto expr_type = (layout_ptr->CannotHaveNull() && !layout_ptr->HasNestedTypes())
	 ? ExpressionType::COMPARE_EQUAL
	 : ExpressionType::COMPARE_NOT_DISTINCT_FROM;
	predicates.resize(layout_ptr->ColumnCount() - 1, expr_type);
	row_matcher.Initialize(true, *layout_ptr, predicates);

One ternary, evaluated once, that changes the comparison function used by every probe of the table, forever.

What does that buy? Look at the comparison operators:

 cpp
 
 
// src/include/duckdb/common/operator/comparison_operators.hpp:105
struct DistinctFrom {
	template <class T>
	static inline bool Operation(const T &left, const T &right, bool left_null, bool right_null) {
		if (left_null || right_null) {
			return left_null != right_null;
		}
		return NotEquals::Operation(left, right);
	}
};

struct NotDistinctFrom {
	template <class T>
	static inline bool Operation(const T &left, const T &right, bool left_null, bool right_null) {
		return !DistinctFrom::Operation(left, right, left_null, right_null);
	}
};

NotDistinctFrom is !DistinctFrom, and DistinctFrom has an extra branch on the two null flags plus a negation on top. Equals has neither. So the cheap predicate is one fewer branch and one fewer boolean inversion, on every column of every group key, for every probe row.

And DuckDB pushes the decision even further down, into the template. The row matcher specialises on four combinations of “left side all valid” and “right side all valid”:

 cpp
 
 
// src/common/row_operations/row_matcher.cpp:64
template <bool NO_MATCH_SEL, class T, class OP>
static idx_t TemplatedMatch(Vector &, const TupleDataVectorFormat &lhs_format, SelectionVector &sel, const idx_t count,
 const TupleDataLayout &rhs_layout, Vector &rhs_row_locations, const idx_t col_idx,
 const vector<MatchFunction> &, SelectionVector *no_match_sel, idx_t &no_match_count) {
#if DUCKDB_SMALLER_BINARY(row_matcher_validity)
	return TemplatedMatchLoop<NO_MATCH_SEL, T, OP>(lhs_format, sel, count, rhs_layout, rhs_row_locations, col_idx,
	 no_match_sel, no_match_count);
#else
	if (lhs_format.unified.validity.CannotHaveNull()) {
		if (rhs_layout.CannotHaveNull()) {
			return TemplatedMatchLoop<NO_MATCH_SEL, T, OP, true, true>(
			 lhs_format, sel, count, rhs_layout, rhs_row_locations, col_idx, no_match_sel, no_match_count);
		} else {
			return TemplatedMatchLoop<NO_MATCH_SEL, T, OP, true, false>(
			 lhs_format, sel, count, rhs_layout, rhs_row_locations, col_idx, no_match_sel, no_match_count);
		}
	} else {
		if (rhs_layout.CannotHaveNull()) {
			return TemplatedMatchLoop<NO_MATCH_SEL, T, OP, false, true>(
			 lhs_format, sel, count, rhs_layout, rhs_row_locations, col_idx, no_match_sel, no_match_count);
		} else {
			return TemplatedMatchLoop<NO_MATCH_SEL, T, OP, false, false>(
			 lhs_format, sel, count, rhs_layout, rhs_row_locations, col_idx, no_match_sel, no_match_count);
		}
	}
#endif
}

This is the payoff in its full glory. The runtime fact “this column has no NULLs” is turned into a compile-time template parameter, and the inner loop then contains literally zero NULL checks:

 cpp
 
 
// src/common/row_operations/row_matcher.cpp:49
		if (COMPARISON_OP::template Operation<T>(
		 lhs_data[lhs_idx], Load<T>(rhs_location + rhs_offset_in_row),
		 LHS_ALL_VALID ? false : !lhs_validity.RowIsValidUnsafe(lhs_idx),
		 RHS_ALL_VALID ? false
		 : !ValidityBytes::RowIsValid(
		 ValidityBytes(rhs_location, rhs_column_count).GetValidityEntryUnsafe(entry_idx),
		 idx_in_entry))) {

LHS_ALL_VALID ? false : ... — when LHS_ALL_VALID is true as a template parameter, the compiler folds the whole expression away. RHS_ALL_VALID ? false : ... likewise. The ?: here is not a runtime branch; it is a compile-time one, because the left operand is a template parameter.

So the chain is:

CannotHaveNull() is a null-pointer test — O(1), no memory access.
It selects the comparison semantics once, at hash-table construction.
That semantics, plus the per-side validity flags, selects a template instantiation of the match loop.
The template instantiation has the NULL checks compiled out entirely.

A runtime fact became a compile-time fact, and the whole cost of it vanished.

The write side propagates the fact

For the above to work, “this column has no NULLs” has to survive being copied around. And it does, because copying a mask that does not exist is a no-op rather than a materialisation:

 cpp
 
 
// src/common/vector/flat_vector.cpp:527
void FlatVector::CopyValidity(Vector &target, const Vector &source, idx_t count) {
	if (source.GetVectorType() == VectorType::FLAT_VECTOR) {
		SetValidity(target, Validity(source));
		return;
	}
	auto &result_validity = ValidityMutable(target);
	if (source.GetVectorType() == VectorType::CONSTANT_VECTOR) {
		if (ConstantVector::IsNull(source)) {
			result_validity.SetAllInvalid(count);
		} else {
			result_validity.Reset(count);
		}
		return;
	}
	auto validity = source.Validity();
	if (!validity.CanHaveNull()) {
		result_validity.Reset(count);
		return;
	}
	for (idx_t r = 0; r < count; r++) {
		if (!validity.IsValid(r)) {
			result_validity.SetInvalid(r);
		}
	}
}

result_validity.Reset(count) — not “allocate a mask and set every bit”. Reset sets the pointer to null. So a NULL-free column propagates its NULL-freeness downstream instead of turning into a 256-byte all-ones bitmap that every downstream kernel would then have to read.

The same thing happens in the validity mask’s own Copy:

 cpp
 
 
// src/include/duckdb/common/types/validity_mask.hpp:351
	inline void Copy(const TemplatedValidityMask &other, idx_t count) {
		capacity = count;
		if (other.CannotHaveNull()) {
			validity_data = nullptr;
			validity_mask = nullptr;
		} else {
			validity_data = make_buffer<ValidityBuffer>(other.validity_mask, count);
			validity_mask = validity_data->owned_data.get();
		}
	}

The same idea, one layer down: MVCC deletes

This “absence encodes meaning” pattern shows up again in DuckDB’s transaction layer, and it is worth looking at because it is the same trick applied to a harder problem.

DuckDB’s MVCC stores, per row, the transaction id that inserted it and the transaction id that deleted it. Storing two uint64_ts per row for a billion rows is 16 GB of metadata, so the representation has to degrade gracefully. It does so with three states:

 cpp
 
 
// src/include/duckdb/storage/table/chunk_info.hpp:47
//! The storage state of the delete side of a ChunkVectorInfo. Exhaustive and mutually exclusive:
//! CONSTANT - all rows share constant_delete_id (NOT_DELETED_ID for none-deleted, or a single delete id)
//! MASKED - partially deleted, all deleted rows share one committed id (mask_delete_id): which rows are
//! deleted is stored in deleted_mask, deleted_data freed
//! ARRAY - per-row delete ids materialized in deleted_data
enum class DeleteIdState : uint8_t { CONSTANT, MASKED, ARRAY };

Read the CONSTANT case again: “all rows share constant_delete_id (NOT_DELETED_ID for none-deleted)”. A single sentinel value means “not deleted” — the absence of a per-row array is the information, exactly like a null validity pointer.

And the MASKED state is the same idea as the validity mask: a bitmap of deleted rows plus one shared transaction id, valid because every deleted row in the vector was deleted by the same committed transaction.

The scan over that mask is word-at-a-time:

 cpp
 
 
// src/storage/table/chunk_info.cpp:104
	case DeleteIdState::MASKED: {
		// every deleted row shares mask_delete_id and alive rows are NOT_DELETED_ID (never deleted), so the
		// delete decision is a single constant for the whole vector
		const bool masked_deleted = DELETE_OP::IsDeleted(view, mask_delete_id);
		if (HasConstantInsertionId()) {
			if (!INSERT_OP::UseInsertedVersion(view, ConstantInsertId())) {
				return 0;
			}
			if (!masked_deleted) {
				// the delete is not visible to this transaction - every row is visible
				return max_count;
			}
			// only the alive (mask-invalid) rows are visible
			if (!sel_vector) {
				return max_count - deleted_mask.CountValid(max_count);
			}
			// scan the mask a word at a time: skip fully-deleted words, take fully-alive words wholesale,
			// only extract bits for mixed words
			idx_t count = 0;
			const idx_t entry_count = ValidityMask::EntryCount(max_count);
			for (idx_t entry_idx = 0; entry_idx < entry_count; entry_idx++) {
				auto entry = deleted_mask.GetValidityEntry(entry_idx);
				if (ValidityMask::AllValid(entry)) {
					// every row in this word is deleted - skip
					continue;
				}
				const idx_t base = entry_idx * ValidityMask::BITS_PER_VALUE;
				const idx_t entry_end = MinValue<idx_t>(base + ValidityMask::BITS_PER_VALUE, max_count);
				if (ValidityMask::NoneValid(entry)) {
					// every row in this word is alive - select them all
					for (idx_t i = base; i < entry_end; i++) {
						sel_vector->set_index(count++, i);
					}
					continue;
				}
				for (idx_t i = base; i < entry_end; i++) {
					if (!ValidityMask::RowIsValid(entry, i - base)) {
						sel_vector->set_index(count++, i);
					}
				}
			}
			return count;
		}

Compare this loop with the one in the scalar executor, which walks the validity mask 64 rows at a time and skips all-ones or all-zeros words wholesale. It is the same algorithm applied to the same data structure. DuckDB reuses ValidityMask as its bitmap type everywhere — for NULLs, for deleted rows, for row-group validity — precisely because the word-at-a-time scan is such a good fit.

And the state machine collapses over time. CompressDeleteToMask folds a per-row id array into a mask plus one shared id:

 cpp
 
 
// src/storage/table/chunk_info.cpp:336
void ChunkVectorInfo::CompressDeleteToMask(transaction_t mask_id) {
	D_ASSERT(delete_state == DeleteIdState::ARRAY);
	// the mask can only carry a single committed id shared by every deleted row
	D_ASSERT(IsCommitted(mask_id));
	// start all-valid (== all deleted), then mark the alive rows invalid
	deleted_mask.Initialize(STANDARD_VECTOR_SIZE);
	{
		auto segment = allocator.GetHandle(deleted_data);
		auto deleted = segment.GetPtr<transaction_t>();
		for (idx_t i = 0; i < STANDARD_VECTOR_SIZE; i++) {
			if (deleted[i] == NOT_DELETED_ID) {
				deleted_mask.SetInvalid(i);
			}
		}
	} // release the read handle before freeing the buffer
	allocator.Free(deleted_data);
	deleted_data = IndexPointer();
	mask_delete_id = mask_id;
	delete_state = DeleteIdState::MASKED;
}

The per-row array is freed. The mask replaces it. The NOT_DELETED_ID sentinel is what made the collapse possible in the first place — one sentinel value, and “which rows are alive” becomes a bitmap.

Seeing it yourself

 sql
 
 
CREATE TABLE strict (id INTEGER NOT NULL, val INTEGER);
CREATE TABLE loose (id INTEGER, val INTEGER);

INSERT INTO strict SELECT range, range % 10 FROM range(100000);
INSERT INTO loose SELECT range, range % 10 FROM range(100000);

-- the aggregate on the NOT NULL column builds a layout whose group columns
-- CannotHaveNull(), so the hash table uses COMPARE_EQUAL for every probe.
SELECT val, count(*) FROM strict GROUP BY val;
SELECT val, count(*) FROM loose GROUP BY val;

-- and once a NULL shows up, the mask is allocated and the semantics change
INSERT INTO loose VALUES (NULL, 99);
SELECT val, count(*) FROM loose GROUP BY val;

The plans look identical. The generated code does not: strict’s group-by has no NULL checks in its probe loop at all, and loose’s does (until the NULL arrives, at which point loose behaves like strict for the rows that are clean and pays the price for the ones that are not).

You can also see the “no NULLs” fact survive a copy:

 sql
 
 
-- the filter produces a dictionary vector; the validity mask is a view,
-- and if nothing was NULL there is nothing to view.
SELECT count(*) FROM strict WHERE id > 10;

What it costs

A branch in the hot path. RowIsValid has an if (!validity_mask) check. It is perfectly predicted, but it is a branch.
One allocation, once. The first SetInvalid on a mask allocates 32 words. If that happens for every vector, you have paid the allocation cost for the “all rows valid” case too — but you have not paid the 256-byte read cost in the common case where the mask is absent.
Loss of the “no NULLs” fact once a NULL appears. A vector with a single NULL in it needs the full mask for all 2048 rows. The mask is all-or-nothing per vector.
The CannotHaveNull() decision is only as good as the layout. If the aggregate’s group columns are nullable, DuckDB must use COMPARE_NOT_DISTINCT_FROM for the whole table, because one NULL anywhere forces the semantics for every row. This is a real trade: a single nullable group column costs the entire table the fast path.
The MVCC version has the same shape. MASKED requires every deleted row to share one committed transaction id. A delete that spans two concurrent transactions cannot use it and stays in ARRAY state until compression can prove otherwise.

The bottom line

The general lesson is that absence can be cheaper than presence, if you design the absence to mean something.

In DuckDB, a null validity_mask pointer means “all rows are valid”. That one convention buys:

Zero bytes of bitmap for a NOT NULL column, and zero mask reads in every kernel that touches it.
A compile-time decision in the aggregate hash table: CannotHaveNull() is checked once and turns into a template parameter that erases the NULL check from the probe loop.
Propagation: copying a NULL-free column keeps it NULL-free instead of materialising an all-ones mask.

And the same convention, applied to MVCC delete state, lets a chunk’s worth of per-row transaction ids collapse into one bitmap and one shared id — freeing the array entirely.

The design rule: when a piece of metadata is usually uniform, do not store it. Store the exception, and let the absence of the exception carry the common case.

The Auxiliary Buffer: How a DuckDB Vector Owns Memory It Never Allocated

Jay Hubbert — Thu, 22 Oct 2026 10:10:00 +0000

TL;DR: A VectorBuffer owns its data bytes. But it can also own other people’s bytes without copying them, by stashing a polymorphic AuxiliaryDataHolder in an AuxiliaryDataSet. The star example: slicing a vector is a pointer bump (data_ptr + type_size * offset) plus a shared_ptr to the parent, so the parent cannot die while the child points into it. No memcpy, no dangling slice — the lifetime is enforced by the type system.

The same mechanism keeps string heaps, pinned storage blocks, and selection-vector data alive across every copy, slice and filter in the engine.

The 16 KiB copy you don’t have to do

Here is a situation the engine hits constantly.

A scan produces a chunk of 2048 rows. A filter runs over it. The filter’s output is the subset of rows that passed — say 37 of them. Downstream operators need a chunk containing those 37 rows.

The naive implementation copies 37 rows into a fresh buffer. That is fine. Now consider what actually happens in a real query: WHERE clauses stack, projections narrow columns, joins combine chunks, and every one of those steps produces a sub-range of the previous chunk. If every step copies, a query over a billion rows does a billion rows’ worth of memcpy, and the engine spends its time in memcpy instead of in the user’s aggregation.

DuckDB’s answer: a sub-range of a chunk is just a chunk whose data pointer starts later. Here is the entire implementation:

 cpp
 
 
// src/common/vector/flat_vector.cpp:97
buffer_ptr<VectorBuffer> StandardVectorBuffer::SliceInternal(const LogicalType &type, idx_t offset, idx_t end) {
	D_ASSERT(end <= capacity);
	auto count = count_t(end - offset);
	auto offset_ptr = data_ptr + type_size * offset;
	auto result = make_buffer<StandardVectorBuffer>(offset_ptr, count, type_size);
	result->GetValidityMask().Slice(validity, offset, count);
	result->AddAuxiliaryData(make_uniq<VectorBufferHolder>(shared_from_this()));
	return result;
}

Four meaningful lines:

offset_ptr = data_ptr + type_size * offset — the new buffer points into the old buffer’s allocation. No data is copied.
make_buffer<StandardVectorBuffer>(offset_ptr, count, type_size) — a new VectorBuffer object is created that wraps that pointer. Note the constructor takes a raw data_ptr, not an allocation, so the new buffer does not own the bytes.
result->GetValidityMask().Slice(validity, offset, count) — the NULL bitmap is sliced the same way: a view, not a copy.
result->AddAuxiliaryData(make_uniq<VectorBufferHolder>(shared_from_this())) — and here is the interesting part.

Step 4 is the entire subject of this article.

The problem with step 4

The new buffer points into the old buffer’s memory. If the old buffer is freed, the new buffer is a dangling pointer, and reading it gives you garbage — or a segfault, or (much worse) plausible-looking wrong answers.

C++’s usual answer is to copy the data. Copying is correct and expensive.

DuckDB’s answer is: make it impossible for the old buffer to be freed while the new one exists. The way to do that in C++ is to hold a shared_ptr to it. And since the parent VectorBuffer already inherits enable_shared_from_this<VectorBuffer>, the parent can hand out a reference to itself:

 cpp
 
 
// src/include/duckdb/common/types/vector_buffer.hpp:76
class VectorBufferHolder : public AuxiliaryDataHolder {
public:
	explicit VectorBufferHolder(buffer_ptr<VectorBuffer> buffer_p) : buffer(std::move(buffer_p)) {
	}

private:
	buffer_ptr<VectorBuffer> buffer;
};

The child buffer stores a shared_ptr<VectorBuffer> to its parent. As long as the child lives, the parent’s refcount is at least one and the parent’s destructor cannot run. When the child dies, the refcount drops and the parent is freed if nobody else wants it.

The lifetime is now a property of the type system, not of programmer discipline. You cannot forget to keep the parent alive, because keeping it alive is what the constructor does.

The auxiliary data hierarchy

VectorBufferHolder is one member of a small family. Here is the whole family:

 cpp
 
 
// src/include/duckdb/common/types/vector_buffer.hpp:46
struct AuxiliaryDataHolder {
	virtual ~AuxiliaryDataHolder() = default;

	virtual idx_t GetAllocationSize() const {
		return 0;
	}
};

struct AuxiliaryDataSet {
	vector<unique_ptr<AuxiliaryDataHolder>> data;
};

class PinnedBufferHolder : public AuxiliaryDataHolder {
public:
	explicit PinnedBufferHolder(BufferHandle handle);
	~PinnedBufferHolder() override;

private:
	BufferHandle handle;
};

class AuxiliaryDataSetHolder : public AuxiliaryDataHolder {
public:
	explicit AuxiliaryDataSetHolder(buffer_ptr<AuxiliaryDataSet> buffer) : auxiliary_data(std::move(buffer)) {
	}

private:
	buffer_ptr<AuxiliaryDataSet> auxiliary_data;
};

The design is a classic type-erased bag:

AuxiliaryDataHolder is an abstract base with a virtual destructor and an optional GetAllocationSize() hook so that memory accounting can walk the bag.
AuxiliaryDataSet is a vector<unique_ptr<AuxiliaryDataHolder>> — the bag itself, heap-allocated on demand.
AuxiliaryDataSetHolder holds a whole bag, not a single item. That is the “share my parent’s bag” holder.

And the bag hangs off the buffer:

 cpp
 
 
// src/include/duckdb/common/types/vector_buffer.hpp:119
	void AddAuxiliaryData(unique_ptr<AuxiliaryDataHolder> aux_data_p) {
		if (!auxiliary_data) {
			auxiliary_data = make_buffer<AuxiliaryDataSet>();
		}
		auxiliary_data->data.push_back(std::move(aux_data_p));
	}
	const buffer_ptr<AuxiliaryDataSet> &GetAuxiliaryData() const {
		return auxiliary_data;
	}
	virtual void ClearAuxiliaryData() {
		auxiliary_data.reset();
	}

Note the lazy allocation: the bag is only created when something is actually put into it. A plain INTEGER column with no strings and no slices has no AuxiliaryDataSet at all — just a null buffer_ptr.

Three kinds of thing you can hold

1. A parent vector buffer (VectorBufferHolder). Used by slicing, as above.

2. Another buffer’s entire auxiliary set (AuxiliaryDataSetHolder). This is how string heaps travel.

When DuckDB stores a VARCHAR column, the 2048 string_t structs live in the vector buffer, but the characters live in a separate heap. If a string is longer than 12 bytes it is a pointer into that heap; if it is shorter it is inlined in the string_t itself.

Now slice a string vector. The string_t array can be sliced with a pointer bump, but the heap cannot — it is a separate allocation that the parent owns. So the child needs to keep the parent’s heap alive:

 cpp
 
 
// src/common/vector/string_vector.cpp:139
buffer_ptr<VectorBuffer> VectorStringBuffer::SliceInternal(const LogicalType &type, idx_t offset, idx_t end) {
	auto type_size = GetTypeIdSize(type.InternalType());
	auto offset_ptr = data_ptr + type_size * offset;
	auto count = count_t(end - offset);
	auto result = make_buffer<VectorStringBuffer>(offset_ptr, count);
	result->GetValidityMask().Slice(validity, offset, count);
	result->AddAuxiliaryData(make_uniq<VectorBufferHolder>(shared_from_this()));
	return result;
}

Same shape as the flat case — but note that VectorStringBuffer’s own auxiliary set contains the string heap (a StringHeapHolder). The child now holds a VectorBufferHolder pointing at the parent, and the parent’s bag contains the heap. One level of indirection keeps the whole chain alive.

3. A pinned storage block (PinnedBufferHolder). This one is different in kind: it holds a BufferHandle, which is DuckDB’s buffer-manager pin. Pinning a block prevents the buffer manager from evicting it. Putting a BufferHandle inside a vector means that as long as the vector is alive, the storage block stays pinned and mapped.

 cpp
 
 
// src/common/vector/string_vector.cpp:235
void StringVector::AddAuxiliaryData(Vector &vector, unique_ptr<AuxiliaryDataHolder> data) {
	vector.AddAuxiliaryData(std::move(data));
}

void StringVector::AddHandle(Vector &vector, BufferHandle handle) {
	AddAuxiliaryData(vector, make_uniq<PinnedBufferHolder>(std::move(handle)));
}

void StringVector::AddHeapReference(Vector &vector, const Vector &other) {
	vector.AddHeapReference(other);
}

AddHandle is the entry point: a vector that reads strings directly out of a Parquet or CSV buffer (rather than copying them into a heap) attaches the pin to the vector. When the vector is recycled, the pin is released with it. The buffer manager never has to be told explicitly.

Walking through dictionaries

There is a subtlety with AddHeapReference. If the source vector is a dictionary vector, the heap it actually uses belongs to the dictionary’s child, not to the dictionary itself. So the reference has to be chased:

 cpp
 
 
// src/common/types/vector.cpp:303
void Vector::AddHeapReference(const Vector &other) {
	if (other.GetVectorType() == VectorType::DICTIONARY_VECTOR) {
		AddHeapReference(DictionaryVector::Child(other));
		return;
	}
	auto &auxiliary_data = other.buffer->GetAuxiliaryData();
	if (!auxiliary_data) {
		return;
	}
	AddAuxiliaryData(make_uniq<AuxiliaryDataSetHolder>(auxiliary_data));
}

Three lines of recursion, then the interesting bit: the target gets an AuxiliaryDataSetHolder pointing at the source’s entire auxiliary set. Not a copy of the heap — a reference to the bag that contains it.

This is why the bag is shareable as a unit. Rather than enumerate “the things a vector might need to keep alive” and copy each one, DuckDB says: “here is the whole bag; hold onto it.” The bag is reference-counted (buffer_ptr<AuxiliaryDataSet>), so this is one refcount bump, not a deep copy of the contents.

Where the slicing actually happens

The buffer-level Slice is where the type dispatch happens, and it is worth reading because of how it handles the constant case:

 cpp
 
 
// src/common/types/vector_buffer.cpp:200
buffer_ptr<VectorBuffer> VectorBuffer::Slice(const LogicalType &type, idx_t offset, idx_t end) {
	if (vector_type == VectorType::CONSTANT_VECTOR) {
		// constant vectors do not need to get sliced - but we do need to update the count
		return ConstantSlice(type, count_t(end - offset));
	}
	auto result = SliceInternal(type, offset, end);
	if (result) {
		if (result->Size() != end - offset) {
			throw InternalException("Slice with offset,end did not set size correctly");
		}
	}
	return result;
}

A constant vector has one physical value, so “slicing” it cannot mean “take a pointer offset” — there is nothing to offset. It means “keep the same pointer, but declare that there are now N logical rows”. Which is exactly what ConstantSliceInternal does, and note that it also attaches a VectorBufferHolder to the parent, for the same lifetime reason:

 cpp
 
 
// src/common/vector/flat_vector.cpp:107
buffer_ptr<VectorBuffer> StandardVectorBuffer::ConstantSliceInternal(const LogicalType &type, count_t count) {
	// a constant has one physical entry - the logical count is tracked separately from the physical capacity
	auto result = make_buffer<StandardVectorBuffer>(data_ptr, count_t(1ULL), type_size);
	result->GetValidityMask().Set(0, validity.RowIsValid(0));
	result->SetVectorType(VectorType::CONSTANT_VECTOR);
	result->SetVectorSize(count);
	result->AddAuxiliaryData(make_uniq<VectorBufferHolder>(shared_from_this()));
	return result;
}

Notice count_t(1ULL) — the new buffer’s capacity is one element, while SetVectorSize(count) sets its logical size to count. Physical capacity and logical cardinality are decoupled for constants. A constant vector of a billion rows is one slot of memory.

And at the Vector level, the slice is a one-liner:

 cpp
 
 
// src/common/types/vector.cpp:209
void Vector::Slice(const Vector &other, idx_t offset, idx_t end) {
	buffer = other.buffer->Slice(GetType(), offset, end);
	if (!buffer) {
		buffer = other.buffer;
	}
}

Which brings us back to the whole design in one line: buffer is a buffer_ptr<VectorBuffer>, so replacing it is a refcount operation, and the vector’s data pointer now points into somebody else’s allocation.

Seeing it yourself

The auxiliary set is invisible from SQL, but you can provoke the paths that use it:

 sql
 
 
CREATE TABLE strings AS SELECT 'prefix-' || range AS s FROM range(100000);

-- every row allocates into a string heap; the heap must survive slicing,
-- filtering and projection.
SELECT length(s) FROM strings WHERE s LIKE 'prefix-1%' LIMIT 5;

-- a constant on the left, a flat vector on the right: the constant is a
-- one-element buffer whose logical size is the whole chunk.
SELECT count(*) FROM strings WHERE 1 = 1;

-- strings that fit inline (<= 12 bytes) never touch the heap at all.
SELECT count(*) FROM strings WHERE length(s) < 13;

The last two queries are the interesting contrast. 1 = 1 produces a CONSTANT_VECTOR on the left, and the comparison runs against a one-slot buffer. length(s) < 13 works on string_t structs whose payload is inline in the struct, so no heap is touched at all.

The fourth kind of thing: selection data

A dictionary vector’s payload is not the values — those live in the child. Its payload is the selection vector: a uint32_t per row saying which child row to look at. And a selection vector is sometimes heap-allocated.

So the same trick applies. When a dictionary vector is sliced with a non-zero offset, the new buffer’s selection vector is a pointer into the old one’s array — again a view, not a copy:

 cpp
 
 
// src/common/vector/dictionary_vector.cpp:99
buffer_ptr<VectorBuffer> DictionaryBuffer::SliceInternal(const LogicalType &type, idx_t offset, idx_t end) {
	// dictionary vector slice: slice the dictionary instead of stacking dictionaries
	if (type.InternalType() == PhysicalType::STRUCT) {
		throw InternalException("Struct vectors cannot be dictionary vectors");
	}
	auto count = end - offset;
	auto &sel_data = GetSelVector().sel_data();
	if (!sel_data) {
		// non-owning sel, we need to create a new selection vector to slice
		SelectionVector new_sel(count);
		for (idx_t i = 0; i < count; i++) {
			new_sel.set_index(i, sel_vector.get_index(offset + i));
		}
		return make_uniq<DictionaryBuffer>(new_sel, count, entry);
	}
	if (offset == 0) {
		// for offset = 0 all we have to do is update the count - so just create a new buffer
		return make_uniq<DictionaryBuffer>(sel_data, end, entry);
	}
	SelectionVector sliced_sel(sel_vector.data() + offset, count);
	auto result = make_uniq<DictionaryBuffer>(sliced_sel, count, entry);
	result->AddAuxiliaryData(make_uniq<SelectionDataHolder>(sel_data));
	return result;
}

Line 120 is the one to notice: make_uniq<SelectionDataHolder>(sel_data) — the new buffer keeps the allocation that owns the selection vector’s uint32_t array alive, because it now only holds a pointer into the middle of it.

The holder itself is the thinnest possible version of the pattern:

 cpp
 
 
// src/include/duckdb/common/vector/dictionary_vector.hpp:106
class SelectionDataHolder : public AuxiliaryDataHolder {
public:
	explicit SelectionDataHolder(buffer_ptr<SelectionData> selection_data_p)
	 : selection_data(std::move(selection_data_p)) {
	}

private:
	buffer_ptr<SelectionData> selection_data;
};

No destructor override, no extra behaviour. Just “I hold a reference to this allocation”. That is the whole abstraction: AuxiliaryDataHolder is a bag of “things my bytes depend on”, and every one of them is a shared_ptr in disguise.

Where the pattern does not apply: the validity mask

The data array can be sliced with a pointer bump because the elements are fixed-width and byte-aligned. The NULL bitmap cannot: a bitmask sliced at row 17 starts in the middle of a 64-bit word.

So the validity mask is the one place where a slice does copy — but only 32 words at most, and only when there is a mask to copy:

 cpp
 
 
// src/common/types/validity_mask.cpp:117
void ValidityMask::Slice(const ValidityMask &other, idx_t source_offset, idx_t count) {
	if (other.CannotHaveNull()) {
		validity_mask = nullptr;
		validity_data.reset();
		return;
	}
	if (source_offset == 0) {
		Initialize(other);
		return;
	}
	ValidityMask new_mask(count);
	new_mask.SliceInPlace(other, 0, source_offset, count);
	Initialize(new_mask);
}

Two details are worth stealing even though this is not an auxiliary-data story:

The other.CannotHaveNull() early-out means a column that never has NULLs slices its mask in zero time. The absence of a mask is itself information, and it propagates.
Initialize(other) on a zero offset still copies — but Initialize shares the underlying buffer when the source has one, so it is a refcount bump, not a copy of the bits.

So the 2048 rows of a vector cost 16 KiB of data plus at most 256 bytes of bitmap, and the bitmap is usually absent entirely.

What it costs

One heap allocation per holder. make_uniq<VectorBufferHolder>(shared_from_this()) allocates a small object. A slice is not quite free — it is one malloc of ~16 bytes plus a shared_ptr refcount bump, instead of a 16 KiB memcpy.
Atomic refcount operations. shared_ptr copy and destruction are atomic increments and decrements. On a many-core machine with 64 threads all slicing the same parent buffer, that is a contended cache line. DuckDB accepts this because the alternative (copying 16 KiB) is far worse.
A layer of indirection when the bag is walked. Reading the heap of a sliced string vector requires going through the parent.
The bag must be cleared when a vector is recycled. Otherwise a recycled vector would keep a stale pin or a stale parent alive. That is what ClearAuxiliaryData() is for, and it is called on every chunk reset.
Shared bags mean shared lifetimes. Because AuxiliaryDataSetHolder shares the entire bag, a child vector keeps alive everything the parent was keeping alive — including pins the child does not need. This is a small over-retention, traded for not having to enumerate what is needed.

The bottom line

The general pattern here is worth stealing: when you need a view onto someone else’s memory, and you cannot copy it, then make the view own a reference to the owner.

DuckDB does that with one polymorphic base class and a vector<unique_ptr<...>>. The consequences:

Slicing a vector is an O(1) pointer bump, not an O(n) copy.
A dangling slice is not something you can write; the parent’s shared_ptr refcount makes it impossible.
The same mechanism extends to string heaps, pinned storage blocks, and selection-vector data, with no extra code in the callers.
The cost is one small allocation and an atomic refcount per slice — paid in preference to 16 KiB of memory traffic.

Ownership, expressed in the type system, is cheaper than copying.

Orrify: How DuckDB Collapses Six Vector Types Into Three Words

Jay Hubbert — Thu, 22 Oct 2026 09:40:00 +0000

TL;DR: DuckDB stores column data in six different physical shapes (flat, constant, dictionary, sequence, FSST, shredded). Writing a filter six times is madness, so every vector is normalised into a UnifiedVectorFormat: a data pointer, a validity mask, and a selection vector. Three words. One kernel. The conversion is free for the three common shapes and only materialises data for the exotic ones — and only when somebody actually asks.

The dictionary case is the neat one: instead of copying 2048 rows, DuckDB keeps one extra indirection per row.

The problem with `WHERE x > 42`

You write this:

 sql
 
SELECT count(*) FROM events WHERE user_id = 42;

By the time the filter runs, the user_id column is not “a column of integers”. It is a Vector, and that vector might be:

a flat array of 2048 int64_ts, or
a single int64_t broadcast to all 2048 rows, or
a selection vector pointing at some other vector’s rows, or
an arithmetic sequence (start, increment) that has not been materialised at all, or
FSST-compressed strings, or
a shredded VARIANT.

A filter that reads data[i] works for the first case and is wrong for the others. A filter that handles all six has six branches, and if you have ten filters you have sixty.

DuckDB’s answer is to make the six shapes converge on a single canonical form before anybody looks at the data. That convergence is called Orrify, after Orri Erling, and it produces the UnifiedVectorFormat.

The unit of work: 2048 rows

Before any of this, the size of a vector. DuckDB does not process one row at a time; it processes batches of exactly STANDARD_VECTOR_SIZE rows:

 cpp
 
 
// src/include/duckdb/common/vector_size.hpp:15
//! The default standard vector size
#define DEFAULT_STANDARD_VECTOR_SIZE 2048U

//! The vector size used in the execution engine
#ifndef STANDARD_VECTOR_SIZE
#define STANDARD_VECTOR_SIZE DEFAULT_STANDARD_VECTOR_SIZE
#endif

#if (STANDARD_VECTOR_SIZE & (STANDARD_VECTOR_SIZE - 1) != 0)
#error The vector size must be a power of two
#endif

2048 rows is a deliberate number, and the #error above tells you the property that matters: the vector size must be a power of two. Everything downstream depends on it:

A vector of int64_t is exactly 16 KiB — four 4 KiB pages, perfectly aligned, no straddling.
A vector of int32_t is exactly 8 KiB.
Row offsets inside a vector are row & 2047 and row >> 11, so masking and shifting replace division.
The validity mask for 2048 rows is exactly 32 uint64_t words — 256 bytes, 4 cache lines.

You can override STANDARD_VECTOR_SIZE at compile time, but it must stay a power of two, or the whole engine’s indexing scheme falls apart.

Six physical shapes

Here is the complete list of physical vector representations in DuckDB:

 cpp
 
 
// src/include/duckdb/common/enums/vector_type.hpp:15
enum class VectorType : uint8_t {
	FLAT_VECTOR, // Flat vectors represent a standard uncompressed vector
	FSST_VECTOR, // Contains string data compressed with FSST
	CONSTANT_VECTOR, // Constant vector represents a single constant
	DICTIONARY_VECTOR, // Dictionary vector represents a selection vector on top of another vector
	SEQUENCE_VECTOR, // Sequence vector represents a sequence with a start point and an increment
	SHREDDED_VECTOR // Shredded variant vector
};

Six. Each exists because there is a real query pattern that makes the general case wasteful:

FLAT — the normal case. 2048 values, in order.
CONSTANT — WHERE user_id = 42, or a SELECT 1 from an empty table. One value, 2048 rows.
DICTIONARY — the output of a filter. “Rows 3, 17, 900 of this chunk survived.” The rows are still in the parent vector; the dictionary just says which ones.
SEQUENCE — range() output, or a sorted integer column with known gaps. Two numbers instead of 2048.
FSST — a column of strings that compresses well with a learned symbol table.
SHREDDED — a VARIANT column where the “interesting” part is a typed payload and the rest is per-row noise.

None of these is a niche. All six show up in ordinary queries. Which is exactly the problem.

Three words

The canonical form is remarkably small. Here is the whole struct:

 cpp
 
 
// src/include/duckdb/common/vector/unified_vector_format.hpp:22
struct UnifiedVectorFormat {
	DUCKDB_API UnifiedVectorFormat();
	// disable copy constructors
	UnifiedVectorFormat(const UnifiedVectorFormat &other) = delete;
	UnifiedVectorFormat &operator=(const UnifiedVectorFormat &) = delete;
	//! enable move constructors
	DUCKDB_API UnifiedVectorFormat(UnifiedVectorFormat &&other) noexcept;
	DUCKDB_API UnifiedVectorFormat &operator=(UnifiedVectorFormat &&) noexcept;

	const SelectionVector *sel;
	const_data_ptr_t data;
	ValidityMask validity;
	SelectionVector owned_sel;
	PhysicalType physical_type;

Strip out the boilerplate and there are three payload words:

field	what it is
`data`	a pointer to the physical values
`validity`	the NULL bitmap (which may itself be absent)
`sel`	the selection vector: `row i` lives at `data[sel[i]]`

owned_sel exists because sometimes the selection vector has to be materialised rather than borrowed, and the struct must own it in that case. physical_type is a safety tag so that a kernel can assert it is not reading int32_ts out of an int64_t array.

A consumer of a unified format never asks what kind of vector it holds. It reads the value at data[sel->get_index(row)], and asks validity.RowIsValid(sel->get_index(row)) whether that row is NULL. That is the whole contract. And the DuckDB header documents the intent in as many words:

 cpp
 
 
// src/include/duckdb/common/types/vector.hpp:121
	//! Creates a UnifiedVectorFormat of a vector
	//! The UnifiedVectorFormat allows efficient reading of vectors regardless of their vector type
	//! It contains (1) a data pointer, (2) a validity mask, and (3) a selection vector
	//! Access to the individual vector elements can be performed through data_pointer[sel_idx[i]]/validity[sel_idx[i]]
	//! The most common vector types (flat, constant & dictionary) can be converted to the canonical format "for free"
	//! ToUnifiedFormat was originally called Orrify, as a tribute to Orri Erling who came up with it
	DUCKDB_API void ToUnifiedFormat(UnifiedVectorFormat &data) const;

Note “for free”. That is the whole design goal: normalisation must not cost anything on the hot path.

The conversion, and when it is free

Here is the entry point:

 cpp
 
 
// src/common/types/vector.cpp:465
void Vector::ToUnifiedFormat(UnifiedVectorFormat &format) const {
	format.physical_type = GetType().InternalType();
	auto vtype = GetVectorType();
	if (vtype != VectorType::FLAT_VECTOR && vtype != VectorType::CONSTANT_VECTOR &&
	 vtype != VectorType::DICTIONARY_VECTOR) {
		// FSST/SEQUENCE/SHREDDED: flatten first so the buffer can provide unified format
		Flatten();
	}
	Buffer().ToUnifiedFormat(format);
}

Read the branches:

FLAT, CONSTANT, DICTIONARY — no work beyond filling in three fields. The buffer’s own ToUnifiedFormat handles them.
FSST, SEQUENCE, SHREDDED — Flatten() is called first, which materialises the data into a plain flat buffer.

That last point is the crux of the design. Normalisation is lazy and one-directional. You never flatten a vector just because you might need to. You flatten it when a consumer actually asks for the unified format and the shape is exotic.

And note the asymmetry: DuckDB flattens the exotic shapes eagerly-on-demand, but it never flattens the cheap shapes. There is no “give me the flat version of this dictionary” fast path — the dictionary stays a dictionary, and the cost is paid as one indirection per row.

The flat and constant cases

A flat buffer’s unified format is the simplest possible thing: the data pointer, the validity mask, and a selection vector that says “row i is at index i”.

 cpp
 
 
// src/common/vector/flat_vector.cpp:220
void StandardVectorBuffer::ToUnifiedFormat(UnifiedVectorFormat &format) const {
	if (vector_type == VectorType::CONSTANT_VECTOR) {
		format.sel = ConstantVector::ZeroSelectionVector(Size(), format.owned_sel);
	} else {
		format.sel = FlatVector::IncrementalSelectionVector();
	}
	format.data = data_ptr;
	format.validity = validity;
}

For a flat vector the selection vector is the identity — IncrementalSelectionVector() returns a process-global static. For a constant vector it is a process-global static array of 2048 zeros. The “broadcast” is literally an array read:

 cpp
 
 
// src/common/types/vector_constants.cpp:7
const SelectionVector *ConstantVector::ZeroSelectionVector() {
	static const SelectionVector ZERO_SELECTION_VECTOR =
	 SelectionVector(const_cast<sel_t *>(ConstantVector::ZERO_VECTOR), STANDARD_VECTOR_SIZE); // NOLINT
	return &ZERO_SELECTION_VECTOR;
}

const SelectionVector *FlatVector::IncrementalSelectionVector() {
	static const SelectionVector INCREMENTAL_SELECTION_VECTOR;
	return &INCREMENTAL_SELECTION_VECTOR;
}

const sel_t ConstantVector::ZERO_VECTOR[STANDARD_VECTOR_SIZE] = {0};

This is a lovely piece of economy. A constant vector of a billion rows still costs one 8-byte slot of data plus a pointer to a static array of zeros. There is no allocation, no per-row storage, and no special case anywhere in the kernel — a consumer cannot tell a constant from a flat vector except by looking at sel[0] and finding a zero.

The dictionary case: one indirection, not a copy

Now the interesting one. A dictionary vector is “a selection vector on top of another vector”. Converting it to a unified format could mean materialising 2048 gathered values into a fresh buffer. DuckDB does not do that. Instead it flattens the child in place and keeps the dictionary’s own selection vector as the indirection:

 cpp
 
 
// src/common/vector/dictionary_vector.cpp:67
void DictionaryBuffer::ToUnifiedFormat(UnifiedVectorFormat &format) const {
	format.owned_sel.Initialize(sel_vector);
	format.sel = &format.owned_sel;

	auto &child = entry->data;
	if (child.GetVectorType() != VectorType::FLAT_VECTOR) {
		// flatten the child in-place
		entry->data.Flatten();
	}
	format.data = FlatVector::GetData(entry->data);
	format.validity = FlatVector::ValidityMutable(entry->data);
}

Read it carefully. Two things happen:

The child is flattened in place, if it is not already flat. Flattening the child is safe and cheap-ish because the child is a whole vector of dense data — the dictionary is only a view onto it. (It is still a copy, which is why the child is only flattened when it is actually not flat.)
The dictionary’s selection vector becomes format.sel.

So the final access pattern for a dictionary vector row i is:

text

idx = sel->get_index(i) // read a uint32_t from the selection vector
value = data[idx] // read 8 bytes

versus a flat vector’s:

text

value = data[i] // read 8 bytes

One extra dependent load per row. That is the entire cost of supporting the dictionary shape. Compare it with the alternative — gathering 2048 values into a fresh buffer is 16 KiB of memory traffic plus an allocation, per vector, per operator.

This is why dictionary vectors are the single most important shape in the engine: SELECT * FROM t WHERE predicate produces one, and everything downstream of a filter sees dictionaries. Making dictionaries free is what makes filters free.

And the flatten-the-child step is also not free, but note the condition: it only happens if the child isn’t flat. In the overwhelmingly common case (a filter over a flat scan result), the child is already flat and this branch is skipped.

The self-referential pointer

One detail in the struct above deserves a closer look. sel is a const SelectionVector *, and owned_sel is a SelectionVector by value inside the same struct. When the dictionary case runs, format.sel = &format.owned_sel — the struct points at itself.

That makes UnifiedVectorFormat non-copyable and very careful about being moved. The move constructor has to repair the self-reference:

 cpp
 
 
// src/common/vector/unified_vector_format.cpp:9
UnifiedVectorFormat::UnifiedVectorFormat(UnifiedVectorFormat &&other) noexcept
 : sel(nullptr), data(nullptr), physical_type(PhysicalType::INVALID) {
	bool refers_to_self = other.sel == &other.owned_sel;
	std::swap(sel, other.sel);
	std::swap(data, other.data);
	std::swap(validity, other.validity);
	std::swap(owned_sel, other.owned_sel);
	std::swap(physical_type, other.physical_type);
	if (refers_to_self) {
		sel = &owned_sel;
	}
}

After the swaps, owned_sel now lives in this object, but sel was swapped over from other and still points into other’s owned_sel. The refers_to_self flag detects that and re-points sel at the new home. Get this wrong and you get a dangling pointer that reads like garbage on the second vector of a batch.

This is the cost of the design: three words of payload, plus a self-referential pointer that has to be managed by hand. It is worth it, because the alternative — a variant type, a virtual dispatch per element, or six copies of every kernel — is worse.

A kernel that uses the three words

Here is the “generic” path in the scalar executor — the fallback used when the inputs are not all flat-or-constant. It is the clearest illustration of the contract, because it is written entirely in terms of the unified format and never mentions a vector type:

 cpp
 
 
// src/include/duckdb/common/vector_operations/scalar_executor.hpp:331
	static void ExecuteGeneric(const std::array<VectorRef, sizeof...(ARGS)> &inputs, Vector &result, idx_t count,
	 ADAPTER &adapter, std::index_sequence<Is...>) {
		constexpr idx_t N = sizeof...(ARGS);
		std::array<UnifiedVectorFormat, N> formats;
		for (idx_t i = 0; i < N; i++) {
			inputs[i].get().ToUnifiedFormat(formats[i]);
		}
		auto input_data = std::make_tuple(UnifiedVectorFormat::GetData<ARGS>(formats[Is])...);
		result.SetVectorType(VectorType::FLAT_VECTOR);
		if (result.size() != count) {
			FlatVector::SetSize(result, count);
		}
		auto result_data = FlatVector::GetDataMutable<RESULT_TYPE>(result);
		auto &result_validity = FlatVector::ValidityMutable(result);
		auto inputs_can_have_null = PrepareGenericResultValidity(formats.data(), N, result, count,
		 PRESERVE_RESULT_VALIDITY, ADAPTER::ADDS_NULLS);

		if (inputs_can_have_null) {
			if constexpr (SPECIALIZE_NULLABLE_GENERIC_SELECTIONS && N == 2) {
				ExecuteGenericBinaryNullableSwitch(std::get<0>(input_data), std::get<1>(input_data), formats,
				 result_data, count, result_validity, adapter);
			} else {
				for (idx_t row = 0; row < count; row++) {
					std::array<idx_t, N> input_indices = {{formats[Is].sel->get_index(row)...}};
					if ((... && formats[Is].validity.RowIsValid(input_indices[Is]))) {
						result_data[row] =
						 adapter.Operation(result_validity, row, std::get<Is>(input_data)[input_indices[Is]]...);
					} else {
						result_validity.SetInvalid(row);
					}
				}
			}
		} else if constexpr (SPECIALIZE_NON_NULL_GENERIC_SELECTIONS && N == 2) {
			ExecuteGenericBinaryNoNullSwitch(std::get<0>(input_data), std::get<1>(input_data), formats, result_data,
			 count, result_validity, adapter);
		} else {
			for (idx_t row = 0; row < count; row++) {
				result_data[row] = adapter.Operation(result_validity, row,
				 std::get<Is>(input_data)[formats[Is].sel->get_index(row)]...);
			}
		}
	}

Three things to notice:

ToUnifiedFormat is called once per input, outside the loop. Normalisation happens at the boundary, not per row.
The inner loop reads formats[Is].sel->get_index(row) — the selection vector — and then indexes input_data. It does not know, and does not care, whether that selection vector is the identity (flat), a table of zeros (constant), or the dictionary’s permutation.
The NULL check is formats[Is].validity.RowIsValid(input_indices[Is]) — again, no vector type in sight.

The whole function is one kernel that covers all six shapes. If Orrify did not exist, this function would need six variants, or a switch on GetVectorType() in the innermost loop, which would defeat every branch predictor in the CPU.

Seeing it yourself

The vector types are an in-memory detail, so EXPLAIN will not show them to you. But you can provoke them:

 sql
 
 
CREATE TABLE events AS SELECT range AS id, range % 100 AS bucket FROM range(100000);

-- the literal 42 is a CONSTANT_VECTOR; events.id is a FLAT_VECTOR.
-- The executor runs the "one constant, one flat" specialised loop.
SELECT count(*) FROM events WHERE id = 42;

-- a filter produces a DICTIONARY_VECTOR for the surviving rows.
-- Everything downstream of the filter reads through a selection vector.
SELECT bucket, count(*) FROM events WHERE id % 7 = 0 GROUP BY bucket;

And to see the shape of the plan the constant is feeding:

 sql
 
 
SET explain_output='all';
EXPLAIN SELECT count(*) FROM events WHERE id = 42;

The point is not the plan text. The point is that the same comparison kernel handles id = 42 (constant on the right), id = other.id (flat on both sides), and id = list_value.id (dictionary on the left), without a single branch on vector type in the inner loop.

What it costs

The trade-offs are worth being explicit about:

One indirection per row for dictionary vectors. A dictionary read is data[sel[i]], not data[i]. The load is dependent — the CPU cannot prefetch past the selection vector — so dictionary-heavy scans are measurably slower per row than flat ones. This is the price of not materialising.
Exotic shapes pay a copy. FSST, sequence, and shredded vectors are flattened on demand. If a query keeps re-flattening the same FSST vector, it keeps paying. DuckDB mitigates this by flattening in place where it can (entry->data.Flatten() replaces the child’s buffer) rather than allocating a fresh copy each time.
A mutable sel is not free either. format.owned_sel.Initialize(sel_vector) copies the selection vector’s 2048 uint32_ts (8 KiB) into the unified format. For a dictionary vector this is a real memcpy, done once per conversion.
Three shapes are free, three are not. Flat, constant, and dictionary cost nothing to normalise. FSST, sequence, and shredded cost a flatten. In practice the exotic ones are rare enough that this is the right trade.

The bottom line

The general lesson is not about vectors. It is that a system with several equally-valid representations of the same data needs exactly one representation that everybody reads.

DuckDB’s trick is that the canonical form is not a fully materialised flat vector. If it were, every filter on a dictionary would cost 16 KiB of copying. Instead the canonical form is “data pointer plus validity mask plus selection vector” — a form that can represent all six shapes without copying any of them, at the price of one indirection for the shapes that need it.

Three words, six shapes, one kernel. The normalisation is free where it can be, lazy where it cannot, and it never lies about what the data is.

Next time you see a DuckDB filter, remember: the filter does not know what it is filtering. It knows it is reading data[sel[i]].

A Tour of DuckDB's Internals: The Forty-Three Tricks

Jay Hubbert — Thu, 22 Oct 2026 09:10:00 +0000

TL;DR: Forty-three deep dives into DuckDB’s source, in the order they appeared. Each one is a specific, self-contained trick: how a validity mask disappears when a column has no NULLs, how a 16-bit salt lets a hash probe skip the tuple entirely, how Node48 buys O(1) child lookup with a byte-indexed side table, how a DELETE of consecutive rows stores zero row IDs. Every entry links to the full write-up.

Start anywhere. There is no dependency between them.

Why DuckDB’s source is worth reading

Most database engines are two systems wearing a trenchcoat: a storage engine that predates the query engine that sits on top of it, and a query engine that inherits the storage engine’s compromises. Read enough of DuckDB and you notice something different. It is an analytical database — columnar, vectorised, single-node, designed for scans and joins over data that does not fit in memory — and every single design decision in it is a bet on the shape of the workload. The bets are not written down in a design document. They are in the code, in the constants, in the if statements that nobody wrote because the alternative never occurs in practice.

That is what makes the code so instructive. There is no legacy to apologise for, so every strange-looking constant is load-bearing. STANDARD_VECTOR_SIZE is 2048 because 2048 rows of an 8-byte column is exactly 16 KiB, which is exactly one page, which is exactly what you want when a vector’s worth of data has to survive a cache line boundary. string_t is 16 bytes with 12 bytes of inline storage because uint32_t + char[12] lands on a 16-byte slot that fits an AVX register. The hash table entry packs a 48-bit pointer and a 16-bit salt into one 64-bit word because the salt lets the probe reject a non-match without touching the tuple, and one cache line is the unit of pain.

The other thing you notice is how much of the work is avoidance rather than acceleration. DuckDB is fast in large part because it declines to do things: it does not decompress a string column to measure it (SELECT strlen(col) never reads the string), it does not build a hash table for a perfectly dense integer range, it does not store keys in an aggregate table when the range is dense enough to be an array, it does not copy strings into a heap when they are 12 bytes or shorter. Each of those is a bet too, and each of those bets is visible in a specific, readable piece of code.

That is what this series is about: forty-three bets, one per article, each taken apart.

#	Trick	Published
1	Orrify: How DuckDB Collapses Six Vector Types Into Three Words	2026-10-22
2	The Auxiliary Buffer: How a DuckDB Vector Owns Memory It Never Allocated	2026-10-22
3	The Validity Mask Is Optional: A Null Pointer Means No NULLs	2026-10-22
4	The 64-Bit Constant Mask: How DuckDB Specialises Its Scalar Executor at Runtime	2026-10-22
5	The Vector Cache: DuckDB Never Frees a DataChunk’s Memory	2026-10-22
6	The Compression Auction: Every Codec in DuckDB Bids on Your Data	2026-10-22
7	FSST: How DuckDB Learns a Symbol Table From a Sample of Your Strings	2026-10-22
8	ALP: How DuckDB Replaces Floating-Point Numbers With Integers	2026-10-22
9	Zone Maps: How DuckDB Prunes a Billion Rows Without Reading Them	2026-10-22
10	The 16-Bit Salt: How DuckDB Packs a Row Pointer and a Hash Into One 64-Bit Word	2026-10-23
11	The Perfect Hash Join: When the Integers Are Dense Enough to Skip the Hash Table	2026-10-23
12	chains_longer_than_one: The Boolean That Simplifies DuckDB’s Join Probe	2026-10-23
13	The Found Flag Lives Inside the Tuple: How DuckDB Does Outer Joins	2026-10-23
14	Spilling the Aggregate State: What DuckDB Does When a GROUP BY Outgrows Memory	2026-10-23
15	The Perfect Aggregate Hash Table Stores No Keys at All	2026-10-23
16	Clustered Aggregation: Updating a State Once Per Run, Not Once Per Row	2026-10-23
17	How count(DISTINCT a) and sum(DISTINCT a) Share One Dedup Table	2026-10-23
18	DuckDB Serialises Your ORDER BY Into One Byte-Comparable Blob	2026-10-23
19	Nine Sort Key Layouts, and the Statistics That Let DuckDB Skip a Radix Pass	2026-10-23
20	TopN: The Bounded Heap and the Boundary Value DuckDB Pushes Down as a Filter	2026-10-24
21	The Merge That Never Allocates: How DuckDB Merges Sorted Runs In Place	2026-10-24
22	The FilterCombiner: Turning a Pile of Predicates Into One Tight Range	2026-10-24
23	The Statistics Propagator: How min/max Proves a Predicate Is False	2026-10-24
24	The Join Order Optimizer: Dynamic Programming With a Greedy Escape Hatch	2026-10-24
25	The Deliminator: Deleting the Join That Exists Only to Carry a Column	2026-10-24
26	Distributivity: How DuckDB Factors a Common Predicate Out of an OR	2026-10-24
27	Pinning Is Ownership: DuckDB’s Buffer Handle Is a shared_ptr	2026-10-24
28	The Lock-Free Eviction Queue: Weak Pointers and Sequence Numbers	2026-10-24
29	The Temporary Memory Manager Solves a Convex Optimization to Split Memory	2026-10-24
30	The Block Allocator: One mmap Arena, Per-Thread Free Lists and Lazy Commit	2026-10-25
31	The Undo Log Is a Self-Describing Byte Stream	2026-10-25
32	An UPDATE Never Touchs the Row Layout	2026-10-25
33	The Consecutive-Rows Fast Path: A DELETE That Stores Zero Row IDs	2026-10-25
34	VisibilityBound: How DuckDB Makes an Off-by-One a Compile Error	2026-10-25
35	All-or-Nothing: How DuckDB Decides Whether a Pipeline Runs in Parallel	2026-10-25
36	Morsels: How DuckDB Hands Out Scan Work Without Locking the Scan	2026-10-25
37	The Query Plan Is an Event DAG With Counter-Based Dependencies	2026-10-25
38	Never Touch the Bytes: How DuckDB Prunes Parquet Row Groups	2026-10-25
39	The Dictionary Is the Filter: How DuckDB Prunes a Parquet Page	2026-10-25
40	SWAR: How DuckDB Skips 64 CSV Bytes at a Time	2026-10-25

The vectorised engine

DuckDB does not process one row at a time. It processes 2048 of them at a time, in a Vector, and every trick in this section is about what that decision buys and what it costs.

Orrify: six vector types into three words. DuckDB has six physical vector types — flat, constant, dictionary, sequence, list and struct — and every operator that consumes a vector has to handle all of them. UnifiedVectorFormat collapses them into a single flat layout of three machine words per column (a data pointer, a validity pointer and a selection-vector pointer), so an operator writes its logic once against the unified form instead of six times against the concrete forms. The trick is that the unified format is a view: it does not copy or flatten anything, it just points at whichever representation the vector happens to be in. Read the deep dive — src/include/duckdb/common/vector/unified_vector_format.hpp.

The auxiliary buffer. Slicing a vector is a pointer bump, not a copy — but a slice points into its parent’s bytes, so the parent must outlive it. VectorBuffer solves this with an AuxiliaryDataHolder: a type-erased bag of extra allocations that a buffer can carry without owning them in the ordinary sense. The slice holds a shared_ptr to the parent, and the lifetime problem becomes a type-system problem instead of a programmer discipline problem. Read the deep dive — src/common/vector/flat_vector.cpp.

The validity mask is optional. A column with no NULLs should not pay for a NULL bitmap. DuckDB’s ValidityMask represents “no NULLs” as a null pointer, and every read of the mask checks for that first. The mask is allocated lazily, on the first NULL, and the whole engine benefits: a scan of a NOT NULL column touches no mask bytes at all. Read the deep dive — src/include/duckdb/common/types/validity_mask.hpp.

The 64-bit constant mask. Scalar expressions are not compiled 2048 separate times. At runtime DuckDB classifies the inputs of a vectorised operation — all constant, all flat, or mixed — and packs that classification into a 64-bit mask that is then used to select among specialisations. The specialisation happens once per vector, and the per-row loop becomes a switch on a bit instead of a nest of ifs. Read the deep dive — src/include/duckdb/common/vector_operations/scalar_executor.hpp.

The vector cache. A DataChunk’s buffers are not freed when the chunk is reset; they are cached and reused. The cache is keyed by type and size, so the same allocation is handed out again and again. The result is that a steady-state query performs essentially zero allocations in the executor. Read the deep dive — src/common/types/vector_cache.cpp.

The storage format

DuckDB’s on-disk format is columnar, compressed, and — this is the interesting part — self-describing enough to be re-read without a schema. The tricks here are about how a column gets encoded, and how a scan manages to skip the parts of it that cannot match.

The compression auction. When DuckDB checkpoints a column it does not pick a codec from a lookup table. Every codec that supports the column’s type is asked to compress a sample of the data, and each one reports how well it did. The winner is chosen by a scoring function that weighs compression ratio against decompression speed. The “auction” is the metaphor, but the mechanism is a real benchmark on real data. Read the deep dive — src/storage/table/column_data_checkpointer.cpp.

FSST learns its symbol table from a sample. Fast Static Symbol Table compression works by replacing common byte sequences with short symbol codes. DuckDB builds the symbol table from a random 25% sample of the column, then measures how well the resulting table compresses the whole column — and if the sample was unrepresentative, it goes back and tries again. The metadata it writes is a backwards metadata table, so a scan can skip an entire 1024-value vector without touching a byte of payload. Read the deep dive — src/storage/compression/fsst.cpp.

ALP replaces floating-point numbers with integers. Adaptive Lossless floating-Point compression works by observing that most float columns are really integers with an exponent and a factor attached. ALP searches for that (exponent, factor) pair by sampling, converts the values to integers, and stores those with ordinary integer compression — losslessly, because the original floats can be reconstructed exactly. Read the deep dive — src/include/duckdb/storage/compression/alp/alp_compress.hpp.

Zone maps. Every column segment and every row group carries min/max statistics. Before a segment is read, the scan asks whether the predicate’s range can possibly intersect the segment’s range; if not, the segment is skipped without a byte being read. The same zone-map logic decides how many rows to prefetch, so DuckDB does not even fetch the blocks it knows it does not need. Read the deep dive — src/storage/table/row_group.cpp.

Joins and aggregates

This is where an analytical database spends most of its time. The tricks here are about making the hash table smaller, the probe cheaper, and the group-by state smaller.

The 16-bit salt. A hash table entry is a single 64-bit word holding a 48-bit pointer to the tuple and 16 bits taken from the tuple’s hash. The probe compares the whole word: if the salts differ, the keys are definitely different, and the engine never dereferences the tuple to find out. Only a salt match triggers the expensive comparison. The same 16 bits are reused to derive the linear-probing increment, so the probe step is hash-dependent without a second hash. Read the deep dive — src/include/duckdb/execution/ht_entry.hpp.

The perfect hash join. When the join key is a dense integer range, a hash table is the wrong data structure: the “hash table” can be a flat array indexed by the key itself, with a bitmap for presence. DuckDB detects this case and switches to a completely different executor. Read the deep dive — src/execution/operator/join/perfect_hash_join_executor.cpp.

chains_longer_than_one. The join probe has two completely different algorithms: one for a build side that fits in cache, one for a build side that does not. Which one to use is decided by a single boolean flag on the hash table — a flag that is itself cheap to maintain, because it only flips when a chain actually gets long. Read the deep dive — src/execution/join_hashtable.cpp.

The found flag lives inside the tuple. A FULL or RIGHT OUTER join has to mark the build rows that found a match. DuckDB does not keep a side bitmap; it writes a boolean into the tuple itself, next to the payload. The “found” state and the data travel together, in the same cache line, and the flag costs nothing because the row layout already had padding. Read the deep dive — src/execution/join_hashtable.cpp.

Spilling the aggregate state. When a GROUP BY outgrows memory, DuckDB does not fail and does not spill the input. It spills the aggregate state: the partially-aggregated groups are partitioned by hash and written to disk, and the final aggregation runs over the spilled partitions. The state is usually much smaller than the input, which is the whole point. Read the deep dive — src/execution/aggregate_state_spilling.cpp.

The perfect aggregate hash table stores no keys. For a GROUP BY over a dense integer range, DuckDB uses a flat array of aggregate states indexed by the group value and stores no keys at all. There is nothing to hash, nothing to compare, and nothing to spill — the array is the table. Read the deep dive — src/execution/perfect_aggregate_hashtable.cpp.

Clustered aggregation. If the input arrives sorted by the grouping column, consecutive rows belong to the same group. DuckDB detects runs of equal group values and updates the aggregate state once per run instead of once per row, using the run length as a multiplier. Read the deep dive — src/common/clustered_aggregate.cpp.

count(DISTINCT a) and sum(DISTINCT a) share one table. Two DISTINCT aggregates over the same column are not two dedup hash sets. They are one: the deduplication happens once, the per-distinct-value aggregation is deferred to the finalize phase, and each aggregate keeps its own state vector indexed by the dedup table’s position. Read the deep dive — src/execution/operator/aggregate/distinct_aggregate_data.cpp.

Sorting

Sorting is where an analytical database is most likely to spill to disk, and where the “avoid work” strategy shows up most clearly.

One byte-comparable blob. DuckDB serialises a whole sort key — several columns, with NULL ordering and collations — into a single byte string whose memcmp order is the desired sort order. The sort then operates on one opaque byte array instead of a tuple of mixed-typed columns, which means one comparison function, one radix pass, and one serialization format for every query. Read the deep dive — src/function/scalar/create_sort_key.cpp.

Nine sort key layouts. There are nine fixed-size sort-key layouts, chosen at bind time from the number of columns, their types and their sizes. Alongside them DuckDB computes which byte positions of the key are provably constant (because the column’s statistics say so), and the radix sort skips those byte positions entirely — one fewer pass over the data. Read the deep dive — src/include/duckdb/common/sorting/sort_key.hpp.

TopN. A LIMIT 10 over a million rows does not sort a million rows. DuckDB keeps a bounded heap, and — this is the good part — pushes the current boundary value down as a dynamic filter, so the scan below can discard rows that could never make the cut. Read the deep dive — src/execution/operator/order/physical_top_n.cpp.

The merge that never allocates. Merging sorted runs does not need a new buffer for the output. DuckDB merges in place inside the buffer manager’s existing blocks, and it degrades gracefully — falling back to a slower path — when the buffer manager refuses to cooperate. Read the deep dive — src/include/duckdb/common/sorting/buffered_merge.hpp.

The optimizer

DuckDB’s optimizer is a pipeline of fairly ordinary rewrite rules. What makes it interesting is that the rules are proving things about the data, not just rewriting syntax.

The FilterCombiner. A pile of predicates on the same column — a > 3 AND a > 5 AND a < 9 AND a <> 7 — is folded into one tight range, and the combiner infers new predicates that were not written: a > 5 implies a > 3. The result is one range that the storage layer can act on. Read the deep dive — src/optimizer/filter_combiner.cpp.

The statistics propagator. If a column’s statistics say max(a) = 100, then a > 200 is not a filter — it is the constant false, and the optimizer can replace the expression with it. The propagator walks expressions bottom-up and turns comparisons that statistics have settled into constants. Read the deep dive — src/optimizer/statistics/expression/propagate_comparison.cpp.

The join order optimizer. DPhyp-style dynamic programming over plan subgraphs, with a cost model and a cardinality estimator — and a greedy escape hatch for the cases where the full enumeration would be too expensive. Read the deep dive — src/optimizer/join_order/plan_enumerator.cpp.

The Deliminator. A correlated subquery becomes a join with a delim join on top. The Deliminator recognises that the join exists only to carry a column up from the correlated side, and deletes it — along with the associated duplication machinery. Read the deep dive — src/optimizer/deliminator.cpp.

Distributivity. a = 1 OR b = 2 becomes a = 1 OR (a = 1 AND b = 2) — and then the common factor a = 1 can be hoisted out, giving the optimizer something it can push down into the scan. Read the deep dive — src/optimizer/rule/distributivity.cpp.

Memory management

DuckDB is a database that is expected to run on a laptop with 16 GB of RAM while scanning a 100 GB file. The memory manager is the machinery that makes that survivable.

Pinning is ownership. The buffer handle’s pin is a shared_ptr; the pin count is only a hint to the eviction policy. A buffer is evicted when nothing holds a shared_ptr to it, which makes “the data must stay alive” a property of the type system rather than of a counter discipline. Read the deep dive — src/include/duckdb/storage/buffer/buffer_handle.hpp.

The lock-free eviction queue. Buffers are evicted through a lock-free multi-producer multi-consumer ring of weak pointers with sequence-number versioning — the classic Vyukov-style queue. Because the nodes are weak, a buffer that was pinned again while sitting in the eviction queue is simply skipped rather than resurrected. Read the deep dive — src/storage/buffer/buffer_pool.cpp.

The temporary memory manager solves a convex optimization. When several operators want temporary memory at once — two hash aggregates, a sort, a join — the manager does not split it evenly. It solves a small convex optimization that weights each request by how much it will actually use, and hands out the split. Read the deep dive — src/storage/temporary_memory_manager.cpp.

The block allocator. Underneath everything is one big mmap arena with per-thread free lists, so the allocator itself never takes a lock on the hot path. It also keeps a decaying estimate of how much memory it should retain, so a burst of allocations does not permanently inflate the resident set. Read the deep dive — src/storage/block_allocator.cpp.

Transactions

DuckDB’s concurrency control is optimistic: readers never block, and conflicts are detected at commit time. The tricks here are all about making the bookkeeping cheap, because the bookkeeping is what every transaction pays for.

The undo log is a self-describing byte stream. The undo log is not a list of allocated objects; it is a contiguous byte stream where each entry describes its own type and length, so the log can be walked backwards and forwards without a side table of offsets. And it lives inside the buffer manager, which means it is pinned, accounted and spillable for free. Read the deep dive — src/transaction/undo_buffer_allocator.cpp.

An UPDATE never touches the row layout. An update does not rewrite the row. It appends the previous values to a version chain stored in the row group’s own memory, and readers who need the old value walk the chain backwards. The row layout never changes, which is what makes updates cheap. Read the deep dive — src/storage/table/update_segment.cpp.

The consecutive-rows fast path. A DELETE of 1000 consecutive rows does not store 1000 row IDs. There is a fast path that stores a range instead, and the row-ID array is often not stored at all. Read the deep dive — src/include/duckdb/transaction/delete_info.hpp.

VisibilityBound. Off-by-one errors in timestamp comparisons are the classic source of subtle MVCC bugs. In DuckDB the “visible up to” bound is a distinct type, so an exclusive timestamp passed where an inclusive one is required is a compile error rather than a wrong answer. Read the deep dive — src/include/duckdb/common/constants.hpp.

Parallelism

DuckDB is single-node but fully parallel. The parallelism is not thread-per-row and not thread-per-query; it is thread-per-pipeline with fine-grained work handoff.

All-or-nothing. A pipeline runs in parallel only if every stage in it agrees to be parallel. If any stage says no, the whole pipeline silently degrades to a single task — no partial parallelism, no correctness bugs from half-parallel operators. Read the deep dive — src/parallel/pipeline.cpp.

Morsels. Scan work is handed out as morsels — row groups — by a cursor that hands each one to whichever thread asks next. The scan itself is never locked; only the cursor is. Read the deep dive — src/storage/table/row_group_collection.cpp.

The query plan is an event DAG. Dependencies between operators are expressed as counters that the producer decrements, not as condition variables that the consumer waits on. The whole plan becomes a DAG of events, and the executor never blocks a thread waiting for data — it parks the task instead. Read the deep dive — src/parallel/event.cpp.

Reading files

Reading Parquet and CSV is a whole separate set of tricks, because the format itself gives you opportunities to avoid work that the query engine would otherwise have to do.

Never touch the bytes. Parquet’s footer carries per-column min/max statistics per row group. DuckDB parses them once at open time and asks, before a single byte of data is read, whether a row group can possibly contain a match. If not, the group is skipped wholesale — no page header parsed, no dictionary built, no column decompressed. The subtle part is the float handling: because NaN is not described by min/max, a float column is pruned twice, and skipped only if both prunes agree. Read the deep dive — extension/parquet/parquet_reader.cpp.

The dictionary is the filter. For a dictionary-encoded Parquet column, DuckDB applies the pushed-down filter to the dictionary values, producing a boolean per dictionary entry, and then turns per-row filtering into a table lookup. The row-level selection is written branchlessly: the code always writes the row index and only advances the counter on a match, so there is no branch misprediction on the hot path. Read the deep dive — extension/parquet/decoder/dictionary_decoder.cpp.

SWAR. The CSV parser is a precomputed 256-bytes-by-19-states transition table, so the hot loop is one array lookup per byte. On top of that, runs of uninteresting bytes are skipped 64 at a time using SIMD-within-a-register: eight uint64_t loads, one MaybeZeroBytes comparison per stop pattern, the per-byte flags packed into a 64-bit mask, and the first candidate found with a single count-trailing-zeros. Read the deep dive — src/include/duckdb/common/swar.hpp.

Three bonus deep dives

These three did not fit the once-a-day schedule, but they are worth the detour.

SELECT strlen(col) never reads the string. A Parquet string is a 4-byte length followed by a payload. DuckDB pushes strlen into the scan, which swaps in a reader that reads the length prefix and skips the payload with a pointer bump. The string data is never copied, never validated as UTF-8, never materialised. Read the deep dive — extension/parquet/read_single_parquet_file.cpp.

The 16-byte string_t and its 4-byte shadow prefix. string_t is a tagged union: 12 bytes of inline storage for short strings, a pointer for long ones — and for long strings, a copy of the first four bytes of the payload sitting next to the pointer. That shadow prefix makes equality a pair of 16-byte loads and makes hashing of a short string a dereference-free affair. Read the deep dive — src/include/duckdb/common/types/string_type.hpp.

The ART’s adaptive fanout. DuckDB’s index is an adaptive radix tree: shared key prefixes collapse into dedicated prefix nodes, and each node grows through Node4, Node16, Node48 and Node256 so that its memory cost tracks its actual branching. Node48 is the interesting one — a 256-byte byte-indexed side table buys O(1) child lookup — and every child reference is a single 64-bit word packing buffer id | offset | node type. Read the deep dive — src/include/duckdb/execution/index/art/node48.hpp.

Where to start

If you are new to DuckDB’s internals, this is the order I would read them in.

Orrify first, then the auxiliary buffer and the validity mask. These three establish the vocabulary — what a vector is, how it owns memory, how NULLs are represented. Everything else in the engine is written in those terms.
The 16-bit salt next. It is the shortest article in the series and it teaches you how DuckDB thinks about hash tables, which then makes the join and aggregate articles much easier.
The vector cache, because it explains why the executor feels allocation-free and why the memory articles matter.
The compression auction and then FSST and ALP. These are self-contained and they change how you think about storage.
Zone maps, then the Parquet row-group pruning article. Together they are the “skip work you do not need” theme, and they are the two that will change how you write queries.
Never touch the bytes and the dictionary is the filter if you mostly care about file formats; the statistics propagator and the filter combiner if you mostly care about the optimizer.
The undo log, then VisibilityBound, if you want to understand DuckDB’s concurrency model. These two are the ones that surprise people coming from a row-store engine.

Everything else is reference material. Come back to it when you need it.

Duckdb on Opensource internals

The Validity Mask Is Optional: A Null Pointer Means No NULLs

The cheapest possible representation of “no NULLs”

The type

Reading: the short circuit

Writing: only one direction costs anything

Where it pays off: the aggregate hash table

The write side propagates the fact

The same idea, one layer down: MVCC deletes

Seeing it yourself

What it costs

The bottom line

The Auxiliary Buffer: How a DuckDB Vector Owns Memory It Never Allocated

The 16 KiB copy you don’t have to do

The problem with step 4

The auxiliary data hierarchy

Three kinds of thing you can hold

Walking through dictionaries

Where the slicing actually happens

Seeing it yourself

The fourth kind of thing: selection data

Where the pattern does not apply: the validity mask

What it costs

The bottom line

Orrify: How DuckDB Collapses Six Vector Types Into Three Words

The problem with WHERE x > 42

The unit of work: 2048 rows

Six physical shapes

Three words

The conversion, and when it is free

The flat and constant cases

The dictionary case: one indirection, not a copy

The self-referential pointer

A kernel that uses the three words

Seeing it yourself

What it costs

The bottom line

A Tour of DuckDB's Internals: The Forty-Three Tricks

Why DuckDB’s source is worth reading

Table of contents

The vectorised engine

The storage format

Joins and aggregates

Sorting

The optimizer

Memory management

Transactions

Parallelism

Reading files

Three bonus deep dives

Where to start

The problem with `WHERE x > 42`