Orrify: How DuckDB Collapses Six Vector Types Into Three Words

TL;DR: DuckDB stores column data in six different physical shapes (flat, constant, dictionary, sequence, FSST, shredded). Writing a filter six times is madness, so every vector is normalised into a UnifiedVectorFormat: a data pointer, a validity mask, and a selection vector. Three words. One kernel. The conversion is free for the three common shapes and only materialises data for the exotic ones — and only when somebody actually asks.

The dictionary case is the neat one: instead of copying 2048 rows, DuckDB keeps one extra indirection per row.

The problem with WHERE x > 42

You write this:

SELECT count(*) FROM events WHERE user_id = 42;

By the time the filter runs, the user_id column is not “a column of integers”. It is a Vector, and that vector might be:

• a flat array of 2048 int64_ts, or
• a single int64_t broadcast to all 2048 rows, or
• a selection vector pointing at some other vector’s rows, or
• an arithmetic sequence (start, increment) that has not been materialised at all, or
• FSST-compressed strings, or
• a shredded VARIANT.

A filter that reads data[i] works for the first case and is wrong for the others. A filter that handles all six has six branches, and if you have ten filters you have sixty.

DuckDB’s answer is to make the six shapes converge on a single canonical form before anybody looks at the data. That convergence is called Orrify, after Orri Erling, and it produces the UnifiedVectorFormat.

The unit of work: 2048 rows

Before any of this, the size of a vector. DuckDB does not process one row at a time; it processes batches of exactly STANDARD_VECTOR_SIZE rows:

// src/include/duckdb/common/vector_size.hpp:15
//! The default standard vector size
#define DEFAULT_STANDARD_VECTOR_SIZE 2048U

//! The vector size used in the execution engine
#ifndef STANDARD_VECTOR_SIZE
#define STANDARD_VECTOR_SIZE DEFAULT_STANDARD_VECTOR_SIZE
#endif

#if (STANDARD_VECTOR_SIZE & (STANDARD_VECTOR_SIZE - 1) != 0)
#error The vector size must be a power of two
#endif

2048 rows is a deliberate number, and the #error above tells you the property that matters: the vector size must be a power of two. Everything downstream depends on it:

• A vector of int64_t is exactly 16 KiB — four 4 KiB pages, perfectly aligned, no straddling.
• A vector of int32_t is exactly 8 KiB.
• Row offsets inside a vector are row & 2047 and row >> 11, so masking and shifting replace division.
• The validity mask for 2048 rows is exactly 32 uint64_t words — 256 bytes, 4 cache lines.

You can override STANDARD_VECTOR_SIZE at compile time, but it must stay a power of two, or the whole engine’s indexing scheme falls apart.

Six physical shapes

Here is the complete list of physical vector representations in DuckDB:

// src/include/duckdb/common/enums/vector_type.hpp:15
enum class VectorType : uint8_t {
	FLAT_VECTOR,       // Flat vectors represent a standard uncompressed vector
	FSST_VECTOR,       // Contains string data compressed with FSST
	CONSTANT_VECTOR,   // Constant vector represents a single constant
	DICTIONARY_VECTOR, // Dictionary vector represents a selection vector on top of another vector
	SEQUENCE_VECTOR,   // Sequence vector represents a sequence with a start point and an increment
	SHREDDED_VECTOR    // Shredded variant vector
};

Six. Each exists because there is a real query pattern that makes the general case wasteful:

• FLAT — the normal case. 2048 values, in order.
• CONSTANT — WHERE user_id = 42, or a SELECT 1 from an empty table. One value, 2048 rows.
• DICTIONARY — the output of a filter. “Rows 3, 17, 900 of this chunk survived.” The rows are still in the parent vector; the dictionary just says which ones.
• SEQUENCE — range() output, or a sorted integer column with known gaps. Two numbers instead of 2048.
• FSST — a column of strings that compresses well with a learned symbol table.
• SHREDDED — a VARIANT column where the “interesting” part is a typed payload and the rest is per-row noise.

None of these is a niche. All six show up in ordinary queries. Which is exactly the problem.

Three words

The canonical form is remarkably small. Here is the whole struct:

// src/include/duckdb/common/vector/unified_vector_format.hpp:22
struct UnifiedVectorFormat {
	DUCKDB_API UnifiedVectorFormat();
	// disable copy constructors
	UnifiedVectorFormat(const UnifiedVectorFormat &other) = delete;
	UnifiedVectorFormat &operator=(const UnifiedVectorFormat &) = delete;
	//! enable move constructors
	DUCKDB_API UnifiedVectorFormat(UnifiedVectorFormat &&other) noexcept;
	DUCKDB_API UnifiedVectorFormat &operator=(UnifiedVectorFormat &&) noexcept;

	const SelectionVector *sel;
	const_data_ptr_t data;
	ValidityMask validity;
	SelectionVector owned_sel;
	PhysicalType physical_type;

Strip out the boilerplate and there are three payload words:

owned_sel exists because sometimes the selection vector has to be materialised rather than borrowed, and the struct must own it in that case. physical_type is a safety tag so that a kernel can assert it is not reading int32_ts out of an int64_t array.

A consumer of a unified format never asks what kind of vector it holds. It reads the value at data[sel->get_index(row)], and asks validity.RowIsValid(sel->get_index(row)) whether that row is NULL. That is the whole contract. And the DuckDB header documents the intent in as many words:

// src/include/duckdb/common/types/vector.hpp:121
	//! Creates a UnifiedVectorFormat of a vector
	//! The UnifiedVectorFormat allows efficient reading of vectors regardless of their vector type
	//! It contains (1) a data pointer, (2) a validity mask, and (3) a selection vector
	//! Access to the individual vector elements can be performed through data_pointer[sel_idx[i]]/validity[sel_idx[i]]
	//! The most common vector types (flat, constant & dictionary) can be converted to the canonical format "for free"
	//! ToUnifiedFormat was originally called Orrify, as a tribute to Orri Erling who came up with it
	DUCKDB_API void ToUnifiedFormat(UnifiedVectorFormat &data) const;

Note “for free”. That is the whole design goal: normalisation must not cost anything on the hot path.

The conversion, and when it is free

Here is the entry point:

// src/common/types/vector.cpp:465
void Vector::ToUnifiedFormat(UnifiedVectorFormat &format) const {
	format.physical_type = GetType().InternalType();
	auto vtype = GetVectorType();
	if (vtype != VectorType::FLAT_VECTOR && vtype != VectorType::CONSTANT_VECTOR &&
	    vtype != VectorType::DICTIONARY_VECTOR) {
		// FSST/SEQUENCE/SHREDDED: flatten first so the buffer can provide unified format
		Flatten();
	}
	Buffer().ToUnifiedFormat(format);
}

Read the branches:

• FLAT, CONSTANT, DICTIONARY — no work beyond filling in three fields. The buffer’s own ToUnifiedFormat handles them.
• FSST, SEQUENCE, SHREDDED — Flatten() is called first, which materialises the data into a plain flat buffer.

That last point is the crux of the design. Normalisation is lazy and one-directional. You never flatten a vector just because you might need to. You flatten it when a consumer actually asks for the unified format and the shape is exotic.

And note the asymmetry: DuckDB flattens the exotic shapes eagerly-on-demand, but it never flattens the cheap shapes. There is no “give me the flat version of this dictionary” fast path — the dictionary stays a dictionary, and the cost is paid as one indirection per row.

The flat and constant cases

A flat buffer’s unified format is the simplest possible thing: the data pointer, the validity mask, and a selection vector that says “row i is at index i”.

// src/common/vector/flat_vector.cpp:220
void StandardVectorBuffer::ToUnifiedFormat(UnifiedVectorFormat &format) const {
	if (vector_type == VectorType::CONSTANT_VECTOR) {
		format.sel = ConstantVector::ZeroSelectionVector(Size(), format.owned_sel);
	} else {
		format.sel = FlatVector::IncrementalSelectionVector();
	}
	format.data = data_ptr;
	format.validity = validity;
}

For a flat vector the selection vector is the identity — IncrementalSelectionVector() returns a process-global static. For a constant vector it is a process-global static array of 2048 zeros. The “broadcast” is literally an array read:

// src/common/types/vector_constants.cpp:7
const SelectionVector *ConstantVector::ZeroSelectionVector() {
	static const SelectionVector ZERO_SELECTION_VECTOR =
	    SelectionVector(const_cast<sel_t *>(ConstantVector::ZERO_VECTOR), STANDARD_VECTOR_SIZE); // NOLINT
	return &ZERO_SELECTION_VECTOR;
}

const SelectionVector *FlatVector::IncrementalSelectionVector() {
	static const SelectionVector INCREMENTAL_SELECTION_VECTOR;
	return &INCREMENTAL_SELECTION_VECTOR;
}

const sel_t ConstantVector::ZERO_VECTOR[STANDARD_VECTOR_SIZE] = {0};

This is a lovely piece of economy. A constant vector of a billion rows still costs one 8-byte slot of data plus a pointer to a static array of zeros. There is no allocation, no per-row storage, and no special case anywhere in the kernel — a consumer cannot tell a constant from a flat vector except by looking at sel[0] and finding a zero.

The dictionary case: one indirection, not a copy

Now the interesting one. A dictionary vector is “a selection vector on top of another vector”. Converting it to a unified format could mean materialising 2048 gathered values into a fresh buffer. DuckDB does not do that. Instead it flattens the child in place and keeps the dictionary’s own selection vector as the indirection:

// src/common/vector/dictionary_vector.cpp:67
void DictionaryBuffer::ToUnifiedFormat(UnifiedVectorFormat &format) const {
	format.owned_sel.Initialize(sel_vector);
	format.sel = &format.owned_sel;

	auto &child = entry->data;
	if (child.GetVectorType() != VectorType::FLAT_VECTOR) {
		// flatten the child in-place
		entry->data.Flatten();
	}
	format.data = FlatVector::GetData(entry->data);
	format.validity = FlatVector::ValidityMutable(entry->data);
}

Read it carefully. Two things happen:

1. The child is flattened in place, if it is not already flat. Flattening the child is safe and cheap-ish because the child is a whole vector of dense data — the dictionary is only a view onto it. (It is still a copy, which is why the child is only flattened when it is actually not flat.)
2. The dictionary’s selection vector becomes format.sel.

So the final access pattern for a dictionary vector row i is:

idx = sel->get_index(i)      // read a uint32_t from the selection vector
value = data[idx]            // read 8 bytes

versus a flat vector’s:

value = data[i]              // read 8 bytes

One extra dependent load per row. That is the entire cost of supporting the dictionary shape. Compare it with the alternative — gathering 2048 values into a fresh buffer is 16 KiB of memory traffic plus an allocation, per vector, per operator.

This is why dictionary vectors are the single most important shape in the engine: SELECT * FROM t WHERE predicate produces one, and everything downstream of a filter sees dictionaries. Making dictionaries free is what makes filters free.

And the flatten-the-child step is also not free, but note the condition: it only happens if the child isn’t flat. In the overwhelmingly common case (a filter over a flat scan result), the child is already flat and this branch is skipped.

The self-referential pointer

One detail in the struct above deserves a closer look. sel is a const SelectionVector *, and owned_sel is a SelectionVector by value inside the same struct. When the dictionary case runs, format.sel = &format.owned_sel — the struct points at itself.

That makes UnifiedVectorFormat non-copyable and very careful about being moved. The move constructor has to repair the self-reference:

// src/common/vector/unified_vector_format.cpp:9
UnifiedVectorFormat::UnifiedVectorFormat(UnifiedVectorFormat &&other) noexcept
    : sel(nullptr), data(nullptr), physical_type(PhysicalType::INVALID) {
	bool refers_to_self = other.sel == &other.owned_sel;
	std::swap(sel, other.sel);
	std::swap(data, other.data);
	std::swap(validity, other.validity);
	std::swap(owned_sel, other.owned_sel);
	std::swap(physical_type, other.physical_type);
	if (refers_to_self) {
		sel = &owned_sel;
	}
}

After the swaps, owned_sel now lives in this object, but sel was swapped over from other and still points into other’s owned_sel. The refers_to_self flag detects that and re-points sel at the new home. Get this wrong and you get a dangling pointer that reads like garbage on the second vector of a batch.

This is the cost of the design: three words of payload, plus a self-referential pointer that has to be managed by hand. It is worth it, because the alternative — a variant type, a virtual dispatch per element, or six copies of every kernel — is worse.

A kernel that uses the three words

Here is the “generic” path in the scalar executor — the fallback used when the inputs are not all flat-or-constant. It is the clearest illustration of the contract, because it is written entirely in terms of the unified format and never mentions a vector type:

// src/include/duckdb/common/vector_operations/scalar_executor.hpp:331
	static void ExecuteGeneric(const std::array<VectorRef, sizeof...(ARGS)> &inputs, Vector &result, idx_t count,
	                           ADAPTER &adapter, std::index_sequence<Is...>) {
		constexpr idx_t N = sizeof...(ARGS);
		std::array<UnifiedVectorFormat, N> formats;
		for (idx_t i = 0; i < N; i++) {
			inputs[i].get().ToUnifiedFormat(formats[i]);
		}
		auto input_data = std::make_tuple(UnifiedVectorFormat::GetData<ARGS>(formats[Is])...);
		result.SetVectorType(VectorType::FLAT_VECTOR);
		if (result.size() != count) {
			FlatVector::SetSize(result, count);
		}
		auto result_data = FlatVector::GetDataMutable<RESULT_TYPE>(result);
		auto &result_validity = FlatVector::ValidityMutable(result);
		auto inputs_can_have_null = PrepareGenericResultValidity(formats.data(), N, result, count,
		                                                         PRESERVE_RESULT_VALIDITY, ADAPTER::ADDS_NULLS);

		if (inputs_can_have_null) {
			if constexpr (SPECIALIZE_NULLABLE_GENERIC_SELECTIONS && N == 2) {
				ExecuteGenericBinaryNullableSwitch(std::get<0>(input_data), std::get<1>(input_data), formats,
				                                   result_data, count, result_validity, adapter);
			} else {
				for (idx_t row = 0; row < count; row++) {
					std::array<idx_t, N> input_indices = {{formats[Is].sel->get_index(row)...}};
					if ((... && formats[Is].validity.RowIsValid(input_indices[Is]))) {
						result_data[row] =
						    adapter.Operation(result_validity, row, std::get<Is>(input_data)[input_indices[Is]]...);
					} else {
						result_validity.SetInvalid(row);
					}
				}
			}
		} else if constexpr (SPECIALIZE_NON_NULL_GENERIC_SELECTIONS && N == 2) {
			ExecuteGenericBinaryNoNullSwitch(std::get<0>(input_data), std::get<1>(input_data), formats, result_data,
			                                 count, result_validity, adapter);
		} else {
			for (idx_t row = 0; row < count; row++) {
				result_data[row] = adapter.Operation(result_validity, row,
				                                     std::get<Is>(input_data)[formats[Is].sel->get_index(row)]...);
			}
		}
	}

Three things to notice:

1. ToUnifiedFormat is called once per input, outside the loop. Normalisation happens at the boundary, not per row.
2. The inner loop reads formats[Is].sel->get_index(row) — the selection vector — and then indexes input_data. It does not know, and does not care, whether that selection vector is the identity (flat), a table of zeros (constant), or the dictionary’s permutation.
3. The NULL check is formats[Is].validity.RowIsValid(input_indices[Is]) — again, no vector type in sight.

The whole function is one kernel that covers all six shapes. If Orrify did not exist, this function would need six variants, or a switch on GetVectorType() in the innermost loop, which would defeat every branch predictor in the CPU.

Seeing it yourself

The vector types are an in-memory detail, so EXPLAIN will not show them to you. But you can provoke them:

CREATE TABLE events AS SELECT range AS id, range % 100 AS bucket FROM range(100000);

-- the literal 42 is a CONSTANT_VECTOR; events.id is a FLAT_VECTOR.
-- The executor runs the "one constant, one flat" specialised loop.
SELECT count(*) FROM events WHERE id = 42;

-- a filter produces a DICTIONARY_VECTOR for the surviving rows.
-- Everything downstream of the filter reads through a selection vector.
SELECT bucket, count(*) FROM events WHERE id % 7 = 0 GROUP BY bucket;

And to see the shape of the plan the constant is feeding:

SET explain_output='all';
EXPLAIN SELECT count(*) FROM events WHERE id = 42;

The point is not the plan text. The point is that the same comparison kernel handles id = 42 (constant on the right), id = other.id (flat on both sides), and id = list_value.id (dictionary on the left), without a single branch on vector type in the inner loop.

What it costs

The trade-offs are worth being explicit about:

• One indirection per row for dictionary vectors. A dictionary read is data[sel[i]], not data[i]. The load is dependent — the CPU cannot prefetch past the selection vector — so dictionary-heavy scans are measurably slower per row than flat ones. This is the price of not materialising.
• Exotic shapes pay a copy. FSST, sequence, and shredded vectors are flattened on demand. If a query keeps re-flattening the same FSST vector, it keeps paying. DuckDB mitigates this by flattening in place where it can (entry->data.Flatten() replaces the child’s buffer) rather than allocating a fresh copy each time.
• A mutable sel is not free either. format.owned_sel.Initialize(sel_vector) copies the selection vector’s 2048 uint32_ts (8 KiB) into the unified format. For a dictionary vector this is a real memcpy, done once per conversion.
• Three shapes are free, three are not. Flat, constant, and dictionary cost nothing to normalise. FSST, sequence, and shredded cost a flatten. In practice the exotic ones are rare enough that this is the right trade.

The bottom line

The general lesson is not about vectors. It is that a system with several equally-valid representations of the same data needs exactly one representation that everybody reads.

DuckDB’s trick is that the canonical form is not a fully materialised flat vector. If it were, every filter on a dictionary would cost 16 KiB of copying. Instead the canonical form is “data pointer plus validity mask plus selection vector” — a form that can represent all six shapes without copying any of them, at the price of one indirection for the shapes that need it.

Three words, six shapes, one kernel. The normalisation is free where it can be, lazy where it cannot, and it never lies about what the data is.

Next time you see a DuckDB filter, remember: the filter does not know what it is filtering. It knows it is reading data[sel[i]].