The Undo Log Is a Self-Describing Byte Stream

Jay Hubbert — Sun, 25 Oct 2026 09:40:00 +0000

TL;DR: A transaction’s undo log in DuckDB is not a list of objects. It is a contiguous range of bytes, bump-allocated, where every entry is prefixed by an 8-byte header: a 4-byte type tag and a 4-byte payload length. Because the length is in the stream, the log can be walked forward by hopping payload + length, and walked backward by collecting the entry offsets into a small vector and replaying them in reverse. Commit, rollback, revert-commit, WAL serialisation and garbage collection are all the same two walks over the same flat bytes. And because those bytes come from the buffer manager, the undo log is memory-accounted like everything else.

What an undo log has to do

A transaction in DuckDB can insert, update, delete, create an index, change the catalog, and bump a sequence. Each of those changes needs an undo record so that the transaction can be undone — either because it rolled back, or because a crash happened, or because an older snapshot needs the old value.

That gives the undo log an awkward set of requirements:

It must be walkable forwards, because committing means applying the changes in the order they were made, and writing them to the WAL means serialising them in that same order.
It must be walkable backwards, because rolling back means undoing the most recent change first.
It must be resumable, because a commit can fail halfway through and the failure handler has to un-apply exactly the part that was applied.
It must be cheap, because a transaction can create a very large number of undo entries — but, crucially, not one per row.

The obvious implementation is a std::vector of nodes, each with a type tag, a payload, and pointers to the next and previous node. It works, and it is not what DuckDB does.

DuckDB’s undo log is a byte stream.

The header

Every entry is prefixed by exactly eight bytes:

 cpp
 
 
// src/transaction/undo_buffer.cpp:22
constexpr uint32_t UNDO_ENTRY_HEADER_SIZE = sizeof(UndoFlags) + sizeof(uint32_t);

UndoFlags is an enum class with an explicit uint32_t underlying type:

 cpp
 
 
// src/include/duckdb/common/enums/undo_flags.hpp:15
enum class UndoFlags : uint32_t { // far too big but aligned (TM)
	EMPTY_ENTRY = 0,
	CATALOG_ENTRY = 1,
	INSERT_TUPLE = 2,
	DELETE_TUPLE = 3,
	UPDATE_TUPLE = 4,
	SEQUENCE_VALUE = 5,
	ATTACHED_DATABASE = 6
};

The comment “far too big but aligned” is the whole story: a uint8_t tag would have made the header 5 bytes and misaligned every payload. A uint32_t tag makes it 4, and 4 + 4 = 8 — the header is exactly one machine word, and every payload that follows it is 8-byte aligned.

So the header is {UndoFlags type; uint32_t length;}.

Writing an entry

 cpp
 
 
// src/transaction/undo_buffer.cpp:28
UndoBufferReference UndoBuffer::CreateEntry(UndoFlags type, idx_t len) {
	idx_t alloc_len = AlignValue<idx_t>(len + UNDO_ENTRY_HEADER_SIZE);
	auto handle = allocator.Allocate(alloc_len);
	auto data = handle.GetDataMutable();
	// write the undo entry metadata
	Store<UndoFlags>(type, data);
	data += sizeof(UndoFlags);
	Store<uint32_t>(UnsafeNumericCast<uint32_t>(alloc_len - UNDO_ENTRY_HEADER_SIZE), data);
	// increment the position of the header past the undo entry metadata
	handle.position += UNDO_ENTRY_HEADER_SIZE;
	return handle;
}

Three things to notice.

The allocation is bump-style. allocator.Allocate(alloc_len) returns a handle whose position is the current write cursor. There is no free list per entry size, no malloc per entry, no object header. The caller then writes its payload into the space that follows.

The stored length is the padded length. alloc_len is the payload length rounded up to a multiple of 8, and the header stores alloc_len - 8. That means the length in the header is exactly the distance from one payload’s start to the next entry’s start. The padding is part of the record, not slack in the middle of it — which is what makes the forward walk work without any extra bookkeeping.

The header is written at the cursor, then the cursor is advanced past it. handle.position += UNDO_ENTRY_HEADER_SIZE moves the handle’s position from the start of the header to the start of the payload, so the returned UndoBufferReference points at the payload, not at the header.

The caller then fills in the payload:

 cpp
 
 
// src/transaction/duck_transaction.cpp:142
UndoBufferReference DuckTransaction::CreateUpdateInfo(DuckTableEntry &table_entry, idx_t type_size, idx_t capacity,
 idx_t value_capacity, idx_t row_group_start) {
	idx_t alloc_size = UpdateInfo::GetAllocSize(type_size, capacity, value_capacity);
	auto undo_entry = undo_buffer.CreateEntry(UndoFlags::UPDATE_TUPLE, alloc_size);
	auto &update_info = UpdateInfo::Get(undo_entry);
	UpdateInfo::Initialize(update_info, table_entry, GetTransactionId(), row_group_start, capacity, value_capacity);
	return undo_entry;
}

Walking forward: length-hopping

The forward walk is the simple one. You are at the start of an entry; you read the length out of the header; you jump that far past the payload; you are at the next entry.

 cpp
 
 
// src/transaction/undo_buffer.cpp:42
void UndoBuffer::IterateEntries(UndoBuffer::IteratorState &state, T &&callback) {
	// iterate in insertion order: start with the tail
	state.current = allocator.tail.get();
	state.started = true;
	while (state.current) {
		state.handle = allocator.buffer_manager.Pin(state.current->block);
		state.start = state.handle.GetDataMutable();
		state.end = state.start + state.current->position;
		while (state.start < state.end) {
			auto len_position = state.start + sizeof(UndoFlags);
			auto payload_position = len_position + sizeof(uint32_t);
			auto type = Load<UndoFlags>(state.start);
			auto len = Load<uint32_t>(len_position);

			callback(type, payload_position);
			state.start = payload_position + len;
		}
		state.current = state.current->prev;
	}
}

Two levels of iteration: across blocks (state.current walks the block chain), and within a block (state.start hops by length).

The within-block walk is branch-free apart from the loop condition: two loads, one add, one compare. No pointer chasing, no switch on node type to find the next node, no virtual dispatch.

The across-block walk goes prev — from the newest block back to the oldest — because the block chain is built by pushing new blocks on the head. So the forward-in-time walk of the entries is a backward walk of the block list. The chain is doubly linked precisely so that both directions are cheap.

Walking backward: replaying offsets in reverse

The backward walk cannot hop by length, because a length tells you where the next entry is, not where the previous one is. There is no back-pointer in the stream.

The trick is to remember where you have been. Walk the block forward, one entry at a time, and push each entry’s offset into a small vector. Then replay that vector backwards.

 cpp
 
 
// src/transaction/undo_buffer.cpp:64
void UndoBuffer::ReverseIterateEntries(UndoBuffer::IteratorState &end_state, T &&callback) {
	if (!end_state.started) {
		return;
	}

	idx_t end_position = 0;
	if (end_state.current) {
		auto block_start = end_state.handle.GetDataMutable();
		D_ASSERT(end_state.start >= block_start);
		end_position = NumericCast<idx_t>(end_state.start - block_start);
		D_ASSERT(end_position <= end_state.current->position);
	}

	// Start at the newest committed entry and iterate back to the oldest one.
	auto current = end_state.current ? end_state.current : allocator.head.get();
	while (current) {
		auto handle = allocator.buffer_manager.Pin(current->block);
		data_ptr_t start = handle.GetDataMutable();
		data_ptr_t end = start + (current == end_state.current ? end_position : current->position);
		vector<pair<UndoFlags, data_ptr_t>> nodes;
		while (start < end) {
			auto type = Load<UndoFlags>(start);
			start += sizeof(UndoFlags);
			auto len = Load<uint32_t>(start);
			start += sizeof(uint32_t);
			nodes.emplace_back(type, start);
			start += len;
		}
		for (idx_t i = nodes.size(); i > 0; i--) {
			callback(nodes[i - 1].first, nodes[i - 1].second);
		}
		current = current->next.get();
	}
}

The inner loop is the forward walk again — same length-hopping — but instead of calling the callback it accumulates (type, payload_pointer) pairs into nodes. Then the callback is invoked on nodes in reverse.

The cost is one small vector per block. A 256 KiB block of undo entries holds thousands of entries, so the vector holds thousands of 16-byte pairs — tens of kilobytes of temporary scratch for a block that is 256 KiB. That is a fine trade for turning “backwards” into “forwards plus a reverse loop”.

Note the direction of the block chain in the reverse walk: current = current->next.get(). The block list is walked in the opposite direction from the forward walk, because the newest block is at the head and the oldest at the tail, and going backwards in time means going from head to tail.

Why “resume from a position” needs an end state

The end_state parameter is how a partially-applied commit is un-applied. IteratorState remembers the block and the exact byte offset the forward walk had reached:

 cpp
 
 
// src/include/duckdb/transaction/undo_buffer.hpp:40
	struct IteratorState {
		BufferHandle handle;
		optional_ptr<UndoBufferEntry> current;
		data_ptr_t start;
		data_ptr_t end;
		bool started = false;
	};

When a commit fails halfway, the state holds the position of the entry that failed. ReverseIterateEntries then starts from exactly that entry and works backwards — which is precisely “un-apply the part that was applied”.

 cpp
 
 
// src/transaction/undo_buffer.cpp:201
void UndoBuffer::RevertCommit(UndoBuffer::IteratorState &end_state, transaction_t transaction_id) {
	CommitState state(transaction, transaction_id, active_transaction_state, CommitMode::REVERT_COMMIT);
	ReverseIterateEntries(end_state, [&](UndoFlags type, data_ptr_t data) { state.RevertCommit(type, data); });

	state.Verify();
}

One iterator state, used by both the commit walk and the revert walk. The machinery is identical; only the direction and the callback differ.

One log, four consumers

Because the representation is flat, all the transaction-lifecycle operations reduce to one of the two walks:

 cpp
 
 
// src/transaction/undo_buffer.cpp:187
void UndoBuffer::WriteToWAL(WriteAheadLog &wal, optional_ptr<StorageCommitState> commit_state) {
	WALWriteState state(transaction, wal, commit_state);
	UndoBuffer::IteratorState iterator_state;
	IterateEntries(iterator_state, [&](UndoFlags type, data_ptr_t data) { state.CommitEntry(type, data); });
}

void UndoBuffer::Commit(UndoBuffer::IteratorState &iterator_state, CommitInfo &info) {
	active_transaction_state = info.active_transactions;

	CommitState state(transaction, info.commit_id, active_transaction_state, CommitMode::COMMIT);
	IterateEntries(iterator_state, [&](UndoFlags type, data_ptr_t data) { state.CommitEntry(type, data, info); });
	state.Verify();
}

The WAL writer and the committer are the same loop with a different callback. The rollback is the reverse loop:

 cpp
 
 
// src/transaction/undo_buffer.cpp:208
void UndoBuffer::Rollback() {
	// rollback needs to be performed in reverse
	RollbackState state(transaction);
	ReverseIterateEntries([&](UndoFlags type, data_ptr_t data) { state.RollbackEntry(type, data); });
}

And “did this transaction change anything at all?” is a pointer test, not a scan:

 cpp
 
 
// src/transaction/undo_buffer.cpp:106
bool UndoBuffer::ChangesMade() {
	// we need to search for any index creation entries
	return allocator.head.get();
}

If the allocator’s head is null, nothing was ever allocated, so there is nothing to undo. That check runs on every commit.

The blocks live in the buffer manager

The undo log is not on the heap. It comes from the buffer manager, which means it is pinned, memory-accounted, and subject to the same memory limit as everything else.

The chain of blocks is a doubly linked list of UndoBufferEntry:

 cpp
 
 
// src/include/duckdb/transaction/undo_buffer_allocator.hpp:21
struct UndoBufferEntry {
	explicit UndoBufferEntry(BufferManager &buffer_manager) : buffer_manager(buffer_manager) {
	}
	~UndoBufferEntry();

	BufferManager &buffer_manager;
	shared_ptr<BlockHandle> block;
	idx_t position = 0;
	idx_t capacity = 0;
	unique_ptr<UndoBufferEntry> next;
	optional_ptr<UndoBufferEntry> prev;
};

block is a shared_ptr<BlockHandle> — the same handle type the buffer manager uses for a file block. position is the write cursor within that block; capacity is how big it is.

The allocator is where the two interesting decisions live:

 cpp
 
 
// src/transaction/undo_buffer_allocator.cpp:30
UndoBufferReference UndoBufferAllocator::Allocate(idx_t alloc_len) {
	lock_guard<mutex> guard(lock);
	D_ASSERT(!head || head->position <= head->capacity);
	BufferHandle handle;
	if (!head || head->position + alloc_len > head->capacity) {
		// no space in current head - allocate a new block
		auto block_size = buffer_manager.GetBlockSize();
		;
		idx_t capacity;
		if (!head && alloc_len <= 4096) {
			capacity = 4096;
		} else {
			capacity = block_size;
		}
		if (capacity < alloc_len) {
			capacity = NextPowerOfTwo(alloc_len);
		}
		auto entry = make_uniq<UndoBufferEntry>(buffer_manager);
		if (capacity < block_size) {
			entry->block = buffer_manager.RegisterSmallMemory(MemoryTag::TRANSACTION, capacity);
			handle = buffer_manager.Pin(entry->block);
		} else {
			handle = buffer_manager.Allocate(MemoryTag::TRANSACTION, capacity, false);
			entry->block = handle.GetBlockHandle();
		}
		entry->capacity = capacity;
		entry->position = 0;
		// add block to the chain
		if (head) {
			head->prev = entry.get();
			entry->next = std::move(head);
		} else {
			tail = entry.get();
		}
		head = std::move(entry);
	} else {
		handle = buffer_manager.Pin(head->block);
	}
	idx_t current_position = head->position;
	head->position += alloc_len;
	return UndoBufferReference(*head, std::move(handle), current_position);
}

Two things stand out.

The first block is small. If the transaction’s first undo entry fits in 4 KiB, the block is 4 KiB, and it is registered via RegisterSmallMemory — which makes it a TINY_BUFFER rather than a full 256 KiB buffer-manager block. A transaction that does one small UPDATE does not reserve a quarter of a megabyte from the pool. Every subsequent block is a full buffer-manager block.

The memory is charged under MemoryTag::TRANSACTION. The undo log shows up in duckdb_memory() like everything else, and it counts against the memory limit. A transaction that accumulates a huge undo log can therefore cause the buffer manager to start evicting other things — which is exactly right, because the undo log is the transaction’s memory.

The false in Allocate(MemoryTag::TRANSACTION, capacity, false) is can_destroy: the buffer must not simply be thrown away when it is unpinned, because the undo log has to survive until the transaction ends.

The non-owning pointer

The UpdateInfo chains that live inside row groups cannot hold a BufferHandle — they would pin the block forever. So they hold an UndoBufferPointer, which is just (entry, position):

 cpp
 
 
// src/include/duckdb/transaction/undo_buffer_allocator.hpp:58
struct UndoBufferPointer {
	UndoBufferPointer() : entry(nullptr), position(0) {
	}
	UndoBufferPointer(UndoBufferEntry &entry_p, idx_t position) : entry(&entry_p), position(position) {
	}

	UndoBufferEntry *entry;
	idx_t position;

	UndoBufferReference Pin() const;
	bool IsSet() const {
		return entry;
	}
};

Nothing is pinned until someone asks. Pin() re-acquires the buffer handle on demand:

 cpp
 
 
// src/transaction/undo_buffer_allocator.cpp:18
UndoBufferReference UndoBufferPointer::Pin() const {
	if (!entry) {
		throw InternalException("UndoBufferPointer::Pin called but no entry was found");
	}
	D_ASSERT(entry->capacity >= position);
	auto handle = entry->buffer_manager.Pin(entry->block);
	return UndoBufferReference(*entry, std::move(handle), position);
}

This is what makes it safe to store undo-buffer references inside persistent row-group structures. The row group holds a raw pointer and an offset; the memory is pinned only for the duration of the read that needs it.

Why “one entry per row” is not a problem

A bulk INSERT of a million rows produces a handful of undo entries, not a million, because the insert’s undo record is a range, not a set of rows:

 cpp
 
 
// src/transaction/duck_transaction.cpp:134
void DuckTransaction::PushAppend(DuckTableEntry &table_entry, idx_t start_row, idx_t row_count) {
	auto undo_entry = undo_buffer.CreateEntry(UndoFlags::INSERT_TUPLE, sizeof(AppendInfo));
	auto append_info = reinterpret_cast<AppendInfo *>(undo_entry.GetDataMutable());
	append_info->table = &table_entry;
	append_info->start_row = start_row;
	append_info->count = row_count;
}

Thirty bytes of undo for a million rows. The byte-stream representation is what makes this natural: the undo record is a value in the stream, and values can be ranges.

Deletes are similar, and they carry a further trick — a consecutive range of row ids is represented by a flag rather than an array:

 cpp
 
 
// src/transaction/duck_transaction.cpp:101
void DuckTransaction::PushDelete(DuckTableEntry &table_entry, RowVersionManager &info, idx_t vector_idx, row_t rows[],
 idx_t count, idx_t base_row) {
	bool is_consecutive = true;
	// check if the rows are consecutive
	for (idx_t i = 0; i < count; i++) {
		if (rows[i] != row_t(i)) {
			is_consecutive = false;
			break;
		}
	}
	idx_t alloc_size = sizeof(DeleteInfo);
	if (!is_consecutive) {
		// if rows are not consecutive we need to allocate row identifiers
		alloc_size += sizeof(uint16_t) * count;
	}

A consecutive delete allocates nothing beyond the fixed-size struct — no array of row ids at all. The allocation size is decided before the payload is written, which is only possible because the entry length is known up front.

What it costs

Eight bytes of overhead per entry. One type tag, one length. For a transaction with a thousand entries that is 8 KB. For a transaction with ten entries it is 80 bytes. Compare to a node-based design, where every entry also pays for a next pointer, a previous pointer, and a size field — and where the allocation of each node is a separate malloc.
One memcpy-free write. The payload is written directly into the bump-allocated space; there is no temporary object that is then copied.
A scratch vector during reverse walks. One vector<pair<UndoFlags, data_ptr_t>> per block, proportional to the number of entries in the block. This is the price of making “backwards” out of “forwards”.
A mutex on every allocation. UndoBufferAllocator::Allocate takes lock_guard<mutex>. A transaction’s undo buffer is normally only touched by its own thread, but a rollback can be triggered from another thread, so the lock is there.
Memory that counts against the limit. The undo log is buffer-manager memory, tagged TRANSACTION. A long-running transaction that touches a lot of rows will push the buffer manager into evicting other blocks. That is correct behaviour, but it means the cost of a long transaction is not hidden.

The upside is that the whole transaction-lifecycle machinery is small and uniform. There is one data structure, two iterators, and four consumers. A bug in one place is a bug in all of them, and there is exactly one place to look.

The bottom line

Flat representations pay off when the operations on them can be expressed as simple arithmetic on the representation itself.

Put the length in the stream. An 8-byte header — 4 bytes of type, 4 bytes of length — turns “find the next record” into pointer + length. No links, no traversal, no per-record allocation.
Walks are arithmetic; reversal is bookkeeping. Forward traversal is length-hopping. Backward traversal is forward traversal plus a vector of offsets, replayed in reverse. Both are O(n), and the second one is a small, local cost.
A self-describing stream is a resumable stream. Because the iterator state is just (block, offset), a failed commit can be un-applied from exactly where it stopped, using the same code that walks it.
Make the lifetime explicit. The bytes come from the buffer manager, tagged and accounted; the pointers that escape into persistent structures are non-owning and re-pin on demand.

The general lesson: before you design a data structure around objects and pointers, check whether the problem is really asking for a stream. When it is — logs, WALs, undo buffers, byte streams — a header and a length go a long way.

Mvcc on Opensource internals