SWAR: How DuckDB Skips 64 CSV Bytes at a Time

Jay Hubbert — Sun, 25 Oct 2026 14:10:00 +0000

TL;DR: CSV parsing is a state machine, and a state machine can be a table. DuckDB precomputes a 256 × 19 transition table per dialect, so the hot loop is one array lookup per byte. But most bytes in a CSV file are ordinary text inside a field, and ordinary text does not need the state machine at all. DuckDB computes a 64-bit “stop mask” over 64 bytes at a time using SWAR (SIMD-within-a-register), finds the first interesting byte with a single count-trailing-zeros, and skips the rest in one go.

The scenario

 sql
 
 
SELECT customer_id, description, amount
FROM 'orders.csv';

A 50 GB CSV. The description column is free text: 80 characters of ordinary words per row, with a comma every so often.

A CSV parser has to look at every byte — but most bytes are uninteresting. Inside an unquoted field, the only bytes that change anything are the delimiter, the quote, the newline, the carriage return, the comment marker, and the escape character. Everything else is “just another character of the field”.

So the parser has two jobs: run the state machine over the interesting bytes, and skip the ordinary ones fast.

The state machine: 256 × 19

DuckDB models CSV parsing as a finite state machine with 19 states:

 cpp
 
 
// src/include/duckdb/execution/operator/csv_scanner/csv_state.hpp:16
enum class CSVState : uint8_t {
	STANDARD = 0, //! Regular unquoted field state
	DELIMITER = 1, //! State after encountering a field separator (e.g., ;) - This is always the last delimiter byte
	//! ------------- Multi-byte Delimiter States (up to 4 bytes) ----------------------------------------------!//
	// ...
	RECORD_SEPARATOR = 5, //! State after encountering a record separator (i.e., \n)
	CARRIAGE_RETURN = 6, //! State after encountering a carriage return(i.e., \r)
	QUOTED = 7, //! State when inside a quoted field
	UNQUOTED = 8, //! State when leaving a quoted field
	ESCAPE = 9, //! State when encountering an escape character (e.g., \)
	INVALID = 10, //! Got to an Invalid State, this should error.
	NOT_SET = 11, //! If the state is not set, usually the first state before getting the first character
	// ...
	MAYBE_QUOTED = 18 //! We are potentially in a quoted value or the end of an unquoted value
};

The states are more than the naive “in a field / in a quoted field” pair, because CSV dialects are a swamp: multi-byte delimiters, \r\n vs \n vs \r, comments, escapes, strict vs lenient mode.

A naive parser writes this as a chain of ifs — if (c == '"') ... else if (c == ',') ... else if (c == '\n') — which is up to six unpredictable branches per byte. DuckDB precomputes the answer instead. The transition table is indexed by byte and by current state:

 cpp
 
 
// src/include/duckdb/execution/operator/csv_scanner/csv_state_machine_cache.hpp:20
class StateMachine {
public:
	static constexpr uint32_t NUM_STATES = 19;
	static constexpr uint32_t NUM_TRANSITIONS = 256;
	CSVState state_machine[NUM_TRANSITIONS][NUM_STATES];
	//! Transitions where we might skip processing
	//! For the Standard State
	bool skip_standard[256];
	//! For the Quoted State
	bool skip_quoted[256];
	//! For the Comment State
	bool skip_comment[256];
	//! The byte patterns that cover every byte the skip tables stop at, for the byte skipper
	vector<SwarBlock::BytePattern> stop_patterns;
	// ...
};

256 × 19 = 4,864 bytes. It fits in L1 cache, and the transition is one indexed load:

 cpp
 
 
// src/include/duckdb/execution/operator/csv_scanner/csv_state_machine.hpp:126
	//! Transition all states to next state, that depends on the current char
	inline void Transition(CSVStates &states, char current_char) const {
		states.states[0] = states.states[1];
		states.states[1] = transition_array[static_cast<uint8_t>(current_char)][static_cast<uint8_t>(states.states[1])];
	}

Two state variables, not one: states[0] is the previous state, states[1] is the current one. The parser needs to distinguish “the delimiter after a quote” from “the delimiter after a value” for strict-mode validation, and the previous state to detect the start of a new row.

The table depends on the dialect: delimiter, quote, escape, comment, newline style, strict mode. Building it is more expensive than parsing a small file, so DuckDB caches it in an object cache keyed by a hash of all those options. The bulk fill is “for all 256 bytes, the transition out of state X is Y” — one loop per state — and then the interesting bytes (delimiter, quote, escape, comment, \r, \n) get their entries overwritten.

The skip tables: which bytes are boring

Next to the transition table, DuckDB builds three bool[256] tables: skip_standard, skip_quoted and skip_comment. A true entry means “this byte does not change anything in this state, so do not run the state machine on it”.

 cpp
 
 
// src/execution/operator/csv_scanner/state_machine/csv_state_machine_cache.cpp:421
	// Initialize characters we can skip during processing, for Standard and Quoted states
	for (idx_t i = 0; i < StateMachine::NUM_TRANSITIONS; i++) {
		transition_array.skip_standard[i] = true;
		transition_array.skip_quoted[i] = true;
		transition_array.skip_comment[i] = true;
	}
	// For standard states we only care for delimiters \r and \n
	transition_array.skip_standard[delimiter_first_byte] = false;
	transition_array.skip_standard[static_cast<uint8_t>('\n')] = false;
	transition_array.skip_standard[static_cast<uint8_t>('\r')] = false;
	transition_array.skip_standard[comment] = false;
	if (enable_unquoted_escape) {
		transition_array.skip_standard[escape] = false;
	}

	// For quoted we only care about quote, escape and for delimiters \r and \n
	transition_array.skip_quoted[quote] = false;
	transition_array.skip_quoted[escape] = false;
	// ...

	auto &stop_patterns = transition_array.stop_patterns;
	stop_patterns.push_back(SwarBlock::BytePattern::Byte(delimiter_first_byte));
	if (quote != '\0') {
		stop_patterns.push_back(SwarBlock::BytePattern::Byte(quote));
	}
	stop_patterns.push_back(SwarBlock::BytePattern::Either('\n', '\r'));
	if (comment != '\0') {
		stop_patterns.push_back(SwarBlock::BytePattern::Byte(comment));
	}
	if (escape != '\0' && escape != quote) {
		stop_patterns.push_back(SwarBlock::BytePattern::Byte(escape));
	}
}

The skip_* tables are the definition of “boring”. The stop_patterns vector is the same information in a form the SWAR skipper can consume: a list of byte patterns that end a run of skippable bytes. There are between 2 and 5 of them: the delimiter (always present), the quote, \n or \r as one Either pattern, the comment marker, and the escape character.

SWAR: 64 bytes at a time

Now the actual trick. Instead of testing one byte at a time against the skip table, DuckDB tests 64 bytes at once using SWAR.

The idea is the classic “SIMD within a register” bit trick. Load eight uint64_ts. For each word, compare it against a repeated pattern using arithmetic, and flag the bytes that match by setting their high bit. Then pack the flags of the eight words into one 64-bit mask, one bit per byte.

The primitives

 cpp
 
 
// src/include/duckdb/common/swar.hpp:20
//! SWAR primitives over the eight bytes of a uint64_t, the masks flag a byte by setting its high bit
struct SwarWord {
	static constexpr idx_t SIZE = sizeof(uint64_t);
	static constexpr uint64_t LSB = 0x0101010101010101ULL;
	static constexpr uint64_t MSB = 0x8080808080808080ULL;

	//! ...
	//! Flags every zero byte of `word` and possibly bytes above one, cheaper than ZeroBytes for callers that verify
	static inline uint64_t MaybeZeroBytes(uint64_t word) {
		return (word - LSB) & ~word & MSB;
	}

MaybeZeroBytes is the workhorse. It is the classic “has zero byte” test, but instead of returning a bool it returns a mask with the high bit set in every byte that is zero (and possibly the byte above a zero byte — hence maybe).

How does it work? word - LSB subtracts one from each byte independently: a byte that was zero becomes 0xFF and borrows from the next byte. The & ~word part isolates the bytes that were zero before the subtraction, and & MSB keeps only the high bit of each byte. The borrows can propagate, so MaybeZeroBytes may flag a byte that is not itself zero — fine, since the caller only needs a superset of the true matches.

From “maybe zero” to “matches a pattern”

To test “does this byte equal c?”, you XOR the word with c repeated in every byte, then look for zero bytes. But the dialect’s bytes are not always a single value — the newline can be \n or \r. DuckDB handles this with a masked pattern: a value and a mask, where a byte matches if it agrees with the value on the bits the mask selects.

 cpp
 
 
// src/include/duckdb/common/swar.hpp:85
	//! A byte pattern repeated over a word, a byte matches when it equals `value` on the bits set in `mask`
	struct BytePattern {
		//! Matches one byte
		static BytePattern Byte(uint8_t byte) {
			return BytePattern(byte, 0xff);
		}
		//! Matches both bytes, and every byte that agrees with them on the bits where they agree
		static BytePattern Either(uint8_t a, uint8_t b) {
			const auto mask = static_cast<uint8_t>(~(a ^ b));
			return BytePattern(a, mask);
		}
		uint64_t value;
		uint64_t mask;
		// ...
	};

Either('\n', '\r') computes ~(0x0A ^ 0x0D) = 0xF7. A byte matches if it equals 0x0A on the bits where 0x0A and 0x0D agree — everywhere except the low four bits. So the pattern matches 0x0A and 0x0D and nothing else in that neighbourhood: one pattern covers both newline conventions. The comparison for a masked pattern is (word & mask) ^ value — zero bytes in the result are the matches.

Building the 64-bit stop mask

 cpp
 
 
// src/include/duckdb/common/swar.hpp:109
	//! Mask of the bytes matching any pattern plus possibly bytes right above a match, no match is ever missed
	static inline uint64_t MaybeAnyMask(const_data_ptr_t block, const vector<BytePattern> &patterns) {
		//! We have between 2 (delimiter/newline) and 5 (delimiter,newline,comment,quote,escape) patterns
		switch (patterns.size()) {
		case 2:
			return MaybeAnyMaskUnrolled<2>(block, patterns);
		// ...
		case 5:
			return MaybeAnyMaskUnrolled<5>(block, patterns);
		default:
			throw InternalException("SwarBlock::MaybeAnyMask takes %d to %d patterns, got %d", MIN_PATTERNS,
			 MAX_PATTERNS, patterns.size());
		}
	}

private:
	//! MaybeAnyMask with the pattern loop unrolled for a count known at compile time
	template <idx_t PATTERN_COUNT>
	static inline uint64_t MaybeAnyMaskUnrolled(const_data_ptr_t block, const vector<BytePattern> &patterns) {
		uint64_t mask = 0;
		for (idx_t i = 0; i < WORDS; i++) {
			const auto word = Load<uint64_t>(block + i * SwarWord::SIZE);
			uint64_t flags = 0;
			for (idx_t p = 0; p < PATTERN_COUNT; p++) {
				flags |= SwarWord::MaybeZeroBytes((word & patterns.get(p).mask) ^ patterns.get(p).value);
			}
			mask |= SwarWord::PackFlags(flags) << (i * SwarWord::SIZE);
		}
		return mask;
	}

Two details worth stealing.

The switch on patterns.size(). The number of patterns is known at runtime but is one of 2, 3, 4 or 5. The switch turns it into a compile-time constant PATTERN_COUNT, so the inner loop unrolls completely and the patterns.get(p).mask / .value loads become constants: 8 words × N patterns of straight-line code, no loop overhead, no bounds checks.

PackFlags. Each word’s flag mask has the high bit of each byte set. PackFlags compresses eight such bytes into eight bits with a multiply by 0x0102040810204080 and a shift by 56 — a well-known bit-gather.

The final mask |= PackFlags(flags) << (i * 8) assembles the eight words into a single 64-bit mask where bit i means “byte i of the block is a stop byte”.

The skipper

 cpp
 
 
// src/include/duckdb/execution/operator/csv_scanner/csv_byte_skipper.hpp:47
	//! Skips bytes that match no stop pattern, `pos` ends on a candidate stop, on `limit`, or in the buffer tail
	inline SkippedTo SkipToCandidate(const idx_t limit, idx_t &pos) {
		while (pos < limit) {
			if (pos < block_start || pos >= block_end) {
				if (pos + SwarBlock::SIZE > buffer_size) {
					return SkippedTo::TAIL;
				}
				MaskBlock(pos);
			}
			const uint64_t candidates = stop_mask >> (pos - block_start);
			if (candidates) {
				const idx_t candidate = pos + CountZeros<uint64_t>::Trailing(candidates);
				if (candidate < limit) {
					pos = candidate;
					return SkippedTo::CANDIDATE_FOUND;
				}
			}
			pos = MinValue<idx_t>(candidates ? limit : block_end, limit);
		}
		return SkippedTo::CANDIDATE_NOT_FOUND;
	}

stop_mask >> (pos - block_start) — the mask is for the whole 64-byte block, so shifting right by the offset within the block aligns bit 0 with the current position.
CountZeros<uint64_t>::Trailing(candidates) — find the lowest set bit, i.e. the first interesting byte in the block. On x86-64 that is the tzcnt/bsf instruction; DuckDB’s implementation is a De Bruijn sequence so it works everywhere:

 cpp
 
 
// src/include/duckdb/common/bit_utils.hpp:49
	inline static idx_t Trailing(uint64_t value_in) {
		if (!value_in) {
			return 64;
		}
		uint64_t value = value_in;

		// ...
		constexpr uint64_t debruijn64lsb = 0x07EDD5E59A4E28C2ULL;
		auto result = index64lsb[((value & -value) * debruijn64lsb) >> 58];
		// ...
		return result;
	}

value & -value isolates the lowest set bit; multiplying by a De Bruijn constant and taking the top 6 bits looks up the index of that bit in a 64-entry table. That is a branch-free, table-lookup count-trailing-zeros that the compiler can turn into a single bsf on architectures that have it.

The last line, pos = MinValue<idx_t>(candidates ? limit : block_end, limit), handles the two ways a block can end: the candidate was past the limit, so stop at the limit; or the mask was empty, so the whole 64-byte block was boring and we jump to its end.

MaskBlock(from) sets stop_mask = SwarBlock::MaybeAnyMask(buffer + from, stop_patterns) and remembers the block bounds.

The arithmetic

For a 500-character free-text field:

Byte-at-a-time with a skip table: 500 iterations, each with a table lookup and a branch. The branch is always not-taken, so it predicts perfectly — but you still pay 500 dependent loads.
SWAR: 8 blocks, each 8 word loads and 8 × N pattern tests of about 4 operations each — roughly 1,300 ALU operations for 64 bytes, but fully pipelined and branch-free.

The win is not the operation count. It is that the branch disappears entirely, and that 64 bytes are examined with eight independent loads instead of 64 dependent ones.

The tail is handled gracefully: if fewer than 64 bytes remain, the skipper returns TAIL and the caller falls back to the byte-at-a-time skip table:

 cpp
 
 
// src/include/duckdb/execution/operator/csv_scanner/csv_byte_skipper.hpp:69
	//! Skips every byte `skip_table` skips, `pos` ends on a stop or on the byte before `to_pos`
	inline void SkipToStop(const bool (&skip_table)[StateMachine::NUM_TRANSITIONS], const idx_t to_pos, idx_t &pos) {
		while (pos + 1 < to_pos) {
			const auto skipped_to = SkipToCandidate(to_pos - 1, pos);
			if (skipped_to == SkippedTo::TAIL) {
				// ...
				return;
			}
			if (skipped_to == SkippedTo::CANDIDATE_NOT_FOUND || !skip_table[ByteAt(pos)]) {
				return;
			}
			// a candidate the skip table skips after all
			pos++;
		}
	}

The !skip_table[ByteAt(pos)] check matters: a candidate is not automatically interesting. The stop patterns are a superset of the skip table’s “interesting” bytes, because the same pattern list is shared across the standard, quoted and comment states. A " is a stop pattern, but inside an unquoted field a quote is just a character.

This is inside the main parse loop:

 cpp
 
 
// src/include/duckdb/execution/operator/csv_scanner/base_scanner.hpp:341
			case CSVState::STANDARD:
				iterator.pos.buffer_pos++;
				skipper.SkipToStop(state_machine->transition_array.skip_standard, to_pos, iterator.pos.buffer_pos);
				break;

The parser reads one byte, transitions the state machine, and then — if the new state is one where long runs of ordinary bytes are possible — it kicks the skipper. So the flow is: transition table for the interesting bytes, SWAR skip for the boring ones.

Making CSV parsing parallel

None of the above would matter if the CSV scan were single-threaded. The other half of the story is how DuckDB splits the file.

CSV is the hard case for parallelism. You cannot split by row count, because finding the row boundaries requires parsing. You cannot split at an arbitrary byte offset, because a value might straddle the cut. DuckDB splits the file into fixed-size byte ranges — “boundaries” — which is O(1) and content-independent:

 cpp
 
 
// src/execution/operator/csv_scanner/scanner/scanner_boundary.cpp:36
idx_t CSVIterator::BytesPerThread(const CSVReaderOptions &reader_options) {
	const idx_t buffer_size = reader_options.buffer_size_option.GetValue();
	const idx_t max_row_size = reader_options.maximum_line_size.GetValue();
	const idx_t bytes_per_thread = buffer_size / CSVBuffer::ROWS_PER_BUFFER * ROWS_PER_THREAD;
	if (bytes_per_thread < max_row_size) {
		// If we are setting up the buffer size directly, we must make sure each thread will read the full buffer.
		return max_row_size;
	}
	if (bytes_per_thread == 0) {
		// Bytes per thread can never be zero, but it might happen if max_row_size = 0
		// Not sure why a human being would do that...
		return 1;
	}
	return bytes_per_thread;
}

With the defaults — CSVBuffer::ROWS_PER_BUFFER = 16 and max_line_size_default = 2,000,000 — the buffer is 32 MB, and ROWS_PER_THREAD = 4 gives 8 MB per thread. The bytes_per_thread < max_row_size guard gives a single thread enough room for one maximal row.

The iterator advances by whole byte ranges:

 cpp
 
 
// src/execution/operator/csv_scanner/scanner/scanner_boundary.cpp:52
bool CSVIterator::Next(CSVBufferManager &buffer_manager, const CSVReaderOptions &reader_options) {
	if (!is_set) {
		return false;
	}
	const auto bytes_per_thread = BytesPerThread(reader_options);

	// If we are calling next this is not the first one anymore
	first_one = false;
	boundary.boundary_idx++;
	// Figure out the shape of our start buffer
	idx_t buffer_actual_size;
	bool is_last_buffer;
	const bool known_ranges = buffer_manager.HasKnownBufferRanges();
	if (known_ranges) {
		// We know exactly the layout of the buffers, can do the whole shebang without IO
		buffer_actual_size = buffer_manager.KnownBufferSize(boundary.buffer_idx);
		is_last_buffer = boundary.buffer_idx + 1 == buffer_manager.KnownBufferCount();
	} else {
		// ...
	}
	if (is_last_buffer && boundary.buffer_pos + bytes_per_thread > buffer_actual_size) {
		// 1) We are done with the current file
		return false;
	} else if (boundary.buffer_pos + bytes_per_thread >= buffer_actual_size) {
		// 2) We still have data to scan in this file, we set the iterator accordingly.
		// We must move the buffer
		boundary.buffer_idx++;
		boundary.buffer_pos = 0;
		// ...
	} else {
		// 3) We are not done with the current buffer, hence we just move where we start within the buffer
		boundary.buffer_pos += bytes_per_thread;
	}
	boundary.end_pos = boundary.buffer_pos + bytes_per_thread;
	SetCurrentPositionToBoundary();
	return true;
}

The known_ranges branch is worth noting: when the file’s buffer layout is known up front, the boundary computation happens with no I/O at all.

A global state hands these boundaries out one at a time:

 cpp
 
 
// src/execution/operator/csv_scanner/table_function/global_csv_state.cpp:106
bool CSVGlobalState::Next(shared_ptr<CSVFileScan> &current_file_ptr, CSVLocalState &lstate) {
	auto &current_file = *current_file_ptr;
	if (!initialized) {
		// initialize the boundary for this file
		// ...
	} else {
		// produce the next boundary for this file
		if (current_boundary.done || !current_boundary.Next(*current_file.buffer_manager, current_file.options)) {
			// finished processing this file - return
			return false;
		}
	}
	// ...
	++current_file.started_tasks;
	// The scanner itself is constructed by the decoding thread when the claim is first scanned
	lstate.scanner_idx = scanner_idx++;
	lstate.iterator = current_boundary;
	// ...
	return true;
}

A boundary is a triple: which buffer, where in it, and where it ends.

The straddling value

The hard case: a value that starts in one boundary and ends in the next. A quoted field containing a newline, a 3 MB text blob, a field split across two 32 MB buffers.

DuckDB’s answer is to re-scan backwards. The scanner remembers where the straddling value started, resets the state machine, and replays the tail of the previous buffer:

 cpp
 
 
// src/execution/operator/csv_scanner/scanner/string_value_scanner.cpp:1420
void StringValueScanner::ProcessOverBufferValue() {
	// Process first string
	if (result.last_position.buffer_pos != previous_buffer_handle->actual_size) {
		states.Initialize();
	}

	string over_buffer_string;
	auto previous_buffer = previous_buffer_handle->Ptr();
	idx_t j = 0;
	result.quoted = false;
	for (idx_t i = result.last_position.buffer_pos; i < previous_buffer_handle->actual_size; i++) {
		state_machine->Transition(states, previous_buffer[i]);
		if (states.EmptyLine() || states.IsCurrentNewRow()) {
			continue;
		}
		if (states.NewRow() || states.NewValue()) {
			break;
		}
		// ...
		j++;
	}

states.Initialize() resets the two-state machine, and the loop replays the tail of the previous buffer with the ordinary Transition call. The value is materialised exactly once, by the thread that owns the boundary where the value ends.

Branchless row selection

The last piece is the output side. When a row fails to parse — a bad cast, a missing column — it has to be removed from the chunk. DuckDB does that with the same compaction trick used everywhere else in the engine: always write, advance only on a match.

 cpp
 
 
// src/execution/operator/csv_scanner/scanner/string_value_scanner.cpp:1228
		if (!result.borked_rows.empty()) {
			// We must remove the borked lines from our chunk
			SelectionVector successful_rows(parse_chunk.size());
			idx_t sel_idx = 0;
			for (idx_t row_idx = 0; row_idx < parse_chunk.size(); row_idx++) {
				if (result.borked_rows.find(row_idx) == result.borked_rows.end()) {
					successful_rows.set_index(sel_idx++, row_idx);
				}
			}
			// Now we slice the result
			insert_chunk.Slice(successful_rows, sel_idx);
			result.borked_rows.clear();
		}

successful_rows.set_index(sel_idx++, row_idx) — the increment only happens on a surviving row, and the index of a dead row is simply overwritten by the next live one. At the end, only the first sel_idx entries are meaningful, and insert_chunk.Slice compacts the chunk through that selection vector.

This is the same “always write, advance only on a match” pattern as the Parquet dictionary filter and the storage layer’s segment pruning, and for the same reason: it turns a data-dependent branch into a data-dependent increment, and data-dependent branches are what the CPU’s predictor is bad at.

What it costs

The transition table: 4,864 bytes per dialect, built once and cached. Building it costs more than parsing a small file, which is why the cache exists.
The SWAR mask: 8 word loads and roughly 8 × N × 5 ALU operations per 64 bytes. For a 2-pattern dialect that is about 2.5 operations per byte.
The tail: the last fewer-than-64 bytes of every buffer are handled byte-at-a-time, and the straddle replay is bounded by one buffer tail per straddling value.

The bottom line

Two ideas do most of the work here.

Precompute the state machine. A CSV parser has 19 states and 256 possible input bytes. That is a finite table, and a finite table can be computed once and indexed instead of branched on. The branch disappears from the hot path; all that is left is a load whose address depends on the previous iteration.

Skip the boring bytes in bulk. Once you know which bytes are boring, you do not have to look at them one at a time. SWAR lets you test 64 bytes with a handful of ALU operations, and one count-trailing-zeros tells you where the interesting byte is. The parser’s cost becomes proportional to the number of interesting bytes in the file rather than the total number of bytes.

The general lesson is that “read every byte” is not the same as “look at every byte”. Whenever a parser has a small set of bytes that matter, you can find them in bulk with bit tricks and turn the positions into a table-driven parse.

Swar on Opensource internals