FSST: How DuckDB Learns a Symbol Table From a Sample of Your Strings

Jay Hubbert — Thu, 22 Oct 2026 12:40:00 +0000

TL;DR: FSST replaces the most common byte sequences in your strings with single bytes, using an alphabet of at most 255 symbols learned once and shared by every string in a block. DuckDB does not learn that alphabet from your whole column: it samples 25% of the rows, builds the alphabet from the sample, actually compresses the sample with it, and extrapolates the column’s size by dividing by the sample rate. It then adds the per-block symbol tables and multiplies the whole estimate by 1.2, so FSST only wins when it is clearly better than the boring codecs.

The scenario

You have a VARCHAR column with four million URLs in it:

text

https://api.internal.example.com/v2/accounts/8817/transactions?page=3
https://api.internal.example.com/v2/accounts/9921/transactions?page=1
https://api.internal.example.com/v2/accounts/4410/transactions?page=7
...

Average length: 40 bytes. Raw size: 160 MB. The strings are highly redundant — the first 44 bytes are the same in every single row — but no two rows are identical, so a dictionary codec is useless. A general-purpose compressor like ZSTD would find the redundancy, but ZSTD’s decompression is a chain of Huffman and FSE decoders, which is a lot of work per byte.

FSST is the middle path: it finds the redundancy, but it does so with a flat table lookup per symbol.

What FSST actually is

FSST stands for Fast Static Symbol Table. The idea is small enough to explain in a paragraph.

You get an alphabet of at most 255 symbols. Each symbol is a byte sequence of 1 to 8 bytes. To compress a string, you walk it from left to right and, at each position, look up the longest symbol that matches. Each match is written as one byte: the symbol’s code. If no symbol matches, you emit the escape code 255 followed by the raw byte:

 c
 
 
// third_party/fsst/fsst.h:92
/* A compressed string is simply a string of 1-byte codes; except for code 255, which is followed by an uncompressed byte. */
#define FSST_ESC 255

The decoder is the whole point. It is a flat array of 255 entries:

 c
 
 
// third_party/fsst/fsst.h:99
typedef struct {
 unsigned long long version; /* version id */
 unsigned char zeroTerminated; /* terminator is a single-byte code that does not appear in longer symbols */
 unsigned char len[255]; /* len[x] is the byte-length of the symbol x (1 < len[x] <= 8). */
 unsigned long long symbol[255]; /* symbol[x] contains in LITTLE_ENDIAN the bytesequence that code x represents (0 <= x < 255). */
} duckdb_fsst_decoder_t;

Decompression is: read a byte, look up len[code] and symbol[code], copy that many bytes, advance. One table lookup per symbol, no branches, no dictionary of strings, no hash table. And duckdb_fsst_decoder_t is 2312 bytes, so it fits comfortably in L1.

The catch is that the alphabet has to be learned, and it has to be learned from the data you are about to compress. That is where DuckDB’s implementation gets interesting.

Learning the alphabet

The upstream FSST library learns the alphabet with a small greedy search that is worth following closely, because it is a nice piece of engineering.

Step 1: pick a terminator

FSST needs a byte value that cannot appear inside a symbol, so that the decoder knows a symbol ended. The library picks the least frequent byte in the sample:

 c
 
 
// third_party/fsst/libfsst.cpp:72
	} else {
		u16 byteHisto[256];
		memset(byteHisto, 0, sizeof(byteHisto));
		for(size_t i=0; i<line.size(); i++) {
			const u8* cur = line[i];
			const u8* end = cur + len[i];
			while(cur < end) byteHisto[*cur++]++;
		}
		u32 minSize = FSST_SAMPLEMAXSZ, i = st->terminator = 256;
		while(i-- > 0) {
			if (byteHisto[i] > minSize) continue;
			st->terminator = i;
			minSize = byteHisto[i];
		}
	}

Rare bytes make good terminators because they are cheap to exclude from the alphabet.

Step 2: five rounds of “count, then rebuild”

The core loop runs five times with an increasing sample fraction:

 c
 
 
// third_party/fsst/libfsst.cpp:230
	for(sampleFrac=8; true; sampleFrac += 30) {
		memset(&counters, 0, sizeof(Counters));
		long gain = compressCount(st, counters);
		if (gain >= bestGain) { // a new best solution!
			counters.backup1(bestCounters);
			*bestTable = *st; bestGain = gain;
		}
		if (sampleFrac >= 128) break; // we do 5 rounds (sampleFrac=8,38,68,98,128)
		makeTable(st, counters);
	}

So the fractions are 8, 38, 68, 98, 128 — the first four rounds look at progressively more of the sample, and the last round (128) looks at all of it and only measures; it does not rebuild the table.

In each round, compressCount walks the sample with the current alphabet and tallies two things into the Counters struct:

single-symbol counts: how often each existing symbol was used, and how often each literal byte was used.
symbol-pair counts: how often symbol x was immediately followed by symbol y. These are the candidates for new, longer symbols.

Step 3: greedily grow the alphabet

makeTable turns the counts into candidate symbols, scores them, and takes the best:

 c
 
 
// third_party/fsst/libfsst.cpp:170
		auto addOrInc = [&](std::unordered_set<QSymbol> &cands, Symbol s, u64 count) {
			if (count < (5*sampleFrac)/128) return; // improves both compression speed (less candidates), but also quality!!
			QSymbol q;
			q.symbol = s;
			q.gain = count * s.length();
			auto it = cands.find(q);
			if (it != cands.end()) {
				q.gain += (*it).gain;
				cands.erase(*it);
			}
			cands.insert(q);
		};

The gain of a candidate symbol is count × length: how many bytes you save by using this symbol instead of its parts. A two-byte symbol used 1000 times saves 1000 bytes; a one-byte symbol used 1000 times saves 1000 bytes too — but a one-byte symbol is also usable in more places, so the library promotes single-byte candidates by a factor of 8:

 c
 
 
// third_party/fsst/libfsst.cpp:189
			Symbol s1 = st->symbols[pos1];
			addOrInc(cands, s1, ((s1.length()==1)?8LL:1LL)*cnt1);

The candidates go into a priority queue ordered by gain, and the table is rebuilt from the top down, stopping at 255 symbols:

 c
 
 
// third_party/fsst/libfsst.cpp:218
		while (st->nSymbols < 255 && !pq.empty()) {
			QSymbol q = pq.top();
			pq.pop();
			st->add(q.symbol);
		}

That is the whole algorithm. It is a crude Huffman: count frequencies, greedily merge the most valuable pairs, stop at a hard limit of 255 symbols so that a symbol is exactly one byte.

DuckDB’s twist: you do not need the whole column

Here is where DuckDB departs from upstream FSST. The upstream library wants you to hand it the entire batch of strings. DuckDB hands it a quarter of the column.

 cpp
 
 
// src/storage/compression/fsst.cpp:104
bool FSSTStorage::StringAnalyze(AnalyzeState &state_p, const Vector &input) {
	auto &state = state_p.Cast<FSSTAnalyzeState>();
	UnifiedVectorFormat vdata;
	input.ToUnifiedFormat(vdata);

	const auto count = input.size();
	state.count += count;
	auto data = UnifiedVectorFormat::GetData<string_t>(vdata);

	// Note that we ignore the sampling in case we have not found any valid strings yet, this solves the issue of
	// not having seen any valid strings here leading to an empty fsst symbol table.
	bool sample_selected = !state.have_valid_row || state.random_engine.NextRandom() < ANALYSIS_SAMPLE_SIZE;

	for (idx_t i = 0; i < count; i++) {
		auto idx = vdata.sel->get_index(i);

		if (!vdata.validity.RowIsValid(idx)) {
			continue;
		}

		// We need to check all strings for this, otherwise we run in to trouble during compression if we miss ones
		auto string_size = data[idx].GetSize();
		if (string_size >= StringUncompressed::GetStringBlockLimit(state.info.GetBlockSize())) {
			return false;
		}

		if (!sample_selected) {
			continue;
		}

Three things to notice.

The sampling is per vector, not per row. sample_selected is decided once for the whole 2048-row vector, using a random number from the analyze state’s own RandomEngine. So the sample is a random subset of vectors, and every string in a selected vector goes in. This matters: it keeps the sample cheap (one random draw per 2048 rows) and it keeps the sample representative of whole vectors, which is what you want if your data is clustered.

The first valid string is always taken. !state.have_valid_row short-circuits the sampling. The comment explains why: if the very first vector happened to be skipped, the analyze state would have no strings at all and the symbol table would come out empty.

The size check happens for every string, sampled or not. A string bigger than the block limit cannot be FSST-encoded at all, so the codec bails out of the auction entirely with return false. This is the early-disqualification path from the previous article.

The constants are right at the top of the file:

 cpp
 
 
// src/storage/compression/fsst.cpp:37
struct FSSTStorage {
	static constexpr double MINIMUM_COMPRESSION_RATIO = 1.2;
	static constexpr double ANALYSIS_SAMPLE_SIZE = 0.25;

ANALYSIS_SAMPLE_SIZE = 0.25 — a quarter of the rows. MINIMUM_COMPRESSION_RATIO = 1.2 — the penalty factor, applied at the very end.

The extrapolation

Here is the interesting part. Having learned an alphabet from the sample, DuckDB does not guess how well it would do on the rest of the column. It actually compresses the sample with the learned encoder and measures the result.

 cpp
 
 
// src/storage/compression/fsst.cpp:171
	state.fsst_encoder = duckdb_fsst_create(string_count, fsst_string_sizes, fsst_string_ptrs, 0);

duckdb_fsst_create is the upstream entry point that runs the five-round search we walked through above. Then:

 cpp
 
 
// src/storage/compression/fsst.cpp:180
	auto res = duckdb_fsst_compress(state.fsst_encoder, string_count, fsst_string_sizes, fsst_string_ptrs,
	 output_buffer_size, compressed_buffer.get(), compressed_sizes, compressed_ptrs);

This is a real compression of the sample. compressed_sizes[i] is the actual compressed length of sample string i. The source even leaves a note about whether this was necessary:

 cpp
 
 
// src/storage/compression/fsst.cpp:173
	// TODO: do we really need to encode to get a size estimate?

It does need to, because the compressed length distribution is what determines the width of the bitpacked offset table, and the offset table is a real cost.

And now the extrapolation:

 cpp
 
 
// src/storage/compression/fsst.cpp:187
	// Sum and and Max compressed lengths
	for (idx_t i = 0; i < string_count; i++) {
		const auto size = compressed_sizes[i];
		compressed_dict_size += size;
		max_compressed_string_length = MaxValue(max_compressed_string_length, size);
	}
	D_ASSERT(compressed_dict_size ==
	 (uint64_t)(compressed_ptrs[res - 1] - compressed_ptrs[0]) + compressed_sizes[res - 1]);

	auto minimum_width = BitpackingPrimitives::MinimumBitWidth(max_compressed_string_length);
	auto bitpacked_offsets_size =
	 BitpackingPrimitives::GetRequiredSize(string_count + state.empty_strings, minimum_width);

	auto estimated_base_size = double(bitpacked_offsets_size + compressed_dict_size) * (1 / ANALYSIS_SAMPLE_SIZE);
	auto num_blocks = estimated_base_size / double(state.info.GetBlockSize() - sizeof(duckdb_fsst_decoder_t));
	auto symtable_size = num_blocks * sizeof(duckdb_fsst_decoder_t);
	auto estimated_size = estimated_base_size + symtable_size;

	return LossyNumericCast<idx_t>(estimated_size * MINIMUM_COMPRESSION_RATIO);

Four steps:

The dictionary size is the sum of the compressed lengths. This is the payload.
The offset table size: the compressed lengths are stored bitpacked at MinimumBitWidth(max_compressed_string_length) bits each. GetRequiredSize rounds the count up to a multiple of BITPACKING_ALGORITHM_GROUP_SIZE (32) and multiplies.
Extrapolation: multiply by 1 / ANALYSIS_SAMPLE_SIZE = 4, because we only measured a quarter of the rows.
The symbol tables: the alphabet has to be stored once per block, because each 256 KiB block has its own compressed strings that need their own decoder. So the estimate adds num_blocks × sizeof(duckdb_fsst_decoder_t).

And finally the penalty: estimated_size * MINIMUM_COMPRESSION_RATIO = ×1.2.

The same arithmetic, with numbers

Let’s run our four-million-URL column through it.

Sample: 25% of 4,000,000 rows ≈ 1,000,000 strings (≈488 of the ~1953 vectors).

Suppose the learned alphabet compresses the average URL from 40 bytes to 24 bytes, and the longest compressed sample string is 40 bytes.

compressed_dict_size = 1,000,000 × 24 = 24,000,000 bytes
max_compressed_string_length = 40 → MinimumBitWidth(40) = 6 bits
bitpacked_offsets_size = GetRequiredSize(1,000,000, 6) = 1,000,000 (already a multiple of 32) × 6 / 8 = 750,000 bytes
estimated_base_size = (750,000 + 24,000,000) × 4 = 99,000,000 bytes
num_blocks = 99,000,000 / (262,136 − 2,312) ≈ 381 blocks
symtable_size = 381 × 2,312 ≈ 880,850 bytes
estimated_size = 99,000,000 + 880,850 = 99,880,850 bytes
final score = 99,880,850 × 1.2 = 119,857,020 bytes

Now compare against the uncompressed codec’s score, which is essentially 4,000,000 × 40 = 160,000,000 bytes. FSST’s bid of ~120 MB beats 160 MB, so FSST wins the auction.

But look at what the 1.2 factor did. The true estimated size was 99.9 MB, i.e. a genuine 1.60x over the raw data. After the penalty the bid is 119.9 MB, an effective 1.335x. FSST has to be at least 1.2x better than raw just to break even against the penalty. That is a deliberate bias in favour of the boring codecs, and it is exactly what you want: FSST’s estimate is an extrapolation from a sample, so it deserves a margin of error.

Why the symbol table term matters

Notice that the symbol tables add 880 KB to a 99.9 MB estimate — about 0.9%. Now consider a much smaller column: 4,000 rows of 40-byte strings. The sample is 1,000 strings.

compressed_dict_size = 1,000 × 24 = 24,000
bitpacked_offsets_size = GetRequiredSize(1,000, 6) = 1,024 × 6/8 = 768
estimated_base_size = (768 + 24,000) × 4 = 99,072 bytes
num_blocks = 99,072 / 259,824 = 0.38 → symtable_size = 0.38 × 2,312 ≈ 883 bytes

Still small, but the ratio is different: 883 bytes of symbol table for 99 KB of data. And if the strings were short — say 8 bytes each — the symbol table would dominate completely. This is exactly why the estimate has to include the per-block overhead rather than just the compressed payload: FSST loses on short strings, and the auction needs to see that.

The on-disk layout

Now that we know what is stored, here is how it is laid out inside a 256 KiB block.

The strings are written into a “dictionary” that grows downwards from the end of the block, and only the lengths are stored in the forward-growing index buffer:

 cpp
 
 
// src/storage/compression/fsst.cpp:242
	void UpdateState(string_t uncompressed_string, unsigned char *compressed_string, size_t compressed_string_len) {
		if (!HasEnoughSpace(compressed_string_len)) {
			Flush();
			if (!HasEnoughSpace(compressed_string_len)) {
				throw InternalException("FSST string compression failed due to insufficient space in empty block");
			};
		}

		stats_writer.Update(uncompressed_string);

		// Write string into dictionary
		current_dictionary.size += compressed_string_len;
		auto dict_pos = current_end_ptr - current_dictionary.size;
		memcpy(dict_pos, compressed_string, compressed_string_len);
		current_dictionary.Verify(info.GetBlockSize());

		// We just push the string length to effectively delta encode the strings
		index_buffer.push_back(NumericCast<uint32_t>(compressed_string_len));

		max_compressed_string_length = MaxValue(max_compressed_string_length, compressed_string_len);

		current_width = BitpackingPrimitives::MinimumBitWidth(max_compressed_string_length);
		current_segment->count++;
	}

Two ideas here.

Storing lengths instead of offsets makes the offsets implicit. The reader keeps a running cumulative offset and recovers each individual length by subtracting the previous one. The lengths are then bitpacked at current_width bits, a width that is recomputed on every string and therefore only ever grows (the maximum is monotonic).

The dictionary grows downwards so that the symbol table can live at a known offset near the end of the block while the payload fills from the start.

When the block is flushed, if it is less than 80% full, the dictionary is memmoved forward so it butts up against the symbol table, and the segment’s reported size shrinks to the exact number of bytes used:

 cpp
 
 
// src/storage/compression/fsst.cpp:369
		if (total_size >= info.GetCompactionFlushLimit()) {
			// the block is full enough, don't bother moving around the dictionary
			// NOTE: We forgot to call SetDictionary here, so dict_size on disk is stale.
			// not worth fixing as it's a legacy encoder, the reader can be bounded by dict_end
			return info.GetBlockSize();
		}

		// the block has space left: figure out how much space we can save
		auto move_amount = info.GetBlockSize() - total_size;
		// move the dictionary so it lines up exactly with the offsets
		auto new_dictionary_offset = symbol_table_offset + fsst_serialized_symbol_table_size;
		memmove(base_ptr + new_dictionary_offset, base_ptr + current_dictionary.end - current_dictionary.size,
		 current_dictionary.size);
		current_dictionary.end -= move_amount;
		D_ASSERT(current_dictionary.end == total_size);
		// write the new dictionary (with the updated "end")
		FSSTStorage::SetDictionary(*current_segment, handle, current_dictionary);

		return total_size;
	}

GetCompactionFlushLimit() is four fifths of the block size. This is an in-place compaction: no second copy of the block is written, the bytes are just shuffled within it.

Scanning it back

Because the offsets are cumulative, decoding string k requires knowing the offset of string k−1. DuckDB handles this with two tricks.

Trick one: a remembered cursor. The scan state caches the last decoded cumulative offset and the row it belongs to. A scan that continues where the previous one stopped resumes the running offset instead of recomputing it from the start of the block.

Trick two: bitunpack alignment. Bitpacking has a hard alignment requirement — groups of 32 values — so a partial scan has to decode a bit more than it needs and then throw the extra away. The offsets struct spells out the arithmetic:

 cpp
 
 
// src/storage/compression/fsst.cpp:24
// Counts and offsets used during scanning/fetching
// | ColumnSegment to be scanned / fetched from				 |
// | untouched | bp align | unused d-values | to scan | bp align | untouched |
typedef struct BPDeltaDecodeOffsets {
	idx_t delta_decode_start_row; // X
	idx_t bitunpack_alignment_offset; //			 <--------->
	idx_t bitunpack_start_row; //	 X
	idx_t unused_delta_decoded_values; //						 <----------------->
	idx_t scan_offset; //			 <---------------------------->
	idx_t total_delta_decode_count; //					 <-------------------------->
	idx_t total_bitunpack_count; // <------------------------------------------------>
} bp_delta_offsets_t;

and the calculation:

 cpp
 
 
// src/storage/compression/fsst.cpp:916
// The calculation of offsets and counts while scanning or fetching is a bit tricky, for two reasons:
// - bitunpacking needs to be aligned to BITPACKING_ALGORITHM_GROUP_SIZE
// - delta decoding needs to decode from the last known value.
bp_delta_offsets_t FSSTStorage::CalculateBpDeltaOffsets(int64_t last_known_row, idx_t start, idx_t scan_count) {

The net effect: a scan of n rows touches O(n + 32) values, not O(rows from the start of the block).

Late decompression

The last trick is the most interesting one for a scan, and it is unique to FSST.

Normally, a string scan decompresses every string it reads, whether or not the consumer cares. FSST can skip that. Instead of producing plain strings, the scanner produces a special vector whose payload is the still-compressed byte ranges plus a pointer to the FSST decoder. Decompression then happens lazily, one string at a time, only when something actually touches the value.

 cpp
 
 
// src/storage/compression/fsst.cpp:733
	if (enable_fsst_vectors) {
		D_ASSERT(result_offset == 0);
		auto string_block_limit = StringUncompressed::GetStringBlockLimit(segment.GetBlockSize());
		FSSTVector::Create(result, scan_state.duckdb_fsst_decoder, string_block_limit, scan_count);
		result_data = FSSTVector::GetCompressedData(result);
	} else {
		D_ASSERT(result.GetVectorType() == VectorType::FLAT_VECTOR);
		result_data = FlatVector::GetDataMutable<string_t>(result);
	}

	auto strings = scan_state.ValidateRange(start, scan_count);
	if (enable_fsst_vectors) {
		// The result could outlive the ScanState, so keep the segment block pinned with the result vector
		auto &buffer_manager = BufferManager::GetBufferManager(segment.GetDatabase());
		StringVector::AddHandle(result, buffer_manager.Pin(state.context, segment.GetBlockHandle()));
		for (idx_t i = 0; i < scan_count; i++) {
			auto compressed_string = strings.GetCompressedString(i);
			result_data[i] = string_t(const_char_ptr_cast(compressed_string.data()),
			 NumericCast<uint32_t>(compressed_string.size()));
		}
		FSSTVector::SetCount(result, scan_count);
	} else {
		// Just decompress
		auto &str_allocator = StringVector::GetStringAllocator(result);
		for (idx_t i = 0; i < scan_count; i++) {
			result_data[i + result_offset] = scan_state.DecompressString(strings.GetCompressedString(i), str_allocator);
		}
	}
}

The lifetime problem is real: this vector can outlive the scan state, so the block holding the compressed bytes must stay pinned. StringVector::AddHandle attaches the BufferHandle to the result vector, so the memory stays valid for exactly as long as the data does.

And the payoff is that a query which filters on this column can throw away rows it never looks at, and the decompression cost for those rows is never paid.

The costs

The sample is real memory. The analyze state keeps a quarter of the column’s strings in memory so it can build an alphabet and compress them. For a 100 GB string column that is 25 GB of string data during the checkpoint. This is bounded by the memory limit, but it is not nothing.

The estimate is an extrapolation. A quarter of the vectors might not be representative. If the column is sorted by the string value, the sample of vectors is spread across the whole value range, which is good. If the column is clustered such that whole vectors share long prefixes, the sample could be skewed.

The 1.2 penalty is a heuristic, not a guarantee. It is a fixed number. If FSST would actually achieve 1.25x, the penalty turns that into a bid of 1.04x, and FSST loses to uncompressed even though it would have been a win. The penalty is a bet that the sample extrapolation is usually slightly optimistic.

FSST does not compress short strings well. The per-block symbol table is 2,312 bytes. On a column of 4-byte codes, that is fatal. The estimate correctly reflects this, so the auction picks something else.

A new format, new reader. The FSST format is only used for storage versions before StorageVersion::V1_3_0; newer versions use dict_fsst (dictionary-encode first, then FSST the unique entries). If you force COMPRESSION_FSST on a modern database you are opting into the old encoder.

The bottom line

FSST is a good example of a compression scheme being designed around the scan, not around the file. It has three layers:

A flat 255-entry decoder table, so decompression is one lookup per symbol and no per-string dictionary is needed.
A learned alphabet, built by a five-round greedy search that counts symbol and symbol-pair frequencies and merges the highest-gain pairs until it hits 255 symbols.
A measurement, not a guess: DuckDB compresses a 25% sample with the learned alphabet, measures the actual result, extrapolates by 4x, adds the per-block symbol tables, and applies a 1.2x margin.

That last layer is the transferable lesson. When you are choosing a format for a column, the interesting question is not “how well does this codec compress in theory” but “how big will this column be on disk, including every overhead I will have to pay”. FSST’s estimate answers exactly that question, in the same currency as every other codec in the auction.

And the payoff is not only bytes: because the scanner can hand back still-compressed strings and expand them lazily, a filter on a string column can discard rows without ever paying to decompress them.

Fsst on Opensource internals