Rust on Opensource internals

A Tour of rustc's Internals: The Thirty-Eight Tricks That Make the Compiler Fast

Jay Hubbert — Wed, 21 Oct 2026 13:10:00 +0000

TL;DR: rustc is a compiler that has spent twenty years turning “the obvious implementation” into “the slow implementation, replaced by something clever”. This is a tour of thirty-eight of those somethings — from an 8-byte Span with four internal layouts to a dataflow engine with no worklist — and where in the source tree each one lives.

rustc --version prints a version number. What it does not print is the several hundred small decisions that make a large crate compile in seconds rather than minutes. This is a tour of the ones that are worth knowing about, grouped by the part of the compiler they live in.

1. Everything is a query, and nothing is cached

rustc’s compiler is built on a query system: every computation is a memoized, dependency-tracked node in a DAG. The interesting parts are the peripherals.

The colour map that is also an index — one u32 per node holds red, green and the current-session index, so there is no side table.
Green means same — incremental compilation re-runs queries and compares 128-bit fingerprints, so fingerprints have to be reversible: DefIds hash to a stable DefPathHash.
Four shapes of cache — a OnceLock for unit keys, a dense vector cache for local DefIds, a sharded map for the rest; the slot packs “initialised?” and the dep-node index into one atomic word.
The lock that disappears — the thread count is a compile-time decision, so a single-threaded build has zero atomic instructions in its cache path.
38 bytes and a mask — the dependency graph file has fixed-size node headers, varint edge lists that stay varint in memory, and an encoder that hands out node indices in blocks of 256.

2. The borrow checker solves in one pass

Region inference is not a fixpoint over regions; it is a fixpoint over an SCC condensation of the region graph.

One pass, no fixpoint — regions in a cycle are equal by construction, so the condensation is a DAG and one topological sweep suffices.
end(X) doesn’t live in the CFG — region values are three independent dimensions (points, free regions, placeholders), so the “end of lifetime X” marker costs no extra CFG points and the interval sets stay contiguous.
The blame is the most interesting constraint — a BFS finds the shortest path through the constraint graph, then a hand-tuned ranking picks the most user-facing constraint on it, and the expensive constraint objects are built only on the error path.
No queue at all — the dataflow framework’s dirty set is a bitset indexed by reverse-postorder rank; rewinding to a lower rank on a back edge is both the correctness rule and the reason loop-free code is analysed in a single pass.
Two-phase borrows are dominator queries — v.push(v.len()) works because “is this borrow active here?” is answered with two tree queries instead of an extra dataflow analysis.

3. The trait solver thinks in lattices

Coinduction for free — cycles are classified Inductive < Unknown < Coinductive, joined with a max-like operation, and the kind is recomputed compositionally on every stack push.
Divide by four — after an overflow the recursion budget is divided rather than decremented, which turns a quadratic blowup into a geometric one.
Proof trees, not errors — the solver never formats a message; it records a replayable trace and reconstructs the error afterwards, with the cache disabled so the replay is faithful.

4. MIR optimisation is where the sharp edges are

GVN refuses to merge two borrows — every pointer value carries a marker type whose Hash and PartialEq are unreachable!(), so “accidentally unify two &muts” is impossible rather than unlikely.
The call graph from the wrong MIR — the inliner computes the call graph from pre-optimization MIR to avoid a query cycle, and gives up with an eighth of the recursion limit so the cheap analysis fails before the expensive one.
Four ways to drop — drop sites are classified dead, static, conditional or open; only the conditional case allocates a flag, so most drops cost nothing.
Drop glue gets four times the budget — the monomorphiser divides the recursion depth by four for drop glue only, because dropping a value legitimately recurses through the type structure.
Merging codegen units by overlap — CGUs are merged by inlined-item overlap, not by size, because duplicated inlined code is what makes LLVM’s job harder.
Sorting fields and hiding a niche — the layout algorithm sorts fields by log2(max(align, size)) and reserves the biggest invalid-value hole as the discriminant, which is why Option<&T> is one pointer.

5. The front end never fails, and never copies

The lexer that never fails — unterminated strings and stray characters are token kinds; the lexer is a total function from bytes to tokens, and it refuses to be greedy so 0..2 and 12.foo() tokenize correctly.
One token, two operators — >> is split on demand with byte-exact spans, so a proc macro sees exactly what the source said.
Capturing tokens by replaying them — the captured range is a half-open interval of bump counts; the tokens are materialised only if something asks for them, which is rare.
macro_rules! is an NFA — the matcher keeps a set of live positions and follows epsilon transitions, so there is no backtracking and no exponential blowup.
Did you mean a hyphen? — a static confusables table doubles as the recovery mechanism and the diagnostic source.

6. Const evaluation is one interpreter with two personalities

One interpreter, two machines — the Machine trait has eight associated types and 57 hooks; Miri and CTFE are two instantiations, so Miri gets provenance tracking for free.
The init mask is a single boolean — uninitialised-byte tracking starts as one bool and only materialises a bitmap on a partial write; a full overwrite de-materialises it again.
Provenance in eight bytes — an allocation id plus two flags fit in one NonZero<u64>, and the provenance of stored pointers is sparse relocations rather than per-byte metadata.

7. The standard library is where the bytes are

Vec’s capacity lives in a niche — RawVec stores the capacity in a UsizeNoHighBit, so Vec is three words instead of four.
collect() reuses your allocation — in-place collection turns map + collect into a loop with no allocation, no copy and no free.
Sixteen buckets at a time — the SwissTable compares 16 control bytes with one SSE2 instruction, and the top 7 bits of the hash filter candidates before any key comparison.
SipHash’s field order is a SIMD hint — v0, v2, v1, v3 instead of v0, v1, v2, v3, so the compiler can find two SIMD lanes by itself.
Thirteen elements, forty-five comparisons — small slices are sorted with a fixed, branchless sorting network.
One XOR and one CLZ — the powersort merge-tree depth is computed from the leading zeros of an XOR, not from rational comparisons.
Sixteen bytes at a time — UTF-8 validation is an ASCII scanner with a Unicode-aware fallback.

8. And the data structures underneath everything

Always bump downwards — arenas allocate downwards from the end of a chunk and use hint::assert_unchecked to tell LLVM the alignment invariant holds, deleting the alignment arithmetic.
How a span fits in eight bytes — four internal layouts selected by a prefix, decoded by a macro because the enum version did not optimise; and SyntaxContext deliberately does not implement Ord, because its allocation order is nondeterministic.
Tables that pick their own integer width — metadata tables choose their element width at encode time by stripping trailing zeros, and byte-interleave integer pairs so both halves compress at once.
Pruning irrelevant pattern columns — a relevancy flag per matrix row turns an exponential exhaustiveness check into a linear one.

The themes

Read the list again and four patterns emerge.

Representation follows the common case. The init mask is a bool until it cannot be; the provenance map is None until it cannot be; the capacity is a niche until it cannot be. The rare case should never cost the common case.

Delete work rather than optimise it. No worklist. No error messages during solving. No token collection for nodes that do not need tokens. No locks in a single-threaded build. Most of the speed comes from not doing the thing.

Change the representation to change the cost. Powersort’s merge depth becomes an XOR and a lzcnt because the number line is rescaled. Region values become three dimensions so end(X) is free. Metadata elements are byte-interleaved so one stripping pass compresses two integers.

Make the illegal state unrepresentable. VnOpaque’s panicking PartialEq. SyntaxContext’s missing Ord. The assert_unchecked alignment invariant. The compile-time size budgets on query keys and values. These are not optimisations; they are the reason the optimisations are safe.

Where to start reading

If you want to read the source with a plan rather than at random:

compiler/rustc_middle/src/dep_graph/ — the incremental machinery, which is self-contained and unusually well commented.
compiler/rustc_mir_dataflow/src/framework/ — a small, elegant framework you can read in one sitting.
compiler/rustc_expand/src/mbe/macro_parser.rs — the NFA, with a module-level tutorial that is better than most blog posts.
library/core/src/slice/sort/ — driftsort and ipnsort, and the 13-element sorting network.
library/alloc/src/vec/ — specialisation-driven collect, in-place collection, the niche-packed capacity.

The compiler’s own documentation is at https://rustc-dev-guide.rust-lang.org/, and the rustc-dev-guide source lives in src/doc/rustc-dev-guide/ in the same repository.

Pruning Irrelevant Pattern Columns: How rustc's Match Checker Went From Exponential to Linear

Jay Hubbert — Wed, 21 Oct 2026 13:05:00 +0000

TL;DR: rustc’s match exhaustiveness checker implements Maranget’s usefulness algorithm, specialised for Rust’s patterns. The interesting part is an extra relevancy rule: when a column contains a wildcard and some constructors are missing, specializing with those missing constructors subsumes the others — so the checker can skip whole subtrees. In one pathological case that turned exponential work into linear work.

The algorithm

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:8-11, 110-135

//! The algorithm implemented here is inspired from the one described in [this
//! paper](http://moscova.inria.fr/~maranget/papers/warn/index.html). We have however changed it in
//! various ways to accommodate the variety of patterns that Rust supports. We thus explain our
//! version here, without being as precise.
...
//! # Specialization
//!
//! The examples in the previous section motivate the operation at the heart of the algorithm:
//! "specialization". It captures this idea of "removing one layer of constructor".
//!
//! `specialize(c, p)` takes a value-only constructor `c` and a pattern `p`, and returns a
//! pattern-tuple or nothing. It works as follows:
//!
//! - Specializing for the wrong constructor returns nothing
//!
//! - `specialize(None, Some(p0)) := <nothing>`
//! - `specialize([,,,], [p0]) := <nothing>`
//!
//! - Specializing for the correct constructor returns a tuple of the fields
//!
//! - `specialize(Variant1, Variant1(p0, p1, p2)) := (p0, p1, p2)`
//! - `specialize(Foo{ bar, baz, quz }, Foo { bar: p0, baz: p1, .. }) := (p0, p1, _)`
//! - `specialize([,,,], [p0, .., p1]) := (p0, _, _, p1)`

The core operation is specialization: peel one constructor off the head of a pattern, replacing it with its sub-fields. A matrix of patterns is specialized column by column; a pattern is useful if it can match something the other patterns cannot; and the whole matrix is exhaustive if a virtual wildcard row is not useful.

The blowup

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:302-361

//! Relevancy allowed us to skip the case `(true, true, _)` entirely. In some cases this pruning can
//! give drastic speedups. The case this was built for is the following (#118437):
//!
//! ```ignore(illustrative)
//! match foo {
//! (true, _, _, _, ..) => 1,
//! (_, true, _, _, ..) => 2,
//! (_, _, true, _, ..) => 3,
//! (_, _, _, true, ..) => 4,
//! ...
//! }
//! ```
//!
//! Without considering relevancy, we would explore all 2^n combinations of the `true` and `Missing`
//! constructors. Relevancy tells us that e.g. `(true, true, false, false, false, ...)` is
//! irrelevant for all the rows. This allows us to skip all cases with more than one `true`
//! constructor, changing the runtime from exponential to linear.

That is the whole story of the optimisation, told in the source. A match with n boolean-ish columns where each row fixes exactly one column would require exploring all 2ⁿ combinations of constructors.

The implementation

Each row of the matrix carries a relevancy flag:

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:1018-1025

struct PatStack<'p, Cx: PatCx> {
 // Rows of len 1 are very common, which is why `SmallVec[_; 2]` works well.
 pats: SmallVec<[PatOrWild<'p, Cx>; 2]>,
 /// Sometimes we know that as far as this row is concerned, the current case is already handled
 /// by a different, more general, case. When the case is irrelevant for all rows this allows us
 /// to skip a case entirely. This is purely an optimization. See at the top for details.
 relevant: bool,
}

The flag is propagated during specialization:

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:1081-1085

// `ctor` is relevant for this row if it is the actual constructor of this row, or if the
// row has a wildcard and `ctor` is relevant for wildcards.
let ctor_is_relevant =
 !matches!(self.head().ctor(), Constructor::Wildcard) || ctor_is_relevant;
Ok(PatStack { pats: new_pats, relevant: self.relevant && ctor_is_relevant })

and the subtree is abandoned as soon as no row is relevant:

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:1704-1709

if !matrix.wildcard_row_is_relevant && matrix.rows().all(|r| !r.pats.relevant) {
 // Here we know that nothing will contribute further to exhaustiveness or usefulness. This
 // is purely an optimization: skipping this check doesn't affect correctness. See the top of the
 // file for details.
 return Ok(WitnessMatrix::empty());
}

The rule that decides relevancy per constructor:

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:1739-1748

for ctor in split_ctors {
 // Dig into rows that match `ctor`.
 debug!("specialize({:?})", ctor);
 // `ctor` is *irrelevant* if there's another constructor in `split_ctors` that matches
 // strictly fewer rows. In that case we can sometimes skip it. See the top of the file for
 // details.
 let ctor_is_relevant = matches!(ctor, Constructor::Missing)
 || missing_ctors.is_empty()
 || mcx.tycx.exhaustive_witnesses();

The intuition: if a column contains a wildcard row and some constructors are missing from that column, then the missing constructor already matches “everything the wildcard matches, plus more”. Specializing with any other constructor therefore yields no more information than specializing with Missing, so it can be skipped.

Constructor splitting

The other half of the algorithm is choosing which constructors to try. That is the “split” step, and it has a precise contract:

 rust
 
 
// compiler/rustc_pattern_analysis/src/constructor.rs:1011-1037

/// Describes the result of analyzing the constructors in a column of a match.
///
/// `present` is morally the set of constructors present in the column, and `missing` is the set of
/// constructors that exist in the type but are not present in the column.
///
/// More formally, if we discard wildcards from the column, this respects the following constraints:
/// 1. the union of `present`, `missing` and `missing_empty` covers all the constructors of the type
/// 2. each constructor in `present` is covered by something in the column
/// 3. no constructor in `missing` or `missing_empty` is covered by anything in the column
/// 4. each constructor in the column is equal to the union of one or more constructors in `present`
/// 5. `missing` does not contain empty constructors (see discussion about emptiness at the top of
/// the file);
/// 6. `missing_empty` contains only empty constructors
/// 7. constructors in `present`, `missing` and `missing_empty` are split for the column; in other
/// words, they are either fully included in or fully disjoint from each constructor in the
/// column. In yet other words, there are no non-trivial intersections like between `0..10` and
/// `5..15`.

Point 7 is the subtle one: the splitting has to make overlapping constructors disjoint before iterating, which is why overlapping integer ranges like 0..10 and 5..15 need special treatment — the split is computed so that every pair is either fully included or fully disjoint.

The overall shape:

 rust
 
 
// compiler/rustc_pattern_analysis/src/constructor.rs:36-49

//! Constructor splitting looks at the constructors in the first column of the matrix and constructs
//! such a sensible set of constructors. Formally, we want to find a smallest disjoint set of
//! constructors:
//! - Whose union covers the whole type, and
//! - That have no non-trivial intersection with any of the constructors in the column (i.e. they're
//! each either disjoint with or covered by any given column constructor).
//!
//! We compute this in two steps: first [`PatCx::ctors_for_ty`] determines the
//! set of all possible constructors for the type. Then [`ConstructorSet::split`] looks at the
//! column of constructors and splits the set into groups accordingly. The precise invariants of
//! [`ConstructorSet::split`] is described in [`SplitConstructorSet`].

Witness reconstruction

The algorithm does not just decide whether a match is exhaustive — it produces the witness, the value that was not matched, which becomes the “missing pattern” in the error message. That is done by “unspecializing”: the witnesses computed on specialized matrices are pushed back through the constructors:

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:1548-1579

fn apply_constructor(
 &mut self,
 pcx: &PlaceCtxt<'_, Cx>,
 missing_ctors: &[Constructor<Cx>],
 ctor: &Constructor<Cx>,
) {
 ...
 if matches!(ctor, Constructor::Missing) {
 // We got the special `Missing` constructor that stands for the constructors not present
 // in the match. For each missing constructor `c`, we add a `c(_, _, _)` witness
 // appropriately filled with wildcards.
 let mut ret = Self::empty();
 for ctor in missing_ctors {
 let pat = pcx.wild_from_ctor(ctor.clone());
 // Clone `self` and add `c(_, _, _)` to each of its witnesses.
 let mut wit_matrix = self.clone();
 wit_matrix.push_pattern(pat);
 ret.extend(wit_matrix);
 }
 *self = ret;
 } else {
 // Any other constructor we unspecialize as expected.
 for witness in std::mem::take(&mut self.0) {
 self.0.extend(witness.apply_constructor(pcx, ctor));
 }
 }
}

The parent_row back-pointer in each specialized row makes unspecialization O(rows) with no search:

 rust
 
 
// compiler/rustc_pattern_analysis/src/usefulness.rs:1317-1334

fn unspecialize(&mut self, specialized: Self) {
 for child_row in specialized.rows() {
 let parent_row_id = child_row.parent_row;
 let parent_row = &mut self.rows[parent_row_id];
 // A parent row is useful if any of its children is.
 parent_row.useful |= child_row.useful;
 ...

Where it lives

compiler/rustc_pattern_analysis/src/usefulness.rs — the algorithm, the relevancy rule, WitnessMatrix.
compiler/rustc_pattern_analysis/src/constructor.rs — ConstructorSet::split and its seven invariants.
compiler/rustc_pattern_analysis/src/pat.rs — the pattern model.

Takeaway

Maranget’s algorithm is elegant but has a known worst case. The fix here is not a better data structure — it is an extra relevance rule that says “this branch of the search cannot produce new information, skip it”. The rule is expressed as a single boolean per matrix row, propagated through the recursion, and checked once per node. The cost is one flag; the saving is exponential.

Tables That Pick Their Own Integer Width: How rustc Encodes Crate Metadata

Jay Hubbert — Wed, 21 Oct 2026 13:00:00 +0000

TL;DR: rustc’s crate metadata is a set of tables whose element width is decided at encode time: the encoder strips trailing zero bytes from each element and takes the maximum, so a table of small values is stored in one byte per entry. And when an element is a pair of integers, the bytes are interleaved so that trailing zeros in both halves line up and one stripping pass compresses both.

The table header

 rust
 
 
// compiler/rustc_metadata/src/rmeta/mod.rs:156-164

struct LazyTable<IdxEncode, IdxDecode, T> {
 position: NonZero<usize>,
 /// The encoded size of the elements of a table is selected at runtime to drop
 /// trailing zeroes. This is the number of bytes used for each table element.
 width: usize,
 /// How many elements are in the table.
 len: usize,
 _marker: PhantomData<fn(IdxEncode, IdxDecode) -> T>,
}

The width is per table, not per element. The decoder knows the stride and reads width bytes at position + width * index.

Choosing the width

 rust
 
 
// compiler/rustc_metadata/src/rmeta/table.rs:453-505

pub(crate) fn set(&mut self, i: Ie, value: T) {
 ...
 if !value.is_default() {
 // FIXME(eddyb) investigate more compact encodings for sparse tables.
 // ...
 let block = self.blocks.ensure_contains_elem(i, || [0; N]);
 value.write_to_bytes(block);
 if self.width != N {
 let width = N - trailing_zeros(block);
 self.width = self.width.max(width);
 }
 }
}

pub(crate) fn encode(&self, buf: &mut FileEncoder<'_>) -> LazyTable<Ie, Id, T> {
 let pos = buf.position();

 let width = self.width;
 for block in &self.blocks {
 buf.write_with(|dest| {
 *dest = *block;
 width
 });
 }
 ...
}

fn trailing_zeros(x: &[u8]) -> usize {
 x.iter().rev().take_while(|b| **b == 0).count()
}

The mechanism:

each element has a maximum width N (1, 8 or 16 bytes);
when an element is written, the encoder computes how many of its trailing bytes are zero and remembers the maximum over all elements;
at encode time, only that many bytes per element are written.

So a table of DefKinds — one byte each — whose values all happen to be below 16 encodes in one byte per entry, and the same table with one large value encodes in the full width.

The is_default() check is a second optimisation: default values are simply not written, leaving zeros. (The comment notes that metadata tables never need to reset a value to default, which is what makes this safe.)

Byte-interleaving the pairs

Here is the genuinely clever part. Many metadata elements are a pair of integers — a position and a length, or a crate index and a local index. If you store them as [pos_bytes | len_bytes], then the high bytes of len (which are usually non-zero) shield the trailing zeros of pos, and the element can never shrink below 8 bytes.

The fix is to interleave:

 rust
 
 
// compiler/rustc_metadata/src/rmeta/table.rs:362-390

// Interleaving the bytes of the two integers exposes trailing bytes in the first integer
// to the varint scheme that we use for tables.
#[inline]
fn decode_interleaved<const N: usize, const M: usize>(encoded: &[u8; N]) -> ([u8; M], [u8; M]) {
 assert_eq!(M * 2, N);
 let mut first = [0u8; M];
 let mut second = [0u8; M];
 for i in 0..M {
 first[i] = encoded[2 * i];
 second[i] = encoded[2 * i + 1];
 }
 (first, second)
}

// Element width is selected at runtime on a per-table basis by omitting trailing
// zero bytes in table elements. This works very naturally when table elements are
// simple numbers but sometimes we have a pair of integers. If naively encoded, the second element
// would shield the trailing zeroes in the first. Interleaving the bytes exposes trailing zeroes in
// both to the optimization.
//
// Prefer passing a and b such that `b` is usually smaller.
#[inline]
fn encode_interleaved<const N: usize, const M: usize>(a: [u8; M], b: [u8; M], dest: &mut [u8; N]) {
 assert_eq!(M * 2, N);
 for i in 0..M {
 dest[2 * i] = a[i];
 dest[2 * i + 1] = b[i];
 }
}

Interleaving pos[0], len[0], pos[1], len[1], ... puts the significant bytes of both integers at the same offsets, so the trailing zeros of both line up at the end of the element and one “strip trailing zeros” pass compresses both halves at once.

The comment “Prefer passing a and b such that b is usually smaller” is the follow-on advice: the smaller number’s high bytes become zeros, which is where the savings come from.

Used for LazyArray<T>:

 rust
 
 
// compiler/rustc_metadata/src/rmeta/table.rs:346-360

impl<T> LazyArray<T> {
 #[inline]
 fn write_to_bytes_impl(self, dest: &mut [u8; 16]) {
 let position = (self.position.get() as u64).to_le_bytes();
 let len = (self.num_elems as u64).to_le_bytes();

 encode_interleaved(position, len, dest)
 }

and for DefIds, where the crate number goes second:

 rust
 
 
// compiler/rustc_metadata/src/rmeta/table.rs:272-288

fn write_to_bytes(self, dest: &mut [u8; 8]) {
 match self {
 None => unreachable!(),
 Some(id) => {
 let (krate, index) = id.decompose();
 debug_assert!(krate < u32::MAX);
 // CrateNum is less than `CrateNum::MAX_AS_U32`.
 let krate = (krate + 1).to_le_bytes();
 let index = index.to_le_bytes();

 // CrateNum is usually much smaller than the index within the crate, so put it in
 // the second slot.
 encode_interleaved(index, krate, dest);
 }
 }
}

A DefId element is 8 bytes: a u32 index and a u32 crate number. The crate number is usually small (a handful of dependencies), so putting it second means the element usually compresses to 5 bytes or fewer.

Decoding straight out of the mmap

 rust
 
 
// compiler/rustc_metadata/src/rmeta/table.rs:507-535

pub(super) fn get<'a, 'tcx, M: MetaBlob<'a>>(&self, metadata: M, i: Id) -> T::Value<'tcx> {
 // Access past the end of the table returns a Default
 if i.index() >= self.len {
 return Default::default();
 }

 let width = self.width;
 let start = self.position.get() + (width * i.index());
 let end = start + width;
 let bytes = &metadata.blob()[start..end];

 if let Ok(fixed) = bytes.try_into() {
 FixedSizeEncoding::from_bytes(fixed)
 } else {
 let mut fixed = [0u8; N];
 fixed[..width].copy_from_slice(bytes);
 FixedSizeEncoding::from_bytes(&fixed)
 }
}

The element is read by indexing the metadata blob directly — no decompression step, no intermediate structure, no copy (in the common case where the width matches the fixed size exactly). And the blob is an mmap:

 rust
 
 
// compiler/rustc_metadata/src/locator.rs:936-953

fn get_rmeta_metadata_section<'a, 'p>(filename: &'p Path) -> Result<OwnedSlice, MetadataError<'a>> {
 // mmap the file, because only a small fraction of it is read.
 let file = std::fs::File::open(filename).map_err(|_| {
 MetadataError::LoadFailure(format!(
 "failed to open rmeta metadata: '{}'",
 filename.display()
 ))
 })?;
 let mmap = unsafe { Mmap::map(file) };
 ...

The comment is the point: only a small fraction of the file is ever read, so the OS only faults in the pages that are touched.

Lazy values and post-order encoding

The table above stores offsets, and those offsets point into a post-order-encoded tree:

 rust
 
 
// compiler/rustc_metadata/src/rmeta/mod.rs:93-118

/// A value of type T referred to by its absolute position
/// in the metadata, and which can be decoded lazily.
///
/// Metadata is effective a tree, encoded in post-order,
/// and with the root's position written next to the header.
/// That means every single `LazyValue` points to some previous
/// location in the metadata and is part of a larger node.
///
/// The first `LazyValue` in a node is encoded as the backwards
/// distance from the position where the containing node
/// starts and where the `LazyValue` points to, while the rest
/// use the forward distance from the previous `LazyValue`.
/// Distances start at 1, as 0-byte nodes are invalid.
/// Also invalid are nodes being referred in a different
/// order than they were encoded in.
#[must_use]
struct LazyValue<T> {
 position: NonZero<usize>,
 _marker: PhantomData<fn() -> T>,
}

A LazyValue is one machine word — an offset, nothing else. The value T is decoded only when somebody asks for it.

And the offsets are encoded as distances, not absolute positions:

 rust
 
 
// compiler/rustc_metadata/src/rmeta/encoder.rs:497-518

fn emit_lazy_distance(&mut self, position: NonZero<usize>) {
 let pos = position.get();
 let distance = match self.lazy_state {
 LazyState::NoNode => bug!("emit_lazy_distance: outside of a metadata node"),
 LazyState::NodeStart(start) => {
 let start = start.get();
 assert!(pos <= start);
 start - pos
 }
 LazyState::Previous(last_pos) => {
 assert!(
 last_pos <= position,
 "make sure that the calls to `lazy*` \
 are in the same order as the metadata fields",
 );
 position.get() - last_pos.get()
 }
 };
 self.lazy_state = LazyState::Previous(NonZero::new(pos).unwrap());
 self.emit_usize(distance);
}

The first LazyValue in a node is a backwards distance from the node’s start; the rest are forward distances from the previous one. Because post-order encoding puts children right next to their parent, these distances are small — typically one to three varint bytes instead of eight for an absolute offset.

The effect: loading a dependency crate’s metadata costs one mmap plus a handful of page faults for the header and the tables. Only the definitions actually referenced get decoded.

Where it lives

compiler/rustc_metadata/src/rmeta/table.rs — TableBuilder::set, encode_interleaved, LazyTable::get.
compiler/rustc_metadata/src/rmeta/mod.rs — LazyTable, LazyValue.
compiler/rustc_metadata/src/rmeta/encoder.rs — emit_lazy_distance.
compiler/rustc_metadata/src/locator.rs — the mmap.

Takeaway

Two reusable ideas. First: choose the encoding width at runtime, when the data is in front of you — the encoder knows the actual values and can pick the smallest width that fits, which a fixed-width format cannot do. Second: interleave to expose common structure — when two numbers are packed together, laying their bytes out alternately lets a single trailing-zero-stripping rule compress both. Neither trick is specific to rustc; both are good tricks for any binary format.

How a Span Fits in Eight Bytes: rustc's Four-Format Packed Encoding

Jay Hubbert — Wed, 21 Oct 2026 12:55:00 +0000

TL;DR: A Span has to say where in the source something is, and which hygiene context it belongs to. rustc packs that into eight bytes using four different layouts, selected by a variable-length prefix in the high bits. And the decoding is a macro, not an enum, because — as the comment says — “converting span to an enum and matching on it doesn’t optimize away properly.”

The four formats

 rust
 
 
// compiler/rustc_span/src/span_encoding.rs:31-50

//! There are four different span formats and each packs data into the 8 bytes
//! in a different way. A variable-length prefix in the high bits identifies
//! which format is being used. It is an invariant that we always use the first
//! listed format whose requirements are satisfied. `len` is `hi - lo`.
//!
//! - Inline-context format (requires 15-bit length, 16-bit context, and no parent):
//!
//! `[ prefix=0 | len:15 | ctxt:16 | lo:32 ]`
//!
//! - Inline-parent format (requires 14-bit length, root context, and 15-bit parent):
//!
//! `[ prefix=110 | len:14 | parent:15 | lo:32 ]`
//!
//! - Inline-pair format (requires 24-bit lo, 7-bit length, 15-bit context, 16-bit parent):
//!
//! `[ prefix=10 | len:7 | ctxt:15 | parent:16 | lo:24 ]`
//!
//! - Interned format (all cases not covered above):
//!
//! `[ prefix=111 | ctxt:29 | index:32 ]`

Read the invariants:

Inline-context is the common case: a 32-bit start offset, a 15-bit length, and a 16-bit hygiene context. No parent needed.
Inline-parent handles the case where the span’s context is the root context but it has a parent expansion: the parent is stored (15 bits) and the context is implied.
Inline-pair is for spans with a 24-bit offset — needed for very large files — trading length and context bits for offset bits.
Interned is the escape hatch: everything goes into a side table and the span holds an index.

The prefix lengths differ per format (1 bit, 3 bits, 2 bits, 3 bits), so the decoding is a small decision tree.

The type

 rust
 
 
// compiler/rustc_span/src/span_encoding.rs:86-95

#[derive(Clone, Copy, Eq, PartialEq, Hash)]
#[rustc_pass_by_value]
#[repr(packed(4))]
pub struct Span(u64);

repr(packed(4)) keeps the alignment at 4 bytes while the size stays 8, so AST nodes containing spans stay smaller. The rustc_pass_by_value attribute tells the compiler to pass spans by value rather than by reference.

The bit-field helper

 rust
 
 
// compiler/rustc_span/src/span_encoding.rs:182-201

impl InlineCtxt {
 const PREFIX_VALUE: u32 = 0b0;
 const PREFIX: SpanField = SpanField::top(1);
 const LEN: SpanField = Self::PREFIX.below(15);
 const CTXT: SpanField = Self::LEN.below(16);
 const LO: SpanField = Self::CTXT.below(32);

 #[inline]
 pub(crate) const fn try_new_span(lo: u32, len: u32, ctxt: u32) -> Option<Span> {
 if Self::LEN.too_big(len) | Self::CTXT.too_big(ctxt) == 0 {
 Some(Span(
 Self::PREFIX.set(Self::PREFIX_VALUE)
 | Self::LEN.set(len)
 | Self::CTXT.set(ctxt)
 | Self::LO.set(lo),
 ))
 } else {
 None
 }
 }
}

The bit-field layout is computed at compile time (shift, width, mask are consts), so each format is just shifts and masks at runtime. And note the |: the two “does it fit?” tests are combined into a single branchless test.

Why a macro and not an enum

 rust
 
 
// compiler/rustc_span/src/span_encoding.rs:346-372

// This code is very hot, and converting span to an enum and matching on it doesn't optimize away
// properly. So we are using a macro emulating such a match, but expand it directly to an if-else
// chain.
macro_rules! match_span_kind {
 (
 $span:expr,
 InlineCtxt($span1:ident) => $arm1:expr,
 InlineParent($span2:ident) => $arm2:expr,
 InlinePair($span3:ident) => $arm3:expr,
 Interned($span4:ident) => $arm4:expr,
 ) => {{
 let span64 = $span.0;
 if InlineCtxt::PREFIX.get(span64) == InlineCtxt::PREFIX_VALUE {
 let $span1 = InlineCtxt::from_span($span);
 $arm1
 } else if InlineParent::PREFIX.get(span64) == InlineParent::PREFIX_VALUE {
 let $span2 = InlineParent::from_span($span);
 $arm2
 } else if InlinePair::PREFIX.get(span64) == InlinePair::PREFIX_VALUE {
 let $span3 = InlinePair::from_span($span);
 $arm3
 } else {
 let $span4 = Interned::from_span($span);
 $arm4
 }
 }};
}

The source looks like an exhaustive enum match — readable, and the compiler will warn if a format is forgotten — but it expands to a straight-line if/else chain over the raw u64.

Why does that matter? Using a real enum would mean constructing the enum (a store to a stack slot) and then matching on it, and LLVM failed to optimize that away in this hot path. The macro keeps the ergonomics and gets the codegen.

The order of the tests matters too: InlineCtxt (prefix 0) is tested first, because it is by far the most common. The source even micro-optimises that case:

 rust
 
 
// compiler/rustc_span/src/span_encoding.rs:486-498

InlineCtxt(span) => {
 // This format occurs 1-2 orders of magnitude more often than others (#125017),
 // so it makes sense to micro-optimize it to avoid `span.data()` and `Span::new()`.

Hygiene as interned mark chains

The ctxt field is a SyntaxContext:

 rust
 
 
// compiler/rustc_span/src/hygiene.rs:51-79

/// A `SyntaxContext` represents a chain of pairs `(ExpnId, Transparency)` named "marks".
///
/// See <https://rustc-dev-guide.rust-lang.org/macro-expansion.html> for more explanation.
#[derive(Default, Clone, Copy, PartialEq, Eq, Hash)]
pub struct SyntaxContext(u32);

Four bytes. It is an index into a global, interned table:

 rust
 
 
// compiler/rustc_span/src/hygiene.rs:81-95

#[derive(Clone, Copy, Debug)]
struct SyntaxContextData {
 /// The last macro expansion in the chain.
 /// (Here we say the most deeply nested macro expansion is the "outermost" expansion.)
 outer_expn: ExpnId,
 /// Transparency of the last macro expansion
 outer_transparency: Transparency,
 parent: SyntaxContext,
 /// This context, but with all transparent and semi-opaque expansions filtered away.
 opaque: SyntaxContext,
 /// This context, but with all transparent expansions filtered away.
 opaque_and_semiopaque: SyntaxContext,
 /// Name of the crate to which `$crate` with this context would resolve.
 dollar_crate_name: Symbol,
}

The mark chain is a linked list of interned nodes, but the useful projections of it are precomputed: opaque is the chain with transparent and semi-opaque marks filtered out, opaque_and_semiopaque with only transparent ones. So “resolve this name under macros 2.0 rules” is an array index rather than a chain walk.

The interning itself:

 rust
 
 
// compiler/rustc_span/src/hygiene.rs:729-750

fn alloc_ctxt(
 &mut self,
 parent: SyntaxContext,
 expn_id: ExpnId,
 transparency: Transparency,
) -> SyntaxContext {
 // Look into the cache first.
 let key = (parent, expn_id, transparency);
 if let Some(ctxt) = self.syntax_context_map.get(&key) {
 return *ctxt;
 }
 ...

Because contexts are hash-consed, two expansions with the same hygiene chain share one u32, and equality is a single integer compare.

Why `Ord` is forbidden

 rust
 
 
// compiler/rustc_span/src/hygiene.rs:71-75

// To ensure correctness of incremental compilation,
// `SyntaxContext` must not implement `Ord` or `PartialOrd`.
// See https://github.com/rust-lang/rust/issues/90317.
impl !Ord for SyntaxContext {}
impl !PartialOrd for SyntaxContext {}

This is one of my favourite pieces of API design in the whole compiler. Hygiene contexts are allocated in an arbitrary, non-deterministic order as expansions are encountered. If SyntaxContext implemented Ord, then any BTreeMap/BTreeSet keyed by it would iterate in an order that depends on that arbitrary allocation order — and the incremental compiler compares query results across sessions. The same set of contexts, in a different order, would look like a different result, causing spurious recompilation and non-deterministic diagnostics.

By forbidding Ord, the type system forces callers to use HashSet / HashMap (order-independent) or to explicitly opt into an ordering (as Span does with its OrdSpan wrapper, which orders only by position).

Where it lives

compiler/rustc_span/src/span_encoding.rs — the four formats, the match_span_kind! macro, SpanField.
compiler/rustc_span/src/hygiene.rs — SyntaxContext, the interning table, the forbidden Ord.

Takeaway

Two lessons. First: when an enum’s discriminant is in the hot path, the source-level enum may not be the best representation — a packed integer with a prefix, decoded by a macro, can be strictly faster, and the macro lets you keep the readable version. Second: impl !Ord is a real technique. Sometimes the best way to guarantee a property (deterministic iteration order) is to remove a trait from a type so that the compiler enforces it everywhere.

Always Bump Downwards: The Arena Allocator rustc Uses for Everything

Jay Hubbert — Wed, 21 Oct 2026 12:50:00 +0000

TL;DR: rustc allocates almost everything — types, HIR, MIR, spans — from bump allocators. The trick is that the arena allocates downwards from the end of a chunk, and tells LLVM that the pointer is aligned with hint::assert_unchecked, which lets the compiler delete the alignment arithmetic entirely.

The arena

 rust
 
 
// compiler/rustc_arena/src/lib.rs:344-362

/// An arena that can hold objects of multiple different types that impl `Copy`
/// and/or satisfy `!mem::needs_drop`.
pub struct DroplessArena {
 /// A pointer to the start of the free space.
 start: Cell<*mut u8>,

 /// A pointer to the end of free space.
 ///
 /// The allocation proceeds downwards from the end of the chunk towards the
 /// start. (This is slightly simpler and faster than allocating upwards,
 /// see <https://fitzgeraldnick.com/2019/11/01/always-bump-downwards.html>.)
 /// When this pointer crosses the start pointer, a new chunk is allocated.
 ///
 /// This is kept aligned to DROPLESS_ALIGNMENT.
 end: Cell<*mut u8>,

 /// A vector of arena chunks.
 chunks: RefCell<Vec<ArenaChunk>>,
}

The allocation proceeds downwards. The comment links to a blog post arguing why, but the short version is that downward allocation is marginally simpler: the alignment fix-up is a subtraction and a mask on a pointer that moves down, and the “have we run out?” check is a single unsigned comparison against the chunk’s start.

The allocation loop

 rust
 
 
// compiler/rustc_arena/src/lib.rs:419-453

#[inline]
pub fn alloc_raw(&self, layout: Layout) -> *mut u8 {
 assert!(layout.size() != 0);

 // This loop executes once or twice: if allocation fails the first
 // time, the `grow` ensures it will succeed the second time.
 loop {
 let start = self.start.get().addr();
 let old_end = self.end.get();
 let end = old_end.addr();

 // Align allocated bytes so that `self.end` stays aligned to
 // DROPLESS_ALIGNMENT.
 let bytes = align_up(layout.size(), DROPLESS_ALIGNMENT);

 // Tell LLVM that `end` is aligned to DROPLESS_ALIGNMENT.
 unsafe { hint::assert_unchecked(end == align_down(end, DROPLESS_ALIGNMENT)) };

 if let Some(sub) = end.checked_sub(bytes) {
 let new_end = align_down(sub, layout.align());
 if start <= new_end {
 let new_end = old_end.with_addr(new_end);
 // `new_end` is aligned to DROPLESS_ALIGNMENT as `align_down`
 // preserves alignment as both `end` and `bytes` are already
 // aligned to DROPLESS_ALIGNMENT.
 self.end.set(new_end);
 return new_end;
 }
 }

 // No free space left. Allocate a new chunk to satisfy the request.
 // On failure the grow will panic or abort.
 self.grow(layout);
 }
}

The hint::assert_unchecked is the interesting line. It is a compile-time-only assertion that LLVM treats as an assume: given end % 8 == 0 and bytes % 8 == 0, LLVM knows end - bytes is already 8-aligned, so align_down(sub, layout.align()) reduces to a plain mask and — crucially — the whole “align the size up to 8” step collapses away.

Without the assume, LLVM must conservatively keep the full align_up / align_down arithmetic in the hot path.

The invariant really holds: grow aligns the chunk end down to DROPLESS_ALIGNMENT, and every allocation subtracts align_up(size, DROPLESS_ALIGNMENT) — a multiple of 8.

Chunk growth

 rust
 
 
// compiler/rustc_arena/src/lib.rs:96-102

// The arenas start with PAGE-sized chunks, and then each new chunk is twice as
// big as its predecessor, up until we reach HUGE_PAGE-sized chunks, whereupon
// we stop growing. This scales well, from arenas that are barely used up to
// arenas that are used for 100s of MiBs. Note also that the chosen sizes match
// the usual sizes of pages and huge pages on Linux.
const PAGE: usize = 4096;
const HUGE_PAGE: usize = 2 * 1024 * 1024;

and:

 rust
 
 
// compiler/rustc_arena/src/lib.rs:380-417

#[inline(never)]
#[cold]
fn grow(&self, layout: Layout) {
 // Add some padding so we can align `self.end` while
 // still fitting in a `layout` allocation.
 let additional = layout.size() + cmp::max(DROPLESS_ALIGNMENT, layout.align()) - 1;

 unsafe {
 let mut chunks = self.chunks.borrow_mut();
 let mut new_cap;
 if let Some(last_chunk) = chunks.last_mut() {
 // There is no need to update `last_chunk.entries` because that
 // field isn't used by `DroplessArena`.

 // If the previous chunk's len is less than HUGE_PAGE
 // bytes, then this chunk will be least double the previous
 // chunk's size.
 new_cap = last_chunk.storage.len().min(HUGE_PAGE / 2);
 new_cap *= 2;
 } else {
 new_cap = PAGE;
 }
 // Also ensure that this chunk can fit `additional`.
 new_cap = cmp::max(additional, new_cap);

 let chunk = chunks.push_mut(ArenaChunk::new(align_up(new_cap, PAGE)));
 self.start.set(chunk.start());

 // Align the end to DROPLESS_ALIGNMENT.
 let end = align_down(chunk.end().addr(), DROPLESS_ALIGNMENT);
 ...

Geometric growth: 4 KiB, 8 KiB, 16 KiB, … capped at 2 MiB, which is Linux’s HUGE_PAGE. The sizes deliberately match the OS’s page sizes so the allocator can hand back huge-page-aligned memory.

Note grow is #[cold] #[inline(never)] — the chunk allocation path never bloats the hot inline path.

Two arenas, one macro

 rust
 
 
// compiler/rustc_arena/src/lib.rs:597-627

/// Declares an `Arena` that can allocate values of a variety of `Copy`, `needs_drop` and
/// `!needs_drop` types.
///
/// The declared arena actually contains a single [`DroplessArena`], plus a separate
/// [`TypedArena`] for each of the types listed in the body of the macro invocation.
///
/// Any type that is `Copy` can be allocated in the arena without needing to be listed
/// explicitly. Those values will be stored in the [`DroplessArena`].
///
/// Types that are `!Copy` can only be allocated if they are listed in the macro invocation.
/// For types that are `!Copy + needs_drop`, values will be stored in the corresponding
/// [`TypedArena`] and will be dropped when the arena is dropped.
///
/// As an optimization, types that are `!Copy + !needs_drop` will actually be stored in the
/// `DroplessArena`, and the corresponding [`TypedArena`] will remain empty. This makes
/// better use of the dropless arena's storage blocks, while the overhead of having a few
/// unused typed-arenas is negligible.
#[rustc_macro_transparency = "semiopaque"]
pub macro declare_arena!(
 // Each of these entries becomes a `$name: TypedArena<$ty>` field in the arena.
 // This allows values of non-copy type $ty to be allocated in the arena.
 // The field names must be distinct, but have no further significance.
 $(
 $name:ident: $ty:ty,
 )*
) {

A declarative macro (macro-rules 2.0) generates the arena type at the use site, so it is fully specialised and inlinable — no vtable, no dynamic dispatch.

And the routing:

 rust
 
 
// compiler/rustc_arena/src/lib.rs:639-679

 // Any type that impls `Copy` can be arena-allocated in the `DroplessArena`.
 impl<'tcx, T: Copy> ArenaAllocitable<'tcx, rustc_arena::IsCopy> for T {
 #[inline]
 #[allow(clippy::mut_from_ref)]
 fn allocate_on(self, arena: &'tcx Arena<'tcx>) -> &'tcx mut Self {
 arena.dropless.alloc(self)
 }
 ...
 }
 $(
 impl<'tcx> ArenaAllocitable<'tcx, rustc_arena::IsNotCopy> for $ty {
 #[inline]
 fn allocate_on(self, arena: &'tcx Arena<'tcx>) -> &'tcx mut Self {
 if !::std::mem::needs_drop::<Self>() {
 arena.dropless.alloc(self)
 } else {
 arena.$name.alloc(self)
 }
 }

DroplessArena never runs destructors, so it is only valid for !needs_drop types. A droppable type goes to its own TypedArena<T>, which knows how many elements were initialised and drops them at teardown. But a !Copy + !needs_drop type — Option<NonZeroU32>, say — still goes into the dropless arena, because needs_drop is a compile-time constant that folds away. The comment says the trade explicitly: better use of the dropless storage, at the cost of a few unused typed arenas.

Why arenas at all?

Because rustc allocates enormous numbers of small, uniform-lifetime objects: every Ty, every HirId, every Span, every MIR statement. A general-purpose allocator’s per-object overhead (size class lookup, free list management, coalescing) is pure waste for objects that are all freed at once at the end of the compilation.

A bump allocator makes allocation two instructions and deallocation free.

Where it lives

compiler/rustc_arena/src/lib.rs — DroplessArena, TypedArena, declare_arena!.
compiler/rustc_middle/src/arena.rs — the compiler’s main arena.
compiler/rustc_hir/src/arena.rs — the HIR arena.

Takeaway

The assert_unchecked trick is the most transferable idea here: when you maintain an alignment invariant by construction, tell the compiler about it, and the alignment arithmetic disappears from the hot path. The downward bump is a smaller win but the same species of trick — choose the representation that makes the invariant easy to see and to state.

Provenance in Eight Bytes: How rustc Packs a Pointer's Identity

Jay Hubbert — Wed, 21 Oct 2026 12:45:00 +0000

TL;DR: A pointer in rustc’s const evaluator needs to say which allocation it points into, whether it is immutable, and whether it is a shared reference. rustc packs all three into one NonZero<u64> using the top two bits as flags — and stores the provenance of the bytes in memory sparsely, as whole-pointer relocations, so ordinary data carries almost no overhead.

The packed provenance

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/pointer.rs:82-116

/// The type of provenance in the compile-time interpreter.
/// This is a packed representation of:
/// - an `AllocId` (non-zero)
/// - an `immutable: bool`
/// - a `shared_ref: bool`
///
/// with the extra invariant that if `immutable` is `true`, then so
/// is `shared_ref`.
#[derive(Copy, Clone, Eq, Hash, Ord, PartialEq, PartialOrd)]
pub struct CtfeProvenance(NonZero<u64>);

impl From<AllocId> for CtfeProvenance {
 fn from(value: AllocId) -> Self {
 let prov = CtfeProvenance(value.0);
 assert!(
 prov.alloc_id() == value,
 "`AllocId` with the highest bits set cannot be used in CTFE"
 );
 prov
 }
}
...
const IMMUTABLE_MASK: u64 = 1 << 63; // the highest bit
const SHARED_REF_MASK: u64 = 1 << 62;
const ALLOC_ID_MASK: u64 = u64::MAX & !IMMUTABLE_MASK & !SHARED_REF_MASK;

impl CtfeProvenance {
 /// Returns the `AllocId` of this provenance.
 #[inline(always)]
 pub fn alloc_id(self) -> AllocId {
 AllocId(NonZero::new(self.0.get() & ALLOC_ID_MASK).unwrap())
 }

Three facts in eight bytes:

the AllocId, which is already a NonZero<u64>;
immutable, in bit 63;
shared_ref, in bit 62.

And the invariant “immutable implies shared_ref” means the pair encodes only four states, not eight — the flags are not independent.

Because it is a NonZero, Option<CtfeProvenance> is niche-optimized to 8 bytes, and the pointer stays small:

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/pointer.rs:224-234

#[derive(Copy, Clone, Eq, PartialEq, TyEncodable, TyDecodable, Hash)]
#[derive(StableHash)]
pub struct Pointer<Prov = CtfeProvenance> {
 pub(super) offset: Size, // kept private to avoid accidental misinterpretation (meaning depends on `Prov` type)
 pub provenance: Prov,
}

static_assert_size!(Pointer, 16);
// `Option<Prov>` pointers are also passed around quite a bit
// (but not stored in permanent machine state).
static_assert_size!(Pointer<Option<CtfeProvenance>>, 16);

Sixteen bytes for a pointer (offset + provenance), whether or not the provenance is optional. That matters because every load and store in the interpreter goes through a Pointer.

The naive design — struct Provenance { alloc: AllocId, immutable: bool, shared: bool } — would be 16 bytes with padding, and Option<Provenance> would be 24. That is a 50% increase in the size of every pointer the interpreter touches.

The sparse provenance map

The provenance of pointers stored in memory is a separate structure:

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/provenance_map.rs:1-42

//! Store the provenance for each byte in the range, with a more efficient
//! representation for the common case where PTR_SIZE consecutive bytes have the same provenance.
...
/// Stores the provenance information of pointers stored in memory.
#[derive(Clone, PartialEq, Eq, Hash, Debug)]
#[derive(StableHash)]
pub struct ProvenanceMap<Prov = CtfeProvenance> {
 /// `Provenance` in this map applies from the given offset for an entire pointer-size worth of
 /// bytes. Two entries in this map are always at least a pointer size apart.
 ptrs: SortedMap<Size, Prov>,
 /// This stores byte-sized provenance fragments.
 bytes: Option<Box<SortedMap<Size, PointerFrag<Prov>>>>,
}

Two levels:

ptrs — the common case. A whole pointer stored at an aligned offset gets one entry, not one per byte. This is exactly the “relocation” model that LLVM’s constant materializer uses, so intern can hand the same list to the backend without translation.
bytes — the rare case of a partially overwritten pointer. It is Option<Box<...>>, so for essentially every allocation it is None: one null pointer.

The alternative — provenance per byte — would need 8 bytes of metadata per 8 bytes of data, doubling the memory of every allocation, plus a hash lookup per byte.

Merging the fragments back

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/provenance_map.rs:151-182

/// Attempt to merge per-byte provenance back into ptr chunks, if the right fragments
/// sit next to each other. Return `false` if that is not possible due to partial pointers.
pub fn merge_bytes(&mut self, cx: &impl HasDataLayout) -> bool {
 let Some(bytes) = self.bytes.as_deref_mut() else {
 return true;
 };
 let ptr_size = cx.data_layout().pointer_size();
 while let Some((offset, first_frag)) = bytes.iter().next() {
 let offset = *offset;
 // Check if this fragment starts a pointer.
 let range = offset..offset + ptr_size;
 let frags = bytes.range(range.clone());
 if frags.len() != ptr_size.bytes_usize() {
 // We can't merge this one, no point in trying to merge the rest.
 return false;
 }
 for (idx, (_offset, frag)) in frags.iter().enumerate() {
 if !(frag.prov == first_frag.prov
 && frag.bytes == first_frag.bytes
 && frag.idx == idx as u8)
 {
 return false;
 }
 }
 // Looks like a pointer! Move it over to the ptr provenance map.
 self.ptrs.insert(offset, first_frag.prov);
 bytes.remove_range(range);
 }
 // We managed to convert everything into whole pointers.
 self.bytes = None;
 true
}

If the byte fragments happen to describe a real pointer (same provenance, contiguous, in order), they are folded back into a single ptrs entry and the byte map is dropped entirely. This is what allows a backend to receive a plain relocation list.

Hashing large allocations

One more place where “big data, small hash” shows up:

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation.rs:228-274

/// This is the maximum size we will hash at a time, when interning an `Allocation` and its
/// `InitMask`. Note, we hash that amount of bytes twice: at the start, and at the end of a buffer.
/// Used when these two structures are large: we only partially hash the larger fields in that
/// situation. See the comment at the top of their respective `Hash` impl for more details.
const MAX_BYTES_TO_HASH: usize = 64;

/// This is the maximum size (in bytes) for which a buffer will be fully hashed, when interning.
/// Otherwise, it will be partially hashed in 2 slices, requiring at least 2 `MAX_BYTES_TO_HASH`
/// bytes.
const MAX_HASHED_BUFFER_LEN: usize = 2 * MAX_BYTES_TO_HASH;
...
impl hash::Hash for Allocation {
 fn hash<H: hash::Hasher>(&self, state: &mut H) {
 ...
 let byte_count = bytes.len();
 if byte_count > MAX_HASHED_BUFFER_LEN {
 // Hash the buffer's length.
 byte_count.hash(state);

 // And its head and tail.
 bytes[..MAX_BYTES_TO_HASH].hash(state);
 bytes[byte_count - MAX_BYTES_TO_HASH..].hash(state);
 } else {
 bytes.hash(state);
 }

The comment above it explains the reasoning:

Const allocations are only hashed for interning. However, they can be large, making the hashing expensive especially since it uses FxHash: it’s better suited to short keys, not potentially big buffers like the actual bytes of allocation.

So for buffers over 128 bytes: hash the length, the first 64 bytes and the last 64 bytes. Collisions are caught by the interner’s equality check anyway, and the length+head+tail triple is highly discriminating in practice. O(size) hashing of every const allocation becomes O(1).

The same trick is applied to the init mask’s blocks.

Where it lives

compiler/rustc_middle/src/mir/interpret/pointer.rs — CtfeProvenance, the masks, Pointer.
compiler/rustc_middle/src/mir/interpret/allocation/provenance_map.rs — the sparse map and merge_bytes.
compiler/rustc_middle/src/mir/interpret/allocation.rs — partial hashing.

Takeaway

Two related lessons here. First: niche packing — AllocId is already a NonZero<u64>, and NonZero has a niche, so the two boolean flags cost nothing. Second: representation follows the common case — provenance is stored per pointer, not per byte, and the per-byte fallback is behind an Option<Box<...>> that is None almost always. Both tricks are about the same thing: the rare case should not cost the common case anything.

The Init Mask Is a Single Boolean Until It Isn't: How rustc Tracks Uninitialised Memory

Jay Hubbert — Wed, 21 Oct 2026 12:40:00 +0000

TL;DR: Rust has to know which bytes of a const are initialised, because reading an uninitialised byte is UB. The naive representation is a bitmap — one bit per byte, so a 1 MiB allocation costs 128 KiB. rustc instead starts with a single boolean and only materialises the bitmap when a partial write happens. A full overwrite materialises it back to a boolean.

The representation

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:14-42

type Block = u64;

/// A bitmask where each bit refers to the byte with the same index. If the bit is `true`, the byte
/// is initialized. If it is `false` the byte is uninitialized.
/// The actual bits are only materialized when needed, and we try to keep this data lazy as long as
/// possible. Currently, if all the blocks have the same value, then the mask represents either a
/// fully initialized or fully uninitialized const allocation, so we can only store that single
/// value.
#[derive(Clone, Debug, Eq, PartialEq, Encodable_NoContext, Decodable_NoContext, Hash, StableHash)]
pub struct InitMask {
 blocks: InitMaskBlocks,
 len: Size,
}

#[derive(Clone, Debug, Eq, PartialEq, Encodable_NoContext, Decodable_NoContext, Hash, StableHash)]
enum InitMaskBlocks {
 Lazy {
 /// Whether the lazy init mask is fully initialized or uninitialized.
 state: bool,
 },
 Materialized(InitMaskMaterialized),
}

The doc comment says it plainly: “The actual bits are only materialized when needed.”

A freshly allocated buffer is InitMask::new(size, false) — one byte of state, zero bits of bitmap.

Reading it

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:48-65

#[inline]
pub fn is_range_initialized(&self, range: AllocRange) -> Result<(), AllocRange> {
 let end = range.end();
 if end > self.len {
 return Err(AllocRange::from(self.len..end));
 }

 match self.blocks {
 InitMaskBlocks::Lazy { state } => {
 // Lazily allocated blocks represent the full mask, and cover the requested range by
 // definition.
 if state { Ok(()) } else { Err(range) }
 }
 InitMaskBlocks::Materialized(ref blocks) => {
 blocks.is_range_initialized(range.start, end)
 }
 }
}

The lazy case is O(1) and touches no memory beyond the single bool.

Writing it, and the two-way transition

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:69-100

/// Sets a specified range to a value. If the range is out-of-bounds, the mask will grow to
/// accommodate it entirely.
pub fn set_range(&mut self, range: AllocRange, new_state: bool) {
 let start = range.start;
 let end = range.end();

 let is_full_overwrite = start == Size::ZERO && end >= self.len;

 // Optimize the cases of a full init/uninit state, while handling growth if needed.
 match self.blocks {
 InitMaskBlocks::Lazy { ref mut state } if is_full_overwrite => {
 // This is fully overwriting the mask, and we'll still have a single initialization
 // state: the blocks can stay lazy.
 *state = new_state;
 self.len = end;
 }
 InitMaskBlocks::Materialized(_) if is_full_overwrite => {
 // This is also fully overwriting materialized blocks with a single initialization
 // state: we'll have no need for these blocks anymore and can make them lazy.
 self.blocks = InitMaskBlocks::Lazy { state: new_state };
 self.len = end;
 }
 InitMaskBlocks::Lazy { state } if state == new_state => {
 // Here we're partially overwriting the mask but the initialization state doesn't
 // change: the blocks can stay lazy.
 if end > self.len {
 self.len = end;
 }
 }
 _ => {
 // Otherwise, we have a partial overwrite that can result in a mix of initialization
 // states, so we'll need materialized blocks.
 let len = self.len;
 let blocks = self.materialize_blocks();

Four cases, and note the second one: a full overwrite of an already materialised mask de-materialises it. So the representation is not a one-way ratchet; a memset-style write over a previously partially-written region returns to the one-byte representation.

The third case is subtle and good: a partial write whose value equals the current uniform state also keeps the mask lazy. Writing zeros into a fully-uninitialised buffer does not make it “partially initialised” — it stays fully uninitialised.

The bit-level search

When the mask is materialised, finding the first bit with a given value uses word-at-a-time tricks:

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:365-548

/// Search one block, returning the index of the first bit equal to `is_init`.
fn search_block(
 bits: Block,
 block: usize,
 start_bit: usize,
 is_init: bool,
) -> Option<Size> {
 // Invert bits so we're always looking for the first set bit.
 // ! 0b00111011
 // bits = 0b11000100
 let bits = if is_init { bits } else { !bits };
 // Mask off unused start bits.
 // 0b11000100
 // & 0b11111000
 // bits = 0b11000000
 let bits = bits & (!0 << start_bit);
 // Find set bit, if any.
 // bit = trailing_zeros(0b11000000)
 // bit = 6
 if bits == 0 {
 None
 } else {
 let bit = bits.trailing_zeros();
 Some(InitMaskMaterialized::size_from_bit_index(block, bit))
 }
}

trailing_zeros on a u64 is a single TZCNT instruction, and whole blocks that are uniformly 0 or 1 are skipped entirely.

And the loop is written for codegen:

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:493-503

if start_block < end_block_inclusive {
 // This loop is written in a specific way for performance.
 // Notably: `..end_block_inclusive + 1` is used for an inclusive range instead of `..=end_block_inclusive`,
 // and `.zip(start_block + 1..)` is used to track the index instead of `.enumerate().skip().take()`,
 // because both alternatives result in significantly worse codegen.

The proof that it is correct

The fast implementation is checked against a naive reference in debug builds:

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:530-547

fn find_bit_slow(
 init_mask: &InitMaskMaterialized,
 start: Size,
 end: Size,
 is_init: bool,
) -> Option<Size> {
 (start..end).find(|&i| init_mask.get(i) == is_init)
}

let result = find_bit_fast(self, start, end, is_init);

debug_assert_eq!(
 result,
 find_bit_slow(self, start, end, is_init),
 "optimized implementation of find_bit is wrong for start={start:?} end={end:?} is_init={is_init} init_mask={self:#?}"
);

Keeping the naive version in the tree as an executable specification is a lovely practice: it costs nothing in release builds and catches any divergence immediately.

The run-length encoding for repeated copies

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:646-724

/// Run-length encoding of the uninit mask.
/// Used to copy parts of a mask multiple times to another allocation.
pub struct InitCopy {
 /// Whether the first range is initialized.
 initial: bool,
 /// The lengths of ranges that are run-length encoded.
 /// The initialization state of the ranges alternate starting with `initial`.
 ranges: smallvec::SmallVec<[u64; 1]>,
}

The comment explains the motivation:

Since we are copying size bytes from src to dest + i * size (for i in 0..repeat), a naive initialization mask copying algorithm would repeatedly have to read the initialization mask from the source and write it to the destination. Even if we optimized the memory accesses, we’d be doing all of this repeat times. Therefore we precompute a compressed version of the initialization mask of the source value and then write it back repeat times without computing any more information from the source.

So [x; 1000] does O(1) work per repeat instead of O(repeat × size). And the uniform case short-circuits entirely:

 rust
 
 
// compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs:699

pub fn apply_copy(&mut self, defined: InitCopy, range: AllocRange, repeat: u64) {
 // An optimization where we can just overwrite an entire range of initialization bits if
 // they are going to be uniformly `1` or `0`. ...
 if defined.ranges.len() <= 1 {
 let start = range.start;
 let end = range.start + range.size * repeat; // `Size` operations
 self.set_range(AllocRange::from(start..end), defined.initial);
 return;
 }

Where it lives

compiler/rustc_middle/src/mir/interpret/allocation/init_mask.rs — the whole file: InitMask, InitMaskBlocks, find_bit, InitCopy.
compiler/rustc_middle/src/mir/interpret/allocation/provenance_map.rs — the sibling trick for provenance (a lazily allocated byte map).

Takeaway

This is the “one bit can carry a lot of information” pattern: most allocations are either fully initialised or fully uninitialised, and the representation should reflect the common case, not the worst case. The transition is bidirectional, so the mask degrades and recovers gracefully instead of monotonically growing.

One Interpreter, Two Machines: How rustc's Const Eval Powers Miri Too

Jay Hubbert — Wed, 21 Oct 2026 12:35:00 +0000

TL;DR: rustc’s constant evaluator is not “the const evaluator” with hooks for Miri bolted on. It is a generic interpreter written against a Machine trait with eight associated types and 57 methods. CTFE and Miri are two instantiations. That is why Miri gets UBSan-grade checking for free, and why the core interpreter has no idea which one it is serving.

The trait

 rust
 
 
// compiler/rustc_const_eval/src/interpret/machine.rs:113-163

/// Methods of this trait signifies a point where CTFE evaluation would fail
/// and some use case dependent behaviour can instead be applied.
pub trait Machine<'tcx>: Sized {
 /// Additional memory kinds a machine wishes to distinguish from the builtin ones
 type MemoryKind: Debug + std::fmt::Display + MayLeak + Eq + 'static;

 /// Pointers are "tagged" with provenance information; typically the `AllocId` they belong to.
 type Provenance: Provenance + Eq + Hash + 'static;

 /// When getting the AllocId of a pointer, some extra data is also obtained from the provenance
 /// that is passed to memory access hooks so they can do things with it.
 type ProvenanceExtra: Copy + 'static;

 /// Machines can define extra (non-instance) things that represent values of function pointers.
 /// For example, Miri uses this to return a function pointer from `dlsym`
 /// that can later be called to execute the right thing.
 type ExtraFnVal: Debug + Copy;

 /// Extra data stored in every call frame.
 type FrameExtra;

 /// Extra data stored in every allocation.
 type AllocExtra: Debug + Clone + 'tcx;

 /// Type for the bytes of the allocation.
 type Bytes: AllocBytes + 'static;

 /// Memory's allocation map
 type MemoryMap: AllocMap<
 AllocId,
 (
 MemoryKind<Self::MemoryKind>,
 Allocation<Self::Provenance, Self::AllocExtra, Self::Bytes>,
 ),
 > + Default
 + Clone;

Eight associated types covering: how memory kinds are distinguished, how provenance is represented, what extra data a function pointer can carry, what goes in a call frame, what goes in an allocation, how bytes are stored, and how the allocation map works.

The design decision that matters: the trait is pure, with associated types rather than &mut dyn objects. Every hook is a static call after monomorphization, so the interpreter’s inner loop — which runs millions of steps during a const evaluation — has no virtual dispatch.

The shared compile-time instantiation

 rust
 
 
// compiler/rustc_const_eval/src/interpret/machine.rs:692-712

/// A lot of the flexibility above is just needed for `Miri`, but all "compile-time" machines
/// (CTFE and ConstProp) use the same instance. Here, we share that code.
pub macro compile_time_machine(<$tcx: lifetime>) {
 type Provenance = CtfeProvenance;
 type ProvenanceExtra = bool; // the "immutable" flag

 type ExtraFnVal = !;

 type MemoryKind = $crate::const_eval::MemoryKind;
 type MemoryMap =
 rustc_data_structures::fx::FxIndexMap<AllocId, (MemoryKind<Self::MemoryKind>, Allocation)>;
 const GLOBAL_KIND: Option<Self::MemoryKind> = None; // no copying of globals from `tcx` to machine memory

 type AllocExtra = ();
 type FrameExtra = ();
 type Bytes = Box<[u8]>;

Three details worth stopping on:

type ExtraFnVal = ! — the empty type. Miri can return a function pointer from dlsym; CTFE cannot. By making the type uninhabited, the CTFE instantiation of call_extra_fn becomes match fn_val {} — the capability is statically impossible, at zero runtime cost.
GLOBAL_KIND = None — globals are never copied out of tcx into the interpreter’s memory. A static that is only read costs nothing; a naive design would clone the whole allocation on every access.
type Bytes = Box<[u8]> — an exactly-sized, contiguous byte store. Miri uses a Vec<u8> plus separate provenance tracking; CTFE does not need per-byte provenance for ordinary data.

The lazy operand model

 rust
 
 
// compiler/rustc_const_eval/src/interpret/operand.rs:25-41

/// An `Immediate` represents a single immediate self-contained Rust value.
///
/// For optimization of a few very common cases, there is also a representation for a pair of
/// primitive values (`ScalarPair`). It allows Miri to avoid making allocations for checked binary
/// operations and wide pointers. This idea was taken from rustc's codegen.
/// In particular, thanks to `ScalarPair`, arithmetic operations and casts can be entirely
/// defined on `Immediate`, and do not have to work with a `Place`.
#[derive(Copy, Clone, Debug)]
pub enum Immediate<Prov: Provenance = CtfeProvenance> {
 /// A single scalar value (must have *initialized* `Scalar` ABI).
 Scalar(Scalar<Prov>),
 /// A pair of two scalar value (must have `ScalarPair` ABI where both fields are
 /// `Scalar::Initialized`).
 ScalarPair(Scalar<Prov>, Scalar<Prov>),
 /// A value of fully uninitialized memory. Can have arbitrary size and layout, but must be sized.
 Uninit,
}

and:

 rust
 
 
// compiler/rustc_const_eval/src/interpret/operand.rs:476-489

/// An `Operand` is the result of computing a `mir::Operand`. It can be immediate,
/// or still in memory. The latter is an optimization, to delay reading that chunk of
/// memory and to avoid having to store arbitrary-sized data here.
#[derive(Copy, Clone, Debug)]
pub(super) enum Operand<Prov: Provenance = CtfeProvenance> {
 Immediate(Immediate<Prov>),
 Indirect(MemPlace<Prov>),
}

Operand::Indirect(MemPlace) means the value is still in memory. So an operand that is never read — the destination of a Drop, an argument that is overwritten before use — costs zero memory traffic.

And ScalarPair means a wide pointer (&[T], &dyn Trait) or a two-field tuple lives in registers, with no allocation at all.

Projecting out of a ScalarPair is register shuffling

 rust
 
 
// compiler/rustc_const_eval/src/interpret/operand.rs:409-431

// This makes several assumptions about what layouts we will encounter; we match what
// codegen does as good as we can (see `extract_field` in `rustc_codegen_ssa/src/mir/operand.rs`).
let inner_val: Immediate<_> = match (**self, self.layout.backend_repr) {
 // If the entire value is uninit, then so is the field (can happen in ConstProp).
 (Immediate::Uninit, _) => Immediate::Uninit,
 // If the field is uninhabited, we can forget the data (can happen in ConstProp).
 // `enum S { A(!), B, C }` is an example of an enum with Scalar layout that
 // has an uninhabited variant, which means this case is possible.
 _ if layout.is_uninhabited() => Immediate::Uninit,
 // the field contains no information, can be left uninit
 // (Scalar/ScalarPair can contain even aligned ZST, not just 1-ZST)
 _ if layout.is_zst() => Immediate::Uninit,
 ...
 // extract fields from types with `ScalarPair` ABI
 (
 Immediate::ScalarPair(a_val, b_val),
 BackendRepr::ScalarPair { a: _, b: _, b_offset },
 ) => Immediate::from(if offset.bytes() == 0 {
 a_val
 } else {
 assert_eq!(offset, b_offset);
 b_val
 }),

Field 0 or field 1 out of a ScalarPair is a branch on the offset, not a memory load. ZST and uninhabited fields collapse to Uninit without ever touching bytes.

A micro-optimisation kept on purpose

 rust
 
 
// compiler/rustc_const_eval/src/interpret/operand.rs:803-807

let mut op = self.local_to_op(mir_place.local, layout)?;
// Using `try_fold` turned out to be bad for performance, hence the loop.
for elem in mir_place.projection.iter() {
 op = self.project(&op, elem)?
}

The comment preserves a decision that a future refactoring would undoubtedly “clean up”: the iterator-combinator version measured slower than the plain loop.

Where it lives

compiler/rustc_const_eval/src/interpret/machine.rs — the Machine trait and compile_time_machine!.
compiler/rustc_const_eval/src/const_eval/machine.rs — the CTFE instantiation.
compiler/rustc_const_eval/src/interpret/operand.rs — Immediate, Operand, ScalarPair.
compiler/rustc_miri/src/const_eval/machine.rs — the Miri instantiation.

Takeaway

Genericity is usually traded against concreteness: the more hooks you put behind a trait, the slower the hot path. rustc avoided that trade by making the trait pure (associated types, not trait objects), so the interpreter’s inner loop stays monomorphised and inlinable. The result is that Miri’s provenance tracking and CTFE’s cheapness coexist in one implementation — and Miri gets its strict aliasing model for free, because the machinery was always there.

Sixteen Bytes at a Time: How Rust Validates UTF-8

Jay Hubbert — Wed, 21 Oct 2026 12:30:00 +0000

TL;DR: Validating UTF-8 is one of the hottest operations in text processing, and the fast path is not a UTF-8 decoder at all — it is an ASCII scanner. Rust reads two usize words at a time, checks them for high bits with a single AND, and only drops into the byte-at-a-time validator when it finds a non-ASCII byte.

The fast path

 rust
 
 
// library/core/src/str/validations.rs:114-120

const NONASCII_MASK: usize = usize::repeat_u8(0x80);

/// Returns `true` if any byte in the word `x` is nonascii (>= 128).
#[inline]
const fn contains_nonascii(x: usize) -> bool {
 (x & NONASCII_MASK) != 0
}

usize::repeat_u8(0x80) is 0x8080...80 — one high bit per byte. So x & NONASCII_MASK != 0 answers “does this word contain any non-ASCII byte?” with a single AND and a compare.

The scanning loop:

 rust
 
 
// library/core/src/str/validations.rs:219-247

} else {
 // Ascii case, try to skip forward quickly.
 // When the pointer is aligned, read 2 words of data per iteration
 // until we find a word containing a non-ascii byte.
 if align != usize::MAX && align.wrapping_sub(index).is_multiple_of(USIZE_BYTES) {
 let ptr = v.as_ptr();
 while index < blocks_end {
 unsafe {
 let block = ptr.add(index) as *const usize;
 // break if there is a nonascii byte
 let zu = contains_nonascii(*block);
 let zv = contains_nonascii(*block.add(1));
 if zu || zv {
 break;
 }
 }
 index += ascii_block_size;
 }
 // step from the point where the wordwise loop stopped
 while index < len && v[index] < 128 {
 index += 1;
 }
 } else {
 index += 1;
 }
}

Two usize reads (16 bytes on 64-bit) per iteration, two ANDs, two compares. The alignment check (align.wrapping_sub(index).is_multiple_of(USIZE_BYTES)) ensures the word reads are properly aligned — unaligned reads are legal but slow on many CPUs.

For a pure-ASCII string — the overwhelming majority of text in most programs — the whole validation is len / 16 iterations of five instructions.

The slow path

When a non-ASCII byte is found, the validator switches to a per-character state machine that tracks how many bytes the current UTF-8 sequence still needs. That part is necessarily byte-at-a-time, because UTF-8’s validity depends on the continuation bytes that follow.

The ASCII skip loop is the only part that can be vectorised, and it is where all the time goes for typical text.

`const_eval_select`

One more detail: the same source has to work at compile time (for const string validation and for include_str! checks), where align_offset is not a const function. The code handles that with a macro that selects between a const-evaluable path and the runtime path:

rust

// library/core/src/str/validations.rs

The align != usize::MAX test is the signal: align_offset returns usize::MAX when the offset cannot be computed at compile time.

Why it matters

UTF-8 validation happens on every str you create from bytes, on every file you read, on every String you build from a slice. It is one of the most-executed pieces of code in a text-heavy program. Making it 16 bytes per iteration instead of 1 is a 16× difference on the dominant path, and it costs about ten lines of code.

The same “ASCII fast path” trick appears elsewhere in the standard library and in str::find/str::split implementations: check for ASCII in bulk, and only fall back to the Unicode-aware machinery when you actually hit a non-ASCII character.

Where it lives

library/core/src/str/validations.rs — contains_nonascii, the ASCII skip loop, the UTF-8 state machine.
library/core/src/str/mod.rs — from_utf8, from_utf8_unchecked.

Takeaway

The fastest special case is the one that handles most of the data. UTF-8 validation looks like it needs a decoder, but the decoder is only needed for the minority of bytes that are not ASCII — and detecting those in bulk is a two-instruction test on eight bytes at a time. This is the same shape as the SwissTable’s group scan or the ASCII fast paths in memchr: find the interesting bytes in bulk, then handle them one at a time.

One XOR and One CLZ: How Rust's Stable Sort Decides Which Runs to Merge

Jay Hubbert — Wed, 21 Oct 2026 12:25:00 +0000

TL;DR: Rust’s stable sort (driftsort) merges runs using the powersort heuristic: build a merge tree over the runs, where each merge happens at the depth that balances the tree. The clever part is computing that depth — instead of comparing rational numbers, it rescales the number line to [0, 2^62) and takes the leading zeros of an XOR.

The theory

 rust
 
 
// library/core/src/slice/sort/stable/drift.rs:122-171

// Nearly-Optimal Mergesorts: Fast, Practical Sorting Methods That Optimally
// Adapt to Existing Runs by J. Ian Munro and Sebastian Wild.
//
// This method forms a binary merge tree, where each internal node corresponds
// to a splitting point between the adjacent runs that have to be merged. ...
// As an optimization we rescale the number line from [0, 1) to [0, 2^62). Then
// finding the simplest dyadic fraction between midpoints corresponds to finding
// the most significant bit difference of the midpoints. We save scale_factor =
// ceil(2^62 / n) to perform this rescaling using a multiplication, avoiding
// having to repeatedly do integer divides. ...
// Thus for the splitting point between two adjacent slices [a, b) and [b, c)
// the desired depth of the corresponding merge node is CLZ((a+b)*f ^ (b+c)*f),

Read the last line again, because it is the whole trick:

the desired depth of the corresponding merge node is CLZ((a+b)*f ^ (b+c)*f)

where f is a scale factor.

The implementation

 rust
 
 
// library/core/src/slice/sort/stable/drift.rs:122-171

#[inline(always)]
fn merge_tree_scale_factor(n: usize) -> u64 {
 if usize::BITS > u64::BITS {
 panic!("Platform not supported");
 }
 (1u64 << 62).div_ceil(n as u64)
}

#[inline(always)]
fn merge_tree_depth(left: usize, mid: usize, right: usize, scale_factor: u64) -> u8 {
 let x = left as u64 + mid as u64;
 let y = mid as u64 + right as u64;
 ((scale_factor * x) ^ (scale_factor * y)).leading_zeros() as u8
}

The classic powersort formulation compares the rational numbers (a+b)/2 and (b+c)/2 to decide which merge tree is most balanced. Doing that with rational arithmetic means multiplications, comparisons and careful overflow handling.

The trick: rescale [0, 1) to [0, 2^62). After rescaling, “which dyadic fraction is simplest between these two numbers?” becomes “what is the most significant bit where they differ?” — which is exactly leading_zeros(x ^ y). The scale factor f = ceil(2^62 / n) is computed once, and the rescaling is a multiplication.

So each merge-tree depth query is: two additions, two multiplications, one XOR, one lzcnt. No division, no rational comparison, no branch.

The sqrt approximation

The same file avoids the FPU for another purpose:

rust

// library/core/src/slice/sort/stable/drift.rs:173-187

sqrt_approx computes √n with an ilog2 and one Newton step rather than calling f64::sqrt. No floating point, no libm call, and — importantly — no floating point in the middle of a sorting algorithm, where the reproducibility of the output matters.

The scratch buffer sizing

 rust
 
 
// library/core/src/slice/sort/stable/mod.rs:94-135

fn driftsort_main<T, F: FnMut(&T, &T) -> bool, BufT: BufGuard<T>>(v: &mut [T], is_less: &mut F) {
 // By allocating n elements of memory we can ensure the entire input can
 // be sorted using stable quicksort, which allows better performance on
 // random and low-cardinality distributions. However, we still want to
 // reduce our memory usage to n - n / 2 for large inputs. We do this by scaling
 // our allocation as max(n - n / 2, min(n, 8MB)), ensuring we scale like n for
 // small inputs and n - n / 2 for large inputs, without a sudden drop off. ...
 const MAX_FULL_ALLOC_BYTES: usize = 8_000_000; // 8MB
 let max_full_alloc = MAX_FULL_ALLOC_BYTES / size_of::<T>();
 let len = v.len();
 let alloc_len = cmp::max(
 cmp::max(len - len / 2, cmp::min(len, max_full_alloc)),
 SMALL_SORT_GENERAL_SCRATCH_LEN,
 );

 // For small inputs 4KiB of stack storage suffices, which allows us to avoid
 // calling the (de-)allocator. Benchmarks showed this was quite beneficial.
 let mut stack_buf = AlignedStorage::<T, 4096>::new();
 let stack_scratch = stack_buf.as_uninit_slice_mut();
 let mut heap_buf;
 let scratch = if stack_scratch.len() >= alloc_len {
 stack_scratch
 } else {
 heap_buf = BufT::with_capacity(alloc_len);
 heap_buf.as_uninit_slice_mut()
 };

Three decisions:

The scratch is n - n/2 for large inputs, not n — half the memory of a textbook mergesort.
Sorts of ≤ 4 KiB of elements never touch the allocator — the scratch is on the stack.
The BufGuard trait is what lets core (which has no allocator) contain the algorithm while std provides the buffer. That is a nice layering trick: the sorting algorithm lives in core, and the allocation policy is injected.

Where it lives

library/core/src/slice/sort/stable/drift.rs — merge_tree_depth, merge_tree_scale_factor, sqrt_approx.
library/core/src/slice/sort/stable/mod.rs — driftsort_main, the scratch sizing, BufGuard.
library/core/src/slice/sort/stable/merge.rs — the merge itself.

Takeaway

The “simplest dyadic fraction between two rationals” is a natural way to balance a merge tree, but computing it with rational arithmetic is expensive. Rescaling the number line so that the answer becomes a bit difference is a classic “change the representation, change the cost” move: the same mathematical quantity, computed with an XOR and a lzcnt instead of multiplications and comparisons. It is the kind of trick that you can only find if you actually look at what the algorithm needs rather than how the paper writes it down.

Thirteen Elements, Forty-Five Comparisons: The Sorting Network in Rust's sort

Jay Hubbert — Wed, 21 Oct 2026 12:20:00 +0000

TL;DR: Rust’s sort and sort_unstable both fall back to a fixed sorting network for small inputs — up to 13 elements, 45 compare-exchanges — instead of insertion sort. A network has no branches at all: the comparison sequence is fixed in advance, so there is nothing to mispredict.

The rule

 rust
 
 
// library/core/src/slice/sort/stable/mod.rs:28-86

#[inline(always)]
pub fn sort<T, F: FnMut(&T, &T) -> bool, BufT: BufGuard<T>>(v: &mut [T], is_less: &mut F) {
 // Arrays of zero-sized types are always all-equal, and thus sorted.
 if T::IS_ZST {
 return;
 }

 // Instrumenting the standard library showed that 90+% of the calls to sort
 // by rustc are either of size 0 or 1.
 let len = v.len();
 if intrinsics::likely(len < 2) {
 return;
 }
 ...
 // More advanced sorting methods than insertion sort are faster if called
 // in a hot loop for small inputs, but for general-purpose code the small
 // binary size of insertion sort is more important. The instruction cache in
 // modern processors is very valuable, and for a single sort call in general
 // purpose code any gains from an advanced method are cancelled by i-cache
 // misses during the sort, and thrashing the i-cache for surrounding code.
 const MAX_LEN_ALWAYS_INSERTION_SORT: usize = 20;
 if intrinsics::likely(len <= MAX_LEN_ALWAYS_INSERTION_SORT) {
 insertion_sort_shift_left(v, 1, is_less);
 return;
 }

Note the reasoning about insertion sort for the stable sort: the reason is not that insertion sort is faster, but that it is small, and the instruction cache is precious. For the unstable sort (ipnsort) the trade goes the other way — a network is branchless and therefore fast.

The network

 rust
 
 
// library/core/src/slice/sort/shared/smallsort.rs:463-530

/// Sorts the first 13 elements of `v` with a fast fixed function.
///
/// Should `is_less` generate substantial amounts of code the compiler can choose to not inline
/// `swap_if_less`. If the code of a sort impl changes so as to call this function in multiple
/// places, `#[inline(never)]` is recommended to keep binary-size in check. ...
fn sort13_optimal<T, F>(v: &mut [T], is_less: &mut F)
where
 F: FnMut(&T, &T) -> bool,
{
 if v.len() < 13 {
 intrinsics::abort_immediate();
 }
 let v_base = v.as_mut_ptr();
 // Optimal sorting network see:
 // https://bertdobbelaere.github.io/sorting_networks.html.
 unsafe {
 swap_if_less(v_base, 0, 12, is_less);
 swap_if_less(v_base, 1, 10, is_less);
 ...
 swap_if_less(v_base, 5, 6, is_less);
 }
}

Dobbelaere’s optimal 13-element network is 45 compare-exchange operations — provably minimal for 13 inputs. The network is data independent: the same 45 comparisons happen regardless of the input, so the branch predictor never has anything to learn.

swap_if_less compiles to a compare and a conditional swap:

rust

// library/core/src/slice/sort/shared/smallsort.rs

typically becoming a cmov-style branchless select for small Copy types.

The specialisation gate

The network is only used for types that are Copy and small enough to be handled with stack buffers:

rust

// library/core/src/slice/sort/shared/smallsort.rs:142-171

with a stack budget:

 rust
 
const MAX_STACK_ARRAY_SIZE: usize = 4096;

The specialisation matters because the network’s swap_if_less needs the elements to be movable without a destructor; for types with Drop the stable path uses a different, copy-based small sort.

Ping-pong scratch

For the stable sort, small sorts use a scratch buffer in a ping-pong fashion, and the merge-back is panic-safe:

 rust
 
 
// library/core/src/slice/sort/shared/smallsort.rs:220-309

fn small_sort_general_with_scratch<T: FreezeMarker, F: FnMut(&T, &T) -> bool>(
 v: &mut [T],
 scratch: &mut [MaybeUninit<T>],
 is_less: &mut F,
) {
 ...
 // SAFETY: see comment in `CopyOnDrop::drop`.
 let drop_guard = CopyOnDrop { src: scratch_base, dst: v_base, len };

CopyOnDrop is the same pattern as the DropGuard in Vec::drain: if the comparison function panics, the guard still copies the scratch data back into the original array so the Vec is left in a valid state.

Where it lives

library/core/src/slice/sort/shared/smallsort.rs — sort13_optimal, CopyOnDrop, the specialisation gates.
library/core/src/slice/sort/stable/mod.rs — the stable sort’s small-input policy.
library/core/src/slice/sort/unstable/mod.rs — ipnsort, the unstable variant.

Takeaway

Data-dependent branches are the enemy of small sorts. A fixed comparison network does strictly more comparisons than insertion sort in the worst case, but it has zero unpredictable branches, and for small inputs the branch mispredictions cost more than the extra work. This is a classic case where “more operations, fewer branches” wins — and where the optimal network (45 rather than 60-something comparisons) is a nice piece of applied combinatorics.

SipHash's Field Order Is a SIMD Hint: How Rust's Default Hasher Vectorises Itself

Jay Hubbert — Wed, 21 Oct 2026 12:15:00 +0000

TL;DR: SipHash’s four state words are v0, v1, v2, v3. Rust declares them as v0, v2, v1, v3 — and the comment says why: “v0, v2 and v1, v3 show up in pairs in the algorithm, and simd implementations of SipHash will use vectors of v02 and v13. By placing them in this order in the struct, the compiler can pick up on just a few simd optimizations by itself.” A field order is a vectorisation hint.

The state

 rust
 
 
// library/core/src/hash/sip.rs:62-94

#[derive(Debug, Clone, Copy)]
#[repr(C)]
struct State {
 // v0, v2 and v1, v3 show up in pairs in the algorithm,
 // and simd implementations of SipHash will use vectors
 // of v02 and v13. By placing them in this order in the struct,
 // the compiler can pick up on just a few simd optimizations by itself.
 v0: u64,
 v2: u64,
 v1: u64,
 v3: u64,
}

macro_rules! compress {
 ($state:expr) => {{ compress!($state.v0, $state.v1, $state.v2, $state.v3) }};
 ($v0:expr, $v1:expr, $v2:expr, $v3:expr) => {{
 $v0 = $v0.wrapping_add($v1);
 $v2 = $v2.wrapping_add($v3);
 $v1 = $v1.rotate_left(13);
 $v1 ^= $v0;
 $v3 = $v3.rotate_left(16);
 $v3 ^= $v2;
 $v0 = $v0.rotate_left(32);

 $v2 = $v2.wrapping_add($v1);
 $v0 = $v0.wrapping_add($v3);
 $v1 = $v1.rotate_left(17);
 $v1 ^= $v2;
 $v3 = $v3.rotate_left(21);
 $v3 ^= $v0;
 $v2 = $v2.rotate_left(32);
 }};
}

Read the field order: v0, v2, v1, v3. Not v0, v1, v2, v3.

The algorithm interleaves the four words as two independent pairs: v0 accumulates from v1 and v3, v2 accumulates from v3 and v1. The rounds are ARX (add, rotate, xor) operations, and the two pairs have no data dependency between them within a round. A SIMD implementation would put v0 and v2 in one 128-bit lane and v1 and v3 in the other.

By declaring the struct in the order v0, v2, v1, v3, the two lanes are contiguous in memory, so LLVM’s SLP vectoriser can find them without having to prove the reordering itself. A naive field order would require the vectoriser to shuffle.

The comment is unusually explicit about the reasoning: “the compiler can pick up on just a few simd optimizations by itself”.

The round function is deliberately written as a macro

The compress! macro takes the four words as parameters, so the same code can be instantiated over a single State or over four separate locals (which is what the unrolled tail handling does). The macro also avoids re-reading the struct fields between rounds.

The tail, without a memcpy

 rust
 
 
// library/core/src/hash/sip.rs:115-144

/// Loads a u64 using up to 7 bytes of a byte slice. It looks clumsy but the
/// `copy_nonoverlapping` calls that occur (via `load_int_le!`) all have fixed
/// sizes and avoid calling `memcpy`, which is good for speed.
#[inline]
unsafe fn u8to64_le(buf: &[u8], start: usize, len: usize) -> u64 {

The comment says it all: the awkward-looking code is written so that every load is a fixed-size copy_nonoverlapping, which LLVM turns into a single mov instead of a call to memcpy.

The prefix-free string hack

 rust
 
 
// library/core/src/hash/sip.rs:302-308

#[inline]
fn write_str(&mut self, s: &str) {
 // This hasher works byte-wise, and `0xFF` cannot show up in a `str`,
 // so just hashing the one extra byte is enough to be prefix-free.
 self.write(s.as_bytes());
 self.write_u8(0xFF);
}

Hashing "ab" followed by "c" must not collide with "a" followed by "bc". The standard fix is to hash the length, which costs 8 bytes of hashing. Rust’s fix costs one byte: 0xFF cannot appear in a str, so appending it makes the encoding prefix-free without a length field.

Why SipHash at all?

The default RandomState uses SipHash-1-3 because HashMap is exposed to attacker-controlled keys. A non-keyed hash like FxHash is trivially collided: an attacker can find keys that all land in the same bucket and turn a lookup into a linear scan. SipHash’s keyed construction makes that require knowledge of the (random, per-thread, per-map) key.

The compiler does not use SipHash — rustc_data_structures has its own FxHash, which is a multiply-xor-shift hash:

The reason is the opposite: inside a compiler, the keys are not attacker-controlled, and the compiler hashes millions of small keys during a build. DoS resistance is worthless there; speed is everything.

Where it lives

library/core/src/hash/sip.rs — the state layout, the compression macro, the tail handling.
library/core/src/hash/mod.rs — the Hasher trait.
library/std/src/hash/random.rs — RandomState.
compiler/rustc_data_structures/src/fx.rs — the compiler’s own hash.

Takeaway

There is a class of performance hint that lives below the level of algorithms: the order of struct fields. Because LLVM’s SLP vectoriser works on adjacent values, putting v0 next to v2 — rather than in “natural” order — lets it discover a two-lane SIMD computation that it would otherwise have to prove. The cost is zero and the effect is measurable. It is also a lovely reminder that in a systems language, the source layout is part of the design.

Sixteen Buckets at a Time: The SwissTable in Rust's HashMap

Jay Hubbert — Wed, 21 Oct 2026 12:10:00 +0000

TL;DR: HashMap in Rust is a SwissTable: a power-of-two bucket array with a parallel array of control bytes, where the top 7 bits of each hash are stored in the control byte. A lookup loads 16 control bytes with one SSE2 instruction, compares them all at once, and uses movemask to find candidates — 16 comparisons for the price of one.

Note: HashMap in current Rust lives in library/std but the table implementation comes from the external hashbrown crate (hashbrown = "0.17.1" in library/std/Cargo.toml). The snippets below are from hashbrown-0.17.1.

Control bytes

 rust
 
 
// hashbrown-0.17.1/src/control/tag.rs:7-49

impl Tag {
 /// Control tag value for an empty bucket.
 pub(crate) const EMPTY: Tag = Tag(0b1111_1111);

 /// Control tag value for a deleted bucket.
 pub(crate) const DELETED: Tag = Tag(0b1000_0000);

 /// Checks whether a control tag represents a full bucket (top bit is clear).
 #[inline]
 pub(crate) const fn is_full(self) -> bool {
 self.0 & 0x80 == 0
 }
 ...
 /// Creates a control tag representing a full bucket with the given hash.
 #[inline]
 pub(crate) const fn full(hash: u64) -> Tag {
 ...
 // Grab the top 7 bits of the hash. While the hash is normally a full 64-bit
 // value, some hash functions (such as FxHash) produce a usize result
 // instead, which means that the top 32 bits are 0 on 32-bit platforms.
 // So we use MIN_HASH_LEN constant to handle this.
 let top7 = hash >> (MIN_HASH_LEN * 8 - 7);
 Tag((top7 & 0x7f) as u8) // truncation
 }
}

The layout is a deliberate bit-budget:

the top bit distinguishes “special” (EMPTY or DELETED) from “full”;
the remaining 7 bits are h2, the top 7 bits of the hash.

That split is what makes the SIMD scan possible: “is this bucket free?” is “does the top bit of this byte have the high bit set?”, which movemask answers for 16 bytes at once.

The group scan

 rust
 
 
// hashbrown-0.17.1/src/control/group/sse2.rs:46-116

/// Loads a group of tags starting at the given address.
#[inline]
pub(crate) unsafe fn load(ptr: *const Tag) -> Self {
 unsafe { Group(x86::_mm_loadu_si128(ptr.cast())) }
}
...
/// Returns a `BitMask` indicating all tags in the group which have
/// the given value.
#[inline]
pub(crate) fn match_tag(self, tag: Tag) -> BitMask {
 unsafe {
 let cmp = x86::_mm_cmpeq_epi8(self.0, x86::_mm_set1_epi8(tag.0 as i8));
 BitMask(x86::_mm_movemask_epi8(cmp) as u16)
 }
}
...
/// Returns a `BitMask` indicating all tags in the group which are `EMPTY` or `DELETED`.
#[inline]
pub(crate) fn match_empty_or_deleted(&self) -> BitMask {
 unsafe {
 // A tag is EMPTY or DELETED iff the high bit is set
 BitMask(x86::_mm_movemask_epi8(self.0) as u16)
 }
}

#[inline]
pub(crate) fn match_full(&self) -> BitMask {
 BitMask(!self.match_empty_or_deleted().0)
}

Two movemask results: which of the 16 bytes equal my h2 tag, and which are free. One _mm_cmpeq_epi8 plus one _mm_movemask_epi8 does the work of 16 byte comparisons.

And the comment about AVX explains why the table stopped at SSE2:

 rust
 
 
// hashbrown-0.17.1/src/control/group/mod.rs:8-23

// Use the SSE2 implementation if possible: it allows us to scan 16 buckets
// at once instead of 8. We don't bother with AVX since it would require
// runtime dispatch and wouldn't gain us much anyways: the probability of
// finding a match drops off drastically after the first few buckets.

Runtime dispatch for AVX would cost more than it saves, because the probability of finding a match drops off quickly.

The lookup loop

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:2007-2045

#[inline(always)]
unsafe fn find_inner(&self, hash: u64, eq: &mut dyn FnMut(usize) -> bool) -> Option<usize> {
 let tag_hash = Tag::full(hash);
 let mut probe_seq = self.probe_seq(hash);

 loop {
 let group = unsafe { Group::load(self.ctrl(probe_seq.pos)) };

 for bit in group.match_tag(tag_hash) {
 // This is the same as `(probe_seq.pos + bit) % self.num_buckets()` because the number
 // of buckets is a power of two, and `self.bucket_mask = self.num_buckets() - 1`.
 let index = (probe_seq.pos + bit) & self.bucket_mask;

 if likely(eq(index)) {
 return Some(index);
 }
 }

 if likely(group.match_empty().any_bit_set()) {
 return None;
 }

 probe_seq.move_next(self.bucket_mask);
 }
}

Three optimisations in twenty lines:

The h2 filter. The expensive key comparison (eq, which may call Eq::eq on a user type) only runs for buckets whose 7-bit tag matches — a 1/128 false-positive rate.
The empty short-circuit. Finding an EMPTY byte means the key is definitely not in the table, so the lookup ends there.
Mask instead of modulo. & self.bucket_mask replaces %, because the bucket count is a power of two.

Triangular probing

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:57-93

/// Primary hash function, used to select the initial bucket to probe from.
#[inline]
#[expect(clippy::cast_possible_truncation)]
fn h1(hash: u64) -> usize {
 // On 32-bit platforms we simply ignore the higher hash bits.
 hash as usize
}

/// Probe sequence based on triangular numbers, which is guaranteed (since our
/// table size is a power of two) to visit every group of elements exactly once.
///
/// A triangular probe has us jump by 1 more group every time. So first we
/// jump by 1 group (meaning we just continue our linear scan), then 2 groups
/// (skipping over 1 group), then 3 groups (skipping over 2 groups), and so on.
///
/// Proof that the probe will visit every group in the table:
/// <https://fgiesen.wordpress.com/2015/02/22/triangular-numbers-mod-2n/>
#[derive(Clone)]
struct ProbeSeq {
 pos: usize,
 stride: usize,
}

impl ProbeSeq {
 #[inline]
 fn move_next(&mut self, bucket_mask: usize) {
 // We should have found an empty bucket by now and ended the probe.
 debug_assert!(
 self.stride <= bucket_mask,
 "Went past end of probe sequence"
 );

 self.stride += Group::WIDTH;
 self.pos += self.stride;
 self.pos &= bucket_mask;
 }
}

The stride grows by the group width (16 on SSE2) each step, so the probe visits every group exactly once — triangular numbers mod 2ⁿ are a permutation. Note this is not robin-hood hashing and not triple hashing; it is group-wise quadratic probing with a triangular stride.

The load factor

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:104-191

// Otherwise require 1/8 buckets to be empty (87.5% load)
//
// Be careful when modifying this, calculate_layout relies on the
// overflow check here.
let adjusted_cap = cap.checked_mul(8)? / 7;
...
// Returns the maximum effective capacity for the given bucket mask, taking
// the maximum load factor into account.
#[inline]
fn bucket_mask_to_capacity(bucket_mask: usize) -> usize {
 if bucket_mask < 8 {
 // For tables with 1/2/4/8 buckets, we always reserve one empty slot.
 // Keep in mind that the bucket mask is one less than the bucket count.
 bucket_mask
 } else {
 // For larger tables we reserve 12.5% of the slots as empty.
 ((bucket_mask + 1) / 8) * 7
 }
}

87.5% maximum load: the slack keeps probe chains short, and the empty-slot reservation is what makes the “found an empty ⇒ absent” shortcut valid.

The backwards data layout

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:660-687

/// Returns pointer to one past last `data` element in the table as viewed from
/// the start point of the allocation.
...
pub(crate) fn data_end(&self) -> NonNull<T> {
 // `self.table.ctrl.cast()` returns pointer that
 // points here (to the end of `T0`)
 // ∨
 // [Pad], T_n, ..., T1, T0, |CT0, CT1, ..., CT_n|, CTa_0, CTa_1, ..., CTa_m
 // \________ ________/
 // \/
 // `n = buckets - 1`, i.e. `RawTable::num_buckets() - 1`
 //
 // where: T0...T_n - our stored data;
 // CT0...CT_n - control bytes or metadata for `data`.
 // CTa_0...CTa_m - additional control bytes, where `m = Group::WIDTH - 1` (so that the search
 // with loading `Group` bytes from the heap works properly, even if the result
 // of `h1(hash) & self.bucket_mask` is equal to `self.bucket_mask`). See also
 // `RawTableInner::set_ctrl` function.
 self.table.ctrl.cast()
}

Elements are stored backwards relative to the control bytes, so that data_end() is simply “the control pointer” — one allocation, and the control array sits at the end where the group loads can over-read safely (the extra CTa_* bytes are the padding that makes a group load at the last position legal).

The iterator then walks the control bytes forward and subtracts to walk the data backwards:

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:460-471

#[inline]
unsafe fn next_n(&self, offset: usize) -> Self {
 let ptr = if T::IS_ZERO_SIZED {
 // invalid pointer is good enough for ZST
 invalid_mut(self.ptr.as_ptr() as usize + offset)
 } else {
 unsafe { self.ptr.as_ptr().sub(offset) }
 };

This backwards layout is also why HashMap’s iteration order is arbitrary: it is the raw bucket order, deliberately unspecified.

The random seed, once per thread

 rust
 
 
// library/std/src/hash/random.rs:41-78

pub fn new() -> RandomState {
 // Historically this function did not cache keys from the OS and instead
 // simply always called `rand::thread_rng().gen()` twice. In #31356 it
 // was discovered, however, that because we re-seed the thread-local RNG
 // from the OS periodically that this can cause excessive slowdown when
 // many hash maps are created on a thread. To solve this performance
 // trap we cache the first set of randomly generated keys per-thread.
 //
 // Later in #36481 it was discovered that exposing a deterministic
 // iteration order allows a form of DOS attack. To counter that we
 // increment one of the seeds on every RandomState creation, giving
 // every corresponding HashMap a different iteration order.
 thread_local!(static KEYS: Cell<(u64, u64)> = {
 Cell::new(hashmap_random_keys())
 });

 KEYS.with(|keys| {
 let (k0, k1) = keys.get();
 keys.set((k0.wrapping_add(1), k1));
 RandomState { k0, k1 }
 })
}

The OS entropy is fetched once per thread (a getrandom syscall is ~1µs; creating a HashMap must not cost a syscall), while the seed is bumped per map so two maps in the same thread still have different iteration orders.

Where it lives

hashbrown-0.17.1/src/control/tag.rs — the control byte encoding.
hashbrown-0.17.1/src/control/group/sse2.rs — the SIMD group scan.
hashbrown-0.17.1/src/raw.rs — probing, growth, the backwards layout.
library/std/src/hash/random.rs — RandomState.

Takeaway

The SwissTable’s core idea is that a hash table’s expensive part is key comparison, and you can avoid most of it by storing a cheap fingerprint (next to the data) and comparing those in bulk with SIMD. Sixteen candidates at a time, with a 1/128 false-positive rate, is a very good trade — and it is the reason Rust’s HashMap is competitive with C++’s unordered_map.

collect() Reuses Your Allocation: Rust's In-Place Collection

Jay Hubbert — Wed, 21 Oct 2026 12:05:00 +0000

TL;DR: vec.into_iter().map(f).collect::<Vec<_>>() can run entirely inside the original allocation. Rust implements this with a specialisation trait (InPlaceCollect) that checks the size and alignment relationship between the source and destination element types, then reads and writes through two cursors into the same buffer.

The idea

Collecting an iterator normally means: allocate a new buffer, write every element into it, free the old one. But if the iterator is already walking a Vec, and the element types are compatible, you can just write the transformed elements back over the source.

The catch is that the source and destination element types may have different sizes. If the destination is bigger, writing in place would overwrite elements that have not been read yet. The solution is to shrink the allocation first, or to bail out to the normal path.

The layout contract

The source documents the rules precisely:

 rust
 
 
// library/alloc/src/vec/in_place_collect.rs:40-61

// Ultimately each step of the iterator must free up enough *bytes* in the source to make room
// for the next output item.
// If `T` and `U` have the same size no fixup is needed.
// If `T`'s size is a multiple of `U` we can compensate by multiplying the capacity accordingly.
// Otherwise the input capacity (and thus layout) in bytes may not be representable by the output
// `Vec<U>`. In that case `alloc.shrink()` is used to update the allocation's layout.
//
// Alignments of `T` must be the same or larger than `U`. ...

So:

same size → trivial;
source size is a multiple of destination size → the destination capacity is scaled up, and the byte budget works out;
otherwise → the allocation is shrunk to fit the new layout, which is allowed for Vec but is a real cost.

The implementation

 rust
 
 
// library/alloc/src/vec/in_place_collect.rs:249-337

fn from_iter_in_place<I, T>(mut iterator: I) -> Vec<T>
where
 I: Iterator<Item = T> + InPlaceCollect,
 <I as SourceIter>::Source: AsVecIntoIter,
{
 unsafe {
 let inner = iterator.as_inner().as_into_iter();
 (
 inner.buf,
 inner.ptr,
 inner.cap,
 inner.buf.cast::<T>(),
 inner.end as *const T,
 // SAFETY: the multiplication can not overflow, since `inner.cap * size_of::<I::SRC>()` is the size of the allocation.
 inner.cap.unchecked_mul(size_of::<I::Src>()) / size_of::<T>(),
 )
 };

 let len = unsafe {
 SpecInPlaceCollect::collect_in_place(&mut iterator, dst_buf.as_ptr() as *mut T, dst_end)
 };
 ...
 if needs_realloc::<I::Src, T>(src_cap, dst_cap) {
 ...
 let result = alloc.shrink(dst_buf.cast(), old_layout, new_layout);
 let Ok(reallocated) = result else { handle_alloc_error(new_layout) };
 dst_buf = reallocated.cast::<T>();
 }
 ...
 unsafe { Vec::from_parts(dst_buf, len, dst_cap) }
}

The key line is inner.cap * size_of::<I::Src>() / size_of::<T>(): the destination capacity is computed in bytes and then converted, which is what makes the “source is a multiple of destination” case work.

And the result is documented in the source, which is unusual and lovely:

rust

// library/alloc/src/vec/in_place_collect.rs:118-126

Does not allocate, free or panic. On optlevel>=2 it does not loop.

So on optimized builds, vec.into_iter().map(|u| u as isize).collect::<Vec<isize>>() compiles to a counted loop that LLVM can even eliminate.

The specialisation hierarchy

The InPlaceCollect trait is implemented for the iterators that can support it:

rust

// library/alloc/src/vec/in_place_collect.rs:389-413

with the TrustedRandomAccessNoCoerce specialisation being the one that allows the “does not loop” claim.

Even when in-place collection does not apply, collect can avoid the allocation:

 rust
 
 
// library/alloc/src/vec/spec_from_iter.rs:37-64

impl<T> SpecFromIter<T, IntoIter<T>> for Vec<T> {
 fn from_iter(iterator: IntoIter<T>) -> Self {
 // A common case is passing a vector into a function which immediately
 // re-collects into a vector. We can short circuit this if the IntoIter
 // has not been advanced at all.
 // When it has been advanced We can also reuse the memory and move the data to the front.
 // But we only do so when the resulting Vec wouldn't have more unused capacity
 // than creating it through the generic FromIterator implementation would. ...
 let has_advanced = iterator.buf != iterator.ptr;
 if !has_advanced || iterator.len() >= iterator.cap / 2 {
 let it = ManuallyDrop::new(iterator);
 unsafe {
 if has_advanced {
 ptr::copy(it.ptr.as_ptr(), it.buf.as_ptr(), it.len());
 }
 return Vec::from_parts(it.buf, it.len(), it.cap);
 }
 }
 let mut vec = Vec::new();
 vec.spec_extend(iterator);
 vec
 }
}

let v2: Vec<_> = v1.into_iter().collect(); is zero allocations and zero copies when the iterator has not been advanced — and at most one memmove otherwise. The iterator.len() >= iterator.cap / 2 guard prevents keeping a mostly-empty allocation alive.

The `SpecExtend` chain

 rust
 
 
// library/alloc/src/vec/spec_extend.rs:13-59

impl<T, I, A: Allocator> SpecExtend<T, I> for Vec<T, A>
where
 I: Iterator<Item = T>,
{
 default fn spec_extend(&mut self, iter: I) {
 self.extend_desugared(iter)
 }
}

impl<T, I, A: Allocator> SpecExtend<T, I> for Vec<T, A>
where
 I: TrustedLen<Item = T>,
{
 default fn spec_extend(&mut self, iterator: I) {
 self.extend_trusted(iterator)
 }
}

impl<T, A1: Allocator, A2: Allocator> SpecExtend<T, IntoIter<T, A2>> for Vec<T, A1> {
 fn spec_extend(&mut self, iterator: IntoIter<T, A2>) {
 unsafe {
 self.append_elements(iterator.as_slice() as _);
 }
 iterator.forget_remaining_elements_and_dealloc();
 }
}

Three tiers:

TrustedLen → reserve exactly once, then write without per-element capacity checks;
IntoIter → a single memcpy plus “forget the rest of the source”;
slice::Iter with a trivially-clonable T → also a single memcpy.

The TrustedLen path is the interesting one:

 rust
 
 
// library/alloc/src/vec/mod.rs:4231-4264

fn extend_trusted(&mut self, iterator: impl iter::TrustedLen<Item = T>) {
 let (low, high) = iterator.size_hint();
 if let Some(additional) = high {
 debug_assert_eq!(low, additional, "TrustedLen iterator's size hint is not exact: {:?}", (low, high));
 self.reserve(additional);
 let ptr = self.as_mut_ptr();
 let mut local_len = SetLenOnDrop::new(&mut self.len);
 unsafe {
 iterator.for_each(move |element| {
 ptr::write(ptr.add(local_len.current_len()), element);
 // Since the loop executes user code which can panic we have to update
 // the length every step to correctly drop what we've written.
 local_len.increment_len(1);
 });
 }
 } else {
 panic!("capacity overflow");
 }
}

One reserve, then a tight loop with no capacity check. SetLenOnDrop keeps self.len accurate even if the iterator body panics, so partially written elements are dropped correctly.

Where it lives

library/alloc/src/vec/in_place_collect.rs — the in-place machinery.
library/alloc/src/vec/spec_from_iter.rs — allocation stealing.
library/alloc/src/vec/spec_extend.rs — the specialisation chain.
library/alloc/src/vec/mod.rs — extend_trusted, extend_desugared.

Takeaway

collect() is usually thought of as “allocate and fill”. But the common patterns are: extend a Vec with another Vec, or map over a Vec. Both can be done without touching the allocator at all, and Rust implements them with the same trick — specialise on the iterator type, then check the layout relationship between source and destination. The result is a collect() that, in the common case, is a loop.

Vec's Capacity Lives in a Niche: How Rust's RawVec Got One Word Smaller

Jay Hubbert — Wed, 21 Oct 2026 12:00:00 +0000

TL;DR: Vec<T> is three machine words: pointer, length, capacity. It manages that by storing the capacity in a niche type — a usize whose top bit is guaranteed to be zero — so the “zero capacity” case is representable without an extra discriminant. And the growth policy is max(2 * cap, required), with a special minimum for tiny vectors.

The growth policy

 rust
 
 
// library/alloc/src/raw_vec/mod.rs:539-555

// Nothing we can really do about these checks, sadly.
let required_cap = len.checked_add(additional).ok_or(CapacityOverflow)?;

// This guarantees exponential growth. The doubling cannot overflow
// because `cap <= isize::MAX` and the type of `cap` is `usize`.
let cap = cmp::max(self.cap.as_inner() * 2, required_cap);
let cap = cmp::max(min_non_zero_cap(elem_layout.size()), cap);

Three lines:

checked_add catches the length overflow and turns it into a CapacityOverflow error rather than a panic with a confusing message.
max(2*cap, required) is the amortized-doubling rule. If the caller asked for a specific capacity, that wins; otherwise double.
max(min_non_zero_cap(size), cap) imposes a floor.

The doubling cannot overflow because cap <= isize::MAX and the multiplication is done in usize. That invariant is why the code can use plain * instead of checked_mul.

The tiny-vector floor

 rust
 
 
// library/alloc/src/raw_vec/mod.rs:163-176

// Tiny Vecs are dumb. Skip to:
// - 8 if the element size is 1, because any heap allocator is likely
// to round up a request of less than 8 bytes to at least 8 bytes.
// - 4 if elements are moderate-sized (<= 1 KiB).
// - 1 otherwise, to avoid wasting too much space for very short Vecs.
const fn min_non_zero_cap(size: usize) -> usize {
 if size == 1 {
 8
 } else if size <= 1024 {
 4
 } else {
 1
 }
}

The reasoning is allocator-aware: malloc rounds a 1-byte request up to a 16-byte chunk anyway, so allocating 1 byte and then reallocating to 8 is pure waste. For medium-sized elements, 4 gives a reasonable first allocation; for large elements, 1 avoids over-allocating.

The niche

 rust
 
 
// library/alloc/src/raw_vec/mod.rs:40-51

type Cap = core::num::niche_types::UsizeNoHighBit;

// SAFETY: 0 *definitely* is less than isize::MAX.
const ZERO_CAP: Cap = unsafe { Cap::new_unchecked(0) };

/// `Cap(cap)`, except if `T` is a ZST then `Cap::ZERO`.
///
/// # Safety: cap must be <= `isize::MAX`.
const unsafe fn new_cap<T>(cap: usize) -> Cap {
 // SAFETY: Upheld by caller.
 if T::IS_ZST { ZERO_CAP } else { unsafe { Cap::new_unchecked(cap) } }
}

UsizeNoHighBit is a usize with an invariant: the top bit is always zero. That is a niche — a value the type cannot represent — and Rust’s niche optimization uses it to make Option<Cap> the same size as Cap.

Why does that matter? Because RawVec needs to represent “no allocation” (distinguished by a null pointer) and a capacity. With the invariant, the capacity’s top bit is free to be used as a flag… and, in fact, Cap::ZERO is the zero-capacity case, which is also the ZST case.

The payoff: Vec<T> is { ptr, cap, len } — three words — instead of four. For a Vec<u8> of 24 bytes on the stack, that is 8 bytes saved on every Vec in the program, including the ones on the stack of every function.

`into_raw_parts` and `from_raw_parts`

 rust
 
 
// library/alloc/src/vec/mod.rs:795-801

#[must_use = "losing the pointer will leak memory"]
#[stable(feature = "vec_into_raw_parts", since = "1.93.0")]
#[rustc_const_unstable(feature = "const_heap", issue = "79507")]
pub const fn into_raw_parts(self) -> (*mut T, usize, usize) {
 let mut me = ManuallyDrop::new(self);
 (me.as_mut_ptr(), me.len(), me.capacity())
}

ManuallyDrop suppresses the destructor so the buffer is not freed, while the pointer/length/capacity are read out of the value. The reconstruction side is the mirror image:

 rust
 
 
// library/alloc/src/vec/mod.rs:1127-1140

pub const unsafe fn from_raw_parts_in(
 ptr: *mut T,
 length: usize,
 capacity: usize,
 alloc: A,
) -> Self {
 ub_checks::assert_unsafe_precondition!(
 check_library_ub,
 "Vec::from_raw_parts_in requires that length <= capacity",
 (length: usize = length, capacity: usize = capacity) => length <= capacity
 );
 // SAFETY: Upheld by caller.
 unsafe { Vec { buf: RawVec::from_raw_parts_in(ptr, capacity, alloc), len: length } }
}

It is const, and the only validation is a debug-mode UB check. That makes it a zero-cost FFI boundary primitive: Vec::into_raw_parts and Vec::from_raw_parts round-trip through C without touching the allocator at all.

`drain` is destructive but reversible

 rust
 
 
// library/alloc/src/vec/drain.rs:88-137

pub fn keep_rest(self) {
 // At this moment layout looks like this:
 //
 // [head] [yielded by next] [unyielded] [yielded by next_back] [tail]
 // ^-- start \_________/-- unyielded_len \____/-- self.tail_len
 // ^-- unyielded_ptr ^-- tail
 ...
 let mut this = ManuallyDrop::new(self);
 unsafe {
 let source_vec = this.vec.as_mut();
 let start = source_vec.len();
 let tail = this.tail_start;
 let unyielded_len = this.iter.len();
 let unyielded_ptr = this.iter.as_slice().as_ptr();
 // ZSTs have no identity, so we don't need to move them around.
 if !T::IS_ZST {
 let start_ptr = source_vec.as_mut_ptr().add(start);
 // memmove back unyielded elements
 if unyielded_ptr != start_ptr {
 ptr::copy(src, dst, unyielded_len);
 }
 // memmove back untouched tail
 if tail != (start + unyielded_len) {
 ptr::copy(src, dst, this.tail_len);
 }
 }
 source_vec.set_len(start + unyielded_len + this.tail_len);
 }
}

The tail is restored with one memmove, not one move per element. And the Drop implementation uses a nested guard struct so the memmove happens even if the user’s Drop impl panics mid-drain:

 rust
 
 
// library/alloc/src/vec/drain.rs:176-200

impl<T, A: Allocator> Drop for Drain<'a, T, A> {
 fn drop(&mut self) {
 /// Moves back the un-`Drain`ed elements to restore the original `Vec`.
 struct DropGuard<'r, 'a, T, A: Allocator>(&'r mut Drain<'a, T, A>);
 impl<'r, 'a, T, A: Allocator> Drop for DropGuard<'r, 'a, T, A> {
 fn drop(&mut self) {
 if self.0.tail_len > 0 {
 unsafe {
 let source_vec = self.0.vec.as_mut();
 // memmove back untouched tail, update to new length
 let start = source_vec.len();
 let tail = self.0.tail_start;
 if tail != start {
 let src = source_vec.as_ptr().add(tail);
 let dst = source_vec.as_mut_ptr().add(start);
 ptr::copy(src, dst, self.0.tail_len);
 }
 source_vec.set_len(start + self.0.tail_len);
 }
 }
 }
 }

The naive alternative — Vec::remove in a loop — is O(n²) in moves.

Where it lives

library/alloc/src/raw_vec/mod.rs — the growth policy, min_non_zero_cap, the Cap niche.
library/alloc/src/vec/mod.rs — into_raw_parts, from_raw_parts_in.
library/alloc/src/vec/drain.rs — the DropGuard trick.
library/core/src/num/niche_types.rs — UsizeNoHighBit.

Takeaway

The niche trick here is a nice illustration of a general Rust idiom: when a type has a value it can never hold, that value is free. usize can never be usize::MAX in a capacity, so the top bit is free, so Option<Cap> is free, so Vec is one word smaller. Across a program with thousands of live Vecs, that is megabytes of stack and heap.

Did You Mean a Hyphen?: How rustc Repairs Unicode Confusables While Parsing

Jay Hubbert — Wed, 21 Oct 2026 11:55:00 +0000

TL;DR: If you paste an en-dash into Rust source, rustc does not just complain — it looks the character up in a static confusables table, substitutes the real ASCII token (-), keeps parsing, and reports the problem as a side effect. The diagnostic is generated from the same table that produced the recovery.

Copy-pasting from a word processor is a common way to introduce Unicode characters that look like ASCII ones. The classic offenders:

– (en dash) instead of -;
‘ / ’ (curly quotes) instead of ' and ";
non-breaking spaces;
full-width characters from CJK input methods.

rustc’s response is unusually good: it does not just point at the character, it fixes the parse and tells you what it did.

The table

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:1-12

//! Characters and their corresponding confusables were collected from
//! <https://www.unicode.org/Public/security/10.0.0/confusables.txt>

use rustc_span::{BytePos, Pos, Span, kw};

use super::Lexer;
use crate::diagnostics::TokenSubstitution;
use crate::token;

#[rustfmt::skip] // for line breaks
pub(super) static UNICODE_ARRAY: &[(char, &str, &str)] = &[
 (' ', "Line Separator", " "),
 (' ', "Paragraph Separator", " "),

Each entry is (unicode char, unicode name, ASCII replacement). The relevant rows:

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:37-59, 131-166, 168-184

 ('‐', "Hyphen", "-"),
 ('‑', "Non-Breaking Hyphen", "-"),
 ('‒', "Figure Dash", "-"),
 ('–', "En Dash", "-"),
 ('—', "Em Dash", "-"),
 ...
 ('‘', "Left Single Quotation Mark", "'"),
 ('’', "Right Single Quotation Mark", "'"),
 ...
 ('“', "Left Double Quotation Mark", "\""),
 ('”', "Right Double Quotation Mark", "\""),

The table is generated from Unicode’s own confusables data file, so it covers thousands of characters.

The lookup returns a token

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:339-380

pub(super) fn check_for_substitution(
 lexer: &Lexer<'_, '_>,
 pos: BytePos,
 ch: char,
 count: usize,
) -> (Option<token::TokenKind>, Option<TokenSubstitution>) {
 let Some(&(_, u_name, ascii_str)) = UNICODE_ARRAY.iter().find(|&&(c, _, _)| c == ch) else {
 return (None, None);
 };

 let span = Span::with_root_ctxt(pos, pos + Pos::from_usize(ch.len_utf8() * count));

 let Some((_, ascii_name, token)) = ASCII_ARRAY.iter().find(|&&(s, _, _)| s == ascii_str) else {
 let msg = format!("substitution character not found for '{ch}'");
 lexer.dcx().span_bug(span, msg);
 };

 // special help suggestion for "directed" double quotes
 let sugg = if let Some(s) = peek_delimited(&lexer.src[lexer.src_index(pos)..], '“', '”') {
 let span = Span::with_root_ctxt(
 pos,
 pos + Pos::from_usize('“'.len_utf8() + s.len() + '”'.len_utf8()),
 );
 Some(TokenSubstitution::DirectedQuotes {
 span,
 suggestion: format!("\"{s}\""),
 ascii_str,
 ascii_name,
 })
 } else {
 let suggestion = ascii_str.to_string().repeat(count);
 Some(TokenSubstitution::Other {
 span,
 suggestion,
 ch: ch.to_string(),
 u_name,
 ascii_str,
 ascii_name,
 })
 };
 (*token, sugg)
}

Three things to notice:

The return value is a real TokenKind. The ASCII_ARRAY maps the ASCII replacement back to a token, so an en-dash becomes token::Minus. The parser continues with a valid token stream.
The diagnostic is a by-product. The same function that performs the substitution also builds the suggestion. No second pass, no second table.
The “directed quotes” special case. If the input has “hello”, the suggestion replaces the whole quoted run with "hello", which is a much better fix than replacing each smart quote separately:

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:383-391

/// Extract string if found at current position with given delimiters
fn peek_delimited(text: &str, from_ch: char, to_ch: char) -> Option<&str> {
 let mut chars = text.chars();
 let first_char = chars.next()?;
 if first_char != from_ch {
 return None;
 }
 let last_char_idx = chars.as_str().find(to_ch)?;
 Some(&chars.as_str()[..last_char_idx])
}

Where it is invoked, and the “swallow the repeats” trick

 rust
 
 
// compiler/rustc_parse/src/lexer/mod.rs:486-514

let repeats = it.take_while(|c1| *c1 == c).count();
// FIXME: the lexer could be used to turn the ASCII version of unicode
// homoglyphs, instead of keeping a table in `check_for_substitution`into the
// token. Ideally, this should be inside `rustc_lexer`. However, we should
// first remove compound tokens like `<<` from `rustc_lexer`, and then add
// fancier error recovery to it, as there will be less overall work to do this
// way.
let (token, sugg) =
 unicode_chars::check_for_substitution(self, start, c, repeats + 1);
self.dcx().emit_err(crate::diagnostics::UnknownTokenStart {
 span: self.mk_sp(start, self.pos + Pos::from_usize(repeats * c.len_utf8())),
 escaped: escaped_char(c),
 sugg,
 null: c == '\x00',
 invisible: INVISIBLE_CHARACTERS.contains(&c),
 repeat: if repeats > 0 {
 swallow_next_invalid = repeats;
 Some(crate::diagnostics::UnknownTokenRepeat { repeats })
 } else {
 None
 },
});

The swallow_next_invalid mechanism suppresses the next N identical invalid characters, so @@@@@@ produces one error (“character appears 5 more times”) instead of six. The sub-diagnostic renders it:

 rust
 
 
// compiler/rustc_parse/src/diagnostics.rs:2890-2899

#[derive(Subdiagnostic)]
#[note(
 "character appears {$repeats ->
 [one] once more
 *[other] {$repeats} more times
 }"
)]
pub(crate) struct UnknownTokenRepeat {
 pub repeats: usize,
}

Note also the FIXME: the authors think this repair machinery should live in rustc_lexer itself, once the lexer has compound tokens removed and better recovery. It is a known piece of technical debt, documented as such.

The suggestion is deliberately not auto-applied

 rust
 
 
// compiler/rustc_parse/src/diagnostics.rs:2858-2888

#[derive(Subdiagnostic)]
pub(crate) enum TokenSubstitution {
 #[suggestion(
 "Unicode characters '“' (Left Double Quotation Mark) and '”' (Right Double Quotation Mark) look like '{$ascii_str}' ({$ascii_name}), but are not",
 code = "{suggestion}",
 applicability = "maybe-incorrect",
 style = "verbose"
 )]
 DirectedQuotes { ... },

applicability = "maybe-incorrect" means rustfix will offer the fix but will not apply it automatically. For a confusable replacement, that is the right call: the character might be intentional.

The other unicode trick: fast-path NFC normalisation

 rust
 
 
// compiler/rustc_parse/src/lexer/mod.rs:1278-1287

pub fn nfc_normalize(string: &str) -> Symbol {
 use unicode_normalization::{IsNormalized, UnicodeNormalization, is_nfc_quick};
 match is_nfc_quick(string.chars()) {
 IsNormalized::Yes => Symbol::intern(string),
 _ => {
 let normalized_str: String = string.chars().nfc().collect();
 Symbol::intern(&normalized_str)
 }
 }
}

Every identifier goes through this. is_nfc_quick is a cheap O(n) check that answers “is this already normalised?”; the expensive full normalisation only runs for identifiers that actually need it. Identifiers are normalised so that é written as one codepoint and as e + combining acute are the same identifier.

Where it lives

compiler/rustc_parse/src/lexer/unicode_chars.rs — the tables and check_for_substitution.
compiler/rustc_parse/src/lexer/mod.rs — the invocation, the repeat suppression, nfc_normalize.
compiler/rustc_parse/src/diagnostics.rs — TokenSubstitution.

Takeaway

Error recovery is a design problem, not a diagnostics problem. The best recovery is the kind that lets the compiler keep going and produces a better message than a plain error would. rustc’s confusable handling does both from a single table: the same lookup that produces the token produces the suggestion, so the recovery is essentially free.

Capturing Tokens by Replaying Them: rustc's Lazy Token Capture

Jay Hubbert — Wed, 21 Oct 2026 11:50:00 +0000

TL;DR: rustc’s parser does not collect tokens into a buffer as it parses. It takes a snapshot of the token cursor and remembers the bump count; the captured range is a half-open interval of bump counts. The tokens are materialised by replaying the cursor, and only if something actually asks for them — which is rare.

The problem

Some AST nodes need their original tokens: anything that will be handed to a proc macro, a derive, or an attribute that reads source text. Most nodes do not.

Collecting tokens eagerly — appending to a Vec as the parser advances — would cost every node in the program. rustc instead does the opposite.

The snapshot

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:17-24

// When collecting tokens, this fully captures the start point. Usually its
// just after outer attributes, but occasionally it's before.
#[derive(Clone, Debug)]
pub(super) struct CollectPos {
 start_token: (Token, Spacing),
 cursor_snapshot: TokenCursor,
 start_pos: u32,
}

and the snapshot operation:

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:94-101

impl<'a> Parser<'a> {
 pub(super) fn collect_pos(&self) -> CollectPos {
 CollectPos {
 start_token: (self.token, self.token_spacing),
 cursor_snapshot: self.token_cursor.clone(),
 start_pos: self.num_bump_calls,
 }
 }

start_pos is a bump counter, not a byte offset or a token index. The captured range is [start_pos, end_pos) in bump-counter space.

Note that TokenCursor::clone() is cheap-ish: the cursor holds a Vec of enclosing delimiters and Rc-shared token trees. The token trees are shared, not copied.

The end position, with the ungluing correction

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:278-294

assert!(
 !(self.break_last_token > 0 && matches!(capture_trailing, Trailing::Yes)),
 "Cannot have break_last_token > 0 and have trailing token"
);
assert!(self.break_last_token <= 2, "cannot break token more than twice");

let end_pos = self.num_bump_calls
 + capture_trailing as u32
 // If we "broke" the last token (e.g. breaking a `>>` token once into `>` + `>`, or
 // breaking a `>>=` token twice into `>` + `>` + `=`), then extend the range of
 // captured tokens to include it, because the parser was not actually bumped past it.
 // (Even if we broke twice, it was still just one token originally, hence the `1`.)
 // When the `LazyAttrTokenStream` gets converted into an `AttrTokenStream`, we will
 // rebreak that final token once or twice.
 + if self.break_last_token == 0 { 0 } else { 1 };

let num_calls = end_pos - collect_pos.start_pos;

The correction: if the last token was unglued (see the previous article), the range must include it, because the parser was never actually bumped past it. Even two breaks of >>= count as one original token.

The lazy stream

 rust
 
 
// compiler/rustc_ast/src/tokenstream.rs:217-239

enum LazyAttrTokenStreamInner {
 // The token stream has already been produced.
 Direct(AttrTokenStream),

 // From a value of this type we can reconstruct the `TokenStream` seen by
 // the `f` callback passed to a call to `Parser::collect_tokens`, by
 // replaying the getting of the tokens. This saves us producing a
 // `TokenStream` if it is never needed, e.g. a captured `macro_rules!`
 // argument that is never passed to a proc macro. In practice, token stream
 // creation happens rarely compared to calls to `collect_tokens` (see some
 // statistics in #78736) so we are doing as little up-front work as
 // possible.
 //
 // This also makes `Parser` very cheap to clone, since there is no
 // intermediate collection buffer to clone.
 Pending {
 start_token: (Token, Spacing),
 cursor_snapshot: TokenCursor,
 num_calls: u32,
 break_last_token: u32,
 node_replacements: ThinVec<NodeReplacement>,
 },
}

The comment mentions the statistics: token stream creation happens rarely compared to calls to collect_tokens. That is the empirical justification for the whole design.

And it gives a second benefit: the Parser becomes cheap to clone, because there is no intermediate collection buffer. And the parser is cloned — for every macro rule attempt during macro_rules! matching.

The replay

 rust
 
 
// compiler/rustc_ast/src/tokenstream.rs:252-263

// The token produced by the final call to `{,inlined_}next_and_bump` was not
// actually consumed by the callback. The combination of chaining the
// initial token and using `take` produces the desired result - we
// produce an empty `TokenStream` if no calls were made, and omit the
// final token otherwise.
let mut cursor_snapshot = cursor_snapshot.clone();
let tokens = iter::once(FlatToken::Token(*start_token))
 .chain(iter::repeat_with(|| FlatToken::Token(cursor_snapshot.next_and_bump())))
 .take(*num_calls as usize);

if node_replacements.is_empty() {
 make_attr_token_stream(tokens, *break_last_token)
} else {

The replay is a lazy iterator: once(start_token).chain(repeat_with(next)).take(n). The take is what stops it. And make_attr_token_stream(tokens, break_last_token) re-splits the final glued token using the same table the parser used — so the tokens handed to a proc macro are byte-identical to the source.

Deciding whether to capture: twice

The decision is made in two places, and the second one is sharper.

The early check, before parsing:

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:163-181

let possible_capture_mode = self.capture_cfg;

// We must collect if anything could observe the collected tokens, i.e.
// if any of the following conditions hold.
// - We are force collecting tokens (because force collection requires
// tokens by definition).
let needs_collection = matches!(force_collect, ForceCollect::Yes)
 // - Any of our outer attributes require tokens.
 || needs_tokens(&attrs.attrs)
 // - Our target supports custom inner attributes (custom
 // inner attribute invocation might require token capturing).
 || R::SUPPORTS_CUSTOM_INNER_ATTRS
 // - We are in "possible capture mode" (which requires tokens if
 // the parsed node has `#[cfg]` or `#[cfg_attr]` attributes).
 || possible_capture_mode;
if !needs_collection {
 return Ok(f(self, attrs.attrs)?.0);
}

and the late check, after parsing, when inner attributes are known:

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:230-266

// When we're not in "definite capture mode", then skip collecting and
// return early if `ret` doesn't support tokens or already has some.
//
// Note that this check is independent of `force_collect`. There's no
// need to collect tokens when we don't support tokens or already have
// tokens.
let definite_capture_mode = self.capture_cfg
 && matches!(self.capture_state.capturing, Capturing::Yes)
 && has_cfg_or_cfg_attr(&ret_attrs);
if !definite_capture_mode && !ret_can_hold_tokens {
 return Ok(ret);
}

The comment in the source is candid about the residual waste:

If we return early here that means the setup, such as cloning the token cursor, was unnecessary. That’s hard to avoid.

What actually triggers capture

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:410-427

fn needs_tokens(attrs: &[ast::Attribute]) -> bool {
 // Tokens are needed if...
 attrs.iter().any(|attr| match &attr.kind {
 AttrKind::Normal(normal) => {
 match normal.item.name() {
 // ... a multi-segment attribute is present, e.g. `rustfmt::skip`.
 None => true,
 // ... `cfg_attr` or a single-segment non-builtin attribute is present, e.g.
 // `derive`, `test`, `global_allocator`.
 Some(name) => name == sym::cfg_attr || !rustc_feature::is_builtin_attr_name(name),
 }
 }
 // Synthetic attributes are created only during expansion, and can't re-enter the parser
 // because they have no token form.
 AttrKind::Synthetic(_) => unreachable!(),
 AttrKind::DocComment(..) => false,
 })
}

Builtin attributes (#[doc], #[inline], #[allow]) do not need tokens. #[derive], #[test], #[global_allocator], #[cfg_attr] and any path-qualified attribute do.

The `IntervalSet` for seen attributes

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:274-283

// This state is used by `Parser::collect_tokens`.
#[derive(Clone, Debug)]
struct CaptureState {
 capturing: Capturing,
 parser_replacements: Vec<ParserReplacement>,
 inner_attr_parser_ranges: FxHashMap<AttrId, ParserRange>,
 // `IntervalSet` is good for perf because attrs are mostly added to this
 // set in contiguous ranges.
 seen_attrs: IntervalSet<AttrId>,
}

Attribute IDs are assigned monotonically, so the “already seen” set is mostly a few contiguous ranges — the same interval-set trick the borrow checker uses for regions, applied to a completely different problem.

Where it lives

compiler/rustc_parse/src/parser/attr_wrapper.rs — CollectPos, collect_pos, the two capture decisions.
compiler/rustc_ast/src/tokenstream.rs — LazyAttrTokenStream, the replay, TokenCursor.
compiler/rustc_parse/src/parser/mod.rs — CaptureState, seen_attrs.

Takeaway

“Capture the tokens” sounds like it needs a buffer. It does not — it needs a recipe for reconstructing them. Recording where you were and how far you went is enough, and the recipe is cheaper to store, cheaper to clone, and (crucially) only executed if someone asks. That is the difference between paying O(n) for every AST node and paying O(1).

One Token, Two Operators: How rustc Splices a >> Back Apart

Jay Hubbert — Wed, 21 Oct 2026 11:45:00 +0000

TL;DR: >> is a single token in Rust’s grammar, but Vec<Vec<u8>> needs two > tokens. rustc’s parser splits the glued token on demand, adjusting the spans so the two halves point at the right bytes, and tracking that it did so so that a proc macro can be handed the original glued token back.

The problem

Rust has no closing-bracket ambiguity problem like C++’s >>, but it has the same tokenization problem: >> is a single shift operator, and Vec<Vec<u8>> ends with two of them.

The parser cannot just re-lex the source, because by the time it is parsing, the source may have come from a macro expansion, a proc macro, or an include — the token stream is the source of truth, not the text.

Ungluing on demand

The parser keeps a counter of how many pieces it has split off:

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:204-222

// During parsing we may sometimes need to "unglue" a glued token into two
// or three component tokens (e.g. `>>` into `>` and `>`, or `>>=` into `>`
// and `>` and `=`), so the parser can consume them one at a time. This
// process bypasses the normal capturing mechanism (e.g. `num_bump_calls`
// will not be incremented), since the "unglued" tokens due not exist in the
// original `TokenStream`.
//
// If we end up consuming all the component tokens, this is not an issue,
// because we'll end up capturing the single "glued" token.
//
// However, sometimes we may want to capture not all of the original
// token. For example, capturing the `Vec<u8>` in `Option<Vec<u8>>`
// requires us to unglue the trailing `>>` token. The `break_last_token`
// field is used to track these tokens. They get appended to the captured
// stream when we evaluate a `LazyAttrTokenStream`.
//
// This value is always 0, 1, or 2. It can only reach 2 when splitting
// `>>=` or `<<=`.
break_last_token: u32 = 0,

The key sentence: “This process bypasses the normal capturing mechanism (e.g. num_bump_calls will not be incremented), since the ‘unglued’ tokens do not exist in the original TokenStream.”

That is what makes the capture accounting work: the captured range is measured in original tokens, and splitting a token does not change the count.

The split, with span surgery

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:753-783

/// Eats the expected token if it's present possibly breaking
/// compound tokens like multi-character operators in process.
/// Returns `true` if the token was eaten.
fn break_and_eat(&mut self, exp: ExpTokenPair) -> bool {
 if self.token == exp.tok {
 self.bump();
 return true;
 }
 match self.token.kind.break_two_token_op(1) {
 Some((first, second)) if first == exp.tok => {
 let first_span = self.psess.source_map().start_point(self.token.span);
 let second_span = self.token.span.with_lo(first_span.hi());
 self.token = Token::new(first, first_span);
 // Keep track of this token - if we end token capturing now,
 // we'll want to append this token to the captured stream.
 //
 // If we consume any additional tokens, then this token
 // is not needed (we'll capture the entire 'glued' token),
 // and `bump` will set this field to 0.
 self.break_last_token += 1;
 // Use the spacing of the glued token as the spacing of the
 // unglued second token.
 self.bump_with((Token::new(second, second_span), self.token_spacing));
 true
 }
 _ => {
 self.expected_token_types.insert(exp.token_type);
 false
 }
 }
}

Four details:

first_span is the start point of the glued token — so the first half points at the first byte.
second_span is the glued token’s span with its low end moved to first_span.hi() — so the second half points at the second byte. The two halves are byte-exact.
break_last_token += 1 records that a split happened, so the capture machinery knows to re-glue it later.
The spacing of the glued token is reused for the second half. Spacing (Joint vs Alone) is what the token stream uses to decide whether >> should be re-glued into one token when handed to a macro; getting it wrong would change the meaning of the macro input.

The gluing table

 rust
 
 
// compiler/rustc_ast/src/token.rs:546-584

/// An approximation to proc-macro-style single-character operators used by
/// rustc parser. If the operator token can be broken into two tokens, the
/// first of which has `n` (1 or 2) chars, then this function performs that
/// operation, otherwise it returns `None`.
pub fn break_two_token_op(&self, n: u32) -> Option<(TokenKind, TokenKind)> {
 assert!(n == 1 || n == 2);
 Some(match (self, n) {
 (Le, 1) => (Lt, Eq),
 (EqEq, 1) => (Eq, Eq),
 (Ne, 1) => (Bang, Eq),
 (Ge, 1) => (Gt, Eq),
 (AndAnd, 1) => (And, And),
 (OrOr, 1) => (Or, Or),
 (Shl, 1) => (Lt, Lt),
 (Shr, 1) => (Gt, Gt),
 ...
 (ShlEq, 1) => (Lt, Le), // `<` + `<=`
 (ShlEq, 2) => (Shl, Eq), // `<<` + `=`
 (ShrEq, 1) => (Gt, Ge), // `>` + `>=`
 (ShrEq, 2) => (Shr, Eq), // `>>` + `=`
 (DotDot, 1) => (Dot, Dot),
 (DotDotDot, 1) => (Dot, DotDot), // `.` + `..`
 (DotDotDot, 2) => (DotDot, Dot), // `..` + `.`
 (DotDotEq, 2) => (DotDot, Eq),
 (PathSep, 1) => (Colon, Colon),
 (RArrow, 1) => (Minus, Gt),
 (LArrow, 1) => (Lt, Minus),
 (FatArrow, 1) => (Eq, Gt),
 _ => return None,
 })
}

The n parameter is the interesting part. >>= can be split as > + >= (n=1) or as >> + = (n=2). The parser asks for the split that matches what it is looking for.

This table is the exact inverse of the gluing performed when a token stream is built:

rust

// compiler/rustc_ast/src/tokenstream.rs:692-720

so the parser and the token stream always agree about what >> means.

The reset on bump

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:1132-1151

/// Advance the parser by one token.
pub fn bump(&mut self) {
 // Note: destructuring here would give nicer code, but it was found in #96210 to be slower
 // than `.0`/`.1` access.
 let mut next = self.token_cursor.inlined_next_and_bump();
 self.num_bump_calls += 1;
 // We got a token from the underlying cursor and no longer need to
 // worry about an unglued token. See `break_and_eat` for more details.
 self.break_last_token = 0;
 if next.0.span.is_dummy() {
 // Tweak the location for better diagnostics, but keep syntactic context intact.
 let fallback_span = self.token.span;
 next.0.span = fallback_span.with_ctxt(next.0.span.ctxt());
 }
 ...

break_last_token = 0 on every real bump: “if we consumed any additional tokens, then this token is not needed (we’ll capture the entire ‘glued’ token)”. So the counter only survives if the split token is the last thing captured.

And the is_dummy() fixup: tokens synthesised by macro expansion have dummy spans; the parser substitutes the previous token’s span for better diagnostics — while keeping the hygiene context, which is the part that must not change.

Why byte-exactness matters

If the spans were not byte-exact, a proc macro receiving Option<Vec<u8>> would see two > tokens pointing at overlapping or wrong source ranges. Proc macros that re-emit their input (which is most of them) would then produce output with mangled spans, and the error messages downstream would point at nonsense.

This is why the split is done with with_lo(first_span.hi()) rather than with a fresh span: the split token must claim exactly the bytes it occupies.

Where it lives

compiler/rustc_parse/src/parser/mod.rs — break_and_eat, bump, break_last_token.
compiler/rustc_ast/src/token.rs — break_two_token_op.
compiler/rustc_ast/src/tokenstream.rs — the gluing side and TokenCursor.

Takeaway

Token-level hacks are usually where compilers accumulate their worst inconsistencies. rustc’s approach — keep the original token stream authoritative, split on demand with byte-exact spans, and record the split so it can be undone — keeps a single source of truth and makes the “give the proc macro back exactly what the user wrote” requirement fall out naturally.

macro_rules! Is an NFA: How Rust Matches Repetitions Without Backtracking

Jay Hubbert — Wed, 21 Oct 2026 11:40:00 +0000

TL;DR: macro_rules! matching in rustc is not a backtracking matcher. It is an NFA: for each input token, rustc maintains a set of live matcher positions, advances all of them, and follows epsilon transitions for repetitions. There is no rewind, no retry, and no exponential blowup from a rule like $($x:expr),*.

Matching macro_rules! is the part of macro expansion that everyone implements badly. The obvious implementation is recursive backtracking: try to match the rule, fail, back up, try the next rule. That is exponential in the worst case and makes the error messages terrible.

rustc took the other road.

The module documentation is the specification

The best documentation of this design is the header of the implementation file, which is a tutorial in itself:

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:1-14

//! This is an NFA-based parser, which calls out to the main Rust parser for named non-terminals
//! (which it commits to fully when it hits one in a grammar). There's a set of current NFA threads
//! and a set of next ones. Instead of NTs, we have a special case for Kleene star. The big-O, in
//! pathological cases, is worse than traditional use of NFA or Earley parsing, but it's an easier
//! fit for Macro-by-Example-style rules.
//!
//! (In order to prevent the pathological case, we'd need to lazily construct the resulting
//! `NamedMatch`es at the very end. It'd be a pain, and require more memory to keep around old
//! matcher positions, but it would also save overhead)
//!
//! We don't say this parser uses the Earley algorithm, because it's unnecessarily inaccurate.
//! The macro parser restricts itself to the features of finite state automata. Earley parsers
//! can be described as an extension of NFAs with completion rules, prediction rules, and recursion.

Two admissions right at the top: the big-O is worse than a textbook NFA in pathological cases, and the fix would require lazily constructing the results. Both are honest engineering notes.

How it works

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:16-40

//! Quick intro to how the parser works:
//!
//! A "matcher position" (a.k.a. "position" or "mp") is a dot in the middle of a matcher, usually
//! written as a `·`. For example `· a $( a )* a b` is one, as is `a $( · a )* a b`.
//!
//! The parser walks through the input a token at a time, maintaining a list
//! of threads consistent with the current position in the input string: `cur_mps`.
//!
//! As it processes them, it fills up `eof_mps` with threads that would be valid if
//! the macro invocation is now over, `bb_mps` with threads that are waiting on
//! a Rust non-terminal like `$e:expr`, and `next_mps` with threads that are waiting
//! on a particular token. Most of the logic concerns moving the · through the
//! repetitions indicated by Kleene stars. The rules for moving the · without
//! consuming any input are called epsilon transitions. It only advances or calls
//! out to the real Rust parser when no `cur_mps` threads remain.

Three sets of threads:

cur_mps — positions that are live at the current input token;
next_mps — positions waiting for a particular token;
bb_mps — positions blocked on a Rust non-terminal ($e:expr);
eof_mps — positions that would accept the input if it ended here.

And the worked example, which is worth reading in full because it makes the algorithm concrete:

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:41-70

//! Example:
//!
//! ```text, ignore
//! Start parsing a a a a b against [· a $( a )* a b].
//!
//! Remaining input: a a a a b
//! next: [· a $( a )* a b]
//!
//! - - - Advance over an a. - - -
//!
//! Remaining input: a a a b
//! cur: [a · $( a )* a b]
//! Descend/Skip (first position).
//! next: [a $( · a )* a b] [a $( a )* · a b].
//!
//! - - - Advance over an a. - - -
//!
//! Remaining input: a a b
//! cur: [a $( a · )* a b] [a $( a )* a · b]
//! Follow epsilon transition: Finish/Repeat (first position)
//! next: [a $( a )* · a b] [a $( · a )* a b] [a $( a )* a · b]
//!
//! - - - Advance over an a. - - - (this looks exactly like the last step)
//!
//! Remaining input: a b
//! cur: [a $( a · )* a b] [a $( a )* a · b]
//! Follow epsilon transition: Finish/Repeat (first position)
//! next: [a $( a )* · a b] [a $( · a )* a b] [a $( a )* a · b]
//!
//! - - - Advance over an a. - - -
//!
//! Remaining input: b
//! cur: [a $( a · )* a b] [a $( a )* a · b]
//! Follow epsilon transition: Finish/Repeat (first position)
//! next: [a $( a )* · a b] [a $( · a )* a b] [a $( a )* a · b]
//!
//! - - - Advance over a b. - - -
//!
//! Remaining input: ''
//! eof: [a $( a )* a b ·]
//! ```

Notice that after the first a, there are already two live positions (the “skip the repetition” and the “descend into the repetition” ones). The set of threads is what makes the matcher non-backtracking: both possibilities are explored simultaneously, and the one that fails simply dies.

The matcher is a flat array of locations

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:99-126

pub(crate) enum MatcherLoc {
 Token {
 token: Token,
 },
 Sequence {
 op: KleeneOp,
 num_metavar_decls: usize,
 idx_first_after: usize,
 next_metavar: usize,
 seq_depth: usize,
 },
 SequenceKleeneOpNoSep {
 op: KleeneOp,
 idx_first: usize,
 },
 SequenceSep {
 separator: Token,
 },
 SequenceKleeneOpAfterSep {
 idx_first: usize,
 },
 MetaVarDecl {
 span: Span,
 bind: Ident,
 kind: NonterminalKind,
 next_metavar: usize,
 seq_depth: usize,
 },
 Eof,
}

and the comment above it explains why this matters:

This means a matcher can be represented by &[MatcherLoc], and traversal mostly involves simply incrementing the current matcher position index by one.

A “dot” in a matcher is an index into a flat array. Advancing is idx += 1. There are no linked lists, no tree walks, no allocation per position. The KleeneOp variants encode the repetition structure with back-references (idx_first_after, idx_first) into the same array, so a repetition is a jump, not a recursive structure.

Threads are cheap, and the matches are shared

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:245-262

/// A single matcher position, representing the state of matching.
#[derive(Debug)]
struct MatcherPos {
 /// The index into `TtParser::locs`, which represents the "dot".
 idx: usize,

 /// The matches made against metavar decls so far. On a successful match, this vector ends up
 /// with one element per metavar decl in the matcher. ...
 ///
 /// It is critical to performance that this is an `Rc`, because it gets cloned frequently when
 /// processing sequences. Mostly for sequence-ending possibilities that must be tried but end
 /// up failing.
 matches: Rc<Vec<NamedMatch>>,
}

// This type is used a lot. Make sure it doesn't unintentionally get bigger.
#[cfg(target_pointer_width = "64")]
rustc_data_structures::static_assert_size!(MatcherPos, 16);

The matches vector is behind an Rc because when the matcher is inside a repetition, each possible “how many repetitions so far” is a separate thread, and they share the common prefix of matches. Rc::make_mut in push_match gives copy-on-write semantics: threads that diverge pay for the copy, threads that stay in sync share it.

And the size assertion: 16 bytes per thread. With a few dozen threads per input token and thousands of input tokens, that is the difference between a few kilobytes and a few megabytes.

Named non-terminals commit

The NFA covers the macro grammar. When it hits a Rust non-terminal like $e:expr, it hands off:

calls out to the main Rust parser for named non-terminals (which it commits to fully when it hits one in a grammar)

“Commits to fully” is the key word. The NFA cannot run a Rust expression parser and then backtrack into the middle of it, so once a non-terminal is encountered, the parse of that fragment must succeed. This is why some macro_rules patterns that “should” work do not — and it is a deliberate trade: committing gives a real Rust parser error message when the fragment does not parse, instead of a macro error.

The pathological case, and the note about it

The header comment admits:

The big-O, in pathological cases, is worse than traditional use of NFA or Earley parsing, but it’s an easier fit for Macro-by-Example-style rules.

The reason is the eager construction of NamedMatches: each thread carries the matches it has made so far, and those are built eagerly rather than lazily. A pathological rule can make the set of threads grow exponentially, each with a matches vector.

The fix would be to construct the matches only at the end, but then the matcher would have to keep every old position alive — more memory, more complexity. The comment says as much.

Where it lives

compiler/rustc_expand/src/mbe/macro_parser.rs — the whole thing, with an excellent module-level tutorial.
compiler/rustc_expand/src/mbe/mod.rs — the rule representation and the expansion driver.

Takeaway

The NFA approach to macro_rules! is a good example of a technique that solves the classic problem (backtracking blowup) by accepting a different, rarer problem (thread explosion). And the implementation documents its own trade-off in the first fourteen lines, which is more than most compiler code manages.

The Lexer That Never Fails: rustc's Tokenizer Returns Errors as Data

Jay Hubbert — Wed, 21 Oct 2026 11:35:00 +0000

TL;DR: rustc_lexer has no Result in its API. An unterminated string, an empty integer literal, a stray ∞ — all of them are token kinds. The lexer is a total function from bytes to a sequence of tokens, and the diagnostic layer decides afterwards which ones deserve a message. It also refuses to be greedy: 0..2 and 12.foo() tokenize correctly because the lexer looks at two characters instead of rewinding.

Tokens are just a kind and a length

 rust
 
 
// compiler/rustc_lexer/src/lib.rs:55-68

/// Parsed token.
/// It doesn't contain information about data that has been parsed,
/// only the type of the token and its size.
#[derive(Debug)]
pub struct Token {
 pub kind: TokenKind,
 pub len: u32,
}

No span. No symbol. The rustc_lexer crate does not depend on rustc_span at all — it has no idea what a Symbol is. The consumer (rustc_parse::lexer) advances a BytePos cursor by token.len and constructs spans itself:

 rust
 
 
// compiler/rustc_parse/src/lexer/mod.rs:161-171

fn next_token_from_cursor(&mut self) -> (Token, bool) {
 let mut preceded_by_whitespace = false;
 let mut swallow_next_invalid = 0;
 // Skip trivial (whitespace & comments) tokens
 loop {
 let str_before = self.cursor.as_str();
 let token = self.cursor.advance_token();
 let start = self.pos;
 self.pos = self.pos + BytePos(token.len);

This keeps rustc_lexer a pure crate, reusable by rustfmt, the pretty-printer and the format-string parser. The size is asserted from the consumer side, because rustc_lexer cannot depend on rustc_data_structures:

 rust
 
 
// compiler/rustc_parse/src/lexer/mod.rs:31-36

// This type is used a lot. Make sure it doesn't unintentionally get bigger.
//
// This assertion is in this crate, rather than in `rustc_lexer`, because that
// crate cannot depend on `rustc_data_structures`.
#[cfg(target_pointer_width = "64")]
rustc_data_structures::static_assert_size!(rustc_lexer::Token, 12);

Errors are token kinds

 rust
 
 
// compiler/rustc_lexer/src/lib.rs:78-93

/// A block comment, e.g. `/* block comment */`.
///
/// Block comments can be recursive, so a sequence like `/* /* */`
/// will not be considered terminated and will result in a parsing error.
BlockComment {
 doc_style: Option<DocStyle>,
 terminated: bool,
},

and:

 rust
 
 
// compiler/rustc_lexer/src/lib.rs:223-247

pub enum LiteralKind {
 /// `12_u8`, `0o100`, `0b120i99`, `1f32`.
 Int { base: Base, empty_int: bool },
 /// `12.34f32`, `1e3`, but not `1f32`.
 Float { base: Base, empty_exponent: bool },
 /// `'a'`, `'\\'`, `'''`, `';`
 Char { terminated: bool },
 ...
 /// `r"abc"`, `r#"abc"#`, `r####"ab"###"c"####`, `r#"a`. `None` indicates
 /// an invalid literal.
 RawStr { n_hashes: Option<u8> },
}

Every failure mode is a field of a token kind: terminated: false for an unterminated block comment, empty_int: true for 0b with no digits, n_hashes: None for a malformed raw string. The lexer never returns Err, never emits a diagnostic, and never aborts. It just keeps going and produces a token stream that contains the mistake.

The error message is then written by the layer above, which knows about spans, suggestions and whether the mistake is worth reporting.

The lexer refuses to be greedy

This is the part I like most. A maximal-munch lexer (the C tradition) would tokenize 0..2 as 0. followed by .2 and then have to rewind. rustc looks at two characters instead:

 rust
 
 
// compiler/rustc_lexer/src/lib.rs:953-981

match self.first() {
 // Don't be greedy if this is actually an
 // integer literal followed by field/method access or a range pattern
 // (`0..2` and `12.foo()`)
 '.' if self.second() != '.' && !is_id_start(self.second()) => {
 // might have stuff after the ., and if it does, it needs to start
 // with a number
 self.bump();
 let mut empty_exponent = false;
 if self.first().is_ascii_digit() {
 self.eat_decimal_digits();
 match self.first() {
 'e' | 'E' => {
 self.bump();
 empty_exponent = !self.eat_float_exponent();
 }
 _ => (),
 }
 }
 Float { base, empty_exponent }
 }
 'e' | 'E' => {
 self.bump();
 let empty_exponent = !self.eat_float_exponent();
 Float { base, empty_exponent }
 }
 _ => Int { base, empty_int: false },
}

The three-way decision on .:

.. → not a float: 0..2 is 0, .., 2;
. followed by an identifier start → not a float: 12.foo() is 12, ., foo;
otherwise → float, and the digits after the . are optional, so 1. is a float but 1.foo is not.

The identifier check uses unicode_ident::is_xid_start, so 1.é is also not a float.

The alternative — a greedy lexer plus rewind — means a token buffer, a save/restore of the cursor, and a class of bugs where the rewind is forgotten. Two characters of lookahead is cheaper and simpler.

The cursor: cloning `Chars` for lookahead

 rust
 
 
// compiler/rustc_lexer/src/lib.rs:461-485

/// Peeks the next symbol from the input stream without consuming it.
/// If requested position doesn't exist, `EOF_CHAR` is returned.
pub fn first(&self) -> char {
 // `.next()` optimizes better than `.nth(0)`
 self.chars.clone().next().unwrap_or(EOF_CHAR)
}

/// Peeks the second symbol from the input stream without consuming it.
pub(crate) fn second(&self) -> char {
 // `.next()` optimizes better than `.nth(1)`
 let mut iter = self.chars.clone();
 iter.next();
 iter.next().unwrap_or(EOF_CHAR)
}

std::str::Chars is Clone and cloning it is cheap (a slice pointer and a length). Cloning gives you a non-consuming lookahead. And the comments about .next() versus .nth(0) are the kind of micro-detail that only matters because someone measured it: nth has a generic loop with a Step bound, next is a simple branch.

Bulk scanning with memchr

 rust
 
 
// compiler/rustc_lexer/src/lib.rs:519-533

/// Eats symbols while predicate returns true or until the end of file is reached.
pub(crate) fn eat_while(&mut self, mut predicate: impl FnMut(char) -> bool) {
 // It was tried making optimized version of this for eg. line comments, but
 // LLVM can inline all of this and compile it down to fast iteration over bytes.
 while predicate(self.first()) && !self.is_eof() {
 self.bump();
 }
}

pub(crate) fn eat_until(&mut self, byte: u8) {
 self.chars = match memchr::memchr(byte, self.as_str().as_bytes()) {
 Some(index) => self.as_str()[index..].chars(),
 None => "".chars(),
 }
}

memchr is the SIMD-accelerated byte search from the memchr crate. It is used for line comments, raw strings and frontmatter fences — anywhere a long run of bytes has to be scanned.

Note the comment on eat_while: a hand-written specialized loop was tried and was not faster, because LLVM already compiles the generic predicate loop down to fast byte iteration. Sometimes the clever thing is the thing you deleted.

The frontmatter flag

rust

// compiler/rustc_lexer/src/lib.rs:541-567

Frontmatter (--- at the top of a file) is lexed as a single token that swallows the whole block. Whether frontmatter is even allowed is a flag on the cursor:

 rust
 
 
// compiler/rustc_lexer/src/lib.rs:684-692

if matches!(self.frontmatter_allowed, FrontmatterAllowed::Yes)
 && !matches!(token_kind, Whitespace)
{
 // stop allowing frontmatters after first non-whitespace token
 self.frontmatter_allowed = FrontmatterAllowed::No;
}

After the first real token, the frontmatter code is dead but still costs one predictable branch per token. The whole feature is a few branches and a flag — no separate scanner, no separate token type.

Where it lives

compiler/rustc_lexer/src/lib.rs — the token kinds, the cursor, the number lexer.
compiler/rustc_parse/src/lexer/mod.rs — the span-producing wrapper and the diagnostics.
compiler/rustc_parse/src/lexer/unicode_chars.rs — the confusables table.

Takeaway

Encoding errors in the data rather than in a Result is a design choice with wide consequences: the lexer becomes total, its error handling becomes policy, and recovery becomes trivial — you just keep going, because a mistake is a token like any other. It also makes the “how far did the bad thing extend?” question free: the token’s length is already known.

Rust on Opensource internals

A Tour of rustc's Internals: The Thirty-Eight Tricks That Make the Compiler Fast

1. Everything is a query, and nothing is cached

2. The borrow checker solves in one pass

3. The trait solver thinks in lattices

4. MIR optimisation is where the sharp edges are

5. The front end never fails, and never copies

6. Const evaluation is one interpreter with two personalities

7. The standard library is where the bytes are

8. And the data structures underneath everything

The themes

Where to start reading

Pruning Irrelevant Pattern Columns: How rustc's Match Checker Went From Exponential to Linear

The algorithm

The blowup

The implementation

Constructor splitting

Witness reconstruction

Where it lives

Takeaway

Tables That Pick Their Own Integer Width: How rustc Encodes Crate Metadata

The table header

Choosing the width

Byte-interleaving the pairs

Decoding straight out of the mmap

Lazy values and post-order encoding

Where it lives

Takeaway

How a Span Fits in Eight Bytes: rustc's Four-Format Packed Encoding

The four formats

The type

The bit-field helper

Why a macro and not an enum

Hygiene as interned mark chains

Why Ord is forbidden

Where it lives

Takeaway

Always Bump Downwards: The Arena Allocator rustc Uses for Everything

The arena

The allocation loop

Chunk growth

Two arenas, one macro

Why arenas at all?

Where it lives

Takeaway

Provenance in Eight Bytes: How rustc Packs a Pointer's Identity

The packed provenance

The sparse provenance map

Merging the fragments back

Hashing large allocations

Where it lives

Takeaway

The Init Mask Is a Single Boolean Until It Isn't: How rustc Tracks Uninitialised Memory

The representation

Reading it

Writing it, and the two-way transition

The bit-level search

The proof that it is correct

The run-length encoding for repeated copies

Where it lives

Takeaway

One Interpreter, Two Machines: How rustc's Const Eval Powers Miri Too

The trait

The shared compile-time instantiation

The lazy operand model

Projecting out of a ScalarPair is register shuffling

A micro-optimisation kept on purpose

Where it lives

Takeaway

Sixteen Bytes at a Time: How Rust Validates UTF-8

The fast path

The slow path

const_eval_select

Why it matters

Where it lives

Takeaway

One XOR and One CLZ: How Rust's Stable Sort Decides Which Runs to Merge

The theory

The implementation

The sqrt approximation

Why `Ord` is forbidden

`const_eval_select`

The `SpecExtend` chain

`into_raw_parts` and `from_raw_parts`

`drain` is destructive but reversible

The `IntervalSet` for seen attributes