Macros on Opensource internals

Capturing Tokens by Replaying Them: rustc's Lazy Token Capture

Jay Hubbert — Wed, 21 Oct 2026 11:50:00 +0000

TL;DR: rustc’s parser does not collect tokens into a buffer as it parses. It takes a snapshot of the token cursor and remembers the bump count; the captured range is a half-open interval of bump counts. The tokens are materialised by replaying the cursor, and only if something actually asks for them — which is rare.

The problem

Some AST nodes need their original tokens: anything that will be handed to a proc macro, a derive, or an attribute that reads source text. Most nodes do not.

Collecting tokens eagerly — appending to a Vec as the parser advances — would cost every node in the program. rustc instead does the opposite.

The snapshot

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:17-24

// When collecting tokens, this fully captures the start point. Usually its
// just after outer attributes, but occasionally it's before.
#[derive(Clone, Debug)]
pub(super) struct CollectPos {
 start_token: (Token, Spacing),
 cursor_snapshot: TokenCursor,
 start_pos: u32,
}

and the snapshot operation:

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:94-101

impl<'a> Parser<'a> {
 pub(super) fn collect_pos(&self) -> CollectPos {
 CollectPos {
 start_token: (self.token, self.token_spacing),
 cursor_snapshot: self.token_cursor.clone(),
 start_pos: self.num_bump_calls,
 }
 }

start_pos is a bump counter, not a byte offset or a token index. The captured range is [start_pos, end_pos) in bump-counter space.

Note that TokenCursor::clone() is cheap-ish: the cursor holds a Vec of enclosing delimiters and Rc-shared token trees. The token trees are shared, not copied.

The end position, with the ungluing correction

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:278-294

assert!(
 !(self.break_last_token > 0 && matches!(capture_trailing, Trailing::Yes)),
 "Cannot have break_last_token > 0 and have trailing token"
);
assert!(self.break_last_token <= 2, "cannot break token more than twice");

let end_pos = self.num_bump_calls
 + capture_trailing as u32
 // If we "broke" the last token (e.g. breaking a `>>` token once into `>` + `>`, or
 // breaking a `>>=` token twice into `>` + `>` + `=`), then extend the range of
 // captured tokens to include it, because the parser was not actually bumped past it.
 // (Even if we broke twice, it was still just one token originally, hence the `1`.)
 // When the `LazyAttrTokenStream` gets converted into an `AttrTokenStream`, we will
 // rebreak that final token once or twice.
 + if self.break_last_token == 0 { 0 } else { 1 };

let num_calls = end_pos - collect_pos.start_pos;

The correction: if the last token was unglued (see the previous article), the range must include it, because the parser was never actually bumped past it. Even two breaks of >>= count as one original token.

The lazy stream

 rust
 
 
// compiler/rustc_ast/src/tokenstream.rs:217-239

enum LazyAttrTokenStreamInner {
 // The token stream has already been produced.
 Direct(AttrTokenStream),

 // From a value of this type we can reconstruct the `TokenStream` seen by
 // the `f` callback passed to a call to `Parser::collect_tokens`, by
 // replaying the getting of the tokens. This saves us producing a
 // `TokenStream` if it is never needed, e.g. a captured `macro_rules!`
 // argument that is never passed to a proc macro. In practice, token stream
 // creation happens rarely compared to calls to `collect_tokens` (see some
 // statistics in #78736) so we are doing as little up-front work as
 // possible.
 //
 // This also makes `Parser` very cheap to clone, since there is no
 // intermediate collection buffer to clone.
 Pending {
 start_token: (Token, Spacing),
 cursor_snapshot: TokenCursor,
 num_calls: u32,
 break_last_token: u32,
 node_replacements: ThinVec<NodeReplacement>,
 },
}

The comment mentions the statistics: token stream creation happens rarely compared to calls to collect_tokens. That is the empirical justification for the whole design.

And it gives a second benefit: the Parser becomes cheap to clone, because there is no intermediate collection buffer. And the parser is cloned — for every macro rule attempt during macro_rules! matching.

The replay

 rust
 
 
// compiler/rustc_ast/src/tokenstream.rs:252-263

// The token produced by the final call to `{,inlined_}next_and_bump` was not
// actually consumed by the callback. The combination of chaining the
// initial token and using `take` produces the desired result - we
// produce an empty `TokenStream` if no calls were made, and omit the
// final token otherwise.
let mut cursor_snapshot = cursor_snapshot.clone();
let tokens = iter::once(FlatToken::Token(*start_token))
 .chain(iter::repeat_with(|| FlatToken::Token(cursor_snapshot.next_and_bump())))
 .take(*num_calls as usize);

if node_replacements.is_empty() {
 make_attr_token_stream(tokens, *break_last_token)
} else {

The replay is a lazy iterator: once(start_token).chain(repeat_with(next)).take(n). The take is what stops it. And make_attr_token_stream(tokens, break_last_token) re-splits the final glued token using the same table the parser used — so the tokens handed to a proc macro are byte-identical to the source.

Deciding whether to capture: twice

The decision is made in two places, and the second one is sharper.

The early check, before parsing:

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:163-181

let possible_capture_mode = self.capture_cfg;

// We must collect if anything could observe the collected tokens, i.e.
// if any of the following conditions hold.
// - We are force collecting tokens (because force collection requires
// tokens by definition).
let needs_collection = matches!(force_collect, ForceCollect::Yes)
 // - Any of our outer attributes require tokens.
 || needs_tokens(&attrs.attrs)
 // - Our target supports custom inner attributes (custom
 // inner attribute invocation might require token capturing).
 || R::SUPPORTS_CUSTOM_INNER_ATTRS
 // - We are in "possible capture mode" (which requires tokens if
 // the parsed node has `#[cfg]` or `#[cfg_attr]` attributes).
 || possible_capture_mode;
if !needs_collection {
 return Ok(f(self, attrs.attrs)?.0);
}

and the late check, after parsing, when inner attributes are known:

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:230-266

// When we're not in "definite capture mode", then skip collecting and
// return early if `ret` doesn't support tokens or already has some.
//
// Note that this check is independent of `force_collect`. There's no
// need to collect tokens when we don't support tokens or already have
// tokens.
let definite_capture_mode = self.capture_cfg
 && matches!(self.capture_state.capturing, Capturing::Yes)
 && has_cfg_or_cfg_attr(&ret_attrs);
if !definite_capture_mode && !ret_can_hold_tokens {
 return Ok(ret);
}

The comment in the source is candid about the residual waste:

If we return early here that means the setup, such as cloning the token cursor, was unnecessary. That’s hard to avoid.

What actually triggers capture

 rust
 
 
// compiler/rustc_parse/src/parser/attr_wrapper.rs:410-427

fn needs_tokens(attrs: &[ast::Attribute]) -> bool {
 // Tokens are needed if...
 attrs.iter().any(|attr| match &attr.kind {
 AttrKind::Normal(normal) => {
 match normal.item.name() {
 // ... a multi-segment attribute is present, e.g. `rustfmt::skip`.
 None => true,
 // ... `cfg_attr` or a single-segment non-builtin attribute is present, e.g.
 // `derive`, `test`, `global_allocator`.
 Some(name) => name == sym::cfg_attr || !rustc_feature::is_builtin_attr_name(name),
 }
 }
 // Synthetic attributes are created only during expansion, and can't re-enter the parser
 // because they have no token form.
 AttrKind::Synthetic(_) => unreachable!(),
 AttrKind::DocComment(..) => false,
 })
}

Builtin attributes (#[doc], #[inline], #[allow]) do not need tokens. #[derive], #[test], #[global_allocator], #[cfg_attr] and any path-qualified attribute do.

The `IntervalSet` for seen attributes

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:274-283

// This state is used by `Parser::collect_tokens`.
#[derive(Clone, Debug)]
struct CaptureState {
 capturing: Capturing,
 parser_replacements: Vec<ParserReplacement>,
 inner_attr_parser_ranges: FxHashMap<AttrId, ParserRange>,
 // `IntervalSet` is good for perf because attrs are mostly added to this
 // set in contiguous ranges.
 seen_attrs: IntervalSet<AttrId>,
}

Attribute IDs are assigned monotonically, so the “already seen” set is mostly a few contiguous ranges — the same interval-set trick the borrow checker uses for regions, applied to a completely different problem.

Where it lives

compiler/rustc_parse/src/parser/attr_wrapper.rs — CollectPos, collect_pos, the two capture decisions.
compiler/rustc_ast/src/tokenstream.rs — LazyAttrTokenStream, the replay, TokenCursor.
compiler/rustc_parse/src/parser/mod.rs — CaptureState, seen_attrs.

Takeaway

“Capture the tokens” sounds like it needs a buffer. It does not — it needs a recipe for reconstructing them. Recording where you were and how far you went is enough, and the recipe is cheaper to store, cheaper to clone, and (crucially) only executed if someone asks. That is the difference between paying O(n) for every AST node and paying O(1).

One Token, Two Operators: How rustc Splices a >> Back Apart

Jay Hubbert — Wed, 21 Oct 2026 11:45:00 +0000

TL;DR: >> is a single token in Rust’s grammar, but Vec<Vec<u8>> needs two > tokens. rustc’s parser splits the glued token on demand, adjusting the spans so the two halves point at the right bytes, and tracking that it did so so that a proc macro can be handed the original glued token back.

The problem

Rust has no closing-bracket ambiguity problem like C++’s >>, but it has the same tokenization problem: >> is a single shift operator, and Vec<Vec<u8>> ends with two of them.

The parser cannot just re-lex the source, because by the time it is parsing, the source may have come from a macro expansion, a proc macro, or an include — the token stream is the source of truth, not the text.

Ungluing on demand

The parser keeps a counter of how many pieces it has split off:

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:204-222

// During parsing we may sometimes need to "unglue" a glued token into two
// or three component tokens (e.g. `>>` into `>` and `>`, or `>>=` into `>`
// and `>` and `=`), so the parser can consume them one at a time. This
// process bypasses the normal capturing mechanism (e.g. `num_bump_calls`
// will not be incremented), since the "unglued" tokens due not exist in the
// original `TokenStream`.
//
// If we end up consuming all the component tokens, this is not an issue,
// because we'll end up capturing the single "glued" token.
//
// However, sometimes we may want to capture not all of the original
// token. For example, capturing the `Vec<u8>` in `Option<Vec<u8>>`
// requires us to unglue the trailing `>>` token. The `break_last_token`
// field is used to track these tokens. They get appended to the captured
// stream when we evaluate a `LazyAttrTokenStream`.
//
// This value is always 0, 1, or 2. It can only reach 2 when splitting
// `>>=` or `<<=`.
break_last_token: u32 = 0,

The key sentence: “This process bypasses the normal capturing mechanism (e.g. num_bump_calls will not be incremented), since the ‘unglued’ tokens do not exist in the original TokenStream.”

That is what makes the capture accounting work: the captured range is measured in original tokens, and splitting a token does not change the count.

The split, with span surgery

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:753-783

/// Eats the expected token if it's present possibly breaking
/// compound tokens like multi-character operators in process.
/// Returns `true` if the token was eaten.
fn break_and_eat(&mut self, exp: ExpTokenPair) -> bool {
 if self.token == exp.tok {
 self.bump();
 return true;
 }
 match self.token.kind.break_two_token_op(1) {
 Some((first, second)) if first == exp.tok => {
 let first_span = self.psess.source_map().start_point(self.token.span);
 let second_span = self.token.span.with_lo(first_span.hi());
 self.token = Token::new(first, first_span);
 // Keep track of this token - if we end token capturing now,
 // we'll want to append this token to the captured stream.
 //
 // If we consume any additional tokens, then this token
 // is not needed (we'll capture the entire 'glued' token),
 // and `bump` will set this field to 0.
 self.break_last_token += 1;
 // Use the spacing of the glued token as the spacing of the
 // unglued second token.
 self.bump_with((Token::new(second, second_span), self.token_spacing));
 true
 }
 _ => {
 self.expected_token_types.insert(exp.token_type);
 false
 }
 }
}

Four details:

first_span is the start point of the glued token — so the first half points at the first byte.
second_span is the glued token’s span with its low end moved to first_span.hi() — so the second half points at the second byte. The two halves are byte-exact.
break_last_token += 1 records that a split happened, so the capture machinery knows to re-glue it later.
The spacing of the glued token is reused for the second half. Spacing (Joint vs Alone) is what the token stream uses to decide whether >> should be re-glued into one token when handed to a macro; getting it wrong would change the meaning of the macro input.

The gluing table

 rust
 
 
// compiler/rustc_ast/src/token.rs:546-584

/// An approximation to proc-macro-style single-character operators used by
/// rustc parser. If the operator token can be broken into two tokens, the
/// first of which has `n` (1 or 2) chars, then this function performs that
/// operation, otherwise it returns `None`.
pub fn break_two_token_op(&self, n: u32) -> Option<(TokenKind, TokenKind)> {
 assert!(n == 1 || n == 2);
 Some(match (self, n) {
 (Le, 1) => (Lt, Eq),
 (EqEq, 1) => (Eq, Eq),
 (Ne, 1) => (Bang, Eq),
 (Ge, 1) => (Gt, Eq),
 (AndAnd, 1) => (And, And),
 (OrOr, 1) => (Or, Or),
 (Shl, 1) => (Lt, Lt),
 (Shr, 1) => (Gt, Gt),
 ...
 (ShlEq, 1) => (Lt, Le), // `<` + `<=`
 (ShlEq, 2) => (Shl, Eq), // `<<` + `=`
 (ShrEq, 1) => (Gt, Ge), // `>` + `>=`
 (ShrEq, 2) => (Shr, Eq), // `>>` + `=`
 (DotDot, 1) => (Dot, Dot),
 (DotDotDot, 1) => (Dot, DotDot), // `.` + `..`
 (DotDotDot, 2) => (DotDot, Dot), // `..` + `.`
 (DotDotEq, 2) => (DotDot, Eq),
 (PathSep, 1) => (Colon, Colon),
 (RArrow, 1) => (Minus, Gt),
 (LArrow, 1) => (Lt, Minus),
 (FatArrow, 1) => (Eq, Gt),
 _ => return None,
 })
}

The n parameter is the interesting part. >>= can be split as > + >= (n=1) or as >> + = (n=2). The parser asks for the split that matches what it is looking for.

This table is the exact inverse of the gluing performed when a token stream is built:

rust

// compiler/rustc_ast/src/tokenstream.rs:692-720

so the parser and the token stream always agree about what >> means.

The reset on bump

 rust
 
 
// compiler/rustc_parse/src/parser/mod.rs:1132-1151

/// Advance the parser by one token.
pub fn bump(&mut self) {
 // Note: destructuring here would give nicer code, but it was found in #96210 to be slower
 // than `.0`/`.1` access.
 let mut next = self.token_cursor.inlined_next_and_bump();
 self.num_bump_calls += 1;
 // We got a token from the underlying cursor and no longer need to
 // worry about an unglued token. See `break_and_eat` for more details.
 self.break_last_token = 0;
 if next.0.span.is_dummy() {
 // Tweak the location for better diagnostics, but keep syntactic context intact.
 let fallback_span = self.token.span;
 next.0.span = fallback_span.with_ctxt(next.0.span.ctxt());
 }
 ...

break_last_token = 0 on every real bump: “if we consumed any additional tokens, then this token is not needed (we’ll capture the entire ‘glued’ token)”. So the counter only survives if the split token is the last thing captured.

And the is_dummy() fixup: tokens synthesised by macro expansion have dummy spans; the parser substitutes the previous token’s span for better diagnostics — while keeping the hygiene context, which is the part that must not change.

Why byte-exactness matters

If the spans were not byte-exact, a proc macro receiving Option<Vec<u8>> would see two > tokens pointing at overlapping or wrong source ranges. Proc macros that re-emit their input (which is most of them) would then produce output with mangled spans, and the error messages downstream would point at nonsense.

This is why the split is done with with_lo(first_span.hi()) rather than with a fresh span: the split token must claim exactly the bytes it occupies.

Where it lives

compiler/rustc_parse/src/parser/mod.rs — break_and_eat, bump, break_last_token.
compiler/rustc_ast/src/token.rs — break_two_token_op.
compiler/rustc_ast/src/tokenstream.rs — the gluing side and TokenCursor.

Takeaway

Token-level hacks are usually where compilers accumulate their worst inconsistencies. rustc’s approach — keep the original token stream authoritative, split on demand with byte-exact spans, and record the split so it can be undone — keeps a single source of truth and makes the “give the proc macro back exactly what the user wrote” requirement fall out naturally.

macro_rules! Is an NFA: How Rust Matches Repetitions Without Backtracking

Jay Hubbert — Wed, 21 Oct 2026 11:40:00 +0000

TL;DR: macro_rules! matching in rustc is not a backtracking matcher. It is an NFA: for each input token, rustc maintains a set of live matcher positions, advances all of them, and follows epsilon transitions for repetitions. There is no rewind, no retry, and no exponential blowup from a rule like $($x:expr),*.

Matching macro_rules! is the part of macro expansion that everyone implements badly. The obvious implementation is recursive backtracking: try to match the rule, fail, back up, try the next rule. That is exponential in the worst case and makes the error messages terrible.

rustc took the other road.

The module documentation is the specification

The best documentation of this design is the header of the implementation file, which is a tutorial in itself:

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:1-14

//! This is an NFA-based parser, which calls out to the main Rust parser for named non-terminals
//! (which it commits to fully when it hits one in a grammar). There's a set of current NFA threads
//! and a set of next ones. Instead of NTs, we have a special case for Kleene star. The big-O, in
//! pathological cases, is worse than traditional use of NFA or Earley parsing, but it's an easier
//! fit for Macro-by-Example-style rules.
//!
//! (In order to prevent the pathological case, we'd need to lazily construct the resulting
//! `NamedMatch`es at the very end. It'd be a pain, and require more memory to keep around old
//! matcher positions, but it would also save overhead)
//!
//! We don't say this parser uses the Earley algorithm, because it's unnecessarily inaccurate.
//! The macro parser restricts itself to the features of finite state automata. Earley parsers
//! can be described as an extension of NFAs with completion rules, prediction rules, and recursion.

Two admissions right at the top: the big-O is worse than a textbook NFA in pathological cases, and the fix would require lazily constructing the results. Both are honest engineering notes.

How it works

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:16-40

//! Quick intro to how the parser works:
//!
//! A "matcher position" (a.k.a. "position" or "mp") is a dot in the middle of a matcher, usually
//! written as a `·`. For example `· a $( a )* a b` is one, as is `a $( · a )* a b`.
//!
//! The parser walks through the input a token at a time, maintaining a list
//! of threads consistent with the current position in the input string: `cur_mps`.
//!
//! As it processes them, it fills up `eof_mps` with threads that would be valid if
//! the macro invocation is now over, `bb_mps` with threads that are waiting on
//! a Rust non-terminal like `$e:expr`, and `next_mps` with threads that are waiting
//! on a particular token. Most of the logic concerns moving the · through the
//! repetitions indicated by Kleene stars. The rules for moving the · without
//! consuming any input are called epsilon transitions. It only advances or calls
//! out to the real Rust parser when no `cur_mps` threads remain.

Three sets of threads:

cur_mps — positions that are live at the current input token;
next_mps — positions waiting for a particular token;
bb_mps — positions blocked on a Rust non-terminal ($e:expr);
eof_mps — positions that would accept the input if it ended here.

And the worked example, which is worth reading in full because it makes the algorithm concrete:

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:41-70

//! Example:
//!
//! ```text, ignore
//! Start parsing a a a a b against [· a $( a )* a b].
//!
//! Remaining input: a a a a b
//! next: [· a $( a )* a b]
//!
//! - - - Advance over an a. - - -
//!
//! Remaining input: a a a b
//! cur: [a · $( a )* a b]
//! Descend/Skip (first position).
//! next: [a $( · a )* a b] [a $( a )* · a b].
//!
//! - - - Advance over an a. - - -
//!
//! Remaining input: a a b
//! cur: [a $( a · )* a b] [a $( a )* a · b]
//! Follow epsilon transition: Finish/Repeat (first position)
//! next: [a $( a )* · a b] [a $( · a )* a b] [a $( a )* a · b]
//!
//! - - - Advance over an a. - - - (this looks exactly like the last step)
//!
//! Remaining input: a b
//! cur: [a $( a · )* a b] [a $( a )* a · b]
//! Follow epsilon transition: Finish/Repeat (first position)
//! next: [a $( a )* · a b] [a $( · a )* a b] [a $( a )* a · b]
//!
//! - - - Advance over an a. - - -
//!
//! Remaining input: b
//! cur: [a $( a · )* a b] [a $( a )* a · b]
//! Follow epsilon transition: Finish/Repeat (first position)
//! next: [a $( a )* · a b] [a $( · a )* a b] [a $( a )* a · b]
//!
//! - - - Advance over a b. - - -
//!
//! Remaining input: ''
//! eof: [a $( a )* a b ·]
//! ```

Notice that after the first a, there are already two live positions (the “skip the repetition” and the “descend into the repetition” ones). The set of threads is what makes the matcher non-backtracking: both possibilities are explored simultaneously, and the one that fails simply dies.

The matcher is a flat array of locations

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:99-126

pub(crate) enum MatcherLoc {
 Token {
 token: Token,
 },
 Sequence {
 op: KleeneOp,
 num_metavar_decls: usize,
 idx_first_after: usize,
 next_metavar: usize,
 seq_depth: usize,
 },
 SequenceKleeneOpNoSep {
 op: KleeneOp,
 idx_first: usize,
 },
 SequenceSep {
 separator: Token,
 },
 SequenceKleeneOpAfterSep {
 idx_first: usize,
 },
 MetaVarDecl {
 span: Span,
 bind: Ident,
 kind: NonterminalKind,
 next_metavar: usize,
 seq_depth: usize,
 },
 Eof,
}

and the comment above it explains why this matters:

This means a matcher can be represented by &[MatcherLoc], and traversal mostly involves simply incrementing the current matcher position index by one.

A “dot” in a matcher is an index into a flat array. Advancing is idx += 1. There are no linked lists, no tree walks, no allocation per position. The KleeneOp variants encode the repetition structure with back-references (idx_first_after, idx_first) into the same array, so a repetition is a jump, not a recursive structure.

Threads are cheap, and the matches are shared

 rust
 
 
// compiler/rustc_expand/src/mbe/macro_parser.rs:245-262

/// A single matcher position, representing the state of matching.
#[derive(Debug)]
struct MatcherPos {
 /// The index into `TtParser::locs`, which represents the "dot".
 idx: usize,

 /// The matches made against metavar decls so far. On a successful match, this vector ends up
 /// with one element per metavar decl in the matcher. ...
 ///
 /// It is critical to performance that this is an `Rc`, because it gets cloned frequently when
 /// processing sequences. Mostly for sequence-ending possibilities that must be tried but end
 /// up failing.
 matches: Rc<Vec<NamedMatch>>,
}

// This type is used a lot. Make sure it doesn't unintentionally get bigger.
#[cfg(target_pointer_width = "64")]
rustc_data_structures::static_assert_size!(MatcherPos, 16);

The matches vector is behind an Rc because when the matcher is inside a repetition, each possible “how many repetitions so far” is a separate thread, and they share the common prefix of matches. Rc::make_mut in push_match gives copy-on-write semantics: threads that diverge pay for the copy, threads that stay in sync share it.

And the size assertion: 16 bytes per thread. With a few dozen threads per input token and thousands of input tokens, that is the difference between a few kilobytes and a few megabytes.

Named non-terminals commit

The NFA covers the macro grammar. When it hits a Rust non-terminal like $e:expr, it hands off:

calls out to the main Rust parser for named non-terminals (which it commits to fully when it hits one in a grammar)

“Commits to fully” is the key word. The NFA cannot run a Rust expression parser and then backtrack into the middle of it, so once a non-terminal is encountered, the parse of that fragment must succeed. This is why some macro_rules patterns that “should” work do not — and it is a deliberate trade: committing gives a real Rust parser error message when the fragment does not parse, instead of a macro error.

The pathological case, and the note about it

The header comment admits:

The big-O, in pathological cases, is worse than traditional use of NFA or Earley parsing, but it’s an easier fit for Macro-by-Example-style rules.

The reason is the eager construction of NamedMatches: each thread carries the matches it has made so far, and those are built eagerly rather than lazily. A pathological rule can make the set of threads grow exponentially, each with a matches vector.

The fix would be to construct the matches only at the end, but then the matcher would have to keep every old position alive — more memory, more complexity. The comment says as much.

Where it lives

compiler/rustc_expand/src/mbe/macro_parser.rs — the whole thing, with an excellent module-level tutorial.
compiler/rustc_expand/src/mbe/mod.rs — the rule representation and the expansion driver.

Takeaway

The NFA approach to macro_rules! is a good example of a technique that solves the classic problem (backtracking blowup) by accepting a different, rarer problem (thread explosion). And the implementation documents its own trade-off in the first fourteen lines, which is more than most compiler code manages.

Macros on Opensource internals

Capturing Tokens by Replaying Them: rustc's Lazy Token Capture

The problem

The snapshot

The end position, with the ungluing correction

The lazy stream

The replay

Deciding whether to capture: twice

What actually triggers capture

The IntervalSet for seen attributes

Where it lives

Takeaway

One Token, Two Operators: How rustc Splices a >> Back Apart

The problem

Ungluing on demand

The split, with span surgery

The gluing table

The reset on bump

Why byte-exactness matters

Where it lives

Takeaway

macro_rules! Is an NFA: How Rust Matches Repetitions Without Backtracking

The module documentation is the specification

How it works

The matcher is a flat array of locations

Threads are cheap, and the matches are shared

Named non-terminals commit

The pathological case, and the note about it

Where it lives

Takeaway

The `IntervalSet` for seen attributes