Did You Mean a Hyphen?: How rustc Repairs Unicode Confusables While Parsing
rustc repairs unicode confusables by looking them up in a static table, substituting the ASCII token and continuing to parse, so recovery costs nothing extra.
Rust
Capturing Tokens by Replaying Them: rustc's Lazy Token Capture
rustc’s parser captures macro tokens by recording a cursor snapshot plus a bump count and replaying them on demand, so tokens are materialised only for nodes that actually need …
One Token, Two Operators: How rustc Splices a >> Back Apart
rustc’s parser unglues compound tokens like » and »= on demand, with exact span surgery so that a proc macro sees the same bytes the source had.
macro_rules! Is an NFA: How Rust Matches Repetitions Without Backtracking
The macro_rules! matcher in rustc is an NFA-based parser: a set of live matcher positions advances one input token at a time, with epsilon transitions for repetitions and no …
The Lexer That Never Fails: rustc's Tokenizer Returns Errors as Data
rustc’s lexer never returns an error - failure modes are encoded in the token kind, and the lexer also refuses to be greedy about numbers so 0..2 and 12.foo() tokenize correctly.