The Lexer That Never Fails: rustc's Tokenizer Returns Errors as Data

TL;DR: rustc_lexer has no Result in its API. An unterminated string, an empty integer literal, a stray ∞ — all of them are token kinds. The lexer is a total function from bytes to a sequence of tokens, and the diagnostic layer decides afterwards which ones deserve a message. It also refuses to be greedy: 0..2 and 12.foo() tokenize correctly because the lexer looks at two characters instead of rewinding.

Tokens are just a kind and a length

rust
// compiler/rustc_lexer/src/lib.rs:55-68

/// Parsed token.
/// It doesn't contain information about data that has been parsed,
/// only the type of the token and its size.
#[derive(Debug)]
pub struct Token {
    pub kind: TokenKind,
    pub len: u32,
}

No span. No symbol. The rustc_lexer crate does not depend on rustc_span at all — it has no idea what a Symbol is. The consumer (rustc_parse::lexer) advances a BytePos cursor by token.len and constructs spans itself:

rust
// compiler/rustc_parse/src/lexer/mod.rs:161-171

fn next_token_from_cursor(&mut self) -> (Token, bool) {
    let mut preceded_by_whitespace = false;
    let mut swallow_next_invalid = 0;
    // Skip trivial (whitespace & comments) tokens
    loop {
        let str_before = self.cursor.as_str();
        let token = self.cursor.advance_token();
        let start = self.pos;
        self.pos = self.pos + BytePos(token.len);

This keeps rustc_lexer a pure crate, reusable by rustfmt, the pretty-printer and the format-string parser. The size is asserted from the consumer side, because rustc_lexer cannot depend on rustc_data_structures:

rust
// compiler/rustc_parse/src/lexer/mod.rs:31-36

// This type is used a lot. Make sure it doesn't unintentionally get bigger.
//
// This assertion is in this crate, rather than in `rustc_lexer`, because that
// crate cannot depend on `rustc_data_structures`.
#[cfg(target_pointer_width = "64")]
rustc_data_structures::static_assert_size!(rustc_lexer::Token, 12);

Errors are token kinds

rust
// compiler/rustc_lexer/src/lib.rs:78-93

/// A block comment, e.g. `/* block comment */`.
///
/// Block comments can be recursive, so a sequence like `/* /* */`
/// will not be considered terminated and will result in a parsing error.
BlockComment {
    doc_style: Option<DocStyle>,
    terminated: bool,
},

and:

rust
// compiler/rustc_lexer/src/lib.rs:223-247

pub enum LiteralKind {
    /// `12_u8`, `0o100`, `0b120i99`, `1f32`.
    Int { base: Base, empty_int: bool },
    /// `12.34f32`, `1e3`, but not `1f32`.
    Float { base: Base, empty_exponent: bool },
    /// `'a'`, `'\\'`, `'''`, `';`
    Char { terminated: bool },
    ...
    /// `r"abc"`, `r#"abc"#`, `r####"ab"###"c"####`, `r#"a`. `None` indicates
    /// an invalid literal.
    RawStr { n_hashes: Option<u8> },
}

Every failure mode is a field of a token kind: terminated: false for an unterminated block comment, empty_int: true for 0b with no digits, n_hashes: None for a malformed raw string. The lexer never returns Err, never emits a diagnostic, and never aborts. It just keeps going and produces a token stream that contains the mistake.

The error message is then written by the layer above, which knows about spans, suggestions and whether the mistake is worth reporting.

The lexer refuses to be greedy

This is the part I like most. A maximal-munch lexer (the C tradition) would tokenize 0..2 as 0. followed by .2 and then have to rewind. rustc looks at two characters instead:

rust
// compiler/rustc_lexer/src/lib.rs:953-981

match self.first() {
    // Don't be greedy if this is actually an
    // integer literal followed by field/method access or a range pattern
    // (`0..2` and `12.foo()`)
    '.' if self.second() != '.' && !is_id_start(self.second()) => {
        // might have stuff after the ., and if it does, it needs to start
        // with a number
        self.bump();
        let mut empty_exponent = false;
        if self.first().is_ascii_digit() {
            self.eat_decimal_digits();
            match self.first() {
                'e' | 'E' => {
                    self.bump();
                    empty_exponent = !self.eat_float_exponent();
                }
                _ => (),
            }
        }
        Float { base, empty_exponent }
    }
    'e' | 'E' => {
        self.bump();
        let empty_exponent = !self.eat_float_exponent();
        Float { base, empty_exponent }
    }
    _ => Int { base, empty_int: false },
}

The three-way decision on .:

.. → not a float: 0..2 is 0, .., 2;
. followed by an identifier start → not a float: 12.foo() is 12, ., foo;
otherwise → float, and the digits after the . are optional, so 1. is a float but 1.foo is not.

The identifier check uses unicode_ident::is_xid_start, so 1.é is also not a float.

The alternative — a greedy lexer plus rewind — means a token buffer, a save/restore of the cursor, and a class of bugs where the rewind is forgotten. Two characters of lookahead is cheaper and simpler.

The cursor: cloning `Chars` for lookahead

rust
// compiler/rustc_lexer/src/lib.rs:461-485

/// Peeks the next symbol from the input stream without consuming it.
/// If requested position doesn't exist, `EOF_CHAR` is returned.
pub fn first(&self) -> char {
    // `.next()` optimizes better than `.nth(0)`
    self.chars.clone().next().unwrap_or(EOF_CHAR)
}

/// Peeks the second symbol from the input stream without consuming it.
pub(crate) fn second(&self) -> char {
    // `.next()` optimizes better than `.nth(1)`
    let mut iter = self.chars.clone();
    iter.next();
    iter.next().unwrap_or(EOF_CHAR)
}

std::str::Chars is Clone and cloning it is cheap (a slice pointer and a length). Cloning gives you a non-consuming lookahead. And the comments about .next() versus .nth(0) are the kind of micro-detail that only matters because someone measured it: nth has a generic loop with a Step bound, next is a simple branch.

Bulk scanning with memchr

rust
// compiler/rustc_lexer/src/lib.rs:519-533

/// Eats symbols while predicate returns true or until the end of file is reached.
pub(crate) fn eat_while(&mut self, mut predicate: impl FnMut(char) -> bool) {
    // It was tried making optimized version of this for eg. line comments, but
    // LLVM can inline all of this and compile it down to fast iteration over bytes.
    while predicate(self.first()) && !self.is_eof() {
        self.bump();
    }
}

pub(crate) fn eat_until(&mut self, byte: u8) {
    self.chars = match memchr::memchr(byte, self.as_str().as_bytes()) {
        Some(index) => self.as_str()[index..].chars(),
        None => "".chars(),
    }
}

memchr is the SIMD-accelerated byte search from the memchr crate. It is used for line comments, raw strings and frontmatter fences — anywhere a long run of bytes has to be scanned.

Note the comment on eat_while: a hand-written specialized loop was tried and was not faster, because LLVM already compiles the generic predicate loop down to fast byte iteration. Sometimes the clever thing is the thing you deleted.

The frontmatter flag

rust

// compiler/rustc_lexer/src/lib.rs:541-567

Frontmatter (--- at the top of a file) is lexed as a single token that swallows the whole block. Whether frontmatter is even allowed is a flag on the cursor:

rust
// compiler/rustc_lexer/src/lib.rs:684-692

if matches!(self.frontmatter_allowed, FrontmatterAllowed::Yes)
    && !matches!(token_kind, Whitespace)
{
    // stop allowing frontmatters after first non-whitespace token
    self.frontmatter_allowed = FrontmatterAllowed::No;
}

After the first real token, the frontmatter code is dead but still costs one predictable branch per token. The whole feature is a few branches and a flag — no separate scanner, no separate token type.

Where it lives

compiler/rustc_lexer/src/lib.rs — the token kinds, the cursor, the number lexer.
compiler/rustc_parse/src/lexer/mod.rs — the span-producing wrapper and the diagnostics.
compiler/rustc_parse/src/lexer/unicode_chars.rs — the confusables table.

Takeaway

Encoding errors in the data rather than in a Result is a design choice with wide consequences: the lexer becomes total, its error handling becomes policy, and recovery becomes trivial — you just keep going, because a mistake is a token like any other. It also makes the “how far did the bad thing extend?” question free: the token’s length is already known.