One Token, Two Operators: How rustc Splices a >> Back Apart

TL;DR: >> is a single token in Rust’s grammar, but Vec<Vec<u8>> needs two > tokens. rustc’s parser splits the glued token on demand, adjusting the spans so the two halves point at the right bytes, and tracking that it did so so that a proc macro can be handed the original glued token back.

The problem

Rust has no closing-bracket ambiguity problem like C++’s >>, but it has the same tokenization problem: >> is a single shift operator, and Vec<Vec<u8>> ends with two of them.

The parser cannot just re-lex the source, because by the time it is parsing, the source may have come from a macro expansion, a proc macro, or an include — the token stream is the source of truth, not the text.

Ungluing on demand

The parser keeps a counter of how many pieces it has split off:

rust
// compiler/rustc_parse/src/parser/mod.rs:204-222

// During parsing we may sometimes need to "unglue" a glued token into two
// or three component tokens (e.g. `>>` into `>` and `>`, or `>>=` into `>`
// and `>` and `=`), so the parser can consume them one at a time. This
// process bypasses the normal capturing mechanism (e.g. `num_bump_calls`
// will not be incremented), since the "unglued" tokens due not exist in the
// original `TokenStream`.
//
// If we end up consuming all the component tokens, this is not an issue,
// because we'll end up capturing the single "glued" token.
//
// However, sometimes we may want to capture not all of the original
// token. For example, capturing the `Vec<u8>` in `Option<Vec<u8>>`
// requires us to unglue the trailing `>>` token. The `break_last_token`
// field is used to track these tokens. They get appended to the captured
// stream when we evaluate a `LazyAttrTokenStream`.
//
// This value is always 0, 1, or 2. It can only reach 2 when splitting
// `>>=` or `<<=`.
break_last_token: u32 = 0,

The key sentence: “This process bypasses the normal capturing mechanism (e.g. num_bump_calls will not be incremented), since the ‘unglued’ tokens do not exist in the original TokenStream.”

That is what makes the capture accounting work: the captured range is measured in original tokens, and splitting a token does not change the count.

The split, with span surgery

rust
// compiler/rustc_parse/src/parser/mod.rs:753-783

/// Eats the expected token if it's present possibly breaking
/// compound tokens like multi-character operators in process.
/// Returns `true` if the token was eaten.
fn break_and_eat(&mut self, exp: ExpTokenPair) -> bool {
    if self.token == exp.tok {
        self.bump();
        return true;
    }
    match self.token.kind.break_two_token_op(1) {
        Some((first, second)) if first == exp.tok => {
            let first_span = self.psess.source_map().start_point(self.token.span);
            let second_span = self.token.span.with_lo(first_span.hi());
            self.token = Token::new(first, first_span);
            // Keep track of this token - if we end token capturing now,
            // we'll want to append this token to the captured stream.
            //
            // If we consume any additional tokens, then this token
            // is not needed (we'll capture the entire 'glued' token),
            // and `bump` will set this field to 0.
            self.break_last_token += 1;
            // Use the spacing of the glued token as the spacing of the
            // unglued second token.
            self.bump_with((Token::new(second, second_span), self.token_spacing));
            true
        }
        _ => {
            self.expected_token_types.insert(exp.token_type);
            false
        }
    }
}

Four details:

first_span is the start point of the glued token — so the first half points at the first byte.
second_span is the glued token’s span with its low end moved to first_span.hi() — so the second half points at the second byte. The two halves are byte-exact.
break_last_token += 1 records that a split happened, so the capture machinery knows to re-glue it later.
The spacing of the glued token is reused for the second half. Spacing (Joint vs Alone) is what the token stream uses to decide whether >> should be re-glued into one token when handed to a macro; getting it wrong would change the meaning of the macro input.

The gluing table

rust
// compiler/rustc_ast/src/token.rs:546-584

/// An approximation to proc-macro-style single-character operators used by
/// rustc parser. If the operator token can be broken into two tokens, the
/// first of which has `n` (1 or 2) chars, then this function performs that
/// operation, otherwise it returns `None`.
pub fn break_two_token_op(&self, n: u32) -> Option<(TokenKind, TokenKind)> {
    assert!(n == 1 || n == 2);
    Some(match (self, n) {
        (Le, 1) => (Lt, Eq),
        (EqEq, 1) => (Eq, Eq),
        (Ne, 1) => (Bang, Eq),
        (Ge, 1) => (Gt, Eq),
        (AndAnd, 1) => (And, And),
        (OrOr, 1) => (Or, Or),
        (Shl, 1) => (Lt, Lt),
        (Shr, 1) => (Gt, Gt),
        ...
        (ShlEq, 1) => (Lt, Le),  // `<` + `<=`
        (ShlEq, 2) => (Shl, Eq), // `<<` + `=`
        (ShrEq, 1) => (Gt, Ge),  // `>` + `>=`
        (ShrEq, 2) => (Shr, Eq), // `>>` + `=`
        (DotDot, 1) => (Dot, Dot),
        (DotDotDot, 1) => (Dot, DotDot), // `.` + `..`
        (DotDotDot, 2) => (DotDot, Dot), // `..` + `.`
        (DotDotEq, 2) => (DotDot, Eq),
        (PathSep, 1) => (Colon, Colon),
        (RArrow, 1) => (Minus, Gt),
        (LArrow, 1) => (Lt, Minus),
        (FatArrow, 1) => (Eq, Gt),
        _ => return None,
    })
}

The n parameter is the interesting part. >>= can be split as > + >= (n=1) or as >> + = (n=2). The parser asks for the split that matches what it is looking for.

This table is the exact inverse of the gluing performed when a token stream is built:

rust

// compiler/rustc_ast/src/tokenstream.rs:692-720

so the parser and the token stream always agree about what >> means.

The reset on bump

rust
// compiler/rustc_parse/src/parser/mod.rs:1132-1151

/// Advance the parser by one token.
pub fn bump(&mut self) {
    // Note: destructuring here would give nicer code, but it was found in #96210 to be slower
    // than `.0`/`.1` access.
    let mut next = self.token_cursor.inlined_next_and_bump();
    self.num_bump_calls += 1;
    // We got a token from the underlying cursor and no longer need to
    // worry about an unglued token. See `break_and_eat` for more details.
    self.break_last_token = 0;
    if next.0.span.is_dummy() {
        // Tweak the location for better diagnostics, but keep syntactic context intact.
        let fallback_span = self.token.span;
        next.0.span = fallback_span.with_ctxt(next.0.span.ctxt());
    }
    ...

break_last_token = 0 on every real bump: “if we consumed any additional tokens, then this token is not needed (we’ll capture the entire ‘glued’ token)”. So the counter only survives if the split token is the last thing captured.

And the is_dummy() fixup: tokens synthesised by macro expansion have dummy spans; the parser substitutes the previous token’s span for better diagnostics — while keeping the hygiene context, which is the part that must not change.

Why byte-exactness matters

If the spans were not byte-exact, a proc macro receiving Option<Vec<u8>> would see two > tokens pointing at overlapping or wrong source ranges. Proc macros that re-emit their input (which is most of them) would then produce output with mangled spans, and the error messages downstream would point at nonsense.

This is why the split is done with with_lo(first_span.hi()) rather than with a fresh span: the split token must claim exactly the bytes it occupies.

Where it lives

compiler/rustc_parse/src/parser/mod.rs — break_and_eat, bump, break_last_token.
compiler/rustc_ast/src/token.rs — break_two_token_op.
compiler/rustc_ast/src/tokenstream.rs — the gluing side and TokenCursor.

Takeaway

Token-level hacks are usually where compilers accumulate their worst inconsistencies. rustc’s approach — keep the original token stream authoritative, split on demand with byte-exact spans, and record the split so it can be undone — keeps a single source of truth and makes the “give the proc macro back exactly what the user wrote” requirement fall out naturally.