Did You Mean a Hyphen?: How rustc Repairs Unicode Confusables While Parsing

TL;DR: If you paste an en-dash into Rust source, rustc does not just complain — it looks the character up in a static confusables table, substitutes the real ASCII token (-), keeps parsing, and reports the problem as a side effect. The diagnostic is generated from the same table that produced the recovery.

Copy-pasting from a word processor is a common way to introduce Unicode characters that look like ASCII ones. The classic offenders:

– (en dash) instead of -;
‘ / ’ (curly quotes) instead of ' and ";
non-breaking spaces;
full-width characters from CJK input methods.

rustc’s response is unusually good: it does not just point at the character, it fixes the parse and tells you what it did.

The table

rust
// compiler/rustc_parse/src/lexer/unicode_chars.rs:1-12

//! Characters and their corresponding confusables were collected from
//! <https://www.unicode.org/Public/security/10.0.0/confusables.txt>

use rustc_span::{BytePos, Pos, Span, kw};

use super::Lexer;
use crate::diagnostics::TokenSubstitution;
use crate::token;

#[rustfmt::skip] // for line breaks
pub(super) static UNICODE_ARRAY: &[(char, &str, &str)] = &[
    (' ', "Line Separator", " "),
    (' ', "Paragraph Separator", " "),

Each entry is (unicode char, unicode name, ASCII replacement). The relevant rows:

rust
// compiler/rustc_parse/src/lexer/unicode_chars.rs:37-59, 131-166, 168-184

    ('‐', "Hyphen", "-"),
    ('‑', "Non-Breaking Hyphen", "-"),
    ('‒', "Figure Dash", "-"),
    ('–', "En Dash", "-"),
    ('—', "Em Dash", "-"),
    ...
    ('‘', "Left Single Quotation Mark", "'"),
    ('’', "Right Single Quotation Mark", "'"),
    ...
    ('“', "Left Double Quotation Mark", "\""),
    ('”', "Right Double Quotation Mark", "\""),

The table is generated from Unicode’s own confusables data file, so it covers thousands of characters.

The lookup returns a token

rust
// compiler/rustc_parse/src/lexer/unicode_chars.rs:339-380

pub(super) fn check_for_substitution(
    lexer: &Lexer<'_, '_>,
    pos: BytePos,
    ch: char,
    count: usize,
) -> (Option<token::TokenKind>, Option<TokenSubstitution>) {
    let Some(&(_, u_name, ascii_str)) = UNICODE_ARRAY.iter().find(|&&(c, _, _)| c == ch) else {
        return (None, None);
    };

    let span = Span::with_root_ctxt(pos, pos + Pos::from_usize(ch.len_utf8() * count));

    let Some((_, ascii_name, token)) = ASCII_ARRAY.iter().find(|&&(s, _, _)| s == ascii_str) else {
        let msg = format!("substitution character not found for '{ch}'");
        lexer.dcx().span_bug(span, msg);
    };

    // special help suggestion for "directed" double quotes
    let sugg = if let Some(s) = peek_delimited(&lexer.src[lexer.src_index(pos)..], '“', '”') {
        let span = Span::with_root_ctxt(
            pos,
            pos + Pos::from_usize('“'.len_utf8() + s.len() + '”'.len_utf8()),
        );
        Some(TokenSubstitution::DirectedQuotes {
            span,
            suggestion: format!("\"{s}\""),
            ascii_str,
            ascii_name,
        })
    } else {
        let suggestion = ascii_str.to_string().repeat(count);
        Some(TokenSubstitution::Other {
            span,
            suggestion,
            ch: ch.to_string(),
            u_name,
            ascii_str,
            ascii_name,
        })
    };
    (*token, sugg)
}

Three things to notice:

The return value is a real TokenKind. The ASCII_ARRAY maps the ASCII replacement back to a token, so an en-dash becomes token::Minus. The parser continues with a valid token stream.
The diagnostic is a by-product. The same function that performs the substitution also builds the suggestion. No second pass, no second table.
The “directed quotes” special case. If the input has “hello”, the suggestion replaces the whole quoted run with "hello", which is a much better fix than replacing each smart quote separately:

rust
// compiler/rustc_parse/src/lexer/unicode_chars.rs:383-391

/// Extract string if found at current position with given delimiters
fn peek_delimited(text: &str, from_ch: char, to_ch: char) -> Option<&str> {
    let mut chars = text.chars();
    let first_char = chars.next()?;
    if first_char != from_ch {
        return None;
    }
    let last_char_idx = chars.as_str().find(to_ch)?;
    Some(&chars.as_str()[..last_char_idx])
}

Where it is invoked, and the “swallow the repeats” trick

rust
// compiler/rustc_parse/src/lexer/mod.rs:486-514

let repeats = it.take_while(|c1| *c1 == c).count();
// FIXME: the lexer could be used to turn the ASCII version of unicode
// homoglyphs, instead of keeping a table in `check_for_substitution`into the
// token. Ideally, this should be inside `rustc_lexer`. However, we should
// first remove compound tokens like `<<` from `rustc_lexer`, and then add
// fancier error recovery to it, as there will be less overall work to do this
// way.
let (token, sugg) =
    unicode_chars::check_for_substitution(self, start, c, repeats + 1);
self.dcx().emit_err(crate::diagnostics::UnknownTokenStart {
    span: self.mk_sp(start, self.pos + Pos::from_usize(repeats * c.len_utf8())),
    escaped: escaped_char(c),
    sugg,
    null: c == '\x00',
    invisible: INVISIBLE_CHARACTERS.contains(&c),
    repeat: if repeats > 0 {
        swallow_next_invalid = repeats;
        Some(crate::diagnostics::UnknownTokenRepeat { repeats })
    } else {
        None
    },
});

The swallow_next_invalid mechanism suppresses the next N identical invalid characters, so @@@@@@ produces one error (“character appears 5 more times”) instead of six. The sub-diagnostic renders it:

rust
// compiler/rustc_parse/src/diagnostics.rs:2890-2899

#[derive(Subdiagnostic)]
#[note(
    "character appears {$repeats ->
        [one] once more
        *[other] {$repeats} more times
    }"
)]
pub(crate) struct UnknownTokenRepeat {
    pub repeats: usize,
}

Note also the FIXME: the authors think this repair machinery should live in rustc_lexer itself, once the lexer has compound tokens removed and better recovery. It is a known piece of technical debt, documented as such.

The suggestion is deliberately not auto-applied

rust
// compiler/rustc_parse/src/diagnostics.rs:2858-2888

#[derive(Subdiagnostic)]
pub(crate) enum TokenSubstitution {
    #[suggestion(
        "Unicode characters '“' (Left Double Quotation Mark) and '”' (Right Double Quotation Mark) look like '{$ascii_str}' ({$ascii_name}), but are not",
        code = "{suggestion}",
        applicability = "maybe-incorrect",
        style = "verbose"
    )]
    DirectedQuotes { ... },

applicability = "maybe-incorrect" means rustfix will offer the fix but will not apply it automatically. For a confusable replacement, that is the right call: the character might be intentional.

The other unicode trick: fast-path NFC normalisation

rust
// compiler/rustc_parse/src/lexer/mod.rs:1278-1287

pub fn nfc_normalize(string: &str) -> Symbol {
    use unicode_normalization::{IsNormalized, UnicodeNormalization, is_nfc_quick};
    match is_nfc_quick(string.chars()) {
        IsNormalized::Yes => Symbol::intern(string),
        _ => {
            let normalized_str: String = string.chars().nfc().collect();
            Symbol::intern(&normalized_str)
        }
    }
}

Every identifier goes through this. is_nfc_quick is a cheap O(n) check that answers “is this already normalised?”; the expensive full normalisation only runs for identifiers that actually need it. Identifiers are normalised so that é written as one codepoint and as e + combining acute are the same identifier.

Where it lives

compiler/rustc_parse/src/lexer/unicode_chars.rs — the tables and check_for_substitution.
compiler/rustc_parse/src/lexer/mod.rs — the invocation, the repeat suppression, nfc_normalize.
compiler/rustc_parse/src/diagnostics.rs — TokenSubstitution.

Takeaway

Error recovery is a design problem, not a diagnostics problem. The best recovery is the kind that lets the compiler keep going and produces a better message than a plain error would. rustc’s confusable handling does both from a single table: the same lookup that produces the token produces the suggestion, so the recovery is essentially free.