Diagnostics on Opensource internals

Did You Mean a Hyphen?: How rustc Repairs Unicode Confusables While Parsing

Jay Hubbert — Wed, 21 Oct 2026 11:55:00 +0000

TL;DR: If you paste an en-dash into Rust source, rustc does not just complain — it looks the character up in a static confusables table, substitutes the real ASCII token (-), keeps parsing, and reports the problem as a side effect. The diagnostic is generated from the same table that produced the recovery.

Copy-pasting from a word processor is a common way to introduce Unicode characters that look like ASCII ones. The classic offenders:

– (en dash) instead of -;
‘ / ’ (curly quotes) instead of ' and ";
non-breaking spaces;
full-width characters from CJK input methods.

rustc’s response is unusually good: it does not just point at the character, it fixes the parse and tells you what it did.

The table

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:1-12

//! Characters and their corresponding confusables were collected from
//! <https://www.unicode.org/Public/security/10.0.0/confusables.txt>

use rustc_span::{BytePos, Pos, Span, kw};

use super::Lexer;
use crate::diagnostics::TokenSubstitution;
use crate::token;

#[rustfmt::skip] // for line breaks
pub(super) static UNICODE_ARRAY: &[(char, &str, &str)] = &[
 (' ', "Line Separator", " "),
 (' ', "Paragraph Separator", " "),

Each entry is (unicode char, unicode name, ASCII replacement). The relevant rows:

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:37-59, 131-166, 168-184

 ('‐', "Hyphen", "-"),
 ('‑', "Non-Breaking Hyphen", "-"),
 ('‒', "Figure Dash", "-"),
 ('–', "En Dash", "-"),
 ('—', "Em Dash", "-"),
 ...
 ('‘', "Left Single Quotation Mark", "'"),
 ('’', "Right Single Quotation Mark", "'"),
 ...
 ('“', "Left Double Quotation Mark", "\""),
 ('”', "Right Double Quotation Mark", "\""),

The table is generated from Unicode’s own confusables data file, so it covers thousands of characters.

The lookup returns a token

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:339-380

pub(super) fn check_for_substitution(
 lexer: &Lexer<'_, '_>,
 pos: BytePos,
 ch: char,
 count: usize,
) -> (Option<token::TokenKind>, Option<TokenSubstitution>) {
 let Some(&(_, u_name, ascii_str)) = UNICODE_ARRAY.iter().find(|&&(c, _, _)| c == ch) else {
 return (None, None);
 };

 let span = Span::with_root_ctxt(pos, pos + Pos::from_usize(ch.len_utf8() * count));

 let Some((_, ascii_name, token)) = ASCII_ARRAY.iter().find(|&&(s, _, _)| s == ascii_str) else {
 let msg = format!("substitution character not found for '{ch}'");
 lexer.dcx().span_bug(span, msg);
 };

 // special help suggestion for "directed" double quotes
 let sugg = if let Some(s) = peek_delimited(&lexer.src[lexer.src_index(pos)..], '“', '”') {
 let span = Span::with_root_ctxt(
 pos,
 pos + Pos::from_usize('“'.len_utf8() + s.len() + '”'.len_utf8()),
 );
 Some(TokenSubstitution::DirectedQuotes {
 span,
 suggestion: format!("\"{s}\""),
 ascii_str,
 ascii_name,
 })
 } else {
 let suggestion = ascii_str.to_string().repeat(count);
 Some(TokenSubstitution::Other {
 span,
 suggestion,
 ch: ch.to_string(),
 u_name,
 ascii_str,
 ascii_name,
 })
 };
 (*token, sugg)
}

Three things to notice:

The return value is a real TokenKind. The ASCII_ARRAY maps the ASCII replacement back to a token, so an en-dash becomes token::Minus. The parser continues with a valid token stream.
The diagnostic is a by-product. The same function that performs the substitution also builds the suggestion. No second pass, no second table.
The “directed quotes” special case. If the input has “hello”, the suggestion replaces the whole quoted run with "hello", which is a much better fix than replacing each smart quote separately:

 rust
 
 
// compiler/rustc_parse/src/lexer/unicode_chars.rs:383-391

/// Extract string if found at current position with given delimiters
fn peek_delimited(text: &str, from_ch: char, to_ch: char) -> Option<&str> {
 let mut chars = text.chars();
 let first_char = chars.next()?;
 if first_char != from_ch {
 return None;
 }
 let last_char_idx = chars.as_str().find(to_ch)?;
 Some(&chars.as_str()[..last_char_idx])
}

Where it is invoked, and the “swallow the repeats” trick

 rust
 
 
// compiler/rustc_parse/src/lexer/mod.rs:486-514

let repeats = it.take_while(|c1| *c1 == c).count();
// FIXME: the lexer could be used to turn the ASCII version of unicode
// homoglyphs, instead of keeping a table in `check_for_substitution`into the
// token. Ideally, this should be inside `rustc_lexer`. However, we should
// first remove compound tokens like `<<` from `rustc_lexer`, and then add
// fancier error recovery to it, as there will be less overall work to do this
// way.
let (token, sugg) =
 unicode_chars::check_for_substitution(self, start, c, repeats + 1);
self.dcx().emit_err(crate::diagnostics::UnknownTokenStart {
 span: self.mk_sp(start, self.pos + Pos::from_usize(repeats * c.len_utf8())),
 escaped: escaped_char(c),
 sugg,
 null: c == '\x00',
 invisible: INVISIBLE_CHARACTERS.contains(&c),
 repeat: if repeats > 0 {
 swallow_next_invalid = repeats;
 Some(crate::diagnostics::UnknownTokenRepeat { repeats })
 } else {
 None
 },
});

The swallow_next_invalid mechanism suppresses the next N identical invalid characters, so @@@@@@ produces one error (“character appears 5 more times”) instead of six. The sub-diagnostic renders it:

 rust
 
 
// compiler/rustc_parse/src/diagnostics.rs:2890-2899

#[derive(Subdiagnostic)]
#[note(
 "character appears {$repeats ->
 [one] once more
 *[other] {$repeats} more times
 }"
)]
pub(crate) struct UnknownTokenRepeat {
 pub repeats: usize,
}

Note also the FIXME: the authors think this repair machinery should live in rustc_lexer itself, once the lexer has compound tokens removed and better recovery. It is a known piece of technical debt, documented as such.

The suggestion is deliberately not auto-applied

 rust
 
 
// compiler/rustc_parse/src/diagnostics.rs:2858-2888

#[derive(Subdiagnostic)]
pub(crate) enum TokenSubstitution {
 #[suggestion(
 "Unicode characters '“' (Left Double Quotation Mark) and '”' (Right Double Quotation Mark) look like '{$ascii_str}' ({$ascii_name}), but are not",
 code = "{suggestion}",
 applicability = "maybe-incorrect",
 style = "verbose"
 )]
 DirectedQuotes { ... },

applicability = "maybe-incorrect" means rustfix will offer the fix but will not apply it automatically. For a confusable replacement, that is the right call: the character might be intentional.

The other unicode trick: fast-path NFC normalisation

 rust
 
 
// compiler/rustc_parse/src/lexer/mod.rs:1278-1287

pub fn nfc_normalize(string: &str) -> Symbol {
 use unicode_normalization::{IsNormalized, UnicodeNormalization, is_nfc_quick};
 match is_nfc_quick(string.chars()) {
 IsNormalized::Yes => Symbol::intern(string),
 _ => {
 let normalized_str: String = string.chars().nfc().collect();
 Symbol::intern(&normalized_str)
 }
 }
}

Every identifier goes through this. is_nfc_quick is a cheap O(n) check that answers “is this already normalised?”; the expensive full normalisation only runs for identifiers that actually need it. Identifiers are normalised so that é written as one codepoint and as e + combining acute are the same identifier.

Where it lives

compiler/rustc_parse/src/lexer/unicode_chars.rs — the tables and check_for_substitution.
compiler/rustc_parse/src/lexer/mod.rs — the invocation, the repeat suppression, nfc_normalize.
compiler/rustc_parse/src/diagnostics.rs — TokenSubstitution.

Takeaway

Error recovery is a design problem, not a diagnostics problem. The best recovery is the kind that lets the compiler keep going and produces a better message than a plain error would. rustc’s confusable handling does both from a single table: the same lookup that produces the token produces the suggestion, so the recovery is essentially free.

rustc's Trait Solver Doesn't Write Error Messages: It Writes Proof Trees

Jay Hubbert — Wed, 21 Oct 2026 11:00:00 +0000

TL;DR: During trait solving, rustc never formats an error message. It records a proof tree — a structured trace of which candidate was tried and what it required — and only if the goal ultimately fails does it replay that trace (with the global cache disabled) to find the obligation that actually failed. The result is a fast solver and much better error messages.

Most compilers build error messages as they go. The solver tries a candidate, the candidate fails, and the failure handler formats a diagnostic right there. It works, but it is expensive: every candidate evaluation may construct spans, allocate strings, and build a Diag that will be thrown away 99.9% of the time.

rustc’s next-generation solver does the opposite.

The proof tree

 rust
 
 
// compiler/rustc_next_trait_solver/src/solve/inspect/build.rs:17-46

/// We need to know whether to build a prove tree while evaluating. We
/// pass a `ProofTreeBuilder` with `state: Some(None)` into the search
/// graph which then causes the initial `EvalCtxt::compute_goal` to actually
/// build a proof tree which then gets written into the `state`.
///
/// Building the proof tree for a single evaluation step happens via the
/// [EvaluationStepBuilder] which is updated by the `EvalCtxt` when
/// appropriate.
pub(crate) struct ProofTreeBuilder<D, I = <D as SolverDelegate>::Interner>
where
 D: SolverDelegate<Interner = I>,
 I: Interner,
{
 state: Option<Box<Option<inspect::Probe<I>>>>,
 _infcx: PhantomData<D>,
}

impl<D: SolverDelegate<Interner = I>, I: Interner> ProofTreeBuilder<D> {
 pub(crate) fn new() -> ProofTreeBuilder<D> {
 ProofTreeBuilder { state: Some(Box::new(None)), _infcx: PhantomData }
 }

 pub(crate) fn new_noop() -> ProofTreeBuilder<D> {
 ProofTreeBuilder { state: None, _infcx: PhantomData }
 }

 pub(crate) fn is_noop(&self) -> bool {
 self.state.is_none()
 }
}

Note the two constructors. new_noop() builds nothing. The normal solving path uses the noop builder — the proof tree is not even allocated. Only when the diagnostic machinery asks for one does the real builder get installed.

The cache is disabled while recording

This is the subtle part. If you replay a proof tree from the global cache, you get a different trace than the one that actually ran, because the cached path skipped work. So the search graph disables the cache when a proof tree is being built:

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:834-852

// Lookup the global cache unless we're building proof trees or are currently
// fuzzing.
let validate_cache = if !D::inspect_is_noop(inspect) {
 None
} else if let Some(scope) = D::enter_validation_scope(cx, input) {
 ...
} else if let Some(result) =
 self.lookup_global_cache(cx, input, step_kind_from_parent, available_depth)
{
 return result;
} else {
 None
};

Read that first branch: if !D::inspect_is_noop(inspect) { None } — when we are building a proof tree, the global cache lookup is skipped entirely. The recorded trace is therefore the true, uncached, from-scratch evaluation.

That makes diagnostics more expensive than solving (the goal is re-run from scratch), but it makes them correct, and it happens only on the error path.

Replaying the tree

When an obligation fails, the diagnostic machinery re-runs the goal with a real builder and walks the resulting tree to find the deepest interesting obligation:

 rust
 
 
// compiler/rustc_trait_selection/src/solve/inspect/analyse.rs:178-200

pub fn instantiate_proof_tree_for_nested_goal(
 &self,
 source: GoalSource,
 goal: Goal<'tcx, ty::Predicate<'tcx>>,
 span: Span,
) -> InspectGoal<'a, 'tcx> {
 let infcx = self.goal.infcx;
 match goal.predicate.kind().no_bound_vars() {
 Some(ty::PredicateKind::NormalizesTo(ty::NormalizesTo { .. })) => {
 // We don't handle `NormalizesTo` as a nested goal
 unreachable!()
 }
 _ => {
 // We're using a probe here as evaluating a goal could constrain
 // inference variables by choosing one candidate. If we then recurse
 // into another candidate who ends up with different inference
 // constraints, we get an ICE if we already applied the constraints
 // from the chosen candidate.
 let proof_tree =
 infcx.probe(|_| infcx.evaluate_root_goal_for_proof_tree(goal, span).1);
 InspectGoal::new(infcx, self.goal.depth + 1, proof_tree, source)
 }
 }
}

and the error derivation walks it looking for the best leaf:

 rust
 
 
// compiler/rustc_trait_selection/src/solve/fulfill/derive_errors.rs:150-178

fn find_best_leaf_obligation<'tcx>(
 infcx: &'tcx InferCtxt<'tcx>,
 obligation: &PredicateObligation<'tcx>,
 consider_ambiguities: bool,
) -> PredicateObligation<'tcx> {
 let obligation = infcx.deeply_resolve_ignoring_regions(obligation.clone());
 // FIXME: we use a probe here as the `BestObligation` visitor does not
 // check whether it uses candidates which get shadowed by where-bounds.
 //
 // We should probably fix the visitor to not do so instead, as this also
 // means the leaf obligation may be incorrect.
 let obligation = infcx
 .fudge_inference_if_ok(|| {
 infcx
 .visit_proof_tree(
 obligation.as_goal(),
 &mut BestObligation { obligation: obligation.clone(), consider_ambiguities },
 )
 .break_value()
 .ok_or(())
 // walk around the fact that the cause in `Obligation` is ignored by folders so that
 // we can properly fudge the infer vars in cause code.
 .map(|o| (o.cause.clone(), o))
 })
 .map(|(cause, o)| PredicateObligation { cause, ..o })
 .unwrap_or(obligation);
 deeply_normalize_for_diagnostics(infcx, obligation.param_env, obligation)
}

visit_proof_tree walks the recorded Probe tree — steps like AddGoal, NestedProbe, MakeCanonicalResponse, RecordImplArgs — and the visitor picks the “best” leaf obligation.

The `probe` wrapper is load-bearing

Notice the infcx.probe(...) in the snippet above. Replaying a candidate can constrain inference variables — choosing a candidate applies its where-clauses to the inference context. If the diagnostic machinery then replays a different candidate with different constraints, it would hit an ICE (“we already applied the constraints from the chosen candidate”).

The fix is to run the replay inside a probe: a snapshot of the inference context that is rolled back afterwards. This is the same snapshot/undo machinery the solver itself uses to try candidates and back out.

The FIXME in the source is an honest admission that this is a workaround: the visitor should not be using shadowed candidates in the first place.

What the solver does build eagerly

Not everything is deferred. The solver does maintain a canonical response for each candidate — the answer, canonicalized so it can be cached and compared:

rust

// compiler/rustc_next_trait_solver/src/solve/mod.rs:321-405

and the merging logic is where the performance lives:

 rust
 
 
let always_applicable = candidates.iter().enumerate().find(|(_, candidate)| {
 candidate.result.value.certainty == Certainty::Yes
 && has_no_inference_or_external_constraints(candidate.result)
});

if let Some((i, candidate)) = always_applicable {
 return Some((candidate.result, MergeCandidateInfo::AlwaysApplicable(i)));
}

If one candidate applies with certainty and imposes no constraints, it subsumes the others — and the solver can then ignore the other candidates' head usages, which prunes fixpoint iterations.

If the candidates are not equal but are equal modulo region constraints, they merge. Otherwise the solver flounders to ambiguity, keeping the strongest “maybe cause”:

 rust
 
 
fn bail_with_ambiguity(&mut self, candidates: &[Candidate<I>]) -> CanonicalResponse<I> {
 debug_assert!(candidates.len() > 1);
 let maybe = candidates.iter().fold(MaybeInfo::AMBIGUOUS, |maybe, candidate| {
 // We pull down the certainty of `Certainty::Yes` to ambiguity when combining
 // these responses, b/c we're combining more than one response and this we
 // don't know which one applies.
 match candidate.result.value.certainty {
 Certainty::Yes => maybe,
 Certainty::Maybe(cand_maybe) => maybe.or(cand_maybe),
 }
 });
 self.make_ambiguous_response_no_constraints(maybe)
}

Note the comment: combining responses lowers certainty to ambiguity because we do not know which candidate applies. But the “maybe cause” is preserved with or, so an overflow is not silently downgraded to plain ambiguity.

Where it lives

compiler/rustc_next_trait_solver/src/solve/inspect/build.rs — the ProofTreeBuilder.
compiler/rustc_type_ir/src/search_graph/mod.rs — cache disabling during proof-tree construction.
compiler/rustc_trait_selection/src/solve/inspect/analyse.rs — replaying the tree.
compiler/rustc_trait_selection/src/solve/fulfill/derive_errors.rs — finding the best leaf obligation.
compiler/rustc_next_trait_solver/src/solve/mod.rs — candidate merging.

Takeaway

Deferred diagnostics are a classic performance technique: do not build the thing you might throw away. What is unusual here is how rustc defers them. It does not merely postpone the message; it records enough structured information to reconstruct the evaluation later, and it deliberately disables its cache during that reconstruction so the replay is faithful. The side effect is better errors: the diagnostic code can ask questions of the trace (“which nested goal failed?”) that eager diagnostics could never answer.

The Borrow Checker Blames the Most Interesting Constraint: BFS, Then a Ranking

Jay Hubbert — Wed, 21 Oct 2026 10:35:00 +0000

TL;DR: Lifetime errors are hard to explain because there are usually many constraints that could be blamed. rustc picks one in two steps: a breadth-first search finds the shortest path through the constraint graph (fewest constraints), and then a hand-tuned “interest” score picks the most user-facing constraint on that path — scanning from the source end or the target end depending on which side is being blamed.

A borrow checker error looks like this:

text

error[E0597]: `x` does not live long enough
 --> src/main.rs:5:20
 |
5 | drop(x);
 | ^ borrowed value does not live long enough

The “borrowed value does not live long enough” part is easy. The hard part is picking which constraint in the chain of lifetime requirements to blame. A function with six lifetime parameters can have dozens of outlives constraints between them, and most of them are compiler-generated noise.

Step one: shortest path

 rust
 
 
// compiler/rustc_borrowck/src/region_infer/region_context.rs:453-475

let mut context = IndexVec::from_elem(Trace::NotVisited, &self.definitions);
context[from_region] = Trace::StartRegion;

let fr_static = self.universal_regions().fr_static;

// Use a deque so that we do a breadth-first search. We will
// stop at the first match, which ought to be the shortest
// path (fewest constraints).
let mut deque = VecDeque::new();
deque.push_back(from_region);

while let Some(r) = deque.pop_front() {

The choice of VecDeque + pop_front over Vec + pop is deliberate and commented: BFS, not DFS. A DFS would find a path, and the path it finds would be arbitrary — often long, often winding through irrelevant constraints. BFS finds the path with the fewest constraints, which is already a decent proxy for “the constraint the user most likely intended”.

Step two: the interest ranking

But shortest is not the same as most useful. rustc goes further and scores each constraint by how interesting it is for a human:

 rust
 
 
// compiler/rustc_borrowck/src/region_infer/region_context.rs:684-768

// To pick a constraint to blame, we organize constraints by how interesting we expect them
// to be in diagnostics, then pick the most interesting one closest to either the source or
// the target on our constraint path.
let constraint_interest = |constraint: &OutlivesConstraint<'tcx>| {
 // Try to avoid blaming constraints from desugarings, since they may not clearly match
 // match what users have written. As an exception, allow blaming returns generated by
 // `?` desugaring, since the correspondence is fairly clear.
 let category = if let Some(kind) = constraint.span.desugaring_kind()
 && (kind != DesugaringKind::QuestionMark
 || !matches!(constraint.category, ConstraintCategory::Return(_)))
 {
 ConstraintCategory::Boring
 } else {
 constraint.category
 };

 let interest = match category {
 // Returns usually provide a type to blame and have specially written diagnostics,
 // so prioritize them.
 ConstraintCategory::Return(_) => 0,

The scores are hand-assigned integers. Zero is “a return expression”, which gets priority because it has a type to show and a dedicated diagnostic message. Everything else is worse, in increasing order, ending with solver-internal constraints.

The final selection picks the best-scoring constraint, scanning the path from one end or the other:

 rust
 
 
// compiler/rustc_borrowck/src/region_infer/region_context.rs:1510-1513 (region_context.rs:750-768)

let best_choice = if blame_source {
 path.iter().enumerate().rev().min_by_key(|(_, c)| constraint_interest(c)).unwrap().0
} else {
 path.iter().enumerate().min_by_key(|(_, c)| constraint_interest(c)).unwrap().0
};

min_by_key finds the lowest interest number, i.e. the most interesting constraint. The only difference between the two branches is the direction of the scan:

blame_source → scan the path in reverse, preferring constraints near the source;
otherwise (blaming a placeholder or a target) → scan forward, preferring constraints near the target.

That asymmetry matters. When you are blaming a sub-type bound, the interesting constraint is near the source; when you are blaming a higher-ranked placeholder, the interesting constraint is near the target.

The cold path is deferred

One more detail worth stealing: the constraints themselves are not built during the search. The search walks a Trace graph, and the actual OutlivesConstraint values are only materialized at the end, on the error path:

 rust
 
 
// compiler/rustc_borrowck/src/region_infer/region_context.rs:477-509

let mut result = vec![];
let mut p = r;
// This loop is cold and runs at the end, which is why we delay
// `OutlivesConstraint` construction until now.
loop {
 match context[p] {
 Trace::FromGraph(c) => {
 p = c.sup;
 result.push(*c);
 }
 ...

The comment is the whole justification: constraint construction involves spans, categories and variance info, and building it for every successful borrow check would be pure waste. The search therefore carries a lightweight Trace enum — NotVisited, StartRegion, FromGraph, FromStatic — and only converts it into real constraints when it is about to complain.

Why not just blame the constraint that failed?

Because there usually isn’t one. The constraint that “failed” is often a derived fact — the region grew too large because of a chain of other constraints. Blaming the derived failure gives the user a message about something they never wrote.

Instead rustc picks the constraint on the path that is closest to the user and most likely to be the real intent. If you have ever seen a lifetime error that pointed at a ? operator, at a return statement, or at an explicit type annotation rather than at the compiler’s internal machinery, this is why.

Where it lives

compiler/rustc_borrowck/src/region_infer/region_context.rs — the BFS (best_blame_constraint), the constraint_interest closure, and the deferred construction of OutlivesConstraint.
compiler/rustc_borrowck/src/region_infer/region_context.rs — ConstraintCategory, the enum the ranking is built on.

Takeaway

Error messages are a performance problem too: they run on the cold path, but they run on the path that users actually see, and a confusing error costs a user far more time than a slow compile. Spending a BFS plus a hand-tuned ranking to pick the right constraint is a good trade, and the “build the expensive thing only on the error path” pattern makes it cheap enough to always do.

Diagnostics on Opensource internals

Did You Mean a Hyphen?: How rustc Repairs Unicode Confusables While Parsing

The table

The lookup returns a token

Where it is invoked, and the “swallow the repeats” trick

The suggestion is deliberately not auto-applied

The other unicode trick: fast-path NFC normalisation

Where it lives

Takeaway

rustc's Trait Solver Doesn't Write Error Messages: It Writes Proof Trees

The proof tree

The cache is disabled while recording

Replaying the tree

The probe wrapper is load-bearing

What the solver does build eagerly

Where it lives

Takeaway

The Borrow Checker Blames the Most Interesting Constraint: BFS, Then a Ranking

Step one: shortest path

Step two: the interest ranking

The cold path is deferred

Why not just blame the constraint that failed?

Where it lives

Takeaway

The `probe` wrapper is load-bearing