Trait-Solver on Opensource internals

rustc's Trait Solver Doesn't Write Error Messages: It Writes Proof Trees

Jay Hubbert — Wed, 21 Oct 2026 11:00:00 +0000

TL;DR: During trait solving, rustc never formats an error message. It records a proof tree — a structured trace of which candidate was tried and what it required — and only if the goal ultimately fails does it replay that trace (with the global cache disabled) to find the obligation that actually failed. The result is a fast solver and much better error messages.

Most compilers build error messages as they go. The solver tries a candidate, the candidate fails, and the failure handler formats a diagnostic right there. It works, but it is expensive: every candidate evaluation may construct spans, allocate strings, and build a Diag that will be thrown away 99.9% of the time.

rustc’s next-generation solver does the opposite.

The proof tree

 rust
 
 
// compiler/rustc_next_trait_solver/src/solve/inspect/build.rs:17-46

/// We need to know whether to build a prove tree while evaluating. We
/// pass a `ProofTreeBuilder` with `state: Some(None)` into the search
/// graph which then causes the initial `EvalCtxt::compute_goal` to actually
/// build a proof tree which then gets written into the `state`.
///
/// Building the proof tree for a single evaluation step happens via the
/// [EvaluationStepBuilder] which is updated by the `EvalCtxt` when
/// appropriate.
pub(crate) struct ProofTreeBuilder<D, I = <D as SolverDelegate>::Interner>
where
 D: SolverDelegate<Interner = I>,
 I: Interner,
{
 state: Option<Box<Option<inspect::Probe<I>>>>,
 _infcx: PhantomData<D>,
}

impl<D: SolverDelegate<Interner = I>, I: Interner> ProofTreeBuilder<D> {
 pub(crate) fn new() -> ProofTreeBuilder<D> {
 ProofTreeBuilder { state: Some(Box::new(None)), _infcx: PhantomData }
 }

 pub(crate) fn new_noop() -> ProofTreeBuilder<D> {
 ProofTreeBuilder { state: None, _infcx: PhantomData }
 }

 pub(crate) fn is_noop(&self) -> bool {
 self.state.is_none()
 }
}

Note the two constructors. new_noop() builds nothing. The normal solving path uses the noop builder — the proof tree is not even allocated. Only when the diagnostic machinery asks for one does the real builder get installed.

The cache is disabled while recording

This is the subtle part. If you replay a proof tree from the global cache, you get a different trace than the one that actually ran, because the cached path skipped work. So the search graph disables the cache when a proof tree is being built:

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:834-852

// Lookup the global cache unless we're building proof trees or are currently
// fuzzing.
let validate_cache = if !D::inspect_is_noop(inspect) {
 None
} else if let Some(scope) = D::enter_validation_scope(cx, input) {
 ...
} else if let Some(result) =
 self.lookup_global_cache(cx, input, step_kind_from_parent, available_depth)
{
 return result;
} else {
 None
};

Read that first branch: if !D::inspect_is_noop(inspect) { None } — when we are building a proof tree, the global cache lookup is skipped entirely. The recorded trace is therefore the true, uncached, from-scratch evaluation.

That makes diagnostics more expensive than solving (the goal is re-run from scratch), but it makes them correct, and it happens only on the error path.

Replaying the tree

When an obligation fails, the diagnostic machinery re-runs the goal with a real builder and walks the resulting tree to find the deepest interesting obligation:

 rust
 
 
// compiler/rustc_trait_selection/src/solve/inspect/analyse.rs:178-200

pub fn instantiate_proof_tree_for_nested_goal(
 &self,
 source: GoalSource,
 goal: Goal<'tcx, ty::Predicate<'tcx>>,
 span: Span,
) -> InspectGoal<'a, 'tcx> {
 let infcx = self.goal.infcx;
 match goal.predicate.kind().no_bound_vars() {
 Some(ty::PredicateKind::NormalizesTo(ty::NormalizesTo { .. })) => {
 // We don't handle `NormalizesTo` as a nested goal
 unreachable!()
 }
 _ => {
 // We're using a probe here as evaluating a goal could constrain
 // inference variables by choosing one candidate. If we then recurse
 // into another candidate who ends up with different inference
 // constraints, we get an ICE if we already applied the constraints
 // from the chosen candidate.
 let proof_tree =
 infcx.probe(|_| infcx.evaluate_root_goal_for_proof_tree(goal, span).1);
 InspectGoal::new(infcx, self.goal.depth + 1, proof_tree, source)
 }
 }
}

and the error derivation walks it looking for the best leaf:

 rust
 
 
// compiler/rustc_trait_selection/src/solve/fulfill/derive_errors.rs:150-178

fn find_best_leaf_obligation<'tcx>(
 infcx: &'tcx InferCtxt<'tcx>,
 obligation: &PredicateObligation<'tcx>,
 consider_ambiguities: bool,
) -> PredicateObligation<'tcx> {
 let obligation = infcx.deeply_resolve_ignoring_regions(obligation.clone());
 // FIXME: we use a probe here as the `BestObligation` visitor does not
 // check whether it uses candidates which get shadowed by where-bounds.
 //
 // We should probably fix the visitor to not do so instead, as this also
 // means the leaf obligation may be incorrect.
 let obligation = infcx
 .fudge_inference_if_ok(|| {
 infcx
 .visit_proof_tree(
 obligation.as_goal(),
 &mut BestObligation { obligation: obligation.clone(), consider_ambiguities },
 )
 .break_value()
 .ok_or(())
 // walk around the fact that the cause in `Obligation` is ignored by folders so that
 // we can properly fudge the infer vars in cause code.
 .map(|o| (o.cause.clone(), o))
 })
 .map(|(cause, o)| PredicateObligation { cause, ..o })
 .unwrap_or(obligation);
 deeply_normalize_for_diagnostics(infcx, obligation.param_env, obligation)
}

visit_proof_tree walks the recorded Probe tree — steps like AddGoal, NestedProbe, MakeCanonicalResponse, RecordImplArgs — and the visitor picks the “best” leaf obligation.

The `probe` wrapper is load-bearing

Notice the infcx.probe(...) in the snippet above. Replaying a candidate can constrain inference variables — choosing a candidate applies its where-clauses to the inference context. If the diagnostic machinery then replays a different candidate with different constraints, it would hit an ICE (“we already applied the constraints from the chosen candidate”).

The fix is to run the replay inside a probe: a snapshot of the inference context that is rolled back afterwards. This is the same snapshot/undo machinery the solver itself uses to try candidates and back out.

The FIXME in the source is an honest admission that this is a workaround: the visitor should not be using shadowed candidates in the first place.

What the solver does build eagerly

Not everything is deferred. The solver does maintain a canonical response for each candidate — the answer, canonicalized so it can be cached and compared:

rust

// compiler/rustc_next_trait_solver/src/solve/mod.rs:321-405

and the merging logic is where the performance lives:

 rust
 
 
let always_applicable = candidates.iter().enumerate().find(|(_, candidate)| {
 candidate.result.value.certainty == Certainty::Yes
 && has_no_inference_or_external_constraints(candidate.result)
});

if let Some((i, candidate)) = always_applicable {
 return Some((candidate.result, MergeCandidateInfo::AlwaysApplicable(i)));
}

If one candidate applies with certainty and imposes no constraints, it subsumes the others — and the solver can then ignore the other candidates' head usages, which prunes fixpoint iterations.

If the candidates are not equal but are equal modulo region constraints, they merge. Otherwise the solver flounders to ambiguity, keeping the strongest “maybe cause”:

 rust
 
 
fn bail_with_ambiguity(&mut self, candidates: &[Candidate<I>]) -> CanonicalResponse<I> {
 debug_assert!(candidates.len() > 1);
 let maybe = candidates.iter().fold(MaybeInfo::AMBIGUOUS, |maybe, candidate| {
 // We pull down the certainty of `Certainty::Yes` to ambiguity when combining
 // these responses, b/c we're combining more than one response and this we
 // don't know which one applies.
 match candidate.result.value.certainty {
 Certainty::Yes => maybe,
 Certainty::Maybe(cand_maybe) => maybe.or(cand_maybe),
 }
 });
 self.make_ambiguous_response_no_constraints(maybe)
}

Note the comment: combining responses lowers certainty to ambiguity because we do not know which candidate applies. But the “maybe cause” is preserved with or, so an overflow is not silently downgraded to plain ambiguity.

Where it lives

compiler/rustc_next_trait_solver/src/solve/inspect/build.rs — the ProofTreeBuilder.
compiler/rustc_type_ir/src/search_graph/mod.rs — cache disabling during proof-tree construction.
compiler/rustc_trait_selection/src/solve/inspect/analyse.rs — replaying the tree.
compiler/rustc_trait_selection/src/solve/fulfill/derive_errors.rs — finding the best leaf obligation.
compiler/rustc_next_trait_solver/src/solve/mod.rs — candidate merging.

Takeaway

Deferred diagnostics are a classic performance technique: do not build the thing you might throw away. What is unusual here is how rustc defers them. It does not merely postpone the message; it records enough structured information to reconstruct the evaluation later, and it deliberately disables its cache during that reconstruction so the replay is faithful. The side effect is better errors: the diagnostic code can ask questions of the trace (“which nested goal failed?”) that eager diagnostics could never answer.

Divide by Four: How rustc's Trait Solver Makes Itself Stop

Jay Hubbert — Wed, 21 Oct 2026 10:55:00 +0000

TL;DR: The new trait solver has a recursion budget. Normally each nested goal costs one unit. But once an overflow has been observed anywhere below, the budget is divided by four instead of decremented. The effective depth after k overflows is depth / 4^k, so the total work is linear rather than quadratic — and the solver gives up almost instantly after the first sign of pathology.

Trait solving is the part of a Rust compiler most likely to run away. The problems are mutually recursive trait bounds, deeply nested associated types, and goals that expand into thousands of sub-goals. The old solver handled this with a recursion limit and a “cycle” error. The new solver handles it with a budget that shrinks geometrically once things go bad.

The budget

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:287-326

impl AvailableDepth {
 /// Returns the remaining depth allowed for nested goals.
 ///
 /// This is generally simply one less than the current depth.
 /// However, if we encountered overflow, we significantly reduce
 /// the remaining depth of all nested goals to prevent hangs
 /// in case there is exponential blowup.
 fn allowed_depth_for_nested<D: Delegate>(
 root_depth: AvailableDepth,
 stack: &Stack<D::Cx>,
 lower_available_depth: LowerAvailableDepth,
 ) -> Option<AvailableDepth> {
 if let Some(last) = stack.last() {
 match lower_available_depth {
 LowerAvailableDepth::Yes => {}
 LowerAvailableDepth::No => {
 return Some(last.available_depth);
 }
 }

 if last.available_depth.0 == 0 {
 return None;
 }

 Some(if last.encountered_overflow {
 AvailableDepth(last.available_depth.0 / D::DIVIDE_AVAILABLE_DEPTH_ON_OVERFLOW)
 } else {
 AvailableDepth(last.available_depth.0 - 1)
 })
 } else {
 Some(root_depth)
 }
 }

and the divisor:

 rust
 
 
// compiler/rustc_next_trait_solver/src/solve/search_graph.rs:43
const DIVIDE_AVAILABLE_DEPTH_ON_OVERFLOW: usize = 4;

Read the difference between the two branches:

normal case: available_depth - 1;
after an overflow: available_depth / 4.

That is the entire trick. The budget is not a counter, it is an amount of remaining work, and once the solver knows it is in a pathological region it takes away three quarters of the remaining work rather than one unit.

Why dividing is the right shape

The classic failure mode of a recursion-limited solver is quadratic behaviour: each level of the recursion can fan out into a constant number of sub-goals, so the total number of goals is proportional to depth^2 in the worst case. With a budget of 128, that is thousands of goals — each of which may be expensive.

Dividing changes the shape. After the first overflow, the budget is depth / 4; after the second (one level deeper), depth / 16; then depth / 64. The total work becomes:

text

depth + depth/4 + depth/16 + ... = depth * 4/3

— linear in the budget rather than quadratic. The solver does not “run for 128 levels”; it runs for 128, then 32, then 8, then 2, then 0. Total nesting explored: about 160 levels, but the branching factor collapses along the way.

The practical effect is that the new solver gives up fast on pathological input, which is what you want from a compiler: a clear “trait bound is not satisfied” (or an explicit overflow error) in 10 ms beats a 40-second hang.

The overflow handler poisons the parent

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:936-957

fn handle_overflow(&self, cx: X, input: X::Input) -> X::Result {
 if let Some(last) = self.stack.last_mut() {
 last.encountered_overflow = true;
 // If computing a goal `B` depends on another goal `A` and
 // `A` has a nested goal which overflows, then computing `B`
 // at the same depth, but with `A` already on the stack,
 // would encounter a solver cycle instead, potentially
 // changing the result.
 //
 // We must therefore not use the global cache entry for `B` in that case.
 // See tests/ui/traits/next-solver/cycles/hidden-by-overflow.rs
 last.nested_goals.insert(last.input, PathsToNested::EMPTY);
 }

 debug!("encountered stack overflow");
 D::stack_overflow_result(cx, input)
}

Two things happen when an overflow is detected. The frame is marked (encountered_overflow = true), which is what makes the next nested goal divide rather than decrement. And the parent’s nested-goal set is invalidated, so the global cache entry for the parent is not reused — because computing the parent at a shallower depth, with the overflowing goal already on the stack, could produce a cycle instead of an overflow, and those two have different answers.

That second point is subtle and important: overflow and cycle are not the same thing, and the solver has to make sure a cached “yes” from one context is not reused in the other.

The `LowerAvailableDepth::No` escape hatch

Not all nested goals should consume depth. NormalizesTo goals behave like function calls rather than independent nested evaluations:

rust

// compiler/rustc_type_ir/src/search_graph/mod.rs:263-275

When LowerAvailableDepth::No is in effect, the child’s budget is the parent’s budget, unchanged. So a chain of normalizations does not burn the budget, while a chain of nested obligations does.

The cache has to know about all this

For the global cache to stay sound, each entry records the goals it transitively touched — but only once something interesting has happened:

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:498-514

/// The nested goals of each stack entry and the path from the
/// stack entry to that nested goal.
///
/// They are used when checking whether reevaluating a global cache
/// would encounter a cycle or use a provisional cache entry given
/// the current search graph state. We need to disable the global cache
/// in this case as it could otherwise result in behavioral differences.
/// Cycles can impact behavior. The cycle ABA may have different final
/// results from a the cycle BAB depending on the cycle root.
///
/// We only start tracking nested goals once we've either encountered
/// overflow or a solver cycle. This is a performance optimization to
/// avoid tracking nested goals on the happy path.
#[derive_where(Debug, Default, Clone; X: Cx)]
struct NestedGoals<X: Cx> {
 nested_goals: HashMap<X::Input, PathsToNested>,
}

and the applicability check rejects entries whose nested goals are on the stack:

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:1190-1206

fn candidate_is_applicable(
 &self,
 step_kind_from_parent: PathKind,
 nested_goals: &NestedGoals<X>,
) -> bool {
 // If the global cache entry didn't depend on any nested goals, it always
 // applies.
 if nested_goals.is_empty() {
 return true;
 }

 // If a nested goal of the global cache entry is on the stack, we would
 // definitely encounter a cycle.
 if self.stack.iter().any(|e| nested_goals.contains(e.input)) {
 debug!("cache entry not applicable due to stack");
 return false;
 }

The lazy tracking is a nice touch: on the happy path (no cycles, no overflow) the map is empty and the check is a single is_empty() test. Only in the pathological case does the dependency graph get materialized.

Where it lives

compiler/rustc_type_ir/src/search_graph/mod.rs — AvailableDepth, handle_overflow, NestedGoals, candidate_is_applicable.
compiler/rustc_next_trait_solver/src/solve/search_graph.rs — DIVIDE_AVAILABLE_DEPTH_ON_OVERFLOW = 4.

Takeaway

A recursion limit answers “how deep may I go?”. A budget answers “how much work may I do?”. The second question is the useful one, and when the answer is “a lot”, the right response to a hint of pathology is to remove most of the remaining budget, not a little of it. Dividing by four is a one-line change with a dramatic effect on worst-case compile times.

Coinduction for Free: The Three-Value Lattice That Makes rustc's Trait Solver Terminate

Jay Hubbert — Wed, 21 Oct 2026 10:50:00 +0000

TL;DR: Trait solving can loop: impl Trait for T where proving T: Trait requires proving T: Trait. rustc’s next-generation solver handles this with coinduction — assume success, then iterate until the answer stops changing. The interesting part is the small lattice that classifies a cycle, and the fact that the cycle’s kind is recomputed from scratch on every stack push instead of being cached.

The cycle

The classic cycle in trait solving:

 rust
 
 
trait Foo {}
impl<T: Foo> Foo for T {} // needs `T: Foo` to prove `T: Foo`

When the solver is asked “does T: Foo hold?”, it picks the impl, which requires T: Foo — the same question. A naive solver detects the cycle and gives up (“overflow”). A coinductive solver assumes the answer is “yes”, finishes the current derivation, and then checks whether the assumption holds.

The lattice

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:115-158

/// In the initial iteration of a cycle, we do not yet have a provisional
/// result. In the case we return an initial provisional result depending
/// on the kind of cycle.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum PathKind {
 /// A path consisting of only inductive/unproductive steps. Their initial
 /// provisional result is `Err(NoSolution)`. We currently treat them as
 /// `PathKind::Unknown` during coherence until we're fully confident in
 /// our approach.
 Inductive,
 /// A path which is not be coinductive right now but we may want
 /// to change of them to be so in the future. We return an ambiguous
 /// result in this case to prevent people from relying on this.
 Unknown,
 /// A path with at least one coinductive step. Such cycles hold.
 Coinductive,
 /// A path which is treated as ambiguous. Once a path has this path kind
 /// any other segment does not change its kind.
 ///
 /// This is currently only used when fuzzing to support negative reasoning.
 /// For more details, see #143054.
 ForcedAmbiguity,
}

impl PathKind {
 /// Returns the path kind when merging `self` with `rest`.
 ///
 /// Given an inductive path `self` and a coinductive path `rest`,
 /// the path `self -> rest` would be coinductive.
 ///
 /// This operation represents an ordering and would be equivalent
 /// to `max(self, rest)`.
 fn extend(self, rest: PathKind) -> PathKind {
 match (self, rest) {
 (PathKind::ForcedAmbiguity, _) | (_, PathKind::ForcedAmbiguity) => {
 PathKind::ForcedAmbiguity
 }
 (PathKind::Coinductive, _) | (_, PathKind::Coinductive) => PathKind::Coinductive,
 (PathKind::Unknown, _) | (_, PathKind::Unknown) => PathKind::Unknown,
 (PathKind::Inductive, PathKind::Inductive) => PathKind::Inductive,
 }
 }
}

Four values, not three — the fourth, ForcedAmbiguity, exists only so the fuzzer can inject negative reasoning. The key property is that extend is a join on the lattice Inductive < Unknown < Coinductive < ForcedAmbiguity, and it behaves like max: a single coinductive step makes the whole path coinductive.

Note that Unknown is deliberately not the bottom. Inductive ∪ Unknown = Unknown, not Inductive. The Unknown value exists because the solver does not yet trust its own inductive-cycle detection, so it refuses to commit:

 rust
 
 
// compiler/rustc_next_trait_solver/src/solve/search_graph.rs:45-90

fn initial_provisional_result(
 cx: I,
 kind: PathKind,
 input: I::CanonicalInput,
) -> (QueryResult<I>, AccessedOpaques<I>) {
 match kind {
 PathKind::Coinductive => response_no_constraints(cx, input, Certainty::Yes),
 PathKind::Unknown | PathKind::ForcedAmbiguity => {
 response_no_constraints(cx, input, Certainty::overflow(false))
 }
 // Even though we know some cycles to be unproductive, we still treat them
 // as unknown for now. This is both as we are not 100% confident in the
 // implementation yet and any incorrect errors would be unsound there.
 //
 // The affected cases are also fairly artificial and not necessarily desirable
 // so keeping this as ambiguity is fine for now.
 //
 // See `tests/ui/traits/next-solver/cycles/unproductive-in-coherence.rs` and
 // `tests/ui/traits/next-solver/overflow/recursive-self-normalization-simple.rs`
 // for examples where this would matter.
 //
 // FIXME(-Znext-solver=coinductive): Long term, we probably do want to
 // return `NoSolution` here. This should happen separately from the
 // stabilization of the new solver.
 PathKind::Inductive => unreachable!(),
 }
}

So a coinductive cycle starts with “yes”, and an unknown cycle starts with “ambiguous, probably overflow”. The PathKind::Inductive arm is unreachable!() because rustc never constructs one yet — the FIXME says what would change.

Why the kind is recomputed rather than cached

The cycle’s kind is not stored on the stack frame; it is folded over the stack segment between the head and the current entry, every time a goal is pushed. The consequence is that the kind of a cycle is always correct with respect to the current path, even when the same head is reached by different routes.

That matters because a head can be reached both inductively and coinductively. In that case, the cycle is coinductive (a single coinductive step suffices) and the solver may shortcut to “yes” — but only if all the usages of that head agree.

The fixpoint, with three escape hatches

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:1370-1445

let mut encountered_overflow = false;
let mut i = 0;
loop {
 let result = D::compute_goal(self, cx, input, inspect);
 let stack_entry = self.stack.pop();
 encountered_overflow |= stack_entry.encountered_overflow;
 debug_assert_eq!(stack_entry.input, input);

 // If the current goal is not a cycle head, we are done.
 //
 // There are no provisional cache entries which depend on this goal.
 let Some(usages) = stack_entry.usages else {
 return EvaluationResult::finalize(stack_entry, encountered_overflow, result);
 };

 // If it is a cycle head, we have to keep trying to prove it until
 // we reach a fixpoint. We need to do so for all cycle heads,
 // not only for the root.
 //
 // See tests/ui/traits/next-solver/cycles/fixpoint-rerun-all-cycle-heads.rs
 // for an example.
 //
 // Check whether we reached a fixpoint, either because the final result
 // is equal to the provisional result of the previous iteration, or because
 // this was only the head of either coinductive or inductive cycles, and the
 // final result is equal to the initial response for that case.
 if let Ok(fixpoint) = self.reached_fixpoint(&stack_entry, usages, result) {
 self.rebase_provisional_cache_entries(
 &stack_entry,
 RebaseReason::ReachedFixpoint(fixpoint),
 );
 return EvaluationResult::finalize(stack_entry, encountered_overflow, result);
 } else if usages.is_empty() {
 self.rebase_provisional_cache_entries(&stack_entry, RebaseReason::NoCycleUsages);
 return EvaluationResult::finalize(stack_entry, encountered_overflow, result);
 }

 // If computing this goal results in ambiguity with no constraints,
 // we do not rerun it. It's incredibly difficult to get a different
 // response in the next iteration in this case. These changes would
 // likely either be caused by incompleteness or can change the maybe
 // cause from ambiguity to overflow. Returning ambiguity always
 // preserves soundness and completeness even if the goal could
 // otherwise succeed or fail.
 //
 // This prevents exponential blowup affecting multiple major crates.
 // As we only get to this branch if we haven't yet reached a fixpoint,
 // we also taint all provisional cache entries which depend on the
 // current goal.
 if let Some(kind) = D::is_ambiguous_result(result) {
 self.rebase_provisional_cache_entries(&stack_entry, RebaseReason::Ambiguity(kind));
 return EvaluationResult::finalize(stack_entry, encountered_overflow, result);
 };

 // If we've reached the fixpoint step limit, we bail with overflow and taint all
 // provisional cache entries which depend on the current goal.
 i += 1;
 if i >= D::FIXPOINT_STEP_LIMIT {
 debug!("canonical cycle overflow");
 let result = D::fixpoint_overflow_result(cx, input);
 self.rebase_provisional_cache_entries(
 &stack_entry,
 RebaseReason::Ambiguity(D::FIXPOINT_OVERFLOW_AMBIGUITY_KIND),
 );
 return EvaluationResult::finalize(stack_entry, encountered_overflow, result);
 }

Four exits:

Not a cycle head → done.
Fixpoint reached → done. The check is that the new result equals the previous provisional result, or (on the first iteration) equals the initial provisional result and all usages had the same path kind.
No usages → the head was referenced but its value did not matter.
Ambiguity without constraints → stop, keep the ambiguity. This is the anti-blowup rule, and the comment is unusually honest about it: “This prevents exponential blowup affecting multiple major crates.”
Step limit (8) → overflow.

Without rule 4, a cycle that keeps producing “ambiguous but constraint-free” answers would be re-run eight times, each time re-running the whole subtree — a multiplicative blowup of up to 8^k.

The lattice again, this time as a bitmask

The same lattice shows up in the cache-validity machinery, where “which kinds of paths reach this goal” is a set, and sets of path kinds are stored as bitflags:

 rust
 
 
// compiler/rustc_type_ir/src/search_graph/mod.rs:428-476

impl PathsToNested {
 /// The implementation of this function is kind of ugly. We check whether
 /// there currently exist 'weaker' paths in the set, if so we upgrade these
 /// paths to at least `path`.
 #[must_use]
 fn extend_with(mut self, path: PathKind) -> Self {
 match path {
 PathKind::Inductive => {
 if self.intersects(PathsToNested::EMPTY) {
 self.remove(PathsToNested::EMPTY);
 self.insert(PathsToNested::INDUCTIVE);
 }
 }
 ...

A set of path kinds as a 5-bit mask, with the join implemented as “upgrade weaker paths to at least this one”. That turns “is this cache entry still applicable?” into an O(1) bitmask test instead of an enumeration of paths.

Where it lives

compiler/rustc_type_ir/src/search_graph/mod.rs — PathKind, extend, the fixpoint loop, PathsToNested.
compiler/rustc_next_trait_solver/src/solve/search_graph.rs — initial_provisional_result, the Delegate policy.
compiler/rustc_next_trait_solver/src/solve/mod.rs — FIXPOINT_STEP_LIMIT.

Takeaway

Coinduction is usually taught as “assume true, then check”. The part that makes it implementable is the classification of the cycle into kinds and the discipline of recomputing that classification compositionally, rather than caching it. Once the kind is a value in a small lattice with a max join, everything else — the initial assumption, the shortcut condition, the cache-validity check — falls out of the algebra.

Trait-Solver on Opensource internals

rustc's Trait Solver Doesn't Write Error Messages: It Writes Proof Trees

The proof tree

The cache is disabled while recording

Replaying the tree

The probe wrapper is load-bearing

What the solver does build eagerly

Where it lives

Takeaway

Divide by Four: How rustc's Trait Solver Makes Itself Stop

The budget

Why dividing is the right shape

The overflow handler poisons the parent

The LowerAvailableDepth::No escape hatch

The cache has to know about all this

Where it lives

Takeaway

Coinduction for Free: The Three-Value Lattice That Makes rustc's Trait Solver Terminate

The cycle

The lattice

Why the kind is recomputed rather than cached

The fixpoint, with three escape hatches

The lattice again, this time as a bitmask

Where it lives

Takeaway

The `probe` wrapper is load-bearing

The `LowerAvailableDepth::No` escape hatch