Loop Peeling, Unrolling and Induction Variables in TurboFan

Jay Hubbert — Thu, 22 Oct 2026 14:10:00 +0000

TL;DR: TurboFan has three separate loop optimisations and they live in different places. src/compiler/loop-peeling.cc peels the first iteration of inner loops so the checks in it can be hoisted out of the loop. src/compiler/turboshaft/loop-unrolling-reducer.cc proves a loop’s trip count by pattern-matching the canonical for (i = cst; i cmp cst; i = i binop cst) shape, and unrolls it if the count is at most 4. src/compiler/loop-variable-optimizer.cc finds the phi that carries the counter, turns it into an InductionVariablePhi with the increment and the loop bounds as extra inputs, and lets the typer compute a range type for it. The interesting part is the price: the range type is only sound because the phi is finally guarded by a TypeGuard on the backedge, and that guard is a deoptimization point.

A loop in TurboFan’s IR is three nodes: a Loop (the control node), a Phi (the counter), and an EffectPhi (the effect chain’s loop-carried value). Everything else hangs off those.

That shape invites a family of optimisations, and V8 has spent years accumulating them. This article walks through the three that matter: peeling, unrolling, and the induction variable. Each one is a proof about the loop, and each proof has to be paid for somewhere.

Finding the loops: the loop tree

Before anything can be done to a loop, the compiler has to know which nodes belong to it. src/compiler/loop-analysis.cc builds a LoopTree of nested loops, with each loop recording its header nodes, its body, and its exits. The tree is built by a DFS from the graph’s end node, marking backedges as it goes. The interesting part of that file is not the tree but an assumption it makes:

 cpp
 
 
// src/compiler/loop-analysis.h:25
// TODO(titzer): don't assume entry edges have a particular index.
static const int kAssumedLoopEntryIndex = 0; // assume loops are entered here.

Index 0 of a Loop node’s control inputs is assumed to be the entry edge, and index 1 the backedge. The same constant appears in loop-peeling.cc, loop-variable-optimizer.cc and loop-analysis.cc. The TODO is honest about it being a shortcut.

HasMarkedExits is the other thing you need to know about a loop before you touch it: whether its exits are properly marked with LoopExit nodes, so that rewriting the body does not break the control-flow graph.

 cpp
 
 
// src/compiler/loop-analysis.cc:705
bool LoopFinder::HasMarkedExits(LoopTree* loop_tree,
 const LoopTree::Loop* loop) {
 // Look for returns and if projections that are outside the loop but whose
 // control input is inside the loop.
 Node* loop_node = loop_tree->GetLoopControl(loop);
 for (Node* node : loop_tree->LoopNodes(loop)) {
 for (Node* use : node->uses()) {
 if (!loop_tree->Contains(loop, use)) {
 bool unmarked_exit;
 switch (node->opcode()) {
 case IrOpcode::kLoopExit:
 unmarked_exit = (node->InputAt(1) != loop_node);
 break;
 ...
 }
 if (unmarked_exit) {
 ...
 return false;
 }
 }
 }
 }
 return true;
}

Peeling: the classic TurboFan peel

src/compiler/loop-peeling.cc peels the first iteration of the innermost loops. The idea, spelled out in the ASCII art at the top of the file, is that the loop body is duplicated: once as a “peeled iteration” that runs before the loop, and once as the loop itself. The checks in the peeled iteration - Smi checks, bounds checks, type checks

then happen once, outside the loop.

The peel is only allowed if the exits are marked, and only for small loops:

 cpp
 
 
// src/compiler/loop-peeling.cc:218
void LoopPeeler::PeelInnerLoops(LoopTree::Loop* loop) {
 // If the loop has nested loops, peel inside those.
 if (!loop->children().empty()) {
 for (LoopTree::Loop* inner_loop : loop->children()) {
 PeelInnerLoops(inner_loop);
 }
 return;
 }
 // Only peel small-enough loops.
 if (loop->TotalSize() > LoopPeeler::kMaxPeeledNodes) return;

and the peeled body is built with a NodeCopier, with the loop’s header nodes mapped to their entry values - the value they have on the way in to the loop, not the value the backedge gives them (src/compiler/loop-peeling.cc:125):

 cpp
 
 
 // Map the loop header nodes to their entry values.
 for (Node* node : loop_tree_->HeaderNodes(loop)) {
 copier.Insert(node, node->InputAt(kAssumedLoopEntryIndex));
 }

The rest of Peel is graph surgery: the peeled iteration’s exit becomes a Merge, the loop’s backedge input is rewired to the peeled iteration, and the loop’s exit markers become a Merge/Phi/EffectPhi pair. A loop with several backedges gets a Merge of the peeled iteration’s outputs, and the header phis are resized with ResizeMergeOrPhi.

Peeling in Turboshaft: hoisting the checks out

The modern backend has its own, cleaner peeling pass, in src/compiler/turboshaft/loop-peeling-reducer.h. Its goal is stated plainly:

 cpp
 
 
// src/compiler/turboshaft/loop-peeling-reducer.h:34
// LoopPeeling "peels" the first iteration of innermost loops (= it extracts the
// first iteration from the loop). The goal of this is mainly to hoist checks
// out of the loop (such as Smi-checks, type-checks, bound-checks, etc).

The mechanism is a CopyingPhase: the loop body is cloned twice, once for the peeled iteration and once for the remaining loop, and the backedge of the peeled copy is turned into a forward edge. The peeled iteration’s stack check is dropped outright (src/compiler/turboshaft/loop-peeling-reducer.h:78), because the loop’s own stack check covers the remaining iterations. The eligibility rules are simple: no inner loops, not too many operations (kMaxSizeForPeeling = 1000), and no nested peeling.

Proving a trip count

Now the unrolling. To unroll a loop you have to know how many times it runs, and the only way TurboFan is willing to know that is to prove it from the shape of the loop. StaticCanonicalForLoopMatcher in src/compiler/turboshaft/loop-unrolling-reducer.cc recognises exactly one shape:

 cpp
 
 
// src/compiler/turboshaft/loop-unrolling-reducer.h:85
class V8_EXPORT_PRIVATE StaticCanonicalForLoopMatcher {
 // In the context of this class, a "static canonical for-loop" is one of the
 // form `for (let i = cst; i cmp cst; i = i binop cst)`. That is, a fairly
 // simple for-loop, for which we can statically compute the number of
 // iterations.

The matcher works backwards from the loop header’s branch. It has to find, in order: a comparison of the loop’s phi against a constant; the phi’s forward edge being a constant; and the phi’s back edge being a binop that involves the phi and a constant. The for (i = 0; i < 42; i = i + 2) case is spelled out in the comments:

 cpp
 
 
// src/compiler/turboshaft/loop-unrolling-reducer.cc:157
 // We have: phi(..., ...) cmp_op cmp_cst
 // eg, for (i = ...; i < 42; ...)
 uint64_t phi_cst;
 if (matcher_.MatchUnsignedIntegralConstant(phi.forward_edge(), &phi_cst)) {
 // We have: phi(phi_cst, ...) cmp_op cmp_cst
 // eg, for (i = 0; i < 42; ...)
 V<Word> left, right;
 BinOp binop_op;
 WordRepresentation binop_rep;
 if (MatchWordBinop(phi.back_edge(), &left, &right, &binop_op, &binop_rep) ||
 MatchCheckedOverflowBinop(phi.back_edge(), &left, &right, &binop_op,
 &binop_rep)) {
 // We have: phi(phi_cst, phi binop_op ...) cmp_op cmp_cst
 // eg, for (i = 0; i < 42; i = ... + ...)
 if (left == phi_idx) {
 // We have: phi(phi_cst, phi binop_op binop_cst) cmp_op cmp_cst
 // eg, for (i = 0; i < 42; i = i + 2)

The i = 2 + i case is only accepted if the operation commutes, because the simulation assumes i = i op cst. Add, Mul and the bitwise ops commute; Sub does not.

The counting itself is a simulation, not a formula, because overflow makes a closed form messy:

 cpp
 
 
// src/compiler/turboshaft/loop-unrolling-reducer.cc:433
 // It's a bit hard to compute the number of iterations without some kind of
 // (simple) SMT solver, especially when taking overflows into account. Thus,
 // we just simulate the evolution of the loop counter: we repeatedly compute
 // `init binop_op binop_cst`, and compare the result with `max`. This is
 // somewhat inefficient, so it should only be done if `kMaxExactIter} is
 // small.
 DCHECK_LE(kMaxExactIter, 10);

 Int curr = init;
 size_t iter_count = 0;
 for (; iter_count < kMaxExactIter; iter_count++) {
 if (Cmp(curr, max, cmp_op) != loop_if_cond_is) {
 return IterationCount::Exact(iter_count);
 }

kMaxExactIter is 5, so the exact trip count is only computed for loops of at most 5 iterations. Anything longer gets an approximate count, computed by division, and approximate counts are good for exactly one thing: removing the stack check.

 cpp
 
 
// src/compiler/turboshaft/loop-unrolling-reducer.h:282
 static constexpr size_t kMaxLoopIterationsForFullUnrolling = 4;
 static constexpr size_t kMaxPartialUnrollingCount = 4;
 static constexpr size_t kMaxIterForStackCheckRemoval = 5000;

 cpp
 
 
// src/compiler/turboshaft/loop-unrolling-reducer.cc:41
 if (iter_count.IsSmallerThan(kMaxIterForStackCheckRemoval)) {
 stack_checks_to_remove_.insert(start->index().id());
 }

A loop that runs at most 4999 times cannot overflow the stack, so the per-iteration stack check can go. That is the under-appreciated win.

Unrolling: full, partial, and removal

With the trip count in hand, three things can happen. A loop that runs zero times is removed entirely - the header is inlined and its branch becomes a Goto (src/compiler/turboshaft/loop-unrolling-reducer.h:247). A loop with an exact count of at most 4 is fully unrolled - the body is cloned that many times, backedge and all. And everything else that is small enough gets partially unrolled - four copies of the body, with the original left as the loop:

 cpp
 
 
// src/compiler/turboshaft/loop-unrolling-reducer.h:196
 bool ShouldFullyUnrollLoop(const Block* loop_header) const {
 LoopFinder::LoopInfo header_info = loop_finder_.GetLoopInfo(loop_header);
 if (header_info.has_inner_loops) return false;
 if (header_info.op_count > kMaxLoopSizeForFullUnrolling) return false;

 auto iter_count = GetIterationCount(loop_header);
 return iter_count.IsExact() &&
 iter_count.exact_count() <= kMaxLoopIterationsForFullUnrolling;
 }

 bool ShouldPartiallyUnrollLoop(const Block* loop_header) const {
 LoopFinder::LoopInfo info = loop_finder_.GetLoopInfo(loop_header);
 return !info.has_inner_loops && !info.has_any_call &&
 info.op_count < kMaxLoopSizeForPartialUnrolling;
 }

The !info.has_any_call condition is the interesting one: a loop with a call in it is not partially unrolled, because the call’s side effects would be reordered with respect to the rest of the body.

The classic (pre-Turboshaft) unroller in src/compiler/loop-unrolling.cc does the same thing with a NodeCopier and a much more manual rewiring of the phis and the loop exits. It also does the stack-check trick in a more direct way: in every copy but the first, the stack check’s value uses are replaced with the constant 1 (src/compiler/loop-unrolling.cc:58).

The induction variable

Now the third optimisation, and the one with the most interesting machinery. src/compiler/loop-variable-optimizer.cc looks for a phi whose backedge is an add or subtract of the phi itself and a constant:

 cpp
 
 
// src/compiler/loop-variable-optimizer.cc:223
InductionVariable* LoopVariableOptimizer::TryGetInductionVariable(Node* phi) {
 DCHECK_EQ(2, phi->op()->ValueInputCount());
 Node* loop = NodeProperties::GetControlInput(phi);
 DCHECK_EQ(IrOpcode::kLoop, loop->opcode());
 Node* initial = phi->InputAt(0);
 Node* arith = phi->InputAt(1);
 InductionVariable::ArithmeticType arithmeticType;
 if (arith->opcode() == IrOpcode::kJSAdd ||
 arith->opcode() == IrOpcode::kNumberAdd ||
 arith->opcode() == IrOpcode::kSpeculativeNumberAdd ||
 arith->opcode() == IrOpcode::kSpeculativeAdditiveSafeIntegerAdd ||
 arith->opcode() == IrOpcode::kSpeculativeSmallIntegerAdd) {
 arithmeticType = InductionVariable::ArithmeticType::kAddition;
 } else if (arith->opcode() == IrOpcode::kJSSubtract ||
 arith->opcode() == IrOpcode::kSpeculativeSmallIntegerSubtract) {
 arithmeticType = InductionVariable::ArithmeticType::kSubtraction;
 } else {
 return nullptr;
 }

The list of accepted opcodes is long because the arithmetic can arrive in several shapes depending on how the feedback was collected.

The bounds are collected too. The optimizer walks the control flow and, for every comparison involving the induction variable, records an upper or lower bound. Then the phi is rewritten into an InductionVariablePhi with extra inputs - the increment and every bound:

 cpp
 
 
// src/compiler/loop-variable-optimizer.cc:290
void LoopVariableOptimizer::ChangeToInductionVariablePhis() {
 for (auto entry : induction_vars_) {
 // It only make sense to analyze the induction variables if
 // there is a bound.
 InductionVariable* induction_var = entry.second;
 if (induction_var->upper_bounds().empty() &&
 induction_var->lower_bounds().empty()) {
 continue;
 }
 // Insert the increment value to the value inputs.
 induction_var->phi()->InsertInput(graph()->zone(),
 induction_var->phi()->InputCount() - 1,
 induction_var->increment());
 // Insert the bound inputs to the value inputs.
 for (auto bound : induction_var->lower_bounds()) {
 induction_var->phi()->InsertInput(
 graph()->zone(), induction_var->phi()->InputCount() - 1, bound.bound);
 }
 for (auto bound : induction_var->upper_bounds()) {
 induction_var->phi()->InsertInput(
 graph()->zone(), induction_var->phi()->InputCount() - 1, bound.bound);
 }
 NodeProperties::ChangeOp(
 induction_var->phi(),
 common()->InductionVariablePhi(induction_var->phi()->InputCount() - 1));
 }
}

So the phi, which had two value inputs (initial, backedge), now has three or more. The extra inputs are pure decoration - the operator is kPure - but they carry the information the typer needs.

The prefixed point: a range type for the counter

With the bounds in hand, the typer can give the counter a range type instead of “some number”. TypeInductionVariablePhi in src/compiler/turbofan-typer.cc computes the range by walking the bounds:

 cpp
 
 
// src/compiler/turbofan-typer.cc:998
 if (increment_min >= 0) {
 // Increasing sequence.
 min = initial_type.Min();
 for (auto bound : induction_var->upper_bounds()) {
 Type bound_type = TypeOrNone(bound.bound);
 // If the type is not an integer, just skip the bound.
 if (!bound_type.Is(typer_->cache_->kInteger)) continue;
 double bound_max = bound_type.Max();
 if (bound.kind == InductionVariable::kStrict) {
 bound_max -= 1;
 }
 max = std::min(max, bound_max + increment_max);
 }

bound_max - 1 is the strictness: i < 10 means i never reaches 10. This is a “prefixed point” computation: the type of the counter is computed from the initial value and the bounds, not by iterating to a fixed point as ordinary phi typing would.

The result is then checked. After typing, the compiler asserts that the type is indeed a prefixed point - that the type of the phi contains the types of all its inputs - with a CHECK, not a DCHECK:

 cpp
 
 
// src/compiler/turbofan-typer.cc:484
 if (induction_vars != nullptr) {
 // Validate the types computed by TypeInductionVariablePhi.
 for (auto entry : induction_vars->induction_variables()) {
 InductionVariable* induction_var = entry.second;
 if (induction_var->phi()->opcode() == IrOpcode::kInductionVariablePhi) {
 CHECK(visitor.InductionVariablePhiTypeIsPrefixedPoint(induction_var));
 }
 }

This is a real correctness check on the optimiser. Getting the range wrong would be a miscompilation, not a missed optimisation.

The price: a guard on the backedge

Here is where the proof is paid for. The range type is only sound if the counter really does stay inside the bounds - an assumption about the runtime, not a fact about the graph. So the InductionVariablePhi is turned back into an ordinary Phi, and a TypeGuard is inserted on the backedge if the backedge value’s type is not already a subtype of the phi’s type:

 cpp
 
 
// src/compiler/loop-variable-optimizer.cc:320
void LoopVariableOptimizer::ChangeToPhisAndInsertGuards() {
 for (auto entry : induction_vars_) {
 InductionVariable* induction_var = entry.second;
 if (induction_var->phi()->opcode() == IrOpcode::kInductionVariablePhi) {
 // Turn the induction variable phi back to normal phi.
 int value_count = 2;
 Node* control = NodeProperties::GetControlInput(induction_var->phi());
 induction_var->phi()->TrimInputCount(value_count + 1);
 induction_var->phi()->ReplaceInput(value_count, control);
 NodeProperties::ChangeOp(
 induction_var->phi(),
 common()->Phi(MachineRepresentation::kTagged, value_count));

 // If the backedge is not a subtype of the phi's type, we insert a sigma
 // to get the typing right.
 Node* backedge_value = induction_var->phi()->InputAt(1);
 Type backedge_type = NodeProperties::GetType(backedge_value);
 Type phi_type = NodeProperties::GetType(induction_var->phi());
 if (!backedge_type.Is(phi_type)) {
 Node* loop = NodeProperties::GetControlInput(induction_var->phi());
 Node* backedge_control = loop->InputAt(1);
 Node* backedge_effect =
 NodeProperties::GetEffectInput(induction_var->effect_phi(), 1);
 Node* rename =
 graph()->NewNode(common()->TypeGuard(phi_type), backedge_value,
 backedge_effect, backedge_control);
 induction_var->effect_phi()->ReplaceInput(1, rename);
 induction_var->phi()->ReplaceInput(1, rename);
 }
 }
 }
}

That TypeGuard sits on the backedge, so it runs once per iteration. If the counter ends up outside the range the analysis computed, the guard fails and the frame is deoptimized. It costs one comparison and one branch per iteration, and it is the price of the range type.

The whole dance is orchestrated by the typer, which calls ChangeToInductionVariablePhis before typing and ChangeToPhisAndInsertGuards after it (src/compiler/turbofan-typer.cc:473-493). The induction variables themselves are found in the typer phase:

 cpp
 
 
// src/compiler/pipeline.cc:1049
 LoopVariableOptimizer induction_vars(data->jsgraph()->graph(),
 data->common(), temp_zone);
 if (v8_flags.turbo_loop_variable) induction_vars.Run();

and the turbo_loop_variable flag (default true) is what lets you turn the whole thing off.

OSR: the assumption you cannot check

The one place where TurboFan makes an assumption it cannot verify is OSR - on-stack replacement. When a hot loop is entered from the interpreter, TurboFan compiles a version of the function that starts in the middle of the loop, with the loop’s state taken from the interpreter’s frames. The entry point is the loop header that the bytecode analyser finds by scanning backwards for the JumpLoop that matches the OSR offset (src/compiler/bytecode-analysis.cc:553).

The graph builder then has to peel the outer loops to get there, because the OSR entry point is inside a nest of loops and the code has to start at the innermost one:

 cpp
 
 
// src/compiler/bytecode-graph-builder.cc:1396
void BytecodeGraphBuilder::AdvanceToOsrEntryAndPeelLoops() {
 environment()->FillWithOsrValues();

FillWithOsrValues is where the assumption is encoded: every live local in the environment is replaced by an OsrValue node (src/compiler/bytecode-graph-builder.cc:913), which reads the value out of the interpreter’s frame at the OSR entry point. OsrHelper then has to make room for the interpreter’s frame in the optimized frame:

 cpp
 
 
// src/compiler/osr.cc:21
void OsrHelper::SetupFrame(Frame* frame) {
 // The optimized frame will subsume the unoptimized frame. Do so by reserving
 // the first spill slots.
 frame->ReserveSpillSlots(UnoptimizedFrameSlots());
}

The assumptions here are: the interpreter’s loop state is exactly what the optimized code expects, and the loop will be entered at the header. The compiler cannot prove either. What it does instead is build the peeled iteration so that the first iteration of the OSR loop is compiled with the checks hoisted - and if an assumption is violated at runtime, the deoptimizer takes over. Same “guard instead of proof” trade as the induction variable’s TypeGuard, at a coarser granularity.

Trade-offs

What it buys. Peeled loops hoist their checks. Unrolled loops have fewer branches and better instruction scheduling. A counter with a range type lets the register allocator use a 32-bit integer instead of a double, and lets the typer prove an array access is in bounds.

What it costs.

Code size. Peeling duplicates the body. Unrolling multiplies it.
Compile time. The trip-count matcher, the induction variable optimizer and the typer’s fixed point are all paid on every compilation.
The CHECK on the prefixed point. Getting the range wrong would be a miscompilation, not just a missed optimisation.
Guards. Every range type is paid for with a TypeGuard on the backedge; every peeled loop with an extra copy of the body.
The loop-entry assumption. kAssumedLoopEntryIndex is a hard assumption about the shape of the control graph, with a TODO next to it.

The bottom line

TurboFan’s loop optimisations are all the same shape: recognise a pattern, compute a fact about it (trip count, counter range), and use the fact to remove work. The pattern recognition is deliberately narrow

one canonical for loop shape, one canonical induction variable shape - because a narrow pattern is a proof you can actually make.

The part worth remembering is the bookkeeping. A fact like “the counter is between 0 and 41” is not free: it has to be re-established at every point where the program could disagree. The induction variable does it with a TypeGuard on the backedge; OSR with a deoptimizer on the side; the trip count by not being used unless it is exact.

The thing people get wrong

People think of loop unrolling as “the compiler makes the loop body bigger so the CPU pipeline has more to chew on”. That is the 1990s version, and it is not what TurboFan does. TurboFan unrolls a loop only when it can prove the trip count, and the count it proves is at most 4. The far more common win is the opposite one: proving the count is small (under 5000) so that the stack check can be deleted.

The other misconception is that the range type on an induction variable is free. It is not. It is a claim about the runtime, re-asserted at the top of every iteration by a TypeGuard that will deoptimize the frame if the claim is ever wrong. The range type buys you a 32-bit register; the TypeGuard is what you pay for it.

Loops on Opensource internals