CPython Specializes Bytecode with Inline Caches: The Adaptive Interpreter

Jay Hubbert — Tue, 05 Aug 2025 09:00:00 +0000

TL;DR: CPython 3.11’s specializing adaptive interpreter (PEP 659) rewrites a function’s bytecode in place while it runs. Hot operations become type-specific versions of themselves, guarded by version tags. When the world changes — a class gains an attribute, a dict is mutated — the tag moves and the generic path takes over again.

By 3.15 this covers attribute access, global lookup, calls, binary operations, iteration, and more. Let’s trace the machinery: counters, quickening, specialization, and the version tags that guard the fast paths.

The shape of a cache

Bytecode is an array of 16-bit code units. Each code unit is a union:

 c
 
 
// Include/internal/pycore_structs.h
typedef union {
 uint16_t cache;
 struct {
 uint8_t code;
 uint8_t arg;
 } op;
 _Py_BackoffCounter counter;
} _Py_CODEUNIT;

So one 16-bit word is either an opcode/arg pair or a cache slot. An instruction occupies 1 + _PyOpcode_Caches[opcode] words, and the trailing words are its inline cache. _PyOpcode_Caches gives the count per opcode — 9 for LOAD_ATTR, 4 for LOAD_GLOBAL, 3 for CALL.

The first cache word is always a _Py_BackoffCounter:

 c
 
 
// Include/internal/pycore_backoff.h
#define BACKOFF_BITS 3
#define BACKOFF_MASK 7
#define MAX_BACKOFF 6
#define UNREACHABLE_BACKOFF 7

#define MAKE_VALUE_AND_BACKOFF(value, backoff) \
 ((uint16_t)(((value) << BACKOFF_BITS) | (backoff)))

A counter packs a 13-bit value and a 3-bit backoff. It triggers when the value is zero and the backoff isn’t the unreachable sentinel.

Exponential backoff

 c
 
 
// Include/internal/pycore_backoff.h
static const uint16_t value_and_backoff_next[] = {
 MAKE_VALUE_AND_BACKOFF(1, 1), // warmup
 MAKE_VALUE_AND_BACKOFF(6, 2),
 MAKE_VALUE_AND_BACKOFF(30, 3),
 MAKE_VALUE_AND_BACKOFF(126, 4),
 MAKE_VALUE_AND_BACKOFF(508, 5),
 MAKE_VALUE_AND_BACKOFF(2052, 6),
 MAKE_VALUE_AND_BACKOFF(8190, 6),
 MAKE_VALUE_AND_BACKOFF(8190, 6),
};

static inline bool backoff_counter_triggers(_Py_BackoffCounter counter) {
 return counter.value_and_backoff < UNREACHABLE_BACKOFF;
}

The first execution after quickening has value 1, backoff 1, so it triggers on the second execution.

After a specialization failure, the counter restarts at the next backoff level, so retries get less and less frequent. After a success, the counter is set to “cooldown” (value 52, backoff 0) — roughly 53 executions before the next attempt after a miss.

That’s the anti-thrash mechanism: code that never stabilizes doesn’t get hammered with re-specialization attempts.

Quickening: one-time setup

When a code object is created, _PyCode_Quicken initializes the cache slots but does not specialize yet:

 c
 
 
// Python/specialize.c
void _PyCode_Quicken(_Py_CODEUNIT *instructions, Py_ssize_t size,
 int enable_counters, int flags) {
 ...
 for (Py_ssize_t i = 0; i < size-1; i++) {
 opcode = instructions[i].op.code;
 int caches = _PyOpcode_Caches[opcode];
 if (caches) {
 switch (opcode) {
 case JUMP_BACKWARD:
 instructions[i + 1].counter = jump_counter;
 break;
 case RESUME:
 instructions[i + 1].counter = resume_counter;
 break;
 case POP_JUMP_IF_FALSE:
 case POP_JUMP_IF_TRUE:
 ...
 instructions[i + 1].cache = 0x5555; // branch history
 break;
 default:
 instructions[i + 1].counter = adaptive_counter;
 break;
 }
 i += caches;
 }
 }
}

If specialization is disabled, counters are set to the unreachable backoff, so backoff_counter_triggers is always false and _SPECIALIZE_* never fires.

And POP_JUMP_* caches start as 0x5555 (alternating bits), updated at runtime as branch-history bitsets for the Tier 2 optimizer.

The generic opcode gate

Every cache-bearing base opcode contains a specializing micro-op:

 c
 
 
// Python/bytecodes.c (LOAD_ATTR)
specializing op(_SPECIALIZE_LOAD_ATTR, (counter/1, owner -- owner)) {
 #if ENABLE_SPECIALIZATION
 if (ADAPTIVE_COUNTER_TRIGGERS(counter)) {
 PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1);
 next_instr = this_instr;
 _Py_Specialize_LoadAttr(owner, next_instr, name);
 DISPATCH_SAME_OPARG();
 }
 OPCODE_DEFERRED_INC(LOAD_ATTR);
 ADVANCE_ADAPTIVE_COUNTER(this_instr[1].counter);
 #endif
}

So each execution of a generic op either calls the specializer — which may rewrite the opcode in place and re-dispatch — or advances the counter and runs the generic implementation.

Rewriting the opcode safely

 c
 
 
// Python/specialize.c
static inline int set_opcode(_Py_CODEUNIT *instr, uint8_t opcode) {
#ifdef Py_GIL_DISABLED
 uint8_t old_op = _Py_atomic_load_uint8_relaxed(&instr->op.code);
 if (old_op >= MIN_INSTRUMENTED_OPCODE) {
 return 0; // lost race with instrumentation
 }
 if (!_Py_atomic_compare_exchange_uint8(&instr->op.code, &old_op, opcode)) {
 assert(old_op >= MIN_INSTRUMENTED_OPCODE);
 return 0;
 }
 return 1;
#else
 instr->op.code = opcode;
 return 1;
#endif
}

static inline void specialize(_Py_CODEUNIT *instr, uint8_t specialized_opcode) {
 if (!set_opcode(instr, specialized_opcode)) { ...; return; }
 set_counter((_Py_BackoffCounter *)instr + 1, adaptive_counter_cooldown());
}

static inline void unspecialize(_Py_CODEUNIT *instr) {
 uint8_t opcode = FT_ATOMIC_LOAD_UINT8_RELAXED(instr->op.code);
 uint8_t generic_opcode = _PyOpcode_Deopt[opcode];
 if (!set_opcode(instr, generic_opcode)) { ...; return; }
 _Py_BackoffCounter *counter = (_Py_BackoffCounter *)instr + 1;
 _Py_BackoffCounter cur = load_counter(counter);
 set_counter(counter, adaptive_counter_backoff(cur));
}

set_opcode uses an atomic compare-exchange in the free-threaded build, so specialization races safely with instrumentation. specialize() flips the opcode and resets the counter to cooldown. unspecialize() maps any specialized opcode back to its base via _PyOpcode_Deopt and increases backoff.

Cache structs and unaligned access

Cache structs put the counter first, then fields packed into 16-bit words:

 c
 
 
// Include/internal/pycore_code.h
typedef struct {
 _Py_BackoffCounter counter;
 uint16_t module_keys_version;
 uint16_t builtin_keys_version;
 uint16_t index;
} _PyLoadGlobalCache; // 4 words

typedef struct {
 _Py_BackoffCounter counter;
 uint16_t version[2]; // type version as uint32
 union {
 uint16_t keys_version[2];
 uint16_t dict_offset;
 };
 uint16_t descr[4]; // descriptor pointer packed as 4 words
} _PyLoadMethodCache; // 9 words

Fields may be unaligned (a uint32 or pointer straddling 16-bit slots), so CPython uses memcpy-based helpers that compilers lower to plain moves:

 c
 
 
// Include/internal/pycore_code.h
static inline void write_u32(uint16_t *p, uint32_t val) {
 memcpy(p, &val, sizeof(val));
}
static inline uint32_t read_u32(uint16_t *p) {
 uint32_t val;
 memcpy(&val, p, sizeof(val));
 return val;
}
static inline PyObject *read_obj(uint16_t *p) {
 PyObject *val;
 memcpy(&val, p, sizeof(val));
 return val;
}

Version tags: the guards

A specialized op starts with guards. Example for LOAD_ATTR:

 c
 
 
// Python/generated_cases.c.h
uint32_t type_version = read_u32(&this_instr[2].cache);
PyTypeObject *tp = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
 UPDATE_MISS_STATS(LOAD_ATTR);
 JUMP_TO_PREDICTED(LOAD_ATTR); // deopt to generic
}

Three version tags are used:

Type version (tp_version_tag): bumped by PyType_Modified, which sets it to 0 for the type and all subclasses and notifies type watchers. Version 0 means “invalid”, and the specializer refuses it.
Dict keys version (dk_version in PyDictKeysObject): mutations set it to 0, forcing respecialization. Used for module globals/builtins in LOAD_GLOBAL.
Function version (func_version in PyFunctionObject): set once from co->co_version; reset to FUNC_VERSION_CLEARED (0) if __code__, __defaults__, __kwdefaults__, or __closure__ changes — permanently disabling specialization for that function. Used for calls.

This is the part people get wrong. The cache isn’t invalidated by clearing it; it’s invalidated by moving a version number. The cached value stays stale in memory, but the comparison fails, so correctness never depends on eagerly wiping anything.

On a guard miss, JUMP_TO_PREDICTED(LOAD_ATTR) lands at the predicted generic implementation at the same instruction, so the specialization gate runs again with the cooldown counter. After ~53 misses the counter triggers and the specializer retries.

What gets specialized

Families in 3.15 include LOAD_ATTR, LOAD_GLOBAL, BINARY_OP, CALL, FOR_ITER, CONTAINS_OP, COMPARE_OP, STORE_ATTR, STORE_SUBSCR, UNPACK_SEQUENCE, TO_BOOL, SEND, LOAD_SUPER_ATTR, GET_ITER, RESUME, JUMP_BACKWARD, CALL_KW, and CALL_FUNCTION_EX.

Each family is a base opcode plus specialized variants that all deopt back to it. The family’s cache size is the maximum of its members’ caches, so all variants share one layout.

Specialization conditions are strict. LOAD_GLOBAL requires globals/builtins to be exact dicts with unicode keys, the name to be found, indices and keys versions to fit in uint16, and no lazy imports. CALL requires the callable to use _PyFunction_Vectorcall (not a replaced vectorcall). RESUME only specializes when not tracing.

Why this is fast

The common case after specialization is a type-tag comparison, a dict-version comparison, and a direct slot read — no hashing, no descriptor search, no tuple building.

When the world changes (a class gains an attribute, a dict is mutated, a function is rebound), the version tag invalidates the cache and the generic path runs again, re-specializing when things stabilize.

The cost of the slow path is paid only on misses, and the backoff machinery keeps the interpreter from thrashing on code that never stabilizes. That’s the whole trade: a few extra instructions in the guard, an order-of-magnitude cheaper operation in the body.

Bytecode on Opensource internals