How CPython Implements Generator (types.GeneratorType) in C

A Python generator is, at the C level, a single object that is its own frame: the executable state of a paused function lives inside the object, so a generator can be suspended, resumed, thrown into, and closed from anywhere. In CPython 3.15 generators, coroutines and async generators share one implementation in Objects/genobject.c and one struct layout defined by _PyGenObject_HEAD.

This article traces that implementation.

The struct: `_PyGenObject`

The head macro is shared by all three flavours:

c
// Include/internal/pycore_interpframe_structs.h
#define _PyGenObject_HEAD(prefix)                                           \
    PyObject_HEAD                                                           \
    PyObject *prefix##_weakreflist;                                         \
    PyObject *prefix##_name;                                                \
    PyObject *prefix##_qualname;                                            \
    _PyErr_StackItem prefix##_exc_state;                                    \
    PyObject *prefix##_origin_or_finalizer;                                 \
    char prefix##_hooks_inited;                                             \
    char prefix##_closed;                                                   \
    char prefix##_running_async;                                            \
    int8_t prefix##_frame_state;                                            \
    _PyInterpreterFrame prefix##_iframe;

struct _PyGenObject { _PyGenObject_HEAD(gi) };
struct _PyCoroObject { _PyGenObject_HEAD(cr) };
struct _PyAsyncGenObject { _PyGenObject_HEAD(ag) };

So a PyGenObject is just PyObject_HEAD plus a weakref list, name/qualname, the saved exception state, a finalizer/origin pointer, a few flags, the frame state byte gi_frame_state, and an embedded _PyInterpreterFrame gi_iframe. The frame has owner == FRAME_OWNED_BY_GENERATOR, which is how _PyGen_GetGeneratorFromFrame recovers the owning generator from a frame pointer (it subtracts offsetof(PyGenObject, gi_iframe)).

Because the frame is embedded, a generator is one allocation: the object is the activation record. There is no separate stack frame to keep alive.

Creating a generator: `RETURN_GENERATOR`

A generator function is detected at compile time (CO_GENERATOR). When it is called, the interpreter runs the function body until the first RETURN_GENERATOR bytecode, which converts the in-flight frame into a generator object:

c
// Python/bytecodes.c
inst(RETURN_GENERATOR, (-- res)) {
    PyFunctionObject *func =
        (PyFunctionObject *)PyStackRef_AsPyObjectBorrow(frame->f_funcobj);
    PyGenObject *gen = (PyGenObject *)_Py_MakeCoro(func);
    ERROR_IF(gen == NULL);
    assert(STACK_LEVEL() <= 2);
    SAVE_STACK();
    _PyInterpreterFrame *gen_frame = &gen->gi_iframe;
    frame->instr_ptr++;
    _PyFrame_Copy(frame, gen_frame);          // copy locals + value stack
    assert(frame->frame_obj == NULL);
    gen->gi_frame_state = FRAME_CREATED;
    gen_frame->owner = FRAME_OWNED_BY_GENERATOR;
    _Py_LeaveRecursiveCallPy(tstate);
    _PyInterpreterFrame *prev = frame->previous;
    _PyThreadState_UpdateLastProfiledFrame(tstate, frame, prev);
    _PyThreadState_PopFrame(tstate, frame);
    frame = tstate->current_frame = prev;
    LOAD_IP(frame->return_offset);
    RELOAD_STACK();
    res = PyStackRef_FromPyObjectStealMortal((PyObject *)gen);
    ...
}

_Py_MakeCoro allocates the object (PyObject_GC_NewVar, sized for the frame’s locals+stack slots) and fills in gi_name/gi_qualname from the function. RETURN_GENERATOR then copies the current thread frame into gi_iframe with _PyFrame_Copy, sets gi_frame_state = FRAME_CREATED, and pops the original frame off the thread stack. From that point the generator owns its own frame; the caller just receives the generator object.

_PyFrame_Copy is worth a look because it is what makes “pausing” work: it copies the executable, globals, builtins, locals, instr_ptr, and the entire value stack into the embedded frame, and converts stack references to heap-safe ones (the generator may outlive the thread stack):

c
// Include/internal/pycore_interpframe.h
static inline void _PyFrame_Copy(_PyInterpreterFrame *src, _PyInterpreterFrame *dest)
{
    dest->f_executable = PyStackRef_MakeHeapSafe(src->f_executable);
    dest->previous = NULL;          // no dangling pointer to old frame
    dest->f_funcobj = PyStackRef_MakeHeapSafe(src->f_funcobj);
    dest->f_globals = src->f_globals;
    dest->f_builtins = src->f_builtins;
    dest->f_locals = src->f_locals;
    dest->frame_obj = src->frame_obj;
    dest->instr_ptr = src->instr_ptr;
    ...
    for (int i = 0; i < stacktop; i++) {
        dest->localsplus[i] = PyStackRef_MakeHeapSafe(src->localsplus[i]);
    }
}

Generator state

gi_frame_state is an int8_t holding one of:

c
// Include/internal/pycore_frame.h
typedef enum _framestate {
    FRAME_CREATED = 0,
    FRAME_SUSPENDED = 1,
    FRAME_SUSPENDED_YIELD_FROM = 2,
    FRAME_SUSPENDED_YIELD_FROM_LOCKED = 3,
    FRAME_EXECUTING = 4,
    FRAME_CLEARED = 5
} PyFrameState;

#define FRAME_STATE_SUSPENDED(S) ((S) >= FRAME_SUSPENDED && (S) <= FRAME_SUSPENDED_YIELD_FROM_LOCKED)
#define FRAME_STATE_FINISHED(S)  ((S) == FRAME_CLEARED)

Exposed to Python as gi_state (GEN_CREATED, GEN_SUSPENDED, GEN_RUNNING, GEN_CLOSED). All state transitions use atomic loads/stores and _Py_GEN_TRY_SET_FRAME_STATE, which is necessary for the free-threaded build.

`gi_code` and `gi_frame`

gi_code is not a stored field; it is derived from the embedded frame:

c
// Objects/genobject.c
static inline PyCodeObject *_PyGen_GetCode(PyGenObject *gen) {
    return _PyFrame_GetCode(&gen->gi_iframe);   // (PyCodeObject*)gi_iframe.f_executable
}

static PyObject *gen_getcode(PyObject *self, void *ignored) {
    PyGenObject *gen = _PyGen_CAST(self);
    return _gen_getcode(gen, "gi_code");        // Py_NewRef(_PyGen_GetCode(gen))
}

gi_frame is similarly lazy: _gen_getframe returns _PyFrame_GetFrameObject(&gen->gi_iframe) unless the generator is finished, in which case it returns None. The frame object is created on demand and cached in gi_iframe.frame_obj.

`send()`: `gen_send_ex` / `gen_send_ex2`

The public send()/__next__() path goes through gen_send_ex, which validates state and atomically moves CREATED/SUSPENDED -> EXECUTING, then calls gen_send_ex2:

c
// Objects/genobject.c
static PySendResult
gen_send_ex2(PyGenObject *gen, PyObject *arg, PyObject **presult, int exc)
{
    assert(FT_ATOMIC_LOAD_INT8_RELAXED(gen->gi_frame_state) == FRAME_EXECUTING);

    PyThreadState *tstate = _PyThreadState_GET();
    _PyInterpreterFrame *frame = &gen->gi_iframe;

    /* Push arg onto the frame's value stack */
    PyObject *arg_obj = arg ? arg : Py_None;
    _PyFrame_StackPush(frame, PyStackRef_FromPyObjectNew(arg_obj));

    _PyErr_StackItem *prev_exc_info = tstate->exc_info;
    gen->gi_exc_state.previous_item = prev_exc_info;
    tstate->exc_info = &gen->gi_exc_state;

    if (exc) {
        assert(_PyErr_Occurred(tstate));
        _PyErr_ChainStackItem();
    }

    EVAL_CALL_STAT_INC(EVAL_CALL_GENERATOR);
    PyObject *result = _PyEval_EvalFrame(tstate, frame, exc);
    ...

    int return_kind = ((_PyThreadStateImpl *)tstate)->generator_return_kind;

    if (return_kind == GENERATOR_YIELD) {
        assert(result != NULL && !_PyErr_Occurred(tstate));
        *presult = result;
        return PYGEN_NEXT;            // yielded
    }

    assert(return_kind == GENERATOR_RETURN);
    ...
    *presult = result;
    return result ? PYGEN_RETURN : PYGEN_ERROR;   // exhausted / error
}

Key points:

The sent value is pushed onto the generator’s own value stack as the result of the yield expression, via _PyFrame_StackPush.
Exception state is swapped in: tstate->exc_info points at gen->gi_exc_state for the duration of the call, and restored afterwards.
_PyEval_EvalFrame runs the embedded frame. When it hits YIELD_VALUE, ceval sets generator_return_kind = GENERATOR_YIELD and returns the yielded object; the frame stays EXECUTING until gen_send_ex2 finishes, then transitions to SUSPENDED.
On exhaustion, gi_frame_state becomes FRAME_CLEARED and gen_clear_frame releases locals.

gen_send_ex also guards against the obvious mistakes: sending a non-None value into a just-created generator ("can't send non-None value to a just-started generator"), re-entering an already-running generator, and reusing an exhausted coroutine.

`throw()`

_gen_throw reuses the same machinery. After validating state and moving to EXECUTING, it handles yield from delegation: if the generator is suspended inside a yield from, the thrown exception is forwarded to the sub-iterator (yf), possibly recursively. Otherwise it lands at throw_here, where the exception is restored into the current error indicator and gen_send_ex2 is called with exc=1:

c
// Objects/genobject.c
throw_here:
    assert(FT_ATOMIC_LOAD_INT8_RELAXED(gen->gi_frame_state) == FRAME_EXECUTING);
    if (gen_set_exception(typ, val, tb) < 0) { ... }
    return gen_throw_current_exception(gen);   // -> gen_send_ex2(gen, Py_None, &result, 1)

gen_set_exception normalizes the (type, value, traceback) triple into the current exception (calling PyErr_NormalizeException when given a class).

`close()`

gen_close is a state machine over gi_frame_state:

FRAME_CREATED: atomically move to FRAME_CLEARED, call gen_clear_frame, return None. The generator never ran.
FRAME_CLEARED (finished): return None.
FRAME_EXECUTING: raise “generator already executing”.
suspended: move to EXECUTING, and if suspended in yield from, close the sub-iterator first. Then, if the resume point is a plain RESUME after a yield with exception depth 1, it can clear the frame directly; otherwise it injects GeneratorExit and runs the generator to completion via gen_send_ex2(..., exc=1).

c
// Objects/genobject.c
if (err == 0) {
    PyErr_SetNone(PyExc_GeneratorExit);
}
PyObject *retval;
if (gen_send_ex2(gen, Py_None, &retval, 1) == PYGEN_RETURN) {
    // generator returned a value while closing
    return retval;
} else if (retval) {
    ... PyErr_SetString(PyExc_RuntimeError, "generator ignored GeneratorExit");
} ...
if (PyErr_ExceptionMatches(PyExc_GeneratorExit)) {
    PyErr_Clear();
    Py_RETURN_NONE;
}

A generator that catches GeneratorExit and yields again produces "generator ignored GeneratorExit".

Clearing and deallocation

gen_clear_frame runs once the state is FRAME_CLEARED:

c
// Objects/genobject.c
static void gen_clear_frame(PyGenObject *gen)
{
    assert(FT_ATOMIC_LOAD_INT8_RELAXED(gen->gi_frame_state) == FRAME_CLEARED);
    _PyInterpreterFrame *frame = &gen->gi_iframe;
    _PyThreadState_UpdateLastProfiledFrame(_PyThreadState_GET(), frame, frame->previous);
    frame->previous = NULL;
    _PyFrame_ClearExceptCode(frame);   // clear locals + value stack
    _PyErr_ClearExcState(&gen->gi_exc_state);
}

_PyFrame_ClearExceptCode clears locals and drops f_funcobj; if a PyFrameObject is still referenced it takes ownership by copying the embedded frame into the heap frame object (so the frame survives the generator).

gen_dealloc (called by the GC) makes sure the frame is cleared before freeing:

c
// Objects/genobject.c
if (gen->gi_frame_state != FRAME_CLEARED) {
    gen->gi_frame_state = FRAME_CLEARED;
    gen_clear_frame(gen);
}
assert(gen->gi_exc_state.exc_value == NULL);
PyStackRef_CLEAR(gen->gi_iframe.f_executable);
Py_CLEAR(gen->gi_name);
Py_CLEAR(gen->gi_qualname);
PyObject_GC_Del(gen);

gen_traverse keeps the GC honest: while the frame is not cleared it traverses gi_iframe (via _PyFrame_Traverse); when cleared it still visits the code object held by f_executable so a deferred reference keeps it alive.

gen_finalize (tp_finalize) handles unawaited coroutines by issuing a RuntimeWarning, and otherwise closes the generator. Async generators may instead invoke their ag_origin_or_finalizer.

`yield from` support

When a generator is suspended inside yield from, gi_frame_state is FRAME_SUSPENDED_YIELD_FROM and the sub-iterator sits on the value stack (slot 2 of localsplus-relative peek). gen_getyieldfrom reads it (locking the state to ..._LOCKED on the free-threaded build to avoid races), and close()/ throw() use it to delegate to the sub-iterator.

Key takeaways

A generator is one GC object whose payload includes an _PyInterpreterFrame; it is created by RETURN_GENERATOR, which copies the in-flight thread frame into the object and pops the original.
gi_frame_state (created / suspended / suspended-yield-from / executing / cleared) tracks execution, with atomic transitions for free-threading.
gi_code and gi_frame are derived from gi_iframe, not stored fields.
send() pushes the argument onto the generator’s value stack, swaps in gi_exc_state, runs _PyEval_EvalFrame, and classifies the result by generator_return_kind.
throw() forwards into a yield from sub-iterator or injects an exception and resumes.
close() injects GeneratorExit (or clears directly) and runs to completion.
Clearing releases locals and, if a frame object escaped, transfers ownership via take_ownership.

The struct: _PyGenObject

Creating a generator: RETURN_GENERATOR

Generator state

gi_code and gi_frame

send(): gen_send_ex / gen_send_ex2

throw()

close()