Generator on Opensource internals

How CPython Implements Generator (types.GeneratorType) in C

Jay Hubbert — Tue, 20 Oct 2026 09:00:00 +0000

A Python generator is, at the C level, a single object that is its own frame: the executable state of a paused function lives inside the object, so a generator can be suspended, resumed, thrown into, and closed from anywhere. In CPython 3.15 generators, coroutines and async generators share one implementation in Objects/genobject.c and one struct layout defined by _PyGenObject_HEAD.

This article traces that implementation.

The struct: `_PyGenObject`

The head macro is shared by all three flavours:

 c
 
 
// Include/internal/pycore_interpframe_structs.h
#define _PyGenObject_HEAD(prefix) \
 PyObject_HEAD \
 PyObject *prefix##_weakreflist; \
 PyObject *prefix##_name; \
 PyObject *prefix##_qualname; \
 _PyErr_StackItem prefix##_exc_state; \
 PyObject *prefix##_origin_or_finalizer; \
 char prefix##_hooks_inited; \
 char prefix##_closed; \
 char prefix##_running_async; \
 int8_t prefix##_frame_state; \
 _PyInterpreterFrame prefix##_iframe;

struct _PyGenObject { _PyGenObject_HEAD(gi) };
struct _PyCoroObject { _PyGenObject_HEAD(cr) };
struct _PyAsyncGenObject { _PyGenObject_HEAD(ag) };

So a PyGenObject is just PyObject_HEAD plus a weakref list, name/qualname, the saved exception state, a finalizer/origin pointer, a few flags, the frame state byte gi_frame_state, and an embedded _PyInterpreterFrame gi_iframe. The frame has owner == FRAME_OWNED_BY_GENERATOR, which is how _PyGen_GetGeneratorFromFrame recovers the owning generator from a frame pointer (it subtracts offsetof(PyGenObject, gi_iframe)).

Because the frame is embedded, a generator is one allocation: the object is the activation record. There is no separate stack frame to keep alive.

Creating a generator: `RETURN_GENERATOR`

A generator function is detected at compile time (CO_GENERATOR). When it is called, the interpreter runs the function body until the first RETURN_GENERATOR bytecode, which converts the in-flight frame into a generator object:

 c
 
 
// Python/bytecodes.c
inst(RETURN_GENERATOR, (-- res)) {
 PyFunctionObject *func =
 (PyFunctionObject *)PyStackRef_AsPyObjectBorrow(frame->f_funcobj);
 PyGenObject *gen = (PyGenObject *)_Py_MakeCoro(func);
 ERROR_IF(gen == NULL);
 assert(STACK_LEVEL() <= 2);
 SAVE_STACK();
 _PyInterpreterFrame *gen_frame = &gen->gi_iframe;
 frame->instr_ptr++;
 _PyFrame_Copy(frame, gen_frame); // copy locals + value stack
 assert(frame->frame_obj == NULL);
 gen->gi_frame_state = FRAME_CREATED;
 gen_frame->owner = FRAME_OWNED_BY_GENERATOR;
 _Py_LeaveRecursiveCallPy(tstate);
 _PyInterpreterFrame *prev = frame->previous;
 _PyThreadState_UpdateLastProfiledFrame(tstate, frame, prev);
 _PyThreadState_PopFrame(tstate, frame);
 frame = tstate->current_frame = prev;
 LOAD_IP(frame->return_offset);
 RELOAD_STACK();
 res = PyStackRef_FromPyObjectStealMortal((PyObject *)gen);
 ...
}

_Py_MakeCoro allocates the object (PyObject_GC_NewVar, sized for the frame’s locals+stack slots) and fills in gi_name/gi_qualname from the function. RETURN_GENERATOR then copies the current thread frame into gi_iframe with _PyFrame_Copy, sets gi_frame_state = FRAME_CREATED, and pops the original frame off the thread stack. From that point the generator owns its own frame; the caller just receives the generator object.

_PyFrame_Copy is worth a look because it is what makes “pausing” work: it copies the executable, globals, builtins, locals, instr_ptr, and the entire value stack into the embedded frame, and converts stack references to heap-safe ones (the generator may outlive the thread stack):

 c
 
 
// Include/internal/pycore_interpframe.h
static inline void _PyFrame_Copy(_PyInterpreterFrame *src, _PyInterpreterFrame *dest)
{
 dest->f_executable = PyStackRef_MakeHeapSafe(src->f_executable);
 dest->previous = NULL; // no dangling pointer to old frame
 dest->f_funcobj = PyStackRef_MakeHeapSafe(src->f_funcobj);
 dest->f_globals = src->f_globals;
 dest->f_builtins = src->f_builtins;
 dest->f_locals = src->f_locals;
 dest->frame_obj = src->frame_obj;
 dest->instr_ptr = src->instr_ptr;
 ...
 for (int i = 0; i < stacktop; i++) {
 dest->localsplus[i] = PyStackRef_MakeHeapSafe(src->localsplus[i]);
 }
}

Generator state

gi_frame_state is an int8_t holding one of:

 c
 
 
// Include/internal/pycore_frame.h
typedef enum _framestate {
 FRAME_CREATED = 0,
 FRAME_SUSPENDED = 1,
 FRAME_SUSPENDED_YIELD_FROM = 2,
 FRAME_SUSPENDED_YIELD_FROM_LOCKED = 3,
 FRAME_EXECUTING = 4,
 FRAME_CLEARED = 5
} PyFrameState;

#define FRAME_STATE_SUSPENDED(S) ((S) >= FRAME_SUSPENDED && (S) <= FRAME_SUSPENDED_YIELD_FROM_LOCKED)
#define FRAME_STATE_FINISHED(S) ((S) == FRAME_CLEARED)

Exposed to Python as gi_state (GEN_CREATED, GEN_SUSPENDED, GEN_RUNNING, GEN_CLOSED). All state transitions use atomic loads/stores and _Py_GEN_TRY_SET_FRAME_STATE, which is necessary for the free-threaded build.

`gi_code` and `gi_frame`

gi_code is not a stored field; it is derived from the embedded frame:

 c
 
 
// Objects/genobject.c
static inline PyCodeObject *_PyGen_GetCode(PyGenObject *gen) {
 return _PyFrame_GetCode(&gen->gi_iframe); // (PyCodeObject*)gi_iframe.f_executable
}

static PyObject *gen_getcode(PyObject *self, void *ignored) {
 PyGenObject *gen = _PyGen_CAST(self);
 return _gen_getcode(gen, "gi_code"); // Py_NewRef(_PyGen_GetCode(gen))
}

gi_frame is similarly lazy: _gen_getframe returns _PyFrame_GetFrameObject(&gen->gi_iframe) unless the generator is finished, in which case it returns None. The frame object is created on demand and cached in gi_iframe.frame_obj.

`send()`: `gen_send_ex` / `gen_send_ex2`

The public send()/__next__() path goes through gen_send_ex, which validates state and atomically moves CREATED/SUSPENDED -> EXECUTING, then calls gen_send_ex2:

 c
 
 
// Objects/genobject.c
static PySendResult
gen_send_ex2(PyGenObject *gen, PyObject *arg, PyObject **presult, int exc)
{
 assert(FT_ATOMIC_LOAD_INT8_RELAXED(gen->gi_frame_state) == FRAME_EXECUTING);

 PyThreadState *tstate = _PyThreadState_GET();
 _PyInterpreterFrame *frame = &gen->gi_iframe;

 /* Push arg onto the frame's value stack */
 PyObject *arg_obj = arg ? arg : Py_None;
 _PyFrame_StackPush(frame, PyStackRef_FromPyObjectNew(arg_obj));

 _PyErr_StackItem *prev_exc_info = tstate->exc_info;
 gen->gi_exc_state.previous_item = prev_exc_info;
 tstate->exc_info = &gen->gi_exc_state;

 if (exc) {
 assert(_PyErr_Occurred(tstate));
 _PyErr_ChainStackItem();
 }

 EVAL_CALL_STAT_INC(EVAL_CALL_GENERATOR);
 PyObject *result = _PyEval_EvalFrame(tstate, frame, exc);
 ...

 int return_kind = ((_PyThreadStateImpl *)tstate)->generator_return_kind;

 if (return_kind == GENERATOR_YIELD) {
 assert(result != NULL && !_PyErr_Occurred(tstate));
 *presult = result;
 return PYGEN_NEXT; // yielded
 }

 assert(return_kind == GENERATOR_RETURN);
 ...
 *presult = result;
 return result ? PYGEN_RETURN : PYGEN_ERROR; // exhausted / error
}

Key points:

The sent value is pushed onto the generator’s own value stack as the result of the yield expression, via _PyFrame_StackPush.
Exception state is swapped in: tstate->exc_info points at gen->gi_exc_state for the duration of the call, and restored afterwards.
_PyEval_EvalFrame runs the embedded frame. When it hits YIELD_VALUE, ceval sets generator_return_kind = GENERATOR_YIELD and returns the yielded object; the frame stays EXECUTING until gen_send_ex2 finishes, then transitions to SUSPENDED.
On exhaustion, gi_frame_state becomes FRAME_CLEARED and gen_clear_frame releases locals.

gen_send_ex also guards against the obvious mistakes: sending a non-None value into a just-created generator ("can't send non-None value to a just-started generator"), re-entering an already-running generator, and reusing an exhausted coroutine.

`throw()`

_gen_throw reuses the same machinery. After validating state and moving to EXECUTING, it handles yield from delegation: if the generator is suspended inside a yield from, the thrown exception is forwarded to the sub-iterator (yf), possibly recursively. Otherwise it lands at throw_here, where the exception is restored into the current error indicator and gen_send_ex2 is called with exc=1:

 c
 
 
// Objects/genobject.c
throw_here:
 assert(FT_ATOMIC_LOAD_INT8_RELAXED(gen->gi_frame_state) == FRAME_EXECUTING);
 if (gen_set_exception(typ, val, tb) < 0) { ... }
 return gen_throw_current_exception(gen); // -> gen_send_ex2(gen, Py_None, &result, 1)

gen_set_exception normalizes the (type, value, traceback) triple into the current exception (calling PyErr_NormalizeException when given a class).

`close()`

gen_close is a state machine over gi_frame_state:

FRAME_CREATED: atomically move to FRAME_CLEARED, call gen_clear_frame, return None. The generator never ran.
FRAME_CLEARED (finished): return None.
FRAME_EXECUTING: raise “generator already executing”.
suspended: move to EXECUTING, and if suspended in yield from, close the sub-iterator first. Then, if the resume point is a plain RESUME after a yield with exception depth 1, it can clear the frame directly; otherwise it injects GeneratorExit and runs the generator to completion via gen_send_ex2(..., exc=1).

 c
 
 
// Objects/genobject.c
if (err == 0) {
 PyErr_SetNone(PyExc_GeneratorExit);
}
PyObject *retval;
if (gen_send_ex2(gen, Py_None, &retval, 1) == PYGEN_RETURN) {
 // generator returned a value while closing
 return retval;
} else if (retval) {
 ... PyErr_SetString(PyExc_RuntimeError, "generator ignored GeneratorExit");
} ...
if (PyErr_ExceptionMatches(PyExc_GeneratorExit)) {
 PyErr_Clear();
 Py_RETURN_NONE;
}

A generator that catches GeneratorExit and yields again produces "generator ignored GeneratorExit".

Clearing and deallocation

gen_clear_frame runs once the state is FRAME_CLEARED:

 c
 
 
// Objects/genobject.c
static void gen_clear_frame(PyGenObject *gen)
{
 assert(FT_ATOMIC_LOAD_INT8_RELAXED(gen->gi_frame_state) == FRAME_CLEARED);
 _PyInterpreterFrame *frame = &gen->gi_iframe;
 _PyThreadState_UpdateLastProfiledFrame(_PyThreadState_GET(), frame, frame->previous);
 frame->previous = NULL;
 _PyFrame_ClearExceptCode(frame); // clear locals + value stack
 _PyErr_ClearExcState(&gen->gi_exc_state);
}

_PyFrame_ClearExceptCode clears locals and drops f_funcobj; if a PyFrameObject is still referenced it takes ownership by copying the embedded frame into the heap frame object (so the frame survives the generator).

gen_dealloc (called by the GC) makes sure the frame is cleared before freeing:

 c
 
 
// Objects/genobject.c
if (gen->gi_frame_state != FRAME_CLEARED) {
 gen->gi_frame_state = FRAME_CLEARED;
 gen_clear_frame(gen);
}
assert(gen->gi_exc_state.exc_value == NULL);
PyStackRef_CLEAR(gen->gi_iframe.f_executable);
Py_CLEAR(gen->gi_name);
Py_CLEAR(gen->gi_qualname);
PyObject_GC_Del(gen);

gen_traverse keeps the GC honest: while the frame is not cleared it traverses gi_iframe (via _PyFrame_Traverse); when cleared it still visits the code object held by f_executable so a deferred reference keeps it alive.

gen_finalize (tp_finalize) handles unawaited coroutines by issuing a RuntimeWarning, and otherwise closes the generator. Async generators may instead invoke their ag_origin_or_finalizer.

`yield from` support

When a generator is suspended inside yield from, gi_frame_state is FRAME_SUSPENDED_YIELD_FROM and the sub-iterator sits on the value stack (slot 2 of localsplus-relative peek). gen_getyieldfrom reads it (locking the state to ..._LOCKED on the free-threaded build to avoid races), and close()/ throw() use it to delegate to the sub-iterator.

Key takeaways

A generator is one GC object whose payload includes an _PyInterpreterFrame; it is created by RETURN_GENERATOR, which copies the in-flight thread frame into the object and pops the original.
gi_frame_state (created / suspended / suspended-yield-from / executing / cleared) tracks execution, with atomic transitions for free-threading.
gi_code and gi_frame are derived from gi_iframe, not stored fields.
send() pushes the argument onto the generator’s value stack, swaps in gi_exc_state, runs _PyEval_EvalFrame, and classifies the result by generator_return_kind.
throw() forwards into a yield from sub-iterator or injects an exception and resumes.
close() injects GeneratorExit (or clears directly) and runs to completion.
Clearing releases locals and, if a frame object escaped, transfers ownership via take_ownership.

How CPython Implements async generators in C

Jay Hubbert — Tue, 20 Oct 2026 09:00:00 +0000

Python’s async def functions containing yield become async generators: objects that implement __anext__, asend, athrow and aclose. Unlike a regular generator, an async generator is also an awaitable: calling anext(ag) does not run code immediately, it returns an async_generator_asend awaitable that is driven by the event loop. In CPython 3.15 all of this lives in Objects/genobject.c (there is no separate asyncgenobject.c), sharing the same interpreter-frame machinery as generators and coroutines.

The object struct

PyAsyncGenObject is defined through the shared _PyGenObject_HEAD macro, so its layout is identical to a generator’s except for the ag_ prefix:

 c
 
 
// Include/internal/pycore_interpframe_structs.h
struct _PyAsyncGenObject {
 PyObject_HEAD
 PyObject *ag_weakreflist;
 PyObject *ag_name;
 PyObject *ag_qualname;
 _PyErr_StackItem ag_exc_state;
 PyObject *ag_origin_or_finalizer;
 char ag_hooks_inited;
 char ag_closed;
 char ag_running_async;
 int8_t ag_frame_state;
 _PyInterpreterFrame ag_iframe;
};

The two async-specific fields are ag_closed (set once the generator has received GeneratorExit) and ag_origin_or_finalizer, which doubles as storage for the thread’s async-generator finalizer hook. The frame itself is the same _PyInterpreterFrame used by every other callable, so an async generator reuses the normal eval loop and yield/YIELD_VALUE machinery. PyAsyncGen_New allocates through gen_new_with_qualname and then initializes the async fields:

 c
 
 
// Objects/genobject.c
PyObject *PyAsyncGen_New(PyFrameObject *f, PyObject *name, PyObject *qualname) {
 PyAsyncGenObject *ag;
 ag = (PyAsyncGenObject *)gen_new_with_qualname(&PyAsyncGen_Type, f,
 name, qualname);
 ...
 ag->ag_origin_or_finalizer = NULL;
 ag->ag_closed = 0;
 ag->ag_hooks_inited = 0;
 ag->ag_running_async = 0;
 return (PyObject*)ag;
}

`ag_code`

Like gi_code/cr_code, ag_code is a read-only attribute backed by the frame’s executable. ag_getcode just forwards to _gen_getcode, which returns a new reference to _PyGen_GetCode(gen) (i.e. _PyFrame_GetCode(&gen->ag_iframe)):

 c
 
 
// Objects/genobject.c
static PyObject *
ag_getcode(PyObject *gen, void *Py_UNUSED(ignored)) {
 return _gen_getcode((PyGenObject*)gen, "ag_code");
}

static PyObject *
_gen_getcode(PyGenObject *gen, const char *const name) {
 if (PySys_Audit("object.__getattr__", "Os", gen, name) < 0) {
 return NULL;
 }
 return Py_NewRef(_PyGen_GetCode(gen));
}

It is registered in async_gen_getsetlist as {"ag_code", ag_getcode, NULL, NULL}. Because the code object is reachable through the (possibly cleared) frame, ag_code remains available even after the generator is closed.

Async generator hooks

CPython exposes two per-thread hooks, sys.set_asyncgen_firstiter and sys.set_asyncgen_finalizer, stored on the thread state as tstate->async_gen_firstiter and tstate->async_gen_finalizer (see Python/ceval.c). The firstiter hook is invoked the first time an async generator is iterated; the finalizer is stored on the object for later use.

Hook initialization is lazy and guarded by ag_hooks_inited, so it happens on the first call to anext/asend/aclose/athrow:

 c
 
 
// Objects/genobject.c
static int async_gen_init_hooks(PyAsyncGenObject *o) {
 PyThreadState *tstate;
 PyObject *finalizer;
 PyObject *firstiter;

 if (o->ag_hooks_inited) return 0;
 o->ag_hooks_inited = 1;

 tstate = _PyThreadState_GET();

 finalizer = tstate->async_gen_finalizer;
 if (finalizer) {
 o->ag_origin_or_finalizer = Py_NewRef(finalizer);
 }

 firstiter = tstate->async_gen_firstiter;
 if (firstiter) {
 PyObject *res;
 Py_INCREF(firstiter);
 res = PyObject_CallOneArg(firstiter, (PyObject *)o);
 Py_DECREF(firstiter);
 if (res == NULL) return 1; /* propagate exception */
 Py_DECREF(res);
 }
 return 0;
}

Note the ordering: the finalizer is stored (ag_origin_or_finalizer), while the firstiter callback is called immediately and its result discarded. If the firstiter callback raises, the error propagates out of anext() etc.

`anext`/`asend`/`athrow` are awaitables

None of these run the generator directly. async_gen_anext returns an async_generator_asend object (with sendval == NULL), and async_gen_asend wraps the supplied argument. These awaitables (PyAsyncGenASend) carry state (AWAITABLE_STATE_INIT / _ITER / _CLOSED) plus the generator and the value to send. Driving them via .send()/__next__() calls gen_send, then async_gen_unwrap_value converts a wrapped yield value into a StopIteration:

 c
 
 
// Objects/genobject.c
static PyObject *async_gen_unwrap_value(PyAsyncGenObject *gen, PyObject *result) {
 if (result == NULL) {
 if (!PyErr_Occurred()) PyErr_SetNone(PyExc_StopAsyncIteration);
 if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration) ||
 PyErr_ExceptionMatches(PyExc_GeneratorExit)) {
 gen->ag_closed = 1;
 }
 gen->ag_running_async = 0;
 return NULL;
 }
 if (_PyAsyncGenWrappedValue_CheckExact(result)) {
 _PyGen_SetStopIterationValue(((_PyAsyncGenWrappedValue*)result)->agw_val);
 Py_DECREF(result);
 gen->ag_running_async = 0;
 return NULL; /* signals "yielded a value" via StopIteration */
 }
 return result;
}

ag_running_async guards re-entrancy: while a generator is running, a second __anext__() raises “asynchronous generator is already running”.

`aclose`

async_gen_aclose is implemented as athrow(GeneratorExit) in “aclose mode” (agt_typ == NULL). It creates an async_generator_athrow awaitable and, on first send, marks the generator closed and throws GeneratorExit with close_on_genexit = 0 (so a suspended yield from is not auto-closed immediately, allowing the async generator to clean up via awaits):

 c
 
 
// Objects/genobject.c (async_gen_athrow_send, aclose mode)
if (o->agt_typ == NULL) {
 /* aclose() mode */
 o->agt_gen->ag_closed = 1;
 retval = _gen_throw((PyGenObject *)gen, 0,
 PyExc_GeneratorExit, NULL, NULL);
 if (retval && _PyAsyncGenWrappedValue_CheckExact(retval)) {
 Py_DECREF(retval);
 goto yield_close; /* generator yielded during close -> RuntimeError */
 }
}

If the generator yields a value while being closed, CPython sets ASYNC_GEN_IGNORED_EXIT_MSG (“async generator ignored GeneratorExit”) and raises RuntimeError. Otherwise, on completion, StopAsyncIteration/GeneratorExit are swallowed and converted to StopIteration, signalling that the aclose() awaitable is done:

 c
 
 
// Objects/genobject.c
if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration) ||
 PyErr_ExceptionMatches(PyExc_GeneratorExit)) {
 if (o->agt_typ == NULL) {
 PyErr_Clear();
 PyErr_SetNone(PyExc_StopIteration);
 }
}

The ag_closed flag is also consulted by subsequent asend/athrow calls, which then immediately raise StopAsyncIteration without running the body again.

Finalization

gen_finalize (registered as tp_finalize) handles unawaited async generators: if the frame is not finished and ag_closed is false, it invokes the stored finalizer (ag_origin_or_finalizer) so that unclosed async generators are reported (and typically closed) rather than silently leaked.

Summary

PyAsyncGenObject shares _PyGenObject_HEAD(ag) with generators; async-specific state is ag_closed, ag_running_async, ag_hooks_inited and ag_origin_or_finalizer.
ag_code is a read-only getset returning the frame’s code object.
Hooks are initialized lazily on first iteration: the finalizer is stored, firstiter is called immediately.
anext/asend/athrow return awaitables (PyAsyncGenASend / PyAsyncGenAThrow) that drive the generator via gen_send/gen_throw.
aclose is athrow(GeneratorExit) in a special mode that marks the generator closed, forbids yielding, and converts terminal exceptions to StopIteration.

Generator on Opensource internals

How CPython Implements Generator (types.GeneratorType) in C

The struct: `_PyGenObject`

Creating a generator: `RETURN_GENERATOR`

Generator state

`gi_code` and `gi_frame`

`send()`: `gen_send_ex` / `gen_send_ex2`

`throw()`

`close()`

Clearing and deallocation

`yield from` support

Key takeaways

Further reading

How CPython Implements async generators in C

The object struct

`ag_code`

Async generator hooks

`anext`/`asend`/`athrow` are awaitables

`aclose`

Finalization

Summary

Further reading

Generator on Opensource internals

How CPython Implements Generator (types.GeneratorType) in C

The struct: _PyGenObject

Creating a generator: RETURN_GENERATOR

Generator state

gi_code and gi_frame

send(): gen_send_ex / gen_send_ex2

throw()

close()

Clearing and deallocation

yield from support

Key takeaways

Further reading

How CPython Implements async generators in C

The object struct

ag_code

Async generator hooks

anext/asend/athrow are awaitables

aclose

Finalization

Summary

Further reading

The struct: `_PyGenObject`

Creating a generator: `RETURN_GENERATOR`

`gi_code` and `gi_frame`

`send()`: `gen_send_ex` / `gen_send_ex2`

`throw()`

`close()`

`yield from` support

`ag_code`

`anext`/`asend`/`athrow` are awaitables

`aclose`