How CPython Implements async generators in C

Python’s async def functions containing yield become async generators: objects that implement __anext__, asend, athrow and aclose. Unlike a regular generator, an async generator is also an awaitable: calling anext(ag) does not run code immediately, it returns an async_generator_asend awaitable that is driven by the event loop. In CPython 3.15 all of this lives in Objects/genobject.c (there is no separate asyncgenobject.c), sharing the same interpreter-frame machinery as generators and coroutines.

The object struct

PyAsyncGenObject is defined through the shared _PyGenObject_HEAD macro, so its layout is identical to a generator’s except for the ag_ prefix:

c
// Include/internal/pycore_interpframe_structs.h
struct _PyAsyncGenObject {
    PyObject_HEAD
    PyObject *ag_weakreflist;
    PyObject *ag_name;
    PyObject *ag_qualname;
    _PyErr_StackItem ag_exc_state;
    PyObject *ag_origin_or_finalizer;
    char ag_hooks_inited;
    char ag_closed;
    char ag_running_async;
    int8_t ag_frame_state;
    _PyInterpreterFrame ag_iframe;
};

The two async-specific fields are ag_closed (set once the generator has received GeneratorExit) and ag_origin_or_finalizer, which doubles as storage for the thread’s async-generator finalizer hook. The frame itself is the same _PyInterpreterFrame used by every other callable, so an async generator reuses the normal eval loop and yield/YIELD_VALUE machinery. PyAsyncGen_New allocates through gen_new_with_qualname and then initializes the async fields:

c
// Objects/genobject.c
PyObject *PyAsyncGen_New(PyFrameObject *f, PyObject *name, PyObject *qualname) {
    PyAsyncGenObject *ag;
    ag = (PyAsyncGenObject *)gen_new_with_qualname(&PyAsyncGen_Type, f,
                                                   name, qualname);
    ...
    ag->ag_origin_or_finalizer = NULL;
    ag->ag_closed = 0;
    ag->ag_hooks_inited = 0;
    ag->ag_running_async = 0;
    return (PyObject*)ag;
}

`ag_code`

Like gi_code/cr_code, ag_code is a read-only attribute backed by the frame’s executable. ag_getcode just forwards to _gen_getcode, which returns a new reference to _PyGen_GetCode(gen) (i.e. _PyFrame_GetCode(&gen->ag_iframe)):

c
// Objects/genobject.c
static PyObject *
ag_getcode(PyObject *gen, void *Py_UNUSED(ignored)) {
    return _gen_getcode((PyGenObject*)gen, "ag_code");
}

static PyObject *
_gen_getcode(PyGenObject *gen, const char *const name) {
    if (PySys_Audit("object.__getattr__", "Os", gen, name) < 0) {
        return NULL;
    }
    return Py_NewRef(_PyGen_GetCode(gen));
}

It is registered in async_gen_getsetlist as {"ag_code", ag_getcode, NULL, NULL}. Because the code object is reachable through the (possibly cleared) frame, ag_code remains available even after the generator is closed.

Async generator hooks

CPython exposes two per-thread hooks, sys.set_asyncgen_firstiter and sys.set_asyncgen_finalizer, stored on the thread state as tstate->async_gen_firstiter and tstate->async_gen_finalizer (see Python/ceval.c). The firstiter hook is invoked the first time an async generator is iterated; the finalizer is stored on the object for later use.

Hook initialization is lazy and guarded by ag_hooks_inited, so it happens on the first call to anext/asend/aclose/athrow:

c
// Objects/genobject.c
static int async_gen_init_hooks(PyAsyncGenObject *o) {
    PyThreadState *tstate;
    PyObject *finalizer;
    PyObject *firstiter;

    if (o->ag_hooks_inited) return 0;
    o->ag_hooks_inited = 1;

    tstate = _PyThreadState_GET();

    finalizer = tstate->async_gen_finalizer;
    if (finalizer) {
        o->ag_origin_or_finalizer = Py_NewRef(finalizer);
    }

    firstiter = tstate->async_gen_firstiter;
    if (firstiter) {
        PyObject *res;
        Py_INCREF(firstiter);
        res = PyObject_CallOneArg(firstiter, (PyObject *)o);
        Py_DECREF(firstiter);
        if (res == NULL) return 1;   /* propagate exception */
        Py_DECREF(res);
    }
    return 0;
}

Note the ordering: the finalizer is stored (ag_origin_or_finalizer), while the firstiter callback is called immediately and its result discarded. If the firstiter callback raises, the error propagates out of anext() etc.

`anext`/`asend`/`athrow` are awaitables

None of these run the generator directly. async_gen_anext returns an async_generator_asend object (with sendval == NULL), and async_gen_asend wraps the supplied argument. These awaitables (PyAsyncGenASend) carry state (AWAITABLE_STATE_INIT / _ITER / _CLOSED) plus the generator and the value to send. Driving them via .send()/__next__() calls gen_send, then async_gen_unwrap_value converts a wrapped yield value into a StopIteration:

c
// Objects/genobject.c
static PyObject *async_gen_unwrap_value(PyAsyncGenObject *gen, PyObject *result) {
    if (result == NULL) {
        if (!PyErr_Occurred()) PyErr_SetNone(PyExc_StopAsyncIteration);
        if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration) ||
            PyErr_ExceptionMatches(PyExc_GeneratorExit)) {
            gen->ag_closed = 1;
        }
        gen->ag_running_async = 0;
        return NULL;
    }
    if (_PyAsyncGenWrappedValue_CheckExact(result)) {
        _PyGen_SetStopIterationValue(((_PyAsyncGenWrappedValue*)result)->agw_val);
        Py_DECREF(result);
        gen->ag_running_async = 0;
        return NULL;   /* signals "yielded a value" via StopIteration */
    }
    return result;
}

ag_running_async guards re-entrancy: while a generator is running, a second __anext__() raises “asynchronous generator is already running”.

`aclose`

async_gen_aclose is implemented as athrow(GeneratorExit) in “aclose mode” (agt_typ == NULL). It creates an async_generator_athrow awaitable and, on first send, marks the generator closed and throws GeneratorExit with close_on_genexit = 0 (so a suspended yield from is not auto-closed immediately, allowing the async generator to clean up via awaits):

c
// Objects/genobject.c (async_gen_athrow_send, aclose mode)
if (o->agt_typ == NULL) {
    /* aclose() mode */
    o->agt_gen->ag_closed = 1;
    retval = _gen_throw((PyGenObject *)gen, 0,
                        PyExc_GeneratorExit, NULL, NULL);
    if (retval && _PyAsyncGenWrappedValue_CheckExact(retval)) {
        Py_DECREF(retval);
        goto yield_close;   /* generator yielded during close -> RuntimeError */
    }
}

If the generator yields a value while being closed, CPython sets ASYNC_GEN_IGNORED_EXIT_MSG (“async generator ignored GeneratorExit”) and raises RuntimeError. Otherwise, on completion, StopAsyncIteration/GeneratorExit are swallowed and converted to StopIteration, signalling that the aclose() awaitable is done:

c
// Objects/genobject.c
if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration) ||
        PyErr_ExceptionMatches(PyExc_GeneratorExit)) {
    if (o->agt_typ == NULL) {
        PyErr_Clear();
        PyErr_SetNone(PyExc_StopIteration);
    }
}

The ag_closed flag is also consulted by subsequent asend/athrow calls, which then immediately raise StopAsyncIteration without running the body again.

Finalization

gen_finalize (registered as tp_finalize) handles unawaited async generators: if the frame is not finished and ag_closed is false, it invokes the stored finalizer (ag_origin_or_finalizer) so that unclosed async generators are reported (and typically closed) rather than silently leaked.

Summary

PyAsyncGenObject shares _PyGenObject_HEAD(ag) with generators; async-specific state is ag_closed, ag_running_async, ag_hooks_inited and ag_origin_or_finalizer.
ag_code is a read-only getset returning the frame’s code object.
Hooks are initialized lazily on first iteration: the finalizer is stored, firstiter is called immediately.
anext/asend/athrow return awaitables (PyAsyncGenASend / PyAsyncGenAThrow) that drive the generator via gen_send/gen_throw.
aclose is athrow(GeneratorExit) in a special mode that marks the generator closed, forbids yielding, and converts terminal exceptions to StopIteration.

The object struct

ag_code

Async generator hooks

anext/asend/athrow are awaitables

aclose

Finalization

Summary

Further reading

`ag_code`

`anext`/`asend`/`athrow` are awaitables

`aclose`