CPython's LazyImport Implementation in C

TL;DR: PEP 810 doesn’t change the import system — it drops a placeholder object into the namespace and swaps it for the real module the first time you touch the name. The swap is atomic, so it’s safe in threaded code.

PEP 810 adds lazy imports: an import statement whose module is not actually loaded until the name is first used. In CPython this is implemented not by changing the import system wholesale, but by storing a placeholder object in the module/global namespace and replacing it with the real imported object on first access. That placeholder is the lazy_import type, exposed publicly as types.LazyImportType.

The entire implementation lives in Objects/lazyimportobject.c. This article traces it: the struct, how roots and projections form chains, how resolution runs, how the placeholder is atomically replaced in the namespace, and how the interpreter and module objects cooperate.

The object layout

A lazy import is a single C struct, GC-tracked:

c
// Objects/lazyimportobject.c
typedef struct {
    PyObject_HEAD
    PyObject *lz_builtins;  // Roots own the mapping; projections retain the root.
    // A root stores its absolute name (PyUnicode) in lz_from, and original
    // fromlist in lz_attr.
    // A projection stores its source placeholder (a PyLazyImportObject)
    // in lz_from, and the attribute to import from (PyUnicode) it in lz_attr.
    PyObject *lz_from;
    PyObject *lz_attr;
    // Declaration location.
    PyCodeObject *lz_code;
    int lz_instr_offset;
    // The root was accessed, so its declared path requires import semantics.
    int lz_active;
} PyLazyImportObject;

The fields have two distinct roles depending on whether the object is a root or a projection:

A root represents a top-level lazy import statement. lz_builtins is the builtins mapping captured at declaration time, lz_from is the absolute module name (a str), and lz_attr is the original fromlist (a tuple, a str, or NULL/None).
A projection represents one attribute taken off an existing placeholder, e.g. from a import b where a is itself lazy. Here lz_builtins is NULL, lz_from is the source placeholder, and lz_attr is the attribute name.

lz_code/lz_instr_offset record where the import was declared, so that a failure during resolution can attach a traceback/note pointing at the original statement. lz_active records that a dotted path had to be traversed eagerly (see below).

Here’s the thing: one struct, two roles. A root is an import statement; a projection is one attribute peeled off another placeholder. Both live in the same fields, and which meaning applies depends on whether lz_builtins is NULL.

The GC hooks are the standard ones:

c
// Objects/lazyimportobject.c
static int
lazy_import_traverse(PyObject *op, visitproc visit, void *arg)
{
    PyLazyImportObject *m = PyLazyImportObject_CAST(op);
    Py_VISIT(m->lz_builtins);
    Py_VISIT(m->lz_from);
    Py_VISIT(m->lz_attr);
    Py_VISIT(m->lz_code);
    return 0;
}

static int
lazy_import_clear(PyObject *op)
{
    PyLazyImportObject *m = PyLazyImportObject_CAST(op);
    Py_CLEAR(m->lz_builtins);
    Py_CLEAR(m->lz_from);
    Py_CLEAR(m->lz_attr);
    Py_CLEAR(m->lz_code);
    return 0;
}

The type itself is GC-enabled and not directly constructible from Python (tp_new is unset):

c
// Objects/lazyimportobject.c
PyTypeObject PyLazyImport_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    .tp_name = "lazy_import",
    .tp_basicsize = sizeof(PyLazyImportObject),
    .tp_dealloc = lazy_import_dealloc,
    .tp_repr = lazy_import_repr,
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
    .tp_doc = lazy_import_doc,
    .tp_getattro = lazy_import_getattro,
    .tp_traverse = lazy_import_traverse,
    .tp_clear = lazy_import_clear,
    .tp_methods = lazy_import_methods,
    .tp_alloc = PyType_GenericAlloc,
    .tp_free = PyObject_GC_Del,
};

It is published as types.LazyImportType via _typesmodule.c (EXPORT_STATIC_TYPE("LazyImportType", PyLazyImport_Type)).

Creating placeholders

_PyLazyImport_New is the single factory. It handles both roots and projections, distinguished by whether name is already a lazy import:

c
// Objects/lazyimportobject.c
PyObject *
_PyLazyImport_New(_PyInterpreterFrame *frame, PyObject *builtins,
                  PyObject *name, PyObject *fromlist)
{
    ...
    PyLazyImportObject *m = PyObject_GC_New(
        PyLazyImportObject, &PyLazyImport_Type);
    ...
    m->lz_builtins = Py_XNewRef(builtins);
    m->lz_from = Py_NewRef(name);
    m->lz_attr = Py_XNewRef(fromlist);

    m->lz_code = NULL;
    m->lz_instr_offset = -1;
    m->lz_active = 0;

    if (frame != NULL) {
        m->lz_code = (PyCodeObject *)Py_NewRef(_PyFrame_GetCode(frame));
        m->lz_instr_offset = _PyInterpreterFrame_LASTI(frame);
    }

    _PyObject_GC_TRACK(m);
    return (PyObject *)m;
}

The debug assertions encode the invariant: a projection has builtins == NULL, a non-NULL string fromlist, and a lazy-import name; a root has builtins != NULL.

When the compiler emits a lazy from a import b, it does not create a new root. Instead it calls _PyEval_LazyImportFrom, which first tries to reuse an already-loaded attribute, and only otherwise builds a projection:

c
// Objects/lazyimportobject.c
PyObject *
_PyEval_LazyImportFrom(PyThreadState *tstate, _PyInterpreterFrame *frame,
                       PyObject *v, PyObject *name)
{
    assert(PyLazyImport_CheckExact(v));
    PyLazyImportObject *lz = PyLazyImportObject_CAST(v);
    // Only `from a import b` can take b off an already imported a;
    // `import a.b as c` has to import a.b first.
    if (lz->lz_attr != NULL && PyTuple_Check(lz->lz_attr) &&
        PyTuple_GET_SIZE(lz->lz_attr) > 0) {
        PyObject *attr = lazy_import_get_loaded_attr(tstate, lz->lz_from, name);
        if (attr != NULL || PyErr_Occurred()) {
            return attr;
        }
    }
    return _PyLazyImport_New(frame, NULL, v, name);
}

lazy_import_get_loaded_attr is a small optimization: if the module is already fully present in sys.modules and initialized, and the attribute is a concrete (non-lazy) object, it is returned immediately without scheduling any import. Failed lookups are cleared and retried later at resolution time.

The name machinery

A placeholder’s dotted path is computed by walking the projection chain. lazy_import_path recurses through lz_from when it is itself a lazy import, joining each link with .:

c
// Objects/lazyimportobject.c
static PyObject *
lazy_import_path(PyLazyImportObject *m)
{
    if (PyLazyImport_CheckExact(m->lz_from)) {
        ...
        PyObject *base = lazy_import_path((PyLazyImportObject *)m->lz_from);
        ...
        PyObject *res = PyUnicode_FromFormat("%U.%U", base, m->lz_attr);
        ...
    }
    if (m->lz_attr != NULL &&
        (!PyTuple_Check(m->lz_attr) || PyTuple_GET_SIZE(m->lz_attr) > 0)) {
        return Py_NewRef(m->lz_from);   // root with fromlist: name is the module
    }
    // __import__("a.b") returns the top-level package `a`.
    Py_ssize_t dot = PyUnicode_FindChar(
        m->lz_from, '.', 0, PyUnicode_GET_LENGTH(m->lz_from), 1);
    ...
    return PyUnicode_Substring(m->lz_from, 0, dot);
}

lazy_import_name is the shorter form used for reprs and error messages, and lazy_import_repr renders <lazy_import 'name...'>.

Accessing an unresolved placeholder

Before resolution, the proxy refuses normal attribute access with a helpful error. lazy_import_getattro first tries the generic slot (which will find nothing, since there are no instance attributes), then raises:

c
// Objects/lazyimportobject.c
static PyObject *
lazy_import_getattro(PyObject *op, PyObject *name)
{
    PyObject *value = _PyObject_GenericGetAttrWithDict(
        op, name, NULL, /* suppress */ 1);
    if (value != NULL || PyErr_Occurred()) {
        return value;
    }
    PyObject *lz_name = lazy_import_name(PyLazyImportObject_CAST(op));
    ...
    PyErr_Format(PyExc_AttributeError,
                 "cannot access attribute %R on unresolved lazy import %R",
                 name, lz_name);
    ...
}

There is one explicit method, resolve(), which forces resolution and returns the imported object:

c
// Objects/lazyimportobject.c
static PyObject *
lazy_import_resolve(PyObject *self, PyObject *args)
{
    return _PyImport_LoadLazyImportTstate(PyThreadState_GET(), self);
}

Resolution

Resolution is the core. _PyImport_LoadLazyImportTstate calls lazy_import_resolve_impl, which takes an optional imported_module out-param used by _PyLazyImport_Reify.

The algorithm:

Walk to the root. Follow lz_from while it is a lazy import. Remember the first projection (first) and the root.

c
// Objects/lazyimportobject.c
PyLazyImportObject *root = lz, *first = NULL;
while (PyLazyImport_CheckExact(root->lz_from)) {
    first = root;
    root = (PyLazyImportObject *)root->lz_from;
}

Cycle detection. The thread state keeps a set ts->lazy_imports of placeholders currently being resolved. If the one being resolved is already in the set, raise ImportCycleError.

c
// Objects/lazyimportobject.c
if (_PyLazyImport_IsResolving(tstate, lazy_import)) {
    ...
    PyErr_SetImportErrorSubclass(PyExc_ImportCycleError, errmsg,
                                 root->lz_from, NULL);
    return NULL;
}

_PyLazyImport_IsResolving is just a set membership test; identity hashing is used, which is why the assertion notes that exact placeholders use identity.

Register the placeholder in the resolving set, and compute the fromlist passed to __import__. For a from a import b, c form, only the single name being resolved is imported; an empty tuple is preserved for custom __import__ hooks.

c
// Objects/lazyimportobject.c
if (first != NULL && root->lz_attr != NULL &&
    PyTuple_Check(root->lz_attr) && PyTuple_GET_SIZE(root->lz_attr) > 0) {
    fromlist = PyTuple_Pack(1, first->lz_attr);
}
else {
    fromlist = Py_NewRef(root->lz_attr != NULL ? root->lz_attr : Py_None);
}

Fetch __import__ from the root’s captured builtins and call _PyEval_ImportNameWithImport. For a plain top-level import of a dotted name (import a.b.c), CPython tries to bind only the top-level package when all intermediate components are already loaded and regular, setting lz_active so children are imported on demand. This is the “lazy submodules” optimization.

c
// Objects/lazyimportobject.c
if (first == NULL &&
    (fromlist == Py_None ||
     (PyTuple_Check(fromlist) && PyTuple_GET_SIZE(fromlist) == 0)) &&
    _PyImport_IsDefaultImportFunc(tstate->interp, import_func) &&
    root->lz_builtins == tstate->interp->builtins) {
    // A plain import binds the root package. Its pending children are
    // imported separately when their attributes are accessed.
    ...
    if (regular && (!complete || loaded == NULL) && !PyErr_Occurred()) {
        Py_SETREF(name, PyUnicode_Substring(name, 0, dot));
    }
    ...
}

Replay projections. If the placeholder is a projection (first != NULL), take the imported module and walk the attribute chain again via lazy_import_replay_from, which calls _PyEval_ImportFrom for each link and resolves any nested lazy imports:

c
// Objects/lazyimportobject.c
if (obj != NULL && first != NULL) {
    PyObject *from = obj;
    obj = lazy_import_replay_from(tstate, from, lz);
    Py_DECREF(from);
}

Discard lazy module bookkeeping (_PyImport_DiscardLazyModule) and unregister from the resolving set.
On failure, lazy_import_add_exception_cause attaches the declaration site: if the exception has no existing cause/context, it wraps it in an ImportError whose traceback points at lz_code/lz_instr_offset; otherwise it adds a __note__.

The real win: point 7 means a lazy import that blows up at use time still points at the import line in your traceback. That’s what lz_code and lz_instr_offset are for.

Reifying: replacing the placeholder in the namespace

Resolution alone returns the imported object; it does not update the namespace. _PyLazyImport_Reify does that, and is called from the eval loop (_PyLoadName/_PyEval_LoadName), module attribute access, and the specialized bytecode caches:

c
// Objects/lazyimportobject.c
PyObject *
_PyLazyImport_Reify(PyThreadState *tstate, PyObject *placeholder,
                    PyObject *name, PyObject *namespace)
{
    PyObject *imported_module = NULL;
    PyObject *value = lazy_import_resolve_impl(
        tstate, placeholder, &imported_module);
    if (value == NULL) {
        Py_XDECREF(imported_module);
        return NULL;
    }
    int rc;
    if (PyDict_CheckExact(namespace)) {
        rc = _PyDict_ReplaceItemIf(namespace, name, placeholder, value);
        if (rc == 0 && imported_module != NULL) {
            rc = lazy_import_replace_child(
                tstate, placeholder, name, namespace, imported_module, value);
        }
    }
    else if (Py_TYPE(namespace)->tp_as_mapping == NULL ||
             Py_TYPE(namespace)->tp_as_mapping->mp_ass_subscript == NULL) {
        // Read-only namespaces can resolve a value without caching it.
        Py_XDECREF(imported_module);
        return value;
    }
    else {
        // Custom namespaces retain their mapping protocol.
        PyObject *current;
        rc = PyMapping_GetOptionalItem(namespace, name, &current);
        if (rc > 0) {
            if (current == placeholder) {
                rc = PyObject_SetItem(namespace, name, value);
            }
            Py_DECREF(current);
        }
    }
    ...
    return value;
}

The atomic _PyDict_ReplaceItemIf is the key to thread safety: the placeholder is replaced only if it is still the current value, so a concurrent resolution cannot clobber a newer binding. lazy_import_replace_child extends this to the parent module’s dictionary, so that loading pkg.child updates pkg.child’s binding too.

This is the part people get wrong. The replace is conditional. If someone rebound the name between the placeholder being created and it being resolved, the resolution result is thrown away rather than stomping the newer binding.

Lazy submodules

Dotted imports are handled specially. When an attribute access on a module finds a pending lazy submodule (registered via _PyImport_TryLoadLazySubmodule, which calls _PyLazyImport_LoadChild), the child is resolved and the module’s dictionary is updated:

c
// Objects/lazyimportobject.c
PyObject *
_PyLazyImport_LoadChild(PyThreadState *tstate, PyObject *declaration,
                        PyObject *name)
{
    ...
    PyObject *lz = _PyLazyImport_New(NULL, source->lz_builtins, name, NULL);
    ...
    // build a projection chain for each dotted component
    ...
    PyObject *result = _PyImport_LoadLazyImportTstate(tstate, lz);
    ...
}

_PyLazyImport_IsActive exposes whether a root was accessed in a way that requires full import semantics (used by import.c when deciding whether to resolve a child eagerly).

Summary

lazy_import is a GC-tracked struct with lz_builtins, lz_from, lz_attr, plus declaration location.
Roots own the import description; projections form a chain via lz_from, enabling from a import b without importing up front.
Resolution walks to the root, guards against cycles with a per-thread set, calls __import__, replays attribute lookups, and attaches declaration-site tracebacks on failure.
_PyLazyImport_Reify atomically swaps the placeholder for the real object in the namespace (and in the parent module for submodules).
The eval loop, module attribute access, and bytecode caches all call _PyLazyImport_Reify when they encounter a placeholder, so resolution is transparent at use sites.
Lazy submodules are imported on demand through _PyLazyImport_LoadChild.