How CPython Implements Weak References (weakref) in C

Python’s weak references – weakref.ref, weakref.proxy, the machinery behind weakref.WeakValueDictionary and __weakref__ – are implemented in CPython in Objects/weakrefobject.c as a single C struct, PyWeakReference, that is reused for three Python types: ReferenceType, ProxyType, and CallableProxyType. A weak reference is a “stealth” pointer: it points at an object without incrementing that object’s reference count, and when the object dies every weak reference to it is cleared (and any callbacks run). This article traces that implementation through the 3.15 source.

The object: `PyWeakReference`

The layout is defined in Include/cpython/weakrefobject.h:

c
// Include/cpython/weakrefobject.h
struct _PyWeakReference {
    PyObject_HEAD

    /* The object to which this is a weak reference, or Py_None if none.
     * Note that this is a stealth reference: wr_object's refcount is
     * not incremented to reflect this pointer. */
    PyObject *wr_object;

    /* A callable to invoke when wr_object dies, or NULL if none. */
    PyObject *wr_callback;

    /* A cache for wr_object's hash code. -1 if unknown. */
    Py_hash_t hash;

    /* Doubly-linked list pointers into the referenced object's weakref list. */
    PyWeakReference *wr_prev;
    PyWeakReference *wr_next;
    vectorcallfunc vectorcall;

#ifdef Py_GIL_DISABLED
    PyMutex *weakrefs_lock;
#endif
};

Invariants:

wr_object is either a live object or Py_None. Py_None is the sentinel for “dead”; it is not a real reference, so no Py_DECREF is owed for it.
wr_object is a stealth reference: its refcount is never incremented to account for this pointer. This is the entire point of a weak reference.
wr_callback is an owned (refcounted) reference, or NULL.
The object is GC-tracked (Py_TPFLAGS_HAVE_GC) because it owns wr_callback (and, transitively, may keep a dead ref’s callback alive).
All three Python types (ReferenceType, ProxyType, CallableProxyType) share this same struct; only their tp_* slots differ.
PyWeakref_CheckRef, PyWeakref_CheckProxy, PyWeakref_Check are the type tests. PyWeakref_CheckRefExact is an exact-type test used for the “basic ref” fast paths.

The per-object weakref list

Every object that supports weak references carries a doubly-linked, NULL-terminated list of the weak references pointing at it. The head of that list is stored in the object itself, at an offset given by the type’s tp_weaklistoffset:

c
// Include/internal/pycore_object.h
static inline PyObject **
_PyObject_GET_WEAKREFS_LISTPTR(PyObject *op)
{
    Py_ssize_t offset = Py_TYPE(op)->tp_weaklistoffset;
    return (PyObject **)((char *)op + offset);
}

Support is gated by _PyType_SUPPORTS_WEAKREFS(type) which simply checks type->tp_weaklistoffset != 0. Concrete examples in the source:

c
// Objects/funcobject.c
.tp_weaklistoffset = offsetof(PyFunctionObject, func_weakreflist),
// Objects/genobject.c
.tp_weaklistoffset = offsetof(PyGenObject, gi_weakreflist),
// Objects/moduleobject.c
.tp_weaklistoffset = offsetof(PyModuleObject, md_weaklist),
// Objects/setobject.c
.tp_weaklistoffset = offsetof(PySetObject, weakreflist),

For heap (user-defined) classes, __weakref__ is added automatically unless __slots__ forbids it. In typeobject.c this is recorded as a managed weakref slot:

c
// Objects/typeobject.c
if (ctx->add_weak) {
    type_add_flags(type, Py_TPFLAGS_MANAGED_WEAKREF);
    type->tp_weaklistoffset = MANAGED_WEAKREF_OFFSET;   // negative sentinel
}

MANAGED_WEAKREF_OFFSET is a negative offset (-2 or -4 sizeof(PyObject*), depending on the build) into the managed pointer area just before the instance dict, so the head pointer lives in the object’s pre-header region rather than in a named field. subtype_getweakref exposes this as the __weakref__ attribute and returns the head weakref (or None):

c
// Objects/typeobject.c
static PyObject *
subtype_getweakref(PyObject *obj, void *context)
{
    ...
    weaklistptr = (PyObject **)((char *)obj + type->tp_weaklistoffset);
    if (*weaklistptr == NULL) {
        result = Py_NewRef(Py_None);
    }
    else if (_Py_TryIncref(*weaklistptr)) {
        result = *weaklistptr;
    }
    else {
        result = Py_NewRef(Py_None);
    }
    ...
}

Construction: `get_or_create_weakref`

The public entry points are PyWeakref_NewRef and PyWeakref_NewProxy:

c
// Objects/weakrefobject.c
PyObject *
PyWeakref_NewRef(PyObject *ob, PyObject *callback)
{
    return (PyObject *)get_or_create_weakref(&_PyWeakref_RefType, ob, callback);
}

PyObject *
PyWeakref_NewProxy(PyObject *ob, PyObject *callback)
{
    PyTypeObject *type = &_PyWeakref_ProxyType;
    if (PyCallable_Check(ob)) {
        type = &_PyWeakref_CallableProxyType;
    }
    return (PyObject *)get_or_create_weakref(type, ob, callback);
}

get_or_create_weakref does the real work. First it validates that the target supports weak references, normalizes callback == Py_None to NULL, then:

c
// Objects/weakrefobject.c
PyWeakReference **list = GET_WEAKREFS_LISTPTR(obj);

if ((type == &_PyWeakref_RefType) ||
    (type == &_PyWeakref_ProxyType) ||
    (type == &_PyWeakref_CallableProxyType))
{
    LOCK_WEAKREFS(obj);
    PyWeakReference *basic_ref = try_reuse_basic_ref(*list, type, callback);
    if (basic_ref != NULL) {
        UNLOCK_WEAKREFS(obj);
        return basic_ref;            // reused an existing callback-less ref
    }
    PyWeakReference *newref = allocate_weakref(type, obj, callback);
    if (newref == NULL) { UNLOCK_WEAKREFS(obj); return NULL; }
    insert_weakref(newref, list);
    UNLOCK_WEAKREFS(obj);
    return newref;
}
else {
    // Subtypes / custom types: allocate outside the lock
    PyWeakReference *newref = allocate_weakref(type, obj, callback);
    if (newref == NULL) return NULL;
    LOCK_WEAKREFS(obj);
    insert_weakref(newref, list);
    UNLOCK_WEAKREFS(obj);
    return newref;
}

Two important optimizations:

Basic-ref reuse. A weak reference with no callback of the exact basic type (weakref.ref / weakref.proxy) is “canonical”: at most one of each kind (ref and proxy) is kept at the head of the list. try_reuse_basic_ref returns an existing one instead of allocating a new object, incrementing its refcount. This is why weakref.ref(obj) is weakref.ref(obj) can be true for callback-less refs.
Lock discipline. For the three builtin types allocation happens inside the lock (cheap, tp_alloc); for subtypes, allocation happens outside the lock because a subtype’s tp_alloc may run arbitrary Python code.

init_weakref stores the stealth pointer and owns the callback:

c
// Objects/weakrefobject.c
static void
init_weakref(PyWeakReference *self, PyObject *ob, PyObject *callback)
{
    self->hash = -1;
    self->wr_object = ob;          // stealth: no incref
    self->wr_prev = NULL;
    self->wr_next = NULL;
    self->wr_callback = Py_XNewRef(callback);
    self->vectorcall = weakref_vectorcall;
#ifdef Py_GIL_DISABLED
    self->weakrefs_lock = &WEAKREF_LIST_LOCK(ob);
    _PyObject_SetMaybeWeakref(ob);
    _PyObject_SetMaybeWeakref((PyObject *)self);
#endif
}

allocate_weakref uses type->tp_alloc(type, 0) (so subtypes allocate correctly) and then init_weakref.

Insertion ordering

The list is kept in a canonical order so the basic (callback-less) refs sit at the head:

c
// Objects/weakrefobject.c
static void
insert_weakref(PyWeakReference *newref, PyWeakReference **list)
{
    PyWeakReference *ref, *proxy;
    get_basic_refs(*list, &ref, &proxy);

    PyWeakReference *prev;
    if (is_basic_ref(newref)) {
        prev = NULL;                      // head
    }
    else if (is_basic_proxy(newref)) {
        prev = ref;                       // after the basic ref
    }
    else {
        prev = (proxy == NULL) ? ref : proxy;   // after both basic refs
    }

    if (prev == NULL) insert_head(newref, list);
    else              insert_after(newref, prev);
}

insert_head/insert_after splice the node into the doubly-linked list and update wr_prev/wr_next. The first node’s wr_prev stays NULL and acts as the list head stored in the object.

Reading a weak reference

The core accessor is _PyWeakref_GET_REF (inlined in Include/internal/pycore_weakref.h):

c
// Include/internal/pycore_weakref.h
static inline PyObject* _PyWeakref_GET_REF(PyObject *ref_obj)
{
    PyWeakReference *ref = _Py_CAST(PyWeakReference*, ref_obj);
    PyObject *obj = FT_ATOMIC_LOAD_PTR(ref->wr_object);
    if (obj == Py_None) {
        return NULL;                       // cleared
    }

    LOCK_WEAKREFS(obj);
#ifdef Py_GIL_DISABLED
    if (ref->wr_object == Py_None) {       // re-check under lock
        UNLOCK_WEAKREFS(obj);
        return NULL;
    }
#endif
    if (_Py_TryIncref(obj)) {              // stealth -> real ref, if alive
        UNLOCK_WEAKREFS(obj);
        return obj;                        // new reference (caller owns)
    }
    UNLOCK_WEAKREFS(obj);
    return NULL;
}

Because wr_object is stealth, reading it requires trying to incref the target under the lock: if the target is already dead (refcount == 0), the incref fails and NULL is returned. On success the caller receives a new owned reference.

The public API wraps this:

c
// Objects/weakrefobject.c
int
PyWeakref_GetRef(PyObject *ref, PyObject **pobj)
{
    ...
    *pobj = _PyWeakref_GET_REF(ref);
    return (*pobj != NULL);   // 1 if live, 0 if dead
}

PyWeakref_IsDead / _PyWeakref_IS_DEAD additionally handle the GC/trashcan delay case: a target whose refcount hit zero but whose weakrefs have not yet been cleared is still reported as dead via _is_dead().

The ref object itself is callable (via vectorcall), so ref() returns the target or None:

c
// Objects/weakrefobject.c
static PyObject *
weakref_vectorcall(PyObject *self, ...)
{
    PyObject *obj = _PyWeakref_GET_REF(self);
    if (obj == NULL) {
        Py_RETURN_NONE;
    }
    return obj;          // new reference
}

Hashing and equality

weakref_hash caches the target’s hash in wr_hash under the weakref’s own critical section; a dead ref raises TypeError: weak object has gone away.
weakref_richcompare compares targets: if either is dead, equality reduces to identity of the two ref objects; otherwise it delegates to PyObject_RichCompare on the targets.

Clearing: when the target dies

The central routine is clear_weakref_lock_held, which removes a node from the target’s list and flips wr_object to Py_None:

c
// Objects/weakrefobject.c
static void
clear_weakref_lock_held(PyWeakReference *self, PyObject **callback)
{
    if (self->wr_object != Py_None) {
        PyWeakReference **list = GET_WEAKREFS_LISTPTR(self->wr_object);
        if (*list == self) {
            FT_ATOMIC_STORE_PTR(*list, self->wr_next);
        }
        FT_ATOMIC_STORE_PTR(self->wr_object, Py_None);
        if (self->wr_prev != NULL) self->wr_prev->wr_next = self->wr_next;
        if (self->wr_next != NULL) self->wr_next->wr_prev = self->wr_prev;
        self->wr_prev = NULL;
        self->wr_next = NULL;
    }
    if (callback != NULL) {
        *callback = self->wr_callback;   // steal
        self->wr_callback = NULL;
    }
}

Note that the callback is stolen, not invoked, here. Clearing happens under the weakref list lock, and invoking arbitrary Python code under that lock is forbidden (especially in free-threaded builds where the lock is _Py_LOCK_DONT_DETACH). So clear_weakref_lock_held only detaches the callback; the caller decides when to run it.

There are three callers:

PyObject_ClearWeakRefs(object) – called from tp_dealloc when the target’s refcount hits zero. It clears every weakref and runs callbacks in two phases:
- First, the callback-less basic ref/proxy at the head of the list are cleared under the lock (no callbacks to run).
- Then each remaining ref is cleared under the lock, its callback stolen, and the (ref, callback) pair stored in a tuple. After the lock is released, callbacks are invoked via handle_callback:
  c
  static void handle_callback(PyWeakReference *ref, PyObject *callback) { PyObject *cbresult = PyObject_CallOneArg(callback, (PyObject *)ref); if (cbresult == NULL) { PyErr_FormatUnraisable("Exception ignored while " "calling weakref callback %R", callback); } else { Py_DECREF(cbresult); } }
- The original exception (if any) is saved and restored around this, so a failing callback does not clobber an in-flight exception.
GC (gc_clear in weakrefobject.c and the inlined copy in gcmodule.c’s handle_weakrefs). During collection, weakrefs are cleared but callbacks are deliberately left intact and uncalled, because running Python code mid-GC is unsafe. _PyWeakref_ClearRef (note: capital R) is the GC-facing variant that clears the ref while leaving the callback for tp_dealloc to decref later.
Deallocation of the weakref itself (weakref_dealloc / proxy_dealloc) calls clear_weakref (which locks and then Py_XDECREFs the stolen callback) and then tp_free.

_PyWeakref_ClearWeakRefsNoCallbacks and _PyStaticType_ClearWeakRefs are further variants used for bulk clearing without callbacks (e.g. finalization of static types).

The callback member

wr_callback is exposed read-only to Python as __callback__:

c
// Objects/weakrefobject.c
static PyMemberDef weakref_members[] = {
    {"__callback__", _Py_T_OBJECT,
     offsetof(PyWeakReference, wr_callback), Py_READONLY},
    {NULL}
};

weakref.ref.__init__ is a no-op (the real construction happens in __new__), so ref(obj, callback) creates a new weakref with a callback that cannot be reused via the basic-ref fast path (callbacks force a fresh node).

Proxy objects

ProxyType and CallableProxyType are the same PyWeakReference struct, but instead of being callable to retrieve the target, they forward every operation to the target. The implementation is a set of macros that unwrap the proxy into the real object, call the generic C API, and decref:

c
// Objects/weakrefobject.c
#define UNWRAP(o) \
    if (PyWeakref_CheckProxy(o)) { \
        o = _PyWeakref_GET_REF(o); \
        if (!proxy_check_ref(o)) { return NULL; }   // ReferenceError if dead \
    } else { \
        Py_INCREF(o); \
    }

#define WRAP_UNARY(method, generic) \
    static PyObject * method(PyObject *proxy) { \
        UNWRAP(proxy); \
        PyObject* res = generic(proxy); \
        Py_DECREF(proxy); \
        return res; \
    }

#define WRAP_BINARY(method, generic) \
    static PyObject * method(PyObject *x, PyObject *y) { \
        UNWRAP(x); UNWRAP(y); \
        PyObject* res = generic(x, y); \
        Py_DECREF(x); Py_DECREF(y); \
        return res; \
    }

proxy_check_ref raises ReferenceError: weakly-referenced object no longer exists when the target is gone. These macros generate the number slots (proxy_add, proxy_sub, …), sequence/mapping slots (proxy_length, proxy_getitem, proxy_setitem, proxy_contains), iterator slots (proxy_iter, proxy_iternext), and attribute access (proxy_getattr, proxy_setattr). CallableProxyType additionally sets tp_call = proxy_call, which unwraps and calls the target.

Notable details:

Proxies have no tp_hash (intentionally omitted) because they are “mutable” once the reference dies.
proxy_bool, proxy_repr, etc. fetch the target via _PyWeakref_GET_REF directly rather than through UNWRAP.
proxy_dealloc is the same as weakref_dealloc (clear_weakref + PyObject_GC_Del).

Free-threading

In the no-GIL build, weakref state is protected by striped locks keyed on the target’s address (WEAKREF_LIST_LOCK(obj)), acquired with _Py_LOCK_DONT_DETACH so that weakref clearing can run during operations like WeakValueDictionary deletion that must not suspend. The weakref’s own weakrefs_lock pointer is captured at creation (so a ref can be locked even after its target has been set to Py_None). _PyWeakref_GET_REF and _PyWeakref_IS_DEAD take and release these locks; _PyObject_SetMaybeWeakref marks both the target and the ref itself so the runtime knows a weakref may exist. The GC is exempt from locking because the world is stopped during collection.

Summary of the data flow

text

weakref.ref(obj, callback)
  -> get_or_create_weakref(RefType, obj, callback)
       -> supports weakrefs? (tp_weaklistoffset != 0)
       -> list = GET_WEAKREFS_LISTPTR(obj)   // head pointer in obj
       -> reuse basic ref if callback is None and one exists
       -> allocate_weakref -> tp_alloc + init_weakref (stealth wr_object,
                                                      owned wr_callback)
       -> insert_weakref into doubly-linked list (basic refs at head)
       -> return ref

ref()  -> _PyWeakref_GET_REF -> try incref target under lock -> new ref or None

target dies (refcount 0) -> tp_dealloc -> PyObject_ClearWeakRefs(target)
  -> clear each node (wr_object = Py_None, unlink)
  -> steal callbacks, invoke them outside the lock
  -> ref() now returns None; ref.__callback__ is None

Key takeaways

weakref.ReferenceType, ProxyType, and CallableProxyType are all PyWeakReference; the only differences are their tp_* slots.
wr_object is a stealth (non-owning) pointer; Py_None marks death. Reading it requires _Py_TryIncref under the target’s lock.
Each weakref-able object stores the head of a doubly-linked list of weakrefs at tp_weaklistoffset (a real offset for builtin types, a managed negative sentinel for heap classes with __weakref__).
Callback-less basic refs (one ref + one proxy) are reused instead of reallocated; refs with callbacks are always new nodes.
Clearing unlinks the node and steals the callback; callbacks run outside the lock (and are deferred entirely during GC).
Proxies forward every operation by unwrapping to the target via UNWRAP, raising ReferenceError when the target is gone.
Free-threaded builds protect the list with striped, non-detaching locks keyed on the target’s address.

The object: PyWeakReference