CPython's pickle.PickleBuffer Implementation in C: Out-of-Band Data Without Copies

TL;DR: pickle.PickleBuffer is a fixed-size C object that holds exactly one Py_buffer view, taken with PyBUF_FULL_RO, and never copies a byte. Its buffer-protocol implementation redirects to the original exporter, so whatever flags the consumer asks for are honored end to end.

Python’s pickle.PickleBuffer wraps a buffer-providing object so pickle can move its data out-of-band (protocol 5+). It isn’t implemented in Lib/pickle.py — CPython puts it in Objects/picklebufobject.c as a small GC object that stores a single Py_buffer view taken with PyBUF_FULL_RO. Let’s trace it through the 3.15 source.

The object struct

PyPickleBufferObject is fixed-size. It keeps the view exported from the original object plus a weakref list:

c
// Objects/picklebufobject.c
typedef struct {
    PyObject_HEAD
    /* The view exported by the original object */
    Py_buffer view;
    PyObject *weakreflist;
} PyPickleBufferObject;

The type object registers it as a GC object with weakref support and buffer procs:

c
// Objects/picklebufobject.c
PyTypeObject PyPickleBuffer_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "pickle.PickleBuffer",
    .tp_doc = PyDoc_STR("Wrapper for potentially out-of-band buffers"),
    .tp_basicsize = sizeof(PyPickleBufferObject),
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
    .tp_new = picklebuf_new,
    .tp_dealloc = picklebuf_dealloc,
    .tp_traverse = picklebuf_traverse,
    .tp_clear = picklebuf_clear,
    .tp_weaklistoffset = offsetof(PyPickleBufferObject, weakreflist),
    .tp_as_buffer = &picklebuf_as_buffer,
    .tp_methods = picklebuf_methods,
};

Here’s what’s interesting: unlike memoryview, there’s no separate managed-buffer object and no private copy of shape/strides/suboffsets. The PickleBuffer simply holds the exporter’s Py_buffer directly.

Creating a PickleBuffer

Both the C API and the Python constructor do the same thing: allocate the object, then obtain a full read-only view from base:

c
// Objects/picklebufobject.c
PyObject *
PyPickleBuffer_FromObject(PyObject *base)
{
    PyTypeObject *type = &PyPickleBuffer_Type;
    PyPickleBufferObject *self;

    self = (PyPickleBufferObject *) type->tp_alloc(type, 0);
    if (self == NULL) {
        return NULL;
    }
    self->view.obj = NULL;
    self->weakreflist = NULL;
    if (PyObject_GetBuffer(base, &self->view, PyBUF_FULL_RO) < 0) {
        Py_DECREF(self);
        return NULL;
    }
    return (PyObject *) self;
}

picklebuf_new is identical in shape, parsing a single positional argument named "":

c
// Objects/picklebufobject.c
static PyObject *
picklebuf_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    ...
    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O:PickleBuffer",
                                     keywords, &base)) {
        return NULL;
    }
    self = (PyPickleBufferObject *) type->tp_alloc(type, 0);
    ...
    self->view.obj = NULL;
    self->weakreflist = NULL;
    if (PyObject_GetBuffer(base, &self->view, PyBUF_FULL_RO) < 0) {
        Py_DECREF(self);
        return NULL;
    }
    return (PyObject *) self;
}

Notice that self->view.obj is set to NULL first. That’s deliberate: if PyObject_GetBuffer fails (or the allocation is interrupted), you can’t leave a dangling pointer that dealloc/clear would later try to release. On success, view.obj points at base with its reference count held by the Py_buffer, so the original exporter stays alive for as long as the PickleBuffer does.

The released state

A PickleBuffer can be released, after which its view is no longer usable. Because PyBuffer_Release sets view->obj = NULL, the code uses exactly that as the released marker:

c
// Objects/picklebufobject.c
const Py_buffer *
PyPickleBuffer_GetBuffer(PyObject *obj)
{
    PyPickleBufferObject *self = (PyPickleBufferObject *) obj;

    if (!PyPickleBuffer_Check(obj)) { ... TypeError ... }
    if (self->view.obj == NULL) {
        PyErr_SetString(PyExc_ValueError,
                        "operation forbidden on released PickleBuffer object");
        return NULL;
    }
    return &self->view;
}

The same guard appears in picklebuf_raw and picklebuf_getbuf. Releasing is exposed both as the C API function and as a method:

c
// Objects/picklebufobject.c
int
PyPickleBuffer_Release(PyObject *obj)
{
    ...
    PyBuffer_Release(&self->view);
    return 0;
}

static PyObject *
picklebuf_release(PyObject *op, PyObject *Py_UNUSED(ignored))
{
    PyPickleBufferObject *self = (PyPickleBufferObject*)op;
    PyBuffer_Release(&self->view);
    Py_RETURN_NONE;
}

PyBuffer_Release is idempotent here: once view.obj is NULL it does nothing, so calling release() twice is safe.

Buffer protocol redirection

A PickleBuffer itself implements the buffer protocol. But instead of copying its stored view, it re-exports from the original object with whatever flags the caller requests:

c
// Objects/picklebufobject.c
static int
picklebuf_getbuf(PyObject *op, Py_buffer *view, int flags)
{
    PyPickleBufferObject *self = (PyPickleBufferObject*)op;
    if (self->view.obj == NULL) {
        PyErr_SetString(PyExc_ValueError,
                        "operation forbidden on released PickleBuffer object");
        return -1;
    }
    return PyObject_GetBuffer(self->view.obj, view, flags);
}

This is an important design choice. The stored view is always PyBUF_FULL_RO, but a consumer can ask for a different level of detail — format, strides, shape, writability — and the PickleBuffer forwards that request straight to the original exporter. The release callback is a no-op that exists only as a sentinel:

c
// Objects/picklebufobject.c
static void
picklebuf_releasebuf(PyObject *self, Py_buffer *view)
{
    /* Since our bf_getbuffer redirects to the original object, this
     * implementation is never called.  It only exists to signal that
     * buffers exported by PickleBuffer have non-trivial releasing
     * behaviour (see check in Python/getargs.c).
     */
}

Because bf_getbuffer redirects, the real release happens against the original object’s buffer — not against the PickleBuffer’s own view.

`raw()`: reinterpreting the buffer as raw bytes

PickleBuffer.raw() returns a memoryview over the underlying memory, but only when the wrapped buffer is contiguous. The implementation builds an ordinary memoryview from self, then mutates it in place to describe the data as a 1-D array of bytes:

c
// Objects/picklebufobject.c
static PyObject *
picklebuf_raw(PyObject *op, PyObject *Py_UNUSED(ignored))
{
    PyPickleBufferObject *self = (PyPickleBufferObject*)op;
    if (self->view.obj == NULL) { ... released error ... }
    if (self->view.suboffsets != NULL
        || !PyBuffer_IsContiguous(&self->view, 'A')) {
        PyErr_SetString(PyExc_BufferError,
                        "cannot extract raw buffer from non-contiguous buffer");
        return NULL;
    }
    PyObject *m = PyMemoryView_FromObject((PyObject *) self);
    if (m == NULL) {
        return NULL;
    }
    PyMemoryViewObject *mv = (PyMemoryViewObject *) m;
    assert(mv->view.suboffsets == NULL);
    /* Mutate memoryview instance to make it a "raw" memoryview */
    mv->view.format = "B";
    mv->view.ndim = 1;
    mv->view.itemsize = 1;
    /* shape = (length,) */
    mv->view.shape = &mv->view.len;
    /* strides = (1,) */
    mv->view.strides = &mv->view.itemsize;
    /* Fix memoryview state flags */
    mv->flags = _Py_MEMORYVIEW_C | _Py_MEMORYVIEW_FORTRAN;
    return m;
}

Here’s the trick: the shape and strides pointers are pointed at the len and itemsize fields of the same Py_buffer. So the memoryview describes exactly len bytes with stride 1, and no arrays are allocated at all. That’s what pickle uses to obtain a contiguous byte view for in-band serialization, and to hand to a buffer callback for out-of-band transfer.

Lifetime, GC, and weakrefs

The PickleBuffer is GC-tracked because its view.obj can participate in reference cycles — the wrapped object may itself refer back to the PickleBuffer. Traversal visits that reference:

c
// Objects/picklebufobject.c
static int
picklebuf_traverse(PyObject *op, visitproc visit, void *arg)
{
    PyPickleBufferObject *self = (PyPickleBufferObject*)op;
    Py_VISIT(self->view.obj);
    return 0;
}

Clearing releases the underlying buffer, and deallocation does the same:

c
// Objects/picklebufobject.c
static int
picklebuf_clear(PyObject *op)
{
    PyPickleBufferObject *self = (PyPickleBufferObject*)op;
    PyBuffer_Release(&self->view);
    return 0;
}

static void
picklebuf_dealloc(PyObject *op)
{
    PyPickleBufferObject *self = (PyPickleBufferObject*)op;
    PyObject_GC_UnTrack(self);
    FT_CLEAR_WEAKREFS(op, self->weakreflist);
    PyBuffer_Release(&self->view);
    Py_TYPE(self)->tp_free((PyObject *) self);
}

FT_CLEAR_WEAKREFS clears any weak references before the object is freed, and PyBuffer_Release both calls the exporter’s release callback (if any) and decrements view.obj. After that, the slot is freed with tp_free.

Use in the pickle protocol

In Modules/_pickle.c, a PickleBuffer is detected by type and routed to save_picklebuffer, which obtains a full view, checks contiguity, then either writes the data in-band (as bytes/bytearray) or writes NEXT_BUFFER/READONLY_BUFFER opcodes for out-of-band transfer and hands the buffer to the user’s buffer callback.

On unpickling, NEXT_BUFFER pops the next buffer from the provided buffers iterable, and READONLY_BUFFER wraps it in a read-only memoryview when needed.

Key takeaways

PickleBuffer is a fixed-size GC object holding one Py_buffer view taken with PyBUF_FULL_RO.
It neither copies nor manages shape/strides. It stores the exporter’s view directly and keeps the exporter alive through view.obj.
Released state is detected by view.obj == NULL, which PyBuffer_Release sets for you.
Its buffer-protocol implementation redirects to the original exporter with the caller’s requested flags.
raw() builds a memoryview and reinterprets it as a contiguous 1-D byte array by pointing shape/strides at the view’s own len/itemsize.
Weakrefs and GC traversal are supported, and deallocation releases the underlying buffer.