How CPython Implements bytearray in C

Python’s bytearray is the mutable cousin of bytes: a sequence of raw bytes you can grow, shrink and overwrite in place. In CPython that is implemented by PyByteArrayObject in Objects/bytearrayobject.c, and the clever part is that its backing store is not a plain malloc’d block — it is a real PyBytesObject, reused so that bytes and bytearray share one allocation scheme. This article traces the representation, the resize strategy and the free-threaded locking through the 3.15 source.

The object struct

Every bytearray is a PyObject_VAR_HEAD (refcount, type, length) plus the fields that describe its buffer:

c
// Include/cpython/bytearrayobject.h
typedef struct {
    PyObject_VAR_HEAD
    /* How many bytes allocated in ob_bytes. Loaded/stored atomically. */
    Py_ssize_t ob_alloc;
    char *ob_bytes;        /* Physical backing buffer */
    char *ob_start;        /* Logical start inside ob_bytes */
    Py_ssize_t ob_exports; /* How many buffer exports */
    PyObject *ob_bytes_object;  /* PyBytes used as zero-copy storage */
} PyByteArrayObject;

ob_size (from PyObject_VAR_HEAD) is the logical length. ob_bytes is the physical allocation base, ob_start is where the live data begins, and ob_alloc is how many bytes are allocated from ob_start’s point of view… in practice, after the compaction step in resize, ob_start == ob_bytes and ob_alloc equals the size of the backing PyBytes.

The trick that makes all of this work is ob_bytes_object: the actual memory lives inside a PyBytesObject. ob_bytes and ob_start are just pointers into PyBytes_AS_STRING(ob_bytes_object). This means a bytearray and a bytes share the same allocator, the same over-allocation heuristics, and the same trailing-NUL guarantee, and bytes(ba) can in some paths hand over the same buffer.

c
// Objects/bytearrayobject.c
static void
bytearray_reinit_from_bytes(PyByteArrayObject *self, Py_ssize_t size)
{
    Py_ssize_t alloc = PyBytes_GET_SIZE(self->ob_bytes_object);
    assert(0 <= size && size <= alloc);

    self->ob_bytes = self->ob_start = PyBytes_AS_STRING(self->ob_bytes_object);
    Py_SET_SIZE(self, size);
    FT_ATOMIC_STORE_SSIZE_RELAXED(self->ob_alloc, alloc);
}

So construction is: allocate a PyBytesObject of the right size, point the bytearray at it, and set the logical length. PyByteArray_FromStringAndSize does exactly that:

c
// Objects/bytearrayobject.c
PyObject *
PyByteArray_FromStringAndSize(const char *bytes, Py_ssize_t size)
{
    PyByteArrayObject *new = PyObject_New(PyByteArrayObject, &PyByteArray_Type);
    ...
    new->ob_exports = 0;
    /* size=0 uses the immortal empty bytes, so no allocation. */
    new->ob_bytes_object = PyBytes_FromStringAndSize(NULL, size);
    if (new->ob_bytes_object == NULL) { Py_DECREF(new); return NULL; }
    bytearray_reinit_from_bytes(new, size);
    if (bytes != NULL && size > 0) {
        memcpy(new->ob_bytes, bytes, size);
    }
    return (PyObject *)new;
}

When size == 0, PyBytes_FromStringAndSize(NULL, 0) returns the immortal empty-bytes singleton, so an empty bytearray costs only the object header. bytearray_new (used by bytearray()) does the same with Py_GetConstant(Py_CONSTANT_EMPTY_BYTES).

Access macros

The public macros all go through ob_start / ob_size:

c
// Include/cpython/bytearrayobject.h
#define PyByteArray_AS_STRING(self) PyByteArray_AS_STRING(_PyObject_CAST(self))
static inline char* PyByteArray_AS_STRING(PyObject *op) {
    return _PyByteArray_CAST(op)->ob_start;
}
static inline Py_ssize_t PyByteArray_GET_SIZE(PyObject *op) {
    ... /* atomic load under free-threaded build, else Py_SIZE(self) */
}

PyByteArray_GET_SIZE is atomic on Py_GIL_DISABLED builds because ob_size is mutated while other threads may be reading it through the buffer protocol.

Buffer exports and `_canresize`

A bytearray can hand out its memory to C code via the buffer protocol. While such a view is alive, the object must not move or shrink its backing store, otherwise the C pointer would dangle. This is tracked by ob_exports:

c
// Objects/bytearrayobject.c
static int
bytearray_getbuffer_lock_held(PyObject *self, Py_buffer *view, int flags)
{
    ...
    void *ptr = (void *) PyByteArray_AS_STRING(obj);
    if (PyBuffer_FillInfo(view, (PyObject*)obj, ptr, Py_SIZE(obj), 0, flags) < 0)
        return -1;
    obj->ob_exports++;
    return 0;
}

static void
bytearray_releasebuffer(PyObject *self, Py_buffer *view)
{
    Py_BEGIN_CRITICAL_SECTION(self);
    PyByteArrayObject *obj = _PyByteArray_CAST(self);
    obj->ob_exports--;
    assert(obj->ob_exports >= 0);
    Py_END_CRITICAL_SECTION();
}

_canresize then simply refuses any resize (and __init__) while exports are outstanding:

c
// Objects/bytearrayobject.c
static int
_canresize(PyByteArrayObject *self)
{
    if (self->ob_exports > 0) {
        PyErr_SetString(PyExc_BufferError,
                "Existing exports of data: object cannot be re-sized");
        return 0;
    }
    return 1;
}

bytearray_dealloc even prints a SystemError if an object is freed while exports remain — that is a programming error in whoever held the buffer.

A neat trick: methods that read the buffer while calling back into Python (such as join or the bytes-search helpers) bump ob_exports themselves to pin the storage for the duration, even without a real external export:

c
// Objects/bytearrayobject.c (bytearray_join_impl)
self->ob_exports++;   // protects `self` from being cleared/resized
ret = stringlib_bytes_join((PyObject*)self, iterable_of_bytes);
self->ob_exports--;   // unexport `self`

Resize: the heart of the type

PyByteArray_Resize is the public entry point and it is the only path that changes the size. It takes the object lock, then runs the strategy in bytearray_resize_lock_held:

c
// Objects/bytearrayobject.c
int
PyByteArray_Resize(PyObject *self, Py_ssize_t requested_size)
{
    int ret;
    Py_BEGIN_CRITICAL_SECTION(self);
    ret = bytearray_resize_lock_held(self, requested_size);
    Py_END_CRITICAL_SECTION();
    return ret;
}

The strategy has several cases:

Same size — nothing to do.
Resize to 0 — reset to the immortal empty bytes.
Current buffer big enough (size + offset <= alloc):
- major downsize (size < alloc/2): shrink the backing PyBytes to exactly size;
- minor downsize: just update ob_size and write the trailing NUL.
Need to grow:
- moderate upsize (size <= alloc * 1.125): over-allocate like list_resize, i.e. alloc = size + (size >> 3) + (size < 9 ? 3 : 6);
- major upsize: allocate exactly size.

Then alloc is clamped to PyByteArray_SIZE_MAX and bytearray_resize_storage does the actual work.

There is also a subtlety: because ob_start may not equal ob_bytes (the data can be shifted inside the buffer after slices/take_bytes), resize first compacts by memmove-ing the live bytes to the base before resizing the PyBytes. On failure it undoes the partial move.

c
// Objects/bytearrayobject.c
static int
bytearray_resize_storage(PyByteArrayObject *self,
                         Py_ssize_t new_size, Py_ssize_t alloc)
{
    _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(self);
    ...
    char *old_start = self->ob_start;
    if (self->ob_start != self->ob_bytes) {
        Py_ssize_t move = Py_MIN(new_size, size);
        memmove(self->ob_bytes, self->ob_start, move);
        self->ob_start = self->ob_bytes;
    }
    if (_PyBytes_ResizeKeepOnError(&self->ob_bytes_object, alloc) < 0) {
        if (old_start != self->ob_bytes && new_size < size) {
            /* undo the compaction */
            Py_ssize_t moved = new_size;
            Py_ssize_t remaining = size - moved;
            memmove(self->ob_bytes + moved, old_start + moved, remaining);
        }
        bytearray_write_trailing_null_byte(self);
        return -1;
    }
    return 0;
}

After storage resize, bytearray_reinit_from_bytes repoints ob_bytes / ob_start at the (possibly new) buffer and refreshes ob_alloc. The trailing NUL is written whenever the allocation differs from the logical size, keeping PyByteArray_AS_STRING C-string safe.

`take_bytes`: zero-copy shrink

bytearray even has a __alloc__-style fast path for splitting off the tail: instead of copying, it can hand the front PyBytes to the caller and re-point the bytearray at a freshly allocated tail. This is the _bytearray_take_bytes logic around line 1640 — it swaps ob_bytes_object so the returned bytes and the surviving bytearray no longer share storage. Worth knowing it exists: it is why repeated pop/take patterns are cheaper than they look.

Locking and free-threading

CPython 3.15 ships the free-threaded (no-GIL) build, and bytearray is a shared, mutable buffer, so almost every method is wrapped in a critical section. The pattern is mechanical:

c
// Objects/bytearrayobject.c
static PyObject *
bytearray_some_op_impl(PyByteArrayObject *self, ...)
{
    _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(self);
    ...
}

// wrapper
PyObject *bytearray_some_op(...) {
    ...
    Py_BEGIN_CRITICAL_SECTION(self);
    ret = bytearray_some_op_impl(self, ...);
    Py_END_CRITICAL_SECTION();
    return ret;
}

Clinic-generated code carries @critical_section annotations that produce these wrappers automatically. Inside the lock, ob_alloc is read with FT_ATOMIC_LOAD_SSIZE_RELAXED and written with FT_ATOMIC_STORE_SSIZE_RELAXED so that even lock-free readers (like __alloc__, which only reads ob_alloc) observe a consistent value:

c
// Objects/bytearrayobject.c (bytearray_alloc)
Py_ssize_t alloc = FT_ATOMIC_LOAD_SSIZE_RELAXED(self->ob_alloc);
if (alloc > 0) {
    alloc += _PyBytesObject_SIZE;
}
return PyLong_FromSsize_t(alloc);

ob_size itself is loaded atomically in PyByteArray_GET_SIZE on free-threaded builds. The net effect: a bytearray is safe to read and resize concurrently with other threads, with resizes serialized per-object and buffer exports acting as a read-lock against mutation.

Type object

c
// Objects/bytearrayobject.c
PyTypeObject PyByteArray_Type = {
    ...
    sizeof(PyByteArrayObject),
    bytearray_dealloc,
    ...
    &bytearray_as_buffer,     /* tp_as_buffer */
    ...
    PyType_GenericAlloc,      /* tp_alloc */
    bytearray_new,            /* tp_new */
    PyObject_Free,            /* tp_free */
};

tp_as_buffer points at bytearray_getbuffer / bytearray_releasebuffer, which is what makes bytearray support the buffer protocol (and therefore memoryview, bytes(ba), file writes, etc.). tp_hash is 0, so bytearray is unhashable, consistent with mutability.

Memory layout summary

text

+----------------------------------+  <- PyObject_New result (PyObject_Free)
| ob_refcnt                        |
+----------------------------------+
| *ob_type  (&PyByteArray_Type)    |
+----------------------------------+
| ob_size   (logical length)       |
+----------------------------------+
| ob_alloc  (atomic; backing size) |
+----------------------------------+
| *ob_bytes   (base of buffer)     |
+----------------------------------+
| *ob_start   (live data start)    |
+----------------------------------+
| ob_exports  (buffer view count)  |
+----------------------------------+
| *ob_bytes_object (PyBytes)       |---+
+----------------------------------+   |
                                       v
                              +-------------------+
                              | PyBytesObject     |
                              | ... ob_sval[]     |  <- the actual bytes
                              +-------------------+

The bytearray object is small and fixed-size; all variable storage lives in the referenced PyBytesObject, which is resized in place (or replaced) during PyByteArray_Resize.

Key takeaways

bytearray is a PyObject_VAR_HEAD plus ob_alloc, ob_bytes, ob_start, ob_exports and ob_bytes_object.
The backing store is a PyBytesObject; ob_bytes/ob_start are pointers into it, so bytearray reuses bytes’ allocator and NUL-termination.
Empty bytearrays reuse the immortal empty bytes, costing only the header.
ob_exports pins the buffer against resize/dealloc while views are live.
Resize uses a list-like over-allocation strategy and compacts ob_start back to ob_bytes first.
Every mutating path is a critical section, with ob_alloc/ob_size read atomically for the free-threaded build.