How CPython Implements bytes in C

Python’s bytes is an immutable sequence of bytes. In CPython it is implemented by PyBytesObject in Objects/bytesobject.c, a variable-size object whose payload is a single trailing char array (ob_sval). This article traces the representation and the hot paths through the 3.15 source.

The object struct

Every bytes object is a PyVarObject: the size lives in ob_size (the number of bytes, not counting the trailing NUL).

c
// Include/cpython/bytesobject.h
typedef struct {
    PyObject_VAR_HEAD
    Py_DEPRECATED(3.11) Py_hash_t ob_shash;
    char ob_sval[1];

    /* Invariants:
     *     ob_sval contains space for 'ob_size+1' elements.
     *     ob_sval[ob_size] == 0.
     *     ob_shash is the hash of the byte string or -1 if not computed yet.
     */
} PyBytesObject;

ob_sval is declared with size 1 and the allocator over-allocates the trailing storage, exactly like PyLongObject’s digit array. The ob_shash field caches the hash; -1 means “not computed yet” (and a real hash of -1 is remapped to -2 by Py_HashBuffer).

Memory layout and `_PyBytesObject_SIZE`

Because ob_sval[1] already accounts for the first byte, the basic size used for allocation is one byte less than sizeof(PyBytesObject):

c
// Include/internal/pycore_bytesobject.h
#define _PyBytesObject_SIZE (offsetof(PyBytesObject, ob_sval) + 1)

That saves 3 or 7 bytes per object on typical systems. bytesobject.c aliases it:

c
// Objects/bytesobject.c
#define PyBytesObject_SIZE _PyBytesObject_SIZE

So an object of length n is allocated as PyBytesObject_SIZE + n bytes, with the final ob_sval[n] byte reserved for the terminating '\0'.

text

+---------------------------+  <- PyObject_Malloc result
| ob_refcnt                 |
+---------------------------+
| *ob_type  (&PyBytes_Type) |
+---------------------------+
| ob_size   (length n)      |
+---------------------------+
| ob_shash  (cached hash)   |
+---------------------------+
| ob_sval[0]                |
| ob_sval[1]                |
| ...                       |
| ob_sval[n-1]              |
| ob_sval[n] == '\0'        |
+---------------------------+

The trailing NUL lets bytes objects be passed directly to C APIs expecting null-terminated strings, while ob_size keeps the true (possibly embedded-NUL) length.

Singletons: empty and single characters

There is no general interning of bytes, but two families of immortal singletons are created at runtime init:

c
// Objects/bytesobject.c
#define CHARACTERS _Py_SINGLETON(bytes_characters)
#define CHARACTER(ch) \
     ((PyBytesObject *)&(CHARACTERS[ch]))
#define EMPTY (&_Py_SINGLETON(bytes_empty))

bytes_empty is the immortal empty bytes object (b""); PyBytes_FromStringAndSize returns it for size 0.
bytes_characters[256] are immortal single-byte objects for every byte value 0..255. PyBytes_FromStringAndSize and PyBytes_FromString return CHARACTER(*str & 255) when size == 1.

c
// Objects/bytesobject.c
if (size == 1 && str != NULL) {
    op = CHARACTER(*str & 255);
    assert(_Py_IsImmortal(op));
    return (PyObject *)op;
}

These are immortal (PEP 683), so they are never freed and their refcounts never hit zero. This is why b"a" is b"a" is always true for a single byte, but b"ab" is b"ab" is generally false.

Allocation

The single allocation routine is _PyBytes_FromSize:

c
// Objects/bytesobject.c
static PyObject *
_PyBytes_FromSize(Py_ssize_t size, int use_calloc)
{
    PyBytesObject *op;
    assert(size >= 0);

    if (size == 0) {
        return bytes_get_empty();
    }

    if ((size_t)size > (size_t)PY_SSIZE_T_MAX - PyBytesObject_SIZE) {
        PyErr_SetString(PyExc_OverflowError,
                        "byte string is too large");
        return NULL;
    }

    /* Inline PyObject_NewVar */
    if (use_calloc)
        op = (PyBytesObject *)PyObject_Calloc(1, PyBytesObject_SIZE + size);
    else
        op = (PyBytesObject *)PyObject_Malloc(PyBytesObject_SIZE + size);
    if (op == NULL) {
        return PyErr_NoMemory();
    }
    _PyObject_InitVar((PyVarObject*)op, &PyBytes_Type, size);
    set_ob_shash(op, -1);
    if (!use_calloc) {
        op->ob_sval[size] = '\0';
    }
    return (PyObject *) op;
}

Points worth noting:

size 0 collapses to the empty singleton.
use_calloc zeroes the whole allocation (so the trailing NUL and data are already zero); otherwise only ob_sval[size] is written.
There is no freelist for bytes objects (unlike int’s compact freelist). Allocation always goes through PyObject_Malloc/Calloc.
PyBytes_FromString skips _PyBytes_FromSize and inlines the PyObject_Malloc + memcpy(str, size+1) for the common null-terminated case.

PyBytes_FromStringAndSize adds the single-byte shortcut and copies str into ob_sval when str != NULL.

Hashing

bytes_hash caches its result in ob_shash:

c
// Objects/bytesobject.c
static Py_hash_t
bytes_hash(PyObject *self)
{
    PyBytesObject *a = _PyBytes_CAST(self);
    Py_hash_t hash = get_ob_shash(a);
    if (hash == -1) {
        /* Can't fail */
        hash = Py_HashBuffer(a->ob_sval, Py_SIZE(a));
        set_ob_shash(a, hash);
    }
    return hash;
}

get_ob_shash/set_ob_shash use relaxed atomics on the free-threaded build (Py_GIL_DISABLED), plain field access otherwise. Py_HashBuffer (in Python/pyhash.c) uses a fast inline DJBX33A path for very short inputs (Py_HASH_CUTOFF) and the configured hash function (SipHash by default) otherwise, and remaps a result of -1 to -2.

Because the hash is cached, _PyBytes_Resize refuses to mutate an object whose hash has already been computed (see below).

Resize (the one mutation escape hatch)

Bytes are immutable from Python, but the C API allows in-place mutation while an object is being built, provided it is uniquely referenced and has not been hashed:

c
// Objects/bytesobject.c
int
_PyBytes_ResizeKeepOnError(PyObject **pv, Py_ssize_t newsize)
{
    PyObject *v = *pv;
    ...
    Py_ssize_t oldsize = PyBytes_GET_SIZE(v);
    if (oldsize == newsize) {
        return 0;
    }

    if (oldsize == 0) {          /* was the empty singleton */
        result = _PyBytes_FromSize(newsize, 0);
        ...
    }
    if (newsize == 0) {          /* becoming empty */
        *pv = bytes_get_empty();
        Py_DECREF(v);
        return 0;
    }

    if (!_PyObject_IsUniquelyReferenced(v)
        || get_ob_shash((PyBytesObject *)v) != -1)
    {
        /* Allocate and then copy so we don't get a shared immortal
           one-character singleton! */
        result = _PyBytes_FromSize(newsize, 0);
        ...
        memcpy(PyBytes_AS_STRING(result), PyBytes_AS_STRING(v),
               Py_MIN(oldsize, newsize));
        *pv = result;
        Py_DECREF(v);
        return 0;
    }

    return bytes_resize_inplace(pv, newsize);
}

bytes_resize_inplace does the actual PyObject_Realloc and re-initializes the object header:

c
// Objects/bytesobject.c
static inline int
bytes_resize_inplace(PyObject **pv, Py_ssize_t newsize)
{
    assert(Py_REFCNT(v) == 1);
    assert(PyBytes_GET_SIZE(v) >= 1);
    assert(newsize >= 1);
    assert(_PyBytes_IsMutable(v));

    ...
    PyObject *result = PyObject_Realloc(v, PyBytesObject_SIZE + newsize);
    ...
    *pv = result;
    v = result;
    _Py_NewReferenceNoTotal(v);
    PyBytesObject *sv = (PyBytesObject *)v;
    Py_SET_SIZE(sv, newsize);
    sv->ob_sval[newsize] = '\0';
    ...
}

_PyBytes_Resize (the older variant) additionally sets *pv = NULL and decrefs on failure. _PyBytes_IsMutable is a debug-only check that the object is uniquely referenced, not immortal, and not one of the empty/single-byte singletons.

PyBytesWriter (in the same file) builds on this: it starts with a 256-byte stack buffer (small_buffer) and promotes to a heap bytes object via the same resize machinery, which is how bytes formatting avoids many small allocations.

Operations

A few representative hot paths:

Concatenation (_PyBytes_Concat) gets buffers from both operands, handles empty inputs by returning the other operand, and otherwise allocates a new bytes object and memcpys both parts.
Repeat (bytes_repeat) returns the same object when n == 1 or the size is 0, otherwise allocates and uses _PyBytes_Repeat (a doubling copy) to fill the result.
Slicing (bytes_subscript) returns the empty singleton for empty results, returns Py_NewRef(self) when the slice is the whole object, and uses PyBytes_FromStringAndSize(ptr, slicelength) otherwise.
Indexing returns a cached single-byte singleton via _PyLong_FromUnsignedChar.
Buffer protocol (bytes_buffer_getbuffer) exposes ob_sval with PyBuffer_FillInfo, so bytes participates in the buffer API with no copy.

Key takeaways

bytes is a PyVarObject whose ob_size is the length and whose payload is ob_sval, a char array of size + 1 bytes ending in a NUL.
Allocation size is _PyBytesObject_SIZE + n, saving a few bytes over sizeof(PyBytesObject).
There is no freelist; allocation always uses PyObject_Malloc/Calloc.
The empty bytes and all 256 single-byte objects are immortal singletons, returned by the construction functions.
The hash is lazily computed and cached in ob_shash.
In-place resize is allowed only for uniquely referenced, unhashed, non-singleton objects; otherwise a copy is made.