Opensource internals

CPython's str Implementation in C (Unicode): One Byte per Character When You Can

10 min read Jay Hubbert

PEP 393's flexible string representation lets CPython store each str in 1, 2, or 4 bytes per code point. Here's the struct layout, the ASCII optimization, hashing, interning, and the PyUnicodeWriter path.

TL;DR: A CPython str doesn’t store characters as 4-byte code points. PEP 393’s flexible representation picks the narrowest width that fits the string’s largest code point — 1, 2, or 4 bytes — so ASCII strings are single-byte and single-allocation.

Python’s str is a sequence of Unicode code points. In CPython that’s PyUnicodeObject in Objects/unicodeobject.c, using the “flexible string representation” from PEP 393: each string stores its characters in the smallest width that can hold its largest code point, and the struct itself changes shape depending on whether the string is ASCII and whether it’s “compact”. Let’s trace the representation and the hot paths through the 3.15 source.

The object structs

Every string is at least a PyASCIIObject. Non-ASCII compact strings add a UTF-8 cache header (PyCompactUnicodeObject), and subclasses add a data pointer (PyUnicodeObject):

c 
// Include/cpython/unicodeobject.h
typedef struct {
    PyObject_HEAD
    Py_ssize_t length;          /* Number of code points */
    Py_hash_t hash;             /* Hash value; -1 if not set */
    _Py_ALIGNED_DEF(4, struct _PyUnicodeObject_state) state;
} PyASCIIObject;

typedef struct {
    PyASCIIObject _base;
    Py_ssize_t utf8_length;     /* bytes in utf8, excluding NUL */
    char *utf8;                 /* cached UTF-8 (null-terminated) */
} PyCompactUnicodeObject;

typedef struct {
    PyCompactUnicodeObject _base;
    union {
        void *any;
        Py_UCS1 *latin1;
        Py_UCS2 *ucs2;
        Py_UCS4 *ucs4;
    } data;                     /* Canonical buffer (legacy only) */
} PyUnicodeObject;

The state bitfield carries everything CPython needs to know about the string without inspecting the data:

c 
// Include/cpython/unicodeobject.h
struct _PyUnicodeObject_state {
    unsigned int interned:2;          /* 0 not interned .. 3 static */
    unsigned int kind:3;              /* 1, 2, or 4 bytes per char */
    unsigned int compact:1;           /* one block vs. two */
    unsigned int ascii:1;             /* all chars < 128 */
    unsigned int statically_allocated:1;
    ...
};

kind is one of PyUnicode_1BYTE_KIND, PyUnicode_2BYTE_KIND, PyUnicode_4BYTE_KIND. And interned distinguishes not-interned, mortal interned, immortal interned, and immortal+statically-allocated interned.

Flexible representations (PEP 393)

Instead of always storing 4 bytes per character (the old Py_UNICODE representation), CPython picks the narrowest width that fits the string’s maximum code point:

  • maxchar < 128 -> 1 byte, ASCII
  • maxchar < 256 -> 1 byte (latin1)
  • maxchar < 65536 -> 2 bytes (BMP)
  • otherwise -> 4 bytes (full range up to U+10FFFF)

That’s decided once, at allocation time, by PyUnicode_New:

c 
// Objects/unicodeobject.c
PyObject *
PyUnicode_New(Py_ssize_t size, Py_UCS4 maxchar)
{
    ...
    is_ascii = 0;
    struct_size = sizeof(PyCompactUnicodeObject);
    if (maxchar < 128) {
        kind = PyUnicode_1BYTE_KIND;
        char_size = 1;
        is_ascii = 1;
        struct_size = sizeof(PyASCIIObject);
    }
    else if (maxchar < 256) {
        kind = PyUnicode_1BYTE_KIND;
        char_size = 1;
    }
    else if (maxchar < 65536) {
        kind = PyUnicode_2BYTE_KIND;
        char_size = 2;
    }
    else {
        kind = PyUnicode_4BYTE_KIND;
        char_size = 4;
    }
    ...
    obj = (PyObject *) PyObject_Malloc(struct_size + (size + 1) * char_size);
    ...
    _PyUnicode_LENGTH(unicode) = size;
    _PyUnicode_HASH(unicode) = -1;
    _PyUnicode_STATE(unicode).interned = 0;
    _PyUnicode_STATE(unicode).kind = kind;
    _PyUnicode_STATE(unicode).compact = 1;
    _PyUnicode_STATE(unicode).ascii = is_ascii;
    ...
}

The +1 in the allocation leaves room for a trailing NUL, so the data can be handed to C APIs as a null-terminated buffer. PyUnicode_DATA(op) returns a pointer just past the struct, and PyUnicode_READ(kind, data, i) / PyUnicode_WRITE(kind, data, i, ch) index into it. Because kind lives in the object, code can switch on width without changing the API.

The real win: a Python program’s identifiers, log messages, and JSON payloads are overwhelmingly ASCII. Those strings cost one byte per character and one allocation.

Compact vs. legacy

A string is compact when the character buffer lives in the same allocation as the struct (compact == 1). PyUnicode_New always produces compact strings, and the data pointer is computed by walking past the struct:

c 
// Include/cpython/unicodeobject.h
static inline void* _PyUnicode_COMPACT_DATA(PyObject *op) {
    if (PyUnicode_IS_ASCII(op)) {
        return _Py_STATIC_CAST(void*, (_PyASCIIObject_CAST(op) + 1));
    }
    return _Py_STATIC_CAST(void*, (_PyCompactUnicodeObject_CAST(op) + 1));
}

Legacy strings (compact == 0) are used by subclasses of str. They keep a separate data.any pointer (PyUnicodeObject.data) allocated with PyMem_Malloc, because the subclass may have a different layout or size. unicode_dealloc frees that second block when present:

c 
// Objects/unicodeobject.c
if (_PyUnicode_HAS_UTF8_MEMORY(unicode)) {
    PyMem_Free(_PyUnicode_UTF8(unicode));
}
if (!PyUnicode_IS_COMPACT(unicode) && _PyUnicode_DATA_ANY(unicode)) {
    PyMem_Free(_PyUnicode_DATA_ANY(unicode));
}
Py_TYPE(unicode)->tp_free(unicode);

So compact strings are one PyObject_Malloc (and one free) for both header and data; legacy strings are two.

ASCII optimization

ASCII strings are the overwhelmingly common case, so CPython gives them a dedicated, smaller struct (PyASCIIObject instead of PyCompactUnicodeObject). No utf8/utf8_length fields are needed, because for ASCII the canonical 1-byte data is the UTF-8 representation. The ascii flag records this, and _PyUnicode_COMPACT_DATA uses the smaller offset.

Latin-1 characters (U+0080..U+00FF) still use 1 byte per character, but they aren’t ASCII, so they carry the UTF-8 cache fields.

There’s also a singleton for every Latin-1 code point, used as a fast path for single-character strings:

c 
// Include/internal/pycore_global_strings.h
#define _Py_LATIN1_CHR(CH) \
    ((CH) < 128 \
     ? (PyObject*)&_Py_SINGLETON(strings).ascii[(CH)] \
     : (PyObject*)&_Py_SINGLETON(strings).latin1[(CH) - 128])

intern_common (below) collapses any length-1 1-byte string to LATIN1(ch), so "a" and chr(97) are the same object.

UTF-8 caching

For non-ASCII strings, the UTF-8 representation is computed lazily and cached in utf8/utf8_length (inside PyCompactUnicodeObject). ASCII strings don’t need a cache because their 1-byte data is already valid UTF-8, so PyUnicode_AsUTF8 returns the data pointer directly.

_PyUnicode_HAS_UTF8_MEMORY tracks whether the cache owns a separate allocation that must be freed on dealloc (see above).

Hashing

The hash is cached in ob_hash (-1 means “not computed yet”). unicode_hash first checks the cache, then hashes the raw character buffer:

c 
// Objects/unicodeobject.c
static Py_hash_t
unicode_hash(PyObject *self)
{
    Py_uhash_t x;
    Py_hash_t hash = PyUnicode_HASH(self);
    if (hash != -1) {
        return hash;
    }
    x = Py_HashBuffer(PyUnicode_DATA(self),
                      PyUnicode_GET_LENGTH(self) * PyUnicode_KIND(self));

    PyUnicode_SET_HASH(self, x);
    return x;
}

Because the buffer is contiguous bytes in canonical form, hashing is just hashing length * kind bytes — the same bytes as the UTF-8 for ASCII, which is why an ASCII str hashes the same as the equivalent bytes. The result is stored back, so subsequent lookups are O(1).

Interning

Interning deduplicates strings and is central to attribute/identifier lookup. CPython keeps two caches: a process-wide hashtable (INTERNED_STRINGS) for statically allocated strings, and a per-interpreter dict for runtime ones. intern_common is the shared implementation:

c 
// Objects/unicodeobject.c
static PyObject *
intern_common(PyInterpreterState *interp, PyObject *s, bool immortalize)
{
    ...
    /* Is it already interned? */
    switch (PyUnicode_CHECK_INTERNED(s)) { ... }

    /* if it's a short string, get the singleton */
    if (PyUnicode_GET_LENGTH(s) == 1 &&
                PyUnicode_KIND(s) == PyUnicode_1BYTE_KIND) {
        PyObject *r = LATIN1(*(unsigned char*)PyUnicode_DATA(s));
        Py_DECREF(s);
        return r;
    }

    /* Look in the global cache now. */
    {
        PyObject *r = (PyObject *)_Py_hashtable_get(INTERNED_STRINGS, s);
        if (r != NULL) {
            Py_DECREF(s);
            return Py_NewRef(r);
        }
    }

    /* Do a setdefault on the per-interpreter cache. */
    PyObject *interned = get_interned_dict(interp);
    ...
    int res = PyDict_SetDefaultRef(interned, s, s, &t);
    ...
    /* NOT_INTERNED -> INTERNED_MORTAL: the two dict references
       (key and value) are not counted in ob_refcnt. */
    if (!_Py_IsImmortal(s)) {
        Py_DECREF(s);
        Py_DECREF(s);
    }
    _PyUnicode_STATE(s).interned = SSTATE_INTERNED_MORTAL;
    if (immortalize) {
        immortalize_interned(s);
    }
    return s;
}

Two details to internalize: the interned dict holds two references (key and value) that are not counted in ob_refcnt, so unicode_dealloc restores the count before popping. And immortal interned strings are promoted via immortalize_interned, which sets the immortal flag and the SSTATE_INTERNED_IMMORTAL state. Empty strings come from the immortal _Py_STR(empty) singleton.

Building strings: PyUnicodeWriter

Because str is immutable, constructing one incrementally is awkward. CPython provides PyUnicodeWriter, which keeps a mutable buffer and grows it:

c 
// Include/cpython/unicodeobject.h
typedef struct PyUnicodeWriter PyUnicodeWriter;
PyAPI_FUNC(PyUnicodeWriter*) PyUnicodeWriter_Create(Py_ssize_t length);
PyAPI_FUNC(int) PyUnicodeWriter_WriteChar(PyUnicodeWriter *writer, Py_UCS4 ch);
PyAPI_FUNC(int) PyUnicodeWriter_WriteStr(PyUnicodeWriter *writer, PyObject *obj);
PyAPI_FUNC(PyObject*) PyUnicodeWriter_Finish(PyUnicodeWriter *writer);

Internally this is backed by a _PyUnicodeWriter struct holding buffer, data, kind, maxchar, size, pos, plus min_length, min_char and an overallocate flag.

_PyUnicodeWriter_Prepare grows the buffer only when needed, and upgrades kind (1 -> 2 -> 4 bytes) if a larger character is written — reallocating the whole buffer. Finish returns the resulting immutable str and clears the writer. This is the path used by str.join, formatting, and decoding.

Conversion to and from UTF-8

PyUnicode_FromString / PyUnicode_FromStringAndSize decode UTF-8 into the narrowest kind; PyUnicode_AsUTF8AndSize encodes (and caches) UTF-8.

For ASCII inputs the kind stays 1 byte and no transcoding beyond validation is needed. For higher code points the encoder chooses 2/4 bytes and the decoder validates UTF-8 sequences.

Surrogate pairs are not combined. Python strings are sequences of code points, so lone surrogates remain individual characters — useful for round-tripping Windows file names.

Operations

Reading is a typed, unchecked index into the canonical buffer:

c 
// Include/cpython/unicodeobject.h
static inline Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index)
{
    int kind = PyUnicode_KIND(unicode);
    if (kind == PyUnicode_1BYTE_KIND)
        return PyUnicode_1BYTE_DATA(unicode)[index];
    if (kind == PyUnicode_2BYTE_KIND)
        return PyUnicode_2BYTE_DATA(unicode)[index];
    return PyUnicode_4BYTE_DATA(unicode)[index];
}

Mutation is gated. _PyUnicode_IsModifiable checks that the object is compact, an exact str, not interned, and has refcount 1; otherwise PyUnicode_CopyCharacters allocates a fresh destination.

PyUnicode_CopyCharacters also performs kind conversion when the destination is wider than the source, and falls back to memcpy when both sides share the same kind. Most “operations” on strings (slicing, concatenation, case conversion, replace) therefore allocate a new PyUnicode_New and copy converted code points into it, choosing the smallest kind that fits the result’s maximum character.

That’s why str is immutable from Python’s point of view even though the C buffer is writable during construction.

Concatenation and repetition

PyUnicode_Concat handles the empty-string fast paths (_PyUnicode_GetEmpty), then allocates a result of len(a)+len(b) and copies both halves with PyUnicode_CopyCharacters, upgrading kind to fit the wider of the two inputs. PyUnicode_Repeat does the same with a loop, overallocating slightly and trimming. Neither ever mutates its inputs.

Comparison

unicode_compare_eq first compares length and (for ASCII) pointer equality, then does a memcmp-style byte comparison over the canonical buffers when kinds match; otherwise it falls back to code-point comparison.

Because the canonical representation is unique for a given string, two equal strings of the same kind compare in O(n) bytes with no per-character branching. Ordering (str.__lt__) compares code points lexicographically after the same kind normalization.

Identifiers and the global string table

CPython pre-creates a large table of interned strings (identifiers, literals, single characters) in Include/internal/pycore_global_strings.h, exposed via _Py_STR(name) and _Py_ID(name). These are statically allocated and marked SSTATE_INTERNED_IMMORTAL_STATIC, so they’re shared across interpreters and never freed.

Attribute and variable name lookups hit this table first (via _Py_hashtable_get(INTERNED_STRINGS, ...) in intern_common), which is why obj.attr and globals["name"] are fast: the name is already an interned singleton.

Memory layout

A compact ASCII string — single allocation, smallest case:

text 
+-------------------------------+  <- PyObject_Malloc
| ob_refcnt                     |
+-------------------------------+
| *ob_type  (&PyUnicode_Type)   |
+-------------------------------+
| length                        |
+-------------------------------+
| hash  (-1 until computed)     |
+-------------------------------+
| state (interned/kind/compact/ |
|        ascii/statically_alloc) |
+-------------------------------+
| data[0]  (Py_UCS1, ASCII)     |
| data[1]                       |
| ...                           |
| data[size] = '\0'             |
+-------------------------------+

Compact non-ASCII adds utf8_length and utf8 between the header and data; legacy (subclass) strings instead store data.any pointing at a separately allocated buffer.

For an ASCII string there’s no second block and no separate UTF-8 cache — the data is the UTF-8.

Key takeaways

  • str is a PyObject whose layout is PyASCIIObject / PyCompactUnicodeObject / PyUnicodeObject depending on ASCII-ness and compactness.
  • PEP 393 stores characters in the narrowest of 1/2/4 bytes per code point, chosen at allocation from maxchar.
  • Compact strings put header and data in one PyObject_Malloc block; legacy (subclass) strings use a second block for data.
  • ASCII strings get the smallest struct and share data with their UTF-8 form; single-character Latin-1 strings are immortal singletons.
  • Hashing caches in ob_hash and hashes the raw canonical buffer; interning deduplicates via a global hashtable plus a per-interpreter dict, with immortal promotion.

Further reading

  • Objects/unicodeobject.c – allocation, hashing, interning, dealloc.
  • Include/cpython/unicodeobject.h – structs, state bitfield, access macros.
  • Include/internal/pycore_global_strings.h – Latin-1/ASCII singletons.

PythonCpythonInternalsStrUnicode