Strings on Opensource internals

ClickHouse Packs Short Strings Into Hash Table Keys: No Allocation Per Key

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse never stores a std::string in a hash table cell. It splits keys into size classes and packs strings of up to 24 bytes into plain integers, so your GROUP BY allocates nothing per key.

Strings are the most common GROUP BY key you’ll ever write. They’re also the worst case for a hash table.

Put a std::string in a cell and you get one heap allocation per distinct key. Put a std::string_view in a cell and the bytes have to outlive the table — which for an aggregation means copying them into an arena first.

ClickHouse’s answer is a hash table that is not a hash table. Instead of one table with a variable-size key, StringHashTable keeps five separate tables, one per key size class, and packs short keys into fixed-width integers.

The payoff: every cell is the same size, no key ever allocates, and the compiler can dispatch on key length at compile time.

The whole thing lives in src/Common/HashTable/StringHashTable.h, with the map specialisation in StringHashMap.h.

Five tables, one per size class

The composite table is five sub-tables:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:234
template <typename SubMaps>
class StringHashTable : private boost::noncopyable
{
protected:
 static constexpr size_t NUM_MAPS = 5;
 // Map for storing empty string
 using T0 = typename SubMaps::T0;

 // Short strings are stored as numbers
 using T1 = typename SubMaps::T1;
 using T2 = typename SubMaps::T2;
 using T3 = typename SubMaps::T3;

 // Long strings are stored as std::string_view along with saved hash
 using Ts = typename SubMaps::Ts;
 using Self = StringHashTable;

and the storage:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:254
 T0 m0;
 T1 m1;
 T2 m2;
 T3 m3;
 Ts ms;

Here’s what lives where:

Table	Key type	Holds
`m0`	—	the empty string, in a dedicated slot
`m1`	`StringKey8` (`UInt64`)	1..8 bytes
`m2`	`StringKey16` (`UInt128`)	9..16 bytes
`m3`	`StringKey24`	17..24 bytes
`ms`	`std::string_view` + saved hash	25+ bytes, or short strings with a trailing zero

And the key types are just integers:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:10
using StringKey8 = UInt64;
using StringKey16 = UInt128;
struct StringKey24
{
 UInt64 a;
 UInt64 b;
 UInt64 c;

 bool operator==(const StringKey24 rhs) const { return a == rhs.a && b == rhs.b && c == rhs.c; }
};

So a 20-byte string becomes three UInt64s and nothing else. No length field. No pointer. No allocation.

The length is implicit — it’s recovered from the position of the first zero byte.

The packing: how a string becomes a number

The heart of the file is dispatchOnKeyClass. It classifies a key, packs it, hashes it, and calls a callback with the right sub-table. It’s written for speed, and the comment says so out loud:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:357
 // Dispatch is written in a way that maximizes the performance:
 // 1. Always memcpy 8 times bytes
 // 2. Use switch case extension to generate fast dispatching table
 // 3. Funcs are named callables that can be force_inlined
 //
 /// Note: the key bytes must be readable in 8-byte chunks (a `std::string` cannot be passed
 /// here, while e.g. `ColumnString::getDataAt` can), because the packing copies 8 bytes at a
 /// time.
 template <typename Resolver, typename KeyHolder, typename HashProvider, typename Func>
 static auto ALWAYS_INLINE dispatchOnKeyClass(Resolver && resolver, KeyHolder && key_holder, HashProvider && hash_of, Func && func)
 {
 const auto & x = keyHolderGetKey(key_holder);
 const size_t sz = x.size();
 if (sz == 0)
 {
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 0>{}, 0), VoidKey{}, 0);
 }

 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

 const char * p = x.data();
 // pending bits that needs to be shifted out
 const char s = (-sz & 7) * 8;

The switch ((sz - 1) >> 3) that follows turns the length into one of four cases, and the compiler turns that into a jump table or a chain of comparisons:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:394
 switch ((sz - 1) >> 3)
 {
 case 0: // 1..8 bytes
 {
 // first half page
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)
 {
 memcpy(&n[0], p, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] &= -1ULL >> s;
 else
 n[0] &= -1ULL << s;
 }
 else
 {
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[0], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[0] >>= s;
 else
 n[0] <<= s;
 }
 const size_t res = hash_of(k8);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 1>{}, res), k8, res);
 }

Three tricks are hiding in those twenty lines. This is the part people get wrong the first time they read it.

Trick one: the trailing-zero test. A fixed-size string key encodes its length implicitly, by the position of the first zero byte. That means a string whose last byte is zero cannot be represented at all: "ab\0" and "ab" would pack to the same 8 bytes. So such strings get pushed to the generic table before the switch:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:376
 if (x[sz - 1] == 0)
 {
 // Strings with trailing zeros are not representable as fixed-size
 // string keys. Put them to the generic table.
 const size_t res = hash_of(x);
 return func(resolver(std::integral_constant<size_t, 4>{}, res), std::forward<KeyHolder>(key_holder), res);
 }

Trick two: the mask/shift pair. s = (-sz & 7) * 8 is the number of padding bits you need to round the length up to a multiple of 8. In the first branch, the 8 bytes are read forwards from the start of the string and the padding bits are cleared with a mask. In the second branch, the 8 bytes are read backwards from the end of the string and the padding bits are shifted out.

Trick three: the “first half page” rule. Reading 8 bytes from a 3-byte string reads 5 bytes past its end. That’s only safe if those bytes are mapped. The condition

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:399
 if ((reinterpret_cast<uintptr_t>(p) & 2048) == 0)

tests whether the address is 2048-byte aligned. If it is, at least 2048 bytes of the page are still ahead of it, so reading 8 bytes forward cannot fault. Otherwise the code reads backwards instead, ending exactly at the last byte of the string.

The bytes before the string that this reads are then shifted out — the >>= s discards exactly the (-sz & 7) bytes that belong to the previous string or to the column’s offset array. So the garbage is read but never used.

The 9..16 and 17..24 byte cases use the same trick with two and three words:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:420
 case 1: // 9..16 bytes
 {
 memcpy(&n[0], p, 8);
 const char * lp = x.data() + x.size() - 8;
 memcpy(&n[1], lp, 8);
 if constexpr (std::endian::native == std::endian::little)
 n[1] >>= s;
 else
 n[1] <<= s;
 const size_t res = hash_of(k16);
 keyHolderDiscardKey(key_holder);
 return func(resolver(std::integral_constant<size_t, 2>{}, res), k16, res);
 }

Notice that the first word is always read forwards from p — for a 9-byte string that’s inside the string — while the last word is read backwards from the end. The overlap between them is harmless: the bytes in the middle get read twice with the same values.

Unpacking: getting the string back

Going the other way, the length is recovered from the number of leading zero bytes:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:21
inline std::string_view ALWAYS_INLINE toStringView(const StringKey8 & n)
{
 chassert(n != 0);
 if constexpr (std::endian::native == std::endian::big)
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countr_zero(n) >> 3)};
 else
 return {reinterpret_cast<const char *>(&n), 8ul - (std::countl_zero(n) >> 3)};
}

countl_zero(n) >> 3 is the number of whole zero bytes at the end of the word, so 8 - that is the string length. On a big-endian machine the zero bytes sit at the other end, hence the countr_zero variant.

The empty-cell rule, specialised per size class

The “empty cell is a zeroed cell” rule from the generic table needs a sensible notion of “zero” for a packed string key. For StringKey16 the rule is “the last word is zero”:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:20
template <typename TMapped>
struct StringHashMapCell<StringKey16, TMapped> : public HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>
{
 using Base = HashMapCell<StringKey16, TMapped, StringHashTableHash, HashTableNoState>;
 using value_type = typename Base::value_type;
 using Base::Base;
 static constexpr bool need_zero_value_storage = false;
 bool isZero(const HashTableNoState & state) const { return isZero(this->value.first, state); }

 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey16 & key, const HashTableNoState &) { return key.items[1] == 0; }
 void setZero() { this->value.first.items[1] = 0; }

The reasoning is the same as in the generic table. A packed key with a zero last word would have to encode the length “16” with no room for a terminating zero byte, so such a key cannot exist. The encoding is injective on all valid keys, and the all-zero pattern is unreachable. That makes it safe to use as the empty marker.

Same for StringKey24, where the marker is the third word:

 cpp
 
 
// src/Common/HashTable/StringHashMap.h:49
 // Zero means unoccupied cells in hash table. Use key with last word = 0 as
 // zero keys, because such keys are unrepresentable (no way to encode length).
 static bool isZero(const StringKey24 & key, const HashTableNoState &)
 { return key.c == 0; }
 void setZero() { this->value.first.c = 0; }

The generic std::string_view cell can’t do this. A string_view is a pointer and a length, and a zeroed one is a null pointer with length 0 — which is a legal key, namely the empty string. So the generic cell declares that it needs no zero storage, and the empty string gets its own slot instead:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:134
template <typename Cell>
struct StringHashTableEmpty // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init) - `zero_value_storage` is raw storage for placement new, only written when `has_zero` becomes true
{
 using Self = StringHashTableEmpty;

 bool has_zero = false;
 alignas(Cell) std::byte zero_value_storage[sizeof(Cell)];

public:
 bool hasZero() const { return has_zero; }

And that empty slot isn’t a hash table at all — it’s a single cell with an emplace that ignores the key entirely:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:172
 template <typename KeyHolder>
 void ALWAYS_INLINE emplace(KeyHolder &&, LookupResult & it, bool & inserted, size_t = 0)
 {
 if (!hasZero())
 {
 setHasZero();
 inserted = true;
 }
 else
 inserted = false;
 it = zeroValue();
 }

Hashing: CRC32C where it can, CityHash where it can’t

The hash function is chosen per class, and for the fixed-size classes it’s the hardware CRC32C instruction:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:46
struct StringHashTableHash
{
#if defined(__SSE4_2__)
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key);
 return res;
 }
 size_t ALWAYS_INLINE operator()(StringKey16 key) const
 {
 size_t res = -1ULL;
 res = _mm_crc32_u64(res, key.items[0]);
 res = _mm_crc32_u64(res, key.items[1]);
 return res;
 }

with an AArch64 CRC32 path, an s390x path, and CityHash as the portable fallback:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:114
#else
 size_t ALWAYS_INLINE operator()(StringKey8 key) const
 {
 return CityHash_v1_0_2::CityHash64(reinterpret_cast<const char *>(&key), 8);
 }

CRC32 is a poor hash function in the abstract — the header of the generic hash file says so — but it’s one instruction per 8 bytes with a three-cycle latency, and the table is small enough to fit in cache most of the time.

There’s another requirement, and it’s the one that actually constrains the design: the hash has to be consistent with the packing. The hash is computed over the packed word, not over the original bytes, so "abc" and "abc\0..." can’t collide by accident. They hash different words.

The lookup result: why you never see the key

The string table doesn’t return a pointer to a cell. It returns a pointer to the mapped value, wrapped in a small type:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:214
template <typename Mapped>
struct StringHashTableLookupResult
{
 Mapped * mapped_ptr;
 StringHashTableLookupResult() {} /// NOLINT
 StringHashTableLookupResult(Mapped * mapped_ptr_) : mapped_ptr(mapped_ptr_) {} /// NOLINT
 StringHashTableLookupResult(std::nullptr_t) {} /// NOLINT
 const VoidKey getKey() const { return {}; } /// NOLINT
 auto & getMapped() { return *mapped_ptr; }
 auto & operator*() { return *this; }
 auto & operator*() const { return *this; }
 auto * operator->() { return this; }
 auto * operator->() const { return this; }
 explicit operator bool() const { return mapped_ptr; }

The reason is that the key lives in the cell, and the caller usually only wants the aggregate state. But the key is still there — just in a different sub-table with a different representation. Returning a pointer to it would mean returning a pointer to a StringKey8 or to a string_view, which have different types depending on the size class.

So the key is erased from the interface: getKey() returns a VoidKey.

Reserving and growing: where the heuristic leaks

The string table grows differently from the generic one:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:207
template <size_t initial_size_degree = 8>
struct StringHashTableGrower : public HashTableGrowerWithPrecalculation<initial_size_degree>
{
 // Smooth growing for string maps
 void increaseSize() { this->increaseSizeDegree(1); }
};

The generic grower doubles the buffer on overflow. The string grower grows by a factor of two in the exponent, i.e. by one power of two — the same thing, but expressed as a single degree increment so it composes with the precalculated mask.

Reserving is divided evenly over the four real sub-maps, because the caller only knows the total number of keys:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:317
 explicit StringHashTable(size_t reserve_for_num_elements)
 : m1{reserve_for_num_elements / 4}
 , m2{reserve_for_num_elements / 4}
 , m3{reserve_for_num_elements / 4}
 , ms{reserve_for_num_elements / 4}
 {
 }

That guess is often wrong. A table of 20-byte strings uses m3 only, and m1, m2, ms reserve space they will never touch.

The aggregator knows this, and refuses to pre-reserve the string tables:

 cpp
 
 
// src/Interpreters/Aggregator.cpp:5214
 /// `StringHashTable::reserve` splits the hint evenly over its four size-class sub-maps,
 /// while a real key set concentrates in one of them, so it is not reserved.
 constexpr bool can_reserve = requires { dst.reserve(size_t{}); } && !requires { dst.emptyStringSlot(); };

And the emptyStringSlot() method exists precisely so that this requires expression can tell a StringHashTable from an ordinary hash table:

 cpp
 
 
// src/Common/HashTable/StringHashTable.h:312
 /// The dedicated slot of the empty-string key, which the sub-table dispatch hashes to 0.
 T0 & emptyStringSlot() { return m0; }

Where it’s used: three string-key aggregation variants

The string map is one of the standard aggregation data variants:

 cpp
 
 
// src/Interpreters/AggregatedData.h:53
using AggregatedDataWithPackedStringKey = HashMap<PackedStringRef, AggregateDataPtr>;
using AggregatedDataWithShortStringKey = StringHashMap<AggregateDataPtr>;
using AggregatedDataWithStringKey = HashMapWithSavedHash<std::string_view, AggregateDataPtr>;

Three flavours of string key, three tables:

AggregatedDataWithPackedStringKey — a PackedStringRef, a pointer plus a length, with the payload inline for short strings;
AggregatedDataWithShortStringKey — the composite StringHashMap of this article;
AggregatedDataWithStringKey — a plain std::string_view map with a saved hash, where the key bytes live in an arena.

The StringHashMap variant is the interesting one, because it’s the only one where the key doesn’t live anywhere else. The string is in the table, as three machine words.

Trade-offs: what you win and what you pay

What the packed string table buys you:

No allocation per key. A 20-byte key is three UInt64s inside the cell. Nothing is copied into an arena, nothing is freed.
A uniform cell size. Every cell of m1 is 8 bytes of key plus the mapped value; every cell of m3 is 24 bytes plus the mapped value. The table can be scanned, and the compiler can generate good code for it.
A compile-time dispatch on length. dispatchOnKeyClass is a template over the size class, so each branch is specialised: the m1 path hashes one word, the m3 path hashes three.
A hash over the packed form. The hash is computed over exactly the bytes that are stored, so the stored key and the hashed key can never disagree.

What it costs you:

The bytes of the key are duplicated. The cell holds the packed key, and if the caller needs the original string it has to be reconstructed with toStringView.
The trailing-zero case is a wart. A string ending in \0 — which happens with binary data, and with FixedString columns padded with zeros — can’t use the packed classes and falls back to the generic table, where the key is a string_view into the arena.
The reservation heuristic is bad. Reserving n/4 for each of the four sub-maps wastes memory whenever the keys concentrate in one class, which is the normal case.
The empty string needs its own slot, and the key type of the generic sub-table isn’t the same as the key type of the packed ones, so the composite has to be written as a template over a SubMaps bundle rather than as a plain HashTable.

This is a good example of a general principle: when the key type has a size distribution, splitting the table by size class turns a variable-size-cell problem into a set of fixed-size-cell problems, and lets the compiler specialise each one.

Wrapping up

Worth internalizing: a GROUP BY over strings is a memory-bound benchmark before it’s a CPU-bound one. ClickHouse’s whole trick here is to make the key be the data — no arena, no pointer chasing, no allocator traffic.

So if you’re tuning an aggregation and it feels allocation-heavy, check what your key type maps to. Short, non-zero-terminated strings hit the fast path. FixedString padded with zeros and raw binary blobs do not — and that’s a silent cliff, not an error message.

Next time you’re profiling, watch for memcpy traffic coming out of the aggregator rather than out of the storage layer. That’s the signature of this code path not doing what you expect.

CPython's Template String Implementation in C (t-strings): Interpolation as Data

Jay Hubbert — Tue, 20 Oct 2026 09:00:00 +0000

TL;DR: Python 3.14+ template strings (t"...") don’t produce a str. They produce a string.templatelib.Template carrying two parallel sequences: the literal string pieces (strings) and the evaluated Interpolation objects (interpolations). Both types are C types, GC-tracked, built by two dedicated bytecodes.

Template holds the literal pieces and the interpolations side by side. Each Interpolation holds the value, the source expression text, an optional conversion flag, and an optional format spec. The implementation lives in Objects/templateobject.c and Objects/interpolationobject.c. Let’s trace it.

The `Interpolation` struct

 c
 
 
// Objects/interpolationobject.c
typedef struct {
 PyObject_HEAD
 PyObject *value;
 PyObject *expression;
 PyObject *conversion;
 PyObject *format_spec;
} interpolationobject;

The four fields are all owned references:

value – the evaluated result of the interpolated expression.
expression – the source text of the expression (a str, default "").
conversion – None or a one-character str ('s', 'r', 'a').
format_spec – the format specifier string (default "").

The type itself is a plain GC type:

 c
 
 
// Objects/interpolationobject.c
PyTypeObject _PyInterpolation_Type = {
 PyVarObject_HEAD_INIT(NULL, 0)
 .tp_name = "string.templatelib.Interpolation",
 .tp_basicsize = sizeof(interpolationobject),
 .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
 .tp_new = interpolation_new,
 .tp_dealloc = interpolation_dealloc,
 .tp_clear = interpolation_clear,
 .tp_free = PyObject_GC_Del,
 .tp_repr = interpolation_repr,
 .tp_members = interpolation_members,
 .tp_methods = interpolation_methods,
 .tp_traverse = interpolation_traverse,
};

tp_members exposes all four fields as read-only attributes, which is why interp.value, interp.expression, interp.conversion, and interp.format_spec work directly from Python:

 c
 
 
// Objects/interpolationobject.c
static PyMemberDef interpolation_members[] = {
 {"value", Py_T_OBJECT_EX, offsetof(interpolationobject, value), Py_READONLY, "Value"},
 {"expression", Py_T_OBJECT_EX, offsetof(interpolationobject, expression), Py_READONLY, "Expression"},
 {"conversion", Py_T_OBJECT_EX, offsetof(interpolationobject, conversion), Py_READONLY, "Conversion"},
 {"format_spec", Py_T_OBJECT_EX, offsetof(interpolationobject, format_spec), Py_READONLY, "Format specifier"},
 {NULL}
};

Constructing an `Interpolation`

The public constructor is generated by Argument Clinic from:

 c
 
 
// Objects/interpolationobject.c (clinic input)
@classmethod
Interpolation.__new__ as interpolation_new

 value: object
 expression: object(subclass_of='&PyUnicode_Type', c_default='&_Py_STR(empty)') = ""
 conversion: object(converter='_conversion_converter') = None
 format_spec: object(subclass_of='&PyUnicode_Type', c_default='&_Py_STR(empty)') = ""

The converter _conversion_converter validates that conversion is either None or a single-character string among 's', 'r', 'a':

 c
 
 
// Objects/interpolationobject.c
static int
_conversion_converter(PyObject *arg, PyObject **conversion)
{
 if (arg == Py_None) {
 return 1;
 }
 if (!PyUnicode_Check(arg)) {
 ...
 return 0;
 }
 Py_ssize_t len;
 const char *conv_str = PyUnicode_AsUTF8AndSize(arg, &len);
 if (len != 1 || !(conv_str[0] == 'a' || conv_str[0] == 'r' || conv_str[0] == 's')) {
 PyErr_SetString(PyExc_ValueError,
 "Interpolation() argument 'conversion' must be one of 's', 'a' or 'r'");
 return 0;
 }
 *conversion = arg;
 return 1;
}

The implementation allocates a GC object and takes new references to all four arguments:

 c
 
 
// Objects/interpolationobject.c
static PyObject *
interpolation_new_impl(PyTypeObject *type, PyObject *value,
 PyObject *expression, PyObject *conversion,
 PyObject *format_spec)
{
 interpolationobject *self = PyObject_GC_New(interpolationobject, type);
 if (!self) {
 return NULL;
 }

 self->value = Py_NewRef(value);
 self->expression = Py_NewRef(expression);
 self->conversion = Py_NewRef(conversion);
 self->format_spec = Py_NewRef(format_spec);
 PyObject_GC_Track(self);
 return (PyObject *) self;
}

There’s also an internal fast builder, _PyInterpolation_Build, used by the bytecode. It takes a C int conversion (one of FVC_NONE, FVC_STR, FVC_REPR, FVC_ASCII) rather than a Python object and maps it to the stored conversion value:

 c
 
 
// Objects/interpolationobject.c
PyObject *
_PyInterpolation_Build(PyObject *value, PyObject *str, int conversion, PyObject *format_spec)
{
 interpolationobject *interpolation = PyObject_GC_New(interpolationobject, &_PyInterpolation_Type);
 ...
 interpolation->value = Py_NewRef(value);
 interpolation->expression = Py_NewRef(str);
 interpolation->format_spec = Py_NewRef(format_spec);
 interpolation->conversion = NULL;

 if (conversion == 0) { // FVC_NONE
 interpolation->conversion = Py_None;
 }
 else {
 switch (conversion) {
 case FVC_ASCII: // 3
 interpolation->conversion = _Py_LATIN1_CHR('a');
 break;
 case FVC_REPR: // 2
 interpolation->conversion = _Py_LATIN1_CHR('r');
 break;
 case FVC_STR: // 1
 interpolation->conversion = _Py_LATIN1_CHR('s');
 break;
 ...
 }
 }
 PyObject_GC_Track(interpolation);
 return (PyObject *) interpolation;
}

Note that FVC_NONE is 0, which is why conversion == 0 is tested directly.

This builder is the hot path: it avoids going through __new__ and the converter on every interpolated value. Instead of allocating a fresh one-character string for 'a'/'r'/'s', it reaches for the immortal Latin-1 singleton.

_PyInterpolation_InitTypes sets __match_args__ to the four field names so that structural pattern matching works:

 c
 
 
// Objects/interpolationobject.c
PyStatus
_PyInterpolation_InitTypes(PyInterpreterState *interp)
{
 PyObject *tuple = Py_BuildValue("(ssss)", "value", "expression", "conversion", "format_spec");
 ...
 int status = PyDict_SetItemString(dict, "__match_args__", tuple);
 ...
}

Pickling is supported through __reduce__, which returns the class and a 4-tuple of fields:

 c
 
 
// Objects/interpolationobject.c
static PyObject*
interpolation_reduce(PyObject *op, PyObject *Py_UNUSED(dummy))
{
 interpolationobject *self = interpolationobject_CAST(op);
 return Py_BuildValue("(O(OOOO))", (PyObject *)Py_TYPE(op),
 self->value, self->expression,
 self->conversion, self->format_spec);
}

The `Template` struct

 c
 
 
// Objects/templateobject.c
typedef struct {
 PyObject_HEAD
 PyObject *strings;
 PyObject *interpolations;
} templateobject;

strings is a tuple of str whose length is len(interpolations) + 1; interpolations is a tuple of Interpolation. Both are owned references, and the object is GC-tracked:

 c
 
 
// Objects/templateobject.c
PyTypeObject _PyTemplate_Type = {
 PyVarObject_HEAD_INIT(NULL, 0)
 .tp_name = "string.templatelib.Template",
 .tp_basicsize = sizeof(templateobject),
 .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
 .tp_as_sequence = &template_as_sequence,
 .tp_new = template_new,
 .tp_dealloc = template_dealloc,
 .tp_clear = template_clear,
 .tp_free = PyObject_GC_Del,
 .tp_repr = template_repr,
 .tp_members = template_members,
 .tp_methods = template_methods,
 .tp_getset = template_getset,
 .tp_iter = template_iter,
 .tp_traverse = template_traverse,
};

The two tuple fields are exposed read-only:

 c
 
 
// Objects/templateobject.c
static PyMemberDef template_members[] = {
 {"strings", Py_T_OBJECT_EX, offsetof(templateobject, strings), Py_READONLY, "Strings"},
 {"interpolations", Py_T_OBJECT_EX, offsetof(templateobject, interpolations), Py_READONLY, "Interpolations"},
 {NULL},
};

A computed values property returns a tuple of the interpolation values, mirroring Template.values in the Python-level API:

 c
 
 
// Objects/templateobject.c
static PyObject *
template_values_get(PyObject *op, void *Py_UNUSED(data))
{
 templateobject *self = templateobject_CAST(op);
 Py_ssize_t len = PyTuple_GET_SIZE(self->interpolations);
 PyObject *values = PyTuple_New(len);
 if (values == NULL) {
 return NULL;
 }
 for (Py_ssize_t i = 0; i < len; i++) {
 PyObject *item = PyTuple_GET_ITEM(self->interpolations, i);
 PyTuple_SET_ITEM(values, i, _PyInterpolation_GetValueRef(item));
 }
 return values;
}

_PyInterpolation_GetValueRef simply returns a new reference to the interpolation’s value field.

`Template.new`: normalizing the argument list

The constructor accepts a flat sequence of str and Interpolation objects and normalizes them into the two parallel tuples. Consecutive strings are concatenated; an Interpolation with no preceding string gets an empty string inserted before it; a trailing interpolation gets an empty string appended:

 c
 
 
// Objects/templateobject.c
static PyObject *
template_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
 if (kwds != NULL) {
 PyErr_SetString(PyExc_TypeError, "Template.__new__ only accepts *args arguments");
 return NULL;
 }

 Py_ssize_t argslen = PyTuple_GET_SIZE(args);
 Py_ssize_t stringslen = 0;
 Py_ssize_t interpolationslen = 0;
 int last_was_str = 0;

 for (Py_ssize_t i = 0; i < argslen; i++) {
 PyObject *item = PyTuple_GET_ITEM(args, i);
 if (PyUnicode_Check(item)) {
 if (!last_was_str) {
 stringslen++;
 }
 last_was_str = 1;
 }
 else if (_PyInterpolation_CheckExact(item)) {
 if (!last_was_str) {
 stringslen++;
 }
 interpolationslen++;
 last_was_str = 0;
 }
 else {
 PyErr_Format(PyExc_TypeError,
 "Template.__new__ *args need to be of type 'str' or 'Interpolation', got %T",
 item);
 return NULL;
 }
 }
 if (!last_was_str) {
 stringslen++;
 }
 ...
}

The second pass performs the concatenation and empty-string insertion, then delegates to _PyTemplate_Build:

 c
 
 
// Objects/templateobject.c (second pass, abbreviated)
for (Py_ssize_t i = 0; i < argslen; i++) {
 PyObject *item = PyTuple_GET_ITEM(args, i);
 if (PyUnicode_Check(item)) {
 if (last_was_str) {
 PyObject *laststring = PyTuple_GET_ITEM(strings, stringsidx - 1);
 PyObject *concat = PyUnicode_Concat(laststring, item);
 ...
 PyTuple_SET_ITEM(strings, stringsidx - 1, concat);
 Py_DECREF(laststring);
 }
 else {
 PyTuple_SET_ITEM(strings, stringsidx++, Py_NewRef(item));
 }
 last_was_str = 1;
 }
 else if (_PyInterpolation_CheckExact(item)) {
 if (!last_was_str) {
 _Py_DECLARE_STR(empty, "");
 PyTuple_SET_ITEM(strings, stringsidx++, &_Py_STR(empty));
 }
 PyTuple_SET_ITEM(interpolations, interpolationsidx++, Py_NewRef(item));
 last_was_str = 0;
 }
}
if (!last_was_str) {
 _Py_DECLARE_STR(empty, "");
 PyTuple_SET_ITEM(strings, stringsidx++, &_Py_STR(empty));
}

PyObject *template = _PyTemplate_Build(strings, interpolations);
Py_DECREF(strings);
Py_DECREF(interpolations);
return template;

This normalization is what lets Template("a", interp, "b") and Template("a", interp, "b", interp2) both satisfy the invariant that len(strings) == len(interpolations) + 1.

`_PyTemplate_Build`: the factory

 c
 
 
// Objects/templateobject.c
PyObject *
_PyTemplate_Build(PyObject *strings, PyObject *interpolations)
{
 templateobject *template = PyObject_GC_New(templateobject, &_PyTemplate_Type);
 if (template == NULL) {
 return NULL;
 }

 template->strings = Py_NewRef(strings);
 template->interpolations = Py_NewRef(interpolations);
 PyObject_GC_Track(template);
 return (PyObject *) template;
}

Like _PyInterpolation_Build, this is the internal fast path used by the bytecode and by template_new itself.

Iteration: `TemplateIter`

Template.__iter__ returns a GC-tracked templateiterobject that alternates between the strings and interpolations iterators, skipping empty strings that result from adjacent interpolations:

 c
 
 
// Objects/templateobject.c
typedef struct {
 PyObject_HEAD
 PyObject *stringsiter;
 PyObject *interpolationsiter;
 int from_strings;
} templateiterobject;

 c
 
 
// Objects/templateobject.c
static PyObject *
templateiter_next(PyObject *op)
{
 templateiterobject *self = templateiterobject_CAST(op);
 PyObject *item;
 if (self->from_strings) {
 item = PyIter_Next(self->stringsiter);
 self->from_strings = 0;
 if (item == NULL) {
 return NULL;
 }
 if (PyUnicode_GET_LENGTH(item) == 0) {
 Py_SETREF(item, PyIter_Next(self->interpolationsiter));
 self->from_strings = 1;
 }
 }
 else {
 item = PyIter_Next(self->interpolationsiter);
 self->from_strings = 1;
 }
 return item;
}

The from_strings flag starts at 1, so iteration begins with a string. When that string is empty (which happens between two interpolations), it’s replaced by the next interpolation and the flag is reset so the following string is consumed next. That yields the familiar str/Interpolation alternation you get from for part in template:.

template_iter constructs the iterator and obtains iterators over both tuples:

 c
 
 
// Objects/templateobject.c
static PyObject *
template_iter(PyObject *op)
{
 templateobject *self = templateobject_CAST(op);
 templateiterobject *iter = PyObject_GC_New(templateiterobject, &_PyTemplateIter_Type);
 ...
 PyObject *stringsiter = PyObject_GetIter(self->strings);
 ...
 PyObject *interpolationsiter = PyObject_GetIter(self->interpolations);
 ...
 iter->stringsiter = stringsiter;
 iter->interpolationsiter = interpolationsiter;
 iter->from_strings = 1;
 PyObject_GC_Track(iter);
 return (PyObject *)iter;
}

Concatenation

Template supports + via tp_as_sequence.sq_concat. When both operands are exact Template instances, their strings and interpolations are concatenated, merging the boundary strings:

 c
 
 
// Objects/templateobject.c
static PyObject *
template_strings_concat(PyObject *left, PyObject *right)
{
 Py_ssize_t left_stringslen = PyTuple_GET_SIZE(left);
 PyObject *left_laststring = PyTuple_GET_ITEM(left, left_stringslen - 1);
 Py_ssize_t right_stringslen = PyTuple_GET_SIZE(right);
 PyObject *right_firststring = PyTuple_GET_ITEM(right, 0);

 PyObject *concat = PyUnicode_Concat(left_laststring, right_firststring);
 ...
 PyObject *newstrings = PyTuple_New(left_stringslen + right_stringslen - 1);
 ...
 // copy left[0..n-2], then concat, then right[1..]
}

 c
 
 
// Objects/templateobject.c
static PyObject *
template_concat_templates(templateobject *self, templateobject *other)
{
 PyObject *newstrings = template_strings_concat(self->strings, other->strings);
 ...
 PyObject *newinterpolations = PySequence_Concat(self->interpolations, other->interpolations);
 ...
 PyObject *newtemplate = _PyTemplate_Build(newstrings, newinterpolations);
 ...
}

_PyTemplate_Concat is the public entry point and rejects non-Template operands with a TypeError.

Pickling

Template.__reduce__ delegates to string.templatelib._template_unpickle, passing the two tuples:

 c
 
 
// Objects/templateobject.c
static PyObject*
template_reduce(PyObject *op, PyObject *Py_UNUSED(dummy))
{
 PyObject *mod = PyImport_ImportModule("string.templatelib");
 ...
 PyObject *func = PyObject_GetAttrString(mod, "_template_unpickle");
 ...
 templateobject *self = templateobject_CAST(op);
 PyObject *result = Py_BuildValue("O(OO)", func, self->strings, self->interpolations);
 ...
}

The Python helper _template_unpickle in Lib/string/templatelib.py zips the strings and interpolations back into the flat str/Interpolation argument list and calls Template(*parts), re-running the normalization in template_new.

From source to object: the compiler and bytecode

The parser produces a TemplateStr expression node whose values sequence contains Constant (string) and Interpolation nodes. The compiler’s codegen_template (in Python/codegen.c) emits the strings as a tuple, the interpolations as a tuple, and then BUILD_TEMPLATE:

 c
 
 
// Python/codegen.c
Py_ssize_t stringslen = 0;
for (Py_ssize_t i = 0; i < value_count; i++) {
 value = asdl_seq_GET(e->v.TemplateStr.values, i);
 if (value->kind == Interpolation_kind) {
 if (last_was_interpolation) {
 ADDOP_LOAD_CONST(c, loc, Py_NewRef(&_Py_STR(empty)));
 stringslen++;
 }
 last_was_interpolation = 1;
 }
 else {
 VISIT(c, expr, value);
 stringslen++;
 last_was_interpolation = 0;
 }
}
if (last_was_interpolation) {
 ADDOP_LOAD_CONST(c, loc, Py_NewRef(&_Py_STR(empty)));
 stringslen++;
}
ADDOP_I(c, loc, BUILD_TUPLE, stringslen);

Py_ssize_t interpolationslen = 0;
for (Py_ssize_t i = 0; i < value_count; i++) {
 value = asdl_seq_GET(e->v.TemplateStr.values, i);
 if (value->kind == Interpolation_kind) {
 VISIT(c, expr, value);
 interpolationslen++;
 }
}
ADDOP_I(c, loc, BUILD_TUPLE, interpolationslen);
ADDOP(c, loc, BUILD_TEMPLATE);

Each Interpolation node is compiled by codegen_interpolation, which visits the value, loads the expression string constant, optionally visits the format spec, encodes the conversion into the oparg, and emits BUILD_INTERPOLATION:

 c
 
 
// Python/codegen.c
VISIT(c, expr, e->v.Interpolation.value);
ADDOP_LOAD_CONST(c, loc, e->v.Interpolation.str);

int oparg = 2;
if (e->v.Interpolation.format_spec) {
 oparg++;
 VISIT(c, expr, e->v.Interpolation.format_spec);
}

int conversion = e->v.Interpolation.conversion;
if (conversion != -1) {
 switch (conversion) {
 case 's': oparg |= FVC_STR << 2; break;
 case 'r': oparg |= FVC_REPR << 2; break;
 case 'a': oparg |= FVC_ASCII << 2; break;
 ...
 }
}

ADDOP_I(c, loc, BUILD_INTERPOLATION, oparg);

The bytecode implementation consumes the stack and calls the C builders:

 c
 
 
// Python/bytecodes.c
inst(BUILD_INTERPOLATION, (value, str, format[oparg & 1] -- interpolation)) {
 PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
 PyObject *str_o = PyStackRef_AsPyObjectBorrow(str);
 int conversion = oparg >> 2;
 PyObject *format_o;
 if (oparg & 1) {
 format_o = PyStackRef_AsPyObjectBorrow(format[0]);
 }
 else {
 format_o = &_Py_STR(empty);
 }
 PyObject *interpolation_o = _PyInterpolation_Build(value_o, str_o, conversion, format_o);
 ...
 interpolation = PyStackRef_FromPyObjectSteal(interpolation_o);
}

inst(BUILD_TEMPLATE, (strings, interpolations -- template)) {
 PyObject *strings_o = PyStackRef_AsPyObjectBorrow(strings);
 PyObject *interpolations_o = PyStackRef_AsPyObjectBorrow(interpolations);
 PyObject *template_o = _PyTemplate_Build(strings_o, interpolations_o);
 PyStackRef_CLOSE(interpolations);
 PyStackRef_CLOSE(strings);
 ERROR_IF(template_o == NULL);
 template = PyStackRef_FromPyObjectSteal(template_o);
}

The oparg & 1 bit of BUILD_INTERPOLATION indicates whether a format spec was supplied; the conversion is carried in the upper bits (oparg >> 2).

Memory layout and GC

Both types are GC objects with tp_traverse/tp_clear visiting every owned reference:

 c
 
 
// Objects/interpolationobject.c
static int
interpolation_traverse(PyObject *op, visitproc visit, void *arg)
{
 interpolationobject *self = interpolationobject_CAST(op);
 Py_VISIT(self->value);
 Py_VISIT(self->expression);
 Py_VISIT(self->conversion);
 Py_VISIT(self->format_spec);
 return 0;
}

 c
 
 
// Objects/templateobject.c
static int
template_traverse(PyObject *op, visitproc visit, void *arg)
{
 templateobject *self = templateobject_CAST(op);
 Py_VISIT(self->strings);
 Py_VISIT(self->interpolations);
 return 0;
}

The iterator is also GC-tracked because it holds two iterators. All three use PyObject_GC_Del as tp_free and untrack themselves in tp_dealloc before clearing.

The on-disk sizes are small: an Interpolation is a PyObject header plus four pointers, and a Template is a header plus two pointers. Both are allocated through PyObject_GC_New, so they participate in cyclic garbage collection like tuples and lists.

Key takeaways

Template and Interpolation are GC-tracked C types defined in Objects/templateobject.c and Objects/interpolationobject.c, exported as string.templatelib.Template and string.templatelib.Interpolation.
An Interpolation stores value, expression, conversion (None or a one-char str), and format_spec; conversion is validated by _conversion_converter in the public constructor and synthesized from FVC_* constants by _PyInterpolation_Build on the bytecode path.
A Template stores strings and interpolations tuples satisfying len(strings) == len(interpolations) + 1; template_new normalizes a flat argument list into that invariant.
Iteration alternates strings and interpolations, skipping empty boundary strings; concatenation merges the boundary strings.
The compiler emits BUILD_INTERPOLATION and BUILD_TEMPLATE, which call _PyInterpolation_Build and _PyTemplate_Build.
Both objects are GC-tracked with full traverse/clear support and support pickling via __reduce__.

CPython Interns Strings

Jay Hubbert — Thu, 22 Jan 2026 09:00:00 +0000

TL;DR: String interning lets CPython compare equal strings by pointer instead of by content. Identifiers, single-byte characters, and constants all live in one of two tables, and the trick is in how the refcounts are fudged to keep them alive.

String interning is the trick that makes a = "foo"; b = "foo"; a is b sometimes true: two equal strings are represented by one object, so equality becomes pointer comparison and hashing is free. CPython does not intern everything – that would be expensive and pointless – but it interns aggressively where it pays off: identifiers, single-byte characters, attribute names, and constants. This article traces the two mechanisms behind it.

Two mechanisms

CPython has two kinds of interned strings:

Singletons: runtime-global, lock-free, shared across interpreters.
- 256 one-byte latin-1 strings.
- _Py_ID(...) for identifier fragments.
- _Py_STR(...) for other static strings.
Dynamic interning: per-interpreter dict, for strings created at runtime.

The state of a string is recorded in two bits:

 c
 
 
// Include/cpython/unicodeobject.h
#define SSTATE_NOT_INTERNED 0
#define SSTATE_INTERNED_MORTAL 1
#define SSTATE_INTERNED_IMMORTAL 2
#define SSTATE_INTERNED_IMMORTAL_STATIC 3

stored in _PyASCIIObject_CAST(op)->state.interned.

Two bits, four states. NOT_INTERNED is the default; the other three say which table you’re in and whether the table owns your life.

Singletons: the global table

Singletons live in a global hashtable shared by all interpreters and threads without synchronization:

 c
 
 
// Objects/unicodeobject.c
#define INTERNED_STRINGS _PyRuntime.cached_objects.interned_strings

static PyStatus init_global_interned_strings(PyInterpreterState *interp) {
 INTERNED_STRINGS = _Py_hashtable_new_full(...);
 _PyUnicode_InitStaticStrings(interp); // _Py_ID / _Py_STR
 for (int i = 0; i < 256; i++) {
 PyObject *s = LATIN1(i);
 _PyUnicode_InternStatic(interp, &s);
 assert(s == LATIN1(i));
 }
 ...
}

The macros that reach them are defined in Include/internal/pycore_global_strings.h:

 c
 
 
#define _Py_ID(NAME) \
 (_Py_SINGLETON(strings.identifiers._py_ ## NAME._ascii.ob_base))
#define _Py_STR(NAME) \
 (_Py_SINGLETON(strings.literals._py_ ## NAME._ascii.ob_base))
#define _Py_LATIN1_CHR(CH) \
 ((CH) < 128 \
 ? (PyObject*)&_Py_SINGLETON(strings).ascii[(CH)] \
 : (PyObject*)&_Py_SINGLETON(strings).latin1[(CH) - 128])

Static interning inserts into the global table and marks the string SSTATE_INTERNED_IMMORTAL_STATIC:

 c
 
 
// Objects/unicodeobject.c
static PyObject* intern_static(PyInterpreterState *interp, PyObject *s /* stolen */) {
 assert(_PyUnicode_STATE(s).statically_allocated);
 assert(!PyUnicode_CHECK_INTERNED(s));
 PyObject *r = (PyObject *)_Py_hashtable_get(INTERNED_STRINGS, s);
 assert(r == NULL);
 if (r != NULL && r != s) { ... return Py_NewRef(r); }
 if (_Py_hashtable_set(INTERNED_STRINGS, s, s) < 0) { ... }
 _PyUnicode_STATE(s).interned = SSTATE_INTERNED_IMMORTAL_STATIC;
 return s;
}

Dynamic interning: the per-interpreter dict

For strings created at runtime, each interpreter has its own dict:

 c
 
 
// Objects/unicodeobject.c
static inline PyObject *get_interned_dict(PyInterpreterState *interp) {
 return _Py_INTERP_CACHED_OBJECT(interp, interned_strings);
}

static int init_interned_dict(PyInterpreterState *interp) {
 interned = PyDict_New(); // or shared main-interpreter dict
 _Py_INTERP_CACHED_OBJECT(interp, interned_strings) = interned;
 return 0;
}

The core routine is intern_common(interp, s, immortalize):

 c
 
 
// Objects/unicodeobject.c (intern_common)
// steals a reference to s, returns a (possibly different) interned object.

Its steps, in order:

Reject subclasses.
If already interned, possibly immortalize it and return.
Single-byte latin-1 strings are redirected to the _Py_LATIN1_CHR singleton.
Check the global static cache for a static copy.
Otherwise PyDict_SetDefaultRef on the per-interpreter dict. On first insert:

 c
 
 
// Objects/unicodeobject.c
if (!_Py_IsImmortal(s)) {
 /* The two references in interned dict (key and value) are not counted. */
 Py_DECREF(s);
 Py_DECREF(s);
}
FT_ATOMIC_STORE_UINT8(_PyUnicode_STATE(s).interned, SSTATE_INTERNED_MORTAL);
if (immortalize) { immortalize_interned(s); }

On a hit, the existing interned string is returned and s is discarded.

The two references in the dict (key and value) are deliberately not counted in ob_refcnt. That is the subtle part: an interned string’s reference count does not include the dict that keeps it alive, so the count reflects only external users. When the last external reference goes away, the count can reach zero even though the dict still holds the string; unicode_dealloc and _PyUnicode_ClearInterned restore the missing counts before freeing.

This is the part people get wrong. Those two Py_DECREF calls look like a bug. They aren’t: the dict holds two uncounted references to s, so the count has to be adjusted to match reality. Without this, every interned string would be freed the moment its last user dropped it.

Mortal vs immortal

 c
 
 
// Objects/unicodeobject.c
static void immortalize_interned(PyObject *s) {
 assert(PyUnicode_CHECK_INTERNED(s) == SSTATE_INTERNED_MORTAL);
 assert(!_Py_IsImmortal(s));
 ...
 _Py_SetImmortal(s);
 FT_ATOMIC_STORE_UINT8(_PyUnicode_STATE(s).interned, SSTATE_INTERNED_IMMORTAL);
}

The public entry points differ in whether they immortalize:

 c
 
 
// Objects/unicodeobject.c
void _PyUnicode_InternImmortal(PyInterpreterState *interp, PyObject **p);
void _PyUnicode_InternMortal(PyInterpreterState *interp, PyObject **p);
void _PyUnicode_InternStatic(PyInterpreterState *interp, PyObject **p);
void _PyUnicode_InternInPlace(PyInterpreterState *interp, PyObject **p); // -> Immortal
void PyUnicode_InternInPlace(PyObject **p); // public -> Mortal

In the free-threaded build all interned strings become immortal, and a copy may be made if the original cannot be safely immortalized.

Shutdown bookkeeping

At interpreter shutdown, mortal interned strings are released by restoring the two dict references and removing the entry. Immortal interned strings are made mortal again with _Py_SetMortal(s, 2) so they can be collected; static strings stay in the global table until the main interpreter tears it down.

Where CPython uses it

Interning is not a user-facing feature; it is a compiler/runtime optimization. The places that call it are the ones where string identity matters:

Code objects (Objects/codeobject.c): identifier-like ASCII strings are interned via should_intern_string, and all names and constants are interned/immortalized in intern_strings / intern_constants.
Marshal (Python/marshal.c): TYPE_INTERNED strings are interned and immortalized on load.
The compiler and optimizer, plus many runtime sites (object, type, module, dict, pickle, …), call _PyUnicode_InternMortal / _PyUnicode_InternImmortal.

The payoff is that attribute lookup, global lookup, and constant loading can compare strings by pointer instead of by content, and interned constants never need to be hashed more than once.

The real win: obj.attr in a hot loop stops being a string comparison and becomes a pointer comparison. That’s the difference between a memcmp and a single load.

Inspecting it

 c
 
 
// Objects/unicodeobject.c
Py_ssize_t _PyUnicode_InternedSize(void) {
 PyObject *dict = get_interned_dict(_PyInterpreterState_GET());
 return _Py_hashtable_len(INTERNED_STRINGS) + PyDict_GET_SIZE(dict);
}

And PyUnicode_CHECK_INTERNED(obj) exposes the four states to C code. The distinction between mortal and immortal matters for reference counting and for whether a string can be shared across threads in the free-threaded build, which is why the API splits into InternMortal and InternImmortal rather than one call.

Wrapping up

Two tables, two lifetimes. The global table holds immortal static strings — the 256 latin-1 singletons, every _Py_ID identifier, every _Py_STR literal — and never needs a lock. The per-interpreter dict holds runtime interned strings, which are mortal by default and immortalized on request.

The one thing to remember: interning is an identity optimization, not a memory optimization in the usual sense. Two equal strings become one object, so you get pointer-equality and a cached hash. And the refcount fudge — two uncounted dict references — is the subtle part that keeps the whole scheme consistent.

Strings on Opensource internals

ClickHouse Packs Short Strings Into Hash Table Keys: No Allocation Per Key

Five tables, one per size class

The packing: how a string becomes a number

Unpacking: getting the string back

The empty-cell rule, specialised per size class

Hashing: CRC32C where it can, CityHash where it can’t

The lookup result: why you never see the key

Reserving and growing: where the heuristic leaks

Where it’s used: three string-key aggregation variants

Trade-offs: what you win and what you pay

Further reading

Wrapping up

CPython's Template String Implementation in C (t-strings): Interpolation as Data

The `Interpolation` struct

Constructing an `Interpolation`

The `Template` struct

`Template.new`: normalizing the argument list

`_PyTemplate_Build`: the factory

Iteration: `TemplateIter`

Concatenation

Pickling

From source to object: the compiler and bytecode

Memory layout and GC

Key takeaways

Further reading

CPython Interns Strings

Two mechanisms

Singletons: the global table

Dynamic interning: the per-interpreter dict

Mortal vs immortal

Shutdown bookkeeping

Where CPython uses it

Inspecting it

Wrapping up

Strings on Opensource internals

ClickHouse Packs Short Strings Into Hash Table Keys: No Allocation Per Key

Five tables, one per size class

The packing: how a string becomes a number

Unpacking: getting the string back

The empty-cell rule, specialised per size class

Hashing: CRC32C where it can, CityHash where it can’t

The lookup result: why you never see the key

Reserving and growing: where the heuristic leaks

Where it’s used: three string-key aggregation variants

Trade-offs: what you win and what you pay

Further reading

Wrapping up

CPython's Template String Implementation in C (t-strings): Interpolation as Data

The Interpolation struct

Constructing an Interpolation

The Template struct

Template.__new__: normalizing the argument list

_PyTemplate_Build: the factory

Iteration: TemplateIter

Concatenation

Pickling

From source to object: the compiler and bytecode

Memory layout and GC

Key takeaways

Further reading

CPython Interns Strings

Two mechanisms

Singletons: the global table

Dynamic interning: the per-interpreter dict

Mortal vs immortal

Shutdown bookkeeping

Where CPython uses it

Inspecting it

Wrapping up

The `Interpolation` struct

Constructing an `Interpolation`

The `Template` struct

`Template.new`: normalizing the argument list

`_PyTemplate_Build`: the factory

Iteration: `TemplateIter`