Polars' Categorical Encoding Hides a Global Mutable Dictionary in Plain Sight

TL;DR: Every pl.Categorical in a process is a view into one of a small number of shared dictionaries that map strings to u32 codes. The dictionary itself is a lock-free concurrent hash table plus a concurrent append-only vector, so nothing takes a global lock on the hot path. Codes are handed out in insertion order, which is why group-by can hash integers instead of strings. The cost is that the dictionary is process-global mutable state: unrelated datasets share it, it never forgets a string, and its contents are visible to every thread that touches a categorical.

The old cache is gone; the dictionary is not

If you learned Polars from older releases, you remember pl.StringCache, a process-global RwLock<HashMap<...>> that owned the string-to-code mapping for every categorical series, and you remember the sync_categoricals dance for making two frames agree. Grep this tree for StringCache or RevMapping and you will find almost nothing: the classic RevMapping type is gone, and the Python-side cache is a deprecated stub.

@deprecated("the string cache has been replaced by pl.Categories")
class StringCache(contextlib.ContextDecorator):
    """
    Does nothing.

    .. deprecated:: 1.41.0
        The string cache was used to maintain the mapping for the Categorical
        dtype, this is now done through ``pl.Categories``.
    """

py-polars/src/polars/string_cache.py:27

So where did the mapping go? Into Categories, a named object that owns a CategoricalMapping, plus a process-global registry that makes two same-named Categories share one dictionary. The mutation never went away; it just moved a layer down, into a structure that is much easier to read.

The whole thing lives in two files:

• crates/polars-dtype/src/categorical/mod.rs — Categories, FrozenCategories, the registry.
• crates/polars-dtype/src/categorical/mapping.rs — the actual string-to-code dictionary.

A global dictionary, right there in the source

The first thing you notice is a pair of process-wide statics. The registry maps a categorical’s identity to the object that owns its mapping:

// Used to maintain a 1:1 mapping between Categories' ID and the Categories objects themselves.
// This is important for serialization.
static CATEGORIES_REGISTRY: LazyLock<Mutex<PlHashMap<CategoricalId, Weak<Categories>>>> =
    LazyLock::new(|| Mutex::new(PlHashMap::new()));

crates/polars-dtype/src/categorical/mod.rs:95

And the “global” categorical — the one you get from pl.Categorical with no arguments — is a single Arc parked in a LazyLock, created on first use and registered under an empty name/namespace:

static GLOBAL_CATEGORIES: LazyLock<Arc<Categories>> = LazyLock::new(|| {
    let mut registry = CATEGORIES_REGISTRY.lock().unwrap();
    let global_id = CategoricalId::global();
    if let Some(cats_ref) = registry.get(&global_id) {
        if let Some(cats) = cats_ref.upgrade() {
            return cats;
        }
    }
    let global = Arc::new(Categories {
        id: CategoricalId::global(),
        mapping: Mutex::new(Weak::new()),
    });
    registry.insert(global_id, Arc::downgrade(&global));
    global
});

crates/polars-dtype/src/categorical/mod.rs:107

Categories itself is a name plus a weak handle to the dictionary:

/// A (named) object which is used to indicate which categorical data types have the same mapping.
pub struct Categories {
    id: CategoricalId,
    mapping: Mutex<Weak<CategoricalMapping>>,
}

crates/polars-dtype/src/categorical/mod.rs:126

The Weak is the crux. Categories is the identity; the dictionary is content that is created on demand and thrown away when the last categorical dtype referencing it goes away:

    /// The mapping for this Categories object. If no mapping currently exists
    /// it creates a new empty mapping.
    pub fn mapping(&self) -> Arc<CategoricalMapping> {
        let mut guard = self.mapping.lock().unwrap();
        if let Some(arc) = guard.upgrade() {
            return arc;
        }
        let arc = Arc::new(CategoricalMapping::new(self.id.physical.max_categories()));
        *guard = Arc::downgrade(&arc);
        arc
    }

crates/polars-dtype/src/categorical/mod.rs:199

So the ownership graph is:

   pl.Categorical()  ->  Categories::global()  --+
                                                 |
   pl.Categories("fruit") -> Categories("fruit") --+--> Arc<CategoricalMapping>
                                                 |      (str -> u32, u32 -> str)
   pl.Enum([...]) -> FrozenCategories -----------+--> Arc<CategoricalMapping> (immutable)

Note the direction of the arrows. Categories objects are shared through the global registry; the mappings are shared through the Arc that every DataType::Categorical carries:

    #[cfg(feature = "dtype-categorical")]
    Categorical(Arc<Categories>, Arc<CategoricalMapping>),

crates/polars-core/src/datatypes/dtype.rs:135

Two rules fall out of this. First, pl.Categories("fruit") called twice in the same process returns the same Categories object, because the registry deduplicates on (name, namespace, physical):

    pub fn new(
        name: PlSmallStr,
        namespace: PlSmallStr,
        physical: CategoricalPhysical,
    ) -> Arc<Self> {
        let id = CategoricalId {
            name,
            namespace,
            physical,
        };
        let mut registry = CATEGORIES_REGISTRY.lock().unwrap();
        if let Some(cats_ref) = registry.get(&id) {
            if let Some(cats) = cats_ref.upgrade() {
                return cats;
            }
        }

crates/polars-dtype/src/categorical/mod.rs:108

Second, the global Categories means every plain pl.Categorical in the process shares one dictionary. pl.Series(["a"], dtype=pl.Categorical) and pl.Series(["b"], dtype=pl.Categorical) in two unrelated modules of your program are looking at the same Arc<CategoricalMapping>. That is the “global mutable dictionary in plain sight”. It is one static away from being a process-wide hash map, and functionally it is one.

Interning a string into a code

The dictionary is deliberately small. Two structures, no locks:

pub struct CategoricalMapping {
    str_to_cat: RawTable<str, CatSize>,
    cat_to_str_and_hash: boxcar::Vec<(&'static str, u64)>,
    max_categories: usize,
    upper_bound: AtomicUsize,
    hasher: PlSeedableRandomStateQuality,
}

crates/polars-dtype/src/categorical/mapping.rs:16

• RawTable is Polars’ own lock-free concurrent hash table (polars-utils/src/parma), the same one used for row encoding.
• boxcar::Vec is a concurrent append-only vector: many threads can push, readers can index.
• upper_bound is an atomic counter of “how many codes have been handed out”.

CatSize is always a u32 in the dictionary, even when the physical array is u8 or u16:

pub type CatSize = u32;

crates/polars-dtype/src/categorical/catsize.rs:1

The insert path is where the concurrency story lives:

    /// Same as to_cat, but with the hash pre-computed.
    #[inline(always)]
    pub fn insert_cat_with_hash(&self, s: &str, hash: u64) -> PolarsResult<CatSize> {
        self.str_to_cat
            .try_get_or_insert_with(
                hash,
                s,
                |k| k == s,
                |k| {
                    let old_upper_bound = self.upper_bound.fetch_add(1, Ordering::Relaxed);
                    if old_upper_bound + 1 > self.max_categories {
                        self.upper_bound.fetch_sub(1, Ordering::Relaxed);
                        polars_bail!(ComputeError: "attempted to insert more categories than the maximum allowed");
                    }
                    let hash = PlFixedStateQuality::default().hash_one(k);
                    let idx = self
                        .cat_to_str_and_hash
                        .push((unsafe { core::mem::transmute::<&str, &'static str>(k) }, hash));
                    Ok(idx as CatSize)
                },
            )
            .copied()
    }

crates/polars-dtype/src/categorical/mapping.rs:83

Three things to notice.

The code is the index in the vector. idx is where the string landed in cat_to_str_and_hash, so codes are dense and handed out in insertion order. Two different dictionaries that saw the same strings in different orders will assign different codes to them. That is the entire reason pl.Categorical series from unrelated sources cannot be compared directly.

val_f runs exactly once per distinct string, even under concurrent inserts. This is the contract of the underlying table:

    /// Finds the value corresponding to a key with the given hash and equality function, or insert
    /// a new one if the key does not exist.
    ///
    /// `val_f` is guaranteed to only be called when inserting a new key not currently found in the
    /// table, even if multiple concurrent inserts occur. The key reference passed to `val_f` lives
    /// as long as the new entry will.

crates/polars-utils/src/parma/raw/mod.rs:659

So the “who gets which code” race is resolved inside the hash table, not by a lock around the whole dictionary. Two threads casting the same new string concurrently produce one code, not two.

The counter and the vector can disagree. upper_bound is bumped before the string is pushed into the vector. Between the fetch_add and the push, cat_to_str(id) returns None for a code that has already been handed out. The Python docs are explicit about the consequence:

    .. note::
        Because the categories are backed by a concurrent data structure, physical
        category values may be reserved before they are assigned a string lexical
        value if concurrent queries are running. As a result, the resulting `Series`
        may contain `None` values.

py-polars/src/polars/datatypes/classes.py:734

And indeed the upper bound is deliberately allowed to over-count:

    /// Returns an upper bound such that all strings inserted into the CategoricalMapping
    /// have a categorical id less than it. Note that due to parallel inserts which
    /// you have not synchronized with, there may be gaps when using `from_cat`.
    #[inline(always)]
    pub fn num_cats_upper_bound(&self) -> usize {
        // We need to clamp to self.max_categories because a `fetch_add` may
        // have (temporarily) pushed it beyond the max allowed.
        self.upper_bound
            .load(Ordering::Relaxed)
            .min(self.max_categories)
    }

crates/polars-dtype/src/categorical/mapping.rs:133

That is the honest cost of a lock-free dictionary: the set of codes is consistent the moment an insert returns, but a concurrent reader can observe a half-published slot. Polars’ answer is that a categorical value with no string is a null, and the validity bitmap is computed from the mapping.

How strings actually get interned

Nothing in the cast path does string interning by hand. The Categorical dtype carries its Arc<CategoricalMapping>, and the chunked-array constructor just pushes into it:

    pub fn append_str(&mut self, val: &str) -> PolarsResult<()> {
        let cat = if self.is_enum {
            self.mapping.get_cat(val).ok_or_else(|| {
                polars_err!(ComputeError: "attempted to insert '{val}' into Enum which does not contain this string")
            })?
        } else {
            self.mapping.insert_cat(val)?
        };
        self.cats.push(T::Native::from_cat(cat));
        self.validity.push(true);
        Ok(())
    }

crates/polars-core/src/chunked_array/builder/categorical.rs:57

Enum is the frozen, immutable variant: it only ever looks up, never inserts, and a string that is not in the enum becomes a null. Categorical inserts, growing the dictionary as it goes. The bulk path is the same thing without the builder:

            DataType::Categorical(cats, mapping) => {
                assert!(cats.physical() == T::physical());
                for opt_s in strings {
                    cat_ids.push(if let Some(s) = opt_s {
                        T::Native::from_cat(mapping.insert_cat(s)?)
                    } else {
                        T::Native::zero()
                    });
                    validity.push(opt_s.is_some());
                }
            },

crates/polars-core/src/chunked_array/logical/categorical.rs:178

One hash per distinct string, then a table probe per row. The dictionary is what makes the “one hash per distinct string” claim true across the whole process rather than per series.

Why group-by becomes cheap

Here is the payoff. Grouping a categorical column is grouping integers. The series implementation simply forwards to the physical representation:

            #[cfg(feature = "algorithm_group_by")]
            fn group_tuples(&self, multithreaded: bool, sorted: bool) -> PolarsResult<GroupsType> {
                self.0.physical().group_tuples(multithreaded, sorted)
            }

crates/polars-core/src/series/implementations/categorical.rs:142

The physical array is u8/u16/u32, which lands in the numeric grouping path: hash-partition the keys across threads, then build groups per partition. No string comparisons, no per-row hashing of variable-length data. A group-by over a categorical key costs the same as a group-by over an integer key of the same cardinality.

The dictionary helps a second time, in the hash itself. Polars’ vec_hash for a categorical column does not hash the codes — it hashes the precomputed string hashes stored alongside the strings in the dictionary:

            let mapping = self.get_mapping();
            for opt_cat in self.phys.iter() {
                if let Some(cat) = opt_cat {
                    let base_h = unsafe { mapping.cat_to_hash_unchecked(cat.as_cat()) };
                    buf.push(folded_multiply(base_h, mult));
                } else {
                    buf.push(null);
                }
            }

crates/polars-core/src/chunked_array/logical/categorical.rs:430

The u64 next to each string in cat_to_str_and_hash is a hash of that string with a fixed seed. It is computed once, at interning time, and reused for every hash-based operation on every column that shares the dictionary. Group-by, join and hash-join all read a u64 out of a vector instead of re-hashing the string bytes.

The row encoder is the exception that proves the rule. When Polars needs the ordering of a categorical to be lexical rather than insertion order, it cannot use the codes, so it writes the strings:

unsafe fn encode_cat_array<T: NativeType + FixedLengthEncoding + CatNative>(
    buffer: &mut [MaybeUninit<u8>],
    keys: &PrimitiveArray<T>,
    opt: RowEncodingOptions,
    ctx: &RowEncodingCategoricalContext,
    offsets: &mut [usize],
) {
    if ctx.is_enum || !opt.is_ordered() {
        numeric::encode(buffer, keys, opt, offsets);
    } else {
        utf8::encode_str(
            buffer,
            keys.iter()
                .map(|k| k.map(|&cat| ctx.mapping.cat_to_str_unchecked(cat.as_cat()))),
            opt,
            offsets,
        );
    }
}

crates/polars-row/src/encode.rs:612

Enum (frozen, ordered by definition) encodes the fixed-width code. A Categorical with is_ordered() encodes the string, because the code order is insertion order and nobody wants that. The mapping is what makes the decode side possible again: decode_str_as_cat looks the string back up in the same dictionary.

Local vs global, and the replacement for sync_categoricals

The old sync_categoricals existed because two categorical series could hold different dictionaries, and concatenating them meant reconciling the codes. The new design pushes the reconciliation earlier, into the dtype, and refuses rather than repairs:

pub fn ensure_same_categories(left: &Arc<Categories>, right: &Arc<Categories>) -> PolarsResult<()> {
    if Arc::ptr_eq(left, right) {
        return Ok(());
    }

    if left.name() != right.name() {
        polars_bail!(SchemaMismatch: "Categories name mismatch, left: '{}', right: '{}'.

Operations mixing different Categories are often not supported, you may have to cast.", left.name(), right.name())
    }

crates/polars-dtype/src/categorical/mod.rs:342

So the taxonomy today is:

• Global — pl.Categorical, i.e. Categories::global(). One dictionary for the whole process, empty at startup, shared by everything that uses the plain categorical dtype. Python picks it up automatically: "Categorical" => DataType::from_categories(Categories::global()) (crates/polars-python/src/conversion/mod.rs:467).
• Named — pl.Categories("fruit"). A dictionary per (name, namespace, physical), shared process-wide by name. Two separately constructed pl.Categories("fruit") objects are the same object, so their codes are comparable.
• Frozen — pl.Enum([...]). An immutable, content-addressed dictionary; two enums with the same strings are the same object, so comparison is a pointer check.

The FrozenCategories registry is a nice piece of engineering in its own right — it hashes the category list and stores Weak references so that content-equal enums deduplicate process-wide (crates/polars-dtype/src/categorical/mod.rs:100).

Where the global mutable state hurts

Now the bill.

The dictionary is shared with everything, including things that have nothing to do with each other. Two independent datasets that both use the plain pl.Categorical dtype share one dictionary. Casting a column of user-supplied strings to pl.Categorical anywhere in your process inserts those strings into the dictionary that your other module’s group-by is reading from. The codes remain valid, so nothing breaks — but the dictionary now contains strings you never intended, and it keeps them.

It never forgets. The RawTable is insert-only in practice; entries are dropped only when the whole CategoricalMapping is dropped, which happens when the last DataType::Categorical referencing it dies. As long as one global-categorical series is alive somewhere in the process, every string ever cast to pl.Categorical is still resident. The Python docs say as much:

    The `Categories` instance is only a weak reference to the actual
    mapping stored in Polars. If no actual data exists using this mapping (like
    a `Series` or `DataFrame`), the mapping is cleaned up by Polars:

py-polars/src/polars/datatypes/classes.py:738

“Cleaned up” means the whole dictionary goes, codes and all — and it only goes when the last dtype referencing it dies, which is why a pl.Categories handle on the Python side is not enough to keep the mapping alive. You need a live series:

    If you wish to keep a persistent mapping, simply keep alive some object which
    uses the mapping, e.g. `keepalive = pl.Series([], dtype=pl.Categorical(fruit))`.

py-polars/src/polars/datatypes/classes.py:746

Codes are not portable. Since the code is the insertion index, the same string has different codes in different dictionaries. Any code that joins or compares two categorical series must first prove they share a Categories object, and if they do not, the answer is an error, not a merge. That is a behavioural change from the old cache, where mismatched caches could be reconciled.

fork() breaks it. CATEGORIES_REGISTRY and GLOBAL_CATEGORIES are process-global statics. After a fork(), the child inherits a copy of the registry with the parent’s Arc/Weak bookkeeping, so the child’s notion of “the same Categories” is a different object with a different address. Polars detects exactly this:

    } else {
        polars_bail!(SchemaMismatch: "Categories which should be equal have different backing objects.

This is a known problem when combining Polars with multiprocessing using fork().")
    }

crates/polars-dtype/src/categorical/mod.rs:352

The registry mutex is a real global lock, even if it is off the hot path. Categories::new and Drop both take it, so code that creates named categories in a hot loop serializes on one mutex for the whole process.

And the codes can be observed half-published. As described above, a concurrent reader can see a code whose string has not landed in the vector yet. The failure mode is not corruption but a None where a category should be.

The bottom line

The categorical dtype in current Polars is a u32 code array plus a pointer to a dictionary that is shared, mutable, and process-global in the default configuration. The dictionary is fast because it is a lock-free concurrent hash table with precomputed string hashes, and it is shared because sharing the dictionary is what makes the codes meaningful in the first place.

The trade is explicit: you get integer-speed grouping, joins and hashing, and you give up the ability to treat a categorical series as a self-contained value. Two series are comparable only if they point at the same Categories object; the global one is shared by everything.

If you want the old, isolated behaviour, do not use the global categorical. Give each logical dataset its own namespace:

import polars as pl

# A dictionary of its own, shared by name across the process but not with
# any other dataset's categoricals.
fruit = pl.Categories("etl", namespace="orders")
df = df.with_columns(pl.col("fruit").cast(pl.Categorical(fruit)))

crates/polars-dtype/src/categorical/mod.rs is worth an hour of reading. It is 370 lines, it is the only place in the engine where a piece of global mutable state is load-bearing for correctness, and it tells you exactly how much Polars is willing to trade for it.