Polars Compares Strings Without Allocating
A BinaryView is 16 bytes: a length, a 4-byte prefix, and either 12 inline bytes or a buffer location. Equality is decided by two u64 compares before any byte of string data is …
Polars
Polars' Categorical Encoding Hides a Global Mutable Dictionary in Plain Sight
The categorical dtype is a thin wrapper over a global, lock-free string-to-u32 dictionary. The trick turns group-by and joins into integer work; the same shared state leaks across …
Polars Chunks Its Columns So Parallelism Has Something to Eat
Every column in Polars is a list of immutable Arrow arrays plus cached length and null-count. Kernels run chunk by chunk, so chunk boundaries decide the unit of work, the unit of …
Polars Shares Columns Instead of Copying Them: Arc, Bytes and Reference Counting
Column bytes live in refcounted buffers; sharing a column clones handles, and the Arrow FFI imports foreign memory by keeping the C array alive as the buffer’s owner.
Polars Stores Short Strings Inline in 16 Bytes: The View Type
The View type packs a 4-byte length, 4-byte prefix, buffer index and offset into 16 bytes, and inlines any string of 12 bytes or less directly into those bytes.