Polars Shares Columns Instead of Copying Them: Arc, Bytes and Reference Counting
Column bytes live in refcounted buffers; sharing a column clones handles, and the Arrow FFI imports foreign memory by keeping the C array alive as the buffer’s owner.
Arrow
Polars Stores Short Strings Inline in 16 Bytes: The View Type
The View type packs a 4-byte length, 4-byte prefix, buffer index and offset into 16 bytes, and inlines any string of 12 bytes or less directly into those bytes.
The Arrow Row Format: How DataFusion Packs a Multi-Column Sort Key Into One Comparable Byte String
Arrow’s RowConverter encodes each key column into a byte layout where lexicographic row order equals byte order, turning a multi-column sort key comparison into a single memcmp.
Polars Buffers Are Immutable: The Refcounted Storage Behind Every Column
The buffer layer of Polars is deliberately dumb: an immutable (ptr, length) view over a hand-rolled refcounted storage, with all growth pushed into the mutable builders.
Dictionary Columns Are a Hash Cache: How DataFusion Groups by Dictionary Index and What That Buys
A dictionary column is a pre-built hash cache: DataFusion hashes each distinct dictionary value once, scatters those hashes to the keys, and memoizes value-index to group-slot so …