Arrow on Opensource internals

Polars Shares Data With Python Over the Arrow C Data Interface

Jay Hubbert — Wed, 21 Oct 2026 08:20:00 +0000

TL;DR: polars-arrow implements the Arrow C Data Interface, which is a contract about pointers and ownership, not about bytes. Exporting a Polars array fills an ArrowArray struct whose buffers point straight at the existing memory, and parks the Rust array in private_data so a release callback can drop it later. Importing does the mirror image: the foreign bytes are borrowed, and the C array itself is kept alive as the owner. The only thing that gets copied is a buffer that cannot legally be borrowed.

Moving a dataframe between Polars and Python looks like it should be a memcpy. It is not, and the reason is that Arrow’s C Data Interface was designed to make the memcpy unnecessary. The interface is three C structs and a set of rules about who owns what; the rules are interesting mostly because they are subtle, and Polars’ implementation is a good place to see them written down.

The three structs

crates/polars-arrow/src/ffi/generated.rs is a rust-bindgen output file containing the ABI-compatible mirrors of the three Arrow C structs. The array one:

 rust
 
 
/// ABI-compatible struct for [`ArrowArray`](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions)
#[repr(C)]
#[derive(Debug)]
pub struct ArrowArray {
 pub(super) length: i64,
 pub(super) null_count: i64,
 pub(super) offset: i64,
 pub(super) n_buffers: i64,
 pub(super) n_children: i64,
 pub(super) buffers: *mut *const ::std::os::raw::c_void,
 pub(super) children: *mut *mut ArrowArray,
 pub(super) dictionary: *mut ArrowArray,
 pub(super) release: ::std::option::Option<unsafe extern "C" fn(arg1: *mut ArrowArray)>,
 pub(super) private_data: *mut ::std::os::raw::c_void,
}

crates/polars-arrow/src/ffi/generated.rs:32

ArrowSchema has the same shape for types (format, name, metadata, flags, children, dictionary, release, private_data), and ArrowArrayStream is the pull-based variant with four function pointers (get_schema, get_next, get_last_error, release) instead of data pointers.

Every one of these has a release field, and that is the load-bearing part of the whole design. The contract is: the producer of the struct owns whatever the struct points at, and the consumer must call release exactly once when it is done. release is where the producer takes its memory back.

Exporting: no bytes move

Exporting an array is ArrowArray::new:

 rust
 
 
 /// creates a new `ArrowArray` from existing data.
 ///
 /// # Safety
 /// This method releases `buffers`. Consumers of this struct *must* call `release` before
 /// releasing this struct, or contents in `buffers` leak.
 pub(crate) fn new(mut array: Box<dyn Array>) -> Self {

crates/polars-arrow/src/ffi/array.rs:103

The buffers array is built from the array’s own buffer pointers:

 rust
 
 
 let buffers_ptr = buffers
 .iter()
 .map(|maybe_buffer| match maybe_buffer {
 Some(b) => *b as *const std::os::raw::c_void,
 None => std::ptr::null(),
 })
 .collect::<Box<[_]>>();
 let n_buffers = buffers.len() as i64;

crates/polars-arrow/src/ffi/array.rs:135

buffers here comes from the array’s ToFfi implementation. For a primitive array that is literally “the address of my values buffer”:

 rust
 
 
unsafe impl<T: NativeType> ToFfi for PrimitiveArray<T> {
 fn buffers(&self) -> Vec<Option<*const u8>> {
 vec![
 self.validity.as_ref().map(|x| x.as_ptr()),
 Some(self.values.storage_ptr().cast::<u8>()),
 ]
 }

crates/polars-arrow/src/array/primitive/ffi.rs:9

storage_ptr() is the address of the allocation, not of the first element. No copy, no allocation, no serialization: the consumer gets a pointer into the same memory Polars is using.

Now the ownership question. The ArrowArray struct has no place to put a Box<dyn Array> — it is a plain C struct. So Polars parks it in private_data:

 rust
 
 
#[allow(dead_code)]
struct PrivateData {
 array: Box<dyn Array>,
 buffers_ptr: Box<[*const std::os::raw::c_void]>,
 children_ptr: Box<[*mut ArrowArray]>,
 dictionary_ptr: Option<*mut ArrowArray>,
 variadic_buffer_sizes: Box<[i64]>,
}

crates/polars-arrow/src/ffi/array.rs:91

and hands the raw pointer to the C struct:

 rust
 
 
 Self {
 length,
 null_count,
 offset: offset as i64,
 n_buffers,
 n_children,
 buffers: private_data.buffers_ptr.as_mut_ptr(),
 children: private_data.children_ptr.as_mut_ptr(),
 dictionary: private_data.dictionary_ptr.unwrap_or(std::ptr::null_mut()),
 release: Some(c_release_array),
 private_data: Box::into_raw(private_data) as *mut ::std::os::raw::c_void,
 }

crates/polars-arrow/src/ffi/array.rs:178

Note the children: a StructArray or ListArray has child arrays, and each child is exported as its own ArrowArray with its own release callback:

 rust
 
 
 let children_ptr = children
 .into_iter()
 .map(|child| {
 Box::into_raw(Box::new(ArrowArray::new(ffi::align_to_c_data_interface(
 child,
 ))))
 })
 .collect::<Box<_>>();
 let n_children = children_ptr.len() as i64;

crates/polars-arrow/src/ffi/array.rs:144

The child’s ArrowArray is leaked into a raw pointer and kept in children_ptr, so the parent’s release callback can free each child in turn. A consumer that wants a nested array’s data has to walk children and call release on each one — the C interface’s tree of structs, each with its own lifetime.

The release callback is the other half of the contract:

 rust
 
 
// callback used to drop [ArrowArray] when it is exported
unsafe extern "C" fn c_release_array(array: *mut ArrowArray) {
 if array.is_null() {
 return;
 }
 let array = &mut *array;

 // take ownership of `private_data`, therefore dropping it
 let private = Box::from_raw(array.private_data as *mut PrivateData);
 for child in private.children_ptr.iter() {
 let _ = Box::from_raw(*child);
 }

 if let Some(ptr) = private.dictionary_ptr {
 let _ = Box::from_raw(ptr);
 }

 array.release = None;
}

crates/polars-arrow/src/ffi/array.rs:76

Dropping PrivateData drops the Box<dyn Array>, which drops its buffers, which decrements their refcounts. The consumer of the C array never has to know any of this; it just calls release. Setting array.release = None at the end is what makes the release idempotent from Rust’s side, because ArrowArray also has a Drop impl that calls release:

 rust
 
 
impl Drop for ArrowArray {
 fn drop(&mut self) {
 match self.release {
 None => (),
 Some(release) => unsafe { release(self) },
 };
 }
}

crates/polars-arrow/src/ffi/array.rs:63

ArrowSchema has the mirror-image callback, c_release_schema, dropping a SchemaPrivateData that holds the type’s CStrings and its children.

The one thing that is copied: alignment

Before exporting, Polars has to make sure the array is in a shape the C interface can express. The awkward case is a validity bitmap whose bit offset does not match the data offset. The C interface requires the validity bitmap to be aligned to the data’s offset, so Polars rebuilds the bitmap:

 rust
 
 
 fn to_ffi_aligned(&self) -> Self {
 let offset = self.values.offset();

 let validity = self.validity.as_ref().map(|bitmap| {
 if bitmap.offset() == offset {
 bitmap.clone()
 } else {
 align(bitmap, offset)
 }
 });

 Self {
 dtype: self.dtype.clone(),
 validity,
 values: self.values.clone(),
 }
 }

crates/polars-arrow/src/array/primitive/ffi.rs:33

align copies the bits into a freshly allocated bitmap; values.clone() is a refcount bump, so the data is still shared. The dispatcher that decides whether an array needs this treatment is in bridge.rs:

 rust
 
 
macro_rules! ffi_dyn {
 ($array:expr, $ty:ty) => {{
 let a = $array.as_any().downcast_ref::<$ty>().unwrap();
 if a.offset().is_some() {
 $array
 } else {
 Box::new(a.to_ffi_aligned())
 }
 }};
}

pub fn align_to_c_data_interface(array: Box<dyn Array>) -> Box<dyn Array> {

crates/polars-arrow/src/ffi/bridge.rs:15

offset() returns Some only when the validity bitmap and the values agree on their offset; when they do not, the array is replaced by an aligned copy.

Importing: borrowing, and keeping the owner alive

Import is the direction where the memory belongs to somebody else, and it is where “zero-copy” is most easily over-claimed. Polars does not copy the bytes, but it does have to keep the C array alive for as long as it uses them.

 rust
 
 
pub unsafe fn import_array_from_c(
 array: ArrowArray,
 dtype: ArrowDataType,
) -> PolarsResult<Box<dyn Array>> {
 try_from(InternalArrowArray::new(array, dtype))
}

crates/polars-arrow/src/ffi/mod.rs:47

Note that import_array_from_c consumes the ArrowArray by value. It is moved into an Arc:

 rust
 
 
#[derive(Debug, Clone)]
pub struct InternalArrowArray {
 // Arc is used for shareability since this is immutable
 array: Arc<ArrowArray>,
 // Arced to reduce cost of cloning
 dtype: Arc<ArrowDataType>,
}

impl InternalArrowArray {
 pub fn new(array: ArrowArray, dtype: ArrowDataType) -> Self {
 Self {
 array: Arc::new(array),
 dtype: Arc::new(dtype),
 }
 }
}

crates/polars-arrow/src/ffi/array.rs:592

Every buffer created from that array carries a clone of the InternalArrowArray as its owner:

 rust
 
 
unsafe fn create_buffer_known_len<T: NativeType>(
 array: &ArrowArray,
 dtype: &ArrowDataType,
 owner: InternalArrowArray,
 len: usize,
 index: usize,
) -> PolarsResult<Buffer<T>> {
 if len == 0 {
 // Zero-length arrays might have invalid pointers for zero-length slices in Rust,
 // so this is more than just an optimization.
 return Ok(Buffer::new());
 }
 let ptr: *mut T = get_buffer_ptr(array, dtype, index)?;
 let slice = core::slice::from_raw_parts(ptr, len);
 let storage = SharedStorage::from_slice_with_owner(slice, owner);
 Ok(Buffer::from_storage(storage))
}

crates/polars-arrow/src/ffi/array.rs:269

SharedStorage::from_slice_with_owner is the ForeignOwner path in polars-buffer: the bytes are not Polars’, but Polars holds the owner so they stay valid. When the last Polars-side handle dies, the Arc<ArrowArray> refcount hits zero, ArrowArray::drop runs, and the producer’s release callback fires — freeing the foreign memory exactly once, in exactly the right place.

The buffer path has an explicit escape hatch for the one case where borrowing is not legal:

 rust
 
 
 // We have to check alignment, for zero-copy to be valid.
 if ptr.is_aligned() {
 let slice = core::slice::from_raw_parts(ptr.add(offset), len);
 let storage = SharedStorage::from_slice_with_owner(slice, owner);
 Ok(Buffer::from_storage(storage))
 } else {
 // Byte-wise copy for misaligned buffers.
 let mut v = Vec::with_capacity(len);
 core::ptr::copy_nonoverlapping(
 ptr.add(offset).cast::<u8>(),
 v.spare_capacity_mut().as_mut_ptr().cast::<u8>(),
 len * size_of::<T>(),
 );
 v.set_len(len);
 Ok(Buffer::from(v))
 }

crates/polars-arrow/src/ffi/array.rs:269

A misaligned pointer cannot legally become a &[T], so Polars copies those bytes into a fresh Vec and owns that instead. This is the honest answer to “is it zero-copy”: yes, except for misaligned buffers, where it has to be or the program is undefined behaviour.

The stream interface

For whole dataframes, Polars uses the pull-based ArrowArrayStream. Exporting one is a small struct with four callbacks:

 rust
 
 
pub fn export_iterator(
 iter: Box<dyn Iterator<Item = PolarsResult<Box<dyn Array>>>>,
 field: Field,
) -> ArrowArrayStream {
 let private_data = Box::new(PrivateData {
 iter,
 field,
 error: None,
 });

 ArrowArrayStream {
 get_schema: Some(get_schema),
 get_next: Some(get_next),
 get_last_error: Some(get_last_error),
 release: Some(release),
 private_data: Box::into_raw(private_data) as *mut ::std::os::raw::c_void,
 }
}

crates/polars-arrow/src/ffi/stream.rs:217

get_next hands out one exported array per call:

 rust
 
 
unsafe extern "C" fn get_next(iter: *mut ArrowArrayStream, array: *mut ArrowArray) -> i32 {
 if iter.is_null() {
 return 2001;
 }
 let private = &mut *((*iter).private_data as *mut PrivateData);

 match private.iter.next() {
 Some(Ok(item)) => {
 // check that the array has the same dtype as field
 let item_dt = item.dtype();
 let expected_dt = private.field.dtype();
 if item_dt != expected_dt {
 private.error = Some(CString::new(format!("The iterator produced an item of data type {item_dt:?} but the producer expects data type {expected_dt:?}").as_bytes().to_vec()).unwrap());
 return 2001; // custom application specific error (since this is never a result of this interface)
 }

 std::ptr::write(array, export_array_to_c(item));

 private.error = None;
 0
 },

crates/polars-arrow/src/ffi/stream.rs:147

The reader side is a Rust iterator over Box<dyn Array>, and the last paragraph of the C stream spec is handled explicitly:

 rust
 
 
 pub unsafe fn next(&mut self) -> Option<PolarsResult<Box<dyn Array>>> {
 let mut array = ArrowArray::empty();
 let status = unsafe { (self.iter.get_next.unwrap())(&mut *self.iter, &mut array) };

 if status != 0 {
 return Some(Err(unsafe { handle_error(&mut self.iter) }));
 }

 // last paragraph of https://arrow.apache.org/docs/format/CStreamInterface.html#c.ArrowArrayStream.get_next
 array.release?;

 // SAFETY: assumed from the C stream interface
 unsafe { import_array_from_c(array, self.field.dtype.clone()) }
 .map(Some)
 .transpose()
 }

crates/polars-arrow/src/ffi/stream.rs:113

array.release? is the end-of-stream signal: a producer that sets release to None says “no more batches”.

The Python side

In Python, this surfaces as the PyCapsule interface. DataFrame.__arrow_c_stream__ is a single call into Rust:

 rust
 
 
 #[allow(unused_variables)]
 #[pyo3(signature = (requested_schema))]
 fn __arrow_c_stream__<'py>(
 &self,
 py: Python<'py>,
 requested_schema: Option<Py<PyAny>>,
 ) -> PyResult<Bound<'py, PyCapsule>> {
 py.enter_polars_ok(|| {
 self.df.write().rechunk_mut_par();
 })?;
 dataframe_to_stream(&self.df.read(), py)
 }

crates/polars-python/src/dataframe/export.rs:197

and dataframe_to_stream turns the frame into a capsule:

 rust
 
 
pub(crate) fn dataframe_to_stream<'py>(
 df: &DataFrame,
 py: Python<'py>,
) -> PyResult<Bound<'py, PyCapsule>> {
 let iter = Box::new(DataFrameStreamIterator::new(df));
 let field = iter.field();
 let stream = ffi::export_iterator(iter, field);
 PyCapsule::new_with_value(py, stream, c"arrow_array_stream")
}

crates/polars-python/src/interop/arrow/to_py.rs:78

The capsule’s name, arrow_array_stream, is the handshake: a consumer checks the name to make sure it knows how to release the pointer inside.

Going the other way, PySeries.from_arrow_c_array and from_arrow_c_stream take the capsule apart. The array variant moves the ArrowArray out of the capsule with a std::ptr::replace that leaves an empty array behind:

 rust
 
 
pub(crate) fn import_array_pycapsules(
 schema_capsule: &Bound<PyCapsule>,
 array_capsule: &Bound<PyCapsule>,
) -> PyResult<(polars_arrow::datatypes::Field, Box<dyn Array>)> {
 let field = import_schema_pycapsule(schema_capsule)?;

 // # Safety
 // array_capsule holds a valid C ArrowArray pointer, as defined by the Arrow PyCapsule
 // Interface
 unsafe {
 let array_ptr = std::ptr::replace(
 array_capsule
 .pointer_checked(Some(c"arrow_array"))?
 .as_ptr() as _,
 ArrowArray::empty(),
 );
 let array = ffi::import_array_from_c(array_ptr, field.dtype().clone()).unwrap();

 Ok((field, array))
 }
}

crates/polars-python/src/series/import.rs:53

The std::ptr::replace is the ownership transfer: the capsule’s pointer slot is overwritten with an empty struct, so whoever drops the capsule afterwards cannot double-free, and Polars now owns the ArrowArray and will call its release when the last handle dies.

On the Python side, pycapsule_to_frame handles both spellings:

 python
 
 
def pycapsule_to_frame(
 obj: Any,
 *,
 schema: SchemaDefinition | None = None,
 schema_overrides: SchemaDict | None = None,
 rechunk: bool = False,
) -> DataFrame:
 """Convert PyCapsule object to DataFrame."""
 if hasattr(obj, "__arrow_c_array__"):
 # This uses the fact that PySeries.from_arrow_c_array will create a
 # struct-typed Series. Then we unpack that to a DataFrame.
 # `struct_unnest` is a single Rust call that releases the GIL; going via
 # `DataFrame.unnest` would run the whole lazy engine while holding it,
 # which serializes construction across Python threads.
 df = wrap_df(PySeries.from_arrow_c_array(obj).struct_unnest())

 elif hasattr(obj, "__arrow_c_stream__"):

py-polars/src/polars/_utils/pycapsule.py:39

The pyarrow bridge: `to_arrow`

DataFrame.to_arrow() does not go through the capsule interface; it goes through pyarrow’s own C data interface import, which is the same trick:

 rust
 
 
pub(crate) fn to_py_array(
 array: ArrayRef,
 field: &ArrowField,
 pyarrow: &Bound<PyModule>,
) -> PyResult<Py<PyAny>> {
 let schema = Box::new(ffi::export_field_to_c(field));
 let array = Box::new(ffi::export_array_to_c(array));

 let schema_ptr: *const ffi::ArrowSchema = &*schema;
 let array_ptr: *const ffi::ArrowArray = &*array;

 let array = pyarrow.getattr("Array")?.call_method1(
 "_import_from_c",
 (array_ptr as Py_uintptr_t, schema_ptr as Py_uintptr_t),
 )?;

 Ok(array.unbind())
}

crates/polars-python/src/interop/arrow/to_py.rs:28

Polars exports an ArrowArray and an ArrowSchema, hands the two raw addresses to pyarrow’s private _import_from_c, and pyarrow takes ownership of them. The buffers are shared; pyarrow’s Array points at Polars’ memory. The same routine builds the RecordBatch for DataFrame.to_arrow(), one column at a time.

This is also why to_arrow is not free of type conversions even though it is free of byte copies: the Arrow type Polars holds has to be expressed in the compat level pyarrow expects, and some Polars types (notably dictionaries with unsigned indices) have no pyarrow equivalent and are cast on the way out.

The older protocol: `polars-ffi`

There is also a second, Polars-specific ABI in the polars-ffi crate, kept for plugins that want to exchange whole Series rather than raw Arrow arrays. It is a SeriesExport struct with its own release callback:

 rust
 
 
/// An FFI exported `Series`.
#[repr(C)]
pub struct SeriesExport {
 field: *mut ArrowSchema,
 // A double ptr, so we can easily release the buffer
 // without dropping the arrays.
 arrays: *mut *mut ArrowArray,
 len: usize,
 release: Option<unsafe extern "C" fn(arg1: *mut SeriesExport)>,
 private_data: *mut std::os::raw::c_void,
}

crates/polars-ffi/src/version_0.rs:14

The version in the module path (version_0) is a promise that the layout will not change under you; polars-ffi exports MAJOR/MINOR constants for plugins to check.

The bottom line

Export copies nothing. The ArrowArray’s buffers are the array’s own pointers; the Rust array is parked in private_data and dropped by the release callback.
Import copies nothing, except misaligned buffers. Those are copied because a misaligned pointer cannot be a &[T]. Ownership is inverted: the imported Buffer keeps the foreign ArrowArray alive as its owner, and the producer’s release fires when the last Polars handle drops.
Two copies are deliberate. A validity bitmap whose offset disagrees with the data offset is rebuilt (bitmap only, not data), and a misaligned buffer is copied byte-wise.
The stream interface is the same contract, pulled. get_next hands out one exported array at a time; a None release means end of stream.
Ownership transfer is a std::ptr::replace. Whoever takes the pointer out of a capsule leaves an empty struct behind, so exactly one party ever calls release.

The practical takeaway: if you are handing data between Polars and another Arrow implementation, use the PyCapsule interface (__arrow_c_stream__ / __arrow_c_array__) and you are sharing memory, not copying it. The one case where you will silently pay for a copy is a column whose buffers are misaligned, which in practice means data that came from somewhere unusual rather than from Polars itself.

Polars Encodes Group Keys as Rows to Make Hashing Cheap

Jay Hubbert — Wed, 21 Oct 2026 06:40:00 +0000

TL;DR: A composite group key is a tuple, and tuples are awkward to hash: you have to combine the hashes of the parts in a way that cannot collide across different arities or orderings. Polars sidesteps that by encoding each row of the key into a byte blob — a null sentinel, big-endian bytes for numbers, a terminator for strings — and then hashing the blob. One hash, one comparison, one allocation.

The encoder is not a black box: it is written so that the encoded bytes sort in the same order as the values, which is what lets the same representation be used for sorting, grouping and joining.

Grouping on one numeric column is easy. The physical value is the key, and the hash of that value is the hash of the key. Grouping on three columns is a different problem. Now the key is a tuple, and a tuple needs a hash function that is consistent across the whole query: two rows are in the same group if and only if their hashes agree and their keys are equal, and the “if and only if” has to survive the hashing.

The tempting shortcut — hash each column and XOR the hashes — is wrong in subtle ways. (a, b) and (b, a) can hash the same; (1, 2) and (1, 2, 0) can hash the same; the fix is to mix in the arity and the position of each column, and then you are maintaining a hand-rolled hash combiner with all the collision risk that implies.

Polars takes the other route: turn the tuple into a byte string, and let a well-tested byte hash do the work.

Encoding a row

The generic encoder lives in the polars-row crate and is used for the “row encoded” key representation. A key column’s contribution to a row is decided by its dtype, and the decision is made by a trait:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:28
/// Encodes a value of a particular fixed width type into bytes
pub trait FixedLengthEncoding: Copy + Debug {
 // 1 is validity 0 or 1
 // bit repr of encoding
 const ENCODED_LEN: usize = 1 + size_of::<Self::Encoded>();

 type Encoded: Sized + Copy + AsRef<[u8]> + AsMut<[u8]>;

 fn encode(self) -> Self::Encoded;

 fn decode(encoded: Self::Encoded) -> Self;

 /// Invert all bits of an encoded value. Used for descending order.
 fn invert(encoded: Self::Encoded) -> Self::Encoded;

 /// Keep the encoded value if `keep`, otherwise all zeros.
 fn keep_or_zero(encoded: Self::Encoded, keep: bool) -> Self::Encoded;

 fn decode_reverse(encoded: Self::Encoded) -> Self {
 Self::decode(Self::invert(encoded))
 }
}

Every fixed-width value is written as one sentinel byte followed by the value’s bytes. The sentinel is 1 for a valid value and 0xFF (or 0x00, when sorting descending) for null:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:230
pub unsafe fn encode<T: NativeType + FixedLengthEncoding>(
 buffer: &mut [MaybeUninit<u8>],
 arr: &PrimitiveArray<T>,
 opt: RowEncodingOptions,
 offsets: &mut [usize],
) {
 if arr.null_count() == 0 {
 encode_slice(buffer, arr.values().as_slice(), opt, offsets)
 } else {
 encode_slice_with_validity(
 buffer,
 arr.values().as_slice(),
 arr.validity().unwrap(),
 opt,
 offsets,
 )
 }
}

/// Write the sentinel byte and the encoded value.
#[inline(always)]
unsafe fn write_value<T: FixedLengthEncoding>(
 dst: *mut MaybeUninit<u8>,
 sentinel: u8,
 encoded: T::Encoded,
) {
 *dst = MaybeUninit::new(sentinel);
 std::ptr::write_unaligned(dst.add(1) as *mut T::Encoded, encoded);
}

/// Write the null sentinel followed by zeros.
#[inline(always)]
unsafe fn write_null<T: FixedLengthEncoding>(dst: *mut MaybeUninit<u8>, null_sentinel: u8) {
 *dst = MaybeUninit::new(null_sentinel);
 std::ptr::write_bytes(dst.add(1), 0, T::ENCODED_LEN - 1);
}

Note that a null is written as the sentinel plus zeros. That matters: it means a null key and a zero-valued key never produce the same bytes, and it means the encoding of a row is fixed width, so the row encoder can compute row offsets before it writes anything.

The value bytes themselves are chosen so that byte order equals value order. Unsigned integers are stored big-endian, which already sorts correctly. Signed integers need one extra step, because in two’s complement -1 sorts after 1 if you compare bytes:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:78
// toggle the sign bit and then encode as big indian
macro_rules! encode_signed {
 ($n:expr, $t:ty) => {
 impl FixedLengthEncoding for $t {
 type Encoded = [u8; $n];

 #[inline(always)]
 fn encode(self) -> [u8; $n] {
 #[cfg(target_endian = "big")]
 {
 todo!()
 }

 let mut b = self.to_be_bytes();
 // Toggle top "sign" bit to ensure consistent sort order
 b[0] ^= 0x80;
 b
 }

Flipping the top bit maps the two’s complement range onto the unsigned range, which is exactly what big-endian byte comparison needs. Floats get the same treatment, plus a canonicalization step so that NaN has one representation and -0.0 sorts with 0.0:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:176
impl FixedLengthEncoding for f64 {
 type Encoded = [u8; 8];

 #[inline]
 fn encode(self) -> [u8; 8] {
 // https://github.com/rust-lang/rust/blob/9c20b2a8cc7588decb6de25ac6a7912dcef24d65/library/core/src/num/f32.rs#L1176-L1260
 let s = canonical_f64(self).to_bits() as i64;
 let val = s ^ (((s >> 63) as u64) >> 1) as i64;
 val.encode()
 }

The ^ (((s >> 63) as u64) >> 1) is the standard “flip all bits if negative” transform, which maps IEEE-754 doubles onto a monotonically increasing integer ordering. Combined with the sign-bit flip in the signed-integer macro, the float encoder reuses the integer encoder.

Descending order is handled by inverting every bit, which turns “ascending” into “descending” without a second code path:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:38
macro_rules! invert_and_keep_or_zero {
 ($t:ty) => {
 #[inline(always)]
 fn invert(encoded: Self::Encoded) -> Self::Encoded {
 (!Self::from_ne_bytes(encoded)).to_ne_bytes()
 }

Strings are the awkward case

Fixed-width values have a known size; strings do not. The string encoding solves this with a trick: since the bytes 0xFC..=0xFF never appear in valid UTF-8, every byte can be shifted by two and the result can never be confused with the terminator or the null sentinel:

 rust
 
 
// crates/polars-row/src/variable/utf8.rs:1
//! Row encoding for UTF-8 strings
//!
//! This encoding is based on the fact that in UTF-8 the bytes 0xFC - 0xFF are never valid bytes.
//! To make this work with the row encoding, we add 2 to each byte which gives us two bytes which
//! never occur in UTF-8 before and after the possible byte range. The values 0x00 and 0xFF are
//! reserved for the null sentinel. The values 0x01 and 0xFE are reserved as a sequence terminator
//! byte.
//!
//! This allows the string row encoding to have a constant 1 byte overhead.

The terminator is what lets a decoder find the end of a string without a length prefix — one byte of overhead per string instead of eight — and it is found with a vectorized byte search:

 rust
 
 
// crates/polars-row/src/variable/utf8.rs:107
/// Encode the bytes of a string followed by the terminator.
///
/// # Safety
/// `dst` must be writable for `s.len() + 1` bytes.
#[inline(always)]
unsafe fn encode_bytes(dst: *mut MaybeUninit<u8>, s: &[u8], xor_mask: u8) {
 if s.len() < BLOCK_SIZE {
 let mut block = [0u8; BLOCK_SIZE];
 std::ptr::copy_nonoverlapping(s.as_ptr(), block.as_mut_ptr(), s.len());
 encode_short(dst, block, s.len(), xor_mask);
 } else {
 for (i, &b) in s.iter().enumerate() {
 *dst.add(i) = MaybeUninit::new(xor_mask ^ (b + 2));
 }
 *dst.add(s.len()) = MaybeUninit::new(xor_mask ^ 0x01);
 }
}

Note that strings are not fixed width, so a row containing a string has a variable length. That is the main structural difference between the generic row encoding and the newer fixed-width key rows described at the end.

Hashing the bytes

Once the rows are encoded, hashing them is trivial — one hash per row:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:146
 if use_row_encoding {
 let keys = df.columns();
 let mut keys_encoded = _get_rows_encoded_unordered(keys).unwrap().into_array();

 if !null_is_valid {
 let validities = keys
 .iter()
 .map(|c| c.as_materialized_series().rechunk_validity())
 .collect_vec();
 let combined = combine_validities_and_many(&validities);
 keys_encoded.set_validity(combined);
 }

 // TODO: use vechash? Not supported yet for lists.
 // let mut hashes = Vec::with_capacity(df.height());
 // columns_to_hashes(df.columns(), Some(random_state), &mut hashes).unwrap();

 let hashes = keys_encoded
 .values_iter()
 .map(|k| random_state.hash_one(k))
 .collect();
 Self::RowEncoded(RowEncodedKeys {
 hashes: PrimitiveArray::from_vec(hashes),
 keys: keys_encoded,
 })

Two properties make this work. First, the hash is computed over the whole row, so the arity and the order of the columns are baked into the bytes — there is no way for (a, b) to collide with (b, a) unless the bytes themselves are equal. Second, the hash is computed once, up front, and stored alongside the keys:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:116
/// Represents a DataFrame plus a hash per row, intended for keys in grouping
/// or joining. The hashes may or may not actually be physically pre-computed,
/// this depends per type.
#[derive(Clone, Debug)]
pub enum HashKeys {
 RowEncoded(RowEncodedKeys),
 KeyRows(KeyRowKeys),
 Binview(BinviewKeys),
 Single(SingleKeys),
}

Pre-hashing is what makes partitioned joins possible: the hash decides which partition a key belongs to, so a probe only has to look at one partition’s table instead of all of them.

The map that stores the groups

The row-encoded grouper keeps a hash map from row bytes to group index. The map is specialised for exactly this shape — pre-hashed byte keys — and it stores the key bytes in an arena of buffers rather than allocating a Vec per key:

 rust
 
 
// crates/polars-utils/src/idx_map/bytes_idx_map.rs:25
/// An IndexMap where the keys are always [u8] slices which are pre-hashed.
pub struct BytesIndexMap<V> {
 table: HashTable<IdxSize>,
 tuples: Vec<(Key, V)>,
 key_data: Vec<Vec<u8>>,

 // Internal random seed used to keep hash iteration order decorrelated.
 // We simply store a random odd number and multiply the canonical hash by it.
 seed: u64,
}

The lookup is a hash compare followed by a byte compare:

 rust
 
 
// crates/polars-utils/src/idx_map/bytes_idx_map.rs:79
 /// Gets the index by insertion order of the given key.
 pub fn get_index_of(&self, hash: u64, key: &[u8]) -> Option<IdxSize> {
 self.table
 .find(hash.wrapping_mul(self.seed), |i| unsafe {
 let t = self.tuples.get_unchecked(*i as usize);
 hash == t.0.key_hash && key == t.0.get(&self.key_data)
 })
 .copied()
 }

That key == t.0.get(&self.key_data) is a memcmp over the row bytes. It is the whole equality check for a composite key — no per-column dispatch, no null handling at comparison time (nulls are in the bytes), no tuple comparison.

And the grouper itself is four lines:

 rust
 
 
// crates/polars-expr/src/groups/row_encoded.rs:23
 fn insert_key(&mut self, hash: u64, key: &[u8]) -> IdxSize {
 match self.idx_map.entry(hash, key) {
 Entry::Occupied(o) => o.index(),
 Entry::Vacant(v) => {
 let index = v.index();
 v.insert(());
 index
 },
 }
 }

The index handed out is the insertion order index, which is why the keys come back in group order for free.

When no encoding is needed

All of the above is machinery that exists only because there is more than one key column, or because the key is a string. For a single numeric key, Polars skips the encoding entirely:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:13
pub struct SingleKeyHashGrouper<T: PolarsDataType> {
 /// Each key with its group.
 table: HashTable<(T::Physical<'static>, IdxSize)>,
 num_groups: IdxSize,
 /// The group of the null key, IdxSize::MAX if there is none.
 null_idx: IdxSize,
 random_state: PlRandomState,
}

The hash table is keyed by the physical value itself, and the null key gets its own group index instead of being hashed at all, so it can never be confused with a real value:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:100
 for (hash, idx) in hashes.iter().zip(block) {
 let idx = *idx as usize;
 if validity.is_some_and(|v| !v.get_bit_unchecked(idx)) {
 if null_is_valid {
 if self.null_idx == IdxSize::MAX {
 self.null_idx = self.num_groups;
 self.num_groups += 1;
 }
 on_group(self.null_idx);
 }
 } else {
 let key = arr.value_unchecked(idx);
 match self.table.entry(*hash, |(k, _)| k.tot_eq(&key), hasher) {
 Entry::Occupied(o) => on_group(o.get().1),
 Entry::Vacant(v) => {
 v.insert((key, self.num_groups));
 on_group(self.num_groups);
 self.num_groups += 1;
 },
 }
 }
 }

The choice between the representations is made once, from the key schema:

 rust
 
 
// crates/polars-expr/src/groups/mod.rs:92
pub fn new_hash_grouper(key_schema: Arc<Schema>) -> Box<dyn Grouper> {
 if let Some(layout) = KeyRowLayout::new(key_schema.iter_values()) {
 Box::new(key_rows::KeyRowHashGrouper::new(Arc::new(layout)))
 } else if key_schema.len() > 1 {
 Box::new(row_encoded::RowEncodedHashGrouper::new())
 } else {
 let (_name, dt) = key_schema.get_at_index(0).unwrap();
 match dt {
 dt if dt.is_primitive_numeric() | dt.is_temporal() => {
 with_match_physical_numeric_polars_type!(dt.to_physical(), |$T| {
 Box::new(single_key::SingleKeyHashGrouper::<$T>::new())
 })
 },

Read it as a ladder: if the schema can be laid out as fixed-width key rows, use that; otherwise if there is more than one column, use generic row encoding; otherwise use the most specialized grouper the single column’s dtype allows.

Key rows: the fixed-width version

The generic row encoding has one structural weakness: string columns make rows variable length, which means row offsets are not known up front and the bytes are not word-aligned. The newer key row representation fixes both by computing a layout from the schema alone:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:72
/// The row format of keys of one schema. It depends only on the key dtypes: the null
/// bits are always present, also when nulls are not keys, so keys that disagree on
/// whether nulls match (the two sides of an anti join) share one layout.
///
/// Rows are written into zeroed words and null values are left zero, so equal keys
/// have equal words, including padding. The one exception is the second word of a
/// long view, which holds where its bytes are stored: equality skips that word and
/// compares the bytes instead.
#[derive(Debug, PartialEq)]
pub(crate) struct KeyRowLayout {
 pub(super) cols: Vec<ColLayout>,
 /// Byte offset of the null bits, one per column.
 pub(super) null_offset: usize,
 null_width: usize,
 pub(super) stride_words: usize,
 /// The first word of each view column, in column order.
 pub(super) view_words: Vec<usize>,
 /// Every word except the second half of each view.
 pub(super) plain_words: Vec<usize>,
}

Three decisions are worth pulling out.

Columns are ordered by width, descending. The layout sorts the fields so that the widest column comes first, which (a) makes the stride a power of two number of words and (b) means the comparison bails out early on the most discriminating column:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:90
 let null_width = cols.len().div_ceil(8).next_power_of_two();
 let mut fields: Vec<(usize, Option<usize>)> = widths
 .into_iter()
 .enumerate()
 .map(|(i, width)| (width, Some(i)))
 .chain([(null_width, None)])
 .collect();
 fields.sort_by_key(|(width, _)| Reverse(*width));

Nulls live inside the row. One bit per key column, always allocated even when no column has nulls. That is deliberate: it means the two sides of an anti join — where one side treats null as a key and the other does not — can share one layout and one hash table.

Equality is an XOR fold over words. Because rows are written into zeroed words, “equal” means “no differing bits”, which is one instruction per word for the fixed part:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:170
 pub(super) unsafe fn rows_eq(
 &self,
 a: &[u64],
 b: &[u64],
 mut long_eq: impl FnMut(View, View) -> bool,
 ) -> bool {
 if self.view_words.is_empty() {
 return a.iter().zip(b).fold(0, |acc, (x, y)| acc | (x ^ y)) == 0;
 }
 for w in &self.plain_words {
 if a.get_unchecked(*w) != b.get_unchecked(*w) {
 return false;
 }
 }

The strings are the awkward case again. An Arrow “view” is 16 bytes: short strings are inlined, long strings point into a buffer, and the second half of the view word holds a location, not data. So the word comparison skips that word and compares the bytes instead.

The hash is derived from the columns rather than from the row bytes, by folding each column’s contribution into a per-row hash:

 rust
 
 
// crates/polars-expr/src/key_rows/keys.rs:338
 let mut hashes = vec![random_state.hash_one(HASH_MULTIPLE); len];
 for col in &cols {
 col.hash_into(&mut hashes, random_state);
 }

The bottom line

The design in one sentence: encode the key once, then let bytes do all the work.

The trade-offs:

Encoding costs memory and a copy. Every distinct key is materialized as bytes and stored in the map’s arena. For a group-by on a wide string key that is a real cost, which is why single string keys get their own grouper that compares Arrow views directly instead of copying bytes.
The encoding is a format, so it has invariants. The sentinel byte, the terminator byte and the shifted UTF-8 bytes all have to be mutually exclusive, and the decoder has to know which convention a column was encoded with. RowEncodingOptions carries the descending and null-placement flags so that a row encoded one way is never decoded as another.
Fixed width buys alignment, at the cost of limits. The key-row layout needs a power-of-two word stride, one null-bit word, and at most two string columns; beyond that it gives up and the generic encoder takes over. The limits are explicit in the source (MAX_KEY_COLUMNS = 64, MAX_VIEW_COLUMNS = 2, crates/polars-expr/src/key_rows/layout.rs:13).
Sorting comes for free. Because the encoded bytes are order-preserving, the same representation can be sorted, compared and hashed. That is not an accident; it is why the sign bit is flipped and why floats are canonicalized before they are written.

Polars Groups Columns So One Pass Feeds Many Aggregations

Jay Hubbert — Wed, 21 Oct 2026 06:20:00 +0000

TL;DR: A group-by in Polars is two stages, not one. The first stage turns key columns into group ids: the keys are encoded (four different ways, depending on their dtype and count), hashed, and looked up in a hash table that hands out a group index per distinct key. The second stage is every aggregation expression, each of which consumes the same group structure. One grouping pass, many aggregations.

The streaming engine pushes this further: a fixed-size hot table keeps the frequent keys in memory, the rare ones are spilled, and the hot table is allowed to evict.

A naive group-by implementation computes the groups once per aggregation: df.group_by("k").agg(sum("a")) and df.group_by("k").agg(mean("b")) would each build their own hash table over the keys. That is a lot of duplicated work, and it scales badly with the number of aggregations.

Polars separates the two concerns. The grouping structure is computed once, and the aggregations are expressions that read it.

Two grouping engines, not one

Before going further: the type you may know as GroupsProxy is not in this tree. It has been split in two, and the split follows the execution engine:

the eager engine (polars-core) groups a materialized DataFrame into group tuples — GroupsType::Idx (a list of row indices per group) or GroupsType::Slice (a list of (offset, len) ranges, for sorted input);
the lazy/streaming engine (polars-expr) groups hash keys into group indices through the Grouper trait, because in a streaming query the keys arrive in morsels and the group table has to grow incrementally.

The eager one is the older design and is worth a look, because it is where the “sorted input” fast path lives:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:20
/// Used to create the tuples for a group_by operation.
pub trait IntoGroupsType {
 /// Create the tuples need for a group_by operation.
 /// * The first value in the tuple is the first index of the group.
 /// * The second value in the tuple is the indexes of the groups including the first value.
 fn group_tuples(&self, _multithreaded: bool, _sorted: bool) -> PolarsResult<GroupsType> {
 unimplemented!()
 }
}

Note the sorted flag: it is a promise from the caller that the key column is already sorted, and it unlocks a path that never builds a hash table at all:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:65
 fn create_groups_from_sorted(&self, multithreaded: bool) -> GroupsSlice {
 if verbose() {
 eprintln!("group_by keys are sorted; running sorted key fast path");
 }

Sorted keys mean the groups are contiguous ranges of rows, so the whole grouping reduces to a scan for run boundaries, which can be done in parallel over partitions of the sorted array.

For everything else, the eager path hashes:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:30
fn num_groups_proxy<T>(ca: &ChunkedArray<T>, multithreaded: bool, sorted: bool) -> GroupsType
where
 T: PolarsNumericType,
 T::Native: TotalHash + TotalEq + DirtyHash + ToTotalOrd,

TotalHash/TotalEq are the important detail: they give floats a total order, so NaN == NaN and -0.0 == 0.0 behave sanely inside a hash table.

The `Grouper` trait

The lazy/streaming side has one trait, and it is small:

 rust
 
 
// crates/polars-expr/src/groups/mod.rs:45
/// A Grouper maps keys to groups, such that duplicate keys map to the same group.
pub trait Grouper: Any + Send + Sync {
 /// Creates a new empty Grouper similar to this one.
 fn new_empty(&self) -> Box<dyn Grouper>;

 /// Reserves space for the given number additional groups.
 fn reserve(&mut self, additional: usize);

 /// Returns the number of groups in this Grouper.
 fn num_groups(&self) -> IdxSize;

 /// Inserts the given subset of keys into this Grouper. If groups_idxs is
 /// passed it is extended such with the group index of keys[subset[i]].
 /// New groups get consecutive indices in the order they first occur.
 ///
 /// # Safety
 /// The subset indexes must be in-bounds.
 unsafe fn insert_keys_subset(
 &mut self,
 keys: &HashKeys,
 subset: &[IdxSize],
 group_idxs: Option<&mut Vec<IdxSize>>,
 );

 /// Returns the keys in this Grouper in group order, that is the key for
 /// group i is returned in row i.
 fn get_keys_in_group_order(&self, schema: &Schema) -> DataFrame;

That is the whole contract: insert keys, get group indices out, hand back the distinct keys in group order. The trait does not mention aggregations at all, which is exactly the point — the grouping structure is reusable by any number of consumers.

The other three methods on the trait (probe_partitioned_groupers, contains_key_partitioned_groupers, mark_groups_partitioned_groupers) exist for partitioned hash joins: they probe other groupers by hash partition, so a join can look up a key in the partition that owns it without searching all of them.

Four encodings, one interface

The interesting decision is how to represent a key so that it can be hashed and compared. Polars picks between four encodings based on the key schema:

 rust
 
 
// crates/polars-expr/src/groups/mod.rs:113
pub fn new_hash_grouper(key_schema: Arc<Schema>) -> Box<dyn Grouper> {
 if let Some(layout) = KeyRowLayout::new(key_schema.iter_values()) {
 Box::new(key_rows::KeyRowHashGrouper::new(Arc::new(layout)))
 } else if key_schema.len() > 1 {
 Box::new(row_encoded::RowEncodedHashGrouper::new())
 } else {
 let (_name, dt) = key_schema.get_at_index(0).unwrap();
 match dt {
 dt if dt.is_primitive_numeric() | dt.is_temporal() => {
 with_match_physical_numeric_polars_type!(dt.to_physical(), |$T| {
 Box::new(single_key::SingleKeyHashGrouper::<$T>::new())
 })
 },

 #[cfg(feature = "dtype-decimal")]
 DataType::Decimal(_, _) => {
 Box::new(single_key::SingleKeyHashGrouper::<Int128Type>::new())
 },
 #[cfg(feature = "dtype-categorical")]
 dt @ (DataType::Enum(_, _) | DataType::Categorical(_, _)) => {
 with_match_categorical_physical_type!(dt.cat_physical().unwrap(), |$C| {
 Box::new(single_key::SingleKeyHashGrouper::<<$C as PolarsCategoricalType>::PolarsPhysical>::new())
 })
 },

 DataType::String | DataType::Binary => Box::new(binview::BinviewHashGrouper::new()),

 _ => Box::new(row_encoded::RowEncodedHashGrouper::new()),
 }
 }
}

Reading that top to bottom: if the key columns can be laid out as fixed-width key rows, use that; otherwise if there is more than one key column, use generic row encoding; otherwise, for a single key, use the most specialized grouper the dtype allows — a direct hash table over the physical values for numbers and dates, a view-aware one for strings, and generic row encoding for anything exotic.

Single key: the values themselves

For one numeric or temporal key, no encoding is needed at all — the physical value is the key:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:13
pub struct SingleKeyHashGrouper<T: PolarsDataType> {
 /// Each key with its group.
 table: HashTable<(T::Physical<'static>, IdxSize)>,
 num_groups: IdxSize,
 /// The group of the null key, IdxSize::MAX if there is none.
 null_idx: IdxSize,
 random_state: PlRandomState,
}

and the insert path is a textbook open-addressing hash table, with two hardware-friendly touches. First, the null key gets its own group index rather than being hashed, so nulls are never confused with a real value. Second, the work is done in blocks: the hashes of a whole block are computed (and prefetched) before the lookups happen, so the CPU has several independent memory accesses in flight:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:64
 unsafe fn insert_impl(
 &mut self,
 arr: &T::Array,
 null_is_valid: bool,
 subset: &[IdxSize],
 mut on_group: impl FnMut(IdxSize),
 ) {

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:82
 let mut hashes = [0; BLOCK_SIZE];
 for block in subset.chunks(BLOCK_SIZE) {
 // Reserve up front so the table doesn't move after prefetching.
 self.table.reserve(block.len(), hasher);
 let prefetch = self.should_prefetch();
 for (hash, idx) in hashes.iter_mut().zip(block) {
 *hash = random_state.tot_hash_one(arr.value_unchecked(*idx as usize));
 if prefetch {
 self.table.prefetch_insert(*hash);
 }
 }

That “reserve up front so the table doesn’t move after prefetching” comment is the kind of detail that only shows up after a profiler: reallocating the table after issuing prefetches would invalidate the addresses you just prefetched.

Key rows: a hand-rolled row format

The newest and most interesting encoding is key_rows. Instead of using the generic polars-row encoder, Polars lays out the key columns itself into a row format with a fixed stride, and the layout is computed once per schema:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:72
/// The row format of keys of one schema. It depends only on the key dtypes: the null
/// bits are always present, also when nulls are not keys, so keys that disagree on
/// whether nulls match (the two sides of an anti join) share one layout.
///
/// Rows are written into zeroed words and null values are left zero, so equal keys
/// have equal words, including padding. The one exception is the second word of a
/// long view, which holds where its bytes are stored: equality skips that word and
/// compares the bytes instead.
#[derive(Debug, PartialEq)]
pub(crate) struct KeyRowLayout {
 pub(super) cols: Vec<ColLayout>,
 /// Byte offset of the null bits, one per column.
 pub(super) null_offset: usize,
 null_width: usize,
 pub(super) stride_words: usize,
 /// The first word of each view column, in column order.
 pub(super) view_words: Vec<usize>,
 /// Every word except the second half of each view.
 pub(super) plain_words: Vec<usize>,
}

Three design decisions are worth spelling out:

Nulls live in the row. One null bit per key column, always present even when nulls are not part of the key. That means the two sides of an anti join (where one side treats null as a key and the other does not) can share one layout and one hash table.
Columns are sorted by width, descending. The layout puts the widest columns first so that the whole row is a power-of-two number of 64-bit words, and so that comparisons can bail out early on the most discriminating word:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:90
 let null_width = cols.len().div_ceil(8).next_power_of_two();
 let mut fields: Vec<(usize, Option<usize>)> = widths
 .into_iter()
 .enumerate()
 .map(|(i, width)| (width, Some(i)))
 .chain([(null_width, None)])
 .collect();
 fields.sort_by_key(|(width, _)| Reverse(*width));

Equality is a word-wise XOR fold. Because rows are written into zeroed words, “equal” means “no differing bits”, and that is one instruction per word for the fixed-width part:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:170
 pub(super) unsafe fn rows_eq(
 &self,
 a: &[u64],
 b: &[u64],
 mut long_eq: impl FnMut(View, View) -> bool,
 ) -> bool {
 if self.view_words.is_empty() {
 return a.iter().zip(b).fold(0, |acc, (x, y)| acc | (x ^ y)) == 0;
 }
 for w in &self.plain_words {
 if a.get_unchecked(*w) != b.get_unchecked(*w) {
 return false;
 }
 }

Strings (Arrow “views”) are the awkward case: a view is a 16-byte struct where short strings are inlined and long strings point into a buffer. The second half of a view word holds a location, not data, so it is skipped by the word comparison and the bytes are compared instead.

Row encoding and binview: the fallbacks

Generic row encoding uses the polars-row crate and stores the encoded bytes in a byte-indexed map:

 rust
 
 
// crates/polars-expr/src/groups/row_encoded.rs:12
#[derive(Default)]
pub struct RowEncodedHashGrouper {
 idx_map: BytesIndexMap<()>,
}

 rust
 
 
// crates/polars-expr/src/groups/row_encoded.rs:23
 fn insert_key(&mut self, hash: u64, key: &[u8]) -> IdxSize {
 match self.idx_map.entry(hash, key) {
 Entry::Occupied(o) => o.index(),
 Entry::Vacant(v) => {
 let index = v.index();
 v.insert(());
 index
 },
 }
 }

For a single string key, there is a dedicated grouper that compares the Arrow views directly, so a 100 MB string column is never copied into a row buffer:

 rust
 
 
// crates/polars-expr/src/groups/binview.rs:33
 /// # Safety
 /// The view must be valid for the given buffer set.
 #[inline(always)]
 unsafe fn insert_key(
 &mut self,
 hash: u64,
 view: View,
 buffers: &Buffer<Buffer<u8>>,
 ) -> IdxSize {
 unsafe {
 match self.idx_map.entry_view(hash, view, buffers) {

The keys themselves

The Grouper does not take a DataFrame of keys; it takes a HashKeys, which is the encoded key material plus a precomputed hash per row:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:116
/// Represents a DataFrame plus a hash per row, intended for keys in grouping
/// or joining. The hashes may or may not actually be physically pre-computed,
/// this depends per type.
#[derive(Clone, Debug)]
pub enum HashKeys {
 RowEncoded(RowEncodedKeys),
 KeyRows(KeyRowKeys),
 Binview(BinviewKeys),
 Single(SingleKeys),
}

The from_df constructor is where the encoding decision is actually made, and it prefers key rows over everything else:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:130
 if !force_row_encoding
 && let Some(layout) = KeyRowLayout::new(df.columns().iter().map(|c| c.dtype()))
 {
 return Self::KeyRows(KeyRowKeys::from_columns(
 df.columns(),
 Arc::new(layout),
 &random_state,
 null_is_valid,
 ));
 }

 let first_col_variant = hash_keys_variant_for_dtype(df[0].dtype());
 let use_row_encoding = force_row_encoding
 || df.width() > 1
 || first_col_variant == HashKeysVariant::RowEncoded;

Precomputing the hash per row is what makes partitioned joins and the partitioned groupers possible: a key’s hash determines its partition, so a probe only has to look in one partition’s table.

One pass, many aggregations

Now the payoff. The eager engine groups once and then runs every aggregation against the same group structure:

 rust
 
 
// crates/polars-mem-engine/src/executors/group_by.rs:68
pub(super) fn group_by_helper(
 mut df: DataFrame,
 keys: Vec<Column>,
 aggs: &[Arc<dyn PhysicalExpr>],
 apply: Option<PlanCallback<DataFrame, DataFrame>>,
 state: &ExecutionState,
 maintain_order: bool,
 output_schema: &SchemaRef,
 slice: Option<(i64, usize)>,
) -> PolarsResult<DataFrame> {
 df.rechunk_mut_par();
 let gb = df.group_by_with_series(keys, true, maintain_order)?;

 rust
 
 
// crates/polars-mem-engine/src/executors/group_by.rs:15
pub(super) fn evaluate_aggs(
 df: &DataFrame,
 aggs: &[Arc<dyn PhysicalExpr>],
 groups: &GroupPositions,
 state: &ExecutionState,
) -> PolarsResult<Vec<Column>> {
 RAYON.install(|| {
 aggs.par_iter()
 .map(|expr| {
 let agg = expr.evaluate_on_groups(df, groups, state)?.finalize();
 polars_ensure!(agg.len() == groups.len(), agg_len = agg.len(), groups.len());
 Ok(agg)
 })
 .collect::<PolarsResult<Vec<_>>>()
 })
}

groups is computed once and then shared, read-only, by every aggregation expression, which are themselves free to run in parallel. The lazy engine does the same thing, one level down: an aggregation expression receives the shared GroupPositions and reduces its own input column against it:

 rust
 
 
// crates/polars-expr/src/expressions/aggregation.rs:232
 fn evaluate_on_groups_impl<'a>(
 &self,
 df: &DataFrame,
 groups: &'a GroupPositions,
 state: &ExecutionState,
 ) -> PolarsResult<AggregationContext<'a>> {
 let mut ac = self.input.evaluate_on_groups(df, groups, state)?;

The GroupedReduction trait is the streaming-engine equivalent, and it is designed so that many reductions can share one grouping structure:

 rust
 
 
// crates/polars-expr/src/reduce/mod.rs:163
 /// Updates the specified group with the given values.
 ///
 /// For order-sensitive grouped reductions, seq_id can be used to resolve
 /// order between calls/multiple reductions.
 fn update_group(
 &mut self,
 values: &[&Column],
 group_idx: IdxSize,
 seq_id: u64,
 ) -> PolarsResult<()>;

and the partition table of the streaming group-by shows the pattern in full: one grouper, N reductions, one pass over the rows:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/table.rs:143
 pub fn add_rows(
 &mut self,
 input: usize,
 frame: &FrameWithKeys,
 rows: Option<&[IdxSize]>,
 ) -> PolarsResult<()> {
 let rows = rows_or_all(rows, &mut self.all_rows, frame.keys.len());
 self.group_idxs.clear();
 unsafe {
 self.grouper
 .insert_keys_subset(&frame.keys, rows, Some(&mut self.group_idxs));
 }
 let num_groups = self.grouper.num_groups();
 let params = &*self.params;
 let mut in_cols = Vec::new();
 for r_idx in &params.reductions_per_input[input] {
 let r = &mut self.reductions[*r_idx];
 r.resize(num_groups);
 in_cols.clear();
 in_cols.extend(
 params.grouped_reduction_cols[*r_idx]
 .iter()
 .map(|col| frame.frame.column(col).unwrap()),
 );
 // Cold rows only exist without order-sensitive reductions.
 unsafe { r.update_groups_subset(&in_cols, rows, &self.group_idxs, 0)? };
 }
 Ok(())
 }

The group indices computed by the grouper are handed to every reduction. r.resize(num_groups) grows the reduction’s state to the number of groups found so far; update_groups_subset then adds the values of the given rows to the right groups. Adding a tenth aggregation to a query costs a tenth reduction state and a tenth pass over the group’s rows — but not a tenth grouping pass.

The hot table

There is a second, more aggressive version of this idea in the streaming engine. Because a group-by over a large table has to hold the group table in memory, Polars splits the keys into hot (frequent) and cold (rare) ones:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/mod.rs:1
//! The streaming group-by.
//!
//! While receiving input, each pipeline aggregates rows with frequent keys in small hot tables
//! and stores the other (cold) rows with their hashed keys in spillable frames. Groups evicted
//! from a hot table, and the hot tables at the end of the input, become pre-aggregates. Once all
//! input is received, the cold rows and pre-aggregates are split into hash partitions, which are
//! aggregated independently. With at most one partition per pipeline, all partitions are built
//! before sending. Otherwise the cold rows are first stored per partition, and the partitions are
//! built a few at a time while the results are sent.

The hot table is a HotGrouper, a sibling of Grouper with one crucial difference — it has a fixed size and evicts rather than growing:

 rust
 
 
// crates/polars-expr/src/hot_groups/mod.rs:21
/// A HotGrouper maps keys to groups, such that duplicate keys map to the same
/// group. Unlike a Grouper it has a fixed size and will cause evictions rather
/// than growing.
pub trait HotGrouper: Any + Send + Sync {
 /// Creates a new empty HotGrouper similar to this one, with the given size.
 fn new_empty(&self, groups: usize) -> Box<dyn HotGrouper>;

 /// Returns the number of groups in this HotGrouper.
 fn num_groups(&self) -> IdxSize;

 /// Returns the number of slots in the hot table.
 fn num_slots(&self) -> usize;

 /// Doubles the number of slots in the hot table. Group indices are unchanged.
 fn double(&mut self);

 /// Inserts the given keys into this Grouper, extending groups_idxs with
 /// the group index of keys[i].
 ///
 /// A missed key that is followed by a key with the same hash is inserted
 /// even if that evicts another key, as is every missed key with
 /// `force_hot`. This keeps sorted keys from first going cold.
 fn insert_keys(
 &mut self,
 keys: &HashKeys,
 hot_idxs: &mut Vec<IdxSize>,
 hot_group_idxs: &mut Vec<EvictIdx>,
 cold_idxs: &mut Vec<IdxSize>,
 force_hot: bool,
 );

The EvictIdx type is the trick that makes this work: a group index whose top bit is set means “this group was evicted, reset it before updating”. The reduction state carries the same flag, so an evicted group’s partial sum is discarded and rebuilt from the cold rows later. The note about “a missed key that is followed by a key with the same hash” is about sorted input: if the keys are sorted, the same key appears in a run, and evicting the first occurrence and re-inserting the second would defeat the purpose, so the grouper is allowed to evict on a hash collision instead.

The sink keeps both halves:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/sink.rs:60
/// The state of one pipeline of the sink.
struct LocalSink {
 // Created when the input first sends this pipeline a non-empty morsel.
 hot_grouper_per_input: Vec<Option<Box<dyn HotGrouper>>>,
 hot_grouped_reductions: Vec<Box<dyn GroupedReduction>>,

 // The number of slots of each hot grouper, and the rows missed by them.
 hot_table_size: usize,
 miss_f2: F2Sketch,

 // The keys of the cold rows and the pre-aggregates.
 key_sketch: CardinalitySketch,

 // Per input, the cold rows not yet in a frame.
 cold_row_builders: Vec<Option<FrameWithKeysBuilder>>,
 // Each frame with the input it came from.
 cold_frames: Vec<(usize, SpillFrameWithKeys)>,
 // The estimated bytes of `cold_frames`.
 cold_bytes: u64,
 // In creation order.
 pre_aggs: Vec<PreAgg>,
}

The hot table grows only when the evidence says it should — the F2 sketch counts how often keys are missed, and the table is enlarged only if the missed keys repeat enough and dominate the misses:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/sink.rs:47
/// The hot tables only grow if the frequently missed keys were missed at least this
/// many times each.
const HOT_TABLE_GROW_MIN_REPEAT: f64 = 4.0;

/// The hot tables only grow if the frequently missed keys make up at least this
/// fraction of the missed rows.
const HOT_TABLE_GROW_MIN_HEAVY_SHARE: f64 = 0.5;

The bottom line

The design in one sentence: grouping is a data structure, not a step. The group structure is built once, by a Grouper that picks an encoding suited to the key schema, and then it is a read-only input to every aggregation.

The trade-offs that come with it:

Encoding choice is per-query, not per-user. A single numeric key gets a direct hash table over the values; a multi-column key gets fixed-width key rows; strings get view comparisons; anything else falls back to generic row encoding. The cost is four implementations of one trait, and the benefit is that the common cases avoid copying key data at all.
The group structure is shared, so it is immutable during aggregation. Every aggregation reads it; none of them may reorder it. That is why evaluate_aggs can run the aggregations in parallel over one groups.
The hot table trades exactness of memory prediction for speed. It is a fixed-size cache of frequent keys, and it evicts. The evicted groups are not lost, they are re-derived from the spilled cold rows, which means the grouping work is done twice for evicted keys — a deliberate exchange of CPU for a bounded memory footprint.

Nulls Are Not Values: How DataFusion Skips Work Using Arrow's Validity Bitmaps

Jay Hubbert — Wed, 21 Oct 2026 06:13:00 +0000

TL;DR: In Arrow, a null is not a value in the data buffer — it is a bit in a separate validity bitmap, and None for the bitmap means “no nulls anywhere”, which is not the same as “all bits set”. That separation is a performance lever, because it lets DataFusion ask a single cheap question (“does this array have any nulls at all?”) and, based on the answer, skip an entire category of per-row work: null-aware comparisons, null-checking appends, null-aware bit unions, and the whole null-buffer allocation. The trick appears in the GROUP BY builders, in the AND/OR short-circuit logic, in IN list evaluation, and in the scatter/filter primitives.

The representation

Arrow’s NullBuffer is a bit-packed validity bitmap plus a cached null count:

 rust
 
 
// arrow-buffer-60.0.0/src/buffer/null.rs:33-44
#[derive(Debug, Clone, Eq, PartialEq)]
pub struct NullBuffer {
 buffer: BooleanBuffer,
 null_count: usize,
}

impl NullBuffer {
 /// Create a new [`NullBuffer`] computing the null count
 pub fn new(buffer: BooleanBuffer) -> Self {
 let null_count = buffer.len() - buffer.count_set_bits();
 Self { buffer, null_count }
 }

Two details matter enormously:

The bitmap is a validity bitmap. A set bit means “this slot has a value”. is_valid(idx) reads the bit; is_null(idx) is its negation.
The bitmap is optional. Array::nulls() returns Option<&NullBuffer>, and None means there are no nulls at all — not “all bits set”. This is the crucial optimization: a column of a million non-null Int32s has a values buffer of 4 MB and no validity bitmap whatsoever.

 rust
 
 
// arrow-buffer-60.0.0/src/buffer/null.rs:46-64
 /// Create a new [`NullBuffer`] of length `len` where all values are null
 pub fn new_null(len: usize) -> Self {
 Self {
 buffer: BooleanBuffer::new_unset(len),
 null_count: len,
 }
 }

 /// Create a new [`NullBuffer`] of length `len` where all values are valid
 ///
 /// Note: it is more efficient to not set the null buffer if it is known to
 /// be all valid (aka all values are not null)
 pub fn new_valid(len: usize) -> Self {

The comment in Arrow’s own source says it outright: don’t materialize an all-ones bitmap; use None instead.

The “all null / all valid / some” trichotomy

Because null_count is cached, the question “is this array entirely null, entirely valid, or mixed?” is answerable in O(1) from the count and the length. DataFusion leans on this constantly, and encodes the answer as a three-valued enum:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs:1271-1279
///Shows how many `null`s there are in an array
enum Nulls {
 /// All array items are `null`s
 All,
 /// There are both `null`s and non-`null`s in the array items
 Some,
 /// There are no `null`s in the array items
 None,
}

Note the naming collision with Option::None — Nulls::None means no nulls, which is the opposite of Option::None meaning no bitmap. That naming is a small readability trap, but the semantics are consistent: both Nulls::None and Option::None mean “no nulls to worry about”.

Every GroupColumn implementation in the multi-column GROUP BY path computes this trichotomy once and then branches on it for the whole batch. The byte-array builder is the clearest example:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/bytes.rs:149-181
 let null_count = array.null_count();
 let num_rows = array.len();
 let all_null_or_non_null = if null_count == 0 {
 Nulls::None
 } else if null_count == num_rows {
 Nulls::All
 } else {
 Nulls::Some
 };

 match all_null_or_non_null {
 Nulls::Some => {
 for &row in rows {
 self.append_val_inner::<B>(array, row)?
 }
 }

 Nulls::None => {
 self.nulls.append_n_non_nulls(rows.len());
 for &row in rows {
 self.do_append_val_inner(arr, row)?;
 }
 }

 Nulls::All => {
 self.nulls.append_n_nulls(rows.len());

 let new_len = self.offsets.len() + rows.len();
 let offset = self.buffer.len();
 self.offsets.resize(new_len, O::usize_as(offset));
 }
 }

Look at what the three arms cost:

Nulls::Some: per-row, check is_null(row), append to the null builder, append the value.
Nulls::None: the null bitmap is not touched at all for the non-null case. append_n_non_nulls(n) is a length bump. The value loop has no null check in it.
Nulls::All: the values are not copied at all. The offsets buffer is resized with a repeated value and the string buffer is untouched. An all-null column of 10,000 strings costs 10,000 offset writes and zero bytes of string data.

The primitive builder does the same thing but goes further — it uses a const NULLABLE: bool type parameter so the null check is compiled out entirely for non-nullable columns:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/primitive.rs:171-188
 fn append_val(&mut self, array: &ArrayRef, row: usize) -> Result<()> {
 // Perf: skip null check if input can't have nulls
 if NULLABLE {
 if array.is_null(row) {
 self.nulls.append_null();
 self.group_values.push(T::default_value());
 } else {
 self.nulls.append_non_null();
 self.group_values
 .push(array.as_primitive::<T>().value(row).canonicalize());
 }
 } else {
 self.group_values
 .push(array.as_primitive::<T>().value(row).canonicalize());
 }

 Ok(())
 }

if NULLABLE where NULLABLE is a const generic is not a runtime branch — it is a compile-time one. There are two monomorphizations of PrimitiveGroupValueBuilder, and the non-nullable one contains no null handling code at all.

The comparison side has the same structure, with an extra trick: a null-aware comparison can be short-circuited by nullness alone.

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs:121-133
/// Determines if the nullability of the existing and new input array can be used
/// to short-circuit the comparison of the two values.
///
/// Returns `Some(result)` if the result of the comparison can be determined
/// from the nullness of the two values, and `None` if the comparison must be
/// done on the values themselves.
pub fn nulls_equal_to(lhs_null: bool, rhs_null: bool) -> Option<bool> {
 match (lhs_null, rhs_null) {
 (true, true) => Some(true),
 (false, true) | (true, false) => Some(false),
 _ => None,
 }
}

Some(true) = “both null, so equal (SQL GROUP BY treats nulls as one group)”. Some(false) = “one null, one not, so different”. None = “both non-null, fall through to the value comparison”. This is a three-way branch replacing a value comparison in 2 out of 3 cases involving nulls.

The `might_have_nulls` trick: a conservative bitmap

There is a subtle but important pattern in the group-value builders. The null builder is a NullBufferBuilder, and DataFusion wraps it in a trait that exposes a conservative nullness predicate:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/null_builder.rs:40-45
 /// Returns true if this builder might have any nulls
 ///
 /// This is guaranteed to be true if there are nulls
 /// but may be true even if there are no nulls
 fn might_have_nulls(&self) -> bool;

implemented as:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/null_builder.rs:88-90
 fn might_have_nulls(&self) -> bool {
 self.as_slice().is_some()
 }

NullBufferBuilder only materializes its underlying bitmap the first time a null is actually appended (Arrow’s materialize_if_needed). So as_slice() is Some iff a null was ever appended. This means might_have_nulls is true whenever the builder has definitely seen a null, and possibly true when it hasn’t — the fuzziness exists because the builder is reused across batches, and truncating its bitmap to preserve a precise “definitely has nulls” property would cost more than the conservative answer. The predicate is used to decide whether the nullable comparison path is required:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/primitive.rs:190-207
 fn vectorized_equal_to(
 &mut self,
 lhs_rows: &[usize],
 array: &ArrayRef,
 rhs_rows: &[usize],
 equal_to_results: &mut BooleanBufferBuilder,
 ) {
 if !NULLABLE || (array.null_count() == 0 && !self.nulls.might_have_nulls()) {
 self.vectorized_equal_to_non_nullable(
 lhs_rows,
 array,
 rhs_rows,
 equal_to_results,
 );
 } else {
 self.vectorized_equal_nullable(lhs_rows, array, rhs_rows, equal_to_results);
 }
 }

The non-nullable path is the fast one: it reads array_values[rhs_row] directly and packs the result into a bitmask with a word-at-a-time AND:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/primitive.rs:79-116
 let array_values = array.as_primitive::<T>().values();
 let n = lhs_rows.len();

 // Build a packed comparison bitmask, then AND it into equal_to_results
 let num_bytes = n.div_ceil(8);
 let mut cmp_buf = vec![0u8; num_bytes];

 for (i, (&lhs_row, &rhs_row)) in lhs_rows.iter().zip(rhs_rows.iter()).enumerate()
 {
 if !equal_to_results.get_bit(i) {
 continue;
 }
 let left = if cfg!(debug_assertions) {
 self.group_values[lhs_row]
 } else {
 unsafe { *self.group_values.get_unchecked(lhs_row) }
 };
 let right = if cfg!(debug_assertions) {
 array_values[rhs_row]
 } else {
 unsafe { *array_values.get_unchecked(rhs_row) }
 };
 // `left` was already canonicalized on append; canonicalize the
 // input so ±0 (and any future equivalence class) compares equal.
 if left.is_eq(right.canonicalize()) {
 cmp_buf[i / 8] |= 1 << (i % 8);
 }
 }

 // AND the comparison result into the existing equal_to_results bitmask
 apply_bitwise_binary_op(
 equal_to_results.as_slice_mut(),
 0,
 &cmp_buf,
 0,
 n,
 |a, b| a & b,
 );

Note the array_values[rhs_row] read: PrimitiveArray::values() returns the raw value buffer, with no null information in it. Reading a null slot’s value is safe (it holds whatever was in the buffer, usually a default), because the null-ness is tracked separately in the bitmap and checked before the value is ever used. This is the whole point of the separation: the value buffer is dense and unbranched, and null-ness lives somewhere else.

Boolean filters: nulls as false, at the bitmap level

Filter predicates are the other place nulls show up constantly. In SQL, a NULL predicate is not TRUE, so it does not pass a WHERE clause. Arrow expresses this as a bitmap operation:

 rust
 
 
// arrow-select-60.0.0/src/filter.rs:159-171
/// Convert all null values in `BooleanArray` to `false`
///
/// This is useful for filter-like operations which select only `true`
/// values, but not `false` or `NULL` values
///
/// Internally this is implemented as a bitwise `AND` operation with null bits
/// and the boolean bits.
pub fn prep_null_mask_filter(filter: &BooleanArray) -> BooleanArray {
 let nulls = filter.nulls().unwrap();
 let mask = filter.values() & nulls.inner();
 BooleanArray::new(mask, None)
}

filter.values() & nulls.inner() is a word-at-a-time AND of two bitmaps producing a new bitmap with no validity bitmap of its own (the nulls have been folded into the values). This is the entire implementation of SQL’s three-valued logic for filtering: one AND of two bitmaps.

DataFusion uses it in the places where a predicate’s nulls must not pass:

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:449-455
 // only update the keys / rows if the filter does not match all rows
 if filter.null_count() > 0 || filter.has_false() {
 // Indices in `set_indices` should be correct if filter contains nulls
 // So we prepare the filter here. Note this is also done in the `FilterBuilder`
 // so there is no overhead to do this here.
 if filter.nulls().is_some() {
 filter = prep_null_mask_filter(&filter);
 }

and in the AND/OR short-circuit logic, where nulls have to be handled at the bit level rather than the row level:

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:1541-1548
/// A null-free mask of the rows of `array` that are `true` or `NULL`.
fn not_false(array: &BooleanArray) -> BooleanArray {
 let values = match array.nulls() {
 Some(nulls) => array.values() | &!nulls.inner(),
 None => array.values().clone(),
 };
 BooleanArray::new(values, None)
}

array.values() | &!nulls.inner() — “true OR null” — is the set of rows that are not yet decided. Again, no per-row work; two bitmap operations.

And the “all false” test, used to short-circuit a whole AND chain:

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:1515-1524
/// Whether every row is already `false`.
fn is_all_false(value: &ColumnarValue) -> bool {
 match value {
 ColumnarValue::Array(array) => {
 let array = array.as_boolean();
 array.null_count() == 0 && !array.has_true()
 }
 ColumnarValue::Scalar(scalar) => *scalar == ScalarValue::Boolean(Some(false)),
 }
}

null_count() == 0 && !has_true() — a single cached count plus a single scan of the values bitmap — answers “is this whole column false?” If so, the entire AND chain can stop.

`IN` lists: computing nullability as a bitmap expression

The IN list implementation is the most bitmap-heavy code in the physical expression layer, because SQL’s IN has genuinely subtle null semantics. DataFusion computes the “contains” bitmap first, then applies the null mask as pure bitmap algebra:

 rust
 
 
// datafusion/physical-expr/src/expressions/in_list/result.rs:72-110
pub(super) fn build_result_from_contains(
 needle_nulls: Option<&NullBuffer>,
 haystack_has_nulls: bool,
 negated: bool,
 contains_buf: BooleanBuffer,
) -> BooleanArray {
 match (needle_nulls, haystack_has_nulls, negated) {
 // Haystack has nulls: result is null unless value is found.
 (Some(v), true, false) => {
 // values: valid & contains, nulls: valid & contains
 let values = v.inner() & &contains_buf;
 BooleanArray::new(values.clone(), Some(NullBuffer::new(values)))
 }
 (None, true, false) => {
 BooleanArray::new(contains_buf.clone(), Some(NullBuffer::new(contains_buf)))
 }
 (Some(v), true, true) => {
 // NOT IN with nulls: false if found, null if not found or needle null.
 // values: valid & !contains, nulls: valid & contains
 let valid = v.inner();
 let values = valid & &(!&contains_buf);
 let nulls = valid & &contains_buf;
 BooleanArray::new(values, Some(NullBuffer::new(nulls)))
 }
 (None, true, true) => {
 BooleanArray::new(!&contains_buf, Some(NullBuffer::new(contains_buf)))
 }
 // Haystack has no nulls: result validity follows needle validity.
 (Some(v), false, false) => {
 // values: valid & contains, nulls: valid
 BooleanArray::new(v.inner() & &contains_buf, Some(v.clone()))
 }
 (Some(v), false, true) => {
 // values: valid & !contains, nulls: valid
 BooleanArray::new(v.inner() & &(!&contains_buf), Some(v.clone()))
 }
 (None, false, false) => BooleanArray::new(contains_buf, None),
 (None, false, true) => BooleanArray::new(!&contains_buf, None),
 }
}

Read the last two arms: (None, false, false) and (None, false, true) produce a BooleanArray with no validity bitmap. If the needle isn’t null and the haystack has no nulls, the result is never null — so no bitmap is allocated. That is the “all valid ⇒ None” convention applied to a computed result, not just an input.

Filters and selection vectors

arrow::compute::filter is the other half of the null story, because a filter predicate is a BooleanArray — which has its own validity bitmap for the NULL slots — and the output has to compact.

Arrow’s FilterBuilder picks between two strategies depending on how selective the predicate is:

 rust
 
 
// arrow-select-60.0.0/src/filter.rs:346-365
 fn default_strategy(filter_length: usize, filter_count: usize) -> Self {
 if filter_length == 0 || filter_count == 0 {
 return IterationStrategy::None;
 }

 if filter_count == filter_length {
 return IterationStrategy::All;
 }

 // Compute the selectivity of the predicate by dividing the number of true
 // bits in the predicate by the predicate's total length
 //
 // This can then be used as a heuristic for the optimal iteration strategy
 let selectivity_frac = filter_count as f64 / filter_length as f64;
 if selectivity_frac > FILTER_SLICES_SELECTIVITY_THRESHOLD {
 return IterationStrategy::SlicesIterator;
 }
 IterationStrategy::IndexIterator
 }

with the threshold at 0.8:

 rust
 
 
// arrow-select-60.0.0/src/filter.rs:39-43
/// If the filter selects more than this fraction of rows, use
/// [`SlicesIterator`] to copy ranges of values. Otherwise iterate
/// over individual rows using [`IndexIterator`]
///
/// Threshold of 0.8 chosen based on <https://dl.acm.org/doi/abs/10.1145/3465998.3466009>
const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.8;

A selective filter produces a selection vector (a list of surviving row indices); a non-selective one produces a list of slices (contiguous ranges) and copies whole runs. This is a genuinely important distinction: for a 5%-selective filter, gathering 50,000 scattered rows from a million-row column is a random-access disaster; for a 95%-selective filter, copying 19 contiguous runs is nearly free.

DataFusion uses this all over the place — the FilterExec calls filter_record_batch, the TopK operator builds a FilterBuilder and uses filter.values().set_indices() to get the selection vector for its heap, and the range-partition router gathers rows with take_arrays in one shot per batch:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1372-1388
 let batches = {
 let _timer = timer.timer();
 let indices_array: PrimitiveArray<UInt32Type> = reordered_indices.into();
 let columns = take_arrays(batch.columns(), &indices_array, None)?;

 let mut options = RecordBatchOptions::new();
 options = options.with_row_count(Some(indices_array.len()));
 let reordered_batch =
 RecordBatch::try_new_with_options(batch.schema(), columns, &options)?;

 partition_ranges
 .into_iter()
 .map(|(partition, start, len)| {
 Ok((partition, reordered_batch.slice(start, len)))
 })
 .collect()
 };

One take for the whole batch, then each partition is a zero-copy slice of the reordered batch. Note that take does preserve the validity bitmap — the nulls follow the rows — so the null handling is implicit in the gather rather than a separate pass.

Where the physical `Column` fits in

The physical Column expression is the leaf that hands the bitmap-bearing array to everything else. It is deliberately trivial:

 rust
 
 
// datafusion/physical-expr/src/expressions/column.rs:120-124
 /// Evaluate the expression
 fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 self.bounds_check(batch.schema().as_ref())?;
 Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
 }

An Arc clone. The values buffer, the validity bitmap, the offsets — all shared, none copied, none inspected. This is what makes the whole design work: because Column does not look at the nulls, every consumer above it is free to make its own decision about whether nulls matter, and the decision is made once per batch rather than per row.

(A note on file layout: the Arrow helper utilities that older DataFusion versions kept under datafusion/common/src/arrow/ now live in datafusion/common/src/cast.rs — the as_boolean_array, as_string_array, as_dictionary_array family of downcasting helpers the physical expression layer leans on — and the null-handling helpers in datafusion/common/src/utils/mod.rs.)

Propagating a parent’s nulls to its children

Arrow’s nested types have a subtlety: a struct’s child arrays have their own validity bitmaps, and the struct’s own bitmap says which rows are null. A null struct row means all of its children are logically null, but the children’s bitmaps do not know that.

DataFusion has a helper that fixes this up while preserving the value-buffer sharing:

 rust
 
 
// datafusion/common/src/utils/mod.rs:1573-1598
/// Apply a struct's nulls to one of its fields.
///
/// Arrays with validity bitmaps share their value buffers; unions and
/// run-end encoded arrays are rebuilt.
pub fn apply_parent_nulls(
 col: &ArrayRef,
 parent_nulls: Option<&NullBuffer>,
) -> Result<ArrayRef> {
 let Some(parent_nulls) = parent_nulls else {
 // If there are no parent nulls to apply, we can just return
 return Ok(Arc::clone(col));
 };

 // NullArray is already entirely null and cannot have a validity bitmap.
 // If we have 0 parent nulls, we can also avoid extra work.
 if col.data_type().is_null() || parent_nulls.null_count() == 0 {
 return Ok(Arc::clone(col));
 }

 if layout(col.data_type()).can_contain_null_mask {
 // `nullif` marks a row null where the mask is true and keeps the
 // field's own nulls. Only the validity bitmap is rebuilt; the value
 // buffers and child arrays are shared with `col`.
 let null_parents = BooleanArray::new(!parent_nulls.inner(), None);
 return Ok(nullif(col.as_ref(), &null_parents)?);
 }

Two shortcuts again: no parent nulls at all → return the column untouched; parent has zero nulls → also return untouched. Only when there is real work to do does it build the inverted mask (!parent_nulls.inner()) and rebuild the validity bitmap. And because the values are untouched, the operation is cheap even for a wide struct.

The can_contain_null_mask check is the important one: unions and run-end-encoded arrays have no validity bitmap of their own — their nulls live in their children — so those need a full rebuild via MutableArrayData instead. The helper is explicit about that:

 rust
 
 
// datafusion/common/src/utils/mod.rs:1600-1613
 // Unions and run-end encoded arrays have no validity bitmap of their own
 // and represent nulls in their children. Rebuild the array so null parents
 // become null values in those children.
 let data = col.to_data();
 let mut mutable = MutableArrayData::new(vec![&data], true, data.len());
 let mut end = 0;
 for (start, valid_end) in parent_nulls.valid_slices() {
 mutable.try_extend_nulls(start - end)?;
 mutable.try_extend(0, start, valid_end)?;
 end = valid_end;
 }
 mutable.try_extend_nulls(data.len() - end)?;

 Ok(make_array(mutable.freeze()))
}

valid_slices() — the valid runs of the parent bitmap — is what makes this a bulk copy rather than a per-row loop.

The reverse operation: scatter

filter compacts; scatter expands. The two are inverses, and the null handling is where they differ. scatter puts the source values into the positions where the mask is true, and fills the rest with nulls:

 rust
 
 
// datafusion/physical-expr-common/src/utils.rs:76-102
pub fn scatter(mask: &BooleanArray, truthy: &dyn Array) -> Result<ArrayRef> {
 let mask = match mask.null_count() {
 0 => Cow::Borrowed(mask),
 n if n == mask.len() => {
 return Ok(new_null_array(truthy.data_type(), mask.len()));
 }
 _ => Cow::Owned(prep_null_mask_filter(mask)),
 };

 let output_len = mask.len();

 // Fast path: no true values mean all-null object
 if !mask.has_true() {
 return Ok(new_null_array(truthy.data_type(), output_len));
 }

 // Fast path: all true means output = truthy
 if mask.null_count() == 0 && !mask.has_false() {
 return Ok(truthy.slice(0, truthy.len()));
 }

 let count = mask.true_count();
 let selectivity = count as f64 / output_len as f64;
 let mask_buffer = mask.values();

 scatter_array(truthy, mask_buffer, output_len, selectivity)
}

Two fast paths again: “mask is all false” → the answer is a null array (no copying of the source at all); “mask is all true” → the answer is a slice of the source (no copying either, just an offset).

The null mask of the scattered result is computed by a dedicated routine that knows the two interesting cases:

 rust
 
 
// datafusion/physical-expr-common/src/utils.rs:205-225
fn scatter_null_mask(
 src_nulls: Option<&NullBuffer>,
 mask: &BooleanBuffer,
 output_len: usize,
 selectivity: f64,
) -> Option<NullBuffer> {
 let false_count = output_len - mask.count_set_bits();
 let src_null_count = src_nulls.map(|n| n.null_count()).unwrap_or(0);

 if src_null_count == 0 {
 if false_count == 0 {
 None
 } else {
 Some(NullBuffer::new(mask.clone()))
 }
 } else {
 let src_nulls = src_nulls.unwrap();
 let scattered = scatter_bits(src_nulls.inner(), mask, output_len, selectivity);
 Some(NullBuffer::new(scattered)).filter(|n| n.null_count() > 0)
 }
}

Three observations:

src_null_count == 0 && false_count == 0 → None. No nulls in the source and every output slot is filled → no validity bitmap needed.
src_null_count == 0 && false_count > 0 → the output’s validity bitmap is literally the mask. The slots that were not selected are null; the ones that were selected are valid. The mask doubles as the validity bitmap — no separate computation, no allocation beyond a NullBuffer wrapper.
The .filter(|n| n.null_count() > 0) at the end is the “don’t materialize an all-ones bitmap” rule applied to a computed result: if the scattered nulls turned out to be zero, drop the bitmap entirely.

Slicing a bitmap is not free

One more Arrow detail that bites people: slicing a validity bitmap is O(1) in the buffer but O(len) in the null count, because the count has to be recomputed:

 rust
 
 
// arrow-buffer-60.0.0/src/buffer/null.rs:281-288
 /// Slices this [`NullBuffer`] by the provided `offset` and `length`
 ///
 /// # Panics
 ///
 /// Panics if `offset + len > self.len()`
 pub fn slice(&self, offset: usize, len: usize) -> Self {
 Self::new(self.buffer.slice(offset, len))
 }

Self::new recomputes null_count by count_set_bits() over the sliced region. The bitmap is not copied — BooleanBuffer::slice just bumps bit_offset — but the popcount is real work. So slicing a million-row array into a hundred thousand-row window costs a popcount over 12.5 KB, not over 125 KB, but it is not free either. This is why DataFusion’s RowValues and friends prefer to keep a whole array and track an offset where possible.

Where the separation leaks: logical vs physical nulls

The bitmap separation creates one genuinely nasty corner case, and DataFusion has a whole API for it: logical nulls.

For a dictionary array, a row can be physically valid (its key is not null) but logically null (its key points at a NULL entry in the dictionary’s values array). Arrow exposes both:

 rust
 
 
// arrow-array-60.0.0/src/array/dictionary_array.rs:769-787
 fn logical_null_count(&self) -> usize {
 match (self.keys.nulls(), self.values.logical_nulls()) {
 (None, None) => 0,
 (Some(key_nulls), None) => key_nulls.null_count(),
 (None, Some(value_nulls)) => self
 .keys
 .values()
 .iter()
 .filter(|k| value_nulls.is_null(k.as_usize()))
 .count(),
 (Some(key_nulls), Some(value_nulls)) => self
 .keys
 .values()
 .iter()
 .enumerate()
 .filter(|(idx, k)| key_nulls.is_null(*idx) || value_nulls.is_null(k.as_usize()))
 .count(),
 }
 }

DataFusion’s hash join uses the logical count to decide whether the build side has nulls, because a physically-valid key pointing at a null value must be treated as a null key for NOT IN semantics:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/exec.rs:3620-3622
 let build_has_null = null_aware == Some(NullAwareMode::RightAnti)
 && !left_values.is_empty()
 && left_values[0].logical_null_count() > 0;

and it caches the logical nulls separately, because computing them can be expensive for nested dictionaries:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/exec.rs:358-362
 /// Logical nulls of the scalar key for uncorrelated null-aware `LeftMark`.
 ///
 /// Computed once while collecting the build side because dictionary
 /// logical-null discovery can scan nested dictionary values.
 left_mark_build_key_nulls: Option<NullBuffer>,

This is a real failure mode: an operator that uses null_count() where it meant logical_null_count() will treat a null-valued dictionary entry as a present value. There are explicit regression tests for it in the hash join test module (datafusion/physical-plan/src/joins/hash_join/exec.rs:9269+, test_..._logical_null_count style).

Failure modes and trade-offs

Option<NullBuffer> semantics are easy to get backwards. None means “no nulls”, not “all nulls”. Code that does nulls.map(|n| n.null_count()).unwrap_or(0) is correct; code that does nulls.is_none() => all nulls is catastrophically wrong. DataFusion’s Nulls enum exists partly to make the distinction explicit at the type level.
The cached null_count can go stale if you mutate the underlying buffer. NullBuffer is immutable once built, so this is mostly a non-issue, but hand-rolled NullBufferBuilder misuse can produce a count that disagrees with the bits.
might_have_nulls is deliberately fuzzy. Code that treats it as “definitely has nulls” will be correct but slow; code that treats it as “definitely has no nulls” will be wrong. The name is the warning.
The Nulls::All fast path skips the values entirely. Any code that later assumes the group’s value buffer contains the actual bytes will find zeros there. This is why the group-value builders keep the null bitmap and the values buffer strictly consistent — the “value” of a null group is defined as a default/empty value.
Row format encoding of nulls is a sentinel, not a bitmap. The row format writes a 0x00 (or 0xFF) byte for null in each field, so a null row is still a full-length row. That is a space cost compared to the bitmap representation, but it buys the memcmp-comparability described in the row-format article.
Dictionary logical nulls are O(D) to discover. logical_null_count has to scan the keys. For a dictionary with a million entries, that is a million reads, done once and cached — but it is not free, and DataFusion explicitly caches it rather than recomputing per batch.

The takeaway

The value/validity split in Arrow is not just a memory optimization; it is an information optimization. Because null-ness lives in a separate, bit-packed, countable buffer, a whole batch can be classified as all-null / all-valid / mixed in O(1), and the answer selects between radically different code paths: copy the values or don’t, allocate a validity bitmap or don’t, run a null-aware comparison or a raw value comparison, fold nulls into a mask with one AND or handle them row by row. DataFusion’s Nulls enum, its nulls_equal_to short-circuit, its might_have_nulls conservatism, Arrow’s prep_null_mask_filter, and the scatter_null_mask routine that reuses the selection mask as the validity bitmap are all instances of the same idea: decide once, at the batch level, whether nulls are a problem — and then take a completely different code path.

Polars Evaluates Expressions in an Arena

Jay Hubbert — Wed, 21 Oct 2026 06:00:00 +0000

TL;DR: A Polars expression exists in three shapes. The one you write is a tree of Arc<Expr>. The one the optimizer reasons about is an AExpr node living in a flat Arena<AExpr>, addressed by an 8-byte Node index. The one that touches data is an Arc<dyn PhysicalExpr> tree.

The middle shape is the interesting one: because the plan is a vector plus integers, cloning it is copying indices, comparing two subtrees is comparing integers, and nothing in it is individually freed until the whole arena drops.

Every dataframe engine has to answer the same question twice: how do I describe an expression, and how do I run it? The naive answer is to use one structure for both, and to run it by walking the tree.

Polars does not. It has three distinct representations, and the transitions between them are explicit, one-way, and cheap. This post is about the middle one: the arena.

The shape you write: a tree of `Arc<Expr>`

The user-facing expression is an enum whose children are Arc<Expr>:

 rust
 
 
// crates/polars-plan/src/dsl/expr/mod.rs:76
pub enum Expr {
 /// Values in a `eval` context.
 ///
 /// Equivalent of `pl.element()`.
 Element,
 Alias(Arc<Expr>, PlSmallStr),
 Column(PlSmallStr),
 Selector(Selector),
 Literal(LiteralValue),
 DataTypeFunction(DataTypeFunction),
 BinaryExpr {
 left: Arc<Expr>,
 op: Operator,
 right: Arc<Expr>,
 },
 Cast {
 expr: Arc<Expr>,
 dtype: DataTypeExpr,
 options: CastOptions,
 },

This is a perfectly good API shape: it is cheap to clone (cloning an Arc is a refcount bump), it is easy to pattern match on, and it is what the Python bindings, the R bindings, and the Expr builder methods all produce.

It is also a bad shape for a plan. A tree of Arc means:

every node lives in its own heap allocation, scattered across the heap;
“the same expression” is a deep structural comparison, not a pointer comparison;
you cannot hand a sub-expression to another subsystem without either cloning it or keeping the Arc alive, which keeps the whole tree alive.

Polars needs all three capabilities constantly. The optimizer compares sub-expressions to find common subplans. The projection pushdown needs to answer “does this expression mention column a?”. The CSE pass needs to deduplicate identical subtrees. So the DSL tree gets converted, once, into something flat.

The arena

The whole arena is 172 lines in crates/polars-utils/src/arena.rs, and the relevant parts are tiny:

 rust
 
 
// crates/polars-utils/src/arena.rs:26
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Ord, PartialOrd)]
#[repr(transparent)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct Node(pub usize);

...

#[derive(Debug, Clone)]
#[cfg_attr(feature = "ir_serde", derive(Serialize, Deserialize))]
pub struct Arena<T> {
 version: u32,
 items: Vec<T>,
}

and adding a node is a Vec::push plus an index:

 rust
 
 
// crates/polars-utils/src/arena.rs:57
 pub fn add(&mut self, val: T) -> Node {
 let idx = self.items.len();
 self.items.push(val);
 Node(idx)
 }

Reading a node is a bounds-checked index:

 rust
 
 
// crates/polars-utils/src/arena.rs:106
 #[inline]
 pub fn get(&self, idx: Node) -> &T {
 self.items.get(idx.0).unwrap()
 }

That is the entire data structure. A Node is a usize — a Copy, Eq, Hash, Ord index into a Vec. Everything else in the design follows from that.

The IR node: `AExpr`

The expression IR is a single enum whose children are Nodes rather than pointers:

 rust
 
 
// crates/polars-plan/src/plans/aexpr/mod.rs:180
/// IR expression node that is allocated in an [`Arena`][polars_utils::arena::Arena].
#[derive(Clone, Debug, Default)]
#[cfg_attr(feature = "ir_serde", derive(serde::Serialize, serde::Deserialize))]
pub enum AExpr {
 /// Values in a `eval` context.
 ///
 /// Equivalent of `pl.element()`.
 Element,
 Explode {
 expr: Node,
 options: ExplodeOptions,
 },
 Column(PlSmallStr),
 ...
 BinaryExpr {
 left: Node,
 op: Operator,
 right: Node,
 },
 Cast {
 expr: Node,
 dtype: DataType,
 options: CastOptions,
 },

Note what is not here: no Box, no Arc, no dyn. The recursive edges of the expression graph are plain usizes. The graph is stored in the arena; the enum only carries the edges.

The handle: `ExprIR`

A plan does not hold AExprs directly. It holds ExprIRs, which pair a node index with the output name and a memoized dtype:

 rust
 
 
// crates/polars-plan/src/plans/expr_ir.rs:70
#[derive(Clone, Debug)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct ExprIR {
 /// Output name of this expression.
 output_name: OutputName,
 /// Output dtype of this expression
 /// Reduced expression.
 /// This expression is pruned from `alias` and already expanded.
 node: Node,
 #[cfg_attr(feature = "serde", serde(skip))]
 output_dtype: OnceLock<DataType>,
}

ExprIR is Clone, and cloning it copies a Node (8 bytes), an OutputName, and a OnceLock<DataType>. It does not copy the expression subtree, because the subtree lives in the arena and is shared by reference. This is the “cheap cloning of expression trees” property, and it is not an optimization bolted on later — it falls out of the representation.

ExprIR also caches the dtype:

 rust
 
 
// crates/polars-plan/src/plans/expr_ir.rs:291
 pub fn dtype(&self, schema: &Schema, expr_arena: &Arena<AExpr>) -> PolarsResult<&DataType> {
 match self.output_dtype.get() {
 Some(dtype) => Ok(dtype),
 None => {
 let dtype = expr_arena
 .get(self.node)
 .to_dtype(&ToFieldContext::new(expr_arena, schema))?;
 let _ = self.output_dtype.set(dtype);
 Ok(self.output_dtype.get().unwrap())
 },
 }
 }

Since the arena is immutable once built, the dtype of a node is a property of the node, so every ExprIR referring to it agrees. The OnceLock turns a repeated schema query into a pointer read.

From `Expr` to `ExprIR`

The conversion is a recursive walk that bottoms out in a single arena.add:

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/expr_to_ir.rs:38
/// Converts expression to AExpr and adds it to the arena, which uses an arena (Vec) for allocation.
#[recursive]
pub(super) fn to_aexpr_impl(
 expr: Expr,
 ctx: &mut ExprToIRContext,
) -> PolarsResult<(Node, PlSmallStr)> {

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/expr_to_ir.rs:170
 Expr::BinaryExpr { left, op, right } => {
 let (l, output_name) = recurse_arc!(left)?;
 let (r, _) = recurse_arc!(right)?;
 (
 AExpr::BinaryExpr {
 left: l,
 op,
 right: r,
 },
 output_name,
 )
 },

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/expr_to_ir.rs:681
 Ok((ctx.arena.add(v), output_name))
}

The recursion is bottom-up: children are converted first, so by the time the parent is built, its children already have indices. The parent is pushed onto the arena last. The resulting arena is in post-order, which is also the order the physical planner wants to walk.

The whole plan uses two arenas — one for plan nodes, one for expressions:

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/mod.rs:58
pub fn to_alp(
 lp: DslPlan,
 expr_arena: &mut Arena<AExpr>,
 lp_arena: &mut Arena<IR>,
 // Only `SIMPLIFY_EXPR`, `TYPE_COERCION`, `TYPE_CHECK`, and `PREDICATE_PUSHDOWN` are respected.
 opt_flags: &mut OptFlags,
) -> PolarsResult<Node> {

Lowering to physical expressions

Now the interesting part: the arena is walked to build the operators that actually touch data. The walk is a single match on the IR enum — this is where the plan-time dispatch happens:

 rust
 
 
// crates/polars-expr/src/planner.rs:132
#[recursive]
fn create_physical_expr_inner(
 expression: Node,
 expr_arena: &mut Arena<AExpr>,
 schema: &SchemaRef, // Schema of the input.
 state: &mut ExpressionConversionState,
) -> PolarsResult<Arc<dyn PhysicalExpr>> {
 use AExpr::*;

 let aexpr = expr_arena.get(expression);
 match aexpr.clone() {

 rust
 
 
// crates/polars-expr/src/planner.rs:301
 BinaryExpr { left, op, right } => {
 let output_field = expr_arena
 .get(expression)
 .to_field(&ToFieldContext::new(expr_arena, schema))?;
 let is_scalar = is_scalar_ae(expression, expr_arena);
 let lhs = create_physical_expr_inner(left, expr_arena, schema, state)?;
 let rhs = create_physical_expr_inner(right, expr_arena, schema, state)?;
 Ok(Arc::new(phys_expr::BinaryExpr::new(
 lhs,
 op,
 rhs,
 node_to_expr(expression, expr_arena),
 state.local.has_lit,
 state.allow_threading,
 is_scalar,
 output_field,
 )))
 },

So the dynamic dispatch that a naive engine would pay per row is paid per plan node, at planning time. The match is on a #[derive(Clone)] enum, so the compiler can, in principle, turn the whole arm dispatch into a jump table or a set of specialized branches. What survives into the hot loop is Arc<dyn PhysicalExpr> — a vtable call per operator, not per row.

Note the node_to_expr(expression, expr_arena) argument. Every physical expression carries a copy of the original DSL expression, purely so that error messages can say “this failed in col("a") + col("b")”. That is a deliberate, expensive-in-the-plan-but-free-at-runtime trade: the arena makes it possible to reconstruct the DSL tree lazily, only on the error path.

 rust
 
 
// crates/polars-plan/src/plans/conversion/ir_to_dsl.rs:7
/// converts a node from the AExpr arena to Expr
#[recursive]
pub fn node_to_expr(node: Node, expr_arena: &Arena<AExpr>) -> Expr {
 let expr = expr_arena.get(node).clone();

 match expr {
 AExpr::Element => Expr::Element,
 AExpr::Explode { expr, options } => Expr::Explode {
 input: Arc::new(node_to_expr(expr, expr_arena)),
 options,
 },

The dispatch table

Not every IR node becomes a bespoke physical operator. Most function expressions are lowered into a generic ApplyExpr that calls a closure. The mapping from IR function to closure lives in one big match in crates/polars-expr/src/dispatch/mod.rs:

 rust
 
 
// crates/polars-expr/src/dispatch/mod.rs:175
/// `input` and `expr_arena` let a function pick a variant based on its arguments, e.g. a
/// constant argument that can be prepared once.
pub fn function_expr_to_udf(
 func: IRFunctionExpr,
 input: &[ExprIR],
 expr_arena: &Arena<AExpr>,
) -> SpecialEq<Arc<dyn ColumnsUdf>> {
 use IRFunctionExpr as F;
 match func {
 // Namespaces
 #[cfg(feature = "dtype-array")]
 F::ArrayExpr(func) => array::function_expr_to_udf(func),
 F::BinaryExpr(func) => binary::function_expr_to_udf(func),
 #[cfg(feature = "dtype-categorical")]
 F::Categorical(func) => cat::function_expr_to_udf(func),
 ...
 F::Boolean(func) => boolean::function_expr_to_udf(func, input, expr_arena),

and the closures are built with a small macro that fixes the calling convention (first column is the root, the rest are arguments):

 rust
 
 
// crates/polars-expr/src/dispatch/mod.rs:58
/// `Fn(&Series, args)`
#[macro_export]
macro_rules! map {
 ($func:path) => {{
 let f = move |c: &mut [::polars_core::prelude::Column]| {
 let c = &c[0];
 $func(c)
 };

 SpecialEq::new(Arc::new(f))
 }};

That is the whole extension mechanism for user-defined and built-in functions: an IR enum variant, one arm in the dispatch table, one function taking &Column.

Where the `dyn` actually is

It is worth being precise about the dynamic dispatch, because the usual claim (“Polars avoids Box<dyn>”) is only half true and the half that is false matters.

The DSL layer is an enum with Arc<Expr> children — there is no Box<dyn Expr> anywhere in it. The IR layer is an enum with Node children, and it has no Box<dyn> either. The dyn appears exactly twice, and both times it is bounded by the number of plan nodes, never by the number of rows:

Arc<dyn PhysicalExpr>, the operator tree. One vtable call per operator per chunk of data, and the kind of operator is resolved at planning time by the match you saw above.
Arc<dyn ColumnsUdf>, the function layer. One vtable call per function application per chunk.

So the cost of dynamic dispatch in Polars is O(operators) per chunk, not O(rows). Compare that with the design where the expression tree itself is Box<dyn Expr> and is walked per row: there the cost is O(rows × operators). The arena is what makes it possible to have the first shape without paying the second.

The other thing the arena changes is where the type information lives. In a Box<dyn Expr> design, the type of a sub-expression is a vtable lookup at runtime. In the IR, the type of a node is a property of the enum variant, known at compile time; the only thing resolved at runtime is the data type, and even that is memoized in the ExprIR.

What the arena buys, concretely

Cheap equality and hashing. Two ExprIRs that refer to the same arena node are the same expression. The canonicalization map exploits this to assign structural equivalence classes to subtrees:

 rust
 
 
// crates/polars-plan/src/plans/aexpr/canonical.rs:23
slotmap::new_key_type! {
 /// Identifies an `AExpr` up to structural equality: two nodes get the same
 /// `CanonicalExprId` if and only if they represent the same expression.
 pub struct CanonicalExprId;
}

Cheap duplication. Duplicating a subtree is one clone of the item plus one push:

 rust
 
 
// crates/polars-utils/src/arena.rs:144
impl<T: Clone> Arena<T> {
 pub fn duplicate(&mut self, node: Node) -> Node {
 let item = self.items[node.0].clone();
 self.add(item)
 }
}

One allocation strategy. Every expression node in the whole plan lives in one Vec. No per-node allocator calls, no Box, no Arc refcount traffic while the plan is being built and rewritten. When the optimizer pushes expressions around — predicate pushdown rewriting a filter, projection pushdown splitting a select, CSE inserting a cache — it copies Nodes, not trees.

Node lifetime. The trade-off is real and worth stating plainly: an arena never frees an individual node. Arena has clear() and pop(), but there is no “free this node”. A rewritten plan leaks the expressions it replaced until the arena itself is dropped. For a query plan, whose lifetime is one collect() call, that is the right trade. It is the wrong trade for a long-lived structure that mutates one node at a time — which is exactly why Arena is not used for, say, the physical operator tree, where each operator has its own lifetime and is dropped independently.

Caching the arena across calls

Because the arena is the plan, and the plan is expensive to rebuild, a LazyFrame keeps it around:

 rust
 
 
// crates/polars-lazy/src/frame/cached_arenas.rs:9
pub(crate) struct CachedArena {
 lp_arena: Arena<IR>,
 expr_arena: Arena<AExpr>,
}

impl LazyFrame {
 pub fn set_cached_arena(&self, lp_arena: Arena<IR>, expr_arena: Arena<AExpr>) {
 let mut cached = self.cached_arena.lock().unwrap();
 *cached = Some(CachedArena {
 lp_arena,
 expr_arena,
 });
 }

 rust
 
 
// crates/polars-lazy/src/frame/cached_arenas.rs:116
 pub(super) fn get_arenas(&mut self) -> (Arena<IR>, Arena<AExpr>) {
 match self.cached_arena.lock().unwrap().as_mut() {
 Some(arenas) => (arenas.lp_arena.clone(), arenas.expr_arena.clone()),
 None => (Arena::with_capacity(16), Arena::with_capacity(16)),
 }
 }

collect_schema() on a LazyFrame converts the DSL plan to IR once, caches both arenas, and reuses them for every subsequent schema query or collect(). Since Arena is Clone and cloning a Vec<AExpr> is a memcpy of the node vector, “reuse the plan” is nearly free. Note the arena has a version counter, bumped on every clear(), so a stale cached plan can be detected rather than silently reused.

The bottom line

The arena is not a performance micro-optimization; it is a representation decision that makes a class of algorithms possible.

The DSL tree (Arc<Expr>) is the right shape for a user API: cheap to build, cheap to share.
The IR (Arena<AExpr> + Node) is the right shape for a plan: flat, index-addressed, comparable by integer, duplicable by memcpy.
The physical tree (Arc<dyn PhysicalExpr>) is the right shape for execution: one vtable call per operator per chunk, and the dispatch on the kind of operator happens at planning time.

The one thing to remember when reading the code: Node is not a pointer, it is an index. That single fact explains why the plan is cheap to copy, why the optimizer can rewrite it freely, why common subplan elimination works, and why every physical operator can afford to carry a copy of the original expression for error messages.

Polars Shares Columns Instead of Copying Them: Arc, Bytes and Reference Counting

Jay Hubbert — Wed, 21 Oct 2026 04:20:00 +0000

TL;DR: Zero-copy in Polars is not one mechanism but three stacked ones. Series is an Arc<dyn SeriesTrait>, ChunkedArray holds Arc<Field> plus a Vec of chunk handles, and the actual bytes live in Buffer<T>, which carries a hand-rolled AtomicU64 refcount with Arc’s memory ordering. Cloning any of these copies handles, never bytes. The Arrow C data interface rides on the same mechanism: an imported buffer does not own its memory, it owns the C array that owns the memory, and releasing the buffer releases the array.

The word “zero-copy” gets used loosely in dataframe discussions, usually to mean “we avoided one memcpy somewhere”. In Polars it means something much more specific: at no point in the pipeline does the engine allocate a second copy of a column’s bytes in order to hand the column to someone else. What moves around is metadata — pointers, lengths, indices — and occasionally a reference to the same bytes.

There are three layers to get right, and each one has a different tool.

Layer 1: the Series is an `Arc`

The most visible layer is the user-facing one. A Series in Polars is a single field:

 rust
 
pub struct Series(pub Arc<dyn SeriesTrait>);

crates/polars-core/src/series/mod.rs:151

That is the entire definition. Cloning a Series bumps a refcount. Passing a Series to a function that takes it by value does not move the data, it moves the handle. This is the layer the Python user sees: df["a"] hands back a Series that shares the column’s buffers with the DataFrame it came from, and both can be alive at once without either being invalidated.

The next layer down is the chunked array, which is what a Series wraps:

 rust
 
 
pub struct ChunkedArray<T: PolarsDataType> {
 pub(crate) field: Arc<Field>,
 pub(crate) chunks: Vec<ArrayRef>,

 pub(crate) flags: StatisticsFlagsIM,

 length: usize,
 null_count: usize,
 _pd: std::marker::PhantomData<T>,
}

crates/polars-core/src/chunked_array/mod.rs:140

The name (Arc<Field>) is shared between chunks; the chunks themselves are ArrayRefs, and ArrayRef is not even an Arc:

 rust
 
pub type ArrayRef = Box<dyn Array>;

crates/polars-arrow/src/array/mod.rs:789

This looks like it should be a copy, but it is not. Box<dyn Array> is a polymorphic handle: cloning it dispatches to the concrete array’s Clone implementation, and every concrete array in polars-arrow stores its data in Buffer<T>. Cloning a PrimitiveArray<u32> clones a Buffer<u32>, and cloning a Buffer<u32> is a refcount increment. So the Vec<ArrayRef> clone allocates a new vector and a new box per chunk, and copies zero bytes of column data.

The ChunkedArray clone says so explicitly:

 rust
 
 
impl<T: PolarsDataType> Clone for ChunkedArray<T> {
 fn clone(&self) -> Self {
 ChunkedArray {
 field: self.field.clone(),
 chunks: self.chunks.clone(),
 flags: self.flags.clone(),

 _pd: Default::default(),
 length: self.length,
 null_count: self.null_count,
 }
 }
}

crates/polars-core/src/chunked_array/mod.rs:1051

The field is an Arc, so cloning it is a refcount bump; the chunks clone is the Vec plus the per-chunk Box<dyn Array> clones, which as established are refcount bumps one level down.

Layer 2: the bytes, and who owns them

The bytes live in Buffer<T>, and the ownership question is answered by SharedStorage. The full enum, from crates/polars-buffer/src/storage.rs:40:

 rust
 
 
enum BackingStorage {
 Vec {
 original_capacity: usize, // Elements, not bytes.
 vtable: &'static VecVTable,
 },
 ForeignOwner(Box<dyn Any + Send + 'static>),

 /// Backed by some external method which we do not need to take care of,
 /// but we still should refcount and drop the SharedStorageInner.
 External,

 /// Both the backing storage and the SharedStorageInner are leaked, no
 /// refcounting is done. This technically should be a flag on
 /// SharedStorageInner instead of being here, but that would add 8 more
 /// bytes to SharedStorageInner, so here it is.
 Leaked,
}

crates/polars-buffer/src/storage.rs:38

This enum is the answer to “how does polars-arrow distinguish owned from borrowed data?”:

Vec — Polars allocated these bytes with a Vec and will free them.
ForeignOwner — the bytes are owned by somebody else, and Polars keeps the owner alive in a Box<dyn Any> purely so it can be dropped later.
External — the bytes are borrowed from a slice; Polars only manages lifetime, never frees.
Leaked — nobody owns them and nobody frees them.

The ForeignOwner variant is what makes the Arrow FFI work (§4 below), and it is also what Buffer::from_owner uses:

 rust
 
 
 /// Creates a [`Buffer`] backed by `owner`.
 pub fn from_owner<O: Send + AsRef<[T]> + 'static>(owner: O) -> Self {
 Self::from_storage(SharedStorage::from_owner(owner))
 }

crates/polars-buffer/src/buffer.rs:122

A Buffer can therefore be about memory that Polars did not allocate, and the refcount still works, because the refcount is on Polars’ bookkeeping object, not on the bytes.

Layer 3: the refcount itself

Polars does not use std::sync::Arc for buffers, and the reason is written into the code — it needs a vtable and a leaked mode that Arc does not have. But the atomic protocol is Arc’s, down to the ordering:

 rust
 
 
impl<T> Clone for SharedStorage<T> {
 fn clone(&self) -> Self {
 let inner = self.inner();
 if !matches!(inner.backing, BackingStorage::Leaked) {
 // Ordering semantics copied from Arc<T>.
 inner.ref_count.fetch_add(1, Ordering::Relaxed);
 }
 Self {
 inner: self.inner,
 phantom: PhantomData,
 }
 }
}

impl<T> Drop for SharedStorage<T> {
 fn drop(&mut self) {
 let inner = self.inner();
 if matches!(inner.backing, BackingStorage::Leaked) {
 return;
 }

 // Ordering semantics copied from Arc<T>.
 if inner.ref_count.fetch_sub(1, Ordering::Release) == 1 {
 std::sync::atomic::fence(Ordering::Acquire);
 unsafe {
 self.drop_slow();
 }
 }
 }
}

crates/polars-buffer/src/storage.rs:491 and crates/polars-buffer/src/storage.rs:483

The Relaxed increment is sound because a clone cannot publish a write that another thread has not yet seen; the Release decrement plus the Acquire fence is what makes the final owner’s destruction of the data safe against concurrent readers.

Because the refcount lives in a heap allocation next to the data (SharedStorageInner), the handle — a Buffer<T> — is three words wide and trivially copyable, and a SharedStorage<T> is a single pointer.

Slicing: the payoff

The reason all of this exists is slicing. Buffer slicing is a pointer bump:

 rust
 
 
 /// Slices this buffer to the given range.
 ///
 /// # Panics
 /// Panics iff the range is out of bounds.
 #[inline]
 pub fn slice_in_place<R: RangeBounds<usize>>(&mut self, range: R) {
 unsafe {
 let Range { start, end } = check_range(range, ..self.len());
 self.ptr = self.ptr.add(start);
 self.length = end - start;
 }
 }

crates/polars-buffer/src/buffer.rs:218

So df.head(5), df["a"].slice(.., 0, 5), a filter that keeps a prefix, or a lazy plan that pushes a slice predicate down to the scan — none of them copy data. They produce a new handle onto the same bytes, and the refcount keeps those bytes alive for exactly as long as somebody needs them. The bytes of a sliced-out region stay allocated until the last handle referencing the storage dies, which is the classic cost of this design: a small slice of a large column pins the whole allocation.

The FFI: importing and exporting without copying

This is where the design pays off most visibly, because the Arrow C data interface is exactly a zero-copy contract: the producer hands out pointers into its own memory, and the consumer must not free them.

Export

Exporting a Polars array to C is a matter of pointing at the buffers that already exist. The Rust array itself is parked in the private_data field so that it outlives the C struct:

 rust
 
 
#[allow(dead_code)]
struct PrivateData {
 array: Box<dyn Array>,
 buffers_ptr: Box<[*const std::os::raw::c_void]>,
 children_ptr: Box<[*mut ArrowArray]>,
 dictionary_ptr: Option<*mut ArrowArray>,
 variadic_buffer_sizes: Box<[i64]>,
}

crates/polars-arrow/src/ffi/array.rs:91

and the ArrowArray struct is filled in with those very pointers:

 rust
 
 
 let buffers_ptr = buffers
 .iter()
 .map(|maybe_buffer| match maybe_buffer {
 Some(b) => *b as *const std::os::raw::c_void,
 None => std::ptr::null(),
 })
 .collect::<Box<[_]>>();
 let n_buffers = buffers.len() as i64;

crates/polars-arrow/src/ffi/array.rs:135

The buffers come straight out of the array:

 rust
 
 
impl<T: ViewType + ?Sized> ToFfi for BinaryViewArrayGeneric<T> {
 fn buffers(&self) -> Vec<Option<*const u8>> {
 let mut buffers = Vec::with_capacity(self.buffers.len() + 2);
 buffers.push(self.validity.as_ref().map(|x| x.as_ptr()));
 buffers.push(Some(self.views.storage_ptr().cast::<u8>()));
 // Export at the offset_adjusted pointer, not the allocation start. Arises whenever
 // `binary_to_binview` sees an inlined value before the first non-inlined one. See #28612.
 buffers.extend(self.buffers.iter().map(|b| Some(b.as_ptr())));
 buffers
 }

crates/polars-arrow/src/array/binview/ffi.rs:10

No bytes are copied on export; the consumer gets a pointer to the same memory Polars is using. The lifetime is handled by the release callback, which frees the Rust array when the consumer is done:

 rust
 
 
// callback used to drop [ArrowArray] when it is exported
unsafe extern "C" fn c_release_array(array: *mut ArrowArray) {
 if array.is_null() {
 return;
 }
 let array = &mut *array;

 // take ownership of `private_data`, therefore dropping it
 let private = Box::from_raw(array.private_data as *mut PrivateData);
 for child in private.children_ptr.iter() {
 let _ = Box::from_raw(*child);
 }

 if let Some(ptr) = private.dictionary_ptr {
 let _ = Box::from_raw(ptr);
 }

 array.release = None;
}

crates/polars-arrow/src/ffi/array.rs:76

private_data holds the Box<dyn Array>, so dropping PrivateData drops the array, which drops its buffers, which decrements the refcounts. The consumer of the C array never has to know any of this — it just calls release.

Import

Importing is the interesting direction, because the memory belongs to somebody else. Polars does not copy it; it borrows it and keeps the C array alive as the owner:

 rust
 
 
unsafe fn create_buffer_known_len<T: NativeType>(
 array: &ArrowArray,
 dtype: &ArrowDataType,
 owner: InternalArrowArray,
 len: usize,
 index: usize,
) -> PolarsResult<Buffer<T>> {
 if len == 0 {
 // Zero-length arrays might have invalid pointers for zero-length slices in Rust,
 // so this is more than just an optimization.
 return Ok(Buffer::new());
 }
 let ptr: *mut T = get_buffer_ptr(array, dtype, index)?;
 let slice = core::slice::from_raw_parts(ptr, len);
 let storage = SharedStorage::from_slice_with_owner(slice, owner);
 Ok(Buffer::from_storage(storage))
}

crates/polars-arrow/src/ffi/array.rs:269

SharedStorage::from_slice_with_owner is the ForeignOwner path: the bytes are not Polars’, but Polars holds an InternalArrowArray (which is an Arc<ArrowArray>) as the owner:

 rust
 
 
 /// # Safety
 /// The slice must be valid as long as owner lives.
 pub unsafe fn from_slice_with_owner<O: Send + 'static>(slice: &[T], owner: O) -> Self {
 #[expect(clippy::manual_slice_size_calculation)]
 let length_in_bytes = slice.len() * size_of::<T>();
 let ptr = slice.as_ptr().cast_mut();
 let inner = SharedStorageInner {
 ref_count: AtomicU64::new(1),
 ptr,
 length_in_bytes,
 backing: BackingStorage::ForeignOwner(owner),
 phantom: PhantomData,
 };
 Self {
 inner: NonNull::new(Box::into_raw(Box::new(inner))).unwrap(),
 phantom: PhantomData,
 }
 }

crates/polars-buffer/src/storage.rs:216

So the ownership chain is: the Polars Buffer holds a SharedStorageInner which holds the InternalArrowArray which holds an Arc<ArrowArray> which owns the C-side memory. When the last Polars handle dies, the inner is dropped, the Arc<ArrowArray> refcount hits zero, and the C array’s release callback runs — freeing the foreign memory exactly once.

There is exactly one case where the import does copy, and it is a genuine correctness requirement rather than a shortcut:

 rust
 
 
 // We have to check alignment, for zero-copy to be valid.
 if ptr.is_aligned() {
 let slice = core::slice::from_raw_parts(ptr.add(offset), len);
 let storage = SharedStorage::from_slice_with_owner(slice, owner);
 Ok(Buffer::from_storage(storage))
 } else {
 // Byte-wise copy for misaligned buffers.
 let mut v = Vec::with_capacity(len);
 core::ptr::copy_nonoverlapping(
 ptr.add(offset).cast::<u8>(),
 v.spare_capacity_mut().as_mut_ptr().cast::<u8>(),
 len * size_of::<T>(),
 );
 v.set_len(len);
 Ok(Buffer::from(v))
 }

crates/polars-arrow/src/ffi/array.rs:269

A misaligned buffer cannot be turned into a &[T] soundly, so Polars copies it into a fresh Vec and takes ownership of that instead. This is the “owned vs borrowed” distinction in action, decided per buffer at import time.

The FFI entry points are thin wrappers:

 rust
 
 
/// Exports an [`Box<dyn Array>`] to the C data interface.
pub fn export_array_to_c(array: Box<dyn Array>) -> ArrowArray {
 ArrowArray::new(bridge::align_to_c_data_interface(array))
}

crates/polars-arrow/src/ffi/mod.rs:22

 rust
 
 
/// Imports an [`Array`] from the C data interface.
///
/// # Safety
/// This function is intrinsically `unsafe` and relies on a [`ArrowArray`]
/// being valid according to the [C data interface](https://arrow.apache.org/docs/format/CDataInterface.html) (FFI).
pub unsafe fn import_array_from_c(
 array: ArrowArray,
 dtype: ArrowDataType,
) -> PolarsResult<Box<dyn Array>> {
 try_from(InternalArrowArray::new(array, dtype))
}

crates/polars-arrow/src/ffi/mod.rs:39

import_array_from_c consumes the ArrowArray — it is moved into an Arc, so the release callback will fire exactly once, when the last Polars-side reference goes away. Ownership transfer, not copying.

What copies, and what does not

Now the honest accounting.

Never copies the bytes:

Cloning a Series (Arc bump), a ChunkedArray (handle clone), or an array (Buffer clone).
Slicing a buffer, a column, or a frame.
Exporting to the Arrow C interface.
Importing from the Arrow C interface (when the buffer is properly aligned).

Copies, on purpose:

Buffer::to_vec, the explicit escape hatch:

 rust
 
 
impl<T: Clone> Buffer<T> {
 pub fn to_vec(self) -> Vec<T> {
 match self.into_mut() {
 Either::Right(v) => v,
 Either::Left(same) => same.as_slice().to_vec(),
 }
 }
}

crates/polars-buffer/src/buffer.rs:357

Misaligned FFI import (above).
Any mutation of shared data, which has to break the sharing first.

Un-sharing on purpose: `deshare` and `gc`

Sharing is not free forever. If a column is sliced down to a fraction of its original size, the shared storage keeps the whole original allocation alive. The string array has an explicit escape hatch for that:

 rust
 
 
 /// Garbage collect
 pub fn gc(self) -> Self {
 if self.buffers.is_empty() {
 return self;
 }
 let mut mutable = MutableBinaryViewArray::with_capacity(self.len());
 let buffers = self.buffers.as_ref();

 for view in self.views.as_ref() {
 unsafe { mutable.push_view_unchecked(*view, buffers) }
 }
 mutable.freeze().with_validity(self.validity)
 }

 pub fn deshare(&self) -> Self {
 if self.buffers.storage_refcount() == 1
 && self.buffers.iter().all(|b| b.storage_refcount() == 1)
 {
 return self.clone();
 }
 self.clone().gc()
 }

crates/polars-arrow/src/array/binview/mod.rs:558

deshare checks the refcounts — the same storage_refcount exposed by Buffer:

 rust
 
 
 /// Since this takes a shared reference to self, beware that others might
 /// increment this after you've checked it's equal to 1.
 pub fn storage_refcount(&self) -> u64 {
 self.storage.refcount()
 }

crates/polars-buffer/src/buffer.rs:324

— and if anyone else is holding the data, it rebuilds the array, copying only the bytes that are still live. maybe_gc is the automatic version, with a 16 KiB minimum saving and a heuristic that avoids copying when the live fraction is too small to be worth it:

 rust
 
 
 pub fn maybe_gc(self) -> Self {
 const GC_MINIMUM_SAVINGS: usize = 16 * 1024; // At least 16 KiB.

 if self.total_buffer_len <= GC_MINIMUM_SAVINGS {
 return self;
 }

 if self.buffers.storage_refcount() != 1 {
 // There are multiple holders of this `buffers`.
 // If we allow gc in this case,
 // it may end up copying the same content multiple times.
 return self;
 }

crates/polars-arrow/src/array/binview/mod.rs:581

Note the second guard: garbage collecting while somebody else holds a reference would risk copying the same content more than once, so Polars skips it.

The bottom line

Sharing is the default. Series is an Arc, ChunkedArray holds Arc<Field> and chunk handles, and the bytes sit in a Buffer with an AtomicU64 refcount that follows Arc’s ordering exactly.
Slicing is free; it is a pointer bump. The cost is that a small slice pins the whole allocation, and Polars has to be told (or to guess) when to garbage collect.
Ownership is explicit. BackingStorage distinguishes Vec (ours), ForeignOwner (somebody else’s, we keep the owner alive), External (borrowed) and Leaked (nobody’s). The FFI import path uses ForeignOwner, so an imported Arrow array is freed exactly once, by whoever drops the last Polars-side handle.
Copies happen only when they must: an explicit to_vec, a misaligned FFI buffer, or an un-sharing (deshare/gc) that trades memory for independence.

The practical takeaway for someone writing Polars code: prefer sliced, clone and Arc over manual copying, and reach for deshare or maybe_gc when you know a column has been narrowed and the shared allocation is now mostly dead weight. If you need to move data across a process boundary, use the Arrow C data interface — it is the one place in the stack where the ownership rules are already worked out, and it moves pointers, not payloads.

Polars Stores Short Strings Inline in 16 Bytes: The View Type

Jay Hubbert — Wed, 21 Oct 2026 04:10:00 +0000

TL;DR: Every string in a Polars string column is a 16-byte View. The first 4 bytes are the length, and the length is the tag: length <= 12 means the string’s bytes are stored in the remaining 12 bytes of the view itself, length > 12 means the view holds a 4-byte prefix plus a (buffer_idx, offset) pair that points into one of the column’s shared byte buffers. There is no separate tag byte, no pointer per string and no Arc<[u8]> per string — the spill target is a refcounted Buffer<u8> shared by the whole column.

Arrow’s Utf8 layout is a pair of arrays: an i32 offset buffer and one contiguous bytes buffer. It works, but it has two costs that matter at scale. The offsets cost 4 bytes per row, and every string access is two dependent loads — read the offset, then read the bytes — which serializes the memory system even when the CPU has plenty of bandwidth left.

Polars does not use Utf8 for string columns. It uses Utf8View, and the whole design of that type is about making the common case (short strings) free.

The type

 rust
 
 
// We use this instead of u128 because we want alignment of <= 8 bytes.
/// A reference to a set of bytes.
///
/// If `length <= 12`, these bytes are inlined over the `prefix`, `buffer_idx` and `offset` fields.
/// If `length > 12`, these fields specify a slice of a buffer.
#[derive(Copy, Clone, Default)]
#[repr(C)]
pub struct View {
 /// The length of the string/bytes.
 pub length: u32,
 /// First 4 bytes of string/bytes data.
 pub prefix: u32,
 /// The buffer index.
 pub buffer_idx: u32,
 /// The offset into the buffer.
 pub offset: u32,
}

crates/polars-arrow/src/array/binview/view.rs:26

Sixteen bytes, four u32 fields, repr(C) so the layout is guaranteed. The inline threshold is right there:

 rust
 
 
impl View {
 pub const MAX_INLINE_SIZE: u32 = 12;

 #[inline(always)]
 pub fn is_inline(&self) -> bool {
 self.length <= Self::MAX_INLINE_SIZE
 }

crates/polars-arrow/src/array/binview/view.rs:55

Twelve, not sixteen, because four bytes are spent on the length. The remaining twelve bytes of the view are the inline payload.

The exact bit layout

On a little-endian machine (and the code assumes little-endian in several places), the 16 bytes of a View mean two different things depending on the length:

text

INLINE (length <= 12)
 byte 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
 +---------------+-------------------------------+
 | length (u32) | string bytes (up to 12) |
 +---------------+-------------------------------+
 ^ zero padded to byte 15

SPILLED (length > 12)
 byte 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
 +---------------+---------------+---------------+---------------+
 | length (u32) | prefix (u32) | buffer_idx | offset (u32) |
 +---------------+---------------+---------------+---------------+

The prefix field is doing double duty: for an inline view it is the first four bytes of the string; for a spilled view it is a copy of the first four bytes of the string, kept next to the metadata so that comparisons and hashes can look at it without chasing the buffer index. The full data of a spilled string lives in buffers[buffer_idx][offset .. offset + length].

Note what is not there: no tag byte, no pointer, no length-prefixed encoding inside the inline payload. The length field is the tag. The branch that distinguishes the two representations is a single compare against a compile-time constant, and it is the same branch everywhere in the codebase.

Building an inline view

The trick that makes this work is that the payload is written over the metadata fields, using raw pointer arithmetic on the view itself:

 rust
 
 
 /// Create a new inline view without verifying the length
 ///
 /// # Safety
 ///
 /// It needs to hold that `bytes.len() <= View::MAX_INLINE_SIZE`.
 #[inline]
 pub unsafe fn new_inline_unchecked(bytes: &[u8]) -> Self {
 debug_assert!(bytes.len() <= u32::MAX as usize);
 debug_assert!(bytes.len() <= Self::MAX_INLINE_SIZE as usize);

 let mut view = Self {
 length: bytes.len() as u32,
 ..Default::default()
 };

 let view_ptr = &mut view as *mut _ as *mut u8;

 // SAFETY:
 // - bytes length <= 12,
 // - size_of::<View> == 16
 // - View is laid out as [length, prefix, buffer_idx, offset] (using repr(C))
 // - By grabbing the view_ptr and adding 4, we have provenance over prefix, buffer_idx and
 // offset. (i.e. the same could not be achieved with &mut self.prefix as *mut _ as *mut u8)
 unsafe {
 let inline_data_ptr = view_ptr.add(4);
 core::ptr::copy_nonoverlapping(bytes.as_ptr(), inline_data_ptr, bytes.len());
 }
 view
 }

crates/polars-arrow/src/array/binview/view.rs:92

The safety comment is worth reading twice, because it explains a subtlety of Rust’s aliasing rules: you cannot get a &mut u8 to the prefix field and then treat it as a *mut u8 with 12 bytes of writable memory behind it — prefix is only 4 bytes, and the compiler is right to complain. Going through the whole view’s pointer and offsetting by 4 gives you provenance over the entire 12-byte tail, which is exactly the region being written.

new_inline_from_block is the branch-free variant, used when you already have a 16-byte block and want a view of its first len bytes:

 rust
 
 
 /// Create an inline view from the first `len` bytes of `block` without branching on `len`.
 ///
 /// # Panics
 ///
 /// Panics if `len > View::MAX_INLINE_SIZE`.
 #[inline(always)]
 pub fn new_inline_from_block(block: [u8; 16], len: usize) -> Self {
 // Masks that keep the first `len` inline bytes and zero the rest.
 const MASKS: [[u8; 16]; View::MAX_INLINE_SIZE as usize + 1] = {
 let mut masks = [[0u8; 16]; View::MAX_INLINE_SIZE as usize + 1];
 let mut len = 0;
 while len <= View::MAX_INLINE_SIZE as usize {
 let mut i = 0;
 while i < len {
 masks[len][4 + i] = 0xFF;
 i += 1;
 }
 len += 1;
 }
 masks
 };

 let mask = MASKS[len];
 let mut raw = [0u8; 16];
 raw[4..].copy_from_slice(&block[..12]);
 for i in 0..16 {
 raw[i] &= mask[i];
 }
 // The length field is little-endian, like the rest of the view.
 raw[..4].copy_from_slice(&(len as u32).to_le_bytes());
 // SAFETY: `View` is `repr(C)` with the length first, then 12 inline bytes.
 unsafe { std::mem::transmute::<[u8; 16], View>(raw) }
 }

crates/polars-arrow/src/array/binview/view.rs:114

The mask table is a compile-time constant, so “zero the padding” becomes an AND with a 16-byte constant — no branch, and the compiler can vectorize the whole thing.

Spilling: the prefix and the buffer index

The spilled representation is created in one place:

 rust
 
 
 /// Create a new inline view
 ///
 /// # Safety
 ///
 /// It needs to hold that `bytes.len() > View::MAX_INLINE_SIZE`.
 #[inline]
 pub unsafe fn new_noninline_unchecked(bytes: &[u8], buffer_idx: u32, offset: u32) -> Self {
 debug_assert!(bytes.len() <= u32::MAX as usize);
 debug_assert!(bytes.len() > View::MAX_INLINE_SIZE as usize);

 // SAFETY: The invariant of this function guarantees that this is safe.
 let prefix = unsafe { u32::from_le_bytes(bytes[0..4].try_into().unwrap_unchecked()) };
 Self {
 length: bytes.len() as u32,
 prefix,
 buffer_idx,
 offset,
 }
 }

crates/polars-arrow/src/array/binview/view.rs:148

and the public constructor is just a branch between the two:

 rust
 
 
 #[inline]
 pub fn new_from_bytes(bytes: &[u8], buffer_idx: u32, offset: u32) -> Self {
 assert!(bytes.len() <= u32::MAX as usize);

 // SAFETY: We verify the invariant with the outer if statement
 unsafe {
 if bytes.len() <= Self::MAX_INLINE_SIZE as usize {
 Self::new_inline_unchecked(bytes)
 } else {
 Self::new_noninline_unchecked(bytes, buffer_idx, offset)
 }
 }
 }

crates/polars-arrow/src/array/binview/view.rs:163

The prefix is not decorative: the validation code uses it to detect corrupted or hand-crafted views, because a view whose prefix disagrees with the bytes in the buffer is a bug or an attack:

 rust
 
 polars_ensure!(b.starts_with(&view.prefix.to_le_bytes()), ComputeError: "prefix does not match string data");

crates/polars-arrow/src/array/binview/view.rs:555

Where the spilled bytes live

They do not live in an Arc<[u8]> per string. The column keeps a list of byte buffers, and the view indexes into that list:

 rust
 
 
pub struct BinaryViewArrayGeneric<T: ViewType + ?Sized> {
 dtype: ArrowDataType,
 views: Buffer<View>,
 buffers: Buffer<Buffer<u8>>,
 validity: Option<Bitmap>,
 phantom: PhantomData<T>,
 /// Total bytes length if we would concatenate them all.
 total_bytes_len: RelaxedCell<u64>,
 /// Total bytes in the buffer (excluding remaining capacity)
 total_buffer_len: usize,
}

crates/polars-arrow/src/array/binview/mod.rs:133

Buffer<Buffer<u8>> is a refcounted buffer whose elements are themselves refcounted buffers. So the layout of a string column is:

text

+---------------------------+ +-------------------------------+
| views: Buffer<View> | | buffers: Buffer<Buffer<u8>> |
| | | |
| [len][abc..][len][xyz..] |-->| [buffer 0: 64 KiB of bytes] |
| ... 16 bytes per row | | [buffer 1: 64 KiB of bytes] |
+---------------------------+ +-------------------------------+

The buffers are not one giant buffer: they are capped chunks, because a single contiguous buffer would have to be reallocated (and therefore copied) as the column grows. The chunking parameters are explicit:

 rust
 
 
// Growth parameters of view array buffers.
pub const BINVIEW_ARROW_BUFFER_LEN_LIMIT: usize = i32::MAX as usize;
pub const BINVIEW_MAX_ROW_BYTE_LEN: usize = (u32::MAX - 1) as usize;
const DEFAULT_BLOCK_SIZE: usize = 8 * 1024;
const MAX_EXP_BLOCK_SIZE: usize = 16 * 1024 * 1024;

crates/polars-arrow/src/array/binview/mod.rs:55

i32::MAX is not arbitrary: the Arrow C data interface describes buffer lengths as i64 but the view’s offset is a u32 and the buffer length limit keeps everything within 2 GiB per buffer, so the offsets stay representable.

How the buffers grow

The mutable builder owns the buffer under construction and doubles it, with the clamp that keeps small columns small and huge columns bounded:

 rust
 
 
 // We want to make sure that we never have to memcopy between buffers. So if the
 // current buffer is not large enough, create a new buffer that is large enough and try
 // to anticipate the larger size.
 if self.in_progress_buffer.len().saturating_add(bytes.len())
 > usize::min(
 BINVIEW_ARROW_BUFFER_LEN_LIMIT,
 self.in_progress_buffer.capacity(),
 )
 {
 const {
 assert!(MAX_EXP_BLOCK_SIZE < BINVIEW_ARROW_BUFFER_LEN_LIMIT);
 }

 // Allocate a new buffer and flush the old buffer.
 let new_capacity = usize::max(
 bytes.len(),
 (self.in_progress_buffer.capacity() * 2)
 .clamp(DEFAULT_BLOCK_SIZE, MAX_EXP_BLOCK_SIZE),
 );

 let in_progress = Vec::with_capacity(new_capacity);
 let flushed = std::mem::replace(&mut self.in_progress_buffer, in_progress);
 if !flushed.is_empty() {
 self.completed_buffers.push(flushed.into())
 }
 }

crates/polars-arrow/src/array/binview/mutable.rs:329

The comment is the design goal in one sentence: never have to memcopy between buffers. Once a buffer is full it is flushed and never touched again, so every spilled string’s bytes are written exactly once and stay where they are. That is what makes offset stable for the lifetime of the column.

There is a second, older copy of this logic in View::new_with_buffers, which is used when building views against a raw Vec<Vec<u8>>:

 rust
 
 
 /// # Safety
 ///
 /// It needs to hold that `bytes.len() > View::MAX_INLINE_SIZE`.
 #[cold]
 unsafe fn new_with_buffers_slow(
 bytes: &[u8],
 buffer_idx_offset: u32,
 buffers: &mut Vec<Vec<u8>>,
 ) -> Self {
 // Resize last buffer instead of making new buffer.
 const RESIZE_LIMIT: usize = 8192;

 if buffers.is_empty() {
 buffers.push(bytes.to_vec());
 return View::new_noninline_unchecked(bytes, buffer_idx_offset, 0);
 }

 let num_buffers = buffers.len();
 let old_cap = buffers.last().unwrap().capacity();
 let new_cap = (old_cap + old_cap).clamp(bytes.len(), u32::MAX as usize);
 if new_cap <= RESIZE_LIMIT {
 let buffer_idx = buffer_idx_offset + num_buffers as u32 - 1;
 let buf = buffers.last_mut().unwrap();
 let offset = buf.len() as u32;
 buf.reserve(new_cap - buf.len());
 buf.extend_from_slice(bytes);
 View::new_noninline_unchecked(bytes, buffer_idx, offset)
 } else {
 let buffer_idx = buffer_idx_offset + num_buffers as u32;
 let mut buf = Vec::with_capacity(new_cap);
 buf.extend_from_slice(bytes);
 buffers.push(buf);
 View::new_noninline_unchecked(bytes, buffer_idx, 0)
 }
 }

crates/polars-arrow/src/array/binview/view.rs:221

Same doubling, but with a RESIZE_LIMIT of 8 KiB deciding between growing the last buffer and starting a new one. The fast path above it — new_with_buffers — checks whether the last buffer still has room and appends in place.

Reading a string back

The read side is the mirror image, and it is where the “no pointer chase for short strings” payoff shows up:

 rust
 
 
 /// Constructs a byteslice from this view.
 ///
 /// # Safety
 /// Assumes that this view is valid for the given buffers.
 #[inline]
 pub unsafe fn get_slice_unchecked<'a, B: AsRef<[u8]>>(&'a self, buffers: &'a [B]) -> &'a [u8] {
 unsafe {
 if self.length <= Self::MAX_INLINE_SIZE {
 self.get_inlined_slice_unchecked()
 } else {
 self.get_external_slice_unchecked(buffers)
 }
 }
 }

crates/polars-arrow/src/array/binview/view.rs:243

and the inline case is a slice of the view’s own memory:

 rust
 
 
 /// Construct a byte slice from an inline view.
 ///
 /// # Safety
 /// Assumes that this view is inlinable.
 #[inline]
 pub unsafe fn get_inlined_slice_unchecked(&self) -> &[u8] {
 debug_assert!(self.length <= View::MAX_INLINE_SIZE);
 let ptr = self as *const View as *const u8;
 unsafe { std::slice::from_raw_parts(ptr.add(4), self.length as usize) }
 }

crates/polars-arrow/src/array/binview/view.rs:291

The spilled case is an index into the buffer list plus a slice:

 rust
 
 
 /// Construct a byte slice from an external view.
 ///
 /// # Safety
 /// Assumes that this view is in the external buffers.
 #[inline]
 pub unsafe fn get_external_slice_unchecked<'a, B: AsRef<[u8]>>(
 &self,
 buffers: &'a [B],
 ) -> &'a [u8] {
 debug_assert!(self.length > View::MAX_INLINE_SIZE);
 let data = buffers.get_unchecked(self.buffer_idx as usize);
 let offset = self.offset as usize;
 data.as_ref()
 .get_unchecked(offset..offset + self.length as usize)
 }

crates/polars-arrow/src/array/binview/view.rs:299

For an inline string, reading the bytes is a load from the same cache line that holds the length. For a spilled string, it is two dependent loads (buffers[buffer_idx], then the bytes) plus the data itself. That is the trade the whole type exists to make.

Comparing and hashing

Because a View is Pod and 16 bytes, it can be compared as a single u128:

 rust
 
 
impl PartialEq for View {
 #[inline(always)]
 fn eq(&self, other: &Self) -> bool {
 self.as_u128() == other.as_u128()
 }
}

crates/polars-arrow/src/array/binview/view.rs:456

For inline views this is exact, because the padding bytes are guaranteed zero — the constructor masks them, and the validator rejects a view whose padding is non-zero:

 rust
 
 
 if let Some(inline_slice) = view.get_inlined_slice() {
 if view.length < View::MAX_INLINE_SIZE && view.as_u128() >> (32 + view.length * 8) != 0
 {
 polars_bail!(ComputeError: "view contained non-zero padding in prefix");
 }

crates/polars-arrow/src/array/binview/view.rs:537

For spilled views, comparing the raw u128 is not enough (the same prefix can appear in different buffers), so any code that needs content equality branches on the same length test first. The hot grouper for string keys does exactly that: inline keys are compared as u128s, spilled keys as byte slices.

 rust
 
 
 if view.is_inline() {
 self.table.insert_key(
 hash,
 (),
 force_hot,
 next_hash,
 |_, b| view == b.1,

crates/polars-expr/src/hot_groups/binview.rs:51

and for the spilled branch:

 rust
 
 
 } else {
 let bytes = view.get_external_slice_unchecked(buffers);
 self.table.insert_key(
 hash,
 (),
 force_hot,
 next_hash,
 |_, b| {
 // We only reach here if the hash matched, so jump straight to full comparison.
 bytes == b.2
 },

crates/polars-expr/src/hot_groups/binview.rs:67

The hash function branches the same way:

 rust
 
 
 #[inline]
 pub unsafe fn hash_with_buffers_unchecked<B: AsRef<[u8]>, S: std::hash::BuildHasher>(
 &self,
 buffers: &[B],
 build_hasher: &S,
 ) -> u64 {
 unsafe {
 if self.length <= Self::MAX_INLINE_SIZE {
 build_hasher.hash_one(self.as_u128())
 } else {
 build_hasher.hash_one(self.get_external_slice_unchecked(buffers))
 }
 }
 }

crates/polars-arrow/src/array/binview/view.rs:259

Filling views at SIMD speed

Building a column of short strings is a memcpy problem, and Polars solves it with portable SIMD. extend_with_inlinable_strided writes 16-byte views directly into the destination buffer, using a shuffle to fan one 16-byte source load out into several views:

 rust
 
 
 // This way we can reuse the same load for multiple views.
 repeat_with!(
 I = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] => {
 if I < BLOCKS_PER_LOAD {
 let zero = u8x16::default();
 const SWIZZLE: [usize; 16] = const {
 let mut swizzle = [16usize; 16];

 let mut i = 0;
 while i < N {
 let idx = i + I * N;
 if idx < 16 {
 swizzle[4+i] = idx;
 }
 i += 1;
 }

 swizzle
 };

 let scattered = simd_swizzle!(loaded, zero, SWIZZLE);
 let view_bytes = (scattered | length_mask).to_array();

 // SAFETY: dst has the capacity reserved and view_bytes is 16
 // bytes long.
 unsafe {
 core::ptr::copy_nonoverlapping(view_bytes.as_ptr(), dst, 16);
 dst = dst.add(16);
 }
 }
 }
 );

crates/polars-arrow/src/array/binview/view.rs:385

The length_mask ORs the length into the low byte of the view’s length field, and the shuffle places the string bytes at offset 4. The net effect: for a column of fixed-width short strings, the views are produced at memory bandwidth with no per-element branch.

A second, smaller inline type

The same trick appears elsewhere in the codebase at a smaller scale. SmallBytes in polars-utils inlines up to 8 bytes and uses the top bit of the length word as a null flag:

 rust
 
 
/// `Option<Box<[u8]>>` with inlining of `<= size_of::<*mut u8>()` bytes.
pub struct SmallBytes {
 data: PointerOrArray,
 len: MarkedUsize,
}

crates/polars-utils/src/small_bytes.rs:19

 rust
 
 
/// A usize where the top bit is used as a marked indicator.
#[derive(Copy, Clone, PartialEq, Eq, Hash)]
pub struct MarkedUsize(usize);

impl MarkedUsize {
 pub const UNMARKED_MAX: usize = ((1 << (usize::BITS - 1)) - 1);

 #[inline(always)]
 pub const fn new(idx: usize, marked: bool) -> Self {
 debug_assert!(idx >> (usize::BITS - 1) == 0);
 Self(idx | ((marked as usize) << (usize::BITS - 1)))
 }

crates/polars-utils/src/marked_usize.rs:1

It is used for partition keys in the streaming engine (crates/polars-stream/src/nodes/io_sinks/components/partition_key.rs:7), where the key is short, hashed per row, and must not allocate. For user-facing string values, Polars uses PlSmallStr, which is a thin wrapper around compact_str::CompactString and inlines up to 24 bytes on 64-bit platforms.

The bottom line

The View type is a 16-byte union without the union:

Cost. Every string costs 16 bytes of view, even a 1-character one, plus the validity bit and the 8 bytes of ChunkedArray bookkeeping. A column of short strings uses more memory than a Utf8 column with the same values.
Benefit. A short string costs no extra memory at all beyond its view — no offset array, no per-string allocation, no pointer chase. Reading it is a load from the cache line you already touched to get the length.
Cost. The length field is the tag, so every consumer pays one well-predicted branch. Code that wants to be branch-free has to specialize (see the as_u128 equality and the SIMD builder).
Benefit. Spilled bytes are written once and never moved: buffers are flushed at 8 KiB and doubled up to 16 MiB, and the buffer_idx/offset pair is stable for the lifetime of the column, which is what makes zero-copy slicing and sharing of the whole column possible.
Cost. A View is 4-byte aligned, not 8 or 16, because it is four u32 fields — kernels that want to load views as u64s must use the Bytes16Alignment4 wrapper from polars-arrow/src/types/aligned_bytes.rs and do their own unaligned loads.

The one-line summary: Polars optimises the common string, not the average one. If your strings are short — country codes, enum labels, UUID prefixes, hashes, IPs — the Utf8View layout is close to free, and the Arrow Utf8 layout would be paying 4 bytes per row for offsets you do not need.

The Arrow Row Format: How DataFusion Packs a Multi-Column Sort Key Into One Comparable Byte String

Jay Hubbert — Wed, 21 Oct 2026 04:05:00 +0000

TL;DR: Arrow’s RowConverter turns N Arrow columns into one flat byte buffer per row, using an order-preserving encoding: signed ints get their sign bit flipped, floats become their IEEE-754 totalOrder integer image, strings are written in self-delimiting blocks, and each field starts with a null sentinel. Because the encoding is order-preserving, memcmp on two concatenated rows is the lexicographic comparison of the original columns. DataFusion uses this in the sort merge path, in range partitioning, in the TopK heap, and in the generic GROUP BY fallback — everywhere a multi-column key has to be compared against itself millions of times.

The problem: comparing a tuple of columns, over and over

Sorting is the canonical case. ORDER BY country, city, population produces a composite key. The naive implementation keeps the three columns as three Arrow arrays and compares them pairwise:

 text
 
 
compare(a, b):
 if a.country != b.country: return a.country < b.country
 if a.city != b.city: return a.city < b.city
 return a.population < b.population

That is fine for one comparison. A sort does O(N log N) of them, and a merge of K sorted runs does O(N log K) more. With 100 million rows and a 3-column key, you are looking at billions of tuple comparisons, each of which has to branch on “which column are we in”, re-dispatch on the runtime type of that column, and handle the possibility that the column is null.

The classic fix in the database world is the “tuple” or “serialized key” representation: pack the whole composite key into one opaque byte string so that comparing keys is a single memcmp. The trouble is doing it without losing information. If you just concatenate the raw bytes, ("a", "bc") and ("ab", "c") become the same bytes. The encoding has to be injective and order-preserving at the same time.

What Arrow’s row format actually looks like

Arrow’s arrow-row crate (vendored here as arrow-row-60.0.0) defines exactly such an encoding. The format is documented at length in arrow-row-60.0.0/src/lib.rs:199-350, and the interesting parts are:

Signed integers get their sign bit flipped and are written big-endian, so the byte sequence is monotonically increasing in the numeric value:

 text
 
 
// arrow-row-60.0.0/src/lib.rs:225-241
/// Signed integers have their most significant sign bit flipped, and are then encoded in the
/// same manner as an unsigned integer.
///
/// ```text
/// ┌──┬──┬──┬──┐ ┌──┬──┬──┬──┐ ┌──┬──┬──┬──┬──┐
/// 5 │05│00│00│00│ │05│00│00│80│ │01│80│00│00│05│
/// -5 │FB│FF│FF│FF│ │FB│FF│FF│7F│ │01│7F│FF│FF│FB│

Nulls are encoded as a leading sentinel byte, 0x00 for nulls-first and 0xFF for nulls-last, with a valid value starting with 0x01:

 text
 
 
// arrow-row-60.0.0/src/fixed.rs:224-241
 let null_sentinel = null_sentinel(opts);
 for (value_idx, is_valid) in nulls.iter().enumerate() {
 ...
 if is_valid {
 to_write[0] = 1;
 ...
 } else {
 data[*offset] = null_sentinel;
 }

Variable-length strings use a COBS-inspired block scheme so the decoder never has to scan for a terminator:

 text
 
 
// arrow-row-60.0.0/src/variable.rs:82-102
/// - single `0_u8` if null
/// - single `1_u8` if empty array
/// - `2_u8` if not empty, followed by one or more blocks
/// where a block is encoded as
/// - [`BLOCK_SIZE`] bytes of string data, padded with 0s
/// - `0xFF_u8` if this is not the last block for this string
/// - otherwise the length of the block as a `u8`

The first four blocks are only 8 bytes wide, to avoid space amplification for short strings (variable.rs:30-38).

Descending order is obtained by negating the encoded bytes of the non-null payload — not by reversing the comparator:

 text
 
 
// arrow-row-60.0.0/src/lib.rs:561-563
/// ## Reverse Column Ordering
///
/// The order of a given column can be reversed by negating the encoded bytes of non-null values

This is the crucial design decision. Because the same byte-comparison code handles ASC and DESC, the merge loop needs no special cases, and the sort direction becomes a property of the encoding, not of the comparison.

Floats are converted to their IEEE-754 totalOrder integer image, so -0.0 sorts before +0.0 and NaNs have a defined position. This is not the same as SQL’s = semantics — DataFusion has to normalize -0.0 to +0.0 explicitly wherever it uses the row format for equality, which it does in the GROUP BY fallback:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/row.rs:124-130
 fn intern(&mut self, cols: &[ArrayRef], groups: &mut Vec<usize>) -> Result<()> {
 // Normalize -0.0 → +0.0 so RowConverter (IEEE 754 totalOrder) and
 // primitive hashing both group ±0 together. No-op for non-float
 // columns.
 let normalized_cols: Vec<ArrayRef> =
 cols.iter().map(normalize_float_zero).collect();

The payoff: one comparison, no dispatch

The Rows type is a flat Vec<u8> plus a Vec<usize> of row offsets:

 rust
 
 
// arrow-row-60.0.0/src/lib.rs:1331-1341
pub struct Rows {
 /// Underlying row bytes
 buffer: Vec<u8>,
 /// Row `i` has data `&buffer[offsets[i]..offsets[i+1]]`
 offsets: Vec<usize>,
 /// The config for these rows
 config: RowConfig,
}

and comparing two rows is literally slice::cmp:

 rust
 
 
// arrow-row-60.0.0/src/lib.rs:1593-1599
impl Ord for Row<'_> {
 #[inline]
 fn cmp(&self, other: &Self) -> Ordering {
 self.data.cmp(other.data)
 }
}

memcmp on two byte strings. No type dispatch, no null branch, no “which column are we in” branch. That is the entire trick.

Where DataFusion uses it

1. The sort merge path

RowCursorStream is the multi-column path of the sort-preserving merge. It builds one RowConverter for the whole ordering and appends each batch’s key columns into a reusable Rows buffer:

 rust
 
 
// datafusion/physical-plan/src/sorts/stream.rs:191-201
 let sort_fields = expressions
 .iter()
 .map(|expr| {
 let data_type = expr.expr.data_type(schema)?;
 Ok(SortField::new_with_options(data_type, expr.options))
 })
 .collect::<Result<Vec<_>>>()?;

 let streams: Vec<_> = streams.into_iter().map(|s| s.fuse()).collect();
 let stream_count = streams.len();
 let converter = RowConverter::new(sort_fields)?;

 rust
 
 
// datafusion/physical-plan/src/sorts/stream.rs:216-240
 fn convert_batch(
 &mut self,
 batch: &RecordBatch,
 stream_idx: usize,
 ) -> Result<RowValues> {
 let cols = evaluate_expressions_to_arrays(&self.column_expressions, batch)?;

 // At this point, ownership should of this Rows should be unique
 let mut rows = self.rows.take_next(stream_idx, &self.converter)?;

 rows.clear();

 self.converter.append(&mut rows, &cols)?;
 self.reservation.try_resize(self.converter.size())?;
 ...

Note that the sort itself does not use the row format — sort_batch uses Arrow’s lexsort_to_indices (datafusion/physical-plan/src/sorts/sort.rs:1097-1116). The row format is for the merge, where the alternative is comparing two streams of differently-shaped columns against each other, repeatedly.

The RowValues cursor goes one step further and caches the raw pointer and length of the current row, so the hot comparison is a pointer dereference plus memcmp rather than two offset lookups:

 rust
 
 
// datafusion/physical-plan/src/sorts/cursor.rs:293-317
 fn compare(l: &Self, l_idx: usize, r: &Self, r_idx: usize) -> Ordering {
 ...
 let _ = (l_idx, r_idx);
 l.current_slice().cmp(r.current_slice())
 }

Interestingly, the comment above it says the compiler’s own inlining choice wins here:

 rust
 
 
// datafusion/physical-plan/src/sorts/cursor.rs:279-281
 // No inline hint on purpose: for the heavyweight `Rows` byte comparison the
 // compiler's own choice wins — both `#[inline]` and `#[inline(never)]`
 // measurably regress the multi-column merge path.

2. Range partitioning

The range-partition router needs to answer “which of the N split points does this row fall after?” for an arbitrary composite key. The trick is to encode the split points into the row format once, at construction time, and then binary-search the encoded rows:

 rust
 
 
// datafusion/physical-plan/src/repartition/range.rs:543-555
 let split_point_rows = if split_points.is_empty() {
 None
 } else {
 let split_point_arrays = (0..num_cols)
 .map(|col_idx| {
 let col_scalars =
 split_points.iter().map(|sp| sp.values()[col_idx].clone());
 ScalarValue::iter_to_array(col_scalars)
 })
 .collect::<Result<Vec<_>>>()?;

 Some(row_converter.convert_columns(&split_point_arrays)?)
 };

 rust
 
 
// datafusion/physical-plan/src/repartition/range.rs:583-596
#[inline]
fn partition_point_rows(sp_rows: &Rows, row: &Row<'_>) -> usize {
 let mut left = 0;
 let mut right = sp_rows.num_rows();
 while left < right {
 let mid = left + (right - left) / 2;
 if sp_rows.row(mid) <= *row {
 left = mid + 1;
 } else {
 right = mid;
 }
 }
 left
}

A composite-key range lookup is now O(log N) memcmps. This is a genuine reuse: the sort path and the shuffle path share one encoding, so a query that both sorts and range-shuffles does not pay two different serialization costs.

3. The TopK heap

TopK keeps a bounded heap of the best k rows. The heap’s comparison key is the row-format bytes, so the heap is a BinaryHeap<Row> and the eviction test is a byte compare:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:515-535
 fn find_new_topk_items(
 &mut self,
 items: impl Iterator<Item = usize>,
 batch_entry: &mut RecordBatchEntry,
 ) -> usize {
 let mut replacements = 0;
 let rows = &mut self.scratch_rows;
 for (index, row) in items.zip(rows.iter()) {
 match self.heap.max() {
 // heap has k items, and the new row is greater (by byte order,
 // after row conversion) than the top K ==> it is not a new topk
 Some(max_row) if row.as_ref() >= max_row.row() => {}
 // don't yet have k items or new item is lower than the currently k low values
 None | Some(_) => {
 self.heap.add(batch_entry, row, index);
 replacements += 1;
 }
 }
 }
 replacements
 }

TopK also keeps a second converter for the common sort prefix between the input ordering and the TopK ordering, so it can early-exit the whole operator when the input has moved past the heap boundary (datafusion/physical-plan/src/topk/mod.rs:393-398 and :708-784). That early exit is only expressible because “past the boundary” is a byte comparison on a prefix row.

4. The generic GROUP BY fallback

GroupValuesRows is the fallback used for nested or unsupported group key types. It interns keys by hashing the columnar input (so the hash is computed on the cheap typed representation) and then uses the row format only for the equality check:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/row.rs:152-163
 for (row, &target_hash) in batch_hashes.iter().enumerate() {
 let entry = self.map.find_mut(target_hash, |(exist_hash, group_idx)| {
 // Somewhat surprisingly, this closure can be called even if the
 // hash doesn't match, so check the hash first with an integer
 // comparison first avoid the more expensive comparison with
 // group value. https://github.com/apache/datafusion/pull/11718
 target_hash == *exist_hash
 // verify that the group that we are inserting with hash is
 // actually the same key value as the group in
 // existing_idx (aka group_values @ row)
 && group_rows.row(row) == group_values.row(*group_idx)
 });

So the hash is over typed columns, and only the tie-break on a hash collision is a memcmp of two row buffers. That is the right split: hashing is O(key width) per row either way, but the equality check only runs on collisions, which are rare.

Why not just use a struct of arrays everywhere?

Because the row format has real costs.

Space. Every row pays a per-field sentinel byte, plus big-endian padding for integers, plus the block padding for strings. A 3-column key of (Utf8, Int32, Utf8) costs roughly 1 + len(country) + 1 + 4 + 1 + len(city) bytes plus block padding, versus the “natural” 4 + 4 + 8 for a tight columnar layout. For a narrow key it is a wash; for a wide string key it can be 2x.

Encode cost. You must serialize every row before you can compare any of them. If the query only needs the first column for comparison (a common case: ORDER BY a with b as a tiebreaker that rarely matters), the row format does strictly more work than a single-column compare. This is why DataFusion keeps a specialized single-column path:

 rust
 
 
// datafusion/physical-plan/src/sorts/streaming_merge.rs:279-292
 // Special case single column comparisons with optimized cursor implementations
 if expressions.len() == 1 {
 let sort = expressions[0].clone();
 let data_type = sort.expr.data_type(schema.as_ref())?;
 downcast_primitive! {
 data_type => (primitive_merge_helper, ...),
 DataType::Utf8 => merge_helper!(StringArray, ...),
 ...
 }
 }

Lossy for some types. The row format hydrates dictionaries and run-end-encoded arrays to their plain value type during encoding (arrow-row-60.0.0/src/lib.rs:305-312). DataFusion has to re-encode the group keys on the way out to restore the declared schema type — see encode_array_if_necessary in datafusion/physical-plan/src/aggregates/group_values/row.rs:329-464, which walks nested list/map/struct types and re-applies dictionary encoding to each child. That is a nontrivial amount of code that exists purely to undo a lossy step in the encoding.

Type-stability. The format is explicitly documented as unstable:

 rust
 
 
// arrow-row-60.0.0/src/lib.rs:187
/// *Note: The encoding of the row format may change from release to release.*

Rows produced by one RowConverter can only be compared with rows produced by the same converter (arrow-row-60.0.0/src/lib.rs:1041-1045 asserts this with Arc::ptr_eq). DataFusion therefore has to plumb a single converter through the whole operator lifetime and never let two of them meet.

Failure modes to watch for

Comparing rows from different converters. The Arc::ptr_eq assertion in RowConverter::append catches the common case, but a Row that outlives its converter’s fields is a use-after-free hazard in principle. DataFusion sidesteps this by keeping the converter alive in the same struct as the rows.
Float totalOrder vs SQL equality. The row format says -0.0 and +0.0 are different; SQL says they are equal. Any use of the row format for equality must normalize first, exactly as GroupValuesRows::intern does.
Null ordering drift. SortField::new_with_options carries nulls_first, and the encoding flips the null sentinel accordingly. Getting the direction wrong silently reorders nulls. DataFusion’s build_sort_fields threads the PhysicalSortExpr’s options straight through, which is the right thing to do but leaves no room for a mistake.
Memory accounting. Rows holds a Vec<u8> that grows to the high water mark of the batch. DataFusion’s ReusableRows in datafusion/physical-plan/src/sorts/stream.rs:97-162 keeps one buffer per partition alive and re-reserves it against the memory pool, precisely because the retained capacity is what the pool is actually charged for.

The takeaway

The row format is a serialization format with a comparison contract. By making the encoding order-preserving and self-deliminating at the same time, Arrow gets a key type whose comparison is memcmp — no type dispatch, no null branches, no per-column loop. DataFusion leans on this everywhere a composite key is compared repeatedly: merges, range routing, bounded heaps, and the generic grouping fallback. The price is encode cost, space, and a documented-unstable wire format — which is why the single column fast path still exists, and why the dictionary flattening in the encoder has to be undone by hand on the way out.

Polars Buffers Are Immutable: The Refcounted Storage Behind Every Column

Jay Hubbert — Wed, 21 Oct 2026 04:00:00 +0000

TL;DR: polars_buffer::Buffer<T> is not a Vec<T> with extra steps. It is three fields — a shared storage, a raw pointer and a length — and it has no capacity, no reserve and no resize. A buffer is frozen the moment it is created; growing it is somebody else’s job (the mutable builders), and every consumer that wants a different view of the same bytes gets one by bumping a pointer. The sharing is not std::sync::Arc either: it is a hand-rolled AtomicU64 refcount with four different ownership modes, because Arrow buffers can be owned by a Vec, by a foreign owner, by nobody, or by nobody at all.

I went looking for Polars’ buffer allocator expecting to find the usual performance folklore: a custom aligned allocator, 64-byte-aligned allocations for SIMD, amortized doubling inside the buffer type. None of that exists in crates/polars-buffer at this commit. What exists is more interesting, and it explains a lot of Polars’ behaviour downstream.

The premise of this article is the one-line summary that opens crates/polars-buffer/src/buffer.rs:

 rust
 
 
/// [`Buffer`] is a contiguous memory region that can be shared across
/// thread boundaries.
///
/// The easiest way to think about [`Buffer<T>`] is being equivalent to
/// a `Arc<Vec<T>>`, with the following differences:
/// * slicing and cloning is `O(1)`.
/// * it supports external allocated memory
///
/// The easiest way to create one is to use its implementation of `From<Vec<T>>`.
///
/// # Examples
/// ```
/// use polars_buffer::Buffer;
///
/// let mut buffer: Buffer<u32> = vec![1, 2, 3].into();
/// assert_eq!(buffer.as_ref(), [1, 2, 3].as_ref());
///
/// // it supports copy-on-write semantics (i.e. back to a `Vec`)
/// let vec: Vec<u32> = buffer.into_mut().right().unwrap();
/// assert_eq!(vec, vec![1, 2, 3]);
///
/// // cloning and slicing is `O(1)` (data is shared)
/// let mut buffer: Buffer<u32> = vec![1, 2, 3].into();
/// let mut sliced = buffer.clone();
/// sliced.slice(1, 1);
/// assert_eq!(sliced.as_ref(), [2].as_ref());
/// // but cloning forbids getting mut since `slice` and `buffer` now share data
/// assert_eq!(buffer.get_mut_slice(), None);
/// ```

crates/polars-buffer/src/buffer.rs:10

Two properties fall out of that doc comment, and they are the whole design: cloning is O(1), and mutability is conditional on being the only owner.

A buffer is a pointer, a length, and a refcount

The struct is deliberately minimal:

 rust
 
 
pub struct Buffer<T> {
 /// The internal byte buffer.
 storage: SharedStorage<T>,

 /// A pointer into the buffer where our data starts.
 ptr: *const T,

 // The length of the buffer.
 length: usize,
}

crates/polars-buffer/src/buffer.rs:41

No capacity. No length-vs-capacity distinction. The pointer may be past the start of the allocation (that is what slicing does), and the storage knows the true extent of the allocation. That is why is_sliced is a comparison of two different quantities:

 rust
 
 
 /// Returns whether underlying data is sliced.
 /// If sliced the [`Buffer`] is backed by
 /// more data than the length of `Self`.
 pub fn is_sliced(&self) -> bool {
 self.storage.len() != self.length
 }

crates/polars-buffer/src/buffer.rs:161

Cloning a Buffer is a refcount bump plus two copies of a pointer and a length:

 rust
 
 
impl<T> Clone for Buffer<T> {
 #[inline(always)]
 fn clone(&self) -> Self {
 Self {
 storage: self.storage.clone(),
 ptr: self.ptr,
 length: self.length,
 }
 }
}

crates/polars-buffer/src/buffer.rs:54

Slicing is a pointer bump, not a copy:

 rust
 
 
 #[inline]
 pub fn slice_in_place<R: RangeBounds<usize>>(&mut self, range: R) {
 unsafe {
 let Range { start, end } = check_range(range, ..self.len());
 self.ptr = self.ptr.add(start);
 self.length = end - start;
 }
 }

crates/polars-buffer/src/buffer.rs:218

This is the single most load-bearing decision in the whole buffer layer. A filter that keeps 3 rows out of a million does not copy three rows; it clones a 16-byte handle and moves a pointer. A take that gathers 1000 scattered rows copies only the view metadata for the 1000 surviving rows if it can, and otherwise rebuilds a buffer — but the unchanged chunks of the column are shared with the source, untouched.

Four ways a buffer can own its bytes

SharedStorage is a hand-rolled Arc. The inner struct is a raw heap allocation behind a NonNull, which means the refcount lives next to the data rather than next to the handle — one allocation for the bytes, one for the refcount, and the Buffer handle itself is trivially copyable.

 rust
 
 
enum BackingStorage {
 Vec {
 original_capacity: usize, // Elements, not bytes.
 vtable: &'static VecVTable,
 },
 ForeignOwner(Box<dyn Any + Send + 'static>),

 /// Backed by some external method which we do not need to take care of,
 /// but we still should refcount and drop the SharedStorageInner.
 External,

 /// Both the backing storage and the SharedStorageInner are leaked, no
 /// refcounting is done. This technically should be a flag on
 /// SharedStorageInner instead of being here, but that would add 8 more
 /// bytes to SharedStorageInner, so here it is.
 Leaked,
}

struct SharedStorageInner<T> {
 ref_count: AtomicU64,
 ptr: *mut T,
 length_in_bytes: usize,
 backing: BackingStorage,
 // https://github.com/rust-lang/rfcs/blob/master/text/0769-sound-generic-drop.md#phantom-data
 phantom: PhantomData<T>,
}

crates/polars-buffer/src/storage.rs:40 and crates/polars-buffer/src/storage.rs:56

The four modes cover the real life of a buffer in a dataframe engine:

Vec — the normal case: Polars allocated it and must free it.
ForeignOwner — the bytes belong to someone else (a Box<dyn Any> that keeps them alive), e.g. an imported Arrow array.
External — the bytes are borrowed from a slice whose lifetime the caller guarantees.
Leaked — nobody owns it and nobody refcounts it.

The Vec variant is where the type system earns its keep. SharedStorage<T> is generic over T, but the allocation was made as a Vec<U> for some other U (that is how try_transmute reinterprets a u64 buffer as f64 bytes). So the drop path cannot call Vec::<T>::from_raw_parts; it calls through a vtable captured at allocation time:

 rust
 
 
// Allows us to transmute between types while also keeping the original
// stats and drop method of the Vec around.
struct VecVTable {
 size: usize,
 align: usize,
 drop_buffer: unsafe fn(*mut (), usize),
}

impl VecVTable {
 const fn new<T>() -> Self {
 unsafe fn drop_buffer<T>(ptr: *mut (), cap: usize) {
 unsafe { drop(Vec::from_raw_parts(ptr.cast::<T>(), 0, cap)) }
 }

 Self {
 size: size_of::<T>(),
 align: align_of::<T>(),
 drop_buffer: drop_buffer::<T>,
 }
 }

crates/polars-buffer/src/storage.rs:22

The vtable also records the size and alignment of the original element type, which is what makes “give me my Vec back” a checked operation rather than UB:

 rust
 
 
 // We may only go back to a Vec if we originally came from a Vec
 // where the desired size/align matches the original.
 let BackingStorage::Vec {
 original_capacity,
 vtable,
 } = &mut inner.backing
 else {
 return None;
 };

 if vtable.size != size_of::<T>() || vtable.align != align_of::<T>() {
 return None;
 }

crates/polars-buffer/src/storage.rs:384

This is where the alignment story actually lives: not in the allocator, but in the type of the element. A Buffer<u8> and a Buffer<f64> have the same size in bytes but different alignments, and the code refuses to transmute between them unless the target’s alignment divides the source’s:

 rust
 
 
 // The pointer must be properly aligned for U.
 // We can skip this check if the alignment of U divides the alignment of T.
 if !align_of::<T>().is_multiple_of(align_of::<U>()) && !inner.ptr.cast::<U>().is_aligned() {
 return Err(self);
 }

crates/polars-buffer/src/storage.rs:438

The refcount, copied from `Arc<T>`

The atomic operations are literally Arc’s, with the comments to prove it:

 rust
 
 
impl<T> Clone for SharedStorage<T> {
 fn clone(&self) -> Self {
 let inner = self.inner();
 if !matches!(inner.backing, BackingStorage::Leaked) {
 // Ordering semantics copied from Arc<T>.
 inner.ref_count.fetch_add(1, Ordering::Relaxed);
 }
 Self {
 inner: self.inner,
 phantom: PhantomData,
 }
 }
}

impl<T> Drop for SharedStorage<T> {
 fn drop(&mut self) {
 let inner = self.inner();
 if matches!(inner.backing, BackingStorage::Leaked) {
 return;
 }

 // Ordering semantics copied from Arc<T>.
 if inner.ref_count.fetch_sub(1, Ordering::Release) == 1 {
 std::sync::atomic::fence(Ordering::Acquire);
 unsafe {
 self.drop_slow();
 }
 }
 }
}

crates/polars-buffer/src/storage.rs:491 and crates/polars-buffer/src/storage.rs:483

Two details are worth pausing on. First, Leaked storage skips the atomic entirely — the global zero pool (§ below) pays no atomic traffic at all. Second, the Release/Acquire fence pair is the standard “publish my writes before the last owner drops the data” pattern: without it, the final Drop of the elements could race with a still-running reader.

The Drop of the inner storage is where the actual bytes are released:

 rust
 
 
impl<T> Drop for SharedStorageInner<T> {
 fn drop(&mut self) {
 match core::mem::replace(&mut self.backing, BackingStorage::External) {
 BackingStorage::ForeignOwner(o) => drop(o),
 BackingStorage::Vec {
 original_capacity,
 vtable,
 } => unsafe {
 // Drop the elements in our slice.
 if std::mem::needs_drop::<T>() {
 core::ptr::drop_in_place(core::ptr::slice_from_raw_parts_mut(
 self.ptr,
 self.length_in_bytes / size_of::<T>(),
 ));
 }

 // Free the buffer.
 if original_capacity > 0 {
 (vtable.drop_buffer)(self.ptr.cast(), original_capacity);
 }
 },
 BackingStorage::External | BackingStorage::Leaked => {},
 }
 }
}

crates/polars-buffer/src/storage.rs:88

Note original_capacity, not length_in_bytes: the deallocation has to hand back exactly the capacity the allocator gave us, and Vec::from_raw_parts with a mismatched capacity is undefined behaviour. That is why the capacity is captured at allocation time, and why the whole VecVTable dance exists in the first place.

Growth is somebody else’s job

Because Buffer has no reserve and no resize, something else has to grow. In the string path that someone is MutableBinaryViewArray, which owns a plain Vec<u8> that it doubles:

 rust
 
 
 /// Get a [`View`] for a specific set of bytes.
 pub fn push_value_into_buffer(&mut self, bytes: &[u8]) -> View {
 assert!(bytes.len() <= u32::MAX as usize);

 if bytes.len() <= View::MAX_INLINE_SIZE as usize {
 View::new_inline(bytes)
 } else {
 self.total_buffer_len += bytes.len();

 // We want to make sure that we never have to memcopy between buffers. So if the
 // current buffer is not large enough, create a new buffer that is large enough and try
 // to anticipate the larger size.
 if self.in_progress_buffer.len().saturating_add(bytes.len())
 > usize::min(
 BINVIEW_ARROW_BUFFER_LEN_LIMIT,
 self.in_progress_buffer.capacity(),
 )
 {
 const {
 assert!(MAX_EXP_BLOCK_SIZE < BINVIEW_ARROW_BUFFER_LEN_LIMIT);
 }

 // Allocate a new buffer and flush the old buffer.
 let new_capacity = usize::max(
 bytes.len(),
 (self.in_progress_buffer.capacity() * 2)
 .clamp(DEFAULT_BLOCK_SIZE, MAX_EXP_BLOCK_SIZE),
 );

 let in_progress = Vec::with_capacity(new_capacity);
 let flushed = std::mem::replace(&mut self.in_progress_buffer, in_progress);
 if !flushed.is_empty() {
 self.completed_buffers.push(flushed.into())
 }
 }

crates/polars-arrow/src/array/binview/mutable.rs:329

The doubling is clamped between DEFAULT_BLOCK_SIZE (8 KiB) and MAX_EXP_BLOCK_SIZE (16 MiB), so tiny columns do not allocate a huge buffer and enormous columns stop doubling. The same pattern appears in View::new_with_buffers_slow, with a RESIZE_LIMIT of 8192 bytes deciding whether to grow the last buffer or start a new one.

The architectural point: mutation happens in builders, which own Vecs; the finished Buffer is a frozen artefact. MutableBinaryViewArray::freeze converts the Vec<View> and Vec<Buffer<u8>> into Buffer<View> and Buffer<Buffer<u8>> and from then on the data is shared, not owned.

The one place Polars aligns memory on purpose

I promised to address the 64-byte claim, so let me be explicit: Polars does not align every Arrow buffer to 64 bytes, and there is no custom aligned allocator in polars-buffer. Buffers are ordinary Vec allocations, and their alignment is whatever the element type requires — 1 byte for u8, 4 for u32, 8 for f64.

There is exactly one over-aligned allocation in this layer, and it is for zeroed memory:

 rust
 
 
#[repr(C, align(4096))]
#[derive(Copy, Clone)]
struct Aligned([u8; 4096]);

// We intentionally leak 8MiB of zeroed memory once so we don't have to
// refcount it.
const GLOBAL_ZERO_SIZE: usize = 8 * 1024 * 1024;
static GLOBAL_ZEROES: LazyLock<SharedStorage<Aligned>> = LazyLock::new(|| {
 const { assert!(GLOBAL_ZERO_SIZE.is_multiple_of(size_of::<Aligned>())) };
 let chunks = GLOBAL_ZERO_SIZE / size_of::<Aligned>();
 let v = vec![Aligned([0; _]); chunks];
 let mut ss = SharedStorage::from_vec(v);
 ss.leak();
 ss
});

crates/polars-buffer/src/buffer.rs:369

Buffer::zeroed then hands out slices of that single 8 MiB block, and the Leaked backing means no refcounting and no free — the block lives for the process lifetime. This is used for validity bitmaps and IPC padding, where “give me 1 MiB of zeros” should not mean “calloc 1 MiB and possibly touch every page”. The 4096-byte alignment of the chunk type means every handed-out slice is page-aligned, which is what you want if the consumer is going to mmap or hand the pages to something that expects page granularity.

There is a second, subtler alignment trick, in the empty storage:

 rust
 
 
 /// Creates an empty SharedStorage.
 pub const fn empty() -> Self {
 assert!(align_of::<T>() <= 1 << 30);
 static INNER: SharedStorageInner<()> = SharedStorageInner {
 ref_count: AtomicU64::new(1),
 ptr: core::ptr::without_provenance_mut(1 << 30), // Very overaligned for any T.
 length_in_bytes: 0,
 backing: BackingStorage::Leaked,
 phantom: PhantomData,
 };

crates/polars-buffer/src/storage.rs:134

An empty buffer still needs a pointer that is valid for reads if anyone ever dereferences it, and the simplest such pointer is one that is absurdly aligned — 1 << 30 is aligned to every power of two up to a gigabyte, so align_of::<T>() is satisfied for any sane T without a per-type branch.

Where alignment actually shows up: the kernel side

Alignment does matter enormously for SIMD, but Polars handles it in the kernel layer rather than the allocator. The trick is a family of byte types that carry the alignment of the type they stand for:

 rust
 
 
/// A representation of a type as raw bytes with the same alignment as the original type.
pub trait AlignedBytes: Pod + Default + Eq {
 const ALIGNMENT: usize;
 const SIZE: usize;
 const SIZE_ALIGNMENT_PAIR: PrimitiveSizeAlignmentPair;

 type Unaligned: AsRef<[u8]>
 + AsMut<[u8]>
 + std::ops::Index<usize, Output = u8>
 + std::ops::IndexMut<usize, Output = u8>
 + for<'a> TryFrom<&'a [u8]>
 + std::fmt::Debug
 + Default
 + IntoIterator<Item = u8>
 + Pod;

crates/polars-arrow/src/types/aligned_bytes.rs:11

and the macro that generates them:

 rust
 
 
 /// Bytes with a size and alignment.
 ///
 /// This is used to reduce the monomorphizations for routines that solely rely on the size
 /// and alignment of types.
 #[derive(Debug, Copy, Clone, PartialEq, Eq, Default, Pod, Zeroable)]
 #[repr(C, align($alignment))]
 pub struct $name([u8; $size]);

crates/polars-arrow/src/types/aligned_bytes.rs:43

The sizes and alignments that exist are the honest ones — 1, 2, 4, 8, 12, 16 and 32 bytes with alignments 1, 2, 4, 8 and 16:

 rust
 
 
impl_aligned_bytes! {
 (Bytes1Alignment1, 1, 1, S1A1, [u8, i8], u8),
 (Bytes2Alignment2, 2, 2, S2A2, [u16, i16, pf16], u16),
 (Bytes4Alignment4, 4, 4, S4A4, [u32, i32, f32], u32),
 (Bytes8Alignment8, 8, 8, S8A8, [u64, i64, f64], u64),
 (Bytes8Alignment4, 8, 4, S8A4, [days_ms]),
 (Bytes12Alignment4, 12, 4, S12A4, [[u32; 3]]),
 (Bytes16Alignment4, 16, 4, S16A4, [View]),
 (Bytes16Alignment8, 16, 8, S16A8, [months_days_ns]),
 (Bytes16Alignment16, 16, 16, S16A16, [u128, i128]),
 (Bytes32Alignment16, 32, 16, S32A16, [i256]),
}

crates/polars-arrow/src/types/aligned_bytes.rs:144

The comment on Bytes16Alignment4 is the punchline: a View is 16 bytes wide but only 4-byte aligned, because it is four u32 fields. A kernel that wants to load 16 bytes of views as two u64s cannot assume 8-byte alignment, so the aligned wrapper tells the compiler the truth and the kernel does the (un)aligned load it has to do.

Getting the bytes back

Since the buffer is shared, the only way to mutate it is to be the sole owner. into_mut and get_mut_slice encode that:

 rust
 
 
 /// Returns a mutable reference to its underlying [`Vec`], if possible.
 ///
 /// This operation returns [`Either::Right`] iff this [`Buffer`]:
 /// * has no alive clones
 /// * has not been imported from the C data interface (FFI)
 #[inline]
 pub fn into_mut(mut self) -> Either<Self, Vec<T>> {
 // We lose information if the data is sliced.
 if self.is_sliced() {
 return Either::Left(self);
 }
 match self.storage.try_into_vec() {
 Ok(v) => Either::Right(v),
 Err(slf) => {
 self.storage = slf;
 Either::Left(self)
 },
 }
 }

crates/polars-buffer/src/buffer.rs:296

Three separate reasons the answer can be “no”: the buffer is sliced (the storage holds more than the buffer exposes), there are other owners, or it came from the C data interface and Polars does not own the allocation. Note that the API never silently copies: it returns the buffer back unchanged and lets the caller decide.

The explicit escape hatch is to_vec, which is the one place a copy is guaranteed:

 rust
 
 
impl<T: Clone> Buffer<T> {
 pub fn to_vec(self) -> Vec<T> {
 match self.into_mut() {
 Either::Right(v) => v,
 Either::Left(same) => same.as_slice().to_vec(),
 }
 }
}

crates/polars-buffer/src/buffer.rs:357

The bottom line

The buffer layer buys three things and pays for them in a specific way:

Cheap slicing and cheap cloning. A filter or a head/tail slice produces a new handle, not new bytes. Cost: the data cannot be mutated in place, so every mutating operation either needs exclusivity or must rebuild.
A single ownership model for four kinds of memory. Vec, ForeignOwner, External and Leaked in one enum, with a vtable to make the generic drop sound. Cost: 40-odd bytes of bookkeeping per allocation, and a vtable pointer chase on every deallocation.
Alignment handled where it matters. Element-type alignment is checked on every transmute; the only over-aligned allocation is the 8 MiB zero pool; SIMD alignment is a property of the kernel’s byte types, not of the allocator.

The trade-off to keep in mind when reading the rest of the codebase: if you see a Vec in a Polars kernel, it is because that kernel is building data. If you see a Buffer, it is data that is finished and shared. The boundary between the two is the freeze.

Dictionary Columns Are a Hash Cache: How DataFusion Groups by Dictionary Index and What That Buys

Jay Hubbert — Wed, 21 Oct 2026 02:37:00 +0000

TL;DR: A DictionaryArray is not a compressed column — it is a pre-built hash cache that the file format already paid for. DataFusion exploits that: it hashes the D distinct dictionary values once, scatters the D hashes out to the N row positions, and memoizes a value index → group slot map, so a row whose key repeats a dictionary index costs one array lookup instead of a string comparison. GROUP BY on a dictionary column does O(D) hashing work per batch, not O(N). And DataFusion will introduce dictionary encoding for you when reading Parquet, because Parquet files already contain the dictionary.

The naive way to group a string column

GROUP BY country over 10 million rows means 10 million string comparisons and 10 million string hashings. The strings are long, the values repeat (maybe 200 distinct countries), and the CPU spends all its time re-deriving facts the data already told it.

The obvious fix is to intern the strings into integers first: build a table value → id, replace each string with its id, group the ids. That is exactly what a dictionary-encoded Arrow column already is:

 text
 
DictionaryArray<Int32, Utf8>

 keys: [ 0, 1, 0, 1, 2, 0, ... ] <- one Int32 per row
 values: [ "US", "DE", "FR" ] <- the distinct values, once

The keys array says “row 0 has value 0”; the values array is the interned table. DataFusion’s job is to use that table rather than throw it away.

The hashing trick: hash the values, scatter to the keys

The first place the dictionary pays off is hashing. A hash aggregation must compute a u64 hash per row. For a plain string array that means hashing 10 million strings. For a dictionary array, the same 200 distinct strings only need hashing once — the hashes are then scattered to the row positions by looking up each row’s key.

 rust
 
 
// datafusion/common/src/hash_utils.rs:626-642
/// Hash the values in a dictionary array.
#[cfg(not(feature = "force_hash_collisions"))]
fn hash_dictionary<K: ArrowDictionaryKeyType>(
 array: &DictionaryArray<K>,
 random_state: &impl HashState,
 hashes_buffer: &mut [u64],
 multi_col: bool,
) -> Result<()> {
 // Hash each dictionary value once, and then use that computed
 // hash for each key value to avoid a potentially expensive
 // redundant hashing for large dictionary elements (e.g. strings)
 let dict_values = array.values();
 let mut dict_hashes = vec![0; dict_values.len()];
 create_hashes([dict_values], random_state, &mut dict_hashes)?;
 dispatch_dictionary_scatter(array, &dict_hashes, hashes_buffer, multi_col);
 Ok(())
}

The shape is O(D) hashing plus O(N) cheap array reads. The scatter itself uses const generics to eliminate runtime branching:

 rust
 
 
// datafusion/common/src/hash_utils.rs:522-539
/// Scatter precomputed dictionary value hashes to key positions.
///
/// Uses const generics to eliminate runtime branching in the hot loop:
/// - `HAS_NULL_KEYS`: Whether to check for null dictionary keys
/// - `HAS_NULL_VALUES`: Whether to check for null dictionary values
/// - `MULTI_COL`: Whether to combine with existing hash (true) or initialize (false)
#[cfg(not(feature = "force_hash_collisions"))]
#[inline(never)]
fn hash_dictionary_scatter<
 K: ArrowDictionaryKeyType,
 const HAS_NULL_KEYS: bool,
 const HAS_NULL_VALUES: bool,
 const MULTI_COL: bool,
>(
 array: &DictionaryArray<K>,
 dict_hashes: &[u64],
 hashes_buffer: &mut [u64],
) {

A dispatcher picks one of eight monomorphizations once per batch, so the hot loop contains zero null checks and zero branches on the multi-column flag. Note the null subtlety the const generics encode: a valid dictionary key can point at a null dictionary value. That row is logically null but physically valid, and HAS_NULL_VALUES handles it.

The grouping trick: memoize value-index → group-slot

Hashing is only half of it. After the hash, the aggregation must decide “is this row’s key the same group as the one in bucket H?”. The natural answer is to compare the values. But the dictionary gives a better answer: within one dictionary, the same index means the same value.

DataFusion’s DictionaryGroupValuesColumn is built around that idea. Its state is three things: the dedup store of values, a hashbrown table that bridges batches, and a per-batch memo:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:37-71
/// [`GroupColumn`] for dictionary-encoded columns with key type `K`.
///
/// `inner` holds one slot per distinct value seen across all batches.
/// `group_to_inner[group_idx]` maps each group to its slot in `inner`,
/// so groups with the same value share a slot rather than duplicating data.
pub struct DictionaryGroupValuesColumn<K: ArrowDictionaryKeyType + Send + Sync> {
 /// Deduplicated store of distinct values.
 inner: Box<dyn GroupColumn>,
 /// Unary null array (length 1) reused for every null appended to `inner`.
 null_array: ArrayRef,
 /// Maps each group index to its slot in `inner`.
 group_to_inner: Vec<usize>,
 /// Lookup table mapping `(value_hash, inner_slot)` for each non-null distinct value.
 value_dedup: HashTable<(u64, usize)>,
 ...
 /// Maps `val_idx → inner_slot` for the values array in `cached_values`.
 ///
 /// Valid only while `cached_values` is `Some` and matches the incoming
 /// values array; `usize::MAX` means "not yet resolved". Entries are only
 /// ever filled in, never invalidated, because `inner` slots are stable
 /// under append. `take_n` remaps `inner` and therefore drops the cache.
 val_to_inner: Vec<usize>,
 /// Hashes of the values array in `cached_values`, one per `val_idx`.
 val_hashes: Vec<u64>,
 /// Weak reference to the values array that `val_hashes` and `val_to_inner`
 /// were built for. A matching live array reuses both caches without
 /// retaining an input dictionary after its batch is released.
 cached_values: Option<Weak<dyn Array>>,
 _phantom: PhantomData<K>,
}

val_to_inner is the memo table. val_hashes is the precomputed hash array. value_dedup is the cross-batch bridge: a hashbrown table mapping (value_hash, inner_slot) so a value seen in a previous batch — with a different dictionary, hence a different index — still resolves to the same inner slot.

The append path is where the memo pays off:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:441-476
 fn vectorized_append(&mut self, array: &ArrayRef, rows: &[usize]) -> Result<()> {
 let dict = array.as_dictionary::<K>();
 let dict_keys = dict.keys();
 let dict_values = dict.values();

 // Reuses `val_hashes`/`val_to_inner` when this batch carries the same
 // values array as the last one, which is the common case downstream of
 // a repartition or filter. Previously this unconditionally re-hashed
 // and re-initialised both buffers, costing O(D) per batch.
 self.sync_value_cache(dict_values);
 ...
 let raw_keys = dict_keys.values();
 if dict_keys.null_count() == 0 {
 for &row in rows {
 let val_idx = raw_keys[row].as_usize();
 if self.val_to_inner[val_idx] == usize::MAX {
 // A non-null key can still point to a null value in the values array.
 self.val_to_inner[val_idx] = if dict_values.is_null(val_idx) {
 self.find_or_insert_null()?
 } else {
 self.find_or_insert_value(
 dict_values,
 val_idx,
 self.val_hashes[val_idx],
 )?
 };
 }
 self.group_to_inner.push(self.val_to_inner[val_idx]);
 }
 } else {

Read the fast path: self.val_to_inner[val_idx]. One array read. If it is not usize::MAX, the group slot is already known — no hashing, no value comparison. For a batch with 10 million rows and 200 distinct countries, 9,999,800 rows take that path.

The slow path — an index never seen before in this batch — does a hash lookup by value hash and, on a miss, appends the value to the inner store:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:208-235
 fn find_or_insert_value(
 &mut self,
 dict_values: &ArrayRef,
 val_idx: usize,
 hash: u64,
 ) -> Result<usize> {
 let inner = &*self.inner;
 let existing = self
 .value_dedup
 .find(hash, |&(entry_hash, slot)| {
 entry_hash == hash && inner.equal_to(slot, dict_values, val_idx)
 })
 .map(|&(_, slot)| slot);

 match existing {
 Some(slot) => Ok(slot),
 None => {
 let slot = self.inner.len();
 self.inner.append_val(dict_values, val_idx)?;
 self.value_dedup.insert_accounted(
 (hash, slot),
 |&(entry_hash, _)| entry_hash,
 &mut self.value_dedup_size,
 );
 Ok(slot)
 }
 }
 }

This is the part that makes the whole thing correct rather than merely fast. The grouping key is the value, not the index — because the same value can arrive with different indices in different batches, and two batches may use entirely different dictionaries. value_dedup bridges that: keyed on the hash of the value, with an equality check to resolve collisions.

So the precise statement of the optimization is:

The dictionary index is a memo key within a batch (a cache of “index → group”); the value hash is the grouping key across batches. The dictionary never changes the grouping semantics — only the cost.

The equality path: memoize, but only positive results

The comparison side has the same structure, with a subtle twist. When the aggregation checks “does this incoming row’s key equal the group in this bucket?”, it can consult val_to_inner — but only if the existing group’s slot is known for that index. DataFusion exploits a fact about its own data structure:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:237-271
 /// Compares the value at `val_idx` against `inner[lhs_slot]` directly,
 /// memorizing `lhs_slot` in `val_to_inner` when they are equal.
 ///
 /// Each distinct value occupies exactly one slot in `inner`, so an equal
 /// comparison proves `lhs_slot` is the slot for `val_idx` without probing
 /// `value_dedup` or hashing. An unequal result is deliberately not
 /// memoized: it says which slot the value is *not* in, not which one it
 /// is. That is what keeps `val_to_inner` fill-only and never stale.
 ///
 /// Only called on a `val_to_inner` miss. It takes the fields it needs
 /// rather than `&mut self`, so the caller's hot loop can keep its slices
 /// of `val_to_inner` and `group_to_inner` in registers across the call.
 #[cold]
 #[inline(never)]
 fn equal_to_uncached(
 val_to_inner: &mut [usize],
 inner: &dyn GroupColumn,
 null_array: &ArrayRef,
 dict_values: &ArrayRef,
 val_idx: usize,
 lhs_slot: usize,
 ) -> bool {
 let equal = if dict_values.is_null(val_idx) {
 inner.equal_to(lhs_slot, null_array, 0)
 } else {
 inner.equal_to(lhs_slot, dict_values, val_idx)
 };

 if equal {
 val_to_inner[val_idx] = lhs_slot;
 }
 equal
 }

Three decisions in one function:

Only positive results are memoized. A negative result (“this index is not that slot”) does not say which slot it is, so caching it would be wrong. This keeps val_to_inner fill-only, so it can never go stale.
The function is #[cold] and #[inline(never)]. It is the rare path; keeping it out of the hot loop lets LLVM pipeline the hot loop’s array reads.
It takes &mut [usize] rather than &mut self. The caller’s hot loop can hold its slices in registers across the call.

The vectorized path puts it together, and adds an adaptive heuristic:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:364-380
 let cached = self.is_cached(dict_values);
 if !cached && rhs_rows.len() < dict_values.len() {
 self.equal_to_per_row(
 lhs_rows,
 dict_values,
 dict,
 rhs_rows,
 equal_to_results,
 );
 return;
 }
 self.sync_value_cache(dict_values);

 let raw_keys = dict_keys.values();
 let group_to_inner = self.group_to_inner.as_slice();
 let val_to_inner = self.val_to_inner.as_mut_slice();

Building the memo costs O(D). If the batch only has a handful of rows to check, O(D) setup to save O(rows) work is a loss — so DataFusion falls back to a per-row comparison path. A genuinely good adaptive heuristic, and easy to get wrong.

The “no null keys” arm of the same loop skips the get_bit(idx) guard, with the comment explaining why: the loop only ever writes false into equal_to_results, so overwriting an already-false bit is harmless. That removes a branch from the hot loop.

The cross-batch cache: `sync_value_cache`

val_hashes and val_to_inner are only valid for a specific dictionary. If the next batch has a different one, they must be rebuilt. DataFusion detects this by pointer identity:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:184-206
 fn is_cached(&self, values: &ArrayRef) -> bool {
 self.cached_values
 .as_ref()
 .is_some_and(|cached| std::ptr::addr_eq(cached.as_ptr(), Arc::as_ptr(values)))
 }

 /// Makes `val_hashes` and `val_to_inner` correspond to `dict_values`,
 /// reusing them when they were already built for this same values array.
 ///
 /// `take`, `filter` and repartitioning clone the values `Arc` and rewrite
 /// only the keys, so batches in a partition usually share one values
 /// array; without this the per-batch cost is O(dictionary cardinality)
 /// rather than O(rows). `take_n` remaps `inner` and clears
 /// `cached_values`, forcing a rebuild.
 fn sync_value_cache(&mut self, dict_values: &ArrayRef) {
 if self.is_cached(dict_values) {
 return;
 }
 self.hash_values(dict_values);
 self.val_to_inner.clear();
 self.val_to_inner.resize(dict_values.len(), usize::MAX);
 self.cached_values = Some(Arc::downgrade(dict_values));
 }

The Weak reference is a nice touch: it detects “same dictionary” without keeping the dictionary alive. If the input batch is dropped, the Weak fails to upgrade and the cache is rebuilt for the next dictionary. Using Weak rather than a strong Arc avoids accidentally pinning an input batch’s dictionary buffer in memory.

The output: a dictionary again, but compacted

On the way out, group values must be emitted as a dictionary again. Group indices are group ids, and the dictionary keys must index into the values array. And the null group must not consume a key index:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:91-140
 /// Build a `DictionaryArray` from `values` (all inner slots) and the
 /// per-group slot mapping. The null inner slot, if any, is excluded from
 /// the values array and its groups emit a null key — so it never consumes
 /// a key index regardless of where it sits in `inner`.
 fn into_dict(
 values: ArrayRef,
 group_to_inner: &[usize],
 null_inner_slot: Option<usize>,
 ) -> ArrayRef {
 let Some(null_slot) = null_inner_slot else {
 // Fast path: no null group — raw slot indices are valid keys.
 let keys: PrimitiveArray<K> = group_to_inner
 .iter()
 .map(|&slot| Some(K::Native::usize_as(slot)))
 .collect();
 return Arc::new(DictionaryArray::<K>::new(keys, values));
 };

 // Build a compact remap: each non-null slot gets a contiguous key
 // starting from 0; the null slot is skipped entirely.
 ...
 // Compact values array: drop the null slot so key indices stay tight.
 let compact_indices: Int64Array = (0..n)
 .filter(|&i| i != null_slot)
 .map(|i| i as i64)
 .collect();
 let compact =
 take(&*values, &compact_indices, None).expect("compact values in into_dict");
 Arc::new(DictionaryArray::<K>::new(keys, compact))
 }

Without the remap, the null group’s slot would leave a hole in the key space, wasting a key index and making the key type overflow sooner. There is an explicit guard:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs:142-155
 // https://github.com/apache/datafusion/issues/23127
 // Null groups emit a null key (None), not a slot index, so the null inner
 // slot never consumes a key index regardless of its position in inner.
 fn check_key_overflow(&self) -> Result<()> {
 let non_null_count = self.inner.len() - self.null_inner_slot.is_some() as usize;
 if !Self::key_type_fits(non_null_count) {
 return exec_err!(
 "Dictionary key type {:?} cannot represent {} distinct values",
 K::DATA_TYPE,
 non_null_count
 );
 }
 Ok(())
 }

An Int8 dictionary key can only address 127 distinct values. Group by a column with 200 distinct values and the Int8 key type overflows. This is a real failure mode of dictionary-based grouping, caught with a clean error rather than a silent wraparound.

Where dictionary encoding comes from

Now the good part: DataFusion introduces dictionary encoding automatically, because Parquet files already contain the dictionary.

Parquet’s dictionary encoding stores, per column chunk, a dictionary page (the distinct values) plus RLE-encoded indices into it. The information is right there in the file. By default the Parquet reader expands it back to plain values, throwing away the dictionary and paying to re-derive it.

DataFusion has a config option to keep it:

 rust
 
 
// datafusion/common/src/config.rs:1429-1438
 /// (reading) If true, top-level string and binary Parquet columns with
 /// dictionary pages are inferred and scanned as
 /// `Dictionary<Int32, Utf8>` / `Dictionary<Int32, Binary>` instead of
 /// their plain value type.
 ///
 /// This applies only when DataFusion infers the table schema. Tables with
 /// a user-supplied schema are not promoted because the Parquet footer is
 /// not read at DDL time, so dictionary pages cannot be detected per column.
 /// See <https://github.com/apache/datafusion/issues/24112>
 pub enable_rle_to_dictionary: bool, default = false

The promotion happens at schema-inference time, by inspecting the Parquet footer for dictionary pages:

 rust
 
 
// datafusion/datasource-parquet/src/metadata.rs:463-505
 let schema = if self.enable_rle_to_dictionary {
 let schema_descr = file_metadata.schema_descr();
 // Top-level columns that have a dictionary page in at least one row group.
 let dict_cols: HashSet<String> = metadata
 .row_groups()
 .iter()
 .flat_map(|rg| {
 rg.columns()
 .iter()
 .enumerate()
 .filter_map(|(col_idx, col)| {
 col.dictionary_page_offset()?;
 let col_desc = schema_descr.column(col_idx);
 let parts = col_desc.path().parts();
 // Skip nested columns: their leaf name doesn't match the
 // Arrow top-level field name.
 (parts.len() == 1).then(|| parts[0].clone())
 })
 })
 .collect();

Note “in at least one row group” and the parts.len() == 1 check — only top-level columns are promoted, because for a nested column the Parquet leaf name does not match the Arrow field name.

Since different files may disagree, the schemas are normalized before merging:

 rust
 
 
// datafusion/datasource-parquet/src/file_format.rs:494-499
 // Normalize dict-promoted schemas before merging so mixed dict/plain files merge cleanly.
 let mut schemas: Vec<Schema> =
 schemas.into_iter().map(|(_, schema)| schema).collect();
 if self.options.global.enable_rle_to_dictionary {
 schemas = crate::schema_coercion::uniform_dict_schemas(schemas);
 }

So the user-visible schema has a Dictionary column, and the whole pipeline — including the GROUP BY — sees a dictionary. The dictionary-aware group column is selected automatically, because the type dispatch in make_group_column handles DataType::Dictionary:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs:1025-1051
 DataType::Dictionary(key_dt, value_dt) => {
 let new_field = Field::new("", *value_dt.clone(), true);
 let inner = make_group_column(&new_field)?;
 macro_rules! dict_col {
 ($T:ty) => {
 Box::new(dictionary::DictionaryGroupValuesColumn::<$T>::new(
 inner, &new_field,
 ))
 };
 }
 let col: Box<dyn GroupColumn> = match key_dt.as_ref() {
 DataType::Int8 => dict_col!(Int8Type),
 DataType::Int16 => dict_col!(Int16Type),
 DataType::Int32 => dict_col!(Int32Type),
 DataType::Int64 => dict_col!(Int64Type),
 DataType::UInt8 => dict_col!(UInt8Type),
 DataType::UInt16 => dict_col!(UInt16Type),
 DataType::UInt32 => dict_col!(UInt32Type),
 DataType::UInt64 => dict_col!(UInt64Type),
 _ => {
 return not_impl_err!(
 "Dictionary key type {key_dt} not supported in GroupValuesColumn"
 );
 }
 };
 Some(col)
 }

The inner is a normal GroupColumn for the value type — the dictionary column delegates storage of the actual values to the ordinary builder and only adds index bookkeeping on top. Clean layering: the dictionary is an index optimization, not a different storage model.

The catch: the row format throws dictionaries away

Arrow’s RowConverter hydrates dictionaries to their plain value type. So the row-format GROUP BY fallback (GroupValuesRows) cannot use the dictionary at all, and must re-encode the output to put it back:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/row.rs:319-334
/// Re-apply dictionary / run-end encoding to `array` so it matches `expected`.
///
/// Arrow's [`RowConverter`] flattens dictionary and run-end-encoded values to
/// their plain value type during row encoding (at [`RowConverter::append`]),
/// so any group-value array produced from the row format is in that plain
/// type and must be re-encoded to match the schema's expected type before
/// being returned. Shared with the generic row-backed `GroupColumn`.
pub(crate) fn encode_array_if_necessary(
 array: &ArrayRef,
 expected: &DataType,
) -> Result<ArrayRef> {
 match (expected, array.data_type()) {
 ...
 (DataType::Dictionary(_, _), _) => Ok(cast(array.as_ref(), expected)?),

The single-line cast re-creates the dictionary from scratch, a full pass over the group values. Not free — but paid once per group on the output path, not per row.

Dictionary awareness is spreading

The same “operate on the values, then scatter to the keys” pattern shows up across the expression layer. IN lists with dictionary needles evaluate the filter once on the dictionary values and then take the result by the keys:

 rust
 
 
// datafusion/physical-expr/src/expressions/in_list/dictionary_filter.rs:41-48
 fn contains(&self, needles: &dyn Array, negated: bool) -> Result<BooleanArray> {
 let Some(dictionary) = needles.as_any_dictionary_opt() else {
 return self.inner.contains(needles, negated);
 };
 let values_contains = self.contains(dictionary.values().as_ref(), negated)?;
 let result = take(&values_contains, dictionary.keys(), None)?;
 Ok(result.as_boolean().clone())
 }

Case conversion transforms only the values and reuses the keys (datafusion/functions/src/string/common.rs:539-547). And the hash join must cope with the logical null problem — a valid dictionary key pointing at a null value is logically null:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/exec.rs:3620-3622
 let build_has_null = null_aware == Some(NullAwareMode::RightAnti)
 && !left_values.is_empty()
 && left_values[0].logical_null_count() > 0;

Failure modes and trade-offs

The dictionary is per-batch. Two batches can have completely different dictionaries, so the index is not a stable group identity. Any code that treats “same index” as “same value” across batches is wrong. value_dedup is exactly the mechanism that prevents this bug, and it is the part you must not remove.
Key type overflow. Int8 keys address 127 values. Exceed that and you get an error, not a wrong answer — but it is an error, and check_key_overflow runs after every append.
The dictionary can be bigger than the data. A column with 10 million rows and 9 million distinct values has a nearly-as-large dictionary, and the memo tables cost 8 and 8 bytes per dictionary entry. The rhs_rows.len() < dict_values.len() heuristic guards the comparison path, but the memo arrays are still allocated at D size.
Dictionary + row format is a losing combination. The row format hydrates the dictionary, so you pay for the dictionary and the row bytes, then re-encode on output.
enable_rle_to_dictionary is off by default. That is a deliberate trade: promoting the column changes the user-visible schema (the column’s type becomes Dictionary<Int32, Utf8>), which can break downstream code expecting Utf8. It also only works when DataFusion infers the schema.
Promotion is per-file-set, not per-file. A column is promoted if any row group in any file has a dictionary page, and then uniform_dict_schemas makes all files agree. Files without a dictionary get their values expanded and re-interned — wasted work, but it keeps the schema consistent.

The takeaway

A dictionary column is a hash table the file format built for you. DataFusion’s job is not to throw it away, and the way it does that is by splitting the problem in two:

Within a batch, the dictionary index is a memo key: val_to_inner turns “which group does this row belong to?” into a single array read for every row whose index was seen before. Only positive results are memoized, so the cache can never go stale.
Across batches, the value is the grouping key, bridged by value_dedup, a hashbrown table keyed on the value’s hash with an equality check for collisions.

The hashing side gets the same treatment: hash_dictionary hashes the D distinct values once and scatters the results to the N row positions, with eight const-generic monomorphizations keeping the hot loop branch-free. Grouping a dictionary column costs O(D) string work per batch rather than O(N), and the only thing DataFusion adds is the bookkeeping to make the index and the value agree. It is a clean example of the general principle: when the input format has already done expensive work, build your data structures so that work is not repeated.

DataFusion's Branchless IN Filter: Why x IN (10, 20, 30) Is Faster Than Building a Hash Table

Jay Hubbert — Wed, 21 Oct 2026 02:05:00 +0000

TL;DR: WHERE status IN ('new', 'pending', 'paid') looks like it should be a hash lookup, but for a short list a hash table is slower: you pay for the table, and the hash is a serial dependency. DataFusion instead compares the input against every value in the list and combines the results with a non-short-circuiting |. The regular, branch-free comparison sequence is something the CPU can vectorise, and for a list of three or ten values that beats a hash lookup outright. The choice of strategy is made per type and per list length, at plan time.

IN lists are everywhere. WHERE id IN (1, 2, 3). WHERE country IN ('US', 'GB', 'DE'). WHERE tenant_id IN (...) with a hundred values. They look like a simple membership test, and Arrow provides arrow::compute::kernels::filter::is_in, which builds a hash table and probes it.

But building a hash table for three values is absurd. The table costs an allocation, a hash per value, a probe per input row, and — crucially — a branch per probe, which the branch predictor cannot learn.

DataFusion’s InListExpr has a whole family of specialised filters for this, and picking the right one is a genuinely interesting piece of engineering. The code lives in datafusion/physical-expr/src/expressions/in_list/ — nine files, about 2,400 lines — and the entry point is a strategy chooser.

The strategy chooser

 rust
 
 
// datafusion/physical-expr/src/expressions/in_list/strategy.rs
pub(super) fn instantiate_static_filter(
 in_array: ArrayRef,
 needle_data_type: &DataType,
) -> Result<StaticFilterRef> {
 let in_array = flatten_dictionary_haystack(in_array)?;

 let filter = if view_types_match(needle_data_type, in_array.data_type())
 && let Some(filter) = instantiate_byte_view_filter(&in_array)?
 {
 filter
 } else if let Some(filter) = instantiate_fixed_size_binary_filter(&in_array)? {
 filter
 } else if let Some(filter) = instantiate_primitive_filter(&in_array)? {
 filter
 } else {
 Arc::new(ArrayStaticFilter::try_new(in_array)?)
 };

 // Plain inputs can call the concrete filter directly. Dictionary inputs
 // share one adapter across all concrete filter types.
 if matches!(needle_data_type, DataType::Dictionary(_, _)) {
 Ok(Arc::new(DictionaryFilter::new(filter)))
 } else {
 Ok(filter)
 }
}

Read it as a cascade: try the most specific filter that applies, fall back to the generic one. The generic fallback is ArrayStaticFilter — an Arrow-array-backed filter that works for any type at the cost of a linear scan.

Note the order: byte-view, then fixed-size-binary, then primitive, then generic. The specialised ones are tried first because they know the memory layout and can do better.

The dictionary handling is a nice touch. A dictionary-encoded column wraps a value array in an index array. The values are what the IN list matches against, so the filter is built on the flattened values and a DictionaryFilter adapter maps indices back. And the flattening is recursive:

 rust
 
 
fn flatten_dictionary_haystack(mut in_array: ArrayRef) -> Result<ArrayRef> {
 // Flatten every dictionary layer so the final value type can use a
 // specialized filter.
 while let Some(dictionary) = in_array.as_any_dictionary_opt() {
 in_array = take(dictionary.values().as_ref(), dictionary.keys(), None)?;
 }

 Ok(in_array)
}

A dictionary of a dictionary is a real thing, and this handles it.

The branchless filter

Here is the centrepiece. branchless_filter.rs opens with an explanation that reads like a textbook on micro-optimisation:

Why use a branchless filter?

For a short list such as x IN (10, 20, 30), it can be faster to compare x with all three values than to build and search a hash table.

“Branchless” means that the filter always checks every list value. It combines the answers with |, while || would stop at the first match. This regular sequence of comparisons is easier for the compiler and CPU to optimize.

The kernel is twelve lines:

 rust
 
 
#[inline]
#[expect(
 clippy::needless_bitwise_bool,
 reason = "the non-short-circuiting `|` is the point of this branchless kernel"
)]
fn check_values<C, const N: usize>(
 in_list_values: &[C],
 input_values: &[C],
) -> BooleanBuffer
where
 C: Copy + PartialEq,
{
 let in_list_values: &[C; N] = in_list_values
 .try_into()
 .expect("comparison length matches IN-list values");

 BooleanBuffer::collect_bool(input_values.len(), |i| {
 // SAFETY: `collect_bool` invokes this closure for indices in
 // `0..input_values.len()`.
 let input_value = unsafe { *input_values.get_unchecked(i) };
 // `|` checks every list value; `||` would stop after the first match.
 in_list_values
 .iter()
 .fold(false, |acc, &value| acc | (value == input_value))
 })
}

Three things to notice.

const N: usize. The list length is a compile-time constant in this function. The list is stored as &[C; N], a fixed-size array reference, so the loop over the list values is fully unrolled by the compiler: N comparisons, no loop counter, no bounds check on the list.

| not ||. The #[expect(clippy::needless_bitwise_bool, ...)] attribute is a confession: Clippy wants ||, and the code deliberately uses |. With ||, the compiler emits a branch per comparison and the loop over the list is data-dependent. With |, every comparison is executed unconditionally, the results are OR-ed together, and the CPU can issue the comparisons in parallel. The expect attribute with an explanation is the right way to fight the linter here — a #[allow] with no comment would just be a mystery later.

get_unchecked. The input index is unchecked, with a SAFETY comment justifying it: collect_bool guarantees the closure is called for indices in 0..input_values.len(). This is the standard Arrow-kernel idiom — eliminate the bounds check from the innermost loop.

The size limits, and why they are what they are

The maximum list length for the branchless path is not arbitrary, and the source explains each one:

 rust
 
 
/// Maximum list size for branchless lookup on 1-byte primitives.
///
/// Sixteen 1-byte values fit in one 128-bit SIMD vector, so this keeps the
/// branchless list small enough for a single vectorized membership check.
const BRANCHLESS_MAX_1B: usize = 16;

/// Maximum list size for branchless lookup on 2-byte primitives.
///
/// Eight 2-byte values fit in one 128-bit SIMD vector, ...
const BRANCHLESS_MAX_2B: usize = 8;

/// Maximum list size for branchless lookup on 4-byte primitives.
///
/// Thirty-two 4-byte values keep the inline list at 128 bytes. Beyond that,
/// the comparison chain and filter footprint grow enough that the
/// hash/generic fallback is a better fit.
const BRANCHLESS_MAX_4B: usize = 32;

/// Maximum list size for branchless lookup on 8-byte primitives.
///
/// Sixteen 8-byte values use the same 128-byte inline-list budget as 4-byte
/// primitives. ...
const BRANCHLESS_MAX_8B: usize = 16;

/// Maximum list size for branchless lookup on 16-byte primitives.
///
/// These comparisons are wider, so this path is limited to four values.
const BRANCHLESS_MAX_16B: usize = 4;

The 1-byte and 2-byte limits are SIMD limits: 16 bytes fit in one XMM register, 8 two-byte values fit in one XMM register. The 4-byte and 8-byte limits are cache-line limits: 32 × 4 bytes = 128 bytes = one cache line, 16 × 8 bytes = 128 bytes = one cache line. The 16-byte limit is 4 values because the comparisons are so wide that the chain gets unwieldy.

The module header explains the interaction with the bitmap filter:

One- and two-byte values have an especially efficient next step: every possible bit pattern fits in a compact bitmap. For a longer list, the bitmap filter turns on one bit for each listed value, then checks membership with a direct bit lookup. This becomes a better fit before a 64- or 128-comparison branchless chain would be useful. Wider types have too many possible values for such a bitmap, so their limits are tuned separately.

The bitmap filter

For 1- and 2-byte types, there is a strictly better structure than both the branchless chain and a hash table: a bitmap of the entire value space. 256 possible byte values, 65,536 possible 16-bit values.

 rust
 
 
struct BitmapFilter<T: BitmapFilterType> {
 null_count: usize,
 bits: T::Storage,
}

fn try_new(in_array: &ArrayRef) -> Result<Self> {
 ...
 let mut bits = T::Storage::new_zeroed();
 let values = prim_array.values();
 match prim_array.nulls() {
 None => {
 for &v in values {
 bits.set_bit(T::index(v));
 }
 }
 Some(nulls) => {
 for i in
 BitIndexIterator::new(nulls.validity(), nulls.offset(), nulls.len())
 {
 bits.set_bit(T::index(values[i]));
 }
 }
 }
 Ok(Self { null_count: prim_array.null_count(), bits })
}

#[inline(always)]
fn check(&self, needle: T::Native) -> bool {
 self.bits.get_bit(T::index(needle))
}

The membership test is a single bit lookup. The entire filter for a 256-element byte domain is 32 bytes — four cache lines, or one if you are lucky. This is the classic “direct lookup table” optimisation, and it is strictly better than a hash set for small domains: no hashing, no probing, no collisions, one memory access.

The construction handles nulls by iterating only the valid rows via BitIndexIterator — the null entries are simply not set in the bitmap, and their count is remembered so the SQL null semantics can be applied later.

The hash set filter

For 4-byte and 8-byte types, the domain is too large for a bitmap, so the choice is between the branchless chain and a hash set. The hash set wins for long lists:

 rust
 
 
DataType::Int32 => {
 Arc::new(PrimitiveHashSetFilter::<Int32Type>::try_new(in_array)?)
}
DataType::Int64 => {
 Arc::new(PrimitiveHashSetFilter::<Int64Type>::try_new(in_array)?)
}

and for floats, the set needs an ordered wrapper to be a valid hash map key:

 rust
 
 
// Float primitive types use ordered wrapper keys for Hash/Eq.
DataType::Float32 => Arc::new(PrimitiveHashSetFilter::<
 Float32Type,
 OrderedFloat32,
>::try_new(in_array)?),

OrderedFloat32 is the standard trick: f32 does not implement Eq or Hash because NaN != NaN and 0.0 != -0.0. Wrapping it in a total-order newtype makes it usable as a key. Note that this is the opposite choice from the branchless filter, which compares floats by their raw bit pattern and deliberately preserves the 0.0 vs -0.0 distinction.

The bit-pattern trick

The branchless filter’s type mapping is one of the more surprising bits of the code:

 rust
 
 
macro_rules! branchless_filter_type {
 ($logical:ty, $compare:ty, $max_len:expr) => {
 // The branchless filter reads the same Arrow value buffer as the
 // comparison type. That is only valid when both native types have
 // the same width, so catch any bad mapping here at compile time.
 const _: () = assert!(
 size_of::<<$logical as ArrowPrimitiveType>::Native>()
 == size_of::<<$compare as ArrowPrimitiveType>::Native>(),
 "BranchlessFilterType::CompareType must use the same native width"
 );

 impl BranchlessFilterType for $logical {
 type CompareType = $compare;
 const MAX_LIST_LEN: usize = $max_len;
 }
 };
}

branchless_filter_type!(Int8Type, UInt8Type, BRANCHLESS_MAX_1B);
branchless_filter_type!(Float32Type, UInt32Type, BRANCHLESS_MAX_4B);
branchless_filter_type!(Int64Type, UInt64Type, BRANCHLESS_MAX_8B);

Float32 is compared as UInt32. Int64 as UInt64. And the reason is in the module header:

Some Arrow types share the same in-memory representation. For example, a Float32 and a UInt32 both use four bytes per value. The filter compares those stored bits through an unsigned type of the same size, without copying the value buffer. A bit pattern is simply the bytes Arrow uses to store a value. Comparing it preserves details such as 0.0 versus -0.0 and different NaN values.

So the filter does not copy the values into a Vec<u32> — it reinterprets the existing Arrow buffer as unsigned integers. branchless_values does this with zero copies:

 rust
 
 
fn branchless_values<T>(array: &PrimitiveArray<T>) -> ScalarBuffer<BranchlessNative<T>>
where
 T: BranchlessFilterType,
{
 let values = array.values();
 // `values.inner()` already reflects any slice applied to the array.
 ScalarBuffer::<BranchlessNative<T>>::new(values.inner().clone(), 0, values.len())
}

Hmm — one thing to note: the clone of the inner Buffer is cheap (it is a reference-counted Bytes handle), but it is not free, and the values buffer is shared rather than copied. The type-level assert! on the size equality is a compile-time guard against a future edit that maps a type to a different-width comparison type, which would be a memory-safety bug.

The float bit-pattern comparison is not the same as float equality: 0.0 and -0.0 have different bit patterns, so x IN (0.0) would be false for -0.0. For a membership test against a list, the set semantics are preserved (the list contains 0.0; -0.0 is not in it), but the ordering is by bit pattern, not by numeric value. Whether that matters depends on the semantics you want. The comment says the choice is deliberate.

SQL null semantics

Every one of these filters has to reproduce SQL’s three-valued logic: x IN (1, NULL) is NULL (not false) when x is not 1, because the list contains a NULL. And x NOT IN (1, NULL) is NULL for every x.

The filters handle this by separating the membership computation from the null computation:

 rust
 
 
Ok(build_result_from_contains(
 v.nulls(), // nulls in the input
 self.null_count > 0, // nulls in the IN list
 negated,
 matches,
))

The membership bitmask is computed first, and the null rules are applied afterwards, in one place, for all filter types. The result.rs module is 111 lines and does exactly that — it is the single source of truth for IN/NOT IN null semantics in the whole engine.

The doc comment in the branchless filter spells out the flow:

Evaluation first records which values matched, then build_result_from_contains combines it with input nulls, list nulls, and NOT IN to produce the usual SQL null behavior.

The catch

The IN list must be constant. The whole static-filter machinery depends on the list being a compile-time-known array. InListExpr::try_new evaluates the list expressions at plan time; if they are not constant, static_filter is None and the expression falls back to a per-batch evaluation.
The filter is built at plan time and lives as long as the plan. For a plan with a million-element IN list, that is a million elements of memory held for the duration of the query — and the hash set is built once, so it amortises.
The specialisation is per type and per list length. The membership_check_for_len function generates a function pointer per supported length:
rust
```
match T::MAX_LIST_LEN {
 4 => choose!(0, 1, 2, 3, 4),
 8 => choose!(0, 1, 2, 3, 4, 5, 6, 7, 8),
 16 => choose!(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16),
 32 => choose!(0, 1, 2, 3, ...),
 _ => unreachable!("list-size limits must be 4, 8, 16, or 32"),
}
```
The comment explains why: “Keeping the length out of BranchlessFilter avoids generating a full copy of the filter for every supported length.” So the kernel is monomorphised per length, but the filter struct is not — the length is a function pointer stored in the struct. A clever compromise between code size and inner-loop efficiency.
The bit-pattern comparison for floats is not numeric equality. It distinguishes 0.0 from -0.0 and different NaN payloads. For IN list membership this is arguably more correct than float equality, but it is a deliberate deviation from == and worth knowing about.
Only primitives, byte views and fixed-size binaries get the good treatment. Strings (Utf8) fall back to ArrayStaticFilter, which does a linear scan with Arrow’s equality kernel. A Utf8 IN list of ten strings is a linear scan per batch.

The general lesson

The lesson is not “avoid hash tables” — it is match the data structure to the domain. A 256-element domain has a perfect data structure: a bitmap. A three-element list has a perfect data structure: three comparisons. A million-element list of 64-bit integers has a perfect data structure: a hash set. DataFusion encodes all three, plus a generic fallback, and picks between them at plan time based on the type and the length.

And the “branchless” trick generalises beyond IN: the same idea — a fixed, regular, data-independent sequence of operations that the CPU can pipeline — is what makes SIMD worthwhile at all. The | is not just shorter than ||; it is the thing that makes the comparison chain vectorisable.

No Row-by-Row Interpreter: How DataFusion's Physical Expression Tree Compiles Into Whole-Batch Arrow Kernel Calls

Jay Hubbert — Wed, 21 Oct 2026 00:29:00 +0000

TL;DR: The classic query-engine mistake is a row-at-a-time interpreter: for each row, walk the expression tree, switch on the operator, switch on the data type, compute. DataFusion does the opposite. The physical expression tree is walked once per RecordBatch, the switch on the operator happens once per node, and the work itself is delegated to Arrow’s precompiled, SIMD-vectorized kernels, which see the entire array at once. a < b is not a Rust for loop over rows — it is one call to arrow::compute::kernels::cmp::lt on two &dyn Datums.

The shape of the problem

A logical expression like

 sql
 
WHERE a + b > 100 AND c LIKE 'foo%'

has a tree of operators and leaves. Every engine has to answer: where does the work happen?

The naive design is a bytecode interpreter:

 text
 
 
for row in rows: # 1_000_000 iterations
 stack = []
 for op in bytecode: # 7 iterations
 match op:
 ADD => stack.push(stack.pop() + stack.pop())
 LT => stack.push(stack.pop() < stack.pop())
 ...

This is what every row-oriented engine did for decades, and it is catastrophically slow on modern CPUs. The inner loop is a match on a discriminant, the operands are boxed, the CPU branch predictor is doing nothing but predicting the match, and the memory access pattern is scattered across a heap-allocated value per intermediate result. None of that is vectorizable.

The fix that won is the “vectorized” or “column-at-a-time” design: don’t evaluate one row, evaluate one column. Compute the whole a + b column, then the whole > 100 column, then the whole AND. Every step is a tight, branch-free, SIMD-friendly loop over a dense buffer.

DataFusion’s actual design: a typed tree, dispatched once per batch

DataFusion’s physical expression is a trait:

 rust
 
 
// datafusion/physical-expr-common/src/physical_expr.rs:76-96
pub trait PhysicalExpr: Any + Send + Sync + Display + Debug + DynEq + DynHash {
 /// Get the data type of this expression, given the schema of the input.
 fn data_type(&self, input_schema: &Schema) -> Result<DataType> { ... }
 /// Determine whether this expression is nullable, given the schema of the input
 fn nullable(&self, input_schema: &Schema) -> Result<bool> { ... }
 /// Evaluate an expression against a RecordBatch
 fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue>;
 ...
}

The return type is ColumnarValue, which has exactly two variants:

 rust
 
 
// datafusion/expr-common/src/columnar_value.rs
pub enum ColumnarValue {
 /// Array of values
 Array(ArrayRef),
 /// A single value
 Scalar(ScalarValue),
}

That enum is the whole abstraction. A node in the tree is asked to evaluate against a batch, and it answers with either “here is the whole column” or “here is one value that is the same for every row”. There is no per-row iteration anywhere in the trait.

A leaf: the physical `Column`

The simplest leaf does almost nothing:

 rust
 
 
// datafusion/physical-expr/src/expressions/column.rs:120-124
 /// Evaluate the expression
 fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 self.bounds_check(batch.schema().as_ref())?;
 Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
 }

Note what this is not: it does not copy the column, does not slice it, does not box it into a Value. It clones an Arc — a refcount bump — and returns a borrowed view of the batch’s buffer. The physical Column is resolved to a schema index at plan time, not a name:

 rust
 
 
// datafusion/physical-expr/src/expressions/column.rs:66-72
pub struct Column {
 /// The name of the column (used for debugging and display purposes)
 name: String,
 /// The index of the column in its schema
 index: usize,
}

The name is carried along only for Display and Debug. At runtime the lookup is batch.column(self.index) — an array index.

An internal node: `BinaryExpr`

BinaryExpr is where the “compile once, dispatch once” trick becomes visible. Its evaluate looks like this:

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:561-583
 fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 use arrow::compute::kernels::numeric::*;

 if self.op == Operator::And {
 return self.evaluate_conjunction(batch);
 }

 // Evaluate left-hand side expression.
 let lhs = self.left.evaluate(batch)?;

 // Check if we can apply short-circuit evaluation.
 match check_short_circuit(&lhs, &self.op) {
 ShortCircuitStrategy::None => {}
 ShortCircuitStrategy::ReturnLeft => return Ok(lhs),
 ...

and then the meat:

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:641-683
 match self.op {
 Operator::Plus if self.fail_on_overflow => return apply(&lhs, &rhs, add),
 Operator::Plus => return apply(&lhs, &rhs, add_wrapping),
 Operator::Minus if self.fail_on_overflow => return apply(&lhs, &rhs, sub),
 Operator::Minus => return apply(&lhs, &rhs, sub_wrapping),
 Operator::Multiply if self.fail_on_overflow => return apply(&lhs, &rhs, mul),
 Operator::Multiply => return apply(&lhs, &rhs, mul_wrapping),
 Operator::Divide => return apply(&lhs, &rhs, div),
 Operator::Modulo => return apply(&lhs, &rhs, rem),

 Operator::Eq
 | Operator::NotEq
 | Operator::Lt
 | Operator::Gt
 | Operator::LtEq
 | Operator::GtEq
 | Operator::IsDistinctFrom
 | Operator::IsNotDistinctFrom
 | Operator::LikeMatch
 | Operator::ILikeMatch
 | Operator::NotLikeMatch
 | Operator::NotILikeMatch => {
 return apply_cmp(self.op, &lhs, &rhs);
 }
 _ => {}
 }

Read that carefully. The match self.op is a Rust match on an enum discriminant. The compiler turns it into a jump table, or inlines the branches. It happens once, not once per row. And each arm names an Arrow kernel function — add, sub_wrapping, div, rem — which are monomorphized over the concrete numeric type inside Arrow.

apply_cmp does the same thing for comparisons, by turning the operator into a function pointer:

 rust
 
 
// datafusion/physical-expr-common/src/datum.rs:62-98
pub fn apply_cmp(
 op: Operator,
 lhs: &ColumnarValue,
 rhs: &ColumnarValue,
) -> Result<ColumnarValue> {
 if lhs.data_type().is_nested() {
 apply_cmp_for_nested(op, lhs, rhs)
 } else {
 let f = match op {
 Operator::Eq => eq,
 Operator::NotEq => neq,
 Operator::Lt => lt,
 Operator::LtEq => lt_eq,
 Operator::Gt => gt,
 Operator::GtEq => gt_eq,
 Operator::IsDistinctFrom => distinct,
 Operator::IsNotDistinctFrom => not_distinct,

 Operator::LikeMatch => like,
 Operator::ILikeMatch => ilike,
 Operator::NotLikeMatch => nlike,
 Operator::NotILikeMatch => nilike,

 _ => {
 return internal_err!("Invalid compare operator: {}", op);
 }
 };
 ...
 apply(&lhs, &rhs, |l, r| Ok(Arc::new(f(l, r)?)))
 }
}

f is a function pointer to an Arrow kernel — arrow_ord::cmp::lt and friends, which take &dyn Datum and return a BooleanArray. Selecting the function pointer happens once per evaluation of the node; the call itself processes the whole array.

`Datum`: the scalar/array duality, resolved inside Arrow

apply is the shim that maps Arrow’s Datum kernels onto ColumnarValue:

 rust
 
 
// datafusion/physical-expr-common/src/datum.rs:38-59
pub fn apply(
 lhs: &ColumnarValue,
 rhs: &ColumnarValue,
 f: impl Fn(&dyn Datum, &dyn Datum) -> Result<ArrayRef, ArrowError>,
) -> Result<ColumnarValue> {
 match (&lhs, &rhs) {
 (ColumnarValue::Array(left), ColumnarValue::Array(right)) => {
 Ok(ColumnarValue::Array(f(&left.as_ref(), &right.as_ref())?))
 }
 (ColumnarValue::Scalar(left), ColumnarValue::Array(right)) => Ok(
 ColumnarValue::Array(f(&left.to_scalar()?, &right.as_ref())?),
 ),
 (ColumnarValue::Array(left), ColumnarValue::Scalar(right)) => Ok(
 ColumnarValue::Array(f(&left.as_ref(), &right.to_scalar()?)?),
 ),
 (ColumnarValue::Scalar(left), ColumnarValue::Scalar(right)) => {
 let array = f(&left.to_scalar()?, &right.to_scalar()?)?;
 let scalar = ScalarValue::try_from_array(array.as_ref(), 0)?;
 Ok(ColumnarValue::Scalar(scalar))
 }
 }
}

Arrow’s Datum trait is the key to the scalar/array duality. It has exactly one method:

 rust
 
 
// arrow-array-60.0.0/src/scalar.rs:79-100
pub trait Datum {
 /// Returns the value for this [`Datum`] and a boolean indicating if the value is scalar
 fn get(&self) -> (&dyn Array, bool);
}

A scalar is just a one-element array plus a flag saying “treat me as a scalar”. Arrow’s kernels then have one code path for the array case and one for the broadcast case, and they are written once, in Arrow, with SIMD intrinsics — not in DataFusion, and not per-row.

So WHERE x = 42 does not materialize a 1-million-element array of 42s. 42 becomes a Scalar, Arrow’s eq kernel sees a scalar on one side and broadcasts inside its own loop.

Where the “compilation” actually happens

The word “compile” in the title is doing real work here, but the compilation is not code generation. It is resolution of everything that can be resolved at plan time, so that the runtime path has nothing left to decide.

Type coercion is resolved by the logical planner. The physical planner explicitly refuses to do it:

 rust
 
 
// datafusion/physical-expr/src/planner.rs:274-281
 // Note that the logical planner is responsible
 // for type coercion on the arguments (e.g. if one
 // argument was originally Int32 and one was
 // Int64 they will both be coerced to Int64).
 //
 // There should be no coercion during physical
 // planning.
 binary(lhs, *op, rhs, input_schema)

So by the time a BinaryExpr exists, its two children are guaranteed to have compatible types, and the runtime match self.op arm can call add/sub/etc. without any type check, any cast, any error path. When a coercion is needed, the logical type-coercion analyzer inserts an Expr::Cast node into the logical tree (datafusion/optimizer/src/analyzer/type_coercion.rs:535-570, via BinaryTypeCoercer::new(...).get_input_types()), and the physical planner lowers that node into a physical CastExpr:

 rust
 
 
// datafusion/physical-expr/src/planner.rs:387-404
 Expr::Cast(Cast { expr, field }) => expressions::cast_with_target_field(
 create_physical_expr(expr, input_dfschema, execution_props, planning_ctx)?,
 input_schema,
 field,
 None,
 ),
 Expr::TryCast(TryCast { expr, field }) => {
 expressions::try_cast_with_target_field(
 create_physical_expr(
 expr,
 input_dfschema,
 execution_props,
 planning_ctx,
 )?,
 input_schema,
 field,
 )
 }

So the cast is a node in the tree, evaluated once per batch by the CastExpr — never a per-row type check buried inside the arithmetic kernel.

The operator-to-kernel mapping is resolved once. The match op above is a compile-time-known set of arms. Arrow’s kernels are pub fn lt(lhs: &dyn Datum, rhs: &dyn Datum) -> Result<BooleanArray, ArrowError> — inside, they downcast once and then run a monomorphized loop.

Null handling is resolved inside the kernel. lt on two Int32Arrays knows how to combine two validity bitmaps, and it does it with bitwise AND over 64-bit words, not with a per-row if is_null.

The physical tree is a different tree from the logical one

This separation is not cosmetic; it is what makes the runtime path fast.

The logical Expr enum (datafusion/expr/src/expr.rs:327+) is a planning language:

 rust
 
 
pub enum Expr {
 Alias(Alias),
 Column(Column), // named, qualified, relation-aware
 ScalarVariable(FieldRef, Vec<String>),
 Literal(ScalarValue, Option<FieldMetadata>),
 BinaryExpr(BinaryExpr),
 ...
 Like(Like),
 SimilarTo(Like),
 Between(Between),
 Case(Case),
 Cast(Cast),
 TryCast(TryCast),
 InList(InList),
 ...
}

It refers to columns by name and qualifier, may contain scalar subqueries and correlated references, and its operators are un-coerced.

The physical PhysicalExpr is a execution language. It refers to columns by index, contains no subqueries (those are resolved into separate operators), and its operators are pre-coerced. The translation happens exactly once, in create_physical_expr:

 rust
 
 
// datafusion/physical-expr/src/planner.rs:133-168
pub fn create_physical_expr(
 e: &Expr,
 input_dfschema: &DFSchema,
 execution_props: &ExecutionProps,
 planning_ctx: &PhysicalPlanningContext,
) -> Result<Arc<dyn PhysicalExpr>> {
 let input_schema = input_dfschema.as_arrow();

 match e {
 ...
 Expr::Column(c) => {
 let idx = input_dfschema.index_of_column(c)?;
 Ok(Arc::new(Column::new(&c.name, idx)))
 }
 Expr::Literal(value, metadata) => Ok(Arc::new(Literal::new_with_metadata(
 value.clone(),
 metadata.clone(),
 ))),

That single function is the entire “compiler”. It walks the logical tree once and produces a tree of Arc<dyn PhysicalExpr> where:

Expr::Column (name + qualifier) becomes Column { name, index } — a resolved slot.

Expr::Literal becomes Literal, whose evaluate is a constant:

 rust
 
 
// datafusion/physical-expr/src/expressions/literal.rs:106-108
 fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
 Ok(ColumnarValue::Scalar(self.value.clone()))
 }

A literal never allocates, never materializes a column. WHERE x = 42 hands 42 to Arrow as a scalar.

Expr::BinaryExpr becomes BinaryExpr, with coercion already applied.

Because the physical tree is a tree of trait objects (Arc<dyn PhysicalExpr>), the runtime walk is a virtual call per node per batch. For a 7-node expression on a 8192-row batch, that is 7 virtual calls for 8192 rows — a per-batch cost, not a per-row cost. This is the crucial distinction from the bytecode interpreter, and it is why the design scales.

Specialization at the node level, not just at the batch level

Once you’re evaluating whole arrays, a whole class of optimizations opens up that are impossible in a row interpreter.

Short-circuiting on the shape of the result, not row-by-row. BinaryExpr::evaluate inspects the boolean LHS array and, if the RHS is not needed at all, skips evaluating it entirely:

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:1285-1316
 ColumnarValue::Array(array) => {
 // Check Boolean arrays directly.
 if let Ok(bool_array) = as_boolean_array(array) {
 // NULLs still need the RHS.
 if bool_array.null_count() > 0 {
 return ShortCircuitStrategy::None;
 }

 let len = bool_array.len();
 if len == 0 {
 return ShortCircuitStrategy::None;
 }

 let true_count = bool_array.values().count_set_bits();
 if true_count == len {
 return ShortCircuitStrategy::ReturnLeft;
 }

 if true_count == 0 {
 return ShortCircuitStrategy::ReturnRight;
 }

 let false_count = len - true_count;
 if false_count as f32 / len as f32 <= PRE_SELECTION_THRESHOLD {
 // Only false rows need the RHS; the LHS has no NULLs here.
 let mask = BooleanArray::new(!bool_array.values(), None);
 return ShortCircuitStrategy::PreSelection {
 mask,
 fill_value: true,
 };
 }
 }
 }

count_set_bits() on the validity/values buffer tells you, in O(n/64) word operations, whether the whole column is true, all false, or mostly false. In the last case, DataFusion filters the batch down to just the false rows, evaluates the RHS on that smaller batch, and scatters the result back — trading a full-width RHS evaluation for a filtered one.

Nested AND flattening. evaluate_conjunction walks a chain of ANDs and filters between conjuncts, so a cheap first conjunct prunes the input for an expensive second one:

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:1210-1254
 /// Evaluates nested `AND`s together to avoid repeated filtering (#25035).
 ///
 /// When at most [`PRE_SELECTION_THRESHOLD`] of original rows remain,
 /// filters before conjuncts that may be costly or fail, so they never see
 /// rows nested evaluation would skip. `NULL` rows stay for a later `false`.
 fn evaluate_conjunction(&mut self, batch: &RecordBatch) -> Result<ColumnarValue> {

Array-vs-scalar kernel specialization. Some operators have a dedicated Arrow kernel for the array-scalar case that is materially faster than the general array-array one (regex matching, bitwise ops):

 rust
 
 
// datafusion/physical-expr/src/expressions/binary.rs:1142-1163
 fn evaluate_array_scalar(
 &self,
 array: &dyn Array,
 scalar: ScalarValue,
 ) -> Result<Option<Result<ArrayRef>>> {
 use Operator::*;
 let scalar_result = match &self.op {
 RegexMatch => regex_match_dyn_scalar(array, &scalar, false, false),
 RegexIMatch => regex_match_dyn_scalar(array, &scalar, false, true),
 ...
 // if scalar operation is not supported - fallback to array implementation
 _ => None,
 };

 Ok(scalar_result)
 }

Selection vectors instead of filtering. PhysicalExpr::evaluate_selection is a trait method that evaluates an expression only on the rows selected by a boolean mask, and scatters the results back:

 rust
 
 
// datafusion/physical-expr-common/src/physical_expr.rs:121-180
 fn evaluate_selection(
 &self,
 batch: &RecordBatch,
 selection: &BooleanArray,
 ) -> Result<ColumnarValue> {
 let row_count = batch.num_rows();
 ...
 // First, check if we can avoid filtering altogether.
 if selection.null_count() == 0 && !selection.has_false() {
 // All values from the `selection` filter are true and match the input batch.
 // No need to perform any filtering.
 return self.evaluate(batch);
 }

 // Next, prepare the result array for each 'true' row in the selection vector.
 let filtered_result = if !selection.has_true() {
 // Do not call `evaluate` when the selection is empty.
 // `evaluate_selection` is used to conditionally evaluate expressions.
 // When the expression in question is fallible, evaluating it with an empty
 // record batch may trigger a runtime error (e.g. division by zero).
 ...
 ColumnarValue::Array(new_empty_array(&datatype))
 } else {
 // If we reach this point, there's no other option than to filter the batch.
 // This is a fairly costly operation since it requires creating partial copies
 // (worst case of length `row_count - 1`) of all the arrays in the record batch.
 let filtered_batch = filter_record_batch(batch, selection)?;
 self.evaluate(&filtered_batch)?
 };

 // Finally, scatter the filtered result array so that the indices match the input rows again.
 match &filtered_result {
 ColumnarValue::Array(a) => {
 scatter(selection, a.as_ref()).map(ColumnarValue::Array)
 }
 ...

This is the vectorized analogue of “don’t evaluate 1/0 for rows that will be discarded”. A row interpreter would need a per-row branch; a columnar evaluator filters the batch, evaluates the cheap subset, and scatters.

CASE becomes a sequence of filtered batches. The CaseExpr evaluates one branch at a time on the remaining rows, which shrink as branches match:

 rust
 
 
// datafusion/physical-expr/src/expressions/case.rs:864-890
 // If the 'when' predicate did not match any rows, continue to the next branch immediately.
 // Only counts valid slots that are true (masked-null predicate slots are ignored),
 // so no `prep_null_mask_filter` needed here.
 if !when_value.has_true() {
 continue;
 }

 // If the 'when' predicate matched all remaining rows, there is no need to filter
 if when_value.null_count() == 0 && !when_value.has_false() {
 let then_expression = &self.when_then_expr[i].1;
 let then_value = then_expression.evaluate(&remainder_batch)?;
 result_builder.add_branch_result(&remainder_rows, then_value)?;
 return result_builder.finish();
 }

 // Filter the remainder batch based on the 'when' value
 // This results in a batch containing only the rows that need to be evaluated
 // for the current branch

There is even a fully-memoized specialization: CASE WHEN <literal> THEN <literal> ... becomes a lookup table over the values (datafusion/physical-expr/src/expressions/case/literal_lookup_table/).

Trade-offs and failure modes

The virtual call per node per batch is not free. For a batch of 8192 rows and a 20-node expression, that’s 20 dynamic dispatches — utterly negligible. But for tiny batches (say 8 rows) the fixed overhead of the walk can dominate. DataFusion mitigates this by coalescing small batches upstream (see the batch-merging article in this series), which raises the row count per evaluation.
The tree must be re-planned for schema changes. Because Column is an index, inserting a column into the batch invalidates the whole physical expression tree. with_new_schema in datafusion/physical-expr/src/expressions/column.rs:246-266 exists precisely to re-index the tree, and it has to be called at every point where the input schema shifts (projections, joins). The alternative — name-based lookup at runtime — would be catastrophically slower.
Filters copy data. filter_record_batch makes partial copies of every column in the batch, as the doc comment above says: “worst case of length row_count - 1”. This is the cost of the “evaluate fewer rows” optimizations. DataFusion guards it with the PRE_SELECTION_THRESHOLD (20%) check so it only filters when it is clearly worth it, and with is_cheap_and_infallible to avoid filtering before a conjunct that cannot fail.

ColumnarValue::Scalar is a footgun. The doc on into_array says plainly that converting a scalar to an array “is not as efficient as handling the scalar value directly”. A UDF that eagerly calls into_array on all its arguments throws away the entire scalar broadcast optimization. The ScalarFunctionExpr wrapper at least checks the result length:

 rust
 
 
// datafusion/physical-expr/src/scalar_function.rs:263-279
 if let ColumnarValue::Array(array) = &output
 && array.len() != batch.num_rows()
 {
 // If the arguments are a non-empty slice of scalar values, we can assume that
 // returning a one-element array is equivalent to returning a scalar.
 let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
 return if preserve_scalar {
 ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
 } else {
 internal_err!(...)
 };
 }

Kernel coverage gaps leak into the physical layer. time ± interval returns a wrapped time (PostgreSQL/DuckDB semantics) which Arrow’s arithmetic kernels do not implement, so DataFusion has to implement it itself in apply_time_interval (datafusion/physical-expr/src/expressions/binary.rs:395-433). Every such gap is a hand-written kernel that does not get Arrow’s SIMD treatment, and it is a maintenance liability.
The physical tree is not the logical tree. Anyone reasoning about “why is this query slow” has to remember that the expression they wrote is not the expression that runs. The logical planner may have rewritten it, the physical planner may have inserted casts, and the physical expression tree may have been re-indexed. This is a genuine source of confusion, and it’s why DataFusion has fmt_sql on PhysicalExpr so you can print the thing that will actually execute.

The takeaway

DataFusion’s expression engine is not an interpreter, and it is not a compiler either. It is a typed tree of node objects, walked once per batch, where each node resolves its operator to an Arrow kernel and hands over whole arrays. All the “deciding” work — name resolution, type coercion, operator selection, null semantics, SIMD details — is pushed either to plan time or into Arrow’s precompiled kernels, so that the only thing left at runtime is a handful of virtual calls and a straight-line call into a vectorized function. The result is that the per-row cost of expression interpretation is essentially zero, and all the time is spent in the kernels doing actual arithmetic on dense buffers.

Arrow on Opensource internals

Polars Shares Data With Python Over the Arrow C Data Interface

The three structs

Exporting: no bytes move

The one thing that is copied: alignment

Importing: borrowing, and keeping the owner alive

The stream interface

The Python side

The pyarrow bridge: to_arrow

The older protocol: polars-ffi

The bottom line

Polars Encodes Group Keys as Rows to Make Hashing Cheap

Encoding a row

Strings are the awkward case

Hashing the bytes

The map that stores the groups

When no encoding is needed

Key rows: the fixed-width version

The bottom line

Polars Groups Columns So One Pass Feeds Many Aggregations

Two grouping engines, not one

The Grouper trait

Four encodings, one interface

Single key: the values themselves

Key rows: a hand-rolled row format

Row encoding and binview: the fallbacks

The keys themselves

One pass, many aggregations

The hot table

The bottom line

Nulls Are Not Values: How DataFusion Skips Work Using Arrow's Validity Bitmaps

The representation

The “all null / all valid / some” trichotomy

The might_have_nulls trick: a conservative bitmap

Boolean filters: nulls as false, at the bitmap level

IN lists: computing nullability as a bitmap expression

Filters and selection vectors

Where the physical Column fits in

Propagating a parent’s nulls to its children

The reverse operation: scatter

Slicing a bitmap is not free

Where the separation leaks: logical vs physical nulls

Failure modes and trade-offs

The takeaway

Polars Evaluates Expressions in an Arena

The shape you write: a tree of Arc<Expr>

The arena

The IR node: AExpr

The handle: ExprIR

From Expr to ExprIR

Lowering to physical expressions

The dispatch table

Where the dyn actually is

What the arena buys, concretely

Caching the arena across calls

The bottom line

Polars Shares Columns Instead of Copying Them: Arc, Bytes and Reference Counting

Layer 1: the Series is an Arc

Layer 2: the bytes, and who owns them

Layer 3: the refcount itself

Slicing: the payoff

The FFI: importing and exporting without copying

Export

Import

What copies, and what does not

Un-sharing on purpose: deshare and gc

The bottom line

Polars Stores Short Strings Inline in 16 Bytes: The View Type

The type

The exact bit layout

Building an inline view

Spilling: the prefix and the buffer index

Where the spilled bytes live

How the buffers grow

Reading a string back

Comparing and hashing

Filling views at SIMD speed

A second, smaller inline type

The bottom line

The Arrow Row Format: How DataFusion Packs a Multi-Column Sort Key Into One Comparable Byte String

The pyarrow bridge: `to_arrow`

The older protocol: `polars-ffi`

The `Grouper` trait

The `might_have_nulls` trick: a conservative bitmap

`IN` lists: computing nullability as a bitmap expression

Where the physical `Column` fits in

The shape you write: a tree of `Arc<Expr>`

The IR node: `AExpr`

The handle: `ExprIR`

From `Expr` to `ExprIR`

Where the `dyn` actually is

Layer 1: the Series is an `Arc`

Un-sharing on purpose: `deshare` and `gc`

The refcount, copied from `Arc<T>`

The cross-batch cache: `sync_value_cache`

A leaf: the physical `Column`

An internal node: `BinaryExpr`

`Datum`: the scalar/array duality, resolved inside Arrow