Polars on Opensource internals

A Tour of Polars' Internals: The Twelve Ideas That Explain the Engine

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: Polars is a stack of small ideas, and almost every performance question you have about Polars can be answered by naming one of them. Chunks are the unit of parallelism. View is the unit of string storage. The arena is the unit of expression allocation. The optimizer is a rewrite engine, not a cost model. The streaming engine is a pipeline of morsels. This is a tour of twelve of them, and where in the source tree each one lives.

pl.LazyFrame(...).explain() prints a plan, and the plan looks like a tree of node names — SLICE, SELECT, SORT, AGGREGATE. It reads like the output of a query planner in any other engine. It is not. Polars’ plan is closer to a compilation unit: a tree that gets rewritten several times before a single byte of data moves, and then executed by one of two quite different runtimes.

The best way to learn the engine is to ignore the node names and ask, for each one, what is the trick underneath it. Here is that list.

1. The DSL, the IR, and the fact that there are two of each

The user-facing plan is DslPlan, a tree of Arc<DslPlan> nodes with Expr leaves — user-level expressions, cheap to clone, awkward to analyze.

 rust
 
 
// crates/polars-plan/src/dsl/plan.rs:32
pub enum DslPlan {
 #[cfg(feature = "python")]
 PythonScan {
 options: crate::dsl::python_dsl::PythonOptionsDsl,
 },
 /// Filter on a boolean mask
 Filter {
 input: Arc<DslPlan>,
 predicate: Expr,
 },

crates/polars-plan/src/dsl/plan.rs:32

The optimizer never touches that tree. It works on IR, where every node lives in an arena and every expression is an ExprIR — a resolved expression with a dtype, no user-facing sugar left in it.

 rust
 
 
// crates/polars-plan/src/plans/ir/mod.rs:66
 Filter {
 input: Node,
 predicate: ExprIR,
 },

crates/polars-plan/src/plans/ir/mod.rs:66

The split matters for a reason that is easy to miss: the DSL tree can be rebuilt cheaply and structurally, while the IR is what the executors see. All the interesting analysis — predicate pushdown, projection pushdown, simplification, common-subplan elimination — happens on the IR, where the engine can ask questions like “is this expression already known to be non-null?” and get a real answer instead of a pattern match on syntax.

The arena itself is a Vec<T> plus integer handles, which is about as cheap as it gets:

 rust
 
 
// crates/polars-utils/src/arena.rs:38
pub struct Arena<T> {
 version: u32,
 items: Vec<T>,
}

crates/polars-utils/src/arena.rs:38

More on this in Polars Evaluates Expressions in an Arena.

2. Two engines, and a switch between them

collect() does not pick an engine by measuring anything. It picks one by looking at flags:

 rust
 
 
// crates/polars-lazy/src/frame/mod.rs:636
 pub fn collect_with_engine(mut self, engine: Engine) -> PolarsResult<QueryResult> {
 let engine = match engine {
 Engine::Streaming => Engine::Streaming,
 _ if std::env::var("POLARS_FORCE_STREAMING").as_deref() == Ok("1") => Engine::Streaming,
 Engine::Auto => {
 if self.opt_state.eager() {
 Engine::InMemory
 } else {
 Engine::Streaming
 }
 },

crates/polars-lazy/src/frame/mod.rs:636

Both engines consume the same optimized IR. The in-memory engine (polars-mem-engine) builds a tree of Box<dyn Executor> nodes and runs it depth-first with rayon fan-out; the streaming engine (polars-stream) lowers the same IR into a graph of nodes connected by pipes and runs it as a dataflow pipeline.

This is the single most interesting architectural fact in the codebase: the IR is the contract, and two very different runtimes honour it. A filter is a closure in one and a pipe in the other.

3. Chunks are the unit of parallelism

A ChunkedArray is a vector of Arrow arrays plus a length. Every kernel walks it chunk by chunk, which means the chunk boundaries are simultaneously the unit of work, the unit of parallelism, and the granularity at which validity and offsets are tracked.

 rust
 
 
// crates/polars-core/src/chunked_array/mod.rs:140
pub struct ChunkedArray<T: PolarsDataType> {
 pub(crate) field: Arc<Field>,
 pub(crate) chunks: Vec<ArrayRef>,

 pub(crate) flags: StatisticsFlagsIM,

 length: usize,
 null_count: usize,
 _pd: std::marker::PhantomData<T>,
}

crates/polars-core/src/chunked_array/mod.rs:140

Appending to a ChunkedArray is free — you push another ArrayRef into the Vec. Random access costs an extra indirection, which is why the engine prefers chunk-local work and only pays for a global index when it has to.

Full treatment: Polars Chunks Its Columns So Parallelism Has Something to Eat.

4. `View`: a string that fits in a register

The most copied type in the engine is not a String. It is the 16-byte View:

 rust
 
 
// crates/polars-arrow/src/array/binview/view.rs:20
#[derive(Copy, Clone, Default)]
#[repr(C)]
pub struct View {
 /// The length of the string/bytes.
 pub length: u32,
 /// First 4 bytes of string/bytes data.
 pub prefix: u32,
 /// The buffer index.
 pub buffer_idx: u32,
 /// The offset into the buffer.
 pub offset: u32,
}

crates/polars-arrow/src/array/binview/view.rs:20

A string of twelve bytes or fewer is stored inside the view. Anything longer spills into a shared buffer and the view keeps an index and an offset. Because the view is 16 bytes, comparing two of them is two integer compares before any string data is touched — the length rejects mismatches for free, and the four-byte prefix rejects most of the rest.

Full treatment: Polars Stores Short Strings Inline in 16 Bytes: The View Type and Polars Compares Strings Without Allocating.

5. Immutable, refcounted buffers

There is no Vec<u8> underneath a column. There is a SharedStorage<T> with an AtomicU64 refcount and a hand-written vtable for the drop glue, and the only mutation path is copy-on-write through a “make this uniquely owned” step. That is what makes zero-copy sharing between a DataFrame, a Python object and an Arrow C stream cheap: they all point at the same bytes.

Full treatment: Polars Buffers Are Immutable: The Refcounted Storage Behind Every Column and Polars Shares Columns Instead of Copying Them.

6. The optimizer is a rewrite engine

Polars’ optimizer is a list of rewrite rules applied to fixed points. There is no cost model, no cardinality estimation driving plan choices, no “which join order is cheaper” search. A rule that fires changes the tree; the rules run again; if nothing changes, the plan is done.

That design has an obvious consequence — Polars will not reorder a join for you — and a less obvious one: because the rules are local and structural, they can be extremely aggressive about shape. A select that is really just a projection becomes a SimpleProjection. A duplicated subplan becomes a Cache node. A filter that is provably always-true disappears.

Full treatment: Polars Simplifies the Expression Tree Before It Runs Anything and Polars Kills Duplicate Subplans by Fingerprinting Them.

7. Predicate pushdown ends at the file footer

Pushdown in Polars is not one feature but three layers: the optimizer moves the filter into the scan node, the scan converts the predicate into per-row-group statistics checks, and the reader skips row groups whose min/max cannot satisfy it.

The conversion is the interesting part, because the filter has to be compiled twice: once against the real columns, and once against a synthetic frame of col_min/col_max columns so the same expression code can be reused to test statistics.

Full treatment: Polars Pushes Predicates Into the Parquet Scanner and Polars Prunes Parquet Row Groups With Chunk Statistics.

8. Group keys have four different encodings

There is no single “group by” implementation. A single integer key groups by comparing integers. A single string key groups by dictionary codes. Multiple keys are encoded into rows — fixed-width byte blobs with null bits — so that equality becomes a memcmp and the hash is the hash of the row bytes.

Four encodings, four code paths, chosen by the shape of the key. The trick that makes the whole family fast is that the encoding is also the hashing input: there is no separate hash function per key type, there is one hash over the encoded bytes.

Full treatment: Polars Encodes Group Keys as Rows to Make Hashing Cheap and Polars Groups Columns So One Pass Feeds Many Aggregations.

9. Sortedness is metadata, and it changes the algorithm

Polars tracks whether a column is sorted as a flag on the chunked array, and the optimizer propagates that through the plan. A group-by on a sorted key does not hash: it walks runs. A merge join only accepts sorted inputs. A top-k on a sorted stream can stop early.

This is the cheapest possible form of query planning — a boolean that unlocks a completely different algorithm — and it is why sort().group_by() can be dramatically faster than group_by().

Full treatment: Polars Aggregates Faster When the Keys Are Already Sorted and Polars Joins Two Tables with a Piecewise Merge Join.

10. The streaming engine is a pipeline of morsels

The streaming engine never materialises a whole partition. A source node emits morsels — fixed-size blocks of rows — and downstream nodes consume them. The pipe between two nodes is a bounded channel, so backpressure is structural: a slow consumer stops the producer.

 rust
 
 
// crates/polars-stream/src/morsel.rs:83
pub struct Morsel {
 /// The data contained in this morsel.
 sf: SpillFrame,

 /// The sequence number of this morsel. May only stay equal or increase
 /// within a pipeline.
 seq: MorselSeq,

crates/polars-stream/src/morsel.rs:83

When a stage has to repartition — a hash join, a group-by — the rows are shuffled: hashed into N partitions, buffered, and exchanged. The hash is a multiply-shift, and the partitioning is a counting sort, not a sort.

Full treatment: Polars’ New Streaming Engine Works in Morsels and Polars Hash-Shuffles Rows Between Streaming Stages.

11. The runtime filter: a join that tells the scan what to skip

The newest trick in the tree is the runtime filter. When a hash join builds its table on one side, the values of the build keys are known — and can be published as a predicate that the probe-side scan applies. A scan that would have read a whole partition can now skip row groups whose statistics cannot match the build key range, or check a bloom filter of the build keys before decoding anything.

It is a join and a scan cooperating through a slot in the plan that is filled at runtime.

Full treatment: Polars Publishes a Join’s Build Side as a Runtime Filter.

12. Python is a guest, not the host

The Python bindings do not own the data. pl.DataFrame wraps a Rust DataFrame; handing it to pandas or Arrow exports it through the Arrow C data interface, which is a set of pointers plus a release callback. The GIL is dropped for the duration of a query and re-taken only where Python code actually runs — inside a UDF.

Full treatment: Polars Shares Data With Python Over the Arrow C Data Interface and Polars Releases the GIL Before It Does Any Real Work.

The bottom line

If you want the mental model in one paragraph: Polars represents data as immutable, chunked, refcounted Arrow buffers with 16-byte inline strings; it compiles a user program into an arena-allocated IR through a stack of structural rewrite rules; and it executes that IR with one of two runtimes, one of which streams morsels through a graph of pipes with hash shuffles at the boundaries.

Almost every performance knob in Polars follows from those three sentences. A columnar layout is what makes the predicate-pushdown and statistics pruning possible. An arena is what makes the optimizer cheap enough to run aggressively. Morsels are what make the streaming engine’s memory usage independent of the input size.

The articles in this series take each of those ideas apart. If you only read one, read Polars’ New Streaming Engine Works in Morsels — it is the newest part of the engine and the least documented.

Polars Spills to Disk Before It Runs Out of Memory

Jay Hubbert — Wed, 21 Oct 2026 08:50:00 +0000

TL;DR: Polars does not measure the memory a particular operator uses. It counts every allocation the process makes with a thin wrapper around the global allocator, and compares that number against a budget (80% of RAM by default). Nodes in the streaming engine register their output frames as “spillable” with a spill context. When the global counter crosses the budget, a central memory manager picks candidates, encodes them as Arrow IPC streams, and writes them to files under a per-process spill directory. A second allocator-level threshold, off by default, aborts the process before the Linux OOM killer ever sees it.

The brief I was given for this article pointed at crates/polars-ooc/src/spill.rs, write.rs, reader.rs, and an OocS3 spill manager. Those files do not exist at this commit (b413f2b653). What exists is crates/polars-ooc/src/ with global_alloc.rs, memory_manager.rs, spill_context/, spill_file.rs, spill_frame.rs and spill_token.rs. There is no S3 backend: the only spill target is the local filesystem, and the “reader” and “writer” are two functions inside spill_frame.rs. So this is what is really there.

One counter for the whole process

The first thing to understand is that Polars’ out-of-core machinery is built on a global memory estimate, not on per-operator accounting. There is a GlobalAlloc implementation in crates/polars-ooc/src/global_alloc.rs that wraps jemalloc (or mimalloc) and keeps a running byte total:

 rust
 
 
// crates/polars-ooc/src/global_alloc.rs:5
static GLOBAL_ALLOC_SIZE: AtomicU64 = AtomicU64::new(0);

/// Returns an estimate of the total amount of bytes allocated.
///
/// This can be up to OOC_DRIFT_THRESHOLD * num_threads bytes less than or
/// greater than the true memory usage.
pub fn estimate_memory_usage() -> u64 {
 let bytes = GLOBAL_ALLOC_SIZE.load(Ordering::Relaxed);
 if bytes > i64::MAX as u64 {
 // Drift + moving allocations between threads allows for underflow,
 // so this is best reported as zero.
 0
 } else {
 bytes
 }
}

crates/polars-ooc/src/global_alloc.rs:5-20

The counter is not bumped on every alloc directly. Each thread accumulates into a thread-local drift counter, and only flushes to the global atomic when the drift exceeds POLARS_OOC_DRIFT_THRESHOLD (4 MiB by default):

 rust
 
 
// crates/polars-ooc/src/global_alloc.rs:28
#[inline(always)]
fn update_alloc_size(bytes: i64) {
 LOCAL_ALLOC_DRIFT.with(|drift| {
 let new = drift.get().wrapping_add(bytes);
 drift.set(new);
 if new.unsigned_abs() > polars_config::get_ooc_drift_threshold() {
 flush_local_drift();
 }
 })
}

#[cold]
#[inline(never)]
fn flush_local_drift() {
 LOCAL_ALLOC_DRIFT.with(|drift| {
 let drift_bytes = drift.get();
 let old = GLOBAL_ALLOC_SIZE.fetch_add(drift_bytes as u64, Ordering::AcqRel);
 drift.set(0);

 let total = old.wrapping_add(drift_bytes as u64);
 if total <= i64::MAX as u64 && total > polars_config::get_oomkill_threshold_bytes() {
 oomkill();
 }
 })
}

crates/polars-ooc/src/global_alloc.rs:28-52

This is the same batching trick every high-performance allocator-backed counter uses: one atomic fetch_add per 4 MiB per thread instead of one per malloc. The cost is that the estimate can be off by up to DRIFT_THRESHOLD * num_threads in either direction, which the doc comment states plainly.

The Allocator itself is four lines of arithmetic wrapped around the underlying allocator:

 rust
 
 
// crates/polars-ooc/src/global_alloc.rs:81
pub struct Allocator;

unsafe impl GlobalAlloc for Allocator {
 unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
 update_alloc_size(layout.size() as i64);
 unsafe { UNDERLYING_ALLOC.alloc(layout) }
 }

 unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
 update_alloc_size(layout.size() as i64);
 unsafe { UNDERLYING_ALLOC.alloc_zeroed(layout) }
 }

 unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
 update_alloc_size(-(layout.size() as i64));
 unsafe { UNDERLYING_ALLOC.dealloc(ptr, layout) }
 }

 unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
 update_alloc_size(new_size as i64 - layout.size() as i64);
 unsafe { UNDERLYING_ALLOC.realloc(ptr, layout, new_size) }
 }
}

crates/polars-ooc/src/global_alloc.rs:81-103

In the Python build this is installed as the process-wide allocator:

 rust
 
 
// crates/polars-python/src/c_api/allocator.rs:7
#[global_allocator]
static ALLOC: polars_ooc::Allocator = polars_ooc::Allocator;

crates/polars-python/src/c_api/allocator.rs:7-8

That last detail has real consequences. Because the allocator is process-global, the estimate includes everything the Python interpreter allocates — your list of strings, the dataframe you already materialized, the interpreter’s own bookkeeping. So the “Polars memory usage” number that drives spilling is really “this process’s memory usage”. Spilling can be triggered by allocations Polars never made. That is a deliberate trade: a single counter is cheap and always available, whereas per-operator accounting would need every allocation site to be instrumented.

The budget

The budget is derived from total system memory, with a cgroup-aware helper (sys.cgroup_limits()) so containers get the right number:

 rust
 
 
// crates/polars-config/src/lib.rs:384
let budget_frac = f64::from_bits(self.ooc_memory_budget_fraction.load(Ordering::Relaxed));
let bytes = u64::min(
 self.ooc_memory_budget_bytes.load(Ordering::Relaxed),
 (total_memory() as f64 * budget_frac) as u64,
);
self.ooc_effective_memory_budget_bytes
 .store(bytes, Ordering::Relaxed);
let frac = f64::from_bits(self.ooc_memory_prefetch_fraction.load(Ordering::Relaxed));
self.ooc_memory_prefetch_bytes
 .store((bytes as f64 * frac) as u64, Ordering::Relaxed);

crates/polars-config/src/lib.rs:384-393

The default fraction is 0.8 and the default absolute cap is u64::MAX (POLARS_OOC_MEMORY_BUDGET_MB), so the effective budget is 80% of RAM unless you set the absolute value. The prefetch threshold is 90% of the budget, not of RAM — the gap between the two is the hysteresis band the engine uses to decide between spilling more and reading some back.

There is a separate, much stricter threshold for the hard abort:

 rust
 
 
// crates/polars-config/src/lib.rs:395
let oomkill_frac = f64::from_bits(self.oomkill_threshold_fraction.load(Ordering::Relaxed));
let oomkill_frac_bytes = if oomkill_frac.is_finite() {
 (total_memory() as f64 * oomkill_frac) as u64
} else {
 u64::MAX
};
let oomkill_mb_bytes = self.oomkill_threshold_bytes.load(Ordering::Relaxed);
OOMKILL_THRESHOLD_BYTES_ATOMIC.store(
 u64::min(oomkill_mb_bytes, oomkill_frac_bytes),
 Ordering::Relaxed,
);

crates/polars-config/src/lib.rs:395-404

Both POLARS_OOMKILL_THRESHOLD_FRACTION (default f64::INFINITY) and POLARS_OOMKILL_THRESHOLD_MB (default u64::MAX) are effectively unlimited out of the box, so the abort path is opt-in. When it does fire, it is deliberately brutal — it cannot allocate, so it writes straight to fd 2 and calls libc::abort():

 rust
 
 
// crates/polars-ooc/src/global_alloc.rs:54
#[cold]
#[inline(never)]
fn oomkill() -> ! {
 // Can't allocate here, so write directly to stderr.
 const MSG: &[u8] = b"polars: memory usage exceeded POLARS_OOMKILL_THRESHOLD_MB or POLARS_OOMKILL_THRESHOLD_FRACTION, aborting\n";
 unsafe {
 libc::write(2, MSG.as_ptr().cast(), MSG.len() as _);
 libc::abort()
 }
}

crates/polars-ooc/src/global_alloc.rs:54-63

abort() rather than a panic, because at that point a panic might need to allocate to format its message, and the whole point is that we are out of memory. Note also that the abort is checked only in flush_local_drift, so it fires at most once per 4 MiB of allocation churn — it is a sampling point, not an exact tripwire.

Making frames spillable

Now the other half. Nodes do not decide to spill; they make their output eligible for spilling and let a global manager decide.

The unit of spilling is a SpillToken<T>, a hand-rolled state machine over a single AtomicU64. The bits are:

 rust
 
 
// crates/polars-ooc/src/spill_token.rs:17
// SpillTokenInner's state
const SPILLED_BIT: u64 = 1; // Set when value = None and spilled is Some.
const DROPPED_BIT: u64 = 2; // Set when the token owner has dropped. Forbids creation of new pins / lock.
const LOCK_BIT: u64 = 4; // When set no new pins may be made, and at most 1 thread may set this bit.
const HAS_WAITERS_BIT: u64 = 8; // Only updated while holding waiters lock, allows checking for waiters without locking.
const REINSERT_WHEN_SPILLABLE_BIT: u64 = 16; // When set the SpillToken should insert into its registered context once spillable.
const RO_PIN_COUNT_UNIT: u64 = 32; // Added to the state for each active pin.
const RO_PIN_MASK: u64 = u64::MAX << 5;

crates/polars-ooc/src/spill_token.rs:17-24

A “pin” is a live reference to the in-memory value. Pin count lives in the upper bits of the same word as the flags, so acquiring a pin is one compare_exchange. Spilling requires upgrading a solo pin into the exclusive lock — the transition that guarantees nobody is reading the buffer while it is being encoded:

 rust
 
 
// crates/polars-ooc/src/spill_token.rs:715
// Try to upgrade solo pin into exclusive lock.
let is_exclusive = slf
 .state
 .try_update(Ordering::Acquire, Ordering::Acquire, |s| {
 if s & (RO_PIN_MASK | LOCK_BIT) == RO_PIN_COUNT_UNIT {
 Some(s - RO_PIN_COUNT_UNIT + LOCK_BIT)
 } else {
 None
 }
 })
 .is_ok();

crates/polars-ooc/src/spill_token.rs:715-725

If the upgrade fails, the spill is abandoned (add_failed_spill) and the frame stays in memory. That failure rate is not incidental: it is measured and fed back into the bandit that picks candidates.

Who decides

Nodes register their frames with a spill context. There are three context flavours — MostRecentSpillContext, LeastRecentSpillContext and RandomSpillContext — differing only in the order in which candidates are popped from the queue. A context is a name plus a policy plus a statistics object:

 rust
 
 
// crates/polars-ooc/src/spill_context/mod.rs:530
/// A context that spills the most-recently registered spillable when asked.
#[derive(Clone)]
#[repr(transparent)]
pub struct MostRecentSpillContext(StrongSpillContext);

impl MostRecentSpillContext {
 pub fn new(name: PlSmallStr, task_metrics: Option<Arc<TaskMetricAggregator>>) -> Self {
 Self(StrongSpillContext::new(
 name,
 SpillContextPolicy::MostRecent,
 task_metrics,
 ))
 }
}

crates/polars-ooc/src/spill_context/mod.rs:530-543

The register call is where the global memory check happens. Note that there are two variants: register_no_spill_check (synchronous, no global check) and register (async, checks):

 rust
 
 
// crates/polars-ooc/src/spill_context/mod.rs:512
pub trait ParameterFreeSpillContext {
 fn register_no_spill_check<T, S>(&self, token: &T)
 where
 T: AsRef<SpillToken<S>>,
 S: Spillable,
 Self: Sized;

 fn register<T, S>(&self, token: &T) -> impl Future<Output = ()>
 where
 T: AsRef<SpillToken<S>>,
 S: Spillable,
 Self: Sized,
 {
 self.register_no_spill_check(token);
 memory_manager().spill()
 }
}

crates/polars-ooc/src/spill_context/mod.rs:512-528

So registering a frame is the only trigger. The engine never asks “does this node exceed its share?”; it asks “are we over budget globally?” and if so, spills whatever is registered and eligible. The global memory manager is the only thing that reads the allocator counter:

 rust
 
 
// crates/polars-ooc/src/memory_manager.rs:61
fn should_spill(&self) -> bool {
 let usage = crate::estimate_memory_usage();
 let likely_dealt_with = self.est_spill_in_progress.load(Ordering::Relaxed);
 let likely_incoming = self.est_prefetch_in_progress.load(Ordering::Relaxed);
 (usage + likely_incoming).saturating_sub(likely_dealt_with)
 > config().ooc_memory_budget_bytes()
}

pub(crate) fn should_prefetch(&self) -> bool {
 if !self.spills_exist.load(Ordering::Acquire) {
 return false;
 }

 let usage = crate::estimate_memory_usage();
 let likely_incoming = self.est_prefetch_in_progress.load(Ordering::Relaxed);
 (usage + likely_incoming) < config().ooc_memory_prefetch_bytes()
}

crates/polars-ooc/src/memory_manager.rs:61-77

The subtraction of est_spill_in_progress matters: bytes that are already being written to disk are still allocated (the encoded buffer lives in RAM until the write completes), so counting them as “not yet freed” would make the manager spill harder the more it spills. The estimate of in-flight spill bytes is tracked by a SpillInProgressTracker whose Drop decrements the counter — a RAII guard, so it cannot leak even if the spill task is cancelled.

Picking a victim

Candidate selection is a multi-armed bandit over spill contexts, not a heuristic. Each context keeps discounted statistics of its own historical performance and is sampled via Thompson sampling:

 rust
 
 
// crates/polars-ooc/src/memory_manager.rs:289
// Thompson sampling.
let score_sample = ctx.0.stats().sample_spill_score(&mut rng);
assert!(!score_sample.is_nan());
live_contexts.push((ctx.clone(), score_sample));

crates/polars-ooc/src/memory_manager.rs:289-292

The score itself is a ratio of relief to IO time — byte-seconds freed per second spent doing I/O:

 rust
 
 
// crates/polars-ooc/src/spill_context/stats.rs:481
if r >= 0.0 && t >= 0.0 {
 return r / (BASE_IO_TIME + t);
}

crates/polars-ooc/src/spill_context/stats.rs:481-483

where r = bytes * seconds_spilled and t = spill_time + unspill_time. A context whose frames are cheap to write and expensive to read back gets a low score and is avoided. All statistics decay with a one-second half-life, because a plan that was a good spill candidate two minutes ago may not be now. The algorithm is documented as an adaptation of “Discounted Thompson Sampling for Non-Stationary Bandit Problems”.

Within the winning context, candidates are drained from the spill queue in policy order and filtered by size:

 rust
 
 
// crates/polars-ooc/src/memory_manager.rs:321
ctx.0.drain_live_while(|rt| {
 let Some(cand) = rt.upgrade() else {
 return true;
 };
 match cand.clone().spill_status() {
 SpillStatus::InMemory(sz) if sz >= min_spill => {
 candidates.push((cand, SpillInProgressTracker::new(sz), rt));
 },
 SpillStatus::InMemory(_) => {
 cand.cancel_spill_attempt_and_reinsert(
 rt.registration_id,
 ctx.1,
 InsertReason::TooSmall(rt.timestamp),
 );
 },
 SpillStatus::Pinned => {
 cand.cancel_spill_attempt_and_reinsert(
 rt.registration_id,
 ctx.1,
 InsertReason::Unpin,
 );
 },
 // A spilled token is re-inserted by whoever unspills it, a
 // dropped one does not need re-inserting at all.
 SpillStatus::Spilled | SpillStatus::Dropped => {},
 }

 num_considered += 1;
 num_considered < SPILL_FRAME_BATCH_SIZE
});

crates/polars-ooc/src/memory_manager.rs:321-350

Frames smaller than POLARS_OOC_SPILL_MIN_BYTES (64 KiB default) are not worth the round trip and go to the back of the queue; pinned frames (being read right now) are skipped and re-inserted when unpinned. The drain is capped at 256 candidates per attempt.

Each surviving candidate is spilled on its own task, bounded by a semaphore of POLARS_OOC_MAX_PARALLEL_SPILL_TASKS (64 by default):

 rust
 
 
// crates/polars-ooc/src/memory_manager.rs:137
for (spillable, spill_in_progress, rt) in spillables {
 let permit = self.spill_semaphore.clone().acquire_owned().await.unwrap();

 let successful_spill = successful_spill.clone();
 let ctx = ctx.clone();

 executor::spawn(TaskPriority::High, task_metrics.as_deref(), async move {
 // Spill, or reinsert if a failure.
 match spillable.clone().try_spill(ctx.clone()) {
 Ok(spill_success) => {
 if spill_success.await {
 ...

crates/polars-ooc/src/memory_manager.rs:137-147

The on-disk format

Spilling a DataFrame means writing an Arrow IPC stream (not a file format with random access — a stream, so it reads back sequentially):

 rust
 
 
// crates/polars-ooc/src/spill_frame.rs:26
async fn spill(&self, context_id: &str) -> Self::Spilled {
 let mut df = self.clone();
 let context_id = context_id.to_owned();

 // Encode in the current task (on computational executor).
 let mut buf = Vec::new();

 let mut writer = IpcStreamWriter::new(&mut buf).with_compat_level(CompatLevel::newest());
 let clvl = polars_config::config().ooc_spill_compression_level();
 if clvl > 0 {
 let zstd_lvl = ZstdLevel::try_new(clvl.try_into().unwrap()).unwrap();
 writer = writer.with_compression(Some(IpcCompression::ZSTD(zstd_lvl)));
 }
 writer
 .finish(&mut df)
 .unwrap_or_else(|e| panic!("failed to encode spill file for '{context_id}': {e}",));

crates/polars-ooc/src/spill_frame.rs:26-41

Two details worth noting. First, compression is off by default: POLARS_OOC_SPILL_COMPRESSION_LEVEL defaults to 0, and the code only attaches ZSTD when the level is positive. Compression costs CPU and the spill file is usually read once, so the default trades disk for time. Second, the encode happens on the task that owns the data (so the buffers are already hot in cache) while the actual file write is pushed onto the tokio runtime:

 rust
 
 
// crates/polars-ooc/src/spill_frame.rs:43
// Do file creation / writing on tokio.
let file = ASYNC
 .spawn(async move {
 let size = buf.len() as u64;
 let spill_file = SpillFile::new(&context_id, "ipc", size);
 if BYTES_SPILLED_TO_DISK.fetch_add(size).saturating_add(size)
 > polars_config::config().ooc_disk_budget_bytes()
 {
 spill_file.creation_aborted();
 polars_error::abort::polars_abort_ooc_out_of_disk();
 }

crates/polars-ooc/src/spill_frame.rs:43-53

The disk budget (POLARS_OOC_DISK_BUDGET_MB, 64 GB by default) is checked against a process-wide counter of bytes spilled. Exceeding it aborts the query rather than filling the disk.

The file layout is deliberately simple, with one directory per process so that a crashed run’s leftovers can be identified and removed:

text

<spill_dir>/
 <pid>/ <- process directory (one per OS process)
 spill-<ctx>-<uuid>.ipc <- individual spill file (unique per spill)

crates/polars-ooc/src/spill_file.rs:9-15

The default spill directory is /var/tmp/polars-$USER/spill on Linux — deliberately not std::env::temp_dir(), because on Linux that is often tmpfs, and spilling to tmpfs defeats the entire purpose of spilling:

 rust
 
 
// crates/polars-config/src/spill_path.rs:16
pub fn default_ooc_spill_dir() -> PathBuf {
 #[cfg(target_os = "linux")]
 let (base, user_var) = (PathBuf::from("/var/tmp"), "USER");
 ...
 let user = std::env::var(user_var).unwrap_or_else(|_| "unknown".to_string());
 base.join(format!("polars-{user}")).join("spill")
}

crates/polars-config/src/spill_path.rs:16-31

The spill file is deleted when the SpillFile is dropped, but deletion is handed off to a background thread so the hot path never blocks on unlink:

 rust
 
 
// crates/polars-ooc/src/spill_file.rs:55
impl Drop for SpillFile {
 fn drop(&mut self) {
 SPILL_CLEANER
 .send_rq
 .send(CleanRequest::File(
 core::mem::take(&mut self.path),
 self.size,
 ))
 .unwrap();
 }
}

crates/polars-ooc/src/spill_file.rs:55-65

The same cleaner thread, started lazily, also sweeps directories left by dead processes at startup — it reads the PID from each subdirectory name and checks whether that PID is still alive:

 rust
 
 
// crates/polars-ooc/src/spill_file.rs:127
fn cleanup_stale_dirs() {
 let spill_dir = polars_config::config().ooc_spill_dir();
 let Ok(entries) = std::fs::read_dir(&spill_dir) else {
 return;
 };

 let our_pid = std::process::id();
 let dead_pid_dirs = entries.flatten().filter_map(|e| {
 let pid = e.file_name().to_str()?.parse::<u32>().ok()?;
 (pid != our_pid && !polars_utils::sys::is_process_alive(pid)).then(|| e.path())
 });
 for path in dead_pid_dirs {
 let _ = std::fs::remove_dir_all(&path);
 }
}

crates/polars-ooc/src/spill_file.rs:127-141

Reading back

Unspilling is the mirror image: open the file and decode it, on a blocking thread since it is filesystem I/O:

 rust
 
 
// crates/polars-ooc/src/spill_frame.rs:71
async fn unspill(location: &Self::Spilled) -> Self {
 let path = location.path().to_owned();
 ASYNC
 .spawn_blocking(move || {
 let file = std::fs::File::open(&path).unwrap_or_else(|e| {
 panic!("failed to open spill file {:?}: {e}", path.display())
 });
 IpcStreamReader::new(file).finish().unwrap_or_else(|e| {
 panic!("failed to read spill file {:?}: {e}", path.display())
 })
 })
 .await
 .unwrap()
}

crates/polars-ooc/src/spill_frame.rs:71-84

But nobody waits for a spill to be read back if they can avoid it. When memory usage drops below the prefetch threshold (90% of budget), the manager proactively pulls spilled frames back into memory on low-priority tasks, so the eventual read is a cache hit:

 rust
 
 
// crates/polars-ooc/src/spill_context/mod.rs:362
for _ in 0..self.stats.suggested_prefetch_amount() {
 let Some(permit) = mm.try_get_prefetch_permit() else {
 return PrefetchScheduleResult::NoPermitsLeft;
 };

 let pop = match self.policy() {
 SpillContextPolicy::MostRecent => shared.unspill_queue.pop_front(),
 SpillContextPolicy::LeastRecent => shared.unspill_queue.pop_back(),
 SpillContextPolicy::Random => shared.unspill_queue.pop_random(&mut rng),
 };

crates/polars-ooc/src/spill_context/mod.rs:362-371

Prefetching is scheduled by a separate bandit (sample_prefetch_score), and the amount of prefetching scales with how long it has been since the last spill: suggested_prefetch_amount returns 1 + prefetch_without_spill_streak. If nothing is being spilled, the engine reads more aggressively ahead of demand.

Wiring it into the streaming engine

Everything above is machinery. The nodes are what make it useful.

The unit of work in the streaming engine is a Morsel, and every morsel carries a SpillFrame:

 rust
 
 
// crates/polars-stream/src/morsel.rs:82
#[derive(Debug)]
pub struct Morsel {
 /// The data contained in this morsel.
 sf: SpillFrame,

 /// The sequence number of this morsel. May only stay equal or increase
 /// within a pipeline.
 seq: MorselSeq,
 ...
}

crates/polars-stream/src/morsel.rs:82-96

A node that buffers its input registers each incoming morsel with its context, so the buffered data becomes spillable while it waits:

 rust
 
 
// crates/polars-stream/src/nodes/in_memory_sink.rs:66
for mut recv in receivers {
 let slf = &*self;
 join_handles.push(scope.spawn_task(TaskPriority::High, async move {
 let mut morsels = Vec::new();
 while let Ok(mut morsel) = recv.recv().await {
 morsel.take_consume_token();
 slf.spill_ctx.register(morsel.sf()).await;
 morsels.push(morsel);
 }

 slf.morsels_per_pipe.lock().push(morsels);
 Ok(())
 }));
}

crates/polars-stream/src/nodes/in_memory_sink.rs:66-79

The sort node does the same for its buffered input, and additionally uses the memory budget to decide when to stop sorting in memory and start partitioning:

 rust
 
 
// crates/polars-stream/src/nodes/sort/tuning.rs:61
/// The buffered size above which the input is partitioned instead of
/// sorted in memory.
pub fn partition_threshold(&self) -> u64 {
 self.partition_threshold_bytes.min(budget() / 2)
}

/// The number of buckets that may be sorted or waiting to be emitted at
/// once while flushing: two per pipeline, as long as they take at most a
/// quarter of the memory budget.
pub fn flush_ahead(&self, bucket_bytes: u64, num_pipelines: usize) -> usize {
 let fit = budget() / 4 / bucket_bytes.max(1);
 usize::try_from(fit)
 .unwrap_or(usize::MAX)
 .clamp(1, 2 * num_pipelines)
}

crates/polars-stream/src/nodes/sort/tuning.rs:61-75

That is the hybrid strategy in one place: the sort node does not spill blindly, it partitions its input when the buffered bytes exceed half the budget, and the partition buckets are individually registered as spillable. The hash shuffle does something similar, keeping per-partition frames registered so the shuffle can spill under pressure:

 rust
 
 
// crates/polars-stream/src/hash_shuffle/store.rs:64
let frame = SpillFrameWithKeys::new_unregistered(builder.freeze_reset());
self.spill_ctx.register_no_spill_check(&frame);
self.estimated_size += frame.estimated_size() as u64;
self.frames_per_input[input].push_back(frame);

crates/polars-stream/src/hash_shuffle/store.rs:64-67

Note register_no_spill_check here: the shuffle registers without asking the global manager to spill, because it is on the hot path and does not want to block on a global decision. It relies on other nodes’ register calls to drive the check, and on the explicit memory_manager().spill() calls sprinkled through the shuffle’s own build and load loops.

The join nodes follow the same pattern with a MostRecentSpillContext (the build side of an equi-join keeps its probe keys in spillable frames), and the group-by keeps cold rows in SpillFrameWithKeys so that the rows and their hashed keys always spill and unspill together — the keys are needed to re-partition the rows, so splitting them across a frame and a side file would be a bug waiting to happen.

Aborting cleanly

Running out of disk is a normal, expected outcome of a query that decided to spill, so it is turned into an ordinary Rust error rather than a crash. polars-error/src/abort.rs implements a panic-based abort protocol with a magic string in the panic payload:

 rust
 
 
// crates/polars-error/src/abort.rs:12
pub enum QueryAborted {
 KeyboardInterrupt,
 OocOutOfDisk,
}

// We use a unique string so we can detect it in backtraces.
static POLARS_ABORT_PREFIX: &str = "__POLARS_ABORT_";
static POLARS_ABORT_KEYBOARD_INTERRUPT_STR: &str = "__POLARS_ABORT_KEYBOARD_INTERRUPT";
static POLARS_ABORT_OOC_OUT_OF_DISK_STR: &str = "__POLARS_ABORT_OOC_OUT_OF_DISK";

crates/polars-error/src/abort.rs:12-20

The out-of-disk abort can be raised from anywhere — including from inside a tokio task with no PolarsResult return path — so it is a panic with a recognisable payload, caught by catch_polars_abort at the Python boundary:

 rust
 
 
// crates/polars-python/src/utils.rs:147
let ret = self.detach(|| catch_polars_abort(AssertUnwindSafe(f)));
match ret {
 Ok(Ok(ret)) => Ok(ret),
 Ok(Err(err)) => Err(PyErr::from(err.into())),
 Err(QueryAborted::KeyboardInterrupt) => Err(PyKeyboardInterrupt::new_err("")),
 Err(QueryAborted::OocOutOfDisk) => Err(ComputeError::new_err(
 "query aborted: spilled data exceeded the out-of-core disk budget, increase POLARS_OOC_DISK_BUDGET_MB",
 )),
}

crates/polars-python/src/utils.rs:147-155

The same mechanism doubles as the Ctrl-C path: Python hooks SIGINT, the Rust side sets a bit in an atomic, and every try_raise_polars_abort() in a hot loop turns that bit into the same kind of panic. The panic hook suppresses the usual panic message when a catcher is installed, so the user sees a clean KeyboardInterrupt rather than a Rust backtrace.

The bottom line

The design is coherent once you see the constraint it is built around: the engine must never let the process run out of memory, and it must never pay for spilling more than necessary.

Accounting is global and approximate. One counting allocator for the whole process, sampled every 4 MiB per thread. The estimate can be off by 4 MiB * threads in either direction, and it includes Python’s own allocations. In exchange it works everywhere with zero instrumentation of individual operators.
Nodes do not decide to spill. They register frames as candidates; a single global manager compares the counter against the budget (POLARS_OOC_MEMORY_BUDGET_FRACTION, default 80% of RAM) and picks victims using a discounted Thompson-sampling bandit over spill contexts, scoring each context by byte-seconds of relief per second of I/O.
The on-disk format is a plain Arrow IPC stream, one file per spill under /var/tmp/polars-$USER/spill/<pid>/, ZSTD only if you ask for it, deleted asynchronously and swept by PID liveness check after a crash.
Reading back is anticipatory. Below 90% of the budget the engine prefetches spilled frames onto low-priority tasks, so the read that eventually happens is a cache hit instead of a disk wait.
There is a hard abort before the OOM killer, but it is off by default (POLARS_OOMKILL_THRESHOLD_MB / _FRACTION are unlimited out of the box). When enabled it calls libc::abort() — no unwinding, no cleanup. Running out of disk, by contrast, is a clean, catchable error.

To use it, set the budget and let the engine work:

 bash
 
 
# Spill aggressively: 25% of RAM, tiny frames still worth spilling.
export POLARS_OOC_MEMORY_BUDGET_FRACTION=0.25
export POLARS_OOC_SPILL_MIN_BYTES=1048576
export POLARS_OOC_SPILL_DIR=/mnt/big-disk/polars-spill

# Compress spill files (costs CPU, saves disk).
export POLARS_OOC_SPILL_COMPRESSION_LEVEL=3

# Die with a clear message instead of being OOM-killed.
export POLARS_OOMKILL_THRESHOLD_FRACTION=0.9

The trade-offs are real. Spilling to disk costs a serialization round trip and a read-back later, and the bandit is explicitly trying to avoid contexts where that round trip is expensive. Frames under 64 KiB are never spilled because the overhead dominates. And because the counter is process-wide, a Python program holding several gigabytes of its own data can trigger spilling of Polars frames that are nowhere near their size limit. If you are memory-constrained, lowering the budget below 100% is usually the right move — you want Polars to start spilling while there is still headroom, not when the allocator’s threshold is already crossed.

Polars Publishes a Join's Build Side as a Runtime Filter

Jay Hubbert — Wed, 21 Oct 2026 08:40:00 +0000

TL;DR: A hash join knows exactly which keys it holds, and the probe side usually does not need to look at rows whose keys cannot be in that set. Polars makes this work by turning the join key on the probe side into a DynamicPred — a predicate slot with no value at plan time — pushing it into the scan below the probe, and having the join fill the slot with the build side’s min/max range (and, when it pays, a bloom filter) once the hash table is built. The scan then skips whole row groups by their statistics and drops the rows outside the range.

The join runtime filter is the most interesting optimizer pass in Polars, because it is not really an optimizer pass at all. It does not rewrite a plan into a cheaper plan; it builds a communication channel between two operators that are far apart in the tree, and then fills that channel at run time with information that only exists once the data has been read.

The module that implements it explains itself better than I could:

text

 plan time run time
 ---------------- -------------
 join key on the probe side build side is hashed
 | |
 v v
 trace the key down min/max of the build keys
 to a parquet scan (+ a bloom filter, maybe)
 | |
 v |
 DynamicPred slot <--------- set ----------+
 (no value yet) (the range)
 |
 v
 parquet scan under the probe side:
 - skip row groups whose stats fall outside the range
 - drop rows whose value falls outside the range

 rust
 
 
//! Let a hash join tell the parquet scan under its probe side which keys the build
//! side holds.
//!
//! A join whose one side is known to be much smaller than a scan under its other
//! side gets that side forced as build side; one whose side is only estimated to be
//! much smaller gets it preferred. Every probe key that
//! reads a parquet scan column unchanged gets a dynamic predicate on that scan. The
//! join publishes the range of its build keys once the build is done. The scan
//! skips the row groups outside the range and drops the rows outside it. When the
//! build keys are estimated to be a small share of the scan's distinct keys the
//! join also publishes a bloom filter over them, which the scan probes per row
//! instead.

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:1

The plan-time side: finding a scan to talk to

The pass runs after the join build side has been chosen, and it starts by deciding whether there is anything to talk to at all:

 rust
 
 
pub(super) fn attach_join_runtime_filters(
 root: Node,
 ir_arena: &mut Arena<IR>,
 expr_arena: &mut Arena<AExpr>,
) {
 if !polars_config::config().join_runtime_filters() {
 return;
 }
 // Inputs before their join, so a join lower in a probe chain is forced before
 // an outer one tries to carry a predicate through it.
 let mut joins = Vec::new();
 let mut has_pruning_scan = false;
 let mut stack = vec![root];
 while let Some(node) = stack.pop() {
 let ir = ir_arena.get(node);
 match ir {
 IR::Join { .. } => joins.push(node),
 IR::Scan { scan_type, .. } => has_pruning_scan |= skips_batches(scan_type),
 _ => {},
 }
 ir.copy_inputs(&mut stack);
 }
 if !has_pruning_scan {
 return;
 }

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:79

Two early exits define the scope of the whole feature. The first is a config flag (POLARS_JOIN_RUNTIME_FILTERS, on by default). The second is has_pruning_scan: the only scans that can use a range are ones that skip batches by their statistics, which in practice means parquet scans with row-group statistics:

 rust
 
 
fn skips_batches(scan_type: &FileScanIR) -> bool {
 scan_type
 .flags()
 .contains(ScanFlags::SKIPS_BATCHES_BY_STATISTICS)
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:475

A CSV scan cannot use a range, so a plan with no parquet scan under the probe side is left completely alone.

Which joins qualify

Not every join can publish a filter. The constraints are in one function:

 rust
 
 
/// Whether the join may publish a filter or be crossed by one: an inner, semi or
/// anti equi join the streaming engine can run as a hash join that blocks its
/// probe side until the build is done. Sorted inputs may still make it a merge
/// join, which drops the filter. A key that may evaluate differently each time
/// gives no filter, as the join evaluates it again when it builds.
fn is_eligible_join(options: &JoinOptionsIR, expr_arena: &Arena<AExpr>) -> bool {
 let args = &options.args;
 let JoinTypeOptionsIR::Equi { on, .. } = &options.options else {
 return false;
 };
 !on.is_empty()
 && on.iter().all(|(left, right)| {
 !is_inherently_nondeterministic(left.node(), expr_arena)
 && !is_inherently_nondeterministic(right.node(), expr_arena)
 })
 && (args.how == JoinType::Inner || args.how.is_semi_anti())
 && args.maintain_order == MaintainOrderJoin::None
 && !args.nulls_equal
 && args.slice.is_none()
 && !args.validation.needs_checks()
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:334

Every clause is a correctness requirement:

Equi only. A range is only meaningful for equality keys.
Inner, semi or anti. A left or right outer join keeps unmatched rows from one side, so pruning the probe side would drop rows the join is supposed to emit.
No maintain_order. Order maintenance changes which side is read first, and the whole scheme depends on the probe side being read after the build.
No nulls_equal. The published range excludes nulls; a join that treats nulls as equal would need nulls in the range.
No slice. A slice takes the first n rows of the join output, and pruning the probe side would change which rows those are.
Deterministic keys. The join evaluates the key expression again when it builds, and the scan evaluates it when it reads. If the two evaluations could differ, the published range would describe a different set of keys than the hash table holds.

Semi and anti joins get a special rule, stated in the module docs:

 rust
 
 
//! A semi join publishes from either side and an anti join from its left side
//! only, as the rows of its right side that match no left key change nothing. A
//! semi join may prefer its right side on an estimate; a left side is only built
//! when forced, as building it keeps its rows. It is not forced when the right keys
//! are known to be much fewer than its estimated rows, as a right build keeps only
//! those.

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:30

Which side becomes the build

The pass picks the build side itself, and it does so from statistics rather than from a user hint. build_candidates produces, for each side, a forced candidate if the side has a hard row bound, and a preferred candidate from its estimate:

 rust
 
 
/// The ways a filtered side may be built from: forced when it has a bound, and
/// preferred on its estimate.
fn build_candidates(left: bool, stats: &NodeStats) -> Vec<BuildCandidate> {
 if stats.filtered >= stats.unfiltered {
 return Vec::new();
 }
 let forced = stats.max_rows().map(|rows| BuildCandidate {
 left,
 rows,
 forced: true,
 });
 let preferred = BuildCandidate {
 left,
 rows: stats.filtered,
 forced: false,
 };
 forced.into_iter().chain([preferred]).collect()
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:273

A side that is filtered below its unfiltered size is the only candidate — an unfiltered side has no interesting range, because its range is just the range of the whole column. The “lopsided” threshold is a factor of eight:

 rust
 
 
/// A side has to be this many times smaller to be picked as the build side.
pub(super) const LOPSIDED_FACTOR: f64 = 8.0;

crates/polars-plan/src/plans/optimizer/join_build_side.rs:14

and the candidate is only taken if the scan under the probe side is big enough for the pruning to be worth it:

 rust
 
 (!filters.is_empty() && *probe_rows >= LOPSIDED_FACTOR * side.rows).then_some(side)

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:173

Tracing the key to a scan

This is the part that makes the feature work: the join’s key on the probe side has to reach a scan that reads that column unchanged. If the scan applies a filter to the key column first, the values that reach the join are not the values in the file, and a range published from the build side would be wrong for the filtered values.

trace_probe_keys walks the probe input and collects the scans each key reaches:

 rust
 
 
fn trace_probe_keys(
 probe_input: Node,
 probe_keys: Vec<Option<PlSmallStr>>,
 ir_arena: &Arena<IR>,
 expr_arena: &mut Arena<AExpr>,
 scratch: &mut UnitVec<Node>,
) -> Vec<TracedKey> {
 let probe_schema = ir_arena.get(probe_input).schema(ir_arena).into_owned();
 let mut traced = Vec::new();
 for (key_idx, name) in probe_keys.into_iter().enumerate() {
 let Some(name) = name else { continue };
 if !probe_schema.get(&name).is_some_and(supports_runtime_range) {
 continue;
 }
 let column = expr_arena.add(AExpr::Column(name));
 let (dyn_node, pred) = new_dynamic_pred(column, expr_arena, None);
 let predicate = ExprIR::from_node(dyn_node, expr_arena);
 let origins = scan_origins(probe_input, predicate, ir_arena, expr_arena, scratch);

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:317

scan_origins is the actual traversal. It pushes a predicate down through the nodes a predicate can be pushed past, follows the column through a union into every input, and stops at anything that would change the column’s values:

 rust
 
 
 IR::SimpleProjection { .. }
 | IR::Filter { .. }
 | IR::Select { .. }
 | IR::HStack { .. } => {
 match push_past(
 node,
 &mut predicate,
 ir_arena,
 expr_arena,
 scratch,
 maintain_errors,
 ) {
 Ok(Some(input)) => node = input,
 _ => continue 'paths,
 }
 },
 // A slice over the union would take other rows once some are pruned.
 IR::Union { inputs, options } if options.slice.is_none() => {
 // Each input rewrites its own expression nodes; the filter they
 // refer to stays the one the join publishes.
 for &input in inputs.iter().rev() {
 let copy = deep_clone_ae(predicate.node(), expr_arena);
 paths.push((input, ExprIR::from_node(copy, expr_arena)));
 }
 continue 'paths;
 },

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:407

Two of the guards are worth reading closely. A Filter is passed through by push_past, which also rewrites the predicate so it refers to the same column after the filter — so the filter’s own restriction is preserved in the predicate that ends up on the scan. But a filter containing a window function is a barrier, because a window function is lowered to a group-by that reads the scan on its own:

 rust
 
 
 IR::Filter {
 predicate: filter, ..
 } if has_window(std::slice::from_ref(filter), expr_arena) => continue 'paths,

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:384

A union is followed into every input, and each input gets its own copy of the predicate expression — but they all refer to the same DynamicPred, so they all see the same published filter. A union with a slice is excluded, because pruning one input would shift the rows the slice takes.

Attaching the slot to the scan

Once a scan is found, the predicate is attached to it, combined with any predicate the scan already had:

 rust
 
 
fn attach_to_scan(
 scan: Node,
 predicate: ExprIR,
 ir_arena: &mut Arena<IR>,
 expr_arena: &mut Arena<AExpr>,
) {
 let IR::Scan {
 predicate: existing,
 predicate_file_skip_applied,
 ..
 } = ir_arena.get_mut(scan)
 else {
 unreachable!()
 };
 // Files are skipped by the scan predicate after these filters are attached.
 debug_assert!(predicate_file_skip_applied.is_none());
 *existing = Some(match existing.take() {
 None => predicate,
 Some(existing) => {
 let node = expr_arena.add(AExpr::BinaryExpr {
 left: existing.node(),
 op: Operator::And,
 right: predicate.node(),
 });
 ExprIR::from_node(node, expr_arena)
 },
 });
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:503

The predicate is an ordinary expression in the plan — a DynamicPred function node wrapping a column reference — which means it flows through the normal expression machinery. Its value, though, is not in the plan. That is what DynamicPred is for:

 rust
 
 
/// A predicate over `node` whose value a producer sets at run time, evaluated
/// per row wherever it lands. A reader reads in `read_order` when it can.
pub fn new_dynamic_pred(
 node: Node,
 arena: &mut Arena<AExpr>,
 read_order: Option<ReadOrder>,
) -> (Node, DynamicPred) {
 let pred = DynamicPred::new(read_order);
 let function = IRFunctionExpr::DynamicPred {
 pred: pred.downgrade(),
 };
 let options = function.function_options();
 let aexpr = AExpr::Function {
 input: vec![ExprIR::from_node(node, arena)],
 function,
 options,
 };

 (arena.add(aexpr), pred)
}

crates/polars-plan/src/plans/optimizer/predicate_pushdown/dynamic.rs:246

The DynamicPred is a shared, mutable slot: an Arc<Inner> with an RwLock<Option<Arc<dyn PredicateExpr>>> and an AtomicBool that says whether the value has been set:

 rust
 
 
 pub fn set(&self, pred: Arc<dyn PredicateExpr>) {
 {
 let mut guard = self.inner.pred.write().unwrap();
 *guard = Some(pred);
 }
 self.inner.is_set.store(true, Ordering::Release);
 }

 pub fn is_set(&self) -> bool {
 self.inner.is_set.load(Ordering::Acquire)
 }

crates/polars-plan/src/plans/optimizer/predicate_pushdown/dynamic.rs:143

Before it is set, the predicate evaluates to “everything passes”, so the scan is correct whether or not the join ever publishes anything.

The run-time side: what the join publishes

The plan carries a list of these filters on the join node:

 rust
 
 
/// The build-side key at `key_idx` of the join's `on`, published through
/// `pred` for the probe side: its range, and a bloom filter over its values
/// when `bloom_keys` gives the number of distinct keys to size it for.
#[derive(Clone, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "ir_serde", derive(Serialize, Deserialize))]
pub struct RuntimeFilter {
 pub key_idx: usize,
 pub pred: DynamicPred,
 pub bloom_keys: Option<usize>,
 /// The probe side's distinct keys, when the plan can estimate them.
 pub probe_distinct: Option<usize>,
}

crates/polars-plan/src/plans/options.rs:414

bloom_keys is the plan’s estimate of how many distinct keys the build side holds, and probe_distinct is the plan’s estimate of how many distinct keys the probe side has. Whether to build a bloom filter at all depends on the ratio:

 rust
 
 
/// Largest ratio of distinct build keys to distinct probe keys for which a
/// bloom filter is published.
pub const MAX_BUILD_PROBE_DISTINCT_RATIO: f64 = 0.3;

crates/polars-plan/src/plans/options.rs:402

A bloom filter over the build keys is only worth probing if the build keys are a small fraction of the probe’s distinct keys. If the build side has almost as many distinct keys as the probe side, the filter would reject almost nothing and only cost time.

At run time, RuntimeFilters owns one builder per filter. The builder tracks two things: the min/max of the key column, and (optionally) the hashes of the key values:

 rust
 
 
pub(super) struct KeyFilterBuilder {
 range: KeyRange,
 bloom: Option<BloomBuilder>,
}

crates/polars-stream/src/nodes/joins/runtime_filter.rs:219

The range is just the running min and max:

 rust
 
 
impl KeyRange {
 /// Widen the range to cover the non-null values of `column`.
 fn extend(&mut self, column: &Column) -> PolarsResult<()> {
 let min = column.min_reduce()?;
 let max = column.max_reduce()?;
 if !min.is_null() && !max.is_null() {
 self.merge(Self {
 bounds: Some((min, max)),
 });
 }
 Ok(())
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:270

and the published filter evaluates it per row with two comparisons:

 rust
 
 
impl PredicateExpr for KeyRange {
 /// Whether each value lies in the range. Nulls never do.
 fn evaluate(&self, columns: &[Column]) -> PolarsResult<Option<Column>> {
 let Some(bounds) = &self.bounds else {
 return Ok(None);
 };
 let column = columns[0].as_materialized_series();
 let Some((lo, hi)) = bounds_for(bounds, column.dtype()) else {
 return Ok(None);
 };
 let mask = (column.gt_eq(&lo)? & column.lt_eq(&hi)?).fill_null_with_values(false)?;
 Ok(Some(mask.into_column()))
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:488

Note fill_null_with_values(false): a null key never matches a join key, so it never matches the range.

The same filter also knows how to skip a whole batch of rows using the batch’s statistics:

 rust
 
 
 fn evaluate_stats(
 &self,
 min: &Column,
 max: &Column,
 _null_count: &Column,
 ) -> PolarsResult<Option<Column>> {
 let Some(bounds) = &self.bounds else {
 return Ok(Some(Column::new_scalar(
 min.name().clone(),
 Scalar::from(true),
 min.len(),
 )));
 };
 let min = min.as_materialized_series();
 let max = max.as_materialized_series();
 let Some((lo, hi)) = bounds_for(bounds, min.dtype()) else {
 return Ok(None);
 };
 // A batch is skipped when it lies entirely below or above the range. An
 // unknown statistic is null and settles nothing.
 let skip = max.lt(&lo)? | min.gt(&hi)?;
 Ok(Some(skip.fill_null_with_values(false)?.into_column()))
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:507

That is the row-group skip: if the batch’s maximum is below the range’s minimum, or its minimum is above the range’s maximum, no row in it can match, and the reader never decodes it.

When the filter is published

There are two moments at which the join can publish, and the distinction matters.

A forced build side is read completely before the probe side is opened, so its range is known before the scan even starts. A preferred build side is only preferred, so the join samples both sides at run time and decides. The sampling code makes the ordering explicit:

 rust
 
 
 fn try_transition_to_build(
 &mut self,
 recv: &[PortState],
 params: &mut EquiJoinParams,
 state: &StreamingExecutionState,
 spill_ctx: &MostRecentSpillContext,
 ) -> PolarsResult<Option<BuildState>> {
 if let Some(left) = self.only_side {
 let idx = if left { 0 } else { 1 };
 let len = self.len(left);
 if len >= params.sample_limit {
 if config::verbose() {
 eprintln!("preferred build side reached the sample limit, sampling both sides");
 }
 } else if recv[idx] == PortState::Done {
 if config::verbose() {
 eprintln!("preferred build side done with {len} rows, publishing its ranges");
 }
 self.publish_runtime_filters(left, params, state)?;

crates/polars-stream/src/nodes/joins/equi_join.rs:437

If the preferred side ends before it reaches the sample limit, it is complete, and its ranges are published before the other side is read at all. If it reaches the limit, both sides are sampled and the filter is only published if the preferred side ends up being the build side. The result is always exact; only the pruning is lost in the second case.

Once the build is done, the local builders are merged and published:

 rust
 
 
 /// Hand every build key to the runtime filters. Filters set from a sample
 /// keep their value, whichever side is built; filters of a side that was
 /// not built get a predicate that skips nothing.
 fn publish_runtime_filters(&mut self, params: &mut EquiJoinParams) {
 if !params.publishes_runtime_filters() {
 params.runtime_filters.publish_nothing();
 return;
 }
 let locals = self
 .local_builders
 .iter_mut()
 .map(|l| std::mem::take(&mut l.key_filters));
 params.runtime_filters.publish_merged(locals);
 }

crates/polars-stream/src/nodes/joins/equi_join.rs:761

publish_nothing is the important case: if the join ends up building the other side, the filter that was prepared for this side must be filled with something that rejects nothing, or the scan would drop rows it should keep.

The keys themselves are collected while the build side is read:

 rust
 
 
 let locals = RAYON.install(|| {
 morsels
 .par_chunks(chunk_len)
 .map(|chunk| {
 let mut builders = self.new_builders();
 for morsel in chunk {
 let df = morsel.df_blocking();
 let keys = ASYNC.block_on(select_key_columns(&df, key_selectors, state))?;
 self.extend(&keys, &mut builders)?;
 }
 PolarsResult::Ok(builders)
 })
 .collect::<PolarsResult<Vec<_>>>()
 })?;
 self.publish_merged(locals);

crates/polars-stream/src/nodes/joins/runtime_filter.rs:109

The bloom filter

The bloom filter is the more expensive half of the published filter, and it has a budget. Each builder buffers key hashes until the total number of build rows exceeds a budget, at which point they all switch to a shared bloom filter of the size the plan estimated:

 rust
 
 
/// Bits per key a bloom filter is sized for.
const BLOOM_BITS_PER_KEY: usize = 8;
/// A bloom filter with fewer bits per distinct key than this is not published.
const BLOOM_MIN_BITS_PER_KEY: usize = 4;
/// Smallest bloom filter that is built.
const BLOOM_MIN_BYTES: usize = 64 << 10;
/// Largest bloom filter that is published.
const BLOOM_MAX_BYTES: usize = 32 << 20;

crates/polars-stream/src/nodes/joins/runtime_filter.rs:30

The point of the budget is that a bloom filter sized from the actual distinct keys is better than one sized from the plan’s estimate, but only if you can afford to hold the hashes until you know the count. The CardinalitySketch tracks the distinct count cheaply while the hashes are buffered, so the filter can be sized exactly at publication time:

 rust
 
 
 let distinct = sketch.estimate();
 let weak = spec
 .probe_distinct
 .is_some_and(|probe| distinct as f64 > probe as f64 * MAX_BUILD_PROBE_DISTINCT_RATIO);
 let bloom = if weak {
 None
 } else if planned {

crates/polars-stream/src/nodes/joins/runtime_filter.rs:328

and if the real distinct count turns out to be too large relative to the probe side’s, the bloom filter is dropped at the last moment rather than published and probed for nothing.

The published filter then probes the bloom per row instead of the range:

 rust
 
 
 fn evaluate(&self, columns: &[Column]) -> PolarsResult<Option<Column>> {
 let Some(KeyBloom { spec, bloom }) = &self.bloom else {
 return self.range.evaluate(columns);
 };
 let column = &columns[0];
 if column.dtype() != &spec.dtype {
 return Ok(None);
 }
 let mut mask = BitmapBuilder::with_capacity(column.len());
 spec.hash_keys(column).for_each_hash(|_, hash| {
 mask.push(hash.is_some_and(|hash| bloom.contains(hash)));
 });
 Ok(Some(
 BooleanChunked::from_bitmap(column.name().clone(), mask.freeze()).into_column(),
 ))
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:402

The bloom filter is in addition to the range, not instead of it: the range is what allows whole row groups to be skipped, the bloom is what catches the rows inside the range that are not in the build side.

The consumer: the scan

The predicate lands on the scan as an ordinary expression, and the expression engine lowers it twice: once into a per-row filter, and once into a statistics-level skip.

The per-row path is DynamicPred:

 rust
 
 
 F::DynamicPred { pred, .. } => {
 map_as_slice!(misc::dynamic_pred, &pred)
 },

crates/polars-expr/src/dispatch/mod.rs:571

and the statistics path is DynamicSkipBatch, which the skip-batch lowering derives from it:

 rust
 
 
 IRFunctionExpr::DynamicPred { pred, .. } => {
 let target = resolve_stat_target(input[0].node(), arena, schema)?;
 let dtype = target_leaf_dtype(&target, schema)?;
 if !supports_runtime_range(dtype) {
 return None;
 }
 let function = IRFunctionExpr::DynamicSkipBatch { pred: pred.clone() };
 let options = function.function_options();
 let input = [
 target.min(arena),
 target.max(arena),
 target.null_count(arena),
 ]
 .into_iter()
 .map(|b| ExprIR::from_node(b.node(), arena))
 .collect();

crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs:795

The dtype restriction is explicit:

 rust
 
 
/// Whether a runtime key range may skip batches of a column with this dtype.
///
/// The range is compared against the batch's min/max with polars' ordering, so the
/// type must be one whose parquet statistics are ordered the same way. Floats are out
/// because the statistics leave out NaN; 128-bit integers because their statistics
/// are byte-wise.
pub(crate) fn supports_runtime_range(dtype: &DataType) -> bool {
 use DataType as D;
 match dtype {
 D::Boolean
 | D::UInt8
 | D::UInt16
 | D::UInt32
 | D::UInt64
 | D::Int8
 | D::Int16
 | D::Int32
 | D::Int64
 | D::String
 | D::Binary
 | D::Date
 | D::Datetime(..)
 | D::Duration(_)
 | D::Time => true,
 #[cfg(feature = "dtype-decimal")]
 D::Decimal(..) => true,
 _ => false,
 }
}

crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs:230

Floats are excluded because parquet’s min/max statistics leave out NaN, and NaN compares strangely; 128-bit integers are excluded because their statistics are byte-wise rather than numeric.

The scan side of the contract is a small enum:

 rust
 
 
pub enum RuntimeRange {
 /// Not published yet. Every batch is kept; a later file may see a range.
 Pending,
 /// Never published. Every batch is kept.
 Disabled,
 /// No value can match. Every batch is skipped.
 Empty,
 /// Only values in `lo..=hi` can match, a missing bound does not limit that
 /// side. Nulls match when `nulls_match`.
 Range {
 lo: Option<Scalar>,
 hi: Option<Scalar>,
 nulls_match: bool,
 },
}

crates/polars-io/src/predicates.rs:509

Pending and Disabled are different on purpose: Pending means “not yet, but maybe later”, so a reader keeps every batch of the current file but checks again for the next one; Disabled means “never”. The parquet reader resolves the range against each file’s statistics and skips the row groups that fall outside it.

One honest limitation

The runtime filter is a streaming engine feature. runtime_filters lives on JoinOptionsIR in polars-plan, but the only consumer is polars-stream: the physical plan nodes that read it are EquiJoin and SemiAntiJoin in crates/polars-stream/src/physical_plan/mod.rs, and crates/polars-mem-engine never looks at the field. So on the classic in-memory engine the optimizer pass still runs and still rewrites the plan, but nothing fills the slot and nothing reads it — the predicate evaluates to “everything passes”.

The bottom line

The mechanism is a shared slot, not a plan rewrite. DynamicPred is a predicate with no value at plan time; the join fills it at run time.
Eligibility is strict. Inner, semi or anti equi joins, no order maintenance, no nulls-equal, no slice, deterministic keys, and a scan that skips batches by statistics. Everything else is left alone.
The trace is the hard part. A key only qualifies if it reaches a scan unchanged; filters are pushed past (and the predicate rewritten to match), window functions and sliced unions are barriers, and a union’s inputs all share one published filter.
The build side is chosen from statistics, forced when there is a hard row bound, preferred otherwise, and only when the probe-side scan is at least eight times bigger.
What is published is a range and, conditionally, a bloom filter. The range skips row groups by statistics and drops rows outside it; the bloom filter, published only when the build keys are under 30% of the probe’s distinct keys, catches the rows inside the range that are not in the build side.
It is a streaming-engine feature. The in-memory engine ignores it.

For a user the lever is POLARS_JOIN_RUNTIME_FILTERS (on by default), and the shape of the data is what decides whether it pays: a join against a parquet scan with a very selective small build side is exactly the case this was built for.

Polars Runs Queries on Rayon's Global Thread Pool

Jay Hubbert — Wed, 21 Oct 2026 08:30:00 +0000

TL;DR: Polars does not have a thread pool of its own for query execution. crates/polars-core/src/runtime.rs defines a zero-sized RAYON marker whose install, join, scope and spawn forward to rayon’s free functions — which, on a native build, means rayon’s global pool. Polars’ own ThreadPools are a single-threaded pool for munmap and a wasm-only pool. Async orchestration lives in a separate global Tokio runtime (ASYNC), and RAYON::block_on is the escape hatch that keeps a blocking call from parking a rayon worker.

I went looking for the pool where Polars runs its queries, expecting something like a POOL global with a per-engine lifecycle. What is actually there is more interesting, and slightly the opposite of what I expected: on a native build Polars does not create a thread pool at all for query execution. It borrows rayon’s.

One marker type for every parallel entry point

The whole abstraction lives in crates/polars-core/src/runtime.rs, and it starts with a type that has no fields:

 rust
 
 
pub struct RAYON;

// Thread locals to allow disabling threading for specific threads.
#[cfg(any(target_os = "emscripten", not(target_family = "wasm")))]
thread_local! {
 static NOOP_POOL: RefCell<ThreadPool> = RefCell::new(
 ThreadPoolBuilder::new()
 .use_current_thread()
 .num_threads(1)
 .build()
 .expect("could not create no-op thread pool")
 );
}

crates/polars-core/src/runtime.rs:13

RAYON is a zero-sized type, so RAYON as a value is a singleton that carries no state. Its methods are the only way the rest of Polars is allowed to touch rayon, and every one of them has two implementations: a native one and a wasm one.

 rust
 
 
impl RAYON {
 pub fn install<OP, R>(&self, op: OP) -> R
 where
 OP: FnOnce() -> R + Send,
 R: Send,
 {
 #[cfg(not(any(target_os = "emscripten", not(target_family = "wasm"))))]
 {
 op()
 }

 #[cfg(any(target_os = "emscripten", not(target_family = "wasm")))]
 {
 self.with(|p| p.install(op))
 }
 }

 pub fn join<A, B, RA, RB>(&self, oper_a: A, oper_b: B) -> (RA, RB)
 where
 A: FnOnce() -> RA + Send,
 B: FnOnce() -> RB + Send,
 RA: Send,
 RB: Send,
 {
 self.install(|| rayon::join(oper_a, oper_b))
 }

crates/polars-core/src/runtime.rs:39

Look at the native branch of install: it is literally op(). No pool, no ThreadPool::install, nothing. The rayon call happens inside op, and because rayon parallel iterators (par_iter) use the current pool — the global one when you are not inside an explicit install — the work lands on rayon’s global thread pool.

spawn and spawn_fifo are the same story:

 rust
 
 
 pub fn spawn<OP>(&self, op: OP)
 where
 OP: FnOnce() + Send + 'static,
 {
 #[cfg(not(any(target_os = "emscripten", not(target_family = "wasm"))))]
 {
 rayon::spawn(op)
 }

 #[cfg(any(target_os = "emscripten", not(target_family = "wasm")))]
 {
 self.with(|p| {
 p.spawn(op);
 if p.current_num_threads() == 1 {
 p.yield_now();
 }
 })
 }
 }

crates/polars-core/src/runtime.rs:70

This is a deliberate design decision, and it has an obvious motivation: a library should not decide how many threads the host application gets. If Polars spun up its own pool of n_cores threads, then a Python process that also uses numpy’s threads, or a Rust application that already has a rayon pool, would end up with two competing thread pools fighting over the same cores. Sharing rayon’s global pool means there is exactly one pool of worker threads in the process, and Polars is a guest in it.

The cost is that Polars cannot bound its own parallelism independently. The pool size is rayon’s, which is POLARS_MAX_THREADS if Polars built the pool first (see below) and otherwise whatever the host application configured.

The pools Polars does own

There are exactly two ThreadPoolBuilders in the whole workspace outside of tests. The first is the wasm-only pool:

 rust
 
 
// this is re-exported in utils for polars child crates
#[cfg(not(target_family = "wasm"))] // only use this on non wasm targets
pub static THREAD_POOL: LazyLock<ThreadPool> = LazyLock::new(|| {
 let thread_name = std::env::var("POLARS_THREAD_NAME").unwrap_or_else(|_| "polars".to_string());
 ThreadPoolBuilder::new()
 .num_threads(polars_config::config().max_threads())
 .thread_name(move |i| format!("{thread_name}-{i}"))
 .build()
 .expect("could not spawn threads")
});

crates/polars-core/src/runtime.rs:194

Read the cfg carefully: on native builds this pool exists, but it is never installed — nothing runs on it — and it is only consulted by RAYON::block_on to ask whether the current thread is one of its workers. The method that does install it, RAYON::with, is compiled only for emscripten/wasm:

 rust
 
 
 #[cfg(any(target_os = "emscripten", not(target_family = "wasm")))]
 pub fn with<OP, R>(&self, op: OP) -> R
 where
 OP: FnOnce(&ThreadPool) -> R + Send,
 R: Send,
 {
 if polars_async::executor::ALLOW_RAYON_THREADS.get()
 || THREAD_POOL.current_thread_index().is_some()
 {
 op(&THREAD_POOL)
 } else {
 NOOP_POOL.with(|v| op(&v.borrow()))
 }
 }

crates/polars-core/src/runtime.rs:139

On wasm there is no shared address space to parallelise over and no real threading, so Polars uses a one-thread pool that runs the work on the current thread, and falls back to the NOOP_POOL when it must not run in parallel at all. This is the one place where the “own pool” design is real, and it exists precisely because the global-pool trick is not available.

The second pool is the UNMAP_POOL in crates/polars-utils/src/mmap.rs, with exactly one thread, and it exists for a reason that has nothing to do with queries:

 rust
 
 
pub static UNMAP_POOL: LazyLock<ThreadPool> = LazyLock::new(|| {
 let thread_name = std::env::var("POLARS_THREAD_NAME").unwrap_or_else(|_| "polars".to_string());
 ThreadPoolBuilder::new()
 .num_threads(1)
 .thread_name(move |i| format!("{thread_name}-unmap-{i}"))
 .build()
 .expect("could not spawn threads")
});

crates/polars-utils/src/mmap.rs:16

munmap takes a lock on the file being unmapped, and unmapping a large mapping can block for a long time. Doing that on a rayon worker would stall the whole pool, so Polars moves it to a dedicated single-threaded pool. This is the pattern to notice: Polars does not create pools for throughput, it creates pools for isolation.

Never block a rayon thread

The same reasoning produces the most interesting function in the file. Rayon workers are supposed to be doing work; parking one on a mutex or a channel receive steals a core from every other job. RAYON::block_on is the escape hatch, and it moves the blocking part onto Tokio’s blocking pool:

 rust
 
 
 /// Calls a blocking function without blocking the rayon thread pool by
 /// moving it to a different thread.
 ///
 /// If this thread isn't a rayon thread this simply calls f directly.
 pub fn block_on<R: Send, F: FnOnce() -> R + Send>(&self, f: F) -> R {
 if THREAD_POOL.current_thread_index().is_some() {
 // We wrap F such that the resulting closure no longer has F or R as
 // parameters, and whose only limitation is that it isn't 'static.
 let mut opt_f: Option<F> = Some(f);
 let mut out: Option<std::thread::Result<R>> = None;
 let mut wrap_f = || {
 let f = AssertUnwindSafe(opt_f.take().unwrap());
 out = Some(catch_unwind(f));
 };

 // SAFETY: we always await the future to completion before returning from here, meaning
 // wrap_f stays alive for as long as it needs to. If for whatever reason we unwind we
 // abort.
 let abort = WithDrop::new((), |()| std::process::abort);

crates/polars-core/src/runtime.rs:152

Two details worth pausing on. First, the WithDrop::new((), |std::process::abort) guard: if the future is dropped before it completes, the closure that owns F would be dropped on the wrong thread, so the process is aborted instead of risking UB. Second, catch_unwind around the moved closure: a panic in the blocking work has to travel back across the thread boundary.

The caller then spins, yielding to rayon rather than blocking it:

 rust
 
 
 while !join_handle.is_finished() {
 match rayon::yield_now() {
 Some(Yield::Executed) => {},
 Some(Yield::Idle) => std::thread::yield_now(),
 None => unreachable!(),
 }
 }

crates/polars-core/src/runtime.rs:175

rayon::yield_now lets the worker run other queued jobs while it waits, which is the whole point: the rayon thread is not idle, it is doing someone else’s work.

Note the pool the guard asks about: THREAD_POOL.current_thread_index(), i.e. the pool Polars builds for itself, not rayon’s global pool. That distinction is worth keeping in mind, because it is the seam between the two worlds: the blocking escape hatch is written for threads that belong to a pool Polars controls, because a thread from an arbitrary pool has no business being parked in ASYNC.spawn_blocking and then unwound through catch_unwind on somebody else’s scheduler.

The async runtime

Everything above is synchronous. The orchestration layer — the streaming engine’s graph of operators, the parquet reader’s I/O — lives on a Tokio runtime, and it is a global one:

 rust
 
 
pub struct RuntimeManager {
 rt: Runtime,
}

impl RuntimeManager {
 fn new() -> Self {
 let n_threads = std::env::var("POLARS_ASYNC_THREAD_COUNT")
 .map(|x| x.parse::<usize>().expect("integer"))
 .unwrap_or(usize::min(polars_config::config().max_threads(), 32));

 let max_blocking = std::env::var("POLARS_MAX_BLOCKING_THREAD_COUNT")
 .map(|x| x.parse::<usize>().expect("integer"))
 .unwrap_or(512);

crates/polars-async/src/lib.rs:23

 rust
 
 
 let rt = Builder::new_multi_thread()
 .worker_threads(n_threads)
 .max_blocking_threads(max_blocking)
 .on_thread_start(move || {
 if tokio_thread_count_start.fetch_add(1) + 1 >= (max_total_threads as i64) && !warned.load() {
 warned.store(true);
 polars_warn!("POLARS_MAX_BLOCKING_THREAD_COUNT reached ({max_blocking}), this may indicate a deadlock");
 }
 })

crates/polars-async/src/lib.rs:37

The warning is worth noting: the blocking pool is the one RAYON::block_on and spawn_blocking feed from, and if it fills up, something is waiting on a blocking call that will never complete — a deadlock waiting to happen. The runtime warns rather than hangs.

The manager is a process global:

 rust
 
 
/// The global Polars async runtime accessor.
pub static ASYNC: LazyLock<RuntimeManager> = LazyLock::new(RuntimeManager::new);

crates/polars-async/src/lib.rs:120

and polars-core re-exports it next to RAYON:

 rust
 
pub use polars_async::ASYNC;

crates/polars-core/src/runtime.rs:211

So the two halves of the parallelism story sit next to each other in one file: rayon for data-parallel CPU work, Tokio for orchestration and I/O.

How the in-memory engine uses it

The in-memory engine’s executors are ordinary rayon users, routed through the marker. The filter executor is a good example because it has to decide whether splitting the frame is even worth it:

 rust
 
 
 /// Evaluates the predicate, in parallel over row chunks if worthwhile.
 fn evaluate_predicate(&self, df: &DataFrame, state: &ExecutionState) -> PolarsResult<Column> {
 let n_chunks =
 (df.height() / MIN_ROWS_PER_PREDICATE_CHUNK).min(RAYON.current_num_threads());
 let is_column = matches!(self.predicate.as_expression(), Some(Expr::Column(_)));
 if !self.streamable || n_chunks < 2 || is_column {
 return self.predicate.evaluate(df, state);
 }

 let chunks = df.clone().split_chunks_by_n(n_chunks, true);
 let columns = RAYON.install(|| {
 chunks
 .into_par_iter()
 .map(|chunk| self.predicate.evaluate(&chunk, state))
 .collect::<PolarsResult<Vec<_>>>()
 })?;

crates/polars-mem-engine/src/executors/filter.rs:67

RAYON.current_num_threads() is the budget: the frame is split into at most as many chunks as there are workers, and only if there are at least two. The group-by executor does the same with a parallel iterator over aggregations:

 rust
 
 
pub(super) fn evaluate_aggs(
 df: &DataFrame,
 aggs: &[Arc<dyn PhysicalExpr>],
 groups: &GroupPositions,
 state: &ExecutionState,
) -> PolarsResult<Vec<Column>> {
 RAYON.install(|| {
 aggs.par_iter()
 .map(|expr| {
 let agg = expr.evaluate_on_groups(df, groups, state)?.finalize();
 polars_ensure!(agg.len() == groups.len(), agg_len = agg.len(), groups.len());
 Ok(agg)
 })
 .collect::<PolarsResult<Vec<_>>>()
 })
}

crates/polars-mem-engine/src/executors/group_by.rs:15

Note that RAYON.install is called around the par_iter, not around each element. On native that is a no-op wrapper; on wasm it is what pins the parallel iterator to Polars’ own single-thread pool.

The streaming engine mixes both worlds. The join runtime filter, for instance, fans a morsel batch over rayon and then blocks on an async key selection inside each chunk:

 rust
 
 
 let chunk_len = morsels.len().div_ceil(RAYON.current_num_threads()).max(1);
 let locals = RAYON.install(|| {
 morsels
 .par_chunks(chunk_len)
 .map(|chunk| {
 let mut builders = self.new_builders();
 for morsel in chunk {
 let df = morsel.df_blocking();
 let keys = ASYNC.block_on(select_key_columns(&df, key_selectors, state))?;
 self.extend(&keys, &mut builders)?;
 }
 PolarsResult::Ok(builders)
 })
 .collect::<PolarsResult<Vec<_>>>()
 })?;

crates/polars-stream/src/nodes/joins/runtime_filter.rs:100

ASYNC.block_on inside a rayon worker is precisely the case RAYON::block_on was written for — but here it is the Tokio runtime’s own block_on, which runs the future to completion on the current thread. It is safe because select_key_columns is genuinely async and the thread is going to be busy either way.

For a query that must not block the caller at all, the lazy engine hands the whole thing to a rayon worker and returns a handle:

 rust
 
 
 } else {
 RAYON.spawn_fifo(move || {
 let result = physical_plan.execute(&mut state);
 tx.send(result).unwrap();
 });
 }

crates/polars-lazy/src/frame/exitable.rs:20

spawn_fifo rather than spawn: FIFO scheduling means the query’s tasks are run in submission order, which keeps memory behaviour predictable for a pipeline that stages data.

Errors across task boundaries

Two small utilities in crates/polars-utils/src/async_utils/ handle the part that is easy to get wrong when work is spread across tasks: making sure an error in one task is not silently swallowed by the others.

error_capture.rs gives every task an ErrorCapture and the parent an ErrorHandle:

 rust
 
 
 /// Wraps a future such that its error result is sent to the associated [`ErrorHandle`].
 pub async fn wrap_future<F, O>(self, fut: F)
 where
 F: Future<Output = Result<O, ErrorT>>,
 {
 let err: Result<(), tokio::sync::mpsc::error::TrySendError<ErrorMessage<ErrorT>>> =
 match AssertUnwindSafe(fut).catch_unwind().await {
 Ok(Ok(_)) => return,
 Ok(Err(err)) => self.tx.try_send(ErrorMessage::Error(err)),
 Err(panic) => self.tx.try_send(ErrorMessage::Panic(panic)),
 };
 drop(err);
 }

crates/polars-utils/src/async_utils/error_capture.rs:30

The catch_unwind is what turns a panic in a spawned task into a message the parent can act on, and ErrorHandle::join re-raises it:

 rust
 
 
 /// Block until either an error is received, or all [`ErrorCapture`]s associated with this
 /// handle are dropped (i.e. successful completion of all wrapped futures).
 ///
 /// # Panics
 /// If a panic is received, this will resume unwinding.
 pub async fn join(self) -> Result<(), ErrorT> {
 let ErrorHandle { mut rx } = self;

 match rx.recv().await {
 None => Ok(()),
 Some(ErrorMessage::Error(e)) => Err(e),
 Some(ErrorMessage::Panic(panic)) => std::panic::resume_unwind(panic),
 }
 }

crates/polars-utils/src/async_utils/error_capture.rs:67

And tokio_handle_ext.rs is the abort-on-drop guard, which matters because a cancelled query must not leave a task running against a half-torn-down plan:

 rust
 
 
/// Calls [`tokio::task::JoinHandle::abort`] on the join handle when dropped.
pub struct AbortOnDropHandle<T>(pub tokio::task::JoinHandle<T>);

impl<T> Future for AbortOnDropHandle<T> {
 type Output = Result<T, tokio::task::JoinError>;

 fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
 Pin::new(&mut self.0).poll(cx)
 }
}

impl<T> Drop for AbortOnDropHandle<T> {
 fn drop(&mut self) {
 self.0.abort();
 }
}

crates/polars-utils/src/async_utils/tokio_handle_ext.rs:11

`Send`, `Sync`, and the one unsafe escape

Since Polars hands work to threads it does not own, the Send/Sync bounds matter a lot, and there is exactly one place where Polars asserts them manually. crates/polars-utils/src/sync.rs is not a general-purpose concurrency toolkit; it holds a single type:

 rust
 
 
/// Utility that allows use to send pointers to another thread.
/// This is better than going through `usize` as MIRI can follow these.
#[derive(Debug, PartialEq, Eq)]
#[repr(transparent)]
pub struct SyncPtr<T>(*mut T);

crates/polars-utils/src/sync.rs:5

The unsafe impl Sync for SyncPtr<T> and unsafe impl Send for SyncPtr<T> at the bottom of the file are the whole point: a raw pointer is neither, and this wrapper launders it into something that can cross a thread boundary while staying visible to Miri. The deref_unchecked accessor is correspondingly unsafe and documents that there are no guarantees at all beyond the pointer being valid.

Everything else in Polars that crosses threads does it with ordinary Arc/Send/Sync, and the Unsync-style opt-outs that older versions of Polars used to mark types as thread-local do not exist in this tree.

The bottom line

Polars has no thread pool for query execution on native builds. crates/polars-core/src/runtime.rs routes every parallel operation through the RAYON marker, whose native branch forwards to rayon’s global pool. One pool per process, not one per library.
The pools Polars does own are for isolation, not throughput: a one-thread UNMAP_POOL so munmap cannot stall a worker, and a wasm-only THREAD_POOL for platforms with no shared-memory parallelism.
Blocking is banned on rayon workers. RAYON::block_on moves the blocking part to Tokio’s blocking pool and yields the rayon thread to other jobs meanwhile, aborting the process rather than risking a cross-thread drop if the future is cancelled.
Orchestration is Tokio, and it is global too. ASYNC is a multi-thread runtime with POLARS_ASYNC_THREAD_COUNT workers (default: min(cores, 32)) and POLARS_MAX_BLOCKING_THREAD_COUNT blocking threads (default 512), and it warns when the blocking pool fills up because that is the shape of a deadlock.
The number of pipelines is Polars’ own. The streaming engine builds polars_config::config().max_threads() pipelines (crates/polars-stream/src/execute.rs:293), so POLARS_MAX_THREADS — which defaults to available_parallelism() — controls how many concurrent operators a streaming query runs, even though the workers underneath come from rayon’s shared pool.

For a user, the practical knobs are therefore POLARS_MAX_THREADS (pipelines and Tokio workers) plus the Tokio thread counts, and the practical rule is: never write a Python UDF or a plugin that blocks, because it will be occupying a shared worker that other queries need.

Polars Shares Data With Python Over the Arrow C Data Interface

Jay Hubbert — Wed, 21 Oct 2026 08:20:00 +0000

TL;DR: polars-arrow implements the Arrow C Data Interface, which is a contract about pointers and ownership, not about bytes. Exporting a Polars array fills an ArrowArray struct whose buffers point straight at the existing memory, and parks the Rust array in private_data so a release callback can drop it later. Importing does the mirror image: the foreign bytes are borrowed, and the C array itself is kept alive as the owner. The only thing that gets copied is a buffer that cannot legally be borrowed.

Moving a dataframe between Polars and Python looks like it should be a memcpy. It is not, and the reason is that Arrow’s C Data Interface was designed to make the memcpy unnecessary. The interface is three C structs and a set of rules about who owns what; the rules are interesting mostly because they are subtle, and Polars’ implementation is a good place to see them written down.

The three structs

crates/polars-arrow/src/ffi/generated.rs is a rust-bindgen output file containing the ABI-compatible mirrors of the three Arrow C structs. The array one:

 rust
 
 
/// ABI-compatible struct for [`ArrowArray`](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions)
#[repr(C)]
#[derive(Debug)]
pub struct ArrowArray {
 pub(super) length: i64,
 pub(super) null_count: i64,
 pub(super) offset: i64,
 pub(super) n_buffers: i64,
 pub(super) n_children: i64,
 pub(super) buffers: *mut *const ::std::os::raw::c_void,
 pub(super) children: *mut *mut ArrowArray,
 pub(super) dictionary: *mut ArrowArray,
 pub(super) release: ::std::option::Option<unsafe extern "C" fn(arg1: *mut ArrowArray)>,
 pub(super) private_data: *mut ::std::os::raw::c_void,
}

crates/polars-arrow/src/ffi/generated.rs:32

ArrowSchema has the same shape for types (format, name, metadata, flags, children, dictionary, release, private_data), and ArrowArrayStream is the pull-based variant with four function pointers (get_schema, get_next, get_last_error, release) instead of data pointers.

Every one of these has a release field, and that is the load-bearing part of the whole design. The contract is: the producer of the struct owns whatever the struct points at, and the consumer must call release exactly once when it is done. release is where the producer takes its memory back.

Exporting: no bytes move

Exporting an array is ArrowArray::new:

 rust
 
 
 /// creates a new `ArrowArray` from existing data.
 ///
 /// # Safety
 /// This method releases `buffers`. Consumers of this struct *must* call `release` before
 /// releasing this struct, or contents in `buffers` leak.
 pub(crate) fn new(mut array: Box<dyn Array>) -> Self {

crates/polars-arrow/src/ffi/array.rs:103

The buffers array is built from the array’s own buffer pointers:

 rust
 
 
 let buffers_ptr = buffers
 .iter()
 .map(|maybe_buffer| match maybe_buffer {
 Some(b) => *b as *const std::os::raw::c_void,
 None => std::ptr::null(),
 })
 .collect::<Box<[_]>>();
 let n_buffers = buffers.len() as i64;

crates/polars-arrow/src/ffi/array.rs:135

buffers here comes from the array’s ToFfi implementation. For a primitive array that is literally “the address of my values buffer”:

 rust
 
 
unsafe impl<T: NativeType> ToFfi for PrimitiveArray<T> {
 fn buffers(&self) -> Vec<Option<*const u8>> {
 vec![
 self.validity.as_ref().map(|x| x.as_ptr()),
 Some(self.values.storage_ptr().cast::<u8>()),
 ]
 }

crates/polars-arrow/src/array/primitive/ffi.rs:9

storage_ptr() is the address of the allocation, not of the first element. No copy, no allocation, no serialization: the consumer gets a pointer into the same memory Polars is using.

Now the ownership question. The ArrowArray struct has no place to put a Box<dyn Array> — it is a plain C struct. So Polars parks it in private_data:

 rust
 
 
#[allow(dead_code)]
struct PrivateData {
 array: Box<dyn Array>,
 buffers_ptr: Box<[*const std::os::raw::c_void]>,
 children_ptr: Box<[*mut ArrowArray]>,
 dictionary_ptr: Option<*mut ArrowArray>,
 variadic_buffer_sizes: Box<[i64]>,
}

crates/polars-arrow/src/ffi/array.rs:91

and hands the raw pointer to the C struct:

 rust
 
 
 Self {
 length,
 null_count,
 offset: offset as i64,
 n_buffers,
 n_children,
 buffers: private_data.buffers_ptr.as_mut_ptr(),
 children: private_data.children_ptr.as_mut_ptr(),
 dictionary: private_data.dictionary_ptr.unwrap_or(std::ptr::null_mut()),
 release: Some(c_release_array),
 private_data: Box::into_raw(private_data) as *mut ::std::os::raw::c_void,
 }

crates/polars-arrow/src/ffi/array.rs:178

Note the children: a StructArray or ListArray has child arrays, and each child is exported as its own ArrowArray with its own release callback:

 rust
 
 
 let children_ptr = children
 .into_iter()
 .map(|child| {
 Box::into_raw(Box::new(ArrowArray::new(ffi::align_to_c_data_interface(
 child,
 ))))
 })
 .collect::<Box<_>>();
 let n_children = children_ptr.len() as i64;

crates/polars-arrow/src/ffi/array.rs:144

The child’s ArrowArray is leaked into a raw pointer and kept in children_ptr, so the parent’s release callback can free each child in turn. A consumer that wants a nested array’s data has to walk children and call release on each one — the C interface’s tree of structs, each with its own lifetime.

The release callback is the other half of the contract:

 rust
 
 
// callback used to drop [ArrowArray] when it is exported
unsafe extern "C" fn c_release_array(array: *mut ArrowArray) {
 if array.is_null() {
 return;
 }
 let array = &mut *array;

 // take ownership of `private_data`, therefore dropping it
 let private = Box::from_raw(array.private_data as *mut PrivateData);
 for child in private.children_ptr.iter() {
 let _ = Box::from_raw(*child);
 }

 if let Some(ptr) = private.dictionary_ptr {
 let _ = Box::from_raw(ptr);
 }

 array.release = None;
}

crates/polars-arrow/src/ffi/array.rs:76

Dropping PrivateData drops the Box<dyn Array>, which drops its buffers, which decrements their refcounts. The consumer of the C array never has to know any of this; it just calls release. Setting array.release = None at the end is what makes the release idempotent from Rust’s side, because ArrowArray also has a Drop impl that calls release:

 rust
 
 
impl Drop for ArrowArray {
 fn drop(&mut self) {
 match self.release {
 None => (),
 Some(release) => unsafe { release(self) },
 };
 }
}

crates/polars-arrow/src/ffi/array.rs:63

ArrowSchema has the mirror-image callback, c_release_schema, dropping a SchemaPrivateData that holds the type’s CStrings and its children.

The one thing that is copied: alignment

Before exporting, Polars has to make sure the array is in a shape the C interface can express. The awkward case is a validity bitmap whose bit offset does not match the data offset. The C interface requires the validity bitmap to be aligned to the data’s offset, so Polars rebuilds the bitmap:

 rust
 
 
 fn to_ffi_aligned(&self) -> Self {
 let offset = self.values.offset();

 let validity = self.validity.as_ref().map(|bitmap| {
 if bitmap.offset() == offset {
 bitmap.clone()
 } else {
 align(bitmap, offset)
 }
 });

 Self {
 dtype: self.dtype.clone(),
 validity,
 values: self.values.clone(),
 }
 }

crates/polars-arrow/src/array/primitive/ffi.rs:33

align copies the bits into a freshly allocated bitmap; values.clone() is a refcount bump, so the data is still shared. The dispatcher that decides whether an array needs this treatment is in bridge.rs:

 rust
 
 
macro_rules! ffi_dyn {
 ($array:expr, $ty:ty) => {{
 let a = $array.as_any().downcast_ref::<$ty>().unwrap();
 if a.offset().is_some() {
 $array
 } else {
 Box::new(a.to_ffi_aligned())
 }
 }};
}

pub fn align_to_c_data_interface(array: Box<dyn Array>) -> Box<dyn Array> {

crates/polars-arrow/src/ffi/bridge.rs:15

offset() returns Some only when the validity bitmap and the values agree on their offset; when they do not, the array is replaced by an aligned copy.

Importing: borrowing, and keeping the owner alive

Import is the direction where the memory belongs to somebody else, and it is where “zero-copy” is most easily over-claimed. Polars does not copy the bytes, but it does have to keep the C array alive for as long as it uses them.

 rust
 
 
pub unsafe fn import_array_from_c(
 array: ArrowArray,
 dtype: ArrowDataType,
) -> PolarsResult<Box<dyn Array>> {
 try_from(InternalArrowArray::new(array, dtype))
}

crates/polars-arrow/src/ffi/mod.rs:47

Note that import_array_from_c consumes the ArrowArray by value. It is moved into an Arc:

 rust
 
 
#[derive(Debug, Clone)]
pub struct InternalArrowArray {
 // Arc is used for shareability since this is immutable
 array: Arc<ArrowArray>,
 // Arced to reduce cost of cloning
 dtype: Arc<ArrowDataType>,
}

impl InternalArrowArray {
 pub fn new(array: ArrowArray, dtype: ArrowDataType) -> Self {
 Self {
 array: Arc::new(array),
 dtype: Arc::new(dtype),
 }
 }
}

crates/polars-arrow/src/ffi/array.rs:592

Every buffer created from that array carries a clone of the InternalArrowArray as its owner:

 rust
 
 
unsafe fn create_buffer_known_len<T: NativeType>(
 array: &ArrowArray,
 dtype: &ArrowDataType,
 owner: InternalArrowArray,
 len: usize,
 index: usize,
) -> PolarsResult<Buffer<T>> {
 if len == 0 {
 // Zero-length arrays might have invalid pointers for zero-length slices in Rust,
 // so this is more than just an optimization.
 return Ok(Buffer::new());
 }
 let ptr: *mut T = get_buffer_ptr(array, dtype, index)?;
 let slice = core::slice::from_raw_parts(ptr, len);
 let storage = SharedStorage::from_slice_with_owner(slice, owner);
 Ok(Buffer::from_storage(storage))
}

crates/polars-arrow/src/ffi/array.rs:269

SharedStorage::from_slice_with_owner is the ForeignOwner path in polars-buffer: the bytes are not Polars’, but Polars holds the owner so they stay valid. When the last Polars-side handle dies, the Arc<ArrowArray> refcount hits zero, ArrowArray::drop runs, and the producer’s release callback fires — freeing the foreign memory exactly once, in exactly the right place.

The buffer path has an explicit escape hatch for the one case where borrowing is not legal:

 rust
 
 
 // We have to check alignment, for zero-copy to be valid.
 if ptr.is_aligned() {
 let slice = core::slice::from_raw_parts(ptr.add(offset), len);
 let storage = SharedStorage::from_slice_with_owner(slice, owner);
 Ok(Buffer::from_storage(storage))
 } else {
 // Byte-wise copy for misaligned buffers.
 let mut v = Vec::with_capacity(len);
 core::ptr::copy_nonoverlapping(
 ptr.add(offset).cast::<u8>(),
 v.spare_capacity_mut().as_mut_ptr().cast::<u8>(),
 len * size_of::<T>(),
 );
 v.set_len(len);
 Ok(Buffer::from(v))
 }

crates/polars-arrow/src/ffi/array.rs:269

A misaligned pointer cannot legally become a &[T], so Polars copies those bytes into a fresh Vec and owns that instead. This is the honest answer to “is it zero-copy”: yes, except for misaligned buffers, where it has to be or the program is undefined behaviour.

The stream interface

For whole dataframes, Polars uses the pull-based ArrowArrayStream. Exporting one is a small struct with four callbacks:

 rust
 
 
pub fn export_iterator(
 iter: Box<dyn Iterator<Item = PolarsResult<Box<dyn Array>>>>,
 field: Field,
) -> ArrowArrayStream {
 let private_data = Box::new(PrivateData {
 iter,
 field,
 error: None,
 });

 ArrowArrayStream {
 get_schema: Some(get_schema),
 get_next: Some(get_next),
 get_last_error: Some(get_last_error),
 release: Some(release),
 private_data: Box::into_raw(private_data) as *mut ::std::os::raw::c_void,
 }
}

crates/polars-arrow/src/ffi/stream.rs:217

get_next hands out one exported array per call:

 rust
 
 
unsafe extern "C" fn get_next(iter: *mut ArrowArrayStream, array: *mut ArrowArray) -> i32 {
 if iter.is_null() {
 return 2001;
 }
 let private = &mut *((*iter).private_data as *mut PrivateData);

 match private.iter.next() {
 Some(Ok(item)) => {
 // check that the array has the same dtype as field
 let item_dt = item.dtype();
 let expected_dt = private.field.dtype();
 if item_dt != expected_dt {
 private.error = Some(CString::new(format!("The iterator produced an item of data type {item_dt:?} but the producer expects data type {expected_dt:?}").as_bytes().to_vec()).unwrap());
 return 2001; // custom application specific error (since this is never a result of this interface)
 }

 std::ptr::write(array, export_array_to_c(item));

 private.error = None;
 0
 },

crates/polars-arrow/src/ffi/stream.rs:147

The reader side is a Rust iterator over Box<dyn Array>, and the last paragraph of the C stream spec is handled explicitly:

 rust
 
 
 pub unsafe fn next(&mut self) -> Option<PolarsResult<Box<dyn Array>>> {
 let mut array = ArrowArray::empty();
 let status = unsafe { (self.iter.get_next.unwrap())(&mut *self.iter, &mut array) };

 if status != 0 {
 return Some(Err(unsafe { handle_error(&mut self.iter) }));
 }

 // last paragraph of https://arrow.apache.org/docs/format/CStreamInterface.html#c.ArrowArrayStream.get_next
 array.release?;

 // SAFETY: assumed from the C stream interface
 unsafe { import_array_from_c(array, self.field.dtype.clone()) }
 .map(Some)
 .transpose()
 }

crates/polars-arrow/src/ffi/stream.rs:113

array.release? is the end-of-stream signal: a producer that sets release to None says “no more batches”.

The Python side

In Python, this surfaces as the PyCapsule interface. DataFrame.__arrow_c_stream__ is a single call into Rust:

 rust
 
 
 #[allow(unused_variables)]
 #[pyo3(signature = (requested_schema))]
 fn __arrow_c_stream__<'py>(
 &self,
 py: Python<'py>,
 requested_schema: Option<Py<PyAny>>,
 ) -> PyResult<Bound<'py, PyCapsule>> {
 py.enter_polars_ok(|| {
 self.df.write().rechunk_mut_par();
 })?;
 dataframe_to_stream(&self.df.read(), py)
 }

crates/polars-python/src/dataframe/export.rs:197

and dataframe_to_stream turns the frame into a capsule:

 rust
 
 
pub(crate) fn dataframe_to_stream<'py>(
 df: &DataFrame,
 py: Python<'py>,
) -> PyResult<Bound<'py, PyCapsule>> {
 let iter = Box::new(DataFrameStreamIterator::new(df));
 let field = iter.field();
 let stream = ffi::export_iterator(iter, field);
 PyCapsule::new_with_value(py, stream, c"arrow_array_stream")
}

crates/polars-python/src/interop/arrow/to_py.rs:78

The capsule’s name, arrow_array_stream, is the handshake: a consumer checks the name to make sure it knows how to release the pointer inside.

Going the other way, PySeries.from_arrow_c_array and from_arrow_c_stream take the capsule apart. The array variant moves the ArrowArray out of the capsule with a std::ptr::replace that leaves an empty array behind:

 rust
 
 
pub(crate) fn import_array_pycapsules(
 schema_capsule: &Bound<PyCapsule>,
 array_capsule: &Bound<PyCapsule>,
) -> PyResult<(polars_arrow::datatypes::Field, Box<dyn Array>)> {
 let field = import_schema_pycapsule(schema_capsule)?;

 // # Safety
 // array_capsule holds a valid C ArrowArray pointer, as defined by the Arrow PyCapsule
 // Interface
 unsafe {
 let array_ptr = std::ptr::replace(
 array_capsule
 .pointer_checked(Some(c"arrow_array"))?
 .as_ptr() as _,
 ArrowArray::empty(),
 );
 let array = ffi::import_array_from_c(array_ptr, field.dtype().clone()).unwrap();

 Ok((field, array))
 }
}

crates/polars-python/src/series/import.rs:53

The std::ptr::replace is the ownership transfer: the capsule’s pointer slot is overwritten with an empty struct, so whoever drops the capsule afterwards cannot double-free, and Polars now owns the ArrowArray and will call its release when the last handle dies.

On the Python side, pycapsule_to_frame handles both spellings:

 python
 
 
def pycapsule_to_frame(
 obj: Any,
 *,
 schema: SchemaDefinition | None = None,
 schema_overrides: SchemaDict | None = None,
 rechunk: bool = False,
) -> DataFrame:
 """Convert PyCapsule object to DataFrame."""
 if hasattr(obj, "__arrow_c_array__"):
 # This uses the fact that PySeries.from_arrow_c_array will create a
 # struct-typed Series. Then we unpack that to a DataFrame.
 # `struct_unnest` is a single Rust call that releases the GIL; going via
 # `DataFrame.unnest` would run the whole lazy engine while holding it,
 # which serializes construction across Python threads.
 df = wrap_df(PySeries.from_arrow_c_array(obj).struct_unnest())

 elif hasattr(obj, "__arrow_c_stream__"):

py-polars/src/polars/_utils/pycapsule.py:39

The pyarrow bridge: `to_arrow`

DataFrame.to_arrow() does not go through the capsule interface; it goes through pyarrow’s own C data interface import, which is the same trick:

 rust
 
 
pub(crate) fn to_py_array(
 array: ArrayRef,
 field: &ArrowField,
 pyarrow: &Bound<PyModule>,
) -> PyResult<Py<PyAny>> {
 let schema = Box::new(ffi::export_field_to_c(field));
 let array = Box::new(ffi::export_array_to_c(array));

 let schema_ptr: *const ffi::ArrowSchema = &*schema;
 let array_ptr: *const ffi::ArrowArray = &*array;

 let array = pyarrow.getattr("Array")?.call_method1(
 "_import_from_c",
 (array_ptr as Py_uintptr_t, schema_ptr as Py_uintptr_t),
 )?;

 Ok(array.unbind())
}

crates/polars-python/src/interop/arrow/to_py.rs:28

Polars exports an ArrowArray and an ArrowSchema, hands the two raw addresses to pyarrow’s private _import_from_c, and pyarrow takes ownership of them. The buffers are shared; pyarrow’s Array points at Polars’ memory. The same routine builds the RecordBatch for DataFrame.to_arrow(), one column at a time.

This is also why to_arrow is not free of type conversions even though it is free of byte copies: the Arrow type Polars holds has to be expressed in the compat level pyarrow expects, and some Polars types (notably dictionaries with unsigned indices) have no pyarrow equivalent and are cast on the way out.

The older protocol: `polars-ffi`

There is also a second, Polars-specific ABI in the polars-ffi crate, kept for plugins that want to exchange whole Series rather than raw Arrow arrays. It is a SeriesExport struct with its own release callback:

 rust
 
 
/// An FFI exported `Series`.
#[repr(C)]
pub struct SeriesExport {
 field: *mut ArrowSchema,
 // A double ptr, so we can easily release the buffer
 // without dropping the arrays.
 arrays: *mut *mut ArrowArray,
 len: usize,
 release: Option<unsafe extern "C" fn(arg1: *mut SeriesExport)>,
 private_data: *mut std::os::raw::c_void,
}

crates/polars-ffi/src/version_0.rs:14

The version in the module path (version_0) is a promise that the layout will not change under you; polars-ffi exports MAJOR/MINOR constants for plugins to check.

The bottom line

Export copies nothing. The ArrowArray’s buffers are the array’s own pointers; the Rust array is parked in private_data and dropped by the release callback.
Import copies nothing, except misaligned buffers. Those are copied because a misaligned pointer cannot be a &[T]. Ownership is inverted: the imported Buffer keeps the foreign ArrowArray alive as its owner, and the producer’s release fires when the last Polars handle drops.
Two copies are deliberate. A validity bitmap whose offset disagrees with the data offset is rebuilt (bitmap only, not data), and a misaligned buffer is copied byte-wise.
The stream interface is the same contract, pulled. get_next hands out one exported array at a time; a None release means end of stream.
Ownership transfer is a std::ptr::replace. Whoever takes the pointer out of a capsule leaves an empty struct behind, so exactly one party ever calls release.

The practical takeaway: if you are handing data between Polars and another Arrow implementation, use the PyCapsule interface (__arrow_c_stream__ / __arrow_c_array__) and you are sharing memory, not copying it. The one case where you will silently pay for a copy is a column whose buffers are misaligned, which in practice means data that came from somewhere unusual rather than from Polars itself.

Polars Releases the GIL Before It Does Any Real Work

Jay Hubbert — Wed, 21 Oct 2026 08:10:00 +0000

TL;DR: Almost every #[pymethods] entry point in crates/polars-python does not compute anything itself. It calls py.enter_polars_df(|| ...) or py.enter_polars(|| ...), and that helper runs the closure through pyo3’s Python::detach, which drops the GIL for the duration of the call. The GIL comes back in exactly one place: Python::attach, used to call back into Python user functions (UDFs, map_batches, group_by apply). Releasing the GIL also breaks Python’s own Ctrl-C handling, so Polars installs its own SIGINT handler and converts the resulting panic back into a KeyboardInterrupt.

The GIL is the reason import polars as pl still feels like a native extension in 2026, and the reason it also does not. A Rust extension module that holds the GIL while it crunches numbers is functionally single-threaded: every other Python thread in the process sits in gil_drop_request waiting for the extension to hand the lock back. That is why pandas.apply on a large frame blocks the whole interpreter, and it is the first thing any dataframe engine written in Rust has to fix.

The shape of the fix, in one picture:

text

 thread A (Python) Polars (Rust) thread B (Python)
 ---------------- ------------- ----------------
 df.collect()
 [GIL held]
 -- detach ------------> compute ... <-- free to run
 (no GIL)
 <-- attach ------------- (GIL re-acquired)
 [GIL held]
 return df

Polars’ fix is not subtle, but it is systematic, and it is worth reading the code because the boundary it draws tells you a lot about how Polars thinks about Python.

One helper, used everywhere

The entry point is a trait called EnterPolarsExt, implemented for Python<'_>:

 rust
 
 
pub trait EnterPolarsExt {
 /// Whenever you have a block of code in the public Python API that
 /// (potentially) takes a long time, wrap it in enter_polars. This will
 /// ensure we release the GIL and catch KeyboardInterrupts.
 ///
 /// This not only can increase performance and usability, it can avoid
 /// deadlocks on the GIL for Python UDFs.
 fn enter_polars<T, E, F>(self, f: F) -> PyResult<T>
 where
 F: Ungil + Send + FnOnce() -> Result<T, E>,
 T: Ungil + Send,
 E: Ungil + Send + Into<PyPolarsErr>;

crates/polars-python/src/utils.rs:87

The implementation is where the GIL actually goes away:

 rust
 
 
impl EnterPolarsExt for Python<'_> {
 fn enter_polars<T, E, F>(self, f: F) -> PyResult<T>
 where
 F: Ungil + Send + FnOnce() -> Result<T, E>,
 T: Ungil + Send,
 E: Ungil + Send + Into<PyPolarsErr>,
 {
 let _timeout_guard = is_timeout_enabled().then(|| {
 std::hint::cold_path();
 schedule_polars_timeout(get_traceback(self).ok())
 });
 let ret = self.detach(|| catch_polars_abort(AssertUnwindSafe(f)));
 match ret {
 Ok(Ok(ret)) => Ok(ret),
 Ok(Err(err)) => Err(PyErr::from(err.into())),
 Err(QueryAborted::KeyboardInterrupt) => Err(PyKeyboardInterrupt::new_err("")),
 Err(QueryAborted::OocOutOfDisk) => Err(ComputeError::new_err(
 "query aborted: spilled data exceeded the out-of-core disk budget, increase POLARS_OOC_DISK_BUDGET_MB",
 )),
 }
 }
}

crates/polars-python/src/utils.rs:153

Python::detach is pyo3 0.29’s name for what used to be Python::allow_threads. Under the hood it creates a SuspendAttach guard, which tells the interpreter that this thread no longer needs the GIL, and then runs the closure. When the closure returns, the guard reacquires the GIL — including on the panic path, because it is a guard.

The Ungil bound is pyo3’s safety mechanism for this. Ungil is implemented for every Send type, and the bound on the closure and its return value means the compiler will not let a Bound<'py, PyAny> (a borrowed Python object) escape into the closure, because a borrowed Python object is not Send. That is not a formality: the closure runs while other Python threads are running, so it must not be holding a pointer into the interpreter’s object graph.

enter_polars has three convenience wrappers that differ only in how they wrap the result:

 rust
 
 
 /// Same as enter_polars, but expects a PolarsResult<DataFrame> as return
 /// which is converted to a PyDataFrame.
 #[inline(always)]
 fn enter_polars_df<F>(self, f: F) -> PyResult<PyDataFrame>
 where
 Self: Sized,
 F: Ungil + Send + FnOnce() -> PolarsResult<DataFrame>,
 {
 self.enter_polars(f).map(PyDataFrame::new)
 }

crates/polars-python/src/utils.rs:108

enter_polars_ok for infallible closures, enter_polars_series for a Series, enter_polars_df for a DataFrame. There are over 200 call sites of these helpers in crates/polars-python/src, which is the point: the pattern is not applied to the expensive operations, it is applied to all of them, so a user never has to guess whether a given DataFrame method holds the GIL.

A `#[pymethods]` function that detaches

Take two of the simplest ones:

 rust
 
 
 pub fn select(&self, py: Python<'_>, columns: Vec<PyBackedStr>) -> PyResult<Self> {
 py.enter_polars_df(|| self.df.read().select(columns.iter().map(|x| &**x)))
 }

 pub fn gather(&self, py: Python<'_>, indices: Wrap<Vec<IdxSize>>) -> PyResult<Self> {
 let indices = indices.0;
 let indices = IdxCa::from_vec("".into(), indices);
 py.enter_polars_df(|| self.df.read().take(&indices))
 }

crates/polars-python/src/dataframe/general.rs:281 and :281

Note the ordering: the Vec<IdxSize> is converted into an Arrow index array before enter_polars, while the GIL is still held, because that conversion allocates and needs Vec. Only the part that actually crunches rows — the take — runs with the GIL released. That split is deliberate and consistent: argument marshalling happens under the GIL, computation happens without it.

The same shape shows up for the heavy path, LazyFrame.collect:

 rust
 
 
 #[pyo3(signature = (engine, lambda_post_opt))]
 fn collect(
 &self,
 py: Python<'_>,
 engine: Wrap<Engine>,
 lambda_post_opt: Option<Py<PyAny>>,
 ) -> PyResult<PyDataFrame> {
 py.enter_polars_df(|| {
 let ldf = self.ldf.read().clone();
 if let Some(lambda) = lambda_post_opt {
 ldf._collect_post_opt(|root, lp_arena, expr_arena| {
 post_opt_callback(&lambda, root, lp_arena, expr_arena)
 })
 } else {
 ldf.collect_with_engine(engine.0).map(|r| match r {
 QueryResult::Single(df) => df,
 // TODO: Should return query results
 QueryResult::Multiple(_) => DataFrame::empty(),
 })
 }
 })
 }

crates/polars-python/src/lazyframe/general.rs:719

The whole query — planning, optimization, physical execution, which in turn spreads work over a thread pool — happens with the GIL released. That is the payoff: while a collect() runs, every other Python thread in the process keeps running normally.

Where the GIL has to come back

There is exactly one category of code that cannot run without the GIL: calling into Python. Polars handles this by re-acquiring the GIL explicitly with Python::attach, at the point of the callback.

The clearest example is group_by_map_groups:

 rust
 
 
 pub fn group_by_map_groups(
 &self,
 py: Python<'_>,
 by: Vec<PyBackedStr>,
 lambda: Py<PyAny>,
 maintain_order: bool,
 ) -> PyResult<Self> {
 py.enter_polars_df(|| {
 let df = self.df.read().clone(); // Clone so we can't deadlock on re-entrance from lambda.
 let gb = if maintain_order {
 df.group_by_stable(by.iter().map(|x| &**x))
 } else {
 df.group_by(by.iter().map(|x| &**x))
 }?;

 let function = move |df: DataFrame| {
 Python::attach(|py| {
 let pypolars = polars(py).bind(py);
 let pydf = PyDataFrame::new(df);
 let python_df_wrapper =
 pypolars.getattr("wrap_df").unwrap().call1((pydf,)).unwrap();

crates/polars-python/src/dataframe/general.rs:388

Read the nesting carefully. The outer enter_polars_df releases the GIL for the duration of the group-by. The inner Python::attach takes it back for exactly as long as the user’s Python lambda runs. That is the boundary, and it is drawn in one place: the closure that Polars hands to Python.

The same pattern appears in the UDF plumbing, which is the more interesting case because it is not per-call but per-plan. At Python startup, Polars registers two function pointers:

 rust
 
 
// Will be overwritten on Python Polars start up.
type PythonColumnUdf =
 fn(s: &[Column], output_dtype: Option<DataType>, lambda: &Py<PyAny>) -> PolarsResult<Column>;
pub static CALL_PYTHON_COLUMNS_UDF: OnceLock<PythonColumnUdf> = OnceLock::new();

type PythonDfUdf = fn(s: DataFrame, lambda: &Py<PyAny>) -> PolarsResult<DataFrame>;
pub static CALL_PYTHON_DF_UDF: OnceLock<PythonDfUdf> = OnceLock::new();

crates/polars-plan/src/dsl/python_dsl/python_udf.rs:17

Those are plain fn pointers, so they can be stored in a OnceLock inside polars-plan, a crate that does not depend on Python at run time. The implementations live in the Python-facing crate:

 rust
 
 
fn python_function_caller_series(
 s: &[Column],
 output_dtype: Option<DataType>,
 lambda: &Py<PyAny>,
) -> PolarsResult<Column> {
 Python::attach(|py| call_lambda_with_series(py, s, output_dtype, lambda))
}

fn python_function_caller_df(df: DataFrame, lambda: &Py<PyAny>) -> PolarsResult<DataFrame> {
 Python::attach(|py| {
 let pypolars = polars(py).bind(py);

crates/polars-python/src/on_startup.rs:42 and :42

And the plan-side UDF trait calls straight through them:

 rust
 
 
impl ColumnsUdf for PythonUdfExpression {
 fn call_udf(&self, s: &mut [Column]) -> PolarsResult<Column> {
 let func = CALL_PYTHON_COLUMNS_UDF.get().unwrap();
 let field = self
 .materialized_field
 .get()
 .expect("should have been materialized at this point");
 let mut out = func(
 s,
 self.materialized_field.get().map(|f| f.dtype.clone()),
 &self.python_function,
 )?;

crates/polars-plan/src/dsl/python_dsl/python_udf.rs:107

So a pl.col("a").map_elements(lambda) in a lazy query is: the engine runs with the GIL released, reaches the UDF node, calls call_udf, which calls the registered function pointer, which does Python::attach, which takes the GIL back, calls your lambda, and releases it again.

The consequence for a Python user is worth stating plainly: while your UDF is running, the GIL is held, and every other Python thread is blocked. A UDF that itself calls back into Polars would deadlock, which is exactly why the group_by_map_groups code clones the DataFrame up front (“Clone so we can’t deadlock on re-entrance from lambda”) and why the Ungil bounds exist at all.

Releasing the GIL breaks Ctrl-C, so Polars reimplements it

Here is the part that surprised me. Releasing the GIL has a side effect that is easy to miss: Python’s KeyboardInterrupt is raised by the interpreter’s own SIGINT handler, and that handler needs the GIL to build the exception and unwind the Python frame. While Polars holds the GIL — which it now deliberately does not — a Ctrl-C cannot be delivered the normal way.

Polars solves this by taking over SIGINT itself:

 rust
 
 
pub fn register_polars_abort_mechanism() {
 let default_hook = std::panic::take_hook();
 std::panic::set_hook(Box::new(move |p| {
 // Suppress output if there is an active catcher and the panic message
 // contains the abort string.
 let num_catchers =
 ABORT_STATE.load(Ordering::Relaxed) >> ABORT_CATCHERS_UNIT.trailing_zeros();
 let suppress = num_catchers > 0 && decode_polars_abort(p.payload()).is_some();
 if !suppress {
 default_hook(p);
 }
 }));

 // WASM doesn't support signals, so we just skip installing the hook there.
 #[cfg(not(target_family = "wasm"))]
 unsafe {
 // SAFETY: we only do an atomic op in the signal handler, which is allowed.
 signal_hook::low_level::register(signal_hook::consts::signal::SIGINT, move || {

crates/polars-error/src/abort.rs:68

The signal handler does one thing: it sets a bit in a global atomic. That bit is checked from inside the computation by try_raise_polars_abort, which is described as “very cheap” and is called from deep inside the kernels:

 rust
 
 
/// Checks if the abort flag is set, and if yes panics. This function is very cheap.
#[inline(always)]
pub fn try_raise_polars_abort() {
 if ABORT_STATE.load(Ordering::Acquire) & (ABORT_CATCHERS_UNIT - 1) != 0 {
 try_raise_polars_abort_slow()
 }
}

crates/polars-error/src/abort.rs:103

The panic unwinds out of the Rust computation, is caught by catch_polars_abort (which enter_polars wraps around the whole closure), and is turned back into a Python KeyboardInterrupt by the match in utils.rs above. The user sees a normal Python traceback; internally it was a Rust panic carrying the string __POLARS_ABORT_KEYBOARD_INTERRUPT.

The same mechanism doubles as the out-of-core disk budget tripwire (QueryAborted::OocOutOfDisk), because a spilled query can run out of disk from anywhere inside the engine and there is no PolarsResult return path for it.

Crossing the boundary with data

Releasing the GIL is only half of it; the data you were holding has to survive the trip. Polars wraps the Arrow C stream reader in a Send newtype explicitly to make that legal:

 rust
 
 
/// Moves an FFI stream reader across a GIL-release boundary.
///
/// # Safety
/// The Arrow C stream interface is a plain C ABI: producers must be callable without
/// the GIL held, and Python-backed producers (pyarrow and friends) re-acquire it
/// internally. The reader is only ever touched by the single thread that owns it.
struct SendStreamReader(ArrowArrayStreamReader<Box<ArrowArrayStream>>);
unsafe impl Send for SendStreamReader {}

crates/polars-python/src/series/import.rs:115

And the same file shows the pattern applied to the actual drain:

 rust
 
 
 let s = py.enter_polars(move || {
 let mut stream = stream;

 let mut produced_arrays: Vec<Box<dyn Array>> = vec![];
 while let Some(array) = unsafe { stream.0.next() } {
 produced_arrays.push(array?);
 }

crates/polars-python/src/series/import.rs:130

On the Python side, the same idea shows up in a comment that explains why pycapsule_to_frame calls struct_unnest instead of DataFrame.unnest:

 python
 
 
 if hasattr(obj, "__arrow_c_array__"):
 # This uses the fact that PySeries.from_arrow_c_array will create a
 # struct-typed Series. Then we unpack that to a DataFrame.
 # `struct_unnest` is a single Rust call that releases the GIL; going via
 # `DataFrame.unnest` would run the whole lazy engine while holding it,
 # which serializes construction across Python threads.
 df = wrap_df(PySeries.from_arrow_c_array(obj).struct_unnest())

py-polars/src/polars/_utils/pycapsule.py:39

Callbacks that arrive from another thread

One more shape is worth showing, because it is the one that is easy to get wrong: a callback that arrives from a worker thread rather than being called from one. collect_with_callback runs the query on a blocking thread and then calls a Python callback on some other thread entirely:

 rust
 
 
 #[cfg(feature = "async")]
 #[pyo3(signature = (engine, lambda))]
 fn collect_with_callback(
 &self,
 py: Python<'_>,
 engine: Wrap<Engine>,
 lambda: Py<PyAny>,
 ) -> PyResult<()> {
 py.enter_polars_ok(|| {
 let ldf = self.ldf.read().clone();

 // We use a tokio spawn_blocking here as it has a high blocking
 // thread pool limit.
 polars_core::runtime::ASYNC.spawn_blocking(move || {
 let result = ldf
 .collect_with_engine(engine.0)
 .map(|r| match r {
 QueryResult::Single(df) => df,
 // TODO: Should return query results
 QueryResult::Multiple(_) => DataFrame::empty(),
 })
 .map(PyDataFrame::new)
 .map_err(PyPolarsErr::from);

 Python::attach(|py| match result {
 Ok(df) => {
 lambda.call1(py, (df,)).map_err(|err| err.restore(py)).ok();
 },

crates/polars-python/src/lazyframe/general.rs:757

Three things happen here. The outer enter_polars_ok releases the GIL so the Python thread that called collect_with_callback is free immediately. The query is handed to ASYNC.spawn_blocking, a Tokio blocking thread, so it does not occupy the calling thread either. And the callback itself runs under Python::attach, which re-acquires the GIL on the worker thread — this is what pyo3’s Python::attach is for, as opposed to Python::with_gil, which assumes the current thread already holds it.

The lambda is captured as Py<PyAny>, not Bound<'py, PyAny>. Py<T> is an owned, atomic reference-counted handle that is Send; Bound<'py, T> is a borrowed reference tied to a GIL epoch and is not. That is the whole trick that lets a Python object be carried into a closure that runs without the GIL: Py is Ungil, Bound is not.

The bottom line

The GIL is released by default, everywhere. enter_polars is the single door, and it calls Python::detach. Argument marshalling happens under the GIL; computation happens without it.
The GIL is re-acquired at exactly one point: the callback into Python user code, via Python::attach. A UDF therefore runs with the GIL held and blocks every other Python thread for its duration. That is the trade Polars makes, and it is unavoidable — you cannot call Python without the GIL.
The Ungil bound is the type-level guard. It makes it impossible to smuggle a borrowed Python object across the released-GIL window.
Ctrl-C is reimplemented. Because Python cannot deliver KeyboardInterrupt while the GIL is released, Polars installs its own SIGINT handler, sets an atomic flag, and turns the resulting panic back into a KeyboardInterrupt at the boundary.

For a Python user, the practical consequences are: a Polars query no longer freezes your interpreter, a Python UDF does, and if you want real parallelism across Python threads, keep the Python side of the work out of the UDF and into the expression language.

Polars Estimates Cardinality With HyperLogLog

Jay Hubbert — Wed, 21 Oct 2026 08:00:00 +0000

TL;DR: Polars has two cardinality sketches. The one in polars-compute has 16 384 one-byte registers, uses the low 14 bits of the hash as the register index and the number of leading zeros of the remaining bits as the register’s value, and estimates the cardinality with Ertl’s tau/sigma correction instead of the bias-correction table of HLL++. It is always dense — there is no sparse representation. The sketch the streaming engine actually uses is much smaller: 256 registers, one byte each, about 6.5% error. Both are used to make decisions: dictionary-encode a Parquet column, pick the build side of a hash join, publish a bloom filter, or size the hot table of a group-by.

Counting distinct values exactly means keeping a set of them, and the set is as big as the answer. A dataframe engine that wants to know “is this column worth dictionary-encoding?” or “which side of this join should I build the hash table on?” usually does not need the exact answer — it needs an answer that is right most of the time and costs a constant amount of memory no matter how many rows go past. That is what a cardinality sketch is, and Polars keeps several of them.

The register trick

The idea behind HyperLogLog is old and simple. Hash every value to 64 bits. Use p bits of the hash to pick one of m = 2^p registers. Use the position of the first 1 bit in the remaining bits as the value stored in that register, keeping the maximum ever seen. A register’s value is then “how far into the hash space did the most extreme value that landed here go”, and the distribution of those values across registers is what encodes the answer.

Polars’ implementation is in crates/polars-compute/src/hyperloglogplus.rs, and the constants that fix its accuracy and its memory are four lines:

 rust
 
 
/// The greater is P, the smaller the error.
const HLL_P: usize = 14_usize;
/// The number of bits of the hash value used determining the number of leading zeros
const HLL_Q: usize = 64_usize - HLL_P;
const NUM_REGISTERS: usize = 1_usize << HLL_P;
/// Mask to obtain index into the registers
const HLL_P_MASK: u64 = (NUM_REGISTERS as u64) - 1;

crates/polars-compute/src/hyperloglogplus.rs:31

And the whole update is three lines:

 rust
 
 
 /// Adds an element to the HyperLogLog.
 pub fn add(&mut self, obj: &T) {
 let hash = self.hash_value(obj);
 let index = (hash & HLL_P_MASK) as usize;
 let p = ((hash >> HLL_P) | (1_u64 << HLL_Q)).trailing_zeros() + 1;
 self.registers[index] = self.registers[index].max(p as u8);
 }

crates/polars-compute/src/hyperloglogplus.rs:88

Two details make this work. First, hash & HLL_P_MASK takes the low 14 bits as the index — the register choice is independent of the rank bits, which is what makes the estimator unbiased. Second, the | (1_u64 << HLL_Q) sets a sentinel bit at position 50, so trailing_zeros() is always defined (a hash of all zeros would otherwise have 64 trailing zeros and overflow the u8 register) and the rank is capped at 51, which fits in a byte.

The registers are one byte each, so the whole sketch is 16 KiB:

 rust
 
 
#[derive(Clone, Debug)]
pub struct HyperLogLog<T>
where
 T: Hash + ?Sized,
{
 registers: [u8; NUM_REGISTERS],
 phantom: PhantomData<T>,
}

crates/polars-compute/src/hyperloglogplus.rs:38

The hash is seeded with a fixed seed, not a random one:

 rust
 
 
/// Fixed seed for the hashing so that values are consistent across runs
///
/// Note that when we later move on to have serialized HLL register binaries
/// shared across cluster, this SEED will have to be consistent across all
/// parties otherwise we might have corruption. So ideally for later this seed
/// shall be part of the serialized form (or stay unchanged across versions).
const SEED: PlFixedStateQuality = PlFixedStateQuality::with_seed(0);

crates/polars-compute/src/hyperloglogplus.rs:57

That is a deliberate trade: a random seed would make two sketches of the same data disagree, which would break the merge operation and any attempt to compare estimates across runs. The comment shows the author thinking about shipping register arrays between machines.

Merging is the operation a sketch is really built for, and it is trivial because the registers are maxima:

 rust
 
 
 /// Merge the other [`HyperLogLog`] into this one
 pub fn merge(&mut self, other: &HyperLogLog<T>) {
 assert!(
 self.registers.len() == other.registers.len(),
 "unexpected got unequal register size, expect {}, got {}",
 self.registers.len(),
 other.registers.len()
 );
 for i in 0..self.registers.len() {
 self.registers[i] = self.registers[i].max(other.registers[i]);
 }
 }

crates/polars-compute/src/hyperloglogplus.rs:113

You can sketch a billion rows on four machines and combine the four sketches into one, and the answer is the answer for all of them.

The estimator, and what “HLL++” means here

The original HyperLogLog paper needed two corrections: a small-range correction (when the sketch is mostly empty registers) and a large-range correction (when the linear-count estimator breaks down and the two-largest harmonic mean takes over). HLL++ added a third thing: an empirically measured bias correction, a table of corrections per register value that removes the systematic error of the estimator in the small-to-medium range.

Polars implements none of that machinery. Its estimator is Ertl’s, from “New cardinality estimation algorithms for HyperLogLog sketches” (arXiv:1702.01284), and it folds the range corrections into the estimator itself:

 rust
 
 
 /// Guess the number of unique elements seen by the HyperLogLog.
 pub fn count(&self) -> usize {
 let histogram = self.get_histogram();
 let m = NUM_REGISTERS as f64;
 let mut z = m * hll_tau((m - histogram[HLL_Q + 1] as f64) / m);
 for i in histogram[1..=HLL_Q].iter().rev() {
 z += *i as f64;
 z *= 0.5;
 }
 z += m * hll_sigma(histogram[0] as f64 / m);
 (0.5 / 2_f64.ln() * m * m / z).round() as usize
 }

crates/polars-compute/src/hyperloglogplus.rs:121

The histogram counts how many registers hold each value. z is built from the register values in a harmonic-mean-like way; hll_tau is the correction for the case where the sketch has seen a lot of high values, and hll_sigma the correction for the case where most registers are still zero. Both are iterative series that stop when they converge:

 rust
 
 
/// Helper function sigma as defined in
/// "New cardinality estimation algorithms for HyperLogLog sketches"
/// Otmar Ertl, arXiv:1702.01284
#[inline]
fn hll_sigma(x: f64) -> f64 {
 if x == 1. {
 f64::INFINITY
 } else {
 let mut y = 1.0;
 let mut z = x;
 let mut x = x;
 loop {
 x *= x;
 let z_prime = z;
 z += x * y;
 y += y;
 if z_prime == z {
 break;
 }
 }
 z
 }
}

crates/polars-compute/src/hyperloglogplus.rs:135

The point of the tau/sigma estimator is that it is unbiased over the whole range, which makes the empirical bias table of HLL++ unnecessary. What the module does not have — and this is where the name is slightly misleading — is the other half of HLL++: the sparse representation. The registers are always a dense [u8; 16384], so every sketch costs 16 KiB, even if it has seen three values. There is no switch from a sparse list of (index, value) pairs to the dense array when the list gets too long. For a column with a handful of distinct values, Polars pays 16 KiB anyway.

The accuracy is the usual HyperLogLog accuracy, and the test suite states it in the only units that matter:

 rust
 
 
 fn compare_with_delta(got: usize, expected: usize) {
 let expected = expected as f64;
 let diff = (got as f64) - expected;
 let diff = diff.abs() / expected;
 // times 6 because we want the tests to be stable
 // so we allow a rather large margin of error
 // this is adopted from redis's unit test version as well
 let margin = 1.04 / ((NUM_REGISTERS as f64).sqrt()) * 6.0;

crates/polars-compute/src/hyperloglogplus.rs:221

1.04 / sqrt(16384) is 0.81% — the standard error of the estimator — and the test allows six times that, because a test that fails one run in a million is not a test.

The smaller sketch the streaming engine uses

The 16 KiB sketch is fine for a one-shot approx_n_unique. It is not fine for the streaming engine, where a sketch per partition per join side adds up, and where the answer only has to be good enough to make a decision. So polars-utils carries a second, much smaller implementation:

 rust
 
 
/// HyperLogLog in Practice: Algorithmic Engineering of
/// a State of The Art Cardinality Estimation Algorithm
/// Stefan Heule, Marc Nunkesser, Alexander Hall
///
/// We use m = 256 which gives a relative error of ~6.5% of the cardinality
/// estimate. We don't bother with stuffing the counts in 6 bits, byte access is
/// fast.
///
/// The bias correction described in the paper is not implemented, so this is
/// somewhere in between HyperLogLog and HyperLogLog++.
#[derive(Clone)]
pub struct CardinalitySketch {
 buckets: [u8; 256],
}

crates/polars-utils/src/cardinality_sketch.rs:20

The comment is unusually honest: this is a HyperLogLog without the bias correction, so “somewhere in between”. 256 bytes and 6.5% error is a completely different point on the accuracy/memory curve from 16 KiB and 0.8%, and for “should I publish this bloom filter?” the cheaper answer is the right one.

The update is the same register trick with different constants:

 rust
 
 
 /// Add a new hash to the sketch.
 #[inline]
 pub fn insert(&mut self, mut h: u64) {
 const ARBITRARY_ODD: u64 = 0x902813a5785dc787;
 // We multiply by this arbitrarily chosen odd number and then take the
 // top bits to ensure the sketch is influenced by all bits of the hash.
 h = h.wrapping_mul(ARBITRARY_ODD);
 let idx = (h >> 56) as usize;
 let p = 1 + (h << 8).leading_zeros() as u8;
 self.buckets[idx] = self.buckets[idx].max(p);
 }

crates/polars-utils/src/cardinality_sketch.rs:40

And the estimator is the classical one with the small-range correction:

 rust
 
 
 pub fn estimate(&self) -> usize {
 let m = 256.0;
 let alpha_m = 0.7123 / (1.0 + 1.079 / m);

 let mut sum = 0.0;
 let mut num_zero = 0;
 for x in self.buckets.iter() {
 sum = alg_add_f64(sum, inv_pow2(*x));
 num_zero += (*x == 0) as usize;
 }

 let est = (alpha_m * m * m) / sum;
 let corr_est = if est <= 5.0 / 2.0 * m && num_zero != 0 {
 // Small cardinality estimate, full 64-bit logarithm is overkill.
 m * (m as f32 / num_zero as f32).ln() as f64
 } else {
 est
 };

 if num_zero == self.buckets.len() {
 0
 } else {
 ((corr_est + 0.5) as usize).max(1)
 }
 }

crates/polars-utils/src/cardinality_sketch.rs:65

Two details are worth noticing. The hash is multiplied by an odd constant before the bits are used, which is a cheap way to mix the hash so that all of its bits influence the register choice. And the sum of 2^-bucket is accumulated with alg_add_f64, which on nightly builds uses Rust’s algebraic float addition so that the result does not depend on the order of the additions:

 rust
 
 
#[inline(always)]
pub fn alg_add_f64(a: f64, b: f64) -> f64 {
 #[cfg(feature = "nightly")]
 {
 a.algebraic_add(b)
 }
 #[cfg(not(feature = "nightly"))]
 {
 a + b
 }
}

crates/polars-utils/src/algebraic_ops.rs:2

With 256 terms of very different magnitudes, the order of the additions is not irrelevant to the last bits of the sum, and the last bits of the sum are the estimate.

The min of one is also there: an empty sketch estimates 0, and a sketch that has seen anything at all estimates at least 1.

Where the estimates are used

`approx_n_unique`

The obvious entry point. The chunked implementation is four lines:

 rust
 
 
 fn approx_n_unique(&self) -> IdxSize {
 let mut hllp = HyperLogLog::new();
 self.iter().for_each(|item| hllp.add(&item.to_total_ord()));
 hllp.count() as IdxSize
 }

crates/polars-core/src/chunked_array/ops/approx_n_unique.rs:17

to_total_ord is the float wrapper from the sorting story: for f64 it produces a TotalOrdWrap, whose Hash canonicalises -0.0 to 0.0 and all NaNs to one value. Without it, -0.0 and 0.0 would be counted as two distinct values and every NaN would be its own distinct value, which is rarely what the user meant.

Deciding whether to dictionary-encode a Parquet column

polars-compute exposes the sketch as an array-level cardinality estimate, and the Parquet writer uses it to decide whether dictionary encoding is worth it:

 rust
 
 
 let estimated_cardinality = polars_compute::cardinality::estimate_cardinality(array);

 if array.len() > 128 && (estimated_cardinality as f64) / (array.len() as f64) > 0.75 {
 return None;
 }

crates/polars-parquet/src/arrow/write/dictionary.rs:256

If more than 75% of the values are distinct, a dictionary is just an indirection with extra steps, so the column is written as plain values. The estimate does not have to be exact for that decision; it has to be right near the boundary, and 0.8% error is plenty.

The estimator itself is a dispatch on the physical type, and it handles strings by hashing the bytes:

 rust
 
 
fn binary_offset_array_estimate<O: Offset>(array: &BinaryArray<O>) -> usize {
 let mut hll = HyperLogLog::new();

 if array.has_nulls() {
 for v in array.iter() {
 let v = v.unwrap_or_default();
 hll.add(v);
 }
 } else {
 for v in array.values_iter() {
 hll.add(v);
 }
 }

 hll.count()
}

crates/polars-compute/src/cardinality.rs:127

Note that nulls are hashed as the default value — the empty string, or zero for a primitive — so they contribute at most one more distinct value, and none at all if the default value is in the data anyway. Booleans are handled exactly rather than sketched, because there are only two of them.

Choosing the build side of a hash join

This is where the small sketch earns its keep. The streaming engine samples both sides of an equi-join and estimates how many bytes each side’s hash table would need:

 rust
 
 
 let rows = rows as f64;
 Ok(Self {
 key_ratio: (sketch.estimate() as f64 / rows).min(1.0),
 min_hash_key_ratio: min_hash.and_then(|m| m.key_ratio()),
 key_width: key_bytes as f64 / rows,
 row_width: row_bytes as f64 / rows,
 })
 }

 /// Bytes retained when building a table of the distinct keys of `rows` rows.
 pub fn distinct_keys_build_bytes(&self, rows: usize) -> f64 {
 rows as f64 * self.key_ratio * (self.key_width + KEY_SLOT_OVERHEAD)
 }

crates/polars-stream/src/nodes/joins/utils.rs:356

Two ratios are computed. key_ratio is distinct keys over rows, straight from the cardinality sketch. min_hash_key_ratio is the same question asked differently: how many of the rows are distinct keys, counting each key once regardless of how often it appears. That is what the min-hash sketch is for:

 rust
 
 
 /// Add a new hash to the sketch.
 #[inline]
 pub fn insert(&mut self, h: u64) {
 const IDX_ODD: u64 = 0x85b0fc582ca15d65; // Randomly chosen.
 const TAG_ODD: u64 = 0x4a0da407caf4e9ad; // Randomly chosen.
 let idx = (h.wrapping_mul(IDX_ODD) >> (64 - NUM_ENTRIES.trailing_zeros())) as usize;
 let tag = (h.wrapping_mul(TAG_ODD) >> 32) as u32;
 let entry = &mut self.entries[idx];
 if tag < entry.tag {
 entry.tag = tag;
 entry.count = 0;
 }
 entry.count += (tag == entry.tag) as u32;
 }

crates/polars-utils/src/min_hash_sketch.rs:38

 rust
 
 
 /// Estimates the number of distinct keys as a fraction of the number of
 /// rows, weighting each distinct key equally.
 pub fn key_ratio(&self) -> Option<f64> {
 let mut nonempty = 0u64;
 let mut total = 0u64;
 for entry in self.entries.iter() {
 nonempty += (entry.count > 0) as u64;
 total += entry.count as u64;
 }
 (nonempty > 0).then(|| nonempty as f64 / total as f64)
 }

crates/polars-utils/src/min_hash_sketch.rs:60

Each entry keeps the smallest tag it has seen for that bucket and a count of how many rows hashed to that same tag. A key that appears many times wins its bucket’s tag competition and is counted once, so the ratio of non-empty entries to total count is the fraction of rows that are distinct keys.

Those two numbers feed the build-side decision:

 rust
 
 
 let left_unique =
 left.min_hash_key_ratio.unwrap_or(0.0) >= UNIQUE_KEY_RATIO;
 let right_unique =
 right.min_hash_key_ratio.unwrap_or(0.0) >= UNIQUE_KEY_RATIO;
 let mut left_score = left.all_rows_build_bytes(params.sample_limit);
 let mut right_score = right.all_rows_build_bytes(params.sample_limit);
 if left.key_ratio <= REF_KEY_RATIO && right_unique {
 left_score *= UNIQUE_BUILD_MARGIN / (left.key_ratio + 1e-6);
 }
 if right.key_ratio <= REF_KEY_RATIO && left_unique {
 right_score *= UNIQUE_BUILD_MARGIN / (right.key_ratio + 1e-6);
 }

crates/polars-stream/src/nodes/joins/equi_join.rs:521

If one side has unique keys and the other repeats them, the repeating side is assumed to reference the unique side, and it is penalised for being the build side. Getting this wrong costs memory and time, and the sample that produces these numbers is a sample — so the sketch is being used to make a decision that is right most of the time, with a bounded cost when it is wrong.

Publishing a bloom filter

The join runtime filter is a bloom filter built on the build side and probed by the other side. It is only worth publishing if it will reject enough probes, and the decision is made with a cardinality sketch of the build keys:

 rust
 
 
 let distinct = sketch.estimate();
 let weak = spec
 .probe_distinct
 .is_some_and(|probe| distinct as f64 > probe as f64 * MAX_BUILD_PROBE_DISTINCT_RATIO);
 let bloom = if weak {
 None
 } else if planned {
 // Every builder holding it was dropped above, so no thread inserts
 // anymore.
 shared
 .and_then(|bloom| {
 debug_assert_eq!(Arc::strong_count(&bloom), 1);
 Arc::into_inner(bloom).map(AtomicSplitBlockBloom::into_inner)
 })
 .filter(|bloom| distinct.saturating_mul(BLOOM_MIN_BITS_PER_KEY) <= bloom.num_bits())
 } else {
 KeyFilterSpec::bloom_for(distinct)
 };

crates/polars-stream/src/nodes/joins/runtime_filter.rs:328

If the build side has too many distinct keys relative to the probe side, the filter would not reject much, so it is dropped. The bloom filter itself is a split-block filter in the Parquet layout, 32-byte blocks and eight bits per key.

The F2 sketch: counting heavy keys

polars-utils also has a sketch for the second frequency moment:

 rust
 
 
const NUM_BUCKETS: usize = 64;

/// Estimates the second frequency moment F2 = sum_k f_k^2 of a stream of hashed
/// keys, where f_k is the number of times key k was inserted.
///
/// This is a single row of a Count-Sketch (the "fast AMS" sketch): every hash
/// adds +1 or -1 to one of 64 buckets, and the sum of squared buckets is an
/// unbiased estimate of F2 with a relative standard error of ~18%.
#[derive(Clone)]
pub struct F2Sketch {
 buckets: [i64; NUM_BUCKETS],
 num_inserts: u64,
}

crates/polars-utils/src/f2_sketch.rs:1

F2 is not a set-similarity measure; it is a measure of skew. If every key appears once, F2 equals the number of rows. If one key appears a thousand times, F2 is a million. So F2 tells you how heavy the tail of the frequency distribution is, and the streaming group-by uses exactly that to decide how big its hot table should be:

 rust
 
 
 fn maybe_grow_hot_tables(&mut self, max_size: usize) {
 let size = self.hot_table_size;
 if size >= max_size {
 return;
 }

 // Cheap early exit before estimating the number of distinct missed keys.
 let misses = self.miss_f2.num_inserts() as f64;
 if misses < HOT_TABLE_GROW_MIN_REPEAT * size as f64 {
 return;
 }

crates/polars-stream/src/nodes/group_by/sink.rs:209

Read the algebra. The model is: the missed rows consist of heavy keys that were each missed repeat times, plus the remaining distinct keys, each missed once. With D distinct keys (the cardinality sketch’s estimate, capped at the number of missed rows) and F2 the second moment over the missed rows only (the F2 sketch only sees the misses):

text

misses = D + heavy * (repeat - 1)
F2 = D + heavy * (repeat^2 - 1)

Subtracting gives F2 - misses = heavy * repeat * (repeat - 1), and misses - D = heavy * (repeat - 1), so their quotient is repeat; and F2 - 2 * misses + D = heavy * (repeat - 1)^2, so the square of misses - D divided by that is heavy. Both are in the code:

 rust
 
 
 let f2 = self.miss_f2.estimate();
 let distinct = (self.key_sketch.estimate() as f64).min(misses);
 let excess = misses - distinct;
 let denom = f2 - 2.0 * misses + distinct;
 if excess <= 0.0 || denom <= 0.0 {
 return;
 }
 let repeat = (f2 - misses) / excess;
 let heavy = excess * excess / denom;

crates/polars-stream/src/nodes/group_by/sink.rs:225

If a few keys are missing over and over, repeat is large and the hot table should grow to keep them; if the misses are spread over many distinct keys, repeat is close to one and growing the table would be wasted memory. That is the whole reason the F2 sketch exists. A cardinality sketch tells you how many distinct keys there are; F2 tells you how skewed they are, and skew is what decides whether a cache is worth growing.

The bottom line

The sketches in Polars are not one algorithm but a family, chosen per decision:

Sketch	Size	Error	Used for
`HyperLogLog` (polars-compute)	16 KiB	~0.8%	`approx_n_unique`, Parquet dictionary decision
`CardinalitySketch` (polars-utils)	256 B	~6.5%	join build side, bloom filter publishing, group-by distinct count
`MinHashSketch`	2 KiB	—	fraction of rows that are distinct keys
`F2Sketch`	512 B	~18%	heavy-key count in the streaming group-by

The trade-off is the usual one, and it is visible in the constants: the big sketch buys an order of magnitude of accuracy for 64 times the memory, and it is only used where the answer is shown to the user. The small sketch is used where the answer only has to tilt a decision — which side to build, whether to publish a filter, how big a table to allocate — and there the 6.5% error is free.

The register trick is what makes all of this possible: 64 bits of hash per value, a byte per register, and a maximum update, so that the whole cost of “how many distinct values are there” is one hash and one byte comparison per row, with a memory footprint that does not depend on the number of rows at all. The estimator on top is where the accuracy comes from, and Polars' choice of Ertl’s tau/sigma over HLL++’s empirical bias table is a nice example of a better algorithm replacing a lookup table.

Polars Finds the Top K With a Bounded Heap

Jay Hubbert — Wed, 21 Oct 2026 07:50:00 +0000

TL;DR: There are two top-k implementations and they are not the same algorithm. The in-memory engine materialises the column and calls select_nth_unstable_by — a quickselect partition, O(n) expected, no ordering among the winners. The streaming engine cannot materialise anything, so it keeps a BinaryHeap holding exactly k keys per pipeline, admits a new key only when it is strictly below the current threshold, and merges the per-pipeline heaps pairwise when the input ends. Ties are broken by arrival order, not by row index, and the threshold is published as a filter so that rows which cannot win are never even read.

top_k looks like a trivial operation. Sort the column, take the first k, done. That is fine when the column already fits in memory and you are willing to pay O(n log n) for a full sort to look at k rows — which is almost never a good trade when k is 10 and n is a billion. The interesting question is what a dataframe engine does when it cannot afford the sort.

In Polars there are two answers, and they live in different crates.

The in-memory answer: partition, don’t sort

The eager engine has the whole column in RAM, so it can afford to copy it and then throw most of it away. This is the numeric path:

 rust
 
 
fn top_k_num_impl<T>(ca: &ChunkedArray<T>, k: usize, descending: bool) -> ChunkedArray<T>
where
 T: PolarsNumericType,
{
 if k >= ca.len() && ca.null_count() == 0 {
 return ca.clone();
 }

 // Get rid of all the nulls and transform into Vec<T::Native>.
 let mut nnca = ca.drop_nulls();
 nnca.rechunk_mut();
 let chunk = nnca.downcast_into_iter().next().unwrap();
 let (_, buffer, _) = chunk.into_inner();
 let mut vec = buffer.to_vec();

 // Partition.
 if k < vec.len() {
 if descending {
 vec.select_nth_unstable_by(k, TotalOrd::tot_cmp);
 } else {
 vec.select_nth_unstable_by(k, |a, b| TotalOrd::tot_cmp(b, a));
 }
 }

 // Reconstruct output (with nulls at the end).
 let out_len = k.min(ca.len());
 let non_null_count = ca.len() - ca.null_count();
 vec.resize(out_len, T::Native::default());
 let validity = first_n_valid_mask(non_null_count, out_len);

 let arr = PrimitiveArray::from_vec(vec).with_validity_typed(validity);
 ChunkedArray::with_chunk_like(ca, arr)
}

crates/polars-ops/src/chunked_array/top_k.rs:66

select_nth_unstable_by is quickselect: it partitions the slice so that the element at index k is the one that would be there in sorted order, and it does that in O(n) expected time instead of O(n log n). The first k elements are then the winners. The result is not sorted — the winners come out in whatever order the partition left them — and ties are broken arbitrarily, because the algorithm is unstable.

The multi-column case cannot use quickselect on a single column, so it row-encodes the keys into bytes and partitions those:

 rust
 
 
 let sorted = if k >= from_n_rows {
 match (sort_options.multithreaded, sort_options.maintain_order) {
 (true, true) => RAYON.install(|| {
 rows.par_sort();
 }),
 (true, false) => RAYON.install(|| {
 rows.par_sort_unstable();
 }),
 (false, true) => rows.sort(),
 (false, false) => rows.sort_unstable(),
 }
 &rows
 } else if sort_options.maintain_order {
 // todo: maybe there is some more efficient method, comparable to select_nth_unstable
 if sort_options.multithreaded {
 RAYON.install(|| {
 rows.par_sort();
 })
 } else {
 rows.sort();
 }
 &rows[..k]
 } else {
 // todo: possible multi threaded `select_nth_unstable`?
 let (lower, _el, _upper) = rows.select_nth_unstable(k);
 if sort_options.multithreaded {
 RAYON.install(|| {
 lower.par_sort_unstable();
 })
 } else {
 lower.sort_unstable();
 }
 &*lower
 };

crates/polars-core/src/chunked_array/ops/sort/arg_bottom_k.rs:63

Note the escape hatch: maintain_order forces a full sort, because quickselect cannot promise anything about the order of the winners. That flag is the difference between “the k largest values” and “the k largest values, in the order they appeared”.

There is a third path, used by arg_sort with a limit, which is the one place in Polars where a top-k keeps a stable order:

 rust
 
 
fn sort_limit_impl<T>(
 vals: &mut [(IdxSize, T)],
 limit: usize,
 options: SortOptions,
) -> &[(IdxSize, T)]
where
 T: TotalOrd + Send + Sync,
{
 if limit >= vals.len() {
 sort_impl(vals, options);
 return vals;
 }
 let cmp = |a: &(IdxSize, T), b: &(IdxSize, T)| {
 let ord = if options.descending {
 b.1.tot_cmp(&a.1)
 } else {
 a.1.tot_cmp(&b.1)
 };
 ord.then(a.0.cmp(&b.0))
 };
 let (lower, _el, _upper) = vals.select_nth_unstable_by(limit, cmp);
 if options.multithreaded {
 RAYON.install(|| lower.par_sort_unstable_by(cmp));
 } else {
 lower.sort_unstable_by(cmp);
 }
 lower
}

crates/polars-core/src/chunked_array/ops/sort/arg_sort.rs:41

The .then(a.0.cmp(&b.0)) is the tie-break: when the values compare equal, the original row index decides. That is what makes the result stable, and it is why the comparison carries the index along.

The streaming answer: a heap of exactly k keys

The streaming engine cannot afford to materialise a sorted copy of the column. It receives morsels, and it must decide, row by row, whether a row is worth keeping. That is a job for a bounded heap: keep the k best keys seen so far, throw everything else away.

The state is a struct in crates/polars-stream/src/nodes/top_k.rs:

 rust
 
 
struct BottomKWithPayload<P> {
 k: usize,
 heap: BinaryHeap<Priority<P, (DfsKey, RowIdxKey)>>,
 df_subsets: SlotMap<DfsKey, DfSubset>,
 row_idxs: SlotMap<RowIdxKey, IdxSize>,
 marked_for_pruning: SecondaryMap<DfsKey, ()>,
 to_prune: Vec<DfsKey>,
 gather_idxs: Vec<IdxSize>,
 shared_optimum: Arc<RwLock<Option<P>>>,
 /// The best `k` keys of all pipelines, when they share them.
 shared_keys: Option<Arc<Mutex<BinaryHeap<P>>>>,
}

crates/polars-stream/src/nodes/top_k.rs:78

The heap holds Priority<P, (DfsKey, RowIdxKey)>. P is the key type — a ReorderWithNulls<TotalOrdWrap<f64>> for numeric keys, a ReorderWithNulls<Vec<u8>> for strings, a row encoding for multi-column keys — and the payload is a pair of handles: which data frame the row came from, and which row inside it.

The Priority wrapper is the trick that makes this work. It is a newtype that compares only on the key:

 rust
 
 
use std::cmp::Ordering;

/// A pair which is ordered exclusively by the first element.
#[derive(Copy, Clone, Debug)]
pub struct Priority<P, T>(pub P, pub T);

impl<P: Ord + Eq, T> Ord for Priority<P, T> {
 #[inline]
 fn cmp(&self, other: &Self) -> Ordering {
 self.0.cmp(&other.0)
 }
}

crates/polars-utils/src/priority.rs:7

So the heap’s ordering is the ordering of the keys, and the payload is carried along without influencing it. That is what allows the heap to be a plain BinaryHeap<Priority<...>> instead of a hand-rolled structure.

The bounded part

The whole algorithm is add_one:

 rust
 
 
 fn add_one<Q>(
 &mut self,
 dfs_key: DfsKey,
 row_idx: IdxSize,
 key: Q,
 is_less: impl Fn(&Q, &P) -> bool,
 to_owned: impl Fn(Q) -> P,
 ) -> bool {
 // We use a max-heap for our bottom k. This means the top element in our heap (peek())
 // is the first to be replaced.
 if !(self.heap.len() < self.k || is_less(&key, self.peek_optimum().unwrap())) {
 return false;
 }
 let opt = self.heap.peek().map(|p| p.1);
 let row_idx_key = self.row_idxs.insert(row_idx);
 let df_subset = &mut self.df_subsets[dfs_key];
 df_subset.subset_len += 1;
 df_subset.rows.push(row_idx_key);
 self.heap
 .push(Priority(to_owned(key), (dfs_key, row_idx_key)));

 if self.heap.len() > self.k {
 let (dfs_key, row_idx_key) = self.heap.pop().unwrap().1;
 self.row_idxs[row_idx_key] = IdxSize::MAX;
 self.df_subsets[dfs_key].subset_len -= 1;
 self.mark_if_sparse(dfs_key);
 }

 opt != self.heap.peek().map(|p| p.1)
 }

crates/polars-stream/src/nodes/top_k.rs:215

Read it as a max-heap holding the k smallest keys seen so far (Polars calls this “bottom k”, and reverses the comparison when the user asked for a top-k). peek() is the largest key in the heap, i.e. the current threshold: the k-th best key so far. A new key is admitted only if the heap is not full yet, or if the new key is strictly below that threshold. If it is admitted, it is pushed and the largest element is popped, so the heap size never exceeds k + 1 for an instant and never exceeds k afterwards.

The comparison is a strict <, which is where the tie behaviour comes from: a key equal to the threshold is not admitted. Once the heap holds k keys, the first k arrivals of a group of equal keys are the ones that stay. (And within the heap, BinaryHeap::pop removes the last element of its vector, so among equal keys the most recently pushed is the one discarded.) The result is that top_k on ties keeps the earliest rows, deterministically, even though the heap itself is not a stable structure.

The heap is also bounded in memory, not just in size:

 rust
 
 
impl<P: Ord + Clone> BottomKWithPayload<P> {
 pub fn new(k: usize, num_pipelines: usize) -> Self {
 let share_keys = num_pipelines > 1 && k <= MAX_SHARED_K;
 Self::with_shared(
 k,
 Arc::default(),
 share_keys.then(|| Arc::new(Mutex::new(BinaryHeap::with_capacity(k + 1)))),
 )
 }

 fn with_shared(
 k: usize,
 shared_optimum: Arc<RwLock<Option<P>>>,
 shared_keys: Option<Arc<Mutex<BinaryHeap<P>>>>,
 ) -> Self {
 let initial_capacity = get_ideal_morsel_size().min(k + 1);
 Self {
 k,
 heap: BinaryHeap::with_capacity(initial_capacity),
 df_subsets: SlotMap::with_key(),
 row_idxs: SlotMap::with_key(),
 marked_for_pruning: SecondaryMap::new(),
 to_prune: Vec::new(),
 gather_idxs: Vec::new(),
 shared_optimum,
 shared_keys,
 }
 }

crates/polars-stream/src/nodes/top_k.rs:97

The heap is pre-allocated for at most a morsel’s worth of keys or k + 1, whichever is smaller, so a pipeline that never sees a competitive key never grows a large allocation.

The rows are not copied

The heap stores handles, not data. A row that is still in the heap lives in the data frame it came from, and the RowIdxKey slotmap remembers where. Rows that fall out of the heap are marked, and the data frames are pruned periodically:

 rust
 
 
 /// Schedules a subset for pruning once none of its rows are in the heap, or once at most half
 /// of its rows are in the heap and a gather would remove enough rows to be worthwhile.
 fn mark_if_sparse(&mut self, dfs_key: DfsKey) {
 let df_subset = &self.df_subsets[dfs_key];
 let in_heap = df_subset.subset_len;
 let removed = df_subset.df.height() - in_heap;
 let should_prune =
 in_heap == 0 || (removed >= in_heap && removed >= MIN_GATHER_REMOVED_ROWS);
 if should_prune && self.marked_for_pruning.insert(dfs_key, ()).is_none() {
 self.to_prune.push(dfs_key);
 }
 }

 pub fn prune(&mut self) {
 for dfs_key in self.to_prune.drain(..) {
 self.marked_for_pruning.remove(dfs_key);
 if self.df_subsets[dfs_key].subset_len == 0 {
 let df_subset = self.df_subsets.remove(dfs_key).unwrap();
 for row_idx in df_subset.rows {
 self.row_idxs.remove(row_idx);
 }
 } else {
 self.df_subsets[dfs_key].gather(&mut self.row_idxs, &mut self.gather_idxs);
 }
 }
 }

crates/polars-stream/src/nodes/top_k.rs:243

Pruning is not free — it is a gather over the surviving rows — so it is only done when at least half the rows are gone and at least sixteen rows would be removed. The constant MIN_GATHER_REMOVED_ROWS = 16 is there because a gather of a handful of rows costs more than the memory it saves.

Merging the per-pipeline heaps

Each pipeline of the streaming engine gets its own reducer with its own heap, because the pipelines run concurrently and cannot share a mutex on the hot path. When the input ends, the heaps are merged pairwise:

 rust
 
 
 pub fn combine(&mut self, other: &BottomKWithPayload<P>) {
 let mut new_df_keys =
 SecondaryMap::<DfsKey, DfsKey>::with_capacity(other.df_subsets.capacity());
 for (dfs_key, dfs) in &other.df_subsets {
 if dfs.subset_len > 0 {
 let subset = DfSubset {
 df: dfs.df.clone(),
 rows: Vec::new(),
 subset_len: 0,
 };
 new_df_keys.insert(dfs_key, self.df_subsets.insert(subset));
 }
 }
 for prio in &other.heap {
 let (dfs_key, row_idx_key) = prio.1;
 self.add_one(
 new_df_keys[dfs_key],
 other.row_idxs[row_idx_key],
 prio.0.clone(),
 |l, r| l < r,
 |x| x,
 );
 }
 for &dfs_key in new_df_keys.values() {
 self.mark_if_sparse(dfs_key);
 }
 self.prune();
 }

crates/polars-stream/src/nodes/top_k.rs:265

Merging two heaps of k keys each is O(k log k): every key of the second heap is offered to the first through the same add_one admission test, so most of them are rejected on the first comparison. The merge is done in parallel with a rayon reduction over the per-pipeline reducers.

So the complexity of the whole streaming top-k is:

Time: O(n log k) for n rows, versus O(n log n) for a full sort. With k = 10 that is roughly log2(10) = 3.3 comparisons per row instead of log2(n), which for a billion rows is the difference between a few seconds and a few minutes.
Memory: O(k) for the heap, plus the rows that are still in it. The heap never holds more than k keys, but the payload of those keys has to survive until the end of the query, which is why the pruning machinery exists.

Publishing the threshold

The heap’s threshold is worth more than the answer. Once the engine knows the k-th best key, every row that is worse than that key can be discarded immediately, wherever it comes from. Polars publishes it as a dynamic predicate that is pushed back into the scan:

 rust
 
 
 let keys = columns[0].as_materialized_series();
 let key_ca: &ChunkedArray<T> = keys.as_phys_any().downcast_ref().unwrap();
 let pred: BooleanChunked = key_ca
 .iter()
 .map(|opt_x| ReorderWithNulls(opt_x.map(TotalOrdWrap)) < v)
 .collect_ca(PlSmallStr::EMPTY);
 Ok(Some(Column::from(pred.into_series())))

crates/polars-stream/src/nodes/top_k.rs:359

For integer and temporal keys the predicate also exposes a value range, so a parquet reader can skip entire row groups using statistics:

 rust
 
 
 fn runtime_range(&self) -> RuntimeRange {
 // Floats order NaN apart from statistics, enums by their categories.
 if !(self.dtype.is_integer() || self.dtype.is_temporal() || self.dtype.is_decimal()) {
 return RuntimeRange::Disabled;
 }
 let Some(ReorderWithNulls(Some(TotalOrdWrap(threshold)))) = *self.shared_optimum.read()
 else {
 return RuntimeRange::Pending;
 };

crates/polars-stream/src/nodes/top_k.rs:438

That is why the streaming implementation is worth the extra machinery: the threshold does not just reduce what is kept, it reduces what is read.

For small k the pipelines go further and share their best keys, so that the published threshold is the k-th key over all pipelines rather than one pipeline’s k-th key:

 rust
 
 
/// Up to this `k` several pipelines share their best `k` keys, so the published optimum is
/// the k-th key over all pipelines instead of the best k-th key of one pipeline.
const MAX_SHARED_K: usize = 128;

crates/polars-stream/src/nodes/top_k.rs:39

The bottom line

top_k in Polars is two algorithms wearing the same name:

In memory, it is a quickselect partition over a materialised copy of the values. O(n) expected time, O(n) extra memory, winners in arbitrary order. maintain_order turns it back into a full sort.
Streaming, it is a BinaryHeap of exactly k keys per pipeline, with the payload carried along out of band so the heap never holds data. O(n log k) time, O(k) heap memory, and — the real prize — a threshold that is published back into the scan so that losing rows are never read.

The tie-breaking rules differ, and that is worth knowing: the in-memory quickselect breaks ties arbitrarily, the streaming heap keeps the earliest arrivals, and the arg_sort-with-limit path keeps the earliest rows by index. If your query depends on which of several equal values comes back, ask for maintain_order and pay for the sort.

And this is the query that makes the streaming engine reach for the heap:

 python
 
import polars as pl

lf = pl.scan_parquet("big.parquet")
result = lf.sort("x", descending=True).head(10).collect()

The streaming engine does not lower that into “sort everything, then slice”. It lowers it into a top-k node — the bounded heap — followed by a sort of the ten survivors:

 rust
 
 
/// Keeps the first `k` rows in sort order, in no particular order.
pub(crate) fn build_top_k_stream(
 input: PhysStream,
 by_column: Vec<ExprIR>,
 sort_options: &SortMultipleOptions,
 k: u64,
 dyn_pred: Option<DynamicPred>,
 expr_arena: &mut Arena<AExpr>,
 phys_sm: &mut DenseSlotMap<PhysNodeKey, PhysNode>,
) -> PhysStream {

crates/polars-stream/src/physical_plan/lower_ir.rs:67

The k rows that come out of the heap are then sorted, so the answer is correctly ordered; what the heap avoids is sorting the other billion rows. And because the heap’s threshold is published as a dynamic predicate, the parquet reader can skip row groups whose statistics cannot contain a winner — so the losing rows are not merely discarded, they are never read.

Polars Sorts Floats the Way Nobody Wants Them Sorted

Jay Hubbert — Wed, 21 Oct 2026 07:40:00 +0000

TL;DR: Rust floats have no total order, so Polars defines one: TotalOrd. Every NaN compares equal to every other NaN, and NaN is the largest value in the order, which means sort(descending=True) puts NaN first. -0.0 and 0.0 compare equal, so the sign of zero is not recoverable from a sorted column. The sort then picks a comparator per dtype, keeps nulls out of the comparator entirely, and flattens the chunks of a chunked array into one vector before sorting it.

Sorting floats is one of those places where a dataframe engine has to pick a behaviour and then live with it forever. The obvious behaviour — the one every language gives you by default — is “NaN compares equal to nothing, so it goes wherever the comparator happens to put it”. That is not a behaviour you can implement with a sort, because a comparator that is not a total order makes the result depend on the algorithm, the input order and the pivot choices of the sort. Polars refuses to have that property, so it invents an order. The question worth asking is not whether that order is correct, but which unconvenient truths it bakes in.

Why Rust cannot sort floats

f64 implements PartialOrd, not Ord, because IEEE 754 says NaN compares false against everything, including itself. Ord requires a total order, so the standard library simply does not offer one. A sort algorithm that needs Ord cannot be written for f64.

Polars solves this with a parallel trait hierarchy in crates/polars-utils/src/total_ord.rs. The TotalOrd trait is a drop-in replacement for Ord with a tot_cmp method:

 rust
 
 
/// Alternative trait for Ord. By consistently using this we can still be
/// generic w.r.t Ord while getting a total ordering for floats.
pub trait TotalOrd: TotalEq {
 fn tot_cmp(&self, other: &Self) -> Ordering;

 #[inline]
 fn tot_lt(&self, other: &Self) -> bool {
 self.tot_cmp(other) == Ordering::Less
 }

 #[inline]
 fn tot_gt(&self, other: &Self) -> bool {
 self.tot_cmp(other) == Ordering::Greater
 }

 #[inline]
 fn tot_le(&self, other: &Self) -> bool {
 self.tot_cmp(other) != Ordering::Greater
 }

 #[inline]
 fn tot_ge(&self, other: &Self) -> bool {
 self.tot_cmp(other) != Ordering::Less
 }
}

crates/polars-utils/src/total_ord.rs:64

The point of the trait is not the comparison itself but the plumbing: the same generic code can then sort i32, String and f64 with one implementation, because the trait is implemented for all of them.

What the float order actually says

Here is the entire float implementation. It is a macro because f16, f32 and f64 all want it:

 rust
 
 
macro_rules! impl_float_eq_ord {
 ($T:ty) => {
 impl TotalEq for $T {
 #[inline]
 fn tot_eq(&self, other: &Self) -> bool {
 if self.is_nan() {
 other.is_nan()
 } else {
 self == other
 }
 }
 }

 impl TotalOrd for $T {
 #[inline(always)]
 fn tot_cmp(&self, other: &Self) -> Ordering {
 if self.tot_lt(other) {
 Ordering::Less
 } else if self.tot_gt(other) {
 Ordering::Greater
 } else {
 Ordering::Equal
 }
 }

 #[inline(always)]
 fn tot_lt(&self, other: &Self) -> bool {
 !self.tot_ge(other)
 }

 #[inline(always)]
 fn tot_gt(&self, other: &Self) -> bool {
 other.tot_lt(self)
 }

 #[inline(always)]
 fn tot_le(&self, other: &Self) -> bool {
 other.tot_ge(self)
 }

 #[inline(always)]
 fn tot_ge(&self, other: &Self) -> bool {
 // We consider all NaNs equal, and NaN is the largest possible
 // value. Thus if self is NaN we always return true. Otherwise
 // self >= other is correct. If other is not NaN it is trivially
 // correct, and if it is we note that nothing can be greater or
 // equal to NaN except NaN itself, which we already handled earlier.
 self.is_nan() | (self >= other)
 }
 }
 };
}

impl_float_eq_ord!(pf16);
impl_float_eq_ord!(f32);
impl_float_eq_ord!(f64);

crates/polars-utils/src/total_ord.rs:259

Everything interesting is in tot_ge. Because tot_lt is defined as !tot_ge, the whole order is decided by one expression:

self.is_nan() short-circuits to true. A NaN is greater than or equal to everything, including another NaN.
Otherwise self >= other is plain IEEE comparison, and it is false for every comparison against a NaN.

So NaN is the maximum of the order, every NaN is equal to every other NaN, and nothing is greater than a NaN. That is a genuine total order: tot_cmp is antisymmetric, transitive and total. The consequences are the inconvenient ones nobody asked for:

NaN sorts last ascending and first descending. The descending branch of the sort swaps the comparator’s arguments, which turns “NaN is the largest” into “NaN comes first”. A sort(descending=True) that puts NaN at the top of the column is not a bug, it is the order you asked for.
-0.0 and 0.0 are equal. IEEE says -0.0 >= 0.0 is true (they compare equal), so neither is tot_lt the other and tot_cmp returns Equal. The sign of a zero is destroyed by a sort. If you need it, keep it in another column.
NaN == NaN under TotalEq. Two NaNs are the same key for unique, group_by and hash joins. That is a choice, and it is the choice that makes NaN usable as a group key at all.

Equality and ordering have to agree, or a HashMap keyed by floats would break. Rust’s Hash for f64 hashes the bit pattern, so 0.0 and -0.0 would hash differently while comparing equal. Polars solves this by canonicalising before hashing:

 rust
 
 
/// Converts an f64 into a canonical form, where -0 == 0 and all NaNs map to
/// the same value.
#[inline]
pub fn canonical_f64(x: f64) -> f64 {
 // -0.0 + 0.0 becomes 0.0.
 let convert_zero = x + 0.0;
 if convert_zero.is_nan() {
 f64::from_bits(0x7ff8000000000000) // Canonical quiet NaN.
 } else {
 convert_zero
 }
}

crates/polars-utils/src/total_ord.rs:44

x + 0.0 is the cheapest possible way to strip a sign bit: IEEE addition gives +0.0 for -0.0 + +0.0. The same helper is used for f32 and f16, and the hashed value is canonical_f64(x).to_bits(), so -0.0 and 0.0 hash identically and every NaN hashes identically.

To use these floats in code that wants Eq + Hash + Ord — a HashMap key, a BTreeMap key — Polars wraps them in TotalOrdWrap, which is #[repr(transparent)] over the value and forwards Ord to tot_cmp. Integers do not need the wrapper; the ToTotalOrd trait exists precisely so the wrapper can be elided for types that are already totally ordered.

Picking a comparator per dtype

With an order in hand, the sort machinery is small. The comparator is a generic closure, and the dtype decides which one gets instantiated: TotalOrd::tot_cmp for the numeric types, a byte comparison for binary views, a row encoding for the nested ones. The descending flag is applied by swapping the arguments:

 rust
 
 
pub(crate) fn sort_by_branch<T, C>(slice: &mut [T], descending: bool, cmp: C, parallel: bool)
where
 T: Send,
 C: Send + Sync + Fn(&T, &T) -> Ordering,
{
 if parallel {
 RAYON.install(|| match descending {
 true => slice.par_sort_by(|a, b| cmp(b, a)),
 false => slice.par_sort_by(cmp),
 })
 } else {
 match descending {
 true => slice.sort_by(|a, b| cmp(b, a)),
 false => slice.sort_by(cmp),
 }
 }
}

fn sort_unstable_by_branch<T, C>(slice: &mut [T], options: SortOptions, cmp: C)
where
 T: Send,
 C: Send + Sync + Fn(&T, &T) -> Ordering,
{
 if options.multithreaded {
 RAYON.install(|| match options.descending {
 true => slice.par_sort_unstable_by(|a, b| cmp(b, a)),
 false => slice.par_sort_unstable_by(cmp),
 })
 } else {
 match options.descending {
 true => slice.sort_unstable_by(|a, b| cmp(b, a)),
 false => slice.sort_unstable_by(cmp),
 }
 }
}

// Reduce monomorphisation.
fn sort_impl_unstable<T>(vals: &mut [T], options: SortOptions)
where
 T: TotalOrd + Send + Sync,
{
 sort_unstable_by_branch(vals, options, TotalOrd::tot_cmp);
}

crates/polars-core/src/chunked_array/ops/sort/mod.rs:73

Note sort_by_branch is stable and sort_unstable_by_branch is not; both are used, and the choice is deliberate per call site. A stable sort allocates a scratch buffer of half the slice length, so the unstable variant exists to avoid that allocation when the caller does not care about tie order.

Not every dtype reaches a comparator at all:

Boolean is sorted by counting. The number of true values is known from the bitmask, so the sorted column is written as a run of true followed by a run of false. There is no sort, no comparator and no allocation beyond the output bitmap.
BinaryView sorts the views, not the bytes. The 16-byte view structs are permuted in place and the data buffers are left untouched.
Struct and List have no natural order, so they are row-encoded first and the encoded bytes are sorted.

Nulls never reach a comparator. They are partitioned out of the values before the sort and re-attached afterwards:

 rust
 
 
fn partition_nulls<T: Copy>(
 values: &mut [T],
 mut validity: Option<Bitmap>,
 options: SortOptions,
) -> (&mut [T], Option<Bitmap>) {
 let partitioned = if let Some(bitmap) = &validity {
 // Partition null last first
 let mut out_len = 0;
 for idx in bitmap.true_idx_iter() {
 unsafe { *values.get_unchecked_mut(out_len) = *values.get_unchecked(idx) };
 out_len += 1;
 }
 let valid_count = out_len;
 let null_count = values.len() - valid_count;
 validity = Some(create_validity(
 bitmap.len(),
 bitmap.unset_bits(),
 options.nulls_last,
 ));

 // Views are correctly partitioned.
 if options.nulls_last {
 &mut values[..valid_count]
 }
 // We need to swap the ends.
 else {
 // swap nulls with end
 let mut end = values.len() - 1;

 for i in 0..null_count {
 unsafe { *values.get_unchecked_mut(end) = *values.get_unchecked(i) };
 end = end.saturating_sub(1);
 }
 &mut values[null_count..]
 }
 } else {
 values
 };
 (partitioned, validity)
}

crates/polars-core/src/chunked_array/ops/sort/mod.rs:41

The null mask is turned into a run of valid values at the front, the values are sorted, and a new validity bitmap is written to match where the nulls ended up. nulls_last is not a comparator flag; it is a partition decision made before the sort starts. This is why the float order can be as weird as it is: nulls are not part of it.

The same philosophy shows up in crates/polars-utils/src/sort.rs, where the comparison helper that the multi-key sort uses keeps nulls out of the value comparison as well:

 rust
 
 
/// Compare two values with support for sort direction and nulls position.
///
/// # Panics
///
/// Panics if `T::partial_cmp(lhs, rhs)` returns `None`.
#[inline]
pub fn reorder_cmp<T: PartialOrd + IsNull>(
 lhs: &T,
 rhs: &T,
 descending: bool,
 nulls_last: bool,
) -> Ordering {
 match PartialOrd::partial_cmp(lhs, rhs).expect("expected total ordering") {
 Ordering::Equal => Ordering::Equal,
 _ if lhs.is_null() && nulls_last => Ordering::Greater,
 _ if rhs.is_null() && nulls_last => Ordering::Less,
 _ if lhs.is_null() => Ordering::Less,
 _ if rhs.is_null() => Ordering::Greater,
 ord if descending => ord.reverse(),
 ord => ord,
 }
}

crates/polars-utils/src/sort.rs:119

Note the order of the arms: the null rules are applied before the descending reversal, so nulls_last and descending do not interact. A null is either smaller than everything or larger than everything, never “reversed”.

That file also holds the one place where Polars sorts a scratch buffer of (value, index) pairs by hand, which is a neat trick worth reading: the pairs are sorted, and then the value half of the memory is overwritten with the index half, in place, because the index is smaller than the value:

 rust
 
 
pub fn arg_sort_ascending<'a, T: TotalOrd + Copy + 'a, Idx, I: IntoIterator<Item = T>>(
 v: I,
 scratch: &'a mut Vec<u8>,
 n: usize,
) -> &'a mut [Idx]
where
 Idx: FromPrimitive + Copy,
{
 let upper_bound = size_of::<(T, Idx)>() * n + align_of::<(T, Idx)>();
 scratch.reserve(upper_bound);
 let scratch_slice = unsafe {
 let cap_slice = scratch.spare_capacity_mut();
 let (_, scratch_slice, _) = cap_slice.align_to_mut::<MaybeUninit<(T, Idx)>>();
 &mut scratch_slice[..n]
 };

 for ((i, v), dst) in v.into_iter().enumerate().zip_eq(scratch_slice.iter_mut()) {
 *dst = MaybeUninit::new((v, Idx::from_usize(i).unwrap()));
 }

 let scratch_slice = unsafe { assume_init_mut(scratch_slice) };
 scratch_slice.sort_by(|key1, key2| key1.0.tot_cmp(&key2.0));

 // Now we write the indexes in the same array. So from (T, Idxsize) to just IdxSize.
 unsafe {
 let src = scratch_slice.as_ptr();
 let (_, scratch_slice_aligned_to_idx, _) = scratch_slice.align_to_mut::<Idx>();
 let dst = scratch_slice_aligned_to_idx.as_mut_ptr();

 for i in 0..n {
 dst.add(i).write((*src.add(i)).1);
 }

 &mut scratch_slice_aligned_to_idx[..n]
 }
}

crates/polars-utils/src/sort.rs:15

The scratch buffer is a Vec<u8> that is reused across calls, and the alignment dance is what lets the same allocation hold (T, Idx) pairs on the way in and bare Idx values on the way out.

What a chunked sort actually costs

Here is the part that surprised me. A ChunkedArray is a list of chunks, but the numeric sort does not sort the chunks and merge them. It flattens everything into one vector:

 rust
 
 
fn sort_with_numeric<T>(ca: &ChunkedArray<T>, options: SortOptions) -> ChunkedArray<T>
where
 T: PolarsNumericType,
{
 sort_with_fast_path!(ca, options);
 if ca.null_count() == 0 {
 let mut vals = ca.to_vec_null_aware().left().unwrap();

 sort_impl_unstable(vals.as_mut_slice(), options);

 let mut ca = ChunkedArray::from_vec(ca.name().clone(), vals);
 let s = if options.descending {
 IsSorted::Descending
 } else {
 IsSorted::Ascending
 };
 ca.set_sorted_flag(s);
 ca
 } else {
 let null_count = ca.null_count();
 let len = ca.len();

 let mut vals = Vec::with_capacity(ca.len());

 if !options.nulls_last {
 let iter = std::iter::repeat_n(T::Native::default(), null_count);
 vals.extend(iter);
 }

 ca.downcast_iter().for_each(|arr| {
 let iter = arr.iter().filter_map(|v| v.copied());
 vals.extend(iter);
 });
 let mut_slice = if options.nulls_last {
 &mut vals[..len - null_count]
 } else {
 &mut vals[null_count..]
 };

 sort_impl_unstable(mut_slice, options);

 if options.nulls_last {
 vals.extend(std::iter::repeat_n(T::Native::default(), ca.null_count()));
 }

 let arr = PrimitiveArray::new(
 T::get_static_dtype().to_arrow(CompatLevel::newest()),
 vals.into(),
 Some(create_validity(len, null_count, options.nulls_last)),
 );
 let mut new_ca = ChunkedArray::with_chunk(ca.name().clone(), arr);
 let s = if options.descending {
 IsSorted::Descending
 } else {
 IsSorted::Ascending
 };
 new_ca.set_sorted_flag(s);
 new_ca
 }
}

crates/polars-core/src/chunked_array/ops/sort/mod.rs:189

So sorting an f64 column of n rows copies 8n bytes out of the chunk buffers into a fresh Vec, sorts that vector in place, and hands the vector to a new Buffer — the buffer takes ownership, so there is no second copy, but the first one is unavoidable: the sort works on a contiguous array, and a chunked array is not contiguous. The chunk boundaries are irrelevant to the result; they only matter for how the data was laid out in memory.

The arg_sort path is more expensive, because it has to carry the index along with the value:

 rust
 
 
 let mut vals = Vec::with_capacity(len - null_count);
 let mut nulls_idx = Vec::with_capacity(null_cap);
 let mut count: IdxSize = 0;

 for arr_iter in iters {
 let iter = arr_iter.into_iter().filter_map(|v| {
 let i = count;
 count += 1;
 match v {
 Some(v) => Some((i, v)),
 None => {
 // SAFETY: we allocated enough.
 unsafe { nulls_idx.push_unchecked(i) };
 None
 },
 }
 });
 vals.extend(iter);
 }

 let vals = if let Some(limit) = options.limit {
 // Overwrite output len.
 len = std::cmp::min(limit as usize, len);
 sort_limit_impl(&mut vals, len, options)
 } else {
 sort_impl(vals.as_mut_slice(), options);
 vals.as_slice()
 };

crates/polars-core/src/chunked_array/ops/sort/arg_sort.rs:150

A (IdxSize, f64) pair is 16 bytes with alignment 8, so an argsort of an f64 column materialises 16n bytes before the first comparison happens. And because sort_impl uses the stable sort_by, Rust’s stable sort adds another n/2 elements of scratch space — about 8n more bytes. The unstable variant is used for the values-only sort precisely because it does not need that scratch buffer.

That is the real price of “sort the chunks and merge them” being avoided: a single global sort of a flattened vector, with one big allocation. A k-way merge of per-chunk sorts would have cost O(n log c) comparisons instead of O(n log n), but it would also have needed the chunks to stay alive and the merge to run over c cursors. Polars does not do that, and the reason shows up in the streaming engine.

The streaming engine sorts by partitioning, not by merging

When the sort cannot fit in memory, the streaming engine takes a different route. Instead of one sort over everything, it samples the key column, picks split keys from the sample, and routes every row into a bucket:

 rust
 
 
/// Computes the `b - 1` split keys separating `b` equally sized buckets.
///
/// The result is sorted in the order the buckets are laid out, so descending
/// when `descending` is set, and contains no nulls. Equal neighbours are kept:
/// they are what marks a key value that holds a whole bucket by itself.
pub fn split_keys(
 samples: Vec<Series>,
 dtype: &DataType,
 descending: bool,
 b: usize,
) -> PolarsResult<Series> {
 let samples: Vec<Series> = samples.into_iter().filter(|s| !s.is_empty()).collect();
 if samples.is_empty() || b <= 1 {
 return Ok(Series::new_empty(PlSmallStr::EMPTY, dtype));
 }

 let s = concat(samples).drop_nulls().sort_with(SortOptions {
 descending,
 nulls_last: false,
 multithreaded: true,
 maintain_order: false,
 limit: None,
 })?;

 let n = s.len();
 if n == 0 {
 return Ok(Series::new_empty(PlSmallStr::EMPTY, dtype));
 }

 let idxs: Vec<IdxSize> = (1..b).map(|j| (j * n / b) as IdxSize).collect();
 s.take_slice(&idxs)
}

crates/polars-stream/src/nodes/sort/sample.rs:135

Each bucket is then sorted on its own, and the buckets are emitted in key order. There is no merge step at all, because the buckets hold disjoint key ranges: the output is a concatenation of sorted runs, not a merge of them. The classifier is a branchless search tree over the split keys in Eytzinger layout, so classifying a row is log2(b) comparisons with no branch mispredictions. A bucket bounded by two equal split keys holds a single key value and is already sorted, so it is emitted without sorting at all.

This is the answer to “how do you avoid materialising the whole column”: you do not sort it as one thing, you cut it into key ranges first. The cost is that the split keys come from a sample, so the buckets are only roughly equal-sized; a heavy-tailed key column can put most of the rows in one bucket, which is exactly the case the “single key value” fast path covers.

Where a real merge does happen

One place Polars merges two sorted streams is the sort-merge join. If both sides of a join are already sorted, the hash table can be replaced by a linear scan of the two sorted runs:

 rust
 
 
 let no_nulls = s_left.null_count() == 0 && s_right.null_count() == 0;
 match (s_left.is_sorted_flag(), s_right.is_sorted_flag(), no_nulls) {
 (IsSorted::Ascending, IsSorted::Ascending, true) if is_numeric => {
 if verbose {
 eprintln!("inner join: keys are sorted: use sorted merge join");
 }
 Ok((par_sorted_merge_inner_no_nulls(s_left, s_right), true))
 },

crates/polars-ops/src/frame/join/hash_join/sort_merge.rs:235

The merge itself is O(n + m) comparisons, which is far cheaper than a hash join’s random memory access pattern — but only if the sortedness is already there. If one side is not sorted, Polars sorts it (paying O(n log n) plus a full copy of the values) and then undoes the permutation on the resulting indices. The engine only takes that route when the size ratio between the two sides makes the extra sort worth it, which is a decision made with a float comparison of lengths and an environment variable override.

The bottom line

The float order in Polars is not the IEEE order, and it is not the “NaN goes last, whatever that means” order either. It is a deliberate total order with three properties worth memorising:

NaN is the maximum. sort(descending=True) puts NaN first.
-0.0 and 0.0 are the same value, in sorting and in hashing.
Every NaN is the same value, so unique collapses them and group_by treats them as one group.

The sort machinery around that order is unremarkable and worth knowing anyway: nulls are partitioned out before sorting rather than handled in the comparator, descending is implemented by swapping comparator arguments, the numeric sort flattens the chunks into one vector (one full copy of the values), an argsort materialises 16 bytes per row for the (index, value) pairs plus the stable sort’s scratch buffer, booleans are sorted by counting, and the streaming engine avoids the global sort altogether by partitioning the key range and emitting sorted buckets back to back.

If you want to see the order for yourself:

 python
 
 
import polars as pl

df = pl.DataFrame({"x": [1.0, float("nan"), -0.0, 0.0, float("nan")]})
print(df.sort("x")) # [1.0, -0.0, 0.0, nan, nan]
print(df.sort("x", descending=True)) # [nan, nan, 1.0, 0.0, -0.0]

Those two lines are the whole article. The two NaNs are adjacent and indistinguishable, the two zeros are adjacent because Polars considers them equal, and in the descending sort the NaNs are first — the largest value under the order that sort actually implements. The relative order of the two zeros is not guaranteed, incidentally: a values-only sort is an unstable sort, so the tie is broken arbitrarily.

Polars Parses JSON Without Building a DOM

Jay Hubbert — Wed, 21 Oct 2026 07:30:00 +0000

TL;DR: A JSON DOM is a tree of owned values, and building one costs an allocation per node plus a pass to walk it afterwards. Polars never builds one: it parses a line into simd_json::BorrowedValues that borrow the source bytes, then walks the object’s keys once and pushes each value straight into the builder of the column that key belongs to. The schema is what makes this possible — the set of columns is known before the first row is parsed, so there is a destination for every value.

The name “deserializer” is misleading in most JSON libraries. serde_json::Value is a DOM: a tree of String, Vec and Map nodes, each one heap-allocated, each one owned. Parsing a file into it means allocating a node per value, and then walking the tree a second time to copy the leaves into columns. Both passes are paid for every byte of the file, and the tree is pure overhead: the data is only wanted in columnar form.

Polars’ JSON reader sidesteps this in three steps: parse into borrowed values, use the schema to route each value to a column, and keep the number of copies to one per string.

Where the code lives

The reader is split across two crates, which is worth knowing before you go looking for it:

crates/polars-json holds the value-level machinery: json::deserialize (values to Arrow arrays), json::infer (type inference), json::ordered (the order-preserving tape conversion for Map columns), and ndjson::deserialize (a row iterator to an Arrow array).
crates/polars-io holds the file-level machinery: ndjson/core.rs (the chunked, parallel reader), ndjson/buffer.rs (the per-column builders), and json/mod.rs (the JsonReader front end).

Polars used to route JSON through arrow2’s json_deserializer module — a hand-written parser with its own Buffer/Bytes value tree. That was replaced by simd-json in 2023 (“use simd-json for all json parsing”, #8922), and today the value parsing is simd-json’s. What plays the role of the old visitor now is polars-io/src/ndjson/buffer.rs: a set of per-column builders that the parsed values are pushed into, one column at a time.

Splitting the file into lines

The first step is embarrassingly simple, and the comment explains why it is not the obvious thing (crates/polars-io/src/ndjson/core.rs:223):

 rust
 
 
pub fn json_lines(bytes: &[u8]) -> impl Iterator<Item = &[u8]> {
 // This previously used `serde_json`'s `RawValue` to deserialize chunks without really deserializing them.
 // However, this convenience comes at a cost. serde_json allocates and parses and does UTF-8 validation, all
 // things we don't need since we use simd_json for them. Also, `serde_json::StreamDeserializer` has a more
 // ambitious goal: it wants to parse potentially *non-delimited* sequences of JSON values, while we know
 // our values are line-delimited. Turns out, custom splitting is very easy, and gives a very nice performance boost.
 bytes
 .split(|&byte| byte == b'\n')
 .filter(|bytes| is_json_line(bytes))
}

serde_json::RawValue was the old trick for “give me the raw text of the next value without parsing it” — it did the job, but it allocated a Box<str> per value and re-validated UTF-8. Since NDJSON is line-delimited by definition, splitting on \n and skipping blank lines is enough, and it costs nothing.

The borrowed value

Each line is then parsed into a value that borrows the buffer (crates/polars-io/src/ndjson/core.rs:194):

 rust
 
 
#[inline(always)]
fn parse_impl(
 bytes: &[u8],
 buffers: &mut PlIndexMap<BufferKey, Buffer>,
 scratch: &mut Scratch,
 guide: Option<&TapeGuide>,
 ignore_errors: bool,
) -> PolarsResult<usize> {
 scratch.json.clear();
 scratch.json.extend_from_slice(bytes);
 let n = scratch.json.len();
 let parse_err = |e| polars_err!(ComputeError: "error parsing line: {}", e);
 let value = match guide {
 Some(guide) => {
 let tape = simd_json::to_tape_with_buffers(&mut scratch.json, &mut scratch.buffers)
 .map_err(parse_err)?;
 polars_json::json::ordered::tape_to_value(&tape, guide, ignore_errors)?
 },
 None => simd_json::to_borrowed_value_with_buffers(&mut scratch.json, &mut scratch.buffers)
 .map_err(parse_err)?,
 };
 match value {
 simd_json::BorrowedValue::Object(value) => {
 buffers.iter_mut().try_for_each(|(s, inner)| {
 match s.0.map_lookup(&value) {
 Some(v) => inner.add(v)?,
 None => inner.add_null(),
 }
 PolarsResult::Ok(())
 })?;
 },
 _ if ignore_errors => {
 buffers.iter_mut().for_each(|(_, inner)| inner.add_null());
 },
 v => {
 polars_bail!(ComputeError: "NDJSON line expected to contain JSON object: {v}");
 },
 };
 Ok(n)
}

Three things to notice.

First, to_borrowed_value_with_buffers parses into a value whose strings point into scratch.json, the buffer that holds the line. The scratch buffer is reused for every line, so nothing is allocated per line for the parse itself.

Second, the routing is a single loop over the columns, not over the keys of the object. For each column the reader looks up the key in the parsed object; if the key is missing, the column gets a null. This is the opposite of the DOM approach, where you would walk the object’s keys and look up the column. Doing it this way means the number of lookups is the number of columns, regardless of how many keys the line has, and a line with extra keys costs nothing.

Third, the map lookup is s.0.map_lookup(&value), where s.0 is a simd_json::KnownKey. The key is hashed once, when the schema is turned into buffers, and the hash is reused for every line (crates/polars-io/src/ndjson/buffer.rs:17):

 rust
 
 
#[derive(Debug, Clone, PartialEq)]
pub(crate) struct BufferKey<'a>(pub(crate) KnownKey<'a>);
impl Eq for BufferKey<'_> {}

impl Hash for BufferKey<'_> {
 fn hash<H: Hasher>(&self, state: &mut H) {
 self.0.key().hash(state)
 }
}

The Hash impl is the interesting part: it hashes the key string, not the KnownKey itself, which keeps the hash consistent with KnownKey’s equality. The KnownKey carries the precomputed hash that map_lookup uses at parse time. The buffers are stored in a PlIndexMap keyed by BufferKey, which keeps the column order stable and gives O(1) lookup by name.

The column builder

Each column gets a builder chosen by its dtype (crates/polars-io/src/ndjson/buffer.rs:152):

 rust
 
 
pub(crate) fn init_buffers(
 schema: &Schema,
 capacity: usize,
 ignore_errors: bool,
) -> PolarsResult<PlIndexMap<BufferKey<'_>, Buffer<'_>>> {
 schema
 .iter()
 .map(|(name, dtype)| {
 let av_buf = (dtype, capacity).into();
 let key = KnownKey::from(name.as_str());
 Ok((
 BufferKey(key),
 Buffer {
 name,
 buf: av_buf,
 ignore_errors,
 },
 ))
 })
 .collect()
}

The builder is an AnyValueBuffer, which is a per-dtype append-only buffer. The add method is a match on that buffer’s variant (crates/polars-io/src/ndjson/buffer.rs:54):

 rust
 
 
 #[inline]
 pub(crate) fn add(&mut self, value: &Value) -> PolarsResult<()> {
 use AnyValueBuffer::*;
 match &mut self.buf {
 _ if value.is_null() => {
 self.buf.add(AnyValue::Null);
 Ok(())
 },
 Boolean(buf) => {
 match value {
 Value::Static(StaticNode::Bool(v)) => buf.append_value(*v),
 Value::Static(StaticNode::Null) => buf.append_null(),
 _ if self.ignore_errors => buf.append_null(),
 v => {
 polars_bail!(ComputeError: "cannot parse '{}' ({}) as Boolean", v, v.value_type())
 },
 }
 Ok(())
 },
 Int32(buf) => {
 let v =
 deserialize_numeric::<Int32Type>(value, value.as_i32(), self.ignore_errors)?;
 buf.append_option(v);
 Ok(())
 },

Note value.as_i32(): the borrowed value knows how to narrow an I64 or U64 to an i32 itself, so the numeric conversion is a couple of instructions, not a parse-and-convert. And AnyValue::Null is pushed for a JSON null, so the null mask is maintained by the builder as it goes.

The capacity argument is the row-count hint, and it is where the reader avoids a reallocation storm: it is estimated from the file size and the mean line length before the first line is parsed (crates/polars-io/src/ndjson/core.rs:80):

 rust
 
 
 fn parse_json(&mut self, mut n_threads: usize, bytes: &[u8]) -> PolarsResult<DataFrame> {
 let mut bytes = bytes;
 let mut total_rows = 128;

 if let Some((mean, std)) = get_line_stats_json(bytes, self.sample_size) {
 let line_length_upper_bound = mean + 1.1 * std;

 total_rows = (bytes.len() as f32 / (mean - 0.01 * std)) as usize;

get_line_stats_json samples the first and the 75% mark of the file, computes the mean and standard deviation of the line lengths, and from those estimates the number of rows. It is a heuristic, and the file is truncated to the estimated row count if n_rows was given — the reader accepts that it might be slightly wrong and corrects the row count afterwards.

Chunking and parallelism

The file is cut into byte ranges at line boundaries, and each range is parsed independently (crates/polars-io/src/ndjson/core.rs:395):

 rust
 
 
pub(crate) fn get_file_chunks_json(bytes: &[u8], n_threads: usize) -> Vec<(usize, usize)> {
 let mut last_pos = 0;
 let total_len = bytes.len();
 let chunk_size = total_len / n_threads;
 let mut offsets = Vec::with_capacity(n_threads);
 for _ in 0..n_threads {
 let search_pos = last_pos + chunk_size;

 if search_pos >= bytes.len() {
 break;
 }

 let end_pos = match next_line_position_naive_json(&bytes[search_pos..]) {
 Some(pos) => search_pos + pos,
 None => {
 break;
 },
 };
 offsets.push((last_pos, end_pos));
 last_pos = end_pos;
 }
 offsets.push((last_pos, total_len));
 offsets
}

The cut points are found with memchr on the newline character — the same quote-unaware search as in the CSV reader, and for the same reason: a chunk boundary that lands in the middle of a quoted string is a bug, but a chunk boundary that is off by a few bytes is repaired by the “is this line a complete line” check that follows. The per-chunk work is then run in parallel with rayon, and each chunk builds its own set of column buffers (crates/polars-io/src/ndjson/core.rs:115):

 rust
 
 
 let row_index = self.row_index.as_ref().map(|ri| ri as &RowIndex);
 let (mut dfs, prepredicate_heights) = RAYON.install(|| {
 file_chunks
 .into_par_iter()
 .map(|(start_pos, stop_at_nbytes)| {
 let mut local_df = parse_ndjson(
 &bytes[start_pos..stop_at_nbytes],
 Some(capacity),
 &self.schema,
 self.ignore_errors,
 )?;

Each chunk is parsed into a complete DataFrame and the frames are stacked afterwards. That is a coarser unit of work than the Arrow-style approach of one batch per chunk, and it is why the NDJSON path materialises a DataFrame per chunk rather than a set of Arrow arrays.

Schema: from JSON objects to columns

The schema is what makes the direct-to-column routing possible. For NDJSON each line is an object, and the object is a struct, and the struct’s fields are the columns (crates/polars-io/src/ndjson/mod.rs:29):

 rust
 
 
pub fn infer_schema<R: std::io::BufRead>(
 reader: &mut R,
 infer_schema_len: Option<NonZeroUsize>,
) -> PolarsResult<Schema> {
 let dtypes = polars_json::ndjson::iter_unique_dtypes(reader, infer_schema_len)?;
 let dtype =
 crate::json::infer::dtypes_to_supertype(dtypes.map(|dt| DataType::from_arrow_dtype(&dt)))?;

 if !matches!(&dtype, DataType::Struct(_)) {
 polars_bail!(ComputeError: "NDJSON line expected to contain JSON object: {dtype}");
 }

 let schema = StructArray::get_fields(&dtype.to_arrow(CompatLevel::newest()))
 .iter()
 .map(Into::<Field>::into)
 .collect();
 Ok(schema)
}

Inference reads a sample of lines, infers an Arrow type for each, and coerces the set of types into one supertype. The coercion rules are in crates/polars-json/src/json/infer_schema.rs:71 and are worth a look, because they decide what a mixed-type column becomes:

 rust
 
 
/// Coerce an heterogeneous set of [`ArrowDataType`] into a single one. Rules:
/// * The empty set is coerced to `Null`
/// * `Int64` and `Float64` are `Float64`
/// * Lists and scalars are coerced to a list of a compatible scalar
/// * Structs contain the union of all fields
/// * All other types are coerced to `Utf8`
pub(crate) fn coerce_dtype<A: Borrow<ArrowDataType>>(datatypes: &[A]) -> ArrowDataType {

Int64 and Float64 mix to Float64; two structs merge into a struct with the union of their fields; anything else that does not match becomes a string. Note that this is a value type inference, not a JSON type inference: the schema says “this column is a float”, not “this column holds numbers”.

The other direction: values to an Arrow array

The polars-json crate also offers a path that goes straight to Arrow arrays rather than to per-column builders. deserialize takes a borrowed value and a dtype and produces an array (crates/polars-json/src/json/deserialize.rs:421):

 rust
 
 
pub(crate) fn _deserialize<'a, A: Borrow<BorrowedValue<'a>>>(
 rows: &[A],
 dtype: ArrowDataType,
 allow_extra_fields_in_struct: bool,
) -> PolarsResult<Box<dyn Array>> {
 match &dtype {
 ArrowDataType::Null => {
 if let Some(err_idx) = (0..rows.len())
 .find(|i| !matches!(rows[*i].borrow(), BorrowedValue::Static(StaticNode::Null)))
 {
 check_err_idx(rows, err_idx, "null")?;
 }

 Ok(Box::new(NullArray::new(dtype, rows.len())))
 },

and for a struct, it fans out over the fields of the dtype, pushing each field’s values into its own list and then deserialising that list (crates/polars-json/src/json/deserialize.rs:240):

 rust
 
 
 rows.iter().enumerate().for_each(|(i, row)| {
 match row.borrow() {
 BorrowedValue::Object(values) => {
 let mut n_matched = 0usize;
 for (&key, &mut (_, ref mut inner)) in out_values.iter_mut() {
 if let Some(v) = values.get(key) {
 n_matched += 1;
 inner.push(v)
 } else {
 inner.push(&JSON_NULL_VALUE)
 }
 }

The same trick as in the NDJSON path, at a different granularity: the dtype’s fields drive the iteration, and a missing key becomes an explicit null value.

Rows as an array

There is a third path, used when the caller already has rows as strings: the rows are concatenated into a single JSON array and parsed in one go (crates/polars-json/src/ndjson/deserialize.rs:33):

 rust
 
 
pub fn deserialize_iter<'a>(
 rows: impl Iterator<Item = &'a str>,
 dtype: ArrowDataType,
 guide: Option<&ArrowDataType>,
 buf_size: usize,
 count: usize,
 allow_extra_fields_in_struct: bool,
) -> PolarsResult<ArrayRef> {
 // The rows are parsed as one JSON array.
 let guide_dtype = guide.map(|guide| guide.clone().to_large_list(true));
 let guide = guide_dtype.as_ref().map(TapeGuide::new);
 let mut arr: Vec<Box<dyn Array>> = Vec::new();
 let mut buf = Vec::with_capacity(std::cmp::min(buf_size + count + 2, u32::MAX as usize));
 buf.push(b'[');

and then, per row, buf.extend_from_slice(row.as_bytes()); buf.push(b','). The buffer is flushed and parsed whenever it approaches u32::MAX, because that is the largest array length the parser can represent. The result is a list of structs, which the caller then transposes into columns.

The JSON (array) path

A JSON file that is one big array is a different problem: there are no lines to split, and the whole document has to be parsed at once. The reader parses the document into a borrowed value, infers a struct type from it, and deserialises the whole thing into one Arrow array (crates/polars-io/src/json/mod.rs:288):

 rust
 
 
 let (tape, json_value) = if map_guided {
 let tape = simd_json::to_tape(json_bytes).map_err(to_compute_err)?;
 let json_value = self
 .schema
 .is_none()
 .then(|| polars_json::json::ordered::tape_to_default_value(&tape));
 (Some(tape), json_value)
 } else {
 let json_value =
 simd_json::to_borrowed_value(json_bytes).map_err(to_compute_err)?;
 (None, Some(json_value))
 };

Note that the whole document is parsed into a value here, so this path does build a tree — but a tree of borrowed values, not a tree of owned ones. The strings still point into the buffer, and the columns are then filled from that tree in one pass. The alternative, streaming a JSON array element by element, would need a different parser; the format does not make it easy, and the document is assumed to fit in memory anyway.

Map order

One wrinkle: simd_json’s Value::Object switches to a hash map above 32 members, which loses the order of the keys and any duplicate keys. Polars needs the order for Map columns, so it parses to a tape and converts that itself (crates/polars-json/src/json/ordered.rs:1):

 rust
 
 
//! Tape conversion that keeps the source order of `Map` objects.
//!
//! [`Value::Object`] switches to an unordered hash table above 32 members, which loses
//! entry order and duplicate keys. This conversion emits every object at a `Map` position as
//! `[{"key": k, "value": v}, ...]` in tape order instead.

The TapeGuide is built from the target dtype, so it knows which positions in the document are Maps and which are plain structs; everything else falls back to the default conversion.

The bottom line

No DOM is built. The parser produces BorrowedValues that point into the source buffer, and the reader consumes them before the buffer is reused.
The schema is the routing table. Columns are visited, not keys, so a line with extra keys costs nothing, and a line with a missing key becomes a null without a special case.
The number of copies per value is one for strings (into the column’s view buffer) and zero for numbers and booleans, which are converted in place.
Allocations are per chunk, not per row: the line buffer, the parser’s scratch buffers, the column builders and the row-count estimate are all computed once per chunk.
The cost of this design is that the schema must be known before the values are read, so a file whose rows disagree about their keys needs an explicit schema or a supertype, and mixed types within a column are coerced by the inference rules rather than preserved.

Polars Parses CSV With Portable SIMD and simdutf8

Jay Hubbert — Wed, 21 Oct 2026 07:20:00 +0000

TL;DR: The CSV reader never builds an intermediate representation of a row. It scans the buffer 64 bytes at a time with std::simd, finds the field and line separators with a bitmask, and computes “am I inside a quoted field?” with a carry-less multiply that XOR-folds the quote mask. Each field is a &[u8] slice into the original buffer, handed to a parse_bytes implementation chosen by the column’s dtype. Strings are the only values that get copied, and they are validated with simdutf8.

CSV looks like the simplest format in the dataframe world, and that is exactly why it is hard. A field may contain the separator, the newline, doubled quotes, or nothing at all. Rows may be short. Lines may be comments. The first line may be a header, or a BOM may precede it, and it may use a different quote character than the rest of the file.

The naive implementation — bytes.split(b'\n') then line.split(b',') — is wrong for any file with quoted fields, which is most files. Polars’ approach is worth studying because it solves the “where does the field end?” question with bitmasks rather than with a state machine over bytes.

The field splitter

SplitFields is the workhorse: an iterator that yields (&[u8], bool) per field, where the bool says whether the field was quoted and therefore needs unescaping. There are two implementations behind one type: a scalar one and a #[cfg(feature = "simd")] one (crates/polars-io/src/csv/read/splitfields.rs:142):

 rust
 
 
#[cfg(feature = "simd")]
mod inner {
 use std::simd::prelude::*;

 use polars_utils::clmul::prefix_xorsum_inclusive;

 const SIMD_SIZE: usize = 64;
 type SimdVec = u8x64;

The interesting part is the quoted-field branch (crates/polars-io/src/csv/read/splitfields.rs:276):

 rust
 
 
 loop {
 let bytes = unsafe { self.v.get_unchecked(total_idx..) };

 if bytes.len() > SIMD_SIZE {
 let lane: [u8; SIMD_SIZE] = unsafe {
 bytes
 .get_unchecked(0..SIMD_SIZE)
 .try_into()
 .unwrap_unchecked()
 };
 let simd_bytes = SimdVec::from(lane);
 let has_eol = simd_bytes.simd_eq(self.simd_eol_char);
 let has_sep = simd_bytes.simd_eq(self.simd_separator);
 let quote_mask = simd_bytes.simd_eq(self.simd_quote_char).to_bitmask();
 let mut end_mask = (has_sep | has_eol).to_bitmask();

 let mut not_in_quote_field = prefix_xorsum_inclusive(quote_mask);

 if not_in_field_previous_iter {
 not_in_quote_field = !not_in_quote_field;
 }
 not_in_field_previous_iter =
 (not_in_quote_field & (1 << (SIMD_SIZE - 1))) > 0;
 end_mask &= not_in_quote_field;

Read that carefully, because it is the whole trick. has_sep | has_eol is a mask of “bytes that could end a field”. quote_mask is a mask of “bytes that are quotes”. prefix_xorsum_inclusive(quote_mask) turns the quote mask into a mask of “the number of quotes seen so far is odd”, i.e. “we are inside a quoted field” — for every position in the 64-byte block, in one instruction sequence. Masking the candidate ends with that gives the positions that are really field ends.

The prefix XOR is a carry-less multiplication by all-ones (crates/polars-utils/src/clmul.rs:89):

 rust
 
 
// Computes for each bit i the XOR of bits[0..=i].
#[inline]
pub fn prefix_xorsum_inclusive(x: u64) -> u64 {
 #[cfg(all(target_arch = "x86_64", target_feature = "pclmulqdq"))]
 return intel_clmul64(x, u64::MAX);

 #[cfg(all(
 target_arch = "aarch64",
 target_feature = "neon",
 target_feature = "aes"
 ))]
 return arm_clmul64(x, u64::MAX);

 #[allow(unreachable_code)]
 portable_prefix_xorsum_inclusive(x)
}

On x86 with pclmulqdq that is a single PCLMULQDQ; on aarch64 with the crypto extensions it is a vmull_p64; everywhere else it falls back to six shift-and-xor steps. The not_in_field_previous_iter flag carries the parity across block boundaries, so a quoted field that spans several 64-byte blocks still terminates at the right place.

One detail that makes this fast: the mask of valid ends found in a block is cached in previous_valid_ends, so the next call to next() pops the earliest bit without re-scanning (crates/polars-io/src/csv/read/splitfields.rs:222):

 rust
 
 
 // Then check cached value as this is hot.
 if self.previous_valid_ends != 0 {
 let pos = self.previous_valid_ends.trailing_zeros() as usize;
 self.previous_valid_ends >>= (pos + 1) as u64;

For an unquoted field the SIMD path is even simpler: there is no parity to compute, so the mask is just “separator or end of line” (crates/polars-io/src/csv/read/splitfields.rs:365):

 rust
 
 
 // Start of an unenclosed field
 let mut total_idx = 0;

 loop {
 let bytes = unsafe { self.v.get_unchecked(total_idx..) };

 if bytes.len() > SIMD_SIZE {
 let lane: [u8; SIMD_SIZE] = unsafe {
 bytes
 .get_unchecked(0..SIMD_SIZE)
 .try_into()
 .unwrap_unchecked()
 };
 let simd_bytes = SimdVec::from(lane);
 let has_eol_char = simd_bytes.simd_eq(self.simd_eol_char);
 let has_separator = simd_bytes.simd_eq(self.simd_separator);
 let has_any_mask = (has_separator | has_eol_char).to_bitmask();

The line splitter

SplitLines does the same thing for newlines, with the same quote-parity trick (crates/polars-io/src/csv/read/parser.rs:594):

 rust
 
 
 loop {
 let bytes = unsafe { self.v.get_unchecked(self.total_index..) };
 if bytes.len() > SIMD_SIZE {
 let lane: [u8; SIMD_SIZE] = unsafe {
 bytes
 .get_unchecked(0..SIMD_SIZE)
 .try_into()
 .unwrap_unchecked()
 };
 let simd_bytes = SimdVec::from(lane);
 let eol_mask = simd_bytes.simd_eq(self.simd_eol_char).to_bitmask();

 let valid_eols = if self.quoting {
 let quote_mask = simd_bytes.simd_eq(self.simd_quote_char).to_bitmask();
 let mut not_in_quote_field = prefix_xorsum_inclusive(quote_mask);

 if not_in_field_previous_iter {
 not_in_quote_field = !not_in_quote_field;
 }
 not_in_field_previous_iter = (not_in_quote_field & (1 << (SIMD_SIZE - 1))) > 0;
 eol_mask & not_in_quote_field
 } else {
 eol_mask
 };

When the file has no quote character configured, the whole quote machinery disappears and the mask is just “where are the newlines” — the if self.quoting branch is a compile-time-ish constant per file, and the branch predictor never gets it wrong.

Counting lines without parsing them

count(*) on a CSV file must not parse a single value, but it still has to respect quoting: a newline inside a quoted field is not a row boundary. The trick is to compute the answer for both possible starting states at once — “as if the chunk starts outside a string” and “as if it starts inside one” — and then pick the right one when the chunks are stitched together (crates/polars-io/src/csv/read/parser.rs:768):

 rust
 
 
 #[cfg(feature = "simd")]
 {
 // 0 if even number of quotes seen so far, u64::MAX otherwise.
 let mut global_quote_parity_mask = 0;
 while scan_offset + 64 <= bytes.len() {
 let block: [u8; 64] = unsafe {
 bytes
 .get_unchecked(scan_offset..scan_offset + 64)
 .try_into()
 .unwrap_unchecked()
 };
 let simd_bytes = SimdVec::from(block);
 let eol_mask = simd_bytes.simd_eq(self.simd_eol_char).to_bitmask();
 if self.quoting {
 let quote_mask = simd_bytes.simd_eq(self.simd_quote_char).to_bitmask();
 let quote_parity =
 prefix_xorsum_inclusive(quote_mask) ^ global_quote_parity_mask;
 global_quote_parity_mask = ((quote_parity as i64) >> 63) as u64;

 let start_outside_string_eol_mask = eol_mask & !quote_parity;
 states[0].newline_count += start_outside_string_eol_mask.count_ones() as usize;

states[0] and states[1] are the two hypotheses. When the chunked reader joins the results, it carries the “am I inside a string at the end of this chunk” bit into the next chunk (crates/polars-io/src/csv/read/parser.rs:200):

 rust
 
 
 let mut n = 0;
 let mut in_string = false;
 for (pair, _) in states {
 n += pair[in_string as usize].newline_count;
 in_string = pair[in_string as usize].end_inside_string;
 }
 n += eof_unterminated_row as usize;
 Ok(n)

This is also why the counting path can be parallel: each chunk is analysed independently, and only the parity bit has to cross the boundary.

The value layer

Splitting is only half of it. Once a field is a &[u8], it has to become a value of the column’s dtype. That is the job of the ParsedBuilder trait (crates/polars-io/src/csv/read/builder.rs:123):

 rust
 
 
trait ParsedBuilder {
 fn parse_bytes(
 &mut self,
 bytes: &[u8],
 ignore_errors: bool,
 _needs_escaping: bool,
 _missing_is_null: bool,
 _time_unit: Option<TimeUnit>,
 ) -> PolarsResult<()>;
}

Numbers go to atoi_simd and fast_float2, which are SIMD parsers for integers and floats respectively (crates/polars-io/src/csv/read/builder.rs:38):

 rust
 
 
impl PrimitiveParser for Float32Type {
 #[inline]
 fn parse(bytes: &[u8]) -> Option<f32> {
 fast_float2::parse(bytes).ok()
 }
}

Strings are where simdutf8 comes in. The Utf8Field builder keeps a MutableBinaryViewArray and a scratch buffer for unescaping (crates/polars-io/src/csv/read/builder.rs:193):

 rust
 
 
#[inline]
pub fn validate_utf8(bytes: &[u8]) -> bool {
 simdutf8::basic::from_utf8(bytes).is_ok()
}

and the parse itself (crates/polars-io/src/csv/read/builder.rs:220):

 rust
 
 
 // note that one branch writes without updating the length, so we must do that later.
 let escaped_bytes = if needs_escaping {
 self.scratch.clear();
 self.scratch.reserve(bytes.len());
 polars_ensure!(bytes.len() > 1 && bytes.last() == Some(&self.quote_char), ComputeError: "invalid csv file\n\nField `{}` is not properly escaped.", std::str::from_utf8(bytes).map_err(to_compute_err)?);

 // SAFETY:
 // we just allocated enough capacity and data_len is correct.
 unsafe {
 let n_written =
 escape_field(bytes, self.quote_char, self.scratch.spare_capacity_mut());
 self.scratch.set_len(n_written);
 }

 self.scratch.as_slice()
 } else {
 bytes
 };

 if matches!(self.encoding, CsvEncoding::LossyUtf8) | ignore_errors {
 // It is important that this happens after escaping, as invalid escaped string can produce
 // invalid utf8.
 let parse_result = validate_utf8(escaped_bytes);

Note the ordering constraint in that comment: the UTF-8 check runs on the unescaped bytes, because the escaped form is not the form that ends up in the column, and an escaped string can be invalid in ways the unescaped one is not. And in the default Utf8 encoding, validate_utf8 is not called per field at all — the reader validates whole chunks up front (crates/polars-io/src/csv/read/read_impl.rs:449):

 rust
 
 
 s.spawn(move |_| {
 if check_utf8 && !super::builder::validate_utf8(b) {
 let mut results = results.lock().unwrap();
 results.push((
 b.as_ptr() as usize,
 Err(polars_err!(ComputeError: "invalid utf-8 sequence")),
 ));
 return;
 }

One simdutf8 pass over 16 MB beats one pass per field, because the per-field call has to re-establish the SIMD state each time.

Where memchr fits

memchr is in the CSV crate, but it is not what makes the reader fast. It is used in the handful of places that need to find a newline without caring about quoting: skipping lines, finding the end of the file’s complete rows, and detecting comment lines (crates/polars-io/src/csv/read/parser.rs:261):

 rust
 
 
/// Find the nearest next line position.
/// Does not check for new line characters embedded in String fields.
pub(super) fn next_line_position_naive(input: &[u8], eol_char: u8) -> Option<usize> {
 let pos = memchr::memchr(eol_char, input)? + 1;
 if input.len() - pos == 0 {
 return None;
 }
 Some(pos)
}

The name says it: naive, i.e. it does not respect quotes. It is used where the caller has already established that quoting is not in play, or where a wrong answer is merely a wrong row count in a count(*) that is already approximate. The hot loops use the portable SIMD path instead, because memchr cannot answer “is this newline inside a quoted field?” in one call.

Copying

The reader is careful about copies. SplitFields yields slices of the original buffer, and the numeric parsers consume those slices in place: no allocation per field, and no copy of the digits. Only strings are copied, once, into the view array’s buffer, and the unescaping scratch buffer is reused across fields of the same column.

The field dispatch itself is where the projection pushdown lands: the parser iterates over all fields but only touches the projected ones (crates/polars-io/src/csv/read/parser.rs:1164):

 rust
 
 
 Some((mut field, needs_escaping)) => {
 let field_len = field.len();

 // +1 is the split character that is consumed by the iterator.
 read_sol += field_len + 1;

 if idx == next_projected as u32 {

and the values of non-projected columns are simply never materialised.

Parallelism

The reader splits the file into chunks of at most 16 MB and parses them in parallel with rayon, but the chunk size is not a constant (crates/polars-io/src/csv/read/read_impl.rs:377):

 rust
 
 
 let n_threads = self
 .n_threads
 .unwrap_or_else(|| RAYON.current_num_threads());

 // This is chosen by benchmarking on ny city trip csv dataset.
 // We want small enough chunks such that threads start working as soon as possible
 // But we also want them large enough, so that we have less chunks related overhead.
 // We minimize chunks to 16 MB to still fit L3 cache.
 //
 // Width-aware adjustment: For wide data (many columns), per-chunk overhead
 // (allocating column buffers) becomes significant. Each chunk must allocate
 // O(n_cols) buffers, so total allocation overhead is O(n_chunks * n_cols).
 // To keep this bounded, we limit n_chunks such that n_chunks * n_cols <= threshold.
 // With threshold ~500K, this gives:
 // - 100 cols: up to 5000 chunks (no practical limit)
 // - 1000 cols: up to 500 chunks
 // - 10000 cols: up to 50 chunks
 // - 30000 cols: up to 16 chunks
 let n_cols = projection.len();
 // Empirically determined to balance allocation overhead and parallelism.
 const ALLOCATION_BUDGET: usize = 500_000;
 let max_chunks_for_width = ALLOCATION_BUDGET / n_cols.max(1);
 let n_parts_hint = std::cmp::min(n_threads * 16, max_chunks_for_width.max(n_threads));
 let chunk_size = std::cmp::min(bytes.len() / n_parts_hint.max(1), 16 * 1024 * 1024);

That is a nice piece of engineering honesty: the chunk count is not only about parallelism, it is about how many column buffers get allocated. A 30 000-column file would otherwise allocate 30 000 builders per chunk.

Chunk boundaries are found with CountLines::find_next, which counts newlines in a window and doubles the window until it finds at least one — the same quote-aware counter, used for chunking rather than for count(*).

Inference reuses the splitter

Schema inference is not a separate parser. The header line is split with the same SplitFields iterator, and the sample lines are split the same way (crates/polars-io/src/csv/read/schema_inference.rs:187):

 rust
 
 
 let record = SplitFields::new(
 line,
 parse_options.separator,
 parse_options.quote_char,
 parse_options.eol_char,
 );

 for (i, (slice, needs_escaping)) in record.enumerate() {

Each field is then classified by a set of regular expressions — boolean, then integer, then float, then date, then string (crates/polars-io/src/csv/read/schema_inference.rs:337):

 rust
 
 
pub fn infer_field_schema(string: &str, try_parse_dates: bool, decimal_comma: bool) -> DataType {
 // when quoting is enabled in the reader, these quotes aren't escaped, we default to
 // String for them
 let bytes = string.as_bytes();
 if bytes.len() >= 2 && *bytes.first().unwrap() == b'"' && *bytes.last().unwrap() == b'"' {

Note the first branch: a value that is still quoted after unescaping is treated as a string, not a number. "123" is a string, 123 is an integer, and getting that wrong is the classic CSV type-inference bug.

The types found in the sample are accumulated per column in a set, and the final dtype is the supertype of the set — so a column that is an integer in the first ten thousand rows and a float in the next ten becomes a float, and a column that is sometimes a string becomes a string. The cost of inference is therefore one pass over infer_schema_length rows, with a regex per field.

Ragged rows and null values

Malformed data is common enough that the reader has to have a policy for it. A row with fewer fields than the schema gets nulls for the missing ones, and the reader notices this by comparing the number of fields it saw with the number it expected (crates/polars-io/src/csv/read/parser.rs:1179):

 rust
 
 
 // there can be lines that miss fields (also the comma values)
 // this means the splitter won't process them.
 // We traverse them to read them as null values.
 while processed_fields < projection.len() {
 debug_assert!(processed_fields < buffers.len());
 let buf = unsafe {
 // SAFETY: processed fields index can never exceed the projection indices.
 buffers.get_unchecked_mut(processed_fields)
 };
 buf.add_null(!parse_options.missing_is_null);
 processed_fields += 1;
 }

A row with more fields than the schema is an error unless truncate_ragged_lines is set, and the error message carries the byte offset of the offending field, which is why the parser keeps a pointer to the start of the chunk rather than only an index.

The null_values option is checked per field, before the value is parsed, against a compiled matcher for the column’s dtype (crates/polars-io/src/csv/read/parser.rs:1196):

 rust
 
 
 // if we have null values argument, check if this field equal null value
 if let Some(null_values) = null_values {
 let field = if needs_escaping && !field.is_empty() {
 unsafe { field.get_unchecked(1..field.len() - 1) }
 } else {
 field
 };

 // SAFETY:
 // process fields is in bounds
 add_null = unsafe { null_values.is_null(field, idx as usize) }
 }

The bottom line

The parser is a zero-copy splitter plus a per-dtype value parser. Fields are &[u8] slices into the mmap’d buffer; the numeric parsers read them in place; only strings are copied.
Field and line boundaries are found with 64-byte SIMD blocks and a carry-less multiply that computes quote parity for all 64 positions at once. The mask of valid ends is cached, so a row with 20 fields costs one block scan, not 20.
simdutf8 validates the string payload — per chunk in the default encoding, per field only when the encoding is lossy or errors are being ignored.
memchr is used for the quote-unaware lookups (line skipping, comment detection, complete-row detection), not for the hot splitting loops.
The trade-off of the SIMD path: the quote-parity logic assumes quotes are balanced by toggling, which is what RFC 4180 says, but a file with a stray quote in an unquoted field will shift the parity. The scalar path has the same assumption, so this is a correctness constraint of the format, not of the implementation.

Polars Prunes Parquet Row Groups With Chunk Statistics, Not the Page Index

Jay Hubbert — Wed, 21 Oct 2026 07:10:00 +0000

TL;DR: Polars prunes Parquet row groups with the column chunk statistics that live in the file footer, not with the page index. Those statistics are decoded into a tiny DataFrame with <col>_min, <col>_max and <col>_nc columns, the user’s predicate is evaluated against that DataFrame, and the row groups whose bit is clear are never fetched. Polars writes the page index (column index and offset index) when it writes a file with statistics enabled, but its own reader never reads it back: page-level skipping is not implemented.

Every reader of a Parquet file faces the same question: the predicate says ts > 2026-01-01, which of the row groups in this file can be thrown away without touching the bytes? The answer has to come from somewhere, and the only metadata a reader is guaranteed to have is the footer.

Before diving in, one correction to the usual framing. Polars’ vendored Parquet implementation does write the page index — the column index and the offset index, the structures that hold per-page min/max and per-page byte offsets. If you write a Parquet file with Polars (with statistics enabled) and read it with DuckDB, Spark or DataFusion, those readers can skip pages inside a row group. Polars’ own reader does not do that. It skips whole row groups, using the statistics attached to each column chunk. The distinction matters: a row group is typically 100k+ rows, a page is typically a few thousand, so page-level skipping is a strictly finer-grained optimisation that Polars writes the data for but does not yet consume.

Which reader is the real one

The Parquet code lives in two crates. crates/polars-parquet holds the deserializers, the statistics decoding, and a vendored copy of the parquet crate itself (crates/polars-parquet/src/parquet/, with its own thrift decoder and page reader). The plan — which row groups, which columns, in what order — lives in the streaming engine, in crates/polars-stream/src/nodes/io_sources/parquet/. That split matters: the plan is where the pruning decision is made, and it is not in the crate you would guess from the name.

The plan is built in init_morsel_distributor, in the prefetch task, before a single byte of column data is read (crates/polars-stream/src/nodes/io_sources/parquet/init.rs:197):

 rust
 
 
 let has_row_index = row_index.is_some();
 let row_group_mask = calculate_row_group_pred_pushdown_skip_mask(
 row_group_slice.clone(),
 use_statistics,
 predicate.as_ref(),
 &metadata,
 projected_arrow_fields.clone(),
 row_index,
 verbose,
 task_metrics.as_deref(),
 runtime_range_skipper.as_deref(),
 )
 .await?;

 let mut row_groups = row_groups_to_read(
 &metadata,
 row_group_slice,
 row_group_mask.as_ref(),
 row_offset,
 slice_range,
 );

row_group_slice is the result of slice pushdown: LIMIT 10 over a file with 200 row groups narrows the slice to the one row group that contains row 10, so the statistics machinery never even looks at the other 199. row_group_mask is the result of statistics pushdown: a bitmap with one bit per row group, set where the row group cannot match. row_groups_to_read turns the two into a deque of row groups to fetch (crates/polars-stream/src/nodes/io_sources/parquet/row_group_data_fetch.rs:65):

 rust
 
 
/// The row groups of `row_group_slice` that `row_group_mask` keeps, in file order. The
/// first starts at `row_offset` and `slice_range` is relative to it.
pub(super) fn row_groups_to_read(
 metadata: &FileMetadata,
 row_group_slice: Range<usize>,
 row_group_mask: Option<&Bitmap>,
 mut row_offset: usize,
 mut slice_range: Option<Range<usize>>,
) -> VecDeque<RowGroupToRead> {
 let mut row_groups = VecDeque::with_capacity(row_group_slice.len());
 for (i, idx) in row_group_slice.enumerate() {
 let num_rows = metadata.row_groups[idx].num_rows();
 let slice = slice_range.as_mut().map(|slice_range| {
 let rg_row_start = slice_range.start;
 let rg_row_end = slice_range.end.min(num_rows);

 *slice_range = slice_range.start.saturating_sub(num_rows)
 ..slice_range.end.saturating_sub(num_rows);

 (rg_row_start, rg_row_end - rg_row_start)
 });
 if !row_group_mask.is_some_and(|mask| mask.get_bit(i)) {
 row_groups.push_back(RowGroupToRead {
 idx,
 row_offset,
 slice,
 });
 }
 row_offset = row_offset.saturating_add(num_rows);
 }
 row_groups
}

Note the mask semantics: a set bit means “skip”. The function keeps the row groups whose bit is clear, and accumulates row_offset over the skipped ones too, because the row offset of a row group is its position in the file, not in the result set. That matters for with_row_index.

The columns in the plan

The other half of the plan is which columns to read, and it is a separate mechanism from the row-group pruning. Projection pushdown has already decided which Arrow fields the query needs; the reader then works out, per row group, which byte ranges those fields live in, and fetches only those (crates/polars-stream/src/nodes/io_sources/parquet/row_group_data_fetch.rs:406):

 rust
 
 
fn get_row_group_byte_ranges_for_projection<'a>(
 row_group_metadata: &'a RowGroupMetadata,
 columns: &'a mut dyn Iterator<Item = &PlSmallStr>,
) -> impl Iterator<Item = ParquetResult<std::ops::Range<usize>>> + 'a {
 columns.flat_map(|col_name| {
 row_group_metadata
 .columns_under_root_iter(col_name)
 // `Option::into_iter` so that we return an empty iterator for the
 // `allow_missing_columns` case
 .into_iter()
 .flatten()
 .map(|col| {
 let byte_range = col.byte_range()?;
 Ok(byte_range.start as usize..byte_range.end as usize)
 })
 })
}

columns_under_root_iter is what makes nested schemas work: a projected struct field is not one column but several leaf columns, and the reader has to walk down to the leaves to find their byte ranges. Empty ranges — chunks without pages — are dropped, because they need no I/O.

This is also where the “full projection” shortcut lives: if the query reads every column, the reader skips the per-column walk and just takes the row group’s whole byte range.

The statistics become a DataFrame

Here is the part I find elegant. Polars does not implement a separate “evaluate predicate against min/max bounds” interpreter. Instead it turns the per-row-group statistics into a small DataFrame — one row per row group, one column per statistic — and then runs the user’s own predicate on that DataFrame (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:184):

 rust
 
 
 for projection in projected_arrow_fields.iter() {
 let c = projection.output_name();

 if !skip_batch_columns.contains(c) {
 continue;
 }

 let mut statistics =
 load_parquet_column_statistics(&metadata, row_group_slice.clone(), projection)?;

 // Note: Order is important here. We re-use the transform for the output column, meaning
 // that it may set the column name.
 statistics.min = projection.apply_transform(statistics.min)?;
 statistics.max = projection.apply_transform(statistics.max)?;
 // The null counts of struct fields have the file's field layout. The transform may
 // reorder, rename or insert fields (e.g. by field ID), and casts the leaves, so it
 // does not apply to them: use unknown null counts.
 if let ArrowFieldProjection::Mapped { output_dtype, .. } = projection
 && matches!(projection.arrow_field().dtype(), ArrowDataType::Struct(_))
 {
 statistics.null_count = Column::full_null(
 PlSmallStr::EMPTY,
 statistics.null_count.len(),
 &null_count_dtype(output_dtype),
 );
 }

 let statistics = statistics.with_base_column_name(c);

 columns.extend([statistics.min, statistics.max, statistics.null_count]);
 }

 if let Some(row_index) = row_index {
 let statistics = build_row_index_statistics(&row_index, row_groups_slice)
 .with_base_column_name(&row_index.name);

 columns.extend([statistics.min, statistics.max, statistics.null_count]);
 }

 let statistics_df = DataFrame::new(num_row_groups, columns)?;

 sbp.evaluate_with_stat_df(&statistics_df)

So a filter like pl.col("ts") > cutoff becomes, internally, a filter over a DataFrame with columns ts_min, ts_max and ts_nc. The predicate compiler already knows how to rewrite ts > cutoff into ts_min > cutoff; the skip batch predicate is that rewrite, evaluated over statistics. One code path for “filter a frame” and “filter a set of row groups”, and no second expression language to maintain.

Two details are worth pulling out. The first is the len column: the number of rows in each row group is pushed into the statistics frame as a plain column, so a predicate like pl.len() > 1000 prunes row groups too, without any special case (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:167):

 rust
 
 
 let mut columns = Vec::with_capacity(1 + skip_batch_columns.len() * 3);

 let lengths: Vec<IdxSize> = row_groups_slice
 .iter()
 .map(|rg| rg.num_rows() as IdxSize)
 .collect();

 columns.push(Column::new("len".into(), lengths));

The second is that the statistics are pushed through the projection’s transform before the predicate sees them. A projection can rename a column, reorder struct fields, or cast a leaf (projection.apply_transform), and the predicate was compiled against the transformed column. Applying the transform to the statistics as well is what keeps the two in agreement.

The columns are named <col>_min, <col>_max and <col>_nc (crates/polars-io/src/predicates.rs:363):

 rust
 
 
 columns.extend([
 Column::new_scalar(format_pl_smallstr!("{col}_min"), min, 1),
 Column::new_scalar(format_pl_smallstr!("{col}_max"), max, 1),
 Column::new_scalar(format_pl_smallstr!("{col}_nc"), nc, 1),
 ]);

The result is a bitmap, one bit per row group, and the verbose log prints the savings (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:130):

 rust
 
 
 if verbose && (mask.is_some() || runtime_range_skipper.is_some()) {
 let num_row_groups = row_group_slice.len();
 eprintln!(
 "[ParquetFileReader]: Predicate pushdown: \
 reading {} / {} row groups",
 mask.as_ref().map_or(num_row_groups, |m| m.unset_bits()),
 num_row_groups,
 );
 }

Where the numbers come from

The footer is kept in memory as a buffer, and the statistics are stored as byte ranges into it. Resolving a bound is therefore a slice, not a copy (crates/polars-parquet/src/parquet/metadata/column_chunk_metadata.rs:111):

 rust
 
 
 /// The plain-encoded min and max of the chunk, borrowed from `footer_buf`;
 /// `None` when the chunk has no statistics.
 pub fn raw_bounds<'a>(&self, footer_buf: &'a [u8]) -> Option<RawBounds<'a>> {
 let stats = self.column_chunk.meta_data.as_ref()?.statistics.as_ref()?;
 Some(RawBounds {
 min: stats.min_value.map(|range| range.resolve(footer_buf)),
 max: stats.max_value.map(|range| range.resolve(footer_buf)),
 min_is_exact: stats.is_min_value_exact != Some(false),
 max_is_exact: stats.is_max_value_exact != Some(false),
 })
 }

The is_exact flags are a nice touch: a writer may record a truncated or approximate bound, and an inexact bound is worthless for pruning (it may exclude rows that do match). Polars validates the bounds and then throws away the inexact ones rather than pruning on them.

Before a bound can be compared, the file’s declared sort order has to match the order Polars compares with — an unsigned column whose statistics were collected as signed cannot be used (crates/polars-parquet/src/arrow/read/statistics/bounds.rs:151):

 rust
 
 
/// Whether the `min_value` and `max_value` of a leaf bound the values polars decodes
/// as `dtype`: the file declares an order for the leaf, and it is the one polars
/// compares the decoded values with.
fn bounds_are_usable(
 column_order: ColumnOrder,
 physical_type: &PhysicalType,
 dtype: &ArrowDataType,
) -> bool {
 let Some(required) = required_sort_order(dtype, physical_type) else {
 return false;
 };
 match column_order {
 ColumnOrder::TypeDefinedOrder(order) => order == required,
 // Total order agrees with the float comparison on every non-NaN value.
 ColumnOrder::IEEE754TotalOrder => {
 matches!(physical_type, PhysicalType::Float | PhysicalType::Double)
 },
 ColumnOrder::Unsupported | ColumnOrder::Undefined => false,
 }
}

The same care shows up for timestamps: if the reader multiplies a microsecond timestamp into nanoseconds and the multiplication overflows, the values have no order, so both bounds are required and both must be in range, otherwise the chunk bounds nothing (crates/polars-parquet/src/arrow/read/statistics/bounds.rs:264).

Nested columns need one more step. A struct column is not one Parquet leaf but several, and a predicate on a single struct field should be able to prune on that field’s statistics. load_struct_column_statistics therefore assembles a per-field min/max/null-count structure from the leaves, and gives up — falls back to all-null statistics, which prune nothing — if the leaf count and the struct field count do not line up (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:732).

The “already applied” flag

There is a second level of pruning that happens before the reader ever sees the file: table statistics. If a Parquet dataset has per-file min/max statistics recorded out of band, a predicate can be evaluated against them and whole files skipped. When that happens, the reader must not evaluate the predicate again — or rather, the plan has to know whether there is anything left to evaluate. That is what PredicateFileSkip is for (crates/polars-descriptions/src/lib.rs:21):

 rust
 
 
pub struct PredicateFileSkip {
 /// If `true` the predicate can be skipped at runtime.
 pub no_residual_predicate: bool,
 /// Number of files before skipping
 pub original_len: usize,
}

It is set when the file-level statistics are applied (crates/polars-mem-engine/src/scan_predicate/functions.rs:563):

 rust
 
 
 let predicate_to_readers = (!can_skip_scan_predicate).then_some(&scan_predicate);

 if let Some(skip_files_mask) = skip_files_mask {
 assert_eq!(skip_files_mask.len(), sources.len());

 let predicate_file_skip = PredicateFileSkip {
 no_residual_predicate: predicate_to_readers.is_none(),
 original_len: sources.len(),
 };

and honoured when the pipeline is initialised: with no_residual_predicate set, the predicate is simply not handed to the reader (crates/polars-stream/src/nodes/io_sources/multi_scan/pipeline/initialization.rs:124):

 rust
 
 
 Some(PredicateFileSkip {
 no_residual_predicate: false,
 original_len: _,
 }) => (None, config.predicate.as_ref()),
 Some(PredicateFileSkip {
 no_residual_predicate: true,
 original_len: _,
 }) => (None, None),

The one subtlety is runtime ranges: a predicate with a runtime range is not consumed by the file-level statistics, because the range can change while the scan runs. That is why can_skip_scan_predicate is predicate.runtime_ranges.is_empty() in the hive-partition branch (crates/polars-mem-engine/src/scan_predicate/functions.rs:407).

Ranges that arrive late

The runtime range is the case where a join discovers a filter after the scan has started. The plan cannot prune on it, but the reader can still check each row group against it just before fetching it. RuntimeRangeSkipper resolves the range against the file’s statistics once (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:540):

 rust
 
 
 fn skips_with(&self, checks: &[RangeCheck], row_group_idx: usize) -> bool {
 checks.iter().any(|check| match check {
 RangeCheck::KeepAll => false,
 RangeCheck::SkipAll => true,
 RangeCheck::Bounds {
 leaf,
 lo,
 hi,
 nulls_match,
 } => {
 let leaf = self.hints[*leaf].leaf.as_ref().unwrap();
 let Some((min, max)) = leaf.row_group_bounds(row_group_idx, &self.metadata) else {
 return false;
 };
 let below = lo
 .as_ref()
 .is_some_and(|lo| max.as_ref().is_some_and(|max| max < lo));
 let above = hi
 .as_ref()
 .is_some_and(|hi| min.as_ref().is_some_and(|min| min > hi));
 (below || above)
 && (!nulls_match
 || has_no_nulls(&self.metadata.row_groups[row_group_idx], leaf.leaf_idx))
 },
 })
 }

and the fetcher re-checks the front of the queue before each fetch, so a range that tightened since the plan was made still skips work (crates/polars-stream/src/nodes/io_sources/parquet/row_group_data_fetch.rs:139):

 rust
 
 
 /// Moves past the row groups at the front that the runtime ranges skip.
 fn skip_front(&mut self) {
 while let Some(row_group) = self.row_groups.front()
 && self
 .runtime_range_skipper
 .as_ref()
 .is_some_and(|s| s.skips(row_group.idx))
 {
 self.row_groups.pop_front();
 }
 }

The page index: written, not read

The write side is where the page index shows up. When the footer is written, Polars serialises a column index and an offset index for every column chunk — the column index only if the writer was configured to collect statistics, the offset index unconditionally (crates/polars-parquet/src/parquet/write/file.rs:205):

 rust
 
 
 if self.options.write_statistics {
 // write column indexes (require page statistics)
 self.row_groups
 .iter_mut()
 .zip(self.page_specs.iter())
 .try_for_each(|(group, pages)| {
 group.columns.iter_mut().zip(pages.iter()).try_for_each(
 |(column, pages)| {
 let offset = self.offset;
 if let Some(length) = write_column_index(&mut self.writer, pages)? {
 self.offset += length;
 column.column_index_offset = Some(offset as i64);
 column.column_index_length = Some(length as i32);
 }
 ParquetResult::Ok(())
 },
 )?;
 ParquetResult::Ok(())
 })?;
 };

and the offset index right after it, unconditionally. So a file written by Polars carries per-page min/max and per-page offsets, and any reader that implements the page index can use them.

Polars’ own reader does not. The vendored page reader still carries the comment from the crate it was forked from (crates/polars-parquet/src/parquet/read/page/reader.rs:68):

 rust
 
 
/// A fallible [`Iterator`] of [`CompressedDataPage`]. This iterator reads pages back
/// to back until all pages have been consumed.
///
/// The pages from this iterator always have [`None`] [`crate::parquet::page::CompressedDataPage::selected_rows()`] since
/// filter pushdown is not supported without a
/// pre-computed [page index](https://github.com/apache/parquet-format/blob/master/PageIndex.md).
pub struct PageReader {

and the read path never opens the column index or the offset index: the only references to them in the reader are the metadata accessors that tell you where the index is, not what is in it.

The bottom line

Pruning is per row group, using the per-column-chunk min/max/null-count in the footer. The bounds are borrowed from the footer buffer, not copied, and are dropped if the file’s sort order or an exactness flag makes them unusable.
The predicate is not re-implemented for statistics. It is evaluated on a DataFrame of <col>_min/<col>_max/<col>_nc columns, which means the filter you wrote and the filter that prunes row groups are the same expression.
PredicateFileSkip::no_residual_predicate is the “already applied” flag: it tells the reader that table statistics already consumed the predicate, so the reader must not evaluate it again.
A range published at run time is checked again per row group, immediately before the fetch.
Polars writes the page index (with statistics enabled) so that other engines can prune at page granularity. If you want that from Polars itself, it is not there yet, and the win would be largest on files with many small row groups and highly selective predicates.

Polars Does Interval Joins with a Sort and a Sweep

Jay Hubbert — Wed, 21 Oct 2026 07:00:00 +0000

TL;DR: An interval join asks for pairs of rows whose intervals overlap: left.start <= right.end and right.start <= left.end. Done naively that is a nested loop, O(n*m) comparisons. Polars instead sorts the combined inputs twice — once by the interval start, once by the interval end — and then sweeps, marking right-hand rows in a bit array and letting each left row look up only the right rows that can overlap it.

The pruning device is a filtered bit array: a bitmap of visited right-hand rows plus a coarse summary bitmap, so that sparse regions are skipped 1024 bits at a time.

Polars has two inequality-join code paths, and both are in crates/polars-ops/src/frame/join/iejoin/. The first is the full IEJoin, which handles two inequality operators at once — the classic interval overlap query. The second is a “piecewise merge join”, which handles a single inequality.

Both are described in the source with a reference to the paper they come from:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/mod.rs:527
/// Inequality join. Matches rows between two DataFrames using two inequality operators
/// (one of [<, <=, >, >=]).
/// Based on Khayyat et al. 2015, "Lightning Fast and Space Efficient Inequality Joins"
/// and extended to work with duplicate values.
fn iejoin_tuples(

The problem

An interval join on two tables asks: for every left interval [x1, x2] and every right interval [y1, y2], find the pairs where the intervals overlap, i.e. x1 <= y2 and y1 <= x2. That is two inequality predicates over the same pair of rows.

The naive implementation evaluates both predicates for every pair, which is O(nm) predicate evaluations and O(nm) memory if you materialize the result. For a million rows on each side that is a trillion comparisons.

The trick the inequality-join literature uses is that the two predicates pull in opposite directions. One of them (x1 <= y2) is satisfied by right rows that are “far enough right”; the other (y1 <= x2) is satisfied by right rows that are “not too far right”. If you sort the rows by their start points, the set of right rows that could possibly match a given left row is a contiguous range — and the algorithm’s job is to find that range and to enumerate the right rows inside it that also satisfy the second predicate.

text

 combined rows: L1 order (by x) L2 order (by y, within L1)
 --------------- --------------- ----------------------------
 L1: [l1] [ l1 l2 r1 ] [ r1 l1 l2 ]
 L2: [l2] | | | | | |
 R1: [r1] | | | | | |
 v v v v v v

 sweep over the L2 order:
 r1 -> right row: set its bit in the visited bitmap
 l1 -> left row: binary search the start of the candidate range,
 then emit every set bit from there on

The two sorts

The algorithm concatenates the two sides into single arrays of start points x and end points y, then sorts:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/mod.rs:589
 let mut x = selected_left[0].to_physical_repr().into_owned();
 let left_height = x.len();

 x.extend(&selected_right[0].to_physical_repr())?;
 // Rechunk because we will gather.
 let x = x.rechunk();

 let mut y = selected_left[1].to_physical_repr().into_owned();
 y.extend(&selected_right[1].to_physical_repr())?;
 // Rechunk because we will gather.
 let y = y.rechunk();

 let l1_sort_options = SortOptions::default()
 .with_maintain_order(true)
 .with_nulls_last(false)
 .with_order_descending(l1_descending);
 // Get ordering of x, skipping any null entries as these cannot be matches
 let l1_order = x
 .arg_sort(l1_sort_options)
 .slice(x.null_count() as i64, x.len() - x.null_count());

 let y_ordered_by_x = unsafe { y.take_unchecked(&l1_order) };
 let l2_sort_options = SortOptions::default()
 .with_maintain_order(true)
 .with_nulls_last(false)
 .with_order_descending(l2_descending);
 // Get the indexes into l1, ordered by y values.
 // l2_order is the same as "p" from Khayyat et al.
 let l2_order = y_ordered_by_x.arg_sort(l2_sort_options).slice(
 y_ordered_by_x.null_count() as i64,
 y_ordered_by_x.len() - y_ordered_by_x.null_count(),
 );

Two sorts, and the second one is the subtle one:

L1 order sorts the combined rows by their start point x. The sort direction depends on the first operator: Lt/LtEq want ascending, Gt/ GtEq want descending. The comment above says it explicitly — “the algorithms published in Khayyat et al. have incorrect logic for determining whether to sort descending”, so Polars computes it from the operator.
L2 order sorts the end points, but in the order established by L1. That is: take y, permute it by l1_order, and then sort that permuted array by y. The result is the paper’s p: an ordering of the rows by end point, expressed as indices into the L1 array.

Both sorts use maintain_order(true) — they are stable sorts. Stability is not an optimization here, it is a correctness requirement, because the algorithm relies on rows with equal end points being processed in a well-defined relative order.

Nulls are dropped by slicing the sort result past the null count, since nulls_last(false) puts them first.

The L1 and L2 arrays

The sort results are then turned into two flat arrays of small structs. The L1 array holds one item per row, in L1 order, with a signed row index that encodes which side the row came from:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/l1_l2.rs:28
pub(super) fn build_l1_array<T>(
 ca: &ChunkedArray<T>,
 order: &IdxCa,
 right_df_offset: IdxSize,
) -> PolarsResult<Vec<L1Item<T::Native>>>
where
 T: PolarsNumericType,
{
 assert_eq!(order.null_count(), 0);
 assert_eq!(ca.chunks().len(), 1);
 let arr = ca.downcast_get(0).unwrap();
 // Even if there are nulls, they will not be selected by order.
 let values = arr.values().as_slice();

 let mut array: Vec<L1Item<T::Native>> = Vec::with_capacity(ca.len());

 for order_arr in order.downcast_iter() {
 for index in order_arr.values().as_slice().iter().copied() {
 debug_assert!(arr.get(index as usize).is_some());
 let value = unsafe { *values.get_unchecked(index as usize) };
 let row_index = if index < right_df_offset {
 // Row from LHS
 index as i64 + 1
 } else {
 // Row from RHS
 -((index - right_df_offset) as i64) - 1
 };
 array.push(L1Item { row_index, value });
 }
 }

 Ok(array)
}

The sign of row_index is the whole trick for distinguishing sides: positive means “left row”, negative means “right row”. A single i64 per row instead of a tag plus an index.

The L2 array does not store the end values at all. It only needs to know where a run of equal end points ends:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/l1_l2.rs:95
/// Item in L1 array used in the IEJoin algorithm
#[derive(Clone, Copy, Debug)]
pub(super) struct L1Item<T> {
 /// 1 based index for entries from the LHS df, or -1 based index for entries from the RHS
 pub(super) row_index: i64,
 /// X value
 pub(super) value: T,
}

/// Item in L2 array used in the IEJoin algorithm
#[derive(Clone, Copy, Debug)]
pub(super) struct L2Item {
 /// Corresponding index into the L1 array of
 pub(super) l1_index: IdxSize,
 /// Whether this is the end of a run of equal y values
 pub(super) run_end: bool,
}

The sweep

The sweep is ie_join_impl_t. It walks the L2 order and does one of two things per element: if the element is a right-hand row, mark it as visited; if it is a left-hand row, emit all the visited right-hand rows that satisfy the first operator.

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/mod.rs:74
 if op2.is_strict() {
 // For strict inequalities, we rely on using a stable sort of l2 so that
 // p values only increase as we traverse a run of equal y values.
 // To handle inclusive comparisons in x and duplicate x values we also need the
 // sort of l1 to be stable, so that the left hand side entries come before the right
 // hand side entries (as we mark visited entries from the right hand side).
 for &p in l2_order {
 match_count += unsafe {
 l1_array.process_entry(
 p as usize,
 &mut bit_array,
 op1,
 &mut left_row_idx,
 &mut right_row_idx,
 )
 };

 if slice_end.is_some_and(|end| match_count >= end) {
 break;
 }
 }
 } else {
 let l2_array = build_l2_array(&y_ordered_by_x, l2_order)?;

 // For non-strict inequalities in l2, we need to track runs of equal y values and only
 // check for matches after we reach the end of the run and have marked all rhs entries
 // in the run as visited.
 let mut run_start = 0;

 for i in 0..l2_array.len() {
 // Elide bound checks
 unsafe {
 let item = l2_array.get_unchecked(i);
 let p = item.l1_index;
 l1_array.mark_visited(p as usize, &mut bit_array);

 if item.run_end {
 for l2_item in l2_array.get_unchecked(run_start..i + 1) {
 let p = l2_item.l1_index;
 match_count += l1_array.process_lhs_entry(
 p as usize,
 &bit_array,
 op1,
 &mut left_row_idx,
 &mut right_row_idx,
 );
 }

 run_start = i + 1;

The two branches exist because of < versus <= on the second operator.

With a strict operator (<), a right row with end point e only matches a left row whose end point is strictly greater than e, so processing the right row as soon as it is encountered is fine — as long as the L1 sort is stable, so that equal-x left rows come before equal-x right rows.

With a non-strict operator (<=), a left row with end point e matches every right row with end point e, including the ones that have not been visited yet. So the algorithm cannot check a left row until the whole run of equal end points has been processed. That is what run_end marks, and it is why the L2 array stores a flag per element instead of the values.

process_entry is where the two cases are unified:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/l1_l2.rs:212
impl<T> L1Array for Vec<L1Item<T>>
where
 T: NumericNative,
{
 unsafe fn process_entry(
 &self,
 l1_index: usize,
 bit_array: &mut FilteredBitArray,
 op1: InequalityOperator,
 left_row_ids: &mut Vec<IdxSize>,
 right_row_ids: &mut Vec<IdxSize>,
 ) -> i64 {
 let row_index = self.get_unchecked(l1_index).row_index;
 let from_lhs = row_index > 0;
 if from_lhs {
 find_matches_in_l1(
 self,
 l1_index,
 row_index,
 bit_array,
 op1,
 left_row_ids,
 right_row_ids,
 )
 } else {
 bit_array.set_bit_unchecked(l1_index);
 0
 }
 }

A right row sets its bit; a left row goes looking for matches.

Finding the matches

find_matches_in_l1 is where the pruning happens. Two steps:

Find the position in the L1 array where the candidate range starts. This is a binary search, but an exponential one — the search doubles its window until it brackets the answer, which is much cheaper than a full binary search when the answer is close:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/l1_l2.rs:149
unsafe fn find_search_start_index<T>(
 l1_array: &[L1Item<T>],
 index: usize,
 operator: InequalityOperator,
) -> usize
where
 T: NumericNative,
 T: TotalOrd,
{
 let sub_l1 = l1_array.get_unchecked(index..);
 let value = l1_array.get_unchecked(index).value;

 match operator {
 InequalityOperator::Gt => {
 sub_l1.partition_point_exponential(|a| a.value.tot_ge(&value)) + index
 },
 InequalityOperator::Lt => {
 sub_l1.partition_point_exponential(|a| a.value.tot_le(&value)) + index
 },
 InequalityOperator::GtEq => {
 sub_l1.partition_point_exponential(|a| value.tot_lt(&a.value)) + index
 },
 InequalityOperator::LtEq => {
 sub_l1.partition_point_exponential(|a| value.tot_gt(&a.value)) + index
 },
 }
}

Because L1 is sorted by x, all rows from that position onwards satisfy the first operator. So the remaining question is: which of those rows have already been visited? — that is, which right rows also satisfy the second operator.

Enumerate the set bits of the bit array from that position:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/l1_l2.rs:184
fn find_matches_in_l1<T>(
 l1_array: &[L1Item<T>],
 l1_index: usize,
 row_index: i64,
 bit_array: &FilteredBitArray,
 op1: InequalityOperator,
 left_row_ids: &mut Vec<IdxSize>,
 right_row_ids: &mut Vec<IdxSize>,
) -> i64
where
 T: NumericNative,
 T: TotalOrd,
{
 debug_assert!(row_index > 0);
 let mut match_count = 0;

 // This entry comes from the left hand side DataFrame.
 // Find all following entries in L1 (meaning they satisfy the first operator)
 // that have already been visited (so satisfy the second operator).
 // Because we use a stable sort for l2, we know that we won't find any
 // matches for duplicate y values when traversing forwards in l1.
 let start_index = unsafe { find_search_start_index(l1_array, l1_index, op1) };
 unsafe {
 bit_array.on_set_bits_from(start_index, |set_bit: usize| {
 // SAFETY
 // set bit is within bounds.
 let right_row_index = l1_array.get_unchecked(set_bit).row_index;
 debug_assert!(right_row_index < 0);
 left_row_ids.push((row_index - 1) as IdxSize);
 right_row_ids.push((-right_row_index) as IdxSize - 1);
 match_count += 1;
 })
 };

 match_count
}

The bit array is the “visited right rows” set. Each set bit is a right row that has been swept past in end-point order, i.e. a right row whose end point is small enough to satisfy the second predicate.

The filtered bit array

The naive way to enumerate set bits is to walk the whole bitmap from the start index to the end, which is O(n) per left row — back to O(n*m). The filtered bit array adds a summary level: one bit per 1024-bit chunk, saying “this chunk contains at least one visited row”.

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/filtered_bit_array.rs:38
/// Bit array with a filter to speed up searching for set bits when sparse,
/// based on section 4.1 from Khayyat et al. 2015,
/// "Lightning Fast and Space Efficient Inequality Joins"
pub struct FilteredBitArray {
 bit_array: MutableBitmap,
 filter: MutableBitmap,
}

impl FilteredBitArray {
 const CHUNK_SIZE: usize = 1024;

 pub fn from_len_zeroed(len: usize) -> Self {
 Self {
 bit_array: MutableBitmap::from_len_zeroed(len),
 filter: MutableBitmap::from_len_zeroed(len.div_ceil(Self::CHUNK_SIZE)),
 }
 }

 pub unsafe fn set_bit_unchecked(&mut self, index: usize) {
 self.bit_array.set_unchecked(index, true);
 self.filter.set_unchecked(index / Self::CHUNK_SIZE, true);
 }

 pub unsafe fn on_set_bits_from<F>(&self, start: usize, mut action: F)
 where
 F: FnMut(usize),
 {
 let start_chunk = start / Self::CHUNK_SIZE;
 let mut chunk_offset = start % Self::CHUNK_SIZE;
 for chunk_idx in start_chunk..self.filter.len() {
 if self.filter.get_unchecked(chunk_idx) {
 // There are some set bits in this chunk
 let start = chunk_idx * Self::CHUNK_SIZE + chunk_offset;
 let end = min((chunk_idx + 1) * Self::CHUNK_SIZE, self.bit_array.len());
 for bit_idx in start..end {
 // SAFETY: `bit_idx` is always less than `self.bit_array.len()`
 if self.bit_array.get_unchecked(bit_idx) {
 action(bit_idx);
 }
 }
 }
 chunk_offset = 0;
 }
 }
}

Empty chunks are skipped with a single bit test, so the cost of enumerating matches is proportional to the number of non-empty chunks plus the number of set bits — not to the length of the array. This is the mechanism that makes the sweep sub-quadratic in practice: the “pruning” of the right side is done by the summary bitmap, which skips regions of the L1 array where no right row has been visited yet.

The single-inequality variant

When there is only one operator, the whole L1/L2 machinery is unnecessary. The condition x1 < y2 is monotone in both arguments once both sides are sorted, so a two-cursor scan suffices:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/mod.rs:133
fn piecewise_merge_join_impl_t<T, P>(
 slice: Option<(i64, usize)>,
 left_order: Option<&[IdxSize]>,
 right_order: Option<&[IdxSize]>,
 left_ordered: Series,
 right_ordered: Series,
 mut pred: P,
) -> PolarsResult<(Vec<IdxSize>, Vec<IdxSize>)>
where
 T: PolarsNumericType,
 P: FnMut(&T::Native, &T::Native) -> bool,
{
 let slice_end = slice_end_index(slice);

 let mut left_row_idx: Vec<IdxSize> = vec![];
 let mut right_row_idx: Vec<IdxSize> = vec![];

 let left_ca: &ChunkedArray<T> = left_ordered.as_ref().as_ref();
 let right_ca: &ChunkedArray<T> = right_ordered.as_ref().as_ref();

 debug_assert!(left_order.is_none_or(|order| order.len() == left_ca.len()));
 debug_assert!(right_order.is_none_or(|order| order.len() == right_ca.len()));

 let mut left_idx = 0;
 let mut right_idx = 0;
 let mut match_count = 0;

 while left_idx < left_ca.len() {
 debug_assert!(left_ca.get(left_idx).is_some());
 let left_val = unsafe { left_ca.value_unchecked(left_idx) };
 while right_idx < right_ca.len() {
 debug_assert!(right_ca.get(right_idx).is_some());
 let right_val = unsafe { right_ca.value_unchecked(right_idx) };
 if pred(&left_val, &right_val) {
 // If the predicate is true, then it will also be true for all
 // remaining rows from the right side.

The comment is the whole reason this is linear: once the predicate holds, it holds for every remaining right row, so they are all emitted at once and the right cursor does not advance. The left cursor then moves on.

The sorting for this variant reuses the column’s sorted flag when it is already there:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/mod.rs:667
 fn get_sorted(series: Series, descending: bool) -> (Series, Option<IdxCa>) {
 let expected_flag = if descending {
 IsSorted::Descending
 } else {
 IsSorted::Ascending
 };
 if (series.is_sorted_flag() == expected_flag || series.len() <= 1) && !series.has_nulls() {
 // Fast path, no need to re-sort
 (series, None)
 } else {
 let sort_options = SortOptions::default()
 .with_nulls_last(false)
 .with_order_descending(descending);

 // Get order and slice to ignore any null values, which cannot be match results
 let mut order = series.arg_sort(sort_options).slice(
 series.null_count() as i64,
 series.len() - series.null_count(),
 );

If the input is already flagged as sorted in the right direction, the sort is skipped entirely and the “order” is the identity — which is why the function returns Option<IdxCa>: None means “no permutation needed”.

Going parallel

The parallel variant partitions both sides by the first sort order and processes the partitions independently, but only after checking that the partition can contain matches at all:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/mod.rs:343
 let Some((min_l, max_l)) = get_extrema(l_l1_idx, sl) else {
 return Ok(None);
 };
 let Some((min_r, max_r)) = get_extrema(r_l1_idx, sr) else {
 return Ok(None);
 };

 let include_block = match options.operator1 {
 InequalityOperator::Lt => min_l < max_r,
 InequalityOperator::LtEq => min_l <= max_r,
 InequalityOperator::Gt => max_l > min_r,
 InequalityOperator::GtEq => max_l >= min_r,
 };

 if include_block {

min_l and max_r are the extreme values of the left and right partition. If the smallest left start is not below the largest right end, no left row in this partition can overlap any right row in the other partition, and the whole block is skipped. That is a cheap interval-overlap test, and it is what keeps the partition count from exploding.

The number of partitions is chosen with the cartesian product in mind, because each partition pair does its own join:

 rust
 
 
// crates/polars-ops/src/frame/join/iejoin/mod.rs:314
 // Because we do a cartesian product, the number of partitions is squared.
 // We take the sqrt, but we don't expect every partition to produce results and work can be
 // imbalanced, so we multiply the number of partitions by 2, which leads to 2^2= 4
 let n_partitions = (_set_partition_size() as f32).sqrt() as usize * 2;

The bottom line

The design in one sentence: sort the problem into a form where the candidate set of every row is a range, then keep the survivors in a bitmap.

The trade-offs:

Two sorts up front. The algorithm pays O(n log n) for the L1 sort and O(n log n) for the L2 sort before it emits a single pair. For small inputs a nested loop would be cheaper, which is why the single-inequality variant exists at all.
The output can still be quadratic. If every interval overlaps every other, the number of output pairs is O(n*m) and no algorithm can avoid writing them out. The IEJoin makes the search sub-quadratic; it cannot make the output sub-quadratic.
Stability is a correctness requirement. Both sorts use maintain_order(true), and the non-strict branch of the sweep depends on it. This is the kind of requirement that is easy to break in a refactor and hard to notice.
The bit array is O(n) memory — one bit per row, plus one bit per 1024 rows of summary. Cheap, but it is another allocation proportional to the input size.
Only numeric types. The implementation goes through with_match_physical_numeric_polars_type!, so interval joins over strings or datetimes are not supported here; the single-inequality variant is likewise numeric-only.

Polars Joins Two Tables with a Piecewise Merge Join

Jay Hubbert — Wed, 21 Oct 2026 06:50:00 +0000

TL;DR: An equi-join on two sorted tables does not need a hash table. It needs a merge: one cursor on the left key column, one on the right, and a comparison that decides which side advances. Polars implements exactly that in match_keys_impl, and the streaming engine goes one step further: it cuts both inputs into pieces that are mergeable independently, so the sweep can be suspended at a morsel boundary and resumed later.

The choice is made from the optimizer’s sortedness metadata, not by inspecting the data at run time.

The default equi-join in Polars is a hash join, and a hash join has a build/probe split: one side is turned into a hash table, the other side is streamed through it. That is a good algorithm for arbitrary data and a poor one for sorted data, because it throws away the ordering that the input already has and replaces it with random access into a table.

If both sides are sorted, the join is a merge, and the merge is a linear scan.

The hash join, for contrast

Here is the hash join’s structure. The build side becomes a partitioned hash table:

 rust
 
 
// crates/polars-ops/src/frame/join/hash_join/single_keys.rs:22
pub(crate) fn build_tables<T, I>(
 keys: Vec<I>,
 nulls_equal: bool,
) -> Vec<PlHashMap<<T as ToTotalOrd>::TotalOrdItem, IdxVec>>
where
 T: TotalHash + TotalEq + ToTotalOrd,
 <T as ToTotalOrd>::TotalOrdItem: Send + Sync + Copy + Hash + Eq + DirtyHash + IsNull,
 I: IntoIterator<Item = T> + Send + Sync + Clone,
{

and the probe side hashes each key, picks the partition that owns it, and looks it up:

 rust
 
 
// crates/polars-ops/src/frame/join/hash_join/single_keys_inner.rs:31
 probe.into_iter().enumerate_idx().for_each(|(idx_a, k)| {
 let k = k.to_total_ord();
 let idx_a = idx_a + local_offset;
 // probe table that contains the hashed value
 let current_probe_table =
 unsafe { hash_tbls.get_unchecked(hash_to_partition(k.dirty_hash(), n_tables)) };

 let value = current_probe_table.get(&k);

 if let Some(indexes_b) = value {
 let tuples = indexes_b.iter().map(|&idx_b| swap_fn(idx_a, idx_b));
 results.extend(tuples);
 }
 });

Two hash computations per row (one when building, one when probing), a hash table allocation, and one random memory access per probe. A merge join does none of that.

The merge sweep

The merge join kernel is short, and it is the whole algorithm:

 rust
 
 
// crates/polars-ops/src/frame/join/merge_join.rs:139
 while *build_row_offset < build_key.len() {
 if gather_build.len() >= limit_results {
 return;
 }

 let build_keyval = unsafe { build_key.get_unchecked(*build_row_offset) };
 let build_keyval = build_keyval.as_ref();
 let mut build_keyval_matched = false;

 if nulls_equal || build_keyval.is_some() {
 for probe_idx in *probe_row_offset..probe_key.len() {
 let probe_keyval = unsafe { probe_key.get_unchecked(probe_idx) };
 let probe_keyval = probe_keyval.as_ref();

 let mut ord: Ordering = match (&build_keyval, &probe_keyval) {
 (None, None) if nulls_equal => Ordering::Equal,
 (Some(l), Some(r)) => TotalOrd::tot_cmp(*l, *r),
 _ => continue,
 };
 if descending {
 ord = ord.reverse();
 }

 match ord {
 Ordering::Equal => {
 if let Some(probe_unmatched) = gather_probe_unmatched.as_mut() {
 // All probe keys up to and *excluding* this matched key are unmatched
 probe_unmatched
 .extend(*probe_first_unmatched as IdxSize..probe_idx as IdxSize);
 *probe_first_unmatched = (*probe_first_unmatched).max(probe_idx + 1);
 }
 gather_build.push(*build_row_offset as IdxSize);
 gather_probe.push(probe_idx as IdxSize);
 build_keyval_matched = true;
 },
 Ordering::Greater => {
 if let Some(probe_unmatched) = gather_probe_unmatched.as_mut() {
 // All probe keys up to and *including* this matched key are unmatched
 probe_unmatched
 .extend(*probe_first_unmatched as IdxSize..=probe_idx as IdxSize);
 *probe_first_unmatched = (*probe_first_unmatched).max(probe_idx + 1);
 }
 *probe_row_offset = probe_idx + 1;
 },
 Ordering::Less => {
 break;
 },
 }
 }
 }
 if build_emit_unmatched && !build_keyval_matched {
 gather_build.push(*build_row_offset as IdxSize);
 gather_probe.push(IdxSize::MAX);
 }
 *build_row_offset += 1;
 }

The shape of it:

text

 build (sorted) probe (sorted)
 ------------------------ ------------------------
 [ 3 3 3 5 5 9 ] [ 3 8 ]

 build cursor -> 0 (key 3) probe cursor -> 0 (key 3)
 3 == 3 -> emit (0, 0)
 3 == 3 -> emit (1, 0) # the probe cursor did NOT advance
 3 == 3 -> emit (2, 0)
 3 < 8 -> break, advance the build cursor (key 5)
 5 > 3 -> advance the probe cursor (key 8)
 5 < 8 -> break, advance the build cursor (key 5)
 5 < 8 -> break, advance the build cursor (key 9)
 9 > 8 -> advance the probe cursor, past the end
 key 9 is unmatched

Three rules, and they are the whole of a merge join:

Equal: emit the pair. Note that the probe cursor does not advance. That is what handles duplicate keys on the probe side: the next probe row with the same key is compared again, and matches too. Duplicates on the build side are handled by the outer loop: the build cursor advances one row at a time, so each duplicate build row gets its own pass over the same run of probe rows.
Greater (build key > probe key): the probe row is behind and can never match this or any later build row, so the probe cursor advances.
Less (build key < probe key): the probe side has already moved past this build key, so no later probe row can match it either; break out of the inner loop and advance the build cursor.

Both cursors only ever move forward, so the whole join is O(n + m) in the number of key comparisons, plus the number of emitted pairs.

Nulls are handled explicitly rather than by accident. A null build key with nulls_equal == false skips the inner loop entirely (the if nulls_equal || build_keyval.is_some() guard), so it matches nothing and is emitted as unmatched if the join type requires that. A null probe key falls into the _ => continue arm, which skips it without advancing the cursor.

Descending keys are handled by reversing the comparison rather than by flipping the comparison sites:

 rust
 
 
// crates/polars-ops/src/frame/join/merge_join.rs:131
 if descending {
 ord = ord.reverse();
 }

That works because both sides are sorted in the same direction, so the merge structure is symmetric.

Making it piecewise

The kernel above is written as a loop with two cursors that are passed in and returned out (build_row_offset, probe_row_offset). That signature is the interesting part: the sweep is resumable.

The streaming engine uses it that way. Because the output of a join can be much larger than either input (a many-to-many join), the engine emits results in morsels, and the sweep has to stop at a morsel boundary and pick up where it left off:

 rust
 
 
// crates/polars-stream/src/nodes/joins/merge_join.rs:498
 let mut build_row_offset = 0;
 let mut probe_row_offset = 0;
 let mut probe_last_matched = 0;
 arenas.gather_probe_unmatched.clear();
 while build_row_offset < build.height() || probe_row_offset < probe.height() {
 arenas.gather_build.clear();
 arenas.gather_probe.clear();
 let gather_probe_unmatched = params
 .probe_params()
 .emit_unmatched
 .then_some(&mut arenas.gather_probe_unmatched);
 match_keys(
 &build_key,
 &probe_key,
 &mut arenas.gather_build,
 &mut arenas.gather_probe,
 gather_probe_unmatched,
 params.build_params().emit_unmatched,
 params.key_descending,
 params.args.nulls_equal,
 morsel_size,
 &mut build_row_offset,
 &mut probe_row_offset,
 &mut probe_last_matched,
 );

The limit_results argument is the morsel size, and the kernel returns as soon as it has produced that many pairs. The cursors keep their values, so the next call resumes exactly where it stopped. That is the “piecewise” in the name: the join is computed as a sequence of independently emitted pieces, but the sweep is one continuous traversal of the two sorted inputs.

There is a second, coarser kind of piece: whole chunks of the input. The streaming node cannot assume it has both sides fully in memory, so it has to decide, given what it has buffered so far, which rows of the build side can be merged with what it has of the probe side. That decision is a binary search, because both sides are sorted:

 rust
 
 
// crates/polars-stream/src/nodes/joins/merge_join.rs:697
 let build_last_idx = usize::min(build.height(), search_limit) - 1;
 let build_last = get_key(build, build_params, params, build_last_idx).await;
 let build_first_incomplete = match build_done {
 false => {
 binary_search_lower(build, &build_last, ..=build_last_idx, params, build_params).await
 },
 true => build.height(),
 };

 let probe_last_idx = usize::min(probe.height(), search_limit) - 1;
 let probe_last = get_key(probe, probe_params, params, probe_last_idx).await;
 let probe_first_incomplete = match probe_done {
 false => {
 binary_search_lower(probe, &probe_last, ..=probe_last_idx, params, probe_params).await
 },
 true => probe.height(),
 };

build_first_incomplete is the index of the first build row whose group might continue into the next morsel — the last group in the buffered window is incomplete if the input has not ended yet, because the next morsel might contain more rows with the same key. The same is computed for the probe side, and then the two are reconciled:

 rust
 
 
// crates/polars-stream/src/nodes/joins/merge_join.rs:729
 let build_mergeable_until; // bound is *exclusive*
 let probe_mergeable_until;
 match keys_cmp(&build_last_completed_val, &probe_last_completed_val, params) {
 Ordering::Equal => {
 build_mergeable_until = build_first_incomplete;
 probe_mergeable_until = probe_first_incomplete;
 },
 Ordering::Less => {
 build_mergeable_until = build_first_incomplete;
 let build_last_mergeable =
 get_key(build, build_params, params, build_mergeable_until - 1).await;
 probe_mergeable_until = binary_search_upper(
 probe,
 &build_last_mergeable,
 ..probe_first_incomplete,
 params,
 probe_params,
 )
 .await;
 },
 Ordering::Greater => {
 probe_mergeable_until = probe_first_incomplete;
 let probe_last_mergeable =
 get_key(probe, probe_params, params, probe_mergeable_until - 1).await;
 build_mergeable_until = binary_search_upper(
 build,
 &probe_last_mergeable,
 ..build_first_incomplete,
 params,
 build_params,
 )
 .await;
 },
 }

If the last completed build key is smaller than the last completed probe key, then every probe row up to that build key has been consumed, so the probe side can be cut at the corresponding position — and that position is found with another binary search. The result is a pair of prefixes, one from each side, that can be merged without ever consulting a row outside the prefix. That is the definition of a mergeable piece.

Once the pieces are known, the build side can be split further so that several pieces are merged in parallel:

 rust
 
 
// crates/polars-stream/src/nodes/joins/merge_join.rs:625
fn find_mergeable_partition(
 build: SpillFrameSearchBuffer,
 probe: SpillFrameSearchBuffer,
 fmp: FindMergeableParams<'_>,
) -> PolarsResult<UnitVec<(SpillFrameSearchBuffer, SpillFrameSearchBuffer)>> {
 let morsel_size = get_ideal_morsel_size();

 if fmp.params.preserve_order_probe() || fmp.params.probe_params().emit_unmatched {
 return Ok(UnitVec::from([(build, probe)]));
 }

 let est_out_rows = build.height() * probe.height() + build.height() + probe.height();
 let normal_out_rows = morsel_size.pow(2);
 let partition_count = est_out_rows.div_ceil(normal_out_rows);
 if partition_count <= 1 {
 return Ok(UnitVec::from([(build, probe)]));
 }

The estimate build.height() * probe.height() is the worst case for the number of output rows — the cartesian product. Dividing that by the square of the morsel size gives a partition count that keeps each partition’s output around one morsel. The build side is then cut into chunks and each chunk is paired with the whole probe side; because the build side is sorted and the chunks are contiguous and disjoint, the chunks’ outputs can be concatenated without reordering.

Note the two early-outs: if the join must preserve the probe side’s order, or if unmatched probe rows must be reported, the partition cannot be split, because the output would no longer be in probe order and the unmatched rows would have to be merged back in order.

How the choice is made

The join type is not decided by inspecting the data. It comes from the optimizer’s sortedness analysis, which tracks which expressions are known to be sorted and in which direction:

 rust
 
 
// crates/polars-stream/src/physical_plan/lower_ir.rs:1391
 let right_on_sorted = ctx.sortedness.are_keys_sorted_any(
 input_right,
 &right_on,
 expr_arena,
 &input_right_schema,
 );
 let join_keys_sorted_together =
 Option::zip(left_on_sorted.as_ref(), right_on_sorted.as_ref())
 .is_some_and(|(ls, rs)| ls == rs);
 let mut key_descending = left_on_sorted
 .as_ref()
 .and_then(|v| v.first())
 .and_then(|s| s.descending);
 let key_nulls_last = left_on_sorted
 .as_ref()
 .and_then(|v| v.first())
 .and_then(|s| s.nulls_last);
 let use_streaming_merge_join = args.how.is_equi()
 && join_keys_sorted_together
 && key_descending.is_some()
 && key_nulls_last.is_some();

Four conditions, and each one is load-bearing:

args.how.is_equi() — a merge join implements equality only. Range joins and asof joins have their own machinery.
join_keys_sorted_together — the left and right key columns must be sorted in the same way. The comparison in the sweep assumes both sides are ordered identically; if the left is ascending and the right is descending, the merge degenerates into a quadratic scan.
key_descending.is_some() — the direction must be known, because it decides how the comparison is interpreted.
key_nulls_last.is_some() — the placement of nulls must be known, because nulls are either at the front or the back of the sorted array and the sweep has to know where to start.

The sortedness metadata is derived from the plan: a sort node produces sorted output, and that fact propagates through the plan tree. If the user writes df.sort("k").join(other, on="k") where other is also sorted on k, the optimizer knows both sides are sorted and the merge join is chosen. If either side is not known to be sorted, the hash join is used and the sortedness is irrelevant.

For multi-column keys, the streaming engine materialises a single row-encoded key column for each side, because the merge kernel compares one key column at a time:

 rust
 
 
// crates/polars-stream/src/physical_plan/lower_ir.rs:1521
 _ if use_streaming_merge_join => {
 let keys_are_row_encoded = left_on_names.len() > 1;
 if keys_are_row_encoded {
 key_descending = Some(false);
 }

Row encoding preserves the ordering of the key tuple, so the encoded column is sorted exactly when the original key columns are — but only in ascending order, which is why the direction is forced to ascending here.

The other merge: `merge_sorted`

A different operation in the same family is worth mentioning, because it uses the same sortedness assumption for a different job. merge_sorted concatenates two frames that are sorted on the same key, instead of joining them:

 rust
 
 
// crates/polars-ops/src/frame/join/merge_sorted.rs:249
fn get_merge_indicator<T>(
 mut a_iter: impl ExactSizeIterator<Item = T>,
 mut b_iter: impl ExactSizeIterator<Item = T>,
) -> Vec<bool>
where
 T: PartialOrd + Default + Copy,
{
 const A_INDICATOR: bool = true;
 const B_INDICATOR: bool = false;

 let a_len = a_iter.size_hint().0;
 let b_len = b_iter.size_hint().0;
 if a_len == 0 {
 return vec![B_INDICATOR; b_len];
 };
 if b_len == 0 {
 return vec![A_INDICATOR; a_len];
 }

 let mut current_a = T::default();
 let cap = a_len + b_len;
 let mut out = Vec::with_capacity(cap);

 let mut current_b = b_iter.next().unwrap();

 for a in &mut a_iter {
 current_a = a;
 if a <= current_b {
 out.push(A_INDICATOR);
 continue;
 }
 out.push(B_INDICATOR);

Instead of emitting pairs, it emits a boolean per output row saying which side that row came from. It is the same two-cursor sweep with a different payload: the merge of two sorted runs into one sorted run, which is the last step of a merge sort. The output frame is then built by interleaving the two sides according to that indicator.

The bottom line

The design in one sentence: a sorted equi-join is a merge, and the merge is resumable.

The trade-offs:

It requires sortedness that the optimizer can prove. Not “the data happens to be sorted” but “this plan produces sorted output here”. A user who sorts one side but not the other gets the hash join, and the sort on one side is then pure overhead.
The kernel handles duplicates, but the output can explode. A many-to-many join on duplicate keys produces the cartesian product of the runs, and the partition estimate in find_mergeable_partition assumes exactly that worst case.
Unmatched rows cost extra bookkeeping. Tracking which probe rows were not matched requires the gather_probe_unmatched vector and the probe_first_unmatched watermark, and it disables the parallel partitioning of the build side.
The state is two cursors and nothing else. Because the sweep is resumable, the join can be suspended and resumed for free, and the output can be produced in morsels of any size without buffering the whole result. That is the payoff of writing the kernel as a function of offsets rather than as a function of two frames.

Polars Encodes Group Keys as Rows to Make Hashing Cheap

Jay Hubbert — Wed, 21 Oct 2026 06:40:00 +0000

TL;DR: A composite group key is a tuple, and tuples are awkward to hash: you have to combine the hashes of the parts in a way that cannot collide across different arities or orderings. Polars sidesteps that by encoding each row of the key into a byte blob — a null sentinel, big-endian bytes for numbers, a terminator for strings — and then hashing the blob. One hash, one comparison, one allocation.

The encoder is not a black box: it is written so that the encoded bytes sort in the same order as the values, which is what lets the same representation be used for sorting, grouping and joining.

Grouping on one numeric column is easy. The physical value is the key, and the hash of that value is the hash of the key. Grouping on three columns is a different problem. Now the key is a tuple, and a tuple needs a hash function that is consistent across the whole query: two rows are in the same group if and only if their hashes agree and their keys are equal, and the “if and only if” has to survive the hashing.

The tempting shortcut — hash each column and XOR the hashes — is wrong in subtle ways. (a, b) and (b, a) can hash the same; (1, 2) and (1, 2, 0) can hash the same; the fix is to mix in the arity and the position of each column, and then you are maintaining a hand-rolled hash combiner with all the collision risk that implies.

Polars takes the other route: turn the tuple into a byte string, and let a well-tested byte hash do the work.

Encoding a row

The generic encoder lives in the polars-row crate and is used for the “row encoded” key representation. A key column’s contribution to a row is decided by its dtype, and the decision is made by a trait:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:28
/// Encodes a value of a particular fixed width type into bytes
pub trait FixedLengthEncoding: Copy + Debug {
 // 1 is validity 0 or 1
 // bit repr of encoding
 const ENCODED_LEN: usize = 1 + size_of::<Self::Encoded>();

 type Encoded: Sized + Copy + AsRef<[u8]> + AsMut<[u8]>;

 fn encode(self) -> Self::Encoded;

 fn decode(encoded: Self::Encoded) -> Self;

 /// Invert all bits of an encoded value. Used for descending order.
 fn invert(encoded: Self::Encoded) -> Self::Encoded;

 /// Keep the encoded value if `keep`, otherwise all zeros.
 fn keep_or_zero(encoded: Self::Encoded, keep: bool) -> Self::Encoded;

 fn decode_reverse(encoded: Self::Encoded) -> Self {
 Self::decode(Self::invert(encoded))
 }
}

Every fixed-width value is written as one sentinel byte followed by the value’s bytes. The sentinel is 1 for a valid value and 0xFF (or 0x00, when sorting descending) for null:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:230
pub unsafe fn encode<T: NativeType + FixedLengthEncoding>(
 buffer: &mut [MaybeUninit<u8>],
 arr: &PrimitiveArray<T>,
 opt: RowEncodingOptions,
 offsets: &mut [usize],
) {
 if arr.null_count() == 0 {
 encode_slice(buffer, arr.values().as_slice(), opt, offsets)
 } else {
 encode_slice_with_validity(
 buffer,
 arr.values().as_slice(),
 arr.validity().unwrap(),
 opt,
 offsets,
 )
 }
}

/// Write the sentinel byte and the encoded value.
#[inline(always)]
unsafe fn write_value<T: FixedLengthEncoding>(
 dst: *mut MaybeUninit<u8>,
 sentinel: u8,
 encoded: T::Encoded,
) {
 *dst = MaybeUninit::new(sentinel);
 std::ptr::write_unaligned(dst.add(1) as *mut T::Encoded, encoded);
}

/// Write the null sentinel followed by zeros.
#[inline(always)]
unsafe fn write_null<T: FixedLengthEncoding>(dst: *mut MaybeUninit<u8>, null_sentinel: u8) {
 *dst = MaybeUninit::new(null_sentinel);
 std::ptr::write_bytes(dst.add(1), 0, T::ENCODED_LEN - 1);
}

Note that a null is written as the sentinel plus zeros. That matters: it means a null key and a zero-valued key never produce the same bytes, and it means the encoding of a row is fixed width, so the row encoder can compute row offsets before it writes anything.

The value bytes themselves are chosen so that byte order equals value order. Unsigned integers are stored big-endian, which already sorts correctly. Signed integers need one extra step, because in two’s complement -1 sorts after 1 if you compare bytes:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:78
// toggle the sign bit and then encode as big indian
macro_rules! encode_signed {
 ($n:expr, $t:ty) => {
 impl FixedLengthEncoding for $t {
 type Encoded = [u8; $n];

 #[inline(always)]
 fn encode(self) -> [u8; $n] {
 #[cfg(target_endian = "big")]
 {
 todo!()
 }

 let mut b = self.to_be_bytes();
 // Toggle top "sign" bit to ensure consistent sort order
 b[0] ^= 0x80;
 b
 }

Flipping the top bit maps the two’s complement range onto the unsigned range, which is exactly what big-endian byte comparison needs. Floats get the same treatment, plus a canonicalization step so that NaN has one representation and -0.0 sorts with 0.0:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:176
impl FixedLengthEncoding for f64 {
 type Encoded = [u8; 8];

 #[inline]
 fn encode(self) -> [u8; 8] {
 // https://github.com/rust-lang/rust/blob/9c20b2a8cc7588decb6de25ac6a7912dcef24d65/library/core/src/num/f32.rs#L1176-L1260
 let s = canonical_f64(self).to_bits() as i64;
 let val = s ^ (((s >> 63) as u64) >> 1) as i64;
 val.encode()
 }

The ^ (((s >> 63) as u64) >> 1) is the standard “flip all bits if negative” transform, which maps IEEE-754 doubles onto a monotonically increasing integer ordering. Combined with the sign-bit flip in the signed-integer macro, the float encoder reuses the integer encoder.

Descending order is handled by inverting every bit, which turns “ascending” into “descending” without a second code path:

 rust
 
 
// crates/polars-row/src/fixed/numeric.rs:38
macro_rules! invert_and_keep_or_zero {
 ($t:ty) => {
 #[inline(always)]
 fn invert(encoded: Self::Encoded) -> Self::Encoded {
 (!Self::from_ne_bytes(encoded)).to_ne_bytes()
 }

Strings are the awkward case

Fixed-width values have a known size; strings do not. The string encoding solves this with a trick: since the bytes 0xFC..=0xFF never appear in valid UTF-8, every byte can be shifted by two and the result can never be confused with the terminator or the null sentinel:

 rust
 
 
// crates/polars-row/src/variable/utf8.rs:1
//! Row encoding for UTF-8 strings
//!
//! This encoding is based on the fact that in UTF-8 the bytes 0xFC - 0xFF are never valid bytes.
//! To make this work with the row encoding, we add 2 to each byte which gives us two bytes which
//! never occur in UTF-8 before and after the possible byte range. The values 0x00 and 0xFF are
//! reserved for the null sentinel. The values 0x01 and 0xFE are reserved as a sequence terminator
//! byte.
//!
//! This allows the string row encoding to have a constant 1 byte overhead.

The terminator is what lets a decoder find the end of a string without a length prefix — one byte of overhead per string instead of eight — and it is found with a vectorized byte search:

 rust
 
 
// crates/polars-row/src/variable/utf8.rs:107
/// Encode the bytes of a string followed by the terminator.
///
/// # Safety
/// `dst` must be writable for `s.len() + 1` bytes.
#[inline(always)]
unsafe fn encode_bytes(dst: *mut MaybeUninit<u8>, s: &[u8], xor_mask: u8) {
 if s.len() < BLOCK_SIZE {
 let mut block = [0u8; BLOCK_SIZE];
 std::ptr::copy_nonoverlapping(s.as_ptr(), block.as_mut_ptr(), s.len());
 encode_short(dst, block, s.len(), xor_mask);
 } else {
 for (i, &b) in s.iter().enumerate() {
 *dst.add(i) = MaybeUninit::new(xor_mask ^ (b + 2));
 }
 *dst.add(s.len()) = MaybeUninit::new(xor_mask ^ 0x01);
 }
}

Note that strings are not fixed width, so a row containing a string has a variable length. That is the main structural difference between the generic row encoding and the newer fixed-width key rows described at the end.

Hashing the bytes

Once the rows are encoded, hashing them is trivial — one hash per row:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:146
 if use_row_encoding {
 let keys = df.columns();
 let mut keys_encoded = _get_rows_encoded_unordered(keys).unwrap().into_array();

 if !null_is_valid {
 let validities = keys
 .iter()
 .map(|c| c.as_materialized_series().rechunk_validity())
 .collect_vec();
 let combined = combine_validities_and_many(&validities);
 keys_encoded.set_validity(combined);
 }

 // TODO: use vechash? Not supported yet for lists.
 // let mut hashes = Vec::with_capacity(df.height());
 // columns_to_hashes(df.columns(), Some(random_state), &mut hashes).unwrap();

 let hashes = keys_encoded
 .values_iter()
 .map(|k| random_state.hash_one(k))
 .collect();
 Self::RowEncoded(RowEncodedKeys {
 hashes: PrimitiveArray::from_vec(hashes),
 keys: keys_encoded,
 })

Two properties make this work. First, the hash is computed over the whole row, so the arity and the order of the columns are baked into the bytes — there is no way for (a, b) to collide with (b, a) unless the bytes themselves are equal. Second, the hash is computed once, up front, and stored alongside the keys:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:116
/// Represents a DataFrame plus a hash per row, intended for keys in grouping
/// or joining. The hashes may or may not actually be physically pre-computed,
/// this depends per type.
#[derive(Clone, Debug)]
pub enum HashKeys {
 RowEncoded(RowEncodedKeys),
 KeyRows(KeyRowKeys),
 Binview(BinviewKeys),
 Single(SingleKeys),
}

Pre-hashing is what makes partitioned joins possible: the hash decides which partition a key belongs to, so a probe only has to look at one partition’s table instead of all of them.

The map that stores the groups

The row-encoded grouper keeps a hash map from row bytes to group index. The map is specialised for exactly this shape — pre-hashed byte keys — and it stores the key bytes in an arena of buffers rather than allocating a Vec per key:

 rust
 
 
// crates/polars-utils/src/idx_map/bytes_idx_map.rs:25
/// An IndexMap where the keys are always [u8] slices which are pre-hashed.
pub struct BytesIndexMap<V> {
 table: HashTable<IdxSize>,
 tuples: Vec<(Key, V)>,
 key_data: Vec<Vec<u8>>,

 // Internal random seed used to keep hash iteration order decorrelated.
 // We simply store a random odd number and multiply the canonical hash by it.
 seed: u64,
}

The lookup is a hash compare followed by a byte compare:

 rust
 
 
// crates/polars-utils/src/idx_map/bytes_idx_map.rs:79
 /// Gets the index by insertion order of the given key.
 pub fn get_index_of(&self, hash: u64, key: &[u8]) -> Option<IdxSize> {
 self.table
 .find(hash.wrapping_mul(self.seed), |i| unsafe {
 let t = self.tuples.get_unchecked(*i as usize);
 hash == t.0.key_hash && key == t.0.get(&self.key_data)
 })
 .copied()
 }

That key == t.0.get(&self.key_data) is a memcmp over the row bytes. It is the whole equality check for a composite key — no per-column dispatch, no null handling at comparison time (nulls are in the bytes), no tuple comparison.

And the grouper itself is four lines:

 rust
 
 
// crates/polars-expr/src/groups/row_encoded.rs:23
 fn insert_key(&mut self, hash: u64, key: &[u8]) -> IdxSize {
 match self.idx_map.entry(hash, key) {
 Entry::Occupied(o) => o.index(),
 Entry::Vacant(v) => {
 let index = v.index();
 v.insert(());
 index
 },
 }
 }

The index handed out is the insertion order index, which is why the keys come back in group order for free.

When no encoding is needed

All of the above is machinery that exists only because there is more than one key column, or because the key is a string. For a single numeric key, Polars skips the encoding entirely:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:13
pub struct SingleKeyHashGrouper<T: PolarsDataType> {
 /// Each key with its group.
 table: HashTable<(T::Physical<'static>, IdxSize)>,
 num_groups: IdxSize,
 /// The group of the null key, IdxSize::MAX if there is none.
 null_idx: IdxSize,
 random_state: PlRandomState,
}

The hash table is keyed by the physical value itself, and the null key gets its own group index instead of being hashed at all, so it can never be confused with a real value:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:100
 for (hash, idx) in hashes.iter().zip(block) {
 let idx = *idx as usize;
 if validity.is_some_and(|v| !v.get_bit_unchecked(idx)) {
 if null_is_valid {
 if self.null_idx == IdxSize::MAX {
 self.null_idx = self.num_groups;
 self.num_groups += 1;
 }
 on_group(self.null_idx);
 }
 } else {
 let key = arr.value_unchecked(idx);
 match self.table.entry(*hash, |(k, _)| k.tot_eq(&key), hasher) {
 Entry::Occupied(o) => on_group(o.get().1),
 Entry::Vacant(v) => {
 v.insert((key, self.num_groups));
 on_group(self.num_groups);
 self.num_groups += 1;
 },
 }
 }
 }

The choice between the representations is made once, from the key schema:

 rust
 
 
// crates/polars-expr/src/groups/mod.rs:92
pub fn new_hash_grouper(key_schema: Arc<Schema>) -> Box<dyn Grouper> {
 if let Some(layout) = KeyRowLayout::new(key_schema.iter_values()) {
 Box::new(key_rows::KeyRowHashGrouper::new(Arc::new(layout)))
 } else if key_schema.len() > 1 {
 Box::new(row_encoded::RowEncodedHashGrouper::new())
 } else {
 let (_name, dt) = key_schema.get_at_index(0).unwrap();
 match dt {
 dt if dt.is_primitive_numeric() | dt.is_temporal() => {
 with_match_physical_numeric_polars_type!(dt.to_physical(), |$T| {
 Box::new(single_key::SingleKeyHashGrouper::<$T>::new())
 })
 },

Read it as a ladder: if the schema can be laid out as fixed-width key rows, use that; otherwise if there is more than one column, use generic row encoding; otherwise use the most specialized grouper the single column’s dtype allows.

Key rows: the fixed-width version

The generic row encoding has one structural weakness: string columns make rows variable length, which means row offsets are not known up front and the bytes are not word-aligned. The newer key row representation fixes both by computing a layout from the schema alone:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:72
/// The row format of keys of one schema. It depends only on the key dtypes: the null
/// bits are always present, also when nulls are not keys, so keys that disagree on
/// whether nulls match (the two sides of an anti join) share one layout.
///
/// Rows are written into zeroed words and null values are left zero, so equal keys
/// have equal words, including padding. The one exception is the second word of a
/// long view, which holds where its bytes are stored: equality skips that word and
/// compares the bytes instead.
#[derive(Debug, PartialEq)]
pub(crate) struct KeyRowLayout {
 pub(super) cols: Vec<ColLayout>,
 /// Byte offset of the null bits, one per column.
 pub(super) null_offset: usize,
 null_width: usize,
 pub(super) stride_words: usize,
 /// The first word of each view column, in column order.
 pub(super) view_words: Vec<usize>,
 /// Every word except the second half of each view.
 pub(super) plain_words: Vec<usize>,
}

Three decisions are worth pulling out.

Columns are ordered by width, descending. The layout sorts the fields so that the widest column comes first, which (a) makes the stride a power of two number of words and (b) means the comparison bails out early on the most discriminating column:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:90
 let null_width = cols.len().div_ceil(8).next_power_of_two();
 let mut fields: Vec<(usize, Option<usize>)> = widths
 .into_iter()
 .enumerate()
 .map(|(i, width)| (width, Some(i)))
 .chain([(null_width, None)])
 .collect();
 fields.sort_by_key(|(width, _)| Reverse(*width));

Nulls live inside the row. One bit per key column, always allocated even when no column has nulls. That is deliberate: it means the two sides of an anti join — where one side treats null as a key and the other does not — can share one layout and one hash table.

Equality is an XOR fold over words. Because rows are written into zeroed words, “equal” means “no differing bits”, which is one instruction per word for the fixed part:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:170
 pub(super) unsafe fn rows_eq(
 &self,
 a: &[u64],
 b: &[u64],
 mut long_eq: impl FnMut(View, View) -> bool,
 ) -> bool {
 if self.view_words.is_empty() {
 return a.iter().zip(b).fold(0, |acc, (x, y)| acc | (x ^ y)) == 0;
 }
 for w in &self.plain_words {
 if a.get_unchecked(*w) != b.get_unchecked(*w) {
 return false;
 }
 }

The strings are the awkward case again. An Arrow “view” is 16 bytes: short strings are inlined, long strings point into a buffer, and the second half of the view word holds a location, not data. So the word comparison skips that word and compares the bytes instead.

The hash is derived from the columns rather than from the row bytes, by folding each column’s contribution into a per-row hash:

 rust
 
 
// crates/polars-expr/src/key_rows/keys.rs:338
 let mut hashes = vec![random_state.hash_one(HASH_MULTIPLE); len];
 for col in &cols {
 col.hash_into(&mut hashes, random_state);
 }

The bottom line

The design in one sentence: encode the key once, then let bytes do all the work.

The trade-offs:

Encoding costs memory and a copy. Every distinct key is materialized as bytes and stored in the map’s arena. For a group-by on a wide string key that is a real cost, which is why single string keys get their own grouper that compares Arrow views directly instead of copying bytes.
The encoding is a format, so it has invariants. The sentinel byte, the terminator byte and the shifted UTF-8 bytes all have to be mutually exclusive, and the decoder has to know which convention a column was encoded with. RowEncodingOptions carries the descending and null-placement flags so that a row encoded one way is never decoded as another.
Fixed width buys alignment, at the cost of limits. The key-row layout needs a power-of-two word stride, one null-bit word, and at most two string columns; beyond that it gives up and the generic encoder takes over. The limits are explicit in the source (MAX_KEY_COLUMNS = 64, MAX_VIEW_COLUMNS = 2, crates/polars-expr/src/key_rows/layout.rs:13).
Sorting comes for free. Because the encoded bytes are order-preserving, the same representation can be sorted, compared and hashed. That is not an accident; it is why the sign bit is flipped and why floats are canonicalized before they are written.

Polars Aggregates Faster When the Keys Are Already Sorted

Jay Hubbert — Wed, 21 Oct 2026 06:30:00 +0000

TL;DR: Every ChunkedArray in Polars carries an IsSorted flag (Ascending, Descending, Not) in its metadata. The eager group-by reads that flag before it builds anything: if the key column is sorted, the groups are contiguous row ranges, so they are produced by a linear run-length scan instead of a hash table. The scan can also be split across threads, because the partitions are cut at group boundaries.

The output of that path is not just a different set of groups: it is a different type of group structure, GroupsType::Slice, which downstream aggregations consume with slice arithmetic instead of gathers.

Group-by is usually a hashing problem. You have N rows and a key column, and you want to know which rows share a key. The standard solution is a hash table: hash the key, look it up, append the row index to the bucket. That is O(N) but with a random-access pattern and a data structure that has to be allocated, grown and probed.

But there is a much cheaper structure available when the keys happen to be sorted, and sorted keys are not exotic: they are what a sort produces, what a parquet file with a sorted column contains, and what a range-partitioned stream delivers. When the keys are sorted, all the rows with the same key are adjacent, so the groups are just run boundaries. Finding them is a single pass over the array, with no allocation beyond the output vector and no hashing at all.

Polars exploits this, and the interesting part is how it decides.

text

 sort("k")
 |
 v
 [ 1 1 1 2 2 7 7 9 ] IsSorted::Ascending
 |
 v
 group_by("k")
 |
 +-- is_sorted_ascending_flag()? --- no ---> hash table
 | |
 | v
 | GroupsType::Idx
 | (row indices per group)
 |
 +-- yes ---> run-length scan ---> GroupsType::Slice
 (offset, len) per group
 [ (0,3) (3,2) (5,2) (7,1) ]

There is an obvious objection: sorting costs O(N log N), which is more than hashing costs. The flag only pays for itself when the sort was going to happen anyway, or when the data arrived sorted. In practice that is common: a sort before a group_by, a parquet file whose key column is physically ordered, a window function’s PARTITION BY output, a table that was already sorted by an upstream system. In those cases the sort is not a cost of the group-by, it is a fact about the input, and the group-by gets to use it.

The flag, not the check

Polars does not verify that the keys are sorted. It trusts a bit of metadata that is set when the column is sorted and cleared whenever a mutation could invalidate it:

 rust
 
 
// crates/polars-core/src/series/series_trait.rs:15
pub enum IsSorted {
 Ascending,
 Descending,
 Not,
}

 rust
 
 
// crates/polars-core/src/chunked_array/ops/sort/mod.rs:199
 let mut ca = ChunkedArray::from_vec(ca.name().clone(), vals);
 let s = if options.descending {
 IsSorted::Descending
 } else {
 IsSorted::Ascending
 };
 ca.set_sorted_flag(s);

The flag lives in the chunked array’s StatisticsFlags, next to things like “can fast explode list”, and it is propagated by operations that preserve order (slice, take with sorted indices, append of two compatible runs). Anything that writes a value in an arbitrary position resets it.

So the group-by’s job is not “is this sorted?” but “does the column claim to be sorted?”. That is a single bit test:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:137
 fn group_tuples(&self, multithreaded: bool, sorted: bool) -> PolarsResult<GroupsType> {
 // sorted path
 if self.is_sorted_ascending_flag() || self.is_sorted_descending_flag() {
 // don't have to pass `sorted` arg, GroupSlice is always sorted.
 let groups = self.rechunk().create_groups_from_sorted(multithreaded);
 return Ok(GroupsType::new_slice(groups, false, true));
 }

Two things to notice. The sorted: bool argument of group_tuples is not about sorted keys — it is the “maintain the order of groups by first index” flag that group_by_stable passes, and it selects how the hash path orders its output. Key sortedness is detected from the column metadata.

Second, the sorted path returns GroupsType::new_slice(groups, false, true) and returns early. Nothing downstream is told “we took the fast path”; the type of the group structure is the message.

The run-length scan

Here is the whole sorted path:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:96
 fn create_groups_from_sorted(&self, multithreaded: bool) -> GroupsSlice {
 if verbose() {
 eprintln!("group_by keys are sorted; running sorted key fast path");
 }
 let arr = self.downcast_iter().next().unwrap();
 if arr.is_empty() {
 return GroupsSlice::default();
 }
 let mut values = arr.values().as_slice();
 let null_count = arr.null_count();
 let length = values.len();

 // all nulls
 if null_count == length {
 return vec![[0, length as IdxSize]];
 }

 let mut nulls_first = false;
 if null_count > 0 {
 nulls_first = arr.get(0).is_none()
 }

 if nulls_first {
 values = &values[null_count..];
 } else {
 values = &values[..length - null_count];
 };

 let n_threads = RAYON.current_num_threads();
 if multithreaded && n_threads > 1 {
 let parts =
 create_clean_partitions(values, n_threads, self.is_sorted_descending_flag());
 let n_parts = parts.len();

 let first_ptr = &values[0] as *const T::Native as usize;
 let groups = parts.par_iter().enumerate().map(|(i, part)| {
 // we go via usize as *const is not send
 let first_ptr = first_ptr as *const T::Native;

 let part_first_ptr = &part[0] as *const T::Native;
 let mut offset = unsafe { part_first_ptr.offset_from(first_ptr) } as IdxSize;

 // nulls first: only add the nulls at the first partition
 if nulls_first && i == 0 {
 partition_to_groups(part, null_count as IdxSize, true, offset)
 }
 // nulls last: only compute at the last partition
 else if !nulls_first && i == n_parts - 1 {
 partition_to_groups(part, null_count as IdxSize, false, offset)
 }
 // other partitions
 else {
 if nulls_first {
 offset += null_count as IdxSize;
 };

 partition_to_groups(part, 0, false, offset)
 }
 });
 let groups = RAYON.install(|| groups.collect::<Vec<_>>());
 flatten_par(&groups)
 } else {
 partition_to_groups(values, null_count as IdxSize, nulls_first, 0)
 }
 }

The scan itself is a single loop over the values, comparing each element to the first element of the current run:

 rust
 
 
// crates/polars-arrow/src/legacy/kernels/sort_partition.rs:91
pub fn partition_to_groups_amortized<T>(
 values: &[T],
 first_group_offset: IdxSize,
 nulls_first: bool,
 offset: IdxSize,
 out: &mut Vec<[IdxSize; 2]>,
) where
 T: Debug + TotalEq + Sized,
{
 if let Some(mut first) = values.first() {
 out.clear();
 if nulls_first && first_group_offset > 0 {
 out.push([0, first_group_offset])
 }

 let mut first_idx = if nulls_first { first_group_offset } else { 0 } + offset;

 for val in values {
 // new group reached
 if val.tot_ne(first) {
 let val_ptr = val as *const T;
 let first_ptr = first as *const T;

 // SAFETY:
 // all pointers suffice the invariants
 let len = unsafe { val_ptr.offset_from(first_ptr) } as IdxSize;
 out.push([first_idx, len]);
 first_idx += len;
 first = val;
 }
 }

The trick in that loop is that the length of a run is computed by pointer arithmetic — val_ptr.offset_from(first_ptr) — rather than by counting. The loop never increments a counter; it just watches the pointer move. The comparison is tot_ne, the total-order equality, so floats behave (NaN is never equal to itself, but it is equal to itself under total order, and -0.0 and 0.0 compare equal).

Nulls get special treatment because Polars sorts them to one end of the array: the null run is emitted as its own group at the front or the back, and the values slice is trimmed to exclude it.

Making the scan parallel

A naive “split the array in half and scan each half” would be wrong: a group could straddle the boundary. Polars cuts the array at group boundaries instead, and the search for those boundaries is itself cheap because the array is sorted:

 rust
 
 
// crates/polars-arrow/src/legacy/kernels/sort_partition.rs:10
/// Find partition indexes such that every partition contains unique groups.
fn find_partition_points<T>(values: &[T], n: usize, descending: bool) -> Vec<usize>
where
 T: Debug + NativeType,
{
 let len = values.len();
 if n > len {
 return find_partition_points(values, len / 2, descending);
 }
 if n < 2 {
 return vec![];
 }
 let chunk_size = len / n;

 let mut partition_points = Vec::with_capacity(n + 1);

 let mut start_idx = 0;
 loop {
 let end_idx = start_idx + chunk_size;
 if end_idx >= len {
 break;
 }
 // first take that partition as a slice
 // and then find the location where the group of the latest value starts
 let part = &values[start_idx..end_idx];

 let latest_val = values[end_idx];
 let idx = if descending {
 part.partition_point(|v| v.tot_gt(&latest_val))
 } else {
 part.partition_point(|v| v.tot_lt(&latest_val))
 };

 if idx != 0 {
 partition_points.push(idx + start_idx)
 }

 start_idx += chunk_size;
 }
 partition_points
}

For each candidate boundary it looks at the value just past the boundary and binary-searches backwards within the chunk for the start of that value’s run. That gives “clean partitions”: every partition contains whole groups. create_clean_partitions then slices the array at those points, and each thread scans one slice independently. The offsets are fixed up by pointer arithmetic in create_groups_from_sorted so that the group offsets are global row indices, and the per-partition results are concatenated.

The null group is handled by only emitting it in one partition (the first if nulls come first, the last if nulls come last), with the other partitions compensating by an offset shift.

What the hash path looks like

For comparison, this is the same operation when the keys are not sorted:

 rust
 
 
// crates/polars-core/src/frame/group_by/hashing.rs:75
pub(crate) fn group_by<K>(keys: impl Iterator<Item = K>, sorted: bool) -> GroupsType
where
 K: TotalHash + TotalEq,
{
 let init_size = get_init_size();
 let (mut first, mut groups);
 if sorted {
 groups = Vec::with_capacity(get_init_size());
 first = Vec::with_capacity(get_init_size());
 let mut hash_tbl = PlHashMap::with_capacity(init_size);
 for (idx, k) in keys.enumerate_idx() {
 match hash_tbl.entry(TotalOrdWrap(k)) {
 Entry::Vacant(entry) => {
 let group_idx = groups.len() as IdxSize;
 entry.insert(group_idx);
 groups.push(unitvec![idx]);
 first.push(idx);
 },
 Entry::Occupied(entry) => unsafe {
 groups.get_unchecked_mut(*entry.get() as usize).push(idx)
 },
 }
 }
 } else {
 let mut hash_tbl = PlHashMap::with_capacity(init_size);
 for (idx, k) in keys.enumerate_idx() {
 match hash_tbl.entry(TotalOrdWrap(k)) {
 Entry::Vacant(entry) => {
 entry.insert((idx, unitvec![idx]));
 },
 Entry::Occupied(mut entry) => entry.get_mut().1.push(idx),
 }
 }
 (first, groups) = hash_tbl.into_values().unzip();
 }
 GroupsType::Idx(GroupsIdx::new(first, groups, sorted))
}

The hash path allocates a hash table, a unitvec per group, and produces GroupsType::Idx — a list of row indices per group. The sorted path produces GroupsType::Slice — a list of (offset, len) pairs, where the rows of a group are a contiguous range.

The multi-threaded variant of the hash path does not partition the rows; it partitions the hash space, so every thread has to look at every key:

 rust
 
 
// crates/polars-core/src/frame/group_by/hashing.rs:146
pub(crate) fn group_by_threaded_slice<T, IntoSlice>(
 keys: Vec<IntoSlice>,
 n_partitions: usize,
 sorted: bool,
) -> GroupsType
where
 T: ToTotalOrd,
 <T as ToTotalOrd>::TotalOrdItem: Send + Sync + Copy + DirtyHash,
 IntoSlice: AsRef<[T]> + Send + Sync,
{
 let init_size = get_init_size();

 // We will create a hashtable in every thread.
 // We use the hash to partition the keys to the matching hashtable.
 // Every thread traverses all keys/hashes and ignores the ones that doesn't fall in that partition.
 let out = RAYON.install(|| {
 (0..n_partitions)
 .into_par_iter()
 .map(|thread_no| {
 let mut hash_tbl = PlHashMap::with_capacity(init_size);

 let mut offset = 0;
 for keys in &keys {
 let keys = keys.as_ref();
 let len = keys.len() as IdxSize;

 for (key_idx, k) in keys.iter().enumerate_idx() {
 let k = k.to_total_ord();
 let idx = key_idx + offset;

 if thread_no == hash_to_partition(k.dirty_hash(), n_partitions) {
 match hash_tbl.entry(k) {
 Entry::Vacant(entry) => {
 entry.insert((idx, unitvec![idx]));
 },
 Entry::Occupied(mut entry) => {
 entry.get_mut().1.push(idx);
 },
 }
 }
 }
 offset += len;
 }
 hash_tbl
 .into_iter()
 .map(|(_k, v)| v)
 .collect_trusted::<Vec<_>>()
 })
 .collect::<Vec<_>>()
 });
 finish_group_order(out, sorted)
}

Note the shape of the work: every thread reads every key, computes its hash, and throws away the keys that are not its own. The hash table is partitioned rather than the data, so there is no work partitioning to do — but each thread touches the whole array. The sorted path does the opposite: the data is partitioned, so a thread only reads its own slice, and the only coordination is choosing where to cut.

That difference propagates all the way into the aggregations. Every aggregation has a match arm per group type:

 rust
 
 
// crates/polars-core/src/frame/group_by/aggregations/dispatch.rs:30
 pub unsafe fn agg_valid_count(&self, groups: &GroupsType) -> Series {
 // Prevent a rechunk for every individual group.
 let valid = self.rechunk_validity();

 match groups {
 GroupsType::Idx(groups) => agg_helper_idx_on_all::<IdxType, _>(groups, |idxs| {
 debug_assert!(idxs.len() <= self.len());
 if let Some(v) = &valid {
 let mut count = 0;
 for idx in idxs.iter() {
 count += unsafe { v.get_bit_unchecked(*idx as usize) as IdxSize };
 }
 Some(count)
 } else {
 Some(idxs.len() as IdxSize)
 }
 }),
 GroupsType::Slice { groups, .. } => {
 _agg_helper_slice::<IdxType, _>(groups, |[first, len]| {
 debug_assert!(len <= self.len() as IdxSize);
 if let Some(v) = &valid {
 let m = BitMask::from_bitmap(v).sliced(first as usize, len as usize);
 Some(m.set_bits() as IdxSize)
 } else {
 Some(len)
 }
 })
 },
 }
 }

The Idx arm walks a list of indices and tests each one’s validity bit. The Slice arm takes a bitmap slice — a contiguous run of bits — and counts the set bits in it, which is a popcount over a machine word rather than a per-row branch. Same answer, very different instruction mix.

The slice representation also enables a second class of optimization: an aggregation over a contiguous range can use rolling kernels, since the ranges are themselves monotone. agg_quantile_generic has a branch that checks whether the group slices are monotonic and, if so, hands the whole thing to a rolling-window kernel instead of computing a quantile per group.

The metadata that travels with the groups

The sorted fast path also leaves a trace in the output. When the group-by produces its key columns, it copies the sorted flag of the input key onto the output:

 rust
 
 
// crates/polars-core/src/frame/group_by/mod.rs:263
 GroupsType::Slice {
 groups,
 overlapping,
 monotonic: _,
 } => {
 if *overlapping && !groups.is_empty() {
 // Groups can be sliced.
 let offset = groups[0][0];
 let [upper_offset, upper_len] = groups[groups.len() - 1];
 return s.slice(
 offset as i64,
 ((upper_offset + upper_len) - offset) as usize,
 );
 }

 let indices = groups
 .iter()
 .map(|&[first, _len]| first)
 .collect_ca(PlSmallStr::EMPTY);
 // SAFETY: groups are always in bounds.
 let mut out = unsafe { s.take_unchecked(&indices) };
 // Sliced groups are always in order of discovery.
 out.set_sorted_flag(s.is_sorted_flag());
 out
 },

So df.sort("k").group_by("k").agg(...) produces a result whose k column is still flagged as sorted, and a subsequent group_by on that result takes the fast path again. The same reasoning is used for the Idx variant, but only when the groups are known to be ordered by first index.

The GroupsType::Slice variant carries two more pieces of metadata:

 rust
 
 
// crates/polars-core/src/frame/group_by/position.rs:260
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum GroupsType {
 Idx(GroupsIdx),
 Slice {
 // the groups slices
 groups: GroupsSlice,
 /// Indicates if the groups may overlap, i.e., at least one index MAY be
 /// included in more than one group slice.
 overlapping: bool,
 /// Indicates if the groups are rolling, i.e. for every consecutive group
 /// slice (offset, len), and given start = offset and end = offset + len,
 /// then both new_start >= start AND new_end >= end MUST be true.
 monotonic: bool,
 },
}

monotonic says the group ranges are themselves ordered and non-shrinking, which is exactly the precondition for a rolling kernel. overlapping says a row may appear in more than one group, which is false for a group-by but true for e.g. a window function. Both invariants are checked in debug builds:

 rust
 
 
// crates/polars-core/src/frame/group_by/position.rs:308
 pub fn new_slice(groups: GroupsSlice, overlapping: bool, monotonic: bool) -> Self {
 #[cfg(debug_assertions)]
 {
 fn groups_overlap(groups: &GroupsSlice) -> bool {
 if groups.len() < 2 {
 return false;
 }
 let mut groups = groups.clone();
 groups.sort();
 let mut prev_end = groups[0][1];

 for g in &groups[1..] {
 let start = g[0];
 let end = g[1];
 if start < prev_end {
 return true;
 }
 if end > prev_end {
 prev_end = end;
 }
 }
 false
 }

 assert!(overlapping || !groups_overlap(&groups));
 assert!(!monotonic || slice_groups_are_monotonic(&groups));
 }

The streaming version

The same idea exists in the streaming engine, where the input arrives in morsels and you cannot assume the whole column is in memory. There is a dedicated node for it, and it does the same run-length scan — but on a buffer that is flushed at group boundaries:

 rust
 
 
// crates/polars-stream/src/nodes/sorted_group_by.rs:86
 let mut offset = 0;
 let groups = idxs
 .iter()
 .map(|i| {
 let start = offset;
 offset += i;
 [start, *i]
 })
 .collect();
 let groups = GroupsType::new_slice(groups, false, true).into_sliceable();

idxs comes from rle_lengths(column, idxs), which computes the length of each run of equal values — the same run-length encoding, but as a list of lengths instead of a list of ranges. The node then buffers morsels until the buffer holds more than one distinct key, and flushes at a point where a group cannot straddle the boundary:

 rust
 
 
// crates/polars-stream/src/nodes/sorted_group_by.rs:247
 if fst == lst {
 continue;
 }

 let mut buf_key_column = buf_key_column.as_materialized_series().clone();
 buf_key_column.set_sorted_flag(IsSorted::Ascending);

 let descending = fst > lst;
 let num_flushable = search_sorted(
 &buf_key_column,
 &buf_key_column.tail(Some(1)),
 SearchSortedSide::Left,
 descending,
 )
 .unwrap();

The bottom line

The design in one sentence: sortedness is metadata, and metadata is exploited at every stage that sees it.

The trade-offs:

The fast path trusts the flag. If the flag is wrong, the group-by produces wrong groups — a silent correctness bug. That is why the flag is cleared on every mutation that could break the order, and why the sorted path rechunks first (it needs one contiguous chunk) but never re-validates.
It only pays off for a single key column. The multi-key path in group_by_with_series encodes all keys into row bytes first, and then hashes those bytes; the row bytes have no sortedness flag, so the sorted fast path is not available there.
The output type changes. GroupsType::Slice instead of GroupsType::Idx is not just a different representation: it selects slice-based aggregation code, enables rolling kernels through the monotonic flag, and lets the sorted flag survive into the output so the next group-by can take the same path.
Nulls are a special case everywhere. Nulls must be at one end of the sorted array, and the scan has to know which end, and the parallel version has to emit the null group exactly once.

Polars Groups Columns So One Pass Feeds Many Aggregations

Jay Hubbert — Wed, 21 Oct 2026 06:20:00 +0000

TL;DR: A group-by in Polars is two stages, not one. The first stage turns key columns into group ids: the keys are encoded (four different ways, depending on their dtype and count), hashed, and looked up in a hash table that hands out a group index per distinct key. The second stage is every aggregation expression, each of which consumes the same group structure. One grouping pass, many aggregations.

The streaming engine pushes this further: a fixed-size hot table keeps the frequent keys in memory, the rare ones are spilled, and the hot table is allowed to evict.

A naive group-by implementation computes the groups once per aggregation: df.group_by("k").agg(sum("a")) and df.group_by("k").agg(mean("b")) would each build their own hash table over the keys. That is a lot of duplicated work, and it scales badly with the number of aggregations.

Polars separates the two concerns. The grouping structure is computed once, and the aggregations are expressions that read it.

Two grouping engines, not one

Before going further: the type you may know as GroupsProxy is not in this tree. It has been split in two, and the split follows the execution engine:

the eager engine (polars-core) groups a materialized DataFrame into group tuples — GroupsType::Idx (a list of row indices per group) or GroupsType::Slice (a list of (offset, len) ranges, for sorted input);
the lazy/streaming engine (polars-expr) groups hash keys into group indices through the Grouper trait, because in a streaming query the keys arrive in morsels and the group table has to grow incrementally.

The eager one is the older design and is worth a look, because it is where the “sorted input” fast path lives:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:20
/// Used to create the tuples for a group_by operation.
pub trait IntoGroupsType {
 /// Create the tuples need for a group_by operation.
 /// * The first value in the tuple is the first index of the group.
 /// * The second value in the tuple is the indexes of the groups including the first value.
 fn group_tuples(&self, _multithreaded: bool, _sorted: bool) -> PolarsResult<GroupsType> {
 unimplemented!()
 }
}

Note the sorted flag: it is a promise from the caller that the key column is already sorted, and it unlocks a path that never builds a hash table at all:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:65
 fn create_groups_from_sorted(&self, multithreaded: bool) -> GroupsSlice {
 if verbose() {
 eprintln!("group_by keys are sorted; running sorted key fast path");
 }

Sorted keys mean the groups are contiguous ranges of rows, so the whole grouping reduces to a scan for run boundaries, which can be done in parallel over partitions of the sorted array.

For everything else, the eager path hashes:

 rust
 
 
// crates/polars-core/src/frame/group_by/into_groups.rs:30
fn num_groups_proxy<T>(ca: &ChunkedArray<T>, multithreaded: bool, sorted: bool) -> GroupsType
where
 T: PolarsNumericType,
 T::Native: TotalHash + TotalEq + DirtyHash + ToTotalOrd,

TotalHash/TotalEq are the important detail: they give floats a total order, so NaN == NaN and -0.0 == 0.0 behave sanely inside a hash table.

The `Grouper` trait

The lazy/streaming side has one trait, and it is small:

 rust
 
 
// crates/polars-expr/src/groups/mod.rs:45
/// A Grouper maps keys to groups, such that duplicate keys map to the same group.
pub trait Grouper: Any + Send + Sync {
 /// Creates a new empty Grouper similar to this one.
 fn new_empty(&self) -> Box<dyn Grouper>;

 /// Reserves space for the given number additional groups.
 fn reserve(&mut self, additional: usize);

 /// Returns the number of groups in this Grouper.
 fn num_groups(&self) -> IdxSize;

 /// Inserts the given subset of keys into this Grouper. If groups_idxs is
 /// passed it is extended such with the group index of keys[subset[i]].
 /// New groups get consecutive indices in the order they first occur.
 ///
 /// # Safety
 /// The subset indexes must be in-bounds.
 unsafe fn insert_keys_subset(
 &mut self,
 keys: &HashKeys,
 subset: &[IdxSize],
 group_idxs: Option<&mut Vec<IdxSize>>,
 );

 /// Returns the keys in this Grouper in group order, that is the key for
 /// group i is returned in row i.
 fn get_keys_in_group_order(&self, schema: &Schema) -> DataFrame;

That is the whole contract: insert keys, get group indices out, hand back the distinct keys in group order. The trait does not mention aggregations at all, which is exactly the point — the grouping structure is reusable by any number of consumers.

The other three methods on the trait (probe_partitioned_groupers, contains_key_partitioned_groupers, mark_groups_partitioned_groupers) exist for partitioned hash joins: they probe other groupers by hash partition, so a join can look up a key in the partition that owns it without searching all of them.

Four encodings, one interface

The interesting decision is how to represent a key so that it can be hashed and compared. Polars picks between four encodings based on the key schema:

 rust
 
 
// crates/polars-expr/src/groups/mod.rs:113
pub fn new_hash_grouper(key_schema: Arc<Schema>) -> Box<dyn Grouper> {
 if let Some(layout) = KeyRowLayout::new(key_schema.iter_values()) {
 Box::new(key_rows::KeyRowHashGrouper::new(Arc::new(layout)))
 } else if key_schema.len() > 1 {
 Box::new(row_encoded::RowEncodedHashGrouper::new())
 } else {
 let (_name, dt) = key_schema.get_at_index(0).unwrap();
 match dt {
 dt if dt.is_primitive_numeric() | dt.is_temporal() => {
 with_match_physical_numeric_polars_type!(dt.to_physical(), |$T| {
 Box::new(single_key::SingleKeyHashGrouper::<$T>::new())
 })
 },

 #[cfg(feature = "dtype-decimal")]
 DataType::Decimal(_, _) => {
 Box::new(single_key::SingleKeyHashGrouper::<Int128Type>::new())
 },
 #[cfg(feature = "dtype-categorical")]
 dt @ (DataType::Enum(_, _) | DataType::Categorical(_, _)) => {
 with_match_categorical_physical_type!(dt.cat_physical().unwrap(), |$C| {
 Box::new(single_key::SingleKeyHashGrouper::<<$C as PolarsCategoricalType>::PolarsPhysical>::new())
 })
 },

 DataType::String | DataType::Binary => Box::new(binview::BinviewHashGrouper::new()),

 _ => Box::new(row_encoded::RowEncodedHashGrouper::new()),
 }
 }
}

Reading that top to bottom: if the key columns can be laid out as fixed-width key rows, use that; otherwise if there is more than one key column, use generic row encoding; otherwise, for a single key, use the most specialized grouper the dtype allows — a direct hash table over the physical values for numbers and dates, a view-aware one for strings, and generic row encoding for anything exotic.

Single key: the values themselves

For one numeric or temporal key, no encoding is needed at all — the physical value is the key:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:13
pub struct SingleKeyHashGrouper<T: PolarsDataType> {
 /// Each key with its group.
 table: HashTable<(T::Physical<'static>, IdxSize)>,
 num_groups: IdxSize,
 /// The group of the null key, IdxSize::MAX if there is none.
 null_idx: IdxSize,
 random_state: PlRandomState,
}

and the insert path is a textbook open-addressing hash table, with two hardware-friendly touches. First, the null key gets its own group index rather than being hashed, so nulls are never confused with a real value. Second, the work is done in blocks: the hashes of a whole block are computed (and prefetched) before the lookups happen, so the CPU has several independent memory accesses in flight:

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:64
 unsafe fn insert_impl(
 &mut self,
 arr: &T::Array,
 null_is_valid: bool,
 subset: &[IdxSize],
 mut on_group: impl FnMut(IdxSize),
 ) {

 rust
 
 
// crates/polars-expr/src/groups/single_key.rs:82
 let mut hashes = [0; BLOCK_SIZE];
 for block in subset.chunks(BLOCK_SIZE) {
 // Reserve up front so the table doesn't move after prefetching.
 self.table.reserve(block.len(), hasher);
 let prefetch = self.should_prefetch();
 for (hash, idx) in hashes.iter_mut().zip(block) {
 *hash = random_state.tot_hash_one(arr.value_unchecked(*idx as usize));
 if prefetch {
 self.table.prefetch_insert(*hash);
 }
 }

That “reserve up front so the table doesn’t move after prefetching” comment is the kind of detail that only shows up after a profiler: reallocating the table after issuing prefetches would invalidate the addresses you just prefetched.

Key rows: a hand-rolled row format

The newest and most interesting encoding is key_rows. Instead of using the generic polars-row encoder, Polars lays out the key columns itself into a row format with a fixed stride, and the layout is computed once per schema:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:72
/// The row format of keys of one schema. It depends only on the key dtypes: the null
/// bits are always present, also when nulls are not keys, so keys that disagree on
/// whether nulls match (the two sides of an anti join) share one layout.
///
/// Rows are written into zeroed words and null values are left zero, so equal keys
/// have equal words, including padding. The one exception is the second word of a
/// long view, which holds where its bytes are stored: equality skips that word and
/// compares the bytes instead.
#[derive(Debug, PartialEq)]
pub(crate) struct KeyRowLayout {
 pub(super) cols: Vec<ColLayout>,
 /// Byte offset of the null bits, one per column.
 pub(super) null_offset: usize,
 null_width: usize,
 pub(super) stride_words: usize,
 /// The first word of each view column, in column order.
 pub(super) view_words: Vec<usize>,
 /// Every word except the second half of each view.
 pub(super) plain_words: Vec<usize>,
}

Three design decisions are worth spelling out:

Nulls live in the row. One null bit per key column, always present even when nulls are not part of the key. That means the two sides of an anti join (where one side treats null as a key and the other does not) can share one layout and one hash table.
Columns are sorted by width, descending. The layout puts the widest columns first so that the whole row is a power-of-two number of 64-bit words, and so that comparisons can bail out early on the most discriminating word:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:90
 let null_width = cols.len().div_ceil(8).next_power_of_two();
 let mut fields: Vec<(usize, Option<usize>)> = widths
 .into_iter()
 .enumerate()
 .map(|(i, width)| (width, Some(i)))
 .chain([(null_width, None)])
 .collect();
 fields.sort_by_key(|(width, _)| Reverse(*width));

Equality is a word-wise XOR fold. Because rows are written into zeroed words, “equal” means “no differing bits”, and that is one instruction per word for the fixed-width part:

 rust
 
 
// crates/polars-expr/src/key_rows/layout.rs:170
 pub(super) unsafe fn rows_eq(
 &self,
 a: &[u64],
 b: &[u64],
 mut long_eq: impl FnMut(View, View) -> bool,
 ) -> bool {
 if self.view_words.is_empty() {
 return a.iter().zip(b).fold(0, |acc, (x, y)| acc | (x ^ y)) == 0;
 }
 for w in &self.plain_words {
 if a.get_unchecked(*w) != b.get_unchecked(*w) {
 return false;
 }
 }

Strings (Arrow “views”) are the awkward case: a view is a 16-byte struct where short strings are inlined and long strings point into a buffer. The second half of a view word holds a location, not data, so it is skipped by the word comparison and the bytes are compared instead.

Row encoding and binview: the fallbacks

Generic row encoding uses the polars-row crate and stores the encoded bytes in a byte-indexed map:

 rust
 
 
// crates/polars-expr/src/groups/row_encoded.rs:12
#[derive(Default)]
pub struct RowEncodedHashGrouper {
 idx_map: BytesIndexMap<()>,
}

 rust
 
 
// crates/polars-expr/src/groups/row_encoded.rs:23
 fn insert_key(&mut self, hash: u64, key: &[u8]) -> IdxSize {
 match self.idx_map.entry(hash, key) {
 Entry::Occupied(o) => o.index(),
 Entry::Vacant(v) => {
 let index = v.index();
 v.insert(());
 index
 },
 }
 }

For a single string key, there is a dedicated grouper that compares the Arrow views directly, so a 100 MB string column is never copied into a row buffer:

 rust
 
 
// crates/polars-expr/src/groups/binview.rs:33
 /// # Safety
 /// The view must be valid for the given buffer set.
 #[inline(always)]
 unsafe fn insert_key(
 &mut self,
 hash: u64,
 view: View,
 buffers: &Buffer<Buffer<u8>>,
 ) -> IdxSize {
 unsafe {
 match self.idx_map.entry_view(hash, view, buffers) {

The keys themselves

The Grouper does not take a DataFrame of keys; it takes a HashKeys, which is the encoded key material plus a precomputed hash per row:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:116
/// Represents a DataFrame plus a hash per row, intended for keys in grouping
/// or joining. The hashes may or may not actually be physically pre-computed,
/// this depends per type.
#[derive(Clone, Debug)]
pub enum HashKeys {
 RowEncoded(RowEncodedKeys),
 KeyRows(KeyRowKeys),
 Binview(BinviewKeys),
 Single(SingleKeys),
}

The from_df constructor is where the encoding decision is actually made, and it prefers key rows over everything else:

 rust
 
 
// crates/polars-expr/src/hash_keys.rs:130
 if !force_row_encoding
 && let Some(layout) = KeyRowLayout::new(df.columns().iter().map(|c| c.dtype()))
 {
 return Self::KeyRows(KeyRowKeys::from_columns(
 df.columns(),
 Arc::new(layout),
 &random_state,
 null_is_valid,
 ));
 }

 let first_col_variant = hash_keys_variant_for_dtype(df[0].dtype());
 let use_row_encoding = force_row_encoding
 || df.width() > 1
 || first_col_variant == HashKeysVariant::RowEncoded;

Precomputing the hash per row is what makes partitioned joins and the partitioned groupers possible: a key’s hash determines its partition, so a probe only has to look in one partition’s table.

One pass, many aggregations

Now the payoff. The eager engine groups once and then runs every aggregation against the same group structure:

 rust
 
 
// crates/polars-mem-engine/src/executors/group_by.rs:68
pub(super) fn group_by_helper(
 mut df: DataFrame,
 keys: Vec<Column>,
 aggs: &[Arc<dyn PhysicalExpr>],
 apply: Option<PlanCallback<DataFrame, DataFrame>>,
 state: &ExecutionState,
 maintain_order: bool,
 output_schema: &SchemaRef,
 slice: Option<(i64, usize)>,
) -> PolarsResult<DataFrame> {
 df.rechunk_mut_par();
 let gb = df.group_by_with_series(keys, true, maintain_order)?;

 rust
 
 
// crates/polars-mem-engine/src/executors/group_by.rs:15
pub(super) fn evaluate_aggs(
 df: &DataFrame,
 aggs: &[Arc<dyn PhysicalExpr>],
 groups: &GroupPositions,
 state: &ExecutionState,
) -> PolarsResult<Vec<Column>> {
 RAYON.install(|| {
 aggs.par_iter()
 .map(|expr| {
 let agg = expr.evaluate_on_groups(df, groups, state)?.finalize();
 polars_ensure!(agg.len() == groups.len(), agg_len = agg.len(), groups.len());
 Ok(agg)
 })
 .collect::<PolarsResult<Vec<_>>>()
 })
}

groups is computed once and then shared, read-only, by every aggregation expression, which are themselves free to run in parallel. The lazy engine does the same thing, one level down: an aggregation expression receives the shared GroupPositions and reduces its own input column against it:

 rust
 
 
// crates/polars-expr/src/expressions/aggregation.rs:232
 fn evaluate_on_groups_impl<'a>(
 &self,
 df: &DataFrame,
 groups: &'a GroupPositions,
 state: &ExecutionState,
 ) -> PolarsResult<AggregationContext<'a>> {
 let mut ac = self.input.evaluate_on_groups(df, groups, state)?;

The GroupedReduction trait is the streaming-engine equivalent, and it is designed so that many reductions can share one grouping structure:

 rust
 
 
// crates/polars-expr/src/reduce/mod.rs:163
 /// Updates the specified group with the given values.
 ///
 /// For order-sensitive grouped reductions, seq_id can be used to resolve
 /// order between calls/multiple reductions.
 fn update_group(
 &mut self,
 values: &[&Column],
 group_idx: IdxSize,
 seq_id: u64,
 ) -> PolarsResult<()>;

and the partition table of the streaming group-by shows the pattern in full: one grouper, N reductions, one pass over the rows:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/table.rs:143
 pub fn add_rows(
 &mut self,
 input: usize,
 frame: &FrameWithKeys,
 rows: Option<&[IdxSize]>,
 ) -> PolarsResult<()> {
 let rows = rows_or_all(rows, &mut self.all_rows, frame.keys.len());
 self.group_idxs.clear();
 unsafe {
 self.grouper
 .insert_keys_subset(&frame.keys, rows, Some(&mut self.group_idxs));
 }
 let num_groups = self.grouper.num_groups();
 let params = &*self.params;
 let mut in_cols = Vec::new();
 for r_idx in &params.reductions_per_input[input] {
 let r = &mut self.reductions[*r_idx];
 r.resize(num_groups);
 in_cols.clear();
 in_cols.extend(
 params.grouped_reduction_cols[*r_idx]
 .iter()
 .map(|col| frame.frame.column(col).unwrap()),
 );
 // Cold rows only exist without order-sensitive reductions.
 unsafe { r.update_groups_subset(&in_cols, rows, &self.group_idxs, 0)? };
 }
 Ok(())
 }

The group indices computed by the grouper are handed to every reduction. r.resize(num_groups) grows the reduction’s state to the number of groups found so far; update_groups_subset then adds the values of the given rows to the right groups. Adding a tenth aggregation to a query costs a tenth reduction state and a tenth pass over the group’s rows — but not a tenth grouping pass.

The hot table

There is a second, more aggressive version of this idea in the streaming engine. Because a group-by over a large table has to hold the group table in memory, Polars splits the keys into hot (frequent) and cold (rare) ones:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/mod.rs:1
//! The streaming group-by.
//!
//! While receiving input, each pipeline aggregates rows with frequent keys in small hot tables
//! and stores the other (cold) rows with their hashed keys in spillable frames. Groups evicted
//! from a hot table, and the hot tables at the end of the input, become pre-aggregates. Once all
//! input is received, the cold rows and pre-aggregates are split into hash partitions, which are
//! aggregated independently. With at most one partition per pipeline, all partitions are built
//! before sending. Otherwise the cold rows are first stored per partition, and the partitions are
//! built a few at a time while the results are sent.

The hot table is a HotGrouper, a sibling of Grouper with one crucial difference — it has a fixed size and evicts rather than growing:

 rust
 
 
// crates/polars-expr/src/hot_groups/mod.rs:21
/// A HotGrouper maps keys to groups, such that duplicate keys map to the same
/// group. Unlike a Grouper it has a fixed size and will cause evictions rather
/// than growing.
pub trait HotGrouper: Any + Send + Sync {
 /// Creates a new empty HotGrouper similar to this one, with the given size.
 fn new_empty(&self, groups: usize) -> Box<dyn HotGrouper>;

 /// Returns the number of groups in this HotGrouper.
 fn num_groups(&self) -> IdxSize;

 /// Returns the number of slots in the hot table.
 fn num_slots(&self) -> usize;

 /// Doubles the number of slots in the hot table. Group indices are unchanged.
 fn double(&mut self);

 /// Inserts the given keys into this Grouper, extending groups_idxs with
 /// the group index of keys[i].
 ///
 /// A missed key that is followed by a key with the same hash is inserted
 /// even if that evicts another key, as is every missed key with
 /// `force_hot`. This keeps sorted keys from first going cold.
 fn insert_keys(
 &mut self,
 keys: &HashKeys,
 hot_idxs: &mut Vec<IdxSize>,
 hot_group_idxs: &mut Vec<EvictIdx>,
 cold_idxs: &mut Vec<IdxSize>,
 force_hot: bool,
 );

The EvictIdx type is the trick that makes this work: a group index whose top bit is set means “this group was evicted, reset it before updating”. The reduction state carries the same flag, so an evicted group’s partial sum is discarded and rebuilt from the cold rows later. The note about “a missed key that is followed by a key with the same hash” is about sorted input: if the keys are sorted, the same key appears in a run, and evicting the first occurrence and re-inserting the second would defeat the purpose, so the grouper is allowed to evict on a hash collision instead.

The sink keeps both halves:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/sink.rs:60
/// The state of one pipeline of the sink.
struct LocalSink {
 // Created when the input first sends this pipeline a non-empty morsel.
 hot_grouper_per_input: Vec<Option<Box<dyn HotGrouper>>>,
 hot_grouped_reductions: Vec<Box<dyn GroupedReduction>>,

 // The number of slots of each hot grouper, and the rows missed by them.
 hot_table_size: usize,
 miss_f2: F2Sketch,

 // The keys of the cold rows and the pre-aggregates.
 key_sketch: CardinalitySketch,

 // Per input, the cold rows not yet in a frame.
 cold_row_builders: Vec<Option<FrameWithKeysBuilder>>,
 // Each frame with the input it came from.
 cold_frames: Vec<(usize, SpillFrameWithKeys)>,
 // The estimated bytes of `cold_frames`.
 cold_bytes: u64,
 // In creation order.
 pre_aggs: Vec<PreAgg>,
}

The hot table grows only when the evidence says it should — the F2 sketch counts how often keys are missed, and the table is enlarged only if the missed keys repeat enough and dominate the misses:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/sink.rs:47
/// The hot tables only grow if the frequently missed keys were missed at least this
/// many times each.
const HOT_TABLE_GROW_MIN_REPEAT: f64 = 4.0;

/// The hot tables only grow if the frequently missed keys make up at least this
/// fraction of the missed rows.
const HOT_TABLE_GROW_MIN_HEAVY_SHARE: f64 = 0.5;

The bottom line

The design in one sentence: grouping is a data structure, not a step. The group structure is built once, by a Grouper that picks an encoding suited to the key schema, and then it is a read-only input to every aggregation.

The trade-offs that come with it:

Encoding choice is per-query, not per-user. A single numeric key gets a direct hash table over the values; a multi-column key gets fixed-width key rows; strings get view comparisons; anything else falls back to generic row encoding. The cost is four implementations of one trait, and the benefit is that the common cases avoid copying key data at all.
The group structure is shared, so it is immutable during aggregation. Every aggregation reads it; none of them may reorder it. That is why evaluate_aggs can run the aggregations in parallel over one groups.
The hot table trades exactness of memory prediction for speed. It is a fixed-size cache of frequent keys, and it evicts. The evicted groups are not lost, they are re-derived from the spilled cold rows, which means the grouping work is done twice for evicted keys — a deliberate exchange of CPU for a bounded memory footprint.

Polars Reorders Work to Avoid Running Out of Memory

Jay Hubbert — Wed, 21 Oct 2026 06:10:00 +0000

TL;DR: Polars’ streaming engine does not execute a plan in plan order. It executes it in phases, and inside a phase it decides which node to run by looking at how much memory that node is holding onto. At most one memory-hungry node runs per phase, and among the candidates the scheduler prefers the one whose output can be consumed immediately, then the one that frees the most held data, then the one with the most unbounded work downstream.

If reordering is not enough, the out-of-core layer spills to disk once the process-wide allocation counter crosses a budget, and a global allocator aborts the process before the kernel’s OOM killer does. Where the engine can trade CPU for memory, it does: a select is split with a minimum cut so that wide intermediates are recomputed rather than stored.

I went looking for a “reorder the plan so we do not run out of memory” pass in the optimizer. There is not one, at least not under that name. What exists is much more interesting: the streaming engine treats execution order as a scheduling problem with memory as the objective function.

The problem with a fixed order

A streaming plan is a graph, not a list. Some nodes are cheap filters that pass data through; others are sort, group_by, join — nodes that must see all of their input before they can emit anything, and that therefore hold their input in memory while they work.

If you run such a graph in plan order, the classic failure is:

text

scan ──► filter ──► sort ──────────► join ──► sink
 └─► group_by ─────────────┘

Run sort first and it buffers the entire dataset. Run group_by first and it builds a hash table over the entire dataset. Meanwhile the other branch could have been streaming through, keeping only a fraction of the data alive at any time. The peak memory of the query is not a property of any single node; it is a property of the order in which nodes are run.

So the engine needs to know, for each node, how much memory it is currently holding, and it needs to be able to change its mind about the order.

The graph

The streaming engine’s graph is two slot maps — nodes and pipes:

 rust
 
 
// crates/polars-stream/src/graph.rs:23
/// Represents the compute graph.
///
/// The `nodes` perform computation and the `pipes` form the connections between nodes
/// that data is sent through.
#[derive(Default)]
pub struct Graph {
 pub nodes: SlotMap<GraphNodeKey, GraphNode>,
 pub pipes: SlotMap<LogicalPipeKey, LogicalPipe>,
}

 rust
 
 
// crates/polars-stream/src/graph.rs:187
pub struct GraphNode {
 pub compute: Box<dyn ComputeNode>,
 pub inputs: Vec<LogicalPipeKey>,
 pub outputs: Vec<LogicalPipeKey>,
}

Every pipe has a state on each side, and there are exactly three states:

 rust
 
 
// crates/polars-stream/src/graph.rs:212
#[derive(Copy, Clone, PartialEq, Eq, Debug, PartialOrd, Ord)]
pub enum PortState {
 Blocked,
 Ready,
 Done,
}

A node can only make progress when its input pipes are Ready (data available) and its output pipes are not Blocked (the downstream is not ready to take more). This is a classic dataflow readiness rule, and it is what makes re-ordering possible in the first place: a node’s eligibility depends on the state of the graph, not on its position in the plan.

Classifying how much memory a node holds

Each node reports its memory behaviour through one small enum:

 rust
 
 
// crates/polars-stream/src/nodes/mod.rs:83
/// How a node's memory use relates to its input size, given its current state.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum NodeMemoryUsage {
 /// Memory does not meaningfully grow with the input size, now or in a later
 /// state.
 Bounded,
 /// Memory may grow with the input size while data streams through the node.
 Unbounded,

 /// Stores nothing meaningful yet, but will be `Accumulating` in a later state.
 WillAccumulate,
 /// Stores (substantial parts of) its input while running. Each phase runs
 /// at most one such node, unless all such nodes remaining are sinks.
 Accumulating,
 /// Holds significant stored data that may be freed once this node runs to completion.
 HoldingUntilDone,
 /// Holds significant stored data that may be freed immediately while this node runs.
 Draining,
}

This is a per-node classification, not a per-node budget. There is no “this node may use 200 MB”; the classification is qualitative, and the scheduler uses it to decide ordering. The distinction between HoldingUntilDone and Draining is the interesting one: both hold a lot, but the first can only release when it finishes, while the second releases as it runs. A Draining node is therefore a priority — run it and you get your memory back sooner.

Nodes implement it themselves. A sort, for instance:

 rust
 
 
// crates/polars-stream/src/nodes/sort/mod.rs:370
 fn memory_usage(&self) -> NodeMemoryUsage {
 match &self.state {
 SortState::BufferAndSample(_) => NodeMemoryUsage::Accumulating,
 SortState::Flush(_) => NodeMemoryUsage::Draining,
 SortState::Source(src) => src.memory_usage(),
 SortState::Done => NodeMemoryUsage::Bounded,

The phase loop

Execution is a loop over phases. Each phase picks a subgraph to run, runs it to completion, and then re-evaluates the graph state. The subgraph selection is where the reordering happens:

 rust
 
 
// crates/polars-stream/src/execute.rs:116
/// Finds a part of the graph which we can run.
fn find_runnable_subgraph(graph: &mut Graph) -> (PlHashSet<GraphNodeKey>, Vec<LogicalPipeKey>) {
 // Find phase sinks, choose a subset with at most one accumulating node, and
 // return the subgraph needed to feed them.
 let phase_sinks = find_runnable_phase_sinks(graph);
 let (accumulating, mut to_run): (Vec<_>, Vec<_>) = phase_sinks
 .into_iter()
 .partition(|n| graph.nodes[*n].compute.memory_usage() == NodeMemoryUsage::Accumulating);

 // If all accumulating nodes left are sinks (InMemorySink), we're not gaining
 // anything by only running a subset.
 let only_accumulating_sinks_left = accumulating
 .iter()
 .all(|node_key| graph.nodes[*node_key].outputs.is_empty());

 if only_accumulating_sinks_left {
 to_run.extend(accumulating);
 } else {
 to_run.push(choose_accumulating_node(graph, &accumulating));
 }

 expand_ready_subgraph(graph, to_run)
}

Read the comment carefully: choose a subset with at most one accumulating node. An Accumulating node is one that stores its input while it runs — a sort, a group-by, a join build side. Running two of them at once means two half-full hash tables in memory at the same time. The engine refuses: at most one per phase, so the peak is one accumulator plus whatever the cheap nodes need.

The exception is worth noting: if every remaining accumulator is a sink (the final in-memory result), there is nothing to be gained by serializing them, so they all run together.

Choosing which accumulator to run

When there is more than one candidate accumulator, the tie is broken by a scoring function that is, in effect, a memory-aware topological sort:

 rust
 
 
// crates/polars-stream/src/execution_order.rs:13
/// Chooses which of the given accumulating nodes to run, in order of preference:
/// 1. nodes whose outputs are ready to be consumed, so the result is not held;
/// 2. nodes that read from the most nodes holding data (see [`num_held_upstream`]), so that data
/// can be released;
/// 3. nodes with the most non-bounded nodes downstream (see [`num_non_bounded_downstream`]), so
/// a short branch does not hold its result while a long branch is processed;
/// 4. nodes with more outputs, so in-memory sinks (which have none) run last.
pub fn choose_accumulating_node(graph: &Graph, candidates: &[GraphNodeKey]) -> GraphNodeKey {
 let num_held = num_held_upstream(graph, candidates);
 let num_non_bounded = num_non_bounded_downstream(graph, candidates);
 *candidates
 .iter()
 .max_by_key(|node_key| {
 let node = &graph.nodes[**node_key];
 let num_outputs_ready_to_recv = node
 .outputs
 .iter()
 .filter(|o| graph.pipes[**o].recv_state == PortState::Ready)
 .count();
 (
 num_outputs_ready_to_recv,
 num_held[**node_key],
 num_non_bounded[**node_key],
 node.outputs.len(),
 )
 })
 .unwrap()
}

Four rules, in priority order. Rule 1 is the anti-footgun rule: if the downstream is ready to consume, run this node now and hand the result over immediately, rather than holding the finished result in memory. Rule 2 is the memory rule proper — count how many holding nodes feed into this one, and prefer the node that, when it finishes, lets the most data go. Rule 3 keeps long branches moving: if a short branch would finish and sit on its result while a long branch is still working, prefer to start the long branch first. Rule 4 is a tiebreak that pushes final sinks last.

The two counting helpers are small recursive graph walks:

 rust
 
 
// crates/polars-stream/src/execution_order.rs:39
/// For each of the given nodes and the nodes feeding them through pipes ready on both sides, the
/// number of [`NodeMemoryUsage::HoldingUntilDone`] or [`NodeMemoryUsage::Draining`] nodes among
/// itself and its inputs through such pipes, recursively. A node reachable through several such
/// paths is counted once per path.
fn num_held_upstream(graph: &Graph, nodes: &[GraphNodeKey]) -> SecondaryMap<GraphNodeKey, usize> {

 rust
 
 
// crates/polars-stream/src/execution_order.rs:45
/// For each of the given nodes and the nodes downstream of them, the largest number of nodes
/// that are not [`NodeMemoryUsage::Bounded`] on a path from that node (inclusive) to a node
/// without outputs.
fn num_non_bounded_downstream(
 graph: &Graph,
 nodes: &[GraphNodeKey],
) -> SecondaryMap<GraphNodeKey, usize> {

Note that both walk only ready pipes. A node that is holding data but is currently blocked cannot contribute to freeing memory right now, so it does not count.

Inside a phase: topological spawn

Once the phase’s subgraph is chosen, the nodes are spawned in topological order — but backwards, from the sinks:

 rust
 
 
// crates/polars-stream/src/execute.rs:163
 // We do a topological sort of the graph: we want to spawn each node,
 // starting with the sinks and moving backwards. This order is important
 // for the initialization of physical pipes - the receive port must be
 // initialized first.

The reason is mechanical: a physical pipe’s receive port must exist before the sender can use it, and the pipe is created between two nodes. Spawning sinks-first means every pipe is half-initialized by the time its other end is spawned. The sort is a simple count of not-yet-ready outputs:

 rust
 
 
// crates/polars-stream/src/execute.rs:176
 let mut ready = Vec::new();
 let mut num_send_ports_not_yet_ready = SecondaryMap::new();
 for node_key in nodes {
 let node = &graph.nodes[*node_key];
 let num_outputs_in_subgraph = node
 .outputs
 .iter()
 .filter(|o| physical_pipes.contains_key(**o))
 .count();
 num_send_ports_not_yet_ready.insert(*node_key, num_outputs_in_subgraph);
 if num_outputs_in_subgraph == 0 {
 ready.push(*node_key);
 }
 }

So there are two levels of ordering: a global one that decides which memory-hungry node goes next, and a local one that makes the spawn valid.

The last resort: spill to disk

Reordering reduces peak memory; it cannot eliminate it. A single sort of a dataset that does not fit in RAM still does not fit in RAM. For that, Polars has an out-of-core layer, polars-ooc, with a global memory budget.

The budget is a fraction of total memory, capped by an absolute value:

 rust
 
 
// crates/polars-config/src/lib.rs:384
 let budget_frac = f64::from_bits(self.ooc_memory_budget_fraction.load(Ordering::Relaxed));
 let bytes = u64::min(
 self.ooc_memory_budget_bytes.load(Ordering::Relaxed),
 (total_memory() as f64 * budget_frac) as u64,
 );
 self.ooc_effective_memory_budget_bytes
 .store(bytes, Ordering::Relaxed);

The decision to spill is made against a process-wide allocation counter, not against any per-node accounting:

 rust
 
 
// crates/polars-ooc/src/memory_manager.rs:64
 fn should_spill(&self) -> bool {
 let usage = crate::estimate_memory_usage();
 let likely_dealt_with = self.est_spill_in_progress.load(Ordering::Relaxed);
 let likely_incoming = self.est_prefetch_in_progress.load(Ordering::Relaxed);
 (usage + likely_incoming).saturating_sub(likely_dealt_with)
 > config().ooc_memory_budget_bytes()
 }

and it is checked at explicit points in the hot loops of the nodes that accumulate — the group-by, the window nodes, the hash-shuffle batches:

 rust
 
 
// crates/polars-stream/src/nodes/group_by/mod.rs:305
 for_each_parallel(
 partitions.iter_mut(),
 state.task_metrics(),
 &|(p, table, _)| async move {
 for (pre_agg, groups) in pre_aggs.in_partition(*p) {
 table.add_pre_agg(pre_agg, groups)?;
 }
 memory_manager().spill().await;
 Ok(())
 },
 )?;

spill() is cheap when there is no pressure — it is a single comparison against the budget — and expensive only when it actually has to move data to disk.

The allocator that refuses to be OOM-killed

The counter that drives all of this is maintained by a custom global allocator, which is also where the hard limit lives:

 rust
 
 
// crates/polars-ooc/src/global_alloc.rs:58
#[cold]
#[inline(never)]
fn flush_local_drift() {
 LOCAL_ALLOC_DRIFT.with(|drift| {
 let drift_bytes = drift.get();
 let old = GLOBAL_ALLOC_SIZE.fetch_add(drift_bytes as u64, Ordering::AcqRel);
 drift.set(0);

 let total = old.wrapping_add(drift_bytes as u64);
 if total <= i64::MAX as u64 && total > polars_config::get_oomkill_threshold_bytes() {
 oomkill();
 }
 })
}

#[cold]
#[inline(never)]
fn oomkill() -> ! {
 // Can't allocate here, so write directly to stderr.
 const MSG: &[u8] = b"polars: memory usage exceeded POLARS_OOMKILL_THRESHOLD_MB or POLARS_OOMKILL_THRESHOLD_FRACTION, aborting\n";
 unsafe {
 libc::write(2, MSG.as_ptr().cast(), MSG.len() as _);
 libc::abort()
 }
}

Per-thread allocation deltas are accumulated in a thread-local and only flushed to the global counter when they exceed a drift threshold, so the common case is one relaxed atomic add per allocation and no contention. The flush is also where the abort happens: rather than letting the Linux OOM killer pick a victim (which is often the wrong process, and gives you no diagnostics), Polars aborts itself with a message, once the allocation total crosses POLARS_OOMKILL_THRESHOLD_MB or its fractional counterpart.

Recompute instead of materialize

The last technique is the most interesting one, because it trades CPU for memory at plan time, and does it optimally.

Consider a group-by whose input is select(a, b, a + b). The naive plan computes a + b, stores it alongside a and b, and the group-by node aggregates over three columns. But the group-by only needs a + b as a reduction input, and it can recompute it from a and b inside the node, over the rows it actually needs. Storing it would mean the intermediate payload of the group-by is wider — and that payload is exactly what gets spilled to disk when memory runs out.

So the streaming engine splits the select into a pre-select (what is materialized) and a post-select (what is recomputed downstream), choosing the split to minimize the width of the intermediate:

 rust
 
 
// crates/polars-stream/src/physical_plan/split_select.rs:266
/// Splits `exprs` into a pre-select and a post-select.
///
/// Selecting the returned pre-select on `input_schema` and then evaluating the returned
/// post-select expressions on that result, in order and each seeing the outputs of those
/// before it, produces exactly what selecting `exprs` on `input_schema` would, and:
///
/// * every post-select expression is elementwise,
/// * every expression in `must_preselect` is an output of the pre-select, under its own
/// output name,
/// * the expected bytes per row of the intermediate schema is minimal, with ties going
/// to materializing input columns rather than computed values,
/// * a post-select expression refers to an earlier post-select expression it contains by
/// its output name instead of computing it again.

The “minimal” claim is not a heuristic. The split is posed as a minimum-cut problem:

 rust
 
 
// crates/polars-stream/src/physical_plan/split_select.rs:331
 // Minimize the intermediate width as a minimum cut. Per node we have two booleans:
 //
 // post(n) the node's operation is applied in the post-select
 // avail(n) the node's value is available to the post-select
 //
 // with `avail(n) && !post(n)` meaning `n` is materialized, at cost `weight(n)`. The
 // only rule tying nodes together is that a node computed in the post-select needs its
 // inputs available there. Nodes are otherwise free to disagree, because an input of a
 // materialized node is recomputed inside the pre-select rather than passed through
 // the intermediate.
 //
 // Assigning "source side" to `true`, each cost is of the form
 // `w * [x == true && y == false]`, which is exactly an edge `x -> y` of capacity `w`,
 // so the minimum cut is the minimum intermediate width.

The graph is built with one node per expression sub-term, weights estimated from the output dtypes (a String column is assumed to cost 128 bits plus a 16-byte payload; a List is 64 bits plus 4 times its inner width), and the minimum cut is computed with Dinic’s algorithm:

 rust
 
 
// crates/polars-stream/src/physical_plan/split_select.rs:356
 let post = |idx: usize| 2 * idx;
 let avail = |idx: usize| 2 * idx + 1;
 let src = 2 * dag.len();
 let dst = 2 * dag.len() + 1;

 // The weights are scaled so that a tie in width goes to the cut that materializes the
 // fewest computed values.
 let scale = dag.len() as u64 + 1;
 let mut network = FlowNetwork::new(2 * dag.len() + 2);
 for (idx, dag_node) in dag.iter().enumerate() {
 let is_computed = !matches!(expr_arena.get(dag_node.node), AExpr::Column(_));
 let cost = dag_node.weight * scale + (dag_node.weight > 0 && is_computed) as u64;
 network.add_edge(avail(idx), post(idx), cost);
 if dag_node.splittable {
 for input in &dag_node.inputs {
 network.add_edge(post(idx), avail(*input), INF);
 }
 } else {
 network.add_edge(post(idx), dst, INF);
 }
 }

The scale factor is a tiebreak: among cuts of equal width, prefer the one that materializes the fewest computed values, i.e. the one that recomputes more and stores less.

And here is the caller, with the reasoning stated in a comment:

 rust
 
 
// crates/polars-stream/src/physical_plan/lower_group_by.rs:928
 // Keep the pre-select narrow: whatever is cheaper to recompute inside the group-by
 // node than to store in its (spilled) payload comes back as a post-select expression,
 // which the node evaluates over the rows that need it.
 let (mut pre_select_exprs, post_select_agg_inputs) = split_pre_post_select_minsize_elementwise(
 &splittable_agg_inputs,
 &must_preselect,
 &input_schema,
 expr_arena,
 )?;

The bottom line

There is no single “OOM protection policy” object in this engine. Memory safety is spread across four cooperating mechanisms, and it is worth seeing them as a ladder:

Reorder. Nodes are classified by how much they hold, and the scheduler picks the next accumulator by how much memory it will release. At most one accumulator per phase.
Narrow. Where a plan materializes an intermediate, the engine computes the minimum-width version of it with a min-cut, preferring to recompute a derived column instead of storing it.
Spill. Past a global budget, the out-of-core layer writes cold data to disk, and the group-by node pre-aggregates hot keys so that what does get spilled is as small as possible.
Abort. A global allocator counts every byte and aborts with a message before the kernel OOM killer does.

The trade-offs are real. Reordering costs a graph walk per phase. The min-cut is a polynomial-time computation at planning time, on a DAG of expression sub-terms. Spilling costs I/O and a re-read. And all of it is global rather than per-node: the budget is a process-wide byte count, and the scheduler’s notion of “holding” is a qualitative classification, not a measurement. What you get in exchange is that a query’s peak memory is a scheduling decision rather than a property of the plan you happened to write.

Polars Evaluates Expressions in an Arena

Jay Hubbert — Wed, 21 Oct 2026 06:00:00 +0000

TL;DR: A Polars expression exists in three shapes. The one you write is a tree of Arc<Expr>. The one the optimizer reasons about is an AExpr node living in a flat Arena<AExpr>, addressed by an 8-byte Node index. The one that touches data is an Arc<dyn PhysicalExpr> tree.

The middle shape is the interesting one: because the plan is a vector plus integers, cloning it is copying indices, comparing two subtrees is comparing integers, and nothing in it is individually freed until the whole arena drops.

Every dataframe engine has to answer the same question twice: how do I describe an expression, and how do I run it? The naive answer is to use one structure for both, and to run it by walking the tree.

Polars does not. It has three distinct representations, and the transitions between them are explicit, one-way, and cheap. This post is about the middle one: the arena.

The shape you write: a tree of `Arc<Expr>`

The user-facing expression is an enum whose children are Arc<Expr>:

 rust
 
 
// crates/polars-plan/src/dsl/expr/mod.rs:76
pub enum Expr {
 /// Values in a `eval` context.
 ///
 /// Equivalent of `pl.element()`.
 Element,
 Alias(Arc<Expr>, PlSmallStr),
 Column(PlSmallStr),
 Selector(Selector),
 Literal(LiteralValue),
 DataTypeFunction(DataTypeFunction),
 BinaryExpr {
 left: Arc<Expr>,
 op: Operator,
 right: Arc<Expr>,
 },
 Cast {
 expr: Arc<Expr>,
 dtype: DataTypeExpr,
 options: CastOptions,
 },

This is a perfectly good API shape: it is cheap to clone (cloning an Arc is a refcount bump), it is easy to pattern match on, and it is what the Python bindings, the R bindings, and the Expr builder methods all produce.

It is also a bad shape for a plan. A tree of Arc means:

every node lives in its own heap allocation, scattered across the heap;
“the same expression” is a deep structural comparison, not a pointer comparison;
you cannot hand a sub-expression to another subsystem without either cloning it or keeping the Arc alive, which keeps the whole tree alive.

Polars needs all three capabilities constantly. The optimizer compares sub-expressions to find common subplans. The projection pushdown needs to answer “does this expression mention column a?”. The CSE pass needs to deduplicate identical subtrees. So the DSL tree gets converted, once, into something flat.

The arena

The whole arena is 172 lines in crates/polars-utils/src/arena.rs, and the relevant parts are tiny:

 rust
 
 
// crates/polars-utils/src/arena.rs:26
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Ord, PartialOrd)]
#[repr(transparent)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct Node(pub usize);

...

#[derive(Debug, Clone)]
#[cfg_attr(feature = "ir_serde", derive(Serialize, Deserialize))]
pub struct Arena<T> {
 version: u32,
 items: Vec<T>,
}

and adding a node is a Vec::push plus an index:

 rust
 
 
// crates/polars-utils/src/arena.rs:57
 pub fn add(&mut self, val: T) -> Node {
 let idx = self.items.len();
 self.items.push(val);
 Node(idx)
 }

Reading a node is a bounds-checked index:

 rust
 
 
// crates/polars-utils/src/arena.rs:106
 #[inline]
 pub fn get(&self, idx: Node) -> &T {
 self.items.get(idx.0).unwrap()
 }

That is the entire data structure. A Node is a usize — a Copy, Eq, Hash, Ord index into a Vec. Everything else in the design follows from that.

The IR node: `AExpr`

The expression IR is a single enum whose children are Nodes rather than pointers:

 rust
 
 
// crates/polars-plan/src/plans/aexpr/mod.rs:180
/// IR expression node that is allocated in an [`Arena`][polars_utils::arena::Arena].
#[derive(Clone, Debug, Default)]
#[cfg_attr(feature = "ir_serde", derive(serde::Serialize, serde::Deserialize))]
pub enum AExpr {
 /// Values in a `eval` context.
 ///
 /// Equivalent of `pl.element()`.
 Element,
 Explode {
 expr: Node,
 options: ExplodeOptions,
 },
 Column(PlSmallStr),
 ...
 BinaryExpr {
 left: Node,
 op: Operator,
 right: Node,
 },
 Cast {
 expr: Node,
 dtype: DataType,
 options: CastOptions,
 },

Note what is not here: no Box, no Arc, no dyn. The recursive edges of the expression graph are plain usizes. The graph is stored in the arena; the enum only carries the edges.

The handle: `ExprIR`

A plan does not hold AExprs directly. It holds ExprIRs, which pair a node index with the output name and a memoized dtype:

 rust
 
 
// crates/polars-plan/src/plans/expr_ir.rs:70
#[derive(Clone, Debug)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct ExprIR {
 /// Output name of this expression.
 output_name: OutputName,
 /// Output dtype of this expression
 /// Reduced expression.
 /// This expression is pruned from `alias` and already expanded.
 node: Node,
 #[cfg_attr(feature = "serde", serde(skip))]
 output_dtype: OnceLock<DataType>,
}

ExprIR is Clone, and cloning it copies a Node (8 bytes), an OutputName, and a OnceLock<DataType>. It does not copy the expression subtree, because the subtree lives in the arena and is shared by reference. This is the “cheap cloning of expression trees” property, and it is not an optimization bolted on later — it falls out of the representation.

ExprIR also caches the dtype:

 rust
 
 
// crates/polars-plan/src/plans/expr_ir.rs:291
 pub fn dtype(&self, schema: &Schema, expr_arena: &Arena<AExpr>) -> PolarsResult<&DataType> {
 match self.output_dtype.get() {
 Some(dtype) => Ok(dtype),
 None => {
 let dtype = expr_arena
 .get(self.node)
 .to_dtype(&ToFieldContext::new(expr_arena, schema))?;
 let _ = self.output_dtype.set(dtype);
 Ok(self.output_dtype.get().unwrap())
 },
 }
 }

Since the arena is immutable once built, the dtype of a node is a property of the node, so every ExprIR referring to it agrees. The OnceLock turns a repeated schema query into a pointer read.

From `Expr` to `ExprIR`

The conversion is a recursive walk that bottoms out in a single arena.add:

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/expr_to_ir.rs:38
/// Converts expression to AExpr and adds it to the arena, which uses an arena (Vec) for allocation.
#[recursive]
pub(super) fn to_aexpr_impl(
 expr: Expr,
 ctx: &mut ExprToIRContext,
) -> PolarsResult<(Node, PlSmallStr)> {

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/expr_to_ir.rs:170
 Expr::BinaryExpr { left, op, right } => {
 let (l, output_name) = recurse_arc!(left)?;
 let (r, _) = recurse_arc!(right)?;
 (
 AExpr::BinaryExpr {
 left: l,
 op,
 right: r,
 },
 output_name,
 )
 },

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/expr_to_ir.rs:681
 Ok((ctx.arena.add(v), output_name))
}

The recursion is bottom-up: children are converted first, so by the time the parent is built, its children already have indices. The parent is pushed onto the arena last. The resulting arena is in post-order, which is also the order the physical planner wants to walk.

The whole plan uses two arenas — one for plan nodes, one for expressions:

 rust
 
 
// crates/polars-plan/src/plans/conversion/dsl_to_ir/mod.rs:58
pub fn to_alp(
 lp: DslPlan,
 expr_arena: &mut Arena<AExpr>,
 lp_arena: &mut Arena<IR>,
 // Only `SIMPLIFY_EXPR`, `TYPE_COERCION`, `TYPE_CHECK`, and `PREDICATE_PUSHDOWN` are respected.
 opt_flags: &mut OptFlags,
) -> PolarsResult<Node> {

Lowering to physical expressions

Now the interesting part: the arena is walked to build the operators that actually touch data. The walk is a single match on the IR enum — this is where the plan-time dispatch happens:

 rust
 
 
// crates/polars-expr/src/planner.rs:132
#[recursive]
fn create_physical_expr_inner(
 expression: Node,
 expr_arena: &mut Arena<AExpr>,
 schema: &SchemaRef, // Schema of the input.
 state: &mut ExpressionConversionState,
) -> PolarsResult<Arc<dyn PhysicalExpr>> {
 use AExpr::*;

 let aexpr = expr_arena.get(expression);
 match aexpr.clone() {

 rust
 
 
// crates/polars-expr/src/planner.rs:301
 BinaryExpr { left, op, right } => {
 let output_field = expr_arena
 .get(expression)
 .to_field(&ToFieldContext::new(expr_arena, schema))?;
 let is_scalar = is_scalar_ae(expression, expr_arena);
 let lhs = create_physical_expr_inner(left, expr_arena, schema, state)?;
 let rhs = create_physical_expr_inner(right, expr_arena, schema, state)?;
 Ok(Arc::new(phys_expr::BinaryExpr::new(
 lhs,
 op,
 rhs,
 node_to_expr(expression, expr_arena),
 state.local.has_lit,
 state.allow_threading,
 is_scalar,
 output_field,
 )))
 },

So the dynamic dispatch that a naive engine would pay per row is paid per plan node, at planning time. The match is on a #[derive(Clone)] enum, so the compiler can, in principle, turn the whole arm dispatch into a jump table or a set of specialized branches. What survives into the hot loop is Arc<dyn PhysicalExpr> — a vtable call per operator, not per row.

Note the node_to_expr(expression, expr_arena) argument. Every physical expression carries a copy of the original DSL expression, purely so that error messages can say “this failed in col("a") + col("b")”. That is a deliberate, expensive-in-the-plan-but-free-at-runtime trade: the arena makes it possible to reconstruct the DSL tree lazily, only on the error path.

 rust
 
 
// crates/polars-plan/src/plans/conversion/ir_to_dsl.rs:7
/// converts a node from the AExpr arena to Expr
#[recursive]
pub fn node_to_expr(node: Node, expr_arena: &Arena<AExpr>) -> Expr {
 let expr = expr_arena.get(node).clone();

 match expr {
 AExpr::Element => Expr::Element,
 AExpr::Explode { expr, options } => Expr::Explode {
 input: Arc::new(node_to_expr(expr, expr_arena)),
 options,
 },

The dispatch table

Not every IR node becomes a bespoke physical operator. Most function expressions are lowered into a generic ApplyExpr that calls a closure. The mapping from IR function to closure lives in one big match in crates/polars-expr/src/dispatch/mod.rs:

 rust
 
 
// crates/polars-expr/src/dispatch/mod.rs:175
/// `input` and `expr_arena` let a function pick a variant based on its arguments, e.g. a
/// constant argument that can be prepared once.
pub fn function_expr_to_udf(
 func: IRFunctionExpr,
 input: &[ExprIR],
 expr_arena: &Arena<AExpr>,
) -> SpecialEq<Arc<dyn ColumnsUdf>> {
 use IRFunctionExpr as F;
 match func {
 // Namespaces
 #[cfg(feature = "dtype-array")]
 F::ArrayExpr(func) => array::function_expr_to_udf(func),
 F::BinaryExpr(func) => binary::function_expr_to_udf(func),
 #[cfg(feature = "dtype-categorical")]
 F::Categorical(func) => cat::function_expr_to_udf(func),
 ...
 F::Boolean(func) => boolean::function_expr_to_udf(func, input, expr_arena),

and the closures are built with a small macro that fixes the calling convention (first column is the root, the rest are arguments):

 rust
 
 
// crates/polars-expr/src/dispatch/mod.rs:58
/// `Fn(&Series, args)`
#[macro_export]
macro_rules! map {
 ($func:path) => {{
 let f = move |c: &mut [::polars_core::prelude::Column]| {
 let c = &c[0];
 $func(c)
 };

 SpecialEq::new(Arc::new(f))
 }};

That is the whole extension mechanism for user-defined and built-in functions: an IR enum variant, one arm in the dispatch table, one function taking &Column.

Where the `dyn` actually is

It is worth being precise about the dynamic dispatch, because the usual claim (“Polars avoids Box<dyn>”) is only half true and the half that is false matters.

The DSL layer is an enum with Arc<Expr> children — there is no Box<dyn Expr> anywhere in it. The IR layer is an enum with Node children, and it has no Box<dyn> either. The dyn appears exactly twice, and both times it is bounded by the number of plan nodes, never by the number of rows:

Arc<dyn PhysicalExpr>, the operator tree. One vtable call per operator per chunk of data, and the kind of operator is resolved at planning time by the match you saw above.
Arc<dyn ColumnsUdf>, the function layer. One vtable call per function application per chunk.

So the cost of dynamic dispatch in Polars is O(operators) per chunk, not O(rows). Compare that with the design where the expression tree itself is Box<dyn Expr> and is walked per row: there the cost is O(rows × operators). The arena is what makes it possible to have the first shape without paying the second.

The other thing the arena changes is where the type information lives. In a Box<dyn Expr> design, the type of a sub-expression is a vtable lookup at runtime. In the IR, the type of a node is a property of the enum variant, known at compile time; the only thing resolved at runtime is the data type, and even that is memoized in the ExprIR.

What the arena buys, concretely

Cheap equality and hashing. Two ExprIRs that refer to the same arena node are the same expression. The canonicalization map exploits this to assign structural equivalence classes to subtrees:

 rust
 
 
// crates/polars-plan/src/plans/aexpr/canonical.rs:23
slotmap::new_key_type! {
 /// Identifies an `AExpr` up to structural equality: two nodes get the same
 /// `CanonicalExprId` if and only if they represent the same expression.
 pub struct CanonicalExprId;
}

Cheap duplication. Duplicating a subtree is one clone of the item plus one push:

 rust
 
 
// crates/polars-utils/src/arena.rs:144
impl<T: Clone> Arena<T> {
 pub fn duplicate(&mut self, node: Node) -> Node {
 let item = self.items[node.0].clone();
 self.add(item)
 }
}

One allocation strategy. Every expression node in the whole plan lives in one Vec. No per-node allocator calls, no Box, no Arc refcount traffic while the plan is being built and rewritten. When the optimizer pushes expressions around — predicate pushdown rewriting a filter, projection pushdown splitting a select, CSE inserting a cache — it copies Nodes, not trees.

Node lifetime. The trade-off is real and worth stating plainly: an arena never frees an individual node. Arena has clear() and pop(), but there is no “free this node”. A rewritten plan leaks the expressions it replaced until the arena itself is dropped. For a query plan, whose lifetime is one collect() call, that is the right trade. It is the wrong trade for a long-lived structure that mutates one node at a time — which is exactly why Arena is not used for, say, the physical operator tree, where each operator has its own lifetime and is dropped independently.

Caching the arena across calls

Because the arena is the plan, and the plan is expensive to rebuild, a LazyFrame keeps it around:

 rust
 
 
// crates/polars-lazy/src/frame/cached_arenas.rs:9
pub(crate) struct CachedArena {
 lp_arena: Arena<IR>,
 expr_arena: Arena<AExpr>,
}

impl LazyFrame {
 pub fn set_cached_arena(&self, lp_arena: Arena<IR>, expr_arena: Arena<AExpr>) {
 let mut cached = self.cached_arena.lock().unwrap();
 *cached = Some(CachedArena {
 lp_arena,
 expr_arena,
 });
 }

 rust
 
 
// crates/polars-lazy/src/frame/cached_arenas.rs:116
 pub(super) fn get_arenas(&mut self) -> (Arena<IR>, Arena<AExpr>) {
 match self.cached_arena.lock().unwrap().as_mut() {
 Some(arenas) => (arenas.lp_arena.clone(), arenas.expr_arena.clone()),
 None => (Arena::with_capacity(16), Arena::with_capacity(16)),
 }
 }

collect_schema() on a LazyFrame converts the DSL plan to IR once, caches both arenas, and reuses them for every subsequent schema query or collect(). Since Arena is Clone and cloning a Vec<AExpr> is a memcpy of the node vector, “reuse the plan” is nearly free. Note the arena has a version counter, bumped on every clear(), so a stale cached plan can be detected rather than silently reused.

The bottom line

The arena is not a performance micro-optimization; it is a representation decision that makes a class of algorithms possible.

The DSL tree (Arc<Expr>) is the right shape for a user API: cheap to build, cheap to share.
The IR (Arena<AExpr> + Node) is the right shape for a plan: flat, index-addressed, comparable by integer, duplicable by memcpy.
The physical tree (Arc<dyn PhysicalExpr>) is the right shape for execution: one vtable call per operator per chunk, and the dispatch on the kind of operator happens at planning time.

The one thing to remember when reading the code: Node is not a pointer, it is an index. That single fact explains why the plan is cheap to copy, why the optimizer can rewrite it freely, why common subplan elimination works, and why every physical operator can afford to carry a copy of the original expression for error messages.

Polars Writes Parquet From a Streaming Pipeline

Jay Hubbert — Wed, 21 Oct 2026 05:50:00 +0000

TL;DR: A Parquet file is a sequence of row groups, and a row group is a sequence of pages, and pages are compressed. That means a Parquet writer cannot just start writing bytes as rows arrive — it has to decide where the row group boundaries are, and it cannot know that until it has seen enough rows. Polars solves this with a resize pipeline between the streaming engine and the writer: morsels arrive at whatever size the upstream produced them, get re-cut into row groups of a target size, and are then encoded column-by-column in parallel while a single task writes the pages to the file in order.

The hard part of streaming Parquet is not the encoding. It is that a row group boundary is a decision, and the decision depends on data you have not seen yet. If you flush a row group too eagerly you get a million tiny row groups and a file that is slow to read. If you wait too long you hold the whole thing in memory.

Let me walk through how the pieces fit, starting from the Python API and going down to the byte writes.

The API

The entry point is sink_parquet on a LazyFrame (crates/polars-python/src/lazyframe/general.rs:796):

 rust
 
 
 #[cfg(feature = "parquet")]
 #[pyo3(signature = (
 target, sink_options, compression, compression_level, statistics, row_group_size, data_page_size,
 metadata, arrow_schema
 ))]
 fn sink_parquet(
 &self,
 py: Python<'_>,
 target: PyFileSinkDestination,
 sink_options: PySinkOptions,
 compression: &str,
 compression_level: Option<i32>,
 statistics: Wrap<StatisticsOptions>,
 row_group_size: Option<usize>,
 data_page_size: Option<usize>,
 metadata: Wrap<Option<KeyValueMetadata>>,
 arrow_schema: Option<Wrap<ArrowSchema>>,
 ) -> PyResult<PyLazyFrame> {

It builds a ParquetWriteOptions and hands it to LazyFrame::sink (crates/polars-lazy/src/frame/mod.rs:807):

 rust
 
 
 pub fn sink(
 mut self,
 sink_type: SinkDestination,
 file_format: FileWriteFormat,
 unified_sink_args: UnifiedSinkArgs,
 ) -> PolarsResult<Self> {
 polars_ensure!(
 !matches!(self.logical_plan, DslPlan::Sink { .. }),
 InvalidOperation: "cannot create a sink on top of another sink"
 );

 self.logical_plan = DslPlan::Sink {

Note that sink is a plan operation, not an execution one. The returned LazyFrame is a plan that ends in a sink; it is executed when you collect() it. This is why sink_parquet returns a PyLazyFrame and not a path.

The options themselves (crates/polars-io/src/parquet/write/options.rs:24):

 rust
 
 
pub struct ParquetWriteOptions {
 /// Data page compression
 pub compression: ParquetCompression,
 /// Compute and write column statistics.
 pub statistics: StatisticsOptions,
 /// If `None` will be all written to a single row group.
 pub row_group_size: Option<usize>,
 /// if `None` will be 1024^2 bytes
 pub data_page_size: Option<usize>,
 /// Custom file-level key value metadata
 pub key_value_metadata: Option<KeyValueMetadata>,
 pub arrow_schema: Option<ArrowSchemaRef>,
 #[cfg_attr(feature = "serde", serde(default))]
 pub compat_level: Option<CompatLevel>,
}

row_group_size: None does not mean “one row group” in the streaming engine — the comment is about the old, non-streaming path. The streaming path has its own default, which we will meet shortly.

The sink node

The sink is a node in the streaming graph (crates/polars-stream/src/nodes/io_sinks/mod.rs:30):

 rust
 
 
pub struct IOSinkNode {
 name: PlSmallStr,
 state: IOSinkNodeState,
 io_metrics: Option<Arc<IOMetrics>>,
 verbose: bool,
 partitioned_by_key: bool,
}

It has a peculiar lifecycle: it does not do any work during the query. It waits for the input pipe to report Done, and only then does it initialise the writer pipeline and wait for it (crates/polars-stream/src/nodes/io_sinks/mod.rs:84):

 rust
 
 
 recv[0] = if recv[0] == PortState::Done {
 // Ensure initialize / writes empty file for empty output.
 self.state
 .initialize(&self.name, execution_state, self.io_metrics.clone())?;

 match std::mem::replace(&mut self.state, IOSinkNodeState::Finished) {
 IOSinkNodeState::Initialized {
 phase_channel_tx,
 task_handle,
 } => {

Wait — that reads as if the sink receives no morsels at all during the query. It does receive them; they just go somewhere else. The sink takes a serial receive port (crates/polars-stream/src/nodes/io_sinks/mod.rs:141):

 rust
 
 let phase_morsel_rx = recv_ports[0].take().unwrap().serial();

and hands it to a “phase” channel. The point of the phase mechanism is that the sink may be executed in several phases by the graph scheduler, and the writer pipeline must survive across those phases. The phase task relays morsels from each phase into a single long-lived channel (crates/polars-stream/src/nodes/io_sinks/mod.rs:222):

 rust
 
 
 executor::spawn(TaskPriority::High, task_metrics, async move {
 let mut morsel_seq: u64 = 1;

 while let Ok(mut phase_rx) = phase_channel_rx.recv().await {
 while let Ok(mut morsel) = phase_rx.recv().await {
 morsel.set_seq(MorselSeq::new(morsel_seq));
 morsel_seq = morsel_seq.saturating_add(1);

 if multi_phase_tx.send(morsel).await.is_err() {
 break;
 }
 }
 }
 });

Note the sequence numbers are renumbered here. The sink does not care about the upstream’s ordering guarantees; it cares about producing a file, and for that it just needs a total order over the morsels it receives. Renumbering makes the writer’s ordering logic trivial.

The sink also always sends one empty morsel at the start (crates/polars-stream/src/nodes/io_sinks/mod.rs:216):

 rust
 
 
 let _ = multi_phase_tx.try_send(Morsel::new_unregistered(
 DataFrame::empty_with_arc_schema(config.input_schema.clone()),
 MorselSeq::new(0),
 SourceToken::default(),
 ));

That is how an empty result still produces a valid, empty Parquet file rather than nothing at all.

The resize pipeline: morsels are the wrong size

Now the interesting part. The upstream produces morsels of about POLARS_IDEAL_MORSEL_SIZE rows (100,000 by default). A Parquet row group wants a different number of rows, and the two are not related: the user may have asked for row_group_size = 1_000_000, or for something byte-driven, or left it alone.

So between the engine and the writer there is a stage that re-cuts the morsels. That stage is MorselResizePipeline (crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs:22):

 rust
 
 
/// Splits large morsels / combines small morsels.
pub struct MorselResizePipeline {
 pub schema: Arc<Schema>,
 pub target_sink_morsel_size: TargetSinkMorselSize,
 pub inflight_morsel_semaphore: Arc<tokio::sync::Semaphore>,
 pub morsel_rx: connector::Receiver<Morsel>,
 pub morsel_tx: connector::Sender<SinkMorsel>,
}

It is not a trivial pass-through. It maintains a buffer of received dataframes, and it decides, for each outgoing chunk, how many rows to take from the buffer (crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs:54):

 rust
 
 
 loop {
 let df_to_send = if let Some(num_rows_to_take) = num_rows_to_take_iter.next() {
 take_n_rows_from_buffered(Arc::clone(&schema), &mut buffered_rows, num_rows_to_take)
 } else if let Some(n_rows) = next_chunk_lengths.pop() {
 let next_chunk_size = logical_received_size
 .calc_delta(n_rows as IdxSize, physical_received_size)
 .unwrap();

 let logical_buffered_size = logical_received_size.checked_sub(sent_size).unwrap();

 let send_buffered_as_one_df;

 (send_buffered_as_one_df, num_rows_to_take_iter) = target_sink_morsel_size
 .calc_next_splits(logical_buffered_size, next_chunk_size);

 logical_received_size = logical_received_size.add(next_chunk_size)?;

 if send_buffered_as_one_df {
 take_n_rows_from_buffered(
 Arc::clone(&schema),
 &mut buffered_rows,
 idxsize_to_u64(logical_buffered_size.num_rows),
 )
 } else {
 continue;
 }
 } else if let Ok(morsel) = {

Three cases, and they are worth reading carefully:

There is a pending split plan. Take that many rows from the buffer and send them.
There is a new chunk of rows. Ask the target size how to split the combination of what is buffered and what just arrived. If the answer is “flush the buffer as one chunk”, do that; otherwise adopt the new split plan and loop.
No more rows. Flush whatever is left as one final chunk.

The logical_received_size / physical_received_size distinction is there because a morsel can have several Arrow chunks, and the split logic works on chunk boundaries where possible. The sent_size tracks what has already gone out.

Taking rows out of the buffer is a stack (crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs:188):

 rust
 
 
fn take_n_rows_from_buffered(
 schema: Arc<Schema>,
 buffered_rows: &mut VecDeque<DfWithOffset>,
 mut n_rows: u64,
) -> DataFrame {
 if n_rows == 0 {
 return DataFrame::empty_with_arc_schema(schema);
 }

 let mut stacked_df: Option<DataFrame> = None;

 while n_rows != 0 {
 let next_df = buffered_rows.front_mut().unwrap();

 let next_df = if next_df.height() as u64 > n_rows {
 next_df.split_off_front(n_rows as usize)
 } else {
 buffered_rows.pop_front().unwrap().into_df()
 };

 if next_df.height() as u64 == n_rows && stacked_df.is_none() {
 return next_df;
 }
 ...

If the requested row count happens to land exactly on a frame boundary, the frame is returned as-is — no copy, no vertical stack. Only when the boundary falls in the middle of a frame does the frame get split and the pieces stacked.

Deciding the row group size

The decision logic is TargetSinkMorselSize::calc_next_splits (crates/polars-stream/src/nodes/io_sinks/components/size.rs:211):

 rust
 
 
impl TargetSinkMorselSize {
 pub fn calc_next_splits(
 &self,
 buffered_size: RowCountAndSize,
 incoming_size: RowCountAndSize,
 ) -> (bool, PartSizesIter) {
 let combined_size = buffered_size.checked_add(incoming_size).unwrap();

 let mut flush_buffered_as_one_split = false;
 let (mut part_sizes_iter, mut limited_by) = self.build_part_sizes_iter(combined_size);

and build_part_sizes_iter is where the two limits — rows and bytes — are arbitrated (crates/polars-stream/src/nodes/io_sinks/components/size.rs:274):

 rust
 
 
 fn build_part_sizes_iter(&self, size: RowCountAndSize) -> (PartSizesIter, LimitedBy) {
 if size.num_rows == 0 {
 return (PartSizesIter::default(), LimitedBy::default());
 }

 let n_parts_by_num_rows = if self.target_num_rows_mode == SplitMode::Exact {
 u64::max(1, idxsize_to_u64(size.num_rows / self.target_num_rows))
 } else {
 calc_n_parts(
 idxsize_to_u64(size.num_rows),
 NonZeroU64::new(idxsize_to_u64(self.target_num_rows.get())).unwrap(),
 )
 };

 let mut max_parts_by_num_bytes = 0;
 let mut n_parts_by_num_bytes = 0;

 if self.target_num_bytes.get() != u64::MAX {
 max_parts_by_num_bytes = idxsize_to_u64(size.num_rows / self.target_num_bytes_min_rows);
 n_parts_by_num_bytes = calc_n_parts(size.num_bytes, self.target_num_bytes);
 };

The SplitMode distinction is the interesting bit:

Exact means “every row group must have exactly target_num_rows rows”. This is what you want when the user set row_group_size explicitly: the file must have the row group layout they asked for.
Approximate means “get close to target_num_rows”. This is the default when the user did not ask for anything, and it lets the pipeline roll a row group as soon as it has roughly enough rows, which keeps the pipeline moving instead of waiting for an exact count.

The calc_n_parts helper is the shared arithmetic (crates/polars-utils/src/calc_morsel_split.rs:69):

 rust
 
 
/// Number of parts to split `size` to minimize the average part size difference to `target_part_size`.
pub fn calc_n_parts(size: u64, target_part_size: NonZeroU64) -> u64 {
 if size <= target_part_size.get() {
 return if size == 0 { 0 } else { 1 };
 }

 let n_parts = size / target_part_size;

 (n_parts..=n_parts.saturating_add(1))
 .min_by_key(|n_parts| (size / *n_parts).abs_diff(target_part_size.get()))
 .unwrap()
}

Only two candidates are considered: n and n+1 parts. The winner is the one whose resulting part size is closest to the target. That is enough, because size / n and size / (n+1) straddle the target.

The part sizes themselves come from PartSizesIter (crates/polars-utils/src/calc_morsel_split.rs:45):

 rust
 
 
impl Iterator for PartSizesIter {
 type Item = u64;

 fn next(&mut self) -> Option<Self::Item> {
 self.remaining_parts = self.remaining_parts.checked_sub(1)?;
 Some(self.base_part_size + (self.remaining_parts >= self.remainder_cutoff) as u64)
 }

It is a counting-sort-style distribution of the remainder: the first remainder_cutoff parts get one extra row, the rest do not. The result is that sum(part_sizes) == total exactly, with the parts differing by at most one row.

There is one more subtlety in calc_next_splits (crates/polars-stream/src/nodes/io_sinks/components/size.rs:236):

 rust
 
 
 if limited_by == LimitedBy::Rows
 && part_sizes_iter.len() <= 1
 && self.target_num_rows_mode != SplitMode::Exact
 && part_sizes_iter.base_part_size().checked_mul(2).is_some_and(
 // base_part_size < (4/3) * target_num_rows
 |double_base_part_size| {
 u64::abs_diff(
 double_base_part_size,
 idxsize_to_u64(self.target_num_rows.get()),
 ) < u64::abs_diff(
 part_sizes_iter.base_part_size(),
 idxsize_to_u64(self.target_num_rows.get()),
 )
 },
 )
 {
 // Wait for more data to have a fuller chunk.
 part_sizes_iter = PartSizesIter::default()
 }

In Approximate mode, if the accumulated data is already close to target_num_rows, the pipeline waits for more rather than flushing a slightly-short row group. The 4/3 heuristic: if doubling the current size would land closer to the target than the current size does, then the current size is “under” the target by more than a factor of 4/3, so it is worth waiting.

The defaults

ParquetWriterStarter decides the target (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:52):

 rust
 
 
 fn target_sink_morsel_size(&self) -> TargetSinkMorselSize {
 let target_num_bytes_min_rows = const { NonZeroIdxSize::new(16384).unwrap() };

 if let Some(row_group_size) = self.row_group_size
 && row_group_size > 0
 {
 TargetSinkMorselSize {
 target_num_rows: row_group_size.try_into().unwrap(),
 target_num_bytes: NonZeroU64::MAX,
 target_num_bytes_min_rows,
 target_num_rows_mode: SplitMode::Exact,
 }
 } else {
 let (env_num_rows, env_num_bytes) = ideal_sink_morsel_size_env();

 TargetSinkMorselSize {
 target_num_rows: env_num_rows
 .unwrap_or(const { NonZeroIdxSize::new(122_880).unwrap() }),
 target_num_bytes: env_num_bytes.unwrap_or(NonZeroU64::MAX),
 target_num_bytes_min_rows,
 target_num_rows_mode: SplitMode::Approximate,
 }
 }
 }

So:

If the user set row_group_size, the mode is Exact and the byte limit is disabled. The file will have exactly the row group layout requested.
Otherwise the default is 122,880 rows in Approximate mode. That number is 120 × 1024, and it is a deliberate choice: Parquet readers and the ecosystem generally cope better with row groups in the low hundreds of thousands of rows.
The byte limit is u64::MAX (disabled) unless POLARS_IDEAL_SINK_MORSEL_SIZE_BYTES is set, in which case the target_num_bytes_min_rows of 16,384 acts as a floor: a row group never gets smaller than 16k rows even if the byte budget says so.

Three tasks: resize, encode, write

The single-file sink pipeline is assembled in crates/polars-stream/src/nodes/io_sinks/pipeline_initialization/single_file.rs:136:

 rust
 
 
 let (writer_tx, writer_rx) = connector::connector();
 let writer_handle = file_writer_starter.start_file_writer(
 writer_rx,
 file_open_task,
 num_pipelines_per_sink,
 execution_state.task_metrics.clone(),
 )?;

 let schema = Arc::clone(&file_schema);
 let inflight_morsel_semaphore =
 Arc::new(tokio::sync::Semaphore::new(inflight_morsel_limit.get()));

 let resize_pipeline = MorselResizePipeline {
 schema,
 target_sink_morsel_size,
 inflight_morsel_semaphore,
 morsel_rx,
 morsel_tx: writer_tx,
 };

The semaphore is the memory bound. Every outgoing sink morsel holds a permit, and the permit is released only when the data has actually been written (crates/polars-stream/src/nodes/io_sinks/components/sink_morsel.rs:3):

 rust
 
 
pub type SinkMorselPermit = tokio::sync::OwnedSemaphorePermit;

/// In-flight morsel in the IO sink. Holds a permit against a semaphore that restricts
/// the total number of sink morsels in memory.
pub struct SinkMorsel {
 df: DataFrame,
 /// Should only be dropped once the data associated with this morsel has been dropped from memory.
 permit: SinkMorselPermit,
}

The limit is num_pipelines + 4 (crates/polars-stream/src/nodes/io_sinks/config.rs:30):

 rust
 
 
 pub fn inflight_morsel_limit(&self, num_pipelines: NonZeroUsize) -> NonZeroUsize {
 ...
 NonZeroUsize::saturating_add(
 num_pipelines,
 // Additional buffer to accommodate head-of-line blocking
 4,
 )
 }

The extra four is for head-of-line blocking: if one row group takes longer to encode than the others, the other permits let the pipeline keep feeding the encoder.

Now the writer itself. ParquetWriterStarter::start_file_writer (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:102) sets up three things: a channel of encoded row groups, a writer task and an encoder task.

 rust
 
 
 let (encoded_row_group_tx, encoded_row_group_rx) = tokio::sync::mpsc::channel::<
 executor::AbortOnDropHandle<PolarsResult<EncodedRowGroup>>,
 >(num_pipelines.get());

The channel’s capacity is the number of pipelines, so up to that many row groups can be in flight — being encoded, or encoded and waiting to be written.

The WriteOptions are built once (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:106):

 rust
 
 
 let key_value_metadata = self.options.key_value_metadata.clone();
 let write_options = WriteOptions {
 statistics: self.options.statistics,
 compression: self.options.compression.into(),
 version: Version::V1,
 data_page_size: self.options.data_page_size,
 };

Version::V1 is hardcoded here — the streaming writer does not emit Parquet v2 files even though the option type allows it. compression is converted from the user-facing ParquetCompression enum (crates/polars-io/src/parquet/write/options.rs:55):

 rust
 
 
impl From<ParquetCompression> for CompressionOptions {
 fn from(value: ParquetCompression) -> Self {
 use ParquetCompression::*;
 match value {
 Uncompressed => CompressionOptions::Uncompressed,
 Snappy => CompressionOptions::Snappy,
 Gzip(level) => CompressionOptions::Gzip(level),
 Brotli(level) => CompressionOptions::Brotli(level),
 Lz4Raw => CompressionOptions::Lz4Raw,
 Zstd(level) => CompressionOptions::Zstd(level),
 }
 }
}

and the default is Zstd (crates/polars-io/src/parquet/write/options.rs:49):

 rust
 
 
impl Default for ParquetCompression {
 fn default() -> Self {
 Self::Zstd(None)
 }
}

The encoder: parallel across columns

The encoder task (crates/polars-stream/src/nodes/io_sinks/writers/parquet/row_group_encoder.rs:52) receives sink morsels and turns each into an encoded row group:

 rust
 
 
 while let Ok(morsel) = morsel_rx.recv().await {
 let arrow_schema = Arc::clone(&arrow_schema);
 let schema_descriptor = Arc::clone(&schema_descriptor);
 let encodings = Buffer::clone(&encodings);
 let column_task_metrics = task_metrics.clone();

 let row_group_encode_handle = executor::AbortOnDropHandle::new(executor::spawn(
 TaskPriority::High,
 task_metrics.as_deref(),
 async move {
 let (df, morsel_permit) = morsel.into_inner();
 let num_rows = df.height();

 let mut data: Vec<Vec<CompressedPage>> = Vec::with_capacity(num_leaf_columns);

The whole row group is encoded in a spawned task, and the handle is sent to the writer rather than the result. That is the key to the pipelining: the writer does not wait for the encoding to finish before it starts the next one.

Inside the task, the columns are encoded in parallel (crates/polars-stream/src/nodes/io_sinks/writers/parquet/row_group_encoder.rs:73):

 rust
 
 
 for fut in parallelize_first_to_local(
 TaskPriority::High,
 column_task_metrics.as_deref(),
 df.into_columns().into_iter().enumerate().map(|(i, c)| {
 let arrow_schema = Arc::clone(&arrow_schema);
 let schema_descriptor = Arc::clone(&schema_descriptor);
 let encodings = Buffer::clone(&encodings);

 async move {
 let parquet_type = &schema_descriptor.fields()[i];
 let encodings = encodings[i].as_slice();
 let array =
 c.as_materialized_series().rechunk().to_arrow_with_field(
 0,
 Cow::Borrowed(arrow_schema.get_at_index(i).unwrap().1),
 true,
 )?;

 let mut data: UnitVec<Vec<CompressedPage>> =
 UnitVec::with_capacity(num_leaf_columns);

 for encode_page_iter in array_to_columns(
 array,
 parquet_type.clone(),
 write_options,
 encodings,
 )? {
 let compressed_pages: Vec<CompressedPage> =
 Compressor::new_from_vec(
 encode_page_iter.map(|result| {
 result.map_err(|e| {
 ParquetError::FeatureNotSupported(format!(
 "reraised in polars: {e}",
 ))
 })
 }),
 write_options.compression,
 vec![],
 )
 .collect::<ParquetResult<_>>()?;

 data.push(compressed_pages)
 }

 PolarsResult::Ok(data)
 }
 }),
 ) {
 data.extend(fut.await?);
 }

parallelize_first_to_local keeps the first column on the current thread and spawns the rest (crates/polars-async/src/primitives/opt_spawned_future.rs:59):

 rust
 
 
/// Parallelizes futures across the computational async runtime.
///
/// As an optimization for cache access, the first future is kept on the current thread. If there
/// is only 1 future, then all data is kept on the current thread and spawn is not called at all.

That is a cache-locality optimisation: the first column’s data is already in this core’s cache, so encoding it here avoids a cross-core transfer. The remaining columns are spawned, so a wide table is encoded on all cores at once.

Note the order of operations inside the column future: rechunk() to a single Arrow chunk, convert to Arrow with the field’s metadata attached, then array_to_columns to produce the encoded (but uncompressed) pages, then Compressor to compress them. The compression happens per page, inside the same parallel task, so the compressed bytes never leave the core that produced them.

The writer: one task, one file

The writer task is deliberately on the I/O runtime, not the compute one (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:117):

 rust
 
 
 let io_handle = tokio_handle_ext::AbortOnDropHandle(
 ASYNC.spawn(
 io_writer::IOWriter {
 file,
 encoded_row_group_rx,
 arrow_schema,
 schema_descriptor: Arc::clone(&schema_descriptor),
 write_options,
 encodings: Buffer::clone(&encodings),
 key_value_metadata,
 num_leaf_columns,
 }
 .run(),
 ),
 );

and its body is short (crates/polars-stream/src/nodes/io_sinks/writers/parquet/io_writer.rs:57):

 rust
 
 
 while let Some(handle) = encoded_row_group_rx.recv().await {
 let EncodedRowGroup {
 num_rows,
 data,
 morsel_permit,
 } = handle.await?;
 assert_eq!(data.len(), num_leaf_columns);
 parquet_writer.write_row_group(num_rows as u64, &data)?;
 drop(data);
 drop(morsel_permit);
 }

handle.await? is where the writer waits for the row group to finish encoding — but by then the next row group is already being encoded on another core. So the pipeline is:

text

 resize task encoder task(s) writer task
 (compute) (compute, N in flight) (IO runtime)
 ┌───────────┐ morsel ┌──────────────┐ pages ┌──────────────┐
 │ cut rows │ ───────► │ encode cols │ ──────► │ write bytes │
 │ to target │ │ + compress │ (chan) │ to file │
 └───────────┘ └──────────────┘ └──────────────┘

The morsel_permit is dropped after the write, so the semaphore that bounds in-flight memory is only released once the data is on disk. That is what keeps the memory bound honest.

The BatchedWriter wraps the actual Parquet file writer (crates/polars-io/src/parquet/write/batched_writer.rs:29):

 rust
 
 
pub struct BatchedWriter<W: Write> {
 // A mutex so that streaming engine can get concurrent read access to
 // compress pages.
 //
 // @TODO: Remove mutex when old streaming engine is removed
 pub(super) writer: Mutex<FileWriter<W>>,
 // @TODO: Remove when old streaming engine is removed
 pub(super) parquet_schema: SchemaDescriptor,
 pub(super) encodings: Buffer<Vec<Encoding>>,
 pub(super) options: WriteOptions,
 pub(super) parallel: bool,
 pub(super) key_value_metadata: Option<KeyValueMetadata>,
}

and write_row_group hands the already-compressed pages straight through (crates/polars-io/src/parquet/write/batched_writer.rs:104):

 rust
 
 
 /// Note: `num_rows` can be passed as `u64::MAX` to infer `num_rows` from the encoded data.
 pub fn write_row_group(
 &mut self,
 num_rows: u64,
 rg: &[Vec<CompressedPage>],
 ) -> PolarsResult<()> {
 let writer = self.writer.get_mut().unwrap();
 let rg = DynIter::new(rg.iter().map(|col_pages| {
 Ok(DynStreamingIterator::new(
 fallible_streaming_iterator::convert(col_pages.iter().map(PolarsResult::Ok)),
 ))
 }));
 writer.write(num_rows, rg)?;
 Ok(())
 }

No re-encoding, no re-compression — the pages arrive ready. The underlying FileWriter::write (crates/polars-parquet/src/parquet/write/file.rs:167) is where the bytes actually hit the file:

 rust
 
 
 /// Writes a row group to the file.
 ///
 /// This call is IO-bounded
 pub fn write<E>(
 &mut self,
 num_rows: u64,
 row_group: RowGroupIterColumns<'_, E>,
 ) -> ParquetResult<()>
 where
 ParquetError: From<E>,
 E: std::error::Error,
 {
 if self.offset == 0 {
 self.start()?;
 }
 let ordinal = self.row_groups.len();
 let (group, specs, size) = write_row_group(
 &mut self.writer,
 num_rows,
 self.offset,
 self.schema.columns(),
 row_group,
 ordinal,
 )?;
 self.offset += size;
 self.row_groups.push(group);
 self.page_specs.push(specs);
 Ok(())
 }

The row group metadata is accumulated in memory (row_groups, page_specs) and written once at the end. That is the one piece of state that grows with the number of row groups, but it is tiny compared to the data.

Finally the footer (crates/polars-stream/src/nodes/io_sinks/writers/parquet/io_writer.rs:73):

 rust
 
 
 parquet_writer.finish()?;
 let parquet_metadata = {
 let writer = parquet_writer.get_writer().lock().unwrap();
 let mut out = Vec::new();
 write_metadata_sidecar(&mut out, writer.metadata().unwrap())?;
 out
 };
 drop(parquet_writer);
 buffered_file.flush()?;
 drop(buffered_file);

 file.set_parquet_metadata(parquet_metadata);
 file.close(sync_on_close).await?;

The metadata sidecar is written to a separate buffer rather than into the file — that is what makes the partitioned sink possible, where several files’ row groups are described by one metadata blob.

What is parallel and what is serialised

Putting it all in one table:

stage	where	parallel?
resize (re-cut morsels)	compute executor	no, one task
encode + compress a row group	compute executor	yes, one task per column (first column kept local)
several row groups at once	compute executor	yes, up to `num_pipelines` in flight
write bytes to the file	I/O runtime	no, one task owns the file
file open / close	I/O runtime	no

The invariant is: the file itself is written by exactly one task, in row group order. Everything else is parallel. That is the whole design of the Parquet sink: parallelism is pushed as far upstream as possible, and the serial part is reduced to “append these bytes to this file”.

The price is that the writer is a bottleneck if the encoding is fast. With a wide table and cheap compression, the encoder can produce row groups faster than the disk can absorb them — and then the channel between them fills up and the encoder blocks. That is correct behaviour, not a bug: the backpressure just moves the bottleneck to where it actually is.

Partitioned sinks

The same machinery drives the partitioned sink, with one extra stage. start_partition_sink_pipeline (crates/polars-stream/src/nodes/io_sinks/pipeline_initialization/partition_by.rs:29) inserts a partitioner between the engine and the resize stage, and a distributor that opens files on demand.

Each partition gets its own writer, its own resize pipeline and its own row group sequence, so a partitioned write produces several independent Parquet files, each with its own row groups. The number of simultaneously open files is bounded (crates/polars-stream/src/nodes/io_sinks/config.rs:46):

 rust
 
 
 pub fn max_open_sinks(&self) -> NonZeroUsize {
 ...
 if self.target.is_cloud_location() {
 const { NonZeroUsize::new(512).unwrap() }
 } else {
 const { NonZeroUsize::new(128).unwrap() }
 }
 }

128 local files, 512 in the cloud. Beyond that the sink has to close and reopen files, which is why partitioned writes to object storage have a per-file size limit option (crates/polars-stream/src/nodes/io_sinks/pipeline_initialization/partition_by.rs:112):

 rust
 
 
 if let Some(file_size_limit) = file_size_limit {
 target_sink_morsel_size.target_num_rows = NonZeroIdxSize::min(
 target_sink_morsel_size.target_num_rows,
 file_size_limit.num_rows,
 );
 target_sink_morsel_size.target_num_bytes = NonZeroU64::min(
 target_sink_morsel_size.target_num_bytes,
 file_size_limit.num_bytes,
 );
 target_sink_morsel_size.target_num_rows_mode = SplitMode::Exact;
 }

Note that the file size limit also forces SplitMode::Exact: if you said “at most 1M rows per file”, the row groups have to add up to exactly that, not approximately.

The old, non-streaming path

For comparison, the classic path is much simpler (crates/polars-io/src/parquet/write/writer.rs:147):

 rust
 
 
 /// Write the given DataFrame in the writer `W`.
 /// Returns the total size of the file.
 pub fn finish(self, df: &mut DataFrame) -> PolarsResult<u64> {
 let chunked_df = chunk_df_for_writing(df, self.row_group_size.unwrap_or(512 * 512))?;
 let mut batched = self.batched(chunked_df.schema())?;
 batched.write_batch(&chunked_df)?;
 batched.finish()
 }

chunk_df_for_writing splits the frame into row groups of row_group_size rows (default 512 × 512 = 262,144), and write_batch encodes and compresses them eagerly (crates/polars-io/src/parquet/write/batched_writer.rs:251):

 rust
 
 
/// This serializer encodes and compresses all eagerly in memory.
/// Used for separating compute from IO.
fn create_eager_serializer(

Note “eagerly in memory” — the old path materialises the entire encoded row group before writing it. The streaming path is the same code with the eagerness removed: the row group is encoded in a spawned task, and the handle is what travels down the channel.

The bottom line

The design of the Parquet sink comes down to four decisions:

Row group boundaries are decided by a separate stage. The streaming engine’s morsels are not row groups. A resize pipeline re-cuts them, using PartSizesIter to distribute the remainder exactly, and using calc_n_parts to pick a split count that lands close to the target.
The mode matters. Exact when the user asked for a specific row_group_size, Approximate otherwise. In approximate mode the writer will even wait for more rows rather than flush a short row group.
Encoding is parallel, writing is serial. Column encoding and compression run on the compute executor, in parallel across columns and across row groups. Exactly one task owns the file and writes the pages in order.
The memory bound is a semaphore, not a guess. Every in-flight sink morsel holds a permit, released only after the data is written.

The knobs, if you want to experiment:

POLARS_IDEAL_SINK_MORSEL_SIZE_ROWS — the default row group size when row_group_size is not set (default 122,880).
POLARS_IDEAL_SINK_MORSEL_SIZE_BYTES — a byte-based limit on the row group, with a 16,384-row floor.
POLARS_INFLIGHT_SINK_MORSEL_LIMIT — how many sink morsels may be in memory at once (default num_pipelines + 4).
POLARS_MAX_OPEN_SINKS — how many files a partitioned sink may hold open at once.

Polars on Opensource internals

A Tour of Polars' Internals: The Twelve Ideas That Explain the Engine

1. The DSL, the IR, and the fact that there are two of each

2. Two engines, and a switch between them

3. Chunks are the unit of parallelism

4. View: a string that fits in a register

5. Immutable, refcounted buffers

6. The optimizer is a rewrite engine

7. Predicate pushdown ends at the file footer

8. Group keys have four different encodings

9. Sortedness is metadata, and it changes the algorithm

10. The streaming engine is a pipeline of morsels

11. The runtime filter: a join that tells the scan what to skip

12. Python is a guest, not the host

The bottom line

Polars Spills to Disk Before It Runs Out of Memory

One counter for the whole process

The budget

Making frames spillable

Who decides

Picking a victim

The on-disk format

Reading back

Wiring it into the streaming engine

Aborting cleanly

The bottom line

Polars Publishes a Join's Build Side as a Runtime Filter

The plan-time side: finding a scan to talk to

Which joins qualify

Which side becomes the build

Tracing the key to a scan

Attaching the slot to the scan

The run-time side: what the join publishes

When the filter is published

The bloom filter

The consumer: the scan

One honest limitation

The bottom line

Polars Runs Queries on Rayon's Global Thread Pool

One marker type for every parallel entry point

The pools Polars does own

Never block a rayon thread

The async runtime

How the in-memory engine uses it

Errors across task boundaries

Send, Sync, and the one unsafe escape

The bottom line

Polars Shares Data With Python Over the Arrow C Data Interface

The three structs

Exporting: no bytes move

The one thing that is copied: alignment

Importing: borrowing, and keeping the owner alive

The stream interface

The Python side

The pyarrow bridge: to_arrow

The older protocol: polars-ffi

The bottom line

Polars Releases the GIL Before It Does Any Real Work

One helper, used everywhere

A #[pymethods] function that detaches

Where the GIL has to come back

Releasing the GIL breaks Ctrl-C, so Polars reimplements it

Crossing the boundary with data

Callbacks that arrive from another thread

The bottom line

Polars Estimates Cardinality With HyperLogLog

The register trick

The estimator, and what “HLL++” means here

The smaller sketch the streaming engine uses

Where the estimates are used

approx_n_unique

Deciding whether to dictionary-encode a Parquet column

Choosing the build side of a hash join

Publishing a bloom filter

The F2 sketch: counting heavy keys

The bottom line

Polars Finds the Top K With a Bounded Heap

The in-memory answer: partition, don’t sort

The streaming answer: a heap of exactly k keys

The bounded part

4. `View`: a string that fits in a register

`Send`, `Sync`, and the one unsafe escape

The pyarrow bridge: `to_arrow`

The older protocol: `polars-ffi`

A `#[pymethods]` function that detaches

`approx_n_unique`

The other merge: `merge_sorted`