Opensource internals

Polars Spills to Disk Before It Runs Out of Memory

18 min read Jay Hubbert

Polars counts every allocation in the process with a global allocator, and when that number crosses a budget, a background manager writes registered frames to IPC files on disk.

TL;DR: Polars does not measure the memory a particular operator uses. It counts every allocation the process makes with a thin wrapper around the global allocator, and compares that number against a budget (80% of RAM by default). Nodes in the streaming engine register their output frames as “spillable” with a spill context. When the global counter crosses the budget, a central memory manager picks candidates, encodes them as Arrow IPC streams, and writes them to files under a per-process spill directory. A second allocator-level threshold, off by default, aborts the process before the Linux OOM killer ever sees it.

The brief I was given for this article pointed at crates/polars-ooc/src/spill.rs, write.rs, reader.rs, and an OocS3 spill manager. Those files do not exist at this commit (b413f2b653). What exists is crates/polars-ooc/src/ with global_alloc.rs, memory_manager.rs, spill_context/, spill_file.rs, spill_frame.rs and spill_token.rs. There is no S3 backend: the only spill target is the local filesystem, and the “reader” and “writer” are two functions inside spill_frame.rs. So this is what is really there.

One counter for the whole process

The first thing to understand is that Polars’ out-of-core machinery is built on a global memory estimate, not on per-operator accounting. There is a GlobalAlloc implementation in crates/polars-ooc/src/global_alloc.rs that wraps jemalloc (or mimalloc) and keeps a running byte total:

rust 
// crates/polars-ooc/src/global_alloc.rs:5
static GLOBAL_ALLOC_SIZE: AtomicU64 = AtomicU64::new(0);

/// Returns an estimate of the total amount of bytes allocated.
///
/// This can be up to OOC_DRIFT_THRESHOLD * num_threads bytes less than or
/// greater than the true memory usage.
pub fn estimate_memory_usage() -> u64 {
    let bytes = GLOBAL_ALLOC_SIZE.load(Ordering::Relaxed);
    if bytes > i64::MAX as u64 {
        // Drift + moving allocations between threads allows for underflow,
        // so this is best reported as zero.
        0
    } else {
        bytes
    }
}

crates/polars-ooc/src/global_alloc.rs:5-20

The counter is not bumped on every alloc directly. Each thread accumulates into a thread-local drift counter, and only flushes to the global atomic when the drift exceeds POLARS_OOC_DRIFT_THRESHOLD (4 MiB by default):

rust 
// crates/polars-ooc/src/global_alloc.rs:28
#[inline(always)]
fn update_alloc_size(bytes: i64) {
    LOCAL_ALLOC_DRIFT.with(|drift| {
        let new = drift.get().wrapping_add(bytes);
        drift.set(new);
        if new.unsigned_abs() > polars_config::get_ooc_drift_threshold() {
            flush_local_drift();
        }
    })
}

#[cold]
#[inline(never)]
fn flush_local_drift() {
    LOCAL_ALLOC_DRIFT.with(|drift| {
        let drift_bytes = drift.get();
        let old = GLOBAL_ALLOC_SIZE.fetch_add(drift_bytes as u64, Ordering::AcqRel);
        drift.set(0);

        let total = old.wrapping_add(drift_bytes as u64);
        if total <= i64::MAX as u64 && total > polars_config::get_oomkill_threshold_bytes() {
            oomkill();
        }
    })
}

crates/polars-ooc/src/global_alloc.rs:28-52

This is the same batching trick every high-performance allocator-backed counter uses: one atomic fetch_add per 4 MiB per thread instead of one per malloc. The cost is that the estimate can be off by up to DRIFT_THRESHOLD * num_threads in either direction, which the doc comment states plainly.

The Allocator itself is four lines of arithmetic wrapped around the underlying allocator:

rust 
// crates/polars-ooc/src/global_alloc.rs:81
pub struct Allocator;

unsafe impl GlobalAlloc for Allocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        update_alloc_size(layout.size() as i64);
        unsafe { UNDERLYING_ALLOC.alloc(layout) }
    }

    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
        update_alloc_size(layout.size() as i64);
        unsafe { UNDERLYING_ALLOC.alloc_zeroed(layout) }
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        update_alloc_size(-(layout.size() as i64));
        unsafe { UNDERLYING_ALLOC.dealloc(ptr, layout) }
    }

    unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
        update_alloc_size(new_size as i64 - layout.size() as i64);
        unsafe { UNDERLYING_ALLOC.realloc(ptr, layout, new_size) }
    }
}

crates/polars-ooc/src/global_alloc.rs:81-103

In the Python build this is installed as the process-wide allocator:

rust 
// crates/polars-python/src/c_api/allocator.rs:7
#[global_allocator]
static ALLOC: polars_ooc::Allocator = polars_ooc::Allocator;

crates/polars-python/src/c_api/allocator.rs:7-8

That last detail has real consequences. Because the allocator is process-global, the estimate includes everything the Python interpreter allocates — your list of strings, the dataframe you already materialized, the interpreter’s own bookkeeping. So the “Polars memory usage” number that drives spilling is really “this process’s memory usage”. Spilling can be triggered by allocations Polars never made. That is a deliberate trade: a single counter is cheap and always available, whereas per-operator accounting would need every allocation site to be instrumented.

The budget

The budget is derived from total system memory, with a cgroup-aware helper (sys.cgroup_limits()) so containers get the right number:

rust 
// crates/polars-config/src/lib.rs:384
let budget_frac = f64::from_bits(self.ooc_memory_budget_fraction.load(Ordering::Relaxed));
let bytes = u64::min(
    self.ooc_memory_budget_bytes.load(Ordering::Relaxed),
    (total_memory() as f64 * budget_frac) as u64,
);
self.ooc_effective_memory_budget_bytes
    .store(bytes, Ordering::Relaxed);
let frac = f64::from_bits(self.ooc_memory_prefetch_fraction.load(Ordering::Relaxed));
self.ooc_memory_prefetch_bytes
    .store((bytes as f64 * frac) as u64, Ordering::Relaxed);

crates/polars-config/src/lib.rs:384-393

The default fraction is 0.8 and the default absolute cap is u64::MAX (POLARS_OOC_MEMORY_BUDGET_MB), so the effective budget is 80% of RAM unless you set the absolute value. The prefetch threshold is 90% of the budget, not of RAM — the gap between the two is the hysteresis band the engine uses to decide between spilling more and reading some back.

There is a separate, much stricter threshold for the hard abort:

rust 
// crates/polars-config/src/lib.rs:395
let oomkill_frac = f64::from_bits(self.oomkill_threshold_fraction.load(Ordering::Relaxed));
let oomkill_frac_bytes = if oomkill_frac.is_finite() {
    (total_memory() as f64 * oomkill_frac) as u64
} else {
    u64::MAX
};
let oomkill_mb_bytes = self.oomkill_threshold_bytes.load(Ordering::Relaxed);
OOMKILL_THRESHOLD_BYTES_ATOMIC.store(
    u64::min(oomkill_mb_bytes, oomkill_frac_bytes),
    Ordering::Relaxed,
);

crates/polars-config/src/lib.rs:395-404

Both POLARS_OOMKILL_THRESHOLD_FRACTION (default f64::INFINITY) and POLARS_OOMKILL_THRESHOLD_MB (default u64::MAX) are effectively unlimited out of the box, so the abort path is opt-in. When it does fire, it is deliberately brutal — it cannot allocate, so it writes straight to fd 2 and calls libc::abort():

rust 
// crates/polars-ooc/src/global_alloc.rs:54
#[cold]
#[inline(never)]
fn oomkill() -> ! {
    // Can't allocate here, so write directly to stderr.
    const MSG: &[u8] = b"polars: memory usage exceeded POLARS_OOMKILL_THRESHOLD_MB or POLARS_OOMKILL_THRESHOLD_FRACTION, aborting\n";
    unsafe {
        libc::write(2, MSG.as_ptr().cast(), MSG.len() as _);
        libc::abort()
    }
}

crates/polars-ooc/src/global_alloc.rs:54-63

abort() rather than a panic, because at that point a panic might need to allocate to format its message, and the whole point is that we are out of memory. Note also that the abort is checked only in flush_local_drift, so it fires at most once per 4 MiB of allocation churn — it is a sampling point, not an exact tripwire.

Making frames spillable

Now the other half. Nodes do not decide to spill; they make their output eligible for spilling and let a global manager decide.

The unit of spilling is a SpillToken<T>, a hand-rolled state machine over a single AtomicU64. The bits are:

rust 
// crates/polars-ooc/src/spill_token.rs:17
// SpillTokenInner's state
const SPILLED_BIT: u64 = 1; // Set when value = None and spilled is Some.
const DROPPED_BIT: u64 = 2; // Set when the token owner has dropped. Forbids creation of new pins / lock.
const LOCK_BIT: u64 = 4; // When set no new pins may be made, and at most 1 thread may set this bit.
const HAS_WAITERS_BIT: u64 = 8; // Only updated while holding waiters lock, allows checking for waiters without locking.
const REINSERT_WHEN_SPILLABLE_BIT: u64 = 16; // When set the SpillToken should insert into its registered context once spillable.
const RO_PIN_COUNT_UNIT: u64 = 32; // Added to the state for each active pin.
const RO_PIN_MASK: u64 = u64::MAX << 5;

crates/polars-ooc/src/spill_token.rs:17-24

A “pin” is a live reference to the in-memory value. Pin count lives in the upper bits of the same word as the flags, so acquiring a pin is one compare_exchange. Spilling requires upgrading a solo pin into the exclusive lock — the transition that guarantees nobody is reading the buffer while it is being encoded:

rust 
// crates/polars-ooc/src/spill_token.rs:715
// Try to upgrade solo pin into exclusive lock.
let is_exclusive = slf
    .state
    .try_update(Ordering::Acquire, Ordering::Acquire, |s| {
        if s & (RO_PIN_MASK | LOCK_BIT) == RO_PIN_COUNT_UNIT {
            Some(s - RO_PIN_COUNT_UNIT + LOCK_BIT)
        } else {
            None
        }
    })
    .is_ok();

crates/polars-ooc/src/spill_token.rs:715-725

If the upgrade fails, the spill is abandoned (add_failed_spill) and the frame stays in memory. That failure rate is not incidental: it is measured and fed back into the bandit that picks candidates.

Who decides

Nodes register their frames with a spill context. There are three context flavours — MostRecentSpillContext, LeastRecentSpillContext and RandomSpillContext — differing only in the order in which candidates are popped from the queue. A context is a name plus a policy plus a statistics object:

rust 
// crates/polars-ooc/src/spill_context/mod.rs:530
/// A context that spills the most-recently registered spillable when asked.
#[derive(Clone)]
#[repr(transparent)]
pub struct MostRecentSpillContext(StrongSpillContext);

impl MostRecentSpillContext {
    pub fn new(name: PlSmallStr, task_metrics: Option<Arc<TaskMetricAggregator>>) -> Self {
        Self(StrongSpillContext::new(
            name,
            SpillContextPolicy::MostRecent,
            task_metrics,
        ))
    }
}

crates/polars-ooc/src/spill_context/mod.rs:530-543

The register call is where the global memory check happens. Note that there are two variants: register_no_spill_check (synchronous, no global check) and register (async, checks):

rust 
// crates/polars-ooc/src/spill_context/mod.rs:512
pub trait ParameterFreeSpillContext {
    fn register_no_spill_check<T, S>(&self, token: &T)
    where
        T: AsRef<SpillToken<S>>,
        S: Spillable,
        Self: Sized;

    fn register<T, S>(&self, token: &T) -> impl Future<Output = ()>
    where
        T: AsRef<SpillToken<S>>,
        S: Spillable,
        Self: Sized,
    {
        self.register_no_spill_check(token);
        memory_manager().spill()
    }
}

crates/polars-ooc/src/spill_context/mod.rs:512-528

So registering a frame is the only trigger. The engine never asks “does this node exceed its share?”; it asks “are we over budget globally?” and if so, spills whatever is registered and eligible. The global memory manager is the only thing that reads the allocator counter:

rust 
// crates/polars-ooc/src/memory_manager.rs:61
fn should_spill(&self) -> bool {
    let usage = crate::estimate_memory_usage();
    let likely_dealt_with = self.est_spill_in_progress.load(Ordering::Relaxed);
    let likely_incoming = self.est_prefetch_in_progress.load(Ordering::Relaxed);
    (usage + likely_incoming).saturating_sub(likely_dealt_with)
        > config().ooc_memory_budget_bytes()
}

pub(crate) fn should_prefetch(&self) -> bool {
    if !self.spills_exist.load(Ordering::Acquire) {
        return false;
    }

    let usage = crate::estimate_memory_usage();
    let likely_incoming = self.est_prefetch_in_progress.load(Ordering::Relaxed);
    (usage + likely_incoming) < config().ooc_memory_prefetch_bytes()
}

crates/polars-ooc/src/memory_manager.rs:61-77

The subtraction of est_spill_in_progress matters: bytes that are already being written to disk are still allocated (the encoded buffer lives in RAM until the write completes), so counting them as “not yet freed” would make the manager spill harder the more it spills. The estimate of in-flight spill bytes is tracked by a SpillInProgressTracker whose Drop decrements the counter — a RAII guard, so it cannot leak even if the spill task is cancelled.

Picking a victim

Candidate selection is a multi-armed bandit over spill contexts, not a heuristic. Each context keeps discounted statistics of its own historical performance and is sampled via Thompson sampling:

rust 
// crates/polars-ooc/src/memory_manager.rs:289
// Thompson sampling.
let score_sample = ctx.0.stats().sample_spill_score(&mut rng);
assert!(!score_sample.is_nan());
live_contexts.push((ctx.clone(), score_sample));

crates/polars-ooc/src/memory_manager.rs:289-292

The score itself is a ratio of relief to IO time — byte-seconds freed per second spent doing I/O:

rust 
// crates/polars-ooc/src/spill_context/stats.rs:481
if r >= 0.0 && t >= 0.0 {
    return r / (BASE_IO_TIME + t);
}

crates/polars-ooc/src/spill_context/stats.rs:481-483

where r = bytes * seconds_spilled and t = spill_time + unspill_time. A context whose frames are cheap to write and expensive to read back gets a low score and is avoided. All statistics decay with a one-second half-life, because a plan that was a good spill candidate two minutes ago may not be now. The algorithm is documented as an adaptation of “Discounted Thompson Sampling for Non-Stationary Bandit Problems”.

Within the winning context, candidates are drained from the spill queue in policy order and filtered by size:

rust 
// crates/polars-ooc/src/memory_manager.rs:321
ctx.0.drain_live_while(|rt| {
    let Some(cand) = rt.upgrade() else {
        return true;
    };
    match cand.clone().spill_status() {
        SpillStatus::InMemory(sz) if sz >= min_spill => {
            candidates.push((cand, SpillInProgressTracker::new(sz), rt));
        },
        SpillStatus::InMemory(_) => {
            cand.cancel_spill_attempt_and_reinsert(
                rt.registration_id,
                ctx.1,
                InsertReason::TooSmall(rt.timestamp),
            );
        },
        SpillStatus::Pinned => {
            cand.cancel_spill_attempt_and_reinsert(
                rt.registration_id,
                ctx.1,
                InsertReason::Unpin,
            );
        },
        // A spilled token is re-inserted by whoever unspills it, a
        // dropped one does not need re-inserting at all.
        SpillStatus::Spilled | SpillStatus::Dropped => {},
    }

    num_considered += 1;
    num_considered < SPILL_FRAME_BATCH_SIZE
});

crates/polars-ooc/src/memory_manager.rs:321-350

Frames smaller than POLARS_OOC_SPILL_MIN_BYTES (64 KiB default) are not worth the round trip and go to the back of the queue; pinned frames (being read right now) are skipped and re-inserted when unpinned. The drain is capped at 256 candidates per attempt.

Each surviving candidate is spilled on its own task, bounded by a semaphore of POLARS_OOC_MAX_PARALLEL_SPILL_TASKS (64 by default):

rust 
// crates/polars-ooc/src/memory_manager.rs:137
for (spillable, spill_in_progress, rt) in spillables {
    let permit = self.spill_semaphore.clone().acquire_owned().await.unwrap();

    let successful_spill = successful_spill.clone();
    let ctx = ctx.clone();

    executor::spawn(TaskPriority::High, task_metrics.as_deref(), async move {
        // Spill, or reinsert if a failure.
        match spillable.clone().try_spill(ctx.clone()) {
            Ok(spill_success) => {
                if spill_success.await {
                    ...

crates/polars-ooc/src/memory_manager.rs:137-147

The on-disk format

Spilling a DataFrame means writing an Arrow IPC stream (not a file format with random access — a stream, so it reads back sequentially):

rust 
// crates/polars-ooc/src/spill_frame.rs:26
async fn spill(&self, context_id: &str) -> Self::Spilled {
    let mut df = self.clone();
    let context_id = context_id.to_owned();

    // Encode in the current task (on computational executor).
    let mut buf = Vec::new();

    let mut writer = IpcStreamWriter::new(&mut buf).with_compat_level(CompatLevel::newest());
    let clvl = polars_config::config().ooc_spill_compression_level();
    if clvl > 0 {
        let zstd_lvl = ZstdLevel::try_new(clvl.try_into().unwrap()).unwrap();
        writer = writer.with_compression(Some(IpcCompression::ZSTD(zstd_lvl)));
    }
    writer
        .finish(&mut df)
        .unwrap_or_else(|e| panic!("failed to encode spill file for '{context_id}': {e}",));

crates/polars-ooc/src/spill_frame.rs:26-41

Two details worth noting. First, compression is off by default: POLARS_OOC_SPILL_COMPRESSION_LEVEL defaults to 0, and the code only attaches ZSTD when the level is positive. Compression costs CPU and the spill file is usually read once, so the default trades disk for time. Second, the encode happens on the task that owns the data (so the buffers are already hot in cache) while the actual file write is pushed onto the tokio runtime:

rust 
// crates/polars-ooc/src/spill_frame.rs:43
// Do file creation / writing on tokio.
let file = ASYNC
    .spawn(async move {
        let size = buf.len() as u64;
        let spill_file = SpillFile::new(&context_id, "ipc", size);
        if BYTES_SPILLED_TO_DISK.fetch_add(size).saturating_add(size)
            > polars_config::config().ooc_disk_budget_bytes()
        {
            spill_file.creation_aborted();
            polars_error::abort::polars_abort_ooc_out_of_disk();
        }

crates/polars-ooc/src/spill_frame.rs:43-53

The disk budget (POLARS_OOC_DISK_BUDGET_MB, 64 GB by default) is checked against a process-wide counter of bytes spilled. Exceeding it aborts the query rather than filling the disk.

The file layout is deliberately simple, with one directory per process so that a crashed run’s leftovers can be identified and removed:

text 
<spill_dir>/
  <pid>/                     <- process directory (one per OS process)
    spill-<ctx>-<uuid>.ipc   <- individual spill file (unique per spill)

crates/polars-ooc/src/spill_file.rs:9-15

The default spill directory is /var/tmp/polars-$USER/spill on Linux — deliberately not std::env::temp_dir(), because on Linux that is often tmpfs, and spilling to tmpfs defeats the entire purpose of spilling:

rust 
// crates/polars-config/src/spill_path.rs:16
pub fn default_ooc_spill_dir() -> PathBuf {
    #[cfg(target_os = "linux")]
    let (base, user_var) = (PathBuf::from("/var/tmp"), "USER");
    ...
    let user = std::env::var(user_var).unwrap_or_else(|_| "unknown".to_string());
    base.join(format!("polars-{user}")).join("spill")
}

crates/polars-config/src/spill_path.rs:16-31

The spill file is deleted when the SpillFile is dropped, but deletion is handed off to a background thread so the hot path never blocks on unlink:

rust 
// crates/polars-ooc/src/spill_file.rs:55
impl Drop for SpillFile {
    fn drop(&mut self) {
        SPILL_CLEANER
            .send_rq
            .send(CleanRequest::File(
                core::mem::take(&mut self.path),
                self.size,
            ))
            .unwrap();
    }
}

crates/polars-ooc/src/spill_file.rs:55-65

The same cleaner thread, started lazily, also sweeps directories left by dead processes at startup — it reads the PID from each subdirectory name and checks whether that PID is still alive:

rust 
// crates/polars-ooc/src/spill_file.rs:127
fn cleanup_stale_dirs() {
    let spill_dir = polars_config::config().ooc_spill_dir();
    let Ok(entries) = std::fs::read_dir(&spill_dir) else {
        return;
    };

    let our_pid = std::process::id();
    let dead_pid_dirs = entries.flatten().filter_map(|e| {
        let pid = e.file_name().to_str()?.parse::<u32>().ok()?;
        (pid != our_pid && !polars_utils::sys::is_process_alive(pid)).then(|| e.path())
    });
    for path in dead_pid_dirs {
        let _ = std::fs::remove_dir_all(&path);
    }
}

crates/polars-ooc/src/spill_file.rs:127-141

Reading back

Unspilling is the mirror image: open the file and decode it, on a blocking thread since it is filesystem I/O:

rust 
// crates/polars-ooc/src/spill_frame.rs:71
async fn unspill(location: &Self::Spilled) -> Self {
    let path = location.path().to_owned();
    ASYNC
        .spawn_blocking(move || {
            let file = std::fs::File::open(&path).unwrap_or_else(|e| {
                panic!("failed to open spill file {:?}: {e}", path.display())
            });
            IpcStreamReader::new(file).finish().unwrap_or_else(|e| {
                panic!("failed to read spill file {:?}: {e}", path.display())
            })
        })
        .await
        .unwrap()
}

crates/polars-ooc/src/spill_frame.rs:71-84

But nobody waits for a spill to be read back if they can avoid it. When memory usage drops below the prefetch threshold (90% of budget), the manager proactively pulls spilled frames back into memory on low-priority tasks, so the eventual read is a cache hit:

rust 
// crates/polars-ooc/src/spill_context/mod.rs:362
for _ in 0..self.stats.suggested_prefetch_amount() {
    let Some(permit) = mm.try_get_prefetch_permit() else {
        return PrefetchScheduleResult::NoPermitsLeft;
    };

    let pop = match self.policy() {
        SpillContextPolicy::MostRecent => shared.unspill_queue.pop_front(),
        SpillContextPolicy::LeastRecent => shared.unspill_queue.pop_back(),
        SpillContextPolicy::Random => shared.unspill_queue.pop_random(&mut rng),
    };

crates/polars-ooc/src/spill_context/mod.rs:362-371

Prefetching is scheduled by a separate bandit (sample_prefetch_score), and the amount of prefetching scales with how long it has been since the last spill: suggested_prefetch_amount returns 1 + prefetch_without_spill_streak. If nothing is being spilled, the engine reads more aggressively ahead of demand.

Wiring it into the streaming engine

Everything above is machinery. The nodes are what make it useful.

The unit of work in the streaming engine is a Morsel, and every morsel carries a SpillFrame:

rust 
// crates/polars-stream/src/morsel.rs:82
#[derive(Debug)]
pub struct Morsel {
    /// The data contained in this morsel.
    sf: SpillFrame,

    /// The sequence number of this morsel. May only stay equal or increase
    /// within a pipeline.
    seq: MorselSeq,
    ...
}

crates/polars-stream/src/morsel.rs:82-96

A node that buffers its input registers each incoming morsel with its context, so the buffered data becomes spillable while it waits:

rust 
// crates/polars-stream/src/nodes/in_memory_sink.rs:66
for mut recv in receivers {
    let slf = &*self;
    join_handles.push(scope.spawn_task(TaskPriority::High, async move {
        let mut morsels = Vec::new();
        while let Ok(mut morsel) = recv.recv().await {
            morsel.take_consume_token();
            slf.spill_ctx.register(morsel.sf()).await;
            morsels.push(morsel);
        }

        slf.morsels_per_pipe.lock().push(morsels);
        Ok(())
    }));
}

crates/polars-stream/src/nodes/in_memory_sink.rs:66-79

The sort node does the same for its buffered input, and additionally uses the memory budget to decide when to stop sorting in memory and start partitioning:

rust 
// crates/polars-stream/src/nodes/sort/tuning.rs:61
/// The buffered size above which the input is partitioned instead of
/// sorted in memory.
pub fn partition_threshold(&self) -> u64 {
    self.partition_threshold_bytes.min(budget() / 2)
}

/// The number of buckets that may be sorted or waiting to be emitted at
/// once while flushing: two per pipeline, as long as they take at most a
/// quarter of the memory budget.
pub fn flush_ahead(&self, bucket_bytes: u64, num_pipelines: usize) -> usize {
    let fit = budget() / 4 / bucket_bytes.max(1);
    usize::try_from(fit)
        .unwrap_or(usize::MAX)
        .clamp(1, 2 * num_pipelines)
}

crates/polars-stream/src/nodes/sort/tuning.rs:61-75

That is the hybrid strategy in one place: the sort node does not spill blindly, it partitions its input when the buffered bytes exceed half the budget, and the partition buckets are individually registered as spillable. The hash shuffle does something similar, keeping per-partition frames registered so the shuffle can spill under pressure:

rust 
// crates/polars-stream/src/hash_shuffle/store.rs:64
let frame = SpillFrameWithKeys::new_unregistered(builder.freeze_reset());
self.spill_ctx.register_no_spill_check(&frame);
self.estimated_size += frame.estimated_size() as u64;
self.frames_per_input[input].push_back(frame);

crates/polars-stream/src/hash_shuffle/store.rs:64-67

Note register_no_spill_check here: the shuffle registers without asking the global manager to spill, because it is on the hot path and does not want to block on a global decision. It relies on other nodes’ register calls to drive the check, and on the explicit memory_manager().spill() calls sprinkled through the shuffle’s own build and load loops.

The join nodes follow the same pattern with a MostRecentSpillContext (the build side of an equi-join keeps its probe keys in spillable frames), and the group-by keeps cold rows in SpillFrameWithKeys so that the rows and their hashed keys always spill and unspill together — the keys are needed to re-partition the rows, so splitting them across a frame and a side file would be a bug waiting to happen.

Aborting cleanly

Running out of disk is a normal, expected outcome of a query that decided to spill, so it is turned into an ordinary Rust error rather than a crash. polars-error/src/abort.rs implements a panic-based abort protocol with a magic string in the panic payload:

rust 
// crates/polars-error/src/abort.rs:12
pub enum QueryAborted {
    KeyboardInterrupt,
    OocOutOfDisk,
}

// We use a unique string so we can detect it in backtraces.
static POLARS_ABORT_PREFIX: &str = "__POLARS_ABORT_";
static POLARS_ABORT_KEYBOARD_INTERRUPT_STR: &str = "__POLARS_ABORT_KEYBOARD_INTERRUPT";
static POLARS_ABORT_OOC_OUT_OF_DISK_STR: &str = "__POLARS_ABORT_OOC_OUT_OF_DISK";

crates/polars-error/src/abort.rs:12-20

The out-of-disk abort can be raised from anywhere — including from inside a tokio task with no PolarsResult return path — so it is a panic with a recognisable payload, caught by catch_polars_abort at the Python boundary:

rust 
// crates/polars-python/src/utils.rs:147
let ret = self.detach(|| catch_polars_abort(AssertUnwindSafe(f)));
match ret {
    Ok(Ok(ret)) => Ok(ret),
    Ok(Err(err)) => Err(PyErr::from(err.into())),
    Err(QueryAborted::KeyboardInterrupt) => Err(PyKeyboardInterrupt::new_err("")),
    Err(QueryAborted::OocOutOfDisk) => Err(ComputeError::new_err(
        "query aborted: spilled data exceeded the out-of-core disk budget, increase POLARS_OOC_DISK_BUDGET_MB",
    )),
}

crates/polars-python/src/utils.rs:147-155

The same mechanism doubles as the Ctrl-C path: Python hooks SIGINT, the Rust side sets a bit in an atomic, and every try_raise_polars_abort() in a hot loop turns that bit into the same kind of panic. The panic hook suppresses the usual panic message when a catcher is installed, so the user sees a clean KeyboardInterrupt rather than a Rust backtrace.

The bottom line

The design is coherent once you see the constraint it is built around: the engine must never let the process run out of memory, and it must never pay for spilling more than necessary.

  • Accounting is global and approximate. One counting allocator for the whole process, sampled every 4 MiB per thread. The estimate can be off by 4 MiB * threads in either direction, and it includes Python’s own allocations. In exchange it works everywhere with zero instrumentation of individual operators.
  • Nodes do not decide to spill. They register frames as candidates; a single global manager compares the counter against the budget (POLARS_OOC_MEMORY_BUDGET_FRACTION, default 80% of RAM) and picks victims using a discounted Thompson-sampling bandit over spill contexts, scoring each context by byte-seconds of relief per second of I/O.
  • The on-disk format is a plain Arrow IPC stream, one file per spill under /var/tmp/polars-$USER/spill/<pid>/, ZSTD only if you ask for it, deleted asynchronously and swept by PID liveness check after a crash.
  • Reading back is anticipatory. Below 90% of the budget the engine prefetches spilled frames onto low-priority tasks, so the read that eventually happens is a cache hit instead of a disk wait.
  • There is a hard abort before the OOM killer, but it is off by default (POLARS_OOMKILL_THRESHOLD_MB / _FRACTION are unlimited out of the box). When enabled it calls libc::abort() — no unwinding, no cleanup. Running out of disk, by contrast, is a clean, catchable error.

To use it, set the budget and let the engine work:

bash 
# Spill aggressively: 25% of RAM, tiny frames still worth spilling.
export POLARS_OOC_MEMORY_BUDGET_FRACTION=0.25
export POLARS_OOC_SPILL_MIN_BYTES=1048576
export POLARS_OOC_SPILL_DIR=/mnt/big-disk/polars-spill

# Compress spill files (costs CPU, saves disk).
export POLARS_OOC_SPILL_COMPRESSION_LEVEL=3

# Die with a clear message instead of being OOM-killed.
export POLARS_OOMKILL_THRESHOLD_FRACTION=0.9

The trade-offs are real. Spilling to disk costs a serialization round trip and a read-back later, and the bandit is explicitly trying to avoid contexts where that round trip is expensive. Frames under 64 KiB are never spilled because the overhead dominates. And because the counter is process-wide, a Python program holding several gigabytes of its own data can trigger spilling of Polars frames that are nowhere near their size limit. If you are memory-constrained, lowering the budget below 100% is usually the right move — you want Polars to start spilling while there is still headroom, not when the allocator’s threshold is already crossed.

PolarsInternalsPerformanceMemoryStreaming