A Tour of Polars' Internals: The Twelve Ideas That Explain the Engine

TL;DR: Polars is a stack of small ideas, and almost every performance question you have about Polars can be answered by naming one of them. Chunks are the unit of parallelism. View is the unit of string storage. The arena is the unit of expression allocation. The optimizer is a rewrite engine, not a cost model. The streaming engine is a pipeline of morsels. This is a tour of twelve of them, and where in the source tree each one lives.

pl.LazyFrame(...).explain() prints a plan, and the plan looks like a tree of node names — SLICE, SELECT, SORT, AGGREGATE. It reads like the output of a query planner in any other engine. It is not. Polars’ plan is closer to a compilation unit: a tree that gets rewritten several times before a single byte of data moves, and then executed by one of two quite different runtimes.

The best way to learn the engine is to ignore the node names and ask, for each one, what is the trick underneath it. Here is that list.

1. The DSL, the IR, and the fact that there are two of each

The user-facing plan is DslPlan, a tree of Arc<DslPlan> nodes with Expr leaves — user-level expressions, cheap to clone, awkward to analyze.

rust
// crates/polars-plan/src/dsl/plan.rs:32
pub enum DslPlan {
    #[cfg(feature = "python")]
    PythonScan {
        options: crate::dsl::python_dsl::PythonOptionsDsl,
    },
    /// Filter on a boolean mask
    Filter {
        input: Arc<DslPlan>,
        predicate: Expr,
    },

crates/polars-plan/src/dsl/plan.rs:32

The optimizer never touches that tree. It works on IR, where every node lives in an arena and every expression is an ExprIR — a resolved expression with a dtype, no user-facing sugar left in it.

rust
// crates/polars-plan/src/plans/ir/mod.rs:66
    Filter {
        input: Node,
        predicate: ExprIR,
    },

crates/polars-plan/src/plans/ir/mod.rs:66

The split matters for a reason that is easy to miss: the DSL tree can be rebuilt cheaply and structurally, while the IR is what the executors see. All the interesting analysis — predicate pushdown, projection pushdown, simplification, common-subplan elimination — happens on the IR, where the engine can ask questions like “is this expression already known to be non-null?” and get a real answer instead of a pattern match on syntax.

The arena itself is a Vec<T> plus integer handles, which is about as cheap as it gets:

rust
// crates/polars-utils/src/arena.rs:38
pub struct Arena<T> {
    version: u32,
    items: Vec<T>,
}

crates/polars-utils/src/arena.rs:38

More on this in Polars Evaluates Expressions in an Arena.

2. Two engines, and a switch between them

collect() does not pick an engine by measuring anything. It picks one by looking at flags:

rust
// crates/polars-lazy/src/frame/mod.rs:636
    pub fn collect_with_engine(mut self, engine: Engine) -> PolarsResult<QueryResult> {
        let engine = match engine {
            Engine::Streaming => Engine::Streaming,
            _ if std::env::var("POLARS_FORCE_STREAMING").as_deref() == Ok("1") => Engine::Streaming,
            Engine::Auto => {
                if self.opt_state.eager() {
                    Engine::InMemory
                } else {
                    Engine::Streaming
                }
            },

crates/polars-lazy/src/frame/mod.rs:636

Both engines consume the same optimized IR. The in-memory engine (polars-mem-engine) builds a tree of Box<dyn Executor> nodes and runs it depth-first with rayon fan-out; the streaming engine (polars-stream) lowers the same IR into a graph of nodes connected by pipes and runs it as a dataflow pipeline.

This is the single most interesting architectural fact in the codebase: the IR is the contract, and two very different runtimes honour it. A filter is a closure in one and a pipe in the other.

3. Chunks are the unit of parallelism

A ChunkedArray is a vector of Arrow arrays plus a length. Every kernel walks it chunk by chunk, which means the chunk boundaries are simultaneously the unit of work, the unit of parallelism, and the granularity at which validity and offsets are tracked.

rust
// crates/polars-core/src/chunked_array/mod.rs:140
pub struct ChunkedArray<T: PolarsDataType> {
    pub(crate) field: Arc<Field>,
    pub(crate) chunks: Vec<ArrayRef>,

    pub(crate) flags: StatisticsFlagsIM,

    length: usize,
    null_count: usize,
    _pd: std::marker::PhantomData<T>,
}

crates/polars-core/src/chunked_array/mod.rs:140

Appending to a ChunkedArray is free — you push another ArrayRef into the Vec. Random access costs an extra indirection, which is why the engine prefers chunk-local work and only pays for a global index when it has to.

Full treatment: Polars Chunks Its Columns So Parallelism Has Something to Eat.

4. `View`: a string that fits in a register

The most copied type in the engine is not a String. It is the 16-byte View:

rust
// crates/polars-arrow/src/array/binview/view.rs:20
#[derive(Copy, Clone, Default)]
#[repr(C)]
pub struct View {
    /// The length of the string/bytes.
    pub length: u32,
    /// First 4 bytes of string/bytes data.
    pub prefix: u32,
    /// The buffer index.
    pub buffer_idx: u32,
    /// The offset into the buffer.
    pub offset: u32,
}

crates/polars-arrow/src/array/binview/view.rs:20

A string of twelve bytes or fewer is stored inside the view. Anything longer spills into a shared buffer and the view keeps an index and an offset. Because the view is 16 bytes, comparing two of them is two integer compares before any string data is touched — the length rejects mismatches for free, and the four-byte prefix rejects most of the rest.

Full treatment: Polars Stores Short Strings Inline in 16 Bytes: The View Type and Polars Compares Strings Without Allocating.

5. Immutable, refcounted buffers

There is no Vec<u8> underneath a column. There is a SharedStorage<T> with an AtomicU64 refcount and a hand-written vtable for the drop glue, and the only mutation path is copy-on-write through a “make this uniquely owned” step. That is what makes zero-copy sharing between a DataFrame, a Python object and an Arrow C stream cheap: they all point at the same bytes.

Full treatment: Polars Buffers Are Immutable: The Refcounted Storage Behind Every Column and Polars Shares Columns Instead of Copying Them.

6. The optimizer is a rewrite engine

Polars’ optimizer is a list of rewrite rules applied to fixed points. There is no cost model, no cardinality estimation driving plan choices, no “which join order is cheaper” search. A rule that fires changes the tree; the rules run again; if nothing changes, the plan is done.

That design has an obvious consequence — Polars will not reorder a join for you — and a less obvious one: because the rules are local and structural, they can be extremely aggressive about shape. A select that is really just a projection becomes a SimpleProjection. A duplicated subplan becomes a Cache node. A filter that is provably always-true disappears.

Full treatment: Polars Simplifies the Expression Tree Before It Runs Anything and Polars Kills Duplicate Subplans by Fingerprinting Them.

7. Predicate pushdown ends at the file footer

Pushdown in Polars is not one feature but three layers: the optimizer moves the filter into the scan node, the scan converts the predicate into per-row-group statistics checks, and the reader skips row groups whose min/max cannot satisfy it.

The conversion is the interesting part, because the filter has to be compiled twice: once against the real columns, and once against a synthetic frame of col_min/col_max columns so the same expression code can be reused to test statistics.

Full treatment: Polars Pushes Predicates Into the Parquet Scanner and Polars Prunes Parquet Row Groups With Chunk Statistics.

8. Group keys have four different encodings

There is no single “group by” implementation. A single integer key groups by comparing integers. A single string key groups by dictionary codes. Multiple keys are encoded into rows — fixed-width byte blobs with null bits — so that equality becomes a memcmp and the hash is the hash of the row bytes.

Four encodings, four code paths, chosen by the shape of the key. The trick that makes the whole family fast is that the encoding is also the hashing input: there is no separate hash function per key type, there is one hash over the encoded bytes.

Full treatment: Polars Encodes Group Keys as Rows to Make Hashing Cheap and Polars Groups Columns So One Pass Feeds Many Aggregations.

9. Sortedness is metadata, and it changes the algorithm

Polars tracks whether a column is sorted as a flag on the chunked array, and the optimizer propagates that through the plan. A group-by on a sorted key does not hash: it walks runs. A merge join only accepts sorted inputs. A top-k on a sorted stream can stop early.

This is the cheapest possible form of query planning — a boolean that unlocks a completely different algorithm — and it is why sort().group_by() can be dramatically faster than group_by().

Full treatment: Polars Aggregates Faster When the Keys Are Already Sorted and Polars Joins Two Tables with a Piecewise Merge Join.

10. The streaming engine is a pipeline of morsels

The streaming engine never materialises a whole partition. A source node emits morsels — fixed-size blocks of rows — and downstream nodes consume them. The pipe between two nodes is a bounded channel, so backpressure is structural: a slow consumer stops the producer.

rust
// crates/polars-stream/src/morsel.rs:83
pub struct Morsel {
    /// The data contained in this morsel.
    sf: SpillFrame,

    /// The sequence number of this morsel. May only stay equal or increase
    /// within a pipeline.
    seq: MorselSeq,

crates/polars-stream/src/morsel.rs:83

When a stage has to repartition — a hash join, a group-by — the rows are shuffled: hashed into N partitions, buffered, and exchanged. The hash is a multiply-shift, and the partitioning is a counting sort, not a sort.

Full treatment: Polars’ New Streaming Engine Works in Morsels and Polars Hash-Shuffles Rows Between Streaming Stages.

11. The runtime filter: a join that tells the scan what to skip

The newest trick in the tree is the runtime filter. When a hash join builds its table on one side, the values of the build keys are known — and can be published as a predicate that the probe-side scan applies. A scan that would have read a whole partition can now skip row groups whose statistics cannot match the build key range, or check a bloom filter of the build keys before decoding anything.

It is a join and a scan cooperating through a slot in the plan that is filled at runtime.

Full treatment: Polars Publishes a Join’s Build Side as a Runtime Filter.

12. Python is a guest, not the host

The Python bindings do not own the data. pl.DataFrame wraps a Rust DataFrame; handing it to pandas or Arrow exports it through the Arrow C data interface, which is a set of pointers plus a release callback. The GIL is dropped for the duration of a query and re-taken only where Python code actually runs — inside a UDF.

Full treatment: Polars Shares Data With Python Over the Arrow C Data Interface and Polars Releases the GIL Before It Does Any Real Work.

The bottom line

If you want the mental model in one paragraph: Polars represents data as immutable, chunked, refcounted Arrow buffers with 16-byte inline strings; it compiles a user program into an arena-allocated IR through a stack of structural rewrite rules; and it executes that IR with one of two runtimes, one of which streams morsels through a graph of pipes with hash shuffles at the boundaries.

Almost every performance knob in Polars follows from those three sentences. A columnar layout is what makes the predicate-pushdown and statistics pruning possible. An arena is what makes the optimizer cheap enough to run aggressively. Morsels are what make the streaming engine’s memory usage independent of the input size.

The articles in this series take each of those ideas apart. If you only read one, read Polars’ New Streaming Engine Works in Morsels — it is the newest part of the engine and the least documented.