Overview on Opensource internals

A Tour of DataFusion's Physical Operators: The Ten That Matter and the Trick Each One Uses

Jay Hubbert — Wed, 21 Oct 2026 09:17:00 +0000

TL;DR: EXPLAIN a DataFusion query and you will see a dozen node names. Most of them are one of ten operators, and each of those ten is built around a single idea. This is a tour of that list: what the operator does, the trick it is built on, and where its own article in this series lives.

EXPLAIN output is a good way to learn an engine, and a bad way to understand it. The node names are opaque and the tree looks like it was assembled by a committee.

It was not. Every node in there is one of a small set of operators, and each of those has a specific idea at its core. Here is the tour.

1. `DataSourceExec` — the scan

The leaf. It knows how to turn a FileScanConfig into a stream of batches, and it is where the file-format-specific work happens: listing the object store, reading the Parquet footer, evaluating pruning predicates, decoding pages.

The trick: decide what to read before reading it. The footer is read first, and it contains enough statistics to skip entire row groups and entire files. The newer push decoder goes further: it interleaves byte-range fetches with decoding, so a filter that tightens at runtime can still prune the row groups that have not been fetched yet.

2. `FilterExec` — the filter

Evaluates a boolean predicate against each batch and emits the rows that pass.

The trick: the filter is pushable. It offers its predicate to the child, and the child may take it — entirely, partially, or not at all. The Parquet scan takes it partially: it evaluates the predicate against row-group statistics to skip data, and keeps a residual filter to guarantee correctness. The two-phase negotiation is gather_filters_for_pushdown / handle_child_pushdown_result.

3. `ProjectionExec` — the projection

Selects and computes output columns.

The trick: it is foldable. try_swapping_with_projection lets a projection be absorbed into the operator below, so the scan reads only the columns that survive — and the projection node disappears from the plan entirely. There is even an embedded variant, where the projection becomes a Vec<usize> inside the join, so the join never materialises the dropped columns at all.

4. `RepartitionExec` — the shuffle

Splits the input into N partitions, either round-robin (no coordination) or by hash (both sides agree without talking).

The trick: slicing, not copying. A batch is routed by computing row indices per partition, doing one take per column, and handing out RecordBatch::slice views. Round-robin skips the take entirely. The StrengthReducedU64 trick replaces a % with a mask or a reciprocal-multiply, because division is expensive and the modulus is constant.

5. `HashJoinExec` — the equi-join

The workhorse. Builds a hash table over the smaller side, probes it with the larger.

The trick: the build side is reused three times. It is the hash table, it is the payload, and it is the dynamic filter published to the probe side — so the probe side can skip rows that cannot match, using a membership test against the same table. There is also a perfect-hash variant: when the build key is a dense integer, the hash table is replaced by a flat array indexed by the key.

6. `SortExec` — the sort

The trick: two modes. Below a threshold, everything is concatenated and sorted in one go — no merge at all. Above it, the input is sorted into runs and merged, with the runs coalesced first when the sort key is a single column, because then the merge’s per-stream overhead dominates its per-comparison cost.

7. `SortPreservingMergeExec` — the merge

Merges N sorted streams into one sorted stream.

The trick: it does not sort. It keeps a cursor per stream and always takes the smallest head. When the streams are sorted ranges (which the range-partitioning machinery produces), the merge is a simple concatenation. And when there are too many runs to merge at once, the multi-level merge spills intermediate results and repeats.

8. `AggregateExec` — the aggregation

Groups and aggregates.

The trick: the accumulator is a state machine, split into update_batch / state / merge_batch / evaluate, which is what makes the partial/final two-phase split possible. And the group table itself can be skipped entirely when the input is already sorted by the grouping key — the group clustering optimisation replaces the hash table with Arrow’s partition kernel.

9. `WindowAggExec` — the window function

The trick: the frame is a state machine over a sorted stream. A ROWS frame is stateless (you know the offset); a RANGE frame needs state, because “the current tie group” has to be tracked. And the operator declares that it needs its input sorted, so the optimizer inserts a sort if there is not one.

10. `CoalescePartitionsExec` — the fan-in

Merges N partition streams into one.

The trick: it is not a computation. It is a promise — “the plan below has N partitions, but the caller wants one stream”. Which is what makes execute_stream on a multi-partition plan possible without pushing fan-in into user code.

The ones you will also see

A few more, for completeness:

GlobalLimitExec / LocalLimitExec — the limit, split so that the global one can be a barrier.
UnionExec — the union of N plans.
UnnestExec — expanding a list column into rows.
ScalarSubqueryExec — evaluating a subquery once and broadcasting the result.
CooperativeExec — the budget that keeps a stream from hogging a worker.
AnalyzeExec — the EXPLAIN ANALYZE wrapper.
BufferExec — decoupling a fast producer from a slow consumer.

How they compose

The interesting part is not the operators individually but how they negotiate. A HashJoinExec says “I need hash-partitioned input on the join key”. A RepartitionExec says “I can give you that”. The optimizer inserts the second between the first and whatever is below it, and if the input already satisfies the requirement, it inserts nothing.

The same negotiation happens for ordering (a SortExec asks for sorted input and is deleted if the input is already sorted), for limits (a SortExec can absorb a LIMIT and stop early), and for filters (a FilterExec pushes its predicate down and keeps a residual).

Which is why the plan you get is not the plan you asked for, and why EXPLAIN is worth reading.

Why it matters

DataFusion’s operator set is small — a few dozen types — and each one is a well-known idea implemented with unusual care. The parts that surprise you are all the same shape:

a statistical fact (min/max, hash, count) used to avoid work rather than to do work;
an asymmetry exploited (the build side is also the filter; the probe side is also the producer of the sort);
a boundary where the data is already in the right shape (a sorted input, a dense key range, a prefix group).

Once you learn to look for those three things, DataFusion’s plans stop looking like a committee’s compromise and start looking like a series of deliberate decisions.

A Tour of Polars' Internals: The Twelve Ideas That Explain the Engine

Jay Hubbert — Wed, 21 Oct 2026 09:00:00 +0000

TL;DR: Polars is a stack of small ideas, and almost every performance question you have about Polars can be answered by naming one of them. Chunks are the unit of parallelism. View is the unit of string storage. The arena is the unit of expression allocation. The optimizer is a rewrite engine, not a cost model. The streaming engine is a pipeline of morsels. This is a tour of twelve of them, and where in the source tree each one lives.

pl.LazyFrame(...).explain() prints a plan, and the plan looks like a tree of node names — SLICE, SELECT, SORT, AGGREGATE. It reads like the output of a query planner in any other engine. It is not. Polars’ plan is closer to a compilation unit: a tree that gets rewritten several times before a single byte of data moves, and then executed by one of two quite different runtimes.

The best way to learn the engine is to ignore the node names and ask, for each one, what is the trick underneath it. Here is that list.

1. The DSL, the IR, and the fact that there are two of each

The user-facing plan is DslPlan, a tree of Arc<DslPlan> nodes with Expr leaves — user-level expressions, cheap to clone, awkward to analyze.

 rust
 
 
// crates/polars-plan/src/dsl/plan.rs:32
pub enum DslPlan {
 #[cfg(feature = "python")]
 PythonScan {
 options: crate::dsl::python_dsl::PythonOptionsDsl,
 },
 /// Filter on a boolean mask
 Filter {
 input: Arc<DslPlan>,
 predicate: Expr,
 },

crates/polars-plan/src/dsl/plan.rs:32

The optimizer never touches that tree. It works on IR, where every node lives in an arena and every expression is an ExprIR — a resolved expression with a dtype, no user-facing sugar left in it.

 rust
 
 
// crates/polars-plan/src/plans/ir/mod.rs:66
 Filter {
 input: Node,
 predicate: ExprIR,
 },

crates/polars-plan/src/plans/ir/mod.rs:66

The split matters for a reason that is easy to miss: the DSL tree can be rebuilt cheaply and structurally, while the IR is what the executors see. All the interesting analysis — predicate pushdown, projection pushdown, simplification, common-subplan elimination — happens on the IR, where the engine can ask questions like “is this expression already known to be non-null?” and get a real answer instead of a pattern match on syntax.

The arena itself is a Vec<T> plus integer handles, which is about as cheap as it gets:

 rust
 
 
// crates/polars-utils/src/arena.rs:38
pub struct Arena<T> {
 version: u32,
 items: Vec<T>,
}

crates/polars-utils/src/arena.rs:38

More on this in Polars Evaluates Expressions in an Arena.

2. Two engines, and a switch between them

collect() does not pick an engine by measuring anything. It picks one by looking at flags:

 rust
 
 
// crates/polars-lazy/src/frame/mod.rs:636
 pub fn collect_with_engine(mut self, engine: Engine) -> PolarsResult<QueryResult> {
 let engine = match engine {
 Engine::Streaming => Engine::Streaming,
 _ if std::env::var("POLARS_FORCE_STREAMING").as_deref() == Ok("1") => Engine::Streaming,
 Engine::Auto => {
 if self.opt_state.eager() {
 Engine::InMemory
 } else {
 Engine::Streaming
 }
 },

crates/polars-lazy/src/frame/mod.rs:636

Both engines consume the same optimized IR. The in-memory engine (polars-mem-engine) builds a tree of Box<dyn Executor> nodes and runs it depth-first with rayon fan-out; the streaming engine (polars-stream) lowers the same IR into a graph of nodes connected by pipes and runs it as a dataflow pipeline.

This is the single most interesting architectural fact in the codebase: the IR is the contract, and two very different runtimes honour it. A filter is a closure in one and a pipe in the other.

3. Chunks are the unit of parallelism

A ChunkedArray is a vector of Arrow arrays plus a length. Every kernel walks it chunk by chunk, which means the chunk boundaries are simultaneously the unit of work, the unit of parallelism, and the granularity at which validity and offsets are tracked.

 rust
 
 
// crates/polars-core/src/chunked_array/mod.rs:140
pub struct ChunkedArray<T: PolarsDataType> {
 pub(crate) field: Arc<Field>,
 pub(crate) chunks: Vec<ArrayRef>,

 pub(crate) flags: StatisticsFlagsIM,

 length: usize,
 null_count: usize,
 _pd: std::marker::PhantomData<T>,
}

crates/polars-core/src/chunked_array/mod.rs:140

Appending to a ChunkedArray is free — you push another ArrayRef into the Vec. Random access costs an extra indirection, which is why the engine prefers chunk-local work and only pays for a global index when it has to.

Full treatment: Polars Chunks Its Columns So Parallelism Has Something to Eat.

4. `View`: a string that fits in a register

The most copied type in the engine is not a String. It is the 16-byte View:

 rust
 
 
// crates/polars-arrow/src/array/binview/view.rs:20
#[derive(Copy, Clone, Default)]
#[repr(C)]
pub struct View {
 /// The length of the string/bytes.
 pub length: u32,
 /// First 4 bytes of string/bytes data.
 pub prefix: u32,
 /// The buffer index.
 pub buffer_idx: u32,
 /// The offset into the buffer.
 pub offset: u32,
}

crates/polars-arrow/src/array/binview/view.rs:20

A string of twelve bytes or fewer is stored inside the view. Anything longer spills into a shared buffer and the view keeps an index and an offset. Because the view is 16 bytes, comparing two of them is two integer compares before any string data is touched — the length rejects mismatches for free, and the four-byte prefix rejects most of the rest.

Full treatment: Polars Stores Short Strings Inline in 16 Bytes: The View Type and Polars Compares Strings Without Allocating.

5. Immutable, refcounted buffers

There is no Vec<u8> underneath a column. There is a SharedStorage<T> with an AtomicU64 refcount and a hand-written vtable for the drop glue, and the only mutation path is copy-on-write through a “make this uniquely owned” step. That is what makes zero-copy sharing between a DataFrame, a Python object and an Arrow C stream cheap: they all point at the same bytes.

Full treatment: Polars Buffers Are Immutable: The Refcounted Storage Behind Every Column and Polars Shares Columns Instead of Copying Them.

6. The optimizer is a rewrite engine

Polars’ optimizer is a list of rewrite rules applied to fixed points. There is no cost model, no cardinality estimation driving plan choices, no “which join order is cheaper” search. A rule that fires changes the tree; the rules run again; if nothing changes, the plan is done.

That design has an obvious consequence — Polars will not reorder a join for you — and a less obvious one: because the rules are local and structural, they can be extremely aggressive about shape. A select that is really just a projection becomes a SimpleProjection. A duplicated subplan becomes a Cache node. A filter that is provably always-true disappears.

Full treatment: Polars Simplifies the Expression Tree Before It Runs Anything and Polars Kills Duplicate Subplans by Fingerprinting Them.

7. Predicate pushdown ends at the file footer

Pushdown in Polars is not one feature but three layers: the optimizer moves the filter into the scan node, the scan converts the predicate into per-row-group statistics checks, and the reader skips row groups whose min/max cannot satisfy it.

The conversion is the interesting part, because the filter has to be compiled twice: once against the real columns, and once against a synthetic frame of col_min/col_max columns so the same expression code can be reused to test statistics.

Full treatment: Polars Pushes Predicates Into the Parquet Scanner and Polars Prunes Parquet Row Groups With Chunk Statistics.

8. Group keys have four different encodings

There is no single “group by” implementation. A single integer key groups by comparing integers. A single string key groups by dictionary codes. Multiple keys are encoded into rows — fixed-width byte blobs with null bits — so that equality becomes a memcmp and the hash is the hash of the row bytes.

Four encodings, four code paths, chosen by the shape of the key. The trick that makes the whole family fast is that the encoding is also the hashing input: there is no separate hash function per key type, there is one hash over the encoded bytes.

Full treatment: Polars Encodes Group Keys as Rows to Make Hashing Cheap and Polars Groups Columns So One Pass Feeds Many Aggregations.

9. Sortedness is metadata, and it changes the algorithm

Polars tracks whether a column is sorted as a flag on the chunked array, and the optimizer propagates that through the plan. A group-by on a sorted key does not hash: it walks runs. A merge join only accepts sorted inputs. A top-k on a sorted stream can stop early.

This is the cheapest possible form of query planning — a boolean that unlocks a completely different algorithm — and it is why sort().group_by() can be dramatically faster than group_by().

Full treatment: Polars Aggregates Faster When the Keys Are Already Sorted and Polars Joins Two Tables with a Piecewise Merge Join.

10. The streaming engine is a pipeline of morsels

The streaming engine never materialises a whole partition. A source node emits morsels — fixed-size blocks of rows — and downstream nodes consume them. The pipe between two nodes is a bounded channel, so backpressure is structural: a slow consumer stops the producer.

 rust
 
 
// crates/polars-stream/src/morsel.rs:83
pub struct Morsel {
 /// The data contained in this morsel.
 sf: SpillFrame,

 /// The sequence number of this morsel. May only stay equal or increase
 /// within a pipeline.
 seq: MorselSeq,

crates/polars-stream/src/morsel.rs:83

When a stage has to repartition — a hash join, a group-by — the rows are shuffled: hashed into N partitions, buffered, and exchanged. The hash is a multiply-shift, and the partitioning is a counting sort, not a sort.

Full treatment: Polars’ New Streaming Engine Works in Morsels and Polars Hash-Shuffles Rows Between Streaming Stages.

11. The runtime filter: a join that tells the scan what to skip

The newest trick in the tree is the runtime filter. When a hash join builds its table on one side, the values of the build keys are known — and can be published as a predicate that the probe-side scan applies. A scan that would have read a whole partition can now skip row groups whose statistics cannot match the build key range, or check a bloom filter of the build keys before decoding anything.

It is a join and a scan cooperating through a slot in the plan that is filled at runtime.

Full treatment: Polars Publishes a Join’s Build Side as a Runtime Filter.

12. Python is a guest, not the host

The Python bindings do not own the data. pl.DataFrame wraps a Rust DataFrame; handing it to pandas or Arrow exports it through the Arrow C data interface, which is a set of pointers plus a release callback. The GIL is dropped for the duration of a query and re-taken only where Python code actually runs — inside a UDF.

Full treatment: Polars Shares Data With Python Over the Arrow C Data Interface and Polars Releases the GIL Before It Does Any Real Work.

The bottom line

If you want the mental model in one paragraph: Polars represents data as immutable, chunked, refcounted Arrow buffers with 16-byte inline strings; it compiles a user program into an arena-allocated IR through a stack of structural rewrite rules; and it executes that IR with one of two runtimes, one of which streams morsels through a graph of pipes with hash shuffles at the boundaries.

Almost every performance knob in Polars follows from those three sentences. A columnar layout is what makes the predicate-pushdown and statistics pruning possible. An arena is what makes the optimizer cheap enough to run aggressively. Morsels are what make the streaming engine’s memory usage independent of the input size.

The articles in this series take each of those ideas apart. If you only read one, read Polars’ New Streaming Engine Works in Morsels — it is the newest part of the engine and the least documented.