A Tour of DataFusion's Physical Operators: The Ten That Matter and the Trick Each One Uses

TL;DR: EXPLAIN a DataFusion query and you will see a dozen node names. Most of them are one of ten operators, and each of those ten is built around a single idea. This is a tour of that list: what the operator does, the trick it is built on, and where its own article in this series lives.

EXPLAIN output is a good way to learn an engine, and a bad way to understand it. The node names are opaque and the tree looks like it was assembled by a committee.

It was not. Every node in there is one of a small set of operators, and each of those has a specific idea at its core. Here is the tour.

1. DataSourceExec — the scan

The leaf. It knows how to turn a FileScanConfig into a stream of batches, and it is where the file-format-specific work happens: listing the object store, reading the Parquet footer, evaluating pruning predicates, decoding pages.

The trick: decide what to read before reading it. The footer is read first, and it contains enough statistics to skip entire row groups and entire files. The newer push decoder goes further: it interleaves byte-range fetches with decoding, so a filter that tightens at runtime can still prune the row groups that have not been fetched yet.

2. FilterExec — the filter

Evaluates a boolean predicate against each batch and emits the rows that pass.

The trick: the filter is pushable. It offers its predicate to the child, and the child may take it — entirely, partially, or not at all. The Parquet scan takes it partially: it evaluates the predicate against row-group statistics to skip data, and keeps a residual filter to guarantee correctness. The two-phase negotiation is gather_filters_for_pushdown / handle_child_pushdown_result.

3. ProjectionExec — the projection

Selects and computes output columns.

The trick: it is foldable. try_swapping_with_projection lets a projection be absorbed into the operator below, so the scan reads only the columns that survive — and the projection node disappears from the plan entirely. There is even an embedded variant, where the projection becomes a Vec<usize> inside the join, so the join never materialises the dropped columns at all.

4. RepartitionExec — the shuffle

Splits the input into N partitions, either round-robin (no coordination) or by hash (both sides agree without talking).

The trick: slicing, not copying. A batch is routed by computing row indices per partition, doing one take per column, and handing out RecordBatch::slice views. Round-robin skips the take entirely. The StrengthReducedU64 trick replaces a % with a mask or a reciprocal-multiply, because division is expensive and the modulus is constant.

5. HashJoinExec — the equi-join

The workhorse. Builds a hash table over the smaller side, probes it with the larger.

The trick: the build side is reused three times. It is the hash table, it is the payload, and it is the dynamic filter published to the probe side — so the probe side can skip rows that cannot match, using a membership test against the same table. There is also a perfect-hash variant: when the build key is a dense integer, the hash table is replaced by a flat array indexed by the key.

6. SortExec — the sort

The trick: two modes. Below a threshold, everything is concatenated and sorted in one go — no merge at all. Above it, the input is sorted into runs and merged, with the runs coalesced first when the sort key is a single column, because then the merge’s per-stream overhead dominates its per-comparison cost.

7. SortPreservingMergeExec — the merge

Merges N sorted streams into one sorted stream.

The trick: it does not sort. It keeps a cursor per stream and always takes the smallest head. When the streams are sorted ranges (which the range-partitioning machinery produces), the merge is a simple concatenation. And when there are too many runs to merge at once, the multi-level merge spills intermediate results and repeats.

8. AggregateExec — the aggregation

Groups and aggregates.

The trick: the accumulator is a state machine, split into update_batch / state / merge_batch / evaluate, which is what makes the partial/final two-phase split possible. And the group table itself can be skipped entirely when the input is already sorted by the grouping key — the group clustering optimisation replaces the hash table with Arrow’s partition kernel.

9. WindowAggExec — the window function

The trick: the frame is a state machine over a sorted stream. A ROWS frame is stateless (you know the offset); a RANGE frame needs state, because “the current tie group” has to be tracked. And the operator declares that it needs its input sorted, so the optimizer inserts a sort if there is not one.

10. CoalescePartitionsExec — the fan-in

Merges N partition streams into one.

The trick: it is not a computation. It is a promise — “the plan below has N partitions, but the caller wants one stream”. Which is what makes execute_stream on a multi-partition plan possible without pushing fan-in into user code.

The ones you will also see

A few more, for completeness:

• GlobalLimitExec / LocalLimitExec — the limit, split so that the global one can be a barrier.
• UnionExec — the union of N plans.
• UnnestExec — expanding a list column into rows.
• ScalarSubqueryExec — evaluating a subquery once and broadcasting the result.
• CooperativeExec — the budget that keeps a stream from hogging a worker.
• AnalyzeExec — the EXPLAIN ANALYZE wrapper.
• BufferExec — decoupling a fast producer from a slow consumer.

How they compose

The interesting part is not the operators individually but how they negotiate. A HashJoinExec says “I need hash-partitioned input on the join key”. A RepartitionExec says “I can give you that”. The optimizer inserts the second between the first and whatever is below it, and if the input already satisfies the requirement, it inserts nothing.

The same negotiation happens for ordering (a SortExec asks for sorted input and is deleted if the input is already sorted), for limits (a SortExec can absorb a LIMIT and stop early), and for filters (a FilterExec pushes its predicate down and keeps a residual).

Which is why the plan you get is not the plan you asked for, and why EXPLAIN is worth reading.

Why it matters

DataFusion’s operator set is small — a few dozen types — and each one is a well-known idea implemented with unusual care. The parts that surprise you are all the same shape:

• a statistical fact (min/max, hash, count) used to avoid work rather than to do work;
• an asymmetry exploited (the build side is also the filter; the probe side is also the producer of the sort);
• a boundary where the data is already in the right shape (a sorted input, a dense key range, a prefix group).

Once you learn to look for those three things, DataFusion’s plans stop looking like a committee’s compromise and start looking like a series of deliberate decisions.