Execution on Opensource internals

A Tour of DataFusion's Physical Operators: The Ten That Matter and the Trick Each One Uses

Jay Hubbert — Wed, 21 Oct 2026 09:17:00 +0000

TL;DR: EXPLAIN a DataFusion query and you will see a dozen node names. Most of them are one of ten operators, and each of those ten is built around a single idea. This is a tour of that list: what the operator does, the trick it is built on, and where its own article in this series lives.

EXPLAIN output is a good way to learn an engine, and a bad way to understand it. The node names are opaque and the tree looks like it was assembled by a committee.

It was not. Every node in there is one of a small set of operators, and each of those has a specific idea at its core. Here is the tour.

1. `DataSourceExec` — the scan

The leaf. It knows how to turn a FileScanConfig into a stream of batches, and it is where the file-format-specific work happens: listing the object store, reading the Parquet footer, evaluating pruning predicates, decoding pages.

The trick: decide what to read before reading it. The footer is read first, and it contains enough statistics to skip entire row groups and entire files. The newer push decoder goes further: it interleaves byte-range fetches with decoding, so a filter that tightens at runtime can still prune the row groups that have not been fetched yet.

2. `FilterExec` — the filter

Evaluates a boolean predicate against each batch and emits the rows that pass.

The trick: the filter is pushable. It offers its predicate to the child, and the child may take it — entirely, partially, or not at all. The Parquet scan takes it partially: it evaluates the predicate against row-group statistics to skip data, and keeps a residual filter to guarantee correctness. The two-phase negotiation is gather_filters_for_pushdown / handle_child_pushdown_result.

3. `ProjectionExec` — the projection

Selects and computes output columns.

The trick: it is foldable. try_swapping_with_projection lets a projection be absorbed into the operator below, so the scan reads only the columns that survive — and the projection node disappears from the plan entirely. There is even an embedded variant, where the projection becomes a Vec<usize> inside the join, so the join never materialises the dropped columns at all.

4. `RepartitionExec` — the shuffle

Splits the input into N partitions, either round-robin (no coordination) or by hash (both sides agree without talking).

The trick: slicing, not copying. A batch is routed by computing row indices per partition, doing one take per column, and handing out RecordBatch::slice views. Round-robin skips the take entirely. The StrengthReducedU64 trick replaces a % with a mask or a reciprocal-multiply, because division is expensive and the modulus is constant.

5. `HashJoinExec` — the equi-join

The workhorse. Builds a hash table over the smaller side, probes it with the larger.

The trick: the build side is reused three times. It is the hash table, it is the payload, and it is the dynamic filter published to the probe side — so the probe side can skip rows that cannot match, using a membership test against the same table. There is also a perfect-hash variant: when the build key is a dense integer, the hash table is replaced by a flat array indexed by the key.

6. `SortExec` — the sort

The trick: two modes. Below a threshold, everything is concatenated and sorted in one go — no merge at all. Above it, the input is sorted into runs and merged, with the runs coalesced first when the sort key is a single column, because then the merge’s per-stream overhead dominates its per-comparison cost.

7. `SortPreservingMergeExec` — the merge

Merges N sorted streams into one sorted stream.

The trick: it does not sort. It keeps a cursor per stream and always takes the smallest head. When the streams are sorted ranges (which the range-partitioning machinery produces), the merge is a simple concatenation. And when there are too many runs to merge at once, the multi-level merge spills intermediate results and repeats.

8. `AggregateExec` — the aggregation

Groups and aggregates.

The trick: the accumulator is a state machine, split into update_batch / state / merge_batch / evaluate, which is what makes the partial/final two-phase split possible. And the group table itself can be skipped entirely when the input is already sorted by the grouping key — the group clustering optimisation replaces the hash table with Arrow’s partition kernel.

9. `WindowAggExec` — the window function

The trick: the frame is a state machine over a sorted stream. A ROWS frame is stateless (you know the offset); a RANGE frame needs state, because “the current tie group” has to be tracked. And the operator declares that it needs its input sorted, so the optimizer inserts a sort if there is not one.

10. `CoalescePartitionsExec` — the fan-in

Merges N partition streams into one.

The trick: it is not a computation. It is a promise — “the plan below has N partitions, but the caller wants one stream”. Which is what makes execute_stream on a multi-partition plan possible without pushing fan-in into user code.

The ones you will also see

A few more, for completeness:

GlobalLimitExec / LocalLimitExec — the limit, split so that the global one can be a barrier.
UnionExec — the union of N plans.
UnnestExec — expanding a list column into rows.
ScalarSubqueryExec — evaluating a subquery once and broadcasting the result.
CooperativeExec — the budget that keeps a stream from hogging a worker.
AnalyzeExec — the EXPLAIN ANALYZE wrapper.
BufferExec — decoupling a fast producer from a slow consumer.

How they compose

The interesting part is not the operators individually but how they negotiate. A HashJoinExec says “I need hash-partitioned input on the join key”. A RepartitionExec says “I can give you that”. The optimizer inserts the second between the first and whatever is below it, and if the input already satisfies the requirement, it inserts nothing.

The same negotiation happens for ordering (a SortExec asks for sorted input and is deleted if the input is already sorted), for limits (a SortExec can absorb a LIMIT and stop early), and for filters (a FilterExec pushes its predicate down and keeps a residual).

Which is why the plan you get is not the plan you asked for, and why EXPLAIN is worth reading.

Why it matters

DataFusion’s operator set is small — a few dozen types — and each one is a well-known idea implemented with unusual care. The parts that surprise you are all the same shape:

a statistical fact (min/max, hash, count) used to avoid work rather than to do work;
an asymmetry exploited (the build side is also the filter; the probe side is also the producer of the sort);
a boundary where the data is already in the right shape (a sorted input, a dense key range, a prefix group).

Once you learn to look for those three things, DataFusion’s plans stop looking like a committee’s compromise and start looking like a series of deliberate decisions.

One take, N slices: how DataFusion's RepartitionExec splits a batch across shuffle partitions

Jay Hubbert — Wed, 21 Oct 2026 07:01:00 +0000

TL;DR: BatchPartitioner never copies a batch once per partition. It computes one row-index vector per partition, concatenates them, performs a single take_arrays over the whole batch, and then hands out RecordBatch::slice(start, len) views over that one reordered buffer. Round-robin partitioning skips the take entirely — the whole batch goes to one output partition, whole.

The problem: a shuffle is a scatter-gather in disguise

Every parallel plan in DataFusion eventually has to move rows between worker threads. RepartitionExec is that operator: it maps N input partitions onto M output partitions according to a Partitioning scheme (datafusion/physical-expr/src/partitioning.rs:122-132):

 rust
 
 
pub enum Partitioning {
 /// Allocate batches using a round-robin algorithm and the specified number of partitions
 RoundRobinBatch(usize),
 /// Allocate rows based on a hash of one of more expressions and the specified number of partitions
 Hash(Vec<Arc<dyn PhysicalExpr>>, usize),
 /// Partition rows by source-declared ranges
 Range(RangePartitioning),
 /// Unknown partitioning scheme with a known number of partitions
 UnknownPartitioning(usize),
}

The naive implementation of “split this 8192-row batch into 16 partitions” is: for each partition, scan the batch and copy the matching rows into a fresh builder. That is 16 passes over the data and 16 output allocations. It is also almost never the right shape, because the expensive part of a shuffle is copying bytes, not deciding where rows go.

DataFusion’s answer splits the problem in two:

Routing (decide which row goes where) is done with plain integer indices — no data movement at all.
Materialization (move the bytes) is done once for the whole batch, and the per-partition results are slices of that one materialized buffer.

The router: `BatchPartitioner`

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:649-673
pub struct BatchPartitioner {
 state: BatchPartitionerState,
 timer: metrics::Time,
}

enum BatchPartitionerState {
 Hash {
 exprs: Vec<Arc<dyn PhysicalExpr>>,
 partition_reducer: StrengthReducedU64,
 hash_buffer: Vec<u64>,
 indices: Vec<Vec<u32>>,
 },
 RoundRobin {
 num_partitions: usize,
 next_idx: usize,
 },
 Range { /* ... */ }
}

Note the shape of the Hash state: there is no per-partition RecordBatch builder, and no per-partition data buffer. There is one hash_buffer: Vec<u64> (one hash per row) and one indices: Vec<Vec<u32>> (one list of row numbers per partition). Everything else is derived.

Round robin: the batch is the unit, not the row

Partitioning::RoundRobinBatch is a bit of a misnomer if you come from Spark: it does not interleave rows. The whole batch goes to one output partition, and the next batch goes to the next partition:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1240-1247
BatchPartitionerState::RoundRobin {
 num_partitions,
 next_idx,
} => {
 let idx = *next_idx;
 *next_idx = (*next_idx + 1) % *num_partitions;
 Box::new(std::iter::once(Ok((idx, batch))))
}

No hashing, no take, no copy — the batch is handed downstream unchanged. The only subtlety is where you start, and DataFusion solves it neatly so that N inputs feeding M outputs don’t all pile onto partition 0:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1055-1068
pub fn new_round_robin_partitioner(
 num_partitions: usize,
 timer: metrics::Time,
 input_partition: usize,
 num_input_partitions: usize,
) -> Self {
 Self {
 state: BatchPartitionerState::RoundRobin {
 num_partitions,
 next_idx: (input_partition * num_partitions) / num_input_partitions,
 },
 timer,
 }
}

With 2 inputs and 4 outputs, input 0 starts at partition 0 and input 1 starts at partition 2. Combined with the fact that each input alternates, the batches land evenly.

Hash: route with integers first

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1257-1279
let arrays = evaluate_expressions_to_arrays(exprs.as_slice(), &batch)?;

hash_buffer.clear();
hash_buffer.resize(batch.num_rows(), 0);

create_hashes(
 &arrays,
 REPARTITION_RANDOM_STATE.random_state(),
 hash_buffer,
)?;

for v in indices.iter_mut() {
 v.clear();
}

partition_reducer.partition_indices(hash_buffer, indices);

// Finished building index-arrays for output partitions
timer.done();

let partitioned_batches =
 Self::partition_grouped_take(&batch, indices, &self.timer)?;

Two details worth pausing on.

The hash seed is fixed. REPARTITION_RANDOM_STATE is a SeededRandomState::with_seed(0) (datafusion/physical-plan/src/repartition/mod.rs:675-677). A random seed per query would make the same key land in different partitions on different runs, breaking co-partitioning assumptions between the two sides of a join and making plans non-reproducible. Fixing the seed makes hash partitioning a pure function of the key, which is exactly what the planner’s “both children must be hash-partitioned on the same keys” rule relies on.

The modulo is strength-reduced. Computing hash % num_partitions for every row means an integer division per row, and integer division is expensive. DataFusion precomputes the compiler’s classic constant-divisor optimization once, at partitioner construction time:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:939-986
enum StrengthReducedU64 {
 PowerOfTwo { mask: u64 },
 Reciprocal { divisor: u64, reciprocal: u128 },
}

fn partition_indices(self, hash_buffer: &[u64], indices: &mut [Vec<u32>]) {
 match self {
 Self::PowerOfTwo { mask } => {
 for (index, hash) in hash_buffer.iter().enumerate() {
 indices[(*hash & mask) as usize].push(index as u32);
 }
 }
 Self::Reciprocal { divisor, reciprocal } => {
 for (index, hash) in hash_buffer.iter().enumerate() {
 let quotient = Self::quotient(*hash, reciprocal);
 let partition = *hash - quotient * divisor;
 indices[partition as usize].push(index as u32);
 }
 }
 }
}

Powers of two become a mask; everything else becomes a 128-bit reciprocal multiply (ceil(2^128 / divisor)) with the remainder recovered by subtraction. Either way the inner loop is a multiply plus a subtract — no div instruction at all.

Multi-column keys: hash-then-combine, not hash-of-a-tuple

A multi-column key (PARTITION BY a, b or a join on (a, b)) is hashed one column at a time, with the second and subsequent columns rehashing the running value rather than computing independent hashes and merging them:

 rust
 
 
// datafusion/common/src/hash_utils.rs:1264-1279
pub fn create_hashes<'a, I, T>(
 arrays: I,
 random_state: &impl HashState,
 hashes_buffer: &'a mut [u64],
) -> Result<&'a mut [u64]> {
 for (i, array) in arrays.into_iter().enumerate() {
 // combine hashes with `combine_hashes` for all columns besides the first
 let rehash = i >= 1;
 hash_single_array(array.as_dyn_array(), random_state, hashes_buffer, rehash)?;
 }
 Ok(hashes_buffer)
}

For primitive arrays the “rehash” path is especially cheap, because it seeds the hasher with the previous hash rather than hashing the value and then combining:

 rust
 
 
// datafusion/common/src/hash_utils.rs:322-333
if array.null_count() == 0 {
 if rehash {
 for (hash, &value) in hashes_buffer.iter_mut().zip(array.values().iter()) {
 let mut hasher = random_state.seeded_state(*hash).build_hasher();
 value.hash_write(&mut hasher);
 *hash = hasher.finish();
 }
 } else { /* ... */ }
}

The alternative — combine_hashes(l, r) = (17*37 + l) * 37 + r (hash_utils.rs:100-103) — is used for the generic (non-primitive) path. Both are order-sensitive, which is what you want: (a=1,b=2) and (a=2,b=1) must not collide.

The payoff: one take, then slices

Here is the whole trick, in one function:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1328-1391
/// Build repartitioned hash/range output batches using one `take` per input batch.
///
/// The routers first fills one index vector per output partition. This method
/// concatenates those index vectors, performs one grouped `take_arrays`, and
/// then returns each output partition as a slice of the reordered batch.
///
/// For example, given partition indices:
///
/// ```text
/// partition 0: [2, 5]
/// partition 1: []
/// partition 2: [0, 3, 4]
/// ```
///
/// this method takes rows in `[2, 5, 0, 3, 4]` order once, then returns
/// `partition 0 = slice(0, 2)` and `partition 2 = slice(2, 3)`.
fn partition_grouped_take(
 batch: &RecordBatch,
 indices: &mut [Vec<u32>],
 timer: &metrics::Time,
) -> Result<Vec<Result<(usize, RecordBatch)>>> {
 let mut partition_ranges = Vec::with_capacity(indices.len());
 let mut reordered_indices = Vec::with_capacity(batch.num_rows());

 for (partition, p_indices) in indices.iter_mut().enumerate() {
 if p_indices.is_empty() {
 continue;
 }
 let start = reordered_indices.len();
 reordered_indices.extend_from_slice(p_indices);
 partition_ranges.push((partition, start, p_indices.len()));
 p_indices.clear();
 }

 if reordered_indices.is_empty() {
 return Ok(vec![]);
 }

 if partition_ranges.len() == 1 && partition_ranges[0].2 == batch.num_rows() {
 let (partition, _, _) = partition_ranges[0];
 return Ok(vec![Ok((partition, batch.clone()))]);
 }

 let batches = {
 let _timer = timer.timer();
 let indices_array: PrimitiveArray<UInt32Type> = reordered_indices.into();
 let columns = take_arrays(batch.columns(), &indices_array, None)?;

 let mut options = RecordBatchOptions::new();
 options = options.with_row_count(Some(indices_array.len()));
 let reordered_batch =
 RecordBatch::try_new_with_options(batch.schema(), columns, &options)?;

 partition_ranges
 .into_iter()
 .map(|(partition, start, len)| {
 Ok((partition, reordered_batch.slice(start, len)))
 })
 .collect()
 };

 Ok(batches)
}

Read it as three distinct optimizations:

The index vectors are concatenated, not applied one at a time. Partition 0’s rows, then partition 1’s rows, then partition 2’s rows, all in one UInt32Array. That array is exactly the permutation we want.
One take_arrays for the entire batch, not one per partition. take_arrays is a gather: it produces new arrays whose contents are the selected rows in the requested order. Doing it once over 8192 rows × M partitions’ worth of indices costs the same per row as doing it M times over smaller index sets, but it allocates one output array per column instead of M, and it keeps the hot loop tight.
The per-partition batches are slices, not copies. RecordBatch::slice(start, len) produces a new RecordBatch that points at the same Arrow buffers with an offset and a length. No bytes are copied to make partition 2. This is why the function is called “grouped_take” and not “grouped_copy”.
text
```
original batch ─┐
 │ indices per partition: p0=[2,5] p1=[] p2=[0,3,4]
reordered order ─┴─► take() once ─► [2,5,0,3,4]
 ├── slice(0,2) ──► partition 0
 └── slice(2,3) ──► partition 2
 (same buffers, different offset)
```

There’s also a fast path: if all rows in the batch land in a single partition, the batch is returned as-is with no gather at all (mod.rs:1367-1370). That’s the degenerate case of a completely skewed key.

Why the slices don’t leak memory

A slice keeps the whole reordered batch alive, not just its slice. If partition 0 gets 2 rows and partition 2 gets 3 rows, both slices keep the 5-row buffer alive — which is fine, because together they are the whole buffer, and the buffer is dropped once all its slices are. The pathological case is one big partition plus many tiny ones: the tiny partitions keep a large buffer alive. This is bounded by the batch size, not by the dataset, so it’s a constant-factor concern rather than a leak.

Downstream: coalescing the shreds

A repartitioned batch is tiny — batch_size / M rows on average. Feeding 8192-row shards downstream would wreck every downstream operator’s throughput assumptions. So RepartitionExec coalesces on the producer side, before the channel, using one shared LimitedBatchCoalescer per output partition:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1461-1480
/// For this, one shared [`LimitedBatchCoalescer`] per output partition is used:
///
/// ```text
/// ┌───┐ ┌───┐
/// ┌─▶│ │────────▶.───────────. │ │ ┌──────────────────┐
/// │ └───┘ ┌───┐ ( Coalescer 0 )──▶ ├───┤ ───▶│ Output 0 │
/// │┌──────▶│ │──▶`───────────' │ │ └──────────────────┘
/// ││ └───┘ └───┘
/// ┌──────────────────┐ ││ ┌──────────────────┐
/// │BatchPartitioner 0│─┘│ │ Output 1 │
/// └──────────────────┘ │ └──────────────────┘
/// ```

The coalescer is shared across all input tasks targeting the same output partition (SharedCoalescer, mod.rs:276-325), with an AtomicUsize counting how many senders are still active so that the last one to finish flushes the residual batch. Coalescing is skipped for unbounded inputs, where withholding a partial batch would stall the stream forever (mod.rs:480).

The rest of the machinery, briefly

The routing above is only the CPU half. RepartitionExec also has to move the resulting batches between threads, and it does that with a deliberately unusual channel design:

 text
 
 
// datafusion/physical-plan/src/repartition/distributor_channels.rs:23-40
/// There are `N` virtual MPSC (multi-producer, single consumer) channels with
/// unbounded capacity. However, if all buffers/channels are non-empty, than a
/// global gate will be closed preventing new data from being written (the
/// sender futures will be [pending](Poll::Pending)) until at least one channel
/// is empty (and not closed).

A single global “gate” counter (empty_channels: AtomicUsize) implements global backpressure across all M output channels with one atomic increment/decrement per send — rather than a semaphore with M permits. The producer loop then yields to the runtime every num_partitions batches, because an endless input stream would otherwise never give the executor a chance to run (mod.rs:2347-2368).

Under memory pressure the whole path degrades gracefully: OutputChannel::send tries reservation.try_grow(size) and, on failure, spills the batch to a per-(input, output) SpillPool and sends a RepartitionBatch::Spilled marker instead of the data (mod.rs:232-252). The reader side runs a small state machine (StreamState::ReadingMemory / ReadingSpilled, mod.rs:2473-2480) that blocks on the spill stream until the spilled batch arrives, so FIFO order survives the memory/spill boundary.

Trade-offs and failure modes

The take is still a full copy. The slicing is free, but the single take_arrays copies every byte of every column into a new buffer. A shuffle therefore costs one full copy of the data, no matter how many partitions there are. This is the irreducible cost of moving data between threads; the optimization is that you pay it once rather than M times, and that the routing decision costs zero bytes.

Skew is the real enemy. Hash partitioning distributes keys, not rows. If 90% of rows share one key, 90% of the batch lands in one partition. The fast path at mod.rs:1367 only helps when the skew is total. With partial skew you get one huge output partition and M-1 nearly empty ones, and the downstream aggregate on that partition becomes the critical path. The planner’s defense is to hash on the exact key tuple rather than a subset (see the “strictest distribution match” note in datafusion/physical-optimizer/src/ensure_requirements/mod.rs:140-147), because adding more key columns spreads the hot key across more partitions. But no amount of planner cleverness fixes a genuinely skewed key: if country = 'US' is 90% of your data, hashing on (country, city) is the only fix, and only if the query semantics allow it.

Round robin trades locality for balance. RoundRobinBatch is perfectly balanced and needs no key evaluation, but it destroys key locality — every downstream group-by or join has to re-shuffle. That is exactly why the planner only inserts a round-robin when the operator has declared Distribution::UnspecifiedDistribution (i.e. it doesn’t care) but does benefit from more parallelism.

Range partitioning is the third option. Partitioning::Range routes rows by sorted split points rather than by hash (repartition/range.rs), so each partition owns a contiguous key range. It gives perfect balance and preserves sort order, at the cost of needing sampled split points and requiring the input to be sorted by the range key.

Bottom line

The shuffle is the single most performance-sensitive piece of DataFusion’s runtime, and its design is a good illustration of a general principle: decide with integers, move bytes once, and hand out views. The row-index vectors cost 4 bytes per row and are thrown away immediately; the one take_arrays produces a single reordered buffer; the per-partition RecordBatchs are zero-copy windows into it. The result is a shuffle whose cost is O(bytes) and whose routing cost is O(rows), with no hidden O(partitions × bytes) term.

How DataFusion Stops Reading After Ten Rows: Limit Pushdown, Limit Absorption, and the Volcano Model's Batch Problem

Jay Hubbert — Wed, 21 Oct 2026 05:25:00 +0000

TL;DR: SELECT * FROM t LIMIT 10 is the classic example of a query that looks cheap and is not. In a batched volcano engine, every operator produces batches of 8,192 rows, so a limit of 10 makes the engine read 8,192 rows, build a hash table of 8,192 rows, and throw away 8,182 of them. DataFusion’s answer has two halves: limit pushdown (move the limit down the plan so the scan itself stops early) and limit absorption (let an operator take the limit as a parameter and stop producing output once it is satisfied).

The problem is structural. DataFusion, like every vectorised query engine, uses a batched volcano model: operators pull RecordBatches of up to datafusion.execution.batch_size rows (8,192 by default) and push them up. The batching is what makes the engine fast — it is what lets Arrow kernels do their work with SIMD and tight loops — but it has a sharp edge at the tail of a query.

The physical optimizer’s LimitPushdown rule describes it exactly:

Background: vectorized volcano execution model

DataFusion uses a batched volcano model. For most operators, output is produced in batches of datafusion.execution.batch_size (default 8192), so the batch sizes typically look like:
text
8192, 8192, ..., 8192, 100 (the final batch may be partial)
Example

For a join with an expensive, selective predicate:
text
GlobalLimitExec: skip=0, fetch=10
-- NestedLoopJoinExec(on=expr_expensive_and_selective)
--- DataSourceExec()
--- DataSourceExec()
Under this model, NestedLoopJoinExec would keep working until it can emit a full batch (8192 rows), even though the query only needs 10.

That is 819 times more work than the query asked for, in the operator that is usually the most expensive one in the plan.

Two mechanisms

The rule’s documentation names them:

Plan Limit Absorption

In addition to pushing down GlobalLimitExec and LocalLimitExec nodes in the plan, some operators can “absorb” a limit and stop early during execution.

Pushdown removes the limit node from the plan and hands the fetch value to an operator that can honour it. Absorption is the operator using that value to stop producing output.

The interface is small:

 rust
 
 
// datafusion/physical-plan/src/execution_plan.rs
fn supports_limit_pushdown(&self) -> bool {
 false
}

and

 rust
 
 
// datafusion/datasource/src/source.rs
/// Return a copy of this DataSource with a new fetch limit
fn with_fetch(&self, _limit: Option<usize>) -> Option<Arc<dyn DataSource>>;

with_fetch returns None if the operator cannot honour the limit, and the optimizer then leaves a LimitExec in place instead. Note that the default with_fetch returns None, so an operator opts in to limit pushdown.

Pushing into the scan

The most valuable target is the file scan, because a scan that knows it only needs 10 rows can stop after 10 rows — no more pages fetched, no more row groups decoded.

The Parquet implementation threads the limit all the way down to the decoder:

 rust
 
 
// datafusion/datasource-parquet/src/push_decoder.rs
pub(crate) struct DecoderBuilderConfig<'a> {
 ...
 pub(crate) decoder_limit: Option<usize>,
 ...
}

if let Some(limit) = self.decoder_limit {
 builder = builder.with_limit(limit);
}

But before the decoder even starts, the row group plan is pruned:

 rust
 
 
// Prune by limit if limit is set and limit order is not sensitive
if let (Some(limit), false) = (prepared.limit, prepared.preserve_order) {
 row_groups.prune_by_limit(limit, rg_metadata, &prepared.file_metrics);
}

and prune_by_limit is a careful little function:

 rust
 
 
pub fn prune_by_limit(
 &mut self,
 limit: usize,
 rg_metadata: &[RowGroupMetaData],
 metrics: &ParquetFileMetrics,
) {
 let mut fully_matched_row_group_indexes: Vec<usize> = Vec::new();
 let mut fully_matched_rows_count: usize = 0;

 // Iterate through the currently accessible row groups and try to
 // find a set of matching row groups that can satisfy the limit
 for &idx in self.access_plan.row_group_indexes().iter() {
 if self.access_plan.is_fully_matched(idx) {
 let row_group_row_count = match &self.access_plan.inner()[idx] {
 RowGroupAccess::Skip => continue,
 RowGroupAccess::Scan => rg_metadata[idx].num_rows() as usize,
 RowGroupAccess::Selection(selection) => selection.row_count(),
 };
 fully_matched_row_group_indexes.push(idx);
 fully_matched_rows_count += row_group_row_count;
 if fully_matched_rows_count >= limit {
 break;
 }
 }
 }

 // If we can satisfy the limit with fully matching row groups,
 // rewrite the plan to do so
 if fully_matched_rows_count >= limit {
 ...
 metrics.limit_pruned_row_groups.add_pruned(pruned_count);
 ...
 }
}

Two things to notice.

First, it only counts fully matched row groups — row groups where the predicate was statically proven to select every row. A row group with a partial selection has an unknown output size, so it cannot be used to guarantee that the limit is satisfied; but it can still be scanned.

Second, the if fully_matched_rows_count >= limit guard. If the fully matched row groups cannot satisfy the limit on their own, the function does nothing — the plan is left alone, because pruning the rest of the row groups would be wrong (the limit might be satisfied by the partially matched ones). This is a case where the conservative behaviour is also the correct one.

The `preserve_order` condition

Note the false in if let (Some(limit), false) = (prepared.limit, prepared.preserve_order).

preserve_order is set when the file must be read in its natural order — for example, when the query is SELECT * FROM t ORDER BY partition_col LIMIT 10 and the ordering of the files matters. In that case you cannot skip row groups, because the first rows in file order are the ones the query wants, and those are in the first row groups — but the pruning logic here counts row groups by accumulated row count, not by position, so it would be wrong.

The comment in the source makes the intent explicit:

rust

// `select * from t order by partition_col limit 10`.

The limit and the row filter

The other half of the Parquet story is that the limit also affects how the per-row filter is installed. The RgPlanEntry struct carries a fully_matched flag precisely so the decoder knows it can skip the per-row RowFilter for that row group:

 rust
 
 
#[derive(Debug, Clone)]
pub(crate) struct RgPlanEntry {
 pub(crate) rg_index: usize,
 /// `true` when static pruning proved every row of this RG satisfies the
 /// predicate, so the per-row `RowFilter` can be skipped as a no-op.
 pub(crate) fully_matched: bool,
 /// On-disk size of this row group, credited to the `bytes_processed` metric
 /// documented on `ParquetFileMetrics` once the scan is done with it.
 pub(crate) bytes: u64,
}

So a limit can turn “evaluate a filter on every row” into “evaluate a filter on no rows”, for the row groups that are entirely selected.

The optimizer rule

The physical rule that drives all of this is datafusion/physical-optimizer/src/limit_pushdown.rs (487 lines). Its state machine is a small struct tracking the global skip/fetch as it descends the plan:

 rust
 
 
if pushdown_plan.fetch().is_some() {
 if global_state.skip == 0 {
 global_state.satisfied = true;
 }
 (global_state.skip, global_state.fetch) = combine_limit(
 global_state.skip,
 global_state.fetch,
 0,
 pushdown_plan.fetch(),
 );
}

combine_limit is where two limits are merged: the tighter one wins, and skips add up. It is the standard “limit algebra” — LIMIT 10 OFFSET 5 is LIMIT 15 OFFSET 0, and two nested limits compose to the tighter one.

The rule also handles the case where the limit is already satisfied by what it found:

 rust
 
 
if let Some(fetch) = fetch
 && limit_satisfied_by_input(&limit_info.input, skip, fetch, stats_ctx)?
{
 // The input already produces at most `fetch` rows, so no new limit
 // node is needed. Mark satisfied so downstream won't re-add one,
 // but preserve skip/fetch so any nested limit nodes (e.g. an inner
 // GlobalLimitExec) can still be merged with the outer constraint.
 global_state.satisfied = true;
 ...

This uses statistics — if the input’s estimated row count is below the limit, the limit node is not needed at all and can be deleted from the plan. The plan gets smaller and the limit disappears entirely.

The catch

A limit is not always pushable. UnionExec advertises supports_limit_pushdown() == true but combines partitions, so the rule has to be careful:

 rust
 
 
// Checking `supports_limit_pushdown()` alone is not enough. For example:
// UnionExec advertises `supports_limit_pushdown() == true` because it can

Skip is harder than fetch. The rule notes: “Execution plans can’t (yet) handle skip, so if we have one, we still need to add a global limit”. A LIMIT 10 OFFSET 100000 cannot be pushed into a scan, because the scan would have to count and discard 100,000 rows — which it could do, but the machinery is not there.
The limit changes the meaning of the plan for the operators above it. A SortExec with a limit produces the top-k rows, not the first-k rows of a full sort. DataFusion handles this with a separate TopKExec and by pushing the limit into the sort as a “fetch” parameter, but the distinction between “limit” and “top-k” is a real semantic difference that the plan has to preserve.
Pushing a limit into a scan changes the metrics. A scan that stopped after 10 rows did not process the whole file, and the bytes_processed metric has to reflect that. The pruned_row_groups metric exists for exactly this.
Limits interact with the dynamic filters. A TopK that tightens its threshold and a pushed-down LIMIT are two different ways of saying “we do not need much more data”, and both can be active at once.

The general point

The batched volcano model is a performance decision: batching is what lets Arrow’s kernels shine. But it creates a tail-latency problem for queries that need few rows. The fix is not to abandon batching — it is to teach every operator about the limit, so that the last operator in the chain can tell the first one to stop.

That is the same shape as the dynamic-filter idea: an operator learns something at runtime and passes it upstream so the work upstream of it can be avoided. The limit is just the simplest possible instance of it.

Window frames over preserved sort order: how DataFusion evaluates PARTITION BY and ORDER BY

Jay Hubbert — Wed, 21 Oct 2026 02:13:00 +0000

TL;DR: A window function is a sorted, grouped fold. DataFusion makes the operator declare what it needs — KeyPartitioned(partition_by) plus an ordering of (partition_by, order_by) — and the planner inserts the repartition and the sort. Then the frame is evaluated with one of two accumulators: a plain one that only adds rows (for UNBOUNDED PRECEDING frames) or a sliding one that adds rows entering the frame and retracts rows leaving it.

The problem: a window function is three things at once

SUM(x) OVER (PARTITION BY dept ORDER BY ts ROWS BETWEEN 2 PRECEDING AND 1 FOLLOWING)

To evaluate this you need:

Grouping — all rows with the same dept must be together.
Ordering — within a group, rows must be in ts order, because the frame is defined relative to “the current row”.
Framing — for each row, a contiguous range of rows (the frame) whose values are aggregated.

Note that (1) and (2) are in tension. Grouping wants rows with equal dept adjacent; ordering wants ts sorted. The resolution is the standard one: sort by (dept, ts). Then equal-dept rows are contiguous and ordered by ts within each group. Partition boundaries are simply the places where dept changes.

The engineering question is: who arranges that? And the answer in DataFusion is that the operator asks, and the planner arranges.

The declaration

 rust
 
 
// datafusion/physical-plan/src/windows/window_agg_exec.rs:233-263
fn maintains_input_order(&self) -> Vec<bool> {
 vec![true]
}

fn required_input_ordering(&self) -> Vec<Option<OrderingRequirements>> {
 let partition_bys = self.window_expr()[0].partition_by();
 let order_keys = self.window_expr()[0].order_by();
 if self.ordered_partition_by_indices.len() < partition_bys.len() {
 vec![calc_requirements(partition_bys, order_keys)]
 } else {
 let partition_bys = self
 .ordered_partition_by_indices
 .iter()
 .map(|idx| &partition_bys[*idx]);
 vec![calc_requirements(partition_bys, order_keys)]
 }
}

fn input_distribution_requirements(&self) -> InputDistributionRequirements {
 if self.partition_keys().is_empty() {
 InputDistributionRequirements::new(vec![Distribution::SinglePartition])
 } else {
 InputDistributionRequirements::new(vec![Distribution::KeyPartitioned(
 self.partition_keys(),
 )])
 }
}

Read that as: “give me rows grouped by partition_by (or all together if there is no PARTITION BY), and within each group in order_by order.”

The ordering requirement is expressed as alternatives:

 rust
 
 
// datafusion/physical-plan/src/windows/mod.rs:309-342
pub(crate) fn calc_requirements<
 T: Borrow<Arc<dyn PhysicalExpr>>,
 S: Borrow<PhysicalSortExpr>,
>(
 partition_by_exprs: impl IntoIterator<Item = T>,
 orderby_sort_exprs: impl IntoIterator<Item = S>,
) -> Option<OrderingRequirements> {
 let mut sort_reqs_with_partition = partition_by_exprs
 .into_iter()
 .map(|partition_by| {
 PhysicalSortRequirement::new(Arc::clone(partition_by.borrow()), None)
 })
 .collect::<Vec<_>>();
 let mut sort_reqs = vec![];
 for element in orderby_sort_exprs.into_iter() {
 let PhysicalSortExpr { expr, options } = element.borrow();
 let sort_req = PhysicalSortRequirement::new(Arc::clone(expr), Some(*options));
 if !sort_reqs_with_partition.iter().any(|e| e.expr.eq(expr)) {
 sort_reqs_with_partition.push(sort_req.clone());
 }
 if !sort_reqs.iter().any(|e: &PhysicalSortRequirement| e.expr.eq(expr)) {
 sort_reqs.push(sort_req);
 }
 }

 let mut alternatives = vec![];
 alternatives.extend(LexRequirement::new(sort_reqs_with_partition));
 alternatives.extend(LexRequirement::new(sort_reqs));

 OrderingRequirements::new_alternatives(alternatives, false)
}

Two alternatives: sort by (partition_by, order_by), or sort by order_by alone. The second alternative is the interesting one — it says “if the input is already sorted by order_by only, that’s also acceptable”, and the operator can then find the partition boundaries itself by scanning for changes in the partition_by values. That is exactly what a BoundedWindowAggExec does.

The plan that results

Because the window operator declares both a distribution and an ordering requirement, EnsureRequirements builds this:

text

SortExec: partition_by ASC, order_by ASC ← ordering requirement
 RepartitionExec: Hash(partition_by) ← distribution requirement
 Scan

Or, with no PARTITION BY at all, the distribution requirement degrades to SinglePartition and everything is coalesced into one stream.

This is the “PartitionBy / SortPreservingMerge trick” in its simplest form: the repartition makes the groups local, the sort makes the order local, and the window operator then only has to walk contiguous runs.

Avoiding a redundant sort

The window operator doesn’t blindly demand (partition_by, order_by). It inspects the input’s existing ordering and finds the longest permutation of the PARTITION BY columns that is already a prefix of it:

 rust
 
 
// datafusion/physical-plan/src/windows/mod.rs:344-357
/// This function calculates the indices such that when partition by expressions reordered with the indices
/// resulting expressions define a preset for existing ordering.
/// For instance, if input is ordered by a, b, c and PARTITION BY b, a is used,
/// this vector will be [1, 0]. It means that when we iterate b, a columns with the order [1, 0]
/// resulting vector (a, b) is a preset of the existing ordering (a, b, c).
pub fn get_ordered_partition_by_indices(
 partition_by_exprs: &[Arc<dyn PhysicalExpr>],
 input: &Arc<dyn ExecutionPlan>,
) -> Result<Vec<usize>> {
 let (_, indices) = input
 .equivalence_properties()
 .find_longest_permutation(partition_by_exprs)?;
 Ok(indices)
}

So if the input is already sorted by (a, b, c) and the window says PARTITION BY b, a, the operator discovers that (a, b) is a prefix of the existing ordering, and the required ordering becomes (a, b, c) — not (b, a, ...). The sort is satisfied by the existing order and the WindowAggExec can run with zero sorting.

The ordered_partition_by_indices are then used consistently: in the declared ordering requirement, in partition_by_sort_keys() (the sort expressions used to detect partition boundaries), and in compute_aggregates(), which evaluates only those partition-by columns when scanning for boundaries.

The frame representation

The frame is two bounds plus a unit:

 rust
 
 
// datafusion/expr/src/window_frame.rs:38-45
#[derive(Clone, PartialEq, Eq, PartialOrd, Hash)]
pub struct WindowFrame {
 /// Frame type - either `ROWS`, `RANGE` or `GROUPS`
 pub units: WindowFrameUnits,
 /// Starting frame boundary
 pub start_bound: WindowFrameBound,
 /// Ending frame boundary
 pub end_bound: WindowFrameBound,

and the bound itself:

 rust
 
 
// datafusion/expr/src/window_frame.rs:312-351
/// There are five ways to describe starting and ending frame boundaries:
///
/// 1. UNBOUNDED PRECEDING
/// 2. `<expr>` PRECEDING
/// 3. CURRENT ROW
/// 4. `<expr>` FOLLOWING
/// 5. UNBOUNDED FOLLOWING
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
pub enum WindowFrameBound {
 /// 1. UNBOUNDED PRECEDING
 /// The frame boundary is the first row in the partition.
 ///
 /// 2. `<expr>` PRECEDING
 /// `<expr>` must be a non-negative constant numeric expression. The boundary is a row that
 /// is `<expr>` "units" prior to the current row.
 Preceding(ScalarValue),
 /// 3. The current row.
 ///
 /// For RANGE and GROUPS frame types, peers of the current row are also
 /// included in the frame, unless specifically excluded by the EXCLUDE clause.
 /// This is true regardless of whether CURRENT ROW is used as the starting or ending frame
 /// boundary.
 CurrentRow,
 /// 4. This is the same as "`<expr>` PRECEDING" except that the boundary is `<expr>` units after the
 /// current rather than before the current row.
 ///
 /// 5. UNBOUNDED FOLLOWING
 /// The frame boundary is the last row in the partition.
 Following(ScalarValue),
}

impl WindowFrameBound {
 pub fn is_unbounded(&self) -> bool {
 match self {
 WindowFrameBound::Preceding(elem) => elem.is_null(),
 WindowFrameBound::CurrentRow => false,
 WindowFrameBound::Following(elem) => elem.is_null(),
 }
 }
}

The encoding trick here is that “UNBOUNDED” is a NULL ScalarValue. Preceding(ScalarValue::UInt64(None)) means UNBOUNDED PRECEDING; Preceding(ScalarValue::UInt64(Some(5))) means 5 PRECEDING. One enum variant covers both the unbounded and the bounded case, and is_unbounded() is a null check. This is why the frame arithmetic below can be written as a single match over three cases.

For ROWS frames, the offset is always a UInt64, so the arithmetic is pure integer work:

 rust
 
 
// datafusion/expr/src/window_state.rs:243-295
fn calculate_range_rows(
 window_frame: &Arc<WindowFrame>,
 length: usize,
 idx: usize,
) -> Result<Range<usize>> {
 let start = match window_frame.start_bound {
 // UNBOUNDED PRECEDING
 WindowFrameBound::Preceding(ScalarValue::UInt64(None)) => 0,
 WindowFrameBound::Preceding(ScalarValue::UInt64(Some(n))) => {
 idx.saturating_sub(n as usize)
 }
 WindowFrameBound::CurrentRow => idx,
 // UNBOUNDED FOLLOWING
 WindowFrameBound::Following(ScalarValue::UInt64(None)) => {
 return internal_err!(
 "Frame start cannot be UNBOUNDED FOLLOWING '{window_frame:?}'"
 );
 }
 WindowFrameBound::Following(ScalarValue::UInt64(Some(n))) => {
 std::cmp::min(idx + n as usize, length)
 }
 ...
 };
 let end = match window_frame.end_bound {
 ...
 WindowFrameBound::CurrentRow => idx + 1,
 // UNBOUNDED FOLLOWING
 WindowFrameBound::Following(ScalarValue::UInt64(None)) => length,
 WindowFrameBound::Following(ScalarValue::UInt64(Some(n))) => {
 std::cmp::min(idx + n as usize + 1, length)
 }
 ...
 };
 Ok(Range { start, end })
}

Note that the ROWS path is stateless while RANGE and GROUPS are stateful:

 rust
 
 
// datafusion/expr/src/window_state.rs:170-189
/// This object stores the window frame state for use in incremental calculations.
#[derive(Debug, Clone)]
pub enum WindowFrameContext {
 /// ROWS frames are inherently stateless.
 Rows(Arc<WindowFrame>),
 /// RANGE frames are stateful, they store indices specifying where the
 /// previous search left off. This amortizes the overall cost to O(n)
 /// where n denotes the row count.
 Range {
 window_frame: Arc<WindowFrame>,
 state: WindowFrameStateRange,
 },
 /// GROUPS frames are stateful, they store group boundaries and indices
 /// specifying where the previous search left off. This amortizes the
 /// overall cost to O(n) where n denotes the row count.
 Groups { ... },
}

A RANGE BETWEEN INTERVAL '1' HOUR PRECEDING AND CURRENT ROW frame has to binary-search the order-by column for every row; remembering where the last search ended turns that into an amortized linear scan.

The default frames

The defaults are where “no ORDER BY” gets decided:

 rust
 
 
// datafusion/expr/src/window_frame.rs:145-176
impl WindowFrame {
 /// Creates a new, default window frame (with the meaning of default
 /// depending on whether the frame contains an `ORDER BY` clause and this
 /// ordering is strict (i.e. no ties).
 pub fn new(order_by: Option<bool>) -> Self {
 if let Some(strict) = order_by {
 // This window frame covers the table (or partition if `PARTITION BY`
 // is used) from beginning to the `CURRENT ROW` (with same rank). It
 // is used when the `OVER` clause contains an `ORDER BY` clause but
 // no frame.
 Self {
 units: if strict {
 WindowFrameUnits::Rows
 } else {
 WindowFrameUnits::Range
 },
 start_bound: WindowFrameBound::Preceding(ScalarValue::UInt64(None)),
 end_bound: WindowFrameBound::CurrentRow,
 causal: strict,
 }
 } else {
 // This window frame covers the whole table (or partition if `PARTITION BY`
 // is used). It is used when the `OVER` clause does not contain an
 // `ORDER BY` clause and there is no frame.
 Self {
 units: WindowFrameUnits::Rows,
 start_bound: WindowFrameBound::Preceding(ScalarValue::UInt64(None)),
 end_bound: WindowFrameBound::Following(ScalarValue::UInt64(None)),
 causal: false,
 }
 }
 }
}

So:

`OVER` clause	default frame	units
`PARTITION BY d`	`UNBOUNDED PRECEDING → UNBOUNDED FOLLOWING` (whole partition)	ROWS
`PARTITION BY d ORDER BY t` (strict)	`UNBOUNDED PRECEDING → CURRENT ROW`	ROWS
`PARTITION BY d ORDER BY t` (ties)	`UNBOUNDED PRECEDING → CURRENT ROW`	RANGE

The third row is the subtle one: with ties in the ORDER BY column, the default frame is RANGE, so that “current row” means “current row and its peers” — the SQL standard’s RANGE ... CURRENT ROW semantics. With a strict ordering (no ties possible), ROWS and RANGE agree, so ROWS is used because it’s cheaper.

Also note causal. The frame is causal if computing row i doesn’t depend on row i+1:

 rust
 
 
// datafusion/expr/src/window_frame.rs:208-252
pub fn new_bounds(
 units: WindowFrameUnits,
 start_bound: WindowFrameBound,
 end_bound: WindowFrameBound,
) -> Self {
 let causal = match units {
 WindowFrameUnits::Rows => match &end_bound {
 WindowFrameBound::Following(value) => {
 if value.is_null() {
 // Unbounded following
 false
 } else { ... }
 }
 _ => true,
 },
 ...
 };
 Self { units, start_bound, end_bound, causal }
}

Causality is what makes streaming window evaluation possible: if the frame never looks forward, row i can be emitted as soon as row i is seen.

Two accumulators: growing frames vs sliding frames

The choice of accumulator is made at expression-construction time, from the frame’s shape:

 rust
 
 
// datafusion/physical-plan/src/windows/mod.rs:139-167
/// Creates an appropriate [`WindowExpr`] based on the window frame and
fn window_expr_from_aggregate_expr(
 partition_by: &[Arc<dyn PhysicalExpr>],
 order_by: &[PhysicalSortExpr],
 window_frame: Arc<WindowFrame>,
 aggregate: Arc<AggregateFunctionExpr>,
 filter: Option<Arc<dyn PhysicalExpr>>,
) -> Arc<dyn WindowExpr> {
 // Is there a potentially unlimited sized window frame?
 let unbounded_window = window_frame.is_ever_expanding();

 if !unbounded_window {
 Arc::new(SlidingAggregateWindowExpr::new(
 aggregate, partition_by, order_by, window_frame, filter,
 ))
 } else {
 Arc::new(PlainAggregateWindowExpr::new(
 aggregate, partition_by, order_by, window_frame, filter,
 ))
 }
}

with

 rust
 
 
// datafusion/expr/src/window_frame.rs:304-309
/// Is the window frame ever-expanding (it always grows in the superset sense).
/// Useful when understanding if set-monotonicity properties of functions can
/// be exploited.
pub fn is_ever_expanding(&self) -> bool {
 self.start_bound.is_unbounded()
}

Ever-expanding means the frame’s start is UNBOUNDED PRECEDING: rows only ever get added to the frame, never removed. SUM(x) OVER (ORDER BY t) is the canonical example. The accumulator is a plain update_batch-only accumulator — no retraction needed.

Sliding frames (ROWS BETWEEN 2 PRECEDING AND 1 FOLLOWING, or anything with a bounded start) need rows to enter and leave. That’s where the sliding accumulator earns its name:

 rust
 
 
// datafusion/physical-expr/src/window/sliding_aggregate.rs:180-235
/// Given current range and the last range, calculates the accumulator
/// result for the range of interest.
fn get_aggregate_result_inside_range(
 &self,
 last_range: &Range<usize>,
 cur_range: &Range<usize>,
 value_slice: &[ArrayRef],
 accumulator: &mut Box<dyn Accumulator>,
 filter_mask: Option<&BooleanArray>,
) -> Result<ScalarValue> {
 if cur_range.start == cur_range.end {
 // Keep the accumulator synchronized with `last_range`. RANGE frames
 // can become empty between two non-empty frames when the ORDER BY
 // values contain gaps.
 let retract_bound = last_range.end - last_range.start;
 if retract_bound > 0 {
 ... accumulator.retract_batch(&retract)? ...
 }
 self.aggregate
 .default_value(self.aggregate.field().data_type())
 } else {
 // Accumulate any new rows that have entered the window:
 let update_bound = cur_range.end - last_range.end;
 if update_bound > 0 {
 ... accumulator.update_batch(&update)? ...
 }

 // Remove rows that have now left the window:
 let retract_bound = cur_range.start - last_range.start;
 if retract_bound > 0 {
 ... accumulator.retract_batch(&retract)? ...
 }
 ...
 }
}

This is the classic sliding-window trick: instead of re-aggregating the whole frame for every row (which is O(N × W)), you keep the accumulator in sync with the previous frame and only apply the delta:

text

frame(i-1) = [s .. e)
frame(i) = [s' .. e')
delta = retract [s .. s'), update [e .. e')

Cost per row: O(1) amortized, versus O(W) for a naive re-aggregation. This requires the aggregate to support retract_batch — hence create_sliding_accumulator vs create_accumulator.

The plain (ever-expanding) accumulator has a corresponding memory trick — it can discard old rows because they’ll never be needed again:

 rust
 
 
// datafusion/physical-expr/src/window/aggregate.rs:196-220
fn evaluate_stateful(
 &self,
 partition_batches: &PartitionBatches,
 window_agg_state: &mut PartitionWindowAggStates,
 eval_ctx: &WindowEvalContext<'_>,
) -> Result<()> {
 self.aggregate_evaluate_stateful(partition_batches, window_agg_state, eval_ctx)?;

 // Update window frame range for each partition. As we know that
 // non-sliding aggregations will never call `retract_batch`, this value
 // can safely increase, and we can remove "old" parts of the state.
 // This enables us to run queries involving UNBOUNDED PRECEDING frames
 // using bounded memory for suitable aggregations.
 for partition_row in partition_batches.keys() {
 ...
 if self.window_frame.start_bound.is_unbounded() {
 state.window_frame_range.start =
 state.window_frame_range.end.saturating_sub(1);
 }
 }
 Ok(())
}

So SUM(x) OVER (ORDER BY t) — which naively needs the whole prefix summed for every row — can run in bounded memory, because only the last row’s prefix is ever needed again.

The “no ORDER BY” special case

OVER (PARTITION BY d) with no ORDER BY gives the whole-partition frame. The PlainAggregateWindowExpr notices that this frame is the same for every row in the partition and skips all per-row work:

 rust
 
 
// datafusion/physical-expr/src/window/aggregate.rs:146-168
// Returns true if every row in the partition has the same window frame. This allows
// for preventing bound + function calculation for every row due to the values being the
// same.
//
// This occurs when both bounds fall under either condition below:
// 1. Bound is unbounded (`Preceding` or `Following`)
// 2. Bound is `CurrentRow` while using `Range` units with no order by clause
// This results in an invalid range specification. Following PostgreSQL's convention,
// We interpret this as the entire partition being used for the current window frame.
fn is_window_constant_in_partition(
 order_by: &[PhysicalSortExpr],
 window_frame: &WindowFrame,
) -> bool {
 let is_constant_bound = |bound: &WindowFrameBound| match bound {
 WindowFrameBound::CurrentRow => {
 window_frame.units == WindowFrameUnits::Range && order_by.is_empty()
 }
 _ => bound.is_unbounded(),
 };

 is_constant_bound(&window_frame.start_bound)
 && is_constant_bound(&window_frame.end_bound)
}

and the evaluation path checks it first:

 rust
 
 
// datafusion/physical-expr/src/window/window_expr.rs:333-348
if self.is_constant_in_partition() {
 if not_end {
 let field = self.field()?;
 let out_type = field.data_type();
 return Ok(new_empty_array(out_type));
 }
 let values = if let Some(mask) = filter_mask { ... } else { values };
 accumulator.update_batch(&values)?;
 let value = accumulator.evaluate()?;
 return value.to_array_of_size(record_batch.num_rows());
}

So SUM(x) OVER (PARTITION BY d) is computed once per partition and then broadcast to every row of the partition. That is the difference between O(N) aggregate calls and O(1) per partition.

Two executors: `WindowAggExec` and `BoundedWindowAggExec`

There are two physical window operators, and the difference is memory behaviour.

WindowAggExec is the simple one — and it buffers everything:

 rust
 
 
// datafusion/physical-plan/src/windows/window_agg_exec.rs:541-582
fn compute_aggregates(&self) -> Result<Option<RecordBatch>> {
 // record compute time on drop
 let _timer = self.baseline_metrics.elapsed_compute().timer();

 let batch = concat_batches(&self.input.schema(), &self.batches)?;
 if batch.num_rows() == 0 {
 return Ok(None);
 }

 let partition_by_sort_keys = self
 .ordered_partition_by_indices
 .iter()
 .map(|idx| self.partition_by_sort_keys[*idx].evaluate_to_sort_column(&batch))
 .collect::<Result<Vec<_>>>()?;
 let partition_points =
 evaluate_partition_ranges(batch.num_rows(), &partition_by_sort_keys)?;

 let mut partition_results = Vec::with_capacity(partition_points.len());
 // Calculate window cols
 for partition_point in partition_points {
 let length = partition_point.end - partition_point.start;
 partition_results.push(compute_window_aggregates(
 &self.window_expr,
 &batch.slice(partition_point.start, length),
 )?)
 }
 let columns = transpose(partition_results)
 .iter()
 .map(|elems| concat(&elems.iter().map(|x| x.as_ref()).collect::<Vec<_>>()))
 .collect::<Result<Vec<ArrayRef>, ArrowError>>()?;
 ...
}

and the stream drains the whole input before computing anything:

 rust
 
 
// datafusion/physical-plan/src/windows/window_agg_exec.rs:607-628
loop {
 return Poll::Ready(Some(match ready!(self.input.poll_next_unpin(cx)) {
 Some(Ok(batch)) => {
 self.batches.push(batch);
 continue;
 }
 Some(Err(e)) => Err(e),
 None => {
 // Release the input pipeline's resources before computing
 // the final aggregates.
 let input_schema = self.input.schema();
 self.input = Box::pin(EmptyRecordBatchStream::new(input_schema));
 let Some(result) = self.compute_aggregates()? else {
 return Poll::Ready(None);
 };
 self.finished = true;
 ...
 Ok(result)
 }
 }));
}

So WindowAggExec is O(partition) memory. It relies on the repartition to make partitions small, and on the sort to make partition boundaries detectable by simple comparison.

BoundedWindowAggExec is the streaming one. It doesn’t require the input to be sorted by PARTITION BY at all — it hashes the partition keys and groups rows itself:

 rust
 
 
// datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs:920-955
fn compute_partition_permutation(
 &mut self,
 columns: &[ArrayRef],
 batch: &RecordBatch,
) -> Result<(Vec<PartitionKey>, Vec<u32>, Vec<usize>)> {
 let num_rows = batch.num_rows();
 self.batch_hashes.clear();
 self.batch_hashes.resize(num_rows, 0);
 create_hashes(columns, &self.random_state, &mut self.batch_hashes)?;
 self.input_buffer_hashes.extend(&self.batch_hashes);
 self.row_map_batch.clear();
 let mut keys: Vec<PartitionKey> = vec![];
 // Partition id of each row, in row order:
 let mut row_partition_ids: Vec<usize> = Vec::with_capacity(num_rows);
 for (&hash, row_idx) in self.batch_hashes.iter().zip(0u32..) {
 let entry = self.row_map_batch.find_mut(hash, |(_, group_idx)| {
 let row = get_row_at_idx(columns, row_idx as usize).unwrap();
 // Handle hash collisions with an equality check:
 row == keys[*group_idx]
 });
 ...
 }
 ...
}

Note the hash-collision handling: the hash is only used to find a candidate group; the actual key values are compared for equality. That’s the standard open-addressing-with-equality-check pattern, and it’s why the grouping is correct even with adversarial hash collisions. The rows of each partition are then scattered into contiguous runs (a counting sort by partition id), and the per-partition state is updated incrementally as batches arrive.

The choice between the two executors is made by get_best_fitting_window:

 rust
 
 
// datafusion/physical-plan/src/windows/mod.rs:657-681
// If all window expressions can run with bounded memory, choose the
// bounded window variant:
if window_expr.iter().all(|e| e.uses_bounded_memory()) {
 Ok(Some(Arc::new(
 BoundedWindowAggExec::try_new(
 window_expr,
 Arc::clone(input),
 input_order_mode,
 !physical_partition_keys.is_empty(),
 )?
 .with_state_observer(state_observer)?,
 ) as _))
} else if input_order_mode != InputOrderMode::Sorted {
 // For `WindowAggExec` to work correctly PARTITION BY columns should be sorted.
 // Hence, if `input_order_mode` is not `Sorted` we should convert
 // input ordering such that it can work with `Sorted` (add `SortExec`).
 // Effectively `WindowAggExec` works only in `Sorted` mode.
 Ok(None)
} else {
 Ok(Some(Arc::new(WindowAggExec::try_new(
 window_expr,
 Arc::clone(input),
 !physical_partition_keys.is_empty(),
 )?) as _))
}

and uses_bounded_memory is decided by the frame’s end bound:

 rust
 
 
// datafusion/physical-expr/src/window/aggregate.rs:244-246
fn uses_bounded_memory(&self) -> bool {
 !self.window_frame.end_bound.is_unbounded()
}

Here’s the resulting taxonomy, and it’s a nice one:

frame	end bound	executor	memory
`PARTITION BY d` (no ORDER BY)	UNBOUNDED FOLLOWING	`WindowAggExec`	whole partition
`ORDER BY t` (default)	CURRENT ROW	`BoundedWindowAggExec` (if aggregate supports it)	bounded
`ROWS BETWEEN 2 PRECEDING AND 1 FOLLOWING`	FOLLOWING(1)	`BoundedWindowAggExec`	bounded

The “no ORDER BY” case is the odd one out: because the frame extends to the end of the partition, you cannot emit a row until you have seen the whole partition — there’s no way to know the final aggregate value otherwise. That’s a semantic fact, not an implementation limitation, and it’s why the default no-ORDER BY frame forces the buffering executor.

The `functions-window` crate: three evaluation APIs

The built-in window functions don’t implement frame evaluation themselves. They implement PartitionEvaluator, which has three methods, and the choice between them is a big part of the performance story:

 rust
 
 
// datafusion/expr/src/partition_evaluator.rs:74-88
/// # Implementation Table
///
/// |[`uses_window_frame`]|[`supports_bounded_execution`]|[`include_rank`]|
/// function_to_implement|
/// |---|---|---|---|
/// |false (default) |false (default) |false (default) | [`evaluate_all`] |
/// |false |true |false | [`evaluate`] |
/// |false |true/false |true | [`evaluate_all_with_rank`] |
/// |true |true/false |true/false | [`evaluate`] |

evaluate_all is the fast path: the whole partition at once, one array out. row_number uses it:

 rust
 
 
// datafusion/functions-window/src/row_number.rs:140-161
fn evaluate_all(
 &mut self,
 _values: &[ArrayRef],
 num_rows: usize,
) -> Result<ArrayRef> {
 Ok(Arc::new(UInt64Array::from_iter_values(
 1..=(num_rows as u64),
 )))
}

fn evaluate(&mut self,
 _values: &[ArrayRef],
 _range: &Range<usize>,
) -> Result<ScalarValue> {
 self.n_rows += 1;
 Ok(ScalarValue::UInt64(Some(self.n_rows as u64)))
}

row_number over a partition of n rows is literally 1..=n — no per-row state, no frame arithmetic, no accumulator. The per-row evaluate exists only as a fallback for the bounded-execution path.

rank uses the third API, because ranks are peer-based: they need the group boundaries, which the executor computes:

 rust
 
 
// datafusion/functions-window/src/rank.rs:314-331
fn evaluate_all_with_rank(
 &self,
 num_rows: usize,
 ranks_in_partition: &[Range<usize>],
) -> Result<ArrayRef> {
 let result: ArrayRef = match self.rank_type {
 RankType::Basic => Arc::new(UInt64Array::from_iter_values(
 ranks_in_partition
 .iter()
 .scan(1_u64, |acc, range| {
 let len = range.end - range.start;
 let result = iter::repeat_n(*acc, len);
 *acc += len as u64;
 Some(result)
 })
 .flatten(),

lead/lag use evaluate_all too, with an explicit shift (functions-window/src/lead_lag.rs:705-722). And cume_dist and percent_rank are rank-based too — they need the total row count of the partition, which is why they cannot be evaluated in a streaming fashion:

 rust
 
 
// datafusion/functions-window/src/rank.rs:308-310
RankType::Percent => {
 exec_err!("Can not execute PERCENT_RANK in a streaming fashion")
}

Trade-offs and failure modes

The repartition is a real cost, and it’s a full data copy. The RepartitionExec(Hash(partition_by)) that the window’s declaration triggers moves every row. For a window over a table that’s already partitioned by the right key (Parquet row groups sorted by dept), that’s pure waste — and the planner can’t know. preserve_file_partitions and Partitioning::Range exist precisely to let the source declare “I’m already partitioned this way” and skip the shuffle.

Skew in PARTITION BY is fatal. A window with PARTITION BY country where one country is 90% of the rows means one hash partition holds 90% of the data, and WindowAggExec will buffer all of it. The window operator inherits the shuffle’s skew problem, and there’s no partial mitigation: unlike a join, you can’t split a partition’s rows across threads without breaking the ordering guarantee.

WindowAggExec buffers the entire partition. The stream pushes every input batch into self.batches and only computes when the input ends. For a 100 GB partition that’s 100 GB of RAM (or an OOM). BoundedWindowAggExec avoids it, but only when the frame’s end bound is bounded and the aggregate supports retraction/streaming. When neither holds, the only options are to spill or to repartition more finely.

The ORDER BY in the OVER clause is not free — and sometimes isn’t needed. OVER (PARTITION BY d ORDER BY t) sorts every partition. If the window function doesn’t actually need the order — SUM(x) OVER (PARTITION BY d) doesn’t — you still pay for the sort, because the frame semantics depend on the ordering even if the function doesn’t. DataFusion does exploit one case: the is_window_constant_in_partition check above means that for the whole-partition frame, the sort contributes nothing to the result.

Reversing the sort is a legitimate optimization. Some window functions can be evaluated backwards — LAG(x, 1) is LEAD(x, 1) evaluated in reverse order. get_window_mode checks whether the existing input ordering satisfies the requirement with the order-by keys reversed, and if so, the operator runs backwards:

 rust
 
 
// datafusion/physical-plan/src/windows/mod.rs:695-732
pub fn get_window_mode(
 partitionby_exprs: &[Arc<dyn PhysicalExpr>],
 orderby_keys: &[PhysicalSortExpr],
 input: &Arc<dyn ExecutionPlan>,
) -> Result<Option<(bool, InputOrderMode>)>> {
 ...
 let reverse_orderby_keys =
 orderby_keys.iter().map(|e| e.reverse()).collect::<Vec<_>>();
 for (should_swap, orderbys) in
 [(false, orderby_keys), (true, reverse_orderby_keys.as_ref())]
 {
 ...
 if req.is_empty() || input_eqs.ordering_satisfy_requirement(req)? {
 // Window can be run with existing ordering
 let mode = if indices.len() == partitionby_exprs.len() {
 InputOrderMode::Sorted
 } else if indices.is_empty() {
 InputOrderMode::Linear
 } else {
 InputOrderMode::PartiallySorted(indices)
 };
 return Ok(Some((should_swap, mode)));
 }
 }
 Ok(None)
}

InputOrderMode is the honest description of what the input actually guarantees: Sorted (partition-by columns are a prefix of the ordering), PartiallySorted(indices) (some prefix of the partition-by columns is), or Linear (no ordering at all — only valid when there’s no PARTITION BY). The bounded executor handles all three; the plain one only handles Sorted.

Bottom line

The window operator is where DataFusion’s declarative design pays off most visibly. Three independent requirements — grouping, ordering, and framing — are expressed as metadata, and one optimizer rule turns them into a repartition, a sort, and (sometimes) nothing at all. The evaluation itself is then a two-accumulator choice: grow the frame and only add rows, or slide it and add-and-retract the delta. The frame bounds are a three-variant enum in which “unbounded” is a NULL scalar, and the “no ORDER BY” case is recognized early enough to collapse an O(N) per-row computation into a single per-partition aggregate.

Operators declare what they need: how DataFusion's planner inserts the shuffle for you

Jay Hubbert — Wed, 21 Oct 2026 01:17:00 +0000

TL;DR: An operator never builds a shuffle itself. It declares a Distribution requirement (Unspecified, SinglePartition, or KeyPartitioned(exprs)), and the EnsureRequirements physical optimizer rule walks the plan bottom-up, checks whether the child’s current Partitioning satisfies the requirement, and inserts a RepartitionExec only when it doesn’t. Joins add a second, harder constraint: both sides must land in the same partition for the same key.

The problem: who decides where data goes?

Consider SELECT ... FROM a JOIN b ON a.k = b.k. A partitioned hash join wants the following invariant:

text

left: Hash([k]) right: Hash([k])
 P0: k ∈ {1, 4, 9} P0: k ∈ {1, 4, 9}
 P1: k ∈ {2, 5} P1: k ∈ {2, 5}

Partition i of the left side may only be joined with partition i of the right side. If the scan of a produces 8 file partitions and the scan of b produces 3, somebody has to move rows until both sides agree on the layout. Where should that decision live?

The naive answer — “in the planner, with hard-coded rules per operator type” — is what most query engines do, and it rots. Every new operator needs its rule; every rule needs to know about every other operator’s behaviour; the join rule has to know that the aggregate below it also repartitions. It’s a web of special cases.

DataFusion inverted this. The operator states what it needs; the optimizer satisfies it generically.

The vocabulary: `Distribution` vs `Partitioning`

Two enums, and the difference matters.

Partitioning describes what a node actually produces:

 rust
 
 
// datafusion/physical-expr/src/partitioning.rs:122-132
pub enum Partitioning {
 RoundRobinBatch(usize),
 Hash(Vec<Arc<dyn PhysicalExpr>>, usize),
 Range(RangePartitioning),
 UnknownPartitioning(usize),
}

Distribution describes what a node requires of its input:

 rust
 
 
// datafusion/physical-expr/src/partitioning.rs:934-946
pub enum Distribution {
 /// Unspecified distribution
 UnspecifiedDistribution,
 /// A single partition is required
 SinglePartition,
 /// Deprecated historical name for [`Distribution::KeyPartitioned`].
 #[deprecated(since = "55.0.0", note = "Use Distribution::KeyPartitioned")]
 HashPartitioned(Vec<Arc<dyn PhysicalExpr>>),
 /// Requires children to be distributed in such a way that the same
 /// values of the keys end up in the same partition
 KeyPartitioned(Vec<Arc<dyn PhysicalExpr>>),
}

HashPartitioned is deprecated in favour of KeyPartitioned — a naming change that reflects a semantic broadening: the requirement is about key locality, not about the literal hash function. The requirement doesn’t say “partition by hash(k) % 8”; it says “rows with the same k must be in the same partition, and it must be the same partition across children”.

The matching side of this is PartitioningSatisfaction:

 rust
 
 
// datafusion/physical-expr/src/partitioning.rs:535-554
pub enum PartitioningSatisfaction {
 /// The partitioning does not satisfy the distribution requirement
 NotSatisfied,
 /// The partitioning exactly matches the distribution requirement
 Exact,
 /// The partitioning satisfies the distribution requirement via subset logic
 Subset,
}

Subset is the interesting one. If a child is already hash-partitioned on (k) and the parent needs (k, j), then (k) is not sufficient (rows with the same k but different j would be split) — that’s NotSatisfied. But if the child is hash-partitioned on (k, j) and the parent only needs (k), the child’s partitioning is stronger than required, so it’s satisfied. The rule is implemented in key_satisfaction (partitioning.rs:649-684), and it normalizes both sides through the plan’s equivalence classes first, so a and b count as the same key if the optimizer knows they’re equal.

Declaring the requirement

The trait method is on ExecutionPlan itself:

 rust
 
 
// datafusion/physical-plan/src/execution_plan.rs:188-210
/// Specifies simple per-child input distribution requirements.
///
/// Deprecated: override [`Self::input_distribution_requirements`] instead.
///
/// By default, each child has [`Distribution::UnspecifiedDistribution`].
#[deprecated(since = "55.0.0", note = "Use input_distribution_requirements")]
fn required_input_distribution(&self) -> Vec<Distribution> {
 vec![Distribution::UnspecifiedDistribution; self.children().len()]
}

/// Specifies the input distribution requirements for this plan.
///
/// The default implementation wraps [`Self::required_input_distribution`].
/// Override this method for richer requirements, such as allowing alternate
/// satisfaction policies or requiring multiple children to be co-partitioned.
fn input_distribution_requirements(&self) -> InputDistributionRequirements {
 InputDistributionRequirements::new(self.required_input_distribution())
}

So the modern API is input_distribution_requirements, and the old required_input_distribution is a compatibility shim for out-of-tree operators. The richer return type exists because per-child independence is not always enough:

 rust
 
 
// datafusion/physical-plan/src/distribution_requirements.rs:59-66
#[non_exhaustive]
#[derive(Debug, Clone)]
pub struct InputDistributionRequirements {
 /// Per-child distribution requirements, indexed by child position.
 children: Vec<ChildDistributionRequirement>,
 /// Child indexes that must also have compatible partition layouts.
 co_partitioned: Option<Vec<usize>>,
}

co_partitioned is the join constraint. InputDistributionRequirements::co_partitioned documents it precisely:

 rust
 
 
// datafusion/physical-plan/src/distribution_requirements.rs:107-132
/// Create a requirement that all children are co-partitioned.
///
/// ```text
/// left: Range(left.a ASC, split_points=[10, 20])
/// right: Range(right.x ASC, split_points=[10, 20])
///
/// partition 0 from both sides contains keys before 10
/// partition 1 from both sides contains keys in [10, 20)
/// partition 2 from both sides contains keys at/after 20
/// ```
///
/// If the split points differ, partition `i` from one side no longer covers
/// the same key range as partition `i` from the other side.

Who declares what

A partial aggregate wants any distribution — it will merge later:

 rust
 
 
// datafusion/physical-plan/src/aggregates/mod.rs:2514-2526
fn input_distribution_requirements(&self) -> InputDistributionRequirements {
 InputDistributionRequirements::new(match &self.mode {
 AggregateMode::Partial | AggregateMode::PartialReduce => {
 vec![Distribution::UnspecifiedDistribution]
 }
 AggregateMode::FinalPartitioned | AggregateMode::SinglePartitioned => {
 vec![Distribution::KeyPartitioned(self.group_by().input_exprs())]
 }
 AggregateMode::Final | AggregateMode::Single => {
 vec![Distribution::SinglePartition]
 }
 })
}

A window function wants its PARTITION BY keys co-located — and if it has no PARTITION BY at all, it wants everything in one place:

 rust
 
 
// datafusion/physical-plan/src/windows/window_agg_exec.rs:255-263
fn input_distribution_requirements(&self) -> InputDistributionRequirements {
 if self.partition_keys().is_empty() {
 InputDistributionRequirements::new(vec![Distribution::SinglePartition])
 } else {
 InputDistributionRequirements::new(vec![Distribution::KeyPartitioned(
 self.partition_keys(),
 )])
 }
}

A global sort wants one partition:

 rust
 
 
// datafusion/physical-plan/src/sorts/sort.rs:1488-1495
vec![Distribution::SinglePartition]

And a SortPreservingMergeExec explicitly wants nothing, because it consumes multiple partitions and merges them:

 rust
 
 
// datafusion/physical-plan/src/sorts/sort_preserving_merge.rs:248-252
fn input_distribution_requirements(&self) -> crate::InputDistributionRequirements {
 crate::InputDistributionRequirements::new(vec![
 Distribution::UnspecifiedDistribution,
 ])
}

Note the discipline here: an operator only declares what it needs. Distribution::UnspecifiedDistribution means “I don’t care” — and that is the hook the optimizer uses to decide it’s allowed to add parallelism.

Satisfying the requirement

The work happens in one physical optimizer rule, EnsureRequirements:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/mod.rs:166-288
/// Optimizer rule that enforces both distribution and sorting requirements.
///
/// This rule inserts the repartitioning and sorting operators needed to
/// satisfy the requirements that operators declare via
/// [`ExecutionPlan::input_distribution_requirements`] and
/// [`ExecutionPlan::required_input_ordering`].
pub struct EnsureRequirements {}

Its pipeline is a carefully ordered set of phases:

Phase 0: normalize InterleaveExec back to UnionExec (interleaves are artifacts of Phase 2 and would break if a child later loses its partitioning).
Phase 1: join key reordering, top-down, so that a parent join’s key order matches its children’s.
Phase 2a: distribution enforcement, bottom-up.
Phase 2b: sorting enforcement, bottom-up, on the distribution-fixed plan.
Phase 3: parallelize sorts, restore order-preserving variants, push sorts down, and convert some sorts to partial sorts.

Combining distribution and sorting into one pass is not cosmetic. The module docs spell out why:

 text
 
 
// datafusion/physical-optimizer/src/ensure_requirements/mod.rs:122-138
/// # Behavior: joint distribution + sorting
///
/// Resolving distribution and sorting together lets Phase 2 produce a
/// parallel sort plan in cases where the two-rule pipeline historically
/// risked a serial one. Given `Sort(DESC) ← Coalesce ← MultiPartitionSource`,
/// `EnsureRequirements` rewrites it into:
///
/// ```text
/// SortPreservingMergeExec: [a DESC] (cheap k-way merge of sorted streams)
/// SortExec: [a DESC], preserve_partitioning=true (N sorts run in parallel)
/// MultiPartitionSource
/// ```
///
/// Each input partition is sorted in parallel, then a `SortPreservingMergeExec`
/// at the top performs a cheap merge of pre-sorted streams. For TopK queries
/// (`fetch=K`), each parallel sort only keeps K rows per partition, so total
/// memory is `N × K` rather than coalescing the entire stream first.

Phase 2a: the actual insertion

The bottom-up pass computes, for each child, whether the requirement is already met:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1717-1748
match &requirement {
 Distribution::SinglePartition => {
 child = add_merge_on_top(child);
 }
 Distribution::HashPartitioned(_) | Distribution::KeyPartitioned(_) => {
 let child_partitions =
 child.plan.output_partitioning().partition_count();
 let partitioning_satisfied = input_distributions
 .child_satisfaction(
 child_idx,
 child.plan.as_ref(),
 ChildSatisfactionOptions::new()
 .with_allow_subset(allow_subset_satisfy_partitioning),
 )?
 .is_satisfied();
 // ...
 let needs_repartition = if allow_subset_satisfy_partitioning {
 !partitioning_satisfied
 } else {
 !partitioning_satisfied
 || (target_partitions > child_partitions
 && !preserve_satisfying_file_partitioning)
 };
 let should_add_repartition = hash_necessary && needs_repartition;

And when it decides a shuffle is needed, it builds one:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1752-1801
if should_add_repartition {
 let partitioning = match child.plan.output_partitioning() {
 Partitioning::Range(range) if partitioning_satisfied => {
 match range.scale(target_partitions) {
 Some(range) => { /* keep the range layout, just widen it */ }
 None => requirement
 .clone()
 .create_partitioning(target_partitions),
 }
 }
 _ => {
 requirement.clone().create_partitioning(target_partitions)
 }
 };
 // When there is an existing ordering, we preserve ordering during
 // repartition. ...
 let repartition = RepartitionExec::try_new(
 Arc::clone(&child.plan),
 partitioning,
 )?
 .with_preserve_order();
 let plan = Arc::new(repartition) as _;
 child = DistributionContext::new(plan, true, vec![child]);
}

Distribution::create_partitioning is the bridge from requirement to concrete operator (partitioning.rs:952-965):

 rust
 
 
pub fn create_partitioning(self, partition_count: usize) -> Partitioning {
 match self {
 Distribution::UnspecifiedDistribution => {
 Partitioning::UnknownPartitioning(partition_count)
 }
 Distribution::SinglePartition => Partitioning::UnknownPartitioning(1),
 Distribution::HashPartitioned(expr) | Distribution::KeyPartitioned(expr) => {
 Partitioning::Hash(expr, partition_count)
 }
 }
}

So a KeyPartitioned([k]) requirement becomes RepartitionExec(Hash([k], target_partitions)) — exactly the operator from the previous article.

The `hash_necessary` alignment trick

A naive implementation checks each child independently: “does the right side need a hash shuffle? Yes. Does the left? No. Then shuffle the right.” But that produces garbage — a partitioned join where one side is hash-partitioned and the other is not will silently drop rows that should have matched. The optimizer handles this by aligning the flags:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1173-1203
let is_hash = matches!(
 requirement,
 Distribution::HashPartitioned(_) | Distribution::KeyPartitioned(_)
);
// Hash re-partitioning is necessary when the input has more than one
// partitions:
let multi_partitions = child.output_partitioning().partition_count() > 1;
let roundrobin_sensible = roundrobin_beneficial && roundrobin_beneficial_stats;
needs_alignment |= is_hash && (multi_partitions || roundrobin_sensible);
repartition_status_flags.push((
 is_hash,
 RepartitionRequirementStatus {
 requirement,
 roundrobin_beneficial,
 roundrobin_beneficial_stats,
 hash_necessary: is_hash && multi_partitions,
 },
));
// ...
// Align hash necessary flags for hash partitions to generate consistent
// hash partitions at each children:
if needs_alignment {
 for (is_hash, status) in &mut repartition_status_flags {
 if *is_hash {
 status.hash_necessary = true;
 }
 }
}

If any child of a partitioned join needs a hash shuffle, all hash-keyed children get one. Consistency beats minimality here.

Co-partitioning: making the two sides agree

Even with both sides hash-partitioned, the number of partitions and the layout must match. enforce_distribution_relationships handles the cross-child constraint:

 rust
 
 
// datafusion/physical-plan/src/distribution_requirements.rs:366-393
fn compatible_co_partitioning_layout(
 first_partitioning: &Partitioning,
 other_partitioning: &Partitioning,
) -> bool {
 if first_partitioning.partition_count() == 1
 && other_partitioning.partition_count() == 1
 {
 return true;
 }
 if first_partitioning.partition_count() != other_partitioning.partition_count() {
 return false;
 }
 match (first_partitioning, other_partitioning) {
 (Partitioning::Hash(_, _), Partitioning::Hash(_, _)) => true,
 (Partitioning::Range(left), Partitioning::Range(right)) => {
 left.split_points() == right.split_points()
 && left.ordering().len() == right.ordering().len()
 && left
 .ordering()
 .iter()
 .zip(right.ordering())
 .all(|(left, right)| left.options == right.options)
 }
 _ => false,
 }
}

Two Hash partitionings with the same count are considered compatible because the hash function is fixed (REPARTITION_RANDOM_STATE with seed 0, see the previous article) — so hash(k) % 8 is the same function on both sides. Two Range partitionings must have identical split points, since range layouts are data-dependent and two different samplings would put the same key in different partitions.

Round robin for pure parallelism

When the requirement is UnspecifiedDistribution — the operator doesn’t care — the optimizer may still add a round-robin repartition just to get more parallelism:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1803-1809
Distribution::UnspecifiedDistribution => {
 if add_roundrobin {
 // Add round-robin repartitioning on top of the operator
 // to increase parallelism.
 child = add_roundrobin_on_top(child, target_partitions)?;
 }
}

But it’s gated on statistics and on the operator’s own opinion:

 rust
 
 
// datafusion/physical-plan/src/execution_plan.rs:245-262
fn benefits_from_input_partitioning(&self) -> Vec<bool> {
 // By default try to maximize parallelism with more CPUs if
 // possible
 self.input_distribution_requirements()
 .per_child_distributions()
 .map(|dist| !matches!(dist, Distribution::SinglePartition))
 .collect()
}

SortExec overrides this to vec![false] — a sort is not helped by more input partitions (it needs them coalesced, not split). And the round-robin is additionally gated on the child having more rows than one batch (roundrobin_beneficial_stats, enforce_distribution.rs:1165-1172), so shuffling a 3-row table to 32 cores is avoided.

Subset satisfaction

The strictest-match policy is documented at the top of the rule:

 text
 
 
// datafusion/physical-optimizer/src/ensure_requirements/mod.rs:140-147
/// # Behavior: strictest distribution match for joins
///
/// Distribution requirements are met in the strictest way. For example, a
/// hash join with keys `(a, b, c)` requires `Distribution(a, b, c)`. This
/// can in principle be satisfied by partitioning on any superset of any
/// subset of `(a, b, c)`, but this rule always partitions on the exact key
/// tuple `(a, b, c)`. This is sometimes more aggressive than strictly
/// necessary, but the strictest match helps avoid data skew in joins.

Subset satisfaction is opt-in per child and is disabled in exactly the cases where it would be unsafe:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1648-1683
// Allow subset satisfaction when:
// 1. Current partition count >= threshold
// 2. Not a partitioned join since must use exact hash matching for joins
// 3. Not a grouping set aggregate (requires exact hash including __grouping_id)
let requires_grouping_id = matches!(&requirement,
 Distribution::HashPartitioned(exprs) | Distribution::KeyPartitioned(exprs)
 if exprs.iter().any(|expr| {
 (expr.as_ref() as &dyn Any)
 .downcast_ref::<Column>()
 .is_some_and(|col| col.name() == Aggregate::INTERNAL_GROUPING_ID)
 })
);

let allow_subset_satisfy_partitioning = (current_partitions
 >= subset_satisfaction_threshold
 || (preserve_file_partition_threshold_met
 && current_partitions < target_partitions))
 && !is_partitioned_join
 && !requires_grouping_id;

__grouping_id is the hidden column that distinguishes grouping sets in ROLLUP/CUBE. If you hash on (a) but the aggregate needs (a, __grouping_id), the partial aggregates from different grouping sets would be merged into the same hash bucket and produce wrong results. Hence the explicit exclusion.

Trade-offs and failure modes

The requirement is a declaration, not a guarantee. The optimizer satisfies it at plan time using the Partitioning metadata. If a node lies about its output partitioning, everything downstream silently produces wrong results. This is why the invariants are checked: InputDistributionRequirements::check_invariants (distribution_requirements.rs:216-240) and ExecutionPlan::check_invariants verify co-partitioning at InvariantLevel::Executable.

SinglePartition is a parallelism black hole. Any operator that requires Distribution::SinglePartition — a global sort, a global aggregate, a window function with no PARTITION BY — collapses the whole pipeline to one thread below it. The optimizer’s only defence is to parallelize the work above the collapse point: sort each partition in parallel and merge with SortPreservingMergeExec. That’s the “joint distribution + sorting” behaviour quoted earlier, and it’s the single most important performance trick in the whole rule.

Skew survives the requirement. KeyPartitioned([k]) says “same key, same partition”. It says nothing about balance. If k is skewed, one partition gets 90% of the rows and the join’s critical path is that one thread. The strictest-match policy mitigates this by hashing on the full key tuple, but a genuinely hot key is unfixable at this layer — it needs range partitioning, salting, or a different algorithm.

Co-partitioning can require a second shuffle. If the left side is already Hash([a, b]) and the right side is Hash([a]), satisfying the join’s co-partitioning requirement means re-shuffling the right side onto (a, b) — even though the join’s own key requirement ((a, b)) was already met by the left. enforce_distribution_relationships (enforce_distribution.rs:1273-1330) picks a reference child that already satisfies the layout and rewrites the others to match, ranking candidates by estimated data size so the biggest child is the one you don’t want to move.

UnspecifiedDistribution is a licence to be greedy. It’s the signal “this operator would like more parallelism”. If an operator declares it but doesn’t actually benefit, the optimizer will happily add a round-robin shuffle that costs a full data copy for nothing. That’s why benefits_from_input_partitioning exists as a separate knob, and why operators like SortExec explicitly opt out.

Bottom line

The declarative-distribution design is what makes DataFusion’s optimizer composable. New operators don’t need new planner rules; they need one method:

 rust
 
fn input_distribution_requirements(&self) -> InputDistributionRequirements

The shuffle that follows is not an implementation detail of the operator that needed it — it’s a generic, reusable artifact (RepartitionExec) that the optimizer inserts, counts, and reuses across the plan. The cost of this abstraction is that the planner now has to reason about global consistency (aligned hash necessity, co-partitioning, subset policies) that individual operators never had to think about. The benefit is that a HashJoinExec doesn’t care whether its input came from a Parquet scan, a UNION, or a subquery — it just says “I need k co-located”, and the optimizer figures out the rest.

Top-K without a sort: how DataFusion keeps a bounded heap of the k smallest rows

Jay Hubbert — Wed, 21 Oct 2026 00:53:00 +0000

TL;DR: SortExec with fetch = Some(k) does not sort. It converts sort keys to Arrow’s row format, keeps the k best rows in a BinaryHeap, and evicts the current worst row whenever a better one arrives. Memory is O(k) rows instead of O(N), time is O(N log k) instead of O(N log N) — and the heap’s boundary is published as a dynamic filter so the scan underneath stops reading rows that cannot possibly make the cut.

The problem: `ORDER BY x LIMIT 3` on 100 million rows

This is the single most common analytical query shape after GROUP BY, and the naive plan for it is absurd:

 sql
 
 
-- datafusion/physical-plan/src/topk/mod.rs:71-79
/// ```sql
/// > explain SELECT customer_id, revenue FROM sales ORDER BY revenue DESC limit 3;
/// +--------------+----------------------------------------+
/// | plan_type | plan |
/// +--------------+----------------------------------------+
/// | logical_plan | Limit: 3 |
/// | | Sort: revenue DESC NULLS FIRST |
/// | | Projection: customer_id, revenue |
/// | | TableScan: sales |
/// +--------------+----------------------------------------+
/// ```

The sort materializes 100 million rows, sorts them (O(N log N) comparisons, O(N) memory), and then throws away 99,999,997 of them. The sort is the entire cost of the query and it is almost entirely wasted work.

The information-theoretic observation is simple: to know the top k rows you only need to compare each new row against the current k-th best. That’s a selection problem, not a sorting problem, and it admits an O(N log k) / O(k) solution.

DataFusion implements exactly that, and then goes further: it turns the running threshold into a filter that the scan can use.

Where it lives

There is no separate TopKExec node in this tree. Top-K is a mode of SortExec, selected at execute time by the presence of fetch:

 rust
 
 
// datafusion/physical-plan/src/sorts/sort.rs:1609-1645
match (sort_satisfied, self.fetch.as_ref()) {
 (true, Some(fetch)) => Ok(Box::pin(LimitStream::new(
 input, 0, Some(*fetch),
 BaselineMetrics::new(&self.metrics_set, partition),
 ))),
 (true, None) => Ok(input),
 (false, Some(fetch)) => {
 let filter = self.filter.clone();
 let mut topk = TopK::try_new(
 partition,
 input.schema(),
 self.common_sort_prefix.clone(),
 self.expr.clone(),
 *fetch,
 context.session_config().batch_size(),
 context.runtime_env(),
 &self.metrics_set,
 Arc::clone(&unwrap_or_internal_err!(filter)),
 )?;
 Ok(Box::pin(RecordBatchStreamAdapter::new(
 self.schema(),
 futures::stream::once(async move {
 while let Some(batch) = input.next().await {
 let batch = batch?;
 topk.insert_batch(batch)?;
 if topk.finished {
 break;
 }
 }
 drop(input);
 topk.emit()
 })
 .try_flatten(),
 )))
 }
 (false, None) => {
 // ... full ExternalSorter path
 }
}

Three cases, and note the third: with no fetch, SortExec builds an ExternalSorter (a spilling, multi-run merge sort). With a fetch, it builds a TopK. With a fetch and an already-sorted input, it just takes the first k rows.

The heap

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:925-953
/// This structure keeps at most the *smallest* k items, using the
/// [arrow::row] format for sort keys. While it is called "topK" for
/// values like `1, 2, 3, 4, 5` the "top 3" really means the
/// *smallest* 3 , `1, 2, 3`, not the *largest* 3 `3, 4, 5`.
///
/// Using the `Row` format handles things such as ascending vs
/// descending and nulls first vs nulls last.
struct TopKHeap {
 /// The maximum number of elements to store in this heap.
 k: usize,
 /// Storage for up at most `k` items using a BinaryHeap. Reversed
 /// so that the smallest k so far is on the top
 inner: BinaryHeap<TopKRow>,
 /// Storage the original row values (TopKRow only has the sort key)
 store: RecordBatchStore,
 /// The size of all owned data held by this heap
 owned_bytes: usize,
}

The “smallest k” wording is the key to the whole design. Arrow’s row format encodes a value so that byte-wise memcmp order equals SQL sort order — including descending order and nulls-first/last, because the SortOptions are baked into the encoder. So for ORDER BY revenue DESC, the smallest row bytes correspond to the highest revenue. “Keep the smallest k row-encodings” is exactly “keep the top k rows”, with no per-comparison special-casing for direction or nulls.

That is a genuinely nice trick. A naive implementation would have to write a custom comparator per sort direction, handle nulls, and probably get NULLS FIRST vs NULLS LAST wrong. Arrow’s RowConverter gives you memcmp semantics for free.

Each heap entry is deliberately not the row itself:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:1111-1126
/// Represents one of the top K rows held in this heap. Orders
/// according to memcmp of row (e.g. the arrow Row format, but could
/// also be primitive values)
///
/// Reuses allocations to minimize runtime overhead of creating new Vecs
#[derive(Debug, PartialEq)]
struct TopKRow {
 /// the value of the sort key for this row. This contains the
 /// bytes that could be stored in `OwnedRow` but uses `Vec<u8>` to
 /// reuse allocations.
 row: Vec<u8>,
 /// the RecordBatch this row came from: an id into a [`RecordBatchStore`]
 batch_id: u32,
 /// the index in this record batch the row came from
 index: usize,
}

So the heap stores sort keys plus a pointer (batch id, row index) to the payload. The payload stays in the original RecordBatch, held alive by RecordBatchStore.

Admission: one comparison per row

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:515-535
fn find_new_topk_items(
 &mut self,
 items: impl Iterator<Item = usize>,
 batch_entry: &mut RecordBatchEntry,
) -> usize {
 let mut replacements = 0;
 let rows = &mut self.scratch_rows;
 for (index, row) in items.zip(rows.iter()) {
 match self.heap.max() {
 // heap has k items, and the new row is greater than the
 // current max in the heap ==> it is not a new topk
 Some(max_row) if row.as_ref() >= max_row.row() => {}
 // don't yet have k items or new item is lower than the currently k low values
 None | Some(_) => {
 self.heap.add(batch_entry, row, index);
 replacements += 1;
 }
 }
 }
 replacements
}

heap.max() returns self.inner.peek() — the largest of the k retained rows in row-byte order, which is the k-th best row and therefore the current admission threshold. A new row is admitted only if it is strictly smaller than that threshold. One memcmp per input row, and O(log k) work only for the rows that actually make the cut.

The eviction path is where the “reuse allocations” comment pays off:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:978-1016
fn add(
 &mut self,
 batch_entry: &mut RecordBatchEntry,
 row: impl AsRef<[u8]>,
 index: usize,
) {
 let batch_id = batch_entry.id;
 batch_entry.uses += 1;

 assert!(self.inner.len() <= self.k);
 let row = row.as_ref();

 // Reuse storage for evicted item if possible
 if self.inner.len() == self.k {
 let mut prev_min = self.inner.peek_mut().unwrap();

 // Update batch use
 if prev_min.batch_id == batch_entry.id {
 batch_entry.uses -= 1;
 } else {
 self.store.unuse(prev_min.batch_id);
 }

 // update memory accounting
 self.owned_bytes -= prev_min.owned_size();

 prev_min.replace_with(row, batch_id, index);

 self.owned_bytes += prev_min.owned_size();
 } else {
 let new_row = TopKRow::new(row, batch_id, index);
 self.owned_bytes += new_row.owned_size();
 // put the new row into the heap
 self.inner.push(new_row);
 }
}

Note peek_mut(): the heap’s worst element is overwritten in place with the new row’s bytes, reusing the Vec<u8> allocation. No allocation, no deallocation, no sift_down from scratch — BinaryHeap::peek_mut handles the reordering. In a steady-state top-K over millions of rows, this is the difference between a garbage-generating loop and an allocation-free one.

Also note the reference counting: batch_entry.uses is incremented when a row from that batch enters the heap and decremented when it leaves. A batch whose rows have all been evicted is dropped from the store entirely:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:1244-1253
/// Insert a record batch entry into this store, tracking its
/// memory use, if it has any uses
pub fn insert(&mut self, entry: RecordBatchEntry) {
 // uses of 0 means that none of the rows in the batch were stored in the topk
 if entry.uses > 0 {
 self.batches_size += get_record_batch_memory_size(&entry.batch);
 self.total_rows += entry.batch.num_rows();
 self.batches.insert(entry.id, entry);
 }
}

Emitting: gather the survivors

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:1024-1051
fn emit_with_state(&mut self) -> Result<(Option<RecordBatch>, Vec<TopKRow>)> {
 // generate sorted rows
 let topk_rows = std::mem::take(&mut self.inner).into_sorted_vec();

 if self.store.is_empty() {
 return Ok((None, topk_rows));
 }

 // The batch ids are not continuous, so `indices` below is built through
 // the store's "batch_id -> array_pos" mapping.
 let (record_batches, batch_id_array_pos) = self.store.positional();

 let indices: Vec<_> = topk_rows
 .iter()
 .map(|k| (batch_id_array_pos[&k.batch_id], k.index))
 .collect();

 // At this point `indices` contains indexes within the
 // rows and `input_arrays` contains a reference to the
 // relevant RecordBatch for that index. `interleave_record_batch` pulls
 // them together into a single new batch
 let new_batch = interleave_record_batch(&record_batches, &indices)?;

 Ok((Some(new_batch), topk_rows))
}

BinaryHeap::into_sorted_vec() gives the k survivors already in sorted order — one O(k log k) operation at the very end, not per batch. Then interleave_record_batch gathers the payloads from the (possibly many) retained batches into one output batch, in sorted order.

Compaction

The store holds whole RecordBatches, but only k of their rows are actually wanted. Over a long scan the store can accumulate many batches each contributing one or two surviving rows, wasting memory. maybe_compact fixes this:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:1053-1069
/// Compact this heap, rewriting all stored batches into a single
/// input batch
pub fn maybe_compact(&mut self) -> Result<()> {
 // Don't compact if there's only one batch (compacting into itself is pointless)
 if self.store.len() <= 1 {
 return Ok(());
 }

 let total_rows = self.store.total_rows;
 let num_rows = self.inner.len();

 // Compact when the store holds more than STORE_COMPACTION_RATIO x what
 // the compacted result would need. The multiplier avoids compacting when
 // the savings would be marginal.
 if total_rows <= num_rows * STORE_COMPACTION_RATIO {
 return Ok(());
 }

with

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:918-923
/// length in [`TopKHeap::maybe_compact`], every partition's retained rows
/// (heap slots, plus ties for `RANK`) in [`RecordBatchStore::compact`]. The
/// multiplier avoids compacting when the savings would be marginal.
const STORE_COMPACTION_RATIO: usize = 2;

So compaction fires only when the store holds more than twice as many rows as the heap actually needs — amortizing the rewrite cost against the savings, and never doing a pointless rewrite.

The dynamic filter: pushing the threshold down

The heap knows a threshold: “no row at or above this row-encoding can enter the top k”. DataFusion publishes that as a dynamic filter expression that the scan below can evaluate, so the scan can skip rows early:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:559-568
/// Update the filter representation of our TopK heap.
/// For example, given the sort expression `ORDER BY a DESC, b ASC LIMIT 3`,
/// and the current heap values `[(1, 5), (1, 4), (2, 3)]`,
/// the filter will be updated to:
///
/// ```sql
/// (a > 1 OR (a = 1 AND b < 5)) AND
/// (a > 1 OR (a = 1 AND b < 4)) AND
/// (a > 2 OR (a = 2 AND b < 3))
/// ```
fn update_filter(&mut self) -> Result<()> {

The predicate is built outside the lock, then installed with a double-check, because multiple partitions share one filter and any of them may tighten it:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:587-610
// Extract scalar values BEFORE acquiring lock to reduce critical section
let thresholds = boundary.threshold_values(&self.expr)?;

// Build the filter expression OUTSIDE any synchronization
let predicate = Self::build_filter_expression(&self.expr, &thresholds)?;
let new_threshold =
 boundary.threshold(self.encode_topk_common_prefix_row(boundary)?);

// update the threshold. Since there was a lock gap, we must check if it is still the best
// may have changed while we were building the expression without the lock
let mut filter = self.filter.write();
let still_needs_update = filter
 .shared_threshold
 .as_ref()
 .map(|current| new_threshold.is_more_selective_than(current))
 .unwrap_or(true);
if !still_needs_update {
 // some other thread updated the threshold to a better one while we
 // were building so there is no need to update the filter
 return Ok(());
}

Note the multi-column predicate construction: for ORDER BY a DESC, b ASC, the predicate on b is only applied when a equals the threshold’s a — otherwise a row with a better a would be wrongly rejected. The builder threads a running equality conjunction (prev_sort_expr) through the columns to get this right (topk/mod.rs:679-699).

Early termination

There’s an even stronger trick. If the input is already sorted on a common prefix of the top-K’s ordering, then once the last row of a batch is strictly past the heap boundary, no subsequent row can ever enter the top k — so the query can stop reading:

 rust
 
 
// datafusion/physical-plan/src/topk/mod.rs:708-717
/// If input ordering shares a common sort prefix with the TopK,
/// check if the computation can be finished early.
///
/// This is the case if the last row of the current batch is strictly
/// greater than either the shared dynamic-filter threshold prefix or the max
/// row in the local heap, comparing only on the shared prefix columns.
fn attempt_early_completion(&mut self, batch: &RecordBatch) -> Result<()> {
 let last_row_prefix = self.last_row_common_prefix(batch)?;
 self.finish_if_prefix_past_boundary(last_row_prefix.as_ref())
}

For SELECT day, ts FROM sensor ORDER BY day DESC, ts DESC LIMIT 10 where sensor is already sorted by day, the scan reads only the first few “days” of data and stops. The finished flag then breaks the input loop (sorts/sort.rs:1636-1638).

Parallel top-K

Top-K has an additional trick that only works because of how the plan is built: k rows per partition is enough, because the merge takes the global top k afterwards.

The optimizer’s parallelize_sorts phase rewrites CoalescePartitions → Sort into Sort(preserve_partitioning) → SortPreservingMerge:

 rust
 
 
// datafusion/physical-optimizer/src/utils.rs:54-58
let mut new_sort = SortExec::new(ordering, Arc::clone(&node.plan)).with_fetch(fetch);
if node.plan.output_partitioning().partition_count() > 1 {
 new_sort = new_sort.with_preserve_partitioning(true);
}

and

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/mod.rs:351-361
requirements = add_sort_above_with_check(requirements, sort_reqs, fetch)?;

let spm =
 SortPreservingMergeExec::new(sort_exprs, Arc::clone(&requirements.plan));
Ok(Transformed::yes(
 PlanWithCorrespondingCoalescePartitions::new(
 Arc::new(spm.with_fetch(fetch)),
 false,
 vec![requirements],
 ),
))

So the plan becomes:

text

SortPreservingMergeExec(fetch=k) ← global top k
 SortExec(fetch=k, preserve_partitioning=true) ← per-partition top k
 ├── partition 0 ─┐
 ├── partition 1 ├─ N parallel scans
 └── partition 2 ─┘

Each of the N partitions keeps only k rows, so total memory is N × k rows instead of the whole dataset — and the merge only has to merge N × k rows, not N full streams. This is exactly the win described in the EnsureRequirements module docs:

 text
 
 
/// Each input partition is sorted in parallel, then a `SortPreservingMergeExec`
/// at the top performs a cheap merge of pre-sorted streams. For TopK queries
/// (`fetch=K`), each parallel sort only keeps K rows per partition, so total
/// memory is `N × K` rather than coalescing the entire stream first.

The fetch is propagated to the merge as well, so the merge itself stops after emitting k rows and the upstream partitions get cancelled.

Sort pushdown: where the `fetch` travels

The fetch is a piece of metadata that the sort-pushdown rule threads through the plan. The rule lives in datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs and carries three things per node:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs:60-80
#[derive(Clone, Debug)]
pub struct ParentRequirements {
 ordering_requirement: Option<OrderingRequirements>,
 fetch: Option<usize>,
 /// The distribution required by the consumer above any SortExec we insert.
 /// When this is `SinglePartition` and the input has multiple partitions,
 /// `add_sort_above_with_distribution` wraps the sort in `SortPreservingMergeExec`.
 distribution_requirement: Distribution,
}

pub type SortPushDown = PlanContext<ParentRequirements>;

The fetch arithmetic is where the subtlety lives. A LIMIT with an OFFSET needs skip more input rows than it returns:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs:103-112
fn input_fetch(
 plan: &Arc<dyn ExecutionPlan>,
 parent_fetch: Option<usize>,
) -> Option<usize> {
 let fetch = min_fetch(plan.fetch(), parent_fetch)?;
 let skip = plan
 .downcast_ref::<GlobalLimitExec>()
 .map_or(0, |limit| limit.skip());
 Some(fetch.saturating_add(skip))
}

The comment explains the trap: min(fetch + skip, parent_fetch) would be too small when the parent fetch is the tighter bound, and using the bare fetch would turn LIMIT 10 OFFSET 5 into TopK(10) below the limit — yielding 5 result rows instead of 10.

Pushing a sort through a filter: refused, on purpose

The pushdown rule decides, per node, whether the parent’s ordering requirement can be pushed into the children. For most operators the answer is “yes, if the operator preserves input order”. For a FilterExec, the answer is a hard no:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs:637-646
} else if maintains_input_order.is_empty()
 || !maintains_input_order.iter().any(|o| *o)
 || plan.is::<RepartitionExec>()
 || plan.is::<FilterExec>()
 || pushdown_would_violate_requirements(&parent_required, plan.as_ref())
{
 // If the current plan is a leaf node or can not maintain any of the input ordering, can not pushed down requirements.
 // For RepartitionExec, we always choose to not push down the sort requirements even the RepartitionExec(input_partition=1) could maintain input ordering.
 // Pushing down is not beneficial
 Ok(None)
}

Returning None means “cannot push down” — and the caller then inserts a SortExec above the filter:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs:435-444
} else {
 // Can not push down requirements, add new `SortExec` (distribution-aware):
 sort_push_down = add_sort_above_with_distribution(
 sort_push_down,
 parent_requirement.into_single(),
 parent_fetch,
 &parent_distribution,
 );
 assign_initial_requirements(&mut sort_push_down);
}

This is the right call, and the reason is a nice illustration of “the optimizer must think about data volume, not just ordering”. Pushing a sort below a filter would mean sorting the unfiltered rows and then discarding most of them — strictly more work. Keeping the sort above the filter means filtering first and sorting only the survivors.

The same logic blocks pushing a limit below a filter:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs:455-460
// A parent fetch can be pushed through a plan only if the plan explicitly
// supports limit pushdown and does not change cardinality. For example, if
// LIMIT 2 is applied below a FilterExec that removes half the rows, only one
// row may remain. Applied after the filter, it correctly returns two rows
// when at least two are available.

FilterExec doesn’t advertise supports_limit_pushdown(), so a LIMIT cannot be pushed through it either. And there’s a second, subtler guard for operators that do support limit pushdown but change cardinality:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs:462-477
// UnionExec advertises `supports_limit_pushdown() == true` because it can
// forward a LIMIT k to each of its children—i.e. apply "LIMIT k" separately
// on each branch before merging them together.
//
// However, UnionExec's `cardinality_effect() == GreaterEqual` (it sums up
// all child row counts), so pushing a global TopK/LIMIT through it would
// break the semantics of "take the first k rows of the combined result."

So can_push_fetch_through requires both:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/sort_pushdown.rs:151-155
fn can_push_fetch_through(plan: &Arc<dyn ExecutionPlan>) -> bool {
 plan.supports_limit_pushdown()
 && matches!(plan.cardinality_effect(), CardinalityEffect::Equal)
}

Trade-offs and failure modes

The row-format conversion is O(N) no matter what. Even though the heap only does O(log k) work per admitted row, every row must be encoded into the row format first (self.row_converter.append(rows, &sort_keys)?, topk/mod.rs:471-473). That’s a full pass over the sort key columns, comparable in cost to a comparison sort’s first phase. The win comes from avoiding the O(N log N) comparison phase and the O(N) memory, not from avoiding all work.

k in the millions degenerates. The heap is O(log k) per admission and holds k rows. If k is a large fraction of N, a plain sort (which is O(N log N) but with excellent locality and no per-row encoding) may actually win. DataFusion doesn’t switch strategies based on k; the fetch is either present or not.

Ties are resolved arbitrarily, and differently across partitions. The admission test is row < max_row — strictly smaller. A row equal to the threshold is rejected. That means the k retained rows are “the k strictly-smallest rows”, and among equal rows the choice is arbitrary (it depends on arrival order). SQL leaves tie-breaking unspecified for LIMIT without ORDER BY on a unique key, so this is legal — but it means LIMIT 10 over rows with duplicate sort keys may return a different subset on different runs or with different partition counts. For deterministic results you need a total order in the ORDER BY clause.

The dynamic filter is a double-edged sword. It lets the scan skip rows, but building and updating the predicate costs a lock acquisition per tightening and a BinaryExpr tree rebuild. The code mitigates this by building the expression outside the lock and re-checking under it, and by only updating when the new threshold is strictly more selective.

Memory is O(k) rows but not O(k) bytes. The heap holds k keys plus a reference to the whole RecordBatch each key came from. If the k survivors are spread across k different 8192-row batches, the store holds k full batches — k × 8192 rows of payload — until compaction fires. That’s why maybe_compact exists, and why it fires at a ratio of 2 rather than 1: compaction is itself a rewrite, and doing it eagerly would cost more than it saves.

Early termination only works on a common prefix. attempt_early_completion compares only the shared prefix columns of the sort key. If the input ordering shares nothing with the top-K’s ordering (the common case), the check is a no-op and the query reads everything. The optimization pays off for time-series-ish data that arrives pre-sorted.

Bottom line

The top-K heap is a small, self-contained piece of systems engineering that pays for itself many times over. Three ideas are doing the work:

Encode once, compare with memcmp. Arrow’s row format turns “compare two sort keys according to these SortOptions” into “compare two byte strings”, with null ordering and direction already baked in.
Keep only the boundary. The heap’s worst element is the admission threshold, so the data structure that implements the selection algorithm is also the thing that implements the filter.
Publish the threshold. Turning the boundary into a dynamic filter expression means the rows that can’t win are never read from storage at all — the top-K operator stops being a consumer-only optimization and starts pruning the scan.

The result is a query plan where ORDER BY revenue DESC LIMIT 3 over a hundred million rows reads a few thousand rows and holds three in memory.

Execution on Opensource internals

A Tour of DataFusion's Physical Operators: The Ten That Matter and the Trick Each One Uses

1. DataSourceExec — the scan

2. FilterExec — the filter

3. ProjectionExec — the projection

4. RepartitionExec — the shuffle

5. HashJoinExec — the equi-join

6. SortExec — the sort

7. SortPreservingMergeExec — the merge

8. AggregateExec — the aggregation

9. WindowAggExec — the window function

10. CoalescePartitionsExec — the fan-in

The ones you will also see

How they compose

Why it matters

One take, N slices: how DataFusion's RepartitionExec splits a batch across shuffle partitions

The problem: a shuffle is a scatter-gather in disguise

The router: BatchPartitioner

Round robin: the batch is the unit, not the row

Hash: route with integers first

Multi-column keys: hash-then-combine, not hash-of-a-tuple

The payoff: one take, then slices

Why the slices don’t leak memory

Downstream: coalescing the shreds

The rest of the machinery, briefly

Trade-offs and failure modes

Bottom line

How DataFusion Stops Reading After Ten Rows: Limit Pushdown, Limit Absorption, and the Volcano Model's Batch Problem

Background: vectorized volcano execution model

Example

Two mechanisms

Plan Limit Absorption

Pushing into the scan

The preserve_order condition

The limit and the row filter

The optimizer rule

The catch

The general point

Window frames over preserved sort order: how DataFusion evaluates PARTITION BY and ORDER BY

The problem: a window function is three things at once

The declaration

The plan that results

Avoiding a redundant sort

The frame representation

The default frames

Two accumulators: growing frames vs sliding frames

The “no ORDER BY” special case

Two executors: WindowAggExec and BoundedWindowAggExec

The functions-window crate: three evaluation APIs

Trade-offs and failure modes

Bottom line

Operators declare what they need: how DataFusion's planner inserts the shuffle for you

The problem: who decides where data goes?

The vocabulary: Distribution vs Partitioning

Declaring the requirement

Who declares what

Satisfying the requirement

Phase 2a: the actual insertion

The hash_necessary alignment trick

Co-partitioning: making the two sides agree

Round robin for pure parallelism

Subset satisfaction

Trade-offs and failure modes

Bottom line

Top-K without a sort: how DataFusion keeps a bounded heap of the k smallest rows

The problem: ORDER BY x LIMIT 3 on 100 million rows

Where it lives

The heap

Admission: one comparison per row

Emitting: gather the survivors

Compaction

The dynamic filter: pushing the threshold down

Early termination

Parallel top-K

Sort pushdown: where the fetch travels

Pushing a sort through a filter: refused, on purpose

Trade-offs and failure modes

Bottom line

1. `DataSourceExec` — the scan

2. `FilterExec` — the filter

3. `ProjectionExec` — the projection

4. `RepartitionExec` — the shuffle

5. `HashJoinExec` — the equi-join

6. `SortExec` — the sort

7. `SortPreservingMergeExec` — the merge

8. `AggregateExec` — the aggregation

9. `WindowAggExec` — the window function

10. `CoalescePartitionsExec` — the fan-in

The router: `BatchPartitioner`

The `preserve_order` condition

Two executors: `WindowAggExec` and `BoundedWindowAggExec`

The `functions-window` crate: three evaluation APIs

The vocabulary: `Distribution` vs `Partitioning`

The `hash_necessary` alignment trick

The problem: `ORDER BY x LIMIT 3` on 100 million rows

Sort pushdown: where the `fetch` travels