Datafusion on Opensource internals

The Contract Every DataFusion Operator Signs: A Tour of ExecutionPlan's Declarative Methods

Jay Hubbert — Wed, 21 Oct 2026 09:25:00 +0000

TL;DR: DataFusion’s physical optimizer has no if plan is HashJoinExec branches. Instead, every operator answers a dozen declarative questions — “what distribution do your inputs need?”, “are your inputs already sorted?”, “can you absorb a LIMIT?”, “does a filter on your output reduce your output?” — and the optimizer combines the answers to decide where to put shuffles, which sorts to delete and which filters to push. It is a nice piece of API design, and it is the reason DataFusion can add a new operator without touching the optimizer.

Most query optimizers are a pile of special cases. if the plan is a HashJoin, do X; if it is a Sort, do Y. It works, and it is why those optimizers are hard to extend.

DataFusion’s physical optimizer takes the other approach. The ExecutionPlan trait has about a dozen declarative methods, and the optimizer is a small amount of logic that combines their answers.

The questions

Here is the full list of what an operator can be asked, from datafusion/physical-plan/src/execution_plan.rs:

 rust
 
 
fn required_input_distribution(&self) -> Vec<Distribution> { ... }
fn input_distribution_requirements(&self) -> InputDistributionRequirements { ... }
fn required_input_ordering(&self) -> Vec<Option<OrderingRequirements>> { ... }
fn maintains_input_order(&self) -> Vec<bool> { ... }
fn benefits_from_input_partitioning(&self) -> Vec<bool> { ... }
fn supports_limit_pushdown(&self) -> bool { ... }
fn with_fetch(&self, _limit: Option<usize>) -> Option<Arc<dyn ExecutionPlan>> { ... }
fn fetch(&self) -> Option<usize> { ... }
fn cardinality_effect(&self) -> CardinalityEffect { ... }
fn try_swapping_with_projection(&self, ...) -> Result<Option<Arc<dyn ExecutionPlan>>> { ... }
fn gather_filters_for_pushdown(&self, ...) -> Result<FilterDescription> { ... }
fn try_pushdown_sort(&self, ...) -> Result<Option<Arc<dyn ExecutionPlan>>> { ... }
fn with_preserve_order(&self, ...) -> Result<Arc<dyn ExecutionPlan>> { ... }

Each one is a question the optimizer asks. Here is what they mean.

“What distribution do your inputs need?”

 rust
 
fn required_input_distribution(&self) -> Vec<Distribution> { ... }

Distribution::SinglePartition, Distribution::HashPartitioned(exprs, n), Distribution::RangePartitioned(..). A hash join says “I need both my inputs hash-partitioned on the join key, in the same way”. The optimizer then inserts a RepartitionExec where that is not already true.

The newer, more expressive version is input_distribution_requirements, which can express co-partitioning:

 rust
 
 
/// `InputDistributionRequirements` describes what distribution an operator
/// requires from each child.
///
/// - [`Self::new`] describes independent per-child requirements.
/// - [`Self::co_partitioned`] additionally requires child partitions with the
/// same index to cover compatible key ranges.

The difference matters: “both sides hash-partitioned on the key” is satisfied by two independent repartitions. “Both sides range-partitioned on the key with the same split points” is not — the second side has to be cut at the same boundaries, which means it needs the first side’s boundaries.

“Are your inputs already sorted?”

 rust
 
fn required_input_ordering(&self) -> Vec<Option<OrderingRequirements>> { ... }

A SortExec says “my input must be sorted by these columns”. The optimizer compares that against the input’s declared output_ordering, and if the requirement is already satisfied, the sort is deleted.

And maintains_input_order answers the complementary question: “does this operator preserve the order of its input?” A FilterExec does; a HashJoinExec does not. Without that, the optimizer could not know whether it is safe to remove a sort.

“Can you absorb a LIMIT?”

 rust
 
 
fn supports_limit_pushdown(&self) -> bool { ... }
fn with_fetch(&self, _limit: Option<usize>) -> Option<Arc<dyn ExecutionPlan>> { ... }

A SortExec says yes, and with_fetch returns a copy of itself that will stop after N rows. An operator that cannot says no, and the optimizer leaves a LimitExec in place.

The default for with_fetch is None — an operator has to opt in to absorbing a limit. And an operator that says it supports limit pushdown but returns None from with_fetch is handled gracefully.

“Does a filter on your output reduce your output?”

 rust
 
fn cardinality_effect(&self) -> CardinalityEffect { ... }

CardinalityEffect::Equal means “the output has the same number of rows as the input” — so a filter pushed above this operator cannot reduce it. CardinalityEffect::Lower (or Unknown) means it might.

This is what stops the optimizer from pushing a filter through an operator where the filter would be a no-op. It is a small method that prevents a whole class of wrong optimisations.

“Can I swap you with a projection?”

 rust
 
fn try_swapping_with_projection(&self, ...) -> Result<Option<Arc<dyn ExecutionPlan>>> { ... }

This is the projection pushdown mechanism, and it is operator-specific in a good way: a ProjectionExec can say “yes, and here is the projection I would rather have” — which is how a projection gets folded into a scan, or a ColumnRewriter gets applied.

“What filters can I take?”

 rust
 
 
fn gather_filters_for_pushdown(&self, ...) -> Result<FilterDescription> { ... }
fn handle_child_pushdown_result(&self, ...) -> Result<FilterPushdownResult> { ... }

The two-phase filter pushdown negotiation. The parent offers its filters to the child; the child says which ones it can handle entirely, which ones it can handle partially, and which ones it cannot. The result is a FilterPushdownResult with the residual filters that must stay above.

The “partially” case is the interesting one: a Parquet scan can push a predicate into the reader and keep a residual filter to guarantee correctness. That is only possible because the negotiation returns “partially handled”, not just “yes” or “no”.

“Can I push a sort below you?”

 rust
 
fn try_pushdown_sort(&self, ...) -> Result<Option<Arc<dyn ExecutionPlan>>> { ... }

A FilterExec says “yes, put the sort below me and I will filter the sorted stream”. The optimizer then has to know that the filter does not change the order — which is what maintains_input_order is for.

The properties object

All of the declarative answers are collected once, at construction time, into a PlanProperties:

 rust
 
fn properties(&self) -> &Arc<PlanProperties>;

which holds the output partitioning, the output ordering, the boundedness and the equivalence properties. The reason is caching: these values never change for a given operator instance, so computing them once at construction beats recomputing them on every optimizer question.

Why it matters

The alternative — special-casing operators in the optimizer — has a familiar failure mode. Every new operator needs a branch in the optimizer, and every branch is a place where the special case and the general case can disagree. DataFusion’s approach means a new operator implements a trait and is immediately optimisable.

It also means the knowledge lives with the operator. A hash join knows what distribution it needs; a sort knows what ordering it needs; a filter knows what it can push down. The optimizer is a policy engine, not a catalogue of operator-specific hacks.

And the method list is a good map of what a modern query engine thinks matters: distribution, ordering, cardinality, limits, projections, filters, and the ability to preserve order. If you were writing an execution engine from scratch, that list is a good place to start.

A Tour of DataFusion's Physical Operators: The Ten That Matter and the Trick Each One Uses

Jay Hubbert — Wed, 21 Oct 2026 09:17:00 +0000

TL;DR: EXPLAIN a DataFusion query and you will see a dozen node names. Most of them are one of ten operators, and each of those ten is built around a single idea. This is a tour of that list: what the operator does, the trick it is built on, and where its own article in this series lives.

EXPLAIN output is a good way to learn an engine, and a bad way to understand it. The node names are opaque and the tree looks like it was assembled by a committee.

It was not. Every node in there is one of a small set of operators, and each of those has a specific idea at its core. Here is the tour.

1. `DataSourceExec` — the scan

The leaf. It knows how to turn a FileScanConfig into a stream of batches, and it is where the file-format-specific work happens: listing the object store, reading the Parquet footer, evaluating pruning predicates, decoding pages.

The trick: decide what to read before reading it. The footer is read first, and it contains enough statistics to skip entire row groups and entire files. The newer push decoder goes further: it interleaves byte-range fetches with decoding, so a filter that tightens at runtime can still prune the row groups that have not been fetched yet.

2. `FilterExec` — the filter

Evaluates a boolean predicate against each batch and emits the rows that pass.

The trick: the filter is pushable. It offers its predicate to the child, and the child may take it — entirely, partially, or not at all. The Parquet scan takes it partially: it evaluates the predicate against row-group statistics to skip data, and keeps a residual filter to guarantee correctness. The two-phase negotiation is gather_filters_for_pushdown / handle_child_pushdown_result.

3. `ProjectionExec` — the projection

Selects and computes output columns.

The trick: it is foldable. try_swapping_with_projection lets a projection be absorbed into the operator below, so the scan reads only the columns that survive — and the projection node disappears from the plan entirely. There is even an embedded variant, where the projection becomes a Vec<usize> inside the join, so the join never materialises the dropped columns at all.

4. `RepartitionExec` — the shuffle

Splits the input into N partitions, either round-robin (no coordination) or by hash (both sides agree without talking).

The trick: slicing, not copying. A batch is routed by computing row indices per partition, doing one take per column, and handing out RecordBatch::slice views. Round-robin skips the take entirely. The StrengthReducedU64 trick replaces a % with a mask or a reciprocal-multiply, because division is expensive and the modulus is constant.

5. `HashJoinExec` — the equi-join

The workhorse. Builds a hash table over the smaller side, probes it with the larger.

The trick: the build side is reused three times. It is the hash table, it is the payload, and it is the dynamic filter published to the probe side — so the probe side can skip rows that cannot match, using a membership test against the same table. There is also a perfect-hash variant: when the build key is a dense integer, the hash table is replaced by a flat array indexed by the key.

6. `SortExec` — the sort

The trick: two modes. Below a threshold, everything is concatenated and sorted in one go — no merge at all. Above it, the input is sorted into runs and merged, with the runs coalesced first when the sort key is a single column, because then the merge’s per-stream overhead dominates its per-comparison cost.

7. `SortPreservingMergeExec` — the merge

Merges N sorted streams into one sorted stream.

The trick: it does not sort. It keeps a cursor per stream and always takes the smallest head. When the streams are sorted ranges (which the range-partitioning machinery produces), the merge is a simple concatenation. And when there are too many runs to merge at once, the multi-level merge spills intermediate results and repeats.

8. `AggregateExec` — the aggregation

Groups and aggregates.

The trick: the accumulator is a state machine, split into update_batch / state / merge_batch / evaluate, which is what makes the partial/final two-phase split possible. And the group table itself can be skipped entirely when the input is already sorted by the grouping key — the group clustering optimisation replaces the hash table with Arrow’s partition kernel.

9. `WindowAggExec` — the window function

The trick: the frame is a state machine over a sorted stream. A ROWS frame is stateless (you know the offset); a RANGE frame needs state, because “the current tie group” has to be tracked. And the operator declares that it needs its input sorted, so the optimizer inserts a sort if there is not one.

10. `CoalescePartitionsExec` — the fan-in

Merges N partition streams into one.

The trick: it is not a computation. It is a promise — “the plan below has N partitions, but the caller wants one stream”. Which is what makes execute_stream on a multi-partition plan possible without pushing fan-in into user code.

The ones you will also see

A few more, for completeness:

GlobalLimitExec / LocalLimitExec — the limit, split so that the global one can be a barrier.
UnionExec — the union of N plans.
UnnestExec — expanding a list column into rows.
ScalarSubqueryExec — evaluating a subquery once and broadcasting the result.
CooperativeExec — the budget that keeps a stream from hogging a worker.
AnalyzeExec — the EXPLAIN ANALYZE wrapper.
BufferExec — decoupling a fast producer from a slow consumer.

How they compose

The interesting part is not the operators individually but how they negotiate. A HashJoinExec says “I need hash-partitioned input on the join key”. A RepartitionExec says “I can give you that”. The optimizer inserts the second between the first and whatever is below it, and if the input already satisfies the requirement, it inserts nothing.

The same negotiation happens for ordering (a SortExec asks for sorted input and is deleted if the input is already sorted), for limits (a SortExec can absorb a LIMIT and stop early), and for filters (a FilterExec pushes its predicate down and keeps a residual).

Which is why the plan you get is not the plan you asked for, and why EXPLAIN is worth reading.

Why it matters

DataFusion’s operator set is small — a few dozen types — and each one is a well-known idea implemented with unusual care. The parts that surprise you are all the same shape:

a statistical fact (min/max, hash, count) used to avoid work rather than to do work;
an asymmetry exploited (the build side is also the filter; the probe side is also the producer of the sort);
a boundary where the data is already in the right shape (a sorted input, a dense key range, a prefix group).

Once you learn to look for those three things, DataFusion’s plans stop looking like a committee’s compromise and start looking like a series of deliberate decisions.

The DataFusion Performance Cheat Sheet: The Config Knobs and Optimizer Rules That Actually Move the Needle

Jay Hubbert — Wed, 21 Oct 2026 09:09:00 +0000

TL;DR: datafusion/common/src/config.rs is a thousand lines of options and most of them you will never touch. The ones that matter fall into four groups: parallelism (target_partitions), pushdown (dynamic filters, filter pushdown, repartition_*), spilling (sort_spill_reservation_bytes, max_spill_file_size_bytes), and the algorithm choices (prefer_hash_join, enable_piecewise_merge_join, hash_join_inlist_pushdown_max_size). This is a tour of the ones with a real performance consequence, and what each one costs you.

DataFusion’s configuration surface is enormous, and that is both a feature and a trap. Every interesting optimisation has a switch, and every switch has a “correctness” mode and a “fast” mode, and the defaults are tuned for a workload that is not yours.

Here is the map.

Parallelism

 rust
 
 
/// Number of partitions for query execution. Increasing partitions can increase
/// ... 
pub target_partitions: usize, transform = ExecutionOptions::normalized_parallelism, default = get_available_parallelism()

target_partitions is the number of parallel tasks DataFusion will use. It defaults to the machine’s available parallelism, which is right for a dedicated box and wrong for a container with 2 CPUs’ worth of quota among 64 vCPUs.

The transform is normalized_parallelism, which means the value is clamped into a sane range rather than taken literally. Setting target_partitions = 1000 on a 4-core box will not give you 1000-way parallelism; it will give you as much as the normalisation allows.

The related knob is planning_concurrency, also normalised parallelism. It controls how many threads run the physical planner, not the query. On a service that plans many small queries, raising it can lower planning latency.

Two more that matter for joins:

 rust
 
 
/// The number of rows under which the hash join will use a single partition
pub hash_join_single_partition_threshold: usize, default = 4 * 1024 * 1024
pub hash_join_single_partition_threshold_rows: usize, default = 1024 * 128

Both the bytes and the rows thresholds must be exceeded before the join is partitioned. A join with a 5 MB build side stays in one partition — one hash table, no shuffle, no repartition of the probe side. For a small dimension table joined against a huge fact table, that is the difference between a local hash join and a distributed one.

Pushdown

This is where most of DataFusion’s recent performance work has gone.

 rust
 
 
/// When set to true attempts to push down dynamic filters generated by operators
/// (TopK, Join & Aggregate) into the file scan phase.
pub enable_dynamic_filter_pushdown: bool, default = true

The master switch. It suppresses the three more specific ones, which is worth knowing because the interaction is surprising:

The config will suppress enable_join_dynamic_filter_pushdown, enable_topk_dynamic_filter_pushdown & enable_aggregate_dynamic_filter_pushdown. So if you disable enable_topk_dynamic_filter_pushdown, then enable enable_dynamic_filter_pushdown, the enable_topk_dynamic_filter_pushdown will be overridden.

The example in the doc is the one worth remembering:

For example, for a query such as SELECT * FROM t ORDER BY timestamp DESC LIMIT 10, the optimizer will attempt to push down the current top 10 timestamps that the TopK operator references into the file scans. This means that if we already have 10 timestamps in the year 2025 any files that only have timestamps in the year 2024 can be skipped / pruned at various stages in the scan.

That is the single highest-leverage optimisation in the list for time-partitioned data, and it is on by default.

Then the repartition switches, which control where shuffles are inserted:

 rust
 
 
pub repartition_aggregations: bool, default = true
pub repartition_joins: bool, default = true
pub repartition_file_scans: bool, default = true
pub repartition_windows: bool, default = true
pub repartition_sorts: bool, default = true

Each of these inserts a RepartitionExec when the operator declares it needs a particular distribution. Turning them off makes plans simpler and can help on a single machine — but they are the mechanism by which DataFusion gets parallelism at all.

And the round-robin one, which is a different kind of repartition:

 rust
 
 
/// When set to true, the physical plan optimizer will try to add round robin
/// repartitioning to increase parallelism to leverage more CPU cores
pub enable_round_robin_repartition: bool, default = true

Round-robin repartitioning is not a shuffle — it sends batch i to partition i % N, with no coordination. It is how DataFusion increases parallelism for a pipeline that has no natural partitioning (a filter over a scan, say). It is nearly free, and it is the difference between using one core and using all of them.

Spilling

 rust
 
 
/// Sort spill reservation size - the amount of memory reserved for the
/// sort's spill buffer
pub sort_spill_reservation_bytes: usize, default = 10 * 1024 * 1024

/// The maximum size of a spill file
pub max_spill_file_size_bytes: ConfigNonZeroUsize, default = non_zero_usize_default(128 * 1024 * 1024)

sort_spill_reservation_bytes is how much memory the sort keeps in reserve for its own spill buffer before it gives up and starts writing runs to disk. A larger value means fewer, bigger runs and a shallower merge tree; a smaller value means more runs and a deeper merge tree. The default of 10 MB is small — if your sorts are spilling, raising it usually helps.

max_spill_file_size_bytes caps the size of a single spill file, which bounds the largest batch a merge has to hold. This is the knob that interacts with the skewed-run re-spilling we looked at: a smaller cap means less memory pressure per merge, at the cost of more files.

There are also:

 rust
 
 
/// Compression to use for spilled data
pub spill_compression: SpillCompression, default = SpillCompression::Uncompressed

Uncompressed by default, which is the right choice for a temporary file you are about to read back — the CPU cost of compression usually exceeds the I/O saving on a local disk. On a slow network filesystem, or with Zstd, it can go the other way.

The join choices

 rust
 
 
/// Prefer hash join over sort merge join
pub prefer_hash_join: bool, default = true

Hash joins are the default because they are more predictable. A sort-merge join needs both sides sorted, and the sort is usually more expensive than the join.

 rust
 
 
/// If the join is small enough, use a single partition
pub hash_join_single_partition_threshold: usize, default = 4 * 1024 * 1024

Covered above.

 rust
 
 
/// The maximum size of the IN list to push down
pub hash_join_inlist_pushdown_max_size: usize, default = 128 * 1024
pub hash_join_inlist_pushdown_max_distinct_values: usize, default = 150

These control the hash join’s dynamic filter strategy — whether the build side’s keys are published as an IN list or as a hash-table membership test. The size limit is in bytes; the distinct-values limit is a separate cap of 150. A build side with 200 distinct keys goes to the hash-membership strategy even if it is small.

This matters because the IN list strategy has to copy the keys. For byte keys (strings), the implementation explicitly disables the IN list path so the filter borrows the hash table instead.

 rust
 
pub allow_symmetric_joins_without_pruning: bool, default = true

The symmetric hash join builds a hash table on both sides. It trades memory for parallelism, and this flag says whether to use it when the pruning opportunity is not there.

The Parquet knobs

 rust
 
 
/// Enable reading the page index (column index / offset index)
pub enable_page_index: bool, default = true
/// Enable pruning
pub pruning: bool, default = true
/// Read bloom filters
pub bloom_filter_on_read: bool, default = true
/// Push filters into the reader
pub pushdown_filters: bool, default = false

enable_page_index and bloom_filter_on_read are the ones that matter. The page index lets DataFusion skip pages within a row group, not just row groups, which is a big win on wide files. Bloom filters let it skip files entirely for a point lookup.

pushdown_filters defaults to false, and that surprises people. The doc explains why the default is conservative: pushing a filter into the reader means the reader evaluates it per row with its own semantics, which can differ from SQL’s. DataFusion keeps a residual filter for correctness, and enabling the pushdown without it would be wrong.

 rust
 
 
/// Maximum size of an IN list that the pruning predicate will rewrite into
/// per-value statistics checks. Longer lists skip container-level pruning.
pub max_in_list_size: usize, default = 20

Small, and it is a safety limit, not a performance one: rewriting a huge IN list into per-value statistics checks would be slow.

The sort and aggregate knobs

 rust
 
 
/// The optimizer will replace Filter(rn<=K) → Window(ROW_NUMBER) → Sort
/// patterns with a PartitionedTopKExec ...
/// When the window partition key has low cardinality, enabling this optimization
/// can improve performance. However, for high cardinality keys, it may
/// cause regressions in both memory usage and runtime.
pub enable_window_topn: bool, default = false

Off by default, and the doc explains the trade-off exactly: it is a win for low-cardinality partition keys and a regression for high-cardinality ones, because you keep K rows per partition and the number of partitions is the problem.

 rust
 
 
/// The optimizer will push TopK (Sort with fetch) below hash repartition
/// when the partition key is a prefix of the sort key, reducing data volume
/// before the shuffle.
pub enable_topk_repartition: bool, default = true

A nice one: if you are sorting by (a, b) and repartitioning by a, you can do the top-k before the shuffle, so the shuffle carries less data.

 rust
 
 
/// The optimizer will push a limit operation into grouped aggregations
/// which have no aggregate expressions, as a soft limit, emitting groups
/// once the limit is reached, before all rows in the group are read.
pub enable_distinct_aggregation_soft_limit: bool, default = true

SELECT DISTINCT with a LIMIT can stop early — a DISTINCT is a grouping, and once you have emitted N groups you are done.

The optimizer knobs

 rust
 
 
/// Maximum number of optimizer passes
pub max_passes: usize, default = 3

Small, and that is a statement of confidence: the fixed-point detection we looked at does the real work, and three passes is usually enough to reach it.

 rust
 
 
/// Skip optimizer rules that fail
pub skip_failed_rules: bool, default = false

Off by default, and it should stay off. When a rule fails, the optimizer normally propagates the error. With this on, it logs a warning and continues — which is exactly how you get a query that returns subtly wrong results. The source code’s own comment says:

Note to future readers: if you see this warning it signals a bug in the DataFusion optimizer. Please consider filing a ticket

 rust
 
 
/// The default selectivity of a filter, used when statistics are unavailable
pub default_filter_selectivity: ConfigFilterSelectivity, default = filter_selectivity_default(20)

This is a planning knob that affects real decisions: with no statistics, a filter is assumed to select 20% of the rows. That number changes join strategy, join order, and whether a repartition is worth it. If your workload has wildly different selectivities (a 0.1% filter in one place, a 90% filter in another), and you have no statistics, this default is doing a lot of damage.

 rust
 
pub use_statistics_registry: bool, warn = "`use_statistics_registry` is deprecated and ignored; ..."

Deprecated and ignored. A good reminder to check the deprecation notes before tuning.

The catch

Most of these are workload-specific. The defaults are tuned for TPC-H-like workloads on a dedicated machine. Yours is not.
Several are correctness-sensitive. pushdown_filters, skip_failed_rules and enable_physical_uncorrelated_scalar_subquery can produce wrong answers if misused. The last one has an explicit warning in the source:

Note disabling this option is not recommended. It restores pre … behavior, which silently produces incorrect results for multi-row subqueries …
The knobs interact. enable_dynamic_filter_pushdown suppresses three others. target_partitions changes which repartitions are inserted, which changes the memory profile, which changes whether things spill.
Measure. EXPLAIN ANALYZE gives you per-operator metrics (rows_produced, elapsed_compute, mem_used), and the array_map_created_count metric tells you whether the perfect hash join fired. Tuning without metrics is guessing.

Why it matters

A query engine’s configuration surface is a map of its design decisions. Reading it tells you what the engine thinks is important — and for DataFusion, the interesting entries are the ones that are off by default, because those are the ones where the trade-off was close enough that the maintainers did not want to pick for you.

Writing Parquet Row Groups in Parallel: Column Chunks, Dictionary Pages and the Sort-Before-Write Trick

Jay Hubbert — Wed, 21 Oct 2026 09:01:00 +0000

TL;DR: The Parquet write path in DataFusion is a streaming, one-pass pipeline: batches flow in, get fanned out to per-column encoders, get sealed into row groups, and get concatenated into a file that is uploaded as it is produced. Row groups are sized by row count (and optionally by bytes), dictionary encoding and page statistics are configured once and inherited by the reader’s pruning machinery, and — the trick worth stealing — when the input plan is already sorted, the sort order is written into the Parquet footer so the reader gets the ordering for free instead of paying for a sort.

The shape of the write path

There are two entry points, and they are not the same thing.

plan_to_parquet is the simple one — one file per output partition, one writer task per partition, each task pulling batches from its partition’s stream and pushing them into an AsyncArrowWriter (writer.rs:41-72). The buffer it uses, objectstore_writer_buffer_size (10 MiB by default, datafusion/common/src/config.rs:1159), exists mostly so that a very large output file does not hit remote-endpoint limits on the size of a single upload. It is a transport knob, not a format knob.

The interesting path is ParquetSink (sink.rs), which is what COPY ... TO and INSERT INTO actually use. It receives a demultiplexed stream of batches tagged with the file they belong to (hive partitioning splits the output into many files), and it has two writer strategies.

Two writer strategies

The sequential path is parquet::arrow::AsyncArrowWriter: one file, one task, batches handed in order. It is used when allow_single_file_parallelism is false, or when content-defined chunking is enabled — CDC needs the chunker state that lives inside ArrowWriter and persists across row groups (sink.rs:304-308). The parallel path is DataFusion’s own code, and it is the interesting one.

Parallel column encoding inside one file

The idea: a Parquet row group is a set of independent column chunks. There is no reason to encode column a of row group 7 before column b of row group 7. So DataFusion spawns one task per leaf column and feeds them arrays through per-column channels:

 rust
 
 
// datafusion/datasource-parquet/src/sink.rs:630-645
for writer in col_writers.into_iter() {
 // Buffer size of this channel limits the number of arrays queued up for column level serialization
 let (send_array, receive_array) = mpsc::channel::<ArrowLeafColumn>(max_buffer_size);
 col_array_channels.push(send_array);

 let reservation =
 MemoryConsumer::new("ParquetSink(ArrowColumnWriter)").register(pool);
 let task = SpawnedTask::spawn(column_serializer_task(
 receive_array, writer, reservation, encoding_time.clone(),
 ));
 col_writer_tasks.push(task);
}

The channel depth is maximum_buffered_record_batches_per_stream (default 2, config.rs:1561-1566) — deliberately small, because this is a streaming plan and the memory budget is the scarce resource. The option’s comment says it plainly: the defaults are “tuned for minimum memory usage in a streaming execution plan”, and raising them “is likely worthwhile when writing out already in-memory data”.

A batch is then shredded into leaf arrays and pushed to the right channel:

 rust
 
 
// datafusion/datasource-parquet/src/sink.rs:684-695
let mut next_channel = 0;
for (array, field) in rb.columns().iter().zip(schema.fields()) {
 for c in compute_leaves(field, array)? {
 if col_array_channels[next_channel].send(c).await.is_err() {
 return Ok(());
 }
 next_channel += 1;
 }
}

compute_leaves is what makes this work for nested schemas: a List<Int32> column is one Arrow column but two Parquet leaves (the offsets and the elements), and each leaf needs its own encoder.

Row groups are sealed concurrently, concatenated in order

When a row group fills up, its column writers are closed and the finished column chunks are handed to a separate task, so the next row group can start encoding without waiting (sink.rs:777-817). The batch that straddles the boundary is split — rb.slice(0, rows_left) — and the remainder is carried into the next row group. That is why the row-group limit can be any number, not just a multiple of the batch size.

The finalizer joins the column tasks, closes each writer, and grows a memory reservation by the encoded size (sink.rs:712-727). The concatenator then writes them into the file, in the order the row groups were started, flushing to the object store when the shared buffer grows past a megabyte (sink.rs:859-876):

 rust
 
 
// datafusion/datasource-parquet/src/sink.rs:859-876
let mut rg_out = parquet_writer.next_row_group()?;
for chunk in serialized_columns {
 chunk.append_to_row_group(&mut rg_out)?;
 rg_reservation.free();

 let mut buff_to_flush = merged_buff.buffer.try_lock().unwrap();
 if buff_to_flush.len() > BUFFER_FLUSH_BYTES {
 object_store_writer.write_all(buff_to_flush.as_slice()).await?;
 buff_to_flush.clear();
 }
}
rg_out.close()?;

So the shape is: N column encoders running concurrently, R row groups in flight, and one writer that stitches the finished row groups into the file in order and ships bytes to the object store as it goes. The file is never materialized in memory, and the input is never read twice.

Row group sizing

Row group size is the single most consequential write-side decision, because it is the granularity at which the reader can prune. max_row_group_size defaults to 1M rows; max_row_group_bytes is None, and when set, bounds the row group by bytes instead, matching parquet.block.size in parquet-mr (config.rs:1494-1512).

The byte limit is the interesting one, and the reason it is None by default is a deliberate trade-off: the parallel writer ignores it. From the option’s own documentation: “Currently only honored when allow_single_file_parallelism is false; by default the parallel file writer ignores this limit.” The parallel writer computes its row-group boundary from max_row_group_row_count() alone (sink.rs:747-750), because bounding bytes in a design where row groups are encoded concurrently and asynchronously would require knowing the encoded size before the row group is finished — which is exactly the information that only exists after encoding.

Dictionary encoding and statistics: write-side decisions, read-side wins

The writer’s defaults come from ParquetOptions and are translated into WriterProperties in one place (datafusion/common/src/file_options/parquet_writer.rs:259-276): page size limit, write batch size, writer version, dictionary page size limit, statistics level, max row group rows and bytes, created_by, column index and statistics truncation lengths, data page row count limit, bloom filter switch, and the NDV-per-row-group switch.

The defaults that matter for a reader:

Option	Default	Reader-side consequence
`dictionary_enabled`	`true`	smaller column chunks; enables dictionary-aware reads
`dictionary_page_size_limit`	1 MiB	bounds the dictionary’s memory cost
`statistics_enabled`	`page`	the column index exists, so page-level pruning works
`statistics_truncate_length`	64	bounds are exact up to 64 bytes, then widened
`column_index_truncate_length`	64	same, for the page index
`data_pagesize_limit`	1 MiB	page granularity for pruning
`data_page_row_count_limit`	20 000	rows per page
`compression`	`zstd(3)`
`bloom_filter_on_write`	`false`	bloom pruning is unavailable unless you turn this on

That last row is the sharpest example of a write-time decision with a read-time payoff. DataFusion’s reader can prune row groups with bloom filters (row_group_filter.rs:460-508), but a bloom filter has to be in the file first, and bloom_filter_on_write defaults to false. Turning it on costs space and write time and buys row-group skipping for equality predicates — a trade that is worth it for selective high-cardinality columns and not worth it for low-cardinality ones (where the min/max statistics already do the job).

Similarly, statistics_truncate_length = 64 is the reason that a string column’s row-group bounds are “Alp”..“Alq” rather than exact, and the reason full-match inference can fail on long strings (row_group_filter.rs:189-203).

The sort-before-write trick

Here is the trick worth stealing. Parquet has a sorting_columns field in its metadata: it declares that the file’s rows are sorted by these columns. DataFusion writes it when the input plan is already ordered, and reads it back to skip the sort.

Writing it (file_format.rs:645-668):

 rust
 
 
// Convert ordering requirements to Parquet SortingColumns for file metadata
let sorting_columns = if let Some(ref requirements) = order_requirements {
 let ordering: LexOrdering = requirements.clone().into();
 let writer_schema = get_writer_schema(&conf);
 // In cases like `COPY (... ORDER BY ...) TO ...` the ORDER BY clause
 // may not be compatible with Parquet sorting columns (e.g. ordering on `random()`).
 // So if we cannot create a Parquet sorting column from the ordering requirement,
 // we skip setting sorting columns on the Parquet sink.
 lex_ordering_to_sorting_columns(&ordering, conf.output_schema(), &writer_schema)
 .ok()
 .filter(|columns| !columns.is_empty())
} else {
 None
};

The order_requirements are not invented by the writer — they come from the plan: input_exec.properties().output_ordering() (datafusion/core/src/physical_planner.rs:930-940).

So: if the plan that feeds the sink already has an ordering — because the user wrote COPY (SELECT ... ORDER BY x) TO ..., or because an upstream operator produces sorted output — the writer records it in the footer. It does not impose an order; it only documents one it was given. That distinction is what makes it safe: the metadata is a claim the writer can make because it saw the data in that order. The conversion is conservative in the right way: only plain column references can become sorting columns (metadata.rs:1145-1151).

Reading it back, ordering_from_parquet_metadata (metadata.rs:1209-1236) turns the footer’s sorting columns into a LexOrdering, which the source advertises as its output ordering — and that is what lets the optimizer delete the SortExec above the scan.

The read side has a second, related trick for files that are not declared sorted: reorder the row groups by their min statistics, so a TopK query finds good row groups first (access_plan.rs:727-749). Row groups are sorted lexicographically by the min values of the longest prefix of the sort order made of plain columns present in the file schema, with secondary keys breaking ties when the leading column’s min is equal across row groups — without them the reorder is a no-op on such files and the TopK dynamic filter converges only as fast as disk order allows.

This is the “sort before write” idea from the other direction: if you cannot afford to sort at write time, you can at least approximate the order at read time using statistics you already paid for, and let the TopK’s dynamic filter prune the rest.

The catch

The parallel writer bypasses ArrowWriter. Content-defined chunking is therefore disabled with it — the chunker state lives in ArrowWriter and persists across row groups (sink.rs:304-308). CDC forces the sequential path.
max_row_group_bytes does not work with parallel writing. Setting it and leaving allow_single_file_parallelism = true gives you a row group limit that is silently not applied. This is documented in the option, but it is the kind of footgun that costs an afternoon.
Ordering metadata is a promise, not a check. If the input plan claims an ordering it does not have, the footer will lie, and the reader will happily skip a sort it should have performed. DataFusion mitigates this by deriving the ordering from the plan rather than from the data, so the promise inherits the plan’s correctness.
Bloom filters cost space and write time for a benefit that only materialises when the reader also has bloom pruning enabled (datafusion.execution.parquet.bloom_filter_on_read).
Statistics truncation is a silent precision loss. 64 bytes is enough for most keys and not enough for long strings; the effect is that pruning gets weaker, never wrong — but “never wrong” here comes from the untrusted-stats machinery on the read side, which is a separate piece of machinery you now depend on.

The general lesson of the write path is that Parquet’s read performance is decided at write time. Row group size, dictionary encoding, statistics granularity, bloom filters, and sorting metadata are all write-side choices that the reader either cashes in or does not. DataFusion exposes all of them as configuration, which is the minimum you owe your future queries.

Signatures, Volatility, and the Type-Coercion Lattice: How DataFusion's UDF Metadata Drives the Optimizer

Jay Hubbert — Wed, 21 Oct 2026 08:53:00 +0000

TL;DR: Signature is the metadata contract between a UDF and the engine. TypeSignature says which argument types have implementations; the coercion lattice turns “user called cos(int_col)” into “cast to Float64 and call the Float64 implementation”. Volatility is the second half of the contract: Immutable/Stable functions may be constant-folded at plan time and reordered by the optimizer; Volatile ones may not, and the optimizer goes out of its way to keep them from being duplicated, hoisted or pushed below an aggregate. And because ScalarUDF wraps Arc<dyn ScalarUDFImpl>, the virtual call happens once per batch, not once per row.

Two problems one struct solves

A scalar UDF in DataFusion has to answer three questions before it can run:

What argument types do you implement? (cos handles Float32/Float64; make_array handles anything.)
What do I do with the types I don’t implement? (Cast the input? Error?)
May I call you twice and get the same answer? (abs(-1) yes, random() no.)

The first two are the “type” question and the third is the “time” question. They are orthogonal, and DataFusion keeps them in one struct because they are always needed together — during planning, during coercion, and during optimization:

 rust
 
 
// datafusion/expr-common/src/signature.rs:1410
pub struct Signature {
 /// The data types that the function accepts. See [TypeSignature] for more information.
 pub type_signature: TypeSignature,
 /// The volatility of the function. See [Volatility] for more information.
 pub volatility: Volatility,
 /// Optional parameter names for the function arguments.
 pub parameter_names: Option<Vec<String>>,
}

The constructors make the common cases one-liners (signature.rs:1426-1540): Signature::exact, Signature::uniform, Signature::variadic, Signature::user_defined, Signature::one_of, Signature::comparable, Signature::coercible, Signature::numeric, Signature::string, Signature::nullary, Signature::any, Signature::variadic_any, plus the array-specialised ones (array_and_element, arrays, array_and_index, …).

`TypeSignature`: arity × type shape

 rust
 
 
// datafusion/expr-common/src/signature.rs:165
pub enum TypeSignature {
 Variadic(Vec<DataType>), // N args, each one of these types
 UserDefined, // the UDF does its own coercion
 VariadicAny, // N args, any types
 Uniform(usize, Vec<DataType>), // exactly N args, all the same type
 Exact(Vec<DataType>), // exactly N args, these types, in order
 Coercible(Vec<Coercion>), // per-argument coercion rules
 Comparable(usize), // N args coerced to one comparable type
 Any(usize), // N args, any types
 OneOf(Vec<TypeSignature>), // try each in order
 ArraySignature(ArrayFunctionSignature),
 Numeric(usize),
 String(usize),
 Nullary, // no arguments
}

The doc comment on the enum is the important part: TypeSignature does not define the types a user may call the function with. It defines the types you wrote code for. Everything else is the engine’s problem, and the engine solves it by inserting casts.

OneOf deserves special mention because it is how a function supports genuinely different shapes. approx_percentile_cont uses it for the 2-argument and 3-argument forms (datafusion/functions-aggregate/src/approx_percentile_cont.rs:137-172):

 rust
 
 
let signature = Signature::one_of(
 vec![
 // 2 args - numeric, percentile (float)
 TypeSignature::Coercible(vec![
 Coercion::new_implicit(
 TypeSignatureClass::Float,
 vec![TypeSignatureClass::Numeric],
 NativeType::Float64,
 ),
 Coercion::new_implicit_native(
 logical_float64(),
 vec![TypeSignatureClass::Numeric],
 ),
 ]),
 // 3 args - numeric, percentile (float), number of centroids (integer)
 // ...
 ],
 Volatility::Immutable,
);

The coercion lattice

The interesting question is: given cos(some_int32_column), what happens?

The entry point is fields_with_udf (datafusion/expr/src/type_coercion/functions.rs:110), which is called during planning for every function call:

 rust
 
 
// datafusion/expr/src/type_coercion/functions.rs:136
let valid_types = get_valid_types_with_udf(type_signature, &current_types, func)?;
if valid_types.iter().any(|data_type| data_type == &current_types) {
 return Ok(current_fields.to_vec()); // already exact: no casts
}
let updated_types =
 try_coerce_types(func.name(), valid_types, &current_types, type_signature)?;

Three steps: enumerate the valid type combinations for the signature, check whether the call already matches one, otherwise find a combination we can cast to.

get_valid_types_with_udf (functions.rs:532) special-cases two signatures: UserDefined delegates straight to the UDF’s own coerce_types hook, and OneOf tries each alternative and collects the successes (so one_of with two 3-arg forms can match either). Everything else goes to get_valid_types, which expands the signature into concrete DataType vectors.

Then try_coerce_types (functions.rs:494) picks a target. It has two paths, and the distinction matters:

 rust
 
 
// datafusion/expr/src/type_coercion/functions.rs:474
fn is_well_supported_signature(type_signature: &TypeSignature) -> bool {
 match type_signature {
 TypeSignature::OneOf(type_signatures) => type_signatures.iter().all(is_well_supported_signature),
 TypeSignature::UserDefined | TypeSignature::Numeric(_) | TypeSignature::String(_)
 | TypeSignature::Coercible(_) | TypeSignature::Any(_) | TypeSignature::Nullary
 | TypeSignature::Comparable(_) => true,
 TypeSignature::Variadic(_) | TypeSignature::VariadicAny | TypeSignature::Uniform(_, _)
 | TypeSignature::Exact(_) | TypeSignature::ArraySignature(_) => false,
 }
}

The “well supported” signatures (the new Coercible/Numeric/String/ Comparable family) have already resolved to a single exact target type, so coercion is a direct cast. The legacy signatures (Exact, Uniform, Variadic) take the old path, which searches for any valid combination and casts to it — more expensive, more magic, and the source of most of the “why did my function get a CAST I didn’t ask for” confusion in DataFusion.

Finding the common type

The actual lattice lives in datafusion/expr-common/src/type_coercion/binary.rs. type_union_resolution (:708) computes the common type of a set of argument types, Postgres/DuckDB style:

 rust
 
 
// datafusion/expr-common/src/type_coercion/binary.rs:719
// If all the data_types are null, return string
if data_types.iter().all(|t| t == &DataType::Null) {
 return Some(DataType::Utf8View);
}
// Ignore Nulls, if any data_type category is not the same, return None
// ...

and the pairwise step (:773) is a cascade of fallbacks:

 rust
 
 
// datafusion/expr-common/src/type_coercion/binary.rs:873
binary_numeric_coercion(lhs_type, rhs_type)
 .or_else(|| list_coercion(...))
 .or_else(|| map_coercion(...))
 .or_else(|| temporal_coercion_nonstrict_timezone(...))
 .or_else(|| string_coercion(...))
 .or_else(|| null_coercion(...))
 .or_else(|| string_numeric_coercion(...))
 .or_else(|| binary_coercion(...))

That is the “lattice”: numeric widening first (i32 + i64 → i64), then lists, then maps, then temporals, then strings, then the escape hatches that let a string be read as a number or a binary be read as a string. Order matters, and that order encodes a priority: it mirrors what a human would guess.

The Coercion enum (signature.rs:1164) is the modern, explicit way to say “this argument should end up as a Float64, and it may be implicitly cast from any numeric type”:

 rust
 
 
// datafusion/expr-common/src/signature.rs:1164
pub enum Coercion {
 Exact { desired_type: TypeSignatureClass, encoding_preservation: EncodingPreservation },
 Implicit {
 desired_type: TypeSignatureClass,
 implicit_coercion: ImplicitCoercion, // allowed_source_types + default_casted_type
 encoding_preservation: EncodingPreservation,
 },
}

TypeSignatureClass (signature.rs:371) is the coarse lattice of type families — Any, Timestamp, Time, Interval, Duration, Native(LogicalTypeRef), Integer, Float, Decimal, Numeric, Binary. Numeric is the union of Integer | Float | Decimal; Integer and Float are the two ways to spell “any numeric that is not a decimal”. Writing Coercion::new_implicit(TypeSignatureClass::Float, vec![TypeSignatureClass::Numeric], NativeType::Float64) says: “accept a float; if the user gave me an int or a decimal, cast it to Float64”. That is one line replacing a page of match arms.

`Volatility`: the promise about time

 rust
 
 
// datafusion/expr-common/src/signature.rs:56
pub enum Volatility {
 /// Always returns the same output when given the same input.
 Immutable,
 /// May return different values given the same input across different
 /// queries but must return the same value for a given input within a query.
 Stable,
 /// May change the return value from evaluation to evaluation.
 Volatile,
}

Three levels, and the difference between Immutable and Stable is subtle but real: Immutable says “same input, same output, forever” (so abs(-1) is folded to 1 at plan time), Stable says “same input, same output within this query” (now() is constant for the duration of one query but changes between queries), and Volatile says “no promises at all” (random()).

The optimizer uses this in two distinct ways.

1. Constant folding. The simplifier’s ConstEvaluator refuses to evaluate volatile functions at plan time:

 rust
 
 
// datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:617
fn volatility_ok(volatility: Volatility) -> bool {
 match volatility {
 Volatility::Immutable => true,
 // Values for functions such as now() are taken from ExecutionProps
 Volatility::Stable => true,
 Volatility::Volatile => false,
 }
}

can_evaluate (expr_simplifier.rs:626) then refuses the whole subtree containing a volatile node, so 1 + random() is not folded and 1 + abs(-1) is. Folding is not just a nicety: it turns a per-row computation into a constant, which lets the optimizer push predicates around it, drop columns, and use statistics.

2. “May I duplicate or move this expression?” Plenty of optimizations are only valid if the expression means the same thing wherever it ends up. DataFusion encodes that check with Expr::is_volatile:

 rust
 
 
// datafusion/expr/src/expr.rs:2208
pub fn is_volatile_node(&self) -> bool {
 let volatility = match self {
 Expr::ScalarFunction(func) => func.func.signature().volatility,
 Expr::AggregateFunction(func) => func.func.signature().volatility,
 // ...
 };
 volatility == Volatility::Volatile
}

Note the distinction between is_volatile_node (this node only) and is_volatile (:2231, the whole subtree). Rewrites use the subtree version, because moving a subtree that contains a volatile node is just as wrong as moving the volatile node itself.

Concretely, the filter pushdown refuses to push a predicate into the grouping keys when the key is volatile:

 rust
 
 
// datafusion/optimizer/src/push_down_filter.rs:1117
// We can push down Predicate which in groupby_expr.
// Volatile group keys are excluded: below the aggregate, the
// predicate would evaluate the key again and see a different
// value than the one used for grouping.
let non_volatile_group_exprs: Vec<Expr> = agg
 .group_expr
 .iter()
 .filter(|expr| !expr.is_volatile())
 .cloned()
 .collect();

That is a correctness fix, not a pessimization: GROUP BY random() would group by one value and then filter on a different random value if the predicate were pushed down. Similarly, outer-join elimination only fires when the join condition is not volatile (datafusion/optimizer/src/eliminate_join.rs:1036), and common-subexpression elimination skips volatile output columns (datafusion/optimizer/src/extract_leaf_expressions.rs:109).

The cost of getting this wrong is asymmetric and brutal:

Declare a non-deterministic function Immutable → the planner folds it to a literal → every row of the query gets the same value. Silently wrong.
Declare a deterministic function Volatile → no folding, no predicate pushdown below aggregates, no CSE, no join elimination. Correct but mysteriously slow, and the profiler will not tell you why.

The rule of thumb in the doc comment is the right one: “You should always define your function to have the strictest possible volatility to maximize performance and avoid unexpected results.”

The dispatch cost: one virtual call per batch, not per row

ScalarUDF is a thin newtype over a trait object:

 rust
 
 
// datafusion/expr/src/udf.rs:86
pub struct ScalarUDF {
 inner: Arc<dyn ScalarUDFImpl>,
}

The vtable lookup happens in ScalarFunctionExpr::evaluate (datafusion/physical-expr/src/scalar_function.rs:236), which evaluates each child expression once and hands the whole batch to the function:

 rust
 
 
// datafusion/physical-expr/src/scalar_function.rs:236
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 let args = self.args.iter().map(|e| e.evaluate(batch)).collect::<Result<Vec<_>>>()?;
 // ...
 let output = self.fun.invoke_with_args(ScalarFunctionArgs {
 args,
 arg_fields,
 number_rows: batch.num_rows(),
 return_field: Arc::clone(&self.return_field),
 config_options: Arc::clone(&self.config_options),
 })?;
 // ...
}

So the dynamic dispatch is O(1) per batch; the body of the function works on ArrayRefs and uses Arrow’s vectorized kernels. A UDF written naively — one ScalarValue in, one ScalarValue out — would instead be O(rows) virtual calls with a ScalarValue allocation per row, and would be the slowest thing in the engine.

There is one more escape hatch worth knowing about, for when the optimizer needs to reason about a specific function rather than a name:

 rust
 
 
// datafusion/physical-expr/src/scalar_function.rs:165
pub fn try_downcast_func<T>(expr: &dyn PhysicalExpr) -> Option<&ScalarFunctionExpr>
where T: ScalarUDFImpl

It matches on the concrete ScalarUDFImpl type, not the function name, so GetFieldFunc can be recognised even if a user registered a different function under the same name. The parquet reader uses it to recognise get_field and file_row_index/input_file_name (datafusion/datasource-parquet/src/projection_read_plan.rs:418,511).

The registry

All of this metadata hangs off the function object, and the functions live in a registry:

 rust
 
 
// datafusion/expr/src/registry.rs:40
pub trait FunctionRegistry {
 fn udf(&self, name: &str) -> Result<Arc<ScalarUDF>>;
 fn udaf(&self, name: &str) -> Result<Arc<AggregateUDF>>;
 fn udwf(&self, name: &str) -> Result<Arc<WindowUDF>>;
 fn register_udf(&mut self, _udf: Arc<ScalarUDF>) -> Result<Option<Arc<ScalarUDF>>>;
 // ...
}

datafusion/functions/src/lib.rs:166 builds the default set (all_default_functions, chaining the per-package functions() lists) and :180 (register_all) installs them into any registry, logging an overwrite if a name collides. Because the registry hands out Arc<ScalarUDF>, the Signature is computed once and shared by every plan that uses the function — the coercion lattice is a planning-time cost, paid once per query, not a per-row cost.

Trade-offs

Metadata is a promise you can break. Signature is unchecked by the compiler. A function that lies about its arity or its volatility produces wrong answers, not errors.
The coercion lattice is a planning-time tax. Every function call enumerates valid type combinations and searches for a cast path. The OneOf and legacy Exact/Uniform paths are the expensive ones; the modern Coercible/Numeric/String/Comparable signatures exist partly to make that search trivial.
OneOf order is semantics. The first matching alternative wins, so reordering the alternatives in a signature can change which implementation runs — and therefore the result type and the accuracy.
Volatility::Stable is under-used. now() is Stable; most UDF authors reach for Immutable or Volatile and never think about the middle rung, which is exactly the rung that says “constant for this query”.
The Arc<dyn ScalarUDFImpl> boundary is a real, if small, cost. It is paid once per batch per function node. For a query that evaluates a function on 8 KiB batches, that is measurable but small; for one that evaluates it on million-row batches, it is noise. Which is another way of saying: keep the batch big, and the dispatch model takes care of itself.

Cooperative Budgets: How DataFusion Stops One Operator From Starving the Whole Runtime

Jay Hubbert — Wed, 21 Oct 2026 08:45:00 +0000

TL;DR: DataFusion treats “one poll_next does a lot of work” as a scheduling bug. Every stream is wrapped in a CooperativeStream that spends one unit of Tokio’s 128-unit task budget per record batch and yields when the budget is exhausted. The EnsureCooperative optimizer rule inserts those wrappers automatically at plan leaves and exchange points. Between the N producer tasks and the M output channels of a shuffle, a global gate keeps at least one channel empty, so the exchange is deadlock-free; the memory bound comes from the pool, not the gate.

The starvation problem

Rust’s Stream trait is pull-based: poll_next is called, and the stream either returns Ready(Some(batch)) or Pending. There is no rule that says poll_next must be cheap. An operator that computes an aggregation over a large dataset can spend milliseconds inside a single poll_next call.

On a Tokio runtime that matters twice over:

Other tasks starve. Tokio’s work-stealing scheduler can only run other tasks when the current one yields. A query plan with 16 partitions, all running on a 4-core box, serializes into 4 long-running tasks.
Cancellation stops working. DataFusion cancels queries by dropping the output stream, which requires the tasks to reach an await point. A task that never yields is uncancellable.

The module doc in datafusion/physical-plan/src/coop.rs:21-30 states the problem plainly:

 rust
 
 
//! A single call to `poll_next` on a top-level [`Stream`] may potentially perform a lot of work
//! before it returns a `Poll::Pending`. Think for instance of calculating an aggregation over a
//! large dataset.
//!
//! If a `Stream` runs for a long period of time without yielding back to the Tokio executor,
//! it can starve other tasks waiting on that executor to execute them.
//! Additionally, this prevents the query execution from being cancelled.

Note the second clause: this is not just a throughput problem, it is a correctness-of-cancellation problem.

The mechanism: spend budget, then yield

Tokio has a built-in cooperative scheduling primitive. Each task has a budget of 128 units; tokio::task::coop::poll_proceed(cx) returns Poll::Pending and schedules a wakeup if the budget is exhausted, and made_progress() gives one unit back. DataFusion’s default implementation uses it directly:

 rust
 
 
// datafusion/physical-plan/src/coop.rs:148-163
#[cfg(any(
 datafusion_coop = "tokio",
 not(any(
 datafusion_coop = "tokio_fallback",
 datafusion_coop = "per_stream"
 ))
))]
{
 let coop = std::task::ready!(tokio::task::coop::poll_proceed(cx));
 let value = self.inner.poll_next_unpin(cx);
 if value.is_ready() {
 coop.made_progress();
 }
 value
}

Read the ordering carefully: poll_proceed is called before the inner stream is polled, and made_progress() is called after it returned Ready. That ordering is the whole game. poll_proceed reserves the right to do one more unit of work; if the budget is gone, it returns Pending without polling the inner stream at all, and Tokio will re-poll later. If you instead check the budget after doing the work, you have already done the expensive thing — you can only yield afterwards, which is exactly the “yield too late” bug.

The three-way cfg is worth noting because it shows the design space. The crate declares the possible values at the workspace level:

 toml
 
 
# Cargo.toml:323-326
'unexpected_cfgs = { level = "warn", check-cfg = [
 'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))',

tokio (the default): the correct poll_proceed/made_progress dance above.
tokio_fallback: a compromise for runtimes without the real primitive — check has_budget_remaining() before polling, and consume the budget afterwards, “and just assume that that succeeded” (coop.rs:164-185). The comment is honest that this “may have slightly worse performance”.
per_stream: a completely local budget, no Tokio involvement:

 rust
 
 
// datafusion/physical-plan/src/coop.rs:114-121
#[cfg(datafusion_coop = "per_stream")]
// Magic value that matches Tokio's task budget value
const YIELD_FREQUENCY: u8 = 128;

with the poll body:

 rust
 
 
// datafusion/physical-plan/src/coop.rs:187-202
#[cfg(datafusion_coop = "per_stream")]
{
 if self.budget == 0 {
 self.budget = YIELD_FREQUENCY;
 cx.waker().wake_by_ref();
 return Poll::Pending;
 }

 let value = { self.inner.poll_next_unpin(cx) };

 if value.is_ready() {
 self.budget -= 1;
 } else {
 self.budget = YIELD_FREQUENCY;
 }
 value
}

The per_stream variant is subtly different in kind: it keeps a private budget per stream rather than sharing Tokio’s task budget. That means it cannot be starved by a sibling operator, but it also means it does not participate in Tokio’s global fairness — and it does the expensive work before checking (it checks at the top of the poll, which is fine, but it doesn’t reserve).

Two wrappers, one for statics and one for trait objects

 rust
 
 
// datafusion/physical-plan/src/coop.rs:452-475
pub fn cooperative<T>(stream: T) -> CooperativeStream<T>
where
 T: RecordBatchStream + Unpin + Send + 'static,
{
 CooperativeStream::new(stream)
}

pub fn make_cooperative(stream: SendableRecordBatchStream) -> SendableRecordBatchStream {
 // TODO is there a more elegant way to overload cooperative
 Box::pin(cooperative(RecordBatchStreamAdapter::new(
 stream.schema(),
 stream,
 )))
}

cooperative is generic, so the inner stream’s poll_next is statically dispatched — no vtable, no dynamic dispatch cost per batch. make_cooperative is the escape hatch for Box<dyn RecordBatchStream>, and it pays for that with an adapter layer. The TODO in the source is a standing invitation to do better.

Making it automatic: `EnsureCooperative`

Wrapping every stream by hand would be unusable. So there is a physical optimizer rule:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_coop.rs:60-121
impl PhysicalOptimizerRule for EnsureCooperative {
 fn name(&self) -> &str { "EnsureCooperative" }

 fn optimize(&self, plan: Arc<dyn ExecutionPlan>, _config: &ConfigOptions)
 -> Result<Arc<dyn ExecutionPlan>>
 {
 use std::cell::RefCell;
 let ancestry_stack = RefCell::new(Vec::<(SchedulingType, EvaluationType)>::new());

 plan.transform_down_up(
 |plan| {
 let props = plan.properties();
 ancestry_stack.borrow_mut()
 .push((props.scheduling_type, props.evaluation_type));
 Ok(Transformed::no(plan))
 },
 |plan| {
 ancestry_stack.borrow_mut().pop();
 let props = plan.properties();
 let is_cooperative = props.scheduling_type == SchedulingType::Cooperative;
 let is_leaf = plan.children().is_empty();
 let is_exchange = props.evaluation_type == EvaluationType::Eager;
 ...

The rule walks the plan maintaining a stack of (SchedulingType, EvaluationType) for the ancestors, and wraps a node in CooperativeExec when it is a leaf (a data source) or an exchange point (an eager operator), provided it is not already cooperative and is not already under a cooperative context.

The “already under a cooperative context” check is the interesting bit:

 rust
 
 
// datafusion/physical-optimizer/src/ensure_coop.rs:94-108
for (scheduling_type, evaluation_type) in ancestry_stack.borrow().iter().rev() {
 // If nearest ancestor is cooperative, we are under a cooperative context
 if *scheduling_type == SchedulingType::Cooperative {
 is_under_cooperative_context = true;
 break;
 // If nearest ancestor is eager, the cooperative context will be reset
 } else if *evaluation_type == EvaluationType::Eager {
 is_under_cooperative_context = false;
 break;
 }
}

A cooperative context ends at an eager operator. That is semantically correct: an eager operator (a repartition, a sort-preserving merge, a buffer) runs its own tasks and drives its own children independently of downstream demand. Making the parent of an eager operator cooperative does nothing for the eager operator’s children, because the eager operator will poll them from a spawned task regardless. So the rule resets at that boundary and starts again below it.

The metadata this relies on lives in PlanProperties:

 rust
 
 
// datafusion/physical-plan/src/execution_plan.rs:1364-1378
pub enum SchedulingType {
 /// The stream generated by [`execute`](ExecutionPlan::execute) does not actively participate in
 /// cooperative scheduling. ...
 NonCooperative,
 /// The stream generated by [`execute`](ExecutionPlan::execute) actively participates in
 /// cooperative scheduling by consuming task budget when it was able to produce a
 /// [`RecordBatch`].
 Cooperative,
}

and the complementary EvaluationType (execution_plan.rs:1389-1413), which distinguishes demand-driven (Lazy) operators from eager ones.

Coordinating N tasks and M channels

Cooperative scheduling is the intra-task half of the problem. The inter-task half is: how do N producer tasks feed M output partitions without deadlocking, and without letting the exchange buffer the whole dataset?

RepartitionExec is where this lives, and the answer splits in two: the memory pool handles the buffering (by spilling), and the gate handles the deadlock:

Each output partition has one channel with an unbounded buffer, shared by all N input tasks (in preserve-order mode it instead has N SPSC channels, one per input partition).
A global Gate counts empty channels. A sender checks empty_channels == 0 before enqueuing and parks itself if the gate is closed (distributor_channels.rs:218-225).
The guarantee is liveness, not memory: “at least one channel is empty”. A sender may still pile batches into an already-non-empty channel while a sibling stays empty, so the channel buffer is unbounded. The bound on in-flight data is the output partition’s MemoryReservation, and the gate knows nothing about bytes.

The producer side also has to yield, because pulling from the input and partitioning is a lot of work:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:2345-2372
// If the input stream is endless, we may spin forever and
// never yield back to tokio. See
// https://github.com/apache/datafusion/issues/5278.
//
// However, yielding on every batch causes a bottleneck
// when running with multiple cores. See
// https://github.com/apache/datafusion/issues/6290
//
// Thus, heuristically yield after producing num_partition
// batches
if batches_until_yield == 0 {
 tokio::task::yield_now().await;
 batches_until_yield = partitioner.num_partitions();
} else {
 batches_until_yield -= 1;
}

This is a heuristic, and the comments link the two GitHub issues that forced it into existence. Yielding after every batch costs throughput (issue #6290); never yielding hangs unbounded inputs (issue #5278). Yielding every num_partitions batches is the compromise, and the comment explains why that particular multiple is reasonable: “In round robin this is ideal as each input will get a new batch.”

The buffer that is exactly the right size

spawn_buffered is a small function with a subtle comment:

 rust
 
 
// datafusion/physical-plan/src/common.rs:186-226
pub fn spawn_buffered(mut input: SendableRecordBatchStream, buffer: usize) -> SendableRecordBatchStream {
 // Use tokio only if running from a multi-thread tokio context
 match tokio::runtime::Handle::try_current() {
 Ok(handle) if handle.runtime_flavor() == tokio::runtime::RuntimeFlavor::MultiThread => {
 let mut builder = RecordBatchReceiverStream::builder(input.schema(), buffer);
 let sender = builder.tx();
 builder.spawn(async move {
 // We call `reserve` (which waits until there's room for at least 1 message in the
 // channel buffer) **before** polling from input to ensure we hold a maximum of
 // `buffer` record batches in memory.
 // Polling from input and then calling send() would block when the channel is full
 // so it would essentially hold `buffer` + 1 record batches:
 // * `buffer`: this many elements would live inside the channel, since this is the
 // channel's capacity
 // * 1 extra RecordBatch which was produced, but there was no room for it in the
 // channel, so it's being owned by the send() future, which keeps the batch in
 // memory while it waits for a slot to free up
 while let Ok(permit) = sender.reserve().await {
 match input.next().await {
 Some(item) => permit.send(item),
 None => break,
 }
 }
 Ok(())
 });
 builder.build()
 }
 _ => input,
 }
}

reserve() before input.next().await is the trick. The naive order (poll, then send) holds buffer + 1 batches: the channel’s buffer plus the one sitting inside the pending send future. With a tokio::sync::mpsc permit held across the poll, the extra batch never materializes. This is the difference between “bounded” and “bounded plus one”, and the plus one is exactly the kind of thing that turns a 64 MiB budget into a 65 MiB OOM.

Also note the runtime_flavor check: on a current-thread runtime there is no point spawning a task, so the input is returned unwrapped.

Panics, cancellation, and `common-runtime`

The stream builder is where DataFusion handles the awkward parts of running work on a shared runtime.

 rust
 
 
// datafusion/physical-plan/src/stream.rs:46-55
/// Creates a stream from a collection of producing tasks, routing panics to the stream.
///
/// Note that this is similar to [`ReceiverStream` from tokio-stream], with the differences being:
///
/// 1. Methods to bound and "detach" tasks (`spawn()` and `spawn_blocking()`).
///
/// 2. Propagates panics, whereas the `tokio` version doesn't propagate panics to the receiver.
///
/// 3. Automatically cancels any outstanding tasks when the receiver stream is dropped.

The panics matter. If a worker task panics and the panic is swallowed, the query hangs forever waiting for a batch that will never arrive. So the builder merges the receiver with a “check” stream that drains the join set:

 rust
 
 
// datafusion/physical-plan/src/stream.rs:132-160
let check = async move {
 while let Some(result) = join_set.join_next().await {
 match result {
 Ok(Ok(())) => {}
 Ok(Err(error)) => return Some(Err(error)),
 Err(e) => {
 if e.is_panic() {
 // resume on the main thread
 std::panic::resume_unwind(e.into_panic());
 } else {
 return Some(exec_err!("Non Panic Task error: {e}"));
 }
 }
 }
 }
 None
};

and merges it with the data stream:

 rust
 
 
// datafusion/physical-plan/src/stream.rs:163-172
let rx_stream = futures::stream::unfold(rx, |mut rx| async move {
 let next_item = rx.recv().await;
 next_item.map(|next_item| (next_item, rx))
});

// Merge the streams together so whichever is ready first
// produces the batch
futures::stream::select(rx_stream, check_stream).boxed()

futures::stream::select polls both and returns whichever is ready first, so an error from a worker surfaces immediately rather than after the remaining workers finish.

datafusion/common-runtime/ wraps Tokio’s task types with DataFusion’s semantics. SpawnedTask aborts on drop:

 rust
 
 
// datafusion/common-runtime/src/common.rs:108-112
impl<R> Drop for SpawnedTask<R> {
 fn drop(&mut self) {
 self.inner.abort();
 }
}

which is what makes query cancellation work: dropping the output stream drops the SpawnedTasks, which abort the worker tasks, which drop their reservations, which return memory to the pool. The chain of RAII is what makes “cancel a query” a single drop.

`spawn_blocking` — and where not to use it

DataFusion does use spawn_blocking, but only for genuinely blocking work: JSON parsing (datasource-json/src/source.rs:510), catalog listing (catalog/src/stream.rs:470).

It explicitly does not use it for CPU-bound work:

 rust
 
 
// datafusion/core/src/lib.rs:644-650
//! Note that DataFusion does not use [`tokio::task::spawn_blocking`] for
//! CPU-bounded work, because `spawn_blocking` is designed for blocking **IO**,
//! not designed CPU bound tasks. Among other challenges, spawned blocking
//! tasks can't yield waiting for input (can't call `await`) so they
//! can't be used to limit the number of concurrent CPU bound tasks or
//! keep the processing pipeline to the same core.

This is a well-argued point and it is the reason the cooperative machinery exists at all. A spawn_blocking task is a thread-pool task: it cannot await, it cannot be cancelled, it cannot participate in Tokio’s budget, and it does not stay on the same core. If you ran a hash aggregation there, you would lose cancellation, lose fairness, and lose the ability to apply backpressure. So CPU-bound operators stay on the async runtime and cooperate.

The memory-table case

MemoryStream is the simplest possible stream: it holds a Vec<RecordBatch> and returns them one at a time, Ready every time:

 rust
 
 
// datafusion/physical-plan/src/memory.rs:94-100
fn poll_next(
 mut self: std::pin::Pin<&mut Self>,
 _: &mut Context<'_>,
) -> Poll<Option<Self::Item>> {
 if self.index >= self.data.len() {
 return Poll::Ready(None);
 }
 self.index += 1;

It never returns Pending, so on its own it would happily hand out a million batches without ever yielding. The fix is at the source:

 rust
 
 
// datafusion/datasource/src/memory.rs:79-93
pub fn open(&self, partition: usize, _context: Arc<TaskContext>) -> Result<SendableRecordBatchStream> {
 Ok(Box::pin(cooperative(
 MemoryStream::try_new(
 self.partitions[partition].clone(),
 Arc::clone(&self.projected_schema),
 self.projection.clone(),
 )?
 .with_fetch(self.fetch),
 )))
}

cooperative(...) wraps it, so every 128 batches the task yields to the runtime. LazyMemoryExec goes further and declares itself cooperative at construction time (memory.rs:228-234):

 rust
 
.with_scheduling_type(SchedulingType::Cooperative)

which tells the EnsureCooperative rule that it does not need to wrap it.

The reservation story for memory tables is handled by ReservationStream (stream.rs:745-806), which shrinks the reservation as batches are consumed and frees it entirely at end-of-stream:

 rust
 
 
// datafusion/physical-plan/src/stream.rs:781-795
Some(Ok(batch)) => {
 self.reservation.shrink(get_record_batch_memory_size(&batch));
 Poll::Ready(Some(Ok(batch)))
}
None => {
 // Stream is done so free the reservation completely
 self.reservation.free();
 // Release the input pipeline's resources.
 let inner_schema = self.inner.schema();
 self.inner = Box::pin(EmptyRecordBatchStream::new(inner_schema));
 Poll::Ready(None)
}

Note the “release the input pipeline’s resources” step: replacing the inner stream with an empty one tears down the upstream chain at end of stream rather than waiting for the outer stream to be dropped. That is what lets a LIMIT 1 over a huge scan release the scan’s memory immediately instead of holding it until the whole plan is dropped.

Trade-offs and failure modes

Cooperative scheduling costs a poll per batch. Every poll_next on a cooperative stream does an atomic load and a branch. For small batches in a tight loop this is measurable. The per_stream variant exists partly to make that cheaper.
The budget is a magic number shared with Tokio. YIELD_FREQUENCY = 128 is hardcoded to match Tokio’s internal task budget (coop.rs:118-121). If Tokio changes its budget, DataFusion’s per_stream mode silently drifts out of sync.
A cooperative stream inside a non-cooperative one is useless. If an operator’s poll_next never returns to the executor, the budget it spends is irrelevant. EnsureCooperative is the safety net, and it depends on operators correctly reporting SchedulingType and EvaluationType — get that wrong and the rule silently skips a subtree.
Gate-based backpressure is a liveness tool, not a memory tool. It keeps at least one channel empty so senders cannot deadlock; it does not cap a channel’s buffer. Memory is bounded by the pool plus the per-operator reservations — and the channel buffer sits outside both.
The reserve-before-poll trick is easy to get wrong. A single send() instead of permit.send() puts you back to buffer + 1 batches, and the regression looks like a mysterious 1-batch memory increase.
spawn_blocking tasks cannot be cancelled once started. SpawnedTask’s doc says it: “Note that if the task was spawned with spawn_blocking, it will only be aborted if it hasn’t started yet” (common-runtime/src/common.rs:28-31). So a blocking task that is stuck in a syscall will outlive the query.
Cooperative scheduling does not bound memory by itself. It bounds CPU monopolization. Memory is bounded by the memory pool plus the exchange gate plus the per-operator reservations. All three have to line up.

The design is worth stealing: treat “how long can a single poll_next run” as a first-class property of the execution model, encode it in plan metadata, and enforce it with an optimizer rule rather than with discipline. Once the runtime has a notion of a budget, backpressure stops being a memory accounting problem and becomes a scheduling problem — and the scheduler already knows how to stop a task from running forever.

The Perfect Hash Join: Turning a Hash Table Into a Plain Array When the Keys Are Dense Integers

Jay Hubbert — Wed, 21 Oct 2026 08:37:00 +0000

TL;DR: A hash join’s probe costs a hash computation, a bucket lookup and a chain walk. But if the build side’s key is an integer in a narrow range — say customer_id between 0 and 100,000 — you do not need a hash table at all. You need an array. DataFusion detects this case, builds a flat Vec<u32> indexed by the key, and turns the probe into a single array load. The interesting part is the detection: the decision has to be made at runtime, from the build side’s actual key distribution, and it has to survive the fact that a partitioned build side may destroy the density.

Hash joins are the workhorse of analytical query processing, and the classic implementation is a hash table: hash the key, index a bucket array, walk a chain of colliding entries. It is a good design, and it handles arbitrary keys.

But it is the wrong design for a large class of real joins.

 sql
 
 
SELECT f.*, d.dim_name
FROM facts f
JOIN dimensions d ON f.dim_id = d.dim_id; -- dim_id is an integer 0..100000

The build side has 100,000 rows with keys drawn from [0, 100000). The keys are dense. A hash table spends a hash computation, a modulo, a bucket load, a chain walk and a comparison to find a row that is sitting at a known offset in an array.

The right structure here is an array.

The array map

 rust
 
 
pub struct ArrayMap {
 // data[probSideVal-offset] -> valIdxInBuildSide + 1; 0 for absent
 data: Vec<u32>,
 // min val in buildSide
 offset: u64,
 // next[buildSideIdx] -> next matching valIdxInBuildSide + 1; 0 for end of chain.
 // If next is empty, it means there are no duplicate keys (no conflicts).
 // It uses the same chain-based conflict resolution as [`JoinHashMapType`].
 next: Vec<u32>,
 num_of_distinct_key: usize,
}

Three arrays, and each one is doing something specific.

data is the lookup table. It has one slot per possible key value in the range [min, max], and each slot holds the build-side row index plus one (so that zero can mean “absent”). The probe is:

 text
 
 
slot = probe_key - offset
if slot out of range: no match
row = data[slot]
if row == 0: no match

That is one memory load and two comparisons. No hash, no modulo, no chain.

offset is the minimum key value, so that the array is indexed from zero even when the keys start at, say, 1,000,000.

next handles duplicate keys. If the build side has two rows with the same key, the array can only point at one of them, so the second is chained. And the comment notes the important optimisation:

If next is empty, it means there are no duplicate keys (no conflicts).

When there are no duplicates — the common case for a dimension table — the chain array is not allocated at all, and the probe is a single load with no chain walk.

When to use it

The decision is made in array_map_key_range (datafusion/physical-plan/src/joins/hash_join/exec.rs), and the conditions are documented on HashJoinExec:

For joins with a single integer-based join key, HashJoinExec may use an ArrayMap (also known as a “perfect hash join”) instead of a general-purpose hash map. This optimization is used when:

There is exactly one join key.

The join key is an integer type up to 64 bits wide that can be losslessly converted to u64 (128-bit integer types such as i128 and u128 are not supported).

The range of keys is small enough (controlled by perfect_hash_join_small_build_threshold) OR the keys are sufficiently dense (controlled by perfect_hash_join_min_key_density).

build_side.num_rows() < u32::MAX

NullEqualsNothing || (NullEqualsNull && build side doesn’t contain null)

Five conditions. Let’s take them one at a time, because each one is a subtle correctness or resource constraint.

Condition 2 — integer types only. The array is indexed by the key, so the key must be an integer that can be losslessly converted to u64. i128 and u128 are excluded because they do not fit.

Condition 4 — num_rows < u32::MAX. The payload index in data is a u32. A build side with more than 4 billion rows cannot be indexed. The source has a TODO: support create ArrayMap<u64> — a 64-bit index would double the memory of the lookup table, which is why it is not done by default.

Condition 5 — null handling. If the join treats NULLs as equal to each other (NullEqualsNull), then a build-side NULL key must be findable. The array has no slot for “null” (nulls are not a number), so the optimisation is disabled if the build side contains any NULL key:

 rust
 
 
if null_equality == NullEquality::NullEqualsNull {
 for batch in batches.iter() {
 let arrays = evaluate_expressions_to_arrays(on_left, batch)?;
 if arrays[0].null_count() > 0 {
 return Ok(None);
 }
 }
}

The density heuristic

Condition 3 is the interesting one, and it has two ways to be satisfied, because there are two different ways the array map can be a bad idea.

The array map’s memory is O(max - min), not O(num_rows). So a build side with 10 rows and keys {0, 1_000_000, 2_000_000, ...} would allocate a 20-million-element array to store 10 rows. Catastrophic.

The two escape hatches:

 rust
 
 
let dense_ratio = (num_row as f64) / ((range + 1) as f64);

if range >= perfect_hash_join_small_build_threshold as u64
 && dense_ratio <= perfect_hash_join_min_key_density
{
 return Ok(None);
}

The small-range escape. If the range is small (below perfect_hash_join_small_build_threshold, default 1024), the array is small no matter how sparse the keys are. A 1024-element array is 4 KB — negligible — so even 3 rows in it is fine. The check is range >= threshold && density <= limit, i.e. both must be bad to reject. If the range is small, the density does not matter.

The density escape. Otherwise, the keys must be dense enough: density > perfect_hash_join_min_key_density (default 0.15). So at least 15% of the slots in the range must be occupied. A build side with 150,000 rows spanning [0, 1_000_000] has a density of 0.15 — the array is 4 MB and holds 150,000 rows, which is 26 bytes per row of array space versus a hash table’s own overhead. Worth it.

And the two thresholds are configurable, because “how dense is dense enough” depends on the hardware, the key width, and the workload.

The overflow trap

There is a bug-shaped detail in the code that is worth pointing out, because it is the kind of thing that produces a wrong answer rather than a crash:

 rust
 
 
// When the key range spans the full integer domain (e.g. i64::MIN to i64::MAX),
// range is u64::MAX and `range + 1` below would overflow.
if range == usize::MAX as u64 {
 return Ok(None);
}

let dense_ratio = (num_row as f64) / ((range + 1) as f64);

range is max - min as a u64. If the build side contains both i64::MIN and i64::MAX, the range is u64::MAX. Then range + 1 overflows — in release mode, u64::MAX + 1 wraps to zero, and the density becomes num_row / 0 = infinity, which would pass the density check and allocate an astronomically large array.

The guard is placed before the division. This is exactly the kind of integer-overflow bug that a fuzzer finds and a careful reviewer prevents.

The partitioning trap

And then there is the subtlety that makes this optimisation much harder than it looks. The doc on HashJoinExec spells it out:

Note that when using PartitionMode::Partitioned, the build side is split into multiple partitions. This can cause a dense build side to become sparse within each partition, potentially disabling this optimization.

For example, consider:
sql
SELECT t1.value, t2.value
FROM range(10000) AS t1
JOIN range(10000) AS t2
 ON t1.value = t2.value;
With 24 partitions, each partition will only receive a subset of the 10,000 rows. The first partition might contain values like 3, 10, 18, 39, 43, which are sparse relative to the original range, even though the overall data set is dense.

This is a beautiful example of a global property becoming a local property. The whole build side is dense — keys 0..10,000 with no gaps. But hash-partitioning by value gives partition 0 the keys 0, 417, 834, ..., and within that partition the density is 1/417, not 1.

So the array map is evaluated per partition, on that partition’s keys, and the density check uses the partition’s own min and max. Partition 0 might have min=0, max=9996, and 10,000/24 = 417 rows — density 0.04, below the 0.15 threshold. Optimisation disabled.

Which is the right answer for a local array map. But it means the perfect hash join mostly applies to CollectLeft mode (one build side, read once) or to cases where the partitioning preserves density.

The measurement

The code has a dedicated metric for this:

 rust
 
const ARRAY_MAP_CREATED_COUNT_METRIC_NAME: &str = "array_map_created_count";

So you can see, in EXPLAIN ANALYZE, whether the optimisation actually fired. Without such a metric, an optimisation that fires unpredictably (because it depends on the runtime key distribution) is impossible to tune. With it, you can look at a workload and see “the array map fired on 12 of 200 joins” and know that the other 188 would not have benefited from a denser key space.

The catch

Memory is O(range), not O(rows). The density check bounds this, but the bound is a heuristic with a configurable threshold.
Only integer keys. Strings, dates and decimals do not get the array map. (Dates could, in principle, since they are integers underneath, but the implementation does not.)
Single key only. A composite key (a, b) would need a hash — unless the pair can be packed into one integer, which the implementation does not do.
The decision is per-partition and therefore unstable. The same query on the same data can take the array-map path on one run and the hash-table path on another, if the partitioning changes. The metric exists precisely because of this.
It is a fast path, not a replacement. The hash table is still there, and the hash table is still needed for the general case. The array map is an optimisation layered on top.

The general lesson

The perfect hash join is a 300-year-old idea — it is what every “lookup table” and every “direct addressing” data structure is — applied to a modern setting where the key distribution is only known at runtime.

The interesting engineering is not the array. It is the decision procedure: two thresholds, an overflow guard, a per-partition re-evaluation, and a metric so you can see what happened. A fast path that fires based on a runtime property is only useful if you can also observe whether it fired.

How DataFusion Turns a Hash Join's Build Side into a Filter the Probe Side Can Execute

Jay Hubbert — Wed, 21 Oct 2026 08:29:00 +0000

TL;DR: A hash join knows, the moment it has read its build side, exactly which key values can possibly match. That is a fact, not a guess — and DataFusion publishes it as a runtime filter that the probe-side scan can execute. Small build sides become an IN list, large ones become a membership test against the build side’s own hash table, and every build contributes min/max bounds as a cheap range pre-filter. The hard parts are the coordination: a partitioned join has N independent build sides whose facts must be merged before any filter is published, and a partially-merged filter would silently drop rows.

Consider the canonical star-schema query:

 sql
 
 
SELECT o.*, c.name
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.amount > 100;

The customer_id filter is not in the SQL. But after the join has built its hash table over customers.id, it knows that only 4,000 distinct customer ids exist. Every orders row whose customer_id is not in that set can never match, and does not need to be read from disk, decoded, or compared.

If customers.id happens to be a primary key with min/max statistics, the join also learns a range: id BETWEEN 17 AND 91,338. If the build side is small enough, it learns an exact set.

That is a runtime-discovered filter, and it is exactly the kind of fact that static pushdown cannot express.

Two flavours of build side, two flavours of filter

DataFusion’s hash join has two partition modes (datafusion/physical-plan/src/joins/hash_join/):

CollectLeft — the build side is read once by partition 0 and shared with every probe partition. One build side, one fact, one filter.
Partitioned — both sides are hash-partitioned on the join key and each partition builds its own hash table independently. N build sides, N facts, and they must be merged.

The merge is the interesting part, and it is where a naive implementation loses rows.

The publication strategy

The filter is chosen per partition at collection time:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs
pub(crate) enum PushdownStrategy {
 /// Use InList for small build sides (< 128MB)
 InList(ArrayRef),
 /// Use map lookup for large build sides. If `Some` the second field
 /// represents the build side's keys using a bucket bitmap.
 Map(Arc<Map>, Option<Arc<KeyRangeBitmap>>),
 /// There was no data in this partition, do not build a dynamic filter for it
 Empty,
}

Three strategies, decided by how big the build side is.

InList — the build side’s key values are materialized into an Arrow array and used to build an InListExpr. The source comment notes the try_new_from_array constructor enables the static_filter optimization, which turns the membership test into a hash lookup rather than a linear scan of the list. The list is also used to build a struct expression for multi-column joins:

 rust
 
 
// Use InList expression for small build sides
PushdownStrategy::InList(in_list_array) => {
 let expr = if on_right.len() == 1 {
 // Single column: col IN (val1, val2, ...)
 Arc::clone(&on_right[0])
 } else {
 let fields = build_struct_fields(
 on_right.iter()
 .map(|r| r.data_type(schema))
 .collect::<Result<Vec<_>>>()?
 .as_ref(),
 )?;
 // The return field name and the function field name don't really matter here.
 let return_field =
 Arc::new(Field::new("struct", DataType::Struct(fields), true));
 ...
 };
 Ok(Some(Arc::new(InListExpr::try_new_from_array(
 expr, in_list_array, false, schema,
 )?)))
}

Struct-packing the multi-column key is the standard trick: struct(a, b) IN (struct(1, 2), struct(3, 4)) gives you one hashable value instead of a tuple comparison, and Arrow’s struct equality is a single kernel call.

Map — for a large build side, an IN list of a million values is worse than useless: materializing it costs memory, and evaluating it per row is slow. Instead, the filter becomes a membership test against the build side’s own hash table:

 rust
 
 
PushdownStrategy::Map(hash_map, pruning_bitmap) => {
 Ok(Some(Arc::new(HashTableLookupExpr::new(
 on_right.to_vec(),
 random_state.clone(),
 hash_map,
 "hash_lookup".to_string(),
 pruning_bitmap,
 )) as Arc<dyn PhysicalExpr>))
}

HashTableLookupExpr (datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs) is a physical expression that hashes its input columns with a preserved seed and probes the map:

 rust
 
 
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 // Evaluate columns
 let join_keys = evaluate_columns(&self.on_columns, batch)?;

 match self.map.as_ref() {
 Map::HashMap(map) => {
 with_hashes(&join_keys, self.random_state.random_state(), |hashes| {
 let array = map.contain_hashes(hashes);
 Ok(ColumnarValue::Array(Arc::new(array)))
 })
 }
 Map::ArrayMap(map) => {
 let array = map.contain_keys(&join_keys)?;
 Ok(ColumnarValue::Array(Arc::new(array)))
 }
 }
}

This is the elegant part: the build side’s hash table is already there, already built, already warm. Reusing it as a filter costs one hash computation and one probe per row — and the probe side would have done exactly that work anyway inside the join. The filter is not extra work; it is the same work, hoisted earlier so that non-matching rows never get decoded.

The pruning_bitmap is a KeyRangeBitmap — a summary of which buckets of the key space are occupied. It exists so the filter can be used for container pruning (skip whole row groups) without materializing the keys.

Empty — the partition had no build rows at all. Nothing to publish.

Bounds: the cheap pre-filter

Independently of the membership strategy, every partition contributes min/max bounds for each join key column:

 rust
 
 
pub(crate) struct ColumnBounds {
 /// The minimum value observed for this column
 pub(crate) min: ScalarValue,
 /// The maximum value observed for this column
 pub(crate) max: ScalarValue,
}

and they are turned into a range predicate:

 rust
 
 
fn create_bounds_predicate(
 on_right: &[PhysicalExprRef],
 bounds: &PartitionBounds,
) -> Option<Arc<dyn PhysicalExpr>> {
 let mut column_predicates = Vec::with_capacity(on_right.len());

 for (col_idx, right_expr) in on_right.iter().enumerate() {
 if let Some(column_bounds) = bounds.get_column_bounds(col_idx) {
 // Create predicate: col >= min AND col <= max
 let min_expr = Arc::new(BinaryExpr::new(
 Arc::clone(right_expr), Operator::GtEq, lit(column_bounds.min.clone()),
 )) as Arc<dyn PhysicalExpr>;
 let max_expr = Arc::new(BinaryExpr::new(
 Arc::clone(right_expr), Operator::LtEq, lit(column_bounds.max.clone()),
 )) as Arc<dyn PhysicalExpr>;
 ...

Bounds are combined with membership using AND, and membership alone is enough when bounds are unavailable:

 rust
 
 
fn combine_membership_and_bounds(
 membership_expr: Option<Arc<dyn PhysicalExpr>>,
 bounds_expr: Option<Arc<dyn PhysicalExpr>>,
) -> Option<Arc<dyn PhysicalExpr>> {
 match (membership_expr, bounds_expr) {
 (Some(membership), Some(bounds)) =>
 Some(Arc::new(BinaryExpr::new(bounds, Operator::And, membership)) ...),
 (Some(membership), None) => Some(membership),
 (None, Some(bounds)) => Some(bounds),
 (None, None) => None,
 }
}

The bounds are not just an optimisation — they are what makes the filter composable with pruning. A Parquet row group knows its customer_id min/max; a filter of the form customer_id BETWEEN 17 AND 91338 can be evaluated against that statistic and the row group skipped without reading a byte. An IN list of 4,000 values cannot be evaluated against a min/max statistic at all.

The barrier: why a partial filter is a bug

Here is the crux. In Partitioned mode, partition 0 of the build side and partition 1 of the build side are different key ranges. A filter derived from partition 0 alone would reject every probe row that should have matched in partition 1.

The source is blunt about it:

We cannot build a partial filter from some partitions - it would incorrectly eliminate valid join results. We must wait until we have complete information from ALL relevant partitions before updating the dynamic filter.

So DataFusion implements a barrier with an elected finalizer. The doc comment on SharedBuildAccumulator describes the protocol:

Each partition computes information from its build-side data (hash maps and/or bounds)

Information is stored in the shared state, which tracks how many partitions have reported

When the last partition reports, one waiter is elected as the finalizer; it merges the collected information, updates the dynamic filter exactly once, and publishes the terminal result by transitioning CompletionState to Ready

A tokio::sync::Notify wakes any other partitions parked in wait_for_completion, which then observe the Ready state under the mutex and return immediately

 rust
 
 
pub(crate) async fn report_build_data(&self, data: PartitionBuildData) -> Result<()> {
 let finalize_input = {
 let mut guard = self.inner.lock();
 self.store_build_data(&mut guard, data)?;
 self.take_finalize_input_if_ready(&mut guard)
 };

 if let Some(finalize_input) = finalize_input {
 self.finish(finalize_input);
 }

 self.wait_for_completion().await
}

Only one thread wins the election — the one that observes the last pending partition turn into a reported one while holding the mutex — and it performs the merge. Every other thread parks on the Notify.

Note that the finalize happens outside the mutex. The finalizer’s work (build the IN list, build the CASE expression, evaluate the bounds) is not free, and holding the lock through it would serialise every partition’s report behind the filter construction.

Routing: one filter, N partitions

The merged partitioned filter has to route each probe row to the right partition’s filter, because partition 3’s build side contains different keys than partition 0’s. DataFusion solves this with a CASE expression keyed on the partition id:

 rust
 
 
// Empty partitions reject their routed rows, while canceled partitions stay
// permissive because their build contents are unknown.
fn build_partitioned_filter(&self, partitions: Vec<PartitionStatus>) -> Result<()> {
 ...
 // WHEN empty_partition_id THEN false -- only when cancellation exists
 // WHEN real_partition_id THEN F(real_partition_id)

The generated expression is roughly:

 sql
 
 
CASE repartition_partition_id(cols)
 WHEN 0 THEN hash_lookup_0(cols) AND col BETWEEN min0 AND max0
 WHEN 1 THEN hash_lookup_1(cols) AND col BETWEEN min1 AND max1
 ...
 ELSE true
END

Two details here are pure defensive engineering:

An empty build partition maps to false — nothing can match it, so rejecting is correct and cheap.
A canceled partition (the build task failed or was cancelled) maps to true — permissive, because we do not know what it contained. Guessing “empty” here would silently drop rows.

NULL handling

A join with NullEqualsNull semantics can match a probe-side NULL against a build-side NULL. A pushed filter col IN (1, 2, 3) evaluates to NULL for a NULL col, and a filter that rejects NULLs would drop rows that the join would have matched. The accumulator tracks this:

 rust
 
 
/// Null equality of the join. Under `NullEqualsNull` a probe-side NULL can match a
/// build-side NULL, so the pushed filter must keep NULL rows here too.
null_equality: NullEquality,
/// Null-aware anti join (`NOT IN`). A probe-side NULL must reach the join so its
/// three-valued logic can collapse the result, so the pushed filter keeps NULL rows.
null_aware: Option<NullAwareMode>,

and there is a preserve_probe_nulls helper applied to the final filter expression. Getting this wrong is a correctness bug that would show up as “missing rows in a NOT IN join”, which is exactly the kind of bug that is hard to notice and expensive to debug.

The catch

The barrier costs latency. The filter cannot be published until all build partitions have reported. For a join whose build side is slow to read, the probe side idles at the filter (or proceeds unfiltered) until the barrier clears. The whole point of the dynamic filter is defeated if the build side takes longer than the probe side.
A partitioned merge is not a merge of IN lists. With N partitions you cannot express “row matches if it matches partition k’s filter” as a single IN list without losing the routing. Hence the CASE. For large partitioned builds the Map strategy wins, and each branch is a hash lookup against that partition’s table.
The hash seed must match. HashTableLookupExpr hashes with the join’s own HASH_JOIN_SEED ('J', 'O', 'I', 'N' — a four-byte ASCII spelling of “JOIN”), and SeededRandomState exists specifically because RandomState does not expose its seed after construction. Without the preserved seed the filter’s hashes would not match the build table’s hashes, and the filter would reject every row.

That last detail is a nice illustration of a general theme in this code base: performance tricks in DataFusion frequently come with a small, boring piece of bookkeeping (a preserved seed, a generation counter, a monotone cache guard) that is easy to overlook and catastrophic to remove.

DataFusion's Interval Analysis: How `BETWEEN` Becomes a Scalar Range and a Selectivity Number

Jay Hubbert — Wed, 21 Oct 2026 08:21:00 +0000

TL;DR: a BETWEEN 10 AND 20 is desugared into a >= 10 AND a <= 20 by the expression simplifier, then re-absorbed into a single Interval (two ScalarValue endpoints) by the constraint-propagation solver. That interval is intersected with the column’s min/max from statistics, and the ratio of the two interval cardinalities becomes the filter selectivity. No histogram, no sampling, no join — just two numbers per column and a divide.

Every cost-based optimizer eventually hits the same wall: to choose between a hash join and a nested loop join, you need to know how many rows will survive a filter. Most systems solve this with histograms, sampling, or a learned model. All three are expensive, all three are approximate, and all three are a source of embarrassing production incidents.

Apache DataFusion takes a much cheaper route. It has exactly two pieces of information per column — a minimum and a maximum — and it derives a filter’s selectivity from the geometric relationship between the range implied by the predicate and the range the data actually occupies.

This article walks through the whole path: how BETWEEN becomes an interval, how the interval is propagated, and where the resulting number ends up.

The vocabulary: `Precision<T>`

Before anything else, the tri-state that everything else is built on:

 rust
 
 
// datafusion/common/src/stats.rs:31-60
pub enum Precision<T: Debug + Clone + PartialEq + Eq + PartialOrd> {
 /// The exact value is known. Used for guaranteeing correctness.
 Exact(T),
 /// The value is not known exactly, but is likely close to this value.
 /// Used for cost-based optimizations.
 Inexact(T),
 /// Nothing is known about the value. This is the default state.
 #[default]
 Absent,
}

Exact is not “approximately right”. The doc comment is explicit: exact values are “used for guaranteeing correctness” — they answer COUNT(*), MIN and MAX without scanning and let the optimizer delete limits. Inexact is the cost-based-optimization tier. Absent is an absorbing element: any arithmetic involving Absent yields Absent.

This distinction matters later, because the pruning layer (covered in a companion piece) refuses to use anything but Exact bounds.

The container is small:

 rust
 
 
// datafusion/common/src/stats.rs:368-383
pub struct Statistics {
 pub num_rows: Precision<usize>,
 pub total_byte_size: Precision<usize>,
 pub column_statistics: Vec<ColumnStatistics>,
}

and the per-column part is where the interesting fields live:

 rust
 
 
// datafusion/common/src/stats.rs:1080-1114
pub struct ColumnStatistics {
 pub null_count: Precision<usize>,
 pub max_value: Precision<ScalarValue>,
 pub min_value: Precision<ScalarValue>,
 pub sum_value: Precision<ScalarValue>,
 pub distinct_count: Precision<usize>,
 pub byte_size: Precision<usize>,
}

Note the shape: min_value and max_value are ScalarValues, not typed numbers. The same struct covers Int64, Utf8, Timestamp and Decimal256 without a type parameter, at the cost of runtime type dispatch everywhere.

Step 1: `BETWEEN` is not special

The first thing DataFusion does with a BETWEEN 10 AND 20 is throw away the Between node entirely. The expression simplifier has a rule for it:

 rust
 
 
// datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:1654-1669
//
// Rules for Between
//
// a between 3 and 5 --> a >= 3 AND a <=5
// a not between 3 and 5 --> a < 3 OR a > 5
Expr::Between(between) => Transformed::yes(if between.negated {
 let l = *between.expr.clone();
 let r = *between.expr;
 or(l.lt(*between.low), r.gt(*between.high))
} else {
 and(
 between.expr.clone().gt_eq(*between.low),
 between.expr.lt_eq(*between.high),
 )
}),

This looks like a loss of information — we now have two independent conjuncts instead of one node. It is not. Two comparisons against the same column is exactly the shape the interval solver wants, and the solver is the thing that puts the range back together.

Step 2: intervals over `ScalarValue`

The interval type is deliberately minimal:

 rust
 
 
// datafusion/expr-common/src/interval_arithmetic.rs:177-181
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Interval {
 lower: ScalarValue,
 upper: ScalarValue,
}

A NULL endpoint means “unbounded on this side”. The constructor is strict about ordering, because an inverted interval is always a bug:

 rust
 
 
// datafusion/expr-common/src/interval_arithmetic.rs:267-285
pub fn try_new(lower: ScalarValue, upper: ScalarValue) -> Result<Self> {
 assert_eq_or_internal_err!(
 lower.data_type(), upper.data_type(),
 "Endpoints of an Interval should have the same type"
 );
 let interval = Self::new(lower, upper);
 assert_or_internal_err!(
 interval.lower.is_null()
 || interval.upper.is_null()
 || interval.lower <= interval.upper,
 "Interval's lower bound {} is greater than the upper bound {}",
 interval.lower, interval.upper
 );
 Ok(interval)
}

Self::new also standardizes the awkward cases: NaN becomes unbounded, +INF becomes f64::MAX for a lower bound and NULL for an upper bound, and unsigned lower bounds default to zero rather than -inf.

Step 3: the column’s interval comes from statistics

The solver’s input is one interval per column, built straight out of ColumnStatistics:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:97-129
pub fn try_from_column(
 schema: &Schema,
 col_stats: &ColumnStatistics,
 col_index: usize,
) -> Result<Self> {
 let field = schema.fields().get(col_index).ok_or_else(|| { ... })?;
 let empty_field =
 ScalarValue::try_from(field.data_type()).unwrap_or(ScalarValue::Null);
 let interval = Interval::try_new(
 col_stats.min_value.get_value().cloned().unwrap_or_else(|| empty_field.clone()),
 col_stats.max_value.get_value().cloned().unwrap_or(empty_field),
 )?;
 ...
}

So ColumnStatistics { min_value: Exact(Int32(0)), max_value: Exact(Int32(999)) } becomes Interval { lower: Int32(0), upper: Int32(999) }, and a column with absent statistics becomes an unbounded interval.

Step 4: propagating the predicate backwards

analyze() is the entry point. It builds an expression DAG, seeds the leaf nodes with the intervals from statistics, and pushes the predicate’s constraints down into them:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:164-232 (excerpt)
pub fn analyze(
 expr: &Arc<dyn PhysicalExpr>,
 context: AnalysisContext,
 schema: &Schema,
) -> Result<AnalysisContext> {
 ...
 let mut target_boundaries = context.boundaries;
 let mut graph = ExprIntervalGraph::try_new(Arc::clone(expr), schema)?;
 ...
 match graph.update_ranges(&mut target_indices_and_boundaries, Interval::TRUE)? {
 PropagationResult::Success => {
 shrink_boundaries(&graph, target_boundaries, &target_expr_and_indices)
 }
 PropagationResult::Infeasible => {
 // If the propagation result is infeasible, set intervals to None
 for bound in &mut target_boundaries { bound.interval = None; }
 Ok(AnalysisContext::new(target_boundaries).with_selectivity(0.0))
 }
 PropagationResult::CannotPropagate => {
 Ok(AnalysisContext::new(target_boundaries).with_selectivity(1.0))
 }
 }
}

Three outcomes, and they map cleanly onto “how selective is this filter”: Success means the constraint could be pushed down and we can compute a selectivity; Infeasible means the predicate contradicts the data (the filter matches nothing); CannotPropagate means we learned nothing, so assume everything passes.

The propagation itself is where the AND of two comparisons gets merged back into one interval. For a comparison under a TRUE parent:

 rust
 
 
// datafusion/physical-expr/src/intervals/cp_solver.rs:353-373
pub fn propagate_comparison(
 op: &Operator,
 parent: &Interval,
 left_child: &Interval,
 right_child: &Interval,
) -> Result<Option<(Interval, Interval)>> {
 if parent == &Interval::TRUE {
 match op {
 Operator::Eq => left_child.intersect(right_child).map(|result| {
 result.map(|intersection| (intersection.clone(), intersection))
 }),
 Operator::Gt => satisfy_greater(left_child, right_child, true),
 Operator::GtEq => satisfy_greater(left_child, right_child, false),
 Operator::Lt => satisfy_greater(right_child, left_child, true)
 .map(|t| t.map(reverse_tuple)),
 Operator::LtEq => satisfy_greater(right_child, left_child, false)
 .map(|t| t.map(reverse_tuple)),
 ...

a >= 10 under TRUE shrinks a’s interval from below; a <= 20 shrinks it from above. Because both are propagated against the same column node, the column’s interval ends up as [10, 20] — the BETWEEN range, reconstructed.

The whole flow:

text

 a BETWEEN 10 AND 20
 │
 ▼ expr_simplifier.rs:1660
 (a >= 10) AND (a <= 20)
 │
 ▼ cp_solver.rs:353 propagate_comparison
 column a interval: [0, 999] ∩ [10, +inf] ∩ (-inf, 20]
 │
 ▼
 column a interval: [10, 20] ← the exact scalar interval
 │
 ▼ cardinality_ratio (interval_arithmetic.rs:1740)
 selectivity = |[10,20]| / |[0,999]|

Step 5: from interval to number

The selectivity is a ratio of cardinalities:

 rust
 
 
// datafusion/expr-common/src/interval_arithmetic.rs:1740-1747
pub fn cardinality_ratio(initial_interval: &Interval, final_interval: &Interval) -> f64 {
 match (final_interval.cardinality(), initial_interval.cardinality()) {
 (Some(final_interval), Some(initial_interval)) => {
 (final_interval as f64) / (initial_interval as f64)
 }
 _ => 1.0,
 }
}

Interval::cardinality() only works for numeric, date, timestamp and decimal types — for a Utf8 column it returns None, and the ratio degrades to 1.0 (no filtering). This is the single biggest limitation of the whole scheme, and it is why DataFusion has a separate string-specific estimator (string_selectivity in datafusion/physical-plan/src/filter.rs:1292) that falls back to a configurable default.

Two more subtleties live in calculate_selectivity:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:305-344
fn calculate_selectivity(
 target_boundaries: &[ExprBoundaries],
 initial_boundaries: &[ExprBoundaries],
) -> Result<f64> {
 // Since the intervals are assumed uniform and the values
 // are not correlated, we need to multiply the selectivities
 // of multiple columns to get the overall selectivity.
 let mut acc: f64 = 1.0;
 for (initial, target) in initial_boundaries.iter().zip(target_boundaries) {
 match (initial.interval.as_ref(), target.interval.as_ref()) {
 (Some(initial_interval), Some(target_interval)) => {
 if let Precision::Exact(distinct_count)
 | Precision::Inexact(distinct_count) = target.distinct_count
 && let Some(s) = singleton_selectivity(
 initial_interval, target_interval, distinct_count,
 )
 {
 acc *= s;
 continue;
 }
 acc *= cardinality_ratio(initial_interval, target_interval);
 }
 ...

Two things to notice. First, the “uniform distribution” assumption is explicit in the comment: values are assumed uniformly spread over the interval and columns are assumed uncorrelated. city = 'Paris' AND state = 'CA' multiplies two independent selectivities, which is wrong whenever they are correlated — and they usually are.

Second, the singleton_selectivity shortcut. If the filter collapsed the column’s interval to a single point, the ratio-based estimate would still be 1/999 even when the column holds only 10 distinct values. The shortcut replaces the ratio with 1 / distinct_count:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:274-300
fn singleton_selectivity(
 initial_interval: &Interval,
 target_interval: &Interval,
 distinct_count: usize,
) -> Option<f64> {
 if distinct_count == 0
 || target_interval.lower().is_null()
 || target_interval.lower() != target_interval.upper()
 {
 return None;
 }
 // Only treat this as a newly-applied equality filter when the initial
 // interval was not already that same singleton.
 let initial_is_same_singleton = !initial_interval.lower().is_null()
 && initial_interval.lower() == initial_interval.upper()
 && initial_interval.lower() == target_interval.lower();
 if initial_is_same_singleton {
 return None;
 }
 Some(1.0 / distinct_count as f64)
}

The initial_is_same_singleton guard is a bug fix worth calling out: if the statistics already said the column was a single value, applying 1/NDV again would double-count the restriction and under-estimate the row count.

Step 6: where the number is consumed

FilterExec is where the estimate becomes a Statistics:

 rust
 
 
// datafusion/physical-plan/src/filter.rs:389-406
if check_support(predicate, schema) {
 let input_analysis_ctx = AnalysisContext::try_from_statistics(
 schema,
 &input_stats.column_statistics,
 )?;
 let analysis_ctx = analyze(predicate, input_analysis_ctx, schema)?;
 let selectivity = analysis_ctx.selectivity.unwrap_or(1.0);
 let filtered_num_rows =
 input_num_rows.with_estimated_selectivity(selectivity);
 let cs = collect_new_statistics(
 schema,
 &input_stats.column_statistics,
 analysis_ctx.boundaries,
 selectivity,
 &null_rejecting_columns,
 filtered_num_rows,
 );
 (selectivity, filtered_num_rows, cs)

with_estimated_selectivity is where the estimate is explicitly downgraded to Inexact, which is the honest thing to do:

 rust
 
 
// datafusion/common/src/stats.rs:200-206
pub fn with_estimated_selectivity(self, selectivity: f64) -> Self {
 if self == Precision::Exact(0) {
 return self;
 }
 self.map(|v| ((v as f64 * selectivity).ceil()) as usize)
 .to_inexact()
}

Notice ceil(): a filter never increases the row count, but rounding down would claim it might. Rounding up is the conservative direction.

The boundaries that come out of analyze are also fed into collect_new_statistics, which tightens the min/max of the surviving columns. So WHERE a BETWEEN 10 AND 20 does not just produce a row count; it produces a narrower ColumnStatistics for a, which then propagates upward to the join, the aggregate, and the next filter.

The memoization that makes this affordable

Interval analysis is not cheap: it builds a petgraph DAG per expression. Running it once per filter per plan would dominate planning time on wide plans. DataFusion’s answer is a memoized bottom-up walk:

 rust
 
 
// datafusion/physical-plan/src/statistics.rs:39-68
type CacheKey = (usize, Option<usize>);

fn cache_key(plan: &dyn ExecutionPlan, partition: Option<usize>) -> CacheKey {
 (from_ref::<dyn ExecutionPlan>(plan).cast::<()>() as usize, partition)
}

/// Per-context memoization cache for statistics computation.
///
/// Keys are `(plan node pointer address, partition)`. Every entry retains the
/// `Arc` whose pointer supplied its key, preventing a dropped plan's address
/// from being reused by a different plan while the entry exists.

The key is a raw pointer, and each cache entry holds the Arc that supplied the pointer. That is a small, clever trick: it makes address reuse impossible while the entry is alive, so the cache can be keyed on identity without inventing a plan-wide unique id. The cost is that the cache keeps dead plans alive until reset_cache() is called.

The walk itself consults a pluggable provider chain before the operator’s built-in statistics_from_inputs, which is the extension point used to feed catalog-level statistics into the optimizer.

Trade-offs and failure modes

The uniform distribution assumption is the big one. A column with a Zipfian or exponential distribution will produce a selectivity that is wrong by an order of magnitude. DataFusion has no histogram, no sketch, no sample. If your amount column has 99% of its mass in [0, 100] and the file’s min/max is [0, 1000000], then amount < 1000 gets a selectivity of 0.001 when the truth is closer to 0.5. The planner then thinks the join build side is tiny and picks the wrong one.

Independence across columns. calculate_selectivity multiplies per-column selectivities. Correlated predicates multiply twice, so the estimated cardinality can be off by orders of magnitude in the low direction.

Non-numeric columns get nothing. cardinality() returns None for Utf8, List, Struct. The interval machinery simply does not apply, and DataFusion falls back to string_selectivity or a flat default.

Exact versus Inexact is a correctness boundary, not a hint. A filter’s output row count is always Inexact unless it was provably zero. The optimizer is allowed to use Exact values to skip work entirely (removing a LIMIT because the input has exactly 5 rows), so a source that lies about Exact gets wrong answers, not just bad plans.

NOT BETWEEN is subtler than it looks. NOT (a BETWEEN -1 AND 1) over a ∈ [-1, 1] is infeasible — no row can satisfy it — and the solver correctly reports None for the interval. But NOT (a BETWEEN -1 AND 1) over a ∈ [-2, 2] does have rows, and the solver has to widen the interval rather than intersect it. The test at datafusion/physical-expr/src/analysis.rs:715-752 pins this behavior down.

Signed zero. NOT (a = +0.0) over a ∈ [-0.0, +0.0] must be infeasible, because +0.0 = +0.0 is TRUE in IEEE 754 even though PartialEq on ScalarValue is bit-wise. The solver has explicit handling for this, and the comment at datafusion/physical-expr/src/analysis.rs:782-790 explains exactly which short-circuit it defeats.

Configuration

There is no single “enable interval analysis” switch. The knobs that matter:

Option	Default	Effect
`datafusion.execution.collect_statistics`	`true`	Whether file/table statistics are gathered at all. Without them, every interval is unbounded and selectivity is 1.0.
`datafusion.optimizer.skip_failed_rules`	`false`	If an optimizer rule throws (e.g. an interval construction error), fail the query rather than silently continuing.
`datafusion.optimizer.max_passes`	`3`	How many times the logical optimizer re-runs the rule set; more passes mean more statistics-driven rewrites get a chance to fire.
`FILTER_EXEC_DEFAULT_SELECTIVITY` (`filter.rs:80`)	`20`	Not a config key but a `FilterExecBuilder::with_default_selectivity` knob: the fallback selectivity percentage for string predicates the interval machinery cannot handle.

The big lever is collect_statistics. Turn it off and every downstream estimate collapses to Absent: no pruning, no build-side selection, no interval analysis. It is also the knob that matters most for listing-heavy workloads, because gathering statistics means reading Parquet footers.

Bottom line

The trick that makes DataFusion’s planner work without histograms is that it does not try to be right about distributions. It only tries to be right about ranges, and it converts a range comparison into a number by dividing interval cardinalities. That is cheap, it is local, and it degrades gracefully — but the moment your data is skewed, or your columns are correlated, or your predicate is on a string, the estimate is fiction.

The honest summary: DataFusion’s statistics are a fast, structured guess, and the codebase is refreshingly explicit about it — every comment in analysis.rs that says “assumed uniform” or “not correlated” is a warning label left by whoever had to debug the resulting plan.

From SQL WHERE to Parquet Row Filter: How DataFusion Decides What It Can Push Down (and What It Must Keep)

Jay Hubbert — Wed, 21 Oct 2026 08:13:00 +0000

TL;DR: WHERE x > 5 reaches the Parquet reader as two independent questions: “can statistics prove this is false here?” (pruning) and “can the decoder evaluate this while it decodes?” (row filter). The support matrices for those two questions are different, non-overlapping, and both are one-sided: unsupported means “keep the data”, never “drop it”. A conjunct that neither system can handle stays in the physical plan as a residual filter, because dropping it would silently change the answer.

One predicate, two questions

When a FilterExec above a Parquet scan pushes its predicate down, the predicate arrives at the source as a single PhysicalExpr. From there it has to serve two masters:

 text
 
 
 WHERE a = 1 AND b = 2 AND c % 2 = 0
 │
 ┌────────┴─────────┐
 ▼ ▼
 statistics pruning decoder pushdown
 "can no row in this "can the decoder evaluate
 row group match?" this while decoding?"
 │ │
 ▼ ▼
 PruningPredicate RowFilter (ArrowPredicate)

The two questions have different answers for the same expression, and conflating them is the classic source of bugs in this area. c % 2 = 0 is a fine row filter (the decoder can compute a modulo) but is useless for statistics pruning (no range of c rules it out). s['x'] > 3 is fine for the decoder but not for statistics, because Parquet statistics for nested fields are not wired into the pruning predicate rewrite yet.

Question 1: which predicates can prune row groups?

Statistics pruning is implemented in the datafusion/pruning crate. The mechanism is a rewrite: the predicate is transformed into an expression over statistics columns (x_min, x_max, x_null_count, x_row_count), which is then evaluated against a record batch of statistics — one row per container (row group, file, …).

The documented rewrite table (datafusion/pruning/src/pruning_predicate.rs:244-252):

Original predicate	Rewritten predicate
`x = 5`	`x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max)`
`x < 5`	`x_null_count != x_row_count AND (x_min < 5)`
`x = 5 AND y = 10`	the two rewrites ANDed
`x IS NULL`	`x_null_count > 0`
`x IS NOT NULL`	`x_null_count != row_count`
`CAST(x as int) = 5`	`x_null_count != x_row_count AND (CAST(x_min as int) <= 5 AND 5 <= CAST(x_max as int))`

Everything else is handled by a rewrite function per expression type, and the entry point for a comparison is rewrite_expr_to_prunable (pruning_predicate.rs:1243-1318). Its rules: the left side must be a monotone-ish column expression and the right side a literal. The column expression may be a column, a cast of a column, a negated column, or a negated literal comparison. The rewrite even normalizes -col > 10 into col < -10 so that only the monotone direction has to be handled, and refuses Not over anything other than an equality-family operator (pruning_predicate.rs:1296-1316).

On top of that, build_predicate_expression (pruning_predicate.rs:1780-1990) recognises a specific set of node types before falling through to the comparison rewrite: IS NULL / IS NOT NULL on a column, a bare column (used for NOT col), IN (...) lists, a HashTableLookupExpr (a semi-join turned into a lookup), boolean CASE expressions (but only with an ELSE branch), case-sensitive LIKE / NOT LIKE, and AND / OR of the above.

`LIKE` becomes a range

The LIKE handling is a nice piece of engineering: a prefix match is exactly a range, so col LIKE 'foo%' is rewritten into col_min <= 'foo' AND 'foo' <= col_max, with the upper bound computed by incrementing the last UTF-8 byte of the prefix (pruning_predicate.rs:2255-2291):

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:2276-2286
let (decoded_prefix, rest) = split_constant_prefix(s);
let has_wildcard = !rest.is_empty();
if has_wildcard && decoded_prefix.is_empty() {
 // there's no filtering we could possibly do, return None and have this be handled by the unhandled hook
 return None;
}
let (lower_bound, upper_bound) = if has_wildcard {
 let incremented_prefix = increment_utf8(&decoded_prefix)?;

Escaped wildcards are handled too ('foo\_%' bounds at 'foo_'), and a pattern with no usable prefix ('%foo') degenerates to “no filtering possible”. ILIKE is explicitly not supported — the TODO says it would need lower()/upper() bounds, which the statistics rewrite cannot express (pruning_predicate.rs:2266-2267).

`IN` lists and the size cap

IN lists are rewritten into a disjunction of equalities, but only up to a cap, documented in the builder (pruning_predicate.rs:462-484): lists up to min(20, C) values use the per-value rewrite; lists between 20 and the configured cap C use a compact pruning expression when the values are ordered literals, and fall back to the per-value rewrite otherwise; lists longer than C hit the unhandled-predicate hook, i.e. “keep the container”.

The default cap is 20 (MAX_IN_LIST_SIZE, pruning_predicate.rs:1697), and query engines wire it from datafusion.execution.parquet.max_in_list_size. The reason for a cap is not paranoia about the list but about the size of the rewritten expression: an IN list of 10,000 values becomes a tree of 10,000 comparisons, evaluated once per row group, over statistics columns. Above the cap, the predicate simply stops pruning — exactly the “keep the container” behaviour, and therefore safe.

The unhandled hook: the constant `true`

Anything the rewriter cannot handle is passed to an UnhandledPredicateHook. The default implementation returns a literal true (pruning_predicate.rs:587-604) — meaning “there may be matching rows”, i.e. keep the container. This is the single most important safety property of the whole design: an expression the rewriter does not understand becomes a proof of nothing.

`AND` is forgiving, `OR` is not

Partial support is allowed, but only in one direction. An AND can use whichever conjuncts it understands, because a row must satisfy all of them. An OR cannot: if one branch is unprunable, the disjunction might be true because of that branch.

The rewriter encodes this structurally: AND/OR combine the rewritten children, and simplification collapses true AND x to x and true OR x to true — i.e. an unprunable branch under OR poisons the whole disjunction back to “keep everything” (pruning_predicate.rs:1934-1970). The behaviour is visible in DataFusion’s own tests: c1 > 15 AND (c2 % 2 = 0) still prunes row group 1, while c1 > 15 OR (c2 % 2 = 0) prunes nothing (row_group_filter.rs:1059-1121).

Question 2: which predicates can the decoder evaluate?

The decoder-level support matrix is a completely separate implementation, living in projection_read_plan.rs (PushdownChecker) and supported_predicates.rs. Its rules:

primitive columns: fine,
list columns: only with a registered list predicate (array_has, array_has_all, array_has_any) or a null check,
struct columns: only via get_field down to a primitive leaf,
columns absent from the file: not pushable,
UDFs that require a rewrite: not pushable (has_unpushable_udfs).

The list-predicate registry is tiny and explicit:

 rust
 
 
// datafusion/datasource-parquet/src/supported_predicates.rs:81-86
fn is_supported_scalar_function(expr: &dyn PhysicalExpr) -> bool {
 scalar_function_name(expr).is_some_and(|name| {
 // Registry of verified array functions
 matches!(name, "array_has" | "array_has_all" | "array_has_any")
 })
}

Expression types defined in external crates cannot implement DataFusion’s trait directly (Rust orphan rules), so the trait has a blanket implementation that delegates to this name-based registry. The registry is a proof obligation list — each entry is a function whose semantics have been verified to be evaluable on a decoded list column.

The residual filter: why “keep it” is mandatory

Neither of the two systems is allowed to drop a conjunct. If a conjunct cannot be evaluated during decoding, it must still be evaluated somewhere, or the query returns rows that do not satisfy the predicate. DataFusion encodes this in the filter pushdown protocol:

 rust
 
 
// datafusion/datasource-parquet/src/source.rs:853-872
let filters: Vec<PushedDownPredicate> = filters.into_iter()
 .map(|filter| {
 if can_expr_be_pushed_down_with_schemas(&filter, pushable_schema) {
 PushedDownPredicate::supported(filter)
 } else {
 PushedDownPredicate::unsupported(filter)
 }
 })
 .collect();

PushedDown::No means “I did not evaluate this; you must”. The parent FilterExec stays in the plan and applies it to the batches the scan produces. (The same holds when pushdown is disabled entirely: the predicate is still used for statistics pruning, but the source reports PushedDown::No for everything so the parent keeps its filter, source.rs:880-896.)

There is a second, subtler reason to keep the residual: the source’s claims about its own output must be true. FileSource distinguishes filter() (the predicate the source uses) from exact_filter() (the part of it that every output row is guaranteed to satisfy):

 rust
 
 
// datafusion/datasource/src/file.rs:113-128
/// Returns the part of [`Self::filter`] that every output row is
/// guaranteed to satisfy.
///
/// [`FileScanConfig`] derives equivalence properties (constants and equal
/// columns) from this filter. A filter that the source uses only to skip
/// data (for example, row group or page pruning with statistics) does not
/// remove every non-matching row, so it must not be returned here.

If the source claimed the whole predicate as exact, the optimizer would be allowed to conclude that c = 5 implies c is constant, and would rewrite the plan accordingly. If the predicate was only used for pruning, that conclusion is false and the rewritten plan returns wrong rows. So exact_filter returns only the conjuncts that were genuinely evaluated during decoding:

 rust
 
 
// datafusion/datasource-parquet/src/source.rs:682-694
fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
 if !self.pushdown_filters() { return None; }
 let predicate = self.predicate.as_ref()?;
 let pushable_schema = self.table_schema.schema_without_virtual_columns();
 let exact = split_conjunction(predicate)
 .into_iter()
 .filter(|expr| can_expr_be_pushed_down_with_schemas(expr, pushable_schema))
 .cloned()
 .collect::<Vec<_>>();
 (!exact.is_empty()).then(|| conjunction(exact))
}

The same discipline shows up inside the pruning rewrite itself. Some rewrites preserve only “which rows can be TRUE” but lose the distinction between FALSE and UNKNOWN — for example NOT IN with a NULL member, or a CASE without an ELSE. The rewriter flags those with has_filter_semantics_only (pruning_predicate.rs:390-396), and such a predicate is fine for pruning (it only needs “can it be true?”), but useless for proving that every row matches — so the inversion that powers full-match inference is refused for it (pruning_predicate.rs:570, can_be_inverted_for_full_match).

The catch

Support is per expression, not per column. A column can be pushable in one predicate and not in another: s['v'] > 5 is pushable, s IS NOT NULL is not; array_has_all(tags, ...) is pushable, tags[1] = 'x' may not be. Any classification logic that works on columns rather than expressions is wrong.
Schema adaptation can change the answer. The pushdown check runs against the physical file schema after casts have been inserted, which is why pushdown_columns takes an allow_struct_casts flag: “Struct casts are accepted only after schema adaptation, not while planning against the table schema” (row_filter.rs:247-249).
Three-valued logic is not optional. The rewrite table above shows x_null_count != x_row_count in every equality. Drop it and a column that is entirely NULL would look like it might contain 5.
The unhandled hook hides bugs. Because an unsupported predicate silently becomes “keep everything”, a regression in the rewriter shows up as a performance regression, not a correctness one. DataFusion mitigates this with counters (predicate_creation_errors, predicate_evaluation_errors) and with PruningPredicateBuilder’s try_build variant that surfaces the error instead of swallowing it.
Unsupported does not mean free. Every unsupported conjunct still costs a residual FilterExec over the output batches — the cheapest place to evaluate it, but not free, and the reason pushdown is off by default.

The through-line: every classification decision in this pipeline is optimistic only about performance and pessimistic about correctness. Supported means “I can prove this or evaluate this”; unsupported means “keep the data and keep the filter”. A system that errs in the other direction would be faster and wrong, and no amount of benchmarking would tell you.

Bitwise Semi Joins: Tracking Matches in a Bitset Instead of Materialising Row Pairs

Jay Hubbert — Wed, 21 Oct 2026 08:05:00 +0000

TL;DR: A semi join (WHERE EXISTS (...)) answers a yes/no question per outer row. The naive sort-merge implementation evaluates the filter for every (outer, inner) pair and materialises the matching pairs. If a key group has a million inner rows and a million outer rows, that is 10^12 pairs. DataFusion’s BitwiseSortMergeJoinStream instead keeps a bit-packed bitmap of which outer rows matched, ORs the filter results into it 64 bits at a time, and counts the set bits with popcnt. The inner rows are streamed past, never materialised.

A semi join is WHERE EXISTS (subquery). An anti join is WHERE NOT EXISTS (subquery). A mark join is SELECT expr, EXISTS(...) FROM ....

All three have the same shape: for each outer row, determine whether some inner row matches. The answer is one bit.

The naive sort-merge implementation is a nested loop over key groups: for each outer key group, for each inner row in the matching inner key group, evaluate the filter, and if it passes, emit the (outer, inner) pair. For an inner join, that is exactly right — you need the pairs. For a semi join, it is absurd: you need one bit per outer row, and you are materialising pairs to get it.

The bitwise stream

datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs (1,328 lines) is the answer:

 rust
 
 
/// Sort-Merge join stream for Semi/Anti/Mark joins.
///
/// Named "bitwise" because it tracks outer-row matches via a per-batch
/// boolean bitset (`BooleanBufferBuilder`) rather than materializing
/// `(outer, inner)` row pairs. Filter results are OR'd into the bitset
/// in `u64` chunks, and emitting applies the bitset directly.
pub(crate) struct BitwiseSortMergeJoinStream {

and the bitset itself, with the reasons spelled out:

 rust
 
 
 // Per-outer-batch match tracking, reused across batches.
 // Bit-packed (not Vec<bool>) so that:
 // - emit: finish() yields a BooleanBuffer directly (no packing iteration)
 // - OR: apply_bitwise_binary_op ORs filter results in u64 chunks
 // - count: UnalignedBitChunk::count_ones uses popcnt
 matched: BooleanBufferBuilder,

Three reasons, one per use. The bitset is not just a compact Vec<bool> — it is a BooleanBufferBuilder, so the output can be a BooleanBuffer directly, with no unpacking pass.

ORing 64 bits at a time

The core operation is merging the filter’s result into the accumulator:

 rust
 
 
// OR filter results into the matched bitset. Both sides are
// bit-packed [u8] buffers, so apply_bitwise_binary_op
// processes 64 bits per loop iteration (not 1 bit at a time).
//
// The offsets handle alignment: outer_offset is the bit
// position within matched where this key group starts,
// and filter_buf.offset() is the BooleanBuffer's internal
// bit offset (usually 0, but not guaranteed by Arrow).
let filter_buf = filter_result.values();
apply_bitwise_binary_op(
 matched.as_slice_mut(),
 outer_offset,
 filter_buf.inner().as_slice(),
 filter_buf.offset(),
 outer_group_len,
 |a, b| a | b,
);

apply_bitwise_binary_op is Arrow’s bitwise kernel: it applies a binary operation to two bit-packed buffers, 64 bits (one u64) per iteration. This is the same family of operation as Arrow’s and/or kernels on BooleanArray, and it is vectorised — the CPU processes a whole u64 of bits per instruction.

The offsets are the fiddly part, and the comment explains why they cannot be ignored: outer_offset is the bit position in the accumulator where this key group starts, and filter_buf.offset() is Arrow’s internal bit offset, “usually 0, but not guaranteed by Arrow”. Arrow’s BooleanBuffer can be a view into a larger buffer at a non-zero bit offset, and a naive implementation that assumes byte alignment produces garbage.

Counting with popcnt

How many outer rows have matched so far? You need to know, because the loop can stop early once every row in the group has matched:

 rust
 
 
// Count already-matched bits using popcnt on u64 chunks (zero-copy).
let mut matched_count = UnalignedBitChunk::new(
 self.matched.as_slice(),
 outer_group_start,
 outer_group_len,
)
.count_ones();

UnalignedBitChunk reads the bytes in place — zero-copy — and uses popcnt to count the set bits. popcnt is a single instruction on any CPU from the last two decades, so counting a million bits is a few thousand instructions.

The recount after each OR is the same:

 rust
 
 
// Recount matched bits after the OR. UnalignedBitChunk is
// zero-copy — it reads the bytes in place and uses popcnt.
matched_count =
 UnalignedBitChunk::new(self.matched.as_slice(), outer_offset, outer_group_len)
 .count_ones();

And the early exit:

 rust
 
 
while matched_count < outer_group_len {
 match spill_stream.next().await {
 ...

If every outer row in the group has already matched, there is no point reading the rest of the inner rows. The inner side is streamed past and discarded.

The nested-loop framing

The doc on the struct explains the terminology, which is helpful because sort-merge join is usually described as a merge rather than a nested loop:

 rust
 
 
 // Input streams — in the nested-loop model that sort-merge join
 // implements, "outer" is the driving loop and "inner" is probed for
 // matches. The existing MaterializingSortMergeJoinStream calls these "streamed"
 // and "buffered" respectively. For Left* joins, outer=left; for
 // Right* joins, outer=right. Output schema equals the outer side.
 outer: SendableRecordBatchStream,
 inner: SendableRecordBatchStream,

So for a RightSemi join, the right side is the “outer” and the output schema is the right side’s — which is why there is a flag:

 rust
 
 
 // Decomposed from JoinType: when RightSemi/RightAnti, outer=right,
 // inner=left, so we swap sides when building the filter batch.
 outer_is_left: bool,

The inner key group still has to be buffered

The bitset saves the output memory. It does not save the input memory, because to evaluate the filter for an outer row against an inner row, you need the inner row.

 rust
 
 
 // Inner key group buffer: all inner rows sharing the current join key.
 // Only populated when a filter is present. Unbounded — a single key
 // with many inner rows will buffer them all. See "Degenerate cases"
 // in exec.rs. On memory pool overflow the buffered slices move to a
 // per-group spill file (see [`Self::buffer_inner_key_group`]).
 inner_key_buffer: Vec<RecordBatch>,

“Unbounded — a single key with many inner rows will buffer them all.” And the escape hatch is a per-group spill file: when the memory pool overflows, the buffered slices are written to a spill file for that key group and read back when the filter is evaluated.

That is a nice design. The spill is scoped to a single key group rather than to the whole join, so at most one group’s worth of inner rows is ever on disk, and the spill file is deleted as soon as the group is done.

And the clock is handled carefully across the spill read:

 rust
 
 
// Note: the clock keeps running across the spill reads — the
// spill file is the join's own data, so reading it back is
// join work (unlike the child inputs' `next()`).

That is a subtle but correct distinction. The join_time metric is stopped while awaiting the child inputs — that is the operator waiting on its producer, not working — but it keeps running while reading back the join’s own spill file, because that is the join doing work.

The comparators are cached

 rust
 
 
 // Cached comparators — pre-built to avoid per-row type dispatch.
 /// Comparator for outer vs inner key comparison
 outer_inner_cmp: Option<JoinKeyComparator>,
 /// Comparator for outer self-comparison (find_key_group_end on outer)
 outer_self_cmp: Option<JoinKeyComparator>,
 /// Comparator for inner self-comparison (find_key_group_end on inner)
 inner_self_cmp: Option<JoinKeyComparator>,

Three comparators, cached. Building an Arrow comparator involves a downcast to the concrete array type and a closure; doing that per row would dominate the join. Caching them is the difference between a sort-merge join that is competitive with a hash join and one that is not.

And the group-boundary search is a binary search over the key:

 rust
 
 
let outer_group_end =
 find_key_group_end(self.get_outer_self_cmp()?, outer_group_start, num_outer);

which is a partition_point-style search — O(log n) instead of a linear scan to find where the key group ends.

The catch

The bitset is per outer batch. It is sized to the current outer batch and reused across batches, so the memory is O(batch size), not O(rows). But it also means the “already matched” state does not survive a batch boundary — which is fine, because the state that matters is per key group, and a key group is within a batch.
The inner key group is still unbounded. The bitset does not help with a key group with a million inner rows; that is what the per-group spill file is for.
It only applies to semi/anti/mark joins. An inner join genuinely needs the pairs. The two implementations coexist:
rust
```
unreachable!("Semi/anti/mark joins are handled by BitwiseSortMergeJoinStream")
```
appears in the other stream, which refuses to handle them.
The alignment handling is subtle. The filter_buf.offset() is “usually 0, but not guaranteed by Arrow”, and getting it wrong produces wrong results rather than a crash.

Why it matters

This is a good example of a general principle: when the answer is much smaller than the work, do not materialise the work.

A semi join’s answer is one bit per outer row. The naive implementation materialises a row pair per match — and in the worst case, a row pair per candidate, matched or not. The bitwise implementation turns the inner side from a source of memory pressure into a source of nothing at all: the inner rows are read, filtered, and discarded, and the only state that survives is a bitmap.

And the bitmap is not a Vec<bool> — it is bit-packed, so the OR is a 64-bit vectorised operation, the count is a popcnt, and the output is the buffer itself. Every part of it was chosen to make the common case free.

DataFusion's Memory Pool: RAII Reservations, Fair Shares, and Who Gets to Spill

Jay Hubbert — Wed, 21 Oct 2026 07:57:00 +0000

TL;DR: DataFusion does not instrument malloc. Instead every “big” operator registers a MemoryConsumer with a MemoryPool and holds a MemoryReservation — an RAII handle backed by an AtomicUsize that returns its bytes to the pool on Drop. try_grow is the negotiation point: success means “allocate now”, failure means “spill, shrink, or error”. FairSpillPool splits the budget between consumers that declare can_spill(true) and those that don’t, so a single operator can’t starve the rest of the plan.

Note: this tree has moved the pool out of datafusion/common/src/memory.rs. The trait and the reservation types now live in datafusion/execution/src/memory_pool/mod.rs, the concrete pools in datafusion/execution/src/memory_pool/pool.rs, and the low-level size estimation helpers in datafusion/common/src/utils/memory.rs.

The problem with counting allocations

The “obvious” design for a query engine memory limiter is to install a counting global allocator and charge every byte to the running query. It is also the design that does not work, because:

A Vec<u8> reallocating from 1 KiB to 2 KiB is charged twice unless you track capacity, not length.
Arrow’s Buffer model means two Arrays in the same RecordBatch can be zero-copy slices of the same allocation. Charging each Array double counts.
Arrow StringView/BinaryView arrays keep views into shared data buffers; slicing them retains far more memory than the slice “uses”.
The interesting question is not “who allocated this” but “which operator is holding a large working set that I could ask to shrink”.

DataFusion takes the pragmatic route, documented at length in the module header of datafusion/execution/src/memory_pool/mod.rs:49-89: streaming memory is not accounted, only the operators whose footprint grows with input rows (GroupByHashExec, SortExec, CrossJoin, …) register and constrain themselves. Everything else is “small”.

That leaves exactly three questions to answer:

How does an operator declare how much it wants?
How does the pool decide who gets it?
What happens when the answer is no?

The three types

`MemoryPool` — the policy

 rust
 
 
// datafusion/execution/src/memory_pool/mod.rs:195-232
pub trait MemoryPool: Any + Send + Sync + std::fmt::Debug + Display {
 fn name(&self) -> &str;
 fn register(&self, _consumer: &MemoryConsumer) {}
 fn unregister(&self, _consumer: &MemoryConsumer) {}
 fn grow(&self, reservation: &MemoryReservation, additional: usize);
 fn shrink(&self, reservation: &MemoryReservation, shrink: usize);
 fn try_grow(&self, reservation: &MemoryReservation, additional: usize) -> Result<()>;
 fn reserved(&self) -> usize;
 fn memory_limit(&self) -> MemoryLimit { MemoryLimit::Unknown }
}

The trait is deliberately tiny and, crucially, the pool is told who is asking. reservation.registration.consumer is reachable from inside the pool, so a policy can make per-consumer decisions. That is the whole trick behind FairSpillPool.

grow is infallible by contract; try_grow is the one that can fail, and it must not increase the reservation on error. Failures are DataFusionError::ResourcesExhausted (pool.rs:304-318).

`MemoryConsumer` — the identity

 rust
 
 
// datafusion/execution/src/memory_pool/mod.rs:270-277
pub struct MemoryConsumer {
 name: String,
 can_spill: bool,
 id: usize,
}

id is a process-unique counter (new_unique_id, mod.rs:296-300) — a consumer is not cloneable, because “the consumer” is the unit of fairness. clone_with_new_id (mod.rs:318-324) exists for the rare case where you want a second, independent registration.

The can_spill flag is the interesting one, and it is doing double duty. Its literal meaning is “this consumer can push its state to disk”. Its effective meaning, as the code comments repeatedly say, is “this consumer can handle memory back pressure”:

 rust
 
 
// datafusion/physical-plan/src/aggregates/grouped_hash_stream.rs:558-563
// We interpret 'can spill' as 'can handle memory back pressure'.
// This value needs to be set to true for the default memory pool implementations
// to ensure fair application of back pressure amongst the memory consumers.
.with_can_spill(oom_mode != OutOfMemoryMode::ReportError),

`MemoryReservation` — the RAII handle

 rust
 
 
// datafusion/execution/src/memory_pool/mod.rs:382-387
pub struct MemoryReservation {
 registration: Arc<SharedRegistration>,
 size: atomic::AtomicUsize,
}

Two fields, and both earn their keep.

Arc<SharedRegistration> holds the pool and the consumer. SharedRegistration has a Drop impl that calls pool.unregister(&self.consumer) (mod.rs:370-374), so the last reservation of a consumer automatically unregisters it — the pool never sees a dangling consumer. That is why MemoryConsumer is not cloneable but MemoryReservation is cheap to derive more of: new_empty() and split() mint sibling reservations that share the same registration (mod.rs:496-522).

size: AtomicUsize is the reservation’s own byte count. grow/try_grow bump it after the pool agrees (mod.rs:472-485); free swaps it to zero and tells the pool to shrink (mod.rs:400-407).

 rust
 
 
// datafusion/execution/src/memory_pool/mod.rs:525-528
impl Drop for MemoryReservation {
 fn drop(&mut self) {
 self.free();
 }
}

That Drop is the entire safety story. An operator cannot leak a reservation by returning early from an error path, by panicking, or by ?-ing its way out of a nested call. The bytes go back to the pool when the handle dies, no matter how.

The counters use Ordering::Relaxed throughout. That is safe because the pool’s own counter is the single source of truth for admission decisions, and the reservation’s size is only read for reporting (FairSpillPool does read reservation.size() inside its try_grow, under the pool mutex, but the reservation is by convention owned by one task at a time).

The three pools

`UnboundedMemoryPool`

pool.rs:33-61. A single AtomicUsize that goes up and down and never says no. It is the default, and it exists so that the accounting is still correct even when the limit is not enforced — useful for observability, and it is the fallback when a MergeMemoryPool wants to bypass the real pool (sorts/streaming_merge.rs:199-208).

`GreedyMemoryPool`

pool.rs:77-129. Classic first-come-first-served:

 rust
 
 
// datafusion/execution/src/memory_pool/pool.rs:106-118
fn try_grow(&self, reservation: &MemoryReservation, additional: usize) -> Result<()> {
 self.used
 .try_update(Ordering::Relaxed, Ordering::Relaxed, |used| {
 let new_used = used + additional;
 (new_used <= self.pool_size).then_some(new_used)
 })
 .map_err(|used| {
 insufficient_capacity_err(reservation, additional,
 self.pool_size.saturating_sub(used), self)
 })?;
 Ok(())
}

The try_update is a compare-and-swap loop, so two operators racing for the last megabyte cannot both win. Greedy is right when there is exactly one spillable operator: it hands out everything to whoever asks first, and the spiller will spill only when it truly must.

`FairSpillPool`

pool.rs:145-284. This is the interesting one, because it does not hand out memory first-come-first-served.

 rust
 
 
// datafusion/execution/src/memory_pool/pool.rs:236-277
fn try_grow(&self, reservation: &MemoryReservation, additional: usize) -> Result<()> {
 let mut state = self.state.lock();
 match reservation.registration.consumer.can_spill {
 true => {
 // The total amount of memory available to spilling consumers
 let spill_available = self.pool_size.saturating_sub(state.unspillable);
 // No spiller may use more than their fraction of the memory available
 let available = spill_available
 .checked_div(state.num_spill)
 .unwrap_or(spill_available);

 if reservation.size() + additional > available {
 return Err(insufficient_capacity_err(reservation, additional, available, self));
 }
 state.spillable += additional;
 }
 false => {
 let available = self
 .pool_size
 .saturating_sub(state.unspillable + state.spillable);
 if available < additional {
 return Err(insufficient_capacity_err(reservation, additional, available, self));
 }
 state.unspillable += additional;
 }
 }
 Ok(())
}

The policy is: memory that nobody can spill is allocated greedily; the remainder is divided equally among the spillable consumers. num_spill is maintained by register/unregister (pool.rs:207-217) — the count of live spillable consumers, not a count of reservations.

The rationale is a classic livelock. Suppose a plan has two hash aggregations and a SortExec that cannot spill. With a greedy pool, the sort grabs everything, the aggregations get nothing, and both spill — writing gigabytes to disk even though the sort would have been fine with a fraction. With the fair pool, each aggregator is capped at (pool_size - unspillable) / num_spill, so both can make progress on the memory they do hold, and the sort keeps what it needs.

The doc comment is refreshingly honest about the cost:

 rust
 
 
// datafusion/execution/src/memory_pool/pool.rs:145-166
/// A [`MemoryPool`] that prevents spillable reservations from using more than
/// an even fraction of the available memory sans any unspillable reservations
/// (i.e. `(pool_size - unspillable_memory) / num_spillable_reservations`)
///
/// This pool works best when you know beforehand the query has
/// multiple spillable operators that will likely all need to
/// spill. Sometimes it will cause spills even when there was
/// sufficient memory (reserved for other operators) to avoid doing
/// so.

“Sometimes it will cause spills even when there was sufficient memory” — that is the price of determinism, and it is paid on purpose.

`MergeMemoryPool`: one consumer, many reservations

A subtler problem appears when a single logical operator needs several reservations that share a budget. The merge phase and the aggregate replay phase of a spilled aggregation are two components of one operator; charging them separately would let FairSpillPool divide the operator’s own quota between its own phases.

 rust
 
 
// datafusion/execution/src/memory_pool/merge_memory_pool.rs:38-52
pub struct MergeMemoryPool {
 parent: Arc<dyn MemoryPool>,
 state: Mutex<MergeMemoryState>,
}

struct MergeMemoryState {
 /// The only reservation charged to the execution pool.
 reservation: MemoryReservation,
 used: usize,
 /// Workspace already acquired for the next merge. It is reusable by any
 /// child reservation, but unavailable to other execution-pool consumers.
 retained: usize,
}

The pool charges one reservation to the parent. Children call try_grow on child reservations and the pool tops up the parent only when usage exceeds what is already reserved. retain(n) keeps workspace alive across child drops; borrow(n) hands out a WorkspaceLoan — a temporary claim on already-reserved bytes, so a merge cursor can hold a buffer without a second round trip to the parent pool (merge_memory_pool.rs:54-135).

This is the pattern the aggregate spill path uses to keep the merge buffers and the replay table under a single consumer registration (aggregates/spill.rs:273-283).

What happens when `try_grow` fails

The failure mode is entirely operator-specific, which is by design: the pool has no opinion about what “recovering” means.

Hash aggregate (aggregates/hash_stream.rs:707-750): on ResourcesExhausted, it takes the entire in-memory state out of the hash table, sorts it, writes it to a spill file, shrinks the reservation, and retries the same input loop. Only if the DiskManager has temp files disabled does it turn into a hard error: "Final hash aggregate cannot spill because temporary files are not enabled in the DiskManager".
Cross join / operators without spilling: the error propagates. The module docs spell out the difference in the three scenarios (memory_pool/mod.rs:90-113).
Repartition (repartition/mod.rs:236-247): falls back to writing the batch to a spill file instead of erroring.

Note the loop shape: it is not “reserve, fail, error”. It is “reserve, fail, release memory, retry”. The MemoryReservation API makes this ergonomic because try_resize (mod.rs:456-468) shrinks and grows in one call, and shrink/try_shrink return the freed bytes to the pool immediately.

Getting the accounting right

Two operators sharing a Buffer must not be charged twice. DataFusion solves this with a buffer-identity set:

 rust
 
 
// datafusion/common/src/utils/memory.rs:152-155
pub fn get_record_batch_memory_size(batch: &RecordBatch) -> usize {
 RecordBatchMemoryCounter::new().count_batch(batch)
}

RecordBatchMemoryCounter (memory.rs:167-200) tracks start addresses of already-counted buffers, so a batch that is a slice of another batch costs only the delta. This matters directly for spilling, where the same underlying buffer can be referenced by a batch and by the state that was derived from it.

Drift: are we lying to ourselves?

MemoryDriftTracker (memory_pool/drift.rs:47-71) compares the process’s actual allocated bytes against the total reserved by all pools and logs when the difference grows. PeakRecordingPool wraps another pool and records peak usage plus the consumer that caused it.

This exists because the pool’s number is a model, not a measurement. A mis-sized estimate (estimate_memory_size, memory.rs:99-121, deliberately over-allocating buckets by 8/7 to keep hashbrown ~1/8 empty) means the pool can be off by a few percent. Drift tracking turns “we think we used 2 GiB” into “the allocator says 2.4 GiB, here is who is responsible”.

Trade-offs and failure modes

Accounting is opt-in and coarse. Memory used by streaming RecordBatches between operators is not tracked at all. The pool’s limit is a limit on state, not on RSS. Configure the pool at ~80% of your container limit, or accept the OOM-killer.
Fair division is static. FairSpillPool divides by the number of spillable consumers, not by their weight. Ten tiny spillers will halve the budget of one large one.
The can_spill flag is a promise. If a consumer claims it can spill and then cannot, the fair pool has already reserved room it will never use.
RAII hides the accounting from readers. reservation.try_grow(n) is not obviously “I am about to allocate n bytes”, and a reservation that is never shrunk keeps its bytes forever. MemoryReservation::size() and the pool’s reserved() are the tools for finding leaks; TrackConsumersPool (pool.rs:423-520) exists precisely to name the top offenders.
Relaxed atomics + Mutex state. FairSpillPool takes a parking_lot mutex on every grow/shrink. That is fine at the granularity of “one reservation change per batch”, and terrible if you call it per row.

The upshot is a system that is not trying to be a general-purpose allocator accountant. It is a quota system with RAII: operators ask, the pool grants or denies, and denial is a normal, expected control-flow event rather than an error. That is the design decision that makes spilling, sorting and joins possible at all — because in a system where memory denial is exceptional, operators must be written to expect it.

DataFusion Swaps Join Sides on Statistics and Applies Non-Equi Filters After the Hash Lookup

Jay Hubbert — Wed, 21 Oct 2026 07:49:00 +0000

TL;DR: A hash join’s build side must fit in memory; the probe side is streamed. So which side is the build side? DataFusion’s JoinSelection physical rule compares total_byte_size (falling back to num_rows) of the two inputs and swaps them if the left is bigger. Swapping is not a free rename: the join type flips, the equi-key pairs flip, the residual filter’s column sides flip, and the output projection is restored. Separately, a join predicate that is not an equality (a.x < b.y) cannot be a hash key, so it becomes a JoinFilter that is evaluated after the hash lookup, on the index pairs that survived it.

Two different “swap” mechanisms

People use “swapped join” to mean two quite different things in DataFusion, and they live in the same file, datafusion/physical-optimizer/src/join_selection.rs:

Statistics-driven swap — the better side becomes the build side. This is a pure performance decision.
Unboundedness-driven swap — the finite side becomes the build side so the query can run at all. This is a correctness/runnability decision.

Both end up calling the same primitive, HashJoinExec::swap_inputs, and both have to undo a surprising amount of metadata surgery.

Statistics-driven swapping

The decision itself is tiny:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:84
pub(crate) fn should_swap_join_order(
 left: &Arc<dyn ExecutionPlan>,
 right: &Arc<dyn ExecutionPlan>,
 context: &dyn PhysicalOptimizerContext,
) -> Result<bool> {
 if !context.config_options().optimizer.join_reordering {
 return Ok(false);
 }

 let left_stats = get_stats(left, context)?;
 let right_stats = get_stats(right, context)?;

 // First compare total_byte_size, then fall back to num_rows if byte
 // sizes are unavailable.
 match (
 left_stats.total_byte_size.get_value(),
 right_stats.total_byte_size.get_value(),
 ) {
 (Some(l), Some(r)) => Ok(l > r),
 _ => match (
 left_stats.num_rows.get_value(),
 right_stats.num_rows.get_value(),
 ) {
 (Some(l), Some(r)) => Ok(l > r),
 _ => Ok(false),
 },
 }
}

Three things to notice, all of them load-bearing:

Bytes beat rows. A table with 10M narrow rows can be smaller than a table with 1M wide rows, and it is the bytes that have to fit in the memory pool. Comparing row counts would pick the wrong build side.
Unknown statistics mean no swap. Absent statistics yield false. DataFusion never guesses: if it does not know, it leaves the plan as the user wrote it.
It is opt-out-able. datafusion.optimizer.join_reordering (default true) disables the whole thing.

Note the deliberate conservatism: the rule compares sizes, not estimated cardinality after the join. It does not try to predict which side produces fewer output rows, because that estimate is far less reliable than a row count.

CollectLeft vs Partitioned: the other half of the decision

Swapping is only half of it. Even with the small side on the left, DataFusion has to decide whether to collect that side into one partition or partition it:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:192
pub(crate) fn try_collect_left(
 hash_join: &HashJoinExec,
 ignore_threshold: bool,
 context: &dyn PhysicalOptimizerContext,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
 ...
 let left_can_collect = ignore_threshold
 || supports_collect_by_thresholds(
 left,
 optimizer_config.hash_join_single_partition_threshold,
 optimizer_config.hash_join_single_partition_threshold_rows,
 context,
 );
 ...
 match (left_can_collect, right_can_collect) {
 (true, true) => { /* swap if the right is smaller, then CollectLeft */ }
 (true, false) => { /* CollectLeft, no swap possible */ }
 (false, true) => { /* swap, then CollectLeft */ }
 (false, false) => Ok(None), // fall through to partitioned
 }
}

The thresholds are datafusion.optimizer.hash_join_single_partition_threshold (4 MiB) and hash_join_single_partition_threshold_rows (128K rows). If the small side fits in one partition, PartitionMode::CollectLeft is used: the build side is gathered into a single partition, and every probe partition shares it. That is strictly better than N independent build sides when the build side is tiny — you hash the build rows once, not N times — but it serialises the build.

If neither side fits, PartitionMode::Partitioned is used and both sides are hash-repartitioned on the join key (see the sibling post on the partitioned hash table).

What `swap_inputs` actually has to fix

Swapping two inputs of a join is not symmetric. This is where most hand-rolled “swap the inputs” code goes wrong, and DataFusion’s implementation is a good catalogue of the cases:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/exec.rs:1521
 pub fn swap_inputs(
 &self,
 partition_mode: PartitionMode,
 ) -> Result<Arc<dyn ExecutionPlan>> {
 assert_or_internal_err!(
 self.dynamic_filter.is_none(),
 "Cannot swap HashJoinExec inputs after dynamic filters have been constructed. \
 Optimizer rules that reorder join inputs must run before optimizer rules `FilterPushdown::new_post_optimization()`"
 );

 let left = self.left();
 let right = self.right();
 let new_join = self
 .builder()
 .with_type(self.join_type.swap())
 .with_new_children(vec![Arc::clone(right), Arc::clone(left)])?
 .with_on(
 self.on()
 .iter()
 .map(|(l, r)| (Arc::clone(r), Arc::clone(l)))
 .collect(),
 )
 .with_filter(self.filter().map(JoinFilter::swap))
 .with_projection(swap_join_projection(
 left.schema().fields().len(),
 right.schema().fields().len(),
 self.projection.as_deref(),
 self.join_type(),
 ))
 .with_partition_mode(partition_mode)
 .build()?;

Five separate fixes:

The join type flips. Left becomes Right, LeftSemi becomes RightSemi, and so on. This is not cosmetic: Left and Right have completely different “emit unmatched rows at the end” behaviour, and the build side of a Right join is the side whose unmatched rows must be produced at the end.
The equi-key pairs flip. (l, r) becomes (r, l). Trivial but mandatory.
The residual filter flips. This is the subtle one:

 rust
 
 
// datafusion/physical-plan/src/joins/join_filter.rs:91
 /// Rewrites the join filter if the inputs to the join are rewritten
 pub fn swap(&self) -> JoinFilter {
 let column_indices = self
 .column_indices()
 .iter()
 .map(|idx| ColumnIndex {
 index: idx.index,
 side: idx.side.negate(),
 })
 .collect();

 JoinFilter::new(
 Arc::clone(self.expression()),
 column_indices,
 Arc::clone(self.schema()),
 )
 }

The expression is unchanged — a.x < b.y is still a.x < b.y — but every column reference records which side it came from, and that side is negated. The JoinFilter keeps an explicit side per column precisely so this rewrite is mechanical rather than a re-planning exercise.

The projection is remapped, and then
the output order is restored:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/exec.rs:1554
 // Semi / anti / mark joins output columns from one side only, so
 // swapping does not reorder them; an embedded projection already fixes
 // the order too. Otherwise a projection restores the original order.
 if is_existence_join(self.join_type) || self.projection.is_some() {
 Ok(Arc::new(new_join))
 } else {
 reorder_output_after_swap(Arc::new(new_join), &left.schema(), &right.schema())
 }

A naive swap would emit the right side’s columns first. The user asked for left.*, right.*; the swapped plan naturally produces right.*, left.*, so a projection is bolted on to put them back. Semi/anti/mark joins are exempt because their output schema comes from one side only and does not move.

When swapping is forbidden

Not every join can be swapped:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:173
/// Determines whether it is possible to swap inputs of a hash join - for null-aware joins, we can only swap an uncorrelated `LeftAnti`
/// (a single join key and no filter), because the swapped `RightAnti` has no per-row NULL handling
fn can_swap_hash_join(hash_join: &HashJoinExec) -> bool {
 hash_join.join_type().supports_swap()
 && (!hash_join.null_aware
 || (*hash_join.join_type() == JoinType::LeftAnti
 && hash_join.on().len() == 1
 && hash_join.filter().is_none()))
}

A null-aware anti join (NOT IN against a nullable subquery) has per-row NULL semantics that only the LeftAnti implementation supports; the swapped RightAnti does not. And once a dynamic filter has been constructed for the join, the build side is already baked in — the assert_or_internal_err! above fires, which is why the JoinSelection rule is documented to run before the filter-pushdown rules.

The unboundedness swap: a different motivation

The other swap is not about performance at all. A hash join is a pipeline breaker: it must consume its build side completely before it can emit anything. If the build side is an unbounded stream, the query never produces a row. The fix is to make sure the finite side is the build side:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:524
pub fn hash_join_swap_subrule(
 mut input: Arc<dyn ExecutionPlan>,
 _config_options: &ConfigOptions,
) -> Result<Arc<dyn ExecutionPlan>> {
 if let Some(hash_join) = input.downcast_ref::<HashJoinExec>()
 && hash_join.dynamic_expressions_produced().is_empty()
 && hash_join.left.boundedness().is_unbounded()
 && !hash_join.right.boundedness().is_unbounded()
 && !hash_join.null_aware
 && matches!(
 *hash_join.join_type(),
 JoinType::Inner | JoinType::Left | JoinType::LeftSemi | JoinType::LeftAnti
 )
 {
 input = swap_join_according_to_unboundedness(hash_join)?;
 }
 Ok(input)
}

The join-type restriction is the interesting part: Right, RightSemi, RightAnti and Full cannot be made runnable by swapping, because the output schema of those joins is anchored to the right side. Swapping would change what the query returns. So for those, the plan stays broken (or the user gets a nested loop join). The comment in the source says it plainly:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:543
/// This function swaps sides of a hash join to make it runnable even if one of
/// its inputs are infinite. Note that this is not always possible; i.e.
/// [`JoinType::Full`], [`JoinType::Right`], [`JoinType::RightAnti`] and
/// [`JoinType::RightSemi`] can not run with an unbounded left side, even if
/// we swap join sides.

Now the residual filter: the predicate that is not a key

An equi-join predicate a.id = b.id is a hash key: DataFusion hashes the key columns and uses the hash to find candidates. A predicate like a.ts < b.ts or a.x + b.y > 10 cannot be a hash key — there is no single value to hash. Such predicates are separated out at planning time and become the join’s filter.

The physical planner counts the range predicates while it builds the filter:

 rust
 
 
// datafusion/core/src/physical_planner.rs:1502
 let join_filter = match filter {
 Some(expr) => {
 let split_expr = split_conjunction(expr);
 for expr in split_expr.iter() {
 match *expr {
 Expr::BinaryExpr(BinaryExpr { left: _, right: _, op }) => {
 if matches!(op,
 Operator::Lt | Operator::LtEq
 | Operator::Gt | Operator::GtEq)
 {
 range_filters.push((**expr).clone());
 num_range_filters += 1;
 }
 total_filters += 1;
 }
 _ => { total_filters += 1; }
 }
 }

and then builds a JoinFilter holding the physical expression, the list of columns it needs (each tagged with the side it came from), and the schema of the intermediate batch those columns form:

 rust
 
 
// datafusion/core/src/physical_planner.rs:1595
 let column_indices = join_utils::JoinFilter::build_column_indices(
 left_field_indices,
 right_field_indices,
 );

 Some(join_utils::JoinFilter::new(
 filter_expr,
 column_indices,
 Arc::new(filter_schema),
 ))

The trick: evaluate the filter on gathered index pairs

The filter is evaluated after the hash lookup, in the join stream, on the matched pairs:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/stream.rs:959
 // apply join filter if exists
 let (left_indices, right_indices) = if let Some(filter) = &self.filter {
 apply_join_filter_to_indices(
 build_side.left_data.batch(),
 &state.batch,
 left_indices,
 right_indices,
 filter,
 JoinSide::Left,
 None,
 self.join_type,
 )?
 } else {
 (left_indices, right_indices)
 };

and the implementation is the whole point:

 rust
 
 
// datafusion/physical-plan/src/joins/utils.rs:1225
pub(crate) fn apply_join_filter_to_indices(
 build_input_buffer: &RecordBatch,
 probe_batch: &RecordBatch,
 build_indices: UInt64Array,
 probe_indices: UInt32Array,
 filter: &JoinFilter,
 build_side: JoinSide,
 max_intermediate_size: Option<usize>,
 join_type: JoinType,
) -> Result<(UInt64Array, UInt32Array)> {
 ...
 let intermediate_batch = build_batch_from_indices(
 filter.schema(),
 build_input_buffer,
 probe_batch,
 &build_indices,
 &probe_indices,
 filter.column_indices(),
 build_side,
 join_type,
 None,
 )?;

 filter
 .expression()
 .evaluate(&intermediate_batch)?
 .into_array(intermediate_batch.num_rows())?
 ...
 let mask = as_boolean_array(&filter_result)?;

 let left_filtered = compute::filter(&build_indices, mask)?;
 let right_filtered = compute::filter(&probe_indices, mask)?;

Read that carefully, because it is a genuinely nice piece of engineering:

The hash lookup produced two index arrays: build_indices (which build row) and probe_indices (which probe row), one entry per candidate pair.
A synthetic intermediate batch is materialized by gathering exactly the columns the filter mentions, from the build side and from the probe side, at those indices. This batch has one row per candidate pair, and the filter expression is evaluated on it as a normal Arrow expression — vectorised, no per-row interpreter.
The resulting boolean mask is applied to both index arrays, so the surviving pairs are the join output.

The upshot: the filter is not a per-pair interpreted predicate. It is a batch Arrow computation over a gathered batch. The cost is one gather of the filter’s columns plus one vectorised comparison, per chunk of candidate pairs.

Note the max_intermediate_size parameter: the intermediate batch is built in slices so that a huge fan-out does not materialise a huge intermediate batch:

 rust
 
 
// datafusion/physical-plan/src/joins/utils.rs:1239
 let filter_result = if let Some(max_size) = max_intermediate_size {
 let mut filter_results =
 Vec::with_capacity(build_indices.len().div_ceil(max_size));

 for i in (0..build_indices.len()).step_by(max_size) {
 let end = min(build_indices.len(), i + max_size);
 let len = end - i;
 let intermediate_batch = build_batch_from_indices(...)?;

How the filter interacts with the probe side

Two details matter for the “does the filter change the probe semantics” question.

Semi/anti joins. The filter can turn a match into a non-match. For a LeftAnti join, “matched” means “matched and passed the filter”. The alignment logic that emits unmatched probe rows therefore has to run after the filter, and the code is explicit about it:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/stream.rs:983
 // The goals of index alignment for different join types are:
 //
 // 1) Right & FullJoin -- to append all missing probe-side indices between
 // previous (excluding) and current joined indices.
 // 2) SemiJoin -- deduplicate probe indices in range between previous
 // (excluding) and current joined indices.
 // 3) AntiJoin -- return only missing indices in range between previous
 // and current joined indices.

and the “last joined probe index” is computed from the filtered indices:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/stream.rs:998
 // if any rows have been joined -- get last joined probe-side (right) row
 // it's important that index counts as "joined" after hash collisions checks
 // and join filters applied.

Visited bitmaps. For outer joins, the build-side “visited” bitmap is updated from the filtered build indices, so a build row that matched the hash but failed the residual filter is still correctly reported as unmatched:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/stream.rs:975
 // mark joined left-side indices as visited, if required by join type
 if need_produce_result_in_final(self.join_type) {
 let mut bitmap = build_side.left_data.visited_indices_bitmap().lock();
 left_indices.iter().flatten().for_each(|x| {
 bitmap.set_bit(x as usize, true);
 });
 }

Note that this code sits after the filter application in the stream, so left_indices here is the filtered array. Getting this ordering wrong is a classic source of “outer join returns the row twice” bugs.

Trade-offs and pathological cases

Statistics can be wrong. The swap is driven by total_byte_size from Statistics, which for a filtered scan can be a gross underestimate. A bad estimate does not merely make the query slow — it makes the build side exceed the memory pool and the query fails, because the hash join has no spill path. Setting datafusion.optimizer.join_reordering = false is the escape hatch.
The filter’s column set drives the intermediate batch size. A filter that references a wide string column from both sides forces a gather of that column for every candidate pair. With a high-fanout key, the gather dominates the join.
A filter that is nearly always false makes the join look like a hash join with extra steps. The hash lookup still produces all the candidates; the filter then discards them. If the filter is selective, a different plan (a nested loop join over a range, or a piecewise merge join) would be better — and DataFusion does have that path: a join with no equi keys and exactly one range predicate is routed to PiecewiseMergeJoinExec when datafusion.optimizer.enable_piecewise_merge_join is on.
Semi/anti joins with a filter are the pathological case for the chained fast path: the filter must be evaluated for every candidate pair, so the per-pair cost does not amortise the way it does for a plain inner join.
Swapping changes the output column order unless the projection is restored. If you write your own optimizer rule that swaps inputs, you inherit all five fixes above, not just the children.

The takeaway

The swap is a memory decision masquerading as a statistics decision: bytes on the build side must fit in the pool, twice, transiently. The residual filter is a post-lookup decision: the hash finds candidates, the filter disqualifies them, and the disqualification happens as a vectorised Arrow computation over a gathered intermediate batch rather than as a per-pair interpreted predicate. Both tricks are about the same thing — keeping the amount of work proportional to the output, rather than to the cross product.

Splitting File Groups by Statistics: Giving Each Partition a Contiguous Slice of the Sort Key

Jay Hubbert — Wed, 21 Oct 2026 07:41:00 +0000

TL;DR: When a table’s files are sorted by a key, you can cut the file list into N partitions by the key ranges rather than arbitrarily. Each partition then owns a contiguous slice of the key space, the partition boundaries are meaningful, and the scan can declare an output ordering — which lets the optimizer delete a downstream sort. DataFusion does this with a greedy best-fit assignment on the files’ min/max statistics.

Consider a table stored as 1,000 Parquet files, each holding a contiguous range of event_date. A query does SELECT ... WHERE event_date BETWEEN '2024-01-01' AND '2024-06-30'.

The obvious plan: list the files, assign them to 16 partitions somehow, scan them, and sort the result. But the files are already sorted by event_date, and if the partition assignment is done by key range rather than arbitrarily, the scan’s output is already sorted — and the sort disappears.

That is the motivation for split_file_groups_by_statistics, a config option that defaults to false because it only applies when the table declares an output ordering.

The entry point

In datafusion/catalog-listing/src/table.rs:

 rust
 
 
let split_file_groups_by_statistics = declared_output_partitioning.is_none()
 && state
 .config_options()
 .execution
 .split_file_groups_by_statistics;
match split_file_groups_by_statistics
 .then(|| {
 output_ordering.first().map(|output_ordering| {
 FileScanConfig::split_groups_by_statistics_with_target_partitions(
 &self.table_schema,
 &partitioned_file_lists,
 output_ordering,
 file_group_count,
 )
 })
 })
 .flatten()
{
 Some(Err(e)) => log::debug!("failed to split file groups by statistics: {e}"),
 Some(Ok(new_groups)) => {
 if new_groups.len() <= file_group_count {
 partitioned_file_lists = new_groups;
 } else {
 log::debug!(
 "attempted to split file groups by statistics, but there were more file groups than target_partitions; falling back to unordered"
 )
 }
 }
 None => {} // no ordering required
}

Note the conditions:

declared_output_partitioning.is_none() — if the table already declares a partitioning (say, hash-partitioned on a join key), the file assignment is not free to choose.
output_ordering.first() — there must be an ordering to split by. The first one is used, because the primary sort key is the one the ranges are cut along.
The result must not exceed the target partition count, or the whole thing is abandoned and the table falls back to unordered groups. The failure is a log::debug!, not an error — the scan is still correct, it just does not get the ordering benefit.

The algorithm

FileScanConfig::split_groups_by_statistics_with_target_partitions (datafusion/datasource/src/file_scan_config/mod.rs):

 rust
 
 
let statistics = MinMaxStatistics::new_from_files(
 sort_order,
 table_schema,
 None,
 files.iter().copied(),
)?;

let indices_sorted_by_min = statistics.min_values_sorted();

// Initialize with target_partitions empty groups
let mut file_groups_indices: Vec<Vec<usize>> = vec![vec![]; target_partitions];

for (idx, min) in indices_sorted_by_min {
 if let Some((_, group)) = file_groups_indices
 .iter_mut()
 .enumerate()
 .filter(|(_, group)| {
 group.is_empty()
 || min
 > statistics
 .max(*group.last().expect("groups should not be empty"))
 })
 .min_by_key(|(_, group)| group.len())
 {
 group.push(idx);
 } else {
 // Create a new group if no existing group fits
 file_groups_indices.push(vec![idx]);
 }
}

Three steps:

1. Sort the files by their minimum value. min_values_sorted() returns the files ordered by the min of the sort key. This is what makes the groups contiguous: a group is built from files that are adjacent in this order.

2. For each file, find a group it can join. The filter is:

 rust
 
 
group.is_empty()
 || min > statistics.max(*group.last().expect("groups should not be empty"))

The group’s last element is the file with the largest min, because the files were processed in increasing min order. And min > max_of_group means the file’s entire range lies strictly above the group’s range. So the group stays a contiguous, non-overlapping range of the key space.

3. Among the groups that fit, pick the smallest.

 rust
 
.min_by_key(|(_, group)| group.len())

That is best-fit: put the file in the group that currently has the fewest files. It is the classic bin-packing heuristic, and it keeps the file counts balanced.

4. If no group fits, create a new one. The else branch. The file’s range overlaps every existing group, so it cannot be added to any of them without breaking the contiguity. A new group is created — and this is where the group count can exceed target_partitions, which is exactly why the caller checks the count and bails out.

Why contiguity matters

The whole point is that the output of the scan is ordered. If partition 0 owns the files for 2024-01 and partition 1 owns 2024-02, and the partitions are merged with a SortPreservingMergeExec, the merged output is sorted by event_date — and the merge does not need to sort, it just needs to compare the partition heads.

That is the EquivalenceProperties machinery from the ordering article: the scan can declare an output ordering, and the optimizer can use that declaration to remove a sort, to skip a repartition, or to satisfy a join’s ordering requirement.

And the min/max ranges of the groups are meaningful: partition 0 covers [2024-01-01, 2024-01-31]. That is a fact about the data that the planner can use.

Empty files

One preliminary step:

 rust
 
 
let (files, empty_files) = partition_known_empty_files(flattened_files);
if files.is_empty() {
 return Ok(file_groups.to_vec());
}

Files that are known to be empty — their statistics say zero rows — are pulled out and handled separately. They would otherwise waste a slot in the best-fit assignment, and a group containing only empty files is a partition that does no work.

The catch

It only applies to an ordered table. output_ordering.first() must exist. An unordered table — the common case — gets nothing.
Overlapping file ranges break it. If two files cover overlapping key ranges (which happens when a table is not strictly sorted, or when files were written out of order), no group can accept the second file and the group count grows. Past target_partitions, the whole thing is abandoned.
It is off by default. The config option split_file_groups_by_statistics defaults to false, which suggests the DataFusion developers found the benefit situational.
The partition count can shrink. The retain(|group| !group.is_empty()) at the end drops empty groups, so you can end up with fewer partitions than target_partitions — and hence less parallelism than you asked for. That is a deliberate trade: correct ordering over maximum parallelism.
The sort order must be compatible with the file layout. The algorithm assumes the files’ min values are meaningfully ordered with respect to the key — if the files’ contents are not actually sorted by the key, the resulting groups are not sorted ranges and the declared output ordering is a lie.

Why it matters

This is a small feature with a disproportionate effect when it applies. The difference between “the scan’s output is sorted” and “the scan’s output is not” is a SortExec — which for a large table is the single most expensive operator in the plan.

And the implementation is a nice piece of engineering: a greedy best-fit over min/max statistics that respects contiguity, handles the empty-file case, and bails out cleanly when the data does not fit the model. The bail-out is the important part — an optimisation that cannot apply must degrade to the old behaviour, not produce a wrong plan.

DataFusion's Aggregate Split: One UDF, Two Accumulator Traits, and a Serializable State Format

Jay Hubbert — Wed, 21 Oct 2026 07:33:00 +0000

TL;DR: An aggregate in DataFusion is not a function, it is a state machine: update_batch folds rows in, state() serializes the machine into Arrow arrays, merge_batch folds another machine’s state back in, and evaluate() produces the answer. That split is what makes partial/final two-phase aggregation possible. GroupsAccumulator is the vectorized sibling: one object holds the state of all groups, so there is no per-group virtual call and no per-group take. GroupsAccumulatorAdapter is the compatibility shim that lets a plain Accumulator pretend to be a GroupsAccumulator — at the price of copying rows into per-group contiguous buffers on every batch.

The problem: an aggregate is two different computations

Write SELECT avg(x) GROUP BY k over 100 million rows in 8 partitions. Two completely different jobs hide behind that one line of SQL:

Folding: for each row, update the state of the group the row belongs to.
Collapsing: for each group, turn its accumulated state into a single value.

The naive implementation does (1) with a function call per row, holding a HashMap<k, State> in memory. That is fine on a toy dataset and catastrophic on a real one, because the per-row call crosses a dynamic-dispatch boundary, the HashMap thrashes, and the state cannot be shipped anywhere.

DataFusion’s answer is to make the aggregate a value that can be serialized, shipped over a network, and merged — and then to compute it in a way that never calls a function per row.

The `Accumulator` trait: one group, one object

The whole contract is 200 lines in datafusion/expr-common/src/accumulator.rs:

 rust
 
 
// datafusion/expr-common/src/accumulator.rs:123
pub trait Accumulator: Send + Sync + Debug + std::any::Any {
 fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()>; // :140
 fn evaluate(&mut self) -> Result<ScalarValue>; // :193
 fn size(&self) -> usize; // :205
 fn state(&mut self) -> Result<Vec<ScalarValue>>; // :374
 fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()>; // :387
 fn retract_batch(&mut self, _values: &[ArrayRef]) -> Result<()> { // :421
 internal_err!("Retract should be implemented ...")
 }
}

Read it as a little algebra. update_batch is the operation, state() is the encoding, merge_batch is the homomorphism over that encoding, and evaluate is the projection of the state into a final value. For SUM the operation and the merge are the same thing (state is the running sum). For COUNT they are not: the state is a count, and merging counts means adding them, not re-counting the underlying rows. For AVG the state is two values — sum and count — and the merge adds both, then evaluate divides. The trait documentation spells this out at accumulator.rs:260-266:

The state can be and often is a different type than the output type of the Accumulator and needs different merge operations (for example, the partial state for COUNT needs to be summed together)

That sentence is the whole reason the trait exists. If SUM(x) were just a function f(accumulator, row) -> (), there would be no way to run it in two phases. Because state() is a separate, serializable representation, the planner can insert a Partial aggregate before a shuffle and a Final aggregate after it, and the two halves only need to agree on the state format.

The two-phase flow

text

 ┌──────────────────────────┐
 │ Partial (per partition) │
 │ update_batch() per row │
 │ state() -> Arrow arrays│
 └────────────┬─────────────┘
 │ (shuffle / repartition)
 ┌────────────▼─────────────┐
 │ Final │
 │ merge_batch(states) │ <- fold partition states together
 │ evaluate() -> value │
 └──────────────────────────┘

The state is plain Arrow data, so it crosses a RecordBatch boundary for free. AVG emits two state columns (sum, count); MEDIAN cannot know its state arity at planning time, so it packs everything into one ScalarValue::List (accumulator.rs:270-273).

The subtlety is that the final phase is not “run the function again on the state”. It is “fold state into state”. merge_batch receives the state()-encoded arrays of other accumulators, not raw rows. If you write an aggregate whose merge_batch is not the algebraic dual of update_batch, you get answers that depend on how many partitions the data had — the classic “works on my laptop” aggregate bug.

`GroupsAccumulator`: the fast path that avoids per-group calls

Now the performance trick. Accumulator handles one group. To aggregate 10 million groups you would need 10 million Box<dyn Accumulator> objects, and every row would cost you one virtual call plus a hash-map-ish lookup.

GroupsAccumulator (datafusion/expr-common/src/groups_accumulator.rs:193) flips this: one object holds the state of all groups, and the caller hands it a vector of group indices:

 rust
 
 
// datafusion/expr-common/src/groups_accumulator.rs:221
fn update_batch(
 &mut self,
 values: &[ArrayRef],
 group_indices: &[usize],
 opt_filter: Option<&BooleanArray>,
 total_num_groups: usize,
) -> Result<()>;

There is no per-group call at all. group_indices[i] says “row i belongs to group group_indices[i]”, and the implementation is expected to fan that out with a tight loop over primitive values. The canonical example is PrimitiveGroupsAccumulator (datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:43):

 rust
 
 
// datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:102
self.values.resize(total_num_groups, self.starting_value);
self.null_state.accumulate(
 group_indices,
 values,
 opt_filter,
 total_num_groups,
 |group_index, new_value| {
 // SAFETY: group_index is guaranteed to be in bounds
 let value = unsafe { self.values.get_unchecked_mut(group_index) };
 (self.prim_fn)(value, new_value);
 },
);

Three things make this fast: the state is a flat Vec<T::Native> indexed by group (cache friendly, no hashing, no allocation per group); prim_fn is a monomorphized closure, so SUM on Int64 compiles to a bare += with no vtable lookup; and the null/filter handling is hoisted out of the inner loop by accumulate (groups_accumulator/accumulate.rs:421), which has four specialized paths (no nulls/no filter, nulls/no filter, no nulls/filter, both) and even processes the null bitmap 64 bits at a time.

NullState (accumulate.rs:134) solves the awkward part: a group that has never seen a value must still produce a row (NULL for SUM, 0 for COUNT). Rather than every aggregate reinventing “did this group see anything”, the shared NullState tracks it and produces the null buffer at evaluate time.

SUM opts in with two lines (datafusion/functions-aggregate/src/sum.rs:301):

 rust
 
 
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
 !args.is_distinct
}

fn create_groups_accumulator(&self, args: AccumulatorArgs) -> Result<Box<dyn GroupsAccumulator>> {
 // ... PrimitiveGroupsAccumulator::<$t, _>::new(&$dt, |x, y| *x = x.add_wrapping(y))
}

The adapter: making old aggregates look new

Not every aggregate has a GroupsAccumulator, and writing one is genuinely hard (the state must be a columnar layout, EmitTo must be honoured, memory must be tracked). So DataFusion ships a bridge: GroupsAccumulatorAdapter (datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:95).

It holds a Vec<AccumulatorState>, one Box<dyn Accumulator> per group, and bridges the vectorized API to the per-group one by reordering the rows:

text

 input rows after take_arrays per-group call
 ┌─────┐ ┌─────────┐
 │ 2 │ │ 200 │ ──┐
 │ 0 │ take ────▶ │ 300 │ ─┤ group 0
 │ 1 │ │ 200 │ ─┘
 │ 0 │ │ 100 │ ──┐
 └─────┘ │ NULL │ ─┘ group 1
 └─────────┘

The implementation (groups_accumulator.rs:281-470) builds a permutation array of u32 row indices, calls Arrow’s compute::take_arrays to produce per-group-contiguous values, and then calls the accumulator once per group through Box<dyn Accumulator>. So the adapter pays:

one take (a full copy of the batch’s values) per update batch, and
one virtual call per group per batch, plus one Accumulator allocation per group.

That is still vastly better than a virtual call per row, but it is dramatically worse than the GroupsAccumulator fast path. The selection between the two happens once, at stream construction (datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:64):

 rust
 
 
pub(in crate::aggregates) fn create_group_accumulator(
 agg_expr: &Arc<AggregateFunctionExpr>,
 metrics: Arc<dyn AggregateMetrics>,
) -> Result<Box<dyn GroupsAccumulator>> {
 if agg_expr.groups_accumulator_supported() {
 agg_expr.create_groups_accumulator_with_metrics(metrics)
 } else {
 debug!("Creating GroupsAccumulatorAdapter for {}: {agg_expr:?}", agg_expr.name());
 // ... GroupsAccumulatorAdapter::new(move || agg_expr.create_accumulator())
 }
}

The debug! line is a hint to anyone profiling: if you see it, you are on the slow path.

`EmitTo`, `take_state`, and the “emit the first N groups” dance

Aggregates cannot always emit everything at once — the hash table drains in batches to respect batch_size. EmitTo (groups_accumulator.rs:31) encodes this:

 rust
 
 
pub enum EmitTo {
 All,
 /// Emit only the first `n` groups and shift all existing group indexes down by `n`.
 First(usize),
}

EmitTo::take_needed (groups_accumulator.rs:52) implements the shift with std::mem::take for All (no copy) and split_vec_min_alloc for First(n). This is why GroupsAccumulator::state takes an emit_to while Accumulator::state does not: the grouped version has to renumber the groups it did not emit, and that renumbering has to be reflected in the hash table’s own group vector. A subtle, easy-to-get-wrong contract, and one more reason the grouped API is opt-in.

first_last has its own take_state helper (datafusion/functions-aggregate/src/first_last.rs:559) that splits three parallel buffers (values, orderings, “is set” bitmap) in lockstep — the same “emit N, keep the rest” problem, but with three columns instead of one.

`convert_to_state`: skipping the hash table entirely

There is one more trick worth stealing. Normally a partial aggregate builds a hash table: rows come in, get hashed, and land in a group slot. But sometimes you already know the grouping key will not reduce anything — e.g. a partial aggregate before a GROUP BY that is only there to pre-compute a state for a downstream aggregate.

GroupsAccumulator::convert_to_state (groups_accumulator.rs:370) converts rows directly into state rows, one state row per input row, with no hash table at all:

 rust
 
 
// datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:669
fn convert_to_state(&self, values: &[ArrayRef], opt_filter: Option<&BooleanArray>)
 -> Result<Vec<ArrayRef>>
{
 // ... for each row: create a fresh accumulator, feed it that one row,
 // then harvest accumulator.state()
}

The PartialSkip mode uses it (datafusion/physical-plan/src/aggregates/aggregate_hash_table/partial_table.rs:128, datafusion/physical-plan/src/aggregates/hash_stream.rs:516). The cost is obvious in hindsight: one accumulator allocation and one state serialization per row. The benefit: no hash table, no group vector, no group-index bookkeeping, and the output is already in the shape the next phase wants.

Trade-offs and failure modes

Two implementations to keep in sync. Every aggregate that wants the fast path must implement both Accumulator and GroupsAccumulator, and the two must agree. A GroupsAccumulator that is subtly wrong produces different answers than the Accumulator path — and the optimizer may pick either one depending on whether DISTINCT is present (see sum.rs:301).
The adapter’s copy is real. The take per batch is O(rows) memory traffic; for wide batches of expensive-to-copy types (strings, lists) this is the dominant cost of the “slow” path, not the virtual call.
evaluate must not consume state. The trait docs are emphatic (accumulator.rs:179-190): evaluate is called repeatedly for window functions with different frames, so it may reorder internal state but must not consume it. Violating this gives correct answers for GROUP BY and garbage for OVER (...).
size() is a hot path. The adapter deliberately maintains allocation_bytes incrementally rather than calling size() per group per batch, with a comment saying exactly that (“We saw size() being a bottleneck in earlier implementations when there were many distinct groups”, groups_accumulator.rs:101-106). If you implement Accumulator, make size() cheap or return a constant.
State format is a public contract. state_fields defines the Arrow schema of the intermediate state, and it is versioned with the engine. If you change a state format you break distributed plans that were planned by an older binary. approx_percentile_cont packs its state into six scalars plus a list (datafusion/functions-aggregate/src/approx_percentile_cont.rs:250) precisely because the state must round-trip through Arrow.

The takeaway: the split into update/state/merge/evaluate is not ceremony, it is what turns “aggregate function” into “algebraic data type” — something you can compute in parallel, ship across a network, and merge without ever knowing the original rows.

DataFusion's Spillable Shuffle: Writing Batches to Disk Without Reordering Them

Jay Hubbert — Wed, 21 Oct 2026 07:25:00 +0000

TL;DR: RepartitionExec gives every output partition one shared memory reservation. When try_grow fails for a batch, the batch is written to that partition’s spill pool on disk and a zero-byte marker travels through the in-memory channel instead. The reader sees the marker and switches to reading the spill file, blocking the channel poll until it has the batch — that is how FIFO order survives a disk round trip. Contrast the aggregation spill, which drains the entire hash table: the shuffle spills selectively, one batch at a time.

The shuffle is the worst operator in the plan

A RepartitionExec is an exchange: N input tasks each produce batches for M output partitions. That makes it simultaneously the most parallel and the most dangerous operator in the plan.

The danger is a deadlock that has nothing to do with locks. Suppose you give each of the M output channels a bounded buffer of size K, and each of the N input tasks writes to all M channels in a fixed order. Input task 1 fills channel 0 and blocks; input task 2 fills channel 1 and blocks; the consumer is waiting on channel 0 but input task 0 is blocked behind input task 1 on channel 1. Classic cyclic wait. Every bounded-buffer exchange has this shape.

DataFusion’s answer is to make the channels unbounded and to put a global gate in front of them:

 rust
 
 
// datafusion/physical-plan/src/repartition/distributor_channels.rs:37-40
//! There are `N` virtual MPSC (multi-producer, single-consumer) channels with unbounded capacity. However, if all
//! buffers/channels are non-empty, than a global gate will be closed preventing new data from being written (the
//! sender futures will be [pending](Poll::Pending)) until at least one channel is empty (and not closed).

The gate is a single atomic counter of empty channels:

 rust
 
 
// datafusion/physical-plan/src/repartition/distributor_channels.rs:412-425
struct Gate {
 /// Number of currently empty (and still open) channels.
 empty_channels: AtomicUsize,
 /// Wakers for the sender side, including their channel IDs.
 ///
 /// This is `None` if the there are non-empty channels.
 send_wakers: Mutex<Option<Vec<(Waker, usize)>>>,
}

And the sender checks it before enqueuing:

 rust
 
 
// datafusion/physical-plan/src/repartition/distributor_channels.rs:218-225
// does ANY receiver need data?
// if so, allow sender to create another
if this.gate.empty_channels.load(Ordering::SeqCst) == 0 {
 let mut guard = this.gate.send_wakers.lock();
 if let Some(send_wakers) = &mut *guard {
 send_wakers.push((cx.waker().clone(), this.channel.id));
 return Poll::Pending;
 }
}

Read the gate’s guarantee precisely, because it is narrower than it looks: at least one channel is empty at any moment. That is a liveness guarantee, not a memory bound. A sender is allowed to push into an already-non-empty channel as long as some other channel is empty, so a single channel can still accumulate a long run of batches while a sibling sits empty.

What the gate buys is the absence of a cyclic wait. When the gate closes, every output channel has data, so the consumer is guaranteed to be able to drain one, which empties it and reopens the gate (distributor_channels.rs:305-320). No sender can be waiting on a resource that only a blocked peer can release, because no peer is blocked — the consumer is always running.

The memory bound comes from somewhere else: the output partition’s MemoryReservation. The gate is what keeps the exchange deadlock-free; the memory pool is what keeps it from eating the machine. Conflating the two is easy, and the code keeps them strictly separate — Gate knows nothing about bytes.

The lock ordering is documented and strict — “channel lock is always acquired prior to gate lock” (distributor_channels.rs:121-123) — because the reverse order would let a sender and a receiver deadlock on the two mutexes.

Where the memory actually goes

Each output partition gets one MemoryConsumer shared by all N input tasks:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:506-511
let reservation = Arc::new(
 MemoryConsumer::new(format!("{name}[{partition}]"))
 .with_can_spill(true)
 .register(context.memory_pool()),
);

The reservation is per output partition, not per (input, output) pair — because the memory that matters is the memory the consumer of that partition is holding. When the consumer reads a batch out of the channel, it shrinks the reservation (repartition/mod.rs:2559-2562).

The try_grow happens on the producer side, at send time:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:232-252
async fn send(&mut self, batch: RecordBatch) -> Result<(), SendError<MaybeBatch>> {
 let size = batch.get_array_memory_size();

 // Decide the payload outside of any await: never hold a MutexGuard
 // across an await point.
 let (payload, is_memory_batch) = {
 match self.reservation.try_grow(size) {
 Ok(_) => (Ok(RepartitionBatch::Memory(batch)), true),
 Err(_) => match self.spill_writer.push_batch(&batch) {
 Ok(()) => (Ok(RepartitionBatch::Spilled), false),
 Err(err) => (Err(err), false),
 },
 }
 };

 let result = self.sender.send(Some(payload)).await;
 if result.is_err() && is_memory_batch {
 self.reservation.shrink(size);
 }
 result
}

Four things to notice:

The fallback is inline. try_grow failing is not an error; it selects the spill path.
The batch is not cloned. RepartitionBatch::Spilled is a marker — it carries no data. The batch itself already went to disk.
The payload decision happens outside any await. The comment is explicit: “never hold a MutexGuard across an await point”. Computing the payload requires the reservation’s mutex; doing that before the send().await keeps the critical section short and avoids the classic SendFuture-holds-lock-across-await bug.
A failed send refunds the memory. If the receiver hung up (a LIMIT dropped the stream), the reservation is shrunk — otherwise a cancelled query would leak its reservation until the whole task was dropped.

The marker and the two-state reader

The reader side is a small state machine:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:2473-2482
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum StreamState {
 /// Reading from the memory channel (normal operation)
 ReadingMemory,
 /// Waiting for a spilled batch from the spill stream.
 /// Must not poll channel until spilled batch is received to preserve ordering.
 ReadingSpilled,
}

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:2546-2590
StreamState::ReadingMemory => {
 let value = match self.receiver.recv().poll_unpin(cx) {
 Poll::Ready(v) => v,
 Poll::Pending => return Poll::Pending,
 };
 match value {
 Some(Some(v)) => match v {
 Ok(RepartitionBatch::Memory(batch)) => {
 // Release memory and return batch
 self.reservation.shrink(batch.get_array_memory_size());
 return Poll::Ready(Some(Ok(batch)));
 }
 Ok(RepartitionBatch::Spilled) => {
 // Batch was spilled, transition to reading from spill stream
 // We must block on spill stream until we get the batch
 // to preserve ordering
 self.state = StreamState::ReadingSpilled;
 }
 Err(e) => return Poll::Ready(Some(Err(e))),
 },
 ...

and

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:2591-2612
StreamState::ReadingSpilled => {
 match self.spill_stream.poll_next_unpin(cx) {
 Poll::Ready(Some(Ok(batch))) => {
 self.state = StreamState::ReadingMemory;
 return Poll::Ready(Some(Ok(batch)));
 }
 ...
 Poll::Ready(None) => {
 // Spill stream ended — release its resources before
 // we go back to draining the memory channel.
 ...
 self.state = StreamState::ReadingMemory;
 }
 Poll::Pending => {
 // Spilled batch not ready yet, must wait
 // This preserves ordering by blocking until spill data arrives
 return Poll::Pending;
 }
 }
}

This is the crux. The channel is a FIFO of slots, and a slot may be either a real batch or a “there is a batch on disk” marker. When the reader hits a marker it must not go back to polling the channel — otherwise it would return the next batch while the spilled one is still in flight, reordering the output. So it blocks on the spill stream until it has the batch, then resumes the channel.

This is why the marker approach beats the alternative (writing the batch to disk and sending a promise that resolves later): the promise version needs a second queue and a join, and the join is where the ordering bugs live. The marker version reuses the channel’s own FIFO as the ordering guarantee.

The spill pool

The spill pool is a small, carefully designed concurrent structure: datafusion/physical-plan/src/spill/spill_pool.rs.

 rust
 
 
// datafusion/physical-plan/src/spill/spill_pool.rs:34-64
/// Shared state between the writer and readers of a spill pool.
/// This contains the queue of files and coordination state.
///
/// # Locking Design
///
/// This struct uses **fine-grained locking** with nested `Arc<Mutex<>>`:
/// - `SpillPoolShared` is wrapped in `Arc<Mutex<>>` (outer lock)
/// - Each `ActiveSpillFileShared` is wrapped in `Arc<Mutex<>>` (inner lock)
///
/// This enables:
/// 1. **Short critical sections**: The outer lock is held only for queue operations
/// 2. **I/O outside locks**: Disk I/O happens while holding only the file-specific lock
/// 3. **Concurrent operations**: Reader can access the queue while writer does I/O
///
/// **Lock ordering discipline**: Never hold both locks simultaneously to prevent deadlock.

The reader can start reading a file before the writer has sealed it. The writer appends a batch, flushes, and wakes the reader; the reader reads whatever has been flushed. Files rotate when they exceed max_spill_file_size_bytes (default 128 MiB, datafusion/common/src/config.rs:1037):

 rust
 
 
// datafusion/physical-plan/src/spill/spill_pool.rs:262-276
let max_file_size_reached = file_shared.estimated_size > self.max_file_size_bytes;

if max_file_size_reached {
 // Finish the IPC writer
 if let Some(mut writer) = file_shared.writer.take() {
 writer.finish()?;
 }
 file_shared.writer_finished = true;
 file_shared.wake();
} else {
 drop(file_shared);
 let mut shared = self.shared.lock();
 shared.open_write_files.push_back(write_file);
}

SPSC vs MPSC: the ordering contract dictates the topology

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:513-546
let (spill_writers, spill_readers) = if preserve_order {
 // preserve_order: one dedicated single-producer FIFO pool per input partition.
 // Each writer is moved into exactly one input task (never cloned), so the ordering
 // the downstream merge relies on is preserved across the spill boundary.
 ...
 for _ in 0..num_input_partitions {
 let (writer, reader) = spill_pool::spsc_channel(max_file_size, Arc::clone(&spill_manager));
 writers.push(Some(writer));
 readers.push(reader);
 }
 (PartitionSpillWriters::PerInput(writers), readers)
} else {
 // non-preserve-order: one shared multi-producer pool per output partition, since
 // all inputs share the same receiver and the output is an unordered multiset.
 let (writer, reader) = spill_pool::mpsc_channel(max_file_size, Arc::clone(&spill_manager));
 (PartitionSpillWriters::Shared(writer), vec![reader])
};

This is the “spill everything vs spill selectively” question, and for the shuffle the answer is dictated by a semantic requirement rather than by preference:

preserve_order: the output must be globally sorted, and the downstream StreamingMerge merges the per-input streams. Each input’s data must stay in order relative to itself. So each (input, output) pair gets its own single-producer pool — spsc_channel — and the writer is moved into the input task, never cloned. The type system enforces it:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:185-213
enum PartitionSpillWriters {
 /// `preserve_order`: one single-producer FIFO writer per input partition. Each is `take`n
 /// exactly once (moved into the matching input task), so the pool always has one writer.
 PerInput(Vec<Option<SpillPoolSink>>),
 /// Non-preserve-order: one shared writer, cloned into every input task.
 Shared(SpillPoolWriter),
}

fn take_for_input(&mut self, input: usize) -> Result<SpillPoolSink> {
 match self {
 PartitionSpillWriters::PerInput(writers) => {
 writers[input].take().ok_or_else(|| {
 internal_datafusion_err!("spill writer for input partition requested more than once")
 })
 }
 PartitionSpillWriters::Shared(writer) => Ok(writer.new_sink()),
 }
}

non-preserve-order: the output is an unordered multiset, so all N inputs can share one multi-producer pool per output partition. mpsc_channel returns a cloneable SpillPoolWriter; new_sink() hands out an extra sink and bumps remaining_writer_count, which is how the pool knows not to signal EOF while another producer is still alive (spill_pool.rs:105-135).

So the number of spill files is N × M in preserve-order mode and M in non-preserve mode. That is a real cost: 16 inputs × 16 outputs = 256 open files in the worst case.

The joins: what spills and what does not

This is where the “spill everything vs spill selectively” split becomes stark, because DataFusion’s joins are on a spectrum.

`HashJoinExec`: reserves, but does not spill

The hash join does register a consumer and reserve for its build side:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/exec.rs:1905-1917
(None, PartitionMode::Partitioned) => {
 let left_stream = self.left.execute(partition, Arc::clone(&context))?;
 let reservation =
 MemoryConsumer::new(format!("HashJoinInput[{partition}]"))
 .register(context.memory_pool());
 OnceFut::new(collect_left_input(...))
}

and the hash table allocation is gated:

 rust
 
 
// datafusion/physical-plan/src/joins/hash_join/exec.rs:3080-3099
let estimated_hashtable_size =
 estimate_memory_size::<(u32, u64)>(num_rows, fixed_size_u32)?
 // Each build row also owns an index in the duplicate-key chain.
 .checked_add(num_rows * size_of::<u32>())
 ...;
reservation.try_grow(estimated_hashtable_size)?;

But there is no spill fallback. When try_grow fails, the ResourcesExhausted error propagates and the query fails. And note that the consumer is not registered with can_spill(true) — because it genuinely cannot spill. A hash map cannot be partially spilled without either (a) serializing it and losing the O(1) probe property, or (b) partitioning it, which is a reshuffle.

The practical consequence: a hash join with a large build side is the hardest query to run under a memory limit in DataFusion. The workaround is to force a sort-merge join instead, which does spill.

`SortMergeJoinExec`: spills selectively, and only what it must

The sort-merge join streams the probe side and buffers the inner side. For semi/anti/mark joins with a filter, it needs to buffer the inner rows of the current key group so each can be evaluated against the outer group. That buffer is the only thing that can grow without bound, and it is spilled — but only when the pool refuses:

 rust
 
 
// datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs:686-700
// Reserve memory for the newly buffered slice. If the pool
// is exhausted, spill the entire buffer to disk.
if self.try_resize_reservation().is_err() {
 if self.runtime_env.disk_manager.tmp_files_enabled() {
 self.spill_inner_key_buffer(&mut writer)?;
 } else {
 // Re-attempt to get the error message
 self.try_resize_reservation().map_err(|e| {
 DataFusionError::Execution(format!("{e}. Disk spilling disabled."))
 })?;
 }
}

The spill is per key group, appended to a single open file for that group, and the file is finished when the group is complete (bitwise_stream.rs:447-466).

The “skip spilling for small sides” idea shows up in the design of this operator rather than in a branch: the no-filter path never buffers inner rows at all — it just advances the cursor — so it cannot OOM and never touches the disk. The doc comment spells out the asymmetry:

 rust
 
 
// datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs:104-112
//! Memory usage is bounded and independent of total input size:
//! - One outer batch at a time (not tracked by reservation — single batch,
//! cannot be spilled since it's needed for filter evaluation)
//! - One inner batch at a time (streaming)
//! - `matched` bitset: one bit per outer row, re-allocated per batch
//! - Inner key group buffer: only for filtered joins, one key group at a time.
//! Tracked via `MemoryReservation`; spilled to disk when the memory pool
//! limit is exceeded.

The degenerate case is documented too: a single hot key means one key group buffers an unbounded number of rows — in memory or on disk (bitwise_stream.rs:114-124).

`NestedLoopJoinExec`: spills the build side in chunks

The nested loop join broadcasts the left side, so it spills the left side when it does not fit:

 rust
 
 
// datafusion/physical-plan/src/joins/nested_loop_join.rs:740-741
let can_spill = context.runtime_env().disk_manager.tmp_files_enabled()
 && !coordinated_fallback_disabled;

and notably it pre-creates the spill manager up front, with a comment that explains why:

 rust
 
 
// datafusion/physical-plan/src/joins/nested_loop_join.rs:743-751
// Built here rather than on demand: by the time the load hits the memory limit the
// stream is partly consumed, and it has to be spillable at that point without going
// back to the left child.

That is a genuinely subtle piece of engineering: the spill decision is deferred (you only know you need to spill when you hit the limit), but the capability to spill must be established before the child stream is consumed, because you cannot re-read a child.

Trade-offs and failure modes

The gate is a liveness mechanism, not a memory bound. It guarantees at least one output channel is empty, which prevents the cyclic wait; it does not limit how many batches a single channel can hold. The memory bound comes from the output partition’s MemoryReservation — and the channel buffer is not accounted to it until the batch is sent. With 1024 output partitions and 1 MiB batches, a single hot partition can accumulate a lot of channel buffer that the memory pool never sees.
Spilling a shuffle is not free and not always a win. You pay an IPC encode, a file write, a file read, and a decode — for a batch that might have been delivered in microseconds. The try_grow fallback triggers on every batch once the pool is full, so a saturated shuffle writes everything to disk. This is the “spill everything” pathology, and unlike the aggregation, the shuffle has no way to avoid it: it cannot batch up and spill less often, because the consumer wants that batch.
The marker scheme requires a strictly FIFO channel. The whole ordering argument collapses if the channel reorders. preserve_order mode pays for this with an SPSC channel per input; non-preserve-order mode gets away with one MPSC pool per output.
File descriptor pressure. N × M spill files in preserve-order mode. With 32 inputs and 32 outputs that is 1024 files, each with its own InProgressSpillFile, IPC writer and buffered encoder.
Disk quota is the real limit. max_temp_directory_size is enforced by the DiskManager, and exceeding it is an I/O error, not a memory error. A shuffle that spills more than the quota dies in a way that looks unrelated to the problem.
Hash joins do not spill. If your plan has a hash join with a big build side and a memory limit, the query fails with ResourcesExhausted and no amount of spilling elsewhere will save it. Force a sort-merge join.
Cancelled queries leak file handles until the abort tasks run. The AsyncIPCStreamWriter drop handler spawns a cleanup task bounded by a semaphore of 8 concurrent aborts (spill/mod.rs:521-527); beyond that, the writer is simply dropped and the backend’s own lifecycle policy takes over.

The through-line: the shuffle spills selectively and reactively, because its unit of work is a self-contained batch that can be written and read back independently. The aggregation spills wholesale, because its unit of work is a hash table that can only be reconstructed by sorting. The join sits in between: it spills the one structure that can grow without bound, and leaves everything else streaming. Each of those choices follows from what the data structure actually is, and that is the interesting part of the design.

How DataFusion Spills a Hash Aggregation to Disk Without Breaking GROUP BY

Jay Hubbert — Wed, 21 Oct 2026 07:17:00 +0000

TL;DR: DataFusion does not “spill the hash table”. It drains the hash table into intermediate aggregate states, sorts those states by the full group key, writes them as Arrow IPC spill files, then does a bounded-memory multi-level merge of the sorted runs and replays the merged stream through a clustered final aggregate. The sort is the whole trick: it makes equal groups adjacent in the merged output so the final aggregate can merge them with O(1) state per group instead of O(distinct groups).

Why “spill the hash table” is not a thing you can do

A hash aggregation’s state is a hash map from group key → accumulator state. The obvious way to spill it is: serialize the map, write it to disk, keep hashing new input into a fresh map. That works until you need the answer, because now you have K on-disk hash maps and you must merge them — and merging hash maps is exactly the aggregation you were trying to avoid, now with random I/O instead of RAM access.

The other obvious way is the classic “hash-partitioned two-phase aggregation”: partition rows by hash(group_key) % K, aggregate each partition independently, then re-aggregate the K partial results. This is what Spark and Flink do. It has a nasty property in DataFusion’s execution model: it is a blocking reshuffle in the middle of a pipelined plan, and the partial results of a partition may still be enormous (all rows with the same hash land in one partition). You have traded a memory problem for a skew problem.

DataFusion’s actual answer is neither. It keeps a single hash table, and when the table can no longer grow, it converts the table’s contents into sorted runs on disk. Sorting is the key move, for two reasons:

Mergeability. Sorted runs can be merged with a k-way merge that holds one batch per run in memory. The memory cost of merging is O(k * batch_size), not O(total data). The number of runs k can be bounded by the merge fan-in, and if k is too large, the merge itself spills intermediate runs and repeats (a multi-level merge).
Semantics. Once the merged output is sorted by the group key, all rows belonging to the same group are contiguous. A streaming aggregate that only needs to compare the current group key with the previous one can merge them in order, keeping exactly one group’s state live at a time.

So spilling a GROUP BY is really “re-implement the GROUP BY as a sort followed by a streaming aggregate”. That is a lot of machinery, and DataFusion has it.

The pieces

`AggregateSpill`: the per-partition spill context

datafusion/physical-plan/src/aggregates/spill.rs is 301 lines and holds the entire strategy. The struct doc comment at spill.rs:49-136 contains a worked example:

 text
 
 
SELECT k, SUM(v) FROM t GROUP BY k

Step 1: OOM round 1
Buffered batch Spill file 1 (sorted)
k partial_sum k partial_sum
1 3 1 3
3 4 -> 2 5
2 5 3 4

Step 3: Global sort and final aggregation
SPM output Final aggregate output
k partial_sum k SUM(v)
1 3 1 5
1 2 -> 2 5
2 5 3 10
3 4
3 6

Three fields do the real work:

 rust
 
 
// datafusion/physical-plan/src/aggregates/spill.rs:50-140
/// Spilled rows already contain evaluated group keys and intermediate
/// aggregate states. Replay must therefore use final aggregation semantics
/// and column-based group expressions rather than evaluating the raw input
/// expressions a second time, so single-stage aggregates are rewritten to
/// their final counterpart here.
replay_agg: AggregateExec,
...
/// Full group-key ordering kept by every spill file and the merged input.
spill_expr: LexOrdering,
...
/// Spill runs waiting to be merged, all sorted by `spill_expr`.
spills: Vec<SortedSpillFile>,

That first comment is the “reproduce the grouping semantics” problem in one paragraph. The rows on disk are not input rows. They are (evaluated group key columns, partial accumulator states). Replaying them through the original AggregateExec would re-evaluate the group expressions against the wrong input and double-count. So AggregateSpill::try_new rewrites the plan:

 rust
 
 
// datafusion/physical-plan/src/aggregates/spill.rs:148-168
let mut replay_agg = agg.clone();
replay_agg.group_clustering_mode = GroupClusteringMode::Full;
let group_schema = match agg.mode {
 AggregateMode::Final | AggregateMode::FinalPartitioned => {
 agg.group_by().group_schema(spill_schema)?
 }
 AggregateMode::Single | AggregateMode::SinglePartitioned => {
 replay_agg.mode = if agg.mode == AggregateMode::Single {
 AggregateMode::Final
 } else {
 AggregateMode::FinalPartitioned
 };
 *replay_agg.group_by_mut() = Arc::new(agg.group_by().as_final());
 agg.group_by().group_schema(&agg.input().schema())?
 }
 mode => {
 return internal_err!("{label}: cannot replay aggregate mode {mode:?}");
 }
};

Single becomes Final; group_by() becomes group_by().as_final(); the group expressions become plain Column references into the spilled schema. Only Final/FinalPartitioned can be replayed — a Partial aggregate has nowhere to merge to.

`sort_and_spill`: drain, sort, write

 rust
 
 
// datafusion/physical-plan/src/aggregates/spill.rs:218-251
pub(super) fn sort_and_spill(&mut self, state_batch: Option<RecordBatch>) -> Result<()> {
 let Some(state_batch) = state_batch else { return Ok(()); };

 let max_batch_rows = state_batch.num_rows().min(self.batch_size);
 let sorted_iter = IncrementalSortIterator::new(
 state_batch,
 self.spill_expr.clone(),
 self.batch_size,
 );
 let spill_file = self.spill_manager
 .spill_record_batch_iter_and_return_max_batch_memory(sorted_iter, self.label)?;
 ...
 self.spills.push(SortedSpillFile { file, max_record_batch_memory });
 self.min_spill_batch_rows = self.min_spill_batch_rows.min(max_batch_rows);
 Ok(())
}

Note IncrementalSortIterator rather than a full sort: it sorts the batch and then yields it back in batch_size chunks (sorts/stream.rs:353-375). This keeps the output batch size bounded so the spill file does not contain one enormous batch, which matters because the merge sizes its buffers from the largest batch it sees.

The state_batch comes from take_state_batch on the aggregate table:

 rust
 
 
// datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:369-411
/// Takes every intermediate aggregate state and resets the table so it can
/// continue accumulating raw input.
///
/// Unlike normal single aggregation output, this materializes intermediate
/// states rather than final values. The states can therefore be merged after
/// spilling without finalizing the same group more than once.
pub(in crate::aggregates) fn take_state_batch(&mut self) -> Result<Option<RecordBatch>> {
 ...
 state.group_values.clear_shrink(0);
 state.batch_group_indices.clear();
 state.batch_group_indices.shrink_to_fit();
 Ok(Some(batch))
}

The clear_shrink(0) is load-bearing: it shrinks the hash table’s own capacity to zero so the memory can actually be returned to the pool, rather than sitting in a Vec that has grown but no longer needs to. Skipping that step is the classic “we spilled but the RSS did not go down” bug.

`into_replay_stream`: merge then re-aggregate

 rust
 
 
// datafusion/physical-plan/src/aggregates/spill.rs:254-300
pub(super) fn into_replay_stream(
 self,
 baseline_metrics: &BaselineMetrics,
 metrics: ClusteredAggregateTableMetrics,
 reservation: MemoryReservation,
) -> Result<SendableRecordBatchStream> {
 ...
 // The merge and replay table are two components of the same aggregate
 // operator. Keep them under one consumer registration so a fair memory
 // pool does not divide this operator's quota between its own phases.
 let merge_reservation = reservation.new_empty();
 let merged = StreamingMergeBuilder::new()
 .with_schema(spill_schema)
 .with_spill_manager(spill_manager)
 .with_sorted_spill_files(spills)
 .with_expressions(&spill_expr)
 .with_metrics(baseline_metrics.intermediate())
 .with_batch_size(batch_size)
 .with_reservation(merge_reservation)
 .with_replay_headroom()
 .with_intermediate_merge_sizing(Some(min_spill_batch_rows))
 .build()?;
 let replay = ClusteredFinalAggregateStream::new_with_input_and_metrics(
 &replay_agg, &context, partition, merged,
 &GroupClusteringMode::Full,
 baseline_metrics.clone(), metrics, None, reservation,
 )?;
 Ok(replay.into_stream())
}

Three subtleties in twenty lines:

reservation.new_empty() creates a sibling reservation sharing the same MemoryConsumer. The merge buffers and the replay table are two phases of one operator; if they had separate consumers, FairSpillPool would split the operator’s own quota between them and each would get half of what it needs.
with_replay_headroom() (sorts/streaming_merge.rs:167-171) makes the merge reserve twice what it needs — the merge buffers plus an equal amount of headroom for the aggregate that is about to consume the merged stream. Without this, the merge happily eats the entire budget and then the replay cannot allocate a single group.
with_intermediate_merge_sizing(Some(min_spill_batch_rows)) passes the minimum across runs of each run’s largest batch row count (sorts/streaming_merge.rs:174-185). A run without a full batch disables sizing for the whole replay, since merging short batches can increase the intermediate run’s budget.

The multi-level merge

datafusion/physical-plan/src/sorts/multi_level_merge.rs is the machinery that makes “merge arbitrarily many sorted runs in bounded memory” possible.

 text
 
 
Initial State: Multiple sorted streams + spill files
 ┌───────────┐
 │ Phase 1 │
 └───────────┘
 (merge as many as fit in memory)
 │
 ▼
 ┌───────────┐
 │ Phase 2 │ ← the intermediate merged run is itself spilled
 └───────────┘

The algorithm, from the module doc (multi_level_merge.rs:51-59):

Take as many sorted runs as the memory budget allows.
Merge them into one sorted stream.
If there are more runs left, write that stream to a spill file, add it back to the pile, and repeat.

The fan-in is bounded by max_spill_merge_fan_in on the DiskManager (multi_level_merge.rs:578-582), so a query that spilled 500 times does not try to open 500 files at once.

The memory strategy is the subtle part:

Worst-case reservation: buffers are sized from the largest batch in each run, not the average (multi_level_merge.rs:127-133).
Adaptive shrinking: if the budget can’t fit the fan-in, the per-stream read buffer shrinks (get_sorted_spill_files_to_merge, multi_level_merge.rs:568-605).
Re-spilling skewed runs: if even a 2-stream merge does not fit because one run has a pathologically wide batch, the larger run is re-spilled with each batch sliced in half, which lowers its largest-batch metric and hence its per-stream reservation. The merge is retried. If a single row is wider than the entire budget, the merge surfaces ResourcesExhausted rather than looping forever (multi_level_merge.rs:139-152).

That last one is the sort of thing you only discover in production.

The spill file format

SpillManager (datafusion/physical-plan/src/spill/spill_manager.rs) is a thin, deliberately dumb layer: it writes RecordBatches to a file and reads them back. It does not interpret the contents — “all records within the same spill file are ordered according to a specific order” is the caller’s contract (spill_manager.rs:38-40).

The format is Arrow IPC stream format, which supports dictionary replacement (spill/mod.rs:469-476). Two details are worth knowing:

View-array GC. Before writing, StringView/BinaryView arrays are garbage-collected, because a sliced view array keeps the entire backing buffer alive. The comment gives the numbers:

 rust
 
 
// datafusion/physical-plan/src/spill/mod.rs:752-762
/// For example, in the ClickBench benchmark (issue #19414), repeated slicing of StringView
/// arrays resulted in 820MB of spill files that could be reduced to just 33MB after GC -
/// a 96% reduction in size.

96% is not a rounding error — without GC, a spilled query can write an order of magnitude more data than it holds.

Exact-size framing. The reader reassembles IPC messages so that each decoded batch pins only its own message, not the 128 KB read chunk it arrived in (spill/mod.rs:96-156). The alternative is subtle and nasty: a 5 KB batch that lands inside a 128 KB chunk keeps all 128 KB alive, and the batch’s accounted size (5 KB) is a lie by a factor of 25. The multi-level merge sizes its buffers from these numbers, so a lie here becomes an OOM.

The DiskManager side

DiskManager (datafusion/execution/src/disk_manager.rs) enforces a disk quota (max_temp_directory_size) and hands out temp files from a randomized choice of local directories (create_tmp_file, disk_manager.rs:342-386). Files are reference counted so the parent directory outlives them, and the disk usage is returned when the last reference drops (RefCountedTempFile::drop, disk_manager.rs:440-453).

Crucially, tmp_files_enabled() (disk_manager.rs:334-336) is the switch that turns “can spill” into a real capability. Aggregates check it before registering as spillable:

 rust
 
 
// datafusion/physical-plan/src/aggregates/hash_stream.rs:578-596
let can_spill = context.runtime_env().disk_manager.tmp_files_enabled();
let spill_context = if can_spill { ... } else { None };
let reservation = MemoryConsumer::new(...)
 .with_can_spill(can_spill)
 .register(context.memory_pool());

So a deployment with temp files disabled produces a pool with no spillable consumers, and FairSpillPool gives the entire budget to whoever asks first.

The trigger

The modern hash aggregate (aggregates/hash_stream.rs:684-760) checks the reservation after each input batch:

 rust
 
 
// datafusion/physical-plan/src/aggregates/hash_stream.rs:705-750
let resize_result = self.reservation.try_resize(
 Self::reservation_size_for_table(hash_table, spill_context.as_deref()));

match resize_result {
 Ok(()) => {}
 Err(e @ DataFusionError::ResourcesExhausted(_)) => {
 let spill_context = spill_context.as_mut().ok_or_else(|| e.context(
 "Final hash aggregate cannot spill because temporary files are not enabled in the DiskManager",
 ))?;
 ...
 let result = hash_table
 .take_state_batch()
 .and_then(|batch| spill_context.sort_and_spill(batch));

 // Spilling shrinks the aggregate table and releases its accumulated
 // memory. Update the reservation accordingly.
 self.reservation.try_resize(hash_table.memory_size())?;
 result?;
 // One sorted run was written; resume reading the original input.
 }
 Err(e) => return Err(e),
}

Three things to notice:

The reservation size includes extra headroom when a spill context exists: table_size + group_count * size_of::<u32>() (hash_stream.rs:661-679) — the index materialization needed to sort the state.
The spill happens after the failed try_resize, and the reservation is then shrunk to the (now much smaller) table size. The failure is not fatal; it is the trigger.
The legacy GroupedHashAggregateStream does the same thing but computes the sort memory budget explicitly, including the “2× worst case” rule:

 rust
 
 
// datafusion/physical-plan/src/aggregates/grouped_hash_stream.rs:1249-1267
// The maximum worst case for a sort is 2X the original underlying buffers(regardless of slicing)
let sort_memory = (batch_memory
 + (emit.get_sliced_size()? as f32 * batch_size_ratio) as usize)
 .min(batch_memory * 2);
self.reservation.try_grow(sort_memory).map_err(|err| {
 resources_datafusion_err!("Failed to reserve memory for sort during spill: {err}")
})?;

And it frees the table before reserving the sort memory — “This must happen before reserving sort memory so the pool has room” (grouped_hash_stream.rs:1242-1248). Order matters: if you reserved first, you would need the memory twice.

Trade-offs and failure modes

Spilling is expensive and it is paid in latency, not just I/O. Each spill event sorts the entire accumulated state: O(n log n) per run, n runs. A query that spills 100 times pays 100 sorts.
The number of runs drives the merge cost. 100 runs means a multi-level merge: merge 16, write the result, merge 16 more… Each level rewrites the whole dataset. This is why max_spill_merge_fan_in exists and why the merge tries hard to fit as many runs as possible in one pass.
Partial aggregates cannot be spilled. The replay needs a “final” target to merge into. A Partial-only plan that OOMs has to error out; the mode rewrite in AggregateSpill::try_new is what makes the whole scheme possible, and it is the part most likely to surprise you when reading the code.
GroupClusteringMode::Full aggregates never spill (spill.rs:169-172 returns an internal error). If groups are already contiguous in the input, spilling would not help — you would read the same data back in the same order and hit the same problem.
Skewed group keys. One hot key means one group’s state never leaves memory. The sort-based spill does not fix that; the group’s state simply stays in the hash table across the whole run.
Disk is not infinite. max_temp_directory_size is enforced, and a query that spills more than the quota gets an I/O error, not an OOM. The failure mode moves, it does not disappear.
The 2× sort-memory rule is a heuristic. It is derived from get_sliced_size and can be wrong for adversarial layouts; the .min(batch_memory * 2) cap is the safety net, and exceeding it produces "Failed to reserve memory for sort during spill".

The overall shape is worth internalizing: DataFusion’s spill implementation is not “write the hash table to disk”. It is “convert an unordered, unbounded in-memory structure into a sequence of sorted runs, then re-derive the aggregation as a streaming operation over the sorted result”. Every hard part — the mode rewrite, the group-key ordering, the merged-stream replay, the bounded-memory multi-level merge — comes from that one decision.

Partial Sort: How DataFusion Sorts a Stream That Is Already Sorted by a Prefix

Jay Hubbert — Wed, 21 Oct 2026 07:09:00 +0000

TL;DR: A full sort has to buffer everything before it can emit anything, because the last row might be the smallest. But if the input is already sorted by a prefix of the required ordering — say the data is ordered by (a, b) and you need (a, b, c) — then only the rows that share a prefix value need to be sorted together. DataFusion’s PartialSortExec buffers one prefix group at a time, sorts just that group, and emits it as soon as it can prove the group is complete. Peak memory drops from O(dataset) to O(largest group), and the first rows come out immediately.

A SortExec is a barrier. It consumes the entire input, sorts it, and then produces output. For a 100 GB table that means 100 GB of memory (or a spill to disk), and it means the first output row arrives only after the last input row has been read.

That is unavoidable for a full sort. If the data is completely unsorted, the smallest row could be anywhere, including the very end.

But real data is rarely completely unsorted. Parquet files written in order, time-series data, data that came out of a previous sort — all of it has some structure. And when the input is already ordered by a prefix of the required ordering, the problem decomposes.

The idea

The module’s header gives the example:

Consider a sort plan having an input with ordering a ASC, b ASC
text
+---+---+---+
| a | b | d |
+---+---+---+
| 0 | 0 | 3 |
| 0 | 0 | 2 |
| 0 | 1 | 1 |
| 0 | 2 | 0 |
+---+---+---+
and required ordering for the plan is a ASC, b ASC, d ASC. The first 3 rows(segment) can be sorted as the segment already has the required information for the sort, but the last row requires further information as the input can continue with a batch with a starting row where a and b does not change

So the input is sorted by (a, b), and we need (a, b, d). The rows with the same (a, b) prefix form a group, and within a group only the d column is out of order. Sort each group by d and concatenate — the result is fully sorted by (a, b, d).

The groups are:

(0, 0): rows with d = 3, 2, 1 → sorted to 1, 2, 3
(0, 1): one row
(0, 2): one row

And the groups appear in the input in prefix order, so the output is globally sorted.

The buffering rule

The subtle part is knowing when a group is complete. You cannot emit the group for prefix (0, 0) until you have seen a row with a different prefix — otherwise more rows with prefix (0, 0) might arrive in the next batch.

The rule, from the source:

[PartialSortExec] buffers rows only until it can prove a prefix group will never be seen again, then sorts and emits buffered rows. A group is guaranteed to never be seen again once a row with a different prefix value arrives. This relies on the input’s existing ordering guarantees.

The doc then walks through the buffer state:

 text
 
 
 Buffer
 +---+---+---+
 | a | b | c |
 +---+---+---+
 | 0 | 0 | 3 |
 | 0 | 0 | 2 |
 | 0 | 0 | 1 |
 +---+---+---+

“As long as the (a, b) prefix keeps repeating, more rows are buffered.”

Then, when a batch arrives with a new prefix:

 text
 
 
 Buffer
 +---+---+---+
 | a | b | c |
 +---+---+---+
 | 0 | 0 | 3 |
 | 0 | 0 | 2 |
 | 0 | 0 | 1 |
 | 0 | 1 | 1 | <-- first row of new batch, new prefix
 | 0 | 2 | 4 | <-- new prefix
 | 0 | 2 | 0 |
 | 1 | 0 | 5 | <-- last row of new batch, new prefix
 +---+---+---+

“Once known complete, each prefix group is sorted by the remaining suffix ordering and emitted as a RecordBatch. Any rows from the most recently seen prefix remain buffered, as more rows with the same prefix may arrive in future batches.”

So the algorithm is:

Accumulate rows into a buffer.
When a new prefix value appears, the previous prefix group is closed: sort it by the suffix and emit it.
The last (still open) group stays buffered.
At end of input, close and emit the final group.

The peak memory is therefore the size of the largest prefix group, not the size of the dataset. For data clustered by a low-cardinality key, that is a factor of thousands.

Why it is a “partial” sort

The word “partial” refers to the ordering guarantee: the input only partially satisfies the required ordering. The operator requires:

the input is ordered by a prefix of the required ordering, and
the operator sorts the suffix within each group.

So PartialSortExec is the composition of a “group by the prefix” and a “sort by the suffix within the group”. The prefix is a prefix of the required ordering, not an arbitrary grouping — that is what makes the concatenation of the sorted groups globally sorted.

The comparison is the expensive part

Within a group, the sort is by the remaining columns only. Arrow’s lexsort_to_indices does the work, but the comparator only needs the suffix columns, which is both less work and a smaller comparator.

The implementation uses Arrow’s make_comparator to build a comparator over the suffix columns and finds the group boundaries by comparing consecutive rows on the prefix columns:

 rust
 
use arrow_ord::ord::make_comparator;

That is the whole trick, expressed in two comparators over disjoint column sets: one to find where the groups are, one to sort within a group.

The catch

The input’s ordering guarantee is a promise. “This relies on the input’s existing ordering guarantees.” If the input is not actually sorted by the prefix, the output is silently wrong. This is the same class of trust as the range-partitioning contract we saw earlier.
A single huge group means no memory saving. If all the rows share the same prefix value, the operator degenerates to a full sort.
It only helps if the prefix is actually a prefix. Sorting by (a, b) when the input is sorted by (a) and then by (c) does not decompose.
The operator has to be placed correctly. It is a physical operator, and the planner has to know that the input’s ordering is a prefix of the required one. That is exactly the kind of fact EquivalenceProperties tracks.

Why it matters

The payoff is not just memory. It is latency: the first output batch comes out as soon as the first group closes, rather than after the entire input has been read and sorted. For a dashboard query that streams results, or for a LIMIT 10 on top of a sort, the difference is the difference between a second and a minute.

And it is a nice illustration of a general idea: sorting is only irreducible when the data is entirely unordered. Every bit of known structure in the input is a chance to break the problem into smaller problems — which is exactly what a partial sort does, one prefix group at a time.

One take, N slices: how DataFusion's RepartitionExec splits a batch across shuffle partitions

Jay Hubbert — Wed, 21 Oct 2026 07:01:00 +0000

TL;DR: BatchPartitioner never copies a batch once per partition. It computes one row-index vector per partition, concatenates them, performs a single take_arrays over the whole batch, and then hands out RecordBatch::slice(start, len) views over that one reordered buffer. Round-robin partitioning skips the take entirely — the whole batch goes to one output partition, whole.

The problem: a shuffle is a scatter-gather in disguise

Every parallel plan in DataFusion eventually has to move rows between worker threads. RepartitionExec is that operator: it maps N input partitions onto M output partitions according to a Partitioning scheme (datafusion/physical-expr/src/partitioning.rs:122-132):

 rust
 
 
pub enum Partitioning {
 /// Allocate batches using a round-robin algorithm and the specified number of partitions
 RoundRobinBatch(usize),
 /// Allocate rows based on a hash of one of more expressions and the specified number of partitions
 Hash(Vec<Arc<dyn PhysicalExpr>>, usize),
 /// Partition rows by source-declared ranges
 Range(RangePartitioning),
 /// Unknown partitioning scheme with a known number of partitions
 UnknownPartitioning(usize),
}

The naive implementation of “split this 8192-row batch into 16 partitions” is: for each partition, scan the batch and copy the matching rows into a fresh builder. That is 16 passes over the data and 16 output allocations. It is also almost never the right shape, because the expensive part of a shuffle is copying bytes, not deciding where rows go.

DataFusion’s answer splits the problem in two:

Routing (decide which row goes where) is done with plain integer indices — no data movement at all.
Materialization (move the bytes) is done once for the whole batch, and the per-partition results are slices of that one materialized buffer.

The router: `BatchPartitioner`

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:649-673
pub struct BatchPartitioner {
 state: BatchPartitionerState,
 timer: metrics::Time,
}

enum BatchPartitionerState {
 Hash {
 exprs: Vec<Arc<dyn PhysicalExpr>>,
 partition_reducer: StrengthReducedU64,
 hash_buffer: Vec<u64>,
 indices: Vec<Vec<u32>>,
 },
 RoundRobin {
 num_partitions: usize,
 next_idx: usize,
 },
 Range { /* ... */ }
}

Note the shape of the Hash state: there is no per-partition RecordBatch builder, and no per-partition data buffer. There is one hash_buffer: Vec<u64> (one hash per row) and one indices: Vec<Vec<u32>> (one list of row numbers per partition). Everything else is derived.

Round robin: the batch is the unit, not the row

Partitioning::RoundRobinBatch is a bit of a misnomer if you come from Spark: it does not interleave rows. The whole batch goes to one output partition, and the next batch goes to the next partition:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1240-1247
BatchPartitionerState::RoundRobin {
 num_partitions,
 next_idx,
} => {
 let idx = *next_idx;
 *next_idx = (*next_idx + 1) % *num_partitions;
 Box::new(std::iter::once(Ok((idx, batch))))
}

No hashing, no take, no copy — the batch is handed downstream unchanged. The only subtlety is where you start, and DataFusion solves it neatly so that N inputs feeding M outputs don’t all pile onto partition 0:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1055-1068
pub fn new_round_robin_partitioner(
 num_partitions: usize,
 timer: metrics::Time,
 input_partition: usize,
 num_input_partitions: usize,
) -> Self {
 Self {
 state: BatchPartitionerState::RoundRobin {
 num_partitions,
 next_idx: (input_partition * num_partitions) / num_input_partitions,
 },
 timer,
 }
}

With 2 inputs and 4 outputs, input 0 starts at partition 0 and input 1 starts at partition 2. Combined with the fact that each input alternates, the batches land evenly.

Hash: route with integers first

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1257-1279
let arrays = evaluate_expressions_to_arrays(exprs.as_slice(), &batch)?;

hash_buffer.clear();
hash_buffer.resize(batch.num_rows(), 0);

create_hashes(
 &arrays,
 REPARTITION_RANDOM_STATE.random_state(),
 hash_buffer,
)?;

for v in indices.iter_mut() {
 v.clear();
}

partition_reducer.partition_indices(hash_buffer, indices);

// Finished building index-arrays for output partitions
timer.done();

let partitioned_batches =
 Self::partition_grouped_take(&batch, indices, &self.timer)?;

Two details worth pausing on.

The hash seed is fixed. REPARTITION_RANDOM_STATE is a SeededRandomState::with_seed(0) (datafusion/physical-plan/src/repartition/mod.rs:675-677). A random seed per query would make the same key land in different partitions on different runs, breaking co-partitioning assumptions between the two sides of a join and making plans non-reproducible. Fixing the seed makes hash partitioning a pure function of the key, which is exactly what the planner’s “both children must be hash-partitioned on the same keys” rule relies on.

The modulo is strength-reduced. Computing hash % num_partitions for every row means an integer division per row, and integer division is expensive. DataFusion precomputes the compiler’s classic constant-divisor optimization once, at partitioner construction time:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:939-986
enum StrengthReducedU64 {
 PowerOfTwo { mask: u64 },
 Reciprocal { divisor: u64, reciprocal: u128 },
}

fn partition_indices(self, hash_buffer: &[u64], indices: &mut [Vec<u32>]) {
 match self {
 Self::PowerOfTwo { mask } => {
 for (index, hash) in hash_buffer.iter().enumerate() {
 indices[(*hash & mask) as usize].push(index as u32);
 }
 }
 Self::Reciprocal { divisor, reciprocal } => {
 for (index, hash) in hash_buffer.iter().enumerate() {
 let quotient = Self::quotient(*hash, reciprocal);
 let partition = *hash - quotient * divisor;
 indices[partition as usize].push(index as u32);
 }
 }
 }
}

Powers of two become a mask; everything else becomes a 128-bit reciprocal multiply (ceil(2^128 / divisor)) with the remainder recovered by subtraction. Either way the inner loop is a multiply plus a subtract — no div instruction at all.

Multi-column keys: hash-then-combine, not hash-of-a-tuple

A multi-column key (PARTITION BY a, b or a join on (a, b)) is hashed one column at a time, with the second and subsequent columns rehashing the running value rather than computing independent hashes and merging them:

 rust
 
 
// datafusion/common/src/hash_utils.rs:1264-1279
pub fn create_hashes<'a, I, T>(
 arrays: I,
 random_state: &impl HashState,
 hashes_buffer: &'a mut [u64],
) -> Result<&'a mut [u64]> {
 for (i, array) in arrays.into_iter().enumerate() {
 // combine hashes with `combine_hashes` for all columns besides the first
 let rehash = i >= 1;
 hash_single_array(array.as_dyn_array(), random_state, hashes_buffer, rehash)?;
 }
 Ok(hashes_buffer)
}

For primitive arrays the “rehash” path is especially cheap, because it seeds the hasher with the previous hash rather than hashing the value and then combining:

 rust
 
 
// datafusion/common/src/hash_utils.rs:322-333
if array.null_count() == 0 {
 if rehash {
 for (hash, &value) in hashes_buffer.iter_mut().zip(array.values().iter()) {
 let mut hasher = random_state.seeded_state(*hash).build_hasher();
 value.hash_write(&mut hasher);
 *hash = hasher.finish();
 }
 } else { /* ... */ }
}

The alternative — combine_hashes(l, r) = (17*37 + l) * 37 + r (hash_utils.rs:100-103) — is used for the generic (non-primitive) path. Both are order-sensitive, which is what you want: (a=1,b=2) and (a=2,b=1) must not collide.

The payoff: one take, then slices

Here is the whole trick, in one function:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1328-1391
/// Build repartitioned hash/range output batches using one `take` per input batch.
///
/// The routers first fills one index vector per output partition. This method
/// concatenates those index vectors, performs one grouped `take_arrays`, and
/// then returns each output partition as a slice of the reordered batch.
///
/// For example, given partition indices:
///
/// ```text
/// partition 0: [2, 5]
/// partition 1: []
/// partition 2: [0, 3, 4]
/// ```
///
/// this method takes rows in `[2, 5, 0, 3, 4]` order once, then returns
/// `partition 0 = slice(0, 2)` and `partition 2 = slice(2, 3)`.
fn partition_grouped_take(
 batch: &RecordBatch,
 indices: &mut [Vec<u32>],
 timer: &metrics::Time,
) -> Result<Vec<Result<(usize, RecordBatch)>>> {
 let mut partition_ranges = Vec::with_capacity(indices.len());
 let mut reordered_indices = Vec::with_capacity(batch.num_rows());

 for (partition, p_indices) in indices.iter_mut().enumerate() {
 if p_indices.is_empty() {
 continue;
 }
 let start = reordered_indices.len();
 reordered_indices.extend_from_slice(p_indices);
 partition_ranges.push((partition, start, p_indices.len()));
 p_indices.clear();
 }

 if reordered_indices.is_empty() {
 return Ok(vec![]);
 }

 if partition_ranges.len() == 1 && partition_ranges[0].2 == batch.num_rows() {
 let (partition, _, _) = partition_ranges[0];
 return Ok(vec![Ok((partition, batch.clone()))]);
 }

 let batches = {
 let _timer = timer.timer();
 let indices_array: PrimitiveArray<UInt32Type> = reordered_indices.into();
 let columns = take_arrays(batch.columns(), &indices_array, None)?;

 let mut options = RecordBatchOptions::new();
 options = options.with_row_count(Some(indices_array.len()));
 let reordered_batch =
 RecordBatch::try_new_with_options(batch.schema(), columns, &options)?;

 partition_ranges
 .into_iter()
 .map(|(partition, start, len)| {
 Ok((partition, reordered_batch.slice(start, len)))
 })
 .collect()
 };

 Ok(batches)
}

Read it as three distinct optimizations:

The index vectors are concatenated, not applied one at a time. Partition 0’s rows, then partition 1’s rows, then partition 2’s rows, all in one UInt32Array. That array is exactly the permutation we want.
One take_arrays for the entire batch, not one per partition. take_arrays is a gather: it produces new arrays whose contents are the selected rows in the requested order. Doing it once over 8192 rows × M partitions’ worth of indices costs the same per row as doing it M times over smaller index sets, but it allocates one output array per column instead of M, and it keeps the hot loop tight.
The per-partition batches are slices, not copies. RecordBatch::slice(start, len) produces a new RecordBatch that points at the same Arrow buffers with an offset and a length. No bytes are copied to make partition 2. This is why the function is called “grouped_take” and not “grouped_copy”.
text
```
original batch ─┐
 │ indices per partition: p0=[2,5] p1=[] p2=[0,3,4]
reordered order ─┴─► take() once ─► [2,5,0,3,4]
 ├── slice(0,2) ──► partition 0
 └── slice(2,3) ──► partition 2
 (same buffers, different offset)
```

There’s also a fast path: if all rows in the batch land in a single partition, the batch is returned as-is with no gather at all (mod.rs:1367-1370). That’s the degenerate case of a completely skewed key.

Why the slices don’t leak memory

A slice keeps the whole reordered batch alive, not just its slice. If partition 0 gets 2 rows and partition 2 gets 3 rows, both slices keep the 5-row buffer alive — which is fine, because together they are the whole buffer, and the buffer is dropped once all its slices are. The pathological case is one big partition plus many tiny ones: the tiny partitions keep a large buffer alive. This is bounded by the batch size, not by the dataset, so it’s a constant-factor concern rather than a leak.

Downstream: coalescing the shreds

A repartitioned batch is tiny — batch_size / M rows on average. Feeding 8192-row shards downstream would wreck every downstream operator’s throughput assumptions. So RepartitionExec coalesces on the producer side, before the channel, using one shared LimitedBatchCoalescer per output partition:

 rust
 
 
// datafusion/physical-plan/src/repartition/mod.rs:1461-1480
/// For this, one shared [`LimitedBatchCoalescer`] per output partition is used:
///
/// ```text
/// ┌───┐ ┌───┐
/// ┌─▶│ │────────▶.───────────. │ │ ┌──────────────────┐
/// │ └───┘ ┌───┐ ( Coalescer 0 )──▶ ├───┤ ───▶│ Output 0 │
/// │┌──────▶│ │──▶`───────────' │ │ └──────────────────┘
/// ││ └───┘ └───┘
/// ┌──────────────────┐ ││ ┌──────────────────┐
/// │BatchPartitioner 0│─┘│ │ Output 1 │
/// └──────────────────┘ │ └──────────────────┘
/// ```

The coalescer is shared across all input tasks targeting the same output partition (SharedCoalescer, mod.rs:276-325), with an AtomicUsize counting how many senders are still active so that the last one to finish flushes the residual batch. Coalescing is skipped for unbounded inputs, where withholding a partial batch would stall the stream forever (mod.rs:480).

The rest of the machinery, briefly

The routing above is only the CPU half. RepartitionExec also has to move the resulting batches between threads, and it does that with a deliberately unusual channel design:

 text
 
 
// datafusion/physical-plan/src/repartition/distributor_channels.rs:23-40
/// There are `N` virtual MPSC (multi-producer, single consumer) channels with
/// unbounded capacity. However, if all buffers/channels are non-empty, than a
/// global gate will be closed preventing new data from being written (the
/// sender futures will be [pending](Poll::Pending)) until at least one channel
/// is empty (and not closed).

A single global “gate” counter (empty_channels: AtomicUsize) implements global backpressure across all M output channels with one atomic increment/decrement per send — rather than a semaphore with M permits. The producer loop then yields to the runtime every num_partitions batches, because an endless input stream would otherwise never give the executor a chance to run (mod.rs:2347-2368).

Under memory pressure the whole path degrades gracefully: OutputChannel::send tries reservation.try_grow(size) and, on failure, spills the batch to a per-(input, output) SpillPool and sends a RepartitionBatch::Spilled marker instead of the data (mod.rs:232-252). The reader side runs a small state machine (StreamState::ReadingMemory / ReadingSpilled, mod.rs:2473-2480) that blocks on the spill stream until the spilled batch arrives, so FIFO order survives the memory/spill boundary.

Trade-offs and failure modes

The take is still a full copy. The slicing is free, but the single take_arrays copies every byte of every column into a new buffer. A shuffle therefore costs one full copy of the data, no matter how many partitions there are. This is the irreducible cost of moving data between threads; the optimization is that you pay it once rather than M times, and that the routing decision costs zero bytes.

Skew is the real enemy. Hash partitioning distributes keys, not rows. If 90% of rows share one key, 90% of the batch lands in one partition. The fast path at mod.rs:1367 only helps when the skew is total. With partial skew you get one huge output partition and M-1 nearly empty ones, and the downstream aggregate on that partition becomes the critical path. The planner’s defense is to hash on the exact key tuple rather than a subset (see the “strictest distribution match” note in datafusion/physical-optimizer/src/ensure_requirements/mod.rs:140-147), because adding more key columns spreads the hot key across more partitions. But no amount of planner cleverness fixes a genuinely skewed key: if country = 'US' is 90% of your data, hashing on (country, city) is the only fix, and only if the query semantics allow it.

Round robin trades locality for balance. RoundRobinBatch is perfectly balanced and needs no key evaluation, but it destroys key locality — every downstream group-by or join has to re-shuffle. That is exactly why the planner only inserts a round-robin when the operator has declared Distribution::UnspecifiedDistribution (i.e. it doesn’t care) but does benefit from more parallelism.

Range partitioning is the third option. Partitioning::Range routes rows by sorted split points rather than by hash (repartition/range.rs), so each partition owns a contiguous key range. It gives perfect balance and preserves sort order, at the cost of needing sampled split points and requiring the input to be sorted by the range key.

Bottom line

The shuffle is the single most performance-sensitive piece of DataFusion’s runtime, and its design is a good illustration of a general principle: decide with integers, move bytes once, and hand out views. The row-index vectors cost 4 bytes per row and are thrown away immediately; the one take_arrays produces a single reordered buffer; the per-partition RecordBatchs are zero-copy windows into it. The result is a shuffle whose cost is O(bytes) and whose routing cost is O(rows), with no hidden O(partitions × bytes) term.

DataFusion Skips the Hash Table When the Input Is Already Sorted by the Group Key

Jay Hubbert — Wed, 21 Oct 2026 06:53:00 +0000

TL;DR: AggregateExec asks the input’s equivalence properties for the longest permutation of the GROUP BY expressions that the existing sort order already satisfies. If that covers every key, the plan enters “full clustering” mode: a state machine knows when a group is complete, and GroupValuesClustered interns keys with no hash table, using Arrow’s partition kernel to find equal-key runs inside each batch. The same family of ideas ends in TopK, where the group set is bounded by the LIMIT.

Sorting is a hint, not a requirement

GROUP BY does not care about input order. But if the input happens to be sorted by the grouping key, then all rows of a group are contiguous, and contiguity is enormously valuable:

You can emit a group the moment you see a different key. No group is ever revisited.
You never need to look up “have I seen this key before?” — the answer is always “the previous row”.
Your memory footprint is one group, not N groups.

DataFusion does not require you to sort. It notices when the input is already sorted, and switches to a completely different execution path.

Detecting the clustering

The decision happens at plan construction, in AggregateExec::try_new_with_schema:

 rust
 
 
// datafusion/physical-plan/src/aggregates/mod.rs:1304-1360
 let input_eq_properties = input.equivalence_properties();
 // Get GROUP BY expressions:
 let groupby_exprs = group_by.input_exprs();
 // If existing ordering satisfies a prefix of the GROUP BY expressions,
 // prefix requirements with this section. In this case, aggregation will
 // work more efficiently.
 // Copy the `PhysicalSortExpr`s to retain the sort options.
 let (new_sort_exprs, indices) =
 input_eq_properties.find_longest_permutation(&groupby_exprs)?;

 ...
 // Constant expressions never change, so they cannot mark a completed group.
 // Exclude them from both the ordering indices and the group expression count.
 let indices: Vec<usize> = indices
 .into_iter()
 .filter(|idx| {
 group_by.groups.iter().all(|group| !group[*idx])
 && input_eq_properties
 .is_expr_constant(&groupby_exprs[*idx])
 .is_none()
 })
 .collect();

 let num_non_constant_groupby_exprs = groupby_exprs
 .iter()
 .filter(|expr| input_eq_properties.is_expr_constant(expr).is_none())
 .count();
 let mut group_clustering_mode = if indices.len() == num_non_constant_groupby_exprs
 && !indices.is_empty()
 && group_by.groups.len() == 1
 {
 GroupClusteringMode::Full
 } else if !indices.is_empty() {
 GroupClusteringMode::Partial(indices)
 } else {
 GroupClusteringMode::None
 };

Three outcomes:

 rust
 
 
// datafusion/physical-plan/src/aggregates/order/mod.rs:47-62
pub enum GroupClusteringMode {
 /// No clustering guarantee is available to complete groups before the input ends.
 None,
 /// Rows with the same values at these grouping-expression indices form one
 /// contiguous range. When those values change, every group in the previous
 /// range is complete and can be emitted.
 ///
 /// For example, with `GROUP BY (a, b)`, `Partial(vec![0])` means all rows
 /// for each value of `a` are contiguous, while an `(a, b)` tuple may recur
 /// within that range.
 Partial(Vec<usize>),
 /// Rows with the same complete grouping tuple form one contiguous range.
 /// When the tuple changes, the previous group can be emitted.
 Full,
}

Full — every group key tuple is contiguous. GROUP BY a, b on input sorted by (a, b).
Partial — some keys are contiguous. GROUP BY a, b on input sorted by a only: all rows for a given a are contiguous, but (a, b) tuples may recur inside that range.
None — no guarantee; use the hash table.

Note the two filters applied to indices: grouping sets are excluded (they rewrite the key expression per set, so the ordering argument does not transfer), and constant group expressions are excluded (a constant never changes, so it cannot mark a boundary — but it also must not be counted towards the “all keys covered” test).

The mode then selects the stream implementation:

 rust
 
 
// datafusion/physical-plan/src/aggregates/mod.rs:1723-1765
 let stream = match (self.mode, &self.group_clustering_mode) {
 (Partial, GroupClusteringMode::None) => StreamType::PartialHash(
 PartialHashAggregateStream::new(self, context, partition)?,
 ),
 (Partial, GroupClusteringMode::Partial(_) | GroupClusteringMode::Full) => {
 StreamType::ClusteredPartialAggregate(
 ClusteredPartialAggregateStream::new(self, context, partition)?,
 )
 }
 ...
 (Final | FinalPartitioned, GroupClusteringMode::None) => {
 StreamType::FinalHash(FinalHashAggregateStream::new(
 self, context, partition,
 )?)
 }
 (
 Final | FinalPartitioned,
 GroupClusteringMode::Partial(_) | GroupClusteringMode::Full,
 ) => StreamType::ClusteredFinalAggregate(ClusteredFinalAggregateStream::new(
 self, context, partition,
 )?),
 (Single | SinglePartitioned, GroupClusteringMode::None) => {
 StreamType::SingleHash(SingleHashAggregateStream::new(
 self, context, partition,
 )?)
 }
 (
 Single | SinglePartitioned,
 GroupClusteringMode::Partial(_) | GroupClusteringMode::Full,
 ) => StreamType::ClusteredSingleAggregate(
 ClusteredSingleAggregateStream::new(self, context, partition)?,
 ),
 };

Knowing when a group is over

The “completion tracker” is deliberately tiny. For the full-clustering case it is a three-state machine that remembers the id of the group currently being accumulated:

 rust
 
 
// datafusion/physical-plan/src/aggregates/order/full.rs:56-96
#[derive(Debug)]
pub struct GroupClusteringFull {
 state: State,
}

#[derive(Debug)]
enum State {
 /// Seen no input yet
 Start,

 /// Data is in progress. `current` is the current group for which
 /// values are being generated. Can emit `current` - 1
 InProgress { current: usize },

 /// Seen end of input: all groups can be emitted
 Complete,
}

impl GroupClusteringFull {
 // How many groups be emitted, or None if no data can be emitted
 pub fn emit_to(&self) -> Option<EmitTo> {
 match &self.state {
 State::Start => None,
 State::InProgress { current, .. } => {
 if *current == 0 {
 // Can not emit if still on the first row
 None
 } else {
 // otherwise emit all rows prior to the current group
 Some(EmitTo::First(*current))
 }
 }
 State::Complete => Some(EmitTo::All),
 }
 }

That is the whole trick: the number of the current group id is the number of groups that are complete. Because group ids are assigned densely in first-seen order and the input is clustered, group g is complete as soon as group g+1 appears. EmitTo::First(current) therefore means “everything before the group I am currently filling is done”.

For the partial-clustering case the tracker is a little richer, because the boundary is defined by a subset of the keys, and it has to compare the key values themselves:

 rust
 
 
// datafusion/physical-plan/src/aggregates/order/partial.rs:88-100
 /// Data is in progress.
 InProgress {
 /// Smallest group index in the current run.
 current_run_start: usize,
 /// The key values of the current run.
 group_key: Vec<ScalarValue>,
 /// index of the current group for which values are being
 /// generated
 current: usize,
 },

 rust
 
 
// order/partial.rs:192-207
 fn updated_group_key(
 current_run_start: usize,
 group_key: Option<Vec<ScalarValue>>,
 range_current_run_start: usize,
 range_group_key: Vec<ScalarValue>,
 ) -> Result<(usize, Vec<ScalarValue>)> {
 if let Some(group_key) = group_key {
 let sort_options = vec![SortOptions::new(false, false); group_key.len()];
 let ordering = compare_rows(&group_key, &range_group_key, &sort_options)?;
 if ordering == Ordering::Equal {
 return Ok((current_run_start, group_key));
 }
 }

 Ok((range_current_run_start, range_group_key))
 }

The boundary comparison uses Arrow’s compare_rows on the subset of key columns that define the run.

No hash table at all

Now the interesting part. When the input is fully clustered, the group key store doesn’t need a hash table. GroupValuesClustered interns keys using adjacency:

 rust
 
 
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/clustered.rs:30-45
/// Columnar group keys for input clustered by the complete grouping tuple.
///
/// Equal keys must be contiguous across input batches. Within a batch Arrow's
/// partition kernel finds those runs. Only the first run can continue the last
/// buffered group; compare it with the stored columns, without retaining the
/// input batch. New group representatives use the existing column builders.
///
/// This removes hashing and hash-table storage, but does not change the dense
/// group-id or emission contracts. In particular, `First(n)` shifts the remaining
/// keys and their ids together. Inputs clustered by only a subset of the keys
/// must use a hash table.
pub(crate) struct GroupValuesClustered {
 schema: SchemaRef,
 columns: Vec<Box<dyn GroupColumn>>,
 new_groups: Vec<usize>,
}

The intern is four steps:

 rust
 
 
// clustered.rs:102-134
impl GroupValues for GroupValuesClustered {
 fn intern(&mut self, cols: &[ArrayRef], groups: &mut Vec<usize>) -> Result<()> {
 groups.clear();
 let ranges = partition(cols)?.ranges();
 if ranges.is_empty() {
 return Ok(());
 }

 let old_len = self.len();
 let continues = old_len > 0
 && self
 .columns
 .iter()
 .zip(cols)
 .all(|(stored, input)| stored.equal_to(old_len - 1, input, 0));

 self.new_groups.clear();
 self.new_groups.extend(
 ranges
 .iter()
 .skip(usize::from(continues))
 .map(|range| range.start),
 );
 for (stored, input) in self.columns.iter_mut().zip(cols) {
 stored.vectorized_append(input, &self.new_groups)?;
 }

 let first_group = old_len - usize::from(continues);
 for (index, range) in ranges.iter().enumerate() {
 groups.resize(range.end, first_group + index);
 }
 Ok(())
 }

arrow_ord::partition::partition(cols) finds the runs of equal keys inside the batch — one vectorized kernel, no hashing.
The first run might be a continuation of the last group of the previous batch, so the stored column builders are compared against input row 0. If they match, that run is not a new group.
One representative row per new run is appended to the column builders — the same GroupColumn machinery as the hash-based path.
The group id for every row is first_group + run_index, computed with a resize (fill-with-value) rather than a per-row loop.

The memory layout is still columnar, but the “hash table” is gone entirely:

 text
 
 
 before (hash-based) after (clustered)

 HashTable<(u64, GroupIndexView)> (no hash table)
 0x9f2c -> group 0
 0x4411 -> group 1
 ... column builders hold one
 columns: [10, 20, 10] representative row per group:
 ["a","b","c"] [10, 20]
 ... ["a", "b"]

The correctness caveat is spelled out in the type’s own docs: the only reason this works is that the caller promised the input is clustered. If it is not, the same key would appear in two runs and become two groups. The promise comes from GroupClusteringMode::Full, which was derived from the input’s ordering properties.

The supported types are constrained:

 rust
 
 
// clustered.rs:47-51
impl GroupValuesClustered {
 /// Types for which adjacent Arrow equality agrees with GROUP BY equality.
 /// Keep single-column specializations and floats/nested/encoded keys on the
 /// established path until their semantics and performance are validated.
 pub(crate) fn supports_schema(schema: &Schema) -> bool {
 schema.fields().len() > 1

“Adjacent Arrow equality agrees with GROUP BY equality” is the invariant that makes partition usable as a group detector. Floats are excluded (because GROUP BY folds -0.0 into +0.0 but Arrow’s equality does not), as are nested and dictionary-encoded types.

Note also fields().len() > 1: the single-column clustered case does not use the hash-free path. It uses GroupValuesColumn::<true> — the streaming variant of the hash-based columnar store, whose scalarized_intern preserves the property that group ids are assigned in input order.

Streaming out completed groups

The clustered partial stream materializes the completed prefix in bulk rather than dribbling groups out one at a time:

 rust
 
 
// datafusion/physical-plan/src/aggregates/clustered_partial_stream.rs:62-72
/// # Group Clustering Optimization
///
/// For the aggregation work, the hash aggregation implementation is reused.
///
/// After each input batch, the group-clustering mode determines whether any
/// groups can be emitted eagerly to improve memory efficiency. For example, if
/// the input is ordered by `k` and the last group key seen is `k = 100`, all
/// groups with keys less than 100 are complete. Materialize that entire completed
/// prefix once, then emit slices of it before reading more input. This avoids
/// repeatedly removing small batches of groups and shifting the remaining hash
/// table and accumulator state.

“Materialize the entire completed prefix once” matters: emitting groups one at a time would require renumbering the surviving group ids and rewriting the hash table on every batch. Bulk emission amortizes that.

The memory story is the headline:

 rust
 
 
// clustered_partial_stream.rs:74-90
/// ## Full group clustering
///
/// Every complete grouping tuple is contiguous. Ordering by every group key is
/// one way to establish this mode ...
///
/// Completed groups can be emitted as soon as the next group is observed. Thus,
/// only the current group remains active after completed groups are emitted, and
/// memory usage does not grow with the total number of groups.
///
/// If a memory reservation nevertheless fails, the stream returns the error
/// directly, indicating an unexpected behavior.
///
/// ## Partial group clustering
///
/// Rows are contiguous for a subset of the group keys ...
///
/// If one `a` value contains many distinct `b` values, the table may accumulate
/// enough groups to exceed the memory limit.
///
/// - `ClusteredPartialAggregateStream`: On reservation failure, it emits all current
/// intermediate states downstream and resets the table. The final stage can
/// merge repeated `(a, b)` state rows, so no disk spill is required.
/// - `ClusteredFinalAggregateStream`: It cannot emit incomplete final results. On
/// reservation failure, it sorts the current intermediate states by the complete
/// group key and spills them as one run. ...

Partial clustering is the interesting middle ground: the runs are complete, but the individual (a, b) groups inside a run are not, so the hash table still lives — but it only has to hold the groups of the current run, and on memory pressure the stream can flush and reset without spilling to disk, because the final stage can merge duplicate (a, b) state rows.

The other way to bound the group set: TopK

There is a second, orthogonal way to avoid an unbounded group set: don’t compute all the groups at all, compute only the ones that can survive the LIMIT.

AggregateExec::try_optimize_topk recognizes two shapes:

 rust
 
 
// datafusion/physical-plan/src/aggregates/mod.rs:949-1002
 /// ## TopK MIN/MAX
 ///
 /// ```txt
 /// SELECT k, MAX(v) AS score FROM t GROUP BY k
 /// ORDER BY score DESC NULLS LAST LIMIT 10;
 ///
 /// Before:
 /// Sort: score DESC, fetch=10
 /// Aggregate: k, MAX(v)
 /// input
 ///
 /// After (Added info in `AggregateExec` to trigger heap optimization):
 /// Sort: score DESC, fetch=10
 /// Aggregate: k, MAX(v), TopKMinMax(10)
 /// input
 /// ```
 /// ...
 /// ## TopK DISTINCT
 ///
 /// ```txt
 /// SELECT DISTINCT k FROM t ORDER BY k ASC NULLS LAST LIMIT 10;
 ///
 /// Before:
 /// Sort: k ASC, fetch=10
 /// Aggregate: k, no aggregates
 /// input
 ///
 /// After:
 /// Sort: k ASC, fetch=10
 /// Aggregate: k, TopKDistinct(10)
 /// input
 /// ```

The optimizer keeps the Sort/Limit above (the aggregate may still produce more than K rows); it just annotates the aggregate so the execution can use a bounded heap instead of a full hash table.

GroupedTopKAggregateStream drives a PriorityMap, which is a hash table and a heap in one:

 rust
 
 
// datafusion/physical-plan/src/aggregates/topk/priority_map.rs:27-37
/// A `Map<K, V>` / `PriorityQueue` combo that evicts the worst values after reaching `capacity`
pub struct PriorityMap {
 map: Box<dyn ArrowHashTable + Send>,
 heap: Box<dyn ArrowHeap + Send>,
 capacity: usize,
 mapper: Vec<(usize, usize)>,
 /// Mirror of the map's all-NULL group count, kept as a plain field so the
 /// per-row `insert` path can check it without a `dyn` call (measured to
 /// regress the topk_aggregate benchmarks when read through the trait)
 null_count: usize,
}

The eviction test is one comparison against the heap’s worst element:

 rust
 
 
// topk/priority_map.rs:60-70
 pub fn insert(&mut self, row_idx: usize) -> Result<()> {
 debug_assert_eq!(self.null_count, 0);

 // if we're full, and the new val is worse than all our values, just bail
 if self.heap.is_worse(row_idx) {
 return Ok(());
 }
 assert!(self.map.len() <= self.capacity, "Overflow");

 self.insert_eligible(row_idx)
 }

 rust
 
 
// topk/heap.rs:179-186
 fn is_worse(&self, row_idx: usize) -> bool {
 if !self.heap.is_full() {
 return false;
 }
 let new_val = self.batch.value(row_idx);
 let worst_val = self.heap.worst_val().expect("Missing root");
 (!self.desc && new_val > *worst_val) || (self.desc && new_val < *worst_val)
 }

The hash table underneath is not a HashMap but an index-addressed store, so the heap can refer to entries by index instead of by hash:

 rust
 
 
// topk/hash_table.rs:41-52
/// A custom wrapper around `hashbrown::HashTable` that:
/// 1. limits the number of entries to the top K
/// 2. Allocates a capacity greater than top K to maintain a low-fill factor and prevent resizing
/// 3. Tracks indexes to allow corresponding heap to refer to entries by index vs hash
struct TopKHashTable<ID> {
 map: HashTable<usize>,
 // Store the actual items separately to allow for index-based access
 store: Vec<HashTableItem<ID>>,
 // Free indexes in the store for reuse
 free_indices: Vec<usize>,
 // Pool of reusable value locations, usually Strings
 free_slots: Vec<ID>,
 // The maximum number of entries allowed
 limit: usize,

Note the free-list for string slots: evicted strings are recycled rather than dropped, which matters when the key is a String and the workload is high-cardinality.

The group-by-only (TopKDistinct) variant uses the group key as both the key and the priority value, so SELECT DISTINCT k ... LIMIT 10 keeps the 10 smallest keys rather than the 10 arbitrary first-seen keys.

Trade-offs

You still pay for the sort. The clustering optimization only removes the hash table; the SortExec that produced the sorted input is still in the plan. For a query that would not otherwise be sorted, adding a sort to enable clustering is usually a loss — which is why DataFusion only uses the clustering guarantee when the input already has it.
The guarantee is per-partition. The clustering mode is derived from the input’s ordering properties, which are per-partition. If the plan has a RepartitionExec that does not preserve order, the guarantee is gone, and the mode reverts to None for the downstream aggregate.
Partial clustering still needs the hash table. Only full clustering removes it. With GROUP BY a, b on input sorted by a only, the (a, b) groups inside an a run still require interning, and the memory bound is “the number of groups in the current run”, not “one group”.
The type whitelist is conservative. Floats, nested types and dictionary-encoded keys stay on the hash path, because Arrow’s adjacent equality does not agree with GROUP BY equality for them. The comment says so: “Keep single-column specializations and floats/nested/encoded keys on the established path until their semantics and performance are validated.”
The hash-free path is multi-column only. Single-column clustering uses the streaming variant of the hash-based store, because the single-column primitive specialization was already faster than a partition-kernel-based approach.
TopK changes the answer’s completeness, not its correctness. The aggregate may produce more than K rows (hence the Limit stays above), and all-NULL groups are tracked separately so they are not silently dropped:

 rust
 
 
// topk/priority_map.rs:128-137
 /// Track a group whose aggregate values are all NULL, so it can be emitted
 /// with a NULL value. MIN/MAX ignore NULL inputs, but an all-NULL group
 /// must still appear in the aggregation output; such groups all tie on the
 /// sort key, so tracking up to `capacity` of them preserves top-k semantics.
 pub fn insert_null(&mut self, row_idx: usize) {

TopK only applies to a narrow shape. One group key, one MIN/MAX aggregate or no aggregate at all, no filter, no grouping sets:

 rust
 
 
// aggregates/mod.rs:1010-1016
 if limit == 0
 || matches!(self.kind, AggregateKind::DistinctLimit { .. })
 || self.group_by().has_grouping_set()
 || self.filter_expr().iter().any(Option::is_some)
 {
 return None;
 }

The takeaway

DataFusion treats input ordering as a capability, not as a requirement. The plan asks “what does the input already guarantee?”, and the answer picks one of three execution paths. In the strongest case — the input is clustered by the complete grouping tuple — the group hash table is not merely optimized, it is absent, replaced by Arrow’s partition kernel and a three-state machine that knows when a group is finished. In the TopK case, the group set is bounded by the query’s own LIMIT, and the hash table becomes an index-addressed store feeding a heap that evicts as it goes.

Both are the same insight: the number of live groups is the thing to minimize, and every input property that bounds it is worth a specialized code path.

Datafusion on Opensource internals

The Contract Every DataFusion Operator Signs: A Tour of ExecutionPlan's Declarative Methods

The questions

“What distribution do your inputs need?”

“Are your inputs already sorted?”

“Can you absorb a LIMIT?”

“Does a filter on your output reduce your output?”

“Can I swap you with a projection?”

“What filters can I take?”

“Can I push a sort below you?”

The properties object

Why it matters

A Tour of DataFusion's Physical Operators: The Ten That Matter and the Trick Each One Uses

1. DataSourceExec — the scan

2. FilterExec — the filter

3. ProjectionExec — the projection

4. RepartitionExec — the shuffle

5. HashJoinExec — the equi-join

6. SortExec — the sort

7. SortPreservingMergeExec — the merge

8. AggregateExec — the aggregation

9. WindowAggExec — the window function

10. CoalescePartitionsExec — the fan-in

The ones you will also see

How they compose

Why it matters

The DataFusion Performance Cheat Sheet: The Config Knobs and Optimizer Rules That Actually Move the Needle

Parallelism

Pushdown

Spilling

The join choices

The Parquet knobs

The sort and aggregate knobs

The optimizer knobs

The catch

Why it matters

Writing Parquet Row Groups in Parallel: Column Chunks, Dictionary Pages and the Sort-Before-Write Trick

The shape of the write path

Two writer strategies

Parallel column encoding inside one file

Row groups are sealed concurrently, concatenated in order

Row group sizing

Dictionary encoding and statistics: write-side decisions, read-side wins

The sort-before-write trick

The catch

Signatures, Volatility, and the Type-Coercion Lattice: How DataFusion's UDF Metadata Drives the Optimizer

Two problems one struct solves

TypeSignature: arity × type shape

The coercion lattice

Finding the common type

Volatility: the promise about time

The dispatch cost: one virtual call per batch, not per row

The registry

Trade-offs

Cooperative Budgets: How DataFusion Stops One Operator From Starving the Whole Runtime

The starvation problem

The mechanism: spend budget, then yield

Two wrappers, one for statics and one for trait objects

Making it automatic: EnsureCooperative

Coordinating N tasks and M channels

The buffer that is exactly the right size

Panics, cancellation, and common-runtime

spawn_blocking — and where not to use it

The memory-table case

Trade-offs and failure modes

The Perfect Hash Join: Turning a Hash Table Into a Plain Array When the Keys Are Dense Integers

The array map

When to use it

The density heuristic

The overflow trap

The partitioning trap

The measurement

The catch

The general lesson

How DataFusion Turns a Hash Join's Build Side into a Filter the Probe Side Can Execute

Two flavours of build side, two flavours of filter

The publication strategy

Bounds: the cheap pre-filter

The barrier: why a partial filter is a bug

Routing: one filter, N partitions

1. `DataSourceExec` — the scan

2. `FilterExec` — the filter

3. `ProjectionExec` — the projection

4. `RepartitionExec` — the shuffle

5. `HashJoinExec` — the equi-join

6. `SortExec` — the sort

7. `SortPreservingMergeExec` — the merge

8. `AggregateExec` — the aggregation

9. `WindowAggExec` — the window function

10. `CoalescePartitionsExec` — the fan-in

`TypeSignature`: arity × type shape

`Volatility`: the promise about time

Making it automatic: `EnsureCooperative`

Panics, cancellation, and `common-runtime`

`spawn_blocking` — and where not to use it

The vocabulary: `Precision<T>`

Step 1: `BETWEEN` is not special

Step 2: intervals over `ScalarValue`

`LIKE` becomes a range

`IN` lists and the size cap

The unhandled hook: the constant `true`

`AND` is forgiving, `OR` is not

`MemoryPool` — the policy

`MemoryConsumer` — the identity

`MemoryReservation` — the RAII handle

`UnboundedMemoryPool`

`GreedyMemoryPool`

`FairSpillPool`

`MergeMemoryPool`: one consumer, many reservations

What happens when `try_grow` fails

What `swap_inputs` actually has to fix

The `Accumulator` trait: one group, one object