DataFusion's Spillable Shuffle: Writing Batches to Disk Without Reordering Them
The shuffle spills selectively, not wholesale: one batch at a time, into a per-(input,output) FIFO pool, with a marker batch telling the reader to switch to disk.
Datafusion
How DataFusion Spills a Hash Aggregation to Disk Without Breaking GROUP BY
Spilling a GROUP BY means re-implementing its semantics on disk: sort the state runs, merge them with bounded memory, then replay through a clustered aggregate.
Partial Sort: How DataFusion Sorts a Stream That Is Already Sorted by a Prefix
DataFusion’s PartialSortExec exploits an input that is already sorted by a prefix of the required ordering: it buffers one prefix group at a time, sorts just the suffix, and emits …
One take, N slices: how DataFusion's RepartitionExec splits a batch across shuffle partitions
The repartition operator hashes each row once, concatenates per-partition row indices, performs exactly one take over the whole batch, and then slices the reordered result into …
DataFusion Skips the Hash Table When the Input Is Already Sorted by the Group Key
When the input is already clustered by the grouping key, DataFusion detects it from the input’s ordering properties, tracks group completion with a tiny state machine, and interns …