A guided tour of DataFusion’s physical operators — the scan, the filter, the projection, the repartition, the joins, the aggregate, the sort and the window — and the trick in each.
The repartition operator hashes each row once, concatenates per-partition row indices, performs exactly one take over the whole batch, and then slices the reordered result into …
DataFusion pushes LIMIT into operators that can absorb it, so a ten-row query stops the scan, the join and the sort after ten rows instead of after a full 8192-row batch.
WindowAggExec asks the planner for hash-partitioned, sorted input, then walks each partition’s frame with a sliding accumulator that adds rows entering the frame and retracts rows …
Instead of hard-coding shuffles, each ExecutionPlan states the distribution it needs from its children; a single optimizer rule turns unmet requirements into RepartitionExec nodes.