DataFusion’s IN-list evaluation picks a filter by list size and type: a branchless comparison chain for short lists, a bitmap for narrow types, a hash set for wide ones.
Scalar UDFs receive whole Arrow arrays per batch, not one ScalarValue per row — ColumnarValue is the currency that makes that possible.
DataFusion’s PartitionedTopK keeps one bounded heap per partition key, sharing a single row converter, memory reservation and row store, so a per-partition top-k costs O(N) instead …
DataFusion pushes row filters into the Parquet decoder itself: the filter’s columns are decoded first, the surviving rows are known before the wide columns are touched, and …
DataFusion’s optimizer detects that it has reached a fixed point by hashing each pass’s plan into a signature of node count plus content hash and checking a set of previous …