Skip to content
Opensource internals
  • Posts

Partitioning

Splitting File Groups by Statistics: Giving Each Partition a Contiguous Slice of the Sort Key

Oct 21, 2026 6 min read Jay Hubbert

DataFusion’s statistics-based file group splitter assigns files to partitions by best-fit on their min/max ranges, preserving the table’s ordering while balancing the file counts.

DatafusionFile scanPartitioningPerformanceInternals

Read more →

Range Partitioning in DataFusion: Replacing the Hash Shuffle with Boundaries Chosen from the Data

Oct 21, 2026 11 min read Jay Hubbert

DataFusion’s range partitioning cuts an ordered key space into N partitions at sampled boundaries, so a shuffle stays balanced even under skew and the partitions keep their order.

DatafusionPartitioningShufflePerformanceInternals

Read more →

© 2026 Jay Hubbert · Built with Hugo 0.168.0

RSS ·