Skip to content
Opensource internals
  • Posts

Datafusion

EquivalenceProperties: How DataFusion Knows Its Data Is Already Sorted Before It Sorts It

Oct 21, 2026 11 min read Jay Hubbert

DataFusion’s EquivalenceProperties tracks equal columns, constants and valid sort orderings through the plan, which lets it remove redundant sorts, repartitions and joins.

DatafusionOptimizerSortingPerformanceInternals

Read more →

Interval Arithmetic in DataFusion: Proving That a Predicate Can Only Match Certain Values

Oct 21, 2026 8 min read Jay Hubbert

DataFusion’s constraint propagator builds a directed acyclic expression graph, computes each node’s value interval bottom-up, then propagates the filter’s constraint back down to …

DatafusionPruningIntervalsPerformanceInternals

Read more →

Prepared Hash Join Builds: Sharing One Build Side Across Many Probes

Oct 21, 2026 8 min read Jay Hubbert

DataFusion’s prepared hash join build freezes a build side into an immutable object with its own memory reservation, so an embedding executor can share one dimension snapshot …

DatafusionJoinsMemoryPerformanceInternals

Read more →

HashJoin vs SortMergeJoin vs NestedLoopJoin: What DataFusion Actually Uses Statistics For

Oct 21, 2026 19 min read Jay Hubbert

DataFusion chooses the join algorithm from predicate shape, not cardinality; statistics then pick the build side, the partition mode, and the output row count.

DatafusionPlanningPerformanceInternals

Read more →

DataFusion's Dynamic Filters: Letting the Producer Tell the Consumer What It Learned at Runtime

Oct 21, 2026 10 min read Jay Hubbert

DataFusion passes runtime facts between operators through a shared, mutable filter expression with a generation counter and a watch channel, so a hash join can narrow a scan that …

DatafusionDynamic filtersPerformanceInternals

Read more →

← Newer Page 8 of 15 Older →

© 2026 Jay Hubbert · Built with Hugo 0.168.0

RSS ·