Planning on Opensource internals

DataFusion's Interval Analysis: How `BETWEEN` Becomes a Scalar Range and a Selectivity Number

Jay Hubbert — Wed, 21 Oct 2026 08:21:00 +0000

TL;DR: a BETWEEN 10 AND 20 is desugared into a >= 10 AND a <= 20 by the expression simplifier, then re-absorbed into a single Interval (two ScalarValue endpoints) by the constraint-propagation solver. That interval is intersected with the column’s min/max from statistics, and the ratio of the two interval cardinalities becomes the filter selectivity. No histogram, no sampling, no join — just two numbers per column and a divide.

Every cost-based optimizer eventually hits the same wall: to choose between a hash join and a nested loop join, you need to know how many rows will survive a filter. Most systems solve this with histograms, sampling, or a learned model. All three are expensive, all three are approximate, and all three are a source of embarrassing production incidents.

Apache DataFusion takes a much cheaper route. It has exactly two pieces of information per column — a minimum and a maximum — and it derives a filter’s selectivity from the geometric relationship between the range implied by the predicate and the range the data actually occupies.

This article walks through the whole path: how BETWEEN becomes an interval, how the interval is propagated, and where the resulting number ends up.

The vocabulary: `Precision<T>`

Before anything else, the tri-state that everything else is built on:

 rust
 
 
// datafusion/common/src/stats.rs:31-60
pub enum Precision<T: Debug + Clone + PartialEq + Eq + PartialOrd> {
 /// The exact value is known. Used for guaranteeing correctness.
 Exact(T),
 /// The value is not known exactly, but is likely close to this value.
 /// Used for cost-based optimizations.
 Inexact(T),
 /// Nothing is known about the value. This is the default state.
 #[default]
 Absent,
}

Exact is not “approximately right”. The doc comment is explicit: exact values are “used for guaranteeing correctness” — they answer COUNT(*), MIN and MAX without scanning and let the optimizer delete limits. Inexact is the cost-based-optimization tier. Absent is an absorbing element: any arithmetic involving Absent yields Absent.

This distinction matters later, because the pruning layer (covered in a companion piece) refuses to use anything but Exact bounds.

The container is small:

 rust
 
 
// datafusion/common/src/stats.rs:368-383
pub struct Statistics {
 pub num_rows: Precision<usize>,
 pub total_byte_size: Precision<usize>,
 pub column_statistics: Vec<ColumnStatistics>,
}

and the per-column part is where the interesting fields live:

 rust
 
 
// datafusion/common/src/stats.rs:1080-1114
pub struct ColumnStatistics {
 pub null_count: Precision<usize>,
 pub max_value: Precision<ScalarValue>,
 pub min_value: Precision<ScalarValue>,
 pub sum_value: Precision<ScalarValue>,
 pub distinct_count: Precision<usize>,
 pub byte_size: Precision<usize>,
}

Note the shape: min_value and max_value are ScalarValues, not typed numbers. The same struct covers Int64, Utf8, Timestamp and Decimal256 without a type parameter, at the cost of runtime type dispatch everywhere.

Step 1: `BETWEEN` is not special

The first thing DataFusion does with a BETWEEN 10 AND 20 is throw away the Between node entirely. The expression simplifier has a rule for it:

 rust
 
 
// datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:1654-1669
//
// Rules for Between
//
// a between 3 and 5 --> a >= 3 AND a <=5
// a not between 3 and 5 --> a < 3 OR a > 5
Expr::Between(between) => Transformed::yes(if between.negated {
 let l = *between.expr.clone();
 let r = *between.expr;
 or(l.lt(*between.low), r.gt(*between.high))
} else {
 and(
 between.expr.clone().gt_eq(*between.low),
 between.expr.lt_eq(*between.high),
 )
}),

This looks like a loss of information — we now have two independent conjuncts instead of one node. It is not. Two comparisons against the same column is exactly the shape the interval solver wants, and the solver is the thing that puts the range back together.

Step 2: intervals over `ScalarValue`

The interval type is deliberately minimal:

 rust
 
 
// datafusion/expr-common/src/interval_arithmetic.rs:177-181
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Interval {
 lower: ScalarValue,
 upper: ScalarValue,
}

A NULL endpoint means “unbounded on this side”. The constructor is strict about ordering, because an inverted interval is always a bug:

 rust
 
 
// datafusion/expr-common/src/interval_arithmetic.rs:267-285
pub fn try_new(lower: ScalarValue, upper: ScalarValue) -> Result<Self> {
 assert_eq_or_internal_err!(
 lower.data_type(), upper.data_type(),
 "Endpoints of an Interval should have the same type"
 );
 let interval = Self::new(lower, upper);
 assert_or_internal_err!(
 interval.lower.is_null()
 || interval.upper.is_null()
 || interval.lower <= interval.upper,
 "Interval's lower bound {} is greater than the upper bound {}",
 interval.lower, interval.upper
 );
 Ok(interval)
}

Self::new also standardizes the awkward cases: NaN becomes unbounded, +INF becomes f64::MAX for a lower bound and NULL for an upper bound, and unsigned lower bounds default to zero rather than -inf.

Step 3: the column’s interval comes from statistics

The solver’s input is one interval per column, built straight out of ColumnStatistics:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:97-129
pub fn try_from_column(
 schema: &Schema,
 col_stats: &ColumnStatistics,
 col_index: usize,
) -> Result<Self> {
 let field = schema.fields().get(col_index).ok_or_else(|| { ... })?;
 let empty_field =
 ScalarValue::try_from(field.data_type()).unwrap_or(ScalarValue::Null);
 let interval = Interval::try_new(
 col_stats.min_value.get_value().cloned().unwrap_or_else(|| empty_field.clone()),
 col_stats.max_value.get_value().cloned().unwrap_or(empty_field),
 )?;
 ...
}

So ColumnStatistics { min_value: Exact(Int32(0)), max_value: Exact(Int32(999)) } becomes Interval { lower: Int32(0), upper: Int32(999) }, and a column with absent statistics becomes an unbounded interval.

Step 4: propagating the predicate backwards

analyze() is the entry point. It builds an expression DAG, seeds the leaf nodes with the intervals from statistics, and pushes the predicate’s constraints down into them:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:164-232 (excerpt)
pub fn analyze(
 expr: &Arc<dyn PhysicalExpr>,
 context: AnalysisContext,
 schema: &Schema,
) -> Result<AnalysisContext> {
 ...
 let mut target_boundaries = context.boundaries;
 let mut graph = ExprIntervalGraph::try_new(Arc::clone(expr), schema)?;
 ...
 match graph.update_ranges(&mut target_indices_and_boundaries, Interval::TRUE)? {
 PropagationResult::Success => {
 shrink_boundaries(&graph, target_boundaries, &target_expr_and_indices)
 }
 PropagationResult::Infeasible => {
 // If the propagation result is infeasible, set intervals to None
 for bound in &mut target_boundaries { bound.interval = None; }
 Ok(AnalysisContext::new(target_boundaries).with_selectivity(0.0))
 }
 PropagationResult::CannotPropagate => {
 Ok(AnalysisContext::new(target_boundaries).with_selectivity(1.0))
 }
 }
}

Three outcomes, and they map cleanly onto “how selective is this filter”: Success means the constraint could be pushed down and we can compute a selectivity; Infeasible means the predicate contradicts the data (the filter matches nothing); CannotPropagate means we learned nothing, so assume everything passes.

The propagation itself is where the AND of two comparisons gets merged back into one interval. For a comparison under a TRUE parent:

 rust
 
 
// datafusion/physical-expr/src/intervals/cp_solver.rs:353-373
pub fn propagate_comparison(
 op: &Operator,
 parent: &Interval,
 left_child: &Interval,
 right_child: &Interval,
) -> Result<Option<(Interval, Interval)>> {
 if parent == &Interval::TRUE {
 match op {
 Operator::Eq => left_child.intersect(right_child).map(|result| {
 result.map(|intersection| (intersection.clone(), intersection))
 }),
 Operator::Gt => satisfy_greater(left_child, right_child, true),
 Operator::GtEq => satisfy_greater(left_child, right_child, false),
 Operator::Lt => satisfy_greater(right_child, left_child, true)
 .map(|t| t.map(reverse_tuple)),
 Operator::LtEq => satisfy_greater(right_child, left_child, false)
 .map(|t| t.map(reverse_tuple)),
 ...

a >= 10 under TRUE shrinks a’s interval from below; a <= 20 shrinks it from above. Because both are propagated against the same column node, the column’s interval ends up as [10, 20] — the BETWEEN range, reconstructed.

The whole flow:

text

 a BETWEEN 10 AND 20
 │
 ▼ expr_simplifier.rs:1660
 (a >= 10) AND (a <= 20)
 │
 ▼ cp_solver.rs:353 propagate_comparison
 column a interval: [0, 999] ∩ [10, +inf] ∩ (-inf, 20]
 │
 ▼
 column a interval: [10, 20] ← the exact scalar interval
 │
 ▼ cardinality_ratio (interval_arithmetic.rs:1740)
 selectivity = |[10,20]| / |[0,999]|

Step 5: from interval to number

The selectivity is a ratio of cardinalities:

 rust
 
 
// datafusion/expr-common/src/interval_arithmetic.rs:1740-1747
pub fn cardinality_ratio(initial_interval: &Interval, final_interval: &Interval) -> f64 {
 match (final_interval.cardinality(), initial_interval.cardinality()) {
 (Some(final_interval), Some(initial_interval)) => {
 (final_interval as f64) / (initial_interval as f64)
 }
 _ => 1.0,
 }
}

Interval::cardinality() only works for numeric, date, timestamp and decimal types — for a Utf8 column it returns None, and the ratio degrades to 1.0 (no filtering). This is the single biggest limitation of the whole scheme, and it is why DataFusion has a separate string-specific estimator (string_selectivity in datafusion/physical-plan/src/filter.rs:1292) that falls back to a configurable default.

Two more subtleties live in calculate_selectivity:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:305-344
fn calculate_selectivity(
 target_boundaries: &[ExprBoundaries],
 initial_boundaries: &[ExprBoundaries],
) -> Result<f64> {
 // Since the intervals are assumed uniform and the values
 // are not correlated, we need to multiply the selectivities
 // of multiple columns to get the overall selectivity.
 let mut acc: f64 = 1.0;
 for (initial, target) in initial_boundaries.iter().zip(target_boundaries) {
 match (initial.interval.as_ref(), target.interval.as_ref()) {
 (Some(initial_interval), Some(target_interval)) => {
 if let Precision::Exact(distinct_count)
 | Precision::Inexact(distinct_count) = target.distinct_count
 && let Some(s) = singleton_selectivity(
 initial_interval, target_interval, distinct_count,
 )
 {
 acc *= s;
 continue;
 }
 acc *= cardinality_ratio(initial_interval, target_interval);
 }
 ...

Two things to notice. First, the “uniform distribution” assumption is explicit in the comment: values are assumed uniformly spread over the interval and columns are assumed uncorrelated. city = 'Paris' AND state = 'CA' multiplies two independent selectivities, which is wrong whenever they are correlated — and they usually are.

Second, the singleton_selectivity shortcut. If the filter collapsed the column’s interval to a single point, the ratio-based estimate would still be 1/999 even when the column holds only 10 distinct values. The shortcut replaces the ratio with 1 / distinct_count:

 rust
 
 
// datafusion/physical-expr/src/analysis.rs:274-300
fn singleton_selectivity(
 initial_interval: &Interval,
 target_interval: &Interval,
 distinct_count: usize,
) -> Option<f64> {
 if distinct_count == 0
 || target_interval.lower().is_null()
 || target_interval.lower() != target_interval.upper()
 {
 return None;
 }
 // Only treat this as a newly-applied equality filter when the initial
 // interval was not already that same singleton.
 let initial_is_same_singleton = !initial_interval.lower().is_null()
 && initial_interval.lower() == initial_interval.upper()
 && initial_interval.lower() == target_interval.lower();
 if initial_is_same_singleton {
 return None;
 }
 Some(1.0 / distinct_count as f64)
}

The initial_is_same_singleton guard is a bug fix worth calling out: if the statistics already said the column was a single value, applying 1/NDV again would double-count the restriction and under-estimate the row count.

Step 6: where the number is consumed

FilterExec is where the estimate becomes a Statistics:

 rust
 
 
// datafusion/physical-plan/src/filter.rs:389-406
if check_support(predicate, schema) {
 let input_analysis_ctx = AnalysisContext::try_from_statistics(
 schema,
 &input_stats.column_statistics,
 )?;
 let analysis_ctx = analyze(predicate, input_analysis_ctx, schema)?;
 let selectivity = analysis_ctx.selectivity.unwrap_or(1.0);
 let filtered_num_rows =
 input_num_rows.with_estimated_selectivity(selectivity);
 let cs = collect_new_statistics(
 schema,
 &input_stats.column_statistics,
 analysis_ctx.boundaries,
 selectivity,
 &null_rejecting_columns,
 filtered_num_rows,
 );
 (selectivity, filtered_num_rows, cs)

with_estimated_selectivity is where the estimate is explicitly downgraded to Inexact, which is the honest thing to do:

 rust
 
 
// datafusion/common/src/stats.rs:200-206
pub fn with_estimated_selectivity(self, selectivity: f64) -> Self {
 if self == Precision::Exact(0) {
 return self;
 }
 self.map(|v| ((v as f64 * selectivity).ceil()) as usize)
 .to_inexact()
}

Notice ceil(): a filter never increases the row count, but rounding down would claim it might. Rounding up is the conservative direction.

The boundaries that come out of analyze are also fed into collect_new_statistics, which tightens the min/max of the surviving columns. So WHERE a BETWEEN 10 AND 20 does not just produce a row count; it produces a narrower ColumnStatistics for a, which then propagates upward to the join, the aggregate, and the next filter.

The memoization that makes this affordable

Interval analysis is not cheap: it builds a petgraph DAG per expression. Running it once per filter per plan would dominate planning time on wide plans. DataFusion’s answer is a memoized bottom-up walk:

 rust
 
 
// datafusion/physical-plan/src/statistics.rs:39-68
type CacheKey = (usize, Option<usize>);

fn cache_key(plan: &dyn ExecutionPlan, partition: Option<usize>) -> CacheKey {
 (from_ref::<dyn ExecutionPlan>(plan).cast::<()>() as usize, partition)
}

/// Per-context memoization cache for statistics computation.
///
/// Keys are `(plan node pointer address, partition)`. Every entry retains the
/// `Arc` whose pointer supplied its key, preventing a dropped plan's address
/// from being reused by a different plan while the entry exists.

The key is a raw pointer, and each cache entry holds the Arc that supplied the pointer. That is a small, clever trick: it makes address reuse impossible while the entry is alive, so the cache can be keyed on identity without inventing a plan-wide unique id. The cost is that the cache keeps dead plans alive until reset_cache() is called.

The walk itself consults a pluggable provider chain before the operator’s built-in statistics_from_inputs, which is the extension point used to feed catalog-level statistics into the optimizer.

Trade-offs and failure modes

The uniform distribution assumption is the big one. A column with a Zipfian or exponential distribution will produce a selectivity that is wrong by an order of magnitude. DataFusion has no histogram, no sketch, no sample. If your amount column has 99% of its mass in [0, 100] and the file’s min/max is [0, 1000000], then amount < 1000 gets a selectivity of 0.001 when the truth is closer to 0.5. The planner then thinks the join build side is tiny and picks the wrong one.

Independence across columns. calculate_selectivity multiplies per-column selectivities. Correlated predicates multiply twice, so the estimated cardinality can be off by orders of magnitude in the low direction.

Non-numeric columns get nothing. cardinality() returns None for Utf8, List, Struct. The interval machinery simply does not apply, and DataFusion falls back to string_selectivity or a flat default.

Exact versus Inexact is a correctness boundary, not a hint. A filter’s output row count is always Inexact unless it was provably zero. The optimizer is allowed to use Exact values to skip work entirely (removing a LIMIT because the input has exactly 5 rows), so a source that lies about Exact gets wrong answers, not just bad plans.

NOT BETWEEN is subtler than it looks. NOT (a BETWEEN -1 AND 1) over a ∈ [-1, 1] is infeasible — no row can satisfy it — and the solver correctly reports None for the interval. But NOT (a BETWEEN -1 AND 1) over a ∈ [-2, 2] does have rows, and the solver has to widen the interval rather than intersect it. The test at datafusion/physical-expr/src/analysis.rs:715-752 pins this behavior down.

Signed zero. NOT (a = +0.0) over a ∈ [-0.0, +0.0] must be infeasible, because +0.0 = +0.0 is TRUE in IEEE 754 even though PartialEq on ScalarValue is bit-wise. The solver has explicit handling for this, and the comment at datafusion/physical-expr/src/analysis.rs:782-790 explains exactly which short-circuit it defeats.

Configuration

There is no single “enable interval analysis” switch. The knobs that matter:

Option	Default	Effect
`datafusion.execution.collect_statistics`	`true`	Whether file/table statistics are gathered at all. Without them, every interval is unbounded and selectivity is 1.0.
`datafusion.optimizer.skip_failed_rules`	`false`	If an optimizer rule throws (e.g. an interval construction error), fail the query rather than silently continuing.
`datafusion.optimizer.max_passes`	`3`	How many times the logical optimizer re-runs the rule set; more passes mean more statistics-driven rewrites get a chance to fire.
`FILTER_EXEC_DEFAULT_SELECTIVITY` (`filter.rs:80`)	`20`	Not a config key but a `FilterExecBuilder::with_default_selectivity` knob: the fallback selectivity percentage for string predicates the interval machinery cannot handle.

The big lever is collect_statistics. Turn it off and every downstream estimate collapses to Absent: no pruning, no build-side selection, no interval analysis. It is also the knob that matters most for listing-heavy workloads, because gathering statistics means reading Parquet footers.

Bottom line

The trick that makes DataFusion’s planner work without histograms is that it does not try to be right about distributions. It only tries to be right about ranges, and it converts a range comparison into a number by dividing interval cardinalities. That is cheap, it is local, and it degrades gracefully — but the moment your data is skewed, or your columns are correlated, or your predicate is on a string, the estimate is fiction.

The honest summary: DataFusion’s statistics are a fast, structured guess, and the codebase is refreshingly explicit about it — every comment in analysis.rs that says “assumed uniform” or “not correlated” is a warning label left by whoever had to debug the resulting plan.

DataFusion's PruningPredicate: Turning a Filter into a Per-File 'Can This Possibly Match?' Test

Jay Hubbert — Wed, 21 Oct 2026 05:17:00 +0000

TL;DR: PruningPredicate compiles your filter into an expression over statistics columns (x_min, x_max, x_null_count, x_row_count) instead of over data columns. x = 5 becomes x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max). The compiled expression is evaluated once, vectorized, against a batch holding one row per container — so pruning 10,000 files costs roughly the same as pruning one.

There is a subtle trap in query engines, and it is easy to fall into.

You have a filter WHERE ts BETWEEN '2024-01-01' AND '2024-01-31'. The naive implementation is: open the file, read the rows, evaluate the predicate per row, discard the ones that fail. That is correct, it is simple, and it buys you exactly nothing — you paid for the I/O you were trying to avoid.

The whole point of a zone map is to answer a different question: can this file possibly contain a matching row? If the answer is provably “no”, you skip the file without opening it.

DataFusion’s implementation lives in the datafusion-pruning crate, and it is more interesting than it looks: not just min/max comparison, but a small compiler that rewrites arbitrary boolean expressions into an equivalent expression over statistics columns, then evaluates that expression in a vectorized batch.

The contract: `PruningStatistics`

The input to the whole machinery is a trait that supplies per-container statistics as Arrow arrays:

 rust
 
 
// datafusion/common/src/pruning.rs:63-97
pub trait PruningStatistics {
 /// Return the minimum values for the named column, if known.
 ///
 /// If the minimum value for a particular container is not known, the
 /// returned array should have `null` in that row. If the minimum value is
 /// not known for any row, return `None`.
 ///
 /// Each non-null entry must be a conservative lower bound for all non-null
 /// values in its container, using Arrow's comparison order for the column
 /// type. For strings, this is unsigned lexicographic UTF-8 byte order.
 /// Bounds that cannot satisfy this contract must be reported as unknown.
 ///
 /// Note: the returned array must contain [`Self::num_containers`] rows
 fn min_values(&self, column: &Column) -> Option<ArrayRef>;
 ...
 fn num_containers(&self) -> usize;

Two design decisions here are worth dwelling on.

First: the arrays are columnar, not per-container. The trait does not have “give me the stats for file 37”. It has “give me the minimums for column x across all 3,412 containers, as one array”. That is the amortization trick, and it is stated in the trait’s own doc comment:

 rust
 
 
// datafusion/common/src/pruning.rs:41-45
/// Information for containers / files are returned as Arrow [`ArrayRef`], so
/// the evaluation happens once on a single [`RecordBatch`], which amortizes the
/// overhead of evaluating the predicate. This is important when pruning 1000s
/// of containers which often happens in analytic systems that have 1000s of
/// potential files to consider.

Second: the contract is on the provider. “Bounds that cannot satisfy this contract must be reported as unknown. Pruning relies on providers to uphold this contract; it does not validate the ordering or bound guarantees of arbitrary statistics sources.” This is the load-bearing sentence of the whole design. DataFusion does not check that your min is really the minimum. If you lie, you get wrong answers — not a warning, not a fallback. Answers.

Compiling the predicate

The rewriting is documented, in prose, right at the top of the crate:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:242-251
/// Original Predicate | Rewritten Predicate
/// ------------------ | --------------------
/// `x = 5` | `x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max)`
/// `x < 5` | `x_null_count != x_row_count AND (x_min < 5)`
/// `x = 5 AND y = 10` | `x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max) AND y_null_count != y_row_count (y_min <= 10 AND 10 <= y_max)`
/// `x IS NULL` | `x_null_count > 0`
/// `x IS NOT NULL` | `x_null_count != row_count`
/// `CAST(x as int) = 5` | `x_null_count != x_row_count (CAST(x_min as int) <= 5 AND 5 <= CAST(x_max as int))`

The null_count != row_count term is the subtle one. It exists to express “this column is not entirely NULL”. If x is all NULLs, then x = 5 is UNKNOWN for every row, and UNKNOWN rows are filtered out — so the container can be pruned. But you can only prove that from the null count. Without the term, an all-NULL container with x_min = NULL would evaluate the rewritten predicate to NULL rather than FALSE, and NULL means “keep”.

That distinction is the crux of the entire design:

true: there MAY be rows that match the predicate → KEEP

NULL: there MAY be rows that match the predicate → KEEP

false: there are NO rows that could possibly match → PRUNE

Note that this is not SQL filter semantics, where NULL means “filtered out”. The pruning predicate inverts it: NULL is unknown, and unknown means “don’t prune, be conservative”.

The rewrite happens in build_predicate_expression, a recursive descent over the physical expression tree:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:1780-1996 (excerpt)
fn build_predicate_expression(
 expr: &Arc<dyn PhysicalExpr>,
 schema: &SchemaRef,
 required_columns: &mut RequiredColumns,
 unhandled_hook: &Arc<dyn UnhandledPredicateHook>,
 max_in_list_size: usize,
 properties: &mut PruningExpressionProperties,
) -> Arc<dyn PhysicalExpr> {
 if is_always_false(expr) {
 // Shouldn't return `unhandled_hook.handle(expr)`
 // Because it will transfer false to true.
 return Arc::clone(expr);
 }
 if let Some(is_null) = expr.downcast_ref::<phys_expr::IsNullExpr>() { ... }
 if let Some(in_list) = expr.downcast_ref::<phys_expr::InListExpr>() { ... }
 ...
 if op == Operator::And || op == Operator::Or {
 let left_expr = build_predicate_expression(&left, ...);
 let right_expr = build_predicate_expression(&right, ...);
 // simplify boolean expression if applicable
 ...
 return expr;
 }
 ...
 build_statistics_expr(&mut expr_builder)
 .unwrap_or_else(|_| unhandled_hook.handle(expr))
}

Two properties of this design are important:

Partial failure is fine. If the left half of an AND can be rewritten and the right half cannot, the unrewritable half is replaced by the UnhandledPredicateHook — which by default returns literal true:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:584-603
/// The default handling for unhandled predicates is to return a constant `true`
/// (meaning don't prune the container)
#[derive(Debug, Clone)]
struct ConstantUnhandledPredicateHook {
 default: Arc<dyn PhysicalExpr>,
}

So an unsupported conjunct degrades to “don’t prune on this side”, and the other conjunct still prunes. AND is the friendliest operator for pruning; OR is the worst, because A OR B can only be pruned if both A and B are provably false.

The rewrite is not just min/max on the column. rewrite_expr_to_prunable normalizes the expression so that the left side is monotonically increasing in the column:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:1243-1317 (excerpt)
fn rewrite_expr_to_prunable(
 column_expr: &PhysicalExprRef,
 op: Operator,
 scalar_expr: &PhysicalExprRef,
 schema: &Schema,
) -> Result<(PhysicalExprRef, Operator, PhysicalExprRef)> {
 if !is_compare_op(op) {
 return plan_err!("rewrite_expr_to_prunable only support compare expression");
 }
 if column_expr.downcast_ref::<phys_expr::Column>().is_some() {
 // `col op lit()`
 Ok((Arc::clone(column_expr), op, Arc::clone(scalar_expr)))
 } else if let Some(cast) = column_expr.downcast_ref::<phys_expr::CastExpr>() {
 ...
 } else if let Some(neg) = column_expr.downcast_ref::<phys_expr::NegativeExpr>() {
 // `-col > lit()` --> `col < -lit()`
 ...
 } else if let Some(not) = column_expr.downcast_ref::<phys_expr::NotExpr>() {
 // `!col = true` --> `col = !true`
 ...
 } else {
 plan_err!("column expression {column_expr:?} is not supported")
 }
}

The reason is obvious in hindsight: x_min is only a valid lower bound for x if x appears monotonically. CAST(x AS INT) = 5 is handled by casting the bounds rather than the column; -x > 5 is rewritten to x < -5; !x = true is rewritten to x = false. But abs(a - 10) > 0 is rejected, because abs is not monotonic and abs(x_min - 10) is not a bound at all.

There is even a type-safety check for casts, because cast ordering is not the same as source ordering:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:1351-1375
fn verify_support_type_for_prune(from_type: &DataType, to_type: &DataType) -> Result<()> {
 ...
 // If both types are strings or both are not strings (number, timestamp, etc)
 // then we can compare them.
 // PruningPredicate does not support casting of strings to numbers and such.
 if from_type.is_string() == to_type.is_string() {
 Ok(())
 } else {
 plan_err!(
 "Try Cast/Cast with from type {from_type} to type {to_type} is not supported"
 )
 }
}

("13" < "3" is true in UTF-8 order but "13" < 3 is nonsense, so string→number casts are rejected.)

The IN-list trick: a sorted domain instead of an OR tree

The default rewrite for x IN (1, 2, 3) is to expand it into (x = 1) OR (x = 2) OR (x = 3) and rewrite each — producing a deep tree of min/max comparisons. For long lists that is expensive to build and slow to evaluate.

DataFusion has a compact alternative, gated behind a raised datafusion.execution.parquet.max_in_list_size:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:459-483
/// Cap on the input length of `IN (...)` lists eligible for min/max pruning.
///
/// For a nonempty list of `N` values (before deduplication) and cap `C`:
///
/// | Condition | Pruning representation |
/// | --- | --- |
/// | `N <= min(20, C)` | Existing per-value rewrite |
/// | `20 < N <= C`, supported ordered literals with optional NULLs | Compact pruning expression |
/// | `20 < N <= C`, unsupported lists | Existing per-value rewrite |
/// | `N > C` | Unhandled-predicate hook, normally "keep the container" |

The compact representation sorts the values once and then does a single binary-search-style probe per container. It is gated behind a raised datafusion.execution.parquet.max_in_list_size — with the default cap of 20 the compact path never engages, because it requires a list longer than 20:

 rust
 
 
// datafusion/pruning/src/primitive_in_list.rs:252-278
fn may_match(&self, min: Option<T::Native>, max: Option<T::Native>) -> Option<bool> {
 if self.membership == SetMembership::NotIn {
 return not_in_may_match(min.as_ref(), max.as_ref(), |value| {
 self.contains(*value)
 });
 }
 match (min, max) {
 (Some(min), Some(max)) => {
 // Inverted statistics are unusable, not proof that the domain
 // and interval are disjoint.
 if min > max {
 return None;
 }
 let index = self.values.partition_point(|value| *value < min);
 Some(self.values.get(index).is_some_and(|value| *value <= max))
 }
 (Some(min), None) if self.values.last().is_some_and(|value| *value < min) => {
 Some(false)
 }
 (None, Some(max))
 if self.values.first().is_some_and(|value| *value > max) =>
 {
 Some(false)
 }
 _ => None,
 }
}

partition_point finds the first value >= min; if that value is <= max, the container’s interval intersects the domain, so keep it; if it is > max, the container’s whole range is above the domain, so prune it. One O(log n) probe replaces n comparisons.

The NOT IN case is deliberately asymmetric, and the reasoning is a gem:

 rust
 
 
// datafusion/pruning/src/in_list.rs:22-32
/// Which `IN` form a sorted domain is pruning for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub(crate) enum SetMembership {
 /// `col IN (...)`. A row matches only where the domain intersects the
 /// interval, so a disjoint interval excludes every row.
 In,
 /// `col NOT IN (...)`. Overlap proves nothing here: values outside the
 /// domain still satisfy the predicate. An interval excludes every row only
 /// when it holds a single value that the domain contains.
 NotIn,
}

For NOT IN, “the container’s range overlaps the domain” says nothing — values outside the domain satisfy NOT IN. The only way to exclude a container is if its entire range is a single value that the domain contains:

 rust
 
 
// datafusion/pruning/src/in_list.rs:53-75
/// Evaluates `NOT IN` against an inclusive statistics interval.
///
/// A container can be excluded only when both bounds identify one value in the
/// domain. A known bound outside the domain proves the container may match and
/// lets an enclosing Boolean expression short-circuit.
///
/// This remains safe when Parquet truncates byte-array statistics. A truncated
/// minimum is no greater than the true minimum, and a truncated maximum is no
/// less than the true maximum. Therefore, equal stored bounds prove that the
/// true minimum and maximum are also equal.
#[inline(always)]
pub(crate) fn not_in_may_match<T: ?Sized + PartialEq>(
 min: Option<&T>,
 max: Option<&T>,
 contains: impl Fn(&T) -> bool,
) -> Option<bool> {
 match (min, max) {
 (Some(min), Some(max)) if contains(min) => Some(min != max),
 (Some(_), Some(_)) => Some(true),
 (Some(bound), None) | (None, Some(bound)) if !contains(bound) => Some(true),
 _ => None,
 }
}

That last doc paragraph is a nice piece of engineering: Parquet truncates long byte-array statistics to a prefix, so min may be less than the true minimum and max may be greater than the true maximum. Equal stored bounds therefore prove equal true bounds (because the true min is >= stored min and the true max is <= stored max, and both are pinned to the same value), so the argument is still sound under truncation.

The string/binary variant does the same thing on raw bytes, with a length comparison first to avoid scanning long common prefixes:

 rust
 
 
// datafusion/pruning/src/string_in_list.rs:206-246
match (min, max) {
 (Some(min), Some(max)) => {
 if min > max { return None; }
 // String and binary values use unsigned lexicographic byte
 // order, as required by the PruningStatistics min/max contract.
 let index = self.values.partition_point(|v| $bytes!(v) < min);
 Some(self.values.get(index).is_some_and(|v| $bytes!(v) <= max))
 }
 // A missing bound makes that end of the interval unbounded.
 // Exclude only when the whole domain lies beyond the known bound;
 // gaps within the domain and equality cannot prove disjointness.
 (Some(min), None)
 if self.values.last().is_some_and(|v| $bytes!(v) < min) => Some(false),
 (None, Some(max))
 if self.values.first().is_some_and(|v| $bytes!(v) > max) => Some(false),
 _ => None,
}

The two-pass prune

Now the actual evaluation. PruningPredicate::prune does two passes:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:657-707
pub fn prune<S: PruningStatistics + ?Sized>(
 &self,
 statistics: &S,
) -> Result<Vec<bool>> {
 let mut builder = BoolVecBuilder::new(statistics.num_containers());

 // Try to prove the predicate can't be true for the containers based on
 // literal guarantees
 for literal_guarantee in &self.literal_guarantees {
 let LiteralGuarantee { column, guarantee, literals } = literal_guarantee;
 if let Some(results) = statistics.contained(column, literals) {
 match guarantee {
 Guarantee::In => builder.combine_array(&results),
 Guarantee::NotIn => {
 builder.combine_array(&arrow::compute::not(&results)?)
 }
 }
 // if all containers are pruned ... can return early
 if builder.check_all_pruned() {
 return Ok(builder.build());
 }
 }
 }

 // Next, try to prove the predicate can't be true for the containers based
 // on min/max values

 // build a RecordBatch that contains the min/max values in the
 // appropriate statistics columns for the min/max predicate
 let statistics_batch =
 build_statistics_record_batch(statistics, &self.required_columns)?;

 // Evaluate the pruning predicate on that record batch and append any results to the builder
 builder.combine_value(self.predicate_expr.evaluate(&statistics_batch)?);

 Ok(builder.build())
}

Pass one uses LiteralGuarantees — a predicate like x IN ('a', 'b') proves that the column can only contain those values, and the contained hook lets a provider answer “does this container contain only values from this set?”. Parquet Bloom filters implement exactly this API. The pass can short-circuit entirely if every container is already proven prunable.

Pass two evaluates the compiled min/max expression against a synthesized RecordBatch of statistics columns. One evaluate call, one batch, all containers.

The flow:

text

 filter expr (x >= lo AND x <= hi)
 │
 ▼ build_predicate_expression
 compiled predicate expr over statistics columns
 │
 ├─ RequiredColumns tells us which columns to fetch:
 │ x_min, x_max, x_null_count, x_row_count
 │
 ▼ build_statistics_record_batch
 ┌────────────┬────────┬────────┬────────────┬───────────┐
 │ x_min │ x_max │ null_ct│ row_count │ (y_min..) │
 ├────────────┼────────┼────────┼────────────┼───────────┤
 │ file0: 0 │ file0: 99 │ ... │ ... │ ... │
 │ file1: 500 │ file1: 600 │ ... │ ... │ ... │
 │ ... │ ... │ ... │ ... │ ... │
 └────────────┴────────┴────────┴────────────┴───────────┘
 │
 ▼ predicate_expr.evaluate(&batch) ← ONE vectorized call
 [true, false, true, ...]
 │
 ▼
 skip file1 entirely; never open it

The amortized cost model

This is the part that is easy to get wrong, so let’s be precise about the costs.

Building the predicate is expensive. It requires expression analysis, a recursive rewrite, a RequiredColumns analysis, and a simplification pass:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:546-561
// build predicate expression once
let mut required_columns = RequiredColumns::new();
let mut properties = PruningExpressionProperties::default();
let predicate_expr = build_predicate_expression(
 &predicate, &file_schema, &mut required_columns,
 &unhandled_hook, self.max_in_list_size, &mut properties,
);
let predicate_schema = required_columns.schema();
// Simplify the newly created predicate to get rid of redundant casts, comparisons, etc.
let predicate_expr =
 PhysicalExprSimplifier::new(&predicate_schema).simplify(predicate_expr)?;
let literal_guarantees = LiteralGuarantee::analyze(&predicate);

Evaluating it per container is cheap, because the compiled expression is pure min/max arithmetic over a few arrays.

So the invariant is: pay the compilation cost once, amortize it over as many containers as possible. Evaluating the original predicate per row costs O(rows). Evaluating a compiled predicate per file costs O(files × predicate_size). Evaluating it once over a vectorized batch costs the same O(files × predicate_size) in SIMD-friendly Arrow kernels, with no per-file dispatch at all.

DataFusion’s file-level pruner makes the “build once” part explicit:

 rust
 
 
// datafusion/pruning/src/file_pruner.rs:127-170
pub fn should_prune(&mut self) -> Result<bool> {
 // Building the pruning predicate is expensive (it involves expression
 // analysis), so we only do it on the first check and whenever a dynamic
 // filter inside the predicate has actually moved.
 let should_build = if self.checked_once {
 self.tracking.watcher().is_some_and(|w| w.changed())
 } else {
 self.checked_once = true;
 true
 };
 if !should_build {
 return Ok(false);
 }
 let pruning_predicate = build_pruning_predicate(
 Arc::clone(&self.predicate),
 &self.file_schema,
 &self.predicate_creation_errors,
 );
 ...
}

DynamicFilterTracking is the interesting bit: a hash join can push a runtime filter (a list of build-side keys) into the probe-side scan. That filter’s value changes as the build side fills, so the pruning predicate must be rebuilt when it moves — but not on every batch. The watcher reports a change at most once per update.

And the Parquet row-group pruner takes the “one call for all row groups” approach:

 rust
 
 
// datafusion/datasource-parquet/src/row_group_filter.rs:367-380
// try to prune the row groups in a single call
match predicate.prune(&pruning_stats) {
 Ok(values) => {
 let mut fully_contained_candidates_original_idx: Vec<usize> =
 Vec::with_capacity(row_group_indexes.len());
 for (idx, &value) in row_group_indexes.iter().zip(values.iter()) {
 if !value {
 self.access_plan.skip(*idx);
 metrics.row_groups_pruned_statistics.add_pruned(1);
 } else {
 metrics.row_groups_pruned_statistics.add_matched(1);
 fully_contained_candidates_original_idx.push(*idx);
 }
 }
 ...

One prune() call, one boolean array, N row groups decided.

Where file statistics come from — and the `Exact` trap

The bridge from “file statistics” to “pruning statistics” is PrunableStatistics:

 rust
 
 
// datafusion/common/src/pruning.rs:342-372
fn get_exact_column_statistics(
 &self,
 column: &Column,
 get_stat: impl Fn(&ColumnStatistics) -> &Precision<ScalarValue>,
) -> Option<ArrayRef> {
 let index = self.schema.index_of(column.name()).ok()?;
 let mut has_value = false;
 match ScalarValue::iter_to_array(self.statistics.iter().map(|s| {
 s.column_statistics.get(index)
 .and_then(|stat| {
 if let Precision::Exact(min) = get_stat(stat) {
 has_value = true;
 Some(min.clone())
 } else {
 None
 }
 })
 .unwrap_or(ScalarValue::Null)
 })) {
 // If there is any non-null value and no errors, return the array
 Ok(array) => has_value.then_some(array),
 ...

Only Precision::Exact bounds are used for pruning. An Inexact min is treated as unknown. This is deliberate and correct: pruning on an approximate bound could skip a file that does contain matching rows, which is a wrong answer, not a slow query. It also means that if your catalog reports approximate statistics, you get no file-level pruning at all — only row-group pruning from Parquet footers, which is exact by construction. null_counts and row_counts behave the same way: they are reported only if at least one container has an exact value, and per-container inexactness is encoded as a NULL in the array.

Partition values fold into the predicate

For Hive-partitioned tables, the partition column values live in the path, not in the file. DataFusion substitutes them into the predicate before pruning:

 rust
 
 
// datafusion/physical-expr-adapter/src/schema_rewriter.rs:64-83
pub fn replace_columns_with_literals<K, V>(
 expr: Arc<dyn PhysicalExpr>,
 replacements: &HashMap<K, V>,
) -> Result<Arc<dyn PhysicalExpr>>
where
 K: Borrow<str> + Eq + Hash,
 V: Borrow<ScalarValue>,
{
 expr.transform_down(|expr| {
 if let Some(column) = expr.downcast_ref::<Column>()
 && let Some(replacement_value) = replacements.get(column.name())
 {
 return Ok(Transformed::yes(expressions::lit(
 replacement_value.borrow().clone(),
 )));
 }
 Ok(Transformed::no(expr))
 })
 .data()
}

So WHERE year = 2024 AND amount < 50 becomes, for a file under year=2024/, 2024 = 2024 AND amount < 50 — and the first conjunct folds to literal true in the simplifier, leaving a pruning predicate that only mentions amount. The FilePruner docstring states the invariant plainly:

 rust
 
 
// datafusion/pruning/src/file_pruner.rs:32-36
/// Prune based on file-level statistics.
///
/// Note: Partition column pruning is handled earlier via `replace_columns_with_literals`
/// which substitutes partition column references with their literal values before
/// the predicate reaches this pruner.

Trade-offs and failure modes

Wrong statistics produce wrong answers. This is not a “slow query” class of bug. The PruningStatistics contract says providers must supply conservative bounds, and the code explicitly declines to validate them. A catalog that reports min = 0 for a file whose true min is 100 will get that file pruned for WHERE x < 50, and the rows in [50, 100) will silently vanish.

Pruning on NULL means “keep”. Every rewritten predicate is careful to produce NULL (unknown) rather than FALSE when the statistics are incomplete. This makes pruning conservative but also means that absent statistics prune nothing. A Parquet file written without statistics, or a column excluded from the footer, disables pruning for that column entirely.

OR prunes badly. A OR B can only exclude a container if both sides are provably false. If one side references an unsupported expression, the whole OR degrades to “keep”. Filter pushdown rewrites that turn a conjunctive filter into a disjunction will silently kill pruning.

All-NULL columns need the null-count term. Without x_null_count != x_row_count, an all-NULL container would evaluate the rewritten x = 5 to NULL and be kept — forever, for every predicate.

String ordering is a footgun. The contract requires unsigned lexicographic UTF-8 byte order, matching Arrow. Parquet’s own ordering conventions differ, and DataFusion’s Parquet adapter has to mask bounds whose ordering it cannot establish:

 rust
 
 
// datafusion/datasource-parquet/src/row_group_filter.rs:289-292
/// This method has no file footer, so it cannot establish the sort order
/// of string or binary min/max statistics. Such bounds are ignored. Use
/// [`Self::prune_by_statistics_with_metadata`] when the full metadata is
/// available.

Pruning is not free. Compiling the predicate involves expression analysis. For a scan over 3 files, pruning may cost more than reading them. There is no “is pruning worth it” heuristic at this layer; the predicate_creation_errors counter exists so you can at least see failures.

Configuration

Option	Default	Effect
`datafusion.execution.parquet.max_in_list_size`	`20`	Cap on `IN (...)` list length eligible for min/max pruning. Lists of ≤ 20 values use the per-value rewrite; the compact sorted-domain form needs a list of > 20 values, so it only engages if you raise the cap above 20. `0` disables the `IN` rewrite entirely (other pruning still works).
`datafusion.execution.collect_statistics`	`true`	Whether file-level statistics are gathered. Without them, `PrunableStatistics` has nothing exact to work with and file-level pruning is dead.

MAX_IN_LIST_SIZE is the hard-coded floor for the per-value path:

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:1686-1693
/// Default maximum number of entries in an `IN (...)` list eligible for
/// statistics pruning.
pub const MAX_IN_LIST_SIZE: usize = 20;

Bottom line

The pruning layer is a small compiler with a very specific target: turn a boolean expression over data columns into a boolean expression over statistics columns, so that the answer to “which files can match?” is computed with vectorized Arrow kernels over one synthesized batch rather than by opening files.

The cleverness is not in the min/max comparison — every columnar engine has that. It is in (a) the rewrite rules that make arbitrary predicates expressible over bounds, including the careful null_count != row_count trick and the IN-list sorted-domain probe, and (b) the refusal to ever guess: NULL means keep, unhandled predicates mean keep, Inexact statistics mean keep.

Which is exactly why the correctness of the whole thing rests on the statistics provider honoring a contract the code chooses not to check.

HashJoin vs SortMergeJoin vs NestedLoopJoin: What DataFusion Actually Uses Statistics For

Jay Hubbert — Wed, 21 Oct 2026 04:21:00 +0000

TL;DR: There is no cardinality-driven algorithm choice in DataFusion’s planner. HashJoin, SortMergeJoin, NestedLoopJoin and CrossJoin are selected by the shape of the join condition: equi-keys present or not, a range filter or not, prefer_hash_join on or off. What the statistics actually decide is (a) which side becomes the hash table, (b) whether that side is collected into one partition or repartitioned, and (c) how many rows the join will emit.

I want to start with a correction, because it is the single most common misconception about DataFusion’s optimizer.

There is no datafusion/optimizer/src/join_reorder.rs in DataFusion. There is no ballistic join enumerator. There is no exhaustive search over join orders. The logical optimizer’s rule list is right there in the source:

 rust
 
 
// datafusion/optimizer/src/optimizer.rs:289-317
Arc::new(RewriteSetComparison::new()),
Arc::new(OptimizeUnions::new()),
Arc::new(UnionsToFilter::new()),
Arc::new(SimplifyExpressions::new()),
Arc::new(ReplaceDistinctWithAggregate::new()),
Arc::new(EliminateJoin::new()),
Arc::new(DecorrelatePredicateSubquery::new()),
Arc::new(ScalarSubqueryToJoin::new()),
Arc::new(DecorrelateLateralJoin::new()),
Arc::new(ExtractEquijoinPredicate::new()),
Arc::new(EliminateDuplicatedExpr::new()),
Arc::new(EliminateFilter::new()),
Arc::new(EliminateCrossJoin::new()),
Arc::new(EliminateLimit::new()),
Arc::new(PropagateEmptyRelation::new()),
Arc::new(FilterNullJoinKeys::default()),
Arc::new(EliminateOuterJoin::new()),
...

No join reorderer. No cost model over join trees. DataFusion will happily execute A ⋈ B ⋈ C ⋈ D in exactly the order you wrote it, forever, no matter how many statistics it has.

What does exist is the physical optimizer rule JoinSelection, and it is much more modest than the name suggests. But before we get to it, we have to talk about how the algorithm is chosen in the first place — because the answer is surprising.

The algorithm decision is structural, not statistical

Here is the whole decision, from the physical planner:

 rust
 
 
// datafusion/core/src/physical_planner.rs:1630-1652 (excerpt)
let join: Arc<dyn ExecutionPlan> = if join_on.is_empty() {
 if join_filter.is_none() && *join_type == JoinType::Inner {
 // cross join if there is no join conditions and no join filter set
 Arc::new(CrossJoinExec::new(physical_left, physical_right))
 } else if num_range_filters == 1
 && total_filters == 1
 && session_state
 .config_options()
 .optimizer
 .enable_piecewise_merge_join
 {
 ... PiecewiseMergeJoinExec ...
 } else {
 // there is no equal join condition, use the nested loop join
 Arc::new(NestedLoopJoinExec::try_new(
 physical_left, physical_right, join_filter, join_type, None,
 )?)
 }
} else if session_state.config().target_partitions() > 1
 && session_state.config().repartition_joins()
 && !prefer_hash_join
 && !*null_aware
{
 // Use SortMergeJoin if hash join is not preferred
 let join_on_len = join_on.len();
 Arc::new(SortMergeJoinExec::try_new(
 physical_left, physical_right, join_on, join_filter,
 *join_type, vec![SortOptions::default(); join_on_len], *null_equality,
 )?)
} else {
 let partition_mode = if can_repartition_join {
 PartitionMode::Auto
 } else {
 PartitionMode::CollectLeft
 };
 Arc::new(
 HashJoinExecBuilder::new(physical_left, physical_right, join_on, *join_type)
 .with_filter(join_filter)
 .with_partition_mode(partition_mode)
 .with_null_equality(*null_equality)
 .with_null_aware(*null_aware)
 .with_null_aware_value_keys(*null_aware_value_keys)
 .build()?,
 )
};

Read it as a decision tree:

text

 join_on.is_empty()?
 / \
 yes no
 | |
 join_filter.is_none() repartition_joins && target_partitions>1
 && join_type == Inner? && !prefer_hash_join && !null_aware?
 / \ / \
 yes no yes no
 | | | |
 CrossJoin range filter? SortMergeJoin HashJoin
 exactly 1? (sorts both sides)
 / \
 yes no
 | |
 PWMJ (opt-in) NestedLoopJoin

No cardinality appears anywhere in that tree. The inputs are:

join_on — the extracted equi-join keys.
join_filter — the residual predicate.
num_range_filters / total_filters — how many of the residual conjuncts are range comparisons.
prefer_hash_join — datafusion.optimizer.prefer_hash_join, default true.
repartition_joins — datafusion.optimizer.repartition_joins, default true.
target_partitions > 1.

So: equi-keys present → hash join (by default). No equi-keys → cross join if there’s also no residual predicate, otherwise nested loop. Sort-merge join is only reachable by explicitly setting prefer_hash_join = false.

That’s a defensible design. Hash join is the right default for OLAP: it’s O(n+m) with a single pass and no sorting, whereas sort-merge join pays a full sort of both inputs. But it means “which join algorithm” is not a cost-based question in DataFusion. The cost-based questions are downstream of it.

Why the range filter branch is so specific

The PiecewiseMergeJoin branch requires exactly one residual filter and that filter being a range comparison:

 rust
 
 
// datafusion/core/src/physical_planner.rs:1497-1531 (excerpt)
let mut num_range_filters = 0;
let mut range_filters: Vec<Expr> = Vec::new();
let mut total_filters = 0;

let join_filter = match filter {
 Some(expr) => {
 let split_expr = split_conjunction(expr);
 for expr in split_expr.iter() {
 match *expr {
 Expr::BinaryExpr(BinaryExpr { left: _, right: _, op }) => {
 if matches!(op, Operator::Lt | Operator::LtEq | Operator::Gt | Operator::GtEq) {
 range_filters.push((**expr).clone());
 num_range_filters += 1;
 }
 total_filters += 1;
 }
 // TODO: Want to deal with `Expr::Between` for IEJoins, it counts as two range predicates
 // which is why it is not dealt with in PWMJ
 // Expr::Between(_) => {},
 _ => { total_filters += 1; }
 }
 }
 ...

The TODO about Between is a nice detail: a BETWEEN 1 AND 10 is two range predicates, so it is not eligible for PWMJ. (And of course, by the time it reaches the physical planner, the expression simplifier has already desugared Between into >= and <= — so the Expr::Between arm is mostly dead code, which is exactly the kind of leftover you find in a codebase that has been refactoring this area for years.)

PWMJ also has to pin one operand to each side:

 rust
 
 
// datafusion/core/src/physical_planner.rs:1668-1720 (excerpt)
// `Neither` covers an operand that references no column from
// either side (e.g. a literal), and `Both` an operand that
// references columns from both. PWMJ needs one operand pinned
// to each side, so both fall back to NestedLoopJoin below rather
// than erroring or (for `Neither`) panicking.
enum Side { Left, Right, Both, Neither }

let side_of = |e: &Expr| -> Side {
 let cols = e.column_refs();
 let any_left = cols.iter().any(|c| left_df_schema.index_of_column(c).is_ok());
 let any_right = cols.iter().any(|c| right_df_schema.index_of_column(c).is_ok());
 match (any_left, any_right) {
 (true, false) => Side::Left,
 (false, true) => Side::Right,
 (true, true) => Side::Both,
 (false, false) => Side::Neither,
 }
};
...
} else if !(left_side == Side::Left && right_side == Side::Right) {
 // Anything other than a clean left/right split -- both
 // operands on one side, one referencing neither side, or
 // either referencing both -- isn't a range predicate PWMJ
 // can plan, so let NestedLoopJoin evaluate it instead.
 return Ok(Arc::new(NestedLoopJoinExec::try_new(...)?));
}

This is the shape of a mature planner: it recognizes “I could use a cleverer algorithm here” and then carefully declines to, falling back to the general one. PWMJ is off by default anyway (enable_piecewise_merge_join = false).

Null-aware joins are pinned

There’s a nice defensive check here. NOT IN over a nullable subquery has null-aware anti-join semantics, and only HashJoinExec implements them:

 rust
 
 
// datafusion/core/src/physical_planner.rs:1611-1627
// Null-aware joins are pinned to CollectLeft hash joins (see
// `HashJoinExec::null_aware`): never repartition them, and
// never route them to the sort-merge path below.
let can_repartition_join = session_state.config().target_partitions() > 1
 && session_state.config().repartition_joins()
 && !*null_aware;

// Only `HashJoinExec` implements null-aware semantics, and it
// needs equi-join keys to do so. Without them the join would be
// planned as a nested loop (or piecewise merge) join, which
// silently ignores the flag and returns wrong results for
// `NOT IN` over a nullable subquery. Fail loudly instead.
if *null_aware && join_on.is_empty() {
 return plan_err!(
 "null_aware {join_type} join requires equi-join keys, but the join has none"
 );
}

“Fail loudly instead” is the right call: the alternative is a silently wrong answer.

Where statistics do matter: `JoinSelection`

Now we get to the physical optimizer rule that people usually mean when they say “DataFusion’s join optimizer”. It runs here:

 rust
 
 
// datafusion/physical-optimizer/src/optimizer.rs:213
Arc::new(JoinSelection::new()),

and its doc comment is admirably precise about its scope:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:70-83
/// Checks whether join inputs should be swapped using available statistics.
///
/// It follows these steps:
/// 1. If a [`datafusion_physical_plan::operator_statistics::StatisticsRegistry`] is
/// provided, use it for cross-operator estimates
/// (e.g., intermediate join outputs that would otherwise have `Absent` statistics).
/// 2. Compare the in-memory sizes of both sides, and place the smaller side on the
/// left (build) side.
/// 3. If in-memory byte sizes are unavailable, fall back to row counts.
/// 4. Do not reorder the join if neither statistic is available, or if
/// `datafusion.optimizer.join_reordering` is disabled.
///
/// Used configurations
/// - `optimizer.join_reordering`: allows or forbids statistics-driven join swapping
pub(crate) fn should_swap_join_order(
 left: &Arc<dyn ExecutionPlan>,
 right: &Arc<dyn ExecutionPlan>,
 context: &dyn PhysicalOptimizerContext,
) -> Result<bool> {
 if !context.config_options().optimizer.join_reordering {
 return Ok(false);
 }
 let left_stats = get_stats(left, context)?;
 let right_stats = get_stats(right, context)?;
 // First compare total_byte_size, then fall back to num_rows if byte
 // sizes are unavailable.
 match (
 left_stats.total_byte_size.get_value(),
 right_stats.total_byte_size.get_value(),
 ) {
 (Some(l), Some(r)) => Ok(l > r),
 _ => match (
 left_stats.num_rows.get_value(),
 right_stats.num_rows.get_value(),
 ) {
 (Some(l), Some(r)) => Ok(l > r),
 _ => Ok(false),
 },
 }
}

That’s the whole “reordering” story: compare the two inputs of one join and put the smaller one on the left. Byte size first, row count as fallback, and if neither is known, do nothing. Note the semantics: Ok(l > r) means “the left is bigger, so swap”.

This is not join-order reordering across a join tree. It is build-side selection for a single join. The config name (join_reordering) oversells it.

`CollectLeft` vs `Partitioned`: the real cost decision

This is where the statistics earn their keep. HashJoinExec has three partition modes:

 rust
 
 
// datafusion/physical-plan/src/joins/mod.rs:119-128
pub enum PartitionMode {
 /// Left/right children are partitioned using the left and right keys
 Partitioned,
 /// Left side will collected into one partition
 CollectLeft,
 /// DataFusion optimizer decides which PartitionMode
 /// mode(Partitioned/CollectLeft) is optimal based on statistics. It will
 /// also consider swapping the left and right inputs for the Join
 Auto,
}

Auto means “the optimizer decides”, and the decision is:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:192-247
pub(crate) fn try_collect_left(
 hash_join: &HashJoinExec,
 ignore_threshold: bool,
 context: &dyn PhysicalOptimizerContext,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
 let left = hash_join.left();
 let right = hash_join.right();
 let optimizer_config = &context.config_options().optimizer;

 let left_can_collect = ignore_threshold
 || supports_collect_by_thresholds(
 left,
 optimizer_config.hash_join_single_partition_threshold,
 optimizer_config.hash_join_single_partition_threshold_rows,
 context,
 );
 let right_can_collect = ignore_threshold
 || supports_collect_by_thresholds(
 right, ... , context,
 );

 match (left_can_collect, right_can_collect) {
 (true, true) => {
 // For null-aware joins, we only swap `LeftAnti` joins where the left side is > right side
 if can_swap_hash_join(hash_join)
 && should_swap_join_order(left, right, context)?
 {
 Ok(Some(hash_join.swap_inputs(PartitionMode::CollectLeft)?))
 } else {
 Ok(Some(Arc::new(
 hash_join.builder()
 .with_partition_mode(PartitionMode::CollectLeft)
 .build()?,
 )))
 }
 }
 (true, false) => Ok(Some(Arc::new(
 hash_join.builder()
 .with_partition_mode(PartitionMode::CollectLeft)
 .build()?,
 ))),
 (false, true) => {
 if optimizer_config.join_reordering && can_swap_hash_join(hash_join) {
 hash_join.swap_inputs(PartitionMode::CollectLeft).map(Some)
 } else {
 Ok(None)
 }
 }
 (false, false) => Ok(None),
 }
}

And the threshold test is a two-tier one:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:113-134
fn supports_collect_by_thresholds(
 plan: &Arc<dyn ExecutionPlan>,
 threshold_byte_size: usize,
 threshold_num_rows: usize,
 context: &dyn PhysicalOptimizerContext,
) -> bool {
 let Ok(stats) = get_stats(plan, context) else { return false; };

 // Stats use `Precision<T>` to represent stats, where `Absent` means unknown.
 // `Exact(0)` and `Inexact(0)` are both valid stats, and we should not treat
 // them as unknown, `Absent` will return None (this is in regards to why
 // `!=0` is not checked)
 if let Some(byte_size) = stats.total_byte_size.get_value() {
 *byte_size < threshold_byte_size
 } else if let Some(num_rows) = stats.num_rows.get_value() {
 *num_rows < threshold_num_rows
 } else {
 false
 }
}

The comment about Exact(0) is a nice bit of care: an empty side is a valid input to CollectLeft — you get a hash table with zero entries — so zero must not be confused with “unknown”.

The thresholds default to 4 MiB / 131,072 rows:

 rust
 
 
// datafusion/common/src/config.rs:1847-1853
/// The maximum estimated size in bytes for one input side of a HashJoin
/// will be collected into a single partition
pub hash_join_single_partition_threshold: usize, default = 4 * 1024 * 1024

/// The maximum estimated size in rows for one input side of a HashJoin
/// will be collected into a single partition
pub hash_join_single_partition_threshold_rows: usize, default = 1024 * 128

Why the threshold exists at all. CollectLeft gathers the entire build side into one partition and builds one hash table there. That is faster than Partitioned (no repartition shuffle, no per-partition hash tables) but it requires the build side to fit in memory and it serializes the build phase. Partitioned repartitions both sides by the join key, so each partition’s build side is a fraction of the total — more memory-efficient, but it pays for a shuffle.

So the rule is: if the build side is small, collect it; if it is large, repartition. That’s a genuine cost model, and it’s the closest thing to a “cost-based join selection” that DataFusion has.

When neither side qualifies, the fallback is:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:257-283
pub(crate) fn partitioned_hash_join(
 hash_join: &HashJoinExec,
 context: &dyn PhysicalOptimizerContext,
) -> Result<Arc<dyn ExecutionPlan>> {
 let left = hash_join.left();
 let right = hash_join.right();
 let partition_mode = if hash_join.null_aware {
 PartitionMode::CollectLeft
 } else {
 PartitionMode::Partitioned
 };
 if can_swap_hash_join(hash_join) && should_swap_join_order(left, right, context)? {
 hash_join.swap_inputs(partition_mode)
 } else {
 Ok(Arc::new(
 hash_join.builder()
 .with_partition_mode(partition_mode)
 .build()?,
 ))
 }
}

Even in the repartitioned case, the smaller input still becomes the build side — because even with partitioning, a smaller build side means smaller hash tables in every partition.

The guards that stop statistics from breaking things

There are three guards in statistical_join_selection_subrule, and each one fixes a real bug:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:293-334
fn statistical_join_selection_subrule(
 plan: Arc<dyn ExecutionPlan>,
 context: &dyn PhysicalOptimizerContext,
) -> Result<Transformed<Arc<dyn ExecutionPlan>>> {
 let transformed = if let Some(hash_join) = plan.downcast_ref::<HashJoinExec>() {
 if !hash_join.dynamic_expressions_produced().is_empty() {
 // Once a HashJoinExec carries a dynamic filter, its build side
 // has already been determined and the dynamic filter has been wired
 // to the probe side. Reordering inputs would invalidate the dynamic
 // filter, so skip this join and leave it unchanged.
 None
 } else {
 match hash_join.partition_mode() { ... }
 }
 } else if let Some(cross_join) = plan.downcast_ref::<CrossJoinExec>() {
 ... cross_join.swap_inputs() ...
 } else if let Some(nl_join) = plan.downcast_ref::<NestedLoopJoinExec>() {
 ... nl_join.swap_inputs() ...
 }

The dynamic-filter guard is the interesting one. A hash join can push a runtime filter (a set of build-side keys, or a min/max range) into the probe-side scan. Once that filter is wired up, the build side is fixed — swapping the inputs would invalidate the filter’s wiring. This is a case where a purely “cost-based” decision would silently break a correctness- adjacent mechanism, so the optimizer refuses to make it.

Note also that the same swapping logic applies to CrossJoinExec and NestedLoopJoinExec — because for those, “left” and “right” are symmetric and putting the smaller input first matters for memory.

Unbounded inputs: a separate, non-statistical rule

The rule also has a “pipeline fixer” phase that runs before the statistical one:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:524-541
pub fn hash_join_swap_subrule(
 mut input: Arc<dyn ExecutionPlan>,
 _config_options: &ConfigOptions,
) -> Result<Arc<dyn ExecutionPlan>> {
 if let Some(hash_join) = input.downcast_ref::<HashJoinExec>()
 && hash_join.dynamic_expressions_produced().is_empty()
 && hash_join.left.boundedness().is_unbounded()
 && !hash_join.right.boundedness().is_unbounded()
 && !hash_join.null_aware // Don't swap null-aware anti joins
 && matches!(
 *hash_join.join_type(),
 JoinType::Inner | JoinType::Left | JoinType::LeftSemi | JoinType::LeftAnti
 )
 {
 input = swap_join_according_to_unboundedness(hash_join)?;
 }
 Ok(input)
}

If one side is an infinite stream and the other is finite, the finite side must be the build side — otherwise the hash table can never be finished. This is not a heuristic; it’s a runnability requirement. The rule refuses to swap Full, Right, RightSemi and RightAnti joins because their semantics don’t survive the swap:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:550-563
pub(crate) fn swap_join_according_to_unboundedness(
 hash_join: &HashJoinExec,
) -> Result<Arc<dyn ExecutionPlan>> {
 ...
 match (*partition_mode, *join_type) {
 (
 _,
 JoinType::Right | JoinType::RightSemi | JoinType::RightAnti
 | JoinType::RightMark | JoinType::Full,
 ) => internal_err!("{join_type} join cannot be swapped for unbounded input."),
 ...

And when both sides are unbounded, a HashJoinExec becomes a SymmetricHashJoinExec, which is a genuinely different algorithm designed for streaming:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:389-405
if let Some(hash_join) = input.downcast_ref::<HashJoinExec>() {
 let left_unbounded = hash_join.left.boundedness().is_unbounded();
 let left_incremental = matches!(
 hash_join.left.pipeline_behavior(),
 EmissionType::Incremental | EmissionType::Both
 );
 let right_unbounded = hash_join.right.boundedness().is_unbounded();
 let right_incremental = matches!(
 hash_join.right.pipeline_behavior(),
 EmissionType::Incremental | EmissionType::Both
 );
 // Process only if both left and right sides are unbounded and incrementally emit.
 if left_unbounded && right_unbounded & left_incremental & right_incremental {

Note the & instead of && — a bitwise AND on two bools. It works (both operands are bool, no short-circuit) but it’s the kind of thing a reviewer should flag. Whether it’s intentional (forcing both sides to be evaluated) or a typo, the semantics happen to be right.

Estimating the join’s output cardinality

The other half of the answer: how big will this join be? That number feeds the aggregate above it, the limit above that, and the choice of build side for the next join.

The estimator is the classic NDV formula:

 rust
 
 
// datafusion/physical-plan/src/operator_statistics/mod.rs:936-968
/// Estimate equi-join output using NDV of join key columns:
/// left_rows * right_rows / product(max(left_ndv_i, right_ndv_i))
/// Falls back to Cartesian product if any key lacks NDV on both sides.
fn equi_join_estimate(
 on: JoinOnRef,
 left: &Statistics,
 right: &Statistics,
 left_rows: usize,
 right_rows: usize,
) -> usize {
 if on.is_empty() {
 return left_rows.saturating_mul(right_rows);
 }
 let mut ndv_divisor: usize = 1;
 for (left_key, right_key) in on {
 let left_ndv = left_key
 .downcast_ref::<Column>()
 .and_then(|c| left.column_statistics.get(c.index()))
 .and_then(|s| s.distinct_count.get_value().copied());
 let right_ndv = right_key
 .downcast_ref::<Column>()
 .and_then(|c| right.column_statistics.get(c.index()))
 .and_then(|s| s.distinct_count.get_value().copied());
 match (left_ndv, right_ndv) {
 (Some(l), Some(r)) if l > 0 && r > 0 => {
 ndv_divisor = ndv_divisor.saturating_mul(l.max(r));
 }
 _ => return left_rows.saturating_mul(right_rows),
 }
 }
 let max_rows = left_rows.saturating_mul(right_rows);
 max_rows.checked_div(ndv_divisor).unwrap_or(max_rows)
}

left_rows × right_rows ÷ Π max(left_ndv_i, right_ndv_i), with the Cartesian product as the fallback whenever any key lacks a usable NDV on both sides.

Then the join type adjusts the base estimate:

 rust
 
 
// datafusion/physical-plan/src/operator_statistics/mod.rs:991-1019
// Apply join-type-aware cardinality bounds
let estimated = match join_type {
 JoinType::Inner => inner_estimate,
 JoinType::Left => inner_estimate.max(left_rows),
 JoinType::Right => inner_estimate.max(right_rows),
 JoinType::Full => {
 // At least left + right - matched, but never less than inner
 let outer_bound = left_rows
 .saturating_add(right_rows)
 .saturating_sub(inner_estimate);
 inner_estimate.max(outer_bound)
 }
 JoinType::LeftSemi => inner_estimate.min(left_rows),
 JoinType::RightSemi => inner_estimate.min(right_rows),
 JoinType::LeftAnti => left_rows.saturating_sub(inner_estimate.min(left_rows)),
 JoinType::RightAnti => {
 right_rows.saturating_sub(inner_estimate.min(right_rows))
 }
 JoinType::LeftMark => left_rows,
 JoinType::RightMark => right_rows,
};

// NL join inner with exact inputs is an exact Cartesian product;
// NDV-based estimates are inherently inexact.
let num_rows = if is_exact_cartesian && join_type == JoinType::Inner {
 Precision::Exact(estimated)
} else {
 Precision::Inexact(estimated)
};

Note the Precision::Exact only for the cross-join-with-exact-inputs case. Everything else is Inexact. That’s the honest labeling: an NDV-based join estimate is never exact, no matter how good the NDVs are.

The same file has the Calcite-derived NDV formulas that feed this:

 rust
 
 
// datafusion/physical-plan/src/operator_statistics/mod.rs:510-577 (doc excerpts)
/// Estimate the number of distinct values when sampling from a population.
///
/// Given a domain with `domain_size` distinct values and `num_selected` rows
/// sampled/filtered from it, estimates how many distinct values will appear
/// in the sample.
///
/// Uses the formula: `Expected distinct = N * [1 - (1 - 1/N)^n]`
///
/// Based on Calcite's `RelMdUtil.numDistinctVals()`:
pub fn num_distinct_vals(domain_size: usize, num_selected: usize) -> usize { ... }

/// Estimate NDV after applying a selectivity factor (filtering).
///
/// When filtering rows, each distinct value has multiple rows. If a value
/// appears `k` times, the probability it survives the filter is `1 - (1-s)^k`
/// where `s` is the selectivity.
///
/// Assuming uniform distribution (each value appears `rows/ndv` times):
/// ```text
/// NDV_after ~ NDV_before * [1 - (1 - selectivity)^(rows/NDV)]
/// ```
pub fn ndv_after_selectivity(...) { ... }

The extension point: `StatisticsRegistry`

Since core DataFusion can’t know your catalog’s statistics, there’s a pluggable provider chain:

 rust
 
 
// datafusion/physical-plan/src/operator_statistics/mod.rs:394-399
/// Registry that chains [`StatisticsProvider`] implementations.
///
/// The registry is a stateless provider chain: it holds no mutable state
/// and is cheaply `Clone`able / `Send` / `Sync`.
pub struct StatisticsRegistry { ... }

and the shipped example datafusion-examples/examples/statistics/join_reorder.rs demonstrates the whole point — a catalog that knows user_id has 100 distinct values and amount lives in [0, 1000]:

 rust
 
 
// datafusion-examples/examples/statistics/join_reorder.rs:60-74
fn catalog_stats(
 plan: &dyn ExecutionPlan,
 _child_stats: &[ExtendedStatistics],
) -> Result<StatisticsResult> {
 let schema = plan.schema();
 let user_id = schema.index_of("user_id")?;
 let amount = schema.index_of("amount")?;
 let mut stats = (*plan.statistics_from_inputs(&[], &StatisticsArgs::new())?).clone();
 stats.column_statistics[amount].min_value =
 Precision::Inexact(ScalarValue::Int32(Some(0)));
 stats.column_statistics[amount].max_value =
 Precision::Inexact(ScalarValue::Int32(Some(999)));
 stats.column_statistics[user_id].distinct_count = Precision::Inexact(100);
 Ok(StatisticsResult::Computed(ExtendedStatistics::new(stats)))
}

The example’s own description is the cleanest summary of what statistics buy you:

the in-memory events table has no column statistics, so the grouped side is estimated at 1000 rows and dims (60 rows) becomes the build side. A provider supplies the amount range and the user_id distinct count a catalog knows; with them the operators estimate the grouped side at 50 rows, below dims, which flips the join build side.

Trade-offs and failure modes

A wrong NDV picks the wrong build side, and it is expensive. The build side is hashed into memory. If you build on the 10-million-row side because you thought it was the 10-thousand-row side, you OOM or thrash. The Precision::Inexact labeling protects you from correctness issues but not from this.

CollectLeft is a memory cliff. The threshold is 4 MiB by estimated size. If the estimate is 10× low, CollectLeft gathers 40 MiB into one partition. And CollectLeft is a pipeline break: nothing downstream of it can start until the whole build side is collected.

Partitioned is a shuffle cliff. Repartitioning both sides of a join by the join key is expensive — a full exchange over the network in a distributed deployment. prefer_hash_join = false (to force sort-merge) trades the hash join’s memory for two sorts, which is usually worse and sometimes better.

No join reordering means a bad plan stays bad. If your SQL has the tables in the wrong order for the data, no amount of statistics will fix it. You have to write the query in the right order, or use a system with a real join optimizer. This is the single biggest limitation in this area.

OR in a join predicate kills the hash join. If the join condition is a.x = b.x OR a.y = b.y, ExtractEquijoinPredicate can’t extract equi-keys, and you fall to NestedLoopJoinExec — O(n×m). DataFusion does not decompose disjunctive equi-joins into a union of hash joins.

NOT BETWEEN and other non-equi predicates go to nested loop. Same story.

Swapping changes semantics for asymmetric join types. can_swap_hash_join is careful:

 rust
 
 
// datafusion/physical-optimizer/src/join_selection.rs:171-179
fn can_swap_hash_join(hash_join: &HashJoinExec) -> bool {
 hash_join.join_type().supports_swap()
 && (!hash_join.null_aware
 || (*hash_join.join_type() == JoinType::LeftAnti
 && hash_join.on().len() == 1
 && hash_join.filter().is_none()))
}

A null-aware LeftAnti join can only be swapped if it has exactly one join key and no residual filter — because the swapped RightAnti has no per-row NULL handling. This is the kind of detail that costs a day to discover.

Configuration

Option	Default	Effect
`datafusion.optimizer.prefer_hash_join`	`true`	`false` routes equi-joins to `SortMergeJoinExec` instead of `HashJoinExec`.
`datafusion.optimizer.join_reordering`	`true`	Allow statistics-driven swapping of a join’s inputs. Does not reorder joins in a join tree.
`datafusion.optimizer.hash_join_single_partition_threshold`	`4 MiB`	Byte threshold under which a side qualifies for `CollectLeft`.
`datafusion.optimizer.hash_join_single_partition_threshold_rows`	`131072`	Row threshold under which a side qualifies for `CollectLeft`.
`datafusion.optimizer.enable_piecewise_merge_join`	`false`	Allow `PiecewiseMergeJoinExec` for a single range filter.
`datafusion.optimizer.repartition_joins`	`true`	Whether joins may be repartitioned at all.
`datafusion.optimizer.top_down_join_key_reordering`	`true`	Reorders the keys within a join, not the joins.
`datafusion.execution.collect_statistics`	`true`	Whether statistics exist at all. Off means every estimate is `Absent` and no swap ever happens.

Bottom line

The honest description of DataFusion’s join planning is: the algorithm is chosen structurally, and statistics tune the parameters.

Structural choice: equi-keys → hash join. No equi-keys → cross or nested loop. Sort-merge only if you ask for it.

Statistical tuning: which side builds the hash table, whether to collect or repartition, and how many rows come out.

That is a real, working cost model — the CollectLeft threshold in particular is a good piece of engineering — but it is not a join-order optimizer, and if you are comparing DataFusion to a system that enumerates join trees, that is the gap you will feel first. The workaround is the classic one: write the joins in the order that makes the small side the build side, and use the StatisticsRegistry to make sure the optimizer sees enough to agree with you.

Inside DataFusion's ListingTable: Listing an Object Store in Parallel and Turning Files into FileGroups

Jay Hubbert — Wed, 21 Oct 2026 03:25:00 +0000

TL;DR: Listing a table is a four-stage pipeline: (1) turn the filter into a path prefix so the object store can skip whole directories, (2) list the prefix and decode Hive-style partition values out of each path, (3) fetch per-file metadata (statistics, ordering) concurrently with meta_fetch_concurrency, and (4) pack the surviving files into FileGroups — one per target_partitions. A LIMIT can cut the pipeline short after stage 3, which is the trick that makes SELECT * FROM t LIMIT 1 cheap on a million-file table.

Everything above the physical plan is a lie of omission. When you write SELECT * FROM 's3://bucket/events/' WHERE dt = '2024-01-01', the planner needs to know which files that resolves to before it can build a plan. That question is answered by the listing layer, and it is where a surprising amount of DataFusion’s real-world performance is won or lost.

The relevant code lives in three crates: datafusion/catalog/ (the TableProvider abstraction), datafusion/catalog-listing/ (ListingTable, the Hive-partitioned file-backed table), and datafusion/datasource/ (PartitionedFile, FileGroup, FileScanConfig).

The contract: `TableProvider`

The catalog is a trait, not a class hierarchy. Everything a table must be able to do is expressed in TableProvider:

 rust
 
 
// datafusion/session/src/table.rs:55-57, 193-199
pub trait TableProvider: Any + Debug + Sync + Send {
 /// Get a reference to the schema for this table
 fn schema(&self) -> SchemaRef;
 ...
 /// Create an [`ExecutionPlan`] for scanning the table with optional
 /// `projection`, `filter`, and `limit`, described below.
 async fn scan(
 &self,
 state: &dyn Session,
 projection: Option<&[usize]>,
 filters: &[Expr],
 limit: Option<usize>,
 ) -> Result<Arc<dyn ExecutionPlan>>;
}

The scan method’s contract is unusually well specified — the doc comment spells out the evaluation order (filters, then limit, then projection), warns that filters may reference columns absent from the projection, and notes that “if any pushed-down filters are Inexact, the LIMIT cannot be pushed down” because inexact filters don’t guarantee that every filtered row was removed. ListingTable overrides the richer scan_with_args variant so it also gets the skip (OFFSET) argument.

Stage 0: split the filter

Before listing anything, ListingTable sorts the incoming filters into two buckets:

 rust
 
 
// datafusion/catalog-listing/src/table.rs:646-651
// If the filters can be resolved using only partition cols, there is no need to
// pushdown it to TableScan, otherwise, `unhandled` pruning predicates will be generated
let (partition_filters, filters): (Vec<_>, Vec<_>) =
 filters.iter().cloned().partition(|filter| {
 can_be_evaluated_for_partition_pruning(&table_partition_col_names, filter)
 });

Partition-only filters go to the listing stage (they can prune whole directories and whole files). Everything else stays as a data filter for the scan, handled by the pruning machinery.

There’s also a subtlety about LIMIT:

 rust
 
 
// datafusion/catalog-listing/src/table.rs:629-662
// The scan must read enough rows to satisfy `skip + limit`, not
// just `limit`, before any rows are skipped below.
let inflated_limit = limit.map(|l| l.saturating_add(skip.unwrap_or(0)));
...
// We should not limit files before assigning declared output partitions
// or before applying non-partition filters.
let statistic_file_limit =
 if filters.is_empty() && declared_output_partitioning.is_none() {
 inflated_limit
 } else {
 None
 };

The file limit is only pushed into the listing when there are no data filters to apply — a file that fails a data filter still has to be counted against the limit’s row budget. Get this wrong and LIMIT 10 returns fewer than 10 rows.

Stage 1: turn the filter into a path prefix

This is the highest-leverage trick in the whole listing layer. Hive-partitioned tables encode partition values as directory names:

text

s3://bucket/events/dt=2024-01-01/region=us-east-1/part-0.parquet
s3://bucket/events/dt=2024-01-02/region=us-west-2/part-0.parquet

A filter dt = '2024-01-01' can be turned into the prefix dt=2024-01-01, and then the object store’s LIST operation never returns the other day’s files at all. No request, no parsing, no filtering.

 rust
 
 
// datafusion/catalog-listing/src/helpers.rs:262-305
pub fn evaluate_partition_prefix<'a>(
 partition_cols: &'a [(String, DataType)],
 filters: &'a [Expr],
) -> Option<Path> {
 let mut partition_values = HashMap::new();
 for filter in filters {
 populate_partition_values(&mut partition_values, filter);
 }
 if partition_values.is_empty() {
 return None;
 }
 let mut parts = Vec::with_capacity(partition_cols.len());
 for (p, _) in partition_cols {
 match partition_values.get(p.as_str()) {
 Some(PartitionValue::Single(val)) => {
 // if a partition only has a single literal value, then it can be added to the
 // prefix
 let encoded = encode_partition_value(val);
 if encoded != val.as_str() {
 // The same decoded value can be represented by both raw and
 // percent-encoded partition directories. Prefix pruning is
 // an optimization, so stop before this partition rather
 // than listing only one spelling and potentially skipping
 // valid rows.
 break;
 }
 parts.push(format!("{p}={encoded}"));
 }
 _ => {
 // break on the first unconstrainted partition to create a common prefix
 // for all covered partitions.
 break;
 }
 }
 }
 if parts.is_empty() { None } else { Some(Path::from_iter(parts)) }
}

Three details worth pausing on:

Only equality constraints count. populate_partition_values only recognises Operator::Eq against a literal, and only AND compositions. A dt > '2024-01-01' filter yields no prefix, because there’s no way to express “all directories after this one” as a LIST prefix.

The prefix is a common prefix, so it stops at the first unconstrained column. If partition_cols is [dt, region] and you filter only on dt, you get prefix dt=2024-01-01 and then stop — because region is unconstrained, all regions under that date must be listed.

The percent-encoding guard is a correctness fix. The same decoded value can be spelled two ways in the store (dt=2024-01-01 and dt=2024-01-01 with a %-encoded variant). If the literal’s encoding doesn’t round-trip, prefixing would list only one spelling and silently drop files. So it bails out. “Prefix pruning is an optimization, so stop before this partition rather than listing only one spelling and potentially skipping valid rows.”

The recursive directory walk

For genuinely hierarchical layouts, there’s a parallel directory walker with an explicit concurrency limit:

 rust
 
 
// datafusion/catalog-listing/src/helpers.rs:142-143
/// The maximum number of concurrent listing requests
const CONCURRENCY_LIMIT: usize = 100;

 rust
 
 
// datafusion/catalog-listing/src/helpers.rs:156-226 (excerpt)
impl Partition {
 /// List the direct children of this partition updating `self.files` with
 /// any child files, and returning a list of child "directories"
 async fn list(mut self, store: &dyn ObjectStore) -> Result<(Self, Vec<Path>)> {
 trace!("Listing partition {}", self.path);
 let prefix = Some(&self.path).filter(|p| !p.as_ref().is_empty());
 let result = store.list_with_delimiter(prefix).await?;
 self.files = Some(
 result.objects.into_iter()
 .filter(|object_meta| object_meta.size > 0)
 .collect(),
 );
 Ok((self, result.common_prefixes))
 }
}

/// Returns a recursive list of the partitions in `table_path` up to `max_depth`
pub async fn list_partitions(
 store: &dyn ObjectStore,
 table_path: &ListingTableUrl,
 max_depth: usize,
 partition_prefix: Option<Path>,
) -> Result<Vec<Partition>> {
 ...
 let mut pending = vec![];
 let mut futures = FuturesUnordered::new();
 futures.push(partition.list(store));

 while let Some((partition, paths)) = futures.next().await.transpose()? {
 // If pending contains a future it implies prior to this iteration
 // `futures.len == CONCURRENCY_LIMIT`. We can therefore add a single
 // future from `pending` to the working set
 if let Some(next) = pending.pop() {
 futures.push(next)
 }
 let depth = partition.depth;
 out.push(partition);
 for path in paths {
 let child = Partition { path, depth: depth + 1, files: None };
 match depth < max_depth {
 true => match futures.len() < CONCURRENCY_LIMIT {
 true => futures.push(child.list(store)),
 false => pending.push(child.list(store)),
 },
 false => out.push(child),
 }
 }
 }
 Ok(out)
}

This is a textbook breadth-first walk with a bounded fan-out: at most 100 in-flight LIST requests, a pending stack for overflow, and FuturesUnordered so that completed requests are reaped immediately rather than in submission order. The list_with_delimiter call is the object store’s “give me the files and the subdirectories of this one directory” operation.

Note the filter(|object_meta| object_meta.size > 0) — zero-byte files are dropped. That is a detail that bites people who write placeholder files into a data directory.

This helper is public API; the default scan path uses the object store’s own recursive LIST instead, but the hierarchical walk is what you’d use for genuinely deep partition trees.

Stage 2: list the prefix and decode partitions

The actual listing goes through ListingTableUrl:

 rust
 
 
// datafusion/datasource/src/url.rs:247-304 (excerpt)
pub async fn list_prefixed_files<'a>(
 &'a self,
 ctx: &'a dyn Session,
 store: &'a dyn ObjectStore,
 prefix: Option<Path>,
 file_extension: &'a str,
) -> Result<BoxStream<'a, Result<ObjectMeta>>> {
 let exec_options = &ctx.config_options().execution;
 let ignore_subdirectory = exec_options.listing_table_ignore_subdirectory;
 ...
 let list: BoxStream<'a, Result<ObjectMeta>> = if self.is_collection() {
 list_with_cache(ctx, store, self.table_ref.as_ref(), &self.prefix, prefix.as_ref()).await?
 } else {
 match store.head(&full_prefix).await {
 Ok(meta) => futures::stream::once(async { Ok(meta) })
 .map_err(|e| DataFusionError::ObjectStore(Box::new(e)))
 .boxed(),
 // If the head command fails, it is likely that object doesn't exist.
 // Retry as though it were a prefix (aka a collection)
 Err(object_store::Error::NotFound { .. }) => {
 list_with_cache(ctx, store, self.table_ref.as_ref(), &self.prefix, prefix.as_ref()).await?
 }
 Err(e) => return Err(e.into()),
 }
 };

 Ok(list
 .try_filter(move |meta| {
 let path = &meta.location;
 let extension_match = path.as_ref().ends_with(file_extension);
 let glob_match = self.contains(path, ignore_subdirectory);
 futures::future::ready(extension_match && glob_match)
 })
 .boxed())
}

Two things to notice. First, the HEAD-then-LIST dance: if the URL points at a single file, HEAD succeeds and the stream is one object; if it 404s, it’s treated as a collection and listed. Second, the listing cache:

 rust
 
 
// datafusion/datasource/src/url.rs:390-423 (excerpt)
match ctx.runtime_env().cache_manager.get_list_files_cache() {
 None => Ok(store.list(Some(&full_prefix))
 .map(|res| res.map_err(|e| DataFusionError::ObjectStore(Box::new(e))))
 .boxed()),
 Some(cache) => {
 ...
 let vec = if let Some(cached) = cache.get(&table_scoped_base_path) {
 debug!("Hit list files cache");
 cached.files_matching_prefix(&filter_prefix)
 } else if let Some(cached) = cache.get(&table_scoped_list_path) {
 debug!("Hit list files cache for requested path");
 cached.files_matching_prefix(&None)
 } else {
 let mut vec = store.list(Some(&full_prefix))
 .try_collect::<Vec<ObjectMeta>>()
 .await?;
 vec.shrink_to_fit(); // Right-size before caching
 let cached: CachedFileList = vec.into();
 let result = cached.files_matching_prefix(&None);
 cache.put(&table_scoped_list_path, cached);
 result
 };

The cache is keyed by {table, path}, so a full-table listing can satisfy a prefix-filtered request by filtering the cached list. The shrink_to_fit() before caching is a nice touch: the vector came from a stream and is over-allocated.

Now the partition values. Each listed object’s path is decoded into partition column values:

 rust
 
 
// datafusion/catalog-listing/src/helpers.rs:466-495 (excerpt)
for (part, expected_partition) in subpath.zip(table_partition_cols) {
 match part.split_once('=') {
 Some((name, val)) if name == expected_partition => {
 // Preserve the original value if percent-decoding produces invalid UTF-8.
 let decoded = percent_decode_str(val)
 .decode_utf8()
 .unwrap_or(Cow::Borrowed(val));
 part_values.push(decoded);
 }
 _ => {
 debug!("Ignoring file: file_path='{file_path}', ...");
 return None;
 }
 }
}

Note the strictness: the path segment’s name= must match the expected partition column, in order. A file that doesn’t fit the declared partition layout is dropped, not guessed at.

Then the partition filters are applied per file, by evaluating the predicate against a one-row batch:

 rust
 
 
// datafusion/catalog-listing/src/helpers.rs:311-330 (excerpt)
pub fn filter_partitioned_file(
 pf: PartitionedFile,
 filters: &[Expr],
 df_schema: &DFSchema,
) -> Result<Option<PartitionedFile>> {
 if pf.partition_values.is_empty() && !filters.is_empty() {
 return Ok(None);
 } else if filters.is_empty() {
 return Ok(Some(pf));
 }
 let arrays = pf.partition_values.iter()
 .map(|v| v.to_array()).collect::<Result<_, _>>()?;
 let batch = RecordBatch::try_new(Arc::clone(df_schema.inner()), arrays)?;
 let filter = utils::conjunction(filters.iter().cloned()).unwrap_or_else(|| lit(true));
 ...

This is a nice trick: the partition values are materialised as a one-row Arrow batch and the ordinary physical expression evaluator is reused to test the filter. No special-case interpreter for partition predicates.

Stage 3: fetch metadata concurrently

Now the interesting concurrency. Multiple table paths are listed in parallel, then per-file metadata (statistics + ordering) is fetched concurrently:

 rust
 
 
// datafusion/catalog-listing/src/table.rs:936-986
async fn collect_files_for_scan<'a>(
 &'a self,
 ctx: &'a dyn Session,
 store: &'a Arc<dyn ObjectStore>,
 listing_time_filters: &'a [Expr],
 file_limit: Option<usize>,
) -> datafusion_common::Result<(FileGroup, bool)> {
 // list files (with partitions)
 let file_list = future::try_join_all(self.table_paths.iter().map(|table_path| {
 pruned_partition_list(
 ctx, store.as_ref(), table_path, listing_time_filters,
 &self.options.file_extension, &self.options.table_partition_cols,
 )
 }))
 .await?;
 let meta_fetch_concurrency =
 ctx.config_options().execution.meta_fetch_concurrency.get();
 // Table paths can overlap, for example when one path is a directory and
 // another names a file inside it. A ListingTable uses one object store,
 // so the object path uniquely identifies a file within this scan.
 let mut seen_files = HashSet::new();
 let file_list = stream::iter(file_list)
 .flatten_unordered(meta_fetch_concurrency)
 .try_filter(move |file| {
 future::ready(seen_files.insert(file.object_meta.location.clone()))
 });
 // collect the statistics and ordering if required by the config
 let files = file_list
 .map(|part_file| async {
 let part_file = part_file?;
 let (statistics, ordering) = if ctx.config().collect_statistics() {
 self.do_collect_statistics_and_ordering(ctx, store, &part_file).await?
 } else {
 (Arc::new(Statistics::new_unknown(&self.file_schema)), None)
 };
 Ok(part_file.with_statistics(statistics).with_ordering(ordering))
 })
 .boxed()
 .buffer_unordered(
 ctx.config_options().execution.meta_fetch_concurrency.get(),
 );

 get_files_with_limit(files, file_limit, ctx.config().collect_statistics()).await
}

Several things to unpack:

try_join_all over table_paths. A ListingTable can point at multiple paths; each is listed concurrently.

flatten_unordered + buffer_unordered with meta_fetch_concurrency. This is the key parallelism knob. The default is 32:

 rust
 
 
// datafusion/common/src/config.rs:1060
pub meta_fetch_concurrency: ConfigNonZeroUsize, default = non_zero_usize_default(32)

flatten_unordered merges the per-path streams while preserving up to N in-flight items; buffer_unordered(N) then bounds the number of concurrent metadata fetches. Both are “unordered” — results come back in completion order, not listing order, which matters because metadata fetch latency is dominated by network round-trips and the natural completion order is skewed.

Deduplication via seen_files. Two table paths can overlap (a directory plus a file inside it), and listing the same file twice would double-count its rows and its statistics. The comment spells out the assumption: “A ListingTable uses one object store, so the object path uniquely identifies a file within this scan.”

Statistics and ordering come from one metadata read. This is an explicit optimization:

 rust
 
 
// datafusion/catalog-listing/src/table.rs:1135-1186 (excerpt)
async fn do_collect_statistics_and_ordering(
 &self,
 ctx: &dyn Session,
 store: &Arc<dyn ObjectStore>,
 part_file: &PartitionedFile,
) -> datafusion_common::Result<(Arc<Statistics>, Option<LexOrdering>)> {
 let path = TableScopedPath {
 table: part_file.table_reference.clone(),
 path: part_file.object_meta.location.clone(),
 };
 let meta = &part_file.object_meta;

 // Check cache first. The key stays `{table, path}` for cheap lookups;
 // the cached value carries the schema fingerprint to prevent reusing
 // stats computed under a different file schema.
 if let Some(cache) = &self.collected_statistics
 && let Some(cached) = cache.get(&path)
 && cached.is_valid_for(meta, &self.file_schema_fingerprint)
 {
 return Ok((Arc::clone(&cached.statistics), cached.ordering.clone()));
 }

 // Cache miss or invalid: fetch both statistics and ordering in a single metadata read
 let file_meta = self.options.format
 .infer_stats_and_ordering(ctx, store, Arc::clone(&self.file_schema), meta)
 .await?;
 ...

For Parquet, “one metadata read” means one footer read that yields both the column statistics and the sorting_columns metadata. Reading the footer twice would double the I/O for every file in the table.

The schema fingerprint in the cache key is a subtle correctness measure: if the file schema changed (schema evolution!), the cached statistics are invalid and must not be reused.

Stage 3.5: the `LIMIT` shortcut

Here’s the trick I mentioned at the top. get_files_with_limit stops pulling files as soon as the accumulated row count exceeds the limit:

 rust
 
 
// datafusion/catalog-listing/src/table.rs:1209-1260 (excerpt)
async fn get_files_with_limit(
 files: impl Stream<Item = datafusion_common::Result<PartitionedFile>>,
 limit: Option<usize>,
 collect_stats: bool,
) -> datafusion_common::Result<(FileGroup, bool)> {
 let mut file_group = FileGroup::default();
 // Fusing the stream allows us to call next safely even once it is finished.
 let mut all_files = Box::pin(files.fuse());
 enum ProcessingState {
 ReadingFiles,
 ReachedLimit,
 }
 let mut state = ProcessingState::ReadingFiles;
 let mut num_rows = Precision::Absent;

 while let Some(file_result) = all_files.next().await {
 // Early exit if we've already reached our limit
 if matches!(state, ProcessingState::ReachedLimit) {
 break;
 }
 let file = file_result?;
 // Update file statistics regardless of state
 if collect_stats && let Some(file_stats) = &file.statistics {
 num_rows = if file_group.is_empty() {
 // For the first file, just take its row count
 file_stats.num_rows
 } else {
 // For subsequent files, accumulate the counts
 num_rows.add(&file_stats.num_rows)
 };
 }
 // Always add the file to our group
 file_group.push(file);
 // Check if we've hit the limit (if one was specified)
 if let Some(limit) = limit
 && let Precision::Exact(row_count) = num_rows
 && row_count > limit
 {
 state = ProcessingState::ReachedLimit;
 }
 }
 // If we still have files in the stream, it means the limit kicked
 // in, and the statistic could have been different had we processed the
 // files in a different order.
 let inexact_stats = all_files.next().await.is_some();
 Ok((file_group, inexact_stats))
}

The Precision::Exact guard is essential: you can only stop early if the row counts are exact. An estimated row count of “100 rows” for a file that actually has 10 million would let you stop after one file and return zero rows.

And inexact_stats is the honesty flag: if the stream still had files when we stopped, then the aggregate statistics for the table are wrong (we didn’t see all the files), so they get downgraded:

 rust
 
 
// datafusion/datasource/src/statistics.rs:408-410
if inexact_stats {
 statistics = statistics.to_inexact()
}

This is exactly the mechanism that makes SELECT * FROM huge_table LIMIT 1 cheap: you list the prefix, you fetch metadata for the first few files, and you stop. The object store’s LIST is still O(prefix) in the worst case, but you avoid the per-file footer reads for the rest of the table.

Stage 4: pack files into `FileGroup`s

Finally the files are distributed across partitions. The FileGroup is a deliberately thin abstraction — a bag of files plus merged statistics:

 rust
 
 
// datafusion/datasource/src/file_groups.rs:379-389
#[derive(Debug, Clone)]
pub struct FileGroup {
 /// The files in this group
 files: Vec<PartitionedFile>,
 /// Optional statistics for the data across all files in the group.
 ///
 /// These statistics cover the full table schema: file columns plus partition columns.
 statistics: Option<Arc<Statistics>>,
}

The split is a chunked round-robin, with a sort first for determinism:

 rust
 
 
// datafusion/datasource/src/file_groups.rs:453-484
/// Partition the list of files into `n` groups
pub fn split_files(mut self, n: usize) -> Vec<FileGroup> {
 if self.is_empty() {
 return vec![];
 }
 // ObjectStore::list does not guarantee any consistent order and for some
 // implementations such as LocalFileSystem, it may be inconsistent. Thus
 // Sort files by path to ensure consistent plans when run more than once.
 self.files.sort_by(|a, b| a.path().cmp(b.path()));

 // effectively this is div with rounding up instead of truncating
 let chunk_size = self.len().div_ceil(n);
 let mut chunks = Vec::with_capacity(n);
 ...

The sort is there for plan reproducibility: object stores don’t promise a stable listing order, so without sorting, the same query would produce different file→partition assignments on different runs, which makes plans non-deterministic and makes performance regressions impossible to bisect.

There’s a second, cleverer grouping mode:

 rust
 
 
// datafusion/datasource/src/file_groups.rs:486-543 (excerpt)
pub fn group_by_partition_values(
 self,
 max_target_partitions: usize,
) -> Vec<FileGroup> {
 ...
 let mut partition_groups: HashMap<Vec<ScalarValue>, Vec<PartitionedFile>> = HashMap::new();
 for file in self.files {
 partition_groups.entry(file.partition_values.clone()).or_default().push(file);
 }
 let num_unique_partitions = partition_groups.len();
 // Sort for deterministic bucket assignment across query executions.
 let mut sorted_partitions: Vec<_> = partition_groups.into_iter().collect();
 ...
 sorted_partitions.sort_by(|a, b| {
 compare_rows(&a.0, &b.0, &sort_options).unwrap_or(Ordering::Equal)
 });
 if num_unique_partitions <= max_target_partitions {
 sorted_partitions.into_iter().map(|(_, files)| FileGroup::new(files)).collect()
 } else {
 // Merge into max_target_partitions buckets using round-robin.
 let mut target_groups = vec![vec![]; max_target_partitions];
 for (idx, (_, files)) in sorted_partitions.into_iter().enumerate() {
 let bucket = idx % max_target_partitions;
 target_groups[bucket].extend(files);
 }
 target_groups.into_iter().map(FileGroup::new).collect()
 }
}

The idea: if all files with the same dt=2024-01-01 land in the same file group, then the scan can declare an output partitioning on dt, and the optimizer can skip repartitioning for an aggregate or join on dt. That’s a real win, and it’s why there’s a threshold:

 rust
 
 
// datafusion/catalog-listing/src/table.rs:1014-1036
// Threshold: 0 = disabled, N > 0 = enabled when distinct_keys >= N
//
// When enabled, files are grouped by their Hive partition column values, allowing
// FileScanConfig to declare output partitioning. This enables the optimizer to
// skip repartitioning for aggregates and joins on partition columns.
let threshold = ctx.config_options().optimizer.preserve_file_partitions;

let (file_groups, grouped_by_partition) =
 if threshold > 0 && !self.options.table_partition_cols.is_empty() {
 let grouped = file_group.group_by_partition_values(file_group_count);
 if grouped.len() >= threshold {
 (grouped, true)
 } else {
 let all_files: Vec<_> = grouped.into_iter().flat_map(|g| g.into_inner()).collect();
 (FileGroup::new(all_files).split_files(file_group_count), false)
 }
 } else {
 (file_group.split_files(file_group_count), false)
 };

datafusion.optimizer.preserve_file_partitions defaults to 0 (disabled). Set it to 1 to always group by partition values, or to N to only do it when there are at least N distinct partition values.

Stage 5: build the scan

Finally the whole thing becomes a FileScanConfig:

 rust
 
 
// datafusion/catalog-listing/src/table.rs:769-786
let file_source = self.create_file_source();
let scan_config = FileScanConfigBuilder::new(object_store_url, file_source)
 .with_file_groups(partitioned_file_lists)
 .with_constraints(self.constraints.clone())
 .with_statistics(statistics)
 .with_projection_indices(projection)?
 .with_limit(inflated_limit)
 .with_output_ordering(output_ordering)
 .with_output_partitioning(output_partitioning)
 .with_expr_adapter(self.expr_adapter_factory.clone())
 .build();

// create the execution plan
let plan = self.options.format
 .create_physical_plan(state, scan_config)
 .await?;

FileScanConfig is the hand-off point between the catalog and the physical plan:

 rust
 
 
// datafusion/datasource/src/file_scan_config/mod.rs:153-181 (excerpt)
pub struct FileScanConfig {
 /// Object store URL, used to get an [`ObjectStore`] instance from
 /// [`RuntimeEnv::object_store`]
 pub object_store_url: ObjectStoreUrl,
 /// List of files to be processed, grouped into partitions
 ///
 /// DataFusion may attempt to read each partition of files
 /// concurrently, however files *within* a partition will be read
 /// sequentially, one after the next.
 ///
 /// Note that when `datafusion.execution.enable_file_stream_work_stealing`
 /// is enabled (the default), files may be reassigned to a different
 /// partition at runtime unless `preserve_order` or
 /// `partitioned_by_file_group` is set, so a file is not guaranteed to be
 /// read by the partition it is grouped under here.
 pub file_groups: Vec<FileGroup>,
 ...

“Files within a partition will be read sequentially, one after the next” — that’s the reason FileGroup exists at all. It’s the unit of sequential work that is distributed in parallel across partitions.

The statistics attached to the config are the merged table-level ones:

 rust
 
 
// datafusion/datasource/src/statistics.rs:380-413 (excerpt)
pub fn compute_all_files_statistics(
 file_groups: Vec<FileGroup>,
 table_schema: SchemaRef,
 collect_stats: bool,
 inexact_stats: bool,
) -> Result<(Vec<FileGroup>, Statistics)> {
 let file_groups_with_stats = file_groups.into_iter()
 .map(|file_group| compute_file_group_statistics(file_group, Arc::clone(&table_schema), collect_stats))
 .collect::<Result<Vec<_>>>()?;
 // Then summary statistics across all file groups
 let file_groups_statistics = file_groups_with_stats.iter()
 .filter_map(|file_group| file_group.file_statistics(None));
 let mut statistics = Statistics::try_merge_iter_with_ndv_fallback(
 file_groups_statistics, &table_schema, NdvFallback::Max,
 )?;
 if inexact_stats {
 statistics = statistics.to_inexact()
 }
 Ok((file_groups_with_stats, statistics))
}

Two levels: per-file-group statistics (merged from the files in the group), then a table-level merge across groups. NdvFallback::Max is the conservative choice for NDV merging — for files from the same table, the NDV of the union is bounded by the larger NDV, not the sum.

The whole pipeline

text

 ListingTable::scan_with_args
 │
 ├─ split filters: partition-only vs data
 ├─ inflate limit by skip
 │
 ▼ list_files_for_scan
 │
 ├─ evaluate_partition_prefix → "dt=2024-01-01"
 ├─ list_prefixed_files (LIST, with cache)
 ├─ parse_partitions_for_path → partition values per file
 ├─ filter_partitioned_file → drop non-matching partitions
 │
 ▼ collect_files_for_scan
 │
 ├─ try_join_all over table_paths (parallel listing)
 ├─ flatten_unordered / buffer_unordered (meta_fetch_concurrency = 32)
 │ └─ infer_stats_and_ordering (one footer read per file)
 │
 ▼ get_files_with_limit ── early exit when Exact rows > limit
 │
 ▼ split_files / group_by_partition_values → Vec<FileGroup>
 │
 ▼ compute_all_files_statistics → table Statistics
 │
 ▼ FileScanConfigBuilder → FileScanConfig → DataSourceExec

Trade-offs and failure modes

Listing is O(files). For a table with a million small files, the LIST operation dominates query latency, and no amount of parallelism in the scan helps. This is the “small files problem”; the mitigations are compaction upstream, or the listing cache.

The listing cache is a correctness hazard if the table changes. The cache key is {table, path} plus a schema fingerprint and the object metadata, and is_valid_for checks those. But if a file is added to the table, a cached listing will not see it.

collect_statistics = false kills everything downstream. Look at the branch again:

 rust
 
 
let (statistics, ordering) = if ctx.config().collect_statistics() {
 self.do_collect_statistics_and_ordering(ctx, store, &part_file).await?
} else {
 (Arc::new(Statistics::new_unknown(&self.file_schema)), None)
};

No statistics means no file-level pruning, no file ordering, no NDV-based join estimates, no CollectLeft decision — and, crucially, no early exit on LIMIT, because the row count would be Absent and the Precision::Exact guard would never fire. So SELECT * FROM t LIMIT 1 would list every file in the table.

Zero-byte files are silently dropped, and so are files whose paths don’t line up with the declared partition layout. If you use zero-byte data files as placeholders, or typo your partition column list, data disappears without an error.

preserve_file_partitions can produce fewer groups than partitions. If you have 3 distinct partition values and target_partitions = 16, you get 3 file groups, the scan has 3 partitions, and the rest of the machine idles.

The LIMIT early exit changes the statistics’ meaning. The table’s reported row count is the sum over the files that were examined, not the whole table. That’s why inexact_stats exists and why the result is downgraded to Precision::Inexact.

meta_fetch_concurrency is a double-edged sword. 32 concurrent footer reads is great for latency and terrible for a store with strict request limits.

Configuration

Option	Default	Effect
`datafusion.execution.collect_statistics`	`true`	Read file footers for statistics. Off ⇒ no pruning, no ordering, no `LIMIT` early exit.
`datafusion.execution.meta_fetch_concurrency`	`32`	Concurrent per-file metadata fetches.
`datafusion.execution.listing_table_ignore_subdirectory`	`true`	Whether the table’s glob may match files in subdirectories.
`datafusion.optimizer.preserve_file_partitions`	`0`	`0` disables grouping by partition values; `1` always groups; `N` groups only when there are ≥ `N` distinct partition values.
`datafusion.execution.enable_file_stream_work_stealing`	`true`	Allows the runtime to reassign files to different partitions than the `FileGroup` they were grouped into.

Bottom line

The listing layer is where DataFusion’s “query engine” part ends and its “data lake client” part begins. Three ideas do most of the work:

Push the filter into the path. A partition equality becomes a LIST prefix, and the object store does the pruning for free. Getting the prefix wrong in the “too narrow” direction loses data, so the code bails out conservatively at every ambiguity.
Amortize the metadata reads. One footer read per file yields both statistics and ordering; the reads happen concurrently; the results are cached by {table, path, schema fingerprint}.
Stop early when you can prove it’s safe. A LIMIT with exact row counts lets the whole pipeline terminate after a handful of files, and the statistics are honestly downgraded to Inexact to reflect that.

The one thing to internalize: the file listing is a full pass over the table’s metadata. Everything downstream — pruning, join selection, limit pushdown — is cheap by comparison. If your query is slow and the table has many files, look at the listing before you look at the plan.

Planning on Opensource internals

DataFusion's Interval Analysis: How `BETWEEN` Becomes a Scalar Range and a Selectivity Number

The vocabulary: Precision<T>

Step 1: BETWEEN is not special

Step 2: intervals over ScalarValue

Step 3: the column’s interval comes from statistics

Step 4: propagating the predicate backwards

Step 5: from interval to number

Step 6: where the number is consumed

The memoization that makes this affordable

Trade-offs and failure modes

Configuration

Bottom line

DataFusion's PruningPredicate: Turning a Filter into a Per-File 'Can This Possibly Match?' Test

The contract: PruningStatistics

Compiling the predicate

The IN-list trick: a sorted domain instead of an OR tree

The two-pass prune

The amortized cost model

Where file statistics come from — and the Exact trap

Partition values fold into the predicate

Trade-offs and failure modes

Configuration

Bottom line

HashJoin vs SortMergeJoin vs NestedLoopJoin: What DataFusion Actually Uses Statistics For

The algorithm decision is structural, not statistical

Why the range filter branch is so specific

Null-aware joins are pinned

Where statistics do matter: JoinSelection

CollectLeft vs Partitioned: the real cost decision

The guards that stop statistics from breaking things

Unbounded inputs: a separate, non-statistical rule

Estimating the join’s output cardinality

The extension point: StatisticsRegistry

Trade-offs and failure modes

Configuration

Bottom line

Inside DataFusion's ListingTable: Listing an Object Store in Parallel and Turning Files into FileGroups

The contract: TableProvider

Stage 0: split the filter

Stage 1: turn the filter into a path prefix

The recursive directory walk

Stage 2: list the prefix and decode partitions

Stage 3: fetch metadata concurrently

Stage 3.5: the LIMIT shortcut

Stage 4: pack files into FileGroups

Stage 5: build the scan

The whole pipeline

Trade-offs and failure modes

Configuration

Bottom line

The vocabulary: `Precision<T>`

Step 1: `BETWEEN` is not special

Step 2: intervals over `ScalarValue`

The contract: `PruningStatistics`

Where file statistics come from — and the `Exact` trap

Where statistics do matter: `JoinSelection`

`CollectLeft` vs `Partitioned`: the real cost decision

The extension point: `StatisticsRegistry`

The contract: `TableProvider`

Stage 3.5: the `LIMIT` shortcut

Stage 4: pack files into `FileGroup`s