DataFusion's PruningPredicate: Turning a Filter into a Per-File 'Can This Possibly Match?' Test

TL;DR: PruningPredicate compiles your filter into an expression over statistics columns (x_min, x_max, x_null_count, x_row_count) instead of over data columns. x = 5 becomes x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max). The compiled expression is evaluated once, vectorized, against a batch holding one row per container — so pruning 10,000 files costs roughly the same as pruning one.

There is a subtle trap in query engines, and it is easy to fall into.

You have a filter WHERE ts BETWEEN '2024-01-01' AND '2024-01-31'. The naive implementation is: open the file, read the rows, evaluate the predicate per row, discard the ones that fail. That is correct, it is simple, and it buys you exactly nothing — you paid for the I/O you were trying to avoid.

The whole point of a zone map is to answer a different question: can this file possibly contain a matching row? If the answer is provably “no”, you skip the file without opening it.

DataFusion’s implementation lives in the datafusion-pruning crate, and it is more interesting than it looks: not just min/max comparison, but a small compiler that rewrites arbitrary boolean expressions into an equivalent expression over statistics columns, then evaluates that expression in a vectorized batch.

The contract: `PruningStatistics`

The input to the whole machinery is a trait that supplies per-container statistics as Arrow arrays:

rust
// datafusion/common/src/pruning.rs:63-97
pub trait PruningStatistics {
    /// Return the minimum values for the named column, if known.
    ///
    /// If the minimum value for a particular container is not known, the
    /// returned array should have `null` in that row. If the minimum value is
    /// not known for any row, return `None`.
    ///
    /// Each non-null entry must be a conservative lower bound for all non-null
    /// values in its container, using Arrow's comparison order for the column
    /// type. For strings, this is unsigned lexicographic UTF-8 byte order.
    /// Bounds that cannot satisfy this contract must be reported as unknown.
    ///
    /// Note: the returned array must contain [`Self::num_containers`] rows
    fn min_values(&self, column: &Column) -> Option<ArrayRef>;
    ...
    fn num_containers(&self) -> usize;

Two design decisions here are worth dwelling on.

First: the arrays are columnar, not per-container. The trait does not have “give me the stats for file 37”. It has “give me the minimums for column x across all 3,412 containers, as one array”. That is the amortization trick, and it is stated in the trait’s own doc comment:

rust
// datafusion/common/src/pruning.rs:41-45
/// Information for containers / files are returned as Arrow [`ArrayRef`], so
/// the evaluation happens once on a single [`RecordBatch`], which amortizes the
/// overhead of evaluating the predicate. This is important when pruning 1000s
/// of containers which often happens in analytic systems that have 1000s of
/// potential files to consider.

Second: the contract is on the provider. “Bounds that cannot satisfy this contract must be reported as unknown. Pruning relies on providers to uphold this contract; it does not validate the ordering or bound guarantees of arbitrary statistics sources.” This is the load-bearing sentence of the whole design. DataFusion does not check that your min is really the minimum. If you lie, you get wrong answers — not a warning, not a fallback. Answers.

Compiling the predicate

The rewriting is documented, in prose, right at the top of the crate:

rust
// datafusion/pruning/src/pruning_predicate.rs:242-251
/// Original Predicate | Rewritten Predicate
/// ------------------ | --------------------
/// `x = 5` | `x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max)`
/// `x < 5` | `x_null_count != x_row_count AND (x_min < 5)`
/// `x = 5 AND y = 10` | `x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max) AND y_null_count != y_row_count (y_min <= 10 AND 10 <= y_max)`
/// `x IS NULL`  | `x_null_count > 0`
/// `x IS NOT NULL`  | `x_null_count != row_count`
/// `CAST(x as int) = 5` | `x_null_count != x_row_count (CAST(x_min as int) <= 5 AND 5 <= CAST(x_max as int))`

The null_count != row_count term is the subtle one. It exists to express “this column is not entirely NULL”. If x is all NULLs, then x = 5 is UNKNOWN for every row, and UNKNOWN rows are filtered out — so the container can be pruned. But you can only prove that from the null count. Without the term, an all-NULL container with x_min = NULL would evaluate the rewritten predicate to NULL rather than FALSE, and NULL means “keep”.

That distinction is the crux of the entire design:

true: there MAY be rows that match the predicate → KEEP
NULL: there MAY be rows that match the predicate → KEEP
false: there are NO rows that could possibly match → PRUNE

Note that this is not SQL filter semantics, where NULL means “filtered out”. The pruning predicate inverts it: NULL is unknown, and unknown means “don’t prune, be conservative”.

The rewrite happens in build_predicate_expression, a recursive descent over the physical expression tree:

rust
// datafusion/pruning/src/pruning_predicate.rs:1780-1996 (excerpt)
fn build_predicate_expression(
    expr: &Arc<dyn PhysicalExpr>,
    schema: &SchemaRef,
    required_columns: &mut RequiredColumns,
    unhandled_hook: &Arc<dyn UnhandledPredicateHook>,
    max_in_list_size: usize,
    properties: &mut PruningExpressionProperties,
) -> Arc<dyn PhysicalExpr> {
    if is_always_false(expr) {
        // Shouldn't return `unhandled_hook.handle(expr)`
        // Because it will transfer false to true.
        return Arc::clone(expr);
    }
    if let Some(is_null) = expr.downcast_ref::<phys_expr::IsNullExpr>() { ... }
    if let Some(in_list) = expr.downcast_ref::<phys_expr::InListExpr>() { ... }
    ...
    if op == Operator::And || op == Operator::Or {
        let left_expr = build_predicate_expression(&left, ...);
        let right_expr = build_predicate_expression(&right, ...);
        // simplify boolean expression if applicable
        ...
        return expr;
    }
    ...
    build_statistics_expr(&mut expr_builder)
        .unwrap_or_else(|_| unhandled_hook.handle(expr))
}

Two properties of this design are important:

Partial failure is fine. If the left half of an AND can be rewritten and the right half cannot, the unrewritable half is replaced by the UnhandledPredicateHook — which by default returns literal true:

rust
// datafusion/pruning/src/pruning_predicate.rs:584-603
/// The default handling for unhandled predicates is to return a constant `true`
/// (meaning don't prune the container)
#[derive(Debug, Clone)]
struct ConstantUnhandledPredicateHook {
    default: Arc<dyn PhysicalExpr>,
}

So an unsupported conjunct degrades to “don’t prune on this side”, and the other conjunct still prunes. AND is the friendliest operator for pruning; OR is the worst, because A OR B can only be pruned if both A and B are provably false.

The rewrite is not just min/max on the column. rewrite_expr_to_prunable normalizes the expression so that the left side is monotonically increasing in the column:

rust
// datafusion/pruning/src/pruning_predicate.rs:1243-1317 (excerpt)
fn rewrite_expr_to_prunable(
    column_expr: &PhysicalExprRef,
    op: Operator,
    scalar_expr: &PhysicalExprRef,
    schema: &Schema,
) -> Result<(PhysicalExprRef, Operator, PhysicalExprRef)> {
    if !is_compare_op(op) {
        return plan_err!("rewrite_expr_to_prunable only support compare expression");
    }
    if column_expr.downcast_ref::<phys_expr::Column>().is_some() {
        // `col op lit()`
        Ok((Arc::clone(column_expr), op, Arc::clone(scalar_expr)))
    } else if let Some(cast) = column_expr.downcast_ref::<phys_expr::CastExpr>() {
        ...
    } else if let Some(neg) = column_expr.downcast_ref::<phys_expr::NegativeExpr>() {
        // `-col > lit()`  --> `col < -lit()`
        ...
    } else if let Some(not) = column_expr.downcast_ref::<phys_expr::NotExpr>() {
        // `!col = true` --> `col = !true`
        ...
    } else {
        plan_err!("column expression {column_expr:?} is not supported")
    }
}

The reason is obvious in hindsight: x_min is only a valid lower bound for x if x appears monotonically. CAST(x AS INT) = 5 is handled by casting the bounds rather than the column; -x > 5 is rewritten to x < -5; !x = true is rewritten to x = false. But abs(a - 10) > 0 is rejected, because abs is not monotonic and abs(x_min - 10) is not a bound at all.

There is even a type-safety check for casts, because cast ordering is not the same as source ordering:

rust
// datafusion/pruning/src/pruning_predicate.rs:1351-1375
fn verify_support_type_for_prune(from_type: &DataType, to_type: &DataType) -> Result<()> {
    ...
    // If both types are strings or both are not strings (number, timestamp, etc)
    // then we can compare them.
    // PruningPredicate does not support casting of strings to numbers and such.
    if from_type.is_string() == to_type.is_string() {
        Ok(())
    } else {
        plan_err!(
            "Try Cast/Cast with from type {from_type} to type {to_type} is not supported"
        )
    }
}

("13" < "3" is true in UTF-8 order but "13" < 3 is nonsense, so string→number casts are rejected.)

The IN-list trick: a sorted domain instead of an OR tree

The default rewrite for x IN (1, 2, 3) is to expand it into (x = 1) OR (x = 2) OR (x = 3) and rewrite each — producing a deep tree of min/max comparisons. For long lists that is expensive to build and slow to evaluate.

DataFusion has a compact alternative, gated behind a raised datafusion.execution.parquet.max_in_list_size:

rust
// datafusion/pruning/src/pruning_predicate.rs:459-483
/// Cap on the input length of `IN (...)` lists eligible for min/max pruning.
///
/// For a nonempty list of `N` values (before deduplication) and cap `C`:
///
/// | Condition | Pruning representation |
/// | --- | --- |
/// | `N <= min(20, C)` | Existing per-value rewrite |
/// | `20 < N <= C`, supported ordered literals with optional NULLs | Compact pruning expression |
/// | `20 < N <= C`, unsupported lists | Existing per-value rewrite |
/// | `N > C` | Unhandled-predicate hook, normally "keep the container" |

The compact representation sorts the values once and then does a single binary-search-style probe per container. It is gated behind a raised datafusion.execution.parquet.max_in_list_size — with the default cap of 20 the compact path never engages, because it requires a list longer than 20:

rust
// datafusion/pruning/src/primitive_in_list.rs:252-278
fn may_match(&self, min: Option<T::Native>, max: Option<T::Native>) -> Option<bool> {
    if self.membership == SetMembership::NotIn {
        return not_in_may_match(min.as_ref(), max.as_ref(), |value| {
            self.contains(*value)
        });
    }
    match (min, max) {
        (Some(min), Some(max)) => {
            // Inverted statistics are unusable, not proof that the domain
            // and interval are disjoint.
            if min > max {
                return None;
            }
            let index = self.values.partition_point(|value| *value < min);
            Some(self.values.get(index).is_some_and(|value| *value <= max))
        }
        (Some(min), None) if self.values.last().is_some_and(|value| *value < min) => {
            Some(false)
        }
        (None, Some(max))
            if self.values.first().is_some_and(|value| *value > max) =>
        {
            Some(false)
        }
        _ => None,
    }
}

partition_point finds the first value >= min; if that value is <= max, the container’s interval intersects the domain, so keep it; if it is > max, the container’s whole range is above the domain, so prune it. One O(log n) probe replaces n comparisons.

The NOT IN case is deliberately asymmetric, and the reasoning is a gem:

rust
// datafusion/pruning/src/in_list.rs:22-32
/// Which `IN` form a sorted domain is pruning for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub(crate) enum SetMembership {
    /// `col IN (...)`. A row matches only where the domain intersects the
    /// interval, so a disjoint interval excludes every row.
    In,
    /// `col NOT IN (...)`. Overlap proves nothing here: values outside the
    /// domain still satisfy the predicate. An interval excludes every row only
    /// when it holds a single value that the domain contains.
    NotIn,
}

For NOT IN, “the container’s range overlaps the domain” says nothing — values outside the domain satisfy NOT IN. The only way to exclude a container is if its entire range is a single value that the domain contains:

rust
// datafusion/pruning/src/in_list.rs:53-75
/// Evaluates `NOT IN` against an inclusive statistics interval.
///
/// A container can be excluded only when both bounds identify one value in the
/// domain. A known bound outside the domain proves the container may match and
/// lets an enclosing Boolean expression short-circuit.
///
/// This remains safe when Parquet truncates byte-array statistics. A truncated
/// minimum is no greater than the true minimum, and a truncated maximum is no
/// less than the true maximum. Therefore, equal stored bounds prove that the
/// true minimum and maximum are also equal.
#[inline(always)]
pub(crate) fn not_in_may_match<T: ?Sized + PartialEq>(
    min: Option<&T>,
    max: Option<&T>,
    contains: impl Fn(&T) -> bool,
) -> Option<bool> {
    match (min, max) {
        (Some(min), Some(max)) if contains(min) => Some(min != max),
        (Some(_), Some(_)) => Some(true),
        (Some(bound), None) | (None, Some(bound)) if !contains(bound) => Some(true),
        _ => None,
    }
}

That last doc paragraph is a nice piece of engineering: Parquet truncates long byte-array statistics to a prefix, so min may be less than the true minimum and max may be greater than the true maximum. Equal stored bounds therefore prove equal true bounds (because the true min is >= stored min and the true max is <= stored max, and both are pinned to the same value), so the argument is still sound under truncation.

The string/binary variant does the same thing on raw bytes, with a length comparison first to avoid scanning long common prefixes:

rust
// datafusion/pruning/src/string_in_list.rs:206-246
match (min, max) {
    (Some(min), Some(max)) => {
        if min > max { return None; }
        // String and binary values use unsigned lexicographic byte
        // order, as required by the PruningStatistics min/max contract.
        let index = self.values.partition_point(|v| $bytes!(v) < min);
        Some(self.values.get(index).is_some_and(|v| $bytes!(v) <= max))
    }
    // A missing bound makes that end of the interval unbounded.
    // Exclude only when the whole domain lies beyond the known bound;
    // gaps within the domain and equality cannot prove disjointness.
    (Some(min), None)
        if self.values.last().is_some_and(|v| $bytes!(v) < min) => Some(false),
    (None, Some(max))
        if self.values.first().is_some_and(|v| $bytes!(v) > max) => Some(false),
    _ => None,
}

The two-pass prune

Now the actual evaluation. PruningPredicate::prune does two passes:

rust
// datafusion/pruning/src/pruning_predicate.rs:657-707
pub fn prune<S: PruningStatistics + ?Sized>(
    &self,
    statistics: &S,
) -> Result<Vec<bool>> {
    let mut builder = BoolVecBuilder::new(statistics.num_containers());

    // Try to prove the predicate can't be true for the containers based on
    // literal guarantees
    for literal_guarantee in &self.literal_guarantees {
        let LiteralGuarantee { column, guarantee, literals } = literal_guarantee;
        if let Some(results) = statistics.contained(column, literals) {
            match guarantee {
                Guarantee::In => builder.combine_array(&results),
                Guarantee::NotIn => {
                    builder.combine_array(&arrow::compute::not(&results)?)
                }
            }
            // if all containers are pruned ... can return early
            if builder.check_all_pruned() {
                return Ok(builder.build());
            }
        }
    }

    // Next, try to prove the predicate can't be true for the containers based
    // on min/max values

    // build a RecordBatch that contains the min/max values in the
    // appropriate statistics columns for the min/max predicate
    let statistics_batch =
        build_statistics_record_batch(statistics, &self.required_columns)?;

    // Evaluate the pruning predicate on that record batch and append any results to the builder
    builder.combine_value(self.predicate_expr.evaluate(&statistics_batch)?);

    Ok(builder.build())
}

Pass one uses LiteralGuarantees — a predicate like x IN ('a', 'b') proves that the column can only contain those values, and the contained hook lets a provider answer “does this container contain only values from this set?”. Parquet Bloom filters implement exactly this API. The pass can short-circuit entirely if every container is already proven prunable.

Pass two evaluates the compiled min/max expression against a synthesized RecordBatch of statistics columns. One evaluate call, one batch, all containers.

The flow:

text

 filter expr (x >= lo AND x <= hi)
        │
        ▼  build_predicate_expression
 compiled predicate expr over statistics columns
        │
        ├─ RequiredColumns tells us which columns to fetch:
        │     x_min, x_max, x_null_count, x_row_count
        │
        ▼  build_statistics_record_batch
 ┌────────────┬────────┬────────┬────────────┬───────────┐
 │ x_min      │ x_max  │ null_ct│ row_count  │ (y_min..) │
 ├────────────┼────────┼────────┼────────────┼───────────┤
 │ file0: 0   │ file0: 99 │ ...  │ ...        │ ...       │
 │ file1: 500 │ file1: 600 │ ...  │ ...        │ ...       │
 │ ...        │ ...    │ ...   │ ...        │ ...       │
 └────────────┴────────┴────────┴────────────┴───────────┘
        │
        ▼  predicate_expr.evaluate(&batch)   ← ONE vectorized call
 [true, false, true, ...]
        │
        ▼
 skip file1 entirely; never open it

The amortized cost model

This is the part that is easy to get wrong, so let’s be precise about the costs.

Building the predicate is expensive. It requires expression analysis, a recursive rewrite, a RequiredColumns analysis, and a simplification pass:

rust
// datafusion/pruning/src/pruning_predicate.rs:546-561
// build predicate expression once
let mut required_columns = RequiredColumns::new();
let mut properties = PruningExpressionProperties::default();
let predicate_expr = build_predicate_expression(
    &predicate, &file_schema, &mut required_columns,
    &unhandled_hook, self.max_in_list_size, &mut properties,
);
let predicate_schema = required_columns.schema();
// Simplify the newly created predicate to get rid of redundant casts, comparisons, etc.
let predicate_expr =
    PhysicalExprSimplifier::new(&predicate_schema).simplify(predicate_expr)?;
let literal_guarantees = LiteralGuarantee::analyze(&predicate);

Evaluating it per container is cheap, because the compiled expression is pure min/max arithmetic over a few arrays.

So the invariant is: pay the compilation cost once, amortize it over as many containers as possible. Evaluating the original predicate per row costs O(rows). Evaluating a compiled predicate per file costs O(files × predicate_size). Evaluating it once over a vectorized batch costs the same O(files × predicate_size) in SIMD-friendly Arrow kernels, with no per-file dispatch at all.

DataFusion’s file-level pruner makes the “build once” part explicit:

rust
// datafusion/pruning/src/file_pruner.rs:127-170
pub fn should_prune(&mut self) -> Result<bool> {
    // Building the pruning predicate is expensive (it involves expression
    // analysis), so we only do it on the first check and whenever a dynamic
    // filter inside the predicate has actually moved.
    let should_build = if self.checked_once {
        self.tracking.watcher().is_some_and(|w| w.changed())
    } else {
        self.checked_once = true;
        true
    };
    if !should_build {
        return Ok(false);
    }
    let pruning_predicate = build_pruning_predicate(
        Arc::clone(&self.predicate),
        &self.file_schema,
        &self.predicate_creation_errors,
    );
    ...
}

DynamicFilterTracking is the interesting bit: a hash join can push a runtime filter (a list of build-side keys) into the probe-side scan. That filter’s value changes as the build side fills, so the pruning predicate must be rebuilt when it moves — but not on every batch. The watcher reports a change at most once per update.

And the Parquet row-group pruner takes the “one call for all row groups” approach:

rust
// datafusion/datasource-parquet/src/row_group_filter.rs:367-380
// try to prune the row groups in a single call
match predicate.prune(&pruning_stats) {
    Ok(values) => {
        let mut fully_contained_candidates_original_idx: Vec<usize> =
            Vec::with_capacity(row_group_indexes.len());
        for (idx, &value) in row_group_indexes.iter().zip(values.iter()) {
            if !value {
                self.access_plan.skip(*idx);
                metrics.row_groups_pruned_statistics.add_pruned(1);
            } else {
                metrics.row_groups_pruned_statistics.add_matched(1);
                fully_contained_candidates_original_idx.push(*idx);
            }
        }
        ...

One prune() call, one boolean array, N row groups decided.

Where file statistics come from — and the `Exact` trap

The bridge from “file statistics” to “pruning statistics” is PrunableStatistics:

rust
// datafusion/common/src/pruning.rs:342-372
fn get_exact_column_statistics(
    &self,
    column: &Column,
    get_stat: impl Fn(&ColumnStatistics) -> &Precision<ScalarValue>,
) -> Option<ArrayRef> {
    let index = self.schema.index_of(column.name()).ok()?;
    let mut has_value = false;
    match ScalarValue::iter_to_array(self.statistics.iter().map(|s| {
        s.column_statistics.get(index)
            .and_then(|stat| {
                if let Precision::Exact(min) = get_stat(stat) {
                    has_value = true;
                    Some(min.clone())
                } else {
                    None
                }
            })
            .unwrap_or(ScalarValue::Null)
    })) {
        // If there is any non-null value and no errors, return the array
        Ok(array) => has_value.then_some(array),
        ...

Only Precision::Exact bounds are used for pruning. An Inexact min is treated as unknown. This is deliberate and correct: pruning on an approximate bound could skip a file that does contain matching rows, which is a wrong answer, not a slow query. It also means that if your catalog reports approximate statistics, you get no file-level pruning at all — only row-group pruning from Parquet footers, which is exact by construction. null_counts and row_counts behave the same way: they are reported only if at least one container has an exact value, and per-container inexactness is encoded as a NULL in the array.

Partition values fold into the predicate

For Hive-partitioned tables, the partition column values live in the path, not in the file. DataFusion substitutes them into the predicate before pruning:

rust
// datafusion/physical-expr-adapter/src/schema_rewriter.rs:64-83
pub fn replace_columns_with_literals<K, V>(
    expr: Arc<dyn PhysicalExpr>,
    replacements: &HashMap<K, V>,
) -> Result<Arc<dyn PhysicalExpr>>
where
    K: Borrow<str> + Eq + Hash,
    V: Borrow<ScalarValue>,
{
    expr.transform_down(|expr| {
        if let Some(column) = expr.downcast_ref::<Column>()
            && let Some(replacement_value) = replacements.get(column.name())
        {
            return Ok(Transformed::yes(expressions::lit(
                replacement_value.borrow().clone(),
            )));
        }
        Ok(Transformed::no(expr))
    })
    .data()
}

So WHERE year = 2024 AND amount < 50 becomes, for a file under year=2024/, 2024 = 2024 AND amount < 50 — and the first conjunct folds to literal true in the simplifier, leaving a pruning predicate that only mentions amount. The FilePruner docstring states the invariant plainly:

rust
// datafusion/pruning/src/file_pruner.rs:32-36
/// Prune based on file-level statistics.
///
/// Note: Partition column pruning is handled earlier via `replace_columns_with_literals`
/// which substitutes partition column references with their literal values before
/// the predicate reaches this pruner.

Trade-offs and failure modes

Wrong statistics produce wrong answers. This is not a “slow query” class of bug. The PruningStatistics contract says providers must supply conservative bounds, and the code explicitly declines to validate them. A catalog that reports min = 0 for a file whose true min is 100 will get that file pruned for WHERE x < 50, and the rows in [50, 100) will silently vanish.

Pruning on NULL means “keep”. Every rewritten predicate is careful to produce NULL (unknown) rather than FALSE when the statistics are incomplete. This makes pruning conservative but also means that absent statistics prune nothing. A Parquet file written without statistics, or a column excluded from the footer, disables pruning for that column entirely.

OR prunes badly. A OR B can only exclude a container if both sides are provably false. If one side references an unsupported expression, the whole OR degrades to “keep”. Filter pushdown rewrites that turn a conjunctive filter into a disjunction will silently kill pruning.

All-NULL columns need the null-count term. Without x_null_count != x_row_count, an all-NULL container would evaluate the rewritten x = 5 to NULL and be kept — forever, for every predicate.

String ordering is a footgun. The contract requires unsigned lexicographic UTF-8 byte order, matching Arrow. Parquet’s own ordering conventions differ, and DataFusion’s Parquet adapter has to mask bounds whose ordering it cannot establish:

rust
// datafusion/datasource-parquet/src/row_group_filter.rs:289-292
/// This method has no file footer, so it cannot establish the sort order
/// of string or binary min/max statistics. Such bounds are ignored. Use
/// [`Self::prune_by_statistics_with_metadata`] when the full metadata is
/// available.

Pruning is not free. Compiling the predicate involves expression analysis. For a scan over 3 files, pruning may cost more than reading them. There is no “is pruning worth it” heuristic at this layer; the predicate_creation_errors counter exists so you can at least see failures.

Configuration

Option	Default	Effect
`datafusion.execution.parquet.max_in_list_size`	`20`	Cap on `IN (...)` list length eligible for min/max pruning. Lists of ≤ 20 values use the per-value rewrite; the compact sorted-domain form needs a list of > 20 values, so it only engages if you raise the cap above 20. `0` disables the `IN` rewrite entirely (other pruning still works).
`datafusion.execution.collect_statistics`	`true`	Whether file-level statistics are gathered. Without them, `PrunableStatistics` has nothing exact to work with and file-level pruning is dead.

MAX_IN_LIST_SIZE is the hard-coded floor for the per-value path:

rust
// datafusion/pruning/src/pruning_predicate.rs:1686-1693
/// Default maximum number of entries in an `IN (...)` list eligible for
/// statistics pruning.
pub const MAX_IN_LIST_SIZE: usize = 20;

Bottom line

The pruning layer is a small compiler with a very specific target: turn a boolean expression over data columns into a boolean expression over statistics columns, so that the answer to “which files can match?” is computed with vectorized Arrow kernels over one synthesized batch rather than by opening files.

The cleverness is not in the min/max comparison — every columnar engine has that. It is in (a) the rewrite rules that make arbitrary predicates expressible over bounds, including the careful null_count != row_count trick and the IN-list sorted-domain probe, and (b) the refusal to ever guess: NULL means keep, unhandled predicates mean keep, Inexact statistics mean keep.

Which is exactly why the correctness of the whole thing rests on the statistics provider honoring a contract the code chooses not to check.

The contract: PruningStatistics