From SQL WHERE to Parquet Row Filter: How DataFusion Decides What It Can Push Down (and What It Must Keep)

TL;DR: WHERE x > 5 reaches the Parquet reader as two independent questions: “can statistics prove this is false here?” (pruning) and “can the decoder evaluate this while it decodes?” (row filter). The support matrices for those two questions are different, non-overlapping, and both are one-sided: unsupported means “keep the data”, never “drop it”. A conjunct that neither system can handle stays in the physical plan as a residual filter, because dropping it would silently change the answer.

One predicate, two questions

When a FilterExec above a Parquet scan pushes its predicate down, the predicate arrives at the source as a single PhysicalExpr. From there it has to serve two masters:

  WHERE a = 1 AND b = 2 AND c % 2 = 0
              │
     ┌────────┴─────────┐
     ▼                  ▼
 statistics pruning   decoder pushdown
 "can no row in this    "can the decoder evaluate
  row group match?"      this while decoding?"
     │                  │
     ▼                  ▼
 PruningPredicate   RowFilter (ArrowPredicate)

The two questions have different answers for the same expression, and conflating them is the classic source of bugs in this area. c % 2 = 0 is a fine row filter (the decoder can compute a modulo) but is useless for statistics pruning (no range of c rules it out). s['x'] > 3 is fine for the decoder but not for statistics, because Parquet statistics for nested fields are not wired into the pruning predicate rewrite yet.

Question 1: which predicates can prune row groups?

Statistics pruning is implemented in the datafusion/pruning crate. The mechanism is a rewrite: the predicate is transformed into an expression over statistics columns (x_min, x_max, x_null_count, x_row_count), which is then evaluated against a record batch of statistics — one row per container (row group, file, …).

The documented rewrite table (datafusion/pruning/src/pruning_predicate.rs:244-252):

Everything else is handled by a rewrite function per expression type, and the entry point for a comparison is rewrite_expr_to_prunable (pruning_predicate.rs:1243-1318). Its rules: the left side must be a monotone-ish column expression and the right side a literal. The column expression may be a column, a cast of a column, a negated column, or a negated literal comparison. The rewrite even normalizes -col > 10 into col < -10 so that only the monotone direction has to be handled, and refuses Not over anything other than an equality-family operator (pruning_predicate.rs:1296-1316).

On top of that, build_predicate_expression (pruning_predicate.rs:1780-1990) recognises a specific set of node types before falling through to the comparison rewrite: IS NULL / IS NOT NULL on a column, a bare column (used for NOT col), IN (...) lists, a HashTableLookupExpr (a semi-join turned into a lookup), boolean CASE expressions (but only with an ELSE branch), case-sensitive LIKE / NOT LIKE, and AND / OR of the above.

LIKE becomes a range

The LIKE handling is a nice piece of engineering: a prefix match is exactly a range, so col LIKE 'foo%' is rewritten into col_min <= 'foo' AND 'foo' <= col_max, with the upper bound computed by incrementing the last UTF-8 byte of the prefix (pruning_predicate.rs:2255-2291):

// datafusion/pruning/src/pruning_predicate.rs:2276-2286
let (decoded_prefix, rest) = split_constant_prefix(s);
let has_wildcard = !rest.is_empty();
if has_wildcard && decoded_prefix.is_empty() {
    // there's no filtering we could possibly do, return None and have this be handled by the unhandled hook
    return None;
}
let (lower_bound, upper_bound) = if has_wildcard {
    let incremented_prefix = increment_utf8(&decoded_prefix)?;

Escaped wildcards are handled too ('foo\_%' bounds at 'foo_'), and a pattern with no usable prefix ('%foo') degenerates to “no filtering possible”. ILIKE is explicitly not supported — the TODO says it would need lower()/upper() bounds, which the statistics rewrite cannot express (pruning_predicate.rs:2266-2267).

IN lists and the size cap

IN lists are rewritten into a disjunction of equalities, but only up to a cap, documented in the builder (pruning_predicate.rs:462-484): lists up to min(20, C) values use the per-value rewrite; lists between 20 and the configured cap C use a compact pruning expression when the values are ordered literals, and fall back to the per-value rewrite otherwise; lists longer than C hit the unhandled-predicate hook, i.e. “keep the container”.

The default cap is 20 (MAX_IN_LIST_SIZE, pruning_predicate.rs:1697), and query engines wire it from datafusion.execution.parquet.max_in_list_size. The reason for a cap is not paranoia about the list but about the size of the rewritten expression: an IN list of 10,000 values becomes a tree of 10,000 comparisons, evaluated once per row group, over statistics columns. Above the cap, the predicate simply stops pruning — exactly the “keep the container” behaviour, and therefore safe.

The unhandled hook: the constant true

Anything the rewriter cannot handle is passed to an UnhandledPredicateHook. The default implementation returns a literal true (pruning_predicate.rs:587-604) — meaning “there may be matching rows”, i.e. keep the container. This is the single most important safety property of the whole design: an expression the rewriter does not understand becomes a proof of nothing.

AND is forgiving, OR is not

Partial support is allowed, but only in one direction. An AND can use whichever conjuncts it understands, because a row must satisfy all of them. An OR cannot: if one branch is unprunable, the disjunction might be true because of that branch.

The rewriter encodes this structurally: AND/OR combine the rewritten children, and simplification collapses true AND x to x and true OR x to true — i.e. an unprunable branch under OR poisons the whole disjunction back to “keep everything” (pruning_predicate.rs:1934-1970). The behaviour is visible in DataFusion’s own tests: c1 > 15 AND (c2 % 2 = 0) still prunes row group 1, while c1 > 15 OR (c2 % 2 = 0) prunes nothing (row_group_filter.rs:1059-1121).

Question 2: which predicates can the decoder evaluate?

The decoder-level support matrix is a completely separate implementation, living in projection_read_plan.rs (PushdownChecker) and supported_predicates.rs. Its rules:

• primitive columns: fine,
• list columns: only with a registered list predicate (array_has, array_has_all, array_has_any) or a null check,
• struct columns: only via get_field down to a primitive leaf,
• columns absent from the file: not pushable,
• UDFs that require a rewrite: not pushable (has_unpushable_udfs).

The list-predicate registry is tiny and explicit:

// datafusion/datasource-parquet/src/supported_predicates.rs:81-86
fn is_supported_scalar_function(expr: &dyn PhysicalExpr) -> bool {
    scalar_function_name(expr).is_some_and(|name| {
        // Registry of verified array functions
        matches!(name, "array_has" | "array_has_all" | "array_has_any")
    })
}

Expression types defined in external crates cannot implement DataFusion’s trait directly (Rust orphan rules), so the trait has a blanket implementation that delegates to this name-based registry. The registry is a proof obligation list — each entry is a function whose semantics have been verified to be evaluable on a decoded list column.

The residual filter: why “keep it” is mandatory

Neither of the two systems is allowed to drop a conjunct. If a conjunct cannot be evaluated during decoding, it must still be evaluated somewhere, or the query returns rows that do not satisfy the predicate. DataFusion encodes this in the filter pushdown protocol:

// datafusion/datasource-parquet/src/source.rs:853-872
let filters: Vec<PushedDownPredicate> = filters.into_iter()
    .map(|filter| {
        if can_expr_be_pushed_down_with_schemas(&filter, pushable_schema) {
            PushedDownPredicate::supported(filter)
        } else {
            PushedDownPredicate::unsupported(filter)
        }
    })
    .collect();

PushedDown::No means “I did not evaluate this; you must”. The parent FilterExec stays in the plan and applies it to the batches the scan produces. (The same holds when pushdown is disabled entirely: the predicate is still used for statistics pruning, but the source reports PushedDown::No for everything so the parent keeps its filter, source.rs:880-896.)

There is a second, subtler reason to keep the residual: the source’s claims about its own output must be true. FileSource distinguishes filter() (the predicate the source uses) from exact_filter() (the part of it that every output row is guaranteed to satisfy):

// datafusion/datasource/src/file.rs:113-128
/// Returns the part of [`Self::filter`] that every output row is
/// guaranteed to satisfy.
///
/// [`FileScanConfig`] derives equivalence properties (constants and equal
/// columns) from this filter. A filter that the source uses only to skip
/// data (for example, row group or page pruning with statistics) does not
/// remove every non-matching row, so it must not be returned here.

If the source claimed the whole predicate as exact, the optimizer would be allowed to conclude that c = 5 implies c is constant, and would rewrite the plan accordingly. If the predicate was only used for pruning, that conclusion is false and the rewritten plan returns wrong rows. So exact_filter returns only the conjuncts that were genuinely evaluated during decoding:

// datafusion/datasource-parquet/src/source.rs:682-694
fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
    if !self.pushdown_filters() { return None; }
    let predicate = self.predicate.as_ref()?;
    let pushable_schema = self.table_schema.schema_without_virtual_columns();
    let exact = split_conjunction(predicate)
        .into_iter()
        .filter(|expr| can_expr_be_pushed_down_with_schemas(expr, pushable_schema))
        .cloned()
        .collect::<Vec<_>>();
    (!exact.is_empty()).then(|| conjunction(exact))
}

The same discipline shows up inside the pruning rewrite itself. Some rewrites preserve only “which rows can be TRUE” but lose the distinction between FALSE and UNKNOWN — for example NOT IN with a NULL member, or a CASE without an ELSE. The rewriter flags those with has_filter_semantics_only (pruning_predicate.rs:390-396), and such a predicate is fine for pruning (it only needs “can it be true?”), but useless for proving that every row matches — so the inversion that powers full-match inference is refused for it (pruning_predicate.rs:570, can_be_inverted_for_full_match).

The catch

• Support is per expression, not per column. A column can be pushable in one predicate and not in another: s['v'] > 5 is pushable, s IS NOT NULL is not; array_has_all(tags, ...) is pushable, tags[1] = 'x' may not be. Any classification logic that works on columns rather than expressions is wrong.
• Schema adaptation can change the answer. The pushdown check runs against the physical file schema after casts have been inserted, which is why pushdown_columns takes an allow_struct_casts flag: “Struct casts are accepted only after schema adaptation, not while planning against the table schema” (row_filter.rs:247-249).
• Three-valued logic is not optional. The rewrite table above shows x_null_count != x_row_count in every equality. Drop it and a column that is entirely NULL would look like it might contain 5.
• The unhandled hook hides bugs. Because an unsupported predicate silently becomes “keep everything”, a regression in the rewriter shows up as a performance regression, not a correctness one. DataFusion mitigates this with counters (predicate_creation_errors, predicate_evaluation_errors) and with PruningPredicateBuilder’s try_build variant that surfaces the error instead of swallowing it.
• Unsupported does not mean free. Every unsupported conjunct still costs a residual FilterExec over the output batches — the cheapest place to evaluate it, but not free, and the reason pushdown is off by default.

The through-line: every classification decision in this pipeline is optimistic only about performance and pessimistic about correctness. Supported means “I can prove this or evaluate this”; unsupported means “keep the data and keep the filter”. A system that errs in the other direction would be faster and wrong, and no amount of benchmarking would tell you.