DataFusion's Parquet Access Plan: How Row Groups, Pages and Files Get Skipped Before a Byte Is Read

TL;DR: DataFusion turns “which bytes do I need?” into a data structure — a ParquetAccessPlan — rather than a sequence of if statements buried in the reader. The plan starts as “scan everything”, then gets narrowed by cheap proofs: file-level statistics, then per-row-group min/max (zone maps), then per-page min/max from the column index, then bloom filters. Every narrowing step only ever removes work; it never adds a scan. The proof obligation is one-sided: a row group may only be skipped when statistics show that no row in it can satisfy the predicate.

The naive reader

The obvious way to answer “which row groups do I need?” is to open the file, look at the footer, and for each row group ask “could this row group contain a row with ts > 2024-01-01?”. That is what a simple Parquet reader does, and it is fine — as long as you have only one kind of evidence and one place to ask the question.

Real queries are not like that. A predicate might be provably false for a whole file (a partition column says region = 'eu', the predicate wants region = 'us'), provably false for a row group (the min/max of ts in the footer), provably false for a page (the min/max of ts in the column index), or provably false only if you consult a bloom filter. And in the other direction: it may be provable that every row in a row group matches, which lets you skip the per-row filter entirely.

DataFusion’s design decision is that all of these must be expressed in one place, and that place must be inspectable, testable, and independent of the decoder: ParquetAccessPlan.

The access plan

// datafusion/datasource-parquet/src/access_plan.rs:145-152
pub enum RowGroupAccess {
    /// Do not read the row group at all
    Skip,
    /// Read all rows from the row group
    Scan,
    /// Scan only the specified rows within the row group
    Selection(RowSelection),
}

Three states, not two. Selection matters: page-index pruning produces row ranges, and those ranges have to survive the rest of the pipeline (including the row-group reordering used for TopK). ParquetAccessPlan keeps the Selection per row group and intersects new selections into it rather than replacing it:

// datafusion/datasource-parquet/src/access_plan.rs:438-463
pub fn scan_selection(&mut self, idx: usize, selection: RowSelection) {
    self.row_groups[idx] = match &self.row_groups[idx] {
        RowGroupAccess::Skip => RowGroupAccess::Skip,
        RowGroupAccess::Scan => RowGroupAccess::Selection(selection),
        RowGroupAccess::Selection(existing_selection) => {
            // ... promote to mask backing when one side is a bitmap, so an
            // external index's bitmap survives the intersection
            RowGroupAccess::Selection(existing_selection.intersection(&selection))
        }
    };
}

Note the bitmap promotion comment: arrow-rs keeps a RowSelection either as run-length selectors or as a boolean bitmap, and intersecting the two forms is not free — so DataFusion converts the selector side into a bitmap first, keeping the cheapest representation for repeated intersections.

The pipeline: state machine, not a pile of ifs

The opener is an explicit state machine, documented as a diagram in the source:

// datafusion/datasource-parquet/src/opener/mod.rs:342-373
      Start
        v
[LoadEncryption]?
        v
   PruneFile          <- file-level stats, before any footer is read
        v
  LoadMetadata
        v
 PrepareFilters      <- now the *file* schema is known
        v
PruneWithStatistics  <- row group min/max (zone maps)
        v
  LoadPageIndex?     <- skipped when all surviving groups are fully matched
        v
 LoadBloomFilters
        v
PruneWithBloomFilters
        v
  BuildStream

The ordering is the whole point. Each stage is cheaper than the next, and each stage can eliminate the work of the stages after it:

• PruneFile uses file-level statistics collected by the catalog (or partition values) and can skip the file without reading even the footer (opener/mod.rs:1021-1053). Because dynamic filters may have tightened since planning, this check is repeated at open time.
• PruneWithStatistics is pure CPU over the footer — no I/O at all.
• LoadPageIndex is I/O: the column/offset index lives outside the footer in modern Parquet files, so fetching it is a second round trip. DataFusion therefore asks for it with PageIndexPolicy::Skip when loading metadata and only pays for it if it will actually be used:

// datafusion/datasource-parquet/src/opener/mod.rs:1062-1063
let mut options =
    ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Skip);

• LoadBloomFilters is more I/O, and it only happens for row groups that statistics have not already resolved.

Whether the page index is worth loading is decided by should_load_page_index (opener/mod.rs:1370-1449): is the feature enabled, is there at least one row group that might contain filtered rows, and does the footer record a column index offset for a predicate column? If every surviving row group was proven “fully matched”, the page index cannot prune anything and its I/O is skipped.

Zone maps: the row-group min/max proof

Row-group pruning goes through RowGroupAccessPlanFilter (row_group_filter.rs). The interesting part is not the mechanics but the proof discipline. PruningPredicate (in datafusion/pruning) rewrites the predicate into an expression over statistics columns and evaluates it vectorised over all row groups at once:

// datafusion/datasource-parquet/src/row_group_filter.rs:367-380
match predicate.prune(&pruning_stats) {
    Ok(values) => {
        for (idx, &value) in row_group_indexes.iter().zip(values.iter()) {
            if !value {
                self.access_plan.skip(*idx);
                metrics.row_groups_pruned_statistics.add_pruned(1);
            } else { ... }
        }

The rewrite is documented in pruning_predicate.rs:244-252:

This is the “zone map” idea from database literature, implemented as an expression rewrite rather than a hand-written comparison per operator. The rewrite must be sound in the right direction: false means “no row in this container can be TRUE”, and NULL means “we don’t know”, which must be treated as “keep the container”.

The inverse trick: proving that everything matches

The cleverest bit in this file is not about skipping work — it is about skipping verification. If statistics can prove that every row in a row group satisfies the predicate, then the per-row filter does not have to be evaluated at all for that row group.

DataFusion does this by pruning with the logical inverse of the predicate:

// datafusion/datasource-parquet/src/row_group_filter.rs:440-448
for (i, &original_row_group_idx) in candidate_row_group_indices.iter().enumerate()
{
    // If the inverted predicate *also* prunes this row group (meaning inverted_values[i] is false),
    // it implies that *all* rows in this group satisfy the original predicate.
    if !inverted_values[i] {
        self.access_plan.mark_fully_matched(original_row_group_idx);

Building the inverse is subtle because SQL is three-valued. A row where the filter evaluates to NULL does not pass it, so the inverse must be NOT pred OR col IS NULL for every nullable column the predicate touches:

// datafusion/datasource-parquet/src/pruning.rs:43-62
let mut inverted_expr: Arc<dyn PhysicalExpr> =
    Arc::new(NotExpr::new(Arc::clone(predicate.orig_expr())));
// ... for each nullable referenced column:
inverted_expr = Arc::new(BinaryExpr::new(
    inverted_expr,
    Operator::Or,
    Arc::new(IsNullExpr::new(Arc::new(column))),
));

The whole inversion is refused if the pruning rewrite only preserved “can be TRUE” but not “is FALSE vs UNKNOWN” (pruning_predicate.rs:570, can_be_inverted_for_full_match). The payoff is real: a fully matched row group gets an empty RowFilter installed in the decoder (opener/mod.rs:1784-1795), so the per-row evaluation cost disappears for it entirely.

Page index: from row groups to pages

Row-group granularity is coarse. The Parquet page index gives per-page min/max (column index) plus per-page row offsets (offset index), which allows skipping row ranges inside a row group:

// datafusion/datasource-parquet/src/page_filter.rs:683-708
// Convert the information of which pages to skip into a RowSelection
// that describes the ranges of rows to skip.
for (i, &f) in values.iter().enumerate().skip(1) {
    if f == selected { sum_row += page_row_counts[i]; }
    else {
        let selector = if selected { RowSelector::select(sum_row) }
                       else { RowSelector::skip(sum_row) };
        vec.push(selector);
        ...

Page row counts come from the offset index’s first_row_index deltas (page_filter.rs:777-795) — the offset index does double duty as a page locator and a row-range index.

Two restrictions are worth noticing. First, the page index must be complete:

// datafusion/datasource-parquet/src/page_filter.rs:246-260
if !parquet_metadata.page_index().is_some_and(|page_index| page_index.is_complete()) {
    debug!("Can not prune pages due to lack of indexes. ...");
    return PagePruningResult::new(access_plan, 0, 0);
}

Partial indexes are useless here because you cannot know which pages were not indexed. Second, page pruning only works for single-column predicates (page_filter.rs:186-190): a page index is per column, so a predicate spanning two columns cannot be resolved from one column’s bounds.

Bloom filters: the last resort, and the false-positive contract

Bloom filters answer a different question: not “is the value in the range?” but “is the value definitely absent?” DataFusion loads them per row group, per predicate column, and feeds them into the same PruningPredicate machinery through a PruningStatistics adapter. The false-positive handling is a one-sided contract:

// datafusion/datasource-parquet/src/bloom_filter.rs:206-237
let known_not_present = values.iter()
    .map(|value| BloomFilterStatistics::check_scalar(...))
    .all(|v| !v);

let contains = if known_not_present { Some(false) } else {
    // Given the bloom filter is probabilistic, we can't be sure that
    // the row group actually contains the values. Return `None` to
    // indicate this uncertainty
    None
};

Some(false) means “definitely not present” → the row group can be pruned. None means “unknown” → keep the row group. There is deliberately no “probably present” answer, because a bloom filter cannot distinguish “present” from “false positive”. Unsupported scalar types return true from check_scalar (bloom_filter.rs:88-97), i.e. “assume it might be there” — a bloom filter over an unsupported type must never cause data loss. Bloom filter I/O is also skipped for row groups that statistics already proved fully matched (opener/mod.rs:1531-1536).

Range assignment: splitting a file for parallelism

One more pruning axis has nothing to do with predicates: a big file may be split into byte ranges so several tasks can read it in parallel. Each row group must land in exactly one range, so row_group_in_range (row_group_filter.rs:57-77) picks an offset inside the row group and tests it against the range. StartOffset uses the row group’s first data page; Midpoint uses the midpoint of the compressed size, matching parquet-java and spreading very large row groups more evenly across ranges.

The catch

Pruning is only as good as the metadata, and metadata lies in several ways.

• Truncated statistics. Min/max strings are truncated (64 bytes by default, datafusion.execution.parquet.statistics_truncate_length), so bounds are wider than reality. row_group_filter.rs:189-203 spells out the consequence: a fully-matching row group can no longer be proven fully matching, and the per-row filter has to stay.
• Untrusted byte-array ordering. The deprecated Parquet min/max fields use signed comparison, while Arrow compares strings as unsigned bytes. Using them for pruning can produce wrong answers, so DataFusion refuses to trust them:

// datafusion/datasource-parquet/src/metadata.rs:69-99
pub(crate) fn has_untrusted_min_max_order(...) -> bool {
    if matches!(column.sort_order(), SortOrder::UNDEFINED | SortOrder::INT96_TIMESTAMP) {
        return true;
    }
    requires_unsigned_byte_array_order(&column)
        && (column.sort_order() != SortOrder::UNSIGNED
            || column_orders.and_then(|orders| orders.get(parquet_column_index).copied())
                != Some(ColumnOrder::TYPE_DEFINED_ORDER(SortOrder::UNSIGNED)))
}

The handling is not “give up on the column”: mask_untrusted_byte_array_stats (row_group_filter.rs:554-572) nulls out the bounds of only the row groups that carry the deprecated signed statistics, keeping the modern bounds usable.

• Every pruning pass costs CPU. The bloom pass explicitly skips evaluation for row groups with no loaded filter, “a cost files written without bloom filters would pay for nothing” (row_group_filter.rs:481-487).

The design invariant that makes all of this safe is worth stating plainly: every pruning step is a proof of absence, and a failed or untrusted proof degenerates to “read it”. Nothing in the pipeline can make a query return fewer rows than it should. That is the property that lets DataFusion be aggressive — including the aggressive part where it decides that a row group is entirely matching and turns the filter off.