Parquet on Opensource internals

Writing Parquet Row Groups in Parallel: Column Chunks, Dictionary Pages and the Sort-Before-Write Trick

Jay Hubbert — Wed, 21 Oct 2026 09:01:00 +0000

TL;DR: The Parquet write path in DataFusion is a streaming, one-pass pipeline: batches flow in, get fanned out to per-column encoders, get sealed into row groups, and get concatenated into a file that is uploaded as it is produced. Row groups are sized by row count (and optionally by bytes), dictionary encoding and page statistics are configured once and inherited by the reader’s pruning machinery, and — the trick worth stealing — when the input plan is already sorted, the sort order is written into the Parquet footer so the reader gets the ordering for free instead of paying for a sort.

The shape of the write path

There are two entry points, and they are not the same thing.

plan_to_parquet is the simple one — one file per output partition, one writer task per partition, each task pulling batches from its partition’s stream and pushing them into an AsyncArrowWriter (writer.rs:41-72). The buffer it uses, objectstore_writer_buffer_size (10 MiB by default, datafusion/common/src/config.rs:1159), exists mostly so that a very large output file does not hit remote-endpoint limits on the size of a single upload. It is a transport knob, not a format knob.

The interesting path is ParquetSink (sink.rs), which is what COPY ... TO and INSERT INTO actually use. It receives a demultiplexed stream of batches tagged with the file they belong to (hive partitioning splits the output into many files), and it has two writer strategies.

Two writer strategies

The sequential path is parquet::arrow::AsyncArrowWriter: one file, one task, batches handed in order. It is used when allow_single_file_parallelism is false, or when content-defined chunking is enabled — CDC needs the chunker state that lives inside ArrowWriter and persists across row groups (sink.rs:304-308). The parallel path is DataFusion’s own code, and it is the interesting one.

Parallel column encoding inside one file

The idea: a Parquet row group is a set of independent column chunks. There is no reason to encode column a of row group 7 before column b of row group 7. So DataFusion spawns one task per leaf column and feeds them arrays through per-column channels:

 rust
 
 
// datafusion/datasource-parquet/src/sink.rs:630-645
for writer in col_writers.into_iter() {
 // Buffer size of this channel limits the number of arrays queued up for column level serialization
 let (send_array, receive_array) = mpsc::channel::<ArrowLeafColumn>(max_buffer_size);
 col_array_channels.push(send_array);

 let reservation =
 MemoryConsumer::new("ParquetSink(ArrowColumnWriter)").register(pool);
 let task = SpawnedTask::spawn(column_serializer_task(
 receive_array, writer, reservation, encoding_time.clone(),
 ));
 col_writer_tasks.push(task);
}

The channel depth is maximum_buffered_record_batches_per_stream (default 2, config.rs:1561-1566) — deliberately small, because this is a streaming plan and the memory budget is the scarce resource. The option’s comment says it plainly: the defaults are “tuned for minimum memory usage in a streaming execution plan”, and raising them “is likely worthwhile when writing out already in-memory data”.

A batch is then shredded into leaf arrays and pushed to the right channel:

 rust
 
 
// datafusion/datasource-parquet/src/sink.rs:684-695
let mut next_channel = 0;
for (array, field) in rb.columns().iter().zip(schema.fields()) {
 for c in compute_leaves(field, array)? {
 if col_array_channels[next_channel].send(c).await.is_err() {
 return Ok(());
 }
 next_channel += 1;
 }
}

compute_leaves is what makes this work for nested schemas: a List<Int32> column is one Arrow column but two Parquet leaves (the offsets and the elements), and each leaf needs its own encoder.

Row groups are sealed concurrently, concatenated in order

When a row group fills up, its column writers are closed and the finished column chunks are handed to a separate task, so the next row group can start encoding without waiting (sink.rs:777-817). The batch that straddles the boundary is split — rb.slice(0, rows_left) — and the remainder is carried into the next row group. That is why the row-group limit can be any number, not just a multiple of the batch size.

The finalizer joins the column tasks, closes each writer, and grows a memory reservation by the encoded size (sink.rs:712-727). The concatenator then writes them into the file, in the order the row groups were started, flushing to the object store when the shared buffer grows past a megabyte (sink.rs:859-876):

 rust
 
 
// datafusion/datasource-parquet/src/sink.rs:859-876
let mut rg_out = parquet_writer.next_row_group()?;
for chunk in serialized_columns {
 chunk.append_to_row_group(&mut rg_out)?;
 rg_reservation.free();

 let mut buff_to_flush = merged_buff.buffer.try_lock().unwrap();
 if buff_to_flush.len() > BUFFER_FLUSH_BYTES {
 object_store_writer.write_all(buff_to_flush.as_slice()).await?;
 buff_to_flush.clear();
 }
}
rg_out.close()?;

So the shape is: N column encoders running concurrently, R row groups in flight, and one writer that stitches the finished row groups into the file in order and ships bytes to the object store as it goes. The file is never materialized in memory, and the input is never read twice.

Row group sizing

Row group size is the single most consequential write-side decision, because it is the granularity at which the reader can prune. max_row_group_size defaults to 1M rows; max_row_group_bytes is None, and when set, bounds the row group by bytes instead, matching parquet.block.size in parquet-mr (config.rs:1494-1512).

The byte limit is the interesting one, and the reason it is None by default is a deliberate trade-off: the parallel writer ignores it. From the option’s own documentation: “Currently only honored when allow_single_file_parallelism is false; by default the parallel file writer ignores this limit.” The parallel writer computes its row-group boundary from max_row_group_row_count() alone (sink.rs:747-750), because bounding bytes in a design where row groups are encoded concurrently and asynchronously would require knowing the encoded size before the row group is finished — which is exactly the information that only exists after encoding.

Dictionary encoding and statistics: write-side decisions, read-side wins

The writer’s defaults come from ParquetOptions and are translated into WriterProperties in one place (datafusion/common/src/file_options/parquet_writer.rs:259-276): page size limit, write batch size, writer version, dictionary page size limit, statistics level, max row group rows and bytes, created_by, column index and statistics truncation lengths, data page row count limit, bloom filter switch, and the NDV-per-row-group switch.

The defaults that matter for a reader:

Option	Default	Reader-side consequence
`dictionary_enabled`	`true`	smaller column chunks; enables dictionary-aware reads
`dictionary_page_size_limit`	1 MiB	bounds the dictionary’s memory cost
`statistics_enabled`	`page`	the column index exists, so page-level pruning works
`statistics_truncate_length`	64	bounds are exact up to 64 bytes, then widened
`column_index_truncate_length`	64	same, for the page index
`data_pagesize_limit`	1 MiB	page granularity for pruning
`data_page_row_count_limit`	20 000	rows per page
`compression`	`zstd(3)`
`bloom_filter_on_write`	`false`	bloom pruning is unavailable unless you turn this on

That last row is the sharpest example of a write-time decision with a read-time payoff. DataFusion’s reader can prune row groups with bloom filters (row_group_filter.rs:460-508), but a bloom filter has to be in the file first, and bloom_filter_on_write defaults to false. Turning it on costs space and write time and buys row-group skipping for equality predicates — a trade that is worth it for selective high-cardinality columns and not worth it for low-cardinality ones (where the min/max statistics already do the job).

Similarly, statistics_truncate_length = 64 is the reason that a string column’s row-group bounds are “Alp”..“Alq” rather than exact, and the reason full-match inference can fail on long strings (row_group_filter.rs:189-203).

The sort-before-write trick

Here is the trick worth stealing. Parquet has a sorting_columns field in its metadata: it declares that the file’s rows are sorted by these columns. DataFusion writes it when the input plan is already ordered, and reads it back to skip the sort.

Writing it (file_format.rs:645-668):

 rust
 
 
// Convert ordering requirements to Parquet SortingColumns for file metadata
let sorting_columns = if let Some(ref requirements) = order_requirements {
 let ordering: LexOrdering = requirements.clone().into();
 let writer_schema = get_writer_schema(&conf);
 // In cases like `COPY (... ORDER BY ...) TO ...` the ORDER BY clause
 // may not be compatible with Parquet sorting columns (e.g. ordering on `random()`).
 // So if we cannot create a Parquet sorting column from the ordering requirement,
 // we skip setting sorting columns on the Parquet sink.
 lex_ordering_to_sorting_columns(&ordering, conf.output_schema(), &writer_schema)
 .ok()
 .filter(|columns| !columns.is_empty())
} else {
 None
};

The order_requirements are not invented by the writer — they come from the plan: input_exec.properties().output_ordering() (datafusion/core/src/physical_planner.rs:930-940).

So: if the plan that feeds the sink already has an ordering — because the user wrote COPY (SELECT ... ORDER BY x) TO ..., or because an upstream operator produces sorted output — the writer records it in the footer. It does not impose an order; it only documents one it was given. That distinction is what makes it safe: the metadata is a claim the writer can make because it saw the data in that order. The conversion is conservative in the right way: only plain column references can become sorting columns (metadata.rs:1145-1151).

Reading it back, ordering_from_parquet_metadata (metadata.rs:1209-1236) turns the footer’s sorting columns into a LexOrdering, which the source advertises as its output ordering — and that is what lets the optimizer delete the SortExec above the scan.

The read side has a second, related trick for files that are not declared sorted: reorder the row groups by their min statistics, so a TopK query finds good row groups first (access_plan.rs:727-749). Row groups are sorted lexicographically by the min values of the longest prefix of the sort order made of plain columns present in the file schema, with secondary keys breaking ties when the leading column’s min is equal across row groups — without them the reorder is a no-op on such files and the TopK dynamic filter converges only as fast as disk order allows.

This is the “sort before write” idea from the other direction: if you cannot afford to sort at write time, you can at least approximate the order at read time using statistics you already paid for, and let the TopK’s dynamic filter prune the rest.

The catch

The parallel writer bypasses ArrowWriter. Content-defined chunking is therefore disabled with it — the chunker state lives in ArrowWriter and persists across row groups (sink.rs:304-308). CDC forces the sequential path.
max_row_group_bytes does not work with parallel writing. Setting it and leaving allow_single_file_parallelism = true gives you a row group limit that is silently not applied. This is documented in the option, but it is the kind of footgun that costs an afternoon.
Ordering metadata is a promise, not a check. If the input plan claims an ordering it does not have, the footer will lie, and the reader will happily skip a sort it should have performed. DataFusion mitigates this by deriving the ordering from the plan rather than from the data, so the promise inherits the plan’s correctness.
Bloom filters cost space and write time for a benefit that only materialises when the reader also has bloom pruning enabled (datafusion.execution.parquet.bloom_filter_on_read).
Statistics truncation is a silent precision loss. 64 bytes is enough for most keys and not enough for long strings; the effect is that pruning gets weaker, never wrong — but “never wrong” here comes from the untrusted-stats machinery on the read side, which is a separate piece of machinery you now depend on.

The general lesson of the write path is that Parquet’s read performance is decided at write time. Row group size, dictionary encoding, statistics granularity, bloom filters, and sorting metadata are all write-side choices that the reader either cashes in or does not. DataFusion exposes all of them as configuration, which is the minimum you owe your future queries.

Polars Publishes a Join's Build Side as a Runtime Filter

Jay Hubbert — Wed, 21 Oct 2026 08:40:00 +0000

TL;DR: A hash join knows exactly which keys it holds, and the probe side usually does not need to look at rows whose keys cannot be in that set. Polars makes this work by turning the join key on the probe side into a DynamicPred — a predicate slot with no value at plan time — pushing it into the scan below the probe, and having the join fill the slot with the build side’s min/max range (and, when it pays, a bloom filter) once the hash table is built. The scan then skips whole row groups by their statistics and drops the rows outside the range.

The join runtime filter is the most interesting optimizer pass in Polars, because it is not really an optimizer pass at all. It does not rewrite a plan into a cheaper plan; it builds a communication channel between two operators that are far apart in the tree, and then fills that channel at run time with information that only exists once the data has been read.

The module that implements it explains itself better than I could:

text

 plan time run time
 ---------------- -------------
 join key on the probe side build side is hashed
 | |
 v v
 trace the key down min/max of the build keys
 to a parquet scan (+ a bloom filter, maybe)
 | |
 v |
 DynamicPred slot <--------- set ----------+
 (no value yet) (the range)
 |
 v
 parquet scan under the probe side:
 - skip row groups whose stats fall outside the range
 - drop rows whose value falls outside the range

 rust
 
 
//! Let a hash join tell the parquet scan under its probe side which keys the build
//! side holds.
//!
//! A join whose one side is known to be much smaller than a scan under its other
//! side gets that side forced as build side; one whose side is only estimated to be
//! much smaller gets it preferred. Every probe key that
//! reads a parquet scan column unchanged gets a dynamic predicate on that scan. The
//! join publishes the range of its build keys once the build is done. The scan
//! skips the row groups outside the range and drops the rows outside it. When the
//! build keys are estimated to be a small share of the scan's distinct keys the
//! join also publishes a bloom filter over them, which the scan probes per row
//! instead.

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:1

The plan-time side: finding a scan to talk to

The pass runs after the join build side has been chosen, and it starts by deciding whether there is anything to talk to at all:

 rust
 
 
pub(super) fn attach_join_runtime_filters(
 root: Node,
 ir_arena: &mut Arena<IR>,
 expr_arena: &mut Arena<AExpr>,
) {
 if !polars_config::config().join_runtime_filters() {
 return;
 }
 // Inputs before their join, so a join lower in a probe chain is forced before
 // an outer one tries to carry a predicate through it.
 let mut joins = Vec::new();
 let mut has_pruning_scan = false;
 let mut stack = vec![root];
 while let Some(node) = stack.pop() {
 let ir = ir_arena.get(node);
 match ir {
 IR::Join { .. } => joins.push(node),
 IR::Scan { scan_type, .. } => has_pruning_scan |= skips_batches(scan_type),
 _ => {},
 }
 ir.copy_inputs(&mut stack);
 }
 if !has_pruning_scan {
 return;
 }

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:79

Two early exits define the scope of the whole feature. The first is a config flag (POLARS_JOIN_RUNTIME_FILTERS, on by default). The second is has_pruning_scan: the only scans that can use a range are ones that skip batches by their statistics, which in practice means parquet scans with row-group statistics:

 rust
 
 
fn skips_batches(scan_type: &FileScanIR) -> bool {
 scan_type
 .flags()
 .contains(ScanFlags::SKIPS_BATCHES_BY_STATISTICS)
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:475

A CSV scan cannot use a range, so a plan with no parquet scan under the probe side is left completely alone.

Which joins qualify

Not every join can publish a filter. The constraints are in one function:

 rust
 
 
/// Whether the join may publish a filter or be crossed by one: an inner, semi or
/// anti equi join the streaming engine can run as a hash join that blocks its
/// probe side until the build is done. Sorted inputs may still make it a merge
/// join, which drops the filter. A key that may evaluate differently each time
/// gives no filter, as the join evaluates it again when it builds.
fn is_eligible_join(options: &JoinOptionsIR, expr_arena: &Arena<AExpr>) -> bool {
 let args = &options.args;
 let JoinTypeOptionsIR::Equi { on, .. } = &options.options else {
 return false;
 };
 !on.is_empty()
 && on.iter().all(|(left, right)| {
 !is_inherently_nondeterministic(left.node(), expr_arena)
 && !is_inherently_nondeterministic(right.node(), expr_arena)
 })
 && (args.how == JoinType::Inner || args.how.is_semi_anti())
 && args.maintain_order == MaintainOrderJoin::None
 && !args.nulls_equal
 && args.slice.is_none()
 && !args.validation.needs_checks()
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:334

Every clause is a correctness requirement:

Equi only. A range is only meaningful for equality keys.
Inner, semi or anti. A left or right outer join keeps unmatched rows from one side, so pruning the probe side would drop rows the join is supposed to emit.
No maintain_order. Order maintenance changes which side is read first, and the whole scheme depends on the probe side being read after the build.
No nulls_equal. The published range excludes nulls; a join that treats nulls as equal would need nulls in the range.
No slice. A slice takes the first n rows of the join output, and pruning the probe side would change which rows those are.
Deterministic keys. The join evaluates the key expression again when it builds, and the scan evaluates it when it reads. If the two evaluations could differ, the published range would describe a different set of keys than the hash table holds.

Semi and anti joins get a special rule, stated in the module docs:

 rust
 
 
//! A semi join publishes from either side and an anti join from its left side
//! only, as the rows of its right side that match no left key change nothing. A
//! semi join may prefer its right side on an estimate; a left side is only built
//! when forced, as building it keeps its rows. It is not forced when the right keys
//! are known to be much fewer than its estimated rows, as a right build keeps only
//! those.

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:30

Which side becomes the build

The pass picks the build side itself, and it does so from statistics rather than from a user hint. build_candidates produces, for each side, a forced candidate if the side has a hard row bound, and a preferred candidate from its estimate:

 rust
 
 
/// The ways a filtered side may be built from: forced when it has a bound, and
/// preferred on its estimate.
fn build_candidates(left: bool, stats: &NodeStats) -> Vec<BuildCandidate> {
 if stats.filtered >= stats.unfiltered {
 return Vec::new();
 }
 let forced = stats.max_rows().map(|rows| BuildCandidate {
 left,
 rows,
 forced: true,
 });
 let preferred = BuildCandidate {
 left,
 rows: stats.filtered,
 forced: false,
 };
 forced.into_iter().chain([preferred]).collect()
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:273

A side that is filtered below its unfiltered size is the only candidate — an unfiltered side has no interesting range, because its range is just the range of the whole column. The “lopsided” threshold is a factor of eight:

 rust
 
 
/// A side has to be this many times smaller to be picked as the build side.
pub(super) const LOPSIDED_FACTOR: f64 = 8.0;

crates/polars-plan/src/plans/optimizer/join_build_side.rs:14

and the candidate is only taken if the scan under the probe side is big enough for the pruning to be worth it:

 rust
 
 (!filters.is_empty() && *probe_rows >= LOPSIDED_FACTOR * side.rows).then_some(side)

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:173

Tracing the key to a scan

This is the part that makes the feature work: the join’s key on the probe side has to reach a scan that reads that column unchanged. If the scan applies a filter to the key column first, the values that reach the join are not the values in the file, and a range published from the build side would be wrong for the filtered values.

trace_probe_keys walks the probe input and collects the scans each key reaches:

 rust
 
 
fn trace_probe_keys(
 probe_input: Node,
 probe_keys: Vec<Option<PlSmallStr>>,
 ir_arena: &Arena<IR>,
 expr_arena: &mut Arena<AExpr>,
 scratch: &mut UnitVec<Node>,
) -> Vec<TracedKey> {
 let probe_schema = ir_arena.get(probe_input).schema(ir_arena).into_owned();
 let mut traced = Vec::new();
 for (key_idx, name) in probe_keys.into_iter().enumerate() {
 let Some(name) = name else { continue };
 if !probe_schema.get(&name).is_some_and(supports_runtime_range) {
 continue;
 }
 let column = expr_arena.add(AExpr::Column(name));
 let (dyn_node, pred) = new_dynamic_pred(column, expr_arena, None);
 let predicate = ExprIR::from_node(dyn_node, expr_arena);
 let origins = scan_origins(probe_input, predicate, ir_arena, expr_arena, scratch);

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:317

scan_origins is the actual traversal. It pushes a predicate down through the nodes a predicate can be pushed past, follows the column through a union into every input, and stops at anything that would change the column’s values:

 rust
 
 
 IR::SimpleProjection { .. }
 | IR::Filter { .. }
 | IR::Select { .. }
 | IR::HStack { .. } => {
 match push_past(
 node,
 &mut predicate,
 ir_arena,
 expr_arena,
 scratch,
 maintain_errors,
 ) {
 Ok(Some(input)) => node = input,
 _ => continue 'paths,
 }
 },
 // A slice over the union would take other rows once some are pruned.
 IR::Union { inputs, options } if options.slice.is_none() => {
 // Each input rewrites its own expression nodes; the filter they
 // refer to stays the one the join publishes.
 for &input in inputs.iter().rev() {
 let copy = deep_clone_ae(predicate.node(), expr_arena);
 paths.push((input, ExprIR::from_node(copy, expr_arena)));
 }
 continue 'paths;
 },

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:407

Two of the guards are worth reading closely. A Filter is passed through by push_past, which also rewrites the predicate so it refers to the same column after the filter — so the filter’s own restriction is preserved in the predicate that ends up on the scan. But a filter containing a window function is a barrier, because a window function is lowered to a group-by that reads the scan on its own:

 rust
 
 
 IR::Filter {
 predicate: filter, ..
 } if has_window(std::slice::from_ref(filter), expr_arena) => continue 'paths,

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:384

A union is followed into every input, and each input gets its own copy of the predicate expression — but they all refer to the same DynamicPred, so they all see the same published filter. A union with a slice is excluded, because pruning one input would shift the rows the slice takes.

Attaching the slot to the scan

Once a scan is found, the predicate is attached to it, combined with any predicate the scan already had:

 rust
 
 
fn attach_to_scan(
 scan: Node,
 predicate: ExprIR,
 ir_arena: &mut Arena<IR>,
 expr_arena: &mut Arena<AExpr>,
) {
 let IR::Scan {
 predicate: existing,
 predicate_file_skip_applied,
 ..
 } = ir_arena.get_mut(scan)
 else {
 unreachable!()
 };
 // Files are skipped by the scan predicate after these filters are attached.
 debug_assert!(predicate_file_skip_applied.is_none());
 *existing = Some(match existing.take() {
 None => predicate,
 Some(existing) => {
 let node = expr_arena.add(AExpr::BinaryExpr {
 left: existing.node(),
 op: Operator::And,
 right: predicate.node(),
 });
 ExprIR::from_node(node, expr_arena)
 },
 });
}

crates/polars-plan/src/plans/optimizer/join_runtime_filter.rs:503

The predicate is an ordinary expression in the plan — a DynamicPred function node wrapping a column reference — which means it flows through the normal expression machinery. Its value, though, is not in the plan. That is what DynamicPred is for:

 rust
 
 
/// A predicate over `node` whose value a producer sets at run time, evaluated
/// per row wherever it lands. A reader reads in `read_order` when it can.
pub fn new_dynamic_pred(
 node: Node,
 arena: &mut Arena<AExpr>,
 read_order: Option<ReadOrder>,
) -> (Node, DynamicPred) {
 let pred = DynamicPred::new(read_order);
 let function = IRFunctionExpr::DynamicPred {
 pred: pred.downgrade(),
 };
 let options = function.function_options();
 let aexpr = AExpr::Function {
 input: vec![ExprIR::from_node(node, arena)],
 function,
 options,
 };

 (arena.add(aexpr), pred)
}

crates/polars-plan/src/plans/optimizer/predicate_pushdown/dynamic.rs:246

The DynamicPred is a shared, mutable slot: an Arc<Inner> with an RwLock<Option<Arc<dyn PredicateExpr>>> and an AtomicBool that says whether the value has been set:

 rust
 
 
 pub fn set(&self, pred: Arc<dyn PredicateExpr>) {
 {
 let mut guard = self.inner.pred.write().unwrap();
 *guard = Some(pred);
 }
 self.inner.is_set.store(true, Ordering::Release);
 }

 pub fn is_set(&self) -> bool {
 self.inner.is_set.load(Ordering::Acquire)
 }

crates/polars-plan/src/plans/optimizer/predicate_pushdown/dynamic.rs:143

Before it is set, the predicate evaluates to “everything passes”, so the scan is correct whether or not the join ever publishes anything.

The run-time side: what the join publishes

The plan carries a list of these filters on the join node:

 rust
 
 
/// The build-side key at `key_idx` of the join's `on`, published through
/// `pred` for the probe side: its range, and a bloom filter over its values
/// when `bloom_keys` gives the number of distinct keys to size it for.
#[derive(Clone, Debug, PartialEq, Hash)]
#[cfg_attr(feature = "ir_serde", derive(Serialize, Deserialize))]
pub struct RuntimeFilter {
 pub key_idx: usize,
 pub pred: DynamicPred,
 pub bloom_keys: Option<usize>,
 /// The probe side's distinct keys, when the plan can estimate them.
 pub probe_distinct: Option<usize>,
}

crates/polars-plan/src/plans/options.rs:414

bloom_keys is the plan’s estimate of how many distinct keys the build side holds, and probe_distinct is the plan’s estimate of how many distinct keys the probe side has. Whether to build a bloom filter at all depends on the ratio:

 rust
 
 
/// Largest ratio of distinct build keys to distinct probe keys for which a
/// bloom filter is published.
pub const MAX_BUILD_PROBE_DISTINCT_RATIO: f64 = 0.3;

crates/polars-plan/src/plans/options.rs:402

A bloom filter over the build keys is only worth probing if the build keys are a small fraction of the probe’s distinct keys. If the build side has almost as many distinct keys as the probe side, the filter would reject almost nothing and only cost time.

At run time, RuntimeFilters owns one builder per filter. The builder tracks two things: the min/max of the key column, and (optionally) the hashes of the key values:

 rust
 
 
pub(super) struct KeyFilterBuilder {
 range: KeyRange,
 bloom: Option<BloomBuilder>,
}

crates/polars-stream/src/nodes/joins/runtime_filter.rs:219

The range is just the running min and max:

 rust
 
 
impl KeyRange {
 /// Widen the range to cover the non-null values of `column`.
 fn extend(&mut self, column: &Column) -> PolarsResult<()> {
 let min = column.min_reduce()?;
 let max = column.max_reduce()?;
 if !min.is_null() && !max.is_null() {
 self.merge(Self {
 bounds: Some((min, max)),
 });
 }
 Ok(())
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:270

and the published filter evaluates it per row with two comparisons:

 rust
 
 
impl PredicateExpr for KeyRange {
 /// Whether each value lies in the range. Nulls never do.
 fn evaluate(&self, columns: &[Column]) -> PolarsResult<Option<Column>> {
 let Some(bounds) = &self.bounds else {
 return Ok(None);
 };
 let column = columns[0].as_materialized_series();
 let Some((lo, hi)) = bounds_for(bounds, column.dtype()) else {
 return Ok(None);
 };
 let mask = (column.gt_eq(&lo)? & column.lt_eq(&hi)?).fill_null_with_values(false)?;
 Ok(Some(mask.into_column()))
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:488

Note fill_null_with_values(false): a null key never matches a join key, so it never matches the range.

The same filter also knows how to skip a whole batch of rows using the batch’s statistics:

 rust
 
 
 fn evaluate_stats(
 &self,
 min: &Column,
 max: &Column,
 _null_count: &Column,
 ) -> PolarsResult<Option<Column>> {
 let Some(bounds) = &self.bounds else {
 return Ok(Some(Column::new_scalar(
 min.name().clone(),
 Scalar::from(true),
 min.len(),
 )));
 };
 let min = min.as_materialized_series();
 let max = max.as_materialized_series();
 let Some((lo, hi)) = bounds_for(bounds, min.dtype()) else {
 return Ok(None);
 };
 // A batch is skipped when it lies entirely below or above the range. An
 // unknown statistic is null and settles nothing.
 let skip = max.lt(&lo)? | min.gt(&hi)?;
 Ok(Some(skip.fill_null_with_values(false)?.into_column()))
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:507

That is the row-group skip: if the batch’s maximum is below the range’s minimum, or its minimum is above the range’s maximum, no row in it can match, and the reader never decodes it.

When the filter is published

There are two moments at which the join can publish, and the distinction matters.

A forced build side is read completely before the probe side is opened, so its range is known before the scan even starts. A preferred build side is only preferred, so the join samples both sides at run time and decides. The sampling code makes the ordering explicit:

 rust
 
 
 fn try_transition_to_build(
 &mut self,
 recv: &[PortState],
 params: &mut EquiJoinParams,
 state: &StreamingExecutionState,
 spill_ctx: &MostRecentSpillContext,
 ) -> PolarsResult<Option<BuildState>> {
 if let Some(left) = self.only_side {
 let idx = if left { 0 } else { 1 };
 let len = self.len(left);
 if len >= params.sample_limit {
 if config::verbose() {
 eprintln!("preferred build side reached the sample limit, sampling both sides");
 }
 } else if recv[idx] == PortState::Done {
 if config::verbose() {
 eprintln!("preferred build side done with {len} rows, publishing its ranges");
 }
 self.publish_runtime_filters(left, params, state)?;

crates/polars-stream/src/nodes/joins/equi_join.rs:437

If the preferred side ends before it reaches the sample limit, it is complete, and its ranges are published before the other side is read at all. If it reaches the limit, both sides are sampled and the filter is only published if the preferred side ends up being the build side. The result is always exact; only the pruning is lost in the second case.

Once the build is done, the local builders are merged and published:

 rust
 
 
 /// Hand every build key to the runtime filters. Filters set from a sample
 /// keep their value, whichever side is built; filters of a side that was
 /// not built get a predicate that skips nothing.
 fn publish_runtime_filters(&mut self, params: &mut EquiJoinParams) {
 if !params.publishes_runtime_filters() {
 params.runtime_filters.publish_nothing();
 return;
 }
 let locals = self
 .local_builders
 .iter_mut()
 .map(|l| std::mem::take(&mut l.key_filters));
 params.runtime_filters.publish_merged(locals);
 }

crates/polars-stream/src/nodes/joins/equi_join.rs:761

publish_nothing is the important case: if the join ends up building the other side, the filter that was prepared for this side must be filled with something that rejects nothing, or the scan would drop rows it should keep.

The keys themselves are collected while the build side is read:

 rust
 
 
 let locals = RAYON.install(|| {
 morsels
 .par_chunks(chunk_len)
 .map(|chunk| {
 let mut builders = self.new_builders();
 for morsel in chunk {
 let df = morsel.df_blocking();
 let keys = ASYNC.block_on(select_key_columns(&df, key_selectors, state))?;
 self.extend(&keys, &mut builders)?;
 }
 PolarsResult::Ok(builders)
 })
 .collect::<PolarsResult<Vec<_>>>()
 })?;
 self.publish_merged(locals);

crates/polars-stream/src/nodes/joins/runtime_filter.rs:109

The bloom filter

The bloom filter is the more expensive half of the published filter, and it has a budget. Each builder buffers key hashes until the total number of build rows exceeds a budget, at which point they all switch to a shared bloom filter of the size the plan estimated:

 rust
 
 
/// Bits per key a bloom filter is sized for.
const BLOOM_BITS_PER_KEY: usize = 8;
/// A bloom filter with fewer bits per distinct key than this is not published.
const BLOOM_MIN_BITS_PER_KEY: usize = 4;
/// Smallest bloom filter that is built.
const BLOOM_MIN_BYTES: usize = 64 << 10;
/// Largest bloom filter that is published.
const BLOOM_MAX_BYTES: usize = 32 << 20;

crates/polars-stream/src/nodes/joins/runtime_filter.rs:30

The point of the budget is that a bloom filter sized from the actual distinct keys is better than one sized from the plan’s estimate, but only if you can afford to hold the hashes until you know the count. The CardinalitySketch tracks the distinct count cheaply while the hashes are buffered, so the filter can be sized exactly at publication time:

 rust
 
 
 let distinct = sketch.estimate();
 let weak = spec
 .probe_distinct
 .is_some_and(|probe| distinct as f64 > probe as f64 * MAX_BUILD_PROBE_DISTINCT_RATIO);
 let bloom = if weak {
 None
 } else if planned {

crates/polars-stream/src/nodes/joins/runtime_filter.rs:328

and if the real distinct count turns out to be too large relative to the probe side’s, the bloom filter is dropped at the last moment rather than published and probed for nothing.

The published filter then probes the bloom per row instead of the range:

 rust
 
 
 fn evaluate(&self, columns: &[Column]) -> PolarsResult<Option<Column>> {
 let Some(KeyBloom { spec, bloom }) = &self.bloom else {
 return self.range.evaluate(columns);
 };
 let column = &columns[0];
 if column.dtype() != &spec.dtype {
 return Ok(None);
 }
 let mut mask = BitmapBuilder::with_capacity(column.len());
 spec.hash_keys(column).for_each_hash(|_, hash| {
 mask.push(hash.is_some_and(|hash| bloom.contains(hash)));
 });
 Ok(Some(
 BooleanChunked::from_bitmap(column.name().clone(), mask.freeze()).into_column(),
 ))
 }

crates/polars-stream/src/nodes/joins/runtime_filter.rs:402

The bloom filter is in addition to the range, not instead of it: the range is what allows whole row groups to be skipped, the bloom is what catches the rows inside the range that are not in the build side.

The consumer: the scan

The predicate lands on the scan as an ordinary expression, and the expression engine lowers it twice: once into a per-row filter, and once into a statistics-level skip.

The per-row path is DynamicPred:

 rust
 
 
 F::DynamicPred { pred, .. } => {
 map_as_slice!(misc::dynamic_pred, &pred)
 },

crates/polars-expr/src/dispatch/mod.rs:571

and the statistics path is DynamicSkipBatch, which the skip-batch lowering derives from it:

 rust
 
 
 IRFunctionExpr::DynamicPred { pred, .. } => {
 let target = resolve_stat_target(input[0].node(), arena, schema)?;
 let dtype = target_leaf_dtype(&target, schema)?;
 if !supports_runtime_range(dtype) {
 return None;
 }
 let function = IRFunctionExpr::DynamicSkipBatch { pred: pred.clone() };
 let options = function.function_options();
 let input = [
 target.min(arena),
 target.max(arena),
 target.null_count(arena),
 ]
 .into_iter()
 .map(|b| ExprIR::from_node(b.node(), arena))
 .collect();

crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs:795

The dtype restriction is explicit:

 rust
 
 
/// Whether a runtime key range may skip batches of a column with this dtype.
///
/// The range is compared against the batch's min/max with polars' ordering, so the
/// type must be one whose parquet statistics are ordered the same way. Floats are out
/// because the statistics leave out NaN; 128-bit integers because their statistics
/// are byte-wise.
pub(crate) fn supports_runtime_range(dtype: &DataType) -> bool {
 use DataType as D;
 match dtype {
 D::Boolean
 | D::UInt8
 | D::UInt16
 | D::UInt32
 | D::UInt64
 | D::Int8
 | D::Int16
 | D::Int32
 | D::Int64
 | D::String
 | D::Binary
 | D::Date
 | D::Datetime(..)
 | D::Duration(_)
 | D::Time => true,
 #[cfg(feature = "dtype-decimal")]
 D::Decimal(..) => true,
 _ => false,
 }
}

crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs:230

Floats are excluded because parquet’s min/max statistics leave out NaN, and NaN compares strangely; 128-bit integers are excluded because their statistics are byte-wise rather than numeric.

The scan side of the contract is a small enum:

 rust
 
 
pub enum RuntimeRange {
 /// Not published yet. Every batch is kept; a later file may see a range.
 Pending,
 /// Never published. Every batch is kept.
 Disabled,
 /// No value can match. Every batch is skipped.
 Empty,
 /// Only values in `lo..=hi` can match, a missing bound does not limit that
 /// side. Nulls match when `nulls_match`.
 Range {
 lo: Option<Scalar>,
 hi: Option<Scalar>,
 nulls_match: bool,
 },
}

crates/polars-io/src/predicates.rs:509

Pending and Disabled are different on purpose: Pending means “not yet, but maybe later”, so a reader keeps every batch of the current file but checks again for the next one; Disabled means “never”. The parquet reader resolves the range against each file’s statistics and skips the row groups that fall outside it.

One honest limitation

The runtime filter is a streaming engine feature. runtime_filters lives on JoinOptionsIR in polars-plan, but the only consumer is polars-stream: the physical plan nodes that read it are EquiJoin and SemiAntiJoin in crates/polars-stream/src/physical_plan/mod.rs, and crates/polars-mem-engine never looks at the field. So on the classic in-memory engine the optimizer pass still runs and still rewrites the plan, but nothing fills the slot and nothing reads it — the predicate evaluates to “everything passes”.

The bottom line

The mechanism is a shared slot, not a plan rewrite. DynamicPred is a predicate with no value at plan time; the join fills it at run time.
Eligibility is strict. Inner, semi or anti equi joins, no order maintenance, no nulls-equal, no slice, deterministic keys, and a scan that skips batches by statistics. Everything else is left alone.
The trace is the hard part. A key only qualifies if it reaches a scan unchanged; filters are pushed past (and the predicate rewritten to match), window functions and sliced unions are barriers, and a union’s inputs all share one published filter.
The build side is chosen from statistics, forced when there is a hard row bound, preferred otherwise, and only when the probe-side scan is at least eight times bigger.
What is published is a range and, conditionally, a bloom filter. The range skips row groups by statistics and drops rows outside it; the bloom filter, published only when the build keys are under 30% of the probe’s distinct keys, catches the rows inside the range that are not in the build side.
It is a streaming-engine feature. The in-memory engine ignores it.

For a user the lever is POLARS_JOIN_RUNTIME_FILTERS (on by default), and the shape of the data is what decides whether it pays: a join against a parquet scan with a very selective small build side is exactly the case this was built for.

From SQL WHERE to Parquet Row Filter: How DataFusion Decides What It Can Push Down (and What It Must Keep)

Jay Hubbert — Wed, 21 Oct 2026 08:13:00 +0000

TL;DR: WHERE x > 5 reaches the Parquet reader as two independent questions: “can statistics prove this is false here?” (pruning) and “can the decoder evaluate this while it decodes?” (row filter). The support matrices for those two questions are different, non-overlapping, and both are one-sided: unsupported means “keep the data”, never “drop it”. A conjunct that neither system can handle stays in the physical plan as a residual filter, because dropping it would silently change the answer.

One predicate, two questions

When a FilterExec above a Parquet scan pushes its predicate down, the predicate arrives at the source as a single PhysicalExpr. From there it has to serve two masters:

 text
 
 
 WHERE a = 1 AND b = 2 AND c % 2 = 0
 │
 ┌────────┴─────────┐
 ▼ ▼
 statistics pruning decoder pushdown
 "can no row in this "can the decoder evaluate
 row group match?" this while decoding?"
 │ │
 ▼ ▼
 PruningPredicate RowFilter (ArrowPredicate)

The two questions have different answers for the same expression, and conflating them is the classic source of bugs in this area. c % 2 = 0 is a fine row filter (the decoder can compute a modulo) but is useless for statistics pruning (no range of c rules it out). s['x'] > 3 is fine for the decoder but not for statistics, because Parquet statistics for nested fields are not wired into the pruning predicate rewrite yet.

Question 1: which predicates can prune row groups?

Statistics pruning is implemented in the datafusion/pruning crate. The mechanism is a rewrite: the predicate is transformed into an expression over statistics columns (x_min, x_max, x_null_count, x_row_count), which is then evaluated against a record batch of statistics — one row per container (row group, file, …).

The documented rewrite table (datafusion/pruning/src/pruning_predicate.rs:244-252):

Original predicate	Rewritten predicate
`x = 5`	`x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max)`
`x < 5`	`x_null_count != x_row_count AND (x_min < 5)`
`x = 5 AND y = 10`	the two rewrites ANDed
`x IS NULL`	`x_null_count > 0`
`x IS NOT NULL`	`x_null_count != row_count`
`CAST(x as int) = 5`	`x_null_count != x_row_count AND (CAST(x_min as int) <= 5 AND 5 <= CAST(x_max as int))`

Everything else is handled by a rewrite function per expression type, and the entry point for a comparison is rewrite_expr_to_prunable (pruning_predicate.rs:1243-1318). Its rules: the left side must be a monotone-ish column expression and the right side a literal. The column expression may be a column, a cast of a column, a negated column, or a negated literal comparison. The rewrite even normalizes -col > 10 into col < -10 so that only the monotone direction has to be handled, and refuses Not over anything other than an equality-family operator (pruning_predicate.rs:1296-1316).

On top of that, build_predicate_expression (pruning_predicate.rs:1780-1990) recognises a specific set of node types before falling through to the comparison rewrite: IS NULL / IS NOT NULL on a column, a bare column (used for NOT col), IN (...) lists, a HashTableLookupExpr (a semi-join turned into a lookup), boolean CASE expressions (but only with an ELSE branch), case-sensitive LIKE / NOT LIKE, and AND / OR of the above.

`LIKE` becomes a range

The LIKE handling is a nice piece of engineering: a prefix match is exactly a range, so col LIKE 'foo%' is rewritten into col_min <= 'foo' AND 'foo' <= col_max, with the upper bound computed by incrementing the last UTF-8 byte of the prefix (pruning_predicate.rs:2255-2291):

 rust
 
 
// datafusion/pruning/src/pruning_predicate.rs:2276-2286
let (decoded_prefix, rest) = split_constant_prefix(s);
let has_wildcard = !rest.is_empty();
if has_wildcard && decoded_prefix.is_empty() {
 // there's no filtering we could possibly do, return None and have this be handled by the unhandled hook
 return None;
}
let (lower_bound, upper_bound) = if has_wildcard {
 let incremented_prefix = increment_utf8(&decoded_prefix)?;

Escaped wildcards are handled too ('foo\_%' bounds at 'foo_'), and a pattern with no usable prefix ('%foo') degenerates to “no filtering possible”. ILIKE is explicitly not supported — the TODO says it would need lower()/upper() bounds, which the statistics rewrite cannot express (pruning_predicate.rs:2266-2267).

`IN` lists and the size cap

IN lists are rewritten into a disjunction of equalities, but only up to a cap, documented in the builder (pruning_predicate.rs:462-484): lists up to min(20, C) values use the per-value rewrite; lists between 20 and the configured cap C use a compact pruning expression when the values are ordered literals, and fall back to the per-value rewrite otherwise; lists longer than C hit the unhandled-predicate hook, i.e. “keep the container”.

The default cap is 20 (MAX_IN_LIST_SIZE, pruning_predicate.rs:1697), and query engines wire it from datafusion.execution.parquet.max_in_list_size. The reason for a cap is not paranoia about the list but about the size of the rewritten expression: an IN list of 10,000 values becomes a tree of 10,000 comparisons, evaluated once per row group, over statistics columns. Above the cap, the predicate simply stops pruning — exactly the “keep the container” behaviour, and therefore safe.

The unhandled hook: the constant `true`

Anything the rewriter cannot handle is passed to an UnhandledPredicateHook. The default implementation returns a literal true (pruning_predicate.rs:587-604) — meaning “there may be matching rows”, i.e. keep the container. This is the single most important safety property of the whole design: an expression the rewriter does not understand becomes a proof of nothing.

`AND` is forgiving, `OR` is not

Partial support is allowed, but only in one direction. An AND can use whichever conjuncts it understands, because a row must satisfy all of them. An OR cannot: if one branch is unprunable, the disjunction might be true because of that branch.

The rewriter encodes this structurally: AND/OR combine the rewritten children, and simplification collapses true AND x to x and true OR x to true — i.e. an unprunable branch under OR poisons the whole disjunction back to “keep everything” (pruning_predicate.rs:1934-1970). The behaviour is visible in DataFusion’s own tests: c1 > 15 AND (c2 % 2 = 0) still prunes row group 1, while c1 > 15 OR (c2 % 2 = 0) prunes nothing (row_group_filter.rs:1059-1121).

Question 2: which predicates can the decoder evaluate?

The decoder-level support matrix is a completely separate implementation, living in projection_read_plan.rs (PushdownChecker) and supported_predicates.rs. Its rules:

primitive columns: fine,
list columns: only with a registered list predicate (array_has, array_has_all, array_has_any) or a null check,
struct columns: only via get_field down to a primitive leaf,
columns absent from the file: not pushable,
UDFs that require a rewrite: not pushable (has_unpushable_udfs).

The list-predicate registry is tiny and explicit:

 rust
 
 
// datafusion/datasource-parquet/src/supported_predicates.rs:81-86
fn is_supported_scalar_function(expr: &dyn PhysicalExpr) -> bool {
 scalar_function_name(expr).is_some_and(|name| {
 // Registry of verified array functions
 matches!(name, "array_has" | "array_has_all" | "array_has_any")
 })
}

Expression types defined in external crates cannot implement DataFusion’s trait directly (Rust orphan rules), so the trait has a blanket implementation that delegates to this name-based registry. The registry is a proof obligation list — each entry is a function whose semantics have been verified to be evaluable on a decoded list column.

The residual filter: why “keep it” is mandatory

Neither of the two systems is allowed to drop a conjunct. If a conjunct cannot be evaluated during decoding, it must still be evaluated somewhere, or the query returns rows that do not satisfy the predicate. DataFusion encodes this in the filter pushdown protocol:

 rust
 
 
// datafusion/datasource-parquet/src/source.rs:853-872
let filters: Vec<PushedDownPredicate> = filters.into_iter()
 .map(|filter| {
 if can_expr_be_pushed_down_with_schemas(&filter, pushable_schema) {
 PushedDownPredicate::supported(filter)
 } else {
 PushedDownPredicate::unsupported(filter)
 }
 })
 .collect();

PushedDown::No means “I did not evaluate this; you must”. The parent FilterExec stays in the plan and applies it to the batches the scan produces. (The same holds when pushdown is disabled entirely: the predicate is still used for statistics pruning, but the source reports PushedDown::No for everything so the parent keeps its filter, source.rs:880-896.)

There is a second, subtler reason to keep the residual: the source’s claims about its own output must be true. FileSource distinguishes filter() (the predicate the source uses) from exact_filter() (the part of it that every output row is guaranteed to satisfy):

 rust
 
 
// datafusion/datasource/src/file.rs:113-128
/// Returns the part of [`Self::filter`] that every output row is
/// guaranteed to satisfy.
///
/// [`FileScanConfig`] derives equivalence properties (constants and equal
/// columns) from this filter. A filter that the source uses only to skip
/// data (for example, row group or page pruning with statistics) does not
/// remove every non-matching row, so it must not be returned here.

If the source claimed the whole predicate as exact, the optimizer would be allowed to conclude that c = 5 implies c is constant, and would rewrite the plan accordingly. If the predicate was only used for pruning, that conclusion is false and the rewritten plan returns wrong rows. So exact_filter returns only the conjuncts that were genuinely evaluated during decoding:

 rust
 
 
// datafusion/datasource-parquet/src/source.rs:682-694
fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
 if !self.pushdown_filters() { return None; }
 let predicate = self.predicate.as_ref()?;
 let pushable_schema = self.table_schema.schema_without_virtual_columns();
 let exact = split_conjunction(predicate)
 .into_iter()
 .filter(|expr| can_expr_be_pushed_down_with_schemas(expr, pushable_schema))
 .cloned()
 .collect::<Vec<_>>();
 (!exact.is_empty()).then(|| conjunction(exact))
}

The same discipline shows up inside the pruning rewrite itself. Some rewrites preserve only “which rows can be TRUE” but lose the distinction between FALSE and UNKNOWN — for example NOT IN with a NULL member, or a CASE without an ELSE. The rewriter flags those with has_filter_semantics_only (pruning_predicate.rs:390-396), and such a predicate is fine for pruning (it only needs “can it be true?”), but useless for proving that every row matches — so the inversion that powers full-match inference is refused for it (pruning_predicate.rs:570, can_be_inverted_for_full_match).

The catch

Support is per expression, not per column. A column can be pushable in one predicate and not in another: s['v'] > 5 is pushable, s IS NOT NULL is not; array_has_all(tags, ...) is pushable, tags[1] = 'x' may not be. Any classification logic that works on columns rather than expressions is wrong.
Schema adaptation can change the answer. The pushdown check runs against the physical file schema after casts have been inserted, which is why pushdown_columns takes an allow_struct_casts flag: “Struct casts are accepted only after schema adaptation, not while planning against the table schema” (row_filter.rs:247-249).
Three-valued logic is not optional. The rewrite table above shows x_null_count != x_row_count in every equality. Drop it and a column that is entirely NULL would look like it might contain 5.
The unhandled hook hides bugs. Because an unsupported predicate silently becomes “keep everything”, a regression in the rewriter shows up as a performance regression, not a correctness one. DataFusion mitigates this with counters (predicate_creation_errors, predicate_evaluation_errors) and with PruningPredicateBuilder’s try_build variant that surfaces the error instead of swallowing it.
Unsupported does not mean free. Every unsupported conjunct still costs a residual FilterExec over the output batches — the cheapest place to evaluate it, but not free, and the reason pushdown is off by default.

The through-line: every classification decision in this pipeline is optimistic only about performance and pessimistic about correctness. Supported means “I can prove this or evaluate this”; unsupported means “keep the data and keep the filter”. A system that errs in the other direction would be faster and wrong, and no amount of benchmarking would tell you.

Polars Prunes Parquet Row Groups With Chunk Statistics, Not the Page Index

Jay Hubbert — Wed, 21 Oct 2026 07:10:00 +0000

TL;DR: Polars prunes Parquet row groups with the column chunk statistics that live in the file footer, not with the page index. Those statistics are decoded into a tiny DataFrame with <col>_min, <col>_max and <col>_nc columns, the user’s predicate is evaluated against that DataFrame, and the row groups whose bit is clear are never fetched. Polars writes the page index (column index and offset index) when it writes a file with statistics enabled, but its own reader never reads it back: page-level skipping is not implemented.

Every reader of a Parquet file faces the same question: the predicate says ts > 2026-01-01, which of the row groups in this file can be thrown away without touching the bytes? The answer has to come from somewhere, and the only metadata a reader is guaranteed to have is the footer.

Before diving in, one correction to the usual framing. Polars’ vendored Parquet implementation does write the page index — the column index and the offset index, the structures that hold per-page min/max and per-page byte offsets. If you write a Parquet file with Polars (with statistics enabled) and read it with DuckDB, Spark or DataFusion, those readers can skip pages inside a row group. Polars’ own reader does not do that. It skips whole row groups, using the statistics attached to each column chunk. The distinction matters: a row group is typically 100k+ rows, a page is typically a few thousand, so page-level skipping is a strictly finer-grained optimisation that Polars writes the data for but does not yet consume.

Which reader is the real one

The Parquet code lives in two crates. crates/polars-parquet holds the deserializers, the statistics decoding, and a vendored copy of the parquet crate itself (crates/polars-parquet/src/parquet/, with its own thrift decoder and page reader). The plan — which row groups, which columns, in what order — lives in the streaming engine, in crates/polars-stream/src/nodes/io_sources/parquet/. That split matters: the plan is where the pruning decision is made, and it is not in the crate you would guess from the name.

The plan is built in init_morsel_distributor, in the prefetch task, before a single byte of column data is read (crates/polars-stream/src/nodes/io_sources/parquet/init.rs:197):

 rust
 
 
 let has_row_index = row_index.is_some();
 let row_group_mask = calculate_row_group_pred_pushdown_skip_mask(
 row_group_slice.clone(),
 use_statistics,
 predicate.as_ref(),
 &metadata,
 projected_arrow_fields.clone(),
 row_index,
 verbose,
 task_metrics.as_deref(),
 runtime_range_skipper.as_deref(),
 )
 .await?;

 let mut row_groups = row_groups_to_read(
 &metadata,
 row_group_slice,
 row_group_mask.as_ref(),
 row_offset,
 slice_range,
 );

row_group_slice is the result of slice pushdown: LIMIT 10 over a file with 200 row groups narrows the slice to the one row group that contains row 10, so the statistics machinery never even looks at the other 199. row_group_mask is the result of statistics pushdown: a bitmap with one bit per row group, set where the row group cannot match. row_groups_to_read turns the two into a deque of row groups to fetch (crates/polars-stream/src/nodes/io_sources/parquet/row_group_data_fetch.rs:65):

 rust
 
 
/// The row groups of `row_group_slice` that `row_group_mask` keeps, in file order. The
/// first starts at `row_offset` and `slice_range` is relative to it.
pub(super) fn row_groups_to_read(
 metadata: &FileMetadata,
 row_group_slice: Range<usize>,
 row_group_mask: Option<&Bitmap>,
 mut row_offset: usize,
 mut slice_range: Option<Range<usize>>,
) -> VecDeque<RowGroupToRead> {
 let mut row_groups = VecDeque::with_capacity(row_group_slice.len());
 for (i, idx) in row_group_slice.enumerate() {
 let num_rows = metadata.row_groups[idx].num_rows();
 let slice = slice_range.as_mut().map(|slice_range| {
 let rg_row_start = slice_range.start;
 let rg_row_end = slice_range.end.min(num_rows);

 *slice_range = slice_range.start.saturating_sub(num_rows)
 ..slice_range.end.saturating_sub(num_rows);

 (rg_row_start, rg_row_end - rg_row_start)
 });
 if !row_group_mask.is_some_and(|mask| mask.get_bit(i)) {
 row_groups.push_back(RowGroupToRead {
 idx,
 row_offset,
 slice,
 });
 }
 row_offset = row_offset.saturating_add(num_rows);
 }
 row_groups
}

Note the mask semantics: a set bit means “skip”. The function keeps the row groups whose bit is clear, and accumulates row_offset over the skipped ones too, because the row offset of a row group is its position in the file, not in the result set. That matters for with_row_index.

The columns in the plan

The other half of the plan is which columns to read, and it is a separate mechanism from the row-group pruning. Projection pushdown has already decided which Arrow fields the query needs; the reader then works out, per row group, which byte ranges those fields live in, and fetches only those (crates/polars-stream/src/nodes/io_sources/parquet/row_group_data_fetch.rs:406):

 rust
 
 
fn get_row_group_byte_ranges_for_projection<'a>(
 row_group_metadata: &'a RowGroupMetadata,
 columns: &'a mut dyn Iterator<Item = &PlSmallStr>,
) -> impl Iterator<Item = ParquetResult<std::ops::Range<usize>>> + 'a {
 columns.flat_map(|col_name| {
 row_group_metadata
 .columns_under_root_iter(col_name)
 // `Option::into_iter` so that we return an empty iterator for the
 // `allow_missing_columns` case
 .into_iter()
 .flatten()
 .map(|col| {
 let byte_range = col.byte_range()?;
 Ok(byte_range.start as usize..byte_range.end as usize)
 })
 })
}

columns_under_root_iter is what makes nested schemas work: a projected struct field is not one column but several leaf columns, and the reader has to walk down to the leaves to find their byte ranges. Empty ranges — chunks without pages — are dropped, because they need no I/O.

This is also where the “full projection” shortcut lives: if the query reads every column, the reader skips the per-column walk and just takes the row group’s whole byte range.

The statistics become a DataFrame

Here is the part I find elegant. Polars does not implement a separate “evaluate predicate against min/max bounds” interpreter. Instead it turns the per-row-group statistics into a small DataFrame — one row per row group, one column per statistic — and then runs the user’s own predicate on that DataFrame (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:184):

 rust
 
 
 for projection in projected_arrow_fields.iter() {
 let c = projection.output_name();

 if !skip_batch_columns.contains(c) {
 continue;
 }

 let mut statistics =
 load_parquet_column_statistics(&metadata, row_group_slice.clone(), projection)?;

 // Note: Order is important here. We re-use the transform for the output column, meaning
 // that it may set the column name.
 statistics.min = projection.apply_transform(statistics.min)?;
 statistics.max = projection.apply_transform(statistics.max)?;
 // The null counts of struct fields have the file's field layout. The transform may
 // reorder, rename or insert fields (e.g. by field ID), and casts the leaves, so it
 // does not apply to them: use unknown null counts.
 if let ArrowFieldProjection::Mapped { output_dtype, .. } = projection
 && matches!(projection.arrow_field().dtype(), ArrowDataType::Struct(_))
 {
 statistics.null_count = Column::full_null(
 PlSmallStr::EMPTY,
 statistics.null_count.len(),
 &null_count_dtype(output_dtype),
 );
 }

 let statistics = statistics.with_base_column_name(c);

 columns.extend([statistics.min, statistics.max, statistics.null_count]);
 }

 if let Some(row_index) = row_index {
 let statistics = build_row_index_statistics(&row_index, row_groups_slice)
 .with_base_column_name(&row_index.name);

 columns.extend([statistics.min, statistics.max, statistics.null_count]);
 }

 let statistics_df = DataFrame::new(num_row_groups, columns)?;

 sbp.evaluate_with_stat_df(&statistics_df)

So a filter like pl.col("ts") > cutoff becomes, internally, a filter over a DataFrame with columns ts_min, ts_max and ts_nc. The predicate compiler already knows how to rewrite ts > cutoff into ts_min > cutoff; the skip batch predicate is that rewrite, evaluated over statistics. One code path for “filter a frame” and “filter a set of row groups”, and no second expression language to maintain.

Two details are worth pulling out. The first is the len column: the number of rows in each row group is pushed into the statistics frame as a plain column, so a predicate like pl.len() > 1000 prunes row groups too, without any special case (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:167):

 rust
 
 
 let mut columns = Vec::with_capacity(1 + skip_batch_columns.len() * 3);

 let lengths: Vec<IdxSize> = row_groups_slice
 .iter()
 .map(|rg| rg.num_rows() as IdxSize)
 .collect();

 columns.push(Column::new("len".into(), lengths));

The second is that the statistics are pushed through the projection’s transform before the predicate sees them. A projection can rename a column, reorder struct fields, or cast a leaf (projection.apply_transform), and the predicate was compiled against the transformed column. Applying the transform to the statistics as well is what keeps the two in agreement.

The columns are named <col>_min, <col>_max and <col>_nc (crates/polars-io/src/predicates.rs:363):

 rust
 
 
 columns.extend([
 Column::new_scalar(format_pl_smallstr!("{col}_min"), min, 1),
 Column::new_scalar(format_pl_smallstr!("{col}_max"), max, 1),
 Column::new_scalar(format_pl_smallstr!("{col}_nc"), nc, 1),
 ]);

The result is a bitmap, one bit per row group, and the verbose log prints the savings (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:130):

 rust
 
 
 if verbose && (mask.is_some() || runtime_range_skipper.is_some()) {
 let num_row_groups = row_group_slice.len();
 eprintln!(
 "[ParquetFileReader]: Predicate pushdown: \
 reading {} / {} row groups",
 mask.as_ref().map_or(num_row_groups, |m| m.unset_bits()),
 num_row_groups,
 );
 }

Where the numbers come from

The footer is kept in memory as a buffer, and the statistics are stored as byte ranges into it. Resolving a bound is therefore a slice, not a copy (crates/polars-parquet/src/parquet/metadata/column_chunk_metadata.rs:111):

 rust
 
 
 /// The plain-encoded min and max of the chunk, borrowed from `footer_buf`;
 /// `None` when the chunk has no statistics.
 pub fn raw_bounds<'a>(&self, footer_buf: &'a [u8]) -> Option<RawBounds<'a>> {
 let stats = self.column_chunk.meta_data.as_ref()?.statistics.as_ref()?;
 Some(RawBounds {
 min: stats.min_value.map(|range| range.resolve(footer_buf)),
 max: stats.max_value.map(|range| range.resolve(footer_buf)),
 min_is_exact: stats.is_min_value_exact != Some(false),
 max_is_exact: stats.is_max_value_exact != Some(false),
 })
 }

The is_exact flags are a nice touch: a writer may record a truncated or approximate bound, and an inexact bound is worthless for pruning (it may exclude rows that do match). Polars validates the bounds and then throws away the inexact ones rather than pruning on them.

Before a bound can be compared, the file’s declared sort order has to match the order Polars compares with — an unsigned column whose statistics were collected as signed cannot be used (crates/polars-parquet/src/arrow/read/statistics/bounds.rs:151):

 rust
 
 
/// Whether the `min_value` and `max_value` of a leaf bound the values polars decodes
/// as `dtype`: the file declares an order for the leaf, and it is the one polars
/// compares the decoded values with.
fn bounds_are_usable(
 column_order: ColumnOrder,
 physical_type: &PhysicalType,
 dtype: &ArrowDataType,
) -> bool {
 let Some(required) = required_sort_order(dtype, physical_type) else {
 return false;
 };
 match column_order {
 ColumnOrder::TypeDefinedOrder(order) => order == required,
 // Total order agrees with the float comparison on every non-NaN value.
 ColumnOrder::IEEE754TotalOrder => {
 matches!(physical_type, PhysicalType::Float | PhysicalType::Double)
 },
 ColumnOrder::Unsupported | ColumnOrder::Undefined => false,
 }
}

The same care shows up for timestamps: if the reader multiplies a microsecond timestamp into nanoseconds and the multiplication overflows, the values have no order, so both bounds are required and both must be in range, otherwise the chunk bounds nothing (crates/polars-parquet/src/arrow/read/statistics/bounds.rs:264).

Nested columns need one more step. A struct column is not one Parquet leaf but several, and a predicate on a single struct field should be able to prune on that field’s statistics. load_struct_column_statistics therefore assembles a per-field min/max/null-count structure from the leaves, and gives up — falls back to all-null statistics, which prune nothing — if the leaf count and the struct field count do not line up (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:732).

The “already applied” flag

There is a second level of pruning that happens before the reader ever sees the file: table statistics. If a Parquet dataset has per-file min/max statistics recorded out of band, a predicate can be evaluated against them and whole files skipped. When that happens, the reader must not evaluate the predicate again — or rather, the plan has to know whether there is anything left to evaluate. That is what PredicateFileSkip is for (crates/polars-descriptions/src/lib.rs:21):

 rust
 
 
pub struct PredicateFileSkip {
 /// If `true` the predicate can be skipped at runtime.
 pub no_residual_predicate: bool,
 /// Number of files before skipping
 pub original_len: usize,
}

It is set when the file-level statistics are applied (crates/polars-mem-engine/src/scan_predicate/functions.rs:563):

 rust
 
 
 let predicate_to_readers = (!can_skip_scan_predicate).then_some(&scan_predicate);

 if let Some(skip_files_mask) = skip_files_mask {
 assert_eq!(skip_files_mask.len(), sources.len());

 let predicate_file_skip = PredicateFileSkip {
 no_residual_predicate: predicate_to_readers.is_none(),
 original_len: sources.len(),
 };

and honoured when the pipeline is initialised: with no_residual_predicate set, the predicate is simply not handed to the reader (crates/polars-stream/src/nodes/io_sources/multi_scan/pipeline/initialization.rs:124):

 rust
 
 
 Some(PredicateFileSkip {
 no_residual_predicate: false,
 original_len: _,
 }) => (None, config.predicate.as_ref()),
 Some(PredicateFileSkip {
 no_residual_predicate: true,
 original_len: _,
 }) => (None, None),

The one subtlety is runtime ranges: a predicate with a runtime range is not consumed by the file-level statistics, because the range can change while the scan runs. That is why can_skip_scan_predicate is predicate.runtime_ranges.is_empty() in the hive-partition branch (crates/polars-mem-engine/src/scan_predicate/functions.rs:407).

Ranges that arrive late

The runtime range is the case where a join discovers a filter after the scan has started. The plan cannot prune on it, but the reader can still check each row group against it just before fetching it. RuntimeRangeSkipper resolves the range against the file’s statistics once (crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:540):

 rust
 
 
 fn skips_with(&self, checks: &[RangeCheck], row_group_idx: usize) -> bool {
 checks.iter().any(|check| match check {
 RangeCheck::KeepAll => false,
 RangeCheck::SkipAll => true,
 RangeCheck::Bounds {
 leaf,
 lo,
 hi,
 nulls_match,
 } => {
 let leaf = self.hints[*leaf].leaf.as_ref().unwrap();
 let Some((min, max)) = leaf.row_group_bounds(row_group_idx, &self.metadata) else {
 return false;
 };
 let below = lo
 .as_ref()
 .is_some_and(|lo| max.as_ref().is_some_and(|max| max < lo));
 let above = hi
 .as_ref()
 .is_some_and(|hi| min.as_ref().is_some_and(|min| min > hi));
 (below || above)
 && (!nulls_match
 || has_no_nulls(&self.metadata.row_groups[row_group_idx], leaf.leaf_idx))
 },
 })
 }

and the fetcher re-checks the front of the queue before each fetch, so a range that tightened since the plan was made still skips work (crates/polars-stream/src/nodes/io_sources/parquet/row_group_data_fetch.rs:139):

 rust
 
 
 /// Moves past the row groups at the front that the runtime ranges skip.
 fn skip_front(&mut self) {
 while let Some(row_group) = self.row_groups.front()
 && self
 .runtime_range_skipper
 .as_ref()
 .is_some_and(|s| s.skips(row_group.idx))
 {
 self.row_groups.pop_front();
 }
 }

The page index: written, not read

The write side is where the page index shows up. When the footer is written, Polars serialises a column index and an offset index for every column chunk — the column index only if the writer was configured to collect statistics, the offset index unconditionally (crates/polars-parquet/src/parquet/write/file.rs:205):

 rust
 
 
 if self.options.write_statistics {
 // write column indexes (require page statistics)
 self.row_groups
 .iter_mut()
 .zip(self.page_specs.iter())
 .try_for_each(|(group, pages)| {
 group.columns.iter_mut().zip(pages.iter()).try_for_each(
 |(column, pages)| {
 let offset = self.offset;
 if let Some(length) = write_column_index(&mut self.writer, pages)? {
 self.offset += length;
 column.column_index_offset = Some(offset as i64);
 column.column_index_length = Some(length as i32);
 }
 ParquetResult::Ok(())
 },
 )?;
 ParquetResult::Ok(())
 })?;
 };

and the offset index right after it, unconditionally. So a file written by Polars carries per-page min/max and per-page offsets, and any reader that implements the page index can use them.

Polars’ own reader does not. The vendored page reader still carries the comment from the crate it was forked from (crates/polars-parquet/src/parquet/read/page/reader.rs:68):

 rust
 
 
/// A fallible [`Iterator`] of [`CompressedDataPage`]. This iterator reads pages back
/// to back until all pages have been consumed.
///
/// The pages from this iterator always have [`None`] [`crate::parquet::page::CompressedDataPage::selected_rows()`] since
/// filter pushdown is not supported without a
/// pre-computed [page index](https://github.com/apache/parquet-format/blob/master/PageIndex.md).
pub struct PageReader {

and the read path never opens the column index or the offset index: the only references to them in the reader are the metadata accessors that tell you where the index is, not what is in it.

The bottom line

Pruning is per row group, using the per-column-chunk min/max/null-count in the footer. The bounds are borrowed from the footer buffer, not copied, and are dropped if the file’s sort order or an exactness flag makes them unusable.
The predicate is not re-implemented for statistics. It is evaluated on a DataFrame of <col>_min/<col>_max/<col>_nc columns, which means the filter you wrote and the filter that prunes row groups are the same expression.
PredicateFileSkip::no_residual_predicate is the “already applied” flag: it tells the reader that table statistics already consumed the predicate, so the reader must not evaluate it again.
A range published at run time is checked again per row group, immediately before the fetch.
Polars writes the page index (with statistics enabled) so that other engines can prune at page granularity. If you want that from Polars itself, it is not there yet, and the win would be largest on files with many small row groups and highly selective predicates.

Polars Writes Parquet From a Streaming Pipeline

Jay Hubbert — Wed, 21 Oct 2026 05:50:00 +0000

TL;DR: A Parquet file is a sequence of row groups, and a row group is a sequence of pages, and pages are compressed. That means a Parquet writer cannot just start writing bytes as rows arrive — it has to decide where the row group boundaries are, and it cannot know that until it has seen enough rows. Polars solves this with a resize pipeline between the streaming engine and the writer: morsels arrive at whatever size the upstream produced them, get re-cut into row groups of a target size, and are then encoded column-by-column in parallel while a single task writes the pages to the file in order.

The hard part of streaming Parquet is not the encoding. It is that a row group boundary is a decision, and the decision depends on data you have not seen yet. If you flush a row group too eagerly you get a million tiny row groups and a file that is slow to read. If you wait too long you hold the whole thing in memory.

Let me walk through how the pieces fit, starting from the Python API and going down to the byte writes.

The API

The entry point is sink_parquet on a LazyFrame (crates/polars-python/src/lazyframe/general.rs:796):

 rust
 
 
 #[cfg(feature = "parquet")]
 #[pyo3(signature = (
 target, sink_options, compression, compression_level, statistics, row_group_size, data_page_size,
 metadata, arrow_schema
 ))]
 fn sink_parquet(
 &self,
 py: Python<'_>,
 target: PyFileSinkDestination,
 sink_options: PySinkOptions,
 compression: &str,
 compression_level: Option<i32>,
 statistics: Wrap<StatisticsOptions>,
 row_group_size: Option<usize>,
 data_page_size: Option<usize>,
 metadata: Wrap<Option<KeyValueMetadata>>,
 arrow_schema: Option<Wrap<ArrowSchema>>,
 ) -> PyResult<PyLazyFrame> {

It builds a ParquetWriteOptions and hands it to LazyFrame::sink (crates/polars-lazy/src/frame/mod.rs:807):

 rust
 
 
 pub fn sink(
 mut self,
 sink_type: SinkDestination,
 file_format: FileWriteFormat,
 unified_sink_args: UnifiedSinkArgs,
 ) -> PolarsResult<Self> {
 polars_ensure!(
 !matches!(self.logical_plan, DslPlan::Sink { .. }),
 InvalidOperation: "cannot create a sink on top of another sink"
 );

 self.logical_plan = DslPlan::Sink {

Note that sink is a plan operation, not an execution one. The returned LazyFrame is a plan that ends in a sink; it is executed when you collect() it. This is why sink_parquet returns a PyLazyFrame and not a path.

The options themselves (crates/polars-io/src/parquet/write/options.rs:24):

 rust
 
 
pub struct ParquetWriteOptions {
 /// Data page compression
 pub compression: ParquetCompression,
 /// Compute and write column statistics.
 pub statistics: StatisticsOptions,
 /// If `None` will be all written to a single row group.
 pub row_group_size: Option<usize>,
 /// if `None` will be 1024^2 bytes
 pub data_page_size: Option<usize>,
 /// Custom file-level key value metadata
 pub key_value_metadata: Option<KeyValueMetadata>,
 pub arrow_schema: Option<ArrowSchemaRef>,
 #[cfg_attr(feature = "serde", serde(default))]
 pub compat_level: Option<CompatLevel>,
}

row_group_size: None does not mean “one row group” in the streaming engine — the comment is about the old, non-streaming path. The streaming path has its own default, which we will meet shortly.

The sink node

The sink is a node in the streaming graph (crates/polars-stream/src/nodes/io_sinks/mod.rs:30):

 rust
 
 
pub struct IOSinkNode {
 name: PlSmallStr,
 state: IOSinkNodeState,
 io_metrics: Option<Arc<IOMetrics>>,
 verbose: bool,
 partitioned_by_key: bool,
}

It has a peculiar lifecycle: it does not do any work during the query. It waits for the input pipe to report Done, and only then does it initialise the writer pipeline and wait for it (crates/polars-stream/src/nodes/io_sinks/mod.rs:84):

 rust
 
 
 recv[0] = if recv[0] == PortState::Done {
 // Ensure initialize / writes empty file for empty output.
 self.state
 .initialize(&self.name, execution_state, self.io_metrics.clone())?;

 match std::mem::replace(&mut self.state, IOSinkNodeState::Finished) {
 IOSinkNodeState::Initialized {
 phase_channel_tx,
 task_handle,
 } => {

Wait — that reads as if the sink receives no morsels at all during the query. It does receive them; they just go somewhere else. The sink takes a serial receive port (crates/polars-stream/src/nodes/io_sinks/mod.rs:141):

 rust
 
 let phase_morsel_rx = recv_ports[0].take().unwrap().serial();

and hands it to a “phase” channel. The point of the phase mechanism is that the sink may be executed in several phases by the graph scheduler, and the writer pipeline must survive across those phases. The phase task relays morsels from each phase into a single long-lived channel (crates/polars-stream/src/nodes/io_sinks/mod.rs:222):

 rust
 
 
 executor::spawn(TaskPriority::High, task_metrics, async move {
 let mut morsel_seq: u64 = 1;

 while let Ok(mut phase_rx) = phase_channel_rx.recv().await {
 while let Ok(mut morsel) = phase_rx.recv().await {
 morsel.set_seq(MorselSeq::new(morsel_seq));
 morsel_seq = morsel_seq.saturating_add(1);

 if multi_phase_tx.send(morsel).await.is_err() {
 break;
 }
 }
 }
 });

Note the sequence numbers are renumbered here. The sink does not care about the upstream’s ordering guarantees; it cares about producing a file, and for that it just needs a total order over the morsels it receives. Renumbering makes the writer’s ordering logic trivial.

The sink also always sends one empty morsel at the start (crates/polars-stream/src/nodes/io_sinks/mod.rs:216):

 rust
 
 
 let _ = multi_phase_tx.try_send(Morsel::new_unregistered(
 DataFrame::empty_with_arc_schema(config.input_schema.clone()),
 MorselSeq::new(0),
 SourceToken::default(),
 ));

That is how an empty result still produces a valid, empty Parquet file rather than nothing at all.

The resize pipeline: morsels are the wrong size

Now the interesting part. The upstream produces morsels of about POLARS_IDEAL_MORSEL_SIZE rows (100,000 by default). A Parquet row group wants a different number of rows, and the two are not related: the user may have asked for row_group_size = 1_000_000, or for something byte-driven, or left it alone.

So between the engine and the writer there is a stage that re-cuts the morsels. That stage is MorselResizePipeline (crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs:22):

 rust
 
 
/// Splits large morsels / combines small morsels.
pub struct MorselResizePipeline {
 pub schema: Arc<Schema>,
 pub target_sink_morsel_size: TargetSinkMorselSize,
 pub inflight_morsel_semaphore: Arc<tokio::sync::Semaphore>,
 pub morsel_rx: connector::Receiver<Morsel>,
 pub morsel_tx: connector::Sender<SinkMorsel>,
}

It is not a trivial pass-through. It maintains a buffer of received dataframes, and it decides, for each outgoing chunk, how many rows to take from the buffer (crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs:54):

 rust
 
 
 loop {
 let df_to_send = if let Some(num_rows_to_take) = num_rows_to_take_iter.next() {
 take_n_rows_from_buffered(Arc::clone(&schema), &mut buffered_rows, num_rows_to_take)
 } else if let Some(n_rows) = next_chunk_lengths.pop() {
 let next_chunk_size = logical_received_size
 .calc_delta(n_rows as IdxSize, physical_received_size)
 .unwrap();

 let logical_buffered_size = logical_received_size.checked_sub(sent_size).unwrap();

 let send_buffered_as_one_df;

 (send_buffered_as_one_df, num_rows_to_take_iter) = target_sink_morsel_size
 .calc_next_splits(logical_buffered_size, next_chunk_size);

 logical_received_size = logical_received_size.add(next_chunk_size)?;

 if send_buffered_as_one_df {
 take_n_rows_from_buffered(
 Arc::clone(&schema),
 &mut buffered_rows,
 idxsize_to_u64(logical_buffered_size.num_rows),
 )
 } else {
 continue;
 }
 } else if let Ok(morsel) = {

Three cases, and they are worth reading carefully:

There is a pending split plan. Take that many rows from the buffer and send them.
There is a new chunk of rows. Ask the target size how to split the combination of what is buffered and what just arrived. If the answer is “flush the buffer as one chunk”, do that; otherwise adopt the new split plan and loop.
No more rows. Flush whatever is left as one final chunk.

The logical_received_size / physical_received_size distinction is there because a morsel can have several Arrow chunks, and the split logic works on chunk boundaries where possible. The sent_size tracks what has already gone out.

Taking rows out of the buffer is a stack (crates/polars-stream/src/nodes/io_sinks/components/morsel_resize_pipeline.rs:188):

 rust
 
 
fn take_n_rows_from_buffered(
 schema: Arc<Schema>,
 buffered_rows: &mut VecDeque<DfWithOffset>,
 mut n_rows: u64,
) -> DataFrame {
 if n_rows == 0 {
 return DataFrame::empty_with_arc_schema(schema);
 }

 let mut stacked_df: Option<DataFrame> = None;

 while n_rows != 0 {
 let next_df = buffered_rows.front_mut().unwrap();

 let next_df = if next_df.height() as u64 > n_rows {
 next_df.split_off_front(n_rows as usize)
 } else {
 buffered_rows.pop_front().unwrap().into_df()
 };

 if next_df.height() as u64 == n_rows && stacked_df.is_none() {
 return next_df;
 }
 ...

If the requested row count happens to land exactly on a frame boundary, the frame is returned as-is — no copy, no vertical stack. Only when the boundary falls in the middle of a frame does the frame get split and the pieces stacked.

Deciding the row group size

The decision logic is TargetSinkMorselSize::calc_next_splits (crates/polars-stream/src/nodes/io_sinks/components/size.rs:211):

 rust
 
 
impl TargetSinkMorselSize {
 pub fn calc_next_splits(
 &self,
 buffered_size: RowCountAndSize,
 incoming_size: RowCountAndSize,
 ) -> (bool, PartSizesIter) {
 let combined_size = buffered_size.checked_add(incoming_size).unwrap();

 let mut flush_buffered_as_one_split = false;
 let (mut part_sizes_iter, mut limited_by) = self.build_part_sizes_iter(combined_size);

and build_part_sizes_iter is where the two limits — rows and bytes — are arbitrated (crates/polars-stream/src/nodes/io_sinks/components/size.rs:274):

 rust
 
 
 fn build_part_sizes_iter(&self, size: RowCountAndSize) -> (PartSizesIter, LimitedBy) {
 if size.num_rows == 0 {
 return (PartSizesIter::default(), LimitedBy::default());
 }

 let n_parts_by_num_rows = if self.target_num_rows_mode == SplitMode::Exact {
 u64::max(1, idxsize_to_u64(size.num_rows / self.target_num_rows))
 } else {
 calc_n_parts(
 idxsize_to_u64(size.num_rows),
 NonZeroU64::new(idxsize_to_u64(self.target_num_rows.get())).unwrap(),
 )
 };

 let mut max_parts_by_num_bytes = 0;
 let mut n_parts_by_num_bytes = 0;

 if self.target_num_bytes.get() != u64::MAX {
 max_parts_by_num_bytes = idxsize_to_u64(size.num_rows / self.target_num_bytes_min_rows);
 n_parts_by_num_bytes = calc_n_parts(size.num_bytes, self.target_num_bytes);
 };

The SplitMode distinction is the interesting bit:

Exact means “every row group must have exactly target_num_rows rows”. This is what you want when the user set row_group_size explicitly: the file must have the row group layout they asked for.
Approximate means “get close to target_num_rows”. This is the default when the user did not ask for anything, and it lets the pipeline roll a row group as soon as it has roughly enough rows, which keeps the pipeline moving instead of waiting for an exact count.

The calc_n_parts helper is the shared arithmetic (crates/polars-utils/src/calc_morsel_split.rs:69):

 rust
 
 
/// Number of parts to split `size` to minimize the average part size difference to `target_part_size`.
pub fn calc_n_parts(size: u64, target_part_size: NonZeroU64) -> u64 {
 if size <= target_part_size.get() {
 return if size == 0 { 0 } else { 1 };
 }

 let n_parts = size / target_part_size;

 (n_parts..=n_parts.saturating_add(1))
 .min_by_key(|n_parts| (size / *n_parts).abs_diff(target_part_size.get()))
 .unwrap()
}

Only two candidates are considered: n and n+1 parts. The winner is the one whose resulting part size is closest to the target. That is enough, because size / n and size / (n+1) straddle the target.

The part sizes themselves come from PartSizesIter (crates/polars-utils/src/calc_morsel_split.rs:45):

 rust
 
 
impl Iterator for PartSizesIter {
 type Item = u64;

 fn next(&mut self) -> Option<Self::Item> {
 self.remaining_parts = self.remaining_parts.checked_sub(1)?;
 Some(self.base_part_size + (self.remaining_parts >= self.remainder_cutoff) as u64)
 }

It is a counting-sort-style distribution of the remainder: the first remainder_cutoff parts get one extra row, the rest do not. The result is that sum(part_sizes) == total exactly, with the parts differing by at most one row.

There is one more subtlety in calc_next_splits (crates/polars-stream/src/nodes/io_sinks/components/size.rs:236):

 rust
 
 
 if limited_by == LimitedBy::Rows
 && part_sizes_iter.len() <= 1
 && self.target_num_rows_mode != SplitMode::Exact
 && part_sizes_iter.base_part_size().checked_mul(2).is_some_and(
 // base_part_size < (4/3) * target_num_rows
 |double_base_part_size| {
 u64::abs_diff(
 double_base_part_size,
 idxsize_to_u64(self.target_num_rows.get()),
 ) < u64::abs_diff(
 part_sizes_iter.base_part_size(),
 idxsize_to_u64(self.target_num_rows.get()),
 )
 },
 )
 {
 // Wait for more data to have a fuller chunk.
 part_sizes_iter = PartSizesIter::default()
 }

In Approximate mode, if the accumulated data is already close to target_num_rows, the pipeline waits for more rather than flushing a slightly-short row group. The 4/3 heuristic: if doubling the current size would land closer to the target than the current size does, then the current size is “under” the target by more than a factor of 4/3, so it is worth waiting.

The defaults

ParquetWriterStarter decides the target (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:52):

 rust
 
 
 fn target_sink_morsel_size(&self) -> TargetSinkMorselSize {
 let target_num_bytes_min_rows = const { NonZeroIdxSize::new(16384).unwrap() };

 if let Some(row_group_size) = self.row_group_size
 && row_group_size > 0
 {
 TargetSinkMorselSize {
 target_num_rows: row_group_size.try_into().unwrap(),
 target_num_bytes: NonZeroU64::MAX,
 target_num_bytes_min_rows,
 target_num_rows_mode: SplitMode::Exact,
 }
 } else {
 let (env_num_rows, env_num_bytes) = ideal_sink_morsel_size_env();

 TargetSinkMorselSize {
 target_num_rows: env_num_rows
 .unwrap_or(const { NonZeroIdxSize::new(122_880).unwrap() }),
 target_num_bytes: env_num_bytes.unwrap_or(NonZeroU64::MAX),
 target_num_bytes_min_rows,
 target_num_rows_mode: SplitMode::Approximate,
 }
 }
 }

So:

If the user set row_group_size, the mode is Exact and the byte limit is disabled. The file will have exactly the row group layout requested.
Otherwise the default is 122,880 rows in Approximate mode. That number is 120 × 1024, and it is a deliberate choice: Parquet readers and the ecosystem generally cope better with row groups in the low hundreds of thousands of rows.
The byte limit is u64::MAX (disabled) unless POLARS_IDEAL_SINK_MORSEL_SIZE_BYTES is set, in which case the target_num_bytes_min_rows of 16,384 acts as a floor: a row group never gets smaller than 16k rows even if the byte budget says so.

Three tasks: resize, encode, write

The single-file sink pipeline is assembled in crates/polars-stream/src/nodes/io_sinks/pipeline_initialization/single_file.rs:136:

 rust
 
 
 let (writer_tx, writer_rx) = connector::connector();
 let writer_handle = file_writer_starter.start_file_writer(
 writer_rx,
 file_open_task,
 num_pipelines_per_sink,
 execution_state.task_metrics.clone(),
 )?;

 let schema = Arc::clone(&file_schema);
 let inflight_morsel_semaphore =
 Arc::new(tokio::sync::Semaphore::new(inflight_morsel_limit.get()));

 let resize_pipeline = MorselResizePipeline {
 schema,
 target_sink_morsel_size,
 inflight_morsel_semaphore,
 morsel_rx,
 morsel_tx: writer_tx,
 };

The semaphore is the memory bound. Every outgoing sink morsel holds a permit, and the permit is released only when the data has actually been written (crates/polars-stream/src/nodes/io_sinks/components/sink_morsel.rs:3):

 rust
 
 
pub type SinkMorselPermit = tokio::sync::OwnedSemaphorePermit;

/// In-flight morsel in the IO sink. Holds a permit against a semaphore that restricts
/// the total number of sink morsels in memory.
pub struct SinkMorsel {
 df: DataFrame,
 /// Should only be dropped once the data associated with this morsel has been dropped from memory.
 permit: SinkMorselPermit,
}

The limit is num_pipelines + 4 (crates/polars-stream/src/nodes/io_sinks/config.rs:30):

 rust
 
 
 pub fn inflight_morsel_limit(&self, num_pipelines: NonZeroUsize) -> NonZeroUsize {
 ...
 NonZeroUsize::saturating_add(
 num_pipelines,
 // Additional buffer to accommodate head-of-line blocking
 4,
 )
 }

The extra four is for head-of-line blocking: if one row group takes longer to encode than the others, the other permits let the pipeline keep feeding the encoder.

Now the writer itself. ParquetWriterStarter::start_file_writer (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:102) sets up three things: a channel of encoded row groups, a writer task and an encoder task.

 rust
 
 
 let (encoded_row_group_tx, encoded_row_group_rx) = tokio::sync::mpsc::channel::<
 executor::AbortOnDropHandle<PolarsResult<EncodedRowGroup>>,
 >(num_pipelines.get());

The channel’s capacity is the number of pipelines, so up to that many row groups can be in flight — being encoded, or encoded and waiting to be written.

The WriteOptions are built once (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:106):

 rust
 
 
 let key_value_metadata = self.options.key_value_metadata.clone();
 let write_options = WriteOptions {
 statistics: self.options.statistics,
 compression: self.options.compression.into(),
 version: Version::V1,
 data_page_size: self.options.data_page_size,
 };

Version::V1 is hardcoded here — the streaming writer does not emit Parquet v2 files even though the option type allows it. compression is converted from the user-facing ParquetCompression enum (crates/polars-io/src/parquet/write/options.rs:55):

 rust
 
 
impl From<ParquetCompression> for CompressionOptions {
 fn from(value: ParquetCompression) -> Self {
 use ParquetCompression::*;
 match value {
 Uncompressed => CompressionOptions::Uncompressed,
 Snappy => CompressionOptions::Snappy,
 Gzip(level) => CompressionOptions::Gzip(level),
 Brotli(level) => CompressionOptions::Brotli(level),
 Lz4Raw => CompressionOptions::Lz4Raw,
 Zstd(level) => CompressionOptions::Zstd(level),
 }
 }
}

and the default is Zstd (crates/polars-io/src/parquet/write/options.rs:49):

 rust
 
 
impl Default for ParquetCompression {
 fn default() -> Self {
 Self::Zstd(None)
 }
}

The encoder: parallel across columns

The encoder task (crates/polars-stream/src/nodes/io_sinks/writers/parquet/row_group_encoder.rs:52) receives sink morsels and turns each into an encoded row group:

 rust
 
 
 while let Ok(morsel) = morsel_rx.recv().await {
 let arrow_schema = Arc::clone(&arrow_schema);
 let schema_descriptor = Arc::clone(&schema_descriptor);
 let encodings = Buffer::clone(&encodings);
 let column_task_metrics = task_metrics.clone();

 let row_group_encode_handle = executor::AbortOnDropHandle::new(executor::spawn(
 TaskPriority::High,
 task_metrics.as_deref(),
 async move {
 let (df, morsel_permit) = morsel.into_inner();
 let num_rows = df.height();

 let mut data: Vec<Vec<CompressedPage>> = Vec::with_capacity(num_leaf_columns);

The whole row group is encoded in a spawned task, and the handle is sent to the writer rather than the result. That is the key to the pipelining: the writer does not wait for the encoding to finish before it starts the next one.

Inside the task, the columns are encoded in parallel (crates/polars-stream/src/nodes/io_sinks/writers/parquet/row_group_encoder.rs:73):

 rust
 
 
 for fut in parallelize_first_to_local(
 TaskPriority::High,
 column_task_metrics.as_deref(),
 df.into_columns().into_iter().enumerate().map(|(i, c)| {
 let arrow_schema = Arc::clone(&arrow_schema);
 let schema_descriptor = Arc::clone(&schema_descriptor);
 let encodings = Buffer::clone(&encodings);

 async move {
 let parquet_type = &schema_descriptor.fields()[i];
 let encodings = encodings[i].as_slice();
 let array =
 c.as_materialized_series().rechunk().to_arrow_with_field(
 0,
 Cow::Borrowed(arrow_schema.get_at_index(i).unwrap().1),
 true,
 )?;

 let mut data: UnitVec<Vec<CompressedPage>> =
 UnitVec::with_capacity(num_leaf_columns);

 for encode_page_iter in array_to_columns(
 array,
 parquet_type.clone(),
 write_options,
 encodings,
 )? {
 let compressed_pages: Vec<CompressedPage> =
 Compressor::new_from_vec(
 encode_page_iter.map(|result| {
 result.map_err(|e| {
 ParquetError::FeatureNotSupported(format!(
 "reraised in polars: {e}",
 ))
 })
 }),
 write_options.compression,
 vec![],
 )
 .collect::<ParquetResult<_>>()?;

 data.push(compressed_pages)
 }

 PolarsResult::Ok(data)
 }
 }),
 ) {
 data.extend(fut.await?);
 }

parallelize_first_to_local keeps the first column on the current thread and spawns the rest (crates/polars-async/src/primitives/opt_spawned_future.rs:59):

 rust
 
 
/// Parallelizes futures across the computational async runtime.
///
/// As an optimization for cache access, the first future is kept on the current thread. If there
/// is only 1 future, then all data is kept on the current thread and spawn is not called at all.

That is a cache-locality optimisation: the first column’s data is already in this core’s cache, so encoding it here avoids a cross-core transfer. The remaining columns are spawned, so a wide table is encoded on all cores at once.

Note the order of operations inside the column future: rechunk() to a single Arrow chunk, convert to Arrow with the field’s metadata attached, then array_to_columns to produce the encoded (but uncompressed) pages, then Compressor to compress them. The compression happens per page, inside the same parallel task, so the compressed bytes never leave the core that produced them.

The writer: one task, one file

The writer task is deliberately on the I/O runtime, not the compute one (crates/polars-stream/src/nodes/io_sinks/writers/parquet/mod.rs:117):

 rust
 
 
 let io_handle = tokio_handle_ext::AbortOnDropHandle(
 ASYNC.spawn(
 io_writer::IOWriter {
 file,
 encoded_row_group_rx,
 arrow_schema,
 schema_descriptor: Arc::clone(&schema_descriptor),
 write_options,
 encodings: Buffer::clone(&encodings),
 key_value_metadata,
 num_leaf_columns,
 }
 .run(),
 ),
 );

and its body is short (crates/polars-stream/src/nodes/io_sinks/writers/parquet/io_writer.rs:57):

 rust
 
 
 while let Some(handle) = encoded_row_group_rx.recv().await {
 let EncodedRowGroup {
 num_rows,
 data,
 morsel_permit,
 } = handle.await?;
 assert_eq!(data.len(), num_leaf_columns);
 parquet_writer.write_row_group(num_rows as u64, &data)?;
 drop(data);
 drop(morsel_permit);
 }

handle.await? is where the writer waits for the row group to finish encoding — but by then the next row group is already being encoded on another core. So the pipeline is:

text

 resize task encoder task(s) writer task
 (compute) (compute, N in flight) (IO runtime)
 ┌───────────┐ morsel ┌──────────────┐ pages ┌──────────────┐
 │ cut rows │ ───────► │ encode cols │ ──────► │ write bytes │
 │ to target │ │ + compress │ (chan) │ to file │
 └───────────┘ └──────────────┘ └──────────────┘

The morsel_permit is dropped after the write, so the semaphore that bounds in-flight memory is only released once the data is on disk. That is what keeps the memory bound honest.

The BatchedWriter wraps the actual Parquet file writer (crates/polars-io/src/parquet/write/batched_writer.rs:29):

 rust
 
 
pub struct BatchedWriter<W: Write> {
 // A mutex so that streaming engine can get concurrent read access to
 // compress pages.
 //
 // @TODO: Remove mutex when old streaming engine is removed
 pub(super) writer: Mutex<FileWriter<W>>,
 // @TODO: Remove when old streaming engine is removed
 pub(super) parquet_schema: SchemaDescriptor,
 pub(super) encodings: Buffer<Vec<Encoding>>,
 pub(super) options: WriteOptions,
 pub(super) parallel: bool,
 pub(super) key_value_metadata: Option<KeyValueMetadata>,
}

and write_row_group hands the already-compressed pages straight through (crates/polars-io/src/parquet/write/batched_writer.rs:104):

 rust
 
 
 /// Note: `num_rows` can be passed as `u64::MAX` to infer `num_rows` from the encoded data.
 pub fn write_row_group(
 &mut self,
 num_rows: u64,
 rg: &[Vec<CompressedPage>],
 ) -> PolarsResult<()> {
 let writer = self.writer.get_mut().unwrap();
 let rg = DynIter::new(rg.iter().map(|col_pages| {
 Ok(DynStreamingIterator::new(
 fallible_streaming_iterator::convert(col_pages.iter().map(PolarsResult::Ok)),
 ))
 }));
 writer.write(num_rows, rg)?;
 Ok(())
 }

No re-encoding, no re-compression — the pages arrive ready. The underlying FileWriter::write (crates/polars-parquet/src/parquet/write/file.rs:167) is where the bytes actually hit the file:

 rust
 
 
 /// Writes a row group to the file.
 ///
 /// This call is IO-bounded
 pub fn write<E>(
 &mut self,
 num_rows: u64,
 row_group: RowGroupIterColumns<'_, E>,
 ) -> ParquetResult<()>
 where
 ParquetError: From<E>,
 E: std::error::Error,
 {
 if self.offset == 0 {
 self.start()?;
 }
 let ordinal = self.row_groups.len();
 let (group, specs, size) = write_row_group(
 &mut self.writer,
 num_rows,
 self.offset,
 self.schema.columns(),
 row_group,
 ordinal,
 )?;
 self.offset += size;
 self.row_groups.push(group);
 self.page_specs.push(specs);
 Ok(())
 }

The row group metadata is accumulated in memory (row_groups, page_specs) and written once at the end. That is the one piece of state that grows with the number of row groups, but it is tiny compared to the data.

Finally the footer (crates/polars-stream/src/nodes/io_sinks/writers/parquet/io_writer.rs:73):

 rust
 
 
 parquet_writer.finish()?;
 let parquet_metadata = {
 let writer = parquet_writer.get_writer().lock().unwrap();
 let mut out = Vec::new();
 write_metadata_sidecar(&mut out, writer.metadata().unwrap())?;
 out
 };
 drop(parquet_writer);
 buffered_file.flush()?;
 drop(buffered_file);

 file.set_parquet_metadata(parquet_metadata);
 file.close(sync_on_close).await?;

The metadata sidecar is written to a separate buffer rather than into the file — that is what makes the partitioned sink possible, where several files’ row groups are described by one metadata blob.

What is parallel and what is serialised

Putting it all in one table:

stage	where	parallel?
resize (re-cut morsels)	compute executor	no, one task
encode + compress a row group	compute executor	yes, one task per column (first column kept local)
several row groups at once	compute executor	yes, up to `num_pipelines` in flight
write bytes to the file	I/O runtime	no, one task owns the file
file open / close	I/O runtime	no

The invariant is: the file itself is written by exactly one task, in row group order. Everything else is parallel. That is the whole design of the Parquet sink: parallelism is pushed as far upstream as possible, and the serial part is reduced to “append these bytes to this file”.

The price is that the writer is a bottleneck if the encoding is fast. With a wide table and cheap compression, the encoder can produce row groups faster than the disk can absorb them — and then the channel between them fills up and the encoder blocks. That is correct behaviour, not a bug: the backpressure just moves the bottleneck to where it actually is.

Partitioned sinks

The same machinery drives the partitioned sink, with one extra stage. start_partition_sink_pipeline (crates/polars-stream/src/nodes/io_sinks/pipeline_initialization/partition_by.rs:29) inserts a partitioner between the engine and the resize stage, and a distributor that opens files on demand.

Each partition gets its own writer, its own resize pipeline and its own row group sequence, so a partitioned write produces several independent Parquet files, each with its own row groups. The number of simultaneously open files is bounded (crates/polars-stream/src/nodes/io_sinks/config.rs:46):

 rust
 
 
 pub fn max_open_sinks(&self) -> NonZeroUsize {
 ...
 if self.target.is_cloud_location() {
 const { NonZeroUsize::new(512).unwrap() }
 } else {
 const { NonZeroUsize::new(128).unwrap() }
 }
 }

128 local files, 512 in the cloud. Beyond that the sink has to close and reopen files, which is why partitioned writes to object storage have a per-file size limit option (crates/polars-stream/src/nodes/io_sinks/pipeline_initialization/partition_by.rs:112):

 rust
 
 
 if let Some(file_size_limit) = file_size_limit {
 target_sink_morsel_size.target_num_rows = NonZeroIdxSize::min(
 target_sink_morsel_size.target_num_rows,
 file_size_limit.num_rows,
 );
 target_sink_morsel_size.target_num_bytes = NonZeroU64::min(
 target_sink_morsel_size.target_num_bytes,
 file_size_limit.num_bytes,
 );
 target_sink_morsel_size.target_num_rows_mode = SplitMode::Exact;
 }

Note that the file size limit also forces SplitMode::Exact: if you said “at most 1M rows per file”, the row groups have to add up to exactly that, not approximately.

The old, non-streaming path

For comparison, the classic path is much simpler (crates/polars-io/src/parquet/write/writer.rs:147):

 rust
 
 
 /// Write the given DataFrame in the writer `W`.
 /// Returns the total size of the file.
 pub fn finish(self, df: &mut DataFrame) -> PolarsResult<u64> {
 let chunked_df = chunk_df_for_writing(df, self.row_group_size.unwrap_or(512 * 512))?;
 let mut batched = self.batched(chunked_df.schema())?;
 batched.write_batch(&chunked_df)?;
 batched.finish()
 }

chunk_df_for_writing splits the frame into row groups of row_group_size rows (default 512 × 512 = 262,144), and write_batch encodes and compresses them eagerly (crates/polars-io/src/parquet/write/batched_writer.rs:251):

 rust
 
 
/// This serializer encodes and compresses all eagerly in memory.
/// Used for separating compute from IO.
fn create_eager_serializer(

Note “eagerly in memory” — the old path materialises the entire encoded row group before writing it. The streaming path is the same code with the eagerness removed: the row group is encoded in a spawned task, and the handle is what travels down the channel.

The bottom line

The design of the Parquet sink comes down to four decisions:

Row group boundaries are decided by a separate stage. The streaming engine’s morsels are not row groups. A resize pipeline re-cuts them, using PartSizesIter to distribute the remainder exactly, and using calc_n_parts to pick a split count that lands close to the target.
The mode matters. Exact when the user asked for a specific row_group_size, Approximate otherwise. In approximate mode the writer will even wait for more rows rather than flush a short row group.
Encoding is parallel, writing is serial. Column encoding and compression run on the compute executor, in parallel across columns and across row groups. Exactly one task owns the file and writes the pages in order.
The memory bound is a semaphore, not a guess. Every in-flight sink morsel holds a permit, released only after the data is written.

The knobs, if you want to experiment:

POLARS_IDEAL_SINK_MORSEL_SIZE_ROWS — the default row group size when row_group_size is not set (default 122,880).
POLARS_IDEAL_SINK_MORSEL_SIZE_BYTES — a byte-based limit on the row group, with a 16,384-row floor.
POLARS_INFLIGHT_SINK_MORSEL_LIMIT — how many sink morsels may be in memory at once (default num_pipelines + 4).
POLARS_MAX_OPEN_SINKS — how many files a partitioned sink may hold open at once.

DataFusion's Parquet Access Plan: How Row Groups, Pages and Files Get Skipped Before a Byte Is Read

Jay Hubbert — Wed, 21 Oct 2026 05:09:00 +0000

TL;DR: DataFusion turns “which bytes do I need?” into a data structure — a ParquetAccessPlan — rather than a sequence of if statements buried in the reader. The plan starts as “scan everything”, then gets narrowed by cheap proofs: file-level statistics, then per-row-group min/max (zone maps), then per-page min/max from the column index, then bloom filters. Every narrowing step only ever removes work; it never adds a scan. The proof obligation is one-sided: a row group may only be skipped when statistics show that no row in it can satisfy the predicate.

The naive reader

The obvious way to answer “which row groups do I need?” is to open the file, look at the footer, and for each row group ask “could this row group contain a row with ts > 2024-01-01?”. That is what a simple Parquet reader does, and it is fine — as long as you have only one kind of evidence and one place to ask the question.

Real queries are not like that. A predicate might be provably false for a whole file (a partition column says region = 'eu', the predicate wants region = 'us'), provably false for a row group (the min/max of ts in the footer), provably false for a page (the min/max of ts in the column index), or provably false only if you consult a bloom filter. And in the other direction: it may be provable that every row in a row group matches, which lets you skip the per-row filter entirely.

DataFusion’s design decision is that all of these must be expressed in one place, and that place must be inspectable, testable, and independent of the decoder: ParquetAccessPlan.

The access plan

 rust
 
 
// datafusion/datasource-parquet/src/access_plan.rs:145-152
pub enum RowGroupAccess {
 /// Do not read the row group at all
 Skip,
 /// Read all rows from the row group
 Scan,
 /// Scan only the specified rows within the row group
 Selection(RowSelection),
}

Three states, not two. Selection matters: page-index pruning produces row ranges, and those ranges have to survive the rest of the pipeline (including the row-group reordering used for TopK). ParquetAccessPlan keeps the Selection per row group and intersects new selections into it rather than replacing it:

 rust
 
 
// datafusion/datasource-parquet/src/access_plan.rs:438-463
pub fn scan_selection(&mut self, idx: usize, selection: RowSelection) {
 self.row_groups[idx] = match &self.row_groups[idx] {
 RowGroupAccess::Skip => RowGroupAccess::Skip,
 RowGroupAccess::Scan => RowGroupAccess::Selection(selection),
 RowGroupAccess::Selection(existing_selection) => {
 // ... promote to mask backing when one side is a bitmap, so an
 // external index's bitmap survives the intersection
 RowGroupAccess::Selection(existing_selection.intersection(&selection))
 }
 };
}

Note the bitmap promotion comment: arrow-rs keeps a RowSelection either as run-length selectors or as a boolean bitmap, and intersecting the two forms is not free — so DataFusion converts the selector side into a bitmap first, keeping the cheapest representation for repeated intersections.

The pipeline: state machine, not a pile of ifs

The opener is an explicit state machine, documented as a diagram in the source:

 text
 
 
// datafusion/datasource-parquet/src/opener/mod.rs:342-373
 Start
 v
[LoadEncryption]?
 v
 PruneFile <- file-level stats, before any footer is read
 v
 LoadMetadata
 v
 PrepareFilters <- now the *file* schema is known
 v
PruneWithStatistics <- row group min/max (zone maps)
 v
 LoadPageIndex? <- skipped when all surviving groups are fully matched
 v
 LoadBloomFilters
 v
PruneWithBloomFilters
 v
 BuildStream

The ordering is the whole point. Each stage is cheaper than the next, and each stage can eliminate the work of the stages after it:

PruneFile uses file-level statistics collected by the catalog (or partition values) and can skip the file without reading even the footer (opener/mod.rs:1021-1053). Because dynamic filters may have tightened since planning, this check is repeated at open time.
PruneWithStatistics is pure CPU over the footer — no I/O at all.
LoadPageIndex is I/O: the column/offset index lives outside the footer in modern Parquet files, so fetching it is a second round trip. DataFusion therefore asks for it with PageIndexPolicy::Skip when loading metadata and only pays for it if it will actually be used:

 rust
 
 
// datafusion/datasource-parquet/src/opener/mod.rs:1062-1063
let mut options =
 ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Skip);

LoadBloomFilters is more I/O, and it only happens for row groups that statistics have not already resolved.

Whether the page index is worth loading is decided by should_load_page_index (opener/mod.rs:1370-1449): is the feature enabled, is there at least one row group that might contain filtered rows, and does the footer record a column index offset for a predicate column? If every surviving row group was proven “fully matched”, the page index cannot prune anything and its I/O is skipped.

Zone maps: the row-group min/max proof

Row-group pruning goes through RowGroupAccessPlanFilter (row_group_filter.rs). The interesting part is not the mechanics but the proof discipline. PruningPredicate (in datafusion/pruning) rewrites the predicate into an expression over statistics columns and evaluates it vectorised over all row groups at once:

 rust
 
 
// datafusion/datasource-parquet/src/row_group_filter.rs:367-380
match predicate.prune(&pruning_stats) {
 Ok(values) => {
 for (idx, &value) in row_group_indexes.iter().zip(values.iter()) {
 if !value {
 self.access_plan.skip(*idx);
 metrics.row_groups_pruned_statistics.add_pruned(1);
 } else { ... }
 }

The rewrite is documented in pruning_predicate.rs:244-252:

Original predicate	Rewritten predicate
`x = 5`	`x_null_count != x_row_count AND (x_min <= 5 AND 5 <= x_max)`
`x < 5`	`x_null_count != x_row_count AND (x_min < 5)`
`x IS NULL`	`x_null_count > 0`

This is the “zone map” idea from database literature, implemented as an expression rewrite rather than a hand-written comparison per operator. The rewrite must be sound in the right direction: false means “no row in this container can be TRUE”, and NULL means “we don’t know”, which must be treated as “keep the container”.

The inverse trick: proving that everything matches

The cleverest bit in this file is not about skipping work — it is about skipping verification. If statistics can prove that every row in a row group satisfies the predicate, then the per-row filter does not have to be evaluated at all for that row group.

DataFusion does this by pruning with the logical inverse of the predicate:

 rust
 
 
// datafusion/datasource-parquet/src/row_group_filter.rs:440-448
for (i, &original_row_group_idx) in candidate_row_group_indices.iter().enumerate()
{
 // If the inverted predicate *also* prunes this row group (meaning inverted_values[i] is false),
 // it implies that *all* rows in this group satisfy the original predicate.
 if !inverted_values[i] {
 self.access_plan.mark_fully_matched(original_row_group_idx);

Building the inverse is subtle because SQL is three-valued. A row where the filter evaluates to NULL does not pass it, so the inverse must be NOT pred OR col IS NULL for every nullable column the predicate touches:

 rust
 
 
// datafusion/datasource-parquet/src/pruning.rs:43-62
let mut inverted_expr: Arc<dyn PhysicalExpr> =
 Arc::new(NotExpr::new(Arc::clone(predicate.orig_expr())));
// ... for each nullable referenced column:
inverted_expr = Arc::new(BinaryExpr::new(
 inverted_expr,
 Operator::Or,
 Arc::new(IsNullExpr::new(Arc::new(column))),
));

The whole inversion is refused if the pruning rewrite only preserved “can be TRUE” but not “is FALSE vs UNKNOWN” (pruning_predicate.rs:570, can_be_inverted_for_full_match). The payoff is real: a fully matched row group gets an empty RowFilter installed in the decoder (opener/mod.rs:1784-1795), so the per-row evaluation cost disappears for it entirely.

Page index: from row groups to pages

Row-group granularity is coarse. The Parquet page index gives per-page min/max (column index) plus per-page row offsets (offset index), which allows skipping row ranges inside a row group:

 rust
 
 
// datafusion/datasource-parquet/src/page_filter.rs:683-708
// Convert the information of which pages to skip into a RowSelection
// that describes the ranges of rows to skip.
for (i, &f) in values.iter().enumerate().skip(1) {
 if f == selected { sum_row += page_row_counts[i]; }
 else {
 let selector = if selected { RowSelector::select(sum_row) }
 else { RowSelector::skip(sum_row) };
 vec.push(selector);
 ...

Page row counts come from the offset index’s first_row_index deltas (page_filter.rs:777-795) — the offset index does double duty as a page locator and a row-range index.

Two restrictions are worth noticing. First, the page index must be complete:

 rust
 
 
// datafusion/datasource-parquet/src/page_filter.rs:246-260
if !parquet_metadata.page_index().is_some_and(|page_index| page_index.is_complete()) {
 debug!("Can not prune pages due to lack of indexes. ...");
 return PagePruningResult::new(access_plan, 0, 0);
}

Partial indexes are useless here because you cannot know which pages were not indexed. Second, page pruning only works for single-column predicates (page_filter.rs:186-190): a page index is per column, so a predicate spanning two columns cannot be resolved from one column’s bounds.

Bloom filters: the last resort, and the false-positive contract

Bloom filters answer a different question: not “is the value in the range?” but “is the value definitely absent?” DataFusion loads them per row group, per predicate column, and feeds them into the same PruningPredicate machinery through a PruningStatistics adapter. The false-positive handling is a one-sided contract:

 rust
 
 
// datafusion/datasource-parquet/src/bloom_filter.rs:206-237
let known_not_present = values.iter()
 .map(|value| BloomFilterStatistics::check_scalar(...))
 .all(|v| !v);

let contains = if known_not_present { Some(false) } else {
 // Given the bloom filter is probabilistic, we can't be sure that
 // the row group actually contains the values. Return `None` to
 // indicate this uncertainty
 None
};

Some(false) means “definitely not present” → the row group can be pruned. None means “unknown” → keep the row group. There is deliberately no “probably present” answer, because a bloom filter cannot distinguish “present” from “false positive”. Unsupported scalar types return true from check_scalar (bloom_filter.rs:88-97), i.e. “assume it might be there” — a bloom filter over an unsupported type must never cause data loss. Bloom filter I/O is also skipped for row groups that statistics already proved fully matched (opener/mod.rs:1531-1536).

Range assignment: splitting a file for parallelism

One more pruning axis has nothing to do with predicates: a big file may be split into byte ranges so several tasks can read it in parallel. Each row group must land in exactly one range, so row_group_in_range (row_group_filter.rs:57-77) picks an offset inside the row group and tests it against the range. StartOffset uses the row group’s first data page; Midpoint uses the midpoint of the compressed size, matching parquet-java and spreading very large row groups more evenly across ranges.

The catch

Pruning is only as good as the metadata, and metadata lies in several ways.

Truncated statistics. Min/max strings are truncated (64 bytes by default, datafusion.execution.parquet.statistics_truncate_length), so bounds are wider than reality. row_group_filter.rs:189-203 spells out the consequence: a fully-matching row group can no longer be proven fully matching, and the per-row filter has to stay.
Untrusted byte-array ordering. The deprecated Parquet min/max fields use signed comparison, while Arrow compares strings as unsigned bytes. Using them for pruning can produce wrong answers, so DataFusion refuses to trust them:

 rust
 
 
// datafusion/datasource-parquet/src/metadata.rs:69-99
pub(crate) fn has_untrusted_min_max_order(...) -> bool {
 if matches!(column.sort_order(), SortOrder::UNDEFINED | SortOrder::INT96_TIMESTAMP) {
 return true;
 }
 requires_unsigned_byte_array_order(&column)
 && (column.sort_order() != SortOrder::UNSIGNED
 || column_orders.and_then(|orders| orders.get(parquet_column_index).copied())
 != Some(ColumnOrder::TYPE_DEFINED_ORDER(SortOrder::UNSIGNED)))
}

The handling is not “give up on the column”: mask_untrusted_byte_array_stats (row_group_filter.rs:554-572) nulls out the bounds of only the row groups that carry the deprecated signed statistics, keeping the modern bounds usable.

Every pruning pass costs CPU. The bloom pass explicitly skips evaluation for row groups with no loaded filter, “a cost files written without bloom filters would pay for nothing” (row_group_filter.rs:481-487).

The design invariant that makes all of this safe is worth stating plainly: every pruning step is a proof of absence, and a failed or untrusted proof degenerates to “read it”. Nothing in the pipeline can make a query return fewer rows than it should. That is the property that lets DataFusion be aggressive — including the aggressive part where it decides that a row group is entirely matching and turns the filter off.

DataFusion's Parquet Push Decoder: Pruning Row Groups in the Middle of a Scan You Already Started

Jay Hubbert — Wed, 21 Oct 2026 05:01:00 +0000

TL;DR: Most Parquet readers are pull-based: you ask for a batch, they decode it. DataFusion’s push decoder inverts that — you push byte ranges into it and it tells you what it needs next. Because the decoder is paused at row-group boundaries rather than in the middle of one, a filter that tightened while the scan was running can still be applied to every row group that has not been fetched yet. The scan does not need to be restarted, and the bytes that were never fetched are never paid for.

Every Parquet reader eventually has to answer the same question: given a predicate, which row groups can be skipped entirely? The standard answer is “decide once, at open time, from the footer statistics”. That works when the predicate is known at plan time.

But DataFusion now has predicates that are not known at plan time — a TopK that tightens its threshold as it fills its heap, a hash join that publishes its build-side bounds once it has read the build side. Those facts arrive mid-scan. If the reader decided everything at open time, the new fact would be useless for the file currently being read.

The push decoder exists to close that gap.

Pull vs push

The classic arrow-rs reader is pull-based:

text

reader.next() -> decode a batch from the current row group
 (fetch bytes as needed, behind the scenes)

The push decoder is the other way round:

 text
 
 
decoder.try_next_reader()
 -> NeedsData(ranges) "give me these byte ranges"
 -> Data(reader) "here is a decoded reader for this row group"
 -> Finished "no more"

The driver in datafusion/datasource-parquet/src/push_decoder.rs is a futures::stream::unfold over a state machine:

 rust
 
 
pub(crate) fn into_stream(self) -> BoxStream<'static, Result<RecordBatch>> {
 futures::stream::unfold(self, |state| async move { state.transition().await })
 .fuse()
 .boxed()
}

and the state machine itself alternates between fetching bytes and decoding:

 rust
 
 
// Step 3: drive the decoder.
let decoder = self.decoder.as_mut().expect("decoder present");
match decoder.try_next_reader() {
 Ok(DecodeResult::NeedsData(ranges)) => {
 // I/O, not compute.
 timer.stop();
 let data = self.reader.get_byte_ranges(ranges.clone()).await ...;
 timer.restart();
 self.decoder.as_mut().unwrap().push_ranges(ranges, data) ...
 }
 Ok(DecodeResult::Data(reader)) => {
 // Pop the RG this reader is for ...
 if let Some(entry) = self.rg_plan.pop_front() {
 self.byte_progress.credit(entry.bytes);
 }
 self.active_reader = Some(reader);
 }
 Ok(DecodeResult::Finished) => return None,
 ...
}

Notice the timer.stop() / timer.restart() around the I/O await. The elapsed_compute metric measures compute, and the fetch is not compute. This is a small thing, but it is the kind of thing that matters: without it, every metric that claims to measure CPU time is really measuring CPU time plus object-store latency, and no one can tell the difference.

Row-group boundaries are the checkpoint

The whole design rests on one property: the decoder can be paused at a row-group boundary and its plan re-derived. The code is explicit about where this is allowed:

 rust
 
 
// `into_builder` errors out mid-row-group, so we gate the prune
// pass on `is_at_row_group_boundary()`. When the decoder is
// mid-RG (e.g. byte ranges have been pushed but no reader has
// been handed back yet), step 3 drives it forward and we get
// another chance at the next boundary — the pruner is stateful
// and idempotent, so deferring loses nothing.
let at_boundary = self
 .decoder
 .as_ref()
 .expect("decoder present")
 .is_at_row_group_boundary();

So the loop is:

Drain a batch from the active reader, if there is one. Return it.
If we are at a row-group boundary, re-derive the row-group plan and rebuild the decoder if anything was dropped.
Drive the decoder: fetch bytes or hand back a reader.

Step 2 is where the dynamic filter pays off.

The row-group plan

The scan keeps a plan of the row groups it intends to read:

 rust
 
 
#[derive(Debug, Clone)]
pub(crate) struct RgPlanEntry {
 pub(crate) rg_index: usize,
 /// `true` when static pruning proved every row of this RG satisfies the
 /// predicate, so the per-row `RowFilter` can be skipped as a no-op.
 pub(crate) fully_matched: bool,
 /// On-disk size of this row group, credited to the `bytes_processed` metric
 /// documented on `ParquetFileMetrics` once the scan is done with it.
 pub(crate) bytes: u64,
}

and at a boundary, the whole remaining plan is scanned and pruned at once:

 rust
 
 
// Step 2: when the decoder is sitting on a row-group boundary,
// scan the entire `rg_plan` and drop every RG the pruner proves
// cannot contribute — head, interior, and tail alike. Evaluating
// per-RG stats against the cached `PruningPredicate` is cheap;
// the expensive part is the `into_builder` rebuild, so we do at
// most one rebuild per boundary regardless of how many RGs were
// dropped. Buffered bytes for already-fetched RGs carry across
// the rebuild.

The phrase “at most one rebuild per boundary” is the key efficiency insight. Rebuilding the decoder is expensive (you have to construct a new reader with new selections). Evaluating a pruning predicate against a row group’s statistics is cheap. So the algorithm prunes many row groups and then pays the rebuild cost once.

The runtime pruner

RowGroupPruner is the piece that answers “should this row group be dropped right now?”:

 rust
 
 
pub(crate) struct RowGroupPruner {
 predicate: Arc<dyn PhysicalExpr>,
 arrow_schema: SchemaRef,
 parquet_metadata: Arc<ParquetMetaData>,
 /// Classifies the predicate's dynamic-filter content.
 tracking: DynamicFilterTracking,
 /// First-call sentinel: forces an initial `pruning_predicate` build
 /// even when `tracking` is `Static` / `AllComplete`.
 needs_initial_build: bool,
 /// Cached pruning predicate. `None` means we couldn't build one for the
 /// current generation (e.g. the predicate has no analyzable bounds);
 /// in that case we conservatively don't prune.
 pruning_predicate: Option<Arc<PruningPredicate>>,
 ...
}

The needs_initial_build flag exists because DynamicFilterTracking::classify distinguishes “no dynamic filters” from “dynamic filters, all complete” — and in the latter case the current value of the filter may already be different from what planning assumed, so a one-shot prune is still worth doing, but there is nothing to watch afterwards.

The check itself is where the “cheap atomic load” trick from the previous article in this series pays off:

 rust
 
 
pub(crate) fn should_prune(&mut self, row_group_indices: &[usize]) -> bool {
 if row_group_indices.is_empty() {
 return false;
 }

 // Refresh the cached `PruningPredicate` on the first call and
 // whenever a watched dynamic filter has advanced since we last
 // looked. `changed()` is a single atomic load per still-incomplete
 // filter — no tree walk on every check.
 let dynamic_changed = self
 .tracking
 .watcher()
 .is_some_and(|tracker| tracker.changed());
 if self.needs_initial_build || dynamic_changed {
 self.pruning_predicate = PruningPredicateBuilder::new()
 .with_file_schema(Arc::clone(&self.arrow_schema))
 .with_error_counter(&self.predicate_creation_errors)
 .with_max_in_list_size(self.max_in_list_size)
 .build(Arc::clone(&self.predicate));
 self.needs_initial_build = false;
 }
 ...

Building a PruningPredicate involves real expression analysis. Doing it once per row group would dominate the cost of pruning. Doing it once per filter generation is nearly free.

And the failure policy is conservative in the right direction:

 rust
 
 
match pp.prune(&stats) {
 // `prune` returns `false` per container that the predicate proves
 // cannot contain matching rows. We can skip the run only when
 // every requested row group is in that state.
 Ok(values) => values.iter().all(|&keep| !keep),
 Err(e) => {
 debug!("Ignoring error evaluating runtime row-group pruning predicate: {e}");
 self.predicate_evaluation_errors.add(1);
 false
 }
}

Two safety properties: a row group is dropped only when every container in it is prunable, and any error means “do not prune”. A flaky pruning path never silently drops data.

The other half: stopping the whole file

The row-group pruner handles the tail of a file. The file-level pruner handles the case where the tightened filter proves that no row group can match — at which point the whole file should be abandoned.

EarlyStoppingStream (datafusion/datasource-parquet/src/opener/early_stop.rs) is a RecordBatchStream adapter that re-checks the file pruner on every batch:

 rust
 
 
fn check_prune(&mut self, input: Result<RecordBatch>) -> Result<Option<RecordBatch>> {
 let batch = input?;

 // Since dynamic filters may have been updated, see if we can stop
 // reading this stream entirely.
 if self.file_pruner.should_prune()? {
 self.files_ranges_pruned_statistics.add_pruned(1);
 // Previously this file range has been counted as matched
 self.files_ranges_pruned_statistics.subtract_matched(1);
 self.finish();
 Ok(None)
 } else {
 // Return the adapted batch
 Ok(Some(batch))
 }
}

The metric adjustment is a nice detail: the file was counted as matched when it was opened, and now it is being reclassified as pruned, so the counters have to be corrected or the metrics will lie.

The struct holds the inner stream as Option<S> specifically so that finishing releases the decoder immediately:

 rust
 
 
/// The inner stream, dropped as soon as this stream is done with it.
///
/// Held as an `Option` so finishing releases the decoder — and the buffers
/// and per-file metric state it owns — at the moment we stop reading, not
/// whenever the caller gets around to dropping this wrapper. Notably the
/// scan's byte-progress accounting is completed by that drop, so deferring
/// it would leave the file reading as partially scanned after the scan had
/// demonstrably finished with it.
inner: Option<S>,

That is a genuinely subtle piece of engineering: the drop of the parquet reader is what finalises the “bytes processed” accounting. If the adapter merely stopped polling and let the drop happen later — or never — the metrics would report a partially-scanned file after the scan had finished with it.

The catch

The decoder must be rebuildable. Pausing at a row-group boundary requires the decoder to support being torn down and reconstructed with a new set of row-group selections. That is a real constraint on arrow-rs’s decoder API, and it is why the push decoder is a separate code path rather than a flag on the ordinary reader.
Statistics must be good enough. Runtime pruning is only as good as the Parquet footer. A file written without statistics, or with statistics that do not reflect the actual data, cannot be pruned at runtime — and, worse, wrong statistics would cause rows to be dropped.
The rebuild is not free. One rebuild per boundary is fine. If a dynamic filter updated on every batch, you would rebuild the decoder on every batch. The design assumes updates are rare (a TopK threshold refresh, a hash join publishing once).
Alignment bugs are the main hazard. The comment on sync_rg_plan_to_decoder_frontier is a war story:

arrow-rs silently finishes row groups whose post-predicate selection is empty without handing back a reader, so without this sync rg_plan trails the decoder by one and a rebuild either re-reads an already delivered row group (#24352) or toggles the per-RG filter for the wrong row group.

A one-row-group skew between “the plan we think we are executing” and “the row group the decoder is actually on” produces either duplicated rows or a filter applied to the wrong data. Both are correctness bugs. The reference to issue #24352 suggests this bit at least once.

Why this matters

Consider a SELECT ... ORDER BY x LIMIT 10 over a Parquet table. The TopK operator learns, after the first few batches, that only values greater than 417 are interesting. If the scan can be told that, the remaining row groups of every file can be skipped at runtime — not because the planner knew the threshold, but because the consumer told the producer.

That is the shape of the whole dynamic-filter idea, and it is why DataFusion treats DynamicFilterPhysicalExpr as a first-class physical expression rather than a special case bolted onto one operator.

Polars Pushes Predicates Into the Parquet Scanner

Jay Hubbert — Wed, 21 Oct 2026 05:00:00 +0000

TL;DR: A Polars filter does not travel to the Parquet reader as a special “skip predicate”. It is the same expression tree, compiled a second time into a predicate over a synthetic frame whose columns are named col_min, col_max and col_nc — one row per row group. The reader evaluates that predicate over the statistics it read from the Parquet footer, and every row group the predicate rejects is never fetched, decompressed or decoded. The optimizer hands the filter to the scan by deleting the IR::Filter node and putting the predicate on the IR::Scan, and a flag on the scan says whether the predicate still has to filter rows or whether it has already been settled at the file level.

Reading a Parquet file and filtering it is the single most common thing a dataframe engine does, and the naive version of it is silly: decode every column of every row group, then throw away 99% of the rows. Parquet’s own format makes that avoidable. A row group is the unit of compression and encoding, and the file footer carries per-row-group statistics — a minimum, a maximum and a null count per column. If those statistics prove that a row group cannot contain a matching row, the row group never has to be read at all.

The interesting part is not that Polars uses the statistics. It is how the predicate gets from the logical plan into that decision, because the two sides of the problem want completely different shapes of expression. The whole path looks like this:

text

 logical plan IR reader
 --------------- --------------- -------------------
 IR::Filter -> IR::Scan -> row group 0 (skip)
 (predicate) { predicate: ... } row group 1 (skip)
 ...
 IR::Filter -> IR::Scan -> row group 7 (read,
 (predicate) { predicate: ... } decode, filter rows)
 ...
 |
 | aexpr_to_skip_batch_predicate
 v
 IR::Scan
 { predicate: (a_min > 3) | (a_max < 3) | ... }
 |
 | evaluated over a frame with one row
 | per row group
 v
 Bitmap { 1, 1, 0, 0, 1, 0, 0, 0 } (1 = skip)

The last line is the entire trick: a second compilation of the same expression, over a namespace of statistics columns, evaluated once per row group.

The optimizer hands the filter to the scan

The rewrite happens in the PredicatePushDown optimizer rule, in the Scan arm of the plan traversal. The rule walks the plan from the top, accumulating predicates in a map keyed by the leaf columns they read, and pushes them towards the leaves. When it reaches a scan, it merges the accumulated predicates with whatever the scan already had and stores the result on the scan node:

 rust
 
 
 Scan {
 sources,
 file_info,
 hive_parts: scan_hive_parts,
 ref predicate,
 predicate_file_skip_applied,
 scan_type,
 unified_scan_args,
 output_schema,
 maintain_order,
 } => {

crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs:364

 rust
 
 
 let predicate = combine_predicates(
 Option::into_iter(predicate.clone()).chain(acc_predicates.into_values()),
 expr_arena,
 );

 let do_optimization = predicate.is_some()
 && match &*scan_type {
 #[cfg(feature = "csv")]
 FileScanIR::Csv { .. } => unified_scan_args.pre_slice.is_none(),
 FileScanIR::ExpandedPaths { .. } => false,
 FileScanIR::Anonymous { function, .. } => {
 function.allows_predicate_pushdown()
 },
 #[allow(unreachable_patterns)]
 _ => true,
 };

crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs:386

and then, crucially, the filter node either disappears or survives above the scan:

 rust
 
 
 let lp = if do_optimization {
 Scan {
 sources,
 file_info,
 hive_parts,
 predicate,
 predicate_file_skip_applied,
 unified_scan_args,
 output_schema,
 scan_type,
 maintain_order,
 }
 } else {
 let lp = Scan {
 sources,
 file_info,
 hive_parts,
 predicate: None,
 predicate_file_skip_applied,
 unified_scan_args,
 output_schema,
 scan_type,
 maintain_order,
 };
 if let Some(predicate) = predicate {
 let input = lp_arena.add(lp);
 Filter { input, predicate }
 } else {
 lp
 }
 };

crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs:429

Note the else branch: if the scan type does not support predicate pushdown, the predicate is put back as a plain IR::Filter above the scan. The pushdown is a move, not a copy — a filter that reaches a scan is no longer a filter node.

The expression is compiled a second time

Now the scan has a predicate expressed in terms of the data columns. The scanner, however, wants to ask a question about statistics: “could this row group possibly contain a row where a == 3?” Asking that with the data expression is impossible, because the data is what you are trying to avoid reading.

Polars solves this by translating the expression into a second expression over a synthetic schema. The module that does this is crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs, and the translation of col(A) == B is spelled out in the source as a comment:

 rust
 
 
 // col(A) == B -> {
 // null_count(A) == 0 , if B.is_null(),
 // null_count(A) == LEN || min(A) > B || max(A) < B, if B.is_not_null(),
 // }

crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs:351

which is then built, literally, out of three columns named A_min, A_max and A_nc plus a len column:

 rust
 
 
 not_null: {
 let col_min = target.min(arena);
 let col_max = target.max(arena);

 let min_is_defined = is_stat_defined(col_min.node(), dtype, arena);
 let max_is_defined = is_stat_defined(col_max.node(), dtype, arena);

 let min_gt = col_min.gt(lv_node, arena);
 let min_gt = min_gt.and(min_is_defined, arena);

 let max_lt = col_max.lt(lv_node, arena);
 let max_lt = max_lt.and(max_is_defined, arena);

 let col_nc = target.null_count(arena);
 let len = col!(len);
 let all_nulls = col_nc.eq(len, arena);

 all_nulls.or(min_gt, arena).or(max_lt, arena).node()
 }

crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs:371

Read that as a boolean: this row group can be skipped if it is entirely null, or if its minimum is above B, or if its maximum is below B. The is_stat_defined guard exists because a null statistic means “unknown”, not “zero” — skipping on an unknown bound would drop real rows.

The translation is conservative by construction: any expression shape the translator does not recognise yields None, and None means “no skip predicate”, which means “read everything”. A rewrite that is wrong in the dangerous direction (skipping a row group that could contain a match) is a data-loss bug, so the translator is written to bail out rather than guess.

There is one subtlety worth calling out, because it is the kind of thing that silently corrupts results: floats. Parquet’s statistics exclude NaN, but the data may contain it, and NaN compares greater than everything under TotalOrd. So col < x is safe to prune on (a NaN never satisfies it) but col > x is not. The translator encodes that:

 rust
 
 
/// Whether min/max statistics are usable for the given dtype, operator, and literal.
///
/// Rejects nested, null, and categorical types. For floats, Parquet stats exclude NaN
/// but data may contain it. Since NaN is largest under TotalOrd, `col < x` is safe
/// (NaN never matches) but `col > x` is not (NaN always matches).
/// col >= NaN matches NaN == NaN, but stats exclude NaN so max < NaN can't detect them -> unsafe
fn can_use_min_max_stats(
 dtype: &DataType,
 op: Option<&Operator>,
 lv: Option<&LiteralValue>,
) -> bool {
 if dtype.is_nested() || dtype.is_null() || dtype.is_categorical() {
 return false;
 }

 if !dtype.is_float() {
 return true;
 }

 let lv_is_nan = lv.is_some_and(|lv| lv.is_nan());

 use Operator as O;
 match op {
 Some(O::Lt | O::LtEq) => true,
 None | Some(O::Eq | O::EqValidity) => !lv_is_nan && lv.is_some(),
 Some(O::Gt) => lv_is_nan,
 _ => false,
 }
}

crates/polars-plan/src/plans/aexpr/predicates/skip_batches.rs:205

The statistics frame

The skip predicate is an ordinary Polars expression, so it needs an ordinary schema. The scan builds that schema when it constructs its predicate, giving every live column three statistics columns plus one row count:

 rust
 
 
 let mut skip_batch_schema = Schema::with_capacity(1 + live_columns.len());

 skip_batch_schema.insert(PlSmallStr::from_static("len"), IDX_DTYPE);
 for (col, dtype) in schema.iter() {
 if !live_columns.contains(col) {
 continue;
 }

 skip_batch_schema.insert(format_pl_smallstr!("{col}_min"), dtype.clone());
 skip_batch_schema.insert(format_pl_smallstr!("{col}_max"), dtype.clone());
 skip_batch_schema.insert(format_pl_smallstr!("{col}_nc"), null_count_dtype(dtype));
 }

 skip_batch_predicate = Some(create_physical_expr(
 &expr,
 expr_arena,
 &Arc::new(skip_batch_schema),
 state,
 )?);

crates/polars-mem-engine/src/scan_predicate/functions.rs:161

So the whole translation reduces to: the skip predicate is a Polars expression over a frame with one row per row group and columns named <col>_min, <col>_max, <col>_nc and len. No new evaluator, no new predicate language, no special-casing in the expression engine.

The statistics the reader gets from the footer are not directly usable as Polars values. They are stored in the file’s physical type, they may be inexact, and the file may declare a sort order that is not the one Polars compares with. crates/polars-parquet/src/arrow/read/statistics/bounds.rs handles all of that, and the guard is a single function:

 rust
 
 
/// Whether the `min_value` and `max_value` of a leaf bound the values polars decodes
/// as `dtype`: the file declares an order for the leaf, and it is the one polars
/// compares the decoded values with.
fn bounds_are_usable(
 column_order: ColumnOrder,
 physical_type: &PhysicalType,
 dtype: &ArrowDataType,
) -> bool {
 let Some(required) = required_sort_order(dtype, physical_type) else {
 return false;
 };
 match column_order {
 ColumnOrder::TypeDefinedOrder(order) => order == required,
 // Total order agrees with the float comparison on every non-NaN value.
 ColumnOrder::IEEE754TotalOrder => {
 matches!(physical_type, PhysicalType::Float | PhysicalType::Double)
 },
 ColumnOrder::Unsupported | ColumnOrder::Undefined => false,
 }
}

crates/polars-parquet/src/arrow/read/statistics/bounds.rs:151

This is the piece that makes the whole scheme safe. A file that stores an Int64 column but declares Unsigned order, or declares no order at all, cannot be pruned with the statistics: the recorded maximum might not be the numerically largest value. Polars simply refuses to prune in that case.

The conversion itself drops bounds that do not survive it:

 rust
 
 
 /// Converts the bounds of a chunk of the leaf. A timestamp the reader
 /// multiplies past `i64` wraps, leaving the chunk's values without order:
 /// such a conversion needs both bounds, and both within range, to hold.
 pub fn convert_bounds<'a>(
 self,
 min: Option<PhysicalBound<'a>>,
 max: Option<PhysicalBound<'a>>,
 ) -> ParquetResult<(Option<ArrowBound<'a>>, Option<ArrowBound<'a>>)> {
 let convert = |bound: Option<_>| bound.map(|b| self.convert(b)).transpose();
 let (min, max) = (convert(min)?.flatten(), convert(max)?.flatten());
 if self.can_wrap() && (min.is_none() || max.is_none()) {
 return Ok((None, None));
 }
 Ok((min, max))
 }

crates/polars-parquet/src/arrow/read/statistics/bounds.rs:272

A nanosecond timestamp stored as milliseconds is multiplied by a million on the way in, which can wrap i64. If it wrapped, the values in the row group have no order, and a “maximum” that is smaller than the “minimum” proves nothing. So the conversion throws both bounds away rather than compare garbage.

The pruning loop

With the statistics in hand, the reader builds a frame with one row per row group and runs the skip predicate on it. The mask is computed in parallel on the compute runtime, because it is pure arithmetic over a handful of thousands of rows:

 rust
 
 
 let mut columns = Vec::with_capacity(1 + skip_batch_columns.len() * 3);

 let lengths: Vec<IdxSize> = row_groups_slice
 .iter()
 .map(|rg| rg.num_rows() as IdxSize)
 .collect();

 columns.push(Column::new("len".into(), lengths));

 for projection in projected_arrow_fields.iter() {
 let c = projection.output_name();

 if !skip_batch_columns.contains(c) {
 continue;
 }

 let mut statistics =
 load_parquet_column_statistics(&metadata, row_group_slice.clone(), projection)?;

 // Note: Order is important here. We re-use the transform for the output column, meaning
 // that it may set the column name.
 statistics.min = projection.apply_transform(statistics.min)?;
 statistics.max = projection.apply_transform(statistics.max)?;
 // The null counts of struct fields have the file's field layout. The transform may
 // reorder, rename or insert fields (e.g. by field ID), and casts the leaves, so it
 // does not apply to them: use unknown null counts.
 if let ArrowFieldProjection::Mapped { output_dtype, .. } = projection
 && matches!(projection.arrow_field().dtype(), ArrowDataType::Struct(_))
 {
 statistics.null_count = Column::full_null(
 PlSmallStr::EMPTY,
 statistics.null_count.len(),
 &null_count_dtype(output_dtype),
 );
 }

 let statistics = statistics.with_base_column_name(c);

 columns.extend([statistics.min, statistics.max, statistics.null_count]);
 }

 if let Some(row_index) = row_index {
 let statistics = build_row_index_statistics(&row_index, row_groups_slice)
 .with_base_column_name(&row_index.name);

 columns.extend([statistics.min, statistics.max, statistics.null_count]);
 }

 let statistics_df = DataFrame::new(num_row_groups, columns)?;

 sbp.evaluate_with_stat_df(&statistics_df)

crates/polars-stream/src/nodes/io_sources/parquet/statistics.rs:184

evaluate_with_stat_df returns a bitmap in which a set bit means “this row group can be skipped”. The mask is then turned into the list of row groups that actually get fetched — and this is the loop that does the saving:

 rust
 
 
pub(super) fn row_groups_to_read(
 metadata: &FileMetadata,
 row_group_slice: Range<usize>,
 row_group_mask: Option<&Bitmap>,
 mut row_offset: usize,
 mut slice_range: Option<Range<usize>>,
) -> VecDeque<RowGroupToRead> {
 let mut row_groups = VecDeque::with_capacity(row_group_slice.len());
 for (i, idx) in row_group_slice.enumerate() {
 let num_rows = metadata.row_groups[idx].num_rows();
 let slice = slice_range.as_mut().map(|slice_range| {
 let rg_row_start = slice_range.start;
 let rg_row_end = slice_range.end.min(num_rows);

 *slice_range = slice_range.start.saturating_sub(num_rows)
 ..slice_range.end.saturating_sub(num_rows);

 (rg_row_start, rg_row_end - rg_row_start)
 });
 if !row_group_mask.is_some_and(|mask| mask.get_bit(i)) {
 row_groups.push_back(RowGroupToRead {
 idx,
 row_offset,
 slice,
 });
 }
 row_offset = row_offset.saturating_add(num_rows);
 }
 row_groups
}

crates/polars-stream/src/nodes/io_sources/parquet/row_group_data_fetch.rs:65

Everything downstream — the byte-range fetcher, the decompressors, the decoders — is driven from that VecDeque. A row group that is not in the deque is never fetched, never decompressed and never decoded.

The flag that says “don’t apply it again”

Pruning a row group is not the same as filtering rows. A row group that survives the statistics check can still contain rows that fail the predicate, so the predicate has to be evaluated on the decoded rows as well. The scanner does both, and the difference is what filters_rows records.

There are two cases in which the predicate does not filter rows. The interesting one is a predicate that only reads hive partition columns: it decides which files to read, not which rows to keep. In that case the scan predicate is built with the row-level part replaced by true:

 rust
 
 
 let mut phys_predicate = create_physical_expr(&predicate, expr_arena, schema, state)?;

 // The hive predicate settles whole files; nothing is left to filter rows by.
 if hive_predicate_is_full_predicate {
 hive_predicate = Some(phys_predicate.clone());
 filters_rows = false;
 let node = expr_arena.add(AExpr::Literal(Scalar::from(true).into()));
 predicate = ExprIR::from_node(node, expr_arena);
 phys_predicate = create_physical_expr(&predicate, expr_arena, schema, state)?;
 }

crates/polars-mem-engine/src/scan_predicate/functions.rs:126

filters_rows travels with the predicate all the way into the reader, and the reader uses it to decide whether to build a mask at all:

 rust
 
 
 let df = if let Some(predicate) = self
 .predicate
 .as_ref()
 .filter(|p| p.filters_rows && !nothing_to_evaluate)
 {
 let mask = predicate.predicate.evaluate_io(&df)?;
 let mask = mask.bool().unwrap();

crates/polars-stream/src/nodes/io_sources/parquet/row_group_decode.rs:231

The plan-level counterpart of the same idea is PredicateFileSkip, which the IR::Scan node carries to say “the file-level skip already settled this predicate”:

 rust
 
 
pub struct PredicateFileSkip {
 /// If `true` the predicate can be skipped at runtime.
 pub no_residual_predicate: bool,
 /// Number of files before skipping
 pub original_len: usize,
}

crates/polars-plan/src/dsl/file_scan/mod.rs:719

It is set when the file-level pruning resolved the whole predicate, and it also serves as a “do not do this again” marker for the optimizer:

 rust
 
 
 let predicate_to_readers = (!can_skip_scan_predicate).then_some(&scan_predicate);

 if let Some(skip_files_mask) = skip_files_mask {
 assert_eq!(skip_files_mask.len(), sources.len());

 let predicate_file_skip = PredicateFileSkip {
 no_residual_predicate: predicate_to_readers.is_none(),
 original_len: sources.len(),
 };

 if verbose {
 eprintln!("apply_scan_predicate_to_scan_ir: {predicate_file_skip:?}");
 }

 *predicate_file_skip_applied = Some(predicate_file_skip);

crates/polars-mem-engine/src/scan_predicate/functions.rs:563

and the guard at the top of the same function refuses to redo the work:

 rust
 
 
 if unified_scan_args.has_row_index_or_slice() || predicate_file_skip_applied.is_some() {
 return Ok(());
 }

crates/polars-mem-engine/src/scan_predicate/functions.rs:525

no_residual_predicate is also what lets projection pushdown take the fast “count the rows without reading them” path for CSV: if the predicate has already been settled at the file level, the scan no longer has a row filter to apply, so the count can be taken from the file metadata.

The bottom line

The trick is that there is no second predicate language. Polars reuses its own expression engine, its own type coercion and its own null semantics, and pays for it by rewriting the expression into a namespace of statistics columns — col_min, col_max, col_nc — that the Parquet reader knows how to fill in with one row per row group.

What it buys: whole row groups are never read. That is not a constant-factor win, it is an I/O win, and for a selective predicate on a well-clustered file it can be the difference between reading 1% of the file and reading all of it.

What it costs:

The skip predicate is conservative by design. It only skips when the statistics prove no row can match. A file with no statistics, a file whose statistics are inexact, a file with a non-default sort order, a categorical column — all of them disable pruning for that column.
Statistics-based pruning is only as good as the data layout. A file written in random order has overlapping min/max ranges in every row group and nothing gets pruned. The win comes from sorted or clustered data.
Two predicate evaluations, not one. The statistics predicate runs once per row group; the row predicate still runs on the rows of the row groups that survive. A predicate that cannot be expressed over statistics costs you the translation and buys you nothing.
The rewrite is not free. Every pushdown-able expression gets a second compilation, and the expression engine has to be able to evaluate a comparison against a nullable statistic column without any special casing.

The practical takeaway is that predicate pushdown into the Parquet scanner is not a single feature but a chain: the optimizer moves the filter onto the scan, the scan compiles the filter into a statistics predicate, the reader evaluates that predicate against the footer, and the row groups that fail it never leave the metadata stage.

Late Materialization in DataFusion's Parquet Reader: Decode the Filter Columns First, Then the Rest

Jay Hubbert — Wed, 21 Oct 2026 01:41:00 +0000

TL;DR: The expensive part of SELECT a, b FROM t WHERE c > 5 is not the comparison — it is decoding a and b for 10 million rows to throw away 9,999,990 of them. DataFusion hands the predicate to the Parquet decoder as a RowFilter: the decoder evaluates c > 5 first, produces a mask, and only decodes a and b for the rows that survive. The columns the filter needs are read before the columns the query wants, which is the opposite of the order a naive reader uses.

The shape of the problem

A Parquet file stores columns independently: a is a column chunk, b is another, c is a third. Decoding any one of them costs I/O, decompression and (for dictionary-encoded, RLE or delta encodings) a fair bit of CPU. Rows, on the other hand, only exist as an alignment between column chunks. The obvious implementation of SELECT a, b FROM t WHERE c = 42 decodes everything and then throws most of it away:

 text
 
 
naive:
 decode a (10M rows) ┐
 decode b (10M rows) ┘→ apply c > 5 → 10 rows survive
 ↑ all that decoding was wasted

The fix is to decode c first, learn which rows survive, and then decode a and b only for those rows: “late materialization” — the filter is evaluated against a subset of the columns, and the wide payload is materialized only for the rows that pass.

arrow-rs’s Parquet decoder supports this via RowFilter, and DataFusion’s row_filter.rs documents the contract in its own words:

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:25-30
//! The `ParquetRecordBatchStream` takes an optional `RowFilter` which is itself
//! a Vec of `Box<dyn ArrowPredicate>`. During decoding, the predicates are
//! evaluated in order, to generate a mask which is used to avoid decoding rows
//! in projected columns which do not pass the filter which can significantly
//! reduce the amount of compute required for decoding and thus improve query
//! performance.

Two things follow. The predicates are evaluated in order, so order matters. And they generate a mask, so the decoder can use row ranges, not just a boolean vector — the second pass over a and b skips whole pages of rows.

Compiling a SQL predicate into a decoder filter

DataFusion’s pushdown has three stages, and each one can fail:

Split the predicate into conjuncts. a = 1 AND b = 2 AND c = 3 becomes three independent predicates. Conjunction is the only boolean operator that distributes cleanly: each conjunct must hold for the row to pass, so each conjunct’s mask can be computed independently and intersected.

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:493-495
// Split into conjuncts:
// `a = 1 AND b = 2 AND c = 3` -> [`a = 1`, `b = 2`, `c = 3`]
let predicates = split_conjunction(expr);

Decide whether each conjunct can be evaluated during decoding. This is the PushdownChecker in projection_read_plan.rs, which walks the expression tree and classifies every column reference it finds.
Order the survivors by cost and install them as an arrow-rs RowFilter.

What “supported” means

The checker is deliberately conservative. A conjunct can be pushed down if every column it touches is a primitive column (projection_read_plan.rs:324-340), or a list column used by one of the registered list predicates:

 rust
 
 
// datafusion/datasource-parquet/src/supported_predicates.rs:81-86
fn is_supported_scalar_function(expr: &dyn PhysicalExpr) -> bool {
 scalar_function_name(expr).is_some_and(|name| {
 // Registry of verified array functions
 matches!(name, "array_has" | "array_has_all" | "array_has_any")
 })
}

or a struct column accessed field-by-field through get_field where the resolved leaf is a primitive:

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:64-66
//! For example, given a struct column `s {name: Utf8, value: Int32}`:
//! - `WHERE s['value'] > 5` — pushed down (accesses a primitive leaf)
//! - `WHERE s IS NOT NULL` — not pushed down (references the whole struct)

and the column must actually exist in the file. A conjunct referencing a missing column sets projected_columns = true and blocks pushdown (projection_read_plan.rs:324-340).

The struct case is where the leaf-level projection pays off. A filter on s['value'] must not drag the entire struct into the read; DataFusion records the access path — root column index plus the field names forming the path — and clips the read to the leaves it touches (projection_read_plan.rs:72-78). A trie of these accesses is built so that shared prefixes collapse (projection_read_plan.rs:112-155), and the resulting leaf set becomes the ProjectionMask installed on the decoder for that predicate.

Ordering the predicates: bytes, not “selectivity”

The module documentation is explicit about the heuristic:

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:32-41
//! Since the predicates are applied serially in the order defined in the
//! `RowFilter`, the optimal ordering depends on the exact filters. The best
//! filters to execute first have two properties:
//!
//! 1. They are relatively inexpensive to evaluate (e.g. they read
//! column chunks which are relatively small)
//!
//! 2. They filter many (contiguous) rows, reducing the amount of decoding
//! required for subsequent filters and projected columns

Property 1 is what DataFusion implements, and it implements it literally: the cost of a predicate is the compressed size of the columns it reads, summed over all row groups in the file.

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:387-402
fn size_of_columns(columns: &[usize], metadata: &ParquetMetaData) -> Result<usize> {
 let mut total_size = 0;
 let row_groups = metadata.row_groups();
 for idx in columns {
 for rg in row_groups.iter() {
 total_size += rg.column(*idx).compressed_size() as usize;
 }
 }
 Ok(total_size)
}

The metric is bytes, not row count, and that matters: a filter on a narrow, highly compressible string column is cheap to evaluate even if it matches nothing, while a filter on a wide, poorly compressible column is expensive per row. Sorting candidates by required_bytes puts the cheap-but-selective filter first:

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:544-547
let mut ordered: Vec<&PrebuiltRowFilterCandidate> = prebuilt.iter().collect();
if reorder_predicates {
 ordered.sort_unstable_by_key(|c| c.required_bytes);
}

Property 2 (selectivity) is deliberately not used, because DataFusion cannot know it: it has no histogram, only min/max. Bytes is knowable from the footer.

The expensive part: don’t rebuild the filter per row group

The filter has to be rebuilt during a scan, because the scan may skip row groups at runtime and the decoder is rebuilt at each boundary. Doing the full construction (tree walks, column resolution, reassign_expr_columns) per row group showed up as a top hot spot in TPC-H profiles. DataFusion split the work in two:

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:477-486
/// Precompute the list of [`PrebuiltRowFilterCandidate`]s for a predicate.
///
/// This is the expensive part of [`build_row_filter`]: split into conjuncts,
/// resolve columns for each conjunct against the file schema, reassign
/// `Column` indices, and compute the sort-order metadata. Doing it once per
/// file (and reusing across row groups) avoids repeated `TreeNode::transform`
/// walks and `Arc<PhysicalExpr>` allocations that showed up as top hot spots
/// in TPCH profiles.

The prebuilt candidate holds an expression whose Column indices have already been rewritten to point into the projected schema, plus the resolved ProjectionMask. Reinstalling the filter is then just “clone some Arcs and allocate counters”:

 rust
 
 
// datafusion/datasource-parquet/src/row_filter.rs:529-537
/// Wrap a list of prebuilt candidates into a fresh [`RowFilter`], assigning
/// per-predicate metric counters and (optionally) reordering by
/// `required_bytes`. This is the cheap per-row-group rebuild path — no tree
/// walks, no column resolution, only counter allocation.

RowFilterContext (push_decoder.rs:351-412) is the cache that makes this possible: it owns the shared Arc<Vec<PrebuiltRowFilterCandidate>> and can mint a fresh RowFilter at any row-group boundary.

Late materialization, taken one step further

There is a second, cheaper form of late materialization available when statistics are good enough: prove that every row in a row group matches, and skip the filter entirely for that row group.

At open time, if the first row group to be read is fully matched, DataFusion installs an empty RowFilter and lets arrow-rs’s “does this decoder have predicates?” check short-circuit the per-row evaluation:

 rust
 
 
// datafusion/datasource-parquet/src/opener/mod.rs:1784-1795
if let Some(ctx) = row_filter_context.as_ref() {
 if first_rg_fully_matched {
 // The first RG is fully matched: install an empty filter
 // and count the suppression ...
 builder = builder.with_row_filter(
 parquet::arrow::arrow_reader::RowFilter::new(vec![]),
 );
 row_filter_skipped_fully_matched.add_one();
 } else {
 builder = builder.with_row_filter(ctx.build_row_filter());

Mid-scan, the same trick is applied per row group: when the decoder reaches the boundary of a fully-matched row group, the stream rebuilds the decoder with the filter removed; when it reaches a row group that does need filtering, the filter is reinstalled from the cached candidates.

 rust
 
 
// datafusion/datasource-parquet/src/push_decoder.rs:685-703
if filter_needs_toggle {
 let want_filter = desired_filter.expect("filter_needs_toggle ⇒ desired Some");
 if want_filter {
 let ctx = self.row_filter_context.as_ref()...;
 builder = builder.with_row_filter(ctx.build_row_filter());
 ...
 } else {
 // Skip per-row filtering for the upcoming fully-matched RG.
 builder = builder.with_row_filter(RowFilter::new(vec![]));
 self.filter_installed = false;
 self.row_filter_skipped_fully_matched.add_one();
 }
}

The rebuild is the expensive part, so DataFusion does at most one into_builder rebuild per row-group boundary, no matter how many row groups were dropped (push_decoder.rs:467-509).

Finally, the decoded batch is projected onto the query’s output schema by a per-batch transform (decoder_projection.rs:141-160), which rebuilds the batch only when the projector’s schema differs from the scan’s output schema in metadata or nullability.

The catch

The residual filter is not a cop-out; it is a correctness requirement. DataFusion only claims a filter is “exact” for the conjuncts it can actually evaluate during decoding:

 rust
 
 
// datafusion/datasource-parquet/src/source.rs:682-694
fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
 if !self.pushdown_filters() { return None; }
 let predicate = self.predicate.as_ref()?;
 let pushable_schema = self.table_schema.schema_without_virtual_columns();
 let exact = split_conjunction(predicate)
 .into_iter()
 .filter(|expr| can_expr_be_pushed_down_with_schemas(expr, pushable_schema))
 .cloned()
 .collect::<Vec<_>>();
 (!exact.is_empty()).then(|| conjunction(exact))
}

and the pushdown protocol marks the unsupported conjuncts as not pushed down, so the parent FilterExec keeps them:

 rust
 
 
// datafusion/datasource-parquet/src/source.rs:853-872
let filters: Vec<PushedDownPredicate> = filters.into_iter()
 .map(|filter| {
 if can_expr_be_pushed_down_with_schemas(&filter, pushable_schema) {
 PushedDownPredicate::supported(filter)
 } else {
 PushedDownPredicate::unsupported(filter)
 }
 })
 .collect();

This matters twice: a conjunct that was not evaluated during decoding must still be evaluated somewhere, or the query returns wrong rows; and exact_filter is what FileScanConfig uses to derive equivalence properties — and a filter that is only used for pruning does not remove non-matching rows, so it must not be advertised as exact:

 rust
 
 
// datafusion/datasource/src/file.rs:113-128
/// Returns the part of [`Self::filter`] that every output row is
/// guaranteed to satisfy.
/// ...
/// A filter that the source uses only to skip data (for example, row group
/// or page pruning with statistics) does not remove every non-matching row,
/// so it must not be returned here.

Other caveats:

Pushdown is off by default. Both datafusion.execution.parquet.pushdown_filters and the per-table option default to false. Pushdown changes the shape of the scan, and for some workloads (a predicate that matches almost everything, on a very wide table) the extra pass over the filter columns is a net loss. reorder_filters is a separate knob for the ordering heuristic.
Late materialization can cost a second pass. If the predicate is unselective, you have paid for an extra scan of the filter columns and decoded everything anyway.
Nested types are a minefield. The rules for lists and structs are per-expression, not per-column: array_has_all(col, ...) is pushable, col[1] = 5 may not be, and col IS NOT NULL on a struct is not pushable at all. Each of those distinctions is a place where a bug would produce wrong answers rather than slow queries.
Virtual columns cannot participate. Predicates referencing the reader-produced row_number column are excluded from pushdown (source.rs:830-845): the reader cannot evaluate them against a mask it has not produced yet.

Parquet on Opensource internals

Writing Parquet Row Groups in Parallel: Column Chunks, Dictionary Pages and the Sort-Before-Write Trick

The shape of the write path

Two writer strategies

Parallel column encoding inside one file

Row groups are sealed concurrently, concatenated in order

Row group sizing

Dictionary encoding and statistics: write-side decisions, read-side wins

The sort-before-write trick

The catch

Polars Publishes a Join's Build Side as a Runtime Filter

The plan-time side: finding a scan to talk to

Which joins qualify

Which side becomes the build

Tracing the key to a scan

Attaching the slot to the scan

The run-time side: what the join publishes

When the filter is published

The bloom filter

The consumer: the scan

One honest limitation

The bottom line

From SQL WHERE to Parquet Row Filter: How DataFusion Decides What It Can Push Down (and What It Must Keep)

One predicate, two questions

Question 1: which predicates can prune row groups?

LIKE becomes a range

IN lists and the size cap

The unhandled hook: the constant true

AND is forgiving, OR is not

Question 2: which predicates can the decoder evaluate?

The residual filter: why “keep it” is mandatory

The catch

Polars Prunes Parquet Row Groups With Chunk Statistics, Not the Page Index

Which reader is the real one

The columns in the plan

The statistics become a DataFrame

Where the numbers come from

The “already applied” flag

Ranges that arrive late

The page index: written, not read

The bottom line

Polars Writes Parquet From a Streaming Pipeline

The API

The sink node

The resize pipeline: morsels are the wrong size

Deciding the row group size

The defaults

Three tasks: resize, encode, write

The encoder: parallel across columns

The writer: one task, one file

What is parallel and what is serialised

Partitioned sinks

The old, non-streaming path

The bottom line

DataFusion's Parquet Access Plan: How Row Groups, Pages and Files Get Skipped Before a Byte Is Read

The naive reader

The access plan

The pipeline: state machine, not a pile of ifs

Zone maps: the row-group min/max proof

The inverse trick: proving that everything matches

Page index: from row groups to pages

Bloom filters: the last resort, and the false-positive contract

Range assignment: splitting a file for parallelism

The catch

DataFusion's Parquet Push Decoder: Pruning Row Groups in the Middle of a Scan You Already Started

Pull vs push

Row-group boundaries are the checkpoint

The row-group plan

The runtime pruner

The other half: stopping the whole file

The catch

Why this matters

Polars Pushes Predicates Into the Parquet Scanner

The optimizer hands the filter to the scan

The expression is compiled a second time

The statistics frame

Reading the real minimum out of the footer

The pruning loop

The flag that says “don’t apply it again”

The bottom line

`LIKE` becomes a range

`IN` lists and the size cap

The unhandled hook: the constant `true`

`AND` is forgiving, `OR` is not