Polars Buffers Are Immutable: The Refcounted Storage Behind Every Column
The buffer layer of Polars is deliberately dumb: an immutable (ptr, length) view over a hand-rolled refcounted storage, with all growth pushed into the mutable builders.
Posts
All posts published on this site.
Everything published here, newest first.
A Memory Ordering Bug in DataFusion's Hash Join, Found by Model Checking Rather Than Testing
DataFusion’s hash join probe partitions coordinate through a single AcqRel counter decrement; a subtle ordering bug was found with loom’s model checker and cannot be reproduced by …
Coalesce Batches, Don't Shrink Them: How DataFusion Rebuilds 8K-Row Batches Out of 3-Row Ones
Tiny batches destroy vectorized throughput; DataFusion merges them with arrow-rs BatchCoalescer, wrapped in a LIMIT-aware coalescer that slices instead of copying, and fused into …
Multi-Level Merge: How DataFusion Sorts More Data Than It Can Hold, One Pass at a Time
DataFusion’s multi-level merge merges the sorted runs it can afford, spills the merged result back to disk, and repeats until one pass remains, re-spilling skewed runs whose widest …
EXPLAIN ANALYZE Without the Overhead: How DataFusion Times Every Operator While the Query Runs
DataFusion records cheap atomic metrics per operator partition, measures only the time spent inside poll(), and renders the annotated plan after the query finishes — so EXPLAIN …