Group-By

Polars Estimates Cardinality With HyperLogLog

Oct 21, 2026 17 min read Jay Hubbert

The HyperLogLog register trick turns a 64-bit hash into an index and a rank, and Polars’ estimator replaces the classic bias correction with Ertl’s tau/sigma correction; a second, …

Polars Internals Joins Group by

Polars Encodes Group Keys as Rows to Make Hashing Cheap

Oct 21, 2026 13 min read Jay Hubbert

The row encoder writes each key column into a self-describing byte layout: a sentinel for null, big-endian bytes with the sign bit flipped for numbers, a terminator for strings. …

Polars Internals Group by Performance Arrow

Polars Aggregates Faster When the Keys Are Already Sorted

Oct 21, 2026 14 min read Jay Hubbert

The eager group-by checks the column’s IsSorted flag before it builds anything. If the keys are sorted, the groups are contiguous row ranges, produced by a run-length scan that can …

Polars Internals Group by Performance Sorting

Polars Groups Columns So One Pass Feeds Many Aggregations

Oct 21, 2026 16 min read Jay Hubbert

Grouping in Polars is a separate stage: keys are encoded (single key, key rows, row encoding, or binview), hashed into group ids by a Grouper, and every aggregation expression then …

Polars Internals Group by Performance Arrow

Polars Hash-Shuffles Rows Between Streaming Stages

Oct 21, 2026 19 min read Jay Hubbert

Rows are partitioned by hash(key) into N buckets, so equal keys always land in the same bucket. Each bucket is aggregated or joined independently, which is a shuffle instead of a …

Polars Internals Streaming Group by Joins Performance