All posts published on this site.
Everything published here, newest first.
DataFusion’s external sorter has two modes: sort the whole buffered input in one concatenated batch when it is small, and otherwise sort per-batch runs and merge them — coalescing …
DataFusion constant-folds expressions by running them against a one-row dummy batch of NULLs, and guards the trick with checks for volatility, leaf nodes, and regexes that would …
Instead of sorting N rows to keep k, DataFusion encodes sort keys in Arrow’s row format, keeps the k best rows in a binary heap, and publishes a dynamic filter so the scan below …
Exact DISTINCT becomes a GROUP BY on every column, and the approximate version keeps a 16 KiB HyperLogLog sketch per group, keyed by the hash of the value rather than the value …
DataFusion’s approx_percentile_cont compresses values into weighted centroids whose state can be merged across partitions — approximately, not exactly.