All posts published on this site.
Everything published here, newest first.
DataFusion’s decorrelator pulls a correlated filter out of a subquery and turns it into a join predicate, adding a __always_true indicator column so the empty-set semantics of …
Instead of hard-coding shuffles, each ExecutionPlan states the distribution it needs from its children; a single optimizer rule turns unmet requirements into RepartitionExec nodes.
DataFusion’s external sorter has two modes: sort the whole buffered input in one concatenated batch when it is small, and otherwise sort per-batch runs and merge them — coalescing …
DataFusion constant-folds expressions by running them against a one-row dummy batch of NULLs, and guards the trick with checks for volatility, leaf nodes, and regexes that would …
Instead of sorting N rows to keep k, DataFusion encodes sort keys in Arrow’s row format, keeps the k best rows in a binary heap, and publishes a dynamic filter so the scan below …