Signatures, Volatility, and the Type-Coercion Lattice: How DataFusion's UDF Metadata Drives the Optimizer
A UDF’s Signature is three orthogonal promises: which types it accepts, how it coerces them, and whether its result is repeatable.
Functions
DataFusion's Aggregate Split: One UDF, Two Accumulator Traits, and a Serializable State Format
DataFusion aggregates are state machines: update_batch feeds them rows, state() serializes them, merge_batch combines them across partitions.
Once Per Batch: How DataFusion's ColumnarValue API Killed the Per-Row UDF Call
Scalar UDFs receive whole Arrow arrays per batch, not one ScalarValue per row — ColumnarValue is the currency that makes that possible.
How DataFusion Approximates Quantiles with a Mergeable t-Digest (and Distinct Counts with HyperLogLog)
DataFusion’s approx_percentile_cont compresses values into weighted centroids whose state can be merged across partitions — approximately, not exactly.