Functions on Opensource internals

Signatures, Volatility, and the Type-Coercion Lattice: How DataFusion's UDF Metadata Drives the Optimizer

Jay Hubbert — Wed, 21 Oct 2026 08:53:00 +0000

TL;DR: Signature is the metadata contract between a UDF and the engine. TypeSignature says which argument types have implementations; the coercion lattice turns “user called cos(int_col)” into “cast to Float64 and call the Float64 implementation”. Volatility is the second half of the contract: Immutable/Stable functions may be constant-folded at plan time and reordered by the optimizer; Volatile ones may not, and the optimizer goes out of its way to keep them from being duplicated, hoisted or pushed below an aggregate. And because ScalarUDF wraps Arc<dyn ScalarUDFImpl>, the virtual call happens once per batch, not once per row.

Two problems one struct solves

A scalar UDF in DataFusion has to answer three questions before it can run:

What argument types do you implement? (cos handles Float32/Float64; make_array handles anything.)
What do I do with the types I don’t implement? (Cast the input? Error?)
May I call you twice and get the same answer? (abs(-1) yes, random() no.)

The first two are the “type” question and the third is the “time” question. They are orthogonal, and DataFusion keeps them in one struct because they are always needed together — during planning, during coercion, and during optimization:

 rust
 
 
// datafusion/expr-common/src/signature.rs:1410
pub struct Signature {
 /// The data types that the function accepts. See [TypeSignature] for more information.
 pub type_signature: TypeSignature,
 /// The volatility of the function. See [Volatility] for more information.
 pub volatility: Volatility,
 /// Optional parameter names for the function arguments.
 pub parameter_names: Option<Vec<String>>,
}

The constructors make the common cases one-liners (signature.rs:1426-1540): Signature::exact, Signature::uniform, Signature::variadic, Signature::user_defined, Signature::one_of, Signature::comparable, Signature::coercible, Signature::numeric, Signature::string, Signature::nullary, Signature::any, Signature::variadic_any, plus the array-specialised ones (array_and_element, arrays, array_and_index, …).

`TypeSignature`: arity × type shape

 rust
 
 
// datafusion/expr-common/src/signature.rs:165
pub enum TypeSignature {
 Variadic(Vec<DataType>), // N args, each one of these types
 UserDefined, // the UDF does its own coercion
 VariadicAny, // N args, any types
 Uniform(usize, Vec<DataType>), // exactly N args, all the same type
 Exact(Vec<DataType>), // exactly N args, these types, in order
 Coercible(Vec<Coercion>), // per-argument coercion rules
 Comparable(usize), // N args coerced to one comparable type
 Any(usize), // N args, any types
 OneOf(Vec<TypeSignature>), // try each in order
 ArraySignature(ArrayFunctionSignature),
 Numeric(usize),
 String(usize),
 Nullary, // no arguments
}

The doc comment on the enum is the important part: TypeSignature does not define the types a user may call the function with. It defines the types you wrote code for. Everything else is the engine’s problem, and the engine solves it by inserting casts.

OneOf deserves special mention because it is how a function supports genuinely different shapes. approx_percentile_cont uses it for the 2-argument and 3-argument forms (datafusion/functions-aggregate/src/approx_percentile_cont.rs:137-172):

 rust
 
 
let signature = Signature::one_of(
 vec![
 // 2 args - numeric, percentile (float)
 TypeSignature::Coercible(vec![
 Coercion::new_implicit(
 TypeSignatureClass::Float,
 vec![TypeSignatureClass::Numeric],
 NativeType::Float64,
 ),
 Coercion::new_implicit_native(
 logical_float64(),
 vec![TypeSignatureClass::Numeric],
 ),
 ]),
 // 3 args - numeric, percentile (float), number of centroids (integer)
 // ...
 ],
 Volatility::Immutable,
);

The coercion lattice

The interesting question is: given cos(some_int32_column), what happens?

The entry point is fields_with_udf (datafusion/expr/src/type_coercion/functions.rs:110), which is called during planning for every function call:

 rust
 
 
// datafusion/expr/src/type_coercion/functions.rs:136
let valid_types = get_valid_types_with_udf(type_signature, &current_types, func)?;
if valid_types.iter().any(|data_type| data_type == &current_types) {
 return Ok(current_fields.to_vec()); // already exact: no casts
}
let updated_types =
 try_coerce_types(func.name(), valid_types, &current_types, type_signature)?;

Three steps: enumerate the valid type combinations for the signature, check whether the call already matches one, otherwise find a combination we can cast to.

get_valid_types_with_udf (functions.rs:532) special-cases two signatures: UserDefined delegates straight to the UDF’s own coerce_types hook, and OneOf tries each alternative and collects the successes (so one_of with two 3-arg forms can match either). Everything else goes to get_valid_types, which expands the signature into concrete DataType vectors.

Then try_coerce_types (functions.rs:494) picks a target. It has two paths, and the distinction matters:

 rust
 
 
// datafusion/expr/src/type_coercion/functions.rs:474
fn is_well_supported_signature(type_signature: &TypeSignature) -> bool {
 match type_signature {
 TypeSignature::OneOf(type_signatures) => type_signatures.iter().all(is_well_supported_signature),
 TypeSignature::UserDefined | TypeSignature::Numeric(_) | TypeSignature::String(_)
 | TypeSignature::Coercible(_) | TypeSignature::Any(_) | TypeSignature::Nullary
 | TypeSignature::Comparable(_) => true,
 TypeSignature::Variadic(_) | TypeSignature::VariadicAny | TypeSignature::Uniform(_, _)
 | TypeSignature::Exact(_) | TypeSignature::ArraySignature(_) => false,
 }
}

The “well supported” signatures (the new Coercible/Numeric/String/ Comparable family) have already resolved to a single exact target type, so coercion is a direct cast. The legacy signatures (Exact, Uniform, Variadic) take the old path, which searches for any valid combination and casts to it — more expensive, more magic, and the source of most of the “why did my function get a CAST I didn’t ask for” confusion in DataFusion.

Finding the common type

The actual lattice lives in datafusion/expr-common/src/type_coercion/binary.rs. type_union_resolution (:708) computes the common type of a set of argument types, Postgres/DuckDB style:

 rust
 
 
// datafusion/expr-common/src/type_coercion/binary.rs:719
// If all the data_types are null, return string
if data_types.iter().all(|t| t == &DataType::Null) {
 return Some(DataType::Utf8View);
}
// Ignore Nulls, if any data_type category is not the same, return None
// ...

and the pairwise step (:773) is a cascade of fallbacks:

 rust
 
 
// datafusion/expr-common/src/type_coercion/binary.rs:873
binary_numeric_coercion(lhs_type, rhs_type)
 .or_else(|| list_coercion(...))
 .or_else(|| map_coercion(...))
 .or_else(|| temporal_coercion_nonstrict_timezone(...))
 .or_else(|| string_coercion(...))
 .or_else(|| null_coercion(...))
 .or_else(|| string_numeric_coercion(...))
 .or_else(|| binary_coercion(...))

That is the “lattice”: numeric widening first (i32 + i64 → i64), then lists, then maps, then temporals, then strings, then the escape hatches that let a string be read as a number or a binary be read as a string. Order matters, and that order encodes a priority: it mirrors what a human would guess.

The Coercion enum (signature.rs:1164) is the modern, explicit way to say “this argument should end up as a Float64, and it may be implicitly cast from any numeric type”:

 rust
 
 
// datafusion/expr-common/src/signature.rs:1164
pub enum Coercion {
 Exact { desired_type: TypeSignatureClass, encoding_preservation: EncodingPreservation },
 Implicit {
 desired_type: TypeSignatureClass,
 implicit_coercion: ImplicitCoercion, // allowed_source_types + default_casted_type
 encoding_preservation: EncodingPreservation,
 },
}

TypeSignatureClass (signature.rs:371) is the coarse lattice of type families — Any, Timestamp, Time, Interval, Duration, Native(LogicalTypeRef), Integer, Float, Decimal, Numeric, Binary. Numeric is the union of Integer | Float | Decimal; Integer and Float are the two ways to spell “any numeric that is not a decimal”. Writing Coercion::new_implicit(TypeSignatureClass::Float, vec![TypeSignatureClass::Numeric], NativeType::Float64) says: “accept a float; if the user gave me an int or a decimal, cast it to Float64”. That is one line replacing a page of match arms.

`Volatility`: the promise about time

 rust
 
 
// datafusion/expr-common/src/signature.rs:56
pub enum Volatility {
 /// Always returns the same output when given the same input.
 Immutable,
 /// May return different values given the same input across different
 /// queries but must return the same value for a given input within a query.
 Stable,
 /// May change the return value from evaluation to evaluation.
 Volatile,
}

Three levels, and the difference between Immutable and Stable is subtle but real: Immutable says “same input, same output, forever” (so abs(-1) is folded to 1 at plan time), Stable says “same input, same output within this query” (now() is constant for the duration of one query but changes between queries), and Volatile says “no promises at all” (random()).

The optimizer uses this in two distinct ways.

1. Constant folding. The simplifier’s ConstEvaluator refuses to evaluate volatile functions at plan time:

 rust
 
 
// datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:617
fn volatility_ok(volatility: Volatility) -> bool {
 match volatility {
 Volatility::Immutable => true,
 // Values for functions such as now() are taken from ExecutionProps
 Volatility::Stable => true,
 Volatility::Volatile => false,
 }
}

can_evaluate (expr_simplifier.rs:626) then refuses the whole subtree containing a volatile node, so 1 + random() is not folded and 1 + abs(-1) is. Folding is not just a nicety: it turns a per-row computation into a constant, which lets the optimizer push predicates around it, drop columns, and use statistics.

2. “May I duplicate or move this expression?” Plenty of optimizations are only valid if the expression means the same thing wherever it ends up. DataFusion encodes that check with Expr::is_volatile:

 rust
 
 
// datafusion/expr/src/expr.rs:2208
pub fn is_volatile_node(&self) -> bool {
 let volatility = match self {
 Expr::ScalarFunction(func) => func.func.signature().volatility,
 Expr::AggregateFunction(func) => func.func.signature().volatility,
 // ...
 };
 volatility == Volatility::Volatile
}

Note the distinction between is_volatile_node (this node only) and is_volatile (:2231, the whole subtree). Rewrites use the subtree version, because moving a subtree that contains a volatile node is just as wrong as moving the volatile node itself.

Concretely, the filter pushdown refuses to push a predicate into the grouping keys when the key is volatile:

 rust
 
 
// datafusion/optimizer/src/push_down_filter.rs:1117
// We can push down Predicate which in groupby_expr.
// Volatile group keys are excluded: below the aggregate, the
// predicate would evaluate the key again and see a different
// value than the one used for grouping.
let non_volatile_group_exprs: Vec<Expr> = agg
 .group_expr
 .iter()
 .filter(|expr| !expr.is_volatile())
 .cloned()
 .collect();

That is a correctness fix, not a pessimization: GROUP BY random() would group by one value and then filter on a different random value if the predicate were pushed down. Similarly, outer-join elimination only fires when the join condition is not volatile (datafusion/optimizer/src/eliminate_join.rs:1036), and common-subexpression elimination skips volatile output columns (datafusion/optimizer/src/extract_leaf_expressions.rs:109).

The cost of getting this wrong is asymmetric and brutal:

Declare a non-deterministic function Immutable → the planner folds it to a literal → every row of the query gets the same value. Silently wrong.
Declare a deterministic function Volatile → no folding, no predicate pushdown below aggregates, no CSE, no join elimination. Correct but mysteriously slow, and the profiler will not tell you why.

The rule of thumb in the doc comment is the right one: “You should always define your function to have the strictest possible volatility to maximize performance and avoid unexpected results.”

The dispatch cost: one virtual call per batch, not per row

ScalarUDF is a thin newtype over a trait object:

 rust
 
 
// datafusion/expr/src/udf.rs:86
pub struct ScalarUDF {
 inner: Arc<dyn ScalarUDFImpl>,
}

The vtable lookup happens in ScalarFunctionExpr::evaluate (datafusion/physical-expr/src/scalar_function.rs:236), which evaluates each child expression once and hands the whole batch to the function:

 rust
 
 
// datafusion/physical-expr/src/scalar_function.rs:236
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 let args = self.args.iter().map(|e| e.evaluate(batch)).collect::<Result<Vec<_>>>()?;
 // ...
 let output = self.fun.invoke_with_args(ScalarFunctionArgs {
 args,
 arg_fields,
 number_rows: batch.num_rows(),
 return_field: Arc::clone(&self.return_field),
 config_options: Arc::clone(&self.config_options),
 })?;
 // ...
}

So the dynamic dispatch is O(1) per batch; the body of the function works on ArrayRefs and uses Arrow’s vectorized kernels. A UDF written naively — one ScalarValue in, one ScalarValue out — would instead be O(rows) virtual calls with a ScalarValue allocation per row, and would be the slowest thing in the engine.

There is one more escape hatch worth knowing about, for when the optimizer needs to reason about a specific function rather than a name:

 rust
 
 
// datafusion/physical-expr/src/scalar_function.rs:165
pub fn try_downcast_func<T>(expr: &dyn PhysicalExpr) -> Option<&ScalarFunctionExpr>
where T: ScalarUDFImpl

It matches on the concrete ScalarUDFImpl type, not the function name, so GetFieldFunc can be recognised even if a user registered a different function under the same name. The parquet reader uses it to recognise get_field and file_row_index/input_file_name (datafusion/datasource-parquet/src/projection_read_plan.rs:418,511).

The registry

All of this metadata hangs off the function object, and the functions live in a registry:

 rust
 
 
// datafusion/expr/src/registry.rs:40
pub trait FunctionRegistry {
 fn udf(&self, name: &str) -> Result<Arc<ScalarUDF>>;
 fn udaf(&self, name: &str) -> Result<Arc<AggregateUDF>>;
 fn udwf(&self, name: &str) -> Result<Arc<WindowUDF>>;
 fn register_udf(&mut self, _udf: Arc<ScalarUDF>) -> Result<Option<Arc<ScalarUDF>>>;
 // ...
}

datafusion/functions/src/lib.rs:166 builds the default set (all_default_functions, chaining the per-package functions() lists) and :180 (register_all) installs them into any registry, logging an overwrite if a name collides. Because the registry hands out Arc<ScalarUDF>, the Signature is computed once and shared by every plan that uses the function — the coercion lattice is a planning-time cost, paid once per query, not a per-row cost.

Trade-offs

Metadata is a promise you can break. Signature is unchecked by the compiler. A function that lies about its arity or its volatility produces wrong answers, not errors.
The coercion lattice is a planning-time tax. Every function call enumerates valid type combinations and searches for a cast path. The OneOf and legacy Exact/Uniform paths are the expensive ones; the modern Coercible/Numeric/String/Comparable signatures exist partly to make that search trivial.
OneOf order is semantics. The first matching alternative wins, so reordering the alternatives in a signature can change which implementation runs — and therefore the result type and the accuracy.
Volatility::Stable is under-used. now() is Stable; most UDF authors reach for Immutable or Volatile and never think about the middle rung, which is exactly the rung that says “constant for this query”.
The Arc<dyn ScalarUDFImpl> boundary is a real, if small, cost. It is paid once per batch per function node. For a query that evaluates a function on 8 KiB batches, that is measurable but small; for one that evaluates it on million-row batches, it is noise. Which is another way of saying: keep the batch big, and the dispatch model takes care of itself.

DataFusion's Aggregate Split: One UDF, Two Accumulator Traits, and a Serializable State Format

Jay Hubbert — Wed, 21 Oct 2026 07:33:00 +0000

TL;DR: An aggregate in DataFusion is not a function, it is a state machine: update_batch folds rows in, state() serializes the machine into Arrow arrays, merge_batch folds another machine’s state back in, and evaluate() produces the answer. That split is what makes partial/final two-phase aggregation possible. GroupsAccumulator is the vectorized sibling: one object holds the state of all groups, so there is no per-group virtual call and no per-group take. GroupsAccumulatorAdapter is the compatibility shim that lets a plain Accumulator pretend to be a GroupsAccumulator — at the price of copying rows into per-group contiguous buffers on every batch.

The problem: an aggregate is two different computations

Write SELECT avg(x) GROUP BY k over 100 million rows in 8 partitions. Two completely different jobs hide behind that one line of SQL:

Folding: for each row, update the state of the group the row belongs to.
Collapsing: for each group, turn its accumulated state into a single value.

The naive implementation does (1) with a function call per row, holding a HashMap<k, State> in memory. That is fine on a toy dataset and catastrophic on a real one, because the per-row call crosses a dynamic-dispatch boundary, the HashMap thrashes, and the state cannot be shipped anywhere.

DataFusion’s answer is to make the aggregate a value that can be serialized, shipped over a network, and merged — and then to compute it in a way that never calls a function per row.

The `Accumulator` trait: one group, one object

The whole contract is 200 lines in datafusion/expr-common/src/accumulator.rs:

 rust
 
 
// datafusion/expr-common/src/accumulator.rs:123
pub trait Accumulator: Send + Sync + Debug + std::any::Any {
 fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()>; // :140
 fn evaluate(&mut self) -> Result<ScalarValue>; // :193
 fn size(&self) -> usize; // :205
 fn state(&mut self) -> Result<Vec<ScalarValue>>; // :374
 fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()>; // :387
 fn retract_batch(&mut self, _values: &[ArrayRef]) -> Result<()> { // :421
 internal_err!("Retract should be implemented ...")
 }
}

Read it as a little algebra. update_batch is the operation, state() is the encoding, merge_batch is the homomorphism over that encoding, and evaluate is the projection of the state into a final value. For SUM the operation and the merge are the same thing (state is the running sum). For COUNT they are not: the state is a count, and merging counts means adding them, not re-counting the underlying rows. For AVG the state is two values — sum and count — and the merge adds both, then evaluate divides. The trait documentation spells this out at accumulator.rs:260-266:

The state can be and often is a different type than the output type of the Accumulator and needs different merge operations (for example, the partial state for COUNT needs to be summed together)

That sentence is the whole reason the trait exists. If SUM(x) were just a function f(accumulator, row) -> (), there would be no way to run it in two phases. Because state() is a separate, serializable representation, the planner can insert a Partial aggregate before a shuffle and a Final aggregate after it, and the two halves only need to agree on the state format.

The two-phase flow

text

 ┌──────────────────────────┐
 │ Partial (per partition) │
 │ update_batch() per row │
 │ state() -> Arrow arrays│
 └────────────┬─────────────┘
 │ (shuffle / repartition)
 ┌────────────▼─────────────┐
 │ Final │
 │ merge_batch(states) │ <- fold partition states together
 │ evaluate() -> value │
 └──────────────────────────┘

The state is plain Arrow data, so it crosses a RecordBatch boundary for free. AVG emits two state columns (sum, count); MEDIAN cannot know its state arity at planning time, so it packs everything into one ScalarValue::List (accumulator.rs:270-273).

The subtlety is that the final phase is not “run the function again on the state”. It is “fold state into state”. merge_batch receives the state()-encoded arrays of other accumulators, not raw rows. If you write an aggregate whose merge_batch is not the algebraic dual of update_batch, you get answers that depend on how many partitions the data had — the classic “works on my laptop” aggregate bug.

`GroupsAccumulator`: the fast path that avoids per-group calls

Now the performance trick. Accumulator handles one group. To aggregate 10 million groups you would need 10 million Box<dyn Accumulator> objects, and every row would cost you one virtual call plus a hash-map-ish lookup.

GroupsAccumulator (datafusion/expr-common/src/groups_accumulator.rs:193) flips this: one object holds the state of all groups, and the caller hands it a vector of group indices:

 rust
 
 
// datafusion/expr-common/src/groups_accumulator.rs:221
fn update_batch(
 &mut self,
 values: &[ArrayRef],
 group_indices: &[usize],
 opt_filter: Option<&BooleanArray>,
 total_num_groups: usize,
) -> Result<()>;

There is no per-group call at all. group_indices[i] says “row i belongs to group group_indices[i]”, and the implementation is expected to fan that out with a tight loop over primitive values. The canonical example is PrimitiveGroupsAccumulator (datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:43):

 rust
 
 
// datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:102
self.values.resize(total_num_groups, self.starting_value);
self.null_state.accumulate(
 group_indices,
 values,
 opt_filter,
 total_num_groups,
 |group_index, new_value| {
 // SAFETY: group_index is guaranteed to be in bounds
 let value = unsafe { self.values.get_unchecked_mut(group_index) };
 (self.prim_fn)(value, new_value);
 },
);

Three things make this fast: the state is a flat Vec<T::Native> indexed by group (cache friendly, no hashing, no allocation per group); prim_fn is a monomorphized closure, so SUM on Int64 compiles to a bare += with no vtable lookup; and the null/filter handling is hoisted out of the inner loop by accumulate (groups_accumulator/accumulate.rs:421), which has four specialized paths (no nulls/no filter, nulls/no filter, no nulls/filter, both) and even processes the null bitmap 64 bits at a time.

NullState (accumulate.rs:134) solves the awkward part: a group that has never seen a value must still produce a row (NULL for SUM, 0 for COUNT). Rather than every aggregate reinventing “did this group see anything”, the shared NullState tracks it and produces the null buffer at evaluate time.

SUM opts in with two lines (datafusion/functions-aggregate/src/sum.rs:301):

 rust
 
 
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
 !args.is_distinct
}

fn create_groups_accumulator(&self, args: AccumulatorArgs) -> Result<Box<dyn GroupsAccumulator>> {
 // ... PrimitiveGroupsAccumulator::<$t, _>::new(&$dt, |x, y| *x = x.add_wrapping(y))
}

The adapter: making old aggregates look new

Not every aggregate has a GroupsAccumulator, and writing one is genuinely hard (the state must be a columnar layout, EmitTo must be honoured, memory must be tracked). So DataFusion ships a bridge: GroupsAccumulatorAdapter (datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:95).

It holds a Vec<AccumulatorState>, one Box<dyn Accumulator> per group, and bridges the vectorized API to the per-group one by reordering the rows:

text

 input rows after take_arrays per-group call
 ┌─────┐ ┌─────────┐
 │ 2 │ │ 200 │ ──┐
 │ 0 │ take ────▶ │ 300 │ ─┤ group 0
 │ 1 │ │ 200 │ ─┘
 │ 0 │ │ 100 │ ──┐
 └─────┘ │ NULL │ ─┘ group 1
 └─────────┘

The implementation (groups_accumulator.rs:281-470) builds a permutation array of u32 row indices, calls Arrow’s compute::take_arrays to produce per-group-contiguous values, and then calls the accumulator once per group through Box<dyn Accumulator>. So the adapter pays:

one take (a full copy of the batch’s values) per update batch, and
one virtual call per group per batch, plus one Accumulator allocation per group.

That is still vastly better than a virtual call per row, but it is dramatically worse than the GroupsAccumulator fast path. The selection between the two happens once, at stream construction (datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:64):

 rust
 
 
pub(in crate::aggregates) fn create_group_accumulator(
 agg_expr: &Arc<AggregateFunctionExpr>,
 metrics: Arc<dyn AggregateMetrics>,
) -> Result<Box<dyn GroupsAccumulator>> {
 if agg_expr.groups_accumulator_supported() {
 agg_expr.create_groups_accumulator_with_metrics(metrics)
 } else {
 debug!("Creating GroupsAccumulatorAdapter for {}: {agg_expr:?}", agg_expr.name());
 // ... GroupsAccumulatorAdapter::new(move || agg_expr.create_accumulator())
 }
}

The debug! line is a hint to anyone profiling: if you see it, you are on the slow path.

`EmitTo`, `take_state`, and the “emit the first N groups” dance

Aggregates cannot always emit everything at once — the hash table drains in batches to respect batch_size. EmitTo (groups_accumulator.rs:31) encodes this:

 rust
 
 
pub enum EmitTo {
 All,
 /// Emit only the first `n` groups and shift all existing group indexes down by `n`.
 First(usize),
}

EmitTo::take_needed (groups_accumulator.rs:52) implements the shift with std::mem::take for All (no copy) and split_vec_min_alloc for First(n). This is why GroupsAccumulator::state takes an emit_to while Accumulator::state does not: the grouped version has to renumber the groups it did not emit, and that renumbering has to be reflected in the hash table’s own group vector. A subtle, easy-to-get-wrong contract, and one more reason the grouped API is opt-in.

first_last has its own take_state helper (datafusion/functions-aggregate/src/first_last.rs:559) that splits three parallel buffers (values, orderings, “is set” bitmap) in lockstep — the same “emit N, keep the rest” problem, but with three columns instead of one.

`convert_to_state`: skipping the hash table entirely

There is one more trick worth stealing. Normally a partial aggregate builds a hash table: rows come in, get hashed, and land in a group slot. But sometimes you already know the grouping key will not reduce anything — e.g. a partial aggregate before a GROUP BY that is only there to pre-compute a state for a downstream aggregate.

GroupsAccumulator::convert_to_state (groups_accumulator.rs:370) converts rows directly into state rows, one state row per input row, with no hash table at all:

 rust
 
 
// datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:669
fn convert_to_state(&self, values: &[ArrayRef], opt_filter: Option<&BooleanArray>)
 -> Result<Vec<ArrayRef>>
{
 // ... for each row: create a fresh accumulator, feed it that one row,
 // then harvest accumulator.state()
}

The PartialSkip mode uses it (datafusion/physical-plan/src/aggregates/aggregate_hash_table/partial_table.rs:128, datafusion/physical-plan/src/aggregates/hash_stream.rs:516). The cost is obvious in hindsight: one accumulator allocation and one state serialization per row. The benefit: no hash table, no group vector, no group-index bookkeeping, and the output is already in the shape the next phase wants.

Trade-offs and failure modes

Two implementations to keep in sync. Every aggregate that wants the fast path must implement both Accumulator and GroupsAccumulator, and the two must agree. A GroupsAccumulator that is subtly wrong produces different answers than the Accumulator path — and the optimizer may pick either one depending on whether DISTINCT is present (see sum.rs:301).
The adapter’s copy is real. The take per batch is O(rows) memory traffic; for wide batches of expensive-to-copy types (strings, lists) this is the dominant cost of the “slow” path, not the virtual call.
evaluate must not consume state. The trait docs are emphatic (accumulator.rs:179-190): evaluate is called repeatedly for window functions with different frames, so it may reorder internal state but must not consume it. Violating this gives correct answers for GROUP BY and garbage for OVER (...).
size() is a hot path. The adapter deliberately maintains allocation_bytes incrementally rather than calling size() per group per batch, with a comment saying exactly that (“We saw size() being a bottleneck in earlier implementations when there were many distinct groups”, groups_accumulator.rs:101-106). If you implement Accumulator, make size() cheap or return a constant.
State format is a public contract. state_fields defines the Arrow schema of the intermediate state, and it is versioned with the engine. If you change a state format you break distributed plans that were planned by an older binary. approx_percentile_cont packs its state into six scalars plus a list (datafusion/functions-aggregate/src/approx_percentile_cont.rs:250) precisely because the state must round-trip through Arrow.

The takeaway: the split into update/state/merge/evaluate is not ceremony, it is what turns “aggregate function” into “algebraic data type” — something you can compute in parallel, ship across a network, and merge without ever knowing the original rows.

Once Per Batch: How DataFusion's ColumnarValue API Killed the Per-Row UDF Call

Jay Hubbert — Wed, 21 Oct 2026 01:57:00 +0000

TL;DR: A DataFusion scalar function is not called per row. PhysicalExpr::evaluate returns a ColumnarValue — either an ArrayRef (the whole batch) or a single ScalarValue — and ScalarFunctionExpr::evaluate evaluates each child expression exactly once, then makes one call into the UDF with the whole batch. The UDF body then works with Arrow kernels. The only place per-row work survives is where a new array has to be constructed (list/struct builders), and even there the work is amortised through Arrow’s MutableArrayData.

The API that used to be slow

For years the shape of a UDF in DataFusion (and in Arrow-rs before it) was:

 rust
 
fn invoke(&self, args: &[ScalarValue]) -> Result<ScalarValue>

One call per row. Every call re-entered the vtable, re-matched on the argument types, allocated a ScalarValue for the result, and then the caller had to un-pack that ScalarValue into a batch. For a UInt8 column, that is 20x the memory traffic of the actual work, plus N virtual calls, plus N allocations.

The modern API is:

 rust
 
 
// datafusion/expr/src/udf.rs:702
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue>;

and the difference is not cosmetic — it changes what the function body can even be. The body receives ArrayRefs and is expected to call Arrow’s vectorized compute kernels (arrow::compute::kernels, take, filter, comparison, …). The per-row loop is gone, not moved.

`ColumnarValue`: the currency

 rust
 
 
// datafusion/expr-common/src/columnar_value.rs:102
pub enum ColumnarValue {
 /// Array of values
 Array(ArrayRef),
 /// A single value
 Scalar(ScalarValue),
}

Two variants, because a value in a query can legitimately be either a column of N rows or one constant. The Scalar variant is not a convenience: it is how literals and constant subexpressions avoid being materialised N times.

The leaves of the physical expression tree produce one or the other, and neither copies the data:

 rust
 
 
// datafusion/physical-expr/src/expressions/column.rs:121
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 self.bounds_check(batch.schema().as_ref())?;
 Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
}

// datafusion/physical-expr/src/expressions/literal.rs:106
fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
 Ok(ColumnarValue::Scalar(self.value.clone()))
}

A column reference is an Arc clone of the batch’s column — O(1), no data movement. A literal is a ScalarValue clone — usually a small enum, no data movement either. So “evaluating the arguments” of a function is O(number of arguments), not O(rows × arguments).

The per-batch contract, precisely

 rust
 
 
// datafusion/physical-expr/src/scalar_function.rs:236
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
 let args = self.args.iter().map(|e| e.evaluate(batch)).collect::<Result<Vec<_>>>()?;
 let arg_fields = self.args.iter().map(|e| e.return_field(batch.schema_ref())).collect::<Result<Vec<_>>>()?;

 // evaluate the function
 let output = self.fun.invoke_with_args(ScalarFunctionArgs {
 args,
 arg_fields,
 number_rows: batch.num_rows(),
 return_field: Arc::clone(&self.return_field),
 config_options: Arc::clone(&self.config_options),
 })?;

 if let ColumnarValue::Array(array) = &output
 && array.len() != batch.num_rows()
 {
 // If the arguments are a non-empty slice of scalar values, we can assume that
 // returning a one-element array is equivalent to returning a scalar.
 let preserve_scalar = array.len() == 1 && !input_empty && input_all_scalar;
 return if preserve_scalar {
 ScalarValue::try_from_array(array, 0).map(ColumnarValue::Scalar)
 } else {
 internal_err!("UDF {} returned a different number of rows than expected. ...")
 };
 }
 Ok(output)
}

Four things to notice:

One call. invoke_with_args is called exactly once per batch per function node.
number_rows is passed explicitly. The function is told how many rows the batch has, so it can produce a result of the right length even if all of its inputs were scalars.
The output length is checked. A function that returns the wrong number of rows is an internal error, caught at the boundary rather than producing a corrupt batch downstream.
The scalar round-trip is preserved. If every argument was a scalar and the function returns a one-element array, it is converted back to a ColumnarValue::Scalar. That keeps SELECT 1 + 1 from becoming a one-element array that then has to be broadcast everywhere.

There is also a debug-only assertion in ScalarUDF::invoke_with_args (datafusion/expr/src/udf.rs:267) that the returned data type matches the type promised at planning time — free correctness checking in debug builds, free in release builds.

The cost of the uniform API: broadcasting scalars

Here is the honest trade-off. To write one array kernel and use it for both f(column) and f(1), the scalar has to be expanded into an array of num_rows:

 rust
 
 
// datafusion/expr-common/src/columnar_value.rs:192
pub fn to_array(&self, num_rows: usize) -> Result<ArrayRef> {
 Ok(match self {
 ColumnarValue::Array(array) => Arc::clone(array),
 ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(num_rows)?,
 })
}

f(1) over a million rows therefore materialises a million copies of 1 before the kernel runs. That is a real cost, and the type’s own documentation says so:

Because all functions must handle ArrayRef, it is convenient to convert ColumnarValue::Scalars using into_array. However, it is often much more performant to provide a different, implementation that handles scalar values differently — datafusion/expr-common/src/columnar_value.rs:90-100

The standard helper handles the common case for you. make_scalar_function (datafusion/functions/src/utils.rs:113) is the glue between the ColumnarValue world and the ArrayRef world:

 rust
 
 
// datafusion/functions/src/utils.rs:120
move |args: &[ColumnarValue]| {
 // first, identify if any of the arguments is an Array. If yes, store its `len`,
 // as any scalar will need to be converted to an array of len `len`.
 let len = args.iter().fold(Option::<usize>::None, |acc, arg| match arg {
 ColumnarValue::Scalar(_) => acc,
 ColumnarValue::Array(a) => Some(a.len()),
 });
 let is_scalar = len.is_none();
 let inferred_length = len.unwrap_or(1);

 let args = args.iter()
 .zip(hints.iter().chain(std::iter::repeat(&Hint::Pad)))
 .map(|(arg, hint)| {
 let expansion_len = match hint {
 Hint::AcceptsSingular => 1,
 Hint::Pad => inferred_length,
 };
 arg.to_array(expansion_len)
 })
 .collect::<Result<Vec<_>>>()?;

 let result = (inner)(&args);
 if is_scalar {
 // If all inputs are scalar, keeps output as scalar
 let result = result.and_then(|arr| ScalarValue::try_from_array(&arr, 0));
 result.map(ColumnarValue::Scalar)
 } else {
 result.map(ColumnarValue::Array)
 }
}

Read it as three rules:

The batch length is inferred from the first array argument.
Scalars are padded to that length (or kept at length 1 if the Hint says the argument is singular, e.g. a “delta” in date_bin).
If every argument was a scalar, the result is collapsed back to a scalar.

That last rule is why SELECT 1 + 1 stays a scalar all the way down, and why SELECT f(1) FROM t does not allocate a million-element array just to read element 0.

The performance-minded escape hatch is to look at args.args directly and match on ColumnarValue::Scalar vs ColumnarValue::Array yourself, writing two code paths. datafusion/functions-nested/src/utils.rs:230 has a simpler variant of the same helper for the nested-function package, and datafusion/functions/src/utils.rs:164 (calculate_binary_math) shows the “handle the scalar operand specially” pattern for binary math.

Nested values: where per-row work is unavoidable

Now the interesting case. make_array(1, 2, 3) does not just compute something per row — it builds a new array whose shape depends on the row. Each output row is a list of N elements, one from each input column:

text

┌─────────┐ ┌─────────┐ ┌──────────────┐
│ ┌─────┐ │ │ ┌─────┐ │ │ ┌──────────┐ │
│ │ A │ │ │ │ X │ │ │ │ [A, X] │ │
│ ├─────┤ │ │ ├─────┤ │ │ ├──────────┤ │
│ │NULL │ │ │ │ Y │ │──────────▶│ │[NULL, Y] │ │
│ ├─────┤ │ │ ├─────┤ │ │ ├──────────┤ │
│ │ C │ │ │ │ Z │ │ │ │ [C, Z] │ │
│ └─────┘ │ │ └─────┘ │ │ └──────────┘ │
└─────────┘ └─────────┘ └──────────────┘
 col1 col2 output

The implementation (datafusion/functions-nested/src/make_array.rs:195) builds a MutableArrayData over all the input arrays and then walks the rows appending one element per column:

 rust
 
 
// datafusion/functions-nested/src/make_array.rs:227
let num_rows = args[0].len();
for row_idx in 0..num_rows {
 for (arr_idx, arg) in args.iter().enumerate() {
 if !arg.as_any().is::<NullArray>() && !arg.is_null(row_idx) && arg.is_valid(row_idx) {
 mutable.try_extend(arr_idx, row_idx, row_idx + 1)?;
 } else {
 mutable.try_extend_nulls(1)?;
 }
 }
 offsets.push(O::usize_as(mutable.len()));
}
let data = mutable.freeze();
Ok(Arc::new(GenericListArray::<O>::try_new(
 Arc::new(Field::new(field_name, data_type, true)),
 OffsetBuffer::new(offsets.into()),
 arrow::array::make_array(data),
 None,
)?))

This is a per-row loop — it has to be, the output shape is per row. But notice what it is not: it is not a virtual call per row, it does not allocate per row (the MutableArrayData buffer grows amortised and is frozen once), and the null handling is inline. The offsets are built in the same pass, so there is no second traversal. The alternative — per-row ScalarValue construction followed by a bulk ListArray build — would allocate a million ScalarValues.

named_struct (datafusion/functions/src/core/named_struct.rs:155) takes the zero-copy route where it can:

 rust
 
 
let values: Vec<ColumnarValue> = args.args.as_chunks::<2>().0.iter()
 .map(|[_name, value]| value.clone()).collect();
let arrays = ColumnarValue::values_to_arrays(&values)?;
Ok(ColumnarValue::Array(Arc::new(StructArray::new(fields.clone(), arrays, None))))

StructArray::new takes the child arrays as they are — for column arguments that is zero copy. Only scalar arguments get broadcast (via values_to_arrays, columnar_value.rs:249), and only because a StructArray requires all children to have the same length.

Note that named_struct also has to reject non-constant field names at planning time (return_field_from_args, named_struct.rs:98-137): the schema of a struct depends on the values of its first argument, so this function’s return type can only be computed from literals. This is a general pattern — when a function’s output type depends on its arguments’ values, the type has to be resolved at plan time, and the argument evaluation has to happen then, not at runtime.

“Per batch” does not mean “per group of rows”

A common misreading: if the function is called once per batch, then random() would return the same value for every row. It does not — because the function is told the row count:

 rust
 
 
// datafusion/functions/src/math/random.rs:82
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
 let mut rng = rng();
 let mut values = vec![0.0; args.number_rows];
 // Equivalent to set each element with rng.random_range(0.0..1.0), but more efficient
 rng.fill(&mut values[..]);
 Ok(ColumnarValue::Array(Arc::new(Float64Array::from(values))))
}

One RNG, one Vec<f64>, one Arrow array — but N distinct values. The batch-at-a-time API does not force constant semantics; it just moves the decision about how to produce per-row variation into the function, where it can be done efficiently. What it does forbid is the optimizer from assuming the call is repeatable, which is exactly what Volatility::Volatile encodes (see the signature article).

From `Expr` to `ColumnarValue`

The logical plan is a tree of Expr (datafusion/expr/src/expr.rs:327), and a scalar call is one node:

 rust
 
 
// datafusion/expr/src/expr.rs:950
pub struct ScalarFunction {
 /// The function
 pub func: Arc<crate::ScalarUDF>,
 /// List of expressions to feed to the functions as arguments
 pub args: Vec<Expr>,
}

The Expr tree is a description of what to compute; it is manipulated by the optimizer, hashed, compared, and rewritten. It never runs. At physical planning time it is lowered into a tree of PhysicalExpr — where the scalar call becomes ScalarFunctionExpr with a Vec<Arc<dyn PhysicalExpr>> of children — and that tree is what runs, once per batch, producing ColumnarValues.

The two trees have different jobs:

Expr is a description with types, spans, and metadata. It is cheap to clone, expensive to hash, and it is what the optimizer reasons about.
PhysicalExpr is an executable plan fragment. It holds an Arc<ScalarUDF> (so the signature and the implementation are shared, not copied), the argument expressions, and the resolved return field. It is built once per query and reused for every batch.

The Expr tree is where a “constant” stays a constant: Expr::Literal is folded by the simplifier (if the function is not volatile) into a ScalarValue, and the physical planner turns it into a Literal physical expression whose evaluate returns ColumnarValue::Scalar. That is the mechanism by which “evaluate the arguments once per batch” also means “don’t evaluate constant arguments at all”.

Trade-offs and rough edges

Broadcasting is a real copy. f(1) over N rows materialises N values. The Hint::AcceptsSingular escape hatch and hand-written scalar fast paths exist precisely to avoid it, but they are opt-in and easy to forget.
The output length check is a runtime cost. One comparison per function node per batch. Cheap, but it is why the “return a one-element array for scalar inputs” case has to be special-cased rather than just allowed.
arg_fields is recomputed per batch. ScalarFunctionExpr::evaluate calls return_field on every child on every batch (scalar_function.rs:243-248). For a Column that is an Arc clone of a schema field; for a Literal it is a field clone. It is not free, and it is pure overhead that could be hoisted into the constructor. It is, however, small compared to the work the function is about to do.
Nested builders are per row by necessity. make_array and friends walk rows. The mitigation is amortisation (MutableArrayData, a single freeze(), offsets built in the same pass) and the avoidance of per-row allocation.
The scalar/array duality leaks into every UDF. Every author has to decide: do I handle ColumnarValue::Scalar specially, or do I let make_scalar_function broadcast for me? Both are legitimate; the second is simpler and the first is faster. The API does not force a choice, which is exactly why performance varies between functions that look identical from the outside.
Bigger batches amortise everything. The per-batch overhead (vtable call, length check, field lookups) is amortised over batch_size rows, so a function that is cheap per row can still be a measurable cost on 8 KiB batches. The vectorised API makes the per-row cost small; it does not make the per-batch cost zero.

How DataFusion Approximates Quantiles with a Mergeable t-Digest (and Distinct Counts with HyperLogLog)

Jay Hubbert — Wed, 21 Oct 2026 00:37:00 +0000

TL;DR: approx_percentile_cont keeps a bounded set of centroids — a mean and a weight each — instead of the raw values. The state is just those centroids plus sum/count/min/max, so a partial aggregate can hand it to a final aggregate over the wire and the final aggregate folds the centroid lists together. The catch: merging re-compresses, so the result depends slightly on how the data was partitioned. HyperLogLog, used by approx_distinct, is exactly mergeable (register-wise max), and DataFusion even gives it a sparse representation so a group with three distinct values does not cost 16 KiB.

The problem: an exact median is not mergeable

SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY x) FROM t needs every value of x, sorted. That is fine until you want to parallelize it: the partial phase on partition 0 has 10 million values, the partial phase on partition 1 has 10 million more, and the final phase has to combine them into one answer. If the aggregate function is “sort everything”, the state is “everything”, and the shuffle moves all the data.

An aggregate is only parallelizable if its state is small and mergeable. sum is the archetype: state is one i64, merge is addition. count likewise. percentile_cont is not: its state is the whole multiset.

Sketch algorithms fix this by keeping an approximate state that is small and mergeable. DataFusion ships two of them: a t-digest for quantiles (datafusion/functions-aggregate-common/src/tdigest.rs) and a HyperLogLog for distinct counts (datafusion/functions-aggregate/src/hyperloglog.rs).

The t-digest data structure

The t-digest (Dunning’s 2019 paper, arXiv:1902.04023) is a “clustering” sketch: instead of storing values, it stores centroids, each of which is a weighted average:

 rust
 
 
// datafusion/functions-aggregate-common/src/tdigest.rs:54
pub struct Centroid {
 mean: f64,
 weight: f64,
}

and the digest itself is:

 rust
 
 
// datafusion/functions-aggregate-common/src/tdigest.rs:98
pub struct TDigest {
 centroids: Vec<Centroid>,
 max_size: usize, // default 100 (DEFAULT_MAX_SIZE, tdigest.rs:39)
 sum: f64,
 count: f64,
 max: f64,
 min: f64,
}

The magic is which values get merged together. A naive “keep the first 100 distinct values” is useless for quantiles. A t-digest merges neighbouring values, but only when the resulting cluster is small enough, and the “small enough” threshold depends on where in the distribution the cluster sits. That is the scale function:

 rust
 
 
// datafusion/functions-aggregate-common/src/tdigest.rs:188
fn k_to_q(k: u64, d: usize) -> f64 {
 let k_div_d = k as f64 / d as f64;
 if k_div_d >= 0.5 {
 let base = 1.0 - k_div_d;
 1.0 - 2.0 * base * base
 } else {
 2.0 * k_div_d * k_div_d
 }
}

k is the index of the centroid, d is max_size. The quantile q a centroid is allowed to “cover” is k_to_q(k, d): near the tails the curve is flat, so a single centroid there may represent a tiny slice of the distribution, and near the median it is steep, so centroids there are allowed to be fat. Practically: the tails are precise and the middle is fuzzy, which is exactly what you want from a quantile sketch — nobody cares whether p50 is off by 0.1%, everybody cares whether p99.9 is right.

The compression loop is a single pass over the sorted values, accumulating weights until the limit for the current centroid index is exceeded:

 rust
 
 
// datafusion/functions-aggregate-common/src/tdigest.rs:285 (inside merge_sorted_f64)
if weight_so_far <= q_limit_times_count {
 sums_to_merge += next_sum;
 weights_to_merge += next.weight();
} else {
 result.sum += curr.add(sums_to_merge, weights_to_merge);
 compressed.push(curr.clone());
 q_limit_times_count = Self::k_to_q(k_limit, self.max_size) * result.count();
 k_limit += 1;
 curr = next;
}

Note q_limit_times_count is a fraction of the total count, so the digest re-balances itself as more data arrives: the same digest that held 100 values in 20 centroids will hold 100 million values in 100 centroids, still with the tail clusters small.

Making the digest mergeable

Merging two digests means merging their centroid lists. The lists are sorted by mean, so the merge is a sorted merge followed by a re-compression pass with the same scale function. For more than two digests, DataFusion does a bottom-up merge sort over the concatenated centroid lists:

 rust
 
 
// datafusion/functions-aggregate-common/src/tdigest.rs:357
pub fn merge_digests<'a>(digests: impl IntoIterator<Item = &'a TDigest>) -> TDigest {
 // ... concatenate all centroids ...
 let mut digests_per_block: usize = 1;
 while digests_per_block < starts.len() {
 for i in (0..starts.len()).step_by(digests_per_block * 2) {
 // ... Self::external_merge(&mut centroids, first, middle, last, &mut merge_buffer)
 }
 digests_per_block *= 2;
 }
 // ... then re-compress under the k_to_q limits ...
}

external_merge (tdigest.rs:310) is a textbook two-pointer merge into a scratch buffer, then copy back. Bottom-up merging (digests_per_block *= 2) keeps the total work O(n log n) in the number of digests, and it is done on flat centroid arrays, not on nested structures — no per-digest allocation in the inner loop.

Serializing the state so it can cross a shuffle

The state has to be Arrow data. TDigest::to_scalar_state (tdigest.rs:576) packs it into six scalars:

text

 ┌────────┬────────┬────────┬───────┬────────┬────────┐
 │max_size│ sum │ count │ max │ min │centroid│
 └────────┴────────┴────────┴───────┴────────┴────────┘
 │
 ┌─────────────────────┘
 ▼
 ┌ List ───┐
 │ mean │
 │ weight │ <- Centroid 1
 │ mean │
 │ weight │ <- Centroid 2
 └─────────┘

and from_scalar_state (tdigest.rs:605) reverses it. The centroids are packed as a flat List<Float64> of [mean, weight, mean, weight, ...] — a deliberately boring layout, because it round-trips through Arrow without custom serialization.

DataFusion’s aggregate wires this up with seven state columns (datafusion/functions-aggregate/src/approx_percentile_cont.rs:250): the six digest fields plus the percentile itself. The percentile has to travel with the state because the final phase does not have the original arguments — it only has the state. STATE_PERCENTILE_IDX = 6 (approx_percentile_cont.rs:52) marks that column, and merge_batch reads it back out:

 rust
 
 
// datafusion/functions-aggregate/src/approx_percentile_cont.rs:461
fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
 if let Some(percentile_array) = states.get(STATE_PERCENTILE_IDX) {
 self.percentile.resolve(percentile_array)?;
 }
 let tdigest_states = &states[..STATE_PERCENTILE_IDX];
 let states = (0..tdigest_states[0].len())
 .map(|index| {
 tdigest_states.iter()
 .map(|array| ScalarValue::try_from_array(array, index))
 .collect::<Result<Vec<_>>>()
 .map(|state| TDigest::from_scalar_state(&state))
 })
 .collect::<Result<Vec<_>>>()?;
 self.merge_digests(&states);
 Ok(())
}

Note that merge_batch reconstructs a TDigest per row of the incoming state, then merges them all. That is the price of a “state is Arrow arrays” design: the per-group state is unpacked, then folded.

The honest caveat: t-digest is approximately mergeable

Here is the thing to internalize. A t-digest is not a CRDT. Merging digests D1 and D2 in the order (D1, D2) does not produce the same centroids as merging in the order (D2, D1), because the re-compression step merges neighbouring centroids differently depending on where the boundary falls. The estimate is stable to within the sketch’s error bar, but it is not bit-exact.

Concretely: SELECT approx_percentile_cont(0.5) FROM t may return a slightly different number depending on how many partitions the table had. The discrepancy shrinks as max_size grows — more centroids means less merging, means less re-compression ambiguity. The default is 100 centroids (DEFAULT_MAX_SIZE, tdigest.rs:39), and the user can raise it as the third argument: approx_percentile_cont(0.9, 500).

This is a design trade, not a bug: exactness would require shipping all the data, which is exactly what the sketch exists to avoid.

The DataFusion-specific wart: sorting every batch

update_batch is:

 rust
 
 
// datafusion/functions-aggregate/src/approx_percentile_cont.rs:427
fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
 if values.len() > 1 {
 self.resolve_percentile(&values[1])?;
 }
 let mut values = Arc::clone(&values[0]);
 if values.null_count() > 0 {
 values = filter(&values, &is_not_null(&values)?)?;
 }
 let sorted_values = &arrow::compute::sort(&values, None)?;
 let sorted_values = ApproxPercentileAccumulator::convert_to_float(sorted_values)?;
 self.digest = self.digest.merge_sorted_f64(&sorted_values);
 Ok(())
}

Read the cost carefully: every batch is fully sorted (arrow::compute::sort), then copied into a Vec<f64>, and then merge_sorted_f64 builds a brand new TDigest (new Vec<Centroid>, new compression pass) rather than incrementally absorbing the values into the existing centroids. So the cost per batch is O(n log n) for the sort, O(n) for the copy, plus a full re-compression of the (up to 100) existing centroids.

A textbook t-digest implementation keeps the digest sorted by centroid mean and inserts new values by binary search + local re-compression, so a batch of n values costs O(n log(max_size)), not O(n log n). DataFusion’s version is simpler to reason about (and reuses Arrow’s vectorized sort), but it is noticeably more expensive than it needs to be for wide batches. If you are benchmarking approx_percentile_cont on a wide table, this is the first place to look.

Bonus: HyperLogLog, and the sparse trick

approx_distinct uses a completely different sketch — one that is exactly mergeable. The state is 2^14 registers of one byte each:

 rust
 
 
// datafusion/functions-aggregate/src/hyperloglog.rs:43
const HLL_P: usize = 14_usize;
const HLL_Q: usize = 64_usize - HLL_P;
pub(crate) const NUM_REGISTERS: usize = 1_usize << HLL_P; // 16384

Each distinct value is hashed; the low 14 bits pick a register, and the remaining bits’ leading-zero count updates it:

 rust
 
 
// datafusion/functions-aggregate/src/hyperloglog.rs:121
pub(crate) fn add_hashed(&mut self, hash: u64) {
 let index = (hash & HLL_P_MASK) as usize;
 let p = ((hash >> HLL_P) | (1_u64 << HLL_Q)).trailing_zeros() + 1;
 self.registers[index] = self.registers[index].max(p as u8);
}

Merging is register-wise max:

 rust
 
 
// datafusion/functions-aggregate/src/hyperloglog.rs:140
pub fn merge(&mut self, other: &HyperLogLog<T>) {
 for i in 0..self.registers.len() {
 self.registers[i] = self.registers[i].max(other.registers[i]);
 }
}

max is commutative, associative and idempotent — a textbook CRDT. The estimate is therefore exactly partition-independent, with a standard error of 1.04/sqrt(16384) = 0.8125% (hyperloglog.rs:31-38).

The clever part is what DataFusion added on top. A dense sketch is 16 KiB per group, which is absurd for a GROUP BY where most groups have three distinct values. So approx_distinct keeps an adaptive per-group state:

 rust
 
 
// datafusion/functions-aggregate/src/approx_distinct.rs:304
const SPARSE_LIMIT: usize = 256;

enum GroupHll {
 Sparse(Vec<u64>), // distinct hashes seen so far
 Dense(Box<HyperLogLog<u8>>),
}

A group starts sparse, accumulating raw hashes; when it exceeds 256 distinct values it is promoted to a dense sketch (compact_or_promote, approx_distinct.rs:360). Because folding the same hashes into a dense sketch produces exactly the same registers as adding the values one by one, the estimate is identical either way. On the read side, a sparse group is estimated without materializing 16 KiB:

 rust
 
 
// datafusion/functions-aggregate/src/hyperloglog.rs:177
pub(crate) fn count_from_hashes(hashes: &[u64]) -> usize {
 // sort (index, rho) pairs, keep the max rho per index,
 // build the register histogram, then apply the HLL estimator
}

The comment says it plainly: “This is used to cheaply estimate the many small groups produced by a high-cardinality GROUP BY, where allocating and scanning a 16 KiB sketch per group would dominate the runtime.” And because the sparse state is a list of hashes, the serialized partial state is also small — the final phase receives a few hundred bytes instead of 16 KiB per group (GroupHll::serialize, approx_distinct.rs:438).

Trade-offs, summarized

t-digest: accurate at the tails, fuzzy in the middle; state is mergeable but the merge is not associative, so results are partition-sensitive; DataFusion’s implementation sorts every batch, which dominates its cost.
HyperLogLog: exactly mergeable, bounded 0.81% error, but the error is relative — useless for small cardinalities (a set of 3 distinct values estimates as 3 ± 0.8, which rounds to 3, fine, but a set of 1 estimates as 1 with a large relative error), and it cannot tell you which values.
Both trade exactness for a state that fits in a shuffle. The general lesson is the same: an aggregate becomes parallelizable exactly when its state becomes small, and its merge becomes an associative operation over that state. t-digest is associative approximately; HyperLogLog is associative exactly. DataFusion ships both, and picks per aggregate.

Functions on Opensource internals

Signatures, Volatility, and the Type-Coercion Lattice: How DataFusion's UDF Metadata Drives the Optimizer

Two problems one struct solves

TypeSignature: arity × type shape

The coercion lattice

Finding the common type

Volatility: the promise about time

The dispatch cost: one virtual call per batch, not per row

The registry

Trade-offs

DataFusion's Aggregate Split: One UDF, Two Accumulator Traits, and a Serializable State Format

The problem: an aggregate is two different computations

The Accumulator trait: one group, one object

The two-phase flow

GroupsAccumulator: the fast path that avoids per-group calls

The adapter: making old aggregates look new

EmitTo, take_state, and the “emit the first N groups” dance

convert_to_state: skipping the hash table entirely

Trade-offs and failure modes

Once Per Batch: How DataFusion's ColumnarValue API Killed the Per-Row UDF Call

The API that used to be slow

ColumnarValue: the currency

The per-batch contract, precisely

The cost of the uniform API: broadcasting scalars

Nested values: where per-row work is unavoidable

“Per batch” does not mean “per group of rows”

From Expr to ColumnarValue

Trade-offs and rough edges

How DataFusion Approximates Quantiles with a Mergeable t-Digest (and Distinct Counts with HyperLogLog)

The problem: an exact median is not mergeable

The t-digest data structure

Making the digest mergeable

Serializing the state so it can cross a shuffle

The honest caveat: t-digest is approximately mergeable

The DataFusion-specific wart: sorting every batch

Bonus: HyperLogLog, and the sparse trick

Trade-offs, summarized

`TypeSignature`: arity × type shape

`Volatility`: the promise about time

The `Accumulator` trait: one group, one object

`GroupsAccumulator`: the fast path that avoids per-group calls

`EmitTo`, `take_state`, and the “emit the first N groups” dance

`convert_to_state`: skipping the hash table entirely

`ColumnarValue`: the currency

From `Expr` to `ColumnarValue`