Signatures, Volatility, and the Type-Coercion Lattice: How DataFusion's UDF Metadata Drives the Optimizer

TL;DR: Signature is the metadata contract between a UDF and the engine. TypeSignature says which argument types have implementations; the coercion lattice turns “user called cos(int_col)” into “cast to Float64 and call the Float64 implementation”. Volatility is the second half of the contract: Immutable/Stable functions may be constant-folded at plan time and reordered by the optimizer; Volatile ones may not, and the optimizer goes out of its way to keep them from being duplicated, hoisted or pushed below an aggregate. And because ScalarUDF wraps Arc<dyn ScalarUDFImpl>, the virtual call happens once per batch, not once per row.

Two problems one struct solves

A scalar UDF in DataFusion has to answer three questions before it can run:

What argument types do you implement? (cos handles Float32/Float64; make_array handles anything.)
What do I do with the types I don’t implement? (Cast the input? Error?)
May I call you twice and get the same answer? (abs(-1) yes, random() no.)

The first two are the “type” question and the third is the “time” question. They are orthogonal, and DataFusion keeps them in one struct because they are always needed together — during planning, during coercion, and during optimization:

rust
// datafusion/expr-common/src/signature.rs:1410
pub struct Signature {
    /// The data types that the function accepts. See [TypeSignature] for more information.
    pub type_signature: TypeSignature,
    /// The volatility of the function. See [Volatility] for more information.
    pub volatility: Volatility,
    /// Optional parameter names for the function arguments.
    pub parameter_names: Option<Vec<String>>,
}

The constructors make the common cases one-liners (signature.rs:1426-1540): Signature::exact, Signature::uniform, Signature::variadic, Signature::user_defined, Signature::one_of, Signature::comparable, Signature::coercible, Signature::numeric, Signature::string, Signature::nullary, Signature::any, Signature::variadic_any, plus the array-specialised ones (array_and_element, arrays, array_and_index, …).

`TypeSignature`: arity × type shape

rust
// datafusion/expr-common/src/signature.rs:165
pub enum TypeSignature {
    Variadic(Vec<DataType>),        // N args, each one of these types
    UserDefined,                    // the UDF does its own coercion
    VariadicAny,                    // N args, any types
    Uniform(usize, Vec<DataType>),  // exactly N args, all the same type
    Exact(Vec<DataType>),           // exactly N args, these types, in order
    Coercible(Vec<Coercion>),      // per-argument coercion rules
    Comparable(usize),              // N args coerced to one comparable type
    Any(usize),                     // N args, any types
    OneOf(Vec<TypeSignature>),      // try each in order
    ArraySignature(ArrayFunctionSignature),
    Numeric(usize),
    String(usize),
    Nullary,                        // no arguments
}

The doc comment on the enum is the important part: TypeSignature does not define the types a user may call the function with. It defines the types you wrote code for. Everything else is the engine’s problem, and the engine solves it by inserting casts.

OneOf deserves special mention because it is how a function supports genuinely different shapes. approx_percentile_cont uses it for the 2-argument and 3-argument forms (datafusion/functions-aggregate/src/approx_percentile_cont.rs:137-172):

rust
let signature = Signature::one_of(
    vec![
        // 2 args - numeric, percentile (float)
        TypeSignature::Coercible(vec![
            Coercion::new_implicit(
                TypeSignatureClass::Float,
                vec![TypeSignatureClass::Numeric],
                NativeType::Float64,
            ),
            Coercion::new_implicit_native(
                logical_float64(),
                vec![TypeSignatureClass::Numeric],
            ),
        ]),
        // 3 args - numeric, percentile (float), number of centroids (integer)
        // ...
    ],
    Volatility::Immutable,
);

The coercion lattice

The interesting question is: given cos(some_int32_column), what happens?

The entry point is fields_with_udf (datafusion/expr/src/type_coercion/functions.rs:110), which is called during planning for every function call:

rust
// datafusion/expr/src/type_coercion/functions.rs:136
let valid_types = get_valid_types_with_udf(type_signature, &current_types, func)?;
if valid_types.iter().any(|data_type| data_type == &current_types) {
    return Ok(current_fields.to_vec());   // already exact: no casts
}
let updated_types =
    try_coerce_types(func.name(), valid_types, &current_types, type_signature)?;

Three steps: enumerate the valid type combinations for the signature, check whether the call already matches one, otherwise find a combination we can cast to.

get_valid_types_with_udf (functions.rs:532) special-cases two signatures: UserDefined delegates straight to the UDF’s own coerce_types hook, and OneOf tries each alternative and collects the successes (so one_of with two 3-arg forms can match either). Everything else goes to get_valid_types, which expands the signature into concrete DataType vectors.

Then try_coerce_types (functions.rs:494) picks a target. It has two paths, and the distinction matters:

rust
// datafusion/expr/src/type_coercion/functions.rs:474
fn is_well_supported_signature(type_signature: &TypeSignature) -> bool {
    match type_signature {
        TypeSignature::OneOf(type_signatures) => type_signatures.iter().all(is_well_supported_signature),
        TypeSignature::UserDefined | TypeSignature::Numeric(_) | TypeSignature::String(_)
        | TypeSignature::Coercible(_) | TypeSignature::Any(_) | TypeSignature::Nullary
        | TypeSignature::Comparable(_) => true,
        TypeSignature::Variadic(_) | TypeSignature::VariadicAny | TypeSignature::Uniform(_, _)
        | TypeSignature::Exact(_) | TypeSignature::ArraySignature(_) => false,
    }
}

The “well supported” signatures (the new Coercible/Numeric/String/ Comparable family) have already resolved to a single exact target type, so coercion is a direct cast. The legacy signatures (Exact, Uniform, Variadic) take the old path, which searches for any valid combination and casts to it — more expensive, more magic, and the source of most of the “why did my function get a CAST I didn’t ask for” confusion in DataFusion.

Finding the common type

The actual lattice lives in datafusion/expr-common/src/type_coercion/binary.rs. type_union_resolution (:708) computes the common type of a set of argument types, Postgres/DuckDB style:

rust
// datafusion/expr-common/src/type_coercion/binary.rs:719
// If all the data_types are null, return string
if data_types.iter().all(|t| t == &DataType::Null) {
    return Some(DataType::Utf8View);
}
// Ignore Nulls, if any data_type category is not the same, return None
// ...

and the pairwise step (:773) is a cascade of fallbacks:

rust
// datafusion/expr-common/src/type_coercion/binary.rs:873
binary_numeric_coercion(lhs_type, rhs_type)
    .or_else(|| list_coercion(...))
    .or_else(|| map_coercion(...))
    .or_else(|| temporal_coercion_nonstrict_timezone(...))
    .or_else(|| string_coercion(...))
    .or_else(|| null_coercion(...))
    .or_else(|| string_numeric_coercion(...))
    .or_else(|| binary_coercion(...))

That is the “lattice”: numeric widening first (i32 + i64 → i64), then lists, then maps, then temporals, then strings, then the escape hatches that let a string be read as a number or a binary be read as a string. Order matters, and that order encodes a priority: it mirrors what a human would guess.

The Coercion enum (signature.rs:1164) is the modern, explicit way to say “this argument should end up as a Float64, and it may be implicitly cast from any numeric type”:

rust
// datafusion/expr-common/src/signature.rs:1164
pub enum Coercion {
    Exact { desired_type: TypeSignatureClass, encoding_preservation: EncodingPreservation },
    Implicit {
        desired_type: TypeSignatureClass,
        implicit_coercion: ImplicitCoercion,   // allowed_source_types + default_casted_type
        encoding_preservation: EncodingPreservation,
    },
}

TypeSignatureClass (signature.rs:371) is the coarse lattice of type families — Any, Timestamp, Time, Interval, Duration, Native(LogicalTypeRef), Integer, Float, Decimal, Numeric, Binary. Numeric is the union of Integer | Float | Decimal; Integer and Float are the two ways to spell “any numeric that is not a decimal”. Writing Coercion::new_implicit(TypeSignatureClass::Float, vec![TypeSignatureClass::Numeric], NativeType::Float64) says: “accept a float; if the user gave me an int or a decimal, cast it to Float64”. That is one line replacing a page of match arms.

`Volatility`: the promise about time

rust
// datafusion/expr-common/src/signature.rs:56
pub enum Volatility {
    /// Always returns the same output when given the same input.
    Immutable,
    /// May return different values given the same input across different
    /// queries but must return the same value for a given input within a query.
    Stable,
    /// May change the return value from evaluation to evaluation.
    Volatile,
}

Three levels, and the difference between Immutable and Stable is subtle but real: Immutable says “same input, same output, forever” (so abs(-1) is folded to 1 at plan time), Stable says “same input, same output within this query” (now() is constant for the duration of one query but changes between queries), and Volatile says “no promises at all” (random()).

The optimizer uses this in two distinct ways.

1. Constant folding. The simplifier’s ConstEvaluator refuses to evaluate volatile functions at plan time:

rust
// datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:617
fn volatility_ok(volatility: Volatility) -> bool {
    match volatility {
        Volatility::Immutable => true,
        // Values for functions such as now() are taken from ExecutionProps
        Volatility::Stable => true,
        Volatility::Volatile => false,
    }
}

can_evaluate (expr_simplifier.rs:626) then refuses the whole subtree containing a volatile node, so 1 + random() is not folded and 1 + abs(-1) is. Folding is not just a nicety: it turns a per-row computation into a constant, which lets the optimizer push predicates around it, drop columns, and use statistics.

2. “May I duplicate or move this expression?” Plenty of optimizations are only valid if the expression means the same thing wherever it ends up. DataFusion encodes that check with Expr::is_volatile:

rust
// datafusion/expr/src/expr.rs:2208
pub fn is_volatile_node(&self) -> bool {
    let volatility = match self {
        Expr::ScalarFunction(func) => func.func.signature().volatility,
        Expr::AggregateFunction(func) => func.func.signature().volatility,
        // ...
    };
    volatility == Volatility::Volatile
}

Note the distinction between is_volatile_node (this node only) and is_volatile (:2231, the whole subtree). Rewrites use the subtree version, because moving a subtree that contains a volatile node is just as wrong as moving the volatile node itself.

Concretely, the filter pushdown refuses to push a predicate into the grouping keys when the key is volatile:

rust
// datafusion/optimizer/src/push_down_filter.rs:1117
// We can push down Predicate which in groupby_expr.
// Volatile group keys are excluded: below the aggregate, the
// predicate would evaluate the key again and see a different
// value than the one used for grouping.
let non_volatile_group_exprs: Vec<Expr> = agg
    .group_expr
    .iter()
    .filter(|expr| !expr.is_volatile())
    .cloned()
    .collect();

That is a correctness fix, not a pessimization: GROUP BY random() would group by one value and then filter on a different random value if the predicate were pushed down. Similarly, outer-join elimination only fires when the join condition is not volatile (datafusion/optimizer/src/eliminate_join.rs:1036), and common-subexpression elimination skips volatile output columns (datafusion/optimizer/src/extract_leaf_expressions.rs:109).

The cost of getting this wrong is asymmetric and brutal:

Declare a non-deterministic function Immutable → the planner folds it to a literal → every row of the query gets the same value. Silently wrong.
Declare a deterministic function Volatile → no folding, no predicate pushdown below aggregates, no CSE, no join elimination. Correct but mysteriously slow, and the profiler will not tell you why.

The rule of thumb in the doc comment is the right one: “You should always define your function to have the strictest possible volatility to maximize performance and avoid unexpected results.”

The dispatch cost: one virtual call per batch, not per row

ScalarUDF is a thin newtype over a trait object:

rust
// datafusion/expr/src/udf.rs:86
pub struct ScalarUDF {
    inner: Arc<dyn ScalarUDFImpl>,
}

The vtable lookup happens in ScalarFunctionExpr::evaluate (datafusion/physical-expr/src/scalar_function.rs:236), which evaluates each child expression once and hands the whole batch to the function:

rust
// datafusion/physical-expr/src/scalar_function.rs:236
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
    let args = self.args.iter().map(|e| e.evaluate(batch)).collect::<Result<Vec<_>>>()?;
    // ...
    let output = self.fun.invoke_with_args(ScalarFunctionArgs {
        args,
        arg_fields,
        number_rows: batch.num_rows(),
        return_field: Arc::clone(&self.return_field),
        config_options: Arc::clone(&self.config_options),
    })?;
    // ...
}

So the dynamic dispatch is O(1) per batch; the body of the function works on ArrayRefs and uses Arrow’s vectorized kernels. A UDF written naively — one ScalarValue in, one ScalarValue out — would instead be O(rows) virtual calls with a ScalarValue allocation per row, and would be the slowest thing in the engine.

There is one more escape hatch worth knowing about, for when the optimizer needs to reason about a specific function rather than a name:

rust
// datafusion/physical-expr/src/scalar_function.rs:165
pub fn try_downcast_func<T>(expr: &dyn PhysicalExpr) -> Option<&ScalarFunctionExpr>
where T: ScalarUDFImpl

It matches on the concrete ScalarUDFImpl type, not the function name, so GetFieldFunc can be recognised even if a user registered a different function under the same name. The parquet reader uses it to recognise get_field and file_row_index/input_file_name (datafusion/datasource-parquet/src/projection_read_plan.rs:418,511).

The registry

All of this metadata hangs off the function object, and the functions live in a registry:

rust
// datafusion/expr/src/registry.rs:40
pub trait FunctionRegistry {
    fn udf(&self, name: &str) -> Result<Arc<ScalarUDF>>;
    fn udaf(&self, name: &str) -> Result<Arc<AggregateUDF>>;
    fn udwf(&self, name: &str) -> Result<Arc<WindowUDF>>;
    fn register_udf(&mut self, _udf: Arc<ScalarUDF>) -> Result<Option<Arc<ScalarUDF>>>;
    // ...
}

datafusion/functions/src/lib.rs:166 builds the default set (all_default_functions, chaining the per-package functions() lists) and :180 (register_all) installs them into any registry, logging an overwrite if a name collides. Because the registry hands out Arc<ScalarUDF>, the Signature is computed once and shared by every plan that uses the function — the coercion lattice is a planning-time cost, paid once per query, not a per-row cost.

Trade-offs

Metadata is a promise you can break. Signature is unchecked by the compiler. A function that lies about its arity or its volatility produces wrong answers, not errors.
The coercion lattice is a planning-time tax. Every function call enumerates valid type combinations and searches for a cast path. The OneOf and legacy Exact/Uniform paths are the expensive ones; the modern Coercible/Numeric/String/Comparable signatures exist partly to make that search trivial.
OneOf order is semantics. The first matching alternative wins, so reordering the alternatives in a signature can change which implementation runs — and therefore the result type and the accuracy.
Volatility::Stable is under-used. now() is Stable; most UDF authors reach for Immutable or Volatile and never think about the middle rung, which is exactly the rung that says “constant for this query”.
The Arc<dyn ScalarUDFImpl> boundary is a real, if small, cost. It is paid once per batch per function node. For a query that evaluates a function on 8 KiB batches, that is measurable but small; for one that evaluates it on million-row batches, it is noise. Which is another way of saying: keep the batch big, and the dispatch model takes care of itself.