Opensource internals

ClickHouse Composes Aggregate Functions with Name Suffixes

19 min read Jay Hubbert

ClickHouse builds new aggregate functions by suffixing names: sumIf, sumState, sumMerge. A factory strips the suffix, recurses into the base function, and wraps it in a decorator that owns a nested function.

TL;DR: sumIf(x, cond), sumState(x) and sumMerge(states) are not three separate functions in the source. There is one sum. A factory looks at the suffix of the name, strips it, resolves the base function recursively, and hands the result to a decorator — an IAggregateFunction that holds another IAggregateFunction and forwards almost everything to it. The decorator is the whole implementation.

Ask a SQL engine for sumIf and you get a function that has never heard of If. Ask it for uniqCombinedIfIf and you get an error, but uniqCombinedIfMergeIf works. Ask for sumState and you get a function whose return type is a data type, not a number.

All three behaviours fall out of one mechanism: the aggregate function name is a program, and the factory is its interpreter.

The interface a combinator implements

A combinator is a small object with four responsibilities, declared in IAggregateFunctionCombinator:

cpp 
// src/AggregateFunctions/Combinators/IAggregateFunctionCombinator.h:18
/** Aggregate function combinator allows to take one aggregate function
  *  and transform it to another aggregate function.
  *
  * In SQL language they are used as suffixes for existing aggregate functions.
  *
  * Example: -If combinator takes an aggregate function and transforms it
  *  to aggregate function with additional argument at end (condition),
  *  that will pass values to original aggregate function when the condition is true.
  *
  * More examples:
  *
  * sum(x) - calculate sum of x
  * sumIf(x, cond) - calculate sum of x for rows where condition is true.
  * sumArray(arr) - calculate sum of all elements of arrays.
  *
  * PS. Please don't mess it with so called "combiner" - totally unrelated notion from Hadoop world.
  * "combining" - merging the states of aggregate functions - is supported naturally in ClickHouse.
  */

The four methods that matter:

cpp 
// src/AggregateFunctions/Combinators/IAggregateFunctionCombinator.h:58
    /** From the arguments for combined function (ex: UInt64, UInt8 for sumIf),
      *  get the arguments for nested function (ex: UInt64 for sum).
      * If arguments are not suitable for combined function, throw an exception.
      */
    virtual DataTypes transformArguments(const DataTypes & arguments) const
    {
        return arguments;
    }
cpp 
// src/AggregateFunctions/Combinators/IAggregateFunctionCombinator.h:76
    /** Create combined aggregate function (ex: sumIf)
      *  from nested function (ex: sum)
      *  and arguments for combined aggregate function (ex: UInt64, UInt8 for sumIf).
      * It's assumed that function transformArguments was called before this function and 'arguments' are validated.
      * Every combinator overrides this method, except combinators with `transformsMultipleNestedFunctions`,
      * which override `transformAggregateFunctionFromMultipleNestedFunctions` instead.
      */
    virtual AggregateFunctionPtr transformAggregateFunction(
        const AggregateFunctionPtr & /*nested_function*/,
        const AggregateFunctionProperties & /*properties*/,
        const DataTypes & /*arguments*/,
        const Array & /*params*/) const
    {
        throw Exception(ErrorCodes::NOT_IMPLEMENTED,
            "Aggregate function combinator '{}' does not wrap a single nested function", getName());
    }

So a combinator answers two questions: what are the base function’s arguments? and what object should represent the combination?

There is also a third, optional question: can I be applied to myself?

cpp 
// src/AggregateFunctions/Combinators/IAggregateFunctionCombinator.h:44
    /** Does combinator supports nesting (of itself, i.e. ArrayArray or IfIf)
     */
    virtual bool supportsNesting() const { return false; }

IfIf is banned, ArrayArray is allowed, and the distinction is enforced centrally (see below).

The suffix table, sorted by length

Combinators are registered in a global factory keyed by their suffix name:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionCombinatorFactory.h:20
    struct CombinatorPair
    {
        std::string name;
        AggregateFunctionCombinatorPtr combinator_ptr;
        Documentation documentation;

        bool operator==(const CombinatorPair & rhs) const { return name == rhs.name; }
        /// Sort by the length of the combinator name for proper tryFindSuffix()
        /// for combiners with common prefix (i.e. "State" and "SimpleState").
        bool operator<(const CombinatorPair & rhs) const { return name.length() > rhs.name.length(); }
    };

That operator< is the interesting bit. It sorts longest suffix first, which matters because one combinator’s name is a suffix of another’s: SimpleState ends with State. If State were checked first, sumSimpleState would be split as sumSimple + State — and there is no sumSimple function, so the query would fail. Sorting by descending length means the most specific suffix wins.

The lookup is a linear scan — there are only a dozen combinators:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionCombinatorFactory.cpp:29
AggregateFunctionCombinatorPtr AggregateFunctionCombinatorFactory::tryFindSuffix(const std::string & name) const
{
    /// O(N) is ok for just a few combinators.
    for (const auto & suffix_value : dict)
        if (endsWith(name, suffix_value.name))
            return suffix_value.combinator_ptr;
    return {};
}

And the registration is a plain loop at startup:

cpp 
// src/AggregateFunctions/registerAggregateFunctions.cpp:226
    {
        auto & factory = AggregateFunctionCombinatorFactory::instance();

        registerAggregateFunctionCombinatorIf(factory);
        registerAggregateFunctionCombinatorArray(factory);
        registerAggregateFunctionCombinatorForEach(factory);
        registerAggregateFunctionCombinatorSimpleState(factory);
        registerAggregateFunctionCombinatorState(factory);
        registerAggregateFunctionCombinatorMerge(factory);
        registerAggregateFunctionCombinatorNull(factory);
        registerAggregateFunctionCombinatorOrFill(factory);
        registerAggregateFunctionCombinatorResample(factory);
        registerAggregateFunctionCombinatorDistinct(factory);
        registerAggregateFunctionCombinatorMap(factory);
        registerAggregateFunctionCombinatorTuple(factory);
        registerAggregateFunctionCombinatorsArgMinArgMax(factory);
    }

Thirteen registration calls, contributing fifteen suffixes between them: If, Array, ForEach, SimpleState, State, Merge, Null, OrNull, OrDefault, Resample, Distinct, Map, Tuple, ArgMin and ArgMax. Two of the registrations contribute two suffixes each. sumOrNull and sumOrDefault come from one registration, parameterised by an enum:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionOrFill.cpp:16
class AggregateFunctionCombinatorOrFill final : public IAggregateFunctionCombinator
{
private:
    Kind kind;

public:
    explicit AggregateFunctionCombinatorOrFill(Kind kind_) : kind(kind_) {}

    /// Due to aggregate_functions_null_for_empty
    bool supportsNesting() const override { return true; }

    String getName() const override
    {
        return kind == Kind::OrNull ? "OrNull" : "OrDefault";
    }

Note supportsNesting() == true for OrNull: sumOrNullOrNull is legal, because the outer one just needs to know whether the inner one saw any rows.

The factory: strip, recurse, wrap

The whole dispatch lives in one function of AggregateFunctionFactory. It is the second thing the factory tries, after the exact-name lookup fails:

cpp 
// src/AggregateFunctions/AggregateFunctionFactory.cpp:302
    /// Combinators of aggregate functions.
    /// For every aggregate function 'agg' and combiner '-Comb' there is a combined aggregate function with the name 'aggComb',
    ///  that can have different number and/or types of arguments, different result type and different behaviour.

    if (AggregateFunctionCombinatorPtr combinator = AggregateFunctionCombinatorFactory::instance().tryFindSuffix(name))
    {
        const std::string & combinator_name = combinator->getName();

        if (combinator->isForInternalUsageOnly())
            throw Exception(ErrorCodes::UNKNOWN_AGGREGATE_FUNCTION,
                "Aggregate function combinator '{}' is only for internal usage",
                combinator_name);

        if (query_context && query_context->getSettingsRef()[Setting::log_queries])
            query_context->addQueryFactoriesInfo(Context::QueryLogFactories::AggregateFunctionCombinator, combinator_name);

        String nested_name = name.substr(0, name.size() - combinator_name.size());

The recursion is the next line that matters. nested_name is the name with the suffix removed — sumIf becomes sum. Then:

cpp 
// src/AggregateFunctions/AggregateFunctionFactory.cpp:334
        Array nested_parameters = combinator->transformParameters(parameters);

        AggregateFunctionPtr combined_function;
        if (combinator->transformsMultipleNestedFunctions())
        {
            /// A combinator that wraps one nested function per argument list (e.g. -Tuple: one per
            /// tuple element).
            auto nested_arguments_list = combinator->transformArgumentsForMultipleNestedFunctions(argument_types);

            VectorWithMemoryTracking<AggregateFunctionPtr> nested_functions;
            nested_functions.reserve(nested_arguments_list.size());
            for (const auto & nested_arguments : nested_arguments_list)
                nested_functions.push_back(get(nested_name, action, nested_arguments, nested_parameters, out_properties, state_variant));

            /// A `-State` round-trip reconstructs every element from this one shared name, so it must be
            /// the action-adjusted base aggregate name, not one element's instantiation (which can collapse
            /// to a placeholder for only-null elements and drop the other elements' state).
            String adjusted_nested_name = getAssociatedNameByNullsAction(getAliasToOrName(nested_name), action);
            combined_function = combinator->transformAggregateFunctionFromMultipleNestedFunctions(
                adjusted_nested_name, std::move(nested_functions), out_properties, argument_types, parameters);
        }
        else
        {
            DataTypes nested_types = combinator->transformArguments(argument_types);

            AggregateFunctionPtr nested_function = get(nested_name, action, nested_types, nested_parameters, out_properties, state_variant);
            combined_function = combinator->transformAggregateFunction(nested_function, out_properties, argument_types, parameters);
        }

Read the else branch carefully — it is three steps:

  1. transformArguments(argument_types) — strip the combinator’s own arguments. For sumIf(x, cond) the nested function sees only [x].
  2. get(nested_name, nested_types, ...) — recursive call into the factory. This resolves sum with the reduced argument list. Because it is the same factory, the nested call can itself find a combinator: sumIfState resolves sumIf first (suffix State wins over If? — no: State is 5 chars, If is 2, so State is checked first), and that call resolves sum with [x].
  3. transformAggregateFunction(nested_function, ...) — build the decorator.

So sumIfState(x, cond) produces a three-deep chain: sum wrapped by If, wrapped by State. Each layer is a separate object, each forwarding to the one below.

Nesting rules, enforced in one place

Right after stripping the suffix, the factory checks whether the result would be nonsense:

cpp 
// src/AggregateFunctions/AggregateFunctionFactory.cpp:318
        String nested_name = name.substr(0, name.size() - combinator_name.size());
        /// Nested identical combinators (i.e. uniqCombinedIfIf) is not
        /// supported (since they don't work -- silently).
        ///
        /// But non-identical is supported and works. For example,
        /// uniqCombinedIfMergeIf is useful in cases when the underlying
        /// storage stores AggregateFunction(uniqCombinedIf) and in SELECT you
        /// need to filter aggregation result based on another column.

        if (!combinator->supportsNesting() && nested_name.ends_with(combinator_name))
        {
            throw Exception(ErrorCodes::ILLEGAL_AGGREGATION,
                "Nested identical combinator '{}' is not supported",
                combinator_name);
        }

sumIfIf is rejected: two nested If combinators would each want to be the last argument, and they cannot both be. But sumIfMergeIf is fine, because If and Merge need different things.

This is why supportsNesting() exists per combinator rather than as a global rule.

-If: the conditional flag argument

The -If combinator is the simplest and the most instructive. Its transformArguments drops the last argument — the condition:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionIf.cpp:25
    DataTypes transformArguments(const DataTypes & arguments) const override
    {
        if (arguments.empty())
            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
                "Incorrect number of arguments for aggregate function with {} suffix", getName());

        /** The last argument is the condition, and saying only that its type is illegal helps nobody: the
          * most common way to get here is forgetting the condition altogether - `sumIf(x)` - where the
          * argument being blamed is a perfectly good argument of `sum`. Name what the argument is for and
          * what it has to be.
          */
        if (!isUInt8(arguments.back()) && !arguments.back()->onlyNull())
            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
                            "Illegal type {} of the last argument of an aggregate function with the {} "
                            "suffix: the last argument is the condition and must be UInt8. If the condition "
                            "is missing, it goes after the arguments of the aggregate function",
                            arguments.back()->getName(), getName());

        return DataTypes(arguments.begin(), std::prev(arguments.end()));
    }

The error message is a small piece of user-facing engineering: it knows that the most likely mistake is sumIf(x) — forgetting the condition — and it says so.

The decorator itself is almost pure forwarding. Its add reads the condition column and delegates only when the condition holds:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionIf.h:135
    void add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena * arena) const override
    {
        if (only_null_condition)
            return;
        if (assert_cast<const ColumnUInt8 &>(*columns[num_arguments - 1]).getData()[row_num])
            nested_func->add(place, columns, row_num, arena);
    }

But the batch path is where the design pays off. Instead of testing the condition itself, the If decorator hands the position of the condition column down to the nested function, which then tests it inside its own (inlinable) loop:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionIf.h:151
    void addBatch(
        size_t row_begin,
        size_t row_end,
        AggregateDataPtr * __restrict places,
        size_t place_offset,
        const IColumn ** columns,
        Arena * arena,
        ssize_t) const override
    {
        if (only_null_condition)
            return;
        nested_func->addBatch(row_begin, row_end, places, place_offset, columns, arena, num_arguments - 1);
    }

num_arguments - 1 is the index of the condition column. The nested function’s generic addBatch (in IAggregateFunctionHelper) knows how to read a UInt8 flag column and skip rows:

cpp 
// src/AggregateFunctions/IAggregateFunction.h:637
    void addBatch( /// NOLINT
        size_t row_begin,
        size_t row_end,
        AggregateDataPtr * places,
        size_t place_offset,
        const IColumn ** columns,
        Arena * arena,
        ssize_t if_argument_pos = -1) const override
    {
        if (if_argument_pos >= 0)
        {
            const auto & flags = assert_cast<const ColumnUInt8 &>(*columns[if_argument_pos]).getData();
            for (size_t i = row_begin; i < row_end; ++i)
            {
                if (flags[i] && places[i])
                    static_cast<const Derived *>(this)->add(places[i] + place_offset, columns, i, arena);
            }
        }

So the -If decorator adds zero per-row overhead relative to the base function, provided the base function uses the generic addBatch. The condition check is one UInt8 load and a branch, folded into a loop that already existed.

Note that the decorator’s own add (the single-row path) still tests the condition itself. Both paths must exist, because add is also used by the merging code and by -Array.

The nested-identical-If special case

Look at addBatchSinglePlace in the If decorator:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionIf.h:180
    void addBatchSinglePlace(
        size_t row_begin,
        size_t row_end,
        AggregateDataPtr __restrict place,
        const IColumn ** columns,
        Arena * arena,
        ssize_t if_argument_pos) const override
    {
        if (only_null_condition)
            return;
        /// The nested function takes a single condition, so the condition of an enclosing combinator is applied row by row.
        if (if_argument_pos >= 0)
        {
            IAggregateFunctionHelper<AggregateFunctionIf>::addBatchSinglePlace(row_begin, row_end, place, columns, arena, if_argument_pos);
            return;
        }
        nested_func->addBatchSinglePlace(row_begin, row_end, place, columns, arena, num_arguments - 1);
    }

The if_argument_pos >= 0 branch means an enclosing combinator is also filtering. In that case the inner If cannot pass a position down (the nested function only understands one condition column), so the inner If falls back to the generic helper, which applies the outer condition row by row, and the inner condition is checked inside add.

This is the combinatorial explosion problem that combinator design has to cope with, and it is handled by making the slow path the fallback.

-State: a function whose result is a state

-State is the odd one out: it does not change how values are accumulated, it changes what comes out. Its result type is a data type:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionState.h:46
    static DataTypePtr createResultType(const AggregateFunctionPtr & nested_)
    {
        DataTypeAggregateFunction::checkSupportedFunctions(nested_);
        return nested_->getStateType();
    }

and its insertResultInto does not compute a value at all — it moves the state pointer into the result column:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionState.h:173
    void insertResultInto(AggregateDataPtr __restrict place, IColumn & to, Arena *) const override
    {
        auto & column = assert_cast<ColumnAggregateFunction &>(to);

        /// Only once the column holds an aliased state, which is the partial transfer to undo.
        if (unlikely(!column.empty()))
        {
            fiu_do_on(FailPoints::aggregate_function_state_transfer_throw,
            {
                throw Exception(ErrorCodes::MEMORY_LIMIT_EXCEEDED, "Injected failure in AggregateFunctionState::insertResultInto");
            });
        }

        column.getData().push_back(place);
    }

push_back(place) — the pointer is stored, not a copy of the bytes. The state is not serialized, not reallocated, not copied. The result column becomes the owner.

This is why -State needs the destroyUpToState machinery from the previous article: after the pointer is transferred, the aggregator must not destroy the state, but must still destroy anything the combinators after -State allocated.

And isState() is true:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionState.h:201
    /// Aggregate function or aggregate function state.
    bool isState() const override { return true; }

which changes downstream behaviour — runningAccumulate and window functions take different paths for a state-producing function.

-Merge: the inverse of -State

-Merge turns a state function back into an aggregating function. Its argument is a column of states, typed as AggregateFunction(...):

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionMerge.cpp:89
    DataTypes transformArguments(const DataTypes & arguments) const override
    {
        if (arguments.size() != 1)
            throw Exception(
                ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
                "Incorrect number of arguments for aggregate function with {} suffix",
                getName());

        const DataTypePtr & argument = arguments[0];

        const DataTypeAggregateFunction * function = typeid_cast<const DataTypeAggregateFunction *>(argument.get());
        if (!function)
            throw Exception(
                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
                "Illegal type {} of argument for aggregate function with {} suffix. It must be AggregateFunction(...)",
                argument->getName(),
                getName());

        return function->getArgumentsDataTypes();
    }

This is the clever bit. The nested function’s argument types are not declared by the user — they are recovered from the type of the state column. sumMerge applied to a column of type AggregateFunction(sum, UInt64) learns that the underlying function is sum over UInt64 because the type says so.

That means the type system carries the information needed to reconstruct the function, which is what makes this work across a network:

sql 
CREATE TABLE hits (
    uid UInt64,
    hits_state AggregateFunction(sum, UInt64)
) ENGINE = AggregatingMergeTree ORDER BY uid;

INSERT INTO hits SELECT number, sumState(number) FROM numbers(10);

-- the stored states are merged automatically by the engine's background merge
SELECT sumMerge(hits_state) FROM hits;

The AggregateFunction(sum, UInt64) in the column type is the recipe. When the engine deserializes a part, it parses that type name, reconstructs the sum function object, and knows exactly how to merge the state bytes.

The add method of the Merge decorator is where the two meet:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionMerge.cpp:65
void AggregateFunctionMerge::add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena * arena) const
{
    const auto & column = assert_cast<const ColumnAggregateFunction &>(*columns[0]);
    auto * const rhs = column.getData()[row_num];

    if (!merge_state_from_different_variant)
    {
        nested_func->merge(place, rhs, arena);
        return;
    }

    nested_func->mergeStateFromDifferentVariant(place, *argument_func, rhs, arena);
}

column.getData()[row_num] is the pointer to the source state. merge takes the destination state and the source state and combines them. No serialization, no copying — the merge happens in place on the raw bytes.

The state-type equality check

Merge’s constructor verifies that the state column’s function and the nested function agree:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionMerge.cpp:33
    if (nested_func->haveSameStateRepresentation(*argument_func))
        return;

    if (data_type->getFunctionName() != nested_func->getName())
        throw Exception(
            ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
            "Illegal type {} of argument for aggregate function with Merge suffix because it corresponds to different aggregate function: "
            "{} instead of {}",
            argument->getName(),
            data_type->getFunctionName(),
            nested_func->getName());

haveSameStateRepresentation is the question “can these two states be merged?” — and it is answered by comparing the state types, which as we saw hash the function name, the parameters, the argument types, the version and the state variant.

Note the subtlety: quantile(0.5)(x) and quantile(0.9)(x) have different parameters but the same state representation, because the parameter only affects finalization, not the state layout. The Merge combinator accepts both.

-OrNull: one extra byte

The -OrNull combinator is the smallest interesting one: it needs to know whether the nested function saw any rows. It answers that with a single byte stored after the nested state:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionOrFill.h:113
    size_t sizeOfData() const override
    {
        /// Pad to alignment so that arrays of states (e.g. in -ForEach) keep each element aligned.
        return ::Memory::alignUp(size_of_data + sizeof(char), alignOfData());
    }

and create initialises it:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionOrFill.h:124
    void create(AggregateDataPtr __restrict place) const override
    {
        nested_function->create(place);
        place[size_of_data] = 0;
    }

The extra byte is at offset size_of_data — immediately after the nested state — and the size is rounded up to the alignment so that arrays of states stay aligned.

The Null combinator does the mirror-image trick, but before the nested state, to keep the nested state’s alignment:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionNull.h:53
    /** In addition to data for nested aggregate function, we keep a flag
      *  indicating - was there at least one non-NULL value accumulated.
      * In case of no not-NULL values, the function will return NULL.
      *
      * We use prefix_size bytes for flag to satisfy the alignment requirement of nested state.
      */

Prefix size for Null, suffix size for OrFill. Same idea, opposite direction, both dictated by alignment.

-ArgMin / -ArgMax: the trailing key

One more combinator is worth a look, because it does the opposite of -If: it appends an argument rather than removing one.

sumArgMin(x, key) computes the sum of x over the rows where key is minimal. The key argument is the last one, and the nested function never sees it:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionCombinatorsArgMinArgMax.cpp:247
    DataTypes transformArguments(const DataTypes & arguments) const override
    {
        if (arguments.empty())
            throw Exception(
                ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
                "Incorrect number of arguments for aggregate function with {} suffix",
                getName());

        return DataTypes(arguments.begin(), arguments.end() - 1);
    }

arguments.end() - 1 — everything except the last one goes to the nested function. The combinator’s own name follows the same pattern as -If:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionCombinatorsArgMinArgMax.cpp:239
    String getName() const override
    {
        if constexpr (isMin)
            return "ArgMin";
        else
            return "ArgMax";
    }

and the decorator’s own name is the nested name with the suffix appended:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionCombinatorsArgMinArgMax.cpp:92
    String getName() const override
    {
        if constexpr (isMin)
            return nested_function->getName() + "ArgMin";
        else
            return nested_function->getName() + "ArgMax";
    }

So sumArgMin is a different aggregate function from argMin — the suffix is on the outside, and the nested function is the plain sum. That is the opposite of the argMin(x, y) aggregate function, which is a single base function taking two arguments.

The Null combinator is applied automatically

One combinator is not applied by the user. If any argument of an aggregate function is Nullable, the factory wraps the function in Null without being asked:

cpp 
// src/AggregateFunctions/AggregateFunctionFactory.cpp:101
    /// If one of the types is Nullable, we apply aggregate function combinator "Null" if it's not window function.
    /// Window functions are not real aggregate functions. Applying combinators doesn't make sense for them,
    /// they must handle the nullability themselves.
    /// Aggregate functions such as any_value_respect_nulls are considered window functions in that sense
    auto properties = tryGetProperties(name, action);
    bool is_window_function = properties.has_value() && properties->is_window_function;
    if (!is_window_function && std::any_of(types_without_low_cardinality.begin(), types_without_low_cardinality.end(),
        [](const auto & type) { return type->isNullable(); }))
    {
        AggregateFunctionCombinatorPtr combinator = AggregateFunctionCombinatorFactory::instance().tryFindSuffix("Null");
        if (!combinator)
            throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot find aggregate function combinator "
                            "to apply a function to Nullable arguments.");

        DataTypes nested_types = combinator->transformArguments(types_without_low_cardinality);
        Array nested_parameters = combinator->transformParameters(parameters);

        bool has_null_arguments = std::any_of(types_without_low_cardinality.begin(), types_without_low_cardinality.end(),
            [](const auto & type) { return type->onlyNull(); });

        AggregateFunctionPtr nested_function = getImpl(name, action, nested_types, nested_parameters, out_properties, has_null_arguments, state_variant);

So sum(x) where x is Nullable(Int64) is really sumNull(x) where Null strips the nullability, and a separate flag byte remembers whether any non-null value was seen.

The name is not the identity

One last wrinkle, and it is a trap. The name of a function is not enough to identify it. Two functions with the same name and different argument types are different functions, and the state type carries the difference:

cpp 
// src/AggregateFunctions/IAggregateFunction.h:137
    /// Returns true if two aggregate functions have the same state representation in memory and the same serialization,
    /// so state of one aggregate function can be safely used with another.
    /// Examples:
    ///  - quantile(x), quantile(a)(x), quantile(b)(x) - parameter doesn't affect state and used for finalization only
    ///  - foo(x) and fooIf(x) - If combinator doesn't affect state
    /// By default returns true only if functions have exactly the same names, combinators and parameters.

That last line is the point: foo(x) and fooIf(x) have different names but the same state representation, because the -If combinator adds no bytes to the state. The If decorator’s sizeOfData simply forwards to the nested function, so a sumIf state and a sum state are byte-identical.

This is why getStateType() for the If combinator delegates:

cpp 
// src/AggregateFunctions/Combinators/AggregateFunctionIf.h:100
    DataTypePtr getStateType() const override
    {
        return this->getStateTypeWithVersionOf(*nested_func);
    }

and why OrNull — which does add a byte — has a different state type.

The trade-offs

What this buys you.

  • An enormous function space from a small codebase. Thirteen combinators multiplied by a few hundred base functions gives thousands of functions, and the base functions are written once. topK, topKIf, topKState, topKMerge are all the same topK with different wrappers.
  • Uniform state semantics. Because every combinator either forwards sizeOfData or adds a well-defined prefix/suffix, the state layout of a combined function is predictable. sumIf and sum share a state type; sumOrNull and sum do not.
  • The type system carries the recipe. AggregateFunction(sum, UInt64) is enough to reconstruct the function, which is what lets a state column survive a round trip through disk, through the network, and through a schema migration.
  • Compositional optimizations. The -If decorator hands the condition column position down so the base function can filter inside its own inlined loop — zero extra cost for a conditional aggregate.

What it costs you.

  • The name is a program, and programs have surprising semantics. sumIfIf is an error while sumIfMergeIf is legal, and the difference is a supportsNesting() boolean per combinator. Users have to learn a small grammar.
  • Combinatorial explosion in the forwarding methods. If alone has to forward add, addManyDefaults, addBatch, addBatchWithNonNullPlaces, addBatchSinglePlace, addBatchSinglePlaceNotNull, mergeImpl, serialize, deserialize, insertResultInto — and each one has to decide what to do with the condition argument. Every new batch variant is a chance to get it wrong.
  • The decorator is not free. Each combinator layer adds a virtual call and a shared_ptr indirection on the hot path. The -If decorator mitigates this by pushing the condition down, but a combinator that cannot do that pays a branch per row.
  • State layout is a public contract. Once a state is on disk, the combinator’s byte layout is frozen. OrFill had to align its extra byte to the end of the nested state, and Null to the beginning — choices that are now permanent.

The essential trick is that a combinator is not a subclass of the base function, not a template parameter of it, and not a wrapper around its values. It is a wrapper around the base function object, and everything else — the argument rewriting, the recursion, the state layout, the serialization — falls out of that one decision.

ClickhouseInternalsAggregationDesign