Query-Plan on Opensource internals

ClickHouse Rewrites the Query Before It Plans It

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: The query you type is not the query that gets planned. ClickHouse first turns the AST into a typed query tree, inlining every alias as it goes, then runs 47 ordered rewrite passes over that tree — pruning unused columns, normalising count(NULL) to count(), collapsing if chains — and finally, in the planner, recognises that SELECT count() FROM t never needs to read a single row and answers it from the parts’ metadata.

A query arrives at the server as an AST: a tree of parser nodes with ASTIdentifier leaves and no type information at all. The plan that actually runs is something else entirely — a typed ActionsDAG per expression, a tree of QueryPlanSteps, and a set of processors reading parts in a particular order.

The gap between the two is closed by a pipeline of rewrites. Most of them happen on the query tree, and the last and most spectacular one happens in the planner, where a count() query is not executed at all.

Stage 1: aliases are inlined

SQL lets you write

 sql
 
SELECT a + b AS s, s * 2 FROM t;

and the standard says the second column may reference the first. ClickHouse resolves this by substituting the aliased expression wherever the alias appears — but there are two different mechanisms for it, depending on which pipeline you are in.

The analyzer: the alias is a cloned expression node

In the current pipeline the query is a tree of typed nodes, and an alias is an entry in a per-scope map from name to expression node:

 cpp
 
 
// src/Analyzer/Resolve/ScopeAliases.h:10
struct ScopeAliases
{
 /// Alias name to query expression node
 std::unordered_map<std::string, QueryTreeNodePtr> alias_name_to_expression_node;

 /// Alias name to lambda node
 std::unordered_map<std::string, QueryTreeNodePtr> alias_name_to_lambda_node;

 /// Alias name to table expression node
 std::unordered_map<std::string, QueryTreeNodePtr> alias_name_to_table_expression_node;

 /// Nodes with duplicated aliases
 QueryTreeNodes nodes_with_duplicated_aliases;

 /// Cloned resolved expressions with aliases that must be removed
 QueryTreeNodes node_to_remove_aliases;

 UnorderedMapWithMemoryTracking<std::string, DataTypePtr> alias_name_to_expression_type;

The map is filled by a visitor that clones every aliased expression, so the stored node is a snapshot of the expression, not a reference to a node that will keep changing:

 cpp
 
 
// src/Analyzer/Resolve/QueryExpressionsAliasVisitor.h:97
 const auto & alias = node->getAlias();
 auto cloned_alias_node = node->clone();

 switch (node_type)
 {
 case QueryTreeNodeType::LAMBDA:
 {
 auto [_, inserted] = aliases.alias_name_to_lambda_node.emplace(alias, cloned_alias_node);
 if (!inserted || aliases.alias_name_to_expression_node.contains(alias))
 addDuplicatingAlias(cloned_alias_node);
 break;
 }

There are three namespaces — expression, lambda, table expression — because SELECT 1 AS x, arrayMap(x -> x + 2, [1,2,3]) must not let the lambda parameter x bind to the projection alias x.

When an identifier is resolved and it turns out to be an alias, the alias node is cloned again and resolved in place of the identifier:

 cpp
 
 
// src/Analyzer/Resolve/QueryAnalyzer.cpp:1462
 auto node_type = alias_node->getNodeType();

 if (!identifier_lookup.isTableExpressionLookup())
 {
 alias_node = alias_node->clone();
 scope_to_resolve_alias_expression->aliases.node_to_remove_aliases.push_back(alias_node);
 }

The clone is essential: the same alias can be referenced from several places, and each reference needs its own resolved copy (a ColumnNode bound to a specific table expression, a function node with a resolved overload). The documentation block above the function spells out the whole algorithm:

 cpp
 
 
// src/Analyzer/Resolve/QueryAnalyzer.cpp:1400
 * Below cases should work:
 * Example:
 * SELECT id AS id FROM test_table;
 * SELECT value.value1 AS value FROM test_table;
 * SELECT (id + 1) AS id FROM test_table;
 * SELECT (1 + (1 + id)) AS id FROM test_table;
 *
 * Below cases should throw cyclic aliases exception:
 * SELECT (id + b) AS id, id as b FROM test_table;
 * SELECT (1 + b + 1 + id) AS id, b as c, id as b FROM test_table;
 *
 * 3. Depending on IdentifierLookupContext select IdentifierResolveScope to resolve aliased expression.
 * 4. Clone query tree of aliased expression (do not clone table expression from join tree).
 * 5. Resolve the node. It is important in case of compound expressions.
 * Example: SELECT value.a, cast('(1)', 'Tuple(a UInt64)') AS value;

The cycle detection is not a separate pass — it falls out of the resolution stack. If the alias currently being resolved is on the stack, resolving it again would be a cycle:

 cpp
 
 
// src/Analyzer/Resolve/QueryAnalyzer.cpp:1470
 /* Do not use alias to resolve identifier when it's part of aliased expression. This is required to support queries like:
 * 1. SELECT dummy + 1 AS dummy
 * 2. SELECT avg(a) OVER () AS a, id FROM test
 */
 if (scope.expressions_in_resolve_process_stack.getExpressionWithAlias(identifier_bind_part) != nullptr)
 {
 /* This is an important fallback in the identifier resolution. In the case of transitive aliases,
 * we may end up in the situation when we try to resolve the same expression, but it's not a cycle.
 * Example: WITH path('clickhouse.com/a/b/c') AS x SELECT x AS path;
 * Here, identifier `x` would be resolved one time as a projection column and one time during `path`
 * function lookup. The second lookup must fail here to break the alias cycle.
 */
 return {};
 }

The AST normaliser: the older way

There is a second, older implementation of the same idea, still used by the legacy interpreter’s tree rewriter and by the rewriting of user-defined SQL functions. It works on the AST and substitutes aliases by cloning the aliased AST node over the identifier:

 cpp
 
 
// src/Interpreters/QueryNormalizer.cpp:106
 if (it_alias != data.aliases.end() && current_alias != node.name())
 {
 if (!IdentifierSemantic::canBeAlias(node))
 return;

 /// We are alias for other column (node.name), but we are alias by
 /// ourselves to some other column
 const auto & alias_node = it_alias->second;

 String our_alias_or_name = alias_node->getAliasOrColumnName();
 std::optional<String> our_name = IdentifierSemantic::getColumnName(alias_node);

 String node_alias = ast->tryGetAlias();

 if (current_asts.contains(alias_node.get()) /// We have loop of multiple aliases
 || (node.name() == our_alias_or_name && our_name && node_alias == *our_name)) /// Our alias points to node.name, direct loop
 throw Exception(ErrorCodes::CYCLIC_ALIASES, "Cyclic aliases");

The current_asts set is the cycle detector: SELECT a AS b, b AS a is a CYCLIC_ALIASES error rather than a stack overflow. The RestoreAliasOnExitScope guard restores the previous alias name when the visitor leaves a subtree, so an alias defined inside a subquery does not leak out of it.

The substitution has to preserve the alias of the substituted node, which is why there are two nearly identical branches:

 cpp
 
 
// src/Interpreters/QueryNormalizer.cpp:124
 /// Let's replace it with the corresponding tree node.
 if (!node_alias.empty() && node_alias != our_alias_or_name)
 {
 /// Avoid infinite recursion here
 auto opt_name = IdentifierSemantic::getColumnName(alias_node);
 bool is_cycle = opt_name && *opt_name == node.name();

 if (!is_cycle)
 {
 /// In a construct like "a AS b", where a is an alias, you must set alias b to the result of substituting alias a.
 /// Check size of the alias before cloning too large alias AST
 alias_node->checkSize(data.settings.max_expanded_ast_elements);
 ast = alias_node->clone();
 ast->setAlias(node_alias);

checkSize is the guard against the quadratic blow-up: an alias that is cloned into a thousand places is a thousand AST nodes, so the size of the cloned subtree is checked against max_expanded_ast_elements before the clone.

Two subtleties worth noting:

Lambdas hide their parameters. SELECT 1 AS x, arrayMap(x -> x + 2, [1,2,3]) must not substitute the outer x into the lambda body. The normaliser extracts the lambda’s own parameter names and removes them from the alias map for the duration of the lambda body:

 cpp
 
 
// src/Interpreters/QueryNormalizer.cpp:240
 /// For lambda functions we need to avoid replacing lambda parameters with external aliases, for example,
 /// Select 1 as x, arrayMap(x -> x + 2, [1, 2, 3])
 /// shouldn't be replaced with Select 1 as x, arrayMap(x -> **(1 as x)** + 2, [1, 2, 3])
 Aliases extracted_aliases;
 if (func_node->name == "lambda")
 {
 Names lambda_aliases = RequiredSourceColumnsMatcher::extractNamesFromLambda(*func_node);
 for (const auto & name : lambda_aliases)
 {
 auto it = data.aliases.find(name);
 if (it != data.aliases.end())
 {
 extracted_aliases.insert(data.aliases.extract(it));
 }
 }
 }

A column name beats an alias if you ask for it. The setting prefer_column_name_to_alias lets SELECT x FROM (SELECT 1 AS x) resolve to the column x rather than the alias.

The normaliser also memoises: finished_asts maps an already-processed AST to its rewritten form, so an expression that appears in a thousand places is rewritten once and shared:

 cpp
 
 
// src/Interpreters/QueryNormalizer.cpp:341
 finished_asts[initial_ast] = ast;

Stage 2: the query tree and its 47 passes

The new pipeline does not use the AST as its working representation. The parser’s output is converted into a query tree — typed nodes (ColumnNode, FunctionNode, ConstantNode, LambdaNode, WindowFunctionNode, QueryNode, JoinNode, …) — and every node in it carries its resolved data type.

The rewriting is then done by a pipeline of passes, and the order of the passes is part of the design:

 cpp
 
 
// src/Analyzer/QueryTreePassManager.cpp:269
void addQueryTreePasses(QueryTreePassManager & manager, bool only_analyze, bool positional_arguments_already_resolved)
{
 manager.addPass(std::make_unique<QueryAnalysisPass>(only_analyze, positional_arguments_already_resolved));
 manager.addPass(std::make_unique<GroupingFunctionsResolvePass>());
 manager.addPass(std::make_unique<AutoFinalOnQueryPass>());
 /// This pass should be run for the secondary queries
 /// to ensure that the only required columns are read from VIEWs on the shards.
 manager.addPass(std::make_unique<RemoveUnusedProjectionColumnsPass>());
 manager.addPass(std::make_unique<PruneArrayJoinColumnsPass>());

 manager.addPass(std::make_unique<DictGetTupleElementPass>());
 manager.addPass(std::make_unique<ConvertEmptyStringComparisonToFunctionPass>());
 manager.addPass(std::make_unique<FunctionToSubcolumnsPass>());

 manager.addPass(std::make_unique<ConvertLogicalExpressionToCNFPass>());
 manager.addPass(std::make_unique<RegexpFunctionRewritePass>());

 manager.addPass(std::make_unique<RewriteSumFunctionWithSumAndCountPass>());
 manager.addPass(std::make_unique<CountDistinctPass>());
 manager.addPass(std::make_unique<UniqToCountPass>());
 manager.addPass(std::make_unique<RewriteArrayExistsToHasPass>());
 manager.addPass(std::make_unique<RewriteArrayFilterLengthToArrayCountPass>());
 manager.addPass(std::make_unique<RewriteHasToInPass>());
 manager.addPass(std::make_unique<NormalizeCountVariantsPass>());

The list continues for another forty passes — AggregateFunctionOfGroupByKeysPass, OptimizeGroupByFunctionKeysPass, MultiIfToIfPass, IfConstantConditionPass, OrderByTupleEliminationPass, FuseFunctionsPass, LogicalExpressionOptimizerPass, OptimizeTrivialGroupByLimitPass — and the ordering constraints are spelled out in the comments:

 cpp
 
 
// src/Analyzer/QueryTreePassManager.cpp:306
 /// These passes can rewrite a predicate so that it depends on base columns instead of
 /// the derived expression that appears in `GROUP BY`. They must run before
 /// `OptimizeGroupByFunctionKeysPass` and `OptimizeGroupByInjectiveFunctionsPass`,
 /// because those passes remove grouping keys based on the current expression tree.
 /// For example, `SELECT toYear(d), toYear(d) = 2024, count() FROM ... GROUP BY ALL`:
 /// `OptimizeGroupByFunctionKeysPass` can keep only `toYear(d)` as the grouping key,
 /// and if `OptimizeDateOrDateTimeConverterWithPreimagePass` runs later it rewrites
 /// `toYear(d) = 2024` to a range on raw `d`. After aggregation only `toYear(d)` and
 /// `count()` remain, so the final projection would need `d` that is no longer present.
 manager.addPass(std::make_unique<InverseDictionaryLookupPass>());
 manager.addPass(std::make_unique<OptimizeDateOrDateTimeConverterWithPreimagePass>());
 manager.addPass(std::make_unique<ComparisonTupleEliminationPass>());

That is a dependency between passes, documented in prose right where the dependency is enforced. There are about a dozen such comments in this one function.

The manager runs them in order and, in debug builds, validates the tree after every pass:

 cpp
 
 
// src/Analyzer/QueryTreePassManager.cpp:195
void QueryTreePassManager::run(QueryTreeNodePtr & query_tree_node)
{
 auto current_context = getContext();
 size_t passes_size = passes.size();

 for (size_t i = 0; i < passes_size; ++i)
 {
 passes[i]->run(query_tree_node, current_context);
#if defined(DEBUG_OR_SANITIZER_BUILD)
 ValidationChecker(passes[i]->getName()).visit(query_tree_node);
#endif
 }
}

and the validator is not a syntax check — it re-checks that every function’s declared argument types still match the types of the arguments it actually has, which is the invariant that a sloppy rewrite pass would break:

 cpp
 
 
// src/Analyzer/QueryTreePassManager.cpp:159
 if (!expected_argument_type->equals(*actual_argument_type))
 {
 /// Aggregate functions remove low cardinality for their argument types
 if ((function->isAggregateFunction() || function->isWindowFunction()) &&
 expected_argument_type->equals(*recursiveRemoveLowCardinality(actual_argument_type)))
 continue;

 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Function {} expects argument {} to have {} type but receives {} after running {} pass",
 function->toAST()->formatForErrorMessage(),
 i + 1,
 expected_argument_type->getName(),
 actual_argument_type->getName(),
 pass_name);
 }

That is a lot of machinery for a pipeline, and it is worth remembering that the first pass is the one that does the actual name resolution: it turns ASTIdentifier leaves into ColumnNodes bound to a specific table expression, and ASTFunction into FunctionNodes with a resolved overload and a result type.

Stage 3: pruning the projection of a subquery

The single most valuable pass for a distributed query is RemoveUnusedProjectionColumnsPass, and it runs fourth — before almost everything else — for exactly the reason its comment gives:

 cpp
 
 
// src/Analyzer/QueryTreePassManager.cpp:274
 /// This pass should be run for the secondary queries
 /// to ensure that the only required columns are read from VIEWs on the shards.
 manager.addPass(std::make_unique<RemoveUnusedProjectionColumnsPass>());

The mechanism is a bottom-up collection of used column names per subquery, followed by a rewrite of the subquery’s projection. The collection is a traversal of the query tree that records, for every query/union node used as a table expression, the names of the columns that the outer query actually reads:

 cpp
 
 
// src/Analyzer/Passes/RemoveUnusedProjectionColumnsPass.cpp:115
 /// Collect information about what columns are used in the query.
 traverseQueryTree(node_to_visit,
 [&subqueries_nodes_to_visit, &node_to_used_columns](
 const QueryTreeNodePtr & /*parent*/,
 const QueryTreeNodePtr & child
 )
 {
 if (isQueryOrUnionNode(child))
 {
 subqueries_nodes_to_visit.insert(child);

 auto * query_node = child->as<QueryNode>();
 auto * union_node = child->as<UnionNode>();

 const auto & correlated_columns = query_node != nullptr ? query_node->getCorrelatedColumns() : union_node->getCorrelatedColumns();
 for (const auto & correlated_column : correlated_columns)
 {
 auto * column_node = correlated_column->as<ColumnNode>();
 auto column_source_node = column_node->getColumnSource();
 auto column_source_node_type = column_source_node->getNodeType();
 if (column_source_node_type == QueryTreeNodeType::QUERY || column_source_node_type == QueryTreeNodeType::UNION)
 {
 if (auto it = node_to_used_columns.find(column_source_node); it != node_to_used_columns.end())
 it->second.insert(column_node->getColumnName());
 }
 }
 return false;
 }
 return true;
 },

and the pruning itself:

 cpp
 
 
// src/Analyzer/Passes/RemoveUnusedProjectionColumnsPass.cpp:167
 /// Pass information about used columns to subqueries and remove unused projection columns
 for (auto & [query_or_union_node, used_columns] : node_to_used_columns)
 {
 /// can't remove columns from distinct, see example - 03023_remove_unused_column_distinct.sql
 if (auto * query_node = query_or_union_node->as<QueryNode>())
 {
 if (query_node->isDistinct())
 continue;
 }

 auto used_projection_indexes = convertUsedColumnNamesToUsedProjectionIndexes(query_or_union_node, used_columns);
 updateUsedProjectionIndexes(query_or_union_node, used_projection_indexes);

 /// Keep at least 1 column if used projection columns are empty
 if (used_projection_indexes.empty())
 used_projection_indexes.insert(0);

Two details show how careful this has to be:

DISTINCT blocks the pruning, because DISTINCT deduplicates over the whole row, so dropping a column changes the number of groups.

Aggregate functions and arrayJoin in the projection block the pruning of that column, even if nobody reads it, because they change the number of rows:

 cpp
 
 
// src/Analyzer/Passes/RemoveUnusedProjectionColumnsPass.cpp:79
 for (size_t i = 0; i < projection_nodes_size; ++i)
 {
 const auto & projection_node = projection_nodes[i];
 if ((!query_node.hasGroupBy() && hasAggregateFunctionNodes(projection_node)) || hasFunctionNode(projection_node, "arrayJoin"))
 used_projection_columns_indexes.insert(i);
 }

The payoff is a distributed query like

 sql
 
SELECT a FROM (SELECT a, b, c FROM t) SETTINGS distributed_group_by_no_merge = 1;

where the subquery is rewritten to SELECT a FROM t before the plan is sent to the shards, so b and c are never read, never serialised over the network, and never joined.

Stage 4: normalising the aggregates

Some passes change the meaning of the query only in the strictest sense. NormalizeCountVariantsPass rewrites count(NULL) to count() and sum(1) to count():

 cpp
 
 
// src/Analyzer/Passes/NormalizeCountVariantsPass.cpp:30
 void enterImpl(QueryTreeNodePtr & node)
 {
 if (!getSettings()[Setting::optimize_normalize_count_variants])
 return;

 auto * function_node = node->as<FunctionNode>();
 if (!function_node || !function_node->isAggregateFunction() || (function_node->getFunctionName() != "count" && function_node->getFunctionName() != "sum"))
 return;

 if (function_node->getArguments().getNodes().size() != 1)
 return;

 /// forbid the optimization if return value of sum() and count() differs:
 /// count() returns only UInt64 type, while sum() could return Nullable().
 if (!function_node->getResultType()->equals(DataTypeUInt64()))
 return;

and then:

 cpp
 
 
// src/Analyzer/Passes/NormalizeCountVariantsPass.cpp:54
 if (function_node->getFunctionName() == "count" && !first_argument_constant_literal.isNull())
 {
 function_node->getArguments().getNodes().clear();
 resolveAggregateFunctionNodeByName(*function_node, "count");
 }
 else if (function_node->getFunctionName() == "sum" &&
 first_argument_constant_literal.getType() == Field::Types::UInt64 &&
 first_argument_constant_literal.safeGet<UInt64>() == 1)
 {
 function_node->getArguments().getNodes().clear();
 resolveAggregateFunctionNodeByName(*function_node, "count");
 }

Why does this matter? Because count() and count(expr) are different aggregates with different implementations: count() has a specialised “count the rows” fast path, and count(expr) has to evaluate the expression on every row and check for NULL. count(NULL) is count() semantically (the argument is always NULL, so nothing is counted), and sum(1) is count() too. Rewriting them lets the trivial-count optimisation fire on queries the user wrote as count(NULL).

Stage 5: the trivial count — a query that reads no data

The last rewrite is the most dramatic, and it happens in the planner rather than in the pass pipeline, because it replaces the plan, not the tree.

 cpp
 
 
// src/Core/Settings.cpp:5792
 DECLARE(Bool, optimize_trivial_count_query, true, R"(
Enables or disables the optimization to trivial query `SELECT count() FROM table` using metadata from MergeTree. If you need to use row-level security, disable this setting.

The planner asks the storage whether the query is a trivial count. The preconditions are checked in one place:

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:576
QueryTreeNodePtr getTrivialCountAggregate(
 const SelectQueryInfo & select_query_info,
 const TableNode * table_node,
 const TableFunctionNode * table_function_node,
 const QueryTreeNodePtr & query_tree,
 const ContextPtr & query_context)
{
 const auto & settings = query_context->getSettingsRef();
 if (!settings[Setting::optimize_trivial_count_query])
 return nullptr;

 /// The rewrite produces a `ReadFromPreparedSource` leaf that the Cascades optimizer cannot
 /// clone; a distributed plan counts the rows with a distributed read instead.
 if (settings[Setting::make_distributed_plan] && settings[Setting::enable_cascades_optimizer])
 return nullptr;

 const auto & storage = table_node ? table_node->getStorage() : table_function_node->getStorage();

 /// `totalRows` and the column stats describe the live table, not the snapshot pinned for this query.
 if (query_context->getPinnedStorageSnapshot(storage->getStorageID().uuid))
 return nullptr;

 if (getEffectiveRowPolicyFilter(*storage, query_context))
 return nullptr;

 if (select_query_info.additional_filter_ast)
 return nullptr;

Each of these exclusions is a correctness rule, not a performance heuristic:

Row-level security — the row policy filters rows, so the metadata count is wrong. Hence the setting’s own advice: “If you need to use row-level security, disable this setting.”
A pinned storage snapshot — totalRows() describes the live table, not the snapshot the query is pinned to.
An additional filter — anything that filters rows invalidates the metadata count.
GROUP BY or PREWHERE — the count is no longer “the number of rows”.

and then the aggregate itself must be exactly one count:

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:615
 QueryTreeNodes aggregates = collectAggregateFunctionNodes(query_tree);
 if (aggregates.size() != 1)
 return nullptr;

 const auto & function_node = aggregates.front()->as<const FunctionNode &>();
 chassert(function_node.getAggregateFunction() != nullptr);
 if (!typeid_cast<const AggregateFunctionCount *>(function_node.getAggregateFunction().get()))
 return nullptr;

 if (!storage->supportsTrivialCountOptimization(
 table_node ? table_node->getStorageSnapshot() : table_function_node->getStorageSnapshot(), query_context))
 return nullptr;

The rewrite itself replaces the entire plan with a one-row source:

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:674
 /// Get number of rows.
 /// MergeTree maintains total count for all parts in Active state and it simply returns that number for trivial
 /// `select count() from table` query. But if we have current transaction, then we should return number of rows in current snapshot
 /// (that may include parts in Outdated state), so the totalRows for transactional case should return the number of rows visible for
 /// the current transaction.
 std::optional<UInt64> num_rows = storage->totalRows(query_context);
 if (!num_rows)
 return false;

and:

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:712
 auto block_with_count = std::make_shared<const Block>(Block{
 {createSingleCountStateColumn(aggregate_function, num_rows.value()),
 std::make_shared<DataTypeAggregateFunction>(aggregate_function, aggregate_function->getArgumentTypes(), Array{}),
 trivial_count_column_name}});

 auto source = std::make_shared<SourceFromSingleChunk>(block_with_count);
 auto prepared_count = std::make_unique<ReadFromPreparedSource>(Pipe(std::move(source)));
 prepared_count->setStepDescription("Optimized trivial count");
 query_plan.addStep(std::move(prepared_count));

Note the shape: the aggregate function’s state is built directly (createSingleCountStateColumn makes a state that already holds the final count), so the plan does not even need an aggregation step. The whole query becomes “read one row from a one-chunk source”.

There is one more trap the code handles explicitly — the distributed and parallel-replica cases, where every replica would answer the same total and the result would be multiplied by the number of replicas:

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:664
 /// Do not apply trivial count optimization if we're the follower. Because if every follower blindly returns totalRows(),
 /// the result will be R times the actual count of rows, where R is the number of replicas.
 /// It's important to make this check after we've set the `select_query_info.optimize_trivial_count` flag, so that storage could apply
 /// some optimization during the read phase.
 /// TODO: We could still avoid reading data for MergeTree on the followers: we could read the parts metadata for the set of parts
 /// we were handed out by the initiator. This would require consuming the `select_query_info.optimize_trivial_count` during the read
 /// phase and acting accordingly.
 if (query_context->canUseParallelReplicasOnFollower())
 return false;

The ordering in the comment is deliberate: the flag optimize_trivial_count is set before the check, because some storages (e.g. StorageURL) act on the flag during the read phase, and the check must not prevent that.

The sparsity-filter variant

A newer, more interesting extension handles SELECT count() FROM t WHERE col <op> const, where the predicate partitions the rows into “rows where col equals its default value” and “rows where it does not”. MergeTree keeps per-column statistics for exactly that:

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:729
bool applyTrivialCountWithSparsityFilterIfPossible(

and the count comes from the statistics:

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:787
 UInt64 num_rows = (classified->predicate_class == SparsityPredicateClass::MatchesDefault)
 ? stats->num_defaults
 : (stats->num_rows - stats->num_defaults);

So SELECT count() FROM t WHERE sparse_col = 0 can be answered from (num_rows, num_defaults) without reading a single row — as long as the predicate is exactly a comparison against a constant that equals the column’s default value, and the aggregate is a zero-argument count():

 cpp
 
 
// src/Planner/PlannerJoinTree.cpp:758
 /// Only zero-argument `count()` / `count(*)` counts rows. `count(expr)` counts non-null
 /// argument values, which the rewrite (which seeds the state with a row count derived from
 /// the per-column `num_defaults`) does not preserve for Nullable/expression counts.
 const auto & function_node = count_aggregate->as<const FunctionNode &>();
 if (!function_node.getArguments().getNodes().empty())
 return false;

The picture

 text
 
 
 SQL text
 |
 | QueryTreeBuilder: AST -> typed query tree
 | QueryAnalysisPass: names and functions resolved,
 | aliases inlined as cloned expression nodes
 v
 query tree
 |
 | ~40 passes, in a fixed order:
 | resolve names & functions
 | RemoveUnusedProjectionColumnsPass (prune subquery projections)
 | NormalizeCountVariantsPass (count(NULL) -> count())
 | MultiIfToIfPass / IfConstantConditionPass / ...
 | LogicalExpressionOptimizerPass (CNF, constant folding)
 v
 resolved query tree
 |
 | Planner: build the plan
 | PlannerJoinTree: is this a trivial count?
 | yes -> ReadFromPreparedSource(one chunk with the count)
 | no -> real plan
 v
 QueryPlan

Trade-offs

What the rewrite pipeline buys:

Correctness by construction for the common cases. count(NULL) and count() are the same thing, but they are different aggregate functions; the pass pipeline makes them the same.
A distributed query reads less on every shard. Column pruning happens before the plan is serialised to the shards, so the shards never even mention the pruned columns.
A count() query costs O(parts), not O(rows). The metadata path reads one number per active part.
Validation as a safety net. The ValidationChecker runs after every pass in debug builds and re-checks type consistency of every function in the tree. For a pipeline of 47 independent rewrites, this is the only practical way to know that pass 37 did not break pass 12’s invariant.

What it costs:

The pass order is load-bearing. A dozen comments in addQueryTreePasses exist purely to explain why pass n must run before pass m. Changing the order is a semantic change, and nothing in the code enforces it — the ordering is a convention documented in prose.
Alias inlining duplicates AST. Cloning an aliased expression into every use site is O(size of the expansion). max_expanded_ast_elements exists precisely because this can explode, and the check happens before the clone, so a malicious query is rejected rather than allowed to allocate.
The trivial count has a long list of preconditions, and each one is a potential bug if a new storage engine or a new setting breaks the assumption. The exclusions are conservative, but they are spread over three functions and two files.
Metadata counts can be stale. totalRows() is the count of parts in the Active state at the time of the query; a concurrent merge or an ongoing transaction makes it approximate relative to a full scan. The code handles the transactional case by asking the storage for a snapshot-aware count.

The one thing to remember: the plan you get is not the query you wrote. Every stage between the parser and the planner is a chance to do less work — and the biggest wins (column pruning, trivial count) happen before the plan is built, when the shape of the data is still cheap to change.

The part people get wrong: SELECT count() FROM t with a WHERE clause is not the trivial count — the metadata only knows the row count, not how many rows pass an arbitrary predicate. It is answered from the metadata only when the predicate is one the storage can evaluate from its own statistics (the sparsity filter), or when there is no predicate at all.

What to go do about it: run EXPLAIN PLAN and EXPLAIN PIPELINE on a query and compare them with the query you wrote. The plan shows what will actually run. When it differs from the query in an interesting way — a projection with fewer columns, an Optimized trivial count step, a read that is not a full scan — the difference is one of the rewrites above.

ClickHouse Pushes ORDER BY into the Storage Read

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: A MergeTree table’s ORDER BY defines its sorting key, and the rows inside every part are physically stored in that order. So SELECT ... ORDER BY k LIMIT 10 does not need to sort the table: it needs the first 10 rows in key order. ClickHouse matches the query’s ORDER BY expression against the sorting key — using the monotonicity of the functions in between — and hands the storage an InputOrderInfo saying “read the first N key columns in this direction, stop after this many rows”. The sort step then becomes a partial sort of an already-sorted stream.

Consider the archetypal analytical query:

 sql
 
SELECT * FROM events ORDER BY event_time LIMIT 10;

The obvious plan reads every row of events, sorts them all by event_time, and takes the first ten. On a table with a billion rows that is a billion rows of I/O to produce ten.

But events is a MergeTree table, and MergeTree tables have a sorting key — the table’s own ORDER BY clause. The rows inside each part are physically stored in that order. If the sorting key is (event_time), then:

reading part 1 from its beginning gives you the earliest rows of part 1;
reading part 2 from its beginning gives you the earliest rows of part 2;
a k-way merge of those gives you the global earliest rows;
and if you only need 10 rows, you can stop as soon as you have 10.

That is the optimisation ClickHouse calls read in order, and it is on by default:

 cpp
 
 
// src/Core/Settings.cpp:5119
 DECLARE(Bool, optimize_read_in_order, true, R"(
Enables [ORDER BY](/reference/statements/select/order-by#optimization-of-data-reading) optimization in [SELECT](/reference/statements/select/index) queries for reading data from [MergeTree](/reference/engines/table-engines/mergetree-family/mergetree) tables.

Possible values:

- 0 — `ORDER BY` optimization is disabled.
- 1 — `ORDER BY` optimization is enabled.

**See Also**

- [ORDER BY Clause](/reference/statements/select/order-by#optimization-of-data-reading)
)", 0) \

The hard part is not the reading. It is deciding whether the query’s ORDER BY can be served by the sorting key, which is a question about expressions, not about columns.

The contract: `InputOrderInfo`

The agreement between the plan optimiser and the storage is a small struct:

 cpp
 
 
// src/Storages/SelectQueryInfo.h:84
struct InputOrderInfo
{
 /// Sort description for merging of already sorted streams.
 /// Always a prefix of ORDER BY or GROUP BY description specified in query.
 SortDescription sort_description_for_merging;

 /** Size of prefix of sorting key that is already
 * sorted before execution of sorting or aggreagation.
 *
 * Contains both columns that scpecified in
 * ORDER BY or GROUP BY clause of query
 * and columns that turned out to be already sorted.
 *
 * E.g. if we have sorting key ORDER BY (a, b, c, d)
 * and query with `WHERE a = 'x' AND b = 'y' ORDER BY c, d` clauses.
 * sort_description_for_merging will be equal to (c, d) and
 * used_prefix_of_sorting_key_size will be equal to 4.
 */
 const size_t used_prefix_of_sorting_key_size;

 const int direction;
 const UInt64 limit;

 InputOrderInfo(
 const SortDescription & sort_description_for_merging_,
 size_t used_prefix_of_sorting_key_size_,
 int direction_, UInt64 limit_)
 : sort_description_for_merging(sort_description_for_merging_)
 , used_prefix_of_sorting_key_size(used_prefix_of_sorting_key_size_)
 , direction(direction_), limit(limit_)
 {
 }

 bool operator==(const InputOrderInfo &) const = default;
};

Four things:

sort_description_for_merging — the sort columns the consumer of the stream needs.
used_prefix_of_sorting_key_size — how many columns of the table’s sorting key are physically ordered in the stream.
direction — 1 for ascending, -1 for descending, 0 for “either”.
limit — how many rows the consumer will take, so the read can stop early.

The used_prefix_of_sorting_key_size is the crucial one, and the doc comment gives the example that makes it click: with sorting key (a, b, c, d) and a query WHERE a = 'x' AND b = 'y' ORDER BY c, d, the merging description is (c, d) — because a and b are constant — but the prefix size is 4, because the stream is physically ordered by all four columns.

Matching the ORDER BY against the sorting key

The matching lives in optimizeReadInOrder.cpp, in the function buildInputOrderFromSortDescription. It walks the query’s sort description and the table’s sorting key in parallel, and for each pair decides whether the sort expression is equivalent to the key expression, possibly up to a monotonic transformation.

The first thing it does is build a tree match between the sorting key’s expression DAG and the DAG of the expressions that produced the sort columns:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:599
 if (dag)
 {
 matches = matchTrees(sorting_key_dag.getOutputs(), *dag);

 for (const auto & [node, match] : matches)
 {
 //std::cerr << "------- matching " << static_cast<const void *>(node) << " " << node->result_name
 // << " to " << static_cast<const void *>(match.node) << " " << (match.node ? match.node->result_name : "") << std::endl;
 if (!match.monotonicity || match.monotonicity->strict)
 {
 if (match.node && fixed_columns.contains(node))
 fixed_key_columns.insert(match.node);
 }
 }

 enrichFixedColumns(sorting_key_dag, fixed_key_columns);
 }

matchTrees is a structural matcher between two ActionsDAGs: it walks the outer DAG and, for each node, looks for a node in the inner DAG that computes the same thing. The interesting part is the monotonicity field of a match — it records how the outer expression relates to the inner one:

 cpp
 
 
// src/Functions/IFunction.h:410
 struct Monotonicity
 {
 bool is_monotonic = false; /// Is the function monotonous (non-decreasing or non-increasing).
 bool is_positive = true; /// true if the function is non-decreasing, false if non-increasing. If is_monotonic = false, then it does not matter.
 bool is_always_monotonic = false; /// Is true if function is monotonic on the whole input range I
 bool is_strict = false; /// true if the function is strictly decreasing or increasing.
 /// Is true if the function is monotonic over the whole subset of the input range on which it
 /// is defined, while the rest of the range either throws an exception or wraps around (so it
 /// is weaker than is_always_monotonic, which requires the whole range to be mapped).
 /// It is enough to push a comparison constant through a sorting or partition key expression:
 /// a constant outside of that subset is rejected by the guards in `applyFunctionChainToColumn`.
 /// It is not enough to map a key range through the function, which needs `is_monotonic`.
 bool is_always_monotonic_where_defined = false;
 };

This is what makes ORDER BY negate(x) work on an ascending key: negate is monotonically decreasing, so the ascending key can be read backwards to produce a descending stream. The matching loop applies the monotonicity’s direction to the read direction:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:739
 current_direction = sort_column_description.direction * reverse_indicator;
 if (match.monotonicity)
 {
 current_direction *= match.monotonicity->direction;
 strict_monotonic = match.monotonicity->strict;
 match_infos.push_back({.source = sort_node, .monotonic = &match});
 }
 else
 match_infos.push_back({.source = sort_node});

current_direction is the direction the key column is physically read in, after applying both the requested sort direction and the monotonicity of the matching expression. reverse_indicator accounts for a key column declared DESC in the table’s ORDER BY.

Once the directions disagree, the match stops:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:810
 /// read_direction == 0 means we can choose any global direction.
 /// current_direction == 0 means current key if fixed and any direction is possible for it.
 if (current_direction && read_direction && current_direction != read_direction)
 break;

 if (read_direction == 0)
 read_direction = current_direction;

 if (current_direction)
 order_key_prefix_descr.push_back(sort_column_description);

 if (current_direction && !strict_monotonic)
 break;

The last line is the reason the loop stops at the first non-strictly monotonic match: if the function is monotonic but not strictly so (e.g. it maps several input values to the same output), then the stream is sorted by the output, but the output is not a bijection of the input, so the physical order of the next key column is not preserved. The stream is still sorted by the prefix, but not by anything beyond it.

Fixed columns: `WHERE x = 42`

A column that the WHERE clause has pinned to a constant is a special case: it is in the sorting key but carries no information. The matcher skips it without giving up the direction:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:716
 /// If the ORDER BY column is fixed (constant due to WHERE clause),
 /// don't let it determine read direction - skip to next columns.
 /// Example: ORDER BY tenant, event_time DESC with WHERE tenant='42'
 /// tenant is constant, so read direction should come from event_time DESC.
 if (fixed_columns.contains(sort_node))
 {
 /// Virtual row for fixed column from order by is not supported now.
 can_optimize_virtual_row = false;

 /// Still add the fixed column to the sort description (required for sort matching)
 /// but don't set current_direction so it won't influence read_direction.
 match_infos.push_back({.source = sort_node, .fixed_column = sort_node});
 order_key_prefix_descr.push_back(sort_column_description);
 ++next_description_column;
 ++next_sort_key;
 continue;
 }

NULLs and NaNs: the physical end

The subtlest constraint is where NULLs and NaNs end up. A MergeTree part stores the NULLs of a Nullable key column and the NaNs of a Float key column at one physical end of the part — the forward end for an ordinary key column, the other end for one declared DESC. Reading the part cannot move them. So whether the requested order is satisfiable depends on the direction the key is read in:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:787
 /** The NULLs and NaNs of this key column come out at the end of the stream when the part is read
 * in the direction that has them last, and at its beginning otherwise; `nulls_direction` asks for
 * them last exactly when it agrees with `direction` (`ASC NULLS LAST`, `DESC NULLS FIRST`). A
 * mismatch cannot be repaired downstream - the sort is elided, which is the point of this
 * optimization - so the match ends here.
 *
 * `current_direction` is the direction the key is read in, with the monotonicity of the match
 * already applied, and `reverse_indicator` says which physical end holds the NULLs and NaNs.
 */
 if (column_is_nullable && current_direction != 0)
 {
 const bool produces_nulls_last = current_direction * reverse_indicator == 1;
 const bool requests_nulls_last = sort_column_description.direction == sort_column_description.nulls_direction;
 if (produces_nulls_last != requests_nulls_last)
 {
 /// `next_description_column` is not read after the loop, so it is left alone.
 next_sort_key = prev_sort_key;
 match_infos.resize(prev_match_infos_size);
 order_key_prefix_descr.resize(prev_order_key_prefix_size);
 break;
 }
 }

ORDER BY negate(x) NULLS FIRST over an ascending x is the case the comment describes: the backward read surfaces the NaNs first, but NULLS FIRST wants them first — that one matches. The opposite combination does not, and the match is rolled back to the previous key.

Note the rollback: the code explicitly restores next_sort_key, match_infos and order_key_prefix_descr to their previous values, because next_sort_key becomes used_prefix_of_sorting_key_size — the width the storage reads with and applies a hard LIMIT to:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:645
 /// The NULL/NaN placement check below can only run after this iteration consumed the key, so
 /// remember how much was consumed and undo it when the check rejects the key. Otherwise the
 /// rejected key would still widen `next_sort_key`, which becomes
 /// `InputOrderInfo::used_prefix_of_sorting_key_size` - the width the storage actually reads with
 /// and applies a hard `LIMIT` to - and the read would stop on a row the advertised prefix does
 /// not order.
 const size_t prev_sort_key = next_sort_key;
 const size_t prev_match_infos_size = match_infos.size();
 const size_t prev_order_key_prefix_size = order_key_prefix_descr.size();

That is a bug class that the comment is guarding against, written down in the code that could cause it.

The result

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:844
 auto order_info = std::make_shared<InputOrderInfo>(order_key_prefix_descr, next_sort_key, read_direction, limit);

and the plan asks the storage whether it can honour it:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:1408
 bool can_read = reading->requestReadingInOrder(
 order_info.input_order->used_prefix_of_sorting_key_size,
 order_info.input_order->direction,
 order_info.input_order->limit,
 query_limit);

 if (!can_read)
 return nullptr;

The storage has the last word:

 cpp
 
 
// src/Processors/QueryPlan/ReadFromMergeTree.cpp:4116
bool ReadFromMergeTree::requestReadingInOrder(size_t prefix_size, int direction, size_t read_limit, size_t query_limit)
{
 /// if direction is not set, use current one
 if (!direction)
 direction = getSortDirection();

 if (direction != 1 && !canReadInReverseOrder())
 return false;

 /// The prefix indexes this snapshot's sorting key, and a clone of its expression list is resized
 /// to `prefix_size`, which appends null `ASTPtr` children when the prefix is longer than the key.
 if (prefix_size > storage_snapshot->metadata->getSortingKey().column_names.size())
 return false;

The read

Once the storage agrees, the read itself changes shape. The parts are read in sorting-key order, and inside each part the rows are read in key order too — ReadType::InOrder or ReadType::InReverseOrder:

 cpp
 
 
// src/Processors/QueryPlan/ReadFromMergeTree.cpp:1356
 sort_description.reserve(input_order_info->used_prefix_of_sorting_key_size);
 for (size_t i = 0; i < input_order_info->used_prefix_of_sorting_key_size; ++i)
 {
 if (!reverse_flags.empty() && reverse_flags[i])
 sort_description.emplace_back(sorting_columns[i], input_order_info->direction * -1);
 else
 sort_description.emplace_back(sorting_columns[i], input_order_info->direction);
 }

 ReadType in_order_read_type = input_order_info->direction > 0 ? ReadType::InOrder : ReadType::InReverseOrder;

The sorting key’s expression is applied to the read stream, because the sorting key may be an expression (ORDER BY (a + b)), not a plain column:

 cpp
 
 
// src/Processors/QueryPlan/ReadFromMergeTree.cpp:1388
 pipe.addSimpleTransform([sorting_expr](const SharedHeader & header)
 {
 return std::make_shared<ExpressionTransform>(header, sorting_expr);
 });

and if there is more than one stream, the streams are merged with a MergingSortedTransform — a k-way merge of already-sorted streams, not a sort:

 cpp
 
 
// src/Processors/QueryPlan/ReadFromMergeTree.cpp:1393
 if (pipe.numOutputPorts() != 1)
 {
 auto transform = std::make_shared<MergingSortedTransform>(
 pipe.getSharedHeader(),
 pipe.numOutputPorts(),
 sort_description,
 block_size.max_block_size_rows,
 /*max_block_size_bytes=*/ 0,

The sort step becomes a partial sort

The other half of the optimisation is on the consumer side. The SortingStep in the plan was a full sort; now that the input is sorted by a prefix, it becomes a FinishSorting:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:1975
 else if (auto order_info = buildInputOrderInfo(*sorting, apply_virtual_row, virtual_row_reader, *node.children.front(), optimization_settings))
 {
 /// Use buffering only if have filter or don't have limit.
 bool use_buffering = order_info->limit == 0;
 sorting->convertToFinishSorting(order_info->sort_description_for_merging, use_buffering, apply_virtual_row);
 }

and convertToFinishSorting just changes the step’s type:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:320
void SortingStep::convertToFinishSorting(SortDescription prefix_description_, bool use_buffering_, bool apply_virtual_row_conversions_)
{
 type = Type::FinishSorting;
 prefix_description = std::move(prefix_description_);
 use_buffering = use_buffering_;
 apply_virtual_row_conversions = apply_virtual_row_conversions_;
}

The pipeline first merges the already-sorted streams by the prefix description, and then, only if the prefix is shorter than the full sort description, installs a partial sort plus a finish sort:

 cpp
 
 
// src/Processors/QueryPlan/SortingStep.cpp:381
void SortingStep::finishSorting(
 QueryPipelineBuilder & pipeline, const SortDescription & input_sort_desc, const SortDescription & result_sort_desc, const UInt64 limit_)
{
 pipeline.addSimpleTransform(
 [&](const SharedHeader & header, QueryPipelineBuilder::StreamType stream_type) -> ProcessorPtr
 {
 if (stream_type != QueryPipelineBuilder::StreamType::Main)
 return nullptr;

 return std::make_shared<PartialSortingTransform>(header, result_sort_desc, limit_, threshold_tracker);
 });

PartialSortingTransform sorts each block but only keeps the rows that could still be in the top-limit (using a threshold tracker), and FinishSortingTransform then completes the ordering of the remaining sort columns within the already-sorted prefix:

 cpp
 
 
// src/Processors/Transforms/FinishSortingTransform.cpp:32
 /// Check for sanity non-modified descriptions
 if (!isPrefix(description_sorted_, description_to_sort_))
 throw Exception(ErrorCodes::LOGICAL_ERROR,
 "Can't finish sorting. SortDescription "
 "of already sorted stream is not prefix of SortDescription needed to sort");

The isPrefix check is the safety net: the finish-sort transform requires that the input is already sorted by a prefix of the requested description, and throws if it is not. This is the last line of defence for the whole optimisation.

The limit is pushed into the read

The limit field of InputOrderInfo is what turns “read in order” into “read only what you need”. The read knows how many rows the consumer wants:

 cpp
 
 
// src/Processors/QueryPlan/ReadFromMergeTree.cpp:1039
 const UInt64 in_order_limit = query_info.input_order_info ? query_info.input_order_info->limit : 0;

and when distributing the marks of the parts among the reading streams, a part that is smaller than the limit is taken whole, while the read stops as soon as the limit is reached:

 cpp
 
 
// src/Processors/QueryPlan/ReadFromMergeTree.cpp:2156
 /// We take full part if it contains enough marks or
 /// if we know limit and part contains less than 'limit' rows.
 bool take_full_part = marks_in_part <= need_marks || (input_order_info->limit && input_order_info->limit < part.getRowsCount());

This is where the 100× difference between “sort a billion rows” and “read ten rows” actually comes from: not fewer bytes sorted, but not reading the parts that cannot contain the answer.

The picture

 text
 
 
 plan before plan after
 ---------- -----------
 ReadFromMergeTree ReadFromMergeTree
 (parts in arbitrary order, (parts in sorting-key order,
 rows in arbitrary order) rows in key order, stops at LIMIT)
 | |
 SortingStep (FULL) SortingStep (FINISH)
 (sorts everything) (partial sort + finish sort)
 | |
 Limit Limit

 cost: O(N log N), reads N rows cost: O(K log P), reads ~K rows
 where K = LIMIT, P = parts

The same machinery, four ways

optimizeReadInOrder is not the only user of this machinery. The same buildInputOrderInfo is reused for four different consumers, each of which has a different idea of what “in order” means:

SortingStep — the ORDER BY case above.
AggregatingStep — optimizeAggregationInOrder: if the data is already sorted by the grouping key, the aggregation can be done in a single pass over the stream instead of a hash table.
DistinctStep — optimizeDistinctInOrder: DISTINCT on sorted data is a “compare with the previous row” operation, not a hash set.
LimitByStep — optimizeLimitByInOrder: LIMIT n BY k on sorted data is a “take the first n rows of each k group” operation.

The aggregation one has an extra requirement — the sort description must cover the grouping keys:

 cpp
 
 
// src/Processors/QueryPlan/Optimizations/optimizeReadInOrder.cpp:2050
 /// The sorted-stream transform needs every key in the sort prefix (and in that order); otherwise a
 /// key not covered by the prefix would be dropped from grouping.
 auto sort_prefix
 = getCollationAwareSortPrefixInColumns(order_info.sort_description, limit_by->getColumns(), *limit_by->getInputHeaders().front());
 if (sort_prefix.size() != limit_by->getColumns().size())
 return;

Trade-offs

What the optimisation buys:

ORDER BY ... LIMIT n becomes O(K log P) instead of O(N log N) where K is the limit and P the number of parts. The parts that cannot contain the answer are never opened.
The sort step becomes a partial sort. The consumer no longer sorts the whole table; it sorts the top-K of each block and merges.
It is not a special case for MergeTree. The same machinery drives aggregation-in-order, distinct-in-order and limit-by-in-order, and works for Merge, ObjectStorage and distributed plans.

What it costs:

The matching logic is genuinely hard. optimizeReadInOrder.cpp is 2060 lines, and the direction/NULL/NaN/monotonicity interactions are the reason. Every one of them is a potential wrong answer if it is got wrong, because the sort is elided — there is no downstream check.
The safety net is a throw. FinishSortingTransform throws a LOGICAL_ERROR if the prefix assumption is violated. That is the right failure mode (loud, not silent) but it means a bug in the matcher is a server error, not a wrong answer.
The read has to be ordered, which constrains the read strategy. Parts must be read in key order, which interacts with parallel replicas, with FINAL (which may need a reverse read), and with the decision of how to spread the marks of the parts among the reading threads.
Not every ORDER BY qualifies. ORDER BY on a non-key column, on a collation, on a non-monotonic expression, or with a conflicting direction simply does not match, and the query falls back to a full sort. The matching is precise, not heuristic — which is why it is worth the complexity, and also why it does not fire as often as you might hope.

The one thing to remember: the sorting key is a physical property of the data, and the optimisation is about discovering that the query’s sort order is already implied by it. The InputOrderInfo is the handshake: “I promise the stream is ordered by these N key columns in this direction, and I will take at most L rows”.

The part people get wrong: the LIMIT is what makes this optimisation worthwhile, and it is easy to lose. SELECT * FROM events ORDER BY event_time (without a limit) still has to read every row, and the benefit is only that the rows arrive already sorted — which saves the sort, but not the read. The dramatic case is ORDER BY ... LIMIT, where the read itself is truncated.

What to go do about it: compare EXPLAIN PIPELINE for the same query with optimize_read_in_order = 1 and = 0. If the plans differ in the ReadFromMergeTree step’s description or in the sort step’s type, the optimisation fired. And check that your table’s sorting key starts with the column you are ordering by — a sorting key of (tenant_id, event_time) will not help ORDER BY event_time, but a sorting key of (event_time, tenant_id) will.

ClickHouse Builds an Actions DAG Once and Runs It on Every Block

Jay Hubbert — Fri, 23 Oct 2026 09:00:00 +0000

TL;DR: A SELECT in ClickHouse is not a loop over rows, it is a graph of actions. That graph — the ActionsDAG — is built once, flattened by a topological sort into a std::vector<Action> where every argument and every result has a fixed integer slot, and then executed once per 65 409-row block by simple array indexing. No per-row dispatch, no tree walking, no intermediate hash maps.

Every columnar database ends up with the same internal representation for a SELECT: a tree of expressions. The obvious way to run that tree is to walk it per row. SELECT a + b * c FROM t becomes, per row, a recursive descent through an expression tree, with a virtual call at every node.

ClickHouse never does that. It has three layers of insulation between the parsed query and the raw bytes:

the query is compiled into a graph of actions (ActionsDAG),
the graph is topologically sorted once into a flat list of actions with pre-assigned slot numbers,
the list is replayed per block, and every intermediate result is a ColumnPtr in a fixed-size array indexed by slot number.

The entire design goal of ActionsDAG is stated in its own header comment:

 cpp
 
 
// src/Interpreters/ActionsDAG.h:60
/// Directed acyclic graph of expressions.
/// This is an intermediate representation of actions which is usually built from expression list AST.
/// Node of DAG describe calculation of a single column with known type, name, and constant value (if applicable).
///
/// DAG representation is useful in case we need to know explicit dependencies between actions.
/// It is helpful when it is needed to optimize actions, remove unused expressions, compile subexpressions,
/// split or merge parts of graph, calculate expressions on partial input.
///
/// Built DAG is used by ExpressionActions, which calculates expressions on block.

Read that last line again: built DAG is used by ExpressionActions, which calculates expressions on block. The DAG is the plan; ExpressionActions is the interpreter that runs it; the interpreter is run once per block.

The node types

A DAG node is “the calculation of a single column with known type, name, and constant value (if applicable)”. There are exactly six kinds:

 cpp
 
 
// src/Interpreters/ActionsDAG.h:73
 enum class ActionType : uint8_t
 {
 /// Column which must be in input.
 INPUT,
 /// Constant column with known value.
 COLUMN,
 /// Another one name for column.
 ALIAS,
 /// Function arrayJoin. Specially separated because it changes the number of rows.
 ARRAY_JOIN,
 FUNCTION,
 /// Placeholder node for correlated column
 PLACEHOLDER,
 };

Two of these are worth pausing on.

ARRAY_JOIN is separated out because it changes the number of rows. Every other action is a pure column-to-column map; arrayJoin is the one operation that can grow the block. That single bit of metadata is what lets the planner split a DAG at an arrayJoin boundary, and what lets removeUnusedActions keep an arrayJoin alive even when nothing downstream reads its result:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:1030
 /// We cannot remove arrayJoin because it changes the number of rows.
 if (node.type == ActionType::ARRAY_JOIN)
 roots.push_back(&node);

PLACEHOLDER is the mark of a correlated column — a reference from a subquery to a column of an outer query. It is a node whose value is not known at plan time. It gets rewritten into an INPUT by decorrelate() once the correlated subquery is turned into a join:

 cpp
 
 
// src/Interpreters/ActionsDAG.h:412
 /// Replace all PLACEHOLDER nodes with INPUT nodes
 /// Not noexcept: appends to the inputs vector, which can throw on allocation.
 void decorrelate();

A node carries its children as raw pointers, its result name and type, and — for functions — both the overload (the IFunctionBase, which knows the signature and the monotonicity) and the prepared instance (the IExecutableFunction, which is what actually gets called at run time):

 cpp
 
 
// src/Interpreters/ActionsDAG.h:92
 struct Node
 {
 NodeRawConstPtrs children;

 ActionType type{};

 std::string result_name;
 DataTypePtr result_type;

 /// Can be used to get function signature or properties like monotonicity.
 FunctionBasePtr function_base;
 /// Prepared function which is used in function execution.
 ExecutableFunctionPtr function;
 /// If function is a compiled statement.
 bool is_function_compiled = false;

The split between function_base and function matters more than it looks. function_base is the overload — it is shared, immutable, and knows things like “am I monotonic?” and “can I be constant-folded?”. function is the prepared instance, built from the concrete argument types and columns, and it is what executeAction calls.

Why `std::list`

The nodes are stored in a std::list, and the header is explicit that this is an implementation detail — but the reason is worth stating, because it explains the whole pointer-based style of the class:

 cpp
 
 
// src/Interpreters/ActionsDAG.h:140
 /// NOTE: std::list is an implementation detail.
 /// It allows to add and remove new nodes inplace without reallocation.
 /// Raw pointers to nodes remain valid.
 using Nodes = std::list<Node>;

Every optimisation in the system — pruning unused columns, splitting the DAG in half to push a filter into the storage, rewriting a subquery’s projection — hands out const Node * pointers into this list and expects them to stay valid. A std::vector<Node> would invalidate every pointer on every push_back. std::list gives O(1) insertion, O(1) erasure, and stable addresses.

Adding a node is therefore trivial:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:546
ActionsDAG::Node & ActionsDAG::addNode(Node node)
{
 auto & res = nodes.emplace_back(std::move(node));
 if (res.type == ActionType::INPUT)
 inputs.emplace_back(&res);

 return res;
}

The DAG keeps three vectors of node pointers: nodes (all of them, in insertion order), inputs (the ones that must be present in the input block), and outputs (the ones that are the result).

Constant folding happens at plan time, not at run time

Before a function node is added, ClickHouse tries to evaluate it right now, at plan time, if all its arguments are constants:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:122
void tryFoldFunctionToConstant(
 ActionsDAG::Node & node,
 const ColumnsWithTypeAndName & arguments,
 bool all_const,
 bool best_effort)
{
 /// If all arguments are constants, and function is suitable to be executed in 'prepare' stage - execute function.
 if (!node.function_base->isSuitableForConstantFolding())
 return;

 ColumnPtr column;
 try
 {
 if (all_const)
 {
 size_t num_rows = arguments.empty() ? 0 : arguments.front().column->size();
 column = node.function->execute(arguments, node.result_type, num_rows, true);
 }
 else
 {
 column = node.function_base->getConstantResultForNonConstArguments(arguments, node.result_type);
 }
 }
 catch (const Exception &)
 {
 if (!best_effort)
 throw;
 return;
 }

and the folded value is stored in the node:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:175
 /// If the result is not a constant, just in case, we will consider the result as unknown.
 if (const auto * column_const = column ? typeid_cast<const ColumnConst *>(column.get()) : nullptr)
 {
 /// Functions may produce a ColumnConst sized to match the input block; in DAG nodes
 /// the size is meaningless and we keep them at zero.
 if (!column_const->empty())
 node.column = ColumnConst::create(column_const->getDataColumnPtr(), 0);
 else
 node.column = column_const->getPtr();
 }

Note the “size is meaningless, keep them at zero” detail: a ColumnConst in a DAG node carries a value, not a row count. The row count is recomputed at execution. That is why SELECT 1 + 1 costs nothing at run time — the whole subtree collapses into one COLUMN node.

The fold is deliberately conservative. Only functions that declare themselves foldable are folded, and the second stage — folding a predicate whose arguments are constants — is restricted to a short list of value-only operators:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:1218
const std::unordered_set<std::string> & foldablePredicateFunctions()
{
 static const std::unordered_set<std::string> functions{
 "equals", "notEquals", "less", "greater", "lessOrEquals", "greaterOrEquals", "and", "or", "not",
 /// `xor` never short-circuits and reads its arguments by value only, the same shape as `not`
 "xor",
 /// `isNull` / `isNotNull` only look at the null map of their argument, so a `ColumnConst`
 /// and the materialized column it wraps give the same answer
 "isNull", "isNotNull"};
 return functions;
}

From graph to flat list: the topological sort

This is the heart of the whole design. ExpressionActions::linearizeActions takes the graph and produces a std::vector<Action> in which:

every action has a result slot — an integer index into a fixed-size array of columns;
every argument is a slot number too;
the order is a topological order, so an action’s arguments are always already computed when it runs.

The algorithm is a Kahn-style topological sort, but with one twist: it does not use a priority queue on “number of children”, it uses two FIFO queues — one for ordinary actions and one for arrayJoin — so that arrayJoins are deliberately delayed as long as possible:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:557
 /// There are independent queues for arrayJoin and other actions.
 /// We delay creation of arrayJoin as long as we can, so that they will be executed closer to end.
 std::queue<const Node *> ready_nodes;
 std::queue<const Node *> ready_array_joins;

The slot allocation is a stack of freed positions:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:568
 /// Every argument will have fixed position in columns list.
 /// If argument is removed, it's position may be reused by other action.
 std::stack<size_t> free_positions;

and each action claims a slot, freeing the slots of arguments that nobody else needs:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:582
 /// Select position for action result.
 size_t free_position = num_columns;
 if (free_positions.empty())
 ++num_columns;
 else
 {
 free_position = free_positions.top();
 free_positions.pop();
 }

The interesting bit is the “is this argument still needed?” decision, which is a reference count over the DAG’s reverse edges:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:605
 ++arg.num_created_parents;

 ExpressionActions::Argument argument;
 argument.pos = arg.position;
 argument.needed_later = arg_info.used_in_result || arg.num_created_parents != arg_info.parents.size();

 if (!argument.needed_later)
 free_positions.push(argument.pos);

needed_later == false means “this is the last consumer of that slot”, so the slot can be recycled by the next action. This is a register allocator for columns. A DAG with 50 nodes but only 4 live values at any point runs with a 4-element scratch array, not a 50-element one.

Executing the plan

The execution model is spelled out in the comment above ExecutionContext:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:798
 /// This struct stores context needed to execute actions.
 ///
 /// Execution model is following:
 /// * execution is performed over list of columns (with fixed size = ExpressionActions::num_columns)
 /// * every argument has fixed position in columns list, every action has fixed position for result
 /// * if argument is not needed anymore (Argument::needed_later == false), it is removed from list
 /// * argument for INPUT is in inputs[inputs_pos[argument.pos]]
 ///
 /// Columns on positions `ExpressionActions::result_positions` are inserted back into block.

So the “intermediate results” are ColumnPtrs in a std::vector of fixed size. The action list is the program; the column vector is the register file. There is no map and no name lookup between the actions — an argument is addressed by its integer slot, and the only hash lookup in the whole execution is the one that binds the input block’s columns to the input slots, once per block.

Running one action is a switch on the node type. For a function:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:868
 ColumnsWithTypeAndName arguments(action.arguments.size());
 for (size_t i = 0; i < arguments.size(); ++i)
 {
 if (!action.arguments[i].needed_later)
 arguments[i] = std::move(columns[action.arguments[i].pos]);
 else
 arguments[i] = columns[action.arguments[i].pos];
 }

Note the std::move: if this is the last use of the argument, the ColumnPtr is moved out of the register file, which releases a reference count. If the argument is needed later, it is copied — a refcount bump, not a data copy. The data itself is never copied; ColumnPtr is a COW handle to a column that was already materialised somewhere upstream.

The actual call is one line:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:894
 res_column.column = action.node->function->execute(arguments, res_column.type, num_rows, dry_run);

function is the prepared IExecutableFunction stored on the node. There is no virtual dispatch per row — there is exactly one virtual call per block per node, and inside it the function loops over the column itself, with SIMD-friendly typed accessors.

Constants skip the call entirely:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:855
 if (action.node->column)
 {
 /// Do not execute function if it's result is already known.
 res_column.column = action.node->column->cloneResized(num_rows);
 /// But still need to remove unused arguments.
 for (const auto & argument : action.arguments)
 {
 if (!argument.needed_later)
 columns[argument.pos] = {};
 }
 break;
 }

cloneResized(num_rows) materialises the constant to the block’s row count — a memset, not a computation.

The plan is built once

The ExpressionActions object is constructed once, at plan time, and then executed over and over. The constructor is where the whole plan-time work happens: constant folding has already happened in the DAG; here the DAG is sorted, the short-circuit analysis runs, and (if enabled) the JIT replaces hot function nodes with compiled ones:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:61
ExpressionActions::ExpressionActions(ActionsDAG actions_dag_, const ExpressionActionsSettings & settings_, bool project_inputs_)
 : actions_dag(std::move(actions_dag_))
 , project_inputs(project_inputs_)
 , settings(settings_)
{
 /// It's important to determine lazy executed nodes before compiling expressions.
 std::unordered_map<const ActionsDAG::Node *, size_t> short_circuit_eager_arguments;
 std::unordered_set<const ActionsDAG::Node *> lazy_executed_nodes = processShortCircuitFunctions(
 actions_dag,
 settings.short_circuit_function_evaluation,
 settings.short_circuit_function_evaluation_reorder_arguments,
 short_circuit_eager_arguments);

#if USE_EMBEDDED_COMPILER
 if (settings.can_compile_expressions && settings.compile_expressions == CompileExpressions::yes)
 actions_dag.compileExpressions(settings.min_count_to_compile_expression, lazy_executed_nodes);
#endif

At run time, ExpressionActions::execute is a loop over the action vector:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:1032
 execution_context.columns.resize(num_columns);

 for (const auto & action : actions)
 {
 try
 {
 executeAction(action, execution_context, dry_run, allow_duplicates_in_input, settings.enable_lazy_columns_replication);
 checkLimits(execution_context.columns);
 }
 catch (Exception & e)
 {
 e.addMessage(fmt::format("while executing '{}'", action.toString()));
 throw;
 }

The modern pipeline does not even build a Block per chunk. ExpressionTransform uses a positional fast path where the input column positions are resolved once, at construction:

 cpp
 
 
// src/Processors/Transforms/ExpressionTransform.cpp:18
ExpressionTransform::ExpressionTransform(
 SharedHeader header_, ExpressionActionsPtr expression_, RuntimeDataflowStatisticsCacheUpdaterPtr updater_)
 : ISimpleTransform(header_, std::make_shared<const Block>(transformHeader(*header_, expression_->getActionsDAG())), false)
 , expression(std::move(expression_))
 , input_positions(expression->getInputPositions(*header_))
 , updater(std::move(updater_))
{
 set_spillables.add(expression->getActionsDAG());
}

and then per chunk:

 cpp
 
 
// src/Processors/Transforms/ExpressionTransform.cpp:66
 /// Fast path: run positionally against the fixed input header, avoiding per-chunk Block name-index work.
 auto columns = expression->executeOnColumns(
 chunk.detachColumns(), getInputPort().getHeader(), input_positions, num_rows, false, &getCancellationFlag());

The comment on getInputPositions in the header explains exactly what was eliminated:

 cpp
 
 
// src/Interpreters/ExpressionActions.h:126
 /// Positional execution for callers whose input structure is fixed (e.g. ExpressionTransform).
 ///
 /// `getInputPositions(header)` precomputes, once, the mapping (required input slot -> position in
 /// `header`); the result is passed to `executeOnColumns` on every chunk. This avoids the per-chunk
 /// name lookups and the construction of a `Block` name index (`index_by_name`) on both the input and
 /// the output, which is significant when the header has very many columns.

This is the whole point of the design in one sentence: the query is analysed once; the data is only ever touched through integer indices.

The DAG is cloned per shard

Because the plan is a value-like object with stable node pointers, it can be copied cheaply and safely. ActionsDAG::clone copies the node list and then repairs the child pointers through a map:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:2453
ActionsDAG ActionsDAG::clone(std::unordered_map<const Node *, const Node *> & old_to_new_nodes) const
{
 ActionsDAG actions;

 for (const auto & node : nodes)
 {
 auto & copy_node = actions.nodes.emplace_back(node);
 old_to_new_nodes[&node] = &copy_node;
 }

 for (auto & node : actions.nodes)
 for (auto & child : node.children)
 child = old_to_new_nodes[child];

 for (const auto & output_node : outputs)
 actions.outputs.push_back(old_to_new_nodes[output_node]);

 for (const auto & input_node : inputs)
 actions.inputs.push_back(old_to_new_nodes[input_node]);

 return actions;
}

The same pattern exists one level up, in ExpressionActions::clone, which also has to fix up the action list, because each action holds a const Node *:

 cpp
 
 
// src/Interpreters/ExpressionActions.cpp:91
ExpressionActionsPtr ExpressionActions::clone() const
{
 auto copy = std::make_shared<ExpressionActions>(ExpressionActions());

 std::unordered_map<const Node *, const Node *> copy_map;
 copy->actions_dag = actions_dag.clone(copy_map);
 copy->actions = actions;
 for (auto & action : copy->actions)
 action.node = copy_map[action.node];

Why clone at all? Because a distributed query sends the same plan to N shards, and because a plan can be executed with different settings (a FINAL variant, a count-only variant, a PREWHERE variant). Because the nodes are immutable once built, a clone is a shallow copy of a std::list plus a pointer fix-up — microseconds, and completely independent of the data.

Pruning: the DAG is edited, not thrown away

Because the DAG is an explicit graph, “remove the columns nobody asked for” is a graph reachability problem rather than a string-matching problem. The implementation is a two-stage iterative DFS:

 cpp
 
// src/Interpreters/ActionsDAG.cpp:1049
 enum class VisitStage { NonDeterministic, Required };

 for (auto stage : {VisitStage::NonDeterministic, VisitStage::Required})

The two stages matter: the first pass classifies which surviving nodes depend on non-deterministic functions (now(), rand()), and the second decides which of them may be treated as constants. A node whose children all became constants is collapsed into a COLUMN node:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:1111
 /// Constant folding. A lambda that captures nothing has no children, but its folded value is a
 /// constant like any other, and a FUNCTION node left behind would cross plan steps as a column.
 /// An INPUT is never folded: `ExpressionActions` binds each entry of `inputs` to a required column as an INPUT.
 if (allow_constant_folding && node->column
 && (!node->children.empty()
 || (node->type == ActionType::FUNCTION && WhichDataType(node->result_type).isFunction())))
 {
 node->type = ActionsDAG::ActionType::COLUMN;
 node->children.clear();
 node->is_deterministic_constant = !non_deterministic_nodes.contains(node);
 }

and the dead nodes are erased from the list:

 cpp
 
 
// src/Interpreters/ActionsDAG.cpp:1126
 const auto erased_count = std::erase_if(nodes, [&](const Node & node) { return !required_nodes.contains(&node); });
 std::erase_if(inputs, [&](const Node * node) { return !required_nodes.contains(node); });

std::erase_if on a std::list is O(1) per element and keeps the surviving nodes’ addresses valid — which is exactly why the nodes are in a list in the first place.

The picture

 text
 
 
 parsed SQL
 |
 v
 ActionsDAG (std::list<Node>, children = Node*)
 | addInput / addColumn / addAlias / addFunction / addArrayJoin
 | constant folding at every add
 | pruning by reachability
 | splitting for filter push-down
 v
 ExpressionActions::linearizeActions()
 | topological sort
 | slot allocation (register allocator over ColumnPtr slots)
 | per-action argument lists = {slot, needed_later}
 v
 std::vector<Action> <-- the plan, built ONCE
 |
 | execute() per block (65 409 rows)
 v
 columns[num_columns] (ColumnPtr slots)
 |
 +--> columns[action.result_position] = function->execute(args)

Trade-offs

What the DAG buys:

No per-row interpretation. One virtual call per node per block, not per row. A 65 409-row block amortises the dispatch 65 409-fold.
Cheap and safe optimisation. Because dependencies are explicit, pruning is exact, filter push-down is a graph split, and constant folding is local. No string matching on column names, no re-derivation of dependencies.
A reusable, clonable plan. The plan is a value; it can be cloned per shard, per replica, per variant (FINAL, count), without re-analysing the query.
A tiny register file. The slot allocator keeps the live-column array as small as the dataflow allows.

What it costs:

Pointer-based, not index-based. The class is full of const Node *, NodeRawConstPtrs, and std::unordered_map<const Node *, ...>. Copying requires a fix-up pass; you cannot serialise a node list and reload it without a relinking step. The alternative — index-based nodes in a vector — is faster to copy but makes every mutation a reindexing problem.
The topological sort is eager. linearizeActions must see the whole DAG before a single action can run. There is no streaming/partial compilation of an expression tree.
One DAG, one shape. The DAG is built for one specific set of inputs. splitActionsForFilter, splitActionsBeforeArrayJoin and friends exist to carve it into pieces that can live in different pipeline steps, but the pieces must still be derived by explicit graph surgery, and the surgery itself has to be correct for every corner case (that is why the codebase has pages of comments about arrayJoin offsets and prewhere inputs).

The one thing to remember: the performance of a ClickHouse expression is not “how fast is the function” but “how many nodes are in the plan and how many blocks do they run over”. A plan is built once and amortised over millions of rows; a node is a virtual call per block, not per row.

The part people get wrong: the DAG is not a lazily-evaluated tree. Every node in it is a slot in a flat array and every function in it is called once per block — unless it is explicitly marked lazy, which is a different mechanism entirely (see the short-circuit article).

Query-Plan on Opensource internals

ClickHouse Rewrites the Query Before It Plans It

Stage 1: aliases are inlined

The analyzer: the alias is a cloned expression node

The AST normaliser: the older way

Stage 2: the query tree and its 47 passes

Stage 3: pruning the projection of a subquery

Stage 4: normalising the aggregates

Stage 5: the trivial count — a query that reads no data

The sparsity-filter variant

The picture

Trade-offs

ClickHouse Pushes ORDER BY into the Storage Read

The contract: InputOrderInfo

Matching the ORDER BY against the sorting key

Fixed columns: WHERE x = 42

NULLs and NaNs: the physical end

The result

The read

The sort step becomes a partial sort

The limit is pushed into the read

The picture

The same machinery, four ways

Trade-offs

ClickHouse Builds an Actions DAG Once and Runs It on Every Block

The node types

Why std::list

Constant folding happens at plan time, not at run time

From graph to flat list: the topological sort

Executing the plan

The plan is built once

The DAG is cloned per shard

Pruning: the DAG is edited, not thrown away

The picture

Trade-offs

The contract: `InputOrderInfo`

Fixed columns: `WHERE x = 42`

Why `std::list`