Ast on Opensource internals

ClickHouse Shares One AST Between Threads

Jay Hubbert — Sun, 25 Oct 2026 09:00:00 +0000

TL;DR: Every ClickHouse AST node is an IAST with a 32-byte header: a vector of children, an atomic refcount, and a 32-bit flag word. Nodes are handed around as boost::intrusive_ptr, so the same subtree can be shared by the analyzer, the formatter and the query log at the same time — and freeing a 1000-deep tree does not recurse 1000 stack frames.

Parsing is the easy half of turning text into a query. The hard half is what happens to the result.

A parsed SELECT is not a private object. In a single server the AST is touched by at least half a dozen components: the analyzer rewrites it, the planner reads it, the formatter prints it back for EXPLAIN, the query log stores a rendering of it, the distributed-query rewriter clones a piece of it for every shard, and the INSERT path looks into it for the data payload. In a cluster, the same AST is then handed to the threads that execute the query, that log it, and that report the result to the client.

So the AST has to be:

shareable — several components hold references to the same subtree without copying it;
mutable — the analyzer rewrites it in place;
small — a query with 10 000 nodes should not cost 10 000 × 100 bytes of header;
safe to destroy — a query can nest 1000 levels deep, and the default max_parser_depth is exactly 1000.

ClickHouse gets all four with an intrusive refcount, a bitfield, and a destructor that is deliberately not recursive.

The node

 cpp
 
 
// src/Parsers/IAST.h:33
/** Element of the syntax tree (hereinafter - directed acyclic graph with elements of semantics)
 */
class IAST : public TypePromotion<IAST>
{
public:
 ASTs children;
private:
 /// We implement intrusive reference counting (based on boost::intrusive_ref_counter) to avoid the padding added by the ref_counter
 /// And we use the extra bytes to store flags_storage which can be used in derived classes.
 mutable std::atomic<UInt32> ref_counter{0};
 UInt32 flags_storage = 0;

 /// Helper to detect if a type has _parent_reserved member
 template <typename T>
 static consteval bool hasParentReserved()
 {
 if constexpr (requires { T::_parent_reserved; })
 return true;
 else
 return false;
 }

public:

 /// Bit 31 of `flags_storage` is reserved for the `parenthesized` flag — see accessors below.
 /// Derived `BitfieldStruct` definitions must keep `RESERVED_BITS <= 31`.
 static constexpr UInt32 PARENTHESIZED_BIT_MASK = UInt32{1} << 31;

 /// If the element has extra parentheses around it, e.g., in "a + (b)", b has extra parentheses.
 bool isParenthesized() const { return (flags_storage & PARENTHESIZED_BIT_MASK) != 0; }
 void setParenthesized(bool value)
 {
 flags_storage = (flags_storage & ~PARENTHESIZED_BIT_MASK) | (value ? PARENTHESIZED_BIT_MASK : 0);
 }

Three members, and the layout is asserted:

 cpp
 
 
// src/Parsers/IAST.cpp:26
/// Verify that we did not increase the size of IAST by accident.
static_assert(sizeof(IAST) <= 32);

32 bytes: 24 for the ASTs vector (three pointers), 4 for the refcount, 4 for the flag word. The static_assert is a tripwire — anyone who adds a member to IAST breaks the build rather than silently making every node 8 bytes bigger.

The comment on the class says “directed acyclic graph”, not “tree”, and that is not a typo. The same node can be referenced from two places. A WITH clause that is used twice, a subquery that the rewriter duplicates, an expression that the common-subexpression eliminator keeps in two slots — the AST is a DAG, and the refcount is what keeps the shared nodes alive.

The vector type is chosen for the same reason:

 cpp
 
 
// src/Parsers/IAST_fwd.h:18
using ASTPtr = boost::intrusive_ptr<IAST>;

/// Boost requires the stored size type to be either strictly smaller than `size_t` or the very same type.
/// On 32-bit platforms `size_t` is already 32-bit, so there is nothing to shrink.
using ASTsStoredSize = std::conditional_t<(sizeof(uint32_t) < sizeof(size_t)), uint32_t, size_t>;

/// Boost vector with smaller stored size to save memory for AST children vectors.
using ASTs = boost::container::vector<
 ASTPtr,
 boost::container::new_allocator<ASTPtr>,
 boost::container::vector_options<boost::container::stored_size<ASTsStoredSize>>::type>;

A std::vector stores its size in a size_t — 8 bytes on a 64-bit build. Every AST node has a children vector, and most of them have one or two children, so the size field is 4 bytes of pure waste in most nodes. The boost::container::vector with stored_size<uint32_t> shrinks it. The std::conditional_t keeps it correct on 32-bit platforms, where size_t is already 32 bits and shrinking would be impossible.

Reference counting

The refcount is intrusive, which means IAST implements the two free functions that boost::intrusive_ptr calls:

 cpp
 
 
// src/Parsers/IAST.cpp:29
void intrusive_ptr_add_ref(const IAST * p) noexcept
{
 p->ref_counter.fetch_add(1, std::memory_order_relaxed);
}

void intrusive_ptr_release(const IAST * p) noexcept
{
 if (p->ref_counter.fetch_sub(1, std::memory_order_acq_rel) == 1)
 {
 struct LinkedList
 {
 ASTs children;
 LinkedList * next = nullptr;

 static LinkedList * create(IAST * ptr)
 {
 if (ptr->children.empty())
 {
 delete ptr;
 return nullptr;
 }

 ASTs children;
 children.swap(ptr->children);
 ptr->~IAST();
 LinkedList * elem = new (dynamic_cast<void *>(ptr)) LinkedList;
 elem->children.swap(children);
 return elem;
 }
 };

 static_assert(sizeof(LinkedList) <= sizeof(IAST));

 const IAST * const_ptr = p;
 IAST * ptr = const_cast<IAST *>(const_ptr);

 LinkedList * list_head = LinkedList::create(ptr);

 while (list_head)
 {
 ASTs children;
 children.swap(list_head->children);
 {
 LinkedList * next = list_head->next;
 list_head->~LinkedList();
 operator delete(list_head);
 list_head = next;
 }

 for (auto & child : children)
 {
 if (child == nullptr || child->use_count() != 1)
 continue;

 ptr = child.detach();
 chassert(ptr->ref_counter.fetch_sub(1, std::memory_order_acq_rel) == 1);

 LinkedList * elem = LinkedList::create(ptr);
 if (elem)
 {
 elem->next = list_head;
 list_head = elem;
 }
 }
 }
 }
}

Read this carefully, because it is the cleverest piece of the whole parser subtree.

A naive intrusive_ptr_release would be:

 cpp
 
if (--refcount == 0) delete this; // ~IAST destroys children -> releases them -> recursion

That is recursive destruction: freeing a node releases its children, which releases their children, and a 1000-deep AST costs 1000 stack frames. With a default max_parser_depth of 1000, that is the difference between “works” and “stack overflow on a malicious query”.

The fix is to turn the recursion into an iteration, and it does it by reusing the node’s own memory:

LinkedList::create(ptr) moves the node’s children vector out, destroys the node object in place (ptr->~IAST()), and then constructs a LinkedList in the same memory using placement new. Because static_assert(sizeof(LinkedList) <= sizeof(IAST)), the recycled node is big enough to hold the list link.
The list of recycled nodes becomes a work queue. Each iteration takes one node off the queue, and for each of its children that is uniquely owned (use_count() != 1 means somebody else holds a reference — skip it), recycles that child the same way, and pushes it onto the queue.
Nothing recurses. The stack depth is constant; the heap holds the work list.

The use_count() != 1 check is what makes the DAG safe: a node referenced from two places is not deleted when the first holder lets go.

chassert(ptr->ref_counter.fetch_sub(1, ...) == 1) is the delicate line. The child was detached from its parent’s vector, so the parent’s reference is gone; the refcount must now drop to zero, and the code asserts it. In a release build the assert compiles away and the value is discarded, but the invariant holds.

The flag word

flags_storage is 32 bits that every derived class can use for its own boolean flags, without adding a member and breaking the 32-byte budget. The accessor is a constrained reinterpret_cast:

 cpp
 
 
// src/Parsers/IAST.h:79
 template <typename BitfieldStruct>
 BitfieldStruct & flags()
 {
 static_assert(std::is_standard_layout_v<BitfieldStruct>);
 static_assert(sizeof(BitfieldStruct) == sizeof(flags_storage), "Bitfield struct must be the same size as flags_storage");
 static_assert(BitfieldStruct::RESERVED_BITS <= 31, "RESERVED_BITS exceeds 31 (bit 31 is reserved for parenthesized)");

 if constexpr (!std::is_void_v<typename BitfieldStruct::ParentFlags>)
 {
 static_assert(hasParentReserved<BitfieldStruct>(), "Derived flags struct must have _parent_reserved field");
 static_assert(
 BitfieldStruct::ParentFlags::RESERVED_BITS < BitfieldStruct::RESERVED_BITS,
 "Derived RESERVED_BITS must be greater than parent's");
 }

 return *reinterpret_cast<BitfieldStruct *>(&flags_storage);
 }

The rules are enforced by the compiler, not by comments:

the bitfield struct must be exactly the size of the flag word;
it may use at most 31 bits, because bit 31 belongs to IAST itself (parenthesized);
a derived struct must reserve the bits its parent already uses, so the ParentFlags chain is checked at compile time.

A derived class declares its flags like this:

 cpp
 
 
// src/Parsers/ASTWithAlias.h:16
 struct ASTWithAliasFlags
 {
 using ParentFlags = void;
 static constexpr UInt32 RESERVED_BITS = 1;

 UInt32 prefer_alias_to_column_name : 1;
 UInt32 unused : 31;
 };

ASTWithAlias has no parent flags (void), so it may use all 32 bits minus the one IAST reserved — hence unused : 31 and RESERVED_BITS = 1. A subclass of ASTWithAlias would declare ParentFlags = ASTWithAliasFlags and RESERVED_BITS = 2, and the static asserts would catch it if it tried to overlap bit 0.

Children and aliases

The children vector owns the child nodes. But most AST classes also want a typed pointer to a particular child — ASTSelectQuery wants ASTExpressionList * select_expression_list, not ASTPtr. So there are two representations, and the invariant is that the vector owns and the member pointer observes.

 cpp
 
 
// src/Parsers/IAST.h:220
 template <typename T>
 void set(T * & field, const ASTPtr & child)
 {
 if (!child)
 return;

 T * cast = dynamic_cast<T *>(child.get());
 if (!cast)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Could not cast AST subtree");

 children.push_back(child);
 field = cast;
 }

 template <typename T>
 void replace(T * & field, const ASTPtr & child)
 {
 if (!child)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Trying to replace AST subtree with nullptr");

 T * cast = dynamic_cast<T *>(child.get());
 if (!cast)
 throw Exception(ErrorCodes::LOGICAL_ERROR, "Could not cast AST subtree");

 for (ASTPtr & current_child : children)
 {
 if (current_child.get() == field)
 {
 current_child = child;
 field = cast;
 return;
 }
 }

 throw Exception(ErrorCodes::LOGICAL_ERROR, "AST subtree not found in children");
 }

set appends to children and points the member at the same object. replace finds the old child in children (by pointer identity — it compares the raw pointers, not the values) and swaps it. setOrReplace and reset complete the set.

The danger of this scheme is obvious: if somebody assigns to the member pointer directly, the children vector and the member disagree, and the refcount leaks. updatePointerToChild exists for the case where a child is replaced behind the class’s back:

 cpp
 
 
// src/Parsers/IAST.h:344
 /// After changing one of `children` elements, update the corresponding member pointer if needed.
 void updatePointerToChild(const IAST * old_ptr, const ASTPtr & new_ptr)
 {
 std::function<void(IAST **, boost::intrusive_ptr<IAST> *)> f = [old_ptr, new_ptr](IAST ** raw, boost::intrusive_ptr<IAST> * smart)
 {
 if (raw && *raw == old_ptr)
 *raw = new_ptr.get();
 else if (smart && smart->get() == old_ptr)
 *smart = new_ptr;
 };
 forEachPointerToChild(f);
 }

forEachPointerToChild is a virtual hook that each class overrides to list its naked pointers, so the base class can repair them after a rewrite.

Copying

Because the tree is shared, “copy” has to be explicit. clone() is a deep copy, and every class implements it:

 cpp
 
 
// src/Parsers/IAST.h:155
 /** Get a deep copy of the tree. Cloned object must have the same range. */
 virtual ASTPtr clone() const = 0;

and the helper that most implementations use is:

 cpp
 
 
// src/Parsers/IAST.cpp:319
void IAST::cloneChildren()
{
 checkStackSize();
 for (auto & child : children)
 child = child->clone();
}

Note the checkStackSize(): clone() recurses, so cloning a 1000-deep tree does use 1000 stack frames. The recursion is bounded by the depth limit that the parser enforced, so it is safe — and the stack check catches the case where an AST was built by a rewriter that did not go through the parser.

Hashing

The tree hash identifies an expression structurally. It is what the query cache keys on, and what common-subexpression elimination compares.

 cpp
 
 
// src/Parsers/IAST.cpp:148
IASTHash IAST::getTreeHash(bool ignore_aliases) const
{
 SipHash hash_state;
 updateTreeHash(hash_state, ignore_aliases);
 return getSipHash128AsPair(hash_state);
}


void IAST::updateTreeHash(SipHash & hash_state, bool ignore_aliases) const
{
 checkStackSize();
 updateTreeHashImpl(hash_state, ignore_aliases);
 hash_state.update(children.size());
 for (const auto & child : children)
 child->updateTreeHash(hash_state, ignore_aliases);
}

The hash is a 128-bit SipHash, computed over the node identifiers and the child count, recursively. ignore_aliases makes SELECT a AS x and SELECT a AS y hash the same, which is what you want for subexpression elimination but not for a query cache key.

The updateTreeHashImpl hook lets a class add its own members to the hash. The base implementation hashes only the getID() string, and the header is explicit about the trap:

 cpp
 
 
// src/Parsers/IAST.h:158
 /** Get hash code, identifying this element and its subtree.
 * Hashing by default ignores aliases (e.g. identifier aliases, function aliases, literal aliases) which is
 * useful for common subexpression elimination. Set 'ignore_aliases = false' if you don't want that behavior.
 *
 * The default implementation only hashes `getID` and `children`. When adding a member that is
 * not a child and is part of the element's meaning, hash it here as well, otherwise two
 * elements that differ only in that member get the same hash, and the callers that use the
 * hash as an identity of the expression treat them as the same expression.
 */

That last paragraph is a trap documented in the source: add a member to ASTFunction, forget to hash it, and two different functions become “the same expression” for the optimizer.

Limits

The parser bounds the recursion depth of the parse, but the AST can end up deeper than the parse was:

 cpp
 
 
// src/Parsers/parseQuery.cpp:569
 /// Also check on the AST level, because the generated AST depth can be greater than the recursion depth of the parser.
 if (res && max_parser_depth)
 res->checkDepth(max_parser_depth);

The check is iterative, not recursive, for the same reason the destructor is:

 cpp
 
 
// src/Parsers/IAST.cpp:173
size_t IAST::checkDepthImpl(size_t max_depth) const
{
 std::vector<std::pair<ASTPtr, size_t>> stack;
 stack.reserve(children.size());

 for (const auto & i: children)
 stack.push_back({i, 1});

 size_t res = 0;

 while (!stack.empty())
 {
 auto top = stack.back();
 stack.pop_back();

 if (top.second >= max_depth)
 throw Exception(ErrorCodes::TOO_DEEP_AST, "AST is too deep. Maximum: {}", max_depth);

 res = std::max(res, top.second);

 for (const auto & i: top.first->children)
 stack.push_back({i, top.second + 1});
 }

 return res;
}

An explicit stack of (node, depth) pairs, allocated on the heap. The same pattern — recursion in the builder, iteration in the checker and the destructor — appears throughout the AST code.

Printing it back

The AST can be turned back into SQL, which is what EXPLAIN, SHOW CREATE TABLE, the query log and the distributed-query rewriter all need. The formatter is a virtual formatImpl on every node, driven by a FormatStateStacked frame that carries the nesting level and a handful of flags:

 cpp
 
 
// src/Parsers/IAST.h:404
 /// The state that is copied when each node is formatted. For example, nesting level.
 struct FormatStateStacked
 {
 UInt16 indent = 0;
 bool need_parens = false;
 bool expression_list_always_start_on_new_line = false; /// Line feed and indent before expression list even if it's of single element.
 bool expression_list_prepend_whitespace = false; /// Prepend whitespace (if it is required)
 bool surround_each_list_element_with_parens = false;
 bool allow_operators = true; /// Format some functions, such as "plus", "in", etc. as operators.
 bool allow_moving_operators_before_parens = true; /// Allow moving operators like "-" before parens: (-...) -> -(...)
 size_t list_element_index = 0;
 std::string create_engine_name;
 const IAST * current_select = nullptr;
 const IAST * current_function = nullptr; /// Pointer to the function whose arguments are being formatted
 bool parent_has_trailing_settings = false; /// A parent ASTQueryWithOutput will append SETTINGS after this node's output.
 bool has_trailing_output_options = false; /// A parent ASTQueryWithOutput has trailing output options (SETTINGS, FORMAT, INTO OUTFILE).
 bool disable_from_first_syntax = false; /// Disable FROM-first syntax for SELECTs inside INSERT (to avoid parsing ambiguity).
 /// Set by a parent formatter that has explicitly emitted `(` around this child to indicate
 /// that the child should not emit its own `parenthesized` parens (which would duplicate the parent's).
 /// Consumed (cleared) by `IAST::format` so it only applies one level deep.
 bool wrapped_in_parens = false;
 /// Set by `IAST::format` when the node has both `isParenthesized()` and a non-empty alias,
 /// to tell `ASTWithAlias::formatImpl` to emit `(expr) AS alias` instead of `(expr AS alias)`.
 /// This keeps the format-parse-format round-trip stable: `(expr) AS alias` re-parses with
 /// `parenthesized=true` on the aliased node, which formats back to `(expr) AS alias`.
 bool parenthesize_alias_inner_only = false;
 };

That is 20-odd flags, all of them one bit or one small integer, all living in a struct that is copied down the tree rather than passed by reference. The copying is the point: a node’s formatter can change the flags for its children without disturbing its siblings.

The parenthesized bit in flags_storage is a good example of why the flag word exists at all. ClickHouse remembers that the user wrote (a + b) rather than a + b, so that the query can be printed back exactly as it was written — but the parentheses carry no meaning for the analyzer, and the storage engine’s table definitions want them gone. Hence a separate formatIgnoringRedundantParentheses() entry point.

Dumping the tree

For debugging, dumpTree prints the tree with indentation and the address of every node:

 cpp
 
 
// src/Parsers/IAST.cpp:424
void IAST::dumpTree(WriteBuffer & ostr, size_t indent) const
{
 checkStackSize();
 String indent_str(indent, '-');
 ostr << indent_str << getID() << ", ";
 writePointerHex(this, ostr);
 writeChar('\n', ostr);
 for (const auto & child : children)
 {
 if (!child)
 throw Exception(ErrorCodes::UNKNOWN_ELEMENT_IN_AST, "Can't dump a nullptr child");
 child->dumpTree(ostr, indent + 1);
 }
}

The indentation is a run of - characters, one per level, and the node’s getID() names it. This is what EXPLAIN AST=1 produces:

 sql
 
EXPLAIN AST = 1 SELECT a + 1 FROM t;

 text
 
 
SelectWithUnionQuery, 0x55f1c0a4e2d0
-ExpressionList, 0x55f1c0a4e330
--Function_plus, 0x55f1c0a4e580
---Identifier_a, 0x55f1c0a4e710
---Literal_1, 0x55f1c0a4e8a0
-ASTTableIdentifier_t, 0x55f1c0a4e940

Each line is the indentation, the node’s getID() (which is the class name plus the node’s own name, joined with an underscore), and the node’s address. The address is how you spot the DAG: the same address appearing twice means the subtree is shared.

The shape of the thing

 text
 
 
 ASTSelectQuery
 (refcount 1)
 |
 +----------+----------+
 | |
 ASTExpressionList ASTTableExpression
 (refcount 1) (refcount 1)
 | |
 | +----------+----------+
 | | |
 ASTIdentifier ASTFunction ASTIdentifier
 (refcount 2) (refcount 1) (refcount 1)
 ^ |
 | |
 +-------+-------+
 |
 shared by two slots

An ASTIdentifier reached from two places has a refcount of 2. When one holder is dropped, the count goes to 1 and nothing happens. When the last one goes, the node is recycled — and, if it has children, its children are recycled too, iteratively.

The cost

Per node: 32 bytes of header, plus the children vector’s heap block (which is one allocation per node that has children), plus the node’s own payload. A SELECT with 10 columns produces on the order of 100 nodes, so a typical query’s AST is a few kilobytes — and it is allocated once and shared everywhere afterwards.

The price is that IAST is a base class with a virtual destructor, a virtual formatImpl, a virtual clone, a virtual getID, a virtual updateTreeHashImpl, a virtual writeJSON/readJSON, and a virtual forEachPointerToChild. That is at least six vtable slots and an indirect call on every operation, on a structure that is traversed constantly.

ClickHouse pays that deliberately: the AST is the interface between the parser, the analyzer, the planner, the formatter, the rewriters and the logs, and all of them are in different translation units and different libraries. A flat struct-of-arrays would be faster and would couple them together. The tree of polymorphic nodes keeps them apart, and the 32-byte header keeps the cost of that decision small.

ClickHouse Resolves Identifiers While It Parses

Jay Hubbert — Sun, 25 Oct 2026 09:00:00 +0000

TL;DR: An identifier is not just a string. db.tbl.col is a qualified reference, col might be a column or an alias or a table, and {id:UInt32} is not a name at all — it is a hole to be filled in later. ClickHouse’s parser resolves all of that eagerly, at parse time: it unquotes the name, splits it into parts, guesses which part is the table qualifier, and builds a parameter node for a placeholder.

Most parsers treat an identifier as an opaque string. The lexer produces QuotedIdentifier or BareWord, the parser copies the bytes into a string, and everything else — is this a column, an alias, a table? which table? — is deferred to the analyzer, which is allowed to ask the catalog.

 sql
 
 
-- Four spellings, four different shapes of node:
SELECT a, `a`, "a", {a:Identifier} FROM db.tbl;
-- ^ ^ ^ ^
-- | | | +-- ASTIdentifier("", [ASTQueryParameter])
-- | | +------------ ASTIdentifier("a") (quoting is forgotten)
-- | +----------------- ASTIdentifier("a")
-- +----------------------- ASTIdentifier("a")

ClickHouse does part of that work in the parser, and the reason is practical: the AST is shared between the parser, the analyzer, the rewriters, the formatter and the query log, and a node that has already resolved its own spelling needs no further interpretation. The price is a set of deliberate approximations, several of which the source marks with FIXME.

One node, three spellings

ParserIdentifier accepts a backquoted name, a double-quoted name, a Unicode-quoted name, a bare word, and — optionally — a query parameter:

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:344
bool ParserIdentifier::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
{
 /// Identifier in backquotes or in double quotes or in English-style Unicode double quotes
 if (pos->type == TokenType::QuotedIdentifier)
 {
 /// The case of Unicode quotes. No escaping is supported. Assuming UTF-8.
 if (*pos->begin == '\xE2' && pos->size() >= 6)
 {
 if (pos->size() == 6) /// Empty Unicode-quoted identifiers are not allowed.
 return false;
 node = make_intrusive<ASTIdentifier>(String(pos->begin + 3, pos->end - 3));
 ++pos;
 return true;
 }

 ReadBufferFromMemory buf(pos->begin, pos->size());
 String s;

 if (*pos->begin == '`')
 readBackQuotedStringWithSQLStyle(s, buf);
 else
 readDoubleQuotedStringWithSQLStyle(s, buf);

 if (s.empty()) /// Identifiers "empty string" are not allowed.
 return false;

 node = make_intrusive<ASTIdentifier>(s);
 ++pos;
 return true;
 }
 if (pos->type == TokenType::BareWord)
 {
 node = make_intrusive<ASTIdentifier>(String(pos->begin, pos->end));
 ++pos;
 return true;
 }

The unquoting happens here, in the parser, not in the lexer. The lexer hands over a token range; readBackQuotedStringWithSQLStyle turns `ab`` intoab and readDoubleQuotedStringWithSQLStyle turns "a\"b" into a"b. By the time the ASTIdentifier exists, the name is the name, not the spelling.

The price is that the spelling is gone. The source is explicit about it:

 cpp
 
 
// src/Parsers/IdentifierQuotingStyle.h:43
 /// A name that the user wrote quoted solely to preserve a mixed-case spelling (`(SELECT "Foo" FROM t)`)
 /// therefore comes out unquoted and is folded to `foo` as well. That is not a property of this rule:
 /// `ParserIdentifier` does not record whether an identifier was quoted, so `"Foo"` and `Foo` are the very
 /// same parsed identifier, and every rule that does not quote unconditionally - `WhenNecessary`, which
 /// this one replaced, included - emits it the same way. Only the `query('...')` form, which is passed to
 /// the external database verbatim, can express a case-sensitive mixed-case identifier.
 AlwaysUnlessUpperCase,

"Foo" and Foo parse to the same AST node. Only the formatter’s quoting rule can put the quotes back, and it decides to do so based on the name’s content (does it contain upper case? is it a keyword?), not on what the user typed.

Compound names

ParserCompoundIdentifier handles a.b.c, and it is where the eager resolution starts. It parses the parts one at a time, and each part may be separated by a dot, or by one of three special two-character delimiters:

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:502
 auto element_parser = std::make_unique<ParserIdentifier>(allow_query_parameter, highlight_type);
 std::vector<std::pair<ParserPtr, SpecialDelimiter>> delimiter_parsers;
 delimiter_parsers.emplace_back(std::make_unique<ParserTokenSequence>(std::vector<TokenType>{TokenType::Dot, TokenType::Colon}), SpecialDelimiter::JSON_PATH_DYNAMIC_TYPE);
 delimiter_parsers.emplace_back(std::make_unique<ParserTokenSequence>(std::vector<TokenType>{TokenType::Dot, TokenType::Caret}), SpecialDelimiter::JSON_PATH_PREFIX);
 delimiter_parsers.emplace_back(std::make_unique<ParserTokenSequence>(std::vector<TokenType>{TokenType::Dot, TokenType::At}), SpecialDelimiter::JSON_PATH_COMBINED);
 delimiter_parsers.emplace_back(std::make_unique<ParserToken>(TokenType::Dot), SpecialDelimiter::NONE);

The three special delimiters exist because of ClickHouse’s JSON column type: json.a.b.:UInt32 is the dynamic type of the subcolumn a.b, json.a.^path is a path prefix, and json.a.@path is a combined one. The parser encodes the delimiter by gluing it to the front of the part and back-quoting the rest:

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:527
 if (last_special_delimiter != SpecialDelimiter::NONE)
 {
 parts.push_back(static_cast<char>(last_special_delimiter) + backQuote(getIdentifierName(element)));
 }
 else
 {
 parts.push_back(getIdentifierName(element));
 /// Check if we have Array of JSON subcolumn additioon after identifier
 /// and replace it with corresponding type subcolumn.
 if (!is_first && array_of_json_identifier_addition.check(pos, expected))
 parts.push_back(array_of_json_identifier_addition.getLastArrayOfJSONSubcolumnIdentifier());
 }

So the part : + `UInt32` is stored as the string :`UInt32`, and the formatter can tell it apart from a part that happens to be spelled UInt32 with a backquote in it. The split is done later, by splitSpecialDelimiterAndIdentifierIfAny, when the node is formatted.

The eager guess

Now the interesting part. The compound constructor stores the parts and guesses which one is the table qualifier:

 cpp
 
 
// src/Parsers/ASTIdentifier.cpp:33
ASTIdentifier::ASTIdentifier(std::vector<String> && name_parts_, bool special, ASTs && name_params)
 : name_parts(name_parts_), semantic(std::make_shared<IdentifierSemanticImpl>())
{
 chassert(!name_parts.empty());
 semantic->special = special;
 semantic->legacy_compound = true;
 if (!name_params.empty())
 {
 [[maybe_unused]] size_t params = 0;
 for (const auto & part [[maybe_unused]] : name_parts)
 {
 if (part.empty())
 ++params;
 }
 chassert(params == name_params.size());
 children = std::move(name_params);
 }
 else
 {
 for (const auto & part [[maybe_unused]] : name_parts)
 chassert(!part.empty());

 if (!special && name_parts.size() >= 2)
 semantic->table = name_parts.end()[-2];

 resetFullName();
 }
}

semantic->table = name_parts.end()[-2]; — the second to last part is declared to be the table qualifier, at construction time, long before the FROM clause has been parsed. For db.tbl.col the guess is tbl; for tbl.col it is tbl.

This is the “resolve eagerly” trick in its purest form, and it is a guess, not a resolution. The parser does not know whether a.b is table.column or alias.column or two parts of a database name; it records the hypothesis, and the analyzer corrects it later using the actual FROM clause. The IdentifierSemantic machinery in src/Interpreters/IdentifierSemantic.h is the corrector: it has setColumnShortName, setColumnLongName, chooseTable, canReferColumnToTable, and a ColumnMatch enum that distinguishes ColumnName, AliasedTableName, TableName, DBAndTable, TableAlias and Ambiguous.

The same constructor also flattens the parts into the dotted name:

 cpp
 
 
// src/Parsers/ASTIdentifier.cpp:283
void ASTIdentifier::resetFullName()
{
 full_name = name_parts[0];
 for (size_t i = 1; i < name_parts.size(); ++i)
 full_name += '.' + name_parts[i];
}

so ASTIdentifier carries both the parts and the joined name. The joined name is what getID() and the tree hash use; the parts are what the analyzer uses.

The semantic is a pimpl

 cpp
 
 
// src/Parsers/ASTIdentifier.h:60
 String full_name;
 std::vector<String> name_parts;

protected:
 std::shared_ptr<IdentifierSemanticImpl> semantic; /// pimpl

IdentifierSemanticImpl lives in src/Interpreters/, not in src/Parsers/. Putting it behind a shared_ptr keeps the parser library independent of the analyzer library — the parser can be built standalone for WebAssembly (utils/wasm-parser), and the whole src/Interpreters tree does not have to come along.

The shared_ptr (rather than a value) has a second purpose. clone() copies the semantic by value:

 cpp
 
 
// src/Parsers/ASTIdentifier.cpp:169
ASTPtr ASTIdentifier::clone() const
{
 auto ret = make_intrusive<ASTIdentifier>(*this);
 ret->semantic = std::make_shared<IdentifierSemanticImpl>(*ret->semantic);
 ret->cloneChildren();
 return ret;
}

A clone must not share its semantic with the original, because the analyzer mutates the semantic in place (setting the table membership, covering the name with an alias, and so on). If the semantic were shared, resolving one copy would resolve the other.

Tables are a different node

ASTTableIdentifier derives from ASTIdentifier and is what the parser builds for a table reference:

 cpp
 
 
// src/Parsers/ASTIdentifier.h:80
class ASTTableIdentifier : public ASTIdentifier
{
public:
 explicit ASTTableIdentifier(const String & table_name, ASTs && name_params = {});
 explicit ASTTableIdentifier(const StorageID & table_id, ASTs && name_params = {});
 ASTTableIdentifier(const String & database_name, const String & table_name, ASTs && name_params = {});

 String getID(char delim) const override { return "TableIdentifier" + (delim + name()); }
 ASTPtr clone() const override;
 void writeJSON(WriteBuffer & out) const override;
 void readJSON(const Poco::JSON::Object & json) override;

 UUID uuid = UUIDHelpers::Nil; // FIXME(ilezhankin): make private
 /// True iff the parser saw an explicit `UUID '...'` clause, set even when the parsed value is `Nil`.
 /// Use this (not `uuid != UUIDHelpers::Nil`) when you need to distinguish "user wrote `UUID '...'`"
 /// from "no UUID clause": explicit `UUID '00000000-0000-0000-0000-000000000000'` parses to `Nil`.
 bool has_uuid = false;

The distinction matters: a table reference may carry a UUID clause, and the has_uuid flag distinguishes “the user wrote UUID '...'” from “there was no UUID clause” even when the UUID is the nil UUID. The parser decides which node to build:

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:563
 ParserKeyword s_uuid(Keyword::UUID);
 UUID uuid = UUIDHelpers::Nil;
 bool has_uuid_clause = false;

 if (table_name_with_optional_uuid)
 {
 if (parts.size() > 2)
 return false;

 if (s_uuid.ignore(pos, expected))
 {
 ParserStringLiteral uuid_p;
 ASTPtr ast_uuid;
 if (!uuid_p.parse(pos, ast_uuid, expected))
 return false;
 uuid = parseFromString<UUID>(ast_uuid->as<ASTLiteral>()->value.safeGet<String>());
 has_uuid_clause = true;
 }

 if (parts.size() == 1) node = make_intrusive<ASTTableIdentifier>(parts[0], std::move(params));
 else node = make_intrusive<ASTTableIdentifier>(parts[0], parts[1], std::move(params));

A table reference may have at most two parts — table or database.table — and the parser rejects a third. The UUID is parsed by the parser, not by the analyzer, and the string is converted to a UUID right there.

Query parameters

Now the interesting case: {name:Type}. The parser accepts it as an identifier, but it is not a name — it is a placeholder.

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:380
 if (allow_query_parameter && pos->type == TokenType::OpeningCurlyBrace)
 {
 ++pos;
 if (pos->type != TokenType::BareWord)
 {
 expected.add(pos, "substitution name (identifier)");
 return false;
 }

 String name(pos->begin, pos->end);
 ++pos;

 if (pos->type != TokenType::Colon)
 {
 expected.add(pos, "colon between name and type");
 return false;
 }

 ++pos;

 if (pos->type != TokenType::BareWord)
 {
 expected.add(pos, "substitution type (identifier)");
 return false;
 }

 String type(pos->begin, pos->end);
 ++pos;

 if (type != "Identifier")
 {
 expected.add(pos, "substitution type (identifier)");
 return false;
 }

Note the last check: the only type accepted in the identifier position is the literal word Identifier. {id:UInt32} is not an identifier — it is a value of a known type, and the value parsers handle it. The identifier parser only handles the case where the whole name is unknown:

 cpp
 
 
// src/Parsers/ExpressionElementParsers.cpp:422
 node = make_intrusive<ASTIdentifier>("", make_intrusive<ASTQueryParameter>(name, type));
 return true;
 }
 return false;
}

An ASTIdentifier with an empty name and one child, which is an ASTQueryParameter:

 cpp
 
 
// src/Parsers/ASTQueryParameter.h:9
/// Parameter in query with name and type of substitution ({name:type}).
/// Example: SELECT * FROM table WHERE id = {pid:UInt16}.
class ASTQueryParameter : public ASTWithAlias
{
public:
 String name;
 String type;

 ASTQueryParameter(const String & name_, const String & type_) : name(name_), type(type_) {}

 /** Get the text that identifies this element. */
 String getID(char delim) const override { return String("QueryParameter") + delim + name + ':' + type; }

For a compound name with a parameter in it — {db}.tbl.col — the same trick is used per part: the part is stored as an empty string in name_parts, and the parameter node is stored as a child. The compound constructor’s chassert(params == name_params.size()) is what keeps the two in step, and the formatter walks them together:

 cpp
 
 
// src/Parsers/ASTIdentifier.cpp:228
 if (compound())
 {
 for (size_t i = 0, j = 0, size = name_parts.size(); i < size; ++i)
 {
 if (i != 0)
 ostr << '.';

 /// Some AST rewriting code, like IdentifierSemantic::setColumnLongName,
 /// does not respect children of identifier.
 /// Here we also ignore children if they are empty.
 if (name_parts[i].empty() && j < children.size())
 {
 children[j]->format(ostr, settings, state, frame);
 ++j;
 }
 else
 format_element(name_parts[i]);
 }
 }

The empty part is a placeholder in the name_parts vector, and the child list is walked in parallel. The comment is a warning from the author: some rewriting code does not respect the children of an identifier, so the formatter has to cope with a part that is empty and has no corresponding child.

The whole mechanism carries a FIXME in the header:

 cpp
 
 
// src/Parsers/ASTIdentifier.h:19
/// FIXME: rewrite code about params - they should be substituted at the parsing stage,
/// or parsed as a separate AST entity.

The parameters are resolved after parsing, by a visitor, and the author thinks the whole approach is wrong. It is still there, and it works, but the source says what it thinks of it.

SETTINGS

SET and the SETTINGS clause of a SELECT share one parser. The parse_only_internals flag is what distinguishes them:

 cpp
 
 
// src/Parsers/ParserSetQuery.cpp:339
bool ParserSetQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
{
 ParserToken s_comma(TokenType::Comma);

 if (!parse_only_internals)
 {
 ParserKeyword s_set(Keyword::SET);

 if (!s_set.ignore(pos, expected))
 return false;

 /// Parse SET TRANSACTION ... queries using ParserTransactionControl
 if (ParserKeyword{Keyword::TRANSACTION}.check(pos, expected))
 return false;

 /// Parse SET TIME ZONE 'tz' as an alias for SET session_timezone = 'tz'
 if (ParserKeyword{Keyword::TIME_ZONE}.ignore(pos, expected))
 {
 ParserToken eq(TokenType::Equals);
 eq.ignore(pos, expected); // optional, for PostgreSQL compatibility
 ASTPtr value_node;
 ParserLiteralOrMap literal_parser;

 if (!literal_parser.parse(pos, value_node, expected))
 return false;

 auto query = make_intrusive<ASTSetQuery>();
 node = query;

 query->is_standalone = !parse_only_internals;

 SettingChange change;
 change.name = "session_timezone";
 change.value = value_node->as<ASTLiteral &>().value;
 query->changes.push_back(std::move(change));

 return true;
 }
 }

SET TIME ZONE 'UTC' is rewritten by the parser into a session_timezone change — a PostgreSQL compatibility alias that never reaches the analyzer as a SET TIME ZONE.

The SETTINGS clause of a SELECT uses the same parser with parse_only_internals = true:

 cpp
 
 
// src/Parsers/ParserSelectQuery.cpp:875
 /// SETTINGS key1 = value1, key2 = value2, ...
 if (s_settings.ignore(pos, expected))
 {
 ParserSetQuery parser_settings(true);

 if (!parser_settings.parse(pos, settings, expected))
 return false;
 }

so SELECT 1 SETTINGS max_threads = 8 and SET max_threads = 8 produce the same ASTSetQuery, differing only in the is_standalone flag.

The pair parser is where the setting values are decided:

 cpp
 
 
// src/Parsers/ParserSetQuery.cpp:239
bool ParserSetQuery::parseNameValuePairWithParameterOrDefault(
 SettingChange & change, String & default_settings, ParserSetQuery::Parameter & parameter, IParser::Pos & pos, Expected & expected, bool enable_shorthand_syntax)
{
 ParserCompoundIdentifier name_p;
 ParserLiteralOrMap value_p;
 ParserToken s_eq(TokenType::Equals);
 ParserFunction function_p;

 ASTPtr node;
 String name;
 ASTPtr function_ast;
 bool have_eq = false;

 if (!name_p.parse(pos, node, expected))
 return false;

 have_eq = s_eq.ignore(pos, expected);

 if (!enable_shorthand_syntax && !have_eq)
 return false;

 tryGetIdentifierNameInto(node, name);

Three things can follow a setting name:

param_ — a query parameter, not a setting:

 cpp
 
 
// src/Parsers/ParserSetQuery.cpp:262
 /// Parameter
 if (name.starts_with(QUERY_PARAMETER_NAME_PREFIX))
 {
 if (!have_eq)
 return false;

 name = name.substr(strlen(QUERY_PARAMETER_NAME_PREFIX));

 if (name.empty())
 throw Exception(ErrorCodes::BAD_ARGUMENTS, "Parameter name cannot be empty");

 String value;

 if (!parseParameterValueIntoString(pos, value, expected))
 return false;

 parameter = {std::move(name), std::move(value)};
 return true;
 }

DEFAULT — reset to the default value;
a value — a literal, a map, a disk(...) call, or a {name:Type} parameter.

And there is the shorthand: a setting name with no value at all means = true.

 cpp
 
 
// src/Parsers/ParserSetQuery.cpp:323
 else
 {
 /// A setting name with no value is shorthand for `= true`. Only a Bool setting can be
 /// written this way, but the parser does not know the settings schema, so it records that
 /// the value was omitted and leaves the check to `BaseSettings::applyChange`.
 node = make_intrusive<ASTLiteral>(Field(true));
 }

 change.name = name;
 change.value = node->as<ASTLiteral &>().value;
 change.shorthand = !have_eq;

The parser records that the value was omitted (change.shorthand = true) rather than pretending it knows the value. The settings schema check happens later, in BaseSettings::applyChange. That is the right division of labour: the parser does not know the list of settings, and there are hundreds of them with dozens of types.

A SET statement and a SETTINGS clause produce the same node:

 sql
 
 
SET max_threads = 8, force_index_by_date; -- shorthand: = true
SET param_rows = 1000; -- a query parameter, not a setting
SELECT 1 SETTINGS max_threads = 8; -- the same ASTSetQuery, is_standalone = false

The result is a small, flat structure:

 cpp
 
 
// src/Parsers/ASTSetQuery.h:17
/** SET query
 */
class ASTSetQuery : public IAST
{
public:
 /// If false, this AST is a part of another query, such as SELECT.
 bool is_standalone = true;

 SettingsChanges changes;
 /// settings that will be reset to default value
 std::vector<String> default_settings;
 NameToNameVector query_parameters;

 /** Get the text that identifies this element. */
 String getID(char) const override { return "Set"; }

Three vectors: the changes, the settings to reset, and the parameters. Note that ASTSetQuery has no children — the settings are not AST nodes at all, they are SettingChange structs with a name and a Field value. The parser decided the value’s type at parse time, so there is nothing left to resolve.

Filling the holes

The parameters are resolved after parsing, by a visitor:

 cpp
 
 
// src/Interpreters/ReplaceQueryParameterVisitor.h:17
class ReplaceQueryParameterVisitor
{
public:
 explicit ReplaceQueryParameterVisitor(const NameToNameMap & parameters)
 : query_parameters(parameters)
 {}

 void visit(ASTPtr & ast);

 /// Resolve query parameters used as setting values (e.g. `max_threads = {threads:UInt64}`)
 /// directly in a SettingsChanges list. Used when settings from a query are applied to a
 /// context outside of the full AST traversal (e.g. on the client side).
 void visitSettingsChanges(SettingsChanges & changes);

Two entry points, because a setting value can hold a parameter without the setting living in the AST: SET max_threads = {threads:UInt64} is a SettingsChanges entry, and the client applies it to a context without walking the AST. visitSettingsChanges handles that case.

The visitor rebuilds the identifiers whose parts were parameters:

 text
 
 
 {db}.tbl.col
 |
 v
 ASTIdentifier(name_parts = ["", "tbl", "col"], children = [ASTQueryParameter("db", "Identifier")])
 |
 | ReplaceQueryParameterVisitor, with db = "mydb"
 v
 ASTIdentifier(name_parts = ["mydb", "tbl", "col"], children = [])

The placeholder is replaced by the value, and the empty part is filled in.

What this buys, and what it costs

The eager resolution buys three things:

the AST is self-describing — ASTIdentifier knows its own parts, its joined name, and its hypothesis about the table qualifier, so the formatter, the tree hash and the log do not need the catalog;
the parser does the work once — the unescaping, the splitting and the joining are done at parse time and never repeated;
the UUID clause and the SET TIME ZONE alias are resolved at parse time, where the surrounding syntax is still known.

The costs are visible in the source:

the FIXME about parameters — a placeholder is encoded as an empty string in a vector of names, and every consumer of name_parts has to know that;
the semantic->table guess — the second-to-last part is assumed to be a table, and the analyzer has to correct it, which means the parser’s answer is sometimes wrong until then;
the loss of the quoting — "Foo" and Foo are the same node, so the formatter cannot reproduce the user’s spelling;
the TODO(ilezhankin) markers on restoreTable, setShortName and createTable, which are the code that patches up the parser’s guesses after the fact.

The pattern is consistent: the parser resolves everything it can resolve without the catalog, and leaves everything that needs the catalog — which column is this, which table does it belong to — to the analyzer, but leaves behind a hypothesis in the node so the analyzer has a starting point.

ClickHouse Parses SQL with a Backtracking Recursive-Descent Parser

Jay Hubbert — Sun, 25 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse’s parser is not a generated LL(k) table. It is a set of ~200 small IParser objects, each of which is handed a mutable position into a token stream, tries to consume something, and rewinds the position if it fails. Every rewind is counted, and a query that rewinds more than 1 000 000 times is rejected.

SQL is a language designed by committee, and it is not LL(1). SELECT a FROM t and SELECT a FROM t WHERE x differ only in what follows FROM, and the parser cannot know until it has looked. A column may be followed by an alias, a table may be followed by a join or by a WHERE, and ( may open a subquery, a tuple, a function argument list, or a parenthesised expression.

Two classic answers:

Backtracking (recursive descent). Try to parse the alternative; if it fails, put the input back and try the next one. Simple to write, exponential in the worst case, and needs a hard budget to stay safe.
Table-driven (yacc/bison). Precompute the decision table. Fast and linear, but you need a grammar compiler, and the grammar of a language like ClickHouse’s SQL is large enough that the tables get big.

ClickHouse takes the first route, and the interesting engineering is not in the grammar — it is in making “put the input back” cheap, and in making “this query is trying too many things” detectable.

The token stream

The parser never sees the query text. It sees a lazily-produced vector of tokens:

 cpp
 
 
// src/Parsers/TokenIterator.h:35
 const Token & operator[] (size_t index)
 {
 while (true)
 {
 if (index < data.size())
 {
 max_pos = std::max(max_pos, index);
 return data[index];
 }

 if (!data.empty() && data.back().isEnd())
 {
 max_pos = data.size() - 1;
 return data.back();
 }

 Token token = lexer.nextToken();

 if (!skip_insignificant || token.isSignificant())
 data.emplace_back(token);
 }
 }

 const Token & max()
 {
 if (data.empty())
 return (*this)[0];
 return data[max_pos];
 }

operator[] is a pull on demand: asking for token 40 lexes tokens 0..40 and stops. Since a failed parse rewinds, the same tokens are re-read rather than re-lexed — the lexer runs at most once per token of the query.

max_pos is the rightmost token anybody has looked at. It is what tryParseQuery uses to decide where the error is:

 cpp
 
 
// src/Parsers/parseQuery.cpp:564
 ASTPtr res;
 const bool parse_res = parser.parse(token_iterator, res, expected);
 const auto last_token = token_iterator.max();
 _out_query_end = last_token.end;

Tokens::reset() exists for the one case where the parser has to be restarted from scratch: the fast path in tryParseQuery that scans for lexical errors before attempting a real parse, and then throws the scan away.

On top of Tokens sits a cursor:

 cpp
 
 
// src/Parsers/TokenIterator.h:82
 ALWAYS_INLINE const Token & get() { return (*tokens)[index]; }
 ALWAYS_INLINE const Token & operator*() { return get(); }
 ALWAYS_INLINE const Token * operator->() { return &get(); }

 ALWAYS_INLINE TokenIterator & operator++()
 {
 ++index;
 return *this;
 }
 ALWAYS_INLINE TokenIterator & operator--()
 {
 --index;
 return *this;
 }

The class is a Tokens * and a size_t index, plus the five comparison operators that compare indices, isValid() (which stops at EndOfStream) and max() (the rightmost token anybody looked at).

A TokenIterator is a Tokens* and a size_t. Copying it is copying 16 bytes, which is what makes “save the position” cheap.

The position is the parser’s state

IParser::Pos is a TokenIterator plus four counters. This is the heart of the whole design:

 cpp
 
 
// src/Parsers/IParser.h:120
 struct Pos : TokenIterator
 {
 uint32_t depth = 0;
 uint32_t max_depth = 0;

 uint32_t backtracks = 0;
 uint32_t max_backtracks = 0;

 Pos(Tokens & tokens_, uint32_t max_depth_, uint32_t max_backtracks_)
 : TokenIterator(tokens_), max_depth(max_depth_), max_backtracks(max_backtracks_)
 {
 }

 Pos(TokenIterator token_iterator_, uint32_t max_depth_, uint32_t max_backtracks_)
 : TokenIterator(token_iterator_), max_depth(max_depth_), max_backtracks(max_backtracks_)
 {
 }

Two budgets, one for each way a recursive-descent parser can go wrong:

depth counts nesting — a subquery inside a subquery inside an expression. The limit is max_parser_depth, default 1000.
backtracks counts rewinds — how many times the parser has said “that alternative did not work, let me try another”. The limit is max_parser_backtracks, default 1 000 000.

 cpp
 
 
// src/Core/Defines.h:91
/// Default limit on recursion depth of recursive descend parser.
static constexpr auto DBMS_DEFAULT_MAX_PARSER_DEPTH = 1000;
/// Default limit on the amount of backtracking of recursive descend parser.
static constexpr auto DBMS_DEFAULT_MAX_PARSER_BACKTRACKS = 1000000;

How a rewind is detected

Here is the part I like. The parser never declares that it is about to backtrack. There is no try_parse helper, no mark()/reset() pair. Instead, the position notices that it is being moved backwards:

 cpp
 
 
// src/Parsers/IParser.cpp:14
/// NOLINTBEGIN(cert-oop54-cpp)
IParser::Pos & IParser::Pos::operator=(const IParser::Pos & rhs)
{
 depth = rhs.depth;
 max_depth = rhs.max_depth;
 backtracks = std::max(backtracks, rhs.backtracks);
 max_backtracks = rhs.max_backtracks;

 if (rhs < *this)
 {
 ++backtracks;
 if (max_backtracks && backtracks > max_backtracks)
 throw Exception(ErrorCodes::TOO_SLOW_PARSING, "Maximum amount of backtracking ({}) exceeded in the parser. "
 "Consider rising max_parser_backtracks parameter.", max_backtracks);
 }

 TokenIterator::operator=(rhs);

 return *this;
}
/// NOLINTEND(cert-oop54-cpp)

rhs < *this is rhs.index < this.index: the new position is behind the old one. That is a rewind, so the counter goes up by one, and if it exceeds the budget the parser throws TOO_SLOW_PARSING.

The consequences are elegant:

Every rewind is counted, including ones nobody thought to instrument. If a new parser is added that rewinds in a loop, it is caught for free.
The counter is monotonic: backtracks = std::max(backtracks, rhs.backtracks) means restoring an old snapshot cannot lower the count.
The check is max_backtracks && ..., so a limit of 0 means “unlimited” — which is how max_parser_depth is documented (“0 — Recursion depth is unlimited”).

The budget is therefore not a property of any one parser; it is a property of the position, and it is shared by the whole tree of parsers.

The same Pos also carries the depth counter, and enforces it:

 cpp
 
 
// src/Parsers/IParser.h:150
 ALWAYS_INLINE void increaseDepth()
 {
 ++depth;
 if (unlikely(max_depth > 0 && depth > max_depth))
 throw Exception(ErrorCodes::TOO_DEEP_RECURSION, "Maximum parse depth ({}) exceeded. "
 "Consider rising max_parser_depth parameter.", max_depth);

and, further down in the same function, a checkStackSize() every 8192 frames (128 under sanitizers or musl, where the frames are bigger):

 cpp
 
 
// src/Parsers/IParser.h:157
 /** Sometimes the maximum parser depth can be set to a high value by the user,
 * but we still want to avoid stack overflow.
 * For this purpose, we can use the checkStackSize function, but it is too heavy.
 * The solution is to check not too frequently.
 * The frequency is arbitrary, but not too large, not too small,
 * and a power of two to simplify the division.
 */
#if defined(USE_MUSL) || defined(SANITIZER) || !defined(NDEBUG)
 static constexpr uint32_t check_frequency = 128;
#else
 static constexpr uint32_t check_frequency = 8192;
#endif
 if (depth % check_frequency == 0)
 checkStackSize();
 }

That call is a guard against the other failure mode: a user who sets max_parser_depth to 100 000 and sends a query with 100 000 nested parentheses will overflow the C++ stack before the counter trips.

The rewind idiom

Now the actual parser code. Every parser that can fail does the same thing:

 cpp
 
 
// src/Parsers/IParserBase.h:17
 template <typename F>
 ALWAYS_INLINE static bool wrapParseImpl(Pos & pos, const F & func)
 {
 Pos begin = pos;
 bool res = func();
 if (!res)
 pos = begin;
 return res;
 }

 struct IncreaseDepthTag {};

 template <typename F>
 ALWAYS_INLINE static bool wrapParseImpl(Pos & pos, IncreaseDepthTag, const F & func)
 {
 Pos begin = pos;
 pos.increaseDepth();
 bool res = func();
 pos.decreaseDepth();
 if (!res)
 pos = begin;
 return res;
 }

Pos begin = pos; is a 32-byte copy (a 16-byte cursor plus four counters). pos = begin; is the rewind, and the operator= above notices it and charges it to the budget. There is no exception handling, no RAII guard, no std::optional — the failure path is one assignment.

The second overload is the depth-tracking one. IParserBase::parse uses it:

 cpp
 
 
// src/Parsers/IParserBase.cpp:7
bool IParserBase::parse(Pos & pos, ASTPtr & node, Expected & expected)
{
 expected.add(pos, getName());

 return wrapParseImpl(pos, IncreaseDepthTag{}, [&]
 {
 const char * begin = pos->begin;
 bool res = parseImpl(pos, node, expected);

and on failure it clears the node:

 cpp
 
 
// src/Parsers/IParserBase.cpp:31
 else
 node = nullptr;
 return res;
 });
}

Three things happen in the wrapper, and every parser in the codebase gets them for free:

expected.add(pos, getName()) — before trying, tell the error machinery what this parser was hoping for, and where.
increaseDepth() / decreaseDepth() — the nesting budget.
On failure, node = nullptr — a failed parse must not leave a half-built AST behind.

The Highlight block records the byte range a keyword or identifier parser consumed, so the query editor can colour the original text; the range is recovered from the token stream, not from the AST.

The interface

IParser is deliberately tiny:

 cpp
 
 
// src/Parsers/IParser.h:196
 virtual bool parse(Pos & pos, ASTPtr & node, Expected & expected) = 0;

 bool ignore(Pos & pos, Expected & expected)
 {
 ASTPtr ignore_node;
 return parse(pos, ignore_node, expected);
 }

The contract, in the source’s own words: parse piece of text from position pos, move pointer pos to the maximum position to which it was possible to parse, in case of success return true and the result in node if it is needed, otherwise false, in expected write what was expected in the maximum position.

The convenience wrappers are what parsers actually call: ignore throws the AST away, and check is the lookahead primitive — parse, but restore the position even on success:

 cpp
 
 
// src/Parsers/IParser.h:212
 bool check(Pos & pos, Expected & expected)
 {
 Pos begin = pos;
 ASTPtr node;
 if (!parse(pos, node, expected))
 {
 pos = begin;
 return false;
 }
 return true;
 }

check answers “does the next thing match this parser?”. Its sibling checkWithoutMoving takes the position by value, so the caller’s cursor is never touched at all — which is why it does not need to restore anything.

Error messages: the rightmost failure wins

A backtracking parser has a problem: it fails in many places, and the user only cares about the last one — the one furthest along the query.

Expected solves this with a simple rule: a failure at a position further right than any previous failure wipes the list of alternatives and starts a new one. The struct holds an InlineVector<const char *, 7> variants (seven const char * fit inline, with a heap fallback beyond that) and a max_parsed_pos pointer into the query text:

 cpp
 
 
// src/Parsers/IParser.h:90
 /// 'description' should be statically allocated string.
 ALWAYS_INLINE void add(const char * current_pos, const char * description)
 {
 if (!max_parsed_pos || current_pos > max_parsed_pos)
 {
 variants.clear();
 max_parsed_pos = current_pos;
 variants.push_back(description);
 return;
 }

 if ((current_pos == max_parsed_pos) && (std::find(variants.begin(), variants.end(), description) == variants.end()))
 variants.push_back(description);
 }

The strings are static literals, so there is nothing to own.

Because every parser calls expected.add(pos, getName()) before it tries, the “expected” list at the end of a failed parse is exactly the set of alternatives that were live at the furthest position reached. SELECT a FROM fails with “expected: table identifier, subquery, function” rather than with the fifty things that were expected three tokens earlier.

Keywords are matched by text

A keyword is not a token type. ParserKeyword matches a word against a string, case-insensitively, and it supports compound keywords with anything in between:

 cpp
 
 
// src/Parsers/CommonParsers.cpp:103
 while (true)
 {
 expected.add(pos, current_word);

 if (pos->type != TokenType::BareWord)
 return false;

 const char * const next_whitespace = find_first_symbols<' ', '\0'>(current_word, s.data() + s.size());
 const size_t word_length = next_whitespace - current_word;

 if (word_length != pos->size())
 return false;

 if (0 != strncasecmp(pos->begin, current_word, word_length))
 return false;

 ++pos;

 if (!*next_whitespace)
 break;

 current_word = next_whitespace + 1;
 }

The while loop is what makes ORDER BY a single keyword: the keyword string is "ORDER BY", and the loop walks the space-separated words one at a time, requiring a BareWord token of exactly the right length for each. Since the lexer throws away whitespace and comments by default, ORDER/*x*/BY matches too.

The keyword itself is one of the 666 entries in the Keyword enum, generated from a macro in src/Parsers/CommonParsers.h:

 cpp
 
 
// src/Parsers/CommonParsers.h:688
enum class Keyword : size_t
{
#define DECLARE_PARSER_KEYWORD_ENUM(identifier, name) identifier,

 APPLY_FOR_PARSER_KEYWORDS(DECLARE_PARSER_KEYWORD_ENUM) APPLY_FOR_PARSER_KEYWORDS_WITH_UNDERSCORES(DECLARE_PARSER_KEYWORD_ENUM)
#undef DECLARE_PARSER_KEYWORD_ENUM
};

Trying the alternatives

With all those pieces in place, the top-level statement parser is just a long chain of alternatives, tried in order:

 cpp
 
 
// src/Parsers/ParserQuery.cpp:132
 bool res = query_with_output_p.parse(pos, node, expected)
 || insert_p.parse(pos, node, expected)
 || use_p.parse(pos, node, expected)
 || parseSetRoleQuery(pos, node, expected)
 || set_p.parse(pos, node, expected)
 || system_p.parse(pos, node, expected)
 || parseCreateAccessEntityQuery(pos, node, expected)
 || create_function_p.parse(pos, node, expected)
 || drop_function_p.parse(pos, node, expected)
 || create_workload_p.parse(pos, node, expected)

… and then fifteen more: named collections, handlers, indexes, hypothetical objects, grants, transactions, DELETE, UPDATE and COPY.

Twenty-six alternatives, each of which must fail cheaply if it is not the right one. ParserInsertQuery starts by matching INSERT; if the token is not INSERT it fails after touching one token. The whole chain is O(number of statement kinds) in the common case.

The order matters, and the source says so:

 cpp
 
 
// src/Parsers/ParserQuery.cpp:51
/** The DCL statements: access management, in three groups because each has to keep its place among
 * the other alternatives. `SET ROLE` has to be tried before `SET`, which would otherwise read it
 * as the shorthand for a `Bool` setting named `ROLE`.
 *
 * A `CLICKHOUSE_PARSER_NO_DCL` build has none of them, and the linker then drops
 * `src/Parsers/Access` and `src/Access/Common` along with them. See `utils/wasm-parser`.
 */

That is a genuine ambiguity, resolved by ordering rather than by lookahead: SET ROLE must be tried before SET, because SET alone also accepts a bareword setting name.

What backtracking costs, and what it buys

The failure mode of a backtracking parser is a query that makes it try an enormous number of alternatives. ClickHouse’s answer is the budget in Pos::operator=, and the error it produces:

 sql
 
 
-- SET max_parser_backtracks = 100;
-- SELECT ...; -- a query crafted to explore many alternatives
-- Code: 407, DB::Exception: Maximum amount of backtracking (100) exceeded in the parser.
-- Consider rising max_parser_backtracks parameter. (TOO_SLOW_PARSING)

The cost of the design is that the worst case is exponential in the nesting depth, and ClickHouse pays for that with a limit rather than with an algorithm. The benefit is that a new SQL feature is a new IParser subclass and nothing else — no grammar file, no generated tables, no build-time codegen. src/Parsers/ contains about 200 of them.

The other benefit is error messages. Because every alternative records what it expected at the position it reached, the error can say “one of: table identifier, subquery, function” — and, since the alternatives are keywords from the language, it can also find the typo:

 cpp
 
 
// src/Parsers/parseQuery.cpp:208
 for (const char * variant : expected.variants)
 {
 /// Only a description that is a keyword itself can be meant literally: the parser also adds prose
 /// such as `SELECT query, possibly with UNION`. A single word cannot be a typo of a multi-word
 /// keyword either, so anything with a space is not a candidate.
 const std::string_view variant_view(variant);
 if (variant_view.size() + max_distance < token.size() || token.size() + max_distance < variant_view.size())
 continue;
 if (!std::all_of(variant_view.begin(), variant_view.end(), [](char c) { return isUpperAlphaASCII(c) || c == '_'; }))
 continue;

 const size_t distance = levenshteinDistanceCaseInsensitive(token, String(variant_view));
 /// Zero distance means the token is that keyword, so it cannot be the reason of the failure.
 if (distance > 0 && distance < best_distance)
 {
 best_distance = distance;
 best_variant = variant_view;
 }
 }

 return best_variant;
}

The thresholds are deliberately tight — one edit for short words, two for longer ones — because the candidate list is keywords, not identifiers from the query. Almost every short word is within a small distance of some keyword, so a loose threshold would suggest nonsense. The comment says it outright: “a wrong guess is worse than no guess”.

The whole picture

 text
 
 
 query text
 |
 v
 +-----------+ +-------------+ +-------------+
 | Lexer | -----> | Tokens | ---> | TokenIterator|
 | (no alloc)| | (lazy vec) | | (index) |
 +-----------+ +-------------+ +------+------+
 |
 v
 +-----------------+
 | Pos |
 | index |
 | depth <= 1000
 | backtracks<=1e6 |
 +--------+--------+
 |
 +----------------------------------------------+-------------+
 | | | |
 parse() succeeds parse() fails ignore() check()
 | | | (rewind)
 v v v
 AST subtree pos = begin -----------> rewind charged
 (ASTPtr) (charged) to budget

The design in one sentence: the parser has exactly one piece of mutable state, and every failure is a rewind of that state.

That is what makes the two budgets meaningful. The depth budget bounds the C++ stack, and the backtrack budget bounds the work. Both are enforced in one place, and both are enforced for every parser in the codebase without any of them opting in.