Parsing on Opensource internals

How the ClickHouse Client Splits a Script into Queries

Jay Hubbert — Sat, 24 Oct 2026 09:00:00 +0000

TL;DR: clickhouse-client --queries-file script.sql does not split the file on ;. It parses one statement at a time, lets the parser tell it where the statement ended, ends inline INSERT data at a blank line, and then walks forward over the semicolons and the trailing comment so the -- { serverError 42 } hint stays attached to the statement it describes.

Splitting a SQL script into statements sounds trivial. It is not, and the reasons are all in the corners of the language:

A semicolon can be data: INSERT INTO t VALUES ('a;b').
A semicolon can be part of a format’s data: INSERT INTO t FORMAT CSV 1,2;3 — is that the number 3 or the end of the statement?
A semicolon can be inside a comment: -- nothing; to see here.
A semicolon can be inside a string: SELECT ';'.
A statement can have no semicolon at all: the last line of the file.
A statement can be followed by a comment that belongs to it: SELECT 1; -- { serverError 42 } — and the client’s test framework depends on that comment being attached to the right statement.

Here is a script that exercises all of it:

 sql
 
 
-- script.sql
SELECT 1; -- statement 1, terminated by ';'
SELECT 'a;b'; -- the ';' is inside a string: still one statement
INSERT INTO t FORMAT CSV
1,2;3
 -- a blank line ends the payload, not the ';'
SELECT 3 -- statement 4, no trailing ';' at all

And the pipeline the client runs to get there:

 text
 
 
 script text
 |
 v
 parseQuery(one statement) -- the parser advances the pointer itself
 |
 v
 INSERT with inline data? --yes--> VALUES: the real VALUES format parser
 | other formats: find the next "\n\n"
 |no
 v
 adjustQueryEnd: skip the ';'s, pull in the trailing comment
 |
 v
 execute | check the "-- { serverError N }" hint

Every SQL client solves this differently, and every solution leaks somewhere. psql uses a lexer. mysql uses a lexer with a mode stack. ClickHouse’s client uses a parser — it asks the real SQL parser where the statement ends — and then applies a small number of special cases for the places where the parser cannot know.

The entry point

--queries-file is a plain option:

 cpp
 
 
// src/Client/ClientBase.cpp:4678
 ("query,q", po::value<std::vector<std::string>>()->multitoken(), R"(Query. Can be specified multiple times (--query "SELECT 1" --query "SELECT 2") or once with multiple semicolon-separated queries (--query "SELECT 1; SELECT 2;"). In the latter case, INSERT queries with non-VALUE format must be separated by empty lines.)")
 ("queries-file", po::value<std::vector<std::string>>()->multitoken(), "File path with queries to execute; multiple files can be specified (--queries-file file1 file2...)")
 ("multiquery,n", "Obsolete, does nothing")

Note the multiquery flag: it is documented as obsolete and does nothing, because the client has always been able to run a multi-statement script. The flag survives only so that old scripts do not break.

Several files can be given, and each can be preceded by “interleave” files that are run before it — a mechanism for setting up state:

 cpp
 
 
// programs/client/Client.cpp:1211
 ("interleave-queries-file",

The whole thing is driven from runNonInteractive:

 cpp
 
 
// src/Client/ClientBase.cpp:5385
 if (!buzz_house && !queries_files.empty())
 {
 for (const auto & queries_file : queries_files)
 {
 for (const auto & interleave_file : interleave_queries_files)
 if (!processMultiQueryFromFile(interleave_file))
 return;

 if (!processMultiQueryFromFile(queries_file))
 return;
 }

 return;
 }

and processMultiQueryFromFile reads the whole file into memory and hands it to the multi-statement executor:

 cpp
 
 
// src/Client/ClientBase.cpp:5356
bool ClientBase::processMultiQueryFromFile(const String & file_name)
{
 if (isEmbeeddedClient())
 throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "Reading queries from file is not supported for an embedded client");

 String queries_from_file;

 ReadBufferFromFile in(file_name);
 readStringUntilEOF(queries_from_file, in);

 /// For `clickhouse-local` only: same entry point as `-q` / stdin so meta-commands (`clear`, `ls`,
 /// `\i`, …) work for whole-file input. Remote `clickhouse-client` keeps `executeMultiQuery` so
 /// `--queries-file` does not apply `exit_strings` and other text-level metas (avoids silent
 /// behavior changes for batch automation).
 if (supportsLocalMetaCommands())
 return processQueryText(queries_from_file);
 return executeMultiQuery(queries_from_file);
}

The difference at the end is a deliberate compatibility decision: clickhouse-local treats a file as “text typed at the prompt”, so \i and clear work inside it; clickhouse-client treats it as a script, and running it through the meta-command processor could change the meaning of an existing pipeline.

The same function is the implementation of the interactive \i command, so “read this file and run it” is one code path everywhere.

The splitting rule, as the code states it

The rule the client actually implements is written down in a comment, right where the multi-statement executor is set up:

 cpp
 
 
// src/Client/ClientBase.cpp:3587
 /// Several queries separated by ';'.
 /// INSERT data is ended by the empty line (\n\n), not ';'.
 /// Unnecessary semicolons may cause data to be parsed containing ';'
 /// e.g. 'insert into xx format csv val;' will insert "val;" instead of "val"
 /// 'insert into xx format csv val\n;' will insert "val" and ";"
 /// An exception is VALUES format where we also support semicolon in
 /// addition to end of line.

That comment is the whole specification, and it is worth reading twice, because the third line is the surprising one: insert into xx format csv val; inserts the string val; — the semicolon is data. The only reliable separator for a data-carrying INSERT is a blank line.

The executor is a loop over a state machine with four states:

 cpp
 
 
// src/Client/ClientBase.cpp:3607
 while (true)
 {
 auto stage = analyzeMultiQueryText(this_query_begin, this_query_end, all_queries_end,
 parsed_query, current_exception);
 switch (stage)
 {
 case MultiQueryProcessingStage::QUERIES_END:
 {
 /// Compatible with old version when run interactive, e.g. "", "\ld"
 if (is_first && is_interactive)
 {
 auto u = processTextAsSingleQuery(all_queries_text);
 UNUSED(u);
 }
 return true;
 }

QUERIES_END, PARSING_FAILED, CONTINUE_PARSING and PARSING_EXCEPTION are the four outcomes; the interesting one is EXECUTE_QUERY, which falls through the switch into the code that runs the statement.

Stage one: let the parser find the boundary

analyzeMultiQueryText is where the work happens. Its first job is to throw away the leading whitespace, so that the queries in the server’s query log do not start with a newline:

 cpp
 
 
// src/Client/ClientBase.cpp:3340
 const char * raw_query_begin = this_query_begin;

 // Remove leading empty newlines and other whitespace, because they
 // are annoying to filter in the query log. This is mostly relevant for
 // the tests.
 while (this_query_begin < all_queries_end && isWhitespaceASCII(*this_query_begin))
 ++this_query_begin;

Its second job is to detect the “only comments left” case, which the parser cannot handle:

 cpp
 
 
// src/Client/ClientBase.cpp:3354
 // If there are only comments left until the end of file, we just
 // stop. The parser can't handle this situation because it always
 // expects that there is some query that it can parse.
 // We can get into this situation because the parser also doesn't
 // skip the trailing comments after parsing a query. This is because
 // they may as well be the leading comments for the next query,
 // and it makes more sense to treat them as such.
 {
 Tokens tokens(this_query_begin, all_queries_end);
 IParser::Pos token_iterator(tokens, max_parser_depth, max_parser_backtracks);
 bool only_comments_left = !token_iterator.isValid();

and its third job — the actual split — is to hand the rest of the script to the parser and let it consume exactly one statement:

 cpp
 
 
// src/Client/ClientBase.cpp:3384
 this_query_end = this_query_begin;
 try
 {
 parsed_query = parseQuery(this_query_end, all_queries_end,
 client_context->getSettingsRef(),
 /*allow_multi_statements=*/ true, raw_query_begin);
 }

parseQuery takes a const char *& and advances it past what it parsed. That is the trick: the parser is the splitter. The client does not look for semicolons; it asks ClickHouse’s own SQL parser to parse one statement and tells it where the end of the buffer is. A ; inside a string literal is inside the string literal, because the lexer said so.

The one thing the parser cannot do is know where a FORMAT CSV payload ends, because the payload is not SQL at all. That is handled separately, below.

Stage two: the INSERT data

After the parser returns, the client checks whether the statement is an INSERT with inline data, and if so, finds the end of the data:

 cpp
 
 
// src/Client/ClientBase.cpp:3444
 // INSERT queries may have the inserted data in the query text that follow the query itself, e.g. "insert into t format CSV 1,2". They
 // need special handling.
 // - If the INSERT statement FORMAT is VALUES, we use the VALUES format parser to skip the inserted data until we reach the trailing single semicolon.
 // - Other formats (e.g. FORMAT CSV) are arbitrarily more complex and tricky to parse. For example, we may be unable to distinguish if the semicolon
 // is part of the data or ends the statement. In this case, we simply assume that the end of the INSERT statement is determined by \n\n (two newlines).

VALUES gets the good treatment: the actual VALUES format parser is used to walk the rows, so a semicolon inside a string literal is handled correctly:

 cpp
 
 
// src/Client/ClientBase.cpp:3461
 if (insert_ast->format == "Values")
 {
 // Invoke the VALUES format parser to skip the inserted data.
 // Skip SQL comments too (as ValuesBlockInputFormat::read does), not just
 // whitespace: a trailing comment after the last row would otherwise be scanned
 // as row data past the terminating ';', swallowing the following queries.
 ReadBufferFromMemory data_in(insert_ast->data, all_queries_end - insert_ast->data);
 skipBOMIfExists(data_in);
 do
 {
 skipWhitespaceAndSQLComments(data_in);
 if (data_in.eof() || *data_in.position() == ';')
 break;
 }
 while (ValuesBlockInputFormat::skipToNextRow(&data_in, 1, 0));

Every other format gets the crude rule — a blank line ends the data:

 cpp
 
 
// src/Client/ClientBase.cpp:3488
 else
 {
 // Handling of generic formats
 auto pos_newline = String(insert_ast->data, all_queries_end).find("\n\n");
 if (pos_newline != std::string::npos)
 this_query_end = insert_ast->data + pos_newline;
 else
 this_query_end = all_queries_end;
 }
 insert_ast->end = this_query_end;

This is the rule the option’s help text warns about: “In the latter case, INSERT queries with non-VALUE format must be separated by empty lines.”

It is a real limitation, and the reason is not laziness: a FORMAT CSV payload has no length, no terminator and no quoting rules the client could rely on. A semicolon in a CSV field is legal. A blank line is the only separator the client and the server can agree on without reimplementing every format’s parser.

Stage three: eat the semicolons and the trailing comment

Now the client knows where the statement ends, but there may be a semicolon and a trailing comment after it. The semicolon has to go, and the comment has to be kept — because the test framework puts the expected error code in it:

 cpp
 
 
// src/Client/ClientBase.cpp:764
void ClientBase::adjustQueryEnd(
 const char *& this_query_end, const char * all_queries_end, uint32_t max_parser_depth, uint32_t max_parser_backtracks)
{
 // We have to skip the trailing semicolon that might be left
 // after VALUES parsing or just after a normal semicolon-terminated query.
 Tokens after_query_tokens(this_query_end, all_queries_end);
 IParser::Pos after_query_iterator(after_query_tokens, max_parser_depth, max_parser_backtracks);
 while (after_query_iterator.isValid() && after_query_iterator->type == TokenType::Semicolon)
 {
 this_query_end = after_query_iterator->end;
 ++after_query_iterator;
 }

Note that it skips all consecutive semicolons, so SELECT 1;; is one statement, not two.

Then the comment logic, which is the subtlest part of the whole file:

 cpp
 
 
// src/Client/ClientBase.cpp:777
 // Now we have to do some extra work to add the trailing
 // same-line comment to the query, but preserve the leading
 // comments of the next query. The trailing comment is important
 // because the test hints are usually written this way, e.g.:
 // select nonexistent_column; -- { serverError 12345 }.
 // The token iterator skips comments and whitespace, so we have
 // to find the newline in the string manually. If it's earlier
 // than the next significant token, it means that the text before
 // newline is some trailing whitespace or comment, and we should
 // add it to our query. There are also several special cases
 // that are described below.
 const auto * newline = find_first_symbols<'\n'>(this_query_end, all_queries_end);
 const char * next_query_begin = after_query_iterator->begin;

 // We include the entire line if the next query starts after
 // it. This is a generic case of trailing in-line comment.
 // The "equals" condition is for case of end of input (they both equal
 // all_queries_end);
 if (newline <= next_query_begin)
 {
 chassert(newline >= this_query_end);
 this_query_end = newline;
 }
 else
 {
 // Many queries on one line, can't do anything. By the way, this
 // syntax is probably going to work as expected:
 // select nonexistent /* { serverError 12345 } */; select 1
 }

The rule is: if the next statement starts on a later line, everything up to the end of the current line belongs to this statement. If two statements share a line, the comment is ambiguous and the client gives up — with the comment above pointing at the syntax that does work.

This matters because of TestHint, the mechanism ClickHouse’s own test suite uses to assert on errors:

 cpp
 
 
// src/Client/TestHint.h:17
/// Checks expected server and client error codes.
///
/// The following comment hints are supported:
///
/// - "-- { serverError 60 }" -- in case of you are expecting server error.
/// - "-- { serverError 16, 36 }" -- in case of you are expecting one of the 2 errors.
///
/// - "-- { clientError 20 }" -- in case of you are expecting client error.
/// - "-- { clientError 20, 60, 92 }" -- It's expected that the client will return one of the 3 errors.

and the hint is checked against the exception the server (or the client) produced:

 cpp
 
 
// src/Client/ClientBase.cpp:3804
 else if (test_hint.hasServerErrors())
 {
 if (!server_exception)
 {
 error_matches_hint = false;
 error_stream << fmt::format("Expected server error code '{}' but got no server error (query: {}).\n",
 test_hint.serverErrors(), full_query);
 }

So the whole comment-attaching dance exists so that a test file can say “this statement must fail with error 60” and the client can check it. The hint is not a nicety; it is the assertion mechanism of the entire ClickHouse integration test suite.

The statement that is executed

After all of that, the client has three views of the same statement, and uses each for a different purpose:

 cpp
 
 
// src/Client/ClientBase.cpp:3694
 // Save query without trailing comment
 auto query_to_execute = std::string_view(all_queries_text).substr(this_query_begin - all_queries_text.data(), this_query_end - this_query_begin);
 size_t insert_query_without_data_length = 0;
 if (const auto * insert = parsed_query->as<ASTInsertQuery>())
 insert_query_without_data_length = insert->data - query_to_execute.data();

 // Try to include the trailing comment with test hints. It is just
 // a guess for now, because we don't yet know where the query ends
 // if it is an INSERT query with inline data. We will do it again
 // after we have processed the query. But even this guess is
 // beneficial so that we see proper trailing comments in "echo" and
 // server log.
 {
 unsigned max_parser_depth = static_cast<unsigned>(client_context->getSettingsRef()[Setting::max_parser_depth]);
 unsigned max_parser_backtracks = static_cast<unsigned>(client_context->getSettingsRef()[Setting::max_parser_backtracks]);
 adjustQueryEnd(this_query_end, all_queries_end, max_parser_depth, max_parser_backtracks);
 }
 // query + inline INSERT data + trailing comments (the latter is our best guess for now).
 auto full_query = std::string_view(all_queries_text).substr(this_query_begin - all_queries_text.data(), this_query_end - this_query_begin);

query_to_execute — the statement without the trailing comment. This is what is sent to the server, so the comment never becomes part of the SQL.
full_query — the statement with the trailing comment. This is what is scanned for the test hint, and what is echoed.
parsed_query — the AST, which is what actually gets planned and sent.

And the script position is tracked for the server’s benefit:

 cpp
 
 
// src/Client/ClientBase.cpp:3714
 ++script_query_number;
 script_line_number += std::count(prev_query_begin, this_query_begin, '\n');
 prev_query_begin = this_query_begin;
 client_context->setScriptQueryAndLineNumber(script_query_number, 1 + script_line_number);

so that an error message from the server can say “statement 7, line 42 of your script” rather than just quoting the statement.

The format follows the file extension

One more piece of “the client guesses” behaviour, and it is the one users notice first. The client infers the format from the file name — for the input, from the file the client is reading; for the output, from the file the query writes to.

For the input, the client has to ask the kernel what file descriptor 0 actually is:

 cpp
 
 
// src/Common/tryGetFileNameByFileDescriptor.cpp:14
std::optional<String> tryGetFileNameFromFileDescriptor(int fd)
{
#ifdef OS_LINUX
 std::string proc_path = fmt::format("/proc/self/fd/{}", fd);
 char file_path[PATH_MAX] = {'\0'};
 if (readlink(proc_path.c_str(), file_path, sizeof(file_path) - 1) != -1)
 return file_path;
 return std::nullopt;
#elif defined(OS_DARWIN)
 char file_path[PATH_MAX] = {'\0'};
 if (fcntl(fd, F_GETPATH, file_path) != -1)
 return file_path;
 return std::nullopt;

readlink("/proc/self/fd/0") gives the original path of the redirected file, even through a pipe (/dev/stdin -> the real file), and F_GETPATH is the macOS equivalent. With that name in hand, the extension is mapped to a format:

 cpp
 
 
// src/Formats/FormatFactory.cpp:1113
std::optional<String> FormatFactory::tryGetFormatFromFileName(String file_name)
{
 if (file_name == "stdin")
 return tryGetFormatFromFileDescriptor(STDIN_FILENO);

 CompressionMethod compression_method = chooseCompressionMethod(file_name, "");
 if (CompressionMethod::None != compression_method)
 {
 auto pos = file_name.find_last_of('.');
 if (pos != String::npos)
 file_name = file_name.substr(0, pos);
 }

 auto pos = file_name.find_last_of('.');
 if (pos == String::npos)
 return std::nullopt;

 String file_extension = file_name.substr(pos + 1, String::npos);
 boost::algorithm::to_lower(file_extension);
 auto it = file_extension_formats.find(file_extension);
 if (it == file_extension_formats.end())
 return std::nullopt;

 return it->second;
}

Three details: the compression suffix is stripped first (so data.csv.gz is a CSV file, not a “gz” file), the extension is lowercased (so DATA.CSV works), and an unknown extension returns nothing rather than guessing.

The client then uses it as the default input format:

 cpp
 
 
// src/Client/ClientBase.cpp:1418
 else
 {
 std::optional<String> format_from_file_name = FormatFactory::instance().tryGetFormatFromFileDescriptor(stdin_fd);
 if (format_from_file_name)
 default_input_format = *format_from_file_name;
 else
 default_input_format = "auto";
 }

so clickhouse-client < data.csv reads CSV, clickhouse-client < data.jsonl reads JSONEachRow, and clickhouse-client < data.whatever falls back to auto (the format that sniffs the input).

The output side is the same trick, applied to INTO OUTFILE:

 cpp
 
 
// src/Client/ClientBase.cpp:1124
 else if (query_with_output->out_file)
 {
 auto format_name = FormatFactory::instance().tryGetFormatFromFileName(out_file);
 if (format_name)
 {
 current_format = *format_name;
 outfile_format_from_extension = true;
 }
 }

SELECT * FROM t INTO OUTFILE 'out.json' writes JSON, because the file is called out.json. And the extension-derived format has a higher priority than the default_format setting, which is why the code needs a flag to remember where the format came from:

 cpp
 
 
// src/Client/ClientBase.cpp:1146
 /// ... only when the client itself was not given an explicit output format. `--output-format`,
 /// `--format` and `--vertical` all set `is_default_format = false` (the last via
 /// `default_output_format = "Vertical"`), and such an explicit choice must win over the
 /// `default_format` *setting* — e.g. the display default the local client now seeds as a
 /// setting. Without this guard, `clickhouse-local --vertical` is silently overridden by that
 /// setting and prints TSV instead of Vertical.
 else if (is_default_format && !has_format_clause && !outfile_format_from_extension
 && !format_settings_ref[Setting::default_format].value.empty())
 current_format = format_settings_ref[Setting::default_format];

The full precedence chain is: an explicit FORMAT clause or --format wins; then the output_format/format settings; then the file extension; then the default_format setting; then the client’s built-in default.

Trade-offs

What the design buys.

Correct splitting for free. Because the real parser does the split, the client agrees with the server about where a statement ends in every case the SQL grammar can express — including semicolons inside strings, inside comments, and inside CASE ... WHEN ... THEN ... END.
No lexer to maintain. A hand-written lexer would have to reimplement the SQL string, identifier and comment rules, and would drift from the server’s.
Errors are attributable. The script query number and line number are tracked and sent to the server, so a failure in a 5 000-line script points at a line.
The test framework rides on it. The -- { serverError 42 } hint mechanism is a direct consequence of attaching the trailing comment to the right statement.
Formats are inferred. clickhouse-client < data.csv and INTO OUTFILE 'x.json' both work without a --format flag, because the client asks the kernel for the file name and looks at the extension.

What it costs.

The whole file is in memory. readStringUntilEOF reads the entire script before running anything. A 2 GB script is a 2 GB allocation.
The INSERT ... FORMAT rule is crude. Everything except VALUES ends at a blank line, so a CSV payload cannot contain a blank line, and a payload whose last row is followed by a single newline runs to the end of the file. The VALUES case is much better, because it uses the real format parser.
Two statements on one line lose their comment. select 1 /* { serverError 5 } */; select 2 cannot be disambiguated, and the client says so in a comment rather than guessing.
The extension inference can surprise you. INTO OUTFILE 'data.txt' writes whatever format txt maps to — or, if txt is not registered, it silently keeps the default format. The inference is a convenience with a sharp edge, and the sharp edge is that the mapping is global and configurable by whatever formats the build has.

A parse error aborts the rest of the script unless --ignore-error is given, in which case the client resynchronises by skipping to the next semicolon:

 cpp
 
 
// src/Client/ClientBase.cpp:3419
 else
 {
 Tokens tokens(this_query_begin, all_queries_end);
 IParser::Pos token_iterator(tokens, max_parser_depth, max_parser_backtracks);
 while (token_iterator->type != TokenType::Semicolon && token_iterator.isValid())
 ++token_iterator;
 this_query_begin = token_iterator->end;
 }

which is another place where a ; inside a string would confuse the resynchronisation.

The one thing to remember: the client does not split scripts on semicolons. It parses them, one statement at a time, and lets the parser tell it where the boundary is. The semicolon handling and the blank-line rule exist only for the two cases the parser cannot see: inline INSERT payloads and trailing comments.

The part people get wrong: INSERT ... FORMAT CSV data is not terminated by a semicolon, and the client will happily insert the ; as data. If your loader produces a file with a trailing semicolon after the last row, the semicolon is now a column value. Put a blank line after the payload, or use VALUES.

ClickHouse Parses URLs with a Hand-Written State Machine

Jay Hubbert — Thu, 22 Oct 2026 09:00:00 +0000

TL;DR: ClickHouse has 27 registered URL functions and none of them uses a regular expression. They share a hand-written byte scanner built on a SIMD find_first_symbols, plus a token-generator protocol (set() / get()) that lets one state machine serve URLHierarchy, URLPathHierarchy, splitByString and friends.

Web analytics is a common ClickHouse workload: a table of request logs, and you want to slice it by host, by path, by query parameter. So ClickHouse ships a family of URL functions:

 sql
 
 
SELECT domain(url), path(url), queryString(url), topLevelDomain(url),
 firstSignificantSubdomain(url), extractURLParameter(url, 'utm_source')
FROM hits;

The obvious way to implement those is with a regular expression. domain would be ^https?://([^/:?#]+), path would be ^[^:]+://[^/]+(/[^?#]*)?, and so on. Every one of them, plus a dozen variants. That approach has two problems.

First, regexes are slow in bulk. RE2 is not a slow engine, but it is a general-purpose one: it compiles the pattern to a program, runs a finite automaton over the input, and pays for capture-group bookkeeping that most of these functions do not need. A domain() call that just needs to find the third / pays for a full regexp engine.

Second, and more subtly, the semantics are wrong. A regex expresses “find a substring shaped like this”. A URL parser has to express “this is the host of this URL”, which means dealing with userinfo (user@host), port separators, IPv6 literals in brackets ([::1]:8080), and the fact that @ can appear in a userinfo section and that a second @ invalidates the first one’s interpretation. Those are not regular in the regex sense.

ClickHouse’s answer is to write the scanner by hand. Here is what that looks like.

The shape of the family

The functions live in src/Functions/URL/. The directory listing is essentially the API surface:

 text
 
 
src/Functions/URL/
├── URLHierarchy.cpp URLHierarchy(url) -> Array(String)
├── URLPathHierarchy.cpp URLPathHierarchy(url) -> Array(String)
├── basename.cpp basename(path) -> String

├── cutFragment.cpp cutFragment(url) -> String
├── cutQueryString.cpp cutQueryString(url) -> String
├── cutQueryStringAndFragment.cpp
├── cutToFirstSignificantSubdomain.cpp
├── cutToFirstSignificantSubdomainCustom.cpp
├── cutURLParameter.cpp cutURLParameter(url, name) -> String
├── cutWWW.cpp cutWWW(url) -> String
├── decodeURLComponent.cpp decodeURLComponent(s) -> String
├── domain.cpp domain(url) -> String
├── domainWithoutWWW.cpp
├── extractURLParameter.cpp extractURLParameter(url, name) -> String
├── extractURLParameterNames.cpp
├── extractURLParameters.cpp
├── firstSignificantSubdomain.cpp
├── firstSignificantSubdomainCustom.cpp
├── fragment.cpp fragment(url) -> String
├── netloc.cpp netloc(url) -> String
├── path.cpp path(url) -> String
├── pathFull.cpp
├── port.cpp port(url) -> String
├── protocol.cpp protocol(url) -> String
├── queryString.cpp queryString(url) -> String
├── queryStringAndFragment.cpp
├── topLevelDomain.cpp topLevelDomain(url) -> String
└── tldLookup.generated.cpp (gperf-generated TLD table)

Note that only a few of these have a .cpp with the implementation in it. Most of the logic is in headers, shared between functions. The reason is that almost all of them are the same function with different parameters: domain and domainWithoutWWW are one template; queryString and queryStringAndFragment are one template; topLevelDomain and topLevelDomainRFC are one template.

One warning before quoting any of it: in this tree the two hierarchy implementation files have swapped contents relative to their names. src/Functions/URL/URLHierarchy.cpp contains URLPathHierarchyImpl and registers URLPathHierarchy; src/Functions/URL/URLPathHierarchy.cpp contains URLHierarchyImpl and registers URLHierarchy. The line numbers below follow the files, not the function names.

The template parameters are bools, and the whole family is generated from a handful of Extractor structs with a execute(data, size, res_data, res_size) signature.

The extraction protocol

Every extractor implements the same four static members:

 cpp
 
 
// src/Functions/URL/path.h:10
template <bool with_query_string>
struct ExtractPath
{
 static size_t getReserveLengthForElement() { return 25; }

 static void execute(Pos data, size_t size, Pos & res_data, size_t & res_size)
 {
 res_data = data;
 res_size = 0;

 Pos pos = data;
 Pos end = pos + size;

 /// We support URLs with and without schema:
 /// 1. http://host/path
 /// 2. host/path
 /// We search for first slash and if there is subsequent slash, then skip and repeat search for the next slash.

 pos = find_first_symbols<'/'>(pos, end);
 if (end == pos)
 return;

 bool has_subsequent_slash = pos + 1 < end && pos[1] == '/';
 if (has_subsequent_slash)
 {
 /// Search for next slash.
 pos = find_first_symbols<'/'>(pos + 2, end);
 if (end == pos)
 return;
 }

 res_data = pos;

 if constexpr (with_query_string)
 {
 res_size = end - res_data;
 }
 else
 {
 Pos query_string_or_fragment = find_first_symbols<'?', '#'>(pos, end);
 res_size = query_string_or_fragment - res_data;
 }
 }
};

Three things to notice.

res_data / res_size is an out-parameter pair, not a string_view. The extractor returns a pointer into the input buffer and a length. No allocation, no copy. The caller (ExtractSubstringImpl) then does the copy into the result column.

The if constexpr (with_query_string) branch. path() and pathFull() are the same scanner with a different answer for the tail. pathFull includes the query string, path does not. if constexpr means the dead branch is not even instantiated — no runtime flag, no branch in the inner loop.

The “first slash, and if the next one is a slash, skip two” rule. This is the entire scheme-vs-no-scheme discrimination, in four lines:

http://host/path — the first / is at index 5, the next char is /, so we skip to index 7 and find the next /, which is the start of the path.
host/path — the first / is at index 4, the next char is p, not /, so the first slash is the path.

That is a heuristic, not a parser — but it is a heuristic on a problem where a heuristic is what the users actually want, and it is exactly the kind of thing a regex would also get “wrong” in the same way.

The SIMD primitive

The scanning primitive is find_first_symbols, which lives in base/base/find_symbols.h. It is the URL functions’ whole performance story:

 cpp
 
 
// base/base/find_symbols.h:19
/** find_first_symbols<c1, c2, ...>(begin, end):
 *
 * Allow to search for next character from the set of 'symbols...' in a string.
 * It is similar to 'strpbrk', 'strcspn' (and 'strchr', 'memchr' in the case of one symbol and '\0'),
 * but with the following differences:
 * - works with any memory ranges, including containing zero bytes;
 * - doesn't require terminating zero byte: end of memory range is passed explicitly;
 * - if not found, returns pointer to end instead of nullptr;
 * - maximum number of symbols to search is 16.
 *
 * Uses SSE 2 in case of small number of symbols for search and SSE 4.2 in the case of large number of symbols,
 * that have more than 2x performance advantage over trivial loop
 * in the case of parsing tab-separated dump with (probably escaped) string fields.
 * In the case of parsing tab separated dump with short strings, there is no performance degradation over trivial loop.
 *
 * Note: the optimal threshold to choose between SSE 2 and SSE 4.2 may depend on CPU model.

Three properties matter for the URL use case:

No terminating NUL required. URL strings in a ColumnString are length-prefixed slices of one big buffer; there is no NUL at the end of the slice.
Returns end rather than nullptr when not found. This makes the caller’s “did I find it” check a single pointer comparison, and it removes the need for a separate “found” flag.
Up to 16 symbols at once. So find_first_symbols<'?', '#'> finds the start of the query string or the fragment in one pass — the two characters that can end a path.

And it is SIMD: SSE2 on x86, NEON on aarch64.

The vectorised driver

The extractors are called from a shared driver that handles the column-at-a-time plumbing. Here is the whole of ExtractSubstringImpl::vector:

 cpp
 
 
// src/Functions/StringHelpers.h:62
template <typename Extractor>
struct ExtractSubstringImpl
{
 static void vector(
 const ColumnString::Chars & data, const ColumnString::Offsets & offsets,
 ColumnString::Chars & res_data, ColumnString::Offsets & res_offsets,
 size_t input_rows_count)
 {
 res_offsets.resize(input_rows_count);
 res_data.reserve(input_rows_count * Extractor::getReserveLengthForElement());

 size_t prev_offset = 0;
 size_t res_offset = 0;

 /// Matched part.
 Pos start = nullptr;
 size_t length = 0;

 for (size_t i = 0; i < input_rows_count; ++i)
 {
 Extractor::execute(reinterpret_cast<const char *>(&data[prev_offset]), offsets[i] - prev_offset, start, length);

 res_data.resize(res_data.size() + length);
 memcpySmallAllowReadWriteOverflow15(&res_data[res_offset], start, length);
 res_offset += length;
 res_offsets[i] = res_offset;
 prev_offset = offsets[i];
 }
 }

The getReserveLengthForElement() call is a per-extractor hint about the typical output length — 25 for a path, 15 for a domain, 10 for a query string, 5 for a top-level domain, 10 for a fragment. It is used to size the reserve() on the result buffer, which avoids repeated reallocation.

The memcpySmallAllowReadWriteOverflow15 is ClickHouse’s small-copy helper: for lengths up to 15 bytes it uses a branchy inline copy, and otherwise it calls memcpy. Since a typical extracted URL part is short, the inline path dominates.

Note also that the loop maintains prev_offset rather than reading offsets[i-1]: the offsets array is monotonically increasing, so the previous offset is the current start.

The authority: the hard part

path is easy — find a slash. domain is where it gets interesting, because the host of a URL is delimited by several different characters depending on the context, and the RFC has a specific grammar for it.

ClickHouse provides two variants. getURLHost is the legacy one, and getURLHostRFC is the RFC-3986-conformant one. They differ substantially. Here is the RFC one’s opening:

 cpp
 
 
// src/Functions/URL/domain.h:59
inline std::string_view getURLHostRFC(const char * data, size_t size)
{
 if (size < 2)
 return std::string_view{};

 Pos pos = data;
 Pos end = data + size;

 if (*pos == '/' && *(pos + 1) == '/')
 {
 pos += 2;
 }
 else if (isAlphaASCII(*pos))
 {
 Pos scheme_end = data + std::min(size, 16UL);
 for (++pos; pos < scheme_end; ++pos)
 {
 if (!isAlphaNumericASCII(*pos))
 {
 switch (*pos)
 {
 case '.':
 case '-':
 case '+':
 break;
 case ' ': /// restricted symbols
 case '\t':
 case '<':
 case '>':
 case '%':
 case '{':
 case '}':
 case '|':
 case '\\':
 case '^':
 case '~':
 case ']':
 case ';':
 case '=':
 case '&':
 return std::string_view{};
 default:
 goto exloop;
 }
 }
 }
exloop: if ((scheme_end - pos) > 2 && *pos == ':' && *(pos + 1) == '/' && *(pos + 2) == '/')
 pos += 3;
 else
 pos = data;
 }

The goto exloop is a deliberate escape from the nested loop: the scheme scan finds the first character that cannot be part of a scheme, and if that character is : followed by //, we have a scheme and skip past it; otherwise the whole thing is reset to data (i.e. “there was no scheme, the host starts at the beginning”).

Note std::min(size, 16UL): the scheme scan is bounded at 16 bytes. An RFC scheme can be longer, but capping it keeps a malformed input like aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.com/x from costing a long scan, and because the fallback is “no scheme”, the cap is safe.

Then the authority scan proper — this is the state machine:

 cpp
 
 
// src/Functions/URL/domain.h:111
 bool has_open_bracket = false;
 bool has_end_bracket = false;
 if (pos < end && *pos == '[') /// IPv6 [2001:db8::1]:80
 {
 has_open_bracket = true;
 ++pos;
 }
 Pos bracket_close_pos = nullptr;
 Pos dot_pos = nullptr;
 Pos colon_pos = nullptr;
 bool has_sub_delims = false;
 bool has_at_symbol = false;
 bool has_terminator_after_colon = false;
 const auto * start_of_host = pos;
 for (; pos < end; ++pos)
 {
 switch (*pos) /// NOLINT(bugprone-switch-missing-default-case)
 {
 case '.':
 if (has_open_bracket)
 continue; /// part of a mixed IPv6/IPv4 tail, e.g. "::ffff:192.0.2.128"; parseIPv6Whole validates it below
 /// Once colon_pos is set (whether or not '@' has been seen), a dot afterward is past
 /// the host, in what would be the port - it must not count as a dot within the host,
 /// e.g. the '.' in "user@foo:80.bar" must not make checkAndReturnHost think "foo" has
 /// a valid dot of its own.
 if (colon_pos == nullptr)
 dot_pos = pos;
 break;
 case ':':
 if (has_open_bracket)
 continue;
 /// Whether or not '@' has been seen yet, this ':' might still turn out to be
 /// followed by a later '@' (e.g. "user@host:80@evil.com"), which would mean it was
 /// never the port separator at all - keep only the first one as a fallback and keep
 /// scanning, instead of stopping the scan here.
 if (colon_pos == nullptr)
 colon_pos = pos;
 break;
 case '/': /// end symbols
 case '?':
 case '#':
 goto done;
 case '@': /// myemail@gmail.com
 /// Inside an IP-literal there is no userinfo: `@` is not allowed there at all.
 if (has_open_bracket) return std::string_view{};
 /// A bracket that already closed before this '@' would have to become part of
 /// userinfo now that '@' turned up - but raw '[' / ']' can never legally appear
 /// there, so an authority like "[::1]:80@evil.com" is invalid, not "host evil.com".
 if (has_end_bracket) return std::string_view{};
 if (has_terminator_after_colon) return std::string_view{};
 if (has_at_symbol) return std::string_view{};
 has_sub_delims = false;
 has_at_symbol = true;
 start_of_host = pos + 1;
 colon_pos = nullptr;
 dot_pos = nullptr;
 has_terminator_after_colon = false;

The state is six booleans plus three remembered positions. The interesting invariant is the has_at_symbol one: a second @ invalidates the authority entirely. The URL http://user@host@evil.com/ is not a URL with userinfo user and host host@evil.com; it is malformed. The scanner detects this and returns an empty host.

That check exists because of a security consideration. A naive parser that takes the last @ as the separator would report evil.com as the host of http://trusted.com@evil.com/ — the classic userinfo-confusion vulnerability. Taking the first @ would report trusted.com@evil.com as the host, which is also wrong but less dangerous. Refusing outright is the safe answer.

The IPv6 handling is the other interesting part. A host can be an IP-literal in brackets: [2001:db8::1]:8080. Inside the brackets, : is part of the address, not a port separator, so the scanner flips into has_open_bracket mode and ignores colons and dots. It also tracks whether the bracket has closed, because after ] a : is an ordinary port separator again.

And it supports RFC 3986’s IPvFuture form, [v1.a], via a helper:

 cpp
 
 
// src/Functions/URL/domain.h:15
inline bool isIPvFuture(const char * begin, const char * end)
{
 if (begin >= end || (*begin != 'v' && *begin != 'V'))
 return false;

 const char * pos = begin + 1;
 const char * version_start = pos;
 while (pos < end && isHexDigit(*pos))
 ++pos;
 if (pos == version_start)
 return false;

 if (pos >= end || *pos != '.')
 return false;
 ++pos;

 const char * address_start = pos;
 for (; pos < end; ++pos)
 {
 char c = *pos;
 bool is_unreserved = isAlphaNumericASCII(c) || c == '-' || c == '.' || c == '_' || c == '~';
 bool is_sub_delim = c == '!' || c == '$' || c == '&' || c == '\'' || c == '(' || c == ')'
 || c == '*' || c == '+' || c == ',' || c == ';' || c == '=';
 if (!is_unreserved && !is_sub_delim && c != ':')
 return false;
 }
 return pos > address_start;
}

Finally, the host is only accepted if it has a dot and a character after it:

 cpp
 
 
// src/Functions/URL/domain.h:44
inline std::string_view checkAndReturnHost(const Pos & pos, const Pos & dot_pos, const Pos & start_of_host)
{
 if (!dot_pos || start_of_host >= pos || pos - dot_pos == 1 || *start_of_host == '.')
 return std::string_view{};

 auto after_dot = *(dot_pos + 1);
 if (after_dot == ':' || after_dot == '/' || after_dot == '?' || after_dot == '#')
 return std::string_view{};

 return std::string_view(start_of_host, pos - start_of_host);
}

“Has a dot, has something after the dot, and does not start with a dot” — the classic rule for “this looks like a hostname, not a bare word”. This is why domain('localhost') returns an empty string: localhost has no dot, so it is not a domain by this definition.

Note the after_dot check: foo.:8080 is rejected because the character after the dot is :. And foo. (dot at the end) is rejected by the pos - dot_pos == 1 check.

The token generator: `URLHierarchy`

URLHierarchy is the odd one out. It does not extract a single substring — it returns an array of progressively longer prefixes:

 sql
 
 
SELECT URLHierarchy('https://example.com/a/b?c=1');
-- ['https://example.com/','https://example.com/a/','https://example.com/a/b?','https://example.com/a/b?c=1']

Producing an array of N substrings means the result is an Array(String), and the implementation cannot use the ExtractSubstringImpl driver (which produces one substring per row). So ClickHouse invented a generator protocol to share the scanning logic.

The protocol is: init() once, then set(begin, end) per input string, then repeated get(token_begin, token_end) calls that yield one token at a time and return false when exhausted. URLPathHierarchyImpl implements it:

 cpp
 
 
// src/Functions/URL/URLHierarchy.cpp:51
 /// Get the next token, if any, or return false.
 bool get(Pos & token_begin, Pos & token_end)
 {
 /// Code from URLParser.

 if (pos == end)
 return false;

 if (pos == begin)
 {
 /// Let's parse everything that goes before the path

 /// Assume that the protocol has already been changed to lowercase.
 while (pos < end && ((*pos >= 'a' && *pos <= 'z') || (*pos >= '0' && *pos <= '9')))
 ++pos;

 /** We will calculate the hierarchy only for URLs in which there is a protocol, and after it there are two slashes.
 * (http, file - fit, mailto, magnet - do not fit), and after two slashes still at least something is there.
 * For the rest, just return an empty array.
 */
 if (pos == begin || pos == end || !(pos + 3 < end && pos[0] == ':' && pos[1] == '/' && pos[2] == '/'))
 {
 pos = end;
 return false;
 }
 pos += 3;

 /// The domain for simplicity is everything that after the protocol and the two slashes, until the next slash or `?` or `#`
 while (pos < end && !(*pos == '/' || *pos == '?' || *pos == '#'))
 ++pos;

 start = pos;

 if (pos != end)
 ++pos;
 }

 /// We go to the next `/` or `?` or `#`, skipping all those at the beginning.
 while (pos < end && (*pos == '/' || *pos == '?' || *pos == '#'))
 ++pos;
 if (pos == end)
 return false;
 while (pos < end && !(*pos == '/' || *pos == '?' || *pos == '#'))
 ++pos;

 if (pos != end)
 ++pos;

 token_begin = start;
 token_end = pos;

 return true;
 }

The state machine has two phases. The first get() call (when pos == begin) consumes the scheme and the authority and sets start to the beginning of the path. Subsequent calls walk the path, emitting one token per segment.

And URLHierarchy is the same protocol with one difference — it emits the scheme+authority as the first token:

 cpp
 
 
// src/Functions/URL/URLPathHierarchy.cpp:83
 token_begin = begin;
 token_end = pos;

 return true;
 }

Compare with URLPathHierarchy, which sets start = pos (the path) and leaves token_begin = start. Two nearly-identical 50-line state machines, differing in three lines, because the protocol is data, not code.

The protocol is generic enough that FunctionTokens — the shared higher-order driver in src/Functions/FunctionTokens.h — implements the column-at-a-time loop for any generator:

 cpp
 
 
// src/Functions/FunctionTokens.h:107
 ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & result_type, size_t input_rows_count) const override
 {
 Generator generator{};
 /// Only generators that opt in (currently `ExtractAllImpl`) take the JIT compile-count threshold.
 if constexpr (requires { generator.init(arguments, max_substrings_includes_remaining_string, regexp_jit_min_count); })
 generator.init(arguments, max_substrings_includes_remaining_string, regexp_jit_min_count);
 else
 generator.init(arguments, max_substrings_includes_remaining_string);

 const auto & array_argument = arguments[generator.strings_argument_position];

 cpp
 
 
// src/Functions/FunctionTokens.h:145
 for (size_t i = 0; i < input_rows_count; ++i)
 {
 Pos pos = reinterpret_cast<Pos>(&src_chars[current_src_offset]);
 current_src_offset = src_offsets[i];
 Pos end = reinterpret_cast<Pos>(&src_chars[current_src_offset]);

 generator.set(pos, end);
 size_t j = 0;
 while (generator.get(token_begin, token_end))
 {
 chassert(token_begin >= pos && token_end >= token_begin);
 chassert(token_end <= end);
 size_t token_size = token_end - token_begin;

 res_strings_chars.resize(res_strings_chars.size() + token_size);
 memcpySmallAllowReadWriteOverflow15(&res_strings_chars[current_dst_strings_offset], token_begin, token_size);

 current_dst_strings_offset += token_size;
 res_strings_offsets.push_back(current_dst_strings_offset);
 ++j;
 }

 current_dst_offset += j;
 res_offsets.push_back(current_dst_offset);
 }

So FunctionTokens<Generator> is a higher-order function over the generator protocol, and it serves:

URLHierarchy
URLPathHierarchy
splitByString, splitByChar, splitByWhitespace, splitByNonAlpha
alphaTokens
extractAll (which is a regexp generator, and which also opts into the regexp JIT threshold)

That is a genuine framework abstraction, not a copy-paste: the token plumbing, the offsets arithmetic, the small-copy helper, and the constant column fast path are all written once.

`extractURLParameter`: a different problem

Not every URL function fits the “find a delimiter” shape. extractURLParameter(url, 'utm_source') has to find a named parameter in the query string, which means a substring search plus a boundary check. It is implemented separately:

 cpp
 
 
// src/Functions/URL/extractURLParameter.cpp:9
struct ExtractURLParameterImpl
{
 static void vector(const ColumnString::Chars & data,
 const ColumnString::Offsets & offsets,
 std::string pattern,
 ColumnString::Chars & res_data, ColumnString::Offsets & res_offsets,
 size_t input_rows_count)
 {
 res_data.reserve(data.size() / 5);
 res_offsets.resize(input_rows_count);

 pattern += '=';
 const char * param_str = pattern.data();
 size_t param_len = pattern.size();

 ColumnString::Offset prev_offset = 0;
 ColumnString::Offset res_offset = 0;

 for (size_t i = 0; i < input_rows_count; ++i)
 {
 ColumnString::Offset cur_offset = offsets[i];

 const char * str = reinterpret_cast<const char *>(&data[prev_offset]);
 const char * end = reinterpret_cast<const char *>(&data[cur_offset]);

 /// Find query string or fragment identifier.
 /// Note that we support parameters in fragment identifier in the same way as in query string.

 const char * const query_string_begin = find_first_symbols<'?', '#'>(str, end);

 /// Will point to the beginning of "name=value" pair. Then it will be reassigned to the beginning of "value".
 const char * param_begin = nullptr;

 if (query_string_begin + 1 < end)
 {
 param_begin = query_string_begin + 1;

 while (true)
 {
 param_begin = static_cast<const char *>(memmem(param_begin, end - param_begin, param_str, param_len));

 if (!param_begin)
 break;

 char prev_char = param_begin[-1];
 if (prev_char != '?' && prev_char != '#' && prev_char != '&')
 {
 /// Parameter name is different but has the same suffix.
 param_begin += param_len;
 continue;
 }

 param_begin += param_len;
 break;
 }
 }

The trick is that the parameter name is turned into a search key by appending =: the code searches for "utm_source=" with memmem (which is itself SIMD-optimised in glibc), and then checks the preceding character to make sure it is a real parameter boundary (?, #, or &) and not the tail of a longer name like notutm_source=.

So the same find_first_symbols primitive handles the delimiter case and memmem handles the substring case, and the boundary check is the extra logic that a regex like [?&]utm_source=([^&#]*) would have given for free.

`firstSignificantSubdomain` and the TLD table

firstSignificantSubdomain needs to know what a top-level domain is, which means a table. ClickHouse generates one with gperf:

 bash
 
 
# src/Functions/URL/tldLookup.sh
#!/usr/bin/env bash

[ ! -f public_suffix_list.dat ] && wget -nv -O public_suffix_list.dat https://publicsuffix.org/list/public_suffix_list.dat

echo '%language=C++
%define lookup-function-name isValid
%define class-name TopLevelDomainLookupHash
%readonly-tables
%includes
%compare-strncmp
%{
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wimplicit-fallthrough"
#pragma GCC diagnostic ignored "-Wzero-as-null-pointer-constant"
#pragma GCC diagnostic ignored "-Wunused-macros"
// NOLINTBEGIN(modernize-macro-to-enum)
%}
# List generated using https://publicsuffix.org/list/public_suffix_list.dat
%%' > tldLookup.gperf
grep -v "//" public_suffix_list.dat | grep . | grep "\." | grep -ve "\..*\..*" >> tldLookup.gperf
echo "%%" >> tldLookup.gperf
echo '// NOLINTEND(modernize-macro-to-enum)' >> tldLookup.gperf

gperf turns the public suffix list into a perfect hash — a static array with no runtime search. The generated file is 110 742 lines, which is why it is checked in rather than built.

The lookup then drives the subdomain extraction:

 cpp
 
 
// src/Functions/URL/ExtractFirstSignificantSubdomain.h:30
 template <class Lookup>
 static void execute(const Lookup & lookup, const Pos data, const size_t size, Pos & res_data, size_t & res_size, Pos * out_domain_end = nullptr)
 {
 res_data = data;
 res_size = 0;

 Pos tmp = nullptr;
 size_t domain_length = 0;
 ExtractDomain<without_www, conform_rfc>::execute(data, size, tmp, domain_length);

 if (domain_length == 0)
 return;

The algorithm walks backwards through the last three dots, and asks the TLD lookup whether the candidate is a registered suffix:

 cpp
 
 
// src/Functions/URL/ExtractFirstSignificantSubdomain.h:84
 const auto * end_of_level_domain = find_first_symbols<'/'>(last_periods[0], end);
 if (!end_of_level_domain)
 {
 end_of_level_domain = end;
 }

 size_t host_len = static_cast<size_t>(end_of_level_domain - last_periods[1] - 1);
 std::string_view host{last_periods[1] + 1, host_len};
 if (lookup(host))
 {
 res_data += last_periods[2] + 1 - begin;
 res_size = last_periods[1] - last_periods[2] - 1;
 }
 else
 {
 res_data += last_periods[1] + 1 - begin;
 res_size = last_periods[0] - last_periods[1] - 1;
 }

For www.foo.example.co.uk, the last three periods bracket example.co.uk; if that is a registered suffix, the significant subdomain is foo, otherwise it is example.

The custom variant (firstSignificantSubdomainCustom) is different because a user-supplied TLD list can have records at any level, so the lookup returns a type rather than a boolean and the scan has to handle TLD_NONE, TLD_REGULAR, TLD_ANY and TLD_EXCLUDE:

 cpp
 
 
// src/Functions/URL/ExtractFirstSignificantSubdomain.h:150
 TLDType tld_type = lookup(host);
 switch (tld_type)
 {
 case TLDType::TLD_NONE:
 break;
 case TLDType::TLD_REGULAR:
 res_data += last_periods[0] + 1 - begin;
 res_size = end - 1 - last_periods[0];
 return;
 case TLDType::TLD_ANY:
 {
 std::string_view regular_host{last_periods[0] + 1, static_cast<size_t>(end - 1 - last_periods[0])};
 if (last_periods[1] && excluded_host != regular_host)
 {
 /// Return TLD_REGULAR + 1
 res_data += last_periods[1] + 1 - begin;
 res_size = end - 1 - last_periods[1];
 }
 else
 {
 /// Same as TLD_REGULAR
 res_data += last_periods[0] + 1 - begin;
 res_size = end - 1 - last_periods[0];
 }
 return;
 }
 case TLDType::TLD_EXCLUDE:
 excluded_host = host;
 break;
 }

TLD_ANY and TLD_EXCLUDE come from the public suffix list’s wildcard and exception rules (*.ck, !www.ck). This is the public suffix list model implemented as a state machine over a suffix hash.

Two host parsers, two behaviours

One last detail worth calling out, because it explains why there are two domain functions. The legacy getURLHost is much simpler — it does not understand userinfo or IPv6:

 cpp
 
 
// src/Functions/URL/domain.h:250
inline std::string_view getURLHost(const char * data, size_t size)
{
 Pos pos = data;
 Pos end = data + size;

 if (size >= 2 && *pos == '/' && *(pos + 1) == '/')
 {
 pos += 2;
 }
 else
 {
 Pos scheme_end = data + std::min(size, 16UL);
 for (++pos; pos < scheme_end; ++pos)
 {
 if (!isAlphaNumericASCII(*pos))
 {
 switch (*pos)
 {
 case '.':
 case '-':
 case '+':
 break;
 case ' ': /// restricted symbols
 case '\t':
 case '<':
 case '>':
 case '%':
 case '{':
 case '}':
 case '|':
 case '\\':
 case '^':
 case '~':
 case '[':
 case ']':
 case ';':
 case '=':
 case '&':
 return std::string_view{};
 default:
 goto exloop;
 }
 }
 }
exloop: if ((scheme_end - pos) > 2 && *pos == ':' && *(pos + 1) == '/' && *(pos + 2) == '/')
 pos += 3;
 else
 pos = data;
 }

 Pos dot_pos = nullptr;
 const auto * start_of_host = pos;
 for (; pos < end; ++pos)
 {
 switch (*pos) /// NOLINT(bugprone-switch-missing-default-case)
 {
 case '.':
 dot_pos = pos;
 break;
 case ':': /// end symbols
 case '/':
 case '?':
 case '#':
 return checkAndReturnHost(pos, dot_pos, start_of_host);
 case '@': /// myemail@gmail.com
 start_of_host = pos + 1;
 break;

Note the difference in the @ handling: the legacy version takes the last @ (it keeps updating start_of_host), while the RFC version refuses a second @ outright. Every function in the family comes in a legacy and an RFC flavour, and the RFC flavour is a separate registered name:

 cpp
 
 
// src/Functions/URL/domain.cpp:11
struct NameDomainRFC { static constexpr auto name = "domainRFC"; };
using FunctionDomainRFC = FunctionStringToString<ExtractSubstringImpl<ExtractDomain<false, true>>, NameDomainRFC>;

So the difference is visible at the call site:

 sql
 
 
SELECT domain('http://user@host@evil.com/'); -- 'evil.com' (legacy)
SELECT domainRFC('http://user@host@evil.com/'); -- '' (RFC)

The same pairing exists for domainWithoutWWW / domainWithoutWWWRFC, topLevelDomain / topLevelDomainRFC, firstSignificantSubdomain / firstSignificantSubdomainRFC, cutToFirstSignificantSubdomain / cutToFirstSignificantSubdomainRFC, and firstSignificantSubdomainCustom / firstSignificantSubdomainCustomRFC.

The topLevelDomain pair shows the difference concretely:

 cpp
 
 
// src/Functions/URL/topLevelDomain.cpp:56
struct NameTopLevelDomain { static constexpr auto name = "topLevelDomain"; };
using FunctionTopLevelDomain = FunctionStringToString<ExtractSubstringImpl<ExtractTopLevelDomain<false>>, NameTopLevelDomain>;

struct NameTopLevelDomainRFC { static constexpr auto name = "topLevelDomainRFC"; };
using FunctionTopLevelDomainRFC = FunctionStringToString<ExtractSubstringImpl<ExtractTopLevelDomain<true>>, NameTopLevelDomainRFC>;

So topLevelDomain uses the legacy host parser and topLevelDomainRFC uses the RFC one, and the difference is visible in the documented example:

 sql
 
 
SELECT topLevelDomain('http://foo:foo%41bar@foo.com'), topLevelDomainRFC('http://foo:foo%41bar@foo.com');
-- '' | 'com'

The legacy parser bails on %41 (a percent-encoded character in the userinfo), while the RFC parser handles the userinfo properly and reports the host.

What the design buys and what it costs

The hand-written approach buys:

Speed. A URL field extraction is a couple of SIMD scans of the input and a memcpy of the result. There is no regexp program, no capture bookkeeping, no per-match allocation.
Correctness on the cases that matter. Userinfo confusion, IPv6 literals, IPvFuture, second-@ rejection — these are the cases where “parse like a browser” and “match like a regex” diverge, and where the divergence has security consequences.
Zero allocation in the common path. The extractors return pointers into the source buffer; only the final result copy allocates.
Composability. The set()/get() generator protocol lets one scanner serve both “give me one field” and “give me a hierarchy of prefixes”, and the same protocol serves the splitBy* family and the regexp-based extractAll.

It costs:

Two implementations of the same idea. getURLHost and getURLHostRFC are separate 150-line functions with different behaviour, and the user-visible difference is a documented example. A regex-based implementation would have had one.
Manual edge-case handling. Every case in the switch is a decision somebody had to make about an ambiguous input. The comments are full of them: the second-@ rule, the dot-after-colon rule, the bracket-closed-then-@ rule. Each is a bug that was found and fixed.
No compositionality with the rest of the SQL. A regex-based implementation would let you write your own pattern against the URL; here you get exactly the twenty-seven functions ClickHouse decided to provide.
The goto exloop pattern. It works, but it is a loop-exit mechanism that has to be understood by anyone modifying the scheme detection.

The lesson

The interesting thing about this code is that it is boring in the best sense: it is a hand-written scanner with a small state machine, sitting on top of a SIMD primitive, wired into a columnar execution model through a generic protocol.

The design decision worth copying is the generator protocol. Rather than write twenty-seven near-identical functions, ClickHouse defined a four-method interface (init, set, get, plus checkArguments) and wrote the column-at-a-time driver once. URLHierarchy and URLPathHierarchy differ by three lines; splitByString and extractAll differ by an entire parsing strategy but share the driver.

The second decision worth copying is the willingness to write two implementations when the semantics genuinely differ, and to document the difference with a worked example, rather than pick one and lose users.