How ClickHouse Recompresses a Part When Its TTL Expires

TL;DR: TTL expr RECOMPRESS CODEC(ZSTD(17)) is a scheduled, whole-part rewrite with a different codec. The TTL expression is evaluated at insert time and only its maximum is kept; when every row of a part has passed that maximum, the merge selector picks the part, the merge writes it out with the new codec, and the codec is written into the part’s own metadata so the next comparison knows whether a rewrite would actually change anything.

Compression is a bet on the data. A column of timestamps and a column of free-text URLs want opposite codecs: the first wants Delta, ZSTD, the second wants LZ4 or nothing at all. ClickHouse normally resolves this once, at write time, and the choice is then baked into the part forever.

But a table’s data is not homogeneous over its lifetime. The recent partition — last week’s events, still being written — has a very different entropy profile from the archive partition, which was written a year ago and will never change again. A codec chosen for the recent data is wrong for the archive, and vice versa. The usual remedy is a manual OPTIMIZE with a CODEC clause, or a migration, or accepting the waste.

The RECOMPRESS TTL makes it automatic and, more importantly, scheduled: the rule says “after this date, this part should be stored with that codec”, and the background merger honours it the same way it honours a delete TTL. This article follows the whole mechanism: how the rule is parsed and validated, how the per-part expiry bound is computed, how the merge selector decides that a rewrite is worth it, how the codec reaches the column streams, and how the part records the codec it was written with.

The syntax

The TTL grammar has an optional action per expression, and RECOMPRESS is one of them, sitting next to DELETE, TO DISK and TO VOLUME:

cpp
// src/Parsers/ExpressionElementParsers.cpp:3007
    else if (s_group_by.ignore(pos, expected))
    {
        mode = TTLMode::GROUP_BY;
    }
    else if (s_recompress.ignore(pos, expected))
    {
        mode = TTLMode::RECOMPRESS;
    }
    else
    {
        /// DELETE is the default mode.
        s_delete.ignore(pos, expected);
        mode = TTLMode::DELETE;
    }

and the codec itself is parsed by the ordinary codec parser, so anything that is a valid CODEC(...) in a column declaration is a valid codec here:

cpp
// src/Parsers/ExpressionElementParsers.cpp:3058
    else if (mode == TTLMode::RECOMPRESS)
    {
        if (!s_codec.ignore(pos, expected))
            return false;

        if (!parser_codec.parse(pos, recompression_codec, expected))
            return false;
    }

The documented shape of a table TTL is therefore:

sql
-- src/Storages/MergeTree/registerStorageMergeTree.cpp:2060
TTL expr
    [DELETE|RECOMPRESS codec_name1|TO DISK 'xxx'|TO VOLUME 'xxx'][, DELETE|RECOMPRESS codec_name2|TO DISK 'aaa'|TO VOLUME 'bbb'] ...
    [WHERE conditions]
    [GROUP BY key_expr [SET v1 = aggr_func(v1) [, v2 = aggr_func(v2) ...]] ]

and a complete example from the engine’s own documentation:

sql
-- src/Storages/MergeTree/registerStorageMergeTree.cpp:2125
CREATE TABLE table_for_recompression
(
    d DateTime,
    key UInt64,
    value String
) ENGINE MergeTree()
ORDER BY tuple()
PARTITION BY key
TTL d + INTERVAL 1 MONTH RECOMPRESS CODEC(ZSTD(17)), d + INTERVAL 1 YEAR RECOMPRESS CODEC(LZ4HC(10))
SETTINGS min_rows_for_wide_part = 0, min_bytes_for_wide_part = 0;

Note the sentence in the same documentation block that defines the semantics precisely (src/Storages/MergeTree/registerStorageMergeTree.cpp:2058):

For parts moving or recompressing, all rows of a part must satisfy the TTL expression criteria.

This is not the semantics of a row TTL. A row TTL fires per row; a recompression TTL fires per part, and only once every row of the part has passed the expression. Recompression is a statement about a part, not about a row, because the codec is a property of the part’s column streams.

Validation

The codec is validated at CREATE time like any other codec, with one relaxation: on ATTACH the checks are relaxed, so that a table created by an older version still loads even if its codec would now be rejected:

cpp
// src/Storages/TTLDescription.cpp:1611
        else if (ttl_element->mode == TTLMode::RECOMPRESS)
        {
            /// On `ATTACH` (loading stored metadata) the codec checks are relaxed the same way column codecs are:
            /// a table created on an earlier version must still load even if its recompression codec would now be
            /// rejected at `CREATE`, otherwise the server could fail to start after an upgrade. A create with
            /// `allow_suspicious_ttl_expressions` also skips them, matching `checkTTLExpression` below.
            result.recompression_codec =
                CompressionCodecFactory::instance().validateCodecAndGetPreprocessedAST(
                    ttl_element->recompression_codec, {},
                    skip_validation ? CodecValidationSettings::trusted() : CodecValidationSettings(context->getSettingsRef()));
        }

The rules of a table are then sorted into buckets by their action — DELETE without a WHERE becomes the table’s single unconditional row TTL, DELETE with a WHERE becomes a conditional row TTL, RECOMPRESS rules go into their own list:

cpp
// src/Storages/TTLDescription.cpp:1701
        else if (ttl.mode == TTLMode::RECOMPRESS)
        {
            result.recompression_ttl.emplace_back(std::move(ttl));
        }
        else if (ttl.mode == TTLMode::GROUP_BY)
        {
            result.group_by_ttl.emplace_back(std::move(ttl));
        }
        else
        {
            result.move_ttl.emplace_back(std::move(ttl));
        }

The per-part bound

Like every other TTL, a recompression rule is evaluated at insert time by TTLCalcTransform, and the result is reduced to a min/max pair in the part’s ttl.txt. The rule is stored under its own map, keyed by the result column of the expression, and — importantly — it does not feed the part’s overall part_min_ttl / part_max_ttl:

cpp
// src/Processors/TTL/TTLUpdateInfoAlgorithm.cpp:35
void TTLUpdateInfoAlgorithm::finalize(const MutableDataPartPtr & data_part) const
{
    if (ttl_update_field == TTLUpdateField::RECOMPRESSION_TTL)
    {
        data_part->ttl_infos.recompression_ttl[ttl_update_key] = new_ttl_info;
    }
    else if (ttl_update_field == TTLUpdateField::MOVES_TTL)
    {
        data_part->ttl_infos.moves_ttl[ttl_update_key] = new_ttl_info;
    }

That is right: a recompression rule must never make a part look like it has expired rows, or the delete-TTL machinery would start rewriting it for the wrong reason. The recompression map is consulted only by the recompression selector.

Because the semantics are “all rows of the part must have expired”, the selector needs the maximum expiry of the part, and the part-level bound is the smallest of those maxima over all the rules:

cpp
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:235
time_t MergeTreeDataPartTTLInfos::getMinimalMaxRecompressionTTL() const
{
    time_t max = std::numeric_limits<time_t>::max();
    for (const auto & [name, info] : recompression_ttl)
        if (info.max != 0)
            max = std::min(info.max, max);

    if (max == std::numeric_limits<time_t>::max())
        return 0;

    return max;
}

So with two rules — one at one month, one at one year — the part becomes eligible for the first rule’s codec after one month (its max for the one-month rule has passed) and for the second rule’s codec after one year.

Choosing the rule

When several rules are due, the engine picks the one that “went into action last”, i.e. the one with the largest expiry that is still in the past:

cpp
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:333
std::optional<TTLDescription> selectTTLDescriptionForTTLInfos(const TTLDescriptions & descriptions, const TTLInfoMap & ttl_info_map, time_t current_time, bool use_max)
{
    time_t best_ttl_time = 0;
    TTLDescriptions::const_iterator best_entry_it;
    for (auto ttl_entry_it = descriptions.begin(); ttl_entry_it != descriptions.end(); ++ttl_entry_it)
    {
        auto ttl_info_it = ttl_info_map.find(ttl_entry_it->result_column);

        if (ttl_info_it == ttl_info_map.end())
        {
            ttl_info_it = tryToFindTTLExpressionInMapByASTMatching(ttl_info_map, ttl_entry_it->result_column);
            if (ttl_info_it == ttl_info_map.end())
                continue;
        }

        time_t ttl_time = 0;

        if (use_max)
            ttl_time = ttl_info_it->second.max;
        else
            ttl_time = ttl_info_it->second.min;

        /// Prefer TTL rule which went into action last.
        if (ttl_time <= current_time
                && best_ttl_time <= ttl_time)
        {
            best_entry_it = ttl_entry_it;
            best_ttl_time = ttl_time;
        }
    }

    return best_ttl_time ? *best_entry_it : std::optional<TTLDescription>();
}

(The tryToFindTTLExpressionInMapByASTMatching fallback exists because the map is keyed by a string rendering of the expression; expired + toIntervalSecond(20) and plus(expired, toIntervalSecond(20)) are the same rule, so the lookup parses both and compares tree hashes.)

A useful side effect of this function is isExplicitRecompression, which answers “is a non-default codec due right now?”:

cpp
// src/Storages/MergeTree/MergeTreeDataPartTTLInfo.cpp:367
bool isExplicitRecompression(
    const TTLDescriptions & recompression_ttl_entries, const TTLInfoMap & recompression_ttl_info, time_t current_time)
{
    auto best_ttl_entry
        = selectTTLDescriptionForTTLInfos(recompression_ttl_entries, recompression_ttl_info, current_time, /*use_max=*/true);
    return best_ttl_entry && !CompressionCodecFactory::isDefaultCodec(best_ttl_entry->recompression_codec);
}

“Would this rewrite change anything?”

A part that is already stored with ZSTD(17) must not be rewritten with ZSTD(17) again: the merge would read every column, recompress it identically, and write it back — burning I/O and CPU to produce a byte-identical part. So the selector compares the codec the rule wants with the codec the part has:

cpp
// src/Storages/MergeTree/Compaction/PartProperties.cpp:34
std::optional<PartProperties::RecompressTTLInfo> buildRecompressTTLInfo(StorageMetadataPtr metadata_snapshot, MergeTreeDataPartPtr part, time_t current_time)
{
    if (!metadata_snapshot->hasAnyRecompressionTTL())
        return std::nullopt;

    const auto & recompression_ttls = metadata_snapshot->getRecompressionTTLs();
    const auto ttl_description = selectTTLDescriptionForTTLInfos(recompression_ttls, part->ttl_infos.recompression_ttl, current_time, true);

    if (ttl_description)
    {
        /// If the part's own default codec could not be recovered exactly (see
        /// `IMergeTreeDataPart::default_codec_is_approximate`), the comparison below cannot be trusted
        /// either way: treat the codec as unknown and always let the merge selector reconsider the
        /// part, rather than risk a wrong guess suppressing a recompression that is still needed.
        if (part->default_codec_is_approximate)
            return PartProperties::RecompressTTLInfo{
                .will_change_codec = true,
                .next_recompress_ttl = part->ttl_infos.getMinimalMaxRecompressionTTL(),
            };

        /// FIXME: Implement in other way -- not string comparison
        const std::string next_codec = astToString(ttl_description->recompression_codec);
        const std::string current_codec = astToString(part->default_codec->getFullCodecDescription());

        return PartProperties::RecompressTTLInfo{
            .will_change_codec = (next_codec != current_codec),
            .next_recompress_ttl = part->ttl_infos.getMinimalMaxRecompressionTTL(),
        };
    }

    return std::nullopt;
}

Two things are worth noting. The comparison is a string comparison of the codec descriptions — the source even carries a FIXME about it. And the “approximate codec” case is handled by giving up and assuming the codec will change: a part whose codec could not be recovered exactly is always reconsidered, because wrongly concluding “no change” would permanently suppress a recompression that is still needed.

The selector then refuses to consider a part whose codec would not change:

cpp
// src/Storages/MergeTree/Compaction/MergeSelectors/TTLMergeSelector.cpp:281
TTLRecompressMergeSelector::TTLRecompressMergeSelector(const PartitionIdToTTLs & merge_due_times_, time_t current_time_)
    : ITTLMergeSelector(&merge_due_times_, current_time_)
{
}

time_t TTLRecompressMergeSelector::getTTLForPart(const PartProperties & part) const
{
    return part.recompression_ttl_info->next_recompress_ttl;
}

bool TTLRecompressMergeSelector::canConsiderPart(const PartProperties & part) const
{
    if (part.is_in_volume_where_merges_avoid)
        return false;

    if (!part.recompression_ttl_info.has_value())
        return false;

    /// Allow part recompression only if it will change codec. Otherwise there will be no difference in bytes size.
    return part.recompression_ttl_info->will_change_codec;
}

getTTLForPart returns the part’s maximum expiry, and the generic TTL range builder only includes a part whose TTL has passed, so a part is only rewritten once every one of its rows has expired. The recompression selector is the fourth priority in the TTL merge scheduler, after drop, row delete and column delete:

cpp
// src/Storages/MergeTree/Compaction/MergeSelectorApplier.cpp:113
    /// Recompression - 4 priority
    if (!ctx.merge_constraints.empty() && ctx.metadata_snapshot.hasAnyRecompressionTTL())
    {
        TTLRecompressMergeSelector recompress_ttl_selector(ctx.next_recompress_times, ctx.current_time);

        if (auto merge_ranges = recompress_ttl_selector.select(ctx.ranges, ctx.merge_constraints, ctx.range_filter); !merge_ranges.empty())
            return pack(ctx, std::move(merge_ranges), MergeType::TTLRecompress);
    }

Recompression is the last priority because it changes nothing about the data — it is pure housekeeping — while the first three change what the data means.

Like the delete TTL, recompression is throttled per partition, with its own timeout setting:

cpp
// src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:225
            case MergeType::TTLRecompress:
            {
                const time_t next_time = current_time + (*settings)[MergeTreeSetting::merge_with_recompression_ttl_timeout];
                last_recompress_ttl_merge_time_advance[partition_id]
                    = {.installed = next_time, .previous = getTTLMergeTime(next_recompress_ttl_merge_times_by_partition, partition_id)};
                next_recompress_ttl_merge_times_by_partition[partition_id] = next_time;
                break;
            }

The codec reaches the column streams

The merge itself is an ordinary merge; the only special thing about it is the codec that the writer uses. The merge asks the table which codec the new part should have, and the answer is the TTL’s codec if a recompression rule is due:

cpp
// src/Storages/MergeTree/MergeTreeData.cpp:12305
MergeTreeData::PartCompressionCodec MergeTreeData::getCompressionCodecForPart(
    const StorageMetadataPtr & metadata_snapshot,
    size_t part_size_compressed,
    const IMergeTreeDataPart::TTLInfos & ttl_infos,
    time_t current_time) const
{
    const auto & recompression_ttl_entries = metadata_snapshot->getRecompressionTTLs();
    auto best_ttl_entry = selectTTLDescriptionForTTLInfos(recompression_ttl_entries, ttl_infos.recompression_ttl, current_time, true);

    if (best_ttl_entry)
        return {
            .codec = CompressionCodecFactory::instance().get(best_ttl_entry->recompression_codec, {}),
            .is_explicit_recompression = !CompressionCodecFactory::isDefaultCodec(best_ttl_entry->recompression_codec)};

    auto codec_setting = (*getSettings())[MergeTreeSetting::default_compression_codec].value;
    if (!codec_setting.empty())
        return {.codec = CompressionCodecFactory::instance().get(codec_setting)};

The rest of the function is the ordinary, adaptive codec selection: if no TTL rule applies, the table-level default_compression_codec setting wins, and otherwise the codec is chosen from the ratio of the part’s size to the table’s total size. The TTL is simply the first rule in that chain.

The merge records the choice on the part, together with the reason:

cpp
// src/Storages/MergeTree/MergeTask.cpp:1169
    else
    {
        auto part_compression_codec = global_ctx->data->getCompressionCodecForPart(
            global_ctx->metadata_snapshot,
            global_ctx->merge_list_element_ptr->total_size_bytes_compressed,
            global_ctx->new_data_part->ttl_infos,
            global_ctx->time_of_merge);
        global_ctx->compression_codec = std::move(part_compression_codec.codec);
        global_ctx->is_explicit_recompression = part_compression_codec.is_explicit_recompression;
    }

    /// Record the chosen codec on the part so that its projections (merged by the sub-merge above,
    /// or rebuilt via `writeTempProjectionPart`) inherit the same codec, together with whether it
    /// was asked for by an explicit `RECOMPRESS` TTL.
    global_ctx->new_data_part->default_codec = global_ctx->compression_codec;
    global_ctx->new_data_part->default_codec_is_explicit_recompression = global_ctx->is_explicit_recompression;

The is_explicit_recompression flag is what stops the adaptive codec selection from overriding the user’s explicit choice. The projections of the part inherit the parent’s codec — and the flag — so that a projection is not silently downgraded to LZ4 because it is small.

The writer then uses the part’s default codec for every column that does not declare its own:

cpp
// src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp:157
ASTPtr IMergeTreeDataPartWriter::getCodecDescriptionOrDefault(const String & column_name, CompressionCodecPtr default_codec) const
{
    /// The `default_codec` is already resolved by `MergeTreeData::getCompressionCodecForPart`, which
    /// honors the table-level `default_compression_codec` setting as well as `RECOMPRESS` TTL codecs.
    /// We must trust it here: re-reading `default_compression_codec` and overriding `default_codec`
    /// would make a `RECOMPRESS` TTL merge write column streams with the setting's codec while the
    /// part metadata (`default_compression_codec.txt`) records the TTL codec, so the metadata and the
    /// actual on-disk data would diverge and recompression would not be applied.
    ASTPtr default_codec_desc = default_codec->getFullCodecDescription();

    if (const auto * column_desc = metadata_snapshot->columns.tryGet(column_name))
        return column_desc->codec ? column_desc->codec : default_codec_desc;

The comment explains exactly the failure mode the design avoids: if the writer re-derived the codec from the setting while the metadata recorded the TTL’s codec, the two would disagree, and the next recompression decision would be made against a fiction.

The part remembers its codec

The chosen codec is written into the part’s own metadata, in default_compression_codec.txt:

cpp
// src/Storages/MergeTree/MergedBlockOutputStream.cpp:484
    if (default_codec != nullptr)
    {
        write_plain_file(IMergeTreeDataPart::DEFAULT_COMPRESSION_CODEC_FILE_NAME, [&](auto & buffer)
        {
            if (new_part->default_codec_is_approximate)
                writeText(IMergeTreeDataPart::UNKNOWN_DEFAULT_COMPRESSION_CODEC, buffer);
            else
                writeText(default_codec->getFullCodecDescription()->formatWithSecretsOneLine(), buffer);
        });
    }

This file is what makes the whole thing idempotent. Without it, the merge scheduler could not tell “this part is already ZSTD(17)” from “this part is ZSTD(3)”, and every part would be rewritten over and over.

The file can be missing or damaged — a part from an older version, a part whose metadata was truncated — and then the codec has to be recovered from the data itself, from the checksums of the column streams. When even that is impossible, the part’s codec is marked approximate:

cpp
// src/Storages/MergeTree/IMergeTreeDataPart.cpp:2186
    else
    {
        /// The `default_compression_codec.txt` file is missing. When no column proves the codec
        /// (every column has an explicit `CODEC`), only a legacy pre-change part can be recovered
        /// safely. A post-change part must contain this file; otherwise its actual default is no
        /// longer recorded on disk, so fail closed instead of reporting a guess.
        default_codec = detectDefaultCompressionCodec([&] { return detectDefaultCompressionCodecFromChecksums(); });
    }

and the “approximate” flag is what the recompression selector respects by assuming the codec will change.

Trade-offs

What the mechanism buys:

A per-part-age codec policy, expressed in one line of DDL. The archive partition ends up compressed the way archive data should be compressed, without anyone running a migration.
No useless rewrites. The will_change_codec check means a part is only rewritten when the codec really differs, and the part’s own metadata is the source of truth for that comparison.
Idempotence and convergence. After the rewrite, the part records the new codec, so the next scheduler pass sees nothing to do. Two replicas that both run the merge end up with the same codec.
No interference with the data semantics. A recompression merge changes bytes on disk, never rows. It is the lowest-priority TTL, so it never delays a delete TTL.

What it costs:

A full rewrite of the part. The rows are read, decompressed, recompressed and written back. The saving is in the steady state — the smaller part costs less to read and less to store — not in the rewrite itself. A part that is rewritten every month pays that I/O every month.
The “all rows must have expired” rule. A part whose rows straddle the boundary is not eligible, so the archive is only recompressed once it is genuinely frozen. For a table with a single active partition, that may never happen.
A string comparison decides whether a rewrite is needed. The source carries a FIXME about it; two spellings of the same codec would be treated as different, and the part would be rewritten once too often.
The four-hour throttle applies, so a rule that fires for many partitions at once will be honoured gradually, partition by partition.
It only changes the default codec. A column that declares its own CODEC(...) keeps it, which is usually what you want but occasionally is not what a user expects when they write RECOMPRESS CODEC(ZSTD(17)).

In short: RECOMPRESS is the compression analogue of the row TTL. It does not do anything at a precise moment — it declares a policy, and the background merger applies it opportunistically, to whole parts, when it is certain that the rewrite will actually change something.