Opensource internals

ClickHouse Keeps a Column Compressed Until Something Reads It

13 min read Jay Hubbert

ColumnCompressed wraps a column in an LZ4 buffer and throws on every accessor until decompress() is called, so a stored block can sit in RAM compressed and only pay for the bytes a query actually touches.

TL;DR: ColumnCompressed is a column whose getDataAt, operator[], filter, sort and every other accessor throw LOGICAL_ERROR until somebody calls decompress(). It holds the payload as one LZ4 buffer plus a std::function that rebuilds the original column. The Memory engine’s compress = true setting, the query result cache and the nested-loop join’s build store all use it, so a block can sit in RAM at a fraction of its size and be inflated only for the columns a query actually reads.

Every in-memory table has the same problem. The data is in RAM, it is readable, and therefore it is occupied. But “occupied” and “read” are not the same event: a Memory table with a WHERE clause that matches 1% of the rows pays for 100% of the bytes when the rows are inserted, and then reads only the columns the predicate needs.

The obvious fix is to compress the data at rest in RAM and decompress it on read. The interesting part is not the compression — it is the shape of the wrapper. It is not a column that contains compressed data. It is a column that refuses to do anything until it has been decompressed:

cpp 
// src/Columns/ColumnCompressed.h:158
    [[noreturn]] static void throwMustBeDecompressed()
    {
        throw Exception(ErrorCodes::LOGICAL_ERROR, "ColumnCompressed must be decompressed before use");
    }

That single decision — fail loudly instead of silently inflating — is what makes the rest of the design work. Because no code path can accidentally read a compressed column, the compression can be pushed all the way down into the storage layer without auditing every consumer of Block.

The class

The class documents its own contract:

cpp 
// src/Columns/ColumnCompressed.h:19
/** Wrapper for compressed column data.
  * The only supported operations are:
  * - decompress (reconstruct the source column)
  * - get the separately compressed parts of the source column, see `getParts`
  * - get size in rows or bytes.

and, further down, the reason it exists at all:

cpp 
// src/Columns/ColumnCompressed.h:28
  * It's often beneficial to store compressed data in-memory and decompress on the fly
  * because it allows to lower memory throughput. More specifically, if:
  *
  * decompression speed * num CPU cores >= memory read throughput
  *
  * Also in-memory compression allows to keep more data in RAM.

The class is a COWHelper over IColumn, like every column, but it holds three things and nothing else:

cpp 
// src/Columns/ColumnCompressed.h:40
    ColumnCompressed(size_t rows_, size_t bytes_, Lazy lazy_, Columns parts_ = {})
        : rows(rows_), bytes(bytes_), lazy(lazy_), parts(std::move(parts_))
    {
    }

where Lazy is just std::function<ColumnPtr()> — the recipe for rebuilding the column. The three fields are the whole trick:

  • rows — how many rows the column has, so that size-sensitive logic (block row counts, Block::rows()) keeps working without inflating anything.
  • bytes — the compressed size. This is what byteSize() reports, which is why the Memory engine can use it as the weight of a column when it orders PREWHERE conditions.
  • lazy — the recipe for rebuilding the column.

And decompress() is one line:

cpp 
// src/Columns/ColumnCompressed.h:51
    ColumnPtr decompress() const override
    {
        return lazy();
    }

Note what is not there: no memoization. Calling decompress() twice runs the LZ4 decompression twice and produces two distinct columns. The compressed column is a recipe, not a cache. Every consumer that needs the data has to be careful to decompress once and share the result — which is exactly what the read path does.

The compression itself: raw LZ4, no codec pipeline

ClickHouse has a whole codec framework for on-disk columns — Delta, Gorilla, ZSTD, CODEC(...) chains — and it is not what ColumnCompressed uses. This is plain LZ4 over a flat byte buffer:

cpp 
// src/Columns/ColumnCompressed.cpp:19
std::shared_ptr<Memory<>> ColumnCompressed::compressBuffer(const void * data, size_t data_size, bool force_compression)
{
    size_t max_dest_size = LZ4_COMPRESSBOUND(data_size);

    if (max_dest_size > std::numeric_limits<int>::max())
        throw Exception(ErrorCodes::CANNOT_COMPRESS, "Cannot compress column of size {}", formatReadableSizeWithBinarySuffix(data_size));

    Memory<> compressed(max_dest_size);

    int compressed_size = LZ4_compress_default(
        reinterpret_cast<const char *>(data),
        compressed.data(),
        static_cast<int>(data_size),
        static_cast<int>(max_dest_size));

    if (compressed_size <= 0)
        throw Exception(ErrorCodes::CANNOT_COMPRESS, "Cannot compress column");

    /// If compression is inefficient.
    const size_t threshold = force_compression ? 1 : 2;
    if (static_cast<size_t>(compressed_size) * threshold > data_size)
        return {};

    /// Shrink to fit.
    auto shrank = std::make_shared<Memory<>>(compressed_size);
    memcpy(shrank->data(), compressed.data(), compressed_size);

    return shrank;
}

Three details worth pausing on:

  1. The threshold. By default the compressed buffer must be at most half the size of the original (threshold = 2). With force_compression the bar drops to “not larger than the original” (threshold = 1). Anything that fails the bar returns nullptr, and the caller falls back to storing the column uncompressed.
  2. Shrink to fit. LZ4’s COMPRESSBOUND is a worst-case bound, typically size + size/255 + 16. The result is copied into an exactly-sized buffer so the RAM accounting is honest. This copy is the price of the “compressed size” claim.
  3. No codec pipeline. The bytes handed to LZ4 are the raw in-memory representation: for a UInt64 column, 65 409 eight-byte integers; for a String column, the char buffer. There is no Delta, no transposed layout. LZ4 finds the repeated bytes of a mostly-constant integer column just fine, and skipping the codec framework keeps the wrapper independent of the on-disk format.

The inverse is LZ4_decompress_safe — the safe variant, which validates the output buffer length instead of trusting the compressed stream (src/Columns/ColumnCompressed.cpp:50).

LZ4_decompress_safe — the safe variant, which validates the output buffer length instead of trusting the compressed stream.

Who implements compress()

The entry point is on the column interface, and the default is “this column type cannot be compressed”:

cpp 
// src/Columns/IColumn.h:816
    /// Compress column in memory to some representation that allows to decompress it back.
    /// Return itself if compression is not applicable for this column type.
    /// The flag `force_compression` indicates that compression should be performed even if it's not efficient (if only compression factor < 1).
    [[nodiscard]] virtual Ptr compress([[maybe_unused]] bool force_compression) const
    {
        /// No compression by default.
        return getPtr();
    }

    /// If it's CompressedColumn, decompress it and return.
    /// Otherwise return itself.
    [[nodiscard]] virtual Ptr decompress() const
    {
        return getPtr();
    }

A flat numeric column is the easy case — one buffer, one LZ4 stream:

cpp 
// src/Columns/ColumnVector.cpp:1331
template <typename T>
ColumnPtr ColumnVector<T>::compress(bool force_compression) const
{
    const size_t data_size = data.size();
    const size_t source_size = data_size * sizeof(T);

    /// Don't compress small blocks.
    if (source_size < 4096) /// A wild guess.
        return ColumnCompressed::wrap(this->getPtr());

    auto compressed = ColumnCompressed::compressBuffer(data.data(), source_size, force_compression);

    if (!compressed)
        return ColumnCompressed::wrap(this->getPtr());

    const size_t compressed_size = compressed->size();
    return ColumnCompressed::create(data_size, compressed_size,
        [my_compressed = std::move(compressed), column_size = data_size]
        {
            auto res = ColumnVector<T>::create(column_size);
            ColumnCompressed::decompressBuffer(
                my_compressed->data(), res->getData().data(), my_compressed->size(), column_size * sizeof(T));
            return res;
        });
}

Two thresholds show up here and they are different from the LZ4 threshold: a column smaller than 4096 bytes is not even attempted (the comment says “A wild guess”), and if LZ4 fails the 50% bar, the column is stored as-is.

ColumnCompressed::wrap is the “give up but stay uniform” path: it wraps the uncompressed column in a ColumnCompressed whose lazy returns the original pointer. Nothing is compressed, but the block’s shape is uniform — every column in a compressed block is a ColumnCompressed, so the reader does not need to branch on the type:

(ColumnCompressed::wrap, src/Columns/ColumnCompressed.h:65, wraps the uncompressed column in a ColumnCompressed whose lazy returns the original pointer.)

A String column is the interesting case, because it is two buffers — the chars and the offsets — and they compress very differently. The chars are text (compressible), the offsets are a monotonically increasing sequence of UInt64 (usually not worth compressing at all):

cpp 
// src/Columns/ColumnString.cpp:726
    auto offsets_compressed = ColumnCompressed::compressBuffer(offsets.data(), source_offsets_size, force_compression);
    const bool offsets_were_compressed = !!offsets_compressed;

    /// Offsets are not compressible. Use the source data.
    if (!offsets_compressed)
    {
        offsets_compressed = std::make_shared<Memory<>>(source_offsets_size);
        memcpy(offsets_compressed->data(), offsets.data(), source_offsets_size);
    }

and the column keeps a part — the offsets alone — as a separately decompressible object:

cpp 
// src/Columns/ColumnString.cpp:739
    ColumnPtr offsets_part = ColumnCompressed::create(
        source_offsets_elements,
        offsets_compressed_size,
        [my_offsets_compressed = offsets_compressed, source_offsets_elements, offsets_were_compressed]
        {
            auto res = ColumnUInt64::create(source_offsets_elements);
            restoreOffsets(*my_offsets_compressed, offsets_were_compressed, res->getData());
            return res;
        });

The whole column is then a second ColumnCompressed that holds both compressed buffers, knows how to rebuild a ColumnString from them, and carries the offsets part in its parts list.

The parts mechanism

getParts() is the reason a compressed column can answer some questions without being fully inflated:

cpp 
// src/Columns/ColumnCompressed.h:56
    /// The parts of the source column, which share the compressed data (not counted in `byteSize`), or empty if it can be
    /// decompressed only as a whole: `ColumnString` {offsets as `ColumnUInt64`}, `ColumnArray` {data, offsets},
    /// `ColumnNullable` {nested, null map}, `ColumnTuple` {elements}, `ColumnMap` {nested array}.
    const Columns & getParts() const { return parts; }

A ColumnNullable is compressed by compressing its two sub-columns and keeping them as parts; a ColumnArray does the same with its data and its offsets (src/Columns/ColumnNullable.cpp:850, src/Columns/ColumnArray.cpp:1444). The parts are the point: they are independently decompressible.

A ColumnTuple keeps one part per element, and a ColumnMap one part for its whole nested array.

Reading a compressed column: partial decompression

Here is where the parts pay off. The Memory engine’s reader has to answer two different kinds of request: “give me the whole column” and “give me the size of each array”. The second one needs only the offsets.

cpp 
// src/Interpreters/getColumnFromBlock.cpp:26
const Columns * getCompressedParts(const IColumn & column, size_t expected)
{
    const auto * compressed = typeid_cast<const ColumnCompressed *>(&column);
    if (!compressed || compressed->getParts().size() != expected)
        return nullptr;
    return &compressed->getParts();
}

and for arrays, the offsets part alone:

cpp 
// src/Interpreters/getColumnFromBlock.cpp:103
ColumnPtr tryDecompressArrayOffsets(const IDataType & type, const IColumn & column)
{
    if (!isArray(type))
        return nullptr;
    const auto * parts = getCompressedParts(column, 2);
    return parts ? (*parts)[1]->decompress() : nullptr;
}

So a query that reads only arr.size0 from a compressed Array column never inflates the array data — it inflates the offsets (8 bytes per row) and answers the question. That is the difference between “the column is compressed” and “the column is compressed except for the part you need”.

The consumers

The Memory engine

The Memory engine has a compress setting:

cpp 
// src/Storages/MemorySettings.cpp:19
    DECLARE(Bool, compress, false, "Compress data in memory", 0) \

and the sink applies it to every column of every inserted block:

cpp 
// src/Storages/StorageMemory.cpp:121
        if ((*memory_settings)[MemorySetting::compress])
        {
            Block compressed_block;
            for (const auto & elem : block)
                compressed_block.insert({elem.column->compress(/*force_compression=*/true), elem.type, elem.name});
            new_blocks.push_back(std::move(compressed_block));
        }

Note force_compression = true here: the Memory engine is storing the data to be compressed, so even a column that only compresses to 95% of its original size is worth wrapping — the wrapper’s byteSize() then reports the compressed size, which is what the engine’s memory accounting uses.

The same file explains why byteSize() of a compressed column is the compressed size, in a comment that is really a design note:

cpp 
// src/Storages/StorageMemory.cpp:810
    for (const auto & block : current_data->blocks)
    {
        for (const auto & elem : block)
        {
            /// For a table with `compress = true` the stored columns are `ColumnCompressed`,
            /// whose `byteSize` is the compressed size - the amount of data a read of the
            /// column has to decompress, which is exactly the weight the WHERE -> PREWHERE
            /// optimization orders conditions by.
            column_sizes[elem.name].data_compressed += elem.column->byteSize();
        }
    }

This is the reason the wrapper reports the compressed size rather than the uncompressed one: the optimizer sorts PREWHERE conditions by “how many bytes will I have to touch to evaluate this predicate”, and for a compressed column that is the compressed size.

The query result cache

The query result cache has a setting of its own, query_cache_compress_entries (default true, src/Core/Settings.cpp:7199), and the writer compresses every column of every chunk on the way in:

cpp 
// src/Interpreters/Cache/QueryResultCache.cpp:978
            const Columns & columns = chunk.getColumns();
            Columns compressed_columns;
            for (const auto & column : columns)
            {
                auto compressed_column = column->compress(/*force_compression=*/false);
                compressed_columns.push_back(compressed_column);
            }

Here force_compression is false: the cache holds many different result sets, and a column that does not compress well should not be wrapped in a wrapper that buys nothing.

The join build store

The nested-loop join keeps its build side in a store that may spill to disk. Before spilling it tries to compress the stored blocks, and it compares the sizes to decide whether the compression is worth anything:

cpp 
// src/Processors/Transforms/BlockNestedLoopJoinData.cpp:113
std::optional<BuildBlock> compressBuildBlock(const BuildBlock & build_block)
{
    Columns columns;
    columns.reserve(build_block.columns.size());
    bool any_compressed = false;
    for (const auto & column : build_block.columns)
    {
        auto compressed_column = column->compress(/*force_compression=*/ false);
        if (compressed_column->byteSize() >= column->byteSize())
            compressed_column = column;
        else
            any_compressed = true;
        columns.push_back(std::move(compressed_column));
    }

The comment is precise about the failure mode: a ColumnCompressed wrapper around an uncompressible column is not free. Because decompress() is not memoized, a reader that decompresses a block gets a fresh copy every time — memory that is allocated, charged to the reader’s chunk, and thrown away.

The scattered block

The parallel hash join shuffles blocks between threads, and the shuffle compresses them on the way (ScatteredBlock::compress() in src/Interpreters/HashJoin/ScatteredBlock.h:250).

The read path: decompress once, share the result

Since decompress() is not memoized, the reader has to decompress once per column and hand the result to everyone who needs it. The Memory engine’s reader does exactly that, and it is careful to avoid decompressing a column that is only needed for one of its subcolumns:

cpp 
// src/Processors/QueryPlan/ReadFromMemoryStorageStep.cpp:249
    std::optional<Block> decompressed;
    for (const auto & column : columns_to_read)
    {
        if (column.isSubcolumn() || !parents_of_subcolumns.contains(column.name))
            continue;

        const auto * elem = src.findByName(column.name);
        if (!elem || !elem->column)
            continue;

        auto decompressed_column = elem->column->decompress();
        if (decompressed_column == elem->column)
            continue;

        if (!decompressed)
            decompressed = src;
        decompressed->getByName(column.name).column = std::move(decompressed_column);
    }

Two things to notice. First, decompressed_column == elem->column — the ColumnCompressed::wrap case returns the same pointer, so nothing is decompressed for an uncompressible column. Second, the decompressed block is a copy of the source block (decompressed = src) whose columns have been replaced; the original block still holds the compressed columns, so the compressed copy stays in the store and can be decompressed again for the next query.

That last point is the whole trick of the Memory engine: the store keeps the compressed block, every query inflates its own copy, and the copy is discarded when the query is done. The peak memory is compressed_size + inflated_size_of_the_columns_this_query_touched.

What it costs

  • No memoization. decompress() runs the LZ4 expansion every time it is called. A block that is decompressed by three different consumers is decompressed three times unless the caller is careful. The read path is careful; third-party code has to be too.
  • LZ4 only, no codec pipeline. A column of slowly-varying integers does not compress under LZ4; the codec framework’s Delta would, and it is not used here. That is a deliberate trade: the codec framework is tied to the on-disk format, and ColumnCompressed must work for arbitrary in-memory columns.
  • A flat buffer, so only “flat” columns benefit. ColumnVector, ColumnFixedString, ColumnString and ColumnDecimal compress their raw bytes; a ColumnArray compresses its data and offsets separately.
  • An extra copy on compression. compressBuffer copies the LZ4 output into an exactly-sized buffer to make the size honest.
  • A wrapper that must be unwrapped. Every consumer of a Block has to know about ColumnCompressed; the throwMustBeDecompressed design turns that into a loud error rather than a silent wrong answer, but it is still a cross-cutting concern.

The escape hatch

sql 
CREATE TABLE t (x UInt64, s String) ENGINE = Memory SETTINGS compress = false;

SELECT sum(x) FROM t;   -- uncompressed: fast scans, more RAM

and, for the query cache:

sql 
SELECT count() FROM hits SETTINGS query_cache_compress_entries = 0;

Wrapping up

The one thing to remember: ColumnCompressed is not a column of compressed data; it is a column that cannot be used until it is decompressed. That distinction is what lets the compression live in the storage layer.

The part people get wrong: decompress() is not a getter that caches. Two calls, two inflations, two copies. If you write code that touches a compressed column from two places, you will pay twice and allocate twice.

What to go do about it: if you have a Memory table (a temporary table, a dictionary source, a Materialized View target) that is much larger than RAM allows, turn on compress = true and measure. The win is largest when your queries are selective — a PREWHERE that touches 1% of the rows inflates 1% of the columns. If your queries read everything, you are paying LZ4 compression on write and LZ4 decompression on read for a memory saving you may not need.

ClickhouseInternalsMemoryCompressionPerformance