ClickHouse's Decimal: the Scale Is a Multiplier, Not Part of the Value
A ClickHouse Decimal is a plain integer plus a scale that lives in the type; the arithmetic is a scale shift, and the 128-bit path has a silent overflow hole that the repo's own tests document.
TL;DR:
Decimal(38, 18)is not a floating-point number with 18 decimals. It is anInt128holding 42420000000000000000, and the “18” is a property of the type that tells you where the decimal point is. Every arithmetic operation is therefore a scale shift — a multiplication by a power of ten — and the 128-bit path has an overflow check that silently does nothing.
Fixed-point arithmetic is old, and the textbook description of it is
uninteresting: an integer, plus a number of digits after the point. The
interesting part is what a database has to decide about it, and ClickHouse’s
decisions are unusually explicit. Where does the scale live? What is the
result scale of a / b? What happens when the answer does not fit?
The answers are: the scale lives in the type; the result scale of a division is the left operand’s scale, which forces an implementation trick; and the overflow behaviour has a hole in it that is documented — with a test — in the repository itself.
The value is an integer
The core type is deliberately thin:
// base/base/Decimal.h:36
/// Own FieldType for Decimal.
/// It is only a "storage" for decimal.
/// To perform operations, you also have to provide a scale (number of digits after point).
template <typename T>
struct Decimal
{
using NativeType = T;
constexpr Decimal() = default;
constexpr Decimal(Decimal<T> &&) noexcept = default;
constexpr Decimal(const Decimal<T> &) = default;
constexpr Decimal(const T & value_): value(value_) {} // NOLINT(google-explicit-constructor)
template <typename U>
constexpr Decimal(const Decimal<U> & x): value(static_cast<T>(x.value)) {} // NOLINT(google-explicit-constructor)
constexpr Decimal<T> & operator=(Decimal<T> &&) noexcept = default;
constexpr Decimal<T> & operator = (const Decimal<T> &) = default;
constexpr operator T () const { return value; } // NOLINT(google-explicit-constructor)
The last line is the whole point: a Decimal is its underlying integer. The
scale is nowhere in the struct — you cannot look at a Decimal128 and know
how many decimals it has. That information lives in the type, and in the
column.
The four underlying types are fixed by how many digits they can hold:
// src/Core/DecimalFunctions.h:27
inline constexpr size_t min_precision = 1;
template <typename T> inline constexpr size_t max_precision = 0;
template <> inline constexpr size_t max_precision<Decimal32> = 9;
template <> inline constexpr size_t max_precision<Decimal64> = 18;
template <> inline constexpr size_t max_precision<DateTime64> = 18;
template <> inline constexpr size_t max_precision<Time64> = 18;
template <> inline constexpr size_t max_precision<Decimal128> = 38;
template <> inline constexpr size_t max_precision<Decimal256> = 76;9, 18, 38, 76 — one more than the number of decimal digits that fit in
Int32, Int64, Int128, Int256. (38 rather than 39 because a signed
128-bit integer holds 39 digits but the top one is only half the range, and a
Decimal must be able to represent both signs of the maximum.)
The scale multiplier is a table lookup, not a pow:
// src/Core/DecimalFunctions.h:36
template <typename T>
inline auto scaleMultiplier(UInt32 scale)
{
if constexpr (std::is_same_v<T, Int32> || std::is_same_v<T, Decimal32>)
return common::exp10_i32(scale);
else if constexpr (std::is_same_v<T, Int64> || std::is_same_v<T, Decimal64> || std::is_same_v<T, DateTime64> || std::is_same_v<T, Time64>)
return common::exp10_i64(scale);
else if constexpr (std::is_same_v<T, Int128> || std::is_same_v<T, Decimal128>)
return common::exp10_i128(scale);
else if constexpr (std::is_same_v<T, Int256> || std::is_same_v<T, Decimal256>)
return common::exp10_i256(scale);
}The type carries the scale
DataTypeDecimalBase is where precision and scale live, and it is
parametric — Decimal(38, 18) and Decimal(38, 6) are different types:
// src/DataTypes/DataTypeDecimalBase.h:47
/// Base class for decimals, like Decimal(P, S), where P is precision, S is scale.
/// Maximum precisions for underlying types are:
/// Int32 9
/// Int64 18
/// Int128 38
/// Int256 76
/// Operation between two decimals leads to Decimal(P, S), where
/// P is one of (9, 18, 38, 76); equals to the maximum precision for the biggest underlying type of operands.
/// S is maximum scale of operands. The allowed valuas are [0, precision]
template <is_decimal T>
class DataTypeDecimalBase : public IDataType
{
public:
using FieldType = T;
using ColumnType = ColumnDecimal<T>;
static constexpr auto type_id = TypeToTypeIndex<T>;
static constexpr bool is_parametric = true;
static constexpr size_t maxPrecision();
DataTypeDecimalBase(UInt32 precision_, UInt32 scale_)
: precision(precision_),
scale(scale_)
{
if (unlikely(precision < 1 || precision > maxPrecision()))
throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND,
"Precision {} is out of bounds (precision range: [1, {}])",
std::to_string(precision), maxPrecision());
if (unlikely(scale > maxPrecision()))
throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND, "Scale {} is out of bounds (max scale: {})",
std::to_string(scale), maxPrecision());
}and the scale is what turns an integer into a number:
// src/DataTypes/DataTypeDecimalBase.h:100
/// Decimal specific
UInt32 getPrecision() const { return precision; }
UInt32 getScale() const { return scale; }
T getScaleMultiplier() const { return getScaleMultiplier(scale); }
T wholePart(T x) const;
T fractionalPart(T x) const;
T maxWholeValue() const { return getScaleMultiplier(precision - scale) - T(1); }maxWholeValue() is the largest integer part that fits: 10^(P-S) - 1. For
Decimal(38, 18) that is 10^20 - 1, so the value is really
[-(10^20-1).999…, +(10^20-1).999…]. The precision constrains the whole
part, and the scale constrains the fraction, and the two are not
independent — 38 digits total.
The column, meanwhile, stores the raw integers and remembers the scale alongside:
// src/Columns/ColumnDecimal.h:13
/// A ColumnVector for Decimals
template <is_decimal T>
class ColumnDecimal final : public COWHelper<IColumnHelper<ColumnDecimal<T>, ColumnFixedSizeHelper>, ColumnDecimal<T>>
{
private:
using Self = ColumnDecimal;
friend class COWHelper<IColumnHelper<Self, ColumnFixedSizeHelper>, Self>;
public:
using ValueType = T;
using NativeT = typename T::NativeType;
using Container = PaddedPODArray<T>;
private:
ColumnDecimal(const size_t n, UInt32 scale_)
: data(n),
scale(scale_)
{}The scale shift
Now the arithmetic. The rule set is short, and the implementation comment states it up front:
// src/Functions/FunctionBinaryArithmetic.h:620
/**
* Binary operations with Decimals (either Decimal OP Decimal or Decimal Op Float) need to scale the args correctly.
* - + (plus), - (minus), * (multiply), least and greatest operations scale one of the args (which scale factor is not 1).
* The resulting scale is either left or the right scale.
* - / (divide) operation scales the first argument.
* The resulting scale is the first one's.
*/Three rules:
| operation | result scale | how |
|---|---|---|
a + b, a - b | max(sa, sb) | multiply the operand with the smaller scale by 10^Δ |
a * b | sa + sb | nothing — the decimal point just moves |
a / b | sa | multiply the left operand by 10^sb, then divide |
The multiplication row is the elegant one: since the values are scaled
integers, a.value * b.value is already the right answer at scale
sa + sb. The decimal point moves; no work is done.
The division row is the awkward one, and the implementation shows the trick.
The result scale is sa, so the result value must be
result.value = (a.value / 10^sa) / (b.value / 10^sb) * 10^sa
= a.value * 10^sb / b.valueso the left operand is multiplied by the right operand’s scale multiplier before the integer division:
// src/Functions/FunctionBinaryArithmetic.h:865
template <bool is_decimal_a>
static NO_SANITIZE_UNDEFINED NativeResultType applyScaledDiv(NativeResultType a, NativeResultType b, NativeResultType scale)
{
if constexpr (is_division)
{
if constexpr (check_overflow)
{
bool overflow = false;
if constexpr (!is_decimal_a)
overflow |= common::mulOverflow(scale, scale, scale);
overflow |= common::mulOverflow(a, scale, a);
if (overflow)
throw Exception(ErrorCodes::DECIMAL_OVERFLOW, "Decimal math overflow");
}
else
{
if constexpr (!is_decimal_a)
scale *= scale;
a *= scale;
}
return Op::template apply<NativeResultType>(a, b);
}
}And where does that scale come from? It is the right operand’s scale
multiplier — chosen in the executor, not the operation:
// src/Functions/FunctionBinaryArithmetic.h:2249
const ResultType scale_a = [&]
{
if constexpr (IsDataTypeDecimal<RightDataType> && is_division)
return right.getScaleMultiplier(); // the division impl uses only the scale_a
else
{
if constexpr (is_multiply)
// the decimal impl uses scales, but if the result is decimal, both of the arguments are decimal,
// so they would multiply correctly, so we need to scale the result to the neutral element (1).
// The explicit type is needed as the int (in contrast with float) can't be implicitly converted
// to decimal.
return ResultType{1};
else
return type.scaleFactorFor(left, false);
}
}();
const ResultType scale_b = [&]
{
if constexpr (is_multiply)
return ResultType{1};
else
return type.scaleFactorFor(right, is_division);
}();Three cases in one place:
- division:
scale_ais the right operand’s scale multiplier, because the division implementation only ever usesscale_a. The naming is confusing; the comment// the division impl uses only the scale_ais the only warning you get. - multiplication: both scales are
1— the neutral element — because the multiplication is already correct and the result scale is carried by the type, not by the operands. - everything else:
type.scaleFactorFor(left, false)— the factor that brings the left operand up to the result scale.
The result scale itself comes from the traits:
// src/Core/DecimalFunctions.h:451
template <bool is_multiply, bool is_division, typename T, typename U, template <typename> typename DecimalType>
inline auto binaryOpResult(const DecimalType<T> & tx, const DecimalType<U> & ty)
{
UInt32 scale{};
if constexpr (is_multiply)
scale = tx.getScale() + ty.getScale();
else if constexpr (is_division)
scale = tx.getScale();
else
scale = (tx.getScale() > ty.getScale() ? tx.getScale() : ty.getScale());
if constexpr (sizeof(T) < sizeof(U))
return DataTypeDecimalTrait<U>(DecimalUtils::max_precision<U>, scale);
else
return DataTypeDecimalTrait<T>(DecimalUtils::max_precision<T>, scale);
}is_multiply → sa + sb; is_division → sa; otherwise → max(sa, sb).
Exactly the table above, in three lines.
The addition path, for completeness, is the generic scale shift:
// src/Functions/FunctionBinaryArithmetic.h:830
template <bool scale_left, bool may_check_overflow = true>
static NO_SANITIZE_UNDEFINED NativeResultType applyScaled(NativeResultType a, NativeResultType b, NativeResultType scale)
{
static_assert(is_plus_minus_compare || is_multiply);
NativeResultType res;
if constexpr (check_overflow && may_check_overflow)
{
bool overflow = false;
if constexpr (scale_left)
overflow |= common::mulOverflow(a, scale, a);
else
overflow |= common::mulOverflow(b, scale, b);
if constexpr (can_overflow)
overflow |= Op::template apply<NativeResultType>(a, b, res);
else
res = Op::template apply<NativeResultType>(a, b);
if (overflow)
throw Exception(ErrorCodes::DECIMAL_OVERFLOW, "Decimal math overflow");
}scale_left says which operand needs the shift; common::mulOverflow
detects whether the shift itself overflowed.
The overflow hole
Here is the part worth reading twice. The overflow detection is delegated to
common::mulOverflow, and that function has a hole for 128-bit integers:
// base/base/arithmeticOverflow.h:174
template <typename T>
inline bool mulOverflow(T x, T y, T & res)
{
return __builtin_mul_overflow(x, y, &res);
}
template <typename T, typename U, typename R>
inline bool mulOverflow(T x, U y, R & res)
{
// not built in type, wide integer
if constexpr (is_big_int_v<T> || is_big_int_v<R> || is_big_int_v<U>)
{
res = mulIgnoreOverflow<R>(x, y);
return false;
}
else
return __builtin_mul_overflow(x, y, &res);
}and the specialisations for the big types say it outright:
// base/base/arithmeticOverflow.h:211
/// Overflow check is not implemented for big integers.
template <>
inline bool mulOverflow(Int128 x, Int128 y, Int128 & res)
{
res = mulIgnoreOverflow(x, y);
return false;
}
template <>
inline bool mulOverflow(Int256 x, Int256 y, Int256 & res)
{
res = mulIgnoreOverflow(x, y);
return false;
}“Overflow check is not implemented for big integers.” So for Decimal32 and
Decimal64, a * b raises DECIMAL_OVERFLOW when the product does not
fit. For Decimal128 and Decimal256, it wraps silently.
And the low-level Decimal operators are designed to wrap:
// base/base/Decimal.h:67
/// The overflow-dependent operators below (`+=`, `-=`, `*=`, plus the free
/// `+`, `-`, `*` and unary `-`) intentionally wrap around — see `Decimal.cpp`
/// for the rationale — and are marked `NO_SANITIZE_UNDEFINED`. `/=` and `%=`
/// keep UBSan instrumentation so `divide-by-zero` and `INT_MIN / -1` are
/// still reported.
NO_SANITIZE_UNDEFINED const Decimal<T> & operator += (const T & x);
NO_SANITIZE_UNDEFINED const Decimal<T> & operator -= (const T & x);
NO_SANITIZE_UNDEFINED const Decimal<T> & operator *= (const T & x);
const Decimal<T> & operator /= (const T & x);
const Decimal<T> & operator %= (const T & x);with the rationale spelled out in the implementation file:
// base/base/Decimal.cpp:22
/// Decimal arithmetic in ClickHouse semantically allows wrap-around overflow for the
/// overflow-dependent operators (`+=`, `-=`, `*=`, plus the free `+`, `-`, `*` and
/// unary `-`): callers that need overflow detection use the helpers in
/// `base/arithmeticOverflow.h` (`addOverflow`, `mulOverflow`, etc.) or check operands
/// explicitly, while paths such as digit accumulation in `readDecimalText` and running
/// sums in aggregate functions rely on two's-complement wrap. Those operators are
/// marked `NO_SANITIZE_UNDEFINED` so UBSan does not flag the wrap as undefined
/// behaviour (matches `addOverflow` / `negateOverflow` below). `/=` and `%=` (and the
/// free `/`) are deliberately left instrumented so UBSan still catches divide-by-zero
/// and `INT_MIN / -1`.
That is a defensible design for the operator: a Decimal is a bag of bits
and the operator is a bag-of-bits operation. The problem is that the
query-level * operator is supposed to raise DECIMAL_OVERFLOW, and for
Int128 it cannot.
What that looks like from SQL
The repository’s own regression test shows it. The test table has a
Decimal(38, 18) column i:
// tests/queries/0_stateless/00700_decimal_arithm.sql:13
i deciMAL(38, 18),and the test runs the full operator set on it:
// tests/queries/0_stateless/00700_decimal_arithm.sql:34
SELECT i + i, i - i, i * i, i / i, intDiv(i, i), intDivOrZero(i, i) FROM decimal WHERE i > 0;with i = 42.42. The expected output, from the reference file, is:
// tests/queries/0_stateless/00700_decimal_arithm.reference:6
84.84 0 98.04456539530768268312696284115894272 1 1 1i + i = 84.84 ✓. i - i = 0 ✓. i / i = 1 ✓. And i * i is
98.04456539530768268312696284115894272, not 1799.4564.
That number is not a rounding artefact. It is the exact value of
(4242 · 10^16)² mod 2^128, divided by 10^36:
i.value = 4242 * 10^16 = 42 420 000 000 000 000 000
i.value² = 1 799 456 400 000 000 000 000 000 000 000 000 000 000
2^128 = 340 282 366 920 938 463 463 374 607 431 768 211 456
i.value² mod 2^128 = 98 044 565 395 307 682 683 126 962 841 158 942 720
/ 10^36 = 98.044565395307682683126962841158942720and the trailing zero is dropped by the decimal formatter, which is why the reference file shows 35 digits after the point rather than 36. The test does not expect an error, because there is no error to expect: the multiplication wrapped around modulo 2^128 and produced a number that looks entirely plausible.
Compare the Decimal(18, 9) column h in the same table, which does
raise:
// tests/queries/0_stateless/00700_decimal_arithm.sql:32
SELECT h + h, h - h, h * h, h / h, intDiv(h, h), intDivOrZero(h, h) FROM decimal WHERE h > 0; -- { serverError DECIMAL_OVERFLOW }h = 42.42 at scale 9 is 42 420 000 000, and its square is
1 799 456 400 000 000 000 000 — which does not fit in Int64
(9 223 372 036 854 775 807). Int64 has a real __builtin_mul_overflow, so
the query raises DECIMAL_OVERFLOW. Same value, same multiplication, one
column wider, opposite behaviour.
Division: a scale-based check instead
Division cannot use a width check, because the intermediate is always computed at the widest possible size. So the check is done on the scales, before the division happens:
// src/Functions/FunctionBinaryArithmetic.h:2818
else if constexpr (IsDataTypeDecimal<LeftDataType> && IsDataTypeDecimal<RightDataType>)
{
if constexpr (is_division)
{
if (decimalCheckArithmeticOverflow(context_))
{
/// Check overflow by using operands scale (based on big decimal division implementation details):
/// big decimal arithmetic is based on big integers, decimal operands are converted to big integers
/// i.e. int_operand = decimal_operand*10^scale
/// For division, left operand will be scaled by right operand scale also to do big integer division,
/// BigInt result = left*10^(left_scale + right_scale) / right * 10^right_scale
/// So, we can check upfront possible overflow just by checking max scale used for left operand
/// Note: it doesn't detect all possible overflow during big decimal division
if (left.getScale() + right.getScale() > ResultDataType::maxPrecision())
throw Exception(ErrorCodes::DECIMAL_OVERFLOW, "Overflow during decimal division");
}
}left.getScale() + right.getScale() > maxPrecision — 39 > 38 for
Decimal(38, 18) / Decimal(38, 1), so it raises. The comment is refreshingly
honest: “it doesn’t detect all possible overflow during big decimal
division”.
The check is controlled by a setting:
// src/DataTypes/DataTypeDecimalBase.cpp:30
bool decimalCheckArithmeticOverflow(ContextPtr context)
{
return !context || context->getSettingsRef()[Setting::decimal_check_overflow];
}and turning it off produces another beautiful number. From the same test file:
// tests/queries/0_stateless/00700_decimal_arithm.sql:86
SELECT toDecimal128(1, 38) / toDecimal128(1, 0) SETTINGS decimal_check_overflow=1;
SELECT toDecimal128(1, 38) / toDecimal128(1, 1) SETTINGS decimal_check_overflow=1; -- { serverError DECIMAL_OVERFLOW }
SELECT toDecimal128(1, 38) / toDecimal128(1, 1) SETTINGS decimal_check_overflow=0;
SELECT toDecimal128(1, 37) / toDecimal128(1, 1) SETTINGS decimal_check_overflow=1;// tests/queries/0_stateless/00700_decimal_arithm.reference:39
SELECT toDecimal128(1, 38) / toDecimal128(1, 0) SETTINGS decimal_check_overflow=1;
1
SELECT toDecimal128(1, 38) / toDecimal128(1, 1) SETTINGS decimal_check_overflow=1; -- { serverError DECIMAL_OVERFLOW }
SELECT toDecimal128(1, 38) / toDecimal128(1, 1) SETTINGS decimal_check_overflow=0;
-0.02084710076281539039012382229530463436The correct answer is 10^38 / 10 = 10^37, which at scale 38 is 0.1. What
comes back instead is -0.02084710076281539039012382229530463436, and that
is again a modular artefact: the implementation multiplies the left operand
by 10^sb = 10, giving 10^39, which does not fit in Int128
(max ≈ 1.7 · 10^38), so it wraps to
-20 847 100 762 815 390 390 123 822 295 304 634 368, and the integer
division by 10 truncates that to
-2 084 710 076 281 539 039 012 382 229 530 463 436, which at scale 38 is
exactly -0.02084710076281539039012382229530463436.
Meanwhile toDecimal128(1, 37) / toDecimal128(1, 1) is 1, because
10^37 · 10 = 10^38 does fit in Int128. The boundary is exactly where
Int128 runs out, and the check is a scale check that happens to agree with
it in this case.
One more check: narrowing
There is a second kind of overflow, and it is not about the size of the result but about the size of an operand. The operation runs in the width of the result, so an operand that does not survive the narrowing would wrap:
// src/Functions/FunctionBinaryArithmetic.h:787
/** The operation runs in the native width of the decimal result, so an operand that does not
* survive the narrowing into that width would wrap and make every row of the result silently
* wrong - `toDecimal32(1.5, 4) * 9223372036854775807` answered `-1.5`, because the multiplier
* narrowed to `-1`. Comparing the same pair already reports `DECIMAL_OVERFLOW`, and the
* documented contract of `Decimal` is that excessive digits in the integer part raise.
*
* Only the conversions that can lose information are checked, so the ones that cannot (a
* narrower integer into a wider native type) stay free.
*/
template <typename T>
static NativeResultType castToNative(const T & value)
{
const auto result = static_cast<NativeResultType>(value);
using Value = std::decay_t<T>;
if constexpr (
check_overflow && is_integer<Value> && is_integer<NativeResultType>
&& (sizeof(Value) > sizeof(NativeResultType) || is_signed_v<Value> != is_signed_v<NativeResultType>))
{
if (!accurate::equalsOp(value, result))
throw Exception(
ErrorCodes::DECIMAL_OVERFLOW,
"Decimal math overflow: the operand does not fit into the {} bits of the result",
sizeof(NativeResultType) * 8);
}
return result;
}The comment is a bug report turned into a regression test: toDecimal32(1.5, 4) * 9223372036854775807 used to answer -1.5, because the multiplier
narrowed to -1. Note the condition: the check only fires when the
conversion can lose information — a narrower integer into a wider type is
free.
The other implementation: digit arithmetic
There is a second, slower, decimal arithmetic engine in the tree, used by the
multiplyDecimal and divideDecimal functions. It exists to support an
explicit result scale, which the ordinary operators do not offer, and it
does the arithmetic the way you would do it on paper: as digits.
// src/Functions/FunctionsDecimalArithmetics.h:27
struct DecimalOpHelpers
{
/* These functions perform main arithmetic logic.
* As soon as intermediate results may not fit Decimal256 (e.g. 1e36, scale 10),
* we may not operate with Decimals. Later on this big number may be shrunk (e.g. result scale is 0 in the case above).
* That's why we need to store intermediate results in a flexible extendable storage (here we use std::vector)
* Here we operate on numbers using simple digit arithmetic.
* This is the reason these functions are slower than traditional ones.
*
* Here and below we use UInt8 for storing digits (0-9 range with maximum carry of 9 will definitely fit this)
*/
static VectorWithMemoryTracking<UInt8> multiply(const VectorWithMemoryTracking<UInt8> & num1, const VectorWithMemoryTracking<UInt8> & num2)Schoolbook long multiplication on UInt8 digit vectors:
// src/Functions/FunctionsDecimalArithmetics.h:49
for (Int32 i = len1 - 1; i >= 0; --i)
{
UInt16 carry = 0;
i_n2 = 0;
for (Int32 j = len2 - 1; j >= 0; --j)
{
if (unlikely(i_n1 + i_n2 >= len1 + len2))
throw DB::Exception(ErrorCodes::DECIMAL_OVERFLOW, "Numeric overflow: result bigger that Decimal256");
UInt16 sum = num1[i] * num2[j] + result[i_n1 + i_n2] + carry;
carry = sum / 10;
result[i_n1 + i_n2] = sum % 10;
++i_n2;
}and long division with a running remainder:
// src/Functions/FunctionsDecimalArithmetics.h:88
static VectorWithMemoryTracking<UInt8> divide(const VectorWithMemoryTracking<UInt8> & number, const Int256 & divisor)
{
VectorWithMemoryTracking<UInt8> result;
const auto max_index = number.size() - 1;
UInt16 idx = 0;
Int256 temp = 0;
while (temp < divisor && max_index >= idx)
{
temp = temp * 10 + number[idx];
++idx;
}
if (unlikely(temp == 0))
return {0};
while (max_index >= idx)
{
result.push_back(static_cast<UInt8>(temp / divisor));
temp = (temp % divisor) * 10 + number[idx];
++idx;
}
result.push_back(static_cast<UInt8>(temp / divisor));
return result;
}The scale shift in this engine is beautiful in its bluntness: it is a matter of pushing and popping zero digits.
// src/Functions/multiplyDecimal.cpp:33
VectorWithMemoryTracking<UInt8> multiplied = DecimalOpHelpers::multiply(a_digits, b_digits);
UInt16 product_scale = scale_a + scale_b;
while (product_scale < result_scale)
{
multiplied.push_back(0);
++product_scale;
}
while (product_scale > result_scale&& !multiplied.empty())
{
multiplied.pop_back();
--product_scale;
}
if (multiplied.empty())
return Decimal256(0);
if (multiplied.size() > DecimalUtils::max_precision<Decimal256>)
throw DB::Exception(ErrorCodes::DECIMAL_OVERFLOW, "Numeric overflow: result bigger that Decimal256");
return Decimal256(sign_a * sign_b * DecimalOpHelpers::fromDigits(multiplied));Multiply, then append zeros to reach the requested scale, or chop digits off the end to reduce it. The decimal point is literally a position in a vector of digits. Division does the same thing on the dividend:
// src/Functions/divideDecimal.cpp:33
VectorWithMemoryTracking<UInt8> a_digits = DecimalOpHelpers::toDigits(a.value * sign_a);
while (scale_a < scale_b + result_scale)
{
a_digits.push_back(0);
++scale_a;
}
while (scale_a > scale_b + result_scale && !a_digits.empty())
{
a_digits.pop_back();
--scale_a;
}This engine is the one with a complete overflow story: the digit vector is
of unbounded length, and the overflow check is “did the result need more than
76 digits”. That is why multiplyDecimal and divideDecimal can promise an
exact answer where the ordinary operators cannot:
SELECT multiply(toDecimal64(-12.647, 3), toDecimal32(2.1239, 4));┌─multiply(toDecimal64(-12.647, 3), toDecimal32(2.1239, 4))─┐
│ -26.8609633 │
└───────────────────────────────────────────────────────────┘
┌─multiplyDecimal(toDecimal64(-12.647, 3), toDecimal32(2.1239, 4))─┐
│ -26.8609 │
└──────────────────────────────────────────────────────────────────┘The ordinary * keeps the full product’s scale (3 + 4 = 7) and fits;
multiplyDecimal was asked for scale 4 and rounds to -26.8609.
DateTime64 is the same machine
It is worth noting that DateTime64 is not a special case bolted onto the
decimal machinery — it is the decimal machinery:
// base/base/Decimal.h:153
/// Distinguishable type to allow function resolution/deduction based on value type,
/// but also relatively easy to convert to/from Decimal64.
class DateTime64 : public Decimal64
{
public:
using Base = Decimal64;
using Base::Base;
using NativeType = Base::NativeType;
constexpr DateTime64(const Base & v): Base(v) {} // NOLINT(google-explicit-constructor)
};DateTime64(3) is a Decimal64 with scale 3, and its value is a count of
milliseconds since the epoch. Which is exactly what makes the DateTime64
timezone story work: the scale is part of the type, and the timezone is
not, because the timezone does not affect the integer.
Trade-offs
What the design buys:
- Exactness.
Decimal(38, 18)is exact to the last bit, and the arithmetic on it is exact too (for widths where the check works). No binary approximation, no0.1 + 0.2 != 0.3. - The scale is metadata. The value is an integer, so the column is a
PODArrayof integers, it compresses, it sorts, it sums, and it ismemcpy-able.getSizeOfValueInMemory()issizeof(T). - Scale-aware types.
Decimal(38, 18)andDecimal(38, 6)are different types, so a schema records the precision the author intended. - A second engine for the hard cases.
multiplyDecimal/divideDecimaluse unbounded digit arithmetic and can therefore offer an exact result with an arbitrary scale, at the cost of speed.
What it costs:
- The scale is not free. Every operation is a multiplication by a power of
ten, and the result scale of a division is a choice (
sa), not the natural one.a / bat scalesais not the same asa / bat scalesb, and users have to know that. - The overflow story is uneven.
Decimal32andDecimal64raiseDECIMAL_OVERFLOW;Decimal128andDecimal256wrap silently, and produce a plausible-looking wrong number rather than an obvious garbage one. The repository’s own test encodes that behaviour as expected output. - The check is a heuristic. The division overflow check is on the scales, and the comment says outright that it “doesn’t detect all possible overflow during big decimal division”.
- Two engines, two behaviours.
a * bandmultiplyDecimal(a, b)are different functions with different overflow semantics, different result scales, and different performance. The fast one is the one with the hole. - Narrowing is a separate hazard. The operation runs in the result’s
width, so an operand that is too wide has to be checked separately, and
the check that was added for it (
castToNative) exists because the original behaviour answered-1.5fortoDecimal32(1.5, 4) * 9223372036854775807.
The bargain, in one sentence: ClickHouse’s Decimal is a well-designed type — an integer plus a scale, with the scale in the type and the arithmetic reduced to shifts — attached to a value layer whose overflow detection is only as wide as the compiler’s builtins, which stop at 64 bits.