Nulls Are Not Values: How DataFusion Skips Work Using Arrow's Validity Bitmaps

TL;DR: In Arrow, a null is not a value in the data buffer — it is a bit in a separate validity bitmap, and None for the bitmap means “no nulls anywhere”, which is not the same as “all bits set”. That separation is a performance lever, because it lets DataFusion ask a single cheap question (“does this array have any nulls at all?”) and, based on the answer, skip an entire category of per-row work: null-aware comparisons, null-checking appends, null-aware bit unions, and the whole null-buffer allocation. The trick appears in the GROUP BY builders, in the AND/OR short-circuit logic, in IN list evaluation, and in the scatter/filter primitives.

The representation

Arrow’s NullBuffer is a bit-packed validity bitmap plus a cached null count:

rust
// arrow-buffer-60.0.0/src/buffer/null.rs:33-44
#[derive(Debug, Clone, Eq, PartialEq)]
pub struct NullBuffer {
    buffer: BooleanBuffer,
    null_count: usize,
}

impl NullBuffer {
    /// Create a new [`NullBuffer`] computing the null count
    pub fn new(buffer: BooleanBuffer) -> Self {
        let null_count = buffer.len() - buffer.count_set_bits();
        Self { buffer, null_count }
    }

Two details matter enormously:

The bitmap is a validity bitmap. A set bit means “this slot has a value”. is_valid(idx) reads the bit; is_null(idx) is its negation.
The bitmap is optional. Array::nulls() returns Option<&NullBuffer>, and None means there are no nulls at all — not “all bits set”. This is the crucial optimization: a column of a million non-null Int32s has a values buffer of 4 MB and no validity bitmap whatsoever.

rust
// arrow-buffer-60.0.0/src/buffer/null.rs:46-64
    /// Create a new [`NullBuffer`] of length `len` where all values are null
    pub fn new_null(len: usize) -> Self {
        Self {
            buffer: BooleanBuffer::new_unset(len),
            null_count: len,
        }
    }

    /// Create a new [`NullBuffer`] of length `len` where all values are valid
    ///
    /// Note: it is more efficient to not set the null buffer if it is known to
    /// be all valid (aka all values are not null)
    pub fn new_valid(len: usize) -> Self {

The comment in Arrow’s own source says it outright: don’t materialize an all-ones bitmap; use None instead.

The “all null / all valid / some” trichotomy

Because null_count is cached, the question “is this array entirely null, entirely valid, or mixed?” is answerable in O(1) from the count and the length. DataFusion leans on this constantly, and encodes the answer as a three-valued enum:

rust
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs:1271-1279
///Shows how many `null`s there are in an array
enum Nulls {
    /// All array items are `null`s
    All,
    /// There are both `null`s and non-`null`s in the array items
    Some,
    /// There are no `null`s in the array items
    None,
}

Note the naming collision with Option::None — Nulls::None means no nulls, which is the opposite of Option::None meaning no bitmap. That naming is a small readability trap, but the semantics are consistent: both Nulls::None and Option::None mean “no nulls to worry about”.

Every GroupColumn implementation in the multi-column GROUP BY path computes this trichotomy once and then branches on it for the whole batch. The byte-array builder is the clearest example:

rust
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/bytes.rs:149-181
        let null_count = array.null_count();
        let num_rows = array.len();
        let all_null_or_non_null = if null_count == 0 {
            Nulls::None
        } else if null_count == num_rows {
            Nulls::All
        } else {
            Nulls::Some
        };

        match all_null_or_non_null {
            Nulls::Some => {
                for &row in rows {
                    self.append_val_inner::<B>(array, row)?
                }
            }

            Nulls::None => {
                self.nulls.append_n_non_nulls(rows.len());
                for &row in rows {
                    self.do_append_val_inner(arr, row)?;
                }
            }

            Nulls::All => {
                self.nulls.append_n_nulls(rows.len());

                let new_len = self.offsets.len() + rows.len();
                let offset = self.buffer.len();
                self.offsets.resize(new_len, O::usize_as(offset));
            }
        }

Look at what the three arms cost:

Nulls::Some: per-row, check is_null(row), append to the null builder, append the value.
Nulls::None: the null bitmap is not touched at all for the non-null case. append_n_non_nulls(n) is a length bump. The value loop has no null check in it.
Nulls::All: the values are not copied at all. The offsets buffer is resized with a repeated value and the string buffer is untouched. An all-null column of 10,000 strings costs 10,000 offset writes and zero bytes of string data.

The primitive builder does the same thing but goes further — it uses a const NULLABLE: bool type parameter so the null check is compiled out entirely for non-nullable columns:

rust
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/primitive.rs:171-188
    fn append_val(&mut self, array: &ArrayRef, row: usize) -> Result<()> {
        // Perf: skip null check if input can't have nulls
        if NULLABLE {
            if array.is_null(row) {
                self.nulls.append_null();
                self.group_values.push(T::default_value());
            } else {
                self.nulls.append_non_null();
                self.group_values
                    .push(array.as_primitive::<T>().value(row).canonicalize());
            }
        } else {
            self.group_values
                .push(array.as_primitive::<T>().value(row).canonicalize());
        }

        Ok(())
    }

if NULLABLE where NULLABLE is a const generic is not a runtime branch — it is a compile-time one. There are two monomorphizations of PrimitiveGroupValueBuilder, and the non-nullable one contains no null handling code at all.

The comparison side has the same structure, with an extra trick: a null-aware comparison can be short-circuited by nullness alone.

rust
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs:121-133
/// Determines if the nullability of the existing and new input array can be used
/// to short-circuit the comparison of the two values.
///
/// Returns `Some(result)` if the result of the comparison can be determined
/// from the nullness of the two values, and `None` if the comparison must be
/// done on the values themselves.
pub fn nulls_equal_to(lhs_null: bool, rhs_null: bool) -> Option<bool> {
    match (lhs_null, rhs_null) {
        (true, true) => Some(true),
        (false, true) | (true, false) => Some(false),
        _ => None,
    }
}

Some(true) = “both null, so equal (SQL GROUP BY treats nulls as one group)”. Some(false) = “one null, one not, so different”. None = “both non-null, fall through to the value comparison”. This is a three-way branch replacing a value comparison in 2 out of 3 cases involving nulls.

The `might_have_nulls` trick: a conservative bitmap

There is a subtle but important pattern in the group-value builders. The null builder is a NullBufferBuilder, and DataFusion wraps it in a trait that exposes a conservative nullness predicate:

rust
// datafusion/physical-plan/src/aggregates/group_values/null_builder.rs:40-45
    /// Returns true if this builder might have any nulls
    ///
    /// This is guaranteed to be true if there are nulls
    /// but may be true even if there are no nulls
    fn might_have_nulls(&self) -> bool;

implemented as:

rust
// datafusion/physical-plan/src/aggregates/group_values/null_builder.rs:88-90
    fn might_have_nulls(&self) -> bool {
        self.as_slice().is_some()
    }

NullBufferBuilder only materializes its underlying bitmap the first time a null is actually appended (Arrow’s materialize_if_needed). So as_slice() is Some iff a null was ever appended. This means might_have_nulls is true whenever the builder has definitely seen a null, and possibly true when it hasn’t — the fuzziness exists because the builder is reused across batches, and truncating its bitmap to preserve a precise “definitely has nulls” property would cost more than the conservative answer. The predicate is used to decide whether the nullable comparison path is required:

rust
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/primitive.rs:190-207
    fn vectorized_equal_to(
        &mut self,
        lhs_rows: &[usize],
        array: &ArrayRef,
        rhs_rows: &[usize],
        equal_to_results: &mut BooleanBufferBuilder,
    ) {
        if !NULLABLE || (array.null_count() == 0 && !self.nulls.might_have_nulls()) {
            self.vectorized_equal_to_non_nullable(
                lhs_rows,
                array,
                rhs_rows,
                equal_to_results,
            );
        } else {
            self.vectorized_equal_nullable(lhs_rows, array, rhs_rows, equal_to_results);
        }
    }

The non-nullable path is the fast one: it reads array_values[rhs_row] directly and packs the result into a bitmask with a word-at-a-time AND:

rust
// datafusion/physical-plan/src/aggregates/group_values/multi_group_by/primitive.rs:79-116
        let array_values = array.as_primitive::<T>().values();
        let n = lhs_rows.len();

        // Build a packed comparison bitmask, then AND it into equal_to_results
        let num_bytes = n.div_ceil(8);
        let mut cmp_buf = vec![0u8; num_bytes];

        for (i, (&lhs_row, &rhs_row)) in lhs_rows.iter().zip(rhs_rows.iter()).enumerate()
        {
            if !equal_to_results.get_bit(i) {
                continue;
            }
            let left = if cfg!(debug_assertions) {
                self.group_values[lhs_row]
            } else {
                unsafe { *self.group_values.get_unchecked(lhs_row) }
            };
            let right = if cfg!(debug_assertions) {
                array_values[rhs_row]
            } else {
                unsafe { *array_values.get_unchecked(rhs_row) }
            };
            // `left` was already canonicalized on append; canonicalize the
            // input so ±0 (and any future equivalence class) compares equal.
            if left.is_eq(right.canonicalize()) {
                cmp_buf[i / 8] |= 1 << (i % 8);
            }
        }

        // AND the comparison result into the existing equal_to_results bitmask
        apply_bitwise_binary_op(
            equal_to_results.as_slice_mut(),
            0,
            &cmp_buf,
            0,
            n,
            |a, b| a & b,
        );

Note the array_values[rhs_row] read: PrimitiveArray::values() returns the raw value buffer, with no null information in it. Reading a null slot’s value is safe (it holds whatever was in the buffer, usually a default), because the null-ness is tracked separately in the bitmap and checked before the value is ever used. This is the whole point of the separation: the value buffer is dense and unbranched, and null-ness lives somewhere else.

Boolean filters: nulls as false, at the bitmap level

Filter predicates are the other place nulls show up constantly. In SQL, a NULL predicate is not TRUE, so it does not pass a WHERE clause. Arrow expresses this as a bitmap operation:

rust
// arrow-select-60.0.0/src/filter.rs:159-171
/// Convert all null values in `BooleanArray` to `false`
///
/// This is useful for filter-like operations which select only `true`
/// values, but not `false` or `NULL` values
///
/// Internally this is implemented as a bitwise `AND` operation with null bits
/// and the boolean bits.
pub fn prep_null_mask_filter(filter: &BooleanArray) -> BooleanArray {
    let nulls = filter.nulls().unwrap();
    let mask = filter.values() & nulls.inner();
    BooleanArray::new(mask, None)
}

filter.values() & nulls.inner() is a word-at-a-time AND of two bitmaps producing a new bitmap with no validity bitmap of its own (the nulls have been folded into the values). This is the entire implementation of SQL’s three-valued logic for filtering: one AND of two bitmaps.

DataFusion uses it in the places where a predicate’s nulls must not pass:

rust
// datafusion/physical-expr/src/expressions/binary.rs:449-455
        // only update the keys / rows if the filter does not match all rows
        if filter.null_count() > 0 || filter.has_false() {
            // Indices in `set_indices` should be correct if filter contains nulls
            // So we prepare the filter here. Note this is also done in the `FilterBuilder`
            // so there is no overhead to do this here.
            if filter.nulls().is_some() {
                filter = prep_null_mask_filter(&filter);
            }

and in the AND/OR short-circuit logic, where nulls have to be handled at the bit level rather than the row level:

rust
// datafusion/physical-expr/src/expressions/binary.rs:1541-1548
/// A null-free mask of the rows of `array` that are `true` or `NULL`.
fn not_false(array: &BooleanArray) -> BooleanArray {
    let values = match array.nulls() {
        Some(nulls) => array.values() | &!nulls.inner(),
        None => array.values().clone(),
    };
    BooleanArray::new(values, None)
}

array.values() | &!nulls.inner() — “true OR null” — is the set of rows that are not yet decided. Again, no per-row work; two bitmap operations.

And the “all false” test, used to short-circuit a whole AND chain:

rust
// datafusion/physical-expr/src/expressions/binary.rs:1515-1524
/// Whether every row is already `false`.
fn is_all_false(value: &ColumnarValue) -> bool {
    match value {
        ColumnarValue::Array(array) => {
            let array = array.as_boolean();
            array.null_count() == 0 && !array.has_true()
        }
        ColumnarValue::Scalar(scalar) => *scalar == ScalarValue::Boolean(Some(false)),
    }
}

null_count() == 0 && !has_true() — a single cached count plus a single scan of the values bitmap — answers “is this whole column false?” If so, the entire AND chain can stop.

`IN` lists: computing nullability as a bitmap expression

The IN list implementation is the most bitmap-heavy code in the physical expression layer, because SQL’s IN has genuinely subtle null semantics. DataFusion computes the “contains” bitmap first, then applies the null mask as pure bitmap algebra:

rust
// datafusion/physical-expr/src/expressions/in_list/result.rs:72-110
pub(super) fn build_result_from_contains(
    needle_nulls: Option<&NullBuffer>,
    haystack_has_nulls: bool,
    negated: bool,
    contains_buf: BooleanBuffer,
) -> BooleanArray {
    match (needle_nulls, haystack_has_nulls, negated) {
        // Haystack has nulls: result is null unless value is found.
        (Some(v), true, false) => {
            // values: valid & contains, nulls: valid & contains
            let values = v.inner() & &contains_buf;
            BooleanArray::new(values.clone(), Some(NullBuffer::new(values)))
        }
        (None, true, false) => {
            BooleanArray::new(contains_buf.clone(), Some(NullBuffer::new(contains_buf)))
        }
        (Some(v), true, true) => {
            // NOT IN with nulls: false if found, null if not found or needle null.
            // values: valid & !contains, nulls: valid & contains
            let valid = v.inner();
            let values = valid & &(!&contains_buf);
            let nulls = valid & &contains_buf;
            BooleanArray::new(values, Some(NullBuffer::new(nulls)))
        }
        (None, true, true) => {
            BooleanArray::new(!&contains_buf, Some(NullBuffer::new(contains_buf)))
        }
        // Haystack has no nulls: result validity follows needle validity.
        (Some(v), false, false) => {
            // values: valid & contains, nulls: valid
            BooleanArray::new(v.inner() & &contains_buf, Some(v.clone()))
        }
        (Some(v), false, true) => {
            // values: valid & !contains, nulls: valid
            BooleanArray::new(v.inner() & &(!&contains_buf), Some(v.clone()))
        }
        (None, false, false) => BooleanArray::new(contains_buf, None),
        (None, false, true) => BooleanArray::new(!&contains_buf, None),
    }
}

Read the last two arms: (None, false, false) and (None, false, true) produce a BooleanArray with no validity bitmap. If the needle isn’t null and the haystack has no nulls, the result is never null — so no bitmap is allocated. That is the “all valid ⇒ None” convention applied to a computed result, not just an input.

Filters and selection vectors

arrow::compute::filter is the other half of the null story, because a filter predicate is a BooleanArray — which has its own validity bitmap for the NULL slots — and the output has to compact.

Arrow’s FilterBuilder picks between two strategies depending on how selective the predicate is:

rust
// arrow-select-60.0.0/src/filter.rs:346-365
    fn default_strategy(filter_length: usize, filter_count: usize) -> Self {
        if filter_length == 0 || filter_count == 0 {
            return IterationStrategy::None;
        }

        if filter_count == filter_length {
            return IterationStrategy::All;
        }

        // Compute the selectivity of the predicate by dividing the number of true
        // bits in the predicate by the predicate's total length
        //
        // This can then be used as a heuristic for the optimal iteration strategy
        let selectivity_frac = filter_count as f64 / filter_length as f64;
        if selectivity_frac > FILTER_SLICES_SELECTIVITY_THRESHOLD {
            return IterationStrategy::SlicesIterator;
        }
        IterationStrategy::IndexIterator
    }

with the threshold at 0.8:

rust
// arrow-select-60.0.0/src/filter.rs:39-43
/// If the filter selects more than this fraction of rows, use
/// [`SlicesIterator`] to copy ranges of values. Otherwise iterate
/// over individual rows using [`IndexIterator`]
///
/// Threshold of 0.8 chosen based on <https://dl.acm.org/doi/abs/10.1145/3465998.3466009>
const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.8;

A selective filter produces a selection vector (a list of surviving row indices); a non-selective one produces a list of slices (contiguous ranges) and copies whole runs. This is a genuinely important distinction: for a 5%-selective filter, gathering 50,000 scattered rows from a million-row column is a random-access disaster; for a 95%-selective filter, copying 19 contiguous runs is nearly free.

DataFusion uses this all over the place — the FilterExec calls filter_record_batch, the TopK operator builds a FilterBuilder and uses filter.values().set_indices() to get the selection vector for its heap, and the range-partition router gathers rows with take_arrays in one shot per batch:

rust
// datafusion/physical-plan/src/repartition/mod.rs:1372-1388
        let batches = {
            let _timer = timer.timer();
            let indices_array: PrimitiveArray<UInt32Type> = reordered_indices.into();
            let columns = take_arrays(batch.columns(), &indices_array, None)?;

            let mut options = RecordBatchOptions::new();
            options = options.with_row_count(Some(indices_array.len()));
            let reordered_batch =
                RecordBatch::try_new_with_options(batch.schema(), columns, &options)?;

            partition_ranges
                .into_iter()
                .map(|(partition, start, len)| {
                    Ok((partition, reordered_batch.slice(start, len)))
                })
                .collect()
        };

One take for the whole batch, then each partition is a zero-copy slice of the reordered batch. Note that take does preserve the validity bitmap — the nulls follow the rows — so the null handling is implicit in the gather rather than a separate pass.

Where the physical `Column` fits in

The physical Column expression is the leaf that hands the bitmap-bearing array to everything else. It is deliberately trivial:

rust
// datafusion/physical-expr/src/expressions/column.rs:120-124
    /// Evaluate the expression
    fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
        self.bounds_check(batch.schema().as_ref())?;
        Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
    }

An Arc clone. The values buffer, the validity bitmap, the offsets — all shared, none copied, none inspected. This is what makes the whole design work: because Column does not look at the nulls, every consumer above it is free to make its own decision about whether nulls matter, and the decision is made once per batch rather than per row.

(A note on file layout: the Arrow helper utilities that older DataFusion versions kept under datafusion/common/src/arrow/ now live in datafusion/common/src/cast.rs — the as_boolean_array, as_string_array, as_dictionary_array family of downcasting helpers the physical expression layer leans on — and the null-handling helpers in datafusion/common/src/utils/mod.rs.)

Propagating a parent’s nulls to its children

Arrow’s nested types have a subtlety: a struct’s child arrays have their own validity bitmaps, and the struct’s own bitmap says which rows are null. A null struct row means all of its children are logically null, but the children’s bitmaps do not know that.

DataFusion has a helper that fixes this up while preserving the value-buffer sharing:

rust
// datafusion/common/src/utils/mod.rs:1573-1598
/// Apply a struct's nulls to one of its fields.
///
/// Arrays with validity bitmaps share their value buffers; unions and
/// run-end encoded arrays are rebuilt.
pub fn apply_parent_nulls(
    col: &ArrayRef,
    parent_nulls: Option<&NullBuffer>,
) -> Result<ArrayRef> {
    let Some(parent_nulls) = parent_nulls else {
        // If there are no parent nulls to apply, we can just return
        return Ok(Arc::clone(col));
    };

    // NullArray is already entirely null and cannot have a validity bitmap.
    // If we have 0 parent nulls, we can also avoid extra work.
    if col.data_type().is_null() || parent_nulls.null_count() == 0 {
        return Ok(Arc::clone(col));
    }

    if layout(col.data_type()).can_contain_null_mask {
        // `nullif` marks a row null where the mask is true and keeps the
        // field's own nulls. Only the validity bitmap is rebuilt; the value
        // buffers and child arrays are shared with `col`.
        let null_parents = BooleanArray::new(!parent_nulls.inner(), None);
        return Ok(nullif(col.as_ref(), &null_parents)?);
    }

Two shortcuts again: no parent nulls at all → return the column untouched; parent has zero nulls → also return untouched. Only when there is real work to do does it build the inverted mask (!parent_nulls.inner()) and rebuild the validity bitmap. And because the values are untouched, the operation is cheap even for a wide struct.

The can_contain_null_mask check is the important one: unions and run-end-encoded arrays have no validity bitmap of their own — their nulls live in their children — so those need a full rebuild via MutableArrayData instead. The helper is explicit about that:

rust
// datafusion/common/src/utils/mod.rs:1600-1613
    // Unions and run-end encoded arrays have no validity bitmap of their own
    // and represent nulls in their children. Rebuild the array so null parents
    // become null values in those children.
    let data = col.to_data();
    let mut mutable = MutableArrayData::new(vec![&data], true, data.len());
    let mut end = 0;
    for (start, valid_end) in parent_nulls.valid_slices() {
        mutable.try_extend_nulls(start - end)?;
        mutable.try_extend(0, start, valid_end)?;
        end = valid_end;
    }
    mutable.try_extend_nulls(data.len() - end)?;

    Ok(make_array(mutable.freeze()))
}

valid_slices() — the valid runs of the parent bitmap — is what makes this a bulk copy rather than a per-row loop.

The reverse operation: scatter

filter compacts; scatter expands. The two are inverses, and the null handling is where they differ. scatter puts the source values into the positions where the mask is true, and fills the rest with nulls:

rust
// datafusion/physical-expr-common/src/utils.rs:76-102
pub fn scatter(mask: &BooleanArray, truthy: &dyn Array) -> Result<ArrayRef> {
    let mask = match mask.null_count() {
        0 => Cow::Borrowed(mask),
        n if n == mask.len() => {
            return Ok(new_null_array(truthy.data_type(), mask.len()));
        }
        _ => Cow::Owned(prep_null_mask_filter(mask)),
    };

    let output_len = mask.len();

    // Fast path: no true values mean all-null object
    if !mask.has_true() {
        return Ok(new_null_array(truthy.data_type(), output_len));
    }

    // Fast path: all true means output = truthy
    if mask.null_count() == 0 && !mask.has_false() {
        return Ok(truthy.slice(0, truthy.len()));
    }

    let count = mask.true_count();
    let selectivity = count as f64 / output_len as f64;
    let mask_buffer = mask.values();

    scatter_array(truthy, mask_buffer, output_len, selectivity)
}

Two fast paths again: “mask is all false” → the answer is a null array (no copying of the source at all); “mask is all true” → the answer is a slice of the source (no copying either, just an offset).

The null mask of the scattered result is computed by a dedicated routine that knows the two interesting cases:

rust
// datafusion/physical-expr-common/src/utils.rs:205-225
fn scatter_null_mask(
    src_nulls: Option<&NullBuffer>,
    mask: &BooleanBuffer,
    output_len: usize,
    selectivity: f64,
) -> Option<NullBuffer> {
    let false_count = output_len - mask.count_set_bits();
    let src_null_count = src_nulls.map(|n| n.null_count()).unwrap_or(0);

    if src_null_count == 0 {
        if false_count == 0 {
            None
        } else {
            Some(NullBuffer::new(mask.clone()))
        }
    } else {
        let src_nulls = src_nulls.unwrap();
        let scattered = scatter_bits(src_nulls.inner(), mask, output_len, selectivity);
        Some(NullBuffer::new(scattered)).filter(|n| n.null_count() > 0)
    }
}

Three observations:

src_null_count == 0 && false_count == 0 → None. No nulls in the source and every output slot is filled → no validity bitmap needed.
src_null_count == 0 && false_count > 0 → the output’s validity bitmap is literally the mask. The slots that were not selected are null; the ones that were selected are valid. The mask doubles as the validity bitmap — no separate computation, no allocation beyond a NullBuffer wrapper.
The .filter(|n| n.null_count() > 0) at the end is the “don’t materialize an all-ones bitmap” rule applied to a computed result: if the scattered nulls turned out to be zero, drop the bitmap entirely.

Slicing a bitmap is not free

One more Arrow detail that bites people: slicing a validity bitmap is O(1) in the buffer but O(len) in the null count, because the count has to be recomputed:

rust
// arrow-buffer-60.0.0/src/buffer/null.rs:281-288
    /// Slices this [`NullBuffer`] by the provided `offset` and `length`
    ///
    /// # Panics
    ///
    /// Panics if `offset + len > self.len()`
    pub fn slice(&self, offset: usize, len: usize) -> Self {
        Self::new(self.buffer.slice(offset, len))
    }

Self::new recomputes null_count by count_set_bits() over the sliced region. The bitmap is not copied — BooleanBuffer::slice just bumps bit_offset — but the popcount is real work. So slicing a million-row array into a hundred thousand-row window costs a popcount over 12.5 KB, not over 125 KB, but it is not free either. This is why DataFusion’s RowValues and friends prefer to keep a whole array and track an offset where possible.

Where the separation leaks: logical vs physical nulls

The bitmap separation creates one genuinely nasty corner case, and DataFusion has a whole API for it: logical nulls.

For a dictionary array, a row can be physically valid (its key is not null) but logically null (its key points at a NULL entry in the dictionary’s values array). Arrow exposes both:

rust
// arrow-array-60.0.0/src/array/dictionary_array.rs:769-787
    fn logical_null_count(&self) -> usize {
        match (self.keys.nulls(), self.values.logical_nulls()) {
            (None, None) => 0,
            (Some(key_nulls), None) => key_nulls.null_count(),
            (None, Some(value_nulls)) => self
                .keys
                .values()
                .iter()
                .filter(|k| value_nulls.is_null(k.as_usize()))
                .count(),
            (Some(key_nulls), Some(value_nulls)) => self
                .keys
                .values()
                .iter()
                .enumerate()
                .filter(|(idx, k)| key_nulls.is_null(*idx) || value_nulls.is_null(k.as_usize()))
                .count(),
        }
    }

DataFusion’s hash join uses the logical count to decide whether the build side has nulls, because a physically-valid key pointing at a null value must be treated as a null key for NOT IN semantics:

rust
// datafusion/physical-plan/src/joins/hash_join/exec.rs:3620-3622
    let build_has_null = null_aware == Some(NullAwareMode::RightAnti)
        && !left_values.is_empty()
        && left_values[0].logical_null_count() > 0;

and it caches the logical nulls separately, because computing them can be expensive for nested dictionaries:

rust
// datafusion/physical-plan/src/joins/hash_join/exec.rs:358-362
    /// Logical nulls of the scalar key for uncorrelated null-aware `LeftMark`.
    ///
    /// Computed once while collecting the build side because dictionary
    /// logical-null discovery can scan nested dictionary values.
    left_mark_build_key_nulls: Option<NullBuffer>,

This is a real failure mode: an operator that uses null_count() where it meant logical_null_count() will treat a null-valued dictionary entry as a present value. There are explicit regression tests for it in the hash join test module (datafusion/physical-plan/src/joins/hash_join/exec.rs:9269+, test_..._logical_null_count style).

Failure modes and trade-offs

Option<NullBuffer> semantics are easy to get backwards. None means “no nulls”, not “all nulls”. Code that does nulls.map(|n| n.null_count()).unwrap_or(0) is correct; code that does nulls.is_none() => all nulls is catastrophically wrong. DataFusion’s Nulls enum exists partly to make the distinction explicit at the type level.
The cached null_count can go stale if you mutate the underlying buffer. NullBuffer is immutable once built, so this is mostly a non-issue, but hand-rolled NullBufferBuilder misuse can produce a count that disagrees with the bits.
might_have_nulls is deliberately fuzzy. Code that treats it as “definitely has nulls” will be correct but slow; code that treats it as “definitely has no nulls” will be wrong. The name is the warning.
The Nulls::All fast path skips the values entirely. Any code that later assumes the group’s value buffer contains the actual bytes will find zeros there. This is why the group-value builders keep the null bitmap and the values buffer strictly consistent — the “value” of a null group is defined as a default/empty value.
Row format encoding of nulls is a sentinel, not a bitmap. The row format writes a 0x00 (or 0xFF) byte for null in each field, so a null row is still a full-length row. That is a space cost compared to the bitmap representation, but it buys the memcmp-comparability described in the row-format article.
Dictionary logical nulls are O(D) to discover. logical_null_count has to scan the keys. For a dictionary with a million entries, that is a million reads, done once and cached — but it is not free, and DataFusion explicitly caches it rather than recomputing per batch.

The takeaway

The value/validity split in Arrow is not just a memory optimization; it is an information optimization. Because null-ness lives in a separate, bit-packed, countable buffer, a whole batch can be classified as all-null / all-valid / mixed in O(1), and the answer selects between radically different code paths: copy the values or don’t, allocate a validity bitmap or don’t, run a null-aware comparison or a raw value comparison, fold nulls into a mask with one AND or handle them row by row. DataFusion’s Nulls enum, its nulls_equal_to short-circuit, its might_have_nulls conservatism, Arrow’s prep_null_mask_filter, and the scatter_null_mask routine that reuses the selection mask as the validity bitmap are all instances of the same idea: decide once, at the batch level, whether nulls are a problem — and then take a completely different code path.