Polars Stores Short Strings Inline in 16 Bytes: The View Type

TL;DR: Every string in a Polars string column is a 16-byte View. The first 4 bytes are the length, and the length is the tag: length <= 12 means the string’s bytes are stored in the remaining 12 bytes of the view itself, length > 12 means the view holds a 4-byte prefix plus a (buffer_idx, offset) pair that points into one of the column’s shared byte buffers. There is no separate tag byte, no pointer per string and no Arc<[u8]> per string — the spill target is a refcounted Buffer<u8> shared by the whole column.

Arrow’s Utf8 layout is a pair of arrays: an i32 offset buffer and one contiguous bytes buffer. It works, but it has two costs that matter at scale. The offsets cost 4 bytes per row, and every string access is two dependent loads — read the offset, then read the bytes — which serializes the memory system even when the CPU has plenty of bandwidth left.

Polars does not use Utf8 for string columns. It uses Utf8View, and the whole design of that type is about making the common case (short strings) free.

The type

rust
// We use this instead of u128 because we want alignment of <= 8 bytes.
/// A reference to a set of bytes.
///
/// If `length <= 12`, these bytes are inlined over the `prefix`, `buffer_idx` and `offset` fields.
/// If `length > 12`, these fields specify a slice of a buffer.
#[derive(Copy, Clone, Default)]
#[repr(C)]
pub struct View {
    /// The length of the string/bytes.
    pub length: u32,
    /// First 4 bytes of string/bytes data.
    pub prefix: u32,
    /// The buffer index.
    pub buffer_idx: u32,
    /// The offset into the buffer.
    pub offset: u32,
}

crates/polars-arrow/src/array/binview/view.rs:26

Sixteen bytes, four u32 fields, repr(C) so the layout is guaranteed. The inline threshold is right there:

rust
impl View {
    pub const MAX_INLINE_SIZE: u32 = 12;

    #[inline(always)]
    pub fn is_inline(&self) -> bool {
        self.length <= Self::MAX_INLINE_SIZE
    }

crates/polars-arrow/src/array/binview/view.rs:55

Twelve, not sixteen, because four bytes are spent on the length. The remaining twelve bytes of the view are the inline payload.

The exact bit layout

On a little-endian machine (and the code assumes little-endian in several places), the 16 bytes of a View mean two different things depending on the length:

text

INLINE  (length <= 12)
 byte 0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
      +---------------+-------------------------------+
      | length (u32)  | string bytes (up to 12)       |
      +---------------+-------------------------------+
                                       ^ zero padded to byte 15

SPILLED (length > 12)
 byte 0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
      +---------------+---------------+---------------+---------------+
      | length (u32)  | prefix (u32)  | buffer_idx    | offset (u32)  |
      +---------------+---------------+---------------+---------------+

The prefix field is doing double duty: for an inline view it is the first four bytes of the string; for a spilled view it is a copy of the first four bytes of the string, kept next to the metadata so that comparisons and hashes can look at it without chasing the buffer index. The full data of a spilled string lives in buffers[buffer_idx][offset .. offset + length].

Note what is not there: no tag byte, no pointer, no length-prefixed encoding inside the inline payload. The length field is the tag. The branch that distinguishes the two representations is a single compare against a compile-time constant, and it is the same branch everywhere in the codebase.

Building an inline view

The trick that makes this work is that the payload is written over the metadata fields, using raw pointer arithmetic on the view itself:

rust
    /// Create a new inline view without verifying the length
    ///
    /// # Safety
    ///
    /// It needs to hold that `bytes.len() <= View::MAX_INLINE_SIZE`.
    #[inline]
    pub unsafe fn new_inline_unchecked(bytes: &[u8]) -> Self {
        debug_assert!(bytes.len() <= u32::MAX as usize);
        debug_assert!(bytes.len() <= Self::MAX_INLINE_SIZE as usize);

        let mut view = Self {
            length: bytes.len() as u32,
            ..Default::default()
        };

        let view_ptr = &mut view as *mut _ as *mut u8;

        // SAFETY:
        // - bytes length <= 12,
        // - size_of::<View> == 16
        // - View is laid out as [length, prefix, buffer_idx, offset] (using repr(C))
        // - By grabbing the view_ptr and adding 4, we have provenance over prefix, buffer_idx and
        // offset. (i.e. the same could not be achieved with &mut self.prefix as *mut _ as *mut u8)
        unsafe {
            let inline_data_ptr = view_ptr.add(4);
            core::ptr::copy_nonoverlapping(bytes.as_ptr(), inline_data_ptr, bytes.len());
        }
        view
    }

crates/polars-arrow/src/array/binview/view.rs:92

The safety comment is worth reading twice, because it explains a subtlety of Rust’s aliasing rules: you cannot get a &mut u8 to the prefix field and then treat it as a *mut u8 with 12 bytes of writable memory behind it — prefix is only 4 bytes, and the compiler is right to complain. Going through the whole view’s pointer and offsetting by 4 gives you provenance over the entire 12-byte tail, which is exactly the region being written.

new_inline_from_block is the branch-free variant, used when you already have a 16-byte block and want a view of its first len bytes:

rust
    /// Create an inline view from the first `len` bytes of `block` without branching on `len`.
    ///
    /// # Panics
    ///
    /// Panics if `len > View::MAX_INLINE_SIZE`.
    #[inline(always)]
    pub fn new_inline_from_block(block: [u8; 16], len: usize) -> Self {
        // Masks that keep the first `len` inline bytes and zero the rest.
        const MASKS: [[u8; 16]; View::MAX_INLINE_SIZE as usize + 1] = {
            let mut masks = [[0u8; 16]; View::MAX_INLINE_SIZE as usize + 1];
            let mut len = 0;
            while len <= View::MAX_INLINE_SIZE as usize {
                let mut i = 0;
                while i < len {
                    masks[len][4 + i] = 0xFF;
                    i += 1;
                }
                len += 1;
            }
            masks
        };

        let mask = MASKS[len];
        let mut raw = [0u8; 16];
        raw[4..].copy_from_slice(&block[..12]);
        for i in 0..16 {
            raw[i] &= mask[i];
        }
        // The length field is little-endian, like the rest of the view.
        raw[..4].copy_from_slice(&(len as u32).to_le_bytes());
        // SAFETY: `View` is `repr(C)` with the length first, then 12 inline bytes.
        unsafe { std::mem::transmute::<[u8; 16], View>(raw) }
    }

crates/polars-arrow/src/array/binview/view.rs:114

The mask table is a compile-time constant, so “zero the padding” becomes an AND with a 16-byte constant — no branch, and the compiler can vectorize the whole thing.

Spilling: the prefix and the buffer index

The spilled representation is created in one place:

rust
    /// Create a new inline view
    ///
    /// # Safety
    ///
    /// It needs to hold that `bytes.len() > View::MAX_INLINE_SIZE`.
    #[inline]
    pub unsafe fn new_noninline_unchecked(bytes: &[u8], buffer_idx: u32, offset: u32) -> Self {
        debug_assert!(bytes.len() <= u32::MAX as usize);
        debug_assert!(bytes.len() > View::MAX_INLINE_SIZE as usize);

        // SAFETY: The invariant of this function guarantees that this is safe.
        let prefix = unsafe { u32::from_le_bytes(bytes[0..4].try_into().unwrap_unchecked()) };
        Self {
            length: bytes.len() as u32,
            prefix,
            buffer_idx,
            offset,
        }
    }

crates/polars-arrow/src/array/binview/view.rs:148

and the public constructor is just a branch between the two:

rust
    #[inline]
    pub fn new_from_bytes(bytes: &[u8], buffer_idx: u32, offset: u32) -> Self {
        assert!(bytes.len() <= u32::MAX as usize);

        // SAFETY: We verify the invariant with the outer if statement
        unsafe {
            if bytes.len() <= Self::MAX_INLINE_SIZE as usize {
                Self::new_inline_unchecked(bytes)
            } else {
                Self::new_noninline_unchecked(bytes, buffer_idx, offset)
            }
        }
    }

crates/polars-arrow/src/array/binview/view.rs:163

The prefix is not decorative: the validation code uses it to detect corrupted or hand-crafted views, because a view whose prefix disagrees with the bytes in the buffer is a bug or an attack:

rust
            polars_ensure!(b.starts_with(&view.prefix.to_le_bytes()), ComputeError: "prefix does not match string data");

crates/polars-arrow/src/array/binview/view.rs:555

Where the spilled bytes live

They do not live in an Arc<[u8]> per string. The column keeps a list of byte buffers, and the view indexes into that list:

rust
pub struct BinaryViewArrayGeneric<T: ViewType + ?Sized> {
    dtype: ArrowDataType,
    views: Buffer<View>,
    buffers: Buffer<Buffer<u8>>,
    validity: Option<Bitmap>,
    phantom: PhantomData<T>,
    /// Total bytes length if we would concatenate them all.
    total_bytes_len: RelaxedCell<u64>,
    /// Total bytes in the buffer (excluding remaining capacity)
    total_buffer_len: usize,
}

crates/polars-arrow/src/array/binview/mod.rs:133

Buffer<Buffer<u8>> is a refcounted buffer whose elements are themselves refcounted buffers. So the layout of a string column is:

text

+---------------------------+   +-------------------------------+
| views: Buffer<View>       |   | buffers: Buffer<Buffer<u8>>   |
|                           |   |                               |
| [len][abc..][len][xyz..]  |-->|  [buffer 0: 64 KiB of bytes]  |
|  ... 16 bytes per row     |   |  [buffer 1: 64 KiB of bytes]  |
+---------------------------+   +-------------------------------+

The buffers are not one giant buffer: they are capped chunks, because a single contiguous buffer would have to be reallocated (and therefore copied) as the column grows. The chunking parameters are explicit:

rust
// Growth parameters of view array buffers.
pub const BINVIEW_ARROW_BUFFER_LEN_LIMIT: usize = i32::MAX as usize;
pub const BINVIEW_MAX_ROW_BYTE_LEN: usize = (u32::MAX - 1) as usize;
const DEFAULT_BLOCK_SIZE: usize = 8 * 1024;
const MAX_EXP_BLOCK_SIZE: usize = 16 * 1024 * 1024;

crates/polars-arrow/src/array/binview/mod.rs:55

i32::MAX is not arbitrary: the Arrow C data interface describes buffer lengths as i64 but the view’s offset is a u32 and the buffer length limit keeps everything within 2 GiB per buffer, so the offsets stay representable.

How the buffers grow

The mutable builder owns the buffer under construction and doubles it, with the clamp that keeps small columns small and huge columns bounded:

rust
            // We want to make sure that we never have to memcopy between buffers. So if the
            // current buffer is not large enough, create a new buffer that is large enough and try
            // to anticipate the larger size.
            if self.in_progress_buffer.len().saturating_add(bytes.len())
                > usize::min(
                    BINVIEW_ARROW_BUFFER_LEN_LIMIT,
                    self.in_progress_buffer.capacity(),
                )
            {
                const {
                    assert!(MAX_EXP_BLOCK_SIZE < BINVIEW_ARROW_BUFFER_LEN_LIMIT);
                }

                // Allocate a new buffer and flush the old buffer.
                let new_capacity = usize::max(
                    bytes.len(),
                    (self.in_progress_buffer.capacity() * 2)
                        .clamp(DEFAULT_BLOCK_SIZE, MAX_EXP_BLOCK_SIZE),
                );

                let in_progress = Vec::with_capacity(new_capacity);
                let flushed = std::mem::replace(&mut self.in_progress_buffer, in_progress);
                if !flushed.is_empty() {
                    self.completed_buffers.push(flushed.into())
                }
            }

crates/polars-arrow/src/array/binview/mutable.rs:329

The comment is the design goal in one sentence: never have to memcopy between buffers. Once a buffer is full it is flushed and never touched again, so every spilled string’s bytes are written exactly once and stay where they are. That is what makes offset stable for the lifetime of the column.

There is a second, older copy of this logic in View::new_with_buffers, which is used when building views against a raw Vec<Vec<u8>>:

rust
    /// # Safety
    ///
    /// It needs to hold that `bytes.len() > View::MAX_INLINE_SIZE`.
    #[cold]
    unsafe fn new_with_buffers_slow(
        bytes: &[u8],
        buffer_idx_offset: u32,
        buffers: &mut Vec<Vec<u8>>,
    ) -> Self {
        // Resize last buffer instead of making new buffer.
        const RESIZE_LIMIT: usize = 8192;

        if buffers.is_empty() {
            buffers.push(bytes.to_vec());
            return View::new_noninline_unchecked(bytes, buffer_idx_offset, 0);
        }

        let num_buffers = buffers.len();
        let old_cap = buffers.last().unwrap().capacity();
        let new_cap = (old_cap + old_cap).clamp(bytes.len(), u32::MAX as usize);
        if new_cap <= RESIZE_LIMIT {
            let buffer_idx = buffer_idx_offset + num_buffers as u32 - 1;
            let buf = buffers.last_mut().unwrap();
            let offset = buf.len() as u32;
            buf.reserve(new_cap - buf.len());
            buf.extend_from_slice(bytes);
            View::new_noninline_unchecked(bytes, buffer_idx, offset)
        } else {
            let buffer_idx = buffer_idx_offset + num_buffers as u32;
            let mut buf = Vec::with_capacity(new_cap);
            buf.extend_from_slice(bytes);
            buffers.push(buf);
            View::new_noninline_unchecked(bytes, buffer_idx, 0)
        }
    }

crates/polars-arrow/src/array/binview/view.rs:221

Same doubling, but with a RESIZE_LIMIT of 8 KiB deciding between growing the last buffer and starting a new one. The fast path above it — new_with_buffers — checks whether the last buffer still has room and appends in place.

Reading a string back

The read side is the mirror image, and it is where the “no pointer chase for short strings” payoff shows up:

rust
    /// Constructs a byteslice from this view.
    ///
    /// # Safety
    /// Assumes that this view is valid for the given buffers.
    #[inline]
    pub unsafe fn get_slice_unchecked<'a, B: AsRef<[u8]>>(&'a self, buffers: &'a [B]) -> &'a [u8] {
        unsafe {
            if self.length <= Self::MAX_INLINE_SIZE {
                self.get_inlined_slice_unchecked()
            } else {
                self.get_external_slice_unchecked(buffers)
            }
        }
    }

crates/polars-arrow/src/array/binview/view.rs:243

and the inline case is a slice of the view’s own memory:

rust
    /// Construct a byte slice from an inline view.
    ///
    /// # Safety
    /// Assumes that this view is inlinable.
    #[inline]
    pub unsafe fn get_inlined_slice_unchecked(&self) -> &[u8] {
        debug_assert!(self.length <= View::MAX_INLINE_SIZE);
        let ptr = self as *const View as *const u8;
        unsafe { std::slice::from_raw_parts(ptr.add(4), self.length as usize) }
    }

crates/polars-arrow/src/array/binview/view.rs:291

The spilled case is an index into the buffer list plus a slice:

rust
    /// Construct a byte slice from an external view.
    ///
    /// # Safety
    /// Assumes that this view is in the external buffers.
    #[inline]
    pub unsafe fn get_external_slice_unchecked<'a, B: AsRef<[u8]>>(
        &self,
        buffers: &'a [B],
    ) -> &'a [u8] {
        debug_assert!(self.length > View::MAX_INLINE_SIZE);
        let data = buffers.get_unchecked(self.buffer_idx as usize);
        let offset = self.offset as usize;
        data.as_ref()
            .get_unchecked(offset..offset + self.length as usize)
    }

crates/polars-arrow/src/array/binview/view.rs:299

For an inline string, reading the bytes is a load from the same cache line that holds the length. For a spilled string, it is two dependent loads (buffers[buffer_idx], then the bytes) plus the data itself. That is the trade the whole type exists to make.

Comparing and hashing

Because a View is Pod and 16 bytes, it can be compared as a single u128:

rust
impl PartialEq for View {
    #[inline(always)]
    fn eq(&self, other: &Self) -> bool {
        self.as_u128() == other.as_u128()
    }
}

crates/polars-arrow/src/array/binview/view.rs:456

For inline views this is exact, because the padding bytes are guaranteed zero — the constructor masks them, and the validator rejects a view whose padding is non-zero:

rust
        if let Some(inline_slice) = view.get_inlined_slice() {
            if view.length < View::MAX_INLINE_SIZE && view.as_u128() >> (32 + view.length * 8) != 0
            {
                polars_bail!(ComputeError: "view contained non-zero padding in prefix");
            }

crates/polars-arrow/src/array/binview/view.rs:537

For spilled views, comparing the raw u128 is not enough (the same prefix can appear in different buffers), so any code that needs content equality branches on the same length test first. The hot grouper for string keys does exactly that: inline keys are compared as u128s, spilled keys as byte slices.

rust
            if view.is_inline() {
                self.table.insert_key(
                    hash,
                    (),
                    force_hot,
                    next_hash,
                    |_, b| view == b.1,

crates/polars-expr/src/hot_groups/binview.rs:51

and for the spilled branch:

rust
            } else {
                let bytes = view.get_external_slice_unchecked(buffers);
                self.table.insert_key(
                    hash,
                    (),
                    force_hot,
                    next_hash,
                    |_, b| {
                        // We only reach here if the hash matched, so jump straight to full comparison.
                        bytes == b.2
                    },

crates/polars-expr/src/hot_groups/binview.rs:67

The hash function branches the same way:

rust
    #[inline]
    pub unsafe fn hash_with_buffers_unchecked<B: AsRef<[u8]>, S: std::hash::BuildHasher>(
        &self,
        buffers: &[B],
        build_hasher: &S,
    ) -> u64 {
        unsafe {
            if self.length <= Self::MAX_INLINE_SIZE {
                build_hasher.hash_one(self.as_u128())
            } else {
                build_hasher.hash_one(self.get_external_slice_unchecked(buffers))
            }
        }
    }

crates/polars-arrow/src/array/binview/view.rs:259

Filling views at SIMD speed

Building a column of short strings is a memcpy problem, and Polars solves it with portable SIMD. extend_with_inlinable_strided writes 16-byte views directly into the destination buffer, using a shuffle to fan one 16-byte source load out into several views:

rust
                    // This way we can reuse the same load for multiple views.
                    repeat_with!(
                        I = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] => {
                            if I < BLOCKS_PER_LOAD {
                                let zero = u8x16::default();
                                const SWIZZLE: [usize; 16] = const {
                                    let mut swizzle = [16usize; 16];

                                    let mut i = 0;
                                    while i < N {
                                        let idx = i + I * N;
                                        if idx < 16 {
                                            swizzle[4+i] = idx;
                                        }
                                        i += 1;
                                    }

                                    swizzle
                                };

                                let scattered = simd_swizzle!(loaded, zero, SWIZZLE);
                                let view_bytes = (scattered | length_mask).to_array();

                                // SAFETY: dst has the capacity reserved and view_bytes is 16
                                // bytes long.
                                unsafe {
                                    core::ptr::copy_nonoverlapping(view_bytes.as_ptr(), dst, 16);
                                    dst = dst.add(16);
                                }
                            }
                        }
                    );

crates/polars-arrow/src/array/binview/view.rs:385

The length_mask ORs the length into the low byte of the view’s length field, and the shuffle places the string bytes at offset 4. The net effect: for a column of fixed-width short strings, the views are produced at memory bandwidth with no per-element branch.

A second, smaller inline type

The same trick appears elsewhere in the codebase at a smaller scale. SmallBytes in polars-utils inlines up to 8 bytes and uses the top bit of the length word as a null flag:

rust
/// `Option<Box<[u8]>>` with inlining of `<= size_of::<*mut u8>()` bytes.
pub struct SmallBytes {
    data: PointerOrArray,
    len: MarkedUsize,
}

crates/polars-utils/src/small_bytes.rs:19

rust
/// A usize where the top bit is used as a marked indicator.
#[derive(Copy, Clone, PartialEq, Eq, Hash)]
pub struct MarkedUsize(usize);

impl MarkedUsize {
    pub const UNMARKED_MAX: usize = ((1 << (usize::BITS - 1)) - 1);

    #[inline(always)]
    pub const fn new(idx: usize, marked: bool) -> Self {
        debug_assert!(idx >> (usize::BITS - 1) == 0);
        Self(idx | ((marked as usize) << (usize::BITS - 1)))
    }

crates/polars-utils/src/marked_usize.rs:1

It is used for partition keys in the streaming engine (crates/polars-stream/src/nodes/io_sinks/components/partition_key.rs:7), where the key is short, hashed per row, and must not allocate. For user-facing string values, Polars uses PlSmallStr, which is a thin wrapper around compact_str::CompactString and inlines up to 24 bytes on 64-bit platforms.

The bottom line

The View type is a 16-byte union without the union:

Cost. Every string costs 16 bytes of view, even a 1-character one, plus the validity bit and the 8 bytes of ChunkedArray bookkeeping. A column of short strings uses more memory than a Utf8 column with the same values.
Benefit. A short string costs no extra memory at all beyond its view — no offset array, no per-string allocation, no pointer chase. Reading it is a load from the cache line you already touched to get the length.
Cost. The length field is the tag, so every consumer pays one well-predicted branch. Code that wants to be branch-free has to specialize (see the as_u128 equality and the SIMD builder).
Benefit. Spilled bytes are written once and never moved: buffers are flushed at 8 KiB and doubled up to 16 MiB, and the buffer_idx/offset pair is stable for the lifetime of the column, which is what makes zero-copy slicing and sharing of the whole column possible.
Cost. A View is 4-byte aligned, not 8 or 16, because it is four u32 fields — kernels that want to load views as u64s must use the Bytes16Alignment4 wrapper from polars-arrow/src/types/aligned_bytes.rs and do their own unaligned loads.

The one-line summary: Polars optimises the common string, not the average one. If your strings are short — country codes, enum labels, UUID prefixes, hashes, IPs — the Utf8View layout is close to free, and the Arrow Utf8 layout would be paying 4 bytes per row for offsets you do not need.