Polars Shares Columns Instead of Copying Them: Arc, Bytes and Reference Counting

TL;DR: Zero-copy in Polars is not one mechanism but three stacked ones. Series is an Arc<dyn SeriesTrait>, ChunkedArray holds Arc<Field> plus a Vec of chunk handles, and the actual bytes live in Buffer<T>, which carries a hand-rolled AtomicU64 refcount with Arc’s memory ordering. Cloning any of these copies handles, never bytes. The Arrow C data interface rides on the same mechanism: an imported buffer does not own its memory, it owns the C array that owns the memory, and releasing the buffer releases the array.

The word “zero-copy” gets used loosely in dataframe discussions, usually to mean “we avoided one memcpy somewhere”. In Polars it means something much more specific: at no point in the pipeline does the engine allocate a second copy of a column’s bytes in order to hand the column to someone else. What moves around is metadata — pointers, lengths, indices — and occasionally a reference to the same bytes.

There are three layers to get right, and each one has a different tool.

Layer 1: the Series is an Arc

The most visible layer is the user-facing one. A Series in Polars is a single field:

pub struct Series(pub Arc<dyn SeriesTrait>);

crates/polars-core/src/series/mod.rs:151

That is the entire definition. Cloning a Series bumps a refcount. Passing a Series to a function that takes it by value does not move the data, it moves the handle. This is the layer the Python user sees: df["a"] hands back a Series that shares the column’s buffers with the DataFrame it came from, and both can be alive at once without either being invalidated.

The next layer down is the chunked array, which is what a Series wraps:

pub struct ChunkedArray<T: PolarsDataType> {
    pub(crate) field: Arc<Field>,
    pub(crate) chunks: Vec<ArrayRef>,

    pub(crate) flags: StatisticsFlagsIM,

    length: usize,
    null_count: usize,
    _pd: std::marker::PhantomData<T>,
}

crates/polars-core/src/chunked_array/mod.rs:140

The name (Arc<Field>) is shared between chunks; the chunks themselves are ArrayRefs, and ArrayRef is not even an Arc:

pub type ArrayRef = Box<dyn Array>;

crates/polars-arrow/src/array/mod.rs:789

This looks like it should be a copy, but it is not. Box<dyn Array> is a polymorphic handle: cloning it dispatches to the concrete array’s Clone implementation, and every concrete array in polars-arrow stores its data in Buffer<T>. Cloning a PrimitiveArray<u32> clones a Buffer<u32>, and cloning a Buffer<u32> is a refcount increment. So the Vec<ArrayRef> clone allocates a new vector and a new box per chunk, and copies zero bytes of column data.

The ChunkedArray clone says so explicitly:

impl<T: PolarsDataType> Clone for ChunkedArray<T> {
    fn clone(&self) -> Self {
        ChunkedArray {
            field: self.field.clone(),
            chunks: self.chunks.clone(),
            flags: self.flags.clone(),

            _pd: Default::default(),
            length: self.length,
            null_count: self.null_count,
        }
    }
}

crates/polars-core/src/chunked_array/mod.rs:1051

The field is an Arc, so cloning it is a refcount bump; the chunks clone is the Vec plus the per-chunk Box<dyn Array> clones, which as established are refcount bumps one level down.

Layer 2: the bytes, and who owns them

The bytes live in Buffer<T>, and the ownership question is answered by SharedStorage. The full enum, from crates/polars-buffer/src/storage.rs:40:

enum BackingStorage {
    Vec {
        original_capacity: usize, // Elements, not bytes.
        vtable: &'static VecVTable,
    },
    ForeignOwner(Box<dyn Any + Send + 'static>),

    /// Backed by some external method which we do not need to take care of,
    /// but we still should refcount and drop the SharedStorageInner.
    External,

    /// Both the backing storage and the SharedStorageInner are leaked, no
    /// refcounting is done. This technically should be a flag on
    /// SharedStorageInner instead of being here, but that would add 8 more
    /// bytes to SharedStorageInner, so here it is.
    Leaked,
}

crates/polars-buffer/src/storage.rs:38

This enum is the answer to “how does polars-arrow distinguish owned from borrowed data?”:

• Vec — Polars allocated these bytes with a Vec and will free them.
• ForeignOwner — the bytes are owned by somebody else, and Polars keeps the owner alive in a Box<dyn Any> purely so it can be dropped later.
• External — the bytes are borrowed from a slice; Polars only manages lifetime, never frees.
• Leaked — nobody owns them and nobody frees them.

The ForeignOwner variant is what makes the Arrow FFI work (§4 below), and it is also what Buffer::from_owner uses:

    /// Creates a [`Buffer`] backed by `owner`.
    pub fn from_owner<O: Send + AsRef<[T]> + 'static>(owner: O) -> Self {
        Self::from_storage(SharedStorage::from_owner(owner))
    }

crates/polars-buffer/src/buffer.rs:122

A Buffer can therefore be about memory that Polars did not allocate, and the refcount still works, because the refcount is on Polars’ bookkeeping object, not on the bytes.

Layer 3: the refcount itself

Polars does not use std::sync::Arc for buffers, and the reason is written into the code — it needs a vtable and a leaked mode that Arc does not have. But the atomic protocol is Arc’s, down to the ordering:

impl<T> Clone for SharedStorage<T> {
    fn clone(&self) -> Self {
        let inner = self.inner();
        if !matches!(inner.backing, BackingStorage::Leaked) {
            // Ordering semantics copied from Arc<T>.
            inner.ref_count.fetch_add(1, Ordering::Relaxed);
        }
        Self {
            inner: self.inner,
            phantom: PhantomData,
        }
    }
}

impl<T> Drop for SharedStorage<T> {
    fn drop(&mut self) {
        let inner = self.inner();
        if matches!(inner.backing, BackingStorage::Leaked) {
            return;
        }

        // Ordering semantics copied from Arc<T>.
        if inner.ref_count.fetch_sub(1, Ordering::Release) == 1 {
            std::sync::atomic::fence(Ordering::Acquire);
            unsafe {
                self.drop_slow();
            }
        }
    }
}

crates/polars-buffer/src/storage.rs:491 and crates/polars-buffer/src/storage.rs:483

The Relaxed increment is sound because a clone cannot publish a write that another thread has not yet seen; the Release decrement plus the Acquire fence is what makes the final owner’s destruction of the data safe against concurrent readers.

Because the refcount lives in a heap allocation next to the data (SharedStorageInner), the handle — a Buffer<T> — is three words wide and trivially copyable, and a SharedStorage<T> is a single pointer.

Slicing: the payoff

The reason all of this exists is slicing. Buffer slicing is a pointer bump:

    /// Slices this buffer to the given range.
    ///
    /// # Panics
    /// Panics iff the range is out of bounds.
    #[inline]
    pub fn slice_in_place<R: RangeBounds<usize>>(&mut self, range: R) {
        unsafe {
            let Range { start, end } = check_range(range, ..self.len());
            self.ptr = self.ptr.add(start);
            self.length = end - start;
        }
    }

crates/polars-buffer/src/buffer.rs:218

So df.head(5), df["a"].slice(.., 0, 5), a filter that keeps a prefix, or a lazy plan that pushes a slice predicate down to the scan — none of them copy data. They produce a new handle onto the same bytes, and the refcount keeps those bytes alive for exactly as long as somebody needs them. The bytes of a sliced-out region stay allocated until the last handle referencing the storage dies, which is the classic cost of this design: a small slice of a large column pins the whole allocation.

The FFI: importing and exporting without copying

This is where the design pays off most visibly, because the Arrow C data interface is exactly a zero-copy contract: the producer hands out pointers into its own memory, and the consumer must not free them.

Export

Exporting a Polars array to C is a matter of pointing at the buffers that already exist. The Rust array itself is parked in the private_data field so that it outlives the C struct:

#[allow(dead_code)]
struct PrivateData {
    array: Box<dyn Array>,
    buffers_ptr: Box<[*const std::os::raw::c_void]>,
    children_ptr: Box<[*mut ArrowArray]>,
    dictionary_ptr: Option<*mut ArrowArray>,
    variadic_buffer_sizes: Box<[i64]>,
}

crates/polars-arrow/src/ffi/array.rs:91

and the ArrowArray struct is filled in with those very pointers:

        let buffers_ptr = buffers
            .iter()
            .map(|maybe_buffer| match maybe_buffer {
                Some(b) => *b as *const std::os::raw::c_void,
                None => std::ptr::null(),
            })
            .collect::<Box<[_]>>();
        let n_buffers = buffers.len() as i64;

crates/polars-arrow/src/ffi/array.rs:135

The buffers come straight out of the array:

impl<T: ViewType + ?Sized> ToFfi for BinaryViewArrayGeneric<T> {
    fn buffers(&self) -> Vec<Option<*const u8>> {
        let mut buffers = Vec::with_capacity(self.buffers.len() + 2);
        buffers.push(self.validity.as_ref().map(|x| x.as_ptr()));
        buffers.push(Some(self.views.storage_ptr().cast::<u8>()));
        // Export at the offset_adjusted pointer, not the allocation start. Arises whenever
        // `binary_to_binview` sees an inlined value before the first non-inlined one. See #28612.
        buffers.extend(self.buffers.iter().map(|b| Some(b.as_ptr())));
        buffers
    }

crates/polars-arrow/src/array/binview/ffi.rs:10

No bytes are copied on export; the consumer gets a pointer to the same memory Polars is using. The lifetime is handled by the release callback, which frees the Rust array when the consumer is done:

// callback used to drop [ArrowArray] when it is exported
unsafe extern "C" fn c_release_array(array: *mut ArrowArray) {
    if array.is_null() {
        return;
    }
    let array = &mut *array;

    // take ownership of `private_data`, therefore dropping it
    let private = Box::from_raw(array.private_data as *mut PrivateData);
    for child in private.children_ptr.iter() {
        let _ = Box::from_raw(*child);
    }

    if let Some(ptr) = private.dictionary_ptr {
        let _ = Box::from_raw(ptr);
    }

    array.release = None;
}

crates/polars-arrow/src/ffi/array.rs:76

private_data holds the Box<dyn Array>, so dropping PrivateData drops the array, which drops its buffers, which decrements the refcounts. The consumer of the C array never has to know any of this — it just calls release.

Import

Importing is the interesting direction, because the memory belongs to somebody else. Polars does not copy it; it borrows it and keeps the C array alive as the owner:

unsafe fn create_buffer_known_len<T: NativeType>(
    array: &ArrowArray,
    dtype: &ArrowDataType,
    owner: InternalArrowArray,
    len: usize,
    index: usize,
) -> PolarsResult<Buffer<T>> {
    if len == 0 {
        // Zero-length arrays might have invalid pointers for zero-length slices in Rust,
        // so this is more than just an optimization.
        return Ok(Buffer::new());
    }
    let ptr: *mut T = get_buffer_ptr(array, dtype, index)?;
    let slice = core::slice::from_raw_parts(ptr, len);
    let storage = SharedStorage::from_slice_with_owner(slice, owner);
    Ok(Buffer::from_storage(storage))
}

crates/polars-arrow/src/ffi/array.rs:269

SharedStorage::from_slice_with_owner is the ForeignOwner path: the bytes are not Polars’, but Polars holds an InternalArrowArray (which is an Arc<ArrowArray>) as the owner:

    /// # Safety
    /// The slice must be valid as long as owner lives.
    pub unsafe fn from_slice_with_owner<O: Send + 'static>(slice: &[T], owner: O) -> Self {
        #[expect(clippy::manual_slice_size_calculation)]
        let length_in_bytes = slice.len() * size_of::<T>();
        let ptr = slice.as_ptr().cast_mut();
        let inner = SharedStorageInner {
            ref_count: AtomicU64::new(1),
            ptr,
            length_in_bytes,
            backing: BackingStorage::ForeignOwner(owner),
            phantom: PhantomData,
        };
        Self {
            inner: NonNull::new(Box::into_raw(Box::new(inner))).unwrap(),
            phantom: PhantomData,
        }
    }

crates/polars-buffer/src/storage.rs:216

So the ownership chain is: the Polars Buffer holds a SharedStorageInner which holds the InternalArrowArray which holds an Arc<ArrowArray> which owns the C-side memory. When the last Polars handle dies, the inner is dropped, the Arc<ArrowArray> refcount hits zero, and the C array’s release callback runs — freeing the foreign memory exactly once.

There is exactly one case where the import does copy, and it is a genuine correctness requirement rather than a shortcut:

    // We have to check alignment, for zero-copy to be valid.
    if ptr.is_aligned() {
        let slice = core::slice::from_raw_parts(ptr.add(offset), len);
        let storage = SharedStorage::from_slice_with_owner(slice, owner);
        Ok(Buffer::from_storage(storage))
    } else {
        // Byte-wise copy for misaligned buffers.
        let mut v = Vec::with_capacity(len);
        core::ptr::copy_nonoverlapping(
            ptr.add(offset).cast::<u8>(),
            v.spare_capacity_mut().as_mut_ptr().cast::<u8>(),
            len * size_of::<T>(),
        );
        v.set_len(len);
        Ok(Buffer::from(v))
    }

crates/polars-arrow/src/ffi/array.rs:269

A misaligned buffer cannot be turned into a &[T] soundly, so Polars copies it into a fresh Vec and takes ownership of that instead. This is the “owned vs borrowed” distinction in action, decided per buffer at import time.

The FFI entry points are thin wrappers:

/// Exports an [`Box<dyn Array>`] to the C data interface.
pub fn export_array_to_c(array: Box<dyn Array>) -> ArrowArray {
    ArrowArray::new(bridge::align_to_c_data_interface(array))
}

crates/polars-arrow/src/ffi/mod.rs:22

/// Imports an [`Array`] from the C data interface.
///
/// # Safety
/// This function is intrinsically `unsafe` and relies on a [`ArrowArray`]
/// being valid according to the [C data interface](https://arrow.apache.org/docs/format/CDataInterface.html) (FFI).
pub unsafe fn import_array_from_c(
    array: ArrowArray,
    dtype: ArrowDataType,
) -> PolarsResult<Box<dyn Array>> {
    try_from(InternalArrowArray::new(array, dtype))
}

crates/polars-arrow/src/ffi/mod.rs:39

import_array_from_c consumes the ArrowArray — it is moved into an Arc, so the release callback will fire exactly once, when the last Polars-side reference goes away. Ownership transfer, not copying.

What copies, and what does not

Now the honest accounting.

Never copies the bytes:

• Cloning a Series (Arc bump), a ChunkedArray (handle clone), or an array (Buffer clone).
• Slicing a buffer, a column, or a frame.
• Exporting to the Arrow C interface.
• Importing from the Arrow C interface (when the buffer is properly aligned).

Copies, on purpose:

• Buffer::to_vec, the explicit escape hatch:

impl<T: Clone> Buffer<T> {
    pub fn to_vec(self) -> Vec<T> {
        match self.into_mut() {
            Either::Right(v) => v,
            Either::Left(same) => same.as_slice().to_vec(),
        }
    }
}

crates/polars-buffer/src/buffer.rs:357

• Misaligned FFI import (above).
• Any mutation of shared data, which has to break the sharing first.

Un-sharing on purpose: deshare and gc

Sharing is not free forever. If a column is sliced down to a fraction of its original size, the shared storage keeps the whole original allocation alive. The string array has an explicit escape hatch for that:

    /// Garbage collect
    pub fn gc(self) -> Self {
        if self.buffers.is_empty() {
            return self;
        }
        let mut mutable = MutableBinaryViewArray::with_capacity(self.len());
        let buffers = self.buffers.as_ref();

        for view in self.views.as_ref() {
            unsafe { mutable.push_view_unchecked(*view, buffers) }
        }
        mutable.freeze().with_validity(self.validity)
    }

    pub fn deshare(&self) -> Self {
        if self.buffers.storage_refcount() == 1
            && self.buffers.iter().all(|b| b.storage_refcount() == 1)
        {
            return self.clone();
        }
        self.clone().gc()
    }

crates/polars-arrow/src/array/binview/mod.rs:558

deshare checks the refcounts — the same storage_refcount exposed by Buffer:

    /// Since this takes a shared reference to self, beware that others might
    /// increment this after you've checked it's equal to 1.
    pub fn storage_refcount(&self) -> u64 {
        self.storage.refcount()
    }

crates/polars-buffer/src/buffer.rs:324

— and if anyone else is holding the data, it rebuilds the array, copying only the bytes that are still live. maybe_gc is the automatic version, with a 16 KiB minimum saving and a heuristic that avoids copying when the live fraction is too small to be worth it:

    pub fn maybe_gc(self) -> Self {
        const GC_MINIMUM_SAVINGS: usize = 16 * 1024; // At least 16 KiB.

        if self.total_buffer_len <= GC_MINIMUM_SAVINGS {
            return self;
        }

        if self.buffers.storage_refcount() != 1 {
            // There are multiple holders of this `buffers`.
            // If we allow gc in this case,
            // it may end up copying the same content multiple times.
            return self;
        }

crates/polars-arrow/src/array/binview/mod.rs:581

Note the second guard: garbage collecting while somebody else holds a reference would risk copying the same content more than once, so Polars skips it.

The bottom line

• Sharing is the default. Series is an Arc, ChunkedArray holds Arc<Field> and chunk handles, and the bytes sit in a Buffer with an AtomicU64 refcount that follows Arc’s ordering exactly.
• Slicing is free; it is a pointer bump. The cost is that a small slice pins the whole allocation, and Polars has to be told (or to guess) when to garbage collect.
• Ownership is explicit. BackingStorage distinguishes Vec (ours), ForeignOwner (somebody else’s, we keep the owner alive), External (borrowed) and Leaked (nobody’s). The FFI import path uses ForeignOwner, so an imported Arrow array is freed exactly once, by whoever drops the last Polars-side handle.
• Copies happen only when they must: an explicit to_vec, a misaligned FFI buffer, or an un-sharing (deshare/gc) that trades memory for independence.

The practical takeaway for someone writing Polars code: prefer sliced, clone and Arc over manual copying, and reach for deshare or maybe_gc when you know a column has been narrowed and the shared allocation is now mostly dead weight. If you need to move data across a process boundary, use the Arrow C data interface — it is the one place in the stack where the ownership rules are already worked out, and it moves pointers, not payloads.