Polars Shares Data With Python Over the Arrow C Data Interface

TL;DR: polars-arrow implements the Arrow C Data Interface, which is a contract about pointers and ownership, not about bytes. Exporting a Polars array fills an ArrowArray struct whose buffers point straight at the existing memory, and parks the Rust array in private_data so a release callback can drop it later. Importing does the mirror image: the foreign bytes are borrowed, and the C array itself is kept alive as the owner. The only thing that gets copied is a buffer that cannot legally be borrowed.

Moving a dataframe between Polars and Python looks like it should be a memcpy. It is not, and the reason is that Arrow’s C Data Interface was designed to make the memcpy unnecessary. The interface is three C structs and a set of rules about who owns what; the rules are interesting mostly because they are subtle, and Polars’ implementation is a good place to see them written down.

The three structs

crates/polars-arrow/src/ffi/generated.rs is a rust-bindgen output file containing the ABI-compatible mirrors of the three Arrow C structs. The array one:

rust
/// ABI-compatible struct for [`ArrowArray`](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions)
#[repr(C)]
#[derive(Debug)]
pub struct ArrowArray {
    pub(super) length: i64,
    pub(super) null_count: i64,
    pub(super) offset: i64,
    pub(super) n_buffers: i64,
    pub(super) n_children: i64,
    pub(super) buffers: *mut *const ::std::os::raw::c_void,
    pub(super) children: *mut *mut ArrowArray,
    pub(super) dictionary: *mut ArrowArray,
    pub(super) release: ::std::option::Option<unsafe extern "C" fn(arg1: *mut ArrowArray)>,
    pub(super) private_data: *mut ::std::os::raw::c_void,
}

crates/polars-arrow/src/ffi/generated.rs:32

ArrowSchema has the same shape for types (format, name, metadata, flags, children, dictionary, release, private_data), and ArrowArrayStream is the pull-based variant with four function pointers (get_schema, get_next, get_last_error, release) instead of data pointers.

Every one of these has a release field, and that is the load-bearing part of the whole design. The contract is: the producer of the struct owns whatever the struct points at, and the consumer must call release exactly once when it is done. release is where the producer takes its memory back.

Exporting: no bytes move

Exporting an array is ArrowArray::new:

rust
    /// creates a new `ArrowArray` from existing data.
    ///
    /// # Safety
    /// This method releases `buffers`. Consumers of this struct *must* call `release` before
    /// releasing this struct, or contents in `buffers` leak.
    pub(crate) fn new(mut array: Box<dyn Array>) -> Self {

crates/polars-arrow/src/ffi/array.rs:103

The buffers array is built from the array’s own buffer pointers:

rust
        let buffers_ptr = buffers
            .iter()
            .map(|maybe_buffer| match maybe_buffer {
                Some(b) => *b as *const std::os::raw::c_void,
                None => std::ptr::null(),
            })
            .collect::<Box<[_]>>();
        let n_buffers = buffers.len() as i64;

crates/polars-arrow/src/ffi/array.rs:135

buffers here comes from the array’s ToFfi implementation. For a primitive array that is literally “the address of my values buffer”:

rust
unsafe impl<T: NativeType> ToFfi for PrimitiveArray<T> {
    fn buffers(&self) -> Vec<Option<*const u8>> {
        vec![
            self.validity.as_ref().map(|x| x.as_ptr()),
            Some(self.values.storage_ptr().cast::<u8>()),
        ]
    }

crates/polars-arrow/src/array/primitive/ffi.rs:9

storage_ptr() is the address of the allocation, not of the first element. No copy, no allocation, no serialization: the consumer gets a pointer into the same memory Polars is using.

Now the ownership question. The ArrowArray struct has no place to put a Box<dyn Array> — it is a plain C struct. So Polars parks it in private_data:

rust
#[allow(dead_code)]
struct PrivateData {
    array: Box<dyn Array>,
    buffers_ptr: Box<[*const std::os::raw::c_void]>,
    children_ptr: Box<[*mut ArrowArray]>,
    dictionary_ptr: Option<*mut ArrowArray>,
    variadic_buffer_sizes: Box<[i64]>,
}

crates/polars-arrow/src/ffi/array.rs:91

and hands the raw pointer to the C struct:

rust
        Self {
            length,
            null_count,
            offset: offset as i64,
            n_buffers,
            n_children,
            buffers: private_data.buffers_ptr.as_mut_ptr(),
            children: private_data.children_ptr.as_mut_ptr(),
            dictionary: private_data.dictionary_ptr.unwrap_or(std::ptr::null_mut()),
            release: Some(c_release_array),
            private_data: Box::into_raw(private_data) as *mut ::std::os::raw::c_void,
        }

crates/polars-arrow/src/ffi/array.rs:178

Note the children: a StructArray or ListArray has child arrays, and each child is exported as its own ArrowArray with its own release callback:

rust
        let children_ptr = children
            .into_iter()
            .map(|child| {
                Box::into_raw(Box::new(ArrowArray::new(ffi::align_to_c_data_interface(
                    child,
                ))))
            })
            .collect::<Box<_>>();
        let n_children = children_ptr.len() as i64;

crates/polars-arrow/src/ffi/array.rs:144

The child’s ArrowArray is leaked into a raw pointer and kept in children_ptr, so the parent’s release callback can free each child in turn. A consumer that wants a nested array’s data has to walk children and call release on each one — the C interface’s tree of structs, each with its own lifetime.

The release callback is the other half of the contract:

rust
// callback used to drop [ArrowArray] when it is exported
unsafe extern "C" fn c_release_array(array: *mut ArrowArray) {
    if array.is_null() {
        return;
    }
    let array = &mut *array;

    // take ownership of `private_data`, therefore dropping it
    let private = Box::from_raw(array.private_data as *mut PrivateData);
    for child in private.children_ptr.iter() {
        let _ = Box::from_raw(*child);
    }

    if let Some(ptr) = private.dictionary_ptr {
        let _ = Box::from_raw(ptr);
    }

    array.release = None;
}

crates/polars-arrow/src/ffi/array.rs:76

Dropping PrivateData drops the Box<dyn Array>, which drops its buffers, which decrements their refcounts. The consumer of the C array never has to know any of this; it just calls release. Setting array.release = None at the end is what makes the release idempotent from Rust’s side, because ArrowArray also has a Drop impl that calls release:

rust
impl Drop for ArrowArray {
    fn drop(&mut self) {
        match self.release {
            None => (),
            Some(release) => unsafe { release(self) },
        };
    }
}

crates/polars-arrow/src/ffi/array.rs:63

ArrowSchema has the mirror-image callback, c_release_schema, dropping a SchemaPrivateData that holds the type’s CStrings and its children.

The one thing that is copied: alignment

Before exporting, Polars has to make sure the array is in a shape the C interface can express. The awkward case is a validity bitmap whose bit offset does not match the data offset. The C interface requires the validity bitmap to be aligned to the data’s offset, so Polars rebuilds the bitmap:

rust
    fn to_ffi_aligned(&self) -> Self {
        let offset = self.values.offset();

        let validity = self.validity.as_ref().map(|bitmap| {
            if bitmap.offset() == offset {
                bitmap.clone()
            } else {
                align(bitmap, offset)
            }
        });

        Self {
            dtype: self.dtype.clone(),
            validity,
            values: self.values.clone(),
        }
    }

crates/polars-arrow/src/array/primitive/ffi.rs:33

align copies the bits into a freshly allocated bitmap; values.clone() is a refcount bump, so the data is still shared. The dispatcher that decides whether an array needs this treatment is in bridge.rs:

rust
macro_rules! ffi_dyn {
    ($array:expr, $ty:ty) => {{
        let a = $array.as_any().downcast_ref::<$ty>().unwrap();
        if a.offset().is_some() {
            $array
        } else {
            Box::new(a.to_ffi_aligned())
        }
    }};
}

pub fn align_to_c_data_interface(array: Box<dyn Array>) -> Box<dyn Array> {

crates/polars-arrow/src/ffi/bridge.rs:15

offset() returns Some only when the validity bitmap and the values agree on their offset; when they do not, the array is replaced by an aligned copy.

Importing: borrowing, and keeping the owner alive

Import is the direction where the memory belongs to somebody else, and it is where “zero-copy” is most easily over-claimed. Polars does not copy the bytes, but it does have to keep the C array alive for as long as it uses them.

rust
pub unsafe fn import_array_from_c(
    array: ArrowArray,
    dtype: ArrowDataType,
) -> PolarsResult<Box<dyn Array>> {
    try_from(InternalArrowArray::new(array, dtype))
}

crates/polars-arrow/src/ffi/mod.rs:47

Note that import_array_from_c consumes the ArrowArray by value. It is moved into an Arc:

rust
#[derive(Debug, Clone)]
pub struct InternalArrowArray {
    // Arc is used for shareability since this is immutable
    array: Arc<ArrowArray>,
    // Arced to reduce cost of cloning
    dtype: Arc<ArrowDataType>,
}

impl InternalArrowArray {
    pub fn new(array: ArrowArray, dtype: ArrowDataType) -> Self {
        Self {
            array: Arc::new(array),
            dtype: Arc::new(dtype),
        }
    }
}

crates/polars-arrow/src/ffi/array.rs:592

Every buffer created from that array carries a clone of the InternalArrowArray as its owner:

rust
unsafe fn create_buffer_known_len<T: NativeType>(
    array: &ArrowArray,
    dtype: &ArrowDataType,
    owner: InternalArrowArray,
    len: usize,
    index: usize,
) -> PolarsResult<Buffer<T>> {
    if len == 0 {
        // Zero-length arrays might have invalid pointers for zero-length slices in Rust,
        // so this is more than just an optimization.
        return Ok(Buffer::new());
    }
    let ptr: *mut T = get_buffer_ptr(array, dtype, index)?;
    let slice = core::slice::from_raw_parts(ptr, len);
    let storage = SharedStorage::from_slice_with_owner(slice, owner);
    Ok(Buffer::from_storage(storage))
}

crates/polars-arrow/src/ffi/array.rs:269

SharedStorage::from_slice_with_owner is the ForeignOwner path in polars-buffer: the bytes are not Polars’, but Polars holds the owner so they stay valid. When the last Polars-side handle dies, the Arc<ArrowArray> refcount hits zero, ArrowArray::drop runs, and the producer’s release callback fires — freeing the foreign memory exactly once, in exactly the right place.

The buffer path has an explicit escape hatch for the one case where borrowing is not legal:

rust
    // We have to check alignment, for zero-copy to be valid.
    if ptr.is_aligned() {
        let slice = core::slice::from_raw_parts(ptr.add(offset), len);
        let storage = SharedStorage::from_slice_with_owner(slice, owner);
        Ok(Buffer::from_storage(storage))
    } else {
        // Byte-wise copy for misaligned buffers.
        let mut v = Vec::with_capacity(len);
        core::ptr::copy_nonoverlapping(
            ptr.add(offset).cast::<u8>(),
            v.spare_capacity_mut().as_mut_ptr().cast::<u8>(),
            len * size_of::<T>(),
        );
        v.set_len(len);
        Ok(Buffer::from(v))
    }

crates/polars-arrow/src/ffi/array.rs:269

A misaligned pointer cannot legally become a &[T], so Polars copies those bytes into a fresh Vec and owns that instead. This is the honest answer to “is it zero-copy”: yes, except for misaligned buffers, where it has to be or the program is undefined behaviour.

The stream interface

For whole dataframes, Polars uses the pull-based ArrowArrayStream. Exporting one is a small struct with four callbacks:

rust
pub fn export_iterator(
    iter: Box<dyn Iterator<Item = PolarsResult<Box<dyn Array>>>>,
    field: Field,
) -> ArrowArrayStream {
    let private_data = Box::new(PrivateData {
        iter,
        field,
        error: None,
    });

    ArrowArrayStream {
        get_schema: Some(get_schema),
        get_next: Some(get_next),
        get_last_error: Some(get_last_error),
        release: Some(release),
        private_data: Box::into_raw(private_data) as *mut ::std::os::raw::c_void,
    }
}

crates/polars-arrow/src/ffi/stream.rs:217

get_next hands out one exported array per call:

rust
unsafe extern "C" fn get_next(iter: *mut ArrowArrayStream, array: *mut ArrowArray) -> i32 {
    if iter.is_null() {
        return 2001;
    }
    let private = &mut *((*iter).private_data as *mut PrivateData);

    match private.iter.next() {
        Some(Ok(item)) => {
            // check that the array has the same dtype as field
            let item_dt = item.dtype();
            let expected_dt = private.field.dtype();
            if item_dt != expected_dt {
                private.error = Some(CString::new(format!("The iterator produced an item of data type {item_dt:?} but the producer expects data type {expected_dt:?}").as_bytes().to_vec()).unwrap());
                return 2001; // custom application specific error (since this is never a result of this interface)
            }

            std::ptr::write(array, export_array_to_c(item));

            private.error = None;
            0
        },

crates/polars-arrow/src/ffi/stream.rs:147

The reader side is a Rust iterator over Box<dyn Array>, and the last paragraph of the C stream spec is handled explicitly:

rust
    pub unsafe fn next(&mut self) -> Option<PolarsResult<Box<dyn Array>>> {
        let mut array = ArrowArray::empty();
        let status = unsafe { (self.iter.get_next.unwrap())(&mut *self.iter, &mut array) };

        if status != 0 {
            return Some(Err(unsafe { handle_error(&mut self.iter) }));
        }

        // last paragraph of https://arrow.apache.org/docs/format/CStreamInterface.html#c.ArrowArrayStream.get_next
        array.release?;

        // SAFETY: assumed from the C stream interface
        unsafe { import_array_from_c(array, self.field.dtype.clone()) }
            .map(Some)
            .transpose()
    }

crates/polars-arrow/src/ffi/stream.rs:113

array.release? is the end-of-stream signal: a producer that sets release to None says “no more batches”.

The Python side

In Python, this surfaces as the PyCapsule interface. DataFrame.__arrow_c_stream__ is a single call into Rust:

rust
    #[allow(unused_variables)]
    #[pyo3(signature = (requested_schema))]
    fn __arrow_c_stream__<'py>(
        &self,
        py: Python<'py>,
        requested_schema: Option<Py<PyAny>>,
    ) -> PyResult<Bound<'py, PyCapsule>> {
        py.enter_polars_ok(|| {
            self.df.write().rechunk_mut_par();
        })?;
        dataframe_to_stream(&self.df.read(), py)
    }

crates/polars-python/src/dataframe/export.rs:197

and dataframe_to_stream turns the frame into a capsule:

rust
pub(crate) fn dataframe_to_stream<'py>(
    df: &DataFrame,
    py: Python<'py>,
) -> PyResult<Bound<'py, PyCapsule>> {
    let iter = Box::new(DataFrameStreamIterator::new(df));
    let field = iter.field();
    let stream = ffi::export_iterator(iter, field);
    PyCapsule::new_with_value(py, stream, c"arrow_array_stream")
}

crates/polars-python/src/interop/arrow/to_py.rs:78

The capsule’s name, arrow_array_stream, is the handshake: a consumer checks the name to make sure it knows how to release the pointer inside.

Going the other way, PySeries.from_arrow_c_array and from_arrow_c_stream take the capsule apart. The array variant moves the ArrowArray out of the capsule with a std::ptr::replace that leaves an empty array behind:

rust
pub(crate) fn import_array_pycapsules(
    schema_capsule: &Bound<PyCapsule>,
    array_capsule: &Bound<PyCapsule>,
) -> PyResult<(polars_arrow::datatypes::Field, Box<dyn Array>)> {
    let field = import_schema_pycapsule(schema_capsule)?;

    // # Safety
    // array_capsule holds a valid C ArrowArray pointer, as defined by the Arrow PyCapsule
    // Interface
    unsafe {
        let array_ptr = std::ptr::replace(
            array_capsule
                .pointer_checked(Some(c"arrow_array"))?
                .as_ptr() as _,
            ArrowArray::empty(),
        );
        let array = ffi::import_array_from_c(array_ptr, field.dtype().clone()).unwrap();

        Ok((field, array))
    }
}

crates/polars-python/src/series/import.rs:53

The std::ptr::replace is the ownership transfer: the capsule’s pointer slot is overwritten with an empty struct, so whoever drops the capsule afterwards cannot double-free, and Polars now owns the ArrowArray and will call its release when the last handle dies.

On the Python side, pycapsule_to_frame handles both spellings:

python
def pycapsule_to_frame(
    obj: Any,
    *,
    schema: SchemaDefinition | None = None,
    schema_overrides: SchemaDict | None = None,
    rechunk: bool = False,
) -> DataFrame:
    """Convert PyCapsule object to DataFrame."""
    if hasattr(obj, "__arrow_c_array__"):
        # This uses the fact that PySeries.from_arrow_c_array will create a
        # struct-typed Series. Then we unpack that to a DataFrame.
        # `struct_unnest` is a single Rust call that releases the GIL; going via
        # `DataFrame.unnest` would run the whole lazy engine while holding it,
        # which serializes construction across Python threads.
        df = wrap_df(PySeries.from_arrow_c_array(obj).struct_unnest())

    elif hasattr(obj, "__arrow_c_stream__"):

py-polars/src/polars/_utils/pycapsule.py:39

The pyarrow bridge: `to_arrow`

DataFrame.to_arrow() does not go through the capsule interface; it goes through pyarrow’s own C data interface import, which is the same trick:

rust
pub(crate) fn to_py_array(
    array: ArrayRef,
    field: &ArrowField,
    pyarrow: &Bound<PyModule>,
) -> PyResult<Py<PyAny>> {
    let schema = Box::new(ffi::export_field_to_c(field));
    let array = Box::new(ffi::export_array_to_c(array));

    let schema_ptr: *const ffi::ArrowSchema = &*schema;
    let array_ptr: *const ffi::ArrowArray = &*array;

    let array = pyarrow.getattr("Array")?.call_method1(
        "_import_from_c",
        (array_ptr as Py_uintptr_t, schema_ptr as Py_uintptr_t),
    )?;

    Ok(array.unbind())
}

crates/polars-python/src/interop/arrow/to_py.rs:28

Polars exports an ArrowArray and an ArrowSchema, hands the two raw addresses to pyarrow’s private _import_from_c, and pyarrow takes ownership of them. The buffers are shared; pyarrow’s Array points at Polars’ memory. The same routine builds the RecordBatch for DataFrame.to_arrow(), one column at a time.

This is also why to_arrow is not free of type conversions even though it is free of byte copies: the Arrow type Polars holds has to be expressed in the compat level pyarrow expects, and some Polars types (notably dictionaries with unsigned indices) have no pyarrow equivalent and are cast on the way out.

The older protocol: `polars-ffi`

There is also a second, Polars-specific ABI in the polars-ffi crate, kept for plugins that want to exchange whole Series rather than raw Arrow arrays. It is a SeriesExport struct with its own release callback:

rust
/// An FFI exported `Series`.
#[repr(C)]
pub struct SeriesExport {
    field: *mut ArrowSchema,
    // A double ptr, so we can easily release the buffer
    // without dropping the arrays.
    arrays: *mut *mut ArrowArray,
    len: usize,
    release: Option<unsafe extern "C" fn(arg1: *mut SeriesExport)>,
    private_data: *mut std::os::raw::c_void,
}

crates/polars-ffi/src/version_0.rs:14

The version in the module path (version_0) is a promise that the layout will not change under you; polars-ffi exports MAJOR/MINOR constants for plugins to check.

The bottom line

Export copies nothing. The ArrowArray’s buffers are the array’s own pointers; the Rust array is parked in private_data and dropped by the release callback.
Import copies nothing, except misaligned buffers. Those are copied because a misaligned pointer cannot be a &[T]. Ownership is inverted: the imported Buffer keeps the foreign ArrowArray alive as its owner, and the producer’s release fires when the last Polars handle drops.
Two copies are deliberate. A validity bitmap whose offset disagrees with the data offset is rebuilt (bitmap only, not data), and a misaligned buffer is copied byte-wise.
The stream interface is the same contract, pulled. get_next hands out one exported array at a time; a None release means end of stream.
Ownership transfer is a std::ptr::replace. Whoever takes the pointer out of a capsule leaves an empty struct behind, so exactly one party ever calls release.

The practical takeaway: if you are handing data between Polars and another Arrow implementation, use the PyCapsule interface (__arrow_c_stream__ / __arrow_c_array__) and you are sharing memory, not copying it. The one case where you will silently pay for a copy is a column whose buffers are misaligned, which in practice means data that came from somewhere unusual rather than from Polars itself.