Polars Releases the GIL Before It Does Any Real Work

TL;DR: Almost every #[pymethods] entry point in crates/polars-python does not compute anything itself. It calls py.enter_polars_df(|| ...) or py.enter_polars(|| ...), and that helper runs the closure through pyo3’s Python::detach, which drops the GIL for the duration of the call. The GIL comes back in exactly one place: Python::attach, used to call back into Python user functions (UDFs, map_batches, group_by apply). Releasing the GIL also breaks Python’s own Ctrl-C handling, so Polars installs its own SIGINT handler and converts the resulting panic back into a KeyboardInterrupt.

The GIL is the reason import polars as pl still feels like a native extension in 2026, and the reason it also does not. A Rust extension module that holds the GIL while it crunches numbers is functionally single-threaded: every other Python thread in the process sits in gil_drop_request waiting for the extension to hand the lock back. That is why pandas.apply on a large frame blocks the whole interpreter, and it is the first thing any dataframe engine written in Rust has to fix.

The shape of the fix, in one picture:

text

  thread A (Python)         Polars (Rust)          thread B (Python)
  ----------------         -------------          ----------------
  df.collect()
  [GIL held]
  -- detach ------------>   compute ...            <-- free to run
                           (no GIL)
  <-- attach -------------   (GIL re-acquired)
  [GIL held]
  return df

Polars’ fix is not subtle, but it is systematic, and it is worth reading the code because the boundary it draws tells you a lot about how Polars thinks about Python.

One helper, used everywhere

The entry point is a trait called EnterPolarsExt, implemented for Python<'_>:

rust
pub trait EnterPolarsExt {
    /// Whenever you have a block of code in the public Python API that
    /// (potentially) takes a long time, wrap it in enter_polars. This will
    /// ensure we release the GIL and catch KeyboardInterrupts.
    ///
    /// This not only can increase performance and usability, it can avoid
    /// deadlocks on the GIL for Python UDFs.
    fn enter_polars<T, E, F>(self, f: F) -> PyResult<T>
    where
        F: Ungil + Send + FnOnce() -> Result<T, E>,
        T: Ungil + Send,
        E: Ungil + Send + Into<PyPolarsErr>;

crates/polars-python/src/utils.rs:87

The implementation is where the GIL actually goes away:

rust
impl EnterPolarsExt for Python<'_> {
    fn enter_polars<T, E, F>(self, f: F) -> PyResult<T>
    where
        F: Ungil + Send + FnOnce() -> Result<T, E>,
        T: Ungil + Send,
        E: Ungil + Send + Into<PyPolarsErr>,
    {
        let _timeout_guard = is_timeout_enabled().then(|| {
            std::hint::cold_path();
            schedule_polars_timeout(get_traceback(self).ok())
        });
        let ret = self.detach(|| catch_polars_abort(AssertUnwindSafe(f)));
        match ret {
            Ok(Ok(ret)) => Ok(ret),
            Ok(Err(err)) => Err(PyErr::from(err.into())),
            Err(QueryAborted::KeyboardInterrupt) => Err(PyKeyboardInterrupt::new_err("")),
            Err(QueryAborted::OocOutOfDisk) => Err(ComputeError::new_err(
                "query aborted: spilled data exceeded the out-of-core disk budget, increase POLARS_OOC_DISK_BUDGET_MB",
            )),
        }
    }
}

crates/polars-python/src/utils.rs:153

Python::detach is pyo3 0.29’s name for what used to be Python::allow_threads. Under the hood it creates a SuspendAttach guard, which tells the interpreter that this thread no longer needs the GIL, and then runs the closure. When the closure returns, the guard reacquires the GIL — including on the panic path, because it is a guard.

The Ungil bound is pyo3’s safety mechanism for this. Ungil is implemented for every Send type, and the bound on the closure and its return value means the compiler will not let a Bound<'py, PyAny> (a borrowed Python object) escape into the closure, because a borrowed Python object is not Send. That is not a formality: the closure runs while other Python threads are running, so it must not be holding a pointer into the interpreter’s object graph.

enter_polars has three convenience wrappers that differ only in how they wrap the result:

rust
    /// Same as enter_polars, but expects a PolarsResult<DataFrame> as return
    /// which is converted to a PyDataFrame.
    #[inline(always)]
    fn enter_polars_df<F>(self, f: F) -> PyResult<PyDataFrame>
    where
        Self: Sized,
        F: Ungil + Send + FnOnce() -> PolarsResult<DataFrame>,
    {
        self.enter_polars(f).map(PyDataFrame::new)
    }

crates/polars-python/src/utils.rs:108

enter_polars_ok for infallible closures, enter_polars_series for a Series, enter_polars_df for a DataFrame. There are over 200 call sites of these helpers in crates/polars-python/src, which is the point: the pattern is not applied to the expensive operations, it is applied to all of them, so a user never has to guess whether a given DataFrame method holds the GIL.

A `#[pymethods]` function that detaches

Take two of the simplest ones:

rust
    pub fn select(&self, py: Python<'_>, columns: Vec<PyBackedStr>) -> PyResult<Self> {
        py.enter_polars_df(|| self.df.read().select(columns.iter().map(|x| &**x)))
    }

    pub fn gather(&self, py: Python<'_>, indices: Wrap<Vec<IdxSize>>) -> PyResult<Self> {
        let indices = indices.0;
        let indices = IdxCa::from_vec("".into(), indices);
        py.enter_polars_df(|| self.df.read().take(&indices))
    }

crates/polars-python/src/dataframe/general.rs:281 and :281

Note the ordering: the Vec<IdxSize> is converted into an Arrow index array before enter_polars, while the GIL is still held, because that conversion allocates and needs Vec. Only the part that actually crunches rows — the take — runs with the GIL released. That split is deliberate and consistent: argument marshalling happens under the GIL, computation happens without it.

The same shape shows up for the heavy path, LazyFrame.collect:

rust
    #[pyo3(signature = (engine, lambda_post_opt))]
    fn collect(
        &self,
        py: Python<'_>,
        engine: Wrap<Engine>,
        lambda_post_opt: Option<Py<PyAny>>,
    ) -> PyResult<PyDataFrame> {
        py.enter_polars_df(|| {
            let ldf = self.ldf.read().clone();
            if let Some(lambda) = lambda_post_opt {
                ldf._collect_post_opt(|root, lp_arena, expr_arena| {
                    post_opt_callback(&lambda, root, lp_arena, expr_arena)
                })
            } else {
                ldf.collect_with_engine(engine.0).map(|r| match r {
                    QueryResult::Single(df) => df,
                    // TODO: Should return query results
                    QueryResult::Multiple(_) => DataFrame::empty(),
                })
            }
        })
    }

crates/polars-python/src/lazyframe/general.rs:719

The whole query — planning, optimization, physical execution, which in turn spreads work over a thread pool — happens with the GIL released. That is the payoff: while a collect() runs, every other Python thread in the process keeps running normally.

Where the GIL has to come back

There is exactly one category of code that cannot run without the GIL: calling into Python. Polars handles this by re-acquiring the GIL explicitly with Python::attach, at the point of the callback.

The clearest example is group_by_map_groups:

rust
    pub fn group_by_map_groups(
        &self,
        py: Python<'_>,
        by: Vec<PyBackedStr>,
        lambda: Py<PyAny>,
        maintain_order: bool,
    ) -> PyResult<Self> {
        py.enter_polars_df(|| {
            let df = self.df.read().clone(); // Clone so we can't deadlock on re-entrance from lambda.
            let gb = if maintain_order {
                df.group_by_stable(by.iter().map(|x| &**x))
            } else {
                df.group_by(by.iter().map(|x| &**x))
            }?;

            let function = move |df: DataFrame| {
                Python::attach(|py| {
                    let pypolars = polars(py).bind(py);
                    let pydf = PyDataFrame::new(df);
                    let python_df_wrapper =
                        pypolars.getattr("wrap_df").unwrap().call1((pydf,)).unwrap();

crates/polars-python/src/dataframe/general.rs:388

Read the nesting carefully. The outer enter_polars_df releases the GIL for the duration of the group-by. The inner Python::attach takes it back for exactly as long as the user’s Python lambda runs. That is the boundary, and it is drawn in one place: the closure that Polars hands to Python.

The same pattern appears in the UDF plumbing, which is the more interesting case because it is not per-call but per-plan. At Python startup, Polars registers two function pointers:

rust
// Will be overwritten on Python Polars start up.
type PythonColumnUdf =
    fn(s: &[Column], output_dtype: Option<DataType>, lambda: &Py<PyAny>) -> PolarsResult<Column>;
pub static CALL_PYTHON_COLUMNS_UDF: OnceLock<PythonColumnUdf> = OnceLock::new();

type PythonDfUdf = fn(s: DataFrame, lambda: &Py<PyAny>) -> PolarsResult<DataFrame>;
pub static CALL_PYTHON_DF_UDF: OnceLock<PythonDfUdf> = OnceLock::new();

crates/polars-plan/src/dsl/python_dsl/python_udf.rs:17

Those are plain fn pointers, so they can be stored in a OnceLock inside polars-plan, a crate that does not depend on Python at run time. The implementations live in the Python-facing crate:

rust
fn python_function_caller_series(
    s: &[Column],
    output_dtype: Option<DataType>,
    lambda: &Py<PyAny>,
) -> PolarsResult<Column> {
    Python::attach(|py| call_lambda_with_series(py, s, output_dtype, lambda))
}

fn python_function_caller_df(df: DataFrame, lambda: &Py<PyAny>) -> PolarsResult<DataFrame> {
    Python::attach(|py| {
        let pypolars = polars(py).bind(py);

crates/polars-python/src/on_startup.rs:42 and :42

And the plan-side UDF trait calls straight through them:

rust
impl ColumnsUdf for PythonUdfExpression {
    fn call_udf(&self, s: &mut [Column]) -> PolarsResult<Column> {
        let func = CALL_PYTHON_COLUMNS_UDF.get().unwrap();
        let field = self
            .materialized_field
            .get()
            .expect("should have been materialized at this point");
        let mut out = func(
            s,
            self.materialized_field.get().map(|f| f.dtype.clone()),
            &self.python_function,
        )?;

crates/polars-plan/src/dsl/python_dsl/python_udf.rs:107

So a pl.col("a").map_elements(lambda) in a lazy query is: the engine runs with the GIL released, reaches the UDF node, calls call_udf, which calls the registered function pointer, which does Python::attach, which takes the GIL back, calls your lambda, and releases it again.

The consequence for a Python user is worth stating plainly: while your UDF is running, the GIL is held, and every other Python thread is blocked. A UDF that itself calls back into Polars would deadlock, which is exactly why the group_by_map_groups code clones the DataFrame up front (“Clone so we can’t deadlock on re-entrance from lambda”) and why the Ungil bounds exist at all.

Releasing the GIL breaks Ctrl-C, so Polars reimplements it

Here is the part that surprised me. Releasing the GIL has a side effect that is easy to miss: Python’s KeyboardInterrupt is raised by the interpreter’s own SIGINT handler, and that handler needs the GIL to build the exception and unwind the Python frame. While Polars holds the GIL — which it now deliberately does not — a Ctrl-C cannot be delivered the normal way.

Polars solves this by taking over SIGINT itself:

rust
pub fn register_polars_abort_mechanism() {
    let default_hook = std::panic::take_hook();
    std::panic::set_hook(Box::new(move |p| {
        // Suppress output if there is an active catcher and the panic message
        // contains the abort string.
        let num_catchers =
            ABORT_STATE.load(Ordering::Relaxed) >> ABORT_CATCHERS_UNIT.trailing_zeros();
        let suppress = num_catchers > 0 && decode_polars_abort(p.payload()).is_some();
        if !suppress {
            default_hook(p);
        }
    }));

    // WASM doesn't support signals, so we just skip installing the hook there.
    #[cfg(not(target_family = "wasm"))]
    unsafe {
        // SAFETY: we only do an atomic op in the signal handler, which is allowed.
        signal_hook::low_level::register(signal_hook::consts::signal::SIGINT, move || {

crates/polars-error/src/abort.rs:68

The signal handler does one thing: it sets a bit in a global atomic. That bit is checked from inside the computation by try_raise_polars_abort, which is described as “very cheap” and is called from deep inside the kernels:

rust
/// Checks if the abort flag is set, and if yes panics. This function is very cheap.
#[inline(always)]
pub fn try_raise_polars_abort() {
    if ABORT_STATE.load(Ordering::Acquire) & (ABORT_CATCHERS_UNIT - 1) != 0 {
        try_raise_polars_abort_slow()
    }
}

crates/polars-error/src/abort.rs:103

The panic unwinds out of the Rust computation, is caught by catch_polars_abort (which enter_polars wraps around the whole closure), and is turned back into a Python KeyboardInterrupt by the match in utils.rs above. The user sees a normal Python traceback; internally it was a Rust panic carrying the string __POLARS_ABORT_KEYBOARD_INTERRUPT.

The same mechanism doubles as the out-of-core disk budget tripwire (QueryAborted::OocOutOfDisk), because a spilled query can run out of disk from anywhere inside the engine and there is no PolarsResult return path for it.

Crossing the boundary with data

Releasing the GIL is only half of it; the data you were holding has to survive the trip. Polars wraps the Arrow C stream reader in a Send newtype explicitly to make that legal:

rust
/// Moves an FFI stream reader across a GIL-release boundary.
///
/// # Safety
/// The Arrow C stream interface is a plain C ABI: producers must be callable without
/// the GIL held, and Python-backed producers (pyarrow and friends) re-acquire it
/// internally. The reader is only ever touched by the single thread that owns it.
struct SendStreamReader(ArrowArrayStreamReader<Box<ArrowArrayStream>>);
unsafe impl Send for SendStreamReader {}

crates/polars-python/src/series/import.rs:115

And the same file shows the pattern applied to the actual drain:

rust
    let s = py.enter_polars(move || {
        let mut stream = stream;

        let mut produced_arrays: Vec<Box<dyn Array>> = vec![];
        while let Some(array) = unsafe { stream.0.next() } {
            produced_arrays.push(array?);
        }

crates/polars-python/src/series/import.rs:130

On the Python side, the same idea shows up in a comment that explains why pycapsule_to_frame calls struct_unnest instead of DataFrame.unnest:

python
    if hasattr(obj, "__arrow_c_array__"):
        # This uses the fact that PySeries.from_arrow_c_array will create a
        # struct-typed Series. Then we unpack that to a DataFrame.
        # `struct_unnest` is a single Rust call that releases the GIL; going via
        # `DataFrame.unnest` would run the whole lazy engine while holding it,
        # which serializes construction across Python threads.
        df = wrap_df(PySeries.from_arrow_c_array(obj).struct_unnest())

py-polars/src/polars/_utils/pycapsule.py:39

Callbacks that arrive from another thread

One more shape is worth showing, because it is the one that is easy to get wrong: a callback that arrives from a worker thread rather than being called from one. collect_with_callback runs the query on a blocking thread and then calls a Python callback on some other thread entirely:

rust
    #[cfg(feature = "async")]
    #[pyo3(signature = (engine, lambda))]
    fn collect_with_callback(
        &self,
        py: Python<'_>,
        engine: Wrap<Engine>,
        lambda: Py<PyAny>,
    ) -> PyResult<()> {
        py.enter_polars_ok(|| {
            let ldf = self.ldf.read().clone();

            // We use a tokio spawn_blocking here as it has a high blocking
            // thread pool limit.
            polars_core::runtime::ASYNC.spawn_blocking(move || {
                let result = ldf
                    .collect_with_engine(engine.0)
                    .map(|r| match r {
                        QueryResult::Single(df) => df,
                        // TODO: Should return query results
                        QueryResult::Multiple(_) => DataFrame::empty(),
                    })
                    .map(PyDataFrame::new)
                    .map_err(PyPolarsErr::from);

                Python::attach(|py| match result {
                    Ok(df) => {
                        lambda.call1(py, (df,)).map_err(|err| err.restore(py)).ok();
                    },

crates/polars-python/src/lazyframe/general.rs:757

Three things happen here. The outer enter_polars_ok releases the GIL so the Python thread that called collect_with_callback is free immediately. The query is handed to ASYNC.spawn_blocking, a Tokio blocking thread, so it does not occupy the calling thread either. And the callback itself runs under Python::attach, which re-acquires the GIL on the worker thread — this is what pyo3’s Python::attach is for, as opposed to Python::with_gil, which assumes the current thread already holds it.

The lambda is captured as Py<PyAny>, not Bound<'py, PyAny>. Py<T> is an owned, atomic reference-counted handle that is Send; Bound<'py, T> is a borrowed reference tied to a GIL epoch and is not. That is the whole trick that lets a Python object be carried into a closure that runs without the GIL: Py is Ungil, Bound is not.

The bottom line

The GIL is released by default, everywhere. enter_polars is the single door, and it calls Python::detach. Argument marshalling happens under the GIL; computation happens without it.
The GIL is re-acquired at exactly one point: the callback into Python user code, via Python::attach. A UDF therefore runs with the GIL held and blocks every other Python thread for its duration. That is the trade Polars makes, and it is unavoidable — you cannot call Python without the GIL.
The Ungil bound is the type-level guard. It makes it impossible to smuggle a borrowed Python object across the released-GIL window.
Ctrl-C is reimplemented. Because Python cannot deliver KeyboardInterrupt while the GIL is released, Polars installs its own SIGINT handler, sets an atomic flag, and turns the resulting panic back into a KeyboardInterrupt at the boundary.

For a Python user, the practical consequences are: a Polars query no longer freezes your interpreter, a Python UDF does, and if you want real parallelism across Python threads, keep the Python side of the work out of the UDF and into the expression language.