Polars Parses JSON Without Building a DOM

TL;DR: A JSON DOM is a tree of owned values, and building one costs an allocation per node plus a pass to walk it afterwards. Polars never builds one: it parses a line into simd_json::BorrowedValues that borrow the source bytes, then walks the object’s keys once and pushes each value straight into the builder of the column that key belongs to. The schema is what makes this possible — the set of columns is known before the first row is parsed, so there is a destination for every value.

The name “deserializer” is misleading in most JSON libraries. serde_json::Value is a DOM: a tree of String, Vec and Map nodes, each one heap-allocated, each one owned. Parsing a file into it means allocating a node per value, and then walking the tree a second time to copy the leaves into columns. Both passes are paid for every byte of the file, and the tree is pure overhead: the data is only wanted in columnar form.

Polars’ JSON reader sidesteps this in three steps: parse into borrowed values, use the schema to route each value to a column, and keep the number of copies to one per string.

Where the code lives

The reader is split across two crates, which is worth knowing before you go looking for it:

• crates/polars-json holds the value-level machinery: json::deserialize (values to Arrow arrays), json::infer (type inference), json::ordered (the order-preserving tape conversion for Map columns), and ndjson::deserialize (a row iterator to an Arrow array).
• crates/polars-io holds the file-level machinery: ndjson/core.rs (the chunked, parallel reader), ndjson/buffer.rs (the per-column builders), and json/mod.rs (the JsonReader front end).

Polars used to route JSON through arrow2’s json_deserializer module — a hand-written parser with its own Buffer/Bytes value tree. That was replaced by simd-json in 2023 (“use simd-json for all json parsing”, #8922), and today the value parsing is simd-json’s. What plays the role of the old visitor now is polars-io/src/ndjson/buffer.rs: a set of per-column builders that the parsed values are pushed into, one column at a time.

Splitting the file into lines

The first step is embarrassingly simple, and the comment explains why it is not the obvious thing (crates/polars-io/src/ndjson/core.rs:223):

pub fn json_lines(bytes: &[u8]) -> impl Iterator<Item = &[u8]> {
    // This previously used `serde_json`'s `RawValue` to deserialize chunks without really deserializing them.
    // However, this convenience comes at a cost. serde_json allocates and parses and does UTF-8 validation, all
    // things we don't need since we use simd_json for them. Also, `serde_json::StreamDeserializer` has a more
    // ambitious goal: it wants to parse potentially *non-delimited* sequences of JSON values, while we know
    // our values are line-delimited. Turns out, custom splitting is very easy, and gives a very nice performance boost.
    bytes
        .split(|&byte| byte == b'\n')
        .filter(|bytes| is_json_line(bytes))
}

serde_json::RawValue was the old trick for “give me the raw text of the next value without parsing it” — it did the job, but it allocated a Box<str> per value and re-validated UTF-8. Since NDJSON is line-delimited by definition, splitting on \n and skipping blank lines is enough, and it costs nothing.

The borrowed value

Each line is then parsed into a value that borrows the buffer (crates/polars-io/src/ndjson/core.rs:194):

#[inline(always)]
fn parse_impl(
    bytes: &[u8],
    buffers: &mut PlIndexMap<BufferKey, Buffer>,
    scratch: &mut Scratch,
    guide: Option<&TapeGuide>,
    ignore_errors: bool,
) -> PolarsResult<usize> {
    scratch.json.clear();
    scratch.json.extend_from_slice(bytes);
    let n = scratch.json.len();
    let parse_err = |e| polars_err!(ComputeError: "error parsing line: {}", e);
    let value = match guide {
        Some(guide) => {
            let tape = simd_json::to_tape_with_buffers(&mut scratch.json, &mut scratch.buffers)
                .map_err(parse_err)?;
            polars_json::json::ordered::tape_to_value(&tape, guide, ignore_errors)?
        },
        None => simd_json::to_borrowed_value_with_buffers(&mut scratch.json, &mut scratch.buffers)
            .map_err(parse_err)?,
    };
    match value {
        simd_json::BorrowedValue::Object(value) => {
            buffers.iter_mut().try_for_each(|(s, inner)| {
                match s.0.map_lookup(&value) {
                    Some(v) => inner.add(v)?,
                    None => inner.add_null(),
                }
                PolarsResult::Ok(())
            })?;
        },
        _ if ignore_errors => {
            buffers.iter_mut().for_each(|(_, inner)| inner.add_null());
        },
        v => {
            polars_bail!(ComputeError: "NDJSON line expected to contain JSON object: {v}");
        },
    };
    Ok(n)
}

Three things to notice.

First, to_borrowed_value_with_buffers parses into a value whose strings point into scratch.json, the buffer that holds the line. The scratch buffer is reused for every line, so nothing is allocated per line for the parse itself.

Second, the routing is a single loop over the columns, not over the keys of the object. For each column the reader looks up the key in the parsed object; if the key is missing, the column gets a null. This is the opposite of the DOM approach, where you would walk the object’s keys and look up the column. Doing it this way means the number of lookups is the number of columns, regardless of how many keys the line has, and a line with extra keys costs nothing.

Third, the map lookup is s.0.map_lookup(&value), where s.0 is a simd_json::KnownKey. The key is hashed once, when the schema is turned into buffers, and the hash is reused for every line (crates/polars-io/src/ndjson/buffer.rs:17):

#[derive(Debug, Clone, PartialEq)]
pub(crate) struct BufferKey<'a>(pub(crate) KnownKey<'a>);
impl Eq for BufferKey<'_> {}

impl Hash for BufferKey<'_> {
    fn hash<H: Hasher>(&self, state: &mut H) {
        self.0.key().hash(state)
    }
}

The Hash impl is the interesting part: it hashes the key string, not the KnownKey itself, which keeps the hash consistent with KnownKey’s equality. The KnownKey carries the precomputed hash that map_lookup uses at parse time. The buffers are stored in a PlIndexMap keyed by BufferKey, which keeps the column order stable and gives O(1) lookup by name.

The column builder

Each column gets a builder chosen by its dtype (crates/polars-io/src/ndjson/buffer.rs:152):

pub(crate) fn init_buffers(
    schema: &Schema,
    capacity: usize,
    ignore_errors: bool,
) -> PolarsResult<PlIndexMap<BufferKey<'_>, Buffer<'_>>> {
    schema
        .iter()
        .map(|(name, dtype)| {
            let av_buf = (dtype, capacity).into();
            let key = KnownKey::from(name.as_str());
            Ok((
                BufferKey(key),
                Buffer {
                    name,
                    buf: av_buf,
                    ignore_errors,
                },
            ))
        })
        .collect()
}

The builder is an AnyValueBuffer, which is a per-dtype append-only buffer. The add method is a match on that buffer’s variant (crates/polars-io/src/ndjson/buffer.rs:54):

    #[inline]
    pub(crate) fn add(&mut self, value: &Value) -> PolarsResult<()> {
        use AnyValueBuffer::*;
        match &mut self.buf {
            _ if value.is_null() => {
                self.buf.add(AnyValue::Null);
                Ok(())
            },
            Boolean(buf) => {
                match value {
                    Value::Static(StaticNode::Bool(v)) => buf.append_value(*v),
                    Value::Static(StaticNode::Null) => buf.append_null(),
                    _ if self.ignore_errors => buf.append_null(),
                    v => {
                        polars_bail!(ComputeError: "cannot parse '{}' ({}) as Boolean", v, v.value_type())
                    },
                }
                Ok(())
            },
            Int32(buf) => {
                let v =
                    deserialize_numeric::<Int32Type>(value, value.as_i32(), self.ignore_errors)?;
                buf.append_option(v);
                Ok(())
            },

Note value.as_i32(): the borrowed value knows how to narrow an I64 or U64 to an i32 itself, so the numeric conversion is a couple of instructions, not a parse-and-convert. And AnyValue::Null is pushed for a JSON null, so the null mask is maintained by the builder as it goes.

The capacity argument is the row-count hint, and it is where the reader avoids a reallocation storm: it is estimated from the file size and the mean line length before the first line is parsed (crates/polars-io/src/ndjson/core.rs:80):

    fn parse_json(&mut self, mut n_threads: usize, bytes: &[u8]) -> PolarsResult<DataFrame> {
        let mut bytes = bytes;
        let mut total_rows = 128;

        if let Some((mean, std)) = get_line_stats_json(bytes, self.sample_size) {
            let line_length_upper_bound = mean + 1.1 * std;

            total_rows = (bytes.len() as f32 / (mean - 0.01 * std)) as usize;

get_line_stats_json samples the first and the 75% mark of the file, computes the mean and standard deviation of the line lengths, and from those estimates the number of rows. It is a heuristic, and the file is truncated to the estimated row count if n_rows was given — the reader accepts that it might be slightly wrong and corrects the row count afterwards.

Chunking and parallelism

The file is cut into byte ranges at line boundaries, and each range is parsed independently (crates/polars-io/src/ndjson/core.rs:395):

pub(crate) fn get_file_chunks_json(bytes: &[u8], n_threads: usize) -> Vec<(usize, usize)> {
    let mut last_pos = 0;
    let total_len = bytes.len();
    let chunk_size = total_len / n_threads;
    let mut offsets = Vec::with_capacity(n_threads);
    for _ in 0..n_threads {
        let search_pos = last_pos + chunk_size;

        if search_pos >= bytes.len() {
            break;
        }

        let end_pos = match next_line_position_naive_json(&bytes[search_pos..]) {
            Some(pos) => search_pos + pos,
            None => {
                break;
            },
        };
        offsets.push((last_pos, end_pos));
        last_pos = end_pos;
    }
    offsets.push((last_pos, total_len));
    offsets
}

The cut points are found with memchr on the newline character — the same quote-unaware search as in the CSV reader, and for the same reason: a chunk boundary that lands in the middle of a quoted string is a bug, but a chunk boundary that is off by a few bytes is repaired by the “is this line a complete line” check that follows. The per-chunk work is then run in parallel with rayon, and each chunk builds its own set of column buffers (crates/polars-io/src/ndjson/core.rs:115):

        let row_index = self.row_index.as_ref().map(|ri| ri as &RowIndex);
        let (mut dfs, prepredicate_heights) = RAYON.install(|| {
            file_chunks
                .into_par_iter()
                .map(|(start_pos, stop_at_nbytes)| {
                    let mut local_df = parse_ndjson(
                        &bytes[start_pos..stop_at_nbytes],
                        Some(capacity),
                        &self.schema,
                        self.ignore_errors,
                    )?;

Each chunk is parsed into a complete DataFrame and the frames are stacked afterwards. That is a coarser unit of work than the Arrow-style approach of one batch per chunk, and it is why the NDJSON path materialises a DataFrame per chunk rather than a set of Arrow arrays.

Schema: from JSON objects to columns

The schema is what makes the direct-to-column routing possible. For NDJSON each line is an object, and the object is a struct, and the struct’s fields are the columns (crates/polars-io/src/ndjson/mod.rs:29):

pub fn infer_schema<R: std::io::BufRead>(
    reader: &mut R,
    infer_schema_len: Option<NonZeroUsize>,
) -> PolarsResult<Schema> {
    let dtypes = polars_json::ndjson::iter_unique_dtypes(reader, infer_schema_len)?;
    let dtype =
        crate::json::infer::dtypes_to_supertype(dtypes.map(|dt| DataType::from_arrow_dtype(&dt)))?;

    if !matches!(&dtype, DataType::Struct(_)) {
        polars_bail!(ComputeError: "NDJSON line expected to contain JSON object: {dtype}");
    }

    let schema = StructArray::get_fields(&dtype.to_arrow(CompatLevel::newest()))
        .iter()
        .map(Into::<Field>::into)
        .collect();
    Ok(schema)
}

Inference reads a sample of lines, infers an Arrow type for each, and coerces the set of types into one supertype. The coercion rules are in crates/polars-json/src/json/infer_schema.rs:71 and are worth a look, because they decide what a mixed-type column becomes:

/// Coerce an heterogeneous set of [`ArrowDataType`] into a single one. Rules:
/// * The empty set is coerced to `Null`
/// * `Int64` and `Float64` are `Float64`
/// * Lists and scalars are coerced to a list of a compatible scalar
/// * Structs contain the union of all fields
/// * All other types are coerced to `Utf8`
pub(crate) fn coerce_dtype<A: Borrow<ArrowDataType>>(datatypes: &[A]) -> ArrowDataType {

Int64 and Float64 mix to Float64; two structs merge into a struct with the union of their fields; anything else that does not match becomes a string. Note that this is a value type inference, not a JSON type inference: the schema says “this column is a float”, not “this column holds numbers”.

The other direction: values to an Arrow array

The polars-json crate also offers a path that goes straight to Arrow arrays rather than to per-column builders. deserialize takes a borrowed value and a dtype and produces an array (crates/polars-json/src/json/deserialize.rs:421):

pub(crate) fn _deserialize<'a, A: Borrow<BorrowedValue<'a>>>(
    rows: &[A],
    dtype: ArrowDataType,
    allow_extra_fields_in_struct: bool,
) -> PolarsResult<Box<dyn Array>> {
    match &dtype {
        ArrowDataType::Null => {
            if let Some(err_idx) = (0..rows.len())
                .find(|i| !matches!(rows[*i].borrow(), BorrowedValue::Static(StaticNode::Null)))
            {
                check_err_idx(rows, err_idx, "null")?;
            }

            Ok(Box::new(NullArray::new(dtype, rows.len())))
        },

and for a struct, it fans out over the fields of the dtype, pushing each field’s values into its own list and then deserialising that list (crates/polars-json/src/json/deserialize.rs:240):

    rows.iter().enumerate().for_each(|(i, row)| {
        match row.borrow() {
            BorrowedValue::Object(values) => {
                let mut n_matched = 0usize;
                for (&key, &mut (_, ref mut inner)) in out_values.iter_mut() {
                    if let Some(v) = values.get(key) {
                        n_matched += 1;
                        inner.push(v)
                    } else {
                        inner.push(&JSON_NULL_VALUE)
                    }
                }

The same trick as in the NDJSON path, at a different granularity: the dtype’s fields drive the iteration, and a missing key becomes an explicit null value.

Rows as an array

There is a third path, used when the caller already has rows as strings: the rows are concatenated into a single JSON array and parsed in one go (crates/polars-json/src/ndjson/deserialize.rs:33):

pub fn deserialize_iter<'a>(
    rows: impl Iterator<Item = &'a str>,
    dtype: ArrowDataType,
    guide: Option<&ArrowDataType>,
    buf_size: usize,
    count: usize,
    allow_extra_fields_in_struct: bool,
) -> PolarsResult<ArrayRef> {
    // The rows are parsed as one JSON array.
    let guide_dtype = guide.map(|guide| guide.clone().to_large_list(true));
    let guide = guide_dtype.as_ref().map(TapeGuide::new);
    let mut arr: Vec<Box<dyn Array>> = Vec::new();
    let mut buf = Vec::with_capacity(std::cmp::min(buf_size + count + 2, u32::MAX as usize));
    buf.push(b'[');

and then, per row, buf.extend_from_slice(row.as_bytes()); buf.push(b','). The buffer is flushed and parsed whenever it approaches u32::MAX, because that is the largest array length the parser can represent. The result is a list of structs, which the caller then transposes into columns.

The JSON (array) path

A JSON file that is one big array is a different problem: there are no lines to split, and the whole document has to be parsed at once. The reader parses the document into a borrowed value, infers a struct type from it, and deserialises the whole thing into one Arrow array (crates/polars-io/src/json/mod.rs:288):

                let (tape, json_value) = if map_guided {
                    let tape = simd_json::to_tape(json_bytes).map_err(to_compute_err)?;
                    let json_value = self
                        .schema
                        .is_none()
                        .then(|| polars_json::json::ordered::tape_to_default_value(&tape));
                    (Some(tape), json_value)
                } else {
                    let json_value =
                        simd_json::to_borrowed_value(json_bytes).map_err(to_compute_err)?;
                    (None, Some(json_value))
                };

Note that the whole document is parsed into a value here, so this path does build a tree — but a tree of borrowed values, not a tree of owned ones. The strings still point into the buffer, and the columns are then filled from that tree in one pass. The alternative, streaming a JSON array element by element, would need a different parser; the format does not make it easy, and the document is assumed to fit in memory anyway.

Map order

One wrinkle: simd_json’s Value::Object switches to a hash map above 32 members, which loses the order of the keys and any duplicate keys. Polars needs the order for Map columns, so it parses to a tape and converts that itself (crates/polars-json/src/json/ordered.rs:1):

//! Tape conversion that keeps the source order of `Map` objects.
//!
//! [`Value::Object`] switches to an unordered hash table above 32 members, which loses
//! entry order and duplicate keys. This conversion emits every object at a `Map` position as
//! `[{"key": k, "value": v}, ...]` in tape order instead.

The TapeGuide is built from the target dtype, so it knows which positions in the document are Maps and which are plain structs; everything else falls back to the default conversion.

The bottom line

• No DOM is built. The parser produces BorrowedValues that point into the source buffer, and the reader consumes them before the buffer is reused.
• The schema is the routing table. Columns are visited, not keys, so a line with extra keys costs nothing, and a line with a missing key becomes a null without a special case.
• The number of copies per value is one for strings (into the column’s view buffer) and zero for numbers and booleans, which are converted in place.
• Allocations are per chunk, not per row: the line buffer, the parser’s scratch buffers, the column builders and the row-count estimate are all computed once per chunk.
• The cost of this design is that the schema must be known before the values are read, so a file whose rows disagree about their keys needs an explicit schema or a supertype, and mixed types within a column are coerced by the inference rules rather than preserved.