Polars Parses CSV With Portable SIMD and simdutf8

TL;DR: The CSV reader never builds an intermediate representation of a row. It scans the buffer 64 bytes at a time with std::simd, finds the field and line separators with a bitmask, and computes “am I inside a quoted field?” with a carry-less multiply that XOR-folds the quote mask. Each field is a &[u8] slice into the original buffer, handed to a parse_bytes implementation chosen by the column’s dtype. Strings are the only values that get copied, and they are validated with simdutf8.

CSV looks like the simplest format in the dataframe world, and that is exactly why it is hard. A field may contain the separator, the newline, doubled quotes, or nothing at all. Rows may be short. Lines may be comments. The first line may be a header, or a BOM may precede it, and it may use a different quote character than the rest of the file.

The naive implementation — bytes.split(b'\n') then line.split(b',') — is wrong for any file with quoted fields, which is most files. Polars’ approach is worth studying because it solves the “where does the field end?” question with bitmasks rather than with a state machine over bytes.

The field splitter

SplitFields is the workhorse: an iterator that yields (&[u8], bool) per field, where the bool says whether the field was quoted and therefore needs unescaping. There are two implementations behind one type: a scalar one and a #[cfg(feature = "simd")] one (crates/polars-io/src/csv/read/splitfields.rs:142):

rust
#[cfg(feature = "simd")]
mod inner {
    use std::simd::prelude::*;

    use polars_utils::clmul::prefix_xorsum_inclusive;

    const SIMD_SIZE: usize = 64;
    type SimdVec = u8x64;

The interesting part is the quoted-field branch (crates/polars-io/src/csv/read/splitfields.rs:276):

rust
                loop {
                    let bytes = unsafe { self.v.get_unchecked(total_idx..) };

                    if bytes.len() > SIMD_SIZE {
                        let lane: [u8; SIMD_SIZE] = unsafe {
                            bytes
                                .get_unchecked(0..SIMD_SIZE)
                                .try_into()
                                .unwrap_unchecked()
                        };
                        let simd_bytes = SimdVec::from(lane);
                        let has_eol = simd_bytes.simd_eq(self.simd_eol_char);
                        let has_sep = simd_bytes.simd_eq(self.simd_separator);
                        let quote_mask = simd_bytes.simd_eq(self.simd_quote_char).to_bitmask();
                        let mut end_mask = (has_sep | has_eol).to_bitmask();

                        let mut not_in_quote_field = prefix_xorsum_inclusive(quote_mask);

                        if not_in_field_previous_iter {
                            not_in_quote_field = !not_in_quote_field;
                        }
                        not_in_field_previous_iter =
                            (not_in_quote_field & (1 << (SIMD_SIZE - 1))) > 0;
                        end_mask &= not_in_quote_field;

Read that carefully, because it is the whole trick. has_sep | has_eol is a mask of “bytes that could end a field”. quote_mask is a mask of “bytes that are quotes”. prefix_xorsum_inclusive(quote_mask) turns the quote mask into a mask of “the number of quotes seen so far is odd”, i.e. “we are inside a quoted field” — for every position in the 64-byte block, in one instruction sequence. Masking the candidate ends with that gives the positions that are really field ends.

The prefix XOR is a carry-less multiplication by all-ones (crates/polars-utils/src/clmul.rs:89):

rust
// Computes for each bit i the XOR of bits[0..=i].
#[inline]
pub fn prefix_xorsum_inclusive(x: u64) -> u64 {
    #[cfg(all(target_arch = "x86_64", target_feature = "pclmulqdq"))]
    return intel_clmul64(x, u64::MAX);

    #[cfg(all(
        target_arch = "aarch64",
        target_feature = "neon",
        target_feature = "aes"
    ))]
    return arm_clmul64(x, u64::MAX);

    #[allow(unreachable_code)]
    portable_prefix_xorsum_inclusive(x)
}

On x86 with pclmulqdq that is a single PCLMULQDQ; on aarch64 with the crypto extensions it is a vmull_p64; everywhere else it falls back to six shift-and-xor steps. The not_in_field_previous_iter flag carries the parity across block boundaries, so a quoted field that spans several 64-byte blocks still terminates at the right place.

One detail that makes this fast: the mask of valid ends found in a block is cached in previous_valid_ends, so the next call to next() pops the earliest bit without re-scanning (crates/polars-io/src/csv/read/splitfields.rs:222):

rust
            // Then check cached value as this is hot.
            if self.previous_valid_ends != 0 {
                let pos = self.previous_valid_ends.trailing_zeros() as usize;
                self.previous_valid_ends >>= (pos + 1) as u64;

For an unquoted field the SIMD path is even simpler: there is no parity to compute, so the mask is just “separator or end of line” (crates/polars-io/src/csv/read/splitfields.rs:365):

rust
                // Start of an unenclosed field
                let mut total_idx = 0;

                loop {
                    let bytes = unsafe { self.v.get_unchecked(total_idx..) };

                    if bytes.len() > SIMD_SIZE {
                        let lane: [u8; SIMD_SIZE] = unsafe {
                            bytes
                                .get_unchecked(0..SIMD_SIZE)
                                .try_into()
                                .unwrap_unchecked()
                        };
                        let simd_bytes = SimdVec::from(lane);
                        let has_eol_char = simd_bytes.simd_eq(self.simd_eol_char);
                        let has_separator = simd_bytes.simd_eq(self.simd_separator);
                        let has_any_mask = (has_separator | has_eol_char).to_bitmask();

The line splitter

SplitLines does the same thing for newlines, with the same quote-parity trick (crates/polars-io/src/csv/read/parser.rs:594):

rust
        loop {
            let bytes = unsafe { self.v.get_unchecked(self.total_index..) };
            if bytes.len() > SIMD_SIZE {
                let lane: [u8; SIMD_SIZE] = unsafe {
                    bytes
                        .get_unchecked(0..SIMD_SIZE)
                        .try_into()
                        .unwrap_unchecked()
                };
                let simd_bytes = SimdVec::from(lane);
                let eol_mask = simd_bytes.simd_eq(self.simd_eol_char).to_bitmask();

                let valid_eols = if self.quoting {
                    let quote_mask = simd_bytes.simd_eq(self.simd_quote_char).to_bitmask();
                    let mut not_in_quote_field = prefix_xorsum_inclusive(quote_mask);

                    if not_in_field_previous_iter {
                        not_in_quote_field = !not_in_quote_field;
                    }
                    not_in_field_previous_iter = (not_in_quote_field & (1 << (SIMD_SIZE - 1))) > 0;
                    eol_mask & not_in_quote_field
                } else {
                    eol_mask
                };

When the file has no quote character configured, the whole quote machinery disappears and the mask is just “where are the newlines” — the if self.quoting branch is a compile-time-ish constant per file, and the branch predictor never gets it wrong.

Counting lines without parsing them

count(*) on a CSV file must not parse a single value, but it still has to respect quoting: a newline inside a quoted field is not a row boundary. The trick is to compute the answer for both possible starting states at once — “as if the chunk starts outside a string” and “as if it starts inside one” — and then pick the right one when the chunks are stitched together (crates/polars-io/src/csv/read/parser.rs:768):

rust
        #[cfg(feature = "simd")]
        {
            // 0 if even number of quotes seen so far, u64::MAX otherwise.
            let mut global_quote_parity_mask = 0;
            while scan_offset + 64 <= bytes.len() {
                let block: [u8; 64] = unsafe {
                    bytes
                        .get_unchecked(scan_offset..scan_offset + 64)
                        .try_into()
                        .unwrap_unchecked()
                };
                let simd_bytes = SimdVec::from(block);
                let eol_mask = simd_bytes.simd_eq(self.simd_eol_char).to_bitmask();
                if self.quoting {
                    let quote_mask = simd_bytes.simd_eq(self.simd_quote_char).to_bitmask();
                    let quote_parity =
                        prefix_xorsum_inclusive(quote_mask) ^ global_quote_parity_mask;
                    global_quote_parity_mask = ((quote_parity as i64) >> 63) as u64;

                    let start_outside_string_eol_mask = eol_mask & !quote_parity;
                    states[0].newline_count += start_outside_string_eol_mask.count_ones() as usize;

states[0] and states[1] are the two hypotheses. When the chunked reader joins the results, it carries the “am I inside a string at the end of this chunk” bit into the next chunk (crates/polars-io/src/csv/read/parser.rs:200):

rust
        let mut n = 0;
        let mut in_string = false;
        for (pair, _) in states {
            n += pair[in_string as usize].newline_count;
            in_string = pair[in_string as usize].end_inside_string;
        }
        n += eof_unterminated_row as usize;
        Ok(n)

This is also why the counting path can be parallel: each chunk is analysed independently, and only the parity bit has to cross the boundary.

The value layer

Splitting is only half of it. Once a field is a &[u8], it has to become a value of the column’s dtype. That is the job of the ParsedBuilder trait (crates/polars-io/src/csv/read/builder.rs:123):

rust
trait ParsedBuilder {
    fn parse_bytes(
        &mut self,
        bytes: &[u8],
        ignore_errors: bool,
        _needs_escaping: bool,
        _missing_is_null: bool,
        _time_unit: Option<TimeUnit>,
    ) -> PolarsResult<()>;
}

Numbers go to atoi_simd and fast_float2, which are SIMD parsers for integers and floats respectively (crates/polars-io/src/csv/read/builder.rs:38):

rust
impl PrimitiveParser for Float32Type {
    #[inline]
    fn parse(bytes: &[u8]) -> Option<f32> {
        fast_float2::parse(bytes).ok()
    }
}

Strings are where simdutf8 comes in. The Utf8Field builder keeps a MutableBinaryViewArray and a scratch buffer for unescaping (crates/polars-io/src/csv/read/builder.rs:193):

rust
#[inline]
pub fn validate_utf8(bytes: &[u8]) -> bool {
    simdutf8::basic::from_utf8(bytes).is_ok()
}

and the parse itself (crates/polars-io/src/csv/read/builder.rs:220):

rust
        // note that one branch writes without updating the length, so we must do that later.
        let escaped_bytes = if needs_escaping {
            self.scratch.clear();
            self.scratch.reserve(bytes.len());
            polars_ensure!(bytes.len() > 1 && bytes.last() == Some(&self.quote_char), ComputeError: "invalid csv file\n\nField `{}` is not properly escaped.", std::str::from_utf8(bytes).map_err(to_compute_err)?);

            // SAFETY:
            // we just allocated enough capacity and data_len is correct.
            unsafe {
                let n_written =
                    escape_field(bytes, self.quote_char, self.scratch.spare_capacity_mut());
                self.scratch.set_len(n_written);
            }

            self.scratch.as_slice()
        } else {
            bytes
        };

        if matches!(self.encoding, CsvEncoding::LossyUtf8) | ignore_errors {
            // It is important that this happens after escaping, as invalid escaped string can produce
            // invalid utf8.
            let parse_result = validate_utf8(escaped_bytes);

Note the ordering constraint in that comment: the UTF-8 check runs on the unescaped bytes, because the escaped form is not the form that ends up in the column, and an escaped string can be invalid in ways the unescaped one is not. And in the default Utf8 encoding, validate_utf8 is not called per field at all — the reader validates whole chunks up front (crates/polars-io/src/csv/read/read_impl.rs:449):

rust
                    s.spawn(move |_| {
                        if check_utf8 && !super::builder::validate_utf8(b) {
                            let mut results = results.lock().unwrap();
                            results.push((
                                b.as_ptr() as usize,
                                Err(polars_err!(ComputeError: "invalid utf-8 sequence")),
                            ));
                            return;
                        }

One simdutf8 pass over 16 MB beats one pass per field, because the per-field call has to re-establish the SIMD state each time.

Where memchr fits

memchr is in the CSV crate, but it is not what makes the reader fast. It is used in the handful of places that need to find a newline without caring about quoting: skipping lines, finding the end of the file’s complete rows, and detecting comment lines (crates/polars-io/src/csv/read/parser.rs:261):

rust
/// Find the nearest next line position.
/// Does not check for new line characters embedded in String fields.
pub(super) fn next_line_position_naive(input: &[u8], eol_char: u8) -> Option<usize> {
    let pos = memchr::memchr(eol_char, input)? + 1;
    if input.len() - pos == 0 {
        return None;
    }
    Some(pos)
}

The name says it: naive, i.e. it does not respect quotes. It is used where the caller has already established that quoting is not in play, or where a wrong answer is merely a wrong row count in a count(*) that is already approximate. The hot loops use the portable SIMD path instead, because memchr cannot answer “is this newline inside a quoted field?” in one call.

Copying

The reader is careful about copies. SplitFields yields slices of the original buffer, and the numeric parsers consume those slices in place: no allocation per field, and no copy of the digits. Only strings are copied, once, into the view array’s buffer, and the unescaping scratch buffer is reused across fields of the same column.

The field dispatch itself is where the projection pushdown lands: the parser iterates over all fields but only touches the projected ones (crates/polars-io/src/csv/read/parser.rs:1164):

rust
                Some((mut field, needs_escaping)) => {
                    let field_len = field.len();

                    // +1 is the split character that is consumed by the iterator.
                    read_sol += field_len + 1;

                    if idx == next_projected as u32 {

and the values of non-projected columns are simply never materialised.

Parallelism

The reader splits the file into chunks of at most 16 MB and parses them in parallel with rayon, but the chunk size is not a constant (crates/polars-io/src/csv/read/read_impl.rs:377):

rust
        let n_threads = self
            .n_threads
            .unwrap_or_else(|| RAYON.current_num_threads());

        // This is chosen by benchmarking on ny city trip csv dataset.
        // We want small enough chunks such that threads start working as soon as possible
        // But we also want them large enough, so that we have less chunks related overhead.
        // We minimize chunks to 16 MB to still fit L3 cache.
        //
        // Width-aware adjustment: For wide data (many columns), per-chunk overhead
        // (allocating column buffers) becomes significant. Each chunk must allocate
        // O(n_cols) buffers, so total allocation overhead is O(n_chunks * n_cols).
        // To keep this bounded, we limit n_chunks such that n_chunks * n_cols <= threshold.
        // With threshold ~500K, this gives:
        //   - 100 cols: up to 5000 chunks (no practical limit)
        //   - 1000 cols: up to 500 chunks
        //   - 10000 cols: up to 50 chunks
        //   - 30000 cols: up to 16 chunks
        let n_cols = projection.len();
        // Empirically determined to balance allocation overhead and parallelism.
        const ALLOCATION_BUDGET: usize = 500_000;
        let max_chunks_for_width = ALLOCATION_BUDGET / n_cols.max(1);
        let n_parts_hint = std::cmp::min(n_threads * 16, max_chunks_for_width.max(n_threads));
        let chunk_size = std::cmp::min(bytes.len() / n_parts_hint.max(1), 16 * 1024 * 1024);

That is a nice piece of engineering honesty: the chunk count is not only about parallelism, it is about how many column buffers get allocated. A 30 000-column file would otherwise allocate 30 000 builders per chunk.

Chunk boundaries are found with CountLines::find_next, which counts newlines in a window and doubles the window until it finds at least one — the same quote-aware counter, used for chunking rather than for count(*).

Inference reuses the splitter

Schema inference is not a separate parser. The header line is split with the same SplitFields iterator, and the sample lines are split the same way (crates/polars-io/src/csv/read/schema_inference.rs:187):

rust
    let record = SplitFields::new(
        line,
        parse_options.separator,
        parse_options.quote_char,
        parse_options.eol_char,
    );

    for (i, (slice, needs_escaping)) in record.enumerate() {

Each field is then classified by a set of regular expressions — boolean, then integer, then float, then date, then string (crates/polars-io/src/csv/read/schema_inference.rs:337):

rust
pub fn infer_field_schema(string: &str, try_parse_dates: bool, decimal_comma: bool) -> DataType {
    // when quoting is enabled in the reader, these quotes aren't escaped, we default to
    // String for them
    let bytes = string.as_bytes();
    if bytes.len() >= 2 && *bytes.first().unwrap() == b'"' && *bytes.last().unwrap() == b'"' {

Note the first branch: a value that is still quoted after unescaping is treated as a string, not a number. "123" is a string, 123 is an integer, and getting that wrong is the classic CSV type-inference bug.

The types found in the sample are accumulated per column in a set, and the final dtype is the supertype of the set — so a column that is an integer in the first ten thousand rows and a float in the next ten becomes a float, and a column that is sometimes a string becomes a string. The cost of inference is therefore one pass over infer_schema_length rows, with a regex per field.

Ragged rows and null values

Malformed data is common enough that the reader has to have a policy for it. A row with fewer fields than the schema gets nulls for the missing ones, and the reader notices this by comparing the number of fields it saw with the number it expected (crates/polars-io/src/csv/read/parser.rs:1179):

rust
        // there can be lines that miss fields (also the comma values)
        // this means the splitter won't process them.
        // We traverse them to read them as null values.
        while processed_fields < projection.len() {
            debug_assert!(processed_fields < buffers.len());
            let buf = unsafe {
                // SAFETY: processed fields index can never exceed the projection indices.
                buffers.get_unchecked_mut(processed_fields)
            };
            buf.add_null(!parse_options.missing_is_null);
            processed_fields += 1;
        }

A row with more fields than the schema is an error unless truncate_ragged_lines is set, and the error message carries the byte offset of the offending field, which is why the parser keeps a pointer to the start of the chunk rather than only an index.

The null_values option is checked per field, before the value is parsed, against a compiled matcher for the column’s dtype (crates/polars-io/src/csv/read/parser.rs:1196):

rust
                        // if we have null values argument, check if this field equal null value
                        if let Some(null_values) = null_values {
                            let field = if needs_escaping && !field.is_empty() {
                                unsafe { field.get_unchecked(1..field.len() - 1) }
                            } else {
                                field
                            };

                            // SAFETY:
                            // process fields is in bounds
                            add_null = unsafe { null_values.is_null(field, idx as usize) }
                        }

The bottom line

The parser is a zero-copy splitter plus a per-dtype value parser. Fields are &[u8] slices into the mmap’d buffer; the numeric parsers read them in place; only strings are copied.
Field and line boundaries are found with 64-byte SIMD blocks and a carry-less multiply that computes quote parity for all 64 positions at once. The mask of valid ends is cached, so a row with 20 fields costs one block scan, not 20.
simdutf8 validates the string payload — per chunk in the default encoding, per field only when the encoding is lossy or errors are being ignored.
memchr is used for the quote-unaware lookups (line skipping, comment detection, complete-row detection), not for the hot splitting loops.
The trade-off of the SIMD path: the quote-parity logic assumes quotes are balanced by toggling, which is what RFC 4180 says, but a file with a stray quote in an unquoted field will shift the parity. The scalar path has the same assumption, so this is a correctness constraint of the format, not of the implementation.