Opensource internals

Sixteen Bytes at a Time: How Rust Validates UTF-8

4 min read Jay Hubbert

Rust's UTF-8 validation skips ASCII in word-sized chunks - two usize per iteration - and only falls back to byte-at-a-time logic when a non-ASCII byte appears.

TL;DR: Validating UTF-8 is one of the hottest operations in text processing, and the fast path is not a UTF-8 decoder at all — it is an ASCII scanner. Rust reads two usize words at a time, checks them for high bits with a single AND, and only drops into the byte-at-a-time validator when it finds a non-ASCII byte.

The fast path

rust 
// library/core/src/str/validations.rs:114-120

const NONASCII_MASK: usize = usize::repeat_u8(0x80);

/// Returns `true` if any byte in the word `x` is nonascii (>= 128).
#[inline]
const fn contains_nonascii(x: usize) -> bool {
    (x & NONASCII_MASK) != 0
}

usize::repeat_u8(0x80) is 0x8080...80 — one high bit per byte. So x & NONASCII_MASK != 0 answers “does this word contain any non-ASCII byte?” with a single AND and a compare.

The scanning loop:

rust 
// library/core/src/str/validations.rs:219-247

} else {
    // Ascii case, try to skip forward quickly.
    // When the pointer is aligned, read 2 words of data per iteration
    // until we find a word containing a non-ascii byte.
    if align != usize::MAX && align.wrapping_sub(index).is_multiple_of(USIZE_BYTES) {
        let ptr = v.as_ptr();
        while index < blocks_end {
            unsafe {
                let block = ptr.add(index) as *const usize;
                // break if there is a nonascii byte
                let zu = contains_nonascii(*block);
                let zv = contains_nonascii(*block.add(1));
                if zu || zv {
                    break;
                }
            }
            index += ascii_block_size;
        }
        // step from the point where the wordwise loop stopped
        while index < len && v[index] < 128 {
            index += 1;
        }
    } else {
        index += 1;
    }
}

Two usize reads (16 bytes on 64-bit) per iteration, two ANDs, two compares. The alignment check (align.wrapping_sub(index).is_multiple_of(USIZE_BYTES)) ensures the word reads are properly aligned — unaligned reads are legal but slow on many CPUs.

For a pure-ASCII string — the overwhelming majority of text in most programs — the whole validation is len / 16 iterations of five instructions.

The slow path

When a non-ASCII byte is found, the validator switches to a per-character state machine that tracks how many bytes the current UTF-8 sequence still needs. That part is necessarily byte-at-a-time, because UTF-8’s validity depends on the continuation bytes that follow.

The ASCII skip loop is the only part that can be vectorised, and it is where all the time goes for typical text.

const_eval_select

One more detail: the same source has to work at compile time (for const string validation and for include_str! checks), where align_offset is not a const function. The code handles that with a macro that selects between a const-evaluable path and the runtime path:

rust 
// library/core/src/str/validations.rs

The align != usize::MAX test is the signal: align_offset returns usize::MAX when the offset cannot be computed at compile time.

Why it matters

UTF-8 validation happens on every str you create from bytes, on every file you read, on every String you build from a slice. It is one of the most-executed pieces of code in a text-heavy program. Making it 16 bytes per iteration instead of 1 is a 16× difference on the dominant path, and it costs about ten lines of code.

The same “ASCII fast path” trick appears elsewhere in the standard library and in str::find/str::split implementations: check for ASCII in bulk, and only fall back to the Unicode-aware machinery when you actually hit a non-ASCII character.

Where it lives

  • library/core/src/str/validations.rs — contains_nonascii, the ASCII skip loop, the UTF-8 state machine.
  • library/core/src/str/mod.rs — from_utf8, from_utf8_unchecked.

Takeaway

The fastest special case is the one that handles most of the data. UTF-8 validation looks like it needs a decoder, but the decoder is only needed for the minority of bytes that are not ASCII — and detecting those in bulk is a two-instruction test on eight bytes at a time. This is the same shape as the SwissTable’s group scan or the ASCII fast paths in memchr: find the interesting bytes in bulk, then handle them one at a time.

RustInternalsStdPerformanceUnicodeSimd