Std on Opensource internals

Sixteen Bytes at a Time: How Rust Validates UTF-8

Jay Hubbert — Wed, 21 Oct 2026 12:30:00 +0000

TL;DR: Validating UTF-8 is one of the hottest operations in text processing, and the fast path is not a UTF-8 decoder at all — it is an ASCII scanner. Rust reads two usize words at a time, checks them for high bits with a single AND, and only drops into the byte-at-a-time validator when it finds a non-ASCII byte.

The fast path

 rust
 
 
// library/core/src/str/validations.rs:114-120

const NONASCII_MASK: usize = usize::repeat_u8(0x80);

/// Returns `true` if any byte in the word `x` is nonascii (>= 128).
#[inline]
const fn contains_nonascii(x: usize) -> bool {
 (x & NONASCII_MASK) != 0
}

usize::repeat_u8(0x80) is 0x8080...80 — one high bit per byte. So x & NONASCII_MASK != 0 answers “does this word contain any non-ASCII byte?” with a single AND and a compare.

The scanning loop:

 rust
 
 
// library/core/src/str/validations.rs:219-247

} else {
 // Ascii case, try to skip forward quickly.
 // When the pointer is aligned, read 2 words of data per iteration
 // until we find a word containing a non-ascii byte.
 if align != usize::MAX && align.wrapping_sub(index).is_multiple_of(USIZE_BYTES) {
 let ptr = v.as_ptr();
 while index < blocks_end {
 unsafe {
 let block = ptr.add(index) as *const usize;
 // break if there is a nonascii byte
 let zu = contains_nonascii(*block);
 let zv = contains_nonascii(*block.add(1));
 if zu || zv {
 break;
 }
 }
 index += ascii_block_size;
 }
 // step from the point where the wordwise loop stopped
 while index < len && v[index] < 128 {
 index += 1;
 }
 } else {
 index += 1;
 }
}

Two usize reads (16 bytes on 64-bit) per iteration, two ANDs, two compares. The alignment check (align.wrapping_sub(index).is_multiple_of(USIZE_BYTES)) ensures the word reads are properly aligned — unaligned reads are legal but slow on many CPUs.

For a pure-ASCII string — the overwhelming majority of text in most programs — the whole validation is len / 16 iterations of five instructions.

The slow path

When a non-ASCII byte is found, the validator switches to a per-character state machine that tracks how many bytes the current UTF-8 sequence still needs. That part is necessarily byte-at-a-time, because UTF-8’s validity depends on the continuation bytes that follow.

The ASCII skip loop is the only part that can be vectorised, and it is where all the time goes for typical text.

`const_eval_select`

One more detail: the same source has to work at compile time (for const string validation and for include_str! checks), where align_offset is not a const function. The code handles that with a macro that selects between a const-evaluable path and the runtime path:

rust

// library/core/src/str/validations.rs

The align != usize::MAX test is the signal: align_offset returns usize::MAX when the offset cannot be computed at compile time.

Why it matters

UTF-8 validation happens on every str you create from bytes, on every file you read, on every String you build from a slice. It is one of the most-executed pieces of code in a text-heavy program. Making it 16 bytes per iteration instead of 1 is a 16× difference on the dominant path, and it costs about ten lines of code.

The same “ASCII fast path” trick appears elsewhere in the standard library and in str::find/str::split implementations: check for ASCII in bulk, and only fall back to the Unicode-aware machinery when you actually hit a non-ASCII character.

Where it lives

library/core/src/str/validations.rs — contains_nonascii, the ASCII skip loop, the UTF-8 state machine.
library/core/src/str/mod.rs — from_utf8, from_utf8_unchecked.

Takeaway

The fastest special case is the one that handles most of the data. UTF-8 validation looks like it needs a decoder, but the decoder is only needed for the minority of bytes that are not ASCII — and detecting those in bulk is a two-instruction test on eight bytes at a time. This is the same shape as the SwissTable’s group scan or the ASCII fast paths in memchr: find the interesting bytes in bulk, then handle them one at a time.

One XOR and One CLZ: How Rust's Stable Sort Decides Which Runs to Merge

Jay Hubbert — Wed, 21 Oct 2026 12:25:00 +0000

TL;DR: Rust’s stable sort (driftsort) merges runs using the powersort heuristic: build a merge tree over the runs, where each merge happens at the depth that balances the tree. The clever part is computing that depth — instead of comparing rational numbers, it rescales the number line to [0, 2^62) and takes the leading zeros of an XOR.

The theory

 rust
 
 
// library/core/src/slice/sort/stable/drift.rs:122-171

// Nearly-Optimal Mergesorts: Fast, Practical Sorting Methods That Optimally
// Adapt to Existing Runs by J. Ian Munro and Sebastian Wild.
//
// This method forms a binary merge tree, where each internal node corresponds
// to a splitting point between the adjacent runs that have to be merged. ...
// As an optimization we rescale the number line from [0, 1) to [0, 2^62). Then
// finding the simplest dyadic fraction between midpoints corresponds to finding
// the most significant bit difference of the midpoints. We save scale_factor =
// ceil(2^62 / n) to perform this rescaling using a multiplication, avoiding
// having to repeatedly do integer divides. ...
// Thus for the splitting point between two adjacent slices [a, b) and [b, c)
// the desired depth of the corresponding merge node is CLZ((a+b)*f ^ (b+c)*f),

Read the last line again, because it is the whole trick:

the desired depth of the corresponding merge node is CLZ((a+b)*f ^ (b+c)*f)

where f is a scale factor.

The implementation

 rust
 
 
// library/core/src/slice/sort/stable/drift.rs:122-171

#[inline(always)]
fn merge_tree_scale_factor(n: usize) -> u64 {
 if usize::BITS > u64::BITS {
 panic!("Platform not supported");
 }
 (1u64 << 62).div_ceil(n as u64)
}

#[inline(always)]
fn merge_tree_depth(left: usize, mid: usize, right: usize, scale_factor: u64) -> u8 {
 let x = left as u64 + mid as u64;
 let y = mid as u64 + right as u64;
 ((scale_factor * x) ^ (scale_factor * y)).leading_zeros() as u8
}

The classic powersort formulation compares the rational numbers (a+b)/2 and (b+c)/2 to decide which merge tree is most balanced. Doing that with rational arithmetic means multiplications, comparisons and careful overflow handling.

The trick: rescale [0, 1) to [0, 2^62). After rescaling, “which dyadic fraction is simplest between these two numbers?” becomes “what is the most significant bit where they differ?” — which is exactly leading_zeros(x ^ y). The scale factor f = ceil(2^62 / n) is computed once, and the rescaling is a multiplication.

So each merge-tree depth query is: two additions, two multiplications, one XOR, one lzcnt. No division, no rational comparison, no branch.

The sqrt approximation

The same file avoids the FPU for another purpose:

rust

// library/core/src/slice/sort/stable/drift.rs:173-187

sqrt_approx computes √n with an ilog2 and one Newton step rather than calling f64::sqrt. No floating point, no libm call, and — importantly — no floating point in the middle of a sorting algorithm, where the reproducibility of the output matters.

The scratch buffer sizing

 rust
 
 
// library/core/src/slice/sort/stable/mod.rs:94-135

fn driftsort_main<T, F: FnMut(&T, &T) -> bool, BufT: BufGuard<T>>(v: &mut [T], is_less: &mut F) {
 // By allocating n elements of memory we can ensure the entire input can
 // be sorted using stable quicksort, which allows better performance on
 // random and low-cardinality distributions. However, we still want to
 // reduce our memory usage to n - n / 2 for large inputs. We do this by scaling
 // our allocation as max(n - n / 2, min(n, 8MB)), ensuring we scale like n for
 // small inputs and n - n / 2 for large inputs, without a sudden drop off. ...
 const MAX_FULL_ALLOC_BYTES: usize = 8_000_000; // 8MB
 let max_full_alloc = MAX_FULL_ALLOC_BYTES / size_of::<T>();
 let len = v.len();
 let alloc_len = cmp::max(
 cmp::max(len - len / 2, cmp::min(len, max_full_alloc)),
 SMALL_SORT_GENERAL_SCRATCH_LEN,
 );

 // For small inputs 4KiB of stack storage suffices, which allows us to avoid
 // calling the (de-)allocator. Benchmarks showed this was quite beneficial.
 let mut stack_buf = AlignedStorage::<T, 4096>::new();
 let stack_scratch = stack_buf.as_uninit_slice_mut();
 let mut heap_buf;
 let scratch = if stack_scratch.len() >= alloc_len {
 stack_scratch
 } else {
 heap_buf = BufT::with_capacity(alloc_len);
 heap_buf.as_uninit_slice_mut()
 };

Three decisions:

The scratch is n - n/2 for large inputs, not n — half the memory of a textbook mergesort.
Sorts of ≤ 4 KiB of elements never touch the allocator — the scratch is on the stack.
The BufGuard trait is what lets core (which has no allocator) contain the algorithm while std provides the buffer. That is a nice layering trick: the sorting algorithm lives in core, and the allocation policy is injected.

Where it lives

library/core/src/slice/sort/stable/drift.rs — merge_tree_depth, merge_tree_scale_factor, sqrt_approx.
library/core/src/slice/sort/stable/mod.rs — driftsort_main, the scratch sizing, BufGuard.
library/core/src/slice/sort/stable/merge.rs — the merge itself.

Takeaway

The “simplest dyadic fraction between two rationals” is a natural way to balance a merge tree, but computing it with rational arithmetic is expensive. Rescaling the number line so that the answer becomes a bit difference is a classic “change the representation, change the cost” move: the same mathematical quantity, computed with an XOR and a lzcnt instead of multiplications and comparisons. It is the kind of trick that you can only find if you actually look at what the algorithm needs rather than how the paper writes it down.

Thirteen Elements, Forty-Five Comparisons: The Sorting Network in Rust's sort

Jay Hubbert — Wed, 21 Oct 2026 12:20:00 +0000

TL;DR: Rust’s sort and sort_unstable both fall back to a fixed sorting network for small inputs — up to 13 elements, 45 compare-exchanges — instead of insertion sort. A network has no branches at all: the comparison sequence is fixed in advance, so there is nothing to mispredict.

The rule

 rust
 
 
// library/core/src/slice/sort/stable/mod.rs:28-86

#[inline(always)]
pub fn sort<T, F: FnMut(&T, &T) -> bool, BufT: BufGuard<T>>(v: &mut [T], is_less: &mut F) {
 // Arrays of zero-sized types are always all-equal, and thus sorted.
 if T::IS_ZST {
 return;
 }

 // Instrumenting the standard library showed that 90+% of the calls to sort
 // by rustc are either of size 0 or 1.
 let len = v.len();
 if intrinsics::likely(len < 2) {
 return;
 }
 ...
 // More advanced sorting methods than insertion sort are faster if called
 // in a hot loop for small inputs, but for general-purpose code the small
 // binary size of insertion sort is more important. The instruction cache in
 // modern processors is very valuable, and for a single sort call in general
 // purpose code any gains from an advanced method are cancelled by i-cache
 // misses during the sort, and thrashing the i-cache for surrounding code.
 const MAX_LEN_ALWAYS_INSERTION_SORT: usize = 20;
 if intrinsics::likely(len <= MAX_LEN_ALWAYS_INSERTION_SORT) {
 insertion_sort_shift_left(v, 1, is_less);
 return;
 }

Note the reasoning about insertion sort for the stable sort: the reason is not that insertion sort is faster, but that it is small, and the instruction cache is precious. For the unstable sort (ipnsort) the trade goes the other way — a network is branchless and therefore fast.

The network

 rust
 
 
// library/core/src/slice/sort/shared/smallsort.rs:463-530

/// Sorts the first 13 elements of `v` with a fast fixed function.
///
/// Should `is_less` generate substantial amounts of code the compiler can choose to not inline
/// `swap_if_less`. If the code of a sort impl changes so as to call this function in multiple
/// places, `#[inline(never)]` is recommended to keep binary-size in check. ...
fn sort13_optimal<T, F>(v: &mut [T], is_less: &mut F)
where
 F: FnMut(&T, &T) -> bool,
{
 if v.len() < 13 {
 intrinsics::abort_immediate();
 }
 let v_base = v.as_mut_ptr();
 // Optimal sorting network see:
 // https://bertdobbelaere.github.io/sorting_networks.html.
 unsafe {
 swap_if_less(v_base, 0, 12, is_less);
 swap_if_less(v_base, 1, 10, is_less);
 ...
 swap_if_less(v_base, 5, 6, is_less);
 }
}

Dobbelaere’s optimal 13-element network is 45 compare-exchange operations — provably minimal for 13 inputs. The network is data independent: the same 45 comparisons happen regardless of the input, so the branch predictor never has anything to learn.

swap_if_less compiles to a compare and a conditional swap:

rust

// library/core/src/slice/sort/shared/smallsort.rs

typically becoming a cmov-style branchless select for small Copy types.

The specialisation gate

The network is only used for types that are Copy and small enough to be handled with stack buffers:

rust

// library/core/src/slice/sort/shared/smallsort.rs:142-171

with a stack budget:

 rust
 
const MAX_STACK_ARRAY_SIZE: usize = 4096;

The specialisation matters because the network’s swap_if_less needs the elements to be movable without a destructor; for types with Drop the stable path uses a different, copy-based small sort.

Ping-pong scratch

For the stable sort, small sorts use a scratch buffer in a ping-pong fashion, and the merge-back is panic-safe:

 rust
 
 
// library/core/src/slice/sort/shared/smallsort.rs:220-309

fn small_sort_general_with_scratch<T: FreezeMarker, F: FnMut(&T, &T) -> bool>(
 v: &mut [T],
 scratch: &mut [MaybeUninit<T>],
 is_less: &mut F,
) {
 ...
 // SAFETY: see comment in `CopyOnDrop::drop`.
 let drop_guard = CopyOnDrop { src: scratch_base, dst: v_base, len };

CopyOnDrop is the same pattern as the DropGuard in Vec::drain: if the comparison function panics, the guard still copies the scratch data back into the original array so the Vec is left in a valid state.

Where it lives

library/core/src/slice/sort/shared/smallsort.rs — sort13_optimal, CopyOnDrop, the specialisation gates.
library/core/src/slice/sort/stable/mod.rs — the stable sort’s small-input policy.
library/core/src/slice/sort/unstable/mod.rs — ipnsort, the unstable variant.

Takeaway

Data-dependent branches are the enemy of small sorts. A fixed comparison network does strictly more comparisons than insertion sort in the worst case, but it has zero unpredictable branches, and for small inputs the branch mispredictions cost more than the extra work. This is a classic case where “more operations, fewer branches” wins — and where the optimal network (45 rather than 60-something comparisons) is a nice piece of applied combinatorics.

SipHash's Field Order Is a SIMD Hint: How Rust's Default Hasher Vectorises Itself

Jay Hubbert — Wed, 21 Oct 2026 12:15:00 +0000

TL;DR: SipHash’s four state words are v0, v1, v2, v3. Rust declares them as v0, v2, v1, v3 — and the comment says why: “v0, v2 and v1, v3 show up in pairs in the algorithm, and simd implementations of SipHash will use vectors of v02 and v13. By placing them in this order in the struct, the compiler can pick up on just a few simd optimizations by itself.” A field order is a vectorisation hint.

The state

 rust
 
 
// library/core/src/hash/sip.rs:62-94

#[derive(Debug, Clone, Copy)]
#[repr(C)]
struct State {
 // v0, v2 and v1, v3 show up in pairs in the algorithm,
 // and simd implementations of SipHash will use vectors
 // of v02 and v13. By placing them in this order in the struct,
 // the compiler can pick up on just a few simd optimizations by itself.
 v0: u64,
 v2: u64,
 v1: u64,
 v3: u64,
}

macro_rules! compress {
 ($state:expr) => {{ compress!($state.v0, $state.v1, $state.v2, $state.v3) }};
 ($v0:expr, $v1:expr, $v2:expr, $v3:expr) => {{
 $v0 = $v0.wrapping_add($v1);
 $v2 = $v2.wrapping_add($v3);
 $v1 = $v1.rotate_left(13);
 $v1 ^= $v0;
 $v3 = $v3.rotate_left(16);
 $v3 ^= $v2;
 $v0 = $v0.rotate_left(32);

 $v2 = $v2.wrapping_add($v1);
 $v0 = $v0.wrapping_add($v3);
 $v1 = $v1.rotate_left(17);
 $v1 ^= $v2;
 $v3 = $v3.rotate_left(21);
 $v3 ^= $v0;
 $v2 = $v2.rotate_left(32);
 }};
}

Read the field order: v0, v2, v1, v3. Not v0, v1, v2, v3.

The algorithm interleaves the four words as two independent pairs: v0 accumulates from v1 and v3, v2 accumulates from v3 and v1. The rounds are ARX (add, rotate, xor) operations, and the two pairs have no data dependency between them within a round. A SIMD implementation would put v0 and v2 in one 128-bit lane and v1 and v3 in the other.

By declaring the struct in the order v0, v2, v1, v3, the two lanes are contiguous in memory, so LLVM’s SLP vectoriser can find them without having to prove the reordering itself. A naive field order would require the vectoriser to shuffle.

The comment is unusually explicit about the reasoning: “the compiler can pick up on just a few simd optimizations by itself”.

The round function is deliberately written as a macro

The compress! macro takes the four words as parameters, so the same code can be instantiated over a single State or over four separate locals (which is what the unrolled tail handling does). The macro also avoids re-reading the struct fields between rounds.

The tail, without a memcpy

 rust
 
 
// library/core/src/hash/sip.rs:115-144

/// Loads a u64 using up to 7 bytes of a byte slice. It looks clumsy but the
/// `copy_nonoverlapping` calls that occur (via `load_int_le!`) all have fixed
/// sizes and avoid calling `memcpy`, which is good for speed.
#[inline]
unsafe fn u8to64_le(buf: &[u8], start: usize, len: usize) -> u64 {

The comment says it all: the awkward-looking code is written so that every load is a fixed-size copy_nonoverlapping, which LLVM turns into a single mov instead of a call to memcpy.

The prefix-free string hack

 rust
 
 
// library/core/src/hash/sip.rs:302-308

#[inline]
fn write_str(&mut self, s: &str) {
 // This hasher works byte-wise, and `0xFF` cannot show up in a `str`,
 // so just hashing the one extra byte is enough to be prefix-free.
 self.write(s.as_bytes());
 self.write_u8(0xFF);
}

Hashing "ab" followed by "c" must not collide with "a" followed by "bc". The standard fix is to hash the length, which costs 8 bytes of hashing. Rust’s fix costs one byte: 0xFF cannot appear in a str, so appending it makes the encoding prefix-free without a length field.

Why SipHash at all?

The default RandomState uses SipHash-1-3 because HashMap is exposed to attacker-controlled keys. A non-keyed hash like FxHash is trivially collided: an attacker can find keys that all land in the same bucket and turn a lookup into a linear scan. SipHash’s keyed construction makes that require knowledge of the (random, per-thread, per-map) key.

The compiler does not use SipHash — rustc_data_structures has its own FxHash, which is a multiply-xor-shift hash:

The reason is the opposite: inside a compiler, the keys are not attacker-controlled, and the compiler hashes millions of small keys during a build. DoS resistance is worthless there; speed is everything.

Where it lives

library/core/src/hash/sip.rs — the state layout, the compression macro, the tail handling.
library/core/src/hash/mod.rs — the Hasher trait.
library/std/src/hash/random.rs — RandomState.
compiler/rustc_data_structures/src/fx.rs — the compiler’s own hash.

Takeaway

There is a class of performance hint that lives below the level of algorithms: the order of struct fields. Because LLVM’s SLP vectoriser works on adjacent values, putting v0 next to v2 — rather than in “natural” order — lets it discover a two-lane SIMD computation that it would otherwise have to prove. The cost is zero and the effect is measurable. It is also a lovely reminder that in a systems language, the source layout is part of the design.

Sixteen Buckets at a Time: The SwissTable in Rust's HashMap

Jay Hubbert — Wed, 21 Oct 2026 12:10:00 +0000

TL;DR: HashMap in Rust is a SwissTable: a power-of-two bucket array with a parallel array of control bytes, where the top 7 bits of each hash are stored in the control byte. A lookup loads 16 control bytes with one SSE2 instruction, compares them all at once, and uses movemask to find candidates — 16 comparisons for the price of one.

Note: HashMap in current Rust lives in library/std but the table implementation comes from the external hashbrown crate (hashbrown = "0.17.1" in library/std/Cargo.toml). The snippets below are from hashbrown-0.17.1.

Control bytes

 rust
 
 
// hashbrown-0.17.1/src/control/tag.rs:7-49

impl Tag {
 /// Control tag value for an empty bucket.
 pub(crate) const EMPTY: Tag = Tag(0b1111_1111);

 /// Control tag value for a deleted bucket.
 pub(crate) const DELETED: Tag = Tag(0b1000_0000);

 /// Checks whether a control tag represents a full bucket (top bit is clear).
 #[inline]
 pub(crate) const fn is_full(self) -> bool {
 self.0 & 0x80 == 0
 }
 ...
 /// Creates a control tag representing a full bucket with the given hash.
 #[inline]
 pub(crate) const fn full(hash: u64) -> Tag {
 ...
 // Grab the top 7 bits of the hash. While the hash is normally a full 64-bit
 // value, some hash functions (such as FxHash) produce a usize result
 // instead, which means that the top 32 bits are 0 on 32-bit platforms.
 // So we use MIN_HASH_LEN constant to handle this.
 let top7 = hash >> (MIN_HASH_LEN * 8 - 7);
 Tag((top7 & 0x7f) as u8) // truncation
 }
}

The layout is a deliberate bit-budget:

the top bit distinguishes “special” (EMPTY or DELETED) from “full”;
the remaining 7 bits are h2, the top 7 bits of the hash.

That split is what makes the SIMD scan possible: “is this bucket free?” is “does the top bit of this byte have the high bit set?”, which movemask answers for 16 bytes at once.

The group scan

 rust
 
 
// hashbrown-0.17.1/src/control/group/sse2.rs:46-116

/// Loads a group of tags starting at the given address.
#[inline]
pub(crate) unsafe fn load(ptr: *const Tag) -> Self {
 unsafe { Group(x86::_mm_loadu_si128(ptr.cast())) }
}
...
/// Returns a `BitMask` indicating all tags in the group which have
/// the given value.
#[inline]
pub(crate) fn match_tag(self, tag: Tag) -> BitMask {
 unsafe {
 let cmp = x86::_mm_cmpeq_epi8(self.0, x86::_mm_set1_epi8(tag.0 as i8));
 BitMask(x86::_mm_movemask_epi8(cmp) as u16)
 }
}
...
/// Returns a `BitMask` indicating all tags in the group which are `EMPTY` or `DELETED`.
#[inline]
pub(crate) fn match_empty_or_deleted(&self) -> BitMask {
 unsafe {
 // A tag is EMPTY or DELETED iff the high bit is set
 BitMask(x86::_mm_movemask_epi8(self.0) as u16)
 }
}

#[inline]
pub(crate) fn match_full(&self) -> BitMask {
 BitMask(!self.match_empty_or_deleted().0)
}

Two movemask results: which of the 16 bytes equal my h2 tag, and which are free. One _mm_cmpeq_epi8 plus one _mm_movemask_epi8 does the work of 16 byte comparisons.

And the comment about AVX explains why the table stopped at SSE2:

 rust
 
 
// hashbrown-0.17.1/src/control/group/mod.rs:8-23

// Use the SSE2 implementation if possible: it allows us to scan 16 buckets
// at once instead of 8. We don't bother with AVX since it would require
// runtime dispatch and wouldn't gain us much anyways: the probability of
// finding a match drops off drastically after the first few buckets.

Runtime dispatch for AVX would cost more than it saves, because the probability of finding a match drops off quickly.

The lookup loop

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:2007-2045

#[inline(always)]
unsafe fn find_inner(&self, hash: u64, eq: &mut dyn FnMut(usize) -> bool) -> Option<usize> {
 let tag_hash = Tag::full(hash);
 let mut probe_seq = self.probe_seq(hash);

 loop {
 let group = unsafe { Group::load(self.ctrl(probe_seq.pos)) };

 for bit in group.match_tag(tag_hash) {
 // This is the same as `(probe_seq.pos + bit) % self.num_buckets()` because the number
 // of buckets is a power of two, and `self.bucket_mask = self.num_buckets() - 1`.
 let index = (probe_seq.pos + bit) & self.bucket_mask;

 if likely(eq(index)) {
 return Some(index);
 }
 }

 if likely(group.match_empty().any_bit_set()) {
 return None;
 }

 probe_seq.move_next(self.bucket_mask);
 }
}

Three optimisations in twenty lines:

The h2 filter. The expensive key comparison (eq, which may call Eq::eq on a user type) only runs for buckets whose 7-bit tag matches — a 1/128 false-positive rate.
The empty short-circuit. Finding an EMPTY byte means the key is definitely not in the table, so the lookup ends there.
Mask instead of modulo. & self.bucket_mask replaces %, because the bucket count is a power of two.

Triangular probing

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:57-93

/// Primary hash function, used to select the initial bucket to probe from.
#[inline]
#[expect(clippy::cast_possible_truncation)]
fn h1(hash: u64) -> usize {
 // On 32-bit platforms we simply ignore the higher hash bits.
 hash as usize
}

/// Probe sequence based on triangular numbers, which is guaranteed (since our
/// table size is a power of two) to visit every group of elements exactly once.
///
/// A triangular probe has us jump by 1 more group every time. So first we
/// jump by 1 group (meaning we just continue our linear scan), then 2 groups
/// (skipping over 1 group), then 3 groups (skipping over 2 groups), and so on.
///
/// Proof that the probe will visit every group in the table:
/// <https://fgiesen.wordpress.com/2015/02/22/triangular-numbers-mod-2n/>
#[derive(Clone)]
struct ProbeSeq {
 pos: usize,
 stride: usize,
}

impl ProbeSeq {
 #[inline]
 fn move_next(&mut self, bucket_mask: usize) {
 // We should have found an empty bucket by now and ended the probe.
 debug_assert!(
 self.stride <= bucket_mask,
 "Went past end of probe sequence"
 );

 self.stride += Group::WIDTH;
 self.pos += self.stride;
 self.pos &= bucket_mask;
 }
}

The stride grows by the group width (16 on SSE2) each step, so the probe visits every group exactly once — triangular numbers mod 2ⁿ are a permutation. Note this is not robin-hood hashing and not triple hashing; it is group-wise quadratic probing with a triangular stride.

The load factor

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:104-191

// Otherwise require 1/8 buckets to be empty (87.5% load)
//
// Be careful when modifying this, calculate_layout relies on the
// overflow check here.
let adjusted_cap = cap.checked_mul(8)? / 7;
...
// Returns the maximum effective capacity for the given bucket mask, taking
// the maximum load factor into account.
#[inline]
fn bucket_mask_to_capacity(bucket_mask: usize) -> usize {
 if bucket_mask < 8 {
 // For tables with 1/2/4/8 buckets, we always reserve one empty slot.
 // Keep in mind that the bucket mask is one less than the bucket count.
 bucket_mask
 } else {
 // For larger tables we reserve 12.5% of the slots as empty.
 ((bucket_mask + 1) / 8) * 7
 }
}

87.5% maximum load: the slack keeps probe chains short, and the empty-slot reservation is what makes the “found an empty ⇒ absent” shortcut valid.

The backwards data layout

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:660-687

/// Returns pointer to one past last `data` element in the table as viewed from
/// the start point of the allocation.
...
pub(crate) fn data_end(&self) -> NonNull<T> {
 // `self.table.ctrl.cast()` returns pointer that
 // points here (to the end of `T0`)
 // ∨
 // [Pad], T_n, ..., T1, T0, |CT0, CT1, ..., CT_n|, CTa_0, CTa_1, ..., CTa_m
 // \________ ________/
 // \/
 // `n = buckets - 1`, i.e. `RawTable::num_buckets() - 1`
 //
 // where: T0...T_n - our stored data;
 // CT0...CT_n - control bytes or metadata for `data`.
 // CTa_0...CTa_m - additional control bytes, where `m = Group::WIDTH - 1` (so that the search
 // with loading `Group` bytes from the heap works properly, even if the result
 // of `h1(hash) & self.bucket_mask` is equal to `self.bucket_mask`). See also
 // `RawTableInner::set_ctrl` function.
 self.table.ctrl.cast()
}

Elements are stored backwards relative to the control bytes, so that data_end() is simply “the control pointer” — one allocation, and the control array sits at the end where the group loads can over-read safely (the extra CTa_* bytes are the padding that makes a group load at the last position legal).

The iterator then walks the control bytes forward and subtracts to walk the data backwards:

 rust
 
 
// hashbrown-0.17.1/src/raw.rs:460-471

#[inline]
unsafe fn next_n(&self, offset: usize) -> Self {
 let ptr = if T::IS_ZERO_SIZED {
 // invalid pointer is good enough for ZST
 invalid_mut(self.ptr.as_ptr() as usize + offset)
 } else {
 unsafe { self.ptr.as_ptr().sub(offset) }
 };

This backwards layout is also why HashMap’s iteration order is arbitrary: it is the raw bucket order, deliberately unspecified.

The random seed, once per thread

 rust
 
 
// library/std/src/hash/random.rs:41-78

pub fn new() -> RandomState {
 // Historically this function did not cache keys from the OS and instead
 // simply always called `rand::thread_rng().gen()` twice. In #31356 it
 // was discovered, however, that because we re-seed the thread-local RNG
 // from the OS periodically that this can cause excessive slowdown when
 // many hash maps are created on a thread. To solve this performance
 // trap we cache the first set of randomly generated keys per-thread.
 //
 // Later in #36481 it was discovered that exposing a deterministic
 // iteration order allows a form of DOS attack. To counter that we
 // increment one of the seeds on every RandomState creation, giving
 // every corresponding HashMap a different iteration order.
 thread_local!(static KEYS: Cell<(u64, u64)> = {
 Cell::new(hashmap_random_keys())
 });

 KEYS.with(|keys| {
 let (k0, k1) = keys.get();
 keys.set((k0.wrapping_add(1), k1));
 RandomState { k0, k1 }
 })
}

The OS entropy is fetched once per thread (a getrandom syscall is ~1µs; creating a HashMap must not cost a syscall), while the seed is bumped per map so two maps in the same thread still have different iteration orders.

Where it lives

hashbrown-0.17.1/src/control/tag.rs — the control byte encoding.
hashbrown-0.17.1/src/control/group/sse2.rs — the SIMD group scan.
hashbrown-0.17.1/src/raw.rs — probing, growth, the backwards layout.
library/std/src/hash/random.rs — RandomState.

Takeaway

The SwissTable’s core idea is that a hash table’s expensive part is key comparison, and you can avoid most of it by storing a cheap fingerprint (next to the data) and comparing those in bulk with SIMD. Sixteen candidates at a time, with a 1/128 false-positive rate, is a very good trade — and it is the reason Rust’s HashMap is competitive with C++’s unordered_map.

collect() Reuses Your Allocation: Rust's In-Place Collection

Jay Hubbert — Wed, 21 Oct 2026 12:05:00 +0000

TL;DR: vec.into_iter().map(f).collect::<Vec<_>>() can run entirely inside the original allocation. Rust implements this with a specialisation trait (InPlaceCollect) that checks the size and alignment relationship between the source and destination element types, then reads and writes through two cursors into the same buffer.

The idea

Collecting an iterator normally means: allocate a new buffer, write every element into it, free the old one. But if the iterator is already walking a Vec, and the element types are compatible, you can just write the transformed elements back over the source.

The catch is that the source and destination element types may have different sizes. If the destination is bigger, writing in place would overwrite elements that have not been read yet. The solution is to shrink the allocation first, or to bail out to the normal path.

The layout contract

The source documents the rules precisely:

 rust
 
 
// library/alloc/src/vec/in_place_collect.rs:40-61

// Ultimately each step of the iterator must free up enough *bytes* in the source to make room
// for the next output item.
// If `T` and `U` have the same size no fixup is needed.
// If `T`'s size is a multiple of `U` we can compensate by multiplying the capacity accordingly.
// Otherwise the input capacity (and thus layout) in bytes may not be representable by the output
// `Vec<U>`. In that case `alloc.shrink()` is used to update the allocation's layout.
//
// Alignments of `T` must be the same or larger than `U`. ...

So:

same size → trivial;
source size is a multiple of destination size → the destination capacity is scaled up, and the byte budget works out;
otherwise → the allocation is shrunk to fit the new layout, which is allowed for Vec but is a real cost.

The implementation

 rust
 
 
// library/alloc/src/vec/in_place_collect.rs:249-337

fn from_iter_in_place<I, T>(mut iterator: I) -> Vec<T>
where
 I: Iterator<Item = T> + InPlaceCollect,
 <I as SourceIter>::Source: AsVecIntoIter,
{
 unsafe {
 let inner = iterator.as_inner().as_into_iter();
 (
 inner.buf,
 inner.ptr,
 inner.cap,
 inner.buf.cast::<T>(),
 inner.end as *const T,
 // SAFETY: the multiplication can not overflow, since `inner.cap * size_of::<I::SRC>()` is the size of the allocation.
 inner.cap.unchecked_mul(size_of::<I::Src>()) / size_of::<T>(),
 )
 };

 let len = unsafe {
 SpecInPlaceCollect::collect_in_place(&mut iterator, dst_buf.as_ptr() as *mut T, dst_end)
 };
 ...
 if needs_realloc::<I::Src, T>(src_cap, dst_cap) {
 ...
 let result = alloc.shrink(dst_buf.cast(), old_layout, new_layout);
 let Ok(reallocated) = result else { handle_alloc_error(new_layout) };
 dst_buf = reallocated.cast::<T>();
 }
 ...
 unsafe { Vec::from_parts(dst_buf, len, dst_cap) }
}

The key line is inner.cap * size_of::<I::Src>() / size_of::<T>(): the destination capacity is computed in bytes and then converted, which is what makes the “source is a multiple of destination” case work.

And the result is documented in the source, which is unusual and lovely:

rust

// library/alloc/src/vec/in_place_collect.rs:118-126

Does not allocate, free or panic. On optlevel>=2 it does not loop.

So on optimized builds, vec.into_iter().map(|u| u as isize).collect::<Vec<isize>>() compiles to a counted loop that LLVM can even eliminate.

The specialisation hierarchy

The InPlaceCollect trait is implemented for the iterators that can support it:

rust

// library/alloc/src/vec/in_place_collect.rs:389-413

with the TrustedRandomAccessNoCoerce specialisation being the one that allows the “does not loop” claim.

Even when in-place collection does not apply, collect can avoid the allocation:

 rust
 
 
// library/alloc/src/vec/spec_from_iter.rs:37-64

impl<T> SpecFromIter<T, IntoIter<T>> for Vec<T> {
 fn from_iter(iterator: IntoIter<T>) -> Self {
 // A common case is passing a vector into a function which immediately
 // re-collects into a vector. We can short circuit this if the IntoIter
 // has not been advanced at all.
 // When it has been advanced We can also reuse the memory and move the data to the front.
 // But we only do so when the resulting Vec wouldn't have more unused capacity
 // than creating it through the generic FromIterator implementation would. ...
 let has_advanced = iterator.buf != iterator.ptr;
 if !has_advanced || iterator.len() >= iterator.cap / 2 {
 let it = ManuallyDrop::new(iterator);
 unsafe {
 if has_advanced {
 ptr::copy(it.ptr.as_ptr(), it.buf.as_ptr(), it.len());
 }
 return Vec::from_parts(it.buf, it.len(), it.cap);
 }
 }
 let mut vec = Vec::new();
 vec.spec_extend(iterator);
 vec
 }
}

let v2: Vec<_> = v1.into_iter().collect(); is zero allocations and zero copies when the iterator has not been advanced — and at most one memmove otherwise. The iterator.len() >= iterator.cap / 2 guard prevents keeping a mostly-empty allocation alive.

The `SpecExtend` chain

 rust
 
 
// library/alloc/src/vec/spec_extend.rs:13-59

impl<T, I, A: Allocator> SpecExtend<T, I> for Vec<T, A>
where
 I: Iterator<Item = T>,
{
 default fn spec_extend(&mut self, iter: I) {
 self.extend_desugared(iter)
 }
}

impl<T, I, A: Allocator> SpecExtend<T, I> for Vec<T, A>
where
 I: TrustedLen<Item = T>,
{
 default fn spec_extend(&mut self, iterator: I) {
 self.extend_trusted(iterator)
 }
}

impl<T, A1: Allocator, A2: Allocator> SpecExtend<T, IntoIter<T, A2>> for Vec<T, A1> {
 fn spec_extend(&mut self, iterator: IntoIter<T, A2>) {
 unsafe {
 self.append_elements(iterator.as_slice() as _);
 }
 iterator.forget_remaining_elements_and_dealloc();
 }
}

Three tiers:

TrustedLen → reserve exactly once, then write without per-element capacity checks;
IntoIter → a single memcpy plus “forget the rest of the source”;
slice::Iter with a trivially-clonable T → also a single memcpy.

The TrustedLen path is the interesting one:

 rust
 
 
// library/alloc/src/vec/mod.rs:4231-4264

fn extend_trusted(&mut self, iterator: impl iter::TrustedLen<Item = T>) {
 let (low, high) = iterator.size_hint();
 if let Some(additional) = high {
 debug_assert_eq!(low, additional, "TrustedLen iterator's size hint is not exact: {:?}", (low, high));
 self.reserve(additional);
 let ptr = self.as_mut_ptr();
 let mut local_len = SetLenOnDrop::new(&mut self.len);
 unsafe {
 iterator.for_each(move |element| {
 ptr::write(ptr.add(local_len.current_len()), element);
 // Since the loop executes user code which can panic we have to update
 // the length every step to correctly drop what we've written.
 local_len.increment_len(1);
 });
 }
 } else {
 panic!("capacity overflow");
 }
}

One reserve, then a tight loop with no capacity check. SetLenOnDrop keeps self.len accurate even if the iterator body panics, so partially written elements are dropped correctly.

Where it lives

library/alloc/src/vec/in_place_collect.rs — the in-place machinery.
library/alloc/src/vec/spec_from_iter.rs — allocation stealing.
library/alloc/src/vec/spec_extend.rs — the specialisation chain.
library/alloc/src/vec/mod.rs — extend_trusted, extend_desugared.

Takeaway

collect() is usually thought of as “allocate and fill”. But the common patterns are: extend a Vec with another Vec, or map over a Vec. Both can be done without touching the allocator at all, and Rust implements them with the same trick — specialise on the iterator type, then check the layout relationship between source and destination. The result is a collect() that, in the common case, is a loop.

Vec's Capacity Lives in a Niche: How Rust's RawVec Got One Word Smaller

Jay Hubbert — Wed, 21 Oct 2026 12:00:00 +0000

TL;DR: Vec<T> is three machine words: pointer, length, capacity. It manages that by storing the capacity in a niche type — a usize whose top bit is guaranteed to be zero — so the “zero capacity” case is representable without an extra discriminant. And the growth policy is max(2 * cap, required), with a special minimum for tiny vectors.

The growth policy

 rust
 
 
// library/alloc/src/raw_vec/mod.rs:539-555

// Nothing we can really do about these checks, sadly.
let required_cap = len.checked_add(additional).ok_or(CapacityOverflow)?;

// This guarantees exponential growth. The doubling cannot overflow
// because `cap <= isize::MAX` and the type of `cap` is `usize`.
let cap = cmp::max(self.cap.as_inner() * 2, required_cap);
let cap = cmp::max(min_non_zero_cap(elem_layout.size()), cap);

Three lines:

checked_add catches the length overflow and turns it into a CapacityOverflow error rather than a panic with a confusing message.
max(2*cap, required) is the amortized-doubling rule. If the caller asked for a specific capacity, that wins; otherwise double.
max(min_non_zero_cap(size), cap) imposes a floor.

The doubling cannot overflow because cap <= isize::MAX and the multiplication is done in usize. That invariant is why the code can use plain * instead of checked_mul.

The tiny-vector floor

 rust
 
 
// library/alloc/src/raw_vec/mod.rs:163-176

// Tiny Vecs are dumb. Skip to:
// - 8 if the element size is 1, because any heap allocator is likely
// to round up a request of less than 8 bytes to at least 8 bytes.
// - 4 if elements are moderate-sized (<= 1 KiB).
// - 1 otherwise, to avoid wasting too much space for very short Vecs.
const fn min_non_zero_cap(size: usize) -> usize {
 if size == 1 {
 8
 } else if size <= 1024 {
 4
 } else {
 1
 }
}

The reasoning is allocator-aware: malloc rounds a 1-byte request up to a 16-byte chunk anyway, so allocating 1 byte and then reallocating to 8 is pure waste. For medium-sized elements, 4 gives a reasonable first allocation; for large elements, 1 avoids over-allocating.

The niche

 rust
 
 
// library/alloc/src/raw_vec/mod.rs:40-51

type Cap = core::num::niche_types::UsizeNoHighBit;

// SAFETY: 0 *definitely* is less than isize::MAX.
const ZERO_CAP: Cap = unsafe { Cap::new_unchecked(0) };

/// `Cap(cap)`, except if `T` is a ZST then `Cap::ZERO`.
///
/// # Safety: cap must be <= `isize::MAX`.
const unsafe fn new_cap<T>(cap: usize) -> Cap {
 // SAFETY: Upheld by caller.
 if T::IS_ZST { ZERO_CAP } else { unsafe { Cap::new_unchecked(cap) } }
}

UsizeNoHighBit is a usize with an invariant: the top bit is always zero. That is a niche — a value the type cannot represent — and Rust’s niche optimization uses it to make Option<Cap> the same size as Cap.

Why does that matter? Because RawVec needs to represent “no allocation” (distinguished by a null pointer) and a capacity. With the invariant, the capacity’s top bit is free to be used as a flag… and, in fact, Cap::ZERO is the zero-capacity case, which is also the ZST case.

The payoff: Vec<T> is { ptr, cap, len } — three words — instead of four. For a Vec<u8> of 24 bytes on the stack, that is 8 bytes saved on every Vec in the program, including the ones on the stack of every function.

`into_raw_parts` and `from_raw_parts`

 rust
 
 
// library/alloc/src/vec/mod.rs:795-801

#[must_use = "losing the pointer will leak memory"]
#[stable(feature = "vec_into_raw_parts", since = "1.93.0")]
#[rustc_const_unstable(feature = "const_heap", issue = "79507")]
pub const fn into_raw_parts(self) -> (*mut T, usize, usize) {
 let mut me = ManuallyDrop::new(self);
 (me.as_mut_ptr(), me.len(), me.capacity())
}

ManuallyDrop suppresses the destructor so the buffer is not freed, while the pointer/length/capacity are read out of the value. The reconstruction side is the mirror image:

 rust
 
 
// library/alloc/src/vec/mod.rs:1127-1140

pub const unsafe fn from_raw_parts_in(
 ptr: *mut T,
 length: usize,
 capacity: usize,
 alloc: A,
) -> Self {
 ub_checks::assert_unsafe_precondition!(
 check_library_ub,
 "Vec::from_raw_parts_in requires that length <= capacity",
 (length: usize = length, capacity: usize = capacity) => length <= capacity
 );
 // SAFETY: Upheld by caller.
 unsafe { Vec { buf: RawVec::from_raw_parts_in(ptr, capacity, alloc), len: length } }
}

It is const, and the only validation is a debug-mode UB check. That makes it a zero-cost FFI boundary primitive: Vec::into_raw_parts and Vec::from_raw_parts round-trip through C without touching the allocator at all.

`drain` is destructive but reversible

 rust
 
 
// library/alloc/src/vec/drain.rs:88-137

pub fn keep_rest(self) {
 // At this moment layout looks like this:
 //
 // [head] [yielded by next] [unyielded] [yielded by next_back] [tail]
 // ^-- start \_________/-- unyielded_len \____/-- self.tail_len
 // ^-- unyielded_ptr ^-- tail
 ...
 let mut this = ManuallyDrop::new(self);
 unsafe {
 let source_vec = this.vec.as_mut();
 let start = source_vec.len();
 let tail = this.tail_start;
 let unyielded_len = this.iter.len();
 let unyielded_ptr = this.iter.as_slice().as_ptr();
 // ZSTs have no identity, so we don't need to move them around.
 if !T::IS_ZST {
 let start_ptr = source_vec.as_mut_ptr().add(start);
 // memmove back unyielded elements
 if unyielded_ptr != start_ptr {
 ptr::copy(src, dst, unyielded_len);
 }
 // memmove back untouched tail
 if tail != (start + unyielded_len) {
 ptr::copy(src, dst, this.tail_len);
 }
 }
 source_vec.set_len(start + unyielded_len + this.tail_len);
 }
}

The tail is restored with one memmove, not one move per element. And the Drop implementation uses a nested guard struct so the memmove happens even if the user’s Drop impl panics mid-drain:

 rust
 
 
// library/alloc/src/vec/drain.rs:176-200

impl<T, A: Allocator> Drop for Drain<'a, T, A> {
 fn drop(&mut self) {
 /// Moves back the un-`Drain`ed elements to restore the original `Vec`.
 struct DropGuard<'r, 'a, T, A: Allocator>(&'r mut Drain<'a, T, A>);
 impl<'r, 'a, T, A: Allocator> Drop for DropGuard<'r, 'a, T, A> {
 fn drop(&mut self) {
 if self.0.tail_len > 0 {
 unsafe {
 let source_vec = self.0.vec.as_mut();
 // memmove back untouched tail, update to new length
 let start = source_vec.len();
 let tail = self.0.tail_start;
 if tail != start {
 let src = source_vec.as_ptr().add(tail);
 let dst = source_vec.as_mut_ptr().add(start);
 ptr::copy(src, dst, self.0.tail_len);
 }
 source_vec.set_len(start + self.0.tail_len);
 }
 }
 }
 }

The naive alternative — Vec::remove in a loop — is O(n²) in moves.

Where it lives

library/alloc/src/raw_vec/mod.rs — the growth policy, min_non_zero_cap, the Cap niche.
library/alloc/src/vec/mod.rs — into_raw_parts, from_raw_parts_in.
library/alloc/src/vec/drain.rs — the DropGuard trick.
library/core/src/num/niche_types.rs — UsizeNoHighBit.

Takeaway

The niche trick here is a nice illustration of a general Rust idiom: when a type has a value it can never hold, that value is free. usize can never be usize::MAX in a capacity, so the top bit is free, so Option<Cap> is free, so Vec is one word smaller. Across a program with thousands of live Vecs, that is megabytes of stack and heap.

Std on Opensource internals

Sixteen Bytes at a Time: How Rust Validates UTF-8

The fast path

The slow path

const_eval_select

Why it matters

Where it lives

Takeaway

One XOR and One CLZ: How Rust's Stable Sort Decides Which Runs to Merge

The theory

The implementation

The sqrt approximation

The scratch buffer sizing

Where it lives

Takeaway

Thirteen Elements, Forty-Five Comparisons: The Sorting Network in Rust's sort

The rule

The network

The specialisation gate

Ping-pong scratch

Where it lives

Takeaway

SipHash's Field Order Is a SIMD Hint: How Rust's Default Hasher Vectorises Itself

The state

The round function is deliberately written as a macro

The tail, without a memcpy

The prefix-free string hack

Why SipHash at all?

Where it lives

Takeaway

Sixteen Buckets at a Time: The SwissTable in Rust's HashMap

Control bytes

The group scan

The lookup loop

Triangular probing

The load factor

The backwards data layout

The random seed, once per thread

Where it lives

Takeaway

collect() Reuses Your Allocation: Rust's In-Place Collection

The idea

The layout contract

The implementation

The specialisation hierarchy

Related: stealing the allocation entirely

The SpecExtend chain

Where it lives

Takeaway

Vec's Capacity Lives in a Niche: How Rust's RawVec Got One Word Smaller

The growth policy

The tiny-vector floor

The niche

into_raw_parts and from_raw_parts

drain is destructive but reversible

Where it lives

Takeaway

`const_eval_select`

The `SpecExtend` chain

`into_raw_parts` and `from_raw_parts`

`drain` is destructive but reversible