How Go's Unicode Tables Are Generated and Why the Runtime Doesn't Use Them

Jay Hubbert — Wed, 21 Oct 2026 04:16:00 +0000

TL;DR: unicode.IsLetter('é') is a binary search over a sorted list of ranges, each with a stride that lets one entry cover a whole arithmetic progression of code points. Below U+0100 there’s a 256-byte property table as a fast path, and utf8 decoding uses a 256-entry first-byte table plus five “accept ranges” that say which second bytes are legal. The runtime imports none of it — it treats strings as bytes.

Every Go program that does anything with text ends up depending on the unicode package, and unicode is a 10,000-line generated file. That file is regenerated every time Unicode publishes a new version of the UCD, and its shape is unusual enough to be worth understanding: it is not a trie, it is not a perfect hash, and it is not a sorted set of code points. It’s a sorted list of ranges, with a compression trick that lets a single entry cover many code points, plus a flat byte table for the first 256 of them.

This is a walk through how it all works, based on reading the Go source tree (Go 1.28 development tree, commit c0be4674).

Where the tables come from

The whole file is generated. Here’s the first six lines of src/unicode/tables.go:

 go
 
 
// src/unicode/tables.go:1
// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.

package unicode

// Version is the Unicode edition from which the tables are derived.
const Version = "17.0.0"

That single constant is the whole version story: Unicode 17.0.0. The unicode package’s tables and the utf8 package’s tables are kept consistent by the unicode/utf8 tests, which is why the latter doesn’t depend on the former.

The generator itself is not in this tree. It used to be — Go had a src/unicode/maketables.go that read UnicodeData.txt and friends from the UCD archive and emitted these tables — but the generation now lives alongside golang.org/x/text, and the only thing left behind in the Go tree is the generated output and the hand-written lookup code around it.

The input files are the standard Unicode Character Database: UnicodeData.txt for the general categories and the code point names, DerivedCoreProperties.txt for properties like White_Space and Unified_Ideograph, CaseFolding.txt for simple case folding, and PropList.txt for the binary properties. The generator turns those into range tables.

The shape of a RangeTable

Here is the central data structure:

 go
 
 
// src/unicode/letter.go:16
// RangeTable defines a set of Unicode code points by listing the ranges of
// code points within the set. The ranges are listed in two slices
// to save space: a slice of 16-bit ranges and a slice of 32-bit ranges.
// The two slices must be in sorted order and non-overlapping.
// Also, R32 should contain only values >= 0x10000 (1<<16).
type RangeTable struct {
	R16 []Range16
	R32 []Range32
	LatinOffset int // number of entries in R16 with Hi <= MaxLatin1
}

// Range16 represents of a range of 16-bit Unicode code points. The range runs from Lo to Hi
// inclusive and has the specified stride.
type Range16 struct {
	Lo uint16
	Hi uint16
	Stride uint16
}

// Range32 represents of a range of Unicode code points and is used when one or
// more of the values will not fit in 16 bits. The range runs from Lo to Hi
// inclusive and has the specified stride. Lo and Hi must always be >= 1<<16.
type Range32 struct {
	Lo uint32
	Hi uint32
	Stride uint32
}

Three fields, and the third is the interesting one. Lo and Hi bound a range. Stride says how to step through it.

Stride == 1 means “every code point in [Lo, Hi] is in the set”. Stride == 2 means “every even code point in [Lo, Hi] is in the set”. Stride == 3 means every third, and so on.

That’s a big compression win, because Unicode categories are full of alternating patterns. Look at the actual generated data for the C (category “Other”) table:

 go
 
 
// src/unicode/tables.go:96
var _C = &RangeTable{
	R16: []Range16{
		{0x0000, 0x001f, 1},
		{0x007f, 0x009f, 1},
		{0x00ad, 0x0378, 715},
		{0x0379, 0x0380, 7},
		{0x0381, 0x0383, 1},
		{0x038b, 0x038d, 2},
		{0x03a2, 0x0530, 398},
		{0x0557, 0x0558, 1},
		{0x058b, 0x058c, 1},
		{0x0590, 0x05c8, 56},
		{0x05c9, 0x05cf, 1},

Read {0x00ad, 0x0378, 715} carefully: from U+00AD to U+0378, step by 715. That’s two code points — U+00AD and U+0378 — covering 715 code points of range. And {0x00ad, 0x0378, 715} is two entries’ worth of information in twelve bytes.

The comment in RangeTable says R32 must contain only values ≥ 0x10000. So the table is split: R16 covers the Basic Multilingual Plane, R32 covers the astral planes. Code points below 0x10000 fit in a uint16; the rest don’t. That’s the “two-level” structure — not a trie, but a split by integer width.

LatinOffset is the third field and it’s a search optimisation: it records how many entries of R16 have Hi <= MaxLatin1, so the caller can skip straight past the Latin-1 portion when it already knows the answer.

Stride compression, in code

The lookup code has to honour the stride:

 go
 
 
// src/unicode/letter.go:90
// is16 reports whether r is in the sorted slice of 16-bit ranges.
func is16(ranges []Range16, r uint16) bool {
	if len(ranges) <= linearMax || r <= MaxLatin1 {
		for i := range ranges {
			range_ := &ranges[i]
			if r < range_.Lo {
				return false
			}
			if r <= range_.Hi {
				return range_.Stride == 1 || (r-range_.Lo)%range_.Stride == 0
			}
		}
		return false
	}

	// binary search over ranges
	lo := 0
	hi := len(ranges)
	for lo < hi {
		m := int(uint(lo+hi) >> 1)
		range_ := &ranges[m]
		if range_.Lo <= r && r <= range_.Hi {
			return range_.Stride == 1 || (r-range_.Lo)%range_.Stride == 0
		}
		if r < range_.Lo {
			hi = m
		} else {
			lo = m + 1
		}
	}
	return false
}

The stride test is (r - range_.Lo) % range_.Stride == 0. If the code point is inside the range but not on the right step, it is not a member.

Two search strategies, chosen by table size:

 go
 
 
// src/unicode/letter.go:86
// linearMax is the maximum size table for linear search for non-Latin1 rune.
// Derived by running 'go test -calibrate'.
const linearMax = 18

Eighteen. Below that a linear scan beats a binary search: the branch predictor learns the loop, and you avoid the data-dependent load of the midpoint. The threshold was found by measurement, not guessed.

Now, the dispatch that decides which search to use:

 go
 
 
// src/unicode/letter.go:156
// Is reports whether the rune is in the specified table of ranges.
func Is(rangeTab *RangeTable, r rune) bool {
	r16 := rangeTab.R16
	// Compare as uint32 to correctly handle negative runes.
	if len(r16) > 0 && uint32(r) <= uint32(r16[len(r16)-1].Hi) {
		return is16(r16, uint16(r))
	}
	r32 := rangeTab.R32
	if len(r32) > 0 && r >= rune(r32[0].Lo) {
		return is32(r32, uint32(r))
	}
	return false
}

func isExcludingLatin(rangeTab *RangeTable, r rune) bool {
	r16 := rangeTab.R16
	// Compare as uint32 to correctly handle negative runes.
	if off := rangeTab.LatinOffset; len(r16) > off && uint32(r) <= uint32(r16[len(r16)-1].Hi) {
		return is16(r16[off:], uint16(r))
	}
	r32 := rangeTab.R32
	if len(r32) > 0 && r >= rune(r32[0].Lo) {
		return is32(r32, uint32(r))
	}
	return false
}

Is is the general entry point. isExcludingLatin is the same thing but starts the R16 scan at LatinOffset, skipping the entries the caller already knows don’t contain the answer.

Both compare as uint32 rather than rune: “Compare as uint32 to correctly handle negative runes.” A negative rune is an invalid code point, and casting to uint32 makes it larger than every valid one, so it falls out of the range checks cleanly.

The Latin-1 fast path

Below U+0100 there is a completely different mechanism: a flat array of 256 bytes, one per code point, each holding a bitmask of properties.

 go
 
 
// src/unicode/graphic.go:7
// Bit masks for each code point under U+0100, for fast lookup.
const (
	pC = 1 << iota // a control character.
	pP // a punctuation character.
	pN // a numeral.
	pS // a symbolic character.
	pZ // a spacing character.
	pLu // an upper-case letter.
	pLl // a lower-case letter.
	pp // a printable character according to Go's definition.
	pg = pp | pZ // a graphical character according to the Unicode definition.
	pLo = pLl | pLu // a letter that is neither upper nor lower case.
	pLmask = pLo
)

And the table itself, at the very bottom of tables.go:

 go
 
 
// src/unicode/tables.go:9426
var properties = [MaxLatin1 + 1]uint8{
	0x00: pC, // '\x00'
	0x01: pC, // '\x01'
	0x02: pC, // '\x02'
	0x03: pC, // '\x03'
	0x04: pC, // '\x04'
	0x05: pC, // '\x05'
	0x06: pC, // '\x06'
	0x07: pC, // '\a'

256 bytes. The whole Latin-1 range, classified, in a quarter of a kilobyte. A single array index.

Which means unicode.IsLetter is two completely different code paths depending on the argument:

 go
 
 
// src/unicode/graphic.go:89
// IsLetter reports whether the rune is a letter (category [L]).
func IsLetter(r rune) bool {
	if uint32(r) <= MaxLatin1 {
		return properties[uint8(r)]&(pLmask) != 0
	}
	return isExcludingLatin(Letter, r)
}

For ASCII and Latin-1, it’s a load and a mask. For anything else, it’s a binary search over a table with a few hundred entries.

The same trick appears in IsDigit:

 go
 
 
// src/unicode/digit.go:7
// IsDigit reports whether the rune is a decimal digit.
func IsDigit(r rune) bool {
	if r <= MaxLatin1 {
		return '0' <= r && r <= '9'
	}
	return isExcludingLatin(Digit, r)
}

Note that IsDigit doesn’t even consult the properties table for Latin-1 — it hardcodes '0' <= r && r <= '9', because decimal digits in Latin-1 are exactly the ASCII digits. The properties table is used for the category Nd, which includes things like superscripts and other numeric forms; IsDigit means ASCII-style digits only.

IsGraphic and IsPrint show the same structure with different masks:

 go
 
 
// src/unicode/graphic.go:33
// IsGraphic reports whether the rune is defined as a Graphic by Unicode.
// Such characters include letters, marks, numbers, punctuation, symbols, and
// spaces, from categories [L], [M], [N], [P], [S], [Zs].
func IsGraphic(r rune) bool {
	// We convert to uint32 to avoid the extra test for negative,
	// and in the index we convert to uint8 to avoid the range check.
	if uint32(r) <= MaxLatin1 {
		return properties[uint8(r)]&pg != 0
	}
	return In(r, GraphicRanges...)
}

// IsPrint reports whether the rune is defined as printable by Go. Such
// characters include letters, marks, numbers, punctuation, symbols, and the
// ASCII space character, from categories [L], [M], [N], [P], [S] and the ASCII space
// character. This categorization is the same as [IsGraphic] except that the
// only spacing character is ASCII space, U+0020.
func IsPrint(r rune) bool {
	if uint32(r) <= MaxLatin1 {
		return properties[uint8(r)]&pp != 0
	}
	return In(r, PrintRanges...)
}

The comment “We convert to uint32 to avoid the extra test for negative, and in the index we convert to uint8 to avoid the range check” is a nice piece of micro-optimisation commentary: casting to uint32 handles the negativity check for free (negatives become huge), and casting to uint8 for the index lets the compiler drop the bounds check because uint32(r) <= MaxLatin1 implies uint8(r) < 256.

Case mapping and the delta trick

Case conversion uses a different table, because the mapping isn’t a membership question — it’s an arithmetic one:

 go
 
 
// src/unicode/letter.go:44
// CaseRange represents a range of Unicode code points for simple (one
// code point to one code point) case conversion.
// The range runs from Lo to Hi inclusive, with a fixed stride of 1. Deltas
// are the number to add to the code point to reach the code point for a
// different case for that character. They may be negative. If zero, it
// means the character is in the corresponding case. There is a special
// case representing sequences of alternating corresponding Upper and Lower
// pairs. It appears with a fixed Delta of
//
//	{UpperLower, UpperLower, UpperLower}
//
// The constant UpperLower has an otherwise impossible delta value.
type CaseRange struct {
	Lo uint32
	Hi uint32
	Delta d
}

Delta is a three-element array — one delta for upper case, one for lower case, one for title case:

 go
 
 
// src/unicode/letter.go:69
// Indices into the Delta arrays inside CaseRanges for case mapping.
const (
	UpperCase = iota
	LowerCase
	TitleCase
	MaxCase
)

type d [MaxCase]rune // to make the CaseRanges text shorter

And the generated data:

 go
 
 
// src/unicode/tables.go:9085
// CaseRanges is the table describing case mappings for all letters with
// non-self mappings.
var CaseRanges = _CaseRanges
var _CaseRanges = []CaseRange{
	{0x0041, 0x005A, d{0, 32, 0}},
	{0x0061, 0x007A, d{-32, 0, -32}},
	{0x00B5, 0x00B5, d{743, 0, 743}},
	{0x00C0, 0x00D6, d{0, 32, 0}},
	{0x00D8, 0x00DE, d{0, 32, 0}},
	{0x00E0, 0x00F6, d{-32, 0, -32}},
	{0x00F8, 0x00FE, d{-32, 0, -32}},
	{0x00FF, 0x00FF, d{121, 0, 121}},
	{0x0100, 0x012F, d{UpperLower, UpperLower, UpperLower}},
	{0x0130, 0x0130, d{0, -199, 0}},
	{0x0131, 0x0131, d{-232, 0, -232}},

{0x0041, 0x005A, d{0, 32, 0}} is the whole of ASCII uppercase: ‘A’ to ‘Z’, and to get lowercase you add 32. d{0, 32, 0} means: upper case delta 0 (already upper), lower case delta 32 (add 32), title case delta 0.

d{121, 0, 121} for U+00FF (ÿ) is the interesting one: to get uppercase you add 121, giving U+0178 (Ÿ). Unicode’s case mappings are not always subtract-32.

And UpperLower is the special case for alternating pairs:

 go
 
 
// src/unicode/letter.go:79
// If the Delta field of a [CaseRange] is UpperLower, it means
// this CaseRange represents a sequence of the form (say)
// [Upper] [Lower] [Upper] [Lower].
const (
	UpperLower = MaxRune + 1 // (Cannot be a valid delta.)
)

MaxRune + 1 — a value that cannot possibly be a valid delta, used as a sentinel. The conversion code detects it and flips the low bit of the offset instead:

 go
 
 
// src/unicode/letter.go:240
		// The correct mapping can be done by clearing or setting the low
		// bit in the sequence offset.
		// The constants UpperCase and TitleCase are even while LowerCase
		// is odd so we take the low bit from _case.
		return rune(cr.Lo) + ((r-rune(cr.Lo))&^1 | rune(_case&1))
	}
	return r + delta
}

UpperCase is 0 (even), LowerCase is 1 (odd), TitleCase is 2 (even). The low bit of the case constant selects upper or lower; the low bit of the offset selects which member of the alternating pair you are. One branchless expression handles the whole alternating range.

Lookup is a plain binary search:

 go
 
 
// src/unicode/letter.go:211
func lookupCaseRange(r rune, caseRange []CaseRange) *CaseRange {
	// binary search over ranges
	lo := 0
	hi := len(caseRange)
	for lo < hi {
		m := int(uint(lo+hi) >> 1)
		cr := &caseRange[m]
		if rune(cr.Lo) <= r && r <= rune(cr.Hi) {
			return cr
		}
		if r < rune(cr.Lo) {
			hi = m
		} else {
			lo = m + 1
		}
	}
	return nil
}

The special cases — Turkish and Azeri — are hand-written, because the generator never got around to them:

 go
 
 
// src/unicode/casetables.go:5
// TODO: This file contains the special casing rules for Turkish and Azeri only.
// It should encompass all the languages with special casing rules
// and be generated automatically, but that requires some API
// development first.

That’s an honest TODO left in the tree, and it’s a good illustration of the gap between “Unicode has a data file for this” and “Go exposes it”.

Simple folding

Case folding is a third thing: mapping a code point to any other code point in its equivalence class. SimpleFold walks that class:

 go
 
 
// src/unicode/letter.go:336
// SimpleFold iterates over Unicode code points equivalent under
// the Unicode-defined simple case folding. Among the code points
// equivalent to rune (including rune itself), SimpleFold returns the
// smallest rune > r if one exists, or else the smallest rune >= 0.
// If r is not a valid Unicode code point, SimpleFold(r) returns r.
//
// For example:
//
//	SimpleFold('A') = 'a'
//	SimpleFold('a') = 'A'
//
//	SimpleFold('K') = 'k'
//	SimpleFold('k') = '\u212A' (Kelvin symbol, K)
//	SimpleFold('\u212A') = 'K'
//

The Kelvin sign is the classic case: ‘K’, ‘k’ and ‘K’ (U+212A) are all equivalent under case folding. A simple delta table can’t express that, because the equivalence class isn’t a simple arithmetic progression. So there’s a separate small table:

 go
 
 
// src/unicode/tables.go:9816
var caseOrbit = []foldPair{
	{0x004B, 0x006B},
	{0x0053, 0x0073},
	{0x006B, 0x212A},
	{0x0073, 0x017F},
	{0x00B5, 0x039C},
	{0x00C5, 0x00E5},
	{0x00DF, 0x1E9E},
	{0x00E5, 0x212B},
	{0x0130, 0x0130},
	{0x0131, 0x0131},
	{0x017F, 0x0053},
	{0x01C4, 0x01C5},
	{0x01C5, 0x01C6},
	{0x01C6, 0x01C4},

An orbit: each entry says “these two are equivalent under folding”. SimpleFold looks up the current rune in this table first, and if it’s not there, falls back to the lower/upper case ranges.

utf8: the accept-ranges trick

The utf8 package is a different kind of table. It doesn’t need Unicode properties at all — it only needs to know what a legal UTF-8 encoding looks like.

The design is a 256-entry table indexed by the first byte, whose entries pack two facts: how long the sequence is, and which “accept range” applies to the second byte.

 go
 
 
// src/unicode/utf8/utf8.go:49
	// These names of these constants are chosen to give nice alignment in the
	// table below. The first nibble is an index into acceptRanges or F for
	// special one-byte cases. The second nibble is the Rune length or the
	// Status for the special one-byte case.
	xx = 0xF1 // invalid: size 1
	as = 0xF0 // ASCII: size 1
	s1 = 0x02 // accept 0, size 2
	s2 = 0x13 // accept 1, size 3
	s3 = 0x03 // accept 0, size 3
	s4 = 0x23 // accept 2, size 3
	s5 = 0x34 // accept 3, size 4
	s6 = 0x04 // accept 0, size 4
	s7 = 0x44 // accept 4, size 4
)

The high nibble is the accept-range index, the low nibble is the sequence length. So s7 = 0x44 means “4 bytes, use accept range 4”. xx = 0xF1 means “invalid, treat as size 1”.

The first-byte table itself:

 go
 
 
// src/unicode/utf8/utf8.go:70
// first is information about the first byte in a UTF-8 sequence.
var first = [256]uint8{
	// 1 2 3 4 5 6 7 8 9 A B C D E F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x00-0x0F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x10-0x1F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x20-0x2F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x30-0x3F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x40-0x4F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x50-0x5F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x60-0x6F
	as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x70-0x7F
	// 1 2 3 4 5 6 7 8 9 A B C D E F
	xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0x80-0x8F
	xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0x90-0x9F
	xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0xA0-0xAF
	xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0xB0-0xBF
	xx, xx, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, // 0xC0-0xCF
	s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, // 0xD0-0xDF
	s2, s3, s3, s3, s3, s3, s3, s3, s3, s3, s3, s3, s3, s4, s3, s3, // 0xE0-0xEF
	s5, s6, s6, s6, s7, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0xF0-0xFF
}

Now the accept ranges:

 go
 
 
// src/unicode/utf8/utf8.go:92
// acceptRange gives the range of valid values for the second byte in a UTF-8
// sequence.
type acceptRange struct {
	lo uint8 // lowest value for second byte.
	hi uint8 // highest value for second byte.
}

// acceptRanges has size 16 to avoid bounds checks in the code that uses it.
var acceptRanges = [16]acceptRange{
	0: {locb, hicb},
	1: {0xA0, hicb},
	2: {locb, 0x9F},
	3: {0x90, hicb},
	4: {locb, 0x8F},
}

Five real ranges. Why do they exist at all? Because UTF-8’s leading bytes encode some of the value, and the second byte’s legal range depends on which bits the first byte already supplied. The classic example: a 3-byte sequence starting with 0xE0 has its second byte restricted to 0xA0..0xBF, because 0xE0 already means the code point starts at 0x0000, and allowing 0x80..0x9F there would let you encode an overlong (invalid) form.

The five ranges cover exactly those boundary cases:

Range	Second byte	Why
0	`0x80..0xBF`	The default continuation byte range
1	`0xA0..0xBF`	After `0xE0` — prevents overlong 3-byte forms
2	`0x80..0x9F`	After `0xED` — prevents UTF-16 surrogates `0xD800..0xDFFF`
3	`0x90..0xBF`	After `0xF0` — prevents overlong 4-byte forms
4	`0x80..0x8F`	After `0xF4` — prevents code points above `0x10FFFF`

That’s the entire validation of UTF-8, in five lines of table.

Now the decoder:

 go
 
 
// src/unicode/utf8/utf8.go:219
func DecodeRuneInString(s string) (r rune, size int) {
	// Inlineable fast path for ASCII characters; see #48195.
	// This implementation is a bit weird but effective at rendering the
	// function inlineable.
	if s != "" && s[0] < RuneSelf {
		return rune(s[0]), 1
	} else {
		r, size = decodeRuneInStringSlow(s)
	}
	return
}

func decodeRuneInStringSlow(s string) (rune, int) {
	n := len(s)
	if n < 1 {
		return RuneError, 0
	}
	s0 := s[0]
	x := first[s0]
	if x >= as {
		// The following code simulates an additional check for x == xx and
		// handling the ASCII and invalid cases accordingly. This mask-and-or
		// approach prevents an additional branch.
		mask := rune(x) << 31 >> 31 // Create 0x0000 or 0xFFFF.
		return rune(s[0])&^mask | RuneError&mask, 1
	}
	sz := int(x & 7)
	accept := acceptRanges[x>>4]
	if n < sz {
		return RuneError, 1
	}
	s1 := s[1]
	if s1 < accept.lo || accept.hi < s1 {
		return RuneError, 1
	}
	if sz <= 2 { // <= instead of == to help the compiler eliminate some bounds checks
		return rune(s0&mask2)<<6 | rune(s1&maskx), 2
	}
	s2 := s[2]
	if s2 < locb || hicb < s2 {
		return RuneError, 1
	}
	if sz <= 3 {
		return rune(s0&mask3)<<12 | rune(s1&maskx)<<6 | rune(s2&maskx), 3
	}
	s3 := s[3]
	if s3 < locb || hicb < s3 {
		return RuneError, 1
	}
	return rune(s0&mask4)<<18 | rune(s1&maskx)<<12 | rune(s2&maskx)<<6 | rune(s3&maskx), 4
}

The mask-and-or trick for the ASCII/invalid case is worth a moment. x is as (0xF0) for ASCII and xx (0xF1) for an invalid starter byte, so the low bit of the table entry distinguishes the two cases. rune(x) << 31 shifts that low bit up to the sign bit, and the arithmetic >> 31 turns it back into a full mask: 0x0000 for ASCII, all-ones for invalid. Then rune(s[0])&^mask | RuneError&mask picks the byte or the replacement rune without a branch.

Now utf8.Valid:

 go
 
 
// src/unicode/utf8/utf8.go:469
// Valid reports whether p consists entirely of valid UTF-8-encoded runes.
func Valid(p []byte) bool {
	// This optimization avoids the need to recompute the capacity
	// when generating code for slicing p, bringing it to parity with
	// ValidString, which was 20% faster on long ASCII strings.
	p = p[:len(p):len(p)]

	for len(p) > 0 {
		p0 := p[0]
		if p0 < RuneSelf {
			p = p[1:]
			// If there's one ASCII byte, there are probably more.
			// Advance quickly through ASCII-only data.
			// Note: using > instead of >= here is intentional. That avoids
			// needing pointing-past-the-end fixup on the slice operations.
			if len(p) > ptrSize && word(p)&hiBits == 0 {
				p = p[ptrSize:]
				if len(p) > 2*ptrSize && (word(p)|word(p[ptrSize:]))&hiBits == 0 {
					p = p[2*ptrSize:]
					for len(p) > 4*ptrSize && ((word(p)|word(p[ptrSize:]))|(word(p[2*ptrSize:])|word(p[3*ptrSize:])))&hiBits == 0 {
						p = p[4*ptrSize:]
					}
				}
			}
			continue
		}

That’s the ASCII fast path: after seeing one ASCII byte, it loads 8 bytes at a time and checks that all the high bits are clear, skipping whole words. On long ASCII strings, utf8.Valid runs at several gigabytes per second.

Why the runtime doesn’t use any of this

Here’s the punchline: the runtime package imports neither unicode nor unicode/utf8. A grep of src/runtime/*.go for those imports turns up exactly one hit, and it’s in runtime/string_test.go.

That’s a deliberate design decision with three reasons.

Size. The runtime is in every binary. unicode/tables.go is over 10,000 lines and several hundred kilobytes of generated data. Importing unicode into the runtime would put all of it into every Go program, including the ones that never look at a non-ASCII character.

The runtime doesn’t need it. Go strings are byte sequences. The runtime manipulates them — slicing, concatenating, hashing, comparing — and none of those operations care whether the bytes are valid UTF-8. The GC traces pointers, not characters. The scheduler doesn’t parse text.

It would be a layering violation. unicode is a user-facing library with a big generated blob and a go:generate pipeline. The runtime is the bottom of the stack. Making the runtime depend on it would mean the runtime’s own binary size is coupled to the Unicode version.

The same reasoning shows up in unicode/utf8, which duplicates a handful of constants rather than importing unicode:

 go
 
 
// src/unicode/utf8/utf8.go:10
// The conditions RuneError==unicode.ReplacementChar and
// MaxRune==unicode.MaxRune are verified in the tests.
// Defining them locally avoids this package depending on package unicode.

Three constants, redefined, with the invariant enforced by a test. That’s the price of keeping the dependency graph acyclic and the binaries small.

Where Unicode knowledge is needed — the compiler, scanning source identifiers, and the strings package — it lives in ordinary library code, not in the runtime.

Trade-offs

What the range-table design buys:

Compactness. The stride trick compresses the categories enormously. A table that would need thousands of individual entries becomes tens of entries.
Speed for the common case. Latin-1 is a single byte lookup. ASCII is a single byte lookup with a mask.
Cache friendliness. A sorted array of 12-byte entries is far more compact than a hash table or a trie, so a binary search touches fewer cache lines.
No initialization. The tables are compile-time constants. No init() function, no lazy building, no lock.

What it costs:

Binary search. IsLetter for a non-Latin-1 rune is O(log n) with data-dependent memory access — the classic profile of a cache-missing binary search.
A lookup is a lookup, not a bit test. For a trie you’d get one memory access; here you get log(n) of them, scattered across the table.
Two code paths everywhere. Every predicate in the package has to handle Latin-1 and non-Latin-1 separately, and the two paths must agree.
Regeneration is a chore. A new Unicode version means regenerating 10,000 lines of Go and reviewing the diff.

Caveats

The tables are only as good as their Unicode version. The Version constant says 17.0.0. Code that depends on the exact set of characters in a category is depending on that version.

IsDigit is not unicode.IsNumber. IsDigit means ASCII-style decimal digits. IsNumber is category N, which includes Roman numerals, superscripts, and fractions. Getting this wrong is a classic source of validation bugs.

Case conversion is “simple” case conversion only. One code point in, one code point out. The German ß uppercases to “SS” — two code points — and unicode.ToUpper('ß') does not do that. The package documentation is explicit:

 go
 
 
// src/unicode/letter.go:66
// BUG(r): There is no mechanism for full case folding, that is, for
// characters that involve multiple runes in the input or output.

Turkish casing is a special case, not a general mechanism. Only Turkish and Azeri have hand-written special tables. Greek final sigma, German sharp s, and Lithuanian rules are not handled.

unicode has no normalisation. NFC and NFD are not in the standard library. If you need them, golang.org/x/text/unicode/norm is where they live.

A rune can hold an invalid code point. rune is just an int32. You can hold -1, or 0x110000, or a surrogate. The range tables handle this by casting to uint32 for comparison, but any code that does its own rune arithmetic has to be careful.

The generated file is generated somewhere else. src/unicode/maketables.go no longer exists in this tree; the header of tables.go credits golang.org/x/text. If you’re looking for the generator in the Go source tree, it isn’t there.

Wrapping up

The one thing to remember: Go’s Unicode tables are sorted range tables with a stride, not a trie. One Range16 entry with Stride == 715 covers two code points 715 apart and stands in for 715 individual membership tests. Below U+0100 there’s a 256-byte property table that bypasses the search entirely. utf8 uses the same idea in a different key: a 256-entry first-byte table plus five “accept ranges” that constrain the second byte to reject overlong encodings, surrogates and out-of-range code points.

The part people get wrong: they assume unicode is a general-purpose character database. It isn’t. It has no normalisation, no full case folding, no grapheme clusters, and only two language-specific casing rules. And unicode.IsDigit is not unicode.IsNumber — the first is ASCII digits, the second is the whole Nd category.

What to go do about it: if you’re validating user input, decide which property you actually mean (IsDigit vs IsNumber vs IsLetter) and write the test with a non-Latin-1 character in it — the Latin-1 fast path and the binary search are different code, and a test suite made entirely of ASCII only ever exercises the first one.

Utf-8 on Opensource internals