How Go's Benchmarking Harness Repeats Until It's Sure

TL;DR: A Go benchmark is not run once. The harness runs it with one iteration to check it works, then guesses an iteration count, measures how long that took, predicts a better count, and repeats until the accumulated time hits -benchtime (one second by default). b.N is what it landed on.

Every Go performance story starts the same way: someone writes a benchmark, gets a number, and the number is wrong. Not because the benchmark was a lie, but because of how the harness works. Go’s testing package does not run your function once and time it. It runs your function many times with a varying number of iterations, and the number of iterations is decided at runtime by looking at how fast your code turned out to be.

Understanding that loop is the difference between a benchmark you can trust and a benchmark that will make you quit your job.

The struct you are handed

The testing.B struct is the whole interface between your code and the harness:

go
// src/testing/benchmark.go:94
type B struct {
	common
	importPath       string // import path of the package containing the benchmark
	bstate           *benchState
	N                int
	previousN        int           // number of iterations in the previous run
	previousDuration time.Duration // total duration of the previous run
	benchFunc        func(b *B)
	benchTime        durationOrCountFlag
	bytes            int64
	missingBytes     bool // one of the subbenchmarks does not have bytes set.
	timerOn          bool
	showAllocResult  bool
	result           BenchmarkResult
	parallelism      int // RunParallel creates parallelism*GOMAXPROCS goroutines
	// The initial states of memStats.Mallocs and memStats.TotalAlloc.
	startAllocs uint64
	startBytes  uint64
	// The net total of this test after being run.
	netAllocs uint64
	netBytes  uint64
	// Extra metrics collected by ReportMetric.
	extra map[string]float64

	// loop tracks the state of B.Loop
	loop struct {
		// n is the target number of iterations. It gets bumped up as we go.
		// When the benchmark loop is done, we commit this to b.N so users can
		// do reporting based on it, but we avoid exposing it until then.
		n uint64
		// i is the current Loop iteration. It's strictly monotonically
		// increasing toward n.
		//
		// The high bit is used to poison the Loop fast path and fall back to
		// the slow path.
		i uint64

		done bool // set when B.Loop return false
	}
}

Two fields matter above all: N, the number of iterations the harness decided on, and timerOn, whether the clock is currently running. Everything else is bookkeeping.

previousN and previousDuration are the memory of the last attempt, and they are what makes the ramp-up converge quickly instead of crawling.

The target duration

The harness needs to know how long to spend on a benchmark. That is -benchtime, and the flag is registered here:

go
// src/testing/benchmark.go:25
func initBenchmarkFlags() {
	matchBenchmarks = flag.String("test.bench", "", "run only benchmarks matching `regexp`")
	benchmarkMemory = flag.Bool("test.benchmem", false, "print memory allocations for benchmarks")
	flag.Var(&benchTime, "test.benchtime", "run each benchmark for duration `d` or N times if `d` is of the form Nx")
}

and the default is one second:

go
// src/testing/benchmark.go:31
var (
	matchBenchmarks *string
	benchmarkMemory *bool

	benchTime = durationOrCountFlag{d: 1 * time.Second} // changed during test of testing package
)

One second per benchmark, not 100 nanoseconds. The 100ns figure that circulates online is not the benchtime target; the smallest unit the harness cares about is the time of a single iteration, and the point of the ramp-up is to get the total to a second.

The flag accepts two forms. A duration is a wall-clock target; an Nx form fixes the iteration count exactly:

go
// src/testing/benchmark.go:38
type durationOrCountFlag struct {
	d         time.Duration
	n         int
	allowZero bool
}

func (f *durationOrCountFlag) String() string {
	if f.n > 0 {
		return fmt.Sprintf("%dx", f.n)
	}
	return f.d.String()
}

func (f *durationOrCountFlag) Set(s string) error {
	if strings.HasSuffix(s, "x") {
		n, err := strconv.ParseInt(s[:len(s)-1], 10, 0)
		if err != nil || n < 0 || (!f.allowZero && n == 0) {
			return fmt.Errorf("invalid count")
		}
		*f = durationOrCountFlag{n: int(n)}
		return nil
	}
	d, err := time.ParseDuration(s)
	if err != nil || d < 0 || (!f.allowZero && d == 0) {
		return fmt.Errorf("invalid duration")
	}
	*f = durationOrCountFlag{d: d}
	return nil
}

-benchtime=100x runs exactly 100 iterations and stops. That is what you want when you are comparing two implementations at a fixed workload, or when you want a fast smoke test in CI. -benchtime=5s gives you a longer, steadier number at the cost of a slower test run.

The ramp-up

Here is the loop that decides b.N. It is short and it is the heart of the whole thing:

go
// src/testing/benchmark.go:327
// launch launches the benchmark function. It gradually increases the number
// of benchmark iterations until the benchmark runs for the requested benchtime.
// launch is run by the doBench function as a separate goroutine.
// run1 must have been called on b.
func (b *B) launch() {
	// Signal that we're done whether we return normally
	// or by FailNow's runtime.Goexit.
	defer func() {
		b.signal <- true
	}()

	// b.Loop does its own ramp-up logic so we just need to run it once.
	// If b.loop.n is non zero, it means b.Loop has already run.
	if b.loop.n == 0 {
		// Run the benchmark for at least the specified amount of time.
		if b.benchTime.n > 0 {
			// We already ran a single iteration in run1.
			// If -benchtime=1x was requested, use that result.
			// See https://golang.org/issue/32051.
			if b.benchTime.n > 1 {
				b.runN(b.benchTime.n)
			}
		} else {
			d := b.benchTime.d
			for n := int64(1); !b.failed && b.duration < d && n < 1e9; {
				last := n
				// Predict required iterations.
				goalns := d.Nanoseconds()
				prevIters := int64(b.N)
				n = int64(predictN(goalns, prevIters, b.duration.Nanoseconds(), last))
				b.runN(int(n))
			}
		}
	}
	b.result = BenchmarkResult{b.N, b.duration, b.bytes, b.netAllocs, b.netBytes, b.extra}
}

Read it carefully. b.duration is the accumulated duration across all the runs so far. The loop keeps going until that total reaches d (one second) or until the iteration count hits a billion.

Each time round, it asks predictN how many iterations it should try next, based on how long the previous attempt took.

go
// src/testing/benchmark.go:301
// Don't run more than 1e9 times. (This also keeps n in int range on 32 bit platforms.)
const maxBenchPredictIters = 1_000_000_000

func predictN(goalns int64, prevIters int64, prevns int64, last int64) int {
	if prevns == 0 {
		// Round up to dodge divide by zero. See https://go.dev/issue/70709.
		prevns = 1
	}

	// Order of operations matters.
	// For very fast benchmarks, prevIters ~= prevns.
	// If you divide first, you get 0 or 1,
	// which can hide an order of magnitude in execution time.
	// So multiply first, then divide.
	n := goalns * prevIters / prevns
	// Run more iterations than we think we'll need (1.2x).
	n += n / 5
	// Don't grow too fast in case we had timing errors previously.
	n = min(n, 100*last)
	// Be sure to run at least one more than last time.
	n = max(n, last+1)
	// Don't run more than 1e9 times. (This also keeps n in int range on 32 bit platforms.)
	n = min(n, maxBenchPredictIters)
	return int(n)
}

That is a proportional controller with safety rails. Multiply first, then divide — the comment explains why, and it is a real bug class: for a benchmark that takes 50ns per iteration, dividing first gives you 0 or 1 and you lose an order of magnitude of accuracy. Then overshoot by 20% so you are more likely to overshoot the target time than undershoot it, but never grow by more than 100x in one step (in case the previous measurement was noise), and always run at least one more iteration than last time so the loop terminates.

In practice this converges in two or three rounds. A benchmark that takes 1µs per iteration gets predicted at roughly a million iterations on the first real attempt, and finishes in one or two rounds.

Note also run1, the very first run, which happens before the loop:

go
// src/testing/benchmark.go:229
// run1 runs the first iteration of benchFunc. It reports whether more
// iterations of this benchmarks should be run.
func (b *B) run1() bool {
	if bstate := b.bstate; bstate != nil {
		// Extend maxLen, if needed.
		if n := len(b.name) + bstate.extLen + 1; n > bstate.maxLen {
			bstate.maxLen = n + 8 // Add additional slack to avoid too many jumps in size.
		}
	}
	go func() {
		// Signal that we're done whether we return normally
		// or by FailNow's runtime.Goexit.
		defer func() {
			b.signal <- true
		}()

		b.runN(1)
	}()
	<-b.signal
	if b.failed {
		fmt.Fprintf(b.w, "%s--- FAIL: %s\n%s", b.chatty.prefix(), b.name, b.output)
		return false
	}

One iteration, N=1, on its own goroutine. It is a smoke test: if your benchmark panics, fails, or prints something enormous, you find out here before the harness starts doing a million iterations of it.

What one “run” means

runN is what actually calls your function:

go
// src/testing/benchmark.go:196
// runN runs a single benchmark for the specified number of iterations.
func (b *B) runN(n int) {
	benchmarkLock.Lock()
	defer benchmarkLock.Unlock()
	ctx, cancelCtx := context.WithCancel(context.Background())
	defer func() {
		b.runCleanup(normalPanic)
		b.checkRaces()
	}()
	// Try to get a comparable environment for each run
	// by clearing garbage from previous runs.
	runtime.GC()
	b.resetRaces()
	b.N = n
	b.loop.n = 0
	b.loop.i = 0
	b.loop.done = false
	b.ctx = ctx
	b.cancelCtx = cancelCtx

	b.parallelism = 1
	b.ResetTimer()
	b.StartTimer()
	b.benchFunc(b)
	b.StopTimer()
	b.previousN = n
	b.previousDuration = b.duration

Four things to notice.

runtime.GC() before every run. The harness tries to give each run the same conditions. That is not perfect, but it means a previous run’s garbage does not skew the next one.

b.N = n is set before your function is called. This is why for i := 0; i < b.N; i++ works: b.N is already the right number when you read it.

b.ResetTimer() then b.StartTimer(). The timer is running by default when your function is entered. The reset zeroes the accumulated duration and the allocation counters.

b.previousN = n and b.previousDuration = b.duration at the end. That is the memory the next predictN call uses.

The timer

Because the timer runs by default, anything you do before the measurement loop is measured too. The classic benchmark bug:

go
func BenchmarkParse(b *testing.B) {
	data, _ := os.ReadFile("big.json")   // this is measured!
	var v Config
	for i := 0; i < b.N; i++ {
		v = Parse(data)                   // so is this
	}
}

The file read happens once but is timed once — which sounds harmless until you realise that on the first iteration b.N is 1, so a single os.ReadFile is a large fraction of a one-millisecond total, and the harness then runs the whole thing again with a million iterations where the read is noise. The reported ns/op is a blend of two different things.

The fix is the timer API:

go
// src/testing/benchmark.go:135
// StartTimer starts timing a test. This function is called automatically
// before a benchmark starts, but it can also be used to resume timing after
// a call to [B.StopTimer].
func (b *B) StartTimer() {
	if !b.timerOn {
		runtime.ReadMemStats(&memStats)
		b.startAllocs = memStats.Mallocs
		b.startBytes = memStats.TotalAlloc
		b.start = time.Now()
		b.timerOn = true
		b.loop.i &^= loopPoisonTimer
	}
}

// StopTimer stops timing a test. This can be used to pause the timer
// while performing steps that you don't want to measure.
func (b *B) StopTimer() {
	if b.timerOn {
		b.duration += time.Since(b.start)
		runtime.ReadMemStats(&memStats)
		b.netAllocs += memStats.Mallocs - b.startAllocs
		b.netBytes += memStats.TotalAlloc - b.startBytes
		b.timerOn = false
		// If we hit B.Loop with the timer stopped, fail.
		b.loop.i |= loopPoisonTimer
	}
}

Note that StopTimer also accumulates the allocation counters, so the allocation statistics exclude the paused region too. And note the poison bit: if you stop the timer inside a b.Loop() body, the harness notices and fails the benchmark, because b.Loop assumes the timer is running.

ResetTimer is the other one:

go
// src/testing/benchmark.go:163
// ResetTimer zeroes the elapsed benchmark time and memory allocation counters
// and deletes user-reported metrics.
// It does not affect whether the timer is running.
func (b *B) ResetTimer() {
	if b.extra == nil {
		// Allocate the extra map before reading memory stats.
		// Pre-size it to make more allocation unlikely.
		b.extra = make(map[string]float64, 16)
	} else {
		clear(b.extra)
	}
	if b.timerOn {
		runtime.ReadMemStats(&memStats)
		b.startAllocs = memStats.Mallocs
		b.startBytes = memStats.TotalAlloc
		b.start = time.Now()
	}
	b.duration = 0
	b.netAllocs = 0
	b.netBytes = 0
}

“Does not affect whether the timer is running” — so ResetTimer on its own does not exclude your setup. You need b.StopTimer() before the setup and b.StartTimer() after it. This is the single most common source of meaningless benchmark numbers in Go, and the reason Go 1.24 added b.Loop.

`b.Loop()`, and why the compiler gets involved

b.Loop() is the modern form, added in Go 1.24, and it solves both the setup-exclusion problem and the “the compiler optimised my benchmark away” problem.

go
// src/testing/benchmark.go:468
// Loop returns true as long as the benchmark should continue running.
//
// A typical benchmark is structured like:
//
//	func Benchmark(b *testing.B) {
//		... setup ...
//		for b.Loop() {
//			... code to measure ...
//		}
//		... cleanup ...
//	}
//
// Loop resets the benchmark timer the first time it is called in a benchmark,
// so any setup performed prior to starting the benchmark loop does not count
// toward the benchmark measurement. Likewise, when it returns false, it stops
// the timer so cleanup code is not measured.
//
// Within the body of a "for b.Loop() { ... }" loop, arguments to and
// results from function calls and assigned variables within the loop are kept
// alive, preventing the compiler from fully optimizing away the loop body.
// Currently, this is implemented as a compiler transformation that wraps such
// variables with a runtime.KeepAlive intrinsic call. This applies only to
// statements syntactically between the curly braces of the loop, and the loop
// condition must be written exactly as "b.Loop()".
//
// After Loop returns false, b.N contains the total number of iterations that
// ran, so the benchmark may use b.N to compute other average metrics.
//
// Prior to the introduction of Loop, benchmarks were expected to contain an
// explicit loop from 0 to b.N. Benchmarks should either use Loop or contain a
// loop to b.N, but not both. Loop offers more automatic management of the
// benchmark timer, and runs each benchmark function only once per measurement,
// whereas b.N-based benchmarks must run the benchmark function (and any
// associated setup and cleanup) several times.
func (b *B) Loop() bool {

Two huge behavioural differences from the old form.

The benchmark function runs once. With b.Loop(), the harness does not re-run your function with different N values. b.Loop does its own ramp-up internally, growing its own target count. Your setup runs once, not three times.

Setup and cleanup are excluded automatically. Loop resets the timer on its first call and stops it when it returns false.

And the compiler gets involved to stop the optimiser eating your benchmark. The implementation is a dedicated compiler pass, cmd/compile/internal/bloop:

go
// src/cmd/compile/internal/bloop/bloop.go:143
// keepAliveAt returns a statement that is either curNode, or a
// block containing curNode followed by a call to runtime.KeepAlive for each
// node in ns. These calls ensure that nodes in ns will be live until
// after curNode's execution.
func keepAliveAt(curfunc *ir.Func, ns ir.Nodes, curNode ir.Node) ir.Node {
	if len(ns) == 0 {
		return curNode
	}

	pos := curNode.Pos()
	calls := ir.Nodes{curNode}
	for _, n := range ns {
		if n == nil || n.Sym() == nil || n.Sym().IsBlank() {
			continue
		}
		if !ir.IsAddressable(n) {
			base.FatalfAt(n.Pos(), "keepAliveAt: node %v is not addressable", n)
		}
		arg := ir.NewConvExpr(pos, ir.OCONV, types.Types[types.TUNSAFEPTR], typecheck.NodAddr(curfunc, n))
		callExpr := typecheck.Call(curfunc, pos, typecheck.LookupRuntime("KeepAlive"), ir.Nodes{arg}, false).(*ir.CallExpr)
		callExpr.IsCompilerVarLive = true
		callExpr.NoInline = true
		calls = append(calls, callExpr)
	}

	return ir.NewBlockStmt(pos, calls)
}

The pass recognises the loop by its shape, and only if it matches exactly:

go
// src/cmd/compile/internal/bloop/bloop.go:339
// isTestingBLoop returns true if it matches the node as a
// testing.(*B).Loop. See issue #61515.
func isTestingBLoop(t ir.Node) bool {
	if t.Op() != ir.OFOR {
		return false
	}
	nFor, ok := t.(*ir.ForStmt)
	if !ok || nFor.Cond == nil || nFor.Cond.Op() != ir.OCALLFUNC {
		return false
	}
	n, ok := nFor.Cond.(*ir.CallExpr)
	if !ok || n.Fun == nil || n.Fun.Op() != ir.OMETHEXPR {
		return false
	}
	name := ir.MethodExprName(n.Fun)
	if name == nil {
		return false
	}
	if fSym := name.Sym(); fSym != nil && name.Class == ir.PFUNC && fSym.Pkg != nil &&
		fSym.Name == "(*B).Loop" && fSym.Pkg.Path == "testing" {
		// Attempting to match a function call to testing.(*B).Loop
		return true
	}
	return false
}

The loop condition must literally be b.Loop(). Write b.Loop() in a variable and the transformation does not apply, and your benchmark body is fair game for dead-code elimination.

The Loop fast path is written to be inlinable:

go
// src/testing/benchmark.go:502
func (b *B) Loop() bool {
	// This is written such that the fast path is as fast as possible and can be
	// inlined.
	//
	// There are three cases where we'll fall out of the fast path:
	//
	// - On the first call, both i and n are 0.
	//
	// - If the loop reaches the n'th iteration, then i == n and we need
	//   to figure out the new target iteration count or if we're done.
	//
	// - If the timer is stopped, it poisons the top bit of i so the slow
	//   path can do consistency checks and fail.
	if b.loop.i < b.loop.n {
		b.loop.i++
		return true
	}
	return b.loopSlowPath()
}

A benchmark that uses b.Loop but returns from inside the loop without breaking it correctly is caught:

go
// src/testing/benchmark.go:224
	if b.loop.n > 0 && !b.loop.done && !b.failed {
		b.Error("benchmark function returned without B.Loop() == false (break or return in loop?)")
	}

Allocations

b.ReportAllocs turns on the allocation reporting for one benchmark:

go
// src/testing/benchmark.go:189
// ReportAllocs enables malloc statistics for this benchmark.
// It is equivalent to setting -test.benchmem, but it only affects the
// benchmark function that calls ReportAllocs.
func (b *B) ReportAllocs() {
	b.showAllocResult = true
}

The counting happens in StartTimer and StopTimer, using runtime.MemStats.Mallocs and runtime.MemStats.TotalAlloc, and the results are divided by b.N at the end:

go
// src/testing/benchmark.go:569
// AllocsPerOp returns the "allocs/op" metric,
// which is calculated as r.MemAllocs / r.N.
func (r BenchmarkResult) AllocsPerOp() int64 {
	if v, ok := r.Extra["allocs/op"]; ok {
		return int64(v)
	}
	if r.N <= 0 {
		return 0
	}
	return int64(r.MemAllocs) / int64(r.N)
}

The Extra map check at the top is how ReportMetric lets you override a built-in metric — you can report your own ns/op if you know better.

The final result carries everything:

go
// src/testing/benchmark.go:535
// BenchmarkResult contains the results of a benchmark run.
type BenchmarkResult struct {
	N         int           // The number of iterations.
	T         time.Duration // The total time taken.
	Bytes     int64         // Bytes processed in one iteration.
	MemAllocs uint64        // The total number of memory allocations.
	MemBytes  uint64        // The total number of bytes allocated.

	// Extra records additional metrics reported by ReportMetric.
	Extra map[string]float64
}

SetBytes is what turns on the MB/s column:

go
// src/testing/benchmark.go:185
// SetBytes records the number of bytes processed in a single operation.
// If this is called, the benchmark will report ns/op and MB/s.
func (b *B) SetBytes(n int64) { b.bytes = n }

`testing.AllocsPerRun`

Sometimes you want the allocation count of a single operation, outside a benchmark. That is AllocsPerRun:

go
// src/testing/allocs.go:20
func AllocsPerRun(runs int, f func()) (avg float64) {
	if parallelStart.Load() != parallelStop.Load() {
		panic("testing: AllocsPerRun called during parallel test")
	}
	defer runtime.GOMAXPROCS(runtime.GOMAXPROCS(1))

	// Warm up the function
	f()

	// Measure the starting statistics
	var memstats runtime.MemStats
	runtime.ReadMemStats(&memstats)
	mallocs := 0 - memstats.Mallocs

	// Run the function the specified number of times
	for i := 0; i < runs; i++ {
		f()
	}

	// Read the final statistics
	runtime.ReadMemStats(&memstats)
	mallocs += memstats.Mallocs

	// Average the mallocs over the runs (not counting the warm-up).
	// We are forced to return a float64 because the API is silly, but do
	// the division as integers so we can ask if AllocsPerRun()==1
	// instead of AllocsPerRun()<2.
	return float64(mallocs / uint64(runs))
}

Four deliberate design decisions here.

It warms up first. The first call to f() is outside the measurement window. First-call costs — lazy initialisation, page faults, the allocator’s first span — are not representative.

It pins GOMAXPROCS to 1 for the duration, and restores it afterwards. Allocation counts can depend on how many Ps are running, because work gets stolen.

It calls f through a function value. f is a func() parameter, so the compiler cannot inline it into AllocsPerRun, and the allocations inside it are real allocations. The current implementation does not use a separate //go:noinline sink function; the indirect call is what keeps the measurements honest.

It returns float64 but divides as integers, so you can compare the result with == 1 exactly rather than with a tolerance. The comment calls the API “silly” and then makes the best of it.

Sub-benchmarks, `-count`, and matching

b.Run creates a sub-benchmark. The rule is that a benchmark with sub-benchmarks is not itself measured:

go
// src/testing/benchmark.go:803
// Run benchmarks f as a subbenchmark with the given name. It reports
// whether there were any failures.
//
// A subbenchmark is like any other benchmark. A benchmark that calls Run at
// least once will not be measured itself and will be called once with N=1.
func (b *B) Run(name string, f func(b *B)) bool {

Each sub-benchmark gets its own B, its own ramp-up, and its own result. The parent’s result is then a synthetic aggregate:

go
// src/testing/benchmark.go:874
// add simulates running benchmarks in sequence in a single iteration. It is
// used to give some meaningful results in case func Benchmark is used in
// combination with Run.
func (b *B) add(other BenchmarkResult) {
	r := &b.result
	// The aggregated BenchmarkResults resemble running all subbenchmarks as
	// in sequence in a single benchmark.
	r.N = 1
	r.T += time.Duration(other.NsPerOp())

The names are matched by a regexp matcher that understands / as a path separator, so -bench=X/Y selects only the Y sub-benchmark of X:

go
// src/testing/match.go:55
func newMatcher(matchString func(pat, str string) (bool, error), patterns, name, skips string) *matcher {
	var filter, skip filterMatch
	if patterns == "" {
		filter = simpleMatch{} // always partial true
	} else {
		filter = splitRegexp(patterns)
		if err := filter.verify(name, matchString); err != nil {
			fmt.Fprintf(os.Stderr, "testing: invalid regexp for %s\n", err)
			os.Exit(1)
		}
	}

-count is the flag that makes the harness run the whole thing more than once:

go
// src/testing/testing.go:461
	count = flag.Uint("test.count", 1, "run tests and benchmarks `n` times")

and the benchmark side honours it:

go
// src/testing/benchmark.go:739
// processBench runs bench b for the configured CPU counts and prints the results.
func (s *benchState) processBench(b *B) {
	for i, procs := range cpuList {
		for j := uint(0); j < *count; j++ {
			runtime.GOMAXPROCS(procs)
			benchName := benchmarkName(b.name, procs)

			// If it's chatty, we've already printed this information.
			if b.chatty == nil {
				fmt.Fprintf(b.w, "%-*s\t", s.maxLen, benchName)
			}
			// Recompute the running time for all but the first iteration.
			if i > 0 || j > 0 {
				b = &B{

Note that each -count repetition starts from a fresh B, so the ramp-up runs again and the numbers are independent samples rather than one long accumulation. That is what you want for a statistical comparison: five independent runs, look at the spread, not one run with a huge N.

There is a similar loop for tests themselves:

go
// src/testing/testing.go:2717
func runTests(modulePath, importPath string, matchString func(pat, str string) (bool, error), tests []InternalTest, deadline time.Time) (ran, ok bool) {
	ok = true
	for _, procs := range cpuList {
		runtime.GOMAXPROCS(procs)
		for i := uint(0); i < *count; i++ {

and the same cpuList loop drives go test -cpu=1,4,8.

Parallel benchmarks

RunParallel splits b.N across goroutines:

go
// src/testing/benchmark.go:950
func (b *B) RunParallel(body func(*PB)) {
	if b.N == 0 {
		return // Nothing to do when probing.
	}
	// Calculate grain size as number of iterations that take ~100µs.
	// 100µs is enough to amortize the overhead and provide sufficient
	// dynamic load balancing.
	grain := uint64(0)
	if b.previousN > 0 && b.previousDuration > 0 {
		grain = 1e5 * uint64(b.previousN) / uint64(b.previousDuration)
	}
	if grain < 1 {
		grain = 1
	}
	// We expect the inner loop and function call to take at least 10ns,
	// so do not do more than 100µs/10ns=1e4 iterations.
	if grain > 1e4 {
		grain = 1e4

The grain is a work-stealing quantum: each goroutine grabs grain iterations at a time from a shared atomic counter, so a goroutine that finishes early can take more work. 100 microseconds is the sweet spot — long enough that the atomic counter contention is negligible, short enough that the load stays balanced.

A note on `testlog`

While we are in the testing package: there is a side channel that most people never see. internal/testlog exists so that cmd/go can see which environment variables and files a test consulted, which is what makes test caching correct:

go
// src/internal/testlog/log.go:5
// Package testlog provides a back-channel communication path
// between tests and package os, so that cmd/go can see which
// environment variables and files a test consults.
package testlog

The logger is swapped in atomically, because test startup races with goroutines started during init:

go
// src/internal/testlog/log.go:23
// logger is the current logger Interface.
// We use an atomic.Pointer in case test startup
// is racing with goroutines started during init.
// That must not cause a race detector failure,
// although it will still result in limited visibility
// into exactly what those goroutines do.
var logger atomic.Pointer[Interface]

It is a small thing, but it is a nice example of atomic.Pointer being the right tool: the interface value is published once, at startup, and read concurrently from arbitrary goroutines.

Trade-offs

What the ramp-up buys you. A measurement over a million iterations is stable to a fraction of a percent. A measurement over one iteration is dominated by scheduler noise, cache state and frequency scaling. The proportional controller gets you to the same place in two or three rounds rather than doubling until you get there.

What it costs. Your benchmark function is called several times, with different N. That means any setup inside the function happens several times, and if the setup has side effects (writing a file, appending to a global) you will be surprised. b.Loop() fixes this by running the function once.

Why -benchtime is one second. Long enough to drown out noise, short enough that go test -bench=. over a whole package finishes in a reasonable time. The ramp-up means the time is spent measuring, not waiting: the harness runs iterations back to back until the total is a second.

Why -count exists. Because a single sample of a noisy measurement is not evidence. Five samples and a look at the spread is.

Why the harness forces a runtime.GC() between runs. Comparability. Each run starts from as close to the same heap state as the harness can manage.

Caveats

The timer is on by default. Setup code before your measurement loop is timed unless you explicitly stop the timer. This is the number one cause of meaningless benchmark results.

b.N changes between runs of the same function. Your function is called with several different values of b.N. If you cache something in a package variable keyed on b.N, or if you branch on b.N, your benchmark is measuring something different each time.

b.Loop() requires the exact literal form. The compiler transformation only matches for b.Loop() { ... }. Any other spelling loses the keep-alive protection and the timer management.

b.Loop() and a b.N loop must not be mixed. The harness checks, but only after the fact, and the error message is a b.Error rather than a compile error.

Benchmarks are not tests of correctness. go test -bench does not check that your code returns the right answer. A benchmark that computes garbage faster is still a fast benchmark.

RunParallel reports wall time for the whole benchmark, not the sum of the per-goroutine times, and it explicitly forbids StartTimer, StopTimer, ResetTimer and Run inside the body.

The ramp-up is per benchmark, per -cpu value, per -count repetition. go test -bench=. -cpu=1,4,8 -count=5 runs your benchmark 15 times, each with its own ramp-up. That is a lot of wall-clock time, and it is why people narrow the benchmark selection when they are iterating.

Wrapping up

The one thing to remember: b.N is not given to you, it is decided by the harness at run time by measuring how fast your code turned out to be. Your benchmark must therefore be a function of b.N, and it must do the same work every iteration, or the number you get back is not a per-operation cost.

The part people get wrong: the timer is running when your function is entered. Unless you stop it, your file read, your map construction and your regex compilation are all inside the measurement, and your ns/op is a blend of setup and work.

What to go do about it: rewrite your benchmarks to use for b.Loop() { ... }. It excludes setup and cleanup, runs the function exactly once, and the compiler keeps your loop body alive so the optimiser cannot delete it. Then run with -benchmem -count=5 and compare the spread, not a single number.