Profiling on Opensource internals

How Go's CPU Profiler Works: SIGPROF, the Signal Stack and 100 Hz Sampling

Jay Hubbert — Wed, 21 Oct 2026 04:16:00 +0000

TL;DR: Go’s CPU profiler is an interval timer plus a signal handler. runtime/pprof.StartCPUProfile hardcodes 100 Hz, the runtime arms setitimer(ITIMER_PROF) with that period, and every time the timer fires the kernel raises SIGPROF on some thread. The handler runs on a dedicated 32 KB signal stack, unwinds the interrupted goroutine’s stack in place, and appends the PCs to a lock-free ring buffer. A separate goroutine drains that buffer and writes a protobuf profile.

Profiling a Go program is easy: call pprof.StartCPUProfile(f) and walk away. What that call actually sets up is a POSIX interval timer, a kernel signal, a signal handler running on a stack the runtime reserves for exactly this purpose, a stack unwinder that has to work while the world is half-frozen, and a lock-free ring buffer shared between a signal handler and an ordinary goroutine.

Every one of those pieces is in the Go tree, and every one of them is a little unusual. Let’s walk through them, based on the Go 1.28 development tree.

The 100 Hz decision

The frequency is not configurable through the public API, and the reason is written down in runtime/pprof:

 go
 
 
// src/runtime/pprof/pprof.go:885
func StartCPUProfile(w io.Writer) error {
	// The runtime routines allow a variable profiling rate,
	// but in practice operating systems cannot trigger signals
	// at more than about 500 Hz, and our processing of the
	// signal is not cheap (mostly getting the stack trace).
	// 100 Hz is a reasonable choice: it is frequent enough to
	// produce useful data, rare enough not to bog down the
	// system, and a nice round number to make it easy to
	// convert sample counts to seconds. Instead of requiring
	// each client to specify the frequency, we hard code it.
	const hz = 100

	cpu.Lock()
	defer cpu.Unlock()
	if cpu.done == nil {
		cpu.done = make(chan bool)
	}
	// Double-check.
	if cpu.profiling {
		return fmt.Errorf("cpu profiling already in use")
	}
	cpu.profiling = true
	pprof_setCPUProfileRate(hz)
	go profileWriter(w)
	return nil
}

Two things to notice. First, 100 is hardcoded — runtime.SetCPUProfileRate exists and does let you pick a rate, but runtime/pprof refuses to use it. Second, the comment tells you the real cost model: the expensive part is not the signal, it is the stack unwinding that follows it. That single sentence explains most of the design decisions below.

profileWriter is the other half of the public API, and it is the goroutine that drains the samples and turns them into a profile:

 go
 
 
// src/runtime/pprof/pprof.go:919
func profileWriter(w io.Writer) {
	b := newProfileBuilder(w)
	var err error
	for {
		if runtime.GOOS == "darwin" || runtime.GOOS == "ios" {
			// see runtime_pprof_readProfile
			time.Sleep(100 * time.Millisecond)
		}
		data, tags, eof := readProfile()
		if e := b.addCPUData(data, tags); e != nil && err == nil {
			err = e
		}
		if eof {
			break
		}
	}
	if err != nil {
		// The runtime should never produce an invalid or truncated profile.
		// It drops records that can't fit into its log buffers.
		panic("runtime/pprof: converting profile: " + err.Error())
	}
	b.build()
	cpu.done <- true
}

The signal handler and this goroutine never touch the same data structure directly. They meet in profBuf, a ring buffer designed for exactly one reader and one writer.

Arming the timer

pprof_setCPUProfileRate is a go:linknamed entry point into the runtime, which lands in setCPUProfileRate in src/runtime/cpuprof.go. That function creates the ring buffer, writes one special record into it (the sampling rate), and then calls setcpuprofilerate, which eventually reaches the OS-specific timer setup:

 go
 
 
// src/runtime/signal_unix.go:276
// setProcessCPUProfilerTimer is called when the profiling timer changes.
// It is called with prof.signalLock held. hz is the new timer, and is 0 if
// profiling is being disabled. Enable or disable the signal as
// required for -buildmode=c-archive.
func setProcessCPUProfilerTimer(hz int32) {
	if hz != 0 {
		// Enable the Go signal handler if not enabled.
		if atomic.Cas(&handlingSig[_SIGPROF], 0, 1) {
			h := getsig(_SIGPROF)
			// If no signal handler was installed before, then we record
			// _SIG_IGN here. When we turn off profiling (below) we'll start
			// ignoring SIGPROF signals. We do this, rather than change
			// to SIG_DFL, because there may be a pending SIGPROF
			// signal that has not yet been delivered to some other thread.
			// If we change to SIG_DFL when turning off profiling, the
			// program will crash when that SIGPROF is delivered. We assume
			// that programs that use profiling don't want to crash on a
			// stray SIGPROF. See issue 19320.
			// We do the change here instead of when turning off profiling,
			// because there we may race with a signal handler running
			// concurrently, in particular, sigfwdgo may observe _SIG_DFL and
			// die. See issue 43828.
			if h == _SIG_DFL {
				h = _SIG_IGN
			}
			atomic.Storeuintptr(&fwdSig[_SIGPROF], h)
			setsig(_SIGPROF, abi.FuncPCABIInternal(sighandler))
		}

		var it itimerval
		it.it_interval.tv_sec = 0
		it.it_interval.set_usec(1000000 / hz)
		it.it_value = it.it_interval
		setitimer(_ITIMER_PROF, &it, nil)
	} else {
		setitimer(_ITIMER_PROF, &itimerval{}, nil)

setitimer(ITIMER_PROF) is the crux. ITIMER_PROF counts CPU time consumed by the process (user time plus system time), not wall-clock time. So a program that spends its life blocked on a socket does not accumulate profiling ticks; a program burning CPU in a tight loop does. The timer is process-wide, and when it expires the kernel picks a thread that does not have the signal blocked and raises SIGPROF there.

That “picks a thread” is why the profiler is biased the way it is. The sample is attributed to whatever goroutine happened to be running on the thread that won the race to receive the signal.

The _SIG_IGN dance above is a lovely piece of defensive engineering: turning the timer off does not cancel an already-pending signal, so switching the handler back to SIG_DFL would let that pending signal kill the program.

The signal stack

Signals are delivered on the thread’s normal stack by default, which is a disaster for a profiler: the interrupted goroutine’s stack is already in use, and the handler needs room to walk it. Go gives every M a dedicated signal stack.

 go
 
 
// src/runtime/os_linux.go:385
// Called to initialize a new m (including the bootstrap m).
// Called on the parent thread (main thread in case of bootstrap), can allocate memory.
func mpreinit(mp *m) {
	mp.gsignal = malg(32 * 1024) // Linux wants >= 2K
	mp.gsignal.m = mp
}

And it installs it as the thread’s alternate signal stack, with SA_ONSTACK:

 go
 
 
// src/runtime/os_linux.go:478
func setsig(i uint32, fn uintptr) {
	var sa sigactiont
	sa.sa_flags = _SA_SIGINFO | _SA_ONSTACK | _SA_RESTORER | _SA_RESTART
	sigfillset(&sa.sa_mask)
	// Although Linux manpage says "sa_restorer element is obsolete and
	// should not be used". x86_64 kernel requires it. Only use it on
	// x86. Note that on 386 this is cleared when using the C sigaction
	// function via cgo; see fixSigactionForCgo.
	if GOARCH == "386" || GOARCH == "amd64" {
		sa.sa_restorer = abi.FuncPCABI0(sigreturn__sigaction)
	}
	if fn == abi.FuncPCABIInternal(sighandler) { // abi.FuncPCABIInternal(sighandler) matches the callers in signal_unix.go
		if iscgo {
			fn = abi.FuncPCABI0(cgoSigtramp)
		} else {
			fn = abi.FuncPCABI0(sigtramp)
		}
	}
	sa.sa_handler = fn
	sigaction(i, &sa, nil)
}

Note sigfillset(&sa.sa_mask): while the handler runs, every signal is blocked. That is why the runtime can rely on a single signal at a time — no reentrancy into the profiler from another signal — and it is also why the profiler’s own locking is a simple compare-and-swap rather than a mutex.

Each thread gets its own 32 KB gsignal goroutine stack, set up when the M is initialised:

 go
 
 
// src/runtime/signal_unix.go:1329
func minitSignals() {
	minitSignalStack()
	minitSignalMask()
}

// minitSignalStack is called when initializing a new m to set the
// alternate signal stack. If the alternate signal stack is not set
// for the thread (the normal case) then set the alternate signal
// stack to the gsignal stack. If the alternate signal stack is set
// for the thread (the case when a non-Go thread sets the alternate
// signal stack and then calls a Go function) then set the gsignal
// stack to the alternate signal stack. We also set the alternate
// signal stack to the gsignal stack if cgo is not used (regardless
// of whether it is already set). Record which choice was made in
// newSigstack, so that it can be undone in unminit.
func minitSignalStack() {
	mp := getg().m
	var st stackt
	sigaltstack(nil, &st)
	if st.ss_flags&_SS_DISABLE != 0 || !iscgo {
		signalstack(&mp.gsignal.stack)
		mp.newSigstack = true
	} else {
		setGsignalStack(&st, &mp.goSigStack)
		mp.newSigstack = false
	}
}

The handler: identify the goroutine, unwind the stack

The handler itself is sighandler, which handles every signal Go cares about. SIGPROF gets an early return:

 go
 
 
// src/runtime/signal_unix.go:674
	if sig == _SIGPROF {
		// Some platforms (Linux) have per-thread timers, which we use in
		// combination with the process-wide timer. Avoid double-counting.
		if !delayedSignal && validSIGPROF(mp, c) {
			sigprof(c.sigpc(), c.sigsp(), c.siglr(), gp, mp)
		}
		return
	}

gp is the G that was running when the signal arrived; mp is the M it was running on. That is the “per-thread goroutine identification”: the sample is attributed to mp.curg, and the goroutine’s pprof labels are attached to the sample as a tag pointer.

sigprof is where the stack is unwound:

 go
 
 
// src/runtime/proc.go:5801
// Called if we receive a SIGPROF signal.
// Called by the signal handler, may run during STW.
//
//go:nowritebarrierrec
func sigprof(pc, sp, lr uintptr, gp *g, mp *m) {
	if prof.hz.Load() == 0 {
		return
	}

	// If mp.profilehz is 0, then profiling is not enabled for this thread.
	// We must check this to avoid a deadlock between setcpuprofilerate
	// and the call to cpuprof.add, below.
	if mp != nil && mp.profilehz == 0 {
		return
	}
	...
	// Profiling runs concurrently with GC, so it must not allocate.
	// Set a trap in case the code does allocate.
	// Note that on windows, one thread takes profiles of all the
	// other threads, so mp is usually not getg().m.
	// In fact mp may not even be stopped.
	// See golang.org/issue/17165.
	getg().m.mallocing++

	var u unwinder
	var stk [maxCPUProfStack]uintptr
	n := 0
	...
	} else {
		u.initAt(pc, sp, lr, gp, unwindSilentErrors|unwindTrap|unwindJumpStack)
	}
	n += tracebackPCs(&u, 0, stk[n:])

Here is a genuinely surprising fact about modern Go: gentraceback is gone. It used to be the workhorse, and every blog post about Go stack unwinding still describes it. In the Go 1.28 tree there is no gentraceback function at all. It has been replaced by the unwinder type, an explicit iterator:

 go
 
 
// src/runtime/traceback.go:84
// An unwinder iterates the physical stack frames of a Go sack.
//
// Typical use of an unwinder looks like:
//
//	var u unwinder
//	for u.init(gp, 0); u.valid(); u.next() {
//		// ... use frame info in u ...
//	}
//
// Implementation note: This is carefully structured to be pointer-free because
// tracebacks happen in places that disallow write barriers (e.g., signals).
// Even if this is stack-allocated, its pointer-receiver methods don't know that
// their receiver is on the stack, so they still emit write barriers. Here we
// address that by carefully avoiding any pointers in this type. Another
// approach would be to split this into a mutable part that's passed by pointer
// but contains no pointers itself and an immutable part that's passed and
// returned by value and can contain pointers. We could potentially hide that
// we're doing that in trivial methods that are inlined into the caller that has
// the stack allocation, but that's fragile.
type unwinder struct {
	// frame is the current physical stack frame, or all 0s if
	// there is no frame.
	frame stkframe

The pointer-free design is not cosmetic. The signal handler runs with write barriers disabled (the GC may be in the middle of a stop-the-world), and the unwinder is deliberately built so it cannot allocate or trigger a barrier.

The result of the unwind is a flat array of return PCs, up to 64 frames:

 go
 
 
// src/runtime/traceback.go:645
func tracebackPCs(u *unwinder, skip int, pcBuf []uintptr) int {
	var cgoBuf [32]uintptr
	n := 0
	for ; n < len(pcBuf) && u.valid(); u.next() {
		f := u.frame.fn
		cgoN := u.cgoCallers(cgoBuf[:])

		// TODO: Why does &u.cache cause u to escape? (Same in traceback2)
		for iu, uf := newInlineUnwinder(f, u.symPC()); n < len(pcBuf) && uf.valid(); uf = iu.next(uf) {
			sf := iu.srcFunc(uf)
			if sf.funcID == abi.FuncIDWrapper && elideWrapperCalling(u.calleeFuncID) {
				// ignore wrappers
			} else if skip > 0 {
				skip--
			} else {
				// Callers expect the pc buffer to contain return addresses
				// and do the -1 themselves, so we add 1 to the call pc to
				// create a "return pc". Since there is no actual call, here
				// "return pc" just means a pc you subtract 1 from to get
				// the pc of the "call". The actual no-op we insert may or
				// may not be 1 byte.
				pcBuf[n] = uf.pc + 1
				n++
			}
			u.calleeFuncID = sf.funcID
		}

If the unwind fails — the sample landed in a JIT blob, or in C code with no Go frames — the profiler falls back to a synthetic two-frame stack of System or GC plus _ExternalCode. Those fake frames show up in go tool pprof output as runtime._System and runtime._ExternalCode, and they are a reliable signal that the time was spent outside Go.

Writing to the ring buffer

The sample is handed to cpuprof.add, which does the least it can get away with:

 go
 
 
// src/runtime/cpuprof.go:108
// add adds the stack trace to the profile.
// It is called from signal handlers and other limited environments
// and cannot allocate memory or acquire locks that might be
// held at the time of the signal, nor can it use substantial amounts
// of stack.
//
//go:nowritebarrierrec
func (p *cpuProfile) add(tagPtr *unsafe.Pointer, stk []uintptr) {
	// Simple cas-lock to coordinate with setcpuprofilerate.
	for !prof.signalLock.CompareAndSwap(0, 1) {
		// TODO: Is it safe to osyield here? https://go.dev/issue/52672
		osyield()
	}

	if prof.hz.Load() != 0 { // implies cpuprof.log != nil
		if p.numExtra > 0 || p.lostExtra > 0 || p.lostAtomic > 0 {
			p.addExtra()
		}
		hdr := [1]uint64{1}
		// Note: write "knows" that the argument is &gp.labels,
		// because otherwise its write barrier behavior may not
		// be correct. See the long comment there before
		// changing the argument here.
		cpuprof.log.write(tagPtr, nanotime(), hdr[:], stk)
	}

	prof.signalLock.Store(0)
}

No allocation, no mutex — a compare-and-swap on prof.signalLock and a copy into a preallocated ring. The ring is profBuf, and its record layout is deliberately trivial:

 go
 
 
// src/runtime/profbuf.go:358
	// Main record.
	// It has to fit in a contiguous section of the slice, so if it doesn't fit at the end,
	// leave a rewind marker (0) and start over at the beginning of the slice.
	wd := int(bw.dataCount() % uint32(len(b.data)))
	nd := countSub(br.dataCount(), bw.dataCount()) + len(b.data)
	skip := 0
	if wd+2+int(b.hdrsize)+len(stk) > len(b.data) {
		b.data[wd] = 0
		skip = len(b.data) - wd
		nd -= skip
		wd = 0
	}
	data := b.data[wd:]
	data[0] = uint64(2 + b.hdrsize + uintptr(len(stk))) // length
	data[1] = uint64(now) // time stamp
	// header, zero-padded
	i := copy(data[2:2+b.hdrsize], hdr)
	clear(data[2+i : 2+b.hdrsize])
	for i, pc := range stk {
		data[2+b.hdrsize+uintptr(i)] = uint64(pc)
	}

Each record is: a length word, a timestamp, a one-word header (the sample count), and the PCs. The buffer is sized in the source with a very human comment:

 go
 
 
// src/runtime/cpuprof.go:21
const (
	maxCPUProfStack = 64

	// profBufWordCount is the size of the CPU profile buffer's storage for the
	// header and stack of each sample, measured in 64-bit words. Every sample
	// has a required header of two words. With a small additional header (a
	// word or two) and stacks at the profiler's maximum length of 64 frames,
	// that capacity can support 1900 samples or 19 thread-seconds at a 100 Hz
	// sample rate, at a cost of 1 MiB.
	profBufWordCount = 1 << 17

19 thread-seconds of samples in a megabyte. If the reader goroutine falls behind and the ring fills up, samples are dropped and the drop count is recorded as a special “overflow” record rather than silently lost.

The three-word header

The very first record the runtime writes is not a sample — it is the sampling rate, and the reader uses it to turn sample counts into nanoseconds:

 go
 
 
// src/runtime/cpuprof.go:94
		cpuprof.on = true
		cpuprof.log = newProfBuf(1, profBufWordCount, profBufTagCount)
		hdr := [1]uint64{uint64(hz)}
		cpuprof.log.write(nil, nanotime(), hdr[:], nil)
		setcpuprofilerate(int32(hz))

That record has a zero-length stack, so it occupies three words: length (3), timestamp, and the frequency. The reader checks for exactly that shape:

 go
 
 
// src/runtime/pprof/proto.go:278
func (b *profileBuilder) addCPUData(data []uint64, tags []unsafe.Pointer) error {
	if !b.havePeriod {
		// first record is period
		if len(data) < 3 {
			return fmt.Errorf("truncated profile")
		}
		if data[0] != 3 || data[2] == 0 {
			return fmt.Errorf("malformed profile")
		}
		// data[2] is sampling rate in Hz. Convert to sampling
		// period in nanoseconds.
		b.period = 1e9 / int64(data[2])
		b.havePeriod = true
		data = data[3:]
		// Consume tag slot. Note that there isn't a meaningful tag
		// value for this record.
		tags = tags[1:]
	}

The same routine also decodes the overflow records, which is how a dropped sample shows up in the profile rather than vanishing:

 go
 
 
// src/runtime/pprof/proto.go:322
		count := data[2]
		stk := data[3:data[0]]
		data = data[data[0]:]
		tag := tags[0]
		tags = tags[1:]

		if count == 0 && len(stk) == 1 {
			// overflow record
			count = uint64(stk[0])
			stk = []uint64{
				// gentraceback guarantees that PCs in the
				// stack can be unconditionally decremented and
				// still be valid, so we must do the same.
				uint64(abi.FuncPCABIInternal(lostProfileEvent) + 1),
			}
		}
		b.m.lookup(stk, tag).count += int64(count)

(The comment mentioning gentraceback is a leftover from before the unwinder rewrite — the property it describes still holds.)

The builder then deduplicates identical stacks on the fly and finally emits the protobuf with two sample types, samples/count and cpu/nanoseconds.

The memory profile is a different animal

The memory profile does not use signals at all. It samples allocations, at a rate set by a global:

 go
 
 
// src/runtime/mprof.go:856
// MemProfileRate controls the fraction of memory allocations
// that are recorded and reported in the memory profile.
// The profiler aims to sample an average of
// one allocation per MemProfileRate bytes allocated.
//
// To include every allocated block in the profile, set MemProfileRate to 1.
// To turn off profiling entirely, set MemProfileRate to 0.
//
// The tools that process the memory profiles assume that the
// profile rate is constant across the lifetime of the program
// and equal to the current value. Programs that change the
// memory profiling rate should do so just once, as early as
// possible in the execution of the program (for example,
// at the beginning of main).
var MemProfileRate int = 512 * 1024

512 KB, by default. The decision of which allocation to sample is made with a counter kept in the mcache — the per-P allocation cache — and the next sample point is drawn from an exponential distribution so that the sampling is Poissonian rather than periodic:

 go
 
 
// src/runtime/malloc.go:2236
// nextSample returns the next sampling point for heap profiling. The goal is
// to sample allocations on average every MemProfileRate bytes, but with a
// completely random distribution over the allocation timeline; this
// corresponds to a Poisson process with parameter MemProfileRate. In Poisson
// processes, the distance between two samples follows the exponential
// distribution (exp(MemProfileRate)), so the best return value is a random
// number taken from an exponential distribution whose mean is MemProfileRate.
func nextSample() int64 {
	if MemProfileRate == 0 {
		// Basically never sample.
		return math.MaxInt64
	}
	if MemProfileRate == 1 {
		// Sample immediately.
		return 0
	}
	return int64(fastexprand(MemProfileRate))
}

The counter lives on the mcache, and the check happens on every allocation:

 go
 
 
// src/runtime/malloc.go:1449
	c.nextSample -= int64(size)
	if c.nextSample < 0 || MemProfileRate != c.memProfRate {
		profilealloc(mp, x, size)
	}

So: the CPU profile samples time using a signal; the memory profile samples bytes using a counter in the allocator cache. Different mechanisms, different biases. The memory profile counts the sampled allocation as if it represented MemProfileRate bytes, which is why pprof scales it up — a 4 KB allocation sampled at 512 KB per sample is reported as 512 KB.

Why CPU-bound goroutines dominate

Because ITIMER_PROF counts CPU time, a goroutine only gets sampled when it is actually on the CPU. A goroutine blocked on a channel, sleeping on a timer, or waiting on a mutex is invisible to the CPU profiler no matter how important it is. The profile answers “where did the CPU time go”, not “where did the program spend its time”.

That is the correct answer for a CPU profiler, but it means a program dominated by I/O will produce a profile full of runtime._System and runtime._ExternalCode frames and nothing else. For “where is my time going”, you want the execution tracer or the block profile instead.

Two more biases worth knowing:

Short goroutines are undersampled. At 100 Hz, a goroutine needs to burn roughly 10 ms of CPU to have a decent chance of appearing at all.
The signal lands on an arbitrary thread. The kernel picks a thread that does not block SIGPROF, so the sample is a snapshot of that thread’s current goroutine. This is unbiased in aggregate, but it means individual samples are attributed to a goroutine that was merely running, not necessarily the one that consumed the CPU time.

Trade-offs

What sampling buys you: near-zero steady-state overhead. The timer fires 100 times a second, each firing costs one signal delivery plus one stack unwind, and nothing else in the program changes.

What it costs:

Statistical error. A short profile is a small sample. 100 samples is 1 second of CPU time; the standard error on a 1% line is roughly 10% relative.
The unwinder is the expensive part. The runtime comment says so outright: “our processing of the signal is not cheap (mostly getting the stack trace)”.
Signal-handler constraints. The handler cannot allocate, cannot take a mutex, and cannot grow the stack. The whole unwinder design is shaped by that.
Dropped samples. If the reader goroutine cannot keep up, samples are dropped and reported as an overflow record rather than blocking the signal handler.
Frames are lost at 64. maxCPUProfStack is 64; deeper stacks are truncated, and the truncated frame is a plain function name.

Caveats

SIGPROF is a normal signal. If your program calls os/signal.Notify(c, syscall.SIGPROF), it will fight the profiler for the signal. On Unix, sigenable explicitly refuses to touch SIGPROF:

 go
 
 
// src/runtime/signal_unix.go:195
	// SIGPROF is handled specially for profiling.
	if sig == _SIGPROF {
		return
	}

c-archive and c-shared builds are different. StartCPUProfile documents that it does not work by default there, because SIGPROF goes to the host program’s handler rather than Go’s.

Linux has per-thread timers. The kernel’s timer_create timers show up as _SI_TIMER signal codes while setitimer produces _SI_KERNEL, and validSIGPROF uses that to avoid counting the same CPU time twice.

Darwin cannot block in the reader. runtime_pprof_readProfile uses a non-blocking read on Darwin and iOS, and profileWriter sleeps 100 ms between reads, because notetsleepg is not safe in a signal handler there.

Profiles are not comparable across runs unless the rate is the same — which it is, since 100 Hz is hardcoded — but they are not comparable if the program was interrupted differently or if samples were dropped.

Wrapping up

The one thing to remember: the CPU profiler is a 100 Hz setitimer(ITIMER_PROF) plus a SIGPROF handler that runs on a per-M 32 KB signal stack, unwinds the interrupted goroutine with a pointer-free unwinder, and copies the PCs into a lock-free ring buffer that a separate goroutine turns into a profile.

The part people get wrong: ITIMER_PROF counts CPU time, so the profile shows you where the CPU went, not where the wall-clock went. A program that is mostly blocked will show almost nothing, and the samples it does show will be attributed to runtime._System.

What to go do about it: turn on a CPU profile and an execution trace at the same time. The profile tells you which function is hot; the trace tells you why the goroutine that called it was asleep. And if the profile is suspiciously empty, check for a SIGPROF handler installed by a library before you go looking for a performance bug.

Go's Execution Tracer: A Ring Buffer of Events Per Goroutine

Jay Hubbert — Wed, 21 Oct 2026 04:16:00 +0000

TL;DR: A CPU profile is a sample of where the CPU was. The execution tracer is a log of what happened: every goroutine creation, every scheduler switch, every channel block, every syscall and every GC phase is recorded as a typed event with a timestamp and a stack. The runtime batches those events into per-M buffers, deduplicates stacks and strings through lock-free hash tries, and hands them to a reader goroutine that writes a versioned trace file that go tool trace renders as a timeline.

If you have ever run go tool trace and watched a waterfall appear, you have seen something quite different from a CPU profile. The profile is a statistical sample; the trace is a history. Everything that happened is in there, in order, with timestamps.

The machinery behind that history is a ring of typed events, and this post is about how it works, based on the Go 1.28 development tree.

What makes it different from pprof

	CPU profile	Execution trace
Trigger	`SIGPROF` at 100 Hz	every interesting runtime event
Granularity	one stack per sample	every transition
Data	PCs per sample	typed events with args
Cost	~0 (statistical)	a few percent (real work)
Question answered	“what is hot?”	“what happened, and why?”

The key word in that last row is why. A profile can tell you that 40% of your CPU went into regexp.MatchString. It cannot tell you that most of that time was spent in one goroutine that was blocked 90% of the way through. Only a timeline can.

Starting a trace

The public API is two functions:

 go
 
 
// src/runtime/trace/trace.go:115
// Start enables tracing for the current program.
// While tracing, the trace will be buffered and written to w.
// Start returns an error if tracing is already enabled.
func Start(w io.Writer) error {
	return tracing.subscribeTraceStartWriter(w)
}

// Stop stops the current tracing, if any.
// Stop only returns after all the writes for the trace have completed.
func Stop() {
	tracing.unsubscribeTraceStartWriter()
}

In Go 1.28 the runtime/trace package is a multiplexer: a traceStartWriter (the file you passed to Start) and a FlightRecorder (a rolling in-memory recorder) can both be attached to the same trace. The package keeps a subscriber count, and the underlying runtime trace is started when the first subscriber arrives and stopped when the last one leaves.

The interesting part is the goroutine that actually drains the trace:

 go
 
 
// src/runtime/trace/subscribe.go:157
	go func() {
		header := runtime.ReadTrace()
		if traceStartWriter != nil {
			traceStartWriter.Write(header)
		}
		if flightRecorder != nil {
			flightRecorder.Write(header)
		}

		for {
			data := runtime.ReadTrace()
			if data == nil {
				break
			}
			if traceStartWriter != nil {
				traceStartWriter.Write(data)
			}
			if flightRecorder != nil {
				flightRecorder.Write(data)
			}
			if len(data) == 1 && tracev2.EventType(data[0]) == tracev2.EvEndOfGeneration {
				if flightRecorder != nil {
					flightRecorder.endGeneration()
				}

The runtime produces a byte stream; this goroutine copies it to the writer(s). Everything the tracer does is designed so that this loop never blocks the program: the runtime hands off full buffers and the reader picks them up.

The overall shape

 text
 
 
 your goroutines
 |
 | traceAcquire() / traceRelease()
 v
 +---------------------------------------------+
 | per-M trace buffers (traceBuf) | <- written by the M,
 | mp.trace.buf[gen%2][experiment] | no locks, no allocation
 +---------------------------------------------+
 | full buffer -> trace.full[gen%2]
 v
 +---------------------------------------------+
 | trace.full[gen%2] (queue of full buffers) |
 +---------------------------------------------+
 | ReadTrace()
 v
 +---------------------------------------------+
 | reader goroutine -> your io.Writer |
 +---------------------------------------------+

The runtime keeps a global trace state, and the buffer management is the first thing in it:

 go
 
 
// src/runtime/trace.go:198
	// Trace buffer management.
	//
	// First we check the empty list for any free buffers. If not, buffers
	// are allocated directly from the OS. Once they're filled up and/or
	// flushed, they end up on the full queue for trace.gen%2.
	//
	// The trace reader takes buffers off the full list one-by-one and
	// places them into reading until they're finished being read from.
	// Then they're placed onto the empty list.
	//
	// Protected by trace.lock.
	reading *traceBuf // buffer currently handed off to user
	empty *traceBuf // stack of empty buffers
	full [2]traceBufQueue
	workAvailable atomic.Bool

gen%2 is the trick that keeps this cheap. There are two sets of buffers, one for the current generation and one for the next, so the reader can be draining generation N while the program is already writing generation N+1. The generations are the tracer’s unit of “everything that happened between two stop-the-worlds”.

Per-M buffers, not per-goroutine buffers

An older description of the tracer — one you will still find in blog posts — says the runtime keeps a buffer of events per goroutine. That was true of the version 1 trace format. The modern tracer, the one behind the version 2 format, writes into per-M buffers:

 go
 
 
// src/runtime/traceruntime.go:26
// mTraceState is per-M state for the tracer.
type mTraceState struct {
	writing atomic.Bool // flag indicating that this M is writing to a trace buffer.
	buf [2][tracev2.NumExperiments]*traceBuf // Per-M traceBuf for writing. Indexed by trace.gen%2.
	link *m // Snapshot of alllink or freelink.
	reentered uint32 // Whether we've reentered tracing from within tracing.
	entryGen uintptr // The generation value on first entry.
	oldthrowsplit bool // gp.throwsplit upon calling traceLocker.writer. For debugging.
}

That is a better design than per-goroutine buffers: an M is a single writer, so the buffer needs no synchronisation at all. The per-goroutine view is reconstructed by the reader from the event stream — GoStart, GoStop, GoBlock and GoUnblock events carry the goroutine ID and a sequence number that lets the reader order events that were produced concurrently.

The per-goroutine bookkeeping that remains in the runtime is small:

 go
 
 
// src/runtime/tracestatus.go:146
type traceSchedResourceState struct {
	// statusTraced indicates whether a status event was traced for this resource
	// a particular generation.
	//
	// There are 3 of these because when transitioning across generations, traceAdvance
	// needs to be able to reliably observe whether a status was traced for the previous
	// generation, while we need to clear the value for the next generation.
	statusTraced [3]atomic.Uint32

	// seq is the sequence counter for this scheduling resource's events.
	// The purpose of the sequence counter is to establish a partial order between
	// events that don't obviously happen serially (same M) in the stream ofevents.
	//
	// There are two of these so that we can reset the counter on each generation.
	// This saves space in the resulting trace by keeping the counter small and allows
	// GoStatus and GoCreate events to omit a sequence number (implicitly 0).
	seq [2]uint64
}

The seq counter is the clever bit: two Ms can emit events about the same goroutine concurrently, and the reader cannot tell which came first from the timestamps alone. The sequence number gives it a tiebreaker.

Writing an event

Events are written through a traceWriter, a value type whose methods return the writer so calls can be chained:

 go
 
 
// src/runtime/tracebuf.go:68
// event writes out the bytes of an event into the event stream.
//
// nosplit because it's part of writing an event for an M, which must not
// have any stack growth.
//
//go:nosplit
func (w traceWriter) event(ev tracev2.EventType, args ...traceArg) traceWriter {
	// N.B. Everything in this call must be nosplit to maintain
	// the stack growth related invariants for writing events.

	// Make sure we have room.
	w, _ = w.ensure(1 + (len(args)+1)*traceBytesPerNumber)

	// Compute the timestamp diff that we'll put in the trace.
	ts := traceClockNow()
	if ts <= w.traceBuf.lastTime {
		ts = w.traceBuf.lastTime + 1
	}
	tsDiff := uint64(ts - w.traceBuf.lastTime)
	w.traceBuf.lastTime = ts

	// Write out event.
	w.byte(byte(ev))
	w.varint(tsDiff)
	for _, arg := range args {
		w.varint(uint64(arg))
	}
	return w
}

Three things to notice:

Timestamps are deltas, not absolute. The first event in a buffer carries its absolute timestamp; every subsequent one carries the difference. The delta is at most a nanosecond or two, so a varint encodes it in one byte.
Timestamps are forced to be strictly increasing. if ts <= lastTime { ts = lastTime + 1 } — the clock is not guaranteed to be monotonic at that granularity, and a trace with equal timestamps would be ambiguous.
nosplit. The whole path is marked nosplit because writing an event happens on the syscall slow path, where the stack must not move.

The buffer itself is a fixed-size byte array with a header:

 go
 
 
// src/runtime/tracebuf.go:258
// traceBufHeader is per-P tracing buffer.
type traceBufHeader struct {
	link *traceBuf // in trace.empty/full
	lastTime traceTime // when we wrote the last event
	pos int // next write offset in arr
	lenPos int // position of batch length value
}

// traceBuf is per-M tracing buffer.
//
// TODO(mknyszek): Rename traceBuf to traceBatch, since they map 1:1 with event batches.
type traceBuf struct {
	_ sys.NotInHeap
	traceBufHeader
	arr [tracev2.MaxBatchSize - unsafe.Sizeof(traceBufHeader{})]byte // underlying buffer for traceBufHeader.buf
}

MaxBatchSize is 64 KiB. When the buffer runs out, refill flushes it to the full queue and grabs a fresh one:

 go
 
 
// src/runtime/tracebuf.go:189
	// Write the buffer's header.
	if w.exp == tracev2.NoExperiment {
		w.byte(byte(tracev2.EvEventBatch))
	} else {
		w.byte(byte(tracev2.EvExperimentalBatch))
		w.byte(byte(w.exp))
	}
	w.varint(uint64(w.gen))
	w.varint(mID)
	w.varint(uint64(ts))
	w.traceBuf.lenPos = w.varintReserve()

The lenPos slot is reserved now and filled in at flush time with the length of the batch — the classic “reserve space, patch it later” trick, because the length is not known until the batch is complete.

The traceLocker and the reentrancy guard

Before writing anything, an M must prove that tracing is on and claim the right to write:

 go
 
 
// src/runtime/traceruntime.go:195
// traceAcquireEnabled is the traceEnabled path for traceAcquire. It's explicitly
// broken out to make traceAcquire inlineable to keep the overhead of the tracer
// when it's disabled low.
//
// nosplit because it's called by traceAcquire, which is nosplit.
//
//go:nosplit
func traceAcquireEnabled() traceLocker {
	// Any time we acquire a traceLocker, we may flush a trace buffer. But
	// buffer flushes are rare. Record the lock edge even if it doesn't happen
	// this time.
	lockRankMayTraceFlush()

	// Prevent preemption.
	mp := acquirem()

	// Check if we're already tracing. It's safe to be reentrant in general,
	// because this function (and the invariants of traceLocker.writer) ensure
	// that it is.
	if mp.trace.writing.Load() {
		mp.trace.reentered++
		return traceLocker{mp, mp.trace.entryGen}
	}

The writing flag is the guard against reentrancy: writing an event can trigger a stack growth, which can trigger a GC, which can emit trace events, which would try to write again. The reentrant call notices the flag and returns a locker bound to the same generation without touching the buffer.

There is also an important subtlety in trace.enabled versus trace.gen:

 go
 
 
// src/runtime/trace.go:264
	// enabled indicates whether tracing is enabled, but it is only an optimization,
	// NOT the source of truth on whether tracing is enabled. Tracing is only truly
	// enabled if gen != 0. This is used as an optimistic fast path check.
	//
	// Transitioning this value from true -> false is easy (once gen is 0)
	// because it's OK for enabled to have a stale "true" value. traceAcquire will
	// always double-check gen.

trace.enabled is a plain bool read without synchronisation — a data race on paper, made safe by the world-stopping barrier used when the trace starts and stops.

The events themselves

The event type list is the vocabulary of the tracer, and it is short enough to read in full:

 go
 
 
// src/internal/trace/tracev2/events.go:16
const (
	EvNone EventType = iota // unused

	// Structural events.
	EvEventBatch // start of per-M batch of events [generation, M ID, timestamp, batch length]
	EvStacks // start of a section of the stack table [...EvStack]
	EvStack // stack table entry [ID, ...{PC, func string ID, file string ID, line #}]
	EvStrings // start of a section of the string dictionary [...EvString]
	EvString // string dictionary entry [ID, length, string]
	EvCPUSamples // start of a section of CPU samples [...EvCPUSample]
	EvCPUSample // CPU profiling sample [timestamp, M ID, P ID, goroutine ID, stack ID]
	EvFrequency // timestamp units per sec [freq]

	// Procs.
	EvProcsChange // current value of GOMAXPROCS [timestamp, GOMAXPROCS, stack ID]
	EvProcStart // start of P [timestamp, P ID, P seq]
	EvProcStop // stop of P [timestamp]
	EvProcSteal // P was stolen [timestamp, P ID, P seq, M ID]
	EvProcStatus // P status at the start of a generation [timestamp, P ID, status]

	// Goroutines.
	EvGoCreate // goroutine creation [timestamp, new goroutine ID, new stack ID, stack ID]
	EvGoCreateSyscall // goroutine appears in syscall (cgo callback) [timestamp, new goroutine ID]
	EvGoStart // goroutine starts running [timestamp, goroutine ID, goroutine seq]
	EvGoDestroy // goroutine ends [timestamp]
	EvGoDestroySyscall // goroutine ends in syscall (cgo callback) [timestamp]
	EvGoStop // goroutine yields its time, but is runnable [timestamp, reason, stack ID]
	EvGoBlock // goroutine blocks [timestamp, reason, stack ID]
	EvGoUnblock // goroutine is unblocked [timestamp, goroutine ID, goroutine seq, stack ID]
	EvGoSyscallBegin // syscall enter [timestamp, P seq, stack ID]
	EvGoSyscallEnd // syscall exit [timestamp]
	EvGoSyscallEndBlocked // syscall exit and it blocked at some point [timestamp]
	EvGoStatus // goroutine status at the start of a generation [timestamp, goroutine ID, M ID, status]

Everything after that is GC, STW, annotations, coroutine switches and the experimental allocation events. Every event is a one-byte type followed by varint-encoded arguments, and every “stack ID” argument is an index into the trace’s stack table.

Emitting one looks like this:

 go
 
 
// src/runtime/traceruntime.go:423
// GoStart emits a GoStart event.
//
// Must be called with a valid P.
func (tl traceLocker) GoStart() {
	gp := getg().m.curg
	pp := gp.m.p
	w := tl.eventWriter(tracev2.GoRunnable, tracev2.ProcRunning)
	w.event(tracev2.EvGoStart, traceArg(gp.goid), gp.trace.nextSeq(tl.gen))
	if pp.ptr().gcMarkWorkerMode != gcMarkWorkerNotWorker {
		w.event(tracev2.EvGoLabel, trace.markWorkerLabels[tl.gen%2][pp.ptr().gcMarkWorkerMode])
	}
}

Note eventWriter rather than rawEventWriter: before the event is written, the tracer emits status events for the G and the P if their status has not already been reported for this generation. That is how the trace reader knows that goroutine 42 was “running” or “waiting” at the start of a generation without having to infer it.

Deduplication: strings, stacks and types

A busy program produces the same stack thousands of times. Writing them all would be absurd, so the tracer keeps three tables — one for strings, one for stacks, one for types — and refers to them by ID.

The backing store is a lock-free, append-only 4-ary hash trie:

 go
 
 
// src/runtime/tracemap.go:26
// traceMap is a map of a variable-sized array of bytes to a unique ID.
//
// Because traceMap just operates on raw bytes, this type is used as the
// backing store for both the trace string table and trace stack table,
// the latter of which is just an array of PCs.
//
// ID 0 is reserved for arrays of bytes of size zero.
type traceMap struct {
	root atomic.UnsafePointer // *traceMapNode (can't use generics because it's notinheap)
	_ cpu.CacheLinePad
	seq atomic.Uint64
	_ cpu.CacheLinePad
	mem traceRegionAlloc
}

and the memory for the table’s contents comes from a bump allocator that lives outside the Go heap:

 go
 
 
// src/runtime/traceregion.go:15
// traceRegionAlloc is a thread-safe region allocator.
// It holds a linked list of traceRegionAllocBlock.
type traceRegionAlloc struct {
	lock mutex
	dropping atomic.Bool // For checking invariants.
	current atomic.UnsafePointer // *traceRegionAllocBlock
	full *traceRegionAllocBlock
}

// traceRegionAllocBlock is a block in traceRegionAlloc.
//
// traceRegionAllocBlock is allocated from non-GC'd memory, so it must not
// contain heap pointers. Writes to pointers to traceRegionAllocBlocks do
// not need write barriers.
type traceRegionAllocBlock struct {
	_ sys.NotInHeap
	traceRegionAllocBlockHeader
	data [traceRegionAllocBlockData]byte
}

Off-heap, not-in-heap memory, no write barriers: the same constraints the signal handler imposes on the CPU profiler, applied here because the tracer must be safe to call from GC-adjacent code.

Generations and the file format

A “generation” is one contiguous stretch of trace data. The tracer starts a new generation when it advances (which happens on Stop, and on subscriber changes), and the file format records which version it is:

 go
 
 
// src/internal/trace/version/version.go:17
const (
	Go111 Version = 11 // v1
	Go119 Version = 19 // v1
	Go121 Version = 21 // v1
	Go122 Version = 22 // v2
	Go123 Version = 23 // v2
	Go125 Version = 25 // v2
	Go126 Version = 26 // v2
	Current = Go126
)

var versions = map[Version][]tracev2.EventSpec{
	// Go 1.11–1.21 use a different parser and are only set here for the sake of
	// Version.Valid.
	Go111: nil,
	Go119: nil,
	Go121: nil,

	Go122: tracev2.Specs()[:tracev2.EvUserLog+1], // All events after are Go 1.23+.
	Go123: tracev2.Specs()[:tracev2.EvExperimentalBatch+1], // All events after are Go 1.25+.
	Go125: tracev2.Specs()[:tracev2.EvClockSnapshot+1], // All events after are Go 1.26+.
	Go126: tracev2.Specs(),
}

That table is the whole compatibility story: each trace version knows which event types exist in it, and events added in later Go versions are simply truncated off the spec list for older versions. The version lives in a sixteen-byte header at the front of the file:

 go
 
 
// src/internal/trace/version/version.go:67
// headerFmt is the format of the header of all Go execution traces.
const headerFmt = "go 1.%d trace\x00\x00\x00"

go tool trace reads that header, picks the matching spec table, and replays the event stream. The tracer is also reachable from outside the standard library: the runtime notes that traceAdvance is called both by runtime/trace and by golang.org/x/exp/trace using go:linkname (src/runtime/trace.go:467), which is how third-party trace consumers drive the same machinery. go tool trace is the supported front end.

What it costs

Tracing is not free, and the cost is not in one place. It is spread across:

Every scheduling transition now emits an event: goroutine start, stop, block, unblock, syscall enter and exit.
Every event is a type byte plus varint-encoded arguments.
Every stack attached to an event has to be walked and hashed.
Start and stop of the trace require stopping the world.

The number people quote is “5–10% overhead”, and that is a rule of thumb rather than a guarantee: a program with a lot of goroutine churn pays more, and a program that spends its life in one tight loop pays almost nothing, because there are no transitions to record. The honest way to find out is to measure your program with and without.

Two knobs make it cheaper. GODEBUG=traceallocfree=1 is off by default because it is expensive; and the flight recorder keeps only the recent generations, so a long-running program does not accumulate an unbounded file.

Trade-offs

What the trace buys you: causality. You can see that goroutine 42 blocked on a channel receive at t=1.2ms, that goroutine 17 sent on it at t=1.1ms, that goroutine 17 was blocked on a mutex held by goroutine 3, and that goroutine 3 was in a GC assist. No amount of CPU profiling gives you that.

What it costs:

Volume. Every transition is recorded. A busy scheduler produces enormous traces.
Storage. Trace files are routinely hundreds of megabytes.
Overhead. A few percent, more if the program churns goroutines.
A non-portable format. The wire format is internal to the Go project and changes between versions, which is why go tool trace from a newer toolchain may refuse an older file.

What you give up by not tracing: the ability to answer “why”. A profile tells you where the CPU is; only a trace tells you why a goroutine was not using it.

Caveats

Traces are big and the UI will melt. Use the go tool trace -pprof=sched flag to extract a scheduler profile from a trace rather than staring at millions of rows in a browser.

The trace contains your data. Log messages and region names are recorded verbatim. If you put a token in a trace.Log, it is in the file.

Stack traces are deduplicated, not free. The first occurrence of a stack carries the cost of walking it; later occurrences cost a hash lookup.

SIGPROF interference. If you run a CPU profile and an execution trace at the same time, the tracer piggybacks the CPU samples onto the trace stream (EvCPUSample), but the two mechanisms still both pay for their own signal delivery.

The world stops at the start and end of the trace. On a program with microsecond latency requirements, that pause is visible.

Version skew. A trace written by Go 1.26 contains event types that Go 1.22’s reader does not know about. The spec tables handle that by truncation, but a very old reader of a very new trace will still lose the newest events.

Wrapping up

The one thing to remember: the execution tracer records events, not samples. Goroutine transitions, syscalls, GC phases and user annotations all become typed, timestamped, varint-encoded entries in per-M ring buffers that are deduplicated through lock-free hash tries and streamed to a reader goroutine.

The part people get wrong: assuming the tracer keeps a buffer per goroutine. It does not — it keeps a buffer per M, and reconstructs the per-goroutine timeline from the event stream using the per-goroutine sequence numbers.

What to go do about it: run go tool trace on a program that is slow, not one that is busy. Then stop looking at the flame graph and look at the “Scheduler latency profile” and the goroutine timeline — that is where the answer usually is.

Profiling on Opensource internals

How Go's CPU Profiler Works: SIGPROF, the Signal Stack and 100 Hz Sampling

The 100 Hz decision

Arming the timer

The signal stack

The handler: identify the goroutine, unwind the stack

Writing to the ring buffer

The three-word header

The memory profile is a different animal

Why CPU-bound goroutines dominate

Trade-offs

Caveats

Wrapping up

Further reading

Go's Execution Tracer: A Ring Buffer of Events Per Goroutine

What makes it different from pprof

Starting a trace

The overall shape

Per-M buffers, not per-goroutine buffers

Writing an event

The traceLocker and the reentrancy guard

The events themselves

Deduplication: strings, stacks and types

Generations and the file format

What it costs

Trade-offs

Caveats

Wrapping up

Further reading