How Go's CPU Profiler Works: SIGPROF, the Signal Stack and 100 Hz Sampling

TL;DR: Go’s CPU profiler is an interval timer plus a signal handler. runtime/pprof.StartCPUProfile hardcodes 100 Hz, the runtime arms setitimer(ITIMER_PROF) with that period, and every time the timer fires the kernel raises SIGPROF on some thread. The handler runs on a dedicated 32 KB signal stack, unwinds the interrupted goroutine’s stack in place, and appends the PCs to a lock-free ring buffer. A separate goroutine drains that buffer and writes a protobuf profile.

Profiling a Go program is easy: call pprof.StartCPUProfile(f) and walk away. What that call actually sets up is a POSIX interval timer, a kernel signal, a signal handler running on a stack the runtime reserves for exactly this purpose, a stack unwinder that has to work while the world is half-frozen, and a lock-free ring buffer shared between a signal handler and an ordinary goroutine.

Every one of those pieces is in the Go tree, and every one of them is a little unusual. Let’s walk through them, based on the Go 1.28 development tree.

The 100 Hz decision

The frequency is not configurable through the public API, and the reason is written down in runtime/pprof:

go
// src/runtime/pprof/pprof.go:885
func StartCPUProfile(w io.Writer) error {
	// The runtime routines allow a variable profiling rate,
	// but in practice operating systems cannot trigger signals
	// at more than about 500 Hz, and our processing of the
	// signal is not cheap (mostly getting the stack trace).
	// 100 Hz is a reasonable choice: it is frequent enough to
	// produce useful data, rare enough not to bog down the
	// system, and a nice round number to make it easy to
	// convert sample counts to seconds. Instead of requiring
	// each client to specify the frequency, we hard code it.
	const hz = 100

	cpu.Lock()
	defer cpu.Unlock()
	if cpu.done == nil {
		cpu.done = make(chan bool)
	}
	// Double-check.
	if cpu.profiling {
		return fmt.Errorf("cpu profiling already in use")
	}
	cpu.profiling = true
	pprof_setCPUProfileRate(hz)
	go profileWriter(w)
	return nil
}

Two things to notice. First, 100 is hardcoded — runtime.SetCPUProfileRate exists and does let you pick a rate, but runtime/pprof refuses to use it. Second, the comment tells you the real cost model: the expensive part is not the signal, it is the stack unwinding that follows it. That single sentence explains most of the design decisions below.

profileWriter is the other half of the public API, and it is the goroutine that drains the samples and turns them into a profile:

go
// src/runtime/pprof/pprof.go:919
func profileWriter(w io.Writer) {
	b := newProfileBuilder(w)
	var err error
	for {
		if runtime.GOOS == "darwin" || runtime.GOOS == "ios" {
			// see runtime_pprof_readProfile
			time.Sleep(100 * time.Millisecond)
		}
		data, tags, eof := readProfile()
		if e := b.addCPUData(data, tags); e != nil && err == nil {
			err = e
		}
		if eof {
			break
		}
	}
	if err != nil {
		// The runtime should never produce an invalid or truncated profile.
		// It drops records that can't fit into its log buffers.
		panic("runtime/pprof: converting profile: " + err.Error())
	}
	b.build()
	cpu.done <- true
}

The signal handler and this goroutine never touch the same data structure directly. They meet in profBuf, a ring buffer designed for exactly one reader and one writer.

Arming the timer

pprof_setCPUProfileRate is a go:linknamed entry point into the runtime, which lands in setCPUProfileRate in src/runtime/cpuprof.go. That function creates the ring buffer, writes one special record into it (the sampling rate), and then calls setcpuprofilerate, which eventually reaches the OS-specific timer setup:

go
// src/runtime/signal_unix.go:276
// setProcessCPUProfilerTimer is called when the profiling timer changes.
// It is called with prof.signalLock held. hz is the new timer, and is 0 if
// profiling is being disabled. Enable or disable the signal as
// required for -buildmode=c-archive.
func setProcessCPUProfilerTimer(hz int32) {
	if hz != 0 {
		// Enable the Go signal handler if not enabled.
		if atomic.Cas(&handlingSig[_SIGPROF], 0, 1) {
			h := getsig(_SIGPROF)
			// If no signal handler was installed before, then we record
			// _SIG_IGN here. When we turn off profiling (below) we'll start
			// ignoring SIGPROF signals. We do this, rather than change
			// to SIG_DFL, because there may be a pending SIGPROF
			// signal that has not yet been delivered to some other thread.
			// If we change to SIG_DFL when turning off profiling, the
			// program will crash when that SIGPROF is delivered. We assume
			// that programs that use profiling don't want to crash on a
			// stray SIGPROF. See issue 19320.
			// We do the change here instead of when turning off profiling,
			// because there we may race with a signal handler running
			// concurrently, in particular, sigfwdgo may observe _SIG_DFL and
			// die. See issue 43828.
			if h == _SIG_DFL {
				h = _SIG_IGN
			}
			atomic.Storeuintptr(&fwdSig[_SIGPROF], h)
			setsig(_SIGPROF, abi.FuncPCABIInternal(sighandler))
		}

		var it itimerval
		it.it_interval.tv_sec = 0
		it.it_interval.set_usec(1000000 / hz)
		it.it_value = it.it_interval
		setitimer(_ITIMER_PROF, &it, nil)
	} else {
		setitimer(_ITIMER_PROF, &itimerval{}, nil)

setitimer(ITIMER_PROF) is the crux. ITIMER_PROF counts CPU time consumed by the process (user time plus system time), not wall-clock time. So a program that spends its life blocked on a socket does not accumulate profiling ticks; a program burning CPU in a tight loop does. The timer is process-wide, and when it expires the kernel picks a thread that does not have the signal blocked and raises SIGPROF there.

That “picks a thread” is why the profiler is biased the way it is. The sample is attributed to whatever goroutine happened to be running on the thread that won the race to receive the signal.

The _SIG_IGN dance above is a lovely piece of defensive engineering: turning the timer off does not cancel an already-pending signal, so switching the handler back to SIG_DFL would let that pending signal kill the program.

The signal stack

Signals are delivered on the thread’s normal stack by default, which is a disaster for a profiler: the interrupted goroutine’s stack is already in use, and the handler needs room to walk it. Go gives every M a dedicated signal stack.

go
// src/runtime/os_linux.go:385
// Called to initialize a new m (including the bootstrap m).
// Called on the parent thread (main thread in case of bootstrap), can allocate memory.
func mpreinit(mp *m) {
	mp.gsignal = malg(32 * 1024) // Linux wants >= 2K
	mp.gsignal.m = mp
}

And it installs it as the thread’s alternate signal stack, with SA_ONSTACK:

go
// src/runtime/os_linux.go:478
func setsig(i uint32, fn uintptr) {
	var sa sigactiont
	sa.sa_flags = _SA_SIGINFO | _SA_ONSTACK | _SA_RESTORER | _SA_RESTART
	sigfillset(&sa.sa_mask)
	// Although Linux manpage says "sa_restorer element is obsolete and
	// should not be used". x86_64 kernel requires it. Only use it on
	// x86. Note that on 386 this is cleared when using the C sigaction
	// function via cgo; see fixSigactionForCgo.
	if GOARCH == "386" || GOARCH == "amd64" {
		sa.sa_restorer = abi.FuncPCABI0(sigreturn__sigaction)
	}
	if fn == abi.FuncPCABIInternal(sighandler) { // abi.FuncPCABIInternal(sighandler) matches the callers in signal_unix.go
		if iscgo {
			fn = abi.FuncPCABI0(cgoSigtramp)
		} else {
			fn = abi.FuncPCABI0(sigtramp)
		}
	}
	sa.sa_handler = fn
	sigaction(i, &sa, nil)
}

Note sigfillset(&sa.sa_mask): while the handler runs, every signal is blocked. That is why the runtime can rely on a single signal at a time — no reentrancy into the profiler from another signal — and it is also why the profiler’s own locking is a simple compare-and-swap rather than a mutex.

Each thread gets its own 32 KB gsignal goroutine stack, set up when the M is initialised:

go
// src/runtime/signal_unix.go:1329
func minitSignals() {
	minitSignalStack()
	minitSignalMask()
}

// minitSignalStack is called when initializing a new m to set the
// alternate signal stack. If the alternate signal stack is not set
// for the thread (the normal case) then set the alternate signal
// stack to the gsignal stack. If the alternate signal stack is set
// for the thread (the case when a non-Go thread sets the alternate
// signal stack and then calls a Go function) then set the gsignal
// stack to the alternate signal stack. We also set the alternate
// signal stack to the gsignal stack if cgo is not used (regardless
// of whether it is already set). Record which choice was made in
// newSigstack, so that it can be undone in unminit.
func minitSignalStack() {
	mp := getg().m
	var st stackt
	sigaltstack(nil, &st)
	if st.ss_flags&_SS_DISABLE != 0 || !iscgo {
		signalstack(&mp.gsignal.stack)
		mp.newSigstack = true
	} else {
		setGsignalStack(&st, &mp.goSigStack)
		mp.newSigstack = false
	}
}

The handler: identify the goroutine, unwind the stack

The handler itself is sighandler, which handles every signal Go cares about. SIGPROF gets an early return:

go
// src/runtime/signal_unix.go:674
	if sig == _SIGPROF {
		// Some platforms (Linux) have per-thread timers, which we use in
		// combination with the process-wide timer. Avoid double-counting.
		if !delayedSignal && validSIGPROF(mp, c) {
			sigprof(c.sigpc(), c.sigsp(), c.siglr(), gp, mp)
		}
		return
	}

gp is the G that was running when the signal arrived; mp is the M it was running on. That is the “per-thread goroutine identification”: the sample is attributed to mp.curg, and the goroutine’s pprof labels are attached to the sample as a tag pointer.

sigprof is where the stack is unwound:

go
// src/runtime/proc.go:5801
// Called if we receive a SIGPROF signal.
// Called by the signal handler, may run during STW.
//
//go:nowritebarrierrec
func sigprof(pc, sp, lr uintptr, gp *g, mp *m) {
	if prof.hz.Load() == 0 {
		return
	}

	// If mp.profilehz is 0, then profiling is not enabled for this thread.
	// We must check this to avoid a deadlock between setcpuprofilerate
	// and the call to cpuprof.add, below.
	if mp != nil && mp.profilehz == 0 {
		return
	}
	...
	// Profiling runs concurrently with GC, so it must not allocate.
	// Set a trap in case the code does allocate.
	// Note that on windows, one thread takes profiles of all the
	// other threads, so mp is usually not getg().m.
	// In fact mp may not even be stopped.
	// See golang.org/issue/17165.
	getg().m.mallocing++

	var u unwinder
	var stk [maxCPUProfStack]uintptr
	n := 0
	...
	} else {
		u.initAt(pc, sp, lr, gp, unwindSilentErrors|unwindTrap|unwindJumpStack)
	}
	n += tracebackPCs(&u, 0, stk[n:])

Here is a genuinely surprising fact about modern Go: gentraceback is gone. It used to be the workhorse, and every blog post about Go stack unwinding still describes it. In the Go 1.28 tree there is no gentraceback function at all. It has been replaced by the unwinder type, an explicit iterator:

go
// src/runtime/traceback.go:84
// An unwinder iterates the physical stack frames of a Go sack.
//
// Typical use of an unwinder looks like:
//
//	var u unwinder
//	for u.init(gp, 0); u.valid(); u.next() {
//		// ... use frame info in u ...
//	}
//
// Implementation note: This is carefully structured to be pointer-free because
// tracebacks happen in places that disallow write barriers (e.g., signals).
// Even if this is stack-allocated, its pointer-receiver methods don't know that
// their receiver is on the stack, so they still emit write barriers. Here we
// address that by carefully avoiding any pointers in this type. Another
// approach would be to split this into a mutable part that's passed by pointer
// but contains no pointers itself and an immutable part that's passed and
// returned by value and can contain pointers. We could potentially hide that
// we're doing that in trivial methods that are inlined into the caller that has
// the stack allocation, but that's fragile.
type unwinder struct {
	// frame is the current physical stack frame, or all 0s if
	// there is no frame.
	frame stkframe

The pointer-free design is not cosmetic. The signal handler runs with write barriers disabled (the GC may be in the middle of a stop-the-world), and the unwinder is deliberately built so it cannot allocate or trigger a barrier.

The result of the unwind is a flat array of return PCs, up to 64 frames:

go
// src/runtime/traceback.go:645
func tracebackPCs(u *unwinder, skip int, pcBuf []uintptr) int {
	var cgoBuf [32]uintptr
	n := 0
	for ; n < len(pcBuf) && u.valid(); u.next() {
		f := u.frame.fn
		cgoN := u.cgoCallers(cgoBuf[:])

		// TODO: Why does &u.cache cause u to escape? (Same in traceback2)
		for iu, uf := newInlineUnwinder(f, u.symPC()); n < len(pcBuf) && uf.valid(); uf = iu.next(uf) {
			sf := iu.srcFunc(uf)
			if sf.funcID == abi.FuncIDWrapper && elideWrapperCalling(u.calleeFuncID) {
				// ignore wrappers
			} else if skip > 0 {
				skip--
			} else {
				// Callers expect the pc buffer to contain return addresses
				// and do the -1 themselves, so we add 1 to the call pc to
				// create a "return pc". Since there is no actual call, here
				// "return pc" just means a pc you subtract 1 from to get
				// the pc of the "call". The actual no-op we insert may or
				// may not be 1 byte.
				pcBuf[n] = uf.pc + 1
				n++
			}
			u.calleeFuncID = sf.funcID
		}

If the unwind fails — the sample landed in a JIT blob, or in C code with no Go frames — the profiler falls back to a synthetic two-frame stack of System or GC plus _ExternalCode. Those fake frames show up in go tool pprof output as runtime._System and runtime._ExternalCode, and they are a reliable signal that the time was spent outside Go.

Writing to the ring buffer

The sample is handed to cpuprof.add, which does the least it can get away with:

go
// src/runtime/cpuprof.go:108
// add adds the stack trace to the profile.
// It is called from signal handlers and other limited environments
// and cannot allocate memory or acquire locks that might be
// held at the time of the signal, nor can it use substantial amounts
// of stack.
//
//go:nowritebarrierrec
func (p *cpuProfile) add(tagPtr *unsafe.Pointer, stk []uintptr) {
	// Simple cas-lock to coordinate with setcpuprofilerate.
	for !prof.signalLock.CompareAndSwap(0, 1) {
		// TODO: Is it safe to osyield here? https://go.dev/issue/52672
		osyield()
	}

	if prof.hz.Load() != 0 { // implies cpuprof.log != nil
		if p.numExtra > 0 || p.lostExtra > 0 || p.lostAtomic > 0 {
			p.addExtra()
		}
		hdr := [1]uint64{1}
		// Note: write "knows" that the argument is &gp.labels,
		// because otherwise its write barrier behavior may not
		// be correct. See the long comment there before
		// changing the argument here.
		cpuprof.log.write(tagPtr, nanotime(), hdr[:], stk)
	}

	prof.signalLock.Store(0)
}

No allocation, no mutex — a compare-and-swap on prof.signalLock and a copy into a preallocated ring. The ring is profBuf, and its record layout is deliberately trivial:

go
// src/runtime/profbuf.go:358
	// Main record.
	// It has to fit in a contiguous section of the slice, so if it doesn't fit at the end,
	// leave a rewind marker (0) and start over at the beginning of the slice.
	wd := int(bw.dataCount() % uint32(len(b.data)))
	nd := countSub(br.dataCount(), bw.dataCount()) + len(b.data)
	skip := 0
	if wd+2+int(b.hdrsize)+len(stk) > len(b.data) {
		b.data[wd] = 0
		skip = len(b.data) - wd
		nd -= skip
		wd = 0
	}
	data := b.data[wd:]
	data[0] = uint64(2 + b.hdrsize + uintptr(len(stk))) // length
	data[1] = uint64(now)                               // time stamp
	// header, zero-padded
	i := copy(data[2:2+b.hdrsize], hdr)
	clear(data[2+i : 2+b.hdrsize])
	for i, pc := range stk {
		data[2+b.hdrsize+uintptr(i)] = uint64(pc)
	}

Each record is: a length word, a timestamp, a one-word header (the sample count), and the PCs. The buffer is sized in the source with a very human comment:

go
// src/runtime/cpuprof.go:21
const (
	maxCPUProfStack = 64

	// profBufWordCount is the size of the CPU profile buffer's storage for the
	// header and stack of each sample, measured in 64-bit words. Every sample
	// has a required header of two words. With a small additional header (a
	// word or two) and stacks at the profiler's maximum length of 64 frames,
	// that capacity can support 1900 samples or 19 thread-seconds at a 100 Hz
	// sample rate, at a cost of 1 MiB.
	profBufWordCount = 1 << 17

19 thread-seconds of samples in a megabyte. If the reader goroutine falls behind and the ring fills up, samples are dropped and the drop count is recorded as a special “overflow” record rather than silently lost.

The three-word header

The very first record the runtime writes is not a sample — it is the sampling rate, and the reader uses it to turn sample counts into nanoseconds:

go
// src/runtime/cpuprof.go:94
		cpuprof.on = true
		cpuprof.log = newProfBuf(1, profBufWordCount, profBufTagCount)
		hdr := [1]uint64{uint64(hz)}
		cpuprof.log.write(nil, nanotime(), hdr[:], nil)
		setcpuprofilerate(int32(hz))

That record has a zero-length stack, so it occupies three words: length (3), timestamp, and the frequency. The reader checks for exactly that shape:

go
// src/runtime/pprof/proto.go:278
func (b *profileBuilder) addCPUData(data []uint64, tags []unsafe.Pointer) error {
	if !b.havePeriod {
		// first record is period
		if len(data) < 3 {
			return fmt.Errorf("truncated profile")
		}
		if data[0] != 3 || data[2] == 0 {
			return fmt.Errorf("malformed profile")
		}
		// data[2] is sampling rate in Hz. Convert to sampling
		// period in nanoseconds.
		b.period = 1e9 / int64(data[2])
		b.havePeriod = true
		data = data[3:]
		// Consume tag slot. Note that there isn't a meaningful tag
		// value for this record.
		tags = tags[1:]
	}

The same routine also decodes the overflow records, which is how a dropped sample shows up in the profile rather than vanishing:

go
// src/runtime/pprof/proto.go:322
		count := data[2]
		stk := data[3:data[0]]
		data = data[data[0]:]
		tag := tags[0]
		tags = tags[1:]

		if count == 0 && len(stk) == 1 {
			// overflow record
			count = uint64(stk[0])
			stk = []uint64{
				// gentraceback guarantees that PCs in the
				// stack can be unconditionally decremented and
				// still be valid, so we must do the same.
				uint64(abi.FuncPCABIInternal(lostProfileEvent) + 1),
			}
		}
		b.m.lookup(stk, tag).count += int64(count)

(The comment mentioning gentraceback is a leftover from before the unwinder rewrite — the property it describes still holds.)

The builder then deduplicates identical stacks on the fly and finally emits the protobuf with two sample types, samples/count and cpu/nanoseconds.

The memory profile is a different animal

The memory profile does not use signals at all. It samples allocations, at a rate set by a global:

go
// src/runtime/mprof.go:856
// MemProfileRate controls the fraction of memory allocations
// that are recorded and reported in the memory profile.
// The profiler aims to sample an average of
// one allocation per MemProfileRate bytes allocated.
//
// To include every allocated block in the profile, set MemProfileRate to 1.
// To turn off profiling entirely, set MemProfileRate to 0.
//
// The tools that process the memory profiles assume that the
// profile rate is constant across the lifetime of the program
// and equal to the current value. Programs that change the
// memory profiling rate should do so just once, as early as
// possible in the execution of the program (for example,
// at the beginning of main).
var MemProfileRate int = 512 * 1024

512 KB, by default. The decision of which allocation to sample is made with a counter kept in the mcache — the per-P allocation cache — and the next sample point is drawn from an exponential distribution so that the sampling is Poissonian rather than periodic:

go
// src/runtime/malloc.go:2236
// nextSample returns the next sampling point for heap profiling. The goal is
// to sample allocations on average every MemProfileRate bytes, but with a
// completely random distribution over the allocation timeline; this
// corresponds to a Poisson process with parameter MemProfileRate. In Poisson
// processes, the distance between two samples follows the exponential
// distribution (exp(MemProfileRate)), so the best return value is a random
// number taken from an exponential distribution whose mean is MemProfileRate.
func nextSample() int64 {
	if MemProfileRate == 0 {
		// Basically never sample.
		return math.MaxInt64
	}
	if MemProfileRate == 1 {
		// Sample immediately.
		return 0
	}
	return int64(fastexprand(MemProfileRate))
}

The counter lives on the mcache, and the check happens on every allocation:

go
// src/runtime/malloc.go:1449
	c.nextSample -= int64(size)
	if c.nextSample < 0 || MemProfileRate != c.memProfRate {
		profilealloc(mp, x, size)
	}

So: the CPU profile samples time using a signal; the memory profile samples bytes using a counter in the allocator cache. Different mechanisms, different biases. The memory profile counts the sampled allocation as if it represented MemProfileRate bytes, which is why pprof scales it up — a 4 KB allocation sampled at 512 KB per sample is reported as 512 KB.

Why CPU-bound goroutines dominate

Because ITIMER_PROF counts CPU time, a goroutine only gets sampled when it is actually on the CPU. A goroutine blocked on a channel, sleeping on a timer, or waiting on a mutex is invisible to the CPU profiler no matter how important it is. The profile answers “where did the CPU time go”, not “where did the program spend its time”.

That is the correct answer for a CPU profiler, but it means a program dominated by I/O will produce a profile full of runtime._System and runtime._ExternalCode frames and nothing else. For “where is my time going”, you want the execution tracer or the block profile instead.

Two more biases worth knowing:

Short goroutines are undersampled. At 100 Hz, a goroutine needs to burn roughly 10 ms of CPU to have a decent chance of appearing at all.
The signal lands on an arbitrary thread. The kernel picks a thread that does not block SIGPROF, so the sample is a snapshot of that thread’s current goroutine. This is unbiased in aggregate, but it means individual samples are attributed to a goroutine that was merely running, not necessarily the one that consumed the CPU time.

Trade-offs

What sampling buys you: near-zero steady-state overhead. The timer fires 100 times a second, each firing costs one signal delivery plus one stack unwind, and nothing else in the program changes.

What it costs:

Statistical error. A short profile is a small sample. 100 samples is 1 second of CPU time; the standard error on a 1% line is roughly 10% relative.
The unwinder is the expensive part. The runtime comment says so outright: “our processing of the signal is not cheap (mostly getting the stack trace)”.
Signal-handler constraints. The handler cannot allocate, cannot take a mutex, and cannot grow the stack. The whole unwinder design is shaped by that.
Dropped samples. If the reader goroutine cannot keep up, samples are dropped and reported as an overflow record rather than blocking the signal handler.
Frames are lost at 64. maxCPUProfStack is 64; deeper stacks are truncated, and the truncated frame is a plain function name.

Caveats

SIGPROF is a normal signal. If your program calls os/signal.Notify(c, syscall.SIGPROF), it will fight the profiler for the signal. On Unix, sigenable explicitly refuses to touch SIGPROF:

go
// src/runtime/signal_unix.go:195
	// SIGPROF is handled specially for profiling.
	if sig == _SIGPROF {
		return
	}

c-archive and c-shared builds are different. StartCPUProfile documents that it does not work by default there, because SIGPROF goes to the host program’s handler rather than Go’s.

Linux has per-thread timers. The kernel’s timer_create timers show up as _SI_TIMER signal codes while setitimer produces _SI_KERNEL, and validSIGPROF uses that to avoid counting the same CPU time twice.

Darwin cannot block in the reader. runtime_pprof_readProfile uses a non-blocking read on Darwin and iOS, and profileWriter sleeps 100 ms between reads, because notetsleepg is not safe in a signal handler there.

Profiles are not comparable across runs unless the rate is the same — which it is, since 100 Hz is hardcoded — but they are not comparable if the program was interrupted differently or if samples were dropped.

Wrapping up

The one thing to remember: the CPU profiler is a 100 Hz setitimer(ITIMER_PROF) plus a SIGPROF handler that runs on a per-M 32 KB signal stack, unwinds the interrupted goroutine with a pointer-free unwinder, and copies the PCs into a lock-free ring buffer that a separate goroutine turns into a profile.

The part people get wrong: ITIMER_PROF counts CPU time, so the profile shows you where the CPU went, not where the wall-clock went. A program that is mostly blocked will show almost nothing, and the samples it does show will be attributed to runtime._System.

What to go do about it: turn on a CPU profile and an execution trace at the same time. The profile tells you which function is hot; the trace tells you why the goroutine that called it was asleep. And if the profile is suspiciously empty, check for a SIGPROF handler installed by a library before you go looking for a performance bug.