Scheduler on Opensource internals

Go's Timers: A Per-P Heap and a Timer That Never Blocks a Thread

Jay Hubbert — Wed, 21 Oct 2026 04:16:00 +0000

TL;DR: A Go timer is not a thread, not a syscall, and not a separate kernel object. It is a struct with a when and a period that lives in a min-heap owned by the P that created it. The scheduler checks that heap at the top of every scheduling round, runs whatever is due, and a goroutine that calls time.Sleep parks itself on a timer attached to its own g.

If you have written Go for more than a week, you have written time.After, time.NewTimer or time.Sleep. If you have written C, you are probably convinced that each of those is secretly an OS thread with a nanosleep in it. It is not. Go’s timers are a data structure.

That data structure has been rewritten fairly recently, and if you have been reading about Go internals for a while you may notice that the function names you remember are gone. There is no addtimer, no deltimer, no modtimer and no checkTimers in the Go 1.28 tree. They were replaced by a per-P heap with timer.modify, timer.maybeAdd and timers.check. Here is how the new design works.

The shape of the problem

A timer has to do three things: know when it is due, wake somebody up at that time, and let the owner change or cancel it without races.

The naive design — one kernel timer per Go timer — fails at scale. A service with a million time.After calls in flight would have a million kernel timer objects, a million interrupts, and enormous scheduler latency. Go’s answer is to keep every timer in user space and let the Go scheduler poll.

The polling is what makes it cheap. When the scheduler has nothing else to do, it computes the earliest due time across all the heaps it knows about, sleeps in netpoll until then, and runs what is due. No dedicated thread, no per-timer signal, no interrupt storm.

The timer struct

 go
 
 
// src/runtime/time.go:52
// A timer is a potentially repeating trigger for calling t.f(t.arg, t.seq).
// Timers are allocated by client code, often as part of other data structures.
// Each P has a heap of pointers to timers that it manages.
//
// A timer is expected to be used by only one client goroutine at a time,
// but there will be concurrent access by the P managing that timer.
// Timer accesses are protected by the lock t.mu, with a snapshot of
// t's state bits published in t.astate to enable certain fast paths to make
// decisions about a timer without acquiring the lock.
type timer struct {
	// mu protects reads and writes to all fields, with exceptions noted below.
	mu mutex

	astate atomic.Uint8 // atomic copy of state bits at last unlock
	state uint8 // state bits
	isChan bool // timer has a channel; immutable; can be read without lock
	isFake bool // timer is using fake time; immutable; can be read without lock

	blocked uint32 // number of goroutines blocked on timer's channel
	rand uint32 // randomizes order of timers at same instant; only set when isFake

	// Timer wakes up at when, and then at when+period, ... (period > 0 only)
	// each time calling f(arg, seq, delay) in the timer goroutine, so f must be
	// a well-behaved function and not block.
	//
	// The arg and seq are client-specified opaque arguments passed back to f.
	// When used from netpoll, arg and seq have meanings defined by netpoll
	// and are completely opaque to this code; in that context, seq is a sequence
	// number to recognize and squelch stale function invocations.
	// When used from package time, arg is a channel (for After, NewTicker)
	// or the function to call (for AfterFunc) and seq is unused (0).
	...
	when int64
	period int64
	f func(arg any, seq uintptr, delay int64)
	arg any
	seq uintptr

	// If non-nil, the timers containing t.
	ts *timers

	// sendLock protects sends on the timer's channel.
	sendLock mutex
	...
	isSending atomic.Int32
}

Four things in there are worth pulling out.

when and period. A one-shot timer has period == 0; a ticker has period > 0 and re-arms itself by when += period.

f and arg. The callback. For time.After and time.NewTimer the callback is sendTime and arg is the channel; for time.AfterFunc the callback is goFunc and arg is your function.

ts *timers. The pointer back to the owning heap. This is what makes “stop a timer that lives on another P” possible: you cannot remove it from the heap directly, so you mark it and let the owner clean it up.

mu plus astate. The timer has its own lock, and a lock-free snapshot of the state bits. The snapshot exists so that the owning P can decide whether a timer needs attention without taking the timer’s lock — which matters, because the P holds its own heap lock at that moment and the lock ordering forbids taking them in the other direction.

The per-P heap

Each P owns one heap of timers:

 go
 
 
// src/runtime/time.go:135
// A timers is a per-P set of timers.
type timers struct {
	// mu protects timers; timers are per-P, but the scheduler can
	// access the timers of another P, so we have to lock.
	mu mutex

	// heap is the set of timers, ordered by heap[i].when.
	// Must hold lock to access.
	heap []timerWhen

	// len is an atomic copy of len(heap).
	len atomic.Uint32

	// zombies is the number of timers in the heap
	// that are marked for removal.
	zombies atomic.Int32

	// raceCtx is the race context used while executing timer functions.
	raceCtx uintptr

	// minWhenHeap is the minimum heap[i].when value (= heap[0].when).
	// The wakeTime method uses minWhenHeap and minWhenModified
	// to determine the next wake time.
	// If minWhenHeap = 0, it means there are no timers in the heap.
	minWhenHeap atomic.Int64

	// minWhenModified is a lower bound on the minimum
	// heap[i].when over timers with the timerModified bit set.
	// If minWhenModified = 0, it means there are no timerModified timers in the heap.
	minWhenModified atomic.Int64
}

type timerWhen struct {
	timer *timer
	when int64
}

The heap stores timerWhen — a pair of a pointer and the time the heap believes the timer is due — rather than the timer itself. That duplication is the key to the whole design, and it is why the state bits exist.

The P struct holds it by value:

 go
 
 
// src/runtime/runtime2.go:896
	// Timer heap.
	timers timers

Ownership diagram:

 text
 
 
 P0 P1 P2
 +---------------+ +---------------+ +---------------+
 | timers.heap | | timers.heap | | timers.heap |
 | [t1, t5, t9] | | [t2] | | [] |
 +---------------+ +---------------+ +---------------+
 | | |
 owned by P0 owned by P1 owned by P2
 ^ ^ ^
 | | |
 a timer created by a goroutine running on P0 lands on P0's heap,
 regardless of which goroutine created it.

Note the comment on mu: “timers are per-P, but the scheduler can access the timers of another P, so we have to lock.” The scheduler steals timers from idle Ps, so the heap lock is not optional.

Three state bits

 go
 
 
// src/runtime/time.go:204
// Timer state field.
const (
	// timerHeaped is set when the timer is stored in some P's heap.
	timerHeaped uint8 = 1 << iota

	// timerModified is set when t.when has been modified
	// but the heap's heap[i].when entry still needs to be updated.
	// That change waits until the heap in which
	// the timer appears can be locked and rearranged.
	// timerModified is only set when timerHeaped is also set.
	timerModified

	// timerZombie is set when the timer has been stopped
	// but is still present in some P's heap.
	// Only set when timerHeaped is also set.
	// It is possible for timerModified and timerZombie to both
	// be set, meaning that the timer was modified and then stopped.
	// A timer sending to a channel may be placed in timerZombie
	// to take it out of the heap even though the timer is not stopped,
	// as long as nothing is reading from the channel.
	timerZombie
)

These three bits are the whole concurrency story.

timerHeaped — “I live in somebody’s heap right now.”

timerModified — “the time in the heap is stale; the real due time is in t.when.” This is what lets Reset be cheap: it updates t.when and sets a bit, and the heap is rearranged later, when someone takes the heap lock anyway. Without the bit, the heap would have to be searched and fixed up on every reset.

timerZombie — “I am stopped but still physically in the heap.” Removing an arbitrary element from a binary heap is O(n), so Stop does not remove anything; it marks the timer and lets the owner sweep it out.

The heap is a plain binary min-heap keyed on when:

 go
 
 
// src/runtime/time.go:172
// less reports whether tw is less than other.
func (tw timerWhen) less(other timerWhen) bool {
	switch {
	case tw.when < other.when:
		return true
	case tw.when > other.when:
		return false
	default:
		// When timers wake at the same time, use a per-timer random value to order them.
		// We only set the random value for timers using fake time, since there's
		// no practical way to schedule real-time timers for the same instant.
		return tw.timer.rand < other.timer.rand
	}
}

Creating a timer

package time does not implement timers; it calls into the runtime through go:linkname. The runtime allocates the struct and calls modify:

 go
 
 
// src/runtime/time.go:390
// newTimer allocates and returns a new time.Timer or time.Ticker (same layout)
// with the given parameters.
//
//go:linkname newTimer time.newTimer
func newTimer(when, period int64, f func(arg any, seq uintptr, delay int64), arg any, c *hchan) *timeTimer {
	t := new(timeTimer)
	t.timer.init(nil, nil)
	t.trace("new")
	if raceenabled {
		racerelease(unsafe.Pointer(&t.timer))
	}
	if c != nil {
		lockInit(&t.sendLock, lockRankTimerSend)
		t.isChan = true
		c.timer = &t.timer
		if c.dataqsiz == 0 {
			throw("invalid timer channel: no capacity")
		}
	}
	if bubble := getg().bubble; bubble != nil {
		t.isFake = true
	}
	t.modify(when, period, f, arg, 0)
	t.self = t
	return t
}

modify is the modern replacement for the old modtimer, and it is where most of the interesting decisions live:

 go
 
 
// src/runtime/time.go:571
	wake := false
	pending := t.when > 0
	t.when = when
	if t.state&timerHeaped != 0 {
		t.state |= timerModified
		if t.state&timerZombie != 0 {
			// In the heap but marked for removal (by a Stop).
			// Unmark it, since it has been Reset and will be running again.
			t.ts.zombies.Add(-1)
			t.state &^= timerZombie
		}
		// The corresponding heap[i].when is updated later.
		// See comment in type timer above and in timers.adjust below.
		if min := t.ts.minWhenModified.Load(); min == 0 || when < min {
			wake = true
			// Force timerModified bit out to t.astate before updating t.minWhenModified,
			// to synchronize with t.ts.adjust. See comment in adjust.
			t.astate.Store(t.state)
			t.ts.updateMinWhenModified(when)
		}
	}

The wake flag is the interesting part. If the timer’s new due time is earlier than anything the heap currently knows about, the netpoller has to be woken up, because it may be sleeping until the old, later time. That is what wakeNetPoller is for.

And when the timer is not in a heap yet, it has to be added — but not necessarily to the P that is running right now:

 go
 
 
// src/runtime/time.go:666
// maybeAdd adds t to the local timers heap if it needs to be in a heap.
// The caller must not hold t's lock nor any timers heap lock.
// The caller probably just unlocked t, but that lock must be dropped
// in order to acquire a ts.lock, to avoid lock inversions.
// (timers.adjust holds ts.lock while acquiring each t's lock,
// so we cannot hold any t's lock while acquiring ts.lock).
//
// Strictly speaking it *might* be okay to hold t.lock and
// acquire ts.lock at the same time, because we know that
// t is not in any ts.heap, so nothing holding a ts.lock would
// be acquiring the t.lock at the same time, meaning there
// isn't a possible deadlock. But it is easier and safer not to be
// too clever and respect the static ordering.
// (If we don't, we have to change the static lock checking of t and ts.)
//
// Concurrent calls to time.Timer.Reset or blockTimerChan
// may result in concurrent calls to t.maybeAdd,
// so we cannot assume that t is not in a heap on entry to t.maybeAdd.
func (t *timer) maybeAdd() {
	// Note: Not holding any locks on entry to t.maybeAdd,
	// so the current g can be rescheduled to a different M and P
	// at any time, including between the ts := assignment and the
	// call to ts.lock. If a reschedule happened then, we would be
	// adding t to some other P's timers, perhaps even a P that the scheduler
	// has marked as idle with no timers, in which case the timer could
	// go unnoticed until long after t.when.
	// Calling acquirem instead of using getg().m makes sure that
	// we end up locking and inserting into the current P's timers.
	mp := acquirem()

That comment is a whole bug report in five lines: if the goroutine migrated to another P between choosing the heap and taking its lock, the timer could land on an idle P that nobody is checking, and would not fire until that P woke up for some other reason. acquirem pins the M so the P cannot change underfoot.

Sleeping

time.Sleep is a different path from time.After, and the difference is elegant. A sleeping goroutine gets a timer of its own, stored on its g:

 go
 
 
// src/runtime/time.go:332
// timeSleep puts the current goroutine to sleep for at least ns nanoseconds.
//
//go:linkname timeSleep time.Sleep
func timeSleep(ns int64) {
	if ns <= 0 {
		return
	}

	gp := getg()
	t := gp.timer
	if t == nil {
		t = new(timer)
		t.init(goroutineReady, gp)
		if gp.bubble != nil {
			t.isFake = true
		}
		gp.timer = t
	}
	var now int64
	if bubble := gp.bubble; bubble != nil {
		now = bubble.now
	} else {
		now = nanotime()
	}
	when := now + ns
	if when < 0 { // check for overflow.
		when = maxWhen
	}
	gp.sleepWhen = when
	if t.isFake {
		// Call timer.reset in this goroutine, since it's the one in a bubble.
		// We don't need to worry about the timer function running before the goroutine
		// is parked, because time won't advance until we park.
		resetForSleep(gp, nil)
		gopark(nil, nil, waitReasonSleep, traceBlockSleep, 1)
	} else {
		gopark(resetForSleep, nil, waitReasonSleep, traceBlockSleep, 1)
	}
}

gopark parks the goroutine; the resetForSleep callback is invoked after the goroutine is parked, which is the subtle part:

 go
 
 
// src/runtime/time.go:372
// resetForSleep is called after the goroutine is parked for timeSleep.
// We can't call timer.reset in timeSleep itself because if this is a short
// sleep and there are many goroutines then the P can wind up running the
// timer function, goroutineReady, before the goroutine has been parked.
func resetForSleep(gp *g, _ unsafe.Pointer) bool {
	gp.timer.reset(gp.sleepWhen, 0)
	return true
}

Without that ordering, a goroutine sleeping for one microsecond could have its timer fire — and goready it — before it ever parked, and then park forever.

The timer’s function is goroutineReady, which is just goready(arg.(*g), 0): the timer is the mechanism that puts the sleeping goroutine back on the run queue.

The scheduler’s timer check

The check happens at the top of findRunnable, before the scheduler looks for anything else to run:

 go
 
 
// src/runtime/proc.go:3430
	// now and pollUntil are saved for work stealing later,
	// which may steal timers. It's important that between now
	// and then, nothing blocks, so these numbers remain mostly
	// relevant.
	now, pollUntil, _ := pp.timers.check(0, nil)

timers.check is the modern checkTimers:

 go
 
 
// src/runtime/time.go:972
// check runs any timers in ts that are ready.
// If now is not 0 it is the current time.
// It returns the passed time or the current time if now was passed as 0.
// and the time when the next timer should run or 0 if there is no next timer,
// and reports whether it ran any timers.
// If the time when the next timer should run is not 0,
// it is always larger than the returned time.
// We pass now in and out to avoid extra calls of nanotime.
//
//go:yeswritebarrierrec
func (ts *timers) check(now int64, bubble *synctestBubble) (rnow, pollUntil int64, ran bool) {
	ts.trace("check")
	// If it's not yet time for the first timer, or the first adjusted
	// timer, then there is nothing to do.
	next := ts.wakeTime()
	if next == 0 {
		// No timers to run or adjust.
		return now, 0, false
	}

	if now == 0 {
		now = nanotime()
	}

	// If this is the local P, and there are a lot of deleted timers,
	// clear them out. We only do this for the local P to reduce
	// lock contention on timersLock.
	zombies := ts.zombies.Load()
	if zombies < 0 {
		badTimer()
	}
	force := ts == &getg().m.p.ptr().timers && int(zombies) > int(ts.len.Load())/4

	if now < next && !force {
		// Next timer is not ready to run, and we don't need to clear deleted timers.
		return now, next, false
	}

	ts.lock()
	if len(ts.heap) > 0 {
		ts.adjust(now, false)
		for len(ts.heap) > 0 {
			// Note that runtimer may temporarily unlock ts.
			if tw := ts.run(now, bubble); tw != 0 {
				if tw > 0 {
					pollUntil = tw
				}
				break
			}
			ran = true
		}

Two return values matter: pollUntil is the time of the next timer, which becomes the deadline the netpoller sleeps until, and ran tells the caller whether anything happened.

adjust is where the stale heap entries get fixed up, and it is the messiest part of the whole file — with an excellent comment explaining why:

 go
 
 
// src/runtime/time.go:852
	// minWhenModified is a lower bound on the earliest t.when
	// among the timerModified timers. We want to make it more precise:
	// we are going to scan the heap and clean out all the timerModified bits,
	// at which point minWhenModified can be set to 0 (indicating none at all).
	//
	// Other P's can be calling ts.wakeTime concurrently, and we'd like to
	// keep ts.wakeTime returning an accurate value throughout this entire process.
	//
	// Setting minWhenModified = 0 *before* the scan could make wakeTime
	// return an incorrect value: if minWhenModified < minWhenHeap, then clearing
	// it to 0 will make wakeTime return minWhenHeap (too late) until the scan finishes.
	// To avoid that, we want to set minWhenModified to 0 *after* the scan.
	//
	// Setting minWhenModified = 0 *after* the scan could result in missing
	// concurrent timer modifications in other goroutines; those will lock
	// the specific timer, set the timerModified bit, and set t.when.
	// To avoid that, we want to set minWhenModified to 0 *before* the scan.
	//
	// The way out of this dilemma is to preserve wakeTime a different way.
	// wakeTime is min(minWhenHeap, minWhenModified), and minWhenHeap
	// is protected by ts.lock, which we hold, so we can modify it however we like
	// in service of keeping wakeTime accurate.

The resolution is to “lock in” the accurate wake time into minWhenHeap before clearing minWhenModified, so the two-step update never exposes a window where the poller would sleep past a timer.

When the scheduler has nothing else to do, it sleeps in the netpoller until pollUntil:

 go
 
 
// src/runtime/proc.go:3750
	// Poll network until next timer.
	if netpollinited() && (netpollAnyWaiters() || pollUntil != 0) && sched.lastpoll.Swap(0) != 0 {
		sched.pollUntil.Store(pollUntil)
		...
		delay := int64(-1)
		if pollUntil != 0 {
			if now == 0 {
				now = nanotime()
			}
			delay = pollUntil - now
			if delay < 0 {
				delay = 0
			}
		}
		...
		list, delta := netpoll(delay) // block until new work is available

One epoll_wait with a timeout serves both the network poller and the timer heap. That is the whole trick: timers and network I/O share a wakeup mechanism, because they share the thread that sleeps waiting for both.

Stopping and resetting without races

The hard part of a channel-based timer is that Stop and Reset can race with the timer actually sending on the channel. The runtime solves this with a sequence number:

 go
 
 
// src/runtime/time.go:1178
	if t.isChan {
		// For a timer channel, we want to make sure that no stale sends
		// happen after a t.stop or t.modify, but we cannot hold t.mu
		// during the actual send (which f does) due to lock ordering.
		// It can happen that we are holding t's lock above, we decide
		// it's time to send a time value (by calling f), grab the parameters,
		// unlock above, and then a t.stop or t.modify changes the timer
		// and returns. At that point, the send needs not to happen after all.
		// The way we arrange for it not to happen is that t.stop and t.modify
		// both increment t.seq while holding both t.mu and t.sendLock.
		// We copied the seq value above while holding t.mu.
		// Now we can acquire t.sendLock (which will be held across the send)
		// and double-check that t.seq is still the seq value we saw above.
		// If not, the timer has been updated and we should skip the send.
		// We skip the send by reassigning f to a no-op function.
		//
		// The isSending field tells t.stop or t.modify that we have
		// started to send the value. That lets them correctly return
		// true meaning that no value was sent.
		lock(&t.sendLock)

		if t.period == 0 {
			// We are committed to possibly sending a value
			// based on seq, so no need to keep telling
			// stop/modify that we are sending.
			if t.isSending.Add(-1) < 0 {
				throw("mismatched isSending updates")
			}
		}

		if t.seq != seq {
			f = func(any, uintptr, int64) {}
		}
	}

	f(arg, seq, delay)

The timer copies t.seq while holding t.mu. Stop and Reset bump t.seq while holding both t.mu and t.sendLock. Just before actually sending, the timer takes t.sendLock and compares the sequence number. If it changed, the send is replaced with a no-op.

This is what makes the modern Timer semantics safe.

time.After, NewTimer, Reset and the drain rule

The user-facing API is in package time:

 go
 
 
// src/time/sleep.go:117
func NewTimer(d Duration) *Timer {
	c := make(chan Time, 1)
	t := newTimer(when(d), 0, sendTime, c, syncTimer(c))
	t.C = c
	return t
}

and After is literally NewTimer(d).C:

 go
 
 
// src/time/sleep.go:174
func After(d Duration) <-chan Time {
	return NewTimer(d).C
}

The old drain rule — “if Stop returns false, you must drain the channel yourself” — is gone as of Go 1.23. The docs say so:

 go
 
 
// src/time/sleep.go:86
// For a chan-based timer created with NewTimer(d), as of Go 1.23,
// any receive from t.C after Stop has returned is guaranteed to block
// rather than receive a stale time value from before the Stop;
// if the program has not received from t.C already and the timer is
// running, Stop is guaranteed to return true.
// Before Go 1.23, the only safe way to use Stop was insert an extra
// <-t.C if Stop returned false to drain a potential stale value.
// See the [NewTimer] documentation for more details.
func (t *Timer) Stop() bool {

Note that this guarantee comes from the runtime’s seq mechanism, not from the channel’s buffer size — NewTimer still allocates a channel with capacity

What changed is that the runtime will not put a stale value in it.

Reset is the same story:

 go
 
 
// src/time/sleep.go:145
func (t *Timer) Reset(d Duration) bool {
	t.checkValid("Reset")
	w := when(d)
	return resetTimer(t, w, 0)
}

One more subtlety: the value sent on the channel is not “now” but “the time the timer was supposed to fire”, adjusted for how late the runtime got to it:

 go
 
 
// src/time/sleep.go:151
// sendTime does a non-blocking send of the current time on c.
func sendTime(c any, seq uintptr, delta int64) {
	// delta is how long ago the channel send was supposed to happen.
	// The current time can be arbitrarily far into the future, because the runtime
	// can delay a sendTime call until a goroutine tries to receive from
	// the channel. Subtract delta to go back to the old time that we
	// used to send.
	select {
	case c.(chan Time) <- Now().Add(Duration(-delta)):
	default:
	}
}

No thread per timer

To be explicit about the thing everybody assumes: there is no timerproc thread, no kernel timer per Go timer, and no interrupt per timer. The runtime has exactly one place where timers turn into wakeups — the netpoller’s timeout — plus a wakeNetPoller call when a timer is moved earlier.

 go
 
 
// src/runtime/proc.go:4020
// wakeNetPoller wakes up the thread sleeping in the network poller if it isn't
// going to wake up before the when argument; or it wakes an idle P to service
// timers and the network poller if there isn't one already.
func wakeNetPoller(when int64) {
	if sched.lastpoll.Load() == 0 {
		// In findRunnable we ensure that when polling the pollUntil
		// field is either zero or the time to which the current
		// poll is expected to run. This can have a spurious wakeup
		// but should never miss a wakeup.
		pollerPollUntil := sched.pollUntil.Load()
		if pollerPollUntil == 0 || pollerPollUntil > when {
			netpollBreak()
		}
	} else {
		// There are no threads in the network poller, try to get
		// one there so it can handle new timers.
		if GOOS != "plan9" { // Temporary workaround - see issue #42303.
			wakep()
		}
	}
}

netpollBreak writes to a pipe to wake the poller out of epoll_wait early. That is the closest thing Go has to a “timer interrupt”.

Trade-offs

What the user-space design buys you: a million timers cost a million heap entries and zero kernel objects. Timer creation is a mutex, a heap insert and a wakeNetPoller — no syscall in the common case.

What it costs:

Latency. A timer can fire late, by however long the scheduler takes to notice. The runtime mitigates this by waking the poller, but under load a timer can be late by milliseconds.
The scheduler has to poll. Every findRunnable checks the heap. That is cheap, but it is not free, and it happens on every scheduling round.
Coarse granularity for tickers. A ticker re-arms itself from the scheduled time, not from the actual fire time, so a ticker that is late does not drift — it catches up.
Lock contention on a hot heap. A program with a million timers on one P has a hot ts.mu.

What the old design bought you: precision, and a guarantee that a timer fires even if the Go scheduler is completely wedged. Go gave that up on purpose: the old timerproc thread existed on Windows and was a known source of latency problems.

Caveats

Timers are not exact. time.Sleep(1 * time.Millisecond) sleeps at least a millisecond, and possibly much longer if the scheduler is busy. Never build anything that depends on timer precision.

A stopped timer is not removed immediately. Stop marks a zombie; the heap entry is swept out later, when the owning P next touches its heap. A program that creates and stops millions of timers will accumulate heap entries until the next sweep.

Reset on a fired timer does not drain the channel for you in the general case — but since Go 1.23 it does not need to, because the runtime suppresses stale sends.

The channel is buffered with capacity 1 even though the docs describe the channel as synchronous. The guarantee comes from the runtime’s sequence numbering, not from the buffer.

AfterFunc runs your function in a new goroutine, and Stop returning false does not mean the function has finished:

 go
 
 
// src/time/sleep.go:79
// For a func-based timer created with [AfterFunc](d, f),
// if t.Stop returns false, then the timer has already expired
// and the function f has been started in its own goroutine;
// Stop does not wait for f to complete before returning.
// If the caller needs to know whether f is completed,
// it must coordinate with f explicitly.

A timer channel send is non-blocking. If nobody is receiving from t.C, the send is dropped:

 go
 
 
// src/time/sleep.go:158
	select {
	case c.(chan Time) <- Now().Add(Duration(-delta)):
	default:
	}

The value is not lost forever — the runtime remembers the timer is pending and delivers when someone blocks on the channel (blockTimerChan / maybeRunChan). But it does mean a timer channel is not a queue.

Timer functions run on the scheduler’s goroutine. The callback is invoked from timers.run, on whatever P owns the heap. If your callback blocks, you block the scheduler. The runtime says as much: “f must be a well-behaved function and not block”.

Wrapping up

The one thing to remember: a Go timer is a struct in a per-P min-heap. The scheduler checks that heap at the top of every scheduling round, the netpoller’s timeout doubles as the sleep-until-next-timer, and time.Sleep parks the goroutine on a timer attached to its own g.

The part people get wrong: the function names. There is no addtimer, deltimer, modtimer or checkTimers any more — they were replaced by timer.modify, timer.maybeAdd and timers.check, and the heap now stores a copy of the due time alongside each timer so that resets can be deferred.

What to go do about it: if you have a program that creates a timer per request, replace the timers with a single ticker and a priority queue, or with time.Timer reused via Reset. The per-timer cost is now a heap insert rather than a syscall, but it is still a heap insert, and it is still a lock.

Go's Scheduler Steals Half-Idle Cores: Work Stealing, Spinning and the Global Run Queue

Jay Hubbert — Wed, 21 Oct 2026 04:16:00 +0000

TL;DR: Go does not hand goroutines to threads and hope for the best. Every processor (P) owns a private 256-slot run queue plus one extra “run next” slot. When a P runs dry, its M turns into a spinning thief: it picks a victim P at random, grabs half that queue in one atomic batch, and keeps going until it finds work or gives up and parks on a futex.

Read the Go source (the 1.28 development tree) and the first thing you notice about src/runtime/proc.go is that it opens with a design essay, not code. That essay is the whole scheduler in English, and it is worth reading before anything else.

The G/M/P model

 go
 
 
// src/runtime/proc.go:24
// Goroutine scheduler
// The scheduler's job is to distribute ready-to-run goroutines over worker threads.
//
// The main concepts are:
// G - goroutine.
// M - worker thread, or machine.
// P - processor, a resource that is required to execute Go code.
// M must have an associated P to execute Go code, however it can be
// blocked or in a syscall w/o an associated P.
//
// Design doc at https://golang.org/s/go11sched.

Three nouns, one rule: a P is the licence to run Go code. An M (an OS thread) can exist without a P — blocked in a syscall, parked, inside cgo — but a goroutine only makes progress while some M holds a P. GOMAXPROCS is the number of Ps, which is the real parallelism limit.

The scheduler’s job is to distribute ready-to-run goroutines over worker threads. The interesting part is what happens when the distribution is uneven: one P with a deep queue, its neighbour with nothing at all.

The local run queue: a 256-slot ring

Each P owns a fixed-size ring buffer of goroutine pointers. Not a slice, not a linked list: a fixed array with a head and a tail index.

 go
 
 
// src/runtime/runtime2.go:804
	// Queue of runnable goroutines. Accessed without lock.
	runqhead uint32
	runqtail uint32
	runq [256]guintptr
	// runnext, if non-nil, is a runnable G that was ready'd by
	// the current G and should be run next instead of what's in
	// runq if there's time remaining in the running G's time
	// slice. It will inherit the time left in the current time
	// slice. If a set of goroutines is locked in a
	// communicate-and-wait pattern, this schedules that set as a
	// unit and eliminates the (potentially large) scheduling
	// latency that otherwise arises from adding the ready'd
	// goroutines to the end of the run queue.
	//
	// Note that while other P's may atomically CAS this to zero,
	// only the owner P can CAS it to a valid G.
	runnext guintptr

 text
 
 
 P0 P1
 +-----------------------------+ +-----------------------------+
 | runnext : *g (one slot) | | runnext : *g |
 +-----------------------------+ +-----------------------------+
 | runq[0..255] (ring) | | runq[0..255] |
 | head=12 tail=19 | | head=3 tail=3 (empty) |
 | G12 G13 G14 G15 G16 G17 G18 | | |
 +-----------------------------+ +-----------------------------+
 ^ ^
 | |
 owner P writes here a thief P may steal half of this

The fixed 256-entry size is deliberate: the owner P can put and take goroutines without taking any lock, because the queue is private and only the owner writes to it. Everything else uses atomics.

 go
 
 
// src/runtime/proc.go:7565
	if next {
	retryNext:
		oldnext := pp.runnext
		if !pp.runnext.cas(oldnext, guintptr(unsafe.Pointer(gp))) {
			goto retryNext
		}
		if oldnext == 0 {
			return
		}
		// Kick the old runnext out to the regular run queue.
		gp = oldnext.ptr()
	}

retry:
	h := atomic.LoadAcq(&pp.runqhead) // load-acquire, synchronize with consumers
	t := pp.runqtail
	if t-h < uint32(len(pp.runq)) {
		pp.runq[t%uint32(len(pp.runq))].set(gp)
		atomic.StoreRel(&pp.runqtail, t+1) // store-release, makes the item available for consumption
		return
	}
	if runqputslow(pp, gp, h, t) {
		return
	}
	// the queue is not full, now the put above must succeed
	goto retry

Note runnext: when a goroutine readies another one (an unbuffered channel handoff, say), the new goroutine goes into the runnext slot and inherits the remaining time slice of the current goroutine. That is how a communicate-and-wait pattern between two goroutines runs as a unit instead of paying a scheduling round trip.

Note also that the ring is a FIFO of readiness, not of goroutine creation, and runnext deliberately jumps the queue.

When the local queue is full, the P spills half of it plus the new goroutine onto the global queue (runqputslow), because moving 129 goroutines to the global queue under a lock is cheaper than moving them one at a time.

The global run queue

The global queue is a linked list of goroutines guarded by sched.lock. It exists for two reasons: overflow from a local queue, and goroutines made runnable by something that is not a P (a timer, the netpoller, a signal handler).

 go
 
 
// src/runtime/proc.go:7320
func globrunqput(gp *g) {
	assertLockHeld(&sched.lock)

	sched.runq.pushBack(gp)
}

A P takes from the global queue in batches, not one goroutine at a time, and dumps the rest into its own local queue:

 go
 
 
// src/runtime/proc.go:7364
func globrunqgetbatch(n int32) (gp *g, q gQueue) {
	assertLockHeld(&sched.lock)

	if sched.runq.size == 0 {
		return
	}

	n = min(n, sched.runq.size, sched.runq.size/gomaxprocs+1)

	gp = sched.runq.pop()
	n--

	for ; n > 0; n-- {
		gp1 := sched.runq.pop()
		q.pushBack(gp1)
	}
	return
}

The batch size is capped at size/gomaxprocs+1: with many Ps, each P takes only a small slice, so one P cannot hoard work while the others starve. One lock acquisition is amortised over many goroutines.

The global queue is not consulted on every scheduling decision — that would serialise everything on one mutex. It is checked once every 61 ticks:

 go
 
 
// src/runtime/proc.go:3459
	// Check the global runnable queue once in a while to ensure fairness.
	// Otherwise two goroutines can completely occupy the local runqueue
	// by constantly respawning each other.
	if pp.schedtick%61 == 0 && !sched.runq.empty() {
		lock(&sched.lock)
		gp := globrunqget()
		unlock(&sched.lock)
		if gp != nil {
			return gp, false, false
		}
	}

61 is a prime, so it does not line up with any common periodic pattern.

Stealing: half a queue at a time

Here is the core of the thing. A thief does not take one goroutine at a time from a victim — that would require the victim to release its queue slot by slot while the thief grabs them. Instead the thief grabs a batch with a single CAS on the victim’s head pointer, taking half of the queue.

 go
 
 
// src/runtime/proc.go:7733
func runqgrab(pp *p, batch *[256]guintptr, batchHead uint32, stealRunNextG bool) uint32 {
	for {
		h := atomic.LoadAcq(&pp.runqhead) // load-acquire, synchronize with other consumers
		t := atomic.LoadAcq(&pp.runqtail) // load-acquire, synchronize with the producer
		n := t - h
		n = n - n/2
		if n == 0 {
			if stealRunNextG {
				// Try to steal from pp.runnext.
				if next := pp.runnext; next != 0 {
					if pp.status == _Prunning {
						if mp := pp.m.ptr(); mp != nil {
							if gp := mp.curg; gp == nil || readgstatus(gp)&^_Gscan != _Gsyscall {
								// Sleep to ensure that pp isn't about to run the g
								// we are about to steal.

The n = n - n/2 line is the whole trick: half the victim’s queue, rounded down. The thief copies the entries into its own ring buffer and commits the theft with one CAS; if the CAS fails, the victim was concurrently taking work and the thief simply retries. No locks, no waiting, no corruption.

The steal takes only half so the victim keeps a share. A thief that took everything could ping-pong work back and forth with the owner and starve it. Half is a compromise: enough to be worth the trip, not enough to starve the victim.

 go
 
 
// src/runtime/proc.go:7806
func runqsteal(pp, p2 *p, stealRunNextG bool) *g {
	t := pp.runqtail
	n := runqgrab(p2, &pp.runq, t, stealRunNextG)
	if n == 0 {
		return nil
	}
	n--
	gp := pp.runq[(t+n)%uint32(len(pp.runq))].ptr()
	if n == 0 {
		return gp
	}
	h := atomic.LoadAcq(&pp.runqhead) // load-acquire, synchronize with consumers
	if t-h+n >= uint32(len(pp.runq)) {
		throw("runqsteal: runq overflow")
	}
	atomic.StoreRel(&pp.runqtail, t+n) // store-release, makes the item available for consumption
	return gp
}

One goroutine from the stolen batch is returned to run immediately; the rest stay in the thief’s local queue as warm work. Note that the stolen goroutines are copied into the thief’s own ring at its runqtail, and the new tail is published with a single store-release.

Picking a victim at random

Stealing from the same P every time would hammer one contended cache line, so the victim is chosen from a pseudo-random permutation of all Ps.

 go
 
 
// src/runtime/proc.go:3847
func stealWork(now int64) (gp *g, inheritTime bool, rnow, pollUntil int64, newWork bool) {
	pp := getg().m.p.ptr()

	ranTimer := false

	const stealTries = 4
	for i := 0; i < stealTries; i++ {
		stealTimersOrRunNextG := i == stealTries-1

		for enum := stealOrder.start(cheaprand()); !enum.done(); enum.next() {
			p2 := allp[enum.position()]
			if pp == p2 {
				continue
			}

Four rounds, each round walking a different permutation of all Ps. The permutation is not a shuffle; it is a walk with a stride coprime to the number of Ps, so every P is visited exactly once per round:

 go
 
 
// src/runtime/proc.go:8087
func (ord *randomOrder) reset(count uint32) {
	ord.count = count
	ord.coprimes = ord.coprimes[:0]
	for i := uint32(1); i <= count; i++ {
		if gcd(i, count) == 1 {
			ord.coprimes = append(ord.coprimes, i)
		}
	}
}

func (ord *randomOrder) start(i uint32) randomEnum {
	return randomEnum{
		count: ord.count,
		pos: i % ord.count,
		inc: ord.coprimes[i/ord.count%uint32(len(ord.coprimes))],
	}
}

Start at a random offset, walk with a random coprime stride. Every P gets visited, in a different order each time, without shuffling an array.

Spinning: the scheduler’s most unusual idea

When a P finds nothing in its own queue, it does not immediately park its thread. It first steals. And stealing means the M is “spinning” — burning CPU looking for work.

 go
 
 
// src/runtime/proc.go:3531
	// Spinning Ms: steal work from other Ps.
	//
	// Limit the number of spinning Ms to half the number of busy Ps.
	// This is necessary to prevent excessive CPU consumption when
	// GOMAXPROCS>>1 but the program parallelism is low.
	if mp.spinning || 2*sched.nmspinning.Load() < gomaxprocs-sched.npidle.Load() {
		if !mp.spinning {
			mp.becomeSpinning()
		}

		gp, inheritTime, tnow, w, newWork := stealWork(now)
		if gp != nil {
			// Successfully stole.
			return gp, inheritTime, false
		}

Note the limit: at most half as many spinning Ms as there are busy Ps. If your program is single-threaded but GOMAXPROCS is 64, the runtime will not let 64 threads burn CPU hunting for work that does not exist. It will let a couple.

Becoming a spinner is a counter increment, and leaving that state is a counter decrement plus a possible wake-up:

 go
 
 
// src/runtime/proc.go:4042
func resetspinning() {
	gp := getg()
	if !gp.m.spinning {
		throw("resetspinning: not a spinning m")
	}
	gp.m.spinning = false
	nmspinning := sched.nmspinning.Add(-1)
	if nmspinning < 0 {
		throw("findRunnable: negative nmspinning")
	}
	// M wakeup policy is deliberately somewhat conservative, so check if we
	// need to wakeup another P here. See "Worker thread parking/unparking"
	// comment at the top of the file for details.
	wakep()
}

So a thread that finds work stops spinning and, in doing so, may start a different thread spinning in its place. The invariant the runtime wants is roughly: if there is runnable work and an idle P, some thread should be looking for it.

The delicate dance before parking

Before a thread gives up and parks, it must release its P, and the moment it releases the P is a race: work could be submitted at the same instant. The runtime’s comment on this is the best piece of prose in the file.

 go
 
 
// src/runtime/proc.go:3641
	// Delicate dance: thread transitions from spinning to non-spinning
	// state, potentially concurrently with submission of new work. We must
	// drop nmspinning first and then check all sources again (with
	// #StoreLoad memory barrier in between). If we do it the other way
	// around, another thread can submit work after we've checked all
	// sources but before we drop nmspinning; as a result nobody will
	// unpark a thread to run the work.

The protocol: drop nmspinning, then re-check every source of work (global queue, all local queues, GC work, timers). If you find work, restore the spinning state and go around again. The ordering is not arbitrary; the comment explains exactly why the other order loses work.

Waking and parking

When goroutines are made runnable, someone has to notice. wakep() starts an M if there is an idle P and nobody is already spinning.

 go
 
 
// src/runtime/proc.go:3221
func wakep() {
	// Be conservative about spinning threads, only start one if none exist
	// already.
	if sched.nmspinning.Load() != 0 || !sched.nmspinning.CompareAndSwap(0, 1) {
		return
	}

	// Disable preemption until ownership of pp transfers to the next M in
	// startm. Otherwise preemption here would leave pp stuck waiting to
	// enter _Pgcstop.
	//
	// See preemption comment on acquirem in startm for more details.
	mp := acquirem()

	var pp *p
	lock(&sched.lock)
	pp, _ = pidlegetSpinning(0)
	if pp == nil {
		if sched.nmspinning.Add(-1) < 0 {
			throw("wakep: negative nmspinning")
		}
		unlock(&sched.lock)
		releasem(mp)
		return
	}

wakep is called from ready(), the function that moves a goroutine from waiting to runnable, and after a timer fires. The policy is deliberately conservative: if a spinner is already out there looking for work, do not wake another thread, because the spinner will probably find it.

When the spinner has nothing, it releases its P, puts itself on the idle M list, and blocks on a futex:

 go
 
 
// src/runtime/proc.go:2999
// Stops execution of the current m until new work is available.
// Returns with acquired P.
func stopm() {
	gp := getg()

	if gp.m.locks != 0 {
		throw("stopm holding locks")
	}
	if gp.m.p != 0 {
		throw("stopm holding p")
	}
	if gp.m.spinning {
		throw("stopm spinning")
	}

	lock(&sched.lock)
	mput(gp.m)
	unlock(&sched.lock)
	mPark()
	acquirep(gp.m.nextp.ptr())
	gp.m.nextp = 0
}

The three throw checks are the invariants: you cannot park while holding locks, holding a P, or while spinning. mPark sleeps on gp.m.park, a note — a one-shot futex word set by notewakeup with a single atomic exchange. Parking an M costs one futex sleep; waking it costs one atomic exchange and one futex wake. That is the price of not burning a core.

Timers are per-P too

Timers live in a per-P heap, and the scheduler checks them at the top of every round with pp.timers.check(0, nil), which returns the time of the next timer to fire. When a timer fires, the goroutines it readies land on a run queue — usually the local one of the P that ran the timer. sysmon, the runtime’s background thread, is what forces long-running goroutines to give up their P, and it deliberately refuses to sleep deeply while schedtrace is enabled:

 go
 
 
// src/runtime/proc.go:6571
		// sysmon should not enter deep sleep if schedtrace is enabled so that
		// it can print that information at the right time.
		//
		// It should also not enter deep sleep if there are any active P's so
		// that it can retake P's from syscalls, preempt long running G's, and
		// poll the network if all P's are busy for long stretches.

Trade-offs

What work stealing buys you. Automatic load balancing with no central queue in the hot path. A goroutine spawned on a busy P ends up on an idle one without anybody scheduling it explicitly. The local queue costs zero locks; the cost of balancing is paid only by threads that have nothing to do.

What it costs. Spinning threads burn CPU. The 2*nmspinning < gomaxprocs-npidle cap bounds that, but a program with GOMAXPROCS=64 and one runnable goroutine can still have a spinner or two turning in the dark. And because a steal grabs half a queue, the first steal on a busy machine copies a lot of pointers to gain one goroutine.

The global queue is a lock. Every overflow and every ready() from a non-P context goes through sched.lock. It is batched to keep that cheap, but it is the one global serialisation point in the design.

Half, not all. Stealing half the victim’s queue is a heuristic balancing “don’t starve the victim” against “don’t make stealing pointless”.

Caveats

The scheduler is not a fairness guarantee. runnext deliberately jumps the queue: a goroutine that readies another one gets that one run next, and inherits its time slice.

Work stealing is invisible to your code. There is no API to ask “which P will run this goroutine”, and there should not be. If you need affinity you need runtime.LockOSThread or a channel-based baton, and you are on your own.

GOMAXPROCS is the parallelism knob, not the thread count. The runtime creates more Ms than Ps (a blocked syscall needs one), and the number of spinning Ms is separately capped. GODEBUG=schedtrace=1 prints a line per scheduling round; scheddetail=1 adds per-P detail.

The global queue is checked only every 61 ticks. A goroutine readied onto the global queue by a non-P context may wait a little while. That is the price of not taking a global lock on every scheduling decision.

Ordering between a local ready() and a global one is undefined. If one goroutine readies another onto the local queue and a third is pushed to the global queue, which runs first is unspecified.

Wrapping up

The one thing to remember: Go’s scheduler is distributed by design. Each P owns a 256-slot lock-free run queue; work that overflows spills to a global queue; and idle Ps steal half of a busy neighbour’s queue in a single atomic batch, which is how a machine with one hot goroutine and one cold P still uses every core.

The part people get wrong: the scheduler spins before it parks. A thread with no work does not immediately sleep — it becomes a spinning thief, and only gives up after the “delicate dance” that re-checks every source of work to make sure it is not about to drop the last thread that could have picked up newly submitted work.

What to go do about it: set GOMAXPROCS deliberately, run with GODEBUG=schedtrace=1000,scheddetail=1 to watch the queues move, and if you suspect scheduler pathology, look first at the global queue size and the number of spinning Ms — both are visible in the schedtrace output.

entersyscall: How Go Hands the P to the Kernel and Gets It Back

Jay Hubbert — Wed, 21 Oct 2026 04:16:00 +0000

TL;DR: A goroutine that calls into the kernel doesn’t stop the world. The runtime flips the G’s status from _Grunning to _Gsyscall but keeps the P attached, so a short syscall costs almost nothing. If the call turns out to be long, sysmon retakes the P about 20 microseconds in, and when the syscall returns exitsyscall either grabs its P back or parks the goroutine on the run queue.

Every Go program makes system calls. Reads and writes on a socket, open, epoll_wait, futex sleeps — the Go runtime itself is nothing but system calls wrapped in a scheduler. And every one of those calls has to answer the same awkward question: the goroutine is now inside the kernel and cannot run Go code, so what happens to the CPU it was using?

The answer is the subject of this article, and it changed significantly in Go 1.14. The modern runtime does not release the P the moment a goroutine enters a syscall. It keeps it, marks the goroutine as _Gsyscall, and waits to see whether the call comes back quickly. Usually it does. When it doesn’t, sysmon reaches in and takes the P away.

This is a walk through how it works, based on reading the Go source (1.28 development tree, commit c0be4674865b).

Four states, one of them special

The runtime tracks a goroutine’s status in a single word. The interesting half of it is here:

 go
 
 
// src/runtime/runtime2.go:35
	// _Gidle means this goroutine was just allocated and has not
	// yet been initialized.
	_Gidle = iota // 0

	// _Grunnable means this goroutine is on a run queue. It is
	// not currently executing user code. The stack is not owned.
	_Grunnable // 1

	// _Grunning means this goroutine may execute user code. The
	// stack is owned by this goroutine. It is not on a run queue.
	// It is assigned an M (g.m is valid) and it usually has a P
	// (g.m.p is valid), but there are small windows of time where
	// it might not, namely upon entering and exiting _Gsyscall.
	_Grunning // 2

	// _Gsyscall means this goroutine is executing a system call.
	// It is not executing user code. The stack is owned by this
	// goroutine. It is not on a run queue. It is assigned an M.
	// It may have a P attached, but it does not own it. Code
	// executing in this state must not touch g.m.p.
	_Gsyscall // 3

	// _Gwaiting means this goroutine is blocked in the runtime.
	// It is not executing user code. It is not on a run queue,
	// but should be recorded somewhere (e.g., a channel wait
	// queue) so it can be ready()d when necessary. The stack is
	// not owned *except* that a channel operation may read or
	// write parts of the stack under the appropriate channel
	// lock. Otherwise, it is not safe to access the stack after a
	// goroutine enters _Gwaiting (e.g., it may get moved).
	_Gwaiting // 4

Four states that matter for this story:

Status	On a run queue?	Has an M?	Has a P?	Running Go code?
`_Grunnable`	yes	no	no	no
`_Grunning`	no	yes	usually yes	yes
`_Gsyscall`	no	yes	maybe, but does not own it	no
`_Gwaiting`	no (parked)	yes	no	no

The comment on _Grunning is the giveaway: “there are small windows of time where it might not [have a P], namely upon entering and exiting _Gsyscall”. That window is the entire subject of this article.

The g struct carries the status in an atomic field, and the runtime flips it with a compare-and-swap so that the GC, the tracer, and sysmon all see a consistent value:

 go
 
 
// src/runtime/proc.go:4737
	// As soon as we switch to _Gsyscall, we are in danger of losing our P.
	// We must not touch it after this point.
	//
	// Try to do a quick CAS to avoid calling into casgstatus in the common case.
	// If we have a bubble, we need to fall into casgstatus.
	if gp.bubble != nil || !gp.atomicstatus.CompareAndSwap(_Grunning, _Gsyscall) {
		casgstatus(gp, _Grunning, _Gsyscall)
	}

Note the wording: “in danger of losing our P”. Not “we lose our P”. The P is still there; we just aren’t allowed to touch it any more, because sysmon might take it away at any moment.

Where the entry point actually lives

The syscall package reaches into the runtime with a linkname:

 go
 
 
// src/syscall/syscall_linux.go:22
// Pull in entersyscall/exitsyscall for Syscall/Syscall6.
//
// Note that this can't be a push linkname because the runtime already has a
// nameless linkname to export to assembly here and in x/sys. Additionally,
// entersyscall fetches the caller PC and SP and thus can't have a wrapper
// inbetween.

//go:linkname runtime_entersyscall runtime.entersyscall
func runtime_entersyscall()

//go:linkname runtime_exitsyscall runtime.exitsyscall
func runtime_exitsyscall()

and syscall.Syscall brackets the trap with them:

 go
 
 
// src/syscall/syscall_linux.go:70
//go:uintptrkeepalive
//go:nosplit
//go:linkname Syscall
func Syscall(trap, a1, a2, a3 uintptr) (r1, r2 uintptr, err Errno) {
	runtime_entersyscall()
	// N.B. Calling RawSyscall here is unsafe with atomic coverage
	// instrumentation and race mode.
	//
	// Coverage instrumentation will add a sync/atomic call to RawSyscall.
	// Race mode will add race instrumentation to sync/atomic. Race
	// instrumentation requires a P, which we no longer have.
	//
	// RawSyscall6 is fine because it is implemented in assembly and thus
	// has no coverage instrumentation.
	//
	// This is typically not a problem in the runtime because cmd/go avoids
	// adding coverage instrumentation to the runtime in race mode.
	r1, r2, err = RawSyscall6(trap, a1, a2, a3, 0, 0, 0)
	runtime_exitsyscall()
	return
}

That comment is worth pausing on: “Race instrumentation requires a P, which we no longer have.” After runtime_entersyscall returns, the goroutine is in _Gsyscall and must not touch g.m.p — because another thread may be running with that P by now. The rule is enforced by convention and by review, not by the type system.

Note also that RawSyscall — the “raw” variant — does not bracket the trap. It’s the fast path for calls that provably cannot block (and it’s the one that breaks under race or coverage instrumentation).

What `entersyscall` actually does

The exported entry point is deliberately tiny:

 go
 
 
// src/runtime/proc.go:4780
//go:nosplit
//go:linkname entersyscall
func entersyscall() {
	// N.B. getcallerfp cannot be written directly as argument in the call
	// to reentersyscall because it forces spilling the other arguments to
	// the stack. This results in exceeding the nosplit stack requirements
	// on some platforms.
	fp := getcallerfp()
	reentersyscall(sys.GetCallerPC(), sys.GetCallerSP(), fp)
}

Everything real happens in reentersyscall, which is shared with the cgo callback path. The first thing it does after disabling preemption is take a snapshot of a counter:

 go
 
 
// src/runtime/proc.go:4686
	// Copy the syscalltick over so we can identify if the P got stolen later.
	gp.m.syscalltick = gp.m.p.ptr().syscalltick

	pp := gp.m.p.ptr()
	if pp.runSafePointFn != 0 {
		// runSafePointFn may stack split if run on this stack
		systemstack(runSafePointFn)
	}
	gp.m.oldp.set(pp)

	// Leave SP around for GC and traceback.
	save(pc, sp, bp)
	gp.syscallsp = sp
	gp.syscallpc = pc
	gp.syscallbp = bp

syscalltick is a per-P counter of “how many syscalls has this P been through”. The M copies it into gp.m.syscalltick. Later, on the way out, the runtime compares the two: if they still match, nobody stole the P.

Then it stores the stack pointer in three separate places — gp.syscallsp, gp.syscallpc, gp.syscallbp — because the GC and the tracer both need to be able to walk a goroutine’s stack while that goroutine is off in the kernel. The stack is still owned by the goroutine; it just isn’t running.

The two flavours: `entersyscall` and `entersyscallblock`

entersyscall assumes the call might come back soon, so it keeps the P. entersyscallblock is the “this is going to block for a long time” version, and it hands the P off immediately:

 go
 
 
// src/runtime/proc.go:4836
func entersyscallblock() {
	gp := getg()

	gp.m.locks++ // see comment in entersyscall
	gp.throwsplit = true
	gp.stackguard0 = stackPreempt // see comment in entersyscall
	gp.m.syscalltick = gp.m.p.ptr().syscalltick
	gp.m.p.ptr().syscalltick++

	addGSyscallNoP(gp.m) // We're going to give up our P.

	// Leave SP around for GC and traceback.
	pc := sys.GetCallerPC()
	sp := sys.GetCallerSP()
	bp := getcallerfp()
	save(pc, sp, bp)
	gp.syscallsp = gp.sched.sp
	gp.syscallpc = gp.sched.pc
	gp.syscallbp = gp.sched.bp

and then, crucially:

 go
 
 
// src/runtime/proc.go:4865
	// Once we switch to _Gsyscall, we can't safely touch
	// our P anymore, so we need to hand it off beforehand.
	// The tracer also needs to see the syscall before the P
	// handoff, so the order here must be (1) trace,
	// (2) handoff, (3) _Gsyscall switch.
	trace := traceAcquire()
	systemstack(func() {
		if trace.ok() {
			trace.GoSysCall()
		}
		handoffp(releasep())
	})
	// <--
	// Caution: we're in a small window where we are in _Grunning without a P.
	// -->
	if debugExtendGrunningNoP {
		usleep(10)
	}
	casgstatus(gp, _Grunning, _Gsyscall)

handoffp(releasep()) — release the P, then hand it to another M so it can start running goroutines right now. This is the “the P is handed off to another M” behaviour, and it only happens here, on the blocking path.

The comment above it is the whole design in four lines: the P must be given away before the status switch, because after the switch the goroutine is not allowed to touch g.m.p anymore. And there’s an explicit “small window where we are in _Grunning without a P” — flagged with a debugExtendGrunningNoP knob that inserts a 10-microsecond sleep to shake out code that assumes that window doesn’t exist.

The two paths side by side:

	`entersyscall`	`entersyscallblock`
Used by	`syscall.Syscall`, cgo calls	long blocking runtime calls
P released immediately?	no	yes, via `handoffp(releasep())`
P can be taken by `sysmon`?	yes, after ~20µs	already gone
Cost when the call returns fast	almost zero	a full P reacquisition

Coming back: `exitsyscall`

The exit path has three cases, and the first thing it does is flip the status optimistically:

 go
 
 
// src/runtime/proc.go:4928
func exitsyscall() {
	gp := getg()

	gp.m.locks++ // see comment in entersyscall
	if sys.GetCallerSP() > gp.syscallsp {
		throw("exitsyscall: syscall frame is no longer valid")
	}
	gp.waitsince = 0

	if sched.stopwait == freezeStopWait {
		// Wedge ourselves if there's an outstanding freezetheworld.
		// If we transition to running, we might end up with our traceback
		// being taken twice.
		systemstack(func() {
			lock(&deadlock)
			lock(&deadlock)
		})
	}

	// Optimistically assume we're going to keep running, and switch to running.
	// Before this point, our P wiring is not ours. Once we get past this point,
	// we can access our P if we have it, otherwise we lost it.

Then the three-way branch:

 go
 
 
// src/runtime/proc.go:4969
	// Grab and clear our old P.
	oldp := gp.m.oldp.ptr()
	gp.m.oldp.set(nil)

	// Check if we still have a P, and if not, try to acquire an idle P.
	pp := gp.m.p.ptr()
	if pp != nil {
		// Fast path: we still have our P. Just emit a syscall exit event.
		if trace := traceAcquire(); trace.ok() {
			systemstack(func() {
				// The truth is we truly never lost the P, but syscalltick
				// is used to indicate whether the P should be treated as
				// lost anyway. For example, when syscalltick is trashed by
				// dropm.
				//
				// TODO(mknyszek): Consider a more explicit mechanism for this.
				// Then syscalltick doesn't need to be trashed, and can be used
				// exclusively by sysmon for deciding when it's time to retake.
				if pp.syscalltick == gp.m.syscalltick {
					trace.GoSysExit(false)
				} else {

Case 1 — the fast path. gp.m.p is still non-nil, so the P was never taken away. The goroutine just resumes where it left off. The syscalltick comparison is how the runtime knows: if pp.syscalltick still equals the snapshot the M took on entry, nobody incremented it, so nobody stole the P.

Case 2 — the P was stolen. gp.m.p is nil, because sysmon took it while the goroutine was in the kernel. Now the goroutine needs a new one:

 go
 
 
// src/runtime/proc.go:5002
	} else {
		// Slow path: we lost our P. Try to get another one.
		systemstack(func() {
			// Try to get some other P.
			if pp := exitsyscallTryGetP(oldp); pp != nil {
				// Install the P.
				acquirepNoTrace(pp)

				// We're going to start running again, so emit all the relevant events.
				if trace := traceAcquire(); trace.ok() {
					trace.ProcStart()
					trace.GoSysExit(true)
					trace.GoStart()
					traceRelease(trace)
				}
			}
		})
		pp = gp.m.p.ptr()
	}

and exitsyscallTryGetP tries two things in order: steal the old P back (the thread that took it may be finished with it), or grab an idle P:

 go
 
 
// src/runtime/proc.go:5078
func exitsyscallTryGetP(oldp *p) *p {
	// Try to steal our old P back.
	if oldp != nil {
		if thread, ok := setBlockOnExitSyscall(oldp); ok {
			thread.takeP()
			decGSyscallNoP(getg().m) // We got a P for ourselves.
			thread.resume()
			return oldp
		}
	}

	// Try to get an idle P.
	if sched.pidle != 0 {
		lock(&sched.lock)
		pp, _ := pidleget(0)
		if pp != nil && sched.sysmonwait.Load() {
			sched.sysmonwait.Store(false)
			notewakeup(&sched.sysmonnote)
		}
		unlock(&sched.lock)
		if pp != nil {
			decGSyscallNoP(getg().m) // We got a P for ourselves.
			return pp
		}
	}
	return nil
}

Case 3 — no P available. The goroutine has to wait for one. This is the exitsyscallNoP path, and it goes through the scheduler like any other deschedule:

 go
 
 
// src/runtime/proc.go:5112
func exitsyscallNoP(gp *g) {
	traceExitingSyscall()
	trace := traceAcquire()
	casgstatus(gp, _Grunning, _Grunnable)
	traceExitedSyscall()
	if trace.ok() {
		// Write out syscall exit eagerly.
		//
		// It's important that we write this *after* we know whether we
		// lost our P or not (determined by exitsyscallfast).
		trace.GoSysExit(true)
		traceRelease(trace)
	}
	decGSyscallNoP(getg().m)
	dropg()

Note casgstatus(gp, _Grunning, _Grunnable) — the goroutine becomes runnable rather than running, and dropg() detaches it from its M:

 go
 
 
// src/runtime/proc.go:4261
func dropg() {
	gp := getg()

	setMNoWB(&gp.m.curg.m, nil)
	setGNoWB(&gp.m.curg, nil)
}

So the full state machine of a goroutine making a syscall looks like this:

 text
 
 
 entersyscall() exitsyscall()
 _Grunning ────────────────────────▶ _Gsyscall ──────────────▶ _Grunning
 │ ▲
 │ sysmon retake │ got a P
 ▼ │
 (P gone) _Grunnable
 │
 │ schedule()
 ▼
 (later) _Grunning

The 20-microsecond retake

So who takes the P away? sysmon, the runtime’s background thread, in a loop called retake:

 go
 
 
// src/runtime/proc.go:6693
func retake(now int64) uint32 {
	n := 0
	// Prevent allp slice changes. This lock will be completely
	// uncontended unless we're already stopping the world.
	lock(&allpLock)
	// We can't use a range loop over allp because we may
	// temporarily drop the allpLock. Hence, we need to re-fetch
	// allp each time around the loop.
	for i := 0; i < len(allp); i++ {
		// Quickly filter out non-running Ps. Running Ps are either
		// in a syscall or are actually executing. Idle Ps don't
		// need to be retaken.
		//
		// This is best-effort, so it's OK that it's racy. Our target
		// is to retake Ps that have been running or in a syscall for
		// a long time (milliseconds), so the state has plenty of time
		// to stabilize.
		pp := allp[i]
		if pp == nil || atomic.Load(&pp.status) != _Prunning {
			// pp can be nil if procresize has grown
			// allp but not yet created new Ps.
			continue
		}

and then the syscall-specific part:

 go
 
 
// src/runtime/proc.go:6755
		// Retake the P if it's there for more than 1 sysmon tick (at least 20us).
		if syst := int64(pp.syscalltick); !sysretake && int64(pd.syscalltick) != syst {
			pd.syscalltick = uint32(syst)
			pd.syscallwhen = now
			thread.resume()
			goto done
		}

		// On the one hand we don't want to retake Ps if there is no other work to do,
		// but on the other hand we want to retake them eventually
		// because they can prevent the sysmon thread from deep sleep.
		if runqempty(pp) && sched.nmspinning.Load()+sched.npidle.Load() > 0 && pd.syscallwhen+10*1000*1000 > now {
			thread.resume()
			goto done
		}

		// Take the P. Note: because we have the scan bit, the goroutine
		// is at worst stuck spinning in exitsyscall.
		thread.takeP()
		thread.resume()
		n++

		// Handoff the P for some other thread to run it.
		handoffp(pp)

Read that carefully, because the logic is subtle:

First observation of a changed syscalltick records syscallwhen = now and does not retake. The syscall might be about to return.
Second observation with the same syscalltick means the syscall is still running. Now the retake is allowed — unless the P’s local run queue is empty and other Ms are idle or spinning and we’re still within 10 milliseconds of the first observation. In that case, there’s nothing useful to run, so there’s no reason to yank the P away from the syscall.

“More than 1 sysmon tick (at least 20us)” is the key phrase. sysmon starts its sleep at 20 microseconds:

 go
 
 
// src/runtime/proc.go:6560
	for {
		if idle == 0 { // start with 20us sleep...
			delay = 20
		} else if idle > 50 { // start doubling the sleep after 1ms...
			delay *= 2
		}
		if delay > 10*1000 { // up to 10ms
			delay = 10 * 1000
		}
		usleep(delay)

So the practical sequence is: syscall enters → sysmon notices within ~20µs → next tick, another ~20µs later, the P is taken. Roughly 20 to 60 microseconds of “the P is hostage” time, and only if there’s work to be done.

The actual theft is thread.takeP(), which is a state transition on the blocked thread:

 go
 
 
// src/runtime/proc.go:6871
// takeP unwires the P attached to the syscalling thread
// and moves it into the _Pidle state.
func (s syscallingThread) takeP() {
	s.releaseP(_Pidle)
}

and releaseP is where the syscalltick bump happens, which is what tells the returning goroutine “you lost your P”:

 go
 
 
// src/runtime/proc.go:6877
// releaseP unwires the P from the syscalling thread, moving
// it to the provided state. Callers should prefer to use
// takeP and gcstopP.
func (s syscallingThread) releaseP(state uint32) {
	if state != _Pidle && state != _Pgcstop {
		throw("attempted to release P into a bad state")
	}
	trace := traceAcquire()
	s.pp.m = 0
	s.mp.p = 0
	atomic.Store(&s.pp.status, state)
	if trace.ok() {
		trace.ProcSteal(s.pp)
		traceRelease(trace)
	}
	addGSyscallNoP(s.mp)
	s.pp.syscalltick++
}

s.pp.syscalltick++ — one increment, and the syscalling goroutine’s snapshot no longer matches. When it comes back, exitsyscall takes the slow path.

The other way to lose a P: `dropm`

There’s one more wrinkle. When a goroutine’s M is destroyed (which happens when a thread exits from inside a syscall), the runtime deliberately corrupts the syscalltick so that the tracer doesn’t get confused:

 go
 
 
// src/runtime/proc.go:2623
	// Trash syscalltick so that it doesn't line up with mp.old.syscalltick anymore.
	//
	// In the new tracer, we model needm and dropm and a goroutine being created and
	// destroyed respectively. The m then might get reused with a different procid but
	// still with a reference to oldp, and still with the same syscalltick. The next
	// time a G is "created" in needm, it'll return and quietly reacquire its P from a
	// different m with a different procid, which will confuse the trace parser. By
	// trashing syscalltick, we ensure that it'll appear as if we lost the P to the
	// tracer parser and that we just reacquired it.
	//
	// Trash the value by decrementing because that gets us as far away from the value
	// the syscall exit code expects as possible. Setting to zero is risky because
	// syscalltick could already be zero (and in fact, is initialized to zero).
	mp.syscalltick--

Decrementing rather than zeroing, because zero is a legitimate value. Nice touch.

`GOMAXPROCS` and the whole thing

Now the part that surprises people. With GOMAXPROCS=1, there is exactly one P. If that P is attached to a goroutine blocked in a syscall, nothing else can run — until sysmon takes the P away and hands it to another thread.

That’s the entire mechanism that keeps a single-CPU Go program alive while one of its goroutines is stuck in a blocking syscall. The cost is up to a few tens of microseconds of stall.

The same logic applies at GOMAXPROCS=8. A blocking syscall doesn’t take the machine down, but it does take a P out of circulation for up to ~20µs before sysmon frees it. If your program has, say, 8 goroutines each occasionally making a 5-millisecond syscall, you can end up with all 8 P’s momentarily attached to syscall-blocked goroutines, and the program stalls until sysmon notices.

This is also why the runtime distinguishes entersyscall from entersyscallblock. If you know your call is going to block for a long time, entersyscallblock gives the P up immediately instead of making sysmon find out in 20µs. The runtime itself uses entersyscallblock for exactly the calls where it knows the answer: waiting on a futex, parking on a channel, waiting for a network poll.

The runtime’s own syscalls don’t go through any of this

One more thing worth knowing: most system calls made by the runtime itself don’t touch the scheduler at all. futexsleep, for example:

 go
 
 
// src/runtime/os_linux.go:65
//go:nosplit
func futexsleep(addr *uint32, val uint32, ns int64) {
	// Some Linux kernels have a bug where futex of
	// FUTEX_WAIT returns an internal error code
	// as an errno. Libpthread ignores the return value
	// here, and so can we: as it says a few lines up,
	// spurious wakeups are allowed.
	if ns < 0 {
		futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, nil, nil, 0)
		return
	}

	var ts timespec
	ts.setNsec(ns)
	futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, &ts, nil, 0)
}

No entersyscall, no exitsyscall. The runtime’s comment on reentersyscall explains why: “This is called only from the go syscall library and cgocall, not from the low-level system calls used by the runtime.” A futex wait inside the scheduler is a blocking operation by design, and the scheduler already knows it’s blocked — it doesn’t need the syscall bookkeeping to tell it.

Trade-offs

What the “keep the P” design buys you: the common case is nearly free. A read() that returns in 5 microseconds causes zero scheduler churn: no P handoff, no M wakeup, no run-queue manipulation. The goroutine is still _Grunning from the scheduler’s point of view except for a CAS on a status word. Under the old design (release the P on entry, reacquire on exit) every syscall was a full P handoff.

What it costs: a syscall that blocks pins a P for up to a few tens of microseconds. At high syscall rates across many goroutines this shows up as scheduling latency, and the fix is entersyscallblock semantics — but only the runtime can call that.

Why the two-tier design (syscalltick) exists: a P can be lost in three different ways (sysmon retake, dropm, explicit release). Rather than have an explicit “I lost my P” flag that has to be kept consistent across all of them, the runtime piggybacks on a counter that’s already being incremented for other reasons. The TODO(mknyszek) in the source admits this is a hack: “Consider a more explicit mechanism for this.”

Why the retake is gated on “is there work to do”: if the program is idle, yanking the P out of a syscall buys nothing and costs a thread wakeup. The runqempty(pp) && nmspinning+npidle > 0 check keeps the common idle case completely hands-off.

Caveats

The names in older articles are stale. exitsyscallfast and keepsyscall no longer exist as functions. exitsyscallfast survives only in a comment (src/runtime/proc.go:5121, “determined by exitsyscallfast”), and keepsyscall does not appear in the tree at all. The old exitsyscallfast returned a bool telling the caller whether it still had its P; that logic is now inline in exitsyscall itself.

entersyscall and exitsyscall are not public API. They’re exported via //go:linkname for syscall, x/sys, and a small list of very large users (the source calls gvisor.dev/gvisor out by name as a “notable member of the hall of shame”). The comment is explicit: “Do not remove or change the type signature. See go.dev/issue/67401.” But you are calling into runtime internals with no compatibility promise.

//go:nosplit is load-bearing. entersyscall cannot split the stack, because it must be able to return immediately and because the runtime can’t know which of the uintptr arguments to the syscall are really pointers into the stack. The slow path therefore has to use systemstack.

The Linux os_linux.go has no entersyscall in it. Older Go trees had per-OS entersyscall/exitsyscall wrappers in os_linux.go. In the 1.28 tree those are gone; the generic implementations in proc.go are used, and the syscall package pulls them in by linkname. If you’re reading an old blog post about os_linux.go, that’s the file to look for.

Latency numbers are approximate. “About 20 microseconds” is a floor, not a guarantee. sysmon’s sleep doubles once the system has been idle for more than a millisecond, so a program that goes quiet and then makes a blocking syscall can see a much longer delay before sysmon retakes the P.

Wrapping up

The one thing to remember: entersyscall does not release the P. It flips the goroutine to _Gsyscall and leaves the P attached, so a fast syscall costs essentially nothing; sysmon takes the P away about 20 microseconds in only if the call is slow and there’s work waiting.

The part people get wrong: that a blocking syscall takes a core away from your program for its whole duration. It doesn’t. It takes a P away for tens of microseconds, and then sysmon hands that P to another thread while the syscall continues in the kernel.

What to go do about it: if you’re writing code that blocks in a syscall and you care about latency, count how many of your goroutines are in _Gsyscall at once. GODEBUG=schedtrace=1000,scheddetail=1 will show you the scheduler’s view, and a program that regularly has every P attached to a syscall-blocked goroutine is a program that’s paying the retake penalty on every call.

Scheduler on Opensource internals

Go's Timers: A Per-P Heap and a Timer That Never Blocks a Thread

The shape of the problem

The timer struct

The per-P heap

Three state bits

Creating a timer

Sleeping

The scheduler’s timer check

Stopping and resetting without races

time.After, NewTimer, Reset and the drain rule

No thread per timer

Trade-offs

Caveats

Wrapping up

Further reading

Go's Scheduler Steals Half-Idle Cores: Work Stealing, Spinning and the Global Run Queue

The G/M/P model

The local run queue: a 256-slot ring

The global run queue

Stealing: half a queue at a time

Picking a victim at random

Spinning: the scheduler’s most unusual idea

The delicate dance before parking

Waking and parking

Timers are per-P too

Trade-offs

Caveats

Wrapping up

Further reading

entersyscall: How Go Hands the P to the Kernel and Gets It Back

Four states, one of them special

Where the entry point actually lives

What entersyscall actually does

The two flavours: entersyscall and entersyscallblock

Coming back: exitsyscall

The 20-microsecond retake

The other way to lose a P: dropm

GOMAXPROCS and the whole thing

The runtime’s own syscalls don’t go through any of this

Trade-offs

Caveats

Wrapping up

Further reading

What `entersyscall` actually does

The two flavours: `entersyscall` and `entersyscallblock`

Coming back: `exitsyscall`

The other way to lose a P: `dropm`

`GOMAXPROCS` and the whole thing