entersyscall: How Go Hands the P to the Kernel and Gets It Back

TL;DR: A goroutine that calls into the kernel doesn’t stop the world. The runtime flips the G’s status from _Grunning to _Gsyscall but keeps the P attached, so a short syscall costs almost nothing. If the call turns out to be long, sysmon retakes the P about 20 microseconds in, and when the syscall returns exitsyscall either grabs its P back or parks the goroutine on the run queue.

Every Go program makes system calls. Reads and writes on a socket, open, epoll_wait, futex sleeps — the Go runtime itself is nothing but system calls wrapped in a scheduler. And every one of those calls has to answer the same awkward question: the goroutine is now inside the kernel and cannot run Go code, so what happens to the CPU it was using?

The answer is the subject of this article, and it changed significantly in Go 1.14. The modern runtime does not release the P the moment a goroutine enters a syscall. It keeps it, marks the goroutine as _Gsyscall, and waits to see whether the call comes back quickly. Usually it does. When it doesn’t, sysmon reaches in and takes the P away.

This is a walk through how it works, based on reading the Go source (1.28 development tree, commit c0be4674865b).

Four states, one of them special

The runtime tracks a goroutine’s status in a single word. The interesting half of it is here:

go
// src/runtime/runtime2.go:35
	// _Gidle means this goroutine was just allocated and has not
	// yet been initialized.
	_Gidle = iota // 0

	// _Grunnable means this goroutine is on a run queue. It is
	// not currently executing user code. The stack is not owned.
	_Grunnable // 1

	// _Grunning means this goroutine may execute user code. The
	// stack is owned by this goroutine. It is not on a run queue.
	// It is assigned an M (g.m is valid) and it usually has a P
	// (g.m.p is valid), but there are small windows of time where
	// it might not, namely upon entering and exiting _Gsyscall.
	_Grunning // 2

	// _Gsyscall means this goroutine is executing a system call.
	// It is not executing user code. The stack is owned by this
	// goroutine. It is not on a run queue. It is assigned an M.
	// It may have a P attached, but it does not own it. Code
	// executing in this state must not touch g.m.p.
	_Gsyscall // 3

	// _Gwaiting means this goroutine is blocked in the runtime.
	// It is not executing user code. It is not on a run queue,
	// but should be recorded somewhere (e.g., a channel wait
	// queue) so it can be ready()d when necessary. The stack is
	// not owned *except* that a channel operation may read or
	// write parts of the stack under the appropriate channel
	// lock. Otherwise, it is not safe to access the stack after a
	// goroutine enters _Gwaiting (e.g., it may get moved).
	_Gwaiting // 4

Four states that matter for this story:

Status	On a run queue?	Has an M?	Has a P?	Running Go code?
`_Grunnable`	yes	no	no	no
`_Grunning`	no	yes	usually yes	yes
`_Gsyscall`	no	yes	maybe, but does not own it	no
`_Gwaiting`	no (parked)	yes	no	no

The comment on _Grunning is the giveaway: “there are small windows of time where it might not [have a P], namely upon entering and exiting _Gsyscall”. That window is the entire subject of this article.

The g struct carries the status in an atomic field, and the runtime flips it with a compare-and-swap so that the GC, the tracer, and sysmon all see a consistent value:

go
// src/runtime/proc.go:4737
	// As soon as we switch to _Gsyscall, we are in danger of losing our P.
	// We must not touch it after this point.
	//
	// Try to do a quick CAS to avoid calling into casgstatus in the common case.
	// If we have a bubble, we need to fall into casgstatus.
	if gp.bubble != nil || !gp.atomicstatus.CompareAndSwap(_Grunning, _Gsyscall) {
		casgstatus(gp, _Grunning, _Gsyscall)
	}

Note the wording: “in danger of losing our P”. Not “we lose our P”. The P is still there; we just aren’t allowed to touch it any more, because sysmon might take it away at any moment.

Where the entry point actually lives

The syscall package reaches into the runtime with a linkname:

go
// src/syscall/syscall_linux.go:22
// Pull in entersyscall/exitsyscall for Syscall/Syscall6.
//
// Note that this can't be a push linkname because the runtime already has a
// nameless linkname to export to assembly here and in x/sys. Additionally,
// entersyscall fetches the caller PC and SP and thus can't have a wrapper
// inbetween.

//go:linkname runtime_entersyscall runtime.entersyscall
func runtime_entersyscall()

//go:linkname runtime_exitsyscall runtime.exitsyscall
func runtime_exitsyscall()

and syscall.Syscall brackets the trap with them:

go
// src/syscall/syscall_linux.go:70
//go:uintptrkeepalive
//go:nosplit
//go:linkname Syscall
func Syscall(trap, a1, a2, a3 uintptr) (r1, r2 uintptr, err Errno) {
	runtime_entersyscall()
	// N.B. Calling RawSyscall here is unsafe with atomic coverage
	// instrumentation and race mode.
	//
	// Coverage instrumentation will add a sync/atomic call to RawSyscall.
	// Race mode will add race instrumentation to sync/atomic. Race
	// instrumentation requires a P, which we no longer have.
	//
	// RawSyscall6 is fine because it is implemented in assembly and thus
	// has no coverage instrumentation.
	//
	// This is typically not a problem in the runtime because cmd/go avoids
	// adding coverage instrumentation to the runtime in race mode.
	r1, r2, err = RawSyscall6(trap, a1, a2, a3, 0, 0, 0)
	runtime_exitsyscall()
	return
}

That comment is worth pausing on: “Race instrumentation requires a P, which we no longer have.” After runtime_entersyscall returns, the goroutine is in _Gsyscall and must not touch g.m.p — because another thread may be running with that P by now. The rule is enforced by convention and by review, not by the type system.

Note also that RawSyscall — the “raw” variant — does not bracket the trap. It’s the fast path for calls that provably cannot block (and it’s the one that breaks under race or coverage instrumentation).

What `entersyscall` actually does

The exported entry point is deliberately tiny:

go
// src/runtime/proc.go:4780
//go:nosplit
//go:linkname entersyscall
func entersyscall() {
	// N.B. getcallerfp cannot be written directly as argument in the call
	// to reentersyscall because it forces spilling the other arguments to
	// the stack. This results in exceeding the nosplit stack requirements
	// on some platforms.
	fp := getcallerfp()
	reentersyscall(sys.GetCallerPC(), sys.GetCallerSP(), fp)
}

Everything real happens in reentersyscall, which is shared with the cgo callback path. The first thing it does after disabling preemption is take a snapshot of a counter:

go
// src/runtime/proc.go:4686
	// Copy the syscalltick over so we can identify if the P got stolen later.
	gp.m.syscalltick = gp.m.p.ptr().syscalltick

	pp := gp.m.p.ptr()
	if pp.runSafePointFn != 0 {
		// runSafePointFn may stack split if run on this stack
		systemstack(runSafePointFn)
	}
	gp.m.oldp.set(pp)

	// Leave SP around for GC and traceback.
	save(pc, sp, bp)
	gp.syscallsp = sp
	gp.syscallpc = pc
	gp.syscallbp = bp

syscalltick is a per-P counter of “how many syscalls has this P been through”. The M copies it into gp.m.syscalltick. Later, on the way out, the runtime compares the two: if they still match, nobody stole the P.

Then it stores the stack pointer in three separate places — gp.syscallsp, gp.syscallpc, gp.syscallbp — because the GC and the tracer both need to be able to walk a goroutine’s stack while that goroutine is off in the kernel. The stack is still owned by the goroutine; it just isn’t running.

The two flavours: `entersyscall` and `entersyscallblock`

entersyscall assumes the call might come back soon, so it keeps the P. entersyscallblock is the “this is going to block for a long time” version, and it hands the P off immediately:

go
// src/runtime/proc.go:4836
func entersyscallblock() {
	gp := getg()

	gp.m.locks++ // see comment in entersyscall
	gp.throwsplit = true
	gp.stackguard0 = stackPreempt // see comment in entersyscall
	gp.m.syscalltick = gp.m.p.ptr().syscalltick
	gp.m.p.ptr().syscalltick++

	addGSyscallNoP(gp.m) // We're going to give up our P.

	// Leave SP around for GC and traceback.
	pc := sys.GetCallerPC()
	sp := sys.GetCallerSP()
	bp := getcallerfp()
	save(pc, sp, bp)
	gp.syscallsp = gp.sched.sp
	gp.syscallpc = gp.sched.pc
	gp.syscallbp = gp.sched.bp

and then, crucially:

go
// src/runtime/proc.go:4865
	// Once we switch to _Gsyscall, we can't safely touch
	// our P anymore, so we need to hand it off beforehand.
	// The tracer also needs to see the syscall before the P
	// handoff, so the order here must be (1) trace,
	// (2) handoff, (3) _Gsyscall switch.
	trace := traceAcquire()
	systemstack(func() {
		if trace.ok() {
			trace.GoSysCall()
		}
		handoffp(releasep())
	})
	// <--
	// Caution: we're in a small window where we are in _Grunning without a P.
	// -->
	if debugExtendGrunningNoP {
		usleep(10)
	}
	casgstatus(gp, _Grunning, _Gsyscall)

handoffp(releasep()) — release the P, then hand it to another M so it can start running goroutines right now. This is the “the P is handed off to another M” behaviour, and it only happens here, on the blocking path.

The comment above it is the whole design in four lines: the P must be given away before the status switch, because after the switch the goroutine is not allowed to touch g.m.p anymore. And there’s an explicit “small window where we are in _Grunning without a P” — flagged with a debugExtendGrunningNoP knob that inserts a 10-microsecond sleep to shake out code that assumes that window doesn’t exist.

The two paths side by side:

	`entersyscall`	`entersyscallblock`
Used by	`syscall.Syscall`, cgo calls	long blocking runtime calls
P released immediately?	no	yes, via `handoffp(releasep())`
P can be taken by `sysmon`?	yes, after ~20µs	already gone
Cost when the call returns fast	almost zero	a full P reacquisition

Coming back: `exitsyscall`

The exit path has three cases, and the first thing it does is flip the status optimistically:

go
// src/runtime/proc.go:4928
func exitsyscall() {
	gp := getg()

	gp.m.locks++ // see comment in entersyscall
	if sys.GetCallerSP() > gp.syscallsp {
		throw("exitsyscall: syscall frame is no longer valid")
	}
	gp.waitsince = 0

	if sched.stopwait == freezeStopWait {
		// Wedge ourselves if there's an outstanding freezetheworld.
		// If we transition to running, we might end up with our traceback
		// being taken twice.
		systemstack(func() {
			lock(&deadlock)
			lock(&deadlock)
		})
	}

	// Optimistically assume we're going to keep running, and switch to running.
	// Before this point, our P wiring is not ours. Once we get past this point,
	// we can access our P if we have it, otherwise we lost it.

Then the three-way branch:

go
// src/runtime/proc.go:4969
	// Grab and clear our old P.
	oldp := gp.m.oldp.ptr()
	gp.m.oldp.set(nil)

	// Check if we still have a P, and if not, try to acquire an idle P.
	pp := gp.m.p.ptr()
	if pp != nil {
		// Fast path: we still have our P. Just emit a syscall exit event.
		if trace := traceAcquire(); trace.ok() {
			systemstack(func() {
				// The truth is we truly never lost the P, but syscalltick
				// is used to indicate whether the P should be treated as
				// lost anyway. For example, when syscalltick is trashed by
				// dropm.
				//
				// TODO(mknyszek): Consider a more explicit mechanism for this.
				// Then syscalltick doesn't need to be trashed, and can be used
				// exclusively by sysmon for deciding when it's time to retake.
				if pp.syscalltick == gp.m.syscalltick {
					trace.GoSysExit(false)
				} else {

Case 1 — the fast path. gp.m.p is still non-nil, so the P was never taken away. The goroutine just resumes where it left off. The syscalltick comparison is how the runtime knows: if pp.syscalltick still equals the snapshot the M took on entry, nobody incremented it, so nobody stole the P.

Case 2 — the P was stolen. gp.m.p is nil, because sysmon took it while the goroutine was in the kernel. Now the goroutine needs a new one:

go
// src/runtime/proc.go:5002
	} else {
		// Slow path: we lost our P. Try to get another one.
		systemstack(func() {
			// Try to get some other P.
			if pp := exitsyscallTryGetP(oldp); pp != nil {
				// Install the P.
				acquirepNoTrace(pp)

				// We're going to start running again, so emit all the relevant events.
				if trace := traceAcquire(); trace.ok() {
					trace.ProcStart()
					trace.GoSysExit(true)
					trace.GoStart()
					traceRelease(trace)
				}
			}
		})
		pp = gp.m.p.ptr()
	}

and exitsyscallTryGetP tries two things in order: steal the old P back (the thread that took it may be finished with it), or grab an idle P:

go
// src/runtime/proc.go:5078
func exitsyscallTryGetP(oldp *p) *p {
	// Try to steal our old P back.
	if oldp != nil {
		if thread, ok := setBlockOnExitSyscall(oldp); ok {
			thread.takeP()
			decGSyscallNoP(getg().m) // We got a P for ourselves.
			thread.resume()
			return oldp
		}
	}

	// Try to get an idle P.
	if sched.pidle != 0 {
		lock(&sched.lock)
		pp, _ := pidleget(0)
		if pp != nil && sched.sysmonwait.Load() {
			sched.sysmonwait.Store(false)
			notewakeup(&sched.sysmonnote)
		}
		unlock(&sched.lock)
		if pp != nil {
			decGSyscallNoP(getg().m) // We got a P for ourselves.
			return pp
		}
	}
	return nil
}

Case 3 — no P available. The goroutine has to wait for one. This is the exitsyscallNoP path, and it goes through the scheduler like any other deschedule:

go
// src/runtime/proc.go:5112
func exitsyscallNoP(gp *g) {
	traceExitingSyscall()
	trace := traceAcquire()
	casgstatus(gp, _Grunning, _Grunnable)
	traceExitedSyscall()
	if trace.ok() {
		// Write out syscall exit eagerly.
		//
		// It's important that we write this *after* we know whether we
		// lost our P or not (determined by exitsyscallfast).
		trace.GoSysExit(true)
		traceRelease(trace)
	}
	decGSyscallNoP(getg().m)
	dropg()

Note casgstatus(gp, _Grunning, _Grunnable) — the goroutine becomes runnable rather than running, and dropg() detaches it from its M:

go
// src/runtime/proc.go:4261
func dropg() {
	gp := getg()

	setMNoWB(&gp.m.curg.m, nil)
	setGNoWB(&gp.m.curg, nil)
}

So the full state machine of a goroutine making a syscall looks like this:

text
                    entersyscall()               exitsyscall()
   _Grunning  ────────────────────────▶ _Gsyscall ──────────────▶ _Grunning
                                          │                          ▲
                                          │  sysmon retake          │ got a P
                                          ▼                          │
                                     (P gone)                  _Grunnable
                                                                  │
                                                                  │ schedule()
                                                                  ▼
                                                             (later) _Grunning

The 20-microsecond retake

So who takes the P away? sysmon, the runtime’s background thread, in a loop called retake:

go
// src/runtime/proc.go:6693
func retake(now int64) uint32 {
	n := 0
	// Prevent allp slice changes. This lock will be completely
	// uncontended unless we're already stopping the world.
	lock(&allpLock)
	// We can't use a range loop over allp because we may
	// temporarily drop the allpLock. Hence, we need to re-fetch
	// allp each time around the loop.
	for i := 0; i < len(allp); i++ {
		// Quickly filter out non-running Ps. Running Ps are either
		// in a syscall or are actually executing. Idle Ps don't
		// need to be retaken.
		//
		// This is best-effort, so it's OK that it's racy. Our target
		// is to retake Ps that have been running or in a syscall for
		// a long time (milliseconds), so the state has plenty of time
		// to stabilize.
		pp := allp[i]
		if pp == nil || atomic.Load(&pp.status) != _Prunning {
			// pp can be nil if procresize has grown
			// allp but not yet created new Ps.
			continue
		}

and then the syscall-specific part:

go
// src/runtime/proc.go:6755
		// Retake the P if it's there for more than 1 sysmon tick (at least 20us).
		if syst := int64(pp.syscalltick); !sysretake && int64(pd.syscalltick) != syst {
			pd.syscalltick = uint32(syst)
			pd.syscallwhen = now
			thread.resume()
			goto done
		}

		// On the one hand we don't want to retake Ps if there is no other work to do,
		// but on the other hand we want to retake them eventually
		// because they can prevent the sysmon thread from deep sleep.
		if runqempty(pp) && sched.nmspinning.Load()+sched.npidle.Load() > 0 && pd.syscallwhen+10*1000*1000 > now {
			thread.resume()
			goto done
		}

		// Take the P. Note: because we have the scan bit, the goroutine
		// is at worst stuck spinning in exitsyscall.
		thread.takeP()
		thread.resume()
		n++

		// Handoff the P for some other thread to run it.
		handoffp(pp)

Read that carefully, because the logic is subtle:

First observation of a changed syscalltick records syscallwhen = now and does not retake. The syscall might be about to return.
Second observation with the same syscalltick means the syscall is still running. Now the retake is allowed — unless the P’s local run queue is empty and other Ms are idle or spinning and we’re still within 10 milliseconds of the first observation. In that case, there’s nothing useful to run, so there’s no reason to yank the P away from the syscall.

“More than 1 sysmon tick (at least 20us)” is the key phrase. sysmon starts its sleep at 20 microseconds:

go
// src/runtime/proc.go:6560
	for {
		if idle == 0 { // start with 20us sleep...
			delay = 20
		} else if idle > 50 { // start doubling the sleep after 1ms...
			delay *= 2
		}
		if delay > 10*1000 { // up to 10ms
			delay = 10 * 1000
		}
		usleep(delay)

So the practical sequence is: syscall enters → sysmon notices within ~20µs → next tick, another ~20µs later, the P is taken. Roughly 20 to 60 microseconds of “the P is hostage” time, and only if there’s work to be done.

The actual theft is thread.takeP(), which is a state transition on the blocked thread:

go
// src/runtime/proc.go:6871
// takeP unwires the P attached to the syscalling thread
// and moves it into the _Pidle state.
func (s syscallingThread) takeP() {
	s.releaseP(_Pidle)
}

and releaseP is where the syscalltick bump happens, which is what tells the returning goroutine “you lost your P”:

go
// src/runtime/proc.go:6877
// releaseP unwires the P from the syscalling thread, moving
// it to the provided state. Callers should prefer to use
// takeP and gcstopP.
func (s syscallingThread) releaseP(state uint32) {
	if state != _Pidle && state != _Pgcstop {
		throw("attempted to release P into a bad state")
	}
	trace := traceAcquire()
	s.pp.m = 0
	s.mp.p = 0
	atomic.Store(&s.pp.status, state)
	if trace.ok() {
		trace.ProcSteal(s.pp)
		traceRelease(trace)
	}
	addGSyscallNoP(s.mp)
	s.pp.syscalltick++
}

s.pp.syscalltick++ — one increment, and the syscalling goroutine’s snapshot no longer matches. When it comes back, exitsyscall takes the slow path.

The other way to lose a P: `dropm`

There’s one more wrinkle. When a goroutine’s M is destroyed (which happens when a thread exits from inside a syscall), the runtime deliberately corrupts the syscalltick so that the tracer doesn’t get confused:

go
// src/runtime/proc.go:2623
	// Trash syscalltick so that it doesn't line up with mp.old.syscalltick anymore.
	//
	// In the new tracer, we model needm and dropm and a goroutine being created and
	// destroyed respectively. The m then might get reused with a different procid but
	// still with a reference to oldp, and still with the same syscalltick. The next
	// time a G is "created" in needm, it'll return and quietly reacquire its P from a
	// different m with a different procid, which will confuse the trace parser. By
	// trashing syscalltick, we ensure that it'll appear as if we lost the P to the
	// tracer parser and that we just reacquired it.
	//
	// Trash the value by decrementing because that gets us as far away from the value
	// the syscall exit code expects as possible. Setting to zero is risky because
	// syscalltick could already be zero (and in fact, is initialized to zero).
	mp.syscalltick--

Decrementing rather than zeroing, because zero is a legitimate value. Nice touch.

`GOMAXPROCS` and the whole thing

Now the part that surprises people. With GOMAXPROCS=1, there is exactly one P. If that P is attached to a goroutine blocked in a syscall, nothing else can run — until sysmon takes the P away and hands it to another thread.

That’s the entire mechanism that keeps a single-CPU Go program alive while one of its goroutines is stuck in a blocking syscall. The cost is up to a few tens of microseconds of stall.

The same logic applies at GOMAXPROCS=8. A blocking syscall doesn’t take the machine down, but it does take a P out of circulation for up to ~20µs before sysmon frees it. If your program has, say, 8 goroutines each occasionally making a 5-millisecond syscall, you can end up with all 8 P’s momentarily attached to syscall-blocked goroutines, and the program stalls until sysmon notices.

This is also why the runtime distinguishes entersyscall from entersyscallblock. If you know your call is going to block for a long time, entersyscallblock gives the P up immediately instead of making sysmon find out in 20µs. The runtime itself uses entersyscallblock for exactly the calls where it knows the answer: waiting on a futex, parking on a channel, waiting for a network poll.

The runtime’s own syscalls don’t go through any of this

One more thing worth knowing: most system calls made by the runtime itself don’t touch the scheduler at all. futexsleep, for example:

go
// src/runtime/os_linux.go:65
//go:nosplit
func futexsleep(addr *uint32, val uint32, ns int64) {
	// Some Linux kernels have a bug where futex of
	// FUTEX_WAIT returns an internal error code
	// as an errno. Libpthread ignores the return value
	// here, and so can we: as it says a few lines up,
	// spurious wakeups are allowed.
	if ns < 0 {
		futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, nil, nil, 0)
		return
	}

	var ts timespec
	ts.setNsec(ns)
	futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, &ts, nil, 0)
}

No entersyscall, no exitsyscall. The runtime’s comment on reentersyscall explains why: “This is called only from the go syscall library and cgocall, not from the low-level system calls used by the runtime.” A futex wait inside the scheduler is a blocking operation by design, and the scheduler already knows it’s blocked — it doesn’t need the syscall bookkeeping to tell it.

Trade-offs

What the “keep the P” design buys you: the common case is nearly free. A read() that returns in 5 microseconds causes zero scheduler churn: no P handoff, no M wakeup, no run-queue manipulation. The goroutine is still _Grunning from the scheduler’s point of view except for a CAS on a status word. Under the old design (release the P on entry, reacquire on exit) every syscall was a full P handoff.

What it costs: a syscall that blocks pins a P for up to a few tens of microseconds. At high syscall rates across many goroutines this shows up as scheduling latency, and the fix is entersyscallblock semantics — but only the runtime can call that.

Why the two-tier design (syscalltick) exists: a P can be lost in three different ways (sysmon retake, dropm, explicit release). Rather than have an explicit “I lost my P” flag that has to be kept consistent across all of them, the runtime piggybacks on a counter that’s already being incremented for other reasons. The TODO(mknyszek) in the source admits this is a hack: “Consider a more explicit mechanism for this.”

Why the retake is gated on “is there work to do”: if the program is idle, yanking the P out of a syscall buys nothing and costs a thread wakeup. The runqempty(pp) && nmspinning+npidle > 0 check keeps the common idle case completely hands-off.

Caveats

The names in older articles are stale. exitsyscallfast and keepsyscall no longer exist as functions. exitsyscallfast survives only in a comment (src/runtime/proc.go:5121, “determined by exitsyscallfast”), and keepsyscall does not appear in the tree at all. The old exitsyscallfast returned a bool telling the caller whether it still had its P; that logic is now inline in exitsyscall itself.

entersyscall and exitsyscall are not public API. They’re exported via //go:linkname for syscall, x/sys, and a small list of very large users (the source calls gvisor.dev/gvisor out by name as a “notable member of the hall of shame”). The comment is explicit: “Do not remove or change the type signature. See go.dev/issue/67401.” But you are calling into runtime internals with no compatibility promise.

//go:nosplit is load-bearing. entersyscall cannot split the stack, because it must be able to return immediately and because the runtime can’t know which of the uintptr arguments to the syscall are really pointers into the stack. The slow path therefore has to use systemstack.

The Linux os_linux.go has no entersyscall in it. Older Go trees had per-OS entersyscall/exitsyscall wrappers in os_linux.go. In the 1.28 tree those are gone; the generic implementations in proc.go are used, and the syscall package pulls them in by linkname. If you’re reading an old blog post about os_linux.go, that’s the file to look for.

Latency numbers are approximate. “About 20 microseconds” is a floor, not a guarantee. sysmon’s sleep doubles once the system has been idle for more than a millisecond, so a program that goes quiet and then makes a blocking syscall can see a much longer delay before sysmon retakes the P.

Wrapping up

The one thing to remember: entersyscall does not release the P. It flips the goroutine to _Gsyscall and leaves the P attached, so a fast syscall costs essentially nothing; sysmon takes the P away about 20 microseconds in only if the call is slow and there’s work waiting.

The part people get wrong: that a blocking syscall takes a core away from your program for its whole duration. It doesn’t. It takes a P away for tens of microseconds, and then sysmon hands that P to another thread while the syscall continues in the kernel.

What to go do about it: if you’re writing code that blocks in a syscall and you care about latency, count how many of your goroutines are in _Gsyscall at once. GODEBUG=schedtrace=1000,scheddetail=1 will show you the scheduler’s view, and a program that regularly has every P attached to a syscall-blocked goroutine is a program that’s paying the retake penalty on every call.

Four states, one of them special

Where the entry point actually lives

What entersyscall actually does

The two flavours: entersyscall and entersyscallblock

Coming back: exitsyscall