Go's netpoller: Why Blocking I/O Doesn't Block a Thread

TL;DR: When a socket has no data, Go does not put the thread to sleep in read(2). The file descriptor is registered with epoll, the goroutine parks on a two-word semaphore inside a pollDesc, and the M goes back to running other goroutines. When epoll_wait reports the socket readable, the runtime flips the semaphore to “ready” and makes the goroutine runnable again. One blocked read costs one goroutine, not one thread.

Blocking I/O is the thing Go is supposed to be good at. A goroutine that reads from a socket should be able to just… wait. And a program with ten thousand of those goroutines should not need ten thousand threads.

The mechanism that makes that work is the netpoller, and it lives in three places: src/runtime/netpoll.go (the generic logic), src/runtime/netpoll_epoll.go (the Linux implementation), and src/internal/poll (the glue that the net package uses).

The problem with blocking

An M running Go code holds a P, and there are only GOMAXPROCS Ps. If a goroutine calls read(2) and the kernel blocks that thread until data arrives, the P goes with it, and the P is now useless for the other goroutines that want to run.

The runtime’s answer is that Go programs should not block in syscalls at all. The scheduler hands a P back to the pool when a goroutine enters a syscall, and takes it back when the syscall returns. That works, but it is a lot of thread churn for something as common as “read from a socket”.

The netpoller removes the blocking entirely. The file descriptor is put in non-blocking mode and registered with the kernel’s readiness notification mechanism — epoll on Linux, kqueue on BSD, IOCP on Windows.

`pollDesc`: the per-descriptor state

Every pollable descriptor gets a pollDesc, allocated from a per-P cache. Here is the whole struct:

go
// src/runtime/netpoll.go:72
// Network poller descriptor.
//
// No heap pointers.
type pollDesc struct {
	_     sys.NotInHeap
	link  *pollDesc      // in pollcache, protected by pollcache.lock
	fd    uintptr        // constant for pollDesc usage lifetime
	fdseq atomic.Uintptr // protects against stale pollDesc

	// atomicInfo holds bits from closing, rd, and wd,
	// which are only ever written while holding the lock,
	// summarized for use by netpollcheckerr,
	// which cannot acquire the lock.
	// After writing these fields under lock in a way that
	// might change the summary, code must call publishInfo
	// before releasing the lock.
	// Code that changes fields and then calls netpollunblock
	// (while still holding the lock) must call publishInfo
	// before calling netpollunblock, because publishInfo is what
	// stops netpollblock from blocking anew
	// (by changing the result of netpollcheckerr).
	// atomicInfo also holds the eventErr bit,
	// recording whether a poll event on the fd got an error;
	// atomicInfo is the only source of truth for that bit.
	atomicInfo atomic.Uint32 // atomic pollInfo

	// rg, wg are accessed atomically and hold g pointers.
	// (Using atomic.Uintptr here is similar to using guintptr elsewhere.)
	rg atomic.Uintptr // pdReady, pdWait, G waiting for read or pdNil
	wg atomic.Uintptr // pdReady, pdWait, G waiting for write or pdNil

	lock    mutex // protects the following fields
	closing bool
	rrun    bool      // whether rt is running
	wrun    bool      // whether wt is running
	user    uint32    // user settable cookie
	rseq    uintptr   // protects from stale read timers
	rt      timer     // read deadline timer
	rd      int64     // read deadline (a nanotime in the future, -1 when expired)
	wseq    uintptr   // protects from stale write timers
	wt      timer     // write deadline timer
	wd      int64     // write deadline (a nanotime in the future, -1 when expired)
	self    *pollDesc // storage for indirect interface. See (*pollDesc).makeArg.
}

The two fields that matter are rg and wg: the “read semaphore” and the “write semaphore”. Each one is a single word that is simultaneously a state flag and a goroutine pointer. The runtime documents the protocol:

go
// src/runtime/netpoll.go:51
// pollDesc contains 2 binary semaphores, rg and wg, to park reader and writer
// goroutines respectively. The semaphore can be in the following states:
//
//	pdReady - io readiness notification is pending;
//	          a goroutine consumes the notification by changing the state to pdNil.
//	pdWait - a goroutine prepares to park on the semaphore, but not yet parked;
//	         the goroutine commits to park by changing the state to G pointer,
//	         or, alternatively, concurrent io notification changes the state to pdReady,
//	         or, alternatively, concurrent timeout/close changes the state to pdNil.
//	G pointer - the goroutine is blocked on the semaphore;
//	            io notification or timeout/close changes the state to pdReady or pdNil respectively
//	            and unparks the goroutine.
//	pdNil - none of the above.
const (
	pdNil   uintptr = 0
	pdReady uintptr = 1
	pdWait  uintptr = 2
)

That is the whole trick: one word, four states, no lock needed on the fast path. pdWait means “a goroutine intends to park here”; the actual goroutine pointer replaces it when the goroutine commits to parking.

text
                 pollDesc.rg (one atomic word)

   pdNil  (0)            nothing pending
      |
      |  CAS(pdNil -> pdWait)      reader prepares to park
      v
   pdWait (2)            intent to park, not yet parked
      |
      |  CAS(pdWait -> &g)         reader commits to park
      v
   &g (pointer)          goroutine parked, waiting for readiness
      |
      |  CAS(&g -> pdReady)        netpoller got an event
      v
   pdReady (1)           event pending
      |
      |  CAS(pdReady -> pdNil)     reader consumes the event
      v
   pdNil  (0)            back to square one

Registering with epoll

On Linux the registration is a one-line epoll_ctl:

go
// src/runtime/netpoll_epoll.go:64
func netpollopen(fd uintptr, pd *pollDesc) uintptr {
	var ev linux.EpollEvent
	ev.Events = linux.EPOLLIN | linux.EPOLLOUT | linux.EPOLLRDHUP | linux.EPOLLET
	netpollPackData(&ev, pd, pd.fdseq.Load())
	return linux.EpollCtl(epfd, linux.EPOLL_CTL_ADD, int32(fd), &ev)
}

Note EPOLLET: edge-triggered. The kernel only tells you about a transition from “not ready” to “ready”, so the runtime must drain the descriptor before parking again. That is why internal/poll retries the read in a loop.

The epoll descriptor itself is created once, lazily, along with an eventfd used to interrupt a blocked epoll_wait:

go
// src/runtime/netpoll_epoll.go:36
func netpollinit() {
	var errno uintptr
	epfd, errno = linux.EpollCreate1(linux.EPOLL_CLOEXEC)
	if errno != 0 {
		println("runtime: epollcreate failed with", errno)
		throw("runtime: netpollinit failed")
	}
	efd, errno := linux.Eventfd(0, linux.EFD_CLOEXEC|linux.EFD_NONBLOCK)
	if errno != 0 {
		println("runtime: eventfd failed with", errno)
		throw("runtime: eventfd failed")
	}
	ev := linux.EpollEvent{
		Events: linux.EPOLLIN,
	}
	netpollPackData(&ev, nil, 0)
	errno = linux.EpollCtl(epfd, linux.EPOLL_CTL_ADD, efd, &ev)

The epoll data field holds a tagged pointer: the pollDesc plus a sequence number. The sequence number is what makes stale events harmless — if the descriptor was closed and its pollDesc recycled, the old event’s sequence will not match and the event is dropped.

Parking: `netpollblock`

This is the function that makes a read “block” without blocking a thread. It is called with the goroutine still running:

go
// src/runtime/netpoll.go:544
// returns true if IO is ready, or false if timed out or closed
// waitio - wait only for completed IO, ignore errors
// Concurrent calls to netpollblock in the same mode are forbidden, as pollDesc
// can hold only a single waiting goroutine for each mode.
func netpollblock(pd *pollDesc, mode int32, waitio bool) bool {
	gpp := &pd.rg
	if mode == 'w' {
		gpp = &pd.wg
	}

	// set the gpp semaphore to pdWait
	for {
		// Consume notification if already ready.
		if gpp.CompareAndSwap(pdReady, pdNil) {
			return true
		}
		if gpp.CompareAndSwap(pdNil, pdWait) {
			break
		}

		// Double check that this isn't corrupt; otherwise we'd loop
		// forever.
		if v := gpp.Load(); v != pdReady && v != pdNil {
			throw("runtime: double wait")
		}
	}

	// need to recheck error states after setting gpp to pdWait
	// this is necessary because runtime_pollUnblock/runtime_pollSetDeadline/deadlineimpl
	// do the opposite: store to closing/rd/wd, publishInfo, load of rg/wg
	if waitio || netpollcheckerr(pd, mode) == pollNoError {
		gopark(netpollblockcommit, unsafe.Pointer(gpp), waitReasonIOWait, traceBlockNet, 5)
	}
	// be careful to not lose concurrent pdReady notification
	old := gpp.Swap(pdNil)
	if old > pdWait {
		throw("runtime: corrupted polldesc")
	}
	return old == pdReady
}

Three steps:

Try to consume a pending “ready” notification (someone got there first).
Claim the semaphore: pdNil → pdWait.
gopark with netpollblockcommit as the commit function, which does the pdWait → &g transition:

go
// src/runtime/netpoll.go:529
func netpollblockcommit(gp *g, gpp unsafe.Pointer) bool {
	r := atomic.Casuintptr((*uintptr)(gpp), pdWait, uintptr(unsafe.Pointer(gp)))
	if r {
		// Bump the count of goroutines waiting for the poller.
		// The scheduler uses this to decide whether to block
		// waiting for the poller if there is nothing else to do.
		netpollAdjustWaiters(1)
	}
	return r
}

gopark is the runtime’s “stop this goroutine and switch to another one” primitive. The M does not block; it goes back to the scheduler and runs something else. The goroutine’s stack is parked with the M’s g0, and when somebody calls goready on it, it comes back.

The commit function returning false means “you were woken up while I was committing, don’t park” — that is the race between a deadline firing and the goroutine actually parking, and it is handled by gopark’s contract.

Waking: `netpollready`

When epoll_wait returns, the runtime converts the event into a runnable goroutine:

go
// src/runtime/netpoll.go:494
func netpollready(toRun *gList, pd *pollDesc, mode int32) int32 {
	delta := int32(0)
	var rg, wg *g
	if mode == 'r' || mode == 'r'+'w' {
		rg = netpollunblock(pd, 'r', true, &delta)
	}
	if mode == 'w' || mode == 'r'+'w' {
		wg = netpollunblock(pd, 'w', true, &delta)
	}
	if rg != nil {
		toRun.push(rg)
	}
	if wg != nil {
		toRun.push(wg)
	}
	return delta
}

and netpollunblock is where the semaphore flips:

go
// src/runtime/netpoll.go:591
func netpollunblock(pd *pollDesc, mode int32, ioready bool, delta *int32) *g {
	gpp := &pd.rg
	if mode == 'w' {
		gpp = &pd.wg
	}

	for {
		old := gpp.Load()
		if old == pdReady {
			return nil
		}
		if old == pdNil && !ioready {
			// Only set pdReady for ioready. runtime_pollWait
			// will check for timeout/cancel before waiting.
			return nil
		}
		new := pdNil
		if ioready {
			new = pdReady
		}
		if gpp.CompareAndSwap(old, new) {
			if old == pdWait {
				old = pdNil
			} else if old != pdNil {
				*delta -= 1
			}
			return (*g)(unsafe.Pointer(old))
		}
	}
}

The CAS returns the old value: if it was a goroutine pointer, that goroutine is the one to wake. The woken goroutines are collected in a gList and handed to the scheduler, which makes them runnable.

The `epoll_wait` loop

The netpoller itself is a plain loop over epoll_wait:

go
// src/runtime/netpoll_epoll.go:146
	var toRun gList
	delta := int32(0)
	for i := int32(0); i < n; i++ {
		ev := events[i]
		if ev.Events == 0 {
			continue
		}

		pd, tag := netpollUnpackData(&ev)
		// pd == nil denotes a netpollBreak eventfd.
		if pd == nil {
			if ev.Events != linux.EPOLLIN {
				println("runtime: netpoll: eventfd ready for", ev.Events)
				throw("runtime: netpoll: eventfd ready for something unexpected")
			}
			if delay != 0 {
				// netpollBreak could be picked up by a
				// nonblocking poll. Only read the 8-byte
				// integer if blocking.
				// Since EFD_SEMAPHORE was not specified,
				// the eventfd counter will be reset to 0.
				var one uint64
				read(int32(netpollEventFd), noescape(unsafe.Pointer(&one)), int32(unsafe.Sizeof(one)))
				netpollWakeSig.Store(0)
			}
			continue
		}

		var mode int32
		if ev.Events&(linux.EPOLLIN|linux.EPOLLRDHUP|linux.EPOLLHUP|linux.EPOLLERR) != 0 {
			mode += 'r'
		}
		if ev.Events&(linux.EPOLLOUT|linux.EPOLLHUP|linux.EPOLLERR) != 0 {
			mode += 'w'
		}
		if mode != 0 {
			if pd.fdseq.Load() == tag {
				pd.setEventErr(ev.Events == linux.EPOLLERR, tag)
				delta += netpollready(&toRun, pd, mode)
			}
		}
	}
	return toRun, delta
}

The pd.fdseq.Load() == tag check is the stale-event guard mentioned above: if the descriptor was closed and the pollDesc reused, the tag will not match and the event is dropped.

Note that only one thread polls at a time. The scheduler checks the netpoller opportunistically before it starts stealing work, and it says so in a comment:

go
// src/runtime/proc.go:3505
	// Poll network.
	// This netpoll is only an optimization before we resort to stealing.
	// We can safely skip it if there are no waiters or a thread is blocked
	// in netpoll already. If there is any kind of logical race with that
	// blocked thread (e.g. it has already returned from netpoll, but does
	// not set lastpoll yet), this thread will do blocking netpoll below
	// anyway.
	// We only poll from one thread at a time to avoid kernel contention
	// on machines with many cores.

The `internal/poll` side

The net package does not talk to the netpoller directly. It goes through internal/poll, which is where the “try the syscall, and only park if it would block” logic lives.

go
// src/internal/poll/fd_unix.go:144
// Read implements io.Reader.
func (fd *FD) Read(p []byte) (int, error) {
	if err := fd.readLock(); err != nil {
		return 0, err
	}
	defer fd.readUnlock()
	if len(p) == 0 {
		// If the caller wanted a zero byte read, return immediately
		// without trying (but after acquiring the readLock).
		// Otherwise syscall.Read returns 0, nil which looks like
		// io.EOF.
		//
		// Waiting for readability instead was proposed in
		// go.dev/cl/22031 and abandoned. Blocking would change
		// the behavior of existing callers, and the netpoller's
		// edge-triggered notifications alone cannot detect data
		// that is already buffered. See go.dev/issue/15735.
		return 0, nil
	}
	if err := fd.pd.prepareRead(fd.isFile); err != nil {
		return 0, err
	}
	if fd.IsStream && len(p) > maxRW {
		p = p[:maxRW]
	}
	for {
		n, err := ignoringEINTRIO(syscall.Read, fd.Sysfd, p)
		if err != nil {
			n = 0
			if err == syscall.EAGAIN && fd.pd.pollable() {
				if err = fd.pd.waitRead(fd.isFile); err == nil {
					continue
				}
			}
		}
		err = fd.eofError(n, err)
		return n, err
	}
}

This is the whole contract in twelve lines: the socket is in non-blocking mode, so syscall.Read returns EAGAIN when there is no data. Only then does the goroutine park, in waitRead, and the loop retries the read when it wakes up.

waitRead is a thin wrapper over the runtime:

go
// src/internal/poll/fd_poll_runtime.go:80
func (pd *pollDesc) wait(mode int, isFile bool) error {
	if pd.runtimeCtx == 0 {
		return errors.New("waiting for unsupported file type")
	}
	res := runtime_pollWait(pd.runtimeCtx, mode)
	return convertErr(res, isFile)
}

func (pd *pollDesc) waitRead(isFile bool) error {
	return pd.wait('r', isFile)
}

The runtimeCtx is the runtime’s *pollDesc, passed as an opaque pointer across the internal/poll → runtime boundary via //go:linkname.

Deadlines

Deadlines are handled with the same per-descriptor timers that the scheduler uses. Setting one arms a timer; when it fires, it unblocks the waiter with an error instead of a readiness event:

go
// src/internal/poll/fd_poll_runtime.go:146
func setDeadlineImpl(fd *FD, t time.Time, mode int) error {
	if err := fd.incref(); err != nil {
		return err
	}
	defer fd.decref()
	if err := fd.ensureInit(); err != nil {
		return err
	}

	var d int64
	if !t.IsZero() {
		d = int64(time.Until(t))
		if d == 0 {
			d = -1 // don't confuse deadline right now with no deadline
		}
	}
	if fd.pd.runtimeCtx == 0 {
		return ErrNoDeadline
	}
	runtime_pollSetDeadline(fd.pd.runtimeCtx, d, mode)
	return nil
}

ErrNoDeadline is the answer for a file that is not pollable — a regular file, for instance, which the kernel does not support readiness notification for:

go
// src/internal/poll/fd.go:39
var ErrNoDeadline = errors.New("file type does not support deadline")

Why goroutines must not block in syscalls

The rule that falls out of all this: a goroutine must never block in a syscall, because the thread it is running on holds a P, and the P is needed by other goroutines.

For network connections the runtime solves this for you. For everything else — reading a file, a pipe, a terminal — the runtime still has to hand the P back while the thread is in the syscall, and take it back when the syscall returns. That handover is the entersyscall/exitsyscall dance, and it is why os.File reads from a regular file are not free.

Trade-offs

What the netpoller buys you. A blocked network read costs a goroutine (2 KB of stack) instead of a thread (8 MB of stack plus a kernel thread). Ten thousand idle connections fit in a few megabytes.

What it costs. Every I/O operation is a non-blocking syscall followed by a possible park and wake-up. For a workload doing lots of tiny reads, the park/wake churn costs more than a blocking read would.

One poller. The runtime polls from one thread at a time to avoid contention on the epoll descriptor, which means that under extreme connection churn a single thread can become the bottleneck.

Edge-triggered notifications. With EPOLLET, the runtime must read until EAGAIN, or it will miss events. That is why internal/poll loops.

Regular files are not pollable. epoll does not support regular files at all, so os.File reads still block the thread.

Caveats

The netpoller is Unix-only in spirit. Windows uses IOCP with a different implementation, and js/wasm has a stub netpoller that does nothing because wasm sockets cannot be made non-blocking.

netpollBreak is how a sleeping poller is woken up. When a timer expires or a new descriptor is registered, the runtime writes to the eventfd to interrupt a blocked epoll_wait.

A goroutine can only be blocked once per descriptor per direction. Two goroutines reading the same socket concurrently is a bug: pollDesc holds one waiting goroutine per mode, and the runtime says so explicitly in netpollblock.

netpollWaiters matters for the scheduler. The count of goroutines waiting on I/O decides whether the scheduler should block in netpoll when there is nothing else to do.

Stale events are handled by sequence numbers, not by locking. If you close a descriptor while another goroutine is blocked on it, the waiter is woken with an error (errClosing), and the pending epoll event is dropped when its sequence number does not match.

A file descriptor that is not in non-blocking mode will block the thread. internal/poll sets the descriptor non-blocking when it initialises the poller; if you hand a raw descriptor to syscall.Read, you are on your own.

Wrapping up

The one thing to remember: a blocking read in Go is not a blocked thread. The goroutine parks on a one-word semaphore inside a pollDesc, the M goes back to the scheduler, and the goroutine is made runnable again when epoll_wait reports the socket readable. The whole mechanism is four states in one atomic word: pdNil, pdWait, pdReady, and “a goroutine pointer”.

The part people get wrong: the pollDesc has two semaphores, one for reads and one for writes, and the readiness event decides which one to wake based on the event mask. A socket that is readable and writable at the same time wakes both waiters.

What to go do about it: if you have a service with many idle connections, check that you are actually using net.Conn (and therefore the netpoller) rather than raw syscall calls, and if you see goroutines stuck in netpollblock, look at GODEBUG=schedtrace=1 output to see whether the poller is even being reached.

The problem with blocking

pollDesc: the per-descriptor state

Registering with epoll

Parking: netpollblock

Waking: netpollready

The epoll_wait loop

The internal/poll side