Opensource internals

The Kubelet Doesn't Poll for Memory Pressure: epoll, eventfd and the memcg Threshold

12 min read Jay Hubbert

For hard memory eviction thresholds the kubelet registers with the kernel's memcg via cgroup.event_control and epolls an eventfd; on cgroup v2 it falls back to polling.

TL;DR: For eviction-hard memory thresholds the kubelet does not poll the cgroup. It writes <eventfd> <watchfd> <threshold> into cgroup.event_control, and the kernel writes to that eventfd every time memory.usage_in_bytes crosses the threshold in either direction. The kubelet then epolls the eventfd and re-runs the eviction manager immediately. The threshold it registers is not the eviction threshold: because memcg counts reclaimable page cache, the kubelet adds back inactive_file to convert a working set threshold into a usage threshold. On cgroup v2 the whole mechanism does not exist and the kubelet installs a no-op notifier and falls back to polling.

Polling a cgroup for memory pressure is expensive and racy. You read a memory file, you compare it to a number, and in between the number moves. On a node with a hundred containers the read itself is the expensive part, and the kubelet’s own default loop only runs every ten seconds, so a node can cross memory.available<100Mi and stay there for a full interval before anyone looks.

The Linux kernel has had a better answer since 3.19: a cgroup can be told “wake me up if this counter crosses this value”, and it will do so via an eventfd. The kubelet’s job is to wire that up correctly, which turns out to be mostly arithmetic.

Two file descriptors and one magic write

cgroup.event_control is a write-only pseudo-file. You write three numbers into it — an eventfd to signal, a watchfd identifying the attribute to watch, and the threshold — and the kernel does the rest. The kubelet’s NewCgroupNotifier is a straight transliteration of that protocol:

go 
// pkg/kubelet/eviction/threshold_notifier_linux.go:48
// NewCgroupNotifier returns a linuxCgroupNotifier, which performs cgroup control operations required
// to receive notifications from the cgroup when the threshold is crossed in either direction.
func NewCgroupNotifier(logger klog.Logger, path, attribute string, threshold int64) (CgroupNotifier, error) {
	// cgroupv2 does not support monitoring cgroup memory thresholds using cgroup.event_control.
	// Instead long term, on cgroupv2 kubelet should rely on combining usage of memory.low on root pods cgroup with inotify notifications on memory.events and or PSI pressure.
	// For now, let's return a fake "disabled" cgroup notifier on cgroupv2.
	// https://github.com/kubernetes/kubernetes/issues/106331
	if libcontainercgroups.IsCgroup2UnifiedMode() {
		return &disabledThresholdNotifier{}, nil
	}

	var watchfd, eventfd, epfd, controlfd int
	var err error
	watchfd, err = unix.Open(fmt.Sprintf("%s/%s", path, attribute), unix.O_RDONLY|unix.O_CLOEXEC, 0)
	if err != nil {
		return nil, err
	}
	defer unix.Close(watchfd)
	controlfd, err = unix.Open(fmt.Sprintf("%s/cgroup.event_control", path), unix.O_WRONLY|unix.O_CLOEXEC, 0)
	if err != nil {
		return nil, err
	}
	defer unix.Close(controlfd)
	eventfd, err = unix.Eventfd(0, unix.EFD_CLOEXEC)

Three fds, and each has a distinct job. watchfd is an fd on the attribute file itself — memory.usage_in_bytes — and it exists purely so the kernel knows which counter you mean; the kubelet closes it immediately afterwards. controlfd is the registration channel. eventfd is the notification channel, and it is the only fd that survives the function.

Then the registration write:

go 
// pkg/kubelet/eviction/threshold_notifier_linux.go:99
	config := fmt.Sprintf("%d %d %d", eventfd, watchfd, threshold)
	_, err = unix.Write(controlfd, []byte(config))
	if err != nil {
		return nil, err
	}
	return &linuxCgroupNotifier{
		eventfd: eventfd,
		epfd:    epfd,
		stop:    make(chan struct{}),
	}, nil
}

One write(2) with three space-separated decimal integers. If the format is wrong the kernel rejects it; there is no negotiation, no “add another watch” call, no way to change the threshold afterwards. The registration is one-shot.

The event loop

The notifier’s Start puts the eventfd into an epoll set and then blocks on epoll_wait in a loop:

go 
// pkg/kubelet/eviction/threshold_notifier_linux.go:111
func (n *linuxCgroupNotifier) Start(ctx context.Context, eventCh chan<- struct{}) {
	logger := klog.FromContext(ctx)
	err := unix.EpollCtl(n.epfd, unix.EPOLL_CTL_ADD, n.eventfd, &unix.EpollEvent{
		Fd:     int32(n.eventfd),
		Events: unix.EPOLLIN,
	})
	if err != nil {
		logger.Info("Eviction manager: error adding epoll eventfd", "err", err)
		return
	}
	buf := make([]byte, eventSize)
	for {
		select {
		case <-n.stop:
			return
		default:
		}
		event, err := wait(n.epfd, n.eventfd, notifierRefreshInterval)
		if err != nil {
			logger.Info("Eviction manager: error while waiting for memcg events", "err", err)
			return
		} else if !event {
			// Timeout on wait.  This is expected if the threshold was not crossed
			continue
		}
		// Consume the event from the eventfd
		_, err = unix.Read(n.eventfd, buf)
		if err != nil {
			logger.Info("Eviction manager: error reading memcg events", "err", err)
			return
		}
		eventCh <- struct{}{}
	}
}

Two details hide in there. The eventfd must be read to drain it — an eventfd is a counter, not a flag, and if you never read it the kernel will keep reporting it as ready. And the read has to happen before the next epoll_wait, which is why the loop reads first and then signals.

The wait helper has a timeout, and treats a timeout as a normal outcome rather than an error:

go 
// pkg/kubelet/eviction/threshold_notifier_linux.go:146
// wait waits up to notifierRefreshInterval for an event on the Epoll FD for the
// eventfd we are concerned about.  It returns an error if one occurs, and true
// if the consumer should read from the eventfd.
func wait(epfd, eventfd int, timeout time.Duration) (bool, error) {
	events := make([]unix.EpollEvent, numFdEvents+1)
	timeoutMS := int(timeout / time.Millisecond)
	n, err := unix.EpollWait(epfd, events, timeoutMS)
	if n == -1 {
		if err == unix.EINTR {
			// Interrupt, ignore the error
			return false, nil
		}
		return false, err
	}
	if n == 0 {
		// Timeout
		return false, nil
	}
	if n > numFdEvents {
		return false, fmt.Errorf("epoll_wait returned more events than we know what to do with")
	}

EINTR is swallowed — a signal handler must not kill the notifier — and the numFdEvents guard exists so a future change that registers more fds cannot silently overflow the events slice.

“In either direction” is the whole point

The doc comment on NewCgroupNotifier says the notifier fires “when the threshold is crossed in either direction”. That is the kernel’s behaviour for memory.usage_in_bytes and it is what makes the design work.

The kernel fires when usage goes above the threshold, and it fires again when usage comes back below it. So the kubelet cannot treat a notification as “we are under pressure right now”. It treats it as “something changed, come look”. The handler that gets invoked does not decide anything; it just re-runs the eviction manager:

go 
// pkg/kubelet/eviction/eviction_manager.go:186
// Start starts the control loop to observe and response to low compute resources.
func (m *managerImpl) Start(ctx context.Context, diskInfoProvider DiskInfoProvider, podFunc ActivePodsFunc, podCleanedUpFunc PodCleanedUpFunc, monitoringInterval time.Duration) {
	logger := klog.FromContext(ctx)
	thresholdHandler := func(message string) {
		logger.Info(message)
		_, _ = m.synchronize(ctx, diskInfoProvider, podFunc)
	}

The real decision — is the threshold met, has the grace period elapsed, which pod dies — happens in synchronize, on the same code path as the ten-second poll. The notification is a wakeup, not a decision. That is why the “either direction” semantics are harmless: a spurious wakeup just costs one extra synchronize call.

The arithmetic: working set versus usage

Here is the part that is easy to get wrong. The eviction threshold is expressed against the node’s available memory and the kubelet’s notion of usage is the working set. But memcg’s memory.usage_in_bytes includes reclaimable page cache — the kernel is perfectly willing to drop clean file pages to satisfy an allocation, so counting them as “used” would fire the notifier constantly on a node with a busy page cache and no real pressure.

So the kubelet has to convert. If the working set is W and the total usage is U, then U - W is the inactive (reclaimable) file portion. The notifier therefore registers a usage threshold that is higher than the working set threshold by exactly that amount:

go 
// pkg/kubelet/eviction/memory_threshold_notifier_others.go:84
	// Set threshold on usage to capacity - eviction_hard + inactive_file,
	// since we want to be notified when working_set = capacity - eviction_hard
	inactiveFile := resource.NewQuantity(int64(*memoryStats.UsageBytes-*memoryStats.WorkingSetBytes), resource.BinarySI)
	capacity := resource.NewQuantity(int64(*memoryStats.AvailableBytes+*memoryStats.WorkingSetBytes), resource.BinarySI)
	evictionThresholdQuantity := evictionapi.GetThresholdQuantity(m.threshold.Value, capacity)
	memcgThreshold := capacity.DeepCopy()
	memcgThreshold.Sub(*evictionThresholdQuantity)
	memcgThreshold.Add(*inactiveFile)

Read it as an equation. capacity = available + workingSet is the total memory the node can hold. evictionThresholdQuantity is the amount of memory that must remain available — either an absolute quantity like 100Mi or a percentage of capacity. So capacity - evictionThreshold is the working set at which the node is in violation. Add inactiveFile and you get the usage at which the working set hits that number.

The threshold value itself comes from the eviction API, and percentage thresholds are resolved against capacity:

go 
// pkg/kubelet/eviction/api/types.go:110
// GetThresholdQuantity returns the expected quantity value for a thresholdValue
func GetThresholdQuantity(value ThresholdValue, capacity *resource.Quantity) *resource.Quantity {
	if value.Quantity != nil {
		res := value.Quantity.DeepCopy()
		return &res
	}
	return resource.NewQuantity(int64(float64(capacity.Value())*float64(value.Percentage)), resource.BinarySI)
}

Two footnotes. The inactive_file figure is a snapshot taken when the threshold was last computed, not a live value — the notifier is periodically rebuilt for exactly that reason (see below). And the whole computation uses memoryStats.AvailableBytes and memoryStats.WorkingSetBytes from the kubelet’s own summary API, which for a non-allocatable threshold is the node’s memory stats.

For allocatable thresholds (memory.available vs allocatable.memory.available) the maths is done against the pods cgroup instead of the node cgroup, and the notifier is pointed at the pods cgroup path:

go 
// pkg/kubelet/eviction/memory_threshold_notifier_others.go:71
func (m *linuxMemoryThresholdNotifier) UpdateThreshold(ctx context.Context, summary *statsapi.Summary) error {
	logger := klog.FromContext(ctx)
	memoryStats := summary.Node.Memory
	if isAllocatableEvictionThreshold(m.threshold) {
		allocatableContainer, err := getSysContainer(summary.Node.SystemContainers, statsapi.SystemContainerPods)
		if err != nil {
			return err
		}
		memoryStats = allocatableContainer.Memory
	}

The registration is torn down and rebuilt

Because cgroup.event_control is write-only and one-shot, there is no way to change a registered threshold. The only way to move it is to tear the whole thing down and register again. That is what UpdateThreshold does:

go 
// pkg/kubelet/eviction/memory_threshold_notifier_others.go:92
	logger.V(3).Info("Eviction manager: setting notifier to capacity", "notifier", m.Description(), "capacity", memcgThreshold.String())
	if m.notifier != nil {
		m.notifier.Stop()
	}
	newNotifier, err := m.factory.NewCgroupNotifier(logger, m.cgroupPath, memoryUsageAttribute, memcgThreshold.Value())
	if err != nil {
		return err
	}
	m.notifier = newNotifier
	go m.notifier.Start(ctx, m.events)
	return nil
}

Stop() closes the eventfd and the epoll fd, which makes the old registration useless, and then a brand new eventfd is created and registered. Rebuilding on every synchronize would be absurd — the eventfd would be recreated every ten seconds for no reason — so the manager rate limits it:

go 
// pkg/kubelet/eviction/memory_threshold_notifier.go:24
const (
	// this prevents constantly updating the memcg notifier if synchronize
	// is run frequently.
	notifierRefreshInterval = 10 * time.Second
)

and the check that enforces the rate limit lives in the eviction manager’s main loop:

go 
// pkg/kubelet/eviction/eviction_manager.go:309
	if m.clock.Since(m.thresholdsLastUpdated) > notifierRefreshInterval {
		m.thresholdsLastUpdated = m.clock.Now()
		for _, notifier := range m.thresholdNotifiers {
			if err := notifier.UpdateThreshold(ctx, summary); err != nil {
				logger.Info("Eviction manager: failed to update notifier", "notifier", notifier.Description(), "err", err)
			}
		}
	}

Ten seconds happens to be the same value as the eviction monitoring period, so in practice the notifier is rebuilt roughly once per poll cycle — but only if a poll actually happened. A node that is being woken by the kernel every few seconds does not rebuild its notifier every time.

cgroup v2 has no answer

The mechanism above is a cgroup v1 feature. There is no cgroup.event_control in cgroup v2; the closest equivalents are memory.events (a notification file with a low/high/max state machine, readable via poll but with much coarser semantics) and PSI. The kubelet does not implement any of that. It checks for unified mode up front and installs a notifier that does nothing:

go 
// pkg/kubelet/eviction/threshold_notifier_linux.go:198
// disabledThresholdNotifier is a fake diasbled threshold notifier that performs no-ops.
type disabledThresholdNotifier struct{}

func (*disabledThresholdNotifier) Start(context.Context, chan<- struct{}) {}
func (*disabledThresholdNotifier) Stop()                                  {}

The comment above it points at the tracking issue (kubernetes/kubernetes#106331) and sketches the intended future: combine memory.low on the pods cgroup with inotify on memory.events, or use PSI.

The same notifier is used on non-Linux platforms, for the same reason — cgroups do not exist:

go 
// pkg/kubelet/eviction/threshold_notifier_unsupported.go:27
// NewCgroupNotifier creates a cgroup notifier that does nothing because cgroups do not exist on non-linux systems.
func NewCgroupNotifier(logger klog.Logger, path, attribute string, threshold int64) (CgroupNotifier, error) {
	logger.V(5).Info("cgroup notifications not supported")
	return &unsupportedThresholdNotifier{}, nil
}

Windows does something different: it polls the commit charge, on a timer, in a goroutine:

go 
// pkg/kubelet/eviction/memory_threshold_notifier_windows.go:50
func (m *windowsMemoryThresholdNotifier) Start(ctx context.Context) {
	logger := klog.FromContext(ctx)
	logger.Info("Eviction manager: starting windowsMemoryThresholdNotifier", "notifier", m.Description())
	go func() {
		for true {
			time.Sleep(notifierRefreshInterval)
			m.checkMemoryUsage(logger)
		}
	}()

The switch that turns it on

All of this is off by default. The kubelet configuration file has a boolean for it:

go 
// staging/src/k8s.io/kubelet/config/v1beta1/types.go:843
	// kernelMemcgNotification, if set, instructs the kubelet to integrate with the
	// kernel memcg notification for determining if memory eviction thresholds are
	// exceeded rather than polling.
	// Default: false
	// +optional
	KernelMemcgNotification bool `json:"kernelMemcgNotification,omitempty"`

and the eviction manager only creates the notifiers when it is set (or on Windows):

go 
// pkg/kubelet/eviction/eviction_manager.go:193
	logger.Info("Eviction manager: starting control loop")
	if m.config.KernelMemcgNotification || runtime.GOOS == "windows" {
		for _, threshold := range m.config.Thresholds {
			if threshold.Signal == evictionapi.SignalMemoryAvailable || threshold.Signal == evictionapi.SignalAllocatableMemoryAvailable {
				notifier, err := NewMemoryThresholdNotifier(logger, threshold, m.config.PodCgroupRoot, &CgroupNotifierFactory{}, thresholdHandler)

Only the memory signals get a notifier. Disk and PID pressure are always polled.

There is one important caveat for anyone enabling this on a modern node: if the node is running cgroup v2, the notifier is the disabled one, so you get the polling behaviour with none of the benefit — and the only visible difference is a log line about a notifier having been created.

Wrapping up

The one thing to remember: the kubelet’s hard memory threshold is not a polling loop, it is a kernel subscription. One write() into cgroup.event_control registers an eventfd with the memcg, and the kernel signals it whenever memory.usage_in_bytes crosses the threshold in either direction; the kubelet epolls that fd and re-runs the eviction manager on the spot.

The part people get wrong: the number registered with the kernel is not the number in evictionHard. The threshold is about working set, memcg counts usage, and the gap between them is reclaimable page cache — so the kubelet registers capacity - threshold + inactive_file. Forget the inactive_file correction and the notifier fires on page-cache churn instead of on real memory pressure.

What to go do about it: check the cgroup mode on your nodes first (/sys/fs/cgroup/cgroup.controllers exists on v2, and kubelet_evictions alone will not tell you which path you are on). If the nodes are cgroup v1 and you want the fast path, set kernelMemcgNotification: true in the kubelet configuration file and restart the kubelet. To confirm it took effect, look for the Eviction manager: setting notifier to capacity line at -v=3 — the capacity field in that message is the converted usage threshold, not your evictionHard value. On cgroup v2 there is nothing to enable yet (issue kubernetes/kubernetes#106331); keep evictionSoft thresholds with a grace period so the ten-second polling loop has room to react.

KubernetesInternalsKubelet