Opensource internals

The Kubelet Evicts the Pod That Misbehaves First: A Three-Key Ranking

12 min read Jay Hubbert

Under memory pressure the kubelet ranks pods with a lexicographic chain: exceeding your request beats pod priority, which beats absolute usage above the request.

TL;DR: When the kubelet has to evict under memory pressure it does not kill the biggest pod. It runs a lexicographic sort with three keys in a fixed order: (1) does the pod’s working set exceed its memory request, (2) the pod’s priority, (3) the absolute usage above the request. The first key is the interesting one: a pod that has blown past what it asked for is evicted before a pod with lower priority that is still within its request. The same three-key combinator (orderedBy/multiSorter) serves disk pressure and PID pressure, only with the keys reshuffled.

Everyone has watched it happen. A node goes into MemoryPressure, one pod disappears, and the postmortem question is always the same: why that one? The intuitive answer — “it was the biggest” — is wrong, and it is wrong in an interesting way rather than an arbitrary way.

The kubelet’s eviction manager is a small, self-contained ranking machine. It collects stats, builds a per-signal ranking function once at startup, sorts the candidate pods with it, and kills exactly one. The ranking function is where all the policy lives, and the policy is not “biggest first”.

The chain, not the heap

The whole ranking abstraction is thirty lines. A multiSorter holds a list of comparison functions and applies them in order, stopping at the first one that produces a non-zero answer:

go 
// pkg/kubelet/eviction/helpers.go:629
// multiSorter implements the Sort interface, sorting changes within.
type multiSorter struct {
	pods []*v1.Pod
	cmp  []cmpFunc
}

// Sort sorts the argument slice according to the less functions passed to OrderedBy.
func (ms *multiSorter) Sort(pods []*v1.Pod) {
	ms.pods = pods
	sort.Sort(ms)
}

// OrderedBy returns a Sorter that sorts using the cmp functions, in order.
// Call its Sort method to sort the data.
func orderedBy(cmp ...cmpFunc) *multiSorter {
	return &multiSorter{
		cmp: cmp,
	}
}

and the Less implementation is a textbook lexicographic chain:

go 
// pkg/kubelet/eviction/helpers.go:659
// Less is part of sort.Interface.
func (ms *multiSorter) Less(i, j int) bool {
	p1, p2 := ms.pods[i], ms.pods[j]
	var k int
	for k = 0; k < len(ms.cmp)-1; k++ {
		cmpResult := ms.cmp[k](p1, p2)
		// p1 is less than p2
		if cmpResult < 0 {
			return true
		}
		// p1 is greater than p2
		if cmpResult > 0 {
			return false
		}
		// we don't know yet
	}
	// the last cmp func is the final decider
	return ms.cmp[k](p1, p2) < 0
}

Because sort.Sort is ascending, the pod at index 0 of the sorted slice is the one that gets killed. Each cmpFunc returns -1 when pod 1 should be evicted first, +1 when pod 2 should be evicted first, and 0 when the key is a tie and the next key should decide.

Three helpers do the actual comparing, and they all share one shape: if the stats for either pod are missing, the pod with missing stats is preferred.

go 
// pkg/kubelet/eviction/helpers.go:805
// cmpBool compares booleans, placing true before false
func cmpBool(a, b bool) int {
	if a == b {
		return 0
	}
	if !b {
		return -1
	}
	return 1
}

That cmpBool is the trick that makes the first key work. cmpBool(true, false) is -1: true sorts first. So a boolean predicate — “exceeds its request” — is enough to dominate the entire rest of the chain.

Key one: over your request, not over the node

The first comparison in the memory chain is exceedMemoryRequests:

go 
// pkg/kubelet/eviction/helpers.go:692
// exceedMemoryRequests compares whether or not pods' memory usage exceeds their requests
func exceedMemoryRequests(stats statsFunc) cmpFunc {
	return func(p1, p2 *v1.Pod) int {
		p1Stats, p1Found := stats(p1)
		p2Stats, p2Found := stats(p2)
		if !p1Found || !p2Found {
			// prioritize evicting the pod for which no stats were found
			return cmpBool(!p1Found, !p2Found)
		}

		p1Memory := memoryUsage(p1Stats.Memory)
		p2Memory := memoryUsage(p2Stats.Memory)
		p1ExceedsRequests := p1Memory.Cmp(v1resource.GetResourceRequestQuantity(p1, v1.ResourceMemory)) == 1
		p2ExceedsRequests := p2Memory.Cmp(v1resource.GetResourceRequestQuantity(p2, v1.ResourceMemory)) == 1
		// prioritize evicting the pod which exceeds its requests
		return cmpBool(p1ExceedsRequests, p2ExceedsRequests)
	}
}

Note == 1: strictly greater than the request. A pod sitting exactly at its request is not “exceeding” it and does not get the free pass to the front of the queue.

And note what “usage” means here. It is the working set, not the RSS:

go 
// pkg/kubelet/eviction/helpers.go:514
// memoryUsage converts working set into a resource quantity.
func memoryUsage(memStats *statsapi.MemoryStats) *resource.Quantity {
	if memStats == nil || memStats.WorkingSetBytes == nil {
		return &resource.Quantity{Format: resource.BinarySI}
	}
	usage := int64(*memStats.WorkingSetBytes)
	return resource.NewQuantity(usage, resource.BinarySI)
}

Working set excludes reclaimable page cache, which is exactly the memory the kernel can take back without killing anybody. Comparing working set against the request is therefore the honest question: “is this container using more than it promised, in memory the kernel cannot just claw back?”

The request side comes from the pod spec, and it is the sum of container requests plus the pod overhead:

go 
// pkg/api/v1/resource/helpers.go:32
// GetResourceRequestQuantity finds and returns the request quantity for a specific resource.
func GetResourceRequestQuantity(pod *v1.Pod, resourceName v1.ResourceName) resource.Quantity {
	requestQuantity := resource.Quantity{}

	switch resourceName {
	case v1.ResourceCPU:
		requestQuantity = resource.Quantity{Format: resource.DecimalSI}
	case v1.ResourceMemory, v1.ResourceStorage, v1.ResourceEphemeralStorage:
		requestQuantity = resource.Quantity{Format: resource.BinarySI}
	default:
		requestQuantity = resource.Quantity{Format: resource.DecimalSI}
	}

Key two: priority, and only if the first key ties

Priority is the second key, and it is a plain integer comparison:

go 
// pkg/kubelet/eviction/helpers.go:679
// priority compares pods by Priority, if priority is enabled.
func priority(p1, p2 *v1.Pod) int {
	priority1 := corev1helpers.PodPriority(p1)
	priority2 := corev1helpers.PodPriority(p2)
	if priority1 == priority2 {
		return 0
	}
	if priority1 > priority2 {
		return 1
	}
	return -1
}

Returns +1 when pod 1 has the higher priority, which in the Less chain means pod 2 sorts first. Lower priority is evicted first. That is the expected behaviour — but only among pods that tied on key one. A low-priority pod that is within its request loses to a high-priority pod that is over its request, every single time.

Key three: the absolute excess

The final tiebreak is how far above the request each pod is running. The comparison subtracts the request from the usage, so it is the excess that matters, not the total:

go 
// pkg/kubelet/eviction/helpers.go:711
// memory compares pods by largest consumer of memory relative to request.
func memory(stats statsFunc) cmpFunc {
	return func(p1, p2 *v1.Pod) int {
		p1Stats, p1Found := stats(p1)
		p2Stats, p2Found := stats(p2)
		if !p1Found || !p2Found {
			// prioritize evicting the pod for which no stats were found
			return cmpBool(!p1Found, !p2Found)
		}

		// adjust p1, p2 usage relative to the request (if any)
		p1Memory := memoryUsage(p1Stats.Memory)
		p1Request := v1resource.GetResourceRequestQuantity(p1, v1.ResourceMemory)
		p1Memory.Sub(p1Request)

		p2Memory := memoryUsage(p2Stats.Memory)
		p2Request := v1resource.GetResourceRequestQuantity(p2, v1.ResourceMemory)
		p2Memory.Sub(p2Request)

		// prioritize evicting the pod which has the larger consumption of memory
		return p2Memory.Cmp(*p1Memory)
	}
}

p2Memory.Cmp(*p1Memory) returns -1 when pod 1 has the bigger excess, so the biggest offender among the tied set comes first.

Put the three together and you have the whole policy in one line:

go 
// pkg/kubelet/eviction/helpers.go:816
// rankMemoryPressure orders the input pods for eviction in response to memory pressure.
// It ranks by whether or not the pod's usage exceeds its requests, then by priority, and
// finally by memory usage above requests.
func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) {
	orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods)
}

One pod per interval, one second of grace

The ranking only decides the order. The eviction manager walks that order and stops at the first pod it can actually kill:

go 
// pkg/kubelet/eviction/eviction_manager.go:428
	// we kill at most a single pod during each eviction interval
	for i := range activePods {
		pod := activePods[i]
		gracePeriodOverride := int64(immediateEvictionGracePeriodSeconds)
		if !isHardEvictionThreshold(thresholdToReclaim) {
			gracePeriodOverride = m.config.MaxPodGracePeriodSeconds
			if pod.Spec.TerminationGracePeriodSeconds != nil {
				gracePeriodOverride = min(m.config.MaxPodGracePeriodSeconds, *pod.Spec.TerminationGracePeriodSeconds)
			}
		}

Two constants matter here. A hard threshold — one with a zero grace period, which is every eviction-hard entry — gets a one-second termination grace period regardless of what the pod asked for:

go 
// pkg/kubelet/eviction/eviction_manager.go:53
const (
	// signalEphemeralContainerFsLimit is amount of storage available on filesystem requested by the container
	signalEphemeralContainerFsLimit string = "ephemeralcontainerfs.limit"
	// signalEphemeralPodFsLimit is amount of storage available on filesystem requested by the pod
	signalEphemeralPodFsLimit string = "ephemeralpodfs.limit"
	// signalEmptyDirFsLimit is amount of storage available on filesystem requested by an emptyDir
	signalEmptyDirFsLimit string = "emptydirfs.limit"
	// immediateEvictionGracePeriodSeconds is how long we give pods to shut down when we
	// need them to evict quickly due to resource pressure
	immediateEvictionGracePeriodSeconds = 1
)

A soft threshold uses the pod’s own terminationGracePeriodSeconds, capped by MaxPodGracePeriodSeconds (the evictionMaxPodGracePeriod field in the kubelet config file, 0 meaning “no cap”).

And “one pod per interval” is literal. After a successful eviction the manager waits for the pod to be cleaned up before it even considers the next one, and that wait is bounded:

go 
// pkg/kubelet/eviction/eviction_manager.go:48
const (
	podCleanupTimeout  = 30 * time.Second
	podCleanupPollFreq = time.Second
)

So under a sustained memory crunch a node loses one pod every few seconds at most, not dozens at once. The default eviction-hard thresholds on Linux are:

go 
// pkg/kubelet/eviction/defaults_linux.go:21
// DefaultEvictionHard includes default options for hard eviction.
var DefaultEvictionHard = map[string]string{
	"memory.available":   "100Mi",
	"nodefs.available":   "10%",
	"nodefs.inodesFree":  "5%",
	"imagefs.available":  "15%",
	"imagefs.inodesFree": "5%",
}

and the monitoring loop that drives all of this ticks every ten seconds:

go 
// pkg/kubelet/kubelet.go:195
	// Period for performing eviction monitoring.
	// ensure this is kept in sync with internal cadvisor housekeeping.
	evictionMonitoringPeriod = time.Second * 10

Before the ranking even runs, the manager tries to reclaim node-level resources — deleting unused images, vacuuming logs — and only escalates to killing pods if that is not enough:

go 
// pkg/kubelet/eviction/eviction_manager.go:480
// reclaimNodeLevelResources attempts to reclaim node level resources.  returns true if thresholds were satisfied and no pod eviction is required.
func (m *managerImpl) reclaimNodeLevelResources(ctx context.Context, signalToReclaim evictionapi.Signal, resourceToReclaim v1.ResourceName) bool {
	logger := klog.FromContext(ctx)
	nodeReclaimFuncs := m.signalToNodeReclaimFuncs[signalToReclaim]
	for _, nodeReclaimFunc := range nodeReclaimFuncs {
		// attempt to reclaim the pressured resource.
		if err := nodeReclaimFunc(ctx); err != nil {
			logger.Info("Eviction manager: unexpected error when attempting to reduce resource pressure", "resourceName", resourceToReclaim, "err", err)
		}

	}

The same combinator, three signals

The memory chain is not special. PID pressure uses two keys, with priority first because there is no meaningful “request exceeded” notion for PIDs:

go 
// pkg/kubelet/eviction/helpers.go:823
// rankPIDPressure orders the input pods by priority in response to PID pressure.
func rankPIDPressure(pods []*v1.Pod, stats statsFunc) {
	orderedBy(priority, process(stats)).Sort(pods)
}

// rankDiskPressureFunc returns a rankFunc that measures the specified fs stats.
func rankDiskPressureFunc(fsStatsToMeasure []fsStatsType, diskResource v1.ResourceName) rankFunc {
	return func(pods []*v1.Pod, stats statsFunc) {
		orderedBy(exceedDiskRequests(stats, fsStatsToMeasure, diskResource), priority, disk(stats, fsStatsToMeasure, diskResource)).Sort(pods)
	}
}

Disk pressure has exactly the same shape as memory pressure: exceed your ephemeral-storage request, then priority, then absolute excess.

The filesystem layout changes the stat set

Which bytes count toward “your disk usage” depends on how the node’s filesystem is laid out, and the ranking functions are built accordingly at startup:

go 
// pkg/kubelet/eviction/helpers.go:1150
// buildSignalToRankFunc returns ranking functions associated with resources
func buildSignalToRankFunc(withImageFs bool, imageContainerSplitFs bool) map[evictionapi.Signal]rankFunc {
	signalToRankFunc := map[evictionapi.Signal]rankFunc{
		evictionapi.SignalMemoryAvailable:            rankMemoryPressure,
		evictionapi.SignalAllocatableMemoryAvailable: rankMemoryPressure,
		evictionapi.SignalPIDAvailable:               rankPIDPressure,
	}
	// usage of an imagefs is optional
	// We have a dedicated Image filesystem (images and containers are on same disk)
	// then we assume it is just a separate imagefs
	if withImageFs && !imageContainerSplitFs {
		// with an imagefs, nodefs pod rank func for eviction only includes logs and local volumes
		signalToRankFunc[evictionapi.SignalNodeFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsLogs, fsStatsLocalVolumeSource}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalNodeFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsLogs, fsStatsLocalVolumeSource}, resourceInodes)
		// with an imagefs, imagefs pod rank func for eviction only includes rootfs
		signalToRankFunc[evictionapi.SignalImageFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsRoot, fsStatsImages}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalImageFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsRoot, fsStatsImages}, resourceInodes)
		signalToRankFunc[evictionapi.SignalContainerFsAvailable] = signalToRankFunc[evictionapi.SignalImageFsAvailable]
		signalToRankFunc[evictionapi.SignalContainerFsInodesFree] = signalToRankFunc[evictionapi.SignalImageFsInodesFree]

		// If both imagefs and container fs are on separate disks
		// we want to track the writeable layer in containerfs signals.
	} else if withImageFs && imageContainerSplitFs {

Read that carefully: with a dedicated image filesystem, nodefs pressure only ranks pods by logs and local volumes, and imagefs pressure ranks by rootfs and images. A pod with a fat writable layer does not show up in the nodefs ranking at all, because its layer lives on the image disk. Without a dedicated imagefs everything shares one device and everything counts:

go 
// pkg/kubelet/eviction/helpers.go:1183
	} else {
		// without an imagefs, nodefs pod rank func for eviction looks at all fs stats.
		// since imagefs and nodefs share a common device, they share common ranking functions.
		signalToRankFunc[evictionapi.SignalNodeFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalNodeFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, resourceInodes)
		signalToRankFunc[evictionapi.SignalImageFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalImageFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, resourceInodes)
		signalToRankFunc[evictionapi.SignalContainerFsAvailable] = signalToRankFunc[evictionapi.SignalNodeFsAvailable]
		signalToRankFunc[evictionapi.SignalContainerFsInodesFree] = signalToRankFunc[evictionapi.SignalNodeFsInodesFree]
	}
	return signalToRankFunc
}

The split between the three cases is discovered once, from the kubelet’s disk info provider, and cached in m.dedicatedImageFs / m.splitContainerImageFs. It is not re-evaluated per eviction.

What the ranking is not

There is no QoS class in this chain. A Burstable pod and a Guaranteed pod are treated identically — the only thing that distinguishes them is whether their working set exceeds their request and what their priority is. The QoS class is, in effect, a derived opinion about the same request number, and the kubelet uses the raw number instead.

There is also no awareness of which pod caused the pressure. Nothing in the ranking looks at when memory went bad or at the node’s other consumers. A pod that has been running at 90% of its request for a week and a pod that blew past its request ten seconds ago are separated only by the tiebreak key.

Wrapping up

The one thing to remember: the eviction order is exceedRequest → priority → excess-above-request, compared lexicographically, and the first key outranks everything else. The kubelet kills the pod that misbehaved, not the pod that happened to be large. If you want a pod to survive memory pressure, give it a request it can live within and a priorityClassName you are willing to defend.

The part people get wrong: “the kubelet evicts the biggest pod” and “the kubelet evicts the lowest-priority pod” are both wrong on their own. A pod at 4 GiB with a 1 GiB request is evicted ahead of a pod at 900 MiB with a 1 GiB request, regardless of priority. Priority only breaks ties between pods that are equally guilty of exceeding their requests.

What to go do about it: audit your requests against actual usage — if container_memory_working_set_bytes routinely sits above the container’s resources.requests.memory, that pod is the first eviction candidate no matter how important it is, so either raise the request or lower the usage. Watch kubelet_evictions{eviction_signal="memory.available"} and kubelet_eviction_stats_age_seconds{eviction_signal="memory.available"} on the node, and tune the evictionHard / evictionSoft / evictionSoftGracePeriod / evictionMinimumReclaim fields of the kubelet configuration file to give soft thresholds time to absorb a spike before a hard one kills someone. If you need the pod to survive a hard eviction, use a priorityClassName with a high value and make it fit inside its request.

KubernetesInternalsKubelet