Opensource internals

A Continuous OOM Score: Why Burstable Containers Don't All Get 0

16 min read Jay Hubbert

The kubelet's QoS policy computes a continuous oom_score_adj for Burstable containers from their memory request, so a container that exceeds its request climbs toward 1000 and becomes the prime kill target.

TL;DR: The naive reading of Kubernetes QoS is “Guaranteed gets -997, Burstable gets 0, BestEffort gets 1000”. That is not what the kubelet does. Burstable containers get a continuous score of 1000 - (1000 * containerMemRequest / nodeMemoryCapacity). A container requesting 10% of node memory scores 900; if it then uses more than its request its effective score approaches 1000 and it becomes the prime OOM kill target. The floor is 1000 + (-997) = 3, so even a Guaranteed pod using 100% of memory outranks every Burstable pod.

The Linux OOM killer does not know about Kubernetes. It looks at each process’s oom_score, a number in the range 0–1000 that the kernel computes roughly as “the percentage of memory this task consumes, times 10”. A process’s effective score is that plus its oom_score_adj, which can be negative. When memory runs out, the kernel kills the process with the highest effective score.

The kubelet’s job is to decide what oom_score_adj to write into each container’s cgroup. That single number is the entire mechanism by which Kubernetes QoS classes turn into “this container died instead of that one”. Getting it wrong is not a subtle performance problem — it is the wrong container getting SIGKILLed.

The constants

Everything starts here:

go 
// pkg/kubelet/qos/policy.go:30
const (
	// KubeletOOMScoreAdj is the OOM score adjustment for Kubelet
	KubeletOOMScoreAdj int = -999
	// KubeProxyOOMScoreAdj is the OOM score adjustment for kube-proxy
	KubeProxyOOMScoreAdj  int = -999
	guaranteedOOMScoreAdj int = -997
	besteffortOOMScoreAdj int = 1000
)

Four numbers, and they are deliberately not symmetric:

  • KubeletOOMScoreAdj and KubeProxyOOMScoreAdj are -999. The kubelet and kube-proxy run as host processes, not in containers, and they must never be the OOM killer’s target — if the kubelet dies, the node goes down entirely.
  • guaranteedOOMScoreAdj is -997. Two points more than -999, so Guaranteed containers are the next thing to die, but still very late.
  • besteffortOOMScoreAdj is 1000. The maximum the kernel allows. A BestEffort container is the first candidate for death, every time.

The gap between -999 and -997 is two points, and it is deliberate. The relative order is what matters, not the absolute values, and the kubelet needs to sit strictly below every container it might have to kill.

The formula

The doc comment on the function explains the whole design:

go 
// pkg/kubelet/qos/policy.go:39
// GetContainerOOMScoreAdjust returns the amount by which the OOM score of all processes in the
// container should be adjusted.
// The OOM score of a process is the percentage of memory it consumes
// multiplied by 10 (barring exceptional cases) + a configurable quantity which is between -1000
// and 1000. Containers with higher OOM scores are killed if the system runs out of memory.
// See https://lwn.net/Articles/391222/ for more information.
// OOMScoreAdjust should be calculated based on the allocated resources, so the pod argument should
// contain the allocated resources in the spec.

Note the last sentence: the score is computed from the allocated (that is, requested) resources, not from live usage. The kubelet cannot know at container-start time how much the container will actually use.

And then the three-way switch:

go 
// pkg/kubelet/qos/policy.go:47
func GetContainerOOMScoreAdjust(pod *v1.Pod, container *v1.Container, memoryCapacity int64) int {
	if types.IsNodeCriticalPod(pod) {
		// Only node critical pod should be the last to get killed.
		return guaranteedOOMScoreAdj
	}

	switch v1qos.GetPodQOS(pod) {
	case v1.PodQOSGuaranteed:
		// Guaranteed containers should be the last to get killed.
		return guaranteedOOMScoreAdj
	case v1.PodQOSBestEffort:
		return besteffortOOMScoreAdj
	}

Node-critical pods first — a pod the kubelet marks as node-critical gets -997 regardless of its QoS class, so it is killed last. Then Guaranteed gets -997, BestEffort gets 1000.

Everything else falls through to the Burstable branch, and that is where the interesting arithmetic lives:

go 
// pkg/kubelet/qos/policy.go:61
	// Burstable containers are a middle tier, between Guaranteed and Best-Effort. Ideally,
	// we want to protect Burstable containers that consume less memory than requested.
	// The formula below is a heuristic. A container requesting for 10% of a system's
	// memory will have an OOM score adjust of 900. If a process in container Y
	// uses over 10% of memory, its OOM score will be 1000. The idea is that containers
	// which use more than their request will have an OOM score of 1000 and will be prime
	// targets for OOM kills.
	// Note that this is a heuristic, it won't work if a container has many small processes.
	containerMemReq := getEffectiveContainerMemoryRequest(pod, container)

	var oomScoreAdjust, remainingReqPerContainer int64

Read that comment twice, because it is the clearest statement of intent in the file. “A container requesting for 10% of a system’s memory will have an OOM score adjust of 900.” The formula is:

go 
// pkg/kubelet/qos/policy.go:79
	if utilfeature.DefaultFeatureGate.Enabled(features.PodLevelResources) &&
		resourcehelper.IsPodLevelRequestsSet(pod) {
		// TODO(ndixita): Refactor to use this formula in all cases, as
		// remainingReqPerContainer will be 0 when pod-level resources are not set.
		remainingReqPerContainer = remainingPodMemReqPerContainer(pod)
		oomScoreAdjust = 1000 - (1000 * (containerMemReq + remainingReqPerContainer) / memoryCapacity)
	} else {
		oomScoreAdjust = 1000 - (1000*containerMemReq)/memoryCapacity
	}

1000 - (1000 * containerMemReq / memoryCapacity).

Work through the arithmetic on a 16 GiB node:

  • Container requests 1 GiB (6.25%): 1000 - (1000 * 1 / 16) = 1000 - 62 = 938.
  • Container requests 4 GiB (25%): 1000 - 250 = 750.
  • Container requests 8 GiB (50%): 1000 - 500 = 500.
  • Container requests 16 GiB (100%): 1000 - 1000 = 0.

So the adjustment decreases as the request grows. A container that requested a large slice of the node has a low adjustment, and therefore a low effective score, and therefore survives.

But wait — the comment says a container requesting 10% gets 900, and if it uses over 10% its OOM score is 1000. How does usage get into this? It does not, directly. The trick is that the kernel’s base score is the percentage of memory consumed, scaled to the 0–1000 range. A container that requested 10% of memory and is using exactly its request has a base score of 100 and an adjustment of 900, for an effective score of 1000. The moment it uses more than its request, its base score rises above 100 and its effective score goes above 1000 — above every other container on the node.

That is the whole point. The adjustment is calibrated so that a Burstable container sitting exactly at its request has an effective score of exactly 1000, and over-consuming pushes it past 1000 into being the sole prime target. The kernel’s own base score does the “am I using too much” arithmetic at runtime; the kubelet’s job is only to set the offset.

Check the formula: base score = 10 × (usage%/100 as a fraction)… let me be precise. The kernel’s base score is roughly usage_fraction * 1000 in modern kernels (the LWN article describes the older 10 × percent formulation). Either way, the calibration is the same: the adjustment 1000 - 1000 * request_fraction means that at exactly the requested fraction, the effective score saturates at the top of the range.

A container that requests less has a higher adjustment, so it can absorb some overshoot before it becomes attractive. A container that requests more has a lower adjustment and is protected.

The last line of the comment is the honest caveat: “Note that this is a heuristic, it won’t work if a container has many small processes.” A container running 1000 processes has each process’s individual usage counted, not the container’s total.

The floor

Now the clamps. The first one is the Guaranteed floor:

go 
// pkg/kubelet/qos/policy.go:112
	// A guaranteed pod using 100% of memory can have an OOM score of 10. Ensure
	// that burstable pods have a higher OOM score adjustment.
	if int(oomScoreAdjust) < (1000 + guaranteedOOMScoreAdj) {
		return (1000 + guaranteedOOMScoreAdj)
	}

1000 + guaranteedOOMScoreAdj = 1000 + (-997) = 3.

So a Burstable container’s adjustment can never go below 3. Why 3 and not -997? Because the comment explains it: “A guaranteed pod using 100% of memory can have an OOM score of 10.” A Guaranteed container using 100% of the node’s memory has a base score of 1000 and an adjustment of -997, for an effective score of 3. For the ordering to hold — Guaranteed must always die after Burstable — the Burstable floor must be at least 3. It is exactly 3, which means a Guaranteed pod at 100% and a Burstable pod requesting 100% both land at an effective score of 3, and the kernel breaks the tie arbitrarily. That is acceptable: in that scenario the Burstable container is using everything it asked for, so there is no “misbehaving” pod to distinguish.

The important consequence: a Burstable container that requests the entire node gets an adjustment of 3, not 0 and not -997. It is still ranked strictly above every other Burstable container that requests less, and it can never outrank a Guaranteed container.

The sidecar clamp, and the BestEffort tie-break

Now the sidecar twist, and it is the cleverest line in the file:

go 
// pkg/kubelet/qos/policy.go:89
	// adapt the sidecarContainer memoryRequest for OOM ADJ calculation
	// calculate the oom score adjustment based on: max-memory( currentSideCarContainer , min-memory(regular containers) ) .
	if isSidecarContainer(pod, container) {
		// check min memory quantity in regular containers
		minMemoryRequest := minRegularContainerMemory(pod)

A sidecar is a restartable init container — one with restartPolicy: Always in spec.initContainers. The helper that identifies it:

go 
// pkg/kubelet/qos/policy.go:124
// isSidecarContainer returns a boolean indicating whether a container is a sidecar or not.
// Since v1.Container does not directly specify whether a container is a sidecar,
// this function uses available indicators (container.RestartPolicy == v1.ContainerRestartPolicyAlways)
// to make that determination.
func isSidecarContainer(pod *v1.Pod, container *v1.Container) bool {
	if container.RestartPolicy != nil && *container.RestartPolicy == v1.ContainerRestartPolicyAlways {
		for _, initContainer := range pod.Spec.InitContainers {
			if initContainer.Name == container.Name {
				return true
			}
		}
	}
	return false
}

The problem the clamp solves: a sidecar typically requests very little memory. Under the normal formula, a sidecar requesting 64 MiB on a 16 GiB node gets an adjustment of 1000 - 3 = 997. It would be the second most attractive OOM kill target on the node, ahead of every real workload. And the sidecar is infrastructure — a log shipper, a proxy, an agent. If it dies, the pod’s actual work keeps running but its plumbing is gone, and the pod never becomes Ready.

So the sidecar’s adjustment is clamped down to what the pod’s smallest regular container would get:

go 
// pkg/kubelet/qos/policy.go:104
		minMemoryOomScoreAdjust := 1000 - (1000*minMemoryRequest)/memoryCapacity
		// the OOM adjustment for sidecar container will match
		// or fall below the OOM score adjustment of regular containers in the Pod.
		if oomScoreAdjust > minMemoryOomScoreAdjust {
			oomScoreAdjust = minMemoryOomScoreAdjust
		}
	}

If the sidecar’s own adjustment (997) is higher than the minimum regular container’s adjustment (say the smallest regular container requests 1 GiB, giving 938), the sidecar’s is lowered to 938. The sidecar can never be ranked as more attractive than the smallest real workload in its own pod.

The minimum is taken over the regular containers only — init containers are explicitly excluded:

go 
// pkg/kubelet/qos/helpers.go:35
// minRegularContainerMemory returns the minimum memory resource quantity
// across all regular containers in pod.Spec.Containers.
// It does not include initContainers (both restartable and non-restartable).
func minRegularContainerMemory(pod *v1.Pod) int64 {
	memoryValue := getEffectiveContainerMemoryRequest(pod, &pod.Spec.Containers[0])
	for i := 1; i < len(pod.Spec.Containers); i++ {
		effectiveMem := getEffectiveContainerMemoryRequest(pod, &pod.Spec.Containers[i])
		if effectiveMem < memoryValue {
			memoryValue = effectiveMem
		}
	}
	return memoryValue
}

Excluding init containers matters: a sidecar is itself an init container, so including them would let a sidecar define its own floor and defeat the whole mechanism.

The BestEffort tie-break. The last clamp is the one people miss:

go 
// pkg/kubelet/qos/policy.go:117
	// Give burstable pods a higher chance of survival over besteffort pods.
	if int(oomScoreAdjust) == besteffortOOMScoreAdj {
		return int(oomScoreAdjust - 1)
	}
	return int(oomScoreAdjust)

If a Burstable container’s adjustment comes out as exactly 1000, it is returned as 999.

When does a Burstable container hit 1000? When its request is zero — but a zero-request container would be classified BestEffort, not Burstable… unless the pod has other containers that are Burstable, in which case the pod’s QoS class is Burstable and every container in it, including the zero-request one, goes through the Burstable branch. Or: the request is non-zero but so small that 1000 * containerMemReq / memoryCapacity truncates to zero.

That is the real reason for the -1: the Burstable branch can produce exactly 1000, and a Burstable container must never tie with a BestEffort container (which is hardcoded at exactly 1000). The -1 guarantees BestEffort always loses the tiebreak, because the kernel kills the highest score and, on a tie, the order is arbitrary.

This is a one-line fix for a class of non-deterministic behaviour, and it is the kind of thing that only shows up in production under memory pressure.

Pod-level resources and DRA

Two newer features perturb the formula. Pod-level resources let a pod declare a memory request at the pod level, in addition to per-container requests. The unallocated remainder is divided equally among the containers:

go 
// pkg/kubelet/qos/helpers.go:49
// remainingPodMemReqPerContainer calculates the remaining pod memory request per
// container by:
// 1. Taking the total pod memory requests
// 2. Subtracting total container memory requests and any DRA memory allocations from pod memory requests
// 3. Dividing the remainder by the number of containers.
// This gives us the additional memory request that is not allocated to any
// containers in the pod. This value will be divided equally among all containers to
// calculate oom score adjustment.
// See https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2837-pod-level-resource-spec/README.md#oom-score-adjustment
// for more details.
func remainingPodMemReqPerContainer(pod *v1.Pod) int64 {

The rationale: a pod-level request is a promise about the pod as a whole. Distributing the unallocated portion equally across the containers means each container’s effective request reflects its share of that promise, and the score calculation stays consistent with how the request was actually made.

Dynamic Resource Allocation changes the effective container request:

go 
// pkg/kubelet/qos/policy.go:139
// getEffectiveContainerMemoryRequest returns the effective container memory request,
// including DRA memory allocations if the feature is enabled.
// If multiple containers share a single DRA memory claim, the claim's memory quantity
// is divided equally among the sharing containers.
func getEffectiveContainerMemoryRequest(pod *v1.Pod, container *v1.Container) int64 {

A DRA memory claim that is shared between two containers contributes half its quantity to each; PerPod overhead is split, PerContainer overhead is not. The point is the same as everywhere else in the kubelet: the OOM score must be computed against the memory the container will actually be accounted for, not against what it asked for in its own spec.

Where the number is applied. The score is written into the container’s OCI runtime spec:

go 
// pkg/kubelet/kuberuntime/kuberuntime_container_linux.go:194
	lcr.OomScoreAdj = int64(qos.GetContainerOOMScoreAdjust(pod, container,
		int64(m.machineInfo.MemoryCapacity)))

memoryCapacity is the node’s memory capacity, not the container’s limit. That is what makes the formula a fraction of the whole machine.

And the kubelet’s own score is validated to the kernel’s legal range:

go 
// pkg/kubelet/apis/config/validation/validation.go:161
	if utilvalidation.IsInRange(int(kc.OOMScoreAdj), -1000, 1000) != nil {
		allErrors = append(allErrors, fmt.Errorf("invalid configuration: oomScoreAdj (--oom-score-adj) %v must be between -1000 and 1000, inclusive", kc.OOMScoreAdj))
	}

--oom-score-adj must be in [-1000, 1000]. The default is -999, which puts the kubelet just below every container it manages.

The eviction ranking is a different mechanism

It is worth being precise about the difference, because “OOM killed” and “evicted” are different events with different causes.

The kubelet’s eviction manager ranks pods for eviction like this:

go 
// pkg/kubelet/eviction/helpers.go:816
// rankMemoryPressure orders the input pods for eviction in response to memory pressure.
// It ranks by whether or not the pod's usage exceeds its requests, then by priority, and
// finally by memory usage above requests.
func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) {
	orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods)
}

Eviction ranks by usage above request — the same signal the OOM adjustment encodes, but computed from live statistics and used to pick a pod to delete (with a graceful termination, respecting terminationGracePeriodSeconds).

The OOM killer ranks by the kernel’s effective score and sends SIGKILL. No grace period, no preStop hook, no cleanup.

They agree on which pod is misbehaving — the one exceeding its request — but they act at different layers: eviction is memory.available crossing eviction-hard/eviction-soft thresholds; the OOM killer is memory.max on the cgroup, or a global node-level OOM.

The trade-offs.

  • It is a heuristic based on requests, not usage. The kubelet cannot know at start time how much a container will use, so it calibrates for the request and lets the kernel’s base score do the rest.
  • The base score is per-process. A container with many processes fragments its memory footprint across them, and the per-process base scores do not add up to the container’s total usage. The source comment says so explicitly.
  • The floor at 3 means a huge Burstable request ties with a Guaranteed pod at 100%. The tie is broken arbitrarily by the kernel, but the scenario is degenerate anyway.
  • Node memory capacity is the denominator, not the container’s limit. On a node with a huge amount of memory but a tight cgroup limit, the calibration is wrong: a container requesting 1% of a 1 TiB node but limited to 1 GiB gets an adjustment near 1000 and is a prime target even though it is well behaved.
  • The sidecar clamp uses the minimum regular container. A pod with one tiny regular container drags the sidecar’s floor down for all of them.

Wrapping up

The one thing to remember: Burstable containers get a continuous oom_score_adj of 1000 - (1000 * containerMemRequest / nodeMemoryCapacity) (pkg/kubelet/qos/policy.go:86), floored at 3 and capped at 999. A container that respects its request sits at an effective score of 1000; one that exceeds it climbs past that and becomes the prime OOM kill target. That is the mechanism that makes over-consuming Burstable pods die before well-behaved ones.

The part people get wrong: that QoS class maps to a fixed oom_score_adj. It does not — only Guaranteed (-997) and BestEffort (1000) are fixed. And separately: the eviction ranking (pkg/kubelet/eviction/helpers.go:816) and the OOM score are different mechanisms. Eviction deletes a pod gracefully when memory.available crosses a threshold; the OOM killer sends SIGKILL when the cgroup’s memory.max is hit. A pod that is “OOMKilled” was not evicted, and its terminationGracePeriodSeconds did not apply.

What to go do about it: if a pod is being OOMKilled, read its resources.requests.memory against the node’s capacity (kubectl describe node → Capacity.memory) and compute the expected adjustment yourself. If the pod is Burstable and its request is a small fraction of the node, its adjustment is high (near 1000) and it is one memory hog away from being the first victim. If it is a sidecar, check whether the pod’s smallest regular container has a tiny request — that drags the sidecar’s floor down too. To keep a critical workload alive, give it a Guaranteed QoS class (equal CPU and memory requests and limits on every container), which pins its adjustment at -997 and puts it last in line. And remember the kernel’s range: the kubelet’s own --oom-score-adj must be in [-1000, 1000] or the kubelet refuses to start.

KubernetesInternalsKubelet