Kubelet on Opensource internals

The Kubelet Evicts the Pod That Misbehaves First: A Three-Key Ranking

Jay Hubbert — Thu, 22 Oct 2026 10:40:00 +0000

TL;DR: When the kubelet has to evict under memory pressure it does not kill the biggest pod. It runs a lexicographic sort with three keys in a fixed order: (1) does the pod’s working set exceed its memory request, (2) the pod’s priority, (3) the absolute usage above the request. The first key is the interesting one: a pod that has blown past what it asked for is evicted before a pod with lower priority that is still within its request. The same three-key combinator (orderedBy/multiSorter) serves disk pressure and PID pressure, only with the keys reshuffled.

Everyone has watched it happen. A node goes into MemoryPressure, one pod disappears, and the postmortem question is always the same: why that one? The intuitive answer — “it was the biggest” — is wrong, and it is wrong in an interesting way rather than an arbitrary way.

The kubelet’s eviction manager is a small, self-contained ranking machine. It collects stats, builds a per-signal ranking function once at startup, sorts the candidate pods with it, and kills exactly one. The ranking function is where all the policy lives, and the policy is not “biggest first”.

The chain, not the heap

The whole ranking abstraction is thirty lines. A multiSorter holds a list of comparison functions and applies them in order, stopping at the first one that produces a non-zero answer:

 go
 
 
// pkg/kubelet/eviction/helpers.go:629
// multiSorter implements the Sort interface, sorting changes within.
type multiSorter struct {
	pods []*v1.Pod
	cmp []cmpFunc
}

// Sort sorts the argument slice according to the less functions passed to OrderedBy.
func (ms *multiSorter) Sort(pods []*v1.Pod) {
	ms.pods = pods
	sort.Sort(ms)
}

// OrderedBy returns a Sorter that sorts using the cmp functions, in order.
// Call its Sort method to sort the data.
func orderedBy(cmp ...cmpFunc) *multiSorter {
	return &multiSorter{
		cmp: cmp,
	}
}

and the Less implementation is a textbook lexicographic chain:

 go
 
 
// pkg/kubelet/eviction/helpers.go:659
// Less is part of sort.Interface.
func (ms *multiSorter) Less(i, j int) bool {
	p1, p2 := ms.pods[i], ms.pods[j]
	var k int
	for k = 0; k < len(ms.cmp)-1; k++ {
		cmpResult := ms.cmp[k](p1, p2)
		// p1 is less than p2
		if cmpResult < 0 {
			return true
		}
		// p1 is greater than p2
		if cmpResult > 0 {
			return false
		}
		// we don't know yet
	}
	// the last cmp func is the final decider
	return ms.cmp[k](p1, p2) < 0
}

Because sort.Sort is ascending, the pod at index 0 of the sorted slice is the one that gets killed. Each cmpFunc returns -1 when pod 1 should be evicted first, +1 when pod 2 should be evicted first, and 0 when the key is a tie and the next key should decide.

Three helpers do the actual comparing, and they all share one shape: if the stats for either pod are missing, the pod with missing stats is preferred.

 go
 
 
// pkg/kubelet/eviction/helpers.go:805
// cmpBool compares booleans, placing true before false
func cmpBool(a, b bool) int {
	if a == b {
		return 0
	}
	if !b {
		return -1
	}
	return 1
}

That cmpBool is the trick that makes the first key work. cmpBool(true, false) is -1: true sorts first. So a boolean predicate — “exceeds its request” — is enough to dominate the entire rest of the chain.

Key one: over your request, not over the node

The first comparison in the memory chain is exceedMemoryRequests:

 go
 
 
// pkg/kubelet/eviction/helpers.go:692
// exceedMemoryRequests compares whether or not pods' memory usage exceeds their requests
func exceedMemoryRequests(stats statsFunc) cmpFunc {
	return func(p1, p2 *v1.Pod) int {
		p1Stats, p1Found := stats(p1)
		p2Stats, p2Found := stats(p2)
		if !p1Found || !p2Found {
			// prioritize evicting the pod for which no stats were found
			return cmpBool(!p1Found, !p2Found)
		}

		p1Memory := memoryUsage(p1Stats.Memory)
		p2Memory := memoryUsage(p2Stats.Memory)
		p1ExceedsRequests := p1Memory.Cmp(v1resource.GetResourceRequestQuantity(p1, v1.ResourceMemory)) == 1
		p2ExceedsRequests := p2Memory.Cmp(v1resource.GetResourceRequestQuantity(p2, v1.ResourceMemory)) == 1
		// prioritize evicting the pod which exceeds its requests
		return cmpBool(p1ExceedsRequests, p2ExceedsRequests)
	}
}

Note == 1: strictly greater than the request. A pod sitting exactly at its request is not “exceeding” it and does not get the free pass to the front of the queue.

And note what “usage” means here. It is the working set, not the RSS:

 go
 
 
// pkg/kubelet/eviction/helpers.go:514
// memoryUsage converts working set into a resource quantity.
func memoryUsage(memStats *statsapi.MemoryStats) *resource.Quantity {
	if memStats == nil || memStats.WorkingSetBytes == nil {
		return &resource.Quantity{Format: resource.BinarySI}
	}
	usage := int64(*memStats.WorkingSetBytes)
	return resource.NewQuantity(usage, resource.BinarySI)
}

Working set excludes reclaimable page cache, which is exactly the memory the kernel can take back without killing anybody. Comparing working set against the request is therefore the honest question: “is this container using more than it promised, in memory the kernel cannot just claw back?”

The request side comes from the pod spec, and it is the sum of container requests plus the pod overhead:

 go
 
 
// pkg/api/v1/resource/helpers.go:32
// GetResourceRequestQuantity finds and returns the request quantity for a specific resource.
func GetResourceRequestQuantity(pod *v1.Pod, resourceName v1.ResourceName) resource.Quantity {
	requestQuantity := resource.Quantity{}

	switch resourceName {
	case v1.ResourceCPU:
		requestQuantity = resource.Quantity{Format: resource.DecimalSI}
	case v1.ResourceMemory, v1.ResourceStorage, v1.ResourceEphemeralStorage:
		requestQuantity = resource.Quantity{Format: resource.BinarySI}
	default:
		requestQuantity = resource.Quantity{Format: resource.DecimalSI}
	}

Key two: priority, and only if the first key ties

Priority is the second key, and it is a plain integer comparison:

 go
 
 
// pkg/kubelet/eviction/helpers.go:679
// priority compares pods by Priority, if priority is enabled.
func priority(p1, p2 *v1.Pod) int {
	priority1 := corev1helpers.PodPriority(p1)
	priority2 := corev1helpers.PodPriority(p2)
	if priority1 == priority2 {
		return 0
	}
	if priority1 > priority2 {
		return 1
	}
	return -1
}

Returns +1 when pod 1 has the higher priority, which in the Less chain means pod 2 sorts first. Lower priority is evicted first. That is the expected behaviour — but only among pods that tied on key one. A low-priority pod that is within its request loses to a high-priority pod that is over its request, every single time.

Key three: the absolute excess

The final tiebreak is how far above the request each pod is running. The comparison subtracts the request from the usage, so it is the excess that matters, not the total:

 go
 
 
// pkg/kubelet/eviction/helpers.go:711
// memory compares pods by largest consumer of memory relative to request.
func memory(stats statsFunc) cmpFunc {
	return func(p1, p2 *v1.Pod) int {
		p1Stats, p1Found := stats(p1)
		p2Stats, p2Found := stats(p2)
		if !p1Found || !p2Found {
			// prioritize evicting the pod for which no stats were found
			return cmpBool(!p1Found, !p2Found)
		}

		// adjust p1, p2 usage relative to the request (if any)
		p1Memory := memoryUsage(p1Stats.Memory)
		p1Request := v1resource.GetResourceRequestQuantity(p1, v1.ResourceMemory)
		p1Memory.Sub(p1Request)

		p2Memory := memoryUsage(p2Stats.Memory)
		p2Request := v1resource.GetResourceRequestQuantity(p2, v1.ResourceMemory)
		p2Memory.Sub(p2Request)

		// prioritize evicting the pod which has the larger consumption of memory
		return p2Memory.Cmp(*p1Memory)
	}
}

p2Memory.Cmp(*p1Memory) returns -1 when pod 1 has the bigger excess, so the biggest offender among the tied set comes first.

Put the three together and you have the whole policy in one line:

 go
 
 
// pkg/kubelet/eviction/helpers.go:816
// rankMemoryPressure orders the input pods for eviction in response to memory pressure.
// It ranks by whether or not the pod's usage exceeds its requests, then by priority, and
// finally by memory usage above requests.
func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) {
	orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods)
}

One pod per interval, one second of grace

The ranking only decides the order. The eviction manager walks that order and stops at the first pod it can actually kill:

 go
 
 
// pkg/kubelet/eviction/eviction_manager.go:428
	// we kill at most a single pod during each eviction interval
	for i := range activePods {
		pod := activePods[i]
		gracePeriodOverride := int64(immediateEvictionGracePeriodSeconds)
		if !isHardEvictionThreshold(thresholdToReclaim) {
			gracePeriodOverride = m.config.MaxPodGracePeriodSeconds
			if pod.Spec.TerminationGracePeriodSeconds != nil {
				gracePeriodOverride = min(m.config.MaxPodGracePeriodSeconds, *pod.Spec.TerminationGracePeriodSeconds)
			}
		}

Two constants matter here. A hard threshold — one with a zero grace period, which is every eviction-hard entry — gets a one-second termination grace period regardless of what the pod asked for:

 go
 
 
// pkg/kubelet/eviction/eviction_manager.go:53
const (
	// signalEphemeralContainerFsLimit is amount of storage available on filesystem requested by the container
	signalEphemeralContainerFsLimit string = "ephemeralcontainerfs.limit"
	// signalEphemeralPodFsLimit is amount of storage available on filesystem requested by the pod
	signalEphemeralPodFsLimit string = "ephemeralpodfs.limit"
	// signalEmptyDirFsLimit is amount of storage available on filesystem requested by an emptyDir
	signalEmptyDirFsLimit string = "emptydirfs.limit"
	// immediateEvictionGracePeriodSeconds is how long we give pods to shut down when we
	// need them to evict quickly due to resource pressure
	immediateEvictionGracePeriodSeconds = 1
)

A soft threshold uses the pod’s own terminationGracePeriodSeconds, capped by MaxPodGracePeriodSeconds (the evictionMaxPodGracePeriod field in the kubelet config file, 0 meaning “no cap”).

And “one pod per interval” is literal. After a successful eviction the manager waits for the pod to be cleaned up before it even considers the next one, and that wait is bounded:

 go
 
 
// pkg/kubelet/eviction/eviction_manager.go:48
const (
	podCleanupTimeout = 30 * time.Second
	podCleanupPollFreq = time.Second
)

So under a sustained memory crunch a node loses one pod every few seconds at most, not dozens at once. The default eviction-hard thresholds on Linux are:

 go
 
 
// pkg/kubelet/eviction/defaults_linux.go:21
// DefaultEvictionHard includes default options for hard eviction.
var DefaultEvictionHard = map[string]string{
	"memory.available": "100Mi",
	"nodefs.available": "10%",
	"nodefs.inodesFree": "5%",
	"imagefs.available": "15%",
	"imagefs.inodesFree": "5%",
}

and the monitoring loop that drives all of this ticks every ten seconds:

 go
 
 
// pkg/kubelet/kubelet.go:195
	// Period for performing eviction monitoring.
	// ensure this is kept in sync with internal cadvisor housekeeping.
	evictionMonitoringPeriod = time.Second * 10

Before the ranking even runs, the manager tries to reclaim node-level resources — deleting unused images, vacuuming logs — and only escalates to killing pods if that is not enough:

 go
 
 
// pkg/kubelet/eviction/eviction_manager.go:480
// reclaimNodeLevelResources attempts to reclaim node level resources. returns true if thresholds were satisfied and no pod eviction is required.
func (m *managerImpl) reclaimNodeLevelResources(ctx context.Context, signalToReclaim evictionapi.Signal, resourceToReclaim v1.ResourceName) bool {
	logger := klog.FromContext(ctx)
	nodeReclaimFuncs := m.signalToNodeReclaimFuncs[signalToReclaim]
	for _, nodeReclaimFunc := range nodeReclaimFuncs {
		// attempt to reclaim the pressured resource.
		if err := nodeReclaimFunc(ctx); err != nil {
			logger.Info("Eviction manager: unexpected error when attempting to reduce resource pressure", "resourceName", resourceToReclaim, "err", err)
		}

	}

The same combinator, three signals

The memory chain is not special. PID pressure uses two keys, with priority first because there is no meaningful “request exceeded” notion for PIDs:

 go
 
 
// pkg/kubelet/eviction/helpers.go:823
// rankPIDPressure orders the input pods by priority in response to PID pressure.
func rankPIDPressure(pods []*v1.Pod, stats statsFunc) {
	orderedBy(priority, process(stats)).Sort(pods)
}

// rankDiskPressureFunc returns a rankFunc that measures the specified fs stats.
func rankDiskPressureFunc(fsStatsToMeasure []fsStatsType, diskResource v1.ResourceName) rankFunc {
	return func(pods []*v1.Pod, stats statsFunc) {
		orderedBy(exceedDiskRequests(stats, fsStatsToMeasure, diskResource), priority, disk(stats, fsStatsToMeasure, diskResource)).Sort(pods)
	}
}

Disk pressure has exactly the same shape as memory pressure: exceed your ephemeral-storage request, then priority, then absolute excess.

The filesystem layout changes the stat set

Which bytes count toward “your disk usage” depends on how the node’s filesystem is laid out, and the ranking functions are built accordingly at startup:

 go
 
 
// pkg/kubelet/eviction/helpers.go:1150
// buildSignalToRankFunc returns ranking functions associated with resources
func buildSignalToRankFunc(withImageFs bool, imageContainerSplitFs bool) map[evictionapi.Signal]rankFunc {
	signalToRankFunc := map[evictionapi.Signal]rankFunc{
		evictionapi.SignalMemoryAvailable: rankMemoryPressure,
		evictionapi.SignalAllocatableMemoryAvailable: rankMemoryPressure,
		evictionapi.SignalPIDAvailable: rankPIDPressure,
	}
	// usage of an imagefs is optional
	// We have a dedicated Image filesystem (images and containers are on same disk)
	// then we assume it is just a separate imagefs
	if withImageFs && !imageContainerSplitFs {
		// with an imagefs, nodefs pod rank func for eviction only includes logs and local volumes
		signalToRankFunc[evictionapi.SignalNodeFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsLogs, fsStatsLocalVolumeSource}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalNodeFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsLogs, fsStatsLocalVolumeSource}, resourceInodes)
		// with an imagefs, imagefs pod rank func for eviction only includes rootfs
		signalToRankFunc[evictionapi.SignalImageFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsRoot, fsStatsImages}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalImageFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsRoot, fsStatsImages}, resourceInodes)
		signalToRankFunc[evictionapi.SignalContainerFsAvailable] = signalToRankFunc[evictionapi.SignalImageFsAvailable]
		signalToRankFunc[evictionapi.SignalContainerFsInodesFree] = signalToRankFunc[evictionapi.SignalImageFsInodesFree]

		// If both imagefs and container fs are on separate disks
		// we want to track the writeable layer in containerfs signals.
	} else if withImageFs && imageContainerSplitFs {

Read that carefully: with a dedicated image filesystem, nodefs pressure only ranks pods by logs and local volumes, and imagefs pressure ranks by rootfs and images. A pod with a fat writable layer does not show up in the nodefs ranking at all, because its layer lives on the image disk. Without a dedicated imagefs everything shares one device and everything counts:

 go
 
 
// pkg/kubelet/eviction/helpers.go:1183
	} else {
		// without an imagefs, nodefs pod rank func for eviction looks at all fs stats.
		// since imagefs and nodefs share a common device, they share common ranking functions.
		signalToRankFunc[evictionapi.SignalNodeFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalNodeFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, resourceInodes)
		signalToRankFunc[evictionapi.SignalImageFsAvailable] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, v1.ResourceEphemeralStorage)
		signalToRankFunc[evictionapi.SignalImageFsInodesFree] = rankDiskPressureFunc([]fsStatsType{fsStatsImages, fsStatsRoot, fsStatsLogs, fsStatsLocalVolumeSource}, resourceInodes)
		signalToRankFunc[evictionapi.SignalContainerFsAvailable] = signalToRankFunc[evictionapi.SignalNodeFsAvailable]
		signalToRankFunc[evictionapi.SignalContainerFsInodesFree] = signalToRankFunc[evictionapi.SignalNodeFsInodesFree]
	}
	return signalToRankFunc
}

The split between the three cases is discovered once, from the kubelet’s disk info provider, and cached in m.dedicatedImageFs / m.splitContainerImageFs. It is not re-evaluated per eviction.

What the ranking is not

There is no QoS class in this chain. A Burstable pod and a Guaranteed pod are treated identically — the only thing that distinguishes them is whether their working set exceeds their request and what their priority is. The QoS class is, in effect, a derived opinion about the same request number, and the kubelet uses the raw number instead.

There is also no awareness of which pod caused the pressure. Nothing in the ranking looks at when memory went bad or at the node’s other consumers. A pod that has been running at 90% of its request for a week and a pod that blew past its request ten seconds ago are separated only by the tiebreak key.

Wrapping up

The one thing to remember: the eviction order is exceedRequest → priority → excess-above-request, compared lexicographically, and the first key outranks everything else. The kubelet kills the pod that misbehaved, not the pod that happened to be large. If you want a pod to survive memory pressure, give it a request it can live within and a priorityClassName you are willing to defend.

The part people get wrong: “the kubelet evicts the biggest pod” and “the kubelet evicts the lowest-priority pod” are both wrong on their own. A pod at 4 GiB with a 1 GiB request is evicted ahead of a pod at 900 MiB with a 1 GiB request, regardless of priority. Priority only breaks ties between pods that are equally guilty of exceeding their requests.

What to go do about it: audit your requests against actual usage — if container_memory_working_set_bytes routinely sits above the container’s resources.requests.memory, that pod is the first eviction candidate no matter how important it is, so either raise the request or lower the usage. Watch kubelet_evictions{eviction_signal="memory.available"} and kubelet_eviction_stats_age_seconds{eviction_signal="memory.available"} on the node, and tune the evictionHard / evictionSoft / evictionSoftGracePeriod / evictionMinimumReclaim fields of the kubelet configuration file to give soft thresholds time to absorb a spike before a hard one kills someone. If you need the pod to survive a hard eviction, use a priorityClassName with a high value and make it fit inside its request.

The Kubelet Writes Its State to Disk: FNV-32a Checksums and a String Rename for Rollback

Jay Hubbert — Thu, 22 Oct 2026 10:10:00 +0000

TL;DR: The kubelet persists CPU pinning, memory pinning and device allocations to /var/lib/kubelet through pkg/kubelet/checkpointmanager. Three decisions are non-obvious. The checksum is FNV-32a over a dump.ForHash of the struct, not over the serialized bytes, so it survives a Go field reorder. Writes go to a temp file, get fsynced, and are then renamed into place, so a reader never sees a partial file. And the memory manager’s V3 checkpoint does a literal strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1) on the hash dump, so an old kubelet can still verify the checksum of a new checkpoint and roll back without draining the node.

The kubelet keeps some state that cannot be reconstructed from the API server. Which containers got which exclusive CPUs, which NUMA nodes a container’s memory was pinned to, which devices were handed to which container — that state lives only in the kubelet’s memory, and if it is lost the kubelet cannot honour the guarantees it already made. So it is written to disk.

That sounds mundane. It is not: checkpointing in the kubelet has a long history of production incidents, and the package that implements it carries an explicit warning from the SIG-Node community about adding more of it.

What is actually on disk

Six submodules use the framework. The files are plain JSON in the kubelet’s root directory (--root-dir, default /var/lib/kubelet):

Submodule	File	Constant
CPU manager	`cpu_manager_state`	`pkg/kubelet/cm/cpumanager/cpu_manager.go:57`
Memory manager	`memory_manager_state`	`pkg/kubelet/cm/memorymanager/memory_manager.go:48`
Device manager	`device-plugins/kubelet_internal_checkpoint`	`pkg/kubelet/cm/devicemanager/types.go:128`
DRA manager	`dra_manager_state`	`pkg/kubelet/cm/dra/manager.go:59`
Allocation manager	`allocated_pods_state`	`pkg/kubelet/allocation/allocation_manager.go:48`

The manager itself is tiny — a mutex, a store, and three verbs:

 go
 
 
// pkg/kubelet/checkpointmanager/checkpoint_manager.go:65-92
// CreateCheckpoint persists checkpoint in CheckpointStore.
func (manager *impl) CreateCheckpoint(checkpointKey string, checkpoint Checkpoint) error {
	manager.mutex.Lock()
	defer manager.mutex.Unlock()
	blob, err := checkpoint.MarshalCheckpoint()
	if err != nil {
		return err
	}
	return manager.store.Write(checkpointKey, blob)
}

// GetCheckpoint retrieves checkpoint from CheckpointStore.
func (manager *impl) GetCheckpoint(checkpointKey string, checkpoint Checkpoint) error {
	manager.mutex.Lock()
	defer manager.mutex.Unlock()
	blob, err := manager.store.Read(checkpointKey)
	if err != nil {
		if err == utilstore.ErrKeyNotFound {
			return errors.ErrCheckpointNotFound
		}
		return err
	}
	err = checkpoint.UnmarshalCheckpoint(blob)
	if err == nil {
		err = checkpoint.VerifyChecksum()
	}
	return err
}

The Checkpoint interface is three methods — MarshalCheckpoint, UnmarshalCheckpoint and VerifyChecksum — and the split matters: MarshalCheckpoint computes and embeds the checksum, and VerifyChecksum is a separate call after unmarshalling. That split exists because the checksum is over the struct, not over the bytes — if it were over the bytes, verification would be a hash of the file and could not be done by the struct’s own method.

The checksum is over a struct dump, not the bytes

The generic checksum helper is four lines:

 go
 
 
// pkg/kubelet/checkpointmanager/checksum/checksum.go:26-48
// Checksum is the data to be stored as checkpoint
type Checksum uint64

// Verify verifies that passed checksum is same as calculated checksum
func (cs Checksum) Verify(data interface{}) error {
	actualCS := New(data)
	if cs != actualCS {
		return &errors.CorruptCheckpointError{ActualCS: uint64(actualCS), ExpectedCS: uint64(cs)}
	}
	return nil
}

// New returns the Checksum of checkpoint data
func New(data interface{}) Checksum {
	return Checksum(getChecksum(data))
}

// Get returns calculated checksum of checkpoint data
func getChecksum(data interface{}) uint64 {
	hash := fnv.New32a()
	hashutil.DeepHashObject(hash, data)
	return uint64(hash.Sum32())
}

Checksum is a uint64 that wraps a 32-bit FNV-1a hash. The 64-bit width is cosmetic; the entropy is 32 bits.

DeepHashObject is the interesting part — it dumps the object with spew and feeds the result to the hasher:

 go
 
 
// pkg/util/hash/hash.go:26-32
// DeepHashObject writes specified object to hash using the spew library
// which follows pointers and prints actual values of the nested objects
// ensuring the hash does not change when a pointer changes.
func DeepHashObject(hasher hash.Hash, objectToWrite interface{}) {
	hasher.Reset()
	fmt.Fprintf(hasher, "%v", dump.ForHash(objectToWrite))
}

and dump.ForHash is a spew configuration with two properties that matter enormously:

 go
 
 
// vendor/k8s.io/utils/dump/dump.go:30-38
// The config MUST NOT be changed because that could change the result of a hash operation
var prettyPrintConfigForHash = &spew.ConfigState{
	Indent: " ",
	SortKeys: true,
	DisableMethods: true,
	SpewKeys: true,
	DisablePointerAddresses: true,
	DisableCapacities: true,
}

SortKeys: true is what makes the hash stable across runs: map iteration order in Go is randomised, and without sorting the dump of a struct with a map field would differ every time. DisablePointerAddresses: true means the dump prints the values a pointer points at rather than its address, so the hash does not depend on where the allocator happened to put the object. DisableMethods: true stops String() methods from being invoked, which would let a type change its own hash just by changing a String() method.

The consequence is that the checksum is a hash of the logical content of the struct. Adding a field, removing a field, or reordering fields changes the hash — which sounds like a problem, and is handled by the version migration machinery discussed below. But changing a field’s type in a way that changes its printed representation, or changing the value of a field, also changes the hash, which is exactly what you want.

The alternative — hashing the JSON bytes — would be immune to struct reordering but would break the moment you added a field, because JSON serialisation of the same logical data changes shape. The struct dump is chosen because it is what the readers care about.

Atomic writes: temp file, fsync, rename

The file store is where the durability story lives:

 go
 
 
// pkg/kubelet/util/store/filestore.go:102-123
// writeFile writes data to path in a single transaction.
func writeFile(fs utilfs.Filesystem, path string, data []byte) (retErr error) {
	// Create a temporary file in the base directory of `path` with a prefix.
	tmpFile, err := fs.TempFile(filepath.Dir(path), tmpPrefix)
	if err != nil {
		return err
	}

	tmpPath := tmpFile.Name()
	shouldClose := true

	defer func() {
		// Close the file.
		if shouldClose {
			if err := tmpFile.Close(); err != nil {
				if retErr == nil {
					retErr = fmt.Errorf("close error: %v", err)
				} else {
					retErr = fmt.Errorf("failed to close temp file after error %v; close error: %v", retErr, err)
				}
			}
		}

The defer closes the temp file if it is still open and removes it if the function is returning an error — a half-written temp file left behind would otherwise accumulate on every failed write.

and the last eleven lines are the part that matters:

 go
 
 
// pkg/kubelet/util/store/filestore.go:133-151
	// Write data.
	if _, err := tmpFile.Write(data); err != nil {
		return err
	}

	// Sync file.
	if err := tmpFile.Sync(); err != nil {
		return err
	}

	// Closing the file before renaming.
	err = tmpFile.Close()
	shouldClose = false
	if err != nil {
		return err
	}

	return fs.Rename(tmpPath, path)
}

Three things are happening, and all three are needed:

Write to a temp file in the same directory. The temp file prefix is . (tmpPrefix = "." in pkg/kubelet/util/store/filestore.go:30), and List() skips anything starting with it, so a crashed kubelet’s leftover temp files do not show up as checkpoints.
Sync() before close. The fsync forces the data out of the page cache and onto the disk. Without it, a node that loses power between the write and the rename can end up with a file whose name exists but whose contents are zero bytes.
rename() into place. rename(2) within a filesystem is atomic. A concurrent reader either sees the old file or the new file, never a half-written one. The rename must be on the same filesystem, which is why the temp file is created in filepath.Dir(path) rather than in /tmp.

Key validation

The store keys are restricted, and the restriction is enforced at the store layer rather than trusted from callers:

 go
 
 
// pkg/kubelet/util/store/store.go:24-39
const (
	keyMaxLength = 250

	keyCharFmt string = "[A-Za-z0-9]"
	keyExtCharFmt string = "[-A-Za-z0-9_.]"
	qualifiedKeyFmt string = "(" + keyCharFmt + keyExtCharFmt + "*)?" + keyCharFmt
)

var (
	// Key must consist of alphanumeric characters, '-', '_' or '.', and must start
	// and end with an alphanumeric character.
	keyRegex = regexp.MustCompile("^" + qualifiedKeyFmt + "$")

	// ErrKeyNotFound is the error returned if key is not found in Store.
	ErrKeyNotFound = fmt.Errorf("key is not found")
)

So a key must start and end with an alphanumeric character, may contain -, _ and . in between, and is at most 250 characters long. The characters that are not allowed are exactly the ones that would let a key escape the checkpoint directory: / and ... The key becomes a filename via filepath.Join(f.directoryPath, key) (pkg/kubelet/util/store/filestore.go:98-100), so this regex is the thing standing between a pod name and a path traversal.

The rollback trick

Here is the gem. The memory manager’s V3 checkpoint embeds the old V1 struct so that older kubelets can still read it:

 go
 
 
// pkg/kubelet/cm/memorymanager/state/checkpoint.go:47-62
// MemoryManagerCheckpoint represents a structure to store memory/pod assignments checkpoint data.
// Dual-checksum format for backward compatibility.
// The Data string with DataChecksum is the authoritative V3 payload.
type MemoryManagerCheckpoint struct {
	// Backward compatibility
	// The embedded V1 checkpoint must not be enhanced and should be removed as soon as the oldest supported version understands V3.
	MemoryManagerCheckpointV1 `json:",inline"`

	// Data is a serialized MemoryManagerCheckpointData
	Data string `json:"data"`
	// DataChecksum is a checksum of Data string
	DataChecksum checksum.Checksum `json:"dataChecksum"`

	// CheckpointData holds actual data, not serialized directly
	CheckpointData MemoryManagerCheckpointData `json:"-"`
}

The embedded V1 struct is a flat struct — PolicyName, MachineState, Entries, Checksum — and the V3 payload lives in the separate Data string with its own DataChecksum. Two checksums, two eras.

And when the V3 checkpoint is written, the legacy checksum is recomputed using a string substitution:

 go
 
 
// pkg/kubelet/cm/memorymanager/state/checkpoint.go:155-176
// MarshalCheckpoint returns marshalled checkpoint in v3 format
func (mp *MemoryManagerCheckpointV3) MarshalCheckpoint() ([]byte, error) {
	data, err := json.Marshal(mp.CheckpointData)
	if err != nil {
		return nil, fmt.Errorf("failed to serialize memory manager checkpoint data: %w", err)
	}
	mp.Data = string(data)
	mp.DataChecksum = checksum.New(mp.Data)

	mp.PolicyName = mp.CheckpointData.PolicyName
	mp.MachineState = mp.CheckpointData.MachineState
	mp.Entries = mp.CheckpointData.Entries

	mp.Checksum = 0
	object := dump.ForHash(&mp.MemoryManagerCheckpointV1)
	object = strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1)
	hash := fnv.New32a()
	_, _ = fmt.Fprintf(hash, "%v", object)
	mp.Checksum = checksum.Checksum(hash.Sum32())

	return json.Marshal(*mp)
}

The strings.Replace is the whole point. dump.ForHash prints the type name of the struct, so hashing &mp.MemoryManagerCheckpointV1 produces a string containing MemoryManagerCheckpointV1. The V1 struct was originally named simply MemoryManagerCheckpoint; it was renamed when V2 arrived. An old kubelet still hashes a type it calls MemoryManagerCheckpoint, and the two printed strings have to match byte for byte or the checksums differ.

So the new kubelet literally renames the type in the hash input to what the old kubelet expects. The identical substitution appears in the V1 and V2 marshalers, with a comment that says so outright:

 go
 
 
// pkg/kubelet/cm/memorymanager/state/checkpoint.go:124-129
	// In order to preserve rollback compatibility,
	// we must generate a checksum using the legacy struct name "MemoryManagerCheckpoint"
	// instead of "MemoryManagerCheckpointV1". Older Kubelets do not have the string
	// replacement logic and expect the original struct name.
	object := dump.ForHash(mp)
	object = strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1)

The practical effect: a node running a new kubelet writes a checkpoint that an old kubelet can still parse, checksum and verify, so a downgrade (rollback) does not require draining the node and deleting the checkpoint file. That is the difference between a rollback being a five-minute operation and being a maintenance window.

The same pattern appears in the CPU manager (V4 embeds V2) and in the V1 and V2 memory manager checkpoints themselves. The V3 reader is also defensive about a corrupt file: if the V3 payload is present but its checksum fails, it refuses to fall back to V2, because a corrupt V3 means the file is newer than the reader understands.

The disclaimer

The package carries an explicit warning, and it is worth reading before you add anything to it:

 text
 
 
// pkg/kubelet/checkpointmanager/README.md:1-6
## DISCLAIMER
- Sig-Node community has reached a general consensus, as a best practice, to
avoid introducing any new checkpointing support. We reached this understanding
after struggling with some hard-to-debug issues in the production environments
caused by the checkpointing.
- Any changes to the checkpointed data structure would be considered incompatible and a component should add its own handling if it needs to ensure backward compatibility of reading old-format checkpoint files.

and the history table below it lists seven subsystems that have carried checkpointing since 2017, with the dockershim one removed in 1.24.

The pattern across all of them is the same: a versioned struct, a checksum, a migration path from the previous version, and a strings.Replace where the type name changed. The CPU manager is at V4, the memory manager at V3. Each version bump is another compatibility surface, and each one is a place where a bug means a node that cannot start.

Wrapping up

The one thing to remember: the checksum is a hash of a struct dump, not of the file. dump.ForHash sorts map keys, hides pointer addresses and suppresses methods, so the hash is a function of the logical content. That is why strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1) is legal at all: the printed type name is part of the hashed string, and renaming it is how a new kubelet keeps its checksum comparable with an old one’s.

The part people get wrong: a corrupt checkpoint does not always mean “delete the file and move on”. The memory manager’s restore path refuses to fall back from a corrupt V3 to V2, and the error it returns tells you exactly what to do — “please drain this node and delete the memory manager checkpoint file %q before restarting Kubelet” (pkg/kubelet/cm/memorymanager/state/state_checkpoint.go:60-61). If you delete a checkpoint file without draining, the kubelet will start with an empty state and happily re-assign exclusive CPUs that another container already owns. That is the failure mode the whole package is defending against.

What to go do about it: look at the files before you touch them — cat /var/lib/kubelet/cpu_manager_state, cat /var/lib/kubelet/memory_manager_state, and ls -la /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint. A checkpoint whose checksum field is 0 is accepted for backwards compatibility with the old file backend, so a zero checksum is not proof of corruption. If you are debugging a checksum mismatch, the two things to check are the struct type name in the dump and the SortKeys behaviour of the spew config — both are part of the hashed string. And before you add a new checkpointed subsystem, read the README disclaimer again: the consensus in SIG-Node is that the answer is no.