The Kubelet Writes Its State to Disk: FNV-32a Checksums and a String Rename for Rollback

TL;DR: The kubelet persists CPU pinning, memory pinning and device allocations to /var/lib/kubelet through pkg/kubelet/checkpointmanager. Three decisions are non-obvious. The checksum is FNV-32a over a dump.ForHash of the struct, not over the serialized bytes, so it survives a Go field reorder. Writes go to a temp file, get fsynced, and are then renamed into place, so a reader never sees a partial file. And the memory manager’s V3 checkpoint does a literal strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1) on the hash dump, so an old kubelet can still verify the checksum of a new checkpoint and roll back without draining the node.

The kubelet keeps some state that cannot be reconstructed from the API server. Which containers got which exclusive CPUs, which NUMA nodes a container’s memory was pinned to, which devices were handed to which container — that state lives only in the kubelet’s memory, and if it is lost the kubelet cannot honour the guarantees it already made. So it is written to disk.

That sounds mundane. It is not: checkpointing in the kubelet has a long history of production incidents, and the package that implements it carries an explicit warning from the SIG-Node community about adding more of it.

What is actually on disk

Six submodules use the framework. The files are plain JSON in the kubelet’s root directory (--root-dir, default /var/lib/kubelet):

The manager itself is tiny — a mutex, a store, and three verbs:

// pkg/kubelet/checkpointmanager/checkpoint_manager.go:65-92
// CreateCheckpoint persists checkpoint in CheckpointStore.
func (manager *impl) CreateCheckpoint(checkpointKey string, checkpoint Checkpoint) error {
	manager.mutex.Lock()
	defer manager.mutex.Unlock()
	blob, err := checkpoint.MarshalCheckpoint()
	if err != nil {
		return err
	}
	return manager.store.Write(checkpointKey, blob)
}

// GetCheckpoint retrieves checkpoint from CheckpointStore.
func (manager *impl) GetCheckpoint(checkpointKey string, checkpoint Checkpoint) error {
	manager.mutex.Lock()
	defer manager.mutex.Unlock()
	blob, err := manager.store.Read(checkpointKey)
	if err != nil {
		if err == utilstore.ErrKeyNotFound {
			return errors.ErrCheckpointNotFound
		}
		return err
	}
	err = checkpoint.UnmarshalCheckpoint(blob)
	if err == nil {
		err = checkpoint.VerifyChecksum()
	}
	return err
}

The Checkpoint interface is three methods — MarshalCheckpoint, UnmarshalCheckpoint and VerifyChecksum — and the split matters: MarshalCheckpoint computes and embeds the checksum, and VerifyChecksum is a separate call after unmarshalling. That split exists because the checksum is over the struct, not over the bytes — if it were over the bytes, verification would be a hash of the file and could not be done by the struct’s own method.

The checksum is over a struct dump, not the bytes

The generic checksum helper is four lines:

// pkg/kubelet/checkpointmanager/checksum/checksum.go:26-48
// Checksum is the data to be stored as checkpoint
type Checksum uint64

// Verify verifies that passed checksum is same as calculated checksum
func (cs Checksum) Verify(data interface{}) error {
	actualCS := New(data)
	if cs != actualCS {
		return &errors.CorruptCheckpointError{ActualCS: uint64(actualCS), ExpectedCS: uint64(cs)}
	}
	return nil
}

// New returns the Checksum of checkpoint data
func New(data interface{}) Checksum {
	return Checksum(getChecksum(data))
}

// Get returns calculated checksum of checkpoint data
func getChecksum(data interface{}) uint64 {
	hash := fnv.New32a()
	hashutil.DeepHashObject(hash, data)
	return uint64(hash.Sum32())
}

Checksum is a uint64 that wraps a 32-bit FNV-1a hash. The 64-bit width is cosmetic; the entropy is 32 bits.

DeepHashObject is the interesting part — it dumps the object with spew and feeds the result to the hasher:

// pkg/util/hash/hash.go:26-32
// DeepHashObject writes specified object to hash using the spew library
// which follows pointers and prints actual values of the nested objects
// ensuring the hash does not change when a pointer changes.
func DeepHashObject(hasher hash.Hash, objectToWrite interface{}) {
	hasher.Reset()
	fmt.Fprintf(hasher, "%v", dump.ForHash(objectToWrite))
}

and dump.ForHash is a spew configuration with two properties that matter enormously:

// vendor/k8s.io/utils/dump/dump.go:30-38
// The config MUST NOT be changed because that could change the result of a hash operation
var prettyPrintConfigForHash = &spew.ConfigState{
	Indent:                  " ",
	SortKeys:                true,
	DisableMethods:          true,
	SpewKeys:                true,
	DisablePointerAddresses: true,
	DisableCapacities:       true,
}

SortKeys: true is what makes the hash stable across runs: map iteration order in Go is randomised, and without sorting the dump of a struct with a map field would differ every time. DisablePointerAddresses: true means the dump prints the values a pointer points at rather than its address, so the hash does not depend on where the allocator happened to put the object. DisableMethods: true stops String() methods from being invoked, which would let a type change its own hash just by changing a String() method.

The consequence is that the checksum is a hash of the logical content of the struct. Adding a field, removing a field, or reordering fields changes the hash — which sounds like a problem, and is handled by the version migration machinery discussed below. But changing a field’s type in a way that changes its printed representation, or changing the value of a field, also changes the hash, which is exactly what you want.

The alternative — hashing the JSON bytes — would be immune to struct reordering but would break the moment you added a field, because JSON serialisation of the same logical data changes shape. The struct dump is chosen because it is what the readers care about.

Atomic writes: temp file, fsync, rename

The file store is where the durability story lives:

// pkg/kubelet/util/store/filestore.go:102-123
// writeFile writes data to path in a single transaction.
func writeFile(fs utilfs.Filesystem, path string, data []byte) (retErr error) {
	// Create a temporary file in the base directory of `path` with a prefix.
	tmpFile, err := fs.TempFile(filepath.Dir(path), tmpPrefix)
	if err != nil {
		return err
	}

	tmpPath := tmpFile.Name()
	shouldClose := true

	defer func() {
		// Close the file.
		if shouldClose {
			if err := tmpFile.Close(); err != nil {
				if retErr == nil {
					retErr = fmt.Errorf("close error: %v", err)
				} else {
					retErr = fmt.Errorf("failed to close temp file after error %v; close error: %v", retErr, err)
				}
			}
		}

The defer closes the temp file if it is still open and removes it if the function is returning an error — a half-written temp file left behind would otherwise accumulate on every failed write.

and the last eleven lines are the part that matters:

// pkg/kubelet/util/store/filestore.go:133-151
	// Write data.
	if _, err := tmpFile.Write(data); err != nil {
		return err
	}

	// Sync file.
	if err := tmpFile.Sync(); err != nil {
		return err
	}

	// Closing the file before renaming.
	err = tmpFile.Close()
	shouldClose = false
	if err != nil {
		return err
	}

	return fs.Rename(tmpPath, path)
}

Three things are happening, and all three are needed:

1. Write to a temp file in the same directory. The temp file prefix is . (tmpPrefix = "." in pkg/kubelet/util/store/filestore.go:30), and List() skips anything starting with it, so a crashed kubelet’s leftover temp files do not show up as checkpoints.
2. Sync() before close. The fsync forces the data out of the page cache and onto the disk. Without it, a node that loses power between the write and the rename can end up with a file whose name exists but whose contents are zero bytes.
3. rename() into place. rename(2) within a filesystem is atomic. A concurrent reader either sees the old file or the new file, never a half-written one. The rename must be on the same filesystem, which is why the temp file is created in filepath.Dir(path) rather than in /tmp.

Key validation

The store keys are restricted, and the restriction is enforced at the store layer rather than trusted from callers:

// pkg/kubelet/util/store/store.go:24-39
const (
	keyMaxLength = 250

	keyCharFmt      string = "[A-Za-z0-9]"
	keyExtCharFmt   string = "[-A-Za-z0-9_.]"
	qualifiedKeyFmt string = "(" + keyCharFmt + keyExtCharFmt + "*)?" + keyCharFmt
)

var (
	// Key must consist of alphanumeric characters, '-', '_' or '.', and must start
	// and end with an alphanumeric character.
	keyRegex = regexp.MustCompile("^" + qualifiedKeyFmt + "$")

	// ErrKeyNotFound is the error returned if key is not found in Store.
	ErrKeyNotFound = fmt.Errorf("key is not found")
)

So a key must start and end with an alphanumeric character, may contain -, _ and . in between, and is at most 250 characters long. The characters that are not allowed are exactly the ones that would let a key escape the checkpoint directory: / and ... The key becomes a filename via filepath.Join(f.directoryPath, key) (pkg/kubelet/util/store/filestore.go:98-100), so this regex is the thing standing between a pod name and a path traversal.

The rollback trick

Here is the gem. The memory manager’s V3 checkpoint embeds the old V1 struct so that older kubelets can still read it:

// pkg/kubelet/cm/memorymanager/state/checkpoint.go:47-62
// MemoryManagerCheckpoint represents a structure to store memory/pod assignments checkpoint data.
// Dual-checksum format for backward compatibility.
// The Data string with DataChecksum is the authoritative V3 payload.
type MemoryManagerCheckpoint struct {
	// Backward compatibility
	// The embedded V1 checkpoint must not be enhanced and should be removed as soon as the oldest supported version understands V3.
	MemoryManagerCheckpointV1 `json:",inline"`

	// Data is a serialized MemoryManagerCheckpointData
	Data string `json:"data"`
	// DataChecksum is a checksum of Data string
	DataChecksum checksum.Checksum `json:"dataChecksum"`

	// CheckpointData holds actual data, not serialized directly
	CheckpointData MemoryManagerCheckpointData `json:"-"`
}

The embedded V1 struct is a flat struct — PolicyName, MachineState, Entries, Checksum — and the V3 payload lives in the separate Data string with its own DataChecksum. Two checksums, two eras.

And when the V3 checkpoint is written, the legacy checksum is recomputed using a string substitution:

// pkg/kubelet/cm/memorymanager/state/checkpoint.go:155-176
// MarshalCheckpoint returns marshalled checkpoint in v3 format
func (mp *MemoryManagerCheckpointV3) MarshalCheckpoint() ([]byte, error) {
	data, err := json.Marshal(mp.CheckpointData)
	if err != nil {
		return nil, fmt.Errorf("failed to serialize memory manager checkpoint data: %w", err)
	}
	mp.Data = string(data)
	mp.DataChecksum = checksum.New(mp.Data)

	mp.PolicyName = mp.CheckpointData.PolicyName
	mp.MachineState = mp.CheckpointData.MachineState
	mp.Entries = mp.CheckpointData.Entries

	mp.Checksum = 0
	object := dump.ForHash(&mp.MemoryManagerCheckpointV1)
	object = strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1)
	hash := fnv.New32a()
	_, _ = fmt.Fprintf(hash, "%v", object)
	mp.Checksum = checksum.Checksum(hash.Sum32())

	return json.Marshal(*mp)
}

The strings.Replace is the whole point. dump.ForHash prints the type name of the struct, so hashing &mp.MemoryManagerCheckpointV1 produces a string containing MemoryManagerCheckpointV1. The V1 struct was originally named simply MemoryManagerCheckpoint; it was renamed when V2 arrived. An old kubelet still hashes a type it calls MemoryManagerCheckpoint, and the two printed strings have to match byte for byte or the checksums differ.

So the new kubelet literally renames the type in the hash input to what the old kubelet expects. The identical substitution appears in the V1 and V2 marshalers, with a comment that says so outright:

// pkg/kubelet/cm/memorymanager/state/checkpoint.go:124-129
	// In order to preserve rollback compatibility,
	// we must generate a checksum using the legacy struct name "MemoryManagerCheckpoint"
	// instead of "MemoryManagerCheckpointV1". Older Kubelets do not have the string
	// replacement logic and expect the original struct name.
	object := dump.ForHash(mp)
	object = strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1)

The practical effect: a node running a new kubelet writes a checkpoint that an old kubelet can still parse, checksum and verify, so a downgrade (rollback) does not require draining the node and deleting the checkpoint file. That is the difference between a rollback being a five-minute operation and being a maintenance window.

The same pattern appears in the CPU manager (V4 embeds V2) and in the V1 and V2 memory manager checkpoints themselves. The V3 reader is also defensive about a corrupt file: if the V3 payload is present but its checksum fails, it refuses to fall back to V2, because a corrupt V3 means the file is newer than the reader understands.

The disclaimer

The package carries an explicit warning, and it is worth reading before you add anything to it:

// pkg/kubelet/checkpointmanager/README.md:1-6
## DISCLAIMER
- Sig-Node community has reached a general consensus, as a best practice, to
avoid introducing any new checkpointing support. We reached this understanding
after struggling with some hard-to-debug issues in the production environments
caused by the checkpointing.
- Any changes to the checkpointed data structure would be considered incompatible and a component should add its own handling if it needs to ensure backward compatibility of reading old-format checkpoint files.

and the history table below it lists seven subsystems that have carried checkpointing since 2017, with the dockershim one removed in 1.24.

The pattern across all of them is the same: a versioned struct, a checksum, a migration path from the previous version, and a strings.Replace where the type name changed. The CPU manager is at V4, the memory manager at V3. Each version bump is another compatibility surface, and each one is a place where a bug means a node that cannot start.

Wrapping up

The one thing to remember: the checksum is a hash of a struct dump, not of the file. dump.ForHash sorts map keys, hides pointer addresses and suppresses methods, so the hash is a function of the logical content. That is why strings.Replace(object, "MemoryManagerCheckpointV1", "MemoryManagerCheckpoint", 1) is legal at all: the printed type name is part of the hashed string, and renaming it is how a new kubelet keeps its checksum comparable with an old one’s.

The part people get wrong: a corrupt checkpoint does not always mean “delete the file and move on”. The memory manager’s restore path refuses to fall back from a corrupt V3 to V2, and the error it returns tells you exactly what to do — “please drain this node and delete the memory manager checkpoint file %q before restarting Kubelet” (pkg/kubelet/cm/memorymanager/state/state_checkpoint.go:60-61). If you delete a checkpoint file without draining, the kubelet will start with an empty state and happily re-assign exclusive CPUs that another container already owns. That is the failure mode the whole package is defending against.

What to go do about it: look at the files before you touch them — cat /var/lib/kubelet/cpu_manager_state, cat /var/lib/kubelet/memory_manager_state, and ls -la /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint. A checkpoint whose checksum field is 0 is accepted for backwards compatibility with the old file backend, so a zero checksum is not proof of corruption. If you are debugging a checksum mismatch, the two things to check are the struct type name in the dump and the SortKeys behaviour of the spew config — both are part of the hashed string. And before you add a new checkpointed subsystem, read the README disclaimer again: the consensus in SIG-Node is that the answer is no.