Controllers on Opensource internals

Before It Evicts Your Node, the Node Controller Checks Its Own Cache

Jay Hubbert — Fri, 23 Oct 2026 14:10:00 +0000

TL;DR: The node controller decides a node is dead from the NodeReady heartbeat or from the kubelet’s Lease renewal. Before it acts on an expired lease in the informer cache, it does a live GET of that lease and compares resource versions. If the live lease differs from the cached one, the cache is stale — so the controller does not just skip the node, it short-circuits the entire pass and refuses to evaluate any node, because a stale lease cache would otherwise produce a mass false-positive eviction storm. The feature is NodeControllerLeaseCircuitBreaker, beta and on by default since 1.37.

Evicting a node is the most destructive thing a Kubernetes controller does. It taints the node NoExecute, the kubelet starts killing pods, and the workloads move. If the controller gets that wrong for the wrong reason — a cache that has not caught up, a watch that stalled — the cost is a cluster-wide workload restart.

The node controller is therefore paranoid in a specific way: it knows that its own informer cache can be stale, and it has a mechanism to detect that staleness and refuse to act on it.

What “dead” means

Two signals say a node is alive: the NodeReady condition’s lastHeartbeatTime on the Node object, and the renewTime on the kube-node-lease Lease object. The controller tracks both and uses the more recent one:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:936
	// Always update the probe time if node lease is renewed.
	// Note: If kubelet never posted the node status, but continues renewing the
	// heartbeat leases, the node controller will assume the node is healthy and
	// take no action.
	observedLease, _ := nc.leaseLister.Leases(v1.NamespaceNodeLease).Get(node.Name)
	if observedLease != nil && (savedLease == nil || savedLease.Spec.RenewTime.Before(observedLease.Spec.RenewTime)) {
		nodeHealth.lease = observedLease
		nodeHealth.probeTimestamp = nc.now()
	}

nc.leaseLister is a cache read. The lease the kubelet renews every ten seconds or so arrives through the informer, so there is always a window in which the cached lease is behind the real one.

The grace period decides when that window becomes fatal. It defaults to 50 seconds for a running node and 60 seconds for one that has not yet posted status:

 go
 
 
// pkg/controller/nodelifecycle/config/v1alpha1/defaults.go:40
	// NodeMonitorGracePeriod is set to a default value of 50 seconds.
	// This value should be greater than the sum of HTTP2_PING_TIMEOUT_SECONDS (30s)
	// and HTTP2_READ_IDLE_TIMEOUT_SECONDS (15s) from the http2 health check
	// to ensure that the server has adequate time to handle slow or idle connections
	// properly before marking a node as unhealthy.
	if obj.NodeMonitorGracePeriod == zero {
		obj.NodeMonitorGracePeriod = metav1.Duration{Duration: 50 * time.Second}
	}
	if obj.NodeStartupGracePeriod == zero {
		obj.NodeStartupGracePeriod = metav1.Duration{Duration: 60 * time.Second}
	}

Both are flags: --node-monitor-grace-period and --node-startup-grace-period on kube-controller-manager (cmd/kube-controller-manager/app/options/nodelifecyclecontroller.go:39). The comment explains the 50 seconds: it must exceed the sum of the kubelet’s HTTP/2 ping timeout (30s) and read idle timeout (15s), so that a merely-slow kubelet is never mistaken for a dead one.

When the probe timestamp falls outside the grace period, the controller believes the node is dead:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:946
	if nc.now().After(nodeHealth.probeTimestamp.Add(gracePeriod)) {
		if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
			var nodeHealthLeaseRV string
			if nodeHealth.lease != nil {
				nodeHealthLeaseRV = nodeHealth.lease.ResourceVersion
			}
			// The lease instance in the informer cache indicates it is expired.
			// Double-check the live lease is actually expired in case our informer cache is stale.
			liveLease, err := nc.kubeClient.CoordinationV1().Leases(v1.NamespaceNodeLease).Get(ctx, node.Name, metav1.GetOptions{})
			if err == nil {
				if liveLease.ResourceVersion != nodeHealthLeaseRV {
					nc.consistencyStore.WroteAt(
						// This is a controller wide consistency check, if the lease cache is stale
						// we need to wait for the cache to catch up before processing any nodes.
						types.NamespacedName{},
						"", // No specific UID for generic name
						leaseResource,
						liveLease.ResourceVersion,
					)
					return 0, v1.NodeCondition{}, nil, shortCircuitError{&consistencyutil.ConsistencyError{
						ReadRV: nodeHealthLeaseRV,
						WroteRV: liveLease.ResourceVersion,
						GroupResource: leaseResource,
					}}
				}
			} else if !apierrors.IsNotFound(err) {
				return 0, v1.NodeCondition{}, nil, shortCircuitError{fmt.Errorf("error looking up lease to verify node %s: %w", node.Name, err)}
			}
		}

Read this carefully, because the semantics are subtle and deliberate.

The cached lease says “expired”. The controller now does a live GET of the same lease. Two outcomes:

The live lease’s resourceVersion equals the cached one. Then the cache is up to date, the lease really is expired, and the controller proceeds to mark the node Unknown and evict.
The live lease’s resourceVersion differs. Then the cache is behind, the “expired” verdict was based on stale data, and the controller bails out of the entire node evaluation — returning a shortCircuitError.

The same bail-out happens if the live GET fails with something other than NotFound. A NotFound is not a bail-out: if the lease is gone, then the node genuinely is not renewing it.

The short circuit

The error type is a marker, and its only job is to be recognisable:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:647
// shortCircuitError is a marker type to signal the polling loop in monitorNodeHealth should return early with error
type shortCircuitError struct {
	error
}

func (n shortCircuitError) Unwrap() error {
	return n.error
}

It is recognised in the retry loop inside monitorNodeHealth, and — crucially — it is not retried:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:705
		if err := wait.PollImmediate(retrySleepTime, retrySleepTime*scheduler.NodeHealthUpdateRetry, func() (bool, error) {
			var err error
			_, observedReadyCondition, currentReadyCondition, err = nc.tryUpdateNodeHealth(ctx, node)
			if err == nil {
				return true, nil
			}
			// If the error is due to a short circuit, don't retry.
			if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
				if err, ok := errors.AsType[shortCircuitError](err); ok {
					return false, err.error
				}
			}
			name := node.Name
			node, err = nc.kubeClient.CoreV1().Nodes().Get(ctx, name, metav1.GetOptions{})
			if err != nil {
				logger.Error(nil, "Failed while getting a Node to retry updating node health. Probably Node was deleted", "node", klog.KRef("", name))
				return false, err
			}
			return false, nil
		}); err != nil {
			logger.Error(err, "Update health of Node from Controller error, Skipping - no pods will be evicted", "node", klog.KObj(node))
			return
		}

The log message is the important part: “Skipping - no pods will be evicted”. A short circuit skips the node, it does not evict it.

But the effect is cluster-wide in the other direction, and that is the point of the design. If the lease cache is stale, it is stale for every node, not just this one. The controller therefore refuses to make eviction decisions for any node until the cache catches up. One stale lease, one aborted pass, zero false evictions.

The consistency store is what makes “until the cache catches up” precise rather than “until the next pass”. The controller records the resource version it saw live:

 go
 
 
// staging/src/k8s.io/client-go/util/consistency/consistency.go:176
// WroteAt increments the written resource version of an ownerRecord if it is
// the newest seen resource version for that resource.
func (w *ownerRecord) WroteAt(resource schema.GroupResource, rv string) {
	w.versionsLock.Lock()
	defer w.versionsLock.Unlock()
	if _, ok := w.versions[resource]; !ok {
		if _, err := resourceversion.CompareResourceVersion(rv, rv); err != nil {
			return
		}
		w.versions[resource] = rv
		return
	}
	cmp, err := resourceversion.CompareResourceVersion(w.versions[resource], rv)
	if err != nil || cmp >= 0 {
		return
	}
	w.versions[resource] = rv
}

and then, at the top of every node’s evaluation, it checks whether the informer store has caught up to that version:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:834
	if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
		// This is a controller-wide consistency check, if the lease cache is stale, then we cannot
		// trust any node leases until the cache has caught back up.
		err := nc.consistencyStore.EnsureReady(types.NamespacedName{})
		// Skip processing this node in this cycle if the node controller cache is not ready yet.
		if err != nil {
			return 0, v1.NodeCondition{}, nil, shortCircuitError{err}
		}
	}

The comparison is against the informer’s store sync resource version, not against a watch event:

 go
 
 
// staging/src/k8s.io/client-go/util/consistency/consistency.go:195
// EnsureReady checks whether or not the ownerRecord is ready compared to the
// read resource versions in the consistency store.
func (w *ownerRecord) EnsureReady(c *consistencyStore) error {
	w.versionsLock.Lock()
	defer w.versionsLock.Unlock()
	for gr, wroteRV := range w.versions {
		store, exists := c.stores[gr]
		if !exists || store == nil {
			return fmt.Errorf("no store registered for group resource %s", gr.String())
		}
		readRV := store.LastStoreSyncResourceVersion()
		if readRV == "" {
			// Store has not observed any sync or bookmark yet (e.g. cache not synced or
			// AtomicFIFO disabled). Fail in the safe direction by reporting not ready.
			return &ConsistencyError{
				WroteRV: wroteRV,
				ReadRV: readRV,
				GroupResource: gr,
			}
		}
		i, err := resourceversion.CompareResourceVersion(wroteRV, readRV)
		if err != nil {
			// comparison errors indicate there's a data problem with resource versions, continue so we don't block syncing
			continue
		}
		if i > 0 {
			// read version is not as new as owner version, not ready
			return &ConsistencyError{
				WroteRV: wroteRV,
				ReadRV: readRV,
				GroupResource: gr,
			}
		}
	}
	return nil
}

Two details worth pausing on. The readRV == "" case fails in the safe direction — if the store has never synced, the controller assumes it is not ready and refuses to act. And the comparison errors are swallowed (continue), because a malformed resource version is a data problem, not a reason to block the entire cluster’s node monitoring.

The store is only created when the feature gate is on; otherwise it is a no-op implementation:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:424
	if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
		nc.consistencyStore = consistencyutil.NewConsistencyStore(map[schema.GroupResource]consistencyutil.LastSyncRVGetter{
			leaseResource: leaseInformer.Informer().GetStore(),
		})
	} else {
		nc.consistencyStore = consistencyutil.NewNoopConsistencyStore()
	}

And the gate itself:

 go
 
 
// pkg/features/kube_features.go:1770
	NodeControllerLeaseCircuitBreaker: {
		{Version: version.MustParse("1.37"), Default: true, PreRelease: featuregate.Beta},
	},

Beta and on by default as of 1.37. It is still a feature gate, so --feature-gates=NodeControllerLeaseCircuitBreaker=false turns the whole mechanism off — which is the first thing to check if you are debugging “the node controller evicted my node”.

Bounded concurrency, and why

The pass over all nodes is parallelised, with a fixed worker count:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:767
	// Marking the pods not ready on a node requires looping over them and
	// updating each pod's status one at a time. This is performed serially, and
	// can take a while if we're processing each node serially as well. So we
	// process them with bounded concurrency instead, since most of the time is
	// spent waiting on io.
	workqueue.ParallelizeUntil(ctx, nc.nodeUpdateWorkerSize, len(nodes), updateNodeFunc)

with nodeUpdateWorkerSize = 8 (pkg/controller/nodelifecycle/node_lifecycle_controller.go:138).

The comment explains the trade-off precisely: marking pods not-ready is a sequence of API calls that spend most of their time waiting on I/O, so serialising them wastes the parallelism, and unbounded parallelism would overload the API server. Eight is the compromise.

The eviction rate limiter

Even with the circuit breaker, the controller does not evict at full speed. Each zone gets its own rate-limited queue, and the rate limiter is hot swappable — the controller replaces it when the zone’s health changes.

 go
 
 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:196
// RateLimitedTimedQueue is a unique item priority queue ordered by
// the expected next time of execution. It is also rate limited.
type RateLimitedTimedQueue struct {
	queue UniqueQueue
	limiterLock sync.Mutex
	limiter flowcontrol.RateLimiter
}

The constants that set the pace:

 go
 
 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:29
const (
	// NodeHealthUpdateRetry controls the number of retries of writing
	// node health update.
	NodeHealthUpdateRetry = 5
	// NodeEvictionPeriod controls how often NodeController will try to
	// evict Pods from non-responsive Nodes.
	NodeEvictionPeriod = 100 * time.Millisecond
	// EvictionRateLimiterBurst is the burst value for all eviction rate
	// limiters
	EvictionRateLimiterBurst = 1
)

A burst of 1 means the token bucket refills at exactly the configured QPS with no allowance for a spike. One node per second, by default.

The queue itself pops items and asks the limiter for permission:

 go
 
 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:231
func (q *RateLimitedTimedQueue) Try(logger klog.Logger, fn ActionFunc) {
	val, ok := q.queue.Head()
	q.limiterLock.Lock()
	defer q.limiterLock.Unlock()
	for ok {
		// rate limit the queue checking
		if !q.limiter.TryAccept() {
			logger.V(10).Info("Try rate limited", "value", val)
			// Try again later
			break
		}

		now := now()
		if now.Before(val.ProcessAt) {
			break
		}

		if ok, wait := fn(val); !ok {
			val.ProcessAt = now.Add(wait + 1)
			q.queue.Replace(val)
		} else {
			q.queue.RemoveFromQueue(val.Value)
		}
		val, ok = q.queue.Head()
	}
}

Swapping the limiter without a burst

Here is the part that is genuinely subtle. When the controller changes a zone’s eviction rate — because the zone got healthier or sicker — it swaps the limiter. A naive swap would be a trap: a token bucket starts full, so the new limiter would immediately allow a burst of evictions at the new rate.

The fix is to drain the new bucket when the old one was saturated:

 go
 
 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:282
// SwapLimiter safely swaps current limiter for this queue with the
// passed one if capacities or qps's differ.
func (q *RateLimitedTimedQueue) SwapLimiter(newQPS float32) {
	q.limiterLock.Lock()
	defer q.limiterLock.Unlock()
	if q.limiter.QPS() == newQPS {
		return
	}
	var newLimiter flowcontrol.RateLimiter
	if newQPS <= 0 {
		newLimiter = flowcontrol.NewFakeNeverRateLimiter()
	} else {
		newLimiter = flowcontrol.NewTokenBucketRateLimiter(newQPS, EvictionRateLimiterBurst)

		// If we're currently waiting on limiter, we drain the new one - this is a good approach when Burst value is 1
		// TODO: figure out if we need to support higher Burst values and decide on the drain logic, should we keep:
		// - saturation (percentage of used tokens)
		// - number of used tokens
		// - number of available tokens
		// - something else
		if q.limiter.TryAccept() == false {
			newLimiter.TryAccept()
		}
	}
	q.limiter.Stop()
	q.limiter = newLimiter
}

The logic: q.limiter.TryAccept() on the old limiter tells you whether the old limiter was saturated. If it was (returns false), the system was already unable to evict at the old rate, so the new limiter is drained with one TryAccept — consuming its single token. The new limiter therefore starts empty and the eviction rate ramps up from zero instead of bursting.

If the old limiter was not saturated, the new one is left full, which is correct: the system was keeping up before, so it can keep up now.

The newQPS <= 0 case installs a FakeNeverRateLimiter — a limiter that never accepts a token, i.e. evictions stop entirely. That is what the “master disruption mode” uses.

The zone state machine

The rate limiter is chosen by the zone state, and the state is computed from the per-node conditions:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:1067
	// At least one node was responding in previous pass or in the current pass. Semantics is as follows:
	// - if the new state is "partialDisruption" we call a user defined function that returns a new limiter to use,
	// - if the new state is "normal" we resume normal operation (go back to default limiter settings),
	// - if new state is "fullDisruption" we restore normal eviction rate,
	// - unless all zones in the cluster are in "fullDisruption" - in that case we stop all evictions.
	if !allAreFullyDisrupted || !allWasFullyDisrupted {
		// We're switching to full disruption mode
		if allAreFullyDisrupted {
			logger.Info("Controller detected that all Nodes are not-Ready. Entering master disruption mode")
			for i := range nodes {
				_, err := nc.markNodeAsReachable(ctx, nodes[i])
				if err != nil {
					logger.Error(nil, "Failed to remove taints from Node", "node", klog.KObj(nodes[i]))
				}
			}
			// We stop all evictions.
			for k := range nc.zoneStates {
				nc.zoneNoExecuteTainter[k].SwapLimiter(0)
			}

SwapLimiter(0) — the “never” limiter. If every zone in the cluster is fully disrupted, the controller stops evicting entirely, because evicting everything at once is strictly worse than evicting nothing: the pods cannot go anywhere else.

And when the cluster recovers:

 go
 
 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:1092
		// We're exiting full disruption mode
		if allWasFullyDisrupted {
			logger.Info("Controller detected that some Nodes are Ready. Exiting master disruption mode")
			// When exiting disruption mode update probe timestamps on all Nodes.
			now := nc.now()
			for i := range nodes {
				v := nc.nodeHealthMap.getDeepCopy(nodes[i].Name)
				v.probeTimestamp = now
				v.readyTransitionTimestamp = now
				nc.nodeHealthMap.set(nodes[i].Name, v)
			}

The probe timestamps are reset so that nodes do not immediately look expired again on the way out of disruption mode.

Wrapping up

The one thing to remember: before acting on an expired lease from the informer cache, the node controller does a live GET and compares resourceVersions (pkg/controller/nodelifecycle/node_lifecycle_controller.go:954). A mismatch means the cache is stale, and the controller returns a shortCircuitError that aborts the evaluation of every node, not just the one it was looking at.

The part people get wrong: that the circuit breaker “skips one node”. It does not. The consistency store records the live resource version it saw and the next pass refuses to evaluate any node until the informer store has caught up to it (staging/src/k8s.io/client-go/util/consistency/consistency.go:197). The whole point is that a stale lease cache is stale for every node, and evicting a subset of nodes based on a stale cache is exactly the failure mode this design exists to prevent.

What to go do about it: if a node is being evicted unexpectedly, check whether the circuit breaker is even enabled — it is a beta feature gate NodeControllerLeaseCircuitBreaker, on by default since 1.37 (pkg/features/kube_features.go:1770), and --feature-gates=NodeControllerLeaseCircuitBreaker=false disables it entirely. Look for the “Update health of Node from Controller error, Skipping - no pods will be evicted” log line, which means the node was skipped rather than evicted. And if you are tuning responsiveness, the grace periods are --node-monitor-grace-period (default 50s) and --node-startup-grace-period (default 60s), the eviction rate is bounded by EvictionRateLimiterBurst = 1 and the --node-eviction-rate (0.1 nodes per second) / --secondary-node-eviction-rate (0.01) settings (cmd/kube-controller-manager/app/options/nodelifecyclecontroller.go:46), the zone is declared unhealthy at --unhealthy-zone-threshold 0.55 of its nodes (minimum 3), and the per-node work runs with 8 workers (pkg/controller/nodelifecycle/node_lifecycle_controller.go:138).

Expectations: How a Controller Waits for Its Own Writes to Come Back

Jay Hubbert — Fri, 23 Oct 2026 13:40:00 +0000

TL;DR: Before a controller writes to the API server it raises an expectation — “I expect N creates and M deletes to come back” — and the informer handler lowers it when the matching watch event arrives. SatisfiedExpectations() gates the sync loop: if the controller’s own writes are not yet visible in the cache, the sync does nothing, which is what stops double-creates and status churn. Deletes are tracked by UID, not by name, so a pod deleted and recreated with the same name is not confused with the old one, and a graceful deletion (an update that sets deletionTimestamp) is not double-counted. The whole thing expires after 5 minutes so a dropped watch event cannot wedge a controller forever.

The Kubernetes control loop has one structural weakness: the controller writes to the API server, and then reads the result back from an informer cache. Between the write returning and the watch event arriving, the cache is stale. If the controller acts on that stale view, it will do the wrong thing — typically create a second copy of a Pod it just created.

This is not a theoretical race. It is the normal case: the API server commits the write, returns 201, and the watch event has to travel through etcd, the apiserver’s watch cache, and the client’s reflector before the informer store is updated. That is tens of milliseconds at best, and the controller’s next sync may land right in the middle of it.

Expectations are the mechanism that closes the gap.

The contract

The design is documented in the source, tersely:

 go
 
 
// pkg/controller/controller_utils.go:128
// Expectations are a way for controllers to tell the controller manager what they expect. eg:
//	ControllerExpectations: {
//		controller1: expects 2 adds in 2 minutes
//		controller2: expects 2 dels in 2 minutes
//		controller3: expects -1 adds in 2 minutes => controller3's expectations have already been met
//	}
//
// Implementation:
//	ControlleeExpectation = pair of atomic counters to track controllee's creation/deletion
//	ControllerExpectationsStore = TTLStore + a ControlleeExpectation per controller
//
// * Once set expectations can only be lowered
// * A controller isn't synced till its expectations are either fulfilled, or expire
// * Controllers that don't set expectations will get woken up for every matching controllee

Three properties, and each one is a design decision worth defending.

“Once set expectations can only be lowered.” The counters are monotone decreasing. A controller can raise them again by setting new expectations (a fresh SetExpectations), but a single expectation record only ever goes down. This makes the counters safe to update from multiple goroutines without a lock around the whole record.

“A controller isn’t synced till its expectations are either fulfilled, or expire.” The second half is the escape hatch.

“Controllers that don’t set expectations will get woken up for every matching controllee.” A controller that does not use expectations still works; it just gets a sync per event, which is what the pre-expectations controllers did.

The data structure and the timeout

Two int64 counters and a timestamp, per controller key:

 go
 
 
// pkg/controller/controller_utils.go:272
// ControlleeExpectations track controllee creates/deletes.
type ControlleeExpectations struct {
	// Important: Since these two int64 fields are using sync/atomic, they have to be at the top of the struct due to a bug on 32-bit platforms
	// See: https://golang.org/pkg/sync/atomic/ for more information
	add int64
	del int64
	key string
	timestamp time.Time
}

The comment about 32-bit platforms is the kind of thing that looks like cargo cult until you hit it: sync/atomic requires 64-bit alignment for 64-bit operations, and a struct whose first field is an int64 is guaranteed to be 8-byte aligned. Move the int64s down behind a string and the atomic operations break on 32-bit ARM.

The store is a cache.Store keyed by the controller key:

 go
 
 
// pkg/controller/controller_utils.go:168
// ControllerExpectations is a cache mapping controllers to what they expect to see before being woken up for a sync.
type ControllerExpectations struct {
	cache.Store
}

and the check is one function:

 go
 
 
// pkg/controller/controller_utils.go:195
func (r *ControllerExpectations) SatisfiedExpectations(logger klog.Logger, controllerKey string) bool {
	if exp, exists, err := r.GetExpectations(controllerKey); exists {
		if exp.Fulfilled() {
			logger.V(4).Info("Controller expectations fulfilled", "expectations", exp)
			return true
		} else if exp.isExpired() {
			logger.V(4).Info("Controller expectations expired", "expectations", exp)
			return true
		} else {
			logger.V(4).Info("Controller still waiting on expectations", "expectations", exp)
			return false
		}
	} else if err != nil {
		logger.V(2).Info("Error encountered while checking expectations, forcing sync", "err", err)
	} else {

Three outcomes, and the third one is the interesting one. If there is no expectation record for this controller — it has never created anything, or its expectations expired and were deleted — the function returns true, and the controller syncs. Expectations only ever delay a sync; they never block one permanently.

The expiry is a flat constant, and the comment above it explains the arithmetic behind it:

 go
 
 
// pkg/controller/controller_utils.go:61
const (
	// If a watch drops a delete event for a pod, it'll take this long
	// before a dormant controller waiting for those packets is woken up anyway. It is
	// specifically targeted at the case where some problem prevents an update
	// of expectations, without it the controller could stay asleep forever. This should
	// be set based on the expected latency of watch events.
	//
	// Currently a controller can service (create *and* observe the watch events for said
	// creation) about 10 pods a second, so it takes about 1 min to service
	// 500 pods. Just creation is limited to 20qps, and watching happens with ~10-30s
	// latency/pod at the scale of 3000 pods over 100 nodes.
	ExpectationsTimeout = 5 * time.Minute

The reasoning is a bit generous: 500 pods at 10/second is a minute, and the watch latency at scale is 10–30 seconds per pod. Five minutes is a safety margin on top of that, because the failure mode being protected against is the controller being stuck, and a stuck controller is much worse than a late one.

The check itself is a wall-clock comparison, not a TTL store:

 go
 
 
// pkg/controller/controller_utils.go:225
func (exp *ControlleeExpectations) isExpired() bool {
	return clock.RealClock{}.Since(exp.timestamp) > ExpectationsTimeout
}

Raising and lowering

Raising happens immediately before the write. In the ReplicaSet controller, it is a single line right before the batch of creates:

 go
 
 
// pkg/controller/replicaset/replica_set.go:666
		// TODO: Track UIDs of creates just like deletes. The problem currently
		// is we'd need to wait on the result of a create to record the pod's
		// UID, which would require locking *across* the create, which will turn
		// into a performance bottleneck. We should generate a UID for the pod
		// beforehand and store it via ExpectCreations.
		rsc.expectations.ExpectCreations(logger, rsKey, diff)

Note the TODO. Creates are tracked by count; deletes are tracked by UID. The reason is spelled out in the comment: to record a created pod’s UID, you would have to hold a lock across the Create call, which serialises the controller. Counting is fine for creates because a create is idempotent from the controller’s point of view — it will be counted as one observation whenever the pod shows up.

Lowering happens in the informer handlers. On the create side:

 go
 
 
// pkg/controller/replicaset/replica_set.go:487
		logger.V(4).Info("Pod created", "pod", klog.KObj(pod), "detail", pod)
		rsc.expectations.CreationObserved(logger, rsKey)
		rsc.queue.Add(rsKey)

and the counter manipulation is atomic:

 go
 
 
// pkg/controller/controller_utils.go:244
// Decrements the expectation counts of the given controller.
func (r *ControllerExpectations) LowerExpectations(logger klog.Logger, controllerKey string, add, del int) {
	if exp, exists, err := r.GetExpectations(controllerKey); err == nil && exists {
		exp.Add(int64(-add), int64(-del))
		// The expectations might've been modified since the update on the previous line.
		logger.V(4).Info("Lowered expectations", "expectations", exp)
	}
}

exp.Add does two atomic adds, and Fulfilled does two atomic loads:

 go
 
 
// pkg/controller/controller_utils.go:282
// Add increments the add and del counters.
func (e *ControlleeExpectations) Add(add, del int64) {
	atomic.AddInt64(&e.add, add)
	atomic.AddInt64(&e.del, del)
}

// Fulfilled returns true if this expectation has been fulfilled.
func (e *ControlleeExpectations) Fulfilled() bool {
	// TODO: think about why this line being atomic doesn't matter
	return atomic.LoadInt64(&e.add) <= 0 && atomic.LoadInt64(&e.del) <= 0
}

The counters can go negative, and that is deliberate: <= 0, not == 0. An extra observed creation (a pod created by someone else, or a duplicate watch event) drives the counter below zero, and the expectation stays satisfied. Trying to match == 0 would mean a controller that observed one too many events could never satisfy its expectations again until the timeout.

Where the gate is

The gate is one boolean in the sync path. In the ReplicaSet controller:

 go
 
 
// pkg/controller/replicaset/replica_set.go:793
	rsNeedsSync := rsc.expectations.SatisfiedExpectations(logger, key)

and then, later, the actual reconciliation is conditional on it:

 go
 
 
// pkg/controller/replicaset/replica_set.go:820
	var manageReplicasErr error
	var nextSyncDuration *time.Duration
	if rsNeedsSync && rs.DeletionTimestamp == nil {
		manageReplicasErr = rsc.manageReplicas(ctx, activePods, rs)
	}

Note what is not conditional: the status update still happens. The controller refuses to create and delete pods while its own writes are in flight, but it still recomputes and publishes status.replicas, status.readyReplicas and friends. That distinction matters — a controller that skips the status update would publish stale replica counts, and the Deployment controller above it would act on those stale counts.

So the gate is not “do nothing”. The gate is “do not mutate while your mutations are unobserved”.

Deletes are tracked by UID

Creates are counted. Deletes are tracked by UID, and the wrapper type that does it explains why:

 go
 
 
// pkg/controller/controller_utils.go:334
// UIDTrackingControllerExpectations tracks the UID of the pods it deletes.
// This cache is needed over plain old expectations to safely handle graceful
// deletion. The desired behavior is to treat an update that sets the
// DeletionTimestamp on an object as a delete. To do so consistently, one needs
// to remember the expected deletes so they aren't double counted.
// TODO: Track creates as well (#22599)
type UIDTrackingControllerExpectations struct {
	ControllerExpectationsInterface
	// TODO: There is a much nicer way to do this that involves a single store,
	// a lock per entry, and a ControlleeExpectationsInterface type.
	uidStoreLock sync.Mutex
	// Store used for the UIDs associated with any expectation tracked via the
	// ControllerExpectationsInterface.
	uidStore cache.Store
}

The mechanism: when the controller deletes pods, it records the UIDs it deleted, not just the count:

 go
 
 
// pkg/controller/controller_utils.go:360
// ExpectDeletions records expectations for the given deleteKeys, against the given controller.
func (u *UIDTrackingControllerExpectations) ExpectDeletions(logger klog.Logger, rcKey string, deletedKeys []string) error {
	expectedUIDs := sets.NewString()
	for _, k := range deletedKeys {
		expectedUIDs.Insert(k)
	}
	logger.V(4).Info("Controller waiting on deletions", "controller", rcKey, "keys", deletedKeys)
	u.uidStoreLock.Lock()
	defer u.uidStoreLock.Unlock()

	if existing := u.GetUIDs(rcKey); existing != nil && existing.Len() != 0 {
		logger.Error(nil, "Clobbering existing delete keys", "keys", existing)
	}
	if err := u.uidStore.Add(&UIDSet{expectedUIDs, rcKey}); err != nil {
		return err
	}
	return u.ControllerExpectationsInterface.ExpectDeletions(logger, rcKey, expectedUIDs.Len())
}

and the observation only counts if the UID is one it was actually waiting for:

 go
 
 
// pkg/controller/controller_utils.go:379
// DeletionObserved records the given deleteKey as a deletion, for the given rc.
func (u *UIDTrackingControllerExpectations) DeletionObserved(logger klog.Logger, rcKey, deleteKey string) {
	u.uidStoreLock.Lock()
	defer u.uidStoreLock.Unlock()

	uids := u.GetUIDs(rcKey)
	if uids != nil && uids.Has(deleteKey) {
		logger.V(4).Info("Controller received delete for pod", "controller", rcKey, "key", deleteKey)
		u.ControllerExpectationsInterface.DeletionObserved(logger, rcKey)
		uids.Delete(deleteKey)
	}
}

The uids.Has(deleteKey) check is the whole point. Without it, a pod that was not one of the ones the controller deleted — a pod that happened to be deleted by the kubelet, or by a human — would lower the delete expectation and make the controller think its deletions had landed. Worse: a pod deleted and recreated with the same name but a different UID would be miscounted, because the key is the name and the UID is the identity.

Now the graceful deletion. A pod with a terminationGracePeriod is not removed from the store the instant the delete is issued: first the deletionTimestamp is set, and only later does the kubelet actually remove it. The controller treats the update as the delete:

 go
 
 
// pkg/controller/replicaset/replica_set.go:519
	labelChanged := !reflect.DeepEqual(curPod.Labels, oldPod.Labels)
	if curPod.DeletionTimestamp != nil {
		// when a pod is deleted gracefully it's deletion timestamp is first modified to reflect a grace period,
		// and after such time has passed, the kubelet actually deletes it from the store. We receive an update
		// for modification of the deletion timestamp and expect an rs to create more replicas asap, not wait
		// until the kubelet actually deletes the pod. This is different from the Phase of a pod changing, because
		// an rs never initiates a phase change, and so is never asleep waiting for the same.
		rsc.deletePod(logger, curPod)
		if labelChanged {
			// we don't need to check the oldPod.DeletionTimestamp because DeletionTimestamp cannot be unset.
			rsc.deletePod(logger, oldPod)
		}
		return
	}

So the deletion is observed twice — once as an update that sets the timestamp, once as the final delete — and the UID tracking is what makes the second one a no-op: the UID was already removed from the expected set, so uids.Has(deleteKey) is false and the counter is not decremented again.

Adoption has the same problem

Expectations are about the controller’s own writes. Adoption is about someone else’s writes, and it has an analogous race: the controller lists orphaned pods from its cache, decides to adopt one, and issues a patch — but between the list and the patch, another controller may have claimed it.

The guard is a live re-read, with the original issue number preserved:

 go
 
 
// pkg/controller/replicaset/replica_set.go:863
func (rsc *ReplicaSetController) claimPods(ctx context.Context, rs *apps.ReplicaSet, selector labels.Selector, filteredPods []*v1.Pod) ([]*v1.Pod, error) {
	// If any adoptions are attempted, we should first recheck for deletion with
	// an uncached quorum read sometime after listing Pods (see #42639).
	canAdoptFunc := controller.RecheckDeletionTimestamp(func(ctx context.Context) (metav1.Object, error) {
		fresh, err := rsc.kubeClient.AppsV1().ReplicaSets(rs.Namespace).Get(ctx, rs.Name, metav1.GetOptions{})
		if err != nil {
			return nil, err
		}
		if fresh.UID != rs.UID {
			return nil, fmt.Errorf("original %v %v/%v is gone: got uid %v, wanted %v", rsc.Kind, rs.Namespace, rs.Name, fresh.UID, rs.UID)
		}
		return fresh, nil
	})

“uncached quorum read” means a GET straight to the API server, bypassing the informer cache, to be sure the object being adopted is not already being deleted. The check itself is a one-liner:

 go
 
 
// pkg/controller/controller_ref_manager.go:391
func RecheckDeletionTimestamp(getObject func(context.Context) (metav1.Object, error)) func(context.Context) error {
	return func(ctx context.Context) error {
		obj, err := getObject(ctx)
		if err != nil {
			return fmt.Errorf("can't recheck DeletionTimestamp: %v", err)
		}
		if obj.GetDeletionTimestamp() != nil {
			return fmt.Errorf("%v/%v has just been deleted at %v", obj.GetNamespace(), obj.GetName(), obj.GetDeletionTimestamp())
		}
		return nil
	}
}

And the whole adoption path is guarded by a sync.Once, because the check is expensive and its answer does not change within a sync:

 go
 
 
// pkg/controller/controller_ref_manager.go:36
type BaseControllerRefManager struct {
	Controller metav1.Object
	Selector labels.Selector

	canAdoptErr error
	canAdoptOnce sync.Once
	CanAdoptFunc func(ctx context.Context) error
}

func (m *BaseControllerRefManager) CanAdopt(ctx context.Context) error {
	m.canAdoptOnce.Do(func() {
		if m.CanAdoptFunc != nil {
			m.canAdoptErr = m.CanAdoptFunc(ctx)
		}
	})
	return m.canAdoptErr
}

The ClaimObject logic that uses it is the standard triad — adopt orphans, release objects we own that no longer match, ignore objects owned by someone else:

 go
 
 
// pkg/controller/controller_ref_manager.go:69
func (m *BaseControllerRefManager) ClaimObject(ctx context.Context, obj metav1.Object, match func(metav1.Object) bool, adopt, release func(context.Context, metav1.Object) error) (bool, error) {
	controllerRef := metav1.GetControllerOfNoCopy(obj)
	if controllerRef != nil {
		if controllerRef.UID != m.Controller.GetUID() {
			// Owned by someone else. Ignore.
			return false, nil
		}
		if match(obj) {
			// We already own it and the selector matches.
			// Return true (successfully claimed) before checking deletion timestamp.
			// We're still allowed to claim things we already own while being deleted
			// because doing so requires taking no actions.
			return true, nil
		}

Slow start, and what it costs

Expectations stop the controller from acting on a stale cache. They do not stop it from hammering the API server when it is being rate-limited or quota-limited. A ReplicaSet that wants 500 pods and has a quota of 10 will issue 500 creates, get 490 403s, and spam 490 events.

The mitigation is a slow-start batch:

 go
 
 
// pkg/controller/controller_utils.go:73
	// When batching pod creates, SlowStartInitialBatchSize is the size of the
	// initial batch. The size of each successive batch is twice the size of
	// the previous batch. For example, for a value of 1, batch sizes would be
	// 1, 2, 4, 8, ... and for a value of 10, batch sizes would be
	// 10, 20, 40, 80, ... Setting the value higher means that quota denials
	// will result in more doomed API calls and associated event spam. Setting
	// the value lower will result in more API call round trip periods for
	// large batches.
	//
	// Given a number of pods to start "N":
	// The number of doomed calls per sync once quota is exceeded is given by:
	// min(N,SlowStartInitialBatchSize)
	// The number of batches is given by:
	// 1+floor(log_2(ceil(N/SlowStartInitialBatchSize)))
	SlowStartInitialBatchSize = 1

With the default of 1, the batch sizes are 1, 2, 4, 8, … and the number of doomed calls is bounded by the first batch, because the whole thing aborts on the first failure:

 go
 
 
// pkg/controller/replicaset/replica_set.go:891
func slowStartBatch(count int, initialBatchSize int, fn func() error) (int, error) {
	remaining := count
	successes := 0
	for batchSize := min(remaining, initialBatchSize); batchSize > 0; batchSize = min(2*batchSize, remaining) {
		errCh := make(chan error, batchSize)
		var wg sync.WaitGroup
		wg.Add(batchSize)
		for i := 0; i < batchSize; i++ {
			go func() {
				defer wg.Done()
				if err := fn(); err != nil {
					errCh <- err
				}
			}()
		}
		wg.Wait()
		curSuccesses := batchSize - len(errCh)
		successes += curSuccesses
		if len(errCh) > 0 {
			return successes, <-errCh
		}
		remaining -= batchSize
	}
	return successes, nil
}

Each batch runs concurrently, the whole batch is awaited, and a single failure in a batch aborts every remaining batch. And the pods that were never attempted have their expectations walked back, so the controller does not sit waiting for creates it never made:

 go
 
 
// pkg/controller/replicaset/replica_set.go:693
		// Any skipped pods that we never attempted to start shouldn't be expected.
		// The skipped pods will be retried later. The next controller resync will
		// retry the slow start process.
		if skippedPods := diff - successfulCreations; skippedPods > 0 {
			logger.V(2).Info("Slow-start failure. Skipping creation of pods, decrementing expectations", "podsSkipped", skippedPods, "kind", rsc.Kind, "replicaSet", klog.KObj(rs))
			for i := 0; i < skippedPods; i++ {
				// Decrement the expected number of creates because the informer won't observe this pod
				rsc.expectations.CreationObserved(logger, rsKey)

That last loop is a nice illustration of the whole design: expectations are a budget, and any code path that decides not to do a write must return the budget.

Expectations buy correctness with latency. A controller that has just created pods will not act again until those pods come back through the watch, which is tens of milliseconds in a small cluster and seconds in a large one. That is the correct trade: a duplicated pod costs more than a delayed reconcile.

They also cost a little complexity, and the complexity has a sharp edge: if you raise expectations and then fail to write, or write and fail to observe, the controller stalls for up to ExpectationsTimeout. Every write path must therefore be paired with either a matching observation or an explicit rollback of the expectation — the skippedPods loop above is that rollback.

Wrapping up

The one thing to remember: expectations are two int64 counters per controller key — adds and deletes — stored in a cache.Store, raised before a write and lowered by the informer handler when the write comes back. The gate is SatisfiedExpectations (pkg/controller/controller_utils.go:195), and it stops the controller from mutating while its own mutations are unobserved, but never stops it from publishing status.

The part people get wrong: that the counters must hit exactly zero. They do not: Fulfilled() is <= 0, so an extra observation drives the counter negative and the expectation stays satisfied. And deletes are tracked by UID, not by count or name — UIDTrackingControllerExpectations (controller_utils.go:340) exists so that a graceful deletion observed twice (once as the deletionTimestamp update, once as the final delete) is counted once, and so that a pod recreated under the same name is a different pod.

What to go do about it: if a controller seems stuck, look at ExpectationsTimeout (5 minutes, pkg/controller/controller_utils.go:72) and ask whether a watch event was dropped — the timeout is the only thing that recovers from that. If you are writing a controller, raise the expectation before the write, lower it in the informer handler, and make sure every early-return path between those two points walks the expectation back — the skippedPods loop in pkg/controller/replicaset/replica_set.go:693 is the canonical example. And if your controller is adoption-happy, take the RecheckDeletionTimestamp re-read from pkg/controller/replicaset/replica_set.go:866: it is a live GET, not a cache read, and it exists because of issue #42639.

Not a Random Suffix: The Deployment Hash and Its Collision Counter

Jay Hubbert — Fri, 23 Oct 2026 13:10:00 +0000

TL;DR: Every ReplicaSet a Deployment creates is named <deployment-name>-<hash>, and the hash is FNV-32a over the pod template — about 4.3 billion possible values, so collisions are a matter of statistics, not of if. The interesting part is what happens on a collision: the controller does not generate a random suffix. It increments a persisted status.collisionCount on the Deployment and rehashes. Because the counter lives in the Deployment’s status, the name stays a pure function of (template, counter) and is identical across controller restarts, resyncs and multiple replicas of the controller manager.

kubectl get rs shows names like nginx-7d9f4b6c8 or coredns-5d78c9869. The suffix looks like a random string. It is not. It is a hash, and the hash is load-bearing in a way that is easy to miss: the Deployment controller uses it to decide whether the ReplicaSet it wants already exists, and the pod-template-hash label it puts on the pod template is how the ReplicaSet controller decides which pods belong to which ReplicaSet.

Get that name wrong and the controller creates duplicate ReplicaSets, which means duplicate pods, which means a rollout that never converges.

The hash

It is short, and it is FNV-32a over the whole pod template:

 go
 
 
// pkg/controller/controller_utils.go:1402
func ComputeHash(template *v1.PodTemplateSpec, collisionCount *int32) string {
	podTemplateSpecHasher := fnv.New32a()
	hashutil.DeepHashObject(podTemplateSpecHasher, *template)

	// Add collisionCount in the hash if it exists.
	if collisionCount != nil {
		collisionCountBytes := make([]byte, 8)
		binary.LittleEndian.PutUint32(collisionCountBytes, uint32(*collisionCount))
		podTemplateSpecHasher.Write(collisionCountBytes)
	}

	return rand.SafeEncodeString(fmt.Sprint(podTemplateSpecHasher.Sum32()))
}

Three details:

fnv.New32a() — FNV-1a, 32 bits. The space is 2^32 ≈ 4.29 × 10^9. With a few thousand distinct pod templates in a cluster, the birthday probability is around 10^-3 per 1000 templates — small, but a Deployment that is recreated from a template farm (one template per tenant, say) can walk into it.
hashutil.DeepHashObject is not json.Marshal. It writes a spew dump of the object into the hasher, with a config that is explicitly frozen:

 go
 
 
// vendor/k8s.io/utils/dump/dump.go:30
// The config MUST NOT be changed because that could change the result of a hash operation
var prettyPrintConfigForHash = &spew.ConfigState{
	Indent: " ",
	SortKeys: true,
	DisableMethods: true,
	SpewKeys: true,
	DisablePointerAddresses: true,
	DisableCapacities: true,
}

and the function that feeds it to the hasher:

 go
 
 
// vendor/k8s.io/utils/dump/dump.go:46
// ForHash keeps the original Spew.Sprintf format to ensure the same checksum
func ForHash(a interface{}) string {
	return prettyPrintConfigForHash.Sprintf("%#v", a)
}

SortKeys: true is what makes the hash stable across map iteration order. DisablePointerAddresses is what makes it stable across pointer identity. DisableMethods means it hashes the structure, not a String() method. And the “MUST NOT be changed” comment is a warning that this is a persistence format: change the spew config and every existing object’s pod-template-hash label becomes a lie, which means every Deployment in the cluster suddenly has a “new” template and rolls all its pods.

The collision counter is mixed in after the template, as 8 little-endian bytes. It is nil for a fresh Deployment, so the hash of the very first template is a pure function of the template alone.

The encoding

The raw hash is a uint32. It is printed in decimal and then run through SafeEncodeString, which is not a base-N encoder but a per-character substitution:

 go
 
 
// staging/src/k8s.io/apimachinery/pkg/util/rand/rand.go:119
// SafeEncodeString encodes s using the same characters as rand.String. This reduces the chances of bad words and
// ensures that strings generated from hash functions appear consistent throughout the API.
func SafeEncodeString(s string) string {
	r := make([]byte, len(s))
	for i, b := range []rune(s) {
		r[i] = alphanums[(int(b) % len(alphanums))]
	}
	return string(r)
}

and the alphabet it substitutes into has vowels removed on purpose:

 go
 
 
// staging/src/k8s.io/apimachinery/pkg/util/rand/rand.go:80
const (
	// We omit vowels from the set of available characters to reduce the chances
	// of "bad words" being formed.
	alphanums = "bcdfghjklmnpqrstvwxz2456789"
	// No. of bits required to index into alphanums string.
	alphanumsIdxBits = 5
	// Mask used to extract last alphanumsIdxBits of an int.
	alphanumsIdxMask = 1<<alphanumsIdxBits - 1
	// No. of random letters we can extract from a single int63.
	maxAlphanumsPerInt = 63 / alphanumsIdxBits
)

Twenty-seven characters, no a, e, i, o, u, y, and no 0, 1 or 3. The rationale is stated plainly in the comment: the hash ends up in object names and in kubectl output, and a name that spells something unfortunate is a support ticket. It also cannot start with a digit, because the decimal digits of the hash are what get substituted, and they map as 0→b, 1→c, 2→d, 3→f, 4→4, 5→5, 6→6, 7→7, 8→8, 9→9 — so the only characters ever fed to SafeEncodeString are 0–9, and the substitution is injective on those ten characters. The encoding therefore never introduces a collision that the hash did not already have, and the resulting suffix is one to ten characters long depending on the magnitude of the hash.

The name

The hash is not the name. The name is the hash plus a prefix:

 go
 
 
// pkg/controller/deployment/sync.go:196
	// new ReplicaSet does not exist, create one.
	newRSTemplate := *d.Spec.Template.DeepCopy()
	podTemplateSpecHash := controller.ComputeHash(&newRSTemplate, d.Status.CollisionCount)
	newRSTemplate.Labels = labelsutil.CloneAndAddLabel(d.Spec.Template.Labels, apps.DefaultDeploymentUniqueLabelKey, podTemplateSpecHash)
	// Add podTemplateHash label to selector.
	newRSSelector := labelsutil.CloneSelectorAndAddLabel(d.Spec.Selector, apps.DefaultDeploymentUniqueLabelKey, podTemplateSpecHash)

	// Create new ReplicaSet
	newRS := apps.ReplicaSet{
		ObjectMeta: metav1.ObjectMeta{
			// Make the name deterministic, to ensure idempotence
			Name: generateReplicaSetName(d.Name, podTemplateSpecHash),
			Namespace: d.Namespace,
			OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(d, controllerKind)},
			Labels: newRSTemplate.Labels,
		},

The comment is the requirement: “Make the name deterministic, to ensure idempotence.” The controller may run several times, from several replicas of kube-controller-manager, and must always converge on the same name.

The prefix is truncated to fit, because the name has to be a DNS-1123 subdomain:

 go
 
 
// pkg/controller/deployment/sync.go:559
func generateReplicaSetName(deploymentName, podTemplateSpecHash string) string {
	maxDeploymentNameLength := validation.DNS1123SubdomainMaxLength - len(replicaSetNameSeparator) - len(podTemplateSpecHash)

	if len(deploymentName) > maxDeploymentNameLength && maxDeploymentNameLength > 0 {
		return deploymentName[:maxDeploymentNameLength] + replicaSetNameSeparator + podTemplateSpecHash
	}

	return deploymentName + replicaSetNameSeparator + podTemplateSpecHash
}

DNS1123SubdomainMaxLength is 253, the separator is -, and the hash is up to ten characters. A Deployment with a 250-character name gets its prefix cut to 242 characters so the whole thing still fits.

What happens on a collision

The collision is discovered the only way it can be: the Create call comes back AlreadyExists.

 go
 
 
// pkg/controller/deployment/sync.go:231
	alreadyExists := false
	createdRS, err := dc.client.AppsV1().ReplicaSets(d.Namespace).Create(ctx, &newRS, metav1.CreateOptions{})
	switch {
	// We may end up hitting this due to a slow cache or a fast resync of the Deployment.
	case errors.IsAlreadyExists(err):
		alreadyExists = true

AlreadyExists is ambiguous. It might mean “this exact ReplicaSet is already here, because you are re-syncing” — the normal, boring case — or it might mean “there is a different ReplicaSet with this name, because your hash collided with someone else’s”. The controller distinguishes the two by comparing the pod templates, ignoring the hash label itself:

 go
 
 
// pkg/controller/deployment/util/deployment_util.go:610
// EqualIgnoreHash returns true if two given podTemplateSpec are equal, ignoring the diff in value of Labels[pod-template-hash]
// We ignore pod-template-hash because:
// 1. The hash result would be different upon podTemplateSpec API changes
// (e.g. the addition of a new field will cause the hash code to change)
// 2. The deployment template won't have hash labels
func EqualIgnoreHash(template1, template2 *v1.PodTemplateSpec) bool {
	t1Copy := template1.DeepCopy()
	t2Copy := template2.DeepCopy()
	// Remove hash labels from template.Labels before comparing
	delete(t1Copy.Labels, apps.DefaultDeploymentUniqueLabelKey)
	delete(t2Copy.Labels, apps.DefaultDeploymentUniqueLabelKey)
	return apiequality.Semantic.DeepEqual(t1Copy, t2Copy)
}

Semantic.DeepEqual is used rather than reflect.DeepEqual so that nil and empty slices, and resource.Quantity values with different internal representations but the same magnitude, compare equal.

If the templates are equal, this is just a re-sync and the existing ReplicaSet is adopted. If they are not equal, it is a genuine hash collision — and here is the interesting part:

 go
 
 
// pkg/controller/deployment/sync.go:255
		// Matching ReplicaSet is not equal - increment the collisionCount in the DeploymentStatus
		// and requeue the Deployment.
		if d.Status.CollisionCount == nil {
			d.Status.CollisionCount = new(int32)
		}
		preCollisionCount := *d.Status.CollisionCount
		*d.Status.CollisionCount++
		// Update the collisionCount for the Deployment and let it requeue by returning the original
		// error.
		_, dErr := dc.client.AppsV1().Deployments(d.Namespace).UpdateStatus(ctx, d, metav1.UpdateOptions{})
		if dErr == nil {
			logger.V(2).Info("Found a hash collision for deployment - bumping collisionCount to resolve it", "deployment", klog.KObj(d), "oldCollisionCount", preCollisionCount, "newCollisionCount", *d.Status.CollisionCount)
		}
		return nil, err

The counter is bumped in the Deployment’s status and written with UpdateStatus. This is the crux of the design. The obvious alternative — a random suffix, or a timestamp — would break idempotence: two controllers racing on the same Deployment would generate two different names and create two ReplicaSets. A persisted counter makes the name a pure function of the inputs, so every controller computes the same next name.

The counter is part of the API, documented as such:

 go
 
 
// staging/src/k8s.io/api/apps/v1/types.go:574
	// collisionCount is the number of hash collisions for the Deployment. The Deployment controller uses this
	// field as a collision avoidance mechanism when it needs to create the name for the newest ReplicaSet.
	// +optional
	CollisionCount *int32 `json:"collisionCount,omitempty" protobuf:"varint,8,opt,name=collisionCount"`

Same field, same purpose, on StatefulSet (line 330) and DaemonSet (line 777).

The ControllerRevision variant

StatefulSet and DaemonSet do not create ReplicaSets, so they roll out via ControllerRevision objects — immutable snapshots of the pod template kept for rollback. The hash is computed differently: plain FNV-1 (not FNV-1a), and the collision counter is mixed in as a decimal string:

 go
 
 
// pkg/controller/history/controller_history.go:132
// HashControllerRevision hashes the contents of revision's Data using FNV hashing. If probe is not nil, the byte value
// of probe is added written to the hash as well. The returned hash will be a safe encoded string to avoid bad words.
func HashControllerRevision(revision *apps.ControllerRevision, probe *int32) string {
	hf := fnv.New32()
	if len(revision.Data.Raw) > 0 {
		hf.Write(revision.Data.Raw)
	}
	if revision.Data.Object != nil {
		hashutil.DeepHashObject(hf, revision.Data.Object)
	}
	if probe != nil {
		hf.Write([]byte(strconv.FormatInt(int64(*probe), 10)))
	}
	return rand.SafeEncodeString(fmt.Sprint(hf.Sum32()))
}

The difference in the collision-count encoding is not cosmetic. ComputeHash writes the counter as 8 fixed-width little-endian bytes; this writes strconv.FormatInt of the counter. With a fixed-width encoding, a counter of 1 and a counter of 256 produce different byte strings; with the decimal string, "1" followed by "256" is b"1256" and a counter of 12 followed by "56" is also b"1256" — a genuine ambiguity, but only reachable by concatenating two different fields, and in practice the revision payload is either raw bytes or an object, never both. The point of mixing the counter in at all is the same as before: it must change the hash.

Creation is a retry loop rather than a single call, and the counter is bumped in memory on AlreadyExists:

 go
 
 
// pkg/controller/history/controller_history.go:293
func (rh *realHistory) CreateControllerRevision(parent metav1.Object, revision *apps.ControllerRevision, collisionCount *int32) (*apps.ControllerRevision, error) {
	if collisionCount == nil {
		return nil, fmt.Errorf("collisionCount should not be nil")
	}

	// Clone the input
	clone := revision.DeepCopy()

	// Continue to attempt to create the revision updating the name with a new hash on each iteration
	for {
		hash := HashControllerRevision(revision, collisionCount)
		// Update the revisions name
		clone.Name = ControllerRevisionName(parent.GetName(), hash)
		ns := parent.GetNamespace()
		created, err := rh.client.AppsV1().ControllerRevisions(ns).Create(context.TODO(), clone, metav1.CreateOptions{})
		if errors.IsAlreadyExists(err) {
			exists, err := rh.client.AppsV1().ControllerRevisions(ns).Get(context.TODO(), clone.Name, metav1.GetOptions{})
			if err != nil {
				return nil, err
			}
			if bytes.Equal(exists.Data.Raw, clone.Data.Raw) {
				return exists, nil
			}
			*collisionCount++
			continue
		}
		return created, err
	}
}

Note the difference in style from the Deployment: the Deployment bumps the counter and persists it to the API and returns, letting the informer wake it up again. The history controller bumps it in memory and loops, because it holds the pointer to the parent’s status object. The DaemonSet does something in between, and it is the one with the interesting guard.

The DaemonSet’s stale-counter guard

The DaemonSet creates ControllerRevisions with the Deployment’s ComputeHash, and it hits AlreadyExists in exactly the same way:

 go
 
 
// pkg/controller/daemon/update.go:564
		// Handle name collisions between different history
		// Get the latest DaemonSet from the API server to make sure collision count is only increased when necessary
		currDS, getErr := dsc.kubeClient.AppsV1().DaemonSets(ds.Namespace).Get(ctx, ds.Name, metav1.GetOptions{})
		if getErr != nil {
			return nil, getErr
		}
		// If the collision count used to compute hash was in fact stale, there's no need to bump collision count; retry again
		if !reflect.DeepEqual(currDS.Status.CollisionCount, ds.Status.CollisionCount) {
			return nil, fmt.Errorf("found a stale collision count (%d, expected %d) of DaemonSet %q while processing; will retry until it is updated", ptr.Deref(ds.Status.CollisionCount, 0), ptr.Deref(currDS.Status.CollisionCount, 0), ds.Name)
		}

This is a guard against bumping the counter for no reason. If the counter the DaemonSet controller was working with is already stale — someone else bumped it — then the collision it detected has already been resolved by the other bump. Bumping again would skip a hash value and, worse, would race another writer on the status.

The DaemonSet therefore does a live GET of its own object before touching the counter, and refuses to bump if the value it is holding differs from the one on the server. It returns an error and lets the informer retry with a fresh copy.

The Deployment controller does not have this guard — it bumps the counter unconditionally when it detects a collision — which is a small inconsistency between the two controllers that is worth knowing about if you are debugging a DaemonSet that seems to bump its counter more than expected.

Why not a random suffix?

Because a random suffix breaks the property the controller depends on: the ReplicaSet name must be derivable. Every sync recomputes it. The Deployment controller does not look for “the ReplicaSet I created last time”; it computes the name and asks the API server. If the name were random, the controller would need to remember it somewhere, and the only somewhere is the Deployment’s status — at which point you have reinvented the collision counter, with more machinery and less determinism.

The counter approach also degrades gracefully. It is a *int32 in the status. It is zero for every object that has never collided. It is bumped at most once per genuine collision, which for a well-behaved cluster is never.

The cost is a 32-bit hash. A 64-bit hash would make collisions essentially impossible, but it would also change every existing pod-template-hash label in every cluster, which is exactly the kind of “harmless” change that rewrites the world. 32 bits it is, and the counter handles the tail.

Wrapping up

The one thing to remember: the Deployment’s ReplicaSet suffix is rand.SafeEncodeString(fmt.Sprint(fnv32a(template))) — FNV-32a over the spew dump of the pod template, printed in decimal and character-substituted into the vowel-free alphabet bcdfghjklmnpqrstvwxz2456789 (staging/src/k8s.io/apimachinery/pkg/util/rand/rand.go:83). It is a deterministic function, not a random string, and the controller relies on that (pkg/controller/deployment/sync.go:207).

The part people get wrong: the collision resolution. It is not a random suffix and not a timestamp. The controller increments status.collisionCount on the Deployment (pkg/controller/deployment/sync.go:255) and rehashes, so the name stays a pure function of (template, counter). And the bump is only triggered by a genuine collision — AlreadyExists plus a template that is not EqualIgnoreHash-equal (pkg/controller/deployment/util/deployment_util.go:610). A duplicate ReplicaSet with the same template is a re-sync, not a collision.

What to go do about it: if a Deployment seems to be creating duplicate ReplicaSets, look at kubectl get deploy <name> -o jsonpath='{.status.collisionCount}' — a non-zero value means a real hash collision happened and the controller worked around it. If the pod-template-hash label changes for every pod in the cluster after an upgrade, suspect the spew hashing config in vendor/k8s.io/utils/dump/dump.go:30 — it is explicitly frozen because changing it invalidates every stored hash. And if you write your own controller that names objects after a hash, put the salt in the object’s status, not in memory: StatefulSet and DaemonSet use the same status.collisionCount field, and the DaemonSet controller re-reads the object before bumping it precisely to avoid clobbering a concurrent bump (pkg/controller/daemon/update.go:566).

Solid, Dangling, Waiting: How the Garbage Collector Decides to Delete

Jay Hubbert — Fri, 23 Oct 2026 12:40:00 +0000

TL;DR: The garbage collector never deletes an object just because it has owner references. It asks the API server — through a metadata-only client, so only PartialObjectMetadata crosses the wire — whether each owner still exists, and sorts the references into three buckets: solid (the owner exists), dangling (it does not) and waiting (it exists, is being deleted, and carries the foregroundDeletion finalizer). Solid owners mean “hands off”; dangling owners mean “delete, or strip the reference”; waiting owners mean “not yet”. There is also a cycle breaker: two objects that are each waiting on the other are unblocked by patching the owner references to non-blocking and deleting with DeletePropagationForeground.

Ownership in Kubernetes is a promise, not a lock. A Deployment sets an owner reference on its ReplicaSets, a ReplicaSet sets one on its Pods, and a Job sets one on the pods it creates. When the owner goes away, the dependents should go away with it — but “should” is doing a lot of work in that sentence, because a dependent can have several owners, one owner can be deleted and recreated with the same name but a different UID, and the API server will happily accept a reference to an object that never existed.

The garbage collector is the component that has to reconcile all of that without ever holding the whole object graph in memory at once, and without racing the API server’s own cascading delete.

Three buckets, not one

The classification happens in one place, and it is short enough to read in full:

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:455
// classify the latestReferences to three categories:
// solid: the owner exists, and is not "waitingForDependentsDeletion"
// dangling: the owner does not exist
// waitingForDependentsDeletion: the owner exists, its deletionTimestamp is non-nil, and it has
// FinalizerDeletingDependents
// This function communicates with the server.
func (gc *GarbageCollector) classifyReferences(ctx context.Context, item *node, latestReferences []metav1.OwnerReference) (
	solid, dangling, waitingForDependentsDeletion []metav1.OwnerReference, err error) {
	for _, reference := range latestReferences {
		isDangling, owner, err := gc.isDangling(ctx, reference, item)
		if err != nil {
			return nil, nil, nil, err
		}
		if isDangling {
			dangling = append(dangling, reference)
			continue
		}

		ownerAccessor, err := meta.Accessor(owner)
		if err != nil {
			return nil, nil, nil, err
		}
		if ownerAccessor.GetDeletionTimestamp() != nil && hasDeleteDependentsFinalizer(ownerAccessor) {
			waitingForDependentsDeletion = append(waitingForDependentsDeletion, reference)
		} else {
			solid = append(solid, reference)
		}
	}
	return solid, dangling, waitingForDependentsDeletion, nil
}

Two things are decided here. “Does the owner exist?” — that is isDangling. And “if it exists, is it in the middle of being torn down?” — that is the DeletionTimestamp != nil && hasDeleteDependentsFinalizer test, which is what separates waiting from solid.

The finalizer in question is metav1.FinalizerDeleteDependents, which the API server adds when a delete is requested with DeletePropagationForeground. Its presence is the API server’s way of saying “I am waiting for my dependents to go away before I disappear”.

Asking the server, cheaply

The liveness question is answered by the API server, not by the GC’s own cache. That matters: the GC’s graph is built from informers and can lag, and a lagging cache would turn into a false “the owner is gone” and a mass deletion.

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:428
	// TODO: It's only necessary to talk to the API server if the owner node
	// is a "virtual" node. The local graph could lag behind the real
	// status, but in practice, the difference is small.
	owner, err = gc.metadataClient.Resource(resource).Namespace(resourceDefaultNamespace(namespaced, item.identity.Namespace)).Get(ctx, reference.Name, metav1.GetOptions{})
	switch {
	case errors.IsNotFound(err):
		gc.absentOwnerCache.Add(absentOwnerCacheKey)
		logger.V(5).Info("item's owner is not found",
			"item", item.identity,
			"owner", reference,
		)
		return true, nil, nil
	case err != nil:
		return false, nil, err
	}

	if owner.GetUID() != reference.UID {
		logger.V(5).Info("item's owner is not found, UID mismatch",
			"item", item.identity,
			"owner", reference,
		)
		gc.absentOwnerCache.Add(absentOwnerCacheKey)
		return true, nil, nil
	}
	return false, owner, nil

Two details are load-bearing.

First, gc.metadataClient is a metadata.Interface — the metadata-only client from k8s.io/client-go/metadata. The returned object is a *metav1.PartialObjectMetadata, which contains only TypeMeta and ObjectMeta. The GC never needs the owner’s spec or status, only its existence, its UID, its deletion timestamp and its finalizers — and fetching a full Pod object to learn four facts about it is exactly the kind of waste that makes a control plane fall over.

Second, the UID check. A name alone is not identity. If a Deployment is deleted and recreated with the same name, its old ReplicaSet still carries an owner reference with the old UID. The name resolves; the UID does not match; the reference is dangling. Without that check, every recreated owner would orphan a generation of dependents.

The absent owner cache

The GET above is a live API call, and the GC makes a lot of them. The absentOwnerCache is the escape hatch — a small LRU of owner references known to be gone:

 go
 
 
// pkg/controller/garbagecollector/uid_cache.go:24
type ReferenceCache struct {
	cache *lru.Cache
}

// NewReferenceCache returns a ReferenceCache.
func NewReferenceCache(maxCacheEntries int) *ReferenceCache {
	return &ReferenceCache{
		cache: lru.New(maxCacheEntries),
	}
}

It is created with a hard-coded size:

 go
 
 
// pkg/controller/garbagecollector/graph_builder.go:165
	absentOwnerCache := NewReferenceCache(500)

500 entries, no flag, no config API. A reference that is known-absent is remembered so the GC does not have to ask again. The cache is checked before the API call, twice, because a cluster-scoped owner and a namespaced owner with the same name are different objects:

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:384
	// check for recorded absent cluster-scoped parent
	absentOwnerCacheKey := objectReference{OwnerReference: ownerReferenceCoordinates(reference)}
	if gc.absentOwnerCache.Has(absentOwnerCacheKey) {
		logger.V(5).Info("according to the absentOwnerCache, item's owner does not exist",
			"item", item.identity,
			"owner", reference,
		)
		return true, nil, nil
	}

	// check for recorded absent namespaced parent
	absentOwnerCacheKey.Namespace = item.identity.Namespace
	if gc.absentOwnerCache.Has(absentOwnerCacheKey) {
		logger.V(5).Info("according to the absentOwnerCache, item's owner does not exist in namespace",
			"item", item.identity,
			"owner", reference,
		)
		return true, nil, nil
	}

The cache is a correctness optimisation, not just a performance one: a dangling reference that is also in the cache short-circuits before the API call, which means the GC does not have to distinguish “owner gone” from “owner gone but a new one exists with the same name” for the cached entries.

The three-way switch

Now the decision. attemptToDeleteItem reads the live object, classifies its owner references, and branches three ways:

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:567
	switch {
	case len(solid) != 0:
		logger.V(2).Info("item has at least one existing owner, will not garbage collect",
			"item", item.identity,
			"owner", solid,
		)
		if len(dangling) == 0 && len(waitingForDependentsDeletion) == 0 {
			return nil
		}
		logger.V(2).Info("remove dangling references and waiting references for item",
			"item", item.identity,
			"dangling", dangling,
			"waitingForDependentsDeletion", waitingForDependentsDeletion,
		)
		// waitingForDependentsDeletion needs to be deleted from the
		// ownerReferences, otherwise the referenced objects will be stuck with
		// the FinalizerDeletingDependents and never get deleted.
		ownerUIDs := append(ownerRefsToUIDs(dangling), ownerRefsToUIDs(waitingForDependentsDeletion)...)
		p, err := c.GenerateDeleteOwnerRefStrategicMergeBytes(item.identity.UID, ownerUIDs)
		if err != nil {
			return err
		}
		_, err = gc.patch(item, p, func(n *node) ([]byte, error) {
			return gc.deleteOwnerRefJSONMergePatch(n, ownerUIDs...)
		})
		return err
	case len(waitingForDependentsDeletion) != 0 && item.dependentsLength() != 0:

Read the first branch carefully. A single solid owner is enough to prevent deletion — the object stays, no matter what else is wrong. But the GC does not simply leave it alone: if some of the other references are dangling, it patches them off. An object with one live owner and three references to long-dead owners keeps its live owner and loses the ghosts, because a dangling reference to a UID that never existed is a permanent trip hazard for anything that walks the graph.

The second branch is the interesting one, and it is the cycle breaker:

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:593
	case len(waitingForDependentsDeletion) != 0 && item.dependentsLength() != 0:
		deps := item.getDependents()
		for _, dep := range deps {
			if dep.isDeletingDependents() {
				// this circle detection has false positives, we need to
				// apply a more rigorous detection if this turns out to be a
				// problem.
				// there are multiple workers run attemptToDeleteItem in
				// parallel, the circle detection can fail in a race condition.
				logger.V(2).Info("processing item, some of its owners and its dependent have FinalizerDeletingDependents, to prevent potential cycle, its ownerReferences are going to be modified to be non-blocking, then the item is going to be deleted with Foreground",
					"item", item.identity,
					"dependent", dep.identity,
				)
				patch, err := item.unblockOwnerReferencesStrategicMergePatch()
				if err != nil {
					return err
				}
				if _, err := gc.patch(item, patch, gc.unblockOwnerReferencesJSONMergePatch); err != nil {
					return err
				}
				break
			}
		}
		logger.V(2).Info("at least one owner of item has FinalizerDeletingDependents, and the item itself has dependents, so it is going to be deleted in Foreground",
			"item", item.identity,
		)
		// the deletion event will be observed by the graphBuilder, so the item
		// will be processed again in processDeletingDependentsItem. If it
		// doesn't have dependents, the function will remove the
		// FinalizerDeletingDependents from the item, resulting in the final
		// deletion of the item.
		policy := metav1.DeletePropagationForeground
		err := gc.deleteObject(item.identity, latest.ResourceVersion, latest.OwnerReferences, &policy)

Here is the deadlock this prevents. Object A owns object B. Both are being deleted with foreground propagation. A is waiting for B to disappear; B is waiting for A to disappear. Neither will ever go away, because each one’s deletion is gated on a dependent that is itself gated.

The fix is to break the blocking nature of the owner references rather than the deletion. unblockOwnerReferencesStrategicMergePatch rewrites the owner references so they no longer carry the blocking flag, and then the item is deleted with DeletePropagationForeground anyway. The foreground deletion still waits for the dependents — it is the owner→dependent direction that gets unblocked.

And when the owner is merely waiting — no cycle, just an owner in the middle of being deleted — the GC does not touch the object at all. It stays put until its owner goes away.

The third branch is the actual delete:

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:631
	default:
		// item doesn't have any solid owner, so it needs to be garbage
		// collected. Also, none of item's owners is waiting for the deletion of
		// the dependents, so set propagationPolicy based on existing finalizers.
		var policy metav1.DeletionPropagation
		switch {
		case hasOrphanFinalizer(latest):
			// if an existing orphan finalizer is already on the object, honor it.
			policy = metav1.DeletePropagationOrphan
		case hasDeleteDependentsFinalizer(latest):
			// if an existing foreground finalizer is already on the object, honor it.
			policy = metav1.DeletePropagationForeground
		default:
			// otherwise, default to background.
			policy = metav1.DeletePropagationBackground
		}

Every owner reference is dangling, so the object is garbage. The propagation policy is not chosen freely: an object that already carries the orphan finalizer gets DeletePropagationOrphan, one that already carries the foreground finalizer gets DeletePropagationForeground, and everything else gets background — which is the cheap one, because background deletion lets the API server cascade without waiting for the dependents to be confirmed gone.

Finishing the foreground deletion

The foreground path needs a second pass. When the GC sees an object that is already deleting its dependents, it checks whether any dependent is still blocking:

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:660
// process item that's waiting for its dependents to be deleted
func (gc *GarbageCollector) processDeletingDependentsItem(logger klog.Logger, item *node) error {
	blockingDependents := item.blockingDependents()
	if len(blockingDependents) == 0 {
		logger.V(2).Info("remove DeleteDependents finalizer for item", "item", item.identity)
		return gc.removeFinalizer(logger, item, metav1.FinalizerDeleteDependents)
	}
	for _, dep := range blockingDependents {
		if !dep.isDeletingDependents() {
			logger.V(2).Info("adding dependent to attemptToDelete, because its owner is deletingDependents",
				"item", item.identity,
				"dependent", dep.identity,
			)
			gc.attemptToDelete.Add(dep)
		}
	}
	return nil
}

If nothing is blocking any more, the foregroundDeletion finalizer is removed, and the API server — which is what actually enforces the finalizer — completes the deletion. If something is still blocking, the dependent is pushed back onto the delete queue and the loop continues.

Orphaning is the mirror image, and it is the one place the GC fans out:

 go
 
 
// pkg/controller/garbagecollector/garbagecollector.go:679
// dependents are copies of pointers to the owner's dependents, they don't need to be locked.
func (gc *GarbageCollector) orphanDependents(logger klog.Logger, owner objectReference, dependents []*node) error {
	errCh := make(chan error, len(dependents))
	wg := sync.WaitGroup{}
	wg.Add(len(dependents))
	for i := range dependents {
		go func(dependent *node) {
			defer wg.Done()
			// the dependent.identity.UID is used as precondition
			p, err := c.GenerateDeleteOwnerRefStrategicMergeBytes(dependent.identity.UID, []types.UID{owner.UID})
			if err != nil {
				errCh <- fmt.Errorf("orphaning %s failed, %v", dependent.identity, err)
				return
			}
			_, err = gc.patch(dependent, p, func(n *node) ([]byte, error) {
				return gc.deleteOwnerRefJSONMergePatch(n, owner.UID)
			})
			// note that if the target ownerReference doesn't exist in the
			// dependent, strategic merge patch will NOT return an error.
			if err != nil && !errors.IsNotFound(err) {
				errCh <- fmt.Errorf("orphaning %s failed, %v", dependent.identity, err)
			}
		}(dependents[i])
	}
	wg.Wait()
	close(errCh)

One goroutine per dependent, because orphaning a Deployment with 200 ReplicaSets means 200 patches, and they are independent. The comment about strategic merge patch not erroring on a missing owner reference is the important part: it makes the orphan operation idempotent, so a retry after a partial failure does not fail.

The graph behind it

All of this runs against a dependency graph that the GC maintains from informers. The graph is a map[UID]*node where each node has a set of dependents, and the concurrency model is spelled out in the type itself:

 go
 
 
// pkg/controller/garbagecollector/graph.go:58
// The single-threaded GraphBuilder.processGraphChanges() is the sole writer of the
// nodes. The multi-threaded GarbageCollector.attemptToDeleteItem() reads the nodes.
// WARNING: node has different locks on different fields. setters and getters
// use the respective locks, so the return values of the getters can be
// inconsistent.
type node struct {
	identity objectReference
	// dependents will be read by the orphan() routine, we need to protect it with a lock.
	dependentsLock sync.RWMutex
	// dependents are the nodes that have node.identity as a
	// metadata.ownerReference.
	dependents map[*node]struct{}
	// this is set by processGraphChanges() if the object has non-nil DeletionTimestamp
	// and has the FinalizerDeleteDependents.
	deletingDependents bool
	deletingDependentsLock sync.RWMutex
	// this records if the object's deletionTimestamp is non-nil.
	beingDeleted bool
	beingDeletedLock sync.RWMutex
	// this records if the object was constructed virtually and never observed via informer event
	virtual bool
	virtualLock sync.RWMutex
	// when processing an Update event, we need to compare the updated
	// ownerReferences with the owners recorded in the graph.
	owners []metav1.OwnerReference
	ownersLock sync.RWMutex
}

One writer, many readers, and per-field locks rather than one lock for the whole node. That is why the comment warns that two getters can be inconsistent with each other: the GC deliberately does not want the cost of a consistent snapshot of the whole node, because the fields it reads are independent facts.

The graph also contains nodes that do not correspond to real objects. When an informer event names an owner the GC has never seen, it creates a virtual node for it:

 go
 
 
// pkg/controller/garbagecollector/graph_builder.go:446
// addDependentToOwners adds n to owners' dependents list. If the owner does not
// exist in the gb.uidToNode yet, a "virtual" node will be created to represent
// the owner. The "virtual" node will be enqueued to the attemptToDelete, so that
// attemptToDeleteItem() will verify if the owner exists according to the API server.
func (gb *GraphBuilder) addDependentToOwners(logger klog.Logger, n *node, owners []metav1.OwnerReference) {
	// track if some of the referenced owners already exist in the graph and have been observed,
	// and the dependent's ownerRef does not match their observed coordinates
	hasPotentiallyInvalidOwnerReference := false

	for _, owner := range owners {
		ownerNode, ok := gb.uidToNode.Read(owner.UID)
		if !ok {
			// Create a "virtual" node in the graph for the owner if it doesn't
			// exist in the graph yet.
			ownerNode = &node{
				identity: objectReference{
					OwnerReference: ownerReferenceCoordinates(owner),
					Namespace: n.identity.Namespace,
				},
				dependents: make(map[*node]struct{}),
				virtual: true,
			}
			logger.V(5).Info("add virtual item", "identity", ownerNode.identity)
			gb.uidToNode.Write(ownerNode)
		}
		ownerNode.addDependent(n)
		if !ok {
			// Enqueue the virtual node into attemptToDelete.
			// The garbage processor will enqueue a virtual delete
			// event to delete it from the graph if API server confirms this
			// owner doesn't exist.
			gb.attemptToDelete.Add(ownerNode)
		} else if !hasPotentiallyInvalidOwnerReference {

The virtual node is the GC’s way of saying “I do not know this owner yet”. It is put on the delete queue, and when the GC processes it, the API server is asked whether it exists. If it does not, the virtual node is removed from the graph with a synthetic delete event:

 go
 
 
// pkg/controller/garbagecollector/graph_builder.go:431
// enqueueVirtualDeleteEvent is used to add a virtual delete event to be processed for virtual nodes
// once it is determined they do not have backing objects in storage
func (gb *GraphBuilder) enqueueVirtualDeleteEvent(ref objectReference) {
	gv, _ := schema.ParseGroupVersion(ref.APIVersion)
	gb.graphChanges.Add(&event{
		virtual: true,
		eventType: deleteEvent,
		gvk: gv.WithKind(ref.Kind),
		obj: &metaonly.MetadataOnlyObject{
			TypeMeta: metav1.TypeMeta{APIVersion: ref.APIVersion, Kind: ref.Kind},
			ObjectMeta: metav1.ObjectMeta{Namespace: ref.Namespace, UID: ref.UID, Name: ref.Name},
		},
	})
}

Note the object in that synthetic event is a metaonly.MetadataOnlyObject — the same metadata-only type used by the API client. The GC’s graph builder never needs the owner’s spec either.

The knobs

There are surprisingly few. The worker count is the main one:

 go
 
 
// pkg/controller/garbagecollector/config/v1alpha1/defaults.go:37
	if obj.ConcurrentGCSyncs == 0 {
		obj.ConcurrentGCSyncs = 20
	}

which is the --concurrent-gc-syncs flag on kube-controller-manager (“The number of garbage collector workers that are allowed to sync concurrently”, cmd/kube-controller-manager/app/options/garbagecollectorcontroller.go:36).

The GC’s client QPS is doubled, because a deletion is two API calls:

 go
 
 
// cmd/kube-controller-manager/app/core.go:664
	// Increase garbage collector controller's throughput: each object deletion takes two API calls,
	// so to get |config.QPS| deletion rate we need to allow 2x more requests for this controller.
	config.QPS *= 2
	metadataClient, err := metadata.NewForConfig(config)

Those two calls are the metadata GET that establishes liveness and the DELETE that removes the object. Both go through the same config, so doubling the QPS doubles both.

The absence of a knob for the absentOwnerCache size (500) is worth noting: it is a fixed constant in graph_builder.go:165, not a config API field.

Wrapping up

The one thing to remember: the GC deletes only when every owner reference is dangling. The classification lives in classifyReferences (pkg/controller/garbagecollector/garbagecollector.go:461), and it is built on a live metadata.Interface GET plus a UID comparison — never on the informer’s cache alone.

The part people get wrong: that a “solid” owner protects the object. It does not protect it cleanly. A solid owner plus some dangling references means the GC still patches the dangling references off the object (garbagecollector.go:589), and an owner that is waiting (being deleted with foreground propagation) can end with the GC unblocking its owner references to break a mutual-delete deadlock (garbagecollector.go:606).

What to go do about it: if an object is not being cleaned up, look at its metadata.ownerReferences and check each owner by UID with kubectl get <owner> -o jsonpath='{.metadata.uid}' — a name that resolves is not enough. If an object is stuck with a foregroundDeletion finalizer, the blocker is a dependent, not the owner; find it with kubectl get <owner> -o jsonpath='{.metadata.finalizers}' and inspect the dependents’ own owner references. And if you are running a large cluster and see GC API traffic dominating the control plane, the QPS budget is --kube-api-qps doubled (cmd/kube-controller-manager/app/core.go:666) and the parallelism is --concurrent-gc-syncs (default 20).