Opensource internals

Before It Evicts Your Node, the Node Controller Checks Its Own Cache

14 min read Jay Hubbert

The node controller verifies an expired lease against a live API GET before declaring a node dead, and short-circuits the entire pass if its informer cache is stale.

TL;DR: The node controller decides a node is dead from the NodeReady heartbeat or from the kubelet’s Lease renewal. Before it acts on an expired lease in the informer cache, it does a live GET of that lease and compares resource versions. If the live lease differs from the cached one, the cache is stale — so the controller does not just skip the node, it short-circuits the entire pass and refuses to evaluate any node, because a stale lease cache would otherwise produce a mass false-positive eviction storm. The feature is NodeControllerLeaseCircuitBreaker, beta and on by default since 1.37.

Evicting a node is the most destructive thing a Kubernetes controller does. It taints the node NoExecute, the kubelet starts killing pods, and the workloads move. If the controller gets that wrong for the wrong reason — a cache that has not caught up, a watch that stalled — the cost is a cluster-wide workload restart.

The node controller is therefore paranoid in a specific way: it knows that its own informer cache can be stale, and it has a mechanism to detect that staleness and refuse to act on it.

What “dead” means

Two signals say a node is alive: the NodeReady condition’s lastHeartbeatTime on the Node object, and the renewTime on the kube-node-lease Lease object. The controller tracks both and uses the more recent one:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:936
	// Always update the probe time if node lease is renewed.
	// Note: If kubelet never posted the node status, but continues renewing the
	// heartbeat leases, the node controller will assume the node is healthy and
	// take no action.
	observedLease, _ := nc.leaseLister.Leases(v1.NamespaceNodeLease).Get(node.Name)
	if observedLease != nil && (savedLease == nil || savedLease.Spec.RenewTime.Before(observedLease.Spec.RenewTime)) {
		nodeHealth.lease = observedLease
		nodeHealth.probeTimestamp = nc.now()
	}

nc.leaseLister is a cache read. The lease the kubelet renews every ten seconds or so arrives through the informer, so there is always a window in which the cached lease is behind the real one.

The grace period decides when that window becomes fatal. It defaults to 50 seconds for a running node and 60 seconds for one that has not yet posted status:

go 
// pkg/controller/nodelifecycle/config/v1alpha1/defaults.go:40
	// NodeMonitorGracePeriod is set to a default value of 50 seconds.
	// This value should be greater than the sum of HTTP2_PING_TIMEOUT_SECONDS (30s)
	// and HTTP2_READ_IDLE_TIMEOUT_SECONDS (15s) from the http2 health check
	// to ensure that the server has adequate time to handle slow or idle connections
	// properly before marking a node as unhealthy.
	if obj.NodeMonitorGracePeriod == zero {
		obj.NodeMonitorGracePeriod = metav1.Duration{Duration: 50 * time.Second}
	}
	if obj.NodeStartupGracePeriod == zero {
		obj.NodeStartupGracePeriod = metav1.Duration{Duration: 60 * time.Second}
	}

Both are flags: --node-monitor-grace-period and --node-startup-grace-period on kube-controller-manager (cmd/kube-controller-manager/app/options/nodelifecyclecontroller.go:39). The comment explains the 50 seconds: it must exceed the sum of the kubelet’s HTTP/2 ping timeout (30s) and read idle timeout (15s), so that a merely-slow kubelet is never mistaken for a dead one.

When the probe timestamp falls outside the grace period, the controller believes the node is dead:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:946
	if nc.now().After(nodeHealth.probeTimestamp.Add(gracePeriod)) {
		if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
			var nodeHealthLeaseRV string
			if nodeHealth.lease != nil {
				nodeHealthLeaseRV = nodeHealth.lease.ResourceVersion
			}
			// The lease instance in the informer cache indicates it is expired.
			// Double-check the live lease is actually expired in case our informer cache is stale.
			liveLease, err := nc.kubeClient.CoordinationV1().Leases(v1.NamespaceNodeLease).Get(ctx, node.Name, metav1.GetOptions{})
			if err == nil {
				if liveLease.ResourceVersion != nodeHealthLeaseRV {
					nc.consistencyStore.WroteAt(
						// This is a controller wide consistency check, if the lease cache is stale
						// we need to wait for the cache to catch up before processing any nodes.
						types.NamespacedName{},
						"", // No specific UID for generic name
						leaseResource,
						liveLease.ResourceVersion,
					)
					return 0, v1.NodeCondition{}, nil, shortCircuitError{&consistencyutil.ConsistencyError{
						ReadRV:        nodeHealthLeaseRV,
						WroteRV:       liveLease.ResourceVersion,
						GroupResource: leaseResource,
					}}
				}
			} else if !apierrors.IsNotFound(err) {
				return 0, v1.NodeCondition{}, nil, shortCircuitError{fmt.Errorf("error looking up lease to verify node %s: %w", node.Name, err)}
			}
		}

Read this carefully, because the semantics are subtle and deliberate.

The cached lease says “expired”. The controller now does a live GET of the same lease. Two outcomes:

  • The live lease’s resourceVersion equals the cached one. Then the cache is up to date, the lease really is expired, and the controller proceeds to mark the node Unknown and evict.
  • The live lease’s resourceVersion differs. Then the cache is behind, the “expired” verdict was based on stale data, and the controller bails out of the entire node evaluation — returning a shortCircuitError.

The same bail-out happens if the live GET fails with something other than NotFound. A NotFound is not a bail-out: if the lease is gone, then the node genuinely is not renewing it.

The short circuit

The error type is a marker, and its only job is to be recognisable:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:647
// shortCircuitError is a marker type to signal the polling loop in monitorNodeHealth should return early with error
type shortCircuitError struct {
	error
}

func (n shortCircuitError) Unwrap() error {
	return n.error
}

It is recognised in the retry loop inside monitorNodeHealth, and — crucially — it is not retried:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:705
		if err := wait.PollImmediate(retrySleepTime, retrySleepTime*scheduler.NodeHealthUpdateRetry, func() (bool, error) {
			var err error
			_, observedReadyCondition, currentReadyCondition, err = nc.tryUpdateNodeHealth(ctx, node)
			if err == nil {
				return true, nil
			}
			// If the error is due to a short circuit, don't retry.
			if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
				if err, ok := errors.AsType[shortCircuitError](err); ok {
					return false, err.error
				}
			}
			name := node.Name
			node, err = nc.kubeClient.CoreV1().Nodes().Get(ctx, name, metav1.GetOptions{})
			if err != nil {
				logger.Error(nil, "Failed while getting a Node to retry updating node health. Probably Node was deleted", "node", klog.KRef("", name))
				return false, err
			}
			return false, nil
		}); err != nil {
			logger.Error(err, "Update health of Node from Controller error, Skipping - no pods will be evicted", "node", klog.KObj(node))
			return
		}

The log message is the important part: “Skipping - no pods will be evicted”. A short circuit skips the node, it does not evict it.

But the effect is cluster-wide in the other direction, and that is the point of the design. If the lease cache is stale, it is stale for every node, not just this one. The controller therefore refuses to make eviction decisions for any node until the cache catches up. One stale lease, one aborted pass, zero false evictions.

The consistency store is what makes “until the cache catches up” precise rather than “until the next pass”. The controller records the resource version it saw live:

go 
// staging/src/k8s.io/client-go/util/consistency/consistency.go:176
// WroteAt increments the written resource version of an ownerRecord if it is
// the newest seen resource version for that resource.
func (w *ownerRecord) WroteAt(resource schema.GroupResource, rv string) {
	w.versionsLock.Lock()
	defer w.versionsLock.Unlock()
	if _, ok := w.versions[resource]; !ok {
		if _, err := resourceversion.CompareResourceVersion(rv, rv); err != nil {
			return
		}
		w.versions[resource] = rv
		return
	}
	cmp, err := resourceversion.CompareResourceVersion(w.versions[resource], rv)
	if err != nil || cmp >= 0 {
		return
	}
	w.versions[resource] = rv
}

and then, at the top of every node’s evaluation, it checks whether the informer store has caught up to that version:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:834
	if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
		// This is a controller-wide consistency check, if the lease cache is stale, then we cannot
		// trust any node leases until the cache has caught back up.
		err := nc.consistencyStore.EnsureReady(types.NamespacedName{})
		// Skip processing this node in this cycle if the node controller cache is not ready yet.
		if err != nil {
			return 0, v1.NodeCondition{}, nil, shortCircuitError{err}
		}
	}

The comparison is against the informer’s store sync resource version, not against a watch event:

go 
// staging/src/k8s.io/client-go/util/consistency/consistency.go:195
// EnsureReady checks whether or not the ownerRecord is ready compared to the
// read resource versions in the consistency store.
func (w *ownerRecord) EnsureReady(c *consistencyStore) error {
	w.versionsLock.Lock()
	defer w.versionsLock.Unlock()
	for gr, wroteRV := range w.versions {
		store, exists := c.stores[gr]
		if !exists || store == nil {
			return fmt.Errorf("no store registered for group resource %s", gr.String())
		}
		readRV := store.LastStoreSyncResourceVersion()
		if readRV == "" {
			// Store has not observed any sync or bookmark yet (e.g. cache not synced or
			// AtomicFIFO disabled). Fail in the safe direction by reporting not ready.
			return &ConsistencyError{
				WroteRV:       wroteRV,
				ReadRV:        readRV,
				GroupResource: gr,
			}
		}
		i, err := resourceversion.CompareResourceVersion(wroteRV, readRV)
		if err != nil {
			// comparison errors indicate there's a data problem with resource versions, continue so we don't block syncing
			continue
		}
		if i > 0 {
			// read version is not as new as owner version, not ready
			return &ConsistencyError{
				WroteRV:       wroteRV,
				ReadRV:        readRV,
				GroupResource: gr,
			}
		}
	}
	return nil
}

Two details worth pausing on. The readRV == "" case fails in the safe direction — if the store has never synced, the controller assumes it is not ready and refuses to act. And the comparison errors are swallowed (continue), because a malformed resource version is a data problem, not a reason to block the entire cluster’s node monitoring.

The store is only created when the feature gate is on; otherwise it is a no-op implementation:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:424
	if utilfeature.DefaultFeatureGate.Enabled(features.NodeControllerLeaseCircuitBreaker) {
		nc.consistencyStore = consistencyutil.NewConsistencyStore(map[schema.GroupResource]consistencyutil.LastSyncRVGetter{
			leaseResource: leaseInformer.Informer().GetStore(),
		})
	} else {
		nc.consistencyStore = consistencyutil.NewNoopConsistencyStore()
	}

And the gate itself:

go 
// pkg/features/kube_features.go:1770
	NodeControllerLeaseCircuitBreaker: {
		{Version: version.MustParse("1.37"), Default: true, PreRelease: featuregate.Beta},
	},

Beta and on by default as of 1.37. It is still a feature gate, so --feature-gates=NodeControllerLeaseCircuitBreaker=false turns the whole mechanism off — which is the first thing to check if you are debugging “the node controller evicted my node”.

Bounded concurrency, and why

The pass over all nodes is parallelised, with a fixed worker count:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:767
	// Marking the pods not ready on a node requires looping over them and
	// updating each pod's status one at a time. This is performed serially, and
	// can take a while if we're processing each node serially as well. So we
	// process them with bounded concurrency instead, since most of the time is
	// spent waiting on io.
	workqueue.ParallelizeUntil(ctx, nc.nodeUpdateWorkerSize, len(nodes), updateNodeFunc)

with nodeUpdateWorkerSize = 8 (pkg/controller/nodelifecycle/node_lifecycle_controller.go:138).

The comment explains the trade-off precisely: marking pods not-ready is a sequence of API calls that spend most of their time waiting on I/O, so serialising them wastes the parallelism, and unbounded parallelism would overload the API server. Eight is the compromise.

The eviction rate limiter

Even with the circuit breaker, the controller does not evict at full speed. Each zone gets its own rate-limited queue, and the rate limiter is hot swappable — the controller replaces it when the zone’s health changes.

go 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:196
// RateLimitedTimedQueue is a unique item priority queue ordered by
// the expected next time of execution. It is also rate limited.
type RateLimitedTimedQueue struct {
	queue       UniqueQueue
	limiterLock sync.Mutex
	limiter     flowcontrol.RateLimiter
}

The constants that set the pace:

go 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:29
const (
	// NodeHealthUpdateRetry controls the number of retries of writing
	// node health update.
	NodeHealthUpdateRetry = 5
	// NodeEvictionPeriod controls how often NodeController will try to
	// evict Pods from non-responsive Nodes.
	NodeEvictionPeriod = 100 * time.Millisecond
	// EvictionRateLimiterBurst is the burst value for all eviction rate
	// limiters
	EvictionRateLimiterBurst = 1
)

A burst of 1 means the token bucket refills at exactly the configured QPS with no allowance for a spike. One node per second, by default.

The queue itself pops items and asks the limiter for permission:

go 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:231
func (q *RateLimitedTimedQueue) Try(logger klog.Logger, fn ActionFunc) {
	val, ok := q.queue.Head()
	q.limiterLock.Lock()
	defer q.limiterLock.Unlock()
	for ok {
		// rate limit the queue checking
		if !q.limiter.TryAccept() {
			logger.V(10).Info("Try rate limited", "value", val)
			// Try again later
			break
		}

		now := now()
		if now.Before(val.ProcessAt) {
			break
		}

		if ok, wait := fn(val); !ok {
			val.ProcessAt = now.Add(wait + 1)
			q.queue.Replace(val)
		} else {
			q.queue.RemoveFromQueue(val.Value)
		}
		val, ok = q.queue.Head()
	}
}

Swapping the limiter without a burst

Here is the part that is genuinely subtle. When the controller changes a zone’s eviction rate — because the zone got healthier or sicker — it swaps the limiter. A naive swap would be a trap: a token bucket starts full, so the new limiter would immediately allow a burst of evictions at the new rate.

The fix is to drain the new bucket when the old one was saturated:

go 
// pkg/controller/nodelifecycle/scheduler/rate_limited_queue.go:282
// SwapLimiter safely swaps current limiter for this queue with the
// passed one if capacities or qps's differ.
func (q *RateLimitedTimedQueue) SwapLimiter(newQPS float32) {
	q.limiterLock.Lock()
	defer q.limiterLock.Unlock()
	if q.limiter.QPS() == newQPS {
		return
	}
	var newLimiter flowcontrol.RateLimiter
	if newQPS <= 0 {
		newLimiter = flowcontrol.NewFakeNeverRateLimiter()
	} else {
		newLimiter = flowcontrol.NewTokenBucketRateLimiter(newQPS, EvictionRateLimiterBurst)

		// If we're currently waiting on limiter, we drain the new one - this is a good approach when Burst value is 1
		// TODO: figure out if we need to support higher Burst values and decide on the drain logic, should we keep:
		// - saturation (percentage of used tokens)
		// - number of used tokens
		// - number of available tokens
		// - something else
		if q.limiter.TryAccept() == false {
			newLimiter.TryAccept()
		}
	}
	q.limiter.Stop()
	q.limiter = newLimiter
}

The logic: q.limiter.TryAccept() on the old limiter tells you whether the old limiter was saturated. If it was (returns false), the system was already unable to evict at the old rate, so the new limiter is drained with one TryAccept — consuming its single token. The new limiter therefore starts empty and the eviction rate ramps up from zero instead of bursting.

If the old limiter was not saturated, the new one is left full, which is correct: the system was keeping up before, so it can keep up now.

The newQPS <= 0 case installs a FakeNeverRateLimiter — a limiter that never accepts a token, i.e. evictions stop entirely. That is what the “master disruption mode” uses.

The zone state machine

The rate limiter is chosen by the zone state, and the state is computed from the per-node conditions:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:1067
	// At least one node was responding in previous pass or in the current pass. Semantics is as follows:
	// - if the new state is "partialDisruption" we call a user defined function that returns a new limiter to use,
	// - if the new state is "normal" we resume normal operation (go back to default limiter settings),
	// - if new state is "fullDisruption" we restore normal eviction rate,
	//   - unless all zones in the cluster are in "fullDisruption" - in that case we stop all evictions.
	if !allAreFullyDisrupted || !allWasFullyDisrupted {
		// We're switching to full disruption mode
		if allAreFullyDisrupted {
			logger.Info("Controller detected that all Nodes are not-Ready. Entering master disruption mode")
			for i := range nodes {
				_, err := nc.markNodeAsReachable(ctx, nodes[i])
				if err != nil {
					logger.Error(nil, "Failed to remove taints from Node", "node", klog.KObj(nodes[i]))
				}
			}
			// We stop all evictions.
			for k := range nc.zoneStates {
				nc.zoneNoExecuteTainter[k].SwapLimiter(0)
			}

SwapLimiter(0) — the “never” limiter. If every zone in the cluster is fully disrupted, the controller stops evicting entirely, because evicting everything at once is strictly worse than evicting nothing: the pods cannot go anywhere else.

And when the cluster recovers:

go 
// pkg/controller/nodelifecycle/node_lifecycle_controller.go:1092
		// We're exiting full disruption mode
		if allWasFullyDisrupted {
			logger.Info("Controller detected that some Nodes are Ready. Exiting master disruption mode")
			// When exiting disruption mode update probe timestamps on all Nodes.
			now := nc.now()
			for i := range nodes {
				v := nc.nodeHealthMap.getDeepCopy(nodes[i].Name)
				v.probeTimestamp = now
				v.readyTransitionTimestamp = now
				nc.nodeHealthMap.set(nodes[i].Name, v)
			}

The probe timestamps are reset so that nodes do not immediately look expired again on the way out of disruption mode.

Wrapping up

The one thing to remember: before acting on an expired lease from the informer cache, the node controller does a live GET and compares resourceVersions (pkg/controller/nodelifecycle/node_lifecycle_controller.go:954). A mismatch means the cache is stale, and the controller returns a shortCircuitError that aborts the evaluation of every node, not just the one it was looking at.

The part people get wrong: that the circuit breaker “skips one node”. It does not. The consistency store records the live resource version it saw and the next pass refuses to evaluate any node until the informer store has caught up to it (staging/src/k8s.io/client-go/util/consistency/consistency.go:197). The whole point is that a stale lease cache is stale for every node, and evicting a subset of nodes based on a stale cache is exactly the failure mode this design exists to prevent.

What to go do about it: if a node is being evicted unexpectedly, check whether the circuit breaker is even enabled — it is a beta feature gate NodeControllerLeaseCircuitBreaker, on by default since 1.37 (pkg/features/kube_features.go:1770), and --feature-gates=NodeControllerLeaseCircuitBreaker=false disables it entirely. Look for the “Update health of Node from Controller error, Skipping - no pods will be evicted” log line, which means the node was skipped rather than evicted. And if you are tuning responsiveness, the grace periods are --node-monitor-grace-period (default 50s) and --node-startup-grace-period (default 60s), the eviction rate is bounded by EvictionRateLimiterBurst = 1 and the --node-eviction-rate (0.1 nodes per second) / --secondary-node-eviction-rate (0.01) settings (cmd/kube-controller-manager/app/options/nodelifecyclecontroller.go:46), the zone is declared unhealthy at --unhealthy-zone-threshold 0.55 of its nodes (minimum 3), and the per-node work runs with 8 workers (pkg/controller/nodelifecycle/node_lifecycle_controller.go:138).

KubernetesInternalsControllers