Expectations: How a Controller Waits for Its Own Writes to Come Back

TL;DR: Before a controller writes to the API server it raises an expectation — “I expect N creates and M deletes to come back” — and the informer handler lowers it when the matching watch event arrives. SatisfiedExpectations() gates the sync loop: if the controller’s own writes are not yet visible in the cache, the sync does nothing, which is what stops double-creates and status churn. Deletes are tracked by UID, not by name, so a pod deleted and recreated with the same name is not confused with the old one, and a graceful deletion (an update that sets deletionTimestamp) is not double-counted. The whole thing expires after 5 minutes so a dropped watch event cannot wedge a controller forever.

The Kubernetes control loop has one structural weakness: the controller writes to the API server, and then reads the result back from an informer cache. Between the write returning and the watch event arriving, the cache is stale. If the controller acts on that stale view, it will do the wrong thing — typically create a second copy of a Pod it just created.

This is not a theoretical race. It is the normal case: the API server commits the write, returns 201, and the watch event has to travel through etcd, the apiserver’s watch cache, and the client’s reflector before the informer store is updated. That is tens of milliseconds at best, and the controller’s next sync may land right in the middle of it.

Expectations are the mechanism that closes the gap.

The contract

The design is documented in the source, tersely:

go
// pkg/controller/controller_utils.go:128
// Expectations are a way for controllers to tell the controller manager what they expect. eg:
//	ControllerExpectations: {
//		controller1: expects  2 adds in 2 minutes
//		controller2: expects  2 dels in 2 minutes
//		controller3: expects -1 adds in 2 minutes => controller3's expectations have already been met
//	}
//
// Implementation:
//	ControlleeExpectation = pair of atomic counters to track controllee's creation/deletion
//	ControllerExpectationsStore = TTLStore + a ControlleeExpectation per controller
//
// * Once set expectations can only be lowered
// * A controller isn't synced till its expectations are either fulfilled, or expire
// * Controllers that don't set expectations will get woken up for every matching controllee

Three properties, and each one is a design decision worth defending.

“Once set expectations can only be lowered.” The counters are monotone decreasing. A controller can raise them again by setting new expectations (a fresh SetExpectations), but a single expectation record only ever goes down. This makes the counters safe to update from multiple goroutines without a lock around the whole record.

“A controller isn’t synced till its expectations are either fulfilled, or expire.” The second half is the escape hatch.

“Controllers that don’t set expectations will get woken up for every matching controllee.” A controller that does not use expectations still works; it just gets a sync per event, which is what the pre-expectations controllers did.

The data structure and the timeout

Two int64 counters and a timestamp, per controller key:

go
// pkg/controller/controller_utils.go:272
// ControlleeExpectations track controllee creates/deletes.
type ControlleeExpectations struct {
	// Important: Since these two int64 fields are using sync/atomic, they have to be at the top of the struct due to a bug on 32-bit platforms
	// See: https://golang.org/pkg/sync/atomic/ for more information
	add       int64
	del       int64
	key       string
	timestamp time.Time
}

The comment about 32-bit platforms is the kind of thing that looks like cargo cult until you hit it: sync/atomic requires 64-bit alignment for 64-bit operations, and a struct whose first field is an int64 is guaranteed to be 8-byte aligned. Move the int64s down behind a string and the atomic operations break on 32-bit ARM.

The store is a cache.Store keyed by the controller key:

go
// pkg/controller/controller_utils.go:168
// ControllerExpectations is a cache mapping controllers to what they expect to see before being woken up for a sync.
type ControllerExpectations struct {
	cache.Store
}

and the check is one function:

go
// pkg/controller/controller_utils.go:195
func (r *ControllerExpectations) SatisfiedExpectations(logger klog.Logger, controllerKey string) bool {
	if exp, exists, err := r.GetExpectations(controllerKey); exists {
		if exp.Fulfilled() {
			logger.V(4).Info("Controller expectations fulfilled", "expectations", exp)
			return true
		} else if exp.isExpired() {
			logger.V(4).Info("Controller expectations expired", "expectations", exp)
			return true
		} else {
			logger.V(4).Info("Controller still waiting on expectations", "expectations", exp)
			return false
		}
	} else if err != nil {
		logger.V(2).Info("Error encountered while checking expectations, forcing sync", "err", err)
	} else {

Three outcomes, and the third one is the interesting one. If there is no expectation record for this controller — it has never created anything, or its expectations expired and were deleted — the function returns true, and the controller syncs. Expectations only ever delay a sync; they never block one permanently.

The expiry is a flat constant, and the comment above it explains the arithmetic behind it:

go
// pkg/controller/controller_utils.go:61
const (
	// If a watch drops a delete event for a pod, it'll take this long
	// before a dormant controller waiting for those packets is woken up anyway. It is
	// specifically targeted at the case where some problem prevents an update
	// of expectations, without it the controller could stay asleep forever. This should
	// be set based on the expected latency of watch events.
	//
	// Currently a controller can service (create *and* observe the watch events for said
	// creation) about 10 pods a second, so it takes about 1 min to service
	// 500 pods. Just creation is limited to 20qps, and watching happens with ~10-30s
	// latency/pod at the scale of 3000 pods over 100 nodes.
	ExpectationsTimeout = 5 * time.Minute

The reasoning is a bit generous: 500 pods at 10/second is a minute, and the watch latency at scale is 10–30 seconds per pod. Five minutes is a safety margin on top of that, because the failure mode being protected against is the controller being stuck, and a stuck controller is much worse than a late one.

The check itself is a wall-clock comparison, not a TTL store:

go
// pkg/controller/controller_utils.go:225
func (exp *ControlleeExpectations) isExpired() bool {
	return clock.RealClock{}.Since(exp.timestamp) > ExpectationsTimeout
}

Raising and lowering

Raising happens immediately before the write. In the ReplicaSet controller, it is a single line right before the batch of creates:

go
// pkg/controller/replicaset/replica_set.go:666
		// TODO: Track UIDs of creates just like deletes. The problem currently
		// is we'd need to wait on the result of a create to record the pod's
		// UID, which would require locking *across* the create, which will turn
		// into a performance bottleneck. We should generate a UID for the pod
		// beforehand and store it via ExpectCreations.
		rsc.expectations.ExpectCreations(logger, rsKey, diff)

Note the TODO. Creates are tracked by count; deletes are tracked by UID. The reason is spelled out in the comment: to record a created pod’s UID, you would have to hold a lock across the Create call, which serialises the controller. Counting is fine for creates because a create is idempotent from the controller’s point of view — it will be counted as one observation whenever the pod shows up.

Lowering happens in the informer handlers. On the create side:

go
// pkg/controller/replicaset/replica_set.go:487
		logger.V(4).Info("Pod created", "pod", klog.KObj(pod), "detail", pod)
		rsc.expectations.CreationObserved(logger, rsKey)
		rsc.queue.Add(rsKey)

and the counter manipulation is atomic:

go
// pkg/controller/controller_utils.go:244
// Decrements the expectation counts of the given controller.
func (r *ControllerExpectations) LowerExpectations(logger klog.Logger, controllerKey string, add, del int) {
	if exp, exists, err := r.GetExpectations(controllerKey); err == nil && exists {
		exp.Add(int64(-add), int64(-del))
		// The expectations might've been modified since the update on the previous line.
		logger.V(4).Info("Lowered expectations", "expectations", exp)
	}
}

exp.Add does two atomic adds, and Fulfilled does two atomic loads:

go
// pkg/controller/controller_utils.go:282
// Add increments the add and del counters.
func (e *ControlleeExpectations) Add(add, del int64) {
	atomic.AddInt64(&e.add, add)
	atomic.AddInt64(&e.del, del)
}

// Fulfilled returns true if this expectation has been fulfilled.
func (e *ControlleeExpectations) Fulfilled() bool {
	// TODO: think about why this line being atomic doesn't matter
	return atomic.LoadInt64(&e.add) <= 0 && atomic.LoadInt64(&e.del) <= 0
}

The counters can go negative, and that is deliberate: <= 0, not == 0. An extra observed creation (a pod created by someone else, or a duplicate watch event) drives the counter below zero, and the expectation stays satisfied. Trying to match == 0 would mean a controller that observed one too many events could never satisfy its expectations again until the timeout.

Where the gate is

The gate is one boolean in the sync path. In the ReplicaSet controller:

go
// pkg/controller/replicaset/replica_set.go:793
	rsNeedsSync := rsc.expectations.SatisfiedExpectations(logger, key)

and then, later, the actual reconciliation is conditional on it:

go
// pkg/controller/replicaset/replica_set.go:820
	var manageReplicasErr error
	var nextSyncDuration *time.Duration
	if rsNeedsSync && rs.DeletionTimestamp == nil {
		manageReplicasErr = rsc.manageReplicas(ctx, activePods, rs)
	}

Note what is not conditional: the status update still happens. The controller refuses to create and delete pods while its own writes are in flight, but it still recomputes and publishes status.replicas, status.readyReplicas and friends. That distinction matters — a controller that skips the status update would publish stale replica counts, and the Deployment controller above it would act on those stale counts.

So the gate is not “do nothing”. The gate is “do not mutate while your mutations are unobserved”.

Deletes are tracked by UID

Creates are counted. Deletes are tracked by UID, and the wrapper type that does it explains why:

go
// pkg/controller/controller_utils.go:334
// UIDTrackingControllerExpectations tracks the UID of the pods it deletes.
// This cache is needed over plain old expectations to safely handle graceful
// deletion. The desired behavior is to treat an update that sets the
// DeletionTimestamp on an object as a delete. To do so consistently, one needs
// to remember the expected deletes so they aren't double counted.
// TODO: Track creates as well (#22599)
type UIDTrackingControllerExpectations struct {
	ControllerExpectationsInterface
	// TODO: There is a much nicer way to do this that involves a single store,
	// a lock per entry, and a ControlleeExpectationsInterface type.
	uidStoreLock sync.Mutex
	// Store used for the UIDs associated with any expectation tracked via the
	// ControllerExpectationsInterface.
	uidStore cache.Store
}

The mechanism: when the controller deletes pods, it records the UIDs it deleted, not just the count:

go
// pkg/controller/controller_utils.go:360
// ExpectDeletions records expectations for the given deleteKeys, against the given controller.
func (u *UIDTrackingControllerExpectations) ExpectDeletions(logger klog.Logger, rcKey string, deletedKeys []string) error {
	expectedUIDs := sets.NewString()
	for _, k := range deletedKeys {
		expectedUIDs.Insert(k)
	}
	logger.V(4).Info("Controller waiting on deletions", "controller", rcKey, "keys", deletedKeys)
	u.uidStoreLock.Lock()
	defer u.uidStoreLock.Unlock()

	if existing := u.GetUIDs(rcKey); existing != nil && existing.Len() != 0 {
		logger.Error(nil, "Clobbering existing delete keys", "keys", existing)
	}
	if err := u.uidStore.Add(&UIDSet{expectedUIDs, rcKey}); err != nil {
		return err
	}
	return u.ControllerExpectationsInterface.ExpectDeletions(logger, rcKey, expectedUIDs.Len())
}

and the observation only counts if the UID is one it was actually waiting for:

go
// pkg/controller/controller_utils.go:379
// DeletionObserved records the given deleteKey as a deletion, for the given rc.
func (u *UIDTrackingControllerExpectations) DeletionObserved(logger klog.Logger, rcKey, deleteKey string) {
	u.uidStoreLock.Lock()
	defer u.uidStoreLock.Unlock()

	uids := u.GetUIDs(rcKey)
	if uids != nil && uids.Has(deleteKey) {
		logger.V(4).Info("Controller received delete for pod", "controller", rcKey, "key", deleteKey)
		u.ControllerExpectationsInterface.DeletionObserved(logger, rcKey)
		uids.Delete(deleteKey)
	}
}

The uids.Has(deleteKey) check is the whole point. Without it, a pod that was not one of the ones the controller deleted — a pod that happened to be deleted by the kubelet, or by a human — would lower the delete expectation and make the controller think its deletions had landed. Worse: a pod deleted and recreated with the same name but a different UID would be miscounted, because the key is the name and the UID is the identity.

Now the graceful deletion. A pod with a terminationGracePeriod is not removed from the store the instant the delete is issued: first the deletionTimestamp is set, and only later does the kubelet actually remove it. The controller treats the update as the delete:

go
// pkg/controller/replicaset/replica_set.go:519
	labelChanged := !reflect.DeepEqual(curPod.Labels, oldPod.Labels)
	if curPod.DeletionTimestamp != nil {
		// when a pod is deleted gracefully it's deletion timestamp is first modified to reflect a grace period,
		// and after such time has passed, the kubelet actually deletes it from the store. We receive an update
		// for modification of the deletion timestamp and expect an rs to create more replicas asap, not wait
		// until the kubelet actually deletes the pod. This is different from the Phase of a pod changing, because
		// an rs never initiates a phase change, and so is never asleep waiting for the same.
		rsc.deletePod(logger, curPod)
		if labelChanged {
			// we don't need to check the oldPod.DeletionTimestamp because DeletionTimestamp cannot be unset.
			rsc.deletePod(logger, oldPod)
		}
		return
	}

So the deletion is observed twice — once as an update that sets the timestamp, once as the final delete — and the UID tracking is what makes the second one a no-op: the UID was already removed from the expected set, so uids.Has(deleteKey) is false and the counter is not decremented again.

Adoption has the same problem

Expectations are about the controller’s own writes. Adoption is about someone else’s writes, and it has an analogous race: the controller lists orphaned pods from its cache, decides to adopt one, and issues a patch — but between the list and the patch, another controller may have claimed it.

The guard is a live re-read, with the original issue number preserved:

go
// pkg/controller/replicaset/replica_set.go:863
func (rsc *ReplicaSetController) claimPods(ctx context.Context, rs *apps.ReplicaSet, selector labels.Selector, filteredPods []*v1.Pod) ([]*v1.Pod, error) {
	// If any adoptions are attempted, we should first recheck for deletion with
	// an uncached quorum read sometime after listing Pods (see #42639).
	canAdoptFunc := controller.RecheckDeletionTimestamp(func(ctx context.Context) (metav1.Object, error) {
		fresh, err := rsc.kubeClient.AppsV1().ReplicaSets(rs.Namespace).Get(ctx, rs.Name, metav1.GetOptions{})
		if err != nil {
			return nil, err
		}
		if fresh.UID != rs.UID {
			return nil, fmt.Errorf("original %v %v/%v is gone: got uid %v, wanted %v", rsc.Kind, rs.Namespace, rs.Name, fresh.UID, rs.UID)
		}
		return fresh, nil
	})

“uncached quorum read” means a GET straight to the API server, bypassing the informer cache, to be sure the object being adopted is not already being deleted. The check itself is a one-liner:

go
// pkg/controller/controller_ref_manager.go:391
func RecheckDeletionTimestamp(getObject func(context.Context) (metav1.Object, error)) func(context.Context) error {
	return func(ctx context.Context) error {
		obj, err := getObject(ctx)
		if err != nil {
			return fmt.Errorf("can't recheck DeletionTimestamp: %v", err)
		}
		if obj.GetDeletionTimestamp() != nil {
			return fmt.Errorf("%v/%v has just been deleted at %v", obj.GetNamespace(), obj.GetName(), obj.GetDeletionTimestamp())
		}
		return nil
	}
}

And the whole adoption path is guarded by a sync.Once, because the check is expensive and its answer does not change within a sync:

go
// pkg/controller/controller_ref_manager.go:36
type BaseControllerRefManager struct {
	Controller metav1.Object
	Selector   labels.Selector

	canAdoptErr  error
	canAdoptOnce sync.Once
	CanAdoptFunc func(ctx context.Context) error
}

func (m *BaseControllerRefManager) CanAdopt(ctx context.Context) error {
	m.canAdoptOnce.Do(func() {
		if m.CanAdoptFunc != nil {
			m.canAdoptErr = m.CanAdoptFunc(ctx)
		}
	})
	return m.canAdoptErr
}

The ClaimObject logic that uses it is the standard triad — adopt orphans, release objects we own that no longer match, ignore objects owned by someone else:

go
// pkg/controller/controller_ref_manager.go:69
func (m *BaseControllerRefManager) ClaimObject(ctx context.Context, obj metav1.Object, match func(metav1.Object) bool, adopt, release func(context.Context, metav1.Object) error) (bool, error) {
	controllerRef := metav1.GetControllerOfNoCopy(obj)
	if controllerRef != nil {
		if controllerRef.UID != m.Controller.GetUID() {
			// Owned by someone else. Ignore.
			return false, nil
		}
		if match(obj) {
			// We already own it and the selector matches.
			// Return true (successfully claimed) before checking deletion timestamp.
			// We're still allowed to claim things we already own while being deleted
			// because doing so requires taking no actions.
			return true, nil
		}

Slow start, and what it costs

Expectations stop the controller from acting on a stale cache. They do not stop it from hammering the API server when it is being rate-limited or quota-limited. A ReplicaSet that wants 500 pods and has a quota of 10 will issue 500 creates, get 490 403s, and spam 490 events.

The mitigation is a slow-start batch:

go
// pkg/controller/controller_utils.go:73
	// When batching pod creates, SlowStartInitialBatchSize is the size of the
	// initial batch.  The size of each successive batch is twice the size of
	// the previous batch.  For example, for a value of 1, batch sizes would be
	// 1, 2, 4, 8, ...  and for a value of 10, batch sizes would be
	// 10, 20, 40, 80, ...  Setting the value higher means that quota denials
	// will result in more doomed API calls and associated event spam.  Setting
	// the value lower will result in more API call round trip periods for
	// large batches.
	//
	// Given a number of pods to start "N":
	// The number of doomed calls per sync once quota is exceeded is given by:
	//      min(N,SlowStartInitialBatchSize)
	// The number of batches is given by:
	//      1+floor(log_2(ceil(N/SlowStartInitialBatchSize)))
	SlowStartInitialBatchSize = 1

With the default of 1, the batch sizes are 1, 2, 4, 8, … and the number of doomed calls is bounded by the first batch, because the whole thing aborts on the first failure:

go
// pkg/controller/replicaset/replica_set.go:891
func slowStartBatch(count int, initialBatchSize int, fn func() error) (int, error) {
	remaining := count
	successes := 0
	for batchSize := min(remaining, initialBatchSize); batchSize > 0; batchSize = min(2*batchSize, remaining) {
		errCh := make(chan error, batchSize)
		var wg sync.WaitGroup
		wg.Add(batchSize)
		for i := 0; i < batchSize; i++ {
			go func() {
				defer wg.Done()
				if err := fn(); err != nil {
					errCh <- err
				}
			}()
		}
		wg.Wait()
		curSuccesses := batchSize - len(errCh)
		successes += curSuccesses
		if len(errCh) > 0 {
			return successes, <-errCh
		}
		remaining -= batchSize
	}
	return successes, nil
}

Each batch runs concurrently, the whole batch is awaited, and a single failure in a batch aborts every remaining batch. And the pods that were never attempted have their expectations walked back, so the controller does not sit waiting for creates it never made:

go
// pkg/controller/replicaset/replica_set.go:693
		// Any skipped pods that we never attempted to start shouldn't be expected.
		// The skipped pods will be retried later. The next controller resync will
		// retry the slow start process.
		if skippedPods := diff - successfulCreations; skippedPods > 0 {
			logger.V(2).Info("Slow-start failure. Skipping creation of pods, decrementing expectations", "podsSkipped", skippedPods, "kind", rsc.Kind, "replicaSet", klog.KObj(rs))
			for i := 0; i < skippedPods; i++ {
				// Decrement the expected number of creates because the informer won't observe this pod
				rsc.expectations.CreationObserved(logger, rsKey)

That last loop is a nice illustration of the whole design: expectations are a budget, and any code path that decides not to do a write must return the budget.

Expectations buy correctness with latency. A controller that has just created pods will not act again until those pods come back through the watch, which is tens of milliseconds in a small cluster and seconds in a large one. That is the correct trade: a duplicated pod costs more than a delayed reconcile.

They also cost a little complexity, and the complexity has a sharp edge: if you raise expectations and then fail to write, or write and fail to observe, the controller stalls for up to ExpectationsTimeout. Every write path must therefore be paired with either a matching observation or an explicit rollback of the expectation — the skippedPods loop above is that rollback.

Wrapping up

The one thing to remember: expectations are two int64 counters per controller key — adds and deletes — stored in a cache.Store, raised before a write and lowered by the informer handler when the write comes back. The gate is SatisfiedExpectations (pkg/controller/controller_utils.go:195), and it stops the controller from mutating while its own mutations are unobserved, but never stops it from publishing status.

The part people get wrong: that the counters must hit exactly zero. They do not: Fulfilled() is <= 0, so an extra observation drives the counter negative and the expectation stays satisfied. And deletes are tracked by UID, not by count or name — UIDTrackingControllerExpectations (controller_utils.go:340) exists so that a graceful deletion observed twice (once as the deletionTimestamp update, once as the final delete) is counted once, and so that a pod recreated under the same name is a different pod.

What to go do about it: if a controller seems stuck, look at ExpectationsTimeout (5 minutes, pkg/controller/controller_utils.go:72) and ask whether a watch event was dropped — the timeout is the only thing that recovers from that. If you are writing a controller, raise the expectation before the write, lower it in the informer handler, and make sure every early-return path between those two points walks the expectation back — the skippedPods loop in pkg/controller/replicaset/replica_set.go:693 is the canonical example. And if your controller is adoption-happy, take the RecheckDeletionTimestamp re-read from pkg/controller/replicaset/replica_set.go:866: it is a live GET, not a cache read, and it exists because of issue #42639.