The Reflector Refuses to Publish a Resource Version It Hasn't Earned

TL;DR: A watch started at resourceVersion: "" or "0" produces a burst of synthetic ADDED events that are not in resource-version order. If the reflector published the RV of that burst immediately, the next watch would resume from a point newer than some of the objects it just cached — and those objects would never be re-delivered. So the reflector keeps a flag, eventReceivedBesidesAdded, and refuses to publish the resource version until it has seen a MODIFIED, a DELETED, or a BOOKMARK. Only then does it know it is past the synthetic set.

Kubernetes is a system where every object carries a monotonically increasing metadata.resourceVersion, and the whole consistency model is built on being able to say “I have seen everything up to version N”. That claim is only true if the version you publish is one you actually earned.

There is exactly one place in client-go where that claim can go wrong, and it is subtle enough that it took a dedicated flag, a boolean threaded through a callback, and a comment to keep it from being “simplified” away later.

The bug this prevents

Imagine a reflector whose store holds three objects: a at resource version 10, b at version 20, c at version 30. The watch starts from resourceVersion: "" — meaning “give me everything, from whatever point you like” — and the server replies with synthetic ADDED events for the whole collection, in store order, not in resource-version order: c at rv=30, then a at rv=10, then b at rv=20.

If the reflector published resourceVersion=30 after seeing the first event, the next watch would resume from 30. Object a, last modified at version 10, would be in the cache with a stale body, and nothing would ever tell anyone. There is no future event for a — it did not change. The reflector would sit there, confidently serving a stale a forever.

The fix is to refuse to publish a version until you know you are past the synthetic burst. The synthetic events all have type ADDED. Real change events have type MODIFIED or DELETED, and a BOOKMARK is a synthetic event too but a different one — it is emitted after the initial set, in strict resource-version order. So the reflector counts “did I see anything that is not an ADDED?”, and only then publishes.

A reflector can start a watch in one of two states, and the difference is the whole bug:

1. From a known resource version (non-empty, not "0"). The server replays events from that exact point. Every event is a real change event, in order. The resource version of any event received is immediately safe to publish.
2. From "" or "0". The server replays the current contents of the collection as synthetic ADDED events, in whatever order the storage layer produced them. The resource version of those events is not a safe resume point.

The reflector encodes this as a single boolean, set before the watch is even opened. The flag starts true; the line below turns it off when the resume token is "" or "0":

// staging/src/k8s.io/client-go/tools/cache/reflector.go:599
			if options.ResourceVersion == "" || options.ResourceVersion == "0" {
				// if we're starting the watch at a resource version that will get synthetic ADDED events in non-rv order,
				// wait until we're through that set of events before propagating the RV
				propagateRVFromStart = false
			}

"" and "0" are both dangerous, and they are dangerous for the same reason. "" means “start from wherever”. "0" means “start from any old version you have” — the reflector uses it for its first list, because “for performance reasons, initial list performed by reflector uses ‘0’ as resource version to allow it to be served from the watch cache if it is enabled”:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:1126
	if r.lastSyncResourceVersion == "" {
		// For performance reasons, initial list performed by reflector uses "0" as resource version to allow it to
		// be served from the watch cache if it is enabled.
		return "0"
	}
	return r.lastSyncResourceVersion

The callback that does the withholding

The decision is not made once, at watch-open time. It is made per event, inside a callback that the event loop invokes. The callback receives two things: the resource version of the event, and whether anything other than an ADDED has been seen so far.

// staging/src/k8s.io/client-go/tools/cache/reflector.go:620
		err = handleWatch(ctx, start, w, r.store, r.expectedType, r.expectedGVK, r.name, r.typeDescription,
			func(rv string, eventReceivedBesidesAdded bool) {
				// We update the resource version in the store only if we have received at least one event that is
				// not an added event, or if the resource version has been set previously. This is because we can
				// encounter 2 scenarios:
				// 1. The watch is started from a resource version specified by the LastSyncResourceVersion field.
				//    In this case, we can update the resource version in the store without worrying about it being
				//    out of order since we will not receive any synthetic added events for resources that may be
				//    out of order.
				// 2. The watch is started when the LastSyncResourceVersion field is empty. In this case, we may not
				//    update the LastSyncResourceVersion until we receive at least one event that is not an added
				//    event, since that is the only way to ensure that the watch has exited the initial list phase.
				if propagateRVFromStart || eventReceivedBesidesAdded {
					r.setLastSyncResourceVersion(rv)
					if rvu, ok := r.store.(ResourceVersionUpdater); ok {
						rvu.UpdateResourceVersion(rv)
					}
				}
			},
			r.clock, resyncerrc)

Two conditions, ORed. propagateRVFromStart means “this watch started from a known-good point, so any event’s RV is fine”. eventReceivedBesidesAdded means “we have now seen a real event, so the synthetic burst is over and this event’s RV is fine”.

The two updates matter differently. setLastSyncResourceVersion is the reflector’s own resume token — what the next watch will start from. rvu.UpdateResourceVersion(rv) pushes it into the store, which is what cache.Store consumers read to know the cache’s freshness. Both are withheld together; publishing to one and not the other would give you a cache that claims a version the reflector does not actually hold.

Where eventReceivedBesidesAdded comes from

The event loop is shared by the watch-list and the ordinary watch path. The flag is set in exactly three branches — MODIFIED, DELETED and BOOKMARK — and the ADDED branch is conspicuously absent:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:1043
			case watch.Modified:
				eventReceivedBesidesAdded = true
				err := store.Update(event.Object)
				if err != nil {
					utilruntime.HandleErrorWithContext(ctx, err, "Unable to update watch event object to store", "reflector", name, "object", event.Object)
				}
			case watch.Deleted:
				// TODO: Will any consumers need access to the "last known
				// state", which is passed in event.Object? If so, may need
				// to change this.
				eventReceivedBesidesAdded = true
				err := store.Delete(event.Object)
				if err != nil {
					utilruntime.HandleErrorWithContext(ctx, err, "Unable to delete watch event object from store", "reflector", name, "object", event.Object)
				}
			case watch.Bookmark:
				// A `Bookmark` means watch has synced here, just update the resourceVersion
				eventReceivedBesidesAdded = true
				if meta.GetAnnotations()[metav1.InitialEventsAnnotationKey] == "true" {
					watchListBookmarkReceived = true
				}

A MODIFIED means “this object changed”, a DELETED means “this object changed (and is gone)”, and a BOOKMARK is emitted by the server strictly in resource-version order — which is exactly the guarantee we need to know we are past the unordered synthetic set.

One more subtlety: the resource version is read from the event object, not from the event envelope. Bookmarks carry their own resource version on the object, so the bookmark’s RV is the server’s current position, which is safe to publish.

And after every event, the loop unconditionally calls back with the current flag:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:1074
			// when eventReceivedBesidesAdded is true, that indicates we are definitely past any initial synthetic Added events
			setLastSyncResourceVersion(resourceVersion, eventReceivedBesidesAdded)
			eventCount++

Note what this does not do: it does not skip the callback when the flag is false. It calls the callback with eventReceivedBesidesAdded=false, and the callback decides. The decision logic lives in one place, next to the explanation of why it exists.

The watch-list variant. The watch-list path uses the same machinery but a different callback, because it needs the opposite behaviour. During the initial burst the resource version is meaningless (the objects are a snapshot, not a sequence), so it is only recorded once a non-ADDED event proves the burst is over:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:870
		watchListBookmarkReceived, err := handleListWatch(ctx, start, w, temporaryStore, r.expectedType, r.expectedGVK, r.name, r.typeDescription,
			func(rv string, eventReceivedBesidesAdded bool) {
				if eventReceivedBesidesAdded {
					resourceVersion = rv
				}
			},
			r.clock, make(chan error))

handleListWatch is handleAnyWatch with exitOnWatchListBookmarkReceived set to true; handleWatch is the same function with it set to false. The two paths share one event loop, one store-writing code path, one resource-version extraction. The only difference is whether the loop returns when the end-of-initial-events bookmark shows up.

A watch that dies instantly

If the watch channel closes in under a second with zero events, the reflector returns a distinctive error rather than treating it as a normal end-of-watch:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:1089
	watchDuration := clock.Since(start)
	if watchDuration < 1*time.Second && eventCount == 0 {
		return watchListBookmarkReceived, &VeryShortWatchError{Name: name}
	}

// staging/src/k8s.io/client-go/tools/cache/reflector.go:1294
// VeryShortWatchError is returned when the watch result channel is closed
// within one second, without having sent any events.
type VeryShortWatchError struct {
	// Name of the Reflector
	Name string
}

Why does this matter? Because a reflector that watches from "" and gets nothing has learned nothing about the state of the collection. It must not conclude “the cache is up to date at version X”. The VeryShortWatchError is a loud signal that the watch was useless, and it is what keeps the backoff machinery from treating a hard failure as a clean run.

The backoff arithmetic

The reflector does not spin when the apiserver is unhealthy. It has a delay function built from a standard exponential backoff with full jitter:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:317
	backoff := options.Backoff
	if backoff == nil {
		backoff = &wait.Backoff{
			Duration: defaultBackoffInit,
			Cap:      defaultBackoffMax,
			Steps:    int(math.Ceil(float64(defaultBackoffMax) / float64(defaultBackoffInit))),
			Factor:   defaultBackoffFactor,
			Jitter:   defaultBackoffJitter,
		}
	}

With defaultBackoffInit = 800ms, defaultBackoffMax = 30s, defaultBackoffFactor = 2.0 and defaultBackoffJitter = 1.0, the step count is ceil(30s / 800ms) = 38. The core algorithm is:

// staging/src/k8s.io/apimachinery/pkg/util/wait/backoff.go:106
	steps--

	// calculate the next step's interval
	if factor != 0 {
		next = time.Duration(float64(duration) * factor)
		if cap > 0 && next > cap {
			next = cap
			steps = 0
		}
	} else {
		next = duration
	}

	// add jitter for this step
	if jitter > 0 {
		duration = Jitter(duration, jitter)
	}

	return duration, next, steps

and the jitter is uniform over [duration, duration + jitter*duration) — wait.Jitter adds rand.Float64() * maxFactor * duration to the base duration, so with maxFactor = 1.0 the sleep is drawn uniformly from [d, 2d).

Walk the arithmetic. The nominal durations are 800ms, 1.6s, 3.2s, 6.4s, 12.8s, 25.6s, 30s (capped) — and each actual sleep is drawn uniformly from [d, 2d). The cap is hit at 51.2s > 30s, so next is clamped to 30s and steps is set to 0. From then on, steps < 1, so every sleep is Jitter(30s, 1.0) — uniform in [30s, 60s). That is the steady state the source comment describes:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:59
	// We used to make the call every 1sec (1 QPS), the goal here is to achieve ~98% traffic reduction when
	// API server is not healthy. With these parameters, backoff will stop at [30,60) sec interval which is
	// 0.22 QPS.

One request every 30 to 60 seconds is roughly 0.02 QPS, so the comment’s “0.22” is off by a factor of ten; the interval it names, not the decimal, is the part to trust. The 98% figure is the real target: 1 QPS down to about 0.02 QPS is a 98% reduction, and it is what protects an overloaded apiserver from being killed by its own clients.

The other half of the design is the reset. The backoff only resets if the apiserver has been healthy for a while:

// staging/src/k8s.io/apimachinery/pkg/util/wait/backoff.go:296
	switch {
	case b.resetInterval == 0:
		b.backoff = b.initialBackoff
	case b.clock.Now().Sub(b.lastStart) > b.resetInterval:
		b.backoff = b.initialBackoff
		b.lastStart = b.clock.Now()
	}
	return b.backoff.Step()
}

defaultBackoffReset = 2 * time.Minute, and the comment says why: “If we don’t backoff for 2min, assume API server is healthy and we reset the backoff.” Without this, a reflector that hit a ten-minute apiserver outage would sit at a 30-60s interval forever afterwards, even though the apiserver was fine.

The whole thing is driven from one place:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:426
	// Until runs the loop immediately (immediate=true) and resets the backoff timer after each
	// successful iteration (sliding=true). See backoff constants at top of file for generalized QPS targets (~0.22 QPS).
	_ = r.delayHandler.Until(ctx, true, true, func(ctx context.Context) (bool, error) {
		if err := r.ListAndWatchWithContext(ctx); err != nil {
			r.watchErrorHandler(ctx, r, err)
		}
		return false, nil
	})

immediate=true means the first attempt happens without delay. sliding=true means the backoff timer is reset after the function returns, so the delay is measured from the end of the previous attempt, not its start.

The timer that had to be stopped

There is one more piece of discipline in this file that has nothing to do with resource versions but everything to do with not leaking. The resync timer is created inside a helper that returns two values — the channel and a cleanup function:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:443
// resyncChan returns a channel which will receive something when a resync is
// required, and a cleanup function.
func (r *Reflector) resyncChan() (<-chan time.Time, func() bool) {
	if r.resyncPeriod == 0 {
		// nothing will ever be sent down this channel
		return nil, func() bool { return false }
	}
	// The cleanup function is required: imagine the scenario where watches
	// always fail so we end up listing frequently. Then, if we don't
	// manually stop the timer, we could end up with many timers active
	// concurrently.
	t := r.clock.NewTimer(r.resyncPeriod)
	return t.C(), t.Stop
}

Read that comment twice, because it describes a genuine bug shape. If watches keep failing, the reflector re-lists, and each list cycle creates a new resync timer. With a resync period of an hour and a failing apiserver, you accumulate one live timer per retry cycle.

So the caller has to stop the previous timer before making a new one:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:516
	resyncCh, cleanup := r.resyncChan()
	defer func() {
		cleanup() // Call the last one written into cleanup
	}()
	for {
		select {
		case <-resyncCh:
		case <-ctx.Done():
			return
		}
		if r.ShouldResync == nil || r.ShouldResync() {
			logger.V(4).Info("Forcing resync", "reflector", r.name)
			if err := r.store.Resync(); err != nil {
				resyncerrc <- err
				return
			}
		}
		cleanup()
		resyncCh, cleanup = r.resyncChan()
	}
}

cleanup() is called immediately before each new timer is created, and the deferred call handles the last one. Note also the resyncerrc channel: it has capacity 1 and is buffered precisely so that a resync error cannot block the resync goroutine forever.

The trade-off

What the withholding buys: a reflector that never resumes from a point it has not earned. The cache and the resume token always agree, and the classic “silently stale object” bug is structurally impossible.

What it costs:

• The first watch after a cold start publishes nothing. Until a real event arrives, LastSyncResourceVersion() stays where it was. For an informer whose collection never changes, the resume token never advances — correct, but it means a “resumed” reflector can sit at an old version.
• One extra field of state threaded through a callback. The eventReceivedBesidesAdded bool parameter exists only so the callback can make the publish/withhold decision in one place.
• The "" case is common, not exotic. Any reflector that starts without a resume token hits the unsafe path, and the safe behaviour is to hold.

Wrapping up

The one thing to remember: the reflector publishes a resource version only when it knows the synthetic ADDED burst is over. The flag is eventReceivedBesidesAdded, set by MODIFIED, DELETED or BOOKMARK in staging/src/k8s.io/client-go/tools/cache/reflector.go:1044, and consulted in the callback at reflector.go:632. Everything else in the file is bookkeeping around that one decision.

The part people get wrong: assuming resourceVersion means “the version of the last object I received”. For a watch that started at "" or "0", the last object received is not a resume point at all — the server ordered that burst by storage order, not by version. Publishing it is how you get an informer that is confidently wrong about one object, forever.

What to go do about it: if you maintain your own watch loop instead of using cache.NewReflector, do not set LastSyncResourceVersion from the first event you receive — wait for a MODIFIED, a DELETED, or a BOOKMARK, exactly as handleAnyWatch does. On the server side, watch the reflector metrics for ListAndWatch calls: the backoff parameters (defaultBackoffInit = 800ms, defaultBackoffMax = 30s, defaultBackoffReset = 2m) mean a reflector that cannot reach the apiserver should be retrying every 30-60 seconds, not every second. If you see a reflector retrying at 1 QPS against a broken apiserver, something outside the standard wait.Backoff path is driving it.