I'm Behind: The Three-Second Grace Period Before the Watch Cache Gives Up

TL;DR: A client can ask the apiserver for a list at a resource version that the watch cache has not observed yet — usually because it just finished a write and wants to read its own write. The cacher waits on a sync.Cond for at most 3 seconds (blockTimeout). If the cache catches up, the request succeeds; if not, the client gets TooLargeResourceVersion with a Retry-After of 1 second. The trick inside the trick: the goroutine that wakes the condvar is only spawned when resourceVersion != 0, because resourceVersion=0 is by far the most common case and can never be satisfied — a goroutine per read request at 10k QPS would be pure waste.

Consistency in the apiserver is not free. The watch cache is a mirror of etcd maintained by a reflector; it is always slightly behind the etcd revision, and it converges asynchronously. A client that writes a Pod and then immediately lists Pods wants to see its own write. If the cacher happens to be a few hundred milliseconds behind, the request is not wrong — it is early.

There are two ways to handle an early request. Serve it from etcd, which is correct but expensive and defeats the whole point of the cache. Or wait a little, because the cacher will catch up in a moment. The cacher waits for exactly three seconds.

The two constants

Both live at the top of the watch cache, and they mean different things:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:44
const (
	// blockTimeout determines how long we're willing to block the request
	// to wait for a given resource version to be propagated to cache,
	// before terminating request and returning Timeout error with retry
	// after suggestion.
	blockTimeout = 3 * time.Second

	// resourceVersionTooHighRetrySeconds is the seconds before a operation should be retried by the client
	// after receiving a 'too high resource version' error.
	resourceVersionTooHighRetrySeconds = 1
)

blockTimeout is how long the server waits. resourceVersionTooHighRetrySeconds is how long the client is told to wait before retrying — one second, which is deliberately shorter than the server’s own timeout, because by then the cache will most likely have caught up.

The error the client receives is a Timeout with a Retry-After, dressed up with a specific cause so that it can be distinguished from other timeouts:

go
// staging/src/k8s.io/apiserver/pkg/storage/errors.go:244
var tooLargeResourceVersionCauseMsg = "Too large resource version"

// NewTooLargeResourceVersionError returns a timeout error with the given retrySeconds for a request for
// a minimum resource version that is larger than the largest currently available resource version for a requested resource.
func NewTooLargeResourceVersionError(minimumResourceVersion, currentRevision uint64, retrySeconds int) error {
	err := apierrors.NewTimeoutError(fmt.Sprintf("Too large resource version: %d, current: %d", minimumResourceVersion, currentRevision), retrySeconds)
	err.ErrStatus.Details.Causes = []metav1.StatusCause{
		{
			Type:    metav1.CauseTypeResourceVersionTooLarge,
			Message: tooLargeResourceVersionCauseMsg,
		},
	}
	return err
}

Note that the message includes both numbers: the resource version the client asked for, and the one the cache currently holds. That pair is enough to diagnose the whole situation — if the gap is small the cache is merely lagging; if the gap is enormous the client is asking for a resource version from a different universe.

The wait

Here is the entire mechanism:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:325
// waitUntilFreshLocked waits until cache is at least as fresh as given resourceVersion.
func (w *watchCache) waitUntilFreshLocked(ctx context.Context, resourceVersion uint64) error {
	if resourceVersion == 0 || resourceVersion <= w.resourceVersion {
		return nil
	}
	w.config.waitingUntilFresh.Add()
	defer w.config.waitingUntilFresh.Remove()
	startTime := w.config.clock.Now()
	defer func() {
		if resourceVersion > 0 {
			metrics.WatchCacheReadWait.WithContext(ctx).WithLabelValues(w.config.groupResource.Group, w.config.groupResource.Resource).Observe(w.config.clock.Since(startTime).Seconds())
		}
	}()

	// In case resourceVersion is 0, we accept arbitrarily stale result.
	// As a result, the condition in the below for loop will never be
	// satisfied (w.resourceVersion is never negative), this call will
	// never hit the w.cond.Wait().
	// As a result - we can optimize the code by not firing the wakeup
	// function (and avoid starting a gorotuine), especially given that
	// resourceVersion=0 is the most common case.
	if resourceVersion > 0 {
		go func() {
			// Wake us up when the time limit has expired.  The docs
			// promise that time.After (well, NewTimer, which it calls)
			// will wait *at least* the duration given. Since this go
			// routine starts sometime after we record the start time, and
			// it will wake up the loop below sometime after the broadcast,
			// we don't need to worry about waking it up before the time
			// has expired accidentally.
			<-w.config.clock.After(blockTimeout)
			w.cond.Broadcast()
		}()
	}

	for w.resourceVersion < resourceVersion {
		if w.config.clock.Since(startTime) >= blockTimeout {
			// Request that the client retry after 'resourceVersionTooHighRetrySeconds' seconds.
			return storage.NewTooLargeResourceVersionError(resourceVersion, w.resourceVersion, resourceVersionTooHighRetrySeconds)
		}
		w.cond.Wait()
	}
	return nil
}

Three things are happening here.

The fast path. If the requested resource version is zero, or the cache already holds it, the function returns immediately. resourceVersion == 0 means “any version will do” — the caller explicitly said it does not care about freshness.

The wakeup goroutine. sync.Cond.Wait() blocks until somebody calls Broadcast(). Normally that somebody is the reflector, when it advances the cache’s resource version. But if the cache never advances — because the requested version is genuinely in the future — the waiter would sleep forever. So a watchdog goroutine is spawned to broadcast after blockTimeout. It is a one-shot goroutine that sleeps and exits.

The loop. After waking, the waiter re-checks the condition. If the cache caught up, the loop exits and the request proceeds. If not, and if the deadline has passed, it returns the error. Note the timeout check happens before Wait(), so the total wait is at least blockTimeout but the error is raised as soon as the clock says so.

The optimisation that matters

The if resourceVersion > 0 guard around the goroutine spawn is the whole point, and the comment above it explains why in unusually blunt terms: “especially given that resourceVersion=0 is the most common case.”

Think about the numbers. An apiserver serving ten thousand reads per second, most of them unpinned (resourceVersion=0 or unset), would spawn ten thousand goroutines per second if the guard were not there. Each one sleeps for three seconds and then broadcasts on a condvar nobody is waiting on — pure waste, and enough to dominate the scheduler.

With the guard, the goroutine is spawned only for requests that will actually block, i.e. requests that pinned a resource version the cache does not have. Those are rare, and they are exactly the requests where the watchdog is load-bearing.

The defer around the metric reinforces the point: the wait histogram is only observed when resourceVersion > 0, because a zero-version request does not wait and recording it would pollute the distribution with zeros.

Who decides whether to wait at all

The waiting lives in the watch cache, but the decision to fall back to etcd lives one level up, in the delegator:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:129
func (c *CacheDelegator) GetList(ctx context.Context, key string, opts storage.ListOptions, listObj runtime.Object) error {
	_, _, err := storage.ValidateListOptions(c.cacher.resourcePrefix, c.cacher.versioner, opts)
	if err != nil {
		return err
	}
	result, err := delegator.ShouldDelegateList(opts, c.cacher)
	if err != nil {
		return err
	}
	if result.ShouldDelegate {
		return c.storage.GetList(ctx, key, opts, listObj)
	}

	if _, err := c.cacher.versioner.ParseResourceVersion(opts.ResourceVersion); err != nil {
		return err
	}

	if !c.cacher.Ready() && shouldDelegateListOnNotReadyCache(opts) {
		// If Cacher is not initialized, delegator List requests to storage
		// as described in https://kep.k8s.io/4568
		return c.storage.GetList(ctx, key, opts, listObj)
	}
	err = c.cacher.GetList(ctx, key, opts, listObj)
	success := "true"
	fallback := "false"
	if err != nil {
		if errors.IsResourceExpired(err) && utilfeature.DefaultFeatureGate.Enabled(features.ListFromCacheSnapshot) {
			return c.storage.GetList(ctx, key, opts, listObj)
		}
		if result.ConsistentRead {
			// IsTooLargeResourceVersion occurs when the requested RV is higher than cache's current RV
			// and cache hasn't caught up within the timeout period.
			if storage.IsTooLargeResourceVersion(err) {
				if utilfeature.DefaultFeatureGate.Enabled(features.ConsistentListFromCacheSkipTimeoutFallback) {
					fallback = "skipped"
					err = errors.NewTooManyRequests(err.Error(), resourceVersionTooHighRetrySeconds)
				} else {
					fallback = "true"
					err = c.storage.GetList(ctx, key, opts, listObj)
				}
			}
			if err != nil {
				success = "false"
			}
			metrics.ConsistentReadTotal.WithLabelValues(c.cacher.groupResource.Group, c.cacher.groupResource.Resource, success, fallback).Add(1)
		}
		return err
	}

The default behaviour when the cache gives up: go to etcd anyway. The client gets an answer, correct and fresh, at the cost of an etcd round trip. The three-second wait was an optimisation, not a policy.

The ConsistentListFromCacheSkipTimeoutFallback feature gate changes that policy. When it is on, the fallback is skipped and the client gets a 429 instead:

go
// staging/src/k8s.io/apiserver/pkg/features/kube_features.go:403
	ConsistentListFromCacheSkipTimeoutFallback: {
		{Version: version.MustParse("1.37"), Default: false, PreRelease: featuregate.Alpha},
	},

The reason is that a fallback to etcd is not free: it pins an etcd snapshot and consumes a connection from the apiserver’s etcd pool. Under a thundering herd of read-after-write clients, the fallback converts a cache miss into an etcd overload. Returning 429 with a one-second Retry-After lets the client back off and lets the cache catch up, instead of stampeding etcd.

The gate is alpha and off by default in 1.37. The metric that tells you which path was taken is apiserver_watch_cache_consistent_read_total, with labels group, resource, success and fallback — where fallback is true, false or skipped.

There is a second fallback for a different failure: if the requested resource version has been compacted away — it is older than the history the cache keeps — the error is ResourceExpired and, with ListFromCacheSnapshot enabled (beta since 1.34), the request is served from etcd:

go
// staging/src/k8s.io/apiserver/pkg/features/kube_features.go:450
	ListFromCacheSnapshot: {
		{Version: version.MustParse("1.33"), Default: false, PreRelease: featuregate.Alpha},
		{Version: version.MustParse("1.34"), Default: true, PreRelease: featuregate.Beta},
	},

The history buffer next door

The reason a request can be “too old” rather than “too new” is the watch cache’s history buffer: a circular buffer of recent events that lets the cache serve a watch from a resource version in the recent past. And that buffer is self-tuning.

The bounds are a hundred events at the bottom and a hundred thousand at the top:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_history.go:31
const (
	// defaultLowerBoundCapacity is a default value for event cache capacity's lower bound.
	// TODO: Figure out, to what value we can decreased it.
	defaultLowerBoundCapacity = 100

	// defaultUpperBoundCapacity should be able to keep the required history.
	defaultUpperBoundCapacity = 100 * 1024
)

and the resize logic doubles when the buffer is full but still fresh, and halves when a quarter of it has gone stale:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_history.go:120
// resizeCacheLocked resizes the cache if necessary:
// - increases capacity by 2x if cache is full and all cached events occurred within last eventFreshDuration.
// - decreases capacity by 2x when recent quarter of events occurred outside of eventFreshDuration(protect watchCache from flapping).
func (w *watchCacheHistory) resizeCacheLocked(eventTime time.Time) {
	if w.isCacheFullLocked() && eventTime.Sub(w.cache[w.startIndex%w.capacity].RecordTime) < w.eventFreshDuration {
		capacity := min(w.capacity*2, w.upperBoundCapacity)
		if capacity > w.capacity {
			w.doCacheResizeLocked(capacity)
		}
		return
	}
	if w.isCacheFullLocked() && eventTime.Sub(w.cache[(w.endIndex-w.capacity/4)%w.capacity].RecordTime) > w.eventFreshDuration {
		capacity := max(w.capacity/2, w.lowerBoundCapacity)
		if capacity < w.capacity {
			w.doCacheResizeLocked(capacity)
		}
		return
	}
}

The asymmetry is deliberate. Growing is conditioned on the oldest event still being fresh — if the buffer is full and the oldest entry is recent, the churn is real and the buffer is too small. Shrinking is conditioned on the quarter point being stale, not the oldest entry — that hysteresis is what stops the buffer from flapping up and down around the threshold on a workload with a bursty event rate.

The “fresh” window is itself derived from the bookmark frequency:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:63
	// DefaultEventFreshDuration is the default time duration of events
	// we want to keep.
	// We set it to defaultBookmarkFrequency plus epsilon to maximize
	// chances that last bookmark was sent within kept history, at the
	// same time, minimizing the needed memory usage.
	DefaultEventFreshDuration = defaultBookmarkFrequency + 15*time.Second

One minute (the bookmark frequency) plus fifteen seconds. The point is that the cache should be able to replay the last bookmark it sent — a client that resumes from a bookmark must be able to find it in the history.

If you raise --min-request-timeout or otherwise lengthen the watch window, the upper bound scales proportionally:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_history.go:55
// capacityUpperBound denotes the maximum possible capacity of the watch cache
// to which it can resize.
func capacityUpperBound(eventFreshDuration time.Duration) int {
	if eventFreshDuration <= DefaultEventFreshDuration {
		return defaultUpperBoundCapacity
	}
	// eventFreshDuration determines how long the watch events are supposed
	// to be stored in the watch cache.
	// In very high churn situations, there is a need to store more events
	// in the watch cache, hence it would have to be upsized accordingly.
	// Because of that, for larger values of eventFreshDuration, we set the
	// upper bound of the watch cache's capacity proportionally to the ratio
	// between eventFreshDuration and DefaultEventFreshDuration.
	// Given that the watch cache size can only double, we round up that
	// proportion to the next power of two.
	exponent := int(math.Ceil((math.Log2(eventFreshDuration.Seconds() / DefaultEventFreshDuration.Seconds()))))
	if maxExponent := int(math.Floor((math.Log2(math.MaxInt32 / defaultUpperBoundCapacity)))); exponent > maxExponent {
		// Making sure that the capacity's upper bound fits in a 32-bit integer.
		exponent = maxExponent
		klog.Warningf("Capping watch cache capacity upper bound to %v", defaultUpperBoundCapacity<<exponent)
	}
	return defaultUpperBoundCapacity << exponent
}

The ratio is rounded up to the next power of two, because the buffer can only double — an intermediate capacity like 150 000 could never be reached by doubling from 100.

The batch buffer

Serving a watch from history means walking the circular buffer under the watch cache lock. Doing that once per event would be brutal, so the interval is materialised into a small batch buffer first:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_interval.go:330
const bufferSize = 100

// watchCacheIntervalBuffer is used to reduce acquiring
// the lock on each invocation of historyCacheIntervalSource.next().
type watchCacheIntervalBuffer struct {
	// buffer is used to hold watchCacheEvents that
	// the interval returns on a call to Next().
	buffer []*watchCacheEvent
	// The first element of buffer is defined by startIndex,
	// its last element is defined by endIndex.
	startIndex int
	endIndex   int
}

One hundred events per lock acquisition. The buffer also nils out the entries it has handed over, so a long-lived watch does not pin the event objects in memory after they have aged out of the cache:

go
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_interval.go:344
// next returns the next event present in the interval buffer provided
// it is not empty.
func (wcib *watchCacheIntervalBuffer) next() (*watchCacheEvent, bool) {
	if wcib.isEmpty() {
		return nil, false
	}
	next := wcib.buffer[wcib.startIndex]
	// clean the unused event reference in the buffer. If this is not
	// done, event if the watch event is aged out from the watch
	// cache, it will not be GCed during the lifetime of the watcher
	// that holds a reference to the buffer.
	wcib.buffer[wcib.startIndex] = nil
	wcib.startIndex++
	return next, true
}

What to watch

Two apiserver metrics tell you whether this code path is hurting you. apiserver_watch_cache_read_wait_seconds is the histogram of how long requests waited for freshness — its buckets top out at three seconds (0.005, 0.025, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.25, 1.5, 2, 3), because three seconds is the wall. If you see a fat tail at the 3-second bucket, the cache is chronically behind and the requests are falling through to etcd. And apiserver_watch_cache_consistent_read_total with fallback="true" counts the requests that gave up waiting and went to etcd instead.

Wrapping up

The one thing to remember: the watch cache’s consistency contract is a three-second wall. Ask for a resource version it has not seen and it waits on a condvar for up to blockTimeout = 3s, then returns TooLargeResourceVersion with a one-second Retry-After. The wakeup goroutine is only spawned when resourceVersion > 0, because a zero resource version can never be satisfied and is the overwhelmingly common case.

The part people get wrong: the default behaviour after the timeout is not to fail. The delegator falls back to etcd and serves the request there, so the client sees a normal response. Enabling ConsistentListFromCacheSkipTimeoutFallback changes that into a 429 — it does not make the cache faster, it trades a slow answer for an honest “not yet” so that etcd is not stampeded.

What to go do about it: watch apiserver_watch_cache_read_wait_seconds — a p99 sitting at 3 seconds means the cacher is chronically behind and you are paying etcd round trips for every late read. Watch apiserver_watch_cache_consistent_read_total{fallback="true"} to see how often that fallback fires; if it is non-trivial, the etcd client pool is taking the hit. On the client side, handle Retry-After on the TooLargeResourceVersion timeout error rather than treating it as a failure — the whole point of the one-second retry hint is that the next attempt will succeed. And if you have a workload that reads its own writes, either pin the resource version explicitly (and accept the wait) or accept the eventual consistency instead.