Apiserver on Opensource internals

Double the Page Size: How etcd LIST Escapes the Selector Death Spiral

Jay Hubbert — Thu, 22 Oct 2026 13:10:00 +0000

TL;DR: kubectl get pods -l app=nginx --limit=100 asks etcd for 100 objects, but etcd does not know what a label selector is. It returns 100 raw keys, the apiserver filters them, and if only 4 of the 100 matched, the response has 4 objects in it, not 100. The naive repair - fetch many more pages of 100 - is a death spiral, because a 1% match rate means 100 round trips. The etcd store instead doubles its page size on every iteration, capped at maxLimit = 10000, so the second page asks for 200 keys, the third for 400, and the selector collapses into a handful of large pages instead of hundreds of tiny ones. The continue token that carries the page boundary is base64-encoded JSON, and the decoder refuses any token whose start key is not already path.Clean-normalised, so a client cannot walk out of its own prefix.

A paginated LIST is a promise: “give me at most N objects, and give me a token that lets me fetch the next N”. The promise is easy when nothing filters anything. It is hard when the client asks for a label selector, because the filtering happens after the storage layer has already chosen which objects to hand back.

That mismatch is the whole problem. The etcd store’s paging loop is written in terms of keys in the range; the API’s paging loop is written in terms of objects that matched. When those two numbers diverge, a naive implementation pages forever.

The limit the client asked for

The page-size cap is a plain constant, separate from whatever limit the client asked for:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go:56
const (
	// maxLimit is a maximum page limit increase used when fetching objects from etcd.
	// This limit is used only for increasing page size by kube-apiserver. If request
	// specifies larger limit initially, it won't be changed.
	maxLimit = 10000
)

Read the comment carefully, because it encodes the whole policy: the cap only bounds increases. If a client asks for limit=50000, the apiserver honours 50000 and the cap never applies - it stops the doubling from running away, it does not truncate legitimate large requests. And there are two “limits” in play: the client’s opts.Predicate.Limit is the API-level contract (at most that many matching objects), while the store’s limit variable is the etcd page size (how many raw keys to ask for).

etcd does not know about your label selector

etcd stores opaque byte strings under opaque keys. It has no idea what app=nginx means, and it cannot apply a selector for you. So the store asks etcd for a page of raw keys and applies the predicate itself, in appendChunk:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go:1063
	for i, kv := range kvs {
		if paging && int64(v.Len()) >= pred.Limit {
			return lastKey, evaluated, true, nil
		}
		lastKey = kv.Key
		ok, err := s.processListItem(ctx, kv, pred, newItemFunc, aggregator, v)
		if err != nil {
			return lastKey, evaluated, false, err
		}
		if ok {
			evaluated++
		}
		// free kv early. Long lists can take O(seconds) to decode.
		kvs[i] = nil
	}
	return lastKey, evaluated, false, nil
}

Two things to notice. lastKey is advanced for every key processed, including the ones the selector rejected - the continue token therefore points at the last key examined, not the last key returned, which is exactly what you want for resumption. And limitReached is reported when the loop finds it already has enough matching objects, which is how the caller learns “there is more, and here is where to resume”. The Count field etcd returns is likewise a count of raw keys in the range, not of matching objects. The apiserver is explicit about the consequences - PrepareContinueToken (continue.go:100) refuses to report a remainingItemCount when a selector is in play, because the number would be a lie: etcd’s count includes every key it walked, matched or not. Only when the predicate is empty does it compute itemsCount - limit and hand it back to the client.

Doubling the page size

The consumer loop in GetList (store.go:825) pulls chunks until it has filled the client’s limit, or the range is exhausted. The conditions at the bottom of that loop are the ones that matter: it keeps going only while the response is under-full, and it stops the moment v.Len() >= opts.Predicate.Limit. The producer of those chunks is the page-size doubler:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go:893
			hasMore := int64(len(getResp.Kvs)) < getResp.Count
			// use the same resource version for subsequent requests
			if withRev == 0 {
				withRev = getResp.Revision
			}
			// capture the next continue key before yielding, the consumer nils out kvs as it decodes them
			if len(getResp.Kvs) > 0 {
				continueKey = string(getResp.Kvs[len(getResp.Kvs)-1].Key) + "\x00"
			}
			revision := withRev
			if !yield(listChunk{kvs: getResp.Kvs, revision: &revision, count: getResp.Count, hasMore: hasMore}, nil) {
				return
			}
			if !hasMore {
				return
			}
			if limit < maxLimit {
				// We got incomplete result due to field/label selector dropping the object.
				// Double page size to reduce total number of calls to etcd.
				limit *= 2
				if limit > maxLimit {
					limit = maxLimit
				}
			}
		}
	}
}

The arithmetic is the whole trick, and it is worth doing out loud. Start with limit=100 and a selector that matches 1% of objects. Page 1 asks etcd for 100 keys, gets 100 keys, 1 matches, the consumer is still under 100, so it wants more. The producer doubles: page 2 asks for 200 keys, 2 match. Page 3 asks for 400. Page 4 for 800. After ten pages the page size is 51 200, which the cap clamps to 10 000, and from then on every page returns 10 000 raw keys - roughly 100 matches at a 1% hit rate. The client fills its 100-object page in one or two more requests, not a hundred.

The doubling is geometric, so the number of round trips is O(log(needed_keys / limit)) rather than O(needed_keys / limit): a LIST that takes 100 etcd round trips takes 10 instead. Two details make this safe. The limit *= 2 happens after the yield, so it is only reached when the consumer explicitly asked for more - a consumer that is satisfied never triggers a second page at all. And the clamp is if limit < maxLimit, so once the page size reaches 10 000 it stops growing rather than growing without bound. Note also what the doubling does not change: the client’s requested limit. The response still contains at most opts.Predicate.Limit matching objects, because appendChunk stops as soon as it has that many.

The continue token

The token is the client’s handle on “where I got to”: a small structured object, JSON-encoded and base64’d.

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/continue.go:82
// EncodeContinue returns a string representing the encoded continuation of the current query.
func EncodeContinue(key, keyPrefix string, resourceVersion int64) (string, error) {
	nextKey := strings.TrimPrefix(key, keyPrefix)
	if nextKey == key {
		return "", fmt.Errorf("unable to encode next field: the key and key prefix do not match")
	}
	out, err := json.Marshal(&continueToken{APIVersion: "meta.k8s.io/v1", ResourceVersion: resourceVersion, StartKey: nextKey})
	if err != nil {
		return "", err
	}
	return base64.RawURLEncoding.EncodeToString(out), nil
}

The struct is fixed at meta.k8s.io/v1 with three fields - v, rv, start - and the version is baked in because a rolling upgrade cannot have half the fleet writing a format the other half cannot read. The start key is stored relative to the key prefix, so a token for /registry/pods/ does not leak the prefix into the payload.

The decoder is where the security-relevant code lives. A continue token is client-supplied input, and it names the key the next page starts at. If the client could put ../../registry/secrets/ in there, the next page would range over a different key space entirely - a cross-resource read:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/continue.go:56
	switch c.APIVersion {
	case "meta.k8s.io/v1":
		if c.ResourceVersion == 0 {
			return "", 0, ErrInvalidStartRV
		}
		if len(c.StartKey) == 0 {
			return "", 0, ErrEmptyStartKey
		}
		// defend against path traversal attacks by clients - path.Clean will ensure that startKey cannot
		// be at a higher level of the hierarchy, and so when we append the key prefix we will end up with
		// continue start key that is fully qualified and cannot range over anything less specific than
		// keyPrefix.
		key := c.StartKey
		if !strings.HasPrefix(key, "/") {
			key = "/" + key
		}
		cleaned := path.Clean(key)
		if cleaned != key {
			return "", 0, fmt.Errorf("%w: %v", ErrGenericInvalidKey, c.StartKey)
		}
		return keyPrefix + cleaned[1:], c.ResourceVersion, nil
	default:
		return "", 0, fmt.Errorf("%w %v", ErrUnrecognizedEncodedVersion, c.APIVersion)
	}
}

The defence is elegant: normalise the key with path.Clean, then reject the token if cleaning changed it. A key that is already clean - /pods/foo

passes. A key with a traversal - /pods/../../secrets - cleans to /secrets, which differs from the input, so the token is rejected outright. The apiserver never repairs a hostile token; it refuses it. The other two guards are equally blunt: a token with rv == 0 is invalid (ErrInvalidStartRV), because a continue token must pin a resourceVersion
that is how the pages stay consistent - and an empty start key is invalid (ErrEmptyStartKey). An unrecognised v is rejected too, which is what makes a future meta.k8s.io/v2 possible without breaking old clients. And ValidateListOptions (interfaces.go:372) rejects a request that carries both a continue token and an explicit resourceVersion, because they could disagree.

The stream path, and the graceful fallback

The paginated path above is the fallback. etcd 3.7 added a RangeStream RPC, and the apiserver prefers it when it can:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go:747
// shouldStream determines whether a list request should use etcd RangeStream.
func shouldStream(opts storage.ListOptions) bool {
	// Only recursive lists stream, a non-recursive request reads a single object.
	if !opts.Recursive {
		return false
	}
	// etcd is unaware of selector matches, so a filtered page can't carry a proper limit.
	// Stream only unfiltered pages.
	if opts.Predicate.Limit > 0 && !opts.Predicate.Empty() {
		return false
	}
	return utilfeature.DefaultFeatureGate.Enabled(features.EtcdRangeStream) &&
		etcdfeature.DefaultFeatureSupportChecker.Supports(storage.RangeStream)
}

Note the second condition, and the reason given: a filtered page cannot carry a proper limit, so the stream path is only used for unfiltered pages. The stream RPC does not understand selectors either, and unlike the paginated path it has no doubling mechanism, so a filtered stream would hit the same wall with fewer options.

The interesting part is what happens when etcd is too old to have RangeStream. The store probes with one call and interprets Unimplemented as “not supported”:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go:951
	stream, streamErr := s.client.KV.GetStream(ctx, startKey, streamOpts...)
	var first clientv3.RangeStreamResponse
	var firstOk bool
	if streamErr == nil {
		first, firstOk = <-stream
		if firstOk && grpcstatus.Code(first.Err()) == grpccodes.Unimplemented {
			return nil, false
		}
	} else if grpcstatus.Code(streamErr) == grpccodes.Unimplemented {
		return nil, false
	}

Unimplemented is a normal gRPC status, not an error condition - it means “this server does not have that RPC”. The store treats it as a capability probe, falls back to the paginated path, and records the fact so the next request does not probe again. The checker’s default for storage.RangeStream is true - optimistic - and MarkUnsupported flips it to false for rangeStreamRecheckInterval (10 minutes), after which the apiserver tries again. That recheck interval is what lets a cluster survive an etcd upgrade: the apiserver keeps working against the paginated path while it waits, and picks the stream path back up when the new etcd is in place. The watch side does the same thing, with the same fallback:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:309
func (wc *watchChan) sync() error {
	if wc.recursive && utilfeature.DefaultFeatureGate.Enabled(features.EtcdRangeStream) &&
		etcdfeature.DefaultFeatureSupportChecker.Supports(storage.RangeStream) {
		err := wc.syncStreamRecursive()
		if err == nil {
			return nil
		}
		if grpcstatus.Code(err) != grpccodes.Unimplemented {
			return err
		}
		etcdfeature.DefaultFeatureSupportChecker.MarkUnsupported(storage.RangeStream)
		klog.V(4).Infof("etcd server does not support RangeStream for %v; falling back to paginated list", wc.watcher.groupResource)
	}
	return wc.syncPaginated()
}

The paginated watch sync is worth contrasting with the LIST path, because it does not double. syncPaginated (watcher.go:325) asks for defaultWatcherMaxLimit - which is maxLimit, 10 000 - raw keys per page and keeps the page size fixed for the whole sync. No doubling, because a watch sync has no client-supplied limit to fill: it wants all the objects, so a bigger page is simply better and there is no “the page came back too small” signal to react to.

The cacher ignores limit for resourceVersion=0

Most LIST traffic never reaches the etcd store at all. The watch cache serves it from memory, and for an unpinned list the cacher deliberately ignores the limit:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:749
// computeListLimit determines whether the cacher should
// apply a limit to an incoming LIST request and returns its value.
//
// note that this function doesn't check RVM nor the Continuation token.
// these parameters are validated by the shouldDelegateList function.
//
// as of today, the limit is ignored for requests that set RV == 0
func computeListLimit(opts storage.ListOptions) int64 {
	if opts.Predicate.Limit <= 0 || opts.ResourceVersion == "0" {
		return 0
	}
	return opts.Predicate.Limit
}

The reason is consistency, not laziness. A list at resourceVersion=0 means “whatever is current right now” - the cache is free to be a moving target, and the result is explicitly not a consistent snapshot. But a paginated list is a different contract: the pages together form one snapshot, and the client must be able to resume mid-way and see the same world. If the cacher applied a limit to an unpinned list, the boundary between page 1 and page 2 would depend on whatever happened to be in the cache between the two requests, and the pages could not be joined into a consistent set. So the cacher refuses to paginate an unpinned list. The delegator draws the same line: a request with a limit and a specific, non-zero resourceVersion goes to etcd (ShouldDelegateList, delegator/interface.go:51), while resourceVersion=0 with no continue token is served from the cache with the limit dropped.

The trade-offs

What it buys. A filtered LIST costs O(log) round trips to etcd instead of O(matching_ratio_inverse). At a 1% match rate, filling a 100-object page goes from roughly 100 etcd calls to about 10; at 0.1%, from 1 000 to about 13. Larger pages also mean more sequential etcd reads, which is what etcd’s own storage layer wants.

What it costs.

Memory. A 10 000-key page is 10 000 decoded objects in flight. The apiserver holds them all before it can filter them down.
Over-fetch. The client asked for 100 objects and the apiserver read 10 000 keys to find them. The trade is explicit: read 100x the data once, or make 100 round trips.
The cap is a cap. maxLimit = 10000 is not configurable. A selector so selective that even 10 000 raw keys rarely contain a match will still crawl, and there is no flag to raise the ceiling.
The stream path gives up on filters entirely. shouldStream refuses to stream a filtered page, so the doubling trick only exists on the paginated path. On etcd 3.7+ an unfiltered LIST takes the stream path and never touches the doubling at all.
The continue token is a security boundary. Clients treat it as opaque; it is not, and the path.Clean check in DecodeContinue is the only thing between a client and a cross-prefix read.

Wrapping up

The one thing to remember: the etcd store’s page size is not the client’s limit. The store starts at the client’s limit and doubles it on every iteration, capped at maxLimit = 10000 (staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go:909), because etcd applies no selector and a filtered page comes back under-full. The response still honours the client’s limit - appendChunk stops as soon as it has enough - so the doubling only changes how many raw keys are read.

The part people get wrong: “the apiserver returns at most limit objects, so it must have asked etcd for limit objects”. It did not. It asked for limit, then 2*limit, then 4*limit, up to 10 000 keys per page, and threw away everything that did not match. And the reverse trap: the remainingItemCount field is nil whenever a selector is present, because etcd’s Count counts raw keys, and reporting it as a count of remaining objects would be a lie.

What to go do about it: measure the gap with apiserver_storage_list_fetched_objects_total versus apiserver_storage_list_returned_objects_total (and the apiserver_storage_list_evaluated_objects_total in between) - a large fetched/returned ratio is exactly the selector death spiral, and it is the number that tells you the doubling is doing work. Per-request cost is in apiserver_etcd_request_duration_seconds with the operation label (list versus listStream - the latter means the RangeStream path is live). To see whether the stream path is available, look for the etcd server does not support RangeStream message at -v=4; the EtcdRangeStream feature gate is Beta and on by default, but the storage.RangeStream capability defaults to true and is only downgraded for 10 minutes after an Unimplemented response. And remember that the cacher ignores limit for resourceVersion=0 lists (cacher.go:756) - if you want a paginated, consistent list you must send a non-zero resourceVersion, and you must treat the returned continue token as opaque.

I'm Behind: The Three-Second Grace Period Before the Watch Cache Gives Up

Jay Hubbert — Thu, 22 Oct 2026 12:40:00 +0000

TL;DR: A client can ask the apiserver for a list at a resource version that the watch cache has not observed yet — usually because it just finished a write and wants to read its own write. The cacher waits on a sync.Cond for at most 3 seconds (blockTimeout). If the cache catches up, the request succeeds; if not, the client gets TooLargeResourceVersion with a Retry-After of 1 second. The trick inside the trick: the goroutine that wakes the condvar is only spawned when resourceVersion != 0, because resourceVersion=0 is by far the most common case and can never be satisfied — a goroutine per read request at 10k QPS would be pure waste.

Consistency in the apiserver is not free. The watch cache is a mirror of etcd maintained by a reflector; it is always slightly behind the etcd revision, and it converges asynchronously. A client that writes a Pod and then immediately lists Pods wants to see its own write. If the cacher happens to be a few hundred milliseconds behind, the request is not wrong — it is early.

There are two ways to handle an early request. Serve it from etcd, which is correct but expensive and defeats the whole point of the cache. Or wait a little, because the cacher will catch up in a moment. The cacher waits for exactly three seconds.

The two constants

Both live at the top of the watch cache, and they mean different things:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:44
const (
	// blockTimeout determines how long we're willing to block the request
	// to wait for a given resource version to be propagated to cache,
	// before terminating request and returning Timeout error with retry
	// after suggestion.
	blockTimeout = 3 * time.Second

	// resourceVersionTooHighRetrySeconds is the seconds before a operation should be retried by the client
	// after receiving a 'too high resource version' error.
	resourceVersionTooHighRetrySeconds = 1
)

blockTimeout is how long the server waits. resourceVersionTooHighRetrySeconds is how long the client is told to wait before retrying — one second, which is deliberately shorter than the server’s own timeout, because by then the cache will most likely have caught up.

The error the client receives is a Timeout with a Retry-After, dressed up with a specific cause so that it can be distinguished from other timeouts:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/errors.go:244
var tooLargeResourceVersionCauseMsg = "Too large resource version"

// NewTooLargeResourceVersionError returns a timeout error with the given retrySeconds for a request for
// a minimum resource version that is larger than the largest currently available resource version for a requested resource.
func NewTooLargeResourceVersionError(minimumResourceVersion, currentRevision uint64, retrySeconds int) error {
	err := apierrors.NewTimeoutError(fmt.Sprintf("Too large resource version: %d, current: %d", minimumResourceVersion, currentRevision), retrySeconds)
	err.ErrStatus.Details.Causes = []metav1.StatusCause{
		{
			Type: metav1.CauseTypeResourceVersionTooLarge,
			Message: tooLargeResourceVersionCauseMsg,
		},
	}
	return err
}

Note that the message includes both numbers: the resource version the client asked for, and the one the cache currently holds. That pair is enough to diagnose the whole situation — if the gap is small the cache is merely lagging; if the gap is enormous the client is asking for a resource version from a different universe.

The wait

Here is the entire mechanism:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:325
// waitUntilFreshLocked waits until cache is at least as fresh as given resourceVersion.
func (w *watchCache) waitUntilFreshLocked(ctx context.Context, resourceVersion uint64) error {
	if resourceVersion == 0 || resourceVersion <= w.resourceVersion {
		return nil
	}
	w.config.waitingUntilFresh.Add()
	defer w.config.waitingUntilFresh.Remove()
	startTime := w.config.clock.Now()
	defer func() {
		if resourceVersion > 0 {
			metrics.WatchCacheReadWait.WithContext(ctx).WithLabelValues(w.config.groupResource.Group, w.config.groupResource.Resource).Observe(w.config.clock.Since(startTime).Seconds())
		}
	}()

	// In case resourceVersion is 0, we accept arbitrarily stale result.
	// As a result, the condition in the below for loop will never be
	// satisfied (w.resourceVersion is never negative), this call will
	// never hit the w.cond.Wait().
	// As a result - we can optimize the code by not firing the wakeup
	// function (and avoid starting a gorotuine), especially given that
	// resourceVersion=0 is the most common case.
	if resourceVersion > 0 {
		go func() {
			// Wake us up when the time limit has expired. The docs
			// promise that time.After (well, NewTimer, which it calls)
			// will wait *at least* the duration given. Since this go
			// routine starts sometime after we record the start time, and
			// it will wake up the loop below sometime after the broadcast,
			// we don't need to worry about waking it up before the time
			// has expired accidentally.
			<-w.config.clock.After(blockTimeout)
			w.cond.Broadcast()
		}()
	}

	for w.resourceVersion < resourceVersion {
		if w.config.clock.Since(startTime) >= blockTimeout {
			// Request that the client retry after 'resourceVersionTooHighRetrySeconds' seconds.
			return storage.NewTooLargeResourceVersionError(resourceVersion, w.resourceVersion, resourceVersionTooHighRetrySeconds)
		}
		w.cond.Wait()
	}
	return nil
}

Three things are happening here.

The fast path. If the requested resource version is zero, or the cache already holds it, the function returns immediately. resourceVersion == 0 means “any version will do” — the caller explicitly said it does not care about freshness.

The wakeup goroutine. sync.Cond.Wait() blocks until somebody calls Broadcast(). Normally that somebody is the reflector, when it advances the cache’s resource version. But if the cache never advances — because the requested version is genuinely in the future — the waiter would sleep forever. So a watchdog goroutine is spawned to broadcast after blockTimeout. It is a one-shot goroutine that sleeps and exits.

The loop. After waking, the waiter re-checks the condition. If the cache caught up, the loop exits and the request proceeds. If not, and if the deadline has passed, it returns the error. Note the timeout check happens before Wait(), so the total wait is at least blockTimeout but the error is raised as soon as the clock says so.

The optimisation that matters

The if resourceVersion > 0 guard around the goroutine spawn is the whole point, and the comment above it explains why in unusually blunt terms: “especially given that resourceVersion=0 is the most common case.”

Think about the numbers. An apiserver serving ten thousand reads per second, most of them unpinned (resourceVersion=0 or unset), would spawn ten thousand goroutines per second if the guard were not there. Each one sleeps for three seconds and then broadcasts on a condvar nobody is waiting on — pure waste, and enough to dominate the scheduler.

With the guard, the goroutine is spawned only for requests that will actually block, i.e. requests that pinned a resource version the cache does not have. Those are rare, and they are exactly the requests where the watchdog is load-bearing.

The defer around the metric reinforces the point: the wait histogram is only observed when resourceVersion > 0, because a zero-version request does not wait and recording it would pollute the distribution with zeros.

Who decides whether to wait at all

The waiting lives in the watch cache, but the decision to fall back to etcd lives one level up, in the delegator:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:129
func (c *CacheDelegator) GetList(ctx context.Context, key string, opts storage.ListOptions, listObj runtime.Object) error {
	_, _, err := storage.ValidateListOptions(c.cacher.resourcePrefix, c.cacher.versioner, opts)
	if err != nil {
		return err
	}
	result, err := delegator.ShouldDelegateList(opts, c.cacher)
	if err != nil {
		return err
	}
	if result.ShouldDelegate {
		return c.storage.GetList(ctx, key, opts, listObj)
	}

	if _, err := c.cacher.versioner.ParseResourceVersion(opts.ResourceVersion); err != nil {
		return err
	}

	if !c.cacher.Ready() && shouldDelegateListOnNotReadyCache(opts) {
		// If Cacher is not initialized, delegator List requests to storage
		// as described in https://kep.k8s.io/4568
		return c.storage.GetList(ctx, key, opts, listObj)
	}
	err = c.cacher.GetList(ctx, key, opts, listObj)
	success := "true"
	fallback := "false"
	if err != nil {
		if errors.IsResourceExpired(err) && utilfeature.DefaultFeatureGate.Enabled(features.ListFromCacheSnapshot) {
			return c.storage.GetList(ctx, key, opts, listObj)
		}
		if result.ConsistentRead {
			// IsTooLargeResourceVersion occurs when the requested RV is higher than cache's current RV
			// and cache hasn't caught up within the timeout period.
			if storage.IsTooLargeResourceVersion(err) {
				if utilfeature.DefaultFeatureGate.Enabled(features.ConsistentListFromCacheSkipTimeoutFallback) {
					fallback = "skipped"
					err = errors.NewTooManyRequests(err.Error(), resourceVersionTooHighRetrySeconds)
				} else {
					fallback = "true"
					err = c.storage.GetList(ctx, key, opts, listObj)
				}
			}
			if err != nil {
				success = "false"
			}
			metrics.ConsistentReadTotal.WithLabelValues(c.cacher.groupResource.Group, c.cacher.groupResource.Resource, success, fallback).Add(1)
		}
		return err
	}

The default behaviour when the cache gives up: go to etcd anyway. The client gets an answer, correct and fresh, at the cost of an etcd round trip. The three-second wait was an optimisation, not a policy.

The ConsistentListFromCacheSkipTimeoutFallback feature gate changes that policy. When it is on, the fallback is skipped and the client gets a 429 instead:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/features/kube_features.go:403
	ConsistentListFromCacheSkipTimeoutFallback: {
		{Version: version.MustParse("1.37"), Default: false, PreRelease: featuregate.Alpha},
	},

The reason is that a fallback to etcd is not free: it pins an etcd snapshot and consumes a connection from the apiserver’s etcd pool. Under a thundering herd of read-after-write clients, the fallback converts a cache miss into an etcd overload. Returning 429 with a one-second Retry-After lets the client back off and lets the cache catch up, instead of stampeding etcd.

The gate is alpha and off by default in 1.37. The metric that tells you which path was taken is apiserver_watch_cache_consistent_read_total, with labels group, resource, success and fallback — where fallback is true, false or skipped.

There is a second fallback for a different failure: if the requested resource version has been compacted away — it is older than the history the cache keeps — the error is ResourceExpired and, with ListFromCacheSnapshot enabled (beta since 1.34), the request is served from etcd:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/features/kube_features.go:450
	ListFromCacheSnapshot: {
		{Version: version.MustParse("1.33"), Default: false, PreRelease: featuregate.Alpha},
		{Version: version.MustParse("1.34"), Default: true, PreRelease: featuregate.Beta},
	},

The history buffer next door

The reason a request can be “too old” rather than “too new” is the watch cache’s history buffer: a circular buffer of recent events that lets the cache serve a watch from a resource version in the recent past. And that buffer is self-tuning.

The bounds are a hundred events at the bottom and a hundred thousand at the top:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_history.go:31
const (
	// defaultLowerBoundCapacity is a default value for event cache capacity's lower bound.
	// TODO: Figure out, to what value we can decreased it.
	defaultLowerBoundCapacity = 100

	// defaultUpperBoundCapacity should be able to keep the required history.
	defaultUpperBoundCapacity = 100 * 1024
)

and the resize logic doubles when the buffer is full but still fresh, and halves when a quarter of it has gone stale:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_history.go:120
// resizeCacheLocked resizes the cache if necessary:
// - increases capacity by 2x if cache is full and all cached events occurred within last eventFreshDuration.
// - decreases capacity by 2x when recent quarter of events occurred outside of eventFreshDuration(protect watchCache from flapping).
func (w *watchCacheHistory) resizeCacheLocked(eventTime time.Time) {
	if w.isCacheFullLocked() && eventTime.Sub(w.cache[w.startIndex%w.capacity].RecordTime) < w.eventFreshDuration {
		capacity := min(w.capacity*2, w.upperBoundCapacity)
		if capacity > w.capacity {
			w.doCacheResizeLocked(capacity)
		}
		return
	}
	if w.isCacheFullLocked() && eventTime.Sub(w.cache[(w.endIndex-w.capacity/4)%w.capacity].RecordTime) > w.eventFreshDuration {
		capacity := max(w.capacity/2, w.lowerBoundCapacity)
		if capacity < w.capacity {
			w.doCacheResizeLocked(capacity)
		}
		return
	}
}

The asymmetry is deliberate. Growing is conditioned on the oldest event still being fresh — if the buffer is full and the oldest entry is recent, the churn is real and the buffer is too small. Shrinking is conditioned on the quarter point being stale, not the oldest entry — that hysteresis is what stops the buffer from flapping up and down around the threshold on a workload with a bursty event rate.

The “fresh” window is itself derived from the bookmark frequency:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:63
	// DefaultEventFreshDuration is the default time duration of events
	// we want to keep.
	// We set it to defaultBookmarkFrequency plus epsilon to maximize
	// chances that last bookmark was sent within kept history, at the
	// same time, minimizing the needed memory usage.
	DefaultEventFreshDuration = defaultBookmarkFrequency + 15*time.Second

One minute (the bookmark frequency) plus fifteen seconds. The point is that the cache should be able to replay the last bookmark it sent — a client that resumes from a bookmark must be able to find it in the history.

If you raise --min-request-timeout or otherwise lengthen the watch window, the upper bound scales proportionally:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_history.go:55
// capacityUpperBound denotes the maximum possible capacity of the watch cache
// to which it can resize.
func capacityUpperBound(eventFreshDuration time.Duration) int {
	if eventFreshDuration <= DefaultEventFreshDuration {
		return defaultUpperBoundCapacity
	}
	// eventFreshDuration determines how long the watch events are supposed
	// to be stored in the watch cache.
	// In very high churn situations, there is a need to store more events
	// in the watch cache, hence it would have to be upsized accordingly.
	// Because of that, for larger values of eventFreshDuration, we set the
	// upper bound of the watch cache's capacity proportionally to the ratio
	// between eventFreshDuration and DefaultEventFreshDuration.
	// Given that the watch cache size can only double, we round up that
	// proportion to the next power of two.
	exponent := int(math.Ceil((math.Log2(eventFreshDuration.Seconds() / DefaultEventFreshDuration.Seconds()))))
	if maxExponent := int(math.Floor((math.Log2(math.MaxInt32 / defaultUpperBoundCapacity)))); exponent > maxExponent {
		// Making sure that the capacity's upper bound fits in a 32-bit integer.
		exponent = maxExponent
		klog.Warningf("Capping watch cache capacity upper bound to %v", defaultUpperBoundCapacity<<exponent)
	}
	return defaultUpperBoundCapacity << exponent
}

The ratio is rounded up to the next power of two, because the buffer can only double — an intermediate capacity like 150 000 could never be reached by doubling from 100.

The batch buffer

Serving a watch from history means walking the circular buffer under the watch cache lock. Doing that once per event would be brutal, so the interval is materialised into a small batch buffer first:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_interval.go:330
const bufferSize = 100

// watchCacheIntervalBuffer is used to reduce acquiring
// the lock on each invocation of historyCacheIntervalSource.next().
type watchCacheIntervalBuffer struct {
	// buffer is used to hold watchCacheEvents that
	// the interval returns on a call to Next().
	buffer []*watchCacheEvent
	// The first element of buffer is defined by startIndex,
	// its last element is defined by endIndex.
	startIndex int
	endIndex int
}

One hundred events per lock acquisition. The buffer also nils out the entries it has handed over, so a long-lived watch does not pin the event objects in memory after they have aged out of the cache:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache_interval.go:344
// next returns the next event present in the interval buffer provided
// it is not empty.
func (wcib *watchCacheIntervalBuffer) next() (*watchCacheEvent, bool) {
	if wcib.isEmpty() {
		return nil, false
	}
	next := wcib.buffer[wcib.startIndex]
	// clean the unused event reference in the buffer. If this is not
	// done, event if the watch event is aged out from the watch
	// cache, it will not be GCed during the lifetime of the watcher
	// that holds a reference to the buffer.
	wcib.buffer[wcib.startIndex] = nil
	wcib.startIndex++
	return next, true
}

What to watch

Two apiserver metrics tell you whether this code path is hurting you. apiserver_watch_cache_read_wait_seconds is the histogram of how long requests waited for freshness — its buckets top out at three seconds (0.005, 0.025, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.25, 1.5, 2, 3), because three seconds is the wall. If you see a fat tail at the 3-second bucket, the cache is chronically behind and the requests are falling through to etcd. And apiserver_watch_cache_consistent_read_total with fallback="true" counts the requests that gave up waiting and went to etcd instead.

Wrapping up

The one thing to remember: the watch cache’s consistency contract is a three-second wall. Ask for a resource version it has not seen and it waits on a condvar for up to blockTimeout = 3s, then returns TooLargeResourceVersion with a one-second Retry-After. The wakeup goroutine is only spawned when resourceVersion > 0, because a zero resource version can never be satisfied and is the overwhelmingly common case.

The part people get wrong: the default behaviour after the timeout is not to fail. The delegator falls back to etcd and serves the request there, so the client sees a normal response. Enabling ConsistentListFromCacheSkipTimeoutFallback changes that into a 429 — it does not make the cache faster, it trades a slow answer for an honest “not yet” so that etcd is not stampeded.

What to go do about it: watch apiserver_watch_cache_read_wait_seconds — a p99 sitting at 3 seconds means the cacher is chronically behind and you are paying etcd round trips for every late read. Watch apiserver_watch_cache_consistent_read_total{fallback="true"} to see how often that fallback fires; if it is non-trivial, the etcd client pool is taking the hit. On the client side, handle Retry-After on the TooLargeResourceVersion timeout error rather than treating it as a failure — the whole point of the one-second retry hint is that the next attempt will succeed. And if you have a workload that reads its own writes, either pin the resource version explicitly (and accept the wait) or accept the eventual consistency instead.

Bookmarks in a Bucket: How the Watch Cache Notifies 100k Watchers That It Is Alive

Jay Hubbert — Thu, 22 Oct 2026 12:10:00 +0000

TL;DR: A watch bookmark is an event with nothing in it but a resourceVersion - it tells a client “you are still alive, resume from here”. The apiserver’s watch cache sends one roughly every minute to every watcher, and it deliberately does not do that with a timer per watcher. Instead, watchers are filed into map[int64][]*cacheWatcher keyed by the whole second in which they want their bookmark, and expired buckets are popped in bulk, so a tick costs O(watchers due now), not O(watchers). On top of that sits a three-state machine - WaitingForBookmark -> BookmarkReceived -> BookmarkSent - because the WatchList protocol demands a bookmark carrying a specific resourceVersion, and until that version shows up the watcher must not get a heartbeat at all.

A watch is a long-lived HTTP response. The client holds it open and expects something to arrive periodically, otherwise it cannot tell “the cluster is quiet” apart from “the network died”. A watch silent for ten minutes is indistinguishable from a broken one, and the client’s only correct response is to reconnect - which costs a full relist.

So the apiserver has to emit periodic proof-of-life to every open watch. On a cluster with a hundred thousand watches that is a hundred thousand proof-of-life events per minute, and the naive implementation of “a timer per watcher” is hopeless: a hundred thousand goroutines, a hundred thousand timers, and a scheduler wake-up storm every minute that lands on every apiserver replica at the same instant.

The watch cache solves this with two tricks that compose. The first is about scheduling: how do you fire 100 000 timers without having 100 000 timers? The second is about correctness: what do you do when a client has asked for a bookmark carrying a specific resourceVersion and that version has not been produced yet?

What a bookmark actually is

A bookmark is a watch.Event of type watch.Bookmark whose object is an empty instance of the resource type, stamped with a resourceVersion and nothing else. No spec, no status, no metadata. It carries exactly one piece of information - “the server has processed everything up to RV X” - and its entire job is to give a client a safe resume point. The frequency is a constant, not a flag: defaultBookmarkFrequency = time.Minute (cacher.go:78). Because the cache keeps defaultBookmarkFrequency + 15*time.Second of event history, a client always has 75 seconds of retained history to resume from - the extra fifteen seconds so the most recent bookmark is still inside it.

There is a second, less obvious source of bookmarks: etcd itself. etcd sends progress notifications on a watch, and the apiserver can request one explicitly via RequestWatchProgress. The apiserver’s own watch layer turns those into watch.Bookmark events too - but the cacher throws them away, and the reason is about fleet-wide amplification:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:953
			// Don't dispatch bookmarks coming from the storage layer.
			// They can be very frequent (even to the level of subseconds)
			// to allow efficient watch resumption on kube-apiserver restarts,
			// and propagating them down may overload the whole system.
			//
			// TODO: If at some point we decide the performance and scalability
			// footprint is acceptable, this is the place to hook them in.
			// However, we then need to check if this was called as a result
			// of a bookmark event or regular Add/Update/Delete operation by
			// checking if resourceVersion here has changed.
			if event.Type != watch.Bookmark {

etcd’s progress notifications are sub-second. If every apiserver forwarded them, one quiet object would generate hundreds of bookmark events per second per watcher, across every replica watching it. The cacher’s one-minute schedule is the rate limit on that.

The problem with one timer per watcher

The data structure the cacher reaches for is deliberately blunt:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:202
// As we don't need a high precision here, we keep all watchers timeout within a
// second in a bucket, and pop up them once at the timeout. To be more specific,
// if you set fire time at X, you can get the bookmark within (X-1,X+1) period.
type watcherBookmarkTimeBuckets struct {
	// the key of watcherBuckets is the number of seconds since createTime
	watchersBuckets map[int64][]*cacheWatcher
	createTime time.Time
	startBucketID int64
	clock clock.Clock
	bookmarkFrequency time.Duration
}

The key is seconds since the cacher was created, not a timestamp. That makes the key a small integer, makes the map cheap, and - crucially - makes the whole thing a calendar: buckets are consecutive integers, and “expired” means “the bucket index is less than the current second”. The comment above it says the trade-off out loud: a bookmark requested at time X arrives somewhere in the window (X-1, X+1). The apiserver gave up sub-second precision on purpose, and bought O(due) scheduling with it.

Adding a watcher means asking the watcher when it next wants a bookmark (nextBookmarkTime) and filing it under that second:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:231
	nextTime, ok := w.nextBookmarkTime(t.clock.Now(), t.bookmarkFrequency)
	if !ok {
		return false
	}
	bucketID := int64(nextTime.Sub(t.createTime) / time.Second)
	if bucketID < t.startBucketID {
		bucketID = t.startBucketID
	}
	watchers := t.watchersBuckets[bucketID]
	t.watchersBuckets[bucketID] = append(watchers, w)
	return true
}

The bucketID < t.startBucketID clamp exists because nextBookmarkTime can legitimately return the zero time.Time - meaning “immediately” - and the zero time is before the cacher was created, so the subtraction goes negative. The clamp folds it into the current bucket. And addWatcherThreadUnsafe can return false - “no bookmark wanted” - when the watcher’s computed heartbeat time has already passed.

Popping the bucket

The expiry side is the mirror image, and the whole trick in six lines: it walks startBucketID forward one second at a time, deleting each bucket it passes and collecting its watchers.

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:244
func (t *watcherBookmarkTimeBuckets) popExpiredWatchersThreadUnsafe() [][]*cacheWatcher {
	currentBucketID := int64(t.clock.Since(t.createTime) / time.Second)
	// There should be one or two elements in almost all cases
	expiredWatchers := make([][]*cacheWatcher, 0, 2)
	for ; t.startBucketID <= currentBucketID; t.startBucketID++ {
		if watchers, ok := t.watchersBuckets[t.startBucketID]; ok {
			delete(t.watchersBuckets, t.startBucketID)
			expiredWatchers = append(expiredWatchers, watchers)
		}
	}
	return expiredWatchers
}

startBucketID is a watermark. Buckets below it have already been fired and are deleted as they are consumed, so the map never holds more than one minute’s worth of watchers - at 100 000 watchers, roughly 1 700 per second. And because the bucket key is a whole second, every watcher that wants a bookmark in the same second gets the same bookmark event, fanned out from one dispatch. A per-watcher timer gives you O(total watchers) work per minute with 100 000 separate wake-ups; the bucket gives you O(watchers due in the expired second) work per tick with one wake-up - a million bookmark events a minute in 60 batches of ~17 000 rather than 100 000 batches of 1.

The watchers that come out of the bucket are queued for dispatch by startDispatchingBookmarkEventsLocked (cacher.go:1088), which skips any watcher that has already stopped, and after the dispatch finishDispatching requeues them via c.bookmarkWatchers.addWatcherThreadUnsafe(watcher) (cacher.go:1205). That requeue is where the watcher’s next deadline gets computed - the one place the per-watcher policy lives.

The one-second tick

The timer that drives all of this is a single one, jittered so a fleet of apiservers does not synchronize:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:925
func (c *Cacher) dispatchEvents() {
	// Jitter to help level out any aggregate load.
	bookmarkTimer := c.clock.NewTimer(wait.Jitter(time.Second, 0.25))
	defer bookmarkTimer.Stop()

wait.Jitter(time.Second, 0.25) means a delay uniformly in [0.75s, 1.25s], re-jittered on every fire. On each tick the cacher emits a single bookmark event stamped with the last resourceVersion it processed, and hands it to every watcher currently in watchersBuffer - and for a bookmark event, that buffer was populated only from the expired buckets. If no bucket is due, the buffer is empty and the tick costs a map lookup and nothing else. A tick that finds 1 700 due watchers is one event broadcast to 1 700 channels, not 1 700 allocations.

The WatchList contract

Now the harder half. The WatchList protocol (sendInitialEvents=true, allowWatchBookmarks=true) lets a client ask the server to do the LIST and the WATCH in one stream: initial objects, then a bookmark marking the end of the initial set, then live events. The critical property is that the client can tell where the snapshot ended - the bookmark’s resourceVersion must be at least the version of the last object in the initial set. The cacher therefore computes, per watcher, a minimum acceptable bookmark resourceVersion:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1324
// getBookmarkAfterResourceVersionLocked returns a ResourceVersion after which
// the bookmark event will be delivered.
//
// Must be called under the watchCache lock.
func (c *Cacher) getBookmarkAfterResourceVersionLocked(parsedResourceVersion, requiredResourceVersion uint64, opts storage.ListOptions) uint64 {
	switch {
	case len(opts.ResourceVersion) == 0:
		return requiredResourceVersion
	case parsedResourceVersion == 0:
		// here we assume that watchCache locked is already held
		return c.watchCache.resourceVersion
	default:
		return parsedResourceVersion
	}
}

Three cases. If the client sent no resourceVersion at all, the bookmark must be at least the version the cache was required to be synced to. If the client sent resourceVersion=0 - “give me the latest” - the bar is whatever the cache currently holds, i.e. the version of the initial event set the client is about to receive. Otherwise the client’s own requested version is the floor, enforced at the very edge in the function that puts an event into a watcher’s input channel:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:158
func (c *cacheWatcher) nonblockingAdd(event *watchCacheEvent) bool {
	// if the bookmarkAfterResourceVersion hasn't been seen
	// we will try to deliver a bookmark event every second.
	// the following check will discard a bookmark event
	// if it is < than the bookmarkAfterResourceVersion
	// so that we don't pollute the input channel
	if event.Type == watch.Bookmark && event.ResourceVersion < c.bookmarkAfterResourceVersion {
		return false
	}
	select {
	case c.input <- event:
		c.markBookmarkAfterRvAsReceived(event)
		return true
	default:
		return false
	}
}

A bookmark that is too old is dropped on the floor. The watcher’s input channel is not polluted with bookmarks the client cannot use, and the watcher does not advance its state machine.

Which brings us to the starvation. nextBookmarkTime (cache_watcher.go:233) is where the policy lives. It tries to send a bookmark (a) right before the watch timeout, two seconds ahead of the deadline, (b) roughly every minute, and (c) immediately when the required resourceVersion has not been confirmed yet - the client has already been sent the initial data and is waiting for one specific RV, and the cacher does not know when it will see that RV, so it polls:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:251
	if !c.wasBookmarkAfterRvReceived() {
		return time.Time{}, true // schedule immediately
	}

	heartbeatTime := now.Add(bookmarkFrequency)
	if c.deadline.IsZero() {
		// Timeout is set by our client libraries (e.g. reflector) as well as defaulted by
		// apiserver if properly configured. So this shoudln't happen in practice.
		return heartbeatTime, true
	}
	if pretimeoutTime := c.deadline.Add(-2 * time.Second); pretimeoutTime.Before(heartbeatTime) {
		heartbeatTime = pretimeoutTime
	}

	if heartbeatTime.Before(now) {
		return time.Time{}, false
	}
	return heartbeatTime, true
}

The check at the top short-circuits everything: if the required bookmark has not arrived yet, the watcher asks to be fired immediately - it returns the zero time, which lands in the current bucket and gets popped on the next tick. The one-minute heartbeat and the pre-deadline bookmark do not even get computed. The watcher asks every second, and keeps asking until the cacher produces a bookmark at or above the required version.

The heartbeatTime.Before(now) check at the bottom returns false - “do not schedule me” - when the computed heartbeat is already in the past, which happens when the deadline is less than two seconds away: the pre-timeout bookmark would be in the past, so the watcher is not requeued and lets its deadline handle the teardown. The other deadline rule is hardcoded: a watch’s deadline comes from the HTTP request context, and the apiserver randomises it between --min-request-timeout and twice that value (1800 seconds by default, so 30 to 60 minutes). Two seconds before that deadline the cacher sends a final bookmark, so a reconnecting client resumes from a point at most a couple of seconds stale.

The three-state machine

The “have we seen the required bookmark yet” question is answered by a three-state integer on the watcher:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:38
// possible states of the cache watcher
const (
	// cacheWatcherWaitingForBookmark indicates the cacher
	// is waiting for a bookmark event with a specific RV set
	cacheWatcherWaitingForBookmark = iota

	// cacheWatcherBookmarkReceived indicates that the cacher
	// has received a bookmark event with required RV
	cacheWatcherBookmarkReceived

	// cacheWatcherBookmarkSent indicates that the cacher
	// has already sent a bookmark event to a client
	cacheWatcherBookmarkSent
)

WaitingForBookmark is the initial state for a WatchList watcher; it fires a request every second until the cacher produces an acceptable bookmark, at which point markBookmarkAfterRvAsReceived (called from nonblockingAdd) moves it to BookmarkReceived, and markBookmarkAfterRvSent - called from sendWatchCacheEvent once the event actually reaches the client’s result channel - moves it to BookmarkSent. The distinction between those last two states matters: a watcher that has received but not yet sent its required bookmark is one whose client is alive but slow, and the cacher will not tear it down abruptly. It drains the buffer and closes the watch gracefully, so the client can reconnect and resume from exactly that resourceVersion. A plain watch with no WatchList semantics gets bookmarkAfterResourceVersion == 0, and setBookmarkAfterResourceVersion (cache_watcher.go:332) puts it straight into the BookmarkSent state - no floor to reach, so it goes immediately onto the one-minute heartbeat.

The end-of-initial-events marker

The WatchList client needs to know where the snapshot stops. That is a bookmark carrying the annotation k8s.io/initial-events-end: "true" - metav1.InitialEventsAnnotationKey at staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go:512, set by storage.AnnotateInitialEventsEndBookmark (staging/src/k8s.io/apiserver/pkg/storage/util.go:92).

The cacher attaches that marker to the first acceptable bookmark a watcher sends - the one where wasBookmarkAfterRvSent() is still false. It also pre-builds an “initial events end” bookmark at the version of the initial event set and stashes it on the cache interval (cacher.go:1374), so the client sees the initial objects, then one bookmark with the annotation at the snapshot’s version, then live events with strictly greater versions.

The trade-offs

What it buys. A cache with 100 000 watchers runs one timer, not 100 000. A tick that finds nothing due costs a map lookup. A tick that finds 1 700 watchers due costs one bookmark event and 1 700 channel sends. There is no per-watcher goroutine, no per-watcher timer wheel, and no thundering herd, because the whole fleet’s timers are jittered by 25%.

What it costs.

Precision. The bucket granularity is one second, so a bookmark can be up to a second early or late relative to the requested time. The apiserver documents this as the (X-1, X+1) window and considers it acceptable.
A hardcoded schedule. defaultBookmarkFrequency is a minute, the pre-deadline offset is two seconds, and neither is a flag. You cannot ask the apiserver for a ten-second heartbeat.
A hardcoded floor for WatchList. The bookmark floor is derived from the client’s requested resourceVersion or the cache’s current version, and it only ever moves forward.
Starvation is a feature. A WatchList watcher that has not reached its required version polls for a bookmark once a second. If the cacher’s resourceVersion is stuck (a hot etcd, a paused write path), that watcher burns a dispatch per second, forever.
The event history window is tied to the bookmark period. The cache keeps 75 seconds of events (defaultBookmarkFrequency + 15*time.Second). If the bookmark frequency were configurable, that constant would have to be too.

Wrapping up

The one thing to remember: the watch cache does not run a timer per watcher. It runs one jittered one-second timer per resource type, and it files watchers into map[int64][]*cacheWatcher keyed by the whole second they want their bookmark, popping expired buckets in bulk - so the cost of a tick is O(watchers due now) and the cost of an idle tick is one map lookup.

The part people get wrong: “the apiserver sends me a bookmark every minute” is only true for a plain watch. A WatchList watcher (sendInitialEvents=true&allowWatchBookmarks=true) is held in cacheWatcherWaitingForBookmark and fires a request every second until the cacher produces a bookmark at or above its required resourceVersion; the one-minute heartbeat does not start until that happens. Watchers starved this way are also the ones the cacher refuses to kill abruptly - it drains their buffer and closes them gracefully, so the client resumes from the exact version.

What to go do about it: if you are debugging a client that seems to stall on a watch, look at apiserver_watch_cache_terminated_watchers_total and apiserver_watch_cache_events_dispatched_total (namespace apiserver, subsystem watch_cache) alongside etcd_bookmark_total, which counts the etcd progress notifications the cacher is discarding - a high number there with no matching events_dispatched_total increase confirms the cacher is swallowing storage-layer bookmarks on purpose. To change the cadence you must change code: defaultBookmarkFrequency in staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go and the c.deadline.Add(-2 * time.Second) offset in nextBookmarkTime. To change when a watch dies, use --min-request-timeout (default 1800 seconds).

Serialize Once, Deliver to a Thousand Watchers: The cachingObject

Jay Hubbert — Thu, 22 Oct 2026 11:40:00 +0000

TL;DR: A watch event that fans out to a thousand watchers would normally be serialized a thousand times, once per connection, because each connection negotiates its own content type and its own framing. The watch cache wraps every event object in a cachingObject that memoizes the encoded bytes per runtime.Identifier (which encodes the content type), so the JSON or protobuf is produced exactly once and then handed out. The deep copy that would normally protect the shared object from mutation is done lazily — only on the first actual mutation, and only if the mutation is not a no-op — because ADDED and MODIFIED events usually need no mutation at all.

Here is the shape of the problem. The watch cache holds one copy of every object. A thousand clients are watching. An object changes. The cacher fans the change out to every watcher, and each watcher writes the object into its own HTTP response stream, encoded in whatever content type that client negotiated.

Naively, that is a thousand JSON encodes of the same object, each producing a fresh byte slice, each of which then has to be copied into that connection’s buffer. On a busy apiserver with a large object and a thousand watchers, the encoding dominates the CPU cost of serving the watch.

The watch cache’s answer is a memo attached to the object itself.

The memo and the object it wraps

The memo is a small struct — a sync.Once, the bytes, and an error:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:45
// serializationResult captures a result of serialization.
type serializationResult struct {
	// once should be used to ensure serialization is computed once.
	once sync.Once

	// raw is serialized object.
	raw []byte
	// err is error from serialization.
	err error
}

// serializationsCache is a type for caching serialization results.
type serializationsCache map[runtime.Identifier]*serializationResult

The key is a runtime.Identifier, and identifiers are not just “json” or “protobuf”. They are a composite of everything that affects the output bytes — the encoder plus the group-versioner, plus, in the versioning codec’s case, the negotiated API version:

 go
 
 
// staging/src/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:96
type codecIdentifier struct {
	EncodeGV string `json:"encodeGV,omitempty"`
	Encoder string `json:"encoder,omitempty"`
	Name string `json:"name,omitempty"`
}

// identifier computes Identifier of Encoder based on codec parameters.
func identifier(encodeGV runtime.GroupVersioner, encoder runtime.Encoder) runtime.Identifier {
	result := codecIdentifier{
		Name: "versioning",
	}

	if encodeGV != nil {
		result.EncodeGV = encodeGV.Identifier()
	}
	if encoder != nil {
		result.Encoder = string(encoder.Identifier())
	}

Two clients watching the same object, one asking for application/json and the other for application/vnd.kubernetes.protobuf, get two different identifiers and therefore two different entries. Two clients asking for the same thing share one entry. That is exactly the granularity you want.

The wrapper itself implements metav1.Object and runtime.CacheableObject. It holds the wrapped object, a flag recording whether a deep copy has happened yet, and the memo map itself:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:64
type cachingObject struct {
	lock sync.RWMutex

	// deepCopied defines whether the object below has already been
	// deep copied. The operation is performed lazily on the first
	// setXxx operation.
	//
	// The lazy deep-copy make is useful, as effectively the only
	// case when we are setting some fields are ResourceVersion for
	// DELETE events, so in all other cases we can effectively avoid
	// performing any deep copies.
	deepCopied bool

	// Object for which serializations are cached.
	object metaRuntimeInterface

	// serializations is a cache containing object`s serializations.
	// The value stored in atomic.Value is of type serializationsCache.
	// The atomic.Value type is used to allow fast-path.
	serializations atomic.Value
}

That comment on deepCopied is the key insight of the whole design, and it is worth reading twice. The only field that ever gets mutated in practice is resourceVersion, and only on DELETE events. Everything else is read-only. So the deep copy — which for a big pod spec is a large allocation and a large memcpy — can be deferred until somebody actually tries to write, and skipped entirely if that write turns out to be a no-op.

The read path needs no lock

The memo map lives inside an atomic.Value, and it is replaced copy-on-write. The read path is a plain map lookup with no lock at all:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:102
func (o *cachingObject) getSerializationResult(id runtime.Identifier) *serializationResult {
	// Fast-path for getting from cache.
	serializations := o.serializations.Load().(serializationsCache)
	if result, exists := serializations[id]; exists {
		return result
	}

	// Slow-path (that may require insert).
	o.lock.Lock()
	defer o.lock.Unlock()

	serializations = o.serializations.Load().(serializationsCache)
	// Check if in the meantime it wasn't inserted.
	if result, exists := serializations[id]; exists {
		return result
	}

	// Insert an entry for <id>. This requires copy of existing map.
	newSerializations := make(serializationsCache)
	for k, v := range serializations {
		newSerializations[k] = v
	}
	result := &serializationResult{}
	newSerializations[id] = result
	o.serializations.Store(newSerializations)
	return result
}

The write path copies the whole map and stores the new one. That is O(size of map), but the map holds one entry per distinct content type — one, two, maybe three — so the copy is trivial. And the read path is a single atomic load plus a map lookup, which is what thousands of concurrent watchers actually hammer.

Encode exactly once, splice zero-copy

The memo is populated by CacheEncode, which is the hook the serializers call instead of writing bytes directly:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:136
func (o *cachingObject) CacheEncode(id runtime.Identifier, encode func(runtime.Object, io.Writer) error, w io.Writer) error {
	result := o.getSerializationResult(id)
	result.once.Do(func() {
		buffer := bytes.NewBuffer(nil)
		// TODO(wojtek-t): This is currently making a copy to avoid races
		// in cases where encoding is making subtle object modifications,
		// e.g. #82497
		// Figure out if we can somehow avoid this under some conditions.
		result.err = encode(o.GetObject(), buffer)
		result.raw = buffer.Bytes()
	})
	// Once invoked, fields of serialization will not change.
	if result.err != nil {
		return result.err
	}
	if b, support := w.(runtime.Splice); support {
		b.Splice(result.raw)
		return nil
	}
	_, err := w.Write(result.raw)
	return err
}

Three things to notice.

First, result.once.Do is the guarantee. The encode closure runs exactly once per identifier, no matter how many goroutines call CacheEncode concurrently. The sync.Once also provides the happens-before edge that makes result.raw safe to read afterwards.

Second, o.GetObject() returns a deep copy — so the encoder cannot mutate the shared object. That copy happens inside the once.Do, so it happens once too, not once per watcher. The TODO about issue #82497 is about that copy: it is defensive, and the author would like to remove it.

Third, the runtime.Splice fast path. If the writer supports splicing, the cached bytes are moved into the writer rather than copied.

Splice is a tiny interface in apimachinery, and the contract is explicit about ownership:

 go
 
 
// staging/src/k8s.io/apimachinery/pkg/runtime/splice.go:24
// Splice is the interface that wraps the Splice method.
//
// Splice moves data from given slice without copying the underlying data for
// efficiency purpose. Therefore, the caller should make sure the underlying
// data is not changed later.
type Splice interface {
	Splice([]byte)
	io.Writer
	Reset()
	Bytes() []byte
}

“Splice moves data from given slice without copying the underlying data.” The watch buffer implements it, so the first watcher to encode an object gets the bytes handed over by reference, and every subsequent watcher gets the same slice handed over again. Instead of a thousand copies of a 200 KiB pod, there is one allocation and a thousand pointer moves.

Lazy deep copy, and the no-op check

Every setter on cachingObject routes through one helper, which does the no-op check twice — once optimistically under a read lock, once properly under the write lock:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:227
func (o *cachingObject) conditionalSet(isNoop func() bool, set func()) {
	if fastPath := func() bool {
		o.lock.RLock()
		defer o.lock.RUnlock()
		return isNoop()
	}(); fastPath {
		return
	}
	o.lock.Lock()
	defer o.lock.Unlock()
	if isNoop() {
		return
	}
	if !o.deepCopied {
		o.object = o.object.DeepCopyObject().(metaRuntimeInterface)
		o.deepCopied = true
	}
	o.invalidateCacheLocked()
	set()
}

Three stages. First, if the write would not change anything, return immediately — no deep copy, no cache invalidation. The no-op predicate is supplied by each setter; for the resource version it is a string compare, for the labels and annotations it is a reflect.DeepEqual:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:363
func (o *cachingObject) SetLabels(labels map[string]string) {
	o.conditionalSet(
		func() bool { return reflect.DeepEqual(o.object.GetLabels(), labels) },
		func() { o.object.SetLabels(labels) },
	)
}

Second, if the write is a real change, deep copy the wrapped object — but only once, ever, guarded by o.deepCopied. Third, invalidate the memo, because the cached bytes no longer describe the object.

The invalidation is treated as a bug, because it should never happen:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:208
func (o *cachingObject) invalidateCacheLocked() {
	if cache, ok := o.serializations.Load().(serializationsCache); ok && len(cache) == 0 {
		return
	}
	// We don't expect cache invalidation to happen - so we want
	// to log the stacktrace to allow debugging if that will happen.
	// OTOH, we don't want to spam logs with it.
	// So we try to log it at most once per second.
	if shouldLogCacheInvalidation(time.Now()) {
		klog.Warningf("Unexpected cache invalidation for %#v\n%s", o.object, string(debug.Stack()))
	}
	o.serializations.Store(make(serializationsCache))
}

A stack trace at warning level, rate-limited to one per second:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/caching_object.go:191
var (
	invalidationCacheTimestampLock sync.Mutex
	invalidationCacheTimestamp time.Time
)

// shouldLogCacheInvalidation allows for logging cache-invalidation
// at most once per second (to avoid spamming logs in case of issues).
func shouldLogCacheInvalidation(now time.Time) bool {
	invalidationCacheTimestampLock.Lock()
	defer invalidationCacheTimestampLock.Unlock()
	if invalidationCacheTimestamp.Add(time.Second).Before(now) {
		invalidationCacheTimestamp = now
		return true
	}
	return false
}

If you ever see Unexpected cache invalidation in a kubelet or apiserver log, something mutated a shared object behind the cache’s back. That is a real bug, and the stack trace is there to help you find it.

Where the wrapping happens

The wrapping is not done for every event. It is decided per event type, at dispatch time, and it is deliberately asymmetric:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:987
func setCachingObjects(event *watchCacheEvent, versioner storage.Versioner) {
	switch event.Type {
	case watch.Added, watch.Modified:
		if object, err := newCachingObject(event.Object); err == nil {
			event.Object = object
		} else {
			klog.Errorf("couldn't create cachingObject from: %#v", event.Object)
		}
		// Don't wrap PrevObject for update event (for create events it is nil).
		// We only encode those to deliver DELETE watch events, so if
		// event.Object is not nil it can be used only for watchers for which
		// selector was satisfied for its previous version and is no longer
		// satisfied for the current version.
		// This is rare enough that it doesn't justify making deep-copy of the
		// object (done by newCachingObject) every time.
	case watch.Deleted:
		// Don't wrap Object for delete events - these are not to deliver any
		// events. Only wrap PrevObject.
		if object, err := newCachingObject(event.PrevObject); err == nil {
			// Update resource version of the object.
			// event.PrevObject is used to deliver DELETE watch events and
			// for them, we set resourceVersion to <current> instead of
			// the resourceVersion of the last modification of the object.
			updateResourceVersion(object, versioner, event.ResourceVersion)
			event.PrevObject = object
		} else {
			klog.Errorf("couldn't create cachingObject from: %#v", event.Object)
		}
	}
}

For ADDED and MODIFIED, only the new object is wrapped — the previous version is left alone, because it is only needed for watchers whose selector matched before and no longer matches, which is rare enough that deep-copying it every time would not pay for itself.

For DELETED, the reverse: the current object is irrelevant (a delete delivers the previous state) and only PrevObject is wrapped. And because the delete event has to carry the current resource version — the one at which the deletion happened, not the one at which the object was last modified — the wrapper’s resource version is rewritten here, which is the one mutation that actually triggers the lazy deep copy.

The wrapping is per-dispatch, not per-watch

The wrapping happens in dispatchEvent, once per event, on a shallow copy of the event struct:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1018
func (c *Cacher) dispatchEvent(event *watchCacheEvent) {
	c.startDispatching(event)
	defer c.finishDispatching()
	// Watchers stopped after startDispatching will be delayed to finishDispatching,

	// Since add() can block, we explicitly add when cacher is unlocked.
	// Dispatching event in nonblocking way first, which make faster watchers
	// not be blocked by slower ones.
	if event.Type == watch.Bookmark {
		for _, watcher := range c.watchersBuffer {
			watcher.nonblockingAdd(event)
		}
	} else {
		// Set up caching of object serializations only for dispatching this event.
		//
		// Storing serializations in memory would result in increased memory usage,
		// but it would help for caching encodings for watches started from old
		// versions. However, we still don't have a convincing data that the gain
		// from it justifies increased memory usage, so for now we drop the cached
		// serializations after dispatching this event.
		//
		// Given that CachingObject is just wrapping the object and not perfoming
		// deep-copying (until some field is explicitly being modified), we create
		// it unconditionally to ensure safety and reduce deep-copying.
		//
		// Make a shallow copy to allow overwriting Object and PrevObject.
		wcEvent := *event
		setCachingObjects(&wcEvent, c.versioner)
		wcEvent.timeline.MarkAt(metrics.PointWatcherEnqueued, c.clock.Now())
		event = &wcEvent

Note the comment: the cached serializations are dropped after the dispatch finishes. The memo lives only as long as the fan-out. It is not a long-lived cache of encoded objects — it is a per-event, per-dispatch memo that dies with the event.

And then the fan-out itself, which is where the thousand watchers meet the one event:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1049
		c.blockedWatchers = c.blockedWatchers[:0]
		for _, watcher := range c.watchersBuffer {
			if !watcher.nonblockingAdd(event) {
				c.blockedWatchers = append(c.blockedWatchers, watcher)
			}
		}

Every watcher gets the same pointer to the same cachingObject. They share it. The memo inside it is what makes sharing safe, and the lazy deep copy is what makes the one mutation that does happen safe.

The watcher’s side

The one place a watcher mutates the shared object is turning a “the object changed” event into an “ADDED”/“MODIFIED”/“DELETED” event, which depends on whether the label selector matched before and after:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:373
func (c *cacheWatcher) convertToWatchEvent(event *watchCacheEvent) *watch.Event {
	if event.Type == watch.Bookmark {
		e := &watch.Event{Type: watch.Bookmark, Object: event.Object.DeepCopyObject()}
		if !c.wasBookmarkAfterRvSent() {
			if err := storage.AnnotateInitialEventsEndBookmark(e.Object); err != nil {
				utilruntime.HandleError(fmt.Errorf("error while accessing object's metadata gr: %v, identifier: %v, obj: %#v, err: %v", c.groupResource, c.identifier, e.Object, err))
				return nil
			}
		}
		return e
	}

	curObjPasses := event.Type != watch.Deleted && c.filter.filterWithAttrsAndPrefix(event.Key, event.ObjLabels, event.ObjFields, event.Object)
	oldObjPasses := false
	if event.PrevObject != nil {
		oldObjPasses = c.filter.filterWithAttrsAndPrefix(event.Key, event.PrevObjLabels, event.PrevObjFields, event.PrevObject)
	}
	if !curObjPasses && !oldObjPasses {
		// Watcher is not interested in that object.
		return nil
	}

	switch {
	case curObjPasses && !oldObjPasses:
		return &watch.Event{Type: watch.Added, Object: getMutableObject(event.Object)}
	case curObjPasses && oldObjPasses:
		return &watch.Event{Type: watch.Modified, Object: getMutableObject(event.Object)}
	case !curObjPasses && oldObjPasses:
		// return a delete event with the previous object content, but with the event's resource version
		oldObj := getMutableObject(event.PrevObject)
		// We know that if oldObj is cachingObject (which can only be set via
		// setCachingObjects), its resourceVersion is already set correctly and
		// we don't need to update it. However, since cachingObject efficiently
		// handles noop updates, we avoid this microoptimization here.
		updateResourceVersion(oldObj, c.versioner, event.ResourceVersion)
		return &watch.Event{Type: watch.Deleted, Object: oldObj}
	}

	return nil
}

The helper that decides whether to hand out the shared object or a copy is the whole design in four lines:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:357
func getMutableObject(object runtime.Object) runtime.Object {
	if _, ok := object.(*cachingObject); ok {
		// It is safe to return without deep-copy, because the underlying
		// object will lazily perform deep-copy on the first try to change
		// any of its fields.
		return object
	}
	return object.DeepCopyObject()
}

A cachingObject is handed out as-is, because it knows how to protect itself. Anything else is deep-copied, because it does not.

The memo map is copied on every new content type for an event. With one or two content types that is one or two tiny map allocations per event, in exchange for skipping N encodes. The lazy deep copy means the expensive allocation happens only for DELETE events, where the resource version actually has to be rewritten.

The one genuinely awkward part is that the memo is dropped after each dispatch, so a watch that starts from an old resource version and replays history re-encodes everything it replays. The comment says so explicitly: the tradeoff was measured and the memory cost of keeping the memos was judged not worth it.

Wrapping up

The one thing to remember: the watch cache does not serialize per watcher. It serializes per event, memoized by content type inside a cachingObject, and then hands the same bytes to every watcher through a spliceBuffer that moves rather than copies.

The part people get wrong: the deep copy is not skipped — it is deferred. cachingObject.conditionalSet performs it on the first mutation that is not a no-op, and invalidateCacheLocked throws the memo away when it happens. Any code path that mutates a watched object without going through the metav1.Object setters will corrupt every watcher sharing that event; the Unexpected cache invalidation warning with a stack trace is the fingerprint of exactly that mistake.

What to go do about it: if you are debugging watch latency, look at apiserver_longrunning_requests and the watch-specific histograms before suspecting the cacher. If you see Unexpected cache invalidation in the apiserver log, treat it as a bug in whatever wrote to the object — grep for direct field assignment on objects obtained from a watch. And when profiling, remember that the encode happens once per event per content type, so a profile that shows JSON marshalling dominating is telling you the number of events is too high, not that the encoding path is inefficient.