Bookmarks in a Bucket: How the Watch Cache Notifies 100k Watchers That It Is Alive

TL;DR: A watch bookmark is an event with nothing in it but a resourceVersion - it tells a client “you are still alive, resume from here”. The apiserver’s watch cache sends one roughly every minute to every watcher, and it deliberately does not do that with a timer per watcher. Instead, watchers are filed into map[int64][]*cacheWatcher keyed by the whole second in which they want their bookmark, and expired buckets are popped in bulk, so a tick costs O(watchers due now), not O(watchers). On top of that sits a three-state machine - WaitingForBookmark -> BookmarkReceived -> BookmarkSent - because the WatchList protocol demands a bookmark carrying a specific resourceVersion, and until that version shows up the watcher must not get a heartbeat at all.

A watch is a long-lived HTTP response. The client holds it open and expects something to arrive periodically, otherwise it cannot tell “the cluster is quiet” apart from “the network died”. A watch silent for ten minutes is indistinguishable from a broken one, and the client’s only correct response is to reconnect - which costs a full relist.

So the apiserver has to emit periodic proof-of-life to every open watch. On a cluster with a hundred thousand watches that is a hundred thousand proof-of-life events per minute, and the naive implementation of “a timer per watcher” is hopeless: a hundred thousand goroutines, a hundred thousand timers, and a scheduler wake-up storm every minute that lands on every apiserver replica at the same instant.

The watch cache solves this with two tricks that compose. The first is about scheduling: how do you fire 100 000 timers without having 100 000 timers? The second is about correctness: what do you do when a client has asked for a bookmark carrying a specific resourceVersion and that version has not been produced yet?

What a bookmark actually is

A bookmark is a watch.Event of type watch.Bookmark whose object is an empty instance of the resource type, stamped with a resourceVersion and nothing else. No spec, no status, no metadata. It carries exactly one piece of information - “the server has processed everything up to RV X” - and its entire job is to give a client a safe resume point. The frequency is a constant, not a flag: defaultBookmarkFrequency = time.Minute (cacher.go:78). Because the cache keeps defaultBookmarkFrequency + 15*time.Second of event history, a client always has 75 seconds of retained history to resume from - the extra fifteen seconds so the most recent bookmark is still inside it.

There is a second, less obvious source of bookmarks: etcd itself. etcd sends progress notifications on a watch, and the apiserver can request one explicitly via RequestWatchProgress. The apiserver’s own watch layer turns those into watch.Bookmark events too - but the cacher throws them away, and the reason is about fleet-wide amplification:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:953
			// Don't dispatch bookmarks coming from the storage layer.
			// They can be very frequent (even to the level of subseconds)
			// to allow efficient watch resumption on kube-apiserver restarts,
			// and propagating them down may overload the whole system.
			//
			// TODO: If at some point we decide the performance and scalability
			// footprint is acceptable, this is the place to hook them in.
			// However, we then need to check if this was called as a result
			// of a bookmark event or regular Add/Update/Delete operation by
			// checking if resourceVersion here has changed.
			if event.Type != watch.Bookmark {

etcd’s progress notifications are sub-second. If every apiserver forwarded them, one quiet object would generate hundreds of bookmark events per second per watcher, across every replica watching it. The cacher’s one-minute schedule is the rate limit on that.

The problem with one timer per watcher

The data structure the cacher reaches for is deliberately blunt:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:202
// As we don't need a high precision here, we keep all watchers timeout within a
// second in a bucket, and pop up them once at the timeout. To be more specific,
// if you set fire time at X, you can get the bookmark within (X-1,X+1) period.
type watcherBookmarkTimeBuckets struct {
	// the key of watcherBuckets is the number of seconds since createTime
	watchersBuckets   map[int64][]*cacheWatcher
	createTime        time.Time
	startBucketID     int64
	clock             clock.Clock
	bookmarkFrequency time.Duration
}

The key is seconds since the cacher was created, not a timestamp. That makes the key a small integer, makes the map cheap, and - crucially - makes the whole thing a calendar: buckets are consecutive integers, and “expired” means “the bucket index is less than the current second”. The comment above it says the trade-off out loud: a bookmark requested at time X arrives somewhere in the window (X-1, X+1). The apiserver gave up sub-second precision on purpose, and bought O(due) scheduling with it.

Adding a watcher means asking the watcher when it next wants a bookmark (nextBookmarkTime) and filing it under that second:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:231
	nextTime, ok := w.nextBookmarkTime(t.clock.Now(), t.bookmarkFrequency)
	if !ok {
		return false
	}
	bucketID := int64(nextTime.Sub(t.createTime) / time.Second)
	if bucketID < t.startBucketID {
		bucketID = t.startBucketID
	}
	watchers := t.watchersBuckets[bucketID]
	t.watchersBuckets[bucketID] = append(watchers, w)
	return true
}

The bucketID < t.startBucketID clamp exists because nextBookmarkTime can legitimately return the zero time.Time - meaning “immediately” - and the zero time is before the cacher was created, so the subtraction goes negative. The clamp folds it into the current bucket. And addWatcherThreadUnsafe can return false - “no bookmark wanted” - when the watcher’s computed heartbeat time has already passed.

Popping the bucket

The expiry side is the mirror image, and the whole trick in six lines: it walks startBucketID forward one second at a time, deleting each bucket it passes and collecting its watchers.

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:244
func (t *watcherBookmarkTimeBuckets) popExpiredWatchersThreadUnsafe() [][]*cacheWatcher {
	currentBucketID := int64(t.clock.Since(t.createTime) / time.Second)
	// There should be one or two elements in almost all cases
	expiredWatchers := make([][]*cacheWatcher, 0, 2)
	for ; t.startBucketID <= currentBucketID; t.startBucketID++ {
		if watchers, ok := t.watchersBuckets[t.startBucketID]; ok {
			delete(t.watchersBuckets, t.startBucketID)
			expiredWatchers = append(expiredWatchers, watchers)
		}
	}
	return expiredWatchers
}

startBucketID is a watermark. Buckets below it have already been fired and are deleted as they are consumed, so the map never holds more than one minute’s worth of watchers - at 100 000 watchers, roughly 1 700 per second. And because the bucket key is a whole second, every watcher that wants a bookmark in the same second gets the same bookmark event, fanned out from one dispatch. A per-watcher timer gives you O(total watchers) work per minute with 100 000 separate wake-ups; the bucket gives you O(watchers due in the expired second) work per tick with one wake-up - a million bookmark events a minute in 60 batches of ~17 000 rather than 100 000 batches of 1.

The watchers that come out of the bucket are queued for dispatch by startDispatchingBookmarkEventsLocked (cacher.go:1088), which skips any watcher that has already stopped, and after the dispatch finishDispatching requeues them via c.bookmarkWatchers.addWatcherThreadUnsafe(watcher) (cacher.go:1205). That requeue is where the watcher’s next deadline gets computed - the one place the per-watcher policy lives.

The one-second tick

The timer that drives all of this is a single one, jittered so a fleet of apiservers does not synchronize:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:925
func (c *Cacher) dispatchEvents() {
	// Jitter to help level out any aggregate load.
	bookmarkTimer := c.clock.NewTimer(wait.Jitter(time.Second, 0.25))
	defer bookmarkTimer.Stop()

wait.Jitter(time.Second, 0.25) means a delay uniformly in [0.75s, 1.25s], re-jittered on every fire. On each tick the cacher emits a single bookmark event stamped with the last resourceVersion it processed, and hands it to every watcher currently in watchersBuffer - and for a bookmark event, that buffer was populated only from the expired buckets. If no bucket is due, the buffer is empty and the tick costs a map lookup and nothing else. A tick that finds 1 700 due watchers is one event broadcast to 1 700 channels, not 1 700 allocations.

The WatchList contract

Now the harder half. The WatchList protocol (sendInitialEvents=true, allowWatchBookmarks=true) lets a client ask the server to do the LIST and the WATCH in one stream: initial objects, then a bookmark marking the end of the initial set, then live events. The critical property is that the client can tell where the snapshot ended - the bookmark’s resourceVersion must be at least the version of the last object in the initial set. The cacher therefore computes, per watcher, a minimum acceptable bookmark resourceVersion:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1324
// getBookmarkAfterResourceVersionLocked returns a ResourceVersion after which
// the bookmark event will be delivered.
//
// Must be called under the watchCache lock.
func (c *Cacher) getBookmarkAfterResourceVersionLocked(parsedResourceVersion, requiredResourceVersion uint64, opts storage.ListOptions) uint64 {
	switch {
	case len(opts.ResourceVersion) == 0:
		return requiredResourceVersion
	case parsedResourceVersion == 0:
		// here we assume that watchCache locked is already held
		return c.watchCache.resourceVersion
	default:
		return parsedResourceVersion
	}
}

Three cases. If the client sent no resourceVersion at all, the bookmark must be at least the version the cache was required to be synced to. If the client sent resourceVersion=0 - “give me the latest” - the bar is whatever the cache currently holds, i.e. the version of the initial event set the client is about to receive. Otherwise the client’s own requested version is the floor, enforced at the very edge in the function that puts an event into a watcher’s input channel:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:158
func (c *cacheWatcher) nonblockingAdd(event *watchCacheEvent) bool {
	// if the bookmarkAfterResourceVersion hasn't been seen
	// we will try to deliver a bookmark event every second.
	// the following check will discard a bookmark event
	// if it is < than the bookmarkAfterResourceVersion
	// so that we don't pollute the input channel
	if event.Type == watch.Bookmark && event.ResourceVersion < c.bookmarkAfterResourceVersion {
		return false
	}
	select {
	case c.input <- event:
		c.markBookmarkAfterRvAsReceived(event)
		return true
	default:
		return false
	}
}

A bookmark that is too old is dropped on the floor. The watcher’s input channel is not polluted with bookmarks the client cannot use, and the watcher does not advance its state machine.

Which brings us to the starvation. nextBookmarkTime (cache_watcher.go:233) is where the policy lives. It tries to send a bookmark (a) right before the watch timeout, two seconds ahead of the deadline, (b) roughly every minute, and (c) immediately when the required resourceVersion has not been confirmed yet - the client has already been sent the initial data and is waiting for one specific RV, and the cacher does not know when it will see that RV, so it polls:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:251
	if !c.wasBookmarkAfterRvReceived() {
		return time.Time{}, true // schedule immediately
	}

	heartbeatTime := now.Add(bookmarkFrequency)
	if c.deadline.IsZero() {
		// Timeout is set by our client libraries (e.g. reflector) as well as defaulted by
		// apiserver if properly configured. So this shoudln't happen in practice.
		return heartbeatTime, true
	}
	if pretimeoutTime := c.deadline.Add(-2 * time.Second); pretimeoutTime.Before(heartbeatTime) {
		heartbeatTime = pretimeoutTime
	}

	if heartbeatTime.Before(now) {
		return time.Time{}, false
	}
	return heartbeatTime, true
}

The check at the top short-circuits everything: if the required bookmark has not arrived yet, the watcher asks to be fired immediately - it returns the zero time, which lands in the current bucket and gets popped on the next tick. The one-minute heartbeat and the pre-deadline bookmark do not even get computed. The watcher asks every second, and keeps asking until the cacher produces a bookmark at or above the required version.

The heartbeatTime.Before(now) check at the bottom returns false - “do not schedule me” - when the computed heartbeat is already in the past, which happens when the deadline is less than two seconds away: the pre-timeout bookmark would be in the past, so the watcher is not requeued and lets its deadline handle the teardown. The other deadline rule is hardcoded: a watch’s deadline comes from the HTTP request context, and the apiserver randomises it between --min-request-timeout and twice that value (1800 seconds by default, so 30 to 60 minutes). Two seconds before that deadline the cacher sends a final bookmark, so a reconnecting client resumes from a point at most a couple of seconds stale.

The three-state machine

The “have we seen the required bookmark yet” question is answered by a three-state integer on the watcher:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go:38
// possible states of the cache watcher
const (
	// cacheWatcherWaitingForBookmark indicates the cacher
	// is waiting for a bookmark event with a specific RV set
	cacheWatcherWaitingForBookmark = iota

	// cacheWatcherBookmarkReceived indicates that the cacher
	// has received a bookmark event with required RV
	cacheWatcherBookmarkReceived

	// cacheWatcherBookmarkSent indicates that the cacher
	// has already sent a bookmark event to a client
	cacheWatcherBookmarkSent
)

WaitingForBookmark is the initial state for a WatchList watcher; it fires a request every second until the cacher produces an acceptable bookmark, at which point markBookmarkAfterRvAsReceived (called from nonblockingAdd) moves it to BookmarkReceived, and markBookmarkAfterRvSent - called from sendWatchCacheEvent once the event actually reaches the client’s result channel - moves it to BookmarkSent. The distinction between those last two states matters: a watcher that has received but not yet sent its required bookmark is one whose client is alive but slow, and the cacher will not tear it down abruptly. It drains the buffer and closes the watch gracefully, so the client can reconnect and resume from exactly that resourceVersion. A plain watch with no WatchList semantics gets bookmarkAfterResourceVersion == 0, and setBookmarkAfterResourceVersion (cache_watcher.go:332) puts it straight into the BookmarkSent state - no floor to reach, so it goes immediately onto the one-minute heartbeat.

The end-of-initial-events marker

The WatchList client needs to know where the snapshot stops. That is a bookmark carrying the annotation k8s.io/initial-events-end: "true" - metav1.InitialEventsAnnotationKey at staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go:512, set by storage.AnnotateInitialEventsEndBookmark (staging/src/k8s.io/apiserver/pkg/storage/util.go:92).

The cacher attaches that marker to the first acceptable bookmark a watcher sends - the one where wasBookmarkAfterRvSent() is still false. It also pre-builds an “initial events end” bookmark at the version of the initial event set and stashes it on the cache interval (cacher.go:1374), so the client sees the initial objects, then one bookmark with the annotation at the snapshot’s version, then live events with strictly greater versions.

The trade-offs

What it buys. A cache with 100 000 watchers runs one timer, not 100 000. A tick that finds nothing due costs a map lookup. A tick that finds 1 700 watchers due costs one bookmark event and 1 700 channel sends. There is no per-watcher goroutine, no per-watcher timer wheel, and no thundering herd, because the whole fleet’s timers are jittered by 25%.

What it costs.

• Precision. The bucket granularity is one second, so a bookmark can be up to a second early or late relative to the requested time. The apiserver documents this as the (X-1, X+1) window and considers it acceptable.
• A hardcoded schedule. defaultBookmarkFrequency is a minute, the pre-deadline offset is two seconds, and neither is a flag. You cannot ask the apiserver for a ten-second heartbeat.
• A hardcoded floor for WatchList. The bookmark floor is derived from the client’s requested resourceVersion or the cache’s current version, and it only ever moves forward.
• Starvation is a feature. A WatchList watcher that has not reached its required version polls for a bookmark once a second. If the cacher’s resourceVersion is stuck (a hot etcd, a paused write path), that watcher burns a dispatch per second, forever.
• The event history window is tied to the bookmark period. The cache keeps 75 seconds of events (defaultBookmarkFrequency + 15*time.Second). If the bookmark frequency were configurable, that constant would have to be too.

Wrapping up

The one thing to remember: the watch cache does not run a timer per watcher. It runs one jittered one-second timer per resource type, and it files watchers into map[int64][]*cacheWatcher keyed by the whole second they want their bookmark, popping expired buckets in bulk - so the cost of a tick is O(watchers due now) and the cost of an idle tick is one map lookup.

The part people get wrong: “the apiserver sends me a bookmark every minute” is only true for a plain watch. A WatchList watcher (sendInitialEvents=true&allowWatchBookmarks=true) is held in cacheWatcherWaitingForBookmark and fires a request every second until the cacher produces a bookmark at or above its required resourceVersion; the one-minute heartbeat does not start until that happens. Watchers starved this way are also the ones the cacher refuses to kill abruptly - it drains their buffer and closes them gracefully, so the client resumes from the exact version.

What to go do about it: if you are debugging a client that seems to stall on a watch, look at apiserver_watch_cache_terminated_watchers_total and apiserver_watch_cache_events_dispatched_total (namespace apiserver, subsystem watch_cache) alongside etcd_bookmark_total, which counts the etcd progress notifications the cacher is discarding - a high number there with no matching events_dispatched_total increase confirms the cacher is swallowing storage-layer bookmarks on purpose. To change the cadence you must change code: defaultBookmarkFrequency in staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go and the c.deadline.Add(-2 * time.Second) offset in nextBookmarkTime. To change when a watch dies, use --min-request-timeout (default 1800 seconds).