One Stream Instead of Two: The Watch List That Killed LIST + WATCH

TL;DR: The classic reflector did two things over two connections: a LIST (possibly N paginated round trips) and then a WATCH. The watch-list path (KEP-3157) collapses both into a single WATCH request carrying sendInitialEvents=true. The apiserver replies with synthetic ADDED events for the entire object set, then a synthetic BOOKMARK carrying the resource version, and then keeps the same stream open for live events. The reflector swaps its store, publishes the resource version, and hands the same watch.Interface to the caller — no reconnect, no gap.

The classic Kubernetes read path has always been two round trips. First you LIST, to get a snapshot of the world and a resourceVersion that names a point in etcd’s history. Then you WATCH from that resourceVersion, to learn about changes. It works, and it has worked for a decade, but it is expensive in ways that only became obvious at scale.

The LIST is paginated, by default 500 items per request (staging/src/k8s.io/client-go/tools/pager/pager.go:31). A namespace with 200 000 pods is 400 requests. Each one is served straight out of etcd, not out of the apiserver’s watch cache, because paginated lists with a continue token are explicitly forbidden from being cached. The apiserver pays for the deserialization of every one of those objects, holds the whole list in memory while it paginates, and the client holds the whole list in memory too. Then the client opens a second connection for the WATCH, and from that moment on there are two live connections per informer instead of one. The watch-list design (KEP-3157) does not optimise this. It removes it.

The request that starts the whole thing

The reflector decides once, at construction time, whether it is allowed to try the watch-list path:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:361
	r.useWatchList = clientfeatures.FeatureGates().Enabled(clientfeatures.WatchListClient)
	if r.useWatchList && watchlist.DoesClientNotSupportWatchListSemantics(lw) {
		r.logger.V(2).Info(
			"The client used to build this informer/reflector doesn't support WatchList semantics. The feature will be disabled. This is expected in unit tests but not in production. For details, see the documentation of watchlist.DoesClientNotSupportWatchListSemantics().",
			"feature", clientfeatures.WatchListClient,
		)
		r.useWatchList = false
	}

WatchListClient is a client-go feature gate, defaulting to false at v1.30 and true at v1.35 (staging/src/k8s.io/client-go/features/known_features.go:111).

Because client-go’s own feature gates are env-var driven, you can flip it without a recompile: KUBE_FEATURE_WatchListClient=true in the environment of your controller. The gate is read exactly once, on the first Enabled() call, so setting it after the first informer is constructed does nothing.

If the reflector is going to try the watch-list path, the request itself is built inside watchList, and it is a watch request with one extra flag:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:847
		resourceVersion = ""
		lastKnownRV := r.rewatchResourceVersion()
		temporaryStore = NewStore(DeletionHandlingMetaNamespaceKeyFunc, storeOpts...)
		// TODO(#115478): large "list", slow clients, slow network, p&f
		//  might slow down streaming and eventually fail.
		//  maybe in such a case we should retry with an increased timeout?
		timeoutSeconds := int64(r.minWatchTimeout.Seconds() + rand.Float64()*(r.maxWatchTimeout.Seconds()-r.minWatchTimeout.Seconds()))
		options := metav1.ListOptions{
			ResourceVersion:      lastKnownRV,
			AllowWatchBookmarks:  true,
			SendInitialEvents:    ptr.To(true),
			ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan,
			TimeoutSeconds:       &timeoutSeconds,
		}

Three of those fields are load-bearing. SendInitialEvents: ptr.To(true) is the whole point — it tells the server “please start the stream by sending me the current contents as ADDED events”. ResourceVersionMatch: NotOlderThan says “give me data at least as new as this version”, which is what makes the stream consistent even if the client was disconnected for a while. And AllowWatchBookmarks: true is what buys the end-of-initial-events marker.

The server-side check for “is this a watch-list request” is a one-liner, and it requires both flags — bookmarks must be enabled or there is no way to signal the end of the initial burst:

// staging/src/k8s.io/apiserver/pkg/endpoints/handlers/get.go:328
// isListWatchRequest is mirrored in staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go
func isListWatchRequest(opts metainternalversion.ListOptions) bool {
	return opts.SendInitialEvents != nil && *opts.SendInitialEvents && opts.AllowWatchBookmarks
}

The server’s answer: ADDED, ADDED, ADDED, bookmark

The response is a stream of watch.Events. Every object in the collection comes back as a synthetic ADDED, and then the server sends a synthetic BOOKMARK to mark the boundary. The bookmark is not just a bookmark — it carries an annotation that says “initial events are over”:

// staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go:507
	// InitialEventsAnnotationKey the name of the key
	// under which an annotation marking the end of
	// a watchlist stream is stored.
	//
	// The annotation is added to a "Bookmark" event.
	InitialEventsAnnotationKey = "k8s.io/initial-events-end"

The reflector’s event loop is shared between the watch-list path and the ordinary watch path — one function, handleAnyWatch, with a flag telling it whether to exit when the end-of-initial-events bookmark arrives:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:1058
			case watch.Bookmark:
				// A `Bookmark` means watch has synced here, just update the resourceVersion
				eventReceivedBesidesAdded = true
				if meta.GetAnnotations()[metav1.InitialEventsAnnotationKey] == "true" {
					watchListBookmarkReceived = true
				}

And here is the hand-off — the moment the bookmark arrives, the loop stops reading, and crucially it does not stop the watcher:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:1074
			// when eventReceivedBesidesAdded is true, that indicates we are definitely past any initial synthetic Added events
			setLastSyncResourceVersion(resourceVersion, eventReceivedBesidesAdded)
			eventCount++
			if exitOnWatchListBookmarkReceived && watchListBookmarkReceived {
				stopWatcher = false
				watchDuration := clock.Since(start)
				klog.FromContext(ctx).V(4).Info("Exiting watch because received the bookmark that marks the end of initial events stream", "reflector", name, "totalItems", eventCount, "duration", watchDuration)
				return watchListBookmarkReceived, nil
			}

stopWatcher = false is the entire trick. The deferred w.Stop() in handleAnyWatch is skipped, so the HTTP connection stays open, and the watch.Interface value is returned up the stack to watchList, which returns it to ListAndWatchWithContext, which passes it to watchWithResync -> watch. The ordinary watch loop finds w != nil, skips opening a new watch entirely, and starts consuming the stream that was already open. Zero additional connections. Zero reconnect latency. Zero window in which a change could be missed between the snapshot and the live stream — because there was no window; there was only ever one stream.

The fallback, and why w = nil matters

If anything goes wrong — the server is too old, the feature is disabled server-side, the request 400s — the reflector must not be left holding a half-broken stream. ListAndWatchWithContext is where that is decided:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:483
	if r.useWatchList {
		w, err = r.watchList(ctx)
		if w == nil && err == nil {
			// stopCh was closed
			return nil
		}
		if err != nil {
			logger.V(4).Info(
				"Data couldn't be fetched in watchlist mode. Falling back to regular list. This is expected if watchlist is not supported or disabled in kube-apiserver.",
				"err", err,
			)
			fallbackToList = true
			// ensure that we won't accidentally pass some garbage down the watch.
			w = nil
		}
	}

	if fallbackToList {
		err = r.list(ctx)
		if err != nil {
			return err
		}
	}

	logger.V(2).Info("Caches populated", "type", r.typeDescription, "reflector", r.name)
	return r.watchWithResync(ctx, w)
}

The w = nil line is the subtle one. watchList can return a non-nil w together with an error: the stream opened fine, the server started streaming, and then something failed mid-stream. If that w were passed down to watch(), the reflector would try to consume a stream that is already broken, and would treat its contents as a complete snapshot. Nil-ing it means the fallback path is the only path, and the classic LIST + WATCH runs from scratch with a known-good starting point. The comment says exactly this: “ensure that we won’t accidentally pass some garbage down the watch.”

Note also that the fallback is silent at V(4). Every informer in every controller that talks to an old apiserver logs this at high verbosity and carries on.

The snapshot swap

The watch-list path cannot write into the live store incrementally — the whole point is that the store should be replaced atomically once the snapshot is complete. So the objects land in a temporary store first:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:893
	initTrace.Step("Objects streamed", trace.Field{Key: "count", Value: len(temporaryStore.List())})
	r.setIsLastSyncResourceVersionUnavailable(false)

	// we utilize the temporaryStore to ensure independence from the current store implementation.
	// as of today, the store is implemented as a queue and will be drained by the higher-level
	// component as soon as it finishes replacing the content.
	checkWatchListDataConsistencyIfRequested(ctx, r.name, resourceVersion, r.listerWatcher.ListWithContext, transformer, temporaryStore.List)

	if err := r.store.Replace(temporaryStore.List(), resourceVersion); err != nil {
		return nil, fmt.Errorf("unable to sync watch-list result: %w", err)
	}
	initTrace.Step("SyncWith done")
	r.setLastSyncResourceVersion(resourceVersion)

	return w, nil

The checkWatchListDataConsistencyIfRequested call is interesting: it is a CI-only detector, gated on KUBE_WATCHLIST_INCONSISTENCY_DETECTOR having been set at process start, that re-LISTs the same collection through the ordinary path and panics if the two disagree. The upstream comment says so: “The consistency check is meant to be enforced only in the CI, not in production.”

The temporary store is a plain Store keyed with DeletionHandlingMetaNamespaceKeyFunc, so tombstones are resolved on the way in rather than leaking into the informer as DeletedFinalStateUnknown.

The timeout is randomised, and it has to be

Watch timeouts exist so that a dead connection is eventually noticed. But if every reflector in the cluster uses the same timeout, they all reconnect at the same instant — a thundering herd against the apiserver, and a periodic spike in watch_cache churn. So the timeout is jittered, uniformly, in [minWatchTimeout, maxWatchTimeout):

// staging/src/k8s.io/client-go/tools/cache/reflector.go:54
var (
	// We try to spread the load on apiserver by setting timeouts for
	// watch requests - it is random in [minWatchTimeout, 2*minWatchTimeout].
	defaultMinWatchTimeout = 5 * time.Minute
	defaultMaxWatchTimeout = 2 * defaultMinWatchTimeout

(The same var block also holds the reflector’s backoff constants: defaultBackoffInit = 800ms, defaultBackoffMax = 30s, defaultBackoffReset = 2m, defaultBackoffFactor = 2.0, defaultBackoffJitter = 1.0.)

Five minutes minimum, ten minutes maximum. The minWatchTimeout is deliberately floored: NewReflectorWithOptions only honours a caller-supplied value above the default, because “values lower than 5m will not be honored to avoid negative performance impact on controlplane”. A watch that reconnects every thirty seconds is a lot of requests for a control plane that is already the busiest thing in the cluster.

The randomisation happens at request-construction time, and it happens twice — once in the watch-list path, once in the ordinary watch path:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:588
			timeoutSeconds := int64(r.minWatchTimeout.Seconds() + rand.Float64()*(r.maxWatchTimeout.Seconds()-r.minWatchTimeout.Seconds()))
			options := metav1.ListOptions{
				ResourceVersion: r.LastSyncResourceVersion(),
				// We want to avoid situations of hanging watchers. Stop any watchers that do not
				// receive any events within the timeout window.
				TimeoutSeconds: &timeoutSeconds,
				// To reduce load on kube-apiserver on watch restarts, you may enable watch bookmarks.
				// Reflector doesn't assume bookmarks are returned at all (if the server do not support
				// watch bookmarks, it will ignore this field).
				AllowWatchBookmarks: true,
			}

Because the range is [300s, 600s), the expected number of reconnects per reflector per hour is 1 - 2, and they are spread uniformly rather than synchronised.

The pager trick: how to not hammer etcd

The watch-list path removes the LIST entirely on the happy path. But the fallback path still exists, and the fallback path still has to decide whether to paginate. And here the reflector does something that looks like a bug and is not: it deliberately switches pagination off in some cases.

// staging/src/k8s.io/client-go/tools/cache/reflector.go:693
		pager := pager.New(pager.SimplePageFunc(func(opts metav1.ListOptions) (runtime.Object, error) {
			return r.listerWatcher.ListWithContext(ctx, opts)
		}))
		switch {
		case r.WatchListPageSize != 0:
			pager.PageSize = r.WatchListPageSize
		case r.paginatedResult:
			// We got a paginated result initially. Assume this resource and server honor
			// paging requests (i.e. watch cache is probably disabled) and leave the default
			// pager size set.
		case options.ResourceVersion != "" && options.ResourceVersion != "0":
			// User didn't explicitly request pagination.
			//
			// With ResourceVersion != "", we have a possibility to list from watch cache,
			// but we do that (for ResourceVersion != "0") only if Limit is unset.
			// To avoid thundering herd on etcd (e.g. on master upgrades), we explicitly
			// switch off pagination to force listing from watch cache (if enabled).
			// With the existing semantic of RV (result is at least as fresh as provided RV),
			// this is correct and doesn't lead to going back in time.
			//
			// We also don't turn off pagination for ResourceVersion="0", since watch cache
			// is ignoring Limit in that case anyway, and if watch cache is not enabled
			// we don't introduce regression.
			pager.PageSize = 0
		}

The default page size is 500 (staging/src/k8s.io/client-go/tools/pager/pager.go:31).

Setting pager.PageSize = 0 means Limit is never set on the request. The apiserver’s watch cache will then serve the whole collection from memory, in one response, without touching etcd. The alternative — a paginated list — always goes to etcd, because the apiserver refuses to serve a paginated request from the cache: the continue token pins a specific etcd revision, and the cache does not have a snapshot of that.

The trade-off is explicit in the comment: “To avoid thundering herd on etcd (e.g. on master upgrades)”. During a control-plane upgrade, every informer in the cluster re-lists at once. If those re-lists are paginated, etcd gets hit by thousands of concurrent 500-item range scans, and the apiserver gets hammered with deserialization work for objects that are about to be thrown away. If they are unpaginated, the watch cache absorbs them.

Note that ResourceVersion == "0" is excluded from this trick, and the reason is subtle: relistResourceVersion returns "0" for the very first list of a reflector, because “for performance reasons, initial list performed by reflector uses ‘0’ as resource version to allow it to be served from the watch cache”. But the watch cache ignores Limit when the requested version is 0, so pagination would be a no-op anyway — and if the watch cache is disabled, turning pagination off would turn one big slow list into one giant one. So the reflector leaves the default in place and learns from the result:

// staging/src/k8s.io/client-go/tools/cache/reflector.go:760
	if options.ResourceVersion == "0" && paginatedResult {
		r.paginatedResult = true
	}

Once the reflector has seen one paginated response, it assumes the server does not have a usable watch cache for this resource and keeps paginating forever after.

There is a third option, WatchListPageSize, which forces a page size for the initial and resync watch lists. Its doc comment is a warning: “NOTE: It should be used carefully as paginated lists are always served directly from etcd, which is significantly less efficient and may lead to serious performance and scalability problems.”

The trade-off

What watch-list buys: one connection instead of two, one apiserver round trip instead of N+1, no full-collection buffering in the apiserver for paginated lists, and no reconnect window between the snapshot and the live stream. On a cluster with ten thousand informers, the difference between “one LIST + one WATCH” and “one WATCH” is roughly ten thousand fewer connections and a large reduction in etcd range-scan load.

What it costs:

• The server must support it. sendInitialEvents is a relatively recent addition to the watch protocol. Older apiservers reject the request, the reflector falls back, and you have paid one failed request per reflector per retry cycle. The fallback is automatic but it is not free.
• The initial burst is a burst. The server streams the entire collection as fast as it can. The TODO in the source says so: “large ’list’, slow clients, slow network, p&f might slow down streaming and eventually fail”. A controller on a bad network link can have the server’s send buffer fill up.
• The snapshot lands in a temporary store first, so peak memory during startup is roughly 2x the object set, not 1x.
• The end-of-initial-events bookmark is load-bearing. Without AllowWatchBookmarks, the server cannot signal the end of the burst and the client has no way to know when the snapshot is complete.

Wrapping up

The one thing to remember: sendInitialEvents=true on a WATCH request turns the snapshot and the stream into one connection. The server sends synthetic ADDED events, then a BOOKMARK annotated with k8s.io/initial-events-end, and then keeps the same stream open — the reflector’s stopWatcher = false at staging/src/k8s.io/client-go/tools/cache/reflector.go:1078 is what keeps it open.

The part people get wrong: the fallback to plain LIST is silent and the failure is at V(4), so an informer that is quietly doing the old two-connection dance looks exactly like one that is not. Watch for "Data couldn't be fetched in watchlist mode" at -v=4, and remember that a server which does not support the feature will make every reflector pay a failed request on every retry cycle.

What to go do about it: set KUBE_FEATURE_WatchListClient=true in the environment of a controller that is pinned to an apiserver older than v1.31 and confirm the reflector logs "Exiting watch because received the bookmark that marks the end of initial events stream". On the apiserver side, watch apiserver_longrunning_timestamps and the watch-cache hit rate for the affected resources. If you have informer memory pressure, remember that WatchListPageSize forces paginated lists — which are always served from etcd — so leave it unset unless you have a specific reason.