Client-Go on Opensource internals

The Informer Will OOM If You Are Too Slow: An Unbounded Ring Buffer, On Purpose

Jay Hubbert — Fri, 23 Oct 2026 10:40:00 +0000

TL;DR: A shared informer fans every event out to every registered handler, and each handler is served by a processorListener with an unbounded growing ring buffer between the distributor and the handler. If your OnUpdate takes longer than events arrive, that ring grows without limit and the process OOMs. The upstream comment says so outright: “a failing/stalled listener will have infinite pendingNotifications added until we OOM.”

This is a design decision, not a bug, and understanding why it is a decision tells you a lot about what Kubernetes optimises for. The alternative — dropping events when the buffer is full, or blocking the distributor — would both be worse in the failure mode that actually happens.

But first, the shape of the pipeline, because “where is the buffer” is the question that matters.

Three goroutines and two unbuffered channels

When you register a handler on a shared informer, you get a processorListener. The source describes the whole arrangement:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1361
// processorListener relays notifications from a sharedProcessor to
// one ResourceEventHandler --- using three goroutines, two unbuffered
// channels, and an unbounded ring buffer. The `add(notification)`
// function sends the given notification to `addCh`. One goroutine
// runs `pop()`, which pumps notifications from `addCh` to `nextCh`
// using storage in the ring buffer while `nextCh` is not keeping up.
// Another goroutine runs `run()`, which receives notifications from
// `nextCh` and synchronously invokes the appropriate handler method.
//
// The third goroutine watches the upstream "has synced" channel
// and notifies a SingleFileTracker instance. That instance then
// combines the upstream state and the processListener state to
// implement the overall "event handler has synced".
//
// processorListener also keeps track of the adjusted requested resync
// period of the listener.
type processorListener struct {
	logger klog.Logger
	nextCh chan interface{}
	addCh chan interface{}
	done chan struct{}
	runFinished chan struct{}

	handler ResourceEventHandler
	handlerName string

	syncTracker *synctrack.SingleFileTracker
	upstreamHasSynced DoneChecker

	// pendingNotifications is an unbounded ring buffer that holds all notifications not yet distributed.
	// There is one per listener, but a failing/stalled listener will have infinite pendingNotifications
	// added until we OOM.
	// TODO: This is no worse than before, since reflectors were backed by unbounded DeltaFIFOs, but
	// we should try to do something better.
	pendingNotifications buffer.RingGrowing
	// pendingNotificationsLength tracks pendingNotifications size and is only mutated by pop().
	// run() reads this to decide when to enable expensive time tracing.
	pendingNotificationsLength atomic.Int64

Three goroutines per listener, and they are started here:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1299
		for listener := range p.listeners {
			p.wg.Start(listener.watchSynced)
			p.wg.Start(listener.run)
			p.wg.Start(listener.pop)
		}

The three roles:

distribute (in the sharedProcessor) is the producer. It holds a read lock over the listener set and calls listener.add(obj) for each listener. add does an unbuffered channel send to addCh.
pop is the pump. It reads from addCh, and hands notifications to nextCh (also unbuffered) for run to consume. When run is behind, pop spills into the ring buffer.
run is the consumer. It reads from nextCh and calls your OnAdd / OnUpdate / OnDelete synchronously, on the goroutine that owns the listener.

That last point is the crux. Your handler runs on the listener’s own goroutine. If your handler blocks for 30 seconds doing API calls, that listener stops consuming, pop starts filling the ring, and the ring never shrinks.

The ring buffer, and the comment that admits it

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1390
	// pendingNotifications is an unbounded ring buffer that holds all notifications not yet distributed.
	// There is one per listener, but a failing/stalled listener will have infinite pendingNotifications
	// added until we OOM.
	// TODO: This is no worse than before, since reflectors were backed by unbounded DeltaFIFOs, but
	// we should try to do something better.
	pendingNotifications buffer.RingGrowing

Three sentences, and the middle one is the entire failure mode. The “TODO: This is no worse than before” is a defence of the status quo, not an endorsement: the informer’s DeltaFIFO was also unbounded, so this change did not make things worse, but it did not make them better either.

The initial size is 1024:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:500
const (
	// syncedPollPeriod controls how often you look at the status of your sync funcs
	syncedPollPeriod = 100 * time.Millisecond

	// initialBufferSize is the initial number of event notifications that can be buffered.
	initialBufferSize = 1024
)

and it is passed in when the listener is created:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1054
	listener := newProcessListener(logger, handler, resyncPeriod, determineResyncPeriod(logger, resyncPeriod, s.resyncCheckPeriod), s.clock.Now(), initialBufferSize, s.HasSyncedChecker())

1024 is the starting size, not a limit. The ring doubles from there.

pop(): the one-item lookahead

The pump is the interesting piece of code, because it contains an optimisation that is easy to misread as a bug:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1468
func (p *processorListener) pop() {
	defer utilruntime.HandleCrashWithLogger(p.logger)
	defer close(p.nextCh) // Tell .run() to stop
	defer close(p.done) // Tell .watchSynced() to stop

	var nextCh chan<- interface{}
	var notification interface{}
	for {
		select {
		case nextCh <- notification:
			// Notification dispatched
			var ok bool
			notification, ok = p.pendingNotifications.ReadOne()
			if ok {
				p.pendingNotificationsLength.Add(-1)
			} else { // Nothing to pop
				nextCh = nil // Disable this select case
			}
		case notificationToAdd, ok := <-p.addCh:
			if !ok {
				return
			}
			if notification == nil { // No notification to pop (and pendingNotifications is empty)
				// Optimize the case - skip adding to pendingNotifications
				notification = notificationToAdd
				nextCh = p.nextCh
			} else { // There is already a notification waiting to be dispatched
				p.pendingNotifications.WriteOne(notificationToAdd)
				p.pendingNotificationsLength.Add(1)
			}
		}
	}
}

Read the local variable notification. It holds one notification that has been received from addCh but not yet handed to run. It is the “hot” item.

When addCh delivers a notification and notification is nil (no hot item), the new notification becomes the hot item. Nothing is written to the ring buffer. The comment says it: “Optimize the case - skip adding to pendingNotifications”.
When addCh delivers a notification and notification is non-nil (there is already a hot item), the old hot item is pushed into the ring buffer and the new one becomes hot.

So the ring buffer only receives an item when a second notification arrives before the first has been delivered. In the common case — events arrive slower than the handler consumes them — the ring buffer is never touched at all. That is the one-item lookahead, and it is the difference between “unbounded buffer” and “unbounded buffer that is usually empty”.

nextCh = nil disables that select case when the ring is empty, so pop blocks on addCh alone instead of spinning on a send to a channel nobody is reading.

The length counter exists only to decide when to turn on expensive tracing, and it is only mutated by pop:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1520
			pendingNotifications := p.pendingNotificationsLength.Load()
			if pendingNotifications > initialBufferSize {
				trace := utiltrace.New("processorListener handler",
					utiltrace.Field{Key: "handler", Value: p.handlerName},
					utiltrace.Field{Key: "pendingNotifications", Value: pendingNotifications},
				)
				defer trace.LogIfLong(100 * time.Millisecond)
			}

Note the threshold: the trace only turns on once the backlog exceeds the initial buffer size of 1024. Below that, a slow handler is invisible.

The ring doubles

The buffer itself is a classic growing ring, and the growth is a straight doubling with a careful wrap-around copy:

 go
 
 
// vendor/k8s.io/utils/buffer/ring_growing.go:79
// WriteOne adds an item to the end of the buffer, growing it if it is full.
func (r *TypedRingGrowing[T]) WriteOne(data T) {
	if r.readable == r.n {
		// Time to grow
		newN := r.n * 2
		if newN == 0 {
			newN = defaultRingSize
		}
		newData := make([]T, newN)
		to := r.beg + r.readable
		if to <= r.n {
			copy(newData, r.data[r.beg:to])
		} else {
			copied := copy(newData, r.data[r.beg:])
			copy(newData[copied:], r.data[:(to%r.n)])
		}
		r.beg = 0
		r.data = newData
		r.n = newN
	}
	r.data[(r.readable+r.beg)%r.n] = data
	r.readable++
}

Growth is amortised O(1): the copy happens only on the powers of two. The wrap-around branch handles the case where the readable region straddles the end of the backing array, which is the case the naive implementation gets wrong.

The default ring size is 16, but it is only used when the initial size is zero:

 go
 
 
// vendor/k8s.io/utils/buffer/ring_growing.go:19
// defaultRingSize defines the default ring size if not specified
const defaultRingSize = 16

In the informer the initial size is 1024, so the sequence of capacities is 1024, 2048, 4096, … Each doubling copies the whole backlog, so the total copy cost over N insertions is O(N) amortised, but the peak memory is 2x the backlog.

And the GC trick again, because this is the same class of bug:

 go
 
 
// vendor/k8s.io/utils/buffer/ring_growing.go:61
// ReadOne reads (consumes) first item from the buffer if it is available, otherwise returns false.
func (r *TypedRingGrowing[T]) ReadOne() (data T, ok bool) {
	if r.readable == 0 {
		return
	}
	r.readable--
	element := r.data[r.beg]
	var zero T
	r.data[r.beg] = zero // Remove reference to the object to help GC
	if r.beg == r.n-1 {
		// Was the last element
		r.beg = 0
	} else {
		r.beg++
	}
	return element, true
}

Without r.data[r.beg] = zero, a consumed Pod object would be pinned by the ring’s backing array until the ring grew past it — which, for a ring that never grows, is forever.

The DeltaFIFO runs the handler under the lock

Here is the contrast that makes the whole design legible. The DeltaFIFO that sits between the reflector and the distributor also runs user code, and it runs it under the queue’s own lock:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:550
// Pop blocks until the queue has some items, and then returns one. If
// multiple items are ready, they are returned in the order in which they were
// added/updated. The item is removed from the queue (and the store) before it
// is returned, so if you don't successfully process it, you need to add it back
// with AddIfNotPresent().
// process function is called under lock, so it is safe to update data structures
// in it that need to be in sync with the queue (e.g. knownKeys).
// process should avoid expensive I/O operation so that other queue operations, i.e.
// Add() and Get(), won't be blocked for too long.

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:591
		// Only log traces if the queue depth is greater than 10 and it takes more than
		// 100 milliseconds to process one item from the queue.
		// Queue depth never goes high because processing an item is locking the queue,
		// and new items can't be added until processing finish.
		// https://github.com/kubernetes/kubernetes/issues/103789
		if depth > 10 {
			trace := utiltrace.New("DeltaFIFO Pop Process",
				utiltrace.Field{Key: "ID", Value: id},
				utiltrace.Field{Key: "Depth", Value: depth},
				utiltrace.Field{Key: "Reason", Value: "slow event handlers blocking the queue"})
			defer trace.LogIfLong(100 * time.Millisecond)
		}
		err := process(item, isInInitialList)

The trace fires only when both conditions hold: the queue depth is greater than 10, and the processing takes longer than 100 milliseconds. And the reason is spelled out in the trace itself: “slow event handlers blocking the queue”.

“Queue depth never goes high because processing an item is locking the queue.” So the DeltaFIFO applies backpressure: a slow handler blocks Add, which blocks the reflector’s writes, which slows the watch. Memory stays bounded; the cost is latency.

The listener path is the opposite choice. There, distribute must not block: one slow listener must not stall the whole shared informer and every other listener on it. So the slow listener gets an unbounded buffer and eats the memory itself.

Two different failure modes, chosen deliberately:

DeltaFIFO: backpressure. Bounded memory, added latency.
processorListener: unbounded buffer. Bounded latency, unbounded memory.

Kubernetes’ stated priority is that one controller’s slow handler must not delay the apiserver watch that feeds every other controller. So the listener path eats memory.

The dedup that only collapses adjacent tombstones

The DeltaFIFO does try to reduce work, but its dedup is much narrower than people assume. It only ever compares the last two deltas:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:441
// re-listing and watching can deliver the same update multiple times in any
// order. This will combine the most recent two deltas if they are the same.
func dedupDeltas(deltas Deltas) Deltas {
	n := len(deltas)
	if n < 2 {
		return deltas
	}
	a := &deltas[n-1]
	b := &deltas[n-2]
	if out := isDup(a, b); out != nil {
		deltas[n-2] = *out
		return deltas[:n-1]
	}
	return deltas
}

“the most recent two deltas” — adjacent only. A MODIFIED, MODIFIED, MODIFIED sequence for the same key is not collapsed, because the dedup only looks at deltas n-1 and n-2 of the list, and the list is per-key, and isDup only knows how to handle deletions:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:468
// keep the one with the most information if both are deletions.
func isDeletionDup(a, b *Delta) *Delta {
	if b.Type != Deleted || a.Type != Deleted {
		return nil
	}
	// Do more sophisticated checks, or is this sufficient?
	if _, ok := b.Object.(DeletedFinalStateUnknown); ok {
		return a
	}
	return b
}

Two deletions of the same key collapse into one. And the survivor is the richer one: if the older delta carried a DeletedFinalStateUnknown tombstone — which contains the last known state of the object — then the tombstone is kept, because it carries strictly more information than a bare deletion event. Otherwise the newer, plainer deletion wins.

So the dedup does exactly one thing: it stops a delete-then-delete pair from being delivered twice, and it prefers the version that knows what the object used to look like. It is not a general-purpose coalescer.

The trade-off

What the unbounded buffer buys: a slow handler never blocks the reflector, never blocks the apiserver watch, and never blocks the other handlers on the same informer. The cost of a slow handler is paid by the slow handler.

What it costs:

Memory is unbounded and the failure is a hard OOM. There is no backpressure, no drop policy, no limit. A handler that is 10x slower than the event rate will grow the ring without bound until the process is killed.
The backlog is invisible until it is large. The trace only turns on above 1024 pending notifications, and the DeltaFIFO trace only fires above depth 10 and over 100ms. A handler that is falling behind but still under 1024 buffered items produces no signal at all.
The lookahead is easy to get wrong when modifying. pop() holds a notification across a select, and the ring buffer only ever receives the older of two pending items. Any change to that logic has to preserve the invariant that exactly one item is in flight outside the ring.
The run goroutine serialises the handler. Your handler is called from a single goroutine, so a slow handler is not just slow, it is serial slow, and you cannot fix it by adding workers inside the handler without breaking the ordering guarantee that OnUpdate(old, new) gives you.

Wrapping up

The one thing to remember: the buffer between the informer and your handler is unbounded, on purpose, and the upstream comment says your handler will OOM the process if it stalls (staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1390). Your handler is not allowed to block, and it is not allowed to be slow.

The part people get wrong: assuming that a ResourceEventHandler is called on a goroutine you do not control and therefore cannot hurt anyone. It is called on the listener’s goroutine, and the ring buffer in front of it grows to absorb the difference. A handler that takes 200ms per object, on an informer seeing 1000 objects/second, is a slow memory leak with a deterministic shape: it will OOM.

What to go do about it: in every OnAdd / OnUpdate / OnDelete, enqueue the object into your own workqueue and return immediately — that is what workqueue is for, and its per-item coalescing gives you back the burst absorption you lose by returning early. While you are there, measure the handler: utiltrace only emits "processorListener handler" with a pendingNotifications field once the backlog exceeds initialBufferSize = 1024, so a trace with a pendingNotifications field is already a warning sign. If you cannot avoid blocking in the handler, the fix is to bound the queue you enqueue into and to make the informer watch a namespace or a label selector rather than the whole cluster.

Two Sets and a FIFO: The Whole Concurrency Model of the Kubernetes Workqueue

Jay Hubbert — Fri, 23 Oct 2026 10:10:00 +0000

TL;DR: The entire concurrency model of client-go’s workqueue is three fields: a FIFO queue, a dirty set, and a processing set. Add marks an item dirty and pushes it only if it is not currently being processed; Get moves it from dirty to processing; Done removes it from processing and, if it was re-added while in flight, pushes it back. The result: one item is never processed by two workers at once, and a thousand rapid Adds of the same key collapse into one or two processing runs.

Every controller in Kubernetes ends up with the same problem. Something happens in the cluster — a Pod is created, a Node goes away — and the controller has to react. But “react” means “call the API server”, and the API server has a limited capacity to be called. Worse, the same event tends to arrive in bursts: a Node’s status update touches fifty Pods at once, and a reconcile loop that re-adds the same key on every error will re-add it forever if nobody stops it.

The workqueue is the small piece of machinery that sits between “an event happened” and “the controller does something about it”. It has three properties, and the package doc states them without elaboration:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/doc.go:19
// - Fair: items processed in the order in which they are added.
// - Stingy: a single item will not be processed multiple times concurrently,
// and if an item is added multiple times before it can be processed, it
// will only be processed once.
// - Multiple consumers and producers. In particular, it is allowed for an
// item to be reenqueued while it is being processed.
// In this case it will be processed again.
// - Shutdown notifications.

“Fair”, “stingy”, and “reenqueueable”. Everything else in the package is bookkeeping for those three.

The three fields

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/queue.go:194
	// queue defines the order in which we will work on items. Every
	// element of queue should be in the dirty set and not in the
	// processing set.
	queue Queue[t]

	// dirty defines all of the items that need to be processed.
	dirty sets.Set[t]

	// Things that are currently being processed are in the processing set.
	// These things may be simultaneously in the dirty set. When we finish
	// processing something and remove it from this set, we'll check if
	// it's in the dirty set, and if so, add it to the queue.
	processing sets.Set[t]

The invariant, stated in the comment on queue: every element of the FIFO is in the dirty set and not in the processing set. A dirty item means “needs to be processed”. A processing item means “a worker has claimed it”. An item can be in both, which is precisely the state that says “it was re-added while a worker was busy with it”.

dirty and processing are sets.Set[T] from k8s.io/apimachinery/pkg/util/sets — plain Go maps with a Set interface. The FIFO is pluggable via the Queue interface, which is how you get a priority queue or a rate-limited variant.

Add: mark dirty, push once

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/queue.go:230
func (q *Typed[T]) Add(item T) {
	q.cond.L.Lock()
	defer q.cond.L.Unlock()
	if q.shuttingDown {
		return
	}
	if q.dirty.Has(item) {
		// the same item is added again before it is processed, call the Touch
		// function if the queue cares about it (for e.g, reset its priority)
		if !q.processing.Has(item) {
			q.queue.Touch(item)
		}
		return
	}

	q.metrics.add(item)

	q.dirty.Insert(item)
	if q.processing.Has(item) {
		return
	}

	q.queue.Push(item)
	q.cond.Signal()
}

Three branches, and each one is a distinct case:

Already dirty and not in flight. Return without pushing. This is the coalescing: the item is already sitting in the FIFO exactly once, and adding it again would be a duplicate. The Touch call gives priority-queue implementations a chance to boost the item, but only when it is not currently being processed — you cannot reprioritise something a worker is already holding.
Already dirty and in flight. Also return without pushing — but the item stays dirty, and it was already removed from the FIFO when the worker claimed it. This is the “re-enqueued while being processed” case.
Not dirty. Insert into dirty, and push to the FIFO — unless the item is currently being processed, in which case do not push.

Case 3’s exception is the important one. If a worker is holding item X and someone calls Add(X), the item must not go into the FIFO, because then two workers could pick up X — violating the “stingy” guarantee. Instead, X is left marked dirty, and the re-add is honoured later, by Done.

Get: dirty to processing

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/queue.go:268
func (q *Typed[T]) Get() (item T, shutdown bool) {
	q.cond.L.Lock()
	defer q.cond.L.Unlock()
	for q.queue.Len() == 0 && !q.shuttingDown {
		q.cond.Wait()
	}
	if q.queue.Len() == 0 {
		// We must be shutting down.
		return *new(T), true
	}

	item = q.queue.Pop()

	q.metrics.get(item)

	q.processing.Insert(item)
	q.dirty.Delete(item)

	return item, false
}

Get pops from the FIFO and immediately flips the item’s membership: out of dirty, into processing. That single pair of set operations is the mutual exclusion. Because the item is no longer dirty, a subsequent Add will take the “not dirty” branch, reach the q.processing.Has(item) check, and decline to push. So the item cannot be handed to a second worker. The sync.Cond handles the blocking: Get waits while the queue is empty and not shutting down, Add signals, and ShutDown broadcasts to wake every waiter.

Done: the deferred re-add.

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/queue.go:292
func (q *Typed[T]) Done(item T) {
	q.cond.L.Lock()
	defer q.cond.L.Unlock()

	q.metrics.done(item)

	q.processing.Delete(item)
	if q.dirty.Has(item) {
		q.queue.Push(item)
		q.cond.Signal()
	} else if q.processing.Len() == 0 {
		q.cond.Signal()
	}
}

Done is where the promise is kept. If the item was re-added while the worker was busy, it is still in dirty, so it goes back into the FIFO and another worker picks it up. If it was not re-added, it leaves processing and the queue is now idle — the else if branch signals the condition so that a Get blocked on an empty queue wakes up.

The end-to-end behaviour: N rapid Adds of the same key collapse into one processing run if they all land before the worker calls Get, and into two if one of them lands while the worker is holding the item. Not N. That is the coalescing property, and it is why controllers can safely Add the same key from a dozen different code paths.

The slice queue and its GC trick

The default FIFO is a plain Go slice:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/queue.go:62
func (q *queue[T]) Touch(item T) {}

func (q *queue[T]) Push(item T) {
	*q = append(*q, item)
}

func (q *queue[T]) Len() int {
	return len(*q)
}

func (q *queue[T]) Pop() (item T) {
	item = (*q)[0]

	// The underlying array still exists and reference this object, so the object will not be garbage collected.
	(*q)[0] = *new(T)
	*q = (*q)[1:]

	return item
}

The Pop implementation is the interesting bit. *q = (*q)[1:] slides the slice header forward, which means the popped slot is still inside the backing array — and the backing array still holds a reference to the object. The object cannot be collected, even though it has been popped.

The fix is one line: (*q)[0] = *new(T) writes the zero value of T over the slot, dropping the reference so the GC can reclaim the object. It costs one store per pop and saves an entire object graph from being pinned for the lifetime of the queue’s backing array.

The same trick appears in the ring buffer behind every informer handler (vendor/k8s.io/utils/buffer/ring_growing.go:69).

AddAfter: one heap, one ticker, zero timers

The delaying queue is what you use when a retry must not be immediate — AddAfter(item, 30*time.Second) after a failed reconcile. The obvious implementation is one time.Timer per pending item. The actual implementation uses one heap and one heartbeat ticker:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/delaying_queue.go:151
	ret := &delayingType[T]{
		TypedInterface: q,
		clock: clock,
		heartbeat: clock.NewTicker(maxWait),
		stopCh: make(chan struct{}),
		waitingForAddCh: make(chan *waitFor[T], 1000),
		metrics: newRetryMetrics(name, provider),
	}

	go ret.waitingLoop(logger)
	return ret
}

One goroutine (waitingLoop) owns one heap. AddAfter does not touch the heap directly; it pushes a waitFor struct onto a buffered channel of capacity 1000 and lets the waiting loop own all the ordering. Two details matter: duration <= 0 short-circuits to a plain Add — no channel, no heap, no waiting loop involvement — and the send is guarded by select on q.stopCh, so a ShutDown racing with an AddAfter cannot wedge the caller:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/delaying_queue.go:258
	q.metrics.retry()

	// immediately add things with no delay
	if duration <= 0 {
		q.Add(item)
		return
	}

	select {
	case <-q.stopCh:
		// unblock if ShutDown() is called
	case q.waitingForAddCh <- &waitFor[T]{data: item, readyAt: q.clock.Now().Add(duration)}:
	}
}

The heartbeat exists because the waiting loop is driven by events, and the heap head’s timer is the only thing that can wake it. The bound on that wait is ten seconds:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/delaying_queue.go:276
const maxWait = 10 * time.Second

The waiting loop itself pops everything that is ready, then arms a single timer for the head of the heap:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/delaying_queue.go:300
		// Add ready entries
		for waitingForQueue.Len() > 0 {
			entry := waitingForQueue.Peek().(*waitFor[T])
			if entry.readyAt.After(now) {
				break
			}

			entry = heap.Pop(waitingForQueue).(*waitFor[T])
			q.Add(entry.data)
			delete(waitingEntryByData, entry.data)
		}

		// Set up a wait for the first item's readyAt (if one exists)
		nextReadyAt := never
		if waitingForQueue.Len() > 0 {
			if nextReadyAtTimer != nil {
				nextReadyAtTimer.Stop()
			}
			entry := waitingForQueue.Peek().(*waitFor[T])
			nextReadyAtTimer = q.clock.NewTimer(entry.readyAt.Sub(now))
			nextReadyAt = nextReadyAtTimer.C()
		}

The select that follows has four arms: stopCh, the 10-second heartbeat.C(), the nextReadyAt timer for the heap head, and waitingForAddCh for new entries — with a nested drain loop that empties the channel of anything that arrived while it was being read.

The heartbeat.C() case is the insurance: every 10 seconds the loop wakes up and re-checks the heap head, even if the timer somehow did not fire. The comment says exactly why: “we know that we’ll never end up with an expired item sitting for more than 10 seconds”. The nextReadyAt timer is stopped and recreated each time the head changes, so at any moment there is exactly one timer, not one per item.

The heap is deduplicated by item:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/delaying_queue.go:360
	existing, exists := knownEntries[entry.data]
	if exists {
		if existing.readyAt.After(entry.readyAt) {
			existing.readyAt = entry.readyAt
			heap.Fix(q, existing.index)
		}

		return
	}

	heap.Push(q, entry)
	knownEntries[entry.data] = entry
}

knownEntries maps item -> *waitFor, and the waitFor struct caches its own heap index, which is what makes heap.Fix an O(log n) reposition rather than a remove-and-reinsert. And the update rule is earlier wins: if the item is already scheduled for a later time, the new, earlier time replaces it, but if it was scheduled for an earlier time, the later request is ignored.

The rate limiter: max of two

The default controller rate limiter composes two independent limiters and takes the worst answer:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/default_rate_limiters.go:48
// DefaultTypedControllerRateLimiter is a no-arg constructor for a default rate limiter for a workqueue. It has
// both overall and per-item rate limiting. The overall is a token bucket and the per-item is exponential
func DefaultTypedControllerRateLimiter[T comparable]() TypedRateLimiter[T] {
	return NewTypedMaxOfRateLimiter(
		NewTypedItemExponentialFailureRateLimiter[T](5*time.Millisecond, 1000*time.Second),
		// 10 qps, 100 bucket size. This is only for retry speed and its only the overall factor (not per item)
		&TypedBucketRateLimiter[T]{Limiter: rate.NewLimiter(rate.Limit(10), 100)},
	)
}

Per-item exponential: 5ms base, doubling per failure of that item, capped at 1000 seconds.
Overall token bucket: 10 QPS sustained, burst of 100, shared by all items.

The MaxOf composition is what makes them complementary rather than redundant. The exponential limiter stops one pathological key from hot-looping; the token bucket stops a thousand different keys from collectively overwhelming the apiserver. MaxOf is not a sum and not a min — it takes the larger delay, so whichever constraint is tighter wins:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/default_rate_limiters.go:224
	ret := time.Duration(0)
	for _, limiter := range r.limiters {
		curr := limiter.When(item)
		if curr > ret {
			ret = curr
		}
	}

	return ret
}

The doc comment is honest about the consequence: “When used with a token bucket limiter, the burst could be apparently exceeded in cases where particular items were separately delayed a longer time.”

The exponential limiter itself:

 go
 
 
// staging/src/k8s.io/client-go/util/workqueue/default_rate_limiters.go:120
	exp := r.failures[item]
	r.failures[item] = r.failures[item] + 1

	// The backoff is capped such that 'calculated' value never overflows.
	backoff := float64(r.baseDelay.Nanoseconds()) * math.Pow(2, float64(exp))
	if backoff > math.MaxInt64 {
		return r.maxDelay
	}

	calculated := time.Duration(backoff)
	if calculated > r.maxDelay {
		return r.maxDelay
	}

	return calculated
}

Note the overflow guard: math.Pow(2, exp) grows without bound, and casting a huge float to time.Duration is undefined behaviour in Go, so the math.MaxInt64 check comes first.

Forget is the other half, and it is the caller’s job: it deletes the failure count for that item out of the limiter’s map. If a controller never calls Forget on success, a key that failed once and then succeeded for a week still carries its first failure count, and its next failure jumps straight to a long delay. The standard controller pattern is defer queue.Forget(key) on the success path and queue.AddRateLimited(key) on the failure path.

The trade-off

What this design buys: per-item mutual exclusion for free, automatic coalescing of duplicate work, fairness (FIFO), and a retry mechanism that needs one goroutine and one heap rather than one timer per item.

What it costs:

The FIFO is not a priority queue. Items are processed in the order they were added. If a critical object and a batch of unimportant ones are queued together, the critical one waits.
Add is O(1) but Done is not free. Done does a set lookup and possibly a Push, all under the same mutex as Add and Get. The queue is a contention point when many workers hammer it.
The rate limiter’s map grows. failures is keyed by item and only shrinks on Forget. A controller that requeues a new key on every failure without ever calling Forget leaks map entries — the delay is capped, but the memory is not.
The delaying queue’s channel is a bottleneck. waitingForAddCh has capacity 1000; beyond that, AddAfter blocks until the waiting loop drains it, and the waiting loop is a single goroutine.
Coalescing loses information. If you need to know that an event happened five times, the workqueue cannot tell you.

Wrapping up

The one thing to remember: the queue, the dirty set and the processing set are the whole model. Add marks dirty, Get moves dirty to processing, Done moves processing back to the queue if the item was re-added. Everything else — the heap, the token bucket, the heartbeat ticker — is an implementation detail layered on top of those three fields.

The part people get wrong: assuming that calling Add twice means the handler runs twice. It usually does not. If the key is already dirty and not in flight, the second Add is a no-op; if it is in flight, the re-add is honoured once, after Done. Controllers that need “one run per change” get that; controllers that need “one run per event” do not.

What to go do about it: call queue.Forget(key) on the success path of every processNextWorkItem, or the exponential limiter’s failure count never resets and your next transient error costs you a 1000-second delay. If AddAfter is your retry path, remember the steady-state delay is baseDelay * 2^failures capped at maxDelay — with the default DefaultTypedControllerRateLimiter that is 5ms doubling, and the overall 10 QPS / burst 100 token bucket is the constraint that actually protects the apiserver when many keys fail at once. Watch workqueue_retries_total and workqueue_queue_duration_seconds to see which side of max(exponential, token bucket) you are living on.

The Reflector Refuses to Publish a Resource Version It Hasn't Earned

Jay Hubbert — Fri, 23 Oct 2026 09:40:00 +0000

TL;DR: A watch started at resourceVersion: "" or "0" produces a burst of synthetic ADDED events that are not in resource-version order. If the reflector published the RV of that burst immediately, the next watch would resume from a point newer than some of the objects it just cached — and those objects would never be re-delivered. So the reflector keeps a flag, eventReceivedBesidesAdded, and refuses to publish the resource version until it has seen a MODIFIED, a DELETED, or a BOOKMARK. Only then does it know it is past the synthetic set.

Kubernetes is a system where every object carries a monotonically increasing metadata.resourceVersion, and the whole consistency model is built on being able to say “I have seen everything up to version N”. That claim is only true if the version you publish is one you actually earned.

There is exactly one place in client-go where that claim can go wrong, and it is subtle enough that it took a dedicated flag, a boolean threaded through a callback, and a comment to keep it from being “simplified” away later.

The bug this prevents

Imagine a reflector whose store holds three objects: a at resource version 10, b at version 20, c at version 30. The watch starts from resourceVersion: "" — meaning “give me everything, from whatever point you like” — and the server replies with synthetic ADDED events for the whole collection, in store order, not in resource-version order: c at rv=30, then a at rv=10, then b at rv=20.

If the reflector published resourceVersion=30 after seeing the first event, the next watch would resume from 30. Object a, last modified at version 10, would be in the cache with a stale body, and nothing would ever tell anyone. There is no future event for a — it did not change. The reflector would sit there, confidently serving a stale a forever.

The fix is to refuse to publish a version until you know you are past the synthetic burst. The synthetic events all have type ADDED. Real change events have type MODIFIED or DELETED, and a BOOKMARK is a synthetic event too but a different one — it is emitted after the initial set, in strict resource-version order. So the reflector counts “did I see anything that is not an ADDED?”, and only then publishes.

A reflector can start a watch in one of two states, and the difference is the whole bug:

From a known resource version (non-empty, not "0"). The server replays events from that exact point. Every event is a real change event, in order. The resource version of any event received is immediately safe to publish.
From "" or "0". The server replays the current contents of the collection as synthetic ADDED events, in whatever order the storage layer produced them. The resource version of those events is not a safe resume point.

The reflector encodes this as a single boolean, set before the watch is even opened. The flag starts true; the line below turns it off when the resume token is "" or "0":

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:599
			if options.ResourceVersion == "" || options.ResourceVersion == "0" {
				// if we're starting the watch at a resource version that will get synthetic ADDED events in non-rv order,
				// wait until we're through that set of events before propagating the RV
				propagateRVFromStart = false
			}

"" and "0" are both dangerous, and they are dangerous for the same reason. "" means “start from wherever”. "0" means “start from any old version you have” — the reflector uses it for its first list, because “for performance reasons, initial list performed by reflector uses ‘0’ as resource version to allow it to be served from the watch cache if it is enabled”:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:1126
	if r.lastSyncResourceVersion == "" {
		// For performance reasons, initial list performed by reflector uses "0" as resource version to allow it to
		// be served from the watch cache if it is enabled.
		return "0"
	}
	return r.lastSyncResourceVersion

The callback that does the withholding

The decision is not made once, at watch-open time. It is made per event, inside a callback that the event loop invokes. The callback receives two things: the resource version of the event, and whether anything other than an ADDED has been seen so far.

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:620
		err = handleWatch(ctx, start, w, r.store, r.expectedType, r.expectedGVK, r.name, r.typeDescription,
			func(rv string, eventReceivedBesidesAdded bool) {
				// We update the resource version in the store only if we have received at least one event that is
				// not an added event, or if the resource version has been set previously. This is because we can
				// encounter 2 scenarios:
				// 1. The watch is started from a resource version specified by the LastSyncResourceVersion field.
				// In this case, we can update the resource version in the store without worrying about it being
				// out of order since we will not receive any synthetic added events for resources that may be
				// out of order.
				// 2. The watch is started when the LastSyncResourceVersion field is empty. In this case, we may not
				// update the LastSyncResourceVersion until we receive at least one event that is not an added
				// event, since that is the only way to ensure that the watch has exited the initial list phase.
				if propagateRVFromStart || eventReceivedBesidesAdded {
					r.setLastSyncResourceVersion(rv)
					if rvu, ok := r.store.(ResourceVersionUpdater); ok {
						rvu.UpdateResourceVersion(rv)
					}
				}
			},
			r.clock, resyncerrc)

Two conditions, ORed. propagateRVFromStart means “this watch started from a known-good point, so any event’s RV is fine”. eventReceivedBesidesAdded means “we have now seen a real event, so the synthetic burst is over and this event’s RV is fine”.

The two updates matter differently. setLastSyncResourceVersion is the reflector’s own resume token — what the next watch will start from. rvu.UpdateResourceVersion(rv) pushes it into the store, which is what cache.Store consumers read to know the cache’s freshness. Both are withheld together; publishing to one and not the other would give you a cache that claims a version the reflector does not actually hold.

Where `eventReceivedBesidesAdded` comes from

The event loop is shared by the watch-list and the ordinary watch path. The flag is set in exactly three branches — MODIFIED, DELETED and BOOKMARK — and the ADDED branch is conspicuously absent:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:1043
			case watch.Modified:
				eventReceivedBesidesAdded = true
				err := store.Update(event.Object)
				if err != nil {
					utilruntime.HandleErrorWithContext(ctx, err, "Unable to update watch event object to store", "reflector", name, "object", event.Object)
				}
			case watch.Deleted:
				// TODO: Will any consumers need access to the "last known
				// state", which is passed in event.Object? If so, may need
				// to change this.
				eventReceivedBesidesAdded = true
				err := store.Delete(event.Object)
				if err != nil {
					utilruntime.HandleErrorWithContext(ctx, err, "Unable to delete watch event object from store", "reflector", name, "object", event.Object)
				}
			case watch.Bookmark:
				// A `Bookmark` means watch has synced here, just update the resourceVersion
				eventReceivedBesidesAdded = true
				if meta.GetAnnotations()[metav1.InitialEventsAnnotationKey] == "true" {
					watchListBookmarkReceived = true
				}

A MODIFIED means “this object changed”, a DELETED means “this object changed (and is gone)”, and a BOOKMARK is emitted by the server strictly in resource-version order — which is exactly the guarantee we need to know we are past the unordered synthetic set.

One more subtlety: the resource version is read from the event object, not from the event envelope. Bookmarks carry their own resource version on the object, so the bookmark’s RV is the server’s current position, which is safe to publish.

And after every event, the loop unconditionally calls back with the current flag:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:1074
			// when eventReceivedBesidesAdded is true, that indicates we are definitely past any initial synthetic Added events
			setLastSyncResourceVersion(resourceVersion, eventReceivedBesidesAdded)
			eventCount++

Note what this does not do: it does not skip the callback when the flag is false. It calls the callback with eventReceivedBesidesAdded=false, and the callback decides. The decision logic lives in one place, next to the explanation of why it exists.

The watch-list variant. The watch-list path uses the same machinery but a different callback, because it needs the opposite behaviour. During the initial burst the resource version is meaningless (the objects are a snapshot, not a sequence), so it is only recorded once a non-ADDED event proves the burst is over:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:870
		watchListBookmarkReceived, err := handleListWatch(ctx, start, w, temporaryStore, r.expectedType, r.expectedGVK, r.name, r.typeDescription,
			func(rv string, eventReceivedBesidesAdded bool) {
				if eventReceivedBesidesAdded {
					resourceVersion = rv
				}
			},
			r.clock, make(chan error))

handleListWatch is handleAnyWatch with exitOnWatchListBookmarkReceived set to true; handleWatch is the same function with it set to false. The two paths share one event loop, one store-writing code path, one resource-version extraction. The only difference is whether the loop returns when the end-of-initial-events bookmark shows up.

A watch that dies instantly

If the watch channel closes in under a second with zero events, the reflector returns a distinctive error rather than treating it as a normal end-of-watch:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:1089
	watchDuration := clock.Since(start)
	if watchDuration < 1*time.Second && eventCount == 0 {
		return watchListBookmarkReceived, &VeryShortWatchError{Name: name}
	}

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:1294
// VeryShortWatchError is returned when the watch result channel is closed
// within one second, without having sent any events.
type VeryShortWatchError struct {
	// Name of the Reflector
	Name string
}

Why does this matter? Because a reflector that watches from "" and gets nothing has learned nothing about the state of the collection. It must not conclude “the cache is up to date at version X”. The VeryShortWatchError is a loud signal that the watch was useless, and it is what keeps the backoff machinery from treating a hard failure as a clean run.

The backoff arithmetic

The reflector does not spin when the apiserver is unhealthy. It has a delay function built from a standard exponential backoff with full jitter:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:317
	backoff := options.Backoff
	if backoff == nil {
		backoff = &wait.Backoff{
			Duration: defaultBackoffInit,
			Cap: defaultBackoffMax,
			Steps: int(math.Ceil(float64(defaultBackoffMax) / float64(defaultBackoffInit))),
			Factor: defaultBackoffFactor,
			Jitter: defaultBackoffJitter,
		}
	}

With defaultBackoffInit = 800ms, defaultBackoffMax = 30s, defaultBackoffFactor = 2.0 and defaultBackoffJitter = 1.0, the step count is ceil(30s / 800ms) = 38. The core algorithm is:

 go
 
 
// staging/src/k8s.io/apimachinery/pkg/util/wait/backoff.go:106
	steps--

	// calculate the next step's interval
	if factor != 0 {
		next = time.Duration(float64(duration) * factor)
		if cap > 0 && next > cap {
			next = cap
			steps = 0
		}
	} else {
		next = duration
	}

	// add jitter for this step
	if jitter > 0 {
		duration = Jitter(duration, jitter)
	}

	return duration, next, steps

and the jitter is uniform over [duration, duration + jitter*duration) — wait.Jitter adds rand.Float64() * maxFactor * duration to the base duration, so with maxFactor = 1.0 the sleep is drawn uniformly from [d, 2d).

Walk the arithmetic. The nominal durations are 800ms, 1.6s, 3.2s, 6.4s, 12.8s, 25.6s, 30s (capped) — and each actual sleep is drawn uniformly from [d, 2d). The cap is hit at 51.2s > 30s, so next is clamped to 30s and steps is set to 0. From then on, steps < 1, so every sleep is Jitter(30s, 1.0) — uniform in [30s, 60s). That is the steady state the source comment describes:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:59
	// We used to make the call every 1sec (1 QPS), the goal here is to achieve ~98% traffic reduction when
	// API server is not healthy. With these parameters, backoff will stop at [30,60) sec interval which is
	// 0.22 QPS.

One request every 30 to 60 seconds is roughly 0.02 QPS, so the comment’s “0.22” is off by a factor of ten; the interval it names, not the decimal, is the part to trust. The 98% figure is the real target: 1 QPS down to about 0.02 QPS is a 98% reduction, and it is what protects an overloaded apiserver from being killed by its own clients.

The other half of the design is the reset. The backoff only resets if the apiserver has been healthy for a while:

 go
 
 
// staging/src/k8s.io/apimachinery/pkg/util/wait/backoff.go:296
	switch {
	case b.resetInterval == 0:
		b.backoff = b.initialBackoff
	case b.clock.Now().Sub(b.lastStart) > b.resetInterval:
		b.backoff = b.initialBackoff
		b.lastStart = b.clock.Now()
	}
	return b.backoff.Step()
}

defaultBackoffReset = 2 * time.Minute, and the comment says why: “If we don’t backoff for 2min, assume API server is healthy and we reset the backoff.” Without this, a reflector that hit a ten-minute apiserver outage would sit at a 30-60s interval forever afterwards, even though the apiserver was fine.

The whole thing is driven from one place:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:426
	// Until runs the loop immediately (immediate=true) and resets the backoff timer after each
	// successful iteration (sliding=true). See backoff constants at top of file for generalized QPS targets (~0.22 QPS).
	_ = r.delayHandler.Until(ctx, true, true, func(ctx context.Context) (bool, error) {
		if err := r.ListAndWatchWithContext(ctx); err != nil {
			r.watchErrorHandler(ctx, r, err)
		}
		return false, nil
	})

immediate=true means the first attempt happens without delay. sliding=true means the backoff timer is reset after the function returns, so the delay is measured from the end of the previous attempt, not its start.

The timer that had to be stopped

There is one more piece of discipline in this file that has nothing to do with resource versions but everything to do with not leaking. The resync timer is created inside a helper that returns two values — the channel and a cleanup function:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:443
// resyncChan returns a channel which will receive something when a resync is
// required, and a cleanup function.
func (r *Reflector) resyncChan() (<-chan time.Time, func() bool) {
	if r.resyncPeriod == 0 {
		// nothing will ever be sent down this channel
		return nil, func() bool { return false }
	}
	// The cleanup function is required: imagine the scenario where watches
	// always fail so we end up listing frequently. Then, if we don't
	// manually stop the timer, we could end up with many timers active
	// concurrently.
	t := r.clock.NewTimer(r.resyncPeriod)
	return t.C(), t.Stop
}

Read that comment twice, because it describes a genuine bug shape. If watches keep failing, the reflector re-lists, and each list cycle creates a new resync timer. With a resync period of an hour and a failing apiserver, you accumulate one live timer per retry cycle.

So the caller has to stop the previous timer before making a new one:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:516
	resyncCh, cleanup := r.resyncChan()
	defer func() {
		cleanup() // Call the last one written into cleanup
	}()
	for {
		select {
		case <-resyncCh:
		case <-ctx.Done():
			return
		}
		if r.ShouldResync == nil || r.ShouldResync() {
			logger.V(4).Info("Forcing resync", "reflector", r.name)
			if err := r.store.Resync(); err != nil {
				resyncerrc <- err
				return
			}
		}
		cleanup()
		resyncCh, cleanup = r.resyncChan()
	}
}

cleanup() is called immediately before each new timer is created, and the deferred call handles the last one. Note also the resyncerrc channel: it has capacity 1 and is buffered precisely so that a resync error cannot block the resync goroutine forever.

The trade-off

What the withholding buys: a reflector that never resumes from a point it has not earned. The cache and the resume token always agree, and the classic “silently stale object” bug is structurally impossible.

What it costs:

The first watch after a cold start publishes nothing. Until a real event arrives, LastSyncResourceVersion() stays where it was. For an informer whose collection never changes, the resume token never advances — correct, but it means a “resumed” reflector can sit at an old version.
One extra field of state threaded through a callback. The eventReceivedBesidesAdded bool parameter exists only so the callback can make the publish/withhold decision in one place.
The "" case is common, not exotic. Any reflector that starts without a resume token hits the unsafe path, and the safe behaviour is to hold.

Wrapping up

The one thing to remember: the reflector publishes a resource version only when it knows the synthetic ADDED burst is over. The flag is eventReceivedBesidesAdded, set by MODIFIED, DELETED or BOOKMARK in staging/src/k8s.io/client-go/tools/cache/reflector.go:1044, and consulted in the callback at reflector.go:632. Everything else in the file is bookkeeping around that one decision.

The part people get wrong: assuming resourceVersion means “the version of the last object I received”. For a watch that started at "" or "0", the last object received is not a resume point at all — the server ordered that burst by storage order, not by version. Publishing it is how you get an informer that is confidently wrong about one object, forever.

What to go do about it: if you maintain your own watch loop instead of using cache.NewReflector, do not set LastSyncResourceVersion from the first event you receive — wait for a MODIFIED, a DELETED, or a BOOKMARK, exactly as handleAnyWatch does. On the server side, watch the reflector metrics for ListAndWatch calls: the backoff parameters (defaultBackoffInit = 800ms, defaultBackoffMax = 30s, defaultBackoffReset = 2m) mean a reflector that cannot reach the apiserver should be retrying every 30-60 seconds, not every second. If you see a reflector retrying at 1 QPS against a broken apiserver, something outside the standard wait.Backoff path is driving it.

One Stream Instead of Two: The Watch List That Killed LIST + WATCH

Jay Hubbert — Fri, 23 Oct 2026 09:10:00 +0000

TL;DR: The classic reflector did two things over two connections: a LIST (possibly N paginated round trips) and then a WATCH. The watch-list path (KEP-3157) collapses both into a single WATCH request carrying sendInitialEvents=true. The apiserver replies with synthetic ADDED events for the entire object set, then a synthetic BOOKMARK carrying the resource version, and then keeps the same stream open for live events. The reflector swaps its store, publishes the resource version, and hands the same watch.Interface to the caller — no reconnect, no gap.

The classic Kubernetes read path has always been two round trips. First you LIST, to get a snapshot of the world and a resourceVersion that names a point in etcd’s history. Then you WATCH from that resourceVersion, to learn about changes. It works, and it has worked for a decade, but it is expensive in ways that only became obvious at scale.

The LIST is paginated, by default 500 items per request (staging/src/k8s.io/client-go/tools/pager/pager.go:31). A namespace with 200 000 pods is 400 requests. Each one is served straight out of etcd, not out of the apiserver’s watch cache, because paginated lists with a continue token are explicitly forbidden from being cached. The apiserver pays for the deserialization of every one of those objects, holds the whole list in memory while it paginates, and the client holds the whole list in memory too. Then the client opens a second connection for the WATCH, and from that moment on there are two live connections per informer instead of one. The watch-list design (KEP-3157) does not optimise this. It removes it.

The request that starts the whole thing

The reflector decides once, at construction time, whether it is allowed to try the watch-list path:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:361
	r.useWatchList = clientfeatures.FeatureGates().Enabled(clientfeatures.WatchListClient)
	if r.useWatchList && watchlist.DoesClientNotSupportWatchListSemantics(lw) {
		r.logger.V(2).Info(
			"The client used to build this informer/reflector doesn't support WatchList semantics. The feature will be disabled. This is expected in unit tests but not in production. For details, see the documentation of watchlist.DoesClientNotSupportWatchListSemantics().",
			"feature", clientfeatures.WatchListClient,
		)
		r.useWatchList = false
	}

WatchListClient is a client-go feature gate, defaulting to false at v1.30 and true at v1.35 (staging/src/k8s.io/client-go/features/known_features.go:111).

Because client-go’s own feature gates are env-var driven, you can flip it without a recompile: KUBE_FEATURE_WatchListClient=true in the environment of your controller. The gate is read exactly once, on the first Enabled() call, so setting it after the first informer is constructed does nothing.

If the reflector is going to try the watch-list path, the request itself is built inside watchList, and it is a watch request with one extra flag:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:847
		resourceVersion = ""
		lastKnownRV := r.rewatchResourceVersion()
		temporaryStore = NewStore(DeletionHandlingMetaNamespaceKeyFunc, storeOpts...)
		// TODO(#115478): large "list", slow clients, slow network, p&f
		// might slow down streaming and eventually fail.
		// maybe in such a case we should retry with an increased timeout?
		timeoutSeconds := int64(r.minWatchTimeout.Seconds() + rand.Float64()*(r.maxWatchTimeout.Seconds()-r.minWatchTimeout.Seconds()))
		options := metav1.ListOptions{
			ResourceVersion: lastKnownRV,
			AllowWatchBookmarks: true,
			SendInitialEvents: ptr.To(true),
			ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan,
			TimeoutSeconds: &timeoutSeconds,
		}

Three of those fields are load-bearing. SendInitialEvents: ptr.To(true) is the whole point — it tells the server “please start the stream by sending me the current contents as ADDED events”. ResourceVersionMatch: NotOlderThan says “give me data at least as new as this version”, which is what makes the stream consistent even if the client was disconnected for a while. And AllowWatchBookmarks: true is what buys the end-of-initial-events marker.

The server-side check for “is this a watch-list request” is a one-liner, and it requires both flags — bookmarks must be enabled or there is no way to signal the end of the initial burst:

 go
 
 
// staging/src/k8s.io/apiserver/pkg/endpoints/handlers/get.go:328
// isListWatchRequest is mirrored in staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go
func isListWatchRequest(opts metainternalversion.ListOptions) bool {
	return opts.SendInitialEvents != nil && *opts.SendInitialEvents && opts.AllowWatchBookmarks
}

The server’s answer: ADDED, ADDED, ADDED, bookmark

The response is a stream of watch.Events. Every object in the collection comes back as a synthetic ADDED, and then the server sends a synthetic BOOKMARK to mark the boundary. The bookmark is not just a bookmark — it carries an annotation that says “initial events are over”:

 go
 
 
// staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go:507
	// InitialEventsAnnotationKey the name of the key
	// under which an annotation marking the end of
	// a watchlist stream is stored.
	//
	// The annotation is added to a "Bookmark" event.
	InitialEventsAnnotationKey = "k8s.io/initial-events-end"

The reflector’s event loop is shared between the watch-list path and the ordinary watch path — one function, handleAnyWatch, with a flag telling it whether to exit when the end-of-initial-events bookmark arrives:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:1058
			case watch.Bookmark:
				// A `Bookmark` means watch has synced here, just update the resourceVersion
				eventReceivedBesidesAdded = true
				if meta.GetAnnotations()[metav1.InitialEventsAnnotationKey] == "true" {
					watchListBookmarkReceived = true
				}

And here is the hand-off — the moment the bookmark arrives, the loop stops reading, and crucially it does not stop the watcher:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:1074
			// when eventReceivedBesidesAdded is true, that indicates we are definitely past any initial synthetic Added events
			setLastSyncResourceVersion(resourceVersion, eventReceivedBesidesAdded)
			eventCount++
			if exitOnWatchListBookmarkReceived && watchListBookmarkReceived {
				stopWatcher = false
				watchDuration := clock.Since(start)
				klog.FromContext(ctx).V(4).Info("Exiting watch because received the bookmark that marks the end of initial events stream", "reflector", name, "totalItems", eventCount, "duration", watchDuration)
				return watchListBookmarkReceived, nil
			}

stopWatcher = false is the entire trick. The deferred w.Stop() in handleAnyWatch is skipped, so the HTTP connection stays open, and the watch.Interface value is returned up the stack to watchList, which returns it to ListAndWatchWithContext, which passes it to watchWithResync -> watch. The ordinary watch loop finds w != nil, skips opening a new watch entirely, and starts consuming the stream that was already open. Zero additional connections. Zero reconnect latency. Zero window in which a change could be missed between the snapshot and the live stream — because there was no window; there was only ever one stream.

The fallback, and why `w = nil` matters

If anything goes wrong — the server is too old, the feature is disabled server-side, the request 400s — the reflector must not be left holding a half-broken stream. ListAndWatchWithContext is where that is decided:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:483
	if r.useWatchList {
		w, err = r.watchList(ctx)
		if w == nil && err == nil {
			// stopCh was closed
			return nil
		}
		if err != nil {
			logger.V(4).Info(
				"Data couldn't be fetched in watchlist mode. Falling back to regular list. This is expected if watchlist is not supported or disabled in kube-apiserver.",
				"err", err,
			)
			fallbackToList = true
			// ensure that we won't accidentally pass some garbage down the watch.
			w = nil
		}
	}

	if fallbackToList {
		err = r.list(ctx)
		if err != nil {
			return err
		}
	}

	logger.V(2).Info("Caches populated", "type", r.typeDescription, "reflector", r.name)
	return r.watchWithResync(ctx, w)
}

The w = nil line is the subtle one. watchList can return a non-nil w together with an error: the stream opened fine, the server started streaming, and then something failed mid-stream. If that w were passed down to watch(), the reflector would try to consume a stream that is already broken, and would treat its contents as a complete snapshot. Nil-ing it means the fallback path is the only path, and the classic LIST + WATCH runs from scratch with a known-good starting point. The comment says exactly this: “ensure that we won’t accidentally pass some garbage down the watch.”

Note also that the fallback is silent at V(4). Every informer in every controller that talks to an old apiserver logs this at high verbosity and carries on.

The snapshot swap

The watch-list path cannot write into the live store incrementally — the whole point is that the store should be replaced atomically once the snapshot is complete. So the objects land in a temporary store first:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:893
	initTrace.Step("Objects streamed", trace.Field{Key: "count", Value: len(temporaryStore.List())})
	r.setIsLastSyncResourceVersionUnavailable(false)

	// we utilize the temporaryStore to ensure independence from the current store implementation.
	// as of today, the store is implemented as a queue and will be drained by the higher-level
	// component as soon as it finishes replacing the content.
	checkWatchListDataConsistencyIfRequested(ctx, r.name, resourceVersion, r.listerWatcher.ListWithContext, transformer, temporaryStore.List)

	if err := r.store.Replace(temporaryStore.List(), resourceVersion); err != nil {
		return nil, fmt.Errorf("unable to sync watch-list result: %w", err)
	}
	initTrace.Step("SyncWith done")
	r.setLastSyncResourceVersion(resourceVersion)

	return w, nil

The checkWatchListDataConsistencyIfRequested call is interesting: it is a CI-only detector, gated on KUBE_WATCHLIST_INCONSISTENCY_DETECTOR having been set at process start, that re-LISTs the same collection through the ordinary path and panics if the two disagree. The upstream comment says so: “The consistency check is meant to be enforced only in the CI, not in production.”

The temporary store is a plain Store keyed with DeletionHandlingMetaNamespaceKeyFunc, so tombstones are resolved on the way in rather than leaking into the informer as DeletedFinalStateUnknown.

The timeout is randomised, and it has to be

Watch timeouts exist so that a dead connection is eventually noticed. But if every reflector in the cluster uses the same timeout, they all reconnect at the same instant — a thundering herd against the apiserver, and a periodic spike in watch_cache churn. So the timeout is jittered, uniformly, in [minWatchTimeout, maxWatchTimeout):

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:54
var (
	// We try to spread the load on apiserver by setting timeouts for
	// watch requests - it is random in [minWatchTimeout, 2*minWatchTimeout].
	defaultMinWatchTimeout = 5 * time.Minute
	defaultMaxWatchTimeout = 2 * defaultMinWatchTimeout

(The same var block also holds the reflector’s backoff constants: defaultBackoffInit = 800ms, defaultBackoffMax = 30s, defaultBackoffReset = 2m, defaultBackoffFactor = 2.0, defaultBackoffJitter = 1.0.)

Five minutes minimum, ten minutes maximum. The minWatchTimeout is deliberately floored: NewReflectorWithOptions only honours a caller-supplied value above the default, because “values lower than 5m will not be honored to avoid negative performance impact on controlplane”. A watch that reconnects every thirty seconds is a lot of requests for a control plane that is already the busiest thing in the cluster.

The randomisation happens at request-construction time, and it happens twice — once in the watch-list path, once in the ordinary watch path:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:588
			timeoutSeconds := int64(r.minWatchTimeout.Seconds() + rand.Float64()*(r.maxWatchTimeout.Seconds()-r.minWatchTimeout.Seconds()))
			options := metav1.ListOptions{
				ResourceVersion: r.LastSyncResourceVersion(),
				// We want to avoid situations of hanging watchers. Stop any watchers that do not
				// receive any events within the timeout window.
				TimeoutSeconds: &timeoutSeconds,
				// To reduce load on kube-apiserver on watch restarts, you may enable watch bookmarks.
				// Reflector doesn't assume bookmarks are returned at all (if the server do not support
				// watch bookmarks, it will ignore this field).
				AllowWatchBookmarks: true,
			}

Because the range is [300s, 600s), the expected number of reconnects per reflector per hour is 1 - 2, and they are spread uniformly rather than synchronised.

The watch-list path removes the LIST entirely on the happy path. But the fallback path still exists, and the fallback path still has to decide whether to paginate. And here the reflector does something that looks like a bug and is not: it deliberately switches pagination off in some cases.

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:693
		pager := pager.New(pager.SimplePageFunc(func(opts metav1.ListOptions) (runtime.Object, error) {
			return r.listerWatcher.ListWithContext(ctx, opts)
		}))
		switch {
		case r.WatchListPageSize != 0:
			pager.PageSize = r.WatchListPageSize
		case r.paginatedResult:
			// We got a paginated result initially. Assume this resource and server honor
			// paging requests (i.e. watch cache is probably disabled) and leave the default
			// pager size set.
		case options.ResourceVersion != "" && options.ResourceVersion != "0":
			// User didn't explicitly request pagination.
			//
			// With ResourceVersion != "", we have a possibility to list from watch cache,
			// but we do that (for ResourceVersion != "0") only if Limit is unset.
			// To avoid thundering herd on etcd (e.g. on master upgrades), we explicitly
			// switch off pagination to force listing from watch cache (if enabled).
			// With the existing semantic of RV (result is at least as fresh as provided RV),
			// this is correct and doesn't lead to going back in time.
			//
			// We also don't turn off pagination for ResourceVersion="0", since watch cache
			// is ignoring Limit in that case anyway, and if watch cache is not enabled
			// we don't introduce regression.
			pager.PageSize = 0
		}

The default page size is 500 (staging/src/k8s.io/client-go/tools/pager/pager.go:31).

Setting pager.PageSize = 0 means Limit is never set on the request. The apiserver’s watch cache will then serve the whole collection from memory, in one response, without touching etcd. The alternative — a paginated list — always goes to etcd, because the apiserver refuses to serve a paginated request from the cache: the continue token pins a specific etcd revision, and the cache does not have a snapshot of that.

The trade-off is explicit in the comment: “To avoid thundering herd on etcd (e.g. on master upgrades)”. During a control-plane upgrade, every informer in the cluster re-lists at once. If those re-lists are paginated, etcd gets hit by thousands of concurrent 500-item range scans, and the apiserver gets hammered with deserialization work for objects that are about to be thrown away. If they are unpaginated, the watch cache absorbs them.

Note that ResourceVersion == "0" is excluded from this trick, and the reason is subtle: relistResourceVersion returns "0" for the very first list of a reflector, because “for performance reasons, initial list performed by reflector uses ‘0’ as resource version to allow it to be served from the watch cache”. But the watch cache ignores Limit when the requested version is 0, so pagination would be a no-op anyway — and if the watch cache is disabled, turning pagination off would turn one big slow list into one giant one. So the reflector leaves the default in place and learns from the result:

 go
 
 
// staging/src/k8s.io/client-go/tools/cache/reflector.go:760
	if options.ResourceVersion == "0" && paginatedResult {
		r.paginatedResult = true
	}

Once the reflector has seen one paginated response, it assumes the server does not have a usable watch cache for this resource and keeps paginating forever after.

There is a third option, WatchListPageSize, which forces a page size for the initial and resync watch lists. Its doc comment is a warning: “NOTE: It should be used carefully as paginated lists are always served directly from etcd, which is significantly less efficient and may lead to serious performance and scalability problems.”

The trade-off

What watch-list buys: one connection instead of two, one apiserver round trip instead of N+1, no full-collection buffering in the apiserver for paginated lists, and no reconnect window between the snapshot and the live stream. On a cluster with ten thousand informers, the difference between “one LIST + one WATCH” and “one WATCH” is roughly ten thousand fewer connections and a large reduction in etcd range-scan load.

What it costs:

The server must support it. sendInitialEvents is a relatively recent addition to the watch protocol. Older apiservers reject the request, the reflector falls back, and you have paid one failed request per reflector per retry cycle. The fallback is automatic but it is not free.
The initial burst is a burst. The server streams the entire collection as fast as it can. The TODO in the source says so: “large ’list’, slow clients, slow network, p&f might slow down streaming and eventually fail”. A controller on a bad network link can have the server’s send buffer fill up.
The snapshot lands in a temporary store first, so peak memory during startup is roughly 2x the object set, not 1x.
The end-of-initial-events bookmark is load-bearing. Without AllowWatchBookmarks, the server cannot signal the end of the burst and the client has no way to know when the snapshot is complete.

Wrapping up

The one thing to remember: sendInitialEvents=true on a WATCH request turns the snapshot and the stream into one connection. The server sends synthetic ADDED events, then a BOOKMARK annotated with k8s.io/initial-events-end, and then keeps the same stream open — the reflector’s stopWatcher = false at staging/src/k8s.io/client-go/tools/cache/reflector.go:1078 is what keeps it open.

The part people get wrong: the fallback to plain LIST is silent and the failure is at V(4), so an informer that is quietly doing the old two-connection dance looks exactly like one that is not. Watch for "Data couldn't be fetched in watchlist mode" at -v=4, and remember that a server which does not support the feature will make every reflector pay a failed request on every retry cycle.

What to go do about it: set KUBE_FEATURE_WatchListClient=true in the environment of a controller that is pinned to an apiserver older than v1.31 and confirm the reflector logs "Exiting watch because received the bookmark that marks the end of initial events stream". On the apiserver side, watch apiserver_longrunning_timestamps and the watch-cache hit rate for the affected resources. If you have informer memory pressure, remember that WatchListPageSize forces paginated lists — which are always served from etcd — so leave it unset unless you have a specific reason.

Client-Go on Opensource internals

The Informer Will OOM If You Are Too Slow: An Unbounded Ring Buffer, On Purpose

Three goroutines and two unbuffered channels

The ring buffer, and the comment that admits it

pop(): the one-item lookahead

The ring doubles

The DeltaFIFO runs the handler under the lock

The dedup that only collapses adjacent tombstones

The trade-off

Wrapping up

Two Sets and a FIFO: The Whole Concurrency Model of the Kubernetes Workqueue

The three fields

Add: mark dirty, push once

Get: dirty to processing

The slice queue and its GC trick

AddAfter: one heap, one ticker, zero timers

The rate limiter: max of two

The trade-off

Wrapping up

The Reflector Refuses to Publish a Resource Version It Hasn't Earned

The bug this prevents

The callback that does the withholding

Where eventReceivedBesidesAdded comes from

A watch that dies instantly

The backoff arithmetic

The timer that had to be stopped

The trade-off

Wrapping up

One Stream Instead of Two: The Watch List That Killed LIST + WATCH

The request that starts the whole thing

The server’s answer: ADDED, ADDED, ADDED, bookmark

The fallback, and why w = nil matters

The snapshot swap

The timeout is randomised, and it has to be

The pager trick: how to not hammer etcd

The trade-off

Wrapping up

Where `eventReceivedBesidesAdded` comes from

The fallback, and why `w = nil` matters