The Informer Will OOM If You Are Too Slow: An Unbounded Ring Buffer, On Purpose

TL;DR: A shared informer fans every event out to every registered handler, and each handler is served by a processorListener with an unbounded growing ring buffer between the distributor and the handler. If your OnUpdate takes longer than events arrive, that ring grows without limit and the process OOMs. The upstream comment says so outright: “a failing/stalled listener will have infinite pendingNotifications added until we OOM.”

This is a design decision, not a bug, and understanding why it is a decision tells you a lot about what Kubernetes optimises for. The alternative — dropping events when the buffer is full, or blocking the distributor — would both be worse in the failure mode that actually happens.

But first, the shape of the pipeline, because “where is the buffer” is the question that matters.

Three goroutines and two unbuffered channels

When you register a handler on a shared informer, you get a processorListener. The source describes the whole arrangement:

// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1361
// processorListener relays notifications from a sharedProcessor to
// one ResourceEventHandler --- using three goroutines, two unbuffered
// channels, and an unbounded ring buffer.  The `add(notification)`
// function sends the given notification to `addCh`.  One goroutine
// runs `pop()`, which pumps notifications from `addCh` to `nextCh`
// using storage in the ring buffer while `nextCh` is not keeping up.
// Another goroutine runs `run()`, which receives notifications from
// `nextCh` and synchronously invokes the appropriate handler method.
//
// The third goroutine watches the upstream "has synced" channel
// and notifies a SingleFileTracker instance. That instance then
// combines the upstream state and the processListener state to
// implement the overall "event handler has synced".
//
// processorListener also keeps track of the adjusted requested resync
// period of the listener.
type processorListener struct {
	logger      klog.Logger
	nextCh      chan interface{}
	addCh       chan interface{}
	done        chan struct{}
	runFinished chan struct{}

	handler     ResourceEventHandler
	handlerName string

	syncTracker       *synctrack.SingleFileTracker
	upstreamHasSynced DoneChecker

	// pendingNotifications is an unbounded ring buffer that holds all notifications not yet distributed.
	// There is one per listener, but a failing/stalled listener will have infinite pendingNotifications
	// added until we OOM.
	// TODO: This is no worse than before, since reflectors were backed by unbounded DeltaFIFOs, but
	// we should try to do something better.
	pendingNotifications buffer.RingGrowing
	// pendingNotificationsLength tracks pendingNotifications size and is only mutated by pop().
	// run() reads this to decide when to enable expensive time tracing.
	pendingNotificationsLength atomic.Int64

Three goroutines per listener, and they are started here:

// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1299
		for listener := range p.listeners {
			p.wg.Start(listener.watchSynced)
			p.wg.Start(listener.run)
			p.wg.Start(listener.pop)
		}

The three roles:

• distribute (in the sharedProcessor) is the producer. It holds a read lock over the listener set and calls listener.add(obj) for each listener. add does an unbuffered channel send to addCh.
• pop is the pump. It reads from addCh, and hands notifications to nextCh (also unbuffered) for run to consume. When run is behind, pop spills into the ring buffer.
• run is the consumer. It reads from nextCh and calls your OnAdd / OnUpdate / OnDelete synchronously, on the goroutine that owns the listener.

That last point is the crux. Your handler runs on the listener’s own goroutine. If your handler blocks for 30 seconds doing API calls, that listener stops consuming, pop starts filling the ring, and the ring never shrinks.

The ring buffer, and the comment that admits it

// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1390
	// pendingNotifications is an unbounded ring buffer that holds all notifications not yet distributed.
	// There is one per listener, but a failing/stalled listener will have infinite pendingNotifications
	// added until we OOM.
	// TODO: This is no worse than before, since reflectors were backed by unbounded DeltaFIFOs, but
	// we should try to do something better.
	pendingNotifications buffer.RingGrowing

Three sentences, and the middle one is the entire failure mode. The “TODO: This is no worse than before” is a defence of the status quo, not an endorsement: the informer’s DeltaFIFO was also unbounded, so this change did not make things worse, but it did not make them better either.

The initial size is 1024:

// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:500
const (
	// syncedPollPeriod controls how often you look at the status of your sync funcs
	syncedPollPeriod = 100 * time.Millisecond

	// initialBufferSize is the initial number of event notifications that can be buffered.
	initialBufferSize = 1024
)

and it is passed in when the listener is created:

// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1054
	listener := newProcessListener(logger, handler, resyncPeriod, determineResyncPeriod(logger, resyncPeriod, s.resyncCheckPeriod), s.clock.Now(), initialBufferSize, s.HasSyncedChecker())

1024 is the starting size, not a limit. The ring doubles from there.

pop(): the one-item lookahead

The pump is the interesting piece of code, because it contains an optimisation that is easy to misread as a bug:

// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1468
func (p *processorListener) pop() {
	defer utilruntime.HandleCrashWithLogger(p.logger)
	defer close(p.nextCh) // Tell .run() to stop
	defer close(p.done)   // Tell .watchSynced() to stop

	var nextCh chan<- interface{}
	var notification interface{}
	for {
		select {
		case nextCh <- notification:
			// Notification dispatched
			var ok bool
			notification, ok = p.pendingNotifications.ReadOne()
			if ok {
				p.pendingNotificationsLength.Add(-1)
			} else { // Nothing to pop
				nextCh = nil // Disable this select case
			}
		case notificationToAdd, ok := <-p.addCh:
			if !ok {
				return
			}
			if notification == nil { // No notification to pop (and pendingNotifications is empty)
				// Optimize the case - skip adding to pendingNotifications
				notification = notificationToAdd
				nextCh = p.nextCh
			} else { // There is already a notification waiting to be dispatched
				p.pendingNotifications.WriteOne(notificationToAdd)
				p.pendingNotificationsLength.Add(1)
			}
		}
	}
}

Read the local variable notification. It holds one notification that has been received from addCh but not yet handed to run. It is the “hot” item.

• When addCh delivers a notification and notification is nil (no hot item), the new notification becomes the hot item. Nothing is written to the ring buffer. The comment says it: “Optimize the case - skip adding to pendingNotifications”.
• When addCh delivers a notification and notification is non-nil (there is already a hot item), the old hot item is pushed into the ring buffer and the new one becomes hot.

So the ring buffer only receives an item when a second notification arrives before the first has been delivered. In the common case — events arrive slower than the handler consumes them — the ring buffer is never touched at all. That is the one-item lookahead, and it is the difference between “unbounded buffer” and “unbounded buffer that is usually empty”.

nextCh = nil disables that select case when the ring is empty, so pop blocks on addCh alone instead of spinning on a send to a channel nobody is reading.

The length counter exists only to decide when to turn on expensive tracing, and it is only mutated by pop:

// staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1520
			pendingNotifications := p.pendingNotificationsLength.Load()
			if pendingNotifications > initialBufferSize {
				trace := utiltrace.New("processorListener handler",
					utiltrace.Field{Key: "handler", Value: p.handlerName},
					utiltrace.Field{Key: "pendingNotifications", Value: pendingNotifications},
				)
				defer trace.LogIfLong(100 * time.Millisecond)
			}

Note the threshold: the trace only turns on once the backlog exceeds the initial buffer size of 1024. Below that, a slow handler is invisible.

The ring doubles

The buffer itself is a classic growing ring, and the growth is a straight doubling with a careful wrap-around copy:

// vendor/k8s.io/utils/buffer/ring_growing.go:79
// WriteOne adds an item to the end of the buffer, growing it if it is full.
func (r *TypedRingGrowing[T]) WriteOne(data T) {
	if r.readable == r.n {
		// Time to grow
		newN := r.n * 2
		if newN == 0 {
			newN = defaultRingSize
		}
		newData := make([]T, newN)
		to := r.beg + r.readable
		if to <= r.n {
			copy(newData, r.data[r.beg:to])
		} else {
			copied := copy(newData, r.data[r.beg:])
			copy(newData[copied:], r.data[:(to%r.n)])
		}
		r.beg = 0
		r.data = newData
		r.n = newN
	}
	r.data[(r.readable+r.beg)%r.n] = data
	r.readable++
}

Growth is amortised O(1): the copy happens only on the powers of two. The wrap-around branch handles the case where the readable region straddles the end of the backing array, which is the case the naive implementation gets wrong.

The default ring size is 16, but it is only used when the initial size is zero:

// vendor/k8s.io/utils/buffer/ring_growing.go:19
// defaultRingSize defines the default ring size if not specified
const defaultRingSize = 16

In the informer the initial size is 1024, so the sequence of capacities is 1024, 2048, 4096, … Each doubling copies the whole backlog, so the total copy cost over N insertions is O(N) amortised, but the peak memory is 2x the backlog.

And the GC trick again, because this is the same class of bug:

// vendor/k8s.io/utils/buffer/ring_growing.go:61
// ReadOne reads (consumes) first item from the buffer if it is available, otherwise returns false.
func (r *TypedRingGrowing[T]) ReadOne() (data T, ok bool) {
	if r.readable == 0 {
		return
	}
	r.readable--
	element := r.data[r.beg]
	var zero T
	r.data[r.beg] = zero // Remove reference to the object to help GC
	if r.beg == r.n-1 {
		// Was the last element
		r.beg = 0
	} else {
		r.beg++
	}
	return element, true
}

Without r.data[r.beg] = zero, a consumed Pod object would be pinned by the ring’s backing array until the ring grew past it — which, for a ring that never grows, is forever.

The DeltaFIFO runs the handler under the lock

Here is the contrast that makes the whole design legible. The DeltaFIFO that sits between the reflector and the distributor also runs user code, and it runs it under the queue’s own lock:

// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:550
// Pop blocks until the queue has some items, and then returns one.  If
// multiple items are ready, they are returned in the order in which they were
// added/updated. The item is removed from the queue (and the store) before it
// is returned, so if you don't successfully process it, you need to add it back
// with AddIfNotPresent().
// process function is called under lock, so it is safe to update data structures
// in it that need to be in sync with the queue (e.g. knownKeys).
// process should avoid expensive I/O operation so that other queue operations, i.e.
// Add() and Get(), won't be blocked for too long.

// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:591
		// Only log traces if the queue depth is greater than 10 and it takes more than
		// 100 milliseconds to process one item from the queue.
		// Queue depth never goes high because processing an item is locking the queue,
		// and new items can't be added until processing finish.
		// https://github.com/kubernetes/kubernetes/issues/103789
		if depth > 10 {
			trace := utiltrace.New("DeltaFIFO Pop Process",
				utiltrace.Field{Key: "ID", Value: id},
				utiltrace.Field{Key: "Depth", Value: depth},
				utiltrace.Field{Key: "Reason", Value: "slow event handlers blocking the queue"})
			defer trace.LogIfLong(100 * time.Millisecond)
		}
		err := process(item, isInInitialList)

The trace fires only when both conditions hold: the queue depth is greater than 10, and the processing takes longer than 100 milliseconds. And the reason is spelled out in the trace itself: “slow event handlers blocking the queue”.

“Queue depth never goes high because processing an item is locking the queue.” So the DeltaFIFO applies backpressure: a slow handler blocks Add, which blocks the reflector’s writes, which slows the watch. Memory stays bounded; the cost is latency.

The listener path is the opposite choice. There, distribute must not block: one slow listener must not stall the whole shared informer and every other listener on it. So the slow listener gets an unbounded buffer and eats the memory itself.

Two different failure modes, chosen deliberately:

• DeltaFIFO: backpressure. Bounded memory, added latency.
• processorListener: unbounded buffer. Bounded latency, unbounded memory.

Kubernetes’ stated priority is that one controller’s slow handler must not delay the apiserver watch that feeds every other controller. So the listener path eats memory.

The dedup that only collapses adjacent tombstones

The DeltaFIFO does try to reduce work, but its dedup is much narrower than people assume. It only ever compares the last two deltas:

// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:441
// re-listing and watching can deliver the same update multiple times in any
// order. This will combine the most recent two deltas if they are the same.
func dedupDeltas(deltas Deltas) Deltas {
	n := len(deltas)
	if n < 2 {
		return deltas
	}
	a := &deltas[n-1]
	b := &deltas[n-2]
	if out := isDup(a, b); out != nil {
		deltas[n-2] = *out
		return deltas[:n-1]
	}
	return deltas
}

“the most recent two deltas” — adjacent only. A MODIFIED, MODIFIED, MODIFIED sequence for the same key is not collapsed, because the dedup only looks at deltas n-1 and n-2 of the list, and the list is per-key, and isDup only knows how to handle deletions:

// staging/src/k8s.io/client-go/tools/cache/delta_fifo.go:468
// keep the one with the most information if both are deletions.
func isDeletionDup(a, b *Delta) *Delta {
	if b.Type != Deleted || a.Type != Deleted {
		return nil
	}
	// Do more sophisticated checks, or is this sufficient?
	if _, ok := b.Object.(DeletedFinalStateUnknown); ok {
		return a
	}
	return b
}

Two deletions of the same key collapse into one. And the survivor is the richer one: if the older delta carried a DeletedFinalStateUnknown tombstone — which contains the last known state of the object — then the tombstone is kept, because it carries strictly more information than a bare deletion event. Otherwise the newer, plainer deletion wins.

So the dedup does exactly one thing: it stops a delete-then-delete pair from being delivered twice, and it prefers the version that knows what the object used to look like. It is not a general-purpose coalescer.

The trade-off

What the unbounded buffer buys: a slow handler never blocks the reflector, never blocks the apiserver watch, and never blocks the other handlers on the same informer. The cost of a slow handler is paid by the slow handler.

What it costs:

• Memory is unbounded and the failure is a hard OOM. There is no backpressure, no drop policy, no limit. A handler that is 10x slower than the event rate will grow the ring without bound until the process is killed.
• The backlog is invisible until it is large. The trace only turns on above 1024 pending notifications, and the DeltaFIFO trace only fires above depth 10 and over 100ms. A handler that is falling behind but still under 1024 buffered items produces no signal at all.
• The lookahead is easy to get wrong when modifying. pop() holds a notification across a select, and the ring buffer only ever receives the older of two pending items. Any change to that logic has to preserve the invariant that exactly one item is in flight outside the ring.
• The run goroutine serialises the handler. Your handler is called from a single goroutine, so a slow handler is not just slow, it is serial slow, and you cannot fix it by adding workers inside the handler without breaking the ordering guarantee that OnUpdate(old, new) gives you.

Wrapping up

The one thing to remember: the buffer between the informer and your handler is unbounded, on purpose, and the upstream comment says your handler will OOM the process if it stalls (staging/src/k8s.io/client-go/tools/cache/shared_informer.go:1390). Your handler is not allowed to block, and it is not allowed to be slow.

The part people get wrong: assuming that a ResourceEventHandler is called on a goroutine you do not control and therefore cannot hurt anyone. It is called on the listener’s goroutine, and the ring buffer in front of it grows to absorb the difference. A handler that takes 200ms per object, on an informer seeing 1000 objects/second, is a slow memory leak with a deterministic shape: it will OOM.

What to go do about it: in every OnAdd / OnUpdate / OnDelete, enqueue the object into your own workqueue and return immediately — that is what workqueue is for, and its per-item coalescing gives you back the burst absorption you lose by returning early. While you are there, measure the handler: utiltrace only emits "processorListener handler" with a pendingNotifications field once the backlog exceeds initialBufferSize = 1024, so a trace with a pendingNotifications field is already a warning sign. If you cannot avoid blocking in the handler, the fix is to bound the queue you enqueue into and to make the informer watch a namespace or a label selector rather than the whole cluster.