ResourceQuota Batches Per Namespace: The LRU That Killed etcd Conflicts

TL;DR: The ResourceQuota admission plugin does not evaluate a quota inline, per request. It enqueues an admission waiter into a workqueue indexed by namespace, and a worker drains all the waiters for one namespace in a single evaluation, charging them all against one “running copy” of the quota and issuing one UpdateStatus per quota document. The gem is updatedQuotas: a 100-entry LRU of quota objects this apiserver just wrote, compared by etcd resourceVersion as a monotonic integer, so the next evaluation sees the object it just wrote instead of the stale informer copy. The source comment records the measured effect: “Before this change: 22 updates with 12 conflicts. after this change: 15 updates with 0 conflicts.”

The naive implementation of quota admission is obvious and wrong. Every request that creates a pod reads the namespace’s ResourceQuota, computes the new usage, writes the quota back, and hopes nobody else did the same thing in the meantime. With ten pods being created at once, ten apiserver replicas read the same quota, all ten compute usage+10, and nine of the ten writes fail with a conflict. The client sees a 409 and retries, and the cluster spends more etcd round-trips than the workload.

The fix has two halves: batch the evaluation so the arithmetic is done once, and make the write not conflict with itself.

The request does not block on the quota

The admission plugin’s Validate hands off to the evaluator and blocks on a channel:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/admission.go:55
			if configuration != nil {
				if errs := validation.ValidateConfiguration(configuration); len(errs) != 0 {
					return nil, errs.ToAggregate()
				}
			}
			return NewResourceQuota(configuration, 5)

Evaluate creates a waiter, enqueues it, and waits for the result with a ten second ceiling:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:650
func (e *quotaEvaluator) Evaluate(a admission.Attributes) error {
	e.init.Do(e.start)

	// is this resource ignored?
	gvr := a.GetResource()
	gr := gvr.GroupResource()
	if _, ok := e.ignoredResources[gr]; ok {
		return nil
	}

	// if we do not know how to evaluate use for this resource, create an evaluator
	evaluator := e.registry.Get(gr)
	if evaluator == nil {
		// create an object count evaluator if no evaluator previously registered
		// note, we do not need aggregate usage here, so we pass a nil informer func
		evaluator = generic.NewObjectCountEvaluator(gr, nil, "")
		e.registry.Add(evaluator)
		klog.Infof("quota admission added evaluator for: %s", gr)
	}
	// for this kind, check if the operation could mutate any quota resources
	// if no resources tracked by quota are impacted, then just return
	if !evaluator.Handles(a) {
		return nil
	}
	waiter := newAdmissionWaiter(a)

	e.addWork(waiter)

	// wait for completion or timeout
	select {
	case <-waiter.finished:
	case <-time.After(10 * time.Second):
		return apierrors.NewInternalError(fmt.Errorf("resource quota evaluation timed out"))
	}

	return waiter.result
}

The waiter starts life as a deny:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:101
func newAdmissionWaiter(a admission.Attributes) *admissionWaiter {
	return &admissionWaiter{
		attributes: a,
		finished:   make(chan struct{}),
		result:     defaultDeny{},
	}
}

defaultDeny is a sentinel, and it exists so that a waiter which was never explicitly resolved is rejected rather than admitted. The evaluator clears it only when it is sure the request fits.

The workqueue, indexed by namespace

The queue is keyed by namespace, and the evaluator keeps three maps: the work currently pending, the work that arrived while a batch was in progress, and the set of namespaces currently being processed.

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:60
	// TODO these are used together to bucket items by namespace and then batch them up for processing.
	// The technique is valuable for rollup activities to avoid fanout and reduce resource contention.
	// We could move this into a library if another component needed it.
	// queue is indexed by namespace, so that we bundle up on a per-namespace basis
	queue      *workqueue.Typed[string]
	workLock   sync.Mutex
	work       map[string][]*admissionWaiter
	dirtyWork  map[string][]*admissionWaiter
	inProgress sets.Set[string]

The dirtyWork map is the interesting one. It is the standard workqueue-double-buffered pattern: while namespace foo is being evaluated, any new waiter for foo goes into dirtyWork rather than work, so it is picked up by the next round instead of being evaluated against a half-updated running quota:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:688
func (e *quotaEvaluator) addWork(a *admissionWaiter) {
	e.workLock.Lock()
	defer e.workLock.Unlock()

	ns := a.attributes.GetNamespace()
	// this Add can trigger a Get BEFORE the work is added to a list, but this is ok because the getWork routine
	// waits the worklock before retrieving the work to do, so the writes in this method will be observed
	e.queue.Add(ns)

	if e.inProgress.Has(ns) {
		e.dirtyWork[ns] = append(e.dirtyWork[ns], a)
		return
	}

	e.work[ns] = append(e.work[ns], a)
}

Five worker goroutines drain the queue, and each one takes a whole namespace’s worth of waiters at a time:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:147
func (e *quotaEvaluator) start() {
	defer utilruntime.HandleCrash()

	for i := 0; i < e.workers; i++ {
		go wait.Until(e.doWork, time.Second, e.stopCh)
	}
}

The worker count is hardcoded at plugin registration:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/admission.go:48
			// load the configuration provided (if any)
			configuration, err := LoadConfiguration(config)
			if err != nil {
				return nil, err
			}
			// validate the configuration (if any)
			if configuration != nil {
				if errs := validation.ValidateConfiguration(configuration); len(errs) != 0 {
					return nil, errs.ToAggregate()
				}
			}
			return NewResourceQuota(configuration, 5)

Five workers, each handling a distinct namespace at a time. Since the batch is per-namespace, two namespaces never contend for the same quota calculation, and a hot namespace serialises only against itself.

The four-phase evaluation

The heart of the plugin is checkQuotas, and the source documents its algorithm in the function comment - which is the clearest statement of the design in the whole file:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:216
// checkQuotas checks the admission attributes against the passed quotas.  If a quota applies, it will attempt to update it
// AFTER it has checked all the admissionAttributes.  The method breaks down into phase like this:
//  0. make a copy of the quotas to act as a "running" quota so we know what we need to update and can still compare against the
//     originals
//  1. check each admission attribute to see if it fits within *all* the quotas.  If it didn't fit, mark the waiter as failed
//     and the running quota doesn't change.  If it did fit, check to see if any quota was changed.  If there was no quota change
//     mark the waiter as succeeded.  If some quota did change, update the running quotas
//  2. If no running quota was changed, return now since no updates are needed.
//  3. for each quota that has changed, attempt an update.  If all updates succeeded, update all unset waiters to success status and return.  If the some
//     updates failed on conflict errors and we have retries left, re-get the failed quota from our cache for the latest version
//     and recurse into this method with the subset.  It's safe for us to evaluate ONLY the subset, because the other quota
//     documents for these waiters have already been evaluated.  Step 1, will mark all the ones that should already have succeeded.
func (e *quotaEvaluator) checkQuotas(quotas []corev1.ResourceQuota, admissionAttributes []*admissionWaiter, remainingRetries int) {

Phase 0 and 1 - the “running quota” - is the batching mechanism. There are two copies of the quota list: originalQuotas, an untouched copy used to detect what actually changed, and quotas, which accumulates the running usage as each waiter is evaluated:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:229
	// yet another copy to compare against originals to see if we actually have deltas
	originalQuotas, err := copyQuotas(quotas)
	if err != nil {
		utilruntime.HandleError(err)
		return
	}

	atLeastOneChanged := false
	for i := range admissionAttributes {
		admissionAttribute := admissionAttributes[i]
		newQuotas, err := e.checkRequest(quotas, admissionAttribute.attributes)
		if err != nil {
			admissionAttribute.result = err
			continue
		}

		// Don't update quota for admissionAttributes that correspond to dry-run requests
		if admissionAttribute.attributes.IsDryRun() {
			admissionAttribute.result = nil
			continue
		}

So for a batch of ten pods in namespace foo, the quota is read once, checkRequest is called ten times, and each call folds the new pod’s usage into the running quota. The tenth call sees the usage of all nine before it. That is the whole point: the running quota is what makes the arithmetic correct even though the writes are coalesced into one.

Phase 2 - if nothing changed, there is nothing to write:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:269
	// if none of the requests changed anything, there's no reason to issue an update, just fail them all now
	if !atLeastOneChanged {
		return
	}

Phase 3 - one write per changed quota document, not per waiter:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:274
	// now go through and try to issue updates.  Things get a little weird here:
	// 1. check to see if the quota changed.  If not, skip.
	// 2. if the quota changed and the update passes, be happy
	// 3. if the quota changed and the update fails, add the original to a retry list
	var updatedFailedQuotas []corev1.ResourceQuota
	var lastErr error
	for i := range quotas {
		newQuota := quotas[i]

		// if this quota didn't have its status changed, skip it
		if quota.Equals(originalQuotas[i].Status.Used, newQuota.Status.Used) {
			continue
		}

		if err := e.quotaAccessor.UpdateQuotaStatus(&newQuota); err != nil {
			updatedFailedQuotas = append(updatedFailedQuotas, newQuota)
			lastErr = err
		}
	}

And only after every write succeeds are the waiters resolved:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:294
	if len(updatedFailedQuotas) == 0 {
		// all the updates succeeded.  At this point, anything with the default deny error was just waiting to
		// get a successful update, so we can mark and notify
		for _, admissionAttribute := range admissionAttributes {
			if IsDefaultDeny(admissionAttribute.result) {
				admissionAttribute.result = nil
			}
		}
		return
	}

The retry depth is 3, and the retry is a partial re-evaluation - only the quota documents that failed are re-fetched and re-evaluated, because the others have already been accounted for:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:341
	e.checkQuotas(quotasToCheck, admissionAttributes, remainingRetries-1)

The updatedQuotas LRU

Here is the part that makes the whole thing stop fighting etcd. The quota objects come from an informer cache, and the informer is behind: it is updated by a watch, and the watch delivers the new object after the write that produced it. So immediately after UpdateQuotaStatus returns, the informer still holds the old quota with the old used numbers.

If the next batch reads that stale object, it computes a usage that is too low, and the write that follows fails with a conflict - because the object in etcd has moved on. This is a self-inflicted conflict: the apiserver conflicts with itself.

The fix is a small LRU of the objects this apiserver just wrote:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/resource_access.go:60
	// updatedQuotas holds a cache of quotas that we've updated.  This is used to pull the "really latest" during back to
	// back quota evaluations that touch the same quota doc.  This only works because we can compare etcd resourceVersions
	// for the same resource as integers.  Before this change: 22 updates with 12 conflicts.  after this change: 15 updates with 0 conflicts
	updatedQuotas *lru.Cache

Twenty-two writes with twelve conflicts became fifteen writes with none. The writes went down because the batching was already there and the conflicts were causing retries that wrote again.

The cache is populated on write:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/resource_access.go:79
func (e *quotaAccessor) UpdateQuotaStatus(newQuota *corev1.ResourceQuota) error {
	updatedQuota, err := e.client.CoreV1().ResourceQuotas(newQuota.Namespace).UpdateStatus(context.TODO(), newQuota, metav1.UpdateOptions{})
	if err != nil {
		return err
	}

	key := newQuota.Namespace + "/" + newQuota.Name
	e.updatedQuotas.Add(key, updatedQuota)
	return nil
}

and consulted on read:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/resource_access.go:92
// checkCache compares the passed quota against the value in the look-aside cache and returns the newer
// if the cache is out of date, it deletes the stale entry.  This only works because of etcd resourceVersions
// being monotonically increasing integers
func (e *quotaAccessor) checkCache(quota *corev1.ResourceQuota) *corev1.ResourceQuota {
	key := quota.Namespace + "/" + quota.Name
	uncastCachedQuota, ok := e.updatedQuotas.Get(key)
	if !ok {
		return quota
	}
	cachedQuota := uncastCachedQuota.(*corev1.ResourceQuota)

	if etcdVersioner.CompareResourceVersion(quota, cachedQuota) >= 0 {
		e.updatedQuotas.Remove(key)
		return quota
	}
	return cachedQuota
}

The comparison is a plain integer comparison, because etcd’s resourceVersion is not an opaque string - it is a monotonic integer:

go
// staging/src/k8s.io/apiserver/pkg/storage/api_object_versioner.go:108
// CompareResourceVersion compares etcd resource versions.  Outside this API they are all strings,
// but etcd resource versions are special, they're actually ints, so we can easily compare them.
func (a APIObjectVersioner) CompareResourceVersion(lhs, rhs runtime.Object) int {

So the rule is: if the informer copy is at least as new as the cached copy, the informer has caught up and the cached entry is dropped. If the cached copy is newer, the cached copy wins. Either way, the evaluator reads the newest thing it knows about.

The cache is bounded. newQuotaAccessor creates both caches with a size of 100 and a 30 second TTL on the live lookups:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/resource_access.go:66
// newQuotaAccessor creates an object that conforms to the QuotaAccessor interface to be used to retrieve quota objects.
func newQuotaAccessor() (*quotaAccessor, error) {
	liveLookupCache := lru.New(100)
	updatedCache := lru.New(100)

	// client and lister will be set when SetInternalKubeClientSet and SetInternalKubeInformerFactory are invoked
	return &quotaAccessor{
		liveLookupCache: liveLookupCache,
		liveTTL:         time.Duration(30 * time.Second),
		updatedQuotas:   updatedCache,
	}, nil
}

The liveLookupCache and singleflight

The second cache is for a different failure mode: the informer has not synced yet, so it returns an empty list for a namespace that actually has quotas. If the evaluator believed that, it would let everything through unmetered.

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/resourcequota/resource_access.go:118
	// if there are no items held in our unsynced lister, check our live-lookup LRU, if that misses, do the live lookup to prime it.
	if len(items) == 0 && !e.hasSynced() {
		lruItemObj, ok := e.liveLookupCache.Get(namespace)
		if !ok || lruItemObj.(liveLookupEntry).expiry.Before(time.Now()) {
			// use singleflight.Group to avoid flooding the apiserver with repeated
			// requests. See #22422 for details.
			lruItemObj, err, _ = e.group.Do(namespace, func() (interface{}, error) {
				liveList, err := e.client.CoreV1().ResourceQuotas(namespace).List(context.TODO(), metav1.ListOptions{})

The singleflight.Group is the part that matters under load. Without it, every request that arrives while the informer is cold triggers its own List against the API server, and a namespace with a quota sees a thundering herd of identical reads. With it, the first caller does the List and the rest wait for its result.

LimitRanger’s sibling trick

LimitRanger - the sibling plugin that sets defaults and enforces min/max limits - uses the same pattern, with the same live-lookup LRU and singleflight, because it has exactly the same “the informer is cold” problem:

go
// plugin/pkg/admission/limitranger/admission.go:173
	// if there are no items held in our indexer, check our live-lookup LRU, if that misses, do the live lookup to prime it.
	if len(items) == 0 {
		lruItemObj, ok := l.liveLookupCache.Get(a.GetNamespace())
		if !ok || lruItemObj.(liveLookupEntry).expiry.Before(time.Now()) {
			// Fixed: #22422
			// use singleflight to alleviate simultaneous calls to
			lruItemObj, err, _ = l.group.Do(a.GetNamespace(), func() (interface{}, error) {

with a bigger cache - 10000 entries, because a node can have a LimitRange in every namespace it might touch:

go
// plugin/pkg/admission/limitranger/admission.go:204
// NewLimitRanger returns an object that enforces limits based on the supplied limit function
func NewLimitRanger(actions LimitRangerActions) (*LimitRanger, error) {
	liveLookupCache := lru.New(10000)

	if actions == nil {
		actions = &DefaultLimitRangerActions{}
	}

	return &LimitRanger{
		Handler:         admission.NewHandler(admission.Create, admission.Update),
		actions:         actions,
		liveLookupCache: liveLookupCache,
		liveTTL:         time.Duration(30 * time.Second),
	}, nil
}

LimitRanger also has a nice piece of arithmetic hygiene. The “limit must not exceed request times ratio” check is done in exact decimal, never in float, because Quantity has no multiply and inf.Dec.Mul adds the two scales rather than multiplying them:

go
// plugin/pkg/admission/limitranger/admission.go:301
// exceedsAllowed reports whether limit is greater than request times ratio.
// Quantity has no multiply and inf.Dec.Mul adds the two scales as an int32, so
// the product is formed here with the scale kept in int64 and handed to Cmp.
func exceedsAllowed(limit, request, ratio resource.Quantity) bool {
	requestDec, ratioDec := request.AsDec(), ratio.AsDec()
	scale := int64(requestDec.Scale()) + int64(ratioDec.Scale())

The product is formed from the unscaled big.Ints with the scale applied afterwards, so no precision is lost anywhere:

go
// plugin/pkg/admission/limitranger/admission.go:313
	product := new(inf.Dec)
	product.SetUnscaledBig(new(big.Int).Mul(requestDec.UnscaledBig(), ratioDec.UnscaledBig()))
	product.SetScale(inf.Scale(scale))
	allowed := resource.NewDecimalQuantity(*product, resource.DecimalSI)
	return limit.Cmp(*allowed) > 0
}

Trade-offs

What the batching buys. One quota read and one quota write per namespace per batch, regardless of how many requests are waiting. No self-inflicted etcd conflicts: the LRU means the apiserver never writes a stale object. Correct arithmetic: the running quota folds each waiter into the next, so the tenth pod in a batch is evaluated against the usage of the first nine.

What it costs.

Admission latency. A request now waits for a worker to pick up its namespace, and the worker’s poll interval is one second. Under heavy churn, requests can sit in the queue. The ten second ceiling in Evaluate is the backstop.
A namespace is a serialisation point. All the quota evaluation for one namespace happens in one worker at a time. A namespace creating thousands of objects per second will have a queue.
The running quota is only as good as the batch. If a waiter arrives after the batch started, it goes into dirtyWork and is evaluated in the next round against the quota that was just written. That is correct, but it means a request can observe a quota state that is one round old.
The LRU is bounded at 100 entries. A cluster with more than 100 actively-churning quota documents will start evicting entries, and the eviction falls back to the informer copy - which is correct, just less efficient.

Wrapping up

The one thing to remember: ResourceQuota evaluation is batched per namespace through a workqueue indexed by namespace, with five workers and a ten second per-request ceiling, and the batch’s writes are made conflict-free by updatedQuotas - a 100-entry LRU compared by etcd resourceVersion as an integer. The measured effect is in the source comment: 22 writes with 12 conflicts became 15 writes with 0.

The part people get wrong: assuming the quota’s status.used is authoritative at the moment a request is admitted. It is not - the admission decision is made against a running copy that folds in the other waiters in the same batch, and the object in etcd is updated once at the end. A kubectl get resourcequota immediately after a burst of creates will show a used value that lags the burst, and it will catch up when the batch is written.

What to go do about it: there is no quota-specific apiserver metric, so go to the source. Run the apiserver with -v=4 and look for “quota admission added evaluator for: ” - that message means a resource type is being metered by object count rather than by a real evaluator, and it is logged once per resource type. To see the state of the running quota, use kubectl get resourcequota -o yaml and compare status.used against status.hard; remember that status.used lags a burst by one batch. If you suspect quota-related write conflicts, watch etcd’s etcd_server_proposals_failed_total with reason="conflict" - the updatedQuotas LRU exists precisely to keep that counter flat. And if you are running LimitRanger alongside ResourceQuota, remember that its liveLookupCache holds 10000 entries (ResourceQuota’s holds 100) with a 30 second TTL, so a node that touches many namespaces with LimitRanges will keep a larger warm cache than the quota evaluator does.