The Admission Chain Runs Twice: Reinvocation and the Webhook That Was Skipped

TL;DR: The mutating admission chain is not run once. It is wrapped by newReinvocationHandler, and after the first pass, if any plugin said the object changed, the whole chain runs a second time. The subtlety is which webhooks get re-run: a webhook that was skipped in round zero - because its objectSelector did not match, or its match conditions were false - is not called in round one even if the object now matches. Kubernetes tracks two sets of webhook UIDs for exactly this: the webhooks that were invoked and are eligible, and the webhooks that must be re-invoked.

The intuition behind admission is that each plugin sees a slightly different object than the one before it saw, and that is fine as long as the plugins are independent. But they are not independent. A mutating webhook that adds a label may enable a second webhook’s objectSelector; a policy that defaults a field may make another policy’s condition true. If the chain only ran once, those interactions would silently not happen.

So the chain runs twice. Getting that right - running twice, but not running everything twice - is the whole difficulty.

The chain, and the second pass

The reinvocation wrapper is not part of the chain; it wraps the chain:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugins.go:162
	return newReinvocationHandler(chainAdmissionHandler(handlers)), nil

and the wrapper is a dozen lines:

go
// staging/src/k8s.io/apiserver/pkg/admission/reinvocation.go:35
func (r *reinvoker) Admit(ctx context.Context, a Attributes, o ObjectInterfaces) error {
	if mutator, ok := r.admissionChain.(MutationInterface); ok {
		err := mutator.Admit(ctx, a, o)
		if err != nil {
			return err
		}
		s := a.GetReinvocationContext()
		if s.ShouldReinvoke() {
			s.SetIsReinvoke()
			// Calling admit a second time will reinvoke all in-tree plugins
			// as well as any webhook plugins that need to be reinvoked based on the
			// reinvocation policy.
			return mutator.Admit(ctx, a, o)
		}
	}
	return nil
}

Two things to note. First, there is no loop - the second pass is exactly one extra pass. Two is enough, because a second pass cannot produce a third distinct change that a third pass would resolve; if it does, the chain has already converged in practice and the API server does not try harder.

Second, the second pass runs the whole chain again, including all the in-tree plugins, not just the webhooks. In-tree plugins that mutate - the defaulting webhook among them - are re-run, and they are supposed to be idempotent.

The decision to re-run is carried on the admission attributes, in a small struct that every plugin can read and write:

go
// staging/src/k8s.io/apiserver/pkg/admission/attributes.go:166
type reinvocationContext struct {
	// isReinvoke is true when admission plugins are being reinvoked
	isReinvoke bool
	// reinvokeRequested is true when an admission plugin requested a re-invocation of the chain
	reinvokeRequested bool
	// values stores reinvoke context values per plugin.
	values map[string]interface{}
}

ShouldReinvoke() is the flag that says “somebody mutated the object”; IsReinvoke() is the flag that says “we are in the second pass now”. A plugin that mutates sets the first; a plugin that wants to behave differently in the second pass reads the second.

Who sets the flag

Three call sites in the tree set SetShouldReinvoke(), and all three do it the same way - compare the object before and after, and if the deep-equal check fails, ask for a re-invocation.

The mutating webhook dispatcher is one of them:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go:192
		if changed {
			// Patch had changed the object. Prepare to reinvoke all previous mutations that are eligible for re-invocation.
			webhookReinvokeCtx.RequireReinvokingPreviouslyInvokedPlugins()
			reinvokeCtx.SetShouldReinvoke()
		}
		if hook.ReinvocationPolicy != nil && *hook.ReinvocationPolicy == admissionregistrationv1.IfNeededReinvocationPolicy {
			webhookReinvokeCtx.AddReinvocableWebhookToPreviouslyInvoked(invocation.Webhook.GetUID())
		}

The CEL-based mutating admission policy dispatcher does the same thing, and also asks for a re-invocation when the object had to be converted back to its internal version:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/policy/mutating/dispatcher.go:208
		if !apiequality.Semantic.DeepEqual(objectBeforeMutations, versionedAttr.VersionedObject.Object()) {
			// The mutation has changed the object. Prepare to reinvoke all previous mutations that are eligible for re-invocation.
			policyReinvokeCtx.RequireReinvokingPreviouslyInvokedPlugins()
			reinvokeCtx.SetShouldReinvoke()
		}

Note that RequireReinvokingPreviouslyInvokedPlugins() and SetShouldReinvoke() are always called together. The first says “the webhooks that already ran should run again”; the second says “the chain should run again at all”. They are separate because an in-tree mutation needs the second without necessarily needing the first - if no reinvocable webhook has run yet, there is nothing to re-invoke.

The two sets of webhook UIDs

The webhook dispatcher keeps its own reinvocation context, stashed in the shared context under the plugin’s name so it survives across the two passes:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/reinvocationcontext.go:25
type webhookReinvokeContext struct {
	// lastWebhookOutput holds the result of the last webhook admission plugin call
	lastWebhookOutput runtime.Object
	// previouslyInvokedReinvocableWebhooks holds the set of webhooks that have been invoked and
	// should be reinvoked if a later mutation occurs
	previouslyInvokedReinvocableWebhooks sets.Set[string]
	// reinvokeWebhooks holds the set of webhooks that should be reinvoked
	reinvokeWebhooks sets.Set[string]
}

Two sets, two different jobs:

previouslyInvokedReinvocableWebhooks is the set of webhooks that have already been called and that declared reinvocationPolicy: IfNeeded. It is a running pool.
reinvokeWebhooks is the set that the next pass should actually call. It is computed from the pool when a mutation happens.

The promotion from one to the other is the whole mechanism:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/reinvocationcontext.go:58
func (rc *webhookReinvokeContext) RequireReinvokingPreviouslyInvokedPlugins() {
	if len(rc.previouslyInvokedReinvocableWebhooks) > 0 {
		if rc.reinvokeWebhooks == nil {
			rc.reinvokeWebhooks = sets.New[string]()
		}
		for s := range rc.previouslyInvokedReinvocableWebhooks {
			rc.reinvokeWebhooks.Insert(s)
		}
		rc.previouslyInvokedReinvocableWebhooks = sets.New[string]()
	}
}

Note the last line: the pool is emptied. A webhook that ran in round zero and mutated the object is added back to the pool by the changed branch above (it is added after RequireReinvokingPreviouslyInvokedPlugins is called), so it is eligible for round one. A webhook that ran and did not mutate is not added back, so it will not run a third time if there were one.

The other half of the mechanism is the in-tree mutation case. If an in-tree plugin changed the object between the webhook passes, then all previously invoked reinvocable webhooks are invalidated, because they may have made their round-zero decisions against a stale object:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go:115
	if reinvokeCtx.IsReinvoke() && webhookReinvokeCtx.IsOutputChangedSinceLastWebhookInvocation(attr.GetObject()) {
		// If the object has changed, we know the in-tree plugin re-invocations have mutated the object,
		// and we need to reinvoke all eligible webhooks.
		webhookReinvokeCtx.RequireReinvokingPreviouslyInvokedPlugins()
	}
	defer func() {
		webhookReinvokeCtx.SetLastWebhookInvocationOutput(attr.GetObject())
	}()

IsOutputChangedSinceLastWebhookInvocation is a DeepEqual against a snapshot of the object taken at the end of the last webhook dispatch:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/reinvocationcontext.go:39
func (rc *webhookReinvokeContext) IsOutputChangedSinceLastWebhookInvocation(object runtime.Object) bool {
	return !apiequality.Semantic.DeepEqual(rc.lastWebhookOutput, object)
}

The webhook that was skipped

Now the important part. A webhook’s eligibility is decided fresh on every pass, by ShouldCallHook - which evaluates the rules, the namespace selector, the object selector and the match conditions against the current state of the object. If the object changed in round zero such that the webhook now matches, ShouldCallHook will return an invocation for it in round one.

And then the dispatcher skips it anyway:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go:145
		// This means that during reinvocation, a webhook will not be
		// called for the first time. For example, if the webhook is
		// skipped in the first round because of mismatching labels,
		// even if the labels become matching, the webhook does not
		// get called during reinvocation.
		if reinvokeCtx.IsReinvoke() && !webhookReinvokeCtx.ShouldReinvokeWebhook(invocation.Webhook.GetUID()) {
			continue
		}

ShouldReinvokeWebhook consults reinvokeWebhooks - the set of webhooks that were invoked in round zero and declared reinvocationPolicy: IfNeeded. A webhook that was skipped in round zero was never added to previouslyInvokedReinvocableWebhooks, so it is not in reinvokeWebhooks, so it is skipped in round one too.

This is deliberate and it is the correct behaviour. The reinvocationPolicy contract is “call me again”, not “call me if I now match”. If a webhook that was skipped in round zero started matching in round one and got called, then the object would change, which would trigger another pass, which could call yet another previously-skipped webhook - and the request could cascade. Bounding the reinvocation to webhooks that already ran keeps the process finite.

The default policy is Never, and the default timeout is 10 seconds:

go
// pkg/apis/admissionregistration/v1/defaults.go:55
func SetDefaults_MutatingWebhook(obj *admissionregistrationv1.MutatingWebhook) {
	if obj.FailurePolicy == nil {
		policy := admissionregistrationv1.Fail
		obj.FailurePolicy = &policy
	}
	if obj.MatchPolicy == nil {
		policy := admissionregistrationv1.Equivalent
		obj.MatchPolicy = &policy
	}
	if obj.NamespaceSelector == nil {
		selector := metav1.LabelSelector{}
		obj.NamespaceSelector = &selector
	}
	if obj.ObjectSelector == nil {
		selector := metav1.LabelSelector{}
		obj.ObjectSelector = &selector
	}
	if obj.TimeoutSeconds == nil {
		obj.TimeoutSeconds = new(int32)
		*obj.TimeoutSeconds = 10
	}
	if obj.ReinvocationPolicy == nil {
		never := admissionregistrationv1.NeverReinvocationPolicy
		obj.ReinvocationPolicy = &never
	}
}

and the timeout is bounded by validation:

go
// pkg/apis/admissionregistration/validation/validation.go:387
	if hook.TimeoutSeconds != nil && (*hook.TimeoutSeconds > 30 || *hook.TimeoutSeconds < 1) {
		allErrors = append(allErrors, field.Invalid(fldPath.Child("timeoutSeconds"), *hook.TimeoutSeconds, "the timeout value must be between 1 and 30 seconds"))
	}

So: failurePolicy defaults to Fail, timeoutSeconds defaults to 10 and must be between 1 and 30, and reinvocationPolicy defaults to Never. Only a webhook that explicitly opts into IfNeeded is ever called twice.

The deadline rounds up

When the dispatcher builds the HTTP request to the webhook, it sets the client-side timeout from the context deadline, and it rounds up to the next whole second:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go:284
	// if the context has a deadline, set it as a parameter to inform the backend
	if deadline, hasDeadline := ctx.Deadline(); hasDeadline {
		// compute the timeout
		if timeout := time.Until(deadline); timeout > 0 {
			// if it's not an even number of seconds, round up to the nearest second
			if truncated := timeout.Truncate(time.Second); truncated != timeout {
				timeout = truncated + time.Second
			}
			// set the timeout
			r.Timeout(timeout)
		}
	}

The reason to round up rather than down is that the timeout is a hint to the remote peer. If the apiserver has 9.4 seconds left and it sends timeout=9, a peer that floors the value to whole seconds will kill the request at 9 seconds - a request the apiserver still considers live, and one whose failure would be attributed to the webhook rather than to the apiserver’s own deadline. Rounding up means the peer always waits at least as long as the apiserver does.

The context deadline itself comes from the webhook’s timeoutSeconds:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go:275
	// if the webhook has a specific timeout, wrap the context to apply it
	if h.TimeoutSeconds != nil {
		var cancel context.CancelFunc
		ctx, cancel = context.WithTimeout(ctx, time.Duration(*h.TimeoutSeconds)*time.Second)
		defer cancel()
	}

failurePolicy: Ignore does not cover the parent deadline

failurePolicy: Ignore means “if the webhook is unreachable or errors, let the request through anyway”. But it does not mean “ignore the fact that the whole request has run out of time”. The dispatcher distinguishes the two:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go:204
		if callErr, ok := err.(*webhookutil.ErrCallingWebhook); ok {
			if ignoreClientCallFailures {
				// Ignore context cancelled from webhook metrics
				if errors.Is(callErr.Reason, context.Canceled) {
					klog.Warningf("Context canceled when calling webhook %v", hook.Name)
				} else {
					klog.Warningf("Failed calling webhook, failing open %v: %v", hook.Name, callErr)
					admissionmetrics.Metrics.ObserveWebhookFailOpen(ctx, hook.Name, "admit")
					annotator.addFailedOpenAnnotation()
				}
				utilruntime.HandleError(callErr)

				select {
				case <-ctx.Done():
					// parent context is canceled or timed out, no point in continuing
					return apierrors.NewTimeoutError("request did not complete within requested timeout", 0)
				default:
					// individual webhook timed out, but parent context did not, continue
					continue
				}
			}

The select is the whole point. If this webhook timed out (its own timeoutSeconds expired) but the parent request context is still alive, failurePolicy: Ignore applies and the chain continues. If the parent context is done, the apiserver returns a TimeoutError regardless of the failure policy - because the client is already out of time, and continuing would just be work whose result nobody will read.

matchConditions have their own budget

Match conditions are CEL expressions evaluated on the apiserver, before the webhook is called. They have their own cost budget, which is a quarter of the budget for validating policies:

go
// staging/src/k8s.io/apiserver/pkg/apis/cel/config.go:19
const (
	// PerCallLimit specify the actual cost limit per CEL validation call
	// current PerCallLimit gives roughly 0.1 second for each expression validation call
	PerCallLimit = 1000000

	// RuntimeCELCostBudget is the overall cost budget for runtime CEL validation cost per ValidatingAdmissionPolicyBinding or CustomResource
	// current RuntimeCELCostBudget gives roughly 1 seconds for the validation
	RuntimeCELCostBudget = 10000000

	// RuntimeCELCostBudgetMatchConditions is the overall cost budget for runtime CEL validation cost on matchConditions per object with matchConditions
	// this is per webhook for validatingwebhookconfigurations and mutatingwebhookconfigurations or per ValidatingAdmissionPolicyBinding
	// current RuntimeCELCostBudgetMatchConditions gives roughly 1/4 seconds for the validation
	RuntimeCELCostBudgetMatchConditions = 2500000

And they have their own failure policy, which is separate from the webhook’s failurePolicy. The matcher takes it as a parameter:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/matchconditions/matcher.go:64
func NewMatcher(filter celplugin.ConditionEvaluator, failPolicy *v1.FailurePolicyType, matcherKind, matcherType, objectName string) Matcher {
	var f v1.FailurePolicyType
	if failPolicy == nil {
		f = v1.Fail
	} else {
		f = *failPolicy
	}

The evaluation itself, with the budget passed in:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/matchconditions/matcher.go:80
func (m *matcher) Match(ctx context.Context, versionedAttr *admission.VersionedAttributes, versionedParams runtime.Object, authz authorizer.UnconditionalAuthorizer) MatchResult {
	t := time.Now()
	evalResults, _, err := m.filter.ForInput(ctx, versionedAttr, celplugin.CreateAdmissionRequest(versionedAttr.Attributes, metav1.GroupVersionResource(versionedAttr.GetResource()), metav1.GroupVersionKind(versionedAttr.VersionedKind)), celplugin.OptionalVariableBindings{
		VersionedParams: versionedParams,
		Authorizer:      authz,
	}, nil, celconfig.RuntimeCELCostBudgetMatchConditions)

A CEL error under failurePolicy: Fail rejects the request without the webhook ever being called:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/matchconditions/matcher.go:124
	if len(errorList) > 0 {
		admissionmetrics.Metrics.ObserveMatchConditionEvaluationTime(ctx, time.Since(t), m.objectName, m.matcherKind, m.matcherType, string(versionedAttr.GetOperation()))
		// If mix of true and eval errors then resort to fail policy
		if m.failPolicy == v1.Fail {
			// mix of true and errors with fail policy fail should fail request without calling webhook
			err = utilerrors.NewAggregate(errorList)
			return MatchResult{
				Error: err,
			}
		} else if m.failPolicy == v1.Ignore {
			// if fail policy ignore then skip call to webhook
			return MatchResult{
				Matches: false,
			}
		}
	}

And the caller turns a MatchResult.Error into a Forbidden, again without calling the webhook:

go
// staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/generic/webhook.go:396
		if matchResult.Error != nil {
			klog.Warningf("Failed evaluating match conditions, failing closed %v: %v", h.GetName(), matchResult.Error)
			return nil, apierrors.NewForbidden(attr.GetResource().GroupResource(), attr.GetName(), matchResult.Error)
		} else if !matchResult.Matches {
			admissionmetrics.Metrics.ObserveMatchConditionExclusion(ctx, h.GetName(), "webhook", h.GetType(), string(attr.GetOperation()))
			// if no match, always skip webhook
			return nil, nil
		}

Note the asymmetry that trips people up: with failurePolicy: Ignore on the match conditions, a CEL evaluation error means “skip the webhook”, not “call the webhook anyway”. Ignore here means “ignore the condition, treat it as not matching”, and the webhook is not called. If you want the webhook to be called when CEL fails, that is failurePolicy: Fail on the webhook - but then the CEL failure itself rejects the request first.

Trade-offs

What reinvocation buys. Mutating webhooks and policies can depend on each other’s output. Without a second pass, “add a label, and then have a policy that reacts to that label” is not expressible.

What it costs.

A second round trip per webhook. The second pass calls the webhook again, over the network, with the full AdmissionReview payload. A webhook with reinvocationPolicy: IfNeeded roughly doubles its request volume on mutating requests.
Webhooks must be idempotent. The second pass sees an object the webhook has already modified. A webhook that appends to a list instead of setting it will corrupt the object on the second pass.
A webhook that was skipped never gets its chance. This is the surprising part, and it means reinvocationPolicy: IfNeeded is not a general “call me when relevant” switch - it is “call me again”.
The two-pass structure is not visible from the webhook’s side. A webhook cannot tell whether it is being called for the first or the second time except by observing that the object already contains its own mutations.

Wrapping up

The one thing to remember: the mutating admission chain runs a second time when any plugin mutated the object, and the second pass re-runs only the webhooks that were already invoked and declared reinvocationPolicy: IfNeeded. A webhook skipped in round zero stays skipped in round one, no matter how much the object changed to match it.

The part people get wrong: reading reinvocationPolicy: IfNeeded as “call me whenever I match”. It means “call me again if I already ran”. If you want a webhook to react to a label that another webhook adds, you need the first webhook to be reinvocable and the second one to have been invoked in round zero - the dependency has to exist in both directions, or the second webhook needs to be an in-tree plugin, which does get re-run unconditionally.

What to go do about it: set reinvocationPolicy: IfNeeded only on webhooks that are written to be idempotent, and set timeoutSeconds explicitly (the default is 10, the valid range is 1 to 30) so the deadline rounding in callAttrMutatingHook is predictable. Watch apiserver_admission_webhook_rejection_count and apiserver_admission_webhook_fail_open_count for the failure policy in action, and apiserver_admission_match_condition_evaluation_errors_total for the CEL path - a CEL evaluation error under failurePolicy: Fail rejects the request with Forbidden before the webhook is ever contacted, which looks exactly like a webhook failure in the client but is not one.