Kubernetes on Opensource internals

The 1/n Lottery: How kube-proxy Load-Balances Without a Single Line of State

Jay Hubbert — Wed, 21 Oct 2026 21:00:00 +0000

TL;DR: The iptables proxier does not keep a counter, a hash table or a round-robin index per service. It writes one iptables rule per endpoint, and rule i carries -m statistic --mode random --probability 1/(n-i); the last rule has no probability at all and matches unconditionally. A connection therefore reaches endpoint i with probability 1/n, exactly, and the kernel’s random number generator makes the choice — once per connection, not once per packet. No state, no lock, no coordination.

Every other load balancer in the data centre keeps state: a connection table, a round-robin cursor, a least-connections counter, a consistent-hash ring. kube-proxy’s iptables mode keeps none of it, and yet the split across endpoints of a Service is exactly uniform. The trick is that the state lives in the kernel’s conntrack table, and the algorithm is expressed as a cascade of probabilistic matches rather than as a lookup.

This is worth understanding properly, because the iptables mode is still the default on most clusters and its failure modes are unusual.

One rule per endpoint

The proxier builds a chain per service port (KUBE-SVC-XXXX) and a chain per endpoint (KUBE-SEP-XXXX), and fills the service chain with one rule per endpoint:

 go
 
 
// pkg/proxy/iptables/proxier.go:1430-1474
func (proxier *Proxier) writeServiceToEndpointRules(natRules proxyutil.LineBuffer, svcPortNameString string, svcInfo proxy.ServicePort, svcChain utiliptables.Chain, endpoints []proxy.Endpoint, args []string) {
	// First write session affinity rules, if applicable.
	if svcInfo.SessionAffinityType() == v1.ServiceAffinityClientIP {
		for _, ep := range endpoints {
			epInfo, ok := ep.(*endpointInfo)
			if !ok {
				continue
			}
			comment := fmt.Sprintf(`"%s -> %s"`, svcPortNameString, epInfo.String())

			args = append(args[:0],
				"-A", string(svcChain),
			)
			args = proxier.appendServiceCommentLocked(args, comment)
			args = append(args,
				"-m", "recent", "--name", string(epInfo.ChainName),
				"--rcheck", "--seconds", strconv.Itoa(svcInfo.StickyMaxAgeSeconds()), "--reap",
				"-j", string(epInfo.ChainName),
			)
			natRules.Write(args)
		}
	}

	// Now write loadbalancing rules.
	numEndpoints := len(endpoints)
	for i, ep := range endpoints {
		epInfo, ok := ep.(*endpointInfo)
		if !ok {
			continue
		}
		comment := fmt.Sprintf(`"%s -> %s"`, svcPortNameString, epInfo.String())

		args = append(args[:0], "-A", string(svcChain))
		args = proxier.appendServiceCommentLocked(args, comment)
		if i < (numEndpoints - 1) {
			// Each rule is a probabilistic match.
			args = append(args,
				"-m", "statistic",
				"--mode", "random",
				"--probability", proxier.probability(numEndpoints-i))
		}
		// The final (or only if n == 1) rule is a guaranteed match.
		natRules.Write(args, "-j", string(epInfo.ChainName))
	}
}

The comment // The final (or only if n == 1) rule is a guaranteed match. is the crux. Every rule except the last is guarded by a probability; the last one always fires. The probability attached to rule i is 1/(n-i) where n is the endpoint count and i is the rule index.

Each endpoint chain ends in a DNAT:

 go
 
 
// pkg/proxy/iptables/proxier.go:1229-1242
			args = append(args[:0], "-A", string(endpointChain))
			args = proxier.appendServiceCommentLocked(args, svcPortNameString)
			// Handle traffic that loops back to the originator with SNAT.
			natRules.Write(
				args,
				"-s", epInfo.IP(),
				"-j", string(kubeMarkMasqChain))
			// Update client-affinity lists.
			if svcInfo.SessionAffinityType() == v1.ServiceAffinityClientIP {
				args = append(args, "-m", "recent", "--name", string(endpointChain), "--set")
			}
			// DNAT to final destination.
			args = append(args, "-m", protocol, "-p", protocol, "-j", "DNAT", "--to-destination", epInfo.String())
			natRules.Write(args)

The cascade computes exactly 1/n

Walk the three-endpoint case. Rule 0 matches with probability 1/3. Rule 1 matches with probability 1/2 — but only if rule 0 did not match, which happens with probability 2/3. Rule 2 always matches, but only if the first two did not, which happens with probability (2/3) x (1/2) = 1/3.

P(endpoint 0) = 1/3
P(endpoint 1) = (2/3) x (1/2) = 1/3
P(endpoint 2) = (2/3) x (1/2) = 1/3

Uniform, exactly, with no floating-point accumulation error beyond the printed precision. For four endpoints the cascade is 1/4, 1/3, 1/2, unconditional, and the products telescope the same way: 1/4, (3/4)(1/3) = 1/4, (3/4)(2/3)(1/2) = 1/4, and the remainder is 1 - 3/4 = 1/4.

This is why the last rule has no probability attached. If it did — say the chain ended with 1/(n-(n-1)) = 1/1 — the kernel’s statistic match would still be consulted, but there is no point: the probability of reaching the last rule is already exactly 1/n, so the rule matches everything that gets to it.

The distribution is uniform per connection, not per packet, and not per byte. A client that opens 1000 connections gets roughly 333 to each endpoint, with binomial variance. A client that opens one connection and streams a gigabyte sends all of it to one endpoint. This is a per-connection load balancer and nothing more.

Precomputing the probability strings

The probabilities themselves are computed once and cached, because fmt.Sprintf is expensive and the proxier calls it for every endpoint of every service on every sync:

 go
 
 
// pkg/proxy/iptables/proxier.go:390-410
func computeProbability(n int) string {
	return fmt.Sprintf("%0.10f", 1.0/float64(n))
}

// This assumes proxier.mu is held
func (proxier *Proxier) precomputeProbabilities(numberOfPrecomputed int) {
	if len(proxier.precomputedProbabilities) == 0 {
		proxier.precomputedProbabilities = append(proxier.precomputedProbabilities, "<bad value>")
	}
	for i := len(proxier.precomputedProbabilities); i <= numberOfPrecomputed; i++ {
		proxier.precomputedProbabilities = append(proxier.precomputedProbabilities, computeProbability(i))
	}
}

// This assumes proxier.mu is held
func (proxier *Proxier) probability(n int) string {
	if n >= len(proxier.precomputedProbabilities) {
		proxier.precomputeProbabilities(n)
	}
	return proxier.precomputedProbabilities[n]
}

The comment on the field explains why:

 go
 
 
// pkg/proxy/iptables/proxier.go:167-170
	// Since converting probabilities (floats) to strings is expensive
	// and we are using only probabilities in the format of 1/n, we are
	// precomputing some number of those and cache for future reuse.
	precomputedProbabilities []string

The slice is pre-sized for 1000 entries and index 0 is a sentinel:

 go
 
 
// pkg/proxy/iptables/proxier.go:283
		precomputedProbabilities: make([]string, 0, 1001),

"%0.10f" gives ten decimal places: 1/3 is 0.3333333333, which is slightly less than the true 1/3. The bias is on the order of 1e-11 per rule and utterly irrelevant next to the kernel’s own 32-bit random value. But it does mean the split is “uniform to ten decimal places”, not exactly uniform — worth knowing if you are trying to explain a 0.0001% discrepancy in a traffic report.

The conntrack gate

The whole cascade runs in the kernel’s NAT table, and the kernel only consults NAT for the first packet of a connection. Subsequent packets of the same flow follow the conntrack entry and skip the NAT chains entirely. That single kernel behaviour is what makes this a per-connection load balancer without a single line of kube-proxy state.

The filter table is where the proxier has to be explicit about it. The jump chains from the built-in INPUT, FORWARD and OUTPUT chains into the kube-proxy chains are gated on the connection being new:

 go
 
 
// pkg/proxy/iptables/proxier.go:364-377
var iptablesJumpChains = []iptablesJumpChain{
	{utiliptables.TableFilter, kubeExternalServicesChain, utiliptables.ChainInput, "kubernetes externally-visible service portals", []string{"-m", "conntrack", "--ctstate", "NEW"}},
	{utiliptables.TableFilter, kubeExternalServicesChain, utiliptables.ChainForward, "kubernetes externally-visible service portals", []string{"-m", "conntrack", "--ctstate", "NEW"}},
	{utiliptables.TableFilter, kubeNodePortsChain, utiliptables.ChainInput, "kubernetes health check service ports", nil},
	{utiliptables.TableFilter, kubeServicesChain, utiliptables.ChainForward, "kubernetes service portals", []string{"-m", "conntrack", "--ctstate", "NEW"}},
	{utiliptables.TableFilter, kubeServicesChain, utiliptables.ChainOutput, "kubernetes service portals", []string{"-m", "conntrack", "--ctstate", "NEW"}},
	{utiliptables.TableFilter, kubeForwardChain, utiliptables.ChainForward, "kubernetes forwarding rules", nil},
	{utiliptables.TableFilter, kubeProxyFirewallChain, utiliptables.ChainInput, "kubernetes load balancer firewall", []string{"-m", "conntrack", "--ctstate", "NEW"}},
	{utiliptables.TableFilter, kubeProxyFirewallChain, utiliptables.ChainOutput, "kubernetes load balancer firewall", []string{"-m", "conntrack", "--ctstate", "NEW"}},
	{utiliptables.TableFilter, kubeProxyFirewallChain, utiliptables.ChainForward, "kubernetes load balancer firewall", []string{"-m", "conntrack", "--ctstate", "NEW"}},
	{utiliptables.TableNAT, kubeServicesChain, utiliptables.ChainOutput, "kubernetes service portals", nil},
	{utiliptables.TableNAT, kubeServicesChain, utiliptables.ChainPrerouting, "kubernetes service portals", nil},
	{utiliptables.TableNAT, kubePostroutingChain, utiliptables.ChainPostrouting, "kubernetes postrouting rules", nil},
}

The pattern is exact: every filter-table jump carries -m conntrack --ctstate NEW, and every NAT-table jump does not. The NAT table does not need the gate because conntrack already restricts NAT to the first packet of a flow — the kernel would not re-evaluate the statistic match for subsequent packets anyway, so the gate would be redundant there. The filter table is traversed by every packet, so the gate is load-bearing: without it, the KUBE-FW- and KUBE-SERVICES accept rules would be evaluated on every packet of every connection for no reason.

The two filter-table jumps without the gate are deliberate. KUBE-NODEPORTS jumping into INPUT is the health-check path, and KUBE-FORWARD into FORWARD handles the masquerade mark — neither is per-connection work.

Session affinity rides on the same chains

If the Service asks for sessionAffinity: ClientIP, the proxier prepends rules that use the recent match to pin a client to an endpoint. The structure above is the real code, but the shape is the important part: for each endpoint, a rule that checks whether this client’s IP was recently seen for that endpoint chain, and if so jumps straight to it.

The recent module keeps a small per-IP list per named set, keyed by the endpoint chain name, with a TTL equal to svcInfo.StickyMaxAgeSeconds(). The --reap flag means expired entries are cleaned up as the list is traversed. The affinity rules come before the probabilistic rules, so a pinned client never takes a chance on the lottery.

INVALID state is dropped

One more rule in the filter table, and it is not about balancing:

 go
 
 
// pkg/proxy/iptables/proxier.go:1316-1335
	// Drop the packets in INVALID state, which would potentially cause
	// unexpected connection reset if nf_conntrack_tcp_be_liberal is not set.
	// Ref: https://github.com/kubernetes/kubernetes/issues/74839
	// Ref: https://github.com/kubernetes/kubernetes/issues/117924
	if !proxier.conntrackTCPLiberal {
		rule := []string{
			"-A", string(kubeForwardChain),
			"-m", "conntrack",
			"--ctstate", "INVALID",
		}
		if proxier.nfAcctCounters[metrics.IPTablesCTStateInvalidDroppedNFAcctCounter] {
			rule = append(rule,
				"-m", "nfacct", "--nfacct-name", metrics.IPTablesCTStateInvalidDroppedNFAcctCounter,
			)
		}
		rule = append(rule,
			"-j", "DROP",
		)
		proxier.filterRules.Write(rule)
	}

A packet in the INVALID conntrack state is one the kernel could not classify — a stray ACK, a malformed handshake. If it reached the service chain it would pick an endpoint at random and get a connection reset, which looks exactly like the service flapping. Dropping it in KUBE-FORWARD before it ever reaches the lottery is what makes the failure mode legible. The proxier reads the nf_conntrack_tcp_be_liberal sysctl at startup and only installs the rule when the kernel is not in liberal mode:

 go
 
 
// pkg/proxy/iptables/proxier.go:237-244
	// Be conservative in what you do, be liberal in what you accept from others.
	// If it's non-zero, we mark only out of window RST segments as INVALID.
	// Ref: https://docs.kernel.org/networking/nf_conntrack-sysctl.html
	conntrackTCPLiberal := false
	if val, err := sysctl.GetSysctl(sysctlNFConntrackTCPBeLiberal); err == nil && val != 0 {
		conntrackTCPLiberal = true
		logger.Info("nf_conntrack_tcp_be_liberal set, not installing DROP rules for INVALID packets")
	}

Note the nfacct counter: when enabled, the drops are counted per node so you can see the rate of INVALID packets. That is the metric to look at if clients are reporting intermittent connection resets.

What it buys and what it costs

The iptables mode’s data path is remarkable for what it does not have:

No per-service state. Adding a Service adds rules. Removing one removes rules. There is no runtime cursor, no counter, no map.
No lock. The kernel evaluates the rules; the proxier is not in the path after iptables-restore.
No synchronization between nodes. Every node’s kernel picks independently, which is exactly what you want for spread.
O(1) state per service. The rules themselves are the state.

The costs are equally clear:

Unevenness at low connection counts. With 3 endpoints and 10 connections, the expected split is 3.3/3.3/3.3 and the actual split might be 6/3/1. Long-lived connections are pinned for their whole life, so the imbalance persists.
Rules scale with services times endpoints. A cluster with 5000 Services and 10 endpoints each has 50 000 endpoint chains. iptables-restore on that takes seconds and the kernel’s rule lookup gets slow — this is the motivation for the nftables and IPVS modes.
No health weighting. Endpoints are in or out of the Service; there is no gradual degradation based on endpoint load.
The lottery is per connection. Long-lived, high-bandwidth connections get an uneven share of the bytes, not just of the connections.

Wrapping up

The one thing to remember: the split is exact and stateless because the probabilities telescope. Rule i fires with probability 1/(n-i), the last rule fires unconditionally, and the product of the surviving probabilities gives every endpoint exactly 1/n. The randomness comes from the kernel’s -m statistic --mode random match, and the “per connection” property comes from conntrack, not from kube-proxy.

The part people get wrong: “kube-proxy load-balances round-robin.” It does not, and it does not even do weighted random. It does probabilistic per-connection balancing, so a single client making one connection gets one endpoint, and a client making ten connections gets a random split. If you are chasing uneven traffic with a long-lived connection, the lottery already picked and you will not get a better answer by waiting.

What to go do about it: look at the rules themselves with iptables-save -t nat | grep KUBE-SVC- on any node and read the --probability values — they should be 1/n, 1/(n-1), … and the last rule should have no probability at all. If the last rule has one, or the values do not match the endpoint count, the rules are stale. Watch kube_proxy_sync_proxy_rules_duration_seconds and kube_proxy_sync_full_proxy_rules_duration_seconds for the rule-count cliff — a full sync re-runs iptables-restore over the entire rule set, and that is what gets slow with tens of thousands of endpoint chains. If it is what is hurting you, switch to the nftables mode (--proxy-mode=nftables) or IPVS. For the INVALID-drop path, the counter is a kernel nfacct counter named ct_state_invalid_dropped_pkts (see pkg/proxy/metrics/metrics.go:157), readable with nfacct -l or from /proc/net/nf_conntrack_acct — look at it if clients are seeing connection resets that look like endpoint flapping. And check net.netfilter.nf_conntrack_tcp_be_liberal: when it is set, kube-proxy logs nf_conntrack_tcp_be_liberal set, not installing DROP rules for INVALID packets and skips that rule entirely.

50 Minus N/125: How the Scheduler Decides to Look at 5% of a 10,000-Node Cluster

Jay Hubbert — Wed, 21 Oct 2026 20:30:00 +0000

TL;DR: percentageOfNodesToScore: 0 is not “score every node” — it is the adaptive mode, and the scheduler computes the percentage itself as max(50 - N/125, 5) percent of the cluster, where N is the number of candidate nodes. On top of that sits an absolute floor of 100 feasible nodes (minFeasibleNodesToFind), so small clusters are always fully explored. A 10 000-node cluster therefore gets 5% — 500 nodes. A 125-node cluster gets 49% — but the 100-node floor means it evaluates 100 of 125. And the starting index rotates by the number of nodes processed each cycle, so no node is permanently starved of being examined.

Filtering every node in a 10 000-node cluster on every scheduling cycle is absurd. The Filter plugins are not free — node affinity, taints, resource fit, volume checks — and the number of pods waiting to be scheduled grows with the cluster. So the scheduler stops early. The interesting question is when it stops, and the answer is a hardcoded linear formula with two floors and a rotating cursor.

There are three separate mechanisms here and they are easy to conflate: the percentage that decides how many feasible nodes to hunt for, the absolute floor that overrides the percentage in small clusters, and the rotating start index that makes the sampling fair over time. Read separately, each is simple.

Zero is a mode, not a number

The default is literally zero:

 go
 
 
// pkg/scheduler/apis/config/types.go:261-265
const (
	// DefaultPercentageOfNodesToScore defines the percentage of nodes of all nodes
	// that once found feasible, the scheduler stops looking for more nodes.
	// A value of 0 means adaptive, meaning the scheduler figures out a proper default.
	DefaultPercentageOfNodesToScore = 0

and the field’s doc comment spells out what zero means:

 go
 
 
// pkg/scheduler/apis/config/types.go:62-70
	// PercentageOfNodesToScore is the percentage of all nodes that once found feasible
	// for running a pod, the scheduler stops its search for more feasible nodes in
	// the cluster. This helps improve scheduler's performance. Scheduler always tries to find
	// at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is.
	// Example: if the cluster size is 500 nodes and the value of this flag is 30,
	// then scheduler stops finding further feasible nodes once it finds 150 feasible ones.
	// When the value is 0, default percentage (5%--50% based on the size of the cluster) of the
	// nodes will be scored. It is overridden by profile level PercentageOfNodesToScore.
	PercentageOfNodesToScore *int32

“5%–50% based on the size of the cluster” is the whole of it. The default is a formula, not a constant.

Note the semantics carefully: the percentage is the number of feasible nodes to find. The scheduler keeps running Filter until it has collected that many feasible nodes, and only those get scored. The Filter pass itself may have examined far more than that many nodes, because a node that fails a filter is not “found”. In a cluster where 95% of the nodes cannot host your pod, the Filter pass visits nearly everything before it accumulates 500 feasible nodes — or gives up and returns a FitError.

The formula

Here is the entire computation:

 go
 
 
// pkg/scheduler/algorithm.go:518-543
// numFeasibleNodesToFind returns the number of feasible nodes that once found, the scheduler stops
// its search for more feasible nodes.
func (a *SchedulingAlgorithm) numFeasibleNodesToFind(percentageOfNodesToScore *int32, numAllNodes int32) (numNodes int32) {
	if numAllNodes < minFeasibleNodesToFind {
		return numAllNodes
	}

	// Use profile percentageOfNodesToScore if it's set. Otherwise, use global percentageOfNodesToScore.
	var percentage int32
	if percentageOfNodesToScore != nil {
		percentage = *percentageOfNodesToScore
	} else {
		percentage = a.percentageOfNodesToScore
	}

	if percentage == 0 {
		percentage = max(int32(50)-numAllNodes/125, minFeasibleNodesPercentageToFind)
	}

	numNodes = numAllNodes * percentage / 100
	if numNodes < minFeasibleNodesToFind {
		return minFeasibleNodesToFind
	}

	return numNodes
}

Four things happen in there, in order:

If the cluster is smaller than 100 nodes, return the cluster size — score everything, no sampling.
Pick the profile-level percentage if there is one, otherwise the global one.
If that percentage is zero, replace it with the adaptive value max(50 - N/125, 5).
Compute N * percentage / 100 and floor it at 100.

The 125 is a magic constant that encodes a design decision: the scheduler is willing to sample half the cluster when the cluster is tiny, and shrinks that linearly until it hits 5%, which it reaches at N = 5625. From that point on the percentage is fixed at 5% and the absolute number of nodes scored grows linearly with the cluster.

And the two constants come from schedule_one.go:

 go
 
 
// pkg/scheduler/schedule_one.go:45-62
const (
	// Percentage of plugin metrics to be sampled.
	pluginMetricsSamplePercent = 10
	// minFeasibleNodesToFind is the minimum number of nodes that would be scored
	// in each scheduling cycle. This is a semi-arbitrary value to ensure that a
	// certain minimum of nodes are checked for feasibility. This in turn helps
	// ensure a minimum level of spreading.
	minFeasibleNodesToFind = 100
	// minFeasibleNodesPercentageToFind is the minimum percentage of nodes that
	// would be scored in each scheduling cycle. This is a semi-arbitrary value
	// to ensure that a certain minimum of nodes are checked for feasibility.
	// This in turn helps ensure a minimum level of spreading.
	minFeasibleNodesPercentageToFind = 5
	// noteLengthLimit is a local copy of NoteLengthLimit in k8s.io/kubernetes/pkg/apis/core/validation, which the API
	// server enforces on notes the scheduler records. A local copy is safe because API validation limits will never
	// become lower than an already released version.
	noteLengthLimit = 1024
)

“semi-arbitrary” is the source’s own word. The reason for the floors is spreading: if the scheduler only ever looked at the same 5 feasible nodes, every pod would pile onto them. A floor of 100 feasible nodes guarantees a minimum diversity of candidates for the scoring plugins to choose from.

The arithmetic

percentage = max(50 - N/125, 5), integer division throughout:

N (nodes)	`N/125`	percentage	`N*pct/100`	nodes actually scored
100	0	50	50	100 (floor)
125	1	49	61	100 (floor)
205	1	49	100	100
250	2	48	120	120
1000	8	42	420	420
1250	10	40	500	500
5000	40	10	500	500
5625	45	5	281	281
10000	80	5 (floor)	500	500
20000	160	5 (floor)	1000	1000

Two regimes are visible. Between roughly 205 and 5625 nodes the absolute count scored falls — 420 at 1000 nodes, 281 at 5625 nodes — because the percentage is shrinking faster than the cluster is growing. Above 5625 nodes the percentage pins at 5% and the count grows linearly again.

Below about 205 nodes, the 100-node floor dominates and the scheduler is effectively examining the whole cluster. That is the point of the floor: the percentage formula would happily return 50 feasible nodes out of a 100-node cluster, which is a 50% sample of a cluster where the marginal cost of checking every node is trivial.

Note also the ordering of the two floors. The percentage floor (5) is applied before the node count is computed, and the node-count floor (100) is applied after. So minFeasibleNodesPercentageToFind can never produce fewer than 100 nodes in a cluster of 2000 or more, and minFeasibleNodesToFind can never produce fewer than 100 nodes in any cluster.

The one shortcut before all of this

numNodesToFind is the caller, and it has an even cheaper case:

 go
 
 
// pkg/scheduler/algorithm.go:504-516
// numNodesToFind decides how many feasible nodes Filter should collect, following
// kube-scheduler's percentageOfNodesToScore policy. A profile that neither filters
// through extenders nor scores has no use for a second candidate, so one is enough.
func (a *SchedulingAlgorithm) numNodesToFind(schedFramework framework.Framework, numAllNodes int32) int32 {
	if numAllNodes == 0 {
		return 0
	}

	if !hasExtenderFilters(schedFramework) && !hasScoring(schedFramework) {
		return 1
	}
	return a.numFeasibleNodesToFind(schedFramework.PercentageOfNodesToScore(), numAllNodes)
}

If the profile has no extender filters and no scoring plugins at all, one feasible node is enough — there is nothing to choose between candidates. The percentage machinery is skipped entirely.

The rotating start index

The percentage decides how many nodes to look at. The start index decides which ones, and it exists to stop the sampling from always hitting the same prefix of the node list.

 go
 
 
// pkg/scheduler/algorithm.go:75-79
	// nextStartNodeIndex is the starting index in the node list for the next filter
	// pass. Advancing this across scheduling cycles distributes evaluations uniformly
	// across all nodes when percentageOfNodesToScore halts the search early.
	nextStartNodeIndex int
}

The cursor advances by the number of nodes actually processed in the cycle:

 go
 
 
// pkg/scheduler/algorithm.go:364-373
	// The budget is computed from the candidate list, which PreFilter may have narrowed —
	// not from allNodes.
	numNodesToFind := a.numNodesToFind(schedFramework, int32(len(nodes)))
	feasibleNodes, err := a.findNodesThatPassFilters(ctx, schedFramework, state, pod, diagnosis, nodes, numNodesToFind)
	// always try to update the a.nextStartNodeIndex regardless of whether an error has occurred
	// this is helpful to make sure that all the nodes have a chance to be searched
	processedNodes := len(feasibleNodes) + diagnosis.NodeToStatus.Len()
	if len(preFilterOutcome.allNodes) > 0 {
		a.nextStartNodeIndex = (a.nextStartNodeIndex + processedNodes) % len(preFilterOutcome.allNodes)
	}

processedNodes is the feasible nodes found plus the nodes that were evaluated and rejected — diagnosis.NodeToStatus holds one entry per node the Filter pass touched. So the cursor moves by roughly the number of nodes examined, which means each cycle starts where the previous one stopped and the whole list is covered in N / processedNodes cycles.

The comment is worth noting: the update happens even if the filter pass errored. Otherwise a failing pod would freeze the cursor and the same prefix would be re-examined forever.

The cursor is consumed by the parallel filter loop:

 go
 
 
// pkg/scheduler/algorithm.go:453-457
	checkNode := func(i int) {
		// We check the nodes starting from where we left off in the previous scheduling cycle,
		// this is to make sure all nodes have the same chance of being examined across pods.
		nodeInfo := nodes[(a.nextStartNodeIndex+i)%numAllNodes]
		status := schedFramework.RunFilterPluginsWithNominatedPods(ctx, state, pod, nodeInfo)

One subtlety: nextStartNodeIndex is taken modulo len(preFilterOutcome.allNodes) but applied modulo len(nodes), and nodes is the candidate list that PreFilter may have narrowed. If PreFilter dropped half the cluster, the two lengths differ and the cursor’s phase relative to the candidate list drifts. That is harmless — the modulo in the index expression keeps it in range — but it means the rotation is exact only when PreFilter does not narrow the list.

The node tree hands out nodes in zone order

The list the cursor walks is not arbitrary: it comes from the node tree, which orders nodes round-robin across zones.

 go
 
 
// pkg/scheduler/backend/cache/node_tree.go:117-143
// list returns the list of names of the node. NodeTree iterates over zones and in each zone iterates
// over nodes in a round robin fashion.
func (nt *nodeTree) list() ([]string, error) {
	if len(nt.zones) == 0 {
		return nil, nil
	}
	nodesList := make([]string, 0, nt.numNodes)
	numExhaustedZones := 0
	nodeIndex := 0
	for len(nodesList) < nt.numNodes {
		if numExhaustedZones >= len(nt.zones) { // all zones are exhausted.
			return nodesList, errors.New("all zones exhausted before reaching count of nodes expected")
		}
		for zoneIndex := 0; zoneIndex < len(nt.zones); zoneIndex++ {
			na := nt.tree[nt.zones[zoneIndex]]
			if nodeIndex >= len(na) { // If the zone is exhausted, continue
				if nodeIndex == len(na) { // If it is the first time the zone is exhausted
					numExhaustedZones++
				}
				continue
			}
			nodesList = append(nodesList, na[nodeIndex])
		}
		nodeIndex++
	}
	return nodesList, nil
}

The outer loop advances the index within each zone; the inner loop walks the zones. So the list is zone0[0], zone1[0], zone2[0], ..., zone0[1], zone1[1], .... The practical consequence: the first 100 feasible nodes the scheduler finds are spread across every zone in the cluster, not clustered in whichever zone happens to sort first. This is a deliberate anti-bias measure layered on top of the percentage.

Combined with the rotating cursor, the sampling is: start somewhere new each cycle, and the node list itself is zone-balanced.

What the sampling costs you

The trade-off is explicit. Scoring 5% of a 10 000-node cluster instead of 100% means:

The best node may never be looked at. A pod that would fit perfectly on node 7421 might be scheduled onto node 3 because 7421 was not in the sample. The scoring plugins — NodeResourcesFit, ImageLocality, InterPodAffinity, TopologySpread — only ever rank the candidates they were given.
Topology spread degrades first. The PodTopologySpread plugin scores based on the domain skew it computes, and it computes it over the candidate set. With a small candidate set, the scheduler has less information about where the pod actually belongs.
The filter pass is not bounded by the percentage. Only the scoring is bounded. The Filter pass keeps running until it has found enough feasible nodes, so on a cluster where most nodes are unsuitable, the expensive part is not avoided at all.

That last point is the one that surprises people. percentageOfNodesToScore is not a knob on how much work the scheduler does. It is a knob on how many viable candidates the scoring plugins get to choose between.

Wrapping up

The one thing to remember: percentageOfNodesToScore: 0 is the adaptive formula max(50 - N/125, 5) percent, floored at 100 feasible nodes, applied to the candidate list after PreFilter narrowing — not to the raw cluster size. At 10 000 nodes that is 5%, or 500 nodes.

The part people get wrong: the percentage bounds the number of feasible nodes found and scored, not the number of nodes filtered. If your cluster is full of nodes that fail a taint or resource check, the Filter pass still walks nearly all of them and percentageOfNodesToScore buys you nothing. And setting it to 100 to “get better placements” is a real option with a real cost — the Filter and Score extension point latency is recorded in scheduler_framework_extension_point_duration_seconds, so you can see what it costs you before you commit.

What to go do about it: the setting lives in KubeSchedulerConfiguration.percentageOfNodesToScore and can be overridden per profile (KubeSchedulerProfile.percentageOfNodesToScore, which wins over the global value — see pkg/scheduler/algorithm.go:525-531). Set it to an explicit percentage to bypass the adaptive formula entirely, and set it to 100 for a small, latency-sensitive cluster where placement quality matters more than throughput. For a large cluster that keeps complaining about topology spread, raise it before you start disabling the spread plugin, and watch scheduler_pending_pods alongside scheduler_framework_extension_point_duration_seconds{extension_point="Score"} to see whether the extra scoring actually helps.

The Scheduler Never Locks the Cache: Generation Stamps and Copy-on-Write NodeInfos

Jay Hubbert — Wed, 21 Oct 2026 20:00:00 +0000

TL;DR: The scheduler cache is written by informer goroutines and read by the scheduling cycle, and the two never block each other. The cache keeps its NodeInfos in a doubly-linked list ordered by recency of update, and every NodeInfo carries a globally monotonic Generation counter. UpdateSnapshot walks that list from the head and stops at the first node whose Generation is not newer than the previous snapshot, cloning only the nodes above that line. Everything below is shared by pointer. A 5000-node cluster with two changed nodes copies two NodeInfos per cycle, not 5000.

The scheduler’s cache is the busiest data structure in the control plane. Every node add, node update, pod assignment, pod eviction and pod deletion lands there, and all of it has to be reflected in the next scheduling cycle before that cycle can make a decision. The obvious implementation is a deep copy of the whole cache at the top of every cycle: take a lock, copy every NodeInfo, release the lock, run the cycle against the copy.

That design is wrong on both ends. The copy is O(nodes x pods) per cycle, which in a large cluster is more work than the scheduling itself. And the copy is taken after the lock is released, so it is a snapshot of a moving target unless you hold the lock for the whole cycle, which would serialise the scheduler against every informer in the cluster.

Kubernetes solves it with a much older idea than copy-on-write: stamp the mutated object, keep the mutations ordered, and copy only the delta.

The cache is written by everyone and read by one

The cache is a cacheImpl with a single sync.RWMutex guarding all of its fields. Writers (the informers) take the write lock; the snapshot takes the write lock too, but only for the duration of the copy — the scheduling cycle itself runs entirely against the snapshot, lock-free:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:66-77
	// This mutex guards all fields within this cache struct.
	mu sync.RWMutex
	// a set of assumed pod keys.
	// The key could further be used to get an entry in podStates.
	assumedPods sets.Set[string]
	// a map from pod key to podState.
	podStates map[string]*podState
	nodes map[string]*nodeInfoListItem
	// headNode points to the most recently updated NodeInfo in "nodes". It is the
	// head of the linked list.
	headNode *nodeInfoListItem
	nodeTree *nodeTree

Note that nodes maps a node name to a *nodeInfoListItem, not to a *NodeInfo. The extra indirection is the list node. The map is for random access; the list is for ordered access, and it is the list that makes the delta copy possible.

A list ordered by recency, not by name

Every time a node’s NodeInfo changes, it is unlinked and pushed to the front of the list:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:131-157
// moveNodeInfoToHead moves a NodeInfo to the head of "cache.nodes" doubly
// linked list. The head is the most recently updated NodeInfo.
// We assume cache lock is already acquired.
func (cache *cacheImpl) moveNodeInfoToHead(logger klog.Logger, name string) {
	ni, ok := cache.nodes[name]
	if !ok {
		utilruntime.HandleErrorWithLogger(logger, nil, "No node info with given name found in the cache", "node", klog.KRef("", name))
		return
	}
	// if the node info list item is already at the head, we are done.
	if ni == cache.headNode {
		return
	}

	if ni.prev != nil {
		ni.prev.next = ni.next
	}
	if ni.next != nil {
		ni.next.prev = ni.prev
	}
	if cache.headNode != nil {
		cache.headNode.prev = ni
	}
	ni.next = cache.headNode
	ni.prev = nil
	cache.headNode = ni
}

The list node itself is deliberately thin — info, next, prev and nothing else — so the map and the list can point at the same object without copying it. The reordering is called from addPod and removePod, immediately after the NodeInfo itself is mutated. So the list is ordered by when the node was last touched, not by node name, not by zone, not by generation. That ordering is what lets the snapshot walk terminate early.

Generation stamps: monotonic, global, never reused

Each NodeInfo carries a Generation field that is bumped on every mutation:

 go
 
 
// pkg/scheduler/framework/types.go:208-218
	// PVCRefCounts contains a mapping of PVC names to the number of pods on the node using it.
	// Keys are in the format "namespace/name".
	PVCRefCounts map[string]int

	// Whenever NodeInfo changes, generation is bumped.
	// This is used to avoid cloning it if the object didn't change.
	Generation int64

	// DeclaredFeatures is a set of features published by the node
	DeclaredFeatures ndf.FeatureSet
}

and the bump comes from a single global atomic counter, not a per-node one:

 go
 
 
// pkg/scheduler/framework/types.go:526-532
// nextGeneration: Let's make sure history never forgets the name...
// Increments the generation number monotonically ensuring that generation numbers never collide.
// Collision of the generation numbers would be particularly problematic if a node was deleted and
// added back with the same name. See issue#63262.
func nextGeneration() int64 {
	return atomic.AddInt64(&generation, 1)
}

The comment about issue #63262 is the whole point. If generations were per-node, a node named node-1 could be deleted and re-added and end up with the same generation as before, which would make the scheduler believe nothing changed when everything had. A single global counter makes generations globally unique and totally ordered, so “is this node newer than the snapshot?” is a single integer comparison. nextGeneration() is called from update (pod added or removed), SetNode and RemoveNode. Nothing else bumps it.

The snapshot walk that stops early

This is the whole mechanism:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:240-271
	// Start from the head of the NodeInfo doubly linked list and update snapshot
	// of NodeInfos updated after the last snapshot.
	for node := cache.headNode; node != nil; node = node.next {
		if node.info.Generation <= snapshotGeneration {
			// all the nodes are updated before the existing snapshot. We are done.
			break
		}
		if np := node.info.Node(); np != nil {
			existing, ok := nodeSnapshot.nodeInfoMap[np.Name]
			if !ok {
				updateAllLists = true
				existing = &framework.NodeInfo{}
				nodeSnapshot.nodeInfoMap[np.Name] = existing
			}
			clone := node.info.SnapshotConcrete()
			// We track nodes that have pods with affinity, here we check if this node changed its
			// status from having pods with affinity to NOT having pods with affinity or the other
			// way around.
			if (len(existing.PodsWithAffinity) > 0) != (len(clone.PodsWithAffinity) > 0) {
				updateNodesHavePodsWithAffinity = true
			}
			if (len(existing.PodsWithRequiredAntiAffinity) > 0) != (len(clone.PodsWithRequiredAntiAffinity) > 0) {
				updateNodesHavePodsWithRequiredAntiAffinity = true
			}
			if (len(existing.PodsWithRequiredNonHostScopedAntiAffinity) > 0) != (len(clone.PodsWithRequiredNonHostScopedAntiAffinity) > 0) {
				updateNodesHavePodsWithRequiredNonHostScopedAntiAffinity = true
			}
			// We need to preserve the original pointer of the NodeInfo struct since it
			// is used in the NodeInfoList, which we may not update.
			*existing = *clone
		}
	}

Read the break condition carefully: <=. A node whose generation is exactly the snapshot generation is not copied — it was already captured. The invariant that makes the break sound is the combination of two facts:

The list is ordered by last-update time, and the generation counter is monotonic in time. Therefore generations along the list are non-increasing.
The snapshot generation is the generation of the head at the time of the previous snapshot, i.e. the maximum generation that existed then.

So walking from the head, every node with Generation > snapshotGeneration was updated after the last snapshot and must be copied. The first node with Generation <= snapshotGeneration was last touched at or before the previous snapshot, and so was every node behind it in the list. The break is correct.

Then the snapshot generation is bumped to the current head:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:272-296
	// Update the snapshot generation with the latest NodeInfo generation.
	if cache.headNode != nil {
		nodeSnapshot.generation = cache.headNode.info.Generation
	}

	// Comparing to pods in nodeTree.
	// Deleted nodes get removed from the tree, but they might remain in the nodes map
	// if they still have non-deleted Pods.
	if len(nodeSnapshot.nodeInfoMap) > cache.nodeTree.numNodes {
		cache.removeDeletedNodesFromSnapshot(nodeSnapshot)
		updateAllLists = true
	}

	// Apply the deltas for PVC reference count to the snapshot.
	// This no-op if the snapshot is built afresh i.e. updateAllLists=true
	if !updateAllLists {
		if err := cache.applyPVCRefCountDelta(nodeSnapshot); err != nil {
			logger.Error(err, "rebuilding node snapshot due to unexpected error from refreshing PVC ref counts")
			updateAllLists = true
		}
	}

	if updateAllLists || updateNodesHavePodsWithAffinity || updateNodesHavePodsWithRequiredAntiAffinity || updateNodesHavePodsWithRequiredNonHostScopedAntiAffinity {
		cache.updateNodeInfoSnapshotList(logger, nodeSnapshot, updateAllLists)
	}

The last line is the interesting one: the lists are only rebuilt when the set of nodes changed (updateAllLists) or when a node changed whether it has pods with affinity at all. Pods moving on and off a node does not rebuild them.

What a clone actually copies

SnapshotConcrete is not free, but it is not a deep copy of the world either:

 go
 
 
// pkg/scheduler/framework/types.go:304-316
// SnapshotConcrete returns a copy of this node, Except that ImageStates is copied without the Nodes field.
func (n *NodeInfo) SnapshotConcrete() *NodeInfo {
	clone := &NodeInfo{
		node: n.node,
		Requested: n.Requested.Clone(),
		NonZeroRequested: n.NonZeroRequested.Clone(),
		Allocatable: n.Allocatable.Clone(),
		UsedPorts: make(fwk.HostPortInfo),
		ImageStates: make(map[string]*fwk.ImageStateSummary),
		PVCRefCounts: make(map[string]int),
		Generation: n.Generation,
		DeclaredFeatures: n.DeclaredFeatures.Clone(),
	}

The *v1.Node object itself is shared by pointer (node: n.node), because node status updates mutate the cache’s copy of the node in place before the generation bump. The pod slices are copied by header, not by element. The resource vectors are cloned because AddPod/RemovePod mutate them in place.

The *existing = *clone line at the end of the walk is the trick that keeps pointer identity stable. nodeInfoList holds fwk.NodeInfo interfaces pointing at the snapshot’s NodeInfo structs. If the walk replaced those pointers, the list would dangle. Instead it copies the contents of the clone into the existing struct, so every list entry keeps pointing at the same object it always did.

The lists are only rebuilt when the topology changes

Three auxiliary lists hang off the snapshot, used by the inter-pod affinity plugin to avoid iterating over every node: havePodsWithAffinityNodeInfoList, havePodsWithRequiredAntiAffinityNodeInfoList and havePodsWithRequiredNonHostScopedAntiAffinityNodeInfoList. They are rebuilt from the node tree, and the expensive branch only runs when updateAll is true:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:363-374
func (cache *cacheImpl) updateNodeInfoSnapshotList(logger klog.Logger, snapshot *Snapshot, updateAll bool) {
	snapshot.havePodsWithAffinityNodeInfoList = make([]fwk.NodeInfo, 0, cache.nodeTree.numNodes)
	snapshot.havePodsWithRequiredAntiAffinityNodeInfoList = make([]fwk.NodeInfo, 0, cache.nodeTree.numNodes)
	snapshot.havePodsWithRequiredNonHostScopedAntiAffinityNodeInfoList = make([]fwk.NodeInfo, 0, cache.nodeTree.numNodes)
	if updateAll {
		snapshot.usedPVCRefCounts = make(map[string]int)
		// Take a snapshot of the nodes order in the tree
		snapshot.nodeInfoList = make([]fwk.NodeInfo, 0, cache.nodeTree.numNodes)
		nodesList, err := cache.nodeTree.list()
		if err != nil {
			utilruntime.HandleErrorWithLogger(logger, err, "Error occurred while retrieving the list of names of the nodes from node tree")
		}

and the loop body walks that list, appending each node to nodeInfoList and to whichever of the three affinity lists it belongs to, while summing its PVC ref counts into snapshot.usedPVCRefCounts.

updateAllLists is set to true in exactly two places: a node that is not in the snapshot map (a node was added), and the snapshot map being larger than the node tree (a node was deleted). So the O(nodes) rebuild happens on node churn, not on pod churn. In a cluster with a steady node count and busy pod scheduling, it essentially never runs.

PVC ref counts as a delta map

The usedPVCRefCounts map is the cluster-wide count of how many pods use each PVC. It answers “is this PVC already bound elsewhere?”, and a naive implementation recomputes it by walking every node and every PVC on it on every cycle — O(nodes x PVCs). Instead, the cache keeps a delta:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:92-97
	// pvcRefCountsDelta contains the delta of changes to PVCRefCounts since the last snapshot.
	// Keys are in the format "namespace/name". This data struct serves as an optimization for avoiding
	// burdensome PVC ref count aggregations during scheduler cycles. PVCRefCountsDelta holds the incoming
	// deltas from events handlers while within the scheduler cycle, we only apply and reset the delta to
	// avoid global re-calculation.
	pvcRefCountsDelta map[string]int

The delta is maintained at the point of mutation, before and after the pod is added or removed, so the change is what gets recorded:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:534-539
	// new_delta = old_delta + (PVCRefCounts_after − PVCRefCounts_before)
	cache.refreshPVCRefCountsDelta(n.info, -1)
	defer cache.refreshPVCRefCountsDelta(n.info, 1)

	n.info.AddPod(pod)
	cache.moveNodeInfoToHead(logger, pod.Spec.NodeName)

and applied to the snapshot at the end of the cycle:

 go
 
 
// pkg/scheduler/backend/cache/cache.go:1112-1127
// applyPVCRefCountDelta merges cache.pvcRefCountsDelta into the snapshot's
// PVC ref counts, removes entries that reach zero, and clears the delta.
func (cache *cacheImpl) applyPVCRefCountDelta(snapshot *Snapshot) error {
	for key, delta := range cache.pvcRefCountsDelta {
		snapshot.usedPVCRefCounts[key] += delta
		if refCount := snapshot.usedPVCRefCounts[key]; refCount <= 0 {
			if refCount < 0 {
				delete(snapshot.usedPVCRefCounts, key)
				return fmt.Errorf("PVC %s had negative ref count %v", key, refCount)
			}
			delete(snapshot.usedPVCRefCounts, key)
		}
	}
	cache.pvcRefCountsDelta = map[string]int{}
	return nil
}

The negative-count branch is a genuine assertion that the delta arithmetic is correct: a negative count means the delta and the snapshot disagreed about the cluster state, and the snapshot is thrown away and rebuilt from scratch (updateAllLists = true). The fast path has a self-check.

The arithmetic: a cluster of 5000 nodes each with 20 PVC references, with 2 nodes changing, does 40 map operations per cycle instead of 100 000. The delta map is O(changed PVCs), not O(nodes x PVCs).

Wrapping up

The one thing to remember: the scheduler does not copy the cache, it copies the delta. The Generation stamp tells you whether a NodeInfo changed; the recency-ordered list tells you where the changed ones end. The snapshot walk breaks on the first unchanged node and everything behind it is shared by pointer with the live cache.

The part people get wrong: the break condition is only sound because the list is ordered by last update time and generations are globally monotonic. If you ever reorder that list — by zone, by name, by allocatable — the early break silently starts missing changes, and the scheduler will schedule against stale node state with no error anywhere. UpdateSnapshot also has a consistency check at pkg/scheduler/backend/cache/cache.go:298 that compares len(snapshot.nodeInfoList) against cache.nodeTree.numNodes; if you see snapshot state is not consistent in the scheduler log, that is this invariant breaking.

What to go do about it: measure the cycle cost before you tune anything — scheduler_framework_extension_point_duration_seconds{extension_point="Filter"} and the scheduler_pending_pods gauge tell you whether the scheduler is cycle-bound or cluster-bound. If the cache is the bottleneck, look at how often nodes churn in your cluster — node churn is what forces the O(nodes) updateNodeInfoSnapshotList rebuild — and at the pod density per node, which is what SnapshotConcrete actually copies.