Above 1000 Endpoints, kube-proxy Deletes Its Own Comments

TL;DR: The iptables proxier counts the total number of endpoints across all services. Above 1000, it flips into “large cluster mode” and stops doing three things: it stops emitting -m comment --comment "<svc>" on every rule, it stops performing the hourly full resync, and it stops garbage-collecting stale chains on every sync. The comments alone were about 40% of the iptables-restore payload in a 10k-endpoint cluster. The trick that makes the partial sync work is a discard line buffer: the proxier still computes every rule, but throws the bytes away and keeps only the count, so the metrics stay honest.

kube-proxy’s job is to translate Services and EndpointSlices into iptables rules. The rules are installed with iptables-restore, in one atomic transaction, from a buffer the proxier builds up in memory. In a small cluster that buffer is a few kilobytes. In a large cluster it is tens of megabytes, and the cost of building and restoring it is paid on every sync.

The proxier has to make trade-offs, and it makes them silently. This is the part that bites people: the same cluster produces different rules depending on how many endpoints it has, and nothing tells you that the switch happened.

The threshold

It is a single constant, and the comment says exactly what it is for:

// pkg/proxy/iptables/proxier.go:84
	// largeClusterEndpointsThreshold is the number of endpoints at which
	// we switch into "large cluster mode" and optimize for iptables
	// performance over iptables debuggability
	largeClusterEndpointsThreshold = 1000
)

and the mode is a boolean on the proxier, set at the start of every sync:

// pkg/proxy/iptables/proxier.go:181
	// largeClusterMode is set at the beginning of syncProxyRules if we are
	// going to end up outputting "lots" of iptables rules and so we need to
	// optimize for performance over debuggability.
	largeClusterMode bool

The count itself is a sum over all services of their endpoint counts — note that it is endpoints, not services:

// pkg/proxy/iptables/proxier.go:799
	// Compute total number of endpoint chains across all services
	// to get a sense of how big the cluster is.
	totalEndpoints := 0
	for svcName := range proxier.svcPortMap {
		totalEndpoints += len(proxier.endpointsMap[svcName])
	}
	proxier.largeClusterMode = (totalEndpoints > largeClusterEndpointsThreshold)

So a cluster with 300 Services and 4 endpoints each crosses the threshold at 1200 endpoints. A cluster with one Service and 1001 endpoints crosses it too. The mode is recomputed on every sync, so a cluster that grows past 1000 endpoints silently changes behaviour.

The comments go away

The first casualty is the human-readable comment attached to each rule. This is the single biggest contributor to the payload size, and the reason is obvious in hindsight: every service port rule, every endpoint chain rule, every external-service rule carries the service name as a comment string.

// pkg/proxy/iptables/proxier.go:601
func (proxier *Proxier) appendServiceCommentLocked(args []string, svcName string) []string {
	// Not printing these comments, can reduce size of iptables (in case of large
	// number of endpoints) even by 40%+. So if total number of endpoint chains
	// is large enough, we simply drop those comments.
	if proxier.largeClusterMode {
		return args
	}
	return append(args, "-m", "comment", "--comment", svcName)
}

“even by 40%+” — that is the proxier’s own estimate of the saving. It is the price of being able to read iptables-save output and know which rule belongs to which service, which is the single most useful thing you can do when debugging a Service that is not receiving traffic.

Note that the other comments are not suppressed. The rules that are commented unconditionally — "kubernetes service traffic requiring SNAT", "kubernetes service nodeports; NOTE: this must be the last rule in this chain", "kubernetes forwarding rules" — are fixed-cost, written once, and carry structural information that is useful even in large clusters. Only the per-service comments are dropped.

The hourly full sync goes away

A full sync rebuilds every rule for every service from scratch. It is the correct thing to do periodically, because it repairs drift — rules that were edited by hand, chains that some other tool flushed. But it is also the most expensive thing the proxier does.

// pkg/proxy/iptables/proxier.go:638
	doFullSync := proxier.needFullSync ||
		// Avoid regular full syncs for large clusters.
		((time.Since(proxier.lastFullSync) > proxyutil.FullSyncPeriod) && !proxier.largeClusterMode)

proxyutil.FullSyncPeriod is one hour:

// pkg/proxy/util/utils.go:41
	// FullSyncPeriod is iptables and nftables proxier full sync period
	FullSyncPeriod = 1 * time.Hour

So in large cluster mode the proxier only does a full sync when needFullSync is set — that is, when the proxier is (re)started or when it detects that the underlying tables have changed wholesale. Partial syncs still happen, they just only rewrite the chains that actually changed.

The trade-off is real: drift accumulates. If something else on the host flushes or edits the KUBE-SERVICES chain, the proxier will not repair it until the next restart. That is a deliberate choice — the alternative is an hourly multi-second iptables-restore on every node in a large cluster, which is exactly the kind of periodic load that makes nodes flap.

Stale chains stop being collected

This is the one most people do not know about, because it has no visible effect on the rules that are installed. iptables chains that are no longer used still exist until someone deletes them, and the proxier does that by running iptables-save, diffing the chain list, and emitting -X rules for the ones it no longer wants:

// pkg/proxy/iptables/proxier.go:1246
	// Delete chains no longer in use. Since "iptables-save" can take several seconds
	// to run on hosts with lots of iptables rules, we don't bother to do this on
	// every sync in large clusters. (Stale chains will not be referenced by any
	// active rules, so they're harmless other than taking up memory.)
	deletedChains := 0
	if !proxier.largeClusterMode || time.Since(proxier.lastIPTablesCleanup) > proxier.syncPeriod {

The condition is worth reading twice. In small cluster mode, the cleanup runs on every sync. In large cluster mode, it runs only if it has not run within syncPeriod. So large clusters do not accumulate stale chains without bound — they just accumulate them a bit more.

The comment’s parenthetical is the justification: a stale chain is not referenced by any active rule, so it is harmless except for the memory it occupies. And the cost of the alternative — running iptables-save, which “can take several seconds on hosts with lots of iptables rules” — is paid on every sync otherwise. A proxier that runs iptables-save every 30 seconds on every node is a proxier that is periodically unable to do its job.

The discard line buffer

Here is the genuinely elegant part. A partial sync must not re-emit the chains that did not change. But the proxier cannot know whether a chain changed until it has computed the rules for it — the computation is a single big function that walks every service and every endpoint.

So the proxier computes everything and then throws most of it away. The mechanism is a LineBuffer implementation that counts writes and discards the bytes:

// pkg/proxy/util/linebuffer.go:121
type discardLineBuffer struct {
	lines int
}

// NewDiscardLineBuffer returns a dummy LineBuffer that counts the number of writes but
// throws away the data. (This is used for iptables proxy partial syncs, to keep track of
// how many rules we managed to avoid having to sync.)
func NewDiscardLineBuffer() LineBuffer {
	return &discardLineBuffer{}
}

// Write is part of LineBuffer
func (buf *discardLineBuffer) Write(args ...interface{}) {
	buf.lines++
}

// WriteBytes is part of LineBuffer
func (buf *discardLineBuffer) WriteBytes(bytes []byte) {
	buf.lines++
}

Two of these buffers are created at the top of every sync:

// pkg/proxy/iptables/proxier.go:726
	skippedNatChains := proxyutil.NewDiscardLineBuffer()
	skippedNatRules := proxyutil.NewDiscardLineBuffer()

and then, per service, the destination buffer is swapped. The service’s chains are written into the discard buffer instead of the real one:

// pkg/proxy/iptables/proxier.go:1060
		// If the SVC/SVL/EXT/FW/SEP chains have not changed since the last sync
		// then we can omit them from the restore input. However, we have to still
		// figure out how many chains we _would_ have written, to make the metrics
		// come out right, so we just compute them and throw them away.
		if !doFullSync && !serviceUpdateResult.UpdatedServices.Has(svcName.NamespacedName) && !endpointUpdateResult.UpdatedServices.Has(svcName.NamespacedName) {
			natChains = skippedNatChains
			natRules = skippedNatRules
		}

This is why the buffers are variables rather than struct fields: natChains and natRules are local variables that every subsequent Write call in the loop body goes through. Swapping them redirects the writes.

And the reason it is done at all is the metrics. The proxier publishes the total number of rules and the number of rules in the last sync:

// pkg/proxy/iptables/proxier.go:1357
	metrics.IPTablesRulesTotal.WithLabelValues(string(utiliptables.TableFilter), string(proxier.ipFamily)).Set(float64(proxier.filterRules.Lines()))
	metrics.IPTablesRulesLastSync.WithLabelValues(string(utiliptables.TableFilter), string(proxier.ipFamily)).Set(float64(proxier.filterRules.Lines()))
	metrics.IPTablesRulesTotal.WithLabelValues(string(utiliptables.TableNAT), string(proxier.ipFamily)).Set(float64(proxier.natRules.Lines() + skippedNatRules.Lines() - deletedChains))
	metrics.IPTablesRulesLastSync.WithLabelValues(string(utiliptables.TableNAT), string(proxier.ipFamily)).Set(float64(proxier.natRules.Lines() - deletedChains))

Look at the arithmetic: natRules.Lines() is what was actually written, skippedNatRules.Lines() is what was computed and thrown away, and deletedChains is what was removed. The total is the sum of the first two minus the third. Without the discard buffer, skippedNatRules.Lines() would be zero and the total would collapse to the size of the last partial sync — the metric would say “12 rules” for a cluster with 12 000.

The - deletedChains term is there because the metric is a count of live rules, and the -X rules that delete chains are themselves written into natRules but do not count as rules.

The masquerade mark

While we are in the NAT table: the masquerade mark is a bit in the packet mark, and the way it is used has a subtlety worth understanding, because it is a classic “packet re-enters the stack” problem.

The mark is a single bit of the fwmark space:

// pkg/proxy/iptables/proxier.go:251
	// Generate the masquerade mark to use for SNAT rules.
	masqueradeValue := 1 << uint(*config.IPTables.MasqueradeBit)
	masqueradeMark := fmt.Sprintf("%#08x", masqueradeValue)
	logger.V(2).Info("Using iptables mark for masquerade", "mark", masqueradeMark)

with the bit index defaulting to 14:

// cmd/kube-proxy/app/options.go:134
	fs.Int32Var(o.config.IPTables.MasqueradeBit, "iptables-masquerade-bit", ptr.Deref(o.config.IPTables.MasqueradeBit, 14), "If using the iptables or ipvs proxy mode, the bit of the fwmark space to mark packets requiring SNAT with.  Must be within the range [0, 31].")

so the default mark is 0x4000. The rules that use it:

// pkg/proxy/iptables/proxier.go:741
	proxier.natRules.Write(
		"-A", string(kubePostroutingChain),
		"-m", "mark", "!", "--mark", fmt.Sprintf("%s/%s", proxier.masqueradeMark, proxier.masqueradeMark),
		"-j", "RETURN",
	)
	// Clear the mark to avoid re-masquerading if the packet re-traverses the network stack.
	proxier.natRules.Write(
		"-A", string(kubePostroutingChain),
		"-j", "MARK", "--xor-mark", proxier.masqueradeMark,
	)
	proxier.natRules.Write(
		"-A", string(kubePostroutingChain),
		"-m", "comment", "--comment", `"kubernetes service traffic requiring SNAT"`,
		"-j", "MASQUERADE",
		"--random-fully",
	)

	// Install the kubernetes-specific masquerade mark rule. We use a whole chain for
	// this so that it is easier to flush and change, for example if the mark
	// value should ever change.
	proxier.natRules.Write(
		"-A", string(kubeMarkMasqChain),
		"-j", "MARK", "--or-mark", proxier.masqueradeMark,
	)

Three rules, three jobs:

1. RETURN if the mark is already set. A packet that has already been marked for masquerade is not masqueraded again. This is the idempotence guard for the case where the packet re-enters the network stack.
2. MARK --xor-mark. XOR, not SET or --set-xmark. XOR clears the bit if it was set and sets it if it was clear — it is a toggle. The comment says it plainly: “Clear the mark to avoid re-masquerading if the packet re-traverses the network stack.” Since the first rule already returned for marked packets, reaching this rule means the bit was clear, and toggling it sets it. But if the packet somehow comes back through POSTROUTING a second time, the bit is set, the first rule does not match (the ! is negated), and the XOR clears it. The mark is a one-shot token.
3. MASQUERADE --random-fully. --random-fully is the important bit. The kernel’s default port selection for MASQUERADE is sequential: it picks the next port after the last one it used, which makes the source port of outgoing connections predictable and creates a well-known vulnerability for services that treat the source port as a weak identifier (SSH host keys, some TLS stacks). --random-fully asks the kernel to pick a genuinely random source port instead.

The KUBE-MARK-MASQ chain exists purely so the mark value can be changed without rewriting the whole POSTROUTING chain — the same reason the postrouting rules are in their own chain at all.

What it costs you

Large cluster mode is a debuggability-for-performance trade, and it is applied silently. Concretely, in a cluster above the threshold:

• iptables-save output no longer tells you which service a rule belongs to. You get chain names (KUBE-SVC-XXXXX) and the mapping lives in the API server, not in the ruleset.
• Stale chains accumulate between cleanups, so the ruleset grows.
• Manual edits to the KUBE- chains are not repaired by the hourly full sync, because there is no hourly full sync.

What you get back is a proxier that does not spend several seconds in iptables-save every thirty seconds, and whose iptables-restore payload is a fraction of the size.

The threshold is not configurable. There is no flag for it: 1000 is a constant in pkg/proxy/iptables/proxier.go:87, and the only way to change the behaviour is to change the number of endpoints in the cluster or to patch the source.

Wrapping up

The one thing to remember: the iptables proxier switches behaviour at 1000 total endpoints (pkg/proxy/iptables/proxier.go:87, totalEndpoints > largeClusterEndpointsThreshold at :805), and in large cluster mode it drops per-service comments, skips the hourly full sync (proxyutil.FullSyncPeriod = 1 * time.Hour), and defers stale-chain cleanup to at most once per syncPeriod.

The part people get wrong: that the rules are the same regardless of cluster size. They are not. iptables-save on a 2000-endpoint cluster shows no --comment on the service rules, and the chain GC no longer runs every sync. If you are diffing iptables-save output between two nodes or two clusters, the size difference is often this, not a bug.

What to go do about it: when debugging a Service in a large cluster, do not expect the comments — map the chain names back instead (kubectl get svc -o wide, then correlate with KUBE-SVC-<hash> in the ruleset, where the hash is the service’s spec.clusterIP rendered as a chain-safe string). To confirm which mode you are in, compare the rule counts in kube-proxy_iptables_rules_total against what iptables-save actually shows: in large cluster mode the total includes the discarded rules, which is exactly why the discard buffer exists (pkg/proxy/util/linebuffer.go:121, and the arithmetic at pkg/proxy/iptables/proxier.go:1359). And if you need the masquerade mark to be something else, it is --iptables-masquerade-bit (default 14, i.e. mark 0x4000, valid range [0, 31]) — change it and restart kube-proxy, not the node.