Opensource internals

One Rule Per Service: How nftables Replaced kube-proxy's Chain of Chains

13 min read Jay Hubbert

The iptables proxier needs N chains and N rules per service; the nftables proxier emits one rule with a numgen random mod N map, so the per-service rule count is O(1).

TL;DR: In iptables mode, a service with N endpoints needs a KUBE-SVC-XXXX chain plus N KUBE-SEP-XXXX chains, and the load balancing is emulated with a cascade of -m statistic --mode random --probability rules that only add up to a uniform distribution once you reach the bottom. In nftables mode, kube-proxy emits one rule per service: numgen random mod N map { 0 : ip.port, 1 : ip.port, ... }. The map is a kernel hash table, so the lookup is a hash, and the distribution is uniform by construction instead of by accumulation.

Every service in a Kubernetes cluster is a load balancer, and every load balancer has to answer the same question: given this packet, which backend? The iptables proxier answers it with a chain of chain of chain. The nftables proxier answers it with a hash map lookup and a uniform random number. The difference is not cosmetic - it is the difference between O(endpoints) rules per service and O(1). That matters because the number of services grows with the number of teams, the number of endpoints per service grows with the number of replicas, and every rule in the iptables layout costs a linear scan of the chain at packet time.

The chain of chains

The iptables proxier names five kinds of chain, and every service gets one of each:

go 
// pkg/proxy/iptables/proxier.go:538

const (
	servicePortPolicyClusterChainNamePrefix = "KUBE-SVC-"
	servicePortPolicyLocalChainNamePrefix   = "KUBE-SVL-"
	serviceFirewallChainNamePrefix          = "KUBE-FW-"
	serviceExternalChainNamePrefix          = "KUBE-EXT-"
	servicePortEndpointChainNamePrefix      = "KUBE-SEP-"
)

KUBE-SVC- is the service’s dispatch chain. It contains one rule per endpoint, and each rule jumps to that endpoint’s own KUBE-SEP- chain, which holds the actual -j DNAT. The suffix is a 16-character base32 truncation of a SHA-256 over the service port name and protocol, because iptables chain names are limited to 28 characters. Here is the entire load balancing logic of the iptables proxier:

go 
// pkg/proxy/iptables/proxier.go:1452

	// Now write loadbalancing rules.
	numEndpoints := len(endpoints)
	for i, ep := range endpoints {
		epInfo, ok := ep.(*endpointInfo)
		if !ok {
			continue
		}
		comment := fmt.Sprintf(`"%s -> %s"`, svcPortNameString, epInfo.String())

		args = append(args[:0], "-A", string(svcChain))
		args = proxier.appendServiceCommentLocked(args, comment)
		if i < (numEndpoints - 1) {
			// Each rule is a probabilistic match.
			args = append(args,
				"-m", "statistic",
				"--mode", "random",
				"--probability", proxier.probability(numEndpoints-i))
		}
		// The final (or only if n == 1) rule is a guaranteed match.
		natRules.Write(args, "-j", string(epInfo.ChainName))
	}

Read the probability values: for N endpoints, the first rule fires with probability 1/N, the second with probability 1/(N-1), the third with probability 1/(N-2), and the last one always fires. The unconditional probability of reaching the k-th rule is 1/N * 1/(N-1) * … which is uniform across the N endpoints - but only because the multiply-out happens implicitly as you fall through. Each of the N rules is a separate kernel matcher, and the packet walks the chain until one fires.

The verdict map

nftables has a first-class data type that iptables does not: a map, a hash table whose values can be verdict-like objects (goto, drop, reject) as well as data. kube-proxy declares several of them for dispatch, all in the nat table setup (pkg/proxy/nftables/proxier.go:723):

go 
// pkg/proxy/nftables/proxier.go:723
	// Set up service dispatch
	tx.Add(&knftables.Map{
		Name:    serviceIPsMap,
		Type:    ipvX_addr + " . inet_proto . inet_service : verdict",
		Comment: ptr.To("ClusterIP, ExternalIP and LoadBalancer IP traffic"),
	})
	tx.Add(&knftables.Map{
		Name:    serviceNodePortsMap,
		Type:    "inet_proto . inet_service : verdict",
		Comment: ptr.To("NodePort traffic"),
	})

The key is a concatenation of three fields: destination address, IP protocol, and destination port. The value is a verdict - in practice a goto to a per-service chain. So the whole dispatch table for every service in the cluster is a single nftables map, and the rule that consults it is a single line:

go 
// pkg/proxy/nftables/proxier.go:761
	tx.Add(&knftables.Rule{
		Chain: servicesChain,
		Rule: knftables.Concat(
			ipX, "daddr", ".", "meta l4proto", ".", "th dport",
			"vmap", "@", serviceIPsMap,
		),
	})

vmap @service-ips is the verdict-map lookup. The packet’s daddr, l4proto and th dport are concatenated into a key, the key is hashed, the hash is looked up in a kernel hash table, and the value is a verdict the rule engine applies - a goto to the per-service chain. One rule, one hash lookup, for every service in the cluster. The per-service chain still exists

  • nftables mode keeps one per service and per endpoint, because externalTrafficPolicy: Local needs somewhere to live - but it is reached through a map lookup and holds exactly one rule for the endpoints.

One rule per service

The dispatch from the service chain to the endpoints is where the O(N) -> O(1) collapse happens. Here is the whole thing:

go 
// pkg/proxy/nftables/proxier.go:1963
	tx.Add(&knftables.Rule{
		Chain: svcChain,
		Rule: knftables.Concat(
			"meta l4proto", strings.ToLower(string(svcInfo.Protocol())),
			"dnat", ipX, "addr . port to",
			"numgen random mod", len(endpoints), "map",
			"{", elements, "}",
		),
	})

Read that left to right. meta l4proto tcp matches the protocol. dnat ip addr . port to says the destination is a concatenation of an IPv4 address and a port. Then numgen random mod N map { 0 : 10.0.0.5.8080, 1 : 10.0.0.6.8080, ... }: the kernel generates a random number, takes it modulo N, and uses the result as the index into an inline map whose values are the endpoint address-and-port pairs.

That is a uniform random choice over N endpoints, computed in one expression. No probability cascade, no fall-through, no per-endpoint chain jump. The elements are built by iterating the endpoint list and emitting index : ip.port, separated by commas (pkg/proxy/nftables/proxier.go:1949).

Note that the map is inline in the rule, not a named nftables map object. That is a deliberate trade-off, and the code says so out loud - inserting elements into a named nftables map is a O(n) (or worse) operation in the kernel, so for the common case of exactly one endpoint, kube-proxy skips the map entirely:

go 
// pkg/proxy/nftables/proxier.go:1929
	// Special case for single endpoint services
	// Inserting maps into nftables is a O(n) (or worse) operation, so we want to avoid it for single endpoint services.
	if len(endpoints) == 1 {
		ep := endpoints[0]
		epInfo, ok := ep.(*endpointInfo)
		if !ok {
			return
		}

		tx.Add(&knftables.Rule{
			Chain: svcChain,
			Rule: knftables.Concat(
				"meta l4proto", strings.ToLower(string(svcInfo.Protocol())),
				"dnat to", net.JoinHostPort(epInfo.IP(), strconv.Itoa(epInfo.Port())),
			),
		})

		return
	}

That single-endpoint fast path is the most common case in most clusters, and it produces literally one rule with no map, no numgen, and no random number generator at all.

Session affinity needs a different shape

ClientIP session affinity cannot use the DNAT map, because the pin has to be looked up before the DNAT decision, and the pin is keyed on the source address. So the affinity variant of the service chain uses a verdict map instead, whose values are chain jumps rather than addresses:

go 
// pkg/proxy/nftables/proxier.go:1988
		tx.Add(&knftables.Rule{
			Chain: svcChain,
			Rule: knftables.Concat(
				ipX, "saddr", "@", epInfo.affinitySetName,
				"goto", epInfo.chainName,
			),
		})

		elements = append(elements,
			strconv.Itoa(i), ":", "goto", epInfo.chainName,
		)
		if i != len(endpoints)-1 {
			elements = append(elements, ",")
		}
	}
	// Now write the vmap, for the case where no affinity rule matched
	tx.Add(&knftables.Rule{
		Chain: svcChain,
		Rule: knftables.Concat(
			"numgen random mod", len(endpoints), "vmap",
			"{", elements, "}",
		),
	})

So an affinity-enabled service has N affinity rules (one per endpoint, each matching a source address against a dynamic, timeout-bearing set), plus one numgen random mod N vmap fallback rule for clients with no pin. The vmap values are goto verdicts pointing at the per-endpoint chain, and the per-endpoint chain does the DNAT and updates the affinity set. The affinity set itself is a dynamic nftables set with a timeout, created with the DynamicFlag and TimeoutFlag set and a timeout equal to the service’s sessionAffinityConfig.clientIP.timeoutSeconds. The DynamicFlag comment in the source is worth repeating: the nftables docs say “dynamic” is only needed for sets containing stateful objects such as counters, but on RHEL8 the set gets mutated to have it later, and the next tx.Add() of the set fails because it looks like a flag change.

A service that loses its endpoints flips one element

The other thing the chain-of-chains layout made expensive was the negative case: a service with no endpoints. In iptables mode that means generating a reject rule per service and tearing it down later. In nftables mode there is a dedicated verdict map for exactly this, and the service’s fate is one map element:

go 
// pkg/proxy/nftables/proxier.go:607
	// Set up "no endpoints" drop/reject handling
	tx.Add(&knftables.Map{
		Name:    noEndpointServicesMap,
		Type:    ipvX_addr + " . inet_proto . inet_service : verdict",
		Comment: ptr.To("vmap to drop or reject packets to services with no endpoints"),
	})
	tx.Add(&knftables.Map{
		Name:    noEndpointNodePortsMap,
		Type:    "inet_proto . inet_service : verdict",
		Comment: ptr.To("vmap to drop or reject packets to service nodeports with no endpoints"),
	})

The check itself is one rule, hooked into the filter table’s input, forward and output chains for new connections - ip daddr . meta l4proto . th dport vmap @no-endpoint-services (pkg/proxy/nftables/proxier.go:631).

And the element itself is written with the verdict the service deserves. When a service has no endpoints at all, both internal and external traffic is sent to a shared reject chain; when it has endpoints but none that are local and the internal traffic policy is Local, internal traffic is dropped while external traffic still works:

go 
// pkg/proxy/nftables/proxier.go:1417
		var internalTrafficFilterVerdict, externalTrafficFilterVerdict string
		if !hasEndpoints {
			// The service has no endpoints at all; hasInternalEndpoints and
			// hasExternalEndpoints will also be false, and we will not
			// generate any chains in the "nat" table for the service; only
			// rules in the "filter" table rejecting incoming packets for
			// the service's IPs.
			internalTrafficFilterVerdict = fmt.Sprintf("goto %s", rejectChain)
			externalTrafficFilterVerdict = fmt.Sprintf("goto %s", rejectChain)
		} else {
			if !hasInternalEndpoints {
				// The internalTrafficPolicy is "Local" but there are no local
				// endpoints. Traffic to the clusterIP will be dropped, but
				// external traffic may still be accepted.
				internalTrafficFilterVerdict = "drop"

So a service that goes from three endpoints to zero does not cause a rule synthesis or a rule teardown. It causes one map element to be deleted and one map element to be inserted, with the verdict flipping from “goto the service chain” to “goto reject-chain” - the element is written to no-endpoint-services with the key <clusterIP> <protocol> <port> and the value being the verdict (pkg/proxy/nftables/proxier.go:1449).

The ensureElem helper is what makes this cheap, and it encodes the only slightly clever part of the design: it compares the desired value against the value it last read out of the kernel, and only emits a delete+add when the value actually changed (pkg/proxy/nftables/proxier.go:1121):

go 
// pkg/proxy/nftables/proxier.go:1121
	if exists {
		if existingValue != newValue {
			// value is different, delete and re-add
			tx.Delete(elem)
			tx.Add(elem)
			s.elements[newKey] = newValue
		}
		delete(s.leftoverKeys, newKey)
	} else {
		tx.Add(elem)
		s.elements[newKey] = newValue
	}

The userspace localhost NodePort proxy

There is one thing nftables cannot do that iptables can, and it has a whole subsystem bolted on for it: NodePorts on localhost.

In iptables mode, kube-proxy turns on net/ipv4/conf/all/route_localnet, a kernel hack that makes the loopback interface route non-local addresses. That is what allows curl 127.0.0.1:30000 to reach a NodePort. nftables mode does not use route_localnet, so when the node port addresses include an explicit loopback address, kube-proxy falls back to a userspace TCP proxy - the KubeProxyNFTablesLocalhostNodePorts feature gate plus useLocalhostNodePortProxy(nodePortAddresses) at pkg/proxy/nftables/proxier.go:293.

The proxy listens on 127.0.0.1:<nodePort> (or ::1) for TCP only, and pipes bytes to a backend picked at random, with a five second dial timeout (const dialTimeout = 5 * time.Second, pkg/proxy/localnodeportproxy/proxy.go:37):

go 
// pkg/proxy/localnodeportproxy/proxy.go:223
func (l *nodePortListener) handleTCPConn(ctx context.Context, clientConn net.Conn) {
	defer clientConn.Close() //nolint:errcheck

	ep := l.pickEndpoint()
	backendConn, err := net.DialTimeout("tcp", ep, dialTimeout)
	if err != nil {
		l.logger.V(4).Info("Failed to connect to backend", "err", err, "protocol", l.protocol, "port", l.port, "endpoint", ep)
		return
	}

implemented as a sliding window - the timer is renewed on every connection - and the source comments are unusually honest about what that means:

go 
// pkg/proxy/localnodeportproxy/proxy.go:268
	// Session affinity is implemented as a sliding window (renewing the expiration
	// timer on every connection). Note that since all localhost traffic shares the
	// same source IP (127.0.0.1 or ::1), any active connection from localhost will
	// keep this pin alive indefinitely for all local clients. This matches the
	// behavior in standard iptables/nftables modes.

So “ClientIP affinity” for localhost traffic is effectively “affinity to whichever endpoint the first connection happened to pick”, and it never expires as long as something on the node keeps connecting. That is not a bug in the nftables port; it is a property of keying affinity on a source address that every local client shares. The protocols the userspace proxy cannot serve - UDP, SCTP - are rejected explicitly and counted in the kubeproxy_nftables_localhost_nodeport_rejected_packets_total metric.

Trade-offs

What nftables buys. One rule per service instead of one per endpoint. A single verdict-map lookup instead of a linear scan of a dispatch chain. A uniform distribution computed in one expression instead of a cascade of -m statistic rules. A service losing all its endpoints costs one map element, not a rule synthesis. What it costs.

  • The inline map is rebuilt on every change. numgen random mod N map {...} lives inside the rule, so changing the endpoint set means rewriting the rule. Named maps would let you patch an element, but the kernel’s map insertion is O(n) or worse, so the single-endpoint fast path avoids the map entirely and the multi-endpoint case accepts the rewrite.
  • Session affinity still needs per-endpoint chains. The DNAT map cannot express “look up the source pin first”, so the affinity path uses a verdict map and keeps N endpoint chains plus N affinity rules.
  • The localhost NodePort path leaves the kernel datapath. In iptables mode it is a route_localnet hack plus an iptables jump; in nftables mode it is a userspace TCP proxy with a 5 second dial timeout, which is a real latency and throughput regression for local clients. This is why KubeProxyNFTablesLocalhostNodePorts is a separate gate.
  • Stale chains cannot be deleted immediately. With kernels before 6.2, if a map element points at a chain, deleting the map element does not make the kernel notice the chain is unreferenced for some time, so stale chains are flushed now and deleted later.

Wrapping up

The one thing to remember: the nftables proxier replaces the per-service chain of N endpoint jumps with a single rule - numgen random mod N map { 0 : ip.port, ... } - and replaces the global dispatch chain with a verdict-map lookup, ip daddr . meta l4proto . th dport vmap @service-ips. The per-service rule count is O(1) in the number of endpoints.

The part people get wrong: assuming the whole nftables mode is O(1). sessionAffinity: ClientIP still needs one rule and one chain per endpoint, because the pin has to be resolved before the DNAT, and the verdict-map fallback (numgen random mod N vmap) can only express “jump to a chain”, not “jump to an address”. And the localhost NodePort path in nftables mode is a userspace TCP proxy, not a kernel datapath, so it is measurably slower than iptables mode’s route_localnet trick.

What to go do about it: run nft list map inet kube-proxy service-ips and nft list ruleset on a node running nftables mode and count the rules in the per-service chains - if you see more than one per service, you are looking at an affinity service or at a stale chain pending deletion. To enable the localhost NodePort path, turn on the KubeProxyNFTablesLocalhostNodePorts feature gate (beta and on by default as of 1.38) and set nodePortAddresses to include 127.0.0.1; the rejections for the protocols the userspace proxy cannot serve are counted in kubeproxy_nftables_localhost_nodeport_rejected_packets_total.

KubernetesInternalsNetworking