Hints by CPU Share: How the Topology Cache Keeps a Zone From Overloading

TL;DR: Topology-aware hints are computed from one number: each zone’s share of the cluster’s total allocatable CPU. The controller turns that into a desired endpoint count per zone and a minimum that is guaranteed not to overload the zone by more than 20 percent (overloadThreshold float64 = 0.2). If the minimums cannot be satisfied with the endpoints available - or if the node information is incomplete - hints are removed from every EndpointSlice and a TopologyAwareHintsDisabled event is emitted. On the consumer side, kube-proxy ignores hints entirely unless every ready endpoint carries one.

Zone-aware routing has an obvious failure mode, and it is not subtle: you route traffic to a zone because it has endpoints there, the zone’s pods get slower, and the zone becomes the bottleneck. Topology-aware hints are the mechanism that is supposed to prevent that, and the mechanism has one hard safety property baked in: no zone’s endpoints may be asked to serve more than 20 percent above its share of the cluster.

The work is split across two components. The EndpointSlice controller produces hints, by deciding how many endpoints of each service should be offered to each zone. kube-proxy consumes them, by refusing to route to an endpoint whose hint does not cover the local node or zone. Both halves have to agree, and both are deliberately conservative: the producer abandons hints entirely rather than compute a partial allocation, and the consumer ignores hints entirely rather than apply them one endpoint at a time.

What a hint actually is

A hint is a field on an endpoint in an EndpointSlice. ForZones says “these endpoints should preferentially be used by clients in these zones”; ForNodes says the same for specific nodes. It is a preference, not a restriction - a client in zone A can still reach an endpoint hinted for zone B, it just will not be sent there by a hint-aware proxy.

The producer is the EndpointSlice controller, and the producer of the producer is the topology cache, which lives in k8s.io/endpointslice/topologycache and is shared between the EndpointSlice controller and the reconciler. The cache holds two things: the CPU totals per zone, and the ratio of each zone to the cluster:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:240

		totalCPU.Add(*nodeCPU)
		if _, ok = cpuByZone[zone]; !ok {
			cpuByZone[zone] = nodeCPU
		} else {
			cpuByZone[zone].Add(*nodeCPU)
		}
	}

The zone’s share of the cluster’s CPU

SetNodes is the only place the ratios are computed. It walks the node list, sums node.Status.Allocatable.Cpu() per zone, and divides by the cluster total:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:252
	if totalCPU.IsZero() || !sufficientNodeInfo || len(cpuByZone) < 2 {
		logger.V(2).Info("Insufficient node info for topology hints", "totalZones", len(cpuByZone), "totalCPU", totalCPU.String(), "sufficientNodeInfo", sufficientNodeInfo)
		t.sufficientNodeInfo = false
		t.cpuByZone = nil
		t.cpuRatiosByZone = nil

	} else {
		t.sufficientNodeInfo = sufficientNodeInfo
		t.cpuByZone = cpuByZone

		t.cpuRatiosByZone = map[string]float64{}
		for zone, cpu := range cpuByZone {
			t.cpuRatiosByZone[zone] = float64(cpu.MilliValue()) / float64(totalCPU.MilliValue())
		}
	}

Three different things make the cache give up entirely, and all three are visible in that one if: no usable CPU information at all, some node missing a zone label or a CPU, and only one zone.

The second one is the interesting one, and the source is frank about it:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:228
		// TODO(robscott): Figure out if there's an acceptable proportion of
		// nodes with inadequate information. The current logic means that as
		// soon as we find any node without a zone or allocatable CPU specified,
		// we bail out entirely. Bailing out at this level will make our cluster
		// wide ratios nil, which would result in slices for all Services having
		// their hints removed.
		if !ok || zone == "" || nodeCPU.IsZero() {
			cpuByZone = map[string]*resource.Quantity{}
			sufficientNodeInfo = false
			logger.Info("Can't get CPU or zone information for node", "node", klog.KObj(node))
			break
		}

One node with a missing topology.kubernetes.io/zone label disables hints for the entire cluster. That is a blunt instrument, and the TODO admits it. The reason it is safe is that the consumer would ignore half-populated hints anyway, so a partial answer would be worse than no answer.

Control-plane nodes are excluded from the capacity math entirely, so a three-node control plane in zone A does not distort zone A’s share: hasExcludedLabels skips any node carrying node-role.kubernetes.io/control-plane or node-role.kubernetes.io/master (staging/src/k8s.io/endpointslice/topologycache/topologycache.go:342).

The allocation formula

getAllocations is the core of the whole feature. The desired count for a zone is its CPU ratio times the number of ready endpoints. The minimum is the desired count divided by 1.2, rounded up:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:31
const (
	// overloadThreshold represents the maximum overload any individual endpoint
	// should be exposed to.
	overloadThreshold float64 = 0.2
)

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:315
	for zone, ratio := range t.cpuRatiosByZone {
		desired := ratio * float64(numEndpoints)
		minimum := int(math.Ceil(desired * (1 / (1 + overloadThreshold))))
		allocations[zone] = allocation{
			minimum: minimum,
			desired: math.Max(desired, float64(minimum)),
		}
		minTotal += minimum
		remainingMinEndpoints -= minimum
		if remainingMinEndpoints < 0 {
			return nil, &EventBuilder{
				EventType: v1.EventTypeWarning,
				Reason:    "TopologyAwareHintsDisabled",
				Message:   fmt.Sprintf("%s (%d endpoints, %d zones)", MinAllocationExceedsOverloadThreshold, numEndpoints, len(t.cpuRatiosByZone)),
			}
		}
	}

So the formula is minimum = ceil(desired / 1.2). The ceiling is what makes it a guarantee rather than a target: if a zone’s share is 50 percent of the cluster CPU, its minimum is ceil(0.5 * N / 1.2), and no consumer may be forced to put more than 20 percent above that on the zone.

The remainingMinEndpoints < 0 check is the abort condition. If the sum of the minimums exceeds the number of endpoints available, the allocation is infeasible - the zones’ minimums cannot all be satisfied - and the whole thing is abandoned. There are two earlier bail-outs in the same function, for the same reason:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:296
	if len(t.cpuRatiosByZone) < 2 {
		return nil, &EventBuilder{
			EventType: v1.EventTypeWarning,
			Reason:    "TopologyAwareHintsDisabled",
			Message:   NodesReadyInOneZoneOnly,
		}
	}
	if len(t.cpuRatiosByZone) > numEndpoints {
		return nil, &EventBuilder{
			EventType: v1.EventTypeWarning,
			Reason:    "TopologyAwareHintsDisabled",
			Message:   fmt.Sprintf("%s (%d endpoints, %d zones)", InsufficientNumberOfEndpoints, numEndpoints, len(t.cpuRatiosByZone)),
		}
	}

More zones than endpoints is hopeless: with three zones and two endpoints, at least one zone gets zero endpoints, and the “prefer same zone” promise becomes “route across zones anyway”.

After the minimums are computed, the leftover endpoints are distributed as the zones’ maximum:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:333
	for zone, allocation := range allocations {
		allocation.maximum = allocation.minimum + numEndpoints - minTotal
		allocations[zone] = allocation
	}

	return allocations, nil

When hints are abandoned entirely

There are three distinct places where the cache decides that hints are not possible, and in every one of them the outcome is the same: hints are removed from the EndpointSlices rather than left half-applied. Here is the removal path, taken straight from AddHints:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:92
	allocations, allocationsEvent := t.getAllocations(totalEndpoints)
	events := []*EventBuilder{}
	if allocationsEvent != nil {
		logger.Info(allocationsEvent.Message+", removing hints", "key", si.ServiceKey, "addressType", si.AddressType)
		allocationsEvent.Message = FormatWithAddressType(allocationsEvent.Message, si.AddressType)
		events = append(events, allocationsEvent)
		t.RemoveHints(si.ServiceKey, si.AddressType)
		slicesToCreate, slicesToUpdate := RemoveHintsFromSlices(si)
		return slicesToCreate, slicesToUpdate, events
	}

The same three-way abandon happens at the per-endpoint level, when an endpoint turns up without a zone at all (staging/src/k8s.io/endpointslice/topologycache/topologycache.go:117): the controller emits a TopologyAwareHintsDisabled event with the reason “NoZoneSpecified”, calls t.RemoveHints, and calls RemoveHintsFromSlices.

The events are named TopologyAwareHintsDisabled, with the matching TopologyAwareHintsEnabled on the happy path, and they show up on the Service object - the only place a cluster operator will ever see that hints silently stopped working.

The redistribution step

Assuming the allocation is feasible, AddHints runs a three-step algorithm. Step 1 is to hint every endpoint for its own zone - the trivially correct starting point:

// staging/src/k8s.io/endpointslice/topologycache/topologycache.go:129
			allocatedHintsByZone[*endpoint.Zone]++
			slice.Endpoints[i].Hints = &discovery.EndpointHints{ForZones: []discovery.ForZone{{Name: *endpoint.Zone}}}
		}
	}

	// step 2. Identify which zones need to donate slices and which need more.
	givingZones, receivingZones := getGivingAndReceivingZones(allocations, allocatedHintsByZone)

	// step 3. Redistribute endpoints based on data from step 2.
	redistributions := redistributeHints(logger, allocatableSlices, givingZones, receivingZones)

Step 2 computes the deficit or surplus per zone against the desired count, and converts the fractional numbers into integer transfers. The loop picks the zone with the largest surplus and the zone with the largest deficit, moves one endpoint from one to the other, and repeats - stopping once either side has less than one endpoint left to give or receive, or less than half an endpoint (staging/src/k8s.io/endpointslice/topologycache/utils.go:151).

Step 3 is the actual mutation - a giving zone’s endpoint gets its hint rewritten to a receiving zone’s name (staging/src/k8s.io/endpointslice/topologycache/utils.go:97):

// staging/src/k8s.io/endpointslice/topologycache/utils.go:97
			givingZone := *endpoint.Zone
			numToGive, ok := givingZones[givingZone]
			if ok && numToGive > 0 {
				for receivingZone, numToReceive := range receivingZones {
					if numToReceive > 0 {
						slice.Endpoints[i].Hints = &discovery.EndpointHints{ForZones: []discovery.ForZone{{Name: receivingZone}}}

So a Deployment with 10 replicas in zone A and 2 in zone B, where zone A has 90 percent of the cluster’s CPU, ends up with roughly 9 endpoints hinted for A and 2 hinted for B (desired = 0.9*12 = 10.8, minimum = ceil(10.8/1.2) = 9). The hint is the distribution of demand, not the distribution of pods.

The consumer: why kube-proxy ignores most hints

Everything above is the producer’s view. The consumer is where the safety property is actually enforced, and it is much stricter than you would guess.

kube-proxy computes a single mode for the whole service, not a per-endpoint decision:

// pkg/proxy/topology.go:154
// topologyModeFromHints returns a topology mode ("", "PreferSameZone", or
// "PreferSameNode") based on the Endpoint hints:
//   - If every ready endpoint has a node hint, and at least one endpoint is hinted for
//     this node, then it returns "PreferSameNode".
//   - Otherwise, if every ready endpoint has a zone hint, and at least one endpoint is
//     hinted for this node's zone, then it returns "PreferSameZone".
//   - Otherwise it returns "" (meaning, no topology / default traffic distribution).
func topologyModeFromHints(svcInfo ServicePort, endpoints []Endpoint, nodeName, zone string) string {

The implementation checks two independent conditions - every ready endpoint must carry the hint, and at least one endpoint must be hinted for this node or zone:

// pkg/proxy/topology.go:191
	if allEndpointsHaveNodeHints {
		if hasEndpointForNode {
			return v1.ServiceTrafficDistributionPreferSameNode
		}
		klog.V(2).InfoS("Ignoring same-node topology hints for service since no hints were provided for node", "service", svcInfo, "node", nodeName)
	} else {
		klog.V(7).InfoS("Ignoring same-node topology hints for service since one or more endpoints is missing a node hint", "service", svcInfo)
	}
	if allEndpointsHaveZoneHints {
		if hasEndpointForZone {
			return v1.ServiceTrafficDistributionPreferSameZone
		}

If either condition fails, the mode is "" and the hints are ignored completely - not partially applied, not applied to the endpoints that happen to match. The filter that consumes the mode is unconditional:

// pkg/proxy/topology.go:218
func availableForTopology(endpoint Endpoint, topologyMode, nodeName, zone string) bool {
	switch topologyMode {
	case "":
		return true
	case v1.ServiceTrafficDistributionPreferSameNode:
		return endpoint.NodeHints().Has(nodeName)
	case v1.ServiceTrafficDistributionPreferSameZone:
		return endpoint.ZoneHints().Has(zone)
	default:
		return false
	}
}

The reason for the all-or-nothing rule is starvation. Suppose a service has 100 endpoints and only 2 of them carry a hint for your zone. If kube-proxy honoured the hints per-endpoint, 98 of 100 connections would go to those two pods - which is precisely the overload the whole feature exists to prevent. The mode is computed once, for the service, and it is only “PreferSameZone” if the hints are complete enough that the zone has a fair share.

The hints themselves are parsed out of the EndpointSlice into sets when the endpoint is added to the cache:

// pkg/proxy/endpointslicecache.go:213
		var zoneHints, nodeHints sets.Set[string]
		if endpoint.Hints != nil {
			if len(endpoint.Hints.ForZones) > 0 {
				zoneHints = sets.New[string]()
				for _, zone := range endpoint.Hints.ForZones {
					zoneHints.Insert(zone.Name)
				}
			}
			if len(endpoint.Hints.ForNodes) > 0 {
				nodeHints = sets.New[string]()
				for _, node := range endpoint.Hints.ForNodes {
					nodeHints.Insert(node.Name)
				}
			}
		}

Trade-offs

What the design buys. A cluster with uneven capacity gets traffic proportional to capacity, not proportional to pod count. And the 20 percent overload ceiling is a hard guarantee, not a heuristic: it is enforced by the minimum in the allocation and by the fact that the consumer treats hints as all-or-nothing.

What it costs.

• One bad node disables the feature. A single node missing its zone label sets sufficientNodeInfo = false and removes hints cluster-wide.
• The allocation is recomputed from scratch on every change. The cache is rebuilt whenever the node set changes, and the endpoint distribution is rewritten slice by slice.
• Hints are advisory, so a non-hint-aware proxy ignores them entirely. An iptables-mode kube-proxy in the same cluster as a Cilium datapath will not respect the hints at all.
• The feature needs the trafficDistribution field to be meaningful. The hints are only consulted when the service opts in with trafficDistribution: PreferSameZone.

Wrapping up

The one thing to remember: hints are allocated from each zone’s share of cluster allocatable CPU, with the formula minimum = ceil(desired / 1.2) where overloadThreshold float64 = 0.2. If the minimums cannot be summed into the endpoints available, or if the node info is incomplete, or if there are more zones than endpoints, the controller removes all hints and emits a TopologyAwareHintsDisabled event.

The part people get wrong: thinking hints are a per-endpoint filter. kube-proxy computes one mode for the whole service and ignores every hint on that service unless all ready endpoints are hinted. A service with two hinted endpoints out of a hundred gets zero topology awareness, not two percent - which is the intended behaviour, because honouring partial hints would send 98 percent of the traffic to two pods.

What to go do about it: set spec.trafficDistribution: PreferSameZone on the services that matter and check kubectl describe svc for the TopologyAwareHintsEnabled event - if you see TopologyAwareHintsDisabled with the reason “Can’t get CPU or zone information for node”, the fix is to label the offending node with topology.kubernetes.io/zone and make sure it reports allocatable CPU. Then check the consumer side with kubectl get endpointslice -o yaml and confirm every ready endpoint in the slice carries a hints.forZones entry - if some do and some do not, kube-proxy is ignoring all of them. On the node, kube-proxy -v=7 prints “Ignoring same-zone topology hints for service since one or more endpoints is missing a zone hint”, which is the exact reason your hints are not being used.