Opensource internals

Weight Zero, Wait, Then Delete: The IPVS Connection Drain

13 min read Jay Hubbert

In IPVS mode kube-proxy sets a departing real server's weight to 0, parks it on a graceful-termination list, and rechecks every minute until the kernel reports zero active and inactive connections.

TL;DR: When an endpoint disappears in IPVS mode, kube-proxy does not delete the real server. It sets the real server’s weight to 0, which stops the scheduler from picking it for new connections but leaves the existing ones alone, and parks it on a graceful-termination list. A background goroutine re-checks the list once a minute and deletes the real server only when the kernel reports zero active and zero inactive connections. UDP and SCTP skip the whole dance, because they have no connection state worth draining.

Every backend removal in a cluster is a race between two kinds of traffic: new connections that should stop going to the endpoint, and old connections that are still legitimately talking to it. Delete the real server and you cut off in-flight requests. Leave it in place and you keep sending new traffic to a pod that is about to disappear.

IPVS gives you exactly the primitive needed to separate those two. A real server has a weight, and a weight of zero means “do not pick me for new connections” without meaning “delete me”. That is the whole design.

Weight zero, not delete

The drain starts in GracefulDeleteRS. Note the order of operations: it first tries to delete the real server outright, and only falls back to the drain if the delete is refused because connections are still open.

go 
// pkg/proxy/ipvs/graceful_termination.go:146
// GracefulDeleteRS to update rs weight to 0, and add rs to graceful terminate list
func (m *GracefulTerminationManager) GracefulDeleteRS(vs *utilipvs.VirtualServer, rs *utilipvs.RealServer) error {
	// Try to delete rs before add it to graceful delete list
	ele := &listItem{
		VirtualServer: vs,
		RealServer:    rs,
	}
	deleted, err := m.deleteRsFunc(ele)
	if err != nil {
		klog.ErrorS(err, "Error in deleting real server", "realServer", ele)
	}
	if deleted {
		return nil
	}
	rs.Weight = 0
	err = m.ipvs.UpdateRealServer(vs, rs)
	if err != nil {
		return err
	}
	klog.V(5).InfoS("Adding real server to graceful delete real server list", "realServer", ele)
	m.rsList.add(ele)

If the real server has no connections at all, the delete succeeds immediately and the endpoint is gone in one shot - no weight fiddling, no list entry. The slow path is only for real servers with live connections.

The delete itself is where the “are we done yet?” decision is made, and the decision is made by asking the kernel, not by asking a timer:

go 
// pkg/proxy/ipvs/graceful_termination.go:170
func (m *GracefulTerminationManager) deleteRsFunc(rsToDelete *listItem) (bool, error) {
	klog.V(5).InfoS("Trying to delete real server", "realServer", rsToDelete)
	rss, err := m.ipvs.GetRealServers(rsToDelete.VirtualServer)
	if err != nil {
		return false, err
	}
	for _, rs := range rss {
		if rsToDelete.RealServer.Equal(rs) {
			// For UDP and SCTP traffic, no graceful termination, we immediately delete the RS
			//     (existing connections will be deleted on the next packet because sysctlExpireNoDestConn=1)
			// For other protocols, don't delete until all connections have expired)
			if utilipvs.IsRsGracefulTerminationNeeded(rsToDelete.VirtualServer.Protocol) && rs.ActiveConn+rs.InactiveConn != 0 {
				klog.V(5).InfoS("Skip deleting real server till all connection have expired", "realServer", rsToDelete, "activeConnection", rs.ActiveConn, "inactiveConnection", rs.InactiveConn)
				return false, nil
			}
			klog.V(5).InfoS("Deleting real server", "realServer", rsToDelete)
			err := m.ipvs.DeleteRealServer(rsToDelete.VirtualServer, rs)
			if err != nil {
				return false, fmt.Errorf("delete destination %q err: %w", rs.String(), err)
			}
			return true, nil
		}
	}

rs.ActiveConn and rs.InactiveConn come straight out of the kernel’s IPVS netlink dump. ActiveConn is connections in ESTABLISHED; InactiveConn is connections in a closing or keepalive state. The condition rs.ActiveConn+rs.InactiveConn != 0 is the entire “is it safe to delete yet” test, and it is a kernel-authoritative test: no heuristic timeout, no grace period guesswork, no clock.

The drain list and the one-minute ticker

Real servers waiting to be deleted live in an in-memory map keyed by a composite name - the virtual server plus the real server - so the same endpoint under two different ClusterIPs is two separate entries:

go 
// pkg/proxy/ipvs/graceful_termination.go:47
// GetUniqueRSName return a string type unique rs name with vs information
func GetUniqueRSName(vs *utilipvs.VirtualServer, rs *utilipvs.RealServer) string {
	return vs.String() + "/" + rs.String()
}

The list is flushed by a goroutine that ticks once a minute:

go 
// pkg/proxy/ipvs/graceful_termination.go:31
const (
	rsCheckDeleteInterval = 1 * time.Minute
)
go 
// pkg/proxy/ipvs/graceful_termination.go:216
// Run start a goroutine to try to delete rs in the graceful delete rsList with an interval 1 minute
func (m *GracefulTerminationManager) Run() {
	go wait.Until(m.tryDeleteRs, rsCheckDeleteInterval, wait.NeverStop)
}

tryDeleteRs calls flushList, which walks every entry, calls the same deleteRsFunc used by the fast path, and removes the ones that report success:

go 
// pkg/proxy/ipvs/graceful_termination.go:93
func (q *graceTerminateRSList) flushList(handler func(rsToDelete *listItem) (bool, error)) bool {
	q.lock.Lock()
	defer q.lock.Unlock()
	success := true
	for name, rs := range q.list {
		deleted, err := handler(rs)
		if err != nil {
			klog.ErrorS(err, "Error in deleting real server", "realServer", name)
			success = false
		}
		if deleted {
			klog.InfoS("Removed real server from graceful delete real server list", "realServer", name)
			delete(q.list, rs.String())
		}
	}
	return success

Note the interval is a floor on the observation rate, not a grace period. A real server with 30-second connections can be deleted within a minute of its last connection closing; a real server with a 2-hour connection lives on the list for two hours. The one-minute granularity costs at most one extra minute of a zero-weight real server sitting in the kernel, which is harmless because it receives no new connections. The manager is started once, when the proxier is constructed (pkg/proxy/ipvs/proxier.go:389).

UDP and SCTP skip the drain

There is one protocol carve-out, and it is a one-liner:

go 
// pkg/proxy/ipvs/util/ipvs.go:111
// IsRsGracefulTerminationNeeded returns true if protocol requires graceful termination for the stale connections
func IsRsGracefulTerminationNeeded(proto string) bool {
	return !strings.EqualFold(proto, "UDP") && !strings.EqualFold(proto, "SCTP")
}

UDP and SCTP are connectionless at the IPVS layer. There is no connection table entry whose lifetime can be waited out, and a UDP “connection” in the conntrack sense is just a cache entry the kernel expires on its own. So the drain is skipped entirely: the real server is deleted on the first attempt, and any in-flight packet gets dropped.

The cleanup that makes that safe is a sysctl, and kube-proxy sets it during proxier construction:

go 
// pkg/proxy/ipvs/proxier.go:258
	// Set the conntrack sysctl we need for
	if err := proxyutil.EnsureSysctl(sysctl, sysctlVSConnTrack, 1); err != nil {
		return nil, err
	}

net/ipv4/vs/expire_nodest_conn = 1 means the IPVS connection table drops entries whose destination no longer exists on the next packet, rather than waiting out the protocol timeout. That is what the source comment means by “existing connections will be deleted on the next packet”.

The resurrection path

The interesting failure mode is a pod that flaps: the endpoint disappears, goes on the drain list with weight 0, and then comes back a few seconds later

  • because a rollout replaced it, or because the informer saw a stale delete followed by a stale add. If kube-proxy handled that naively, the endpoint would be weight-0 on the list and weight-1 in the desired state simultaneously. The sync code notices this and yanks the real server back out of the drain list, deleting it from the kernel so it can be re-added cleanly:
go 
// pkg/proxy/ipvs/proxier.go:1667
			// check if newEndpoint is in gracefulDelete list, if true, delete this ep immediately
			uniqueRS := GetUniqueRSName(vs, newDest)
			if !proxier.gracefuldeleteManager.InTerminationList(uniqueRS) {
				continue
			}
			proxier.logger.V(5).Info("new ep is in graceful delete list", "uniqueRealServer", uniqueRS)
			err := proxier.gracefuldeleteManager.MoveRSOutofGracefulDeleteList(uniqueRS)
			if err != nil {
				proxier.logger.Error(err, "Failed to delete endpoint in gracefulDeleteQueue", "endpoint", ep)
				continue
			}

MoveRSOutofGracefulDeleteList does the delete-then-forget - it calls ipvs.DeleteRealServer and removes the entry from the list (pkg/proxy/ipvs/graceful_termination.go:202) - and the AddRealServer that follows on the next line re-adds it with weight 1.

The delete is deliberate: you cannot update a real server’s weight back to 1 in place and leave the drain goroutine running, because the drain goroutine would then delete it out from under you at the next tick. Deleting and re-adding is the only way to make the two code paths agree.

The same list membership check is used on the removal side, so an endpoint already on the drain list is not drained twice (pkg/proxy/ipvs/proxier.go:1686).

The initialSync weight shortcut

There is a related optimisation that has nothing to do with draining but solves the same kind of problem: how do you learn a real server’s weight without asking the kernel about every single one?

Normally kube-proxy has to ask. On every sync of every service, it lists the real servers of the virtual server, because it needs to know which endpoints already exist in the kernel so it can diff against the desired state:

go 
// pkg/proxy/ipvs/proxier.go:1584
	// curEndpoints represents IPVS destinations listed from current system.
	curEndpoints := sets.New[string]()
	curDests, err := proxier.ipvs.GetRealServers(appliedVirtualServer)
	if err != nil {
		proxier.logger.Error(err, "Failed to list IPVS destinations")
		return err
	}

That listing is O(endpoints) netlink round-trips per service per sync. But the weight of a real server is always 1, except during a drain - and during a drain the weight is 0, and kube-proxy is the only thing that ever sets it. So the query is redundant, and kube-proxy skips it after the first sync:

go 
// pkg/proxy/ipvs/proxier.go:162
	// initialSync is a bool indicating if the proxier is syncing for the first time.
	// It is set to true when a new proxier is initialized and then set to false on all
	// future syncs.
	// This lets us run specific logic that's required only during proxy startup.
	// For eg: it enables us to update weights of existing destinations only on startup
	// saving us the cost of querying and updating real servers during every sync.
	initialSync bool

The flag is cleared at the end of the first sync, under the lock (pkg/proxy/ipvs/proxier.go:668), and the only place it is used is the one-time weight reset, which happens only when a real server already exists in the kernel:

go 
// pkg/proxy/ipvs/proxier.go:1653
		if curEndpoints.Has(ep) {
			// if we are syncing for the first time, loop through all current destinations and
			// reset their weight.
			if proxier.initialSync {
				for _, dest := range curDests {
					if dest.Weight != newDest.Weight {
						err = proxier.ipvs.UpdateRealServer(appliedVirtualServer, newDest)
						if err != nil {
							proxier.logger.Error(err, "Failed to update destination", "newDest", newDest)
							continue
						}
					}
				}
			}

The reason this matters: kube-proxy may restart while the kernel still holds the IPVS table. Real servers left over from the previous process could have weight 0 from a drain that was in flight when the process died. On startup kube-proxy resets every weight to 1. Doing this on every sync would cost an O(services x endpoints) netlink query per sync cycle for no benefit, since weights are otherwise always 1.

IPVS needs a local address

IPVS is a load balancer, not a NAT device. It does not rewrite destination addresses the way iptables’ DNAT does - it picks a real server and lets the kernel’s routing deliver the packet. That means the virtual server’s address has to be a local address on the node, or the kernel will not accept traffic for it. kube-proxy solves this with a dummy interface and binds every ClusterIP to it:

go 
// pkg/proxy/ipvs/proxier.go:94
	// defaultDummyDevice is the default dummy interface which ipvs service address will bind to it.
	defaultDummyDevice = "kube-ipvs0"
)
go 
// pkg/proxy/ipvs/proxier.go:709
	// make sure dummy interface exists in the system where ipvs Proxier will bind service address on it
	_, err := proxier.netlinkHandle.EnsureDummyDevice(defaultDummyDevice)
	if err != nil {
		proxier.logger.Error(err, "Failed to create dummy interface", "interface", defaultDummyDevice)
		return
	}

and then each virtual server’s address is bound to it:

go 
// pkg/proxy/ipvs/proxier.go:1555
	// bind service address to dummy interface
	if bindAddr {
		// always attempt to bind if alreadyBoundAddrs is nil,
		// otherwise check if it's already binded and return early
		if alreadyBoundAddrs != nil && alreadyBoundAddrs.Has(vs.Address.String()) {
			return nil
		}

		proxier.logger.V(4).Info("Bind address", "address", vs.Address)
		_, err := proxier.netlinkHandle.EnsureAddressBind(vs.Address.String(), defaultDummyDevice)
		if err != nil {
			proxier.logger.Error(err, "Failed to bind service address to dummy device", "serviceName", svcName)
			return err
		}

The same trick has a security consequence, which is why there is a dedicated filter chain. Because the ClusterIP is a local address on the node, a pod on the node could connect to a ClusterIP and reach whatever is behind it - the host’s own port 443, for example - rather than the service. The KUBE-IPVS-FILTER chain closes that hole:

go 
// pkg/proxy/ipvs/proxier.go:1434
	// Add rules to the filter/KUBE-IPVS-FILTER chain to prevent access to ports on the host through VIP addresses.
	// https://github.com/kubernetes/kubernetes/issues/72236
	proxier.filterRules.Write(
		"-A", string(kubeIPVSFilterChain),
		"-m", "set", "--match-set", proxier.ipsetList[kubeLoadBalancerSet].Name, "dst,dst", "-j", "RETURN")
	proxier.filterRules.Write(
		"-A", string(kubeIPVSFilterChain),
		"-m", "set", "--match-set", proxier.ipsetList[kubeClusterIPSet].Name, "dst,dst", "-j", "RETURN")

Service destinations return early (legitimate traffic). Everything else that looks like a VIP - a NEW connection to an address in the IPVS set - is rejected:

go 
// pkg/proxy/ipvs/proxier.go:1451
	proxier.filterRules.Write(
		"-A", string(kubeIPVSFilterChain),
		"-m", "conntrack", "--ctstate", "NEW",
		"-m", "set", "--match-set", proxier.ipsetList[kubeIPVSSet].Name, "dst", "-j", "REJECT")

Trade-offs

What the drain buys. Zero dropped connections on a rolling update. The decision to delete is made by the kernel’s own connection counters, so it is exact - there is no heuristic grace period to tune, and no case where a long-lived connection is cut off because a timer expired.

What it costs.

  • A minute of latency in the worst case. The check runs once a minute, so a real server can linger on the list with weight 0 for up to 60 seconds after its last connection closes. It receives no new traffic, so the only cost is a stale entry in the kernel’s IPVS table.
  • An unbounded drain list. A real server with a connection that never closes stays on the list forever, and the list is only in memory. There is no maximum drain time.
  • Extra netlink traffic during a churn-heavy sync. Every endpoint removal costs one UpdateRealServer (to zero the weight) plus one GetRealServers listing per check.
  • IPVS mode is on its way out. The KubeProxyIPVS feature gate is deprecated as of 1.37, so nftables and iptables are where new behaviour lands.

Wrapping up

The one thing to remember: a departing IPVS real server is not deleted - it is weighed at 0 and left in the kernel until rs.ActiveConn+rs.InactiveConn == 0, checked once a minute by rsCheckDeleteInterval. The kernel’s connection counters are the only clock in the decision.

The part people get wrong: thinking that “weight 0” means the endpoint is gone. Weight 0 only means the scheduler will not pick it for new connections; the real server is still in the kernel, still holding conntrack state, and still listed by ipvsadm -Ln. Expect to see drained real servers sitting at weight 0 for up to a minute after they should have disappeared - and expect them to be there for as long as the connection they are draining.

What to go do about it: run kube-proxy with -v=5 and watch for the log line “Skip deleting real server till all connection have expired”, which carries the activeConnection and inactiveConnection counts - that tells you exactly which real servers are being held and why. To see what is actually in the kernel, run ipvsadm -Ln and look for real servers with weight 0 - those are the drain list, live. If you need UDP NodePorts to disappear instantly, remember that expire_nodest_conn=1 is what makes that safe; check it with sysctl net/ipv4/vs/expire_nodest_conn. And if you are choosing a proxy mode for a new cluster, note that KubeProxyIPVS is deprecated as of 1.37 - nftables mode has no equivalent drain, because its DNAT happens in the kernel’s conntrack layer rather than in a stateful load-balancer table.

KubernetesInternalsNetworking