The Compactor Elects Itself With a Compare-and-Swap on a Magic Key

TL;DR: Every kube-apiserver wants to compact etcd, but only one should. Instead of a lease or client-go leader election, the compactor uses a single well-known etcd key, compact_rev_key, whose version number acts as a logical clock. Each API server does a compare-and-swap — Txn.If(Version(compactRevKey) == expectVersion) — and exactly one wins and bumps the key’s version by 1. The losers read the winner’s version back out and retry in five minutes. If the winner crashes, the next round’s winner takes over. No leases, no clock skew, no coordination service.

etcd is a multi-version key-value store: every write creates a new revision, and the old revisions stay around until somebody compacts them. Compaction is what reclaims the disk space, and it is also what makes old revisions unreadable — after compaction, a watch that has fallen behind gets an ErrCompacted instead of a stream of missed events.

Kubernetes needs compaction to happen regularly. But etcd is shared: in a typical cluster there are three or more API server instances, all talking to the same etcd cluster, all of which would happily compact on a timer. Three API servers compacting every five minutes is three times the work and no benefit. So somebody has to be elected.

The obvious solution is a lease-based leader election, and Kubernetes already has one in client-go. The compactor does not use it, and the reason is interesting: the election does not need to be timely, it only needs to be atomic.

The protocol, spelled out in the source

The whole algorithm is written out in a comment above the compact loop, and it is worth reading in full because it is unusually precise about what is and is not guaranteed:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:153
func (c *compactor) runCompactLoop(stopCh chan struct{}) {
	// Technical definitions:
	// We have a special key in etcd defined as *compactRevKey*.
	// compactRevKey's value will be set to the string of last compacted revision.
	// compactRevKey's version will be used as logical time for comparison. THe version is referred as compact time.
	// Initially, because the key doesn't exist, the compact time (version) is 0.
	//
	// Algorithm:
	// - Compare to see if (local compact_time) = (remote compact_time).
	// - If yes, increment both local and remote compact_time, and do a compaction.
	// - If not, set local to remote compact_time.
	//
	// Technical details/insights:
	//
	// The protocol here is lease based. If one compactor CAS successfully, the others would know it when they fail in
	// CAS later and would try again in 5 minutes. If an APIServer crashed, another one would "take over" the lease.

Note the trick: etcd’s own key version is the clock. Every write to compact_rev_key bumps its version by one. The version is maintained by etcd itself, is globally consistent, and — crucially — is not a wall-clock timestamp. It is a monotonically increasing counter that only advances when somebody writes the key. Two API servers can compare their local notion of “compact time” against the remote version without any clock agreement whatsoever.

The key is a single constant:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:34
const (
	compactRevKey = "compact_rev_key"
)

and the guarantees are stated right there:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:183
	// We have the guarantees:
	// - in normal cases, the interval is 5 minutes.
	// - in failover, the interval is >5m and <10m
	//
	// FAQ:
	// - What if time is not accurate? We don't care as long as someone did the compaction. Atomicity is ensured using
	//   etcd API.
	// - What happened under heavy load scenarios? Initially, each apiserver will do only one compaction
	//   every 5 minutes. This is very unlikely affecting or affected w.r.t. server load.

“What if time is not accurate? We don’t care as long as someone did the compaction.” That is the whole design in one sentence. The election is not about when; it is about not doing it twice. And the failover guarantee is bounded: >5m and <10m. If the leader crashes, the next API server to try wins the CAS — but only when its own five-minute timer fires, hence the 5–10 minute window.

The diagram in the comment shows the timeline: C1 compacts at t1 and t2; C2’s attempt at t1’ fails the CAS, C2 resyncs its local compact time to the remote version, and tries again at t2’ — after C2 has won it, C1’s next attempt at t2 would fail and C1 would resync.

The compare-and-swap

The election is one etcd transaction:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:219
func Compact(ctx context.Context, client *clientv3.Client, expectVersion, rev int64) (currentVersion, currentRev, compactRev int64, err error) {
	resp, err := client.KV.Txn(ctx).If(
		clientv3.Compare(clientv3.Version(compactRevKey), "=", expectVersion),
	).Then(
		clientv3.OpPut(compactRevKey, strconv.FormatInt(rev, 10)), // Expect side effect: increment Version
	).Else(
		clientv3.OpGet(compactRevKey),
	).Commit()

Read it carefully — every part is load-bearing:

• If(Version(compactRevKey) == expectVersion) — the compare is against the key’s version, not its value. The value is the last compacted revision (a number like 4815162342); the version is the logical clock.
• Then(OpPut(...)) — the side effect of a successful CAS is a write, and the comment says so explicitly: // Expect side effect: increment Version. The write is what advances the logical clock. The value being written is the revision that was just compacted.
• Else(OpGet(compactRevKey)) — the loser’s transaction does a read instead, and the read is part of the same atomic transaction. There is no separate GET round trip; the losing API server learns the winner’s version and the winner’s compacted revision from the failed transaction’s response.

That Else branch is the elegant part. The loser does not need to retry the CAS with a fresh read; the read is already part of the same atomic operation, so there is no window between “observe” and “act”.

The result handling mirrors that:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:231
	currentRev = resp.Header.Revision

	if !resp.Succeeded {
		currentVersion = resp.Responses[0].GetResponseRange().Kvs[0].Version
		compactRev, err = strconv.ParseInt(string(resp.Responses[0].GetResponseRange().Kvs[0].Value), 10, 64)
		if err != nil {
			return currentVersion, currentRev, 0, nil
		}
		return currentVersion, currentRev, compactRev, nil
	}
	currentVersion = expectVersion + 1

	if rev == 0 {
		// We don't compact on bootstrap.
		return currentVersion, currentRev, 0, nil
	}
	if _, err = client.Compact(ctx, rev); err != nil {
		return currentVersion, currentRev, 0, err
	}
	klog.V(4).Infof("etcd: compacted rev (%d), endpoints (%v)", rev, client.Endpoints())
	return currentVersion, currentRev, rev, nil
}

On failure the loser reads the winner’s Version (its new local compact time) and the winner’s compacted revision out of the Else response. On success it advances its own compact time by exactly one (expectVersion + 1) — because it knows the write bumped the version by one.

There is one important detail: the CAS happens before the actual compaction. The winner wins the election first, then calls client.Compact(ctx, rev). If the compaction itself fails, the election has still been consumed — the version has advanced, and nobody else will try this round. That is deliberate: the alternative (compact first, then CAS) would let two API servers both compact the same revision.

And note rev == 0: on the very first round there is nothing to compact (the local revision counter starts at zero because the compactor has never read a revision), so the CAS still runs — establishing the initial version — but no compaction is issued. The comment says exactly that: “We don’t compact on bootstrap.”

One compactor per endpoint, per process

Before the CAS protocol even matters, there is a process-level guard. The etcd3 package keeps a global map from endpoint to compactor:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:38
var (
	endpointsMapMu sync.Mutex
	endpointsMap   map[string]*compactor
)

func init() {
	endpointsMap = make(map[string]*compactor)
}

and the constructor deduplicates on the client’s endpoints:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:47
// StartCompactorPerEndpoint starts a compactor but prevents multiple compactors per client endpoint.
// Calling the function twice on single endpoint will return the same compactor.
// Compactor removes old version of keys from etcd.
// By default, we save the most recent 5 minutes data and compact versions > 5minutes ago.
// It should be enough for slow watchers and to tolerate burst.
func StartCompactorPerEndpoint(client *clientv3.Client, compactInterval time.Duration) *compactor {
	endpointsMapMu.Lock()
	defer endpointsMapMu.Unlock()

	// In one process, we can have only one compactor for one cluster.
	// Currently we rely on endpoints to differentiate clusters.
	for _, ep := range client.Endpoints() {
		if c, ok := endpointsMap[ep]; ok {
			klog.V(4).Infof("compactor already exists for endpoints %v", client.Endpoints())
			return c
		}
	}

This matters because a single API server process may open several etcd clients — one for the core resources, one per CRD, one per aggregated API server. Without this guard, each of those clients would start its own compactor goroutine. The comment admits the design is crude: “Currently we rely on endpoints to differentiate clusters.”

Above that, the storage factory reference-counts compactors per transport configuration:

// staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go:404
// startCompactorOnce start one compactor per transport. If the interval get smaller on repeated calls, the
// compactor is replaced. A destroy func is returned. If all destroy funcs with the same transport are called,
// the compactor is stopped.
func startCompactorOnce(c storagebackend.TransportConfig, interval time.Duration) (etcd3.Compactor, func(), error) {
	compactorsMu.Lock()
	defer compactorsMu.Unlock()

	if interval == 0 {
		// short circuit, if the compaction request from apiserver is disabled
		return nil, func() {}, nil
	}
	key := fmt.Sprintf("%v", c) // gives: {[server1 server2] keyFile certFile caFile}
	if compactor, foundBefore := compactors[key]; !foundBefore || compactor.interval > interval {

The key is the whole TransportConfig formatted as a string — server list plus TLS file paths. Each storage backend that is created bumps a reference count, and the compactor is only stopped when the last reference goes away. Note the “interval get smaller on repeated calls, the compactor is replaced” rule: if a later registration asks for a more aggressive interval, the existing compactor is cancelled and replaced with a faster one. A later registration asking for a longer interval is ignored — the more aggressive compactor stays.

What the compaction revision is used for

The compactor does more than reclaim disk. Its CompactRevision() is consumed by the watch cache, which uses it to know how far it can safely truncate:

// staging/src/k8s.io/apiserver/pkg/storage/cacher/compactor.go:65
func (c *compactor) compactIfNeeded() {
	rev := c.store.CompactRevision()
	if rev == 0 {
		return
	}
	c.lock.Lock()
	defer c.lock.Unlock()
	if rev <= c.compactRevision {
		return
	}
	c.wc.storage.Compact(uint64(rev))
	c.compactRevision = rev
}

So the watch cache polls the store’s compact revision on a timer and drops the watch events that are now older than the compacted revision. This is why the compacted revision has to be correct: if the watch cache thinks it can keep events that etcd has already discarded, a lagging watcher will get an ErrCompacted it did not expect.

There is a second consumer. With the ListFromCacheSnapshot feature gate enabled, the compactor runs a watch on the compact key rather than polling, so a compactor in one API server learns immediately about a compaction done by another:

// staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:254
// runWatchLoops gets and watches changes to compact_rev_key and updates the compaction revision based on it.
func (c *compactor) runWatchLoop(stopCh chan struct{}) {
	ctx := wait.ContextForChannel(stopCh)
	for {
		select {
		case <-time.After(time.Second):
		case <-ctx.Done():
			return
		}
		compactRev, currentRev, err := c.getCompactRev(ctx)

The watch is on compact_rev_key, so any write to it — by any API server — fans out the new compacted revision.

The numbers

The defaults are all in one place:

// staging/src/k8s.io/apiserver/pkg/storage/storagebackend/config.go:33
const (
	StorageTypeUnset = ""
	StorageTypeETCD2 = "etcd2"
	StorageTypeETCD3 = "etcd3"

	DefaultCompactInterval      = 5 * time.Minute
	DefaultDBMetricPollInterval = 30 * time.Second
	DefaultEventsHistoryWindow  = 75 * time.Second
	DefaultHealthcheckTimeout   = 2 * time.Second
	DefaultReadinessTimeout     = 2 * time.Second
)

• --etcd-compaction-interval defaults to DefaultCompactInterval = 5 minutes, and setting it to 0 disables apiserver-initiated compaction entirely:

// staging/src/k8s.io/apiserver/pkg/server/options/etcd.go:192
	fs.DurationVar(&s.StorageConfig.CompactionInterval, "etcd-compaction-interval", s.StorageConfig.CompactionInterval,
		"The interval of compaction requests. If 0, the compaction request from apiserver is disabled.")

• DefaultEventsHistoryWindow is 75 seconds — the minimum history the storage layer keeps. Note it is not the same as the compaction interval. The compaction interval says how often etcd is told to drop old revisions; the events history window says how much history the storage layer promises to keep for the watch cache. 75 seconds > 60 seconds, which matters because the watch cache polls on a timer and must not be told to truncate events it still needs.

The etcd client’s timeouts are also constants, and the reasons are recorded:

// staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go:60
const (
	// The short keepalive timeout and interval have been chosen to aggressively
	// detect a failed etcd server without introducing much overhead.
	keepaliveTime    = 30 * time.Second
	keepaliveTimeout = 10 * time.Second

	// dialTimeout is the timeout for failing to establish a connection.
	// It is set to 20 seconds as times shorter than that will cause TLS connections to fail
	// on heavily loaded arm64 CPUs (issue #64649)
	dialTimeout = 20 * time.Second
)

A 30-second keepalive interval with a 10-second timeout means a dead etcd member is detected in at most 40 seconds. The dial timeout of 20 seconds is the interesting one: shorter timeouts cause TLS handshakes to fail on heavily loaded arm64 CPUs. That is issue #64649 — a real bug where the symptom (connection refused, apiserver fails to start) had nothing to do with etcd being down.

The health probe is rate limited so that a flapping etcd does not turn the API server’s /healthz endpoint into a load generator:

// staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go:196
	// limit to a request every half of the configured timeout with a maximum burst of one
	// rate limited requests will receive the last request sent error (note: not the last received response)
	limiter := rate.NewLimiter(rate.Every(timeout/2), 1)
	// initial state is the clientErr
	lastError := &atomicLastError{err: fmt.Errorf("etcd client connection not yet established")}

	return func() error {

With timeout = 2 seconds (the healthcheck timeout), that is one real probe per second, burst of one. A rate-limited call does not probe again — it returns the last error rather than re-probing. The distinction in the comment is precise: “the last request sent error (note: not the last received response)”. The atomicLastError stores the error with a timestamp and only overwrites it if the new one is newer, so concurrent probes cannot regress the reported error.

Trade-offs

• The election is coarse. The losers learn they lost only when their own timer fires, five minutes later. A compactor that dies mid-interval means compaction is delayed until the next round — hence the 5–10 minute failover window.
• The version is the clock, so no clock is needed. The design has explicitly traded “compact at exactly t” for “compact at some point in each window”. For compaction, that trade is obviously right: being 30 seconds late costs nothing.
• One key, one cluster. The compact_rev_key lives at the root of the etcd keyspace, outside any resource prefix. That means it is shared across all resources — which is correct, since compaction is global. But it also means the etcd keyspace now contains a key that is not a Kubernetes object, and it shows up in etcdctl get "" --prefix output.
• The watch cache depends on the compacted revision being right. Get the election wrong and lagging watchers get spurious ErrCompacted errors. The cacher/compactor.go polling is the safety net for that.

Wrapping up

The one thing to remember: the API server does not use leader election to decide who compacts etcd. It uses a compare-and-swap on the version of the single etcd key compact_rev_key (staging/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:219). etcd’s own version counter is the logical clock, so clock skew is irrelevant — the source comment says it outright: “What if time is not accurate? We don’t care as long as someone did the compaction.”

The part people get wrong: that compaction is per-API-server. It is not. The CAS guarantees exactly one compactor per interval across the whole cluster, and the failover window is bounded at >5m and <10m. If you are watching etcdctl compact output and see several API servers compacting, they are not racing — they are the losers of the election, which do not compact at all, only resync their local compact time.

What to go do about it: if etcd disk usage is climbing, check --etcd-compaction-interval (default 5m, and 0 turns off API-server-driven compaction entirely) and the etcd_server_compaction_revision / apiserver_compaction_revision metrics. If you run compaction yourself with etcdctl compact, remember that you are racing the API server’s own compactor for the same revision — which is safe, but means the API server’s CompactRevision() and your manual compaction can disagree. To see the election in action, read the compact_rev_key version with etcdctl get compact_rev_key --write-out=json and watch the version field increment by one every five minutes; that counter is the entire leader election.