Leader Election Ignores the Clocks: observedTime and the Skew-Rate Ratio

TL;DR: The client-go leader elector never trusts the timestamps stored in the Lease record — those were produced by another machine’s clock. It uses the record’s content change only as a signal that somebody else renewed the lease, and records observedTime locally whenever it observes a change. Lease expiry is then observedTime + leaseDuration, computed entirely from the local clock. That makes the algorithm tolerant to arbitrary clock skew but not to arbitrary clock skew rate, and the tolerance to skew rate is a configurable ratio: LeaseDuration to RenewDeadline.

Every controller in a Kubernetes cluster that must run on exactly one instance — the kube-controller-manager’s individual controllers, the scheduler, the cloud-controller-manager, every operator you have ever installed — needs a leader election. The client-go implementation is one of the most widely-read pieces of Go in the ecosystem, and its design decision is unusual enough to be worth understanding: it throws away the timestamps other nodes write.

That sounds like a bug and is not. A leader election has to answer one question — “is the current leader still alive?” — and the obvious way to answer it is to read the lease’s expiry time and compare it to your clock. But the lease was written by another machine, whose clock may be wildly wrong. So the elector refuses to read a timestamp it cannot trust, and rebuilds the answer out of change detection plus its own monotonic observations.

The design, in the source comment

The package doc comment is the design document, and it is unusually honest about what the algorithm does and does not guarantee:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:22
// A client only acts on timestamps captured locally to infer the state of the
// leader election. The client does not consider timestamps in the leader
// election record to be accurate because these timestamps may not have been
// produced by a local clock. The implemention does not depend on their
// accuracy and only uses their change to indicate that another client has
// renewed the leader lease. Thus the implementation is tolerant to arbitrary
// clock skew, but is not tolerant to arbitrary clock skew rate.
//
// However the level of tolerance to skew rate can be configured by setting
// RenewDeadline and LeaseDuration appropriately. The tolerance expressed as a
// maximum tolerated ratio of time passed on the fastest node to time passed on
// the slowest node can be approximately achieved with a configuration that sets
// the same ratio of LeaseDuration to RenewDeadline. For example if a user wanted
// to tolerate some nodes progressing forward in time twice as fast as other nodes,
// the user could set LeaseDuration to 60 seconds and RenewDeadline to 30 seconds.
//
// While not required, some method of clock synchronization between nodes in the
// cluster is highly recommended. It's important to keep in mind when configuring
// this client that the tolerance to skew rate varies inversely to master
// availability.

Read that carefully. There are two distinct failure modes and the algorithm handles them differently:

Skew: node A’s clock is 10 minutes ahead of node B’s. Completely tolerated. Node A’s RenewTime is never compared to node B’s clock.
Skew rate: node A’s clock runs at 2x node B’s. Not tolerated by default, but tunable.

And the tuning knob is a ratio, not a duration. The example is concrete: “if a user wanted to tolerate some nodes progressing forward in time twice as fast as other nodes, the user could set LeaseDuration to 60 seconds and RenewDeadline to 30 seconds.” The defaults are 15s and 10s — a ratio of 1.5.

The state, and the change detection

The elector’s internal state is four fields:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:188
type LeaderElector struct {
	config LeaderElectionConfig
	// internal bookkeeping
	observedRecord    rl.LeaderElectionRecord
	observedRawRecord []byte
	observedTime      time.Time
	// used to implement OnNewLeader(), may lag slightly from the
	// value observedRecord.HolderIdentity if the transition has
	// not yet been reported.
	reportedLeader string

	// clock is wrapper around time to allow for less flaky testing
	clock clock.Clock

	// used to lock the observedRecord and the observedTime
	observedRecordLock sync.RWMutex

	metrics leaderMetricsAdapter
}

observedTime is the key. It is a local timestamp, taken by this node, at the moment this node last observed a change to the record. It is never read from the Lease.

The lease validity check uses only the local clock and the local observedTime:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:548
func (le *LeaderElector) isLeaseValid(now time.Time) bool {
	// Lock to safely read both the time and the record
	le.observedRecordLock.RLock()
	defer le.observedRecordLock.RUnlock()

	return le.observedTime.Add(time.Second * time.Duration(le.observedRecord.LeaseDurationSeconds)).After(now)
}

observedTime + LeaseDurationSeconds > now. Both terms are local. The LeaseDurationSeconds field comes from the record, but it is a duration, not a timestamp — durations are immune to skew by construction.

And the setter, which is where observedTime comes from:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:558
func (le *LeaderElector) setObservedRecord(observedRecord *rl.LeaderElectionRecord) {
	le.observedRecordLock.Lock()
	defer le.observedRecordLock.Unlock()

	le.observedRecord = *observedRecord
	le.observedTime = le.clock.Now()
}

The comment is precise: “update observedTime to the current time”. The record’s own RenewTime is copied into observedRecord but never used for arithmetic.

So the actual algorithm is: the record is a change-detection channel, not a clock. If the raw bytes of the record change, somebody renewed it, and observedTime is reset to now. If they do not change, the lease is expiring.

The change detection. The raw bytes matter. The elector compares the serialised record, not the decoded struct:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:486
	// 3. Record obtained, check the Identity & Time
	if !bytes.Equal(le.observedRawRecord, oldLeaderElectionRawRecord) {
		le.setObservedRecord(oldLeaderElectionRecord)

		le.observedRawRecord = oldLeaderElectionRawRecord
	}

Why raw bytes and not a field comparison? Because the record is written by a different implementation of the client — possibly a different language, possibly an older Kubernetes version. The only reliable way to detect “did anybody touch this” across versions is to compare the bytes. The cost is that a change to any field — including a field the elector does not care about — resets observedTime. That is the conservative direction: it makes the lease look fresher than it might be, so it delays a takeover rather than triggering a spurious one.

IsLeader is similarly local — it compares the observed holder identity against this client’s own identity:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:246
func (le *LeaderElector) IsLeader() bool {
	return le.getObservedRecord().HolderIdentity == le.config.Lock.Identity()
}

The optimistic fast path

Now the performance trick. In the steady state — we are the leader, the lease is valid, nothing else is competing — the elector does one write to the Lock and no read at all:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:454
	// 1. fast path for the leader to update optimistically assuming that the record observed
	// last time is the current version.
	if le.IsLeader() && le.isLeaseValid(now.Time) {
		oldObservedRecord := le.getObservedRecord()
		leaderElectionRecord.AcquireTime = oldObservedRecord.AcquireTime
		leaderElectionRecord.LeaderTransitions = oldObservedRecord.LeaderTransitions

		err := le.config.Lock.Update(ctx, leaderElectionRecord)
		if err == nil {
			le.setObservedRecord(&leaderElectionRecord)
			return true
		}
		logger.V(2).Info("Failed to update lease optimistically, falling back to slow path", "lock", le.config.Lock.Describe(), "err", err)
	}

The reasoning: “I believe I am the leader and my lease is still valid, so the record I saw last time is almost certainly still the current version. Let me just write it.” The Update is a plain write — no resourceVersion precondition — so it succeeds even if somebody else wrote in between. If it fails (conflict, network error), the elector falls through to the slow path.

The slow path reads the record first, compares, and then writes:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:469
	// 2. obtain or create the ElectionRecord
	oldLeaderElectionRecord, oldLeaderElectionRawRecord, err := le.config.Lock.Get(ctx)
	if err != nil {
		if !errors.IsNotFound(err) {
			logger.Error(err, "Error retrieving lease lock", "lock", le.config.Lock.Describe())
			return false
		}
		if err = le.config.Lock.Create(ctx, leaderElectionRecord); err != nil {
			logger.Error(err, "Error initially creating lease lock", "lock", le.config.Lock.Describe())
			return false
		}

		le.setObservedRecord(&leaderElectionRecord)

		return true
	}

Note that Get returns both the decoded record and the raw bytes — that is what makes the change detection in step 3 possible.

Then the identity and validity check, and the two branches that decide whether to preserve or increment the transition counter:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:492
	if len(oldLeaderElectionRecord.HolderIdentity) > 0 && le.isLeaseValid(now.Time) && !le.IsLeader() {
		logger.V(4).Info("Lease is held by and has not yet expired", "lock", le.config.Lock.Describe(), "holder", oldLeaderElectionRecord.HolderIdentity)
		return false
	}

	// 4. We're going to try to update. The leaderElectionRecord is set to it's default
	// here. Let's correct it before updating.
	if le.IsLeader() {
		leaderElectionRecord.AcquireTime = oldLeaderElectionRecord.AcquireTime
		leaderElectionRecord.LeaderTransitions = oldLeaderElectionRecord.LeaderTransitions
		le.metrics.slowpathExercised(le.config.Name)
	} else {
		leaderElectionRecord.LeaderTransitions = oldLeaderElectionRecord.LeaderTransitions + 1
	}

The slowpathExercised metric is worth noting: it is the signal that the fast path is failing, which usually means contention or a lock that is being written by something other than this client.

So the steady state is: one PUT every RetryPeriod, zero GETs. The slow path costs a GET plus a PUT, and only happens when the optimistic write fails or when the elector is not the leader.

The renew loop

The renew loop ties the timings together:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:285
		err := wait.PollUntilContextTimeout(ctx, le.config.RetryPeriod, le.config.RenewDeadline, true, func(ctx context.Context) (done bool, err error) {
			// PollUntilContextTimeout invokes condition even when the context is canceled when immediate=true.
			// Short-circuit this to prevent unnecessary processing and error log messages.
			if err := ctx.Err(); err != nil {
				return false, err
			}
			if !le.config.Coordinated {
				return le.tryAcquireOrRenew(ctx), nil
			} else {
				return le.tryCoordinatedRenew(ctx), nil
			}
		})
		le.maybeReportTransition()
		desc := le.config.Lock.Describe()
		if err == nil {
			logger.V(5).Info("Successfully renewed lease", "lock", desc)
			return
		}
		le.metrics.leaderOff(le.config.Name)
		logger.Info("Failed to renew lease", "lock", desc, "err", err)
		cancel()

The inner poll has a RenewDeadline timeout. If tryAcquireOrRenew does not succeed within RenewDeadline, the whole thing is cancelled and OnStoppedLeading fires. So RenewDeadline is the tolerance to a hung API server: it is the maximum time the leader will keep acting as leader without a successful renewal.

That is why the client timeout is set to half the renew deadline:

go
// staging/src/k8s.io/client-go/tools/leaderelection/resourcelock/interface.go:140
// NewFromKubeconfig will create a lock of a given type according to the input parameters.
// Timeout set for a client used to contact to Kubernetes should be lower than
// RenewDeadline to keep a single hung request from forcing a leader loss.
// Setting it to max(time.Second, RenewDeadline/2) as a reasonable heuristic.
func NewFromKubeconfig(lockType string, ns string, name string, rlc ResourceLockConfig, kubeconfig *restclient.Config, renewDeadline time.Duration) (Interface, error) {
	// shallow copy, do not modify the kubeconfig
	config := *kubeconfig
	timeout := renewDeadline / 2
	if timeout < time.Second {
		timeout = time.Second
	}
	config.Timeout = timeout

A single hung request must not be able to eat the whole RenewDeadline budget. With a client timeout of RenewDeadline/2, two consecutive hung requests still leave time for a third attempt to succeed. That is the “reasonable heuristic”.

The config validation

The constructor rejects configurations that cannot work:

go
// staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:71
const (
	JitterFactor = 1.2
)

// NewLeaderElector creates a LeaderElector from a LeaderElectionConfig
func NewLeaderElector(lec LeaderElectionConfig) (*LeaderElector, error) {
	if lec.LeaseDuration <= lec.RenewDeadline {
		return nil, fmt.Errorf("leaseDuration must be greater than renewDeadline")
	}
	if lec.RenewDeadline <= time.Duration(JitterFactor*float64(lec.RetryPeriod)) {
		return nil, fmt.Errorf("renewDeadline must be greater than retryPeriod*JitterFactor")
	}

The JitterFactor of 1.2 exists because acquire uses wait.JitterUntilWithContext — the retry interval is randomised by up to ±20% so that a fleet of controllers starting at the same moment does not stampede the API server with simultaneous lock attempts. The validation guarantees that even at the top of the jitter band, several retries fit inside RenewDeadline:

RenewDeadline > 1.2 × RetryPeriod.

The defaults come from component-base:

go
// staging/src/k8s.io/component-base/config/v1alpha1/defaults.go:35
func RecommendedDefaultLeaderElectionConfiguration(obj *LeaderElectionConfiguration) {
	zero := metav1.Duration{}
	if obj.LeaseDuration == zero {
		obj.LeaseDuration = metav1.Duration{Duration: 15 * time.Second}
	}
	if obj.RenewDeadline == zero {
		obj.RenewDeadline = metav1.Duration{Duration: 10 * time.Second}
	}
	if obj.RetryPeriod == zero {
		obj.RetryPeriod = metav1.Duration{Duration: 2 * time.Second}
	}

LeaseDuration 15s, RenewDeadline 10s, RetryPeriod 2s. The ratio 15/10 = 1.5 is the default tolerance to skew rate: a node whose clock runs up to 1.5x as fast as the slowest node will not cause a spurious takeover. Note that the default ResourceLock is still EndpointsResourceLock, with a TODO (#80289) about migrating to LeaseLock.

The new coordinated leader election

There is a second, newer implementation in the tree, and it takes a completely different approach: the election itself moves into the API server as a controller, and the candidates merely respond to pings.

The constants are small and fixed: electionDuration = 5 * time.Second, defaultLeaseDurationSeconds = 5, defaultRequeueInterval = 5 * time.Second and leaseCandidateValidDuration = 30 * time.Minute (pkg/controlplane/controller/leaderelection/leaderelection_controller.go:46).

The protocol is a two-phase ping/ack. The controller stamps candidate.spec.pingTime; the candidate acknowledges by setting renewTime >= pingTime:

go
// pkg/controlplane/controller/leaderelection/leaderelection_controller.go:288
		if candidate.Spec.PingTime == nil ||
			// If PingTime is outdated, send another PingTime only if it already acked the first one.
			// This checks for pingTime <= renewTime because equality is possible in unit tests using a fake clock.
			(candidate.Spec.PingTime.Add(electionDuration).Before(now) && !candidate.Spec.RenewTime.Before(candidate.Spec.PingTime)) {
			clone := candidate.DeepCopy()
			clone.Spec.PingTime = &metav1.MicroTime{Time: now}
			g.Go(func() error {
				_, err := c.leaseCandidateClient.LeaseCandidates(clone.Namespace).Update(gCtx, clone, metav1.UpdateOptions{})
				return err
			})
			canVoteYet = false
		}
	}

The state machine is spelled out in the function’s doc comment (pkg/controlplane/controller/leaderelection/leaderelection_controller.go:245): “Instead of keeping a map and lock on election, the state is calculated every time by looking at the lease, and set of available candidates. PingTime + electionDuration > time.Now: We just asked all candidates to ack and are still waiting for response. PingTime + electionDuration < time.Now: Candidate has not responded within the appropriate PingTime. Continue the election.”

Note the contrast with the client-go elector: this one does compare timestamps, but it does so on a single node — the API server that runs the controller. All candidates compare their clocks against one clock, which is the API server’s. Skew rate matters, but skew between candidates does not, because only the controller’s clock is used to decide expiry.

The winner is decided by version, not by timestamp — the oldest emulation version wins:

go
// pkg/controlplane/controller/leaderelection/election.go:29
func pickBestLeaderOldestEmulationVersion(candidates []*v1beta1.LeaseCandidate) *v1beta1.LeaseCandidate {
	var electee *v1beta1.LeaseCandidate
	for _, c := range candidates {
		if !validLeaseCandidateForOldestEmulationVersion(c) {
			continue
		}
		if electee == nil || compare(electee, c) > 0 {
			electee = c
		}
	}
	return electee
}

with a strict lexicographic comparison:

go
// pkg/controlplane/controller/leaderelection/election.go:92
// -1: lhs better, 1: rhs better
func compare(lhs, rhs *v1beta1.LeaseCandidate) int {
	l := getEmulationVersionOrZero(lhs)
	r := getEmulationVersionOrZero(rhs)
	result := l.Compare(r)
	if result == 0 {
		l := getBinaryVersionOrZero(lhs)
		r := getBinaryVersionOrZero(rhs)
		result = l.Compare(r)
	}
	if result == 0 {
		if lhs.CreationTimestamp.After(rhs.CreationTimestamp.Time) {
			return 1
		}
		return -1
	}
	return result
}

Emulation version first (so an old binary running in compatibility mode wins over a new one — the point is to keep behaviour stable during an upgrade), then binary version, then creation timestamp as a final tiebreak. The strategy name says it: OldestEmulationVersion.

This matters operationally: the leader is deterministic given the same set of candidates, so a restart does not reshuffle leadership. The LeaseCandidate objects are garbage collected after leaseCandidateValidDuration = 30 * time.Minute, and an election is forced every 15 minutes to refresh them.

The trade-offs. Skew tolerance costs availability — the doc comment says it directly: “the tolerance to skew rate varies inversely to master availability”. A node running twice as fast forces you to double LeaseDuration, which doubles the time a takeover takes after a crash. The optimistic fast path can lose a race: the Update has no resourceVersion precondition, so it will happily overwrite a record that another client changed in the meantime. That is why the slow path exists, and why the slow path is the one that checks the identity. And the raw-byte comparison is coarse: any change to any field resets observedTime, including changes the elector does not care about. The coordinated elector, meanwhile, is not skew-tolerant between candidates at all — it trades that for a single-clock comparison and deterministic leader selection.

Wrapping up

The one thing to remember: the client-go leader elector does not read the clock out of the Lease. It uses observedTime — a local timestamp recorded whenever the raw bytes of the record change — and computes expiry as observedTime + LeaseDurationSeconds (staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go:548). The record’s RenewTime is copied around but never used for arithmetic.

The part people get wrong: that the elector is immune to clock problems. It is immune to skew, not to skew rate. The tolerance is the ratio LeaseDuration : RenewDeadline, which defaults to 15s : 10s = 1.5. A node whose clock runs 1.5x faster than the slowest node can cause a spurious takeover; a node running 10x faster will cause flapping.

What to go do about it: if you see leader flaps, look at leader_election_master_transitions on the leader_election metric and at the Lease object’s leaderTransitions field — the transition counter is the direct evidence. The fix is to widen the ratio, exactly as the doc comment says: set LeaseDuration to 60s and RenewDeadline to 30s to tolerate a 2x skew rate. Validate it: NewLeaderElector will reject leaseDuration <= renewDeadline or renewDeadline <= 1.2 * retryPeriod, so your RetryPeriod has to shrink too. And check that the client timeout is at most RenewDeadline/2 (that is what NewFromKubeconfig sets) — a hung API server request that consumes the whole renew deadline will cost you the leadership even though nothing else is wrong.