A Tour of Kubernetes Internals: The Thirty-Six Tricks That Make It Scale

TL;DR: Kubernetes is a distributed system that pretends to be a declarative API. The pretence is the whole design: a YAML file is a diff against a desired state, and thirty different reconcilers are each responsible for making one part of the world match it. What makes that work at scale is a pile of small, unglamorous decisions - a copy-on-write snapshot so the scheduler never locks, a memoized serializer so one etcd event can feed a thousand watchers, a workqueue built from two sets and a FIFO. This is a tour of thirty-six of them, and where in the source tree each one lives.

kubectl apply -f deployment.yaml is not a request to create a Deployment. It is a request to make the world look like a Deployment, and the interesting question is always how much machinery stands between that sentence and the running pod. The answer is a lot: a hash, a watch cache, an informer, a workqueue, a scheduler snapshot, a cgroup, a conntrack entry.

This is a tour of thirty-six of the decisions inside that path - the ones that are worth knowing about, grouped by the part of the system they live in. Every article quotes the source it describes.

1. The scheduler: sampling, snapshots, and stealing

The scheduler runs one cycle at a time, on a cache that other goroutines are writing to, and has to answer “where should this pod go” in milliseconds without locking anything and without looking at every node.

• The Scheduler Never Locks the Cache: Generation Stamps and Copy-on-Write NodeInfos - the cache keeps its NodeInfos on a recency-ordered list and stamps each with a monotonic generation, so a cycle clones the two nodes that changed instead of the five thousand that did not.
• 50 Minus N/125: How the Scheduler Decides to Look at 5% of a 10,000-Node Cluster - percentageOfNodesToScore: 0 is not “score everything”, it is a formula that shrinks as the cluster grows, plus a rotating start index so no node is starved forever.
• Remove Everything, Then Reprieve: How Preemption Finds the Smallest Victim Set - preemption does not search combinations; it removes every candidate victim, checks whether the preemptor fits, then re-adds victims in importance order until it stops fitting.
• The Hostname Fast Path: Inter-Pod Affinity Without the Cluster-Wide Topology Map - when every affinity term is scoped to kubernetes.io/hostname, the cluster-wide topology map is pure waste, and the plugin skips it entirely.

2. The API server: one event, a thousand watchers

The API server’s job is to turn etcd writes into streams of events, and its hot path is fan-out. Almost every trick here exists because one etcd event can reach thousands of watchers.

• Serialize Once, Deliver to a Thousand Watchers: The cachingObject - the watch event wraps its object in a memoizing wrapper, so the JSON is produced once and the deep copy happens only if something actually mutates it.
• Bookmarks in a Bucket: How the Watch Cache Notifies 100k Watchers That It Is Alive - instead of a timer per watcher, progress notifications are bucketed by whole second and popped in bulk; and the WatchList protocol gets a three-state machine so no client can miss an event.
• I’m Behind: The Three-Second Grace Period Before the Watch Cache Gives Up - the cache waits three seconds for etcd to catch up before returning “too old resource version”, and skips a goroutine entirely in the most common case of all.
• Double the Page Size: How etcd LIST Escapes the Selector Death Spiral - a filtered LIST that returns too few matches makes etcd issue tiny pages forever, so the store doubles the page size on every iteration and caps it at ten thousand.
• One Allocation Per Object: The Reverse Marshal Inside the Protobuf Envelope - the protobuf wire format nests the object in a runtime.Unknown, and the serializer marshals backwards from the end of a pre-sized buffer so the object is never copied.
• The Admission Chain Runs Twice: Reinvocation and the Webhook That Was Skipped - mutating admission gets a second pass so a later webhook can fix an earlier one, and the bookkeeping makes sure a webhook that was skipped the first time is not suddenly called the second.
• ResourceQuota Batches Per Namespace: The LRU That Killed etcd Conflicts - quota evaluation is batched per namespace, and a hundred-entry cache of recently written quotas - compared by resourceVersion - took the etcd conflict rate to zero.
• One Wildcard RBAC Probe Instead of N Authorizations - a single update */* finalizers probe short-circuits the per-ownerReference authorization loop, which otherwise has to resolve every owner through the RESTMapper.
• The Compactor Elects Itself With a Compare-and-Swap on a Magic Key - no lease and no leader election record: the version number of one etcd key is the clock, and the FAQ in the source comment is “what if time is not accurate? we don’t care”.
• Leader Election Ignores the Clocks: observedTime and the Skew-Rate Ratio - the elector refuses to trust timestamps written by another machine, and the tolerance for clock skew rate is nothing more than the ratio of two timeouts.

3. client-go: the machinery everybody copies

Every controller in the wild is a variation on the same four components: a reflector, a DeltaFIFO, an informer, and a workqueue. They are also the components with the sharpest edges.

• One Stream Instead of Two: The Watch List That Killed LIST + WATCH - sendInitialEvents=true turns the list into the first half of a watch, and the reflector deliberately turns pagination off so the server answers from memory instead of etcd.
• The Reflector Refuses to Publish a Resource Version It Hasn’t Earned - synthetic ADDED events are not in resource-version order, so the reflector withholds the version until it sees something else, and its backoff is tuned by traffic arithmetic rather than feel.
• Two Sets and a FIFO: The Whole Concurrency Model of the Kubernetes Workqueue - a dirty set, a processing set and a queue give you coalescing, mutual exclusion and requeue-on-race with no extra bookkeeping, and AddAfter uses one heap instead of a thousand timers.
• The Informer Will OOM If You Are Too Slow: An Unbounded Ring Buffer, On Purpose - each handler gets an unbounded growing ring, the source comment says so outright, and the DeltaFIFO’s Pop runs your handler under the queue lock to stop the backlog from growing.
• Kubernetes Forked encoding/json Because the Standard One Loses Integers - the stdlib decoder matches fields case-insensitively and turns every number into a float64, so the API server carries its own fork with strict error accumulation and a hundred-error cap.
• Field Ownership as a Sorted Slice: The Data Structure Behind Server-Side Apply - managedFields is a trie, the trie is a sorted slice rather than a map so it serializes deterministically, and union and difference are merge-joins over two sorted slices.
• $setElementOrder: How a Strategic Merge Patch Remembers the Order of a List - merging a list by key loses the order, so the patch carries a parallel directive list, and reconciling three different orderings is a partition-then-interleave.

4. The kubelet: the part that actually touches the hardware

Everything above runs in a control plane that someone else can reboot. The kubelet is the part that has to be right about cgroups, memory pressure and the OOM killer, because nobody is watching.

• The Kubelet Writes Its State to Disk: FNV-32a Checksums and a String Rename for Rollback - checkpoints are checksummed over a hash dump rather than the serialized bytes, and the memory manager renames a type in a string so an old kubelet can still verify a new checkpoint after a rollback.
• The Kubelet Evicts the Pod That Misbehaves First: A Three-Key Ranking - memory-pressure eviction ranks by “does this pod exceed its request” before it ranks by size, so the culprit goes first and the merely large goes last.
• The Kubelet Doesn’t Poll for Memory Pressure: epoll, eventfd and the memcg Threshold - the kernel writes an eventfd when a memcg threshold is crossed, the kubelet epolls it, and it converts a working-set threshold into a usage threshold by subtracting the reclaimable page cache.
• Anding the NUMA Bitmasks: How the Topology Manager Merges Three Providers - the CPU, memory and device managers each propose a NUMA affinity, and the merge is a bitwise AND; a score of zero means “no signal”, not “zero utilization”, and that distinction changes the average.
• A Continuous OOM Score: Why Burstable Containers Don’t All Get 0 - the OOM score of a Burstable container is computed continuously from its request, so the container that exceeds its request is the prime kill target, and a sidecar is clamped so it never dies first.

5. The controllers: acting on a cache that lags

A controller reads from an informer cache, decides, and writes to the API server. Between those two things is a window in which the world can change underneath it, and the code is full of ways of not caring.

• Expectations: How a Controller Waits for Its Own Writes to Come Back - the controller raises an expectation before it writes and lowers it when its own write comes back through the watch, which is what stops the double-create; and the delete path is tracked by UID.
• Not a Random Suffix: The Deployment Hash and Its Collision Counter - the ReplicaSet name is an FNV-32a hash of the pod template, safe-encoded into a vowel-free alphabet so it cannot spell a profanity, and collisions are escaped with a persisted counter rather than a random suffix.
• Solid, Dangling, Waiting: How the Garbage Collector Decides to Delete - owner references are classified into three buckets using a metadata-only client, and there is a cycle breaker for the case where two objects in foreground deletion are blocking each other forever.
• Before It Evicts Your Node, the Node Controller Checks Its Own Cache - a stale lease in the informer cache would mass-evict every node, so the controller re-reads the lease live and, if the cache is stale, aborts the entire pass instead of acting on it.

6. Networking: a service is a rule

The service proxy is the oldest part of Kubernetes and the one that has to survive the most constraints: 28-character chain names, a kernel that will happily accept thousands of rules, and a datapath that has to stay connection-safe.

• The 1/n Lottery: How kube-proxy Load-Balances Without a Single Line of State - the iptables proxier load-balances with a cascade of probabilistic rules whose probabilities telescope into an exact uniform split, and the kernel picks the branch per new connection.
• Above 1000 Endpoints, kube-proxy Deletes Its Own Comments - in large clusters the proxier drops the comment on every rule, skips the hourly full sync and defers chain garbage collection, and a discard buffer keeps the metrics honest while the bytes go nowhere.
• One Rule Per Service: How nftables Replaced kube-proxy’s Chain of Chains - the nftables proxier replaces N chains with one numgen random mod N map rule and a verdict map, and runs a real userspace proxy for localhost NodePorts because nftables cannot do route_localnet.
• Weight Zero, Wait, Then Delete: The IPVS Connection Drain - IPVS mode sets a departing backend’s weight to zero and waits for the kernel to drain its connections, while UDP and SCTP skip the whole thing.
• Hints by CPU Share: How the Topology Cache Keeps a Zone From Overloading - zone-aware hints are allocated by CPU share with a 20% overload guarantee, and the consumer ignores the hints entirely unless every endpoint has one, which is the difference between spreading traffic and starving two pods.
• Random ClusterIPs on Purpose: The Allocator That Fights eBPF Hash Collisions - sequential ClusterIPs are pathological for eBPF datapaths that hash the 5-tuple, so the allocator picks a random offset and scans, and reserves a lower subrange for statically requested addresses.

How to read the Kubernetes source

The tree is big, but it is organised by the same boundaries this tour follows:

Two habits make the code readable. First, almost every interesting decision is a named constant with a comment explaining the number - minFeasibleNodesToFind = 100, overloadThreshold = 0.2, maxLimit = 10000, blockTimeout = 3 * time.Second. Second, the comments explain why, usually by naming the bug or the incident that caused the code to exist (“this adds ~1s of latency on all transitions from sync to terminating”, “we don’t want to do this on every sync in large clusters”).

Trade-offs

Almost every trick here buys its speed with something. The watch cache trades freshness for throughput: a LIST from the cache can be a few milliseconds behind etcd, and it can be arbitrarily behind if the cache is unhealthy, which is why it eventually returns an error instead of lying. The scheduler trades exactness for latency: it samples nodes and rotates the starting point, so a pod might not land on the theoretically best node, but no node is starved. The informer trades memory for correctness: an unbounded queue means a slow handler eventually takes the process down rather than silently dropping events. The large-cluster mode of kube-proxy trades debuggability for rule count, which is why iptables-save stops being useful past a thousand endpoints.

The pattern is consistent: Kubernetes optimises for the steady state at scale, and pays for it in the corners.

The bottom line

The one thing to remember: the interesting parts of Kubernetes are not the API. They are the places where the code refuses to do the obvious thing - clone two nodes instead of five thousand, serialize once instead of a thousand times, use one heap instead of a thousand timers, and re-read a lease from the server before it kills your node.

The part people get wrong: most of these behaviours are silent. The scheduler does not tell you it sampled 5% of the cluster. The watch cache does not tell you it served you a slightly stale list. kube-proxy does not tell you it stopped writing comments. The defaults are tuned for the cluster size that was current when they were written, and the knobs that would tell you otherwise are often not exposed.

What to go do about it: read the constants. grep -rn "= 100" pkg/scheduler/ and grep -rn "threshold" pkg/kubelet/eviction/ take ten minutes and explain more about how Kubernetes behaves than most of the documentation does. Then watch the metrics that those constants drive - scheduler_pending_pods, apiserver_watch_cache_read_wait_seconds, kubelet_evictions, kube_proxy_iptables_rules_total - and remember that each of them is a window into one of the tricks above.

The thing people get wrong

Kubernetes is often described as a declarative shell over an imperative system, and the phrase makes people look for reconciliation loops in the wrong place. The reconciliation loops are the easy part; they are a Get, a comparison and a Write. The hard part is everything the source does to make those three operations safe at ten thousand nodes per second: the snapshots, the expectations, the caches, the generation stamps, the counters, the frozen hashes, the discarded buffers.

None of it is exotic. It is all a set of small, well-commented decisions about what to do when the world changes underneath you - which, in a distributed system, it always does.