Engineering Atlas › System Design Interviews

Kubernetes for system design interviews

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Objects and desired state

Everything is a typed object with metadata, spec (what you want) and status (what is). A Deployment's spec says "4 replicas of this pod template, roll out with maxSurge 25 percent and maxUnavailable 25 percent", which are also the defaults (Kubernetes docs). You never tell Kubernetes to start a container; you change an object and wait.

The control plane

The API server is the only component that talks to etcd: it authenticates, authorises with RBAC, runs admission webhooks, validates and stores objects. kube-controller-manager runs dozens of controllers (Deployment, ReplicaSet, Job, Node, EndpointSlice, garbage collector) in one process, and kube-scheduler assigns pods to nodes; each uses a Lease for leader election.

The nodes

Each node runs the kubelet, a container runtime such as containerd, and usually kube-proxy or an eBPF replacement. The kubelet watches for pods bound to its node, starts them, runs probes and reports status. Every pod gets its own IP from a flat network where pods reach each other without NAT; a CNI plugin (Calico, Cilium, the cloud's VPC plugin) provides it. A Service gives a stable virtual IP that kube-proxy turns into iptables, IPVS or nftables rules spreading traffic over ready endpoints. iptables is still the default mode, nftables has been GA since 1.33, and IPVS was deprecated in 1.35 and will be off by default from 1.40 (Kubernetes docs).

A Deployment rollout

The Deployment controller owns ReplicaSets, one per pod template, named by a hash of the template. On a template change it creates a new ReplicaSet and shifts replica counts between old and new within the surge and unavailability budget, waiting for new pods to become ready. The ReplicaSet controller creates or deletes pods to match each count.

Down the stack

Layer 1The contract

  • Declarative and eventually consistent. A successful write means the object is durably stored, not that the world has changed. Status catches up later.
  • Optimistic concurrency. Every object has a resourceVersion. An update with a stale one fails with 409 Conflict, so controllers re-read and retry instead of overwriting.
  • List and watch. A client can list a resource at a version and then watch from that version, receiving every later change in order, unless the version has been compacted, in which case it gets 410 Gone and must re-list (API concepts).
  • Scale envelope. Upstream tests target up to 5,000 nodes, 150,000 pods, 300,000 containers and 110 pods per node (Kubernetes docs). Each object is limited to about 1.5 MiB by etcd's default request size (etcd docs).
  • Failure behaviour. If the control plane is down, running pods keep running and Services keep routing; nothing new is scheduled or healed.

Layer 2The mechanism

API server and etcd. Objects are serialised (protobuf for built-in types) and stored under keys like /registry/pods/default/web-7d9f-abc. etcd is a Raft-replicated key-value store with MVCC on bbolt: each write gets a cluster-wide revision, which becomes the object's resourceVersion. The API server keeps a watch cache per resource type, so thousands of watchers are served from memory rather than from etcd, and it compacts old revisions every 5 minutes by default. Since 1.31 the cache also answers consistent ("most recent") lists, after confirming with etcd that it is up to date, so a plain LIST no longer has to read etcd.

Informers and work queues. A controller does not poll. A shared informer lists once, then watches, maintaining a local cache and calling handlers on add, update and delete. Handlers only push an object key (namespace/name) onto a rate-limited work queue, which deduplicates: ten updates to one Deployment while a worker is busy collapse into one pending key. Workers pop keys and call reconcile(key), which reads current state from the cache, computes the difference from spec, issues API writes and returns. Errors re-queue with exponential backoff. A periodic resync re-queues everything, so a missed event is eventually corrected.

Reconcile is level-triggered. Because the loop asks "how many pods should exist and how many do", it is idempotent and safe to run twice or after a crash. The Deployment controller also records observedGeneration in status so clients can tell whether the controller has seen the latest spec.

Scheduler. Unscheduled pods (empty spec.nodeName) enter a priority queue. The scheduling framework runs plugins in phases: PreFilter and Filter remove infeasible nodes (resources, taints, node affinity, topology spread), Score ranks the rest (balanced allocation, image locality, inter-pod affinity), then Reserve, Permit and finally Bind, which writes a Binding subresource setting nodeName. In large clusters it stops after finding enough feasible nodes (percentageOfNodesToScore). Decisions use requests, not actual usage (Scheduling Framework).

Kubelet, CRI and CNI. The kubelet syncs each pod: it asks the runtime over CRI gRPC to RunPodSandbox, which creates a pause container holding the pod's network namespace. The runtime calls the CNI plugin's ADD command with that namespace path; the plugin creates a veth pair or ENI attachment, assigns an IP from IPAM and returns it. Then the kubelet pulls images and calls CreateContainer and StartContainer for each container, placing them in a cgroup under kubepods/<qos>/pod<uid>. It heartbeats by renewing a Lease in kube-node-lease every 10 s (Kubernetes docs); if heartbeats stop for the 50 s node-monitor-grace-period, the node controller sets its Ready condition to Unknown and taints it unreachable, and pods are evicted after an automatically added 300 s toleration (Kubernetes docs).

Layer 3The machine

etcd is the throughput limit. Every write is a Raft append that must be fsynced on a quorum, so etcd wants WAL fsync p99 under 10 ms (etcd FAQ); on slow network disks, leader elections and API timeouts follow. Its default space quota is 2 GiB (8 GiB is the suggested maximum), and once exceeded the cluster becomes read-only until compaction and defragmentation. A cluster with 100,000 pods holds a few hundred MB in etcd, and each pod update rewrites the whole object.

Unbounded LIST calls are the classic overload. A list of all pods that bypasses the watch cache (an exact resourceVersion, or an old cluster) makes the API server read every object from etcd, decode it and re-encode JSON, using hundreds of MB and seconds of CPU. Even from cache, the full response is built in memory, which is why streaming lists (beta and on by default since 1.34) let informers start with a watch that sends the initial state item by item. API Priority and Fairness, stable since 1.29, queues requests by flow so one noisy client cannot starve the controllers (Kubernetes docs).

On the node, pod startup is dominated by image pulls (seconds to minutes), then CNI setup (tens to hundreds of ms). iptables-mode Service routing makes rule updates take seconds with tens of thousands of Services, and packet latency grows with the rule count; IPVS, nftables and eBPF datapaths use hash lookups (Kubernetes blog).

Why it's built this way

  1. Machines and networks fail all the time at scale, so components must recover from any state, which makes level-triggered reconciliation safer than event-driven scripts.
  2. Many independent controllers need a consistent shared truth, so all state goes through one API backed by a consensus store.
  3. Consensus writes are slow and etcd is small, so the API server caches and fans out watches, and objects stay small.
  4. Polling thousands of objects from dozens of controllers would crush the API, so informers list once and watch, and work queues collapse bursts.
  5. Scheduling and running containers are different problems at different scales, so the scheduler only binds and each kubelet owns its node, with CRI and CNI keeping runtimes and networks pluggable.

Diagram

sequenceDiagram
    participant U as kubectl
    participant A as "API server"
    participant E as etcd
    participant C as "Deployment + RS controllers"
    participant S as Scheduler
    participant K as Kubelet
    U->>A: apply Deployment (replicas 3)
    A->>E: write, revision 1042
    A-->>C: watch event
    C->>A: create ReplicaSet, then 3 Pods
    A-->>S: watch: pods with no node
    S->>A: Binding pod to node-7
    A-->>K: watch: pod bound here
    K->>K: CRI RunPodSandbox, CNI ADD
    K->>K: pull image, start containers
    K->>A: status Ready

Worked example

A Deployment has 100 replicas, maxSurge 25 percent and maxUnavailable 25 percent, so during a rollout there may be up to 125 pods and must be at least 75 ready.

The controller first scales the new ReplicaSet to 25 (surge) and the old one down to 75 (the unavailability budget allows losing 25). As new pods become ready, it moves the next 25 from old to new. If a new pod takes 30 s to pull, start and pass readiness, the rollout needs about four waves, roughly 2 to 3 minutes. Setting maxUnavailable 0 and maxSurge 10 percent keeps full capacity but needs ten waves and 10 extra pods of headroom on the nodes.

Control-plane cost: each pod generates roughly 5 to 10 writes over its life (create, bind, several status updates, delete). Replacing 100 pods is about 1,000 writes, trivial for etcd, which sustains thousands of writes per second on good disks. The same rollout across 2,000 Deployments at once is 2 million writes and can saturate it, which is why platform teams stagger large rollouts.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Managed Kubernetes (EKS, GKE, AKS)Standard API, huge ecosystem, bin packing, self-healingSteep learning curve, many moving parts, cluster upgradesDozens of services, a platform team
Self-managed KubernetesFull control, on-premYou run etcd and upgradesRegulatory or on-prem requirements
Serverless containers (Cloud Run, Fargate)No nodes, scale to zeroLess control, cold starts, per-request costFew services, spiky traffic
NomadSimpler, mixed workloadsSmaller ecosystemSmall ops team, VMs plus containers
VMs with autoscaling groupsSimple mental model, strong isolationSlow scaling, poor packingA few large stateless services

Pitfalls in production

Leadership lens

Check yourself

  1. Why are Kubernetes controllers level-triggered rather than edge-triggered?

    Show answer

    If they acted only on events, a missed or duplicated event or a crash would leave the system wrong forever. Comparing full desired and observed state on every pass corrects any drift.

  2. What happens to running pods if the whole control plane goes down for 10 minutes?

    Show answer

    They keep running and Services keep routing via existing rules, but nothing is scheduled, scaled or replaced, and status goes stale.

  3. Why does the scheduler write a Binding instead of telling the kubelet to start a pod?

    Show answer

    All coordination goes through the API; the binding is durable state the kubelet watches, so either side can crash and resume without a direct RPC.

  4. Why does a work queue deduplicate keys rather than queue every event?

    Show answer

    Reconcile reads the latest state anyway, so processing one key once covers all pending changes and bursts do not multiply work.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →