Engineering Atlas › System Design Fundamentals

Updating servers a few at a time

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The loop

The orchestrator starts new-version instances, waits for readiness, adds them to the load balancer, then drains and terminates old instances, repeating until none remain. Kubernetes expresses this as maxSurge (extra pods above desired) and maxUnavailable (pods below desired that may be not ready). With 20 replicas, maxSurge=5 and maxUnavailable=0, it runs up to 25 pods and always keeps 20 ready. Outside Kubernetes the same idea holds: Amazon's pipelines typically replace at most a third of a fleet at once, keeping at least two thirds healthy, and some teams go as low as 5% (AWS Builders' Library).

Readiness versus liveness

Readiness means "send me traffic". Liveness means "restart me". A readiness probe should check that the process has loaded config, warmed critical caches, and can reach hard dependencies. If readiness returns true too early, the rollout races ahead and users hit cold instances. Add a minimum ready time (for example 30 seconds) so a pod that crashes 10 seconds after start does not count as healthy; Kubernetes' minReadySeconds defaults to 0, so this is opt-in (Kubernetes docs). Load balancers can help too: an AWS target group's slow start ramps a new target's share of traffic linearly over 30 to 900 seconds instead of giving it a full share at once (AWS docs).

Draining

Terminating an old instance must be graceful. The orchestrator removes it from the load balancer, the instance stops accepting new connections, finishes in-flight requests within a grace period, then exits. Endpoint removal propagates asynchronously, so a short pre-stop delay of a few seconds avoids sending requests to a process that has already closed its listener. Kubernetes now has a built-in sleep action for preStop hooks, stable since v1.34, so images no longer need a sleep binary just for this (KEP-3960). The grace period clock starts before the hook runs, so the hook and the app's own shutdown must both fit inside terminationGracePeriodSeconds (Kubernetes docs).

Progress and halt

A rollout needs a deadline. If new instances do not become ready within, say, 10 minutes, the controller marks the rollout failed and stops creating more. It does not roll back on its own unless your tooling adds that. Better pipelines watch error rates and latency on new instances during the rollout and trigger an automated rollback.

Mixed versions

For the whole rollout window, some requests hit v1 and some hit v2. A user can bounce between them on consecutive requests. Anything shared, such as the database schema, cache entries, message formats, and session data, must be readable by both versions.

Down the stack

Layer 1The contract

In Kubernetes, the Deployment controller promises two bounds throughout an update: available pods never fall below replicas - maxUnavailable, and pods counted toward the rollout never exceed replicas + maxSurge. Terminating pods are not counted, so while old pods drain the real pod count and resource use can exceed replicas + maxSurge until their grace period ends; since v1.35 (beta) the Deployment reports them in .status.terminatingReplicas (Kubernetes docs). Percentages round surge up and unavailability down. "Available" means ready for at least minReadySeconds. If no progress happens within progressDeadlineSeconds (600 by default), the Deployment reports Progressing=False with reason ProgressDeadlineExceeded. It does not roll back on its own. The contract says nothing about request success: dropping in-flight requests is your problem.

Layer 2The mechanism

A Deployment owns ReplicaSets, one per pod template, named with a pod-template-hash label. An update creates a new ReplicaSet and the controller's reconcile loop scales it up and the old one down in steps that respect the two bounds. Old ReplicaSets are kept at zero replicas (revisionHistoryLimit, default 10), so rollout undo simply scales an old one back up; setting the limit to 0 deletes that history and with it the ability to roll back (Kubernetes docs). When a pod becomes ready, the EndpointSlice controller adds its IP; when it starts terminating, the IP is removed. Termination runs the preStop hook, then sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds (30 by default).

Layer 3The machine

Endpoint removal is not instantaneous. The change goes from the API server through watches to kube-proxy on every node, which rewrites its packet-forwarding rules: iptables is still the default mode in Kubernetes 1.37, with IPVS and nftables as alternatives and nftables planned as the future default (Kubernetes docs). On a 1,000-node cluster with many services, this can take several seconds, and existing TCP connections keep flowing through conntrack entries to the old pod IP. Clients using HTTP keep-alive stay pinned to the old pod until the connection closes, which is why servers send Connection: close or an HTTP/2 GOAWAY during drain. A new pod pays for image pull (hundreds of MB at roughly 100 MB/s if layers are not cached on the node), process start, JIT warmup on the JVM, empty CPU caches, and new TLS handshakes to every dependency for its connection pools. Its first requests can be several times slower than a warm pod's.

Why it's built this way

Controllers reconcile desired state in loops, so a rollout survives controller restarts and is resumable. Pod templates are immutable, so each version gets its own ReplicaSet, and rollback is just scaling. Endpoint state spreads to many nodes asynchronously, so pods need a pre-stop delay to avoid receiving traffic after shutdown. Nodes are shared and bin-packed, so surge capacity is limited by free cluster space, which is why both knobs exist.

Diagram

sequenceDiagram
  participant O as Orchestrator
  participant N as New pod v2
  participant LB as Load balancer
  participant Old as Old pod v1
  O->>N: create (surge slot)
  N-->>O: readiness OK after warmup
  O->>LB: add v2 endpoint
  O->>LB: remove v1 endpoint
  O->>Old: SIGTERM after preStop delay
  Old-->>Old: finish in-flight requests
  Old-->>O: exited
  O->>O: next batch until v1 count is 0

Worked example

A checkout API runs 40 pods at 60 percent CPU at peak, each handling 250 RPS, 10,000 RPS total. Pods take 45 seconds to start and become ready.

With maxUnavailable=10% and maxSurge=0, the deploy removes 4 pods at a time. The other 36 must absorb 10,000 RPS, about 278 RPS each, roughly 67 percent CPU. That is safe. The rollout needs 10 batches of about 60 seconds each including drain, so about 10 minutes.

With maxUnavailable=25%, 30 pods carry 333 RPS each at 80 percent CPU. Latency climbs and autoscaling may fire mid-deploy, adding noise. Faster, but riskier at peak.

With maxSurge=25% and maxUnavailable=0, the cluster briefly runs 50 pods. No capacity loss, and 4 batches finish in about 4 minutes, but the cluster needs room for 10 extra pods. If the node pool is full, the surge pods stay pending and the rollout stalls. The usual choice is surge-based rolling with spare headroom reserved, or a smaller unavailability budget off-peak.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Rolling with surgeNo capacity loss, no full duplicate fleetNeeds spare cluster headroom, mixed versionsDefault for stateless services
Rolling with unavailabilityNo extra capacity neededReduced capacity during deployTight clusters, off-peak deploys
Recreate (stop all, start all)No version mixingFull downtimeIncompatible changes, dev environments
Blue-greenInstant switch and instant rollbackDouble capacity, still mixing at the data layerRollback speed matters more than cost

Pitfalls in production

Leadership lens

Check yourself

  1. With 10 replicas, maxSurge=0 and maxUnavailable=0, what happens?

    Show answer

    The rollout cannot make progress: it may not add a pod and may not remove one. Kubernetes rejects this combination for that reason.

  2. Why can a readiness probe that returns 200 as soon as the HTTP server binds make a rolling deploy dangerous?

    Show answer

    Old pods are removed as soon as new ones report ready, so traffic shifts to instances that have not loaded config or warmed caches, causing errors or latency spikes across the whole fleet.

  3. Why is rollback with a rolling strategy slower than with blue-green?

    Show answer

    Rolling back replaces instances batch by batch with the same pacing as the rollout, while blue-green keeps the old fleet running and just switches traffic back.

  4. A v2 writes a new enum value to a shared table during the rollout. What breaks?

    Show answer

    v1 instances still serving traffic may read the unknown value and fail. The value should only be written after every instance can read it.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →