Engineering Atlas › System Design Fundamentals

Safe rollbacks and immutable servers

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Mutable versus immutable

In the mutable model, a configuration tool changes long-lived servers in place. Over months they drift: a hand hotfix on one host, a failed run on another. Rollback means running the untested inverse of every change.

In the immutable model, the build produces an image (container image or VM image) containing the OS layer, runtime, app, and default config. Instances are created from it and never modified. To change anything, you build a new image and replace instances. Drift cannot accumulate, and any instance can be killed and recreated identically.

Identity by digest

Tags like api:latest or even api:2.4.1 can be moved to point at different content. A content digest (sha256:...) cannot. Deploy manifests should pin digests. Then "what is running" has a precise answer, and "roll back to what ran yesterday" is a lookup in deploy history. Kubernetes says this directly: tags can be moved but digests are fixed, and :latest in production makes a proper rollback harder (Kubernetes docs). Registries can also refuse to overwrite tags; Amazon ECR's immutable mode rejects a push to an existing tag with ImageTagAlreadyExistsException (ECR docs).

What gets rolled back

A release has several layers with different rollback stories:

  1. Code and image: redeploy the previous digest. Fast.
  2. Configuration: version it with the release, or store it with history so it can be reverted.
  3. Feature flags: flip off. Fastest of all, no deploy.
  4. Schema: only reversible if the change was additive. Destructive changes are one-way.
  5. Data and side effects: emails sent, payments captured, messages published to other teams. Not reversible; needs compensating actions.

Making rollback safe

The previous version must understand anything the new version wrote. If v2 starts writing a new message format, v1 consumers must already accept it, or rolling back leaves unreadable messages in queues. The usual technique is a two-phase change: first ship a version that can read the new format but still writes the old one; once that is everywhere, ship the version that writes the new format. Rolling back the second step is then safe. Amazon calls these the prepare and activate phases, leaves a bake of usually a few days between them, and proves the property with upgrade-downgrade tests: deploy to half the fleet, finish, then roll back and check for errors (AWS Builders' Library).

Roll back or roll forward

Roll back when the cause is unclear or impact is ongoing. Roll forward when the fix is tiny and well understood, or when rollback is unsafe because of data changes. The decision rule should be written down before the incident. Google's canarying guidance shows why the default leans to rollback: a team with no rollback path has to patch and redeploy during the outage, which almost always lengthens user impact (SRE Workbook). A third option is to deploy fresh infrastructure from the last known good configuration (Azure Well-Architected).

Down the stack

Layer 1The contract

An OCI registry promises that a manifest digest always resolves to the same bytes. The spec makes the digest the content identifier and says clients should verify retrieved bytes against it (OCI image spec). A tag promises nothing beyond "currently points here", unless the registry enforces immutable tags. A deploy system promises that a release id reproduces exactly one artifact plus one configuration. Rollback promises to restore that pair. It does not promise anything about data, caches, queues, or calls made to other systems.

Layer 2The mechanism

An image manifest is a small JSON document listing a config blob and an ordered list of layer descriptors, each with a media type, size, and SHA-256 digest. Multi-architecture images add an index that points to one manifest per platform. A Kubernetes pod spec can reference registry/api@sha256:..., and the container runtime checks that pulled content hashes to that value. A Deployment keeps each revision as an old ReplicaSet, 10 by default; kubectl rollout undo scales one back up, and once that ReplicaSet is pruned the revision cannot be restored. Kubernetes does not roll back on its own: a stalled rollout only gets a ProgressDeadlineExceeded condition, after 600 seconds by default, and some higher-level tool has to act on it (Kubernetes docs). Helm stores each release revision by default as a Secret in the release namespace, containing the rendered manifests, so helm rollback re-applies an old revision (Helm docs). In GitOps, Argo CD or Flux sync the cluster to a Git commit, so rollback is a git revert. For VMs, an image builder such as Packer bakes a machine image, a launch template version points to it, and an instance refresh replaces the fleet. Registry garbage collection deletes blobs no manifest references, so untagged old releases can disappear unless retention rules keep them.

Layer 3The machine

containerd keeps pulled layers in a content store on the node's disk. Rolling back to an image the node ran yesterday is fast because its layers are still there, unless kubelet image garbage collection removed them when disk usage crossed its high threshold (85 percent by default, freeing down to 80) (kubelet config). A cold pull of a 500 MB image at 100 MB/s takes 5 seconds, plus single-threaded gzip decompression. When 300 nodes pull at once, the registry's egress becomes the bottleneck, which is why large fleets use pull-through mirrors or peer-to-peer distribution. The container root filesystem is an overlayfs: read-only image layers with a writable upper directory. Anything written there disappears when the container is replaced, which is immutability enforced by the filesystem. readOnlyRootFilesystem: true makes the rule explicit.

Why it's built this way

Hashing gives content a verifiable identity at almost no cost, so immutability can be checked rather than trusted. Commodity machines are cheaper to replace than to repair, so fleets are rebuilt instead of patched. Layers are cached on nodes, so recent releases roll back quickly. Data is shared and stateful, so replacing compute can never undo what the data layer saw.

Diagram

flowchart TD
  C[Commit] --> B[Build image]
  B --> D["Digest sha256:ab12"]
  D --> R[(Registry, immutable)]
  R --> H["Deploy history: r41 = ab12, r40 = 9f3c"]
  H --> P[Prod fleet runs ab12]
  P --> X{Alarm fires}
  X -->|rollback| Q["Redeploy r40 digest 9f3c"]
  X -->|flag off| F[Disable feature]
  Q --> P2[Fleet replaced from 9f3c]

Worked example

A service has 80 instances. With mutable deploys via SSH, a release takes 25 minutes and a rollback is a manual revert of packages and config, historically 40 to 60 minutes with mistakes. Mean time to restore across the last 6 incidents was 52 minutes.

After moving to immutable images and digest-pinned manifests, a rollback is a pipeline job that redeploys the previous digest. With images cached on nodes and a surge of 25 percent, replacement completes in about 4 minutes. Alarm to decision takes about 3 minutes with an automatic trigger on error rate. Restore time drops to about 7 minutes.

At 12,000 RPS with 4 percent of requests failing during a bad release, 52 minutes means about 1.5 million failed requests; 7 minutes means about 200,000. Keeping 50 releases of a 400 MB image costs 20 GB of registry, trivial by comparison.

The remaining risk is data: in one incident v2 wrote a cache entry shape v1 could not parse, so the rolled-back v1 crashed. The fix was the two-phase rule.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Immutable images, replace on changeNo drift, fast predictable rollbackImage build time, more registry storageDefault for services
Mutable config managementSmall changes are quickDrift, untested rollbackLegacy fleets, stateful snowflakes
Roll backKnown-good state restored quicklyDoes not undo data changesCause unclear, impact ongoing
Roll forwardKeeps data changes consistentFix under pressure, may add riskFix is trivial, or rollback is unsafe
Flag offSeconds, no deployOnly covers flagged codeFeature-level problems

Pitfalls in production

Leadership lens

Check yourself

  1. Why is a deploy pinned to api:2.4.1 weaker than one pinned to a digest?

    Show answer

    A tag is a mutable pointer and can be re-pushed to different content. A digest identifies exact bytes, so the rollback target is guaranteed to be what ran before.

  2. v2 publishes events with a new required field. Why can rolling back to v1 still fail?

    Show answer

    Events already in the queue use the new shape, and if v1 consumers reject unknown or changed fields they will fail. Readers must accept the new shape before writers emit it.

  3. When is rolling forward the better choice?

    Show answer

    When the release made data changes the old version cannot handle, or when the fix is small, understood, and faster than replacing the fleet.

  4. How does immutable infrastructure reduce drift?

    Show answer

    Instances are never changed after creation, so every instance from the same image is identical; changes only arrive by replacing instances from a new image.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →