Engineering Atlas › System Design Fundamentals

Choosing how to roll out a release

16 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The menu

Recreate. Stop every old instance, then start the new ones. No extra capacity and no version mixing, but downtime equals startup time and every user meets the new version at once. Kubernetes calls it strategy: Recreate; it fits batch workers and software that cannot run two versions against the same state.

Rolling. Replace instances in batches: start a few new ones, wait for readiness, remove a few old ones, repeat. Kubernetes' default RollingUpdate uses maxSurge and maxUnavailable (25% each by default) (Kubernetes docs). It needs little spare capacity, but each batch meets real traffic uncompared, and rollback is another full rolling deploy. The rolling deployments lesson covers the mechanics.

Blue-green. Stand up a complete second fleet with the new version, test it against production dependencies, then flip all traffic in one routing change. Rollback is the same flip, in seconds, as long as the old fleet is still running. The cost is double capacity during the release, and the flip still exposes every user at once. See blue-green deployments.

Canary. Route a small percentage of traffic, say 1%, to the new version, compare its error rate and latency against a baseline, and widen in steps (1%, 10%, 50%, 100%) only while it stays healthy. Blast radius is tiny and rollback is a weight change, but the rollout is slow and the analysis must be statistically sound. See canary releases.

Shadow (dark launch). Mirror real requests to the new version and discard its responses, comparing them with the live version's output. Users are never affected, so it suits rewrites, new databases and new models. It cannot test side effects unless writes are stubbed, and it doubles downstream load.

Feature flags. Ship the code dark and turn behaviour on per user, per percentage or per tenant at runtime. Flags separate deploy from release, so the deploy itself can be boring. See feature flags.

Combining them: waves and cells

At scale, the question shifts from "which strategy" to "in what order do we expose the fleet". Amazon's published practice deploys to one box first, then one availability zone, then one region, then progressively larger waves of regions, with automated rollback on alarm and a bake time at each stage that gives slow failures (memory leaks, daily batch jobs, certificate expiry paths) a chance to show up. The numbers are deliberately slow: the one box serves at most about 10% of a region's or zone's requests and bakes for at least an hour, the first region bakes for at least 12 hours, later waves for 2 to 4 hours, and bake time also waits for a minimum count of requests. With default settings a change takes about four or five business days to reach every region, and deployments skip nights, weekends and holidays when on-call engineers are slower to respond (AWS Builders' Library). Microsoft's guidance is similar: bake times measured in hours and days, growing with each ring (Azure Well-Architected). Inside each wave, a rolling or blue-green mechanism does the actual replacement. Cell-based architectures make this cleaner: each cell is an independent copy of the stack serving a fixed slice of customers, so a wave is a set of cells and blast radius is literally the share of customers in those cells.

Problem shape: partial failure and blast radius

Google's SRE team found that roughly 70% of outages come from changes to a live system (Google SRE book, ch. 1), so the rollout is where most risk lives. Choosing a strategy is a way of bounding partial failure. The same reasoning applies to rolling out a new prompt or model in an AI product: shadow it on logged traffic, canary it to 1% of conversations, compare quality metrics against the incumbent, and keep the old model loaded so rollback is a routing change rather than a redeploy.

Down the stack

Layer 1The contract

A deployment strategy's contract can be written as four numbers per release. Exposure: the fraction of traffic served by the new version before the first health gate is evaluated (recreate and blue-green: 100%; rolling: one batch; canary: the first step). Detection-to-rollback time: from an alarm firing to all traffic on the old version (canary and blue-green: one routing update; rolling and recreate: a redeploy). Capacity overhead: peak instances relative to steady state (recreate 1x, rolling 1x plus surge, blue-green and mesh-weighted canary up to 2x). Mixed-version window: how long both versions serve at once (zero for recreate; the whole rollout for rolling and canary; the drain period for blue-green). A good release process states these numbers up front.

Layer 2The mechanism

Where the switch lives decides how fast it moves. DNS weights and records are cached by resolvers and clients for at least the TTL, and some clients ignore TTLs entirely, so a DNS cutover drains over minutes to hours. Load balancer changes, such as re-weighting AWS ALB target groups, apply to new connections within seconds. Service mesh routing (Istio VirtualService weights, Envoy weighted_clusters) splits per request, which allows 1% canaries. The Kubernetes Gateway API gives the same split in a vendor-neutral form: an HTTPRoute lists backendRefs with proportional weights such as 90 and 10 (Gateway API). Envoy's request_mirror_policies implement shadow traffic: mirroring is fire-and-forget, Envoy does not wait for the shadow, and it appends -shadow to the Host header so the shadow service can tell copies apart (Envoy docs). Managed platforms now ship these strategies too: Amazon ECS added built-in blue/green and, in October 2025, built-in linear and canary deployments with bake times and lifecycle hooks (AWS). Orchestrators encode the strategy declaratively: Kubernetes Deployments support Recreate and RollingUpdate, and controllers such as Argo Rollouts or Flagger add blueGreen and canary strategies with analysis steps that query Prometheus and abort automatically. In-process, a flag SDK evaluates rules from a locally cached ruleset.

A health gate is a function from metrics to a verdict: wait, proceed or roll back. It compares the new version against a baseline running the old version at the same time (not against last week), and it needs a minimum request count before it can judge anything. Google's canary guidance puts that floor at about 50 data points per metric for the statistical tests to mean anything, which at one-minute resolution can stretch an analysis to hours (Google Cloud).

Layer 3The machine

Deploy speed is dominated by startup, not routing. Pulling a 1 GB container image at 100 MB/s takes 10 seconds per node unless it is cached; a JVM service may need 30 to 90 seconds of JIT warm-up before its latency matches the old fleet, which can fail a naive canary on p99 latency even when the code is fine. Empty in-process caches send a burst of misses to the database for the first minutes, so a blue-green flip of a large fleet can briefly double database load. Removing old instances needs connection draining: the load balancer stops new requests, in-flight requests finish, HTTP/2 servers send GOAWAY, and only then does the process exit, often after a preStop sleep of a few seconds so endpoint removal propagates to every proxy. Long-lived connections (WebSockets, gRPC streams) do not move on a routing change at all until they reconnect, so "100% switched" at the balancer can still mean thousands of clients on the old version.

Why it's built this way

  1. Tests cannot reproduce production traffic, data and dependencies, so some failures only appear when real users hit the new version.
  2. If failures will reach users anyway, the controllable variable is how many users and for how long, which is exposure times detection time.
  3. Fast rollback needs the old version still running, which costs capacity; slow rollback is cheap but extends every incident, so strategies trade money for recovery time.
  4. Some failures are slow (leaks, daily jobs, rare paths), so gates add bake time, and wide fleets deploy in waves so a slow failure stays contained to the early waves.
  5. Running two versions at once is unavoidable for zero-downtime strategies, so compatibility between versions becomes a requirement of every change.

Diagram

flowchart TD
  Q1{"Can two versions run at once?"} -->|"no"| RC["Recreate (accept downtime)"]
  Q1 -->|"yes"| Q2{"Need rollback in seconds?"}
  Q2 -->|"no"| RO["Rolling update"]
  Q2 -->|"yes"| Q3{"Can afford 2x capacity briefly?"}
  Q3 -->|"no"| RO
  Q3 -->|"yes"| Q4{"Need small blast radius?"}
  Q4 -->|"no"| BG["Blue-green flip"]
  Q4 -->|"yes"| CA["Canary with health gates"]
  CA --> W["Wider waves: zone, region, all"]

Worked example

A checkout service runs 40 instances across three zones and serves 6,000 requests per second. New instances take 90 seconds to pull, start and warm up. The team deploys twice a day and wants a bad build to cost fewer than 1,000 failed checkouts.

Suppose a bad build fails 20% of requests and alarms fire 3 minutes after the bad version starts serving.

Recreate. Every deploy causes 90 seconds of downtime: 540,000 failed requests, before the build is even known to be bad. Ruled out.

Rolling, batches of 4. The first batch serves 10% of traffic, 600 requests per second, 120 of which fail. In the 3 minutes to detection that is 21,600 failures, and the rollout has meanwhile moved on to more batches unless each batch bakes. Rollback is another rolling deploy, about 10 batches of 90 seconds each, 15 minutes. Far above budget.

Blue-green. 40 extra instances for the release window, about 30 minutes per deploy including bake, so 1 hour a day of double capacity. Detection still happens with 100% exposure: 1,200 failures per second for 3 minutes is 216,000. Rollback takes 5 seconds, but the exposure already blew the budget.

Canary at 1%, then 10%, 50%, 100%, with 10-minute bakes. At 1%, 60 requests per second hit the new version and 12 fail. A gate with a minimum of 3,000 canary requests can judge after 50 seconds; with a 2-minute evaluation window, about 1,440 failures in the worst case. Tightening the first gate to abort when the canary's error rate exceeds twice the baseline after 1,500 requests (25 seconds) brings it under 400. Rollback is a weight change. The cost is a rollout of about 45 minutes and a few spare instances during the early steps.

The team picks canary with automated gates for code, keeps blue-green for runtime upgrades, and puts risky features behind flags.

In the wild

Trade-offs

StrategyStrengthsWeaknessesChoose when
RecreateSimplest, no mixing, no extra capacityDowntime, full exposureBatch workers, dev, incompatible versions
RollingLittle extra capacity, built into orchestratorsSlow rollback, uncompared exposure per batchStateless services with good readiness checks
Blue-greenRollback in seconds, test the full new fleet first2x capacity, full exposure at the flipInfrastructure changes, few releases a day
CanaryTiny blast radius, metric-driven progressionSlow, needs traffic volume and good metricsHigh-traffic user-facing services
ShadowZero user impact, real load and inputsNo side effects tested, doubles downstream loadRewrites, new datastores, new models
Feature flagsRelease without deploy, per-user controlFlag debt, combinations to testProduct features, kill switches

Pitfalls in production

Leadership lens

Check yourself

  1. Why does blue-green give fast rollback but not a small blast radius?

    Show answer

    The old fleet stays running, so rollback is one routing change. But the flip moves all traffic at once, so every user meets the new version before any production signal exists.

  2. A canary fails its latency gate on every release, and the problem disappears after ten minutes. What is likely happening, and what is the fix?

    Show answer

    Fresh instances are cold (JIT, caches, connection pools) while the baseline is warm. Warm the canary before measuring, exclude a warm-up window, or compare against freshly started baseline instances.

  3. Why are database schema changes a constraint on every zero-downtime strategy?

    Show answer

    Rolling, canary and blue-green all run old and new code against the same database at some point, so each schema state must work with both versions, which forces expand-and-contract migrations.

  4. When is shadow traffic better than a canary?

    Show answer

    When the change has no user-visible side effects to test, but a failure would be costly, such as a new storage engine or a rewritten pricing service. Shadowing compares outputs on real traffic without exposing anyone.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →