Switching traffic between two environments
Open in the interactive app: practice, quiz, mock interview →In one minute
- Blue-green runs two complete production environments. One (say blue) serves all traffic; the new version is deployed to the idle one (green).
- After green passes checks against production dependencies, a single routing change moves all traffic from blue to green.
- Rollback is the same switch in reverse, which takes seconds as long as blue is still running and compatible.
- The cost is up to double compute during the release window, plus discipline around shared state such as databases and queues.
- Blue-green removes version mixing at the compute layer but not at the data layer.
How it works
Two fleets, one pointer
Each environment is a full copy of the stateless tier: instances, config, and a target group behind the load balancer. Martin Fowler's 2010 description notes a side benefit: the switch is the same mechanism a hot standby needs, so every release also rehearses the disaster-recovery cutover (Fowler). A pointer decides which color is live. The pointer can be a load balancer listener rule, a weighted DNS record, a service mesh route, a Kubernetes Service selector that matches color: green instead of color: blue, or a Gateway API HTTPRoute whose backend weights go from 1/0 to 0/1 (Gateway API). Some platforms now run the whole sequence for you: Amazon ECS has built-in blue/green with test-traffic routing, lifecycle hooks and a bake time during which blue keeps running after the shift (AWS docs).
The release sequence
- Deploy the new artifact to the idle color and scale it to production size.
- Run smoke tests against it through a private route or a test header, hitting real dependencies.
- Warm it: prime caches, open connection pools, let JIT compilation settle.
- Flip the pointer. New connections go to green.
- Drain blue: let in-flight requests finish, but keep blue running for a bake period.
- If metrics stay healthy through the bake period, scale blue down or keep it as the next idle target.
Where the switch lives matters
Load balancer and mesh switches take effect in seconds. DNS-based switches depend on resolver caching and client behaviour. Even with a 60-second TTL, some clients and corporate resolvers hold records for much longer, so a DNS flip is never truly atomic. Even AWS's own load balancer names resolve with a 60-second TTL, and clients may hold old addresses until it expires (AWS docs). Prefer switching at a layer you control.
Shared state
Both colors usually talk to the same database, cache, and queues. That means the schema must work for both versions during the window (Fowler's rule is to deploy schema changes separately from application upgrades, schema first), and background consumers in the idle color must not start processing messages before it is live, or both versions will consume from the same queue. A common pattern is to gate workers on an "active color" flag read at runtime.
When a release changes a data format, split it into two deployments so the flip back stays safe. Amazon's version: first deploy code that can read both the old and new format but still writes the old one, let it bake, then deploy the change that starts writing the new format. At every point the version you might roll back to can read everything on disk (AWS Builders' Library).
Long-lived connections
The flip affects new connections only; WebSockets and gRPC streams stay on blue until they close or are told to reconnect.
Down the stack
Layer 1The contract
After the flip, every new request or connection reaches exactly one color, the live one. The flip is a single configuration change, so rollback is the same operation with the old value. What it does not promise: existing connections are not moved, shared state is not versioned, and background work in the idle color is not paused unless you build that.
Layer 2The mechanism
On AWS, an Application Load Balancer listener rule forwards to target groups with weights (0 to 999), and one ModifyListener call moves weight from blue's group to green's (AWS docs). Weighted routing is not health-aware between groups: if green's group is empty or all unhealthy, the balancer does not fall back to blue. Target group stickiness, an encrypted AWSALBTG cookie, keeps a client on the color it first reached. Deregistered targets enter a deregistration delay (300 seconds by default, configurable from 0 to 3,600) during which in-flight requests complete (AWS docs). In Kubernetes, changing a Service's selector from color: blue to color: green makes the EndpointSlice controller rewrite the endpoint list. In Envoy-based meshes, the control plane pushes a new route configuration over xDS with weighted_clusters (Envoy docs), and every sidecar applies it independently, usually within about a second. DNS-based switching changes weighted records and waits for TTLs to expire.
Layer 3The machine
An ALB terminates HTTP and chooses a target per request, so a client's keep-alive connection to the load balancer survives the flip and its next request goes to green. A Network Load Balancer works at layer 4 and pins each TCP flow to a target until it closes, so long connections stay on blue. DNS is worse: operating system resolvers, language runtimes (the JVM's networkaddress.cache.ttl), and connection pools all cache addresses independently of the record's TTL. Green also starts cold. If 60 green instances each open 20 database connections, PostgreSQL forks 1,200 new backend processes at roughly 5 to 10 MB each, on top of blue's 1,200, which can hit max_connections or exhaust memory during the overlap.
Why it's built this way
Only the router sees every request, so the switch lives there. Stateless compute is cheap to duplicate but databases are not, so both colors share one data layer and compatibility becomes the developer's job. Connections are long-lived, so draining is needed. DNS caching is outside your control, so serious blue-green setups avoid DNS as the switch.
Diagram
flowchart LR
U[Users] --> R{"Router (live = blue)"}
R -->|100%| B["Blue v1"]
R -.->|"0%, test header only"| G["Green v2"]
B --> DB[(Shared database)]
G --> DB
Q[Smoke tests] --> G
F[Flip pointer] --> RWorked example
A search API runs 60 instances at 0.10 USD per hour each, 6 USD per hour. Releases happen twice a week.
If green is created fresh for each release and lives for 1 hour of testing plus a 2-hour bake, the extra cost is 60 x 3 x 0.10 = 18 USD per release, about 150 USD a month. Keeping a permanent second fleet would cost 6 x 730 = 4,380 USD a month, so ephemeral green is the obvious choice here.
Now consider failure. A bad release raises the error rate from 0.1 percent to 5 percent at 20,000 RPS. With a rolling deploy of 60 instances in batches of 6, rollback takes about 10 minutes: roughly 10 x 60 x 20,000 x 0.049 = 588,000 extra failed requests. With blue-green, detection takes 2 minutes and the flip takes 10 seconds: about 2.2 x 60 x 20,000 x 0.049 = 129,000. The 150 USD a month buys a large cut in blast radius.
The catch: if the release also ran a destructive migration, such as dropping a column v1 reads, the flip back fails just as badly. Blue-green only gives fast rollback when the data layer stays compatible.
In the wild
- Slack, 2021: a DNS-level change whose rollback was capped by caches. Slack enabled DNSSEC on slack.com, and a Route 53 bug in signed answers for wildcard names made some validating resolvers cache "no such records". Rolling back made it worse, because resolvers had cached the DS record for up to 24 hours and returned SERVFAIL until it expired. Most users recovered within hours, but a long tail on smaller resolvers waited out the 24-hour TTL. It is the same reason a DNS flip is a weak blue-green switch: the old answer lives in caches you do not control (Slack).
- Stripe, 2019: a rollback that was not compatible with the present. A latent failover bug in a database version shipped months earlier left one shard unable to elect a primary, and most API requests failed for about 27 minutes. Responders then rolled back the election protocol, but the old version clashed with a configuration change made since, starved CPU on every shard and caused a second outage of about 93 minutes. Going back is only fast and safe if the old version still fits the current config and data (Stripe).
Trade-offs
| Option | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Blue-green | Seconds to roll back, full pre-release test on prod infra | Double capacity during window, all users switch at once | Rollback speed is critical, deploy is risky |
| Rolling | No duplicate fleet | Slow rollback, mixed versions during rollout | Routine releases of stateless services |
| Canary | Limits blast radius to a small fraction | Needs good metrics and routing control, slower | High traffic with clear health signals |
| Blue-green plus canary step | Fast rollback and gradual exposure | Most moving parts | Critical services with mature tooling |
Pitfalls in production
- Background workers in the idle color consume from production queues. Gate consumers on the live-color flag.
- Green is tested at zero load, then gets full traffic cold. Warm caches and pools, or shift through 10 and 50 percent first.
- DNS flips leave stragglers on blue for minutes or hours. Switch at the load balancer or mesh instead.
- Schema changes are not covered by the flip. Use expand-and-contract migrations so both versions work against the same schema.
- Blue gets scaled down right after the flip, so "instant rollback" is no longer available. Keep it warm through a defined bake period.
- The flip points at a group nobody checked. On an ALB, weight 100 to an empty or unhealthy green group fails requests rather than falling back to blue. Gate the flip on green's healthy-target count, not just on the deploy finishing.
Leadership lens
- Decide how long the previous color stays warm and who can trigger the flip back without approval.
- Ask what stateful components are shared and how each one behaves when two versions touch it.
- Weigh the cost of a second fleet against the measured cost of slow rollback for this service.
Check yourself
Why does blue-green give near-instant rollback for code but not for a migration that drops a column?
Show answer
The flip only changes which compute fleet gets traffic. Both fleets share the database, so once the column is gone v1 fails as well, and the flip back does not restore it.
Why is a DNS-based switch weaker than switching at the load balancer?
Show answer
Resolvers and clients cache records, sometimes beyond the TTL, so traffic moves gradually and unpredictably. A load balancer rule change is applied centrally within seconds.
What goes wrong if green's queue consumers start as soon as green boots?
Show answer
Both versions process messages from the same queue before release, so untested v2 code touches production data and duplicates or conflicts are possible.
When is keeping a permanent second fleet worth it?
Show answer
When releases are frequent enough that spin-up time dominates, or when the warm standby doubles as disaster recovery capacity.
Sources and further study
Primary
- Slack: what happened during Slack's DNSSEC rollout: why DNS changes cannot be rolled back faster than resolver caches expire.
- Stripe: root cause analysis, July 2019: a rollback that collided with newer configuration.
Systems docs
- AWS: listener rule action types: weighted target groups, their limits and target group stickiness.
- Amazon ECS blue/green deployments: managed blue/green with lifecycle hooks and bake time.
- Kubernetes Gateway API: traffic splitting: weight-based switching between two backends.
Books
- Release It! (2nd edition), Michael T. Nygard, chapter 13 "Design for Deployment": the fallacy of planned downtime and deploying in phases.
- Building Microservices (2nd edition), Sam Newman, chapter 8 "Deployment": deployment options and principles for independently deployable services.
Other curricula
- Azure Well-Architected: safe deployment practices: treats blue-green as progressive, shifting traffic to green in growing waves rather than a single flip.
- donnemartin/system-design-primer: has no section on blue-green; its nearest topic is active-passive fail-over, which uses a similar standby but for failures rather than releases.
Watch
- An in-depth introduction to Blue Green Deployments (Arpit Bhayani): walks through the switch, rollback, and the database caveat.
- Top 5 Most-Used Deployment Strategies (ByteByteGo): compares blue-green with rolling, canary, and others.
Free resources
- BlueGreenDeployment (Martin Fowler): the original short description of the technique and its database concerns.
- Blue/Green Deployments on AWS (whitepaper): concrete switch mechanisms at the DNS, load balancer, and autoscaling levels.
- Ensuring rollback safety during deployments (AWS Builders' Library): why rollback needs compatible data formats, which blue-green alone does not give you.