Engineering Atlas › System Design Fundamentals

Testing a release on a small slice of users

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Routing a slice

The router splits traffic by weight: 1 percent to canary, 99 percent to stable. Weight can be set in a service mesh, an ingress controller, or a load balancer with weighted target groups. Without a smart router you can approximate by replica count (1 canary pod among 50), but then the split is coarse and changes with autoscaling. Sticky assignment by user or session keeps one user from flipping between versions on each request.

Baseline, not just stable

Comparing the canary to the whole stable fleet is unfair. Stable pods have warm caches and have run for days; the canary just started. The better design deploys a fresh baseline of the old version at the same size and time as the canary, so the only difference is the code. Netflix popularised this "baseline plus canary" pattern; when it and Google open-sourced Kayenta in 2018, the typical setup was three new instances each for baseline and canary, alongside the existing production instances (Google Cloud). Google's later guidance is blunt: don't compare the canary with production, match the baseline on deploy time, instance count and traffic (Google Cloud).

Analysis

At each step the analyser pulls metrics for both groups: error rate, p99 latency, saturation, and business signals such as checkouts per minute. It applies rules like "canary error rate may not exceed baseline by more than 1.5x, with an absolute floor" and requires a minimum sample before deciding; Google suggests at least 50 data points per metric (Google Cloud). Results are pass, fail, or not enough data yet. Choose few metrics, and ones users would feel: Google's SRE Workbook suggests stack-ranking candidates and keeping perhaps a dozen at most, and warns that raw CPU usage is a noisy canary signal because other processes on the host move it (SRE Workbook, ch. 16).

Progression

A typical schedule is 1, 5, 25, 50, 100 percent with a bake time at each step, often 10 to 30 minutes. A fail at any step routes all traffic back to stable and scales the canary down. Total rollout for a critical service can be an hour or more, which is fine if it is automated. Run one canary at a time: overlapping canaries contaminate each other's signal (SRE Workbook, ch. 16).

What canaries cannot see

A canary on 1 percent of traffic may never execute a rare code path, a nightly batch job, or a migration. If v2 writes subtly corrupt records, error rates stay flat while damage accumulates. Pair canaries with data validation and keep risky behaviour behind flags.

Down the stack

Layer 1The contract

The router promises that a request reaches the canary with probability close to its weight, and with sticky routing that a given user stays on one side. The analyser promises a verdict per step from predefined metrics and thresholds, with a known tolerance for false alarms. The controller promises that a fail verdict sets canary weight to zero within seconds. Nothing promises that a bug will show up in the metrics you chose.

Layer 2The mechanism

In Envoy, a route uses weighted_clusters, for example stable at 99 and canary at 1. For each HTTP request the proxy draws a random number and picks a cluster by cumulative weight; for raw TCP it picks once per connection. Header or cookie matches can override the draw for sticky or internal traffic, and use_hash_policy makes the draw come from the route's hash policy (for example a user ID header), so a user lands on the same side at every proxy hop. The old total_weight field is deprecated; weights are simply summed (Envoy docs). Argo Rollouts drives the steps: a rollout lists setWeight and pause steps, and an AnalysisTemplate runs Prometheus queries such as the 5xx rate for pods labelled version=canary, with a successCondition, an interval, and a failureLimit. A run that is neither success nor failure is Inconclusive, which pauses the rollout for a human (Argo Rollouts). Netflix's Kayenta compares each metric's distribution in canary and baseline with a Mann-Whitney U test, and marks a metric high or low only when the 98% confidence interval lies wholly outside a tolerance band and the effect exceeds the allowed change. Each group's score is the share of passing metrics, groups combine into a weighted 0 to 100 score compared against pass and marginal thresholds, and a failing critical metric or 50% missing data forces the score to 0 (Spinnaker docs).

Layer 3The machine

Detection latency is set by the metrics pipeline. Prometheus is commonly configured to scrape every 15 to 30 seconds (the built-in default is one minute (Prometheus docs)), and a rate() over a 2-minute window needs at least two samples, so a hard failure takes 1 to 2 minutes to appear in a query. Latency percentiles come from histogram buckets, so histogram_quantile interpolates within a bucket: with buckets at 250 and 500 ms, a p99 of "310 ms" could be anything in that range. A canary of two pods lands on two nodes; if one shares a host with a noisy neighbour, CPU steal or a long GC pause on that JVM shows as a latency regression that has nothing to do with the code. Counters reset when a pod restarts, which is why queries use rate() instead of raw totals.

Why it's built this way

Failures are rare events, so meaningful comparison needs volume, and volume needs time or weight. Metrics are aggregated in windows, so every step needs a bake period longer than the window. Hosts are noisy, so the comparison group must be the same size and age as the canary. Per-request random routing is cheap but not sticky, so user consistency requires hashing on a key.

Diagram

stateDiagram-v2
  [*] --> Deploy
  Deploy --> Step1: 1 percent traffic
  Step1 --> Analyse
  Analyse --> Step1: not enough data
  Analyse --> Rollback: fail
  Analyse --> NextStep: pass
  NextStep --> Analyse: raise weight
  NextStep --> Promote: weight was 100 percent
  Rollback --> [*]
  Promote --> [*]

Worked example

A feed service serves 50,000 RPS with a baseline error rate of 0.2 percent. A team wants to detect a regression to 0.4 percent.

At 1 percent canary weight the canary sees 500 RPS, 30,000 requests a minute. Expected errors at 0.2 percent are 60 per minute, at 0.4 percent 120. Poisson noise on 60 is about plus or minus 8, so a doubling is obvious within a minute or two. A 10 percent regression, from 0.2 to 0.22, is not: the difference of 6 errors a minute sits inside the noise, and you would need many minutes or a larger slice.

For a smaller service at 200 RPS, 1 percent is 2 RPS, 120 requests a minute, and at 0.2 percent errors that is a quarter of an error per minute. One error proves nothing. Such a service needs a larger canary share, say 10 to 20 percent, longer bakes, or a synthetic load generator. Canary sizing is a statistics problem, and it scales with traffic.

Blast radius at 50,000 RPS: if v2 fails 100 percent of requests and the analyser catches it in 2 minutes at 1 percent, about 60,000 requests fail. Without a canary, a full rolling deploy would fail far more before anyone noticed.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Weighted canary with automated analysisSmall blast radius, objective decisionsNeeds mesh or smart LB and good metricsHigh-traffic critical services
Replica-count canaryWorks with plain load balancingCoarse weights, drifts with autoscalingNo traffic-splitting layer
Cohort canary (internal users, one region)Easy to reason about, low exposureCohort may not represent real trafficEarly dogfooding, regional rollouts
Blue-greenInstant rollbackEveryone switches at onceLow traffic where canary stats are weak

Pitfalls in production

Leadership lens

Check yourself

  1. Why compare the canary to a freshly deployed baseline instead of the existing stable fleet?

    Show answer

    The stable fleet differs in uptime, cache warmth, and host placement, so differences appear even with identical code. A fresh baseline isolates the code change.

  2. Why can client retries hide a bad canary?

    Show answer

    A request that fails on the canary is retried, often lands on stable, and succeeds, so user-facing error rate stays flat while the canary fails many first attempts.

  3. A service gets 100 RPS. Why might a 1 percent canary be pointless?

    Show answer

    1 RPS gives too few events per minute to distinguish a regression from noise, so decisions are random. It needs a larger slice, longer bake, or synthetic traffic.

  4. What class of bug does a canary reliably miss?

    Show answer

    Bugs that do not move monitored metrics quickly, such as silent data corruption, slow memory leaks, or rarely exercised paths.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →