Chaos Engineering and Game Days

In one minute

• Chaos engineering is running experiments that inject failure on purpose, to find weaknesses before they cause unplanned incidents. Game days are scheduled exercises where a team practises responding to such a failure.
• An experiment has a steady-state hypothesis ("checkout success stays above 99.5 percent"), a fault ("terminate 1 of 3 database replicas"), a small blast radius, abort conditions, and someone ready to stop it.
• Start small and safe: staging first, then a small share of production traffic, during working hours, with the team watching. Expand only when the system and the team have earned it.
• The output is not "it survived". It is what you learned: missing alerts, wrong timeouts, runbooks that do not work, dependencies nobody knew about.
• Game days test people and process as much as software: who gets paged, whether dashboards answer the question, whether the runbook's commands still exist.

How it works

Why break things on purpose

Systems have failure modes nobody has seen yet: a timeout that is too long, a retry that amplifies load, a cache that becomes a single point of failure, a fallback that was never exercised. You will discover them either during an incident, at 3 a.m., with customers affected, or during an experiment, at 2 p.m., with a stop button and the team watching. Netflix popularised the idea with Chaos Monkey, which randomly terminates production instances so engineers build services that tolerate it; it is configured per application through Spinnaker (Chaos Monkey docs). Google runs annual DiRT (Disaster Recovery Testing) drills that push production to its limits and cause real outages, on the principle that hope is not a strategy (SRE book, ch. 33). The same chapter notes that other industries rely on live drills too, and that "what if" thought exercises alone are not considered enough.

The experiment loop

The Principles of Chaos Engineering describe the method:

1. Define steady state as measurable output: request success rate, latency percentiles, orders per minute. Not "CPU is fine", but what users experience. The Principles say to focus on the system's measurable output rather than internal attributes.
2. Hypothesise that steady state continues in both a control group and an experimental group.
3. Introduce real-world faults: instance termination, network latency or packet loss, a dependency returning errors, a disk filling up, clock skew, a region becoming unreachable.
4. Look for a difference between control and experiment. A difference is a finding.
5. Minimise blast radius: affect a small fraction of traffic or hosts, have automatic abort conditions, and be able to stop instantly.

The Principles add three "advanced" practices on top. Run experiments in production, because systems behave differently under real traffic and the Principles say chaos "strongly prefers" production traffic. Automate experiments to run continuously, so a fix cannot silently rot. And vary the events you inject, as any event that can disrupt steady state is a candidate. This is why the advice below to start in staging is a way to build confidence and tooling, not the end state.

Game days

A game day is a planned session, often half a day, where a team runs one or more scenarios. A facilitator injects the failure (or announces a simulated one), the on-call responds as they would for real, and observers note what happens: time to detect, time to diagnose, confusion, missing access, wrong documentation. Afterwards, a short review turns observations into action items, exactly like a postmortem. Scenarios can be technical ("the primary database fails over") or organisational ("the only engineer who knows the billing system is unreachable").

Safety first

Experiments in production need guardrails: a clear owner, a time window, stakeholders informed, abort conditions tied to steady-state metrics (for example, stop if checkout success drops below 99 percent for 1 minute), automatic rollback of the fault, and a small starting blast radius. Never run an experiment you cannot stop, and do not run chaos experiments during an ongoing incident or a big launch.

Game days for the AI era

Agents and LLM features add scenarios worth practising: the model provider is down or slow (does the feature degrade gracefully?), an agent is prompt-injected (do the guardrails hold?), an agent runs a destructive command (does the kill switch work, and how long does restore take?). These are cheap to simulate and expensive to discover live.

The same shape everywhere

Fire drills, flight simulators and military exercises follow the same logic: rehearse failure in controlled conditions, measure the response, fix what failed. The goal is that the real event feels familiar.

Down the stack

Diagram

flowchart TD
  H["Steady-state hypothesis"] --> P["Plan: fault, blast radius, abort conditions"]
  P --> R["Run on small slice with team watching"]
  R --> M{"Abort condition hit?"}
  M -->|yes| S["Stop fault, restore"]
  M -->|no| C{"Steady state held?"}
  C -->|yes| E["Expand blast radius next time"]
  C -->|no| F["Finding: fix and retest"]
  S --> F

Worked example

A team runs its first game day on the order service: 3 API pods, a Postgres primary with one replica, and Redis for sessions. Hypothesis: if one API pod is terminated, order success stays above 99.5 percent and p99 latency below 800 ms. Abort if success drops below 98 percent for 60 seconds.

Experiment 1, terminate one pod: success stays at 99.9 percent; p99 rises briefly to 650 ms. Hypothesis holds. Experiment 2, inject 300 ms latency on Redis for 25 percent of connections: success falls to 97 percent within 40 seconds and the abort fires. Finding: the session lookup has a 250 ms timeout with three retries and no fallback, so slow Redis turned into failed orders. Experiment 3, fail over Postgres: the failover completes in 35 seconds, but the on-call engineer cannot find the runbook (it moved to a new wiki), and the alert for replica lag points to a dashboard that no longer exists.

Action items: a single retry with a 400 ms budget and a fallback to re-authenticate via the token for session lookups; fix runbook links and add them to alerts; repair the dashboard. A follow-up game day a month later reruns experiment 2 and holds at 99.6 percent.

In the wild

• AWS, 2017: a restart nobody had rehearsed. An S3 capacity-removal command with a mistyped input took out the index and placement subsystems in us-east-1. AWS wrote that S3 is built to survive removal of capacity, but that it had not completely restarted those two subsystems in its larger regions for many years, and that the restart and its safety checks took longer than expected because of growth. S3 was unavailable for hours, and EC2 launches, EBS restores and Lambda were affected. A game day that restarts a whole subsystem is the experiment that would have measured this (AWS).
• AWS Lambda, 2023: a size nobody had run. On 13 June, normal growth pushed one Lambda Frontend cell past a capacity threshold it had never reached, which triggered a latent defect: execution environments were allocated but never used, so invocations in that cell saw errors and latency. Services such as STS, console sign-in and EventBridge were degraded for hours. AWS fixed the bug and planned to keep cells to well-tested sizes. Scale is a fault you can inject on purpose (AWS).
• Google, ongoing: DiRT as a standing programme. Instead of waiting for such surprises, Google SREs run annual drills that cause real outages, to confirm systems react as expected and to find weaknesses before an uncontrolled outage does (SRE book, ch. 33).

Trade-offs

Pitfalls in production

• Experiments without abort conditions. Fix: bind stop conditions to steady-state metrics, automatically.
• Starting with a big blast radius. Fix: one host, one percent, one zone, then expand.
• Treating "nothing broke" as the goal. Fix: write down what was learned, including gaps in monitoring and runbooks.
• Findings never fixed. Fix: track them in the backlog with owners, like postmortem items.
• Surprising people. Fix: announce game days to stakeholders, and keep a clear stop authority.

Leadership lens

• Start with a quarterly game day per critical service; the first one almost always finds something important.
• Make steady-state metrics user-facing and the same ones used for SLOs and alerts.
• Include people scenarios: the expert is unavailable, the usual tool is down, it is a new on-call's first shift.
• Add agent and LLM failure scenarios to the rotation as those features grow.

Check yourself

1. Why define steady state as user-facing output rather than CPU or memory?

   Show answer

   User-facing metrics show whether the system still does its job; internal metrics may change without affecting users, or stay normal while users suffer.

2. What is the purpose of a control group in a chaos experiment?

   Show answer

   To compare against the experimental group at the same time, so changes caused by the fault are separated from normal variation in traffic.

3. Why can 1 percent injected packet loss cause much larger latency effects?

   Show answer

   Lost segments wait for retransmission timeouts, and congestion control slows the connection, so the latency tail grows far beyond the average delay.

4. What should you do if an experiment's abort condition fires?

   Show answer

   Stop the fault immediately, confirm recovery, and treat it as a finding to investigate and fix, not a failed experiment.

Sources and further study

Primary

• AWS: summary of the S3 service disruption (2017): a primary account of a recovery path that had not been exercised for years.

Systems docs

• AWS Fault Injection Service: stop conditions: alarms that halt an experiment automatically.
• Envoy fault injection filter: injected delays and aborts at the proxy.

Books

• Site Reliability Engineering, Beyer et al.: chapter 33 (Lessons Learned from Other Industries), on DiRT and live drills in aviation, the nuclear Navy and lifeguarding.
• Release It! (2nd ed.), Nygard: chapter 17 (Chaos Engineering).
• Building Secure and Reliable Systems, Adkins et al.: chapter 16 (Disaster Planning), on rehearsing responses before an incident.

Other curricula

• Google SRE book: Lessons Learned from Other Industries: frames drills as the only convincing test, since thought exercises alone are not enough.

Free resources

• Principles of Chaos Engineering: the definition and the experiment method.
• Chaos Monkey (Netflix): the original instance-termination tool and how it is used.
• AWS Fault Injection Service: managed fault injection with stop conditions.
• How to run a GameDay (Gremlin): planning, roles and running the session.
• Game days (AWS Well-Architected): game days as part of operational excellence.