Root Cause Analysis: 5 Whys, Fishbone and Causal Graphs
Open in the interactive app: practice, quiz, mock interview →In one minute
- Root cause analysis asks why an incident happened deeply enough that the fix prevents a whole class of incidents, not just this one.
- In complex systems there is rarely a single root cause. Incidents come from a trigger meeting several contributing factors: a latent bug, a missing alert, a risky default, a rushed process. Model them as a causal graph, not a chain.
- The 5 whys is a useful habit but a weak method on its own: it follows one chain and tends to stop at a person ("the engineer ran the wrong command"). Ask "how" and "what made this possible" as well.
- A fishbone (Ishikawa) diagram helps you look across categories (code, config, process, people and tools, environment, dependencies) so you do not miss branches.
- Good action items attack the graph at several levels: remove the trigger, add detection, limit blast radius and fix the conditions that made the error likely.
How it works
Trigger, causes, contributing factors
Separate three things. The trigger is the event that started it: a deploy, a traffic spike, a certificate expiring. Causes are the mechanisms that turned the trigger into impact: a missing timeout, a retry storm, a shared database. Contributing factors are conditions that made it worse or more likely: no canary, an alert that went to an unmonitored channel, a runbook that was out of date, an on-call engineer on their first shift. Removing the trigger stops a repeat of this exact incident; fixing causes and contributing factors prevents its relatives.
5 whys, used well
Start from the impact and ask why until you reach something you can change systemically. "Checkout failed for 40 minutes. Why? The payment service ran out of database connections. Why? A new query held connections for 30 seconds. Why? It scanned a large table without an index. Why? The migration adding the index failed silently. Why? Migration failures are not checked by the deploy pipeline." The last answer leads to a fix that prevents many future incidents. Two common mistakes: stopping at a person ("why? because Sam forgot") and following only one branch. When an answer is "someone made a mistake", ask what made the mistake easy and the consequences large. John Allspaw makes the same point about the question itself: "why" invites answers about individuals, while "how" draws out the conditions and how decisions made sense at the time (The Infinite Hows).
Fishbone
Draw the impact at the head of a fish and categories along the spine: code, configuration, data, infrastructure, dependencies, process, people and tools, monitoring. For each category ask what contributed. It forces breadth, which is exactly what 5 whys lacks, and it works well as a group exercise in a postmortem meeting.
Causal graphs
Combine both into a directed graph: nodes are facts, edges mean "contributed to". Several roots can lead to the impact, and some nodes have several parents. Reading the graph shows where one fix removes many paths. Safety science has gone further in this direction: Richard Cook's "How Complex Systems Fail" argues that catastrophe requires multiple failures (point 3), that post-accident attribution to a single root cause is fundamentally wrong (point 7), and that hindsight bias makes after-the-fact analysis of human performance inaccurate (point 8) (Cook); Nancy Leveson's CAST method analyses accidents as failures of control structures rather than chains of events. You do not need the full methods to borrow the core lesson: look for the conditions, not the culprit.
Counterfactuals and hindsight
After an incident, everything looks obvious. "They should have noticed the error rate" assumes the signal was as clear then as it is now. Good analysis asks what people knew and saw at the time, why their actions made sense to them, and what would have made the right action easier. This is what blameless means: not ignoring mistakes, but treating them as symptoms of the system. Google's SRE book states the aim as finding contributing causes without indicting any individual or team, because blame teaches people to hide problems (Postmortem Culture).
The same shape everywhere
Root cause analysis is the same skill in a manufacturing defect, a missed sales target or a failed project: separate the trigger from the conditions, look across categories, and fix the system rather than blaming the last person to touch it.
Down the stack
Layer 1The contract
A root cause analysis promises a factual timeline, a causal graph (or at least multiple contributing factors) that explains the impact, action items mapped to graph nodes with owners and dates, and language that describes systems and decisions rather than blaming individuals. A reviewer should be able to trace every action item back to the part of the graph it removes.
Layer 2The mechanism
The evidence comes from the same sources as debugging: deploy and config change logs, metrics, traces, logs, chat transcripts and the incident timeline. Change events are especially valuable: a system that records every deploy, flag flip and config push with timestamps turns "what changed?" into a query, and Google's troubleshooting chapter names recent changes as a productive place to start (SRE book, ch. 12). The analysis itself is graph work: collect candidate facts, link them with "contributed to" edges, and test each edge with a counterfactual: if this node had not been present, would the impact have happened, or been smaller? Edges that fail the test are removed.
Layer 3The machine
Many root causes live in low-level behaviour that the timeline only hints at: a kernel's TCP retransmission behaviour turning packet loss into seconds of latency, a garbage collector pause exceeding a lease timeout so a node is wrongly declared dead, a file system filling up so write() returns ENOSPC and a service fails in an untested path. Reaching that level means reading kernel logs (dmesg for OOM kills and disk errors), garbage collector logs, and system-call failures, and it is often where the systemic fix is: a timeout tuned against real GC pauses, disk alerts at 80 percent, error paths tested.
Why it's built this way
- Complex systems fail through combinations of conditions, not single faults.
- A single chain finds one combination and misses the rest.
- Graphs show every path to impact, and where a fix removes many paths.
- Blaming individuals hides the conditions that made errors likely, and discourages honest reporting.
- So: separate trigger from conditions, map them as a graph, and fix at several levels.
Diagram
flowchart LR T["Trigger: new query deployed"] --> Q["Query scans 40M rows"] M["Migration failed silently"] --> I["Index missing"] P["Pipeline ignores migration errors"] --> M I --> Q Q --> H["Connections held 30 s"] S["Pool shared by all endpoints"] --> X["Pool exhausted"] H --> X X --> O["Checkout fails 40 min"] A["Alert routed to old channel"] --> D["Detection delayed 25 min"] D --> O
Worked example
Checkout fails for 40 minutes. The timeline: at 10:02 a release adds an order-history query; at 10:05 database connections hit the pool limit of 200; errors start; at 10:30 someone notices the alert in an old channel; at 10:38 the release is rolled back; at 10:42 checkout recovers.
A 5 whys chain finds the missing index and the pipeline that ignored migration failures. The fishbone adds other branches. Configuration: the connection pool is shared across all endpoints, so a slow history page could starve checkout. Monitoring: the alert went to a channel the team stopped using in a reorganisation. Process: there was no canary, so the release went to 100 percent at once. Data: the table had grown from 2 million to 40 million rows since the query was tested.
The causal graph has four roots: the silent migration failure, the shared pool, the misrouted alert and the lack of a canary. Action items, one per root: fail the pipeline on migration errors; separate pools per endpoint class so checkout keeps its connections; route alerts by service ownership from a single source of truth, with a weekly test page; canary releases with automatic rollback on error rate. Any one of them would have reduced this incident: the pipeline fix would have prevented it, the pool split would have kept checkout working, and the alert and canary fixes would have cut 40 minutes to about 5.
In the wild
- AWS Kinesis, 2020: trigger versus cause. Adding servers to the front-end fleet pushed each past an OS thread limit, because every server keeps a thread per peer. AWS explicitly called the capacity addition the trigger, not the root cause, and listed fixes at several levels: thread-consumption alarms, larger servers, a dedicated shard-map cache fleet, and cellularisation. Recovery took about 17 hours from first alarms (AWS summary).
- AWS S3, 2017: one mistyped input. During a playbook step to remove a few billing servers, one input was entered wrong and a much larger set was removed, forcing restarts of the index and placement subsystems; S3 was down about 4 hours 17 minutes. The report's fixes target the system: the tool removes capacity more slowly, refuses removals below a minimum, and other tools are audited for similar checks (AWS summary).
- Google Cloud, 2021: a race that validation could not see. The bug existed for about six months and was found internally on 12 November; the team chose a gradual patch rollout, and the race struck an unpatched cluster on 16 November. The write-up lists the fix, input validation, alerting and architectural isolation, not a person (incident report).
Trade-offs
| Method | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| 5 whys | Fast, easy to teach | One chain, stops at people | Small incidents, as a starting point |
| Fishbone | Breadth across categories | Can produce long lists without links | Group postmortems |
| Causal graph | Shows multiple paths and shared fixes | Takes more time | Significant incidents |
| Formal methods (CAST) | Deep systemic insight | Expertise and time | Major or repeated incidents |
Pitfalls in production
- Stopping at human error. Fix: ask what made the error easy and its consequences large.
- One root cause in the report because the template has one field. Fix: list contributing factors and draw the graph.
- Action items that only remove the trigger. Fix: add items for detection, blast radius and the conditions; AWS's Kinesis review treated the capacity addition as the trigger and fixed the thread-limit design behind it (AWS).
- Hindsight bias in the timeline ("they should have seen it"). Fix: describe what people saw and knew at the time.
- Action items without owners or dates. Fix: track them like any other work, and review completion.
Leadership lens
- Change the postmortem template: contributing factors and a graph instead of a single root cause field.
- Record every change (deploys, flags, config, migrations) as events with timestamps; it makes analysis much faster.
- Review action items across incidents for repeated themes; the most common contributing factor is your next investment.
- Model blameless language yourself, especially when the mistake was made by someone senior or by you.
Check yourself
What is the difference between a trigger and a contributing factor?
Show answer
The trigger starts the incident; contributing factors are conditions that allowed it to cause impact or made it worse, and fixing them prevents related incidents too.
Why is "the engineer ran the wrong command" a poor place to stop a 5 whys?
Show answer
It blames a person and leads to "be more careful", while the real question is why the system made the wrong command easy and so damaging.
How does a counterfactual test help build a causal graph?
Show answer
For each candidate cause, ask whether the impact would have happened or been smaller without it; if removing it changes nothing, it is not a contributing factor.
Why might four action items be better than one strong fix?
Show answer
Each addresses a different path or layer (prevention, detection, blast radius, recovery), so the system stays safe even if one defence fails.
Sources and further study
Systems docs
- Effective Troubleshooting (Google SRE book): hypothesis testing, bisection and recent changes.
- Summary of the Amazon S3 Service Disruption, 2017: a postmortem whose fixes are tooling limits.
Books
- Site Reliability Engineering (Google), Ch 15: Postmortem Culture: Learning from Failure.
- The Site Reliability Workbook (Google), Ch 10: Postmortem Culture: Learning from Failure, with good and bad postmortem examples.
- Release It!, 2nd ed. (Nygard), Ch 2: Case Study: The Exception That Grounded an Airline, a worked postmortem that traces a chain of failures.
Other curricula
- Cook and Allspaw above both reject the single-chain model; the 5 whys on Wikipedia notes the same criticism of the technique.
Free resources
- How Complex Systems Fail (Richard Cook): short, influential essay on why there is rarely a single root cause.
- The Infinite Hows (John Allspaw): why asking "how" works better than "why" in incident analysis.
- Postmortem Culture (Google SRE book): blameless postmortems at scale.
- CAST Handbook (Nancy Leveson, PDF): analysing accidents as failures of control, not chains of events.
- Ishikawa diagram (Wikipedia): the fishbone method and its categories.
- Five whys (Wikipedia): the technique, its origins at Toyota and its criticisms.