Engineering Atlas › Problem Solving & Incidents (AI era)

Hypothesis-Driven Investigation

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Multiple working hypotheses

The most common investigative failure is falling in love with the first explanation. Once you believe "it's the database", every slow query looks like proof, and evidence that doesn't fit gets explained away. The counter is old: geologist T. C. Chamberlin argued in 1890 for keeping several working hypotheses alive at once (Wikipedia), and John Platt's 1964 essay "Strong Inference" turned it into a loop (summary): list alternative hypotheses, devise an experiment whose possible outcomes exclude one or more of them, run it cleanly, and repeat.

In engineering terms, write the list before you start digging:

For each, write what you would expect to see if it were true and if it were false.

Choosing the next test

Rate each candidate test on three things: how differently the hypotheses predict its outcome (discrimination), how long it takes (cost), and whether it can make things worse (risk). Rolling back the deploy discriminates H1 from the rest strongly but takes 10 minutes and has risk. Looking at per-AZ error rates takes 30 seconds and splits H3 from H1 and H2. Do the cheap, safe, discriminating tests first, and use mitigations like rollback when the cost of waiting is high, accepting that they also act as tests.

A test that every hypothesis predicts the same way ("CPU is high") feels productive and changes nothing. Before running a query, say out loud what result would make you drop a hypothesis.

Priors and base rates

Not all hypotheses start equal. In most organisations, a large share of incidents follow a change: a deploy, a config push, a feature flag, a dependency upgrade, a certificate expiry. Google's analysis of thousands of postmortems from 2010 to 2017 attributes 37% of outage triggers to binary pushes and 31% to configuration pushes, 68% together (SRE Workbook, appendix C). "What changed?" is the highest-yield first question because its prior is high and it's cheap to answer. Hardware faults and compiler bugs exist, but their prior is low; they deserve attention only once common causes are excluded or the evidence points there specifically.

The investigation log

Keep a running log in the incident channel or a document: timestamp, hypothesis, test, result, conclusion. Google's troubleshooting model asks for the same: design tests that rule hypotheses in or out, run the likeliest and least risky first, and document every step and result (SRE book, ch. 12). "14:22 H3 network: per-AZ 5xx rates equal across AZs, H3 unlikely." This prevents the team from re-testing ruled-out ideas, lets a new responder catch up in a minute, and turns into the postmortem timeline with no extra work.

Working with an agent

An agent with log and metrics access can list hypotheses, run queries and summarise evidence quickly. It will also anchor on the first plausible story, and it can't know that the batch job was rescheduled yesterday unless someone tells it. Give it the hypothesis list, ask it to report evidence for and against each, and check that its "confirmed" claims rest on a discriminating test, not a coincidence in time.

Down the stack

Layer 1The contract

The process promises that every conclusion is backed by a test that could have come out the other way, and that the reasoning is visible to others. Testable form: for each claim in the log there is a recorded prediction, an observation and a timestamp; every hypothesis is in one of three states (open, ruled out, supported); and "root cause found" requires the cause to explain all observations, including the ones where the problem is absent.

Layer 2The mechanism

Underneath, this is Bayesian updating. Each hypothesis H has a prior P(H). A test result E has likelihood P(E | H) under each hypothesis. The posterior is P(H | E) ∝ P(H) × P(E | H), normalised over the list. A test's value is its expected information gain: the entropy of the current distribution minus the expected entropy after seeing the result. Dividing by cost (minutes, risk) gives a ranking of what to do next. The same chapter warns about the failure mode this ranking avoids: chasing irrelevant symptoms, proposing improbable theories, and mistaking coincidence or a shared cause for causation. Nobody computes this to three decimal places at 2 a.m., but the shape guides intuition: tests where likelihoods differ a lot across likely hypotheses are worth more than tests where they don't, and one strong, cheap test beats three weak ones.

In production the discriminating tests are concrete: compare canary and baseline instances, toggle a feature flag for 1% of traffic, shift traffic away from one zone, replay a captured request against old and new builds, or diff configuration between a good and a bad host.

Layer 3The machine

Observation tools have costs that can change the system you are testing. strace uses ptrace: the kernel stops the traced process at every syscall entry and exit and switches to the tracer. Brendan Gregg measured a syscall-heavy dd at 442x slower under strace and warns of 100x slowdowns more generally, which is enough to hide a race or cause timeouts; perf trace avoids the per-syscall stops (Gregg). perf record instead samples: the CPU's performance monitoring unit raises an interrupt at a set frequency (for example -F 99, 99 times per second per CPU), and the kernel records the instruction pointer and stack, costing a few percent. eBPF tools such as bpftrace or BCC's biolatency attach small verified programs to kprobes, tracepoints or uprobes; they aggregate in kernel maps and copy only summaries to user space, so overhead is usually low, though very hot probes (millions of events per second) still add measurable cost. tcpdump copies packets through a ring buffer and can drop them under load; check the "packets dropped by kernel" line before concluding a packet was never sent. Choose the cheapest tool that answers the discriminating question, and record its overhead in the log.

Why it's built this way

  1. People see patterns that confirm what they already believe, so hypotheses must be explicit and plural.
  2. Time during an incident is expensive, so tests are ranked by how much they discriminate per minute.
  3. Observation perturbs systems, so tool choice is part of test design.
  4. Investigations get handed over and later reviewed, so the reasoning is written down as it happens.

Diagram

flowchart TD
    O["Observations"] --> L["List hypotheses with priors (include 'other')"]
    L --> P["Write predictions per hypothesis"]
    P --> T["Pick test: discrimination / cost / risk"]
    T --> R["Run test, record result in log"]
    R --> U["Update: rule out or strengthen"]
    U --> C{"One hypothesis explains all observations?"}
    C -->|"no"| L
    C -->|"yes"| V["Confirm with an intervention, then fix"]

Worked example

At 14:20, checkout error rate rises from 0.2% to 4%. The on-call lists three hypotheses with rough priors: H1 the 14:05 deploy (0.5, since recent changes dominate), H2 database saturation (0.3), H3 inter-AZ network trouble (0.2).

Candidate tests and the probability each hypothesis predicts a "positive" result:

TestH1 deployH2 databaseH3 networkCost
Roll back, errors drop0.950.100.1010 min, some risk
DB metrics show saturation0.100.900.202 min
Packet capture shows loss0.050.050.955 min
CPU is high on app hosts0.50.50.51 min

The CPU check predicts the same result under every hypothesis, so its information gain is zero, even though it's the first thing many people look at. Rollback has the highest information gain (about 0.62 bits of the starting 1.49), but per minute the DB metrics check wins (0.42 bits in 2 minutes). The on-call checks DB metrics first: connection count normal, no lock waits. Negative result: H2 drops to about 0.05. With H1 now dominant and customer impact rising, rollback is both the best test and the best mitigation. Errors fall to 0.3% within 4 minutes of the rollback finishing.

That supports H1 but doesn't prove the mechanism. The follow-up is to reproduce: replay a sample of failed requests against the new build in staging, which fails on orders with more than 50 items because of a new validation query with no index.

In the wild

Trade-offs

ApproachStrengthsWeaknessesChoose when
Single hunch, dig deepFast when the hunch is rightConfirmation bias; slow when wrongVery familiar systems, obvious signals
Explicit hypotheses and discriminating testsSystematic, explainable, handover-friendlyNeeds discipline under pressureAny incident lasting more than a few minutes
Mitigate first, investigate laterLimits customer impactMay destroy evidenceImpact is high and a safe mitigation exists
Broad data collectionCaptures evidence before it is lostCostly; drowning in dataRare or intermittent issues

Pitfalls in production

Leadership lens

Check yourself

  1. Why is "CPU is high" often a useless first test?

    Show answer

    If every candidate cause predicts high CPU, observing it doesn't change their relative probabilities, so it carries no information for choosing between them.

  2. Rollback fixed the errors. Is the root cause proven?

    Show answer

    No. It strongly supports "something in the change", but the mechanism is unknown until a reproduction or a targeted test explains why the change caused the failure.

  3. Why might perf be a better choice than strace for a latency problem?

    Show answer

    strace stops the process on every syscall via ptrace, adding large overhead that can mask or create timing problems, while perf samples at a fixed rate with low overhead.

  4. How does the base rate of change-induced incidents affect your first move?

    Show answer

    It gives "recent change" a high prior, so checking deploys, configs and flags is both cheap and likely to be informative, making it the natural first test.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →