Making Decisions Under Uncertainty
Open in the interactive app: practice, quiz, mock interview →In one minute
- Engineering decisions are almost always made with incomplete information: during incidents, in design reviews, when choosing what to build. Waiting for certainty is itself a decision, usually an expensive one.
- First ask whether the decision is reversible. Two-way doors (a flag, a rollback, a config) should be made fast; one-way doors (deleting data, a public API, a vendor contract) deserve more care.
- Compare options by expected cost: probability times impact, summed over the ways things could turn out. It turns arguments about opinions into arguments about numbers you can check.
- Information has a price and a value. Investigate further only when what you could learn might change the decision and is worth more than the delay.
- Separate decision quality from outcome. A good decision can turn out badly, and a bad one can get lucky; judge decisions by the information and reasoning available at the time.
How it works
Reversible and irreversible
Jeff Bezos's shareholder letters distinguish one-way doors (hard or impossible to undo) from two-way doors (cheap to reverse). Many decisions are two-way doors, but organisations tend to treat them all with one-way caution, which makes them slow. His 2016 letter says that many decisions are reversible and can use a lightweight process, that most should be made with around 70 percent of the information you wish you had, and that "disagree and commit" helps when there is conviction but no consensus (Amazon 2016 letter). In engineering, reversibility is something you can design: feature flags, expand-and-contract migrations, backups and canaries all turn one-way doors into two-way doors, which is why they make teams faster as well as safer.
Expected cost
For each option, list the possible states of the world, estimate their probabilities and the cost of the option in each state, and sum the products. In an incident: you see errors that started 4 minutes after a deploy. Option A, roll back: costs 6 minutes of deploy time whether or not the deploy is the cause. Option B, investigate first: if the deploy is the cause (say 70 percent likely), you lose perhaps 30 minutes of outage before rolling back anyway; if not, you saved a pointless rollback. With 2,000 dollars of impact per minute of outage, rolling back costs about 12,000 dollars in expectation; investigating first costs about 0.7 × 60,000 = 42,000. Even rough numbers make the right move obvious, and they make disagreements specific ("I think the probability is 30 percent, not 70, because..."). Check that the cheap option really is reversible, though. Knight Capital's responders uninstalled the new code from the seven healthy servers, and the SEC found this worsened the problem by making those servers run the old Power Peg code too (SEC order). A rollback is only a two-way door if you know what state it returns you to.
The value of information
Every investigation step costs time. It is worth taking only if its result could change your choice. A simple test: would any answer to this question make me do something different? If not, skip it. Decision theory formalises this as the expected value of perfect information: the difference between the expected cost of deciding now and the expected cost if you could learn the true state first. It is never negative and caps what any real study can be worth, because imperfect information is worth less (EVPI). If even perfect information is worth less than the time it takes to get it, decide now.
Pre-mortems
Before committing to a significant decision, imagine it is six months later and the decision failed badly, then ask everyone to write down why. Psychologist Gary Klein's pre-mortem technique (HBR, 2007) surfaces risks that people hesitate to raise in a normal review, because it makes finding failure modes the task rather than a criticism. Feed the results into the Risk and Change boxes of the decision's SDRC.
Deciding as a group
Under time pressure, consensus is slow. Name the decision owner, gather input briefly, decide, and record the reasoning and the triggers to revisit. "Disagree and commit" means people who argued for another option still support the decision fully once it is made, and the recorded triggers give everyone a legitimate way to reopen it if evidence changes.
Down the stack
Layer 1The contract
A well-made decision has a named owner, a recorded set of options, the key uncertainties with rough probabilities, the chosen option with reasoning, and explicit triggers for revisiting. Reviewers can later judge it against what was known at the time rather than against the outcome.
Layer 2The mechanism
The tools are simple: a decision record (an ADR or an SDRC), a small expected-cost table, and a timebox. Incident processes encode the same logic as rules of thumb: "if a deploy happened in the last hour and errors are user-facing, roll back first" is an expected-cost calculation done in advance, because the probability that the most recent change is responsible is high and the cost of a rollback is low. Error budgets do the same for release decisions: when the budget is spent, the expected cost of more risk is judged too high, and releases pause. Google's guidance lists the typical responses as prioritising recent reliability bugs, working only on reliability until the service is back within its SLO, and a production freeze, with named owners and an escalation path for disagreements (SRE Workbook, Implementing SLOs).
Layer 3The machine
Many engineering uncertainties come from measurement limits on the machine itself. Sampled metrics (a 10-second scrape interval) cannot show sub-second spikes; histogram buckets blur latency between bucket edges (a quantile estimate is only as precise as the bucket it falls in, per the Prometheus docs); log sampling at 1 percent can miss rare errors entirely. Knowing these limits tells you how much to trust a signal: an error that appears in 1 percent sampled logs 5 times in an hour probably happened about 500 times, with wide uncertainty. Good decisions account for how the evidence was produced, not just what it says.
Why it's built this way
- Certainty is rarely available in time, and delay has a cost.
- So decisions must weigh options under uncertainty, explicitly.
- Reversibility changes how much certainty you need, and can be engineered.
- Information is worth gathering only if it could change the choice.
- Recording reasoning lets teams learn from decisions rather than from luck.
Diagram
flowchart TD
D["Decision needed"] --> R{"Reversible cheaply?"}
R -->|yes| F["Decide fast, monitor, revisit"]
R -->|no| O["List options and states"]
O --> E["Estimate probabilities and costs"]
E --> V{"Could more info change the choice and is it worth the delay?"}
V -->|yes| I["Timeboxed investigation"]
I --> E
V -->|no| C["Choose lowest expected cost"]
C --> REC["Record reasoning and triggers"]Worked example
A team must choose a queue for a new event pipeline: managed Kafka at about 9,000 dollars a month, or a cloud queue service at about 2,500. The uncertainty is volume: product forecasts 20,000 events a second within a year, but such forecasts are often wrong. Engineers estimate a 30 percent chance of reaching that volume and a 70 percent chance of staying under 5,000 events a second.
If volume stays low, the cloud queue is fine; Kafka wastes 6,500 dollars a month, about 78,000 a year. If volume is high, the cloud queue would need a migration, estimated at 3 engineer-months, about 75,000 dollars plus risk. Expected first-year cost of choosing Kafka now: 0.7 × 78,000 = 54,600 in wasted spend. Expected cost of the cloud queue: 0.3 × 75,000 = 22,500 in likely migration. Cloud queue wins.
Is more information worth it? If they knew volume would be high, they would pick Kafka and avoid the migration, saving 75,000 in that 30 percent case: perfect information is worth at most 0.3 × 75,000 = 22,500. A two-week load study would cost about 15,000 in engineering time and not give perfect information. They decide now, make the choice more reversible by putting a thin publishing interface in front of the queue, and record a trigger: revisit if sustained volume passes 8,000 events a second.
In the wild
- Knight Capital, 2012: the rollback that made it worse. On 1 August 2012 new order-routing code reached only seven of eight servers, and the eighth ran retired Power Peg logic. In about 45 minutes Knight made 4 million executions in 154 stocks and lost over $460M. The SEC found Knight had no incident-response procedures, relied on its technology team in a live market, and made things worse by uninstalling the new code from the seven good servers. It fined Knight $12M (SEC order). Under pressure the cheap-looking reversible action was not reversible.
- Meta, 2021: a deliberate trade-off meets a rare failure. A maintenance command meant to assess backbone capacity took down every backbone connection. A bug in the audit tool let it through, and DNS servers withdrew their BGP routes when they judged the backbone unhealthy. Normal and out-of-band access both failed, so engineers went on site, where physical security slowed recovery. Meta called the hardening an accepted trade-off worth its day-to-day security (Meta Engineering). Decisions about rare failures need explicit costs, written down before the incident.
Trade-offs
| Approach | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Decide fast on gut | Very quick | Biased, hard to review | Trivial and reversible choices |
| Expected cost table | Explicit, debatable numbers | Estimates can be poor | Moderate to high stakes |
| Gather more data first | Reduces uncertainty | Delay has a cost | When info could flip the choice cheaply |
| Make it reversible, then decide | Lowers the stakes | Engineering effort | One-way doors you can redesign |
| Pre-mortem | Surfaces hidden risks | Takes a meeting | Significant, hard-to-reverse decisions |
Pitfalls in production
- Treating every decision as a one-way door, so the team waits for certainty. Fix: ask "how would we undo this?" first.
- Analysis that could not change the outcome. Fix: ask what answer would change your choice before investigating.
- Judging decisions by outcomes, which punishes good bets that lost. Fix: review the reasoning against what was known.
- Hidden probabilities ("this probably won't happen"). Fix: write a number; disagreement about it is useful.
- No triggers to revisit. Fix: every significant decision records the signals that would reopen it.
Leadership lens
- Push for reversibility in designs; it is the cheapest way to make the team faster.
- In design reviews, ask for the options, the main uncertainty and the expected-cost reasoning, even roughly.
- Timebox investigations and name the decision owner, especially in incidents.
- Run occasional decision reviews separate from outcomes, to build the team's calibration.
Check yourself
Why should rollback usually come before diagnosis when errors start right after a deploy?
Show answer
The deploy is the most likely cause and rolling back is cheap and reversible, so its expected cost is far lower than the expected cost of extended outage while investigating.
What question tells you whether an investigation is worth doing?
Show answer
Could any result change the decision, and is the value of that change greater than the cost of the delay?
Why is "it turned out badly" not proof of a bad decision?
Show answer
Outcomes include luck; a decision is good if it was the best choice given the information and reasoning available at the time.
How can you turn a one-way door into a two-way door in a migration?
Show answer
Use expand-and-contract steps, dual writes, flags and backups so each step can be undone, instead of a single irreversible cutover.
Sources and further study
Primary
- Amazon 2016 letter to shareholders: two-way doors, the 70 percent rule and "disagree and commit".
- Klein, Performing a Project Premortem (HBR, 2007): the original premortem technique.
- SEC order against Knight Capital (2013): a regulator's account of decisions made without incident procedures.
Systems docs
- Google SRE Workbook: Implementing SLOs: error budget policies as pre-agreed decisions.
- Meta Engineering: more details about the October 2021 outage: trade-offs in recovery access.
Books
- Site Reliability Engineering, Beyer et al., chapter 3 "Embracing Risk": risk tolerance and error budgets as a way to decide how much reliability to buy.
- Site Reliability Engineering, chapter 14 "Managing Incidents" and chapter 15 "Postmortem Culture": clear roles and blameless reviews, the basis for judging a decision by what was known.
- The Staff Engineer's Path, Tanya Reilly, chapter 4 "Finite Time": choosing what to do when time and attention are limited.
Other curricula
- Azure Well-Architected: architecture decision records: an append-only log of options, outcome, trade-offs and confidence level, the same recording habit this lesson asks for.
Free resources
- Reversible and irreversible decisions (Farnam Street): one-way and two-way doors explained.
- Jeff Bezos' 2016 letter to shareholders: deciding with about 70 percent of the information, and disagree and commit.
- Performing a Project Premortem (Gary Klein, HBR): the original pre-mortem technique.
- Expected value of perfect information (Wikipedia): the maths behind "is this investigation worth it?".
- OODA loop (Wikipedia): observe, orient, decide, act under time pressure.