Blameless Postmortems That Change Things

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

A postmortem exists to change the system so the next incident is less likely, smaller or shorter. A document that is written, read once and filed has failed, however good it reads.
Blameless does not mean nobody made a mistake. It means treating mistakes as symptoms of the system: what made the action reasonable at the time, and what made its consequences so large?
Blame destroys the information you need. People who fear punishment leave out details, and the organisation learns the wrong lesson ("be more careful").
A good postmortem has a factual timeline, impact in numbers, contributing factors rather than one root cause, what went well, and a small number of action items with owners, dates and a type (prevent, detect, mitigate, process).
Close the loop: track action items to completion, review themes across incidents, and share the learning beyond the team.

How it works

Why blameless

After an incident, it is tempting to find the person who pushed the button. It feels like accountability and it is quick. But it has three costs. People hide information in the next incident, so the organisation learns less. The fix becomes "retrain" or "add an approval", which rarely addresses the conditions that made the mistake easy. And the same mistake happens again with someone else. John Allspaw's writing on blameless postmortems at Etsy, building on the "just culture" idea from aviation and healthcare, made this mainstream in software: people give a full account when they trust it will be used to improve the system rather than to punish them. Google's SRE book makes the same point from the same two industries: you cannot fix people, but you can fix systems and processes (SRE book, ch. 15). Richard Cook adds the reason hindsight misleads: knowing the outcome makes the warning signs look more obvious than they were to the people who had to act (How Complex Systems Fail).

Blameless is not consequence-free. Deliberate harm or repeated reckless disregard of safety is handled separately. The postmortem process covers the vast majority of incidents, which involve competent people making reasonable decisions with the information they had.

When to write one

Decide the triggers before the incident, not during it. Google's list: user-visible downtime or degradation past a threshold, any data loss, an on-call intervention such as a rollback or traffic reroute, a long time to resolve, and a monitoring failure (the incident was found by a person, not an alert). Any stakeholder may also request one (SRE book, ch. 15).

The structure

Summary: two or three sentences anyone in the company can understand.
Impact: in numbers. Users affected, duration, requests failed, revenue, data lost, SLO budget consumed.
Timeline: times, events and decisions, from first trigger to full recovery, built from logs and chat, written in neutral language.
Contributing factors: the causal graph from root cause analysis, not a single line. Cook's argument is that overt failure needs several faults that are each insufficient alone, so "the root cause" is a story we choose after the fact (How Complex Systems Fail).
What went well: detection that worked, a runbook that helped, a quick rollback. It shows what to protect.
Where we got lucky: things that would have made it worse. These are free warnings.
Action items: each with an owner, a due date, a type and a link to the factor it addresses.

Writing without blame

Language matters. "The engineer failed to check the config" assigns fault. "The config change was applied without a diff preview; the tool does not show the target cluster" describes the system. Avoid "should have", "failed to", "careless", "human error" and counterfactuals that rely on hindsight. Describe what people saw and knew at the time, and why their actions made sense to them.

Action items that happen

Most postmortems produce too many action items, and most of them never get done. Better: three to five items, prioritised, each with a single owner and a date, covering different layers. Prevent (remove the trigger or condition), detect (find it faster next time), mitigate (limit blast radius or speed recovery) and process (change how work is done). Track them in the normal work tracker, review completion weekly, and report the completion rate. An item without an owner is a wish. Check the mix, too. Google's workbook critiques a sample postmortem whose action items were mostly mitigative and whose one "preventative" item was to make humans less error-prone; changing human behaviour is less reliable than changing the system (SRE workbook, ch. 10). The same chapter notes that action items from the first outage of a kind "dramatically reduced" the blast radius when a similar one happened three years later, which is the payoff you are tracking for.

Learning across incidents

One postmortem teaches about one incident. Many postmortems together show patterns: the same dependency, the same missing alert, the same kind of change. Tag contributing factors consistently, review them quarterly, and invest in the most common themes. Share widely: a short monthly digest of incidents and lessons is one of the most effective learning tools an engineering organisation has. Google features a "postmortem of the month" in a newsletter, runs reading clubs over old postmortems, and has new SREs re-enact a past incident in the Wheel of Misfortune with the original incident commander present. It also warns that an unreviewed postmortem might as well never have existed, so drafts get a senior review for data collected, impact assessed, depth of root cause and fit of the action plan (SRE book, ch. 15).

The same shape everywhere

Hospitals run morbidity and mortality reviews; aviation runs confidential incident reporting; militaries run after-action reviews. All found the same thing: systems improve when people can report honestly, and the fixes target conditions, not individuals.

Down the stack

Layer 1The contract

The postmortem process promises: every incident above a severity threshold gets a written review within a fixed time (often 5 working days); the review is blameless in language and practice; action items have owners and dates and are tracked to completion; and findings are shared beyond the team.

Layer 2The mechanism

Most teams use a template in a shared doc tool and an incident management system that collects the timeline automatically: chat messages, alerts, deploys and flag changes with timestamps. Action items are created as tickets with a label linking them to the incident, so completion can be reported by query. Some organisations add a review meeting, facilitated by someone not involved in the incident, where participants walk through the timeline and the facilitator asks "what made sense at the time?" questions. Etsy published its facilitation guide for this, now archived, under a Creative Commons licence (Etsy Debriefing Facilitation Guide).

Layer 3The machine

The timeline's accuracy depends on the clocks and logs behind it. Events from different systems carry timestamps from different hosts, synchronised by NTP to within milliseconds usually but sometimes seconds off, and chat timestamps reflect when a message was sent, not when the person saw the alert. Paging systems record exact acknowledgement times; deploy systems record start and finish. When building the timeline, prefer system-generated timestamps from authoritative sources and note when ordering is uncertain, so the analysis does not rest on a sequence that never happened.

Why it's built this way

Incidents come from conditions in the system, so learning requires honest detail about how work actually happened.
Honest detail requires safety from blame.
Written structure ensures impact, factors and actions are covered consistently.
Action items fail without owners, dates and tracking.
Patterns appear only across many incidents, so tagging and periodic review matter.

Diagram

flowchart LR
  I["Incident resolved"] --> T["Timeline from logs and chat"]
  T --> F["Contributing factors"]
  F --> W["What went well, where we got lucky"]
  W --> A["3 to 5 action items: owner, date, type"]
  A --> R["Review meeting, blameless facilitation"]
  R --> TR["Track to completion"]
  TR --> Q["Quarterly theme review across incidents"]

Worked example

A team's postmortem draft for a 50-minute outage says: "Root cause: Alex deployed a config with a typo. Alex should have tested it. Action items: 1. Alex to be more careful. 2. Add a second approval for config changes. 3. Improve monitoring. 4. Write more tests. 5. Update docs. 6. Consider a config validation tool. 7. Review deploy process."

A reviewer rewrites it. Summary and impact in numbers: 50 minutes, 18 percent of API requests failed, roughly 400,000 errors. Contributing factors: config changes skipped the canary stage that code changes go through; the config schema allowed an unknown key without error; the alert fired after 22 minutes because its window was 20 minutes; rollback of config required a manual edit rather than a revert button. What went well: the on-call engineer declared within 2 minutes of the page. Where we got lucky: the change was made at a low-traffic hour.

Action items, reduced from seven vague ones to four specific ones: route config changes through the canary stage (prevent, owner and date); reject unknown keys in config validation (prevent); a 5-minute burn-rate alert (detect); one-click config revert (mitigate). Alex's name appears only in the timeline as the person who made the change, alongside what the tooling showed them. Six weeks later, all four items are done, and a similar typo by another engineer is caught by validation before deploy.

In the wild

GitLab, 2017: a database deletion, written up with the engineer's name redacted next time. During a replication problem an engineer ran a delete on the wrong database host, removing about 300 GB of production data. Every backup path then turned out to be broken: pg_dump 9.2 was producing errors against a 9.6 server, so the S3 bucket was empty. GitLab restored from a snapshot taken about 6 hours earlier and lost data created between 17:20 and 00:00 UTC. It kept a public working document and streamed the recovery to around 5,000 viewers. The postmortem notes the document had named the engineer, with their consent, and that names would be redacted in future (GitLab).
AWS, 2017: a typo fixed in the tool, not the person. An authorised engineer following an established playbook mistyped one input and removed far more S3 servers than intended, taking S3 in us-east-1 down for hours. The published fixes target the tool: it now removes capacity more slowly and refuses to go below a subsystem's minimum, and other tools were audited for the same gap (AWS).
Google, 2014: the "Diskerase" case study. Google's workbook uses a real outage in which all satellite machines were sent to a disk-erase process, and shows a weak and a strong postmortem of it side by side. The strong version replaces "make humans less error-prone" with preventative changes, and the workbook says the action items cut the blast radius of a similar incident three years later (SRE workbook, ch. 10).

Trade-offs

Practice	Strengths	Weaknesses	Choose when
Written postmortem for every incident	Complete record	Overhead, fatigue	Small orgs with few incidents
Severity threshold for postmortems	Focuses effort	Small incidents' lessons missed	Most orgs
Facilitated review meeting	Deeper insight, shared learning	Time from several people	Significant incidents
Lightweight review for minor incidents	Cheap, still captures learning	Less depth	Frequent small incidents
Many action items	Feels thorough	Most never get done	Avoid; prioritise 3 to 5

Pitfalls in production

Naming and blaming in the document. Fix: describe systems and decisions; people appear only in the timeline, neutrally.
A single root cause field. Fix: contributing factors from a causal analysis.
Action items like "be more careful" or "improve monitoring". Fix: specific, testable items with owners and dates.
Action items never completed. Fix: track in the normal backlog and report completion rates to leadership.
Postmortems nobody outside the team reads. Fix: a monthly digest and a searchable archive.

Leadership lens

Model blameless language, especially when the mistake was yours; teams copy what leaders do under pressure.
Protect time for action items: they compete with features, and they lose unless someone senior defends them.
Run a quarterly review of contributing factors across incidents and pick one theme to invest in.
Celebrate early declarations and honest accounts, not only fast fixes.

Check yourself

Why does blaming individuals make future incidents more likely?
Show answer
People hide details to avoid punishment, so the organisation misses the real conditions, and fixes target individuals instead of the system that will trip someone else.
What makes an action item likely to be done?
Show answer
It is specific and testable, has one owner and a due date, is tracked in the normal work system, and is one of a small, prioritised set.
Why include "where we got lucky"?
Show answer
It reveals conditions that would have made the incident worse, giving warnings to act on before they cause a bigger incident.
How would you rewrite "the engineer failed to validate the input"?
Show answer
Describe the system: "The input was accepted without validation; the form and API do not check the field's format", and ask why validation was missing.

Sources and further study

Primary

GitLab: postmortem of the database outage of 31 January 2017: a public postmortem with a timeline, recovery steps and the decision to redact names.

Systems docs

Atlassian: April 2022 outage post-incident review: a long public review with a timeline, workstreams and action items.
AWS: summary of the S3 service disruption (2017): an example of fixes aimed at tooling.
Etsy Debriefing Facilitation Guide: how to run a learning-focused review meeting (archived, read-only).

Books

Site Reliability Engineering, Beyer et al.: chapter 15 (Postmortem Culture: Learning from Failure) and chapter 33 (Lessons Learned from Other Industries, including its postmortem culture section).
The Site Reliability Workbook: chapter 10 (Postmortem Culture), with the bad and good postmortem pair.

Free resources

Postmortem Culture: Learning from Failure (Google SRE book): why blameless postmortems work and how Google runs them.
Postmortem Culture (Google SRE workbook): good and bad postmortem examples, side by side.
Incident postmortems (Atlassian): templates and process from a large incident tool vendor.
Moving Past Shallow Incident Data (Adaptive Capacity Labs): why counts and single causes miss what matters.
Howie: The Post-Incident Guide (Jeli): a detailed guide to learning-focused incident reviews.
How Complex Systems Fail (Richard Cook): the safety-science basis for blameless analysis.

Open in the interactive app: practice, quiz, mock interview →