Engineering Atlas › Problem Solving & Incidents (AI era)

Incident Response: The First 15 Minutes

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Minute 0 to 2: acknowledge and look

A page arrives. The on-call engineer acknowledges it so the escalation policy stops paging the next person, then spends two minutes confirming the signal is real. Check the alert's own graph, one user-facing indicator (the status of the login page, the checkout success rate, a synthetic probe) and the deploy log. The only question is: incident or noisy alert?

Minute 2 to 5: declare

If users are affected or likely to be, declare. Declaring means three things: a named incident with an ID, a dedicated channel or bridge, and a severity. Make declaring cheap, with a chat command that creates the channel, posts the template and pages the incident commander rotation, so nobody needs permission. Google's SRE book gives concrete triggers: declare if a second team must get involved, if customers can see the problem, or if it is still unsolved after an hour of focused analysis (Google SRE book).

Minute 5 to 8: take command

Someone becomes incident commander (IC). On a small team the on-call engineer often holds the role at first, then hands off when a dedicated IC joins. The IC does not debug. The IC states the current understanding in one sentence, assigns an operations lead to drive the fix, assigns someone to communications, and sets the next checkpoint time. "We are seeing 30 percent checkout failures since 14:02, started after the 13:58 payments deploy. Priya is ops lead. Sam owns updates. Next sync at 14:15."

Minute 8 to 15: stabilise

The ops lead looks for the cheapest reversible action that reduces impact: roll back the last deploy, flip the feature flag off, drain the bad zone, shed a class of traffic, scale out. Diagnosis continues in parallel only if it does not slow the mitigation. Comms posts the first internal update and, for customer-facing impact, the first status page entry, even if it only says "we are investigating elevated errors in checkout".

Severity from signals

Severity is a routing decision. It decides who gets paged, how often updates go out, and whether executives are told. Tie it to observable impact: fraction of requests failing, fraction of users affected, data loss, security exposure, revenue paths. Upgrades are free; downgrades are a deliberate IC decision.

The four questions

In the first quarter hour, every useful action answers one of four questions. What is broken, from the user's point of view? Since when? What changed near that time (deploys, config pushes, flag flips, traffic shifts, certificate expiry, a dependency's incident)? What can we turn off or roll back right now? Everything else waits.

Down the stack

Layer 1The contract

The human process promises: an acknowledged page within a target time (often 5 minutes), a declared incident with a named IC within 10, a first internal update within 15, and a severity that maps to a written paging and update policy. It also promises that anyone may declare and that nobody is punished for declaring an incident that turns out to be small. These promises are testable: measure time to acknowledge, time to declare and time to first update from your incident tool's event log.

Layer 2The mechanism

Most pages come from an alerting pipeline. In Prometheus, an alert rule has an expr, a for duration that the condition must hold before the alert fires, and labels such as severity. Alertmanager receives firing alerts, groups them by labels (group_by), waits group_wait (default 30 s) to batch related alerts, then sends a notification, and repeats it every repeat_interval (default 4 hours) while the alert keeps firing; group_interval (default 5 minutes) spaces notifications about new alerts joining the group (Alertmanager configuration). Inhibition rules suppress child alerts when a parent fires, for example silencing per-host alerts while a whole zone is down. The paging service then walks an escalation policy: page the primary, wait N minutes for an acknowledgement, page the secondary, then a manager. Multi-window burn-rate alerts from the Google SRE workbook page when the error budget burns 14.4 times faster than sustainable over both a 1-hour and a 5-minute window (for a 99.9 percent SLO, that is 2 percent of the budget gone), which catches fast burns quickly without paging on brief blips. A 6x burn over 6 hours and 30 minutes also pages, and a 1x burn over 3 days and 6 hours only opens a ticket (SRE workbook).

Layer 3The machine

Detection latency is physical. With a 30 s scrape interval, a rate() over 2 minutes and for: 2m, a hard failure needs roughly 3 to 5 minutes before the page goes out, plus group_wait. That is a third of your first 15 minutes before a human knows. Once on a host, the first minute of looking is a known sequence: uptime for 1, 5 and 15 minute load averages (is load rising or falling?), dmesg -T | tail for the OOM killer and disk errors, vmstat 1 for run queue length, swapping and CPU steal, ss -s for socket counts and TIME_WAIT buildup, df -h for full disks, and top for the hungry process. These read /proc or the kernel ring buffer and are safe on a struggling machine, unlike a heavy profiler.

Why it's built this way

Human attention is the scarcest resource in an outage, so the process spends it on coordination first. Many people debugging in parallel without a coordinator duplicate work and step on each other, so one person holds command. Alerting has inherent delay from sampling and windows, so the human side must be fast to compensate. Impact grows linearly with time while root cause can take hours, so the first goal is stopping the bleeding, not understanding it.

Diagram

flowchart TD
  A["Page fires"] --> B["Ack and confirm signal"]
  B --> C{"Users affected?"}
  C -->|no| D["Ticket, tune alert"]
  C -->|yes or unsure| E["Declare: ID, channel, severity"]
  E --> F["Assign IC, ops lead, comms"]
  F --> G["What changed? Blast radius?"]
  G --> H["Cheapest reversible mitigation"]
  H --> I{"Impact reduced?"}
  I -->|no| G
  I -->|yes| J["Stabilise, then diagnose"]

Worked example

At 14:02 a burn-rate alert fires for the checkout service. The on-call acknowledges at 14:04, sees checkout success at 76 percent (normally 99.5), and notices a payments-service deploy finished at 13:58. At 14:05 she runs the /incident declare command: SEV2 at first, because about 24 percent of checkout attempts fail, just under the 25 percent line the team uses for SEV1. Two minutes later the dashboard shows 34 percent failures and she upgrades to SEV1.

At 14:07 the IC rotation's primary joins and takes command. She names the on-call as ops lead and a support engineer as comms. The ops lead proposes rolling back the 13:58 deploy, which takes 6 minutes through the pipeline, or disabling the new payment-routing flag, which takes 30 seconds. The IC picks the flag. At 14:09 the flag is off; by 14:11 success rate is back to 99.3 percent.

Cost arithmetic: at 400 checkouts a minute and an average basket of 60 dollars, a 30 percent failure rate loses about 7,200 dollars a minute of attempted revenue. Nine minutes from page to mitigation cost roughly 65,000 dollars. Had the team debugged the stack trace first and rolled back at 14:40, the bill would have been near 280,000.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Declare on any user impactFast mobilisation, good data on incident ratesMore small incidents, some noiseDefault for customer-facing systems
Declare only above a thresholdFewer interruptionsSlow start on incidents that growInternal tools with tolerant users
On-call is IC until handoffNo waiting for another personIC and debugger in one headSmall teams, first minutes of any incident
Dedicated IC rotationClear command, debuggers stay focusedNeeds trained people and a rotaMore than a few incidents a month

Pitfalls in production

Leadership lens

Check yourself

  1. Why should the incident commander avoid debugging?

    Show answer

    Debugging consumes attention and narrows focus to one hypothesis, while the IC's job is to see the whole response, assign work and make decisions such as when to roll back.

  2. Your alert has for: 5m on a 1-minute rate window with a 60 s scrape. What does that cost you in an outage?

    Show answer

    At least 6 to 7 minutes from failure to page, so nearly half of the first 15 minutes passes before anyone is involved. Shorter windows or burn-rate alerts trade a little noise for faster detection.

  3. Why are severity upgrades free but downgrades deliberate?

    Show answer

    Early signals tend to understate impact, and a premature downgrade drops paging and update cadence just when the incident might be growing.

  4. A deploy finished 4 minutes before errors began, but a teammate is sure the cause is a database issue. What do you do first?

    Show answer

    Revert or disable the recent change anyway if it is cheap and safe, because it is the most likely cause and the cost of being wrong is small; investigate the database in parallel.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →