Engineering Atlas › Problem Solving & Incidents (AI era)

Mitigate First, Diagnose Second

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why diagnosis is the slow path

Diagnosis is open-ended. You do not know how many hypotheses you will test or how long each takes. Mitigation is a short list of known actions with known costs. If a rollback takes 6 minutes and diagnosis takes somewhere between 10 minutes and 4 hours, the expected impact is far lower if you roll back first and diagnose a healthy system later.

Generic mitigations

A small set of actions fixes most incidents without knowing why they happened:

Choosing among them

Rank options by three things: how long until it takes effect, how likely it is to help, and what happens if it is wrong. A reversible action that takes 30 seconds and has a 40 percent chance of working is often better than a one-way action that takes 20 minutes and has a 90 percent chance. Try the cheap reversible guess first, check impact, then move on. Correlate with change: if a deploy landed in the 30 minutes before onset, rolling it back is the first candidate even when "it can't be that".

Preserve evidence

Mitigation and evidence can conflict. Before you restart a leaking process, take a heap dump or at least record its memory graph. Before you terminate a misbehaving host, take it out of the load balancer instead and leave it running for later inspection. Save the logs that rotate quickly. Write every action into the incident timeline with a timestamp so the later analysis can tell cause from effect. Azure's guidance makes the same trade: keep failed resources in a quarantined state for forensic analysis rather than discarding them, and build toggles that disable or isolate a faulty component without a redeploy (Azure Well-Architected).

When mitigation is not enough

Some incidents have no generic fix: data corruption that is already written, a dependency that is fully down with no alternative, a security breach. Then the question becomes "how do we limit the damage?": stop writes, put the system into read-only mode, fail closed rather than open, or communicate a workaround. Even here, damage limitation comes before full diagnosis.

Down the stack

Layer 1The contract

The process promises that the incident team will pursue the fastest available action that reduces user impact, that every mitigation is announced before it runs, that each has a stated rollback, and that evidence needed for diagnosis is preserved when doing so costs less than a few minutes. It also promises that the team re-checks impact after each action, rather than assuming success, and keeps going until user-facing indicators are back within SLO.

Layer 2The mechanism

Each generic mitigation has a concrete mechanism. In Kubernetes, kubectl rollout undo deployment/api restores the previous pod template from the Deployment's revision history (old ReplicaSets kept up to revisionHistoryLimit, 10 by default; Kubernetes docs) and performs a rolling update governed by maxSurge and maxUnavailable. A feature-flag kill switch is a value in a flag service; SDKs keep a local cache and receive changes over a streaming connection (server-sent events in several commercial products) or by polling every N seconds. A traffic drain changes load-balancer weights, sets a target to draining so in-flight connections finish within a deregistration delay (300 seconds by default on AWS Application Load Balancer target groups; AWS docs), or removes a DNS record. Load shedding is a check at the front door: if the in-flight request count or queue time exceeds a limit, return 503 with Retry-After immediately, before doing work. The rejection has to be far cheaper than the work, and the aim is to keep latency low for the requests you accept, not to be fair to all of them (AWS Builders' Library).

Layer 3The machine

These actions differ by orders of magnitude in time on real machines. A flag change over a streaming connection reaches SDKs in about a second; a polling SDK at a 30 s interval averages 15 s. A rollback has to schedule pods, pull images (seconds if the old image is still on the node's disk, a minute or more for a large image from a registry), start the process, warm the JIT and caches, and pass readiness probes; for a JVM service with 200 pods and maxSurge: 25% that is often 5 to 15 minutes. A DNS-based drain is bounded by TTL, and some resolvers and client libraries hold records longer than the TTL. Scaling out on fresh VMs takes 1 to 3 minutes to boot plus warmup, and cold caches can push extra load to the database just as you add capacity. Restarting a process discards its heap, open file descriptors and in-memory queues: exactly the state you would want for a memory-leak or deadlock diagnosis, which is why jmap, gcore or a goroutine dump comes first if you can afford a minute.

Why it's built this way

User impact accrues every minute, so time to mitigate dominates incident cost. Root-cause search has unbounded duration, so it should not block the bounded actions. Recent change is the most common trigger of incidents (Google's SRE book estimates roughly 70 percent of outages are due to changes in a live system; Google SRE book), so undoing change is the highest-yield generic fix. Reversible actions let a team try several guesses quickly, so systems are built with flags, rollbacks and drains in advance, before anyone needs them at 3 a.m.

Diagram

flowchart TD
  A["Impact confirmed"] --> B{"Change in last 30-60 min?"}
  B -->|yes| C["Roll back or disable flag"]
  B -->|no| D{"Localised to host, zone or region?"}
  D -->|yes| E["Drain it"]
  D -->|no| F{"Overload?"}
  F -->|yes| G["Shed load or scale out"]
  F -->|no| H["Block bad input or degrade feature"]
  C --> I{"Impact reduced?"}
  E --> I
  G --> I
  H --> I
  I -->|no| B
  I -->|yes| J["Stabilise, preserve evidence, diagnose"]

Worked example

At 09:12 search latency p99 jumps from 180 ms to 2.4 s and 8 percent of requests time out. The change log shows three entries in the hour before: a search-ranking deploy at 08:51, a config change raising the cache TTL at 09:05, and a new index build that finished at 09:09.

The ops lead lists options:

OptionMinutes to effectChance it helpsReversible
Revert cache TTL config10.3yes
Roll back ranking deploy90.3yes
Switch to previous index30.5yes
Add 50 percent more search nodes120.2yes

Dividing time by chance of success gives an expected cost per attempt: TTL revert about 3.3, index switch 6, rollback 30, scale-out 60. The IC approves the TTL revert at 09:16; no change by 09:18. The index switch goes at 09:19 and p99 drops to 210 ms by 09:22. Total impact is 10 minutes.

Before switching, the ops lead saved the slow-query log and kept one node on the new index out of rotation. The postmortem later finds the new index had a field mapping that disabled a filter cache, which the preserved node made easy to confirm. Had the team started by profiling queries, the investigation would have taken around 90 minutes of continued timeouts.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Roll backUndoes the likeliest causeSlow for large fleets; not possible after one-way migrationsA deploy preceded onset
Feature flag offSeconds, preciseOnly covers flagged codeThe new feature is suspected
Drain host or zoneRemoves a sick part quicklyRemaining capacity must absorb loadImpact is localised
Shed loadProtects the core pathSome users get errors on purposeOverload or a retry storm
RestartClears bad stateDestroys evidence, often recursNothing else is available

Pitfalls in production

Leadership lens

Check yourself

  1. Why is a reversible 30-second action with a low chance of success often tried before a slow, likely fix?

    Show answer

    Its expected cost (time divided by chance of success) is small, and if it fails you lose little and still learn something before trying the slower option.

  2. What evidence would you capture before restarting a process suspected of leaking memory?

    Show answer

    A heap dump or core (jmap, gcore), memory and file descriptor counts over time, and recent logs; or take the instance out of rotation and leave it running.

  3. Why can a rollback fail to mitigate even when the deploy was the cause?

    Show answer

    The bad version may have written data or run a migration that persists, caches may hold bad entries, or the old version may be incompatible with the new schema.

  4. When is restarting a reasonable mitigation?

    Show answer

    When state is corrupted or exhausted and no cheaper action exists, ideally after evidence is captured and with the expectation that the problem may return.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →