Engineering Atlas › System Design Fundamentals

Alerting that wakes people for the right reasons

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Black-box and white-box monitoring

Black-box probes test the system from outside like a user, catching total outages and DNS or certificate failures that internal metrics miss. White-box monitoring uses internal metrics, logs, and traces to explain why. You need both: black-box tells you it is broken, white-box tells you where. A cheap middle ground is a health endpoint probed from locations that represent your users: keep the liveness check shallow and let only the readiness check touch required dependencies, so a database outage does not restart every instance (Azure health endpoint pattern). Google's rule for anything that pages is short: every page should be actionable and should need human judgement, not a scripted response (Google SRE book).

From SLO to error budget

An SLO such as "99.9 percent of checkout requests succeed over 30 days" allows 0.1 percent failures, the error budget. At 10 million requests per day, that is 300,000 failed requests per 30 days. Burn rate is the observed error ratio divided by the allowed ratio. A burn rate of 1 spends exactly the budget in 30 days; a burn rate of 14.4 spends 2 percent of the monthly budget in one hour.

Multi-window, multi-burn-rate alerts

A single threshold such as "error rate above 0.1 percent for 5 minutes" pages on every brief blip and is slow to notice gradual leaks. A better pattern combines a long and a short window for each severity:

The long window gives significance; the short window makes the alert reset quickly once the problem is fixed. These are the defaults the Google SRE Workbook recommends for a 99.9 percent SLO, with the short window set to about one twelfth of the long one (SRE Workbook).

Routing, grouping, and silences

Alert rules fire into a manager (Alertmanager, PagerDuty, Jira Service Management; Atlassian stopped selling Opsgenie in June 2025 and shuts it down on 5 April 2027 (Atlassian)) that deduplicates identical alerts, groups related ones (all pods of one service into one page), inhibits downstream alerts when an upstream one is firing (do not page for 40 services when the database is down), and routes by team and severity. Silences cover maintenance and must expire automatically.

Down the stack

Layer 1The contract

An alerting pipeline promises that when a rule's condition has held for its for duration, a notification reaches the routed receiver within a bounded delay (typically under a minute plus the group wait), that identical alerts are not re-sent more often than the repeat interval, and that a resolved notification follows when the condition clears. It promises nothing if the pipeline itself is down, which is why a heartbeat to an external service is part of the design. Rules evaluate on a fixed interval, so detection time is quantised to that interval.

Layer 2The mechanism

Prometheus evaluates each rule group every evaluation_interval (often 15 to 60 seconds). An alert rule's expression returns a vector; each element becomes an alert instance identified by its labels and moves through a state machine: inactive, pending (condition true but for not yet elapsed), then firing. Prometheus re-sends each firing alert to Alertmanager no more often than --rules.alert.resend-delay (1 minute by default), and each send pushes the alert's end time forward, so Alertmanager resolves it if the updates stop (Prometheus flags). An optional keep_firing_for keeps an alert firing for a while after the condition clears, which stops flapping alerts from resolving and re-paging (Prometheus alerting rules). Alertmanager walks a routing tree of matchers to pick a receiver, groups alerts by configured labels (for example alertname, service), waits group_wait (30 seconds by default) to collect related alerts into one notification, sends further changes after group_interval (5 minutes), and resends unchanged firing groups after repeat_interval (4 hours) (Alertmanager configuration). Routes and inhibitions now use matchers; the older match and match_re fields are deprecated. Inhibition rules suppress alerts whose labels match a firing source alert. In a highly available pair, each Alertmanager instance gossips notification logs and silences so only one sends each notification. Burn-rate rules are usually built on recording rules that precompute error ratios over each window, so alert evaluation reads a few cheap series instead of raw histograms.

Layer 3The machine

A 30-day burn-rate window evaluated directly would scan 172,800 samples per series at 15-second resolution every evaluation; across hundreds of series and dozens of rules that is billions of sample decodes per minute, which is why long windows rely on recording rules or downsampled data. Notification delivery is ordinary network I/O: an HTTPS POST to a paging provider with a TLS handshake costing one or two round trips, then the provider's own push to phone networks, which commonly adds 5 to 30 seconds. End to end, a page lands roughly evaluation interval plus for plus group_wait plus delivery after the condition becomes true, often 1 to 3 minutes even for a perfect rule.

Why it's built this way

  1. Humans are the scarcest resource in operations, so alerts must be few and meaningful.
  2. Brief blips are normal in distributed systems, so rules require conditions to persist and use windows.
  3. One failure triggers many correlated alerts, so the manager groups and inhibits them into one notification.
  4. Any monitoring component can fail silently, so the pipeline sends heartbeats to an independent system that pages on their absence.

Diagram

flowchart LR
  M["Metrics: good and total events"] --> R["Rule engine: burn rates per window"]
  P["Black-box probes"] --> R
  R -->|"firing"| AM["Alert manager"]
  AM -->|"dedupe, group, inhibit"| RT{"Severity"}
  RT -->|"page"| OC["On-call phone"]
  RT -->|"ticket"| TQ["Team queue"]
  OC --> RB["Runbook and dashboard links"]
  DMS["Dead man's switch"] -.->|"no heartbeat"| EXT["External pager"]
  AM -.->|"heartbeat"| DMS

Worked example

Checkout serves 10 million requests per day with a 99.9 percent 30-day success SLO, a budget of 300,000 errors per month.

Incident 1: a bad deploy makes 5 percent of requests fail. Burn rate is 5 / 0.1 = 50. The 1-hour window crosses 14.4 within about 17 minutes (because 50 times 17/60 is about 14.4); the 5-minute window crossed almost immediately. The page arrives around 17 minutes in. To page faster, many teams add a third tier: burn rate above 14.4 over 5 minutes alone for very high burn, accepting more noise.

Incident 2: a slow memory leak causes 0.3 percent errors. Burn rate is 3. The page tiers never fire, but the ticket tier fires about 24 hours in, when the 3-day average burn passes 1, with 90 percent of the budget left. A naive "0.5 percent for 5 minutes" alert never fires at all.

Noise check: during a normal week, error rate fluctuates between 0.02 and 0.08 percent, with occasional one-minute spikes to 0.6 percent during deploys. A 5-minute threshold at 0.5 percent would page on most deploys. The multi-window rules do not, because a one-minute spike barely moves the 1-hour burn rate.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Static threshold on a cause (CPU over 90 percent)Easy to write and understandPages when users are fine; misses novel failuresTickets for capacity, never for paging
Symptom threshold (error rate over X)Tied to user painFlaky on low traffic; picks one windowSimple services without SLOs yet
Multi-window burn rateHigh precision and recall; budget-awareNeeds SLO definitions and good SLIsAny user-facing service with an SLO
Anomaly detectionCatches unknown patternsHard to explain; false positives on seasonalitySupplementary signals, not primary paging
Synthetic probesDetect total outages and external failuresCover only scripted pathsCritical journeys, DNS, TLS, regions

Pitfalls in production

Leadership lens

Check yourself

  1. Why combine a long and a short window in a burn-rate alert?

    Show answer

    The long window ensures enough budget was spent to matter, and the short window confirms it is still happening so the alert resets quickly after recovery.

  2. What does a burn rate of 1 mean?

    Show answer

    The service is failing at exactly the allowed rate and will use all of its error budget by the end of the SLO period.

  3. Why page on symptoms rather than causes?

    Show answer

    Symptoms directly reflect user pain and cover unknown failure modes, while cause alerts fire when users are unaffected and miss failures nobody anticipated.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →