Dashboards and runbooks for on-call
Open in the interactive app: practice, quiz, mock interview →In one minute
- Dashboards and runbooks are the interface between your telemetry and a tired human during an incident. Their job is to shorten time to diagnosis and time to mitigation.
- Good dashboards are layered: a service overview that answers "is it broken and how badly" in 10 seconds, then drill-downs for dependencies, resources, and individual components.
- A runbook is a decision procedure for a specific alert: what it means, how to confirm it, how to mitigate safely, and when to escalate.
- Both decay. Dashboards and runbooks must be versioned as code, reviewed after incidents, and owned like any other production artifact.
- Repeated runbook steps should become automation, with the runbook documenting the override.
How it works
Dashboard layering
Level 1, service overview. One per service, built around the user-facing SLIs: request rate, error ratio, latency percentiles, and remaining error budget, with deploy markers overlaid. These are close to Google's four golden signals of latency, traffic, errors and saturation, and to the RED method (rate, errors, duration) that Grafana recommends for services (Google SRE book, ch. 6, Grafana). Measure the latency of failed requests separately, since a slow error is worse than a fast one. Someone paged at 3 a.m. should know within seconds whether users are hurting and since when.
Level 2, dependencies and saturation. Per-dependency call rate, errors, and latency (database, cache, downstream services), plus saturation signals: thread pools, connection pools, queue depth, CPU throttling, memory. For resources, the USE method (utilisation, saturation, errors) is the usual checklist.
Level 3, component detail. Per-instance or per-partition views for finding a single bad host, shard, or availability zone.
Put the most important panels top-left, keep units consistent, draw SLO thresholds on graphs, and use template variables (region, cluster) instead of copied dashboards. Amazon's guidance adds a few practical rules: default to UTC when responders sit in several time zones, give graphs in a section the same time range and width, and split availability, latency percentiles and extreme latencies into adjacent graphs (AWS Builders' Library). If nobody can say what decision a panel informs, delete it.
Runbook anatomy
A useful runbook, linked directly from the alert, contains:
- What this alert means in user terms, and its severity.
- Triage: the dashboard link and two or three checks that confirm or rule out common causes, each with the exact query or command.
- Mitigation: safe, reversible actions in order of preference, such as roll back the last deploy, shift traffic away from a zone, enable a feature-flag kill switch, scale out.
- Escalation: who to call and when, for example if not mitigated in 30 minutes or if data loss is suspected.
- Background: architecture notes and links to past incidents.
Write for a responder who has never seen the system: copy-pasteable commands, and dangerous steps marked in bold. Google reports that writing best practices down ahead of time in a playbook gives roughly a 3x improvement in mean time to repair over improvising (Google SRE book, ch. 1). The alert itself must earn the runbook: every page should be actionable, and a page that only needs a robotic response should be automated instead (Google SRE book, ch. 6).
Keeping them alive
Store dashboards as code in git and runbooks next to the service code. At Amazon, reviewers ask during code review whether a change needs a dashboard update, and their weekly operations reviews use a wheel of fortune to choose which audit dashboard to examine (AWS Builders' Library). Each incident review asks whether the dashboard showed the problem and whether the runbook helped; fixes become action items.
Don't let the dashboard share fate with the service
A dashboard is useless if it fails with the thing it watches. In the 2017 S3 outage, AWS could not update its own status dashboard for about two hours because the admin console depended on S3 (AWS). Host monitoring, dashboards and runbooks on infrastructure with different failure domains, and keep a fallback such as direct queries against the metrics store written into the runbook.
Down the stack
Layer 1The contract
A Level 1 dashboard promises that, within its refresh interval, it shows whether each SLI is within objective, how much error budget remains, and when the last change happened, and that it loads in a few seconds even during an incident. A runbook promises that each listed step is safe for any on-call engineer to run under its stated preconditions, that it either improves the symptom or can be reverted, and that it says when to stop and escalate. Both promise to be current as of a stated review date.
Layer 2The mechanism
A Grafana dashboard is a JSON document: a list of panels, each with a data source, one or more queries, a visualisation type, thresholds, and links. Template variables such as $cluster are defined once and substituted into every query, so one dashboard serves all regions. Annotations overlay events, such as deploys pulled from CI or feature-flag changes, as vertical markers. When stored as code, the JSON is generated by a library and provisioned from git, so a pull request reviews a change to a panel like any code change. Grafonnet (Jsonnet) was the long-time choice, but Grafana now marks it as not officially supported; Grafana 12 added versioned dashboard APIs, the official Foundation SDK (Go, TypeScript, Python, Java and PHP) and Git Sync, which ties dashboards and folders to a GitHub repository (Grafana docs). Alert rules carry a runbook_url annotation and a dashboard link, which the paging tool renders in the notification; in Prometheus, annotations are the free-form fields meant for exactly this (Prometheus docs). Runbooks increasingly live as executable notebooks or scripts where each step runs a query or command and shows output inline, with dangerous steps gated behind a confirmation.
Layer 3The machine
Every panel is at least one query per refresh. A 40-panel dashboard refreshing every 10 seconds, open on 30 screens during an incident, issues 7,200 queries per minute against the metrics backend, at exactly the moment that backend may be under stress. Range queries over 7 days at 15-second resolution touch 40,320 samples per series; the backend caps points per query and the dashboard should set a step so a panel returns at most a few thousand points, since browsers render SVG or canvas charts slowly beyond tens of thousands of points. Amazon's own example of a sensible density is a three-hour window at one-minute resolution, about 180 points per line (AWS Builders' Library). Query caching in front of the backend (results cached by aligned time step) and longer refresh intervals keep incident dashboards fast; if the data changes hourly, a 30-second refresh only adds load (Grafana). Runbook commands touch real systems: a kubectl drain evicts pods respecting disruption budgets and can take minutes; a rollback replaces pods at the rollout's surge rate. Runbooks should state those expected durations so responders do not assume failure and act twice.
Why it's built this way
- Incidents happen under stress and often to people new to the system, so knowledge must be written down and one click away.
- Raw telemetry is too large to read during an incident, so dashboards summarise it in layers from symptom to cause.
- Hand-edited dashboards and wiki pages drift, so both are stored as code and reviewed.
- Repeating manual steps is slow and error-prone, so frequent safe steps become automation, with the runbook documenting the manual fallback.
Diagram
flowchart TB
A["Alert fires"] --> RB["Runbook: meaning and severity"]
RB --> D1["Level 1: SLI overview"]
D1 --> Q{"Started at a deploy?"}
Q -->|"yes"| RBK["Roll back"]
Q -->|"no"| D2["Level 2: dependencies, saturation"]
D2 --> Q2{"One zone or shard?"}
Q2 -->|"yes"| DR["Drain zone or shard"]
Q2 -->|"no"| ESC["Escalate to owning team"]
RBK --> V["Verify SLI recovery"]
DR --> V
V --> PIR["Post-incident: update runbook and dashboard"]Worked example
A search service has a p99 latency SLO of 400 ms. An alert fires at 02:10: burn rate 16 over 1 hour, above the 14.4 that Google's SRE Workbook suggests for a paging alert on a one-hour window, which corresponds to spending 2% of a 30-day error budget in that hour (SRE Workbook, ch. 5).
The on-call opens the runbook from the page. Step 1 links to the Level 1 dashboard: error ratio is normal, p99 is 1.3 s, p50 is unchanged at 60 ms, and there was no deploy in 6 hours. Tail-only latency with stable median points to a subset of requests or instances, which the runbook says to check next.
Step 2: the Level 2 panel shows that calls to the index shard service have p99 1.2 s. The Level 3 panel, templated by shard, shows that shard 17 of 64 is slow, and its node shows a disk at 100 percent utilisation from a compaction that started at 02:02.
Step 3, mitigation: the runbook lists "mark shard replica unhealthy so queries use the other replica" with the exact command and a note that it is safe while at least one replica is healthy. The on-call runs it at 02:24; p99 recovers to 280 ms by 02:27.
Time to mitigate: 14 minutes. A similar incident before the layered dashboards took 70, mostly spent finding the slow shard. The review adds a compaction panel per shard and a ticket to automate the replica switch.
In the wild
- AWS, 2017: a playbook command with one wrong input. At 9:37 a.m. PST on 28 February an engineer followed an established S3 playbook to remove a few servers, but mistyped an input and removed far more, including servers behind the index and placement subsystems. Both needed full restarts, and S3 in us-east-1 took about four hours to recover. The status dashboard could not be updated until 11:37 because its console depended on S3. AWS changed the tool to remove capacity slowly and refuse to go below a subsystem's minimum, and audited other tools for the same safeguard (AWS).
- Slack, 2021: debugging without dashboards. Packet loss in an overloaded AWS Transit Gateway took Slack down around 7:00 a.m. PST on 4 January. The dashboarding and alerting service failed early because it reached its database through the same gateways, so responders queried metrics backends directly and SSHed into hosts, some of which autoscaling then removed mid-session. Slack moved the dashboard and alerting instances into the same VPC as their databases (Slack).
- GitLab, 2017: a runbook step run on the wrong host. An engineer meant to wipe the PostgreSQL data directory on the secondary but ran it on the primary, deleting about 300 GB. None of the backup methods worked as expected, the restore took about 18 hours, and roughly six hours of data was lost. Follow-ups included documented replication runbooks, automated restore testing and a named owner for data durability, because nobody had been responsible for testing the procedure (GitLab).
Trade-offs
| Option | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Hand-built dashboards in the UI | Fast to create, flexible | Drift, duplication, no review | Exploration and one-off investigations |
| Dashboards as code with templates | Consistent, reviewable, reusable | Upfront effort, tooling | Every production service |
| Runbooks in the repo, linked from alerts | Versioned with the code, reviewed in PRs | Engineers must maintain them | Services with on-call rotations |
| Automated remediation | Fast, consistent, no human at 3 a.m. | Can amplify failures; needs guardrails | Well-understood, frequent, reversible fixes |
Pitfalls in production
- Dashboard sprawl. Hundreds of dashboards, half broken, and nobody knows which is canonical. Mark official dashboards, template them, and delete unused ones on a schedule.
- Wall of graphs. A 60-panel dashboard where the key signal is on page three. Lead with SLIs and keep Level 1 to roughly ten panels.
- Averages hiding problems. Fleet-average latency looks fine while one zone is failing. Show percentiles and break down by zone or shard.
- Stale runbooks. Commands reference a cluster decommissioned last year. Review runbooks after every use and at least annually, and test them in game days.
- Missing alert-to-runbook link. The page arrives without context and the on-call searches the wiki. Make the runbook URL a required field on every alert rule.
- Monitoring that shares fate with production. Slack's dashboards failed during its January 2021 outage because they sat behind the same failing network path (Slack). Run monitoring in a separate failure domain.
Leadership lens
- Treat time to mitigate as the metric that dashboards and runbooks exist to improve, and review it after incidents.
- Require a runbook and a Level 1 dashboard as part of a service's production-readiness review.
- Run game days where someone unfamiliar with the service follows the runbook. The gaps they hit are the gaps a new on-call will hit.
- Prefer turning repeated runbook steps into safe automation, and keep the manual path documented for when automation fails.
Check yourself
Why should a Level 1 dashboard lead with SLIs rather than resource metrics?
Show answer
SLIs show immediately whether users are affected and how badly, which decides urgency; resource metrics explain causes and belong lower in the drill-down.
What makes a mitigation step suitable for a runbook used by any on-call engineer?
Show answer
It is safe, reversible, precisely specified with copy-pasteable commands, and states its preconditions and risks.
Why store dashboards and runbooks as code?
Show answer
Versioning and review prevent drift, templates keep services consistent, and changes can be tied to incidents and code changes.
Sources and further study
Primary
- Google SRE Workbook, ch. 5: Alerting on SLOs: multiwindow burn-rate alerts and the thresholds behind them.
- AWS: Summary of the Amazon S3 service disruption (2017): a playbook command gone wrong and a status page that shared fate with S3.
Systems docs
- Grafana: observability as code: Foundation SDK, Git Sync, Terraform and the other ways to keep dashboards in git.
- Prometheus: alerting rules: annotations for descriptions and runbook links, plus
forandkeep_firing_forto stop flapping pages.
Books
- Site Reliability Engineering, Beyer et al., chapter 6 "Monitoring Distributed Systems" and chapter 11 "Being On-Call": golden signals, actionable pages and on-call load.
- The Site Reliability Workbook, Beyer et al., chapter 4 "Monitoring" and chapter 9 "Incident Response".
- Observability Engineering, Majors, Fong-Jones and Miranda, chapter 12 "Using Service-Level Objectives for Reliability": why SLO-based alerts beat threshold dashboards.
Other curricula
- Azure Well-Architected: incident management: frames runbooks as documented procedures inside an incident plan, with one shared incident dashboard and regular drills.
- roadmap.sh System Design: lists Monitoring, Health Monitoring and Availability Monitoring as cloud design patterns; it does not cover runbooks.
- donnemartin/system-design-primer: does not cover dashboards or runbooks; operations appear only as a note that microservices add operational complexity.
Watch
- Dashboards and Runbooks: Scrapbooking for Engineers (SREcon22 Asia/Pacific): practical guidance on building both as operational tools.
- Observability: it's not just an ops thing (Charity Majors, NDC): why static dashboards are not enough and how to explore unknown failures.
Free resources
- Building dashboards for operational visibility (AWS Builders' Library): how Amazon layers and reviews its dashboards.
- Grafana dashboard best practices: maturity levels, templating, and reducing sprawl.
- Use playbooks to investigate issues (AWS Well-Architected): structure for playbooks and runbooks.
- PagerDuty Incident Response: an open incident response guide with on-call practices.