Engineering Atlas › System Design Interviews

Design a metrics monitoring and alerting platform

20 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope: logs and traces (see Log Aggregation and Three Pillars of Observability), anomaly detection with machine learning, on-call scheduling and escalation policy (we hand off to a paging provider), and billing.

Back-of-envelope

Round assumed numbers for a large company, not anyone's published figures.

API

# Collection: targets expose, collectors scrape (pull)
GET  /metrics            -> text exposition
     http_requests_total{service="cart",code="500"} 1027 1729000000000

# Collection: agents and short-lived jobs push (remote write)
POST /api/v1/push        body: snappy(protobuf WriteRequest)
     WriteRequest { timeseries: [ { labels: [{name, value}], samples: [{value, timestamp_ms}] } ] }
     -> 200 | 429 { reason: "per-tenant series limit 5000000 exceeded" }

# Query
GET  /api/v1/query_range?query=sum by (code)(rate(http_requests_total[5m]))&start=..&end=..&step=30s
     -> { resultType: "matrix", result: [ { metric: {code: "500"}, values: [[t, "12.4"], ...] } ] }

# Rules
PUT  /api/v1/rules/{namespace}/{group}
     { interval: "30s", rules: [ { alert: "CartErrors", expr: "...", for: "5m",
       keep_firing_for: "2m", labels: {severity: "page", team: "cart"},
       annotations: {summary: "...", runbook: "https://..."} } ] }
GET  /api/v1/alerts      -> [ { labels, state: "pending"|"firing", activeAt, value } ]

# Notification
POST /api/v2/silences    { matchers: [{name, value, isRegex}], startsAt, endsAt, createdBy, comment }
     -> { silenceID }

The WriteRequest above is the Remote-Write 1.0 shape. The 2.0 spec, still marked experimental, sends io.prometheus.write.v2.Request with every label string interned once in a symbols table, and adds native histograms, metadata, exemplars and a start timestamp for counters (Remote-Write 2.0 spec). Remote write returns 429 instead of dropping silently, so a sender knows to back off and an operator can see which tenant hit which limit. Rules are stored per group because a group is the unit of scheduling: all rules in a group run in order, at one interval, on one evaluator.

Data model

Series identity. A series is its sorted label set, including __name__, and its fingerprint is a 64-bit hash of that set. Every component routes, deduplicates and merges by fingerprint.

Inverted index, per time block: (label_name, label_value) -> sorted postings list of series IDs, plus a symbol table storing each string once. A query such as {service="cart", code=~"5.."} intersects postings lists, like a search engine over labels.

Chunks: (series_id, chunk_start) -> compressed bytes, about 120 samples each. The open chunk lives in ingester memory; closed chunks are memory-mapped, then packed into immutable 2-hour blocks in object storage.

Partitioning. Writes are spread across ingesters by fingerprint on a consistent-hash ring (see Consistent Hashing) with replication factor 3. Object storage is partitioned by tenant, then time range, because nearly every query names both.

Rule groups: (tenant, namespace, group) -> versioned rule document, assigned to evaluators by hashing the group key.

Alert state: (rule_id, series_fingerprint) -> state, active_since, last_value, written back as a synthetic series (Prometheus calls it ALERTS_FOR_STATE) so a restarted evaluator restores pending timers.

Notification log (receiver, group_key) -> hash of alerts sent, sent_at and silences id -> matchers, starts_at, ends_at, creator, comment are small and gossiped to every notifier.

High-level design

flowchart LR
  T["Targets (/metrics)"] --> C["Collectors (scrape)"]
  A["Agents and jobs"] --> D
  C --> D["Distributors (validate, limit)"]
  D --> I["Ingesters x3 (WAL + head)"]
  I --> O[("Object storage: 2h blocks")]
  O --> K["Compactor (merge, downsample)"]
  K --> O
  Q["Query frontend (split, cache)"] --> QR["Queriers"]
  QR --> I
  QR --> SG["Store gateways"]
  SG --> O
  G["Dashboards"] --> Q
  R["Rule evaluators"] --> QR
  R --> N["Notifier cluster (gossip)"]
  N --> P["Pager, chat, email"]
  W["External watchdog"] -.-> N

Write path. Collectors find targets through service discovery, scrape every 15 s and batch samples into remote-write requests. A distributor validates labels, applies per-tenant limits, hashes each series to three ingesters and returns success once two have appended to their write-ahead log. Every two hours each ingester cuts its head into an immutable block and uploads it; the compactor merges replicas' blocks, removes duplicates and writes rollups.

Read path. The query frontend splits long ranges into one-day pieces, serves cached pieces and queues the rest per tenant. Queriers fan out to ingesters for recent data and to store gateways for older blocks, merge replicas and evaluate the expression. Queries over months read rollups.

Alert path. Each evaluator owns some rule groups. Every 30 s it runs each rule as an instant query, updates and persists per-series state, and sends firing alerts to every notifier instance, which group, silence, inhibit and deliver.

Deep dives

1. Ingestion: pull or push, and the cardinality budget

Pull, where the collector scrapes /metrics, gives a free health signal (a failed scrape means the target is down or unreachable), lets the monitoring side control load, and lets an engineer curl the endpoint to see what is exported. Push suits short-lived jobs, serverless functions and devices behind NAT. Google's Monarch began by discovering and pulling from entities and moved to push because it improved robustness and scalability (Monarch, VLDB 2020); Prometheus-style fleets scrape locally and push upstream with remote write. I would scrape inside each cluster and push from there to the central distributors, so the hop that crosses the WAN carries batched, compressed, retried data.

The harder decision is cardinality. Samples are cheap; series are expensive, because each carries index entries and an open chunk. A path label with unbounded values, or a pod label on a deployment that rolls hourly, creates series that live briefly and then sit in the index for the rest of the block. I would enforce limits at three points: per metric name in the collector (drop or relabel before sending), per tenant at the distributor (active series and samples per second, answered with 429s), and per query (series touched and chunk bytes fetched). Hitting any limit raises an alert to the owning team. Cloudflare, running about 4.9 billion series across 916 Prometheus servers, defaults each scrape's sample_limit to 200 and patched Prometheus to drop only the excess new series rather than fail the whole scrape (Cloudflare).

2. Storage: head block, blocks, compaction and rollups

Writes arrive for every series at once, in time order; queries read a few series over a range. Writing each sample where it finally belongs would mean millions of tiny random writes per second. Instead the ingester appends every sample to a sequential WAL and keeps per-series chunks in memory. A full chunk (about 120 samples, 30 minutes at 15 s) is written to disk and memory-mapped. Every two hours the head is cut into an immutable block: an index file, chunk files and metadata. This matches Prometheus's own layout: two-hour blocks, a WAL in 128 MB segments, and a reminder that local storage is neither clustered nor replicated, which is why the replicated ingester tier exists (Prometheus storage docs).

Immutability makes the rest simple. Blocks go to object storage, are cached anywhere and are compacted in the background without coordinating with writers. The compactor merges them into 12-hour and 24-hour blocks, which shrinks indexes and the number of objects a long query opens, and computes rollups. A rollup keeps min, max, sum and count per window, not an average, so dashboards still show peaks and means across windows stay correct. I considered a wide-column store with one row per series per hour, which pays for compaction and tombstones this data does not need, and columnar Parquet, which is great for scans across many series but weak at "this series, last hour, every 15 seconds".

3. Alert evaluation and delivery

A rule is a query plus a state machine per output series. When the expression first returns a series, it becomes pending and records active_since. If it is still returned after the for duration, it fires. If it disappears while pending, the timer resets silently; if it disappears while firing, it resolves, unless keep_firing_for holds it a little longer to stop a flapping metric from paging repeatedly. The practice below implements this.

Three details make alerting trustworthy. Pending timers must survive restarts, or every evaluator deploy resets every five-minute for; persisting and restoring active_since fixes that. Each rule group must have exactly one evaluator, assigned by consistent hashing, because a missed evaluation is cheaper than two owners racing on state. And evaluation must read data that has arrived: rules run with a small delay so a late sample does not look like a dip. Prometheus exposes this as a per-group query_offset, and a per-rule limit that discards a rule's output if it would produce too many alerts (Prometheus rules docs).

What to alert on matters as much as how. A fixed threshold with for: 5m either pages on every blip or misses slow leaks. The SRE Workbook's answer is multiwindow, multi-burn-rate alerts on an SLO: for a 99.9% target, page when the error budget burns 14.4 times too fast over both the last hour and the last 5 minutes (2% of the monthly budget gone), page at 6x over 6 hours and 30 minutes, and open a ticket at 1x over 3 days. The short window, one twelfth of the long one, makes the alert stop about 5 minutes after errors stop instead of an hour later (SRE Workbook, ch. 5).

Delivery groups alerts by labels such as (cluster, alertname). A new group waits group_wait (30 s by default in Alertmanager) to collect siblings and sends one message; changes wait group_interval (5 minutes), and an unchanged firing group is re-sent every repeat_interval (4 hours) (Alertmanager configuration). Inhibition suppresses symptom alerts while a cause alert fires, and silences mute matching alerts during maintenance. Notifiers run as a cluster of three with no leader. Every evaluator sends every alert to every notifier, never through a load balancer, which is exactly what Alertmanager's HA mode expects (Alertmanager README); instance n waits n × 15 s (the default --cluster.peer-timeout), then checks the gossiped notification log to see whether a peer already sent it. A partition can produce a duplicate page but not a lost one, a deliberate choice of availability over consistency.

Down the stack

Layer 1The contract

A remote-write request that returned 200 is on at least two ingester WALs and is queryable within one scrape interval plus a few seconds. Queries are eventually consistent across replicas and merge by fingerprint and timestamp. A rule with for: 5m pages no sooner than five minutes after the condition first holds and, when healthy, no later than for + evaluation interval + group_wait: 5 min + 30 s + 30 s. Delivery is at least once.

Layer 2The mechanism

Within a chunk, timestamps are stored as a delta of deltas, so a steady 15 s interval costs one bit per sample. Values are XORed with the previous value and only the meaningful bits are written. Both tricks come from Facebook's Gorilla, which kept 26 hours of data in memory this way (Gorilla, VLDB 2015). The WAL is a sequence of 128 MB segments holding series records (labels, written once) and sample records (series reference, timestamp, value); a restarted ingester replays segments since the last checkpoint. A label query intersects sorted postings lists in a linear merge; regex matchers first expand against every value of that label, which is why path=~".*" on a high-cardinality label is slow. Notifiers replicate the notification log and silences with a gossip library (memberlist in Alertmanager), so a late joiner converges.

Layer 3The machine

At 6.7M samples/s and 16 raw bytes each, the fleet's WALs take about 107 MB/s, under 3 MB/s per ingester across 40 hosts: trivial sequential I/O. The cost is fsync, so WAL writes are batched and synced every few milliseconds rather than per request. Memory is the real limit. Each active series holds label strings, postings entries and an open chunk buffer, and the Go garbage collector scans all of it, so a cardinality spike shows up first as longer GC pauses, then as an out-of-memory kill. Memory-mapping closed chunks moves them from the heap to the page cache, which the kernel can evict. Store gateways fetch chunk ranges from object storage with HTTP range requests at 20 to 50 ms each, so they batch ranges and keep index headers on local disk.

Why it's built this way

  1. Every series gets a sample every 15 seconds and disks are slow at small random writes, so append to one sequential log and keep per-series data in memory.
  2. Memory is finite and alerts read recent data, so keep only a few hours hot and cut the rest into immutable blocks.
  3. Immutable blocks fit cheap object storage, but each read pays network latency, so compact to fewer objects and keep indexes local.
  4. Five years at full resolution costs petabytes nobody reads, so downsample, keeping min, max, sum and count.
  5. A missed page is worse than a duplicate, so notifiers replicate without consensus.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Pull (scrape)Free up/down signal, collector controls load, easy to debugNeeds service discovery and network reach to every targetLong-lived services inside your network
Push (remote write, agents)Works for short jobs and through NAT, batching across the WANSender can overload receiver; no free liveness signalBatch jobs, serverless, edge devices, cross-region
Single Prometheus per clusterSimple, fast, few moving partsNo global view, retention limited by one diskUnder a few million series
Sharded ingesters plus object storage (Mimir, Cortex, Thanos)Horizontal scale, long retention, global queriesMany components to run and tuneTens of millions of series and up
Raw retention onlyExact answers foreverStorage grows linearly, slow long queriesShort retention or regulatory need
Downsampled rollupsCheap long history, fast year-long graphsLoses short spikes unless max is keptCapacity planning and trend dashboards
Notifiers with consensusExactly one notificationUnavailable during partitionsRarely; billing-style events
Gossip notifiers, at-least-onceAlways pagesOccasional duplicate pagesPaging humans

Leadership lens

Check yourself

  1. Why does the lesson size ingesters by active series rather than by samples per second, and roughly how much ingester memory do 100M active series need?

    Show answer

    Each series carries index entries, label strings and an open chunk in memory, while samples compress to about 1.5 bytes each and stream to a sequential WAL. At about 4 KB per series, 100M series need 400 GB, or 1.2 TB with replication factor 3.

  1. A release adds a customer_id label to a request counter. What happens with per-tenant limits in place, and what happens without them?

    Show answer

    The distributor rejects the new series with 429s, existing series keep flowing, and the owning team is alerted with the metric's name. Without limits, ingesters run out of memory one by one as the ring shifts load onto the survivors.

  1. Why does the notifier cluster use gossip and at-least-once delivery instead of consensus for exactly-once notifications?

    Show answer

    A missed page is worse than a duplicate one, and a consensus-based notifier would be unavailable during a partition. With gossip, instance n waits n times 15 s and checks the replicated notification log, so a partition can cause a duplicate page but not a lost one.

  1. What goes wrong if an evaluator restarts without persisting alert state, and how does the design prevent it?

    Show answer

    Every pending alert's timer resets, so a deploy restarts every five-minute for and delays pages. The evaluator writes active_since back as a synthetic series (ALERTS_FOR_STATE) so a restarted evaluator restores its pending timers.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →