Engineering Atlas › System Design Interviews

Prometheus for system design interviews

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Pull scraping

Each target exposes /metrics in a plain-text format: lines like http_requests_total{method="GET",code="200"} 18234. Prometheus discovers targets through service discovery (Kubernetes, Consul, EC2, files), rewrites labels with relabel_configs, and scrapes each target every scrape_interval, usually 15 or 30 s. Every scrape also records a synthetic up series, so a dead target is itself a signal. Pull keeps targets simple and lets you curl what Prometheus sees. Short-lived batch jobs that end before a scrape push to a Pushgateway instead. Since 3.0, Prometheus can also accept pushes directly: an OTLP receiver (--web.enable-otlp-receiver) for OpenTelemetry SDKs, and Remote Write 2.0, which carries metadata, exemplars and native histograms (Prometheus 3.0). Metric and label names may now be any UTF-8, so OpenTelemetry names with dots survive.

Metric types

Counters only go up (and reset to zero on restart); you query their rate. Gauges go up and down. Histograms export cumulative bucket counters (_bucket{le="0.25"}) plus _sum and _count, so quantiles are estimated server-side with histogram_quantile and can be aggregated across instances. Native histograms store exponential buckets in a single series and are much cheaper per resolution. They became a stable, opt-in feature in 3.8 (November 2025), enabled per job with scrape_native_histograms (CHANGELOG).

Rules and alerts

Recording rules precompute expensive expressions into new series. Alerting rules evaluate every evaluation_interval; once true for the for duration they go to Alertmanager, which deduplicates, groups, silences and routes them (Alertmanager docs).

Down the stack

Layer 1The contract

  • Best-effort sampling. A sample exists only if a scrape succeeded; missed scrapes leave gaps. Counters make this tolerable because the next sample still carries the cumulative total.
  • Local, single-node durability. An appended sample is in the WAL, but not necessarily fsynced; a crash can lose a short tail of samples. There is no replication: high availability means running two identical servers that scrape the same targets.
  • Query semantics. An instant selector returns, for each series, the latest sample within the lookback window (5 minutes by default) before the evaluation time. Since 3.0, range selectors and the lookback window are left-open: a sample exactly on the lower boundary is excluded, so a [1m] window over a 1-minute scrape can now hold a single sample, too few for rate (migration guide). A series that disappears from a scrape gets a staleness marker and drops out of instant queries immediately.
  • Approximate functions. rate, increase and histogram_quantile are estimates by design.
  • Scale envelope. A single well-provisioned server commonly handles a few million active series and several hundred thousand samples per second. Cloudflare runs 916 servers averaging about 5 million series each, with the largest near 30 million (Cloudflare); hardware, churn and query load move the limit a lot.

Layer 2The mechanism

Head block. New samples go to the head, an in-memory structure covering the newest data. Each series has a 64-bit reference, its label set, and a current chunk. Labels are interned in a symbol table, and an in-memory inverted index maps each label pair to a sorted postings list of series references.

Chunks. A chunk holds up to about 120 samples of one series (30 minutes at a 15 s scrape, Vernekar), compressed in the style of Facebook's Gorilla (Pelkonen et al., VLDB 2015): timestamps as delta-of-delta (a regular 15 s interval costs about 1 bit per sample), values as the XOR with the previous value with leading and trailing zeros elided. Typical cost is around 1.3 bytes per sample. Once full, a head chunk is written to chunks_head/ and memory-mapped, so the head keeps only the open chunk per series in RAM.

WAL. Before samples are acknowledged in the head they are appended to the WAL in wal/, in 128 MB segments of 32 KB pages. Record types include Series (new label sets with their references), Samples (reference, timestamp, value) and Tombstones. On restart, Prometheus replays the WAL to rebuild the head; periodic checkpoints drop records for series that no longer exist, keeping replay bounded.

Persistent blocks. Every 2 hours, once the head spans enough time, the oldest 2 hours are compacted into a block directory: meta.json (time range, stats, compaction level), chunks/ (segment files of up to 512 MB), index (symbols, series with their chunk references, and postings) and tombstones (deletions applied lazily). Blocks are immutable. Background compaction merges small blocks into larger ones, up to 10 percent of the retention or 31 days, whichever is smaller, and retention deletes whole blocks past --storage.tsdb.retention.time (15 days by default).

Query evaluation. For rate(http_requests_total{job="api"}[5m]) the engine looks up the postings for __name__="http_requests_total" and job="api", intersects the sorted lists, then for each series reads only chunks that overlap the window, from the head or from the blocks covering that time range. A range query repeats this at every step. rate walks the samples, adds back each counter reset (a drop means the counter restarted, so the previous value is added), and extrapolates the increase to the window edges, limited so it never extrapolates below zero.

Layer 3The machine

Memory is dominated by series in the head: labels, postings entries, the open chunk and bookkeeping, often quoted at a few KB per active series (hedge: 3 to 8 KB depending on label sizes). Three million active series therefore need roughly 10 to 25 GB of RAM before queries. Churn makes it worse: every pod restart in Kubernetes creates new series with a new pod label, and old ones stay in the head until the next compaction.

Disk writes are sequential: WAL appends and occasional block writes, so a modest SSD handles hundreds of thousands of samples per second. Reads of persistent blocks go through mmap, so a hot query range lives in the page cache and a cold one costs page faults on the index and chunks. A dashboard querying 30 days across a million series can touch gigabytes of chunks; the query engine bounds this with --query.max-samples (50 million by default) and a timeout (Prometheus flags).

Prometheus 3.0 also sets GOMAXPROCS and GOMEMLIMIT from the container's CPU quota and memory limit, so the Go runtime collects garbage harder before the cgroup OOM killer fires, instead of sizing itself from the host.

Scrape parsing is real CPU: a target with 10,000 series sends roughly 1 MB of text per scrape, so 1,000 such targets at 15 s means about 70 MB/s to parse.

Why it's built this way

  1. Monitoring must work when everything else is broken, so each server is standalone with local disk and no dependency on a cluster or consensus.
  2. Metrics arrive as many small appends to millions of series, so writes go to an in-memory head with a sequential WAL, not random disk updates.
  3. Monitoring data is huge and very regular, so Gorilla-style compression turns 16 raw bytes per sample into about 1.3.
  4. Time-ranged queries and retention by age dominate, so data is partitioned into immutable time blocks that are cheap to read, merge and delete.
  5. Scrapes are periodic and can be missed, so counters are cumulative and rate tolerates resets and gaps at the price of being approximate.

Diagram

flowchart LR
    T["Targets /metrics"] -->|"scrape every 15s"| S["Scrape loop"]
    S --> A["Appender"]
    A --> W["WAL segments"]
    A --> H["Head block (in memory)"]
    H -->|"full chunk"| M["chunks_head (mmap)"]
    H -->|"every 2h"| B["Block: index + chunks"]
    B -->|"compaction"| BB["Larger blocks"]
    Q["PromQL engine"] --> P["Postings intersect"]
    P --> H
    P --> B
    Q --> R["Rules and alerts"]

Worked example

A platform runs 2,000 pods, each exposing about 1,500 series, scraped every 15 s.

Active series: 2,000 × 1,500 = 3 million. Ingestion: 3,000,000 / 15 = 200,000 samples per second. Disk: 200,000 × 86,400 s × about 1.3 bytes is roughly 22 GB per day, so 15 days of retention is about 340 GB plus index overhead. Memory: at around 4 KB per series, about 12 GB for the head, so a 32 GB server leaves headroom for queries and the page cache. Run two of them for HA.

Now a developer adds a customer_id label to a request-latency histogram with 12 buckets on 200 pods, with 20,000 active customers. Each pod could emit 20,000 × 14 series (12 buckets plus sum and count), up to 56 million new series. Memory needs grow by an order of magnitude and the server OOMs, losing all monitoring at the moment you need it. The fix is a cardinality budget: per-scrape sample_limit, alerts on prometheus_tsdb_head_series, and per-customer data in logs or traces instead.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Single Prometheus pairSimple, fast, no dependenciesLocal retention only, manual sharding, no global viewOne cluster, weeks of retention
Prometheus plus ThanosObject-storage retention, global query, dedup of HA pairsMore components to runMany clusters, months to years of data
Mimir or CortexHorizontally scalable, multi-tenant remote writeSignificant operational complexityCentral metrics platform for many teams
Managed (AMP, Grafana Cloud, Datadog)No operationsPer-series cost, vendor lock-inSmall platform team
Push systems (StatsD, OTLP to a collector)Works for short-lived jobs and serverlessHarder to know a target is downEphemeral workloads, mixed telemetry

Pitfalls in production

Leadership lens

Check yourself

  1. Why does adding one label with 1,000 values sometimes increase memory 1,000-fold rather than slightly?

    Show answer

    Each unique label combination is a new series with its own head state, so a label multiplies the series count of every metric it is attached to.

  2. A counter goes 100, 110, 5, 15 within a window. What increase does rate assume, and why?

    Show answer

    25: it treats the drop to 5 as a restart from zero, adding back the 110 before the reset, then extrapolates to the window edges.

  3. Why can Prometheus lose recent samples on a crash even though it has a WAL?

    Show answer

    WAL writes are buffered in pages and not fsynced per sample, so the last unflushed data can be lost; durability is traded for ingestion throughput.

  4. Why are persistent blocks immutable and 2 hours long?

    Show answer

    Immutable time-partitioned blocks make reads lock-free, compaction a merge, and retention a directory delete, while 2 hours bounds the in-memory head.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →