Engineering Atlas › System Design Fundamentals

Instrumenting code to emit metrics

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The data model

A series looks like http_requests_total{service="checkout", method="POST", status="500"}. Each series is a stream of (timestamp, value) samples, typically every 10 to 60 seconds. Time series databases compress these extremely well: Gorilla-style encoding stores timestamps as delta-of-deltas and values as XOR against the previous value. Facebook's Gorilla paper measured 1.37 bytes per point against 16 raw (Gorilla, VLDB 2015), and Prometheus averages 1 to 2 bytes per sample (Prometheus storage). Name metrics in base units (seconds, bytes) and end counters in _total (Prometheus naming).

Push versus pull

In the pull model (Prometheus), the server scrapes an HTTP endpoint on every target and notices when one is down. In the push model (StatsD, OTLP to a collector), the application sends metrics outward, which suits short-lived jobs and serverless. The line has blurred: since 3.0 (November 2024) Prometheus can itself receive OTLP pushes on /api/v1/otlp/v1/metrics once started with --web.enable-otlp-receiver (Prometheus OTLP guide).

Push brings a second difference: temporality. Prometheus counters are cumulative (every sample repeats the same start time), while OpenTelemetry SDKs may send delta points that each cover only the last interval (OTel metrics data model). A delta point that gets lost is lost data rather than a wider interval, and Prometheus needs a conversion option to ingest deltas (Prometheus feature flags).

Counters and rates

A counter is a monotonically increasing total since process start. You query its rate over a window: rate(http_requests_total[5m]). Counters survive scrape loss gracefully, since a missed scrape just lengthens the interval, and the query engine handles resets when a process restarts.

Histograms and percentiles

A histogram keeps a counter per bucket (le="0.1", le="0.25", and so on) plus a sum and count. To compute p99 across 200 instances, sum the bucket rates across instances, then interpolate within the bucket where the 99th percentile falls. Accuracy depends on bucket boundaries, so put boundaries near your SLO thresholds. Precomputed client-side summaries give exact per-instance quantiles but cannot be combined across instances (Prometheus histograms). Native (exponential) histograms remove the bucket-choice problem: boundaries grow by a fixed factor, with 2^scale buckets per power of two, so a higher scale means finer resolution (OTel metrics data model). They were experimental for years and became stable in Prometheus 3.8; from 3.9 the feature flag does nothing and each scrape config must set scrape_native_histograms: true (Prometheus native histograms).

Cardinality

Total series is roughly the product of each label's distinct values. method (5) times status (10) times route (50) times 300 pods is 750,000 series for one metric. Add user_id and it becomes unbounded. Each active series costs memory in the TSDB (a few KB in Prometheus), so cardinality, not sample count, is what crashes metric systems. Prometheus's own guidance is never to put unbounded values such as user IDs or email addresses in labels (Prometheus naming).

Down the stack

Layer 1The contract

A metrics system promises that a series is identified exactly by its name and label set, that samples for a series are stored in timestamp order at the scrape or push resolution, and that queries over a range return values computed from those samples. It does not promise every event is counted: a sample summarises an interval, and a scrape that fails leaves a gap. Counter semantics promise that rate() is correct across process restarts, because a drop in value is treated as a reset. Recent data is typically queryable within one scrape interval.

Layer 2The mechanism

Prometheus keeps the last two to three hours in a head block in memory. Each series has an entry in an in-memory index mapping label pairs to series IDs through postings lists, and each series appends samples to a chunk of up to 120 samples (or two hours) (Vernekar, Prometheus TSDB), compressed with the Gorilla scheme: the timestamp is stored as a delta-of-delta (zero bits when the scrape interval is regular), and the value as an XOR with the previous value, storing only the meaningful bits. Since v2.19, full chunks are written to disk and memory-mapped, so only the open chunk per series stays in RAM. All incoming samples are also appended to a write-ahead log in 128 MB segments for crash recovery (Prometheus storage), and local retention defaults to 15 days. Every two hours the head is cut into an immutable block directory with a chunks/ folder, an index file, and meta.json, and older blocks are compacted into larger ones. A query resolves label matchers by intersecting postings lists, then decodes only the chunks overlapping the time range.

Layer 3The machine

The XOR scheme exploits how floats change slowly: if a counter goes from 10,452 to 10,460, the XOR has few significant bits, often encoded in under 2 bytes, compared with 16 bytes for a raw timestamp and float pair. At about 1.3 bytes per sample, a million series at 15-second resolution (5,760 samples each per day) take roughly 7.5 GB per day on disk. Memory is the binding resource: each active series costs its label strings, index postings, and an open chunk, roughly 3 to 8 KB, so 10 million active series need tens of GB of RAM regardless of how few samples they hold. WAL writes are sequential and batched, so ingestion of hundreds of thousands of samples per second needs only modest disk throughput, but WAL replay after a crash reads every segment and can take minutes on large heads.

Why it's built this way

  1. Operators need cheap, fast answers to "how much, how fast" across fleets, so data is aggregated into numeric samples instead of stored as events.
  2. Consecutive samples are similar, so delta and XOR encoding shrink them by roughly ten times.
  3. Recent data is queried most and changes constantly, so it lives in memory with a WAL for durability.
  4. Each unique label set needs its own index entry and open chunk, so cardinality, not sample volume, sets the cost.

Diagram

flowchart LR
  APP["Service with client library"] -->|"/metrics endpoint"| SCR["Scraper every 15s"]
  JOB["Batch job"] -->|"push OTLP"| COL["Collector"]
  COL --> SCR
  SCR --> TSDB["TSDB: head block in memory"]
  TSDB --> BLK["Compressed blocks on disk"]
  BLK --> LTS["Long-term store, downsampled"]
  TSDB --> QE["Query engine: rate, histogram_quantile"]
  QE --> DASH["Dashboards and alerts"]

Worked example

A checkout service runs 300 pods and serves 12,000 requests per second. The team instruments RED metrics:

REQS = Counter("http_requests_total", "Requests", ["route", "method", "code"])
LAT = Histogram("http_request_seconds", "Latency", ["route"],
                buckets=[.025, .05, .1, .25, .5, 1, 2.5])

def handle(route, method, fn):
    with LAT.labels(route).time():
        code = fn()
    REQS.labels(route, method, str(code)).inc()

Cardinality: 40 routes, 3 methods, 8 status codes gives 960 counter series per pod, though only about 150 combinations occur in practice. The histogram has 40 routes times 10 series (8 buckets including +Inf, plus sum and count), so 400. Per pod roughly 550 series, times 300 pods is 165,000 series. At 15-second scrapes that is 11,000 samples per second, trivial for one Prometheus server.

A developer proposes adding customer_id (2 million values) to the counter. Even if each pod only sees 20,000 customers a day, that is 6 million new series per day, enough to exhaust memory. The answer is to keep per-customer analysis in logs or traces, or a dedicated analytics store.

The SLO is "99 percent of requests under 250 ms". Because 0.25 is a bucket boundary, the fraction of fast requests is exact: sum(rate(bucket{le="0.25"})) divided by sum(rate(count)). No interpolation needed.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Pull (Prometheus scrape)Detects down targets, simple clients, central controlNeeds service discovery; awkward for short jobsLong-running services in Kubernetes
Push (OTLP, StatsD)Works for serverless, batch, and behind NATSilent when a source dies; needs aggregation tierEphemeral workloads
Fixed-bucket histogramsAggregatable, cheapAccuracy limited by bucketsLatency SLOs with known thresholds
Client-side summariesExact per-instance quantilesCannot aggregate across instancesSingle-instance diagnostics only
High-cardinality events (wide events)Arbitrary slicing by any fieldStorage and query cost much higherDebugging unknown unknowns alongside metrics

Pitfalls in production

Leadership lens

Check yourself

  1. Why are counters preferred over gauges for request counts?

    Show answer

    A counter's rate can be computed over any window and missed scrapes only widen the interval, whereas per-interval gauges lose whatever happened between missed samples.

  2. Why can you not average p99 values across instances?

    Show answer

    Percentiles are not linear; the fleet p99 depends on the full distribution, so you must combine bucket counts and compute the quantile from the merged histogram.

  3. What determines the memory cost of a metric in a TSDB?

    Show answer

    The number of active series, which is the product of distinct values across its labels, not the number of samples.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →