Engineering Atlas › System Design Fundamentals

Storing metrics in time series databases

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Data model

A series is a metric name plus a set of label pairs: http_requests_total{service="cart", status="500", pod="cart-7f9"}. Every unique label combination is a new series. Points within a series are ordered by time and usually arrive in order. Two kinds of index matter: an inverted index from label pairs to series ids (to answer "all series where service=cart"), and per-series storage of the points themselves.

Histograms are where cardinality hides. A classic Prometheus histogram is stored as a _sum series, a _count series and one _bucket series per boundary, so 12 buckets times every label combination is a lot of series. Native histograms store the whole distribution as one series with sparse exponential buckets. They became stable in Prometheus 3.8, though scraping them still has to be switched on with scrape_native_histograms (Prometheus spec). Prometheus 3.0 (November 2024), the first major release in seven years, also allowed UTF-8 metric and label names and added Remote Write 2.0 and a native OTLP receiver (Prometheus blog).

Write path

Incoming points land in an in-memory head block, grouped per series, backed by a WAL. Every couple of hours (two hours in Prometheus) the head is cut into an immutable block on disk containing compressed chunks per series and the label index for that window. Background compaction merges small blocks into larger ones, which reduces index overhead for long-range queries; Prometheus caps a compacted block at 10% of the retention time or 31 days, whichever is smaller (Prometheus storage).

Compression

Gorilla-style encoding works on a single series at a time:

Together these get a 16-byte raw point down to about 1.4 bytes on typical monitoring data. Columnar TSDBs on top of general formats get similar results with dictionary, run-length, and delta encodings per column. TimescaleDB (whose company renamed itself Tiger Data in 2025) now calls this the columnstore: convert_to_columnstore() replaced compress_chunk() in version 2.18 (Tiger Data docs). InfluxDB 3 Core and Enterprise went GA in April 2025 on a Rust engine built on Apache Arrow, DataFusion and Parquet (InfluxData).

Query path

A query like sum by (status) (rate(http_requests_total{service="cart"}[5m])) resolves label matchers to series ids via the inverted index, loads only the chunks overlapping the time range, decompresses, applies per-series functions (rate), then aggregates across series. Cost scales with the number of series touched times points per series, so a query that matches 2 million series is slow no matter how well data is compressed.

Retention and downsampling

Because data is partitioned by time, deleting 30-day-old data means deleting whole block directories, which is instant compared with row deletes. Long-term storage usually keeps downsampled data: 5-minute and 1-hour aggregates (min, max, sum, count) that make year-long dashboards cheap.

Down the stack

Layer 1The contract

Prometheus's TSDB stores samples of (millisecond timestamp, float64 value) per series. Appends within a series must be in time order; out-of-order samples are rejected unless out_of_order_time_window is raised from its default of 0s, after which anything newer than the head's max time minus the window is accepted (Prometheus configuration). A sample is queryable as soon as it is appended to the in-memory head. Durability is best effort: the WAL is not fsynced per sample, so a crash can lose the most recent samples, which is acceptable because metrics are periodically re-scraped and the system optimises for ingest throughput rather than per-sample guarantees. A single server is also not clustered or replicated, so the docs say plainly that it is not durable against drive or node loss; that is the job of remote storage. Queries are evaluated at fixed steps with a 5-minute lookback by default, so a series that stops reporting goes stale rather than disappearing instantly.

Layer 2The mechanism

In the head block, each series keeps its labels and an open chunk. A chunk is closed after about 120 samples (around 30 minutes at a 15-second interval) and then memory-mapped from disk, so the head does not hold every closed chunk in RAM. Every append is also written to the WAL, which is split into segment files of 128 MB. Every two hours the oldest head range is compacted into a block directory holding chunks/ (segment files of concatenated chunks), an index file, meta.json, and tombstones. The index contains a symbol table (every label string stored once and referenced by number), series entries that list each series' labels and chunk references, and postings: for every label pair, the sorted list of series ids that carry it. A selector such as {service="cart", status="500"} is answered by intersecting two sorted postings lists, exactly as a search engine intersects terms. Each index section ends in a CRC32 checksum, and the file opens with the magic number 0xBAAAD700 (Prometheus index format).

Inside a chunk, Gorilla encoding applies. The timestamp's delta-of-delta is written with a variable-length prefix code: a single 0 bit when it is zero, and progressively longer prefixes followed by wider fields for larger values. Prometheus uses wider buckets than the original Gorilla paper because its timestamps are in milliseconds. Each value is XORed with the previous one. If the XOR is zero the encoder writes one 0 bit. Otherwise it writes the meaningful bits between the leading and trailing zeros, reusing the previous window's position when the new bits fit inside it, and emitting the leading-zero count and length only when they do not.

Layer 3The machine

A raw sample is 16 bytes. Gorilla's paper reports about 1.37 bytes per sample on Facebook's monitoring data, a 12x reduction (Pelkonen et al., VLDB 2015), and Prometheus documents an average of 1 to 2 bytes per sample, a roughly tenfold saving in both memory and disk. The price is that decoding is strictly sequential and bit-aligned: to read sample 100 you decode samples 1 to 99, and the inner loop is branchy bit manipulation that cannot use SIMD well. Capping chunks at around 120 samples bounds that cost and lets queries skip whole chunks by their time range.

Blocks are read through mmap, so query performance depends on the page cache holding the index and recently used chunks. Memory pressure comes mostly from the head, where every active series costs label storage, an open chunk, and index entries, typically a few kilobytes per series. That is why cardinality, not sample rate, is what exhausts a Prometheus server. Retention is cheap because a block is a directory: expiring 15 days of data is one directory removal per block, with no compaction or rewrite.

Why it's built this way

  1. Metrics arrive as huge numbers of tiny, ordered points, so storage is organised per series where neighbouring values are similar and compress well.
  2. Bit-level compression makes random access impossible, so chunks are kept small and indexed by time range.
  3. Data is written once and only ever queried or expired by time, so it is cut into immutable time-partitioned blocks that can be deleted whole.
  4. Queries select series by labels, so an inverted index of sorted postings resolves selectors by intersection.
  5. Re-scraping makes small losses tolerable, so the WAL trades strict durability for ingest throughput.

Diagram

flowchart LR
    S["Scrapers / agents"] --> W["WAL"]
    S --> H["Head block (RAM, per-series chunks)"]
    H -->|"every 2h"| B1["Block: 2h, compressed chunks + label index"]
    B1 -->|compaction| B2["Block: 12h"]
    B2 -->|compaction| B3["Block: days"]
    B3 -->|"age > retention"| DEL["Delete directory"]
    B3 -->|downsample| DS["5m / 1h rollups"]
    Q["Query"] --> IDX["Label index -> series ids"]
    IDX --> B1
    IDX --> B3

Worked example

A platform team monitors 4,000 Kubernetes pods. Each pod exposes about 500 series, so the system holds 2 million active series. With a 15-second scrape interval, ingestion is about 133,000 points per second.

At 1.5 bytes per point after compression, that is 200 KB/s, or 17 GB per day. Thirty days of local retention needs about 520 GB, which fits on one well-provisioned Prometheus server. Memory is the tighter constraint: each active series costs a few KB in the head block and index, so 2 million series need roughly 8 to 12 GB of RAM.

Then a developer adds a user_id label to a request counter on a service with 500,000 daily users. Each pod now produces hundreds of thousands of new series, and active series jump past 50 million within hours. Head-block memory blows up, the server OOMs, and restarts take minutes because WAL replay must rebuild all those series. The fix is not a bigger server: remove the label, enforce per-target series limits, and send per-user analytics to an event store or OLAP database built for high cardinality.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Prometheus (local TSDB)Simple, fast, pull-based, PromQLSingle node, limited retentionPer-cluster monitoring
Prometheus plus remote storage (Thanos, Mimir, VictoriaMetrics)Global view, long retention, object storageMore moving parts to runMany clusters, months of history
TimescaleDBFull SQL, joins with business tablesHigher write cost than purpose-built TSDBsMetrics mixed with relational data
Columnar OLAP (ClickHouse, Druid)Handles high-cardinality events wellLess natural for rate and counter mathPer-user or per-request analytics
Wide column store with time bucketsMassive scale, multi-region writesYou build compression, rollups, query layerIoT at very large scale

Pitfalls in production

Leadership lens

Check yourself

  1. Why does delta-of-delta encoding work so well for scraped metrics?

    Show answer

    Scrapes arrive at near-constant intervals, so the delta barely changes and its delta is usually zero, which takes about one bit to store.

  2. A TSDB has plenty of disk but keeps running out of memory. What is the likely cause?

    Show answer

    Too many active series; per-series index and head-block memory grow with cardinality regardless of point volume.

  3. Why is retention cheap in a TSDB compared with a row store?

    Show answer

    Data is partitioned into time blocks, so expiring old data means deleting whole files rather than individual rows.

  4. Why do long-range dashboards need downsampled data?

    Show answer

    A year of raw 15-second points is about 2 million points per series; hourly rollups cut that to under 9,000 with little loss for trend views.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →