Engineering Atlas › System Design Interviews

Design usage metering and billing

19 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope: payment collection and retries on cards, tax calculation, and the rate limiter that protects the API (a separate, approximate mechanism).

Back-of-envelope

API

Producers send batches. Each event has an id the producer generates and keeps stable across retries.

POST /v1/usage-events
Authorization: Bearer <service-token>
{
  "events": [
    {"event_id": "gw-7f3a-000912", "customer_id": "cus_8832", "metric": "llm_tokens_out",
     "quantity": 1840, "timestamp": "2026-03-14T09:15:02.113Z",
     "dimensions": {"model": "large", "region": "eu"}}
  ]
}
-> 207 {"results": [{"event_id": "gw-7f3a-000912", "status": "accepted"}]}
   status is one of: accepted | duplicate | rejected(reason)
GET  /v1/customers/cus_8832/usage?metric=llm_tokens_out&from=2026-03-01&to=2026-03-15&granularity=day
POST /v1/usage-events/adjustments    {"cancel": {"event_id": "gw-7f3a-000912"}, "reason": "double-emitted by gateway bug"}
POST /v1/invoices/preview            {"customer_id": "cus_8832", "period": "2026-03"}
POST /v1/invoices/{id}/finalize      Idempotency-Key: close-cus_8832-2026-03
POST /v1/credit-grants               {"customer_id": "cus_8832", "amount_cents": 50000, "expires_at": "2027-03-01"}

Stripe's meter-event API has the same shape: an event name, a customer in the payload, a numeric value, a caller-chosen identifier, and a timestamp accepted only within the past 35 days or up to 5 minutes ahead (Stripe API). Its adjustment endpoint cancels a single event by identifier and reports pending until processed (Stripe API). A bound on how old an event may be is what makes closing a window possible at all.

Data model

Store money as integers in the smallest unit that matters (cents for amounts, micro-dollars for unit prices) or as Postgres numeric, which is exact, and never as float or double precision, which the Postgres docs describe as inexact (PostgreSQL docs).

High-level design

flowchart LR
  GW["Services / API gateway"] --> BUF["Local buffer + retry"]
  BUF --> ING["Ingest API"]
  ING --> LOG["Durable log (partitioned by customer)"]
  LOG --> AGG["Dedup + window aggregator"]
  LOG --> RAW["Raw event archive (object store)"]
  AGG --> AGGDB["Usage aggregates"]
  AGGDB --> DASH["Dashboards, quotas, spend alerts"]
  AGGDB --> CLOSE["Period close job"]
  PLAN["Price plan versions"] --> CLOSE
  CLOSE --> INV["Invoices + ledger"]
  INV --> PAY["Payment + credit notes"]
  GW --> CNT["Independent counters"]
  CNT --> REC["Reconciliation"]
  AGGDB --> REC
  INV --> REC

Write path. A service emits an event with a stable id into a local buffer (a disk-backed queue), which retries until the ingest API acknowledges. The ingest API validates the schema, rejects timestamps outside the allowed window, and appends to a log partitioned by customer. It acknowledges only after the log has replicated the write. The aggregator reads each partition, drops duplicate ids using local keyed state, and adds the quantity to the hourly window for the event's time. It commits state and offsets together so a restart does not double-add. A second consumer writes the same stream to the raw archive.

Read path. Dashboards and spend alerts read usage_aggregates, which lag by a minute or two and show the current window as provisional. The period-close job runs after the lateness bound has passed, reads aggregates for each customer, rates them against the plan versions in force, writes the invoice and ledger postings in one transaction, and emits an event to payments. Reconciliation runs alongside: it compares independent counters, aggregates, and invoiced quantities, and flags drift beyond a threshold.

Deep dives

1. Idempotent ingestion and duplicate suppression

Duplicates come from three places: a producer retries after a timeout that hid a success, the log redelivers after a consumer restart, and a bug emits an event twice with different ids. Mechanism handles the first two; reconciliation handles the third.

Options for the first two:

Choice: the third. The key is producer-assigned and stable across retries, which the idempotent-consumer pattern depends on (microservices.io) and which Stripe applies to payment requests (Stripe blog). Stripe's meter API enforces identifier uniqueness over a rolling period of at least 24 hours, not forever; Lago's docs describe the same trade-off, with its high-volume pipeline using both transaction_id and timestamp for uniqueness (Lago docs). A retry after the dedup window looks new, so the window must exceed the longest retry horizon. The same id with a different quantity is flagged as a conflict, not silently dropped.

2. Windows, late events and the closing rule

Aggregate by event time, or a backed-up queue bills a midnight burst to the wrong day. That needs a notion of "no more events older than T are expected", which stream systems call a watermark. Flink's bounded-out-of-orderness strategy states it directly: the watermark trails the maximum timestamp seen by a fixed delay (Flink docs). The Dataflow paper adds the key insight that a watermark is a heuristic, so a window can fire again for late data, with accumulating or retracting output (Akidau et al., VLDB 2015).

Billing adds a twist: the invoice has a hard close. The scheme has three clocks:

  1. Dashboard windows stay open and update as late data arrives. Nobody is charged yet.
  2. The invoice close happens after a lateness bound, for example 48 hours past the period end, which is also the maximum event age the API accepts.
  3. Post-close late events (a mobile client offline for a week, a replayed backfill) are still stored, marked late, and billed as an adjustment line on the next invoice.

Adjustment lines should be re-rating deltas. Rate the whole original period with the late usage included, subtract what was already billed, and bill the difference. Rating the late quantity alone is wrong under graduated tiers, because its tier depends on everything before it. The practice problem implements this.

3. Rating, credits, and money correctness

Rating is a pure function: (plan version, aggregated quantity) -> line amount. Keep it pure, versioned and re-runnable, so an auditor or a support engineer can reproduce any line.

Down the stack

Layer 1The contract

  • Exactly-once effect, at-least-once delivery. An event with a given (customer_id, event_id) contributes to at most one invoice line, once. A retry within the dedup window returns duplicate and changes nothing; the same id with a different payload returns a conflict.
  • Bounded staleness. Dashboards include events acknowledged more than 2 minutes ago. Invoices include every in-period event acknowledged before the close; later arrivals are billed as adjustments within one cycle.
  • Reproducibility. Re-running the close for a period with the same raw events and plan versions yields byte-identical lines. Finalised invoices are never edited; corrections are new documents that reference the old one.
  • Acknowledgement means durable. Success is returned only after the log has replicated the batch.

Layer 2The mechanism

  • Ingest. Batches of up to 500 events are appended to a replicated log partitioned by hash(customer_id), so one aggregator owns each customer and dedup stays local.
  • Aggregator state machine. For each event: look up (customer, event_id) in the keyed store; if present, count a duplicate and stop; else add the id with a 48-hour TTL, add the quantity to the (customer, metric, hour) cell, and emit the updated cell. Offsets and state commit atomically, which is the same exactly-once-by-checkpoint idea stream engines use. Kleppmann describes this reliance on idempotence and atomic commit as the way stream processors get exactly-once effects (Designing Data-Intensive Applications, 1st ed., ch. 11).
  • Window closing. A watermark per partition equals the maximum event time minus the lateness bound. The close job requires watermark ≥ period end plus bound for every partition of the customer, so a stalled partition delays invoices instead of producing short ones.
  • Rating. Tiers are rows (up_to, unit_price_micros). Graduated rating walks them, taking min(remaining, up_to - previous_cap) units in each. Everything is int64, whose maximum of 9.22 × 10^18 micro-dollars is about $9.2 trillion.
  • Close transaction. One database transaction inserts the invoice, lines, ledger postings, credit drawdowns and an outbox row for the payment event.

Layer 3The machine

  • Durability cost. A replicated log append waits on an fsync or a replica quorum, around 1 to 5 ms in a zone. A batch of 500 events amortises that to microseconds per event, and 35 MB/s is trivial for sequential writes on NVMe, which sustains over 1 GB/s.
  • State store. Dedup state in an embedded LSM tree on local SSD takes a write to the memtable plus a sequential write-ahead log append per new id, and a point read, mostly served by Bloom filters and block cache, for each lookup.
  • Clocks. Event timestamps come from many machines. NTP keeps most hosts within tens of milliseconds, but a drifting host can be seconds or minutes off. Stamp ingest time too, and reject events outside the accepted window, or one wrong clock moves usage across a month boundary.
  • Arithmetic. An IEEE 754 double has 53 bits of mantissa, so integers are exact only to 9.0 × 10^15, and 0.1 + 0.2 ≠ 0.3. Float sums depend on addition order, which differs between a streaming run and a batch rebuild. Integer sums do not.

Why it's built this way

  1. A lost acknowledgement can only be answered by retrying, so duplicates exist and idempotency keys come first.
  2. Clocks differ and queues back up, so billing uses event time, whose watermark is a guess. A lateness bound and an adjustment path are therefore part of the product.
  3. Customers dispute bills, so raw facts are immutable, aggregates are rebuildable, and invoices are corrected by new documents.
  4. Money must add up to the cent, so integers, one rounding step and a double-entry ledger that fails loudly.
  5. A counter cannot see its own bugs, so an independent count reconciles before money moves.

Failure modes

In the wild

Trade-offs

DecisionOption AOption BChoose when
Dedup locationStream processor state with TTLUnique key in a databaseA for volume with a bounded retry horizon, B when events are low volume and retries can be arbitrarily old
Billing time basisEvent time with lateness boundIngest (processing) timeA when customers dispute usage dates, B only if producers are unreliable and you accept mis-dated usage
Late dataAdjustment line on next invoiceHold invoice open longerA for predictable invoice dates, B for contracts that demand exactness and tolerate delay
Money typeInteger micro/minor unitsDecimal type in the databaseA in code that rates in the pipeline, B in SQL-heavy reporting; never floats
Build or buyOwn pipelineMetering vendor or Stripe-style metersBuy until pricing logic or volume outgrows it

Pitfalls in production

Leadership lens

Check yourself

  1. A producer times out on the ingest call and retries with the same event_id. The first attempt had in fact succeeded. What does the customer get billed, and what stops a double charge?

    Show answer

    One event's worth. The retry carries the same producer-assigned id, so the aggregator's keyed state (and the archive's unique key) reports a duplicate and adds nothing. This only holds within the dedup window and if the id is stable across retries.

  2. A customer on graduated tiers has 15 units billed for March. Three late units for March arrive after the invoice closed. Why not simply rate the three units on their own?

    Show answer

    Tier boundaries depend on cumulative quantity. The late units may fall in a more expensive tier than a fresh start would price them. Re-rate 18 units, subtract the amount already billed for 15, and bill the difference.

  3. Why round once per invoice line, not once per event?

    Show answer

    Per-event rounding builds a systematic bias over millions of events and makes totals depend on batching. One rounding of the exact micro-unit total is deterministic and off by at most half a cent per line.

  4. The aggregator, the invoice job, and the gateway all agree in March, yet customers report being overcharged. Where could the error live and what would find it?

    Show answer

    A service that emitted every event twice with different ids passes all internal checks. An independent count (provider logs, upstream request counters) compared by customer and day exposes the ratio drift. Internal agreement is not proof.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →