Engineering Atlas › System Design Interviews

Design a payment processing system

21 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional:

Non-functional:

Out of scope: building a card network or acquirer, fraud model internals (a risk service is called), currency conversion pricing, tax, and subscription billing logic.

Back-of-envelope

Assume a large marketplace with 50 million payments per day. Treat this as a planning number.

Throughput: 50M / 86,400 is about 580 payments per second on average. Shopping peaks (a sale event) can be 10x, so design for about 6,000 per second at peak. Each payment makes 2 to 3 external calls over its life (authorize, capture, maybe refund) and several internal writes.

Ledger volume: each payment produces about 3 ledger transactions (authorization hold, capture with fee split, later payout) with an average of 3 entries each, so roughly 9 entries per payment, or 450M entries per day. At about 200 bytes per entry including indexes, that is 90 GB per day, 33 TB per year and about 165 TB over five years, before replication. Payments rows: 50M x 2 KB = 100 GB per day, about 180 TB over five years. Both are large but well within a sharded relational database or a purpose-built ledger store.

Idempotency records: 50M per day plus internal retries, about 1 KB each with the stored response, 50 GB per day. Keep them for 24 hours to 7 days; retention, not volume, sets the size (a 7-day window is about 350 GB).

Bandwidth: requests are small (a few KB), so even at 6,000 per second the API carries tens of MB/s. Bandwidth is not a concern; database write latency and PSP latency are.

API

POST /v1/payments
  headers: Idempotency-Key: <uuid>
  body: { order_id, amount_minor: 12999, currency: "USD",
          payment_method_token, merchant_id, capture: "auto" | "manual" }
  -> 201 { payment_id, status: "AUTHORIZED" | "CAPTURED" | "PENDING" | "FAILED" }

POST /v1/payments/{id}/capture   headers: Idempotency-Key  body: { amount_minor }
POST /v1/payments/{id}/refunds   headers: Idempotency-Key  body: { amount_minor, reason }
GET  /v1/payments/{id}           -> { payment_id, status, amount_minor, captured_minor, refunded_minor }

POST /internal/psp-webhooks/{psp}   (signed; idempotent by PSP event id)

Amounts are integers in minor units. Floating point is never used for money.

Idempotency-Key is a convention, not a standard: the IETF draft for it expired at revision 07 (October 2025) without becoming an RFC (IETF datatracker), so document your own retention window and conflict responses.

Data model

High-level design

flowchart LR
  CO["Checkout service"] --> API["Payment API"]
  API --> IDK["Idempotency store"]
  API --> RISK["Risk service"]
  API --> PS["Payment state machine"]
  PS --> PDB["Payments DB + outbox"]
  PS --> PSP["PSP adapter"]
  PSP --> EXT["External PSP"]
  EXT --> WH["Webhook receiver"]
  WH --> PS
  PS --> LED["Ledger service"]
  LED --> LDB["Ledger DB"]
  PDB --> EV["Event log"]
  REC["Reconciliation"] --> LDB
  REC --> SF["PSP settlement files"]
  PO["Payout scheduler"] --> LED

Write path: checkout calls POST /payments with an idempotency key. The API inserts the key row as in progress (a unique constraint makes a concurrent duplicate fail fast and wait or return 409). It calls risk, then creates the payment row in CREATED and a payment_attempt with a fresh PSP idempotency key derived from the payment ID and attempt number, all in one transaction. It then calls the PSP. On success, it moves the payment to AUTHORIZED or CAPTURED, posts the matching ledger transaction, writes an outbox event, and stores the response on the idempotency key, all in one local transaction. On a definite decline it moves to FAILED. On a timeout or 5xx it moves to PENDING and a recovery worker resolves it.

Read path: status reads go to the payments table or a read replica with a version check. Merchant balances come from account_balances, and statements are range scans of ledger entries by account.

Asynchronous path: PSP webhooks arrive, are verified by signature, deduplicated by PSP event ID, and applied to the state machine as transitions, which are no-ops if already applied. Nightly, reconciliation matches each settled PSP transaction with our ledger and flags any mismatch.

Deep dives

Exactly-once effects with idempotency keys

Networks lose responses, so the same request will arrive twice. Three layers of protection compose. First, the client's idempotency key: the first request inserts it, later requests with the same key and same body return the stored response, and a different body returns an error, which catches client bugs. If the first request is still running, the duplicate gets a 409 or waits. Second, a business-level uniqueness constraint on (merchant_id, order_id) stops two different keys from charging the same order. Third, the PSP's own idempotency key on every outbound call, deterministic per attempt, so our retry of a timed-out authorize does not create a second authorization. Keys expire after a window agreed with clients, typically a day: Stripe may prune keys once they are at least 24 hours old, and a key reused after pruning is treated as a new request (Stripe API docs). The key insert and the result must be in the same database as the payment so there is no gap where one exists without the other; AWS states the same rule for EC2's client tokens, where recording the token and creating the resources are all or nothing (AWS Builders' Library).

Decide precisely what gets stored. Stripe saves the response once the endpoint starts executing, so a retry replays even a saved 500; a request that failed validation or collided with a concurrent one on the same key saved nothing and may simply be retried. Use time-ordered keys (ULID or UUIDv7) for the key table's primary index: Shopify measured a 50 percent drop in INSERT duration after moving from random UUIDv4 keys to ULIDs, because random keys scatter writes across B-tree pages (Shopify Engineering).

The unknown outcome and the state machine

A payment moves through CREATED -> AUTHORIZED -> CAPTURED -> (PARTIALLY_)REFUNDED, with FAILED, CANCELED and PENDING on the side. PENDING means we sent a request and do not know the result. Never mark it failed and let the user retry with a new key, because the first charge may have succeeded. Resolve it by querying the PSP with the same reference, retrying with the same PSP idempotency key, or waiting for the webhook. A recovery worker scans PENDING rows older than a few seconds with exponential backoff. Transitions are compare-and-set on version, so a webhook and the recovery worker racing on the same payment cannot both apply. If the order is canceled while a payment is pending, schedule a void or refund once it resolves rather than assuming it never happened.

AUTHORIZED is not a resting state either, because a hold expires. For online card payments the hold usually lasts 7 days, and Visa merchant-initiated authorizations last only about 4 days 18 hours; an uncaptured authorization then lapses and the funds go back to the customer. A partial capture releases the rest, and most card payments allow only one capture (Stripe docs). So the state machine needs an EXPIRED transition, and a sweeper must capture or void every authorization before its deadline, which you store on the payment row.

The double-entry ledger and hot accounts

Single-entry "balance += amount" loses history and allows silent drift. In double-entry, a $100 capture with a $3 fee posts one transaction: debit psp_clearing 100, credit merchant_payable 97, credit platform_fees 3. The entries sum to zero, so the sum over all accounts is always zero; any non-zero total is a bug detected instantly. Corrections are reversing transactions, never updates. The ledger post is idempotent by its own key (the payment ID plus event type), so replaying the event log cannot double-post.

Concurrency is the hard part: a large merchant's payable account receives thousands of credits per second, and updating its balance row serialises on a lock. Options are optimistic version checks with retry (works at low contention), sharding the hot account into N sub-accounts and summing on read, or buffering credits and posting them in batches every second. Debits that must not overdraw (payouts, refunds from merchant balance) still check the aggregate balance, so the usual choice is: credits go to sub-accounts without contention, and debits take a lock on the aggregate or go through a single-writer queue per account.

Variant: digital wallet (stored value and transfers)

A wallet changes the shape of the problem. The payment system above moves money between a customer's card and a merchant through an outside processor. A wallet holds the money itself: users top up, pay each other and merchants, and withdraw, so the platform's own ledger is the truth and no processor sits in the middle to settle disagreements.

Requirements and numbers. Every user is both payer and payee, and a balance must never go below zero (except for system accounts). Planning numbers for a large regional wallet: 60M daily actives making about 2.5 transfers each is 150M a day, roughly 1,700 per second on average and 17,000 to 20,000 at a payday or campaign peak. Balance reads run about ten times higher, so they come from the owning shard's memory or a replica with a version check. Each transfer writes about 700 bytes, roughly 105 GB of ledger a day.

Cross-shard transfers. With wallets partitioned by wallet_id, the sender and receiver rarely share a database. Two-phase commit would hold row locks on both wallets across a network round trip and leave them locked if the coordinator dies. The fix is a reservation protocol, a saga whose first step is cheap to undo:

  1. Reserve: place a hold on the sender, lowering the spendable amount without moving money.
  2. Credit: apply the credit to the receiver, which is the point of no return.
  3. Capture: turn the hold into a posted debit.

Each step is idempotent on one transfer_id, so a crashed coordinator is simply replaced and re-runs the transfer. A sweeper releases holds that expire, but never one whose transfer already reached credited. In saga vocabulary the hold is a semantic lock, the credit is the pivot and the capture is a retryable step (Azure saga pattern). TigerBeetle models the same idea natively: a pending transfer raises debits_pending, and a later transfer posts, voids or times it out exactly once (TigerBeetle two-phase transfers).

Balance versus ledger. A stored balance column is updated in the same local transaction as the entry. Each entry records balance_after, a nightly job re-folds entries per wallet, and a global check confirms posted amounts sum to zero and held totals match pending holds.

Hot wallets. One row lock held until durable commit caps a wallet near 1,000 changes a second per millisecond of commit time. A popular merchant takes credits into N sub-wallets chosen by hashing transfer_id, summed for display; debits such as refunds and payouts go through a single-writer queue. Unlike the merchant accounts above, here every user is also a payer, so every reserve must check holds on that wallet.

Down the stack

Layer 1The contract

For a given (client_id, idempotency_key), the system performs the effect at most once and returns the same response every time. Every acknowledged state change is durable before the response is sent. Ledger transactions are atomic (all entries or none), balanced to zero per currency, immutable once committed, and accounts flagged non-negative never go below zero. Events are delivered at least once, so consumers dedupe by event ID. External PSP state may diverge from ours temporarily, but reconciliation detects every divergence within one settlement cycle.

Layer 2The mechanism

The core is one relational transaction: INSERT the idempotency key, UPDATE payments SET status=..., version=version+1 WHERE payment_id=? AND version=?, INSERT ledger transaction and entries, UPDATE account_balances ... WHERE balance + delta >= 0 for guarded accounts, INSERT into outbox, then COMMIT. A unique index on the idempotency key enforces at-most-once; the conditional update enforces state machine order; the check constraint or conditional update enforces non-negative balances. The outbox relay reads committed rows in order and publishes them, recording its position, so events are never published for rolled-back changes. The relay can still crash after publishing and before recording its position, so it republishes on restart and consumers must dedupe (microservices.io). Specialised ledgers such as TigerBeetle put the same rules (balanced transfers, idempotent IDs, balance limits) into a purpose-built engine: a create_transfers request carries up to 8,189 transfers, which amortises consensus and I/O, and a repeated transfer ID returns exists rather than posting twice (TigerBeetle docs).

Layer 3The machine

The commit is the latency floor: Postgres or MySQL writes the transaction's records to the write-ahead log and calls fsync (or fdatasync) before acknowledging. On NVMe with power-loss protection that is roughly 20 to 100 microseconds; on cloud network block storage it is often 0.5 to 2 ms. Synchronous replication to a standby in another zone adds a network round trip of about 1 ms. Group commit amortises one fsync over many concurrent transactions, which is why throughput scales while per-transaction latency stays flat. In Postgres this happens even with commit_delay at its default of zero, because sessions that arrive during an in-progress flush join the next one (PostgreSQL docs). A hot balance row is a single 8 KB heap page under a row lock; every writer queues on it, so throughput on one account is bounded by roughly one commit latency per update, a few hundred to a few thousand per second. External PSP calls take hundreds of milliseconds over TLS, so they must never run while holding a database transaction or row lock.

Why it's built this way

  • Networks drop responses, so every operation must be safely repeatable; hence idempotency keys at every boundary.
  • Remote outcomes can be unknown, so state must be persisted before and after each external call and resolved later.
  • Money must be explainable years later, so history is append-only and balances are derived.
  • Bugs are inevitable, so a zero-sum invariant and independent reconciliation detect them quickly.
  • Durable commits cost an fsync and locks serialise, so contention is designed away with sub-accounts and batching rather than bigger machines.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Synchronous charge in checkoutSimple UX, immediate resultCheckout latency tied to PSPCard payments with fast PSPs
Async with status pollingResilient to PSP slownessMore complex client and UXBank transfers, slow methods
Relational ledger tablesFamiliar, transactional, flexibleHot-row contention, manual shardingMost companies, early stage
Purpose-built ledger engineVery high transfer throughput, built-in invariantsNew operational skill setVery high volume ledgers
Single PSPSimple integrationSingle point of failure, weak pricing leverageEarly stage
Multiple PSPs with routingAvailability, cost optimisationReconciliation across providersLarge volume, global reach

Leadership lens

Check yourself

  1. An authorize call to the PSP times out. Why is it wrong to mark the payment FAILED and let the customer retry?

    Show answer

    A timeout means the outcome is unknown, and the first charge may have succeeded, so a fresh retry with a new key risks charging twice. The payment goes to PENDING and is resolved by querying the PSP, retrying with the same PSP idempotency key, or waiting for the webhook.

  2. One merchant's payable account receives thousands of credits per second. What are the options, and what is the usual choice?

    Show answer

    Optimistic version checks with retry (fine at low contention), sharding the account into sub-accounts summed on read, or batching credits every second. The usual choice is to send credits to sub-accounts without contention while debits that must not overdraw take a lock on the aggregate or go through a single-writer queue.

  3. Why do idempotency records cost about 350 GB rather than growing without bound, given 50M payments a day?

    Show answer

    At about 1 KB each with the stored response that is roughly 50 GB per day, but keys are kept only for 24 hours to 7 days, so retention sets the size and a 7-day window is about 350 GB.

  4. Why must the idempotency key row live in the same database as the payment, and why must PSP calls never run inside a database transaction?

    Show answer

    Committing the key and the payment result in one local transaction leaves no gap where one exists without the other. PSP calls take hundreds of milliseconds, so holding a transaction or row lock across them would serialise writers behind slow external calls.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →