Engineering Atlas › System Design Interviews

Avoiding duplicate processing

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Where duplicates come from

A client sends POST /payments, the server charges the card, and the response is lost on the way back. The client times out and retries. A message consumer processes a message, writes to the database, and crashes before committing its offset; on restart it reprocesses the message. A user taps "Order" twice. In every case the sender cannot distinguish "it never happened" from "it happened and I did not hear".

Idempotency keys at the API

The client generates a UUID per logical operation (not per attempt) and sends it in a header such as Idempotency-Key. The server:

  1. Inserts a row (key, request_fingerprint, status=IN_PROGRESS) with a unique constraint on the key. If the insert fails, the key exists.
  2. If the existing row is COMPLETED, return the stored response. If it is IN_PROGRESS, return 409 Conflict so the client retries later. If the fingerprint differs, reject with 422: the client reused a key for a different request. These codes come from the IETF's Idempotency-Key header draft, which also suggests 400 when a required key is missing; the draft (version 07, October 2025) has expired without becoming an RFC, so treat it as convention, not standard (IETF draft).
  3. Performs the operation and, in the same transaction where possible, stores the response and marks the row COMPLETED.
  4. On a failure that is safe to retry, deletes or resets the row so the next attempt can proceed.

Keys expire after a window long enough to cover client retries (often a day or more), so the table does not grow forever.

Decide which outcomes you store. Stripe's v1 API saves the status and body once the endpoint starts executing, so a retry replays even a 500; requests that fail validation or collide with a concurrent request are not saved and can be retried. Keys are at most 255 characters, are pruned after at least 24 hours, and a reused key with different parameters is an error (Stripe API docs). Stripe's newer v2 API changes the rule: a retry finishes failed or partial work instead of replaying the failure. AWS goes further on retention: EC2 honours a ClientToken for the lifetime of the resource plus an interval, so a very late retry still gets an answer that means the same as the first one, not a confusing "already exists" (AWS Builders' Library).

Idempotent consumers

For message processing, the consumer stores processed message IDs in a table inside the same transaction as its business writes (microservices.io): INSERT INTO processed(message_id) ...; UPDATE accounts ...; COMMIT. A redelivered message hits the unique constraint and is skipped. Two details are easy to miss. Key on an ID the producer chose, never on a delivery or receipt ID the broker regenerates on redelivery. And if several consumer groups share one dedup table, key it on (consumer, message_id), or the first consumer to see a message suppresses it for all the others (Azure Architecture Center). Or make the write itself idempotent: an upsert keyed by the event's natural ID, or a conditional update on a version.

When the side effect is external

If the side effect is an email or a call to a payment provider, you cannot put it in your database transaction. Pass your idempotency key downstream (most payment APIs accept one), or record intent first and have a separate worker perform the call idempotently, or accept a small duplicate risk where the impact is low (a duplicate push notification).

Large-scale stream dedup

For billions of events (ad clicks, analytics), pipelines keep IDs only for a bounded time window, or use a Bloom filter that may wrongly flag a few new events as duplicates but never misses a real one. Stream processors with transactional sinks commit output and input offsets atomically.

How it shows up in interviews

Triggers: design a payment system, an order service, Uber ride requests, a notification service ("don't send twice"), a job scheduler, an ad click counter, a webhook delivery system. A strong answer says delivery is at-least-once, places idempotency keys at each boundary, explains the atomic record-plus-effect write, handles concurrent duplicates (in-progress state), sets a retention window, and treats external side effects explicitly. Typical probes: two identical requests arrive at the same millisecond on different servers; the server crashes after charging but before recording; the client reuses a key with a different amount; how long do you keep keys and why.

Down the stack

Layer 1The contract

An idempotent endpoint promises that any number of requests with the same key and the same payload, within the retention window, produce the effect of exactly one execution and return the same response. Concurrent duplicates must not both execute: one wins and the others see in-progress or the final result. A request with a reused key and a different payload is rejected, not silently replayed. A deduplicating consumer promises at most one application of each message ID's effect, even across crashes and rebalances.

Layer 2The mechanism

In SQL: INSERT INTO idempotency_keys (key, fingerprint, status, created_at) VALUES ($1, $2, 'IN_PROGRESS', now()) ON CONFLICT (key) DO NOTHING RETURNING key; no returned row means a duplicate, and you SELECT the existing row. The unique B-tree index on key is what serialises concurrent duplicates: the second inserter blocks on the first transaction and then sees the conflict. Kafka's idempotent producer gets a producer ID and epoch from the broker, attaches a per-partition sequence number to each batch, and the partition leader rejects batches whose sequence it has already appended; brokers track recent batch sequences per producer, which is why max.in.flight.requests.per.connection must be at most 5. Since Kafka 3.0 the producer defaults to enable.idempotence=true and acks=all (KIP-679), but a validation bug left idempotence off unless set explicitly until 3.0.1, 3.1.1 and 3.2.0 (KAFKA-13598), so older clients should set it by hand. Kafka transactions add a transactional.id, a coordinator writing to the internal __transaction_state topic, and commit markers in each partition, so a consumer reading with isolation.level=read_committed sees only committed output, and consumed offsets are committed in the same transaction. SQS FIFO queues drop messages with a repeated MessageDeduplicationId sent within a 5-minute interval; with content-based deduplication the ID is a SHA-256 of the message body only, so two messages that differ only in attributes count as duplicates (AWS SQS docs).

Layer 3The machine

The dedup record is only as durable as its commit: Postgres considers a transaction committed once its WAL record is flushed with fsync (or fdatasync) to stable storage, which costs from tens of microseconds on NVMe with power-loss protection to several milliseconds on cloud network disks. The unique index lookup is a B-tree descent of 3 to 4 pages, usually in the buffer cache, so the check costs microseconds; the expensive part is the write and its WAL. Random UUIDv4 keys scatter inserts across the index and cause page splits and cache misses at high rates; time-ordered IDs (UUIDv7, ULID) keep inserts on the right edge of the index. Shopify measured a 50 percent drop in INSERT duration after moving idempotency keys from UUIDv4 to ULID (Shopify Engineering). A Bloom filter for 1 billion IDs at a 1% false-positive rate needs roughly 1.2 GB (about 9.6 bits per element), small enough to hold in memory on one machine.

Why it's built this way

Senders cannot tell a lost request from a lost reply, so retries and duplicates are unavoidable. Detecting duplicates requires remembering what has happened, so a durable record keyed by an ID chosen before the first attempt is needed. A crash can fall between any two writes, so the record and the effect must share one atomic commit. Memory and disk are finite, so records expire after a window that covers realistic retry horizons.

Diagram

sequenceDiagram
    participant C as Client
    participant S as Payment API
    participant DB as Database
    participant P as Card provider
    C->>S: POST /payments (Idempotency-Key K)
    S->>DB: insert K IN_PROGRESS (unique)
    S->>P: charge (provider key K)
    P-->>S: charged
    S->>DB: store response, K COMPLETED
    S--xC: response lost
    C->>S: retry with key K
    S->>DB: insert K fails, read row
    S-->>C: stored response, no second charge

Worked example

A food-delivery app processes 3,000 orders per second at dinner peak. Mobile networks are flaky: about 1% of requests time out on the client after the server has already succeeded, and the app retries up to 3 times over 30 seconds. Without dedup, roughly 30 duplicate orders per second would reach restaurants.

The app generates an idempotency key when the user taps "Place order" and reuses it for retries. The order service writes the key row and the order row in one Postgres transaction. Keys are kept for 48 hours: 3,000 per second at peak, perhaps 100 million per day, about 200 million rows, at roughly 200 bytes each including the stored response and index, around 40 GB. Expired keys are dropped by day partition rather than a giant DELETE.

The order service then emits an OrderPlaced event to Kafka with the order ID as message key. The restaurant-notification consumer keeps a processed_events table and inserts the event ID in the same transaction as it updates the restaurant's queue. When a consumer rebalance replays 2,000 events from the last committed offset, every replayed insert conflicts and is skipped; restaurants see each order once.

Payment uses the same order key as the provider's idempotency key, so even if the order service crashes after charging but before committing, the retry produces one charge.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Idempotency key tableExact, returns the original responseStorage, retention managementAPIs with money or orders
Natural idempotence (upsert, set)No extra stateOnly for operations that can be restatedState updates, not increments
Unique constraint on business keySimple, enforced by the databaseNeeds a natural unique keyOne order per cart, one vote per user
Broker dedup (Kafka EOS, SQS FIFO)Handles producer retries transparentlyLimited windows and scopesPipelines on that broker
Bloom filter or TTL cacheCheap at huge scaleFalse positives or bounded windowAnalytics, clicks, telemetry

Pitfalls in production

Leadership lens

Check yourself

  1. Why can't a sender simply avoid retries to prevent duplicates?

    Show answer

    Without retries, lost requests are never completed; since the sender cannot tell lost requests from lost replies, it must retry and the receiver must dedupe.

  2. Why must the dedup record and the business write commit together?

    Show answer

    If they commit separately, a crash between them either loses the record (allowing a duplicate) or loses the effect (marking undone work as done).

  3. Two identical requests arrive concurrently on different servers. What stops both executing?

    Show answer

    A unique constraint or conditional write on the key; only one insert succeeds, and the other sees the in-progress row.

  4. Why use time-ordered IDs for a high-volume dedup table?

    Show answer

    Sequential keys append to the right side of the B-tree index, avoiding random page splits and cache misses from random UUIDs.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →