Making retries safe with idempotency

In one minute

• An operation is idempotent when doing it twice leaves the system in the same state as doing it once. That property is what makes retries safe, and retries are the only general answer to a network that loses requests and responses.
• Some operations are idempotent by nature (set a value, delete by ID, PUT a whole resource). Others (increment, append, charge, send) must be made idempotent by giving each intent an identity and remembering what was already done.
• The main tools are deterministic IDs with unique constraints, conditional writes on a version, idempotency keys with stored results, and per-sender sequence numbers. They differ in how much state they keep and for how long.
• Every deduplication scheme has a window. A retry that arrives after the window is a new request, so the window must outlast your longest retry path, including queues and manual replays.
• Idempotency must cover every side effect, not just the database row: emails, calls to third parties and events published downstream.

How it works

Why retries need it

A client sends a request and the network goes quiet. Three things could have happened: the request never arrived, the server is still working, or the server finished and the response was lost. The client cannot tell which. Its only choices are to give up, which risks losing the operation, or to retry, which risks doing it twice. Idempotency removes the second risk, so retrying becomes the right default. This is the duplicates problem shape, and it looks the same whether a payment API times out, a rider double-taps "book", or an AI agent re-runs a tool call that creates a ticket because the first call timed out.

Same state, not same response

Mathematically, f is idempotent if f(f(x)) = f(x). For APIs the useful definition is about effects: N deliveries of the same request have the same effect on server state as one. The responses may differ. DELETE /files/9 is idempotent even though the first call returns 204 and the second 404, because the file is gone either way. HTTP defines it the same way: a method is idempotent when many identical requests have the intended effect of one, and PUT, DELETE and the safe methods qualify (RFC 9110). Idempotent is also different from safe: a safe operation (GET) changes nothing, while an idempotent one may change state but only once.

Natural idempotency by design

The cheapest idempotency needs no extra machinery, because the operation is shaped to be repeatable:

• Set instead of add. balance = 120 is idempotent; balance += 20 is not. Send the target state, not the delta, when you can.
• Client-chosen IDs. PUT /orders/7f3c... with an ID the client generated creates the order once; a repeat finds it already exists. AWS uses the same idea with ClientToken on EC2 RunInstances (EC2 idempotency).
• Unique constraints. A unique index on (merchant_id, order_id) makes a second insert fail, so "charge order 512" cannot happen twice even if two code paths try.
• Conditional writes. UPDATE ... WHERE id = 7 AND version = 4 succeeds once; a replay finds version 5 and changes nothing. HTTP expresses this with If-Match: "<etag>" and 412 Precondition Failed.

Idempotency keys for everything else

Operations with no natural identity ("send this email", "charge this card") need an idempotency key: the client names the intent, and the server records the key with the outcome and replays it on a repeat. The REST API Design lesson covers the basic store; the hard parts are elsewhere. There is still no standard for this header: the IETF's Idempotency-Key draft reached version 07 in October 2025 and then expired without becoming an RFC (IETF Datatracker). Its conventions are widely copied anyway: 400 when a required key is missing, 422 when a key is reused with a different payload, and 409 while the first request is still running (draft 07). Stripe's v1 API stores the status and body of the first attempt, replays it even if it was a 500, and may forget keys once they are 24 hours old (Stripe docs). The key must be generated once per intent and reused on every retry. Two copies can arrive at once, so the first must claim the key atomically and the second must wait or get 409 Conflict. And a request that crashed halfway leaves a claimed key with no result, so the claim needs an expiring lease plus recovery points that tell a retry where to resume.

Sequence numbers for streams

When one sender sends many messages in order, a key per message is wasteful. Instead the sender numbers its messages, and the receiver remembers the last number it accepted from each sender. Anything at or below that number is a duplicate; anything above the next expected number reveals a gap. TCP has done this since 1981 with byte sequence numbers, which is why a retransmitted segment never appears twice in your socket. Kafka's idempotent producer applies the same idea to records.

Down the stack

Diagram

sequenceDiagram
  participant P as Producer (id 42, epoch 3)
  participant B as Broker partition
  P->>B: batch seq=7
  B-->>P: ack lost (timeout)
  P->>B: retry batch seq=7
  Note over B: seq 7 in last 5 batches
  B-->>P: duplicate, original offset 1001
  P->>B: batch seq=9
  B-->>P: OutOfOrderSequence (expected 8)
  P->>B: batch seq=8 (epoch 2, zombie)
  B-->>P: fenced, epoch too old

Worked example

A notifications service sends 2,000 push and email messages per second on behalf of other services. Callers use a 2-second timeout and retry up to 3 times. In a normal hour 0.5% of calls time out; during a database failover 10% do.

Without idempotency, most timeouts happen after the message was already sent, so 0.5% of 2,000 is 10 duplicate notifications per second, about 860,000 a day. During a failover it is 200 per second, and every user in that window gets the same password-reset email two or three times.

With idempotency keys derived from the business event, for example password-reset:<user_id>:<token_id>, even a caller that crashes and restarts reuses the same key. The client retry horizon is about 15 seconds, but upstream services can replay their queues for up to 3 days, so the window is set to 4 days. Storage is 2,000 × 345,600 seconds, about 690 million keys; at 150 bytes each that is about 100 GB in a key-value store with TTL expiry, which is cheap next to the support cost of duplicate messages.

The email provider is the remaining gap: if the service crashes after the provider accepted the message but before marking the key done, the retry sends again. Passing the same key to the provider's own idempotency header closes it, because the provider then deduplicates too. Each hop has to carry the identity forward.

In the wild

• Twilio, 2013: a charge that could not record itself repeated. On 18 July a misdiagnosed restart of the Redis master behind billing loaded a bad configuration and left the master read-only with balances at zero. Every billable action then triggered an auto-recharge against the customer's card, and because the new balance could not be written, recharges were attempted again and again. The incident affected 1.4% of customers. Twilio refunded the charges, stopped restarting the master in place, and added fail-safes so cards are not charged when a balance is missing or cannot be written (Twilio).
• Stripe, 2017: idempotency keys as a public API contract. Stripe's engineering post sets out why a charge must happen once even when a network failure hides whether the request arrived, half-ran or succeeded with the response lost. Clients send an Idempotency-Key on each POST; the server replays the stored result or resumes the work, and clients retry with exponential backoff and jitter so a recovering server is not stampeded (Stripe).
• Jepsen, 2020: retries that ignored their own setting. Testing MongoDB 4.2.6 transactions, Jepsen saw duplicate writes among other anomalies, and found that transaction retries ignored the retryWrites setting, a likely source of the duplicates. MongoDB traced the anomalies to a retry bug fixed in 4.2.8. A driver-level retry is only safe when the operation it retries is idempotent (Jepsen).

Trade-offs

Pitfalls in production

• Key generated per attempt. Creating the UUID inside the retry loop gives every retry a new key, so nothing is deduplicated. Generate the key when the intent is created and persist it with the intent.
• Key and effect in different stores. Writing the key to Redis and the effect to Postgres leaves a crash window between them. Commit both in one transaction, or make the effect itself idempotent by using the key as its ID.
• Window shorter than the retry horizon. A dead-letter queue replayed after a week sails past a 24-hour window and duplicates every effect. Size the window from the longest replay path, or keep dedupe in a unique constraint that never expires.
• Uncovered side effects. The database write is deduplicated but the email or the downstream event is sent before the commit and again on retry. Use the outbox pattern for events and pass the key to every external provider.
• Treating "in progress" as success. Returning 200 to a duplicate while the first attempt is still running tells the client the work is done when it may yet fail. Return 409 or make the duplicate wait for the outcome.
• Unscoped keys. Two tenants that both use order-1 as a key collide. Scope keys by account or caller identity.

Leadership lens

• Make idempotency part of the API contract for every mutating endpoint and every message consumer, written down with the window, and check it in design review rather than after the first duplicate charge.
• Ask: what is the longest path by which this request can be retried or replayed, and is our dedupe window longer than that?
• Budget the storage explicitly (rate times window times bytes per key). It is usually cheap, and putting a number on it stops someone shortening the window to save money.
• Test it on purpose: chaos tests that drop responses and replay messages find missing idempotency faster than code review does.

Check yourself

1. DELETE /items/5 returns 204 the first time and 404 the second. Is it idempotent?

   Show answer

   Yes. Idempotency is about the effect on state, and after one or two calls the item is gone. Clients should treat 404 on a retried delete as success.

2. Why does Kafka limit an idempotent producer to 5 in-flight requests per connection?

   Show answer

   The broker remembers only the last 5 batches per producer and partition. With more in flight, a retried batch could fall outside that cache and be rejected instead of being recognised as a duplicate.

3. A consumer deduplicates by message ID in a 24-hour Redis set. What scenario still produces duplicates?

   Show answer

   Any redelivery after 24 hours, such as a dead-letter queue replay, plus a crash between performing the effect and writing the ID, because Redis and the effect are not in one transaction.

4. When is a version-conditioned update better than an idempotency key?

   Show answer

   For read-modify-write updates where several writers may race: the version check deduplicates retries and also prevents one writer silently overwriting another, without a separate key store.

Sources and further study

Primary

• RFC 9110, section 9.2.2: Idempotent Methods: the HTTP definition of idempotence and which methods have it.
• draft-ietf-httpapi-idempotency-key-header-07: the last text of the Idempotency-Key draft, with its fingerprint and error-code rules.
• RFC 7323: TCP Extensions for High Performance: timestamps and PAWS, the guard against wrapped sequence numbers.

Systems docs

• Apache Kafka producer configuration: enable.idempotence and exactly when it is silently disabled.
• Amazon SQS: exactly-once processing: deduplication ids, content-based deduplication and the 5-minute interval.

Books

• Designing Distributed Systems (2nd ed.), Brendan Burns, Ch 2 "Important Distributed System Concepts": idempotency as a basic building block, next to latency and reliability.
• Designing Data-Intensive Applications (1st ed.), Martin Kleppmann, Ch 12 "The Future of Data Systems": the "Aiming for Correctness" section on end-to-end operation ids.

Other curricula

• Azure Architecture Center: Idempotent Consumer pattern: stable keys, atomic markers, concurrent duplicates and how long to keep dedupe records.
• roadmap.sh: System Design: lists "Idempotent Operations" as its own node in the system design path.
• donnemartin/system-design-primer: marks which HTTP verbs are idempotent in its REST table and stops there; keys and windows are not covered.

Watch

• Idempotency - What it is and How to Implement it (Alex Hyett): idempotent HTTP methods and how to make POST and PATCH idempotent with keys.
• Idempotency and Intelligent Retry | PayPal (ByteMonk): how a payments company combines idempotency keys with retries.

Free resources

• Making retries safe with idempotent APIs (AWS Builders' Library): how Amazon designs idempotent create calls with client tokens.
• Implementing Stripe-like idempotency keys in Postgres (Brandur Leach): atomic phases, recovery points and locking for in-flight requests.
• The Idempotency-Key HTTP header field (IETF draft): the expired 2025 draft that most APIs still follow for key semantics, expiry and error codes.
• KIP-98: Exactly once delivery and transactional messaging (Apache Kafka): the design of producer IDs, epochs and sequence numbers.
• Ensuring idempotency of Amazon EC2 API requests (AWS): client tokens on a real create API, including what happens on mismatch.
• Idempotent requests (Stripe API docs): exact semantics of key reuse, expiry and conflicting parameters.