Engineering Atlas › System Design Fundamentals

At-most-once, at-least-once and exactly-once delivery

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why duplicates and losses happen

A producer sends a message and the acknowledgement is lost. It cannot tell whether the broker stored the message, so it must either retry (risking a duplicate) or give up (risking a loss). The same dilemma repeats between broker and consumer: the consumer processes a message and crashes before committing its offset or acking. On restart, the broker redelivers. No protocol over a lossy channel can remove this ambiguity; it can only choose which side to err on. This is the Two Generals' Problem in miniature.

At most once

The consumer commits or acks before processing. If it crashes mid-processing, the message is gone. Producers send without retries. This is acceptable for metrics samples, telemetry and cache invalidation hints where a missing message is cheaper than a duplicate or extra latency.

At least once

The consumer processes then commits. A crash between the two causes redelivery and reprocessing. Producers retry until acknowledged. Nothing is lost, but every downstream effect must tolerate repeats. This is the default in SQS, RabbitMQ with manual acks, and Kafka with commit-after-processing.

Defaults move, so check the version you run. Kafka producers used to default to acks=1 with idempotence off, which could lose a write on leader failover and duplicate one on retry. KIP-679 changed the defaults to acks=all and enable.idempotence=true for Kafka 3.0, but a config-validation bug kept idempotence off until 3.0.1, 3.1.1 and 3.2.0 (KAFKA-13598). It is still switched off silently if you set a conflicting acks, retries or max.in.flight.requests.per.connection without enabling it explicitly (Kafka producer configs). Set it explicitly.

Exactly-once effect

Two techniques get you there:

  1. Idempotent processing. Give each message a stable id (generated by the producer, not the broker) and record processed ids in the same transaction as the side effect. A redelivered message finds its id and is skipped. Alternatively make the operation naturally idempotent: SET balance = 50 instead of ADD 10, upserts keyed by business id.
  2. Transactions spanning input and output. Kafka's idempotent producer (sequence numbers per partition so the broker drops retried duplicates) plus transactions let a consume-transform-produce job write output records and commit input offsets atomically (KIP-98). Downstream readers with read_committed never see aborted output. This covers Kafka-to-Kafka only; the moment you call an external API or database, you are back to idempotency.

The outbox problem

A service must update its database and publish an event. Doing both directly risks one succeeding without the other. The transactional outbox writes the event into an outbox table in the same database transaction, and a relay publishes it afterwards, at least once (microservices.io). Consumers deduplicate by event id.

Down the stack

Layer 1The contract

Each system states its guarantee per hop. A Kafka producer with acks=all gets an acknowledgement only after every in-sync replica has the record, and with min.insync.replicas=2 an acknowledged write survives the loss of any one broker; if the in-sync set shrinks below that, the producer gets NotEnoughReplicas instead of a weak ack (Kafka topic configs). A committed consumer offset means "everything before this has been handled"; on restart, consumption resumes exactly there. SQS standard queues promise at-least-once delivery and best-effort ordering; SQS FIFO queues drop a message whose deduplication id was seen in the previous 5 minutes (SQS exactly-once processing).

Read "exactly once" labels closely. Google Pub/Sub's exactly-once delivery promises no redelivery before the ack deadline and none after a successful ack, but a publisher that sends twice still produces two messages with different IDs (Pub/Sub API reference). None of these promises extends past the system boundary to your database or an email provider.

Layer 2The mechanism

Kafka's idempotent producer is assigned a producer id and epoch, and numbers each batch per partition. The partition leader remembers the last five batch sequences per producer, which is why idempotence requires at most 5 in-flight requests per connection; a retried batch with a known sequence is acknowledged without being written again, and a gap raises an out-of-order error. Transactions add a coordinator that logs state in the internal __transaction_state topic. The producer registers partitions with AddPartitionsToTxn, adds the consumed offsets with sendOffsetsToTransaction (so offset commits to __consumer_offsets join the transaction), and on commit the coordinator writes COMMIT or ABORT control markers into every touched partition. A read_committed consumer reads only up to the last stable offset, holding back records from open transactions. On the consumer side, the dedup pattern is a unique index: the insert of the event id fails or succeeds atomically with the business update.

Layer 3The machine

acks=all costs a replication round trip inside the cluster, typically 1 to 3 ms. Kafka does not fsync per message by default; durability comes from replicas holding the data in their page caches, and the OS flushes to disk later. That trade is safe only while replicas fail independently. NATS JetStream makes a similar one, acknowledging at once but flushing about every two minutes by default, and Jepsen showed that a simulated power loss then drops acknowledged writes (Jepsen: NATS 2.12.1). Transactions add control records and coordinator round trips per commit, which is why stream processors commit every 100 ms or so and accept that much extra latency. In PostgreSQL, the dedup insert touches a B-tree index page and appends a WAL record; the commit waits for the WAL fsync, around 0.1 to 1 ms on an SSD, and group commit lets many concurrent transactions share one flush. That is why batching 100 events per transaction raises throughput by an order of magnitude.

Why it's built this way

  1. Networks lose acknowledgements, so senders retry, and retries create duplicates; no protocol avoids this.
  2. Machines crash between acting and recording, so the order of "act" and "ack" chooses between loss and duplication.
  3. A broker can only deduplicate what it can identify, so producers carry ids and sequence numbers.
  4. External systems cannot join a broker's transaction, so exactly-once effects end up implemented at the sink with idempotency.
  5. Remembering every id forever is too expensive, so dedup state is windowed, and the window bounds the guarantee.

Diagram

stateDiagram-v2
    [*] --> Received
    Received --> Processing: at-least-once (act first)
    Received --> Committed: at-most-once (ack first)
    Committed --> Processing
    Processing --> Done: success then commit
    Processing --> Crash: failure
    Crash --> Received: redelivered (duplicate risk)
    Crash --> Lost: offset already committed
    Done --> [*]
    Lost --> [*]

Worked example

A payments ledger consumes 5,000 PaymentCaptured events per second from Kafka and credits merchant balances in PostgreSQL.

Consumers commit offsets every 5 seconds. A consumer crashing just before a commit has processed up to 5 seconds of its partition's events, say 500 events if it owns 10% of traffic, which will all be redelivered to the new owner after rebalance. Without deduplication, 500 merchants get double credits. During a rolling deploy of 20 consumers, that could happen 20 times.

Fix: the consumer runs, per event, a single transaction:

BEGIN;
INSERT INTO processed_events (event_id) VALUES ($1)
  ON CONFLICT DO NOTHING;            -- 0 rows means duplicate: ROLLBACK and skip
UPDATE balances SET amount = amount + $2 WHERE merchant_id = $3;
COMMIT;

The processed_events table grows by 432 million rows a day, so it is partitioned by day and partitions older than 7 days are dropped. That is safe only because Kafka redelivery can never reach back further than the consumer's committed offset plus the commit interval, and producer retries settle within minutes. The extra insert adds about 0.3 ms per event; batching 100 events per transaction keeps throughput well above 5,000 per second.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
At most onceSimplest, lowest latencySilent loss on crashTelemetry, best-effort notifications
At least onceNo loss, simple to runDuplicates must be handledDefault for most pipelines
At least once + idempotent consumerExactly-once effect, works with any sinkDedup storage and key designMoney, inventory, anything counted
Broker transactions (Kafka EOS)Atomic read-process-write within KafkaOnly covers Kafka, adds latencyStream processing between topics

Pitfalls in production

Leadership lens

Check yourself

  1. Why is exactly-once delivery impossible but exactly-once processing possible?

    Show answer

    A sender cannot distinguish a lost message from a lost ack, so it must risk duplicates or loss. The receiver can, however, detect and discard duplicates so the effect happens once.

  2. Why must the dedup record and the side effect share a transaction?

    Show answer

    Otherwise a crash between them either applies the effect without recording it, causing a later duplicate, or records it without applying, causing a loss.

  3. Does Kafka's exactly-once mode protect a consumer that writes to an external database?

    Show answer

    No. It only makes Kafka reads and writes atomic together; the external write needs its own idempotency.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →