Reliable events with the outbox pattern

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

A service often needs to change its database and publish an event. Doing both directly (a "dual write") loses or invents events whenever one succeeds and the other fails.
The outbox pattern writes the event into an outbox table in the same local transaction as the business change. A separate relay later reads the table and publishes to the broker.
The guarantee becomes: if the change commits, the event is eventually published at least once; if it rolls back, the event never exists.
Consumers must be idempotent, because the relay can publish the same event more than once.

How it works

The dual-write problem

Consider INSERT order; COMMIT; publish OrderPlaced. If the process crashes after the commit but before the publish, downstream services never learn about the order. Reverse the order and a crash after publishing but before commit announces an order that does not exist. Wrapping both in a distributed transaction (2PC with the broker) is possible in theory but rarely supported well and couples availability. The outbox avoids the problem by making the event part of the data the database already commits atomically (microservices.io).

Writing to the outbox

In the same transaction as the business change, insert a row into outbox with an id, the aggregate type and id (for example order, 8812), an event type, and the payload. The commit makes both durable together. The service's request path does not talk to the broker at all, so a broker outage no longer fails user requests; events just queue up in the table.

Relaying: polling or log tailing

A polling publisher queries unpublished rows in order, publishes them, then marks them published or deletes them. It is simple and works with any database, but adds query load and latency equal to the poll interval. Transaction log tailing uses change data capture: a connector such as Debezium reads inserts to the outbox table from the database's replication log and publishes them. Latency is lower and there is no polling load, but you now run a CDC pipeline.

Two variants are worth knowing. In a NoSQL store without multi-item transactions, the outbox becomes a field on the record itself, which a change stream then publishes (microservices.io). In PostgreSQL you can skip the table entirely: pg_logical_emit_message(true, prefix, content) writes a message into the WAL as part of the current transaction, so it appears in the logical decoding stream only if the transaction commits, and nothing needs deleting afterwards (Postgres docs).

Ordering and duplicates

The relay can crash after publishing and before recording that it did, so on restart it publishes again: delivery is at-least-once. Consumers keep a record of processed event ids (an "inbox" table) and skip repeats, or make their updates naturally idempotent. To preserve order per entity, publish with the aggregate id as the partition key so all events for one order land on one partition, and never let the relay publish a later event for an aggregate before an earlier failed one.

Housekeeping

The outbox grows with every write. Delete published rows promptly, or partition the table by time and drop old partitions, to avoid bloat that slows both the relay and the business tables' vacuum.

Down the stack

Layer 1The contract

The outbox pattern promises: an event exists if and only if its business transaction committed; every committed event is published at least once, eventually, as long as the relay and broker eventually run; and events for the same aggregate are published in commit order, provided the relay preserves order and partitions by aggregate. It does not promise exactly-once delivery or a latency bound; latency is whatever the relay achieves.

Layer 2The mechanism

A common schema, and the default the Debezium outbox event router expects, has columns id, aggregatetype, aggregateid, type, and payload. The router reads each insert from the change stream and sends it to the topic outbox.event.<aggregatetype> by default, using aggregateid as the message key, so all events for one aggregate share a Kafka partition. It drops DELETE events on the outbox table, so the relay's cleanup produces no extra messages, and it exposes the row id so consumers can deduplicate (Debezium docs). A polling relay typically runs SELECT ... FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED, so several relay instances can work without taking the same rows. One subtle trap: sequence values are allocated when a row is inserted, not when its transaction commits, so a row with id 101 can commit after the relay has already processed id 102. A relay that keeps a "highest id seen" cursor will skip 101 forever; selecting by an unpublished flag, or tailing the log in commit order, avoids this. On the broker side, enable.idempotence=true and acks=all stop producer retries from adding duplicates of their own. Both have been the Java producer's defaults since Kafka 3.0, so older advice to switch them on is now mostly about checking nobody switched them off (KIP-679). They do not help with the relay's own crash-and-resend duplicates, which carry a new producer session.

Layer 3The machine

The outbox insert rides on the business transaction's commit: one extra heap tuple, one extra index entry, and perhaps 200 to 500 bytes more WAL, all flushed by the same fsync. That is why the pattern adds almost no latency to the request. Deleting or updating published rows creates dead tuples that vacuum must reclaim; at 5,000 events per second that is 430 million dead tuples per day. Log tailing uses a logical replication slot, which makes PostgreSQL retain WAL until the connector confirms it, even when nobody is connected (Postgres docs); if the connector is down for a day at 20 MB per second of WAL, the primary holds about 1.7 TB of extra WAL and can fill its disk. The cap, max_slot_wal_keep_size, defaults to -1, meaning unlimited (Postgres docs). A polling relay at a 100 ms interval adds an average of 50 ms of delay plus query time; CDC pipelines usually deliver in tens of milliseconds.

Why it's built this way

A database and a broker cannot commit atomically together without 2PC, so events are stored where the business data is.
The database already guarantees atomic commits, so putting the event in the same transaction ties its fate to the change.
Something must move events to the broker, and it can crash between sending and recording, so delivery is at-least-once.
At-least-once delivery creates duplicates, so consumers deduplicate by event id.
The outbox grows with every write and retained WAL fills disks, so cleanup and connector lag must be monitored.

Diagram

sequenceDiagram
    participant API as Order service
    participant DB as Database
    participant R as Relay (poller or CDC)
    participant K as Broker
    participant C as Consumer
    API->>DB: BEGIN, INSERT order, INSERT outbox row, COMMIT
    R->>DB: read unpublished outbox rows
    R->>K: publish OrderPlaced (key = order id)
    K-->>R: ack
    R->>DB: mark row published
    K->>C: deliver OrderPlaced
    C->>C: skip if event id already processed

Worked example

An order service handles 2,000 orders per second, each producing one OrderPlaced event of about 1 KB. Previously it committed and then published directly to Kafka. During a 4-minute broker incident, the publish calls failed after commit, and about 480,000 orders never reached the fulfilment service. Reconciling them took two days.

With the outbox: each order transaction inserts one more 1 KB row, adding under 0.2 ms to commit time. A relay with two instances polls every 50 ms, fetches up to 500 rows using SKIP LOCKED, publishes them in a batch, and deletes them. Normal end-to-end delay is under 100 ms. During the same broker incident, rows accumulate: 2,000 per second for 240 seconds is 480,000 rows, about 500 MB. When the broker recovers, the relay drains them at 20,000 per second in about 25 seconds. No order is lost.

The fulfilment service receives a few hundred duplicates after a relay restart; its inbox table (event id primary key, checked in the same transaction as its own update) drops them. An alert fires when the oldest unpublished row is more than 30 seconds old.

In the wild

Brandur Leach, 2017: transactionally staged job drains. Enqueuing background jobs into Redis inside a Postgres transaction let workers run jobs before commit (and keep jobs for rolled-back work), while enqueuing after commit lost jobs on a crash. The fix was a staged_jobs table written in the business transaction, drained by a single enqueuer that selects a batch in a repeatable-read transaction, sends it to Sidekiq, then deletes up to the last id sent. Delivery became at least once, so job handlers had to tolerate repeats (brandur.org).
Segment, 2017: what deduplicating at the consumer actually costs. About 0.6% of ingested events arrived more than once, largely from clients on flaky mobile networks resending what the server had already received. Segment partitioned Kafka by message id and gave each worker an embedded RocksDB store of seen ids with a four-week window. After three months it held about 60 billion keys in 1.5 TB on disk, across roughly 200 billion messages. An outbox consumer's inbox table is the same idea at smaller scale (Segment via Twilio).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Direct dual write	Simplest, lowest latency	Loses or invents events on failure	Never for events others depend on
Outbox with polling relay	Works with any database, easy to understand	Poll load, latency equal to interval	Moderate volume, no CDC infrastructure
Outbox with log tailing (CDC)	Low latency, no polling load	CDC pipeline to run, WAL retention risk	High volume, CDC already in place
Event sourcing	Events are the source of truth	Big change in data model	Domain benefits from full history
2PC with broker	Atomic in theory	Poor support, blocking, coupling	Almost never in modern stacks

Pitfalls in production

Skipped rows from id cursors. A relay that tracks "last id processed" misses rows that commit out of id order. Select by an unpublished flag or use log tailing.
Outbox bloat. Millions of published rows slow the relay query and vacuum. Delete promptly or drop time partitions, and index the unpublished rows.
WAL retention from a stalled connector. A broken CDC connector makes the primary keep WAL until the disk fills. A healthy one can do it too: if the outbox is quiet while other tables or databases on the server are busy, the slot only advances when an outbox change arrives, so Debezium recommends periodic heartbeat events (heartbeat.interval.ms) (Debezium docs). Alert on replication slot lag and set a maximum retained WAL size.
Non-idempotent consumers. Duplicate events double-ship or double-charge. Store processed event ids in the consumer's transaction.
Order broken by parallel relays. Two relay workers publish events for the same order out of sequence. Partition work by aggregate id, or use a single ordered stream per partition.
Payloads that leak internal schema. Publishing table rows directly couples consumers to your schema. Publish explicit, versioned event contracts.

Leadership lens

Make "no dual writes" a team rule for events that other services depend on, and check for it in code review.
Choose polling or CDC based on volume and existing platform; a shared CDC platform amortizes its cost across teams.
Require metrics for outbox depth, oldest unpublished age, relay errors, and replication slot lag, with alerts.
Treat event payloads as public APIs with versioning and ownership, not as copies of table rows.

Check yourself

Why does publishing after commit not solve the dual-write problem?
Show answer
A crash between the commit and the publish leaves a committed change with no event, and nothing remembers to send it.
Why do outbox consumers need to be idempotent?
Show answer
The relay may publish an event and crash before recording that it did, so the event is published again on restart.
Why can a relay that reads rows with id > last_seen_id lose events?
Show answer
Ids are assigned at insert time but transactions commit in a different order, so a lower id can become visible after the relay has moved past it.

Sources and further study

Primary

KIP-679: Producer will enable the strongest delivery guarantee by default: the Kafka 3.0 change to acks=all and idempotent producers.
Helland, Life beyond Distributed Transactions (CIDR 2007): why messages between entities must be retried and processed idempotently.

Systems docs

PostgreSQL: pg_logical_emit_message: a transactional WAL message that can replace an outbox table for log-tailing relays.
PostgreSQL: Logical Decoding Concepts: how slots retain WAL even when no consumer is connected.
Debezium connector for PostgreSQL: WAL disk consumption, heartbeats and at-least-once behaviour after faults.

Books

Designing Data-Intensive Applications (1st edition), Martin Kleppmann, chapter 11 "Stream Processing", section "Databases and Streams": dual writes, change data capture and keeping derived systems in sync.
Understanding Distributed Systems (2nd edition), Roberto Vitillo, chapter "Asynchronous transactions": the outbox and sagas as alternatives to atomic commit.

Other curricula

Azure Architecture Center: Transactional Outbox with Azure Cosmos DB: the NoSQL variant, using a single-partition transactional batch, the change feed, TTL cleanup and broker-side duplicate detection.

Watch

What is the Transactional Outbox Pattern? (Confluent): the dual-write problem and how the outbox solves it with Kafka.

Free resources

microservices.io: Transactional outbox: the pattern, its forces, and related patterns.
microservices.io: Transaction log tailing: the CDC-based relay option.
Debezium: Outbox Event Router: the outbox table format and routing options.
Debezium blog: Reliable microservices data exchange with the outbox pattern: an end-to-end implementation walk-through.
AWS Prescriptive Guidance: Transactional outbox: the pattern on AWS, including duplicate handling.

Open in the interactive app: practice, quiz, mock interview →