Engineering Atlas › System Design Fundamentals

Parking failed messages in a dead letter queue

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why messages fail repeatedly

Failures fall into two groups. Transient: a dependency timed out, a database was failing over, a rate limit hit. Retrying later usually succeeds. Permanent: malformed payload, schema version the consumer does not understand, a reference to an entity that does not exist, a bug triggered by specific data. Retrying a permanent failure only wastes capacity, and in an ordered queue it blocks everything behind it.

Mechanics in common brokers

In SQS, each receive increments a message's receive count; when it exceeds maxReceiveCount the queue moves it to the configured DLQ, and a redrive operation can move messages back (SQS dead-letter queues). RabbitMQ routes rejected or expired messages to a dead letter exchange configured on the queue, with headers recording the reason and original queue. Watch the guarantee: classic queues republish dead letters without confirms, so a message can be lost if the target is unavailable, while quorum queues can dead-letter at least once (RabbitMQ dead lettering). Azure Service Bus has a built-in dead-letter subqueue per queue, with a reason and description; the default delivery limit is 10, and messages in the DLQ never expire, so someone must drain it (Service Bus DLQ).

Kafka consumer groups have no built-in DLQ because consumers own their offsets; frameworks and consumers publish failed records to a separate topic, often orders.dlq, then commit the offset and move on. Kafka Connect does this only for sink connectors (Confluent: Kafka Connect concepts), and Kafka Streams gained DLQ support in its exception handlers in 4.2 (Kafka 4.2 release). Kafka's newer share groups count deliveries per record, but a record that hits the limit (5 by default) or is rejected is simply archived; there is no DLQ for it yet (KIP-932).

Retry tiers

A common pattern is a ladder: try inline a few times with short backoff; then publish to a retry topic or delayed queue (for example 1 minute, then 10 minutes, then 1 hour); then the DLQ. This keeps the main consumer fast while still giving transient failures room to recover. Each hop should carry metadata: original topic, partition and offset, attempt count, first failure time, last error class and message.

Ordering

Moving a message aside breaks ordering for its key. If event 2 for order 42 is dead-lettered and event 3 is processed, state can go wrong. Options: accept it where events are independent; park all later messages for that key until the failed one is resolved; or design events as full-state snapshots so later ones supersede earlier ones.

Replay

Replaying sends DLQ messages back to the main queue once the cause is fixed. It needs idempotent consumers, because some of these messages may have partially applied. It also needs rate limiting, because 2 million messages released at once can overload the system that just recovered.

Down the stack

Layer 1The contract

In SQS, a received message becomes invisible for the visibility timeout; if the consumer does not delete it in time, it becomes visible again and its receive count rises. Once a receive would push the count past maxReceiveCount, the queue moves the message to the DLQ instead of delivering it. In RabbitMQ, a message is dead-lettered when it is rejected without requeue, expires by TTL, overflows a length limit, or exceeds a quorum queue's delivery limit. Since RabbitMQ 4.0 that limit defaults to 20, and without a dead-letter exchange the message is dropped, not parked (RabbitMQ quorum queues). The DLQ itself promises nothing beyond storage: no alerting, no replay, no ordering relative to the source.

Layer 2The mechanism

An SQS queue's redrive policy is a small JSON document naming the DLQ's ARN and maxReceiveCount. Every receive returns a fresh receipt handle, which is what DeleteMessage and ChangeMessageVisibility need, so a slow consumer can extend its lease. Redrive back to the source is a message move task with an optional rate cap of at most 500 messages per second, and only one task can run per queue at a time (StartMessageMoveTask). One quirk: on a standard queue with maxReceiveCount above 3, a message received 3 times without being deleted is moved to the back of the queue, so ApproximateAgeOfOldestMessage stops showing it. In RabbitMQ, queue arguments x-dead-letter-exchange and x-dead-letter-routing-key route dead messages, and each message gains an x-death header listing reason, source queue, count and time; quorum queues track a delivery count in their Raft log. Kafka has no broker-side DLQ because a partition is an append-only log: a consumer cannot skip one record and return to it later without committing past it. So frameworks copy the record to another topic: Kafka Connect with errors.tolerance=all and errors.deadletterqueue.topic.name, adding __connect.errors.* headers, or retry topics with suffixes such as -retry-0.

Layer 3The machine

Visibility is a per-message timestamp and receive count stored in a replicated backend, so each receive and delete is a network round trip of a few milliseconds. Poison messages often hurt more than they fail: a payload that triggers an out-of-memory error crashes the consumer process, and a JVM restart takes 10 to 30 seconds, so one bad message can take a consumer offline repeatedly until the count trips. SQS caps a message at 1 MiB including attributes (it was 256 KiB until August 2025), so a large payload plus a full stack trace can still fail to fit (SQS quotas); keep error metadata short and put big payloads in object storage with a pointer. Kafka DLQ topics keep the default 7-day retention unless changed, which is often shorter than the time it takes someone to notice (Kafka topic configs).

Why it's built this way

  1. A broker cannot tell a slow consumer from a dead one, so it uses leases and redelivers on timeout.
  2. Redelivery of a message that always fails would loop forever, so brokers count receives and stop at a limit.
  3. Logs cannot skip one entry and come back, so log-based systems copy failures to a separate topic and move on.
  4. Failures that code cannot fix need humans, so dead messages must keep their context and live long enough to be triaged.
  5. Recovery itself creates load spikes, so replay must be rate-limited.

Diagram

flowchart LR
    P[Producer] --> Q["Main queue"]
    Q --> C[Consumer]
    C -->|success| A["Ack, delete"]
    C -->|"transient, attempts < 5"| R["Retry queue, backoff"]
    R --> Q
    C -->|"permanent, or attempts = 5"| D["Dead letter queue"]
    D --> M["Alert + inspect tool"]
    M -->|"fix deployed, rate-limited redrive"| Q

Worked example

An order service processes 3,000 messages per second from SQS. A deploy introduces a bug that fails on orders with a discount code, about 4% of traffic.

With maxReceiveCount 5 and a visibility timeout of 30 seconds, each failing message is received 5 times over about 2.5 minutes before landing in the DLQ. Failing traffic is 120 per second, but each consumes 5 attempts, so consumers spend 600 attempts per second, 20% of capacity, on doomed work. If the consumer fleet ran at 85% utilisation, the healthy traffic now queues and latency climbs.

The DLQ depth alarm (any message older than 5 minutes, or depth above 100) fires 3 minutes after the deploy. The team rolls back after 20 minutes: 120 x 1,200 seconds = 144,000 messages in the DLQ. They inspect a sample, confirm all share the error DiscountNotFound, deploy the fix, and redrive at 500 messages per second (the highest fixed rate SQS allows), taking about 5 minutes, while watching consumer error rates. Because order creation is idempotent on order_id, the few messages that had partially succeeded do not create duplicates.

Without a DLQ, those messages would have retried forever, consuming 20% of capacity until the fix, or been dropped by a catch-and-log handler.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Broker-native DLQ (SQS, Service Bus)Built in, automatic receive countingCoarse: same retry policy for all errorsSimple queues with independent messages
Retry topics plus DLQ topic (Kafka)Delayed retries without blocking partitionMore topics, ordering per key is lostHigh-throughput streams
Immediate DLQ for permanent errorsNo wasted retriesRequires accurate error classificationValidation and schema failures
Block partition on failurePreserves orderingOne bad message stalls everythingStrict ordering, low volume
Drop and logSimplestSilent data lossTruly disposable messages only

Pitfalls in production

Leadership lens

Check yourself

  1. Why should a schema validation error skip retries and go straight to the DLQ?

    Show answer

    It is permanent: the same payload will fail every time, so retries only waste capacity and delay other messages.

  2. Why must consumers be idempotent before you redrive a DLQ?

    Show answer

    Some messages may have partially succeeded before failing, so reprocessing must not duplicate effects that already happened.

  3. How can a too-short visibility timeout fill a DLQ with valid messages?

    Show answer

    Slow but healthy processing exceeds the timeout, the message is redelivered and its receive count climbs until it crosses the limit.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →