Decoupling work with message queues

16 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

A message queue is a durable buffer between producers and consumers. Each message is meant to be processed by exactly one consumer from a pool of competing consumers (point-to-point).
It decouples the two sides in time, rate and failure: producers keep accepting work while consumers are slow, deploying or down, and a burst becomes a backlog instead of a wall of errors.
Delivery works by lease and acknowledge: a consumer takes a message, it becomes invisible to others, and it is removed only on ack. If the consumer dies, the lease expires and someone else gets it. That is at-least-once delivery.
Ordering is the expensive guarantee. Real systems order per key (a message group or partition) and pay with head-of-line blocking inside that key.
A queue's health is the age of its oldest message and the gap between arrival and completion rates, not its depth alone.

How it works

Point-to-point and competing consumers

A producer sends a message and moves on. The queue stores it durably and hands it to one of N consumers that pull from it. Adding consumers adds throughput, because they compete for messages rather than each receiving a copy. That is the defining difference from pub/sub (see the Pub/Sub lesson), where every subscription gets its own copy.

This is the overload and partial-failure problem shape. A queue lets the slow or failing side of a system fall behind without the fast side noticing. The same structure appears in the Linux kernel: a listening socket has an accept queue sized by the listen() backlog (capped by net.core.somaxconn, 4096 by default since kernel 5.4). Handshaken connections wait there for accept(), and when it is full the kernel drops new ones. Both fail the same way: too small a buffer sheds work, too large a one hides an outage behind latency.

Lease, ack, redelivery

A message moves through a small state machine. It is ready after it is sent. A receive moves it to in flight, leased to one consumer until a deadline. SQS calls this the visibility timeout: 30 seconds by default, and extensions can never push it past 12 hours from the first receive (SQS visibility timeout). RabbitMQ calls the message unacknowledged and guards it with consumer_timeout, 30 minutes by default; when it fires, the broker closes the whole channel and requeues everything that channel held (RabbitMQ consumers). An ack deletes it. A nack, or an expired lease, makes it ready again and increments its receive count. After too many receives it goes to a dead letter queue (see the Dead Letter Queues lesson).

The broker cannot tell a slow consumer from a dead one, so once the lease runs out it redelivers. A merely slow consumer then finds its ack rejected while another consumer holds the same message. SQS standard queues go further and say a message may occasionally arrive twice even inside the visibility timeout. Duplicates are therefore part of the contract, which is why the Delivery Semantics lesson ends at at-least-once plus idempotent handlers (see the Idempotency lesson).

Ordering and its price

A FIFO queue with one consumer is strictly ordered and has the throughput of one consumer. Add a second and message 2 can finish before message 1; any retry also reorders.

The practical answer is ordering per key. SQS FIFO calls the key a message group ID: while any message from group order-42 is in flight, no other message from that group is handed out, but other groups flow freely. Kafka gets the same effect with partitions. The cost is head-of-line blocking: one slow or poisoned message stalls everything behind it in its group until it is acked or dead-lettered. Key by what truly needs order, such as one customer's ledger, not all payments. A FIFO queue without high-throughput mode is also capped at 300 calls per second per API action, or 3,000 messages per second with batches of 10 (SQS quotas).

Fairness is a different problem from order. In a queue shared by many tenants, one tenant's burst fills the backlog and everyone else waits behind it. Since July 2025 SQS standard queues treat a MessageGroupId as a tenant label: when one tenant holds far more in-flight messages than the rest, SQS hands out the other tenants' messages first, without promising any order (SQS fair queues). Elsewhere you get the same effect with one queue per tenant class and a worker that polls them in turn.

Flow control

Consumers pull, so the queue never pushes more than they ask for. Prefetch (RabbitMQ's basic.qos, or up to 10 messages per SQS receive) caps how many unacked messages a consumer may hold: too low and it idles between round trips, too high and one consumer hoards work while others starve. Long polling (SQS WaitTimeSeconds up to 20) holds an empty receive open until a message arrives, cutting empty responses and cost.

On the producer side, a bounded queue is backpressure: when full, it rejects or blocks the producer instead of growing forever. Google's SRE book gives the same advice for request queues inside servers: under steady load, keep the queue at half the thread pool or less, so excess work is rejected early instead of waiting until it is useless (Google SRE book).

Down the stack

Layer 1The contract

Once the broker acknowledges a send, the message survives a broker crash (for durable queues) and will be delivered to some consumer at least once. It stays until it is acked, exceeds its maximum receive count, or ages past retention (4 days by default in SQS, at most 14, per the SQS quotas). While leased, no other consumer receives it; after the lease expires it can be redelivered, so a consumer must finish and ack within the lease or extend it. Standard queues promise no order. FIFO queues promise order within a group, and only while consumers ack in order. Nothing promises exactly-once processing.

Layer 2The mechanism

Underneath, a queue is two structures: a ready list in arrival order, and an in-flight set indexed by lease deadline (a heap or timer wheel) so expired leases can be found without scanning. Each message carries an ID, a receive count, and a per-delivery token: SQS returns a new receipt handle on every receive, and only the latest handle can delete the message, which is how a slow consumer's late ack is rejected. RabbitMQ numbers deliveries per channel with a delivery-tag and acks with basic.ack or basic.nack with requeue.

Redis Streams show the structure plainly. XREADGROUP moves entries into the consumer group's pending entries list (PEL), with owner, delivery count and last-delivery time. XACK removes them. XAUTOCLAIM transfers entries idle longer than a threshold to another consumer, which is lease expiry done by the client (Redis Streams).

Kafka now has the same state machine. Share groups ("queues for Kafka", KIP-932) became production-ready in Kafka 4.2 in February 2026 (Kafka 4.2 release). Consumers in a share group read the same partitions at once. Each record they fetch is locked to them for 30 seconds by default and carries a delivery count. The consumer answers ACCEPT, RELEASE (try again) or REJECT (never again). After 5 deliveries by default the record is archived, not redelivered, and redeliveries can arrive after later offsets (KIP-932). That is a lease-and-ack queue on top of a log.

A relational database can be a queue too. With Postgres's SELECT ... FOR UPDATE SKIP LOCKED (since 9.5), each worker locks the first ready row nobody else holds, so workers do not serialise on the head of the table (Postgres SELECT). The ack is a DELETE in the same transaction as the work, which gives exactly-once effects when the work touches the same database.

Layer 3The machine

Every message costs at least two durable writes: the send, and the ack that deletes it. A log such as Kafka avoids the second by storing a consumer offset once per batch, which is a big part of why logs are cheaper per message. A durable send means an fsync: tens of microseconds on an NVMe drive with power-loss protection, and milliseconds on drives without it. Brokers therefore group-commit, batching many messages into one fsync, and producers batch sends to amortise round trips (an SQS batch carries up to 10 messages).

Queues are fastest when nearly empty. RabbitMQ classic queues page old messages to disk as the backlog grows, adding disk reads exactly when the system is behind. In Postgres, each lease UPDATE and ack DELETE leaves a dead tuple, so a long-running transaction that holds back vacuum makes every dequeue scan past dead rows.

Why it's built this way

Producers and consumers fail, deploy and scale on different schedules, so a durable buffer sits between them.
A consumer can crash halfway through work, and nobody can tell slow from dead, so delivery is a lease that expires instead of a delete on hand-off. That makes duplicates unavoidable.
Each delivery and ack is a durable write and a round trip, so brokers group-commit, batch and prefetch.
Strict global order would mean one consumer, so ordering is scoped to a key and its cost stays inside that key.
Memory, disk and patience are finite, so queues get length limits, retention, and dead-lettering.

Diagram

stateDiagram-v2
  [*] --> Ready: send
  Ready --> InFlight: receive (lease starts)
  InFlight --> Acked: ack before deadline
  InFlight --> Ready: nack or lease expires
  InFlight --> InFlight: extend lease
  Ready --> Dead: receive count reached max
  Acked --> [*]
  Dead --> [*]

Worked example

A photo service generates thumbnails. Uploads average 300 per second and peak at 1,500 per second for 10 minutes after a marketing push. A job takes 200 ms on one core, and the fleet runs 100 single-core workers, so capacity is 100 × 5 = 500 jobs per second.

During the peak the queue grows at 1,500 − 500 = 1,000 messages per second, reaching 600,000 messages after 10 minutes. By Little's law, a message arriving at the end waits about 600,000 / 500 = 1,200 seconds, 20 minutes. Afterwards the spare capacity is 500 − 300 = 200 per second, so draining takes 3,000 seconds, 50 minutes. Depth alone (600,000) does not tell you whether that is acceptable; the age of the oldest message (20 minutes) does. If thumbnails must appear within 2 minutes, autoscale on age and be ready to triple the fleet to 300 workers within a minute or two, or throttle the push at the source.

The p99 job takes 2 seconds, so a 30-second visibility timeout leaves slack. A 3-second one would redeliver on every GC pause, doubling work exactly during the peak.

Cost on SQS standard at about $0.40 per million requests: 300 per second is 26 million messages a day. Sending, receiving and deleting one at a time is 78 million requests, about $31 a day. Batching 10 per call cuts that to roughly $3, plus empty receives, which long polling keeps small.

In the wild

Slack, about 2016: a full Redis job queue could neither enqueue nor dequeue. Database contention slowed job execution until the Redis-backed queue hit its memory limit. New jobs were refused, so every feature that used the queue failed. Dequeuing also needed spare memory to move a job onto an in-flight list, so the queue stayed stuck after the database recovered and needed lengthy manual repair. At the time the queue ran over 1.4 billion jobs on busy days, peaking at 33,000 per second. Slack put Kafka in front of Redis as a durable buffer, so enqueues keep working during a backlog (Slack Engineering).
AWS, 2025: queued retries push EC2's lease manager into congestive collapse. After the DynamoDB DNS failure on 19–20 October, EC2's DropletWorkflow Manager had to re-establish leases with a huge fleet of hosts at once. Lease attempts timed out, their retries were queued, and the service stopped making progress. Engineers throttled incoming work and restarted hosts to clear the queues; launches recovered at 5:28 AM PDT, and every EC2 API was normal only by 1:50 PM. AWS's fix is to rate-limit incoming work based on the size of the waiting queue (AWS).
GitHub, 2018: a backlog outlives its usefulness. During the 24-hour degradation after a 43-second network partition, more than 5 million webhook events and 80,000 Pages builds queued up, and about 200,000 webhook payloads passed their internal TTL and were dropped. A backlog that drains too late is the stale-work problem in the Pitfalls below (GitHub).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Managed queue (SQS, Cloud Tasks)	No servers, scales without planning, DLQ built in	Per-request cost, polling latency, limited ordering	Decoupling services in one cloud
Broker queue (RabbitMQ, ActiveMQ)	Low latency push, prefetch, rich routing	You run it; deep backlogs hurt; one queue has a throughput ceiling	Task queues at moderate scale with routing needs
Database as queue (Postgres `SKIP LOCKED`)	Enqueue in the same transaction as business data; no new system	Vacuum and lock overhead past thousands per second	Jobs tied to database writes, modest rates
Log used as queue (Kafka, Redis Streams)	Replay, cheap per message, ordered partitions	Consumer groups cap parallelism at the partition count and have no per-message retry; Kafka share groups (4.2+) add both but give up order	High volume, ordered processing, replay needed

Pitfalls in production

Lease shorter than processing. Work regularly outlives the visibility timeout, so every slow job runs twice or more and load spirals. Set the lease from p99 processing time with margin, and extend it from a heartbeat for long jobs.
Requeue at the head forever. A basic.nack with requeue=true on a message that always fails puts it straight back, and consumers spin on it at full CPU. Cap receives and dead-letter, and add a delay before retry.
Scaling on depth. A depth of 10,000 is fine for a fast consumer and a crisis for a slow one. Alarm and autoscale on age of oldest message (SQS exposes ApproximateAgeOfOldestMessage).
Stale backlog. After a 6-hour outage the queue replays 6 hours of password-reset emails nobody wants. Give messages a deadline and drop expired ones at the consumer.
One hot ordering key. All messages share a group ID such as the tenant name, and the largest tenant serialises everyone. Order only what must be ordered and choose a finer key.

Leadership lens

Ask what the oldest acceptable message age is for each queue, and make that the alarm and the autoscaling target. It turns "the queue is big" into a business decision.
Require every consumer to be idempotent before it ships. Duplicates are part of the contract, and retrofitting idempotency after a double-charge is expensive.
A Postgres-backed queue is often the right first step for a small team because it removes a whole system to operate. Write down the rate (a few thousand jobs per second) at which you would move off it.

Check yourself

A consumer finishes a job in 45 seconds with a 30-second visibility timeout. What happens to its ack and why?
Show answer
The lease expired at 30 seconds and the message became visible, so another consumer may already hold it with a new receipt. The old receipt is stale and the delete fails or is ignored, and the job runs twice.
Why does adding consumers to a FIFO queue with a single message group not increase throughput?
Show answer
Only one message from a group can be in flight at a time, so the group is processed serially no matter how many consumers exist. Parallelism comes from having many groups.
Why is a high prefetch count risky for slow, uneven jobs?
Show answer
One consumer reserves many messages while others sit idle, and if it crashes all of them wait for their leases to expire before anyone else can work on them.
Why do message queues usually cost more per message than a log like Kafka?
Show answer
A queue tracks state per message, so each ack is another durable write and round trip. A log stores one offset per consumer group and commits it per batch.

Sources and further study

Primary

KIP-932: Queues for Kafka: the design of share groups, acquisition locks, delivery counts and the ACCEPT, RELEASE and REJECT outcomes.
Apache Kafka 4.2.0 release announcement: share groups declared production-ready, plus the RENEW acknowledgement for long jobs.

Systems docs

Amazon SQS message quotas: retention, the 1 MiB message size, batch size and FIFO throughput limits in one table.
Amazon SQS fair queues: how SQS detects a noisy tenant and what fairness does not promise.
RabbitMQ: Consumers: the delivery acknowledgement timeout, prefetch and consumer capacity.

Books

Designing Distributed Systems (2nd ed.), Brendan Burns, Ch 11 "Work Queue Systems": a generic work queue, a worked thumbnailer, and scaling workers dynamically.
Designing Data-Intensive Applications (2nd ed.), Martin Kleppmann and Chris Riccomini, Ch 12 "Stream Processing": its "Transmitting Event Streams" section sets per-message brokers against log-based ones.
Site Reliability Engineering, Google, Ch 22 "Addressing Cascading Failures": read free, including how big a server's request queue should be.

Other curricula

donnemartin/system-design-primer: treats queues under "Asynchronism" with short notes on message queues, task queues and back pressure, and flags SQS's possible duplicate delivery; it does not cover leases or ordering.
roadmap.sh: System Design: lists Message Queues, Task Queues and Queue-Based Load Leveling as separate nodes under asynchronism and cloud patterns.

Watch

Message Queues in System Design Interviews w/ Meta Staff Engineer (Hello Interview): when to reach for a queue and the follow-up questions interviewers ask about it.
Publish-Subscribe Pattern vs Message Queues vs Request Response (Hussein Nasser): the three communication styles side by side, with their costs.

Free resources

Amazon SQS visibility timeout: the lease model, in-flight limits and how to extend a lease.
Avoiding insurmountable queue backlogs (AWS Builders' Library): how Amazon handles backlogs, fairness and stale work in multi-tenant queues.
Competing Consumers pattern (Azure Architecture Center): the pattern, its ordering and poison-message issues.
Redis Streams: consumer groups, the pending entries list and claiming idle messages.
Postgres job queues and failure by MVCC (Brandur Leach): how a database-backed queue degrades when vacuum falls behind.
Point-to-Point Channel (Enterprise Integration Patterns): the classic definition of one message, one receiver.

Open in the interactive app: practice, quiz, mock interview →