Engineering Atlas › System Design Fundamentals

What a message broker does

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

What a broker does

Without a broker, a producer must know every recipient, retry each one, and hold undelivered messages itself. A broker takes that over with four jobs: accept (authenticate the producer and confirm receipt), store (durably, until delivered or expired), route (decide which queues or subscriptions get a copy), and deliver (hand messages to consumers, track acks, and redeliver).

The Message Queues and Pub/Sub lessons cover the two delivery shapes a broker exposes. This lesson is about the machine that implements them and the policies that sit around them.

The same structure is older than software brokers. An email server is a store-and-forward broker: Postfix accepts mail over SMTP, puts it in its incoming queue, moves it to the active queue for delivery, parks failed deliveries in the deferred queue for retry, and after maximal_queue_lifetime (5 days by default) bounces it back to the sender, which is dead-lettering by another name. The partial-failure and overload shapes are identical: what to do with a message nobody can take right now.

Smart broker or log broker

A smart broker keeps state per message. RabbitMQ knows which messages in a queue are ready, which are unacked and by which channel, how many times each was delivered, and when each expires. That enables per-message routing, priorities, delays, TTLs and dead-lettering, and lets consumers stay simple. The cost is bookkeeping per message, which caps throughput per queue and makes deep backlogs expensive.

A log broker keeps state per partition. Kafka appends records and serves reads by offset; it does not know whether any single record was processed. Consumers commit one offset per partition. That makes the broker cheap per message and makes replay trivial, but per-message retry, delay and dead-lettering move into consumer code.

The line between the two has blurred. RabbitMQ added streams, an append-only log that consumers read without removing messages and can replay from an offset or a timestamp (RabbitMQ streams). Kafka 4.2 went the other way: its share groups, production-ready since February 2026, keep a lock and a delivery count per record, so a Kafka cluster can act as a work queue (Kafka 4.2 release). Pick the design by access pattern, then check which product offers it.

Policies on the broker

Every serious broker exposes the same set of decisions, whatever it calls them:

Where brokers sit

Most systems run a central cluster; larger estates link clusters across regions (RabbitMQ federation and shovels, Kafka MirrorMaker 2). Inside a cluster, replication now runs on consensus logs. RabbitMQ 4.0 removed classic queue mirroring entirely after three years of deprecation, leaving quorum queues and streams as the replicated types (RabbitMQ 4.0.1 release notes). Kafka 4.0 (March 2025) likewise runs entirely without ZooKeeper and keeps its metadata in its own Raft log, KRaft (Kafka 4.0 release). Brokerless libraries such as ZeroMQ keep routing but drop storage, trading durability for latency.

Down the stack

Layer 1The contract

A broker with publisher confirms promises that once it acks a message, the message is stored with the configured durability (on disk, and on a quorum of replicas for replicated queues) and will be delivered at least once to each queue it was routed to, unless a declared policy removes it: TTL expiry, a length limit with drop-head, or a delivery limit. A rejected publish (a basic.nack under reject-publish overflow, or a returned unroutable message) means the broker did not take responsibility, and the producer must retry or handle it. Every removal by policy is either silent or dead-lettered, depending on configuration; the contract is only as good as that configuration.

Layer 2The mechanism

AMQP 0-9-1 makes the policies concrete as queue arguments: x-max-length or x-max-length-bytes, x-overflow set to drop-head, reject-publish or reject-publish-dlx, x-message-ttl, x-dead-letter-exchange and x-dead-letter-routing-key, and on quorum queues x-delivery-limit. Since RabbitMQ 4.0 that limit defaults to 20 rather than unlimited, so a message redelivered 20 times is dropped, or dead-lettered if a dead-letter exchange is set (RabbitMQ quorum queues). Upgrading teams that relied on endless requeue lose messages silently unless they add a DLX. When a message is dead-lettered, RabbitMQ adds an x-death header recording the source queue, the reason (rejected, expired, maxlen or delivery_limit) and a count, so the consumer of the dead-letter queue knows what happened (RabbitMQ dead lettering).

Expiry has a subtle mechanism. A classic queue with a per-queue TTL expires from the head, which is exact because every message has the same TTL and messages arrive in order. With per-message TTLs, a long-lived message at the head can shield already-expired messages behind it until it is consumed or expires itself.

Flow control is protocol-specific. AMQP 0-9-1 relies on TCP backpressure and the broker blocking publishing connections (connection.blocked) when a memory or disk alarm fires. AMQP 1.0 uses explicit link credit: the receiver sends a flow frame granting N credits, and the sender may send N transfer frames, settled by disposition frames (AMQP 1.0 spec). This is no longer niche: RabbitMQ 4.0 made AMQP 1.0 a core protocol that is always on. Kafka has no per-message flow control at all; consumers pull batches by byte limits such as fetch.max.bytes.

Replication differs too: RabbitMQ quorum queues confirm once a Raft majority has the message, while Kafka followers fetch from the partition leader and acks=all waits for every in-sync replica.

Layer 3The machine

In RabbitMQ every queue is an Erlang process, scheduled on the BEAM's per-core schedulers. One queue's throughput is bounded by one process handling messages serially, typically tens of thousands per second, so throughput comes from many queues, not one big one. Messages pass between connection, channel and queue processes with an internal credit-flow scheme, so a slow queue process slows the publishing connection rather than letting mailboxes grow without limit.

A quorum queue confirm costs a Raft round trip plus a write to the log, which RabbitMQ syncs to disk in batches: roughly 1 to 2 ms across availability zones before disk time. Kafka, by default, does not fsync per write. It acknowledges once the record is in the page cache of all in-sync replicas and relies on replication rather than the disk for durability; its docs advise leaving the forced-flush settings off (Kafka topic configs), which is why Kafka sustains hundreds of MB per second per broker with sequential writes.

The broker's memory is the shared resource every tenant competes for. A RabbitMQ node raises a memory alarm at its high watermark, 60% of detected RAM by default (RabbitMQ memory). Memory and disk alarms are cluster-wide: once one node is over, every node blocks every publishing connection, while connections that only consume keep going (RabbitMQ alarms). A single unconsumed queue growing to millions of messages on one node can therefore stall unrelated producers on all of them, which is the strongest argument for length limits on every queue.

Why it's built this way

  1. Producers should not track consumers or hold undelivered messages, so an intermediary takes custody and confirms it.
  2. Custody means storage, and storage is finite, so brokers need limits, expiry, and a defined fate for messages that hit them.
  3. Tracking every message enables rich routing but costs bookkeeping per message, so high-volume systems moved to logs that track one offset per consumer.
  4. A shared broker turns one tenant's backlog into everyone's memory pressure, so brokers block publishers at the edge and isolate tenants with per-queue limits.

Diagram

flowchart LR
  P["Producer"] -->|"publish key=pay"| X["Exchange"]
  X -->|"binding pay"| Q1["payments queue (max 2, drop-head, TTL 10s)"]
  X -->|"binding pay"| Q2["audit queue (max 5, reject-publish)"]
  X -->|"no binding"| AE["Alternate queue"]
  Q1 -->|"maxlen or expired"| DLQ["Dead letter queue"]
  Q2 -->|"full: confirm nack"| P
  Q1 --> C["Consumer"]

Worked example

A food-delivery company runs one RabbitMQ cluster of three nodes with 16 GB of RAM each, shared by 40 teams. Orders publish 3,000 messages per second at dinner peak, average 2 KB.

The promotions team's consumer crashes on Friday at 18:00. Its queue receives 1,500 messages per second and nobody notices. After an hour it holds 5.4 million messages, about 11 GB of payload. Messages are paged to disk, but the queue index and in-flight data push the node's memory past its high watermark. The alarm is cluster-wide, so all three nodes block publishing connections. The order service stops being able to publish, even though it connects to a different node. A promotions outage has become an ordering outage.

The fix is policy, set centrally. Promotions are worthless after 15 minutes, so the queue gets x-message-ttl of 900,000 ms, which caps the backlog at 15 × 60 × 1,500 = 1.35 million messages, about 2.7 GB. That is still too much for a shared node, so x-max-length-bytes of 512 MB with drop-head bounds it at about 260,000 messages, dead-lettering the overflow to a small capped queue for inspection. The order queues, where every message matters, get reject-publish instead: if they fill, producers receive basic.nack and can retry or alert, rather than losing orders silently. Finally, an alarm on consumer count reaching zero for any queue catches the crash in the first minute.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Smart broker (RabbitMQ, ActiveMQ, Service Bus)Per-message routing, TTL, priorities, DLX, simple consumersPer-message bookkeeping, deep backlogs hurt, per-queue ceilingTask distribution and routing at moderate volume
Log broker (Kafka, Pulsar, Redpanda)Huge throughput, replay, many consumers cheaplyNo per-message retry or delay, partitions limit parallelismEvent streams, CDC, analytics feeds
Managed broker (SQS/SNS, Google Pub/Sub, Amazon MQ)No operations, elasticFewer knobs, per-request cost, vendor semanticsTeams without broker expertise
Brokerless (ZeroMQ, NATS core)Lowest latency, no central storageNo durability, peers handle failureEphemeral signals inside one system

Pitfalls in production

Leadership lens

Check yourself

  1. A RabbitMQ publish to an exchange with no matching binding returns no error. Why, and how do you detect it?

    Show answer

    By default, AMQP 0-9-1 discards unroutable messages, and a confirm only means the broker handled the publish. Set mandatory so the broker returns it, or configure an alternate exchange and monitor it.

  2. When should a queue use drop-head rather than reject-publish?

    Show answer

    Drop-head suits data whose value decays, such as telemetry or promotions, where the newest message matters most. Reject-publish suits data that must not be lost silently, because it pushes the problem back to the producer as a nack.

  3. Why can one team's stalled consumer stop another team's producer on the same RabbitMQ cluster?

    Show answer

    The backlog grows one node's memory past its high watermark, and because memory alarms are cluster-wide, every node then blocks all publishing connections, not just the one feeding the backlog.

  4. Why is a log broker cheaper per message than a smart broker?

    Show answer

    It stores one offset per consumer group and partition instead of per-message delivery state, so there is no per-message ack write, and reads are sequential and cache-friendly.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →