Publish-subscribe messaging

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

In publish/subscribe, a publisher sends a message to a topic, and every subscription to that topic gets its own copy. The publisher does not know who, or how many, the subscribers are.
Fan-out happens across subscriptions; inside one subscription, consumers usually compete like a message queue (see the Message Queues lesson). That pairing is the core idea in Google Pub/Sub, SNS with SQS, Kafka consumer groups and MQTT 5 shared subscriptions.
There are two families. Ephemeral pub/sub (Redis PUBLISH, NATS core, MQTT QoS 0) delivers only to subscribers connected right now. Durable pub/sub keeps a backlog per subscription, so a subscriber that is down catches up later.
Subscribers choose what they get by topic hierarchy with wildcards, or by attribute filters. Matching has to be fast, because it runs once per message for every subscription.
Fan-out multiplies cost: bandwidth, copies or cursors, and retention that the slowest subscriber pins.

How it works

Topics, subscriptions and subscribers

Three names keep the model straight. A topic is the named stream publishers write to. A subscription is a named, independent view of that topic with its own position and its own unacknowledged messages. A subscriber is a process attached to a subscription. When the billing team and the analytics team both need order-events, each gets its own subscription; billing can run five consumers that share the work, and analytics can be down for an hour without touching billing.

This is the fan-out problem shape. Kubernetes shows it in a second domain: controllers watch resources through the API server, each watch carries its own resourceVersion cursor, and a watcher that falls too far behind gets 410 Gone and must relist. With etcd 3, the history kept is the last 5 minutes by default (Kubernetes API concepts). That is a subscription whose cursor fell off the end of retention.

Ephemeral versus durable

Ephemeral pub/sub is a router. Redis PUBLISH returns the number of clients that received the message, and a client that reconnects one second later has missed it forever; Redis documents its pub/sub as at-most-once (Redis pub/sub). It is ideal for signals where only the latest state matters: cache invalidation hints, presence, live cursor positions, "wake up and check the database" nudges.

Ephemeral fan-out still has a scaling trap. In a Redis Cluster, a classic PUBLISH is broadcast to every node over the cluster bus, so adding shards adds traffic instead of capacity. Redis 7.0 added sharded pub/sub (SPUBLISH, SSUBSCRIBE), which hashes each channel to one slot and keeps its messages inside that shard.

Durable pub/sub stores messages until each subscription acknowledges them or retention expires. A new subscription normally starts at "now": Google Pub/Sub does not deliver messages published before the subscription existed (Google Pub/Sub subscriptions), and a Kafka consumer group with auto.offset.reset=latest starts at the end of the log. The usual mistake is deploying the producer before the subscription exists and losing the first hour of events.

Choosing what to receive

Topic hierarchies with wildcards encode routing into the topic name. MQTT uses / as the separator, + for exactly one level and # for any number of trailing levels, so factory/+/temp matches factory/line3/temp, and factory/# matches everything under factory, including factory itself. NATS uses . with and >; AMQP topic exchanges use . with and # (see the RabbitMQ lesson). MQTT also hides topics that start with $, such as $SYS/broker/load, from filters whose first level is a wildcard, so a # subscriber does not drown in broker internals (MQTT 5.0).

Attribute filters match on message metadata instead, such as SNS filter policies ({"region": ["eu-west-1"]}, see SNS message filtering) or Google Pub/Sub filter expressions. They keep topic names simple but are evaluated per message per subscription.

Retained messages solve the late-joiner problem for state. An MQTT publish with the retain flag replaces the broker's stored "last value" for that topic, and a new subscriber immediately receives the retained message for every topic its filter matches. A thermostat display that subscribes to home/+/temp gets every room's current temperature at once instead of waiting for the next reading.

Push or pull

Durable systems deliver either by pull (subscribers fetch, which gives natural flow control) or by push (the broker calls an HTTPS endpoint, Lambda or queue, and treats a 2xx as an ack). Push is convenient across organisational boundaries and behaves much like webhooks (see the Webhooks lesson); pull scales better for heavy consumers because they control their own rate.

Down the stack

Layer 1The contract

Durable pub/sub promises that every message published after a subscription is created is delivered at least once to that subscription, independently of every other subscription, until it is acked or ages past retention. In Google Pub/Sub that is seven days by default, configurable from 10 minutes to 31 days, and an unacked message is redelivered after its ack deadline, 10 seconds by default and at most 600 (Pub/Sub API reference). Order is not promised unless you use an ordering key or partition key, and then only per key and per subscription. One subscriber's failure must not slow or lose messages for another. Ephemeral pub/sub promises only at-most-once delivery to subscribers connected at publish time. Neither promises that subscribers see a message at the same moment.

Layer 2The mechanism

There are two ways to implement fan-out. Copy on publish writes a copy, or a reference, into each subscription's own queue: SNS delivering to SQS queues, a RabbitMQ fanout exchange enqueueing into each bound queue. Each subscription then has normal per-message ack state, filtering and dead-lettering. Shared log with cursors writes the message once and gives each subscription a position. Kafka stores each group's committed offset in the internal __consumer_offsets topic. Pulsar keeps a cursor per subscription with a mark-delete position plus ranges of individually acknowledged messages beyond it, so out-of-order acks do not force redelivery (Pulsar messaging concepts).

Wildcard matching uses a trie keyed by topic level. Brokers such as Mosquitto and EMQX store subscriptions in a tree where each node has children for literal levels plus + and # edges. Matching factory/line3/temp walks one path per level and branches at most into the wildcard edges, so cost grows with topic depth, not with the number of subscriptions.

MQTT delivery state is per client session. QoS 1 uses PUBLISH and PUBACK and may duplicate; QoS 2 adds PUBREC, PUBREL and PUBCOMP so each side can discard duplicates by packet identifier. A session started with Clean Start = 0 and a session expiry interval keeps queueing QoS 1 and 2 messages while the client is offline.

Layer 3The machine

Fan-out multiplies egress bytes: a 1 KB message to 10,000 subscribers is 10 MB on the wire, and the NIC and TLS encryption pay for every copy. Copy on publish also multiplies storage writes; a shared log writes once and lets readers share the page cache, and Kafka sends file data to sockets with sendfile without copying it through user space. The price is that a subscriber reading hours behind pulls cold segments from disk and evicts hot ones that up-to-date readers need.

Slow ephemeral subscribers are handled brutally, by design. Redis buffers outgoing pub/sub messages per client and, with the default client-output-buffer-limit pubsub 32mb 8mb 60 (redis.conf), disconnects a client whose buffer exceeds 32 MB, or stays over 8 MB for 60 seconds. The messages in that buffer are lost. The broker protects its memory, and the subscriber has to resynchronise from a source of truth.

Why it's built this way

Producers should not change when a new consumer appears, so they address a topic, not recipients.
Consumers run at different speeds and fail independently, so each subscription needs its own position and ack state.
Copying every message N times is expensive in disk and memory, so high fan-out systems store once and keep N cursors, while systems needing per-message filters and DLQs per subscriber accept the copies.
Storage is finite, so retention is bounded, and the system must choose between dropping a slow subscriber's backlog and letting it pin storage for everyone. Mature systems drop the subscriber, not the stream.

Diagram

flowchart LR
  P1["Order service"] --> T["Topic order-events (log)"]
  T --> S1["Sub billing (cursor 1042)"]
  T --> S2["Sub analytics (cursor 870)"]
  T --> S3["Sub emails (filter: type=placed)"]
  S1 --> C1["Billing worker 1"]
  S1 --> C2["Billing worker 2"]
  S2 --> C3["Analytics job"]
  S3 --> C4["Email sender"]

Worked example

A sports app sends live scores. Matches publish 50 updates per second in total, each about 400 bytes, on topics like scores/football/m123. Two million phones are connected to 40 gateway servers, 50,000 each, and a cup final has 500,000 watchers.

If each phone were a durable subscription, the broker would track 2 million cursors and back up messages for phones that went to sleep. Instead the design uses two tiers. Each gateway holds one subscription to scores/# on the internal broker, so a goal in the final is published once and fanned out 40 times. Each gateway then fans out to its own sockets: on average 12,500 local watchers of the final, 5 MB per goal per gateway, about 40 ms at 1 Gbit/s. The heavy fan-out happens in the cheap, ephemeral last mile.

Phones that reconnect do not replay missed updates; the score is state, not history. The broker retains the last message per match topic, so a reconnecting gateway, or a phone that opens the app mid-match, gets the current score immediately. The events that truly need durability, such as push notifications for goals, statistics and the archive, are three durable subscriptions on the internal log. If the archive job stops for 2 hours, it falls 50 × 400 B × 7,200 s, about 144 MB, behind: trivial to retain and replay.

In the wild

Discord, 2017: fan-out from one process to 30,000 sessions. Each Discord server (guild) is an Erlang process that sends every event to every connected session process. That was fine for small groups, but busy guilds reached about 30,000 concurrent users, and each send cost 30 to 70 microseconds, so publishing one event took 900 ms to 2.1 s and guilds fell behind at peak. Discord's Manifold library groups recipients by node and sends one batch per node, where local workers finish the fan-out, keeping order while spreading the CPU cost (Discord).
AWS, 2020: the stream everyone subscribes to goes down. Adding capacity to Kinesis Data Streams front-end servers pushed each server past the operating system's thread limit, because every server kept a thread for each of its peers. Kinesis was impaired for about 17 hours, and the services built on it as a shared event backbone were hit too: Cognito, CloudWatch, Lambda and EventBridge, and through EventBridge, ECS and EKS. Fixes included fewer, larger servers, thread alarms and faster cellularisation (AWS).
GitHub, 2018: push subscribers miss a window. After a 43-second partition forced a cross-coast database failover, GitHub ran degraded for 24 hours. More than 5 million webhook events queued, and about 200,000 webhook payloads expired before they could be delivered, so those subscribers never received them (GitHub).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Ephemeral (Redis pub/sub, NATS core, MQTT QoS 0)	Microsecond routing, no storage, simple	Offline or slow subscribers lose messages	Signals, presence, live state with a source of truth elsewhere
Copy per subscription (SNS to SQS, RabbitMQ fanout)	Per-subscriber acks, filters, DLQs, retry	Storage and requests multiply with subscriber count	Few subscribers, each with its own retry and failure policy
Shared log with cursors (Kafka, Pulsar)	Write once, replay, cheap to add subscribers	Ordering and parallelism per partition, coarse acks	Many subscribers, high volume, replay needed
Managed pub/sub (Google Pub/Sub)	No capacity planning, push or pull, filters	Per-message cost, limited replay window	Cloud-native services without a streaming platform team

Pitfalls in production

Subscription created after the producer starts. Messages published in between are gone for that subscription. Create subscriptions as part of infrastructure, before deploying producers, and backfill from the source of truth if needed.
One slow subscriber, invisible. Analytics falls days behind and its backlog expires silently. Alarm on oldest unacked message age per subscription, not on the topic.
Over-broad wildcards. A debugging subscriber on # receives the whole firehose and falls over, or a filter matches far more than intended after topic names change. Review topic naming as an API and restrict wildcard subscriptions with ACLs.
Ephemeral treated as durable. A cache-invalidation channel on Redis pub/sub silently drops messages when a client reconnects or exceeds its buffer limit, and stale data lives forever. Pair ephemeral signals with TTLs or periodic resync.
Fan-out cost surprises. One SNS topic feeding 30 SQS queues turns 1,000 publishes per second into 30,000 deliveries. Count subscriptions when estimating cost and load.

Leadership lens

Treat topics and their message schemas as public APIs with owners, versioning rules and a deprecation process. Subscribers you have never heard of will depend on them.
Ask each team which subscriptions it owns and what lag it tolerates; make that per-subscription age the alarm.
Decide explicitly whether each stream is state (latest value wins, ephemeral and retained is enough) or history (every event matters, needs durable subscriptions). The two cost very different amounts.
Limit who can create wildcard subscriptions on shared topics; a single careless consumer can become the slowest subscriber that pins retention for everyone.

Check yourself

Billing and analytics both subscribe to order-events. Billing runs three consumers. How many times is each message processed?
Show answer
Once by billing, by whichever of its three consumers gets it, and once by analytics. Fan-out is across subscriptions; competition is within one.
Why does an MQTT broker match subscriptions with a trie rather than testing each filter?
Show answer
Testing every filter costs time proportional to the number of subscriptions per message. A trie walk costs time proportional to topic depth, plus the wildcard branches at each level.
When is a retained message a better fix for a late subscriber than durable replay?
Show answer
When the topic carries state where only the latest value matters, such as a temperature or a score. The newcomer needs the current value, not every past change.
Why can one lagging subscriber hurt others on a shared log?
Show answer
It reads cold data from disk, evicting page-cache pages that up-to-date readers need, and if retention is tied to the slowest cursor it forces the log to keep growing.

Sources and further study

Primary

Eugster et al., The Many Faces of Publish/Subscribe (ACM Computing Surveys, 2003): the classic survey that frames pub/sub as decoupling in space, time and synchronisation, and compares topic and content filtering.

Systems docs

Redis pub/sub: at-most-once delivery, pattern subscriptions and sharded pub/sub in Redis 7.
Google Pub/Sub subscription API reference: retention, ack deadlines, filters and exactly-once delivery as fields you set.
Kubernetes API concepts: watches, resource versions and the 410 Gone relist.
Apache Kafka topic configuration: time-based retention (7 days by default), which is what a lagging consumer group races against.

Books

Designing Distributed Systems (2nd ed.), Brendan Burns, Ch 12 "Event-Driven Batch Processing": its "Publisher/Subscriber Infrastructure" section, with a hands-on Kafka deployment.
Designing Data-Intensive Applications (2nd ed.), Martin Kleppmann and Chris Riccomini, Ch 12 "Stream Processing": its "Transmitting Event Streams" section covers delivering one stream to many consumers, by broker and by log.

Other curricula

Azure Architecture Center: Publisher-Subscriber pattern: one input channel, one output channel per subscriber, and a checklist of issues (ordering, poison messages, expiry, schema evolution, backpressure).
roadmap.sh: System Design: lists Publisher/Subscriber as a messaging pattern next to queue-based load leveling and competing consumers.

Watch

What is the Publisher Subscriber Model? (Gaurav Sen): why services move from direct calls to pub/sub, and what that costs.
Pub Sub Model, MQTT Essentials Part 3 (HiveMQ): decoupling in space, time and synchronisation, as MQTT applies it.

Free resources

Google Cloud Pub/Sub overview: topics, subscriptions, push and pull, and the delivery contract.
Google Cloud Pub/Sub subscriptions: subscription types, acknowledgement deadlines, retention and filtering.
MQTT Version 5.0 (OASIS standard): the normative rules for topic filters, wildcards, retained messages and shared subscriptions.
Apache Pulsar messaging concepts: subscription types, cursors and acknowledgement in a log-based system.
Amazon SNS message filtering: attribute-based filter policies as an alternative to topic hierarchies.
NATS subject-based messaging: another hierarchy and wildcard syntax, with naming advice.

Open in the interactive app: practice, quiz, mock interview →