Engineering Atlas › System Design Interviews

Fan-out on write versus on read

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Push: fan-out on write

When Alice posts, a fan-out service reads her follower list and appends the post ID to each follower's timeline (a capped list in Redis or a wide-row store). Reading a timeline is one lookup, then a batch fetch of post bodies. Cost lands on the write: a user with 2,000 followers causes 2,000 inbox writes. It works well while follower counts are modest and reads vastly outnumber writes, which is the case for most social products. Twitter's 2013 numbers show the shape: about 300,000 timeline reads per second against about 6,000 write requests per second, with each home timeline an 800-entry list of tweet IDs in Redis, replicated three times (High Scalability, from Raffi Krikorian's talk).

Pull: fan-out on read

Posts are stored once, indexed by author. A reader's timeline query fetches recent posts from each followee and merges them by time. Writes cost one insert. Reads cost one query per followee, so a user following 1,000 accounts triggers a scatter-gather on every refresh. Pull is also how you handle recipients who rarely log in: no work is done for them until they show up.

Hybrid

Mark senders above a threshold (say 100,000 followers) as high fan-out. Their posts are not pushed; they are stored in the author's outbox. A reader's timeline is their precomputed inbox merged with the recent outboxes of the few high-fan-out accounts they follow. Typical readers follow a handful of such accounts, so the read-time merge is small, and the write-time cost of a celebrity post drops from tens of millions of inbox writes to one. Twitter reached this hybrid after pure push left a 31-million-follower account's tweets taking up to 5 minutes to reach everyone, and replies sometimes arriving before the tweet they answered (High Scalability). Kleppmann uses the same Twitter example to define load parameters in DDIA Ch 1.

Two further refinements are common. Skip pushing to inactive users (not seen in 30 days, Twitter's own definition of active) and rebuild their timelines on next login. And cap inboxes at a few hundred entries, falling back to pull for deep scrolling.

Real-time delivery

For chat, collaborative editing or live dashboards, recipients hold open connections (WebSocket, SSE, long-poll) to gateway servers. A presence or session registry maps user to gateway. When an event arrives, a router looks up recipients' gateways and publishes to those gateways only, typically through a pub/sub system keyed by gateway or by channel, which decouples senders from however many receivers exist (Azure Publisher-Subscriber pattern). Large channels (a 100,000-member Slack-like channel) invert the problem: rather than resolving each member, every gateway subscribes to the channels its connected users are in, and the event is published once per channel. Slack works this way: channel servers own channels by consistent hashing, gateway servers in each region hold users' channel subscriptions, and a message reaches clients worldwide in about 500 ms (Slack Engineering).

In one huge live group the cost is quadratic, because more members means both more senders and more recipients per message: 1,000 online users sending one message each make 1 million deliveries, and 100,000 make 10 billion. Discord's answer was to treat members who are not looking at the server as passive sessions that get far fewer updates (about 90% of connections in large servers), and to put relay processes between the guild and up to 15,000 sessions each to split the fan-out and permission checks (Discord).

How it shows up in interviews

Triggers: design Twitter, Instagram feed, Facebook News Feed, WhatsApp or Slack groups, a notification system, a live comments system, a stock ticker. A strong answer computes fan-out volume, chooses push, pull or hybrid with a threshold, describes the inbox data structure and its cap, handles inactive users, and separates persistent fan-out (timelines) from transient fan-out (online sockets). Typical probes: what happens when a user with 50 million followers posts, how long until the last follower sees it, how do you handle deletes and unfollows, and how do you order events.

Down the stack

Layer 1The contract

A push timeline promises that a post appears in each active follower's inbox within some delivery latency (often a target of seconds, with the largest pushed fan-outs taking longer), at least once, in roughly time order. It usually does not promise strict global ordering: two posts seconds apart may land in different orders in different inboxes, and the read path re-sorts by timestamp or Snowflake-style ID. Deletes are best effort and asynchronous, so a read path must filter against a deleted-posts check. Real-time delivery promises "at most once to the socket, durable elsewhere" (Redis Pub/Sub, a common router, is explicitly at-most-once (Redis docs)): if the client is disconnected, it catches up from the persistent store on reconnect.

Layer 2The mechanism

The write path enqueues a FanoutJob{post_id, author_id, created_at} on a partitioned log. Fan-out workers page through the follower list in chunks of, say, 1,000 IDs, and for each chunk issue a pipelined batch of LPUSH timeline:{uid} post_id followed by LTRIM timeline:{uid} 0 799 (keep the newest 800). Because jobs are chunked, a large fan-out runs on many workers in parallel, and a crash replays only the failed chunk; pushing the same post ID twice is harmless if the reader de-duplicates IDs. The read path takes the inbox IDs, merges in outbox IDs from followed high-fan-out accounts with a k-way merge on time-sortable IDs, filters deleted or blocked items, and multi-gets bodies from a post cache. For sockets, a gateway keeps a map of user_id -> connections, and the router uses Redis Pub/Sub, NATS or Kafka topics per gateway to deliver.

Layer 3The machine

A Redis LPUSH on a small list costs microseconds of CPU, but a separate network round trip is on the order of a few hundred microseconds inside a data centre. Without pipelining, 10,000 inbox writes serialised take seconds; with batches of 1,000 commands per round trip they take tens of milliseconds of wall time per worker. Pipelining also cuts the server's own cost, since many commands share one read() and one write() system call; Redis reports throughput rising towards ten times the unpipelined baseline, and suggests batches of about 10,000 commands so queued replies do not eat memory (Redis docs). Memory is the bigger bill: 800 entries of 8-byte IDs per user is about 6.4 KB raw, plus per-element overhead in Redis's listpack and quicklist encoding, so 200 million active users need terabytes of RAM for timelines. That is why inboxes hold IDs, not post bodies, and why inactive users are dropped. In process-per-connection runtimes the send itself costs: in Discord's Erlang guilds each send took 30 to 70 microseconds, so a publish to 30,000 sessions took 0.9 to 2.1 seconds until they batched sends per remote node (Discord). On the socket side, each idle WebSocket costs a file descriptor and kernel socket buffers of several kilobytes, so a gateway with a million connections spends gigabytes on buffers alone and needs ulimit -n and net.ipv4.tcp_rmem tuned.

Why it's built this way

Reads outnumber writes by orders of magnitude, so precompute on write to make reads one lookup. Follower counts follow a power law, so a few senders would dominate write cost, and they are pulled instead. Memory is expensive, so inboxes store compact IDs and are capped. Network round trips dominate per-write cost, so writes are batched and pipelined. Workers fail, so fan-out is chunked and idempotent.

Diagram

sequenceDiagram
    participant A as Author
    participant P as Post service
    participant L as Fanout log
    participant W as Fanout workers
    participant T as Timeline store
    participant R as Reader
    A->>P: create post
    P->>L: FanoutJob (if author below threshold)
    L->>W: chunks of 1000 follower IDs
    W->>T: pipelined LPUSH and LTRIM
    R->>T: read inbox IDs
    R->>P: read outboxes of followed celebrities
    R->>R: merge by time, filter, hydrate

Worked example

A social app has 150 million daily active users. They create 50 million posts per day, about 600 per second on average and perhaps 2,000 per second at peak. The median author has 200 followers; the mean is pulled up by large accounts.

With pure push, if the mean fan-out is 400, the system performs 600 x 400 = 240,000 inbox writes per second on average. Now add one account with 60 million followers posting: that single post is 60 million writes. At 500,000 writes per second of total fan-out capacity, it takes two minutes and delays everyone else's posts. With a hybrid threshold of 100,000 followers, roughly the top 0.01% of accounts are pulled. Their posts cost one write. A reader following 20 such accounts adds 20 outbox reads per timeline load, each served from cache and merged in under a millisecond of CPU.

Skipping users inactive for 30 days cuts fan-out by perhaps a third in a typical social product, since many followers are dormant. The inbox cap of 800 IDs keeps memory per active user under 10 KB including overhead, about 1.5 TB across 150 million users, sharded across a Redis cluster with replicas.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Fan-out on writeOne-lookup reads, low read latencyWrite amplification, slow for huge sendersModerate fan-out, read-heavy
Fan-out on readCheap writes, always freshExpensive scatter-gather readsFew readers or few followees
HybridBounded cost on both pathsTwo code paths, merge logicPower-law follower graphs
Per-channel pub/sub to gatewaysOne publish per channelGateways receive events for channels they might not needLarge live channels
Per-user routing to gatewaysPrecise deliveryRegistry lookups per recipientSmall groups, direct messages

Pitfalls in production

Leadership lens

Check yourself

  1. Why does pure fan-out on write fail for accounts with tens of millions of followers?

    Show answer

    One post becomes tens of millions of writes, which takes minutes and delays everyone else's deliveries.

  2. Why store post IDs rather than post bodies in inboxes?

    Show answer

    IDs are tiny, so inbox memory stays small, and edits or deletes only need to change the one stored body.

  3. How does the hybrid read path stay cheap?

    Show answer

    Readers follow only a few high-fan-out accounts, so merging their recent outboxes with the precomputed inbox is a small k-way merge over cached data.

  4. Why is fan-out work chunked into batches of follower IDs?

    Show answer

    Chunks parallelise across workers and make retries cover only a small piece, while pipelining each chunk amortises network round trips.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →