Handling read-heavy workloads
Open in the interactive app: practice, quiz, mock interview →In one minute
- Most products read far more than they write, often 100 to 1. Scaling reads means serving answers from many cheap copies and making each answer cheap to compute.
- Think of the read path as a funnel: browser, CDN, in-process cache, distributed cache, read replicas, primary. Each layer removes a fraction of traffic, and the fractions multiply.
- Every copy is stale by some amount. A strong answer sets a staleness budget per data type and keeps read-your-writes for the person who just wrote.
- When reads are expensive rather than merely frequent, precompute: denormalised rows, materialised views, read models fed by change data capture.
- Designs fail in the tail: hot keys, stampedes, lag spikes and cold caches. Name them before the interviewer does.
How it works
The read funnel
Each layer answers what it can and passes the rest inward. If 1 million requests per second arrive, a CDN that answers 80 percent leaves 200,000; a cache tier that answers 95 percent of those leaves 10,000 for the database. Raising any layer's hit ratio from 95 to 99 percent cuts the load behind it by a factor of five, which is why interviewers want hit-ratio arithmetic, not a list of boxes. The Caching chapter covers patterns and invalidation inside one layer; this chapter is about combining layers.
Read replicas and lag
A replica replays the primary's log. Replicas multiply read throughput, but most replication is asynchronous, so a replica can be milliseconds or minutes behind. Three guarantees matter:
- Read-your-writes. After I post a review, I see it (Jepsen). Return the commit position (a Postgres LSN or a MySQL GTID) with the write, store it in the session, and route my reads only to replicas that have replayed past it, or to the primary.
- Monotonic reads. I never see a value disappear on refresh. Pin a session to one replica, or carry the highest position seen so far.
- Bounded staleness. Everyone sees data at most N seconds old. Remove replicas from rotation when lag exceeds N.
Make each read cheap
A page that joins five tables and aggregates 2,000 rows is expensive even when rare. Denormalise: keep the average rating and review count on the restaurant row, updated on each review write. Use materialised views for heavier aggregates. At the far end is CQRS: a separate read model (search index, per-user feed table, page-shaped document) maintained from the write side's events. The cost moves to the write path, and each precomputed copy is one more thing to keep in step (Azure Materialized View pattern).
Protect the origin
When a popular key expires, every server misses at once; request coalescing (one in-flight load per key, others wait) turns 5,000 identical misses into one query, and stale-while-revalidate serves the old value while one request refreshes it. Facebook's memcache leases do the same job: a server hands out a fill token for a key at most once every 10 seconds, which cut the peak database query rate on herd-prone keys from 17,000 to 1,300 per second (Nishtala et al., NSDI 2013). A viral item can saturate one cache shard however many exist; the Hot Keys chapter covers splitting and replicating such keys. A cache that restarts empty, or fails, sends its whole miss stream inward at once, so AWS advises capping calls to dependencies and testing with the cache switched off (AWS Builders' Library). Load shedding in front of the database turns a cold cache into slow pages instead of an outage.
Down the stack
Layer 1The contract
The read path promises a latency envelope (for example p99 under 100 ms at the edge), a staleness bound per data class (prices within 60 seconds, profile photos within 5 minutes, account balance never stale), and read-your-writes for the author of a change. It does not promise that two users see the same value at once, or freshness across regions. A CDN hit within s-maxage is correct under this contract; a read missing the author's own write is not.
Layer 2The mechanism
Postgres streaming replication. On commit the primary writes WAL; a walsender process streams records to the replica's walreceiver, and the startup process replays them into data pages. pg_current_wal_lsn() on the primary and pg_last_wal_replay_lsn() on a replica give positions a router can compare. When a replica query conflicts with replay (it reads rows being vacuumed away), replay waits up to max_standby_streaming_delay (default 30 seconds), then cancels the query: a direct trade between query success and lag (Postgres docs). The other escape, hot_standby_feedback, stops the primary's vacuum from removing rows the replica still reads, which avoids cancellations but bloats tables on the primary (Postgres hot standby). MySQL offers the same idea with GTIDs: WAIT_FOR_EXECUTED_GTID_SET blocks a replica read until a given transaction has applied, returning 1 if an optional timeout expires first (MySQL 8.4 docs).
HTTP caching. Cache-Control: public, s-maxage=60, stale-while-revalidate=30 lets shared caches keep a response for 60 seconds, then serve it stale for up to 30 more while revalidating in the background. s-maxage is defined in RFC 9111, which replaced RFC 7234 in 2022 (RFC 9111); stale-while-revalidate comes from the RFC 5861 extension. CDNs collapse concurrent misses for a URL into one origin fetch (Fastly docs), and tiered caching lets only upper-tier data centers ask the origin, so the origin sees one request per region, not per point of presence (Cloudflare docs).
Cache tiers with ordering. Facebook's TAO, which served a billion reads and millions of writes per second, puts follower caches in front of leader caches per region. Followers forward writes to the leader, which updates the database and invalidates followers asynchronously; the follower that handled a write is updated synchronously on the leader's reply, which gives the writer read-your-writes within that tier while other users converge (Bronson et al., USENIX ATC 2013).
Materialised views. Postgres REFRESH MATERIALIZED VIEW CONCURRENTLY avoids blocking readers but still recomputes the whole query; incremental maintenance from change events updates only affected rows.
Layer 3The machine
Latency per layer spans five orders of magnitude: an in-process map lookup is about 100 ns, a Redis hit in the same zone 0.2 to 0.5 ms, an indexed database read 1 to 5 ms, and a cross-region hop 60 to 150 ms. Postgres replays WAL in a single startup process, so a replica on a 32-core box can fall behind a primary that writes in parallel, even while the replica's CPUs look idle. Since Postgres 15, recovery_prefetch (default try) reads ahead blocks that upcoming WAL records will touch, which hides some of that I/O wait but does not add replay parallelism (Postgres docs). Replicas must also absorb every write's I/O: a replica on slower disks than the primary lags under write bursts. A new replica has a cold buffer pool, so its first reads hit storage: roughly 100 us per 4 KB random read on NVMe, several ms on network block storage. Bandwidth often binds before CPU: 100,000 responses per second of 20 KB is about 16 Gbit/s of egress.
Why it's built this way
- A read changes nothing, so it can be answered by any copy, and copies are cheap compared with making one machine faster.
- Copies are updated after the write commits, because waiting for every copy would put the slowest replica on every write's latency.
- Asynchronous copies diverge, so systems carry positions (LSN, GTID, version) that let a reader demand "at least this fresh".
- Recomputing an aggregate per read wastes work, so read-dominated systems precompute once per write.
- A failing layer dumps its traffic inward, so inner layers need coalescing and shedding.
Diagram
flowchart LR
U["1M req/s"] --> CDN["CDN edge (80% hit)"]
CDN -->|"200k"| APP["App server L1 cache"]
APP -->|"miss"| RC["Redis cluster (95% hit)"]
RC -->|"10k"| RT["Replica router (LSN check)"]
RT -->|"fresh enough"| R1["Replica 1"]
RT -->|"fresh enough"| R2["Replica 2"]
RT -->|"author or lagging"| P["Primary"]
P -->|"WAL stream"| R1
P -->|"WAL stream"| R2Worked example
A restaurant review site has 300,000 page reads per second at peak and 500 review writes per second; 70 percent of traffic is anonymous.
Anonymous pages (210,000 per second) are cached at the CDN with s-maxage=60; at 95 percent edge hits, 10,500 per second reach the origin. Logged-in pages (90,000 per second) split into a shared restaurant fragment and a small personal one, so the shared part still hits Redis.
The restaurant fragment is one denormalised document (hours, rating average, review count, top 10 reviews) of about 8 KB; 400,000 active restaurants take 3.2 GB of Redis. At 97 percent hits on about 100,000 origin requests, the database sees about 3,000 reads per second, within two replicas. Without the stored count and rating sum, each miss would aggregate up to 2,000 review rows: 6 million row reads per second.
Read-your-writes: a review write returns its LSN, kept in the session for 30 seconds, and the router sends that user to a replica past it. Lag is usually under 50 ms, so the primary sees at most about 1,000 extra reads per second.
In the wild
- Slack, 2022: a cache refill that the database could not survive. A gradual Consul agent upgrade dropped memcached nodes from the catalog, and they were replaced with empty ones, so the hit rate fell. Group DM membership was cached by channel, but Vitess was sharded by user, so every miss became a query to every shard. The database overloaded, could not refill the cache, and client retries kept it down; many users could not boot Slack that morning. Slack moved the query to a channel-sharded table and audited other cache-fronted scatter queries (Slack Engineering).
- Facebook, 2010: invalid-value handling turned into a read storm. A system that replaced cached config values it judged invalid met an invalid value in the database itself, so every client queried one cluster at hundreds of thousands of queries per second, and clients deleted cache keys on database errors, which fed the loop. Facebook had to take the site down for about 2.5 hours to stop the traffic and disabled the system (Facebook Engineering).
- Facebook, 2013: leases and a Gutter pool for a read-dominated cache. Facebook's memcache paper describes leases that allow one refill per key every 10 seconds, cutting herd-prone database load from 17,000 to 1,300 queries per second, and a Gutter pool of about 1% of servers that takes over for failed hosts so their misses do not land on the database (USENIX NSDI 2013).
Trade-offs
| Option | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| CDN with short TTL | Removes origin load, cuts latency globally | Public content only, purge delay | Anonymous or shared responses |
| In-process L1 | Nanosecond reads, absorbs hot keys | Per-server staleness, memory per instance | Tiny, very hot, tolerant data |
| Distributed cache | Shared, sub-ms, large capacity | Invalidation, stampedes, cold starts | Default for hot entities |
| Read replicas | Full query power, no invalidation code | Lag, write I/O on every replica | Complex queries, moderate scale |
| Denormalised rows or materialised views | Cheap reads for expensive queries | Write amplification, consistency of copies | Aggregates read far more than written |
| Separate read model (CQRS) | Each read shaped for its page | Two systems, eventual consistency | Very different read and write shapes |
Pitfalls in production
- Replica lag spikes. Backfills or long analytical queries push lag to minutes. Throttle backfills, isolate analytics, and drain replicas above the lag budget.
- Read-your-writes lost behind a load balancer. The next read hit a random lagging replica. Carry an LSN in the session and route on it.
- Caches hide missing capacity. A database sized for 3 percent misses meets 100 percent after a flush. Load-test cold, warm critical keys, and shed load.
- Personalised responses cached publicly. A CDN serves one user's page to another. Mark them
privateand strip cookies on cacheable routes. - Full view refreshes. A refresh every minute becomes the heaviest query. Maintain incrementally from change events.
Leadership lens
- Ask for a staleness budget per data class in the design doc, signed off by product, before choosing layers.
- Make replica lag an SLO, and decide in advance which reads fall back to the primary when it is breached.
- Every precomputed copy needs an owner and a rebuild procedure; count read models as systems in your on-call map.
- Track cost per million reads per layer; moving traffic outward to the CDN is often the cheapest capacity you can buy.
Check yourself
Your CDN hit ratio falls from 90 to 80 percent. What happens to origin load?
Show answer
Misses go from 10 to 20 percent, so origin traffic doubles.
Why can a replica fall behind even though its CPUs are mostly idle?
Show answer
Replay is often single-threaded (Postgres replays WAL in one process) while the primary commits in parallel, so the replica is bound by one core and its I/O latency.
How do you give a user read-your-writes without sending all their reads to the primary?
Show answer
Keep the write's log position in the session and route reads to any replica that has replayed past it, falling back to the primary only when none has.
When should you denormalise instead of adding replicas?
Show answer
When each read is expensive (large joins or aggregates), because replicas multiply capacity but every replica still pays the full query cost.
Sources and further study
Primary
- RFC 9111: HTTP Caching: the current definition of
max-age,s-maxageand when a cache may serve stale responses. - Scaling Memcache at Facebook (NSDI 2013): leases, regional pools and the Gutter pool for a read-heavy cache tier.
Systems docs
- PostgreSQL: replication settings:
max_standby_streaming_delay,hot_standby_feedbackand the other standby knobs. - MySQL 8.4: GTID functions:
WAIT_FOR_EXECUTED_GTID_SETfor read-your-writes on replicas. - Cloudflare Tiered Cache: how lower and upper tiers cut origin fetches.
- Fastly: request collapsing: one origin fetch for many concurrent misses, and when it backfires.
Books
- Designing Data-Intensive Applications (1st ed.), Martin Kleppmann, Ch 5 "Replication": the section on problems with replication lag covers read-your-writes and monotonic reads.
- Understanding Distributed Systems (2nd ed.), Roberto Vitillo, chapters "HTTP caching", "Content delivery networks" and "Caching": the funnel layer by layer.
Other curricula
- donnemartin/system-design-primer: frames read scaling as replication, denormalization and cache-aside, with the 100:1 to 1000:1 read ratio as the reason.
- Azure Architecture Center: Cache-Aside pattern: frames the cache as loaded on demand by the application, with expiry and consistency as the main costs.
Watch
- Scaling Instagram Infrastructure (InfoQ): caches, replicas and thundering herds across regions at very large read volume.
- Scaling Memcache at Facebook (NSDI talk): look-aside caching, leases and cross-region invalidation for a read-dominated workload.
- When to Shard Database vs Replicate? (KodeKloud): when replicas are enough and the lag they introduce.
Free resources
- TAO: Facebook's Distributed Data Store for the Social Graph (USENIX ATC 2013): leader and follower cache tiers with read-after-write for the writer.
- Caching challenges and strategies (AWS Builders' Library): cold caches, sizing and what happens when a cache layer fails.
- Hot standby (PostgreSQL docs): querying replicas, replay conflicts and their settings.
- RFC 5861: HTTP Cache-Control extensions for stale content: the definition of
stale-while-revalidateandstale-if-error. - Read your writes (Jepsen consistency models): the precise guarantee and how it relates to other session models.