Engineering Atlas › System Design Fundamentals

Designing a cache that spans many nodes

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why one box stops being enough

A single cache node runs out of something: memory (a 1.5 TB working set does not fit in a 128 GB machine), throughput (Redis executes commands on one thread, roughly 100,000 to 200,000 simple operations per second without pipelining), network (a 25 Gbit/s NIC moves about 3 GB/s), or blast radius (when the only node restarts, every read hits the database).

This lesson is about the architecture once you split. How entries get in and out (cache-aside, write-through) and what to evict are separate topics; Consistent Hashing, Hot Keys and Design a Distributed Cache go deeper on placement, skew and a full design.

Three places to put the routing

Client-side sharding. Each application process holds the server list and hashes keys to servers itself. Memcached has always worked this way: the servers do not know about each other. It is one hop, but every language needs an identical hashing library, and processes with different configs disagree about where keys live.

A routing proxy. Applications talk to a local or shared proxy (Twemproxy, Facebook's mcrouter, Envoy's Redis proxy) that owns the pool layout. Mcrouter expresses policy as composable route handles: PoolRoute for a sharded pool, AllSyncRoute to write to several replicas, FailoverRoute to try a backup pool, WarmUpRoute to fill a cold pool from a warm one. It costs a hop of tens of microseconds and a fleet to run, but layout changes need no application deploys.

Server-side clustering. In Redis Cluster the nodes agree among themselves who owns which of 16,384 hash slots. A client may send a command to any node; a node that does not own the slot replies MOVED <slot> <host:port>, and a smart client caches the slot map so it usually goes straight to the right node. This avoids a proxy while keeping one source of truth, at the cost of smarter clients. Managed services often let you pick: Azure Managed Redis offers an OSS clustering policy, where clients follow MOVED and ASK themselves, and a proxy-based policy behind a single endpoint that costs a little routing overhead (Azure caching guidance).

Replication and failure

Cache replicas serve two purposes. For availability, a replica in another zone is promoted when its primary dies, so the shard does not go cold. For read capacity, reads of hot keys spread across replicas. Both are asynchronous in Redis, so a replica can serve a value the primary has already overwritten, and a promotion can lose acknowledged writes; the cluster specification says there is always a window in which writes can be lost during partitions (Redis cluster spec). That is acceptable for a cache only because the database still holds the truth.

Memcached has no replication of its own. Facebook's answer was the gutter pool: about 1% of servers that take over the keys of any failed server with short TTLs (Facebook NSDI 2013), so a dead node causes a burst of short-lived entries instead of a burst of database reads. The general lesson is to decide explicitly what a failed shard's traffic hits next.

Multi-key reads and fan-out

A product page that needs 40 keys issues one multiget per shard involved. With 16 shards, 40 random keys touch about 15 of them, and the page waits for the slowest. Adding shards makes this worse, not better: more shards means more round trips per page for the same work. The fix is to co-locate keys that are read together. Redis Cluster supports this with hash tags: only the part of the key between the first { and the next } is hashed, so {user:42}:profile and {user:42}:cart land in the same slot and can be fetched, or updated atomically, in one command.

This is the fan-out problem shape; it reappears when a search query scatters to 50 index shards, or an agent waits on ten parallel tool calls.

Near caches

A small in-process L1 in front of the distributed L2 removes the network hop for the hottest keys and shields their shard. The cost is a weaker consistency layer: invalidations must reach every process, or the L1 uses TTLs of seconds.

Down the stack

Layer 1The contract

A distributed cache promises much less than a database. Any GET may miss, even immediately after a SET, because of eviction, a restart or a failover. Writes are acknowledged by the primary before replication, so a failover can discard them. Reads from replicas can be stale by the replication lag, usually milliseconds but unbounded under load. Atomic multi-key operations exist only within one slot; Redis rejects a multi-key command spanning slots with CROSSSLOT. The Redis cluster specification says plainly that it does not guarantee strong consistency, so correct code treats every answer as a hint and every miss as normal.

Layer 2The mechanism

Redis Cluster maps a key to CRC16(key) mod 16384 using the XMODEM CRC (polynomial 0x1021), after applying the hash-tag rule. Each node keeps the full slot table and gossips PING and PONG messages on a cluster bus at the client port plus 10,000 (or cluster-port) (Redis cluster spec), carrying its view of slot ownership and a configuration epoch that resolves conflicting claims. A node that does not answer for cluster-node-timeout (15,000 ms by default) is flagged PFAIL (possibly failing); when a majority of primaries report it within twice the node timeout, it becomes FAIL, and one of its replicas wins an election and takes over its slots with a higher epoch.

Resharding moves one slot at a time. The source marks the slot MIGRATING and the target IMPORTING, and keys move with MIGRATE. During the move, a key already moved produces ASK <slot> <target> from the source. The client sends ASKING and then the command to the target, but does not update its slot map, because the slot still belongs to the source until the move finishes. Only MOVED changes the map. Clients refresh the whole map with CLUSTER SHARDS (or the older CLUSTER SLOTS). A connection that sends READONLY to a replica may read the slots that replica serves.

Key-by-key migration is no longer the only way. Redis 8.4 added CLUSTER MIGRATION IMPORT, run on the target, which copies whole slot ranges as a snapshot plus a replication stream; when the remaining lag drops under cluster-slot-migration-handoff-max-lag-bytes (1 MB by default), the source pauses writes, for at most 10 seconds by default, and ownership flips in one step (Redis docs). Valkey 9.0 shipped the same idea as CLUSTER MIGRATESLOTS in October 2025: the source keeps every key until the whole slot has moved, so there is no half-migrated slot for multi-key commands to trip over and big collections stream item by item instead of as one huge MIGRATE payload (Valkey). Clients still need to handle ASK, because older servers and tools use the classic path.

Layer 3The machine

A GET of a 1 KB value costs the Redis thread a few microseconds: parse the RESP request, one hash table lookup, copy into the output buffer, one write. Within an availability zone a round trip is about 100 to 300 microseconds; across zones it is closer to 1 ms, and AWS bills cross-zone traffic, including to ElastiCache, at $0.01 per GB in each direction (AWS EC2 pricing). So network, not CPU, dominates, and the levers are fewer round trips (pipelining and multiget turn 40 reads into one write and one read syscall per shard) and fewer bytes.

Connections are the hidden cost: a thousand application servers with 4 connections to each of 32 nodes is 128,000 TCP connections, each with kernel socket buffers and a Redis client struct. Memory has traps too: allocator fragmentation after churn in value sizes, and an RDB snapshot fork() that shares pages copy-on-write, so a write-heavy node can briefly need close to double its memory.

Why it's built this way

  1. RAM per machine and packets per core are fixed, so a large hot set must be partitioned.
  2. Partitioning needs a stable key-to-node map that survives membership changes, so the cluster uses a fixed slot space (or a hash ring) and moves slots, not keys one by one.
  3. Moving data while serving traffic means some keys are temporarily in two places, so the protocol separates a one-off redirect (ASK) from a permanent one (MOVED).
  4. Synchronous replication would double write latency for data the database already holds, so replicas are asynchronous and the contract admits loss.

Diagram

sequenceDiagram
  participant C as Client
  participant A as Node A (slots 0-8191)
  participant B as Node B (importing 866)
  C->>A: GET hello (slot 866)
  A-->>C: ASK 866 B
  C->>B: ASKING
  C->>B: GET hello
  B-->>C: value
  Note over C: slot map unchanged
  Note over A,B: migration finishes, B owns 866
  C->>A: GET hello
  A-->>C: MOVED 866 B
  Note over C: update slot 866 to B
  C->>B: GET hello
  B-->>C: value

Worked example

A catalogue service needs 800,000 reads per second at peak over 1.2 TB of hot values averaging 1 KB.

Memory. Keys, hash table entries and allocator overhead add roughly 25%, so plan for 1.5 TB. Nodes with about 100 GB usable each need 15; use 16 primaries, 1,024 slots each.

Throughput. 800,000 / 16 is 50,000 reads per second per primary, comfortably inside one Redis thread, and 50 MB/s of values per node is a small fraction of the NIC.

Replicas and zones. One replica per primary in a second zone gives 32 nodes. If clients read from whichever copy is in their own zone, no read crosses zones. If instead half the reads cross zones, that is about 400 MB/s, roughly 34,500 GB per day, which at $0.01 per GB in each direction costs about $690 per day, or $250,000 per year, for nothing.

Fan-out. A product page reads 40 keys. Spread randomly, they touch about 15 of the 16 shards. If each shard's p99 is 2 ms, the chance that a page hits at least one p99 response (The Tail at Scale) is 1 - 0.99^15, about 14%, so the page's p90 is already above the shard's p99. Grouping a product's keys under one hash tag, {product:981}, collapses the page to one or two shards.

Failure. A primary dies; detection takes 15 seconds plus an election. Meanwhile 50,000 reads per second for its slots fall through to a database sized for 20,000. Decide in advance: read from the replica, shed, or use a gutter pool.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Client-side shardingOne hop, no extra infrastructureIdentical hashing in every language, config skewFew languages, simple pools (classic Memcached)
Routing proxyCentral policy, replication, failover and warm-up without app changesExtra hop, proxy fleet to operateMany services and languages, frequent pool changes
Server-side clusterOne source of truth, online reshardingSmart clients, redirects, cross-slot limitsRedis at scale, keys that need co-location
Read from replicasMore read capacity, survives a primary lossStale reads, more memoryHot keys, read-heavy traffic
Near cache (L1)No network hop for the hottest keysInvalidation to every process, more stalenessRead-mostly keys with tolerant staleness

Pitfalls in production

Leadership lens

Check yourself

  1. Why does a Redis Cluster client update its slot map on MOVED but not on ASK?

    Show answer

    MOVED means the slot now permanently belongs to another node. ASK happens mid-migration: only some keys have moved, the source still owns the slot, so the redirect applies to that one request.

  2. You double the number of cache shards to cut latency, and page p99 gets worse. Why?

    Show answer

    Pages that read many keys now fan out to more shards, so each page waits for the slowest of more round trips. Throughput per shard dropped, but tail latency is driven by fan-out.

  3. Why is asynchronous replication acceptable for a cache when it would not be for a payments database?

    Show answer

    The cache is not the source of truth. A lost write produces a miss or a stale value that TTLs and invalidation repair, while the database still holds the correct data.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →