Design a content delivery network

In one minute

• A CDN is a routing problem first and a caching problem second. Getting each user to a nearby, healthy point of presence (PoP) decides latency. Anycast steers coarsely; the PoP's load balancer steers finely.
• Cache each object once per PoP, not once per server. Hash the cache key to one owner server so the PoP acts as one large cache, and copy only the hottest objects more widely.
• Hit ratio comes from the hierarchy. Edge, regional shield, then origin, with request collapsing at each level, so a miss in 250 PoPs costs a handful of origin fetches.
• Purge is a write path with a deadline. Use tag generation counters, so purging a million objects worldwide is one small message rather than a scan of every disk.
• Every config push is a global deploy. Rules and WAF signatures reach every PoP within seconds, so stage them the way you stage code.

Requirements

Functional

• Customers create a distribution: hostnames, origins and cache rules (TTLs, which query parameters and headers go into the key).
• Serve GET and HEAD over HTTPS from the edge, with conditional requests (If-None-Match) and range requests (Range: bytes=...).
• Purge by exact URL, by prefix and by surrogate tag (for example every object tagged product-4411), plus a "soft purge" that marks objects stale instead of deleting them.
• Serve stale content when the origin fails, if the customer allows it.
• Per-customer logs and metrics (requests, bytes, hit ratio, status codes).

Non-functional

• Time to first byte (TTFB) under 50 ms at p95 for cache hits for users on a decent network.
• 99.99% availability for cached content, which means surviving the loss of a server, a PoP or an upstream transit link without customer action.
• Purges take effect in every PoP within 5 seconds at p99. Config changes take effect within 30 seconds.
• Origin offload of 95% or more by bytes for cacheable content.
• One customer's flash crowd or attack must not degrade other customers.

Out of scope

• Edge compute, video transcoding, DNS hosting as a product, and billing.

Back-of-envelope

These are round planning numbers for a mid-sized commercial CDN, not any vendor's published figures.

• Requests: 100 billion per day. 100 × 10^9 / 86,400 s ≈ 1.16M requests per second on average. Daily peaks run about 3x, so plan for 3.5M per second.
• Bandwidth: the average response is about 150 KB (small API and image responses, plus large video segments that dominate bytes). 1.16M × 150 KB ≈ 174 GB/s ≈ 1.4 Tbit/s on average and about 4.2 Tbit/s at peak.
• PoPs: 250 PoPs, unevenly loaded. The largest carries about 5% of traffic, 210 Gbit/s at peak. If a server sustains about 40 Gbit/s of TLS traffic, that PoP needs 6 servers, and about 12 with headroom for failures and attacks.
• Cache storage: 2 billion distinct objects a day at 150 KB is a 300 TB daily working set. A large PoP with 12 servers of 8 × 7.68 TB NVMe holds about 740 TB, and each server keeps its hottest few hundred GB in RAM.
• Origin traffic: at 95% byte offload, origins receive about 70 Gbit/s on average. Without a shield tier, every PoP would miss separately and that would be several times higher.
• Logs: 100 billion requests × about 300 bytes per log record ≈ 30 TB per day. Keeping 30 days of raw logs is 900 TB. Rollups kept for 5 years at about 1% of raw size are about 550 TB. Logs are the only large durable storage; the cache itself is disposable.
• Purges: 10,000 purge requests per minute at peak, each fanned out to 250 PoPs, gives about 42,000 PoP-level messages per second. That is small, but the 5-second deadline makes it latency-critical.

API

# Control plane (customer-facing, authenticated)
PUT /v1/distributions/{dist_id}
  { "hostnames": ["static.shop.example"],
    "origins": [{"host": "origin.shop.example", "weight": 100}],
    "cache_key": {"query_params": ["v", "size"], "headers": ["accept-encoding"]},
    "default_ttl_s": 3600, "serve_stale_if_error_s": 86400 }
  -> 200 { "version": 418, "status": "propagating" }

POST /v1/distributions/{dist_id}/purge
  { "type": "tag", "values": ["product-4411"], "mode": "soft" }
  -> 202 { "purge_id": "pg_9a1c", "status": "propagating" }

GET /v1/purges/{purge_id}
  -> 200 { "status": "complete", "pops_acked": 250, "pops_total": 250 }

# Data plane (end users)
GET https://static.shop.example/img/hero.jpg?v=12
  Range: bytes=0-1048575
  -> 206  Content-Range: bytes 0-1048575/8388608
          Cache-Status: shield-fra; fwd=uri-miss; stored, edge-fra3; hit
          Age: 532

The data plane is ordinary HTTP under RFC 9110 and RFC 9111. Cache-Status follows RFC 9211, which lists caches from the one nearest the origin to the one nearest the user, so the shield's entry (a miss it stored earlier) comes before the edge's hit (RFC 9211). The control plane is asynchronous on purpose: "applied everywhere" is a distributed acknowledgement the caller polls for, not something one request can promise.

Data model

• distributions (Postgres, the control-plane source of truth): dist_id, customer_id, version, config_json, created_at. Changes are rare and need transactions and history, so each one writes a new immutable version.
• hostname_map (compiled and shipped to every server): hostname -> (dist_id, version), a read-only in-memory table, so a lookup never leaves the box.
• purge_log (a replicated, ordered log such as Kafka or a Raft log): seq, dist_id, type (url, prefix, tag), value, mode, issued_at. PoPs consume it in order and acknowledge the highest seq they have applied, which is what GET /v1/purges/{id} reports.
• Per-server cache index (in RAM on each cache server): hash(cache_key) -> {disk_offset, size, stored_at, ttl, etag, tags[], tag_generations[]}, at about 100 to 150 bytes per entry. With 30 million objects per server that is 3 to 5 GB of RAM, which is why the index stays in memory while the bodies live on SSD.
• tag_generations (in RAM on each server, fed by the purge log): (dist_id, tag) -> generation. Deep dive 3 explains why this beats enumerating objects.
• Request logs (a columnar store such as ClickHouse): partitioned by (customer_id, hour), because every query is "my traffic, for this time range".

High-level design

flowchart LR
  U["User"] --> AC["Anycast IP"]
  AC --> L4["PoP L4 balancer"]
  L4 --> FE["Edge server (TLS)"]
  FE --> OW["Owner server for key"]
  OW --> SH["Regional shield"]
  SH --> O["Customer origin"]
  API["Control API"] --> PG["Config DB"]
  API --> PL["Purge log"]
  PG --> CD["Config distributor"]
  CD --> FE
  PL --> FE
  FE --> LG["Log shipper"]
  LG --> CH["Analytics store"]

Read path. DNS returns an anycast address and BGP delivers packets to a nearby PoP, where a layer 4 balancer spreads connections across servers. The receiving server terminates TLS, maps the hostname to a distribution, computes the cache key and hashes it onto the PoP's ring to find the owner server, forwarding to it if needed (about 100 µs inside the PoP). On a miss, the owner collapses concurrent requests for the same key into one fetch from the regional shield. The shield runs the same logic and fetches from the origin only on its own miss. Each layer stores the response as it streams back, if the headers allow.

Write path. A config change becomes a new version in Postgres. The distributor compiles a snapshot and pushes it in stages (canary PoPs, then 10%, then everyone), and servers swap snapshots atomically between requests. A purge is appended to the purge log; servers apply entries in seq order and report their applied seq.

Deep dives

1. Getting users to the right PoP, and to the right server inside it

DNS steering answers each resolver with a PoP chosen by geography, measured latency and load. It gives fine control, but it routes by the resolver's location rather than the user's, and every change waits for DNS TTLs. Anycast announces the same prefix from every PoP and lets BGP pick, so a PoP that withdraws its routes stops receiving new connections within seconds, and a volumetric attack spreads across all PoPs instead of landing on one. Its weakness is that "nearest" means best BGP path, not lowest latency, and a route change mid-connection can land packets on a PoP with no state for them. I'd use anycast by default and DNS steering for coarse shifts such as draining a region. The Anycast Routing lesson covers the BGP mechanics.

Inside the PoP, the L4 balancer uses ECMP from the routers plus a consistent-hash table (in the style of Maglev, or Cloudflare's Unimog) so a server added or removed moves few connections. The more interesting choice is the second hash, from cache key to owner server. If every server cached everything it served, a 12-server PoP would hold only one server's worth of distinct content. Hashing keys to owners makes the PoP one large cache. The cost is an internal hop and hot keys: a viral object would pin one server. So each server also keeps a small local cache of keys it forwards above, say, 1,000 requests per second.

2. The cache hierarchy, collapsing and admission

Tiers. Each PoP is assigned to one shield PoP in its region, so a miss goes edge, then shield, then origin. With 250 edge PoPs and 12 shields, a cold object costs at most 12 origin fetches instead of 250, and the shields' large disks hold the long tail that small PoPs evict. A customer with one origin can pick a single shield next to it for maximum offload. Cloudflare's Tiered Cache offers both shapes: a latency-chosen single upper tier per origin, or regional upper-tier pools chosen by hashing the cache key (Cloudflare docs).

Collapsing. When a popular object expires, hundreds of requests can miss in the same millisecond. The owner keeps an in-flight table keyed by cache key: the first miss fetches, and later ones wait on that fetch and stream the same bytes as they arrive. If the response turns out to be uncacheable (private, or a Set-Cookie), it must not be shared: the waiters fetch on their own and the key is marked "do not collapse" for a while. The Cache Stampede lesson covers the general pattern.

Admission and eviction. Plain LRU on SSD lets one-hit wonders, such as a crawler walking a catalogue, push out objects that are requested every hour. Two cheap defences work well: admit an object to SSD only on its second request within a window (tracked with a small counting filter), and split the cache into probation and protected segments so a newcomer can't evict anything that has proven itself. Akamai found that 74% of about 400 million cached objects on a typical cluster were requested only once in two days, and cache-on-second-hit with two rotating Bloom filters cut disk writes by 44% while raising byte hit rate (Maggs and Sitaraman, CCR 2015). Facebook's photo caching study found that a segmented LRU (S4LRU) raised the edge object-hit ratio by 8.5% over FIFO, cutting downstream requests by 20.8% (Huang et al., SOSP 2013).

Large objects. A 4 GB software download or a long video file is not cached as one unit. The edge fetches it in fixed slices (for example 1 MB, using range requests to the origin) and stores each slice as its own cache entry, the approach nginx ships as its slice module (nginx docs). A seek costs one slice fetch, and eviction can drop a file's cold end while keeping its popular start. The danger is stitching slices from two versions, so every slice records the ETag and a mismatch discards them. The practice builds exactly this.

3. Purging worldwide in five seconds

URL purges are easy: hash the key and delete the index entry. Tag and prefix purges are the hard part. "Purge everything tagged product-4411" might match 3 objects or 3 million, spread across 250 PoPs and many servers, and scanning indexes to find them misses the deadline.

Tag generations. Each server keeps a counter per (distribution, tag). When an object is stored, its index entry records the current generation of each of its tags. A tag purge simply increments the counter. On lookup, the server compares the entry's recorded generations with the current ones, and a mismatch means the entry is treated as purged (hard mode) or stale and must be revalidated (soft mode). A purge therefore costs one tiny message and one hash-table write per server, no matter how many objects it covers. Dead entries are reclaimed by normal eviction. Prefix purges use a generation per configured prefix, and "purge all" is a generation on the distribution.

Delivery. Each PoP streams the replicated purge log from a nearby relay and resumes from its last applied seq after a disconnect, so no purge is lost. Meeting 5 seconds is mostly about not queuing: keep purges off the log and config channels, and alert when any PoP's applied seq lags.

Soft purge as the default. A hard purge of a popular tag empties caches worldwide at once and stampedes the origin. Marking objects stale lets the edge serve the old copy while one collapsed revalidation per PoP runs. Fastly works this way: a soft purge marks content stale and a hard purge makes it unusable at once, and a purge-all can never be soft, so Fastly suggests tagging every object with a shared surrogate key to get a soft purge-all (Fastly docs).

Down the stack

Failure modes

• 10x load from a flash crowd. One object jumps to 500,000 requests per second. Collapsing makes it one fetch per PoP, hot-key replication spreads it across servers, and fair queuing on origin fetches protects other customers.
• PoP or region outage. The PoP withdraws its anycast routes when health checks fail, and neighbours absorb the traffic at a lower hit ratio. Plan capacity for absorbing a neighbour.
• Origin outage. Serve stale content within the customer's stale-if-error window, and back off origin retries with jitter so recovery is not a stampede. Soft purge keeps old copies available for exactly this case.
• Bad config push. In July 2019, Cloudflare deployed a WAF rule containing a regular expression that backtracked badly, and CPU exhaustion took down its proxies globally for about 27 minutes (Cloudflare). Stage pushes through canary PoPs, gate on CPU and error rates, and keep a rollback path independent of the failing component.
• Cache poisoning. An unkeyed header such as X-Forwarded-Host changes the origin's response, and the edge caches the malicious variant for everyone (PortSwigger research). Strip headers that are not in the key before forwarding, and don't cache responses that Vary on headers the key ignores.
• Purge storm. A deploy script purges /* every minute and the refill load swamps shield and origin. Rate-limit hard purges and steer customers to versioned URLs.

In the wild

• Fastly, 2021: a valid customer config trips a latent bug everywhere. A software change shipped on 12 May lay dormant until a customer pushed a valid configuration on 8 June that met its exact trigger conditions. About 85% of Fastly's network then returned errors. Fastly detected it within a minute and 95% of the network recovered within 49 minutes once the configuration was disabled (Fastly).
• Valve Steam, 2015: a caching rule serves logged-in pages to strangers. During a DoS attack that pushed store traffic 2,000% above normal sale levels, a second caching configuration was deployed that wrongly cached responses for authenticated users. For about 90 minutes, store pages for about 34,000 users may have been shown to others, including partial card digits and email addresses. Valve closed the store, fixed the configuration and confirmed edge caches were purged before reopening (Valve).
• Akamai, 2015: cache-on-second-hit in production (write-up). With 74% of objects on a typical cluster requested only once in two days, Akamai stopped writing first-time objects to disk. Byte hit rate rose from about 74% to 83% and disk writes fell from 10,209 to 5,738 per second, using two Bloom filters that rotate so old history is forgotten (Maggs and Sitaraman, CCR 2015).

Trade-offs

Leadership lens

• Build first. One PoP with TLS termination, key hashing, collapsing and a single shield, plus the purge log with acknowledgements. Routing sophistication and fancy admission policies can come later; correct purging and safe config pushes cannot.
• Rollout. Start with a few PoPs in the busiest regions, run them dark against mirrored traffic, then grow where latency measurements show the largest gains.
• Ownership. An edge team owns the cache server. A network team owns PoPs, BGP and capacity. A control-plane team owns the API, purge log and staged config rollout, and security ships WAF rules through that same pipeline.
• Cost drivers. Transit and peering bandwidth dominate, then servers. Hit ratio is a cost lever too. Settlement-free peering, or embedding caches inside ISPs as Netflix's Open Connect does, cuts bandwidth cost.
• Design review questions. What is the blast radius of a bad config push, and how fast can it be rolled back? What does a 0% hit ratio in one region do to customer origins?

Check yourself

1. Why do tag purges use per-tag generation counters instead of scanning the cache index for matching objects?

   Show answer

   A tag can match 3 objects or 3 million across 250 PoPs, and scanning indexes misses the 5-second deadline. Incrementing a (distribution, tag) counter costs one small message and one hash-table write per server; entries whose recorded generation no longer matches are treated as purged or stale, and dead entries are reclaimed by normal eviction.

2. A customer hard-purges a popular tag. What goes wrong, and why is soft purge the default?

   Show answer

   Caches worldwide empty at once and the synchronized refill stampedes the origin. Soft purge marks objects stale, so the edge keeps serving the old copy while one collapsed revalidation per PoP runs, at the cost of briefly serving stale content.

3. Why hash cache keys to an owner server inside a PoP instead of letting each server cache what it serves, and what is the cost?

   Show answer

   If every server cached everything it served, a 12-server PoP would hold only one server's worth of distinct content; key hashing makes the PoP one large cache. The cost is an extra internal hop and hot keys pinning one server, handled by a small local cache for keys forwarded above about 1,000 requests per second.

4. With 250 edge PoPs and 12 shields, what does a cold object cost the origin, and why does the per-server cache index live in RAM?

   Show answer

   At most 12 origin fetches instead of 250, because each PoP misses to its regional shield. The index is about 100 to 150 bytes per entry, so 30 million objects per server is only 3 to 5 GB of RAM, while bodies stay on SSD.

Sources and further study

Primary

• RFC 9211: The Cache-Status HTTP Response Header Field: the standard way for each cache tier to report hit, forward reason, collapsing and storage.
• RFC 5861: HTTP Cache-Control Extensions for Stale Content: the stale-while-revalidate and stale-if-error directives behind soft purge and origin-outage serving.
• Practical Web Cache Poisoning (PortSwigger, 2018): unkeyed inputs such as X-Forwarded-Host, with real exploits and the defences.

Systems docs

• Fastly: Soft purges: how a production CDN separates marking stale from deleting.
• Fastly: Purging with surrogate keys: tag-based purging, the feature deep dive 3 builds.

Books

• Understanding Distributed Systems, Roberto Vitillo, chapters "HTTP caching" and "Content delivery networks": freshness and validation, then overlay routing and caching tiers in a few pages.
• Computer Networking: A Top-Down Approach (8th edition), Kurose and Ross, chapter 2 "Application Layer", section on video streaming and CDNs: cluster placement (enter deep versus bring home) and how DNS redirects users.

Other curricula

• donnemartin/system-design-primer: frames a CDN as a component and splits it into push and pull CDNs by who uploads content.
• Azure Architecture Center: CDN guidance: the customer's view, covering versioning, cache control, CORS and a fallback when the CDN itself fails.

Watch

• What is a CDN & why Developers should Care about using one (GOTO 2016): Fastly's founder Artur Bergman on how a CDN works from the inside and how to use one well.
• System Design: Content Delivery Networks (Gaurav Sen): an interview-oriented walkthrough of edge caching and routing.
• What Is A CDN? How Does It Work? (ByteByteGo): a short refresher on PoPs, routing and caching.

Free resources

• RFC 9111: HTTP Caching: the rules every shared cache must follow, including freshness, validation and Vary.
• Unimog: Cloudflare's edge load balancer: how traffic is spread across servers inside a PoP without breaking connections.
• Tiered Cache (Cloudflare docs): how upper-tier PoPs reduce origin traffic in a real CDN.
• ngx_http_slice_module (nginx docs): the range-slicing technique from the practice, as shipped in nginx.
• An Analysis of Facebook Photo Caching (SOSP 2013): measured hit ratios across browser, edge and origin caches, and what better eviction buys.
• Netflix Open Connect: a CDN that places its appliances inside ISP networks.