Engineering Atlas › System Design Fundamentals

Serving content from the edge with a CDN

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Edges, routing and the round-trip tax

A user in Jakarta fetching from an origin in Virginia pays roughly 230 ms per round trip. A new HTTPS connection needs one round trip for TCP and one for TLS 1.3, and a third for the response: about 700 ms before the first byte. Put a PoP 10 ms away and those three round trips cost 30 ms. On a miss the edge reaches the origin over an already open, warmed-up connection, so the user pays one long round trip instead of three. HTTP/3 shortens the setup further, because QUIC folds the transport and TLS 1.3 handshakes into one exchange (RFC 9000); that saves one round trip, but distance still multiplies every round trip that remains.

Users reach a PoP in one of two ways. DNS-based routing answers the CDN hostname with the address of a PoP chosen from the resolver's location (or the client subnet if the resolver sends EDNS Client Subnet). Anycast announces one IP prefix from every PoP and lets BGP deliver each packet to the nearest one; the anycast lesson covers why that converges in seconds and where "nearest" goes wrong. Many CDNs use anycast for the address and DNS steering on top for load.

What a CDN caches

Static assets (images, scripts, fonts, video segments) are the easy case. HTML and API responses can be cached if you say who may share them. Even uncacheable pages benefit, because TLS terminates near the user and the edge forwards over warm connections.

Most CDNs are pull caches: the edge fetches from the origin on the first miss. Push places content before anyone asks; Netflix's Open Connect appliances inside ISP networks are filled off-peak with what is predicted to be watched. Netflix describes this as proactive, directed caching: it predicts what members will watch and downloads most new content to its appliances during configurable off-peak fill windows (Netflix Open Connect overview). Push fits huge objects with predictable demand.

Cache keys

A cache key is the string the proxy hashes to find a stored response. The default is roughly scheme, host and the full path with query string. That default is almost always wrong for real traffic: ?utm_source=newsletter and ?w=200&h=100 versus ?h=100&w=200 each create separate entries for the same bytes. Normalising the key (lowercase the host, drop tracking parameters, sort the rest) can move a hit ratio from 70% to 95% with no other change.

The Vary response header adds request headers to the key. Vary: Accept-Encoding is fine once the CDN normalises it to br, gzip or identity; Vary: User-Agent or Vary: Cookie effectively disables caching. The same problem shape, deciding when two requests are "the same", appears in payment idempotency keys and LLM prompt caching: too little in the key returns wrong answers, too much never reuses anything.

Freshness, staleness and validation

HTTP caching (RFC 9111) says a stored response is fresh while its age is below its freshness lifetime. Shared caches take the lifetime from s-maxage, falling back to max-age, then Expires. When a response goes stale, the cache can revalidate with a conditional request (If-None-Match with the stored ETag, or If-Modified-Since); a 304 Not Modified costs a round trip but no body. If the origin sends no explicit lifetime at all, a cache may invent one: RFC 9111 suggests about 10% of the time since Last-Modified as a typical heuristic, so a file last changed 30 days ago may be cached for about 3 days without anyone asking for it. Send explicit Cache-Control on every response.

RFC 5861 adds two extensions that matter most for CDNs. stale-while-revalidate=N lets the cache serve a stale copy for up to N seconds past expiry while it refreshes in the background, so no user waits on the origin. stale-if-error=N lets it serve a stale copy for up to N seconds when the origin returns errors or is unreachable. Together they turn origin latency spikes and short outages into non-events. RFC 9213 adds CDN-Cache-Control, a header that only CDNs read, so you can tell the CDN "cache for a day" while telling browsers "cache for a minute".

Purging

Sometimes content must go now: a price error, a takedown. Purge by URL works for single objects. For "every page that shows product 42", tag responses with Surrogate-Key (Fastly) or Cache-Tag (Cloudflare) and purge by tag. Purges are now fast on large CDNs; Cloudflare reported a global median under 150 ms for purge by tag, hostname and prefix in 2024 (Cloudflare). Versioned asset URLs (app.3f9c2a.js) never need purging: new content gets a new name.

Down the stack

Layer 1The contract

A CDN promises that a response served from cache was stored from an origin response that allowed sharing (public, or s-maxage, and not private or no-store), and that its Age header plus the stored lifetime say how stale it may be. Within the freshness lifetime it serves without contacting the origin. Past it, the response is served only if a stale-* directive permits it, otherwise it is revalidated or refetched. Requests whose cache keys match are interchangeable; that is the whole correctness boundary. A purge is eventually consistent: the API returns before every PoP has dropped the object, typically within a fraction of a second to a few seconds depending on the provider, so the contract is "soon everywhere", never "now".

Layer 2The mechanism

Inside a PoP, a request first hits a load balancer, then a cache server chosen by consistent hashing on the cache key, so each object lives on one or two machines per PoP rather than all of them. NGINX's default key is $scheme$proxy_host$request_uri; it stores the MD5 of the key in a shared-memory keys_zone index and the body in a file named by that hash. Varnish builds the key in vcl_hash from the URL and the Host header. On a miss, request collapsing makes concurrent requests for the same key wait for the single in-flight fetch: NGINX's proxy_cache_lock (off by default; once on, waiters give up and go to the origin themselves after a default 5 seconds) (NGINX docs), or Varnish's waiting list. That timeout matters: a collapse with no timeout turns one stuck fetch into thousands of stuck users.

Stale serving maps onto NGINX's proxy_cache_use_stale updating error timeout plus proxy_cache_background_update on. Tiering is another hop: an edge miss goes to an upper-tier PoP (Cloudflare Tiered Cache, CloudFront Origin Shield, Fastly shielding) running the same logic. CloudFront already has regional edge caches as a middle tier; Origin Shield adds a third tier in one region and consolidates concurrent misses, so the origin can see as few as one request per object (CloudFront docs). Purge by tag is a reverse index from tag to keys, fanned out over the control plane.

Layer 3The machine

A cache hit should never copy the body through user space. Serving from disk uses sendfile so the kernel moves pages from the page cache to the socket; with kernel TLS (kTLS) the encryption also happens in the kernel or on the NIC, which is how video CDNs push hundreds of Gbit/s from one box; Netflix's FreeBSD work targeted 400 Gbit/s of TLS video per server and listed memory bandwidth as a limit for software kTLS (Gallatin, EuroBSDCon 2021). Storage is tiered by heat: the hottest objects sit in RAM, the rest on NVMe drives that deliver hundreds of thousands of random reads per second.

On the wire, TCP's initial congestion window is 10 segments (RFC 6928), about 14 KB, so the first round trip can carry a small image in full but a 200 KB script needs several more round trips while the window grows. Short edge RTTs make those round trips cheap, and warm origin connections skip slow start.

Why it's built this way

  1. Light in fibre covers about 200 km per millisecond, and handshakes multiply RTT, so the only way to cut latency is to move the server closer.
  2. Storage at the edge is finite and spread across hundreds of sites, so only reusable responses are kept, and the cache key defines reuse.
  3. Each PoP misses independently, so origin load grows with the number of PoPs; a shield tier collapses those misses back to a few.
  4. Purging hundreds of sites synchronously is slow and fragile, so the design prefers bounded staleness (TTLs plus stale windows) and immutable URLs, with purge as the exception.

Diagram

sequenceDiagram
  participant U1 as User A
  participant U2 as User B
  participant E as Edge PoP
  participant S as Shield tier
  participant O as Origin
  U1->>E: GET /img/42.jpg
  E->>E: key lookup: miss
  U2->>E: GET /img/42.jpg
  E->>E: collapse: B waits on A's fetch
  E->>S: GET (keep-alive)
  S->>S: key lookup: miss
  S->>O: GET
  O-->>S: 200, s-maxage=86400
  S-->>E: 200, Age: 0
  E-->>U1: 200 (miss)
  E-->>U2: 200 (collapsed)

Worked example

A product catalogue serves 30,000 image requests per second at peak, averaging 200 KB, so users download 6 GB/s, about 48 Gbit/s. There are 2 million images and 150 PoPs.

With the default cache key, marketing parameters and unordered resize parameters split each image into many entries. The measured edge hit ratio is 72%, so the origin sees 8,400 requests per second and 1.68 GB/s. At an illustrative cloud egress price of $0.05 per GB, that is about $7,300 per day in origin egress, and the origin fleet is sized for 8,400 requests per second of image resizing.

Normalising the key (drop utm_*, sort w and h, normalise Accept-Encoding) raises the edge hit ratio to 92%: 2,400 requests per second reach the next tier. Adding a shield in three regions, which itself hits 80% of the time because it pools misses from 50 PoPs each, cuts origin traffic to about 480 requests per second and 96 MB/s, about $415 per day. The origin shrinks by a factor of 17.

Add stale-if-error=86400 and a 20-minute origin outage only fails first-ever requests for cold images.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Pull CDN, single tierZero setup, origin unchangedEvery PoP misses on its own, long-tail origin loadSmall catalogues, few PoPs in use
Pull with shield tierOrigin sees a handful of misses per objectExtra hop on misses, shield region is a dependencyLarge catalogues, expensive origins
Push / pre-positioningPredictable fills off-peak, no first-user missMust predict demand, storage planningVideo, game patches, software releases
Long TTL plus versioned URLsNever purge, maximal hit ratioNeeds build tooling, HTML must stay short-livedStatic assets
Short TTL plus stale-while-revalidateFresh within seconds, no user waitsMore origin revalidationsHTML, API responses shared by many users
Tag-based purgePrecise invalidation across many URLsIndex cost, vendor-specificCatalogues, CMS pages

Pitfalls in production

Leadership lens

Check yourself

  1. Your edge hit ratio is 95% but the origin still sees heavy load from cache misses after you add 100 new PoPs. Why, and what fixes it?

    Show answer

    Each PoP misses independently, so origin load scales with the number of PoPs for long-tail objects. A shield (upper) tier pools those misses so each object is fetched from the origin roughly once per shield region.

  2. Why is Vary: Accept-Encoding safe on most CDNs but Vary: User-Agent is not?

    Show answer

    CDNs normalise Accept-Encoding to a few values, so it adds at most three variants. User-Agent strings have thousands of distinct values, so every variant is nearly unique and the hit ratio collapses.

  3. A response has Cache-Control: max-age=60, s-maxage=600, stale-while-revalidate=30. At age 620 seconds, what does a shared cache do?

    Show answer

    Its lifetime is 600 (from s-maxage), so it is 20 seconds stale and inside the 30-second window. It serves the stale copy and refreshes in the background.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →