Engineering Atlas › Foundations

HTTP and How the Web Works

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

What happens when you open a URL

Take https://shop.example.com/orders/42?view=full. The browser splits it into a scheme (https), a host (shop.example.com), a path (/orders/42) and a query string (view=full). It asks DNS for the host's IP address, opens a TCP connection to port 443, runs a TLS handshake to agree on encryption keys and check the server's certificate, and only then sends the HTTP request. The server replies, and the browser parses the HTML, finds the CSS, JavaScript and images it references, and fetches those too, often dozens of extra requests.

Anatomy of a request

An HTTP/1.1 request is lines of text separated by carriage return and line feed (\r\n):

GET /orders/42?view=full HTTP/1.1
Host: shop.example.com
Accept: application/json
Authorization: Bearer eyJhbGciOi...

The first line is the request line: method, target and version. Then come headers, one Name: value per line, then an empty line that marks the end of the headers. A request with a body (such as a POST) adds a Content-Length header so the server knows exactly how many bytes of body follow. Header names are case insensitive: content-length and Content-Length are the same header.

Anatomy of a response

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 27
Cache-Control: max-age=60

{"id":42,"status":"paid"}

The status line holds a three-digit code and a human-readable reason. The first digit is the class: 2xx success, 3xx redirect or "use your cached copy", 4xx the client did something wrong (400 bad request, 401 not logged in, 403 not allowed, 404 not found, 429 slow down), and 5xx the server failed (500 bug, 502 bad gateway, 503 overloaded or down, 504 an upstream timed out). Clients, proxies and monitoring all key off that first digit, so choosing the right class matters more than the exact code.

Methods and their promises

MethodTypical useSafeIdempotent
GETRead a resourceYesYes
HEADRead only the headersYesYes
POSTCreate something, run an actionNoNo
PUTReplace a resource at a known URLNoYes
PATCHChange part of a resourceNoNot guaranteed
DELETERemove a resourceNoYes

The standard lists GET, HEAD, OPTIONS and TRACE as safe, PUT and DELETE as idempotent, and POST and PATCH as neither (RFC 9110). These are promises the server makes, not things HTTP enforces. If your GET endpoint sends an email, a browser prefetch or a crawler will send emails. Keep the promises and the rest of the web (caches, retries, proxies) can work for you.

Stateless, then cookies

The server does not remember the previous request on its own. To keep you logged in, it sends a Set-Cookie header, and the browser returns that value in a Cookie header on every later request to the same site. APIs more often use an Authorization header with a token. Either way, the state lives in the request or in a store the server looks up, which is why any server behind a load balancer can answer any request.

Caching

A response can say how long it may be reused: Cache-Control: max-age=60 means "fresh for 60 seconds". Within that window, the browser or a CDN serves its copy without asking the origin (RFC 9111). After it, the client can revalidate: it sends If-None-Match with the response's ETag and the server answers 304 Not Modified with no body if nothing changed. This is the same small-fast-store-in-front-of-a-big-slow-one shape as a CPU cache, with the same questions: what to keep, for how long, and how to invalidate.

HTTP/1.1, HTTP/2 and HTTP/3

HTTP/1.1 sends one request at a time per connection, so browsers open about six connections per host. HTTP/2 keeps one connection and interleaves many requests on it as binary frames. HTTP/3 does the same over QUIC, which runs on UDP so one lost packet stalls only its own stream, not every request on the connection (RFC 9114). A server advertises it with an Alt-Svc header, and clients fall back to TCP when UDP is blocked. The meaning of methods, headers and status codes is the same in all three; only the framing changes.

Down the stack

Layer 1The contract

A client sends a request and gets back at most one final response. HTTP promises the format and meaning of messages, not delivery. If the connection drops or the client times out, the client cannot know whether the server never saw the request, processed it and failed to reply, or is still working on it. Safe methods may be repeated freely; idempotent methods may be retried after a timeout, and a proxy must never retry a non-idempotent request on its own (RFC 9110). A POST may only be retried safely if the server deduplicates, usually with an Idempotency-Key header the client generates once per logical operation, as Stripe's API does (Stripe). That header is still only an IETF Internet-Draft, now expired, not a standard (draft-ietf-httpapi-idempotency-key-header), so agree on its exact behaviour in your own API docs.

Layer 2The mechanism

In HTTP/1.1 (RFC 9112), the server reads bytes until it sees \r\n\r\n, parses the request line and headers, then finds the end of the body in one of two ways: Content-Length: N (read exactly N more bytes) or Transfer-Encoding: chunked (read hex-sized chunks until a zero-length chunk). If both headers are present, Transfer-Encoding wins, the message may be an attempted request smuggling, and a server that accepts it must close the connection afterwards (RFC 9112). Getting this right matters: when a proxy and a server disagree about where a body ends, an attacker can smuggle a second request inside the first. Connections are persistent by default, so the next request follows on the same socket. HTTP/2 (RFC 9113, which also recommends allowing at least 100 concurrent streams) replaces text lines with binary frames (HEADERS, DATA, SETTINGS, WINDOW_UPDATE, RST_STREAM, GOAWAY), each tagged with a stream ID, and compresses headers with HPACK (RFC 7541), a table of previously sent headers so a repeated Authorization costs a few bytes instead of hundreds.

Layer 3The machine

On the wire, a request is TCP segments of up to about 1,460 bytes of payload on a typical Ethernet path. A new TCP connection costs one round trip (SYN, SYN-ACK, ACK). TLS 1.3 adds one more for a full handshake (RFC 8446); resuming with 0-RTT early data can skip it but offers no replay protection across connections, so only send safe requests that way. A new connection can send only about 10 segments (roughly 14 KB) before waiting for acknowledgements, because the initial window is 10 segments (RFC 6928), so a 100 KB response on a fresh connection takes several round trips even on a fast link. On the server, a process like nginx or a Go program calls accept to take a connection, then epoll_wait to learn which of thousands of sockets have data, and read to copy bytes from the kernel's socket buffer into user memory. Parsing a request takes microseconds. The round trips around it take tens to hundreds of milliseconds.

Why it's built this way

  1. Text messages were easy to write, read and debug with a terminal, which helped the web spread in the 1990s.
  2. Statelessness lets any server answer any request, so a site scales by adding servers behind a load balancer.
  3. Round trips, not bandwidth, dominate page load time, so later versions focus on reusing connections and sending many requests in parallel.
  4. Binary framing in HTTP/2 made multiplexing possible, but its own spec admits TCP head-of-line blocking is not addressed (RFC 9113): TCP still delivers bytes in order, so one lost packet stalls everything.
  5. QUIC moves streams into the transport over UDP, so HTTP/3 removes that last head-of-line block and merges the transport and TLS handshakes.

Diagram

sequenceDiagram
  participant B as Browser
  participant D as DNS resolver
  participant S as Server
  B->>D: Where is shop.example.com?
  D-->>B: 203.0.113.10
  B->>S: TCP SYN
  S-->>B: SYN-ACK
  B->>S: ACK + TLS ClientHello
  S-->>B: TLS ServerHello, certificate, Finished
  B->>S: GET /orders/42 HTTP/1.1
  S-->>B: 200 OK + JSON body
  B->>S: GET /app.js (same connection)
  S-->>B: 304 Not Modified

Worked example

A shop's servers are in Virginia and a customer opens the site from Jakarta, about 220 ms round trip away. The browser's DNS answer comes from a nearby resolver in 20 ms. Then TCP costs one round trip (220 ms), TLS 1.3 another (220 ms) and the request itself a third (220 ms) plus 30 ms of server work. The first byte of HTML arrives after about 690 ms, before any CSS, scripts or images. If the page then fetches 12 API calls one after another, each on the warm connection, that is another 12 × 250 ms = 3 seconds.

Three fixes, in order of payoff. First, put a CDN edge in Singapore, 15 ms away: TCP and TLS now finish with the edge in 30 ms, and cached assets come back in one more 15 ms round trip. Only uncached API calls still travel to Virginia, over connections the CDN keeps open. Second, issue the 12 API calls in parallel over HTTP/2, so they cost about one round trip instead of twelve. Third, add Cache-Control: max-age=31536000 to versioned static files such as app.3f9c.js, so repeat visitors fetch nothing at all. The page goes from about 4 seconds to under 1 without making the server any faster.

In the wild

Trade-offs

ChoiceStrengthsWeaknessesChoose when
HTTP/1.1Simple, universal, readable on the wireOne request at a time per connectionInternal tools, simple clients, debugging
HTTP/2Many requests on one connection, header compressionA lost packet stalls all streamsDefault for browsers and gRPC services
HTTP/3 (QUIC)No transport head-of-line blocking, faster setupUDP sometimes blocked, more CPU per byteMobile and lossy networks, global audiences
Long max-age cachingZero requests for repeat visitsStale content until expiryVersioned static assets
Short max-age plus ETagFresh data, cheap 304 checksStill one round trip per checkPages and API responses that change
Cookies for sessionsSent automatically by browsersNeed CSRF protection, size limitsBrowser apps on one site
Bearer tokensWork for any client, easy across servicesClient must store and attach themPublic APIs, mobile apps, service to service

Pitfalls in production

Leadership lens

Check yourself

  1. Why is it safe for a client library to retry a PUT after a timeout but not a plain POST?

    Show answer

    PUT is idempotent, so applying it twice leaves the same state as once. A POST may create a second order or charge, unless the server deduplicates with an idempotency key.

  2. A response has Cache-Control: max-age=60 and an ETag. What happens on a request 30 seconds later, and 90 seconds later?

    Show answer

    At 30 seconds the cached copy is fresh and is used without contacting the server. At 90 seconds the client revalidates with If-None-Match, and gets either 304 Not Modified with no body or a new 200 response.

  3. How does an HTTP/1.1 server know where a request body ends?

    Show answer

    From Content-Length, which gives the exact byte count, or from Transfer-Encoding: chunked, where the body ends with a zero-length chunk.

  4. Why can moving servers closer to users speed up a page more than making the server code faster?

    Show answer

    A fresh HTTPS request needs several round trips (TCP, TLS, request), each costing the full network distance, while server processing is often only tens of milliseconds.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →