Engineering Atlas › System Design Fundamentals

Spreading traffic with load balancers

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

One address, many servers

Without a load balancer, every client must know every server, notice when one dies, and pick among them. A load balancer moves that knowledge into one place: clients connect to a virtual IP (VIP) or hostname, and behind it sits a pool (HAProxy calls it a backend, Envoy a cluster, AWS a target group) of servers that can each handle any request.

This is the overload and partial-failure problem shape at its plainest. The same pattern routes payment calls across gateway instances and an agent's model calls across inference endpoints: one front door, a pool, and a rule for who may receive work.

Layer 4 versus Layer 7

A Layer 4 balancer looks at the IP and TCP or UDP headers, picks a backend for each new flow, and forwards every packet of that flow to the same place. It never reads the payload, so it is fast and protocol-agnostic, but it cannot route by URL, retry a failed request, or see that one long-lived connection carries thousands of requests.

A Layer 7 balancer is a reverse proxy. It accepts the client's TCP and TLS connection, reads each HTTP request, then sends it over a separate connection to a backend it chooses. That makes path- and header-based routing, retries, timeouts, compression, auth and per-request metrics possible. The price is CPU for parsing and TLS, and two sets of socket buffers per client.

Health checks and ejection

Active health checks probe each backend on a timer, for example an HTTP GET /healthz every 2 seconds. A backend is marked down after fall consecutive failures and back up after rise consecutive successes; the thresholds stop one dropped packet from flapping a server in and out. Passive checks, which Envoy calls outlier detection, watch real traffic and eject a backend after, say, 5 consecutive 5xx responses. Passive checks react fast under load; active checks catch dead servers that get no traffic. Use both. Managed balancers now do the passive half too: AWS's Application Load Balancer runs anomaly detection on every target group, comparing each target's error rate with its peers', and with the weighted random algorithm it can shift traffic away from the outlier automatically (AWS ALB docs).

How deep a check should go is a real decision. A check that only proves the process is listening misses a full disk; a check that calls the database fails on every server at once when the database blips. Amazon's guidance is to keep fast load-balancer checks to things local to the host and react to dependency failures centrally, and its NLB, ALB and Route 53 fail open, routing to all targets, when none look healthy (AWS Builders' Library).

Draining

Deploys and scale-in remove healthy servers on purpose. Draining stops new connections or requests going to a backend while existing ones finish, up to a deadline. AWS calls it the deregistration delay (300 seconds by default) (AWS ALB docs); HTTP/2 and gRPC servers send GOAWAY so clients finish in-flight streams and open new ones elsewhere (RFC 9113). Without draining, every deploy shows up as a burst of resets.

Where the balancer lives

Balancing can happen in a dedicated appliance or software tier (HAProxy, NGINX, Envoy, a cloud load balancer), in a sidecar next to each client (a service mesh), or inside the client library itself (gRPC's client-side balancing). Fewer hops favour the client side; central control and simpler clients favour a proxy tier.

Down the stack

Layer 1The contract

An L4 balancer promises that every packet of a flow, identified by its 5-tuple (source IP and port, destination IP and port, protocol), reaches the same backend for the flow's life, and that new flows go only to backends currently believed healthy. An L7 balancer promises per-request routing with explicit failure semantics: a request that cannot be served gets a 502 (bad upstream response), 503 (no healthy upstream) or 504 (upstream timeout) instead of hanging. Neither promises instant detection. The worst-case time to stop routing to a dead backend is about interval × fall for active checks, plus any time a cached decision lives on in an open connection.

Layer 2The mechanism

HAProxy configures checks per server, as in server app1 10.0.0.11:8080 check inter 2s rise 2 fall 3 slowstart 30s, where slowstart ramps a recovered server's weight up over 30 seconds; inter 2s, rise 2 and fall 3 are also the defaults (HAProxy docs). Envoy's outlier detection uses consecutive_5xx (default 5), base_ejection_time (default 30 s, multiplied by the number of past ejections and capped by max_ejection_time, 300 s by default) and max_ejection_percent (default 10%) so a bad deploy cannot eject the whole cluster (Envoy outlier detection API). Envoy also has a panic threshold: if fewer than 50% of hosts are healthy, it balances across all hosts, on the theory that health checking is more likely broken than half the fleet; setting it to 0% makes Envoy return 503 no healthy upstream instead (Envoy panic threshold). On the L4 side, Linux IPVS keeps a connection table keyed by 5-tuple; Google's Maglev and Facebook's Katran hash the 5-tuple into a lookup table so any balancer instance picks the same backend without shared state. Maglev has served Google's traffic since 2008, and one machine saturates a 10 Gbps link with small packets (Maglev, NSDI 2016). Katran, which replaced an IPVS design at Meta, keeps an LRU cache of recent flows on top of the hash, and routers spread VIP traffic across Katran hosts with ECMP (Meta). Backends learn the real client address from X-Forwarded-For or the Forwarded header (RFC 7239) at L7, or from the PROXY protocol header at L4.

Layer 3The machine

An L4 balancer built on XDP or DPDK handles millions of packets per second per core, because it rewrites a few header bytes and never copies the payload into user space. With direct server return, responses go from backend to client without passing back through the balancer, which matters because responses are often ten times larger than requests. Katran works only this way: backends hold the VIP on their loopback interface, and the balancer wraps each inbound packet in an IP-in-IP header to reach them. An L7 proxy instead runs an epoll_wait loop over two sockets per client, each with kernel buffers of tens to hundreds of KB, and pays for TLS: a full handshake costs roughly 0.1 ms of CPU with ECDSA P-256 and closer to 1 ms with RSA-2048, so keep-alive and session resumption are what make TLS termination affordable. Each upstream connection from one proxy IP to one backend IP and port uses an ephemeral port; Linux's default range of 32768 to 60999 allows about 28,000 (kernel docs), which is why proxies pool upstream connections instead of opening one per request.

Why it's built this way

  1. One machine cannot serve all traffic, and clients should not track a changing fleet, so a stable address fronts a pool.
  2. Servers fail and deploy constantly, so the balancer owns health and draining instead of every client.
  3. Reading requests costs CPU, so the cheap L4 tier spreads flows and the expensive L7 tier scales out behind it.
  4. A balancer tier that keeps per-flow state is hard to scale and fail over, so modern L4 designs use consistent hashing on the 5-tuple and stay nearly stateless.

Diagram

sequenceDiagram
  participant C as Client
  participant L4 as L4 tier
  participant L7 as L7 proxy
  participant A as Backend A
  participant B as Backend B
  L7->>A: GET /healthz (every 2s)
  A-->>L7: timeout x3, mark DOWN
  C->>L4: TCP SYN to VIP
  L4->>L7: forward flow (hash 5-tuple)
  C->>L7: TLS + GET /orders
  L7->>B: pick healthy backend B
  B-->>L7: 200 OK
  L7-->>C: 200 OK
  L7->>B: drain: no new requests

Worked example

An orders API serves 24,000 requests per second at peak across 12 backends, about 2,000 each. Each backend saturates at 3,000, so the fleet has 50% headroom.

Backend 7 hangs on a full disk. It still accepts TCP connections, so an L4-only setup keeps sending it one twelfth of new flows. With active checks at inter 2s fall 3 and a 1-second timeout, detection takes up to 9 seconds. During that window 2,000 requests per second time out, about 18,000 failed requests. Adding passive checks with consecutive_5xx: 5 and a 2-second request timeout ejects it after the first few timed-out requests, within a couple of seconds, cutting failures to a few thousand. A single retry on a different backend for idempotent GETs hides most of the rest.

Now the remaining 11 backends carry 24,000 / 11, about 2,180 each, which is fine. If three backends failed, the remaining nine would each take 2,670, close to saturation. That is the number that sizes the fleet: how many simultaneous failures it must absorb at peak.

For the balancer itself, perhaps 1,500 new TLS connections per second arrive with keep-alive. At 0.1 ms of CPU each with ECDSA that is 0.15 cores, small next to proxying. Run at least two proxies per zone, sized so one zone can be lost.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
L4 balancerVery fast, any protocol, nearly stateless with hashingNo request routing, retries or per-request metricsFront tier, non-HTTP protocols, raw throughput
L7 proxyRouting by path or header, retries, TLS, observabilityCPU and latency cost, must parse every protocolHTTP and gRPC services, edge and internal
Client-side balancingNo extra hop, no central bottleneckLogic in every client and language; harder to changeInternal gRPC with a uniform client stack
Sidecar meshUniform policy without changing appsExtra proxy per pod, memory and operations costMany services in many languages
DNS onlySimple, globalCoarse, cached, slow to reactChoosing a region, not a server

Pitfalls in production

Leadership lens

Check yourself

  1. A backend accepts TCP connections but every request hangs. Which kinds of health checking catch it, and how quickly?

    Show answer

    An HTTP active check with a timeout catches it after about interval × fall seconds. Passive checks catch it after a few failed or timed-out real requests, which is faster under load. A TCP-only check never catches it.

  2. Why might an L4 balancer leave one gRPC backend at 90% CPU while others idle?

    Show answer

    gRPC multiplexes many requests on a few long-lived connections, and an L4 balancer places connections, not requests. A few busy connections pinned to one backend stay there.

  3. What is the purpose of Envoy's panic threshold?

    Show answer

    When most hosts look unhealthy, the health checking is likely wrong or a shared dependency is down. Spreading traffic across all hosts is better than piling it on the few that still pass.

  4. Why does direct server return help an L4 tier scale?

    Show answer

    Responses are usually much larger than requests, and with DSR they go straight from backend to client, so the balancer only handles the small inbound half of the traffic.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →