Engineering Atlas › System Design Interviews

Design a scalable load balancer

21 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope

Back-of-envelope

Design for one large region. The numbers are planning assumptions, not any provider's figures.

API

The data plane has no API: it is the VIP. The control plane looks like a cloud load-balancer API.

POST /v1/load-balancers
  { "name": "checkout", "listeners": [
      { "port": 443, "protocol": "HTTPS", "cert_id": "c-91", "default_target_group": "tg-7" },
      { "port": 5432, "protocol": "TCP", "default_target_group": "tg-8" } ] }
  -> 201 { "lb_id": "lb-12", "vip": "203.0.113.10", "dns_name": "lb-12.region.lb.example" }

POST /v1/target-groups
  { "protocol": "HTTP", "port": 8080, "algorithm": "p2c_least_request",
    "health_check": { "path": "/healthz", "interval_s": 2, "timeout_ms": 1000,
                      "healthy_threshold": 2, "unhealthy_threshold": 3 },
    "slow_start_s": 30, "deregistration_delay_s": 60 }
  -> 201 { "tg_id": "tg-7" }

PUT    /v1/target-groups/tg-7/targets   { "targets": [{ "ip": "10.1.4.21", "port": 8080, "weight": 100 }] }
DELETE /v1/target-groups/tg-7/targets/10.1.4.21:8080   -> 202 { "state": "draining", "until": "..." }
GET    /v1/target-groups/tg-7/health   -> 200 [{ "target": "10.1.4.21:8080", "state": "healthy", "reason": null }]

Deregistration returns 202 because removal is a process: the target stops getting new work immediately and is dropped once its connections finish or the drain deadline passes.

Data model

High-level design

flowchart LR
  CL["Clients"] --> RT["Edge routers (ECMP)"]
  RT --> L4A["L4 node A (XDP, Maglev)"]
  RT --> L4B["L4 node B (XDP, Maglev)"]
  L4A -->|"IP-in-IP"| P1["L7 proxy 1"]
  L4B -->|"IP-in-IP"| P2["L7 proxy 2"]
  P1 --> BE["Tenant backends"]
  P2 --> BE
  P1 -.->|"direct return"| CL
  P2 -.->|"direct return"| CL
  API["Control API"] --> DB["Config store"]
  HC["Health checkers"] --> AGG["Health aggregator"]
  DB --> CP["Snapshot builder"]
  AGG --> CP
  CP --> L4A
  CP --> L4B
  CP --> P1
  CP --> P2

Packet path. Every L4 node announces each VIP to the edge routers over BGP, so the routers treat the nodes as equal-cost next hops and hash each packet's 5-tuple to pick one. The L4 node looks the flow up in its connection table; on a miss it hashes the 5-tuple into the VIP's Maglev table to choose an L7 proxy and records the choice. It wraps the packet in an outer IP header addressed to that proxy and sends it back out. The proxy, which has the VIP configured on a loopback interface, strips the outer header, terminates TCP and TLS, picks a tenant backend per request with its own algorithm, and sends the response straight to the client with the VIP as the source address.

Config path. A deregistration writes admin_state = draining and bumps the config version. The snapshot builder rebuilds the Maglev tables of affected VIPs only and streams the delta to every node, which swaps a pointer. Health flips take the same path from the aggregator.

Deep dives

1. Keeping flows on the same backend across a changing fleet

Three things change underneath a live TCP connection: the set of L4 nodes, the set of backends, and the router's ECMP hash. A connection survives only if its packets keep reaching the same backend.

Option A: shared connection state. Replicate the flow table across L4 nodes. Exact affinity, but every new flow becomes a cross-node write at millions per second.

Option B: a consistent hash alone. If every node computes the backend from the 5-tuple with the same function, an ECMP reshuffle doesn't matter. But a plain hash mod N remaps almost every flow when N changes, and a hash ring with virtual nodes (see Consistent Hashing) costs a binary search and pointer-heavy memory per lookup at 10M packets per second.

Option C: Maglev table plus a local connection table. This is what Google's Maglev and Facebook's Katran do; Katran's README describes a fixed-size LRU connection table and a modified Maglev hash that supports unequal weights (Katran). Each backend gets a pseudo-random preference order over the M table slots, derived from two hashes of its name (offset and skip), and backends take turns claiming their next free preferred slot until the table is full. Lookup is one array index. Each backend owns either ⌊M/N⌋ or ⌈M/N⌉ slots, so load is balanced to within one slot. Removing a backend mostly reassigns only that backend's slots; a few others shift because the turn order changes. The connection table covers the rest: a flow already in the table keeps its backend even if the new Maglev table disagrees.

Choice: Option C. A connection breaks only when two things happen at once: its packets move to a different L4 node (which has no table entry) and the Maglev entry for its hash changed. Both are rare, and their coincidence is rarer. One detail matters: the table depends on the order in which backends take turns, so every node must sort backends by name before building, or two nodes with the same set build different tables.

A stateless alternative. GitHub's GLB Director and Cloudflare's Unimog keep no per-flow table in the balancer at all. Each table bucket names a current and a previous server, and a server that receives a mid-connection packet it has no socket for forwards it to the previous one (GitHub GLB, Unimog). This moves the state to the servers that already hold it, at the cost of an extra hop during changes and a limit of one change per bucket at a time.

2. Health checking without self-inflicted outages

The basics (active probes with rise and fall thresholds, passive outlier ejection, draining) are covered in What are Load Balancers?. At this scale the design questions are different.

Who probes. If every proxy probes every backend, probes become load (1.7M per second above) and each proxy forms its own opinion, so a backend can be up on half the fleet and down on the other. I'd run a few checkers per zone, mark a backend down when a majority of zones agree, and push one health view to every node. Proxies still apply fast passive ejection locally. AWS makes the same argument: let a central actor decide, so the whole fleet never makes the same wrong decision at once (AWS Builders' Library). The same arithmetic applies to connections: 170 proxies each holding pools to thousands of backends wastes memory and CPU on both ends, so Google's SRE book has each client connect to a deterministic subset of 20 to 100 backends (Google SRE book, ch. 20).

What a probe means. A deep check that also calls the backend's database fails on every backend at once when the database is slow, and the balancer ejects the whole pool. Probes should check the process; dependency health belongs in metrics and alerts.

Failing safe. Two guards stop the balancer from amplifying an outage. First, cap ejection: Envoy's outlier detection ejects at most 10% of a cluster by default (Envoy docs). Second, a panic threshold: when fewer than 50% of a pool's backends are healthy (Envoy's default), ignore health entirely and spread traffic across all of them, on the theory that a degraded backend serves some requests and a removed one serves none (Envoy panic threshold). AWS load balancers do the same and call it failing open (AWS ALB docs). New backends get slow start, a weight that ramps up over 30 seconds.

3. The return path, encapsulation and draining

Direct server return. If responses passed back through the L4 tier it would need to carry 2 to 4 Tbit/s instead of 400 Gbit/s. With DSR the proxy replies directly. That requires the L4 node to deliver the packet with the original destination (the VIP) intact, so it encapsulates rather than rewriting addresses: GRE in Maglev, IP-in-IP in Katran. The outer header costs 20 bytes (IP-in-IP) or 24 bytes (GRE), so a 1,500-byte client packet becomes too large for a 1,500-byte data-centre MTU. Either run the data-centre network at a larger MTU (Cloudflare uses jumbo frames for this) or clamp the TCP MSS the proxies advertise, which Katran recommends even with a larger MTU because it cannot forward fragmented packets (Katran).

Draining. An L4 node leaves by withdrawing its BGP route and staying up until the routers converge. An L7 proxy leaves by dropping out of the Maglev input (no new flows), sending HTTP/2 GOAWAY on open connections, and waiting up to the deregistration delay. Google calls this the lame duck state and suggests 10 to 150 seconds before exit (Google SRE book, ch. 20); AWS's default deregistration delay is 300 seconds (AWS ALB docs). Without this, every deploy shows up as a burst of resets.

Down the stack

Layer 1The contract

Every packet of a TCP flow reaches the same L7 proxy for the flow's lifetime, as long as that proxy stays in the pool and at most one of {L4 node membership, backend membership} changes at a time. New flows go only to backends the aggregated health view considers up, within 5 seconds of a change, unless the pool is in panic mode. Each backend receives within about 1% of its weighted share of new flows. Packets the L4 tier cannot place (no backends at all) are dropped rather than queued, so clients see a timeout or a reset, never a stall inside the balancer. Added latency is tens of microseconds at L4.

Layer 2The mechanism

For each backend name the table builder computes offset = h1(name) mod M and skip = h2(name) mod (M - 1) + 1, so backend i's preference list is (offset + j × skip) mod M for j = 0, 1, 2, .... Because M is prime, every skip is coprime with M and each list visits every slot exactly once. The builder loops over backends in sorted order, each claiming its next unclaimed preferred slot, until all M slots are full. The Maglev paper reports building a 65,537-entry table in about 1.8 ms and a 655,373-entry table in 22.9 ms, which is why table size isn't raised without limit (Maglev paper). In a Katran-style data plane, the Maglev table (an eBPF array) and the connection table (a per-CPU LRU hash) are kernel maps that a user-space process updates and the XDP program reads per packet.

Layer 3The machine

The NIC writes each packet into a receive ring in memory with DMA. In driver-mode XDP, the eBPF program runs on that buffer before the kernel allocates an sk_buff, which saves hundreds of nanoseconds per packet. It parses the Ethernet, IP and TCP headers, hashes the 5-tuple, checks the per-CPU LRU map, falls back to the Maglev array, grows the packet headroom with bpf_xdp_adjust_head to prepend the outer IP header, and returns XDP_TX to send it out the same NIC. At 10M packets per second per core on a 3 GHz CPU, the budget is 300 cycles per packet. One last-level cache miss to DRAM costs around 80 to 100 ns, which is most of that budget, so a 256 KB Maglev table that stays in cache matters as much as the algorithm. Receive-side scaling hashes flows to NIC queues, one per core, so cores never share a flow and per-CPU maps need no locks.

Why it's built this way

  1. Requests are small and responses are large, so the balancer handles only the inbound direction and lets responses return directly.
  2. Inbound packets arrive at hundreds of millions per second, so per-packet work must fit in a few hundred cycles, which rules out shared state and favours a precomputed table.
  3. Without shared state, nodes must agree by computing the same answer, so the backend choice is a deterministic function of the 5-tuple and a sorted backend list.
  4. That function changes when backends change, so a local connection table protects established flows, and the hash is designed to change as little as possible.
  5. The balancer itself can turn a partial failure into a total one, so health decisions are aggregated, capped and bypassed in panic mode.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Two tiers (L4 then L7)Each tier scales on its own bottleneckMore moving partsLarge shared platforms
Single L7 tier behind DNSSimpleDNS TTLs slow failover; TLS CPU on every hopSmall deployments
Maglev tableO(1) lookup, even load, cache-friendlySlightly more than minimal disruption; rebuild costPacket-rate L4 forwarding
Ring hash with virtual nodesMinimal disruption; easy weightsLarger memory, binary search per lookupRequest-level sharding in proxies
Direct server returnL4 tier carries inbound bytes onlyNeeds encapsulation, MTU care and VIPs on backendsResponse-heavy traffic
Full proxy (NAT both ways)Works with any backend, sees both directionsBalancer carries all bytesSmall scale or when L4 must see responses
Central health checkersOne consistent view; low probe loadAnother service to runMany proxies and many backends

Leadership lens

Check yourself

  1. An L4 node fails and the routers rehash its flows onto other L4 nodes that have no connection-table entry for them. Why do those connections survive, and when would one break?

    Show answer

    Every node builds the same Maglev table from the same sorted backend list, so the new node hashes the 5-tuple to the same proxy. A connection breaks only when its packets move to a node without a table entry and the Maglev entry for its hash changed at the same time, which is rare.

  2. Why does the design use a Maglev table plus a local connection table rather than replicating the flow table across L4 nodes or using a ring hash with virtual nodes?

    Show answer

    A shared flow table gives exact affinity but turns every new flow into a cross-node write at millions per second. A ring hash costs a binary search and pointer-heavy memory per lookup, while Maglev is one array index with load balanced to within one slot, at the cost of slightly more than minimal disruption and a rebuild on changes.

  3. Why is direct server return worth the encapsulation and MTU trouble, using the lesson's numbers?

    Show answer

    Inbound traffic is about 400 Gbit/s while responses are perhaps 2 to 4 Tbit/s, so routing responses back through the L4 tier would multiply its load several times. With DSR the L4 tier carries only inbound bytes, about 100M packets per second, which roughly 24 nodes handle.

  4. A bad deploy fails health checks on 70% of a pool. What stops the balancer from turning that brownout into a blackout?

    Show answer

    Ejection is capped (Envoy ejects at most 10% of a cluster by default), and below 50% healthy the pool enters panic mode and spreads traffic across all backends, since a degraded backend serves some requests and a removed one serves none. Probes also check only the process, not dependencies, so a slow database does not eject every backend at once.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →