Engineering Atlas › System Design Fundamentals

Round robin, least connections and other balancing rules

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Static: round robin and its relatives

Round robin hands requests to backends in turn, which is ideal only when every request and server is alike. Weighted round robin gives a server with weight 3 three turns for every one of a weight-1 server, which handles mixed instance sizes and canaries at 5% of traffic. The naive version sends three requests in a row to the heavy server; NGINX's smooth weighted round robin interleaves them (for weights 5, 1, 1 it produces a, a, b, a, c, a, a) so no server sees bursts (NGINX source).

Random is round robin without the shared cursor: as even over time, with no coordination between balancers.

Hashing for affinity

Hashing the client IP, a session cookie, a user ID or the URL sends the same key to the same backend. That keeps per-user caches warm and lets in-memory sessions work. Plain hash mod N reshuffles almost everything when N changes, so production balancers use consistent hashing: a ring, rendezvous hashing, or a Maglev table (see the Consistent Hashing lesson). Affinity trades away balance: one heavy user or a popular URL lands on one machine.

Dynamic: least connections and least requests

Least connections picks the backend with the fewest open connections. It suits long-lived connections of varied length, such as database proxies or WebSockets. For HTTP, where one connection carries many requests, least outstanding requests counts in-flight requests instead. By Little's law, in-flight requests equal arrival rate times latency, so a backend that slows down accumulates in-flight work and automatically receives less. It makes the balancer route around slowness, not just death.

Latency-aware: EWMA

Some balancers track an exponentially weighted moving average of each backend's response time and multiply it by in-flight requests to estimate the cost of the next one. Finagle and Linkerd use a "peak EWMA" that jumps up immediately on a slow response and decays slowly, so a backend that hiccups is avoided at once and trusted again gradually. Finagle's docs note the catch: the signal assumes slow means loaded, which breaks for long-polling clients whose requests are slow by design (Finagle).

Power of two choices

Picking the global minimum needs a fresh view of every backend, and when many balancers share a stale view they all pick the same "least loaded" server at once: a herd. Instead, sample two backends at random and take the less loaded. Randomness spreads the choices; comparing two still avoids busy servers. Envoy's LEAST_REQUEST, HAProxy's random (two draws by default) and open-source NGINX's random two least_conn (since 1.15.1) all work this way (HAProxy docs, NGINX upstream module).

Slow start and warm-up

A fresh backend with an empty cache, a cold JIT or an unwarmed connection pool looks idle to least-requests and gets flooded. Slow start ramps its effective weight up over 30 to 60 seconds. The same problem shape appears in ride-hailing, where a driver who just came online should not be offered every nearby trip at once.

Down the stack

Layer 1The contract

The algorithm receives the set of currently eligible backends, already filtered by health and draining, and returns one of them, in constant or near-constant time and without blocking. It promises a distribution, not an outcome: round robin promises equal shares over a cycle, weighted round robin shares proportional to weights, hashing that equal keys map to equal backends while membership is fixed, and load-aware algorithms only that they prefer backends with lower observed load. The inputs are local and stale. A balancer knows its own in-flight counts, not those of the other 40 instances sending to the same backends.

Layer 2The mechanism

NGINX's smooth weighted round robin, in ngx_http_upstream_round_robin.c, keeps weight, effective_weight and current_weight per peer. Each pick adds every peer's effective weight to its current weight, chooses the peer with the highest current weight, and subtracts the total from the winner. effective_weight drops on failures and recovers on successes, giving soft penalties. Envoy's LEAST_REQUEST samples choice_count hosts (default 2) and, when weights differ, divides weight by (active_requests + 1) raised to active_request_bias (default 1.0; at 0 it degrades to plain round robin) (Envoy load balancers). HAProxy offers roundrobin (dynamic weights, capped at 4,095 active servers per backend), static-rr, leastconn, source, uri, hdr(name) and random(<draws>). AWS Application Load Balancers let a target group choose round_robin, least_outstanding_requests or weighted_random, the last able to shift weight away from targets whose error rate is anomalous compared with their peers (AWS ALB docs). Google's internal balancing, described in the SRE book, uses weighted round robin where backends report query rate, error rate and utilisation in every response and failures count as a penalty, and gives each client a deterministic subset of 20 to 100 backends so connection counts stay bounded (Google SRE book). The same idea is now open: gRPC's weighted_round_robin policy and Envoy's client-side weighted round robin compute each backend's weight as qps / (utilization + eps/qps × penalty) from ORCA load reports, so a backend that fails fast gets a low weight instead of a high one, and a new backend gets no weight until it has reported for a 10-second blackout period (gRFC A58).

Layer 3The machine

Envoy runs one load balancer per worker thread, each with its own counters, so a proxy with 16 workers is really 16 independent balancers. Sharing one counter array across threads would cost an atomic increment on a contended cache line per request, roughly 20 to 100 ns under contention instead of about 1 ns locally, and a lock around a global minimum search would serialise every request. Random sampling needs a fast thread-local generator and two memory reads. The deeper limit is information: with 40 balancer instances and 16 workers each, there are 640 independent deciders. Mitzenmacher's analysis (thesis) shows why two choices is enough: placing n jobs on n servers at random leaves the busiest server with about log n / log log n jobs, while picking the less loaded of two random servers cuts that to about log log n / log 2, an exponential improvement for one extra read.

Why it's built this way

  1. Requests differ in cost and servers differ in speed, so equal counts do not mean equal load, and the balancer needs a load signal.
  2. In-flight requests reflect both rate and latency (Little's law), so they are a cheap, local load signal that needs no reporting protocol.
  3. Each balancer's view is local and stale, so always choosing the global minimum herds; random sampling of two breaks the herd.
  4. Some workloads need cache or session locality, so hashing exists, and consistent hashing limits the damage when membership changes.

Diagram

flowchart TD
  R["new request"] --> H{"needs affinity?"}
  H -- "yes" --> CH["consistent hash on key"]
  H -- "no" --> E["eligible backends: healthy, not draining"]
  E --> S["sample two at random"]
  S --> C{"compare in-flight x EWMA latency"}
  C --> P["pick the cheaper one"]
  CH --> P2["owner from ring"]
  P --> F["increment in-flight, send"]
  P2 --> F
  F --> D["on response: decrement, update EWMA"]

Worked example

A search service runs 10 backends behind one balancer tier at 5,000 requests per second, 500 per backend. Normal latency is 20 ms, so each backend has about 500 × 0.02 = 10 requests in flight.

Backend 4 enters a long garbage collection phase and its latency rises to 500 ms. With round robin it still receives 500 requests per second. Its in-flight count climbs to 500 × 0.5 = 250, and 10% of all requests now take half a second. The service's p99 latency jumps from about 40 ms to 500 ms, even though nine of ten servers are fine.

With least outstanding requests, backend 4's in-flight count rises above everyone else's and it stops winning comparisons. Traffic settles where its in-flight count roughly matches the others, about 10 requests, so it receives about 10 / 0.5 = 20 requests per second. That is 0.4% of traffic. The p99 returns close to normal, and only the p99.9 shows the slow server. The other nine absorb 4,980 requests per second, about 553 each, well within capacity.

Now the trap. Suppose backend 4 instead returns HTTP 500 in 1 ms because its database pool is exhausted. Its in-flight count is near zero, so least-requests sends it more traffic, and the error rate climbs. Load-aware algorithms need passive health checking or error-weighted penalties, or a fast-failing server becomes a black hole. Utilisation-weighted round robin with an error penalty, as in gRPC's policy above, would give this backend a weight near zero instead.

In the wild

Trade-offs

AlgorithmStrengthsWeaknessesChoose when
Round robinSimple, predictable, no state beyond a cursorIgnores request cost and server speedUniform requests on identical servers
Weighted round robinHandles mixed capacity and canary splitsWeights are static guessesKnown capacity differences, traffic shifting
RandomNo coordination, even over timeShort-term imbalance, ignores loadMany independent balancers, simple stacks
Least connectionsGood for long, varied connectionsWrong signal for multiplexed HTTP/2WebSockets, database proxies, TCP
Least requests with P2CRoutes around slow servers, no herdingFast-failing servers attract trafficDefault for HTTP and gRPC services
Peak EWMAReacts to latency directlyMore state and tuningLatency-sensitive RPC with variable backends
Consistent hashingCache and session affinityHot keys, uneven loadSharded caches, sticky sessions

Pitfalls in production

Leadership lens

Check yourself

  1. Why does least outstanding requests route around a slow server without any explicit latency measurement?

    Show answer

    In-flight requests equal arrival rate times latency, so a slower server accumulates more in-flight work at the same rate and loses comparisons until its rate drops.

  2. Why can choosing the single least loaded backend be worse than choosing the better of two random ones?

    Show answer

    Many balancers with similar stale views all pick the same minimum at once and overload it. Random sampling spreads their choices while still avoiding busy servers.

  3. A server starts answering every request with an error in 1 ms. What does least-requests do, and how do you fix it?

    Show answer

    It sends that server more traffic because it always looks idle. Add outlier detection that ejects it after consecutive errors, or penalise errors in the load estimate.

  4. For weights 3 and 1, what problem does smooth weighted round robin solve compared with the naive version?

    Show answer

    The naive version sends three requests in a row to the heavy server, producing bursts. The smooth version spreads them so the light server's turn falls between them.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →