Engineering Atlas › System Design Interviews

Nginx for system design interviews

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Master and workers

The master process runs as root, parses nginx.conf, opens listening sockets and forks workers. It does no request work. Each worker is an unprivileged process with its own event loop and connection table. worker_processes auto gives one worker per core, so nothing contends for a lock on the hot path.

Reload is a signal, not a restart. On SIGHUP the master parses the new config (rolling back and keeping the old one if it fails to apply), forks new workers, and tells the old ones to stop accepting and exit when in-flight requests finish. Long-lived connections keep old workers alive until worker_shutdown_timeout, so a reload can briefly double memory (nginx: controlling nginx).

A request's path through a worker

A request moves through phases: server selection by Host or SNI, rewrite, location match, access checks (allow, auth_request, limit_req), content, then logging. The content handler serves a file or hands off to the upstream module, which picks a backend and streams the response back through output filters (gzip, chunked encoding, SSL).

Load balancing

An upstream block lists servers with weights. The default algorithm is smooth weighted round-robin; alternatives are least_conn, ip_hash, hash $key consistent (a ketama-style ring), random two least_conn (power of two choices) and, since open-source 1.31.0, least_time, which was previously NGINX Plus only. Each server has max_fails (default 1) and fail_timeout (default 10 s). If a server fails max_fails times within fail_timeout, it is skipped for fail_timeout, then given a single trial request (ngx_http_upstream_module). What counts as a failure is set by proxy_next_upstream, which by default is error timeout: a 500 from the backend is passed to the client and is not a failure unless you add http_500 (ngx_http_proxy_module).

Down the stack

Layer 1The contract

For the client, Nginx is an HTTP/1.1, HTTP/2 or HTTP/3 server that returns a response or a 5xx it generated itself: 502 when the upstream connection failed or returned garbage, 504 when proxy_read_timeout (default 60 s) expired, 503 when limit_req rejected it. It retries a request on the next upstream after an error or timeout (since 1.9.13, POST, PATCH and LOCK only if non_idempotent is set), bounded by proxy_next_upstream_tries and proxy_next_upstream_timeout. Without that flag a POST is not retried once it was sent upstream, and no request is retried after response bytes reached the client, so a backend crash mid-request surfaces as a 502.

Balancing state is per worker: each keeps its own round-robin counters and failure counts unless a zone directive moves peer state into shared memory. A proxied hop within a rack adds tens to a few hundred microseconds when the upstream connection is already open.

Layer 2The mechanism

Event loop. Each worker runs ngx_process_events_and_timers: find the nearest timer in a red-black tree of timeouts, call epoll_wait (the method chosen on Linux, nginx docs) with that as the timeout, dispatch each ready event to its handler, then expire timers. Every connection is an ngx_connection_t with a read event and a write event; handlers are state machines that consume whatever bytes are available and return NGX_AGAIN when a socket would block. Nothing in the loop may block. Disk reads that might miss the page cache can be moved to a thread pool with aio threads.

Memory. Allocation uses per-connection and per-request pools (ngx_pool_t) freed in one step when the request ends (AOSA: nginx). Data moves as chains of buffers (ngx_chain_t of ngx_buf_t); a buffer can point at a file range, which is how sendfile avoids copying.

Accepting. Workers share the listening socket by default. listen 443 reuseport gives each worker its own SO_REUSEPORT socket so the kernel spreads new connections evenly.

Smooth weighted round-robin. Each peer has a static weight, an effective_weight and a current_weight. On each pick, every eligible peer adds its effective_weight to its current_weight; the peer with the highest current_weight wins and has the total subtracted. For weights 5, 1, 1 this yields a a b a c a a, spreading the light servers out instead of sending five requests in a row to a. On failure, effective_weight drops by weight / max_fails and climbs back by one per selection, so a flaky server is eased back in. Maxim Dounin added the algorithm in 2012 (nginx commit).

Upstream keepalive. keepalive 32 in an upstream block gives each worker a cache of up to 32 idle connections to that group. When a response completes cleanly, the connection goes back into the cache instead of being closed. Before 1.29.7 it only worked with proxy_http_version 1.1 and proxy_set_header Connection "", because the default proxy protocol was HTTP/1.0 with Connection: close. From 1.29.7 (and so in 1.30 stable) the defaults flipped: keepalive 32 local is on in every upstream, proxy_http_version defaults to 1.1 and no Connection header is sent (nginx CHANGES). The local flag stops locations from sharing cached connections. keepalive_requests (default 1000 since 1.19.10) and keepalive_timeout (default 60 s) bound each cached connection's life.

Layer 3The machine

epoll_wait returns only ready file descriptors, so the cost of a loop iteration scales with active connections, not total ones. That is the difference from select/poll, which scan every descriptor on every call. Nginx registers sockets edge-triggered (EPOLLET), so it must read until EAGAIN or it will not be woken again for the same data.

An idle client costs a few KB of user memory plus kernel socket buffers (hedge: TLS and HTTP/2 settings change this a lot), so 100,000 idle clients fit in a few hundred MB, provided worker_rlimit_nofile allows the descriptors.

Static files go out with sendfile, page cache to socket inside the kernel, unless TLS forces a trip through user space (kTLS avoids it). A full TLS handshake costs roughly 0.1 to 0.3 ms of CPU with ECDSA P-256 and 1 ms or more with RSA-2048 (hedge: varies by CPU), so session resumption and keepalive matter more than raw request throughput.

New upstream connections hit a hard limit. Linux's default ephemeral range (32768 to 60999, kernel docs) is about 28,000 ports, and the side that closes holds each socket in TIME_WAIT for 60 s. Toward one backend IP and port, that sustains only about 470 new connections per second from one source IP. Keepalive turns that into a handful of long-lived connections. Per-worker pools still waste reuse: when Cloudflare replaced Nginx with Pingora, whose pool is shared by all threads, one large customer's connection reuse rose from 87.1% to 99.92%, or 160 times fewer new origin connections (Cloudflare).

Why it's built this way

  1. In 2002 the C10K problem was that a thread or process per connection cost megabytes of stack and a context switch per event, so Nginx uses one process per core and nonblocking sockets.
  2. One thread per worker means no locks, but anything blocking stalls thousands of connections, so blocking work goes to thread pools.
  3. Workers do not share memory by default, which keeps them independent and crash-isolated, so load-balancing state is per worker unless you opt into a shared zone.
  4. TCP and TLS setup cost more than proxying a small request, so connections are reused on both sides.
  5. Config is compiled into in-memory structures at load time for speed, so changes require a reload; that is the root of the "static config" trade-off against xDS-driven proxies.

Diagram

sequenceDiagram
    participant C as Client
    participant W as "Worker (epoll loop)"
    participant P as "Keepalive pool"
    participant U as Upstream
    C->>W: TLS + GET /api
    W->>W: phases: rewrite, access, content
    W->>W: pick peer (smooth WRR)
    W->>P: idle conn to peer?
    P-->>W: reuse socket
    W->>U: HTTP/1.1 request
    U-->>W: 200 + body
    W->>P: return conn
    W-->>C: response via filters

Worked example

An API peaks at 60,000 requests per second. Four Nginx boxes with 8 cores each sit behind a cloud L4 load balancer, proxying to 30 app servers.

Per box that is 15,000 requests per second, under 2,000 per worker, well within one worker's capacity for small proxied responses, even with one box down.

Without upstream keepalive, each box opens 15,000 new connections per second spread over 30 backends: 500 per second per backend. That is above the roughly 470 per second that TIME_WAIT allows per source and destination pair, so after a minute or so connects start failing with EADDRNOTAVAIL and users see 502s.

With keepalive 64, each worker caches up to 64 idle connections. At 20 ms per upstream request a worker needs about 2,000 × 0.02 = 40 concurrent connections, so nearly every request reuses one and p99 drops by about one RTT.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Nginx OSSFast, tiny, well understood, great for static files and TLSReload for config changes, passive health checks only, per-worker LB stateEdge proxy with mostly static upstreams
NGINX PlusActive health checks, API for upstream changes, metricsLicence costYou want Nginx and dynamic backends
EnvoyxDS dynamic config, rich observability, retries, circuit breaking, gRPCHeavier, more complex configService mesh or fast-changing backends
HAProxyExcellent L4/L7 LB, runtime API, strong health checksNot a web server for static contentPure load balancing at high connection counts
Cloud LB (ALB, GCLB)No servers, autoscaling, managed TLSLess control, per-LCU cost, vendor limitsYou do not want to run proxies

Pitfalls in production

Leadership lens

Check yourself

  1. Why does Nginx run one worker per core instead of one thread per connection?

    Show answer

    Threads cost stack memory and context switches per event; one nonblocking event loop per core handles thousands of connections with no locks and only pays for sockets that are ready.

  2. A backend returned 500 ten times in a row, yet Nginx keeps sending it traffic. Why?

    Show answer

    By default only connection errors and timeouts count as failures; a 500 is a valid response. Add http_500 to proxy_next_upstream if it should count.

  3. With weights 5, 1, 1, why does smooth weighted round-robin produce a a b a c a a instead of five as then b, c?

    Show answer

    Each pick subtracts the total from the winner's current weight, so the heavy server falls behind the light ones briefly and the picks interleave.

  4. What breaks first at high RPS if upstream keepalive is off?

    Show answer

    Ephemeral ports in TIME_WAIT toward each backend run out at a few hundred new connections per second per pair, producing connect errors and 502s, alongside wasted handshake CPU.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →