Engineering Atlas › System Design Fundamentals

Putting an API gateway at the front door

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Data plane and control plane

The data plane is a fleet of identical proxy nodes (Envoy, NGINX, HAProxy, or a managed service's hidden fleet). Each node holds a compiled copy of the configuration in memory and never asks anyone a question on the hot path if it can avoid it. The control plane owns the source of truth: routes, upstream clusters, API keys, plugin settings. Kong stores this in Postgres or declarative YAML; Envoy-based gateways such as Envoy Gateway, Contour and Istio's ingress translate Kubernetes objects into Envoy config and stream it out.

The two halves fail differently. If the control plane dies, proxies keep serving the last good config: you lose the ability to change things, not to serve. If the data plane dies, everything stops.

The configuration model

A GatewayClass names the implementation, owned by the platform team. A Gateway declares listeners (ports, hostnames, TLS certificates), owned by whoever runs the edge. An HTTPRoute attaches to a Gateway and says which paths, headers and methods go to which backends with what weights, and it is owned by the service team. Platform teams own the door; product teams own the rooms behind it. Gateway, GatewayClass and HTTPRoute are stable, and a GRPCRoute does the same job for gRPC (Gateway API docs). This model is now the default path: the community Ingress NGINX controller was retired in March 2026 with no further releases or security fixes, and the Kubernetes project points users to Gateway API (Kubernetes blog).

The edge contract

Azure's architecture guide splits gateway work into three patterns: routing one endpoint to many services, aggregation of several backend calls into one response, and offloading cross-cutting work such as TLS, authentication and throttling (Azure Architecture Center). Offloading is what makes a gateway worth having. What it gives you, beyond routing, is a set of promises every service can rely on:

The problem shapes

A gateway is the overload shape (it must shed and limit load before it reaches services) combined with trust but verify (it turns an untrusted request into a verified one). The same shape appears in an LLM gateway that fronts several model providers: one door, a verified tenant, a token budget, and a weighted route to a cheaper model for a slice of traffic.

Down the stack

Layer 1The contract

For each request the gateway promises: a request whose credential fails verification never reaches an upstream; internal identity headers seen by an upstream were set by the gateway, never by the client; the upstream receives a deadline no larger than the route's configured timeout; the same user hashes to the same side of a weighted split while the weight is unchanged; and the request either gets an upstream response or a gateway-generated status (401, 404, 429, 502, 503, 504) within the route timeout. For configuration the promise is weaker: a change becomes visible on each node within seconds, nodes may disagree during that window, and a config that a node rejects leaves that node on its previous version.

Layer 2The mechanism

Envoy receives config through xDS, a set of gRPC streams: LDS for listeners, RDS for route tables, CDS for clusters, EDS for endpoints. Each DiscoveryResponse carries a version_info and a nonce. The proxy replies with a DiscoveryRequest echoing the nonce; if it accepted the update it echoes the new version (an ACK), and if validation failed it repeats the old version with an error_detail (a NACK). That is how a control plane learns that half the fleet rejected a route. An ACK only means the resources passed validation, not that traffic is already flowing through them (Envoy xDS protocol). Delta xDS sends only changed resources, which matters once you have tens of thousands of routes.

On the request path, the HTTP connection manager runs filters in order, typically jwt_authn, ext_authz, local_ratelimit, then router. The router enforces a route timeout (15 seconds by default, and it covers all retries) and optional per_try_timeout (Envoy route API), and tells the upstream its budget with the x-envoy-expected-rq-timeout-ms header on internal requests (Envoy router filter). Retries are capped by the cluster's circuit breaker (3 concurrent retries by default) or by an optional retry budget, whose defaults let retries be at most 20% of active requests to a cluster, with a floor of 3 concurrent retries, so a failing upstream cannot be hit with three times its normal load (Envoy circuit breakers). Weighted clusters choose a backend per request by a random draw, or by a hash of a header when you configure the policy to make it sticky.

Layer 3The machine

Envoy runs one worker thread per core, and a connection lives on one worker for its whole life, so the hot path takes no locks. New configuration is built on the main thread and published to workers through thread-local slots, so a route update is a pointer swap per worker, not a pause. Bodies are the expensive part. A gateway that buffers a request body (to verify a signature over it, or transcode it) holds it in memory; Envoy's default per_connection_buffer_limit_bytes is 1 MiB (Envoy listener API), so 5,000 concurrent 1 MiB uploads would need about 5 GB of RAM if fully buffered. Streaming avoids that, which is why body-inspecting plugins should be the exception.

Restarting a proxy without dropping connections relies on the kernel. Envoy's hot restart passes listening sockets from the old process to the new one over a Unix domain socket using SCM_RIGHTS, then the old process drains for a configurable period. NGINX does the same with a master process that forks new workers on reload while old workers finish their requests. Either way the listen socket never closes, so the kernel's accept queue keeps taking SYNs throughout.

Why it's built this way

  1. Every service needs the same edge rules, and implementing them in each language drifts, so the rules move into one proxy.
  2. That proxy is on every request, so it must answer from memory, which forces config to be compiled and pushed rather than looked up.
  3. Pushed config can be wrong, so the protocol carries versions and ACK or NACK, and proxies keep serving the last good version.
  4. A shared retry or timeout policy multiplies load across every service, since attempts multiply at each layer (Google SRE book), so retries are budgeted as a fraction of traffic rather than counted per request.
  5. Memory per connection is the scaling limit, so gateways stream bodies and buffer only when a plugin demands it.

Diagram

sequenceDiagram
  participant CP as Control plane
  participant GW as Gateway node
  participant C as Client
  participant S as Orders v1
  participant K as Orders v2 canary
  CP->>GW: RDS update v42 (canary 20%)
  GW-->>CP: ACK v42
  C->>GW: GET /orders/7, Bearer token, x-user-id spoofed
  GW->>GW: strip x-internal and x-user-id, verify token
  GW->>GW: hash(user, route) mod 100 = 88, stable
  GW->>S: GET /orders/7, x-internal-user, deadline 800ms
  S-->>GW: 200
  GW-->>C: 200

Worked example

A retail API serves 30,000 requests per second at peak through 10 gateway nodes. The orders team ships v2 and asks for a 5% canary. With per-request random weighting, a user making 20 calls in a session has a 1 - 0.95^20, about 64%, chance of touching v2 at least once, and v1 and v2 disagree about a cart field. Switching to a sticky split, a hash of the user ID modulo 100 compared against 5, puts exactly the same 5% of users on v2 every time, and raising the weight to 20% only adds users; nobody who was on v2 leaves it.

Deadlines next. The mobile client times out at 2 seconds. The gateway route timeout is 1.5 seconds, and the client sends x-timeout-ms: 1200, so the gateway forwards a budget of 1,200 ms. Orders calls inventory and pricing in sequence; after 700 ms in inventory, orders knows it has 500 ms left and passes that down. Without propagation, pricing would run on its own 3 second timeout for a response nobody reads.

Retries last. During a v2 bug, 20% of orders requests fail. With a policy of 2 retries per request and no budget, failing requests triple, adding about 30,000 x 0.2 x 2 = 12,000 extra requests per second to an already sick service. With a 20% retry budget, the retries are capped at a fifth of active requests, which is roughly what the healthy fleet can absorb.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Managed gateway (AWS API Gateway, Apigee)No fleet to run, built-in keys, quotas and usage plansPer-request price, payload limits, a 29-second default integration timeout that only Regional and private REST APIs can raise, at the cost of throttle quota (AWS)Public APIs with moderate traffic and a small platform team
Self-hosted Envoy-based gatewayFast, xDS-driven, rich routing and retry controlYou run the control plane and the upgrade cadenceHigh traffic, Kubernetes, need for fine control
Kong or NGINX with pluginsHuge plugin ecosystem, simple to extendPlugin code runs on the hot path; Lua and config sprawlMany teams needing auth, keys and transforms quickly
Thin edge plus service meshEdge stays small, policy lives near servicesTwo systems to understand and debugMany internal services that also need mTLS and retries

Pitfalls in production

Leadership lens

Check yourself

  1. The control plane is down for 20 minutes. What still works and what does not?

    Show answer

    Proxies keep serving with their last accepted config, so traffic flows. You cannot add routes, rotate keys pushed through config, or change weights until it returns.

  2. Why must the gateway delete x-user-id from the client request even on routes that require authentication?

    Show answer

    If the filter that sets it is skipped or misconfigured on one route, the client's own value would reach the service. Stripping first makes the safe outcome the default.

  3. Why hash user ID and route together for a canary split rather than user ID alone?

    Show answer

    With the route in the hash, the same users are not the canary population for every service at once, so one unlucky group does not absorb every experiment.

  4. How does a retry budget differ from a retry count?

    Show answer

    A count limits retries per request, so total retries grow with failures. A budget caps retries as a fraction of current traffic, so retry load stays bounded however many requests fail.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →