Engineering Atlas › System Design Interviews

Design an API gateway service

20 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope

Back-of-envelope

API

The data plane accepts any HTTP request. The control plane is a small admin API.

PUT /admin/v1/routes/{route_id}
{ "hosts": ["api.example.com"], "method": "GET",
  "path": "/v2/orders/:order_id", "upstream": "orders-svc",
  "plugins": [ {"name": "jwt_auth", "on_error": "closed"},
               {"name": "rate_limit", "limit": 100, "window_s": 1, "on_error": "open"} ],
  "timeout_ms": 2000, "retry": {"on": ["connect-failure"], "max": 1} }
-> 200 { "route_id": "...", "config_version": 41872 }

GET /admin/v1/config/versions/{n}          -> 200 { "version": n, "sha256": "...", "created_by": "...", "diff": [...] }
POST /admin/v1/config/rollback {"to": n}   -> 202 { "config_version": 41873 }

Data plane responses added by the gateway:
429 { "error": "rate_limited" }  + Retry-After, RateLimit, RateLimit-Policy
401 { "error": "unauthenticated" }  503 { "error": "upstream_unavailable" }

The rate-limit headers follow the current IETF draft, which replaced the older RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset trio with two structured fields, for example RateLimit: "default";r=50;t=30. It is still an Internet-Draft, not an RFC, so clients must tolerate changes (IETF draft-11).

Nodes receive config over a streaming RPC similar to Envoy's xDS: the node sends the version it has, the control plane streams the next snapshot, and the node ACKs or NACKs it (Envoy xDS protocol).

Data model

High-level design

flowchart LR
  C["Clients"] --> DNS["GeoDNS / Anycast"]
  DNS --> L4["L4 load balancer"]
  L4 --> GW["Gateway node: TLS, router, plugin chain"]
  GW --> RL["Rate-limit store (sharded)"]
  GW --> JW["JWKS cache"]
  GW --> UP["Upstream service clusters"]
  GW --> TEL["Log and metrics pipeline"]
  ADM["Admin API"] --> PG["Postgres config"]
  PG --> COMP["Config compiler and validator"]
  COMP --> SNAP["Snapshot store"]
  SNAP --> XDS["Config distribution (streaming)"]
  XDS --> GW

Request path. DNS or anycast sends the client to the nearest region. An L4 load balancer spreads TCP connections across gateway nodes. The node terminates TLS, parses HTTP, and looks up the route in its current snapshot. It then runs the route's plugin chain: authenticate (verify the JWT signature with a cached public key, or hash the API key and look it up in a local cache backed by the consumer store), then rate limit (one round trip to the counter shard, or a local token bucket for hot keys), then transforms. If every plugin passes, the node picks a healthy upstream endpoint, reuses a pooled connection, and forwards the request with the consumer id and trace context in headers. The response streams back through response-phase plugins.

Config path. An engineer changes a route through the admin API. The change is written to Postgres with a new version number. The compiler loads the full config, validates it (no pattern conflicts, all clusters exist, plugin configs parse), compiles it into the snapshot format, stores it by hash, and notifies the distribution service, which streams it to every connected node. Each node builds the new route trie off the hot path and swaps a single pointer. In-flight requests finish on the old snapshot.

Deep dives

1. Route matching and precedence

Options: a linear scan of regexes in priority order, a hash map on exact path, or a segment trie. A regex scan costs O(routes) per request and makes precedence depend on list order. A hash map can't do parameters. I'd compile a trie per (host, method), where each node has three kinds of children: literal segments in a hash map, a single parameter child, and a single wildcard terminal. Lookup tries literal first, then parameter, then wildcard, backtracking if a deeper branch fails. That yields the rule "the most specific route wins, compared segment by segment": /users/me beats /users/:id, /static/css/ beats /static/, and /orgs/:org/repos/:repo beats /orgs/*. Lookup is O(path segments), which is a few hundred nanoseconds. The compiler rejects ambiguous shapes such as /a/:x and /a/:y on the same method, because nothing in the URL can disambiguate them. Regex routes, if allowed at all, run after the trie fails and use a linear-time engine such as RE2.

2. The plugin chain, auth and rate limiting

Order matters and is fixed by the platform, not by each team: request-size limit, authentication, authorization, rate limiting, transforms, then the upstream call. Authentication comes before rate limiting so the quota is charged to a known consumer; unauthenticated traffic gets a separate, much smaller per-IP bucket so a credential-stuffing burst can't spend a real customer's quota. JWTs are verified locally against a cached JWKS in tens of microseconds, so the identity provider is off the hot path; the cost is that revocation waits for token expiry. High-risk routes can instead ask the identity provider on each call through token introspection (RFC 7662), paying a round trip for instant revocation.

Two security rules come with forwarding identity in headers. The gateway must strip any consumer-id or identity headers the client sent before adding its own, and backends must accept traffic only from the gateway (mutual TLS or a network policy), or a caller can bypass auth and quotas by going around it. Azure's Gateway Offloading pattern lists both: prevent gateway bypass, and never treat unverified forwarded headers as proof of identity (Azure Architecture Center).

For rate limiting I considered three options. Purely local buckets need no network but each node sees only 1/N of a consumer's traffic, so a limit of 100 RPS becomes anywhere from 100 to 100 × N. A central counter store is accurate but adds a round trip (around 200 to 500 µs in-region) and a dependency. Hybrid: a central sliding-window counter for normal keys, plus local token leases for the few very hot keys, where a node fetches a batch of 50 tokens and spends them locally. I'd choose the hybrid. Every plugin declares on_error: auth fails closed, rate limiting fails open to a conservative local limit, because a counter outage should not become an API outage. Envoy's global rate limit filter makes the same choice explicit: it allows traffic when the rate limit service doesn't answer unless failure_mode_deny is set, and its call times out after 20 ms by default (Envoy docs).

3. Config distribution and safe change

A bad route can take down every API at once, so config is treated like code. Changes are validated by the compiler, diffed, and rolled out in waves: first to a canary node group carrying about 1% of traffic, watched for error rate and latency for a few minutes, then to everyone. Nodes NACK a snapshot they can't load and keep the old one. Snapshots are immutable and content addressed, so every node can report exactly which hash it's serving and rollback is a pointer move. Below a few tens of MB, push full snapshots rather than deltas, because a full snapshot is idempotent and can't drift. If the control plane is down, nodes keep serving and new nodes boot from the latest snapshot in object storage.

Down the stack

Layer 1The contract

A request entering a healthy node is matched against exactly one config version and either forwarded to one upstream endpoint or rejected with a gateway-generated status (401, 404, 413, 429, 503, 504). Non-idempotent methods are retried only when the gateway can prove the upstream never received the request, such as a connect failure or a refused stream, which is the rule HTTP itself sets for automatic retries (RFC 9110, section 9.2.2). Gateway overhead is under 1 ms p50 and 5 ms p99. Rate limits are enforced within about 10% of the configured value inside a region and are best effort across regions. Managed gateways promise no more: AWS API Gateway documents its token-bucket throttles as targets applied on a best-effort basis, not guaranteed ceilings (AWS docs). A config change becomes visible on all nodes within 10 seconds, and a node never mixes routes from two versions within one request.

Layer 2The mechanism

In an Envoy-style proxy, a listener accepts connections and runs a network filter chain. The HTTP connection manager decodes HTTP/1.1 or HTTP/2 frames into a stream and runs HTTP filters in order (decodeHeaders, decodeData, then encodeHeaders on the way back). Any filter can stop iteration and send a local reply, which is how auth and rate limiting short-circuit. The route table is a list of virtual hosts, each with routes. The config arrives over xDS as DiscoveryResponse messages carrying version_info, a nonce and typed resources (listeners, routes, clusters, endpoints). The node answers with a DiscoveryRequest that echoes the nonce and either the new version (ACK) or the old version plus error_detail (NACK). Each worker thread holds a thread-local pointer to the active snapshot, and the main thread publishes a new one by posting it to every worker's event loop, a read-copy-update pattern with no locks on the request path (Envoy threading model). Upstream clusters hold per-worker connection pools, outlier detection (eject an endpoint after N consecutive 5xx) and circuit breaker counters (max pending requests, max retries in flight).

Layer 3The machine

Each worker runs a single-threaded event loop around epoll_wait. With SO_REUSEPORT, the kernel hashes incoming SYNs across one listen socket per worker, so no lock is taken on accept. A full TLS 1.3 handshake with an ECDHE key exchange and an ECDSA certificate costs very roughly 0.3 to 1 ms of CPU on the server; session resumption cuts most of that, which is why resumption rates are a capacity metric. Reads land in kernel socket buffers (default sizes are tens of KB, and autotuning grows them up to a few MB), and the proxy copies them into its own buffer slices. Per-connection buffer limits ("watermarks") matter: 600k connections each buffering 64 KB would need 38 GB. Upstream connections are kept alive so a request skips the TCP handshake (one in-region RTT, around 0.2 to 0.5 ms) and the TLS handshake. Stats counters are per worker and merged periodically, because a shared atomic counter incremented by 16 cores bounces one 64-byte cache line between them and can cost 50 to 100 ns per increment under contention. Access logs go to an in-memory ring buffer and are flushed with batched writev calls rather than one write per request.

Why it's built this way

  1. Every API pays the gateway's latency on every call, so the hot path must do no blocking I/O except what's unavoidable (the upstream call and at most one counter lookup).
  2. Blocking I/O is avoided by event loops, and event loops can't share mutable state cheaply, so config is immutable and swapped by pointer per worker.
  3. Immutable snapshots need a producer, so a control plane compiles and validates them; because it is off the hot path, it can be slow, strongly consistent and sometimes unavailable.
  4. A shared gateway is a shared blast radius, so config changes are staged, versioned and reversible, and each plugin has a declared failure mode.
  5. CPU, mostly TLS and parsing, is the scarce resource, so connection reuse and session resumption decide the node count more than bandwidth does.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Central gateway fleetOne place for auth, quotas and telemetry; easy to auditShared blast radius; extra hopPublic APIs, many teams, compliance needs
Per-service gateway (backend for frontend)Teams own their edge; isolationDuplicated policy; drift between teamsFew large clients with very different needs
Local rate limits onlyNo network hop; never failsInaccurate by up to N timesCoarse abuse protection
Central counter storeAccurate limitsExtra round trip; a dependencyPaid quotas and contractual limits
JWT verified locallyNo IdP call per requestRevocation waits for expiryShort-lived tokens (5 to 15 minutes)
Opaque token introspectionInstant revocationIdP on the hot pathHigh-risk admin APIs
Full snapshot pushSimple; no driftLarge payloads at scaleConfig under tens of MB

Leadership lens

Check yourself

  1. The rate-limit counter store goes down. What should the gateway do, and why is that plugin configured differently from authentication?

    Show answer

    Rate limiting fails open to a local per-node bucket set to the limit divided by the node count, which bounds the overshoot, because a counter outage should not become an API outage. Authentication fails closed, since letting unauthenticated calls through is not an acceptable degradation.

  2. Why choose a hybrid of a central sliding-window counter plus local token leases over purely local buckets or a purely central store?

    Show answer

    Purely local buckets need no network hop, but each node sees only 1/N of a consumer's traffic, so a limit can be exceeded up to N times. A central store is accurate but adds a round trip and a dependency for every key; the hybrid keeps central accuracy for normal keys and gives the few very hot keys batches of local tokens so they do not melt one shard.

  3. Walk through the node-count estimate. Why run nodes at 50% CPU instead of 90%?

    Show answer

    At about 4k RPS per core, a 16-core node does about 60k RPS at 90% CPU, or 30k at 50%, so 500k RPS needs about 17 nodes, rounded up to 10 per region across three regions. The 50% target leaves headroom for spikes and lets two regions carry the full load if a third is withdrawn.

  4. Why are routes compiled into a segment trie instead of a priority-ordered list of regexes?

    Show answer

    A regex scan costs O(routes) per request and makes precedence depend on list order, while the trie matches in O(path segments) with a fixed rule: literal beats parameter beats wildcard. That makes matching deterministic across nodes, and the compiler can reject ambiguous shapes like /a/:x and /a/:y up front.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →