Engineering Atlas › System Design Fundamentals

Routing and policy at the API gateway

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The request path

A typical request goes through these stages: TLS termination, request parsing and size limits, authentication (validate a JWT signature or an API key), authorisation at coarse grain (does this key have the orders:read scope?), rate limiting per key or tenant, route matching to an upstream service, request transformation (header injection, path rewriting), the upstream call with a timeout and retry policy, response transformation, and finally logging and metrics. Each stage adds microseconds to a few milliseconds; a well-run gateway adds 1 to 5 ms at p50.

Routing

Routes match on host, path prefix, method, and sometimes headers. Longest-prefix matching is the norm, so /orders/returns can go to a different service than /orders. Routing tables can also do weighted splits for canary releases (5% to orders-v2; Envoy calls this weighted_clusters) or tenant-based routing for dedicated deployments.

Policy enforcement at the edge

Centralising authentication means services can trust an injected identity header (for example a verified user ID and tenant) instead of each re-implementing token validation. Envoy's JWT filter, for instance, can forward the verified claims upstream in a header of your choosing (Envoy docs). Rate limiting at the gateway protects every service from abusive clients; a distributed limiter uses a shared store such as Redis or a local token bucket per gateway node with limits divided by node count. Envoy's own guidance is to run both: a coarse local bucket absorbs large bursts so they never reach the global service, which then applies the precise limit (Envoy docs). Request size limits, schema validation, and WAF rules stop bad input early.

Aggregation

A gateway can fan out to several services and merge results, turning six mobile round trips of 150 ms each into one. Keep aggregation shallow and generic; if it starts needing client-specific shaping, that belongs in a Backend for Frontend.

Deployment

Gateways run as a horizontally scaled, stateless fleet behind a layer 4 load balancer, with configuration pushed from a control plane or a declarative repository. Products include managed services (AWS API Gateway, Azure API Management, Apigee) and self-hosted proxies (Kong, Envoy-based gateways, NGINX).

On Kubernetes, the configuration surface is changing. The Gateway API reached v1.0 in October 2023 and is the successor to the older Ingress resource (Kubernetes blog). The community ingress-nginx controller, once the default choice, was retired: best-effort maintenance ended in March 2026, with no further releases or security fixes (Kubernetes blog). A gateway design written today should target the Gateway API rather than Ingress annotations.

Down the stack

Layer 1The contract

For each request the gateway promises: reject unauthenticated calls before any upstream sees them; return 429 when a caller exceeds its limit (the spec allows, but does not require, a Retry-After header, and says 429 responses must not be cached; RFC 6585); route to exactly one upstream cluster according to current config; apply a bounded timeout and a declared retry policy; and emit one access log line and metrics. Config changes are eventually consistent: different nodes may run different versions for a few seconds during a rollout.

Layer 2The mechanism

In Envoy, a request flows through a listener, a filter chain, and the HTTP connection manager, which runs an ordered list of HTTP filters. A typical chain is jwt_authn (verifies the token signature against a cached JWKS and checks iss, exp and nbf, plus aud only if you list audiences: leave that out and any audience is accepted), ext_authz for custom policy, local_ratelimit (a token bucket per node), and finally router, which matches the route table (prefix, path, header matchers), picks a cluster, asks its load balancer for an endpoint, and takes a connection from that cluster's pool. Global limits call a rate-limit service over gRPC (ShouldRateLimit) with descriptors such as ("api_key", "abc"), backed by Redis counters. Route and cluster config arrives from a control plane over xDS streams, so updates apply without restarts.

Layer 3The machine

Envoy runs one worker thread per hardware thread (set by --concurrency), each with its own non-blocking event loop over epoll; an accepted connection stays on one worker for its whole life, avoiding locks on the hot path (Envoy threading model). TLS 1.3 costs one round trip and, on the server, an X25519 key exchange plus a certificate signature, tens to a few hundred microseconds of CPU per full handshake, which is why session resumption and keep-alive matter. Verifying an RS256 JWT is an RSA public-key operation of tens of microseconds; fetching JWKS on the request path would add a network round trip, so keys are cached. A Redis rate-limit check adds a same-zone round trip of about 0.2 to 0.5 ms. Upstream connections are pooled and multiplexed over HTTP/2, so the gateway does not pay TCP and TLS setup per request.

Why it's built this way

  1. Every request must be authenticated and limited, and doing it once at the edge is cheaper than in every service.
  2. Cryptography and connection setup cost CPU and round trips, so TLS is terminated once and upstream connections are reused.
  3. Shared counters require a network hop, so most limiting is local and only strict limits use a central store.
  4. Nodes must scale horizontally and fail independently, so they are stateless and receive config from a control plane.

Diagram

flowchart LR
    C["Mobile / web / partner"] --> LB["L4 load balancer"]
    LB --> GW["Gateway node"]
    GW --> A{"Auth valid?"}
    A -->|"no"| R401["401"]
    A -->|"yes"| RL{"Within rate limit?"}
    RL -->|"no"| R429["429"]
    RL -->|"yes"| RT["Route match"]
    RT -->|"/orders"| O["Orders service"]
    RT -->|"/catalog"| CAT["Catalog service"]
    RT -->|"/users"| U["Users service"]
    GW -.-> OBS["Logs, metrics, traces"]

Worked example

A SaaS company exposes a public API to 3,000 partner integrations, peaking at 40,000 requests per second. Before the gateway, six services each implemented API-key checks differently, and one partner's bug flooded the search service with 15,000 requests per second, causing an outage.

They deploy an Envoy-based gateway fleet: 12 nodes, each comfortably handling 8,000 requests per second, leaving headroom for a node failure and a 2x spike. Rate limits are 100 requests per second per partner by default, enforced by a local token bucket on each node (100 / 12, about 8.3 per second per node, with burst capacity of 50) plus a global Redis check for the top 50 partners with custom contracts. The local approach avoids a Redis round trip on 95% of requests.

Gateway latency is 1.8 ms p50 and 6 ms p99. JWT validation uses cached signing keys refreshed every 5 minutes, so no call to the identity provider is on the hot path. The next time a partner misbehaves, it receives 429s within a second and no backend sees the excess load.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
No gateway (clients call services)No extra hop, no central bottleneckPolicy duplicated, internal topology exposedInternal tools with few services
Single shared gatewayOne place for auth, limits, observabilityBottleneck for changes and capacity, shared blast radiusModerate service count, one platform team
Gateway per domain or per client typeSmaller blast radius, team autonomyMore fleets to run, policy drift riskLarge organisations, distinct client needs
Managed gateway serviceNo fleet to run, built-in featuresPer-request cost, hard vendor limits (AWS REST APIs: 29 s integration timeout by default, 10 MB payload; AWS quotas)Bursty or moderate traffic, small ops team

Pitfalls in production

Leadership lens

Check yourself

  1. Why can services trust an identity header injected by the gateway, and what must be true for that to be safe?

    Show answer

    The gateway validated the credential before injecting it. It is only safe if services are unreachable except through the gateway (or via mTLS-authenticated callers) and the gateway strips any client-supplied copy of that header.

  2. Why do many gateways enforce rate limits locally per node instead of always using a central store?

    Show answer

    A central check adds a network round trip and a dependency to every request; dividing the limit across nodes is accurate enough for most keys and much faster.

  3. When should aggregation move out of the gateway into a BFF?

    Show answer

    When it becomes specific to one client's screens, because then the gateway accumulates client logic and couples all clients to one team's release cycle.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →