Engineering Atlas › Building AI Systems

LLM Gateways, Caching and Rate Limits

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why a gateway

Without one, every team writes its own client with its own retry logic, its own API keys, no shared rate limiting and no unified view of cost. A gateway centralises the cross-cutting concerns: the same reason companies use an API gateway for HTTP services. Open-source proxies such as LiteLLM and managed services such as Cloudflare AI Gateway offer much of this; larger companies often build their own on top of an existing proxy. Azure's architecture guidance lists the same duties (federated authentication, rate limiting, load balancing with circuit breaking, queueing low-priority work, caching, cost attribution) and also warns not to add a gateway if it would put your availability or latency targets at risk (Azure Architecture Center).

Rate limits at two levels

Providers enforce limits per organisation, typically requests per minute (RPM) and input and output tokens per minute (ITPM, OTPM). Anthropic's API uses a token bucket, so capacity refills continuously instead of resetting on a fixed minute, and a nominal 60 RPM can be enforced as one request per second, so short bursts fail. Exceeding a limit returns HTTP 429, usually with a retry-after header, and a sharp traffic ramp can trigger separate acceleration limits (Anthropic rate limits). One 429 should not be retried at all: when a monthly spend cap is reached, the response has no retry-after and every retry fails until the cap resets (same page). Your gateway adds its own limits beneath the provider's:

Token-based limits matter more than request counts, because one request with 100,000 tokens costs as much as a thousand small ones. Know how your provider counts, because it changes your gateway's accounting. For most Claude models only uncached input tokens count toward ITPM (cache reads do not), and OTPM is measured on tokens actually generated, with max_tokens not part of the calculation (Anthropic rate limits). A gateway that budgets its own tenants still needs a pessimistic estimate up front, because output length is unknown in advance, so it reserves input + max_tokens and corrects the count afterwards. Read the provider's anthropic-ratelimit-*-remaining response headers to see how close you are.

Three kinds of caching

Never share cached answers across users when the prompt includes user-specific data or permissions; include the tenant and permission scope in the cache key.

Budgets and attribution

Tag every call with team, feature, tenant and environment. The gateway records input, output and cached tokens and computes cost, so dashboards show spend by feature and tenant in near real time. Budgets with soft alerts (80 percent) and hard actions (downgrade model, queue, or block) prevent month-end surprises and runaway agents.

The same shapes as any system

The gateway is the overload and hot-spot shapes: shared capacity, fair allocation, backpressure, caching for hot requests. It is the same design as an API gateway or a database connection pooler, applied to a new expensive dependency.

Down the stack

Layer 1The contract

The gateway promises: every model call is authenticated, attributed and logged; no team or tenant exceeds its allocation; interactive traffic gets priority; transient provider errors are retried within a deadline and failed over where configured; cached responses are only served when safe by policy; and added latency stays small (for example under 10 ms at p99 excluding the model call).

Layer 2The mechanism

Rate limiting uses token buckets or windowed counters per key (team, tenant, model), usually in Redis with atomic scripts so many gateway replicas share state. Token reservation subtracts input + max_tokens up front and refunds the unused output after the response. Exact caching stores a hash key (for example SHA-256 of a canonical JSON request including tenant scope) to the response with a TTL. Semantic caching stores question embeddings in a vector index per scope, returning a cached answer only above a similarity threshold. Streaming responses complicate caching and accounting, so the gateway buffers token counts from the final usage event, and it must allow long-lived connections without premature timeouts or buffering (Azure Architecture Center).

Layer 3The machine

The gateway sits on every request, so its own cost must be tiny relative to a model call that takes seconds: a Redis round trip within a data centre is about 0.2 to 1 ms, a SHA-256 hash of a 50 KB prompt takes microseconds, and an embedding call for semantic caching costs tens of milliseconds, which is why semantic lookups are only worth it when they avoid multi-second generations. Streaming responses are passed through as server-sent events with small buffers, so the gateway holds many long-lived connections; an event-driven proxy (such as Envoy or one built on async I/O) handles tens of thousands of concurrent streams per node.

Why it's built this way

  1. Many teams share limited, expensive model capacity.
  2. Without central control, limits, retries and costs are inconsistent and unfair.
  3. A gateway enforces allocation, priority and budgets in one place.
  4. Repeated work is common, so caching at the right level saves money and time.
  5. The gateway is on every request, so it must be fast and highly available itself.

Diagram

flowchart LR
  A["Feature A (chat)"] --> GW["LLM gateway"]
  B["Feature B (batch)"] --> GW
  C["Feature C (agent)"] --> GW
  GW --> AU["Auth and attribution"]
  AU --> RL{"Tenant and team limits"}
  RL -->|over| Q["429 or queue (low priority first)"]
  RL -->|ok| CA{"Cache hit (scoped key)?"}
  CA -->|yes| RESP["Cached response"]
  CA -->|no| RT["Route: model, retries, fallback"]
  RT --> P1["Provider 1"]
  RT --> P2["Provider 2"]
  RT --> LOG["Usage, cost, traces"]

Worked example

A company with 6 AI features shares one provider organisation with a limit of 2 million input tokens per minute. Every Monday at 09:00 a reporting job submits 40,000 summarisation requests, hits the limit within minutes, and customer-facing chat starts failing with 429 errors for half an hour.

They introduce a gateway. Limits: chat gets a guaranteed 1.2 million tokens per minute; the reporting job is tagged low priority and capped at 400,000 tokens per minute, and moved to the provider's batch API overnight. Per-tenant limits stop any single customer from using more than 5 percent of chat capacity. An exact cache for the classification feature (identical ticket texts are common because of templates) serves 18 percent of its requests from cache. Prompt caching is enabled for chat's 3,000-token system prompt. Monthly cost falls by 31 percent, and the Monday incident does not recur. A budget alert later catches an agent bug that loops on a tool call, capping its spend at 200 dollars instead of the 9,000 it would have reached over the weekend.

In the wild

Trade-offs

ChoiceStrengthsWeaknessesChoose when
Direct provider calls per teamSimple startNo shared limits, cost visibility or policyOne team, one feature
Central gatewayFairness, budgets, observability, failoverNew critical-path serviceSeveral teams or features
Exact response cacheSafe, simpleLow hit rate for varied inputDeterministic tasks, repeated inputs
Semantic cacheHigh hit rate on FAQ-like trafficWrong-answer risk, extra embedding costLow-risk, non-personalised answers
Token reservation by max_tokensPrevents overshootConservative, needs refundsToken-based provider limits

Pitfalls in production

Leadership lens

Check yourself

  1. Why rate limit by tokens rather than requests?

    Show answer

    Requests vary enormously in size, and providers limit and charge by tokens, so token budgets reflect real capacity and cost.

  2. Why reserve max_tokens before a call?

    Show answer

    Output length is unknown in advance; reserving the maximum prevents exceeding the limit, and the unused part is refunded afterwards.

  3. When is semantic caching dangerous?

    Show answer

    When small differences in questions change the correct answer, or answers depend on the user, so a similar question can get a wrong or another user's answer.

  4. What must be in an exact cache key?

    Show answer

    Everything that affects the output: model, parameters, full prompt including system and tools, and the tenant or permission scope.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →