Engineering Atlas › Building AI Systems

Choosing and Calling Models: Cost, Latency and Quality

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The three-way trade-off

Every model sits somewhere on quality, cost and latency. Within one provider's family, larger models score higher on hard reasoning and writing, cost several times more per token, and generate fewer tokens per second. The right choice depends on the task: a support router that labels tickets into 12 categories needs consistency and speed, while a code review assistant needs reasoning. Using the biggest model everywhere is the most common and expensive mistake; using the smallest everywhere is the second most common.

Choosing with evals

Public benchmarks and arenas tell you roughly how models compare in general. Your task is not general. Build an eval set of 100 to 500 real inputs with checks (see the evals lesson), run each candidate, and record pass rate, cost per request and latency percentiles. Often a smaller model reaches 95 percent of the larger one's quality at a fifth of the cost; sometimes the gap is large on exactly the cases that matter. Rerun when providers release new models or change versions.

Routing and cascades

Calling models reliably

Model APIs fail in familiar ways: rate limits (HTTP 429), overloads (Anthropic returns 529), server errors (500) and timeouts (504) (Anthropic errors). Treat them like any dependency: set a timeout that matches the expected generation time, retry transient errors with exponential backoff and jitter, honour retry-after headers, and never retry indefinitely. Two 429 traps: a monthly spend-cap 429 carries no retry-after and keeps failing until the next month, so retrying it is pointless, and a spend limit you set yourself returns a 400 instead (Anthropic rate limits). Anthropic's official SDKs already retry twice by default with backoff and honour retry-after, so stacking your own retry loop on top multiplies attempts; the Azure Architecture Center warns against nested retry layers for the same reason (Retry pattern). A sharp traffic ramp can also trigger 429s from acceleration limits, so ramp up gradually. Keep a fallback: a second model, a second region or provider, or a degraded non-AI path ("we could not generate a summary; here is the raw ticket"). Make side-effecting tool calls idempotent, because a retried request can repeat them.

Batch and asynchronous work

Many AI workloads do not need an answer in seconds: nightly classification, document enrichment, eval runs, backfills. Batch APIs accept a set of requests and return results within a window of up to 24 hours at 50% of the standard price. Anthropic says most batches finish within an hour, caps a batch at 100,000 requests or 256 MB, and keeps results for 29 days (Anthropic batch docs). OpenAI's equivalent has a 24-hour window, up to 50,000 requests and 200 MB per file, and its own separate rate-limit pool (OpenAI Batch API). Anthropic's batches have their own queue limits too (for example 200,000 queued batch requests on the Start tier), so they are not unlimited. Moving even a quarter of traffic to batch can noticeably lower the bill.

The same shapes as any system

Model selection is a classic capacity and cost trade-off, and the cascade is the same pattern as a cache in front of a database: handle the common case cheaply and fall through to the expensive path only when needed.

Down the stack

Layer 1The contract

For each AI feature, the team commits to a quality target on its eval set, a cost per request and a latency target (often p95 time to first token and total time). The model choice, routing and retry policy are the mechanism to meet that contract, and they change whenever a cheaper or better option meets it.

Layer 2The mechanism

A model client library usually provides: per-feature configuration (model, max tokens, temperature, timeout), retry policy with backoff, fallback chains, token and cost accounting, and routing hooks. Rate limits are applied by the provider per organisation in requests per minute and tokens per minute; the client should track remaining quota from response headers (Anthropic sends anthropic-ratelimit-*-remaining and -reset values) and slow down before hitting the limit, rather than after. Limits use a token bucket that refills continuously, so a short burst can fail even when your per-minute average is fine, and on most Anthropic models cache reads do not count toward the input-token limit, which makes prompt caching a rate-limit tool as well as a cost tool (Anthropic rate limits). Cascades need a cheap, reliable escalation signal: a schema validation failure, a self-reported confidence field, a check against retrieved sources, or a classifier.

Layer 3The machine

The price difference between models reflects hardware time. Larger models have more parameters to read from GPU memory for every generated token and need more GPUs per replica, so each output token occupies more expensive hardware for longer. Batch pricing is lower because the provider gains scheduling freedom: with a 24-hour window it can fill spare capacity and form larger GPU batches. The providers document the discount and the window, not the scheduling internals, so treat that explanation as a reasonable inference. Latency differences come from the same place: more parameters per token means more memory traffic per decode step.

Why it's built this way

  1. Model capability, cost and speed trade against each other because of hardware limits.
  2. Tasks differ in how much capability they need.
  3. So each feature should use the cheapest model that meets its measured quality bar.
  4. Mixed difficulty within a feature makes routing and cascades worthwhile.
  5. Model APIs are remote dependencies with limits, so standard resilience patterns apply.

Diagram

flowchart TD
  R["Request"] --> C{"Route by task"}
  C -->|classify, extract| S["Small model"]
  C -->|reason, write| L["Large model"]
  S --> V{"Valid and confident?"}
  V -->|yes| OUT["Answer"]
  V -->|no| L
  L --> E{"Error?"}
  E -->|429 or 5xx| B["Backoff and retry, then fallback provider"]
  E -->|ok| OUT
  B --> OUT

Worked example

A company tags incoming support tickets with one of 15 categories and a priority, about 400,000 tickets a month, and also drafts replies for agents. Today both use the largest model.

They build a 300-ticket eval for tagging and a 150-ticket eval for drafting, graded by support leads. For tagging, the small model scores 96 percent against the large model's 97 percent at about one tenth of the cost and a third of the latency. For drafting, the small model scores 71 percent against 89 percent; the mid-size model scores 87 percent at about a third of the large model's cost.

New design: tagging on the small model, with escalation to the mid-size model when the small model's output fails schema validation or picks "other" (about 6 percent of tickets). Drafting on the mid-size model, with the large model available for escalated VIP tickets. Overnight re-tagging of old tickets moves to the batch API. The monthly bill falls by about 70 percent, p95 tagging latency drops from 2.4 to 0.8 seconds, and measured quality stays within one point on both evals.

In the wild

Trade-offs

ApproachStrengthsWeaknessesChoose when
Largest model everywhereBest quality, simpleHighest cost and latencyPrototypes, low volume
Per-feature model chosen by evalRight-sized costEval upkeepMost production features
Cascade small to largeBig savings when escalation is rareExtra latency on escalated requestsMixed-difficulty traffic
Multiple providers with fallbackResilience to outagesMore integration and evalsCritical features
Batch APIMuch cheaper, no rate-limit pressureHours of latencyOffline and bulk work

Pitfalls in production

Leadership lens

Check yourself

  1. Why not use the most capable model for every feature?

    Show answer

    It costs more and is slower, and many tasks such as classification or extraction reach nearly the same quality on a smaller model, so you pay for capability you do not need.

  2. When does a cascade save money?

    Show answer

    When the small model handles most requests well and escalation is rare, so the small model's cost plus escalation rate times the large model's cost is below the large model's cost alone.

  3. How should a client respond to HTTP 429 from a model API?

    Show answer

    Back off exponentially with jitter, respect any retry-after header, and slow down overall request rate rather than retrying immediately.

  4. Why is batch processing cheaper?

    Show answer

    The provider gets up to 24 hours of scheduling freedom, which lets it improve GPU utilisation, so it charges 50% less in exchange for slower turnaround.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →