Engineering Atlas › Building AI Systems

How LLM Applications Work: Tokens, Context and Latency

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Tokens, not words

Models do not see characters or words; they see tokens from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries, produced by an algorithm such as byte-pair encoding, which was adapted from data compression to split rare words into subword units (Sennrich et al., 2015). Common English words are usually one token, rarer words split into several, and code, numbers and non-English text often use more tokens per character. A rough rule for English prose is about 4 characters or 0.75 words per token, but always measure with the provider's token counter for your real data: a JSON-heavy prompt can cost twice what the word count suggests.

The request

An API call sends a list of messages (system instructions, user turns, assistant turns, tool results) plus parameters: the model, the maximum output tokens, sampling settings such as temperature, and optionally tool definitions. The provider turns this into one long token sequence, runs the model, and returns generated tokens and usage counts. Nothing is remembered afterwards: the next call must resend whatever context it needs. If you pass tools, the provider also adds a hidden tool-use system prompt: Anthropic lists roughly 286 to 675 extra input tokens depending on the model, on top of your own schemas (Anthropic tool use docs).

Prefill and decode

The model processes all input tokens together in one parallel pass, called prefill. That is fast per token, since GPUs are good at large matrix multiplications, but it still grows with input size. Then it generates output one token at a time, called decode, where each step depends on the previous one. Decode speed is often tens to a few hundred tokens per second per request. So time to first token is dominated by prefill plus queueing and network, and total time by output length. A 2,000-token answer at 80 tokens per second takes 25 seconds no matter how fast prefill was.

Streaming

Streaming returns tokens as they are generated, using server-sent events. One catch is that an error can arrive after the HTTP 200 has already been sent, so streaming clients need a separate path for mid-stream errors (Anthropic errors). The same page recommends streaming or batch for requests expected to run past 10 minutes, since idle connections may be dropped. It does not make the total faster, but users perceive a response that starts in under a second as fast. Any UI that shows model output to a person should stream; background jobs usually do not need to.

The context window

Every model has a maximum context, the sum of input and output tokens it can handle in one call, from tens of thousands to a million or more depending on the model. Plan the budget explicitly: system prompt and tool definitions (often thousands of tokens), conversation history, retrieved documents, and room for the answer. Long contexts cost more, take longer to prefill and can reduce answer quality, as the context engineering lesson explains, so bigger is not automatically better. In one study, models answered best when the relevant passage sat at the start or end of the input and noticeably worse when it was buried in the middle (Liu et al., 2023).

Prompt caching

Providers can cache the processed prefix of a prompt. If many requests start with the same system prompt, tools and documents, cached input tokens are billed at a fraction of the normal price and prefill is faster. Order your prompt with stable content first and per-request content last to benefit.

The details differ by provider and change often. On Anthropic's API a cache read costs 0.1x the base input price for most models (0.05x on the newest Sonnet and Opus), a 5-minute cache write costs 1.25x and a 1-hour write 2x. The default lifetime is 5 minutes and each hit refreshes it. The prefix is built in the order tools, system, messages, and a change at one level invalidates everything after it. Prompts below a model-specific minimum (512 to 4,096 tokens) are simply not cached, with no error, so check cache_read_input_tokens in the usage block to confirm hits (Anthropic prompt caching). OpenAI caches automatically for prefixes of 1,024 tokens or more, and reuse needs the whole rendered prefix to match (OpenAI prompt caching). Cached reads also do not count toward Anthropic's input-tokens-per-minute rate limit on most models, so caching raises throughput as well as cutting cost (Anthropic rate limits).

The same shapes as any system

An LLM call is a slow, expensive, variable-latency remote dependency. Everything you know applies: timeouts, retries with idempotency, caching, rate limits, fallbacks and budgets. The model is just a new kind of backend.

Down the stack

Layer 1The contract

A model API promises: given messages and parameters, return a completion of at most max_tokens output tokens, with usage counts and a stop reason (natural end, a stop sequence, the token limit or a tool call). It does not promise determinism, a fixed latency or that the content is correct.

Layer 2The mechanism

Inside, a transformer turns each token into a vector, passes the sequence through dozens of layers of attention and feed-forward networks, and outputs a probability for every token in the vocabulary. A sampler picks the next token: greedy picks the most likely; temperature reshapes the distribution; top-p limits choices to the most likely tokens whose probabilities sum to p. Do not assume you can set these: Anthropic's API now returns a 400 for non-default temperature, top_p or top_k on Claude 4.7 and later models (Anthropic deprecations). During decode, a key-value cache stores each layer's attention keys and values for previous tokens, so each new token needs only incremental work. This cache is the main memory consumer in serving: it grows and shrinks per request, and naive allocation wasted so much GPU memory to fragmentation that the vLLM authors' paged layout raised throughput 2 to 4 times at similar latency (Kwon et al., 2023). Serving systems batch many users' requests together on each GPU to keep it busy, which is why your latency depends partly on other people's traffic.

Layer 3The machine

Decode is limited by memory bandwidth more than arithmetic: for every generated token, the GPU must read the model's weights from high-bandwidth memory. A model with tens of billions of parameters stored in 16-bit precision is tens to over a hundred gigabytes, and a GPU with a few terabytes per second of bandwidth can read it only so many times per second; batching amortises each read across many requests. The key-value cache also grows with context length, consuming GPU memory per request, which is one reason long contexts cost more and reduce how many requests fit on a GPU at once.

Why it's built this way

  1. Text is split into tokens so a fixed vocabulary can represent any input.
  2. Input tokens can be processed in parallel, output tokens cannot, so prefill and decode behave differently.
  3. GPU memory bandwidth and capacity limit decode speed and context length, so providers price by tokens.
  4. The model has no memory between calls, so applications manage context explicitly.
  5. Caching repeated prefixes saves the most expensive repeated work.

Diagram

sequenceDiagram
  participant App
  participant API as Model API
  participant GPU
  App->>API: messages, tools, max_tokens
  API->>GPU: queue and batch
  GPU->>GPU: prefill all input tokens in parallel
  GPU-->>App: first token (time to first token)
  loop decode, one token per step
    GPU-->>App: next token (streamed)
  end
  API-->>App: stop reason and usage

Worked example

A team builds a summarise-this-ticket button. Each request sends a 1,500-token system prompt and tools, a ticket thread averaging 3,000 tokens, and asks for a 300-token summary. They see time to first token of about 1.2 seconds and total latency of about 5 seconds, with decode at about 80 tokens per second.

Cost at an illustrative price of 3 dollars per million input tokens and 15 per million output tokens: input 4,500 tokens is 0.0135 dollars, output 300 tokens is 0.0045, so about 1.8 cents per summary. At 200,000 summaries a month, that is about 3,600 dollars.

Three changes. Put the static system prompt and tool definitions first and enable prompt caching: cached reads cost a fraction of the base input price, so the 1,500 stable tokens drop to around a tenth of their cost on a read (the first request that writes the cache pays a 1.25x premium on Anthropic, and some models only cache prompts above 2,048 or 4,096 tokens, so check the minimum for yours). Trim quoted email signatures and repeated history from tickets, cutting average input from 3,000 to 1,800 tokens. Ask for a 120-word summary instead of open-ended text, cutting output to about 170 tokens. Total cost falls by roughly half, time to first token improves because prefill is shorter, and total latency drops below 3 seconds because there are fewer output tokens. Streaming makes the remaining wait feel instant.

In the wild

Trade-offs

ChoiceStrengthsWeaknessesChoose when
Long context, include everythingSimple, nothing missedCost, latency, distractionSmall inputs, prototypes
Selective context (retrieval, trimming)Cheaper, faster, focusedCan miss informationProduction features
Streaming responsesFast perceived latencyMore complex client handlingAnything a human waits for
Prompt cachingLower cost and prefill timeNeeds a stable prefixRepeated system prompts, tools, documents
Short, structured outputsFaster and cheaperLess nuanceSummaries, extraction, classification

Pitfalls in production

Leadership lens

Check yourself

  1. Why does a long answer take much longer than a long prompt?

    Show answer

    Input tokens are processed in parallel during prefill, but output tokens are generated one at a time during decode, so total time grows with output length.

  2. What does streaming improve, and what does it not?

    Show answer

    It improves perceived latency by showing the first tokens quickly; it does not reduce total generation time or cost.

  3. Why should static content come first in a prompt?

    Show answer

    Prompt caching reuses the processed prefix, so a stable beginning can be cached and reused across requests, cutting cost and prefill time.

  4. What should your code do if the stop reason says the output hit the token limit?

    Show answer

    Treat the output as incomplete: retry with a higher limit, ask for a shorter answer, or handle it explicitly rather than using a truncated result.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →