Debugging LLM-Powered Systems
Open in the interactive app: practice, quiz, mock interview →In one minute
- LLM features fail in new ways: the same input can give different outputs, failures are often "wrong but fluent" rather than errors, and a change to a prompt, a model version or the retrieved documents can shift behaviour everywhere at once.
- The fundamentals still apply: reproduce, isolate, form hypotheses, test them. You just need different evidence: full traces of prompts, retrieved context, tool calls and outputs.
- Split the pipeline and find the failing stage: was the right context retrieved, was the prompt assembled correctly, did the model reason wrongly, did a tool fail, did post-processing break the output?
- Evals are your test suite: a set of real cases with checks, run on every prompt, model or retrieval change, compared against the previous version case by case.
- Treat prompts, model versions and retrieval settings as code: versioned, reviewed, rolled out gradually and rolled back when evals or production metrics drop.
How it works
What is different
Classic bugs are deterministic: the same input gives the same wrong output, so a failing test reproduces it. LLM systems add three complications. Nondeterminism: sampling, and even batching on the provider's side, can change outputs between runs. Silent failure: a wrong answer looks exactly like a right one, so there is no exception or 500 to alert on. Global coupling: one prompt or model change affects every request, so a fix for one case can quietly break twenty others.
Trace everything
You cannot debug what you cannot see. For each request, record the model and version, sampling parameters, the full assembled prompt (system prompt, instructions, retrieved documents, conversation), every tool call with arguments and results, the raw output, the post-processed output, token counts and latency. Store traces with a request ID, so a user complaint ("it told me the wrong refund policy") becomes a trace you can open and read. Redact personal data before storage, as you would for any log.
Find the failing stage
A typical retrieval-augmented pipeline has stages: query rewriting, retrieval, prompt assembly, generation, tool use and post-processing. Localise the failure like any other bug, stage by stage:
- Retrieval: were the right documents in the context? If the correct refund policy was never retrieved, no prompt change will fix it. Check the query, the top results and their scores.
- Assembly: was the context actually included, in the right order, without truncation? Token limits silently cut the end of long prompts.
- Generation: given good context, did the model answer wrongly? Try the same prompt several times to see if it is consistent or flaky.
- Tools: did a tool return an error the model ignored, or did the model call it with wrong arguments?
- Post-processing: did a parser drop fields, or did a guardrail filter strip the answer?
This is bisection applied to a pipeline: test the midpoint, keep the half that is broken.
Evals as regression tests
An eval set is a list of real inputs, each with a check: an exact expected value, a rule ("must cite a policy document", "must not mention competitor names"), or a grader (a human or a model judging against a rubric). Build it from production failures and from the cases that matter most. Run it on every change and compare case by case with the last version, not just the average: a pass rate that goes from 91 to 92 percent can hide five critical cases that started failing. Because outputs vary, run important cases several times and track their pass rates.
Two habits make evals useful. Start from real failures and keep looking at traces: Hamel Husain describes cheap unit-test assertions first, then human review of logged traces and a model judge calibrated against human labels (Husain). For agents, run several trials per task. Anthropic's example: a 75% per-trial success rate gives only about 42% odds of passing three trials in a row, so measure consistency, not just best-of-k, when users need reliability, and read the transcripts to check that graders are not rejecting valid answers (Anthropic).
Change control
Version prompts in the repository. Pin model versions rather than floating aliases, so a provider update does not change behaviour under you unexpectedly. Roll changes out behind flags to a small share of traffic, compare production metrics (thumbs-down rate, escalations to humans, task completion), and keep the old version ready to restore. Pinning is not enough on its own: in 2025 Anthropic had three infrastructure bugs (misrouted requests, corrupted token output, a compiler bug in approximate top-k) that degraded answers for weeks while its standard evals stayed quiet, partly because the model often recovers from an isolated mistake. Its fixes were more sensitive evals and continuous quality evals on production traffic (Anthropic postmortem). Watch the provider's behaviour too, not just your own diffs.
Down the stack
Layer 1The contract
An LLM feature's contract is statistical: for its eval set, at least X percent of cases pass, critical cases always pass, latency and cost stay within budgets, and production quality metrics stay above thresholds. A change ships only if it keeps that contract, judged by evals run before rollout and metrics watched during it.
Layer 2The mechanism
Tracing uses the same machinery as distributed tracing: each request gets a trace ID, and each stage (retrieval, model call, tool call) is a span with attributes such as model name, input and output token counts and finish reason. OpenTelemetry defines semantic conventions for generative AI spans, now maintained in their own repository (OpenTelemetry GenAI conventions), so traces can flow into standard backends; check which version of the conventions your instrumentation library emits. Evals are a harness that loads cases, runs the pipeline at a fixed configuration, applies checks, and writes results keyed by case ID, so two runs can be diffed. A model's finish reason is a useful field: max_tokens (or length) means the output was cut off, which explains many "the JSON is invalid" bugs.
Layer 3The machine
Nondeterminism has concrete causes. With temperature above zero, the model samples the next token from a probability distribution, so different runs pick different tokens and diverge. Even at temperature zero, outputs can differ, because floating-point addition is not associative and inference servers batch requests together: different batch sizes change the order of operations in matrix multiplications on the GPU, which changes the last bits of the logits, which occasionally flips which token is most likely. That is why "it worked when I tried it" is weak evidence, and why evals run each important case several times. The batching part is the real driver: Thinking Machines ran one prompt 1,000 times at temperature 0 on a 235B model and got 80 distinct outputs, first diverging at token 103. They traced it to kernels that are not batch-invariant, so an answer depends on how busy the server was; with batch-invariant kernels all 1,000 matched, at a speed cost (Thinking Machines, 2025). The popular "concurrent threads plus floating point" story is incomplete, since the forward pass is deterministic for a fixed batch.
Why it's built this way
- Outputs are sampled and can vary, so single runs are anecdotes, not evidence.
- Failures are silent, so you need explicit checks rather than error rates.
- Pipelines have many stages, so traces of each stage are needed to localise faults.
- Changes are global, so per-case comparison against the previous version catches hidden regressions.
- Therefore: trace, evaluate case by case, version and roll out gradually.
Diagram
flowchart LR Q["User question"] --> RW["Query rewrite"] RW --> RT["Retrieve docs"] RT --> AS["Assemble prompt"] AS --> G["Model generates"] G --> TL["Tool calls"] TL --> G G --> PP["Post-process"] PP --> A["Answer"] RT -.->|"right docs?"| CHK["Check each stage in the trace"] AS -.->|"truncated?"| CHK G -.->|"consistent?"| CHK
Worked example
A support assistant answers 40,000 questions a day. After a release, thumbs-down feedback rises from 6 to 9 percent, mostly on refund questions. The release changed two things: a new prompt section about tone, and a switch to a cheaper embedding model for retrieval.
The engineer opens 30 thumbs-down traces. In 22 of them, the retrieved documents are about shipping, not refunds: the refund policy document is ranked 7th, and only the top 5 go into the prompt. In the other 8, the right document was retrieved but the answer was still wrong; rerunning those prompts 10 times gives the right answer 9 times out of 10, so those are mostly ordinary model variance that existed before the release.
Hypothesis: the new embedding model ranks the refund policy lower. Test: run the 400-case eval set with the old and new embedding models while keeping the new prompt. Pass rate on refund cases: 94 percent with the old embeddings, 71 percent with the new ones; other categories barely change. The team rolls back the embedding change, adds 50 refund questions from production to the eval set, and later reintroduces the cheaper model with a reranker, which recovers 95 percent on the refund cases. The tone change was innocent; without traces and per-category evals, it would likely have been blamed first, because it was the change people remembered.
In the wild
- Anthropic, 2025: three bugs quietly degraded Claude's answers. A routing error sent some Sonnet 4 requests to servers built for a longer context (16% of requests at the worst hour, and about 30% of Claude Code users hit it at least once); a TPU misconfiguration produced stray tokens such as Thai text in English replies; a latent compiler bug could drop the most likely token. Evals missed them and privacy controls limited how much engineers could inspect, so detection was slow (Anthropic).
- Air Canada, 2024: a chatbot invented a refund rule and the airline was held to it. The website chatbot told a grieving customer he could claim a bereavement fare retroactively. Staff later said that was not allowed. A British Columbia tribunal rejected the airline's argument that it was not responsible for what its chatbot said and awarded $650.88 in damages (summary of Moffatt v. Air Canada, 2024 BCCRT 149). A trace of the answer and its retrieved sources is how you would have caught the gap before a customer did.
Trade-offs
| Technique | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Full prompt and output tracing | Debuggable, auditable | Storage cost, privacy handling | Always, with redaction and retention limits |
| Exact-match evals | Cheap, deterministic | Only fits structured outputs | Classification, extraction, JSON |
| Rule-based checks | Clear, fast | Miss subtle quality issues | Must-include and must-not-include rules |
| Model-graded evals | Scales to open-ended answers | Grader can be wrong or biased | Large sets, calibrated against humans |
| Human review | Highest judgment | Slow, expensive | Critical cases, calibrating graders |
Pitfalls in production
- Debugging from a single run. Fix: repeat the case several times and look at the distribution.
- Fixing retrieval failures with prompt edits. Fix: check the trace first; if the right context is missing, fix retrieval.
- Floating model aliases that change under you. Fix: pin versions, and upgrade deliberately with evals.
- Comparing only average eval scores. Fix: per-case and per-category diffs, with critical cases that must always pass.
- Silent truncation from token limits. Fix: alert on finish reason
max_tokensand on prompt sizes near the limit.
Leadership lens
- Make tracing and an eval set a launch requirement for any LLM feature, the same as logs and tests for a service.
- Grow the eval set from production failures every week; it is the team's memory of what went wrong.
- Review prompt and model changes like code, and roll them out behind flags with a fast rollback.
- Watch quality metrics in production, not just latency and errors: feedback, escalations, task completion.
Check yourself
Why might a prompt change not fix a wrong answer about refund policy?
Show answer
If retrieval never put the policy document into the context, the model cannot use it; the failure is in retrieval, not generation.
Why can outputs differ between runs even at temperature zero?
Show answer
Batching and floating-point non-associativity on the inference servers change computations slightly, which can flip the most likely token.
Your eval pass rate rose from 91 to 92 percent after a change. Why is that not enough to ship?
Show answer
Averages hide per-case regressions; some important cases may have started failing while others improved, so compare case by case.
What does a finish reason of
max_tokenstell you?Show answer
The output was cut off at the token limit, which explains truncated answers and invalid structured output.
Sources and further study
Primary
- Thinking Machines, Defeating Nondeterminism in LLM Inference (2025): why temperature-0 outputs still vary, with measurements and a fix.
- Anthropic, A postmortem of three recent issues: a provider's own account of silent quality regressions and why evals missed them.
Systems docs
- OpenTelemetry GenAI semantic conventions: span, metric and event names for model and tool calls.
- Summary of Moffatt v. Air Canada: the tribunal's reasoning on a chatbot's wrong answer.
Books
- Site Reliability Engineering, Google, chapter 12 "Effective Troubleshooting": hypothesis-driven debugging that maps onto stage-by-stage localisation.
- Observability Engineering, Majors, Fong-Jones and Miranda, chapter 8 "Analyzing Events to Achieve Observability": debugging from first principles using rich per-request events.
- Observability Engineering, chapter 5 "Structured Events Are the Building Blocks of Observability": why wide, per-request records beat aggregate metrics for finding odd failures.
Free resources
- Your AI Product Needs Evals (Hamel Husain): building evals from real failures, levels of testing.
- A Field Guide to Rapidly Improving AI Products (Hamel Husain): error analysis and looking at your data.
- Demystifying evals for AI agents (Anthropic): how to evaluate multi-step agent behaviour.
- Patterns for Building LLM-based Systems (Eugene Yan): evals, retrieval, guardrails and caching.
- Define success criteria and build evaluations (Anthropic docs): writing measurable criteria and test cases.