Evals: Testing Non-Deterministic Systems

In one minute

• Evals are the test suite of an AI feature: a set of real inputs, a way to grade each output, and a number you track over time. Without them, every prompt or model change is a guess.
• Grade with the cheapest reliable method: code checks first (exact match, schema, contains a citation), then a model as judge with a written rubric, then humans for the cases that matter most and to calibrate the judge.
• Treat scores as measurements with error bars. A pass rate of 85 percent on 100 cases could plausibly be anywhere from about 77 to 91 percent; small eval sets cannot detect small changes.
• Calibrate model judges against human labels and track agreement. A judge you have not checked is an untested test.
• Run evals offline before every change and online on production traffic, and feed production failures back into the eval set.

How it works

What goes in an eval set

Start from real inputs: logged production requests (anonymised), support tickets, user-reported failures, and hand-written cases for important edge cases. For each case, store what a good output must satisfy: an expected value, required facts, forbidden content, or a rubric. Tag cases by category and importance so you can report per-slice results, and mark critical cases that must always pass. Fifty good cases beat five hundred random ones at the start; grow the set from every failure you find. Husain suggests starting near 30 examples and adding until new ones stop revealing new failure modes, then about 100 labelled examples per failure mode when validating a judge; below roughly 60, he notes, the confidence intervals are usually too wide to conclude anything (Husain).

Three kinds of graders

• Code-based: exact match for classification, JSON schema validation for extraction, "contains a citation that exists in the retrieved sources", length limits, regexes for forbidden content. Cheap, fast, deterministic: use them wherever possible.
• Model-based (LLM as judge): a model scores the output against a rubric ("Does the answer state the refund window? Does it avoid promising things not in the policy?"). Scales to open-ended outputs. Prefer binary or few-level judgments on specific criteria over vague 1 to 10 scores, ask the judge to explain before deciding, and use a capable model. Hamel Husain's field guide makes the same case for pass/fail with a written critique from a domain expert, and uses those critiques as few-shot examples in the judge prompt (Husain).
• Human: domain experts label a sample. Expensive and slow, but the ground truth for calibrating everything else.

Judges need calibration

Model judges have known biases (Zheng et al.): preferring longer answers, preferring their own model's style, being swayed by answer position when comparing two. The MT-Bench study (Zheng et al., 2023) found that strong judges such as GPT-4 reach over 80 percent agreement with human preferences, the same level humans reach with each other, and named position, verbosity and self-enhancement bias as the failure modes (Zheng et al.). The biases can be large: in studies Eugene Yan reviews, some judges preferred padded, longer answers more than 90 percent of the time, and swapping the order of the two answers and checking for a consistent verdict is the standard control for position bias (Yan). Measure your own judge: have humans label 100 to 200 outputs, run the judge on the same ones, and compute agreement and Cohen's kappa (agreement corrected for chance). Also report the judge's true positive rate and true negative rate separately, because raw agreement misleads when failures are rare: if 5 percent of cases fail, a judge that always says pass agrees 95 percent of the time and catches nothing (Husain). If kappa or the failure-catching rate is low, refine the rubric, split criteria, or use humans for that criterion. Keep some labelled data out of the judge prompt and test on it once, as you would with a model.

Statistics, briefly

A pass rate from n cases is an estimate. The Wilson score interval gives a sensible 95 percent range even for small n or rates near 0 or 1. On 100 cases, 85 percent passing has an interval of roughly 77 to 91 percent; on 1,000 cases, roughly 83 to 87 percent. Two versions whose intervals overlap heavily may not really differ. Practical rules: compare versions on the same cases (paired comparison), look at which cases changed rather than only the totals, run noisy cases several times, and size the set for the smallest difference you care about. Anthropic's research note recommends exactly this toolkit: report the standard error of the mean and a 95 percent interval of 1.96 standard errors, use paired differences because model scores on the same questions correlate (0.3 to 0.7 for frontier models, so pairing removes real noise), resample and average chain-of-thought answers per question, and run a power analysis to decide how many questions a given difference needs (Anthropic). One more trap from the same note: if questions share a unit, such as several questions about one document, cluster the standard errors on that unit, because naive errors can be over three times too small. The Wilson interval above is the better choice for small n and rates near 0 or 1 (Wikipedia).

Offline and online

Offline evals run in CI on every prompt, model or retrieval change and block releases that regress critical cases. Online evals score a sample of production traffic with the same graders, plus product signals: thumbs up or down, escalation to humans, task completion, follow-up questions that indicate confusion. Online failures become new offline cases, which is how the eval set stays representative. The need is real: Anthropic's 2025 postmortem found that offline evals alone missed three overlapping serving bugs, and the fix was continuous quality evaluation on production traffic (Anthropic).

The same shapes as any system

Evals are trust but verify, made quantitative. They play the role tests and monitoring play for deterministic software, with the extra step that the grader itself must be verified.

Down the stack

Diagram

flowchart LR
  P["Production logs and failures"] --> S["Eval set: cases, tags, critical flags"]
  H["Hand-written edge cases"] --> S
  S --> R["Run system version"]
  R --> G1["Code graders"]
  R --> G2["Model judge with rubric"]
  HL["Human labels sample"] --> CAL["Judge calibration: agreement, kappa"]
  CAL --> G2
  G1 --> D["Per-slice pass rates with intervals"]
  G2 --> D
  D --> GATE{"Critical pass and no regression?"}
  GATE -->|yes| SHIP["Ship"]
  GATE -->|no| FIX["Fix"]

Worked example

A legal-tech company's contract-summary feature has an eval of 120 contracts with lawyer-written key points. Version A scores 82 percent of key points found; version B, a new prompt, scores 86 percent. The team is about to ship B.

Before deciding, they compute intervals: 82 percent of 1,440 key points is roughly 80 to 84 percent, and 86 percent roughly 84 to 88 percent. Those intervals treat key points as independent, but they cluster by contract, so the honest interval is wider; computed over 120 contracts, it is perhaps twice as wide, and the improvement is plausible rather than certain. But the per-slice view shows B dropping from 95 to 81 percent on indemnity clauses, a critical category, while improving on boilerplate. They also check the judge: lawyers had labelled 150 outputs, and the model judge agrees with them on 88 percent with kappa 0.71, acceptable overall, but only 70 percent on indemnity clauses, where the judge misses subtle carve-outs.

Decisions: fix the prompt's handling of indemnity before shipping; mark indemnity cases as critical; for that category, use a stricter rubric that asks the judge to quote the clause and list each carve-out, raising agreement to 86 percent; and add 40 more indemnity contracts to the set. B ships a week later with 87 percent overall and 96 percent on indemnity.

In the wild

• Anthropic, 2025: three bugs that evals missed. Between August and September 2025 three infrastructure bugs degraded Claude's answers: requests misrouted to the wrong server type (0.8 percent of Sonnet 4 requests at first, 16 percent in the worst hour, and about 30 percent of Claude Code users hit at least once), a TPU misconfiguration and a compiler bug. Anthropic's evals did not catch them, in part because the model recovers well from isolated mistakes, and privacy controls limited what engineers could inspect. The fix included more sensitive evals and continuous quality evaluation on real production traffic (Anthropic postmortem).
• Google, 2024: AI Overviews and satire. After launch, AI Overviews answered some odd queries from satirical pages and forum jokes. Google said it had red-teamed and tested on a share of traffic, but real use at scale produced novel queries testing missed. It reported policy violations on fewer than 1 in 7 million overview queries and shipped more than a dozen fixes, including limits on satire and user-generated content (Google).
• Air Canada, 2024: the chatbot's wrong refund advice. A chatbot told a customer he could claim a bereavement fare retroactively within 90 days, contradicting the airline's own policy page. The British Columbia tribunal held the airline responsible for what its chatbot said and awarded CA$812 plus costs and interest. An eval case built from policy questions, graded against the policy text, is exactly the check that was missing (Moffatt v. Air Canada, 2024 BCCRT 149).

Trade-offs

Pitfalls in production

• Tiny eval sets that cannot detect real changes. Fix: grow the set, use intervals and paired comparisons.
• Uncalibrated model judges. Fix: measure agreement and kappa against human labels, per criterion.
• Only averages, no slices. Fix: per-category results and critical cases that must pass.
• Eval sets that stop reflecting production. Fix: add production failures continuously.
• Vague 1 to 10 scores. Fix: specific binary criteria with explanations.

Leadership lens

• Make an eval set and grading plan a launch requirement for AI features.
• Version eval sets, judge prompts and results; store per-case outcomes for diffing.
• Budget human labelling time every month for calibration and new cases.
• Report quality with intervals, not single numbers, to avoid overreacting to noise.

Check yourself

1. Why prefer code-based graders where possible?

   Show answer

   They are cheap, fast and deterministic, so they add no grader noise; model or human grading is only needed where code cannot judge.

2. What does Cohen's kappa add over raw agreement?

   Show answer

   It corrects for agreement expected by chance, which matters when one label is much more common than the other.

3. Two versions score 84 and 86 percent on 100 cases. Can you conclude the second is better?

   Show answer

   Not confidently; with 100 cases the intervals overlap heavily, so look at paired per-case differences or use more cases.

4. Why should production failures be added to the eval set?

   Show answer

   They show real failure modes the set missed, keeping it representative and preventing the same failure from returning.

Sources and further study

Primary

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023): over 80 percent agreement with human preferences, and the position, verbosity and self-enhancement biases.
• A statistical approach to model evaluations (Anthropic): standard errors, clustering, paired differences and power analysis for evals.

Systems docs

• A postmortem of three recent issues (Anthropic engineering): why offline evals missed real degradation and what monitoring was added.
• Creating a LLM-as-a-Judge That Drives Business Results (Hamel Husain): pass/fail judgments, critiques, data splits and reporting true positive and negative rates.
• Evaluating the Effectiveness of LLM-Evaluators (Eugene Yan): measured biases and when to use pairwise versus direct scoring.

Books

• Chip Huyen, AI Engineering (O'Reilly, 2025), Ch 3 "Evaluation Methodology" (including "AI as a Judge") and Ch 4 "Evaluate AI Systems" ("Design Your Evaluation Pipeline").
• Charity Majors, Liz Fong-Jones and George Miranda, Observability Engineering (O'Reilly, 2022), Ch 12 "Using Service-Level Objectives for Reliability", for turning a quality target into an SLO.

Other curricula

• Google SRE Workbook, Ch 2 "Implementing SLOs": the same measure-then-gate habit applied to reliability instead of answer quality.

Free resources

• A statistical approach to model evaluations (Anthropic): confidence intervals, paired differences and sample sizes for evals.
• Judging LLM-as-a-Judge (Zheng et al.): how well model judges agree with humans, and their biases.
• Creating a LLM-as-a-Judge That Drives Business Results (Hamel Husain): building and calibrating judges with domain experts.
• Evaluating the Effectiveness of LLM-Evaluators (Eugene Yan): a review of LLM-as-judge techniques and results.
• Binomial proportion confidence interval (Wikipedia): the Wilson interval and alternatives.
• Cohen's kappa (Wikipedia): agreement corrected for chance.