AI Evals: How to Know Your Output Is Actually Right

You can tell a demo works. You watched it. The input was one you chose, the conditions were good, and you were paying attention. What a demo cannot tell you is how the same feature behaves on the next thousand inputs, sent by people who did not build it, who phrase things badly, who paste in a wall of text, who ask something adjacent to what you designed for. That gap is where AI features quietly rot. Not with an outage, not with a stack trace, just with a slowly rising share of answers that are subtly wrong and that nobody is counting. Traditional software fails loudly. A null pointer throws. A migration breaks the build. An AI feature fails politely: it returns a fluent, well-formatted, confidently worded answer that happens to be wrong, and it returns it with the same tone it uses when it is right. There is no exception to catch. Unless you are measuring output quality on purpose, the only signal you get is the slow drift of users who stop trusting the feature and never tell you why. Husain's advice on this is blunt and it is the single most useful sentence in the field: "You must remove all friction from the process of looking at data." Elsewhere he puts it as "The most important part that people should learn is how to look at data. It tends to be the one place where 95% of problems are found and resolved and clarity is brought to the eval process." Evals are how you replace the vibe with a number. And if your feature chains several model calls together, the number matters more than you think, because per-step reliability compounds rather than averages. The arithmetic of that is worked through in full in our agent reliability calculator guide.

People use the word "eval" to mean four quite different things. Conflating them is why teams end up with a lot of evaluation activity and very little signal. | Kind | What it actually checks | Cost and speed | Reach for it when | |---|---|---|---| | Assertions | Deterministic, code-checkable properties: valid JSON, required field present, under a length cap, banned phrase absent, tool called before the write endpoint | Milliseconds, effectively free | Always. This is the layer most teams badly under-invest in because it feels too simple to count | | LLM-as-judge | Subjective quality against a rubric: tone, faithfulness to a source document, whether the answer addressed the question | Seconds, a model call per case | You need a repeatable score on subjective quality and you already have a reference answer to ground the judge against | | Human review | Everything the other two miss, including failure modes you have not thought to name yet | Slow, expensive, does not scale | Early on, and periodically forever, to calibrate the automated graders | | Online and A-B evals | A real outcome: resolution rate, escalation rate, conversion, thumbs-down rate | Slowest and most expensive signal | The product is mature enough that you need to measure value rather than a proxy for it | The mistake worth avoiding is reaching for the judge first because it feels like the sophisticated option. A large share of real production failures are things a five-line assertion would have caught. Husain's client Rechat maintains hundreds of unit tests, added continuously as new failures appear in production, and that suite runs in seconds and costs nothing. The judge layer earns its place on the questions assertions genuinely cannot answer. Human review never goes away entirely, because it is the only layer that can tell you your rubric is measuring the wrong thing. Anthropic's own write-up puts it plainly: "Human evaluation catches what automation misses."

This is the technique most teams adopt and the one most often oversold, so it is worth stating both halves of the evidence. The foundational result is Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023. Their finding: "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans." The evidence base was substantial: 3,000 controlled expert votes from 58 experts on MT-Bench, plus roughly 3,000 crowdsourced votes from Chatbot Arena, against a corpus of 30,000 conversations. That is a real result and it is why the technique works at all. But two things get dropped from every summary of it. First, the 80% figure is a ceiling reached after mitigating known biases, not the number you get from a naive setup. Second, it measures agreement on preference between two answers, not correctness. The same paper documents the failure modes in the same breath, and later work has measured them properly. Position bias. The judge favours whichever answer it sees first. Shi et al., "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" (IJCNLP-AACL), is the strongest measurement of this: 15 LLM judges across MT-Bench and DevBench, 22 tasks, roughly 40 solution-generating models, producing over 150,000 evaluation instances. They found position bias is "not due to random chance", varies significantly across judges and tasks, and is only weakly influenced by the length of prompt components. The finding that matters practically is what does drive it: position bias is strongly affected by the quality gap between the two solutions. When the answers are close, the judge falls back on position. When one is clearly better, position matters less. Your judge is least trustworthy exactly where you need it most, on the close calls. Verbosity bias. Longer answers score better regardless of whether they are better. If you are comparing a terse prompt against a discursive one, some of your measured difference is length. Self-preference. Wataoka, Takahashi and colleagues, "Self-Preference Bias in LLM-as-a-Judge" (2024), defined a bias metric from 0 (neutral) to 1 (strong self-preference) and measured eight models. GPT-4 scored 0.749, by a wide margin the highest of the eight. GPT-3.5-turbo scored 0.191. Their proposed mechanism is worth knowing because it generalises: the bias appears to track perplexity. Models rate text they find familiar more highly than humans do, whether or not they generated it. Self-preference is a special case of familiarity-preference, which means it does not disappear just because you switch judge vendors. The finding that should change how you build. Krumdick, Lovering, Reddy, Ebner and Tanner, "No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding" (2025), human-annotated correctness labels for 1,200 LLM responses across a finance and business benchmark and a hard subset of MT-Bench. Their result: "when not provided with a correct reference, judges show high agreement with human experts only on questions the judges were able to correctly answer themselves." And the warning about your dashboard: "Although aggregate level statistics might imply a judge has high agreement with human annotators, it will struggle on the subset of questions it could not answer." Read that twice. An ungrounded judge is reliable on the easy cases and unreliable on the hard ones, and the aggregate score averages the two into a number that looks fine. Their fix is cheap: give the judge a correct, human-written reference answer. They also showed that a weaker judge with high-quality human references beats a stronger judge with synthetic ones, which means the effort belongs in writing good reference answers rather than in buying a bigger judge model. As they put it: "Using LLM-as-a-Judge does not alleviate the need for human oversight." The honest summary: an LLM judge is a regression detector, not an oracle. It is good enough to tell you this release got worse than last release on a fixed dataset. It is not good enough to define what good means.

This is the question that stops most teams from starting, and the honest answer is that there is no peer-reviewed guidance on eval set sizing. The specific numbers circulating online, 50 to 100, 30 to 50, 200 to 500, trace back to vendor blogs with no study behind them. Treat them as opinion. Two anchors are genuinely attributable, and they are enough: - Anthropic, writing up a production agent system in 2025, start with about 20 queries representing real usage patterns. Their stated reason is that small eval sets catch large effects. Early in development, the changes you are making are dramatic enough that 20 cases will surface them. - Hamel Husain, working across client products, generates around 50 synthetic instructions per feature, reviews human batches of 25 to 50 examples, and accumulates hundreds of assertion-style unit tests over the life of a product. The failure mode here is not too few test cases. It is waiting. Teams delay building evals until they can build a proper set, and the proper set never gets built, and eighteen months later there is still no way to tell whether last week's prompt change helped. Twenty real cases running in CI beats a perfect suite that exists only in a planning document. One statistical caution before you over-read a small suite. A 200-example eval showing 84% for one config and 82% for another is telling you nothing: that difference sits inside the noise. Small sets are for catching large regressions, not for choosing between two similar options. If you need to resolve a two-point difference, you need either a much larger set or a real online measurement.

The eval set that works is the one assembled from things that have already gone wrong. The one that does not work is the one written in a planning session by someone imagining how users might behave. Husain's error-analysis process is the method worth copying, and note that choosing metrics comes last, not first: 1. Pull real production traces. Not the final answer only. The original request, any clarification turns, the retrieved context, the tool calls, the intermediate decisions, and the final action. 2. Read them and write freeform notes on what went wrong, in concrete language. Not "bad response". Something like "called the write tool before receiving approval". 3. Group those notes into a taxonomy of failure modes. Count how often each occurs. 4. Build evals for the categories that are both frequent and costly. Only then. His summary of why this works: "This structured process is effective because it ensures you're building evaluations for the problems your users are actually having, not the ones you imagine." There is a research finding that explains why you cannot skip step 2. Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran and Arawjo, "Who Validates the Validators?" (UIST 2024), named the phenomenon criteria drift: people's evaluation criteria are a function of the outputs they see. You cannot write your rubric before you look at outputs, because grading outputs is the process that teaches you what you actually wanted. Any plan that starts with "define the quality criteria, then collect examples" has the order backwards. When you sample traces, oversample on failure signals you already have: thumbs-down, session abandonment, repeated clarification requests, downstream parse errors, support tickets. Those are free labels sitting in your logs. Then adopt the one rule that compounds: every incident becomes a permanent test case. Something broke in front of a customer, you fixed it, and now it is a row in the eval set forever. Do that for a year and you have a suite that is specific to your product, impossible for a competitor to copy, and grounded entirely in real breakage rather than imagination.

If you take one engineering insight away, make it this one. Error rate is mostly a function of how tightly the output is grounded in supplied data. It is not a fixed property of the model. The measured spread is enormous, and it is between tasks rather than between vendors. | Task type | What is being measured | Measured error rate | Source | |---|---|---|---| | Grounded summarisation | Claims in a summary that the supplied source document does not support | 1.8% for the best listed model; strong models cluster in the 3 to 8% band | Vectara Hallucination Leaderboard, live GitHub leaderboard, pulled August 2026 | | Grounded summarisation, frontier reasoning models | Same measurement, notable models | A number sit above 10%, including claude-sonnet-4-6 at 10.6% and gpt-5.2-high at 10.8% | Same leaderboard, same pull | | Ungrounded factual recall | Hallucination on PersonQA prompts, no source document supplied | 33% for o3, 16% for o1 | OpenAI o3 and o4-mini System Card, 16 April 2025 | | Ungrounded domain recall | Hallucination on specific legal queries | 69% for ChatGPT 3.5 up to 88% for Llama 2 | Dahl, Magesh, Suzgun and Ho, Stanford RegLab and HAI, 2024 | Sit with the middle two rows. Same company, same generation of models. Roughly 2% error when the model is handed the source material, and 33% when it is asked to recall facts about people from memory. That is more than an order of magnitude, driven entirely by task design rather than model choice. This is the whole practical argument for retrieval, for citations, and for building features that hand the model its source material instead of asking it to remember. It is also a design rule you can apply before writing any code: if your feature requires ungrounded recall, you have chosen a task class with a high floor on error rate, and no amount of prompt work will move it much. Redesign the task so the answer is derivable from something you supply. The Vectara leaderboard is continuously updated, so if you cite it, cite it with your pull date. The point stands regardless of which model currently sits on top. One related constraint belongs here. Hong, Troynikov and Huber's "Context Rot" work at Chroma Research (2025) evaluated 18 models while deliberately holding task complexity constant and varying only input length. Every model tested degraded as input grew, and the degradation was uneven and model-specific. The relevance to evals is direct: if your eval set uses short inputs and production sends long ones, your eval is measuring a different system than the one your users have. Match the input length distribution of your eval set to production, or accept that you are testing a friendlier version of your own product. There is more on how input length behaves in our guide on context windows and token limits.

The tooling question matters less than the process question, but the wrong tool does add friction, and friction is the enemy here. | Tool | Maintainer | Licence | What it is for | |---|---|---|---| | Inspect | UK AI Security Institute with Meridian Labs | MIT, open source | Frontier-model and agentic evaluation. Declarative dataset, task, solver and scorer primitives, sandboxed execution with Docker built in, and a log viewer. Ships more than 200 pre-built evaluations. The first government-backed open-source AI eval framework | | Promptfoo | Promptfoo | MIT, open source | YAML and CLI-first prompt and model comparison, plus red-teaming for prompt injection, jailbreaks, PII leakage and unsafe tool use. Fastest of these to set up | | DeepEval | Confident AI | Apache 2.0, open source | Pytest-style Python framework for LLM applications: test cases, metrics, thresholds, CI runner. Broadest metric coverage of the open-source options | | Ragas | Exploding Gradients | Apache 2.0, open source | RAG-specific metrics: faithfulness, context precision, answer relevance. Purely open source, no paid tier | | Arize Phoenix | Arize AI | Elastic License 2.0 | OpenTelemetry-native self-hostable tracing plus evaluation | | Langfuse | Langfuse | Open source, self-hostable, managed option | Lightweight observability: trace visualisation, prompt versioning, cost and latency tracking per model and per trace | | LangSmith | LangChain | Proprietary SaaS | Tracing, datasets and evals native to LangChain and LangGraph. Path of least resistance if you already live there | | Braintrust | Braintrust | Proprietary SaaS | Dataset management through to scoring, production logging and CI release gates in one system | Two notes on reading that table. Most of the tool-comparison articles you will find in search results are content marketing published by the vendors themselves, so their factual descriptions are roughly reliable and their comparative claims are not. And the pattern almost every team converges on is two tools rather than one: a lightweight framework that gates deploys in CI, plus an observability platform that captures production traces so you have raw material for the next round of evals. If you are starting from nothing and want the smallest useful step, it is not a tool at all. It is a folder of 20 real inputs, a script that runs them, and a set of assertions.

Evals are worth building. They are also routinely oversold, and a team that trusts a green dashboard uncritically is in a worse position than one that knows it is guessing. Six honest limitations. Overfitting to the eval set is the default, not the exception. Every time you debug a failure by adjusting the prompt against a specific eval example, you are fitting to that example. After six months of iterating against the same fixed set, those cases are no longer blind. The prompt has been shaped around them, and the score is partly measuring how well you have memorised your own test suite. This is ordinary train-test leakage wearing new clothes. The mitigation is to hold back a portion of cases you never debug against, and to keep adding fresh production failures so the set does not calcify. Public benchmarks are contaminated. Benchmark items leak into training corpora through web crawls, so a model's rank on a public benchmark is partly a measure of benchmark exposure rather than capability. The practical implication is a sequencing rule: use public benchmarks once, to eliminate obviously unsuitable candidates, then build your own 50-odd examples from your actual use case and make every decision after that on your set. High judge agreement is not evidence of correctness. Two findings converge here. Krumdick and colleagues showed that aggregate agreement hides failure on exactly the subset of questions the judge could not answer itself. And a 2026 paper by Xu, Zheng, Wu and Ma, "Beyond Surface Judgments", argues that LLM judges "form a coherent evaluative group that is much more aligned internally than it is with human readers, indicating that internal agreement is not evidence of validity." A panel of judges can be confidently and consistently wrong together, and adding more judges does not fix it. Only human labels break the loop. An eval measures a proxy you chose. Nothing guarantees that proxy correlates with the outcome your customer cares about. A green dashboard alongside unhappy users is a standard and unremarkable failure. This is the argument for eventually running online evals: at some point you have to measure a real business outcome rather than a rubric score, because the rubric is your opinion about quality and the outcome is the customer's. Maintenance is real, recurring, and usually unowned. Eval sets decay. Product behaviour changes and expected outputs go stale. Judge models get silently updated by the provider and your scores shift with no code change on your side. Somebody has to re-calibrate the judge against expert labels periodically, and in most teams that somebody is nobody. Budget for it or accept that your suite has a shelf life. Small deltas are noise. Repeating this because it is where teams lose the most credibility: shipping on a two-point difference in a 200-case eval and then seeing no production change is not bad luck, it is what the statistics predicted. None of this argues against building evals. It argues for holding them at the right confidence: an eval suite tells you reliably when you have made things worse, tells you roughly when you have made things better, and never tells you that your feature is good.

What is an eval, in one sentence? A fixed set of test cases, plus a grader, plus a pass threshold, run automatically so that a quality regression blocks a deploy the way a failing unit test does. The grader can be code, a model, or a person. How many test cases do I need before evals are worth running? About 20 is a defensible starting point. That is the number Anthropic's engineering team recommend for a production agent system, on the reasoning that early changes are large enough for a small set to detect. Hamel Husain works in human review batches of 25 to 50. There is no peer-reviewed guidance on set sizing, so treat any more specific number you see online as opinion rather than research. The set should be small enough to run in CI in a couple of minutes and should grow every time something breaks. Can I trust an LLM to grade another LLM's output? For regression detection, mostly yes. GPT-4 judges matched human preferences at over 80% agreement in the NeurIPS 2023 MT-Bench study, which is the same level humans agree with each other. For deciding what "good" means, no. Judges show position bias, verbosity bias, and self-preference, and the 2025 "No Free Labels" study found that without a correct reference answer, judges only agree with human experts on questions they could have answered themselves. Always give the judge a human-written reference answer, and re-check it against human labels periodically. Why does my AI feature hallucinate so much more on some tasks than others? Because grounding, not model choice, is the dominant factor. On grounded summarisation, where the source document is supplied, the best model on Vectara's leaderboard sits at 1.8% and strong models cluster between 3 and 8%. On ungrounded factual recall, OpenAI's own system card reports o3 hallucinating on 33% of PersonQA prompts, and Stanford RegLab measured 69% to 88% on specific legal queries. If your feature depends on the model remembering facts, redesign it so the facts are supplied. Should I build evals before or after launching? Build a small assertion suite before launch, because assertions cost nothing and catch the embarrassing failures. Build the real eval set after launch, from actual production failures. You cannot write good test cases for failures you have not seen yet, and the research on criteria drift says you cannot even write a good rubric until you have graded some real outputs. What is the difference between an eval and a unit test? A unit test asserts a deterministic property and either passes or fails. An eval scores a non-deterministic output against a threshold, so it is measured as a rate across a dataset rather than a single pass or fail. Assertion-style evals sit right on the boundary and are the cheapest, most under-used layer of the whole stack.