Information in this post reflects publicly available sources as of October 17, 2026.
A customer complains that your support assistant gave a refund policy that is two years out of date. You look at the prompt, add a line telling the model to prefer the newest policy documents, and try the customer's question: the answer is now correct. You try four more questions you can think of, and they look fine too. You ship it.
A week later, a different complaint arrives. The assistant has started ignoring older documents entirely, including the warranty terms that have not changed since 2023. Your one-line fix solved one problem and quietly created another, and nothing in your process could have caught it.
Trying a handful of examples and deciding it looks right has a name in the industry: the vibe check. Every team starts there. The teams whose AI features keep getting better have replaced it with evals: a repeatable way to measure whether a change made things better or worse before users find out.
Why AI features need their own kind of testing
Traditional software has unit tests. Given this input, the function returns exactly that output, every time. When a test fails, something is broken.
Language model features break that model in three ways. The output is open-ended, so there is rarely one correct string to compare against. The output is non-deterministic, so the same input can produce different answers on different runs. And changes have non-local effects: a prompt tweak, a new retrieval setting, or a model upgrade can improve one kind of question while degrading another that looks unrelated.
That last point is the one that hurts. Every earlier post in this series involved knobs that interact: chunk sizes in RAG, tool descriptions in agents, what goes into the context window. Without a way to measure the effect of turning a knob, every change is a guess, and every model upgrade is a leap of faith.
An eval suite is the AI equivalent of a test suite. It runs your application against a fixed set of inputs, grades the outputs, and reports how the results compare to the last version. It cannot tell you that your system is correct. It can tell you, reliably, whether it got better or worse.
How it works
The dataset
Everything starts with a set of test cases, often called a golden set. Each case has an input, such as a user question, and some description of a good outcome: a reference answer, a list of facts that must appear, things that must not appear, or a rubric.
The best cases come from real usage. Pull questions from logs, support tickets, and bug reports, especially the ones that went wrong. Synthetic cases generated by a model are useful for filling gaps, but they tend to be cleaner and easier than what real users type, which makes the eval look better than reality.
Start smaller than you think. Fifty to a hundred well-chosen cases, tagged by category such as billing, warranty, or account access, will catch far more regressions than no eval at all. Make sure the set includes the hard cases: ambiguous questions, questions with no answer in your documents, and inputs that previously caused bugs.
Graders
Each output needs a verdict. There are three kinds of grader, and most real eval suites use all three.
Code checks come first wherever they apply. Did the output parse as valid JSON, did the SQL query return the right rows, did the generated code pass its tests, and does the answer cite at least one source? These checks are free, instant, and never disagree with themselves.
For open-ended text, the workhorse is the LLM judge: a second model call that reads the output and grades it against explicit criteria. Judges scale to thousands of cases and can assess things code cannot, such as whether an answer is faithful to its sources or actually addresses the question.
Human review is the ground truth that keeps the other two honest. It is too slow to grade every run, so its main jobs are labeling a calibration set for the judge and spot-checking production samples.
Designing a judge you can trust
An LLM judge is a model, so it has the usual weaknesses of models, plus some specific to grading. Research has documented several consistent biases. Judges tend to favor longer answers, prefer whichever answer appears first in a side-by-side comparison, and rate outputs written in their own style more highly than humans do.
A few design choices make judges far more reliable. Ask for a binary pass or fail on one specific criterion, rather than a score from 1 to 10; models are poor at using numeric scales consistently. Give the judge a reference answer or a list of required facts whenever you have one. Grade separate criteria, such as factual accuracy and tone, with separate questions.
In pairwise comparisons, run each pair in both orders and only count verdicts that agree. The rate at which verdicts flip when you swap the order is a direct measure of how much position bias you are dealing with.
A judge prompt for the support assistant might look like this:
You are grading an answer from a customer support assistant.
Question: {question}
Reference answer: {reference}
Candidate answer: {answer}
Criterion: Every factual claim in the candidate is consistent with the
reference, and the candidate answers the question that was asked.
Extra detail is fine if it is accurate. Ignore tone and formatting.
Reply with PASS or FAIL on the first line, then one sentence explaining why.Then validate the judge before relying on it. Have a person label a hundred or so outputs, run the judge on the same outputs, and measure how often they agree. If agreement is high, the judge can grade at scale. If it is not, fix the rubric before you trust any number it produces.
Treat the judge's prompt and model as versioned code: if either changes, re-check agreement. A judge that silently moves to a new model version can shift every score in your history.
A minimal eval harness
Eval frameworks add dashboards and history, but the core is small. This version runs two versions of an application over the same cases and compares the results metric by metric:
import json, re
from statistics import mean
CASES = [json.loads(line) for line in open("evals/support.jsonl")]
# each case: {"question": ..., "reference": ..., "must_not": [...], "tag": "billing"}
def grade(case, answer):
return {
# Code checks: free and deterministic.
"cites_source": bool(re.search(r"\[\d+\]", answer)),
"no_forbidden": not any(p in answer.lower() for p in case["must_not"]),
# LLM judge: returns True for PASS, using the rubric prompt above.
"correct": judge(case["question"], case["reference"], answer),
}
def run(app):
results = [grade(case, app(case["question"])) for case in CASES]
return {metric: mean(r[metric] for r in results) for metric in results[0]}
baseline, candidate = run(app_current), run(app_candidate)
for metric in baseline:
print(f"{metric:14} {baseline[metric]:5.0%} -> {candidate[metric]:5.0%}")The judge, app_current, and app_candidate functions stand in for your own model calls. Everything else is real. Each metric is reported separately, so a gain in correctness cannot hide a drop in citations. In practice you would also break results down by tag, because an overall score can hold steady while one category collapses.
Is the difference real?
Suppose the candidate scores 86% and the baseline 79%. It is tempting to ship. But eval scores are measurements on a sample, and small samples are noisy.
A useful rule of thumb: with n test cases, the margin of error on a pass rate is roughly 1/√n at worst. That works out to about ±10 points on 100 cases, ±5 on 400, and ±3 on 1,000. A small eval set can catch big regressions, but it cannot reliably detect a 2-point improvement, so do not celebrate one.
Two habits shrink the noise. Compare versions on the same cases and look at which individual cases flipped from pass to fail, rather than only comparing totals. And because outputs vary between runs, run each case two or three times, or set the temperature to zero where your application allows it. Reading the flipped cases is also where most of the learning happens.
Evaluating agents
Agents need two kinds of grading. The outcome is whether the task got done, ideally checked in code: the tests pass, the ticket was created with the right fields, the database ended in the expected state. The trajectory is how it got there: which tools it called, how many steps it took, what it cost, and whether it did anything it should not have.
Because agents are long and variable, a single run tells you little. Run each task several times and report how often it succeeds. An agent that succeeds on 3 of 5 attempts is a very different product from one that succeeds on 5 of 5, even though both "can do the task."
Offline and online
Everything so far is offline evaluation: testing before you ship. Online evaluation watches what happens after. It includes user signals such as thumbs-down ratings and escalations to a human, a judge run on a sample of production traffic, and A/B tests between versions.
The two connect into a loop. Every failure found in production becomes a new test case, so the offline suite grows to cover exactly the problems real users hit. Over time, the eval set becomes one of the most valuable assets the team owns: a precise record of everything the system must keep getting right.
When to invest, and when not to
Build an eval set as soon as more than one person is changing a prompt, or as soon as you plan to switch models. Model upgrades are the clearest case: a new version may be better on average and worse on your specific use, and an eval suite answers that question in minutes rather than through customer complaints.
The investment should match the stakes. A small internal tool can live with thirty cases and code checks. A customer-facing assistant handling billing needs hundreds of cases, a validated judge, and online monitoring. And for one-off prompts you will run a few times by hand, a vibe check is genuinely fine.
Do not confuse your evals with public benchmarks. Leaderboard scores tell you which models are strong in general. They say almost nothing about whether a model handles your documents, your users, and your edge cases. Use benchmarks to build a shortlist, then use your own evals to choose.
What goes wrong
The eval set is easier than production. Cases written by the team that built the feature tend to be clear, well-formed, and answerable. Real users are vague, misspell things, and ask questions your documents do not cover. If your eval scores are high and users are unhappy, the dataset is the first suspect, so refresh it from production regularly.
Teams overfit to the eval set. After enough rounds of tweaking a prompt until every case passes, the prompt is tuned to those specific cases rather than to the underlying task. Keep a held-out set that nobody tunes against, and check it before major releases.
Nobody validated the judge. A judge that agrees with humans only 60% of the time produces numbers that look precise and mean very little. Worse, if the judge comes from the same model family as the application, its self-preference can inflate scores. Measure agreement first, and consider a judge from a different provider.
One number hides regressions. An overall pass rate that moves from 84% to 85% can conceal a category that fell from 90% to 60%. Always break results down by tag and read the cases that flipped.
The evals never run. A suite that someone has to remember to run will stop being run within a month. Put it in continuous integration for prompt and configuration changes, and run it automatically when a new model version becomes available.
Cost and complexity
Evals are cheap next to the problems they catch. Take a 300-case suite where each case costs about $0.01 for the application call and $0.005 for the judge. One full run costs about $4.50. Running each case three times to smooth out variance brings it to about $13.50, still less than an hour of an engineer's time.
Human labeling is the expensive part. At a minute per case, labeling 300 outputs takes five hours of an expert's time. That is why humans label a calibration set once and spot-check after that, while the judge does the daily grading.
The real cost is ongoing ownership. Datasets go stale, judges need re-validation when their model changes, and somebody has to read the flipped cases after each run. Teams that treat the eval suite as a product of its own, with an owner and a review cadence, get far more from it than teams that treat it as a one-time project.
The current landscape
Hosted platforms such as Braintrust and LangSmith combine datasets, experiment tracking, judges, and production tracing in one place, with SDKs for Python and TypeScript. Weights & Biases offers Weave for teams already using its tooling.
Open-source options cover most needs without a vendor. Arize Phoenix provides tracing and evals built on OpenTelemetry. DeepEval offers a pytest-style API for running evals in CI. Promptfoo runs test cases from YAML configuration across many providers; OpenAI announced its acquisition in March 2026 and said the open-source project will continue.
Ragas provides retrieval-specific metrics such as faithfulness and context recall, and the UK AI Security Institute's Inspect framework is widely used for structured evaluations.
Model providers also build eval tooling into their consoles, which is a reasonable place to start for a single prompt. Whichever tool you pick, the dataset and the grading criteria are what carry the value. Keep them in formats you can move, such as JSONL files in your repository, so switching tools never means starting over.
Summary
Evals turn AI development from guesswork into engineering. Without them, every prompt change and model upgrade is a bet you settle by waiting for complaints. With them, you know within minutes whether a change helped, which categories it hurt, and whether the difference is real or just noise.
The hard part is not the tooling. It is the dataset: real cases, refreshed from production, tagged so regressions cannot hide, and graded by checks you have validated. Build that first, wire it into your release process, and every other technique in this series becomes something you can measure rather than something you hope works.
Part of the Applied AI series: AI from the builder's side.