AI Atlas
Intermediate · ~2 min read #evals #evaluation #testing

Evals

Repeatable tests for LLM applications

Repeatable tests that measure how well an LLM application performs on its own tasks: inputs, expected behavior and a scoring method. The answer to 'is it better or worse?' when a prompt, model or tool changes.

THE EVAL LOOP · PRODUCTION → SET → CIEVAL SETreal casesversionedRUN × ksame input,varying outputSCOREcode → judge→ humanCI GATEbelow threshold= no mergebillingrefundtechnicalread slices, not the totalPRODUCTIONtraces · cheap checks on every call · judge on a sampleshipfailed traces→ new testsEvery failure seen in production joins the eval set; the set catches the next change in CI
Definition

An eval (short for evaluation) is the LLM-world counterpart of a software test suite. The difference: LLM output often has no single correct answer, and the same input can produce different answers twice. So an eval has three parts:

  • Input: ideally an example from real usage (a production conversation, a support ticket, a prompt the team tried by hand).
  • Expected behavior: sometimes an exact answer, more often a criterion ("the refund must not exceed the order amount").
  • Scoring method: a code-based check (exact match, JSON schema, unit test), LLM-as-a-judge or human review.

Evals run in two places. Offline evals measure a change on a fixed, versioned eval set before it ships (usually in CI). Online evals watch quality on a sample of live traffic. Together they form a loop: failures seen in production go into the eval set, and the set catches the next change.

The difference from a benchmark: a benchmark is a shared exam where everyone compares models on the same questions; an eval measures how your application behaves on your tasks.

Analogy

Like a restaurant's tasting routine. The Michelin guide (a benchmark) tells you which restaurants are good in general; but before a chef puts a new recipe on the menu, they compare it against the dishes regulars order most, check the salt, time the service. Once it's on the menu, they still pull a few plates every night and taste them. The first is an offline eval, the second an online eval.

Real-world example

A prompt change is planned for an e-commerce support bot. The team:

  1. Picks 40 conversations from last month's support tickets where the user was unhappy and adds them to the eval set (personal data masked).
  2. Writes code checks for each: is the reply valid JSON, is the order number well-formed, is the refund amount a number.
  3. Uses a binary (pass/fail) LLM judge for "does it follow the refund policy?", after first measuring the judge on an expert-labelled sample.
  4. Runs each example several times, because the same input can give different outputs, and reports the score as a rate.
  5. Sets a CI threshold: if the score drops below main's, the PR fails.

After launch, a share of traffic goes to the same judge asynchronously; conversations that fail are reviewed and added to the set.

Code examples
A minimal eval loop: code check + repeats + rate python
import json

CASES = [
    {"id": "refund-001", "input": "Order ORD-123456 arrived damaged, I want a refund.",
     "must_contain": "ORD-123456"},
    {"id": "refund-002", "input": "How many days do I have to return an item?", "must_contain": "days"},
]
RUNS = 3  # the same input can give different outputs

def run_app(text: str) -> str:
    ...  # call your app here; return a JSON string

def check(case, output) -> bool:
    try:
        data = json.loads(output)
    except json.JSONDecodeError:
        return False
    return case["must_contain"] in data.get("message", "")

passed = sum(check(c, run_app(c["input"])) for c in CASES for _ in range(RUNS))
print(f"pass rate: {passed / (len(CASES) * RUNS):.0%}")
When to use
  • Before and after changing a prompt, model, temperature or tool set
  • When choosing between two models for your own tasks
  • To make sure a failure seen in production never comes back (regression testing)
  • For agents: to measure whether the task was really completed, from the final state of the environment
When not to use
  • Building elaborate infrastructure for a one-off experiment nobody will rerun
  • Measuring general model capability: that's what benchmarks are for
  • As a replacement for user satisfaction: an eval set complements live signals, it doesn't replace them
Common pitfalls

Starting with invented examples only

Synthetic examples generated by an LLM widen coverage but don't represent real users. Anthropic's post on agent evals (January 2026) says 20-50 simple tasks drawn from real failures is a great start.

Trusting a single run

Even temperature 0 doesn't guarantee determinism: OpenAI describes its seed parameter as best effort, and Thinking Machines (2025) showed batch size can change the result. Run each example several times, report rates, and don't trust small differences between versions.

Looking only at the total score

One category can collapse while the overall score holds steady. Label examples and read results by slice.

Deleting examples that fail

The set is versioned like code. Deleting a failing example hides a regression; if the example is wrong, fix it and write down why.

Using a judge without measuring it

LLM judges have documented position, verbosity and self-enhancement biases (Zheng et al., 2023). Measure the judge's agreement on a separate human-labelled sample before trusting its scores.