Evaluating Agents (Evals as Code)

What you'll build

By the end of this lab you'll have a working eval harness — the thing that turns "the demo looked good" into "we can ship this with confidence." You'll define a golden test set, score an assistant's answers with a mix of deterministic checks and an LLM-as-judge, roll the results into a scorecard, and wire a pass/fail gate you could drop straight into CI to block a bad change.

The system you're evaluating is the Acme Orders assistant from Lab 04, so the series stays continuous. The harness treats it as a black box (question in, answer out), which means everything you learn applies equally to a single model call, a tool-using agent, or a whole multi-step pipeline.

Crucially, the entire lab runs offline with no third-party packages: the assistant is a deterministic mock, and the LLM-as-judge defaults to a deterministic mock judge. A real-model adapter is included so you can see exactly how to plug in a live model when you want to.

Why this matters (the 5-minute business case)

Read this section before handing the lab to your team.

Evals are the single biggest gap between "we built an AI demo" and "we run AI in production." Traditional software is tested with assertions that are true or false: add(2,2) == 4. AI systems are different — outputs are probabilistic, "correct" is often a spectrum, and a prompt tweak that fixes one case can silently break ten others. Without a systematic way to measure quality, teams fly blind: they ship on vibes, regressions slip through unnoticed, and "is it good enough?" becomes an argument instead of a number. Evals make quality measurable, repeatable, and enforceable — they are to AI what unit tests are to software.

Three reasons this is a leadership-level concern, not just an engineering nicety:

1. It converts a risk decision into a number. "Should we ship this assistant to customers?" is, without evals, a gut call. With evals it's "it scores 94% on our golden set, above our 90% bar, with zero regressions on safety cases." That's a decision an accountable leader can actually make — and defend.
2. It makes iteration safe and fast. The reason teams are scared to change a working prompt is that they can't tell what they broke. An eval suite in CI means anyone can improve the system and get immediate, objective feedback on whether they helped or hurt. That turns AI development from cautious and slow into fast and confident — the same shift automated testing brought to software.
3. It catches the failures that actually hurt you. The cases that matter most in production — hallucinating a fact, taking an unsafe action, leaking out of scope — are exactly the ones a happy-path demo never exercises. A golden set lets you weight those cases heavily and gate on them specifically, so a change that makes the assistant confidently wrong can't ship.

The honest caveats, because pragmatic adoption means naming them:

• Your eval is only as good as your golden set. A high score on an unrepresentative or stale dataset is worse than no score — it's false confidence. Golden sets need curation, real-world cases (including past incidents), and ongoing maintenance. Treat the dataset as a first-class asset, not a one-time chore.
• LLM-as-judge is powerful but not free or neutral. Using a model to grade open-ended answers scales beautifully, but judges are non-deterministic and carry biases (favoring longer or more confident answers, or their own phrasing). A judge must itself be validated against human labels periodically, or you're measuring the judge's quirks, not your system's quality.
• Don't optimize the metric into meaninglessness. Once a score becomes a target, there's pressure to game it (Goodhart's law). Keep a held-out set, rotate cases, and remember the score is a proxy for user value, not the value itself.

A useful framing for a non-technical stakeholder: "Evals are automated tests for our AI. They give us a quality score we trust, let us change the system without fear of silently breaking it, and let us block any change that makes it worse — especially on the safety cases that matter most."

Concepts you'll use (2-minute primer)

• System Under Test (SUT) — whatever you're evaluating (a model call, an agent, a pipeline). The harness only needs question -> answer.
• Golden set — a curated list of cases: an input plus how to judge the output. Your source of truth for "good."
• Scorer — a function that decides whether one answer passes. Two families: deterministic (string/regex checks — cheap, fast, repeatable) and LLM-as-judge (a model grades against a rubric — flexible, for open-ended answers).
• Scorecard — the aggregate: an overall (weighted) score plus breakdowns, e.g. by tag.
• Gate — the ship/no-ship rule: a score threshold and/or a regression check against a saved baseline. In CI, the gate is an exit code.

Prerequisites

Get the starter repo. This lab refers to files in evals-lab/: sut.py, cases.json, scorers.py, judge.py, run_evals.py. Unzip it and cd in.

Setup (1 minute)

cd evals-lab
python3 --version       # 3.9+; nothing to install

That's it — the harness is pure standard library. (Only the optional real-model judge needs the Lab 01 SDK.)

Exercise 1 — A golden set and your first run (12 min)

Goal: Define cases, run the SUT over them, and read a scorecard.

1a. Look at a case

Open cases.json. Each case pairs an input with how to score the output:

{
  "id": "order-total",
  "question": "How much was order A-1024?",
  "scorer": "contains",
  "expected": "$149.99",
  "weight": 1,
  "tags": ["total"]
}

The fields that matter: a stable id, the question fed to the SUT, the scorer to use, the expected value it needs, a weight (how much this case counts), and tags for slicing the results. This is your specification of correct behavior, in data — which means non-engineers (support leads, PMs) can contribute cases without touching code.

1b. Run the suite

python run_evals.py --sut good --save-baseline

SUT: good   judge: mock
  [PASS] status-summary-shipped             all required substrings present
  [PASS] order-total                        contains '$149.99'
  ...
Score: 100.0%  (12/12 weighted, 9/9 cases)
Threshold gate: 100.0% >= 85%  -> PASS
RESULT: PASS — safe to ship.

The harness ran the assistant on every case, scored each, and printed a scorecard. --save-baseline recorded this run to baseline.json for later regression checks. It also wrote report.md and report.json — open report.md to see the shareable version (the artifact you'd attach to a PR or a launch review).

Business callout: notice the SUT is a black box — the harness never looked inside sut.py. That's deliberate: the same harness evaluates a mock today, a real model tomorrow, and a multi-agent system next quarter, unchanged. Your evaluation outlives any particular implementation.

Checkpoint: You can express "good behavior" as data and get a score. Now make the scoring smart enough for real AI outputs.

Exercise 2 — Scorers beyond exact match (12 min)

Goal: Handle the reality that good answers vary in wording, and that some criteria are about what an answer must not say.

Exact-match testing breaks immediately on AI outputs — "Order A-1024 is shipped" and "A-1024 has shipped" are both correct but textually different. Open scorers.py and look at the family of deterministic scorers:

• `contains` — the answer includes a required substring (case/space-insensitive).
• `includes_all` — the answer includes every substring in a list (e.g. both a status and a delivery date).
• `regex` — the answer matches a pattern (e.g. a dollar amount \$\d+\.\d{2}).
• `excludes` — the answer contains none of a forbidden list. This is how you catch hallucination.

That last one is worth dwelling on. Look at this safety case in cases.json:

{
  "id": "unknown-order-excludes-fake-date",
  "question": "When will order A-9999 arrive?",
  "scorer": "excludes",
  "expected": ["estimated delivery", "shipped"],
  "weight": 2,
  "tags": ["safety", "hallucination"]
}

Order A-9999 doesn't exist. A good assistant says so; a bad one invents a delivery date. The excludes scorer fails the answer if it confidently states a status or ETA for an order that isn't real — and weight: 2 makes that failure count double.

Rule of thumb: reach for the cheapest scorer that captures the criterion. Deterministic checks are fast, free, and perfectly repeatable — use them for anything expressible as a string rule. Save the LLM judge (next) for genuinely open-ended quality.

Business callout: weighting is how you encode what your business actually cares about. A wrong order total is bad; a hallucinated answer that a customer acts on is worse. Weighting safety and hallucination cases higher means your single headline score reflects real-world risk, not just an unweighted average of easy and hard cases.

Checkpoint: You can score varied phrasings and forbid dangerous content. Next, grade the answers that no string check can.

Exercise 3 — LLM-as-judge (12 min)

Goal: Grade open-ended answers against a rubric, and understand when (and how carefully) to trust a model to do the grading.

Some criteria can't be a substring check: "is this summary clear and complete for a customer?" That's what an LLM-as-judge is for. Look at the rubric case in cases.json:

{
  "id": "customer-summary-rubric",
  "question": "Summarize order A-1024 for the customer.",
  "scorer": "llm_judge",
  "rubric": "PASS only if the answer states the order's status AND its estimated delivery date, and is reasonably polite/clear.",
  "weight": 2
}

The llm_judge scorer (in scorers.py) hands the question, answer, and rubric to a judge (judge.py).

3a. The judge interface

judge.py exposes one function — judge_answer(question, answer, rubric) -> (passed, reason) — with two implementations:

• `_mock_judge` (default): a deterministic stand-in that encodes our rubric's intent as checks. It runs offline and gives the same answer every time, which is what makes this lab reproducible.
• `_llm_judge`: the real thing — it prompts a model to return a strict PASS/FAIL verdict plus a reason. Enable it with EVAL_JUDGE=llm (needs the Lab 01 SDK and an API key).

Run it both ways if you have a key:

python run_evals.py --sut good                       # mock judge (default)
EVAL_JUDGE=llm python run_evals.py --sut good        # real model judge

3b. Treat the judge as a system that itself needs evaluating

This is the subtle, important part. A model grading a model is convenient but not automatically trustworthy. In production you should:

• Validate the judge against human labels. Have people grade a sample, then check how often the judge agrees. If agreement is low, your "scores" are noise.
• Watch for known biases — judges tend to favor longer, more confident, or more familiar-sounding answers. Write rubrics that pin down specific criteria rather than asking "is this good?"
• Keep judges deterministic where you can (low temperature, strict output format) so re-runs are comparable.

Business callout: LLM-as-judge is what makes evals scale to thousands of open-ended cases no human could grade by hand — but an uncalibrated judge gives confident, wrong numbers, which is more dangerous than no number at all. Budget for periodically checking the judge against humans; treat it as instrumentation that needs its own calibration, like any sensor you'd make a decision on.

Checkpoint: You can grade open-ended quality and you know not to trust a judge blindly. Now turn the scorecard into a gate.

Exercise 4 — Gate it in CI: thresholds and regressions (12 min)

Goal: Make the eval block a bad change — the step that turns measurement into protection.

A score is interesting; a gate is what protects production. run_evals.py exits non-zero unless the run passes, so CI can fail a pull request automatically.

4a. Watch a regression get caught

You saved a baseline from the good assistant in Exercise 1. Now evaluate a regressed version — one where someone "improved" the summaries but accidentally dropped the delivery date and made it guess on unknown orders:

python run_evals.py --sut regressed --baseline baseline.json
echo "exit code: $?"

  [FAIL] status-summary-shipped             missing: ['2026-06-25']
  [FAIL] unknown-order-no-hallucination     missing: ["couldn't find"]
  [FAIL] unknown-order-excludes-fake-date   contains forbidden: ['estimated delivery', 'shipped']
  [FAIL] customer-summary-rubric            [mock-judge] no estimated delivery date
  ...
Score: 33.3%  (4/12 weighted, 4/9 cases)
Threshold gate: 33.3% >= 85%  -> FAIL
Regression check: baseline 100.0% -> now 33.3% (drop +66.7%, allowed 0%)  -> FAIL
RESULT: FAIL — do not ship.
exit code: 1

Two independent guards fired: the absolute threshold (below 85%) and the regression check (a drop versus the recorded baseline). Either alone would have blocked the change. The weighted safety cases failed loudly — exactly the failures you most want a gate to stop.

4b. The two-guard pattern

• Threshold gate (--threshold): "never ship below this quality bar." Good for an absolute floor.
• Regression gate (--baseline + --max-regression): "never ship something worse than what we have now." Good for catching subtle backslides even when you're still above the floor.

Use both. The threshold stops obviously-broken changes; the regression check stops quiet erosion.

4c. Wire it into CI (conceptually)

In a CI pipeline you'd run python run_evals.py --baseline baseline.json on every pull request. Because it exits 1 on failure, the PR goes red and can't merge. You'd publish report.md as a build artifact or PR comment so reviewers see which cases regressed, and update baseline.json deliberately (a reviewed commit) whenever you intend to move the bar.

Business callout: this is the whole payoff. "Don't ship AI that's worse than what we have, or below our quality bar" stops being a hope enforced by careful humans and becomes a rule enforced by the build. That is what lets an organization let many people iterate on an AI system quickly without quality drifting — the same leverage CI/CD gave software teams.

Checkpoint: Your evals don't just measure quality — they enforce it, automatically, on every change.

Wrap-up

You built an eval harness end to end. You can now:

• Express "good behavior" as a golden set of data-defined cases
• Score answers with deterministic scorers and an LLM-as-judge, and weight the cases that matter most
• Roll results into a scorecard with shareable reports
• Gate changes on an absolute threshold and on regression versus a baseline, with a CI-ready exit code

The one idea to carry forward: evals are unit tests for probabilistic systems. They convert "is the AI good enough?" from an argument into a number, make iteration safe by catching regressions automatically, and let you enforce — not just hope for — quality on the cases that carry real risk. The harness is easy; the discipline that makes it valuable is curating a golden set that reflects reality and keeping your judge honest.

Stretch challenges (for the labs series)

1. Evaluate the real assistant. Point the SUT at the Lab 04 MCP server (or the Lab 01 Interactions API) instead of the mock, and run the same cases.json. The harness shouldn't need to change — that's the black-box payoff.
2. Calibrate the judge. Hand-label the rubric cases yourself, then run with EVAL_JUDGE=llm and measure how often the model judge agrees with you. Report the agreement rate alongside the score.
3. Add cost & latency to the scorecard. Have the runner record tokens/time per case and report p50/p95 latency next to quality — so a change that's slightly better but twice as slow is visible in the gate decision.
4. Grow the golden set from incidents. Add a case for every real failure you find, so the suite gets stronger exactly where the system has been weak. This is how a golden set earns its name over time.

Troubleshooting

References

This lab is methodology-focused; these are good companions:

• Anthropic — building evaluations / test-driven development for prompts
• OpenAI Evals (open-source eval framework, for comparison)
• Lab 01 (Interactions API) in this series — for wiring a real model as SUT or judge
• Lab 04 (MCP) in this series — the assistant this lab evaluates

A note for your readers: this lab uses a deterministic mock SUT and mock judge so it runs offline and reproducibly. Both are one-function swaps for real models (see sut.llm_sut and judge._llm_judge). When you move to real models, the headline change is non-determinism — run multiple samples, report variance, and calibrate your judge against human labels before trusting the numbers.