Evaluations

Before you start: this is a billed Azure service

• Cost and account. An Azure subscription, a Foundry project, and a model deployment. Model-based (AI-assisted) evaluators make model calls — each run bills proportional to dataset size. Start small.
• Authored against the docs, not executed live. Code is written against the official docs and syntax-checked, not run during authoring. Evaluator names/inputs move — pin your SDK version and verify.
• Two languages. Python-primary (the azure-ai-evaluation SDK) with .NET noted.

About this series

This is Part 6 of the twelve-part Microsoft Foundry series. The last labs improved the agent — grounding, tools, memory. This lab answers the question they raise: is it actually better? You'll use Foundry's evaluation tooling to measure quality with managed evaluators and gate ship/no-ship on evidence.

What you'll build

A real evaluation loop for the Acme assistant:

1. Define an eval dataset — representative questions with reference answers.
2. Run local evaluation with managed evaluators (groundedness, relevance) plus a custom check.
3. Run hosted (cloud) evaluation in the project, with results linked to the agent trace.
4. Read the results, compare two versions, and gate a release on a threshold.

Why this matters (the 5-minute business case)

Read this before handing the lab to your team.

You can't manage what you can't measure, and "it seems better" is not measurement. Every change you've made — grounding, tools, memory — should help, but intuition about LLM quality is unreliable, and a prompt tweak that fixes one case routinely breaks three you didn't check. Evaluation turns agent quality from a vibe into a number you can defend.

Three reasons this is a leadership-level concern:

1. Evidence-based ship/no-ship. Without evals, "is this good enough?" is settled by whoever's most confident in the room. With them, it's settled by a score against a representative dataset. That's shipping on data, not hope.
2. Regression protection at scale. Models, prompts, and tools drift. An eval you run on every change catches the silent degradations before a customer does — how you move fast without breaking quality.
3. Managed evaluators beat rolling your own. Good evaluation needs judge models, calibrated metric prompts, and aggregation. Foundry ships an Evaluators catalog — groundedness, relevance, coherence, safety, and agent-specific evaluators — and links results to the agent trace, so a failed score jumps straight to the steps that caused it.

The honest caveats:

• The dataset is the hard part. Managed evaluators are easy; a representative eval set is the real work. Twenty real cases beat a thousand synthetic ones.
• AI-assisted evaluators are estimates, not oracles. Calibrate them against human judgment on a sample before trusting a threshold blindly.
• Evaluation costs money. Every judged run makes model calls. Keep eval sets focused; run the full suite on meaningful changes.

A framing for a non-technical stakeholder: "Evaluation is how we prove the agent is good enough before it ships and stays good after — scored against real questions, with releases gated on the score, instead of finding out from customers."

Concepts you'll use (2-minute primer)

• Evaluators catalog — ready-made evaluators: groundedness, relevance, coherence, fluency, safety, and agent-specific ones (tool-call accuracy, intent resolution). Some are AI-assisted (a judge model); some are computational.
• `evaluate()` — the local entry point: give it a dataset and a set of evaluators, get scored results.
• Hosted (cloud) evaluation — run the same evaluators in the project at scale, with results recorded and linked to the agent trace.
• Gate — turn a metric threshold into a pass/fail check, like a test.

One sentence: score a representative dataset with managed evaluators — locally or hosted — read the results linked to traces, and gate releases on the numbers.

Prerequisites

Step 1 — Define an eval dataset

A dataset is a JSONL file of cases — start small and real, each with the answer a knowledgeable human would give. Include cases the agent can fail so evaluators have something to catch:

{"query": "Can I return opened headphones after 20 days?", "response": "Opened electronics have a 15-day window, so 20 days is past it.", "context": "Opened electronics: 15-day return window, 10% restocking fee.", "ground_truth": "No — opened electronics are returnable within 15 days."}
{"query": "Is accidental damage covered under warranty?", "response": "Yes, accidental damage is fully covered.", "context": "The 1-year warranty covers manufacturing defects, not accidental damage.", "ground_truth": "No — the warranty covers defects, not accidental damage."}

Step 2 — Run local evaluation

Point evaluate() at the dataset with managed evaluators:

import os
from azure.ai.evaluation import evaluate, GroundednessEvaluator, RelevanceEvaluator

model_config = {
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "azure_deployment": "gpt-5-mini",  # judge model deployment
}

result = evaluate(
    data="acme_eval.jsonl",
    evaluators={
        "groundedness": GroundednessEvaluator(model_config),
        "relevance": RelevanceEvaluator(model_config),
    },
)

print(result["metrics"])       # aggregate scores
print(result["rows"][0])       # per-row scores + reasoning

Groundedness scores run 1–5 (higher = better grounded); the warranty row should score low, with reasoning pointing at the wrong claim. The per-row reasoning is where the value is: a number says something changed; the reasoning says what.

.NET note: evaluation in C# runs via the Agent Framework / Azure.AI.* evaluation surface; the evaluator set mirrors Python. Check the .NET docs for your version.

Step 3 — Run hosted evaluation (linked to traces)

Run the same evaluators in the project so results are recorded centrally and linked to the agent trace — for scale and for debugging:

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient(
    endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
)

# Submit a cloud evaluation against a dataset + evaluators (API shape varies by
# SDK version — see the docs). Results appear in the portal's Evaluations view,
# each linked to the underlying agent trace for debugging.

In the portal's Evaluations view, a low-scoring case links straight to its trace (Lab 10) — so you go from "this case failed" to "here's the tool call that caused it" in one click.

Step 4 — Compare versions and gate

When you change the agent, the real question is "is v2 better than v1?" Run both over the same dataset and compare; then gate, like a test:

BASELINE = 4.0  # groundedness must average at least this (1–5 scale)

score = result["metrics"]["groundedness.gpt_groundedness"]  # key varies by version
assert score >= BASELINE, f"Quality regression: {score:.2f} < {BASELINE:.2f}"
print(f"Passed eval gate at {score:.2f}.")

Wire this into CI and a change can't ship unless it clears the bar — quality enforced, not hoped for.

Cost & cleanup

• AI-assisted evaluators make model calls — cost scales with dataset size and evaluator count. Keep eval sets focused.
• Hosted evaluation results live in the project; delete runs you don't need.
• Confirm in Cost Management that nothing unexpected is accruing.

Verified against

• Microsoft Foundry evaluation docs (Evaluators catalog, agent evaluators, hosted eval) and the `azure-ai-evaluation` SDK, June 2026.
• Evaluator names, input fields, and result keys vary by SDK version; code is syntax-checked against the docs but not executed during authoring. Pin your version and inspect the result keys for your run.

What's next

You can measure quality and gate on it. The next block makes the agent safe and governed:

• Lab 7 — Content Safety & Prompt Shields: apply platform guardrails to inputs and outputs — prompt-injection protection, content filtering, PII detection.
• Evaluation proves quality; safety and identity (Labs 7–8) make the agent trustworthy to put in front of real users.