Gen AI Evaluation: Measure and Gate Quality

Before you start: this is a billed cloud service

• Cost and account. A Google Cloud project with billing enabled and the Vertex AI APIs on. Model-based metrics use an LLM judge, so each evaluation run makes model calls and incurs cost proportional to your dataset size. Start with a small dataset.
• Authored against the docs, not executed live. The Python is written against the official docs and syntax-checked, not run during authoring. The evaluation SDK evolves — pin your SDK version and adjust metric/field names if yours differs.
• SDK-first.

About this series

This is Part 7 of the eleven-part Gemini Enterprise Agent Platform series. The last two labs improved the agent — memory and curated examples. This lab answers the question those raise: is it actually better? You'll use the platform's Gen AI Evaluation service to measure quality with managed metrics and gate ship/no-ship on evidence instead of a hunch.

What you'll build

You'll build a real evaluation loop for the Acme assistant:

1. Define an eval dataset — representative questions with reference answers.
2. Run a pointwise evaluation with managed metrics (groundedness, answer quality) plus a computation-based check.
3. Read the results — both the summary scores and the per-row table with explanations.
4. Compare two versions of the agent head-to-head with a pairwise metric.
5. Gate — turn a metric threshold into an automated ship/no-ship check.

By the end you'll have a repeatable way to prove a change made the agent better — and to catch the changes that quietly made it worse.

Why this matters (the 5-minute business case)

Read this before handing the lab to your team.

You cannot manage what you cannot measure, and "it seems better" is not measurement. Every change you've made in this series — grounding, memory, examples — should help, but intuition about LLM quality is unreliable, and a prompt tweak that fixes one case routinely breaks three others you didn't check. Evaluation is the instrument that turns agent quality from a vibe into a number you can defend.

Three reasons this is a leadership-level concern:

1. Evidence-based ship/no-ship. Without evals, "is this good enough to ship?" is settled by whoever is most confident in the room. With them, it's settled by a score against a representative dataset. That's the difference between shipping on data and shipping on hope — the same argument the standalone Evals lab makes, now as a managed service.
2. Regression protection at scale. Models, prompts, and providers drift. An eval you run on every change is a safety net that catches the silent degradations — the ones that don't show up until a customer complaint. This is how you move fast without breaking quality.
3. Managed metrics beat rolling your own. Good evaluation needs judge models, calibrated metric prompts, and aggregation. The platform ships those — groundedness, coherence, answer quality, safety, plus custom and pairwise metrics — so your team measures quality instead of building a measurement framework.

The honest caveats:

• The dataset is the hard part. Managed metrics are easy; a representative eval set is the real work. Twenty real cases from your actual workflow are worth more than a thousand synthetic ones — garbage in, confident-garbage out.
• LLM-judge metrics are estimates, not oracles. Model-based metrics are powerful but not perfect; calibrate them against human judgment on a sample before you trust a threshold blindly.
• Evaluation costs money. Every judged run makes model calls. Keep eval sets focused, and run the full suite on meaningful changes rather than every keystroke.

A framing for a non-technical stakeholder: "Evaluation is how we prove the agent is good enough before it ships and stays good after. We score it against a set of real questions, gate releases on the score, and catch regressions automatically — instead of finding out from customers."

Concepts you'll use (2-minute primer)

• Eval dataset — a table of cases. For pointwise evaluation, typically the prompt/instruction, the agent's response, and a reference (ground-truth) answer.
• `EvalTask` — the orchestrator: you give it a dataset and metrics, call evaluate(), and get results.
• Pointwise metrics — score each response on its own (groundedness, coherence, answer quality, safety). Some are model-based (an LLM judge scores them); some are computation-based (e.g., exact match).
• Pairwise metrics — compare two responses head-to-head and pick a winner. This is how you tell whether v2 actually beats v1.
• Results — summary_metrics (aggregated scores) and metrics_table (per-row scores with explanations, so you can see why a case failed).

One sentence: score a representative dataset with managed metrics, read the summary and the per-row explanations, compare versions pairwise, and gate releases on the numbers.

Prerequisites

Step 1 — Define an eval dataset

A dataset is just a table. Start small and real — a handful of representative Acme questions, each with the answer a knowledgeable human would give:

import pandas as pd

dataset = pd.DataFrame({
    "prompt": [
        "Can I return opened headphones after 20 days?",
        "How long is standard shipping?",
        "Is accidental damage covered under warranty?",
    ],
    "response": [   # the agent's answers (precomputed, or generated by a model)
        "Opened electronics are returnable within 15 days, so 20 days is past the window.",
        "Standard shipping is 3–5 business days, free over $50.",
        "Yes, accidental damage is fully covered.",   # <- wrong on purpose
    ],
    "reference": [  # ground truth
        "Opened electronics: 15-day return window with a 10% restocking fee.",
        "Standard shipping: 3–5 business days; free on orders over $50.",
        "The 1-year warranty covers manufacturing defects, not accidental damage.",
    ],
})

That third row is wrong on purpose — a good eval set should contain cases your agent can fail, so the metrics have something to catch.

Step 2 — Run a pointwise evaluation

Point an EvalTask at the dataset with a mix of managed model-based metrics and a computation-based one:

import vertexai
from vertexai.evaluation import EvalTask, MetricPromptTemplateExamples

vertexai.init(project="your-project-id", location="us-central1")

eval_task = EvalTask(
    dataset=dataset,
    metrics=[
        MetricPromptTemplateExamples.Pointwise.GROUNDEDNESS,
        MetricPromptTemplateExamples.Pointwise.QUESTION_ANSWERING_QUALITY,
        "exact_match",  # computation-based
    ],
    experiment="acme-agent-eval",
)

result = eval_task.evaluate()

The model-based metrics (groundedness, answer quality) are scored by an LLM judge; exact_match is a deterministic string check. Mixing both gives you a fuller picture than either alone.

Step 3 — Read the results

# Aggregated scores across the dataset:
print(result.summary_metrics)
# e.g. {"question_answering_quality/mean": 0.78, "groundedness/mean": 0.67, ...}

# Per-row scores WITH explanations — this is where you learn why a case failed:
print(result.metrics_table)

The summary_metrics tell you the headline; the metrics_table is where the value is — it shows each case, its score, and the judge's explanation. The warranty row should score low on groundedness, with an explanation pointing at the wrong claim. That's the loop: a number tells you something changed, the explanation tells you what.

Step 4 — Compare two versions head-to-head

When you change the agent (a new prompt, grounding, examples from Lab 6), the real question is "is v2 better than v1?" A pairwise metric answers it directly:

from vertexai.evaluation import EvalTask, MetricPromptTemplateExamples

# Dataset has both the candidate `response` (v2) and a `baseline_model_response` (v1).
pairwise_task = EvalTask(
    dataset=versions_dataset,
    metrics=[MetricPromptTemplateExamples.Pairwise.QUESTION_ANSWERING_QUALITY],
    experiment="acme-agent-v1-vs-v2",
)
result = pairwise_task.evaluate()
print(result.summary_metrics)  # win rate of v2 over v1

A pairwise win rate is the cleanest evidence for a release decision: not "v2 scored 0.81," but "v2 beat v1 on 78% of cases." That's the number to put in front of a stakeholder.

Step 5 — Gate ship/no-ship on the score

Turn the metric into an automated gate, exactly like a test:

BASELINE = 0.75  # the bar a release must clear

score = result.summary_metrics["question_answering_quality/mean"]
assert score >= BASELINE, f"Quality regression: {score:.2f} < {BASELINE:.2f}"
print(f"Passed eval gate at {score:.2f}.")

Wire this into CI and a change can't ship unless it clears the bar — the same "evals as code" discipline as the standalone lab, backed by managed metrics. Now quality is enforced, not hoped for.

Cost & cleanup

• Evaluation runs make model calls (the LLM judge) — cost scales with dataset size and metric count. Keep eval sets focused.
• Evaluation results are logged to a Vertex AI experiment; delete experiments/runs you don't need from the console.
• There's no long-running resource to delete here, but confirm in Billing → Reports that nothing unexpected is accruing.

Verified against

• Vertex AI Gen AI Evaluation service docs (EvalTask, MetricPromptTemplateExamples, pointwise/pairwise metrics) and the `google-cloud-aiplatform[evaluation]` SDK, June 2026.
• Metric names, dataset field names, and result keys vary by SDK version; code is syntax-checked against the docs but not executed during authoring. Pin your version and inspect summary_metrics keys for your run.

What's next

You can now measure quality and gate on it. The next block of the series is about making the agent safe and governed:

• Lab 8 — Model Armor & safety filters: apply platform-level guardrails — prompt-injection protection, content filtering, data-loss controls — as a managed layer.
• Cross-links: the standalone Evaluating Agents lab (evals as code, by hand) and ADK Lab 4 (evaluation & safety). This lab is the managed eval service; that's the same idea you own end to end.