Observability, Tracing & Cost

Before you start: this is a billed cloud service

• Cost and account. A Google Cloud project with billing enabled, a deployed agent (Lab 2), and the Telemetry API (traces) and Logging API (logs) enabled. Trace/log ingestion and retention have their own (small) cost.
• Authored against the docs, not executed live. The Python and config are written against the official docs and checked, not run during authoring. Telemetry env vars and trace APIs evolve — pin your SDK version and verify.
• Hybrid: console + SDK.

About this series

This is Part 10 of the eleven-part Gemini Enterprise Agent Platform series — the operations chapter. Your agent is built, grounded, equipped, remembered, evaluated, guarded, and governed. This lab is about running it in the open: seeing what it does, debugging across steps, and controlling what it costs. An agent you can't observe or afford isn't production-ready, however good it is.

What you'll build

You'll make the deployed Acme assistant fully observable and cost-controlled:

1. Enable tracing and telemetry so every request emits a trace.
2. Inspect a trace — read the spans to see exactly what the agent did, step by step.
3. Use the built-in dashboard — token consumption, latency, error rate, and tool calls over time.
4. Analyze traces programmatically — pull them into a DataFrame for your own analysis.
5. Alert on the things that matter (errors, latency).
6. Control cost — token visibility, budgets, and quotas.

By the end you can answer the two questions every production owner gets: "what is this agent doing?" and "what is it costing us?"

Why this matters (the 5-minute business case)

Read this before handing the lab to your team.

An agent is a multi-step, non-deterministic system calling models and tools — which makes "why did it do that?" genuinely hard without tracing. When a normal service misbehaves you read the logs. When an agent misbehaves you need to see the sequence of reasoning, tool calls, and model responses that led there. Observability is what turns an opaque agent into one you can actually operate.

Three reasons this is a leadership-level concern:

1. You can't run in production what you can't debug. A trace — the timeline of spans for a request — shows which tool was called, what it returned, and where time and tokens went. That's the difference between fixing a production issue in minutes and guessing for a day. No tracing, no real production.
2. FinOps for agents is a new, real cost center. Agents call models repeatedly and call tools that cost money; spend is usage-based and can surprise you. The platform's dashboard gives you token consumption, latency, and error rates so cost is legible per agent — and budgets plus quotas keep it from running away. This is the agent-era version of the scoreboard argument: measure real token spend, not vibes.
3. Observability is also your audit trail. Traces and logs are how you answer "what did the agent do, and when?" for security, compliance, and incident review — the operational complement to the identity and governance from Lab 9.

The honest caveats:

• Capturing prompt/response content is powerful and sensitive. Telemetry can include the actual inputs and outputs — invaluable for debugging, a privacy and compliance question for sensitive data. Turn content capture on deliberately, with retention and access controls.
• Observability has its own cost. Trace/log ingestion and retention bill, and high-volume tracing adds up. Use sampling and sensible retention rather than capturing everything forever.
• Dashboards tell you *what*, traces tell you *why*. Watch the metrics for trends; reach for traces to diagnose a specific bad request. You need both.

A framing for a non-technical stakeholder: "This is how we see what our agents are doing and what they cost. We trace each request to debug problems, watch dashboards for token spend and errors, alert when something's wrong, and set budgets so costs don't surprise us."

Concepts you'll use (2-minute primer)

• Trace & spans — a trace is the timeline of one request; it's made of spans, each a unit of work (an LLM call, a tool call). Reading spans is how you see the agent's actual steps.
• The three pillars — Cloud Trace (request timelines, OpenTelemetry-based), Cloud Monitoring (metrics, dashboards, alerts), and Cloud Logging (logs).
• Telemetry env vars — GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY turns on traces and logs (without prompt/response content); OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT adds the actual prompts and responses.
• The built-in dashboard — Agent Engine ships one showing token consumption, latency, error rate, and tool calls over time.
• Cost controls — budgets and alerts (you saw these in the bootstrapped-startup lab) plus quotas to cap blast radius.

One sentence: enable telemetry, read traces to debug a request, watch the dashboard for trends, alert on the bad ones, and bound spend with budgets and quotas.

Prerequisites

Step 1 — Enable tracing and telemetry

You can turn on basic tracing at the app level (you saw enable_tracing=True back in Lab 2's local test). For a deployed agent, set the telemetry env vars when you deploy:

remote_app = client.agent_engines.create(
    agent=root_agent,
    config={
        "requirements": ["google-cloud-aiplatform[agent_engines,adk]"],
        "env_vars": {
            # Traces + logs, WITHOUT prompt/response content:
            "GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY": "true",
            # Add this ONLY if you intend to capture prompt/response content:
            "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT": "true",
        },
    },
)

Start with telemetry without content capture, and only add content capture deliberately — it records the actual prompts and responses, which is gold for debugging and a liability for sensitive data.

Step 2 — Inspect a trace

Send the agent a few requests, then open the Traces tab for your agent (in the console / Agent Engine, backed by Cloud Trace). Pick a request and read its spans:

• The first span is the overall request.
• Child spans show each step — model calls, tool calls — with their duration and often token usage.

This is where "why did it answer that?" becomes answerable. You can see it grounded (or didn't), which tool it called, what came back, and where the time went. A slow request usually has one fat span that explains it; a wrong answer usually has a tool span with surprising input or output.

Step 3 — Watch the built-in dashboard

Agent Engine ships a dashboard for your deployed agent that tracks, over time:

• Token consumption — your primary cost driver.
• Latency — p50/p95 of how long requests take.
• Error rate — failed requests.
• Tool calls — which tools fire and how often.

Watch these for trends, not single requests. A creeping token count or error rate is an early warning; a tool-call spike might mean the agent is looping. The dashboard is the "is everything healthy?" view; traces are the "what happened in this request?" view.

Step 4 — Analyze traces programmatically

For deeper analysis, pull traces with the Cloud Trace SDK and work with them as data:

import pandas as pd
from google.cloud import trace_v1

client = trace_v1.TraceServiceClient()

traces = client.list_traces(request={
    "project_id": "your-project-id",
    # add filters/time range as needed
})

rows = []
for t in traces:
    for span in t.spans:
        rows.append({
            "trace_id": t.trace_id,
            "span": span.name,
            "start": span.start_time,
            "end": span.end_time,
        })

df = pd.DataFrame(rows)
print(df.head())
# Now compute span durations, find the slowest tools, aggregate by day, etc.

Turning traces into a DataFrame lets you answer questions the dashboard doesn't — slowest tool by p95, token cost per conversation type, error clustering. This is how you go from watching metrics to actually optimizing the agent.

Step 5 — Alert on what matters

Use Cloud Monitoring to alert before users complain. Create alerting policies on the metrics that signal trouble:

• Error rate above a threshold — something's broken.
• Latency (p95) above your SLO — it's gotten slow.
• Token consumption spiking — a cost or looping problem.

Route alerts to the channel your on-call actually watches. An alert that fires into an unread inbox is the same as no alert.

Step 6 — Control cost

Observability tells you what you're spending; these keep it bounded:

• Token visibility per agent — the dashboard already gives you this; review it weekly so spend stays legible.
• Budgets and alerts — set a Cloud Billing budget with thresholds (50/90/100%), exactly as in the bootstrapped-startup lab. Budgets alert; for non-prod, wire a kill-switch.
• Quotas — set model/API quotas to cap the blast radius of a runaway loop or a bad deploy. A quota is a ceiling that a bug can't exceed.

Agents make usage-based spend easy to rack up; these three make it impossible to be surprised by.

Cost & cleanup

• Trace/log ingestion and retention bill — set sensible retention and use sampling for high-volume agents.
• Delete test alerting policies and dashboards you don't need.
• Delete the deployed agent (Lab 2 cleanup) if it was only for this lab.
• Confirm in Billing → Reports that nothing is still accruing.

Verified against

• Agent Engine observability docs (Cloud Trace / Monitoring / Logging, telemetry env vars, the built-in dashboard), the Cloud Trace SDK, and the `google-cloud-aiplatform[agent_engines,adk]` SDK, June 2026.
• Telemetry env var names, trace API shapes, and dashboard metrics vary by version; code is syntax-checked against the docs but not executed during authoring. Pin your version.

What's next

Your agent is now observable, alertable, and affordable. One lab remains — tying the managed platform to the open ecosystem:

• Lab 11 — Interoperability capstone: expose a platform-hosted agent over A2A and have it consume MCP and peer agents — managed and open, without lock-in.
• Cross-links: the bootstrapped-startup GCP lab (budgets, quotas, kill-switches) and the "scoreboard" post (measure real token spend, not vibes) — the same FinOps discipline, applied to agents.