Observability & Cost

Before you start: this is a billed Azure service

• Cost and account. An Azure subscription, a Foundry project, an agent to observe, and an Application Insights resource connected to the project. Trace/log ingestion and retention have their own (small) cost.
• Authored against the docs, not executed live. Code is written against the official docs and syntax-checked, not run during authoring. Pin your SDK version and verify.
• Two languages, hybrid. Python-primary with .NET noted; you'll use the portal (Azure Monitor) and the SDK.

About this series

This is Part 10 of the twelve-part Microsoft Foundry series — the operations chapter. Your agent (now a multi-agent system) is built, grounded, equipped, safe, and governed. This lab is about running it in the open: seeing what it does, debugging across steps, and controlling what it costs.

What you'll build

You'll make the Acme system fully observable and cost-controlled:

1. Enable tracing so every run emits an OpenTelemetry trace.
2. Inspect a trace — read the spans: model calls, tool invocations, and sub-agent hops.
3. Build a dashboard and alerts in Azure Monitor / Application Insights.
4. Control cost — token visibility, budgets, and quotas.

Why this matters (the 5-minute business case)

Read this before handing the lab to your team.

A multi-agent system is a chain of model and tool calls — "why did it do that?" is unanswerable without tracing. When a normal service misbehaves you read logs. When an agent misbehaves you need the sequence of reasoning, tool calls, hand-offs, and responses that led there. Observability is what turns an opaque system into one you can operate.

Three reasons this is a leadership-level concern:

1. You can't run in production what you can't debug. A trace shows which agent ran, which tool it called, what came back, and where time and tokens went — across sub-agent hops and hand-offs, all through one OpenTelemetry pipeline. That's minutes-to-fix instead of a day of guessing.
2. FinOps for agents is a real, new cost center. Agents call models repeatedly and call tools that cost money; spend is usage-based and can surprise you. Token and latency metrics make cost legible per agent; budgets and quotas keep it from running away. (This is the agent-era version of "measure real token spend, not vibes.")
3. Observability is also your audit trail. Traces and logs answer "what did the system do, and when?" for security, compliance, and incident review — the operational complement to the identity and governance from Lab 8.

The honest caveats:

• Capturing prompt/response content is powerful and sensitive. Traces can include the actual inputs/outputs — invaluable for debugging, a privacy question for sensitive data. Enable content capture deliberately, with retention and access controls.
• Observability has its own cost. Ingestion and retention bill; high-volume tracing adds up. Use sampling and sensible retention.
• Dashboards tell you *what*, traces tell you *why*. Watch metrics for trends; reach for traces to diagnose a specific bad run. You need both.

A framing for a non-technical stakeholder: "This is how we see what our agents do and what they cost — trace each request to debug problems, watch dashboards for token spend and errors, alert when something's wrong, and set budgets so costs don't surprise us."

Concepts you'll use (2-minute primer)

• Trace & spans — a trace is the timeline of one request; spans are units of work (a model call, a tool call, a sub-agent hop). Foundry emits these via OpenTelemetry.
• The three pillars — Azure Monitor / Application Insights for metrics, dashboards, and alerts, plus distributed tracing and logs.
• Content capture — opt-in inclusion of actual prompts/responses in traces (sensitive).
• Cost controls — budgets and alerts in Cost Management, plus model/API quotas.

One sentence: enable tracing, read spans to debug a run, watch Azure Monitor for trends, alert on the bad ones, and bound spend with budgets and quotas.

Prerequisites

Step 1 — Enable tracing

Connect your project's Application Insights and turn on OpenTelemetry. The agent's runs then emit traces:

import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.monitor.opentelemetry import configure_azure_monitor

project = AIProjectClient(
    endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
)

# Wire traces/logs to the project's Application Insights:
conn = project.telemetry.get_connection_string()
configure_azure_monitor(connection_string=conn)

# Enable prompt/response CONTENT capture only when you intend to (sensitive):
# os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"

Start with tracing without content capture; add content capture deliberately — it records the actual prompts and responses.

.NET note: the same wiring uses Azure.Monitor.OpenTelemetry.AspNetCore / the OpenTelemetry SDK with the project's Application Insights connection string.

Step 2 — Inspect a trace

Send the agent a few requests, then open Application Insights → Transaction search / traces (or the project's tracing view) and pick a run. Read its spans:

• The first span is the overall request.
• Child spans show each step — model calls, tool calls, and sub-agent hops (the concierge → returns-specialist delegation from Lab 9), with durations and token usage.

This is where "why did it answer that?" becomes answerable. A slow run has one fat span that explains it; a wrong answer has a tool or sub-agent span with surprising input or output.

Step 3 — Dashboards and alerts

In Azure Monitor, build a dashboard over the metrics that matter and set alert rules:

• Token consumption — your primary cost driver.
• Latency (p95) — against your SLO.
• Error rate — failed runs.
• Sub-agent hops / tool calls — a spike can mean a loop or mis-routing.

Route alerts to the channel your on-call actually watches. An alert into an unread inbox is no alert.

Step 4 — Control cost

Observability tells you what you're spending; these keep it bounded:

• Token visibility per agent — review weekly so spend stays legible.
• Budgets and alerts — set a Cost Management budget with thresholds (50/90/100%).
• Quotas — set model/API quotas to cap the blast radius of a runaway loop or bad deploy. A quota is a ceiling a bug can't exceed.

Agents make usage-based spend easy to rack up; these three make it impossible to be surprised by.

Cost & cleanup

• Trace/log ingestion and retention bill — set sensible retention and use sampling for high-volume agents.
• Delete test dashboards and alert rules you don't need.
• Delete test agents and the Application Insights resource if they were only for this lab.
• Confirm in Cost Management that nothing is still accruing.

Verified against

• Microsoft Foundry observability docs (OpenTelemetry tracing, Application Insights, content-capture setting) and the `azure-ai-projects` / azure-monitor-opentelemetry SDKs, June 2026.
• Telemetry setup, the content-capture env var, and metric names vary by SDK version; code is syntax-checked against the docs but not executed during authoring. Pin your version.

What's next

Your system is observable, alertable, and affordable. Two labs remain:

• Lab 11 — Fine-tuning & model customization: when (and how) to customize a model rather than prompt or ground.
• Lab 12 — Interoperability capstone: MCP + A2A + Microsoft 365 publishing — managed and open.