Microsoft Agent Framework Dashboard

This dashboard provides a comprehensive view of applications built on Microsoft Agent Framework (Python) using trace data. It is built on the gen_ai.* span attributes the framework emits through configure_otel_providers(), plus its workflow.* and executor.* workflow spans, and covers agent runs, token usage per model and per agent, latency percentiles, tool activity, workflows, and errors.

Every panel is scoped through two template variables, $service_name and $environment, so one dashboard serves every agent service you run. Select the services you want rather than leaving the picker on ALL, which applies no filter at all and mixes in every other service in the workspace.

Dashboard Preview

What This Dashboard Monitors

This dashboard tracks the trace data Microsoft Agent Framework emits to help you:

• Track Token Usage: Break down input, output, reasoning, and cache-read tokens per model and per agent, without the double counting that summing every span produces.
• Compare Models: See call volume and token totals for every served model, including the dated snapshot the provider actually used.
• Monitor Latency: Watch p50, p95, and p99 for model calls, and p95 per agent and per workflow, to separate a slow model from an agent taking too many turns.
• Understand Agent Activity: Count agent runs, see errors and average latency per agent, and read finish reasons to spot agents looping through tools.
• Watch Tool Behavior: Track calls and failures per tool. Tool failures never reach the agent span, so this is the only place they show up.
• Follow Workflows: See runs and duration per workflow, and every executor step inside sequential, concurrent, and handoff workflows.
• Catch Errors Early: Separate tool, model, and agent error rates, see errors by type over time, and read the failure text on recent errored spans.

Panels Included

Overview (Top Row)

Token panels read chat spans only. Each invoke_agent span repeats the summed usage of its chat spans, so a query that sums every span reports double the real figure. Agents run inside a workflow count once each, so a three-agent concurrent workflow adds three to Agent Runs and one to Workflow Runs.

Tokens and Models

• Token Usage Over Time: Input, output, reasoning, and cache-read tokens over time. Input dominates because each turn resends the conversation and tool results. The cache-read line is the part of input the provider served from its prompt cache at a discount, so a growing gap between it and input is a caching opportunity.
• LLM Calls by Model: Call volume grouped by gen_ai.response.model, which carries the dated snapshot the provider served, so a silent model upgrade shows up as a new series.
• Tokens by Model: Calls, input, output, and cache-read tokens per served model, the figures to multiply against your own per-model rates.
• Tokens by Agent: Runs and token totals per agent, read from invoke_agent spans. This is where to find the agent that dominates spend, and input tokens per run points at agents with long instructions or growing context.

Latency

• LLM Call Latency: p50, p95, and p99 of chat span duration. A p99 that spikes while p50 stays flat usually means timeouts or rate-limit retries on a few calls rather than a slower model.
• Agent Latency (p95): p95 end-to-end duration per agent, including its model and tool calls. Agents with booking or search tools sit well above chat-only agents, so compare each agent to its own baseline.
• Workflow Latency (p95): p95 duration per workflow. A concurrent workflow is as slow as its slowest agent, and a handoff workflow varies with how many turns the specialist agent takes.
• Finish Reasons: Why each model call ended. An even split between tool_calls and stop is the normal shape for tool-using agents. A growing length slice means responses are being cut off at the token limit.

Agents, Tools and Workflows

• Agents: Runs, errors, and average latency per agent. Errors here are model call failures only, because tool failures leave the agent span clean.
• Tools: Calls, failures, and average latency per tool. A flaky backend behind a tool shows up here even when every agent run succeeds.
• Tool Calls Over Time: Tool invocations per tool. A sustained climb on one tool without a matching rise in Agent Runs is the usual early sign of an agent looping on a tool instead of answering.
• Workflows: Runs and average duration per workflow. Workflow spans carry no token counts, so a workflow's cost is the sum of its agents in Tokens by Agent.
• Workflow Executors: Every executor step inside workflows, both agents and built-in steps such as input-conversation, dispatcher, and aggregator. Handoff executors show more steps than their agent has runs, because each handoff briefly visits every participant.

Errors

• Tool Error Rate: Share of tool executions that raised an exception. The agent usually recovers and answers, which is why this rate needs its own panel.
• LLM Error Rate: Share of model calls that failed with a rate limit, server error, or timeout. Each of these also fails the parent agent span, and the workflow around it.
• Errors by Type: Errored spans over time by error.type and operation. A model failure appears on both the chat span and its invoke_agent parent, so the two ChatClientException series overlap. Read the chart per operation rather than summing it.
• Recent Errors: The latest errored spans with their error class and message. A failed chat span is followed by its invoke_agent parent at the same timestamp, and tool errors such as ConnectionError appear on their own.