Introducing AI Observability: Debug Agents with Full App and Infra Context
TL;DR
AI Observability is live in SigNoz Cloud. It’s OpenTelemetry-native. You can now track your agent runs and LLM signals next to your logs, services and infrastructure. Here are the highlights:
- AI Observability dashboard: track a new class of LLM signals, like token usage, cost, latency, errors and tool calls.
- Thread view: see a complete agent run as a conversation, with the input and output messages at each step.
- Correlated debugging: connect LLM signals with your services, logs, databases and infra metrics to debug agents across the whole workflow.
- LLM-aware trace view: LLM calls are labeled in the trace, and the span card shows input and output messages and token usage up front.
- Attribute mapping: map your existing instrumentation, from any framework or model provider, to the OpenTelemetry GenAI conventions (
gen_ai.*). - Custom model pricing: model costs update daily from OpenRouter, and you get an option to set your own.

Our applications are changing. It’s no longer services calling other services. We are embedding more and more agents and LLM calls inside our application workflows.
Gartner expects 40% of enterprise applications to include task-specific AI agents by the end of 2026, up from less than 5% in 2025. By 2028, it expects a third of user experiences to shift from native apps to agentic front ends (Gartner).
At its core, observability is seeing your entire system, every request, every service, every hop, through the data it already emits.
With agents in the loop, what “the system” is, and how a request flows, looks radically different. As software changes, observability will shift with it. That’s where AI observability comes in.
We need a unified portal, not multiple portals… where my logs, traces, and metrics are covered. It's not only logs and all, AI observability is also one part…And I want all this in one place. - E2E Networks
How AI agents are changing observability: 4 key shifts
-
LLM calls are non-deterministic and come at a cost.
For the same request, you may get 2 different answers, 2 different paths to answer, and 2 different set of tool calls. Plus, LLM calls bring their own class of signals.
In LLM monitoring, you need to track token usage, cost, errors, models, tool calls, and latency. You also need to evaluate the agent’s reasoning, using the input and output messages and quality checks.
-
Agentic workflows turn traces from lines into trees.
A traditional trace is a single chain - API, service, database. An agent breaks that shape. One agent call spreads into parallel lines of LLM calls, tool calls, sub-agents, and more.
That brings new debugging questions. Is the agent stuck in a loop? Is it taking the right path to the answer? This is making us rethink how to interact with traces.
-
Agents write the code. Production becomes the final review.
Google says about 75% of its new code is now AI-generated, up from about 25% in October 2024 (TechSpot). As agents write more of our code, the volume of code is exploding.
No team can read and review all of it. Production becomes the final test. Observability’s job expands from keeping systems running to checking whether the code is working as intended.
-
Observability is the feedback signal in the loop.
Building on the previous point, software development is heading toward a loop: agents write code, review it, test it, ship it, and monitor it. On top of that, there are agents running inside the application, serving users.
This loop needs a feedback signal to improve. Observability is that signal, bringing real production insight back to the humans and agents in the loop.
AI Observability is our first big step on this path.
AI Observability in SigNoz Cloud
We’re launching AI observability in SigNoz Cloud. Monitor your agent runs and LLM calls alongside your logs, services and infrastructure.
AI observability can’t live on a separate platform, siloed from the rest of your application and infrastructure data. When you debug an agent, the issue may be in the agent itself, or outside it. It could be a tool, a database or the infrastructure underneath.
We’ve heard from teams who hop between an AI observability tool and SigNoz Cloud or another observability platform, lining up timestamps by hand to piece one request back together.
With SigNoz Cloud, all your LLM signals sit right next to your logs, services and infrastructure. No more tab-hopping. And of course 😉, it’s OpenTelemetry-native - no vendor lock-in, no proprietary SDK, and it works with any framework.
Key features of SigNoz Cloud AI Observability
AI Observability Dashboard to track all your LLM signals
You’ll find a new AI Observability section in the left navigation. It opens a ready-made dashboard that tracks every LLM and agent signal.
It includes charts for,
- Token usage - tokens split into input, output and cached.
- Cost - total spend and cost over time.
- Latency - p50 to p99 latency per model for every inference request, plus time to first token.
- Errors - grouped by type, like rate limit, timeout and auth.
- Tool calls - call rate, error rate and duration for each tool.
You can filter every chart by model, provider, environment or service.
See the telemetry requirements for the attributes each panel needs.

The Explorer tab also lists every AI trace in your system. Each trace gets roll-up totals for tool calls, tokens, cost and more, like trace.total_tokens.
You see these totals next to each trace in the Explorer table. Use the same fields in your queries and filters. For example, find every trace over 10K tokens.

End-to-end agent tracing with LLM-aware spans
We’ve updated the trace view and span card to make LLM calls and AI workloads easier to read.
First, as all your signals are in one place, you can see the complete trace. Trace a request from your web app, through your backend, into the inference service, and see exactly where it slowed down or failed.
The SigNoz Cloud trace view already supports unlimited spans, so large AI traces load in full, with no span limits. The LLM filter isolates every LLM call in even the densest trace.
From LLM spans, you can jump to logs and infra around them.
On top of that, we’ve added:
- AI span labels: All AI spans get a label, so they stand out in the waterfall.
- Input and output messages: the span card for an LLM call shows what went into the model and its response.
- Token usage and cost up front: both sit at the top of the span card for every LLM call.

Follow agent reasoning step by step in the new thread view
The new thread view in trace details shows an AI trace as a conversation, so you can follow what an agent did at each step.
As we covered earlier, AI workflows are breaking the shape of a trace. The waterfall shows each step and when each span ran, but it doesn’t show the messages behind them.
The thread view lays out the full exchange in order:
- System prompt: the instructions the agent started with.
- Inputs and outputs: each message sent to the model and what it returned.
- Tool calls: each tool the agent called, with the arguments and the response it got back.
For example, a user asks your customer support agent to cancel their subscription, but the agent ends up pitching a discount. With each input and output in sequence, you can see exactly where the agent drifted off course.
In trace details, open the Thread tab next to Overview. When you open a trace from AI Observability, you land on the Thread tab directly.

Correlate LLM signals with your services, logs and infra
Not every agent failure starts in the agent or the LLM. It could be an overloaded pod, a slow database, a downstream endpoint that hangs, or the inference service itself.
SigNoz Cloud already has your application, service and infrastructure data. With AI Observability, your LLM and agent signals live in the same place, so you see the complete workflow your agents run in and the infrastructure they run on.
That means you can answer questions that span your agent runs, LLM signals, app and infra in one platform:
- Is my agent slow because of the inference service, or the storage or the pods behind it?
- Did these LLM timeouts start when the pods restarted?
- Which endpoint or customer is driving our token spend?
Set up dashboards and alerts on correlated signals. Put LLM latency next to pod CPU and memory, or alert when the pods behind your agent service start restarting, so you know before your agents fail.
Instrument any framework with attribute mapping
SigNoz Cloud AI Observability is built on the OpenTelemetry GenAI semantic conventions. It works with any framework that emits OTLP traces, with no SigNoz SDK or proprietary agent in your app.
However, not every library uses the same attribute names. OpenInference, custom SDKs and different model providers each follow their own conventions.
So we’ve added Attribute Mapping, which lets you keep your existing instrumentation and map its fields to the OpenTelemetry GenAI conventions (gen_ai.*). Once mapped, those fields power the dashboard, queries and alerts.
- Easily portable. Because everything is built on standard
gen_ai.*fields, you can switch frameworks or add a new provider with minimal changes. - Mappings run on ingest. Each mapping has one target
gen_ai.*attribute and a list of source attributes in priority order. For each span, SigNoz takes the first source that exists and writes it to the target. If the span already has the target, SigNoz leaves it alone. - Defaults cover common libraries. SigNoz ships default mappings for common LLM libraries, and you can add your own groups, scoped by conditions, for in-house instrumentation.

Set your own model pricing
SigNoz Cloud calculates the cost of every LLM call for you. It reads the input, output and cached token counts on each span. Then it applies the model’s price and writes the cost back to the span as signoz.gen_ai.usage.tokens.cost.
Prices for known models are filled in automatically. SigNoz Cloud updates them daily from OpenRouter. New models and price changes are picked up without any manual effort.
Set custom prices when the defaults don’t fit. You may be using a model with no public price, like a fine-tuned or self-hosted one. Or you may have negotiated a better rate with your provider.
In the Model pricing tab, set your own price per million tokens. Your cost dashboards, queries and alerts use it from then on.

Availability
AI Observability is available on SigNoz Cloud and self-hosted SigNoz. Self-hosted users with a custom collector config need to add the signozspanmapper and signozllmpricing processors to the traces pipeline. The earlier /llm-observability/* routes are now /ai-observability/*.
What’s next in AI Observability - Session View
Most of the agent workflows spans over multiple traces. So, we're building a session view that strings every trace in a conversation together, so you can follow the complete chain of agent worklow across the whole session.
Get started with SigNoz AI Observability
SigNoz is OpenTelemetry-native, with traces, logs, metrics and now your agents in one place. See every agent run alongside the services and infrastructure it runs on.
Start with SigNoz Cloud and get 30 days of unrestricted access to every feature. To send your first LLM traces, follow the AI Observability docs.
You can also run SigNoz as a self-hosted Enterprise edition or the open-source Community edition.