The Trace Details interface in SigNoz offers a powerful and intuitive interface to visualize and analyze the journey of a request through your system. By displaying detailed trace information, it enables you to quickly diagnose performance issues, pinpoint errors, and understand the intricate flow of operations in your systems. It is engineered to handle massive trace data supporting more than a million spans per trace making it an ideal solution for high-volume environments.
The Trace Details Page enables you to:
- Visualize the Entire Trace: Render large traces seamlessly using synchronized flame graphs and waterfall charts.
- Drill Down Into Details: Inspect individual spans to view metadata, span percentile, context, attributes and events associated with each span.
- Efficient Navigation and Search: Use the dynamic search bar and filtering options to quickly jump to specific spans or error events making it easier to analyze and troubleshoot.
- Responsive and Scalable: Handle traces from your architectures (monolithic or microservices) without performance lag, ensuring smooth and real-time analysis.
Key Concepts
- Trace: A trace represents the complete path a request takes through your system, helping you understand its flow across various components and services.
- Span: Spans are the building blocks of a trace, representing individual units of work performed by an application or service during the request's lifecycle. Key components of a span include:
- Span Context: Metadata that links spans across services, enabling distributed tracing and hierarchical structuring.
- Span Attributes: Key-value pairs that store additional metadata about the operation performed by the span.
- Span Links: References that establish causal relationships between spans, useful for correlating asynchronous operations or background tasks with the main trace.
- Span Events: Structured log messages that record important events at specific timestamps during the span's execution.
Opening a Trace
Select a trace from the Trace Explorer to open the Trace Details page. You will see two synchronized views:
- Flame Graph — displays a hierarchical representation of the trace, showing the nested relationships between spans.
- Waterfall Chart — provides a detailed timeline view, illustrating the sequence and duration of each span.
Inspecting a Span
Click on a specific span in either the flame graph or waterfall view to inspect detailed information, including:
- Span context and attributes — key-value metadata attached to the span.
- Span events — timestamped log messages recorded during the span.
- Span links — references to related spans across async operations.
- Span kind — client, server, producer, consumer, or internal.
Attributes
The span details panel displays all resource and span attributes as key-value pairs. Hover over any attribute value to see quick action icons that let you interact with the data directly:
- Pin Attribute — pin the attribute to the top of the list for easy reference
- Filter for Value — add the attribute as an
INfilter to the current query - Filter out Value — add the attribute as a
NOT INfilter to exclude matching spans - Group By Attribute — add the attribute to the Group By clause
- Copy Field Name — copy the attribute key to your clipboard
- Copy Field Value — copy the attribute value to your clipboard
Span Percentile
When you view a span's details, you can see its percentile compared to other spans from the same service over the past hour (for example, p50, p90, p95, p99). The percentile information may take a few seconds to appear after opening the span details. This helps you quickly identify performance outliers and understand where the span falls within the typical latency distribution.
Click on the percentile to view more details, including the exact p50, p90, and p99 values for the matching spans.
You can further refine this analysis by adding more resource attributes to narrow down the set of similar spans, or by increasing the time range to observe percentile behavior over a longer period.
Synchronized Navigation
The flame graph and waterfall views remain perfectly synchronized. Navigating in one view automatically updates the other, so you can always see where a span fits into the overall trace timeline.
Search and Filter
The Trace Detail interface allows you to filter spans for quick analysis while dynamically loading additional spans as you scroll, ensuring smooth navigation even in large traces.
For example, to debug errors:
- Enable the
has_error = truefilter to highlight error spans. - Click on an error span to view associated error messages and logs.
- Analyze the affected services and identify bottlenecks causing failures.
Adjusting the Interface
If span names or details appear cramped, use the draggable anchor to adjust the size of the display area. This ensures that you can comfortably read and analyze all information.
Related Signals
When you inspect a span's details, a Related Signals section appears at the bottom of the span details panel. This lets you jump directly to logs or metrics correlated to that specific span — without leaving the trace view.
Logs
Click Logs in the Related Signals section to open an inline panel showing log lines correlated to the span's time window and service. The panel shows log entries with timestamps and message bodies, and an Open in Logs Explorer button takes you to the full Logs Explorer pre-filtered to that trace context and timestamp.
Clicking Open in Logs Explorer opens the Logs Explorer pre-filtered to the relevant trace ID and timestamp, so you can search and analyze the logs in full detail.
Metrics
Click Metrics in the Related Signals section to see infrastructure metrics (Node CPU usage, Node memory usage, Pod metrics, etc.) for the host or pod that served this span. Use the Node and Pod tabs to switch between host-level and pod-level views.
