Triggered alerts history

Alerts History

The Alerts History feature helps you analyze and understand the behavior of your alerts over time. It provides insights into alert frequency, resolution times, and top contributing resources, enabling you to optimize your alerting rules and respond to incidents more effectively.

You can access the Alerts History by clicking the History button next to the Overview tab on any alert rule's detail page.

Overview

The Alerts History page provides three main sections:

1. Summary Metrics

• Total Triggered: Shows the total number of times alerts were triggered in the selected time period. An alert is counted as triggered when the alert condition is met. The subsequent triggers of the same alert without any resolution in between are not counted.
• Average Resolution Time: Displays how long it typically takes to resolve alerts. An alert is considered resolved when the alert condition is no longer met. The resolution time is calculated from the time the alert is triggered to the time it is resolved. For the multi-instance alerts, the resolution time is the total time taken to resolve all instances of the alert. The individual instance resolution time is not considered for this metric.
• Top Contributors: Lists the most frequent sources of alerts (e.g., specific services, hosts, or endpoints)

2. Timeline View

The timeline provides a visual representation of alert states over time:

• Green segments indicate normal operation
• Red segments show when alerts were firing
• You can toggle between "Overall Status" and "Top 5 Contributors" views
• Filter options: All, Fired, and Resolved states

3. Events Table

A detailed list of all alert events showing:

• Current state (Firing/Resolved)
• Associated labels and metadata
• Timestamp information
• Available actions for each event. If available, you can navigate to the related logs, traces, or metrics directly from the alert events.

Using Alert History

Time Range Selection

1. Use the time range selector in the top-right corner to choose your desired time period
2. The "Reset" button (when enabled) returns you to the default time range
3. Last selected time range is automatically saved for future sessions

Analyzing Alert Patterns

1. View the summary metrics to understand:

   • Alert frequency trends (compared to previous periods)
   • Resolution time patterns
   • Most problematic sources

2. Use the timeline to:

   • Identify patterns in alert occurrences
   • Spot correlation between different alert instances
   • Understand the duration of alert states

3. Filter and search capabilities:

   • Use the search bar to filter events by specific criteria
   • Apply state filters (All/Fired/Resolved) to focus on relevant events
   • Sort by different columns in the events table

Investigating Alert Details

1. Click on a specific event in the table to view detailed information
2. Use the "View all" option in Top Contributors to see the complete list
3. Navigate to related logs, traces, or metrics directly from alert events

Best Practices

1. Regular Review

   • Monitor alert frequency and patterns regularly
   • Look for recurring issues in top contributors
   • Analyze resolution times to identify areas for improvement

2. Alert Optimization

   • Use historical data to adjust alert thresholds and conditions to avoid frequent triggers
   • Identify and address frequently triggering alerts
   • Focus on top contributors for system improvements

3. Incident Analysis

   • Use the timeline view to understand the sequence of events
   • Cross-reference with other monitoring data
   • Document patterns for future reference

Related Resources

• Alert Management Guide
• Creating New Alerts
• Alert Notification Channels