Log based alerts

A Log-based alert allows you to define conditions based on log data, triggering alerts when these conditions are met. Here's a breakdown of the various sections and options available when configuring a Log-based alert:

Step 1: Define the Log Metric

In this step, you use the Logs Query Builder to apply filters and operations on your logs to define conditions which triggers log based alert Some of the fields that are available in Logs Query Builder includes:

  • Logs: A field to filter the specific log data to monitor.

  • Aggregate Attribute: Allows you to select how the log data should be aggregated (e.g., "Count").

  • Group by: Provides options to group log data by various attributes, such as "service.name", "method" or custom attributes.

  • Legend Format: Lets you define the format for the legend in the visual representation of the alert.

  • Having: Apply conditions to filter the results further based on aggregate value.

Using Query Builder to perform operations on your logs
Using Query Builder to perform operations on your logs

Step 2: Define Alert Conditions

In this step, you define the specific conditions for triggering the alert, as well as the frequency of checking those conditions:

  • Send a notification when [A] is [above/below] the threshold [in total] during the last [X mins]: A template to set the threshold and define when the alert condition should be checked.

  • Alert Threshold: A field to specify the threshold value for the alert condition.

  • More Options :

    • Run alert every [X mins]: This option determines the frequency at which the alert condition is checked and notifications are sent.

    • Send a notification if data is missing for [X] mins: A field to specify if a notification should be sent when data is missing for a certain period.

Define the alert conditions
Define the alert conditions

Step 3: Alert Configuration

This step is for setting alert metadata like severity, description, and additional details:

  • Severity: Choose the severity of the alert (e.g., "Warning," "Critical").

  • Alert Name: A field to name the alert.

  • Alert Description: Add a detailed description of the alert, explaining its purpose and trigger conditions.

  • Labels: A field to add labels or tags to the alert for categorization. The label is a key-value pair to annotate the alert. Use key:value format to add labels.

  • Notification channels: A field to choose the notification channels from those configured in the Alert Channel settings.

  • Test Notification: A button to test the alert to ensure that it works as expected.

Configure the alert
Setting the alert metadata

Result labels in alert description

You can incorporate result labels in the alert descriptions to make the alerts more informative:

Syntax: Use {{.Labels.<label-name>}} to insert label values.

Example: If you have a query that returns the label service_name then to use it in the alert description, you will use {{.Labels.service_name}}which creates an alert that is specific to the particular service.

Examples

1. Alert when percentage of redis timeout error logs greater than 7% in last 5 mins

Here's a video tutorial for creating this alert:


Step 1: Write Query Builder query to define alert metric

logs builder query for redis timeout logs percentage
Redis timeout query

Here we write 2 queries to calculate error logs percent. First query to count logs which are redis timeout error logs. Second query to count total logs. Then we add a formula to calculate percentage.

error logs percentage chart
Error log percentage chart
✅ Info

Remember to select y-axis unit as Percent(0-100) as we want to apply threshold in percent.


Step 2: Set alert conditions

redis timeout alert condition
Error logs percentage alert condition

The condition is set to trigger a notification if the per-minute error logs percentage exceeds the threshold of 1 second on average in the last five minutes.

Step 3: Set alert configuration

redis timeout alert configuration
Error logs percentage alert configuration

At last configure the alert as Warning, add a name and notification channel.