Exceptions based alerts

An Exceptions-based alert in SigNoz allows you to define conditions based on exception data, triggering alerts when these conditions are met. Here's a breakdown of the various sections and options available when configuring an Exceptions-based alert:

Step 1: Define the Metric Using Clickhouse Query

In this step, you define the Clickhouse query to retrieve the exception data and set conditions for triggering the alert. The following elements are available:

• Clickhouse Query: A field to write a Clickhouse SQL query that selects and aggregates exception data. The query should define the exception type, time range, and other necessary conditions.

• Legend Format: An optional field to define the format for the legend in the visual representation of the alert.

• Having: Apply conditions to filter the results further based on aggregate value.

Step 2: Define Alert Conditions

In this step, you define the specific conditions for triggering the alert, as well as the frequency of checking those conditions. The condition configuration of an alert in SigNoz consists of 5 core parts:

Query

An alert can consist of multiple queries and formulas. But only 1 of them can be put into consideration while determining the alert condition.

You can define one or more queries or formulas to fetch the data you want to evaluate. However, only one of them can be used as the trigger for the alert condition.

For example:

• A = Total request count
• B = Total error count
• C = B / A (Error rate)

You can use query C as the evaluation target to trigger alerts based on error rate.

Condition

This defines the logical condition to check against the selected query’s value.

Match Type

Specifies how the condition must hold over the evaluation window. This allows for flexible evaluation logic.

Evaluation Window (For)

Specifies how long the condition must be true before the alert is triggered.

e.g. For 5 minutes = The condition must remain true continuously for 5 minutes before the alert is triggered.

This helps reduce false positives due to short-lived spikes.

Threshold

This is the value you are comparing the query result against.

e.g. If you choose Condition = Above and set Threshold = 500, the alert will fire when the query result exceeds 500.

Threshold Unit

Specifies the unit of the threshold, such as:

• ms (milliseconds) for latency
• % for CPU usage
• Count for request totals

Helps interpret the threshold in the correct context and also for correct scaling while comparing 2 values.

Advanced Options

In addition, there are 3 more advanced options:

Alert Frequency

• How frequently SigNoz evaluates the alert condition.
• Default is 1 min
• e.g. If set to 1 min the alert will run once every minute.

Notification for missing data points

• Triggers an alert if no data is received for the configured time period.
• Useful for services where consistent data is expected.
• E.g. If set to 5 minutes, and no metric data is received during that period, the alert will fire.

Minimum Data Points in Result Group

• Ensures the alert condition is evaluated only when there's enough data for statistical significance.
• Helps avoid false alerts due to missing or sparse data points.
• E.g. If set to 3, the query must return at least 3 data points in the evaluation window for the alert to be considered.

Step 3: Alert Configuration

In this step, you set the alert's metadata, including severity, name, and description:

Severity

Set the severity level for the alert (e.g., "Warning" or "Critical").

Alert Name

A field to name the alert for easy identification.

Alert Description

Add a detailed description for the alert, explaining its purpose and trigger conditions.

You can incorporate result attributes in the alert descriptions to make the alerts more informative:

Syntax: Use $<attribute-name> to insert attribute values. Attribute values can be any attribute used in group by.

Example: If you have a query that has the attribute service.name in the group by clause then to use it in the alert description, you will use $service.name.

Slack alert format

Using advanced slack formatting is supported if you are using Slack as a notification channel.

Labels

A field to add static labels or tags for categorization. Labels should be added in key value pairs. First enter key (avoid space in key) and set value.

Notification channels

A field to choose the notification channels from those configured in the Alert Channel settings.

Test Notification

A button to test the alert to ensure that it works as expected.

Examples

1. Alert when exception of type ConnectionError occurs

Here's a video tutorial for creating this alert:

• ClickHouse Query: Counts occurrences of 'ConnectionError' exceptions within one-minute intervals, grouped by service name. The ClickHouse Query would look like:

    SELECT 
        count() as value,
        toStartOfInterval(timestamp, toIntervalMinute(1)) AS interval,
        serviceName
    FROM signoz_traces.distributed_signoz_error_index_v2
    WHERE exceptionType !='ConnectionError'
    AND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
    GROUP BY serviceName, interval;

• Alert Threshold: Set to 0
• Alert Name: "Exceptions Alert"
• Severity: "Warning"
• Notification Channels: signoz-slack-alerts (Slack channel)