Routing Policies

Overview

Routing policies in SigNoz allow you to dynamically route alerts to different notification channels based on alert labels using custom expressions. Instead of hardcoding alert destinations, you can create flexible rules that automatically send alerts to the appropriate teams or channels based on conditions like service name, threshold, environment, or any other alert label.

Types of Labels in Routing Policy Expressions

Routing policies can use the following types of labels in their expressions:

1. User-Defined Static Labels

These are custom labels you define when creating alert rules. They remain constant for all alerts generated by that rule.

Examples:

• type: Categorize alerts (e.g., "application", "infrastructure", "business")
• team: Assign ownership (e.g., "payments", "platform")
• sla: Business impact classification (e.g., "customer-facing", "internal")

Usage in expressions:

type = "infrastructure" AND team = "platform"

2. Platform-Provided Labels

SigNoz automatically adds these labels to provide context about the alert's state and configuration.

Available platform labels:

• threshold.name: The severity threshold that triggered the alert (e.g., "critical", "warning")
• alertname: Name of the alert rule

Usage in expressions:

threshold.name = "critical" AND type = "application"

3. Dynamic Query Labels

These labels come from the GROUP BY clause in your alert query. They represent the actual metric dimensions that triggered the alert and will vary with each alert instance.

Common examples:

• Kubernetes labels: k8s.pod.name, k8s.namespace.name, k8s.deployment.name
• Service labels: service.name, service.version
• Infrastructure labels: host.name, cloud.region, cloud.availability_zone
• HTTP labels: http.method, http.status_code, http.route
• Database labels: db.name, db.operation, db.system
• Custom labels: Any attribute you've added to your telemetry

Usage in expressions:

k8s.namespace.name = "production" AND service.name CONTAINS "api"
host.name IN ["prod-server-1", "prod-server-2"] AND cloud.region = "us-east-1"

Creating a New Routing Policy

To create a routing policy:

1. Navigate to Alerts > Configuration > Routing Policies
2. Click the "+ New routing policy" button
3. Fill in the following fields:

Required Fields

• Routing Policy Name: A descriptive name for your policy (e.g., "Checkout team critical alerts")
• Expression: The condition that determines when this policy applies (e.g., service.name = "checkout" AND type = "application" AND threshold.name = "critical")
• Notification Channels: Select one or more channels where matching alerts should be sent

Optional Fields

• Description: Additional context about the policy's purpose

4. Click "Save Routing Policy" to create the policy

Managing Routing Policies

The main routing policies page displays all your configured policies with:

• Policy name (click to expand details)
• Creation and update metadata
• Expression syntax
• Associated notification channels
• Quick actions (edit, delete)

You can search policies by name using the search bar.

Editing a Routing Policy

To modify an existing policy:

1. Click the edit icon (✏️) next to the policy
2. Update any fields as needed
3. Save your changes

Note: Changes take effect immediately for new alerts. Existing active alerts continue using their original routing.

Deleting a Routing Policy

To remove a policy:

1. Click the delete icon (🗑️) next to the policy
2. Confirm deletion in the dialog

How Routing Policies Work

When an alert fires, SigNoz evaluates it against all enabled routing policies:

1. Label Extraction: Alert labels are extracted from the alert instance
2. Expression Evaluation: Each policy's expression is evaluated using the alert's labels
3. Channel Selection: Matching policies' channels are collected
4. Notification Dispatch: Alerts are sent to all matched channels

Routing Process Example

Consider an alert with these labels:

type: "application"
threshold.name: "critical"
deployment.environment: "production"
team: "checkout"

This alert would match policies with expressions like:

• team = "checkout"
• threshold.name = "critical" AND deployment.environment = "production"
• team = "checkout" AND type = "application"

Expression Syntax

Routing policy expressions support the subset of the search syntax:

Basic Operators

• Equality: = (e.g., service.name = "api")
• Inequality: != (e.g., environment != "development")
• Contains: CONTAINS (e.g., service.name CONTAINS "auth")
• Regular expression: REGEXP (e.g., service.name REGEXP "auth")
• In: in (e.g., severity in ["critical", "warning"])

Logical Operators

• AND: AND (both conditions must be true)
• OR: OR (at least one condition must be true)
• Parentheses: () for grouping

Detailed Example

Let's walk through a complete example with multiple services and thresholds:

Example

You have:

• Services: checkout, payment, inventory, user-auth
• Environments: production, staging, development
• Threshold levels: critical, warning, info
• Notification channels:
  • checkout-pagerduty
  • payment-pagerduty
  • platform-oncall
  • general-slack
  • ops-email

Sample Routing Policies

Policy 1: Critical Production Alerts

Name: Production Critical Escalation
Expression: deployment.environment = "production" AND threshold.name = "critical"
Channels: [ops-email, platform-oncall]

Policy 2: Checkout Service Alerts

Name: Checkout Team Alerts
Expression: service.name = "checkout" AND threshold.name IN ["critical", "warning"]
Channels: [checkout-pagerduty]

Policy 3: Payment Service Monitoring

Name: Payment Service All Envs
Expression: service.name = "payment" AND (threshold.name = "critical" OR (threshold.name = "warning" AND deployment.environment = "production"))
Channels: [payment-pagerduty]

Policy 4: General Development Alerts

Name: Dev Environment Notifications
Expression: deployment.environment = "development"
Channels: [general-slack]

Policy 5: Infrastructure Alerts

Name: Infrastructure Critical
Expression: type = "infrastructure" AND threshold.name = "critical"
Channels: [platform-oncall, ops-email]

Alert Routing Examples

Alert 1: Checkout service critical error in production

Labels:
  service.name: "checkout"
  deployment.environment: "production"
  threshold.name: "critical"
  type: "application"
  team: "checkout"
  error_type: "timeout"

Matched Policies: Policy 1, Policy 2
Channels notified: ops-email, platform-oncall, checkout-pagerduty

Alert 2: Payment service warning in staging

Labels:
  service.name: "payment"
  deployment.environment: "staging"
  threshold.name: "warning"
  type: "application"
  team: "payment"

Matched Policies: None (Policy 3 requires production for warnings)
Channels notified: None (falls back to default routing)

Alert 3: Database infrastructure critical alert

Labels:
  service.name: "postgres-primary"
  deployment.environment: "production"
  threshold.name: "critical"
  type: "infrastructure"
  db.name: "orders_db"

Matched Policies: Policy 1, Policy 5
Channels notified: ops-email, platform-oncall

Alert 4: Inventory service info in development

Labels:
  service.name: "inventory"
  deployment.environment: "development"
  threshold.name: "info"
  type: "application"
  team: "inventory"

Matched Policies: Policy 4
Channels notified: general-slack