What is a Prometheus rule? - A Comprehensive Guide

Last Updated: October 27, 202611 min read

A Prometheus rule tells the Prometheus monitoring system to evaluate a PromQL expression at regular intervals. These rules help you put frequently used calculations and alert conditions into your monitoring configuration, so you don't have to run the same queries by hand.

There are two types of rules. Recording rules save the results of a query as new time series, which can reduce repeated query work on dashboards. Alerting rules detect conditions that need attention and trigger alerts when those conditions are met. Together, they help you keep dashboards responsive and find problems without watching them constantly.

This guide covers how to configure and test both types of rules, how Prometheus alerts move through their lifecycle, and how to connect Prometheus to Alertmanager for notifications. It also covers ways to reduce alert noise and manage rules over time.

Understanding recording rules in Prometheus

Recording rules in Prometheus are designed to precompute frequently used or resource-intensive queries. They save the results as new time series, so you can query those results instead of repeatedly running the original complex query.

For example, create a file named rules.yml with this recording rule to calculate the request rate for each service:

groups:
  - name: service_requests
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total[5m]))

In this example, rate() calculates the average per-second increase in http_requests_total over five minutes for each input series. sum by (service) adds those rates across instances and other labels, leaving one result per service.

The recording rule saves that result as service:http_requests:rate5m. You can query this metric instead of repeating the full expression. If your metrics use different names or labels, update the expression accordingly.

The name service:http_requests:rate5m follows the level:metric:operations convention: service is the grouping level, http_requests identifies the source metric, and rate5m describes the calculation.

Alerting rules in Prometheus

An alerting rule defines a condition that Prometheus checks at regular intervals. When the condition is met, Prometheus creates an alert for each matching time series. If the condition remains active for the configured duration, the alert fires.

An alerting rule can include the following fields:

- alert: <alert_name>
  expr: <PromQL_expression>
  for: <duration>
  labels:
    <label_key>: <label_value>
  annotations:
    <annotation_key>: <annotation_value>

Example: Alert on high CPU usage

Add the following cpu_alerts group to rules.yml, below the service_requests group from the recording rule example. The rule alerts when an instance's average CPU usage stays above 90% for five minutes:

  - name: cpu_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High CPU usage detected on {{ $labels.instance }}"
          description: "CPU usage has been above 90% for five minutes on {{ $labels.instance }}."

The expression calculates the five-minute rate of idle CPU time, averages it across CPU cores for each instance, and subtracts that percentage from 100. The result represents non-idle CPU usage. If it remains above 90% across evaluations for five minutes, the alert fires for that instance. This example uses node_cpu_seconds_total, a metric exposed by Node Exporter.

Prometheus sends firing alerts to configured Alertmanager instances. Alertmanager can then group and route notifications. An alert can fire without a notification being sent immediately.

Use label matching to select services

You can use a regular expression in a label selector to check more than one job. For example, this expression finds failed scrapes for targets in either job:

up{job=~"important-service|critical-app"} == 0

A failed scrape tells you that Prometheus could not collect metrics from a target, but it does not reveal the cause. A target that disappears from service discovery may no longer have an up series, so this expression does not detect every missing target.

Combine alert conditions

You can combine conditions when an alert should fire only if both are true. For example, the following expression selects services with a 5xx error rate above 5% and a 90th-percentile request duration above one second:

(
  sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum by (service) (rate(http_requests_total[5m]))
  > 0.05
)
and on (service)
max by (service) (http_request_duration_seconds{quantile="0.9"}) > 1

The error-rate calculation uses http_requests_total with service and status labels. The duration expression uses a summary metric with a service label, that takes the highest reported 90th percentile among instances, rather than calculating a service-wide percentile.and on (service) matches these two conditions by service.

Before using the expression in your own setup, check the metric names and labels and make adjustments accordingly. To quickly become familiar with PromQL query syntax, check out our PromQL cheat sheet.

Use a for duration to avoid firing on brief spikes. For example, changing the CPU rule to for: 15m requires the condition to remain active for 15 minutes. Alertmanager can also group related alerts to reduce notification noise.

Load, validate, and reload your rules

To make Prometheus evaluate the recording and alerting rules, add rules.yml to the rule_files list in prometheus.yml:

global:
  evaluation_interval: 30s
 
rule_files:
  - rules.yml

In this example, evaluation_interval: 30s tells Prometheus to evaluate rules every 30 seconds. If you don't set it, the default is one minute. An individual rule group can set its own interval; rule evaluation and scraping run on separate schedules.

Before loading the rules, check both files with promtool:

promtool check rules rules.yml
promtool check config prometheus.yml

Run these commands with paths to your files. The checks can catch syntax and configuration errors, but they cannot tell you whether the metrics exist or whether your alert threshold is appropriate.

Next, reload the configuration by sending SIGHUP to the Prometheus process. If Prometheus was started with --web.enable-lifecycle, you can instead send a reload request:

curl -fsS -X POST http://localhost:9090/-/reload

Use your Prometheus address and authentication settings if they differ from the example. After the reload, open http://localhost:9090/rules and check that the service_requests and cpu_alerts groups are loaded without evaluation errors. Query service:http_requests:rate5m to inspect the recording rule's output. You can also check http://localhost:9090/alerts for pending or firing HighCPUUsage alerts.

Connect Prometheus to Alertmanager

Prometheus evaluates alerting rules, but it needs Alertmanager to route notifications. To send a notification when the HighCPUUsage alert fires, configure Prometheus to send alerts to Alertmanager, then configure Alertmanager with a notification receiver.

If Alertmanager is not installed, get it from the Prometheus downloads page. Add its address to your existing prometheus.yml configuration:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

In this example, localhost:9093 points to an Alertmanager running on the same host as Prometheus. If they run in separate containers or on different machines, use an address that Prometheus can reach. Check and reload the Prometheus configuration after adding the address, as described in the previous section.

Configure a notification receiver

Alertmanager needs a route to decide where to send alerts and a receiver to deliver the notifications. For example, create an alertmanager.yml file with a webhook receiver:

route:
  receiver: platform-webhook
  group_by: ['alertname', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
 
receivers:
  - name: platform-webhook
    webhook_configs:
      - url: 'https://alerts.example.com/prometheus'
        send_resolved: true

The route sends alerts to platform-webhook and groups them by alert name and job. It waits 30 seconds before sending the first notification for a new group. Later notification updates are checked every five minutes, and unchanged firing alerts can be repeated after four hours. With send_resolved: true, the receiver can also send notifications when alerts resolve.

Replace the example URL with an endpoint that accepts Alertmanager's webhook payload. For Slack, email, or PagerDuty, use the corresponding receiver configuration instead. Check the configuration, then start Alertmanager with it:

amtool check-config alertmanager.yml
alertmanager --config.file=alertmanager.yml

If Alertmanager is already running, reload its configuration with SIGHUP or an HTTP POST to its /-/reload endpoint. Open the Alertmanager UI, usually on port 9093, to check that firing alerts arrive. Then test delivery with a controlled alert and confirm that the receiver gets the notification. An alert appearing in the UI alone does not confirm delivery.

How the Prometheus alert lifecycle works

Prometheus tracks each alert by its labels and evaluates its state at regular intervals. For the HighCPUUsage rule, each instance moves through these states:

State	What it means
Inactive	The instance does not match the alert expression.
Pending	CPU usage is above 90%, but the condition has not remained active for five minutes.
Firing	CPU usage has remained above 90% across evaluations for five minutes.

If the condition clears while the alert is pending, the alert returns to inactive. Without a for duration, an alert can fire on the first matching evaluation. A firing alert becomes inactive when it stops matching, unless keep_firing_for extends it. Alertmanager may then send a resolved notification, depending on the receiver configuration.

Once an alert fires, notification timing depends on Alertmanager as well. For example, if HighCPUUsage becomes pending at 12:00 and stays active through each 30-second evaluation, it can fire at 12:05. With group_wait: 30s for a new alert group, Alertmanager can send the first notification around 12:05:30, plus delivery time. If the alert resolves before group_wait ends, no initial notification is sent for it.

For an existing group, group_interval controls when Alertmanager checks for changes, and repeat_interval controls when an unchanged firing alert is eligible for another notification. In the configuration above, those values are five minutes and four hours. Scrape timing, routing, and delivery can also affect when a notification arrives.

Reduce alert noise and flapping

An alert can repeatedly fire and resolve when a metric moves back and forth across a threshold. This is called flapping. The for: 5m setting in the CPU rule helps by waiting for high CPU usage to persist before firing.

You can also add keep_firing_for to the HighCPUUsage rule, alongside for:

for: 5m
keep_firing_for: 2m

With this setting, the alert keeps firing for two minutes after the condition was last met. This can prevent repeated resolutions during brief recoveries or gaps in data, but it also delays resolution.

Alertmanager provides other ways to manage notifications. Grouping combines related alerts into a notification, while Silences temporarily mute alerts that match selected labels. Inhibition suppresses matching notifications while another alert is active, such as an alert for a dependency failure. Silences and Inhibition do not stop Prometheus from evaluating rules.

Choose delays based on how quickly your team needs to respond. Review noisy alerts before increasing thresholds or wait times, and include enough context in annotations for the recipient to investigate.

Test and troubleshoot Prometheus rules

Use promtool test rules tests.yml to test rules against synthetic input series and expected results. Include the pending period, threshold boundaries, and recovery in your tests.

Problem	What to check
Rule missing from the Rules page	Check `rule_files`, file paths, permissions, and reload errors.
Recording rule returns no data	Run its expression directly and check metric names, label selectors, and recent samples.
Alert stays pending	Check whether it has remained active for the full `for` duration and whether changing labels keep creating new alert instances.
Alert does not fire	Check the expression's returned series, missing data, thresholds, and evaluation errors.
Alert fires but no notification arrives	Check the Prometheus-to-Alertmanager connection, matching routes, receiver errors, grouping delays, silences, and inhibition.
Rule evaluations are slow or fail	Inspect the expression's cost, cardinality, query timeout, and Prometheus resources. Reuse recording rules where they avoid repeated work.

Monitor prometheus_rule_evaluation_failures_total for evaluation failures. Increasing the evaluation interval reduces how often rules run; it does not itself increase the query execution timeout.

FAQs

Can I use PromQL functions in rules?

Yes, you can use any valid PromQL expression in both recording and alerting rules, including functions.

Does a pending alert send a notification?

Prometheus does not send pending alerts to Alertmanager for notification. The alert must fire first. Alertmanager can then apply routing, grouping delays, silences, and inhibition before delivering a notification.

Do I need Alertmanager for recording rules?

No. Prometheus evaluates and stores recording rule results itself. Alertmanager handles alert notifications, not recording rule evaluation.

Monitor and alert on metrics with SigNoz

While Prometheus offers powerful monitoring capabilities, setting up and managing rules can be complex. SigNoz Cloud provides a user-friendly alternative that simplifies monitoring and alerting for your applications and infrastructure.

With SigNoz, you can:

Easily set up alerts without complex PromQL queries
Visualize your metrics with intuitive dashboards
Correlate metrics, traces, and logs in a single platform
Scale your monitoring effortlessly with cloud-native architecture

SigNoz Cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.

You can also install and self-host SigNoz yourself since it is open-source. With 24,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.