Every query on this page goes into the PromQL tab of a metric alert. Copy one, replace the metric and label values with your own, then set a threshold.
Where PromQL fits in an alert
- Go to Alerts in the left navigation and click New Alert.
- Select Metric based Alert.
- Under Define the Query, open the PromQL tab and paste the query.
- Set the threshold, the match type, and the evaluation window under Alert Configuration.
SigNoz saves the rule as a PromQL rule. The rest of the form behaves the same as it does for a Query Builder alert: threshold tiers, match type, evaluation window, the Alert when data stops coming option, re-notification, and notification grouping. The labels your query returns become the alert labels, so the by (...) clause decides how many separate alerts the rule can produce.
When to use PromQL instead of the Query Builder
| Use PromQL when | Reason |
|---|---|
| You need explicit control over label matching in a join | on (...) selects the matching labels, and group_left (label) includes a label from the right side in a many-to-one join. |
| You already have a Prometheus alerting rule | The expression works as written, apart from the rules below. |
You need forecasting with predict_linear | Query Builder does not expose an equivalent forecasting function. |
The Query Builder covers most alerts, including alerts that combine several metrics in a formula. Use PromQL for the cases above.
Rules that apply to every query here
- Quote any name that contains a dot, and put the metric name first inside the braces.
rate({"http.server.request.duration.count"}[5m])returns data.rate(http_server_request_duration_count[5m])returns nothing, because SigNoz stores the original dotted name. See the PromQL UTF-8 syntax guide. - PromQL reads only metrics stored with cumulative or unspecified temporality. A query on a delta metric returns no series. SigNoz span metrics such as
signoz_calls_totalandsignoz_latency.bucketare delta in a default setup, so alert on those with the Query Builder instead. Open the metric in Metrics to read its temporality. - Histogram parts use dot suffixes.
.bucket,.count, and.sum, not_bucket,_count, and_sum. - Regex matchers are fully anchored.
=~"5.."matches a three-character value that starts with5. For a substring match, write=~".*5.*". - Rule evaluation runs the query at fixed 60-second steps across the alert evaluation window. A range selector such as
[5m]is a separate lookback: at each step, the function reads the preceding five minutes of samples.
Service health
A service stopped reporting
absent_over_time() returns 1 when the selector matched no sample across its range, and returns nothing while data flows.
absent_over_time({"http.server.request.duration.count", "service.name"="checkout"}[10m])Threshold above 0, match type at least once. The gap you accept is the range in the selector, not the evaluation window, so change [10m] to change how long a silence you tolerate.
A new rule can fire on its first evaluation. The evaluation window can start before the service sent its first sample, and at least once needs only one step in the window where the selector matched nothing. Create the rule after the service has reported for one full evaluation window.
Do not use bare absent() here. At each step, SigNoz looks back 5 minutes for the latest sample, so absent() returns 1 only after 5 minutes of silence. You cannot set that gap in the query. absent_over_time() takes the gap from its range.
Keep every label matcher an equality match. Both functions copy only equality matchers into their output, so a regex matcher leaves the alert with no service name in the message.
This fires when no requests reach the service. A crashed service and an idle service look identical to it. To tell them apart, point the selector at a metric the process emits regardless of traffic, such as a runtime or JVM metric.
Request rate dropped to near zero
sum by ("service.name") (rate({"http.server.request.duration.count"}[5m]))rate() returns requests per second, so read the normal rate in Metrics before you pick a number. A service that handles 40 requests a second and one that handles 0.1 need different thresholds. Set the threshold to a fraction of the normal rate, comparison below, match type all the times. Every other recipe here compares above, which is also the form default, so this one needs changing by hand.
A service that stops emitting produces no series, and a threshold alone never fires on a missing series. Alert when data stops coming covers that only when the whole query goes silent. SigNoz asks whether the result holds any series at all, so this grouped query still counts as having data while one other service reports.
absent_over_time can detect one named service when the selector includes an equality matcher such as "service.name"="checkout". It cannot identify a missing service from a selector that matches several services because any remaining series makes the function return no result. Create one rule per expected service, or follow No Data Alerts for Groups.
Server error ratio above a percentage
100 * sum by ("service.name") (
rate({"http.server.request.duration.count", "http.response.status_code"=~"5.."}[5m])
)
/
sum by ("service.name") (
rate({"http.server.request.duration.count"}[5m])
)Set the threshold to above 1 for a 1% error budget, and set the y-axis unit to Percent (0 - 100). Both sides group by service.name alone, so / pairs each numerator series with its own denominator. Add an on (...) clause only when the two sides carry different labels, to name the labels that have to match.
A service with no 5xx series has no numerator, so the query returns nothing for it. If no service has a 5xx series, the whole query returns no series. Leave Alert when data stops coming off for this rule. Otherwise it can fire while every service is healthy.
One failed request out of three puts a service at 33%, so a quiet service crosses any threshold you set. Filter those series out with a floor on the denominator:
100 * sum by ("service.name") (
rate({"http.server.request.duration.count", "http.response.status_code"=~"5.."}[5m])
)
/
(
sum by ("service.name") (rate({"http.server.request.duration.count"}[5m])) > 0.5
)The floor counts requests per second, not requests per window. A floor of 0.5 drops any service under one request every two seconds. Set it below the normal rate of the quietest service you want covered. A floor above that rate mutes the alert for that service.
Latency
Use the Query Builder for the p95 and p99 examples below. It computes these percentiles in ClickHouse. Use PromQL for a quantile the Query Builder does not offer, such as histogram_quantile(0.73, ...).
p99 latency for each service
histogram_quantile(0.99,
sum by (le, "service.name") (rate({"http.server.request.duration.bucket"}[5m]))
)le carries the bucket boundary and must stay in the by (...) clause. Drop it and histogram_quantile has nothing to interpolate over.
The unit comes from the metric, not from the alert form. The instrument records http.server.request.duration in seconds, so a threshold of 1.2 means 1.2 seconds. Set the y-axis unit to Seconds (s) so the alert message reads the same way.
p95 latency for one endpoint
histogram_quantile(0.95,
sum by (le) (rate({"http.server.request.duration.bucket", "http.route"="/api/v1/search"}[5m]))
)Infrastructure
Node CPU against allocatable capacity
100 * max by ("k8s.node.name") ({"k8s.node.cpu.usage"})
/
max by ("k8s.node.name") ({"k8s.node.allocatable_cpu"})Both metrics report CPU in cores, so the ratio is the fraction of the node in use. k8s.node.allocatable_cpu needs allocatable_types_to_report on the k8s_cluster receiver. For the receiver settings and the equivalent Query Builder alert, see Kubernetes CPU Alerts.
Pod memory against its limit
max by ("k8s.namespace.name", "k8s.pod.name") ({"k8s.pod.memory_limit_utilization"})The metric is already a ratio, so a threshold of 0.9 means 90% of the total container limits in the pod. Values above 1 do occur, so do not treat 1 as a ceiling. The receiver drops the metric for any pod where a container declares no memory limit. The kubeletstats receiver does not emit this metric by default, so enable k8s.pod.memory_limit_utilization in its metrics section.
For per-container granularity, swap in k8s.container.memory_limit_utilization and add k8s.container.name to the by (...) clause. This metric is also off by default.
Memory use will cross the limit in four hours
predict_linear({"process.memory.usage", "service.name"="checkout"}[6h], 4 * 3600)predict_linear fits a straight line over the last six hours and projects it forward by the second argument, in seconds. process.memory.usage measures one process, so this works when the container runs a single process. Set the threshold to the container memory limit in bytes and the y-axis unit to Bytes (B).
Two samples in the range are enough for a fit. The query starts projecting minutes after a fresh install, and those early fits swing wildly. Six hours is the history you want, not a warm-up the query enforces. Use match type all the times so one wild projection does not page anyone.
Queues and dependencies
Kafka consumer group lag
max by (topic, partition) ({"kafka.partition.current_offset"})
- on (topic, partition) group_right ()
max by (group, topic, partition) ({"kafka.consumer_group.offset"})The Kafka metrics receiver emits both metrics. group_right () allows many rows on the right side for each matched key, which is what happens when several consumer groups read one partition. Threshold above 1000, match type all the times.
The same receiver also emits kafka.consumer_group.lag with the group, topic, and partition labels. Alert on it directly when you do not need the join.
Log processor queue near capacity
100 * max by ("service.name", "otel.component.name") ({"otel.sdk.processor.log.queue.size"})
/
max by ("service.name", "otel.component.name") ({"otel.sdk.processor.log.queue.capacity"})A queue that stays near 100% drops telemetry. Threshold above 80 with match type all the times.
These are OpenTelemetry SDK metrics, so this watches the log processor inside an instrumented service rather than a Collector. otel.component.name identifies the processor instance and belongs in the by (...) clause on both sides. Drop it and a service with two processors pairs the size of one against the capacity of the other.
For the Collector's own exporter queues, alert on otelcol_exporter_queue_size against otelcol_exporter_queue_capacity instead. See Collector internal telemetry.
Alert settings that go with these queries
| Setting | Recommendation |
|---|---|
| Match type | At least once for spikes such as an error ratio. All the times for sustained conditions such as saturation or a rate that collapsed. |
| Evaluation window | The time range the match type reads. The rule runs the query at 60-second steps across the window, so a 5-minute window has six evaluation steps, because it includes both ends. Each one computes its own [5m] lookback, so the window and the selector range are independent. |
| Alert when data stops coming | On for anything that means "a thing stopped". The time period next to it counts in minutes from the last result that had data, so a value of 15 waits 15 evaluations at a 1m frequency. A rule that has never returned data fires on its first evaluation. |
| Threshold value | Keep the number in the threshold field, not in the query. A comparison such as > 0.9 inside the query removes every series below 0.9 before SigNoz applies the threshold, so the chart hides them too. |
| Y-axis unit | Match the raw unit the query produces: seconds for OpenTelemetry duration histograms, a 0 to 1 ratio for Kubernetes utilization metrics, bytes for memory. |
| Rate thresholds | rate() and irate() return a per-second value. A threshold of 1 on a request rate means one request per second, not one request per window. |
| Notification grouping | Group by the same labels the query returns. Grouping by a label the query dropped merges every series into one notification. |
Troubleshooting
The chart preview is empty
Work through these in order:
- Check the metric temporality in Metrics. A delta metric returns nothing through PromQL. Rebuild the alert with the Query Builder.
- Confirm the metric name sits first inside the braces, and quote it when it contains a dot.
- Confirm the histogram part uses a dot suffix, such as
.bucket. - Replace each regex matcher with an equality matcher. A matcher such as
=~"checkout"fails oncheckout-api, because SigNoz anchors the pattern. - Remove the label matchers one at a time. A misspelled label value returns an empty result with no error.
The quantile is empty but the bucket metric has data
le is missing from the by (...) clause, or a processing pipeline renamed it. Check the label list for the metric in Metrics.
The alert fires once for every pod that restarts
A new pod is a new series, and one noisy point is enough to satisfy a match type of at least once. Switch to all the times so the condition has to hold across the whole window. Then group by a label that survives the restart, such as k8s.deployment.name, instead of k8s.pod.name.
Next steps
- Metric-based alerts for every configuration field on the alert form
- PromQL UTF-8 syntax guide for the quoting rules and migration examples
- Alert notification channels to route the alerts you create
