Kubernetes CPU Alerts - Monitor Node and Container Usage

Kubernetes CPU pressure shows up at two levels. A node running near its allocatable CPU throttles the workloads on it and leaves little room to absorb a reschedule. A container running near its own CPU limit gets throttled even when the node has spare capacity. This guide sets up one alert for each.

Prerequisites

• An instance of SigNoz (either Cloud or Self-Hosted)
• Kubernetes metrics reaching SigNoz through the k8s-infra Helm chart or a manually configured collector
• A notification channel to receive the alert

The k8s-infra chart collects every metric used below with its default values. If you run your own collector, check the receiver notes in each section.

Alert on node CPU utilization

Kubernetes reports node CPU in cores. Divide usage by the node's allocatable capacity to get a percentage.

k8s.node.allocatable_cpu comes from the k8s_cluster receiver, which is disabled by default upstream. The k8s-infra chart enables it (presets.clusterMetrics.allocatableTypesToReport includes cpu). On a manually configured collector, add the opt-in to your existing k8s_cluster receiver block:

receivers:
  k8s_cluster:
    allocatable_types_to_report: [cpu, memory]

Restart the collector after the change, then confirm the metric appears in Metrics before you build the alert.

Step 1: Build the utilization query

1. Go to Alerts in the left navigation and click New Alert, then select Metric based Alert.
2. Under Define the Query, configure query A:
   • Metrics: k8s.node.cpu.usage
   • Aggregate within time series: Avg
   • Aggregate across time series: Max by k8s.node.name. On a multi-cluster setup, add k8s.cluster.name as a second key. The k8s-infra chart attaches it from the clusterName value; a manually configured collector does not emit it unless you add it yourself.
3. Add query B and configure it the same way, with k8s.node.allocatable_cpu as the metric.
4. Click Add Formula and set the expression to (A / B) * 100.
5. Set the unit dropdown above the chart to Percent (0 - 100) so the threshold and the alert message read as percentages.

Click the eye icon on queries A and B to hide them from the chart. Both still feed the formula, and the preview drops to one line per node.

A node can read above 100%. k8s.node.cpu.usage covers everything on the node, kubelet and system processes included, while allocatable CPU excludes the capacity reserved for them. Where kube-reserved or system-reserved takes a large share, a busy node crosses 100% of allocatable. That reading is correct.

Step 2: Set the alert condition

Under Set Alert Conditions, read the row as a sentence: send a notification when F1 is ABOVE the threshold ALL THE TIME during the Last 5 Minutes, evaluated as a Rolling window. Then set the threshold row to warning on value > 60, and pick a channel under send to on that same row.

ALL THE TIME means every data point in the window must exceed 60% before the alert fires, which filters out short spikes from a batch job or an image pull.

Treat 60% as a starting point. The headroom you need depends on cluster size, the CPU requests your pods declare, and your scheduling constraints. A three-node cluster at 60% cannot absorb a node loss; a thirty-node cluster can. Raise the threshold toward 80% to alert closer to saturation, or lower it for more warning.

The k8s-infra chart scrapes kubelet and cluster metrics every 30 seconds, so a 5 minute window holds roughly 10 data points per node. If you shorten the window, keep it well above the scrape interval. Evaluation frequency sits under Advanced Options and defaults to 1 minute.

Step 3: Name the alert and group the notifications

1. Under Notification Settings, set Group alerts by to k8s.node.name so each node gets its own notification instead of one combined message.
2. At the top of the form, name the alert and add a severity label next to it. The threshold name you set in Step 2 (warning) is what routing policies match on; the severity label is separate and drives your own filtering.
3. Click Test Notification to confirm the channel works, then Save Alert Rule.

Alert on container CPU against its limit

Node-level alerts miss a container that is throttled while the node sits idle. For that, alert on how much of its own CPU limit a container uses.

k8s.container.cpu_limit_utilization comes from the kubeletstats receiver and reports a ratio from 0 to 1. The receiver emits it only for containers that define a CPU limit.

The metric is disabled by default upstream. The k8s-infra chart turns it on; on a manually configured collector, enable it on your existing kubeletstats receiver:

receivers:
  # On Collector v0.152.0 and newer, use "kubelet_stats" to avoid a deprecation warning.
  kubeletstats:
    metrics:
      k8s.container.cpu_limit_utilization:
        enabled: true

Utilization metrics also need wider RBAC. Give the collector's service account get on nodes/pods alongside the nodes/stats permission it already has, or the receiver collects nothing. See the kubeletstats RBAC reference. Restart the collector, then check Metrics for the metric before building the alert.

Step 1: Build the query

1. Create a new Metric based Alert.
2. Configure query A:
   • Metrics: k8s.container.cpu_limit_utilization
   • Aggregate within time series: Avg
   • Aggregate across time series: Max by k8s.cluster.name, k8s.namespace.name, k8s.pod.name, and k8s.container.name
3. Set the query unit to Percent (0.0-1.0).

Avg smooths spikes inside the evaluation window. Max reports the busiest container in each group, so quiet neighbors cannot mask a throttled one.

Step 2: Set the alert condition

Send a notification when A is ABOVE the threshold ALL THE TIME during the Last 5 Minutes, with the threshold row set to warning on value > 0.8. Pick a channel under send to on the threshold row, the same as the node alert, or turn on routing policies.

The metric is a ratio, so 0.8 means 80% of the container's limit. Group notifications by k8s.namespace.name and k8s.pod.name to keep each firing pod separate.

Validate

• The chart preview in the alert form plots one series per node or container, with the threshold drawn across it. Node values above 100% are legitimate on nodes with a large reserved share; values near zero across every series usually mean a query problem.
• After saving, open Alerts and confirm the rule is enabled. A rule that has never evaluated shows no state.
• Check Alerts > Triggered Alerts once a threshold is crossed, and use alert history to see past firings and how long each lasted.

Next steps

• Apply the same pattern to memory with k8s.node.memory.usage over k8s.node.allocatable_memory, or k8s.container.memory_limit_utilization for containers
• Read time aggregation best practices before tuning windows and thresholds
• Watch nodes and pods in Infrastructure Monitoring

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.