What is the difference between Prometheus Rate vs Increase Functions

Prometheus rate() tells you how quickly a counter is increasing per second. increase() tells you how much it increased over a time period. Use rate() for questions such as "How many requests per second is this service handling?" and increase() for "How many requests did it handle in the last hour?"

This guide explains how the two functions relate, how to count events across instances, and how to handle counters that reset when an application restarts.

Understanding Prometheus rate() and increase()

A counter records a running total, such as requests processed, errors encountered, or bytes sent. Its value increases as events occur and normally resets to zero when the process restarts. The current value alone doesn't tell you how much activity occurred during a particular period.

Both rate() and increase() use samples from a time window to answer that question. They account for observed counter resets and extrapolate to the window's boundaries to estimate activity across the full period. The difference is the unit of the result.

How the rate() function works

To calculate a rate, pass a counter and a time window to rate():

rate(metric_name[time_range])

The time window determines which samples contribute to the average. The result is still expressed per second, even if you choose a longer window.

Real-World Examples of Rate() Usage

1. Calculating request rates in web servers:

   Copy

   rate(http_requests_total[5m])

   This query shows the average number of HTTP requests per second over the last five minutes for each matching series. If the estimated increase for a series is 600 requests over that window, its rate is 2 requests per second.

2. Monitoring network traffic:

   Copy

   rate(network_transmit_bytes_total[1h])

   This displays the average network transmission rate in bytes per second over the last hour for each series.

3. Analyzing CPU usage trends:

   Copy

   rate(node_cpu_seconds_total{mode="user"}[10m])

   This query shows CPU seconds spent in user mode per second, averaged over ten minutes for each CPU on each instance.

Deep Dive into the Increase() Function

The increase() function calculates the estimated total increase in a counter's value over a specified time range. Its syntax is similar to rate():

increase(http_requests_total[5m])

Increase() is particularly useful when you need to know the total change in a metric over time, rather than the rate of change. It's ideal for analyzing cumulative metrics or when you need to understand the absolute change in a value.

Practical Applications of Increase()

1. Measuring total requests over a specific time period:

   Copy

   increase(http_requests_total[24h])

   This query estimates the number of HTTP requests received in the last 24 hours for each matching series.

2. Analyzing cumulative errors in a system:

   Copy

   increase(error_count_total[1h])

   This displays the estimated number of errors that occurred in the last hour for each matching series.

3. Tracking resource consumption in cloud environments:

   Copy

   increase(cloud_cpu_usage_seconds_total[7d])

   If your application exposes a counter named cloud_cpu_usage_seconds_total, this query estimates the CPU time consumed over the past week for each matching series, in seconds.

Count total requests over a period

If your application exposes a request counter named http_requests_total, you can use it to estimate the requests handled in the last hour:

increase(http_requests_total[1h])

This returns a separate result for each label combination. If the counter has instance, method, and status labels, you'll see a count for each combination rather than one total for the application.

To combine those counts across all matching series, use sum():

sum(increase(http_requests_total[1h]))

This returns one estimated total for the last hour. To limit it to a particular service, add a label filter such as {job="api-server"} to the metric selector.

If you need a separate total for each instance, keep that label when aggregating:

sum by (instance) (increase(http_requests_total[1h]))

This combines methods and status codes within each instance, so you can compare how much traffic each instance handled. In each query, increase() runs before sum(). This lets Prometheus account for resets in individual series before combining their results.

Compare request counts with a previous period

To compare the last hour with the same hour yesterday, use offset 1d for the earlier window:

sum(increase(http_requests_total[1h])) /
sum(increase(http_requests_total[1h] offset 1d))

A result of 1.2 means the current count is 20% higher than yesterday's count. This comparison needs data for both periods and a nonzero count for the earlier period.

Handle counter resets after restarts

An application restart normally resets its counters to zero. Subtracting the first value in a window from the last can therefore give a misleading result. For example, a counter might rise from 100 to 150, reset, and then reach 20. Simple subtraction gives a decrease of 80, despite activity both before and after the restart.

rate() and increase() treat an observed drop in a counter as a reset. In this example, they account for the observed increases of 50 before the reset and 20 after it, then apply extrapolation for the query window. You don't need to restore the old counter value just to query activity across a restart.

However, these functions can only work with the samples Prometheus collected. Events between the last scrape and a restart may go unrecorded. Reset handling doesn't make a scraped counter an exact event ledger.

Check for unexpected resets

If a counter's results look unusual, check whether Prometheus detected resets during the same period:

resets(http_requests_total[1h])

This returns the number of detected resets in the last hour for each series. Repeated resets can be a reason to investigate application restarts or code that resets the counter unexpectedly. Use resets() for this check because changes() also counts ordinary increases.

When combining request rates across instances, follow the same order as the request-total examples:

sum(rate(http_requests_total[5m]))

Calculating the rate for each series first preserves its reset handling. Aggregating raw counters before calculating a rate can hide a reset when another instance's counter increases at the same time.

rate() vs increase(): when to use each

Use rate() when the result should be expressed per second, such as requests per second, errors per second, or bytes per second. Use increase() when the result should be a count or amount over a specified period.

The same distinction applies to alerts. An alert for throughput above 100 requests per second calls for rate(). An alert for more than 50 errors in an hour calls for increase(). Choose the function that matches the threshold you want to express.

For recording rules, Prometheus recommends recording rates so the results stay on a consistent per-second basis. You can multiply a rate by a duration when you need an estimated total for that same window.

Best Practices for Using Rate() and Increase()

1. Choose appropriate time ranges: Shorter windows make rates more responsive to recent activity but can be noisier. Longer windows smooth out fluctuations but may hide short-term issues.
2. Handle counter resets: Both functions account for observed resets. They cannot recover events that occurred between the last scrape and a restart.
3. Combine with other PromQL functions: Use sum(), avg(), or max() around the rate or increase to combine results across series. Choose the aggregation that matches the total, average, or maximum you need.
4. Optimize query performance: Avoid unnecessarily long windows, which require Prometheus to process more samples. Keep the window long enough to answer your monitoring question.

Key Takeaways

• Rate() calculates per-second average change; increase() shows total change over time.
• Both functions are essential for analyzing counter metrics in Prometheus.
• Choose between rate() and increase() based on your specific monitoring needs.
• Proper use of these functions is crucial for accurate data interpretation and alerting.

For more on Prometheus, the Prometheus alternatives roundup goes broader, and the Prometheus vs Grafana, OpenTelemetry vs Prometheus, Loki vs Prometheus, and Prometheus vs InfluxDB comparisons cover adjacent matchups.

FAQs

What is the main difference between rate() and increase() in Prometheus?

Rate() calculates the per-second rate of change, while increase() computes the total change over a specified time range.

Can I use rate() and increase() with gauge metrics?

No, these functions are designed for counter metrics. Gauge metrics should be analyzed using different PromQL functions.

How do rate() and increase() handle counter resets?

Both functions automatically handle counter resets by calculating the rate or increase based on the available data points.

What time range should I use with rate() and increase() functions?

The ideal time range depends on your specific use case. Generally, use shorter ranges (e.g., 5m) for more granular data and longer ranges (e.g., 1h) for smoother trends.

Enhance Your Monitoring with SigNoz Cloud

While Prometheus offers powerful monitoring capabilities, managing retention and scaling can become challenging as your infrastructure grows. SigNoz Cloud provides a comprehensive monitoring solution that builds upon Prometheus' strengths while addressing its limitations.

SigNoz Cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.

You can also install and self-host SigNoz yourself since it is open-source. With 24,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.

With SigNoz Cloud, you can:

• Scale your monitoring infrastructure effortlessly
• Access advanced querying and visualization capabilities
• Benefit from integrated tracing and logging alongside metrics.
• Get high performance with the clickhouse database
• Take advantage of SigNoz Cloud's exceptional exception monitoring capabilities